[==========] Running 7 tests from 2 test suites.
[----------] Global test environment set-up.
[----------] 5 tests from LeaderRebalancerTest
[ RUN ] LeaderRebalancerTest.FunctionalTestForDivided
WARNING: Logging before InitGoogleLogging() is written to STDERR
I20260129 02:57:41.064646 19276 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.18.211.62:39923
I20260129 02:57:41.066277 19276 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260129 02:57:41.067287 19276 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260129 02:57:41.084362 19285 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260129 02:57:41.084391 19283 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260129 02:57:41.084542 19282 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260129 02:57:41.085528 19276 server_base.cc:1060] running on GCE node
I20260129 02:57:42.285449 19276 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260129 02:57:42.285737 19276 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260129 02:57:42.285935 19276 hybrid_clock.cc:648] HybridClock initialized: now 1769655462285916 us; error 0 us; skew 500 ppm
I20260129 02:57:42.286791 19276 server_base.cc:860] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260129 02:57:42.292131 19276 webserver.cc:492] Webserver started at http://127.18.211.62:42165/ using document root <none> and password file <none>
I20260129 02:57:42.293041 19276 fs_manager.cc:362] Metadata directory not provided
I20260129 02:57:42.293288 19276 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260129 02:57:42.293730 19276 server_base.cc:908] This appears to be a new deployment of Kudu; creating new FS layout
I20260129 02:57:42.299821 19276 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskrTP_Jo/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1769655460890924-19276-0/minicluster-data/master-0-root/instance:
uuid: "ec7858395ca04e12bff51324a29a761f"
format_stamp: "Formatted at 2026-01-29 02:57:42 on dist-test-slave-vngr"
I20260129 02:57:42.307734 19276 fs_manager.cc:696] Time spent creating directory manager: real 0.007s user 0.006s sys 0.000s
I20260129 02:57:42.313073 19291 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260129 02:57:42.314261 19276 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.005s sys 0.000s
I20260129 02:57:42.314616 19276 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskrTP_Jo/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1769655460890924-19276-0/minicluster-data/master-0-root
uuid: "ec7858395ca04e12bff51324a29a761f"
format_stamp: "Formatted at 2026-01-29 02:57:42 on dist-test-slave-vngr"
I20260129 02:57:42.314962 19276 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrTP_Jo/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1769655460890924-19276-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-taskrTP_Jo/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1769655460890924-19276-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-taskrTP_Jo/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1769655460890924-19276-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260129 02:57:42.376797 19276 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260129 02:57:42.378500 19276 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260129 02:57:42.378955 19276 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260129 02:57:42.450619 19276 rpc_server.cc:307] RPC server started. Bound to: 127.18.211.62:39923
I20260129 02:57:42.450703 19342 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.18.211.62:39923 every 8 connection(s)
I20260129 02:57:42.458679 19343 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260129 02:57:42.480495 19343 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ec7858395ca04e12bff51324a29a761f: Bootstrap starting.
I20260129 02:57:42.486400 19343 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ec7858395ca04e12bff51324a29a761f: Neither blocks nor log segments found. Creating new log.
I20260129 02:57:42.488310 19343 log.cc:826] T 00000000000000000000000000000000 P ec7858395ca04e12bff51324a29a761f: Log is configured to *not* fsync() on all Append() calls
I20260129 02:57:42.492754 19343 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ec7858395ca04e12bff51324a29a761f: No bootstrap required, opened a new log
I20260129 02:57:42.513236 19343 raft_consensus.cc:359] T 00000000000000000000000000000000 P ec7858395ca04e12bff51324a29a761f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ec7858395ca04e12bff51324a29a761f" member_type: VOTER }
I20260129 02:57:42.513793 19343 raft_consensus.cc:385] T 00000000000000000000000000000000 P ec7858395ca04e12bff51324a29a761f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260129 02:57:42.514078 19343 raft_consensus.cc:740] T 00000000000000000000000000000000 P ec7858395ca04e12bff51324a29a761f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ec7858395ca04e12bff51324a29a761f, State: Initialized, Role: FOLLOWER
I20260129 02:57:42.514884 19343 consensus_queue.cc:260] T 00000000000000000000000000000000 P ec7858395ca04e12bff51324a29a761f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ec7858395ca04e12bff51324a29a761f" member_type: VOTER }
I20260129 02:57:42.515396 19343 raft_consensus.cc:399] T 00000000000000000000000000000000 P ec7858395ca04e12bff51324a29a761f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260129 02:57:42.515678 19343 raft_consensus.cc:493] T 00000000000000000000000000000000 P ec7858395ca04e12bff51324a29a761f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260129 02:57:42.516029 19343 raft_consensus.cc:3060] T 00000000000000000000000000000000 P ec7858395ca04e12bff51324a29a761f [term 0 FOLLOWER]: Advancing to term 1
I20260129 02:57:42.520105 19343 raft_consensus.cc:515] T 00000000000000000000000000000000 P ec7858395ca04e12bff51324a29a761f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ec7858395ca04e12bff51324a29a761f" member_type: VOTER }
I20260129 02:57:42.520877 19343 leader_election.cc:304] T 00000000000000000000000000000000 P ec7858395ca04e12bff51324a29a761f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ec7858395ca04e12bff51324a29a761f; no voters:
I20260129 02:57:42.522596 19343 leader_election.cc:290] T 00000000000000000000000000000000 P ec7858395ca04e12bff51324a29a761f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260129 02:57:42.522948 19346 raft_consensus.cc:2804] T 00000000000000000000000000000000 P ec7858395ca04e12bff51324a29a761f [term 1 FOLLOWER]: Leader election won for term 1
I20260129 02:57:42.525275 19346 raft_consensus.cc:697] T 00000000000000000000000000000000 P ec7858395ca04e12bff51324a29a761f [term 1 LEADER]: Becoming Leader. State: Replica: ec7858395ca04e12bff51324a29a761f, State: Running, Role: LEADER
I20260129 02:57:42.526261 19346 consensus_queue.cc:237] T 00000000000000000000000000000000 P ec7858395ca04e12bff51324a29a761f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ec7858395ca04e12bff51324a29a761f" member_type: VOTER }
I20260129 02:57:42.527326 19343 sys_catalog.cc:565] T 00000000000000000000000000000000 P ec7858395ca04e12bff51324a29a761f [sys.catalog]: configured and running, proceeding with master startup.
I20260129 02:57:42.537211 19347 sys_catalog.cc:455] T 00000000000000000000000000000000 P ec7858395ca04e12bff51324a29a761f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ec7858395ca04e12bff51324a29a761f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ec7858395ca04e12bff51324a29a761f" member_type: VOTER } }
I20260129 02:57:42.537292 19348 sys_catalog.cc:455] T 00000000000000000000000000000000 P ec7858395ca04e12bff51324a29a761f [sys.catalog]: SysCatalogTable state changed. Reason: New leader ec7858395ca04e12bff51324a29a761f. Latest consensus state: current_term: 1 leader_uuid: "ec7858395ca04e12bff51324a29a761f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ec7858395ca04e12bff51324a29a761f" member_type: VOTER } }
I20260129 02:57:42.537978 19347 sys_catalog.cc:458] T 00000000000000000000000000000000 P ec7858395ca04e12bff51324a29a761f [sys.catalog]: This master's current role is: LEADER
I20260129 02:57:42.538120 19348 sys_catalog.cc:458] T 00000000000000000000000000000000 P ec7858395ca04e12bff51324a29a761f [sys.catalog]: This master's current role is: LEADER
I20260129 02:57:42.541615 19355 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20260129 02:57:42.552680 19355 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20260129 02:57:42.554742 19276 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20260129 02:57:42.570181 19355 catalog_manager.cc:1357] Generated new cluster ID: 573f0ea08837426cb279adde4adbc3bd
I20260129 02:57:42.570479 19355 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20260129 02:57:42.593043 19355 catalog_manager.cc:1380] Generated new certificate authority record
I20260129 02:57:42.594545 19355 catalog_manager.cc:1514] Loading token signing keys...
I20260129 02:57:42.614825 19355 catalog_manager.cc:6027] T 00000000000000000000000000000000 P ec7858395ca04e12bff51324a29a761f: Generated new TSK 0
I20260129 02:57:42.615813 19355 catalog_manager.cc:1524] Initializing in-progress tserver states...
I20260129 02:57:42.621763 19276 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260129 02:57:42.631238 19365 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260129 02:57:42.632877 19366 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260129 02:57:42.638067 19276 server_base.cc:1060] running on GCE node
W20260129 02:57:42.638084 19368 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260129 02:57:42.639130 19276 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260129 02:57:42.639375 19276 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260129 02:57:42.639546 19276 hybrid_clock.cc:648] HybridClock initialized: now 1769655462639525 us; error 0 us; skew 500 ppm
I20260129 02:57:42.640169 19276 server_base.cc:860] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260129 02:57:42.644552 19276 webserver.cc:492] Webserver started at http://127.18.211.1:37225/ using document root <none> and password file <none>
I20260129 02:57:42.645085 19276 fs_manager.cc:362] Metadata directory not provided
I20260129 02:57:42.645300 19276 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260129 02:57:42.645586 19276 server_base.cc:908] This appears to be a new deployment of Kudu; creating new FS layout
I20260129 02:57:42.646701 19276 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskrTP_Jo/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1769655460890924-19276-0/minicluster-data/ts-0-root/instance:
uuid: "a6aef613bf534ff9ae6ddbba06bb3cf1"
format_stamp: "Formatted at 2026-01-29 02:57:42 on dist-test-slave-vngr"
I20260129 02:57:42.652271 19276 fs_manager.cc:696] Time spent creating directory manager: real 0.005s user 0.001s sys 0.006s
I20260129 02:57:42.656912 19373 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260129 02:57:42.657774 19276 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.002s sys 0.003s
I20260129 02:57:42.658056 19276 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskrTP_Jo/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1769655460890924-19276-0/minicluster-data/ts-0-root
uuid: "a6aef613bf534ff9ae6ddbba06bb3cf1"
format_stamp: "Formatted at 2026-01-29 02:57:42 on dist-test-slave-vngr"
I20260129 02:57:42.658349 19276 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrTP_Jo/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1769655460890924-19276-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-taskrTP_Jo/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1769655460890924-19276-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-taskrTP_Jo/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1769655460890924-19276-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260129 02:57:42.690732 19276 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260129 02:57:42.692121 19276 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260129 02:57:42.694986 19276 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260129 02:57:42.699692 19276 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260129 02:57:42.699966 19276 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.001s
I20260129 02:57:42.700251 19276 ts_tablet_manager.cc:616] Registered 0 tablets
I20260129 02:57:42.700412 19276 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260129 02:57:42.805619 19276 rpc_server.cc:307] RPC server started. Bound to: 127.18.211.1:45187
I20260129 02:57:42.805703 19435 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.18.211.1:45187 every 8 connection(s)
I20260129 02:57:42.812990 19276 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260129 02:57:42.822373 19440 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260129 02:57:42.824047 19441 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260129 02:57:42.828997 19276 server_base.cc:1060] running on GCE node
W20260129 02:57:42.829815 19443 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260129 02:57:42.830731 19276 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260129 02:57:42.830915 19276 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260129 02:57:42.831112 19276 hybrid_clock.cc:648] HybridClock initialized: now 1769655462831082 us; error 0 us; skew 500 ppm
I20260129 02:57:42.831693 19276 server_base.cc:860] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260129 02:57:42.834540 19276 webserver.cc:492] Webserver started at http://127.18.211.2:42441/ using document root <none> and password file <none>
I20260129 02:57:42.835208 19276 fs_manager.cc:362] Metadata directory not provided
I20260129 02:57:42.835464 19276 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260129 02:57:42.835803 19276 server_base.cc:908] This appears to be a new deployment of Kudu; creating new FS layout
I20260129 02:57:42.837212 19276 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskrTP_Jo/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1769655460890924-19276-0/minicluster-data/ts-1-root/instance:
uuid: "895c9e1595924e769a25faeed7e34644"
format_stamp: "Formatted at 2026-01-29 02:57:42 on dist-test-slave-vngr"
I20260129 02:57:42.838728 19436 heartbeater.cc:344] Connected to a master server at 127.18.211.62:39923
I20260129 02:57:42.839247 19436 heartbeater.cc:461] Registering TS with master...
I20260129 02:57:42.840464 19436 heartbeater.cc:507] Master 127.18.211.62:39923 requested a full tablet report, sending...
I20260129 02:57:42.844094 19308 ts_manager.cc:194] Registered new tserver with Master: a6aef613bf534ff9ae6ddbba06bb3cf1 (127.18.211.1:45187)
I20260129 02:57:42.847090 19308 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:36546
I20260129 02:57:42.849172 19276 fs_manager.cc:696] Time spent creating directory manager: real 0.011s user 0.005s sys 0.008s
I20260129 02:57:42.854326 19448 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260129 02:57:42.855242 19276 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.003s sys 0.002s
I20260129 02:57:42.855516 19276 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskrTP_Jo/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1769655460890924-19276-0/minicluster-data/ts-1-root
uuid: "895c9e1595924e769a25faeed7e34644"
format_stamp: "Formatted at 2026-01-29 02:57:42 on dist-test-slave-vngr"
I20260129 02:57:42.855788 19276 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrTP_Jo/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1769655460890924-19276-0/minicluster-data/ts-1-root
metadata directory: /tmp/dist-test-taskrTP_Jo/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1769655460890924-19276-0/minicluster-data/ts-1-root
1 data directories: /tmp/dist-test-taskrTP_Jo/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1769655460890924-19276-0/minicluster-data/ts-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260129 02:57:42.874534 19276 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260129 02:57:42.875792 19276 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260129 02:57:42.878388 19276 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260129 02:57:42.881278 19276 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260129 02:57:42.881469 19276 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260129 02:57:42.881754 19276 ts_tablet_manager.cc:616] Registered 0 tablets
I20260129 02:57:42.881909 19276 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260129 02:57:42.983884 19276 rpc_server.cc:307] RPC server started. Bound to: 127.18.211.2:40083
I20260129 02:57:42.983996 19510 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.18.211.2:40083 every 8 connection(s)
I20260129 02:57:42.990092 19276 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260129 02:57:43.000747 19515 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260129 02:57:43.005299 19511 heartbeater.cc:344] Connected to a master server at 127.18.211.62:39923
I20260129 02:57:43.005944 19511 heartbeater.cc:461] Registering TS with master...
I20260129 02:57:43.007014 19511 heartbeater.cc:507] Master 127.18.211.62:39923 requested a full tablet report, sending...
I20260129 02:57:43.008251 19276 server_base.cc:1060] running on GCE node
W20260129 02:57:43.008636 19514 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260129 02:57:43.009601 19517 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260129 02:57:43.009646 19308 ts_manager.cc:194] Registered new tserver with Master: 895c9e1595924e769a25faeed7e34644 (127.18.211.2:40083)
I20260129 02:57:43.010802 19276 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260129 02:57:43.011046 19276 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260129 02:57:43.011159 19308 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:36554
I20260129 02:57:43.011258 19276 hybrid_clock.cc:648] HybridClock initialized: now 1769655463011240 us; error 0 us; skew 500 ppm
I20260129 02:57:43.012027 19276 server_base.cc:860] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260129 02:57:43.014811 19276 webserver.cc:492] Webserver started at http://127.18.211.3:40941/ using document root <none> and password file <none>
I20260129 02:57:43.015439 19276 fs_manager.cc:362] Metadata directory not provided
I20260129 02:57:43.015690 19276 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260129 02:57:43.016011 19276 server_base.cc:908] This appears to be a new deployment of Kudu; creating new FS layout
I20260129 02:57:43.017473 19276 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskrTP_Jo/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1769655460890924-19276-0/minicluster-data/ts-2-root/instance:
uuid: "309a53f12a534385a7e5a88fb373722a"
format_stamp: "Formatted at 2026-01-29 02:57:43 on dist-test-slave-vngr"
I20260129 02:57:43.023047 19276 fs_manager.cc:696] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260129 02:57:43.027082 19522 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260129 02:57:43.027891 19276 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260129 02:57:43.028215 19276 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskrTP_Jo/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1769655460890924-19276-0/minicluster-data/ts-2-root
uuid: "309a53f12a534385a7e5a88fb373722a"
format_stamp: "Formatted at 2026-01-29 02:57:43 on dist-test-slave-vngr"
I20260129 02:57:43.028575 19276 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrTP_Jo/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1769655460890924-19276-0/minicluster-data/ts-2-root
metadata directory: /tmp/dist-test-taskrTP_Jo/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1769655460890924-19276-0/minicluster-data/ts-2-root
1 data directories: /tmp/dist-test-taskrTP_Jo/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1769655460890924-19276-0/minicluster-data/ts-2-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260129 02:57:43.071653 19276 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260129 02:57:43.072929 19276 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260129 02:57:43.074419 19276 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260129 02:57:43.076702 19276 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260129 02:57:43.076875 19276 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260129 02:57:43.077178 19276 ts_tablet_manager.cc:616] Registered 0 tablets
I20260129 02:57:43.077316 19276 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260129 02:57:43.173244 19276 rpc_server.cc:307] RPC server started. Bound to: 127.18.211.3:34213
I20260129 02:57:43.173399 19584 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.18.211.3:34213 every 8 connection(s)
I20260129 02:57:43.190312 19585 heartbeater.cc:344] Connected to a master server at 127.18.211.62:39923
I20260129 02:57:43.190733 19585 heartbeater.cc:461] Registering TS with master...
I20260129 02:57:43.191458 19585 heartbeater.cc:507] Master 127.18.211.62:39923 requested a full tablet report, sending...
I20260129 02:57:43.193305 19308 ts_manager.cc:194] Registered new tserver with Master: 309a53f12a534385a7e5a88fb373722a (127.18.211.3:34213)
I20260129 02:57:43.194172 19276 internal_mini_cluster.cc:371] 3 TS(s) registered with all masters after 0.016026807s
I20260129 02:57:43.194607 19308 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:36558
I20260129 02:57:43.246157 19308 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36570:
name: "test-workload"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "\004\001\000\216\3438\016\004\001\000\034\307q\034\004\001\000\252\252\252*\004\001\0008\216\3438\004\001\000\306q\034G\004\001\000TUUU\004\001\000\3428\216c\004\001\000p\034\307q""\004\001\000\216\3438\016\004\001\000\034\307q\034\004\001\000\252\252\252*\004\001\0008\216\3438\004\001\000\306q\034G\004\001\000TUUU\004\001\000\3428\216c\004\001\000p\034\307q"
indirect_data: """"
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260129 02:57:43.249892 19308 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-workload in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260129 02:57:43.366015 19550 tablet_service.cc:1505] Processing CreateTablet for tablet 2b5edf7ea566411caea265dcfff5a86a (DEFAULT_TABLE table=test-workload [id=a737fb7069a54dd589688af14107b257]), partition=RANGE (key) PARTITION VALUES < 238609294
I20260129 02:57:43.369735 19550 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2b5edf7ea566411caea265dcfff5a86a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260129 02:57:43.369365 19549 tablet_service.cc:1505] Processing CreateTablet for tablet e5ce9ed62ba242cabb3d6e5371679ba7 (DEFAULT_TABLE table=test-workload [id=a737fb7069a54dd589688af14107b257]), partition=RANGE (key) PARTITION 238609294 <= VALUES < 477218588
I20260129 02:57:43.373673 19549 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e5ce9ed62ba242cabb3d6e5371679ba7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260129 02:57:43.375098 19547 tablet_service.cc:1505] Processing CreateTablet for tablet a78e86f5781f41bba56333c01607277a (DEFAULT_TABLE table=test-workload [id=a737fb7069a54dd589688af14107b257]), partition=RANGE (key) PARTITION 715827882 <= VALUES < 954437176
I20260129 02:57:43.391412 19542 tablet_service.cc:1505] Processing CreateTablet for tablet 3e4edd649791453f902a5ac6d5658c12 (DEFAULT_TABLE table=test-workload [id=a737fb7069a54dd589688af14107b257]), partition=RANGE (key) PARTITION 1908874352 <= VALUES
I20260129 02:57:43.392345 19548 tablet_service.cc:1505] Processing CreateTablet for tablet 2e068f6d10f04c9ea17dba1375ae9b3c (DEFAULT_TABLE table=test-workload [id=a737fb7069a54dd589688af14107b257]), partition=RANGE (key) PARTITION 477218588 <= VALUES < 715827882
I20260129 02:57:43.391499 19397 tablet_service.cc:1505] Processing CreateTablet for tablet 9d5a1dc0b90642309290802698ec59f2 (DEFAULT_TABLE table=test-workload [id=a737fb7069a54dd589688af14107b257]), partition=RANGE (key) PARTITION 954437176 <= VALUES < 1193046470
I20260129 02:57:43.396327 19547 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a78e86f5781f41bba56333c01607277a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260129 02:57:43.395498 19545 tablet_service.cc:1505] Processing CreateTablet for tablet c4689cdf800d479ba8645b96278c1713 (DEFAULT_TABLE table=test-workload [id=a737fb7069a54dd589688af14107b257]), partition=RANGE (key) PARTITION 1193046470 <= VALUES < 1431655764
I20260129 02:57:43.391464 19546 tablet_service.cc:1505] Processing CreateTablet for tablet 9d5a1dc0b90642309290802698ec59f2 (DEFAULT_TABLE table=test-workload [id=a737fb7069a54dd589688af14107b257]), partition=RANGE (key) PARTITION 954437176 <= VALUES < 1193046470
I20260129 02:57:43.389393 19400 tablet_service.cc:1505] Processing CreateTablet for tablet e5ce9ed62ba242cabb3d6e5371679ba7 (DEFAULT_TABLE table=test-workload [id=a737fb7069a54dd589688af14107b257]), partition=RANGE (key) PARTITION 238609294 <= VALUES < 477218588
I20260129 02:57:43.394521 19396 tablet_service.cc:1505] Processing CreateTablet for tablet c4689cdf800d479ba8645b96278c1713 (DEFAULT_TABLE table=test-workload [id=a737fb7069a54dd589688af14107b257]), partition=RANGE (key) PARTITION 1193046470 <= VALUES < 1431655764
I20260129 02:57:43.401669 19397 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9d5a1dc0b90642309290802698ec59f2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260129 02:57:43.390017 19543 tablet_service.cc:1505] Processing CreateTablet for tablet f7d1ccc38ee8428aa16f7c82b86c0f3c (DEFAULT_TABLE table=test-workload [id=a737fb7069a54dd589688af14107b257]), partition=RANGE (key) PARTITION 1670265058 <= VALUES < 1908874352
I20260129 02:57:43.396546 19398 tablet_service.cc:1505] Processing CreateTablet for tablet a78e86f5781f41bba56333c01607277a (DEFAULT_TABLE table=test-workload [id=a737fb7069a54dd589688af14107b257]), partition=RANGE (key) PARTITION 715827882 <= VALUES < 954437176
I20260129 02:57:43.395447 19401 tablet_service.cc:1505] Processing CreateTablet for tablet 2b5edf7ea566411caea265dcfff5a86a (DEFAULT_TABLE table=test-workload [id=a737fb7069a54dd589688af14107b257]), partition=RANGE (key) PARTITION VALUES < 238609294
I20260129 02:57:43.393335 19544 tablet_service.cc:1505] Processing CreateTablet for tablet 6431d8f509f84fa6a63a3aad36a6d39d (DEFAULT_TABLE table=test-workload [id=a737fb7069a54dd589688af14107b257]), partition=RANGE (key) PARTITION 1431655764 <= VALUES < 1670265058
I20260129 02:57:43.408329 19542 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3e4edd649791453f902a5ac6d5658c12. 1 dirs total, 0 dirs full, 0 dirs failed
I20260129 02:57:43.403461 19395 tablet_service.cc:1505] Processing CreateTablet for tablet 6431d8f509f84fa6a63a3aad36a6d39d (DEFAULT_TABLE table=test-workload [id=a737fb7069a54dd589688af14107b257]), partition=RANGE (key) PARTITION 1431655764 <= VALUES < 1670265058
I20260129 02:57:43.397398 19399 tablet_service.cc:1505] Processing CreateTablet for tablet 2e068f6d10f04c9ea17dba1375ae9b3c (DEFAULT_TABLE table=test-workload [id=a737fb7069a54dd589688af14107b257]), partition=RANGE (key) PARTITION 477218588 <= VALUES < 715827882
I20260129 02:57:43.411362 19545 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c4689cdf800d479ba8645b96278c1713. 1 dirs total, 0 dirs full, 0 dirs failed
I20260129 02:57:43.411134 19394 tablet_service.cc:1505] Processing CreateTablet for tablet f7d1ccc38ee8428aa16f7c82b86c0f3c (DEFAULT_TABLE table=test-workload [id=a737fb7069a54dd589688af14107b257]), partition=RANGE (key) PARTITION 1670265058 <= VALUES < 1908874352
I20260129 02:57:43.415378 19544 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6431d8f509f84fa6a63a3aad36a6d39d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260129 02:57:43.417871 19393 tablet_service.cc:1505] Processing CreateTablet for tablet 3e4edd649791453f902a5ac6d5658c12 (DEFAULT_TABLE table=test-workload [id=a737fb7069a54dd589688af14107b257]), partition=RANGE (key) PARTITION 1908874352 <= VALUES
I20260129 02:57:43.418273 19400 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e5ce9ed62ba242cabb3d6e5371679ba7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260129 02:57:43.426191 19548 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2e068f6d10f04c9ea17dba1375ae9b3c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260129 02:57:43.430212 19546 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9d5a1dc0b90642309290802698ec59f2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260129 02:57:43.429999 19473 tablet_service.cc:1505] Processing CreateTablet for tablet a78e86f5781f41bba56333c01607277a (DEFAULT_TABLE table=test-workload [id=a737fb7069a54dd589688af14107b257]), partition=RANGE (key) PARTITION 715827882 <= VALUES < 954437176
I20260129 02:57:43.433522 19473 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a78e86f5781f41bba56333c01607277a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260129 02:57:43.425900 19474 tablet_service.cc:1505] Processing CreateTablet for tablet 2e068f6d10f04c9ea17dba1375ae9b3c (DEFAULT_TABLE table=test-workload [id=a737fb7069a54dd589688af14107b257]), partition=RANGE (key) PARTITION 477218588 <= VALUES < 715827882
I20260129 02:57:43.435891 19474 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2e068f6d10f04c9ea17dba1375ae9b3c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260129 02:57:43.426883 19476 tablet_service.cc:1505] Processing CreateTablet for tablet 2b5edf7ea566411caea265dcfff5a86a (DEFAULT_TABLE table=test-workload [id=a737fb7069a54dd589688af14107b257]), partition=RANGE (key) PARTITION VALUES < 238609294
I20260129 02:57:43.436764 19543 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f7d1ccc38ee8428aa16f7c82b86c0f3c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260129 02:57:43.437654 19471 tablet_service.cc:1505] Processing CreateTablet for tablet c4689cdf800d479ba8645b96278c1713 (DEFAULT_TABLE table=test-workload [id=a737fb7069a54dd589688af14107b257]), partition=RANGE (key) PARTITION 1193046470 <= VALUES < 1431655764
I20260129 02:57:43.442234 19401 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2b5edf7ea566411caea265dcfff5a86a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260129 02:57:43.446153 19476 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2b5edf7ea566411caea265dcfff5a86a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260129 02:57:43.431389 19472 tablet_service.cc:1505] Processing CreateTablet for tablet 9d5a1dc0b90642309290802698ec59f2 (DEFAULT_TABLE table=test-workload [id=a737fb7069a54dd589688af14107b257]), partition=RANGE (key) PARTITION 954437176 <= VALUES < 1193046470
I20260129 02:57:43.452136 19398 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a78e86f5781f41bba56333c01607277a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260129 02:57:43.457362 19472 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9d5a1dc0b90642309290802698ec59f2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260129 02:57:43.458396 19475 tablet_service.cc:1505] Processing CreateTablet for tablet e5ce9ed62ba242cabb3d6e5371679ba7 (DEFAULT_TABLE table=test-workload [id=a737fb7069a54dd589688af14107b257]), partition=RANGE (key) PARTITION 238609294 <= VALUES < 477218588
I20260129 02:57:43.458798 19471 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c4689cdf800d479ba8645b96278c1713. 1 dirs total, 0 dirs full, 0 dirs failed
I20260129 02:57:43.466315 19395 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6431d8f509f84fa6a63a3aad36a6d39d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260129 02:57:43.476431 19470 tablet_service.cc:1505] Processing CreateTablet for tablet 6431d8f509f84fa6a63a3aad36a6d39d (DEFAULT_TABLE table=test-workload [id=a737fb7069a54dd589688af14107b257]), partition=RANGE (key) PARTITION 1431655764 <= VALUES < 1670265058
I20260129 02:57:43.461176 19468 tablet_service.cc:1505] Processing CreateTablet for tablet 3e4edd649791453f902a5ac6d5658c12 (DEFAULT_TABLE table=test-workload [id=a737fb7069a54dd589688af14107b257]), partition=RANGE (key) PARTITION 1908874352 <= VALUES
I20260129 02:57:43.492076 19475 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e5ce9ed62ba242cabb3d6e5371679ba7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260129 02:57:43.465057 19469 tablet_service.cc:1505] Processing CreateTablet for tablet f7d1ccc38ee8428aa16f7c82b86c0f3c (DEFAULT_TABLE table=test-workload [id=a737fb7069a54dd589688af14107b257]), partition=RANGE (key) PARTITION 1670265058 <= VALUES < 1908874352
I20260129 02:57:43.508720 19399 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2e068f6d10f04c9ea17dba1375ae9b3c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260129 02:57:43.509861 19396 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c4689cdf800d479ba8645b96278c1713. 1 dirs total, 0 dirs full, 0 dirs failed
I20260129 02:57:43.511155 19470 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6431d8f509f84fa6a63a3aad36a6d39d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260129 02:57:43.514163 19468 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3e4edd649791453f902a5ac6d5658c12. 1 dirs total, 0 dirs full, 0 dirs failed
I20260129 02:57:43.516415 19469 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f7d1ccc38ee8428aa16f7c82b86c0f3c. 1 dirs total, 0 dirs full, 0 dirs failed
W20260129 02:57:43.547276 19356 auto_rebalancer.cc:229] Could not retrieve cluster info: Service unavailable: Tablet not running
I20260129 02:57:43.544479 19393 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3e4edd649791453f902a5ac6d5658c12. 1 dirs total, 0 dirs full, 0 dirs failed
I20260129 02:57:43.566709 19394 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f7d1ccc38ee8428aa16f7c82b86c0f3c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260129 02:57:43.755465 19604 tablet_bootstrap.cc:492] T e5ce9ed62ba242cabb3d6e5371679ba7 P a6aef613bf534ff9ae6ddbba06bb3cf1: Bootstrap starting.
I20260129 02:57:43.763196 19603 tablet_bootstrap.cc:492] T 2b5edf7ea566411caea265dcfff5a86a P 309a53f12a534385a7e5a88fb373722a: Bootstrap starting.
I20260129 02:57:43.769073 19605 tablet_bootstrap.cc:492] T a78e86f5781f41bba56333c01607277a P 895c9e1595924e769a25faeed7e34644: Bootstrap starting.
I20260129 02:57:43.782994 19604 tablet_bootstrap.cc:654] T e5ce9ed62ba242cabb3d6e5371679ba7 P a6aef613bf534ff9ae6ddbba06bb3cf1: Neither blocks nor log segments found. Creating new log.
I20260129 02:57:43.792927 19605 tablet_bootstrap.cc:654] T a78e86f5781f41bba56333c01607277a P 895c9e1595924e769a25faeed7e34644: Neither blocks nor log segments found. Creating new log.
I20260129 02:57:43.793072 19603 tablet_bootstrap.cc:654] T 2b5edf7ea566411caea265dcfff5a86a P 309a53f12a534385a7e5a88fb373722a: Neither blocks nor log segments found. Creating new log.
I20260129 02:57:43.803241 19604 tablet_bootstrap.cc:492] T e5ce9ed62ba242cabb3d6e5371679ba7 P a6aef613bf534ff9ae6ddbba06bb3cf1: No bootstrap required, opened a new log
I20260129 02:57:43.803872 19604 ts_tablet_manager.cc:1403] T e5ce9ed62ba242cabb3d6e5371679ba7 P a6aef613bf534ff9ae6ddbba06bb3cf1: Time spent bootstrapping tablet: real 0.052s user 0.017s sys 0.004s
I20260129 02:57:43.804332 19605 tablet_bootstrap.cc:492] T a78e86f5781f41bba56333c01607277a P 895c9e1595924e769a25faeed7e34644: No bootstrap required, opened a new log
I20260129 02:57:43.804786 19605 ts_tablet_manager.cc:1403] T a78e86f5781f41bba56333c01607277a P 895c9e1595924e769a25faeed7e34644: Time spent bootstrapping tablet: real 0.050s user 0.014s sys 0.004s
I20260129 02:57:43.805402 19603 tablet_bootstrap.cc:492] T 2b5edf7ea566411caea265dcfff5a86a P 309a53f12a534385a7e5a88fb373722a: No bootstrap required, opened a new log
I20260129 02:57:43.806236 19603 ts_tablet_manager.cc:1403] T 2b5edf7ea566411caea265dcfff5a86a P 309a53f12a534385a7e5a88fb373722a: Time spent bootstrapping tablet: real 0.059s user 0.012s sys 0.007s
I20260129 02:57:43.807302 19604 raft_consensus.cc:359] T e5ce9ed62ba242cabb3d6e5371679ba7 P a6aef613bf534ff9ae6ddbba06bb3cf1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6aef613bf534ff9ae6ddbba06bb3cf1" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 45187 } } peers { permanent_uuid: "895c9e1595924e769a25faeed7e34644" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 40083 } } peers { permanent_uuid: "309a53f12a534385a7e5a88fb373722a" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 34213 } }
I20260129 02:57:43.807437 19605 raft_consensus.cc:359] T a78e86f5781f41bba56333c01607277a P 895c9e1595924e769a25faeed7e34644 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6aef613bf534ff9ae6ddbba06bb3cf1" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 45187 } } peers { permanent_uuid: "895c9e1595924e769a25faeed7e34644" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 40083 } } peers { permanent_uuid: "309a53f12a534385a7e5a88fb373722a" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 34213 } }
I20260129 02:57:43.808259 19604 raft_consensus.cc:385] T e5ce9ed62ba242cabb3d6e5371679ba7 P a6aef613bf534ff9ae6ddbba06bb3cf1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260129 02:57:43.808588 19605 raft_consensus.cc:385] T a78e86f5781f41bba56333c01607277a P 895c9e1595924e769a25faeed7e34644 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260129 02:57:43.808697 19604 raft_consensus.cc:740] T e5ce9ed62ba242cabb3d6e5371679ba7 P a6aef613bf534ff9ae6ddbba06bb3cf1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a6aef613bf534ff9ae6ddbba06bb3cf1, State: Initialized, Role: FOLLOWER
I20260129 02:57:43.808941 19605 raft_consensus.cc:740] T a78e86f5781f41bba56333c01607277a P 895c9e1595924e769a25faeed7e34644 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 895c9e1595924e769a25faeed7e34644, State: Initialized, Role: FOLLOWER
I20260129 02:57:43.808851 19603 raft_consensus.cc:359] T 2b5edf7ea566411caea265dcfff5a86a P 309a53f12a534385a7e5a88fb373722a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "309a53f12a534385a7e5a88fb373722a" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 34213 } } peers { permanent_uuid: "a6aef613bf534ff9ae6ddbba06bb3cf1" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 45187 } } peers { permanent_uuid: "895c9e1595924e769a25faeed7e34644" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 40083 } }
I20260129 02:57:43.809844 19603 raft_consensus.cc:385] T 2b5edf7ea566411caea265dcfff5a86a P 309a53f12a534385a7e5a88fb373722a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260129 02:57:43.810236 19603 raft_consensus.cc:740] T 2b5edf7ea566411caea265dcfff5a86a P 309a53f12a534385a7e5a88fb373722a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 309a53f12a534385a7e5a88fb373722a, State: Initialized, Role: FOLLOWER
I20260129 02:57:43.809677 19605 consensus_queue.cc:260] T a78e86f5781f41bba56333c01607277a P 895c9e1595924e769a25faeed7e34644 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6aef613bf534ff9ae6ddbba06bb3cf1" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 45187 } } peers { permanent_uuid: "895c9e1595924e769a25faeed7e34644" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 40083 } } peers { permanent_uuid: "309a53f12a534385a7e5a88fb373722a" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 34213 } }
I20260129 02:57:43.809643 19604 consensus_queue.cc:260] T e5ce9ed62ba242cabb3d6e5371679ba7 P a6aef613bf534ff9ae6ddbba06bb3cf1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6aef613bf534ff9ae6ddbba06bb3cf1" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 45187 } } peers { permanent_uuid: "895c9e1595924e769a25faeed7e34644" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 40083 } } peers { permanent_uuid: "309a53f12a534385a7e5a88fb373722a" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 34213 } }
I20260129 02:57:43.810863 19603 consensus_queue.cc:260] T 2b5edf7ea566411caea265dcfff5a86a P 309a53f12a534385a7e5a88fb373722a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "309a53f12a534385a7e5a88fb373722a" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 34213 } } peers { permanent_uuid: "a6aef613bf534ff9ae6ddbba06bb3cf1" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 45187 } } peers { permanent_uuid: "895c9e1595924e769a25faeed7e34644" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 40083 } }
I20260129 02:57:43.819033 19605 ts_tablet_manager.cc:1434] T a78e86f5781f41bba56333c01607277a P 895c9e1595924e769a25faeed7e34644: Time spent starting tablet: real 0.014s user 0.008s sys 0.004s
I20260129 02:57:43.819165 19585 heartbeater.cc:499] Master 127.18.211.62:39923 was elected leader, sending a full tablet report...
I20260129 02:57:43.819684 19511 heartbeater.cc:499] Master 127.18.211.62:39923 was elected leader, sending a full tablet report...
I20260129 02:57:43.820178 19603 ts_tablet_manager.cc:1434] T 2b5edf7ea566411caea265dcfff5a86a P 309a53f12a534385a7e5a88fb373722a: Time spent starting tablet: real 0.014s user 0.003s sys 0.010s
I20260129 02:57:43.821066 19603 tablet_bootstrap.cc:492] T e5ce9ed62ba242cabb3d6e5371679ba7 P 309a53f12a534385a7e5a88fb373722a: Bootstrap starting.
I20260129 02:57:43.823252 19604 ts_tablet_manager.cc:1434] T e5ce9ed62ba242cabb3d6e5371679ba7 P a6aef613bf534ff9ae6ddbba06bb3cf1: Time spent starting tablet: real 0.019s user 0.006s sys 0.007s
I20260129 02:57:43.825686 19605 tablet_bootstrap.cc:492] T e5ce9ed62ba242cabb3d6e5371679ba7 P 895c9e1595924e769a25faeed7e34644: Bootstrap starting.
I20260129 02:57:43.828944 19611 raft_consensus.cc:493] T 2b5edf7ea566411caea265dcfff5a86a P 309a53f12a534385a7e5a88fb373722a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260129 02:57:43.829497 19611 raft_consensus.cc:515] T 2b5edf7ea566411caea265dcfff5a86a P 309a53f12a534385a7e5a88fb373722a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "309a53f12a534385a7e5a88fb373722a" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 34213 } } peers { permanent_uuid: "a6aef613bf534ff9ae6ddbba06bb3cf1" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 45187 } } peers { permanent_uuid: "895c9e1595924e769a25faeed7e34644" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 40083 } }
I20260129 02:57:43.832415 19611 leader_election.cc:290] T 2b5edf7ea566411caea265dcfff5a86a P 309a53f12a534385a7e5a88fb373722a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a6aef613bf534ff9ae6ddbba06bb3cf1 (127.18.211.1:45187), 895c9e1595924e769a25faeed7e34644 (127.18.211.2:40083)
I20260129 02:57:43.834254 19603 tablet_bootstrap.cc:654] T e5ce9ed62ba242cabb3d6e5371679ba7 P 309a53f12a534385a7e5a88fb373722a: Neither blocks nor log segments found. Creating new log.
I20260129 02:57:43.834859 19604 tablet_bootstrap.cc:492] T f7d1ccc38ee8428aa16f7c82b86c0f3c P a6aef613bf534ff9ae6ddbba06bb3cf1: Bootstrap starting.
I20260129 02:57:43.835094 19436 heartbeater.cc:499] Master 127.18.211.62:39923 was elected leader, sending a full tablet report...
I20260129 02:57:43.844290 19604 tablet_bootstrap.cc:654] T f7d1ccc38ee8428aa16f7c82b86c0f3c P a6aef613bf534ff9ae6ddbba06bb3cf1: Neither blocks nor log segments found. Creating new log.
I20260129 02:57:43.845165 19605 tablet_bootstrap.cc:654] T e5ce9ed62ba242cabb3d6e5371679ba7 P 895c9e1595924e769a25faeed7e34644: Neither blocks nor log segments found. Creating new log.
I20260129 02:57:43.869611 19605 tablet_bootstrap.cc:492] T e5ce9ed62ba242cabb3d6e5371679ba7 P 895c9e1595924e769a25faeed7e34644: No bootstrap required, opened a new log
I20260129 02:57:43.871944 19609 raft_consensus.cc:493] T a78e86f5781f41bba56333c01607277a P 895c9e1595924e769a25faeed7e34644 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260129 02:57:43.873461 19609 raft_consensus.cc:515] T a78e86f5781f41bba56333c01607277a P 895c9e1595924e769a25faeed7e34644 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6aef613bf534ff9ae6ddbba06bb3cf1" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 45187 } } peers { permanent_uuid: "895c9e1595924e769a25faeed7e34644" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 40083 } } peers { permanent_uuid: "309a53f12a534385a7e5a88fb373722a" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 34213 } }
I20260129 02:57:43.876072 19609 leader_election.cc:290] T a78e86f5781f41bba56333c01607277a P 895c9e1595924e769a25faeed7e34644 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a6aef613bf534ff9ae6ddbba06bb3cf1 (127.18.211.1:45187), 309a53f12a534385a7e5a88fb373722a (127.18.211.3:34213)
I20260129 02:57:43.878756 19604 tablet_bootstrap.cc:492] T f7d1ccc38ee8428aa16f7c82b86c0f3c P a6aef613bf534ff9ae6ddbba06bb3cf1: No bootstrap required, opened a new log
I20260129 02:57:43.882352 19604 ts_tablet_manager.cc:1403] T f7d1ccc38ee8428aa16f7c82b86c0f3c P a6aef613bf534ff9ae6ddbba06bb3cf1: Time spent bootstrapping tablet: real 0.048s user 0.010s sys 0.010s
I20260129 02:57:43.883117 19605 ts_tablet_manager.cc:1403] T e5ce9ed62ba242cabb3d6e5371679ba7 P 895c9e1595924e769a25faeed7e34644: Time spent bootstrapping tablet: real 0.058s user 0.009s sys 0.005s
I20260129 02:57:43.888228 19486 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2b5edf7ea566411caea265dcfff5a86a" candidate_uuid: "309a53f12a534385a7e5a88fb373722a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "895c9e1595924e769a25faeed7e34644" is_pre_election: true
I20260129 02:57:43.887267 19604 raft_consensus.cc:359] T f7d1ccc38ee8428aa16f7c82b86c0f3c P a6aef613bf534ff9ae6ddbba06bb3cf1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6aef613bf534ff9ae6ddbba06bb3cf1" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 45187 } } peers { permanent_uuid: "895c9e1595924e769a25faeed7e34644" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 40083 } } peers { permanent_uuid: "309a53f12a534385a7e5a88fb373722a" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 34213 } }
I20260129 02:57:43.889423 19604 raft_consensus.cc:385] T f7d1ccc38ee8428aa16f7c82b86c0f3c P a6aef613bf534ff9ae6ddbba06bb3cf1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260129 02:57:43.890055 19604 raft_consensus.cc:740] T f7d1ccc38ee8428aa16f7c82b86c0f3c P a6aef613bf534ff9ae6ddbba06bb3cf1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a6aef613bf534ff9ae6ddbba06bb3cf1, State: Initialized, Role: FOLLOWER
W20260129 02:57:43.891185 19524 leader_election.cc:343] T 2b5edf7ea566411caea265dcfff5a86a P 309a53f12a534385a7e5a88fb373722a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 895c9e1595924e769a25faeed7e34644 (127.18.211.2:40083): Illegal state: must be running to vote when last-logged opid is not known
I20260129 02:57:43.897935 19604 consensus_queue.cc:260] T f7d1ccc38ee8428aa16f7c82b86c0f3c P a6aef613bf534ff9ae6ddbba06bb3cf1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6aef613bf534ff9ae6ddbba06bb3cf1" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 45187 } } peers { permanent_uuid: "895c9e1595924e769a25faeed7e34644" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 40083 } } peers { permanent_uuid: "309a53f12a534385a7e5a88fb373722a" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 34213 } }
I20260129 02:57:43.899117 19605 raft_consensus.cc:359] T e5ce9ed62ba242cabb3d6e5371679ba7 P 895c9e1595924e769a25faeed7e34644 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6aef613bf534ff9ae6ddbba06bb3cf1" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 45187 } } peers { permanent_uuid: "895c9e1595924e769a25faeed7e34644" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 40083 } } peers { permanent_uuid: "309a53f12a534385a7e5a88fb373722a" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 34213 } }
I20260129 02:57:43.901477 19603 tablet_bootstrap.cc:492] T e5ce9ed62ba242cabb3d6e5371679ba7 P 309a53f12a534385a7e5a88fb373722a: No bootstrap required, opened a new log
I20260129 02:57:43.901721 19604 ts_tablet_manager.cc:1434] T f7d1ccc38ee8428aa16f7c82b86c0f3c P a6aef613bf534ff9ae6ddbba06bb3cf1: Time spent starting tablet: real 0.019s user 0.006s sys 0.004s
I20260129 02:57:43.899812 19605 raft_consensus.cc:385] T e5ce9ed62ba242cabb3d6e5371679ba7 P 895c9e1595924e769a25faeed7e34644 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260129 02:57:43.901932 19603 ts_tablet_manager.cc:1403] T e5ce9ed62ba242cabb3d6e5371679ba7 P 309a53f12a534385a7e5a88fb373722a: Time spent bootstrapping tablet: real 0.081s user 0.010s sys 0.012s
I20260129 02:57:43.910306 19411 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2b5edf7ea566411caea265dcfff5a86a" candidate_uuid: "309a53f12a534385a7e5a88fb373722a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a6aef613bf534ff9ae6ddbba06bb3cf1" is_pre_election: true
I20260129 02:57:43.911553 19605 raft_consensus.cc:740] T e5ce9ed62ba242cabb3d6e5371679ba7 P 895c9e1595924e769a25faeed7e34644 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 895c9e1595924e769a25faeed7e34644, State: Initialized, Role: FOLLOWER
I20260129 02:57:43.912264 19605 consensus_queue.cc:260] T e5ce9ed62ba242cabb3d6e5371679ba7 P 895c9e1595924e769a25faeed7e34644 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6aef613bf534ff9ae6ddbba06bb3cf1" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 45187 } } peers { permanent_uuid: "895c9e1595924e769a25faeed7e34644" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 40083 } } peers { permanent_uuid: "309a53f12a534385a7e5a88fb373722a" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 34213 } }
W20260129 02:57:43.912663 19524 leader_election.cc:343] T 2b5edf7ea566411caea265dcfff5a86a P 309a53f12a534385a7e5a88fb373722a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a6aef613bf534ff9ae6ddbba06bb3cf1 (127.18.211.1:45187): Illegal state: must be running to vote when last-logged opid is not known
I20260129 02:57:43.913196 19524 leader_election.cc:304] T 2b5edf7ea566411caea265dcfff5a86a P 309a53f12a534385a7e5a88fb373722a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 309a53f12a534385a7e5a88fb373722a; no voters: 895c9e1595924e769a25faeed7e34644, a6aef613bf534ff9ae6ddbba06bb3cf1
I20260129 02:57:43.913956 19603 raft_consensus.cc:359] T e5ce9ed62ba242cabb3d6e5371679ba7 P 309a53f12a534385a7e5a88fb373722a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6aef613bf534ff9ae6ddbba06bb3cf1" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 45187 } } peers { permanent_uuid: "895c9e1595924e769a25faeed7e34644" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 40083 } } peers { permanent_uuid: "309a53f12a534385a7e5a88fb373722a" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 34213 } }
I20260129 02:57:43.914462 19605 ts_tablet_manager.cc:1434] T e5ce9ed62ba242cabb3d6e5371679ba7 P 895c9e1595924e769a25faeed7e34644: Time spent starting tablet: real 0.030s user 0.002s sys 0.003s
I20260129 02:57:43.914641 19603 raft_consensus.cc:385] T e5ce9ed62ba242cabb3d6e5371679ba7 P 309a53f12a534385a7e5a88fb373722a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260129 02:57:43.914961 19603 raft_consensus.cc:740] T e5ce9ed62ba242cabb3d6e5371679ba7 P 309a53f12a534385a7e5a88fb373722a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 309a53f12a534385a7e5a88fb373722a, State: Initialized, Role: FOLLOWER
I20260129 02:57:43.915644 19603 consensus_queue.cc:260] T e5ce9ed62ba242cabb3d6e5371679ba7 P 309a53f12a534385a7e5a88fb373722a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6aef613bf534ff9ae6ddbba06bb3cf1" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 45187 } } peers { permanent_uuid: "895c9e1595924e769a25faeed7e34644" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 40083 } } peers { permanent_uuid: "309a53f12a534385a7e5a88fb373722a" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 34213 } }
I20260129 02:57:43.917269 19605 tablet_bootstrap.cc:492] T 2e068f6d10f04c9ea17dba1375ae9b3c P 895c9e1595924e769a25faeed7e34644: Bootstrap starting.
I20260129 02:57:43.917675 19611 raft_consensus.cc:2749] T 2b5edf7ea566411caea265dcfff5a86a P 309a53f12a534385a7e5a88fb373722a [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260129 02:57:43.925064 19605 tablet_bootstrap.cc:654] T 2e068f6d10f04c9ea17dba1375ae9b3c P 895c9e1595924e769a25faeed7e34644: Neither blocks nor log segments found. Creating new log.
I20260129 02:57:43.927599 19604 tablet_bootstrap.cc:492] T 6431d8f509f84fa6a63a3aad36a6d39d P a6aef613bf534ff9ae6ddbba06bb3cf1: Bootstrap starting.
I20260129 02:57:43.927416 19411 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a78e86f5781f41bba56333c01607277a" candidate_uuid: "895c9e1595924e769a25faeed7e34644" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a6aef613bf534ff9ae6ddbba06bb3cf1" is_pre_election: true
W20260129 02:57:43.931208 19586 tablet.cc:2378] T 2b5edf7ea566411caea265dcfff5a86a P 309a53f12a534385a7e5a88fb373722a: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260129 02:57:43.933293 19450 leader_election.cc:343] T a78e86f5781f41bba56333c01607277a P 895c9e1595924e769a25faeed7e34644 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a6aef613bf534ff9ae6ddbba06bb3cf1 (127.18.211.1:45187): Illegal state: must be running to vote when last-logged opid is not known
I20260129 02:57:43.934001 19603 ts_tablet_manager.cc:1434] T e5ce9ed62ba242cabb3d6e5371679ba7 P 309a53f12a534385a7e5a88fb373722a: Time spent starting tablet: real 0.022s user 0.014s sys 0.005s
I20260129 02:57:43.939983 19603 tablet_bootstrap.cc:492] T c4689cdf800d479ba8645b96278c1713 P 309a53f12a534385a7e5a88fb373722a: Bootstrap starting.
I20260129 02:57:43.940574 19604 tablet_bootstrap.cc:654] T 6431d8f509f84fa6a63a3aad36a6d39d P a6aef613bf534ff9ae6ddbba06bb3cf1: Neither blocks nor log segments found. Creating new log.
I20260129 02:57:43.946619 19603 tablet_bootstrap.cc:654] T c4689cdf800d479ba8645b96278c1713 P 309a53f12a534385a7e5a88fb373722a: Neither blocks nor log segments found. Creating new log.
I20260129 02:57:43.948896 19605 tablet_bootstrap.cc:492] T 2e068f6d10f04c9ea17dba1375ae9b3c P 895c9e1595924e769a25faeed7e34644: No bootstrap required, opened a new log
I20260129 02:57:43.949368 19605 ts_tablet_manager.cc:1403] T 2e068f6d10f04c9ea17dba1375ae9b3c P 895c9e1595924e769a25faeed7e34644: Time spent bootstrapping tablet: real 0.032s user 0.010s sys 0.001s
I20260129 02:57:43.951723 19560 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a78e86f5781f41bba56333c01607277a" candidate_uuid: "895c9e1595924e769a25faeed7e34644" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "309a53f12a534385a7e5a88fb373722a" is_pre_election: true
I20260129 02:57:43.952165 19605 raft_consensus.cc:359] T 2e068f6d10f04c9ea17dba1375ae9b3c P 895c9e1595924e769a25faeed7e34644 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6aef613bf534ff9ae6ddbba06bb3cf1" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 45187 } } peers { permanent_uuid: "895c9e1595924e769a25faeed7e34644" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 40083 } } peers { permanent_uuid: "309a53f12a534385a7e5a88fb373722a" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 34213 } }
I20260129 02:57:43.952946 19605 raft_consensus.cc:385] T 2e068f6d10f04c9ea17dba1375ae9b3c P 895c9e1595924e769a25faeed7e34644 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260129 02:57:43.953289 19605 raft_consensus.cc:740] T 2e068f6d10f04c9ea17dba1375ae9b3c P 895c9e1595924e769a25faeed7e34644 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 895c9e1595924e769a25faeed7e34644, State: Initialized, Role: FOLLOWER
W20260129 02:57:43.953322 19450 leader_election.cc:343] T a78e86f5781f41bba56333c01607277a P 895c9e1595924e769a25faeed7e34644 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 309a53f12a534385a7e5a88fb373722a (127.18.211.3:34213): Illegal state: must be running to vote when last-logged opid is not known
I20260129 02:57:43.953802 19450 leader_election.cc:304] T a78e86f5781f41bba56333c01607277a P 895c9e1595924e769a25faeed7e34644 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 895c9e1595924e769a25faeed7e34644; no voters: 309a53f12a534385a7e5a88fb373722a, a6aef613bf534ff9ae6ddbba06bb3cf1
I20260129 02:57:43.954581 19609 raft_consensus.cc:2749] T a78e86f5781f41bba56333c01607277a P 895c9e1595924e769a25faeed7e34644 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260129 02:57:43.954120 19605 consensus_queue.cc:260] T 2e068f6d10f04c9ea17dba1375ae9b3c P 895c9e1595924e769a25faeed7e34644 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6aef613bf534ff9ae6ddbba06bb3cf1" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 45187 } } peers { permanent_uuid: "895c9e1595924e769a25faeed7e34644" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 40083 } } peers { permanent_uuid: "309a53f12a534385a7e5a88fb373722a" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 34213 } }
I20260129 02:57:43.955657 19604 tablet_bootstrap.cc:492] T 6431d8f509f84fa6a63a3aad36a6d39d P a6aef613bf534ff9ae6ddbba06bb3cf1: No bootstrap required, opened a new log
I20260129 02:57:43.956107 19604 ts_tablet_manager.cc:1403] T 6431d8f509f84fa6a63a3aad36a6d39d P a6aef613bf534ff9ae6ddbba06bb3cf1: Time spent bootstrapping tablet: real 0.029s user 0.010s sys 0.004s
I20260129 02:57:43.959453 19604 raft_consensus.cc:359] T 6431d8f509f84fa6a63a3aad36a6d39d P a6aef613bf534ff9ae6ddbba06bb3cf1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6aef613bf534ff9ae6ddbba06bb3cf1" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 45187 } } peers { permanent_uuid: "895c9e1595924e769a25faeed7e34644" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 40083 } } peers { permanent_uuid: "309a53f12a534385a7e5a88fb373722a" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 34213 } }
I20260129 02:57:43.960239 19604 raft_consensus.cc:385] T 6431d8f509f84fa6a63a3aad36a6d39d P a6aef613bf534ff9ae6ddbba06bb3cf1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260129 02:57:43.960543 19604 raft_consensus.cc:740] T 6431d8f509f84fa6a63a3aad36a6d39d P a6aef613bf534ff9ae6ddbba06bb3cf1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a6aef613bf534ff9ae6ddbba06bb3cf1, State: Initialized, Role: FOLLOWER
I20260129 02:57:43.961161 19604 consensus_queue.cc:260] T 6431d8f509f84fa6a63a3aad36a6d39d P a6aef613bf534ff9ae6ddbba06bb3cf1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6aef613bf534ff9ae6ddbba06bb3cf1" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 45187 } } peers { permanent_uuid: "895c9e1595924e769a25faeed7e34644" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 40083 } } peers { permanent_uuid: "309a53f12a534385a7e5a88fb373722a" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 34213 } }
I20260129 02:57:43.963511 19605 ts_tablet_manager.cc:1434] T 2e068f6d10f04c9ea17dba1375ae9b3c P 895c9e1595924e769a25faeed7e34644: Time spent starting tablet: real 0.014s user 0.004s sys 0.003s
I20260129 02:57:43.963608 19604 ts_tablet_manager.cc:1434] T 6431d8f509f84fa6a63a3aad36a6d39d P a6aef613bf534ff9ae6ddbba06bb3cf1: Time spent starting tablet: real 0.007s user 0.006s sys 0.000s
I20260129 02:57:43.964541 19604 tablet_bootstrap.cc:492] T c4689cdf800d479ba8645b96278c1713 P a6aef613bf534ff9ae6ddbba06bb3cf1: Bootstrap starting.
I20260129 02:57:43.965108 19603 tablet_bootstrap.cc:492] T c4689cdf800d479ba8645b96278c1713 P 309a53f12a534385a7e5a88fb373722a: No bootstrap required, opened a new log
I20260129 02:57:43.965570 19603 ts_tablet_manager.cc:1403] T c4689cdf800d479ba8645b96278c1713 P 309a53f12a534385a7e5a88fb373722a: Time spent bootstrapping tablet: real 0.026s user 0.007s sys 0.011s
I20260129 02:57:43.968341 19603 raft_consensus.cc:359] T c4689cdf800d479ba8645b96278c1713 P 309a53f12a534385a7e5a88fb373722a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6aef613bf534ff9ae6ddbba06bb3cf1" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 45187 } } peers { permanent_uuid: "895c9e1595924e769a25faeed7e34644" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 40083 } } peers { permanent_uuid: "309a53f12a534385a7e5a88fb373722a" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 34213 } }
I20260129 02:57:43.969019 19603 raft_consensus.cc:385] T c4689cdf800d479ba8645b96278c1713 P 309a53f12a534385a7e5a88fb373722a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260129 02:57:43.969316 19603 raft_consensus.cc:740] T c4689cdf800d479ba8645b96278c1713 P 309a53f12a534385a7e5a88fb373722a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 309a53f12a534385a7e5a88fb373722a, State: Initialized, Role: FOLLOWER
I20260129 02:57:43.969918 19603 consensus_queue.cc:260] T c4689cdf800d479ba8645b96278c1713 P 309a53f12a534385a7e5a88fb373722a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6aef613bf534ff9ae6ddbba06bb3cf1" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 45187 } } peers { permanent_uuid: "895c9e1595924e769a25faeed7e34644" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 40083 } } peers { permanent_uuid: "309a53f12a534385a7e5a88fb373722a" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 34213 } }
I20260129 02:57:43.972246 19603 ts_tablet_manager.cc:1434] T c4689cdf800d479ba8645b96278c1713 P 309a53f12a534385a7e5a88fb373722a: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260129 02:57:43.973300 19603 tablet_bootstrap.cc:492] T 2e068f6d10f04c9ea17dba1375ae9b3c P 309a53f12a534385a7e5a88fb373722a: Bootstrap starting.
I20260129 02:57:43.975929 19604 tablet_bootstrap.cc:654] T c4689cdf800d479ba8645b96278c1713 P a6aef613bf534ff9ae6ddbba06bb3cf1: Neither blocks nor log segments found. Creating new log.
I20260129 02:57:43.979616 19603 tablet_bootstrap.cc:654] T 2e068f6d10f04c9ea17dba1375ae9b3c P 309a53f12a534385a7e5a88fb373722a: Neither blocks nor log segments found. Creating new log.
I20260129 02:57:43.985630 19605 tablet_bootstrap.cc:492] T f7d1ccc38ee8428aa16f7c82b86c0f3c P 895c9e1595924e769a25faeed7e34644: Bootstrap starting.
I20260129 02:57:43.989272 19604 tablet_bootstrap.cc:492] T c4689cdf800d479ba8645b96278c1713 P a6aef613bf534ff9ae6ddbba06bb3cf1: No bootstrap required, opened a new log
I20260129 02:57:43.989535 19603 tablet_bootstrap.cc:492] T 2e068f6d10f04c9ea17dba1375ae9b3c P 309a53f12a534385a7e5a88fb373722a: No bootstrap required, opened a new log
I20260129 02:57:43.989857 19604 ts_tablet_manager.cc:1403] T c4689cdf800d479ba8645b96278c1713 P a6aef613bf534ff9ae6ddbba06bb3cf1: Time spent bootstrapping tablet: real 0.025s user 0.011s sys 0.007s
I20260129 02:57:43.990023 19603 ts_tablet_manager.cc:1403] T 2e068f6d10f04c9ea17dba1375ae9b3c P 309a53f12a534385a7e5a88fb373722a: Time spent bootstrapping tablet: real 0.017s user 0.010s sys 0.004s
I20260129 02:57:43.991796 19605 tablet_bootstrap.cc:654] T f7d1ccc38ee8428aa16f7c82b86c0f3c P 895c9e1595924e769a25faeed7e34644: Neither blocks nor log segments found. Creating new log.
I20260129 02:57:43.992786 19603 raft_consensus.cc:359] T 2e068f6d10f04c9ea17dba1375ae9b3c P 309a53f12a534385a7e5a88fb373722a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6aef613bf534ff9ae6ddbba06bb3cf1" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 45187 } } peers { permanent_uuid: "895c9e1595924e769a25faeed7e34644" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 40083 } } peers { permanent_uuid: "309a53f12a534385a7e5a88fb373722a" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 34213 } }
I20260129 02:57:43.993134 19604 raft_consensus.cc:359] T c4689cdf800d479ba8645b96278c1713 P a6aef613bf534ff9ae6ddbba06bb3cf1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6aef613bf534ff9ae6ddbba06bb3cf1" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 45187 } } peers { permanent_uuid: "895c9e1595924e769a25faeed7e34644" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 40083 } } peers { permanent_uuid: "309a53f12a534385a7e5a88fb373722a" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 34213 } }
I20260129 02:57:43.993587 19603 raft_consensus.cc:385] T 2e068f6d10f04c9ea17dba1375ae9b3c P 309a53f12a534385a7e5a88fb373722a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260129 02:57:43.993986 19604 raft_consensus.cc:385] T c4689cdf800d479ba8645b96278c1713 P a6aef613bf534ff9ae6ddbba06bb3cf1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260129 02:57:43.994019 19603 raft_consensus.cc:740] T 2e068f6d10f04c9ea17dba1375ae9b3c P 309a53f12a534385a7e5a88fb373722a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 309a53f12a534385a7e5a88fb373722a, State: Initialized, Role: FOLLOWER
I20260129 02:57:43.994511 19604 raft_consensus.cc:740] T c4689cdf800d479ba8645b96278c1713 P a6aef613bf534ff9ae6ddbba06bb3cf1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a6aef613bf534ff9ae6ddbba06bb3cf1, State: Initialized, Role: FOLLOWER
I20260129 02:57:43.994895 19603 consensus_queue.cc:260] T 2e068f6d10f04c9ea17dba1375ae9b3c P 309a53f12a534385a7e5a88fb373722a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6aef613bf534ff9ae6ddbba06bb3cf1" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 45187 } } peers { permanent_uuid: "895c9e1595924e769a25faeed7e34644" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 40083 } } peers { permanent_uuid: "309a53f12a534385a7e5a88fb373722a" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 34213 } }
I20260129 02:57:43.995507 19604 consensus_queue.cc:260] T c4689cdf800d479ba8645b96278c1713 P a6aef613bf534ff9ae6ddbba06bb3cf1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6aef613bf534ff9ae6ddbba06bb3cf1" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 45187 } } peers { permanent_uuid: "895c9e1595924e769a25faeed7e34644" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 40083 } } peers { permanent_uuid: "309a53f12a534385a7e5a88fb373722a" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 34213 } }
I20260129 02:57:43.997794 19603 ts_tablet_manager.cc:1434] T 2e068f6d10f04c9ea17dba1375ae9b3c P 309a53f12a534385a7e5a88fb373722a: Time spent starting tablet: real 0.007s user 0.003s sys 0.003s
I20260129 02:57:43.998477 19604 ts_tablet_manager.cc:1434] T c4689cdf800d479ba8645b96278c1713 P a6aef613bf534ff9ae6ddbba06bb3cf1: Time spent starting tablet: real 0.008s user 0.006s sys 0.000s
I20260129 02:57:43.998819 19603 tablet_bootstrap.cc:492] T 9d5a1dc0b90642309290802698ec59f2 P 309a53f12a534385a7e5a88fb373722a: Bootstrap starting.
I20260129 02:57:43.999385 19604 tablet_bootstrap.cc:492] T 9d5a1dc0b90642309290802698ec59f2 P a6aef613bf534ff9ae6ddbba06bb3cf1: Bootstrap starting.
I20260129 02:57:44.005724 19603 tablet_bootstrap.cc:654] T 9d5a1dc0b90642309290802698ec59f2 P 309a53f12a534385a7e5a88fb373722a: Neither blocks nor log segments found. Creating new log.
I20260129 02:57:44.006472 19604 tablet_bootstrap.cc:654] T 9d5a1dc0b90642309290802698ec59f2 P a6aef613bf534ff9ae6ddbba06bb3cf1: Neither blocks nor log segments found. Creating new log.
I20260129 02:57:44.015659 19605 tablet_bootstrap.cc:492] T f7d1ccc38ee8428aa16f7c82b86c0f3c P 895c9e1595924e769a25faeed7e34644: No bootstrap required, opened a new log
I20260129 02:57:44.016134 19605 ts_tablet_manager.cc:1403] T f7d1ccc38ee8428aa16f7c82b86c0f3c P 895c9e1595924e769a25faeed7e34644: Time spent bootstrapping tablet: real 0.031s user 0.010s sys 0.003s
I20260129 02:57:44.016883 19603 tablet_bootstrap.cc:492] T 9d5a1dc0b90642309290802698ec59f2 P 309a53f12a534385a7e5a88fb373722a: No bootstrap required, opened a new log
I20260129 02:57:44.017434 19603 ts_tablet_manager.cc:1403] T 9d5a1dc0b90642309290802698ec59f2 P 309a53f12a534385a7e5a88fb373722a: Time spent bootstrapping tablet: real 0.019s user 0.016s sys 0.000s
I20260129 02:57:44.018733 19605 raft_consensus.cc:359] T f7d1ccc38ee8428aa16f7c82b86c0f3c P 895c9e1595924e769a25faeed7e34644 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6aef613bf534ff9ae6ddbba06bb3cf1" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 45187 } } peers { permanent_uuid: "895c9e1595924e769a25faeed7e34644" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 40083 } } peers { permanent_uuid: "309a53f12a534385a7e5a88fb373722a" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 34213 } }
I20260129 02:57:44.019389 19605 raft_consensus.cc:385] T f7d1ccc38ee8428aa16f7c82b86c0f3c P 895c9e1595924e769a25faeed7e34644 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260129 02:57:44.019381 19604 tablet_bootstrap.cc:492] T 9d5a1dc0b90642309290802698ec59f2 P a6aef613bf534ff9ae6ddbba06bb3cf1: No bootstrap required, opened a new log
I20260129 02:57:44.019752 19605 raft_consensus.cc:740] T f7d1ccc38ee8428aa16f7c82b86c0f3c P 895c9e1595924e769a25faeed7e34644 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 895c9e1595924e769a25faeed7e34644, State: Initialized, Role: FOLLOWER
I20260129 02:57:44.019997 19604 ts_tablet_manager.cc:1403] T 9d5a1dc0b90642309290802698ec59f2 P a6aef613bf534ff9ae6ddbba06bb3cf1: Time spent bootstrapping tablet: real 0.021s user 0.011s sys 0.007s
I20260129 02:57:44.020131 19603 raft_consensus.cc:359] T 9d5a1dc0b90642309290802698ec59f2 P 309a53f12a534385a7e5a88fb373722a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6aef613bf534ff9ae6ddbba06bb3cf1" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 45187 } } peers { permanent_uuid: "895c9e1595924e769a25faeed7e34644" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 40083 } } peers { permanent_uuid: "309a53f12a534385a7e5a88fb373722a" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 34213 } }
I20260129 02:57:44.020856 19603 raft_consensus.cc:385] T 9d5a1dc0b90642309290802698ec59f2 P 309a53f12a534385a7e5a88fb373722a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260129 02:57:44.020524 19605 consensus_queue.cc:260] T f7d1ccc38ee8428aa16f7c82b86c0f3c P 895c9e1595924e769a25faeed7e34644 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6aef613bf534ff9ae6ddbba06bb3cf1" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 45187 } } peers { permanent_uuid: "895c9e1595924e769a25faeed7e34644" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 40083 } } peers { permanent_uuid: "309a53f12a534385a7e5a88fb373722a" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 34213 } }
I20260129 02:57:44.021202 19603 raft_consensus.cc:740] T 9d5a1dc0b90642309290802698ec59f2 P 309a53f12a534385a7e5a88fb373722a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 309a53f12a534385a7e5a88fb373722a, State: Initialized, Role: FOLLOWER
I20260129 02:57:44.022991 19604 raft_consensus.cc:359] T 9d5a1dc0b90642309290802698ec59f2 P a6aef613bf534ff9ae6ddbba06bb3cf1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6aef613bf534ff9ae6ddbba06bb3cf1" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 45187 } } peers { permanent_uuid: "895c9e1595924e769a25faeed7e34644" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 40083 } } peers { permanent_uuid: "309a53f12a534385a7e5a88fb373722a" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 34213 } }
I20260129 02:57:44.023716 19604 raft_consensus.cc:385] T 9d5a1dc0b90642309290802698ec59f2 P a6aef613bf534ff9ae6ddbba06bb3cf1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260129 02:57:44.023932 19605 ts_tablet_manager.cc:1434] T f7d1ccc38ee8428aa16f7c82b86c0f3c P 895c9e1595924e769a25faeed7e34644: Time spent starting tablet: real 0.008s user 0.007s sys 0.000s
I20260129 02:57:44.024044 19604 raft_consensus.cc:740] T 9d5a1dc0b90642309290802698ec59f2 P a6aef613bf534ff9ae6ddbba06bb3cf1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a6aef613bf534ff9ae6ddbba06bb3cf1, State: Initialized, Role: FOLLOWER
I20260129 02:57:44.025135 19605 tablet_bootstrap.cc:492] T c4689cdf800d479ba8645b96278c1713 P 895c9e1595924e769a25faeed7e34644: Bootstrap starting.
I20260129 02:57:44.024856 19604 consensus_queue.cc:260] T 9d5a1dc0b90642309290802698ec59f2 P a6aef613bf534ff9ae6ddbba06bb3cf1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6aef613bf534ff9ae6ddbba06bb3cf1" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 45187 } } peers { permanent_uuid: "895c9e1595924e769a25faeed7e34644" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 40083 } } peers { permanent_uuid: "309a53f12a534385a7e5a88fb373722a" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 34213 } }
I20260129 02:57:44.027534 19604 ts_tablet_manager.cc:1434] T 9d5a1dc0b90642309290802698ec59f2 P a6aef613bf534ff9ae6ddbba06bb3cf1: Time spent starting tablet: real 0.007s user 0.005s sys 0.000s
I20260129 02:57:44.028515 19604 tablet_bootstrap.cc:492] T 3e4edd649791453f902a5ac6d5658c12 P a6aef613bf534ff9ae6ddbba06bb3cf1: Bootstrap starting.
I20260129 02:57:44.030346 19610 raft_consensus.cc:493] T e5ce9ed62ba242cabb3d6e5371679ba7 P a6aef613bf534ff9ae6ddbba06bb3cf1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260129 02:57:44.030824 19610 raft_consensus.cc:515] T e5ce9ed62ba242cabb3d6e5371679ba7 P a6aef613bf534ff9ae6ddbba06bb3cf1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6aef613bf534ff9ae6ddbba06bb3cf1" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 45187 } } peers { permanent_uuid: "895c9e1595924e769a25faeed7e34644" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 40083 } } peers { permanent_uuid: "309a53f12a534385a7e5a88fb373722a" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 34213 } }
I20260129 02:57:44.032356 19605 tablet_bootstrap.cc:654] T c4689cdf800d479ba8645b96278c1713 P 895c9e1595924e769a25faeed7e34644: Neither blocks nor log segments found. Creating new log.
I20260129 02:57:44.034246 19610 leader_election.cc:290] T e5ce9ed62ba242cabb3d6e5371679ba7 P a6aef613bf534ff9ae6ddbba06bb3cf1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 895c9e1595924e769a25faeed7e34644 (127.18.211.2:40083), 309a53f12a534385a7e5a88fb373722a (127.18.211.3:34213)
I20260129 02:57:44.045723 19609 raft_consensus.cc:493] T f7d1ccc38ee8428aa16f7c82b86c0f3c P 895c9e1595924e769a25faeed7e34644 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260129 02:57:44.045396 19603 consensus_queue.cc:260] T 9d5a1dc0b90642309290802698ec59f2 P 309a53f12a534385a7e5a88fb373722a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6aef613bf534ff9ae6ddbba06bb3cf1" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 45187 } } peers { permanent_uuid: "895c9e1595924e769a25faeed7e34644" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 40083 } } peers { permanent_uuid: "309a53f12a534385a7e5a88fb373722a" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 34213 } }
I20260129 02:57:44.045902 19604 tablet_bootstrap.cc:654] T 3e4edd649791453f902a5ac6d5658c12 P a6aef613bf534ff9ae6ddbba06bb3cf1: Neither blocks nor log segments found. Creating new log.
I20260129 02:57:44.046172 19609 raft_consensus.cc:515] T f7d1ccc38ee8428aa16f7c82b86c0f3c P 895c9e1595924e769a25faeed7e34644 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6aef613bf534ff9ae6ddbba06bb3cf1" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 45187 } } peers { permanent_uuid: "895c9e1595924e769a25faeed7e34644" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 40083 } } peers { permanent_uuid: "309a53f12a534385a7e5a88fb373722a" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 34213 } }
I20260129 02:57:44.047984 19611 raft_consensus.cc:493] T c4689cdf800d479ba8645b96278c1713 P 309a53f12a534385a7e5a88fb373722a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260129 02:57:44.048425 19611 raft_consensus.cc:515] T c4689cdf800d479ba8645b96278c1713 P 309a53f12a534385a7e5a88fb373722a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6aef613bf534ff9ae6ddbba06bb3cf1" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 45187 } } peers { permanent_uuid: "895c9e1595924e769a25faeed7e34644" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 40083 } } peers { permanent_uuid: "309a53f12a534385a7e5a88fb373722a" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 34213 } }
I20260129 02:57:44.050261 19611 leader_election.cc:290] T c4689cdf800d479ba8645b96278c1713 P 309a53f12a534385a7e5a88fb373722a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a6aef613bf534ff9ae6ddbba06bb3cf1 (127.18.211.1:45187), 895c9e1595924e769a25faeed7e34644 (127.18.211.2:40083)
I20260129 02:57:44.051367 19411 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c4689cdf800d479ba8645b96278c1713" candidate_uuid: "309a53f12a534385a7e5a88fb373722a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a6aef613bf534ff9ae6ddbba06bb3cf1" is_pre_election: true
I20260129 02:57:44.052090 19411 raft_consensus.cc:2468] T c4689cdf800d479ba8645b96278c1713 P a6aef613bf534ff9ae6ddbba06bb3cf1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 309a53f12a534385a7e5a88fb373722a in term 0.
I20260129 02:57:44.052241 19609 leader_election.cc:290] T f7d1ccc38ee8428aa16f7c82b86c0f3c P 895c9e1595924e769a25faeed7e34644 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a6aef613bf534ff9ae6ddbba06bb3cf1 (127.18.211.1:45187), 309a53f12a534385a7e5a88fb373722a (127.18.211.3:34213)
I20260129 02:57:44.053730 19524 leader_election.cc:304] T c4689cdf800d479ba8645b96278c1713 P 309a53f12a534385a7e5a88fb373722a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 309a53f12a534385a7e5a88fb373722a, a6aef613bf534ff9ae6ddbba06bb3cf1; no voters:
I20260129 02:57:44.053977 19560 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f7d1ccc38ee8428aa16f7c82b86c0f3c" candidate_uuid: "895c9e1595924e769a25faeed7e34644" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "309a53f12a534385a7e5a88fb373722a" is_pre_election: true
I20260129 02:57:44.055284 19621 raft_consensus.cc:2804] T c4689cdf800d479ba8645b96278c1713 P 309a53f12a534385a7e5a88fb373722a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260129 02:57:44.055395 19486 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c4689cdf800d479ba8645b96278c1713" candidate_uuid: "309a53f12a534385a7e5a88fb373722a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "895c9e1595924e769a25faeed7e34644" is_pre_election: true
W20260129 02:57:44.056593 19450 leader_election.cc:343] T f7d1ccc38ee8428aa16f7c82b86c0f3c P 895c9e1595924e769a25faeed7e34644 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 309a53f12a534385a7e5a88fb373722a (127.18.211.3:34213): Illegal state: must be running to vote when last-logged opid is not known
I20260129 02:57:44.055727 19621 raft_consensus.cc:493] T c4689cdf800d479ba8645b96278c1713 P 309a53f12a534385a7e5a88fb373722a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260129 02:57:44.057233 19603 ts_tablet_manager.cc:1434] T 9d5a1dc0b90642309290802698ec59f2 P 309a53f12a534385a7e5a88fb373722a: Time spent starting tablet: real 0.039s user 0.008s sys 0.000s
I20260129 02:57:44.057560 19621 raft_consensus.cc:3060] T c4689cdf800d479ba8645b96278c1713 P 309a53f12a534385a7e5a88fb373722a [term 0 FOLLOWER]: Advancing to term 1
W20260129 02:57:44.058231 19524 leader_election.cc:343] T c4689cdf800d479ba8645b96278c1713 P 309a53f12a534385a7e5a88fb373722a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 895c9e1595924e769a25faeed7e34644 (127.18.211.2:40083): Illegal state: must be running to vote when last-logged opid is not known
I20260129 02:57:44.056843 19411 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f7d1ccc38ee8428aa16f7c82b86c0f3c" candidate_uuid: "895c9e1595924e769a25faeed7e34644" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a6aef613bf534ff9ae6ddbba06bb3cf1" is_pre_election: true
I20260129 02:57:44.059294 19411 raft_consensus.cc:2468] T f7d1ccc38ee8428aa16f7c82b86c0f3c P a6aef613bf534ff9ae6ddbba06bb3cf1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 895c9e1595924e769a25faeed7e34644 in term 0.
I20260129 02:57:44.060427 19450 leader_election.cc:304] T f7d1ccc38ee8428aa16f7c82b86c0f3c P 895c9e1595924e769a25faeed7e34644 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 895c9e1595924e769a25faeed7e34644, a6aef613bf534ff9ae6ddbba06bb3cf1; no voters: 309a53f12a534385a7e5a88fb373722a
I20260129 02:57:44.065163 19621 raft_consensus.cc:515] T c4689cdf800d479ba8645b96278c1713 P 309a53f12a534385a7e5a88fb373722a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6aef613bf534ff9ae6ddbba06bb3cf1" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 45187 } } peers { permanent_uuid: "895c9e1595924e769a25faeed7e34644" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 40083 } } peers { permanent_uuid: "309a53f12a534385a7e5a88fb373722a" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 34213 } }
I20260129 02:57:44.066061 19609 raft_consensus.cc:2804] T f7d1ccc38ee8428aa16f7c82b86c0f3c P 895c9e1595924e769a25faeed7e34644 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260129 02:57:44.068321 19609 raft_consensus.cc:493] T f7d1ccc38ee8428aa16f7c82b86c0f3c P 895c9e1595924e769a25faeed7e34644 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260129 02:57:44.068814 19609 raft_consensus.cc:3060] T f7d1ccc38ee8428aa16f7c82b86c0f3c P 895c9e1595924e769a25faeed7e34644 [term 0 FOLLOWER]: Advancing to term 1
I20260129 02:57:44.069628 19603 tablet_bootstrap.cc:492] T a78e86f5781f41bba56333c01607277a P 309a53f12a534385a7e5a88fb373722a: Bootstrap starting.
I20260129 02:57:44.069805 19486 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e5ce9ed62ba242cabb3d6e5371679ba7" candidate_uuid: "a6aef613bf534ff9ae6ddbba06bb3cf1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "895c9e1595924e769a25faeed7e34644" is_pre_election: true
I20260129 02:57:44.070406 19486 raft_consensus.cc:2468] T e5ce9ed62ba242cabb3d6e5371679ba7 P 895c9e1595924e769a25faeed7e34644 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a6aef613bf534ff9ae6ddbba06bb3cf1 in term 0.
I20260129 02:57:44.067823 19605 tablet_bootstrap.cc:492] T c4689cdf800d479ba8645b96278c1713 P 895c9e1595924e769a25faeed7e34644: No bootstrap required, opened a new log
I20260129 02:57:44.071749 19375 leader_election.cc:304] T e5ce9ed62ba242cabb3d6e5371679ba7 P a6aef613bf534ff9ae6ddbba06bb3cf1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 895c9e1595924e769a25faeed7e34644, a6aef613bf534ff9ae6ddbba06bb3cf1; no voters:
I20260129 02:57:44.072741 19610 raft_consensus.cc:2804] T e5ce9ed62ba242cabb3d6e5371679ba7 P a6aef613bf534ff9ae6ddbba06bb3cf1 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260129 02:57:44.073240 19610 raft_consensus.cc:493] T e5ce9ed62ba242cabb3d6e5371679ba7 P a6aef613bf534ff9ae6ddbba06bb3cf1 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260129 02:57:44.073674 19610 raft_consensus.cc:3060] T e5ce9ed62ba242cabb3d6e5371679ba7 P a6aef613bf534ff9ae6ddbba06bb3cf1 [term 0 FOLLOWER]: Advancing to term 1
I20260129 02:57:44.072175 19605 ts_tablet_manager.cc:1403] T c4689cdf800d479ba8645b96278c1713 P 895c9e1595924e769a25faeed7e34644: Time spent bootstrapping tablet: real 0.047s user 0.011s sys 0.006s
I20260129 02:57:44.077152 19603 tablet_bootstrap.cc:654] T a78e86f5781f41bba56333c01607277a P 309a53f12a534385a7e5a88fb373722a: Neither blocks nor log segments found. Creating new log.
I20260129 02:57:44.079988 19605 raft_consensus.cc:359] T c4689cdf800d479ba8645b96278c1713 P 895c9e1595924e769a25faeed7e34644 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6aef613bf534ff9ae6ddbba06bb3cf1" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 45187 } } peers { permanent_uuid: "895c9e1595924e769a25faeed7e34644" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 40083 } } peers { permanent_uuid: "309a53f12a534385a7e5a88fb373722a" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 34213 } }
I20260129 02:57:44.080678 19605 raft_consensus.cc:385] T c4689cdf800d479ba8645b96278c1713 P 895c9e1595924e769a25faeed7e34644 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260129 02:57:44.080942 19605 raft_consensus.cc:740] T c4689cdf800d479ba8645b96278c1713 P 895c9e1595924e769a25faeed7e34644 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 895c9e1595924e769a25faeed7e34644, State: Initialized, Role: FOLLOWER
I20260129 02:57:44.082449 19560 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e5ce9ed62ba242cabb3d6e5371679ba7" candidate_uuid: "a6aef613bf534ff9ae6ddbba06bb3cf1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "309a53f12a534385a7e5a88fb373722a" is_pre_election: true
I20260129 02:57:44.082998 19560 raft_consensus.cc:2468] T e5ce9ed62ba242cabb3d6e5371679ba7 P 309a53f12a534385a7e5a88fb373722a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a6aef613bf534ff9ae6ddbba06bb3cf1 in term 0.
I20260129 02:57:44.082863 19610 raft_consensus.cc:515] T e5ce9ed62ba242cabb3d6e5371679ba7 P a6aef613bf534ff9ae6ddbba06bb3cf1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6aef613bf534ff9ae6ddbba06bb3cf1" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 45187 } } peers { permanent_uuid: "895c9e1595924e769a25faeed7e34644" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 40083 } } peers { permanent_uuid: "309a53f12a534385a7e5a88fb373722a" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 34213 } }
I20260129 02:57:44.085327 19621 leader_election.cc:290] T c4689cdf800d479ba8645b96278c1713 P 309a53f12a534385a7e5a88fb373722a [CANDIDATE]: Term 1 election: Requested vote from peers a6aef613bf534ff9ae6ddbba06bb3cf1 (127.18.211.1:45187), 895c9e1595924e769a25faeed7e34644 (127.18.211.2:40083)
I20260129 02:57:44.085970 19609 raft_consensus.cc:515] T f7d1ccc38ee8428aa16f7c82b86c0f3c P 895c9e1595924e769a25faeed7e34644 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6aef613bf534ff9ae6ddbba06bb3cf1" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 45187 } } peers { permanent_uuid: "895c9e1595924e769a25faeed7e34644" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 40083 } } peers { permanent_uuid: "309a53f12a534385a7e5a88fb373722a" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 34213 } }
I20260129 02:57:44.087114 19411 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c4689cdf800d479ba8645b96278c1713" candidate_uuid: "309a53f12a534385a7e5a88fb373722a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a6aef613bf534ff9ae6ddbba06bb3cf1"
I20260129 02:57:44.081593 19605 consensus_queue.cc:260] T c4689cdf800d479ba8645b96278c1713 P 895c9e1595924e769a25faeed7e34644 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6aef613bf534ff9ae6ddbba06bb3cf1" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 45187 } } peers { permanent_uuid: "895c9e1595924e769a25faeed7e34644" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 40083 } } peers { permanent_uuid: "309a53f12a534385a7e5a88fb373722a" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 34213 } }
I20260129 02:57:44.087833 19411 raft_consensus.cc:3060] T c4689cdf800d479ba8645b96278c1713 P a6aef613bf534ff9ae6ddbba06bb3cf1 [term 0 FOLLOWER]: Advancing to term 1
I20260129 02:57:44.089344 19609 leader_election.cc:290] T f7d1ccc38ee8428aa16f7c82b86c0f3c P 895c9e1595924e769a25faeed7e34644 [CANDIDATE]: Term 1 election: Requested vote from peers a6aef613bf534ff9ae6ddbba06bb3cf1 (127.18.211.1:45187), 309a53f12a534385a7e5a88fb373722a (127.18.211.3:34213)
I20260129 02:57:44.090976 19634 raft_consensus.cc:493] T 9d5a1dc0b90642309290802698ec59f2 P a6aef613bf534ff9ae6ddbba06bb3cf1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260129 02:57:44.091483 19634 raft_consensus.cc:515] T 9d5a1dc0b90642309290802698ec59f2 P a6aef613bf534ff9ae6ddbba06bb3cf1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6aef613bf534ff9ae6ddbba06bb3cf1" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 45187 } } peers { permanent_uuid: "895c9e1595924e769a25faeed7e34644" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 40083 } } peers { permanent_uuid: "309a53f12a534385a7e5a88fb373722a" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 34213 } }
I20260129 02:57:44.093051 19560 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f7d1ccc38ee8428aa16f7c82b86c0f3c" candidate_uuid: "895c9e1595924e769a25faeed7e34644" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "309a53f12a534385a7e5a88fb373722a"
I20260129 02:57:44.093657 19410 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f7d1ccc38ee8428aa16f7c82b86c0f3c" candidate_uuid: "895c9e1595924e769a25faeed7e34644" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a6aef613bf534ff9ae6ddbba06bb3cf1"
I20260129 02:57:44.094296 19410 raft_consensus.cc:3060] T f7d1ccc38ee8428aa16f7c82b86c0f3c P a6aef613bf534ff9ae6ddbba06bb3cf1 [term 0 FOLLOWER]: Advancing to term 1
W20260129 02:57:44.094354 19450 leader_election.cc:343] T f7d1ccc38ee8428aa16f7c82b86c0f3c P 895c9e1595924e769a25faeed7e34644 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 309a53f12a534385a7e5a88fb373722a (127.18.211.3:34213): Illegal state: must be running to vote when last-logged opid is not known
I20260129 02:57:44.095947 19486 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c4689cdf800d479ba8645b96278c1713" candidate_uuid: "309a53f12a534385a7e5a88fb373722a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "895c9e1595924e769a25faeed7e34644"
I20260129 02:57:44.098232 19485 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e5ce9ed62ba242cabb3d6e5371679ba7" candidate_uuid: "a6aef613bf534ff9ae6ddbba06bb3cf1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "895c9e1595924e769a25faeed7e34644"
I20260129 02:57:44.099198 19486 raft_consensus.cc:3060] T c4689cdf800d479ba8645b96278c1713 P 895c9e1595924e769a25faeed7e34644 [term 0 FOLLOWER]: Advancing to term 1
I20260129 02:57:44.099341 19485 raft_consensus.cc:3060] T e5ce9ed62ba242cabb3d6e5371679ba7 P 895c9e1595924e769a25faeed7e34644 [term 0 FOLLOWER]: Advancing to term 1
I20260129 02:57:44.101456 19605 ts_tablet_manager.cc:1434] T c4689cdf800d479ba8645b96278c1713 P 895c9e1595924e769a25faeed7e34644: Time spent starting tablet: real 0.024s user 0.007s sys 0.000s
I20260129 02:57:44.102409 19605 tablet_bootstrap.cc:492] T 9d5a1dc0b90642309290802698ec59f2 P 895c9e1595924e769a25faeed7e34644: Bootstrap starting.
I20260129 02:57:44.105065 19634 leader_election.cc:290] T 9d5a1dc0b90642309290802698ec59f2 P a6aef613bf534ff9ae6ddbba06bb3cf1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 895c9e1595924e769a25faeed7e34644 (127.18.211.2:40083), 309a53f12a534385a7e5a88fb373722a (127.18.211.3:34213)
I20260129 02:57:44.105018 19610 leader_election.cc:290] T e5ce9ed62ba242cabb3d6e5371679ba7 P a6aef613bf534ff9ae6ddbba06bb3cf1 [CANDIDATE]: Term 1 election: Requested vote from peers 895c9e1595924e769a25faeed7e34644 (127.18.211.2:40083), 309a53f12a534385a7e5a88fb373722a (127.18.211.3:34213)
I20260129 02:57:44.109239 19484 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9d5a1dc0b90642309290802698ec59f2" candidate_uuid: "a6aef613bf534ff9ae6ddbba06bb3cf1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "895c9e1595924e769a25faeed7e34644" is_pre_election: true
I20260129 02:57:44.112470 19560 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9d5a1dc0b90642309290802698ec59f2" candidate_uuid: "a6aef613bf534ff9ae6ddbba06bb3cf1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "309a53f12a534385a7e5a88fb373722a" is_pre_election: true
I20260129 02:57:44.113020 19560 raft_consensus.cc:2468] T 9d5a1dc0b90642309290802698ec59f2 P 309a53f12a534385a7e5a88fb373722a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a6aef613bf534ff9ae6ddbba06bb3cf1 in term 0.
I20260129 02:57:44.114656 19560 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e5ce9ed62ba242cabb3d6e5371679ba7" candidate_uuid: "a6aef613bf534ff9ae6ddbba06bb3cf1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "309a53f12a534385a7e5a88fb373722a"
I20260129 02:57:44.115151 19560 raft_consensus.cc:3060] T e5ce9ed62ba242cabb3d6e5371679ba7 P 309a53f12a534385a7e5a88fb373722a [term 0 FOLLOWER]: Advancing to term 1
I20260129 02:57:44.116286 19486 raft_consensus.cc:2468] T c4689cdf800d479ba8645b96278c1713 P 895c9e1595924e769a25faeed7e34644 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 309a53f12a534385a7e5a88fb373722a in term 1.
I20260129 02:57:44.121538 19560 raft_consensus.cc:2468] T e5ce9ed62ba242cabb3d6e5371679ba7 P 309a53f12a534385a7e5a88fb373722a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a6aef613bf534ff9ae6ddbba06bb3cf1 in term 1.
W20260129 02:57:44.122627 19375 leader_election.cc:343] T 9d5a1dc0b90642309290802698ec59f2 P a6aef613bf534ff9ae6ddbba06bb3cf1 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 895c9e1595924e769a25faeed7e34644 (127.18.211.2:40083): Illegal state: must be running to vote when last-logged opid is not known
I20260129 02:57:44.123394 19375 leader_election.cc:304] T 9d5a1dc0b90642309290802698ec59f2 P a6aef613bf534ff9ae6ddbba06bb3cf1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 309a53f12a534385a7e5a88fb373722a, a6aef613bf534ff9ae6ddbba06bb3cf1; no voters: 895c9e1595924e769a25faeed7e34644
I20260129 02:57:44.124508 19375 leader_election.cc:304] T e5ce9ed62ba242cabb3d6e5371679ba7 P a6aef613bf534ff9ae6ddbba06bb3cf1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 309a53f12a534385a7e5a88fb373722a, a6aef613bf534ff9ae6ddbba06bb3cf1; no voters:
I20260129 02:57:44.125073 19610 raft_consensus.cc:2804] T 9d5a1dc0b90642309290802698ec59f2 P a6aef613bf534ff9ae6ddbba06bb3cf1 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260129 02:57:44.125682 19634 raft_consensus.cc:2804] T e5ce9ed62ba242cabb3d6e5371679ba7 P a6aef613bf534ff9ae6ddbba06bb3cf1 [term 1 FOLLOWER]: Leader election won for term 1
I20260129 02:57:44.125790 19610 raft_consensus.cc:493] T 9d5a1dc0b90642309290802698ec59f2 P a6aef613bf534ff9ae6ddbba06bb3cf1 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260129 02:57:44.126243 19610 raft_consensus.cc:3060] T 9d5a1dc0b90642309290802698ec59f2 P a6aef613bf534ff9ae6ddbba06bb3cf1 [term 0 FOLLOWER]: Advancing to term 1
I20260129 02:57:44.130563 19524 leader_election.cc:304] T c4689cdf800d479ba8645b96278c1713 P 309a53f12a534385a7e5a88fb373722a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 309a53f12a534385a7e5a88fb373722a, 895c9e1595924e769a25faeed7e34644; no voters:
I20260129 02:57:44.131332 19621 raft_consensus.cc:2804] T c4689cdf800d479ba8645b96278c1713 P 309a53f12a534385a7e5a88fb373722a [term 1 FOLLOWER]: Leader election won for term 1
I20260129 02:57:44.131551 19603 tablet_bootstrap.cc:492] T a78e86f5781f41bba56333c01607277a P 309a53f12a534385a7e5a88fb373722a: No bootstrap required, opened a new log
I20260129 02:57:44.131810 19621 raft_consensus.cc:697] T c4689cdf800d479ba8645b96278c1713 P 309a53f12a534385a7e5a88fb373722a [term 1 LEADER]: Becoming Leader. State: Replica: 309a53f12a534385a7e5a88fb373722a, State: Running, Role: LEADER
I20260129 02:57:44.134218 19411 raft_consensus.cc:2468] T c4689cdf800d479ba8645b96278c1713 P a6aef613bf534ff9ae6ddbba06bb3cf1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 309a53f12a534385a7e5a88fb373722a in term 1.
I20260129 02:57:44.134006 19621 consensus_queue.cc:237] T c4689cdf800d479ba8645b96278c1713 P 309a53f12a534385a7e5a88fb373722a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6aef613bf534ff9ae6ddbba06bb3cf1" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 45187 } } peers { permanent_uuid: "895c9e1595924e769a25faeed7e34644" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 40083 } } peers { permanent_uuid: "309a53f12a534385a7e5a88fb373722a" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 34213 } }
I20260129 02:57:44.135555 19485 raft_consensus.cc:2468] T e5ce9ed62ba242cabb3d6e5371679ba7 P 895c9e1595924e769a25faeed7e34644 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a6aef613bf534ff9ae6ddbba06bb3cf1 in term 1.
I20260129 02:57:44.132097 19603 ts_tablet_manager.cc:1403] T a78e86f5781f41bba56333c01607277a P 309a53f12a534385a7e5a88fb373722a: Time spent bootstrapping tablet: real 0.063s user 0.011s sys 0.009s
I20260129 02:57:44.140902 19410 raft_consensus.cc:2468] T f7d1ccc38ee8428aa16f7c82b86c0f3c P a6aef613bf534ff9ae6ddbba06bb3cf1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 895c9e1595924e769a25faeed7e34644 in term 1.
I20260129 02:57:44.142140 19450 leader_election.cc:304] T f7d1ccc38ee8428aa16f7c82b86c0f3c P 895c9e1595924e769a25faeed7e34644 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 895c9e1595924e769a25faeed7e34644, a6aef613bf534ff9ae6ddbba06bb3cf1; no voters: 309a53f12a534385a7e5a88fb373722a
I20260129 02:57:44.141592 19634 raft_consensus.cc:697] T e5ce9ed62ba242cabb3d6e5371679ba7 P a6aef613bf534ff9ae6ddbba06bb3cf1 [term 1 LEADER]: Becoming Leader. State: Replica: a6aef613bf534ff9ae6ddbba06bb3cf1, State: Running, Role: LEADER
I20260129 02:57:44.142937 19604 tablet_bootstrap.cc:492] T 3e4edd649791453f902a5ac6d5658c12 P a6aef613bf534ff9ae6ddbba06bb3cf1: No bootstrap required, opened a new log
I20260129 02:57:44.143301 19635 raft_consensus.cc:2804] T f7d1ccc38ee8428aa16f7c82b86c0f3c P 895c9e1595924e769a25faeed7e34644 [term 1 FOLLOWER]: Leader election won for term 1
I20260129 02:57:44.144279 19604 ts_tablet_manager.cc:1403] T 3e4edd649791453f902a5ac6d5658c12 P a6aef613bf534ff9ae6ddbba06bb3cf1: Time spent bootstrapping tablet: real 0.116s user 0.018s sys 0.019s
I20260129 02:57:44.144495 19635 raft_consensus.cc:697] T f7d1ccc38ee8428aa16f7c82b86c0f3c P 895c9e1595924e769a25faeed7e34644 [term 1 LEADER]: Becoming Leader. State: Replica: 895c9e1595924e769a25faeed7e34644, State: Running, Role: LEADER
I20260129 02:57:44.145627 19635 consensus_queue.cc:237] T f7d1ccc38ee8428aa16f7c82b86c0f3c P 895c9e1595924e769a25faeed7e34644 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6aef613bf534ff9ae6ddbba06bb3cf1" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 45187 } } peers { permanent_uuid: "895c9e1595924e769a25faeed7e34644" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 40083 } } peers { permanent_uuid: "309a53f12a534385a7e5a88fb373722a" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 34213 } }
I20260129 02:57:44.146754 19634 consensus_queue.cc:237] T e5ce9ed62ba242cabb3d6e5371679ba7 P a6aef613bf534ff9ae6ddbba06bb3cf1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6aef613bf534ff9ae6ddbba06bb3cf1" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 45187 } } peers { permanent_uuid: "895c9e1595924e769a25faeed7e34644" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 40083 } } peers { permanent_uuid: "309a53f12a534385a7e5a88fb373722a" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 34213 } }
I20260129 02:57:44.154556 19604 raft_consensus.cc:359] T 3e4edd649791453f902a5ac6d5658c12 P a6aef613bf534ff9ae6ddbba06bb3cf1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6aef613bf534ff9ae6ddbba06bb3cf1" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 45187 } } peers { permanent_uuid: "895c9e1595924e769a25faeed7e34644" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 40083 } } peers { permanent_uuid: "309a53f12a534385a7e5a88fb373722a" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 34213 } }
I20260129 02:57:44.155836 19604 raft_consensus.cc:385] T 3e4edd649791453f902a5ac6d5658c12 P a6aef613bf534ff9ae6ddbba06bb3cf1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260129 02:57:44.157467 19604 raft_consensus.cc:740] T 3e4edd649791453f902a5ac6d5658c12 P a6aef613bf534ff9ae6ddbba06bb3cf1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a6aef613bf534ff9ae6ddbba06bb3cf1, State: Initialized, Role: FOLLOWER
I20260129 02:57:44.156991 19603 raft_consensus.cc:359] T a78e86f5781f41bba56333c01607277a P 309a53f12a534385a7e5a88fb373722a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6aef613bf534ff9ae6ddbba06bb3cf1" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 45187 } } peers { permanent_uuid: "895c9e1595924e769a25faeed7e34644" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 40083 } } peers { permanent_uuid: "309a53f12a534385a7e5a88fb373722a" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 34213 } }
I20260129 02:57:44.158785 19605 tablet_bootstrap.cc:654] T 9d5a1dc0b90642309290802698ec59f2 P 895c9e1595924e769a25faeed7e34644: Neither blocks nor log segments found. Creating new log.
I20260129 02:57:44.159396 19603 raft_consensus.cc:385] T a78e86f5781f41bba56333c01607277a P 309a53f12a534385a7e5a88fb373722a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260129 02:57:44.163805 19603 raft_consensus.cc:740] T a78e86f5781f41bba56333c01607277a P 309a53f12a534385a7e5a88fb373722a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 309a53f12a534385a7e5a88fb373722a, State: Initialized, Role: FOLLOWER
I20260129 02:57:44.165186 19603 consensus_queue.cc:260] T a78e86f5781f41bba56333c01607277a P 309a53f12a534385a7e5a88fb373722a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6aef613bf534ff9ae6ddbba06bb3cf1" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 45187 } } peers { permanent_uuid: "895c9e1595924e769a25faeed7e34644" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 40083 } } peers { permanent_uuid: "309a53f12a534385a7e5a88fb373722a" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 34213 } }
I20260129 02:57:44.159427 19604 consensus_queue.cc:260] T 3e4edd649791453f902a5ac6d5658c12 P a6aef613bf534ff9ae6ddbba06bb3cf1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6aef613bf534ff9ae6ddbba06bb3cf1" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 45187 } } peers { permanent_uuid: "895c9e1595924e769a25faeed7e34644" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 40083 } } peers { permanent_uuid: "309a53f12a534385a7e5a88fb373722a" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 34213 } }
I20260129 02:57:44.172336 19610 raft_consensus.cc:515] T 9d5a1dc0b90642309290802698ec59f2 P a6aef613bf534ff9ae6ddbba06bb3cf1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6aef613bf534ff9ae6ddbba06bb3cf1" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 45187 } } peers { permanent_uuid: "895c9e1595924e769a25faeed7e34644" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 40083 } } peers { permanent_uuid: "309a53f12a534385a7e5a88fb373722a" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 34213 } }
I20260129 02:57:44.176318 19603 ts_tablet_manager.cc:1434] T a78e86f5781f41bba56333c01607277a P 309a53f12a534385a7e5a88fb373722a: Time spent starting tablet: real 0.037s user 0.009s sys 0.000s
I20260129 02:57:44.176596 19610 leader_election.cc:290] T 9d5a1dc0b90642309290802698ec59f2 P a6aef613bf534ff9ae6ddbba06bb3cf1 [CANDIDATE]: Term 1 election: Requested vote from peers 895c9e1595924e769a25faeed7e34644 (127.18.211.2:40083), 309a53f12a534385a7e5a88fb373722a (127.18.211.3:34213)
I20260129 02:57:44.177191 19603 tablet_bootstrap.cc:492] T 3e4edd649791453f902a5ac6d5658c12 P 309a53f12a534385a7e5a88fb373722a: Bootstrap starting.
I20260129 02:57:44.178578 19604 ts_tablet_manager.cc:1434] T 3e4edd649791453f902a5ac6d5658c12 P a6aef613bf534ff9ae6ddbba06bb3cf1: Time spent starting tablet: real 0.034s user 0.006s sys 0.003s
I20260129 02:57:44.179728 19604 tablet_bootstrap.cc:492] T 2b5edf7ea566411caea265dcfff5a86a P a6aef613bf534ff9ae6ddbba06bb3cf1: Bootstrap starting.
I20260129 02:57:44.180646 19485 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9d5a1dc0b90642309290802698ec59f2" candidate_uuid: "a6aef613bf534ff9ae6ddbba06bb3cf1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "895c9e1595924e769a25faeed7e34644"
I20260129 02:57:44.181265 19560 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9d5a1dc0b90642309290802698ec59f2" candidate_uuid: "a6aef613bf534ff9ae6ddbba06bb3cf1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "309a53f12a534385a7e5a88fb373722a"
I20260129 02:57:44.181779 19560 raft_consensus.cc:3060] T 9d5a1dc0b90642309290802698ec59f2 P 309a53f12a534385a7e5a88fb373722a [term 0 FOLLOWER]: Advancing to term 1
W20260129 02:57:44.182176 19375 leader_election.cc:343] T 9d5a1dc0b90642309290802698ec59f2 P a6aef613bf534ff9ae6ddbba06bb3cf1 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 895c9e1595924e769a25faeed7e34644 (127.18.211.2:40083): Illegal state: must be running to vote when last-logged opid is not known
I20260129 02:57:44.190634 19560 raft_consensus.cc:2468] T 9d5a1dc0b90642309290802698ec59f2 P 309a53f12a534385a7e5a88fb373722a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a6aef613bf534ff9ae6ddbba06bb3cf1 in term 1.
I20260129 02:57:44.192399 19375 leader_election.cc:304] T 9d5a1dc0b90642309290802698ec59f2 P a6aef613bf534ff9ae6ddbba06bb3cf1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 309a53f12a534385a7e5a88fb373722a, a6aef613bf534ff9ae6ddbba06bb3cf1; no voters: 895c9e1595924e769a25faeed7e34644
I20260129 02:57:44.194342 19610 raft_consensus.cc:2804] T 9d5a1dc0b90642309290802698ec59f2 P a6aef613bf534ff9ae6ddbba06bb3cf1 [term 1 FOLLOWER]: Leader election won for term 1
I20260129 02:57:44.195888 19610 raft_consensus.cc:697] T 9d5a1dc0b90642309290802698ec59f2 P a6aef613bf534ff9ae6ddbba06bb3cf1 [term 1 LEADER]: Becoming Leader. State: Replica: a6aef613bf534ff9ae6ddbba06bb3cf1, State: Running, Role: LEADER
I20260129 02:57:44.199707 19610 consensus_queue.cc:237] T 9d5a1dc0b90642309290802698ec59f2 P a6aef613bf534ff9ae6ddbba06bb3cf1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6aef613bf534ff9ae6ddbba06bb3cf1" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 45187 } } peers { permanent_uuid: "895c9e1595924e769a25faeed7e34644" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 40083 } } peers { permanent_uuid: "309a53f12a534385a7e5a88fb373722a" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 34213 } }
I20260129 02:57:44.209846 19603 tablet_bootstrap.cc:654] T 3e4edd649791453f902a5ac6d5658c12 P 309a53f12a534385a7e5a88fb373722a: Neither blocks nor log segments found. Creating new log.
I20260129 02:57:44.210275 19604 tablet_bootstrap.cc:654] T 2b5edf7ea566411caea265dcfff5a86a P a6aef613bf534ff9ae6ddbba06bb3cf1: Neither blocks nor log segments found. Creating new log.
I20260129 02:57:44.210937 19605 tablet_bootstrap.cc:492] T 9d5a1dc0b90642309290802698ec59f2 P 895c9e1595924e769a25faeed7e34644: No bootstrap required, opened a new log
I20260129 02:57:44.211422 19605 ts_tablet_manager.cc:1403] T 9d5a1dc0b90642309290802698ec59f2 P 895c9e1595924e769a25faeed7e34644: Time spent bootstrapping tablet: real 0.109s user 0.007s sys 0.011s
I20260129 02:57:44.215476 19605 raft_consensus.cc:359] T 9d5a1dc0b90642309290802698ec59f2 P 895c9e1595924e769a25faeed7e34644 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6aef613bf534ff9ae6ddbba06bb3cf1" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 45187 } } peers { permanent_uuid: "895c9e1595924e769a25faeed7e34644" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 40083 } } peers { permanent_uuid: "309a53f12a534385a7e5a88fb373722a" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 34213 } }
I20260129 02:57:44.216557 19605 raft_consensus.cc:385] T 9d5a1dc0b90642309290802698ec59f2 P 895c9e1595924e769a25faeed7e34644 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260129 02:57:44.217015 19605 raft_consensus.cc:740] T 9d5a1dc0b90642309290802698ec59f2 P 895c9e1595924e769a25faeed7e34644 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 895c9e1595924e769a25faeed7e34644, State: Initialized, Role: FOLLOWER
I20260129 02:57:44.218283 19605 consensus_queue.cc:260] T 9d5a1dc0b90642309290802698ec59f2 P 895c9e1595924e769a25faeed7e34644 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6aef613bf534ff9ae6ddbba06bb3cf1" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 45187 } } peers { permanent_uuid: "895c9e1595924e769a25faeed7e34644" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 40083 } } peers { permanent_uuid: "309a53f12a534385a7e5a88fb373722a" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 34213 } }
I20260129 02:57:44.221802 19605 ts_tablet_manager.cc:1434] T 9d5a1dc0b90642309290802698ec59f2 P 895c9e1595924e769a25faeed7e34644: Time spent starting tablet: real 0.010s user 0.009s sys 0.000s
I20260129 02:57:44.223742 19605 tablet_bootstrap.cc:492] T 2b5edf7ea566411caea265dcfff5a86a P 895c9e1595924e769a25faeed7e34644: Bootstrap starting.
I20260129 02:57:44.224520 19604 tablet_bootstrap.cc:492] T 2b5edf7ea566411caea265dcfff5a86a P a6aef613bf534ff9ae6ddbba06bb3cf1: No bootstrap required, opened a new log
I20260129 02:57:44.224995 19604 ts_tablet_manager.cc:1403] T 2b5edf7ea566411caea265dcfff5a86a P a6aef613bf534ff9ae6ddbba06bb3cf1: Time spent bootstrapping tablet: real 0.045s user 0.020s sys 0.002s
I20260129 02:57:44.227662 19604 raft_consensus.cc:359] T 2b5edf7ea566411caea265dcfff5a86a P a6aef613bf534ff9ae6ddbba06bb3cf1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "309a53f12a534385a7e5a88fb373722a" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 34213 } } peers { permanent_uuid: "a6aef613bf534ff9ae6ddbba06bb3cf1" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 45187 } } peers { permanent_uuid: "895c9e1595924e769a25faeed7e34644" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 40083 } }
I20260129 02:57:44.228405 19604 raft_consensus.cc:385] T 2b5edf7ea566411caea265dcfff5a86a P a6aef613bf534ff9ae6ddbba06bb3cf1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260129 02:57:44.228785 19604 raft_consensus.cc:740] T 2b5edf7ea566411caea265dcfff5a86a P a6aef613bf534ff9ae6ddbba06bb3cf1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a6aef613bf534ff9ae6ddbba06bb3cf1, State: Initialized, Role: FOLLOWER
I20260129 02:57:44.229543 19604 consensus_queue.cc:260] T 2b5edf7ea566411caea265dcfff5a86a P a6aef613bf534ff9ae6ddbba06bb3cf1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "309a53f12a534385a7e5a88fb373722a" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 34213 } } peers { permanent_uuid: "a6aef613bf534ff9ae6ddbba06bb3cf1" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 45187 } } peers { permanent_uuid: "895c9e1595924e769a25faeed7e34644" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 40083 } }
I20260129 02:57:44.232108 19604 ts_tablet_manager.cc:1434] T 2b5edf7ea566411caea265dcfff5a86a P a6aef613bf534ff9ae6ddbba06bb3cf1: Time spent starting tablet: real 0.007s user 0.008s sys 0.000s
I20260129 02:57:44.232820 19605 tablet_bootstrap.cc:654] T 2b5edf7ea566411caea265dcfff5a86a P 895c9e1595924e769a25faeed7e34644: Neither blocks nor log segments found. Creating new log.
I20260129 02:57:44.233047 19604 tablet_bootstrap.cc:492] T a78e86f5781f41bba56333c01607277a P a6aef613bf534ff9ae6ddbba06bb3cf1: Bootstrap starting.
I20260129 02:57:44.233732 19308 catalog_manager.cc:5654] T f7d1ccc38ee8428aa16f7c82b86c0f3c P 895c9e1595924e769a25faeed7e34644 reported cstate change: term changed from 0 to 1, leader changed from <none> to 895c9e1595924e769a25faeed7e34644 (127.18.211.2). New cstate: current_term: 1 leader_uuid: "895c9e1595924e769a25faeed7e34644" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6aef613bf534ff9ae6ddbba06bb3cf1" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 45187 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "895c9e1595924e769a25faeed7e34644" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 40083 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "309a53f12a534385a7e5a88fb373722a" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 34213 } health_report { overall_health: UNKNOWN } } }
I20260129 02:57:44.235106 19306 catalog_manager.cc:5654] T e5ce9ed62ba242cabb3d6e5371679ba7 P a6aef613bf534ff9ae6ddbba06bb3cf1 reported cstate change: term changed from 0 to 1, leader changed from <none> to a6aef613bf534ff9ae6ddbba06bb3cf1 (127.18.211.1). New cstate: current_term: 1 leader_uuid: "a6aef613bf534ff9ae6ddbba06bb3cf1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6aef613bf534ff9ae6ddbba06bb3cf1" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 45187 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "895c9e1595924e769a25faeed7e34644" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 40083 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "309a53f12a534385a7e5a88fb373722a" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 34213 } health_report { overall_health: UNKNOWN } } }
I20260129 02:57:44.228227 19307 catalog_manager.cc:5654] T c4689cdf800d479ba8645b96278c1713 P 309a53f12a534385a7e5a88fb373722a reported cstate change: term changed from 0 to 1, leader changed from <none> to 309a53f12a534385a7e5a88fb373722a (127.18.211.3). New cstate: current_term: 1 leader_uuid: "309a53f12a534385a7e5a88fb373722a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6aef613bf534ff9ae6ddbba06bb3cf1" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 45187 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "895c9e1595924e769a25faeed7e34644" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 40083 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "309a53f12a534385a7e5a88fb373722a" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 34213 } health_report { overall_health: HEALTHY } } }
I20260129 02:57:44.251961 19605 tablet_bootstrap.cc:492] T 2b5edf7ea566411caea265dcfff5a86a P 895c9e1595924e769a25faeed7e34644: No bootstrap required, opened a new log
I20260129 02:57:44.254384 19605 ts_tablet_manager.cc:1403] T 2b5edf7ea566411caea265dcfff5a86a P 895c9e1595924e769a25faeed7e34644: Time spent bootstrapping tablet: real 0.031s user 0.014s sys 0.002s
I20260129 02:57:44.257412 19604 tablet_bootstrap.cc:654] T a78e86f5781f41bba56333c01607277a P a6aef613bf534ff9ae6ddbba06bb3cf1: Neither blocks nor log segments found. Creating new log.
I20260129 02:57:44.257863 19603 tablet_bootstrap.cc:492] T 3e4edd649791453f902a5ac6d5658c12 P 309a53f12a534385a7e5a88fb373722a: No bootstrap required, opened a new log
I20260129 02:57:44.259845 19603 ts_tablet_manager.cc:1403] T 3e4edd649791453f902a5ac6d5658c12 P 309a53f12a534385a7e5a88fb373722a: Time spent bootstrapping tablet: real 0.083s user 0.013s sys 0.017s
I20260129 02:57:44.257978 19605 raft_consensus.cc:359] T 2b5edf7ea566411caea265dcfff5a86a P 895c9e1595924e769a25faeed7e34644 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "309a53f12a534385a7e5a88fb373722a" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 34213 } } peers { permanent_uuid: "a6aef613bf534ff9ae6ddbba06bb3cf1" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 45187 } } peers { permanent_uuid: "895c9e1595924e769a25faeed7e34644" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 40083 } }
I20260129 02:57:44.264190 19605 raft_consensus.cc:385] T 2b5edf7ea566411caea265dcfff5a86a P 895c9e1595924e769a25faeed7e34644 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260129 02:57:44.264585 19605 raft_consensus.cc:740] T 2b5edf7ea566411caea265dcfff5a86a P 895c9e1595924e769a25faeed7e34644 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 895c9e1595924e769a25faeed7e34644, State: Initialized, Role: FOLLOWER
I20260129 02:57:44.264742 19604 tablet_bootstrap.cc:492] T a78e86f5781f41bba56333c01607277a P a6aef613bf534ff9ae6ddbba06bb3cf1: No bootstrap required, opened a new log
I20260129 02:57:44.265326 19604 ts_tablet_manager.cc:1403] T a78e86f5781f41bba56333c01607277a P a6aef613bf534ff9ae6ddbba06bb3cf1: Time spent bootstrapping tablet: real 0.032s user 0.011s sys 0.003s
I20260129 02:57:44.265834 19605 consensus_queue.cc:260] T 2b5edf7ea566411caea265dcfff5a86a P 895c9e1595924e769a25faeed7e34644 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "309a53f12a534385a7e5a88fb373722a" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 34213 } } peers { permanent_uuid: "a6aef613bf534ff9ae6ddbba06bb3cf1" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 45187 } } peers { permanent_uuid: "895c9e1595924e769a25faeed7e34644" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 40083 } }
I20260129 02:57:44.266629 19603 raft_consensus.cc:359] T 3e4edd649791453f902a5ac6d5658c12 P 309a53f12a534385a7e5a88fb373722a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6aef613bf534ff9ae6ddbba06bb3cf1" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 45187 } } peers { permanent_uuid: "895c9e1595924e769a25faeed7e34644" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 40083 } } peers { permanent_uuid: "309a53f12a534385a7e5a88fb373722a" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 34213 } }
I20260129 02:57:44.267391 19603 raft_consensus.cc:385] T 3e4edd649791453f902a5ac6d5658c12 P 309a53f12a534385a7e5a88fb373722a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260129 02:57:44.267729 19603 raft_consensus.cc:740] T 3e4edd649791453f902a5ac6d5658c12 P 309a53f12a534385a7e5a88fb373722a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 309a53f12a534385a7e5a88fb373722a, State: Initialized, Role: FOLLOWER
I20260129 02:57:44.268585 19605 ts_tablet_manager.cc:1434] T 2b5edf7ea566411caea265dcfff5a86a P 895c9e1595924e769a25faeed7e34644: Time spent starting tablet: real 0.014s user 0.002s sys 0.006s
I20260129 02:57:44.268361 19604 raft_consensus.cc:359] T a78e86f5781f41bba56333c01607277a P a6aef613bf534ff9ae6ddbba06bb3cf1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6aef613bf534ff9ae6ddbba06bb3cf1" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 45187 } } peers { permanent_uuid: "895c9e1595924e769a25faeed7e34644" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 40083 } } peers { permanent_uuid: "309a53f12a534385a7e5a88fb373722a" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 34213 } }
I20260129 02:57:44.268469 19603 consensus_queue.cc:260] T 3e4edd649791453f902a5ac6d5658c12 P 309a53f12a534385a7e5a88fb373722a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6aef613bf534ff9ae6ddbba06bb3cf1" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 45187 } } peers { permanent_uuid: "895c9e1595924e769a25faeed7e34644" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 40083 } } peers { permanent_uuid: "309a53f12a534385a7e5a88fb373722a" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 34213 } }
I20260129 02:57:44.269150 19604 raft_consensus.cc:385] T a78e86f5781f41bba56333c01607277a P a6aef613bf534ff9ae6ddbba06bb3cf1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260129 02:57:44.269562 19604 raft_consensus.cc:740] T a78e86f5781f41bba56333c01607277a P a6aef613bf534ff9ae6ddbba06bb3cf1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a6aef613bf534ff9ae6ddbba06bb3cf1, State: Initialized, Role: FOLLOWER
I20260129 02:57:44.269609 19611 raft_consensus.cc:493] T 2e068f6d10f04c9ea17dba1375ae9b3c P 309a53f12a534385a7e5a88fb373722a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260129 02:57:44.270226 19611 raft_consensus.cc:515] T 2e068f6d10f04c9ea17dba1375ae9b3c P 309a53f12a534385a7e5a88fb373722a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6aef613bf534ff9ae6ddbba06bb3cf1" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 45187 } } peers { permanent_uuid: "895c9e1595924e769a25faeed7e34644" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 40083 } } peers { permanent_uuid: "309a53f12a534385a7e5a88fb373722a" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 34213 } }
I20260129 02:57:44.271730 19603 ts_tablet_manager.cc:1434] T 3e4edd649791453f902a5ac6d5658c12 P 309a53f12a534385a7e5a88fb373722a: Time spent starting tablet: real 0.008s user 0.000s sys 0.005s
I20260129 02:57:44.272099 19611 leader_election.cc:290] T 2e068f6d10f04c9ea17dba1375ae9b3c P 309a53f12a534385a7e5a88fb373722a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a6aef613bf534ff9ae6ddbba06bb3cf1 (127.18.211.1:45187), 895c9e1595924e769a25faeed7e34644 (127.18.211.2:40083)
I20260129 02:57:44.272742 19603 tablet_bootstrap.cc:492] T 6431d8f509f84fa6a63a3aad36a6d39d P 309a53f12a534385a7e5a88fb373722a: Bootstrap starting.
I20260129 02:57:44.273478 19410 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2e068f6d10f04c9ea17dba1375ae9b3c" candidate_uuid: "309a53f12a534385a7e5a88fb373722a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a6aef613bf534ff9ae6ddbba06bb3cf1" is_pre_election: true
I20260129 02:57:44.274830 19485 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2e068f6d10f04c9ea17dba1375ae9b3c" candidate_uuid: "309a53f12a534385a7e5a88fb373722a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "895c9e1595924e769a25faeed7e34644" is_pre_election: true
I20260129 02:57:44.275460 19485 raft_consensus.cc:2468] T 2e068f6d10f04c9ea17dba1375ae9b3c P 895c9e1595924e769a25faeed7e34644 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 309a53f12a534385a7e5a88fb373722a in term 0.
W20260129 02:57:44.276517 19524 leader_election.cc:343] T 2e068f6d10f04c9ea17dba1375ae9b3c P 309a53f12a534385a7e5a88fb373722a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a6aef613bf534ff9ae6ddbba06bb3cf1 (127.18.211.1:45187): Illegal state: must be running to vote when last-logged opid is not known
I20260129 02:57:44.277758 19524 leader_election.cc:304] T 2e068f6d10f04c9ea17dba1375ae9b3c P 309a53f12a534385a7e5a88fb373722a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 309a53f12a534385a7e5a88fb373722a, 895c9e1595924e769a25faeed7e34644; no voters: a6aef613bf534ff9ae6ddbba06bb3cf1
I20260129 02:57:44.279263 19611 raft_consensus.cc:2804] T 2e068f6d10f04c9ea17dba1375ae9b3c P 309a53f12a534385a7e5a88fb373722a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260129 02:57:44.279632 19611 raft_consensus.cc:493] T 2e068f6d10f04c9ea17dba1375ae9b3c P 309a53f12a534385a7e5a88fb373722a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260129 02:57:44.280076 19611 raft_consensus.cc:3060] T 2e068f6d10f04c9ea17dba1375ae9b3c P 309a53f12a534385a7e5a88fb373722a [term 0 FOLLOWER]: Advancing to term 1
I20260129 02:57:44.288036 19603 tablet_bootstrap.cc:654] T 6431d8f509f84fa6a63a3aad36a6d39d P 309a53f12a534385a7e5a88fb373722a: Neither blocks nor log segments found. Creating new log.
I20260129 02:57:44.288343 19604 consensus_queue.cc:260] T a78e86f5781f41bba56333c01607277a P a6aef613bf534ff9ae6ddbba06bb3cf1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6aef613bf534ff9ae6ddbba06bb3cf1" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 45187 } } peers { permanent_uuid: "895c9e1595924e769a25faeed7e34644" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 40083 } } peers { permanent_uuid: "309a53f12a534385a7e5a88fb373722a" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 34213 } }
I20260129 02:57:44.291143 19604 ts_tablet_manager.cc:1434] T a78e86f5781f41bba56333c01607277a P a6aef613bf534ff9ae6ddbba06bb3cf1: Time spent starting tablet: real 0.025s user 0.005s sys 0.000s
I20260129 02:57:44.269583 19605 tablet_bootstrap.cc:492] T 3e4edd649791453f902a5ac6d5658c12 P 895c9e1595924e769a25faeed7e34644: Bootstrap starting.
I20260129 02:57:44.287458 19611 raft_consensus.cc:515] T 2e068f6d10f04c9ea17dba1375ae9b3c P 309a53f12a534385a7e5a88fb373722a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6aef613bf534ff9ae6ddbba06bb3cf1" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 45187 } } peers { permanent_uuid: "895c9e1595924e769a25faeed7e34644" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 40083 } } peers { permanent_uuid: "309a53f12a534385a7e5a88fb373722a" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 34213 } }
I20260129 02:57:44.294580 19410 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2e068f6d10f04c9ea17dba1375ae9b3c" candidate_uuid: "309a53f12a534385a7e5a88fb373722a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a6aef613bf534ff9ae6ddbba06bb3cf1"
I20260129 02:57:44.296522 19485 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2e068f6d10f04c9ea17dba1375ae9b3c" candidate_uuid: "309a53f12a534385a7e5a88fb373722a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "895c9e1595924e769a25faeed7e34644"
I20260129 02:57:44.297102 19485 raft_consensus.cc:3060] T 2e068f6d10f04c9ea17dba1375ae9b3c P 895c9e1595924e769a25faeed7e34644 [term 0 FOLLOWER]: Advancing to term 1
I20260129 02:57:44.302965 19605 tablet_bootstrap.cc:654] T 3e4edd649791453f902a5ac6d5658c12 P 895c9e1595924e769a25faeed7e34644: Neither blocks nor log segments found. Creating new log.
I20260129 02:57:44.303617 19610 raft_consensus.cc:493] T 6431d8f509f84fa6a63a3aad36a6d39d P a6aef613bf534ff9ae6ddbba06bb3cf1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260129 02:57:44.304081 19610 raft_consensus.cc:515] T 6431d8f509f84fa6a63a3aad36a6d39d P a6aef613bf534ff9ae6ddbba06bb3cf1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6aef613bf534ff9ae6ddbba06bb3cf1" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 45187 } } peers { permanent_uuid: "895c9e1595924e769a25faeed7e34644" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 40083 } } peers { permanent_uuid: "309a53f12a534385a7e5a88fb373722a" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 34213 } }
I20260129 02:57:44.305831 19610 leader_election.cc:290] T 6431d8f509f84fa6a63a3aad36a6d39d P a6aef613bf534ff9ae6ddbba06bb3cf1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 895c9e1595924e769a25faeed7e34644 (127.18.211.2:40083), 309a53f12a534385a7e5a88fb373722a (127.18.211.3:34213)
I20260129 02:57:44.306663 19485 raft_consensus.cc:2468] T 2e068f6d10f04c9ea17dba1375ae9b3c P 895c9e1595924e769a25faeed7e34644 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 309a53f12a534385a7e5a88fb373722a in term 1.
I20260129 02:57:44.307006 19486 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6431d8f509f84fa6a63a3aad36a6d39d" candidate_uuid: "a6aef613bf534ff9ae6ddbba06bb3cf1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "895c9e1595924e769a25faeed7e34644" is_pre_election: true
I20260129 02:57:44.307826 19524 leader_election.cc:304] T 2e068f6d10f04c9ea17dba1375ae9b3c P 309a53f12a534385a7e5a88fb373722a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 309a53f12a534385a7e5a88fb373722a, 895c9e1595924e769a25faeed7e34644; no voters:
I20260129 02:57:44.308439 19560 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6431d8f509f84fa6a63a3aad36a6d39d" candidate_uuid: "a6aef613bf534ff9ae6ddbba06bb3cf1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "309a53f12a534385a7e5a88fb373722a" is_pre_election: true
W20260129 02:57:44.308722 19375 leader_election.cc:343] T 6431d8f509f84fa6a63a3aad36a6d39d P a6aef613bf534ff9ae6ddbba06bb3cf1 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 895c9e1595924e769a25faeed7e34644 (127.18.211.2:40083): Illegal state: must be running to vote when last-logged opid is not known
I20260129 02:57:44.309245 19611 leader_election.cc:290] T 2e068f6d10f04c9ea17dba1375ae9b3c P 309a53f12a534385a7e5a88fb373722a [CANDIDATE]: Term 1 election: Requested vote from peers a6aef613bf534ff9ae6ddbba06bb3cf1 (127.18.211.1:45187), 895c9e1595924e769a25faeed7e34644 (127.18.211.2:40083)
I20260129 02:57:44.309516 19621 raft_consensus.cc:2804] T 2e068f6d10f04c9ea17dba1375ae9b3c P 309a53f12a534385a7e5a88fb373722a [term 1 FOLLOWER]: Leader election won for term 1
I20260129 02:57:44.313372 19621 raft_consensus.cc:697] T 2e068f6d10f04c9ea17dba1375ae9b3c P 309a53f12a534385a7e5a88fb373722a [term 1 LEADER]: Becoming Leader. State: Replica: 309a53f12a534385a7e5a88fb373722a, State: Running, Role: LEADER
I20260129 02:57:44.314507 19621 consensus_queue.cc:237] T 2e068f6d10f04c9ea17dba1375ae9b3c P 309a53f12a534385a7e5a88fb373722a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6aef613bf534ff9ae6ddbba06bb3cf1" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 45187 } } peers { permanent_uuid: "895c9e1595924e769a25faeed7e34644" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 40083 } } peers { permanent_uuid: "309a53f12a534385a7e5a88fb373722a" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 34213 } }
W20260129 02:57:44.310539 19375 leader_election.cc:343] T 6431d8f509f84fa6a63a3aad36a6d39d P a6aef613bf534ff9ae6ddbba06bb3cf1 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 309a53f12a534385a7e5a88fb373722a (127.18.211.3:34213): Illegal state: must be running to vote when last-logged opid is not known
I20260129 02:57:44.317766 19375 leader_election.cc:304] T 6431d8f509f84fa6a63a3aad36a6d39d P a6aef613bf534ff9ae6ddbba06bb3cf1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a6aef613bf534ff9ae6ddbba06bb3cf1; no voters: 309a53f12a534385a7e5a88fb373722a, 895c9e1595924e769a25faeed7e34644
I20260129 02:57:44.318806 19610 raft_consensus.cc:2749] T 6431d8f509f84fa6a63a3aad36a6d39d P a6aef613bf534ff9ae6ddbba06bb3cf1 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260129 02:57:44.310556 19603 tablet_bootstrap.cc:492] T 6431d8f509f84fa6a63a3aad36a6d39d P 309a53f12a534385a7e5a88fb373722a: No bootstrap required, opened a new log
W20260129 02:57:44.312414 19524 leader_election.cc:343] T 2e068f6d10f04c9ea17dba1375ae9b3c P 309a53f12a534385a7e5a88fb373722a [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer a6aef613bf534ff9ae6ddbba06bb3cf1 (127.18.211.1:45187): Illegal state: must be running to vote when last-logged opid is not known
I20260129 02:57:44.312806 19604 tablet_bootstrap.cc:492] T 2e068f6d10f04c9ea17dba1375ae9b3c P a6aef613bf534ff9ae6ddbba06bb3cf1: Bootstrap starting.
I20260129 02:57:44.319802 19603 ts_tablet_manager.cc:1403] T 6431d8f509f84fa6a63a3aad36a6d39d P 309a53f12a534385a7e5a88fb373722a: Time spent bootstrapping tablet: real 0.047s user 0.010s sys 0.007s
I20260129 02:57:44.323805 19603 raft_consensus.cc:359] T 6431d8f509f84fa6a63a3aad36a6d39d P 309a53f12a534385a7e5a88fb373722a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6aef613bf534ff9ae6ddbba06bb3cf1" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 45187 } } peers { permanent_uuid: "895c9e1595924e769a25faeed7e34644" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 40083 } } peers { permanent_uuid: "309a53f12a534385a7e5a88fb373722a" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 34213 } }
I20260129 02:57:44.325546 19603 raft_consensus.cc:385] T 6431d8f509f84fa6a63a3aad36a6d39d P 309a53f12a534385a7e5a88fb373722a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260129 02:57:44.325835 19603 raft_consensus.cc:740] T 6431d8f509f84fa6a63a3aad36a6d39d P 309a53f12a534385a7e5a88fb373722a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 309a53f12a534385a7e5a88fb373722a, State: Initialized, Role: FOLLOWER
I20260129 02:57:44.326385 19604 tablet_bootstrap.cc:654] T 2e068f6d10f04c9ea17dba1375ae9b3c P a6aef613bf534ff9ae6ddbba06bb3cf1: Neither blocks nor log segments found. Creating new log.
I20260129 02:57:44.326918 19605 tablet_bootstrap.cc:492] T 3e4edd649791453f902a5ac6d5658c12 P 895c9e1595924e769a25faeed7e34644: No bootstrap required, opened a new log
I20260129 02:57:44.326444 19603 consensus_queue.cc:260] T 6431d8f509f84fa6a63a3aad36a6d39d P 309a53f12a534385a7e5a88fb373722a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6aef613bf534ff9ae6ddbba06bb3cf1" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 45187 } } peers { permanent_uuid: "895c9e1595924e769a25faeed7e34644" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 40083 } } peers { permanent_uuid: "309a53f12a534385a7e5a88fb373722a" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 34213 } }
I20260129 02:57:44.327343 19605 ts_tablet_manager.cc:1403] T 3e4edd649791453f902a5ac6d5658c12 P 895c9e1595924e769a25faeed7e34644: Time spent bootstrapping tablet: real 0.058s user 0.008s sys 0.016s
I20260129 02:57:44.329082 19603 ts_tablet_manager.cc:1434] T 6431d8f509f84fa6a63a3aad36a6d39d P 309a53f12a534385a7e5a88fb373722a: Time spent starting tablet: real 0.008s user 0.006s sys 0.000s
I20260129 02:57:44.329907 19603 tablet_bootstrap.cc:492] T f7d1ccc38ee8428aa16f7c82b86c0f3c P 309a53f12a534385a7e5a88fb373722a: Bootstrap starting.
I20260129 02:57:44.329751 19605 raft_consensus.cc:359] T 3e4edd649791453f902a5ac6d5658c12 P 895c9e1595924e769a25faeed7e34644 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6aef613bf534ff9ae6ddbba06bb3cf1" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 45187 } } peers { permanent_uuid: "895c9e1595924e769a25faeed7e34644" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 40083 } } peers { permanent_uuid: "309a53f12a534385a7e5a88fb373722a" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 34213 } }
I20260129 02:57:44.330458 19605 raft_consensus.cc:385] T 3e4edd649791453f902a5ac6d5658c12 P 895c9e1595924e769a25faeed7e34644 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260129 02:57:44.330708 19605 raft_consensus.cc:740] T 3e4edd649791453f902a5ac6d5658c12 P 895c9e1595924e769a25faeed7e34644 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 895c9e1595924e769a25faeed7e34644, State: Initialized, Role: FOLLOWER
I20260129 02:57:44.331267 19605 consensus_queue.cc:260] T 3e4edd649791453f902a5ac6d5658c12 P 895c9e1595924e769a25faeed7e34644 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6aef613bf534ff9ae6ddbba06bb3cf1" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 45187 } } peers { permanent_uuid: "895c9e1595924e769a25faeed7e34644" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 40083 } } peers { permanent_uuid: "309a53f12a534385a7e5a88fb373722a" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 34213 } }
I20260129 02:57:44.333609 19605 ts_tablet_manager.cc:1434] T 3e4edd649791453f902a5ac6d5658c12 P 895c9e1595924e769a25faeed7e34644: Time spent starting tablet: real 0.006s user 0.000s sys 0.005s
I20260129 02:57:44.334523 19605 tablet_bootstrap.cc:492] T 6431d8f509f84fa6a63a3aad36a6d39d P 895c9e1595924e769a25faeed7e34644: Bootstrap starting.
I20260129 02:57:44.335930 19604 tablet_bootstrap.cc:492] T 2e068f6d10f04c9ea17dba1375ae9b3c P a6aef613bf534ff9ae6ddbba06bb3cf1: No bootstrap required, opened a new log
I20260129 02:57:44.336436 19603 tablet_bootstrap.cc:654] T f7d1ccc38ee8428aa16f7c82b86c0f3c P 309a53f12a534385a7e5a88fb373722a: Neither blocks nor log segments found. Creating new log.
I20260129 02:57:44.340022 19605 tablet_bootstrap.cc:654] T 6431d8f509f84fa6a63a3aad36a6d39d P 895c9e1595924e769a25faeed7e34644: Neither blocks nor log segments found. Creating new log.
I20260129 02:57:44.342247 19604 ts_tablet_manager.cc:1403] T 2e068f6d10f04c9ea17dba1375ae9b3c P a6aef613bf534ff9ae6ddbba06bb3cf1: Time spent bootstrapping tablet: real 0.050s user 0.011s sys 0.000s
I20260129 02:57:44.345548 19604 raft_consensus.cc:359] T 2e068f6d10f04c9ea17dba1375ae9b3c P a6aef613bf534ff9ae6ddbba06bb3cf1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6aef613bf534ff9ae6ddbba06bb3cf1" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 45187 } } peers { permanent_uuid: "895c9e1595924e769a25faeed7e34644" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 40083 } } peers { permanent_uuid: "309a53f12a534385a7e5a88fb373722a" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 34213 } }
I20260129 02:57:44.346367 19604 raft_consensus.cc:385] T 2e068f6d10f04c9ea17dba1375ae9b3c P a6aef613bf534ff9ae6ddbba06bb3cf1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260129 02:57:44.346679 19604 raft_consensus.cc:740] T 2e068f6d10f04c9ea17dba1375ae9b3c P a6aef613bf534ff9ae6ddbba06bb3cf1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a6aef613bf534ff9ae6ddbba06bb3cf1, State: Initialized, Role: FOLLOWER
I20260129 02:57:44.347431 19604 consensus_queue.cc:260] T 2e068f6d10f04c9ea17dba1375ae9b3c P a6aef613bf534ff9ae6ddbba06bb3cf1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6aef613bf534ff9ae6ddbba06bb3cf1" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 45187 } } peers { permanent_uuid: "895c9e1595924e769a25faeed7e34644" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 40083 } } peers { permanent_uuid: "309a53f12a534385a7e5a88fb373722a" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 34213 } }
I20260129 02:57:44.352244 19604 ts_tablet_manager.cc:1434] T 2e068f6d10f04c9ea17dba1375ae9b3c P a6aef613bf534ff9ae6ddbba06bb3cf1: Time spent starting tablet: real 0.010s user 0.007s sys 0.000s
I20260129 02:57:44.362218 19603 tablet_bootstrap.cc:492] T f7d1ccc38ee8428aa16f7c82b86c0f3c P 309a53f12a534385a7e5a88fb373722a: No bootstrap required, opened a new log
I20260129 02:57:44.362658 19603 ts_tablet_manager.cc:1403] T f7d1ccc38ee8428aa16f7c82b86c0f3c P 309a53f12a534385a7e5a88fb373722a: Time spent bootstrapping tablet: real 0.033s user 0.009s sys 0.002s
I20260129 02:57:44.365038 19603 raft_consensus.cc:359] T f7d1ccc38ee8428aa16f7c82b86c0f3c P 309a53f12a534385a7e5a88fb373722a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6aef613bf534ff9ae6ddbba06bb3cf1" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 45187 } } peers { permanent_uuid: "895c9e1595924e769a25faeed7e34644" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 40083 } } peers { permanent_uuid: "309a53f12a534385a7e5a88fb373722a" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 34213 } }
I20260129 02:57:44.365686 19603 raft_consensus.cc:385] T f7d1ccc38ee8428aa16f7c82b86c0f3c P 309a53f12a534385a7e5a88fb373722a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260129 02:57:44.365927 19603 raft_consensus.cc:740] T f7d1ccc38ee8428aa16f7c82b86c0f3c P 309a53f12a534385a7e5a88fb373722a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 309a53f12a534385a7e5a88fb373722a, State: Initialized, Role: FOLLOWER
I20260129 02:57:44.366462 19603 consensus_queue.cc:260] T f7d1ccc38ee8428aa16f7c82b86c0f3c P 309a53f12a534385a7e5a88fb373722a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6aef613bf534ff9ae6ddbba06bb3cf1" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 45187 } } peers { permanent_uuid: "895c9e1595924e769a25faeed7e34644" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 40083 } } peers { permanent_uuid: "309a53f12a534385a7e5a88fb373722a" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 34213 } }
I20260129 02:57:44.367030 19605 tablet_bootstrap.cc:492] T 6431d8f509f84fa6a63a3aad36a6d39d P 895c9e1595924e769a25faeed7e34644: No bootstrap required, opened a new log
I20260129 02:57:44.367450 19605 ts_tablet_manager.cc:1403] T 6431d8f509f84fa6a63a3aad36a6d39d P 895c9e1595924e769a25faeed7e34644: Time spent bootstrapping tablet: real 0.033s user 0.003s sys 0.023s
I20260129 02:57:44.368417 19603 ts_tablet_manager.cc:1434] T f7d1ccc38ee8428aa16f7c82b86c0f3c P 309a53f12a534385a7e5a88fb373722a: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20260129 02:57:44.370060 19605 raft_consensus.cc:359] T 6431d8f509f84fa6a63a3aad36a6d39d P 895c9e1595924e769a25faeed7e34644 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6aef613bf534ff9ae6ddbba06bb3cf1" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 45187 } } peers { permanent_uuid: "895c9e1595924e769a25faeed7e34644" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 40083 } } peers { permanent_uuid: "309a53f12a534385a7e5a88fb373722a" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 34213 } }
I20260129 02:57:44.370675 19605 raft_consensus.cc:385] T 6431d8f509f84fa6a63a3aad36a6d39d P 895c9e1595924e769a25faeed7e34644 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260129 02:57:44.370929 19605 raft_consensus.cc:740] T 6431d8f509f84fa6a63a3aad36a6d39d P 895c9e1595924e769a25faeed7e34644 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 895c9e1595924e769a25faeed7e34644, State: Initialized, Role: FOLLOWER
I20260129 02:57:44.371534 19605 consensus_queue.cc:260] T 6431d8f509f84fa6a63a3aad36a6d39d P 895c9e1595924e769a25faeed7e34644 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6aef613bf534ff9ae6ddbba06bb3cf1" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 45187 } } peers { permanent_uuid: "895c9e1595924e769a25faeed7e34644" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 40083 } } peers { permanent_uuid: "309a53f12a534385a7e5a88fb373722a" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 34213 } }
I20260129 02:57:44.372044 19611 raft_consensus.cc:493] T 6431d8f509f84fa6a63a3aad36a6d39d P 309a53f12a534385a7e5a88fb373722a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260129 02:57:44.372488 19611 raft_consensus.cc:515] T 6431d8f509f84fa6a63a3aad36a6d39d P 309a53f12a534385a7e5a88fb373722a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6aef613bf534ff9ae6ddbba06bb3cf1" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 45187 } } peers { permanent_uuid: "895c9e1595924e769a25faeed7e34644" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 40083 } } peers { permanent_uuid: "309a53f12a534385a7e5a88fb373722a" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 34213 } }
I20260129 02:57:44.373622 19605 ts_tablet_manager.cc:1434] T 6431d8f509f84fa6a63a3aad36a6d39d P 895c9e1595924e769a25faeed7e34644: Time spent starting tablet: real 0.006s user 0.001s sys 0.005s
I20260129 02:57:44.374265 19611 leader_election.cc:290] T 6431d8f509f84fa6a63a3aad36a6d39d P 309a53f12a534385a7e5a88fb373722a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a6aef613bf534ff9ae6ddbba06bb3cf1 (127.18.211.1:45187), 895c9e1595924e769a25faeed7e34644 (127.18.211.2:40083)
I20260129 02:57:44.371464 19308 catalog_manager.cc:5654] T 2e068f6d10f04c9ea17dba1375ae9b3c P 309a53f12a534385a7e5a88fb373722a reported cstate change: term changed from 0 to 1, leader changed from <none> to 309a53f12a534385a7e5a88fb373722a (127.18.211.3). New cstate: current_term: 1 leader_uuid: "309a53f12a534385a7e5a88fb373722a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6aef613bf534ff9ae6ddbba06bb3cf1" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 45187 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "895c9e1595924e769a25faeed7e34644" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 40083 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "309a53f12a534385a7e5a88fb373722a" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 34213 } health_report { overall_health: HEALTHY } } }
I20260129 02:57:44.375464 19485 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6431d8f509f84fa6a63a3aad36a6d39d" candidate_uuid: "309a53f12a534385a7e5a88fb373722a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "895c9e1595924e769a25faeed7e34644" is_pre_election: true
I20260129 02:57:44.376111 19485 raft_consensus.cc:2468] T 6431d8f509f84fa6a63a3aad36a6d39d P 895c9e1595924e769a25faeed7e34644 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 309a53f12a534385a7e5a88fb373722a in term 0.
I20260129 02:57:44.376659 19410 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6431d8f509f84fa6a63a3aad36a6d39d" candidate_uuid: "309a53f12a534385a7e5a88fb373722a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a6aef613bf534ff9ae6ddbba06bb3cf1" is_pre_election: true
I20260129 02:57:44.377426 19410 raft_consensus.cc:2468] T 6431d8f509f84fa6a63a3aad36a6d39d P a6aef613bf534ff9ae6ddbba06bb3cf1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 309a53f12a534385a7e5a88fb373722a in term 0.
I20260129 02:57:44.377259 19524 leader_election.cc:304] T 6431d8f509f84fa6a63a3aad36a6d39d P 309a53f12a534385a7e5a88fb373722a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 309a53f12a534385a7e5a88fb373722a, 895c9e1595924e769a25faeed7e34644; no voters:
I20260129 02:57:44.378583 19611 raft_consensus.cc:2804] T 6431d8f509f84fa6a63a3aad36a6d39d P 309a53f12a534385a7e5a88fb373722a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260129 02:57:44.378856 19611 raft_consensus.cc:493] T 6431d8f509f84fa6a63a3aad36a6d39d P 309a53f12a534385a7e5a88fb373722a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260129 02:57:44.379093 19611 raft_consensus.cc:3060] T 6431d8f509f84fa6a63a3aad36a6d39d P 309a53f12a534385a7e5a88fb373722a [term 0 FOLLOWER]: Advancing to term 1
I20260129 02:57:44.382974 19611 raft_consensus.cc:515] T 6431d8f509f84fa6a63a3aad36a6d39d P 309a53f12a534385a7e5a88fb373722a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6aef613bf534ff9ae6ddbba06bb3cf1" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 45187 } } peers { permanent_uuid: "895c9e1595924e769a25faeed7e34644" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 40083 } } peers { permanent_uuid: "309a53f12a534385a7e5a88fb373722a" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 34213 } }
I20260129 02:57:44.384501 19611 leader_election.cc:290] T 6431d8f509f84fa6a63a3aad36a6d39d P 309a53f12a534385a7e5a88fb373722a [CANDIDATE]: Term 1 election: Requested vote from peers a6aef613bf534ff9ae6ddbba06bb3cf1 (127.18.211.1:45187), 895c9e1595924e769a25faeed7e34644 (127.18.211.2:40083)
I20260129 02:57:44.385608 19485 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6431d8f509f84fa6a63a3aad36a6d39d" candidate_uuid: "309a53f12a534385a7e5a88fb373722a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "895c9e1595924e769a25faeed7e34644"
I20260129 02:57:44.385608 19410 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6431d8f509f84fa6a63a3aad36a6d39d" candidate_uuid: "309a53f12a534385a7e5a88fb373722a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a6aef613bf534ff9ae6ddbba06bb3cf1"
I20260129 02:57:44.386268 19485 raft_consensus.cc:3060] T 6431d8f509f84fa6a63a3aad36a6d39d P 895c9e1595924e769a25faeed7e34644 [term 0 FOLLOWER]: Advancing to term 1
I20260129 02:57:44.386279 19410 raft_consensus.cc:3060] T 6431d8f509f84fa6a63a3aad36a6d39d P a6aef613bf534ff9ae6ddbba06bb3cf1 [term 0 FOLLOWER]: Advancing to term 1
I20260129 02:57:44.390364 19485 raft_consensus.cc:2468] T 6431d8f509f84fa6a63a3aad36a6d39d P 895c9e1595924e769a25faeed7e34644 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 309a53f12a534385a7e5a88fb373722a in term 1.
I20260129 02:57:44.391279 19524 leader_election.cc:304] T 6431d8f509f84fa6a63a3aad36a6d39d P 309a53f12a534385a7e5a88fb373722a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 309a53f12a534385a7e5a88fb373722a, 895c9e1595924e769a25faeed7e34644; no voters:
I20260129 02:57:44.392117 19611 raft_consensus.cc:2804] T 6431d8f509f84fa6a63a3aad36a6d39d P 309a53f12a534385a7e5a88fb373722a [term 1 FOLLOWER]: Leader election won for term 1
I20260129 02:57:44.393316 19611 raft_consensus.cc:697] T 6431d8f509f84fa6a63a3aad36a6d39d P 309a53f12a534385a7e5a88fb373722a [term 1 LEADER]: Becoming Leader. State: Replica: 309a53f12a534385a7e5a88fb373722a, State: Running, Role: LEADER
I20260129 02:57:44.394110 19611 consensus_queue.cc:237] T 6431d8f509f84fa6a63a3aad36a6d39d P 309a53f12a534385a7e5a88fb373722a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6aef613bf534ff9ae6ddbba06bb3cf1" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 45187 } } peers { permanent_uuid: "895c9e1595924e769a25faeed7e34644" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 40083 } } peers { permanent_uuid: "309a53f12a534385a7e5a88fb373722a" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 34213 } }
I20260129 02:57:44.394110 19410 raft_consensus.cc:2468] T 6431d8f509f84fa6a63a3aad36a6d39d P a6aef613bf534ff9ae6ddbba06bb3cf1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 309a53f12a534385a7e5a88fb373722a in term 1.
I20260129 02:57:44.398193 19307 catalog_manager.cc:5654] T 9d5a1dc0b90642309290802698ec59f2 P a6aef613bf534ff9ae6ddbba06bb3cf1 reported cstate change: term changed from 0 to 1, leader changed from <none> to a6aef613bf534ff9ae6ddbba06bb3cf1 (127.18.211.1). New cstate: current_term: 1 leader_uuid: "a6aef613bf534ff9ae6ddbba06bb3cf1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6aef613bf534ff9ae6ddbba06bb3cf1" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 45187 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "895c9e1595924e769a25faeed7e34644" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 40083 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "309a53f12a534385a7e5a88fb373722a" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 34213 } health_report { overall_health: UNKNOWN } } }
I20260129 02:57:44.409621 19308 catalog_manager.cc:5654] T 6431d8f509f84fa6a63a3aad36a6d39d P 309a53f12a534385a7e5a88fb373722a reported cstate change: term changed from 0 to 1, leader changed from <none> to 309a53f12a534385a7e5a88fb373722a (127.18.211.3). New cstate: current_term: 1 leader_uuid: "309a53f12a534385a7e5a88fb373722a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6aef613bf534ff9ae6ddbba06bb3cf1" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 45187 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "895c9e1595924e769a25faeed7e34644" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 40083 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "309a53f12a534385a7e5a88fb373722a" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 34213 } health_report { overall_health: HEALTHY } } }
I20260129 02:57:44.422279 19611 raft_consensus.cc:493] T f7d1ccc38ee8428aa16f7c82b86c0f3c P 309a53f12a534385a7e5a88fb373722a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260129 02:57:44.422765 19611 raft_consensus.cc:515] T f7d1ccc38ee8428aa16f7c82b86c0f3c P 309a53f12a534385a7e5a88fb373722a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6aef613bf534ff9ae6ddbba06bb3cf1" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 45187 } } peers { permanent_uuid: "895c9e1595924e769a25faeed7e34644" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 40083 } } peers { permanent_uuid: "309a53f12a534385a7e5a88fb373722a" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 34213 } }
I20260129 02:57:44.424633 19611 leader_election.cc:290] T f7d1ccc38ee8428aa16f7c82b86c0f3c P 309a53f12a534385a7e5a88fb373722a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a6aef613bf534ff9ae6ddbba06bb3cf1 (127.18.211.1:45187), 895c9e1595924e769a25faeed7e34644 (127.18.211.2:40083)
I20260129 02:57:44.425345 19410 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f7d1ccc38ee8428aa16f7c82b86c0f3c" candidate_uuid: "309a53f12a534385a7e5a88fb373722a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a6aef613bf534ff9ae6ddbba06bb3cf1" is_pre_election: true
I20260129 02:57:44.425593 19485 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f7d1ccc38ee8428aa16f7c82b86c0f3c" candidate_uuid: "309a53f12a534385a7e5a88fb373722a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "895c9e1595924e769a25faeed7e34644" is_pre_election: true
I20260129 02:57:44.426059 19410 raft_consensus.cc:2393] T f7d1ccc38ee8428aa16f7c82b86c0f3c P a6aef613bf534ff9ae6ddbba06bb3cf1 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 309a53f12a534385a7e5a88fb373722a in current term 1: Already voted for candidate 895c9e1595924e769a25faeed7e34644 in this term.
I20260129 02:57:44.427186 19524 leader_election.cc:304] T f7d1ccc38ee8428aa16f7c82b86c0f3c P 309a53f12a534385a7e5a88fb373722a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 309a53f12a534385a7e5a88fb373722a; no voters: 895c9e1595924e769a25faeed7e34644, a6aef613bf534ff9ae6ddbba06bb3cf1
I20260129 02:57:44.427937 19611 raft_consensus.cc:3060] T f7d1ccc38ee8428aa16f7c82b86c0f3c P 309a53f12a534385a7e5a88fb373722a [term 0 FOLLOWER]: Advancing to term 1
I20260129 02:57:44.432093 19611 raft_consensus.cc:2749] T f7d1ccc38ee8428aa16f7c82b86c0f3c P 309a53f12a534385a7e5a88fb373722a [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260129 02:57:44.434900 19610 raft_consensus.cc:493] T a78e86f5781f41bba56333c01607277a P a6aef613bf534ff9ae6ddbba06bb3cf1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260129 02:57:44.435284 19610 raft_consensus.cc:515] T a78e86f5781f41bba56333c01607277a P a6aef613bf534ff9ae6ddbba06bb3cf1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6aef613bf534ff9ae6ddbba06bb3cf1" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 45187 } } peers { permanent_uuid: "895c9e1595924e769a25faeed7e34644" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 40083 } } peers { permanent_uuid: "309a53f12a534385a7e5a88fb373722a" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 34213 } }
I20260129 02:57:44.436805 19610 leader_election.cc:290] T a78e86f5781f41bba56333c01607277a P a6aef613bf534ff9ae6ddbba06bb3cf1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 895c9e1595924e769a25faeed7e34644 (127.18.211.2:40083), 309a53f12a534385a7e5a88fb373722a (127.18.211.3:34213)
I20260129 02:57:44.437568 19485 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a78e86f5781f41bba56333c01607277a" candidate_uuid: "a6aef613bf534ff9ae6ddbba06bb3cf1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "895c9e1595924e769a25faeed7e34644" is_pre_election: true
I20260129 02:57:44.437872 19560 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a78e86f5781f41bba56333c01607277a" candidate_uuid: "a6aef613bf534ff9ae6ddbba06bb3cf1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "309a53f12a534385a7e5a88fb373722a" is_pre_election: true
I20260129 02:57:44.438079 19485 raft_consensus.cc:2468] T a78e86f5781f41bba56333c01607277a P 895c9e1595924e769a25faeed7e34644 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a6aef613bf534ff9ae6ddbba06bb3cf1 in term 0.
I20260129 02:57:44.438481 19560 raft_consensus.cc:2468] T a78e86f5781f41bba56333c01607277a P 309a53f12a534385a7e5a88fb373722a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a6aef613bf534ff9ae6ddbba06bb3cf1 in term 0.
I20260129 02:57:44.438939 19375 leader_election.cc:304] T a78e86f5781f41bba56333c01607277a P a6aef613bf534ff9ae6ddbba06bb3cf1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 895c9e1595924e769a25faeed7e34644, a6aef613bf534ff9ae6ddbba06bb3cf1; no voters:
I20260129 02:57:44.439652 19610 raft_consensus.cc:2804] T a78e86f5781f41bba56333c01607277a P a6aef613bf534ff9ae6ddbba06bb3cf1 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260129 02:57:44.440022 19610 raft_consensus.cc:493] T a78e86f5781f41bba56333c01607277a P a6aef613bf534ff9ae6ddbba06bb3cf1 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260129 02:57:44.440311 19610 raft_consensus.cc:3060] T a78e86f5781f41bba56333c01607277a P a6aef613bf534ff9ae6ddbba06bb3cf1 [term 0 FOLLOWER]: Advancing to term 1
I20260129 02:57:44.441185 19635 raft_consensus.cc:493] T 2b5edf7ea566411caea265dcfff5a86a P 895c9e1595924e769a25faeed7e34644 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260129 02:57:44.441581 19635 raft_consensus.cc:515] T 2b5edf7ea566411caea265dcfff5a86a P 895c9e1595924e769a25faeed7e34644 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "309a53f12a534385a7e5a88fb373722a" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 34213 } } peers { permanent_uuid: "a6aef613bf534ff9ae6ddbba06bb3cf1" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 45187 } } peers { permanent_uuid: "895c9e1595924e769a25faeed7e34644" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 40083 } }
I20260129 02:57:44.443316 19635 leader_election.cc:290] T 2b5edf7ea566411caea265dcfff5a86a P 895c9e1595924e769a25faeed7e34644 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 309a53f12a534385a7e5a88fb373722a (127.18.211.3:34213), a6aef613bf534ff9ae6ddbba06bb3cf1 (127.18.211.1:45187)
I20260129 02:57:44.444437 19560 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2b5edf7ea566411caea265dcfff5a86a" candidate_uuid: "895c9e1595924e769a25faeed7e34644" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "309a53f12a534385a7e5a88fb373722a" is_pre_election: true
I20260129 02:57:44.445032 19560 raft_consensus.cc:2468] T 2b5edf7ea566411caea265dcfff5a86a P 309a53f12a534385a7e5a88fb373722a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 895c9e1595924e769a25faeed7e34644 in term 0.
I20260129 02:57:44.445206 19410 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2b5edf7ea566411caea265dcfff5a86a" candidate_uuid: "895c9e1595924e769a25faeed7e34644" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a6aef613bf534ff9ae6ddbba06bb3cf1" is_pre_election: true
I20260129 02:57:44.445920 19410 raft_consensus.cc:2468] T 2b5edf7ea566411caea265dcfff5a86a P a6aef613bf534ff9ae6ddbba06bb3cf1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 895c9e1595924e769a25faeed7e34644 in term 0.
I20260129 02:57:44.446115 19450 leader_election.cc:304] T 2b5edf7ea566411caea265dcfff5a86a P 895c9e1595924e769a25faeed7e34644 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 309a53f12a534385a7e5a88fb373722a, 895c9e1595924e769a25faeed7e34644; no voters:
I20260129 02:57:44.446082 19610 raft_consensus.cc:515] T a78e86f5781f41bba56333c01607277a P a6aef613bf534ff9ae6ddbba06bb3cf1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6aef613bf534ff9ae6ddbba06bb3cf1" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 45187 } } peers { permanent_uuid: "895c9e1595924e769a25faeed7e34644" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 40083 } } peers { permanent_uuid: "309a53f12a534385a7e5a88fb373722a" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 34213 } }
I20260129 02:57:44.447263 19635 raft_consensus.cc:2804] T 2b5edf7ea566411caea265dcfff5a86a P 895c9e1595924e769a25faeed7e34644 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260129 02:57:44.447705 19635 raft_consensus.cc:493] T 2b5edf7ea566411caea265dcfff5a86a P 895c9e1595924e769a25faeed7e34644 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260129 02:57:44.448119 19635 raft_consensus.cc:3060] T 2b5edf7ea566411caea265dcfff5a86a P 895c9e1595924e769a25faeed7e34644 [term 0 FOLLOWER]: Advancing to term 1
I20260129 02:57:44.448994 19610 leader_election.cc:290] T a78e86f5781f41bba56333c01607277a P a6aef613bf534ff9ae6ddbba06bb3cf1 [CANDIDATE]: Term 1 election: Requested vote from peers 895c9e1595924e769a25faeed7e34644 (127.18.211.2:40083), 309a53f12a534385a7e5a88fb373722a (127.18.211.3:34213)
I20260129 02:57:44.453872 19635 raft_consensus.cc:515] T 2b5edf7ea566411caea265dcfff5a86a P 895c9e1595924e769a25faeed7e34644 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "309a53f12a534385a7e5a88fb373722a" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 34213 } } peers { permanent_uuid: "a6aef613bf534ff9ae6ddbba06bb3cf1" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 45187 } } peers { permanent_uuid: "895c9e1595924e769a25faeed7e34644" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 40083 } }
I20260129 02:57:44.455371 19485 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a78e86f5781f41bba56333c01607277a" candidate_uuid: "a6aef613bf534ff9ae6ddbba06bb3cf1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "895c9e1595924e769a25faeed7e34644"
I20260129 02:57:44.455632 19560 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a78e86f5781f41bba56333c01607277a" candidate_uuid: "a6aef613bf534ff9ae6ddbba06bb3cf1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "309a53f12a534385a7e5a88fb373722a"
I20260129 02:57:44.455941 19485 raft_consensus.cc:3060] T a78e86f5781f41bba56333c01607277a P 895c9e1595924e769a25faeed7e34644 [term 0 FOLLOWER]: Advancing to term 1
I20260129 02:57:44.456167 19560 raft_consensus.cc:3060] T a78e86f5781f41bba56333c01607277a P 309a53f12a534385a7e5a88fb373722a [term 0 FOLLOWER]: Advancing to term 1
I20260129 02:57:44.457194 19559 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2b5edf7ea566411caea265dcfff5a86a" candidate_uuid: "895c9e1595924e769a25faeed7e34644" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "309a53f12a534385a7e5a88fb373722a"
I20260129 02:57:44.457793 19559 raft_consensus.cc:3060] T 2b5edf7ea566411caea265dcfff5a86a P 309a53f12a534385a7e5a88fb373722a [term 0 FOLLOWER]: Advancing to term 1
I20260129 02:57:44.462479 19485 raft_consensus.cc:2468] T a78e86f5781f41bba56333c01607277a P 895c9e1595924e769a25faeed7e34644 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a6aef613bf534ff9ae6ddbba06bb3cf1 in term 1.
I20260129 02:57:44.462555 19560 raft_consensus.cc:2468] T a78e86f5781f41bba56333c01607277a P 309a53f12a534385a7e5a88fb373722a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a6aef613bf534ff9ae6ddbba06bb3cf1 in term 1.
I20260129 02:57:44.463533 19375 leader_election.cc:304] T a78e86f5781f41bba56333c01607277a P a6aef613bf534ff9ae6ddbba06bb3cf1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 309a53f12a534385a7e5a88fb373722a, a6aef613bf534ff9ae6ddbba06bb3cf1; no voters:
I20260129 02:57:44.464181 19610 raft_consensus.cc:2804] T a78e86f5781f41bba56333c01607277a P a6aef613bf534ff9ae6ddbba06bb3cf1 [term 1 FOLLOWER]: Leader election won for term 1
I20260129 02:57:44.465124 19635 leader_election.cc:290] T 2b5edf7ea566411caea265dcfff5a86a P 895c9e1595924e769a25faeed7e34644 [CANDIDATE]: Term 1 election: Requested vote from peers 309a53f12a534385a7e5a88fb373722a (127.18.211.3:34213), a6aef613bf534ff9ae6ddbba06bb3cf1 (127.18.211.1:45187)
I20260129 02:57:44.464834 19559 raft_consensus.cc:2468] T 2b5edf7ea566411caea265dcfff5a86a P 309a53f12a534385a7e5a88fb373722a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 895c9e1595924e769a25faeed7e34644 in term 1.
I20260129 02:57:44.465396 19610 raft_consensus.cc:697] T a78e86f5781f41bba56333c01607277a P a6aef613bf534ff9ae6ddbba06bb3cf1 [term 1 LEADER]: Becoming Leader. State: Replica: a6aef613bf534ff9ae6ddbba06bb3cf1, State: Running, Role: LEADER
I20260129 02:57:44.466149 19410 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2b5edf7ea566411caea265dcfff5a86a" candidate_uuid: "895c9e1595924e769a25faeed7e34644" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a6aef613bf534ff9ae6ddbba06bb3cf1"
I20260129 02:57:44.466499 19450 leader_election.cc:304] T 2b5edf7ea566411caea265dcfff5a86a P 895c9e1595924e769a25faeed7e34644 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 309a53f12a534385a7e5a88fb373722a, 895c9e1595924e769a25faeed7e34644; no voters:
I20260129 02:57:44.466672 19410 raft_consensus.cc:3060] T 2b5edf7ea566411caea265dcfff5a86a P a6aef613bf534ff9ae6ddbba06bb3cf1 [term 0 FOLLOWER]: Advancing to term 1
I20260129 02:57:44.466391 19610 consensus_queue.cc:237] T a78e86f5781f41bba56333c01607277a P a6aef613bf534ff9ae6ddbba06bb3cf1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6aef613bf534ff9ae6ddbba06bb3cf1" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 45187 } } peers { permanent_uuid: "895c9e1595924e769a25faeed7e34644" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 40083 } } peers { permanent_uuid: "309a53f12a534385a7e5a88fb373722a" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 34213 } }
I20260129 02:57:44.467355 19635 raft_consensus.cc:2804] T 2b5edf7ea566411caea265dcfff5a86a P 895c9e1595924e769a25faeed7e34644 [term 1 FOLLOWER]: Leader election won for term 1
I20260129 02:57:44.468492 19635 raft_consensus.cc:697] T 2b5edf7ea566411caea265dcfff5a86a P 895c9e1595924e769a25faeed7e34644 [term 1 LEADER]: Becoming Leader. State: Replica: 895c9e1595924e769a25faeed7e34644, State: Running, Role: LEADER
I20260129 02:57:44.469300 19635 consensus_queue.cc:237] T 2b5edf7ea566411caea265dcfff5a86a P 895c9e1595924e769a25faeed7e34644 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "309a53f12a534385a7e5a88fb373722a" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 34213 } } peers { permanent_uuid: "a6aef613bf534ff9ae6ddbba06bb3cf1" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 45187 } } peers { permanent_uuid: "895c9e1595924e769a25faeed7e34644" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 40083 } }
I20260129 02:57:44.473641 19410 raft_consensus.cc:2468] T 2b5edf7ea566411caea265dcfff5a86a P a6aef613bf534ff9ae6ddbba06bb3cf1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 895c9e1595924e769a25faeed7e34644 in term 1.
I20260129 02:57:44.482815 19307 catalog_manager.cc:5654] T a78e86f5781f41bba56333c01607277a P a6aef613bf534ff9ae6ddbba06bb3cf1 reported cstate change: term changed from 0 to 1, leader changed from <none> to a6aef613bf534ff9ae6ddbba06bb3cf1 (127.18.211.1). New cstate: current_term: 1 leader_uuid: "a6aef613bf534ff9ae6ddbba06bb3cf1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6aef613bf534ff9ae6ddbba06bb3cf1" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 45187 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "895c9e1595924e769a25faeed7e34644" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 40083 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "309a53f12a534385a7e5a88fb373722a" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 34213 } health_report { overall_health: UNKNOWN } } }
I20260129 02:57:44.484486 19308 catalog_manager.cc:5654] T 2b5edf7ea566411caea265dcfff5a86a P 895c9e1595924e769a25faeed7e34644 reported cstate change: term changed from 0 to 1, leader changed from <none> to 895c9e1595924e769a25faeed7e34644 (127.18.211.2). New cstate: current_term: 1 leader_uuid: "895c9e1595924e769a25faeed7e34644" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "309a53f12a534385a7e5a88fb373722a" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 34213 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a6aef613bf534ff9ae6ddbba06bb3cf1" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 45187 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "895c9e1595924e769a25faeed7e34644" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 40083 } health_report { overall_health: HEALTHY } } }
I20260129 02:57:44.537843 19610 raft_consensus.cc:493] T 2e068f6d10f04c9ea17dba1375ae9b3c P a6aef613bf534ff9ae6ddbba06bb3cf1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260129 02:57:44.538416 19610 raft_consensus.cc:515] T 2e068f6d10f04c9ea17dba1375ae9b3c P a6aef613bf534ff9ae6ddbba06bb3cf1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6aef613bf534ff9ae6ddbba06bb3cf1" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 45187 } } peers { permanent_uuid: "895c9e1595924e769a25faeed7e34644" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 40083 } } peers { permanent_uuid: "309a53f12a534385a7e5a88fb373722a" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 34213 } }
I20260129 02:57:44.540370 19610 leader_election.cc:290] T 2e068f6d10f04c9ea17dba1375ae9b3c P a6aef613bf534ff9ae6ddbba06bb3cf1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 895c9e1595924e769a25faeed7e34644 (127.18.211.2:40083), 309a53f12a534385a7e5a88fb373722a (127.18.211.3:34213)
I20260129 02:57:44.541102 19485 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2e068f6d10f04c9ea17dba1375ae9b3c" candidate_uuid: "a6aef613bf534ff9ae6ddbba06bb3cf1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "895c9e1595924e769a25faeed7e34644" is_pre_election: true
I20260129 02:57:44.541379 19559 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2e068f6d10f04c9ea17dba1375ae9b3c" candidate_uuid: "a6aef613bf534ff9ae6ddbba06bb3cf1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "309a53f12a534385a7e5a88fb373722a" is_pre_election: true
I20260129 02:57:44.541795 19485 raft_consensus.cc:2393] T 2e068f6d10f04c9ea17dba1375ae9b3c P 895c9e1595924e769a25faeed7e34644 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a6aef613bf534ff9ae6ddbba06bb3cf1 in current term 1: Already voted for candidate 309a53f12a534385a7e5a88fb373722a in this term.
I20260129 02:57:44.542980 19375 leader_election.cc:304] T 2e068f6d10f04c9ea17dba1375ae9b3c P a6aef613bf534ff9ae6ddbba06bb3cf1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a6aef613bf534ff9ae6ddbba06bb3cf1; no voters: 309a53f12a534385a7e5a88fb373722a, 895c9e1595924e769a25faeed7e34644
I20260129 02:57:44.543690 19610 raft_consensus.cc:3060] T 2e068f6d10f04c9ea17dba1375ae9b3c P a6aef613bf534ff9ae6ddbba06bb3cf1 [term 0 FOLLOWER]: Advancing to term 1
I20260129 02:57:44.547803 19610 raft_consensus.cc:2749] T 2e068f6d10f04c9ea17dba1375ae9b3c P a6aef613bf534ff9ae6ddbba06bb3cf1 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20260129 02:57:44.553797 19356 auto_rebalancer.cc:229] Could not retrieve cluster info: Service unavailable: Tablet not running
I20260129 02:57:44.560246 19635 consensus_queue.cc:1048] T f7d1ccc38ee8428aa16f7c82b86c0f3c P 895c9e1595924e769a25faeed7e34644 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a6aef613bf534ff9ae6ddbba06bb3cf1" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 45187 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260129 02:57:44.564178 19609 consensus_queue.cc:1048] T f7d1ccc38ee8428aa16f7c82b86c0f3c P 895c9e1595924e769a25faeed7e34644 [LEADER]: Connected to new peer: Peer: permanent_uuid: "309a53f12a534385a7e5a88fb373722a" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 34213 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260129 02:57:44.578367 19634 consensus_queue.cc:1048] T e5ce9ed62ba242cabb3d6e5371679ba7 P a6aef613bf534ff9ae6ddbba06bb3cf1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "309a53f12a534385a7e5a88fb373722a" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 34213 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260129 02:57:44.603075 19634 consensus_queue.cc:1048] T e5ce9ed62ba242cabb3d6e5371679ba7 P a6aef613bf534ff9ae6ddbba06bb3cf1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "895c9e1595924e769a25faeed7e34644" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 40083 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260129 02:57:44.632201 19611 raft_consensus.cc:493] T 3e4edd649791453f902a5ac6d5658c12 P 309a53f12a534385a7e5a88fb373722a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260129 02:57:44.632690 19611 raft_consensus.cc:515] T 3e4edd649791453f902a5ac6d5658c12 P 309a53f12a534385a7e5a88fb373722a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6aef613bf534ff9ae6ddbba06bb3cf1" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 45187 } } peers { permanent_uuid: "895c9e1595924e769a25faeed7e34644" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 40083 } } peers { permanent_uuid: "309a53f12a534385a7e5a88fb373722a" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 34213 } }
I20260129 02:57:44.634444 19611 leader_election.cc:290] T 3e4edd649791453f902a5ac6d5658c12 P 309a53f12a534385a7e5a88fb373722a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a6aef613bf534ff9ae6ddbba06bb3cf1 (127.18.211.1:45187), 895c9e1595924e769a25faeed7e34644 (127.18.211.2:40083)
I20260129 02:57:44.635483 19410 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3e4edd649791453f902a5ac6d5658c12" candidate_uuid: "309a53f12a534385a7e5a88fb373722a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a6aef613bf534ff9ae6ddbba06bb3cf1" is_pre_election: true
I20260129 02:57:44.635615 19486 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3e4edd649791453f902a5ac6d5658c12" candidate_uuid: "309a53f12a534385a7e5a88fb373722a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "895c9e1595924e769a25faeed7e34644" is_pre_election: true
I20260129 02:57:44.635628 19485 raft_consensus.cc:3060] T 9d5a1dc0b90642309290802698ec59f2 P 895c9e1595924e769a25faeed7e34644 [term 0 FOLLOWER]: Advancing to term 1
I20260129 02:57:44.636139 19410 raft_consensus.cc:2468] T 3e4edd649791453f902a5ac6d5658c12 P a6aef613bf534ff9ae6ddbba06bb3cf1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 309a53f12a534385a7e5a88fb373722a in term 0.
I20260129 02:57:44.636255 19486 raft_consensus.cc:2468] T 3e4edd649791453f902a5ac6d5658c12 P 895c9e1595924e769a25faeed7e34644 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 309a53f12a534385a7e5a88fb373722a in term 0.
I20260129 02:57:44.637398 19524 leader_election.cc:304] T 3e4edd649791453f902a5ac6d5658c12 P 309a53f12a534385a7e5a88fb373722a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 309a53f12a534385a7e5a88fb373722a, a6aef613bf534ff9ae6ddbba06bb3cf1; no voters:
I20260129 02:57:44.638125 19611 raft_consensus.cc:2804] T 3e4edd649791453f902a5ac6d5658c12 P 309a53f12a534385a7e5a88fb373722a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260129 02:57:44.638433 19611 raft_consensus.cc:493] T 3e4edd649791453f902a5ac6d5658c12 P 309a53f12a534385a7e5a88fb373722a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260129 02:57:44.638734 19611 raft_consensus.cc:3060] T 3e4edd649791453f902a5ac6d5658c12 P 309a53f12a534385a7e5a88fb373722a [term 0 FOLLOWER]: Advancing to term 1
I20260129 02:57:44.644285 19634 consensus_queue.cc:1048] T 9d5a1dc0b90642309290802698ec59f2 P a6aef613bf534ff9ae6ddbba06bb3cf1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "895c9e1595924e769a25faeed7e34644" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 40083 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260129 02:57:44.644757 19611 raft_consensus.cc:515] T 3e4edd649791453f902a5ac6d5658c12 P 309a53f12a534385a7e5a88fb373722a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6aef613bf534ff9ae6ddbba06bb3cf1" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 45187 } } peers { permanent_uuid: "895c9e1595924e769a25faeed7e34644" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 40083 } } peers { permanent_uuid: "309a53f12a534385a7e5a88fb373722a" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 34213 } }
I20260129 02:57:44.645257 19621 consensus_queue.cc:1048] T c4689cdf800d479ba8645b96278c1713 P 309a53f12a534385a7e5a88fb373722a [LEADER]: Connected to new peer: Peer: permanent_uuid: "895c9e1595924e769a25faeed7e34644" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 40083 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260129 02:57:44.647099 19410 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3e4edd649791453f902a5ac6d5658c12" candidate_uuid: "309a53f12a534385a7e5a88fb373722a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a6aef613bf534ff9ae6ddbba06bb3cf1"
I20260129 02:57:44.647642 19410 raft_consensus.cc:3060] T 3e4edd649791453f902a5ac6d5658c12 P a6aef613bf534ff9ae6ddbba06bb3cf1 [term 0 FOLLOWER]: Advancing to term 1
I20260129 02:57:44.649670 19484 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3e4edd649791453f902a5ac6d5658c12" candidate_uuid: "309a53f12a534385a7e5a88fb373722a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "895c9e1595924e769a25faeed7e34644"
I20260129 02:57:44.650205 19484 raft_consensus.cc:3060] T 3e4edd649791453f902a5ac6d5658c12 P 895c9e1595924e769a25faeed7e34644 [term 0 FOLLOWER]: Advancing to term 1
I20260129 02:57:44.656183 19484 raft_consensus.cc:2468] T 3e4edd649791453f902a5ac6d5658c12 P 895c9e1595924e769a25faeed7e34644 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 309a53f12a534385a7e5a88fb373722a in term 1.
I20260129 02:57:44.656388 19410 raft_consensus.cc:2468] T 3e4edd649791453f902a5ac6d5658c12 P a6aef613bf534ff9ae6ddbba06bb3cf1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 309a53f12a534385a7e5a88fb373722a in term 1.
I20260129 02:57:44.657449 19524 leader_election.cc:304] T 3e4edd649791453f902a5ac6d5658c12 P 309a53f12a534385a7e5a88fb373722a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 309a53f12a534385a7e5a88fb373722a, 895c9e1595924e769a25faeed7e34644; no voters:
I20260129 02:57:44.658269 19621 raft_consensus.cc:2804] T 3e4edd649791453f902a5ac6d5658c12 P 309a53f12a534385a7e5a88fb373722a [term 1 FOLLOWER]: Leader election won for term 1
I20260129 02:57:44.660691 19621 raft_consensus.cc:697] T 3e4edd649791453f902a5ac6d5658c12 P 309a53f12a534385a7e5a88fb373722a [term 1 LEADER]: Becoming Leader. State: Replica: 309a53f12a534385a7e5a88fb373722a, State: Running, Role: LEADER
I20260129 02:57:44.661549 19621 consensus_queue.cc:237] T 3e4edd649791453f902a5ac6d5658c12 P 309a53f12a534385a7e5a88fb373722a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6aef613bf534ff9ae6ddbba06bb3cf1" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 45187 } } peers { permanent_uuid: "895c9e1595924e769a25faeed7e34644" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 40083 } } peers { permanent_uuid: "309a53f12a534385a7e5a88fb373722a" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 34213 } }
I20260129 02:57:44.665143 19610 consensus_queue.cc:1048] T 9d5a1dc0b90642309290802698ec59f2 P a6aef613bf534ff9ae6ddbba06bb3cf1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "309a53f12a534385a7e5a88fb373722a" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 34213 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260129 02:57:44.668926 19611 leader_election.cc:290] T 3e4edd649791453f902a5ac6d5658c12 P 309a53f12a534385a7e5a88fb373722a [CANDIDATE]: Term 1 election: Requested vote from peers a6aef613bf534ff9ae6ddbba06bb3cf1 (127.18.211.1:45187), 895c9e1595924e769a25faeed7e34644 (127.18.211.2:40083)
I20260129 02:57:44.685202 19308 catalog_manager.cc:5654] T 3e4edd649791453f902a5ac6d5658c12 P 309a53f12a534385a7e5a88fb373722a reported cstate change: term changed from 0 to 1, leader changed from <none> to 309a53f12a534385a7e5a88fb373722a (127.18.211.3). New cstate: current_term: 1 leader_uuid: "309a53f12a534385a7e5a88fb373722a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6aef613bf534ff9ae6ddbba06bb3cf1" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 45187 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "895c9e1595924e769a25faeed7e34644" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 40083 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "309a53f12a534385a7e5a88fb373722a" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 34213 } health_report { overall_health: HEALTHY } } }
I20260129 02:57:44.692808 19621 consensus_queue.cc:1048] T c4689cdf800d479ba8645b96278c1713 P 309a53f12a534385a7e5a88fb373722a [LEADER]: Connected to new peer: Peer: permanent_uuid: "a6aef613bf534ff9ae6ddbba06bb3cf1" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 45187 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260129 02:57:44.776868 19621 consensus_queue.cc:1048] T 6431d8f509f84fa6a63a3aad36a6d39d P 309a53f12a534385a7e5a88fb373722a [LEADER]: Connected to new peer: Peer: permanent_uuid: "a6aef613bf534ff9ae6ddbba06bb3cf1" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 45187 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260129 02:57:44.777108 19485 tablet_service.cc:2066] Received LeaderStepDown RPC: tablet_id: "2b5edf7ea566411caea265dcfff5a86a"
dest_uuid: "895c9e1595924e769a25faeed7e34644"
mode: GRACEFUL
new_leader_uuid: "a6aef613bf534ff9ae6ddbba06bb3cf1"
from {username='slave'} at 127.0.0.1:46906
I20260129 02:57:44.777751 19485 raft_consensus.cc:606] T 2b5edf7ea566411caea265dcfff5a86a P 895c9e1595924e769a25faeed7e34644 [term 1 LEADER]: Received request to transfer leadership to TS a6aef613bf534ff9ae6ddbba06bb3cf1
I20260129 02:57:44.792045 19621 consensus_queue.cc:1048] T 2e068f6d10f04c9ea17dba1375ae9b3c P 309a53f12a534385a7e5a88fb373722a [LEADER]: Connected to new peer: Peer: permanent_uuid: "a6aef613bf534ff9ae6ddbba06bb3cf1" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 45187 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260129 02:57:44.794986 19611 consensus_queue.cc:1048] T 6431d8f509f84fa6a63a3aad36a6d39d P 309a53f12a534385a7e5a88fb373722a [LEADER]: Connected to new peer: Peer: permanent_uuid: "895c9e1595924e769a25faeed7e34644" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 40083 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260129 02:57:44.810528 19670 consensus_queue.cc:1048] T 2e068f6d10f04c9ea17dba1375ae9b3c P 309a53f12a534385a7e5a88fb373722a [LEADER]: Connected to new peer: Peer: permanent_uuid: "895c9e1595924e769a25faeed7e34644" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 40083 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260129 02:57:44.817018 19559 tablet_service.cc:2066] Received LeaderStepDown RPC: tablet_id: "2e068f6d10f04c9ea17dba1375ae9b3c"
dest_uuid: "309a53f12a534385a7e5a88fb373722a"
mode: GRACEFUL
new_leader_uuid: "a6aef613bf534ff9ae6ddbba06bb3cf1"
from {username='slave'} at 127.0.0.1:38384
I20260129 02:57:44.817636 19559 raft_consensus.cc:606] T 2e068f6d10f04c9ea17dba1375ae9b3c P 309a53f12a534385a7e5a88fb373722a [term 1 LEADER]: Received request to transfer leadership to TS a6aef613bf534ff9ae6ddbba06bb3cf1
I20260129 02:57:44.818506 19670 raft_consensus.cc:993] T 2e068f6d10f04c9ea17dba1375ae9b3c P 309a53f12a534385a7e5a88fb373722a: : Instructing follower a6aef613bf534ff9ae6ddbba06bb3cf1 to start an election
I20260129 02:57:44.818955 19670 raft_consensus.cc:1081] T 2e068f6d10f04c9ea17dba1375ae9b3c P 309a53f12a534385a7e5a88fb373722a [term 1 LEADER]: Signalling peer a6aef613bf534ff9ae6ddbba06bb3cf1 to start an election
I20260129 02:57:44.826436 19410 tablet_service.cc:2038] Received Run Leader Election RPC: tablet_id: "2e068f6d10f04c9ea17dba1375ae9b3c"
dest_uuid: "a6aef613bf534ff9ae6ddbba06bb3cf1"
from {username='slave'} at 127.0.0.1:40686
I20260129 02:57:44.826992 19410 raft_consensus.cc:493] T 2e068f6d10f04c9ea17dba1375ae9b3c P a6aef613bf534ff9ae6ddbba06bb3cf1 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260129 02:57:44.827361 19410 raft_consensus.cc:3060] T 2e068f6d10f04c9ea17dba1375ae9b3c P a6aef613bf534ff9ae6ddbba06bb3cf1 [term 1 FOLLOWER]: Advancing to term 2
I20260129 02:57:44.834162 19410 raft_consensus.cc:515] T 2e068f6d10f04c9ea17dba1375ae9b3c P a6aef613bf534ff9ae6ddbba06bb3cf1 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6aef613bf534ff9ae6ddbba06bb3cf1" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 45187 } } peers { permanent_uuid: "895c9e1595924e769a25faeed7e34644" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 40083 } } peers { permanent_uuid: "309a53f12a534385a7e5a88fb373722a" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 34213 } }
I20260129 02:57:44.837776 19485 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2e068f6d10f04c9ea17dba1375ae9b3c" candidate_uuid: "a6aef613bf534ff9ae6ddbba06bb3cf1" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "895c9e1595924e769a25faeed7e34644"
I20260129 02:57:44.839097 19485 raft_consensus.cc:3060] T 2e068f6d10f04c9ea17dba1375ae9b3c P 895c9e1595924e769a25faeed7e34644 [term 1 FOLLOWER]: Advancing to term 2
I20260129 02:57:44.839620 19559 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2e068f6d10f04c9ea17dba1375ae9b3c" candidate_uuid: "a6aef613bf534ff9ae6ddbba06bb3cf1" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "309a53f12a534385a7e5a88fb373722a"
I20260129 02:57:44.839452 19410 leader_election.cc:290] T 2e068f6d10f04c9ea17dba1375ae9b3c P a6aef613bf534ff9ae6ddbba06bb3cf1 [CANDIDATE]: Term 2 election: Requested vote from peers 895c9e1595924e769a25faeed7e34644 (127.18.211.2:40083), 309a53f12a534385a7e5a88fb373722a (127.18.211.3:34213)
I20260129 02:57:44.840234 19559 raft_consensus.cc:3055] T 2e068f6d10f04c9ea17dba1375ae9b3c P 309a53f12a534385a7e5a88fb373722a [term 1 LEADER]: Stepping down as leader of term 1
I20260129 02:57:44.840553 19559 raft_consensus.cc:740] T 2e068f6d10f04c9ea17dba1375ae9b3c P 309a53f12a534385a7e5a88fb373722a [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 309a53f12a534385a7e5a88fb373722a, State: Running, Role: LEADER
I20260129 02:57:44.841322 19559 consensus_queue.cc:260] T 2e068f6d10f04c9ea17dba1375ae9b3c P 309a53f12a534385a7e5a88fb373722a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6aef613bf534ff9ae6ddbba06bb3cf1" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 45187 } } peers { permanent_uuid: "895c9e1595924e769a25faeed7e34644" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 40083 } } peers { permanent_uuid: "309a53f12a534385a7e5a88fb373722a" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 34213 } }
I20260129 02:57:44.842761 19559 raft_consensus.cc:3060] T 2e068f6d10f04c9ea17dba1375ae9b3c P 309a53f12a534385a7e5a88fb373722a [term 1 FOLLOWER]: Advancing to term 2
I20260129 02:57:44.847414 19410 tablet_service.cc:2066] Received LeaderStepDown RPC: tablet_id: "9d5a1dc0b90642309290802698ec59f2"
dest_uuid: "a6aef613bf534ff9ae6ddbba06bb3cf1"
mode: GRACEFUL
new_leader_uuid: "895c9e1595924e769a25faeed7e34644"
from {username='slave'} at 127.0.0.1:40704
I20260129 02:57:44.848031 19410 raft_consensus.cc:606] T 9d5a1dc0b90642309290802698ec59f2 P a6aef613bf534ff9ae6ddbba06bb3cf1 [term 1 LEADER]: Received request to transfer leadership to TS 895c9e1595924e769a25faeed7e34644
I20260129 02:57:44.849125 19485 raft_consensus.cc:2468] T 2e068f6d10f04c9ea17dba1375ae9b3c P 895c9e1595924e769a25faeed7e34644 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a6aef613bf534ff9ae6ddbba06bb3cf1 in term 2.
I20260129 02:57:44.850196 19375 leader_election.cc:304] T 2e068f6d10f04c9ea17dba1375ae9b3c P a6aef613bf534ff9ae6ddbba06bb3cf1 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 895c9e1595924e769a25faeed7e34644, a6aef613bf534ff9ae6ddbba06bb3cf1; no voters:
I20260129 02:57:44.849995 19559 raft_consensus.cc:2468] T 2e068f6d10f04c9ea17dba1375ae9b3c P 309a53f12a534385a7e5a88fb373722a [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a6aef613bf534ff9ae6ddbba06bb3cf1 in term 2.
I20260129 02:57:44.851060 19610 raft_consensus.cc:2804] T 2e068f6d10f04c9ea17dba1375ae9b3c P a6aef613bf534ff9ae6ddbba06bb3cf1 [term 2 FOLLOWER]: Leader election won for term 2
I20260129 02:57:44.851506 19610 raft_consensus.cc:697] T 2e068f6d10f04c9ea17dba1375ae9b3c P a6aef613bf534ff9ae6ddbba06bb3cf1 [term 2 LEADER]: Becoming Leader. State: Replica: a6aef613bf534ff9ae6ddbba06bb3cf1, State: Running, Role: LEADER
I20260129 02:57:44.852310 19610 consensus_queue.cc:237] T 2e068f6d10f04c9ea17dba1375ae9b3c P a6aef613bf534ff9ae6ddbba06bb3cf1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6aef613bf534ff9ae6ddbba06bb3cf1" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 45187 } } peers { permanent_uuid: "895c9e1595924e769a25faeed7e34644" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 40083 } } peers { permanent_uuid: "309a53f12a534385a7e5a88fb373722a" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 34213 } }
I20260129 02:57:44.854593 19559 tablet_service.cc:2066] Received LeaderStepDown RPC: tablet_id: "c4689cdf800d479ba8645b96278c1713"
dest_uuid: "309a53f12a534385a7e5a88fb373722a"
mode: GRACEFUL
new_leader_uuid: "895c9e1595924e769a25faeed7e34644"
from {username='slave'} at 127.0.0.1:38384
I20260129 02:57:44.856320 19559 raft_consensus.cc:606] T c4689cdf800d479ba8645b96278c1713 P 309a53f12a534385a7e5a88fb373722a [term 1 LEADER]: Received request to transfer leadership to TS 895c9e1595924e769a25faeed7e34644
I20260129 02:57:44.861421 19306 catalog_manager.cc:5654] T 2e068f6d10f04c9ea17dba1375ae9b3c P a6aef613bf534ff9ae6ddbba06bb3cf1 reported cstate change: term changed from 1 to 2, leader changed from 309a53f12a534385a7e5a88fb373722a (127.18.211.3) to a6aef613bf534ff9ae6ddbba06bb3cf1 (127.18.211.1). New cstate: current_term: 2 leader_uuid: "a6aef613bf534ff9ae6ddbba06bb3cf1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6aef613bf534ff9ae6ddbba06bb3cf1" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 45187 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "895c9e1595924e769a25faeed7e34644" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 40083 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "309a53f12a534385a7e5a88fb373722a" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 34213 } health_report { overall_health: UNKNOWN } } }
I20260129 02:57:44.864118 19559 tablet_service.cc:2066] Received LeaderStepDown RPC: tablet_id: "6431d8f509f84fa6a63a3aad36a6d39d"
dest_uuid: "309a53f12a534385a7e5a88fb373722a"
mode: GRACEFUL
new_leader_uuid: "895c9e1595924e769a25faeed7e34644"
from {username='slave'} at 127.0.0.1:38384
I20260129 02:57:44.864701 19559 raft_consensus.cc:606] T 6431d8f509f84fa6a63a3aad36a6d39d P 309a53f12a534385a7e5a88fb373722a [term 1 LEADER]: Received request to transfer leadership to TS 895c9e1595924e769a25faeed7e34644
I20260129 02:57:44.980723 19667 consensus_queue.cc:1048] T 2b5edf7ea566411caea265dcfff5a86a P 895c9e1595924e769a25faeed7e34644 [LEADER]: Connected to new peer: Peer: permanent_uuid: "309a53f12a534385a7e5a88fb373722a" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 34213 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260129 02:57:44.990592 19609 consensus_queue.cc:1048] T 2b5edf7ea566411caea265dcfff5a86a P 895c9e1595924e769a25faeed7e34644 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a6aef613bf534ff9ae6ddbba06bb3cf1" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 45187 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260129 02:57:44.999317 19635 raft_consensus.cc:993] T 2b5edf7ea566411caea265dcfff5a86a P 895c9e1595924e769a25faeed7e34644: : Instructing follower a6aef613bf534ff9ae6ddbba06bb3cf1 to start an election
I20260129 02:57:44.999728 19635 raft_consensus.cc:1081] T 2b5edf7ea566411caea265dcfff5a86a P 895c9e1595924e769a25faeed7e34644 [term 1 LEADER]: Signalling peer a6aef613bf534ff9ae6ddbba06bb3cf1 to start an election
I20260129 02:57:45.001216 19411 tablet_service.cc:2038] Received Run Leader Election RPC: tablet_id: "2b5edf7ea566411caea265dcfff5a86a"
dest_uuid: "a6aef613bf534ff9ae6ddbba06bb3cf1"
from {username='slave'} at 127.0.0.1:40698
I20260129 02:57:45.001809 19411 raft_consensus.cc:493] T 2b5edf7ea566411caea265dcfff5a86a P a6aef613bf534ff9ae6ddbba06bb3cf1 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260129 02:57:45.002189 19411 raft_consensus.cc:3060] T 2b5edf7ea566411caea265dcfff5a86a P a6aef613bf534ff9ae6ddbba06bb3cf1 [term 1 FOLLOWER]: Advancing to term 2
I20260129 02:57:45.009420 19411 raft_consensus.cc:515] T 2b5edf7ea566411caea265dcfff5a86a P a6aef613bf534ff9ae6ddbba06bb3cf1 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "309a53f12a534385a7e5a88fb373722a" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 34213 } } peers { permanent_uuid: "a6aef613bf534ff9ae6ddbba06bb3cf1" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 45187 } } peers { permanent_uuid: "895c9e1595924e769a25faeed7e34644" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 40083 } }
I20260129 02:57:45.010416 19610 consensus_queue.cc:1048] T a78e86f5781f41bba56333c01607277a P a6aef613bf534ff9ae6ddbba06bb3cf1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "309a53f12a534385a7e5a88fb373722a" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 34213 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260129 02:57:45.012090 19559 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2b5edf7ea566411caea265dcfff5a86a" candidate_uuid: "a6aef613bf534ff9ae6ddbba06bb3cf1" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "309a53f12a534385a7e5a88fb373722a"
I20260129 02:57:45.012655 19559 raft_consensus.cc:3060] T 2b5edf7ea566411caea265dcfff5a86a P 309a53f12a534385a7e5a88fb373722a [term 1 FOLLOWER]: Advancing to term 2
I20260129 02:57:45.012774 19485 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2b5edf7ea566411caea265dcfff5a86a" candidate_uuid: "a6aef613bf534ff9ae6ddbba06bb3cf1" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "895c9e1595924e769a25faeed7e34644"
I20260129 02:57:45.013345 19485 raft_consensus.cc:3055] T 2b5edf7ea566411caea265dcfff5a86a P 895c9e1595924e769a25faeed7e34644 [term 1 LEADER]: Stepping down as leader of term 1
I20260129 02:57:45.013633 19485 raft_consensus.cc:740] T 2b5edf7ea566411caea265dcfff5a86a P 895c9e1595924e769a25faeed7e34644 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 895c9e1595924e769a25faeed7e34644, State: Running, Role: LEADER
I20260129 02:57:45.015802 19485 consensus_queue.cc:260] T 2b5edf7ea566411caea265dcfff5a86a P 895c9e1595924e769a25faeed7e34644 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "309a53f12a534385a7e5a88fb373722a" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 34213 } } peers { permanent_uuid: "a6aef613bf534ff9ae6ddbba06bb3cf1" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 45187 } } peers { permanent_uuid: "895c9e1595924e769a25faeed7e34644" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 40083 } }
I20260129 02:57:45.016955 19485 raft_consensus.cc:3060] T 2b5edf7ea566411caea265dcfff5a86a P 895c9e1595924e769a25faeed7e34644 [term 1 FOLLOWER]: Advancing to term 2
I20260129 02:57:45.023277 19559 raft_consensus.cc:2468] T 2b5edf7ea566411caea265dcfff5a86a P 309a53f12a534385a7e5a88fb373722a [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a6aef613bf534ff9ae6ddbba06bb3cf1 in term 2.
I20260129 02:57:45.023895 19411 leader_election.cc:290] T 2b5edf7ea566411caea265dcfff5a86a P a6aef613bf534ff9ae6ddbba06bb3cf1 [CANDIDATE]: Term 2 election: Requested vote from peers 309a53f12a534385a7e5a88fb373722a (127.18.211.3:34213), 895c9e1595924e769a25faeed7e34644 (127.18.211.2:40083)
I20260129 02:57:45.024750 19375 leader_election.cc:304] T 2b5edf7ea566411caea265dcfff5a86a P a6aef613bf534ff9ae6ddbba06bb3cf1 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 309a53f12a534385a7e5a88fb373722a, a6aef613bf534ff9ae6ddbba06bb3cf1; no voters:
I20260129 02:57:45.025730 19610 raft_consensus.cc:2804] T 2b5edf7ea566411caea265dcfff5a86a P a6aef613bf534ff9ae6ddbba06bb3cf1 [term 2 FOLLOWER]: Leader election won for term 2
I20260129 02:57:45.024474 19485 raft_consensus.cc:2468] T 2b5edf7ea566411caea265dcfff5a86a P 895c9e1595924e769a25faeed7e34644 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a6aef613bf534ff9ae6ddbba06bb3cf1 in term 2.
I20260129 02:57:45.026196 19610 raft_consensus.cc:697] T 2b5edf7ea566411caea265dcfff5a86a P a6aef613bf534ff9ae6ddbba06bb3cf1 [term 2 LEADER]: Becoming Leader. State: Replica: a6aef613bf534ff9ae6ddbba06bb3cf1, State: Running, Role: LEADER
I20260129 02:57:45.027000 19610 consensus_queue.cc:237] T 2b5edf7ea566411caea265dcfff5a86a P a6aef613bf534ff9ae6ddbba06bb3cf1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "309a53f12a534385a7e5a88fb373722a" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 34213 } } peers { permanent_uuid: "a6aef613bf534ff9ae6ddbba06bb3cf1" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 45187 } } peers { permanent_uuid: "895c9e1595924e769a25faeed7e34644" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 40083 } }
I20260129 02:57:45.033453 19610 consensus_queue.cc:1048] T a78e86f5781f41bba56333c01607277a P a6aef613bf534ff9ae6ddbba06bb3cf1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "895c9e1595924e769a25faeed7e34644" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 40083 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260129 02:57:45.035347 19306 catalog_manager.cc:5654] T 2b5edf7ea566411caea265dcfff5a86a P a6aef613bf534ff9ae6ddbba06bb3cf1 reported cstate change: term changed from 1 to 2, leader changed from 895c9e1595924e769a25faeed7e34644 (127.18.211.2) to a6aef613bf534ff9ae6ddbba06bb3cf1 (127.18.211.1). New cstate: current_term: 2 leader_uuid: "a6aef613bf534ff9ae6ddbba06bb3cf1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "309a53f12a534385a7e5a88fb373722a" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 34213 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a6aef613bf534ff9ae6ddbba06bb3cf1" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 45187 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "895c9e1595924e769a25faeed7e34644" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 40083 } health_report { overall_health: UNKNOWN } } }
I20260129 02:57:45.095790 19621 consensus_queue.cc:1048] T 3e4edd649791453f902a5ac6d5658c12 P 309a53f12a534385a7e5a88fb373722a [LEADER]: Connected to new peer: Peer: permanent_uuid: "895c9e1595924e769a25faeed7e34644" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 40083 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260129 02:57:45.107736 19670 consensus_queue.cc:1048] T 3e4edd649791453f902a5ac6d5658c12 P 309a53f12a534385a7e5a88fb373722a [LEADER]: Connected to new peer: Peer: permanent_uuid: "a6aef613bf534ff9ae6ddbba06bb3cf1" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 45187 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260129 02:57:45.235190 19560 raft_consensus.cc:1275] T 2e068f6d10f04c9ea17dba1375ae9b3c P 309a53f12a534385a7e5a88fb373722a [term 2 FOLLOWER]: Refusing update from remote peer a6aef613bf534ff9ae6ddbba06bb3cf1: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260129 02:57:45.236604 19610 consensus_queue.cc:1048] T 2e068f6d10f04c9ea17dba1375ae9b3c P a6aef613bf534ff9ae6ddbba06bb3cf1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "309a53f12a534385a7e5a88fb373722a" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 34213 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260129 02:57:45.243791 19610 raft_consensus.cc:993] T 9d5a1dc0b90642309290802698ec59f2 P a6aef613bf534ff9ae6ddbba06bb3cf1: : Instructing follower 895c9e1595924e769a25faeed7e34644 to start an election
I20260129 02:57:45.244294 19634 raft_consensus.cc:1081] T 9d5a1dc0b90642309290802698ec59f2 P a6aef613bf534ff9ae6ddbba06bb3cf1 [term 1 LEADER]: Signalling peer 895c9e1595924e769a25faeed7e34644 to start an election
I20260129 02:57:45.244506 19484 raft_consensus.cc:1275] T 2e068f6d10f04c9ea17dba1375ae9b3c P 895c9e1595924e769a25faeed7e34644 [term 2 FOLLOWER]: Refusing update from remote peer a6aef613bf534ff9ae6ddbba06bb3cf1: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260129 02:57:45.245716 19634 consensus_queue.cc:1048] T 2e068f6d10f04c9ea17dba1375ae9b3c P a6aef613bf534ff9ae6ddbba06bb3cf1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "895c9e1595924e769a25faeed7e34644" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 40083 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260129 02:57:45.245937 19484 tablet_service.cc:2038] Received Run Leader Election RPC: tablet_id: "9d5a1dc0b90642309290802698ec59f2"
dest_uuid: "895c9e1595924e769a25faeed7e34644"
from {username='slave'} at 127.0.0.1:46902
I20260129 02:57:45.246430 19484 raft_consensus.cc:493] T 9d5a1dc0b90642309290802698ec59f2 P 895c9e1595924e769a25faeed7e34644 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260129 02:57:45.246796 19484 raft_consensus.cc:3060] T 9d5a1dc0b90642309290802698ec59f2 P 895c9e1595924e769a25faeed7e34644 [term 1 FOLLOWER]: Advancing to term 2
I20260129 02:57:45.258297 19670 raft_consensus.cc:993] T c4689cdf800d479ba8645b96278c1713 P 309a53f12a534385a7e5a88fb373722a: : Instructing follower 895c9e1595924e769a25faeed7e34644 to start an election
I20260129 02:57:45.258702 19670 raft_consensus.cc:1081] T c4689cdf800d479ba8645b96278c1713 P 309a53f12a534385a7e5a88fb373722a [term 1 LEADER]: Signalling peer 895c9e1595924e769a25faeed7e34644 to start an election
I20260129 02:57:45.259804 19484 raft_consensus.cc:515] T 9d5a1dc0b90642309290802698ec59f2 P 895c9e1595924e769a25faeed7e34644 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6aef613bf534ff9ae6ddbba06bb3cf1" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 45187 } } peers { permanent_uuid: "895c9e1595924e769a25faeed7e34644" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 40083 } } peers { permanent_uuid: "309a53f12a534385a7e5a88fb373722a" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 34213 } }
I20260129 02:57:45.262398 19411 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9d5a1dc0b90642309290802698ec59f2" candidate_uuid: "895c9e1595924e769a25faeed7e34644" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "a6aef613bf534ff9ae6ddbba06bb3cf1"
I20260129 02:57:45.262691 19484 leader_election.cc:290] T 9d5a1dc0b90642309290802698ec59f2 P 895c9e1595924e769a25faeed7e34644 [CANDIDATE]: Term 2 election: Requested vote from peers a6aef613bf534ff9ae6ddbba06bb3cf1 (127.18.211.1:45187), 309a53f12a534385a7e5a88fb373722a (127.18.211.3:34213)
I20260129 02:57:45.262952 19411 raft_consensus.cc:3055] T 9d5a1dc0b90642309290802698ec59f2 P a6aef613bf534ff9ae6ddbba06bb3cf1 [term 1 LEADER]: Stepping down as leader of term 1
I20260129 02:57:45.262791 19485 tablet_service.cc:2038] Received Run Leader Election RPC: tablet_id: "c4689cdf800d479ba8645b96278c1713"
dest_uuid: "895c9e1595924e769a25faeed7e34644"
from {username='slave'} at 127.0.0.1:46898
I20260129 02:57:45.263327 19411 raft_consensus.cc:740] T 9d5a1dc0b90642309290802698ec59f2 P a6aef613bf534ff9ae6ddbba06bb3cf1 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: a6aef613bf534ff9ae6ddbba06bb3cf1, State: Running, Role: LEADER
I20260129 02:57:45.263515 19485 raft_consensus.cc:493] T c4689cdf800d479ba8645b96278c1713 P 895c9e1595924e769a25faeed7e34644 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260129 02:57:45.263877 19485 raft_consensus.cc:3060] T c4689cdf800d479ba8645b96278c1713 P 895c9e1595924e769a25faeed7e34644 [term 1 FOLLOWER]: Advancing to term 2
I20260129 02:57:45.264055 19411 consensus_queue.cc:260] T 9d5a1dc0b90642309290802698ec59f2 P a6aef613bf534ff9ae6ddbba06bb3cf1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6aef613bf534ff9ae6ddbba06bb3cf1" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 45187 } } peers { permanent_uuid: "895c9e1595924e769a25faeed7e34644" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 40083 } } peers { permanent_uuid: "309a53f12a534385a7e5a88fb373722a" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 34213 } }
I20260129 02:57:45.265323 19411 raft_consensus.cc:3060] T 9d5a1dc0b90642309290802698ec59f2 P a6aef613bf534ff9ae6ddbba06bb3cf1 [term 1 FOLLOWER]: Advancing to term 2
I20260129 02:57:45.265509 19560 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9d5a1dc0b90642309290802698ec59f2" candidate_uuid: "895c9e1595924e769a25faeed7e34644" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "309a53f12a534385a7e5a88fb373722a"
I20260129 02:57:45.266160 19560 raft_consensus.cc:3060] T 9d5a1dc0b90642309290802698ec59f2 P 309a53f12a534385a7e5a88fb373722a [term 1 FOLLOWER]: Advancing to term 2
I20260129 02:57:45.270728 19485 raft_consensus.cc:515] T c4689cdf800d479ba8645b96278c1713 P 895c9e1595924e769a25faeed7e34644 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6aef613bf534ff9ae6ddbba06bb3cf1" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 45187 } } peers { permanent_uuid: "895c9e1595924e769a25faeed7e34644" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 40083 } } peers { permanent_uuid: "309a53f12a534385a7e5a88fb373722a" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 34213 } }
I20260129 02:57:45.271849 19411 raft_consensus.cc:2468] T 9d5a1dc0b90642309290802698ec59f2 P a6aef613bf534ff9ae6ddbba06bb3cf1 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 895c9e1595924e769a25faeed7e34644 in term 2.
I20260129 02:57:45.272665 19560 raft_consensus.cc:2468] T 9d5a1dc0b90642309290802698ec59f2 P 309a53f12a534385a7e5a88fb373722a [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 895c9e1595924e769a25faeed7e34644 in term 2.
I20260129 02:57:45.272912 19450 leader_election.cc:304] T 9d5a1dc0b90642309290802698ec59f2 P 895c9e1595924e769a25faeed7e34644 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 895c9e1595924e769a25faeed7e34644, a6aef613bf534ff9ae6ddbba06bb3cf1; no voters:
I20260129 02:57:45.273509 19410 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c4689cdf800d479ba8645b96278c1713" candidate_uuid: "895c9e1595924e769a25faeed7e34644" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "a6aef613bf534ff9ae6ddbba06bb3cf1"
I20260129 02:57:45.273975 19635 raft_consensus.cc:2804] T 9d5a1dc0b90642309290802698ec59f2 P 895c9e1595924e769a25faeed7e34644 [term 2 FOLLOWER]: Leader election won for term 2
I20260129 02:57:45.274060 19410 raft_consensus.cc:3060] T c4689cdf800d479ba8645b96278c1713 P a6aef613bf534ff9ae6ddbba06bb3cf1 [term 1 FOLLOWER]: Advancing to term 2
I20260129 02:57:45.274504 19635 raft_consensus.cc:697] T 9d5a1dc0b90642309290802698ec59f2 P 895c9e1595924e769a25faeed7e34644 [term 2 LEADER]: Becoming Leader. State: Replica: 895c9e1595924e769a25faeed7e34644, State: Running, Role: LEADER
I20260129 02:57:45.275342 19635 consensus_queue.cc:237] T 9d5a1dc0b90642309290802698ec59f2 P 895c9e1595924e769a25faeed7e34644 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6aef613bf534ff9ae6ddbba06bb3cf1" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 45187 } } peers { permanent_uuid: "895c9e1595924e769a25faeed7e34644" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 40083 } } peers { permanent_uuid: "309a53f12a534385a7e5a88fb373722a" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 34213 } }
I20260129 02:57:45.275810 19485 leader_election.cc:290] T c4689cdf800d479ba8645b96278c1713 P 895c9e1595924e769a25faeed7e34644 [CANDIDATE]: Term 2 election: Requested vote from peers a6aef613bf534ff9ae6ddbba06bb3cf1 (127.18.211.1:45187), 309a53f12a534385a7e5a88fb373722a (127.18.211.3:34213)
I20260129 02:57:45.278484 19560 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c4689cdf800d479ba8645b96278c1713" candidate_uuid: "895c9e1595924e769a25faeed7e34644" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "309a53f12a534385a7e5a88fb373722a"
I20260129 02:57:45.279270 19560 raft_consensus.cc:3055] T c4689cdf800d479ba8645b96278c1713 P 309a53f12a534385a7e5a88fb373722a [term 1 LEADER]: Stepping down as leader of term 1
I20260129 02:57:45.279651 19560 raft_consensus.cc:740] T c4689cdf800d479ba8645b96278c1713 P 309a53f12a534385a7e5a88fb373722a [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 309a53f12a534385a7e5a88fb373722a, State: Running, Role: LEADER
I20260129 02:57:45.280606 19410 raft_consensus.cc:2468] T c4689cdf800d479ba8645b96278c1713 P a6aef613bf534ff9ae6ddbba06bb3cf1 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 895c9e1595924e769a25faeed7e34644 in term 2.
I20260129 02:57:45.280453 19560 consensus_queue.cc:260] T c4689cdf800d479ba8645b96278c1713 P 309a53f12a534385a7e5a88fb373722a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6aef613bf534ff9ae6ddbba06bb3cf1" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 45187 } } peers { permanent_uuid: "895c9e1595924e769a25faeed7e34644" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 40083 } } peers { permanent_uuid: "309a53f12a534385a7e5a88fb373722a" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 34213 } }
I20260129 02:57:45.281742 19560 raft_consensus.cc:3060] T c4689cdf800d479ba8645b96278c1713 P 309a53f12a534385a7e5a88fb373722a [term 1 FOLLOWER]: Advancing to term 2
I20260129 02:57:45.281720 19450 leader_election.cc:304] T c4689cdf800d479ba8645b96278c1713 P 895c9e1595924e769a25faeed7e34644 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 895c9e1595924e769a25faeed7e34644, a6aef613bf534ff9ae6ddbba06bb3cf1; no voters:
I20260129 02:57:45.282676 19635 raft_consensus.cc:2804] T c4689cdf800d479ba8645b96278c1713 P 895c9e1595924e769a25faeed7e34644 [term 2 FOLLOWER]: Leader election won for term 2
I20260129 02:57:45.283318 19635 raft_consensus.cc:697] T c4689cdf800d479ba8645b96278c1713 P 895c9e1595924e769a25faeed7e34644 [term 2 LEADER]: Becoming Leader. State: Replica: 895c9e1595924e769a25faeed7e34644, State: Running, Role: LEADER
I20260129 02:57:45.283759 19308 catalog_manager.cc:5654] T 9d5a1dc0b90642309290802698ec59f2 P 895c9e1595924e769a25faeed7e34644 reported cstate change: term changed from 1 to 2, leader changed from a6aef613bf534ff9ae6ddbba06bb3cf1 (127.18.211.1) to 895c9e1595924e769a25faeed7e34644 (127.18.211.2). New cstate: current_term: 2 leader_uuid: "895c9e1595924e769a25faeed7e34644" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6aef613bf534ff9ae6ddbba06bb3cf1" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 45187 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "895c9e1595924e769a25faeed7e34644" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 40083 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "309a53f12a534385a7e5a88fb373722a" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 34213 } health_report { overall_health: UNKNOWN } } }
I20260129 02:57:45.284144 19635 consensus_queue.cc:237] T c4689cdf800d479ba8645b96278c1713 P 895c9e1595924e769a25faeed7e34644 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6aef613bf534ff9ae6ddbba06bb3cf1" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 45187 } } peers { permanent_uuid: "895c9e1595924e769a25faeed7e34644" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 40083 } } peers { permanent_uuid: "309a53f12a534385a7e5a88fb373722a" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 34213 } }
I20260129 02:57:45.288651 19560 raft_consensus.cc:2468] T c4689cdf800d479ba8645b96278c1713 P 309a53f12a534385a7e5a88fb373722a [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 895c9e1595924e769a25faeed7e34644 in term 2.
I20260129 02:57:45.296959 19308 catalog_manager.cc:5654] T c4689cdf800d479ba8645b96278c1713 P 895c9e1595924e769a25faeed7e34644 reported cstate change: term changed from 1 to 2, leader changed from 309a53f12a534385a7e5a88fb373722a (127.18.211.3) to 895c9e1595924e769a25faeed7e34644 (127.18.211.2). New cstate: current_term: 2 leader_uuid: "895c9e1595924e769a25faeed7e34644" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6aef613bf534ff9ae6ddbba06bb3cf1" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 45187 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "895c9e1595924e769a25faeed7e34644" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 40083 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "309a53f12a534385a7e5a88fb373722a" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 34213 } health_report { overall_health: UNKNOWN } } }
I20260129 02:57:45.392210 19670 raft_consensus.cc:993] T 6431d8f509f84fa6a63a3aad36a6d39d P 309a53f12a534385a7e5a88fb373722a: : Instructing follower 895c9e1595924e769a25faeed7e34644 to start an election
I20260129 02:57:45.392627 19621 raft_consensus.cc:1081] T 6431d8f509f84fa6a63a3aad36a6d39d P 309a53f12a534385a7e5a88fb373722a [term 1 LEADER]: Signalling peer 895c9e1595924e769a25faeed7e34644 to start an election
I20260129 02:57:45.393994 19485 tablet_service.cc:2038] Received Run Leader Election RPC: tablet_id: "6431d8f509f84fa6a63a3aad36a6d39d"
dest_uuid: "895c9e1595924e769a25faeed7e34644"
from {username='slave'} at 127.0.0.1:46898
I20260129 02:57:45.394534 19485 raft_consensus.cc:493] T 6431d8f509f84fa6a63a3aad36a6d39d P 895c9e1595924e769a25faeed7e34644 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260129 02:57:45.394906 19485 raft_consensus.cc:3060] T 6431d8f509f84fa6a63a3aad36a6d39d P 895c9e1595924e769a25faeed7e34644 [term 1 FOLLOWER]: Advancing to term 2
I20260129 02:57:45.399573 19485 raft_consensus.cc:515] T 6431d8f509f84fa6a63a3aad36a6d39d P 895c9e1595924e769a25faeed7e34644 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6aef613bf534ff9ae6ddbba06bb3cf1" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 45187 } } peers { permanent_uuid: "895c9e1595924e769a25faeed7e34644" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 40083 } } peers { permanent_uuid: "309a53f12a534385a7e5a88fb373722a" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 34213 } }
I20260129 02:57:45.401186 19485 leader_election.cc:290] T 6431d8f509f84fa6a63a3aad36a6d39d P 895c9e1595924e769a25faeed7e34644 [CANDIDATE]: Term 2 election: Requested vote from peers a6aef613bf534ff9ae6ddbba06bb3cf1 (127.18.211.1:45187), 309a53f12a534385a7e5a88fb373722a (127.18.211.3:34213)
I20260129 02:57:45.401849 19410 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6431d8f509f84fa6a63a3aad36a6d39d" candidate_uuid: "895c9e1595924e769a25faeed7e34644" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "a6aef613bf534ff9ae6ddbba06bb3cf1"
I20260129 02:57:45.402125 19560 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6431d8f509f84fa6a63a3aad36a6d39d" candidate_uuid: "895c9e1595924e769a25faeed7e34644" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "309a53f12a534385a7e5a88fb373722a"
I20260129 02:57:45.402441 19410 raft_consensus.cc:3060] T 6431d8f509f84fa6a63a3aad36a6d39d P a6aef613bf534ff9ae6ddbba06bb3cf1 [term 1 FOLLOWER]: Advancing to term 2
I20260129 02:57:45.402645 19560 raft_consensus.cc:3055] T 6431d8f509f84fa6a63a3aad36a6d39d P 309a53f12a534385a7e5a88fb373722a [term 1 LEADER]: Stepping down as leader of term 1
I20260129 02:57:45.402990 19560 raft_consensus.cc:740] T 6431d8f509f84fa6a63a3aad36a6d39d P 309a53f12a534385a7e5a88fb373722a [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 309a53f12a534385a7e5a88fb373722a, State: Running, Role: LEADER
I20260129 02:57:45.403772 19560 consensus_queue.cc:260] T 6431d8f509f84fa6a63a3aad36a6d39d P 309a53f12a534385a7e5a88fb373722a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6aef613bf534ff9ae6ddbba06bb3cf1" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 45187 } } peers { permanent_uuid: "895c9e1595924e769a25faeed7e34644" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 40083 } } peers { permanent_uuid: "309a53f12a534385a7e5a88fb373722a" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 34213 } }
I20260129 02:57:45.405006 19560 raft_consensus.cc:3060] T 6431d8f509f84fa6a63a3aad36a6d39d P 309a53f12a534385a7e5a88fb373722a [term 1 FOLLOWER]: Advancing to term 2
I20260129 02:57:45.408965 19410 raft_consensus.cc:2468] T 6431d8f509f84fa6a63a3aad36a6d39d P a6aef613bf534ff9ae6ddbba06bb3cf1 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 895c9e1595924e769a25faeed7e34644 in term 2.
I20260129 02:57:45.409976 19450 leader_election.cc:304] T 6431d8f509f84fa6a63a3aad36a6d39d P 895c9e1595924e769a25faeed7e34644 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 895c9e1595924e769a25faeed7e34644, a6aef613bf534ff9ae6ddbba06bb3cf1; no voters:
I20260129 02:57:45.410703 19635 raft_consensus.cc:2804] T 6431d8f509f84fa6a63a3aad36a6d39d P 895c9e1595924e769a25faeed7e34644 [term 2 FOLLOWER]: Leader election won for term 2
I20260129 02:57:45.411121 19635 raft_consensus.cc:697] T 6431d8f509f84fa6a63a3aad36a6d39d P 895c9e1595924e769a25faeed7e34644 [term 2 LEADER]: Becoming Leader. State: Replica: 895c9e1595924e769a25faeed7e34644, State: Running, Role: LEADER
I20260129 02:57:45.411317 19560 raft_consensus.cc:2468] T 6431d8f509f84fa6a63a3aad36a6d39d P 309a53f12a534385a7e5a88fb373722a [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 895c9e1595924e769a25faeed7e34644 in term 2.
I20260129 02:57:45.411947 19635 consensus_queue.cc:237] T 6431d8f509f84fa6a63a3aad36a6d39d P 895c9e1595924e769a25faeed7e34644 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6aef613bf534ff9ae6ddbba06bb3cf1" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 45187 } } peers { permanent_uuid: "895c9e1595924e769a25faeed7e34644" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 40083 } } peers { permanent_uuid: "309a53f12a534385a7e5a88fb373722a" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 34213 } }
I20260129 02:57:45.417475 19308 catalog_manager.cc:5654] T 6431d8f509f84fa6a63a3aad36a6d39d P 895c9e1595924e769a25faeed7e34644 reported cstate change: term changed from 1 to 2, leader changed from 309a53f12a534385a7e5a88fb373722a (127.18.211.3) to 895c9e1595924e769a25faeed7e34644 (127.18.211.2). New cstate: current_term: 2 leader_uuid: "895c9e1595924e769a25faeed7e34644" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6aef613bf534ff9ae6ddbba06bb3cf1" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 45187 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "895c9e1595924e769a25faeed7e34644" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 40083 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "309a53f12a534385a7e5a88fb373722a" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 34213 } health_report { overall_health: UNKNOWN } } }
I20260129 02:57:45.476735 19485 raft_consensus.cc:1275] T 2b5edf7ea566411caea265dcfff5a86a P 895c9e1595924e769a25faeed7e34644 [term 2 FOLLOWER]: Refusing update from remote peer a6aef613bf534ff9ae6ddbba06bb3cf1: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260129 02:57:45.477929 19634 consensus_queue.cc:1048] T 2b5edf7ea566411caea265dcfff5a86a P a6aef613bf534ff9ae6ddbba06bb3cf1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "895c9e1595924e769a25faeed7e34644" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 40083 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260129 02:57:45.484851 19560 raft_consensus.cc:1275] T 2b5edf7ea566411caea265dcfff5a86a P 309a53f12a534385a7e5a88fb373722a [term 2 FOLLOWER]: Refusing update from remote peer a6aef613bf534ff9ae6ddbba06bb3cf1: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260129 02:57:45.486194 19637 consensus_queue.cc:1048] T 2b5edf7ea566411caea265dcfff5a86a P a6aef613bf534ff9ae6ddbba06bb3cf1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "309a53f12a534385a7e5a88fb373722a" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 34213 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260129 02:57:45.666519 19410 raft_consensus.cc:1275] T c4689cdf800d479ba8645b96278c1713 P a6aef613bf534ff9ae6ddbba06bb3cf1 [term 2 FOLLOWER]: Refusing update from remote peer 895c9e1595924e769a25faeed7e34644: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260129 02:57:45.667637 19635 consensus_queue.cc:1048] T c4689cdf800d479ba8645b96278c1713 P 895c9e1595924e769a25faeed7e34644 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a6aef613bf534ff9ae6ddbba06bb3cf1" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 45187 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260129 02:57:45.674652 19560 raft_consensus.cc:1275] T c4689cdf800d479ba8645b96278c1713 P 309a53f12a534385a7e5a88fb373722a [term 2 FOLLOWER]: Refusing update from remote peer 895c9e1595924e769a25faeed7e34644: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260129 02:57:45.676208 19689 consensus_queue.cc:1048] T c4689cdf800d479ba8645b96278c1713 P 895c9e1595924e769a25faeed7e34644 [LEADER]: Connected to new peer: Peer: permanent_uuid: "309a53f12a534385a7e5a88fb373722a" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 34213 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260129 02:57:45.686506 19560 raft_consensus.cc:1275] T 9d5a1dc0b90642309290802698ec59f2 P 309a53f12a534385a7e5a88fb373722a [term 2 FOLLOWER]: Refusing update from remote peer 895c9e1595924e769a25faeed7e34644: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260129 02:57:45.687919 19689 consensus_queue.cc:1048] T 9d5a1dc0b90642309290802698ec59f2 P 895c9e1595924e769a25faeed7e34644 [LEADER]: Connected to new peer: Peer: permanent_uuid: "309a53f12a534385a7e5a88fb373722a" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 34213 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260129 02:57:45.696290 19410 raft_consensus.cc:1275] T 9d5a1dc0b90642309290802698ec59f2 P a6aef613bf534ff9ae6ddbba06bb3cf1 [term 2 FOLLOWER]: Refusing update from remote peer 895c9e1595924e769a25faeed7e34644: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260129 02:57:45.697539 19689 consensus_queue.cc:1048] T 9d5a1dc0b90642309290802698ec59f2 P 895c9e1595924e769a25faeed7e34644 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a6aef613bf534ff9ae6ddbba06bb3cf1" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 45187 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260129 02:57:45.798244 19560 raft_consensus.cc:1275] T 6431d8f509f84fa6a63a3aad36a6d39d P 309a53f12a534385a7e5a88fb373722a [term 2 FOLLOWER]: Refusing update from remote peer 895c9e1595924e769a25faeed7e34644: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260129 02:57:45.799305 19609 consensus_queue.cc:1048] T 6431d8f509f84fa6a63a3aad36a6d39d P 895c9e1595924e769a25faeed7e34644 [LEADER]: Connected to new peer: Peer: permanent_uuid: "309a53f12a534385a7e5a88fb373722a" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 34213 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260129 02:57:45.807118 19410 raft_consensus.cc:1275] T 6431d8f509f84fa6a63a3aad36a6d39d P a6aef613bf534ff9ae6ddbba06bb3cf1 [term 2 FOLLOWER]: Refusing update from remote peer 895c9e1595924e769a25faeed7e34644: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260129 02:57:45.808332 19689 consensus_queue.cc:1048] T 6431d8f509f84fa6a63a3aad36a6d39d P 895c9e1595924e769a25faeed7e34644 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a6aef613bf534ff9ae6ddbba06bb3cf1" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 45187 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260129 02:57:47.900234 19276 auto_leader_rebalancer-test.cc:272] The leader distribution is
I20260129 02:57:47.900954 19276 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260129 02:57:47.919358 19411 tablet_service.cc:2066] Received LeaderStepDown RPC: tablet_id: "2b5edf7ea566411caea265dcfff5a86a"
dest_uuid: "a6aef613bf534ff9ae6ddbba06bb3cf1"
mode: GRACEFUL
new_leader_uuid: "309a53f12a534385a7e5a88fb373722a"
from {username='slave'} at 127.0.0.1:40714
I20260129 02:57:47.919900 19411 raft_consensus.cc:606] T 2b5edf7ea566411caea265dcfff5a86a P a6aef613bf534ff9ae6ddbba06bb3cf1 [term 2 LEADER]: Received request to transfer leadership to TS 309a53f12a534385a7e5a88fb373722a
I20260129 02:57:47.929318 19484 tablet_service.cc:2066] Received LeaderStepDown RPC: tablet_id: "6431d8f509f84fa6a63a3aad36a6d39d"
dest_uuid: "895c9e1595924e769a25faeed7e34644"
mode: GRACEFUL
new_leader_uuid: "309a53f12a534385a7e5a88fb373722a"
from {username='slave'} at 127.0.0.1:46922
I20260129 02:57:47.929795 19484 raft_consensus.cc:606] T 6431d8f509f84fa6a63a3aad36a6d39d P 895c9e1595924e769a25faeed7e34644 [term 2 LEADER]: Received request to transfer leadership to TS 309a53f12a534385a7e5a88fb373722a
I20260129 02:57:47.930800 19276 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 2
I20260129 02:57:47.931245 19276 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20260129 02:57:48.244483 19635 raft_consensus.cc:993] T 6431d8f509f84fa6a63a3aad36a6d39d P 895c9e1595924e769a25faeed7e34644: : Instructing follower 309a53f12a534385a7e5a88fb373722a to start an election
I20260129 02:57:48.244935 19699 raft_consensus.cc:1081] T 6431d8f509f84fa6a63a3aad36a6d39d P 895c9e1595924e769a25faeed7e34644 [term 2 LEADER]: Signalling peer 309a53f12a534385a7e5a88fb373722a to start an election
I20260129 02:57:48.246276 19560 tablet_service.cc:2038] Received Run Leader Election RPC: tablet_id: "6431d8f509f84fa6a63a3aad36a6d39d"
dest_uuid: "309a53f12a534385a7e5a88fb373722a"
from {username='slave'} at 127.0.0.1:38378
I20260129 02:57:48.246721 19560 raft_consensus.cc:493] T 6431d8f509f84fa6a63a3aad36a6d39d P 309a53f12a534385a7e5a88fb373722a [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20260129 02:57:48.246973 19560 raft_consensus.cc:3060] T 6431d8f509f84fa6a63a3aad36a6d39d P 309a53f12a534385a7e5a88fb373722a [term 2 FOLLOWER]: Advancing to term 3
I20260129 02:57:48.252060 19560 raft_consensus.cc:515] T 6431d8f509f84fa6a63a3aad36a6d39d P 309a53f12a534385a7e5a88fb373722a [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6aef613bf534ff9ae6ddbba06bb3cf1" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 45187 } } peers { permanent_uuid: "895c9e1595924e769a25faeed7e34644" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 40083 } } peers { permanent_uuid: "309a53f12a534385a7e5a88fb373722a" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 34213 } }
I20260129 02:57:48.253582 19560 leader_election.cc:290] T 6431d8f509f84fa6a63a3aad36a6d39d P 309a53f12a534385a7e5a88fb373722a [CANDIDATE]: Term 3 election: Requested vote from peers a6aef613bf534ff9ae6ddbba06bb3cf1 (127.18.211.1:45187), 895c9e1595924e769a25faeed7e34644 (127.18.211.2:40083)
I20260129 02:57:48.254248 19411 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6431d8f509f84fa6a63a3aad36a6d39d" candidate_uuid: "309a53f12a534385a7e5a88fb373722a" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "a6aef613bf534ff9ae6ddbba06bb3cf1"
I20260129 02:57:48.254611 19484 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6431d8f509f84fa6a63a3aad36a6d39d" candidate_uuid: "309a53f12a534385a7e5a88fb373722a" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "895c9e1595924e769a25faeed7e34644"
I20260129 02:57:48.254863 19411 raft_consensus.cc:3060] T 6431d8f509f84fa6a63a3aad36a6d39d P a6aef613bf534ff9ae6ddbba06bb3cf1 [term 2 FOLLOWER]: Advancing to term 3
I20260129 02:57:48.255159 19484 raft_consensus.cc:3055] T 6431d8f509f84fa6a63a3aad36a6d39d P 895c9e1595924e769a25faeed7e34644 [term 2 LEADER]: Stepping down as leader of term 2
I20260129 02:57:48.255491 19484 raft_consensus.cc:740] T 6431d8f509f84fa6a63a3aad36a6d39d P 895c9e1595924e769a25faeed7e34644 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 895c9e1595924e769a25faeed7e34644, State: Running, Role: LEADER
I20260129 02:57:48.256289 19484 consensus_queue.cc:260] T 6431d8f509f84fa6a63a3aad36a6d39d P 895c9e1595924e769a25faeed7e34644 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6aef613bf534ff9ae6ddbba06bb3cf1" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 45187 } } peers { permanent_uuid: "895c9e1595924e769a25faeed7e34644" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 40083 } } peers { permanent_uuid: "309a53f12a534385a7e5a88fb373722a" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 34213 } }
I20260129 02:57:48.257525 19484 raft_consensus.cc:3060] T 6431d8f509f84fa6a63a3aad36a6d39d P 895c9e1595924e769a25faeed7e34644 [term 2 FOLLOWER]: Advancing to term 3
I20260129 02:57:48.261756 19411 raft_consensus.cc:2468] T 6431d8f509f84fa6a63a3aad36a6d39d P a6aef613bf534ff9ae6ddbba06bb3cf1 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 309a53f12a534385a7e5a88fb373722a in term 3.
I20260129 02:57:48.262957 19524 leader_election.cc:304] T 6431d8f509f84fa6a63a3aad36a6d39d P 309a53f12a534385a7e5a88fb373722a [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 309a53f12a534385a7e5a88fb373722a, a6aef613bf534ff9ae6ddbba06bb3cf1; no voters:
I20260129 02:57:48.263693 19621 raft_consensus.cc:2804] T 6431d8f509f84fa6a63a3aad36a6d39d P 309a53f12a534385a7e5a88fb373722a [term 3 FOLLOWER]: Leader election won for term 3
I20260129 02:57:48.263955 19484 raft_consensus.cc:2468] T 6431d8f509f84fa6a63a3aad36a6d39d P 895c9e1595924e769a25faeed7e34644 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 309a53f12a534385a7e5a88fb373722a in term 3.
I20260129 02:57:48.264998 19621 raft_consensus.cc:697] T 6431d8f509f84fa6a63a3aad36a6d39d P 309a53f12a534385a7e5a88fb373722a [term 3 LEADER]: Becoming Leader. State: Replica: 309a53f12a534385a7e5a88fb373722a, State: Running, Role: LEADER
I20260129 02:57:48.265830 19621 consensus_queue.cc:237] T 6431d8f509f84fa6a63a3aad36a6d39d P 309a53f12a534385a7e5a88fb373722a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6aef613bf534ff9ae6ddbba06bb3cf1" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 45187 } } peers { permanent_uuid: "895c9e1595924e769a25faeed7e34644" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 40083 } } peers { permanent_uuid: "309a53f12a534385a7e5a88fb373722a" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 34213 } }
I20260129 02:57:48.274718 19306 catalog_manager.cc:5654] T 6431d8f509f84fa6a63a3aad36a6d39d P 309a53f12a534385a7e5a88fb373722a reported cstate change: term changed from 2 to 3, leader changed from 895c9e1595924e769a25faeed7e34644 (127.18.211.2) to 309a53f12a534385a7e5a88fb373722a (127.18.211.3). New cstate: current_term: 3 leader_uuid: "309a53f12a534385a7e5a88fb373722a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6aef613bf534ff9ae6ddbba06bb3cf1" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 45187 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "895c9e1595924e769a25faeed7e34644" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 40083 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "309a53f12a534385a7e5a88fb373722a" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 34213 } health_report { overall_health: HEALTHY } } }
I20260129 02:57:48.350935 19706 raft_consensus.cc:993] T 2b5edf7ea566411caea265dcfff5a86a P a6aef613bf534ff9ae6ddbba06bb3cf1: : Instructing follower 309a53f12a534385a7e5a88fb373722a to start an election
I20260129 02:57:48.351375 19637 raft_consensus.cc:1081] T 2b5edf7ea566411caea265dcfff5a86a P a6aef613bf534ff9ae6ddbba06bb3cf1 [term 2 LEADER]: Signalling peer 309a53f12a534385a7e5a88fb373722a to start an election
I20260129 02:57:48.352602 19560 tablet_service.cc:2038] Received Run Leader Election RPC: tablet_id: "2b5edf7ea566411caea265dcfff5a86a"
dest_uuid: "309a53f12a534385a7e5a88fb373722a"
from {username='slave'} at 127.0.0.1:38382
I20260129 02:57:48.353080 19560 raft_consensus.cc:493] T 2b5edf7ea566411caea265dcfff5a86a P 309a53f12a534385a7e5a88fb373722a [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20260129 02:57:48.353384 19560 raft_consensus.cc:3060] T 2b5edf7ea566411caea265dcfff5a86a P 309a53f12a534385a7e5a88fb373722a [term 2 FOLLOWER]: Advancing to term 3
I20260129 02:57:48.357853 19560 raft_consensus.cc:515] T 2b5edf7ea566411caea265dcfff5a86a P 309a53f12a534385a7e5a88fb373722a [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "309a53f12a534385a7e5a88fb373722a" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 34213 } } peers { permanent_uuid: "a6aef613bf534ff9ae6ddbba06bb3cf1" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 45187 } } peers { permanent_uuid: "895c9e1595924e769a25faeed7e34644" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 40083 } }
I20260129 02:57:48.359264 19560 leader_election.cc:290] T 2b5edf7ea566411caea265dcfff5a86a P 309a53f12a534385a7e5a88fb373722a [CANDIDATE]: Term 3 election: Requested vote from peers a6aef613bf534ff9ae6ddbba06bb3cf1 (127.18.211.1:45187), 895c9e1595924e769a25faeed7e34644 (127.18.211.2:40083)
I20260129 02:57:48.360096 19411 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2b5edf7ea566411caea265dcfff5a86a" candidate_uuid: "309a53f12a534385a7e5a88fb373722a" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "a6aef613bf534ff9ae6ddbba06bb3cf1"
I20260129 02:57:48.360361 19484 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2b5edf7ea566411caea265dcfff5a86a" candidate_uuid: "309a53f12a534385a7e5a88fb373722a" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "895c9e1595924e769a25faeed7e34644"
I20260129 02:57:48.360661 19411 raft_consensus.cc:3055] T 2b5edf7ea566411caea265dcfff5a86a P a6aef613bf534ff9ae6ddbba06bb3cf1 [term 2 LEADER]: Stepping down as leader of term 2
I20260129 02:57:48.360877 19484 raft_consensus.cc:3060] T 2b5edf7ea566411caea265dcfff5a86a P 895c9e1595924e769a25faeed7e34644 [term 2 FOLLOWER]: Advancing to term 3
I20260129 02:57:48.361016 19411 raft_consensus.cc:740] T 2b5edf7ea566411caea265dcfff5a86a P a6aef613bf534ff9ae6ddbba06bb3cf1 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: a6aef613bf534ff9ae6ddbba06bb3cf1, State: Running, Role: LEADER
I20260129 02:57:48.361701 19411 consensus_queue.cc:260] T 2b5edf7ea566411caea265dcfff5a86a P a6aef613bf534ff9ae6ddbba06bb3cf1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "309a53f12a534385a7e5a88fb373722a" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 34213 } } peers { permanent_uuid: "a6aef613bf534ff9ae6ddbba06bb3cf1" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 45187 } } peers { permanent_uuid: "895c9e1595924e769a25faeed7e34644" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 40083 } }
I20260129 02:57:48.362535 19411 raft_consensus.cc:3060] T 2b5edf7ea566411caea265dcfff5a86a P a6aef613bf534ff9ae6ddbba06bb3cf1 [term 2 FOLLOWER]: Advancing to term 3
I20260129 02:57:48.365530 19484 raft_consensus.cc:2468] T 2b5edf7ea566411caea265dcfff5a86a P 895c9e1595924e769a25faeed7e34644 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 309a53f12a534385a7e5a88fb373722a in term 3.
I20260129 02:57:48.366519 19524 leader_election.cc:304] T 2b5edf7ea566411caea265dcfff5a86a P 309a53f12a534385a7e5a88fb373722a [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 309a53f12a534385a7e5a88fb373722a, 895c9e1595924e769a25faeed7e34644; no voters:
I20260129 02:57:48.366937 19411 raft_consensus.cc:2468] T 2b5edf7ea566411caea265dcfff5a86a P a6aef613bf534ff9ae6ddbba06bb3cf1 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 309a53f12a534385a7e5a88fb373722a in term 3.
I20260129 02:57:48.367142 19621 raft_consensus.cc:2804] T 2b5edf7ea566411caea265dcfff5a86a P 309a53f12a534385a7e5a88fb373722a [term 3 FOLLOWER]: Leader election won for term 3
I20260129 02:57:48.367590 19621 raft_consensus.cc:697] T 2b5edf7ea566411caea265dcfff5a86a P 309a53f12a534385a7e5a88fb373722a [term 3 LEADER]: Becoming Leader. State: Replica: 309a53f12a534385a7e5a88fb373722a, State: Running, Role: LEADER
I20260129 02:57:48.368355 19621 consensus_queue.cc:237] T 2b5edf7ea566411caea265dcfff5a86a P 309a53f12a534385a7e5a88fb373722a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "309a53f12a534385a7e5a88fb373722a" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 34213 } } peers { permanent_uuid: "a6aef613bf534ff9ae6ddbba06bb3cf1" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 45187 } } peers { permanent_uuid: "895c9e1595924e769a25faeed7e34644" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 40083 } }
I20260129 02:57:48.374982 19306 catalog_manager.cc:5654] T 2b5edf7ea566411caea265dcfff5a86a P 309a53f12a534385a7e5a88fb373722a reported cstate change: term changed from 2 to 3, leader changed from a6aef613bf534ff9ae6ddbba06bb3cf1 (127.18.211.1) to 309a53f12a534385a7e5a88fb373722a (127.18.211.3). New cstate: current_term: 3 leader_uuid: "309a53f12a534385a7e5a88fb373722a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "309a53f12a534385a7e5a88fb373722a" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 34213 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a6aef613bf534ff9ae6ddbba06bb3cf1" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 45187 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "895c9e1595924e769a25faeed7e34644" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 40083 } health_report { overall_health: UNKNOWN } } }
I20260129 02:57:48.657871 19411 raft_consensus.cc:1275] T 6431d8f509f84fa6a63a3aad36a6d39d P a6aef613bf534ff9ae6ddbba06bb3cf1 [term 3 FOLLOWER]: Refusing update from remote peer 309a53f12a534385a7e5a88fb373722a: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260129 02:57:48.659045 19621 consensus_queue.cc:1048] T 6431d8f509f84fa6a63a3aad36a6d39d P 309a53f12a534385a7e5a88fb373722a [LEADER]: Connected to new peer: Peer: permanent_uuid: "a6aef613bf534ff9ae6ddbba06bb3cf1" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 45187 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260129 02:57:48.668996 19484 raft_consensus.cc:1275] T 6431d8f509f84fa6a63a3aad36a6d39d P 895c9e1595924e769a25faeed7e34644 [term 3 FOLLOWER]: Refusing update from remote peer 309a53f12a534385a7e5a88fb373722a: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260129 02:57:48.671241 19711 consensus_queue.cc:1048] T 6431d8f509f84fa6a63a3aad36a6d39d P 309a53f12a534385a7e5a88fb373722a [LEADER]: Connected to new peer: Peer: permanent_uuid: "895c9e1595924e769a25faeed7e34644" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 40083 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260129 02:57:48.765559 19484 raft_consensus.cc:1275] T 2b5edf7ea566411caea265dcfff5a86a P 895c9e1595924e769a25faeed7e34644 [term 3 FOLLOWER]: Refusing update from remote peer 309a53f12a534385a7e5a88fb373722a: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260129 02:57:48.766767 19711 consensus_queue.cc:1048] T 2b5edf7ea566411caea265dcfff5a86a P 309a53f12a534385a7e5a88fb373722a [LEADER]: Connected to new peer: Peer: permanent_uuid: "895c9e1595924e769a25faeed7e34644" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 40083 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260129 02:57:48.774606 19411 raft_consensus.cc:1275] T 2b5edf7ea566411caea265dcfff5a86a P a6aef613bf534ff9ae6ddbba06bb3cf1 [term 3 FOLLOWER]: Refusing update from remote peer 309a53f12a534385a7e5a88fb373722a: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260129 02:57:48.776384 19621 consensus_queue.cc:1048] T 2b5edf7ea566411caea265dcfff5a86a P 309a53f12a534385a7e5a88fb373722a [LEADER]: Connected to new peer: Peer: permanent_uuid: "a6aef613bf534ff9ae6ddbba06bb3cf1" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 45187 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260129 02:57:48.932135 19276 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260129 02:57:48.937219 19276 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260129 02:57:48.937592 19276 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20260129 02:57:49.938446 19276 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260129 02:57:49.944031 19276 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260129 02:57:49.944388 19276 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20260129 02:57:50.945217 19276 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260129 02:57:50.949743 19276 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260129 02:57:50.950139 19276 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20260129 02:57:51.951259 19276 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260129 02:57:51.956061 19276 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260129 02:57:51.956460 19276 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20260129 02:57:52.957360 19276 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260129 02:57:52.962148 19276 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260129 02:57:52.962553 19276 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20260129 02:57:53.963577 19276 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260129 02:57:53.968327 19276 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260129 02:57:53.968734 19276 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20260129 02:57:54.969571 19276 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260129 02:57:54.974205 19276 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260129 02:57:54.974599 19276 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20260129 02:57:55.975548 19276 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260129 02:57:55.980687 19276 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260129 02:57:55.981082 19276 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20260129 02:57:56.982164 19276 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260129 02:57:56.987063 19276 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260129 02:57:56.987489 19276 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20260129 02:57:58.014150 19276 auto_leader_rebalancer-test.cc:291] The leader distribution is
I20260129 02:57:58.018290 19560 tablet_service.cc:2066] Received LeaderStepDown RPC: tablet_id: "2b5edf7ea566411caea265dcfff5a86a"
dest_uuid: "309a53f12a534385a7e5a88fb373722a"
mode: GRACEFUL
new_leader_uuid: "895c9e1595924e769a25faeed7e34644"
from {username='slave'} at 127.0.0.1:38384
I20260129 02:57:58.018901 19560 raft_consensus.cc:606] T 2b5edf7ea566411caea265dcfff5a86a P 309a53f12a534385a7e5a88fb373722a [term 3 LEADER]: Received request to transfer leadership to TS 895c9e1595924e769a25faeed7e34644
I20260129 02:57:58.023855 19410 tablet_service.cc:2066] Received LeaderStepDown RPC: tablet_id: "e5ce9ed62ba242cabb3d6e5371679ba7"
dest_uuid: "a6aef613bf534ff9ae6ddbba06bb3cf1"
mode: GRACEFUL
new_leader_uuid: "895c9e1595924e769a25faeed7e34644"
from {username='slave'} at 127.0.0.1:40704
I20260129 02:57:58.024277 19410 raft_consensus.cc:606] T e5ce9ed62ba242cabb3d6e5371679ba7 P a6aef613bf534ff9ae6ddbba06bb3cf1 [term 1 LEADER]: Received request to transfer leadership to TS 895c9e1595924e769a25faeed7e34644
I20260129 02:57:58.028375 19410 tablet_service.cc:2066] Received LeaderStepDown RPC: tablet_id: "2e068f6d10f04c9ea17dba1375ae9b3c"
dest_uuid: "a6aef613bf534ff9ae6ddbba06bb3cf1"
mode: GRACEFUL
new_leader_uuid: "895c9e1595924e769a25faeed7e34644"
from {username='slave'} at 127.0.0.1:40704
I20260129 02:57:58.028818 19410 raft_consensus.cc:606] T 2e068f6d10f04c9ea17dba1375ae9b3c P a6aef613bf534ff9ae6ddbba06bb3cf1 [term 2 LEADER]: Received request to transfer leadership to TS 895c9e1595924e769a25faeed7e34644
I20260129 02:57:58.033090 19410 tablet_service.cc:2066] Received LeaderStepDown RPC: tablet_id: "a78e86f5781f41bba56333c01607277a"
dest_uuid: "a6aef613bf534ff9ae6ddbba06bb3cf1"
mode: GRACEFUL
new_leader_uuid: "895c9e1595924e769a25faeed7e34644"
from {username='slave'} at 127.0.0.1:40704
I20260129 02:57:58.033524 19410 raft_consensus.cc:606] T a78e86f5781f41bba56333c01607277a P a6aef613bf534ff9ae6ddbba06bb3cf1 [term 1 LEADER]: Received request to transfer leadership to TS 895c9e1595924e769a25faeed7e34644
I20260129 02:57:58.042423 19560 tablet_service.cc:2066] Received LeaderStepDown RPC: tablet_id: "6431d8f509f84fa6a63a3aad36a6d39d"
dest_uuid: "309a53f12a534385a7e5a88fb373722a"
mode: GRACEFUL
new_leader_uuid: "895c9e1595924e769a25faeed7e34644"
from {username='slave'} at 127.0.0.1:38384
I20260129 02:57:58.042842 19560 raft_consensus.cc:606] T 6431d8f509f84fa6a63a3aad36a6d39d P 309a53f12a534385a7e5a88fb373722a [term 3 LEADER]: Received request to transfer leadership to TS 895c9e1595924e769a25faeed7e34644
I20260129 02:57:58.311589 19721 raft_consensus.cc:993] T e5ce9ed62ba242cabb3d6e5371679ba7 P a6aef613bf534ff9ae6ddbba06bb3cf1: : Instructing follower 895c9e1595924e769a25faeed7e34644 to start an election
I20260129 02:57:58.311996 19721 raft_consensus.cc:1081] T e5ce9ed62ba242cabb3d6e5371679ba7 P a6aef613bf534ff9ae6ddbba06bb3cf1 [term 1 LEADER]: Signalling peer 895c9e1595924e769a25faeed7e34644 to start an election
I20260129 02:57:58.313462 19485 tablet_service.cc:2038] Received Run Leader Election RPC: tablet_id: "e5ce9ed62ba242cabb3d6e5371679ba7"
dest_uuid: "895c9e1595924e769a25faeed7e34644"
from {username='slave'} at 127.0.0.1:46902
I20260129 02:57:58.313917 19485 raft_consensus.cc:493] T e5ce9ed62ba242cabb3d6e5371679ba7 P 895c9e1595924e769a25faeed7e34644 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260129 02:57:58.314226 19485 raft_consensus.cc:3060] T e5ce9ed62ba242cabb3d6e5371679ba7 P 895c9e1595924e769a25faeed7e34644 [term 1 FOLLOWER]: Advancing to term 2
I20260129 02:57:58.318946 19485 raft_consensus.cc:515] T e5ce9ed62ba242cabb3d6e5371679ba7 P 895c9e1595924e769a25faeed7e34644 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6aef613bf534ff9ae6ddbba06bb3cf1" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 45187 } } peers { permanent_uuid: "895c9e1595924e769a25faeed7e34644" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 40083 } } peers { permanent_uuid: "309a53f12a534385a7e5a88fb373722a" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 34213 } }
I20260129 02:57:58.320324 19485 leader_election.cc:290] T e5ce9ed62ba242cabb3d6e5371679ba7 P 895c9e1595924e769a25faeed7e34644 [CANDIDATE]: Term 2 election: Requested vote from peers a6aef613bf534ff9ae6ddbba06bb3cf1 (127.18.211.1:45187), 309a53f12a534385a7e5a88fb373722a (127.18.211.3:34213)
I20260129 02:57:58.321442 19560 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e5ce9ed62ba242cabb3d6e5371679ba7" candidate_uuid: "895c9e1595924e769a25faeed7e34644" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "309a53f12a534385a7e5a88fb373722a"
I20260129 02:57:58.321439 19411 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e5ce9ed62ba242cabb3d6e5371679ba7" candidate_uuid: "895c9e1595924e769a25faeed7e34644" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "a6aef613bf534ff9ae6ddbba06bb3cf1"
I20260129 02:57:58.322055 19560 raft_consensus.cc:3060] T e5ce9ed62ba242cabb3d6e5371679ba7 P 309a53f12a534385a7e5a88fb373722a [term 1 FOLLOWER]: Advancing to term 2
I20260129 02:57:58.322118 19411 raft_consensus.cc:3055] T e5ce9ed62ba242cabb3d6e5371679ba7 P a6aef613bf534ff9ae6ddbba06bb3cf1 [term 1 LEADER]: Stepping down as leader of term 1
I20260129 02:57:58.322499 19411 raft_consensus.cc:740] T e5ce9ed62ba242cabb3d6e5371679ba7 P a6aef613bf534ff9ae6ddbba06bb3cf1 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: a6aef613bf534ff9ae6ddbba06bb3cf1, State: Running, Role: LEADER
I20260129 02:57:58.323098 19411 consensus_queue.cc:260] T e5ce9ed62ba242cabb3d6e5371679ba7 P a6aef613bf534ff9ae6ddbba06bb3cf1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6aef613bf534ff9ae6ddbba06bb3cf1" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 45187 } } peers { permanent_uuid: "895c9e1595924e769a25faeed7e34644" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 40083 } } peers { permanent_uuid: "309a53f12a534385a7e5a88fb373722a" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 34213 } }
I20260129 02:57:58.324167 19411 raft_consensus.cc:3060] T e5ce9ed62ba242cabb3d6e5371679ba7 P a6aef613bf534ff9ae6ddbba06bb3cf1 [term 1 FOLLOWER]: Advancing to term 2
I20260129 02:57:58.327323 19560 raft_consensus.cc:2468] T e5ce9ed62ba242cabb3d6e5371679ba7 P 309a53f12a534385a7e5a88fb373722a [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 895c9e1595924e769a25faeed7e34644 in term 2.
I20260129 02:57:58.328507 19450 leader_election.cc:304] T e5ce9ed62ba242cabb3d6e5371679ba7 P 895c9e1595924e769a25faeed7e34644 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 309a53f12a534385a7e5a88fb373722a, 895c9e1595924e769a25faeed7e34644; no voters:
I20260129 02:57:58.328956 19411 raft_consensus.cc:2468] T e5ce9ed62ba242cabb3d6e5371679ba7 P a6aef613bf534ff9ae6ddbba06bb3cf1 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 895c9e1595924e769a25faeed7e34644 in term 2.
I20260129 02:57:58.329370 19723 raft_consensus.cc:2804] T e5ce9ed62ba242cabb3d6e5371679ba7 P 895c9e1595924e769a25faeed7e34644 [term 2 FOLLOWER]: Leader election won for term 2
I20260129 02:57:58.330564 19723 raft_consensus.cc:697] T e5ce9ed62ba242cabb3d6e5371679ba7 P 895c9e1595924e769a25faeed7e34644 [term 2 LEADER]: Becoming Leader. State: Replica: 895c9e1595924e769a25faeed7e34644, State: Running, Role: LEADER
I20260129 02:57:58.331471 19723 consensus_queue.cc:237] T e5ce9ed62ba242cabb3d6e5371679ba7 P 895c9e1595924e769a25faeed7e34644 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6aef613bf534ff9ae6ddbba06bb3cf1" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 45187 } } peers { permanent_uuid: "895c9e1595924e769a25faeed7e34644" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 40083 } } peers { permanent_uuid: "309a53f12a534385a7e5a88fb373722a" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 34213 } }
I20260129 02:57:58.337970 19306 catalog_manager.cc:5654] T e5ce9ed62ba242cabb3d6e5371679ba7 P 895c9e1595924e769a25faeed7e34644 reported cstate change: term changed from 1 to 2, leader changed from a6aef613bf534ff9ae6ddbba06bb3cf1 (127.18.211.1) to 895c9e1595924e769a25faeed7e34644 (127.18.211.2). New cstate: current_term: 2 leader_uuid: "895c9e1595924e769a25faeed7e34644" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6aef613bf534ff9ae6ddbba06bb3cf1" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 45187 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "895c9e1595924e769a25faeed7e34644" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 40083 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "309a53f12a534385a7e5a88fb373722a" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 34213 } health_report { overall_health: UNKNOWN } } }
I20260129 02:57:58.342810 19726 raft_consensus.cc:993] T a78e86f5781f41bba56333c01607277a P a6aef613bf534ff9ae6ddbba06bb3cf1: : Instructing follower 895c9e1595924e769a25faeed7e34644 to start an election
I20260129 02:57:58.343278 19721 raft_consensus.cc:1081] T a78e86f5781f41bba56333c01607277a P a6aef613bf534ff9ae6ddbba06bb3cf1 [term 1 LEADER]: Signalling peer 895c9e1595924e769a25faeed7e34644 to start an election
I20260129 02:57:58.344808 19485 tablet_service.cc:2038] Received Run Leader Election RPC: tablet_id: "a78e86f5781f41bba56333c01607277a"
dest_uuid: "895c9e1595924e769a25faeed7e34644"
from {username='slave'} at 127.0.0.1:46902
I20260129 02:57:58.345376 19485 raft_consensus.cc:493] T a78e86f5781f41bba56333c01607277a P 895c9e1595924e769a25faeed7e34644 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260129 02:57:58.345732 19485 raft_consensus.cc:3060] T a78e86f5781f41bba56333c01607277a P 895c9e1595924e769a25faeed7e34644 [term 1 FOLLOWER]: Advancing to term 2
I20260129 02:57:58.351918 19485 raft_consensus.cc:515] T a78e86f5781f41bba56333c01607277a P 895c9e1595924e769a25faeed7e34644 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6aef613bf534ff9ae6ddbba06bb3cf1" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 45187 } } peers { permanent_uuid: "895c9e1595924e769a25faeed7e34644" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 40083 } } peers { permanent_uuid: "309a53f12a534385a7e5a88fb373722a" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 34213 } }
I20260129 02:57:58.353096 19485 leader_election.cc:290] T a78e86f5781f41bba56333c01607277a P 895c9e1595924e769a25faeed7e34644 [CANDIDATE]: Term 2 election: Requested vote from peers a6aef613bf534ff9ae6ddbba06bb3cf1 (127.18.211.1:45187), 309a53f12a534385a7e5a88fb373722a (127.18.211.3:34213)
I20260129 02:57:58.353927 19411 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a78e86f5781f41bba56333c01607277a" candidate_uuid: "895c9e1595924e769a25faeed7e34644" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "a6aef613bf534ff9ae6ddbba06bb3cf1"
I20260129 02:57:58.354148 19560 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a78e86f5781f41bba56333c01607277a" candidate_uuid: "895c9e1595924e769a25faeed7e34644" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "309a53f12a534385a7e5a88fb373722a"
I20260129 02:57:58.354504 19411 raft_consensus.cc:3055] T a78e86f5781f41bba56333c01607277a P a6aef613bf534ff9ae6ddbba06bb3cf1 [term 1 LEADER]: Stepping down as leader of term 1
I20260129 02:57:58.354615 19560 raft_consensus.cc:3060] T a78e86f5781f41bba56333c01607277a P 309a53f12a534385a7e5a88fb373722a [term 1 FOLLOWER]: Advancing to term 2
I20260129 02:57:58.354912 19411 raft_consensus.cc:740] T a78e86f5781f41bba56333c01607277a P a6aef613bf534ff9ae6ddbba06bb3cf1 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: a6aef613bf534ff9ae6ddbba06bb3cf1, State: Running, Role: LEADER
I20260129 02:57:58.355659 19411 consensus_queue.cc:260] T a78e86f5781f41bba56333c01607277a P a6aef613bf534ff9ae6ddbba06bb3cf1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6aef613bf534ff9ae6ddbba06bb3cf1" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 45187 } } peers { permanent_uuid: "895c9e1595924e769a25faeed7e34644" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 40083 } } peers { permanent_uuid: "309a53f12a534385a7e5a88fb373722a" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 34213 } }
I20260129 02:57:58.356904 19411 raft_consensus.cc:3060] T a78e86f5781f41bba56333c01607277a P a6aef613bf534ff9ae6ddbba06bb3cf1 [term 1 FOLLOWER]: Advancing to term 2
I20260129 02:57:58.360898 19560 raft_consensus.cc:2468] T a78e86f5781f41bba56333c01607277a P 309a53f12a534385a7e5a88fb373722a [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 895c9e1595924e769a25faeed7e34644 in term 2.
I20260129 02:57:58.361903 19450 leader_election.cc:304] T a78e86f5781f41bba56333c01607277a P 895c9e1595924e769a25faeed7e34644 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 309a53f12a534385a7e5a88fb373722a, 895c9e1595924e769a25faeed7e34644; no voters:
I20260129 02:57:58.362607 19723 raft_consensus.cc:2804] T a78e86f5781f41bba56333c01607277a P 895c9e1595924e769a25faeed7e34644 [term 2 FOLLOWER]: Leader election won for term 2
I20260129 02:57:58.363128 19723 raft_consensus.cc:697] T a78e86f5781f41bba56333c01607277a P 895c9e1595924e769a25faeed7e34644 [term 2 LEADER]: Becoming Leader. State: Replica: 895c9e1595924e769a25faeed7e34644, State: Running, Role: LEADER
I20260129 02:57:58.363111 19411 raft_consensus.cc:2468] T a78e86f5781f41bba56333c01607277a P a6aef613bf534ff9ae6ddbba06bb3cf1 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 895c9e1595924e769a25faeed7e34644 in term 2.
I20260129 02:57:58.364238 19723 consensus_queue.cc:237] T a78e86f5781f41bba56333c01607277a P 895c9e1595924e769a25faeed7e34644 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6aef613bf534ff9ae6ddbba06bb3cf1" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 45187 } } peers { permanent_uuid: "895c9e1595924e769a25faeed7e34644" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 40083 } } peers { permanent_uuid: "309a53f12a534385a7e5a88fb373722a" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 34213 } }
I20260129 02:57:58.371996 19734 raft_consensus.cc:993] T 6431d8f509f84fa6a63a3aad36a6d39d P 309a53f12a534385a7e5a88fb373722a: : Instructing follower 895c9e1595924e769a25faeed7e34644 to start an election
I20260129 02:57:58.372555 19724 raft_consensus.cc:1081] T 6431d8f509f84fa6a63a3aad36a6d39d P 309a53f12a534385a7e5a88fb373722a [term 3 LEADER]: Signalling peer 895c9e1595924e769a25faeed7e34644 to start an election
I20260129 02:57:58.372153 19306 catalog_manager.cc:5654] T a78e86f5781f41bba56333c01607277a P 895c9e1595924e769a25faeed7e34644 reported cstate change: term changed from 1 to 2, leader changed from a6aef613bf534ff9ae6ddbba06bb3cf1 (127.18.211.1) to 895c9e1595924e769a25faeed7e34644 (127.18.211.2). New cstate: current_term: 2 leader_uuid: "895c9e1595924e769a25faeed7e34644" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6aef613bf534ff9ae6ddbba06bb3cf1" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 45187 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "895c9e1595924e769a25faeed7e34644" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 40083 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "309a53f12a534385a7e5a88fb373722a" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 34213 } health_report { overall_health: UNKNOWN } } }
I20260129 02:57:58.373988 19485 tablet_service.cc:2038] Received Run Leader Election RPC: tablet_id: "6431d8f509f84fa6a63a3aad36a6d39d"
dest_uuid: "895c9e1595924e769a25faeed7e34644"
from {username='slave'} at 127.0.0.1:46898
I20260129 02:57:58.374456 19485 raft_consensus.cc:493] T 6431d8f509f84fa6a63a3aad36a6d39d P 895c9e1595924e769a25faeed7e34644 [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20260129 02:57:58.374826 19485 raft_consensus.cc:3060] T 6431d8f509f84fa6a63a3aad36a6d39d P 895c9e1595924e769a25faeed7e34644 [term 3 FOLLOWER]: Advancing to term 4
I20260129 02:57:58.379671 19485 raft_consensus.cc:515] T 6431d8f509f84fa6a63a3aad36a6d39d P 895c9e1595924e769a25faeed7e34644 [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6aef613bf534ff9ae6ddbba06bb3cf1" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 45187 } } peers { permanent_uuid: "895c9e1595924e769a25faeed7e34644" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 40083 } } peers { permanent_uuid: "309a53f12a534385a7e5a88fb373722a" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 34213 } }
I20260129 02:57:58.381145 19485 leader_election.cc:290] T 6431d8f509f84fa6a63a3aad36a6d39d P 895c9e1595924e769a25faeed7e34644 [CANDIDATE]: Term 4 election: Requested vote from peers a6aef613bf534ff9ae6ddbba06bb3cf1 (127.18.211.1:45187), 309a53f12a534385a7e5a88fb373722a (127.18.211.3:34213)
I20260129 02:57:58.382118 19560 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6431d8f509f84fa6a63a3aad36a6d39d" candidate_uuid: "895c9e1595924e769a25faeed7e34644" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: true dest_uuid: "309a53f12a534385a7e5a88fb373722a"
I20260129 02:57:58.382190 19411 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6431d8f509f84fa6a63a3aad36a6d39d" candidate_uuid: "895c9e1595924e769a25faeed7e34644" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: true dest_uuid: "a6aef613bf534ff9ae6ddbba06bb3cf1"
I20260129 02:57:58.382558 19560 raft_consensus.cc:3055] T 6431d8f509f84fa6a63a3aad36a6d39d P 309a53f12a534385a7e5a88fb373722a [term 3 LEADER]: Stepping down as leader of term 3
I20260129 02:57:58.382740 19411 raft_consensus.cc:3060] T 6431d8f509f84fa6a63a3aad36a6d39d P a6aef613bf534ff9ae6ddbba06bb3cf1 [term 3 FOLLOWER]: Advancing to term 4
I20260129 02:57:58.382851 19560 raft_consensus.cc:740] T 6431d8f509f84fa6a63a3aad36a6d39d P 309a53f12a534385a7e5a88fb373722a [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 309a53f12a534385a7e5a88fb373722a, State: Running, Role: LEADER
I20260129 02:57:58.383466 19560 consensus_queue.cc:260] T 6431d8f509f84fa6a63a3aad36a6d39d P 309a53f12a534385a7e5a88fb373722a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6aef613bf534ff9ae6ddbba06bb3cf1" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 45187 } } peers { permanent_uuid: "895c9e1595924e769a25faeed7e34644" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 40083 } } peers { permanent_uuid: "309a53f12a534385a7e5a88fb373722a" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 34213 } }
I20260129 02:57:58.384392 19560 raft_consensus.cc:3060] T 6431d8f509f84fa6a63a3aad36a6d39d P 309a53f12a534385a7e5a88fb373722a [term 3 FOLLOWER]: Advancing to term 4
I20260129 02:57:58.387007 19411 raft_consensus.cc:2468] T 6431d8f509f84fa6a63a3aad36a6d39d P a6aef613bf534ff9ae6ddbba06bb3cf1 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 895c9e1595924e769a25faeed7e34644 in term 4.
I20260129 02:57:58.387914 19450 leader_election.cc:304] T 6431d8f509f84fa6a63a3aad36a6d39d P 895c9e1595924e769a25faeed7e34644 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 895c9e1595924e769a25faeed7e34644, a6aef613bf534ff9ae6ddbba06bb3cf1; no voters:
I20260129 02:57:58.388029 19560 raft_consensus.cc:2468] T 6431d8f509f84fa6a63a3aad36a6d39d P 309a53f12a534385a7e5a88fb373722a [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 895c9e1595924e769a25faeed7e34644 in term 4.
I20260129 02:57:58.388574 19723 raft_consensus.cc:2804] T 6431d8f509f84fa6a63a3aad36a6d39d P 895c9e1595924e769a25faeed7e34644 [term 4 FOLLOWER]: Leader election won for term 4
I20260129 02:57:58.389038 19723 raft_consensus.cc:697] T 6431d8f509f84fa6a63a3aad36a6d39d P 895c9e1595924e769a25faeed7e34644 [term 4 LEADER]: Becoming Leader. State: Replica: 895c9e1595924e769a25faeed7e34644, State: Running, Role: LEADER
I20260129 02:57:58.389832 19723 consensus_queue.cc:237] T 6431d8f509f84fa6a63a3aad36a6d39d P 895c9e1595924e769a25faeed7e34644 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6aef613bf534ff9ae6ddbba06bb3cf1" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 45187 } } peers { permanent_uuid: "895c9e1595924e769a25faeed7e34644" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 40083 } } peers { permanent_uuid: "309a53f12a534385a7e5a88fb373722a" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 34213 } }
I20260129 02:57:58.395704 19306 catalog_manager.cc:5654] T 6431d8f509f84fa6a63a3aad36a6d39d P 895c9e1595924e769a25faeed7e34644 reported cstate change: term changed from 3 to 4, leader changed from 309a53f12a534385a7e5a88fb373722a (127.18.211.3) to 895c9e1595924e769a25faeed7e34644 (127.18.211.2). New cstate: current_term: 4 leader_uuid: "895c9e1595924e769a25faeed7e34644" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6aef613bf534ff9ae6ddbba06bb3cf1" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 45187 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "895c9e1595924e769a25faeed7e34644" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 40083 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "309a53f12a534385a7e5a88fb373722a" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 34213 } health_report { overall_health: UNKNOWN } } }
I20260129 02:57:58.427532 19734 raft_consensus.cc:993] T 2b5edf7ea566411caea265dcfff5a86a P 309a53f12a534385a7e5a88fb373722a: : Instructing follower 895c9e1595924e769a25faeed7e34644 to start an election
I20260129 02:57:58.427978 19734 raft_consensus.cc:1081] T 2b5edf7ea566411caea265dcfff5a86a P 309a53f12a534385a7e5a88fb373722a [term 3 LEADER]: Signalling peer 895c9e1595924e769a25faeed7e34644 to start an election
I20260129 02:57:58.429395 19485 tablet_service.cc:2038] Received Run Leader Election RPC: tablet_id: "2b5edf7ea566411caea265dcfff5a86a"
dest_uuid: "895c9e1595924e769a25faeed7e34644"
from {username='slave'} at 127.0.0.1:46898
I20260129 02:57:58.429798 19485 raft_consensus.cc:493] T 2b5edf7ea566411caea265dcfff5a86a P 895c9e1595924e769a25faeed7e34644 [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20260129 02:57:58.430091 19485 raft_consensus.cc:3060] T 2b5edf7ea566411caea265dcfff5a86a P 895c9e1595924e769a25faeed7e34644 [term 3 FOLLOWER]: Advancing to term 4
I20260129 02:57:58.433975 19485 raft_consensus.cc:515] T 2b5edf7ea566411caea265dcfff5a86a P 895c9e1595924e769a25faeed7e34644 [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "309a53f12a534385a7e5a88fb373722a" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 34213 } } peers { permanent_uuid: "a6aef613bf534ff9ae6ddbba06bb3cf1" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 45187 } } peers { permanent_uuid: "895c9e1595924e769a25faeed7e34644" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 40083 } }
I20260129 02:57:58.435367 19485 leader_election.cc:290] T 2b5edf7ea566411caea265dcfff5a86a P 895c9e1595924e769a25faeed7e34644 [CANDIDATE]: Term 4 election: Requested vote from peers 309a53f12a534385a7e5a88fb373722a (127.18.211.3:34213), a6aef613bf534ff9ae6ddbba06bb3cf1 (127.18.211.1:45187)
I20260129 02:57:58.436146 19560 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2b5edf7ea566411caea265dcfff5a86a" candidate_uuid: "895c9e1595924e769a25faeed7e34644" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: true dest_uuid: "309a53f12a534385a7e5a88fb373722a"
I20260129 02:57:58.436419 19411 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2b5edf7ea566411caea265dcfff5a86a" candidate_uuid: "895c9e1595924e769a25faeed7e34644" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: true dest_uuid: "a6aef613bf534ff9ae6ddbba06bb3cf1"
I20260129 02:57:58.436663 19560 raft_consensus.cc:3055] T 2b5edf7ea566411caea265dcfff5a86a P 309a53f12a534385a7e5a88fb373722a [term 3 LEADER]: Stepping down as leader of term 3
I20260129 02:57:58.436988 19411 raft_consensus.cc:3060] T 2b5edf7ea566411caea265dcfff5a86a P a6aef613bf534ff9ae6ddbba06bb3cf1 [term 3 FOLLOWER]: Advancing to term 4
I20260129 02:57:58.437004 19560 raft_consensus.cc:740] T 2b5edf7ea566411caea265dcfff5a86a P 309a53f12a534385a7e5a88fb373722a [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 309a53f12a534385a7e5a88fb373722a, State: Running, Role: LEADER
I20260129 02:57:58.437736 19560 consensus_queue.cc:260] T 2b5edf7ea566411caea265dcfff5a86a P 309a53f12a534385a7e5a88fb373722a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "309a53f12a534385a7e5a88fb373722a" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 34213 } } peers { permanent_uuid: "a6aef613bf534ff9ae6ddbba06bb3cf1" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 45187 } } peers { permanent_uuid: "895c9e1595924e769a25faeed7e34644" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 40083 } }
I20260129 02:57:58.438530 19560 raft_consensus.cc:3060] T 2b5edf7ea566411caea265dcfff5a86a P 309a53f12a534385a7e5a88fb373722a [term 3 FOLLOWER]: Advancing to term 4
I20260129 02:57:58.441468 19411 raft_consensus.cc:2468] T 2b5edf7ea566411caea265dcfff5a86a P a6aef613bf534ff9ae6ddbba06bb3cf1 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 895c9e1595924e769a25faeed7e34644 in term 4.
I20260129 02:57:58.442353 19450 leader_election.cc:304] T 2b5edf7ea566411caea265dcfff5a86a P 895c9e1595924e769a25faeed7e34644 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 895c9e1595924e769a25faeed7e34644, a6aef613bf534ff9ae6ddbba06bb3cf1; no voters:
I20260129 02:57:58.442469 19560 raft_consensus.cc:2468] T 2b5edf7ea566411caea265dcfff5a86a P 309a53f12a534385a7e5a88fb373722a [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 895c9e1595924e769a25faeed7e34644 in term 4.
I20260129 02:57:58.443078 19723 raft_consensus.cc:2804] T 2b5edf7ea566411caea265dcfff5a86a P 895c9e1595924e769a25faeed7e34644 [term 4 FOLLOWER]: Leader election won for term 4
I20260129 02:57:58.443562 19723 raft_consensus.cc:697] T 2b5edf7ea566411caea265dcfff5a86a P 895c9e1595924e769a25faeed7e34644 [term 4 LEADER]: Becoming Leader. State: Replica: 895c9e1595924e769a25faeed7e34644, State: Running, Role: LEADER
I20260129 02:57:58.444535 19723 consensus_queue.cc:237] T 2b5edf7ea566411caea265dcfff5a86a P 895c9e1595924e769a25faeed7e34644 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "309a53f12a534385a7e5a88fb373722a" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 34213 } } peers { permanent_uuid: "a6aef613bf534ff9ae6ddbba06bb3cf1" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 45187 } } peers { permanent_uuid: "895c9e1595924e769a25faeed7e34644" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 40083 } }
I20260129 02:57:58.450522 19306 catalog_manager.cc:5654] T 2b5edf7ea566411caea265dcfff5a86a P 895c9e1595924e769a25faeed7e34644 reported cstate change: term changed from 3 to 4, leader changed from 309a53f12a534385a7e5a88fb373722a (127.18.211.3) to 895c9e1595924e769a25faeed7e34644 (127.18.211.2). New cstate: current_term: 4 leader_uuid: "895c9e1595924e769a25faeed7e34644" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "309a53f12a534385a7e5a88fb373722a" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 34213 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a6aef613bf534ff9ae6ddbba06bb3cf1" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 45187 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "895c9e1595924e769a25faeed7e34644" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 40083 } health_report { overall_health: HEALTHY } } }
I20260129 02:57:58.463079 19726 raft_consensus.cc:993] T 2e068f6d10f04c9ea17dba1375ae9b3c P a6aef613bf534ff9ae6ddbba06bb3cf1: : Instructing follower 895c9e1595924e769a25faeed7e34644 to start an election
I20260129 02:57:58.463469 19721 raft_consensus.cc:1081] T 2e068f6d10f04c9ea17dba1375ae9b3c P a6aef613bf534ff9ae6ddbba06bb3cf1 [term 2 LEADER]: Signalling peer 895c9e1595924e769a25faeed7e34644 to start an election
I20260129 02:57:58.464794 19485 tablet_service.cc:2038] Received Run Leader Election RPC: tablet_id: "2e068f6d10f04c9ea17dba1375ae9b3c"
dest_uuid: "895c9e1595924e769a25faeed7e34644"
from {username='slave'} at 127.0.0.1:46902
I20260129 02:57:58.465242 19485 raft_consensus.cc:493] T 2e068f6d10f04c9ea17dba1375ae9b3c P 895c9e1595924e769a25faeed7e34644 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20260129 02:57:58.465459 19485 raft_consensus.cc:3060] T 2e068f6d10f04c9ea17dba1375ae9b3c P 895c9e1595924e769a25faeed7e34644 [term 2 FOLLOWER]: Advancing to term 3
I20260129 02:57:58.469316 19485 raft_consensus.cc:515] T 2e068f6d10f04c9ea17dba1375ae9b3c P 895c9e1595924e769a25faeed7e34644 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6aef613bf534ff9ae6ddbba06bb3cf1" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 45187 } } peers { permanent_uuid: "895c9e1595924e769a25faeed7e34644" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 40083 } } peers { permanent_uuid: "309a53f12a534385a7e5a88fb373722a" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 34213 } }
I20260129 02:57:58.470541 19485 leader_election.cc:290] T 2e068f6d10f04c9ea17dba1375ae9b3c P 895c9e1595924e769a25faeed7e34644 [CANDIDATE]: Term 3 election: Requested vote from peers a6aef613bf534ff9ae6ddbba06bb3cf1 (127.18.211.1:45187), 309a53f12a534385a7e5a88fb373722a (127.18.211.3:34213)
I20260129 02:57:58.471334 19411 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2e068f6d10f04c9ea17dba1375ae9b3c" candidate_uuid: "895c9e1595924e769a25faeed7e34644" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "a6aef613bf534ff9ae6ddbba06bb3cf1"
I20260129 02:57:58.471556 19560 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2e068f6d10f04c9ea17dba1375ae9b3c" candidate_uuid: "895c9e1595924e769a25faeed7e34644" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "309a53f12a534385a7e5a88fb373722a"
I20260129 02:57:58.471930 19411 raft_consensus.cc:3055] T 2e068f6d10f04c9ea17dba1375ae9b3c P a6aef613bf534ff9ae6ddbba06bb3cf1 [term 2 LEADER]: Stepping down as leader of term 2
I20260129 02:57:58.472034 19560 raft_consensus.cc:3060] T 2e068f6d10f04c9ea17dba1375ae9b3c P 309a53f12a534385a7e5a88fb373722a [term 2 FOLLOWER]: Advancing to term 3
I20260129 02:57:58.472282 19411 raft_consensus.cc:740] T 2e068f6d10f04c9ea17dba1375ae9b3c P a6aef613bf534ff9ae6ddbba06bb3cf1 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: a6aef613bf534ff9ae6ddbba06bb3cf1, State: Running, Role: LEADER
I20260129 02:57:58.473021 19411 consensus_queue.cc:260] T 2e068f6d10f04c9ea17dba1375ae9b3c P a6aef613bf534ff9ae6ddbba06bb3cf1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6aef613bf534ff9ae6ddbba06bb3cf1" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 45187 } } peers { permanent_uuid: "895c9e1595924e769a25faeed7e34644" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 40083 } } peers { permanent_uuid: "309a53f12a534385a7e5a88fb373722a" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 34213 } }
I20260129 02:57:58.474186 19411 raft_consensus.cc:3060] T 2e068f6d10f04c9ea17dba1375ae9b3c P a6aef613bf534ff9ae6ddbba06bb3cf1 [term 2 FOLLOWER]: Advancing to term 3
I20260129 02:57:58.478169 19560 raft_consensus.cc:2468] T 2e068f6d10f04c9ea17dba1375ae9b3c P 309a53f12a534385a7e5a88fb373722a [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 895c9e1595924e769a25faeed7e34644 in term 3.
I20260129 02:57:58.479142 19450 leader_election.cc:304] T 2e068f6d10f04c9ea17dba1375ae9b3c P 895c9e1595924e769a25faeed7e34644 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 309a53f12a534385a7e5a88fb373722a, 895c9e1595924e769a25faeed7e34644; no voters:
I20260129 02:57:58.479763 19723 raft_consensus.cc:2804] T 2e068f6d10f04c9ea17dba1375ae9b3c P 895c9e1595924e769a25faeed7e34644 [term 3 FOLLOWER]: Leader election won for term 3
I20260129 02:57:58.480173 19411 raft_consensus.cc:2468] T 2e068f6d10f04c9ea17dba1375ae9b3c P a6aef613bf534ff9ae6ddbba06bb3cf1 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 895c9e1595924e769a25faeed7e34644 in term 3.
I20260129 02:57:58.480216 19723 raft_consensus.cc:697] T 2e068f6d10f04c9ea17dba1375ae9b3c P 895c9e1595924e769a25faeed7e34644 [term 3 LEADER]: Becoming Leader. State: Replica: 895c9e1595924e769a25faeed7e34644, State: Running, Role: LEADER
I20260129 02:57:58.481254 19723 consensus_queue.cc:237] T 2e068f6d10f04c9ea17dba1375ae9b3c P 895c9e1595924e769a25faeed7e34644 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6aef613bf534ff9ae6ddbba06bb3cf1" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 45187 } } peers { permanent_uuid: "895c9e1595924e769a25faeed7e34644" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 40083 } } peers { permanent_uuid: "309a53f12a534385a7e5a88fb373722a" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 34213 } }
I20260129 02:57:58.488124 19306 catalog_manager.cc:5654] T 2e068f6d10f04c9ea17dba1375ae9b3c P 895c9e1595924e769a25faeed7e34644 reported cstate change: term changed from 2 to 3, leader changed from a6aef613bf534ff9ae6ddbba06bb3cf1 (127.18.211.1) to 895c9e1595924e769a25faeed7e34644 (127.18.211.2). New cstate: current_term: 3 leader_uuid: "895c9e1595924e769a25faeed7e34644" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6aef613bf534ff9ae6ddbba06bb3cf1" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 45187 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "895c9e1595924e769a25faeed7e34644" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 40083 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "309a53f12a534385a7e5a88fb373722a" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 34213 } health_report { overall_health: UNKNOWN } } }
I20260129 02:57:58.799412 19411 raft_consensus.cc:1275] T a78e86f5781f41bba56333c01607277a P a6aef613bf534ff9ae6ddbba06bb3cf1 [term 2 FOLLOWER]: Refusing update from remote peer 895c9e1595924e769a25faeed7e34644: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260129 02:57:58.800483 19723 consensus_queue.cc:1048] T a78e86f5781f41bba56333c01607277a P 895c9e1595924e769a25faeed7e34644 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a6aef613bf534ff9ae6ddbba06bb3cf1" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 45187 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260129 02:57:58.803988 19560 raft_consensus.cc:1275] T e5ce9ed62ba242cabb3d6e5371679ba7 P 309a53f12a534385a7e5a88fb373722a [term 2 FOLLOWER]: Refusing update from remote peer 895c9e1595924e769a25faeed7e34644: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260129 02:57:58.805214 19723 consensus_queue.cc:1048] T e5ce9ed62ba242cabb3d6e5371679ba7 P 895c9e1595924e769a25faeed7e34644 [LEADER]: Connected to new peer: Peer: permanent_uuid: "309a53f12a534385a7e5a88fb373722a" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 34213 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260129 02:57:58.817222 19559 raft_consensus.cc:1275] T a78e86f5781f41bba56333c01607277a P 309a53f12a534385a7e5a88fb373722a [term 2 FOLLOWER]: Refusing update from remote peer 895c9e1595924e769a25faeed7e34644: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260129 02:57:58.818686 19560 raft_consensus.cc:1275] T 6431d8f509f84fa6a63a3aad36a6d39d P 309a53f12a534385a7e5a88fb373722a [term 4 FOLLOWER]: Refusing update from remote peer 895c9e1595924e769a25faeed7e34644: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20260129 02:57:58.823947 19744 consensus_queue.cc:1048] T a78e86f5781f41bba56333c01607277a P 895c9e1595924e769a25faeed7e34644 [LEADER]: Connected to new peer: Peer: permanent_uuid: "309a53f12a534385a7e5a88fb373722a" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 34213 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260129 02:57:58.825330 19411 raft_consensus.cc:1275] T e5ce9ed62ba242cabb3d6e5371679ba7 P a6aef613bf534ff9ae6ddbba06bb3cf1 [term 2 FOLLOWER]: Refusing update from remote peer 895c9e1595924e769a25faeed7e34644: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260129 02:57:58.825217 19744 consensus_queue.cc:1048] T 6431d8f509f84fa6a63a3aad36a6d39d P 895c9e1595924e769a25faeed7e34644 [LEADER]: Connected to new peer: Peer: permanent_uuid: "309a53f12a534385a7e5a88fb373722a" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 34213 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260129 02:57:58.825641 19558 raft_consensus.cc:1275] T 2b5edf7ea566411caea265dcfff5a86a P 309a53f12a534385a7e5a88fb373722a [term 4 FOLLOWER]: Refusing update from remote peer 895c9e1595924e769a25faeed7e34644: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20260129 02:57:58.827761 19727 consensus_queue.cc:1048] T e5ce9ed62ba242cabb3d6e5371679ba7 P 895c9e1595924e769a25faeed7e34644 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a6aef613bf534ff9ae6ddbba06bb3cf1" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 45187 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260129 02:57:58.827970 19723 consensus_queue.cc:1048] T 2b5edf7ea566411caea265dcfff5a86a P 895c9e1595924e769a25faeed7e34644 [LEADER]: Connected to new peer: Peer: permanent_uuid: "309a53f12a534385a7e5a88fb373722a" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 34213 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260129 02:57:58.843372 19411 raft_consensus.cc:1275] T 2b5edf7ea566411caea265dcfff5a86a P a6aef613bf534ff9ae6ddbba06bb3cf1 [term 4 FOLLOWER]: Refusing update from remote peer 895c9e1595924e769a25faeed7e34644: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20260129 02:57:58.844642 19723 consensus_queue.cc:1048] T 2b5edf7ea566411caea265dcfff5a86a P 895c9e1595924e769a25faeed7e34644 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a6aef613bf534ff9ae6ddbba06bb3cf1" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 45187 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260129 02:57:58.857623 19410 raft_consensus.cc:1275] T 6431d8f509f84fa6a63a3aad36a6d39d P a6aef613bf534ff9ae6ddbba06bb3cf1 [term 4 FOLLOWER]: Refusing update from remote peer 895c9e1595924e769a25faeed7e34644: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20260129 02:57:58.859372 19742 consensus_queue.cc:1048] T 6431d8f509f84fa6a63a3aad36a6d39d P 895c9e1595924e769a25faeed7e34644 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a6aef613bf534ff9ae6ddbba06bb3cf1" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 45187 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260129 02:57:58.934758 19560 raft_consensus.cc:1275] T 2e068f6d10f04c9ea17dba1375ae9b3c P 309a53f12a534385a7e5a88fb373722a [term 3 FOLLOWER]: Refusing update from remote peer 895c9e1595924e769a25faeed7e34644: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260129 02:57:58.935820 19723 consensus_queue.cc:1048] T 2e068f6d10f04c9ea17dba1375ae9b3c P 895c9e1595924e769a25faeed7e34644 [LEADER]: Connected to new peer: Peer: permanent_uuid: "309a53f12a534385a7e5a88fb373722a" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 34213 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260129 02:57:58.943528 19411 raft_consensus.cc:1275] T 2e068f6d10f04c9ea17dba1375ae9b3c P a6aef613bf534ff9ae6ddbba06bb3cf1 [term 3 FOLLOWER]: Refusing update from remote peer 895c9e1595924e769a25faeed7e34644: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260129 02:57:58.944872 19727 consensus_queue.cc:1048] T 2e068f6d10f04c9ea17dba1375ae9b3c P 895c9e1595924e769a25faeed7e34644 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a6aef613bf534ff9ae6ddbba06bb3cf1" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 45187 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260129 02:58:01.210036 19276 auto_leader_rebalancer-test.cc:306] The leader distribution is
I20260129 02:58:01.210793 19276 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260129 02:58:01.219342 19485 tablet_service.cc:2066] Received LeaderStepDown RPC: tablet_id: "2e068f6d10f04c9ea17dba1375ae9b3c"
dest_uuid: "895c9e1595924e769a25faeed7e34644"
mode: GRACEFUL
new_leader_uuid: "a6aef613bf534ff9ae6ddbba06bb3cf1"
from {username='slave'} at 127.0.0.1:46922
I20260129 02:58:01.220111 19485 raft_consensus.cc:606] T 2e068f6d10f04c9ea17dba1375ae9b3c P 895c9e1595924e769a25faeed7e34644 [term 3 LEADER]: Received request to transfer leadership to TS a6aef613bf534ff9ae6ddbba06bb3cf1
I20260129 02:58:01.222337 19485 tablet_service.cc:2066] Received LeaderStepDown RPC: tablet_id: "6431d8f509f84fa6a63a3aad36a6d39d"
dest_uuid: "895c9e1595924e769a25faeed7e34644"
mode: GRACEFUL
new_leader_uuid: "a6aef613bf534ff9ae6ddbba06bb3cf1"
from {username='slave'} at 127.0.0.1:46922
I20260129 02:58:01.222774 19485 raft_consensus.cc:606] T 6431d8f509f84fa6a63a3aad36a6d39d P 895c9e1595924e769a25faeed7e34644 [term 4 LEADER]: Received request to transfer leadership to TS a6aef613bf534ff9ae6ddbba06bb3cf1
I20260129 02:58:01.224668 19485 tablet_service.cc:2066] Received LeaderStepDown RPC: tablet_id: "9d5a1dc0b90642309290802698ec59f2"
dest_uuid: "895c9e1595924e769a25faeed7e34644"
mode: GRACEFUL
new_leader_uuid: "a6aef613bf534ff9ae6ddbba06bb3cf1"
from {username='slave'} at 127.0.0.1:46922
I20260129 02:58:01.225100 19485 raft_consensus.cc:606] T 9d5a1dc0b90642309290802698ec59f2 P 895c9e1595924e769a25faeed7e34644 [term 2 LEADER]: Received request to transfer leadership to TS a6aef613bf534ff9ae6ddbba06bb3cf1
I20260129 02:58:01.226912 19485 tablet_service.cc:2066] Received LeaderStepDown RPC: tablet_id: "a78e86f5781f41bba56333c01607277a"
dest_uuid: "895c9e1595924e769a25faeed7e34644"
mode: GRACEFUL
new_leader_uuid: "309a53f12a534385a7e5a88fb373722a"
from {username='slave'} at 127.0.0.1:46922
I20260129 02:58:01.227305 19485 raft_consensus.cc:606] T a78e86f5781f41bba56333c01607277a P 895c9e1595924e769a25faeed7e34644 [term 2 LEADER]: Received request to transfer leadership to TS 309a53f12a534385a7e5a88fb373722a
I20260129 02:58:01.229142 19485 tablet_service.cc:2066] Received LeaderStepDown RPC: tablet_id: "e5ce9ed62ba242cabb3d6e5371679ba7"
dest_uuid: "895c9e1595924e769a25faeed7e34644"
mode: GRACEFUL
new_leader_uuid: "309a53f12a534385a7e5a88fb373722a"
from {username='slave'} at 127.0.0.1:46922
I20260129 02:58:01.229552 19485 raft_consensus.cc:606] T e5ce9ed62ba242cabb3d6e5371679ba7 P 895c9e1595924e769a25faeed7e34644 [term 2 LEADER]: Received request to transfer leadership to TS 309a53f12a534385a7e5a88fb373722a
I20260129 02:58:01.230579 19276 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 5
I20260129 02:58:01.231118 19276 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20260129 02:58:01.408749 19754 raft_consensus.cc:993] T e5ce9ed62ba242cabb3d6e5371679ba7 P 895c9e1595924e769a25faeed7e34644: : Instructing follower 309a53f12a534385a7e5a88fb373722a to start an election
I20260129 02:58:01.409324 19756 raft_consensus.cc:1081] T e5ce9ed62ba242cabb3d6e5371679ba7 P 895c9e1595924e769a25faeed7e34644 [term 2 LEADER]: Signalling peer 309a53f12a534385a7e5a88fb373722a to start an election
I20260129 02:58:01.410775 19558 tablet_service.cc:2038] Received Run Leader Election RPC: tablet_id: "e5ce9ed62ba242cabb3d6e5371679ba7"
dest_uuid: "309a53f12a534385a7e5a88fb373722a"
from {username='slave'} at 127.0.0.1:38378
I20260129 02:58:01.411275 19558 raft_consensus.cc:493] T e5ce9ed62ba242cabb3d6e5371679ba7 P 309a53f12a534385a7e5a88fb373722a [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20260129 02:58:01.411789 19558 raft_consensus.cc:3060] T e5ce9ed62ba242cabb3d6e5371679ba7 P 309a53f12a534385a7e5a88fb373722a [term 2 FOLLOWER]: Advancing to term 3
I20260129 02:58:01.418700 19558 raft_consensus.cc:515] T e5ce9ed62ba242cabb3d6e5371679ba7 P 309a53f12a534385a7e5a88fb373722a [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6aef613bf534ff9ae6ddbba06bb3cf1" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 45187 } } peers { permanent_uuid: "895c9e1595924e769a25faeed7e34644" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 40083 } } peers { permanent_uuid: "309a53f12a534385a7e5a88fb373722a" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 34213 } }
I20260129 02:58:01.420684 19558 leader_election.cc:290] T e5ce9ed62ba242cabb3d6e5371679ba7 P 309a53f12a534385a7e5a88fb373722a [CANDIDATE]: Term 3 election: Requested vote from peers a6aef613bf534ff9ae6ddbba06bb3cf1 (127.18.211.1:45187), 895c9e1595924e769a25faeed7e34644 (127.18.211.2:40083)
I20260129 02:58:01.421766 19485 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e5ce9ed62ba242cabb3d6e5371679ba7" candidate_uuid: "309a53f12a534385a7e5a88fb373722a" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "895c9e1595924e769a25faeed7e34644"
I20260129 02:58:01.421712 19409 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e5ce9ed62ba242cabb3d6e5371679ba7" candidate_uuid: "309a53f12a534385a7e5a88fb373722a" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "a6aef613bf534ff9ae6ddbba06bb3cf1"
I20260129 02:58:01.422525 19485 raft_consensus.cc:3055] T e5ce9ed62ba242cabb3d6e5371679ba7 P 895c9e1595924e769a25faeed7e34644 [term 2 LEADER]: Stepping down as leader of term 2
I20260129 02:58:01.422631 19409 raft_consensus.cc:3060] T e5ce9ed62ba242cabb3d6e5371679ba7 P a6aef613bf534ff9ae6ddbba06bb3cf1 [term 2 FOLLOWER]: Advancing to term 3
I20260129 02:58:01.422796 19485 raft_consensus.cc:740] T e5ce9ed62ba242cabb3d6e5371679ba7 P 895c9e1595924e769a25faeed7e34644 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 895c9e1595924e769a25faeed7e34644, State: Running, Role: LEADER
I20260129 02:58:01.423620 19485 consensus_queue.cc:260] T e5ce9ed62ba242cabb3d6e5371679ba7 P 895c9e1595924e769a25faeed7e34644 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6aef613bf534ff9ae6ddbba06bb3cf1" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 45187 } } peers { permanent_uuid: "895c9e1595924e769a25faeed7e34644" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 40083 } } peers { permanent_uuid: "309a53f12a534385a7e5a88fb373722a" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 34213 } }
I20260129 02:58:01.424642 19485 raft_consensus.cc:3060] T e5ce9ed62ba242cabb3d6e5371679ba7 P 895c9e1595924e769a25faeed7e34644 [term 2 FOLLOWER]: Advancing to term 3
I20260129 02:58:01.427989 19409 raft_consensus.cc:2468] T e5ce9ed62ba242cabb3d6e5371679ba7 P a6aef613bf534ff9ae6ddbba06bb3cf1 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 309a53f12a534385a7e5a88fb373722a in term 3.
I20260129 02:58:01.429216 19524 leader_election.cc:304] T e5ce9ed62ba242cabb3d6e5371679ba7 P 309a53f12a534385a7e5a88fb373722a [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 309a53f12a534385a7e5a88fb373722a, a6aef613bf534ff9ae6ddbba06bb3cf1; no voters:
I20260129 02:58:01.430150 19734 raft_consensus.cc:2804] T e5ce9ed62ba242cabb3d6e5371679ba7 P 309a53f12a534385a7e5a88fb373722a [term 3 FOLLOWER]: Leader election won for term 3
I20260129 02:58:01.430066 19485 raft_consensus.cc:2468] T e5ce9ed62ba242cabb3d6e5371679ba7 P 895c9e1595924e769a25faeed7e34644 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 309a53f12a534385a7e5a88fb373722a in term 3.
I20260129 02:58:01.431375 19734 raft_consensus.cc:697] T e5ce9ed62ba242cabb3d6e5371679ba7 P 309a53f12a534385a7e5a88fb373722a [term 3 LEADER]: Becoming Leader. State: Replica: 309a53f12a534385a7e5a88fb373722a, State: Running, Role: LEADER
I20260129 02:58:01.432281 19734 consensus_queue.cc:237] T e5ce9ed62ba242cabb3d6e5371679ba7 P 309a53f12a534385a7e5a88fb373722a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6aef613bf534ff9ae6ddbba06bb3cf1" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 45187 } } peers { permanent_uuid: "895c9e1595924e769a25faeed7e34644" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 40083 } } peers { permanent_uuid: "309a53f12a534385a7e5a88fb373722a" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 34213 } }
I20260129 02:58:01.441975 19307 catalog_manager.cc:5654] T e5ce9ed62ba242cabb3d6e5371679ba7 P 309a53f12a534385a7e5a88fb373722a reported cstate change: term changed from 2 to 3, leader changed from 895c9e1595924e769a25faeed7e34644 (127.18.211.2) to 309a53f12a534385a7e5a88fb373722a (127.18.211.3). New cstate: current_term: 3 leader_uuid: "309a53f12a534385a7e5a88fb373722a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6aef613bf534ff9ae6ddbba06bb3cf1" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 45187 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "895c9e1595924e769a25faeed7e34644" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 40083 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "309a53f12a534385a7e5a88fb373722a" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 34213 } health_report { overall_health: HEALTHY } } }
I20260129 02:58:01.523097 19754 raft_consensus.cc:993] T a78e86f5781f41bba56333c01607277a P 895c9e1595924e769a25faeed7e34644: : Instructing follower 309a53f12a534385a7e5a88fb373722a to start an election
I20260129 02:58:01.523536 19756 raft_consensus.cc:1081] T a78e86f5781f41bba56333c01607277a P 895c9e1595924e769a25faeed7e34644 [term 2 LEADER]: Signalling peer 309a53f12a534385a7e5a88fb373722a to start an election
I20260129 02:58:01.524848 19558 tablet_service.cc:2038] Received Run Leader Election RPC: tablet_id: "a78e86f5781f41bba56333c01607277a"
dest_uuid: "309a53f12a534385a7e5a88fb373722a"
from {username='slave'} at 127.0.0.1:38378
I20260129 02:58:01.525309 19558 raft_consensus.cc:493] T a78e86f5781f41bba56333c01607277a P 309a53f12a534385a7e5a88fb373722a [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20260129 02:58:01.525601 19558 raft_consensus.cc:3060] T a78e86f5781f41bba56333c01607277a P 309a53f12a534385a7e5a88fb373722a [term 2 FOLLOWER]: Advancing to term 3
I20260129 02:58:01.530020 19558 raft_consensus.cc:515] T a78e86f5781f41bba56333c01607277a P 309a53f12a534385a7e5a88fb373722a [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6aef613bf534ff9ae6ddbba06bb3cf1" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 45187 } } peers { permanent_uuid: "895c9e1595924e769a25faeed7e34644" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 40083 } } peers { permanent_uuid: "309a53f12a534385a7e5a88fb373722a" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 34213 } }
I20260129 02:58:01.531482 19558 leader_election.cc:290] T a78e86f5781f41bba56333c01607277a P 309a53f12a534385a7e5a88fb373722a [CANDIDATE]: Term 3 election: Requested vote from peers a6aef613bf534ff9ae6ddbba06bb3cf1 (127.18.211.1:45187), 895c9e1595924e769a25faeed7e34644 (127.18.211.2:40083)
I20260129 02:58:01.532310 19409 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a78e86f5781f41bba56333c01607277a" candidate_uuid: "309a53f12a534385a7e5a88fb373722a" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "a6aef613bf534ff9ae6ddbba06bb3cf1"
I20260129 02:58:01.532550 19485 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a78e86f5781f41bba56333c01607277a" candidate_uuid: "309a53f12a534385a7e5a88fb373722a" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "895c9e1595924e769a25faeed7e34644"
I20260129 02:58:01.533010 19409 raft_consensus.cc:3060] T a78e86f5781f41bba56333c01607277a P a6aef613bf534ff9ae6ddbba06bb3cf1 [term 2 FOLLOWER]: Advancing to term 3
I20260129 02:58:01.533041 19485 raft_consensus.cc:3055] T a78e86f5781f41bba56333c01607277a P 895c9e1595924e769a25faeed7e34644 [term 2 LEADER]: Stepping down as leader of term 2
I20260129 02:58:01.533533 19485 raft_consensus.cc:740] T a78e86f5781f41bba56333c01607277a P 895c9e1595924e769a25faeed7e34644 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 895c9e1595924e769a25faeed7e34644, State: Running, Role: LEADER
I20260129 02:58:01.534268 19485 consensus_queue.cc:260] T a78e86f5781f41bba56333c01607277a P 895c9e1595924e769a25faeed7e34644 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6aef613bf534ff9ae6ddbba06bb3cf1" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 45187 } } peers { permanent_uuid: "895c9e1595924e769a25faeed7e34644" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 40083 } } peers { permanent_uuid: "309a53f12a534385a7e5a88fb373722a" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 34213 } }
I20260129 02:58:01.535509 19485 raft_consensus.cc:3060] T a78e86f5781f41bba56333c01607277a P 895c9e1595924e769a25faeed7e34644 [term 2 FOLLOWER]: Advancing to term 3
I20260129 02:58:01.539671 19409 raft_consensus.cc:2468] T a78e86f5781f41bba56333c01607277a P a6aef613bf534ff9ae6ddbba06bb3cf1 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 309a53f12a534385a7e5a88fb373722a in term 3.
I20260129 02:58:01.540746 19524 leader_election.cc:304] T a78e86f5781f41bba56333c01607277a P 309a53f12a534385a7e5a88fb373722a [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 309a53f12a534385a7e5a88fb373722a, a6aef613bf534ff9ae6ddbba06bb3cf1; no voters:
I20260129 02:58:01.541481 19734 raft_consensus.cc:2804] T a78e86f5781f41bba56333c01607277a P 309a53f12a534385a7e5a88fb373722a [term 3 FOLLOWER]: Leader election won for term 3
I20260129 02:58:01.541749 19485 raft_consensus.cc:2468] T a78e86f5781f41bba56333c01607277a P 895c9e1595924e769a25faeed7e34644 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 309a53f12a534385a7e5a88fb373722a in term 3.
I20260129 02:58:01.541903 19734 raft_consensus.cc:697] T a78e86f5781f41bba56333c01607277a P 309a53f12a534385a7e5a88fb373722a [term 3 LEADER]: Becoming Leader. State: Replica: 309a53f12a534385a7e5a88fb373722a, State: Running, Role: LEADER
I20260129 02:58:01.543195 19734 consensus_queue.cc:237] T a78e86f5781f41bba56333c01607277a P 309a53f12a534385a7e5a88fb373722a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6aef613bf534ff9ae6ddbba06bb3cf1" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 45187 } } peers { permanent_uuid: "895c9e1595924e769a25faeed7e34644" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 40083 } } peers { permanent_uuid: "309a53f12a534385a7e5a88fb373722a" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 34213 } }
I20260129 02:58:01.549768 19307 catalog_manager.cc:5654] T a78e86f5781f41bba56333c01607277a P 309a53f12a534385a7e5a88fb373722a reported cstate change: term changed from 2 to 3, leader changed from 895c9e1595924e769a25faeed7e34644 (127.18.211.2) to 309a53f12a534385a7e5a88fb373722a (127.18.211.3). New cstate: current_term: 3 leader_uuid: "309a53f12a534385a7e5a88fb373722a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6aef613bf534ff9ae6ddbba06bb3cf1" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 45187 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "895c9e1595924e769a25faeed7e34644" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 40083 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "309a53f12a534385a7e5a88fb373722a" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 34213 } health_report { overall_health: HEALTHY } } }
I20260129 02:58:01.560283 19754 raft_consensus.cc:993] T 9d5a1dc0b90642309290802698ec59f2 P 895c9e1595924e769a25faeed7e34644: : Instructing follower a6aef613bf534ff9ae6ddbba06bb3cf1 to start an election
I20260129 02:58:01.560704 19756 raft_consensus.cc:1081] T 9d5a1dc0b90642309290802698ec59f2 P 895c9e1595924e769a25faeed7e34644 [term 2 LEADER]: Signalling peer a6aef613bf534ff9ae6ddbba06bb3cf1 to start an election
I20260129 02:58:01.561868 19409 tablet_service.cc:2038] Received Run Leader Election RPC: tablet_id: "9d5a1dc0b90642309290802698ec59f2"
dest_uuid: "a6aef613bf534ff9ae6ddbba06bb3cf1"
from {username='slave'} at 127.0.0.1:40698
I20260129 02:58:01.562239 19409 raft_consensus.cc:493] T 9d5a1dc0b90642309290802698ec59f2 P a6aef613bf534ff9ae6ddbba06bb3cf1 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20260129 02:58:01.562460 19409 raft_consensus.cc:3060] T 9d5a1dc0b90642309290802698ec59f2 P a6aef613bf534ff9ae6ddbba06bb3cf1 [term 2 FOLLOWER]: Advancing to term 3
I20260129 02:58:01.566816 19409 raft_consensus.cc:515] T 9d5a1dc0b90642309290802698ec59f2 P a6aef613bf534ff9ae6ddbba06bb3cf1 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6aef613bf534ff9ae6ddbba06bb3cf1" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 45187 } } peers { permanent_uuid: "895c9e1595924e769a25faeed7e34644" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 40083 } } peers { permanent_uuid: "309a53f12a534385a7e5a88fb373722a" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 34213 } }
I20260129 02:58:01.568847 19409 leader_election.cc:290] T 9d5a1dc0b90642309290802698ec59f2 P a6aef613bf534ff9ae6ddbba06bb3cf1 [CANDIDATE]: Term 3 election: Requested vote from peers 895c9e1595924e769a25faeed7e34644 (127.18.211.2:40083), 309a53f12a534385a7e5a88fb373722a (127.18.211.3:34213)
I20260129 02:58:01.569592 19485 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9d5a1dc0b90642309290802698ec59f2" candidate_uuid: "a6aef613bf534ff9ae6ddbba06bb3cf1" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "895c9e1595924e769a25faeed7e34644"
I20260129 02:58:01.569823 19558 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9d5a1dc0b90642309290802698ec59f2" candidate_uuid: "a6aef613bf534ff9ae6ddbba06bb3cf1" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "309a53f12a534385a7e5a88fb373722a"
I20260129 02:58:01.570178 19485 raft_consensus.cc:3055] T 9d5a1dc0b90642309290802698ec59f2 P 895c9e1595924e769a25faeed7e34644 [term 2 LEADER]: Stepping down as leader of term 2
I20260129 02:58:01.570326 19558 raft_consensus.cc:3060] T 9d5a1dc0b90642309290802698ec59f2 P 309a53f12a534385a7e5a88fb373722a [term 2 FOLLOWER]: Advancing to term 3
I20260129 02:58:01.570513 19485 raft_consensus.cc:740] T 9d5a1dc0b90642309290802698ec59f2 P 895c9e1595924e769a25faeed7e34644 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 895c9e1595924e769a25faeed7e34644, State: Running, Role: LEADER
I20260129 02:58:01.571295 19485 consensus_queue.cc:260] T 9d5a1dc0b90642309290802698ec59f2 P 895c9e1595924e769a25faeed7e34644 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6aef613bf534ff9ae6ddbba06bb3cf1" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 45187 } } peers { permanent_uuid: "895c9e1595924e769a25faeed7e34644" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 40083 } } peers { permanent_uuid: "309a53f12a534385a7e5a88fb373722a" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 34213 } }
I20260129 02:58:01.572417 19485 raft_consensus.cc:3060] T 9d5a1dc0b90642309290802698ec59f2 P 895c9e1595924e769a25faeed7e34644 [term 2 FOLLOWER]: Advancing to term 3
I20260129 02:58:01.575013 19756 raft_consensus.cc:993] T 2e068f6d10f04c9ea17dba1375ae9b3c P 895c9e1595924e769a25faeed7e34644: : Instructing follower a6aef613bf534ff9ae6ddbba06bb3cf1 to start an election
I20260129 02:58:01.575429 19756 raft_consensus.cc:1081] T 2e068f6d10f04c9ea17dba1375ae9b3c P 895c9e1595924e769a25faeed7e34644 [term 3 LEADER]: Signalling peer a6aef613bf534ff9ae6ddbba06bb3cf1 to start an election
I20260129 02:58:01.577029 19558 raft_consensus.cc:2468] T 9d5a1dc0b90642309290802698ec59f2 P 309a53f12a534385a7e5a88fb373722a [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a6aef613bf534ff9ae6ddbba06bb3cf1 in term 3.
I20260129 02:58:01.577159 19410 tablet_service.cc:2038] Received Run Leader Election RPC: tablet_id: "2e068f6d10f04c9ea17dba1375ae9b3c"
dest_uuid: "a6aef613bf534ff9ae6ddbba06bb3cf1"
from {username='slave'} at 127.0.0.1:40698
I20260129 02:58:01.577721 19410 raft_consensus.cc:493] T 2e068f6d10f04c9ea17dba1375ae9b3c P a6aef613bf534ff9ae6ddbba06bb3cf1 [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20260129 02:58:01.578063 19410 raft_consensus.cc:3060] T 2e068f6d10f04c9ea17dba1375ae9b3c P a6aef613bf534ff9ae6ddbba06bb3cf1 [term 3 FOLLOWER]: Advancing to term 4
I20260129 02:58:01.578619 19375 leader_election.cc:304] T 9d5a1dc0b90642309290802698ec59f2 P a6aef613bf534ff9ae6ddbba06bb3cf1 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 309a53f12a534385a7e5a88fb373722a, a6aef613bf534ff9ae6ddbba06bb3cf1; no voters:
I20260129 02:58:01.579012 19756 raft_consensus.cc:993] T 6431d8f509f84fa6a63a3aad36a6d39d P 895c9e1595924e769a25faeed7e34644: : Instructing follower a6aef613bf534ff9ae6ddbba06bb3cf1 to start an election
I20260129 02:58:01.579015 19485 raft_consensus.cc:2468] T 9d5a1dc0b90642309290802698ec59f2 P 895c9e1595924e769a25faeed7e34644 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a6aef613bf534ff9ae6ddbba06bb3cf1 in term 3.
I20260129 02:58:01.579705 19753 raft_consensus.cc:1081] T 6431d8f509f84fa6a63a3aad36a6d39d P 895c9e1595924e769a25faeed7e34644 [term 4 LEADER]: Signalling peer a6aef613bf534ff9ae6ddbba06bb3cf1 to start an election
I20260129 02:58:01.580756 19764 raft_consensus.cc:2804] T 9d5a1dc0b90642309290802698ec59f2 P a6aef613bf534ff9ae6ddbba06bb3cf1 [term 3 FOLLOWER]: Leader election won for term 3
I20260129 02:58:01.581461 19409 tablet_service.cc:2038] Received Run Leader Election RPC: tablet_id: "6431d8f509f84fa6a63a3aad36a6d39d"
dest_uuid: "a6aef613bf534ff9ae6ddbba06bb3cf1"
from {username='slave'} at 127.0.0.1:40698
I20260129 02:58:01.582170 19409 raft_consensus.cc:493] T 6431d8f509f84fa6a63a3aad36a6d39d P a6aef613bf534ff9ae6ddbba06bb3cf1 [term 4 FOLLOWER]: Starting forced leader election (received explicit request)
I20260129 02:58:01.582134 19764 raft_consensus.cc:697] T 9d5a1dc0b90642309290802698ec59f2 P a6aef613bf534ff9ae6ddbba06bb3cf1 [term 3 LEADER]: Becoming Leader. State: Replica: a6aef613bf534ff9ae6ddbba06bb3cf1, State: Running, Role: LEADER
I20260129 02:58:01.582662 19409 raft_consensus.cc:3060] T 6431d8f509f84fa6a63a3aad36a6d39d P a6aef613bf534ff9ae6ddbba06bb3cf1 [term 4 FOLLOWER]: Advancing to term 5
I20260129 02:58:01.583271 19764 consensus_queue.cc:237] T 9d5a1dc0b90642309290802698ec59f2 P a6aef613bf534ff9ae6ddbba06bb3cf1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6aef613bf534ff9ae6ddbba06bb3cf1" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 45187 } } peers { permanent_uuid: "895c9e1595924e769a25faeed7e34644" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 40083 } } peers { permanent_uuid: "309a53f12a534385a7e5a88fb373722a" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 34213 } }
I20260129 02:58:01.583963 19410 raft_consensus.cc:515] T 2e068f6d10f04c9ea17dba1375ae9b3c P a6aef613bf534ff9ae6ddbba06bb3cf1 [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6aef613bf534ff9ae6ddbba06bb3cf1" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 45187 } } peers { permanent_uuid: "895c9e1595924e769a25faeed7e34644" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 40083 } } peers { permanent_uuid: "309a53f12a534385a7e5a88fb373722a" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 34213 } }
I20260129 02:58:01.585469 19410 leader_election.cc:290] T 2e068f6d10f04c9ea17dba1375ae9b3c P a6aef613bf534ff9ae6ddbba06bb3cf1 [CANDIDATE]: Term 4 election: Requested vote from peers 895c9e1595924e769a25faeed7e34644 (127.18.211.2:40083), 309a53f12a534385a7e5a88fb373722a (127.18.211.3:34213)
I20260129 02:58:01.586372 19558 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2e068f6d10f04c9ea17dba1375ae9b3c" candidate_uuid: "a6aef613bf534ff9ae6ddbba06bb3cf1" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: true dest_uuid: "309a53f12a534385a7e5a88fb373722a"
I20260129 02:58:01.586953 19558 raft_consensus.cc:3060] T 2e068f6d10f04c9ea17dba1375ae9b3c P 309a53f12a534385a7e5a88fb373722a [term 3 FOLLOWER]: Advancing to term 4
I20260129 02:58:01.586412 19485 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2e068f6d10f04c9ea17dba1375ae9b3c" candidate_uuid: "a6aef613bf534ff9ae6ddbba06bb3cf1" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: true dest_uuid: "895c9e1595924e769a25faeed7e34644"
I20260129 02:58:01.587513 19485 raft_consensus.cc:3055] T 2e068f6d10f04c9ea17dba1375ae9b3c P 895c9e1595924e769a25faeed7e34644 [term 3 LEADER]: Stepping down as leader of term 3
I20260129 02:58:01.587795 19485 raft_consensus.cc:740] T 2e068f6d10f04c9ea17dba1375ae9b3c P 895c9e1595924e769a25faeed7e34644 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 895c9e1595924e769a25faeed7e34644, State: Running, Role: LEADER
I20260129 02:58:01.588517 19485 consensus_queue.cc:260] T 2e068f6d10f04c9ea17dba1375ae9b3c P 895c9e1595924e769a25faeed7e34644 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6aef613bf534ff9ae6ddbba06bb3cf1" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 45187 } } peers { permanent_uuid: "895c9e1595924e769a25faeed7e34644" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 40083 } } peers { permanent_uuid: "309a53f12a534385a7e5a88fb373722a" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 34213 } }
I20260129 02:58:01.589625 19485 raft_consensus.cc:3060] T 2e068f6d10f04c9ea17dba1375ae9b3c P 895c9e1595924e769a25faeed7e34644 [term 3 FOLLOWER]: Advancing to term 4
I20260129 02:58:01.591068 19409 raft_consensus.cc:515] T 6431d8f509f84fa6a63a3aad36a6d39d P a6aef613bf534ff9ae6ddbba06bb3cf1 [term 5 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6aef613bf534ff9ae6ddbba06bb3cf1" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 45187 } } peers { permanent_uuid: "895c9e1595924e769a25faeed7e34644" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 40083 } } peers { permanent_uuid: "309a53f12a534385a7e5a88fb373722a" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 34213 } }
I20260129 02:58:01.593330 19558 raft_consensus.cc:2468] T 2e068f6d10f04c9ea17dba1375ae9b3c P 309a53f12a534385a7e5a88fb373722a [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a6aef613bf534ff9ae6ddbba06bb3cf1 in term 4.
I20260129 02:58:01.593689 19409 leader_election.cc:290] T 6431d8f509f84fa6a63a3aad36a6d39d P a6aef613bf534ff9ae6ddbba06bb3cf1 [CANDIDATE]: Term 5 election: Requested vote from peers 895c9e1595924e769a25faeed7e34644 (127.18.211.2:40083), 309a53f12a534385a7e5a88fb373722a (127.18.211.3:34213)
I20260129 02:58:01.594712 19484 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6431d8f509f84fa6a63a3aad36a6d39d" candidate_uuid: "a6aef613bf534ff9ae6ddbba06bb3cf1" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: true dest_uuid: "895c9e1595924e769a25faeed7e34644"
I20260129 02:58:01.595278 19484 raft_consensus.cc:3055] T 6431d8f509f84fa6a63a3aad36a6d39d P 895c9e1595924e769a25faeed7e34644 [term 4 LEADER]: Stepping down as leader of term 4
I20260129 02:58:01.595451 19375 leader_election.cc:304] T 2e068f6d10f04c9ea17dba1375ae9b3c P a6aef613bf534ff9ae6ddbba06bb3cf1 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 309a53f12a534385a7e5a88fb373722a, a6aef613bf534ff9ae6ddbba06bb3cf1; no voters:
I20260129 02:58:01.596460 19764 raft_consensus.cc:2804] T 2e068f6d10f04c9ea17dba1375ae9b3c P a6aef613bf534ff9ae6ddbba06bb3cf1 [term 4 FOLLOWER]: Leader election won for term 4
I20260129 02:58:01.597023 19764 raft_consensus.cc:697] T 2e068f6d10f04c9ea17dba1375ae9b3c P a6aef613bf534ff9ae6ddbba06bb3cf1 [term 4 LEADER]: Becoming Leader. State: Replica: a6aef613bf534ff9ae6ddbba06bb3cf1, State: Running, Role: LEADER
I20260129 02:58:01.597972 19485 raft_consensus.cc:2468] T 2e068f6d10f04c9ea17dba1375ae9b3c P 895c9e1595924e769a25faeed7e34644 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a6aef613bf534ff9ae6ddbba06bb3cf1 in term 4.
I20260129 02:58:01.597910 19764 consensus_queue.cc:237] T 2e068f6d10f04c9ea17dba1375ae9b3c P a6aef613bf534ff9ae6ddbba06bb3cf1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6aef613bf534ff9ae6ddbba06bb3cf1" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 45187 } } peers { permanent_uuid: "895c9e1595924e769a25faeed7e34644" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 40083 } } peers { permanent_uuid: "309a53f12a534385a7e5a88fb373722a" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 34213 } }
I20260129 02:58:01.598907 19558 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6431d8f509f84fa6a63a3aad36a6d39d" candidate_uuid: "a6aef613bf534ff9ae6ddbba06bb3cf1" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: true dest_uuid: "309a53f12a534385a7e5a88fb373722a"
I20260129 02:58:01.595657 19484 raft_consensus.cc:740] T 6431d8f509f84fa6a63a3aad36a6d39d P 895c9e1595924e769a25faeed7e34644 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 895c9e1595924e769a25faeed7e34644, State: Running, Role: LEADER
I20260129 02:58:01.599488 19558 raft_consensus.cc:3060] T 6431d8f509f84fa6a63a3aad36a6d39d P 309a53f12a534385a7e5a88fb373722a [term 4 FOLLOWER]: Advancing to term 5
I20260129 02:58:01.600140 19484 consensus_queue.cc:260] T 6431d8f509f84fa6a63a3aad36a6d39d P 895c9e1595924e769a25faeed7e34644 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 4.4, Last appended by leader: 4, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6aef613bf534ff9ae6ddbba06bb3cf1" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 45187 } } peers { permanent_uuid: "895c9e1595924e769a25faeed7e34644" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 40083 } } peers { permanent_uuid: "309a53f12a534385a7e5a88fb373722a" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 34213 } }
I20260129 02:58:01.601501 19484 raft_consensus.cc:3060] T 6431d8f509f84fa6a63a3aad36a6d39d P 895c9e1595924e769a25faeed7e34644 [term 4 FOLLOWER]: Advancing to term 5
I20260129 02:58:01.601390 19307 catalog_manager.cc:5654] T 9d5a1dc0b90642309290802698ec59f2 P a6aef613bf534ff9ae6ddbba06bb3cf1 reported cstate change: term changed from 2 to 3, leader changed from 895c9e1595924e769a25faeed7e34644 (127.18.211.2) to a6aef613bf534ff9ae6ddbba06bb3cf1 (127.18.211.1). New cstate: current_term: 3 leader_uuid: "a6aef613bf534ff9ae6ddbba06bb3cf1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6aef613bf534ff9ae6ddbba06bb3cf1" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 45187 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "895c9e1595924e769a25faeed7e34644" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 40083 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "309a53f12a534385a7e5a88fb373722a" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 34213 } health_report { overall_health: UNKNOWN } } }
I20260129 02:58:01.605742 19558 raft_consensus.cc:2468] T 6431d8f509f84fa6a63a3aad36a6d39d P 309a53f12a534385a7e5a88fb373722a [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a6aef613bf534ff9ae6ddbba06bb3cf1 in term 5.
I20260129 02:58:01.606983 19375 leader_election.cc:304] T 6431d8f509f84fa6a63a3aad36a6d39d P a6aef613bf534ff9ae6ddbba06bb3cf1 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 309a53f12a534385a7e5a88fb373722a, a6aef613bf534ff9ae6ddbba06bb3cf1; no voters:
I20260129 02:58:01.607944 19765 raft_consensus.cc:2804] T 6431d8f509f84fa6a63a3aad36a6d39d P a6aef613bf534ff9ae6ddbba06bb3cf1 [term 5 FOLLOWER]: Leader election won for term 5
I20260129 02:58:01.610469 19484 raft_consensus.cc:2468] T 6431d8f509f84fa6a63a3aad36a6d39d P 895c9e1595924e769a25faeed7e34644 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a6aef613bf534ff9ae6ddbba06bb3cf1 in term 5.
I20260129 02:58:01.610935 19765 raft_consensus.cc:697] T 6431d8f509f84fa6a63a3aad36a6d39d P a6aef613bf534ff9ae6ddbba06bb3cf1 [term 5 LEADER]: Becoming Leader. State: Replica: a6aef613bf534ff9ae6ddbba06bb3cf1, State: Running, Role: LEADER
I20260129 02:58:01.611799 19765 consensus_queue.cc:237] T 6431d8f509f84fa6a63a3aad36a6d39d P a6aef613bf534ff9ae6ddbba06bb3cf1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 4.4, Last appended by leader: 4, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6aef613bf534ff9ae6ddbba06bb3cf1" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 45187 } } peers { permanent_uuid: "895c9e1595924e769a25faeed7e34644" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 40083 } } peers { permanent_uuid: "309a53f12a534385a7e5a88fb373722a" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 34213 } }
I20260129 02:58:01.619256 19307 catalog_manager.cc:5654] T 2e068f6d10f04c9ea17dba1375ae9b3c P a6aef613bf534ff9ae6ddbba06bb3cf1 reported cstate change: term changed from 3 to 4, leader changed from 895c9e1595924e769a25faeed7e34644 (127.18.211.2) to a6aef613bf534ff9ae6ddbba06bb3cf1 (127.18.211.1). New cstate: current_term: 4 leader_uuid: "a6aef613bf534ff9ae6ddbba06bb3cf1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6aef613bf534ff9ae6ddbba06bb3cf1" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 45187 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "895c9e1595924e769a25faeed7e34644" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 40083 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "309a53f12a534385a7e5a88fb373722a" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 34213 } health_report { overall_health: UNKNOWN } } }
I20260129 02:58:01.620409 19307 catalog_manager.cc:5654] T 6431d8f509f84fa6a63a3aad36a6d39d P a6aef613bf534ff9ae6ddbba06bb3cf1 reported cstate change: term changed from 4 to 5, leader changed from 895c9e1595924e769a25faeed7e34644 (127.18.211.2) to a6aef613bf534ff9ae6ddbba06bb3cf1 (127.18.211.1). New cstate: current_term: 5 leader_uuid: "a6aef613bf534ff9ae6ddbba06bb3cf1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6aef613bf534ff9ae6ddbba06bb3cf1" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 45187 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "895c9e1595924e769a25faeed7e34644" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 40083 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "309a53f12a534385a7e5a88fb373722a" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 34213 } health_report { overall_health: UNKNOWN } } }
I20260129 02:58:01.826053 19484 raft_consensus.cc:1275] T e5ce9ed62ba242cabb3d6e5371679ba7 P 895c9e1595924e769a25faeed7e34644 [term 3 FOLLOWER]: Refusing update from remote peer 309a53f12a534385a7e5a88fb373722a: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260129 02:58:01.827199 19734 consensus_queue.cc:1048] T e5ce9ed62ba242cabb3d6e5371679ba7 P 309a53f12a534385a7e5a88fb373722a [LEADER]: Connected to new peer: Peer: permanent_uuid: "895c9e1595924e769a25faeed7e34644" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 40083 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260129 02:58:01.838295 19409 raft_consensus.cc:1275] T e5ce9ed62ba242cabb3d6e5371679ba7 P a6aef613bf534ff9ae6ddbba06bb3cf1 [term 3 FOLLOWER]: Refusing update from remote peer 309a53f12a534385a7e5a88fb373722a: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260129 02:58:01.839854 19734 consensus_queue.cc:1048] T e5ce9ed62ba242cabb3d6e5371679ba7 P 309a53f12a534385a7e5a88fb373722a [LEADER]: Connected to new peer: Peer: permanent_uuid: "a6aef613bf534ff9ae6ddbba06bb3cf1" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 45187 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260129 02:58:01.927928 19409 raft_consensus.cc:1275] T a78e86f5781f41bba56333c01607277a P a6aef613bf534ff9ae6ddbba06bb3cf1 [term 3 FOLLOWER]: Refusing update from remote peer 309a53f12a534385a7e5a88fb373722a: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260129 02:58:01.929181 19772 consensus_queue.cc:1048] T a78e86f5781f41bba56333c01607277a P 309a53f12a534385a7e5a88fb373722a [LEADER]: Connected to new peer: Peer: permanent_uuid: "a6aef613bf534ff9ae6ddbba06bb3cf1" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 45187 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260129 02:58:01.938349 19484 raft_consensus.cc:1275] T a78e86f5781f41bba56333c01607277a P 895c9e1595924e769a25faeed7e34644 [term 3 FOLLOWER]: Refusing update from remote peer 309a53f12a534385a7e5a88fb373722a: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260129 02:58:01.940313 19734 consensus_queue.cc:1048] T a78e86f5781f41bba56333c01607277a P 309a53f12a534385a7e5a88fb373722a [LEADER]: Connected to new peer: Peer: permanent_uuid: "895c9e1595924e769a25faeed7e34644" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 40083 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260129 02:58:02.029556 19558 raft_consensus.cc:1275] T 9d5a1dc0b90642309290802698ec59f2 P 309a53f12a534385a7e5a88fb373722a [term 3 FOLLOWER]: Refusing update from remote peer a6aef613bf534ff9ae6ddbba06bb3cf1: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260129 02:58:02.030622 19765 consensus_queue.cc:1048] T 9d5a1dc0b90642309290802698ec59f2 P a6aef613bf534ff9ae6ddbba06bb3cf1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "309a53f12a534385a7e5a88fb373722a" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 34213 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260129 02:58:02.039196 19484 raft_consensus.cc:1275] T 9d5a1dc0b90642309290802698ec59f2 P 895c9e1595924e769a25faeed7e34644 [term 3 FOLLOWER]: Refusing update from remote peer a6aef613bf534ff9ae6ddbba06bb3cf1: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260129 02:58:02.041054 19765 consensus_queue.cc:1048] T 9d5a1dc0b90642309290802698ec59f2 P a6aef613bf534ff9ae6ddbba06bb3cf1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "895c9e1595924e769a25faeed7e34644" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 40083 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
W20260129 02:58:02.089565 19581 debug-util.cc:398] Leaking SignalData structure 0x7b080016d060 after lost signal to thread 19279
W20260129 02:58:02.090309 19581 debug-util.cc:398] Leaking SignalData structure 0x7b080016d740 after lost signal to thread 19342
W20260129 02:58:02.091028 19581 debug-util.cc:398] Leaking SignalData structure 0x7b0800139560 after lost signal to thread 19435
W20260129 02:58:02.091733 19581 debug-util.cc:398] Leaking SignalData structure 0x7b08001dc340 after lost signal to thread 19510
W20260129 02:58:02.092530 19581 debug-util.cc:398] Leaking SignalData structure 0x7b0800109ac0 after lost signal to thread 19584
I20260129 02:58:02.140883 19484 raft_consensus.cc:1275] T 2e068f6d10f04c9ea17dba1375ae9b3c P 895c9e1595924e769a25faeed7e34644 [term 4 FOLLOWER]: Refusing update from remote peer a6aef613bf534ff9ae6ddbba06bb3cf1: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20260129 02:58:02.142268 19765 consensus_queue.cc:1048] T 2e068f6d10f04c9ea17dba1375ae9b3c P a6aef613bf534ff9ae6ddbba06bb3cf1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "895c9e1595924e769a25faeed7e34644" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 40083 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260129 02:58:02.142773 19558 raft_consensus.cc:1275] T 2e068f6d10f04c9ea17dba1375ae9b3c P 309a53f12a534385a7e5a88fb373722a [term 4 FOLLOWER]: Refusing update from remote peer a6aef613bf534ff9ae6ddbba06bb3cf1: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20260129 02:58:02.144173 19765 consensus_queue.cc:1048] T 2e068f6d10f04c9ea17dba1375ae9b3c P a6aef613bf534ff9ae6ddbba06bb3cf1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "309a53f12a534385a7e5a88fb373722a" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 34213 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260129 02:58:02.206192 19558 raft_consensus.cc:1275] T 6431d8f509f84fa6a63a3aad36a6d39d P 309a53f12a534385a7e5a88fb373722a [term 5 FOLLOWER]: Refusing update from remote peer a6aef613bf534ff9ae6ddbba06bb3cf1: Log matching property violated. Preceding OpId in replica: term: 4 index: 4. Preceding OpId from leader: term: 5 index: 5. (index mismatch)
I20260129 02:58:02.207365 19764 consensus_queue.cc:1048] T 6431d8f509f84fa6a63a3aad36a6d39d P a6aef613bf534ff9ae6ddbba06bb3cf1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "309a53f12a534385a7e5a88fb373722a" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 34213 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260129 02:58:02.217149 19484 raft_consensus.cc:1275] T 6431d8f509f84fa6a63a3aad36a6d39d P 895c9e1595924e769a25faeed7e34644 [term 5 FOLLOWER]: Refusing update from remote peer a6aef613bf534ff9ae6ddbba06bb3cf1: Log matching property violated. Preceding OpId in replica: term: 4 index: 4. Preceding OpId from leader: term: 5 index: 5. (index mismatch)
I20260129 02:58:02.219175 19768 consensus_queue.cc:1048] T 6431d8f509f84fa6a63a3aad36a6d39d P a6aef613bf534ff9ae6ddbba06bb3cf1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "895c9e1595924e769a25faeed7e34644" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 40083 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20260129 02:58:02.232206 19276 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260129 02:58:02.238314 19276 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260129 02:58:02.238765 19276 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20260129 02:58:03.240001 19276 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260129 02:58:03.246133 19276 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260129 02:58:03.246628 19276 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20260129 02:58:04.247805 19276 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260129 02:58:04.255067 19276 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260129 02:58:04.255548 19276 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20260129 02:58:05.256779 19276 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260129 02:58:05.263635 19276 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260129 02:58:05.264081 19276 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20260129 02:58:06.264987 19276 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260129 02:58:06.269985 19276 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260129 02:58:06.270447 19276 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20260129 02:58:07.271505 19276 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260129 02:58:07.276325 19276 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260129 02:58:07.276729 19276 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20260129 02:58:08.277667 19276 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260129 02:58:08.282347 19276 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260129 02:58:08.282804 19276 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20260129 02:58:09.283815 19276 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260129 02:58:09.288537 19276 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260129 02:58:09.288964 19276 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20260129 02:58:10.289882 19276 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260129 02:58:10.294867 19276 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260129 02:58:10.295351 19276 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20260129 02:58:11.323494 19276 auto_leader_rebalancer-test.cc:318] The leader distribution is
I20260129 02:58:11.324121 19276 tablet_server.cc:178] TabletServer@127.18.211.1:0 shutting down...
I20260129 02:58:11.350950 19276 ts_tablet_manager.cc:1507] Shutting down tablet manager...
I20260129 02:58:11.351787 19276 tablet_replica.cc:333] T 2e068f6d10f04c9ea17dba1375ae9b3c P a6aef613bf534ff9ae6ddbba06bb3cf1: stopping tablet replica
I20260129 02:58:11.352731 19276 raft_consensus.cc:2243] T 2e068f6d10f04c9ea17dba1375ae9b3c P a6aef613bf534ff9ae6ddbba06bb3cf1 [term 4 LEADER]: Raft consensus shutting down.
I20260129 02:58:11.353863 19276 raft_consensus.cc:2272] T 2e068f6d10f04c9ea17dba1375ae9b3c P a6aef613bf534ff9ae6ddbba06bb3cf1 [term 4 FOLLOWER]: Raft consensus is shut down!
I20260129 02:58:11.356205 19276 tablet_replica.cc:333] T a78e86f5781f41bba56333c01607277a P a6aef613bf534ff9ae6ddbba06bb3cf1: stopping tablet replica
I20260129 02:58:11.356735 19276 raft_consensus.cc:2243] T a78e86f5781f41bba56333c01607277a P a6aef613bf534ff9ae6ddbba06bb3cf1 [term 3 FOLLOWER]: Raft consensus shutting down.
I20260129 02:58:11.357275 19276 raft_consensus.cc:2272] T a78e86f5781f41bba56333c01607277a P a6aef613bf534ff9ae6ddbba06bb3cf1 [term 3 FOLLOWER]: Raft consensus is shut down!
I20260129 02:58:11.359105 19276 tablet_replica.cc:333] T 2b5edf7ea566411caea265dcfff5a86a P a6aef613bf534ff9ae6ddbba06bb3cf1: stopping tablet replica
I20260129 02:58:11.359678 19276 raft_consensus.cc:2243] T 2b5edf7ea566411caea265dcfff5a86a P a6aef613bf534ff9ae6ddbba06bb3cf1 [term 4 FOLLOWER]: Raft consensus shutting down.
I20260129 02:58:11.360203 19276 raft_consensus.cc:2272] T 2b5edf7ea566411caea265dcfff5a86a P a6aef613bf534ff9ae6ddbba06bb3cf1 [term 4 FOLLOWER]: Raft consensus is shut down!
I20260129 02:58:11.361784 19276 tablet_replica.cc:333] T 3e4edd649791453f902a5ac6d5658c12 P a6aef613bf534ff9ae6ddbba06bb3cf1: stopping tablet replica
I20260129 02:58:11.362176 19276 raft_consensus.cc:2243] T 3e4edd649791453f902a5ac6d5658c12 P a6aef613bf534ff9ae6ddbba06bb3cf1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260129 02:58:11.362522 19276 raft_consensus.cc:2272] T 3e4edd649791453f902a5ac6d5658c12 P a6aef613bf534ff9ae6ddbba06bb3cf1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260129 02:58:11.364106 19276 tablet_replica.cc:333] T 9d5a1dc0b90642309290802698ec59f2 P a6aef613bf534ff9ae6ddbba06bb3cf1: stopping tablet replica
I20260129 02:58:11.364531 19276 raft_consensus.cc:2243] T 9d5a1dc0b90642309290802698ec59f2 P a6aef613bf534ff9ae6ddbba06bb3cf1 [term 3 LEADER]: Raft consensus shutting down.
I20260129 02:58:11.365250 19276 raft_consensus.cc:2272] T 9d5a1dc0b90642309290802698ec59f2 P a6aef613bf534ff9ae6ddbba06bb3cf1 [term 3 FOLLOWER]: Raft consensus is shut down!
I20260129 02:58:11.366802 19276 tablet_replica.cc:333] T 6431d8f509f84fa6a63a3aad36a6d39d P a6aef613bf534ff9ae6ddbba06bb3cf1: stopping tablet replica
I20260129 02:58:11.367210 19276 raft_consensus.cc:2243] T 6431d8f509f84fa6a63a3aad36a6d39d P a6aef613bf534ff9ae6ddbba06bb3cf1 [term 5 LEADER]: Raft consensus shutting down.
I20260129 02:58:11.367995 19276 raft_consensus.cc:2272] T 6431d8f509f84fa6a63a3aad36a6d39d P a6aef613bf534ff9ae6ddbba06bb3cf1 [term 5 FOLLOWER]: Raft consensus is shut down!
I20260129 02:58:11.369632 19276 tablet_replica.cc:333] T f7d1ccc38ee8428aa16f7c82b86c0f3c P a6aef613bf534ff9ae6ddbba06bb3cf1: stopping tablet replica
I20260129 02:58:11.370049 19276 raft_consensus.cc:2243] T f7d1ccc38ee8428aa16f7c82b86c0f3c P a6aef613bf534ff9ae6ddbba06bb3cf1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260129 02:58:11.370404 19276 raft_consensus.cc:2272] T f7d1ccc38ee8428aa16f7c82b86c0f3c P a6aef613bf534ff9ae6ddbba06bb3cf1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260129 02:58:11.371963 19276 tablet_replica.cc:333] T c4689cdf800d479ba8645b96278c1713 P a6aef613bf534ff9ae6ddbba06bb3cf1: stopping tablet replica
I20260129 02:58:11.372421 19276 raft_consensus.cc:2243] T c4689cdf800d479ba8645b96278c1713 P a6aef613bf534ff9ae6ddbba06bb3cf1 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260129 02:58:11.372841 19276 raft_consensus.cc:2272] T c4689cdf800d479ba8645b96278c1713 P a6aef613bf534ff9ae6ddbba06bb3cf1 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260129 02:58:11.374303 19276 tablet_replica.cc:333] T e5ce9ed62ba242cabb3d6e5371679ba7 P a6aef613bf534ff9ae6ddbba06bb3cf1: stopping tablet replica
I20260129 02:58:11.374680 19276 raft_consensus.cc:2243] T e5ce9ed62ba242cabb3d6e5371679ba7 P a6aef613bf534ff9ae6ddbba06bb3cf1 [term 3 FOLLOWER]: Raft consensus shutting down.
I20260129 02:58:11.375069 19276 raft_consensus.cc:2272] T e5ce9ed62ba242cabb3d6e5371679ba7 P a6aef613bf534ff9ae6ddbba06bb3cf1 [term 3 FOLLOWER]: Raft consensus is shut down!
W20260129 02:58:11.379488 19450 proxy.cc:239] Call had error, refreshing address and retrying: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
W20260129 02:58:11.382581 19450 consensus_peers.cc:597] T f7d1ccc38ee8428aa16f7c82b86c0f3c P 895c9e1595924e769a25faeed7e34644 -> Peer a6aef613bf534ff9ae6ddbba06bb3cf1 (127.18.211.1:45187): Couldn't send request to peer a6aef613bf534ff9ae6ddbba06bb3cf1. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20260129 02:58:11.388340 19524 consensus_peers.cc:597] T e5ce9ed62ba242cabb3d6e5371679ba7 P 309a53f12a534385a7e5a88fb373722a -> Peer a6aef613bf534ff9ae6ddbba06bb3cf1 (127.18.211.1:45187): Couldn't send request to peer a6aef613bf534ff9ae6ddbba06bb3cf1. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260129 02:58:11.455468 19276 tablet_server.cc:195] TabletServer@127.18.211.1:0 shutdown complete.
I20260129 02:58:11.476442 19276 tablet_server.cc:178] TabletServer@127.18.211.2:0 shutting down...
I20260129 02:58:11.504592 19276 ts_tablet_manager.cc:1507] Shutting down tablet manager...
I20260129 02:58:11.505244 19276 tablet_replica.cc:333] T 6431d8f509f84fa6a63a3aad36a6d39d P 895c9e1595924e769a25faeed7e34644: stopping tablet replica
I20260129 02:58:11.505954 19276 raft_consensus.cc:2243] T 6431d8f509f84fa6a63a3aad36a6d39d P 895c9e1595924e769a25faeed7e34644 [term 5 FOLLOWER]: Raft consensus shutting down.
I20260129 02:58:11.506508 19276 raft_consensus.cc:2272] T 6431d8f509f84fa6a63a3aad36a6d39d P 895c9e1595924e769a25faeed7e34644 [term 5 FOLLOWER]: Raft consensus is shut down!
I20260129 02:58:11.508415 19276 tablet_replica.cc:333] T 3e4edd649791453f902a5ac6d5658c12 P 895c9e1595924e769a25faeed7e34644: stopping tablet replica
I20260129 02:58:11.508940 19276 raft_consensus.cc:2243] T 3e4edd649791453f902a5ac6d5658c12 P 895c9e1595924e769a25faeed7e34644 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260129 02:58:11.509351 19276 raft_consensus.cc:2272] T 3e4edd649791453f902a5ac6d5658c12 P 895c9e1595924e769a25faeed7e34644 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260129 02:58:11.510963 19276 tablet_replica.cc:333] T 9d5a1dc0b90642309290802698ec59f2 P 895c9e1595924e769a25faeed7e34644: stopping tablet replica
I20260129 02:58:11.511500 19276 raft_consensus.cc:2243] T 9d5a1dc0b90642309290802698ec59f2 P 895c9e1595924e769a25faeed7e34644 [term 3 FOLLOWER]: Raft consensus shutting down.
I20260129 02:58:11.512041 19276 raft_consensus.cc:2272] T 9d5a1dc0b90642309290802698ec59f2 P 895c9e1595924e769a25faeed7e34644 [term 3 FOLLOWER]: Raft consensus is shut down!
I20260129 02:58:11.513733 19276 tablet_replica.cc:333] T c4689cdf800d479ba8645b96278c1713 P 895c9e1595924e769a25faeed7e34644: stopping tablet replica
I20260129 02:58:11.514206 19276 raft_consensus.cc:2243] T c4689cdf800d479ba8645b96278c1713 P 895c9e1595924e769a25faeed7e34644 [term 2 LEADER]: Raft consensus shutting down.
I20260129 02:58:11.515071 19276 raft_consensus.cc:2272] T c4689cdf800d479ba8645b96278c1713 P 895c9e1595924e769a25faeed7e34644 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260129 02:58:11.516711 19276 tablet_replica.cc:333] T 2b5edf7ea566411caea265dcfff5a86a P 895c9e1595924e769a25faeed7e34644: stopping tablet replica
I20260129 02:58:11.517179 19276 raft_consensus.cc:2243] T 2b5edf7ea566411caea265dcfff5a86a P 895c9e1595924e769a25faeed7e34644 [term 4 LEADER]: Raft consensus shutting down.
I20260129 02:58:11.518004 19276 raft_consensus.cc:2272] T 2b5edf7ea566411caea265dcfff5a86a P 895c9e1595924e769a25faeed7e34644 [term 4 FOLLOWER]: Raft consensus is shut down!
I20260129 02:58:11.519578 19276 tablet_replica.cc:333] T a78e86f5781f41bba56333c01607277a P 895c9e1595924e769a25faeed7e34644: stopping tablet replica
I20260129 02:58:11.520115 19276 raft_consensus.cc:2243] T a78e86f5781f41bba56333c01607277a P 895c9e1595924e769a25faeed7e34644 [term 3 FOLLOWER]: Raft consensus shutting down.
I20260129 02:58:11.520550 19276 raft_consensus.cc:2272] T a78e86f5781f41bba56333c01607277a P 895c9e1595924e769a25faeed7e34644 [term 3 FOLLOWER]: Raft consensus is shut down!
I20260129 02:58:11.522132 19276 tablet_replica.cc:333] T 2e068f6d10f04c9ea17dba1375ae9b3c P 895c9e1595924e769a25faeed7e34644: stopping tablet replica
I20260129 02:58:11.522575 19276 raft_consensus.cc:2243] T 2e068f6d10f04c9ea17dba1375ae9b3c P 895c9e1595924e769a25faeed7e34644 [term 4 FOLLOWER]: Raft consensus shutting down.
I20260129 02:58:11.522958 19276 raft_consensus.cc:2272] T 2e068f6d10f04c9ea17dba1375ae9b3c P 895c9e1595924e769a25faeed7e34644 [term 4 FOLLOWER]: Raft consensus is shut down!
I20260129 02:58:11.524626 19276 tablet_replica.cc:333] T e5ce9ed62ba242cabb3d6e5371679ba7 P 895c9e1595924e769a25faeed7e34644: stopping tablet replica
I20260129 02:58:11.525049 19276 raft_consensus.cc:2243] T e5ce9ed62ba242cabb3d6e5371679ba7 P 895c9e1595924e769a25faeed7e34644 [term 3 FOLLOWER]: Raft consensus shutting down.
I20260129 02:58:11.525475 19276 raft_consensus.cc:2272] T e5ce9ed62ba242cabb3d6e5371679ba7 P 895c9e1595924e769a25faeed7e34644 [term 3 FOLLOWER]: Raft consensus is shut down!
I20260129 02:58:11.526909 19276 tablet_replica.cc:333] T f7d1ccc38ee8428aa16f7c82b86c0f3c P 895c9e1595924e769a25faeed7e34644: stopping tablet replica
I20260129 02:58:11.527305 19276 raft_consensus.cc:2243] T f7d1ccc38ee8428aa16f7c82b86c0f3c P 895c9e1595924e769a25faeed7e34644 [term 1 LEADER]: Raft consensus shutting down.
I20260129 02:58:11.528141 19276 raft_consensus.cc:2272] T f7d1ccc38ee8428aa16f7c82b86c0f3c P 895c9e1595924e769a25faeed7e34644 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260129 02:58:11.555555 19276 tablet_server.cc:195] TabletServer@127.18.211.2:0 shutdown complete.
I20260129 02:58:11.575469 19276 tablet_server.cc:178] TabletServer@127.18.211.3:0 shutting down...
W20260129 02:58:11.600535 19524 consensus_peers.cc:597] T a78e86f5781f41bba56333c01607277a P 309a53f12a534385a7e5a88fb373722a -> Peer a6aef613bf534ff9ae6ddbba06bb3cf1 (127.18.211.1:45187): Couldn't send request to peer a6aef613bf534ff9ae6ddbba06bb3cf1. Status: Network error: Client connection negotiation failed: client connection to 127.18.211.1:45187: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260129 02:58:11.603155 19276 ts_tablet_manager.cc:1507] Shutting down tablet manager...
I20260129 02:58:11.603996 19276 tablet_replica.cc:333] T f7d1ccc38ee8428aa16f7c82b86c0f3c P 309a53f12a534385a7e5a88fb373722a: stopping tablet replica
I20260129 02:58:11.604631 19276 raft_consensus.cc:2243] T f7d1ccc38ee8428aa16f7c82b86c0f3c P 309a53f12a534385a7e5a88fb373722a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260129 02:58:11.605141 19276 raft_consensus.cc:2272] T f7d1ccc38ee8428aa16f7c82b86c0f3c P 309a53f12a534385a7e5a88fb373722a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260129 02:58:11.606856 19276 tablet_replica.cc:333] T 6431d8f509f84fa6a63a3aad36a6d39d P 309a53f12a534385a7e5a88fb373722a: stopping tablet replica
I20260129 02:58:11.607404 19276 raft_consensus.cc:2243] T 6431d8f509f84fa6a63a3aad36a6d39d P 309a53f12a534385a7e5a88fb373722a [term 5 FOLLOWER]: Raft consensus shutting down.
I20260129 02:58:11.607844 19276 raft_consensus.cc:2272] T 6431d8f509f84fa6a63a3aad36a6d39d P 309a53f12a534385a7e5a88fb373722a [term 5 FOLLOWER]: Raft consensus is shut down!
I20260129 02:58:11.609460 19276 tablet_replica.cc:333] T 3e4edd649791453f902a5ac6d5658c12 P 309a53f12a534385a7e5a88fb373722a: stopping tablet replica
I20260129 02:58:11.609963 19276 raft_consensus.cc:2243] T 3e4edd649791453f902a5ac6d5658c12 P 309a53f12a534385a7e5a88fb373722a [term 1 LEADER]: Raft consensus shutting down.
I20260129 02:58:11.610678 19276 raft_consensus.cc:2272] T 3e4edd649791453f902a5ac6d5658c12 P 309a53f12a534385a7e5a88fb373722a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260129 02:58:11.612223 19276 tablet_replica.cc:333] T a78e86f5781f41bba56333c01607277a P 309a53f12a534385a7e5a88fb373722a: stopping tablet replica
I20260129 02:58:11.612713 19276 raft_consensus.cc:2243] T a78e86f5781f41bba56333c01607277a P 309a53f12a534385a7e5a88fb373722a [term 3 LEADER]: Raft consensus shutting down.
I20260129 02:58:11.613526 19276 raft_consensus.cc:2272] T a78e86f5781f41bba56333c01607277a P 309a53f12a534385a7e5a88fb373722a [term 3 FOLLOWER]: Raft consensus is shut down!
I20260129 02:58:11.615103 19276 tablet_replica.cc:333] T 2b5edf7ea566411caea265dcfff5a86a P 309a53f12a534385a7e5a88fb373722a: stopping tablet replica
I20260129 02:58:11.615571 19276 raft_consensus.cc:2243] T 2b5edf7ea566411caea265dcfff5a86a P 309a53f12a534385a7e5a88fb373722a [term 4 FOLLOWER]: Raft consensus shutting down.
I20260129 02:58:11.616089 19276 raft_consensus.cc:2272] T 2b5edf7ea566411caea265dcfff5a86a P 309a53f12a534385a7e5a88fb373722a [term 4 FOLLOWER]: Raft consensus is shut down!
I20260129 02:58:11.617733 19276 tablet_replica.cc:333] T 2e068f6d10f04c9ea17dba1375ae9b3c P 309a53f12a534385a7e5a88fb373722a: stopping tablet replica
I20260129 02:58:11.618252 19276 raft_consensus.cc:2243] T 2e068f6d10f04c9ea17dba1375ae9b3c P 309a53f12a534385a7e5a88fb373722a [term 4 FOLLOWER]: Raft consensus shutting down.
I20260129 02:58:11.618698 19276 raft_consensus.cc:2272] T 2e068f6d10f04c9ea17dba1375ae9b3c P 309a53f12a534385a7e5a88fb373722a [term 4 FOLLOWER]: Raft consensus is shut down!
I20260129 02:58:11.620316 19276 tablet_replica.cc:333] T c4689cdf800d479ba8645b96278c1713 P 309a53f12a534385a7e5a88fb373722a: stopping tablet replica
I20260129 02:58:11.620780 19276 raft_consensus.cc:2243] T c4689cdf800d479ba8645b96278c1713 P 309a53f12a534385a7e5a88fb373722a [term 2 FOLLOWER]: Raft consensus shutting down.
I20260129 02:58:11.621212 19276 raft_consensus.cc:2272] T c4689cdf800d479ba8645b96278c1713 P 309a53f12a534385a7e5a88fb373722a [term 2 FOLLOWER]: Raft consensus is shut down!
I20260129 02:58:11.622714 19276 tablet_replica.cc:333] T e5ce9ed62ba242cabb3d6e5371679ba7 P 309a53f12a534385a7e5a88fb373722a: stopping tablet replica
I20260129 02:58:11.623173 19276 raft_consensus.cc:2243] T e5ce9ed62ba242cabb3d6e5371679ba7 P 309a53f12a534385a7e5a88fb373722a [term 3 LEADER]: Raft consensus shutting down.
I20260129 02:58:11.624017 19276 raft_consensus.cc:2272] T e5ce9ed62ba242cabb3d6e5371679ba7 P 309a53f12a534385a7e5a88fb373722a [term 3 FOLLOWER]: Raft consensus is shut down!
I20260129 02:58:11.625504 19276 tablet_replica.cc:333] T 9d5a1dc0b90642309290802698ec59f2 P 309a53f12a534385a7e5a88fb373722a: stopping tablet replica
I20260129 02:58:11.625934 19276 raft_consensus.cc:2243] T 9d5a1dc0b90642309290802698ec59f2 P 309a53f12a534385a7e5a88fb373722a [term 3 FOLLOWER]: Raft consensus shutting down.
I20260129 02:58:11.626410 19276 raft_consensus.cc:2272] T 9d5a1dc0b90642309290802698ec59f2 P 309a53f12a534385a7e5a88fb373722a [term 3 FOLLOWER]: Raft consensus is shut down!
I20260129 02:58:11.673314 19276 tablet_server.cc:195] TabletServer@127.18.211.3:0 shutdown complete.
I20260129 02:58:11.691234 19276 master.cc:561] Master@127.18.211.62:39923 shutting down...
I20260129 02:58:11.823073 19276 raft_consensus.cc:2243] T 00000000000000000000000000000000 P ec7858395ca04e12bff51324a29a761f [term 1 LEADER]: Raft consensus shutting down.
I20260129 02:58:11.823736 19276 raft_consensus.cc:2272] T 00000000000000000000000000000000 P ec7858395ca04e12bff51324a29a761f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260129 02:58:11.824116 19276 tablet_replica.cc:333] T 00000000000000000000000000000000 P ec7858395ca04e12bff51324a29a761f: stopping tablet replica
I20260129 02:58:11.843066 19276 master.cc:583] Master@127.18.211.62:39923 shutdown complete.
309a53f12a534385a7e5a88fb373722a 1
895c9e1595924e769a25faeed7e34644 4
a6aef613bf534ff9ae6ddbba06bb3cf1 4
309a53f12a534385a7e5a88fb373722a 3
895c9e1595924e769a25faeed7e34644 3
a6aef613bf534ff9ae6ddbba06bb3cf1 3
309a53f12a534385a7e5a88fb373722a 1
895c9e1595924e769a25faeed7e34644 8
a6aef613bf534ff9ae6ddbba06bb3cf1 0
309a53f12a534385a7e5a88fb373722a 3
895c9e1595924e769a25faeed7e34644 3
a6aef613bf534ff9ae6ddbba06bb3cf1 3
[ OK ] LeaderRebalancerTest.FunctionalTestForDivided (30820 ms)
[ RUN ] LeaderRebalancerTest.FunctionalTestForNotDivided
I20260129 02:58:11.882380 19276 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.18.211.62:37849
I20260129 02:58:11.883463 19276 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260129 02:58:11.888926 19794 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260129 02:58:11.889009 19795 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260129 02:58:11.890337 19797 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260129 02:58:11.891222 19276 server_base.cc:1060] running on GCE node
I20260129 02:58:11.892251 19276 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260129 02:58:11.892477 19276 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260129 02:58:11.892653 19276 hybrid_clock.cc:648] HybridClock initialized: now 1769655491892630 us; error 0 us; skew 500 ppm
I20260129 02:58:11.893149 19276 server_base.cc:860] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260129 02:58:11.895421 19276 webserver.cc:492] Webserver started at http://127.18.211.62:44511/ using document root <none> and password file <none>
I20260129 02:58:11.895984 19276 fs_manager.cc:362] Metadata directory not provided
I20260129 02:58:11.896193 19276 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260129 02:58:11.896462 19276 server_base.cc:908] This appears to be a new deployment of Kudu; creating new FS layout
I20260129 02:58:11.897611 19276 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskrTP_Jo/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1769655460890924-19276-0/minicluster-data/master-0-root/instance:
uuid: "2dea75bcbf3c4f95b35d28ec2ed08a2c"
format_stamp: "Formatted at 2026-01-29 02:58:11 on dist-test-slave-vngr"
I20260129 02:58:11.902287 19276 fs_manager.cc:696] Time spent creating directory manager: real 0.004s user 0.004s sys 0.002s
I20260129 02:58:11.905784 19802 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260129 02:58:11.906433 19276 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.000s sys 0.001s
I20260129 02:58:11.906749 19276 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskrTP_Jo/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1769655460890924-19276-0/minicluster-data/master-0-root
uuid: "2dea75bcbf3c4f95b35d28ec2ed08a2c"
format_stamp: "Formatted at 2026-01-29 02:58:11 on dist-test-slave-vngr"
I20260129 02:58:11.907013 19276 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrTP_Jo/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1769655460890924-19276-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-taskrTP_Jo/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1769655460890924-19276-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-taskrTP_Jo/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1769655460890924-19276-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260129 02:58:11.925172 19276 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260129 02:58:11.926419 19276 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260129 02:58:11.963438 19276 rpc_server.cc:307] RPC server started. Bound to: 127.18.211.62:37849
I20260129 02:58:11.963532 19853 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.18.211.62:37849 every 8 connection(s)
I20260129 02:58:11.967299 19854 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260129 02:58:11.976831 19854 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2dea75bcbf3c4f95b35d28ec2ed08a2c: Bootstrap starting.
I20260129 02:58:11.981453 19854 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2dea75bcbf3c4f95b35d28ec2ed08a2c: Neither blocks nor log segments found. Creating new log.
I20260129 02:58:11.985575 19854 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2dea75bcbf3c4f95b35d28ec2ed08a2c: No bootstrap required, opened a new log
I20260129 02:58:11.987582 19854 raft_consensus.cc:359] T 00000000000000000000000000000000 P 2dea75bcbf3c4f95b35d28ec2ed08a2c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2dea75bcbf3c4f95b35d28ec2ed08a2c" member_type: VOTER }
I20260129 02:58:11.988052 19854 raft_consensus.cc:385] T 00000000000000000000000000000000 P 2dea75bcbf3c4f95b35d28ec2ed08a2c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260129 02:58:11.988301 19854 raft_consensus.cc:740] T 00000000000000000000000000000000 P 2dea75bcbf3c4f95b35d28ec2ed08a2c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2dea75bcbf3c4f95b35d28ec2ed08a2c, State: Initialized, Role: FOLLOWER
I20260129 02:58:11.988835 19854 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2dea75bcbf3c4f95b35d28ec2ed08a2c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2dea75bcbf3c4f95b35d28ec2ed08a2c" member_type: VOTER }
I20260129 02:58:11.989271 19854 raft_consensus.cc:399] T 00000000000000000000000000000000 P 2dea75bcbf3c4f95b35d28ec2ed08a2c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260129 02:58:11.989530 19854 raft_consensus.cc:493] T 00000000000000000000000000000000 P 2dea75bcbf3c4f95b35d28ec2ed08a2c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260129 02:58:11.989814 19854 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 2dea75bcbf3c4f95b35d28ec2ed08a2c [term 0 FOLLOWER]: Advancing to term 1
I20260129 02:58:11.993824 19854 raft_consensus.cc:515] T 00000000000000000000000000000000 P 2dea75bcbf3c4f95b35d28ec2ed08a2c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2dea75bcbf3c4f95b35d28ec2ed08a2c" member_type: VOTER }
I20260129 02:58:11.994338 19854 leader_election.cc:304] T 00000000000000000000000000000000 P 2dea75bcbf3c4f95b35d28ec2ed08a2c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2dea75bcbf3c4f95b35d28ec2ed08a2c; no voters:
I20260129 02:58:11.995409 19854 leader_election.cc:290] T 00000000000000000000000000000000 P 2dea75bcbf3c4f95b35d28ec2ed08a2c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260129 02:58:11.995803 19857 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 2dea75bcbf3c4f95b35d28ec2ed08a2c [term 1 FOLLOWER]: Leader election won for term 1
I20260129 02:58:11.997156 19857 raft_consensus.cc:697] T 00000000000000000000000000000000 P 2dea75bcbf3c4f95b35d28ec2ed08a2c [term 1 LEADER]: Becoming Leader. State: Replica: 2dea75bcbf3c4f95b35d28ec2ed08a2c, State: Running, Role: LEADER
I20260129 02:58:11.997833 19857 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2dea75bcbf3c4f95b35d28ec2ed08a2c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2dea75bcbf3c4f95b35d28ec2ed08a2c" member_type: VOTER }
I20260129 02:58:11.998466 19854 sys_catalog.cc:565] T 00000000000000000000000000000000 P 2dea75bcbf3c4f95b35d28ec2ed08a2c [sys.catalog]: configured and running, proceeding with master startup.
I20260129 02:58:12.003153 19859 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2dea75bcbf3c4f95b35d28ec2ed08a2c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2dea75bcbf3c4f95b35d28ec2ed08a2c. Latest consensus state: current_term: 1 leader_uuid: "2dea75bcbf3c4f95b35d28ec2ed08a2c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2dea75bcbf3c4f95b35d28ec2ed08a2c" member_type: VOTER } }
I20260129 02:58:12.003546 19858 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2dea75bcbf3c4f95b35d28ec2ed08a2c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2dea75bcbf3c4f95b35d28ec2ed08a2c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2dea75bcbf3c4f95b35d28ec2ed08a2c" member_type: VOTER } }
I20260129 02:58:12.003767 19859 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2dea75bcbf3c4f95b35d28ec2ed08a2c [sys.catalog]: This master's current role is: LEADER
I20260129 02:58:12.004086 19858 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2dea75bcbf3c4f95b35d28ec2ed08a2c [sys.catalog]: This master's current role is: LEADER
I20260129 02:58:12.005968 19866 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20260129 02:58:12.011020 19866 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20260129 02:58:12.013269 19276 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20260129 02:58:12.019992 19866 catalog_manager.cc:1357] Generated new cluster ID: e070e2a12bfc4ed9942cc85cf5f0a506
I20260129 02:58:12.020246 19866 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20260129 02:58:12.046438 19866 catalog_manager.cc:1380] Generated new certificate authority record
I20260129 02:58:12.047648 19866 catalog_manager.cc:1514] Loading token signing keys...
I20260129 02:58:12.058837 19866 catalog_manager.cc:6027] T 00000000000000000000000000000000 P 2dea75bcbf3c4f95b35d28ec2ed08a2c: Generated new TSK 0
I20260129 02:58:12.059409 19866 catalog_manager.cc:1524] Initializing in-progress tserver states...
I20260129 02:58:12.079492 19276 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260129 02:58:12.085675 19875 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260129 02:58:12.086297 19876 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260129 02:58:12.089275 19878 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260129 02:58:12.089527 19276 server_base.cc:1060] running on GCE node
I20260129 02:58:12.090430 19276 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260129 02:58:12.090644 19276 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260129 02:58:12.090770 19276 hybrid_clock.cc:648] HybridClock initialized: now 1769655492090759 us; error 0 us; skew 500 ppm
I20260129 02:58:12.091216 19276 server_base.cc:860] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260129 02:58:12.093328 19276 webserver.cc:492] Webserver started at http://127.18.211.1:33731/ using document root <none> and password file <none>
I20260129 02:58:12.093761 19276 fs_manager.cc:362] Metadata directory not provided
I20260129 02:58:12.093907 19276 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260129 02:58:12.094108 19276 server_base.cc:908] This appears to be a new deployment of Kudu; creating new FS layout
I20260129 02:58:12.095144 19276 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskrTP_Jo/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1769655460890924-19276-0/minicluster-data/ts-0-root/instance:
uuid: "eafca908701c410ab9488970fcb3d74e"
format_stamp: "Formatted at 2026-01-29 02:58:12 on dist-test-slave-vngr"
I20260129 02:58:12.099386 19276 fs_manager.cc:696] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260129 02:58:12.102478 19883 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260129 02:58:12.103192 19276 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260129 02:58:12.103541 19276 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskrTP_Jo/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1769655460890924-19276-0/minicluster-data/ts-0-root
uuid: "eafca908701c410ab9488970fcb3d74e"
format_stamp: "Formatted at 2026-01-29 02:58:12 on dist-test-slave-vngr"
I20260129 02:58:12.104007 19276 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrTP_Jo/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1769655460890924-19276-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-taskrTP_Jo/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1769655460890924-19276-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-taskrTP_Jo/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1769655460890924-19276-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260129 02:58:12.117121 19276 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260129 02:58:12.118265 19276 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260129 02:58:12.119580 19276 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260129 02:58:12.121757 19276 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260129 02:58:12.121933 19276 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260129 02:58:12.122108 19276 ts_tablet_manager.cc:616] Registered 0 tablets
I20260129 02:58:12.122226 19276 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260129 02:58:12.162639 19276 rpc_server.cc:307] RPC server started. Bound to: 127.18.211.1:45001
I20260129 02:58:12.162730 19945 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.18.211.1:45001 every 8 connection(s)
I20260129 02:58:12.167057 19276 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260129 02:58:12.174858 19950 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260129 02:58:12.176373 19951 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260129 02:58:12.178349 19276 server_base.cc:1060] running on GCE node
W20260129 02:58:12.178756 19953 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260129 02:58:12.180267 19276 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260129 02:58:12.180544 19276 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260129 02:58:12.180734 19276 hybrid_clock.cc:648] HybridClock initialized: now 1769655492180698 us; error 0 us; skew 500 ppm
I20260129 02:58:12.181432 19276 server_base.cc:860] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260129 02:58:12.182917 19946 heartbeater.cc:344] Connected to a master server at 127.18.211.62:37849
I20260129 02:58:12.183261 19946 heartbeater.cc:461] Registering TS with master...
I20260129 02:58:12.184058 19946 heartbeater.cc:507] Master 127.18.211.62:37849 requested a full tablet report, sending...
I20260129 02:58:12.184180 19276 webserver.cc:492] Webserver started at http://127.18.211.2:43291/ using document root <none> and password file <none>
I20260129 02:58:12.184769 19276 fs_manager.cc:362] Metadata directory not provided
I20260129 02:58:12.184974 19276 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260129 02:58:12.185250 19276 server_base.cc:908] This appears to be a new deployment of Kudu; creating new FS layout
I20260129 02:58:12.186335 19819 ts_manager.cc:194] Registered new tserver with Master: eafca908701c410ab9488970fcb3d74e (127.18.211.1:45001)
I20260129 02:58:12.186807 19276 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskrTP_Jo/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1769655460890924-19276-0/minicluster-data/ts-1-root/instance:
uuid: "544851f9b7954fd2962675f957e8836a"
format_stamp: "Formatted at 2026-01-29 02:58:12 on dist-test-slave-vngr"
I20260129 02:58:12.188791 19819 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:52748
I20260129 02:58:12.192085 19276 fs_manager.cc:696] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260129 02:58:12.195228 19958 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260129 02:58:12.196039 19276 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260129 02:58:12.196327 19276 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskrTP_Jo/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1769655460890924-19276-0/minicluster-data/ts-1-root
uuid: "544851f9b7954fd2962675f957e8836a"
format_stamp: "Formatted at 2026-01-29 02:58:12 on dist-test-slave-vngr"
I20260129 02:58:12.196599 19276 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrTP_Jo/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1769655460890924-19276-0/minicluster-data/ts-1-root
metadata directory: /tmp/dist-test-taskrTP_Jo/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1769655460890924-19276-0/minicluster-data/ts-1-root
1 data directories: /tmp/dist-test-taskrTP_Jo/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1769655460890924-19276-0/minicluster-data/ts-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260129 02:58:12.209841 19276 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260129 02:58:12.211018 19276 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260129 02:58:12.212536 19276 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260129 02:58:12.215273 19276 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260129 02:58:12.215487 19276 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260129 02:58:12.215749 19276 ts_tablet_manager.cc:616] Registered 0 tablets
I20260129 02:58:12.215961 19276 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.001s sys 0.000s
I20260129 02:58:12.254415 19276 rpc_server.cc:307] RPC server started. Bound to: 127.18.211.2:34199
I20260129 02:58:12.254513 20020 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.18.211.2:34199 every 8 connection(s)
I20260129 02:58:12.259013 19276 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260129 02:58:12.266149 20024 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260129 02:58:12.267616 20025 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260129 02:58:12.270150 20027 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260129 02:58:12.272066 19276 server_base.cc:1060] running on GCE node
I20260129 02:58:12.272228 20021 heartbeater.cc:344] Connected to a master server at 127.18.211.62:37849
I20260129 02:58:12.272653 20021 heartbeater.cc:461] Registering TS with master...
I20260129 02:58:12.273176 19276 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260129 02:58:12.273440 19276 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260129 02:58:12.273475 20021 heartbeater.cc:507] Master 127.18.211.62:37849 requested a full tablet report, sending...
I20260129 02:58:12.273670 19276 hybrid_clock.cc:648] HybridClock initialized: now 1769655492273652 us; error 0 us; skew 500 ppm
I20260129 02:58:12.274353 19276 server_base.cc:860] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260129 02:58:12.275774 19819 ts_manager.cc:194] Registered new tserver with Master: 544851f9b7954fd2962675f957e8836a (127.18.211.2:34199)
I20260129 02:58:12.276926 19276 webserver.cc:492] Webserver started at http://127.18.211.3:40517/ using document root <none> and password file <none>
I20260129 02:58:12.277313 19819 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:52762
I20260129 02:58:12.277423 19276 fs_manager.cc:362] Metadata directory not provided
I20260129 02:58:12.277711 19276 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260129 02:58:12.277988 19276 server_base.cc:908] This appears to be a new deployment of Kudu; creating new FS layout
I20260129 02:58:12.279170 19276 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskrTP_Jo/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1769655460890924-19276-0/minicluster-data/ts-2-root/instance:
uuid: "a634c60a5ca74b26b0d1f679d0166d0c"
format_stamp: "Formatted at 2026-01-29 02:58:12 on dist-test-slave-vngr"
I20260129 02:58:12.283449 19276 fs_manager.cc:696] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260129 02:58:12.286718 20032 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260129 02:58:12.287441 19276 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260129 02:58:12.287747 19276 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskrTP_Jo/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1769655460890924-19276-0/minicluster-data/ts-2-root
uuid: "a634c60a5ca74b26b0d1f679d0166d0c"
format_stamp: "Formatted at 2026-01-29 02:58:12 on dist-test-slave-vngr"
I20260129 02:58:12.288064 19276 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrTP_Jo/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1769655460890924-19276-0/minicluster-data/ts-2-root
metadata directory: /tmp/dist-test-taskrTP_Jo/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1769655460890924-19276-0/minicluster-data/ts-2-root
1 data directories: /tmp/dist-test-taskrTP_Jo/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1769655460890924-19276-0/minicluster-data/ts-2-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260129 02:58:12.304437 19276 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260129 02:58:12.305621 19276 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260129 02:58:12.307446 19276 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260129 02:58:12.309846 19276 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260129 02:58:12.310061 19276 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260129 02:58:12.310298 19276 ts_tablet_manager.cc:616] Registered 0 tablets
I20260129 02:58:12.310468 19276 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260129 02:58:12.348809 19276 rpc_server.cc:307] RPC server started. Bound to: 127.18.211.3:39941
I20260129 02:58:12.348898 20094 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.18.211.3:39941 every 8 connection(s)
I20260129 02:58:12.361835 20095 heartbeater.cc:344] Connected to a master server at 127.18.211.62:37849
I20260129 02:58:12.362174 20095 heartbeater.cc:461] Registering TS with master...
I20260129 02:58:12.362823 20095 heartbeater.cc:507] Master 127.18.211.62:37849 requested a full tablet report, sending...
I20260129 02:58:12.364611 19819 ts_manager.cc:194] Registered new tserver with Master: a634c60a5ca74b26b0d1f679d0166d0c (127.18.211.3:39941)
I20260129 02:58:12.365191 19276 internal_mini_cluster.cc:371] 3 TS(s) registered with all masters after 0.013222627s
I20260129 02:58:12.365967 19819 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:52774
I20260129 02:58:12.388573 19819 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:52780:
name: "test-workload"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "\004\001\000\314\314\314\014\004\001\000\230\231\231\031\004\001\000dff&\004\001\0000333\004\001\000\374\377\377?\004\001\000\310\314\314L\004\001\000\224\231\231Y\004\001\000`fff\004\001\000,33s""\004\001\000\314\314\314\014\004\001\000\230\231\231\031\004\001\000dff&\004\001\0000333\004\001\000\374\377\377?\004\001\000\310\314\314L\004\001\000\224\231\231Y\004\001\000`fff\004\001\000,33s"
indirect_data: """"
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260129 02:58:12.391423 19819 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-workload in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260129 02:58:12.458639 20060 tablet_service.cc:1505] Processing CreateTablet for tablet 7d91c3094df94b47aa9a512d13490daa (DEFAULT_TABLE table=test-workload [id=e92ceea777da4bd8982f0b57cf98dc37]), partition=RANGE (key) PARTITION VALUES < 214748364
I20260129 02:58:12.459210 20059 tablet_service.cc:1505] Processing CreateTablet for tablet 4562a703e5a44a1585b332b851bfd6f2 (DEFAULT_TABLE table=test-workload [id=e92ceea777da4bd8982f0b57cf98dc37]), partition=RANGE (key) PARTITION 214748364 <= VALUES < 429496728
I20260129 02:58:12.460150 20060 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7d91c3094df94b47aa9a512d13490daa. 1 dirs total, 0 dirs full, 0 dirs failed
I20260129 02:58:12.460021 20058 tablet_service.cc:1505] Processing CreateTablet for tablet d5edd0a419ee46c9a6a223f8056afb4c (DEFAULT_TABLE table=test-workload [id=e92ceea777da4bd8982f0b57cf98dc37]), partition=RANGE (key) PARTITION 429496728 <= VALUES < 644245092
I20260129 02:58:12.460821 20059 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4562a703e5a44a1585b332b851bfd6f2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260129 02:58:12.462355 20057 tablet_service.cc:1505] Processing CreateTablet for tablet 47ad34405a6f4d508c9613c07a572cc3 (DEFAULT_TABLE table=test-workload [id=e92ceea777da4bd8982f0b57cf98dc37]), partition=RANGE (key) PARTITION 644245092 <= VALUES < 858993456
I20260129 02:58:12.463670 20057 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 47ad34405a6f4d508c9613c07a572cc3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260129 02:58:12.466022 20056 tablet_service.cc:1505] Processing CreateTablet for tablet 79355f4f5d7942b6ac6f420256fd35b7 (DEFAULT_TABLE table=test-workload [id=e92ceea777da4bd8982f0b57cf98dc37]), partition=RANGE (key) PARTITION 858993456 <= VALUES < 1073741820
I20260129 02:58:12.467296 20056 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 79355f4f5d7942b6ac6f420256fd35b7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260129 02:58:12.468307 20058 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d5edd0a419ee46c9a6a223f8056afb4c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260129 02:58:12.470484 20055 tablet_service.cc:1505] Processing CreateTablet for tablet 620c6d80e1964c51a628ffb43f645798 (DEFAULT_TABLE table=test-workload [id=e92ceea777da4bd8982f0b57cf98dc37]), partition=RANGE (key) PARTITION 1073741820 <= VALUES < 1288490184
I20260129 02:58:12.471771 20055 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 620c6d80e1964c51a628ffb43f645798. 1 dirs total, 0 dirs full, 0 dirs failed
I20260129 02:58:12.477034 20054 tablet_service.cc:1505] Processing CreateTablet for tablet 901f619b02f843b4a2747de956182f12 (DEFAULT_TABLE table=test-workload [id=e92ceea777da4bd8982f0b57cf98dc37]), partition=RANGE (key) PARTITION 1288490184 <= VALUES < 1503238548
I20260129 02:58:12.478379 20054 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 901f619b02f843b4a2747de956182f12. 1 dirs total, 0 dirs full, 0 dirs failed
I20260129 02:58:12.494210 20053 tablet_service.cc:1505] Processing CreateTablet for tablet 1dc801ccecc44ed79d85b0522e49d3e5 (DEFAULT_TABLE table=test-workload [id=e92ceea777da4bd8982f0b57cf98dc37]), partition=RANGE (key) PARTITION 1503238548 <= VALUES < 1717986912
I20260129 02:58:12.495656 20053 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1dc801ccecc44ed79d85b0522e49d3e5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260129 02:58:12.502878 19905 tablet_service.cc:1505] Processing CreateTablet for tablet 901f619b02f843b4a2747de956182f12 (DEFAULT_TABLE table=test-workload [id=e92ceea777da4bd8982f0b57cf98dc37]), partition=RANGE (key) PARTITION 1288490184 <= VALUES < 1503238548
I20260129 02:58:12.504303 19905 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 901f619b02f843b4a2747de956182f12. 1 dirs total, 0 dirs full, 0 dirs failed
I20260129 02:58:12.505029 20052 tablet_service.cc:1505] Processing CreateTablet for tablet 1bd5a7f0c64f4ef6bc2bb544cd69843c (DEFAULT_TABLE table=test-workload [id=e92ceea777da4bd8982f0b57cf98dc37]), partition=RANGE (key) PARTITION 1932735276 <= VALUES
I20260129 02:58:12.496665 19911 tablet_service.cc:1505] Processing CreateTablet for tablet 7d91c3094df94b47aa9a512d13490daa (DEFAULT_TABLE table=test-workload [id=e92ceea777da4bd8982f0b57cf98dc37]), partition=RANGE (key) PARTITION VALUES < 214748364
I20260129 02:58:12.507059 19911 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7d91c3094df94b47aa9a512d13490daa. 1 dirs total, 0 dirs full, 0 dirs failed
I20260129 02:58:12.507303 20052 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1bd5a7f0c64f4ef6bc2bb544cd69843c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260129 02:58:12.497079 19910 tablet_service.cc:1505] Processing CreateTablet for tablet 4562a703e5a44a1585b332b851bfd6f2 (DEFAULT_TABLE table=test-workload [id=e92ceea777da4bd8982f0b57cf98dc37]), partition=RANGE (key) PARTITION 214748364 <= VALUES < 429496728
I20260129 02:58:12.498836 19909 tablet_service.cc:1505] Processing CreateTablet for tablet d5edd0a419ee46c9a6a223f8056afb4c (DEFAULT_TABLE table=test-workload [id=e92ceea777da4bd8982f0b57cf98dc37]), partition=RANGE (key) PARTITION 429496728 <= VALUES < 644245092
I20260129 02:58:12.499389 19907 tablet_service.cc:1505] Processing CreateTablet for tablet 79355f4f5d7942b6ac6f420256fd35b7 (DEFAULT_TABLE table=test-workload [id=e92ceea777da4bd8982f0b57cf98dc37]), partition=RANGE (key) PARTITION 858993456 <= VALUES < 1073741820
I20260129 02:58:12.500602 19908 tablet_service.cc:1505] Processing CreateTablet for tablet 47ad34405a6f4d508c9613c07a572cc3 (DEFAULT_TABLE table=test-workload [id=e92ceea777da4bd8982f0b57cf98dc37]), partition=RANGE (key) PARTITION 644245092 <= VALUES < 858993456
I20260129 02:58:12.501199 19906 tablet_service.cc:1505] Processing CreateTablet for tablet 620c6d80e1964c51a628ffb43f645798 (DEFAULT_TABLE table=test-workload [id=e92ceea777da4bd8982f0b57cf98dc37]), partition=RANGE (key) PARTITION 1073741820 <= VALUES < 1288490184
I20260129 02:58:12.511221 19910 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4562a703e5a44a1585b332b851bfd6f2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260129 02:58:12.516306 19907 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 79355f4f5d7942b6ac6f420256fd35b7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260129 02:58:12.520411 19906 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 620c6d80e1964c51a628ffb43f645798. 1 dirs total, 0 dirs full, 0 dirs failed
I20260129 02:58:12.524463 19909 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d5edd0a419ee46c9a6a223f8056afb4c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260129 02:58:12.528339 19908 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 47ad34405a6f4d508c9613c07a572cc3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260129 02:58:12.531455 20059 tablet_service.cc:1505] Processing CreateTablet for tablet 5d11df549a83429a87d3e7a62fabaa21 (DEFAULT_TABLE table=test-workload [id=e92ceea777da4bd8982f0b57cf98dc37]), partition=RANGE (key) PARTITION 1717986912 <= VALUES < 1932735276
I20260129 02:58:12.533528 19904 tablet_service.cc:1505] Processing CreateTablet for tablet 1dc801ccecc44ed79d85b0522e49d3e5 (DEFAULT_TABLE table=test-workload [id=e92ceea777da4bd8982f0b57cf98dc37]), partition=RANGE (key) PARTITION 1503238548 <= VALUES < 1717986912
I20260129 02:58:12.534178 19902 tablet_service.cc:1505] Processing CreateTablet for tablet 1bd5a7f0c64f4ef6bc2bb544cd69843c (DEFAULT_TABLE table=test-workload [id=e92ceea777da4bd8982f0b57cf98dc37]), partition=RANGE (key) PARTITION 1932735276 <= VALUES
I20260129 02:58:12.534878 19904 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1dc801ccecc44ed79d85b0522e49d3e5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260129 02:58:12.535619 19902 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1bd5a7f0c64f4ef6bc2bb544cd69843c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260129 02:58:12.537068 19903 tablet_service.cc:1505] Processing CreateTablet for tablet 5d11df549a83429a87d3e7a62fabaa21 (DEFAULT_TABLE table=test-workload [id=e92ceea777da4bd8982f0b57cf98dc37]), partition=RANGE (key) PARTITION 1717986912 <= VALUES < 1932735276
I20260129 02:58:12.538379 19903 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5d11df549a83429a87d3e7a62fabaa21. 1 dirs total, 0 dirs full, 0 dirs failed
I20260129 02:58:12.542934 19986 tablet_service.cc:1505] Processing CreateTablet for tablet 7d91c3094df94b47aa9a512d13490daa (DEFAULT_TABLE table=test-workload [id=e92ceea777da4bd8982f0b57cf98dc37]), partition=RANGE (key) PARTITION VALUES < 214748364
I20260129 02:58:12.544314 19986 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7d91c3094df94b47aa9a512d13490daa. 1 dirs total, 0 dirs full, 0 dirs failed
I20260129 02:58:12.544741 19982 tablet_service.cc:1505] Processing CreateTablet for tablet 79355f4f5d7942b6ac6f420256fd35b7 (DEFAULT_TABLE table=test-workload [id=e92ceea777da4bd8982f0b57cf98dc37]), partition=RANGE (key) PARTITION 858993456 <= VALUES < 1073741820
I20260129 02:58:12.546020 19982 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 79355f4f5d7942b6ac6f420256fd35b7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260129 02:58:12.549039 19981 tablet_service.cc:1505] Processing CreateTablet for tablet 620c6d80e1964c51a628ffb43f645798 (DEFAULT_TABLE table=test-workload [id=e92ceea777da4bd8982f0b57cf98dc37]), partition=RANGE (key) PARTITION 1073741820 <= VALUES < 1288490184
I20260129 02:58:12.550410 19981 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 620c6d80e1964c51a628ffb43f645798. 1 dirs total, 0 dirs full, 0 dirs failed
I20260129 02:58:12.550215 19984 tablet_service.cc:1505] Processing CreateTablet for tablet d5edd0a419ee46c9a6a223f8056afb4c (DEFAULT_TABLE table=test-workload [id=e92ceea777da4bd8982f0b57cf98dc37]), partition=RANGE (key) PARTITION 429496728 <= VALUES < 644245092
I20260129 02:58:12.551501 19984 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d5edd0a419ee46c9a6a223f8056afb4c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260129 02:58:12.553984 19979 tablet_service.cc:1505] Processing CreateTablet for tablet 1dc801ccecc44ed79d85b0522e49d3e5 (DEFAULT_TABLE table=test-workload [id=e92ceea777da4bd8982f0b57cf98dc37]), partition=RANGE (key) PARTITION 1503238548 <= VALUES < 1717986912
I20260129 02:58:12.554349 19980 tablet_service.cc:1505] Processing CreateTablet for tablet 901f619b02f843b4a2747de956182f12 (DEFAULT_TABLE table=test-workload [id=e92ceea777da4bd8982f0b57cf98dc37]), partition=RANGE (key) PARTITION 1288490184 <= VALUES < 1503238548
I20260129 02:58:12.549042 19985 tablet_service.cc:1505] Processing CreateTablet for tablet 4562a703e5a44a1585b332b851bfd6f2 (DEFAULT_TABLE table=test-workload [id=e92ceea777da4bd8982f0b57cf98dc37]), partition=RANGE (key) PARTITION 214748364 <= VALUES < 429496728
I20260129 02:58:12.557156 19977 tablet_service.cc:1505] Processing CreateTablet for tablet 1bd5a7f0c64f4ef6bc2bb544cd69843c (DEFAULT_TABLE table=test-workload [id=e92ceea777da4bd8982f0b57cf98dc37]), partition=RANGE (key) PARTITION 1932735276 <= VALUES
I20260129 02:58:12.560060 19978 tablet_service.cc:1505] Processing CreateTablet for tablet 5d11df549a83429a87d3e7a62fabaa21 (DEFAULT_TABLE table=test-workload [id=e92ceea777da4bd8982f0b57cf98dc37]), partition=RANGE (key) PARTITION 1717986912 <= VALUES < 1932735276
I20260129 02:58:12.553984 19983 tablet_service.cc:1505] Processing CreateTablet for tablet 47ad34405a6f4d508c9613c07a572cc3 (DEFAULT_TABLE table=test-workload [id=e92ceea777da4bd8982f0b57cf98dc37]), partition=RANGE (key) PARTITION 644245092 <= VALUES < 858993456
I20260129 02:58:12.567303 19979 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1dc801ccecc44ed79d85b0522e49d3e5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260129 02:58:12.568313 19983 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 47ad34405a6f4d508c9613c07a572cc3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260129 02:58:12.569306 19985 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4562a703e5a44a1585b332b851bfd6f2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260129 02:58:12.576216 19980 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 901f619b02f843b4a2747de956182f12. 1 dirs total, 0 dirs full, 0 dirs failed
I20260129 02:58:12.576937 19977 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1bd5a7f0c64f4ef6bc2bb544cd69843c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260129 02:58:12.580267 19978 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5d11df549a83429a87d3e7a62fabaa21. 1 dirs total, 0 dirs full, 0 dirs failed
I20260129 02:58:12.595002 20109 tablet_bootstrap.cc:492] T 7d91c3094df94b47aa9a512d13490daa P a634c60a5ca74b26b0d1f679d0166d0c: Bootstrap starting.
I20260129 02:58:12.594030 20059 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5d11df549a83429a87d3e7a62fabaa21. 1 dirs total, 0 dirs full, 0 dirs failed
I20260129 02:58:12.722030 20110 tablet_bootstrap.cc:492] T 5d11df549a83429a87d3e7a62fabaa21 P eafca908701c410ab9488970fcb3d74e: Bootstrap starting.
I20260129 02:58:12.728631 20109 tablet_bootstrap.cc:654] T 7d91c3094df94b47aa9a512d13490daa P a634c60a5ca74b26b0d1f679d0166d0c: Neither blocks nor log segments found. Creating new log.
W20260129 02:58:12.722033 19804 outbound_call.cc:321] RPC callback for RPC call kudu.tserver.TabletServerAdminService.CreateTablet -> {remote=127.18.211.3:39941, user_credentials={real_user=slave}} blocked reactor thread for 101650us
I20260129 02:58:12.745189 20110 tablet_bootstrap.cc:654] T 5d11df549a83429a87d3e7a62fabaa21 P eafca908701c410ab9488970fcb3d74e: Neither blocks nor log segments found. Creating new log.
I20260129 02:58:12.761982 20109 tablet_bootstrap.cc:492] T 7d91c3094df94b47aa9a512d13490daa P a634c60a5ca74b26b0d1f679d0166d0c: No bootstrap required, opened a new log
I20260129 02:58:12.762039 20111 tablet_bootstrap.cc:492] T 7d91c3094df94b47aa9a512d13490daa P 544851f9b7954fd2962675f957e8836a: Bootstrap starting.
I20260129 02:58:12.762707 20109 ts_tablet_manager.cc:1403] T 7d91c3094df94b47aa9a512d13490daa P a634c60a5ca74b26b0d1f679d0166d0c: Time spent bootstrapping tablet: real 0.168s user 0.008s sys 0.005s
I20260129 02:58:12.767386 20109 raft_consensus.cc:359] T 7d91c3094df94b47aa9a512d13490daa P a634c60a5ca74b26b0d1f679d0166d0c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a634c60a5ca74b26b0d1f679d0166d0c" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 39941 } } peers { permanent_uuid: "eafca908701c410ab9488970fcb3d74e" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 45001 } } peers { permanent_uuid: "544851f9b7954fd2962675f957e8836a" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34199 } }
I20260129 02:58:12.768357 20109 raft_consensus.cc:385] T 7d91c3094df94b47aa9a512d13490daa P a634c60a5ca74b26b0d1f679d0166d0c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260129 02:58:12.768721 20109 raft_consensus.cc:740] T 7d91c3094df94b47aa9a512d13490daa P a634c60a5ca74b26b0d1f679d0166d0c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a634c60a5ca74b26b0d1f679d0166d0c, State: Initialized, Role: FOLLOWER
I20260129 02:58:12.770401 20111 tablet_bootstrap.cc:654] T 7d91c3094df94b47aa9a512d13490daa P 544851f9b7954fd2962675f957e8836a: Neither blocks nor log segments found. Creating new log.
I20260129 02:58:12.769743 20109 consensus_queue.cc:260] T 7d91c3094df94b47aa9a512d13490daa P a634c60a5ca74b26b0d1f679d0166d0c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a634c60a5ca74b26b0d1f679d0166d0c" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 39941 } } peers { permanent_uuid: "eafca908701c410ab9488970fcb3d74e" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 45001 } } peers { permanent_uuid: "544851f9b7954fd2962675f957e8836a" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34199 } }
I20260129 02:58:12.774996 20110 tablet_bootstrap.cc:492] T 5d11df549a83429a87d3e7a62fabaa21 P eafca908701c410ab9488970fcb3d74e: No bootstrap required, opened a new log
I20260129 02:58:12.775394 20110 ts_tablet_manager.cc:1403] T 5d11df549a83429a87d3e7a62fabaa21 P eafca908701c410ab9488970fcb3d74e: Time spent bootstrapping tablet: real 0.054s user 0.007s sys 0.027s
I20260129 02:58:12.775815 20111 tablet_bootstrap.cc:492] T 7d91c3094df94b47aa9a512d13490daa P 544851f9b7954fd2962675f957e8836a: No bootstrap required, opened a new log
I20260129 02:58:12.776388 20111 ts_tablet_manager.cc:1403] T 7d91c3094df94b47aa9a512d13490daa P 544851f9b7954fd2962675f957e8836a: Time spent bootstrapping tablet: real 0.015s user 0.007s sys 0.005s
I20260129 02:58:12.777052 20095 heartbeater.cc:499] Master 127.18.211.62:37849 was elected leader, sending a full tablet report...
I20260129 02:58:12.777688 20110 raft_consensus.cc:359] T 5d11df549a83429a87d3e7a62fabaa21 P eafca908701c410ab9488970fcb3d74e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eafca908701c410ab9488970fcb3d74e" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 45001 } } peers { permanent_uuid: "544851f9b7954fd2962675f957e8836a" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34199 } } peers { permanent_uuid: "a634c60a5ca74b26b0d1f679d0166d0c" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 39941 } }
I20260129 02:58:12.778220 20110 raft_consensus.cc:385] T 5d11df549a83429a87d3e7a62fabaa21 P eafca908701c410ab9488970fcb3d74e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260129 02:58:12.778455 20110 raft_consensus.cc:740] T 5d11df549a83429a87d3e7a62fabaa21 P eafca908701c410ab9488970fcb3d74e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eafca908701c410ab9488970fcb3d74e, State: Initialized, Role: FOLLOWER
I20260129 02:58:12.779047 20110 consensus_queue.cc:260] T 5d11df549a83429a87d3e7a62fabaa21 P eafca908701c410ab9488970fcb3d74e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eafca908701c410ab9488970fcb3d74e" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 45001 } } peers { permanent_uuid: "544851f9b7954fd2962675f957e8836a" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34199 } } peers { permanent_uuid: "a634c60a5ca74b26b0d1f679d0166d0c" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 39941 } }
I20260129 02:58:12.778954 20111 raft_consensus.cc:359] T 7d91c3094df94b47aa9a512d13490daa P 544851f9b7954fd2962675f957e8836a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a634c60a5ca74b26b0d1f679d0166d0c" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 39941 } } peers { permanent_uuid: "eafca908701c410ab9488970fcb3d74e" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 45001 } } peers { permanent_uuid: "544851f9b7954fd2962675f957e8836a" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34199 } }
I20260129 02:58:12.779804 20111 raft_consensus.cc:385] T 7d91c3094df94b47aa9a512d13490daa P 544851f9b7954fd2962675f957e8836a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260129 02:58:12.780180 20111 raft_consensus.cc:740] T 7d91c3094df94b47aa9a512d13490daa P 544851f9b7954fd2962675f957e8836a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 544851f9b7954fd2962675f957e8836a, State: Initialized, Role: FOLLOWER
I20260129 02:58:12.781364 19946 heartbeater.cc:499] Master 127.18.211.62:37849 was elected leader, sending a full tablet report...
I20260129 02:58:12.781002 20111 consensus_queue.cc:260] T 7d91c3094df94b47aa9a512d13490daa P 544851f9b7954fd2962675f957e8836a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a634c60a5ca74b26b0d1f679d0166d0c" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 39941 } } peers { permanent_uuid: "eafca908701c410ab9488970fcb3d74e" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 45001 } } peers { permanent_uuid: "544851f9b7954fd2962675f957e8836a" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34199 } }
I20260129 02:58:12.781966 20110 ts_tablet_manager.cc:1434] T 5d11df549a83429a87d3e7a62fabaa21 P eafca908701c410ab9488970fcb3d74e: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260129 02:58:12.783131 20110 tablet_bootstrap.cc:492] T d5edd0a419ee46c9a6a223f8056afb4c P eafca908701c410ab9488970fcb3d74e: Bootstrap starting.
I20260129 02:58:12.787349 20021 heartbeater.cc:499] Master 127.18.211.62:37849 was elected leader, sending a full tablet report...
I20260129 02:58:12.788152 20111 ts_tablet_manager.cc:1434] T 7d91c3094df94b47aa9a512d13490daa P 544851f9b7954fd2962675f957e8836a: Time spent starting tablet: real 0.011s user 0.005s sys 0.007s
I20260129 02:58:12.789120 20111 tablet_bootstrap.cc:492] T d5edd0a419ee46c9a6a223f8056afb4c P 544851f9b7954fd2962675f957e8836a: Bootstrap starting.
I20260129 02:58:12.790496 20110 tablet_bootstrap.cc:654] T d5edd0a419ee46c9a6a223f8056afb4c P eafca908701c410ab9488970fcb3d74e: Neither blocks nor log segments found. Creating new log.
I20260129 02:58:12.795352 20111 tablet_bootstrap.cc:654] T d5edd0a419ee46c9a6a223f8056afb4c P 544851f9b7954fd2962675f957e8836a: Neither blocks nor log segments found. Creating new log.
I20260129 02:58:12.795368 20110 tablet_bootstrap.cc:492] T d5edd0a419ee46c9a6a223f8056afb4c P eafca908701c410ab9488970fcb3d74e: No bootstrap required, opened a new log
I20260129 02:58:12.796321 20110 ts_tablet_manager.cc:1403] T d5edd0a419ee46c9a6a223f8056afb4c P eafca908701c410ab9488970fcb3d74e: Time spent bootstrapping tablet: real 0.013s user 0.008s sys 0.004s
I20260129 02:58:12.796459 20109 ts_tablet_manager.cc:1434] T 7d91c3094df94b47aa9a512d13490daa P a634c60a5ca74b26b0d1f679d0166d0c: Time spent starting tablet: real 0.032s user 0.009s sys 0.004s
I20260129 02:58:12.797606 20109 tablet_bootstrap.cc:492] T 4562a703e5a44a1585b332b851bfd6f2 P a634c60a5ca74b26b0d1f679d0166d0c: Bootstrap starting.
I20260129 02:58:12.801532 20110 raft_consensus.cc:359] T d5edd0a419ee46c9a6a223f8056afb4c P eafca908701c410ab9488970fcb3d74e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eafca908701c410ab9488970fcb3d74e" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 45001 } } peers { permanent_uuid: "544851f9b7954fd2962675f957e8836a" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34199 } } peers { permanent_uuid: "a634c60a5ca74b26b0d1f679d0166d0c" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 39941 } }
I20260129 02:58:12.802211 20110 raft_consensus.cc:385] T d5edd0a419ee46c9a6a223f8056afb4c P eafca908701c410ab9488970fcb3d74e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260129 02:58:12.802492 20110 raft_consensus.cc:740] T d5edd0a419ee46c9a6a223f8056afb4c P eafca908701c410ab9488970fcb3d74e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eafca908701c410ab9488970fcb3d74e, State: Initialized, Role: FOLLOWER
I20260129 02:58:12.803550 20109 tablet_bootstrap.cc:654] T 4562a703e5a44a1585b332b851bfd6f2 P a634c60a5ca74b26b0d1f679d0166d0c: Neither blocks nor log segments found. Creating new log.
I20260129 02:58:12.803187 20110 consensus_queue.cc:260] T d5edd0a419ee46c9a6a223f8056afb4c P eafca908701c410ab9488970fcb3d74e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eafca908701c410ab9488970fcb3d74e" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 45001 } } peers { permanent_uuid: "544851f9b7954fd2962675f957e8836a" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34199 } } peers { permanent_uuid: "a634c60a5ca74b26b0d1f679d0166d0c" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 39941 } }
I20260129 02:58:12.805099 20110 ts_tablet_manager.cc:1434] T d5edd0a419ee46c9a6a223f8056afb4c P eafca908701c410ab9488970fcb3d74e: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20260129 02:58:12.805881 20110 tablet_bootstrap.cc:492] T 7d91c3094df94b47aa9a512d13490daa P eafca908701c410ab9488970fcb3d74e: Bootstrap starting.
I20260129 02:58:12.811180 20110 tablet_bootstrap.cc:654] T 7d91c3094df94b47aa9a512d13490daa P eafca908701c410ab9488970fcb3d74e: Neither blocks nor log segments found. Creating new log.
I20260129 02:58:12.816762 20111 tablet_bootstrap.cc:492] T d5edd0a419ee46c9a6a223f8056afb4c P 544851f9b7954fd2962675f957e8836a: No bootstrap required, opened a new log
I20260129 02:58:12.817379 20111 ts_tablet_manager.cc:1403] T d5edd0a419ee46c9a6a223f8056afb4c P 544851f9b7954fd2962675f957e8836a: Time spent bootstrapping tablet: real 0.028s user 0.009s sys 0.002s
I20260129 02:58:12.820668 20111 raft_consensus.cc:359] T d5edd0a419ee46c9a6a223f8056afb4c P 544851f9b7954fd2962675f957e8836a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eafca908701c410ab9488970fcb3d74e" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 45001 } } peers { permanent_uuid: "544851f9b7954fd2962675f957e8836a" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34199 } } peers { permanent_uuid: "a634c60a5ca74b26b0d1f679d0166d0c" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 39941 } }
I20260129 02:58:12.821542 20111 raft_consensus.cc:385] T d5edd0a419ee46c9a6a223f8056afb4c P 544851f9b7954fd2962675f957e8836a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260129 02:58:12.821869 20111 raft_consensus.cc:740] T d5edd0a419ee46c9a6a223f8056afb4c P 544851f9b7954fd2962675f957e8836a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 544851f9b7954fd2962675f957e8836a, State: Initialized, Role: FOLLOWER
I20260129 02:58:12.822791 20111 consensus_queue.cc:260] T d5edd0a419ee46c9a6a223f8056afb4c P 544851f9b7954fd2962675f957e8836a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eafca908701c410ab9488970fcb3d74e" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 45001 } } peers { permanent_uuid: "544851f9b7954fd2962675f957e8836a" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34199 } } peers { permanent_uuid: "a634c60a5ca74b26b0d1f679d0166d0c" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 39941 } }
I20260129 02:58:12.825762 20111 ts_tablet_manager.cc:1434] T d5edd0a419ee46c9a6a223f8056afb4c P 544851f9b7954fd2962675f957e8836a: Time spent starting tablet: real 0.008s user 0.008s sys 0.000s
I20260129 02:58:12.826983 20111 tablet_bootstrap.cc:492] T 47ad34405a6f4d508c9613c07a572cc3 P 544851f9b7954fd2962675f957e8836a: Bootstrap starting.
I20260129 02:58:12.836675 20111 tablet_bootstrap.cc:654] T 47ad34405a6f4d508c9613c07a572cc3 P 544851f9b7954fd2962675f957e8836a: Neither blocks nor log segments found. Creating new log.
I20260129 02:58:12.839062 20109 tablet_bootstrap.cc:492] T 4562a703e5a44a1585b332b851bfd6f2 P a634c60a5ca74b26b0d1f679d0166d0c: No bootstrap required, opened a new log
I20260129 02:58:12.839594 20109 ts_tablet_manager.cc:1403] T 4562a703e5a44a1585b332b851bfd6f2 P a634c60a5ca74b26b0d1f679d0166d0c: Time spent bootstrapping tablet: real 0.042s user 0.007s sys 0.028s
I20260129 02:58:12.842183 20109 raft_consensus.cc:359] T 4562a703e5a44a1585b332b851bfd6f2 P a634c60a5ca74b26b0d1f679d0166d0c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eafca908701c410ab9488970fcb3d74e" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 45001 } } peers { permanent_uuid: "544851f9b7954fd2962675f957e8836a" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34199 } } peers { permanent_uuid: "a634c60a5ca74b26b0d1f679d0166d0c" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 39941 } }
I20260129 02:58:12.842887 20109 raft_consensus.cc:385] T 4562a703e5a44a1585b332b851bfd6f2 P a634c60a5ca74b26b0d1f679d0166d0c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260129 02:58:12.843207 20109 raft_consensus.cc:740] T 4562a703e5a44a1585b332b851bfd6f2 P a634c60a5ca74b26b0d1f679d0166d0c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a634c60a5ca74b26b0d1f679d0166d0c, State: Initialized, Role: FOLLOWER
I20260129 02:58:12.844098 20109 consensus_queue.cc:260] T 4562a703e5a44a1585b332b851bfd6f2 P a634c60a5ca74b26b0d1f679d0166d0c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eafca908701c410ab9488970fcb3d74e" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 45001 } } peers { permanent_uuid: "544851f9b7954fd2962675f957e8836a" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34199 } } peers { permanent_uuid: "a634c60a5ca74b26b0d1f679d0166d0c" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 39941 } }
I20260129 02:58:12.846621 20109 ts_tablet_manager.cc:1434] T 4562a703e5a44a1585b332b851bfd6f2 P a634c60a5ca74b26b0d1f679d0166d0c: Time spent starting tablet: real 0.007s user 0.001s sys 0.004s
I20260129 02:58:12.847487 20109 tablet_bootstrap.cc:492] T 901f619b02f843b4a2747de956182f12 P a634c60a5ca74b26b0d1f679d0166d0c: Bootstrap starting.
I20260129 02:58:12.848490 20114 raft_consensus.cc:493] T 4562a703e5a44a1585b332b851bfd6f2 P a634c60a5ca74b26b0d1f679d0166d0c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260129 02:58:12.848912 20114 raft_consensus.cc:515] T 4562a703e5a44a1585b332b851bfd6f2 P a634c60a5ca74b26b0d1f679d0166d0c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eafca908701c410ab9488970fcb3d74e" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 45001 } } peers { permanent_uuid: "544851f9b7954fd2962675f957e8836a" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34199 } } peers { permanent_uuid: "a634c60a5ca74b26b0d1f679d0166d0c" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 39941 } }
I20260129 02:58:12.851406 20114 leader_election.cc:290] T 4562a703e5a44a1585b332b851bfd6f2 P a634c60a5ca74b26b0d1f679d0166d0c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers eafca908701c410ab9488970fcb3d74e (127.18.211.1:45001), 544851f9b7954fd2962675f957e8836a (127.18.211.2:34199)
I20260129 02:58:12.853658 20109 tablet_bootstrap.cc:654] T 901f619b02f843b4a2747de956182f12 P a634c60a5ca74b26b0d1f679d0166d0c: Neither blocks nor log segments found. Creating new log.
I20260129 02:58:12.858925 20110 tablet_bootstrap.cc:492] T 7d91c3094df94b47aa9a512d13490daa P eafca908701c410ab9488970fcb3d74e: No bootstrap required, opened a new log
I20260129 02:58:12.859495 20110 ts_tablet_manager.cc:1403] T 7d91c3094df94b47aa9a512d13490daa P eafca908701c410ab9488970fcb3d74e: Time spent bootstrapping tablet: real 0.054s user 0.031s sys 0.018s
I20260129 02:58:12.862957 20110 raft_consensus.cc:359] T 7d91c3094df94b47aa9a512d13490daa P eafca908701c410ab9488970fcb3d74e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a634c60a5ca74b26b0d1f679d0166d0c" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 39941 } } peers { permanent_uuid: "eafca908701c410ab9488970fcb3d74e" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 45001 } } peers { permanent_uuid: "544851f9b7954fd2962675f957e8836a" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34199 } }
I20260129 02:58:12.863956 20110 raft_consensus.cc:385] T 7d91c3094df94b47aa9a512d13490daa P eafca908701c410ab9488970fcb3d74e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260129 02:58:12.864332 20110 raft_consensus.cc:740] T 7d91c3094df94b47aa9a512d13490daa P eafca908701c410ab9488970fcb3d74e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eafca908701c410ab9488970fcb3d74e, State: Initialized, Role: FOLLOWER
I20260129 02:58:12.865336 20110 consensus_queue.cc:260] T 7d91c3094df94b47aa9a512d13490daa P eafca908701c410ab9488970fcb3d74e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a634c60a5ca74b26b0d1f679d0166d0c" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 39941 } } peers { permanent_uuid: "eafca908701c410ab9488970fcb3d74e" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 45001 } } peers { permanent_uuid: "544851f9b7954fd2962675f957e8836a" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34199 } }
I20260129 02:58:12.871083 20110 ts_tablet_manager.cc:1434] T 7d91c3094df94b47aa9a512d13490daa P eafca908701c410ab9488970fcb3d74e: Time spent starting tablet: real 0.011s user 0.009s sys 0.001s
I20260129 02:58:12.872339 20110 tablet_bootstrap.cc:492] T 4562a703e5a44a1585b332b851bfd6f2 P eafca908701c410ab9488970fcb3d74e: Bootstrap starting.
I20260129 02:58:12.880400 20110 tablet_bootstrap.cc:654] T 4562a703e5a44a1585b332b851bfd6f2 P eafca908701c410ab9488970fcb3d74e: Neither blocks nor log segments found. Creating new log.
I20260129 02:58:12.893761 20116 raft_consensus.cc:493] T d5edd0a419ee46c9a6a223f8056afb4c P eafca908701c410ab9488970fcb3d74e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260129 02:58:12.894461 20116 raft_consensus.cc:515] T d5edd0a419ee46c9a6a223f8056afb4c P eafca908701c410ab9488970fcb3d74e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eafca908701c410ab9488970fcb3d74e" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 45001 } } peers { permanent_uuid: "544851f9b7954fd2962675f957e8836a" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34199 } } peers { permanent_uuid: "a634c60a5ca74b26b0d1f679d0166d0c" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 39941 } }
I20260129 02:58:12.907272 20116 leader_election.cc:290] T d5edd0a419ee46c9a6a223f8056afb4c P eafca908701c410ab9488970fcb3d74e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 544851f9b7954fd2962675f957e8836a (127.18.211.2:34199), a634c60a5ca74b26b0d1f679d0166d0c (127.18.211.3:39941)
I20260129 02:58:12.918140 20110 tablet_bootstrap.cc:492] T 4562a703e5a44a1585b332b851bfd6f2 P eafca908701c410ab9488970fcb3d74e: No bootstrap required, opened a new log
I20260129 02:58:12.918869 20110 ts_tablet_manager.cc:1403] T 4562a703e5a44a1585b332b851bfd6f2 P eafca908701c410ab9488970fcb3d74e: Time spent bootstrapping tablet: real 0.047s user 0.018s sys 0.001s
I20260129 02:58:12.924952 20110 raft_consensus.cc:359] T 4562a703e5a44a1585b332b851bfd6f2 P eafca908701c410ab9488970fcb3d74e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eafca908701c410ab9488970fcb3d74e" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 45001 } } peers { permanent_uuid: "544851f9b7954fd2962675f957e8836a" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34199 } } peers { permanent_uuid: "a634c60a5ca74b26b0d1f679d0166d0c" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 39941 } }
I20260129 02:58:12.925868 20110 raft_consensus.cc:385] T 4562a703e5a44a1585b332b851bfd6f2 P eafca908701c410ab9488970fcb3d74e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260129 02:58:12.926218 20110 raft_consensus.cc:740] T 4562a703e5a44a1585b332b851bfd6f2 P eafca908701c410ab9488970fcb3d74e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eafca908701c410ab9488970fcb3d74e, State: Initialized, Role: FOLLOWER
I20260129 02:58:12.928272 19996 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4562a703e5a44a1585b332b851bfd6f2" candidate_uuid: "a634c60a5ca74b26b0d1f679d0166d0c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "544851f9b7954fd2962675f957e8836a" is_pre_election: true
I20260129 02:58:12.929934 20111 tablet_bootstrap.cc:492] T 47ad34405a6f4d508c9613c07a572cc3 P 544851f9b7954fd2962675f957e8836a: No bootstrap required, opened a new log
W20260129 02:58:12.930284 20033 leader_election.cc:343] T 4562a703e5a44a1585b332b851bfd6f2 P a634c60a5ca74b26b0d1f679d0166d0c [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 544851f9b7954fd2962675f957e8836a (127.18.211.2:34199): Illegal state: must be running to vote when last-logged opid is not known
I20260129 02:58:12.930817 20111 ts_tablet_manager.cc:1403] T 47ad34405a6f4d508c9613c07a572cc3 P 544851f9b7954fd2962675f957e8836a: Time spent bootstrapping tablet: real 0.104s user 0.052s sys 0.044s
I20260129 02:58:12.934438 19921 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4562a703e5a44a1585b332b851bfd6f2" candidate_uuid: "a634c60a5ca74b26b0d1f679d0166d0c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eafca908701c410ab9488970fcb3d74e" is_pre_election: true
I20260129 02:58:12.927119 20110 consensus_queue.cc:260] T 4562a703e5a44a1585b332b851bfd6f2 P eafca908701c410ab9488970fcb3d74e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eafca908701c410ab9488970fcb3d74e" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 45001 } } peers { permanent_uuid: "544851f9b7954fd2962675f957e8836a" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34199 } } peers { permanent_uuid: "a634c60a5ca74b26b0d1f679d0166d0c" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 39941 } }
I20260129 02:58:12.934486 20111 raft_consensus.cc:359] T 47ad34405a6f4d508c9613c07a572cc3 P 544851f9b7954fd2962675f957e8836a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eafca908701c410ab9488970fcb3d74e" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 45001 } } peers { permanent_uuid: "544851f9b7954fd2962675f957e8836a" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34199 } } peers { permanent_uuid: "a634c60a5ca74b26b0d1f679d0166d0c" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 39941 } }
I20260129 02:58:12.936223 19921 raft_consensus.cc:2468] T 4562a703e5a44a1585b332b851bfd6f2 P eafca908701c410ab9488970fcb3d74e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a634c60a5ca74b26b0d1f679d0166d0c in term 0.
I20260129 02:58:12.936349 20111 raft_consensus.cc:385] T 47ad34405a6f4d508c9613c07a572cc3 P 544851f9b7954fd2962675f957e8836a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260129 02:58:12.938555 20111 raft_consensus.cc:740] T 47ad34405a6f4d508c9613c07a572cc3 P 544851f9b7954fd2962675f957e8836a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 544851f9b7954fd2962675f957e8836a, State: Initialized, Role: FOLLOWER
I20260129 02:58:12.939352 20111 consensus_queue.cc:260] T 47ad34405a6f4d508c9613c07a572cc3 P 544851f9b7954fd2962675f957e8836a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eafca908701c410ab9488970fcb3d74e" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 45001 } } peers { permanent_uuid: "544851f9b7954fd2962675f957e8836a" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34199 } } peers { permanent_uuid: "a634c60a5ca74b26b0d1f679d0166d0c" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 39941 } }
I20260129 02:58:12.941880 20111 ts_tablet_manager.cc:1434] T 47ad34405a6f4d508c9613c07a572cc3 P 544851f9b7954fd2962675f957e8836a: Time spent starting tablet: real 0.011s user 0.006s sys 0.000s
I20260129 02:58:12.941820 20033 leader_election.cc:304] T 4562a703e5a44a1585b332b851bfd6f2 P a634c60a5ca74b26b0d1f679d0166d0c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: a634c60a5ca74b26b0d1f679d0166d0c, eafca908701c410ab9488970fcb3d74e; no voters: 544851f9b7954fd2962675f957e8836a
I20260129 02:58:12.942988 20111 tablet_bootstrap.cc:492] T 1dc801ccecc44ed79d85b0522e49d3e5 P 544851f9b7954fd2962675f957e8836a: Bootstrap starting.
I20260129 02:58:12.943061 20114 raft_consensus.cc:2804] T 4562a703e5a44a1585b332b851bfd6f2 P a634c60a5ca74b26b0d1f679d0166d0c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260129 02:58:12.943514 20114 raft_consensus.cc:493] T 4562a703e5a44a1585b332b851bfd6f2 P a634c60a5ca74b26b0d1f679d0166d0c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260129 02:58:12.943895 20114 raft_consensus.cc:3060] T 4562a703e5a44a1585b332b851bfd6f2 P a634c60a5ca74b26b0d1f679d0166d0c [term 0 FOLLOWER]: Advancing to term 1
I20260129 02:58:12.944454 20110 ts_tablet_manager.cc:1434] T 4562a703e5a44a1585b332b851bfd6f2 P eafca908701c410ab9488970fcb3d74e: Time spent starting tablet: real 0.025s user 0.010s sys 0.000s
I20260129 02:58:12.945530 20110 tablet_bootstrap.cc:492] T 79355f4f5d7942b6ac6f420256fd35b7 P eafca908701c410ab9488970fcb3d74e: Bootstrap starting.
I20260129 02:58:12.949620 20111 tablet_bootstrap.cc:654] T 1dc801ccecc44ed79d85b0522e49d3e5 P 544851f9b7954fd2962675f957e8836a: Neither blocks nor log segments found. Creating new log.
I20260129 02:58:12.950908 20114 raft_consensus.cc:515] T 4562a703e5a44a1585b332b851bfd6f2 P a634c60a5ca74b26b0d1f679d0166d0c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eafca908701c410ab9488970fcb3d74e" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 45001 } } peers { permanent_uuid: "544851f9b7954fd2962675f957e8836a" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34199 } } peers { permanent_uuid: "a634c60a5ca74b26b0d1f679d0166d0c" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 39941 } }
I20260129 02:58:12.951669 20110 tablet_bootstrap.cc:654] T 79355f4f5d7942b6ac6f420256fd35b7 P eafca908701c410ab9488970fcb3d74e: Neither blocks nor log segments found. Creating new log.
I20260129 02:58:12.953832 19921 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4562a703e5a44a1585b332b851bfd6f2" candidate_uuid: "a634c60a5ca74b26b0d1f679d0166d0c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eafca908701c410ab9488970fcb3d74e"
I20260129 02:58:12.954473 19921 raft_consensus.cc:3060] T 4562a703e5a44a1585b332b851bfd6f2 P eafca908701c410ab9488970fcb3d74e [term 0 FOLLOWER]: Advancing to term 1
I20260129 02:58:12.943903 20109 tablet_bootstrap.cc:492] T 901f619b02f843b4a2747de956182f12 P a634c60a5ca74b26b0d1f679d0166d0c: No bootstrap required, opened a new log
I20260129 02:58:12.962749 20109 ts_tablet_manager.cc:1403] T 901f619b02f843b4a2747de956182f12 P a634c60a5ca74b26b0d1f679d0166d0c: Time spent bootstrapping tablet: real 0.115s user 0.054s sys 0.031s
I20260129 02:58:12.966140 19921 raft_consensus.cc:2468] T 4562a703e5a44a1585b332b851bfd6f2 P eafca908701c410ab9488970fcb3d74e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a634c60a5ca74b26b0d1f679d0166d0c in term 1.
I20260129 02:58:12.966419 20109 raft_consensus.cc:359] T 901f619b02f843b4a2747de956182f12 P a634c60a5ca74b26b0d1f679d0166d0c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eafca908701c410ab9488970fcb3d74e" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 45001 } } peers { permanent_uuid: "544851f9b7954fd2962675f957e8836a" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34199 } } peers { permanent_uuid: "a634c60a5ca74b26b0d1f679d0166d0c" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 39941 } }
I20260129 02:58:12.967257 20109 raft_consensus.cc:385] T 901f619b02f843b4a2747de956182f12 P a634c60a5ca74b26b0d1f679d0166d0c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260129 02:58:12.967428 20033 leader_election.cc:304] T 4562a703e5a44a1585b332b851bfd6f2 P a634c60a5ca74b26b0d1f679d0166d0c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a634c60a5ca74b26b0d1f679d0166d0c, eafca908701c410ab9488970fcb3d74e; no voters:
I20260129 02:58:12.969219 20114 leader_election.cc:290] T 4562a703e5a44a1585b332b851bfd6f2 P a634c60a5ca74b26b0d1f679d0166d0c [CANDIDATE]: Term 1 election: Requested vote from peers eafca908701c410ab9488970fcb3d74e (127.18.211.1:45001), 544851f9b7954fd2962675f957e8836a (127.18.211.2:34199)
I20260129 02:58:12.969885 20114 raft_consensus.cc:493] T 7d91c3094df94b47aa9a512d13490daa P a634c60a5ca74b26b0d1f679d0166d0c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260129 02:58:12.970376 20114 raft_consensus.cc:515] T 7d91c3094df94b47aa9a512d13490daa P a634c60a5ca74b26b0d1f679d0166d0c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a634c60a5ca74b26b0d1f679d0166d0c" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 39941 } } peers { permanent_uuid: "eafca908701c410ab9488970fcb3d74e" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 45001 } } peers { permanent_uuid: "544851f9b7954fd2962675f957e8836a" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34199 } }
I20260129 02:58:12.967753 19996 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d5edd0a419ee46c9a6a223f8056afb4c" candidate_uuid: "eafca908701c410ab9488970fcb3d74e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "544851f9b7954fd2962675f957e8836a" is_pre_election: true
I20260129 02:58:12.971897 20111 tablet_bootstrap.cc:492] T 1dc801ccecc44ed79d85b0522e49d3e5 P 544851f9b7954fd2962675f957e8836a: No bootstrap required, opened a new log
I20260129 02:58:12.972223 20114 leader_election.cc:290] T 7d91c3094df94b47aa9a512d13490daa P a634c60a5ca74b26b0d1f679d0166d0c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers eafca908701c410ab9488970fcb3d74e (127.18.211.1:45001), 544851f9b7954fd2962675f957e8836a (127.18.211.2:34199)
I20260129 02:58:12.971971 19996 raft_consensus.cc:2468] T d5edd0a419ee46c9a6a223f8056afb4c P 544851f9b7954fd2962675f957e8836a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eafca908701c410ab9488970fcb3d74e in term 0.
I20260129 02:58:12.972466 20111 ts_tablet_manager.cc:1403] T 1dc801ccecc44ed79d85b0522e49d3e5 P 544851f9b7954fd2962675f957e8836a: Time spent bootstrapping tablet: real 0.030s user 0.015s sys 0.006s
I20260129 02:58:12.973487 19884 leader_election.cc:304] T d5edd0a419ee46c9a6a223f8056afb4c P eafca908701c410ab9488970fcb3d74e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 544851f9b7954fd2962675f957e8836a, eafca908701c410ab9488970fcb3d74e; no voters:
I20260129 02:58:12.974308 20116 raft_consensus.cc:2804] T d5edd0a419ee46c9a6a223f8056afb4c P eafca908701c410ab9488970fcb3d74e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260129 02:58:12.974653 20116 raft_consensus.cc:493] T d5edd0a419ee46c9a6a223f8056afb4c P eafca908701c410ab9488970fcb3d74e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260129 02:58:12.974965 20116 raft_consensus.cc:3060] T d5edd0a419ee46c9a6a223f8056afb4c P eafca908701c410ab9488970fcb3d74e [term 0 FOLLOWER]: Advancing to term 1
I20260129 02:58:12.975230 20111 raft_consensus.cc:359] T 1dc801ccecc44ed79d85b0522e49d3e5 P 544851f9b7954fd2962675f957e8836a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eafca908701c410ab9488970fcb3d74e" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 45001 } } peers { permanent_uuid: "544851f9b7954fd2962675f957e8836a" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34199 } } peers { permanent_uuid: "a634c60a5ca74b26b0d1f679d0166d0c" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 39941 } }
I20260129 02:58:12.975942 20111 raft_consensus.cc:385] T 1dc801ccecc44ed79d85b0522e49d3e5 P 544851f9b7954fd2962675f957e8836a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260129 02:58:12.976222 20111 raft_consensus.cc:740] T 1dc801ccecc44ed79d85b0522e49d3e5 P 544851f9b7954fd2962675f957e8836a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 544851f9b7954fd2962675f957e8836a, State: Initialized, Role: FOLLOWER
I20260129 02:58:12.976969 20111 consensus_queue.cc:260] T 1dc801ccecc44ed79d85b0522e49d3e5 P 544851f9b7954fd2962675f957e8836a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eafca908701c410ab9488970fcb3d74e" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 45001 } } peers { permanent_uuid: "544851f9b7954fd2962675f957e8836a" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34199 } } peers { permanent_uuid: "a634c60a5ca74b26b0d1f679d0166d0c" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 39941 } }
I20260129 02:58:12.979278 20111 ts_tablet_manager.cc:1434] T 1dc801ccecc44ed79d85b0522e49d3e5 P 544851f9b7954fd2962675f957e8836a: Time spent starting tablet: real 0.006s user 0.001s sys 0.003s
I20260129 02:58:12.981150 20111 tablet_bootstrap.cc:492] T 620c6d80e1964c51a628ffb43f645798 P 544851f9b7954fd2962675f957e8836a: Bootstrap starting.
I20260129 02:58:12.967567 20109 raft_consensus.cc:740] T 901f619b02f843b4a2747de956182f12 P a634c60a5ca74b26b0d1f679d0166d0c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a634c60a5ca74b26b0d1f679d0166d0c, State: Initialized, Role: FOLLOWER
I20260129 02:58:12.988752 20117 raft_consensus.cc:493] T 47ad34405a6f4d508c9613c07a572cc3 P 544851f9b7954fd2962675f957e8836a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260129 02:58:12.989334 20117 raft_consensus.cc:515] T 47ad34405a6f4d508c9613c07a572cc3 P 544851f9b7954fd2962675f957e8836a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eafca908701c410ab9488970fcb3d74e" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 45001 } } peers { permanent_uuid: "544851f9b7954fd2962675f957e8836a" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34199 } } peers { permanent_uuid: "a634c60a5ca74b26b0d1f679d0166d0c" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 39941 } }
I20260129 02:58:12.990690 20070 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d5edd0a419ee46c9a6a223f8056afb4c" candidate_uuid: "eafca908701c410ab9488970fcb3d74e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a634c60a5ca74b26b0d1f679d0166d0c" is_pre_election: true
I20260129 02:58:12.992488 20110 tablet_bootstrap.cc:492] T 79355f4f5d7942b6ac6f420256fd35b7 P eafca908701c410ab9488970fcb3d74e: No bootstrap required, opened a new log
I20260129 02:58:12.993247 20110 ts_tablet_manager.cc:1403] T 79355f4f5d7942b6ac6f420256fd35b7 P eafca908701c410ab9488970fcb3d74e: Time spent bootstrapping tablet: real 0.048s user 0.022s sys 0.017s
I20260129 02:58:12.993494 20117 leader_election.cc:290] T 47ad34405a6f4d508c9613c07a572cc3 P 544851f9b7954fd2962675f957e8836a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers eafca908701c410ab9488970fcb3d74e (127.18.211.1:45001), a634c60a5ca74b26b0d1f679d0166d0c (127.18.211.3:39941)
W20260129 02:58:12.994540 19885 leader_election.cc:343] T d5edd0a419ee46c9a6a223f8056afb4c P eafca908701c410ab9488970fcb3d74e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a634c60a5ca74b26b0d1f679d0166d0c (127.18.211.3:39941): Illegal state: must be running to vote when last-logged opid is not known
I20260129 02:58:12.997380 20110 raft_consensus.cc:359] T 79355f4f5d7942b6ac6f420256fd35b7 P eafca908701c410ab9488970fcb3d74e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eafca908701c410ab9488970fcb3d74e" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 45001 } } peers { permanent_uuid: "544851f9b7954fd2962675f957e8836a" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34199 } } peers { permanent_uuid: "a634c60a5ca74b26b0d1f679d0166d0c" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 39941 } }
I20260129 02:58:13.001724 20110 raft_consensus.cc:385] T 79355f4f5d7942b6ac6f420256fd35b7 P eafca908701c410ab9488970fcb3d74e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260129 02:58:12.989301 20133 raft_consensus.cc:493] T 5d11df549a83429a87d3e7a62fabaa21 P eafca908701c410ab9488970fcb3d74e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260129 02:58:13.003227 20133 raft_consensus.cc:515] T 5d11df549a83429a87d3e7a62fabaa21 P eafca908701c410ab9488970fcb3d74e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eafca908701c410ab9488970fcb3d74e" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 45001 } } peers { permanent_uuid: "544851f9b7954fd2962675f957e8836a" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34199 } } peers { permanent_uuid: "a634c60a5ca74b26b0d1f679d0166d0c" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 39941 } }
I20260129 02:58:12.986024 20114 raft_consensus.cc:2804] T 4562a703e5a44a1585b332b851bfd6f2 P a634c60a5ca74b26b0d1f679d0166d0c [term 1 FOLLOWER]: Leader election won for term 1
I20260129 02:58:13.006599 20110 raft_consensus.cc:740] T 79355f4f5d7942b6ac6f420256fd35b7 P eafca908701c410ab9488970fcb3d74e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eafca908701c410ab9488970fcb3d74e, State: Initialized, Role: FOLLOWER
I20260129 02:58:13.007136 20109 consensus_queue.cc:260] T 901f619b02f843b4a2747de956182f12 P a634c60a5ca74b26b0d1f679d0166d0c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eafca908701c410ab9488970fcb3d74e" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 45001 } } peers { permanent_uuid: "544851f9b7954fd2962675f957e8836a" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34199 } } peers { permanent_uuid: "a634c60a5ca74b26b0d1f679d0166d0c" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 39941 } }
W20260129 02:58:13.008203 19867 auto_rebalancer.cc:229] Could not retrieve cluster info: Service unavailable: Tablet not running
I20260129 02:58:13.007259 20133 leader_election.cc:290] T 5d11df549a83429a87d3e7a62fabaa21 P eafca908701c410ab9488970fcb3d74e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 544851f9b7954fd2962675f957e8836a (127.18.211.2:34199), a634c60a5ca74b26b0d1f679d0166d0c (127.18.211.3:39941)
I20260129 02:58:13.008934 19996 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5d11df549a83429a87d3e7a62fabaa21" candidate_uuid: "eafca908701c410ab9488970fcb3d74e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "544851f9b7954fd2962675f957e8836a" is_pre_election: true
I20260129 02:58:13.009639 20109 ts_tablet_manager.cc:1434] T 901f619b02f843b4a2747de956182f12 P a634c60a5ca74b26b0d1f679d0166d0c: Time spent starting tablet: real 0.047s user 0.007s sys 0.001s
I20260129 02:58:13.010671 20109 tablet_bootstrap.cc:492] T 620c6d80e1964c51a628ffb43f645798 P a634c60a5ca74b26b0d1f679d0166d0c: Bootstrap starting.
W20260129 02:58:13.011044 19884 leader_election.cc:343] T 5d11df549a83429a87d3e7a62fabaa21 P eafca908701c410ab9488970fcb3d74e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 544851f9b7954fd2962675f957e8836a (127.18.211.2:34199): Illegal state: must be running to vote when last-logged opid is not known
I20260129 02:58:13.009291 20110 consensus_queue.cc:260] T 79355f4f5d7942b6ac6f420256fd35b7 P eafca908701c410ab9488970fcb3d74e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eafca908701c410ab9488970fcb3d74e" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 45001 } } peers { permanent_uuid: "544851f9b7954fd2962675f957e8836a" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34199 } } peers { permanent_uuid: "a634c60a5ca74b26b0d1f679d0166d0c" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 39941 } }
I20260129 02:58:13.014931 20111 tablet_bootstrap.cc:654] T 620c6d80e1964c51a628ffb43f645798 P 544851f9b7954fd2962675f957e8836a: Neither blocks nor log segments found. Creating new log.
I20260129 02:58:13.018654 20116 raft_consensus.cc:515] T d5edd0a419ee46c9a6a223f8056afb4c P eafca908701c410ab9488970fcb3d74e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eafca908701c410ab9488970fcb3d74e" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 45001 } } peers { permanent_uuid: "544851f9b7954fd2962675f957e8836a" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34199 } } peers { permanent_uuid: "a634c60a5ca74b26b0d1f679d0166d0c" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 39941 } }
I20260129 02:58:13.020576 20116 leader_election.cc:290] T d5edd0a419ee46c9a6a223f8056afb4c P eafca908701c410ab9488970fcb3d74e [CANDIDATE]: Term 1 election: Requested vote from peers 544851f9b7954fd2962675f957e8836a (127.18.211.2:34199), a634c60a5ca74b26b0d1f679d0166d0c (127.18.211.3:39941)
I20260129 02:58:13.006647 20114 raft_consensus.cc:697] T 4562a703e5a44a1585b332b851bfd6f2 P a634c60a5ca74b26b0d1f679d0166d0c [term 1 LEADER]: Becoming Leader. State: Replica: a634c60a5ca74b26b0d1f679d0166d0c, State: Running, Role: LEADER
I20260129 02:58:13.021279 19996 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d5edd0a419ee46c9a6a223f8056afb4c" candidate_uuid: "eafca908701c410ab9488970fcb3d74e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "544851f9b7954fd2962675f957e8836a"
I20260129 02:58:13.021888 19996 raft_consensus.cc:3060] T d5edd0a419ee46c9a6a223f8056afb4c P 544851f9b7954fd2962675f957e8836a [term 0 FOLLOWER]: Advancing to term 1
I20260129 02:58:13.022063 20114 consensus_queue.cc:237] T 4562a703e5a44a1585b332b851bfd6f2 P a634c60a5ca74b26b0d1f679d0166d0c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eafca908701c410ab9488970fcb3d74e" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 45001 } } peers { permanent_uuid: "544851f9b7954fd2962675f957e8836a" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34199 } } peers { permanent_uuid: "a634c60a5ca74b26b0d1f679d0166d0c" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 39941 } }
I20260129 02:58:13.025645 20110 ts_tablet_manager.cc:1434] T 79355f4f5d7942b6ac6f420256fd35b7 P eafca908701c410ab9488970fcb3d74e: Time spent starting tablet: real 0.032s user 0.008s sys 0.000s
W20260129 02:58:13.028393 20033 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.211.1:45001, user_credentials={real_user=slave}} blocked reactor thread for 61119.8us
I20260129 02:58:13.030279 20110 tablet_bootstrap.cc:492] T 1dc801ccecc44ed79d85b0522e49d3e5 P eafca908701c410ab9488970fcb3d74e: Bootstrap starting.
I20260129 02:58:13.032136 20070 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5d11df549a83429a87d3e7a62fabaa21" candidate_uuid: "eafca908701c410ab9488970fcb3d74e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a634c60a5ca74b26b0d1f679d0166d0c" is_pre_election: true
W20260129 02:58:13.039942 19885 leader_election.cc:343] T 5d11df549a83429a87d3e7a62fabaa21 P eafca908701c410ab9488970fcb3d74e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a634c60a5ca74b26b0d1f679d0166d0c (127.18.211.3:39941): Illegal state: must be running to vote when last-logged opid is not known
I20260129 02:58:13.035735 19996 raft_consensus.cc:2468] T d5edd0a419ee46c9a6a223f8056afb4c P 544851f9b7954fd2962675f957e8836a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eafca908701c410ab9488970fcb3d74e in term 1.
I20260129 02:58:13.040290 20070 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d5edd0a419ee46c9a6a223f8056afb4c" candidate_uuid: "eafca908701c410ab9488970fcb3d74e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a634c60a5ca74b26b0d1f679d0166d0c"
I20260129 02:58:13.040578 19885 leader_election.cc:304] T 5d11df549a83429a87d3e7a62fabaa21 P eafca908701c410ab9488970fcb3d74e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: eafca908701c410ab9488970fcb3d74e; no voters: 544851f9b7954fd2962675f957e8836a, a634c60a5ca74b26b0d1f679d0166d0c
I20260129 02:58:13.046970 20110 tablet_bootstrap.cc:654] T 1dc801ccecc44ed79d85b0522e49d3e5 P eafca908701c410ab9488970fcb3d74e: Neither blocks nor log segments found. Creating new log.
I20260129 02:58:13.047487 20133 raft_consensus.cc:2749] T 5d11df549a83429a87d3e7a62fabaa21 P eafca908701c410ab9488970fcb3d74e [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260129 02:58:13.047953 19819 catalog_manager.cc:5654] T 4562a703e5a44a1585b332b851bfd6f2 P a634c60a5ca74b26b0d1f679d0166d0c reported cstate change: term changed from 0 to 1, leader changed from <none> to a634c60a5ca74b26b0d1f679d0166d0c (127.18.211.3). New cstate: current_term: 1 leader_uuid: "a634c60a5ca74b26b0d1f679d0166d0c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eafca908701c410ab9488970fcb3d74e" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 45001 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "544851f9b7954fd2962675f957e8836a" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34199 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a634c60a5ca74b26b0d1f679d0166d0c" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 39941 } health_report { overall_health: HEALTHY } } }
I20260129 02:58:13.049538 19921 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7d91c3094df94b47aa9a512d13490daa" candidate_uuid: "a634c60a5ca74b26b0d1f679d0166d0c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eafca908701c410ab9488970fcb3d74e" is_pre_election: true
I20260129 02:58:13.050323 19921 raft_consensus.cc:2468] T 7d91c3094df94b47aa9a512d13490daa P eafca908701c410ab9488970fcb3d74e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a634c60a5ca74b26b0d1f679d0166d0c in term 0.
I20260129 02:58:13.053316 20033 leader_election.cc:304] T 7d91c3094df94b47aa9a512d13490daa P a634c60a5ca74b26b0d1f679d0166d0c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a634c60a5ca74b26b0d1f679d0166d0c, eafca908701c410ab9488970fcb3d74e; no voters:
I20260129 02:58:13.054701 20109 tablet_bootstrap.cc:654] T 620c6d80e1964c51a628ffb43f645798 P a634c60a5ca74b26b0d1f679d0166d0c: Neither blocks nor log segments found. Creating new log.
I20260129 02:58:13.054740 20131 raft_consensus.cc:2804] T 7d91c3094df94b47aa9a512d13490daa P a634c60a5ca74b26b0d1f679d0166d0c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260129 02:58:13.055564 20131 raft_consensus.cc:493] T 7d91c3094df94b47aa9a512d13490daa P a634c60a5ca74b26b0d1f679d0166d0c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260129 02:58:13.056119 20131 raft_consensus.cc:3060] T 7d91c3094df94b47aa9a512d13490daa P a634c60a5ca74b26b0d1f679d0166d0c [term 0 FOLLOWER]: Advancing to term 1
W20260129 02:58:13.053529 19885 leader_election.cc:343] T d5edd0a419ee46c9a6a223f8056afb4c P eafca908701c410ab9488970fcb3d74e [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer a634c60a5ca74b26b0d1f679d0166d0c (127.18.211.3:39941): Illegal state: must be running to vote when last-logged opid is not known
I20260129 02:58:13.062531 20117 raft_consensus.cc:493] T 1dc801ccecc44ed79d85b0522e49d3e5 P 544851f9b7954fd2962675f957e8836a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260129 02:58:13.063215 20117 raft_consensus.cc:515] T 1dc801ccecc44ed79d85b0522e49d3e5 P 544851f9b7954fd2962675f957e8836a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eafca908701c410ab9488970fcb3d74e" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 45001 } } peers { permanent_uuid: "544851f9b7954fd2962675f957e8836a" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34199 } } peers { permanent_uuid: "a634c60a5ca74b26b0d1f679d0166d0c" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 39941 } }
I20260129 02:58:13.066331 20117 leader_election.cc:290] T 1dc801ccecc44ed79d85b0522e49d3e5 P 544851f9b7954fd2962675f957e8836a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers eafca908701c410ab9488970fcb3d74e (127.18.211.1:45001), a634c60a5ca74b26b0d1f679d0166d0c (127.18.211.3:39941)
I20260129 02:58:13.068137 19996 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4562a703e5a44a1585b332b851bfd6f2" candidate_uuid: "a634c60a5ca74b26b0d1f679d0166d0c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "544851f9b7954fd2962675f957e8836a"
I20260129 02:58:13.070096 19996 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7d91c3094df94b47aa9a512d13490daa" candidate_uuid: "a634c60a5ca74b26b0d1f679d0166d0c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "544851f9b7954fd2962675f957e8836a" is_pre_election: true
I20260129 02:58:13.070902 19996 raft_consensus.cc:2468] T 7d91c3094df94b47aa9a512d13490daa P 544851f9b7954fd2962675f957e8836a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a634c60a5ca74b26b0d1f679d0166d0c in term 0.
W20260129 02:58:13.072503 20033 leader_election.cc:343] T 4562a703e5a44a1585b332b851bfd6f2 P a634c60a5ca74b26b0d1f679d0166d0c [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 544851f9b7954fd2962675f957e8836a (127.18.211.2:34199): Illegal state: must be running to vote when last-logged opid is not known
I20260129 02:58:13.077822 20110 tablet_bootstrap.cc:492] T 1dc801ccecc44ed79d85b0522e49d3e5 P eafca908701c410ab9488970fcb3d74e: No bootstrap required, opened a new log
I20260129 02:58:13.078274 20110 ts_tablet_manager.cc:1403] T 1dc801ccecc44ed79d85b0522e49d3e5 P eafca908701c410ab9488970fcb3d74e: Time spent bootstrapping tablet: real 0.048s user 0.006s sys 0.017s
I20260129 02:58:13.080622 20110 raft_consensus.cc:359] T 1dc801ccecc44ed79d85b0522e49d3e5 P eafca908701c410ab9488970fcb3d74e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eafca908701c410ab9488970fcb3d74e" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 45001 } } peers { permanent_uuid: "544851f9b7954fd2962675f957e8836a" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34199 } } peers { permanent_uuid: "a634c60a5ca74b26b0d1f679d0166d0c" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 39941 } }
I20260129 02:58:13.081334 20110 raft_consensus.cc:385] T 1dc801ccecc44ed79d85b0522e49d3e5 P eafca908701c410ab9488970fcb3d74e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260129 02:58:13.081676 20110 raft_consensus.cc:740] T 1dc801ccecc44ed79d85b0522e49d3e5 P eafca908701c410ab9488970fcb3d74e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eafca908701c410ab9488970fcb3d74e, State: Initialized, Role: FOLLOWER
I20260129 02:58:13.081961 19884 leader_election.cc:304] T d5edd0a419ee46c9a6a223f8056afb4c P eafca908701c410ab9488970fcb3d74e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 544851f9b7954fd2962675f957e8836a, eafca908701c410ab9488970fcb3d74e; no voters: a634c60a5ca74b26b0d1f679d0166d0c
I20260129 02:58:13.077183 20131 raft_consensus.cc:515] T 7d91c3094df94b47aa9a512d13490daa P a634c60a5ca74b26b0d1f679d0166d0c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a634c60a5ca74b26b0d1f679d0166d0c" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 39941 } } peers { permanent_uuid: "eafca908701c410ab9488970fcb3d74e" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 45001 } } peers { permanent_uuid: "544851f9b7954fd2962675f957e8836a" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34199 } }
I20260129 02:58:13.083060 20133 raft_consensus.cc:2804] T d5edd0a419ee46c9a6a223f8056afb4c P eafca908701c410ab9488970fcb3d74e [term 1 FOLLOWER]: Leader election won for term 1
I20260129 02:58:13.083551 20133 raft_consensus.cc:697] T d5edd0a419ee46c9a6a223f8056afb4c P eafca908701c410ab9488970fcb3d74e [term 1 LEADER]: Becoming Leader. State: Replica: eafca908701c410ab9488970fcb3d74e, State: Running, Role: LEADER
I20260129 02:58:13.084362 20133 consensus_queue.cc:237] T d5edd0a419ee46c9a6a223f8056afb4c P eafca908701c410ab9488970fcb3d74e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eafca908701c410ab9488970fcb3d74e" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 45001 } } peers { permanent_uuid: "544851f9b7954fd2962675f957e8836a" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34199 } } peers { permanent_uuid: "a634c60a5ca74b26b0d1f679d0166d0c" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 39941 } }
I20260129 02:58:13.084767 20131 leader_election.cc:290] T 7d91c3094df94b47aa9a512d13490daa P a634c60a5ca74b26b0d1f679d0166d0c [CANDIDATE]: Term 1 election: Requested vote from peers eafca908701c410ab9488970fcb3d74e (127.18.211.1:45001), 544851f9b7954fd2962675f957e8836a (127.18.211.2:34199)
I20260129 02:58:13.088027 19996 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7d91c3094df94b47aa9a512d13490daa" candidate_uuid: "a634c60a5ca74b26b0d1f679d0166d0c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "544851f9b7954fd2962675f957e8836a"
I20260129 02:58:13.088378 19921 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7d91c3094df94b47aa9a512d13490daa" candidate_uuid: "a634c60a5ca74b26b0d1f679d0166d0c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eafca908701c410ab9488970fcb3d74e"
I20260129 02:58:13.088603 19996 raft_consensus.cc:3060] T 7d91c3094df94b47aa9a512d13490daa P 544851f9b7954fd2962675f957e8836a [term 0 FOLLOWER]: Advancing to term 1
I20260129 02:58:13.082383 20110 consensus_queue.cc:260] T 1dc801ccecc44ed79d85b0522e49d3e5 P eafca908701c410ab9488970fcb3d74e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eafca908701c410ab9488970fcb3d74e" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 45001 } } peers { permanent_uuid: "544851f9b7954fd2962675f957e8836a" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34199 } } peers { permanent_uuid: "a634c60a5ca74b26b0d1f679d0166d0c" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 39941 } }
I20260129 02:58:13.088941 19921 raft_consensus.cc:3060] T 7d91c3094df94b47aa9a512d13490daa P eafca908701c410ab9488970fcb3d74e [term 0 FOLLOWER]: Advancing to term 1
I20260129 02:58:13.091084 20110 ts_tablet_manager.cc:1434] T 1dc801ccecc44ed79d85b0522e49d3e5 P eafca908701c410ab9488970fcb3d74e: Time spent starting tablet: real 0.013s user 0.005s sys 0.001s
I20260129 02:58:13.092077 20110 tablet_bootstrap.cc:492] T 901f619b02f843b4a2747de956182f12 P eafca908701c410ab9488970fcb3d74e: Bootstrap starting.
I20260129 02:58:13.094362 19920 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "47ad34405a6f4d508c9613c07a572cc3" candidate_uuid: "544851f9b7954fd2962675f957e8836a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eafca908701c410ab9488970fcb3d74e" is_pre_election: true
I20260129 02:58:13.095217 19919 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1dc801ccecc44ed79d85b0522e49d3e5" candidate_uuid: "544851f9b7954fd2962675f957e8836a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eafca908701c410ab9488970fcb3d74e" is_pre_election: true
W20260129 02:58:13.095850 19959 leader_election.cc:343] T 47ad34405a6f4d508c9613c07a572cc3 P 544851f9b7954fd2962675f957e8836a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer eafca908701c410ab9488970fcb3d74e (127.18.211.1:45001): Illegal state: must be running to vote when last-logged opid is not known
I20260129 02:58:13.095903 19919 raft_consensus.cc:2468] T 1dc801ccecc44ed79d85b0522e49d3e5 P eafca908701c410ab9488970fcb3d74e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 544851f9b7954fd2962675f957e8836a in term 0.
I20260129 02:58:13.096931 20111 tablet_bootstrap.cc:492] T 620c6d80e1964c51a628ffb43f645798 P 544851f9b7954fd2962675f957e8836a: No bootstrap required, opened a new log
I20260129 02:58:13.097385 20111 ts_tablet_manager.cc:1403] T 620c6d80e1964c51a628ffb43f645798 P 544851f9b7954fd2962675f957e8836a: Time spent bootstrapping tablet: real 0.117s user 0.012s sys 0.005s
I20260129 02:58:13.097216 19959 leader_election.cc:304] T 1dc801ccecc44ed79d85b0522e49d3e5 P 544851f9b7954fd2962675f957e8836a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 544851f9b7954fd2962675f957e8836a, eafca908701c410ab9488970fcb3d74e; no voters:
I20260129 02:58:13.098179 20110 tablet_bootstrap.cc:654] T 901f619b02f843b4a2747de956182f12 P eafca908701c410ab9488970fcb3d74e: Neither blocks nor log segments found. Creating new log.
I20260129 02:58:13.098129 20117 raft_consensus.cc:2804] T 1dc801ccecc44ed79d85b0522e49d3e5 P 544851f9b7954fd2962675f957e8836a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260129 02:58:13.098637 20117 raft_consensus.cc:493] T 1dc801ccecc44ed79d85b0522e49d3e5 P 544851f9b7954fd2962675f957e8836a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260129 02:58:13.098732 19921 raft_consensus.cc:2468] T 7d91c3094df94b47aa9a512d13490daa P eafca908701c410ab9488970fcb3d74e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a634c60a5ca74b26b0d1f679d0166d0c in term 1.
I20260129 02:58:13.099010 20117 raft_consensus.cc:3060] T 1dc801ccecc44ed79d85b0522e49d3e5 P 544851f9b7954fd2962675f957e8836a [term 0 FOLLOWER]: Advancing to term 1
I20260129 02:58:13.099994 20033 leader_election.cc:304] T 7d91c3094df94b47aa9a512d13490daa P a634c60a5ca74b26b0d1f679d0166d0c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a634c60a5ca74b26b0d1f679d0166d0c, eafca908701c410ab9488970fcb3d74e; no voters:
I20260129 02:58:13.102511 20111 raft_consensus.cc:359] T 620c6d80e1964c51a628ffb43f645798 P 544851f9b7954fd2962675f957e8836a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eafca908701c410ab9488970fcb3d74e" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 45001 } } peers { permanent_uuid: "544851f9b7954fd2962675f957e8836a" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34199 } } peers { permanent_uuid: "a634c60a5ca74b26b0d1f679d0166d0c" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 39941 } }
I20260129 02:58:13.103318 20111 raft_consensus.cc:385] T 620c6d80e1964c51a628ffb43f645798 P 544851f9b7954fd2962675f957e8836a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260129 02:58:13.101416 20131 raft_consensus.cc:2804] T 7d91c3094df94b47aa9a512d13490daa P a634c60a5ca74b26b0d1f679d0166d0c [term 1 FOLLOWER]: Leader election won for term 1
I20260129 02:58:13.104199 20131 raft_consensus.cc:697] T 7d91c3094df94b47aa9a512d13490daa P a634c60a5ca74b26b0d1f679d0166d0c [term 1 LEADER]: Becoming Leader. State: Replica: a634c60a5ca74b26b0d1f679d0166d0c, State: Running, Role: LEADER
I20260129 02:58:13.104702 20111 raft_consensus.cc:740] T 620c6d80e1964c51a628ffb43f645798 P 544851f9b7954fd2962675f957e8836a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 544851f9b7954fd2962675f957e8836a, State: Initialized, Role: FOLLOWER
I20260129 02:58:13.106346 20111 consensus_queue.cc:260] T 620c6d80e1964c51a628ffb43f645798 P 544851f9b7954fd2962675f957e8836a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eafca908701c410ab9488970fcb3d74e" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 45001 } } peers { permanent_uuid: "544851f9b7954fd2962675f957e8836a" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34199 } } peers { permanent_uuid: "a634c60a5ca74b26b0d1f679d0166d0c" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 39941 } }
I20260129 02:58:13.104229 20109 tablet_bootstrap.cc:492] T 620c6d80e1964c51a628ffb43f645798 P a634c60a5ca74b26b0d1f679d0166d0c: No bootstrap required, opened a new log
I20260129 02:58:13.111099 20109 ts_tablet_manager.cc:1403] T 620c6d80e1964c51a628ffb43f645798 P a634c60a5ca74b26b0d1f679d0166d0c: Time spent bootstrapping tablet: real 0.101s user 0.014s sys 0.016s
I20260129 02:58:13.108357 20117 raft_consensus.cc:515] T 1dc801ccecc44ed79d85b0522e49d3e5 P 544851f9b7954fd2962675f957e8836a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eafca908701c410ab9488970fcb3d74e" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 45001 } } peers { permanent_uuid: "544851f9b7954fd2962675f957e8836a" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34199 } } peers { permanent_uuid: "a634c60a5ca74b26b0d1f679d0166d0c" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 39941 } }
I20260129 02:58:13.111747 20070 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "47ad34405a6f4d508c9613c07a572cc3" candidate_uuid: "544851f9b7954fd2962675f957e8836a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a634c60a5ca74b26b0d1f679d0166d0c" is_pre_election: true
I20260129 02:58:13.108568 19996 raft_consensus.cc:2468] T 7d91c3094df94b47aa9a512d13490daa P 544851f9b7954fd2962675f957e8836a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a634c60a5ca74b26b0d1f679d0166d0c in term 1.
I20260129 02:58:13.111721 20131 consensus_queue.cc:237] T 7d91c3094df94b47aa9a512d13490daa P a634c60a5ca74b26b0d1f679d0166d0c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a634c60a5ca74b26b0d1f679d0166d0c" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 39941 } } peers { permanent_uuid: "eafca908701c410ab9488970fcb3d74e" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 45001 } } peers { permanent_uuid: "544851f9b7954fd2962675f957e8836a" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34199 } }
I20260129 02:58:13.114462 19921 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1dc801ccecc44ed79d85b0522e49d3e5" candidate_uuid: "544851f9b7954fd2962675f957e8836a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eafca908701c410ab9488970fcb3d74e"
I20260129 02:58:13.114810 20069 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1dc801ccecc44ed79d85b0522e49d3e5" candidate_uuid: "544851f9b7954fd2962675f957e8836a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a634c60a5ca74b26b0d1f679d0166d0c" is_pre_election: true
I20260129 02:58:13.115231 19921 raft_consensus.cc:3060] T 1dc801ccecc44ed79d85b0522e49d3e5 P eafca908701c410ab9488970fcb3d74e [term 0 FOLLOWER]: Advancing to term 1
W20260129 02:58:13.116338 19960 leader_election.cc:343] T 47ad34405a6f4d508c9613c07a572cc3 P 544851f9b7954fd2962675f957e8836a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a634c60a5ca74b26b0d1f679d0166d0c (127.18.211.3:39941): Illegal state: must be running to vote when last-logged opid is not known
I20260129 02:58:13.116822 19960 leader_election.cc:304] T 47ad34405a6f4d508c9613c07a572cc3 P 544851f9b7954fd2962675f957e8836a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 544851f9b7954fd2962675f957e8836a; no voters: a634c60a5ca74b26b0d1f679d0166d0c, eafca908701c410ab9488970fcb3d74e
I20260129 02:58:13.118309 20070 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1dc801ccecc44ed79d85b0522e49d3e5" candidate_uuid: "544851f9b7954fd2962675f957e8836a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a634c60a5ca74b26b0d1f679d0166d0c"
W20260129 02:58:13.122023 19960 leader_election.cc:343] T 1dc801ccecc44ed79d85b0522e49d3e5 P 544851f9b7954fd2962675f957e8836a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a634c60a5ca74b26b0d1f679d0166d0c (127.18.211.3:39941): Illegal state: must be running to vote when last-logged opid is not known
I20260129 02:58:13.122150 20144 raft_consensus.cc:2749] T 47ad34405a6f4d508c9613c07a572cc3 P 544851f9b7954fd2962675f957e8836a [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20260129 02:58:13.122963 19960 leader_election.cc:343] T 1dc801ccecc44ed79d85b0522e49d3e5 P 544851f9b7954fd2962675f957e8836a [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer a634c60a5ca74b26b0d1f679d0166d0c (127.18.211.3:39941): Illegal state: must be running to vote when last-logged opid is not known
I20260129 02:58:13.120147 20117 leader_election.cc:290] T 1dc801ccecc44ed79d85b0522e49d3e5 P 544851f9b7954fd2962675f957e8836a [CANDIDATE]: Term 1 election: Requested vote from peers eafca908701c410ab9488970fcb3d74e (127.18.211.1:45001), a634c60a5ca74b26b0d1f679d0166d0c (127.18.211.3:39941)
I20260129 02:58:13.131825 20110 tablet_bootstrap.cc:492] T 901f619b02f843b4a2747de956182f12 P eafca908701c410ab9488970fcb3d74e: No bootstrap required, opened a new log
I20260129 02:58:13.132503 20110 ts_tablet_manager.cc:1403] T 901f619b02f843b4a2747de956182f12 P eafca908701c410ab9488970fcb3d74e: Time spent bootstrapping tablet: real 0.041s user 0.015s sys 0.000s
I20260129 02:58:13.134606 20109 raft_consensus.cc:359] T 620c6d80e1964c51a628ffb43f645798 P a634c60a5ca74b26b0d1f679d0166d0c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eafca908701c410ab9488970fcb3d74e" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 45001 } } peers { permanent_uuid: "544851f9b7954fd2962675f957e8836a" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34199 } } peers { permanent_uuid: "a634c60a5ca74b26b0d1f679d0166d0c" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 39941 } }
I20260129 02:58:13.135564 20109 raft_consensus.cc:385] T 620c6d80e1964c51a628ffb43f645798 P a634c60a5ca74b26b0d1f679d0166d0c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260129 02:58:13.139087 20110 raft_consensus.cc:359] T 901f619b02f843b4a2747de956182f12 P eafca908701c410ab9488970fcb3d74e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eafca908701c410ab9488970fcb3d74e" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 45001 } } peers { permanent_uuid: "544851f9b7954fd2962675f957e8836a" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34199 } } peers { permanent_uuid: "a634c60a5ca74b26b0d1f679d0166d0c" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 39941 } }
I20260129 02:58:13.139645 20109 raft_consensus.cc:740] T 620c6d80e1964c51a628ffb43f645798 P a634c60a5ca74b26b0d1f679d0166d0c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a634c60a5ca74b26b0d1f679d0166d0c, State: Initialized, Role: FOLLOWER
I20260129 02:58:13.141693 20110 raft_consensus.cc:385] T 901f619b02f843b4a2747de956182f12 P eafca908701c410ab9488970fcb3d74e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260129 02:58:13.142338 20110 raft_consensus.cc:740] T 901f619b02f843b4a2747de956182f12 P eafca908701c410ab9488970fcb3d74e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eafca908701c410ab9488970fcb3d74e, State: Initialized, Role: FOLLOWER
I20260129 02:58:13.141733 19817 catalog_manager.cc:5654] T d5edd0a419ee46c9a6a223f8056afb4c P eafca908701c410ab9488970fcb3d74e reported cstate change: term changed from 0 to 1, leader changed from <none> to eafca908701c410ab9488970fcb3d74e (127.18.211.1). New cstate: current_term: 1 leader_uuid: "eafca908701c410ab9488970fcb3d74e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eafca908701c410ab9488970fcb3d74e" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 45001 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "544851f9b7954fd2962675f957e8836a" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34199 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a634c60a5ca74b26b0d1f679d0166d0c" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 39941 } health_report { overall_health: UNKNOWN } } }
I20260129 02:58:13.140931 19921 raft_consensus.cc:2468] T 1dc801ccecc44ed79d85b0522e49d3e5 P eafca908701c410ab9488970fcb3d74e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 544851f9b7954fd2962675f957e8836a in term 1.
I20260129 02:58:13.142814 20109 consensus_queue.cc:260] T 620c6d80e1964c51a628ffb43f645798 P a634c60a5ca74b26b0d1f679d0166d0c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eafca908701c410ab9488970fcb3d74e" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 45001 } } peers { permanent_uuid: "544851f9b7954fd2962675f957e8836a" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34199 } } peers { permanent_uuid: "a634c60a5ca74b26b0d1f679d0166d0c" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 39941 } }
I20260129 02:58:13.144924 20111 ts_tablet_manager.cc:1434] T 620c6d80e1964c51a628ffb43f645798 P 544851f9b7954fd2962675f957e8836a: Time spent starting tablet: real 0.047s user 0.004s sys 0.033s
I20260129 02:58:13.145188 19959 leader_election.cc:304] T 1dc801ccecc44ed79d85b0522e49d3e5 P 544851f9b7954fd2962675f957e8836a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 544851f9b7954fd2962675f957e8836a, eafca908701c410ab9488970fcb3d74e; no voters: a634c60a5ca74b26b0d1f679d0166d0c
I20260129 02:58:13.146221 20109 ts_tablet_manager.cc:1434] T 620c6d80e1964c51a628ffb43f645798 P a634c60a5ca74b26b0d1f679d0166d0c: Time spent starting tablet: real 0.035s user 0.006s sys 0.000s
I20260129 02:58:13.147034 20143 raft_consensus.cc:2804] T 1dc801ccecc44ed79d85b0522e49d3e5 P 544851f9b7954fd2962675f957e8836a [term 1 FOLLOWER]: Leader election won for term 1
I20260129 02:58:13.147255 20109 tablet_bootstrap.cc:492] T 1bd5a7f0c64f4ef6bc2bb544cd69843c P a634c60a5ca74b26b0d1f679d0166d0c: Bootstrap starting.
I20260129 02:58:13.143230 20110 consensus_queue.cc:260] T 901f619b02f843b4a2747de956182f12 P eafca908701c410ab9488970fcb3d74e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eafca908701c410ab9488970fcb3d74e" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 45001 } } peers { permanent_uuid: "544851f9b7954fd2962675f957e8836a" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34199 } } peers { permanent_uuid: "a634c60a5ca74b26b0d1f679d0166d0c" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 39941 } }
I20260129 02:58:13.148739 20143 raft_consensus.cc:697] T 1dc801ccecc44ed79d85b0522e49d3e5 P 544851f9b7954fd2962675f957e8836a [term 1 LEADER]: Becoming Leader. State: Replica: 544851f9b7954fd2962675f957e8836a, State: Running, Role: LEADER
I20260129 02:58:13.149981 20143 consensus_queue.cc:237] T 1dc801ccecc44ed79d85b0522e49d3e5 P 544851f9b7954fd2962675f957e8836a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eafca908701c410ab9488970fcb3d74e" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 45001 } } peers { permanent_uuid: "544851f9b7954fd2962675f957e8836a" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34199 } } peers { permanent_uuid: "a634c60a5ca74b26b0d1f679d0166d0c" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 39941 } }
I20260129 02:58:13.153970 20111 tablet_bootstrap.cc:492] T 901f619b02f843b4a2747de956182f12 P 544851f9b7954fd2962675f957e8836a: Bootstrap starting.
I20260129 02:58:13.154423 20110 ts_tablet_manager.cc:1434] T 901f619b02f843b4a2747de956182f12 P eafca908701c410ab9488970fcb3d74e: Time spent starting tablet: real 0.022s user 0.007s sys 0.000s
I20260129 02:58:13.155833 20109 tablet_bootstrap.cc:654] T 1bd5a7f0c64f4ef6bc2bb544cd69843c P a634c60a5ca74b26b0d1f679d0166d0c: Neither blocks nor log segments found. Creating new log.
I20260129 02:58:13.159709 20110 tablet_bootstrap.cc:492] T 620c6d80e1964c51a628ffb43f645798 P eafca908701c410ab9488970fcb3d74e: Bootstrap starting.
I20260129 02:58:13.168381 20110 tablet_bootstrap.cc:654] T 620c6d80e1964c51a628ffb43f645798 P eafca908701c410ab9488970fcb3d74e: Neither blocks nor log segments found. Creating new log.
I20260129 02:58:13.173446 20111 tablet_bootstrap.cc:654] T 901f619b02f843b4a2747de956182f12 P 544851f9b7954fd2962675f957e8836a: Neither blocks nor log segments found. Creating new log.
I20260129 02:58:13.181584 19818 catalog_manager.cc:5654] T 1dc801ccecc44ed79d85b0522e49d3e5 P 544851f9b7954fd2962675f957e8836a reported cstate change: term changed from 0 to 1, leader changed from <none> to 544851f9b7954fd2962675f957e8836a (127.18.211.2). New cstate: current_term: 1 leader_uuid: "544851f9b7954fd2962675f957e8836a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eafca908701c410ab9488970fcb3d74e" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 45001 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "544851f9b7954fd2962675f957e8836a" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34199 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a634c60a5ca74b26b0d1f679d0166d0c" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 39941 } health_report { overall_health: UNKNOWN } } }
I20260129 02:58:13.182538 19816 catalog_manager.cc:5654] T 7d91c3094df94b47aa9a512d13490daa P a634c60a5ca74b26b0d1f679d0166d0c reported cstate change: term changed from 0 to 1, leader changed from <none> to a634c60a5ca74b26b0d1f679d0166d0c (127.18.211.3). New cstate: current_term: 1 leader_uuid: "a634c60a5ca74b26b0d1f679d0166d0c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a634c60a5ca74b26b0d1f679d0166d0c" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 39941 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "eafca908701c410ab9488970fcb3d74e" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 45001 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "544851f9b7954fd2962675f957e8836a" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34199 } health_report { overall_health: UNKNOWN } } }
I20260129 02:58:13.189458 20109 tablet_bootstrap.cc:492] T 1bd5a7f0c64f4ef6bc2bb544cd69843c P a634c60a5ca74b26b0d1f679d0166d0c: No bootstrap required, opened a new log
I20260129 02:58:13.190168 20109 ts_tablet_manager.cc:1403] T 1bd5a7f0c64f4ef6bc2bb544cd69843c P a634c60a5ca74b26b0d1f679d0166d0c: Time spent bootstrapping tablet: real 0.043s user 0.006s sys 0.016s
I20260129 02:58:13.193928 20109 raft_consensus.cc:359] T 1bd5a7f0c64f4ef6bc2bb544cd69843c P a634c60a5ca74b26b0d1f679d0166d0c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eafca908701c410ab9488970fcb3d74e" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 45001 } } peers { permanent_uuid: "544851f9b7954fd2962675f957e8836a" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34199 } } peers { permanent_uuid: "a634c60a5ca74b26b0d1f679d0166d0c" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 39941 } }
I20260129 02:58:13.194905 20109 raft_consensus.cc:385] T 1bd5a7f0c64f4ef6bc2bb544cd69843c P a634c60a5ca74b26b0d1f679d0166d0c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260129 02:58:13.195271 20109 raft_consensus.cc:740] T 1bd5a7f0c64f4ef6bc2bb544cd69843c P a634c60a5ca74b26b0d1f679d0166d0c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a634c60a5ca74b26b0d1f679d0166d0c, State: Initialized, Role: FOLLOWER
I20260129 02:58:13.200485 20109 consensus_queue.cc:260] T 1bd5a7f0c64f4ef6bc2bb544cd69843c P a634c60a5ca74b26b0d1f679d0166d0c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eafca908701c410ab9488970fcb3d74e" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 45001 } } peers { permanent_uuid: "544851f9b7954fd2962675f957e8836a" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34199 } } peers { permanent_uuid: "a634c60a5ca74b26b0d1f679d0166d0c" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 39941 } }
I20260129 02:58:13.203763 20109 ts_tablet_manager.cc:1434] T 1bd5a7f0c64f4ef6bc2bb544cd69843c P a634c60a5ca74b26b0d1f679d0166d0c: Time spent starting tablet: real 0.013s user 0.008s sys 0.000s
I20260129 02:58:13.204298 20110 tablet_bootstrap.cc:492] T 620c6d80e1964c51a628ffb43f645798 P eafca908701c410ab9488970fcb3d74e: No bootstrap required, opened a new log
I20260129 02:58:13.204902 20110 ts_tablet_manager.cc:1403] T 620c6d80e1964c51a628ffb43f645798 P eafca908701c410ab9488970fcb3d74e: Time spent bootstrapping tablet: real 0.045s user 0.014s sys 0.002s
I20260129 02:58:13.205993 20109 tablet_bootstrap.cc:492] T 47ad34405a6f4d508c9613c07a572cc3 P a634c60a5ca74b26b0d1f679d0166d0c: Bootstrap starting.
I20260129 02:58:13.213279 20110 raft_consensus.cc:359] T 620c6d80e1964c51a628ffb43f645798 P eafca908701c410ab9488970fcb3d74e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eafca908701c410ab9488970fcb3d74e" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 45001 } } peers { permanent_uuid: "544851f9b7954fd2962675f957e8836a" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34199 } } peers { permanent_uuid: "a634c60a5ca74b26b0d1f679d0166d0c" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 39941 } }
I20260129 02:58:13.213966 20110 raft_consensus.cc:385] T 620c6d80e1964c51a628ffb43f645798 P eafca908701c410ab9488970fcb3d74e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260129 02:58:13.214231 20110 raft_consensus.cc:740] T 620c6d80e1964c51a628ffb43f645798 P eafca908701c410ab9488970fcb3d74e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eafca908701c410ab9488970fcb3d74e, State: Initialized, Role: FOLLOWER
I20260129 02:58:13.215049 20109 tablet_bootstrap.cc:654] T 47ad34405a6f4d508c9613c07a572cc3 P a634c60a5ca74b26b0d1f679d0166d0c: Neither blocks nor log segments found. Creating new log.
I20260129 02:58:13.215601 20110 consensus_queue.cc:260] T 620c6d80e1964c51a628ffb43f645798 P eafca908701c410ab9488970fcb3d74e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eafca908701c410ab9488970fcb3d74e" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 45001 } } peers { permanent_uuid: "544851f9b7954fd2962675f957e8836a" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34199 } } peers { permanent_uuid: "a634c60a5ca74b26b0d1f679d0166d0c" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 39941 } }
I20260129 02:58:13.218801 20133 raft_consensus.cc:493] T 79355f4f5d7942b6ac6f420256fd35b7 P eafca908701c410ab9488970fcb3d74e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260129 02:58:13.219259 20133 raft_consensus.cc:515] T 79355f4f5d7942b6ac6f420256fd35b7 P eafca908701c410ab9488970fcb3d74e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eafca908701c410ab9488970fcb3d74e" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 45001 } } peers { permanent_uuid: "544851f9b7954fd2962675f957e8836a" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34199 } } peers { permanent_uuid: "a634c60a5ca74b26b0d1f679d0166d0c" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 39941 } }
I20260129 02:58:13.221273 20133 leader_election.cc:290] T 79355f4f5d7942b6ac6f420256fd35b7 P eafca908701c410ab9488970fcb3d74e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 544851f9b7954fd2962675f957e8836a (127.18.211.2:34199), a634c60a5ca74b26b0d1f679d0166d0c (127.18.211.3:39941)
I20260129 02:58:13.222529 20111 tablet_bootstrap.cc:492] T 901f619b02f843b4a2747de956182f12 P 544851f9b7954fd2962675f957e8836a: No bootstrap required, opened a new log
I20260129 02:58:13.222573 20109 tablet_bootstrap.cc:492] T 47ad34405a6f4d508c9613c07a572cc3 P a634c60a5ca74b26b0d1f679d0166d0c: No bootstrap required, opened a new log
I20260129 02:58:13.223526 20109 ts_tablet_manager.cc:1403] T 47ad34405a6f4d508c9613c07a572cc3 P a634c60a5ca74b26b0d1f679d0166d0c: Time spent bootstrapping tablet: real 0.018s user 0.013s sys 0.002s
I20260129 02:58:13.222723 20070 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "79355f4f5d7942b6ac6f420256fd35b7" candidate_uuid: "eafca908701c410ab9488970fcb3d74e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a634c60a5ca74b26b0d1f679d0166d0c" is_pre_election: true
I20260129 02:58:13.224184 20111 ts_tablet_manager.cc:1403] T 901f619b02f843b4a2747de956182f12 P 544851f9b7954fd2962675f957e8836a: Time spent bootstrapping tablet: real 0.070s user 0.019s sys 0.035s
I20260129 02:58:13.224336 19996 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "79355f4f5d7942b6ac6f420256fd35b7" candidate_uuid: "eafca908701c410ab9488970fcb3d74e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "544851f9b7954fd2962675f957e8836a" is_pre_election: true
W20260129 02:58:13.225085 19885 leader_election.cc:343] T 79355f4f5d7942b6ac6f420256fd35b7 P eafca908701c410ab9488970fcb3d74e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a634c60a5ca74b26b0d1f679d0166d0c (127.18.211.3:39941): Illegal state: must be running to vote when last-logged opid is not known
W20260129 02:58:13.225603 19884 leader_election.cc:343] T 79355f4f5d7942b6ac6f420256fd35b7 P eafca908701c410ab9488970fcb3d74e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 544851f9b7954fd2962675f957e8836a (127.18.211.2:34199): Illegal state: must be running to vote when last-logged opid is not known
I20260129 02:58:13.226008 19884 leader_election.cc:304] T 79355f4f5d7942b6ac6f420256fd35b7 P eafca908701c410ab9488970fcb3d74e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: eafca908701c410ab9488970fcb3d74e; no voters: 544851f9b7954fd2962675f957e8836a, a634c60a5ca74b26b0d1f679d0166d0c
I20260129 02:58:13.226243 20109 raft_consensus.cc:359] T 47ad34405a6f4d508c9613c07a572cc3 P a634c60a5ca74b26b0d1f679d0166d0c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eafca908701c410ab9488970fcb3d74e" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 45001 } } peers { permanent_uuid: "544851f9b7954fd2962675f957e8836a" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34199 } } peers { permanent_uuid: "a634c60a5ca74b26b0d1f679d0166d0c" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 39941 } }
I20260129 02:58:13.226823 20110 ts_tablet_manager.cc:1434] T 620c6d80e1964c51a628ffb43f645798 P eafca908701c410ab9488970fcb3d74e: Time spent starting tablet: real 0.016s user 0.005s sys 0.000s
I20260129 02:58:13.226897 20109 raft_consensus.cc:385] T 47ad34405a6f4d508c9613c07a572cc3 P a634c60a5ca74b26b0d1f679d0166d0c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260129 02:58:13.227025 20133 raft_consensus.cc:2749] T 79355f4f5d7942b6ac6f420256fd35b7 P eafca908701c410ab9488970fcb3d74e [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260129 02:58:13.227313 20109 raft_consensus.cc:740] T 47ad34405a6f4d508c9613c07a572cc3 P a634c60a5ca74b26b0d1f679d0166d0c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a634c60a5ca74b26b0d1f679d0166d0c, State: Initialized, Role: FOLLOWER
I20260129 02:58:13.227320 20111 raft_consensus.cc:359] T 901f619b02f843b4a2747de956182f12 P 544851f9b7954fd2962675f957e8836a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eafca908701c410ab9488970fcb3d74e" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 45001 } } peers { permanent_uuid: "544851f9b7954fd2962675f957e8836a" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34199 } } peers { permanent_uuid: "a634c60a5ca74b26b0d1f679d0166d0c" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 39941 } }
I20260129 02:58:13.228279 20111 raft_consensus.cc:385] T 901f619b02f843b4a2747de956182f12 P 544851f9b7954fd2962675f957e8836a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260129 02:58:13.228107 20109 consensus_queue.cc:260] T 47ad34405a6f4d508c9613c07a572cc3 P a634c60a5ca74b26b0d1f679d0166d0c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eafca908701c410ab9488970fcb3d74e" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 45001 } } peers { permanent_uuid: "544851f9b7954fd2962675f957e8836a" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34199 } } peers { permanent_uuid: "a634c60a5ca74b26b0d1f679d0166d0c" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 39941 } }
I20260129 02:58:13.228647 20110 tablet_bootstrap.cc:492] T 1bd5a7f0c64f4ef6bc2bb544cd69843c P eafca908701c410ab9488970fcb3d74e: Bootstrap starting.
I20260129 02:58:13.228626 20111 raft_consensus.cc:740] T 901f619b02f843b4a2747de956182f12 P 544851f9b7954fd2962675f957e8836a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 544851f9b7954fd2962675f957e8836a, State: Initialized, Role: FOLLOWER
I20260129 02:58:13.229588 20111 consensus_queue.cc:260] T 901f619b02f843b4a2747de956182f12 P 544851f9b7954fd2962675f957e8836a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eafca908701c410ab9488970fcb3d74e" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 45001 } } peers { permanent_uuid: "544851f9b7954fd2962675f957e8836a" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34199 } } peers { permanent_uuid: "a634c60a5ca74b26b0d1f679d0166d0c" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 39941 } }
I20260129 02:58:13.232637 20109 ts_tablet_manager.cc:1434] T 47ad34405a6f4d508c9613c07a572cc3 P a634c60a5ca74b26b0d1f679d0166d0c: Time spent starting tablet: real 0.009s user 0.006s sys 0.000s
I20260129 02:58:13.233569 20109 tablet_bootstrap.cc:492] T 79355f4f5d7942b6ac6f420256fd35b7 P a634c60a5ca74b26b0d1f679d0166d0c: Bootstrap starting.
I20260129 02:58:13.234395 20111 ts_tablet_manager.cc:1434] T 901f619b02f843b4a2747de956182f12 P 544851f9b7954fd2962675f957e8836a: Time spent starting tablet: real 0.010s user 0.001s sys 0.005s
I20260129 02:58:13.235440 20111 tablet_bootstrap.cc:492] T 1bd5a7f0c64f4ef6bc2bb544cd69843c P 544851f9b7954fd2962675f957e8836a: Bootstrap starting.
I20260129 02:58:13.237597 20110 tablet_bootstrap.cc:654] T 1bd5a7f0c64f4ef6bc2bb544cd69843c P eafca908701c410ab9488970fcb3d74e: Neither blocks nor log segments found. Creating new log.
I20260129 02:58:13.239717 20109 tablet_bootstrap.cc:654] T 79355f4f5d7942b6ac6f420256fd35b7 P a634c60a5ca74b26b0d1f679d0166d0c: Neither blocks nor log segments found. Creating new log.
I20260129 02:58:13.242241 20111 tablet_bootstrap.cc:654] T 1bd5a7f0c64f4ef6bc2bb544cd69843c P 544851f9b7954fd2962675f957e8836a: Neither blocks nor log segments found. Creating new log.
I20260129 02:58:13.248941 20110 tablet_bootstrap.cc:492] T 1bd5a7f0c64f4ef6bc2bb544cd69843c P eafca908701c410ab9488970fcb3d74e: No bootstrap required, opened a new log
I20260129 02:58:13.249374 20111 tablet_bootstrap.cc:492] T 1bd5a7f0c64f4ef6bc2bb544cd69843c P 544851f9b7954fd2962675f957e8836a: No bootstrap required, opened a new log
I20260129 02:58:13.249481 20110 ts_tablet_manager.cc:1403] T 1bd5a7f0c64f4ef6bc2bb544cd69843c P eafca908701c410ab9488970fcb3d74e: Time spent bootstrapping tablet: real 0.021s user 0.014s sys 0.003s
I20260129 02:58:13.249861 20111 ts_tablet_manager.cc:1403] T 1bd5a7f0c64f4ef6bc2bb544cd69843c P 544851f9b7954fd2962675f957e8836a: Time spent bootstrapping tablet: real 0.015s user 0.008s sys 0.003s
I20260129 02:58:13.251932 20143 raft_consensus.cc:493] T 620c6d80e1964c51a628ffb43f645798 P 544851f9b7954fd2962675f957e8836a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260129 02:58:13.252372 20143 raft_consensus.cc:515] T 620c6d80e1964c51a628ffb43f645798 P 544851f9b7954fd2962675f957e8836a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eafca908701c410ab9488970fcb3d74e" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 45001 } } peers { permanent_uuid: "544851f9b7954fd2962675f957e8836a" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34199 } } peers { permanent_uuid: "a634c60a5ca74b26b0d1f679d0166d0c" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 39941 } }
I20260129 02:58:13.252996 20111 raft_consensus.cc:359] T 1bd5a7f0c64f4ef6bc2bb544cd69843c P 544851f9b7954fd2962675f957e8836a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eafca908701c410ab9488970fcb3d74e" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 45001 } } peers { permanent_uuid: "544851f9b7954fd2962675f957e8836a" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34199 } } peers { permanent_uuid: "a634c60a5ca74b26b0d1f679d0166d0c" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 39941 } }
I20260129 02:58:13.253724 20111 raft_consensus.cc:385] T 1bd5a7f0c64f4ef6bc2bb544cd69843c P 544851f9b7954fd2962675f957e8836a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260129 02:58:13.253997 20111 raft_consensus.cc:740] T 1bd5a7f0c64f4ef6bc2bb544cd69843c P 544851f9b7954fd2962675f957e8836a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 544851f9b7954fd2962675f957e8836a, State: Initialized, Role: FOLLOWER
I20260129 02:58:13.254449 20143 leader_election.cc:290] T 620c6d80e1964c51a628ffb43f645798 P 544851f9b7954fd2962675f957e8836a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers eafca908701c410ab9488970fcb3d74e (127.18.211.1:45001), a634c60a5ca74b26b0d1f679d0166d0c (127.18.211.3:39941)
I20260129 02:58:13.254678 20111 consensus_queue.cc:260] T 1bd5a7f0c64f4ef6bc2bb544cd69843c P 544851f9b7954fd2962675f957e8836a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eafca908701c410ab9488970fcb3d74e" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 45001 } } peers { permanent_uuid: "544851f9b7954fd2962675f957e8836a" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34199 } } peers { permanent_uuid: "a634c60a5ca74b26b0d1f679d0166d0c" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 39941 } }
I20260129 02:58:13.255849 20070 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "620c6d80e1964c51a628ffb43f645798" candidate_uuid: "544851f9b7954fd2962675f957e8836a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a634c60a5ca74b26b0d1f679d0166d0c" is_pre_election: true
I20260129 02:58:13.256254 19921 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "620c6d80e1964c51a628ffb43f645798" candidate_uuid: "544851f9b7954fd2962675f957e8836a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eafca908701c410ab9488970fcb3d74e" is_pre_election: true
I20260129 02:58:13.256671 20070 raft_consensus.cc:2468] T 620c6d80e1964c51a628ffb43f645798 P a634c60a5ca74b26b0d1f679d0166d0c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 544851f9b7954fd2962675f957e8836a in term 0.
I20260129 02:58:13.256946 20111 ts_tablet_manager.cc:1434] T 1bd5a7f0c64f4ef6bc2bb544cd69843c P 544851f9b7954fd2962675f957e8836a: Time spent starting tablet: real 0.007s user 0.005s sys 0.002s
I20260129 02:58:13.256867 19921 raft_consensus.cc:2468] T 620c6d80e1964c51a628ffb43f645798 P eafca908701c410ab9488970fcb3d74e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 544851f9b7954fd2962675f957e8836a in term 0.
I20260129 02:58:13.257936 20111 tablet_bootstrap.cc:492] T 79355f4f5d7942b6ac6f420256fd35b7 P 544851f9b7954fd2962675f957e8836a: Bootstrap starting.
I20260129 02:58:13.258232 19960 leader_election.cc:304] T 620c6d80e1964c51a628ffb43f645798 P 544851f9b7954fd2962675f957e8836a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 544851f9b7954fd2962675f957e8836a, a634c60a5ca74b26b0d1f679d0166d0c; no voters:
I20260129 02:58:13.259056 20110 raft_consensus.cc:359] T 1bd5a7f0c64f4ef6bc2bb544cd69843c P eafca908701c410ab9488970fcb3d74e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eafca908701c410ab9488970fcb3d74e" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 45001 } } peers { permanent_uuid: "544851f9b7954fd2962675f957e8836a" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34199 } } peers { permanent_uuid: "a634c60a5ca74b26b0d1f679d0166d0c" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 39941 } }
I20260129 02:58:13.259778 20110 raft_consensus.cc:385] T 1bd5a7f0c64f4ef6bc2bb544cd69843c P eafca908701c410ab9488970fcb3d74e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260129 02:58:13.260506 20110 raft_consensus.cc:740] T 1bd5a7f0c64f4ef6bc2bb544cd69843c P eafca908701c410ab9488970fcb3d74e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eafca908701c410ab9488970fcb3d74e, State: Initialized, Role: FOLLOWER
I20260129 02:58:13.261292 20110 consensus_queue.cc:260] T 1bd5a7f0c64f4ef6bc2bb544cd69843c P eafca908701c410ab9488970fcb3d74e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eafca908701c410ab9488970fcb3d74e" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 45001 } } peers { permanent_uuid: "544851f9b7954fd2962675f957e8836a" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34199 } } peers { permanent_uuid: "a634c60a5ca74b26b0d1f679d0166d0c" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 39941 } }
I20260129 02:58:13.265403 20111 tablet_bootstrap.cc:654] T 79355f4f5d7942b6ac6f420256fd35b7 P 544851f9b7954fd2962675f957e8836a: Neither blocks nor log segments found. Creating new log.
I20260129 02:58:13.266462 20109 tablet_bootstrap.cc:492] T 79355f4f5d7942b6ac6f420256fd35b7 P a634c60a5ca74b26b0d1f679d0166d0c: No bootstrap required, opened a new log
I20260129 02:58:13.266893 20109 ts_tablet_manager.cc:1403] T 79355f4f5d7942b6ac6f420256fd35b7 P a634c60a5ca74b26b0d1f679d0166d0c: Time spent bootstrapping tablet: real 0.034s user 0.017s sys 0.014s
I20260129 02:58:13.270824 20109 raft_consensus.cc:359] T 79355f4f5d7942b6ac6f420256fd35b7 P a634c60a5ca74b26b0d1f679d0166d0c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eafca908701c410ab9488970fcb3d74e" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 45001 } } peers { permanent_uuid: "544851f9b7954fd2962675f957e8836a" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34199 } } peers { permanent_uuid: "a634c60a5ca74b26b0d1f679d0166d0c" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 39941 } }
I20260129 02:58:13.265432 20143 raft_consensus.cc:2804] T 620c6d80e1964c51a628ffb43f645798 P 544851f9b7954fd2962675f957e8836a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260129 02:58:13.271517 20109 raft_consensus.cc:385] T 79355f4f5d7942b6ac6f420256fd35b7 P a634c60a5ca74b26b0d1f679d0166d0c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260129 02:58:13.271756 20143 raft_consensus.cc:493] T 620c6d80e1964c51a628ffb43f645798 P 544851f9b7954fd2962675f957e8836a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260129 02:58:13.272137 20143 raft_consensus.cc:3060] T 620c6d80e1964c51a628ffb43f645798 P 544851f9b7954fd2962675f957e8836a [term 0 FOLLOWER]: Advancing to term 1
I20260129 02:58:13.271804 20109 raft_consensus.cc:740] T 79355f4f5d7942b6ac6f420256fd35b7 P a634c60a5ca74b26b0d1f679d0166d0c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a634c60a5ca74b26b0d1f679d0166d0c, State: Initialized, Role: FOLLOWER
I20260129 02:58:13.272995 20110 ts_tablet_manager.cc:1434] T 1bd5a7f0c64f4ef6bc2bb544cd69843c P eafca908701c410ab9488970fcb3d74e: Time spent starting tablet: real 0.023s user 0.006s sys 0.000s
I20260129 02:58:13.273090 20109 consensus_queue.cc:260] T 79355f4f5d7942b6ac6f420256fd35b7 P a634c60a5ca74b26b0d1f679d0166d0c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eafca908701c410ab9488970fcb3d74e" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 45001 } } peers { permanent_uuid: "544851f9b7954fd2962675f957e8836a" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34199 } } peers { permanent_uuid: "a634c60a5ca74b26b0d1f679d0166d0c" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 39941 } }
I20260129 02:58:13.279446 20109 ts_tablet_manager.cc:1434] T 79355f4f5d7942b6ac6f420256fd35b7 P a634c60a5ca74b26b0d1f679d0166d0c: Time spent starting tablet: real 0.012s user 0.008s sys 0.001s
I20260129 02:58:13.280421 20109 tablet_bootstrap.cc:492] T d5edd0a419ee46c9a6a223f8056afb4c P a634c60a5ca74b26b0d1f679d0166d0c: Bootstrap starting.
I20260129 02:58:13.282181 20143 raft_consensus.cc:515] T 620c6d80e1964c51a628ffb43f645798 P 544851f9b7954fd2962675f957e8836a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eafca908701c410ab9488970fcb3d74e" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 45001 } } peers { permanent_uuid: "544851f9b7954fd2962675f957e8836a" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34199 } } peers { permanent_uuid: "a634c60a5ca74b26b0d1f679d0166d0c" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 39941 } }
I20260129 02:58:13.285005 19921 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "620c6d80e1964c51a628ffb43f645798" candidate_uuid: "544851f9b7954fd2962675f957e8836a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eafca908701c410ab9488970fcb3d74e"
I20260129 02:58:13.285598 19921 raft_consensus.cc:3060] T 620c6d80e1964c51a628ffb43f645798 P eafca908701c410ab9488970fcb3d74e [term 0 FOLLOWER]: Advancing to term 1
I20260129 02:58:13.286620 20109 tablet_bootstrap.cc:654] T d5edd0a419ee46c9a6a223f8056afb4c P a634c60a5ca74b26b0d1f679d0166d0c: Neither blocks nor log segments found. Creating new log.
I20260129 02:58:13.289006 20110 tablet_bootstrap.cc:492] T 47ad34405a6f4d508c9613c07a572cc3 P eafca908701c410ab9488970fcb3d74e: Bootstrap starting.
I20260129 02:58:13.291493 20111 tablet_bootstrap.cc:492] T 79355f4f5d7942b6ac6f420256fd35b7 P 544851f9b7954fd2962675f957e8836a: No bootstrap required, opened a new log
I20260129 02:58:13.291960 20111 ts_tablet_manager.cc:1403] T 79355f4f5d7942b6ac6f420256fd35b7 P 544851f9b7954fd2962675f957e8836a: Time spent bootstrapping tablet: real 0.034s user 0.025s sys 0.007s
I20260129 02:58:13.294534 20111 raft_consensus.cc:359] T 79355f4f5d7942b6ac6f420256fd35b7 P 544851f9b7954fd2962675f957e8836a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eafca908701c410ab9488970fcb3d74e" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 45001 } } peers { permanent_uuid: "544851f9b7954fd2962675f957e8836a" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34199 } } peers { permanent_uuid: "a634c60a5ca74b26b0d1f679d0166d0c" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 39941 } }
I20260129 02:58:13.295142 20111 raft_consensus.cc:385] T 79355f4f5d7942b6ac6f420256fd35b7 P 544851f9b7954fd2962675f957e8836a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260129 02:58:13.295410 20111 raft_consensus.cc:740] T 79355f4f5d7942b6ac6f420256fd35b7 P 544851f9b7954fd2962675f957e8836a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 544851f9b7954fd2962675f957e8836a, State: Initialized, Role: FOLLOWER
I20260129 02:58:13.295605 20110 tablet_bootstrap.cc:654] T 47ad34405a6f4d508c9613c07a572cc3 P eafca908701c410ab9488970fcb3d74e: Neither blocks nor log segments found. Creating new log.
I20260129 02:58:13.296077 20111 consensus_queue.cc:260] T 79355f4f5d7942b6ac6f420256fd35b7 P 544851f9b7954fd2962675f957e8836a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eafca908701c410ab9488970fcb3d74e" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 45001 } } peers { permanent_uuid: "544851f9b7954fd2962675f957e8836a" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34199 } } peers { permanent_uuid: "a634c60a5ca74b26b0d1f679d0166d0c" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 39941 } }
I20260129 02:58:13.298519 20111 ts_tablet_manager.cc:1434] T 79355f4f5d7942b6ac6f420256fd35b7 P 544851f9b7954fd2962675f957e8836a: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20260129 02:58:13.299074 19921 raft_consensus.cc:2468] T 620c6d80e1964c51a628ffb43f645798 P eafca908701c410ab9488970fcb3d74e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 544851f9b7954fd2962675f957e8836a in term 1.
I20260129 02:58:13.299678 20111 tablet_bootstrap.cc:492] T 4562a703e5a44a1585b332b851bfd6f2 P 544851f9b7954fd2962675f957e8836a: Bootstrap starting.
I20260129 02:58:13.300184 19959 leader_election.cc:304] T 620c6d80e1964c51a628ffb43f645798 P 544851f9b7954fd2962675f957e8836a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 544851f9b7954fd2962675f957e8836a, eafca908701c410ab9488970fcb3d74e; no voters:
I20260129 02:58:13.301025 20117 raft_consensus.cc:2804] T 620c6d80e1964c51a628ffb43f645798 P 544851f9b7954fd2962675f957e8836a [term 1 FOLLOWER]: Leader election won for term 1
I20260129 02:58:13.301492 20117 raft_consensus.cc:697] T 620c6d80e1964c51a628ffb43f645798 P 544851f9b7954fd2962675f957e8836a [term 1 LEADER]: Becoming Leader. State: Replica: 544851f9b7954fd2962675f957e8836a, State: Running, Role: LEADER
I20260129 02:58:13.302333 20117 consensus_queue.cc:237] T 620c6d80e1964c51a628ffb43f645798 P 544851f9b7954fd2962675f957e8836a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eafca908701c410ab9488970fcb3d74e" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 45001 } } peers { permanent_uuid: "544851f9b7954fd2962675f957e8836a" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34199 } } peers { permanent_uuid: "a634c60a5ca74b26b0d1f679d0166d0c" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 39941 } }
I20260129 02:58:13.306922 20111 tablet_bootstrap.cc:654] T 4562a703e5a44a1585b332b851bfd6f2 P 544851f9b7954fd2962675f957e8836a: Neither blocks nor log segments found. Creating new log.
I20260129 02:58:13.307194 20110 tablet_bootstrap.cc:492] T 47ad34405a6f4d508c9613c07a572cc3 P eafca908701c410ab9488970fcb3d74e: No bootstrap required, opened a new log
I20260129 02:58:13.307709 20110 ts_tablet_manager.cc:1403] T 47ad34405a6f4d508c9613c07a572cc3 P eafca908701c410ab9488970fcb3d74e: Time spent bootstrapping tablet: real 0.019s user 0.007s sys 0.009s
I20260129 02:58:13.310644 20070 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "620c6d80e1964c51a628ffb43f645798" candidate_uuid: "544851f9b7954fd2962675f957e8836a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a634c60a5ca74b26b0d1f679d0166d0c"
I20260129 02:58:13.310479 20110 raft_consensus.cc:359] T 47ad34405a6f4d508c9613c07a572cc3 P eafca908701c410ab9488970fcb3d74e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eafca908701c410ab9488970fcb3d74e" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 45001 } } peers { permanent_uuid: "544851f9b7954fd2962675f957e8836a" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34199 } } peers { permanent_uuid: "a634c60a5ca74b26b0d1f679d0166d0c" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 39941 } }
I20260129 02:58:13.311317 20070 raft_consensus.cc:3060] T 620c6d80e1964c51a628ffb43f645798 P a634c60a5ca74b26b0d1f679d0166d0c [term 0 FOLLOWER]: Advancing to term 1
I20260129 02:58:13.311728 20110 raft_consensus.cc:385] T 47ad34405a6f4d508c9613c07a572cc3 P eafca908701c410ab9488970fcb3d74e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260129 02:58:13.312242 20110 raft_consensus.cc:740] T 47ad34405a6f4d508c9613c07a572cc3 P eafca908701c410ab9488970fcb3d74e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eafca908701c410ab9488970fcb3d74e, State: Initialized, Role: FOLLOWER
I20260129 02:58:13.313278 20110 consensus_queue.cc:260] T 47ad34405a6f4d508c9613c07a572cc3 P eafca908701c410ab9488970fcb3d74e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eafca908701c410ab9488970fcb3d74e" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 45001 } } peers { permanent_uuid: "544851f9b7954fd2962675f957e8836a" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34199 } } peers { permanent_uuid: "a634c60a5ca74b26b0d1f679d0166d0c" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 39941 } }
I20260129 02:58:13.317894 20111 tablet_bootstrap.cc:492] T 4562a703e5a44a1585b332b851bfd6f2 P 544851f9b7954fd2962675f957e8836a: No bootstrap required, opened a new log
I20260129 02:58:13.318315 20111 ts_tablet_manager.cc:1403] T 4562a703e5a44a1585b332b851bfd6f2 P 544851f9b7954fd2962675f957e8836a: Time spent bootstrapping tablet: real 0.019s user 0.012s sys 0.004s
I20260129 02:58:13.320676 20111 raft_consensus.cc:359] T 4562a703e5a44a1585b332b851bfd6f2 P 544851f9b7954fd2962675f957e8836a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eafca908701c410ab9488970fcb3d74e" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 45001 } } peers { permanent_uuid: "544851f9b7954fd2962675f957e8836a" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34199 } } peers { permanent_uuid: "a634c60a5ca74b26b0d1f679d0166d0c" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 39941 } }
I20260129 02:58:13.321245 20131 raft_consensus.cc:493] T 79355f4f5d7942b6ac6f420256fd35b7 P a634c60a5ca74b26b0d1f679d0166d0c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260129 02:58:13.321699 20131 raft_consensus.cc:515] T 79355f4f5d7942b6ac6f420256fd35b7 P a634c60a5ca74b26b0d1f679d0166d0c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eafca908701c410ab9488970fcb3d74e" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 45001 } } peers { permanent_uuid: "544851f9b7954fd2962675f957e8836a" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34199 } } peers { permanent_uuid: "a634c60a5ca74b26b0d1f679d0166d0c" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 39941 } }
I20260129 02:58:13.322662 20070 raft_consensus.cc:2468] T 620c6d80e1964c51a628ffb43f645798 P a634c60a5ca74b26b0d1f679d0166d0c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 544851f9b7954fd2962675f957e8836a in term 1.
I20260129 02:58:13.323565 20131 leader_election.cc:290] T 79355f4f5d7942b6ac6f420256fd35b7 P a634c60a5ca74b26b0d1f679d0166d0c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers eafca908701c410ab9488970fcb3d74e (127.18.211.1:45001), 544851f9b7954fd2962675f957e8836a (127.18.211.2:34199)
I20260129 02:58:13.324970 20110 ts_tablet_manager.cc:1434] T 47ad34405a6f4d508c9613c07a572cc3 P eafca908701c410ab9488970fcb3d74e: Time spent starting tablet: real 0.017s user 0.006s sys 0.000s
I20260129 02:58:13.325656 19996 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "79355f4f5d7942b6ac6f420256fd35b7" candidate_uuid: "a634c60a5ca74b26b0d1f679d0166d0c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "544851f9b7954fd2962675f957e8836a" is_pre_election: true
I20260129 02:58:13.326216 19996 raft_consensus.cc:2468] T 79355f4f5d7942b6ac6f420256fd35b7 P 544851f9b7954fd2962675f957e8836a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a634c60a5ca74b26b0d1f679d0166d0c in term 0.
I20260129 02:58:13.327299 20033 leader_election.cc:304] T 79355f4f5d7942b6ac6f420256fd35b7 P a634c60a5ca74b26b0d1f679d0166d0c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 544851f9b7954fd2962675f957e8836a, a634c60a5ca74b26b0d1f679d0166d0c; no voters:
I20260129 02:58:13.328073 20131 raft_consensus.cc:2804] T 79355f4f5d7942b6ac6f420256fd35b7 P a634c60a5ca74b26b0d1f679d0166d0c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260129 02:58:13.328405 20131 raft_consensus.cc:493] T 79355f4f5d7942b6ac6f420256fd35b7 P a634c60a5ca74b26b0d1f679d0166d0c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260129 02:58:13.328287 19921 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "79355f4f5d7942b6ac6f420256fd35b7" candidate_uuid: "a634c60a5ca74b26b0d1f679d0166d0c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eafca908701c410ab9488970fcb3d74e" is_pre_election: true
I20260129 02:58:13.328815 20131 raft_consensus.cc:3060] T 79355f4f5d7942b6ac6f420256fd35b7 P a634c60a5ca74b26b0d1f679d0166d0c [term 0 FOLLOWER]: Advancing to term 1
I20260129 02:58:13.329134 19921 raft_consensus.cc:2468] T 79355f4f5d7942b6ac6f420256fd35b7 P eafca908701c410ab9488970fcb3d74e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a634c60a5ca74b26b0d1f679d0166d0c in term 0.
I20260129 02:58:13.331038 20143 leader_election.cc:290] T 620c6d80e1964c51a628ffb43f645798 P 544851f9b7954fd2962675f957e8836a [CANDIDATE]: Term 1 election: Requested vote from peers eafca908701c410ab9488970fcb3d74e (127.18.211.1:45001), a634c60a5ca74b26b0d1f679d0166d0c (127.18.211.3:39941)
I20260129 02:58:13.334746 20131 raft_consensus.cc:515] T 79355f4f5d7942b6ac6f420256fd35b7 P a634c60a5ca74b26b0d1f679d0166d0c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eafca908701c410ab9488970fcb3d74e" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 45001 } } peers { permanent_uuid: "544851f9b7954fd2962675f957e8836a" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34199 } } peers { permanent_uuid: "a634c60a5ca74b26b0d1f679d0166d0c" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 39941 } }
I20260129 02:58:13.321254 20111 raft_consensus.cc:385] T 4562a703e5a44a1585b332b851bfd6f2 P 544851f9b7954fd2962675f957e8836a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260129 02:58:13.336396 20111 raft_consensus.cc:740] T 4562a703e5a44a1585b332b851bfd6f2 P 544851f9b7954fd2962675f957e8836a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 544851f9b7954fd2962675f957e8836a, State: Initialized, Role: FOLLOWER
I20260129 02:58:13.337175 19919 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "79355f4f5d7942b6ac6f420256fd35b7" candidate_uuid: "a634c60a5ca74b26b0d1f679d0166d0c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eafca908701c410ab9488970fcb3d74e"
I20260129 02:58:13.337090 20111 consensus_queue.cc:260] T 4562a703e5a44a1585b332b851bfd6f2 P 544851f9b7954fd2962675f957e8836a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eafca908701c410ab9488970fcb3d74e" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 45001 } } peers { permanent_uuid: "544851f9b7954fd2962675f957e8836a" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34199 } } peers { permanent_uuid: "a634c60a5ca74b26b0d1f679d0166d0c" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 39941 } }
I20260129 02:58:13.337743 19919 raft_consensus.cc:3060] T 79355f4f5d7942b6ac6f420256fd35b7 P eafca908701c410ab9488970fcb3d74e [term 0 FOLLOWER]: Advancing to term 1
I20260129 02:58:13.339362 20111 ts_tablet_manager.cc:1434] T 4562a703e5a44a1585b332b851bfd6f2 P 544851f9b7954fd2962675f957e8836a: Time spent starting tablet: real 0.021s user 0.005s sys 0.001s
I20260129 02:58:13.340320 20111 tablet_bootstrap.cc:492] T 5d11df549a83429a87d3e7a62fabaa21 P 544851f9b7954fd2962675f957e8836a: Bootstrap starting.
I20260129 02:58:13.344327 19919 raft_consensus.cc:2468] T 79355f4f5d7942b6ac6f420256fd35b7 P eafca908701c410ab9488970fcb3d74e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a634c60a5ca74b26b0d1f679d0166d0c in term 1.
I20260129 02:58:13.345403 20033 leader_election.cc:304] T 79355f4f5d7942b6ac6f420256fd35b7 P a634c60a5ca74b26b0d1f679d0166d0c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a634c60a5ca74b26b0d1f679d0166d0c, eafca908701c410ab9488970fcb3d74e; no voters:
I20260129 02:58:13.346063 20111 tablet_bootstrap.cc:654] T 5d11df549a83429a87d3e7a62fabaa21 P 544851f9b7954fd2962675f957e8836a: Neither blocks nor log segments found. Creating new log.
I20260129 02:58:13.346143 20114 raft_consensus.cc:2804] T 79355f4f5d7942b6ac6f420256fd35b7 P a634c60a5ca74b26b0d1f679d0166d0c [term 1 FOLLOWER]: Leader election won for term 1
I20260129 02:58:13.346657 20114 raft_consensus.cc:697] T 79355f4f5d7942b6ac6f420256fd35b7 P a634c60a5ca74b26b0d1f679d0166d0c [term 1 LEADER]: Becoming Leader. State: Replica: a634c60a5ca74b26b0d1f679d0166d0c, State: Running, Role: LEADER
I20260129 02:58:13.347500 20114 consensus_queue.cc:237] T 79355f4f5d7942b6ac6f420256fd35b7 P a634c60a5ca74b26b0d1f679d0166d0c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eafca908701c410ab9488970fcb3d74e" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 45001 } } peers { permanent_uuid: "544851f9b7954fd2962675f957e8836a" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34199 } } peers { permanent_uuid: "a634c60a5ca74b26b0d1f679d0166d0c" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 39941 } }
I20260129 02:58:13.347410 19817 catalog_manager.cc:5654] T 620c6d80e1964c51a628ffb43f645798 P 544851f9b7954fd2962675f957e8836a reported cstate change: term changed from 0 to 1, leader changed from <none> to 544851f9b7954fd2962675f957e8836a (127.18.211.2). New cstate: current_term: 1 leader_uuid: "544851f9b7954fd2962675f957e8836a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eafca908701c410ab9488970fcb3d74e" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 45001 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "544851f9b7954fd2962675f957e8836a" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34199 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a634c60a5ca74b26b0d1f679d0166d0c" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 39941 } health_report { overall_health: UNKNOWN } } }
I20260129 02:58:13.357126 19996 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "79355f4f5d7942b6ac6f420256fd35b7" candidate_uuid: "a634c60a5ca74b26b0d1f679d0166d0c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "544851f9b7954fd2962675f957e8836a"
I20260129 02:58:13.357777 20109 tablet_bootstrap.cc:492] T d5edd0a419ee46c9a6a223f8056afb4c P a634c60a5ca74b26b0d1f679d0166d0c: No bootstrap required, opened a new log
I20260129 02:58:13.357885 19996 raft_consensus.cc:3060] T 79355f4f5d7942b6ac6f420256fd35b7 P 544851f9b7954fd2962675f957e8836a [term 0 FOLLOWER]: Advancing to term 1
I20260129 02:58:13.358237 20109 ts_tablet_manager.cc:1403] T d5edd0a419ee46c9a6a223f8056afb4c P a634c60a5ca74b26b0d1f679d0166d0c: Time spent bootstrapping tablet: real 0.078s user 0.028s sys 0.033s
I20260129 02:58:13.360873 20109 raft_consensus.cc:359] T d5edd0a419ee46c9a6a223f8056afb4c P a634c60a5ca74b26b0d1f679d0166d0c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eafca908701c410ab9488970fcb3d74e" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 45001 } } peers { permanent_uuid: "544851f9b7954fd2962675f957e8836a" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34199 } } peers { permanent_uuid: "a634c60a5ca74b26b0d1f679d0166d0c" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 39941 } }
I20260129 02:58:13.361510 20109 raft_consensus.cc:385] T d5edd0a419ee46c9a6a223f8056afb4c P a634c60a5ca74b26b0d1f679d0166d0c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260129 02:58:13.361758 20109 raft_consensus.cc:740] T d5edd0a419ee46c9a6a223f8056afb4c P a634c60a5ca74b26b0d1f679d0166d0c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a634c60a5ca74b26b0d1f679d0166d0c, State: Initialized, Role: FOLLOWER
I20260129 02:58:13.362315 20109 consensus_queue.cc:260] T d5edd0a419ee46c9a6a223f8056afb4c P a634c60a5ca74b26b0d1f679d0166d0c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eafca908701c410ab9488970fcb3d74e" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 45001 } } peers { permanent_uuid: "544851f9b7954fd2962675f957e8836a" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34199 } } peers { permanent_uuid: "a634c60a5ca74b26b0d1f679d0166d0c" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 39941 } }
I20260129 02:58:13.365401 20109 ts_tablet_manager.cc:1434] T d5edd0a419ee46c9a6a223f8056afb4c P a634c60a5ca74b26b0d1f679d0166d0c: Time spent starting tablet: real 0.007s user 0.003s sys 0.004s
I20260129 02:58:13.366247 20109 tablet_bootstrap.cc:492] T 1dc801ccecc44ed79d85b0522e49d3e5 P a634c60a5ca74b26b0d1f679d0166d0c: Bootstrap starting.
I20260129 02:58:13.366357 20111 tablet_bootstrap.cc:492] T 5d11df549a83429a87d3e7a62fabaa21 P 544851f9b7954fd2962675f957e8836a: No bootstrap required, opened a new log
I20260129 02:58:13.366919 20111 ts_tablet_manager.cc:1403] T 5d11df549a83429a87d3e7a62fabaa21 P 544851f9b7954fd2962675f957e8836a: Time spent bootstrapping tablet: real 0.027s user 0.018s sys 0.007s
I20260129 02:58:13.367403 20131 leader_election.cc:290] T 79355f4f5d7942b6ac6f420256fd35b7 P a634c60a5ca74b26b0d1f679d0166d0c [CANDIDATE]: Term 1 election: Requested vote from peers eafca908701c410ab9488970fcb3d74e (127.18.211.1:45001), 544851f9b7954fd2962675f957e8836a (127.18.211.2:34199)
I20260129 02:58:13.366750 19996 raft_consensus.cc:2468] T 79355f4f5d7942b6ac6f420256fd35b7 P 544851f9b7954fd2962675f957e8836a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a634c60a5ca74b26b0d1f679d0166d0c in term 1.
I20260129 02:58:13.369316 20111 raft_consensus.cc:359] T 5d11df549a83429a87d3e7a62fabaa21 P 544851f9b7954fd2962675f957e8836a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eafca908701c410ab9488970fcb3d74e" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 45001 } } peers { permanent_uuid: "544851f9b7954fd2962675f957e8836a" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34199 } } peers { permanent_uuid: "a634c60a5ca74b26b0d1f679d0166d0c" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 39941 } }
I20260129 02:58:13.369792 20111 raft_consensus.cc:385] T 5d11df549a83429a87d3e7a62fabaa21 P 544851f9b7954fd2962675f957e8836a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260129 02:58:13.369998 20111 raft_consensus.cc:740] T 5d11df549a83429a87d3e7a62fabaa21 P 544851f9b7954fd2962675f957e8836a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 544851f9b7954fd2962675f957e8836a, State: Initialized, Role: FOLLOWER
I20260129 02:58:13.370549 20111 consensus_queue.cc:260] T 5d11df549a83429a87d3e7a62fabaa21 P 544851f9b7954fd2962675f957e8836a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eafca908701c410ab9488970fcb3d74e" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 45001 } } peers { permanent_uuid: "544851f9b7954fd2962675f957e8836a" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34199 } } peers { permanent_uuid: "a634c60a5ca74b26b0d1f679d0166d0c" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 39941 } }
I20260129 02:58:13.372466 20111 ts_tablet_manager.cc:1434] T 5d11df549a83429a87d3e7a62fabaa21 P 544851f9b7954fd2962675f957e8836a: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260129 02:58:13.374786 20109 tablet_bootstrap.cc:654] T 1dc801ccecc44ed79d85b0522e49d3e5 P a634c60a5ca74b26b0d1f679d0166d0c: Neither blocks nor log segments found. Creating new log.
I20260129 02:58:13.377653 19817 catalog_manager.cc:5654] T 79355f4f5d7942b6ac6f420256fd35b7 P a634c60a5ca74b26b0d1f679d0166d0c reported cstate change: term changed from 0 to 1, leader changed from <none> to a634c60a5ca74b26b0d1f679d0166d0c (127.18.211.3). New cstate: current_term: 1 leader_uuid: "a634c60a5ca74b26b0d1f679d0166d0c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eafca908701c410ab9488970fcb3d74e" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 45001 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "544851f9b7954fd2962675f957e8836a" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34199 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a634c60a5ca74b26b0d1f679d0166d0c" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 39941 } health_report { overall_health: HEALTHY } } }
I20260129 02:58:13.379472 20109 tablet_bootstrap.cc:492] T 1dc801ccecc44ed79d85b0522e49d3e5 P a634c60a5ca74b26b0d1f679d0166d0c: No bootstrap required, opened a new log
I20260129 02:58:13.379921 20109 ts_tablet_manager.cc:1403] T 1dc801ccecc44ed79d85b0522e49d3e5 P a634c60a5ca74b26b0d1f679d0166d0c: Time spent bootstrapping tablet: real 0.014s user 0.004s sys 0.004s
I20260129 02:58:13.381771 20109 raft_consensus.cc:359] T 1dc801ccecc44ed79d85b0522e49d3e5 P a634c60a5ca74b26b0d1f679d0166d0c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eafca908701c410ab9488970fcb3d74e" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 45001 } } peers { permanent_uuid: "544851f9b7954fd2962675f957e8836a" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34199 } } peers { permanent_uuid: "a634c60a5ca74b26b0d1f679d0166d0c" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 39941 } }
I20260129 02:58:13.382252 20109 raft_consensus.cc:385] T 1dc801ccecc44ed79d85b0522e49d3e5 P a634c60a5ca74b26b0d1f679d0166d0c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260129 02:58:13.382477 20109 raft_consensus.cc:740] T 1dc801ccecc44ed79d85b0522e49d3e5 P a634c60a5ca74b26b0d1f679d0166d0c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a634c60a5ca74b26b0d1f679d0166d0c, State: Initialized, Role: FOLLOWER
I20260129 02:58:13.383074 20109 consensus_queue.cc:260] T 1dc801ccecc44ed79d85b0522e49d3e5 P a634c60a5ca74b26b0d1f679d0166d0c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eafca908701c410ab9488970fcb3d74e" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 45001 } } peers { permanent_uuid: "544851f9b7954fd2962675f957e8836a" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34199 } } peers { permanent_uuid: "a634c60a5ca74b26b0d1f679d0166d0c" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 39941 } }
I20260129 02:58:13.384743 20109 ts_tablet_manager.cc:1434] T 1dc801ccecc44ed79d85b0522e49d3e5 P a634c60a5ca74b26b0d1f679d0166d0c: Time spent starting tablet: real 0.005s user 0.004s sys 0.000s
I20260129 02:58:13.385522 20109 tablet_bootstrap.cc:492] T 5d11df549a83429a87d3e7a62fabaa21 P a634c60a5ca74b26b0d1f679d0166d0c: Bootstrap starting.
I20260129 02:58:13.390636 20109 tablet_bootstrap.cc:654] T 5d11df549a83429a87d3e7a62fabaa21 P a634c60a5ca74b26b0d1f679d0166d0c: Neither blocks nor log segments found. Creating new log.
I20260129 02:58:13.394501 20133 raft_consensus.cc:493] T 47ad34405a6f4d508c9613c07a572cc3 P eafca908701c410ab9488970fcb3d74e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260129 02:58:13.395038 20109 tablet_bootstrap.cc:492] T 5d11df549a83429a87d3e7a62fabaa21 P a634c60a5ca74b26b0d1f679d0166d0c: No bootstrap required, opened a new log
I20260129 02:58:13.394877 20133 raft_consensus.cc:515] T 47ad34405a6f4d508c9613c07a572cc3 P eafca908701c410ab9488970fcb3d74e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eafca908701c410ab9488970fcb3d74e" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 45001 } } peers { permanent_uuid: "544851f9b7954fd2962675f957e8836a" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34199 } } peers { permanent_uuid: "a634c60a5ca74b26b0d1f679d0166d0c" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 39941 } }
I20260129 02:58:13.395363 20109 ts_tablet_manager.cc:1403] T 5d11df549a83429a87d3e7a62fabaa21 P a634c60a5ca74b26b0d1f679d0166d0c: Time spent bootstrapping tablet: real 0.010s user 0.004s sys 0.004s
I20260129 02:58:13.396571 20133 leader_election.cc:290] T 47ad34405a6f4d508c9613c07a572cc3 P eafca908701c410ab9488970fcb3d74e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 544851f9b7954fd2962675f957e8836a (127.18.211.2:34199), a634c60a5ca74b26b0d1f679d0166d0c (127.18.211.3:39941)
I20260129 02:58:13.397316 19996 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "47ad34405a6f4d508c9613c07a572cc3" candidate_uuid: "eafca908701c410ab9488970fcb3d74e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "544851f9b7954fd2962675f957e8836a" is_pre_election: true
I20260129 02:58:13.397531 20070 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "47ad34405a6f4d508c9613c07a572cc3" candidate_uuid: "eafca908701c410ab9488970fcb3d74e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a634c60a5ca74b26b0d1f679d0166d0c" is_pre_election: true
I20260129 02:58:13.397439 20109 raft_consensus.cc:359] T 5d11df549a83429a87d3e7a62fabaa21 P a634c60a5ca74b26b0d1f679d0166d0c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eafca908701c410ab9488970fcb3d74e" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 45001 } } peers { permanent_uuid: "544851f9b7954fd2962675f957e8836a" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34199 } } peers { permanent_uuid: "a634c60a5ca74b26b0d1f679d0166d0c" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 39941 } }
I20260129 02:58:13.397900 19996 raft_consensus.cc:2468] T 47ad34405a6f4d508c9613c07a572cc3 P 544851f9b7954fd2962675f957e8836a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eafca908701c410ab9488970fcb3d74e in term 0.
I20260129 02:58:13.398111 20109 raft_consensus.cc:385] T 5d11df549a83429a87d3e7a62fabaa21 P a634c60a5ca74b26b0d1f679d0166d0c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260129 02:58:13.398147 20070 raft_consensus.cc:2468] T 47ad34405a6f4d508c9613c07a572cc3 P a634c60a5ca74b26b0d1f679d0166d0c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eafca908701c410ab9488970fcb3d74e in term 0.
I20260129 02:58:13.398598 20109 raft_consensus.cc:740] T 5d11df549a83429a87d3e7a62fabaa21 P a634c60a5ca74b26b0d1f679d0166d0c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a634c60a5ca74b26b0d1f679d0166d0c, State: Initialized, Role: FOLLOWER
I20260129 02:58:13.399080 19884 leader_election.cc:304] T 47ad34405a6f4d508c9613c07a572cc3 P eafca908701c410ab9488970fcb3d74e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 544851f9b7954fd2962675f957e8836a, eafca908701c410ab9488970fcb3d74e; no voters:
I20260129 02:58:13.399260 20109 consensus_queue.cc:260] T 5d11df549a83429a87d3e7a62fabaa21 P a634c60a5ca74b26b0d1f679d0166d0c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eafca908701c410ab9488970fcb3d74e" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 45001 } } peers { permanent_uuid: "544851f9b7954fd2962675f957e8836a" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34199 } } peers { permanent_uuid: "a634c60a5ca74b26b0d1f679d0166d0c" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 39941 } }
I20260129 02:58:13.399823 20133 raft_consensus.cc:2804] T 47ad34405a6f4d508c9613c07a572cc3 P eafca908701c410ab9488970fcb3d74e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260129 02:58:13.400203 20133 raft_consensus.cc:493] T 47ad34405a6f4d508c9613c07a572cc3 P eafca908701c410ab9488970fcb3d74e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260129 02:58:13.400563 20133 raft_consensus.cc:3060] T 47ad34405a6f4d508c9613c07a572cc3 P eafca908701c410ab9488970fcb3d74e [term 0 FOLLOWER]: Advancing to term 1
I20260129 02:58:13.401520 20109 ts_tablet_manager.cc:1434] T 5d11df549a83429a87d3e7a62fabaa21 P a634c60a5ca74b26b0d1f679d0166d0c: Time spent starting tablet: real 0.006s user 0.003s sys 0.003s
I20260129 02:58:13.406430 20133 raft_consensus.cc:515] T 47ad34405a6f4d508c9613c07a572cc3 P eafca908701c410ab9488970fcb3d74e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eafca908701c410ab9488970fcb3d74e" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 45001 } } peers { permanent_uuid: "544851f9b7954fd2962675f957e8836a" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34199 } } peers { permanent_uuid: "a634c60a5ca74b26b0d1f679d0166d0c" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 39941 } }
I20260129 02:58:13.408000 20133 leader_election.cc:290] T 47ad34405a6f4d508c9613c07a572cc3 P eafca908701c410ab9488970fcb3d74e [CANDIDATE]: Term 1 election: Requested vote from peers 544851f9b7954fd2962675f957e8836a (127.18.211.2:34199), a634c60a5ca74b26b0d1f679d0166d0c (127.18.211.3:39941)
I20260129 02:58:13.408797 19996 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "47ad34405a6f4d508c9613c07a572cc3" candidate_uuid: "eafca908701c410ab9488970fcb3d74e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "544851f9b7954fd2962675f957e8836a"
I20260129 02:58:13.408922 20070 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "47ad34405a6f4d508c9613c07a572cc3" candidate_uuid: "eafca908701c410ab9488970fcb3d74e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a634c60a5ca74b26b0d1f679d0166d0c"
I20260129 02:58:13.409312 19996 raft_consensus.cc:3060] T 47ad34405a6f4d508c9613c07a572cc3 P 544851f9b7954fd2962675f957e8836a [term 0 FOLLOWER]: Advancing to term 1
I20260129 02:58:13.409461 20070 raft_consensus.cc:3060] T 47ad34405a6f4d508c9613c07a572cc3 P a634c60a5ca74b26b0d1f679d0166d0c [term 0 FOLLOWER]: Advancing to term 1
I20260129 02:58:13.414170 19996 raft_consensus.cc:2468] T 47ad34405a6f4d508c9613c07a572cc3 P 544851f9b7954fd2962675f957e8836a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eafca908701c410ab9488970fcb3d74e in term 1.
I20260129 02:58:13.414300 20070 raft_consensus.cc:2468] T 47ad34405a6f4d508c9613c07a572cc3 P a634c60a5ca74b26b0d1f679d0166d0c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eafca908701c410ab9488970fcb3d74e in term 1.
I20260129 02:58:13.415094 19884 leader_election.cc:304] T 47ad34405a6f4d508c9613c07a572cc3 P eafca908701c410ab9488970fcb3d74e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 544851f9b7954fd2962675f957e8836a, eafca908701c410ab9488970fcb3d74e; no voters:
I20260129 02:58:13.415725 20133 raft_consensus.cc:2804] T 47ad34405a6f4d508c9613c07a572cc3 P eafca908701c410ab9488970fcb3d74e [term 1 FOLLOWER]: Leader election won for term 1
I20260129 02:58:13.416133 20133 raft_consensus.cc:697] T 47ad34405a6f4d508c9613c07a572cc3 P eafca908701c410ab9488970fcb3d74e [term 1 LEADER]: Becoming Leader. State: Replica: eafca908701c410ab9488970fcb3d74e, State: Running, Role: LEADER
I20260129 02:58:13.416682 20133 consensus_queue.cc:237] T 47ad34405a6f4d508c9613c07a572cc3 P eafca908701c410ab9488970fcb3d74e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eafca908701c410ab9488970fcb3d74e" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 45001 } } peers { permanent_uuid: "544851f9b7954fd2962675f957e8836a" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34199 } } peers { permanent_uuid: "a634c60a5ca74b26b0d1f679d0166d0c" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 39941 } }
I20260129 02:58:13.423105 19817 catalog_manager.cc:5654] T 47ad34405a6f4d508c9613c07a572cc3 P eafca908701c410ab9488970fcb3d74e reported cstate change: term changed from 0 to 1, leader changed from <none> to eafca908701c410ab9488970fcb3d74e (127.18.211.1). New cstate: current_term: 1 leader_uuid: "eafca908701c410ab9488970fcb3d74e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eafca908701c410ab9488970fcb3d74e" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 45001 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "544851f9b7954fd2962675f957e8836a" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34199 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a634c60a5ca74b26b0d1f679d0166d0c" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 39941 } health_report { overall_health: UNKNOWN } } }
I20260129 02:58:13.456619 20117 raft_consensus.cc:493] T 5d11df549a83429a87d3e7a62fabaa21 P 544851f9b7954fd2962675f957e8836a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260129 02:58:13.457051 20117 raft_consensus.cc:515] T 5d11df549a83429a87d3e7a62fabaa21 P 544851f9b7954fd2962675f957e8836a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eafca908701c410ab9488970fcb3d74e" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 45001 } } peers { permanent_uuid: "544851f9b7954fd2962675f957e8836a" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34199 } } peers { permanent_uuid: "a634c60a5ca74b26b0d1f679d0166d0c" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 39941 } }
I20260129 02:58:13.458643 20117 leader_election.cc:290] T 5d11df549a83429a87d3e7a62fabaa21 P 544851f9b7954fd2962675f957e8836a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers eafca908701c410ab9488970fcb3d74e (127.18.211.1:45001), a634c60a5ca74b26b0d1f679d0166d0c (127.18.211.3:39941)
I20260129 02:58:13.459470 19919 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5d11df549a83429a87d3e7a62fabaa21" candidate_uuid: "544851f9b7954fd2962675f957e8836a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eafca908701c410ab9488970fcb3d74e" is_pre_election: true
I20260129 02:58:13.459702 20070 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5d11df549a83429a87d3e7a62fabaa21" candidate_uuid: "544851f9b7954fd2962675f957e8836a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a634c60a5ca74b26b0d1f679d0166d0c" is_pre_election: true
I20260129 02:58:13.460057 19919 raft_consensus.cc:2468] T 5d11df549a83429a87d3e7a62fabaa21 P eafca908701c410ab9488970fcb3d74e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 544851f9b7954fd2962675f957e8836a in term 0.
I20260129 02:58:13.460281 20070 raft_consensus.cc:2468] T 5d11df549a83429a87d3e7a62fabaa21 P a634c60a5ca74b26b0d1f679d0166d0c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 544851f9b7954fd2962675f957e8836a in term 0.
I20260129 02:58:13.460963 19959 leader_election.cc:304] T 5d11df549a83429a87d3e7a62fabaa21 P 544851f9b7954fd2962675f957e8836a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 544851f9b7954fd2962675f957e8836a, eafca908701c410ab9488970fcb3d74e; no voters:
I20260129 02:58:13.461670 20117 raft_consensus.cc:2804] T 5d11df549a83429a87d3e7a62fabaa21 P 544851f9b7954fd2962675f957e8836a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260129 02:58:13.461936 20117 raft_consensus.cc:493] T 5d11df549a83429a87d3e7a62fabaa21 P 544851f9b7954fd2962675f957e8836a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260129 02:58:13.462216 20117 raft_consensus.cc:3060] T 5d11df549a83429a87d3e7a62fabaa21 P 544851f9b7954fd2962675f957e8836a [term 0 FOLLOWER]: Advancing to term 1
I20260129 02:58:13.467000 20117 raft_consensus.cc:515] T 5d11df549a83429a87d3e7a62fabaa21 P 544851f9b7954fd2962675f957e8836a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eafca908701c410ab9488970fcb3d74e" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 45001 } } peers { permanent_uuid: "544851f9b7954fd2962675f957e8836a" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34199 } } peers { permanent_uuid: "a634c60a5ca74b26b0d1f679d0166d0c" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 39941 } }
I20260129 02:58:13.469090 20117 leader_election.cc:290] T 5d11df549a83429a87d3e7a62fabaa21 P 544851f9b7954fd2962675f957e8836a [CANDIDATE]: Term 1 election: Requested vote from peers eafca908701c410ab9488970fcb3d74e (127.18.211.1:45001), a634c60a5ca74b26b0d1f679d0166d0c (127.18.211.3:39941)
I20260129 02:58:13.469429 19919 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5d11df549a83429a87d3e7a62fabaa21" candidate_uuid: "544851f9b7954fd2962675f957e8836a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eafca908701c410ab9488970fcb3d74e"
I20260129 02:58:13.469571 20131 raft_consensus.cc:493] T 1bd5a7f0c64f4ef6bc2bb544cd69843c P a634c60a5ca74b26b0d1f679d0166d0c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260129 02:58:13.469864 20114 raft_consensus.cc:493] T 901f619b02f843b4a2747de956182f12 P a634c60a5ca74b26b0d1f679d0166d0c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260129 02:58:13.469995 19919 raft_consensus.cc:3060] T 5d11df549a83429a87d3e7a62fabaa21 P eafca908701c410ab9488970fcb3d74e [term 0 FOLLOWER]: Advancing to term 1
I20260129 02:58:13.470075 20131 raft_consensus.cc:515] T 1bd5a7f0c64f4ef6bc2bb544cd69843c P a634c60a5ca74b26b0d1f679d0166d0c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eafca908701c410ab9488970fcb3d74e" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 45001 } } peers { permanent_uuid: "544851f9b7954fd2962675f957e8836a" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34199 } } peers { permanent_uuid: "a634c60a5ca74b26b0d1f679d0166d0c" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 39941 } }
I20260129 02:58:13.470304 20114 raft_consensus.cc:515] T 901f619b02f843b4a2747de956182f12 P a634c60a5ca74b26b0d1f679d0166d0c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eafca908701c410ab9488970fcb3d74e" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 45001 } } peers { permanent_uuid: "544851f9b7954fd2962675f957e8836a" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34199 } } peers { permanent_uuid: "a634c60a5ca74b26b0d1f679d0166d0c" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 39941 } }
I20260129 02:58:13.471175 20070 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5d11df549a83429a87d3e7a62fabaa21" candidate_uuid: "544851f9b7954fd2962675f957e8836a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a634c60a5ca74b26b0d1f679d0166d0c"
I20260129 02:58:13.471727 20070 raft_consensus.cc:3060] T 5d11df549a83429a87d3e7a62fabaa21 P a634c60a5ca74b26b0d1f679d0166d0c [term 0 FOLLOWER]: Advancing to term 1
I20260129 02:58:13.471853 20131 leader_election.cc:290] T 1bd5a7f0c64f4ef6bc2bb544cd69843c P a634c60a5ca74b26b0d1f679d0166d0c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers eafca908701c410ab9488970fcb3d74e (127.18.211.1:45001), 544851f9b7954fd2962675f957e8836a (127.18.211.2:34199)
I20260129 02:58:13.473361 19996 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1bd5a7f0c64f4ef6bc2bb544cd69843c" candidate_uuid: "a634c60a5ca74b26b0d1f679d0166d0c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "544851f9b7954fd2962675f957e8836a" is_pre_election: true
I20260129 02:58:13.473938 19995 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "901f619b02f843b4a2747de956182f12" candidate_uuid: "a634c60a5ca74b26b0d1f679d0166d0c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "544851f9b7954fd2962675f957e8836a" is_pre_election: true
I20260129 02:58:13.473995 19996 raft_consensus.cc:2468] T 1bd5a7f0c64f4ef6bc2bb544cd69843c P 544851f9b7954fd2962675f957e8836a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a634c60a5ca74b26b0d1f679d0166d0c in term 0.
I20260129 02:58:13.474030 19921 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1bd5a7f0c64f4ef6bc2bb544cd69843c" candidate_uuid: "a634c60a5ca74b26b0d1f679d0166d0c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eafca908701c410ab9488970fcb3d74e" is_pre_election: true
I20260129 02:58:13.474694 19995 raft_consensus.cc:2468] T 901f619b02f843b4a2747de956182f12 P 544851f9b7954fd2962675f957e8836a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a634c60a5ca74b26b0d1f679d0166d0c in term 0.
I20260129 02:58:13.474844 19920 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "901f619b02f843b4a2747de956182f12" candidate_uuid: "a634c60a5ca74b26b0d1f679d0166d0c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eafca908701c410ab9488970fcb3d74e" is_pre_election: true
I20260129 02:58:13.475167 19921 raft_consensus.cc:2468] T 1bd5a7f0c64f4ef6bc2bb544cd69843c P eafca908701c410ab9488970fcb3d74e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a634c60a5ca74b26b0d1f679d0166d0c in term 0.
I20260129 02:58:13.475414 19920 raft_consensus.cc:2468] T 901f619b02f843b4a2747de956182f12 P eafca908701c410ab9488970fcb3d74e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a634c60a5ca74b26b0d1f679d0166d0c in term 0.
I20260129 02:58:13.476040 20033 leader_election.cc:304] T 1bd5a7f0c64f4ef6bc2bb544cd69843c P a634c60a5ca74b26b0d1f679d0166d0c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 544851f9b7954fd2962675f957e8836a, a634c60a5ca74b26b0d1f679d0166d0c; no voters:
I20260129 02:58:13.471911 20114 leader_election.cc:290] T 901f619b02f843b4a2747de956182f12 P a634c60a5ca74b26b0d1f679d0166d0c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers eafca908701c410ab9488970fcb3d74e (127.18.211.1:45001), 544851f9b7954fd2962675f957e8836a (127.18.211.2:34199)
I20260129 02:58:13.476641 19919 raft_consensus.cc:2468] T 5d11df549a83429a87d3e7a62fabaa21 P eafca908701c410ab9488970fcb3d74e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 544851f9b7954fd2962675f957e8836a in term 1.
I20260129 02:58:13.476997 20114 raft_consensus.cc:2804] T 1bd5a7f0c64f4ef6bc2bb544cd69843c P a634c60a5ca74b26b0d1f679d0166d0c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260129 02:58:13.476919 20033 leader_election.cc:304] T 901f619b02f843b4a2747de956182f12 P a634c60a5ca74b26b0d1f679d0166d0c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 544851f9b7954fd2962675f957e8836a, a634c60a5ca74b26b0d1f679d0166d0c; no voters:
I20260129 02:58:13.477706 19959 leader_election.cc:304] T 5d11df549a83429a87d3e7a62fabaa21 P 544851f9b7954fd2962675f957e8836a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 544851f9b7954fd2962675f957e8836a, eafca908701c410ab9488970fcb3d74e; no voters:
I20260129 02:58:13.477409 20114 raft_consensus.cc:493] T 1bd5a7f0c64f4ef6bc2bb544cd69843c P a634c60a5ca74b26b0d1f679d0166d0c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260129 02:58:13.478323 20114 raft_consensus.cc:3060] T 1bd5a7f0c64f4ef6bc2bb544cd69843c P a634c60a5ca74b26b0d1f679d0166d0c [term 0 FOLLOWER]: Advancing to term 1
I20260129 02:58:13.477841 20132 raft_consensus.cc:2804] T 901f619b02f843b4a2747de956182f12 P a634c60a5ca74b26b0d1f679d0166d0c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260129 02:58:13.480079 20132 raft_consensus.cc:493] T 901f619b02f843b4a2747de956182f12 P a634c60a5ca74b26b0d1f679d0166d0c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260129 02:58:13.480432 20132 raft_consensus.cc:3060] T 901f619b02f843b4a2747de956182f12 P a634c60a5ca74b26b0d1f679d0166d0c [term 0 FOLLOWER]: Advancing to term 1
I20260129 02:58:13.481235 20070 raft_consensus.cc:2468] T 5d11df549a83429a87d3e7a62fabaa21 P a634c60a5ca74b26b0d1f679d0166d0c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 544851f9b7954fd2962675f957e8836a in term 1.
I20260129 02:58:13.482040 20117 raft_consensus.cc:2804] T 5d11df549a83429a87d3e7a62fabaa21 P 544851f9b7954fd2962675f957e8836a [term 1 FOLLOWER]: Leader election won for term 1
I20260129 02:58:13.482744 20117 raft_consensus.cc:697] T 5d11df549a83429a87d3e7a62fabaa21 P 544851f9b7954fd2962675f957e8836a [term 1 LEADER]: Becoming Leader. State: Replica: 544851f9b7954fd2962675f957e8836a, State: Running, Role: LEADER
I20260129 02:58:13.483795 20114 raft_consensus.cc:515] T 1bd5a7f0c64f4ef6bc2bb544cd69843c P a634c60a5ca74b26b0d1f679d0166d0c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eafca908701c410ab9488970fcb3d74e" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 45001 } } peers { permanent_uuid: "544851f9b7954fd2962675f957e8836a" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34199 } } peers { permanent_uuid: "a634c60a5ca74b26b0d1f679d0166d0c" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 39941 } }
I20260129 02:58:13.483618 20117 consensus_queue.cc:237] T 5d11df549a83429a87d3e7a62fabaa21 P 544851f9b7954fd2962675f957e8836a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eafca908701c410ab9488970fcb3d74e" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 45001 } } peers { permanent_uuid: "544851f9b7954fd2962675f957e8836a" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34199 } } peers { permanent_uuid: "a634c60a5ca74b26b0d1f679d0166d0c" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 39941 } }
I20260129 02:58:13.486130 20114 leader_election.cc:290] T 1bd5a7f0c64f4ef6bc2bb544cd69843c P a634c60a5ca74b26b0d1f679d0166d0c [CANDIDATE]: Term 1 election: Requested vote from peers eafca908701c410ab9488970fcb3d74e (127.18.211.1:45001), 544851f9b7954fd2962675f957e8836a (127.18.211.2:34199)
I20260129 02:58:13.486785 19919 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1bd5a7f0c64f4ef6bc2bb544cd69843c" candidate_uuid: "a634c60a5ca74b26b0d1f679d0166d0c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eafca908701c410ab9488970fcb3d74e"
I20260129 02:58:13.486619 20132 raft_consensus.cc:515] T 901f619b02f843b4a2747de956182f12 P a634c60a5ca74b26b0d1f679d0166d0c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eafca908701c410ab9488970fcb3d74e" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 45001 } } peers { permanent_uuid: "544851f9b7954fd2962675f957e8836a" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34199 } } peers { permanent_uuid: "a634c60a5ca74b26b0d1f679d0166d0c" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 39941 } }
I20260129 02:58:13.487048 19995 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1bd5a7f0c64f4ef6bc2bb544cd69843c" candidate_uuid: "a634c60a5ca74b26b0d1f679d0166d0c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "544851f9b7954fd2962675f957e8836a"
I20260129 02:58:13.487295 19919 raft_consensus.cc:3060] T 1bd5a7f0c64f4ef6bc2bb544cd69843c P eafca908701c410ab9488970fcb3d74e [term 0 FOLLOWER]: Advancing to term 1
I20260129 02:58:13.487633 19995 raft_consensus.cc:3060] T 1bd5a7f0c64f4ef6bc2bb544cd69843c P 544851f9b7954fd2962675f957e8836a [term 0 FOLLOWER]: Advancing to term 1
I20260129 02:58:13.489063 19920 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "901f619b02f843b4a2747de956182f12" candidate_uuid: "a634c60a5ca74b26b0d1f679d0166d0c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eafca908701c410ab9488970fcb3d74e"
I20260129 02:58:13.489578 19920 raft_consensus.cc:3060] T 901f619b02f843b4a2747de956182f12 P eafca908701c410ab9488970fcb3d74e [term 0 FOLLOWER]: Advancing to term 1
I20260129 02:58:13.494907 20132 leader_election.cc:290] T 901f619b02f843b4a2747de956182f12 P a634c60a5ca74b26b0d1f679d0166d0c [CANDIDATE]: Term 1 election: Requested vote from peers eafca908701c410ab9488970fcb3d74e (127.18.211.1:45001), 544851f9b7954fd2962675f957e8836a (127.18.211.2:34199)
I20260129 02:58:13.495128 19996 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "901f619b02f843b4a2747de956182f12" candidate_uuid: "a634c60a5ca74b26b0d1f679d0166d0c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "544851f9b7954fd2962675f957e8836a"
I20260129 02:58:13.495709 19996 raft_consensus.cc:3060] T 901f619b02f843b4a2747de956182f12 P 544851f9b7954fd2962675f957e8836a [term 0 FOLLOWER]: Advancing to term 1
I20260129 02:58:13.496141 19995 raft_consensus.cc:2468] T 1bd5a7f0c64f4ef6bc2bb544cd69843c P 544851f9b7954fd2962675f957e8836a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a634c60a5ca74b26b0d1f679d0166d0c in term 1.
I20260129 02:58:13.496217 19920 raft_consensus.cc:2468] T 901f619b02f843b4a2747de956182f12 P eafca908701c410ab9488970fcb3d74e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a634c60a5ca74b26b0d1f679d0166d0c in term 1.
I20260129 02:58:13.493575 19919 raft_consensus.cc:2468] T 1bd5a7f0c64f4ef6bc2bb544cd69843c P eafca908701c410ab9488970fcb3d74e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a634c60a5ca74b26b0d1f679d0166d0c in term 1.
I20260129 02:58:13.497411 20033 leader_election.cc:304] T 901f619b02f843b4a2747de956182f12 P a634c60a5ca74b26b0d1f679d0166d0c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a634c60a5ca74b26b0d1f679d0166d0c, eafca908701c410ab9488970fcb3d74e; no voters:
I20260129 02:58:13.498252 20132 raft_consensus.cc:2804] T 901f619b02f843b4a2747de956182f12 P a634c60a5ca74b26b0d1f679d0166d0c [term 1 FOLLOWER]: Leader election won for term 1
I20260129 02:58:13.498416 20033 leader_election.cc:304] T 1bd5a7f0c64f4ef6bc2bb544cd69843c P a634c60a5ca74b26b0d1f679d0166d0c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 544851f9b7954fd2962675f957e8836a, a634c60a5ca74b26b0d1f679d0166d0c; no voters:
I20260129 02:58:13.498757 20132 raft_consensus.cc:697] T 901f619b02f843b4a2747de956182f12 P a634c60a5ca74b26b0d1f679d0166d0c [term 1 LEADER]: Becoming Leader. State: Replica: a634c60a5ca74b26b0d1f679d0166d0c, State: Running, Role: LEADER
I20260129 02:58:13.498101 19817 catalog_manager.cc:5654] T 5d11df549a83429a87d3e7a62fabaa21 P 544851f9b7954fd2962675f957e8836a reported cstate change: term changed from 0 to 1, leader changed from <none> to 544851f9b7954fd2962675f957e8836a (127.18.211.2). New cstate: current_term: 1 leader_uuid: "544851f9b7954fd2962675f957e8836a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eafca908701c410ab9488970fcb3d74e" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 45001 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "544851f9b7954fd2962675f957e8836a" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34199 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a634c60a5ca74b26b0d1f679d0166d0c" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 39941 } health_report { overall_health: UNKNOWN } } }
I20260129 02:58:13.499590 20132 consensus_queue.cc:237] T 901f619b02f843b4a2747de956182f12 P a634c60a5ca74b26b0d1f679d0166d0c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eafca908701c410ab9488970fcb3d74e" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 45001 } } peers { permanent_uuid: "544851f9b7954fd2962675f957e8836a" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34199 } } peers { permanent_uuid: "a634c60a5ca74b26b0d1f679d0166d0c" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 39941 } }
I20260129 02:58:13.500252 20114 raft_consensus.cc:2804] T 1bd5a7f0c64f4ef6bc2bb544cd69843c P a634c60a5ca74b26b0d1f679d0166d0c [term 1 FOLLOWER]: Leader election won for term 1
I20260129 02:58:13.500718 20114 raft_consensus.cc:697] T 1bd5a7f0c64f4ef6bc2bb544cd69843c P a634c60a5ca74b26b0d1f679d0166d0c [term 1 LEADER]: Becoming Leader. State: Replica: a634c60a5ca74b26b0d1f679d0166d0c, State: Running, Role: LEADER
I20260129 02:58:13.501473 20114 consensus_queue.cc:237] T 1bd5a7f0c64f4ef6bc2bb544cd69843c P a634c60a5ca74b26b0d1f679d0166d0c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eafca908701c410ab9488970fcb3d74e" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 45001 } } peers { permanent_uuid: "544851f9b7954fd2962675f957e8836a" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34199 } } peers { permanent_uuid: "a634c60a5ca74b26b0d1f679d0166d0c" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 39941 } }
I20260129 02:58:13.507553 19996 raft_consensus.cc:2468] T 901f619b02f843b4a2747de956182f12 P 544851f9b7954fd2962675f957e8836a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a634c60a5ca74b26b0d1f679d0166d0c in term 1.
I20260129 02:58:13.515806 19817 catalog_manager.cc:5654] T 901f619b02f843b4a2747de956182f12 P a634c60a5ca74b26b0d1f679d0166d0c reported cstate change: term changed from 0 to 1, leader changed from <none> to a634c60a5ca74b26b0d1f679d0166d0c (127.18.211.3). New cstate: current_term: 1 leader_uuid: "a634c60a5ca74b26b0d1f679d0166d0c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eafca908701c410ab9488970fcb3d74e" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 45001 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "544851f9b7954fd2962675f957e8836a" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34199 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a634c60a5ca74b26b0d1f679d0166d0c" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 39941 } health_report { overall_health: HEALTHY } } }
I20260129 02:58:13.527150 20132 consensus_queue.cc:1048] T 7d91c3094df94b47aa9a512d13490daa P a634c60a5ca74b26b0d1f679d0166d0c [LEADER]: Connected to new peer: Peer: permanent_uuid: "eafca908701c410ab9488970fcb3d74e" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 45001 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260129 02:58:13.527226 20133 consensus_queue.cc:1048] T d5edd0a419ee46c9a6a223f8056afb4c P eafca908701c410ab9488970fcb3d74e [LEADER]: Connected to new peer: Peer: permanent_uuid: "544851f9b7954fd2962675f957e8836a" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34199 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260129 02:58:13.527501 19817 catalog_manager.cc:5654] T 1bd5a7f0c64f4ef6bc2bb544cd69843c P a634c60a5ca74b26b0d1f679d0166d0c reported cstate change: term changed from 0 to 1, leader changed from <none> to a634c60a5ca74b26b0d1f679d0166d0c (127.18.211.3). New cstate: current_term: 1 leader_uuid: "a634c60a5ca74b26b0d1f679d0166d0c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eafca908701c410ab9488970fcb3d74e" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 45001 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "544851f9b7954fd2962675f957e8836a" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34199 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a634c60a5ca74b26b0d1f679d0166d0c" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 39941 } health_report { overall_health: HEALTHY } } }
I20260129 02:58:13.529342 20132 consensus_queue.cc:1048] T 7d91c3094df94b47aa9a512d13490daa P a634c60a5ca74b26b0d1f679d0166d0c [LEADER]: Connected to new peer: Peer: permanent_uuid: "544851f9b7954fd2962675f957e8836a" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34199 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260129 02:58:13.536715 20070 raft_consensus.cc:3060] T d5edd0a419ee46c9a6a223f8056afb4c P a634c60a5ca74b26b0d1f679d0166d0c [term 0 FOLLOWER]: Advancing to term 1
I20260129 02:58:13.550601 20133 consensus_queue.cc:1048] T d5edd0a419ee46c9a6a223f8056afb4c P eafca908701c410ab9488970fcb3d74e [LEADER]: Connected to new peer: Peer: permanent_uuid: "a634c60a5ca74b26b0d1f679d0166d0c" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 39941 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260129 02:58:13.620772 20174 consensus_queue.cc:1048] T 4562a703e5a44a1585b332b851bfd6f2 P a634c60a5ca74b26b0d1f679d0166d0c [LEADER]: Connected to new peer: Peer: permanent_uuid: "eafca908701c410ab9488970fcb3d74e" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 45001 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260129 02:58:13.621718 19994 raft_consensus.cc:3060] T 4562a703e5a44a1585b332b851bfd6f2 P 544851f9b7954fd2962675f957e8836a [term 0 FOLLOWER]: Advancing to term 1
I20260129 02:58:13.629520 20114 consensus_queue.cc:1048] T 4562a703e5a44a1585b332b851bfd6f2 P a634c60a5ca74b26b0d1f679d0166d0c [LEADER]: Connected to new peer: Peer: permanent_uuid: "544851f9b7954fd2962675f957e8836a" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34199 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260129 02:58:13.681871 20114 raft_consensus.cc:493] T 1dc801ccecc44ed79d85b0522e49d3e5 P a634c60a5ca74b26b0d1f679d0166d0c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260129 02:58:13.682367 20114 raft_consensus.cc:515] T 1dc801ccecc44ed79d85b0522e49d3e5 P a634c60a5ca74b26b0d1f679d0166d0c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eafca908701c410ab9488970fcb3d74e" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 45001 } } peers { permanent_uuid: "544851f9b7954fd2962675f957e8836a" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34199 } } peers { permanent_uuid: "a634c60a5ca74b26b0d1f679d0166d0c" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 39941 } }
I20260129 02:58:13.684015 20114 leader_election.cc:290] T 1dc801ccecc44ed79d85b0522e49d3e5 P a634c60a5ca74b26b0d1f679d0166d0c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers eafca908701c410ab9488970fcb3d74e (127.18.211.1:45001), 544851f9b7954fd2962675f957e8836a (127.18.211.2:34199)
I20260129 02:58:13.684942 19920 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1dc801ccecc44ed79d85b0522e49d3e5" candidate_uuid: "a634c60a5ca74b26b0d1f679d0166d0c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eafca908701c410ab9488970fcb3d74e" is_pre_election: true
I20260129 02:58:13.685222 19994 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1dc801ccecc44ed79d85b0522e49d3e5" candidate_uuid: "a634c60a5ca74b26b0d1f679d0166d0c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "544851f9b7954fd2962675f957e8836a" is_pre_election: true
I20260129 02:58:13.685745 19920 raft_consensus.cc:2393] T 1dc801ccecc44ed79d85b0522e49d3e5 P eafca908701c410ab9488970fcb3d74e [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a634c60a5ca74b26b0d1f679d0166d0c in current term 1: Already voted for candidate 544851f9b7954fd2962675f957e8836a in this term.
I20260129 02:58:13.686682 20033 leader_election.cc:304] T 1dc801ccecc44ed79d85b0522e49d3e5 P a634c60a5ca74b26b0d1f679d0166d0c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a634c60a5ca74b26b0d1f679d0166d0c; no voters: 544851f9b7954fd2962675f957e8836a, eafca908701c410ab9488970fcb3d74e
I20260129 02:58:13.687299 20114 raft_consensus.cc:3060] T 1dc801ccecc44ed79d85b0522e49d3e5 P a634c60a5ca74b26b0d1f679d0166d0c [term 0 FOLLOWER]: Advancing to term 1
I20260129 02:58:13.689073 20143 consensus_queue.cc:1048] T 620c6d80e1964c51a628ffb43f645798 P 544851f9b7954fd2962675f957e8836a [LEADER]: Connected to new peer: Peer: permanent_uuid: "a634c60a5ca74b26b0d1f679d0166d0c" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 39941 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260129 02:58:13.693408 20114 raft_consensus.cc:2749] T 1dc801ccecc44ed79d85b0522e49d3e5 P a634c60a5ca74b26b0d1f679d0166d0c [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260129 02:58:13.699307 20144 consensus_queue.cc:1048] T 620c6d80e1964c51a628ffb43f645798 P 544851f9b7954fd2962675f957e8836a [LEADER]: Connected to new peer: Peer: permanent_uuid: "eafca908701c410ab9488970fcb3d74e" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 45001 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260129 02:58:13.721954 20144 consensus_queue.cc:1048] T 1dc801ccecc44ed79d85b0522e49d3e5 P 544851f9b7954fd2962675f957e8836a [LEADER]: Connected to new peer: Peer: permanent_uuid: "a634c60a5ca74b26b0d1f679d0166d0c" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 39941 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260129 02:58:13.724318 20069 tablet_service.cc:2066] Received LeaderStepDown RPC: tablet_id: "7d91c3094df94b47aa9a512d13490daa"
dest_uuid: "a634c60a5ca74b26b0d1f679d0166d0c"
mode: GRACEFUL
new_leader_uuid: "eafca908701c410ab9488970fcb3d74e"
from {username='slave'} at 127.0.0.1:42438
I20260129 02:58:13.724895 20069 raft_consensus.cc:606] T 7d91c3094df94b47aa9a512d13490daa P a634c60a5ca74b26b0d1f679d0166d0c [term 1 LEADER]: Received request to transfer leadership to TS eafca908701c410ab9488970fcb3d74e
I20260129 02:58:13.729975 20069 tablet_service.cc:2066] Received LeaderStepDown RPC: tablet_id: "4562a703e5a44a1585b332b851bfd6f2"
dest_uuid: "a634c60a5ca74b26b0d1f679d0166d0c"
mode: GRACEFUL
new_leader_uuid: "eafca908701c410ab9488970fcb3d74e"
from {username='slave'} at 127.0.0.1:42438
I20260129 02:58:13.730464 20069 raft_consensus.cc:606] T 4562a703e5a44a1585b332b851bfd6f2 P a634c60a5ca74b26b0d1f679d0166d0c [term 1 LEADER]: Received request to transfer leadership to TS eafca908701c410ab9488970fcb3d74e
I20260129 02:58:13.738587 20144 consensus_queue.cc:1048] T 1dc801ccecc44ed79d85b0522e49d3e5 P 544851f9b7954fd2962675f957e8836a [LEADER]: Connected to new peer: Peer: permanent_uuid: "eafca908701c410ab9488970fcb3d74e" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 45001 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260129 02:58:13.744346 20070 tablet_service.cc:2066] Received LeaderStepDown RPC: tablet_id: "79355f4f5d7942b6ac6f420256fd35b7"
dest_uuid: "a634c60a5ca74b26b0d1f679d0166d0c"
mode: GRACEFUL
new_leader_uuid: "eafca908701c410ab9488970fcb3d74e"
from {username='slave'} at 127.0.0.1:42438
I20260129 02:58:13.747466 20070 raft_consensus.cc:606] T 79355f4f5d7942b6ac6f420256fd35b7 P a634c60a5ca74b26b0d1f679d0166d0c [term 1 LEADER]: Received request to transfer leadership to TS eafca908701c410ab9488970fcb3d74e
I20260129 02:58:13.757354 20070 tablet_service.cc:2066] Received LeaderStepDown RPC: tablet_id: "901f619b02f843b4a2747de956182f12"
dest_uuid: "a634c60a5ca74b26b0d1f679d0166d0c"
mode: GRACEFUL
new_leader_uuid: "544851f9b7954fd2962675f957e8836a"
from {username='slave'} at 127.0.0.1:42438
I20260129 02:58:13.757840 20070 raft_consensus.cc:606] T 901f619b02f843b4a2747de956182f12 P a634c60a5ca74b26b0d1f679d0166d0c [term 1 LEADER]: Received request to transfer leadership to TS 544851f9b7954fd2962675f957e8836a
I20260129 02:58:13.826781 20180 consensus_queue.cc:1048] T 47ad34405a6f4d508c9613c07a572cc3 P eafca908701c410ab9488970fcb3d74e [LEADER]: Connected to new peer: Peer: permanent_uuid: "a634c60a5ca74b26b0d1f679d0166d0c" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 39941 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260129 02:58:13.836022 20116 consensus_queue.cc:1048] T 47ad34405a6f4d508c9613c07a572cc3 P eafca908701c410ab9488970fcb3d74e [LEADER]: Connected to new peer: Peer: permanent_uuid: "544851f9b7954fd2962675f957e8836a" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34199 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260129 02:58:13.888706 20114 consensus_queue.cc:1048] T 79355f4f5d7942b6ac6f420256fd35b7 P a634c60a5ca74b26b0d1f679d0166d0c [LEADER]: Connected to new peer: Peer: permanent_uuid: "eafca908701c410ab9488970fcb3d74e" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 45001 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260129 02:58:13.894527 20131 raft_consensus.cc:993] T 79355f4f5d7942b6ac6f420256fd35b7 P a634c60a5ca74b26b0d1f679d0166d0c: : Instructing follower eafca908701c410ab9488970fcb3d74e to start an election
I20260129 02:58:13.895005 20174 raft_consensus.cc:1081] T 79355f4f5d7942b6ac6f420256fd35b7 P a634c60a5ca74b26b0d1f679d0166d0c [term 1 LEADER]: Signalling peer eafca908701c410ab9488970fcb3d74e to start an election
I20260129 02:58:13.896659 19919 tablet_service.cc:2038] Received Run Leader Election RPC: tablet_id: "79355f4f5d7942b6ac6f420256fd35b7"
dest_uuid: "eafca908701c410ab9488970fcb3d74e"
from {username='slave'} at 127.0.0.1:50448
I20260129 02:58:13.897256 19919 raft_consensus.cc:493] T 79355f4f5d7942b6ac6f420256fd35b7 P eafca908701c410ab9488970fcb3d74e [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260129 02:58:13.897605 19919 raft_consensus.cc:3060] T 79355f4f5d7942b6ac6f420256fd35b7 P eafca908701c410ab9488970fcb3d74e [term 1 FOLLOWER]: Advancing to term 2
I20260129 02:58:13.900818 20131 consensus_queue.cc:1048] T 79355f4f5d7942b6ac6f420256fd35b7 P a634c60a5ca74b26b0d1f679d0166d0c [LEADER]: Connected to new peer: Peer: permanent_uuid: "544851f9b7954fd2962675f957e8836a" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34199 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260129 02:58:13.905139 19919 raft_consensus.cc:515] T 79355f4f5d7942b6ac6f420256fd35b7 P eafca908701c410ab9488970fcb3d74e [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eafca908701c410ab9488970fcb3d74e" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 45001 } } peers { permanent_uuid: "544851f9b7954fd2962675f957e8836a" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34199 } } peers { permanent_uuid: "a634c60a5ca74b26b0d1f679d0166d0c" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 39941 } }
I20260129 02:58:13.910853 19994 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "79355f4f5d7942b6ac6f420256fd35b7" candidate_uuid: "eafca908701c410ab9488970fcb3d74e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "544851f9b7954fd2962675f957e8836a"
I20260129 02:58:13.911340 19919 leader_election.cc:290] T 79355f4f5d7942b6ac6f420256fd35b7 P eafca908701c410ab9488970fcb3d74e [CANDIDATE]: Term 2 election: Requested vote from peers 544851f9b7954fd2962675f957e8836a (127.18.211.2:34199), a634c60a5ca74b26b0d1f679d0166d0c (127.18.211.3:39941)
I20260129 02:58:13.911396 19994 raft_consensus.cc:3060] T 79355f4f5d7942b6ac6f420256fd35b7 P 544851f9b7954fd2962675f957e8836a [term 1 FOLLOWER]: Advancing to term 2
I20260129 02:58:13.911580 20070 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "79355f4f5d7942b6ac6f420256fd35b7" candidate_uuid: "eafca908701c410ab9488970fcb3d74e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "a634c60a5ca74b26b0d1f679d0166d0c"
I20260129 02:58:13.912962 20070 raft_consensus.cc:3055] T 79355f4f5d7942b6ac6f420256fd35b7 P a634c60a5ca74b26b0d1f679d0166d0c [term 1 LEADER]: Stepping down as leader of term 1
I20260129 02:58:13.913347 20070 raft_consensus.cc:740] T 79355f4f5d7942b6ac6f420256fd35b7 P a634c60a5ca74b26b0d1f679d0166d0c [term 1 LEADER]: Becoming Follower/Learner. State: Replica: a634c60a5ca74b26b0d1f679d0166d0c, State: Running, Role: LEADER
I20260129 02:58:13.914115 20070 consensus_queue.cc:260] T 79355f4f5d7942b6ac6f420256fd35b7 P a634c60a5ca74b26b0d1f679d0166d0c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eafca908701c410ab9488970fcb3d74e" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 45001 } } peers { permanent_uuid: "544851f9b7954fd2962675f957e8836a" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34199 } } peers { permanent_uuid: "a634c60a5ca74b26b0d1f679d0166d0c" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 39941 } }
I20260129 02:58:13.915223 20070 raft_consensus.cc:3060] T 79355f4f5d7942b6ac6f420256fd35b7 P a634c60a5ca74b26b0d1f679d0166d0c [term 1 FOLLOWER]: Advancing to term 2
I20260129 02:58:13.916911 19994 raft_consensus.cc:2468] T 79355f4f5d7942b6ac6f420256fd35b7 P 544851f9b7954fd2962675f957e8836a [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eafca908701c410ab9488970fcb3d74e in term 2.
I20260129 02:58:13.917798 19884 leader_election.cc:304] T 79355f4f5d7942b6ac6f420256fd35b7 P eafca908701c410ab9488970fcb3d74e [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 544851f9b7954fd2962675f957e8836a, eafca908701c410ab9488970fcb3d74e; no voters:
I20260129 02:58:13.918569 20116 raft_consensus.cc:2804] T 79355f4f5d7942b6ac6f420256fd35b7 P eafca908701c410ab9488970fcb3d74e [term 2 FOLLOWER]: Leader election won for term 2
I20260129 02:58:13.919024 20116 raft_consensus.cc:697] T 79355f4f5d7942b6ac6f420256fd35b7 P eafca908701c410ab9488970fcb3d74e [term 2 LEADER]: Becoming Leader. State: Replica: eafca908701c410ab9488970fcb3d74e, State: Running, Role: LEADER
I20260129 02:58:13.919783 20116 consensus_queue.cc:237] T 79355f4f5d7942b6ac6f420256fd35b7 P eafca908701c410ab9488970fcb3d74e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eafca908701c410ab9488970fcb3d74e" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 45001 } } peers { permanent_uuid: "544851f9b7954fd2962675f957e8836a" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34199 } } peers { permanent_uuid: "a634c60a5ca74b26b0d1f679d0166d0c" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 39941 } }
I20260129 02:58:13.921378 20131 consensus_queue.cc:1048] T 1bd5a7f0c64f4ef6bc2bb544cd69843c P a634c60a5ca74b26b0d1f679d0166d0c [LEADER]: Connected to new peer: Peer: permanent_uuid: "544851f9b7954fd2962675f957e8836a" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34199 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260129 02:58:13.922178 20070 raft_consensus.cc:2468] T 79355f4f5d7942b6ac6f420256fd35b7 P a634c60a5ca74b26b0d1f679d0166d0c [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eafca908701c410ab9488970fcb3d74e in term 2.
I20260129 02:58:13.927155 19816 catalog_manager.cc:5654] T 79355f4f5d7942b6ac6f420256fd35b7 P eafca908701c410ab9488970fcb3d74e reported cstate change: term changed from 1 to 2, leader changed from a634c60a5ca74b26b0d1f679d0166d0c (127.18.211.3) to eafca908701c410ab9488970fcb3d74e (127.18.211.1). New cstate: current_term: 2 leader_uuid: "eafca908701c410ab9488970fcb3d74e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eafca908701c410ab9488970fcb3d74e" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 45001 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "544851f9b7954fd2962675f957e8836a" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34199 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a634c60a5ca74b26b0d1f679d0166d0c" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 39941 } health_report { overall_health: UNKNOWN } } }
I20260129 02:58:13.932060 20114 consensus_queue.cc:1048] T 1bd5a7f0c64f4ef6bc2bb544cd69843c P a634c60a5ca74b26b0d1f679d0166d0c [LEADER]: Connected to new peer: Peer: permanent_uuid: "eafca908701c410ab9488970fcb3d74e" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 45001 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260129 02:58:13.939972 20174 raft_consensus.cc:993] T 7d91c3094df94b47aa9a512d13490daa P a634c60a5ca74b26b0d1f679d0166d0c: : Instructing follower eafca908701c410ab9488970fcb3d74e to start an election
I20260129 02:58:13.940429 20174 raft_consensus.cc:1081] T 7d91c3094df94b47aa9a512d13490daa P a634c60a5ca74b26b0d1f679d0166d0c [term 1 LEADER]: Signalling peer eafca908701c410ab9488970fcb3d74e to start an election
I20260129 02:58:13.942163 19920 tablet_service.cc:2038] Received Run Leader Election RPC: tablet_id: "7d91c3094df94b47aa9a512d13490daa"
dest_uuid: "eafca908701c410ab9488970fcb3d74e"
from {username='slave'} at 127.0.0.1:50448
I20260129 02:58:13.942730 19920 raft_consensus.cc:493] T 7d91c3094df94b47aa9a512d13490daa P eafca908701c410ab9488970fcb3d74e [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260129 02:58:13.942968 19920 raft_consensus.cc:3060] T 7d91c3094df94b47aa9a512d13490daa P eafca908701c410ab9488970fcb3d74e [term 1 FOLLOWER]: Advancing to term 2
I20260129 02:58:13.948197 19920 raft_consensus.cc:515] T 7d91c3094df94b47aa9a512d13490daa P eafca908701c410ab9488970fcb3d74e [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a634c60a5ca74b26b0d1f679d0166d0c" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 39941 } } peers { permanent_uuid: "eafca908701c410ab9488970fcb3d74e" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 45001 } } peers { permanent_uuid: "544851f9b7954fd2962675f957e8836a" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34199 } }
I20260129 02:58:13.949927 19920 leader_election.cc:290] T 7d91c3094df94b47aa9a512d13490daa P eafca908701c410ab9488970fcb3d74e [CANDIDATE]: Term 2 election: Requested vote from peers a634c60a5ca74b26b0d1f679d0166d0c (127.18.211.3:39941), 544851f9b7954fd2962675f957e8836a (127.18.211.2:34199)
I20260129 02:58:13.950655 20070 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7d91c3094df94b47aa9a512d13490daa" candidate_uuid: "eafca908701c410ab9488970fcb3d74e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "a634c60a5ca74b26b0d1f679d0166d0c"
I20260129 02:58:13.950960 19995 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7d91c3094df94b47aa9a512d13490daa" candidate_uuid: "eafca908701c410ab9488970fcb3d74e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "544851f9b7954fd2962675f957e8836a"
I20260129 02:58:13.951225 20070 raft_consensus.cc:3055] T 7d91c3094df94b47aa9a512d13490daa P a634c60a5ca74b26b0d1f679d0166d0c [term 1 LEADER]: Stepping down as leader of term 1
I20260129 02:58:13.951543 19995 raft_consensus.cc:3060] T 7d91c3094df94b47aa9a512d13490daa P 544851f9b7954fd2962675f957e8836a [term 1 FOLLOWER]: Advancing to term 2
I20260129 02:58:13.951592 20070 raft_consensus.cc:740] T 7d91c3094df94b47aa9a512d13490daa P a634c60a5ca74b26b0d1f679d0166d0c [term 1 LEADER]: Becoming Follower/Learner. State: Replica: a634c60a5ca74b26b0d1f679d0166d0c, State: Running, Role: LEADER
I20260129 02:58:13.952477 20070 consensus_queue.cc:260] T 7d91c3094df94b47aa9a512d13490daa P a634c60a5ca74b26b0d1f679d0166d0c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a634c60a5ca74b26b0d1f679d0166d0c" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 39941 } } peers { permanent_uuid: "eafca908701c410ab9488970fcb3d74e" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 45001 } } peers { permanent_uuid: "544851f9b7954fd2962675f957e8836a" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34199 } }
I20260129 02:58:13.953533 20070 raft_consensus.cc:3060] T 7d91c3094df94b47aa9a512d13490daa P a634c60a5ca74b26b0d1f679d0166d0c [term 1 FOLLOWER]: Advancing to term 2
I20260129 02:58:13.957249 19995 raft_consensus.cc:2468] T 7d91c3094df94b47aa9a512d13490daa P 544851f9b7954fd2962675f957e8836a [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eafca908701c410ab9488970fcb3d74e in term 2.
I20260129 02:58:13.958209 19884 leader_election.cc:304] T 7d91c3094df94b47aa9a512d13490daa P eafca908701c410ab9488970fcb3d74e [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 544851f9b7954fd2962675f957e8836a, eafca908701c410ab9488970fcb3d74e; no voters:
I20260129 02:58:13.958637 20070 raft_consensus.cc:2468] T 7d91c3094df94b47aa9a512d13490daa P a634c60a5ca74b26b0d1f679d0166d0c [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eafca908701c410ab9488970fcb3d74e in term 2.
I20260129 02:58:13.958842 20116 raft_consensus.cc:2804] T 7d91c3094df94b47aa9a512d13490daa P eafca908701c410ab9488970fcb3d74e [term 2 FOLLOWER]: Leader election won for term 2
I20260129 02:58:13.959362 20116 raft_consensus.cc:697] T 7d91c3094df94b47aa9a512d13490daa P eafca908701c410ab9488970fcb3d74e [term 2 LEADER]: Becoming Leader. State: Replica: eafca908701c410ab9488970fcb3d74e, State: Running, Role: LEADER
I20260129 02:58:13.960176 20116 consensus_queue.cc:237] T 7d91c3094df94b47aa9a512d13490daa P eafca908701c410ab9488970fcb3d74e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a634c60a5ca74b26b0d1f679d0166d0c" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 39941 } } peers { permanent_uuid: "eafca908701c410ab9488970fcb3d74e" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 45001 } } peers { permanent_uuid: "544851f9b7954fd2962675f957e8836a" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34199 } }
I20260129 02:58:13.965853 19816 catalog_manager.cc:5654] T 7d91c3094df94b47aa9a512d13490daa P eafca908701c410ab9488970fcb3d74e reported cstate change: term changed from 1 to 2, leader changed from a634c60a5ca74b26b0d1f679d0166d0c (127.18.211.3) to eafca908701c410ab9488970fcb3d74e (127.18.211.1). New cstate: current_term: 2 leader_uuid: "eafca908701c410ab9488970fcb3d74e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a634c60a5ca74b26b0d1f679d0166d0c" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 39941 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "eafca908701c410ab9488970fcb3d74e" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 45001 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "544851f9b7954fd2962675f957e8836a" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34199 } health_report { overall_health: UNKNOWN } } }
I20260129 02:58:14.008132 20131 consensus_queue.cc:1048] T 901f619b02f843b4a2747de956182f12 P a634c60a5ca74b26b0d1f679d0166d0c [LEADER]: Connected to new peer: Peer: permanent_uuid: "eafca908701c410ab9488970fcb3d74e" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 45001 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260129 02:58:14.021306 20131 consensus_queue.cc:1048] T 901f619b02f843b4a2747de956182f12 P a634c60a5ca74b26b0d1f679d0166d0c [LEADER]: Connected to new peer: Peer: permanent_uuid: "544851f9b7954fd2962675f957e8836a" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34199 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260129 02:58:14.027042 20188 consensus_queue.cc:1048] T 5d11df549a83429a87d3e7a62fabaa21 P 544851f9b7954fd2962675f957e8836a [LEADER]: Connected to new peer: Peer: permanent_uuid: "a634c60a5ca74b26b0d1f679d0166d0c" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 39941 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260129 02:58:14.032444 20132 raft_consensus.cc:993] T 901f619b02f843b4a2747de956182f12 P a634c60a5ca74b26b0d1f679d0166d0c: : Instructing follower 544851f9b7954fd2962675f957e8836a to start an election
I20260129 02:58:14.032912 20132 raft_consensus.cc:1081] T 901f619b02f843b4a2747de956182f12 P a634c60a5ca74b26b0d1f679d0166d0c [term 1 LEADER]: Signalling peer 544851f9b7954fd2962675f957e8836a to start an election
I20260129 02:58:14.034622 19995 tablet_service.cc:2038] Received Run Leader Election RPC: tablet_id: "901f619b02f843b4a2747de956182f12"
dest_uuid: "544851f9b7954fd2962675f957e8836a"
from {username='slave'} at 127.0.0.1:35886
I20260129 02:58:14.035084 19995 raft_consensus.cc:493] T 901f619b02f843b4a2747de956182f12 P 544851f9b7954fd2962675f957e8836a [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260129 02:58:14.035454 19995 raft_consensus.cc:3060] T 901f619b02f843b4a2747de956182f12 P 544851f9b7954fd2962675f957e8836a [term 1 FOLLOWER]: Advancing to term 2
I20260129 02:58:14.040473 20188 consensus_queue.cc:1048] T 5d11df549a83429a87d3e7a62fabaa21 P 544851f9b7954fd2962675f957e8836a [LEADER]: Connected to new peer: Peer: permanent_uuid: "eafca908701c410ab9488970fcb3d74e" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 45001 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260129 02:58:14.042676 19995 raft_consensus.cc:515] T 901f619b02f843b4a2747de956182f12 P 544851f9b7954fd2962675f957e8836a [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eafca908701c410ab9488970fcb3d74e" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 45001 } } peers { permanent_uuid: "544851f9b7954fd2962675f957e8836a" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34199 } } peers { permanent_uuid: "a634c60a5ca74b26b0d1f679d0166d0c" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 39941 } }
I20260129 02:58:14.045399 19919 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "901f619b02f843b4a2747de956182f12" candidate_uuid: "544851f9b7954fd2962675f957e8836a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "eafca908701c410ab9488970fcb3d74e"
I20260129 02:58:14.045670 20070 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "901f619b02f843b4a2747de956182f12" candidate_uuid: "544851f9b7954fd2962675f957e8836a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "a634c60a5ca74b26b0d1f679d0166d0c"
I20260129 02:58:14.045964 19919 raft_consensus.cc:3060] T 901f619b02f843b4a2747de956182f12 P eafca908701c410ab9488970fcb3d74e [term 1 FOLLOWER]: Advancing to term 2
I20260129 02:58:14.046278 20070 raft_consensus.cc:3055] T 901f619b02f843b4a2747de956182f12 P a634c60a5ca74b26b0d1f679d0166d0c [term 1 LEADER]: Stepping down as leader of term 1
I20260129 02:58:14.046610 20070 raft_consensus.cc:740] T 901f619b02f843b4a2747de956182f12 P a634c60a5ca74b26b0d1f679d0166d0c [term 1 LEADER]: Becoming Follower/Learner. State: Replica: a634c60a5ca74b26b0d1f679d0166d0c, State: Running, Role: LEADER
I20260129 02:58:14.047408 20070 consensus_queue.cc:260] T 901f619b02f843b4a2747de956182f12 P a634c60a5ca74b26b0d1f679d0166d0c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eafca908701c410ab9488970fcb3d74e" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 45001 } } peers { permanent_uuid: "544851f9b7954fd2962675f957e8836a" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34199 } } peers { permanent_uuid: "a634c60a5ca74b26b0d1f679d0166d0c" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 39941 } }
I20260129 02:58:14.048466 20070 raft_consensus.cc:3060] T 901f619b02f843b4a2747de956182f12 P a634c60a5ca74b26b0d1f679d0166d0c [term 1 FOLLOWER]: Advancing to term 2
I20260129 02:58:14.055110 20070 raft_consensus.cc:2468] T 901f619b02f843b4a2747de956182f12 P a634c60a5ca74b26b0d1f679d0166d0c [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 544851f9b7954fd2962675f957e8836a in term 2.
I20260129 02:58:14.056185 19960 leader_election.cc:304] T 901f619b02f843b4a2747de956182f12 P 544851f9b7954fd2962675f957e8836a [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 544851f9b7954fd2962675f957e8836a, a634c60a5ca74b26b0d1f679d0166d0c; no voters:
I20260129 02:58:14.056918 19995 leader_election.cc:290] T 901f619b02f843b4a2747de956182f12 P 544851f9b7954fd2962675f957e8836a [CANDIDATE]: Term 2 election: Requested vote from peers eafca908701c410ab9488970fcb3d74e (127.18.211.1:45001), a634c60a5ca74b26b0d1f679d0166d0c (127.18.211.3:39941)
I20260129 02:58:14.057433 20188 raft_consensus.cc:2804] T 901f619b02f843b4a2747de956182f12 P 544851f9b7954fd2962675f957e8836a [term 2 FOLLOWER]: Leader election won for term 2
I20260129 02:58:14.058231 20188 raft_consensus.cc:697] T 901f619b02f843b4a2747de956182f12 P 544851f9b7954fd2962675f957e8836a [term 2 LEADER]: Becoming Leader. State: Replica: 544851f9b7954fd2962675f957e8836a, State: Running, Role: LEADER
I20260129 02:58:14.060747 20188 consensus_queue.cc:237] T 901f619b02f843b4a2747de956182f12 P 544851f9b7954fd2962675f957e8836a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eafca908701c410ab9488970fcb3d74e" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 45001 } } peers { permanent_uuid: "544851f9b7954fd2962675f957e8836a" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34199 } } peers { permanent_uuid: "a634c60a5ca74b26b0d1f679d0166d0c" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 39941 } }
I20260129 02:58:14.061340 19919 raft_consensus.cc:2468] T 901f619b02f843b4a2747de956182f12 P eafca908701c410ab9488970fcb3d74e [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 544851f9b7954fd2962675f957e8836a in term 2.
I20260129 02:58:14.067337 19816 catalog_manager.cc:5654] T 901f619b02f843b4a2747de956182f12 P 544851f9b7954fd2962675f957e8836a reported cstate change: term changed from 1 to 2, leader changed from a634c60a5ca74b26b0d1f679d0166d0c (127.18.211.3) to 544851f9b7954fd2962675f957e8836a (127.18.211.2). New cstate: current_term: 2 leader_uuid: "544851f9b7954fd2962675f957e8836a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eafca908701c410ab9488970fcb3d74e" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 45001 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "544851f9b7954fd2962675f957e8836a" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34199 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a634c60a5ca74b26b0d1f679d0166d0c" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 39941 } health_report { overall_health: UNKNOWN } } }
I20260129 02:58:14.228235 20174 raft_consensus.cc:993] T 4562a703e5a44a1585b332b851bfd6f2 P a634c60a5ca74b26b0d1f679d0166d0c: : Instructing follower eafca908701c410ab9488970fcb3d74e to start an election
I20260129 02:58:14.228657 20132 raft_consensus.cc:1081] T 4562a703e5a44a1585b332b851bfd6f2 P a634c60a5ca74b26b0d1f679d0166d0c [term 1 LEADER]: Signalling peer eafca908701c410ab9488970fcb3d74e to start an election
I20260129 02:58:14.229929 19919 tablet_service.cc:2038] Received Run Leader Election RPC: tablet_id: "4562a703e5a44a1585b332b851bfd6f2"
dest_uuid: "eafca908701c410ab9488970fcb3d74e"
from {username='slave'} at 127.0.0.1:50448
I20260129 02:58:14.230324 19919 raft_consensus.cc:493] T 4562a703e5a44a1585b332b851bfd6f2 P eafca908701c410ab9488970fcb3d74e [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260129 02:58:14.230561 19919 raft_consensus.cc:3060] T 4562a703e5a44a1585b332b851bfd6f2 P eafca908701c410ab9488970fcb3d74e [term 1 FOLLOWER]: Advancing to term 2
I20260129 02:58:14.234841 19919 raft_consensus.cc:515] T 4562a703e5a44a1585b332b851bfd6f2 P eafca908701c410ab9488970fcb3d74e [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eafca908701c410ab9488970fcb3d74e" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 45001 } } peers { permanent_uuid: "544851f9b7954fd2962675f957e8836a" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34199 } } peers { permanent_uuid: "a634c60a5ca74b26b0d1f679d0166d0c" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 39941 } }
I20260129 02:58:14.236327 19919 leader_election.cc:290] T 4562a703e5a44a1585b332b851bfd6f2 P eafca908701c410ab9488970fcb3d74e [CANDIDATE]: Term 2 election: Requested vote from peers 544851f9b7954fd2962675f957e8836a (127.18.211.2:34199), a634c60a5ca74b26b0d1f679d0166d0c (127.18.211.3:39941)
I20260129 02:58:14.237097 19995 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4562a703e5a44a1585b332b851bfd6f2" candidate_uuid: "eafca908701c410ab9488970fcb3d74e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "544851f9b7954fd2962675f957e8836a"
I20260129 02:58:14.237385 20070 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4562a703e5a44a1585b332b851bfd6f2" candidate_uuid: "eafca908701c410ab9488970fcb3d74e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "a634c60a5ca74b26b0d1f679d0166d0c"
I20260129 02:58:14.237586 19995 raft_consensus.cc:3060] T 4562a703e5a44a1585b332b851bfd6f2 P 544851f9b7954fd2962675f957e8836a [term 1 FOLLOWER]: Advancing to term 2
I20260129 02:58:14.237923 20070 raft_consensus.cc:3055] T 4562a703e5a44a1585b332b851bfd6f2 P a634c60a5ca74b26b0d1f679d0166d0c [term 1 LEADER]: Stepping down as leader of term 1
I20260129 02:58:14.238173 20070 raft_consensus.cc:740] T 4562a703e5a44a1585b332b851bfd6f2 P a634c60a5ca74b26b0d1f679d0166d0c [term 1 LEADER]: Becoming Follower/Learner. State: Replica: a634c60a5ca74b26b0d1f679d0166d0c, State: Running, Role: LEADER
I20260129 02:58:14.238726 20070 consensus_queue.cc:260] T 4562a703e5a44a1585b332b851bfd6f2 P a634c60a5ca74b26b0d1f679d0166d0c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eafca908701c410ab9488970fcb3d74e" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 45001 } } peers { permanent_uuid: "544851f9b7954fd2962675f957e8836a" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34199 } } peers { permanent_uuid: "a634c60a5ca74b26b0d1f679d0166d0c" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 39941 } }
I20260129 02:58:14.239540 20070 raft_consensus.cc:3060] T 4562a703e5a44a1585b332b851bfd6f2 P a634c60a5ca74b26b0d1f679d0166d0c [term 1 FOLLOWER]: Advancing to term 2
I20260129 02:58:14.242127 19995 raft_consensus.cc:2468] T 4562a703e5a44a1585b332b851bfd6f2 P 544851f9b7954fd2962675f957e8836a [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eafca908701c410ab9488970fcb3d74e in term 2.
I20260129 02:58:14.243088 19884 leader_election.cc:304] T 4562a703e5a44a1585b332b851bfd6f2 P eafca908701c410ab9488970fcb3d74e [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 544851f9b7954fd2962675f957e8836a, eafca908701c410ab9488970fcb3d74e; no voters:
I20260129 02:58:14.243942 20116 raft_consensus.cc:2804] T 4562a703e5a44a1585b332b851bfd6f2 P eafca908701c410ab9488970fcb3d74e [term 2 FOLLOWER]: Leader election won for term 2
I20260129 02:58:14.244334 20070 raft_consensus.cc:2468] T 4562a703e5a44a1585b332b851bfd6f2 P a634c60a5ca74b26b0d1f679d0166d0c [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eafca908701c410ab9488970fcb3d74e in term 2.
I20260129 02:58:14.244366 20116 raft_consensus.cc:697] T 4562a703e5a44a1585b332b851bfd6f2 P eafca908701c410ab9488970fcb3d74e [term 2 LEADER]: Becoming Leader. State: Replica: eafca908701c410ab9488970fcb3d74e, State: Running, Role: LEADER
I20260129 02:58:14.245267 20116 consensus_queue.cc:237] T 4562a703e5a44a1585b332b851bfd6f2 P eafca908701c410ab9488970fcb3d74e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eafca908701c410ab9488970fcb3d74e" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 45001 } } peers { permanent_uuid: "544851f9b7954fd2962675f957e8836a" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34199 } } peers { permanent_uuid: "a634c60a5ca74b26b0d1f679d0166d0c" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 39941 } }
I20260129 02:58:14.251051 19816 catalog_manager.cc:5654] T 4562a703e5a44a1585b332b851bfd6f2 P eafca908701c410ab9488970fcb3d74e reported cstate change: term changed from 1 to 2, leader changed from a634c60a5ca74b26b0d1f679d0166d0c (127.18.211.3) to eafca908701c410ab9488970fcb3d74e (127.18.211.1). New cstate: current_term: 2 leader_uuid: "eafca908701c410ab9488970fcb3d74e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eafca908701c410ab9488970fcb3d74e" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 45001 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "544851f9b7954fd2962675f957e8836a" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34199 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a634c60a5ca74b26b0d1f679d0166d0c" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 39941 } health_report { overall_health: UNKNOWN } } }
I20260129 02:58:14.426676 20070 raft_consensus.cc:1275] T 79355f4f5d7942b6ac6f420256fd35b7 P a634c60a5ca74b26b0d1f679d0166d0c [term 2 FOLLOWER]: Refusing update from remote peer eafca908701c410ab9488970fcb3d74e: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260129 02:58:14.427927 20116 consensus_queue.cc:1048] T 79355f4f5d7942b6ac6f420256fd35b7 P eafca908701c410ab9488970fcb3d74e [LEADER]: Connected to new peer: Peer: permanent_uuid: "a634c60a5ca74b26b0d1f679d0166d0c" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 39941 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260129 02:58:14.435032 19995 raft_consensus.cc:1275] T 79355f4f5d7942b6ac6f420256fd35b7 P 544851f9b7954fd2962675f957e8836a [term 2 FOLLOWER]: Refusing update from remote peer eafca908701c410ab9488970fcb3d74e: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260129 02:58:14.438751 20180 consensus_queue.cc:1048] T 79355f4f5d7942b6ac6f420256fd35b7 P eafca908701c410ab9488970fcb3d74e [LEADER]: Connected to new peer: Peer: permanent_uuid: "544851f9b7954fd2962675f957e8836a" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34199 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260129 02:58:14.440639 19995 raft_consensus.cc:1275] T 7d91c3094df94b47aa9a512d13490daa P 544851f9b7954fd2962675f957e8836a [term 2 FOLLOWER]: Refusing update from remote peer eafca908701c410ab9488970fcb3d74e: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260129 02:58:14.442729 20200 consensus_queue.cc:1048] T 7d91c3094df94b47aa9a512d13490daa P eafca908701c410ab9488970fcb3d74e [LEADER]: Connected to new peer: Peer: permanent_uuid: "544851f9b7954fd2962675f957e8836a" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34199 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260129 02:58:14.452037 20070 raft_consensus.cc:1275] T 7d91c3094df94b47aa9a512d13490daa P a634c60a5ca74b26b0d1f679d0166d0c [term 2 FOLLOWER]: Refusing update from remote peer eafca908701c410ab9488970fcb3d74e: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260129 02:58:14.453254 20116 consensus_queue.cc:1048] T 7d91c3094df94b47aa9a512d13490daa P eafca908701c410ab9488970fcb3d74e [LEADER]: Connected to new peer: Peer: permanent_uuid: "a634c60a5ca74b26b0d1f679d0166d0c" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 39941 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260129 02:58:14.648768 20070 raft_consensus.cc:1275] T 901f619b02f843b4a2747de956182f12 P a634c60a5ca74b26b0d1f679d0166d0c [term 2 FOLLOWER]: Refusing update from remote peer 544851f9b7954fd2962675f957e8836a: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260129 02:58:14.650231 20143 consensus_queue.cc:1048] T 901f619b02f843b4a2747de956182f12 P 544851f9b7954fd2962675f957e8836a [LEADER]: Connected to new peer: Peer: permanent_uuid: "a634c60a5ca74b26b0d1f679d0166d0c" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 39941 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260129 02:58:14.654477 19919 raft_consensus.cc:1275] T 901f619b02f843b4a2747de956182f12 P eafca908701c410ab9488970fcb3d74e [term 2 FOLLOWER]: Refusing update from remote peer 544851f9b7954fd2962675f957e8836a: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260129 02:58:14.656805 20143 consensus_queue.cc:1048] T 901f619b02f843b4a2747de956182f12 P 544851f9b7954fd2962675f957e8836a [LEADER]: Connected to new peer: Peer: permanent_uuid: "eafca908701c410ab9488970fcb3d74e" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 45001 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260129 02:58:14.780648 19995 raft_consensus.cc:1275] T 4562a703e5a44a1585b332b851bfd6f2 P 544851f9b7954fd2962675f957e8836a [term 2 FOLLOWER]: Refusing update from remote peer eafca908701c410ab9488970fcb3d74e: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260129 02:58:14.781745 20116 consensus_queue.cc:1048] T 4562a703e5a44a1585b332b851bfd6f2 P eafca908701c410ab9488970fcb3d74e [LEADER]: Connected to new peer: Peer: permanent_uuid: "544851f9b7954fd2962675f957e8836a" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34199 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260129 02:58:14.789796 20070 raft_consensus.cc:1275] T 4562a703e5a44a1585b332b851bfd6f2 P a634c60a5ca74b26b0d1f679d0166d0c [term 2 FOLLOWER]: Refusing update from remote peer eafca908701c410ab9488970fcb3d74e: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260129 02:58:14.791556 20200 consensus_queue.cc:1048] T 4562a703e5a44a1585b332b851bfd6f2 P eafca908701c410ab9488970fcb3d74e [LEADER]: Connected to new peer: Peer: permanent_uuid: "a634c60a5ca74b26b0d1f679d0166d0c" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 39941 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260129 02:58:16.796329 19276 auto_leader_rebalancer-test.cc:343] The leader distribution is
I20260129 02:58:16.797008 19276 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260129 02:58:16.814749 19920 tablet_service.cc:2066] Received LeaderStepDown RPC: tablet_id: "4562a703e5a44a1585b332b851bfd6f2"
dest_uuid: "eafca908701c410ab9488970fcb3d74e"
mode: GRACEFUL
new_leader_uuid: "a634c60a5ca74b26b0d1f679d0166d0c"
from {username='slave'} at 127.0.0.1:57002
I20260129 02:58:16.815203 19920 raft_consensus.cc:606] T 4562a703e5a44a1585b332b851bfd6f2 P eafca908701c410ab9488970fcb3d74e [term 2 LEADER]: Received request to transfer leadership to TS a634c60a5ca74b26b0d1f679d0166d0c
I20260129 02:58:16.817200 19920 tablet_service.cc:2066] Received LeaderStepDown RPC: tablet_id: "d5edd0a419ee46c9a6a223f8056afb4c"
dest_uuid: "eafca908701c410ab9488970fcb3d74e"
mode: GRACEFUL
new_leader_uuid: "a634c60a5ca74b26b0d1f679d0166d0c"
from {username='slave'} at 127.0.0.1:57002
I20260129 02:58:16.817581 19920 raft_consensus.cc:606] T d5edd0a419ee46c9a6a223f8056afb4c P eafca908701c410ab9488970fcb3d74e [term 1 LEADER]: Received request to transfer leadership to TS a634c60a5ca74b26b0d1f679d0166d0c
I20260129 02:58:16.818406 19276 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 2
I20260129 02:58:16.818825 19276 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20260129 02:58:16.874567 20116 raft_consensus.cc:993] T 4562a703e5a44a1585b332b851bfd6f2 P eafca908701c410ab9488970fcb3d74e: : Instructing follower a634c60a5ca74b26b0d1f679d0166d0c to start an election
I20260129 02:58:16.875029 20116 raft_consensus.cc:1081] T 4562a703e5a44a1585b332b851bfd6f2 P eafca908701c410ab9488970fcb3d74e [term 2 LEADER]: Signalling peer a634c60a5ca74b26b0d1f679d0166d0c to start an election
I20260129 02:58:16.876246 20070 tablet_service.cc:2038] Received Run Leader Election RPC: tablet_id: "4562a703e5a44a1585b332b851bfd6f2"
dest_uuid: "a634c60a5ca74b26b0d1f679d0166d0c"
from {username='slave'} at 127.0.0.1:43236
I20260129 02:58:16.876669 20070 raft_consensus.cc:493] T 4562a703e5a44a1585b332b851bfd6f2 P a634c60a5ca74b26b0d1f679d0166d0c [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20260129 02:58:16.876930 20070 raft_consensus.cc:3060] T 4562a703e5a44a1585b332b851bfd6f2 P a634c60a5ca74b26b0d1f679d0166d0c [term 2 FOLLOWER]: Advancing to term 3
I20260129 02:58:16.880901 20070 raft_consensus.cc:515] T 4562a703e5a44a1585b332b851bfd6f2 P a634c60a5ca74b26b0d1f679d0166d0c [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eafca908701c410ab9488970fcb3d74e" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 45001 } } peers { permanent_uuid: "544851f9b7954fd2962675f957e8836a" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34199 } } peers { permanent_uuid: "a634c60a5ca74b26b0d1f679d0166d0c" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 39941 } }
I20260129 02:58:16.882236 20070 leader_election.cc:290] T 4562a703e5a44a1585b332b851bfd6f2 P a634c60a5ca74b26b0d1f679d0166d0c [CANDIDATE]: Term 3 election: Requested vote from peers eafca908701c410ab9488970fcb3d74e (127.18.211.1:45001), 544851f9b7954fd2962675f957e8836a (127.18.211.2:34199)
I20260129 02:58:16.883046 19920 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4562a703e5a44a1585b332b851bfd6f2" candidate_uuid: "a634c60a5ca74b26b0d1f679d0166d0c" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "eafca908701c410ab9488970fcb3d74e"
I20260129 02:58:16.883284 19995 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4562a703e5a44a1585b332b851bfd6f2" candidate_uuid: "a634c60a5ca74b26b0d1f679d0166d0c" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "544851f9b7954fd2962675f957e8836a"
I20260129 02:58:16.883603 19920 raft_consensus.cc:3055] T 4562a703e5a44a1585b332b851bfd6f2 P eafca908701c410ab9488970fcb3d74e [term 2 LEADER]: Stepping down as leader of term 2
I20260129 02:58:16.883916 19995 raft_consensus.cc:3060] T 4562a703e5a44a1585b332b851bfd6f2 P 544851f9b7954fd2962675f957e8836a [term 2 FOLLOWER]: Advancing to term 3
I20260129 02:58:16.883972 19920 raft_consensus.cc:740] T 4562a703e5a44a1585b332b851bfd6f2 P eafca908701c410ab9488970fcb3d74e [term 2 LEADER]: Becoming Follower/Learner. State: Replica: eafca908701c410ab9488970fcb3d74e, State: Running, Role: LEADER
I20260129 02:58:16.884812 19920 consensus_queue.cc:260] T 4562a703e5a44a1585b332b851bfd6f2 P eafca908701c410ab9488970fcb3d74e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eafca908701c410ab9488970fcb3d74e" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 45001 } } peers { permanent_uuid: "544851f9b7954fd2962675f957e8836a" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34199 } } peers { permanent_uuid: "a634c60a5ca74b26b0d1f679d0166d0c" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 39941 } }
I20260129 02:58:16.885696 19920 raft_consensus.cc:3060] T 4562a703e5a44a1585b332b851bfd6f2 P eafca908701c410ab9488970fcb3d74e [term 2 FOLLOWER]: Advancing to term 3
I20260129 02:58:16.888512 19995 raft_consensus.cc:2468] T 4562a703e5a44a1585b332b851bfd6f2 P 544851f9b7954fd2962675f957e8836a [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a634c60a5ca74b26b0d1f679d0166d0c in term 3.
I20260129 02:58:16.889592 20033 leader_election.cc:304] T 4562a703e5a44a1585b332b851bfd6f2 P a634c60a5ca74b26b0d1f679d0166d0c [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 544851f9b7954fd2962675f957e8836a, a634c60a5ca74b26b0d1f679d0166d0c; no voters:
I20260129 02:58:16.890340 20132 raft_consensus.cc:2804] T 4562a703e5a44a1585b332b851bfd6f2 P a634c60a5ca74b26b0d1f679d0166d0c [term 3 FOLLOWER]: Leader election won for term 3
I20260129 02:58:16.890357 19920 raft_consensus.cc:2468] T 4562a703e5a44a1585b332b851bfd6f2 P eafca908701c410ab9488970fcb3d74e [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a634c60a5ca74b26b0d1f679d0166d0c in term 3.
I20260129 02:58:16.891655 20132 raft_consensus.cc:697] T 4562a703e5a44a1585b332b851bfd6f2 P a634c60a5ca74b26b0d1f679d0166d0c [term 3 LEADER]: Becoming Leader. State: Replica: a634c60a5ca74b26b0d1f679d0166d0c, State: Running, Role: LEADER
I20260129 02:58:16.892390 20132 consensus_queue.cc:237] T 4562a703e5a44a1585b332b851bfd6f2 P a634c60a5ca74b26b0d1f679d0166d0c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eafca908701c410ab9488970fcb3d74e" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 45001 } } peers { permanent_uuid: "544851f9b7954fd2962675f957e8836a" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34199 } } peers { permanent_uuid: "a634c60a5ca74b26b0d1f679d0166d0c" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 39941 } }
I20260129 02:58:16.898844 19816 catalog_manager.cc:5654] T 4562a703e5a44a1585b332b851bfd6f2 P a634c60a5ca74b26b0d1f679d0166d0c reported cstate change: term changed from 2 to 3, leader changed from eafca908701c410ab9488970fcb3d74e (127.18.211.1) to a634c60a5ca74b26b0d1f679d0166d0c (127.18.211.3). New cstate: current_term: 3 leader_uuid: "a634c60a5ca74b26b0d1f679d0166d0c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eafca908701c410ab9488970fcb3d74e" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 45001 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "544851f9b7954fd2962675f957e8836a" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34199 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a634c60a5ca74b26b0d1f679d0166d0c" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 39941 } health_report { overall_health: HEALTHY } } }
I20260129 02:58:16.943253 20206 raft_consensus.cc:993] T d5edd0a419ee46c9a6a223f8056afb4c P eafca908701c410ab9488970fcb3d74e: : Instructing follower a634c60a5ca74b26b0d1f679d0166d0c to start an election
I20260129 02:58:16.943704 20116 raft_consensus.cc:1081] T d5edd0a419ee46c9a6a223f8056afb4c P eafca908701c410ab9488970fcb3d74e [term 1 LEADER]: Signalling peer a634c60a5ca74b26b0d1f679d0166d0c to start an election
I20260129 02:58:16.945109 20070 tablet_service.cc:2038] Received Run Leader Election RPC: tablet_id: "d5edd0a419ee46c9a6a223f8056afb4c"
dest_uuid: "a634c60a5ca74b26b0d1f679d0166d0c"
from {username='slave'} at 127.0.0.1:43236
I20260129 02:58:16.945586 20070 raft_consensus.cc:493] T d5edd0a419ee46c9a6a223f8056afb4c P a634c60a5ca74b26b0d1f679d0166d0c [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260129 02:58:16.945863 20070 raft_consensus.cc:3060] T d5edd0a419ee46c9a6a223f8056afb4c P a634c60a5ca74b26b0d1f679d0166d0c [term 1 FOLLOWER]: Advancing to term 2
I20260129 02:58:16.949724 20070 raft_consensus.cc:515] T d5edd0a419ee46c9a6a223f8056afb4c P a634c60a5ca74b26b0d1f679d0166d0c [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eafca908701c410ab9488970fcb3d74e" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 45001 } } peers { permanent_uuid: "544851f9b7954fd2962675f957e8836a" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34199 } } peers { permanent_uuid: "a634c60a5ca74b26b0d1f679d0166d0c" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 39941 } }
I20260129 02:58:16.951006 20070 leader_election.cc:290] T d5edd0a419ee46c9a6a223f8056afb4c P a634c60a5ca74b26b0d1f679d0166d0c [CANDIDATE]: Term 2 election: Requested vote from peers eafca908701c410ab9488970fcb3d74e (127.18.211.1:45001), 544851f9b7954fd2962675f957e8836a (127.18.211.2:34199)
I20260129 02:58:16.951819 19920 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d5edd0a419ee46c9a6a223f8056afb4c" candidate_uuid: "a634c60a5ca74b26b0d1f679d0166d0c" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "eafca908701c410ab9488970fcb3d74e"
I20260129 02:58:16.952129 19995 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d5edd0a419ee46c9a6a223f8056afb4c" candidate_uuid: "a634c60a5ca74b26b0d1f679d0166d0c" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "544851f9b7954fd2962675f957e8836a"
I20260129 02:58:16.952358 19920 raft_consensus.cc:3055] T d5edd0a419ee46c9a6a223f8056afb4c P eafca908701c410ab9488970fcb3d74e [term 1 LEADER]: Stepping down as leader of term 1
I20260129 02:58:16.952703 19995 raft_consensus.cc:3060] T d5edd0a419ee46c9a6a223f8056afb4c P 544851f9b7954fd2962675f957e8836a [term 1 FOLLOWER]: Advancing to term 2
I20260129 02:58:16.952697 19920 raft_consensus.cc:740] T d5edd0a419ee46c9a6a223f8056afb4c P eafca908701c410ab9488970fcb3d74e [term 1 LEADER]: Becoming Follower/Learner. State: Replica: eafca908701c410ab9488970fcb3d74e, State: Running, Role: LEADER
I20260129 02:58:16.953714 19920 consensus_queue.cc:260] T d5edd0a419ee46c9a6a223f8056afb4c P eafca908701c410ab9488970fcb3d74e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eafca908701c410ab9488970fcb3d74e" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 45001 } } peers { permanent_uuid: "544851f9b7954fd2962675f957e8836a" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34199 } } peers { permanent_uuid: "a634c60a5ca74b26b0d1f679d0166d0c" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 39941 } }
I20260129 02:58:16.954756 19920 raft_consensus.cc:3060] T d5edd0a419ee46c9a6a223f8056afb4c P eafca908701c410ab9488970fcb3d74e [term 1 FOLLOWER]: Advancing to term 2
I20260129 02:58:16.957564 19995 raft_consensus.cc:2468] T d5edd0a419ee46c9a6a223f8056afb4c P 544851f9b7954fd2962675f957e8836a [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a634c60a5ca74b26b0d1f679d0166d0c in term 2.
I20260129 02:58:16.958458 20033 leader_election.cc:304] T d5edd0a419ee46c9a6a223f8056afb4c P a634c60a5ca74b26b0d1f679d0166d0c [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 544851f9b7954fd2962675f957e8836a, a634c60a5ca74b26b0d1f679d0166d0c; no voters:
I20260129 02:58:16.959149 20132 raft_consensus.cc:2804] T d5edd0a419ee46c9a6a223f8056afb4c P a634c60a5ca74b26b0d1f679d0166d0c [term 2 FOLLOWER]: Leader election won for term 2
I20260129 02:58:16.959203 19920 raft_consensus.cc:2468] T d5edd0a419ee46c9a6a223f8056afb4c P eafca908701c410ab9488970fcb3d74e [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a634c60a5ca74b26b0d1f679d0166d0c in term 2.
I20260129 02:58:16.959695 20132 raft_consensus.cc:697] T d5edd0a419ee46c9a6a223f8056afb4c P a634c60a5ca74b26b0d1f679d0166d0c [term 2 LEADER]: Becoming Leader. State: Replica: a634c60a5ca74b26b0d1f679d0166d0c, State: Running, Role: LEADER
I20260129 02:58:16.960561 20132 consensus_queue.cc:237] T d5edd0a419ee46c9a6a223f8056afb4c P a634c60a5ca74b26b0d1f679d0166d0c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eafca908701c410ab9488970fcb3d74e" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 45001 } } peers { permanent_uuid: "544851f9b7954fd2962675f957e8836a" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34199 } } peers { permanent_uuid: "a634c60a5ca74b26b0d1f679d0166d0c" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 39941 } }
I20260129 02:58:16.967429 19816 catalog_manager.cc:5654] T d5edd0a419ee46c9a6a223f8056afb4c P a634c60a5ca74b26b0d1f679d0166d0c reported cstate change: term changed from 1 to 2, leader changed from eafca908701c410ab9488970fcb3d74e (127.18.211.1) to a634c60a5ca74b26b0d1f679d0166d0c (127.18.211.3). New cstate: current_term: 2 leader_uuid: "a634c60a5ca74b26b0d1f679d0166d0c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eafca908701c410ab9488970fcb3d74e" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 45001 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "544851f9b7954fd2962675f957e8836a" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34199 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a634c60a5ca74b26b0d1f679d0166d0c" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 39941 } health_report { overall_health: HEALTHY } } }
I20260129 02:58:17.328140 19920 raft_consensus.cc:1275] T 4562a703e5a44a1585b332b851bfd6f2 P eafca908701c410ab9488970fcb3d74e [term 3 FOLLOWER]: Refusing update from remote peer a634c60a5ca74b26b0d1f679d0166d0c: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260129 02:58:17.329226 20132 consensus_queue.cc:1048] T 4562a703e5a44a1585b332b851bfd6f2 P a634c60a5ca74b26b0d1f679d0166d0c [LEADER]: Connected to new peer: Peer: permanent_uuid: "eafca908701c410ab9488970fcb3d74e" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 45001 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260129 02:58:17.337750 19995 raft_consensus.cc:1275] T 4562a703e5a44a1585b332b851bfd6f2 P 544851f9b7954fd2962675f957e8836a [term 3 FOLLOWER]: Refusing update from remote peer a634c60a5ca74b26b0d1f679d0166d0c: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260129 02:58:17.339246 20132 consensus_queue.cc:1048] T 4562a703e5a44a1585b332b851bfd6f2 P a634c60a5ca74b26b0d1f679d0166d0c [LEADER]: Connected to new peer: Peer: permanent_uuid: "544851f9b7954fd2962675f957e8836a" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34199 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260129 02:58:17.386834 19995 raft_consensus.cc:1275] T d5edd0a419ee46c9a6a223f8056afb4c P 544851f9b7954fd2962675f957e8836a [term 2 FOLLOWER]: Refusing update from remote peer a634c60a5ca74b26b0d1f679d0166d0c: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260129 02:58:17.388077 20132 consensus_queue.cc:1048] T d5edd0a419ee46c9a6a223f8056afb4c P a634c60a5ca74b26b0d1f679d0166d0c [LEADER]: Connected to new peer: Peer: permanent_uuid: "544851f9b7954fd2962675f957e8836a" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34199 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260129 02:58:17.396535 19920 raft_consensus.cc:1275] T d5edd0a419ee46c9a6a223f8056afb4c P eafca908701c410ab9488970fcb3d74e [term 2 FOLLOWER]: Refusing update from remote peer a634c60a5ca74b26b0d1f679d0166d0c: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260129 02:58:17.398566 20220 consensus_queue.cc:1048] T d5edd0a419ee46c9a6a223f8056afb4c P a634c60a5ca74b26b0d1f679d0166d0c [LEADER]: Connected to new peer: Peer: permanent_uuid: "eafca908701c410ab9488970fcb3d74e" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 45001 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260129 02:58:17.843494 19276 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260129 02:58:18.048573 19276 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260129 02:58:18.049437 19276 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20260129 02:58:18.525226 20223 raft_consensus.cc:493] T 79355f4f5d7942b6ac6f420256fd35b7 P 544851f9b7954fd2962675f957e8836a [term 2 FOLLOWER]: Starting pre-election (detected failure of leader eafca908701c410ab9488970fcb3d74e)
I20260129 02:58:18.526026 20223 raft_consensus.cc:515] T 79355f4f5d7942b6ac6f420256fd35b7 P 544851f9b7954fd2962675f957e8836a [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eafca908701c410ab9488970fcb3d74e" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 45001 } } peers { permanent_uuid: "544851f9b7954fd2962675f957e8836a" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34199 } } peers { permanent_uuid: "a634c60a5ca74b26b0d1f679d0166d0c" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 39941 } }
I20260129 02:58:18.528682 20223 leader_election.cc:290] T 79355f4f5d7942b6ac6f420256fd35b7 P 544851f9b7954fd2962675f957e8836a [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers eafca908701c410ab9488970fcb3d74e (127.18.211.1:45001), a634c60a5ca74b26b0d1f679d0166d0c (127.18.211.3:39941)
I20260129 02:58:18.624106 20207 raft_consensus.cc:493] T 7d91c3094df94b47aa9a512d13490daa P a634c60a5ca74b26b0d1f679d0166d0c [term 2 FOLLOWER]: Starting pre-election (detected failure of leader eafca908701c410ab9488970fcb3d74e)
I20260129 02:58:18.624651 20207 raft_consensus.cc:515] T 7d91c3094df94b47aa9a512d13490daa P a634c60a5ca74b26b0d1f679d0166d0c [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a634c60a5ca74b26b0d1f679d0166d0c" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 39941 } } peers { permanent_uuid: "eafca908701c410ab9488970fcb3d74e" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 45001 } } peers { permanent_uuid: "544851f9b7954fd2962675f957e8836a" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34199 } }
I20260129 02:58:18.626751 20207 leader_election.cc:290] T 7d91c3094df94b47aa9a512d13490daa P a634c60a5ca74b26b0d1f679d0166d0c [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers eafca908701c410ab9488970fcb3d74e (127.18.211.1:45001), 544851f9b7954fd2962675f957e8836a (127.18.211.2:34199)
I20260129 02:58:18.626694 19918 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "79355f4f5d7942b6ac6f420256fd35b7" candidate_uuid: "544851f9b7954fd2962675f957e8836a" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "eafca908701c410ab9488970fcb3d74e" is_pre_election: true
I20260129 02:58:18.627693 19917 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7d91c3094df94b47aa9a512d13490daa" candidate_uuid: "a634c60a5ca74b26b0d1f679d0166d0c" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "eafca908701c410ab9488970fcb3d74e" is_pre_election: true
W20260129 02:58:18.639436 19850 debug-util.cc:398] Leaking SignalData structure 0x7b0800054de0 after lost signal to thread 19279
W20260129 02:58:18.643252 19850 debug-util.cc:398] Leaking SignalData structure 0x7b080004d4c0 after lost signal to thread 19853
I20260129 02:58:18.644459 19995 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7d91c3094df94b47aa9a512d13490daa" candidate_uuid: "a634c60a5ca74b26b0d1f679d0166d0c" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "544851f9b7954fd2962675f957e8836a" is_pre_election: true
I20260129 02:58:18.646576 20070 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "79355f4f5d7942b6ac6f420256fd35b7" candidate_uuid: "544851f9b7954fd2962675f957e8836a" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "a634c60a5ca74b26b0d1f679d0166d0c" is_pre_election: true
W20260129 02:58:18.648136 19850 debug-util.cc:398] Leaking SignalData structure 0x7b08000cb900 after lost signal to thread 19945
W20260129 02:58:18.649060 19850 debug-util.cc:398] Leaking SignalData structure 0x7b08000cd640 after lost signal to thread 20020
W20260129 02:58:18.649999 19850 debug-util.cc:398] Leaking SignalData structure 0x7b08000d86c0 after lost signal to thread 20094
I20260129 02:58:18.663160 19959 leader_election.cc:304] T 79355f4f5d7942b6ac6f420256fd35b7 P 544851f9b7954fd2962675f957e8836a [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 544851f9b7954fd2962675f957e8836a; no voters: a634c60a5ca74b26b0d1f679d0166d0c, eafca908701c410ab9488970fcb3d74e
I20260129 02:58:18.664882 20223 raft_consensus.cc:2749] T 79355f4f5d7942b6ac6f420256fd35b7 P 544851f9b7954fd2962675f957e8836a [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260129 02:58:18.667744 20033 leader_election.cc:304] T 7d91c3094df94b47aa9a512d13490daa P a634c60a5ca74b26b0d1f679d0166d0c [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a634c60a5ca74b26b0d1f679d0166d0c; no voters: 544851f9b7954fd2962675f957e8836a, eafca908701c410ab9488970fcb3d74e
I20260129 02:58:18.668540 20228 raft_consensus.cc:2749] T 7d91c3094df94b47aa9a512d13490daa P a634c60a5ca74b26b0d1f679d0166d0c [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260129 02:58:19.050655 19276 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260129 02:58:19.059716 19276 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260129 02:58:19.060307 19276 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20260129 02:58:20.061357 19276 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260129 02:58:20.066545 19276 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260129 02:58:20.066970 19276 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20260129 02:58:21.068073 19276 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260129 02:58:21.073446 19276 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260129 02:58:21.073909 19276 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20260129 02:58:22.075157 19276 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260129 02:58:22.081302 19276 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260129 02:58:22.081679 19276 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20260129 02:58:23.082726 19276 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260129 02:58:23.087842 19276 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260129 02:58:23.088299 19276 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20260129 02:58:24.089141 19276 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260129 02:58:24.095388 19276 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260129 02:58:24.095820 19276 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20260129 02:58:25.096870 19276 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260129 02:58:25.102531 19276 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260129 02:58:25.102912 19276 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20260129 02:58:26.103780 19276 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260129 02:58:26.108707 19276 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260129 02:58:26.109059 19276 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20260129 02:58:27.136723 19276 auto_leader_rebalancer-test.cc:362] The leader distribution is
I20260129 02:58:27.143069 20070 tablet_service.cc:2066] Received LeaderStepDown RPC: tablet_id: "4562a703e5a44a1585b332b851bfd6f2"
dest_uuid: "a634c60a5ca74b26b0d1f679d0166d0c"
mode: GRACEFUL
new_leader_uuid: "eafca908701c410ab9488970fcb3d74e"
from {username='slave'} at 127.0.0.1:42438
I20260129 02:58:27.143677 20070 raft_consensus.cc:606] T 4562a703e5a44a1585b332b851bfd6f2 P a634c60a5ca74b26b0d1f679d0166d0c [term 3 LEADER]: Received request to transfer leadership to TS eafca908701c410ab9488970fcb3d74e
I20260129 02:58:27.148012 20070 tablet_service.cc:2066] Received LeaderStepDown RPC: tablet_id: "d5edd0a419ee46c9a6a223f8056afb4c"
dest_uuid: "a634c60a5ca74b26b0d1f679d0166d0c"
mode: GRACEFUL
new_leader_uuid: "eafca908701c410ab9488970fcb3d74e"
from {username='slave'} at 127.0.0.1:42438
I20260129 02:58:27.148403 20070 raft_consensus.cc:606] T d5edd0a419ee46c9a6a223f8056afb4c P a634c60a5ca74b26b0d1f679d0166d0c [term 2 LEADER]: Received request to transfer leadership to TS eafca908701c410ab9488970fcb3d74e
I20260129 02:58:27.166951 19995 tablet_service.cc:2066] Received LeaderStepDown RPC: tablet_id: "620c6d80e1964c51a628ffb43f645798"
dest_uuid: "544851f9b7954fd2962675f957e8836a"
mode: GRACEFUL
new_leader_uuid: "eafca908701c410ab9488970fcb3d74e"
from {username='slave'} at 127.0.0.1:38640
I20260129 02:58:27.167440 19995 raft_consensus.cc:606] T 620c6d80e1964c51a628ffb43f645798 P 544851f9b7954fd2962675f957e8836a [term 1 LEADER]: Received request to transfer leadership to TS eafca908701c410ab9488970fcb3d74e
I20260129 02:58:27.172223 19995 tablet_service.cc:2066] Received LeaderStepDown RPC: tablet_id: "901f619b02f843b4a2747de956182f12"
dest_uuid: "544851f9b7954fd2962675f957e8836a"
mode: GRACEFUL
new_leader_uuid: "eafca908701c410ab9488970fcb3d74e"
from {username='slave'} at 127.0.0.1:38640
I20260129 02:58:27.172695 19995 raft_consensus.cc:606] T 901f619b02f843b4a2747de956182f12 P 544851f9b7954fd2962675f957e8836a [term 2 LEADER]: Received request to transfer leadership to TS eafca908701c410ab9488970fcb3d74e
I20260129 02:58:27.172765 20239 raft_consensus.cc:993] T 620c6d80e1964c51a628ffb43f645798 P 544851f9b7954fd2962675f957e8836a: : Instructing follower eafca908701c410ab9488970fcb3d74e to start an election
I20260129 02:58:27.173465 20236 raft_consensus.cc:1081] T 620c6d80e1964c51a628ffb43f645798 P 544851f9b7954fd2962675f957e8836a [term 1 LEADER]: Signalling peer eafca908701c410ab9488970fcb3d74e to start an election
I20260129 02:58:27.175096 19921 tablet_service.cc:2038] Received Run Leader Election RPC: tablet_id: "620c6d80e1964c51a628ffb43f645798"
dest_uuid: "eafca908701c410ab9488970fcb3d74e"
from {username='slave'} at 127.0.0.1:50460
I20260129 02:58:27.175719 19921 raft_consensus.cc:493] T 620c6d80e1964c51a628ffb43f645798 P eafca908701c410ab9488970fcb3d74e [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260129 02:58:27.176191 19921 raft_consensus.cc:3060] T 620c6d80e1964c51a628ffb43f645798 P eafca908701c410ab9488970fcb3d74e [term 1 FOLLOWER]: Advancing to term 2
I20260129 02:58:27.178269 19995 tablet_service.cc:2066] Received LeaderStepDown RPC: tablet_id: "1dc801ccecc44ed79d85b0522e49d3e5"
dest_uuid: "544851f9b7954fd2962675f957e8836a"
mode: GRACEFUL
new_leader_uuid: "eafca908701c410ab9488970fcb3d74e"
from {username='slave'} at 127.0.0.1:38640
I20260129 02:58:27.178846 19995 raft_consensus.cc:606] T 1dc801ccecc44ed79d85b0522e49d3e5 P 544851f9b7954fd2962675f957e8836a [term 1 LEADER]: Received request to transfer leadership to TS eafca908701c410ab9488970fcb3d74e
I20260129 02:58:27.183349 19921 raft_consensus.cc:515] T 620c6d80e1964c51a628ffb43f645798 P eafca908701c410ab9488970fcb3d74e [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eafca908701c410ab9488970fcb3d74e" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 45001 } } peers { permanent_uuid: "544851f9b7954fd2962675f957e8836a" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34199 } } peers { permanent_uuid: "a634c60a5ca74b26b0d1f679d0166d0c" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 39941 } }
I20260129 02:58:27.185427 19921 leader_election.cc:290] T 620c6d80e1964c51a628ffb43f645798 P eafca908701c410ab9488970fcb3d74e [CANDIDATE]: Term 2 election: Requested vote from peers 544851f9b7954fd2962675f957e8836a (127.18.211.2:34199), a634c60a5ca74b26b0d1f679d0166d0c (127.18.211.3:39941)
I20260129 02:58:27.186411 20070 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "620c6d80e1964c51a628ffb43f645798" candidate_uuid: "eafca908701c410ab9488970fcb3d74e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "a634c60a5ca74b26b0d1f679d0166d0c"
I20260129 02:58:27.186226 19995 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "620c6d80e1964c51a628ffb43f645798" candidate_uuid: "eafca908701c410ab9488970fcb3d74e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "544851f9b7954fd2962675f957e8836a"
I20260129 02:58:27.187017 20070 raft_consensus.cc:3060] T 620c6d80e1964c51a628ffb43f645798 P a634c60a5ca74b26b0d1f679d0166d0c [term 1 FOLLOWER]: Advancing to term 2
I20260129 02:58:27.187101 19995 raft_consensus.cc:3055] T 620c6d80e1964c51a628ffb43f645798 P 544851f9b7954fd2962675f957e8836a [term 1 LEADER]: Stepping down as leader of term 1
I20260129 02:58:27.187471 19995 raft_consensus.cc:740] T 620c6d80e1964c51a628ffb43f645798 P 544851f9b7954fd2962675f957e8836a [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 544851f9b7954fd2962675f957e8836a, State: Running, Role: LEADER
I20260129 02:58:27.188266 19995 consensus_queue.cc:260] T 620c6d80e1964c51a628ffb43f645798 P 544851f9b7954fd2962675f957e8836a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eafca908701c410ab9488970fcb3d74e" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 45001 } } peers { permanent_uuid: "544851f9b7954fd2962675f957e8836a" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34199 } } peers { permanent_uuid: "a634c60a5ca74b26b0d1f679d0166d0c" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 39941 } }
I20260129 02:58:27.189201 19995 raft_consensus.cc:3060] T 620c6d80e1964c51a628ffb43f645798 P 544851f9b7954fd2962675f957e8836a [term 1 FOLLOWER]: Advancing to term 2
I20260129 02:58:27.191922 20070 raft_consensus.cc:2468] T 620c6d80e1964c51a628ffb43f645798 P a634c60a5ca74b26b0d1f679d0166d0c [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eafca908701c410ab9488970fcb3d74e in term 2.
I20260129 02:58:27.193168 19885 leader_election.cc:304] T 620c6d80e1964c51a628ffb43f645798 P eafca908701c410ab9488970fcb3d74e [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a634c60a5ca74b26b0d1f679d0166d0c, eafca908701c410ab9488970fcb3d74e; no voters:
I20260129 02:58:27.194097 20233 raft_consensus.cc:2804] T 620c6d80e1964c51a628ffb43f645798 P eafca908701c410ab9488970fcb3d74e [term 2 FOLLOWER]: Leader election won for term 2
I20260129 02:58:27.194443 19995 raft_consensus.cc:2468] T 620c6d80e1964c51a628ffb43f645798 P 544851f9b7954fd2962675f957e8836a [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eafca908701c410ab9488970fcb3d74e in term 2.
I20260129 02:58:27.195194 20233 raft_consensus.cc:697] T 620c6d80e1964c51a628ffb43f645798 P eafca908701c410ab9488970fcb3d74e [term 2 LEADER]: Becoming Leader. State: Replica: eafca908701c410ab9488970fcb3d74e, State: Running, Role: LEADER
I20260129 02:58:27.196177 20233 consensus_queue.cc:237] T 620c6d80e1964c51a628ffb43f645798 P eafca908701c410ab9488970fcb3d74e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eafca908701c410ab9488970fcb3d74e" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 45001 } } peers { permanent_uuid: "544851f9b7954fd2962675f957e8836a" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34199 } } peers { permanent_uuid: "a634c60a5ca74b26b0d1f679d0166d0c" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 39941 } }
I20260129 02:58:27.206102 19817 catalog_manager.cc:5654] T 620c6d80e1964c51a628ffb43f645798 P eafca908701c410ab9488970fcb3d74e reported cstate change: term changed from 1 to 2, leader changed from 544851f9b7954fd2962675f957e8836a (127.18.211.2) to eafca908701c410ab9488970fcb3d74e (127.18.211.1). New cstate: current_term: 2 leader_uuid: "eafca908701c410ab9488970fcb3d74e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eafca908701c410ab9488970fcb3d74e" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 45001 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "544851f9b7954fd2962675f957e8836a" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34199 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a634c60a5ca74b26b0d1f679d0166d0c" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 39941 } health_report { overall_health: UNKNOWN } } }
I20260129 02:58:27.384435 20236 raft_consensus.cc:993] T 1dc801ccecc44ed79d85b0522e49d3e5 P 544851f9b7954fd2962675f957e8836a: : Instructing follower eafca908701c410ab9488970fcb3d74e to start an election
I20260129 02:58:27.384856 20239 raft_consensus.cc:1081] T 1dc801ccecc44ed79d85b0522e49d3e5 P 544851f9b7954fd2962675f957e8836a [term 1 LEADER]: Signalling peer eafca908701c410ab9488970fcb3d74e to start an election
I20260129 02:58:27.386118 19921 tablet_service.cc:2038] Received Run Leader Election RPC: tablet_id: "1dc801ccecc44ed79d85b0522e49d3e5"
dest_uuid: "eafca908701c410ab9488970fcb3d74e"
from {username='slave'} at 127.0.0.1:50460
I20260129 02:58:27.386590 19921 raft_consensus.cc:493] T 1dc801ccecc44ed79d85b0522e49d3e5 P eafca908701c410ab9488970fcb3d74e [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260129 02:58:27.386870 19921 raft_consensus.cc:3060] T 1dc801ccecc44ed79d85b0522e49d3e5 P eafca908701c410ab9488970fcb3d74e [term 1 FOLLOWER]: Advancing to term 2
I20260129 02:58:27.391409 19921 raft_consensus.cc:515] T 1dc801ccecc44ed79d85b0522e49d3e5 P eafca908701c410ab9488970fcb3d74e [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eafca908701c410ab9488970fcb3d74e" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 45001 } } peers { permanent_uuid: "544851f9b7954fd2962675f957e8836a" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34199 } } peers { permanent_uuid: "a634c60a5ca74b26b0d1f679d0166d0c" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 39941 } }
I20260129 02:58:27.392863 19921 leader_election.cc:290] T 1dc801ccecc44ed79d85b0522e49d3e5 P eafca908701c410ab9488970fcb3d74e [CANDIDATE]: Term 2 election: Requested vote from peers 544851f9b7954fd2962675f957e8836a (127.18.211.2:34199), a634c60a5ca74b26b0d1f679d0166d0c (127.18.211.3:39941)
I20260129 02:58:27.393759 19995 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1dc801ccecc44ed79d85b0522e49d3e5" candidate_uuid: "eafca908701c410ab9488970fcb3d74e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "544851f9b7954fd2962675f957e8836a"
I20260129 02:58:27.393962 20070 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1dc801ccecc44ed79d85b0522e49d3e5" candidate_uuid: "eafca908701c410ab9488970fcb3d74e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "a634c60a5ca74b26b0d1f679d0166d0c"
I20260129 02:58:27.394306 19995 raft_consensus.cc:3055] T 1dc801ccecc44ed79d85b0522e49d3e5 P 544851f9b7954fd2962675f957e8836a [term 1 LEADER]: Stepping down as leader of term 1
I20260129 02:58:27.394408 20070 raft_consensus.cc:3060] T 1dc801ccecc44ed79d85b0522e49d3e5 P a634c60a5ca74b26b0d1f679d0166d0c [term 1 FOLLOWER]: Advancing to term 2
I20260129 02:58:27.394657 19995 raft_consensus.cc:740] T 1dc801ccecc44ed79d85b0522e49d3e5 P 544851f9b7954fd2962675f957e8836a [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 544851f9b7954fd2962675f957e8836a, State: Running, Role: LEADER
I20260129 02:58:27.395423 19995 consensus_queue.cc:260] T 1dc801ccecc44ed79d85b0522e49d3e5 P 544851f9b7954fd2962675f957e8836a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eafca908701c410ab9488970fcb3d74e" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 45001 } } peers { permanent_uuid: "544851f9b7954fd2962675f957e8836a" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34199 } } peers { permanent_uuid: "a634c60a5ca74b26b0d1f679d0166d0c" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 39941 } }
I20260129 02:58:27.396585 19995 raft_consensus.cc:3060] T 1dc801ccecc44ed79d85b0522e49d3e5 P 544851f9b7954fd2962675f957e8836a [term 1 FOLLOWER]: Advancing to term 2
I20260129 02:58:27.400564 20070 raft_consensus.cc:2468] T 1dc801ccecc44ed79d85b0522e49d3e5 P a634c60a5ca74b26b0d1f679d0166d0c [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eafca908701c410ab9488970fcb3d74e in term 2.
I20260129 02:58:27.401582 19885 leader_election.cc:304] T 1dc801ccecc44ed79d85b0522e49d3e5 P eafca908701c410ab9488970fcb3d74e [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a634c60a5ca74b26b0d1f679d0166d0c, eafca908701c410ab9488970fcb3d74e; no voters:
I20260129 02:58:27.402159 19995 raft_consensus.cc:2468] T 1dc801ccecc44ed79d85b0522e49d3e5 P 544851f9b7954fd2962675f957e8836a [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eafca908701c410ab9488970fcb3d74e in term 2.
I20260129 02:58:27.402302 20233 raft_consensus.cc:2804] T 1dc801ccecc44ed79d85b0522e49d3e5 P eafca908701c410ab9488970fcb3d74e [term 2 FOLLOWER]: Leader election won for term 2
I20260129 02:58:27.402720 20233 raft_consensus.cc:697] T 1dc801ccecc44ed79d85b0522e49d3e5 P eafca908701c410ab9488970fcb3d74e [term 2 LEADER]: Becoming Leader. State: Replica: eafca908701c410ab9488970fcb3d74e, State: Running, Role: LEADER
I20260129 02:58:27.403596 20233 consensus_queue.cc:237] T 1dc801ccecc44ed79d85b0522e49d3e5 P eafca908701c410ab9488970fcb3d74e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eafca908701c410ab9488970fcb3d74e" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 45001 } } peers { permanent_uuid: "544851f9b7954fd2962675f957e8836a" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34199 } } peers { permanent_uuid: "a634c60a5ca74b26b0d1f679d0166d0c" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 39941 } }
I20260129 02:58:27.410477 19817 catalog_manager.cc:5654] T 1dc801ccecc44ed79d85b0522e49d3e5 P eafca908701c410ab9488970fcb3d74e reported cstate change: term changed from 1 to 2, leader changed from 544851f9b7954fd2962675f957e8836a (127.18.211.2) to eafca908701c410ab9488970fcb3d74e (127.18.211.1). New cstate: current_term: 2 leader_uuid: "eafca908701c410ab9488970fcb3d74e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eafca908701c410ab9488970fcb3d74e" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 45001 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "544851f9b7954fd2962675f957e8836a" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34199 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a634c60a5ca74b26b0d1f679d0166d0c" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 39941 } health_report { overall_health: UNKNOWN } } }
I20260129 02:58:27.425289 20232 raft_consensus.cc:993] T 4562a703e5a44a1585b332b851bfd6f2 P a634c60a5ca74b26b0d1f679d0166d0c: : Instructing follower eafca908701c410ab9488970fcb3d74e to start an election
I20260129 02:58:27.425804 20248 raft_consensus.cc:1081] T 4562a703e5a44a1585b332b851bfd6f2 P a634c60a5ca74b26b0d1f679d0166d0c [term 3 LEADER]: Signalling peer eafca908701c410ab9488970fcb3d74e to start an election
I20260129 02:58:27.427037 19921 tablet_service.cc:2038] Received Run Leader Election RPC: tablet_id: "4562a703e5a44a1585b332b851bfd6f2"
dest_uuid: "eafca908701c410ab9488970fcb3d74e"
from {username='slave'} at 127.0.0.1:50448
I20260129 02:58:27.427485 19921 raft_consensus.cc:493] T 4562a703e5a44a1585b332b851bfd6f2 P eafca908701c410ab9488970fcb3d74e [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20260129 02:58:27.427775 19921 raft_consensus.cc:3060] T 4562a703e5a44a1585b332b851bfd6f2 P eafca908701c410ab9488970fcb3d74e [term 3 FOLLOWER]: Advancing to term 4
I20260129 02:58:27.432400 19921 raft_consensus.cc:515] T 4562a703e5a44a1585b332b851bfd6f2 P eafca908701c410ab9488970fcb3d74e [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eafca908701c410ab9488970fcb3d74e" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 45001 } } peers { permanent_uuid: "544851f9b7954fd2962675f957e8836a" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34199 } } peers { permanent_uuid: "a634c60a5ca74b26b0d1f679d0166d0c" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 39941 } }
I20260129 02:58:27.433907 19921 leader_election.cc:290] T 4562a703e5a44a1585b332b851bfd6f2 P eafca908701c410ab9488970fcb3d74e [CANDIDATE]: Term 4 election: Requested vote from peers 544851f9b7954fd2962675f957e8836a (127.18.211.2:34199), a634c60a5ca74b26b0d1f679d0166d0c (127.18.211.3:39941)
I20260129 02:58:27.434691 19995 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4562a703e5a44a1585b332b851bfd6f2" candidate_uuid: "eafca908701c410ab9488970fcb3d74e" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: true dest_uuid: "544851f9b7954fd2962675f957e8836a"
I20260129 02:58:27.434984 20069 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4562a703e5a44a1585b332b851bfd6f2" candidate_uuid: "eafca908701c410ab9488970fcb3d74e" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: true dest_uuid: "a634c60a5ca74b26b0d1f679d0166d0c"
I20260129 02:58:27.435269 19995 raft_consensus.cc:3060] T 4562a703e5a44a1585b332b851bfd6f2 P 544851f9b7954fd2962675f957e8836a [term 3 FOLLOWER]: Advancing to term 4
I20260129 02:58:27.435561 20069 raft_consensus.cc:3055] T 4562a703e5a44a1585b332b851bfd6f2 P a634c60a5ca74b26b0d1f679d0166d0c [term 3 LEADER]: Stepping down as leader of term 3
I20260129 02:58:27.435899 20069 raft_consensus.cc:740] T 4562a703e5a44a1585b332b851bfd6f2 P a634c60a5ca74b26b0d1f679d0166d0c [term 3 LEADER]: Becoming Follower/Learner. State: Replica: a634c60a5ca74b26b0d1f679d0166d0c, State: Running, Role: LEADER
I20260129 02:58:27.436683 20069 consensus_queue.cc:260] T 4562a703e5a44a1585b332b851bfd6f2 P a634c60a5ca74b26b0d1f679d0166d0c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eafca908701c410ab9488970fcb3d74e" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 45001 } } peers { permanent_uuid: "544851f9b7954fd2962675f957e8836a" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34199 } } peers { permanent_uuid: "a634c60a5ca74b26b0d1f679d0166d0c" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 39941 } }
I20260129 02:58:27.437682 20069 raft_consensus.cc:3060] T 4562a703e5a44a1585b332b851bfd6f2 P a634c60a5ca74b26b0d1f679d0166d0c [term 3 FOLLOWER]: Advancing to term 4
I20260129 02:58:27.439771 19995 raft_consensus.cc:2468] T 4562a703e5a44a1585b332b851bfd6f2 P 544851f9b7954fd2962675f957e8836a [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eafca908701c410ab9488970fcb3d74e in term 4.
I20260129 02:58:27.440865 19884 leader_election.cc:304] T 4562a703e5a44a1585b332b851bfd6f2 P eafca908701c410ab9488970fcb3d74e [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 544851f9b7954fd2962675f957e8836a, eafca908701c410ab9488970fcb3d74e; no voters:
I20260129 02:58:27.441534 20233 raft_consensus.cc:2804] T 4562a703e5a44a1585b332b851bfd6f2 P eafca908701c410ab9488970fcb3d74e [term 4 FOLLOWER]: Leader election won for term 4
I20260129 02:58:27.441918 20233 raft_consensus.cc:697] T 4562a703e5a44a1585b332b851bfd6f2 P eafca908701c410ab9488970fcb3d74e [term 4 LEADER]: Becoming Leader. State: Replica: eafca908701c410ab9488970fcb3d74e, State: Running, Role: LEADER
I20260129 02:58:27.442500 20069 raft_consensus.cc:2468] T 4562a703e5a44a1585b332b851bfd6f2 P a634c60a5ca74b26b0d1f679d0166d0c [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eafca908701c410ab9488970fcb3d74e in term 4.
I20260129 02:58:27.442732 20233 consensus_queue.cc:237] T 4562a703e5a44a1585b332b851bfd6f2 P eafca908701c410ab9488970fcb3d74e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eafca908701c410ab9488970fcb3d74e" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 45001 } } peers { permanent_uuid: "544851f9b7954fd2962675f957e8836a" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34199 } } peers { permanent_uuid: "a634c60a5ca74b26b0d1f679d0166d0c" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 39941 } }
I20260129 02:58:27.449370 19817 catalog_manager.cc:5654] T 4562a703e5a44a1585b332b851bfd6f2 P eafca908701c410ab9488970fcb3d74e reported cstate change: term changed from 3 to 4, leader changed from a634c60a5ca74b26b0d1f679d0166d0c (127.18.211.3) to eafca908701c410ab9488970fcb3d74e (127.18.211.1). New cstate: current_term: 4 leader_uuid: "eafca908701c410ab9488970fcb3d74e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eafca908701c410ab9488970fcb3d74e" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 45001 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "544851f9b7954fd2962675f957e8836a" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34199 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a634c60a5ca74b26b0d1f679d0166d0c" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 39941 } health_report { overall_health: UNKNOWN } } }
I20260129 02:58:27.581889 20069 raft_consensus.cc:1275] T 620c6d80e1964c51a628ffb43f645798 P a634c60a5ca74b26b0d1f679d0166d0c [term 2 FOLLOWER]: Refusing update from remote peer eafca908701c410ab9488970fcb3d74e: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260129 02:58:27.583104 20233 consensus_queue.cc:1048] T 620c6d80e1964c51a628ffb43f645798 P eafca908701c410ab9488970fcb3d74e [LEADER]: Connected to new peer: Peer: permanent_uuid: "a634c60a5ca74b26b0d1f679d0166d0c" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 39941 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260129 02:58:27.597748 19995 raft_consensus.cc:1275] T 620c6d80e1964c51a628ffb43f645798 P 544851f9b7954fd2962675f957e8836a [term 2 FOLLOWER]: Refusing update from remote peer eafca908701c410ab9488970fcb3d74e: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260129 02:58:27.599120 20242 consensus_queue.cc:1048] T 620c6d80e1964c51a628ffb43f645798 P eafca908701c410ab9488970fcb3d74e [LEADER]: Connected to new peer: Peer: permanent_uuid: "544851f9b7954fd2962675f957e8836a" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34199 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260129 02:58:27.665074 20236 raft_consensus.cc:993] T 901f619b02f843b4a2747de956182f12 P 544851f9b7954fd2962675f957e8836a: : Instructing follower eafca908701c410ab9488970fcb3d74e to start an election
I20260129 02:58:27.665531 20239 raft_consensus.cc:1081] T 901f619b02f843b4a2747de956182f12 P 544851f9b7954fd2962675f957e8836a [term 2 LEADER]: Signalling peer eafca908701c410ab9488970fcb3d74e to start an election
I20260129 02:58:27.667133 19921 tablet_service.cc:2038] Received Run Leader Election RPC: tablet_id: "901f619b02f843b4a2747de956182f12"
dest_uuid: "eafca908701c410ab9488970fcb3d74e"
from {username='slave'} at 127.0.0.1:50460
I20260129 02:58:27.667979 19921 raft_consensus.cc:493] T 901f619b02f843b4a2747de956182f12 P eafca908701c410ab9488970fcb3d74e [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20260129 02:58:27.668341 19921 raft_consensus.cc:3060] T 901f619b02f843b4a2747de956182f12 P eafca908701c410ab9488970fcb3d74e [term 2 FOLLOWER]: Advancing to term 3
I20260129 02:58:27.672894 19921 raft_consensus.cc:515] T 901f619b02f843b4a2747de956182f12 P eafca908701c410ab9488970fcb3d74e [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eafca908701c410ab9488970fcb3d74e" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 45001 } } peers { permanent_uuid: "544851f9b7954fd2962675f957e8836a" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34199 } } peers { permanent_uuid: "a634c60a5ca74b26b0d1f679d0166d0c" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 39941 } }
I20260129 02:58:27.674427 19921 leader_election.cc:290] T 901f619b02f843b4a2747de956182f12 P eafca908701c410ab9488970fcb3d74e [CANDIDATE]: Term 3 election: Requested vote from peers 544851f9b7954fd2962675f957e8836a (127.18.211.2:34199), a634c60a5ca74b26b0d1f679d0166d0c (127.18.211.3:39941)
I20260129 02:58:27.675208 19995 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "901f619b02f843b4a2747de956182f12" candidate_uuid: "eafca908701c410ab9488970fcb3d74e" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "544851f9b7954fd2962675f957e8836a"
I20260129 02:58:27.675540 20069 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "901f619b02f843b4a2747de956182f12" candidate_uuid: "eafca908701c410ab9488970fcb3d74e" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "a634c60a5ca74b26b0d1f679d0166d0c"
I20260129 02:58:27.675675 19995 raft_consensus.cc:3055] T 901f619b02f843b4a2747de956182f12 P 544851f9b7954fd2962675f957e8836a [term 2 LEADER]: Stepping down as leader of term 2
I20260129 02:58:27.676095 19995 raft_consensus.cc:740] T 901f619b02f843b4a2747de956182f12 P 544851f9b7954fd2962675f957e8836a [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 544851f9b7954fd2962675f957e8836a, State: Running, Role: LEADER
I20260129 02:58:27.676092 20069 raft_consensus.cc:3060] T 901f619b02f843b4a2747de956182f12 P a634c60a5ca74b26b0d1f679d0166d0c [term 2 FOLLOWER]: Advancing to term 3
I20260129 02:58:27.676790 19995 consensus_queue.cc:260] T 901f619b02f843b4a2747de956182f12 P 544851f9b7954fd2962675f957e8836a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eafca908701c410ab9488970fcb3d74e" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 45001 } } peers { permanent_uuid: "544851f9b7954fd2962675f957e8836a" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34199 } } peers { permanent_uuid: "a634c60a5ca74b26b0d1f679d0166d0c" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 39941 } }
I20260129 02:58:27.677860 19995 raft_consensus.cc:3060] T 901f619b02f843b4a2747de956182f12 P 544851f9b7954fd2962675f957e8836a [term 2 FOLLOWER]: Advancing to term 3
I20260129 02:58:27.680814 20232 raft_consensus.cc:993] T d5edd0a419ee46c9a6a223f8056afb4c P a634c60a5ca74b26b0d1f679d0166d0c: : Instructing follower eafca908701c410ab9488970fcb3d74e to start an election
I20260129 02:58:27.681269 20248 raft_consensus.cc:1081] T d5edd0a419ee46c9a6a223f8056afb4c P a634c60a5ca74b26b0d1f679d0166d0c [term 2 LEADER]: Signalling peer eafca908701c410ab9488970fcb3d74e to start an election
I20260129 02:58:27.682789 19921 tablet_service.cc:2038] Received Run Leader Election RPC: tablet_id: "d5edd0a419ee46c9a6a223f8056afb4c"
dest_uuid: "eafca908701c410ab9488970fcb3d74e"
from {username='slave'} at 127.0.0.1:50448
I20260129 02:58:27.682897 20069 raft_consensus.cc:2468] T 901f619b02f843b4a2747de956182f12 P a634c60a5ca74b26b0d1f679d0166d0c [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eafca908701c410ab9488970fcb3d74e in term 3.
I20260129 02:58:27.683339 19921 raft_consensus.cc:493] T d5edd0a419ee46c9a6a223f8056afb4c P eafca908701c410ab9488970fcb3d74e [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20260129 02:58:27.683648 19995 raft_consensus.cc:2468] T 901f619b02f843b4a2747de956182f12 P 544851f9b7954fd2962675f957e8836a [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eafca908701c410ab9488970fcb3d74e in term 3.
I20260129 02:58:27.683794 19921 raft_consensus.cc:3060] T d5edd0a419ee46c9a6a223f8056afb4c P eafca908701c410ab9488970fcb3d74e [term 2 FOLLOWER]: Advancing to term 3
I20260129 02:58:27.684026 19885 leader_election.cc:304] T 901f619b02f843b4a2747de956182f12 P eafca908701c410ab9488970fcb3d74e [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a634c60a5ca74b26b0d1f679d0166d0c, eafca908701c410ab9488970fcb3d74e; no voters:
I20260129 02:58:27.684896 20242 raft_consensus.cc:2804] T 901f619b02f843b4a2747de956182f12 P eafca908701c410ab9488970fcb3d74e [term 3 FOLLOWER]: Leader election won for term 3
I20260129 02:58:27.685346 20242 raft_consensus.cc:697] T 901f619b02f843b4a2747de956182f12 P eafca908701c410ab9488970fcb3d74e [term 3 LEADER]: Becoming Leader. State: Replica: eafca908701c410ab9488970fcb3d74e, State: Running, Role: LEADER
I20260129 02:58:27.686203 20242 consensus_queue.cc:237] T 901f619b02f843b4a2747de956182f12 P eafca908701c410ab9488970fcb3d74e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eafca908701c410ab9488970fcb3d74e" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 45001 } } peers { permanent_uuid: "544851f9b7954fd2962675f957e8836a" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34199 } } peers { permanent_uuid: "a634c60a5ca74b26b0d1f679d0166d0c" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 39941 } }
I20260129 02:58:27.690382 19921 raft_consensus.cc:515] T d5edd0a419ee46c9a6a223f8056afb4c P eafca908701c410ab9488970fcb3d74e [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eafca908701c410ab9488970fcb3d74e" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 45001 } } peers { permanent_uuid: "544851f9b7954fd2962675f957e8836a" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34199 } } peers { permanent_uuid: "a634c60a5ca74b26b0d1f679d0166d0c" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 39941 } }
I20260129 02:58:27.691859 19921 leader_election.cc:290] T d5edd0a419ee46c9a6a223f8056afb4c P eafca908701c410ab9488970fcb3d74e [CANDIDATE]: Term 3 election: Requested vote from peers 544851f9b7954fd2962675f957e8836a (127.18.211.2:34199), a634c60a5ca74b26b0d1f679d0166d0c (127.18.211.3:39941)
I20260129 02:58:27.692703 19995 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d5edd0a419ee46c9a6a223f8056afb4c" candidate_uuid: "eafca908701c410ab9488970fcb3d74e" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "544851f9b7954fd2962675f957e8836a"
I20260129 02:58:27.692920 20069 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d5edd0a419ee46c9a6a223f8056afb4c" candidate_uuid: "eafca908701c410ab9488970fcb3d74e" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "a634c60a5ca74b26b0d1f679d0166d0c"
I20260129 02:58:27.693281 19995 raft_consensus.cc:3060] T d5edd0a419ee46c9a6a223f8056afb4c P 544851f9b7954fd2962675f957e8836a [term 2 FOLLOWER]: Advancing to term 3
I20260129 02:58:27.693557 20069 raft_consensus.cc:3055] T d5edd0a419ee46c9a6a223f8056afb4c P a634c60a5ca74b26b0d1f679d0166d0c [term 2 LEADER]: Stepping down as leader of term 2
I20260129 02:58:27.693861 20069 raft_consensus.cc:740] T d5edd0a419ee46c9a6a223f8056afb4c P a634c60a5ca74b26b0d1f679d0166d0c [term 2 LEADER]: Becoming Follower/Learner. State: Replica: a634c60a5ca74b26b0d1f679d0166d0c, State: Running, Role: LEADER
I20260129 02:58:27.694623 20069 consensus_queue.cc:260] T d5edd0a419ee46c9a6a223f8056afb4c P a634c60a5ca74b26b0d1f679d0166d0c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eafca908701c410ab9488970fcb3d74e" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 45001 } } peers { permanent_uuid: "544851f9b7954fd2962675f957e8836a" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34199 } } peers { permanent_uuid: "a634c60a5ca74b26b0d1f679d0166d0c" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 39941 } }
I20260129 02:58:27.695674 20069 raft_consensus.cc:3060] T d5edd0a419ee46c9a6a223f8056afb4c P a634c60a5ca74b26b0d1f679d0166d0c [term 2 FOLLOWER]: Advancing to term 3
I20260129 02:58:27.696817 19817 catalog_manager.cc:5654] T 901f619b02f843b4a2747de956182f12 P eafca908701c410ab9488970fcb3d74e reported cstate change: term changed from 2 to 3, leader changed from 544851f9b7954fd2962675f957e8836a (127.18.211.2) to eafca908701c410ab9488970fcb3d74e (127.18.211.1). New cstate: current_term: 3 leader_uuid: "eafca908701c410ab9488970fcb3d74e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eafca908701c410ab9488970fcb3d74e" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 45001 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "544851f9b7954fd2962675f957e8836a" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34199 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a634c60a5ca74b26b0d1f679d0166d0c" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 39941 } health_report { overall_health: UNKNOWN } } }
I20260129 02:58:27.699239 19995 raft_consensus.cc:2468] T d5edd0a419ee46c9a6a223f8056afb4c P 544851f9b7954fd2962675f957e8836a [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eafca908701c410ab9488970fcb3d74e in term 3.
I20260129 02:58:27.700284 19884 leader_election.cc:304] T d5edd0a419ee46c9a6a223f8056afb4c P eafca908701c410ab9488970fcb3d74e [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 544851f9b7954fd2962675f957e8836a, eafca908701c410ab9488970fcb3d74e; no voters:
I20260129 02:58:27.700569 20069 raft_consensus.cc:2468] T d5edd0a419ee46c9a6a223f8056afb4c P a634c60a5ca74b26b0d1f679d0166d0c [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eafca908701c410ab9488970fcb3d74e in term 3.
I20260129 02:58:27.701018 20242 raft_consensus.cc:2804] T d5edd0a419ee46c9a6a223f8056afb4c P eafca908701c410ab9488970fcb3d74e [term 3 FOLLOWER]: Leader election won for term 3
I20260129 02:58:27.701462 20242 raft_consensus.cc:697] T d5edd0a419ee46c9a6a223f8056afb4c P eafca908701c410ab9488970fcb3d74e [term 3 LEADER]: Becoming Leader. State: Replica: eafca908701c410ab9488970fcb3d74e, State: Running, Role: LEADER
I20260129 02:58:27.702251 20242 consensus_queue.cc:237] T d5edd0a419ee46c9a6a223f8056afb4c P eafca908701c410ab9488970fcb3d74e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eafca908701c410ab9488970fcb3d74e" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 45001 } } peers { permanent_uuid: "544851f9b7954fd2962675f957e8836a" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34199 } } peers { permanent_uuid: "a634c60a5ca74b26b0d1f679d0166d0c" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 39941 } }
I20260129 02:58:27.710078 19817 catalog_manager.cc:5654] T d5edd0a419ee46c9a6a223f8056afb4c P eafca908701c410ab9488970fcb3d74e reported cstate change: term changed from 2 to 3, leader changed from a634c60a5ca74b26b0d1f679d0166d0c (127.18.211.3) to eafca908701c410ab9488970fcb3d74e (127.18.211.1). New cstate: current_term: 3 leader_uuid: "eafca908701c410ab9488970fcb3d74e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eafca908701c410ab9488970fcb3d74e" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 45001 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "544851f9b7954fd2962675f957e8836a" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34199 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a634c60a5ca74b26b0d1f679d0166d0c" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 39941 } health_report { overall_health: UNKNOWN } } }
I20260129 02:58:27.854688 19995 raft_consensus.cc:1275] T 1dc801ccecc44ed79d85b0522e49d3e5 P 544851f9b7954fd2962675f957e8836a [term 2 FOLLOWER]: Refusing update from remote peer eafca908701c410ab9488970fcb3d74e: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260129 02:58:27.855902 20242 consensus_queue.cc:1048] T 1dc801ccecc44ed79d85b0522e49d3e5 P eafca908701c410ab9488970fcb3d74e [LEADER]: Connected to new peer: Peer: permanent_uuid: "544851f9b7954fd2962675f957e8836a" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34199 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260129 02:58:27.864509 20069 raft_consensus.cc:1275] T 1dc801ccecc44ed79d85b0522e49d3e5 P a634c60a5ca74b26b0d1f679d0166d0c [term 2 FOLLOWER]: Refusing update from remote peer eafca908701c410ab9488970fcb3d74e: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260129 02:58:27.866398 20254 consensus_queue.cc:1048] T 1dc801ccecc44ed79d85b0522e49d3e5 P eafca908701c410ab9488970fcb3d74e [LEADER]: Connected to new peer: Peer: permanent_uuid: "a634c60a5ca74b26b0d1f679d0166d0c" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 39941 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260129 02:58:27.976783 20069 raft_consensus.cc:1275] T 4562a703e5a44a1585b332b851bfd6f2 P a634c60a5ca74b26b0d1f679d0166d0c [term 4 FOLLOWER]: Refusing update from remote peer eafca908701c410ab9488970fcb3d74e: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20260129 02:58:27.977996 20254 consensus_queue.cc:1048] T 4562a703e5a44a1585b332b851bfd6f2 P eafca908701c410ab9488970fcb3d74e [LEADER]: Connected to new peer: Peer: permanent_uuid: "a634c60a5ca74b26b0d1f679d0166d0c" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 39941 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260129 02:58:27.987361 19995 raft_consensus.cc:1275] T 4562a703e5a44a1585b332b851bfd6f2 P 544851f9b7954fd2962675f957e8836a [term 4 FOLLOWER]: Refusing update from remote peer eafca908701c410ab9488970fcb3d74e: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20260129 02:58:27.988585 20242 consensus_queue.cc:1048] T 4562a703e5a44a1585b332b851bfd6f2 P eafca908701c410ab9488970fcb3d74e [LEADER]: Connected to new peer: Peer: permanent_uuid: "544851f9b7954fd2962675f957e8836a" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34199 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260129 02:58:28.233196 20069 raft_consensus.cc:1275] T 901f619b02f843b4a2747de956182f12 P a634c60a5ca74b26b0d1f679d0166d0c [term 3 FOLLOWER]: Refusing update from remote peer eafca908701c410ab9488970fcb3d74e: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260129 02:58:28.234367 20233 consensus_queue.cc:1048] T 901f619b02f843b4a2747de956182f12 P eafca908701c410ab9488970fcb3d74e [LEADER]: Connected to new peer: Peer: permanent_uuid: "a634c60a5ca74b26b0d1f679d0166d0c" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 39941 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260129 02:58:28.243191 19995 raft_consensus.cc:1275] T 901f619b02f843b4a2747de956182f12 P 544851f9b7954fd2962675f957e8836a [term 3 FOLLOWER]: Refusing update from remote peer eafca908701c410ab9488970fcb3d74e: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260129 02:58:28.244735 20264 consensus_queue.cc:1048] T 901f619b02f843b4a2747de956182f12 P eafca908701c410ab9488970fcb3d74e [LEADER]: Connected to new peer: Peer: permanent_uuid: "544851f9b7954fd2962675f957e8836a" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34199 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260129 02:58:28.267354 19995 raft_consensus.cc:1275] T d5edd0a419ee46c9a6a223f8056afb4c P 544851f9b7954fd2962675f957e8836a [term 3 FOLLOWER]: Refusing update from remote peer eafca908701c410ab9488970fcb3d74e: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260129 02:58:28.268466 20264 consensus_queue.cc:1048] T d5edd0a419ee46c9a6a223f8056afb4c P eafca908701c410ab9488970fcb3d74e [LEADER]: Connected to new peer: Peer: permanent_uuid: "544851f9b7954fd2962675f957e8836a" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34199 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260129 02:58:28.276511 20069 raft_consensus.cc:1275] T d5edd0a419ee46c9a6a223f8056afb4c P a634c60a5ca74b26b0d1f679d0166d0c [term 3 FOLLOWER]: Refusing update from remote peer eafca908701c410ab9488970fcb3d74e: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260129 02:58:28.277894 20264 consensus_queue.cc:1048] T d5edd0a419ee46c9a6a223f8056afb4c P eafca908701c410ab9488970fcb3d74e [LEADER]: Connected to new peer: Peer: permanent_uuid: "a634c60a5ca74b26b0d1f679d0166d0c" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 39941 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260129 02:58:30.215559 19276 auto_leader_rebalancer-test.cc:377] The leader distribution is
I20260129 02:58:30.216409 19276 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260129 02:58:30.224102 19921 tablet_service.cc:2066] Received LeaderStepDown RPC: tablet_id: "1dc801ccecc44ed79d85b0522e49d3e5"
dest_uuid: "eafca908701c410ab9488970fcb3d74e"
mode: GRACEFUL
new_leader_uuid: "a634c60a5ca74b26b0d1f679d0166d0c"
from {username='slave'} at 127.0.0.1:57002
I20260129 02:58:30.224588 19921 raft_consensus.cc:606] T 1dc801ccecc44ed79d85b0522e49d3e5 P eafca908701c410ab9488970fcb3d74e [term 2 LEADER]: Received request to transfer leadership to TS a634c60a5ca74b26b0d1f679d0166d0c
I20260129 02:58:30.226873 19921 tablet_service.cc:2066] Received LeaderStepDown RPC: tablet_id: "4562a703e5a44a1585b332b851bfd6f2"
dest_uuid: "eafca908701c410ab9488970fcb3d74e"
mode: GRACEFUL
new_leader_uuid: "544851f9b7954fd2962675f957e8836a"
from {username='slave'} at 127.0.0.1:57002
I20260129 02:58:30.227348 19921 raft_consensus.cc:606] T 4562a703e5a44a1585b332b851bfd6f2 P eafca908701c410ab9488970fcb3d74e [term 4 LEADER]: Received request to transfer leadership to TS 544851f9b7954fd2962675f957e8836a
I20260129 02:58:30.229310 19921 tablet_service.cc:2066] Received LeaderStepDown RPC: tablet_id: "79355f4f5d7942b6ac6f420256fd35b7"
dest_uuid: "eafca908701c410ab9488970fcb3d74e"
mode: GRACEFUL
new_leader_uuid: "544851f9b7954fd2962675f957e8836a"
from {username='slave'} at 127.0.0.1:57002
I20260129 02:58:30.229727 19921 raft_consensus.cc:606] T 79355f4f5d7942b6ac6f420256fd35b7 P eafca908701c410ab9488970fcb3d74e [term 2 LEADER]: Received request to transfer leadership to TS 544851f9b7954fd2962675f957e8836a
I20260129 02:58:30.231664 19921 tablet_service.cc:2066] Received LeaderStepDown RPC: tablet_id: "901f619b02f843b4a2747de956182f12"
dest_uuid: "eafca908701c410ab9488970fcb3d74e"
mode: GRACEFUL
new_leader_uuid: "a634c60a5ca74b26b0d1f679d0166d0c"
from {username='slave'} at 127.0.0.1:57002
I20260129 02:58:30.232156 19921 raft_consensus.cc:606] T 901f619b02f843b4a2747de956182f12 P eafca908701c410ab9488970fcb3d74e [term 3 LEADER]: Received request to transfer leadership to TS a634c60a5ca74b26b0d1f679d0166d0c
I20260129 02:58:30.232985 19276 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 4
I20260129 02:58:30.233390 19276 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20260129 02:58:30.322880 20268 raft_consensus.cc:993] T 901f619b02f843b4a2747de956182f12 P eafca908701c410ab9488970fcb3d74e: : Instructing follower a634c60a5ca74b26b0d1f679d0166d0c to start an election
I20260129 02:58:30.323303 20270 raft_consensus.cc:1081] T 901f619b02f843b4a2747de956182f12 P eafca908701c410ab9488970fcb3d74e [term 3 LEADER]: Signalling peer a634c60a5ca74b26b0d1f679d0166d0c to start an election
I20260129 02:58:30.324609 20069 tablet_service.cc:2038] Received Run Leader Election RPC: tablet_id: "901f619b02f843b4a2747de956182f12"
dest_uuid: "a634c60a5ca74b26b0d1f679d0166d0c"
from {username='slave'} at 127.0.0.1:43236
I20260129 02:58:30.325019 20069 raft_consensus.cc:493] T 901f619b02f843b4a2747de956182f12 P a634c60a5ca74b26b0d1f679d0166d0c [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20260129 02:58:30.325253 20069 raft_consensus.cc:3060] T 901f619b02f843b4a2747de956182f12 P a634c60a5ca74b26b0d1f679d0166d0c [term 3 FOLLOWER]: Advancing to term 4
I20260129 02:58:30.329802 20069 raft_consensus.cc:515] T 901f619b02f843b4a2747de956182f12 P a634c60a5ca74b26b0d1f679d0166d0c [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eafca908701c410ab9488970fcb3d74e" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 45001 } } peers { permanent_uuid: "544851f9b7954fd2962675f957e8836a" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34199 } } peers { permanent_uuid: "a634c60a5ca74b26b0d1f679d0166d0c" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 39941 } }
I20260129 02:58:30.331579 20069 leader_election.cc:290] T 901f619b02f843b4a2747de956182f12 P a634c60a5ca74b26b0d1f679d0166d0c [CANDIDATE]: Term 4 election: Requested vote from peers eafca908701c410ab9488970fcb3d74e (127.18.211.1:45001), 544851f9b7954fd2962675f957e8836a (127.18.211.2:34199)
I20260129 02:58:30.332434 19921 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "901f619b02f843b4a2747de956182f12" candidate_uuid: "a634c60a5ca74b26b0d1f679d0166d0c" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: true dest_uuid: "eafca908701c410ab9488970fcb3d74e"
I20260129 02:58:30.332821 19994 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "901f619b02f843b4a2747de956182f12" candidate_uuid: "a634c60a5ca74b26b0d1f679d0166d0c" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: true dest_uuid: "544851f9b7954fd2962675f957e8836a"
I20260129 02:58:30.332947 19921 raft_consensus.cc:3055] T 901f619b02f843b4a2747de956182f12 P eafca908701c410ab9488970fcb3d74e [term 3 LEADER]: Stepping down as leader of term 3
I20260129 02:58:30.333240 19921 raft_consensus.cc:740] T 901f619b02f843b4a2747de956182f12 P eafca908701c410ab9488970fcb3d74e [term 3 LEADER]: Becoming Follower/Learner. State: Replica: eafca908701c410ab9488970fcb3d74e, State: Running, Role: LEADER
I20260129 02:58:30.333300 19994 raft_consensus.cc:3060] T 901f619b02f843b4a2747de956182f12 P 544851f9b7954fd2962675f957e8836a [term 3 FOLLOWER]: Advancing to term 4
I20260129 02:58:30.333849 19921 consensus_queue.cc:260] T 901f619b02f843b4a2747de956182f12 P eafca908701c410ab9488970fcb3d74e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eafca908701c410ab9488970fcb3d74e" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 45001 } } peers { permanent_uuid: "544851f9b7954fd2962675f957e8836a" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34199 } } peers { permanent_uuid: "a634c60a5ca74b26b0d1f679d0166d0c" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 39941 } }
I20260129 02:58:30.334715 19921 raft_consensus.cc:3060] T 901f619b02f843b4a2747de956182f12 P eafca908701c410ab9488970fcb3d74e [term 3 FOLLOWER]: Advancing to term 4
I20260129 02:58:30.338510 19994 raft_consensus.cc:2468] T 901f619b02f843b4a2747de956182f12 P 544851f9b7954fd2962675f957e8836a [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a634c60a5ca74b26b0d1f679d0166d0c in term 4.
I20260129 02:58:30.338973 19921 raft_consensus.cc:2468] T 901f619b02f843b4a2747de956182f12 P eafca908701c410ab9488970fcb3d74e [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a634c60a5ca74b26b0d1f679d0166d0c in term 4.
I20260129 02:58:30.339557 20033 leader_election.cc:304] T 901f619b02f843b4a2747de956182f12 P a634c60a5ca74b26b0d1f679d0166d0c [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 544851f9b7954fd2962675f957e8836a, a634c60a5ca74b26b0d1f679d0166d0c; no voters:
I20260129 02:58:30.340211 20248 raft_consensus.cc:2804] T 901f619b02f843b4a2747de956182f12 P a634c60a5ca74b26b0d1f679d0166d0c [term 4 FOLLOWER]: Leader election won for term 4
I20260129 02:58:30.341037 20248 raft_consensus.cc:697] T 901f619b02f843b4a2747de956182f12 P a634c60a5ca74b26b0d1f679d0166d0c [term 4 LEADER]: Becoming Leader. State: Replica: a634c60a5ca74b26b0d1f679d0166d0c, State: Running, Role: LEADER
I20260129 02:58:30.341908 20248 consensus_queue.cc:237] T 901f619b02f843b4a2747de956182f12 P a634c60a5ca74b26b0d1f679d0166d0c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eafca908701c410ab9488970fcb3d74e" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 45001 } } peers { permanent_uuid: "544851f9b7954fd2962675f957e8836a" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34199 } } peers { permanent_uuid: "a634c60a5ca74b26b0d1f679d0166d0c" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 39941 } }
I20260129 02:58:30.348035 19816 catalog_manager.cc:5654] T 901f619b02f843b4a2747de956182f12 P a634c60a5ca74b26b0d1f679d0166d0c reported cstate change: term changed from 3 to 4, leader changed from eafca908701c410ab9488970fcb3d74e (127.18.211.1) to a634c60a5ca74b26b0d1f679d0166d0c (127.18.211.3). New cstate: current_term: 4 leader_uuid: "a634c60a5ca74b26b0d1f679d0166d0c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eafca908701c410ab9488970fcb3d74e" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 45001 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "544851f9b7954fd2962675f957e8836a" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34199 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a634c60a5ca74b26b0d1f679d0166d0c" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 39941 } health_report { overall_health: HEALTHY } } }
I20260129 02:58:30.627995 20268 raft_consensus.cc:993] T 1dc801ccecc44ed79d85b0522e49d3e5 P eafca908701c410ab9488970fcb3d74e: : Instructing follower a634c60a5ca74b26b0d1f679d0166d0c to start an election
I20260129 02:58:30.628438 20270 raft_consensus.cc:1081] T 1dc801ccecc44ed79d85b0522e49d3e5 P eafca908701c410ab9488970fcb3d74e [term 2 LEADER]: Signalling peer a634c60a5ca74b26b0d1f679d0166d0c to start an election
I20260129 02:58:30.629678 20069 tablet_service.cc:2038] Received Run Leader Election RPC: tablet_id: "1dc801ccecc44ed79d85b0522e49d3e5"
dest_uuid: "a634c60a5ca74b26b0d1f679d0166d0c"
from {username='slave'} at 127.0.0.1:43236
I20260129 02:58:30.630187 20069 raft_consensus.cc:493] T 1dc801ccecc44ed79d85b0522e49d3e5 P a634c60a5ca74b26b0d1f679d0166d0c [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20260129 02:58:30.630493 20069 raft_consensus.cc:3060] T 1dc801ccecc44ed79d85b0522e49d3e5 P a634c60a5ca74b26b0d1f679d0166d0c [term 2 FOLLOWER]: Advancing to term 3
I20260129 02:58:30.635098 20069 raft_consensus.cc:515] T 1dc801ccecc44ed79d85b0522e49d3e5 P a634c60a5ca74b26b0d1f679d0166d0c [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eafca908701c410ab9488970fcb3d74e" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 45001 } } peers { permanent_uuid: "544851f9b7954fd2962675f957e8836a" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34199 } } peers { permanent_uuid: "a634c60a5ca74b26b0d1f679d0166d0c" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 39941 } }
I20260129 02:58:30.636615 20069 leader_election.cc:290] T 1dc801ccecc44ed79d85b0522e49d3e5 P a634c60a5ca74b26b0d1f679d0166d0c [CANDIDATE]: Term 3 election: Requested vote from peers eafca908701c410ab9488970fcb3d74e (127.18.211.1:45001), 544851f9b7954fd2962675f957e8836a (127.18.211.2:34199)
I20260129 02:58:30.637387 19921 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1dc801ccecc44ed79d85b0522e49d3e5" candidate_uuid: "a634c60a5ca74b26b0d1f679d0166d0c" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "eafca908701c410ab9488970fcb3d74e"
I20260129 02:58:30.637645 19994 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1dc801ccecc44ed79d85b0522e49d3e5" candidate_uuid: "a634c60a5ca74b26b0d1f679d0166d0c" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "544851f9b7954fd2962675f957e8836a"
I20260129 02:58:30.637882 19921 raft_consensus.cc:3055] T 1dc801ccecc44ed79d85b0522e49d3e5 P eafca908701c410ab9488970fcb3d74e [term 2 LEADER]: Stepping down as leader of term 2
I20260129 02:58:30.638214 19921 raft_consensus.cc:740] T 1dc801ccecc44ed79d85b0522e49d3e5 P eafca908701c410ab9488970fcb3d74e [term 2 LEADER]: Becoming Follower/Learner. State: Replica: eafca908701c410ab9488970fcb3d74e, State: Running, Role: LEADER
I20260129 02:58:30.638252 19994 raft_consensus.cc:3060] T 1dc801ccecc44ed79d85b0522e49d3e5 P 544851f9b7954fd2962675f957e8836a [term 2 FOLLOWER]: Advancing to term 3
I20260129 02:58:30.638833 19921 consensus_queue.cc:260] T 1dc801ccecc44ed79d85b0522e49d3e5 P eafca908701c410ab9488970fcb3d74e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eafca908701c410ab9488970fcb3d74e" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 45001 } } peers { permanent_uuid: "544851f9b7954fd2962675f957e8836a" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34199 } } peers { permanent_uuid: "a634c60a5ca74b26b0d1f679d0166d0c" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 39941 } }
I20260129 02:58:30.639680 19921 raft_consensus.cc:3060] T 1dc801ccecc44ed79d85b0522e49d3e5 P eafca908701c410ab9488970fcb3d74e [term 2 FOLLOWER]: Advancing to term 3
I20260129 02:58:30.644353 19994 raft_consensus.cc:2468] T 1dc801ccecc44ed79d85b0522e49d3e5 P 544851f9b7954fd2962675f957e8836a [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a634c60a5ca74b26b0d1f679d0166d0c in term 3.
I20260129 02:58:30.644984 19921 raft_consensus.cc:2468] T 1dc801ccecc44ed79d85b0522e49d3e5 P eafca908701c410ab9488970fcb3d74e [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a634c60a5ca74b26b0d1f679d0166d0c in term 3.
I20260129 02:58:30.645401 20033 leader_election.cc:304] T 1dc801ccecc44ed79d85b0522e49d3e5 P a634c60a5ca74b26b0d1f679d0166d0c [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 544851f9b7954fd2962675f957e8836a, a634c60a5ca74b26b0d1f679d0166d0c; no voters:
I20260129 02:58:30.646142 20248 raft_consensus.cc:2804] T 1dc801ccecc44ed79d85b0522e49d3e5 P a634c60a5ca74b26b0d1f679d0166d0c [term 3 FOLLOWER]: Leader election won for term 3
I20260129 02:58:30.646603 20248 raft_consensus.cc:697] T 1dc801ccecc44ed79d85b0522e49d3e5 P a634c60a5ca74b26b0d1f679d0166d0c [term 3 LEADER]: Becoming Leader. State: Replica: a634c60a5ca74b26b0d1f679d0166d0c, State: Running, Role: LEADER
I20260129 02:58:30.647387 20248 consensus_queue.cc:237] T 1dc801ccecc44ed79d85b0522e49d3e5 P a634c60a5ca74b26b0d1f679d0166d0c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eafca908701c410ab9488970fcb3d74e" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 45001 } } peers { permanent_uuid: "544851f9b7954fd2962675f957e8836a" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34199 } } peers { permanent_uuid: "a634c60a5ca74b26b0d1f679d0166d0c" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 39941 } }
I20260129 02:58:30.653625 19816 catalog_manager.cc:5654] T 1dc801ccecc44ed79d85b0522e49d3e5 P a634c60a5ca74b26b0d1f679d0166d0c reported cstate change: term changed from 2 to 3, leader changed from eafca908701c410ab9488970fcb3d74e (127.18.211.1) to a634c60a5ca74b26b0d1f679d0166d0c (127.18.211.3). New cstate: current_term: 3 leader_uuid: "a634c60a5ca74b26b0d1f679d0166d0c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eafca908701c410ab9488970fcb3d74e" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 45001 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "544851f9b7954fd2962675f957e8836a" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34199 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a634c60a5ca74b26b0d1f679d0166d0c" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 39941 } health_report { overall_health: HEALTHY } } }
I20260129 02:58:30.708974 20268 raft_consensus.cc:993] T 4562a703e5a44a1585b332b851bfd6f2 P eafca908701c410ab9488970fcb3d74e: : Instructing follower 544851f9b7954fd2962675f957e8836a to start an election
I20260129 02:58:30.709420 20270 raft_consensus.cc:1081] T 4562a703e5a44a1585b332b851bfd6f2 P eafca908701c410ab9488970fcb3d74e [term 4 LEADER]: Signalling peer 544851f9b7954fd2962675f957e8836a to start an election
I20260129 02:58:30.710759 19994 tablet_service.cc:2038] Received Run Leader Election RPC: tablet_id: "4562a703e5a44a1585b332b851bfd6f2"
dest_uuid: "544851f9b7954fd2962675f957e8836a"
from {username='slave'} at 127.0.0.1:35902
I20260129 02:58:30.711216 19994 raft_consensus.cc:493] T 4562a703e5a44a1585b332b851bfd6f2 P 544851f9b7954fd2962675f957e8836a [term 4 FOLLOWER]: Starting forced leader election (received explicit request)
I20260129 02:58:30.711510 19994 raft_consensus.cc:3060] T 4562a703e5a44a1585b332b851bfd6f2 P 544851f9b7954fd2962675f957e8836a [term 4 FOLLOWER]: Advancing to term 5
I20260129 02:58:30.717463 19994 raft_consensus.cc:515] T 4562a703e5a44a1585b332b851bfd6f2 P 544851f9b7954fd2962675f957e8836a [term 5 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eafca908701c410ab9488970fcb3d74e" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 45001 } } peers { permanent_uuid: "544851f9b7954fd2962675f957e8836a" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34199 } } peers { permanent_uuid: "a634c60a5ca74b26b0d1f679d0166d0c" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 39941 } }
I20260129 02:58:30.719470 19994 leader_election.cc:290] T 4562a703e5a44a1585b332b851bfd6f2 P 544851f9b7954fd2962675f957e8836a [CANDIDATE]: Term 5 election: Requested vote from peers eafca908701c410ab9488970fcb3d74e (127.18.211.1:45001), a634c60a5ca74b26b0d1f679d0166d0c (127.18.211.3:39941)
I20260129 02:58:30.720140 19921 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4562a703e5a44a1585b332b851bfd6f2" candidate_uuid: "544851f9b7954fd2962675f957e8836a" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: true dest_uuid: "eafca908701c410ab9488970fcb3d74e"
I20260129 02:58:30.720712 19921 raft_consensus.cc:3055] T 4562a703e5a44a1585b332b851bfd6f2 P eafca908701c410ab9488970fcb3d74e [term 4 LEADER]: Stepping down as leader of term 4
I20260129 02:58:30.720613 20069 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4562a703e5a44a1585b332b851bfd6f2" candidate_uuid: "544851f9b7954fd2962675f957e8836a" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: true dest_uuid: "a634c60a5ca74b26b0d1f679d0166d0c"
I20260129 02:58:30.721042 19921 raft_consensus.cc:740] T 4562a703e5a44a1585b332b851bfd6f2 P eafca908701c410ab9488970fcb3d74e [term 4 LEADER]: Becoming Follower/Learner. State: Replica: eafca908701c410ab9488970fcb3d74e, State: Running, Role: LEADER
I20260129 02:58:30.721148 20069 raft_consensus.cc:3060] T 4562a703e5a44a1585b332b851bfd6f2 P a634c60a5ca74b26b0d1f679d0166d0c [term 4 FOLLOWER]: Advancing to term 5
I20260129 02:58:30.721735 19921 consensus_queue.cc:260] T 4562a703e5a44a1585b332b851bfd6f2 P eafca908701c410ab9488970fcb3d74e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 4.4, Last appended by leader: 4, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eafca908701c410ab9488970fcb3d74e" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 45001 } } peers { permanent_uuid: "544851f9b7954fd2962675f957e8836a" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34199 } } peers { permanent_uuid: "a634c60a5ca74b26b0d1f679d0166d0c" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 39941 } }
I20260129 02:58:30.722723 19921 raft_consensus.cc:3060] T 4562a703e5a44a1585b332b851bfd6f2 P eafca908701c410ab9488970fcb3d74e [term 4 FOLLOWER]: Advancing to term 5
I20260129 02:58:30.725785 20069 raft_consensus.cc:2468] T 4562a703e5a44a1585b332b851bfd6f2 P a634c60a5ca74b26b0d1f679d0166d0c [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 544851f9b7954fd2962675f957e8836a in term 5.
I20260129 02:58:30.726943 19960 leader_election.cc:304] T 4562a703e5a44a1585b332b851bfd6f2 P 544851f9b7954fd2962675f957e8836a [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 544851f9b7954fd2962675f957e8836a, a634c60a5ca74b26b0d1f679d0166d0c; no voters:
I20260129 02:58:30.727391 19921 raft_consensus.cc:2468] T 4562a703e5a44a1585b332b851bfd6f2 P eafca908701c410ab9488970fcb3d74e [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 544851f9b7954fd2962675f957e8836a in term 5.
I20260129 02:58:30.727701 20239 raft_consensus.cc:2804] T 4562a703e5a44a1585b332b851bfd6f2 P 544851f9b7954fd2962675f957e8836a [term 5 FOLLOWER]: Leader election won for term 5
I20260129 02:58:30.728930 20239 raft_consensus.cc:697] T 4562a703e5a44a1585b332b851bfd6f2 P 544851f9b7954fd2962675f957e8836a [term 5 LEADER]: Becoming Leader. State: Replica: 544851f9b7954fd2962675f957e8836a, State: Running, Role: LEADER
I20260129 02:58:30.729660 20239 consensus_queue.cc:237] T 4562a703e5a44a1585b332b851bfd6f2 P 544851f9b7954fd2962675f957e8836a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 4.4, Last appended by leader: 4, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eafca908701c410ab9488970fcb3d74e" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 45001 } } peers { permanent_uuid: "544851f9b7954fd2962675f957e8836a" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34199 } } peers { permanent_uuid: "a634c60a5ca74b26b0d1f679d0166d0c" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 39941 } }
I20260129 02:58:30.737080 19816 catalog_manager.cc:5654] T 4562a703e5a44a1585b332b851bfd6f2 P 544851f9b7954fd2962675f957e8836a reported cstate change: term changed from 4 to 5, leader changed from eafca908701c410ab9488970fcb3d74e (127.18.211.1) to 544851f9b7954fd2962675f957e8836a (127.18.211.2). New cstate: current_term: 5 leader_uuid: "544851f9b7954fd2962675f957e8836a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eafca908701c410ab9488970fcb3d74e" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 45001 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "544851f9b7954fd2962675f957e8836a" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34199 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a634c60a5ca74b26b0d1f679d0166d0c" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 39941 } health_report { overall_health: UNKNOWN } } }
I20260129 02:58:30.792124 20268 raft_consensus.cc:993] T 79355f4f5d7942b6ac6f420256fd35b7 P eafca908701c410ab9488970fcb3d74e: : Instructing follower 544851f9b7954fd2962675f957e8836a to start an election
I20260129 02:58:30.792611 20270 raft_consensus.cc:1081] T 79355f4f5d7942b6ac6f420256fd35b7 P eafca908701c410ab9488970fcb3d74e [term 2 LEADER]: Signalling peer 544851f9b7954fd2962675f957e8836a to start an election
I20260129 02:58:30.793968 19994 tablet_service.cc:2038] Received Run Leader Election RPC: tablet_id: "79355f4f5d7942b6ac6f420256fd35b7"
dest_uuid: "544851f9b7954fd2962675f957e8836a"
from {username='slave'} at 127.0.0.1:35902
I20260129 02:58:30.794471 19994 raft_consensus.cc:493] T 79355f4f5d7942b6ac6f420256fd35b7 P 544851f9b7954fd2962675f957e8836a [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20260129 02:58:30.794744 19994 raft_consensus.cc:3060] T 79355f4f5d7942b6ac6f420256fd35b7 P 544851f9b7954fd2962675f957e8836a [term 2 FOLLOWER]: Advancing to term 3
I20260129 02:58:30.800796 19994 raft_consensus.cc:515] T 79355f4f5d7942b6ac6f420256fd35b7 P 544851f9b7954fd2962675f957e8836a [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eafca908701c410ab9488970fcb3d74e" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 45001 } } peers { permanent_uuid: "544851f9b7954fd2962675f957e8836a" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34199 } } peers { permanent_uuid: "a634c60a5ca74b26b0d1f679d0166d0c" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 39941 } }
I20260129 02:58:30.802286 19994 leader_election.cc:290] T 79355f4f5d7942b6ac6f420256fd35b7 P 544851f9b7954fd2962675f957e8836a [CANDIDATE]: Term 3 election: Requested vote from peers eafca908701c410ab9488970fcb3d74e (127.18.211.1:45001), a634c60a5ca74b26b0d1f679d0166d0c (127.18.211.3:39941)
I20260129 02:58:30.803023 19921 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "79355f4f5d7942b6ac6f420256fd35b7" candidate_uuid: "544851f9b7954fd2962675f957e8836a" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "eafca908701c410ab9488970fcb3d74e"
I20260129 02:58:30.803287 20069 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "79355f4f5d7942b6ac6f420256fd35b7" candidate_uuid: "544851f9b7954fd2962675f957e8836a" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "a634c60a5ca74b26b0d1f679d0166d0c"
I20260129 02:58:30.803666 19921 raft_consensus.cc:3055] T 79355f4f5d7942b6ac6f420256fd35b7 P eafca908701c410ab9488970fcb3d74e [term 2 LEADER]: Stepping down as leader of term 2
I20260129 02:58:30.803880 20069 raft_consensus.cc:3060] T 79355f4f5d7942b6ac6f420256fd35b7 P a634c60a5ca74b26b0d1f679d0166d0c [term 2 FOLLOWER]: Advancing to term 3
I20260129 02:58:30.804103 19921 raft_consensus.cc:740] T 79355f4f5d7942b6ac6f420256fd35b7 P eafca908701c410ab9488970fcb3d74e [term 2 LEADER]: Becoming Follower/Learner. State: Replica: eafca908701c410ab9488970fcb3d74e, State: Running, Role: LEADER
I20260129 02:58:30.805070 19921 consensus_queue.cc:260] T 79355f4f5d7942b6ac6f420256fd35b7 P eafca908701c410ab9488970fcb3d74e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eafca908701c410ab9488970fcb3d74e" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 45001 } } peers { permanent_uuid: "544851f9b7954fd2962675f957e8836a" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34199 } } peers { permanent_uuid: "a634c60a5ca74b26b0d1f679d0166d0c" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 39941 } }
I20260129 02:58:30.806337 19921 raft_consensus.cc:3060] T 79355f4f5d7942b6ac6f420256fd35b7 P eafca908701c410ab9488970fcb3d74e [term 2 FOLLOWER]: Advancing to term 3
I20260129 02:58:30.809726 20069 raft_consensus.cc:2468] T 79355f4f5d7942b6ac6f420256fd35b7 P a634c60a5ca74b26b0d1f679d0166d0c [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 544851f9b7954fd2962675f957e8836a in term 3.
I20260129 02:58:30.810842 19960 leader_election.cc:304] T 79355f4f5d7942b6ac6f420256fd35b7 P 544851f9b7954fd2962675f957e8836a [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 544851f9b7954fd2962675f957e8836a, a634c60a5ca74b26b0d1f679d0166d0c; no voters:
I20260129 02:58:30.811668 20239 raft_consensus.cc:2804] T 79355f4f5d7942b6ac6f420256fd35b7 P 544851f9b7954fd2962675f957e8836a [term 3 FOLLOWER]: Leader election won for term 3
I20260129 02:58:30.811739 19921 raft_consensus.cc:2468] T 79355f4f5d7942b6ac6f420256fd35b7 P eafca908701c410ab9488970fcb3d74e [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 544851f9b7954fd2962675f957e8836a in term 3.
I20260129 02:58:30.812253 20239 raft_consensus.cc:697] T 79355f4f5d7942b6ac6f420256fd35b7 P 544851f9b7954fd2962675f957e8836a [term 3 LEADER]: Becoming Leader. State: Replica: 544851f9b7954fd2962675f957e8836a, State: Running, Role: LEADER
I20260129 02:58:30.813174 20239 consensus_queue.cc:237] T 79355f4f5d7942b6ac6f420256fd35b7 P 544851f9b7954fd2962675f957e8836a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eafca908701c410ab9488970fcb3d74e" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 45001 } } peers { permanent_uuid: "544851f9b7954fd2962675f957e8836a" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34199 } } peers { permanent_uuid: "a634c60a5ca74b26b0d1f679d0166d0c" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 39941 } }
I20260129 02:58:30.820412 19816 catalog_manager.cc:5654] T 79355f4f5d7942b6ac6f420256fd35b7 P 544851f9b7954fd2962675f957e8836a reported cstate change: term changed from 2 to 3, leader changed from eafca908701c410ab9488970fcb3d74e (127.18.211.1) to 544851f9b7954fd2962675f957e8836a (127.18.211.2). New cstate: current_term: 3 leader_uuid: "544851f9b7954fd2962675f957e8836a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eafca908701c410ab9488970fcb3d74e" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 45001 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "544851f9b7954fd2962675f957e8836a" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34199 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a634c60a5ca74b26b0d1f679d0166d0c" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 39941 } health_report { overall_health: UNKNOWN } } }
I20260129 02:58:30.854303 19921 raft_consensus.cc:1275] T 901f619b02f843b4a2747de956182f12 P eafca908701c410ab9488970fcb3d74e [term 4 FOLLOWER]: Refusing update from remote peer a634c60a5ca74b26b0d1f679d0166d0c: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20260129 02:58:30.854305 19994 raft_consensus.cc:1275] T 901f619b02f843b4a2747de956182f12 P 544851f9b7954fd2962675f957e8836a [term 4 FOLLOWER]: Refusing update from remote peer a634c60a5ca74b26b0d1f679d0166d0c: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20260129 02:58:30.855852 20248 consensus_queue.cc:1048] T 901f619b02f843b4a2747de956182f12 P a634c60a5ca74b26b0d1f679d0166d0c [LEADER]: Connected to new peer: Peer: permanent_uuid: "eafca908701c410ab9488970fcb3d74e" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 45001 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260129 02:58:30.856540 20271 consensus_queue.cc:1048] T 901f619b02f843b4a2747de956182f12 P a634c60a5ca74b26b0d1f679d0166d0c [LEADER]: Connected to new peer: Peer: permanent_uuid: "544851f9b7954fd2962675f957e8836a" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34199 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260129 02:58:31.133136 19994 raft_consensus.cc:1275] T 1dc801ccecc44ed79d85b0522e49d3e5 P 544851f9b7954fd2962675f957e8836a [term 3 FOLLOWER]: Refusing update from remote peer a634c60a5ca74b26b0d1f679d0166d0c: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260129 02:58:31.134173 20283 consensus_queue.cc:1048] T 1dc801ccecc44ed79d85b0522e49d3e5 P a634c60a5ca74b26b0d1f679d0166d0c [LEADER]: Connected to new peer: Peer: permanent_uuid: "544851f9b7954fd2962675f957e8836a" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34199 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260129 02:58:31.143584 19921 raft_consensus.cc:1275] T 1dc801ccecc44ed79d85b0522e49d3e5 P eafca908701c410ab9488970fcb3d74e [term 3 FOLLOWER]: Refusing update from remote peer a634c60a5ca74b26b0d1f679d0166d0c: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260129 02:58:31.145124 20283 consensus_queue.cc:1048] T 1dc801ccecc44ed79d85b0522e49d3e5 P a634c60a5ca74b26b0d1f679d0166d0c [LEADER]: Connected to new peer: Peer: permanent_uuid: "eafca908701c410ab9488970fcb3d74e" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 45001 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260129 02:58:31.158144 20069 raft_consensus.cc:1275] T 4562a703e5a44a1585b332b851bfd6f2 P a634c60a5ca74b26b0d1f679d0166d0c [term 5 FOLLOWER]: Refusing update from remote peer 544851f9b7954fd2962675f957e8836a: Log matching property violated. Preceding OpId in replica: term: 4 index: 4. Preceding OpId from leader: term: 5 index: 5. (index mismatch)
I20260129 02:58:31.159304 20239 consensus_queue.cc:1048] T 4562a703e5a44a1585b332b851bfd6f2 P 544851f9b7954fd2962675f957e8836a [LEADER]: Connected to new peer: Peer: permanent_uuid: "a634c60a5ca74b26b0d1f679d0166d0c" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 39941 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260129 02:58:31.170946 19921 raft_consensus.cc:1275] T 4562a703e5a44a1585b332b851bfd6f2 P eafca908701c410ab9488970fcb3d74e [term 5 FOLLOWER]: Refusing update from remote peer 544851f9b7954fd2962675f957e8836a: Log matching property violated. Preceding OpId in replica: term: 4 index: 4. Preceding OpId from leader: term: 5 index: 5. (index mismatch)
I20260129 02:58:31.172433 20291 consensus_queue.cc:1048] T 4562a703e5a44a1585b332b851bfd6f2 P 544851f9b7954fd2962675f957e8836a [LEADER]: Connected to new peer: Peer: permanent_uuid: "eafca908701c410ab9488970fcb3d74e" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 45001 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260129 02:58:31.234284 19276 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260129 02:58:31.239776 19276 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260129 02:58:31.240175 19276 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20260129 02:58:31.332356 20069 raft_consensus.cc:1275] T 79355f4f5d7942b6ac6f420256fd35b7 P a634c60a5ca74b26b0d1f679d0166d0c [term 3 FOLLOWER]: Refusing update from remote peer 544851f9b7954fd2962675f957e8836a: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260129 02:58:31.333853 20291 consensus_queue.cc:1048] T 79355f4f5d7942b6ac6f420256fd35b7 P 544851f9b7954fd2962675f957e8836a [LEADER]: Connected to new peer: Peer: permanent_uuid: "a634c60a5ca74b26b0d1f679d0166d0c" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 39941 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260129 02:58:31.347180 19921 raft_consensus.cc:1275] T 79355f4f5d7942b6ac6f420256fd35b7 P eafca908701c410ab9488970fcb3d74e [term 3 FOLLOWER]: Refusing update from remote peer 544851f9b7954fd2962675f957e8836a: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260129 02:58:31.349150 20291 consensus_queue.cc:1048] T 79355f4f5d7942b6ac6f420256fd35b7 P 544851f9b7954fd2962675f957e8836a [LEADER]: Connected to new peer: Peer: permanent_uuid: "eafca908701c410ab9488970fcb3d74e" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 45001 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260129 02:58:32.241195 19276 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260129 02:58:32.246538 19276 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260129 02:58:32.246997 19276 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20260129 02:58:33.248168 19276 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260129 02:58:33.253304 19276 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260129 02:58:33.253741 19276 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20260129 02:58:34.254722 19276 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260129 02:58:34.259773 19276 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260129 02:58:34.260222 19276 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20260129 02:58:35.261211 19276 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260129 02:58:35.266484 19276 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260129 02:58:35.266937 19276 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20260129 02:58:36.291972 19276 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260129 02:58:36.581133 19276 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260129 02:58:36.582723 19276 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
W20260129 02:58:37.125844 19942 debug-util.cc:398] Leaking SignalData structure 0x7b0800095f80 after lost signal to thread 19279
W20260129 02:58:37.126854 19942 debug-util.cc:398] Leaking SignalData structure 0x7b08001cbbe0 after lost signal to thread 19853
W20260129 02:58:37.127754 19942 debug-util.cc:398] Leaking SignalData structure 0x7b08000bb040 after lost signal to thread 19945
W20260129 02:58:37.128585 19942 debug-util.cc:398] Leaking SignalData structure 0x7b08000a5060 after lost signal to thread 20020
W20260129 02:58:37.129308 19942 debug-util.cc:398] Leaking SignalData structure 0x7b080014cfe0 after lost signal to thread 20094
I20260129 02:58:37.690063 19276 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260129 02:58:37.698913 19276 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260129 02:58:37.699483 19276 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
W20260129 02:58:38.318782 20091 debug-util.cc:398] Leaking SignalData structure 0x7b080016dbe0 after lost signal to thread 19279
W20260129 02:58:38.319581 20091 debug-util.cc:398] Leaking SignalData structure 0x7b080016d4e0 after lost signal to thread 19853
W20260129 02:58:38.320462 20091 debug-util.cc:398] Leaking SignalData structure 0x7b08001cd9e0 after lost signal to thread 19945
W20260129 02:58:38.321228 20091 debug-util.cc:398] Leaking SignalData structure 0x7b08000c9ae0 after lost signal to thread 20020
W20260129 02:58:38.321985 20091 debug-util.cc:398] Leaking SignalData structure 0x7b08000c3c20 after lost signal to thread 20094
I20260129 02:58:38.700563 19276 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260129 02:58:38.709857 19276 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260129 02:58:38.710440 19276 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20260129 02:58:39.711484 19276 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260129 02:58:39.717821 19276 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260129 02:58:39.718219 19276 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20260129 02:58:40.746174 19276 auto_leader_rebalancer-test.cc:388] The leader distribution is
I20260129 02:58:40.746673 19276 tablet_server.cc:178] TabletServer@127.18.211.1:0 shutting down...
I20260129 02:58:40.772920 19276 ts_tablet_manager.cc:1507] Shutting down tablet manager...
I20260129 02:58:40.773725 19276 tablet_replica.cc:333] T 1bd5a7f0c64f4ef6bc2bb544cd69843c P eafca908701c410ab9488970fcb3d74e: stopping tablet replica
I20260129 02:58:40.774739 19276 raft_consensus.cc:2243] T 1bd5a7f0c64f4ef6bc2bb544cd69843c P eafca908701c410ab9488970fcb3d74e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260129 02:58:40.775429 19276 raft_consensus.cc:2272] T 1bd5a7f0c64f4ef6bc2bb544cd69843c P eafca908701c410ab9488970fcb3d74e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260129 02:58:40.777493 19276 tablet_replica.cc:333] T 620c6d80e1964c51a628ffb43f645798 P eafca908701c410ab9488970fcb3d74e: stopping tablet replica
I20260129 02:58:40.778234 19276 raft_consensus.cc:2243] T 620c6d80e1964c51a628ffb43f645798 P eafca908701c410ab9488970fcb3d74e [term 2 LEADER]: Raft consensus shutting down.
I20260129 02:58:40.779211 19276 raft_consensus.cc:2272] T 620c6d80e1964c51a628ffb43f645798 P eafca908701c410ab9488970fcb3d74e [term 2 FOLLOWER]: Raft consensus is shut down!
I20260129 02:58:40.781243 19276 tablet_replica.cc:333] T 1dc801ccecc44ed79d85b0522e49d3e5 P eafca908701c410ab9488970fcb3d74e: stopping tablet replica
I20260129 02:58:40.781708 19276 raft_consensus.cc:2243] T 1dc801ccecc44ed79d85b0522e49d3e5 P eafca908701c410ab9488970fcb3d74e [term 3 FOLLOWER]: Raft consensus shutting down.
I20260129 02:58:40.782063 19276 raft_consensus.cc:2272] T 1dc801ccecc44ed79d85b0522e49d3e5 P eafca908701c410ab9488970fcb3d74e [term 3 FOLLOWER]: Raft consensus is shut down!
I20260129 02:58:40.783692 19276 tablet_replica.cc:333] T 4562a703e5a44a1585b332b851bfd6f2 P eafca908701c410ab9488970fcb3d74e: stopping tablet replica
I20260129 02:58:40.784224 19276 raft_consensus.cc:2243] T 4562a703e5a44a1585b332b851bfd6f2 P eafca908701c410ab9488970fcb3d74e [term 5 FOLLOWER]: Raft consensus shutting down.
I20260129 02:58:40.784634 19276 raft_consensus.cc:2272] T 4562a703e5a44a1585b332b851bfd6f2 P eafca908701c410ab9488970fcb3d74e [term 5 FOLLOWER]: Raft consensus is shut down!
I20260129 02:58:40.786332 19276 tablet_replica.cc:333] T 7d91c3094df94b47aa9a512d13490daa P eafca908701c410ab9488970fcb3d74e: stopping tablet replica
I20260129 02:58:40.786746 19276 raft_consensus.cc:2243] T 7d91c3094df94b47aa9a512d13490daa P eafca908701c410ab9488970fcb3d74e [term 2 LEADER]: Raft consensus shutting down.
I20260129 02:58:40.787451 19276 raft_consensus.cc:2272] T 7d91c3094df94b47aa9a512d13490daa P eafca908701c410ab9488970fcb3d74e [term 2 FOLLOWER]: Raft consensus is shut down!
I20260129 02:58:40.789023 19276 tablet_replica.cc:333] T 79355f4f5d7942b6ac6f420256fd35b7 P eafca908701c410ab9488970fcb3d74e: stopping tablet replica
I20260129 02:58:40.789435 19276 raft_consensus.cc:2243] T 79355f4f5d7942b6ac6f420256fd35b7 P eafca908701c410ab9488970fcb3d74e [term 3 FOLLOWER]: Raft consensus shutting down.
I20260129 02:58:40.789780 19276 raft_consensus.cc:2272] T 79355f4f5d7942b6ac6f420256fd35b7 P eafca908701c410ab9488970fcb3d74e [term 3 FOLLOWER]: Raft consensus is shut down!
I20260129 02:58:40.791497 19276 tablet_replica.cc:333] T d5edd0a419ee46c9a6a223f8056afb4c P eafca908701c410ab9488970fcb3d74e: stopping tablet replica
I20260129 02:58:40.792057 19276 raft_consensus.cc:2243] T d5edd0a419ee46c9a6a223f8056afb4c P eafca908701c410ab9488970fcb3d74e [term 3 LEADER]: Raft consensus shutting down.
I20260129 02:58:40.792793 19276 raft_consensus.cc:2272] T d5edd0a419ee46c9a6a223f8056afb4c P eafca908701c410ab9488970fcb3d74e [term 3 FOLLOWER]: Raft consensus is shut down!
I20260129 02:58:40.794486 19276 tablet_replica.cc:333] T 47ad34405a6f4d508c9613c07a572cc3 P eafca908701c410ab9488970fcb3d74e: stopping tablet replica
I20260129 02:58:40.794991 19276 raft_consensus.cc:2243] T 47ad34405a6f4d508c9613c07a572cc3 P eafca908701c410ab9488970fcb3d74e [term 1 LEADER]: Raft consensus shutting down.
I20260129 02:58:40.795703 19276 raft_consensus.cc:2272] T 47ad34405a6f4d508c9613c07a572cc3 P eafca908701c410ab9488970fcb3d74e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260129 02:58:40.797354 19276 tablet_replica.cc:333] T 901f619b02f843b4a2747de956182f12 P eafca908701c410ab9488970fcb3d74e: stopping tablet replica
I20260129 02:58:40.797780 19276 raft_consensus.cc:2243] T 901f619b02f843b4a2747de956182f12 P eafca908701c410ab9488970fcb3d74e [term 4 FOLLOWER]: Raft consensus shutting down.
I20260129 02:58:40.798096 19276 raft_consensus.cc:2272] T 901f619b02f843b4a2747de956182f12 P eafca908701c410ab9488970fcb3d74e [term 4 FOLLOWER]: Raft consensus is shut down!
I20260129 02:58:40.799660 19276 tablet_replica.cc:333] T 5d11df549a83429a87d3e7a62fabaa21 P eafca908701c410ab9488970fcb3d74e: stopping tablet replica
I20260129 02:58:40.800096 19276 raft_consensus.cc:2243] T 5d11df549a83429a87d3e7a62fabaa21 P eafca908701c410ab9488970fcb3d74e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260129 02:58:40.800429 19276 raft_consensus.cc:2272] T 5d11df549a83429a87d3e7a62fabaa21 P eafca908701c410ab9488970fcb3d74e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260129 02:58:40.846181 19276 tablet_server.cc:195] TabletServer@127.18.211.1:0 shutdown complete.
I20260129 02:58:40.864094 19276 tablet_server.cc:178] TabletServer@127.18.211.2:0 shutting down...
I20260129 02:58:40.889142 19276 ts_tablet_manager.cc:1507] Shutting down tablet manager...
I20260129 02:58:40.889758 19276 tablet_replica.cc:333] T 4562a703e5a44a1585b332b851bfd6f2 P 544851f9b7954fd2962675f957e8836a: stopping tablet replica
I20260129 02:58:40.890447 19276 raft_consensus.cc:2243] T 4562a703e5a44a1585b332b851bfd6f2 P 544851f9b7954fd2962675f957e8836a [term 5 LEADER]: Raft consensus shutting down.
I20260129 02:58:40.891332 19276 raft_consensus.cc:2272] T 4562a703e5a44a1585b332b851bfd6f2 P 544851f9b7954fd2962675f957e8836a [term 5 FOLLOWER]: Raft consensus is shut down!
I20260129 02:58:40.893235 19276 tablet_replica.cc:333] T 79355f4f5d7942b6ac6f420256fd35b7 P 544851f9b7954fd2962675f957e8836a: stopping tablet replica
I20260129 02:58:40.893780 19276 raft_consensus.cc:2243] T 79355f4f5d7942b6ac6f420256fd35b7 P 544851f9b7954fd2962675f957e8836a [term 3 LEADER]: Raft consensus shutting down.
I20260129 02:58:40.894472 19276 raft_consensus.cc:2272] T 79355f4f5d7942b6ac6f420256fd35b7 P 544851f9b7954fd2962675f957e8836a [term 3 FOLLOWER]: Raft consensus is shut down!
I20260129 02:58:40.896163 19276 tablet_replica.cc:333] T 1bd5a7f0c64f4ef6bc2bb544cd69843c P 544851f9b7954fd2962675f957e8836a: stopping tablet replica
I20260129 02:58:40.896656 19276 raft_consensus.cc:2243] T 1bd5a7f0c64f4ef6bc2bb544cd69843c P 544851f9b7954fd2962675f957e8836a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260129 02:58:40.897044 19276 raft_consensus.cc:2272] T 1bd5a7f0c64f4ef6bc2bb544cd69843c P 544851f9b7954fd2962675f957e8836a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260129 02:58:40.898734 19276 tablet_replica.cc:333] T 620c6d80e1964c51a628ffb43f645798 P 544851f9b7954fd2962675f957e8836a: stopping tablet replica
I20260129 02:58:40.899235 19276 raft_consensus.cc:2243] T 620c6d80e1964c51a628ffb43f645798 P 544851f9b7954fd2962675f957e8836a [term 2 FOLLOWER]: Raft consensus shutting down.
I20260129 02:58:40.899657 19276 raft_consensus.cc:2272] T 620c6d80e1964c51a628ffb43f645798 P 544851f9b7954fd2962675f957e8836a [term 2 FOLLOWER]: Raft consensus is shut down!
I20260129 02:58:40.901253 19276 tablet_replica.cc:333] T 1dc801ccecc44ed79d85b0522e49d3e5 P 544851f9b7954fd2962675f957e8836a: stopping tablet replica
I20260129 02:58:40.901859 19276 raft_consensus.cc:2243] T 1dc801ccecc44ed79d85b0522e49d3e5 P 544851f9b7954fd2962675f957e8836a [term 3 FOLLOWER]: Raft consensus shutting down.
I20260129 02:58:40.902320 19276 raft_consensus.cc:2272] T 1dc801ccecc44ed79d85b0522e49d3e5 P 544851f9b7954fd2962675f957e8836a [term 3 FOLLOWER]: Raft consensus is shut down!
I20260129 02:58:40.904376 19276 tablet_replica.cc:333] T 5d11df549a83429a87d3e7a62fabaa21 P 544851f9b7954fd2962675f957e8836a: stopping tablet replica
I20260129 02:58:40.904879 19276 raft_consensus.cc:2243] T 5d11df549a83429a87d3e7a62fabaa21 P 544851f9b7954fd2962675f957e8836a [term 1 LEADER]: Raft consensus shutting down.
I20260129 02:58:40.905628 19276 raft_consensus.cc:2272] T 5d11df549a83429a87d3e7a62fabaa21 P 544851f9b7954fd2962675f957e8836a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260129 02:58:40.907259 19276 tablet_replica.cc:333] T 901f619b02f843b4a2747de956182f12 P 544851f9b7954fd2962675f957e8836a: stopping tablet replica
I20260129 02:58:40.907754 19276 raft_consensus.cc:2243] T 901f619b02f843b4a2747de956182f12 P 544851f9b7954fd2962675f957e8836a [term 4 FOLLOWER]: Raft consensus shutting down.
I20260129 02:58:40.908188 19276 raft_consensus.cc:2272] T 901f619b02f843b4a2747de956182f12 P 544851f9b7954fd2962675f957e8836a [term 4 FOLLOWER]: Raft consensus is shut down!
I20260129 02:58:40.909745 19276 tablet_replica.cc:333] T 47ad34405a6f4d508c9613c07a572cc3 P 544851f9b7954fd2962675f957e8836a: stopping tablet replica
I20260129 02:58:40.910183 19276 raft_consensus.cc:2243] T 47ad34405a6f4d508c9613c07a572cc3 P 544851f9b7954fd2962675f957e8836a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260129 02:58:40.910550 19276 raft_consensus.cc:2272] T 47ad34405a6f4d508c9613c07a572cc3 P 544851f9b7954fd2962675f957e8836a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260129 02:58:40.912246 19276 tablet_replica.cc:333] T d5edd0a419ee46c9a6a223f8056afb4c P 544851f9b7954fd2962675f957e8836a: stopping tablet replica
I20260129 02:58:40.912647 19276 raft_consensus.cc:2243] T d5edd0a419ee46c9a6a223f8056afb4c P 544851f9b7954fd2962675f957e8836a [term 3 FOLLOWER]: Raft consensus shutting down.
I20260129 02:58:40.913009 19276 raft_consensus.cc:2272] T d5edd0a419ee46c9a6a223f8056afb4c P 544851f9b7954fd2962675f957e8836a [term 3 FOLLOWER]: Raft consensus is shut down!
I20260129 02:58:40.914583 19276 tablet_replica.cc:333] T 7d91c3094df94b47aa9a512d13490daa P 544851f9b7954fd2962675f957e8836a: stopping tablet replica
I20260129 02:58:40.914956 19276 raft_consensus.cc:2243] T 7d91c3094df94b47aa9a512d13490daa P 544851f9b7954fd2962675f957e8836a [term 2 FOLLOWER]: Raft consensus shutting down.
I20260129 02:58:40.915298 19276 raft_consensus.cc:2272] T 7d91c3094df94b47aa9a512d13490daa P 544851f9b7954fd2962675f957e8836a [term 2 FOLLOWER]: Raft consensus is shut down!
I20260129 02:58:40.944141 19276 tablet_server.cc:195] TabletServer@127.18.211.2:0 shutdown complete.
I20260129 02:58:40.963557 19276 tablet_server.cc:178] TabletServer@127.18.211.3:0 shutting down...
I20260129 02:58:40.987251 19276 ts_tablet_manager.cc:1507] Shutting down tablet manager...
I20260129 02:58:40.987903 19276 tablet_replica.cc:333] T d5edd0a419ee46c9a6a223f8056afb4c P a634c60a5ca74b26b0d1f679d0166d0c: stopping tablet replica
I20260129 02:58:40.988408 19276 raft_consensus.cc:2243] T d5edd0a419ee46c9a6a223f8056afb4c P a634c60a5ca74b26b0d1f679d0166d0c [term 3 FOLLOWER]: Raft consensus shutting down.
I20260129 02:58:40.988822 19276 raft_consensus.cc:2272] T d5edd0a419ee46c9a6a223f8056afb4c P a634c60a5ca74b26b0d1f679d0166d0c [term 3 FOLLOWER]: Raft consensus is shut down!
I20260129 02:58:40.990558 19276 tablet_replica.cc:333] T 79355f4f5d7942b6ac6f420256fd35b7 P a634c60a5ca74b26b0d1f679d0166d0c: stopping tablet replica
I20260129 02:58:40.991020 19276 raft_consensus.cc:2243] T 79355f4f5d7942b6ac6f420256fd35b7 P a634c60a5ca74b26b0d1f679d0166d0c [term 3 FOLLOWER]: Raft consensus shutting down.
I20260129 02:58:40.991374 19276 raft_consensus.cc:2272] T 79355f4f5d7942b6ac6f420256fd35b7 P a634c60a5ca74b26b0d1f679d0166d0c [term 3 FOLLOWER]: Raft consensus is shut down!
I20260129 02:58:40.993153 19276 tablet_replica.cc:333] T 5d11df549a83429a87d3e7a62fabaa21 P a634c60a5ca74b26b0d1f679d0166d0c: stopping tablet replica
I20260129 02:58:40.993620 19276 raft_consensus.cc:2243] T 5d11df549a83429a87d3e7a62fabaa21 P a634c60a5ca74b26b0d1f679d0166d0c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260129 02:58:40.993989 19276 raft_consensus.cc:2272] T 5d11df549a83429a87d3e7a62fabaa21 P a634c60a5ca74b26b0d1f679d0166d0c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260129 02:58:40.995595 19276 tablet_replica.cc:333] T 47ad34405a6f4d508c9613c07a572cc3 P a634c60a5ca74b26b0d1f679d0166d0c: stopping tablet replica
I20260129 02:58:40.996068 19276 raft_consensus.cc:2243] T 47ad34405a6f4d508c9613c07a572cc3 P a634c60a5ca74b26b0d1f679d0166d0c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260129 02:58:40.996387 19276 raft_consensus.cc:2272] T 47ad34405a6f4d508c9613c07a572cc3 P a634c60a5ca74b26b0d1f679d0166d0c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260129 02:58:40.997807 19276 tablet_replica.cc:333] T 901f619b02f843b4a2747de956182f12 P a634c60a5ca74b26b0d1f679d0166d0c: stopping tablet replica
I20260129 02:58:40.998242 19276 raft_consensus.cc:2243] T 901f619b02f843b4a2747de956182f12 P a634c60a5ca74b26b0d1f679d0166d0c [term 4 LEADER]: Raft consensus shutting down.
I20260129 02:58:40.999267 19276 raft_consensus.cc:2272] T 901f619b02f843b4a2747de956182f12 P a634c60a5ca74b26b0d1f679d0166d0c [term 4 FOLLOWER]: Raft consensus is shut down!
I20260129 02:58:41.001289 19276 tablet_replica.cc:333] T 1bd5a7f0c64f4ef6bc2bb544cd69843c P a634c60a5ca74b26b0d1f679d0166d0c: stopping tablet replica
I20260129 02:58:41.001802 19276 raft_consensus.cc:2243] T 1bd5a7f0c64f4ef6bc2bb544cd69843c P a634c60a5ca74b26b0d1f679d0166d0c [term 1 LEADER]: Raft consensus shutting down.
W20260129 02:58:41.002179 20033 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.18.211.1:45001: connect: Connection refused (error 111) [suppressed 2 similar messages]
I20260129 02:58:41.002619 19276 raft_consensus.cc:2272] T 1bd5a7f0c64f4ef6bc2bb544cd69843c P a634c60a5ca74b26b0d1f679d0166d0c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260129 02:58:41.004695 19276 tablet_replica.cc:333] T 1dc801ccecc44ed79d85b0522e49d3e5 P a634c60a5ca74b26b0d1f679d0166d0c: stopping tablet replica
I20260129 02:58:41.005225 19276 raft_consensus.cc:2243] T 1dc801ccecc44ed79d85b0522e49d3e5 P a634c60a5ca74b26b0d1f679d0166d0c [term 3 LEADER]: Raft consensus shutting down.
I20260129 02:58:41.006006 19276 raft_consensus.cc:2272] T 1dc801ccecc44ed79d85b0522e49d3e5 P a634c60a5ca74b26b0d1f679d0166d0c [term 3 FOLLOWER]: Raft consensus is shut down!
I20260129 02:58:41.007723 19276 tablet_replica.cc:333] T 620c6d80e1964c51a628ffb43f645798 P a634c60a5ca74b26b0d1f679d0166d0c: stopping tablet replica
I20260129 02:58:41.008229 19276 raft_consensus.cc:2243] T 620c6d80e1964c51a628ffb43f645798 P a634c60a5ca74b26b0d1f679d0166d0c [term 2 FOLLOWER]: Raft consensus shutting down.
I20260129 02:58:41.008714 19276 raft_consensus.cc:2272] T 620c6d80e1964c51a628ffb43f645798 P a634c60a5ca74b26b0d1f679d0166d0c [term 2 FOLLOWER]: Raft consensus is shut down!
I20260129 02:58:41.010572 19276 tablet_replica.cc:333] T 4562a703e5a44a1585b332b851bfd6f2 P a634c60a5ca74b26b0d1f679d0166d0c: stopping tablet replica
I20260129 02:58:41.011023 19276 raft_consensus.cc:2243] T 4562a703e5a44a1585b332b851bfd6f2 P a634c60a5ca74b26b0d1f679d0166d0c [term 5 FOLLOWER]: Raft consensus shutting down.
I20260129 02:58:41.011411 19276 raft_consensus.cc:2272] T 4562a703e5a44a1585b332b851bfd6f2 P a634c60a5ca74b26b0d1f679d0166d0c [term 5 FOLLOWER]: Raft consensus is shut down!
I20260129 02:58:41.013134 19276 tablet_replica.cc:333] T 7d91c3094df94b47aa9a512d13490daa P a634c60a5ca74b26b0d1f679d0166d0c: stopping tablet replica
I20260129 02:58:41.013590 19276 raft_consensus.cc:2243] T 7d91c3094df94b47aa9a512d13490daa P a634c60a5ca74b26b0d1f679d0166d0c [term 2 FOLLOWER]: Raft consensus shutting down.
I20260129 02:58:41.013999 19276 raft_consensus.cc:2272] T 7d91c3094df94b47aa9a512d13490daa P a634c60a5ca74b26b0d1f679d0166d0c [term 2 FOLLOWER]: Raft consensus is shut down!
I20260129 02:58:41.059829 19276 tablet_server.cc:195] TabletServer@127.18.211.3:0 shutdown complete.
I20260129 02:58:41.079351 19276 master.cc:561] Master@127.18.211.62:37849 shutting down...
I20260129 02:58:41.105363 19276 raft_consensus.cc:2243] T 00000000000000000000000000000000 P 2dea75bcbf3c4f95b35d28ec2ed08a2c [term 1 LEADER]: Raft consensus shutting down.
I20260129 02:58:41.105981 19276 raft_consensus.cc:2272] T 00000000000000000000000000000000 P 2dea75bcbf3c4f95b35d28ec2ed08a2c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260129 02:58:41.106307 19276 tablet_replica.cc:333] T 00000000000000000000000000000000 P 2dea75bcbf3c4f95b35d28ec2ed08a2c: stopping tablet replica
I20260129 02:58:41.143647 19276 master.cc:583] Master@127.18.211.62:37849 shutdown complete.
544851f9b7954fd2962675f957e8836a 4
a634c60a5ca74b26b0d1f679d0166d0c 1
eafca908701c410ab9488970fcb3d74e 5
544851f9b7954fd2962675f957e8836a 4
a634c60a5ca74b26b0d1f679d0166d0c 3
eafca908701c410ab9488970fcb3d74e 3
544851f9b7954fd2962675f957e8836a 1
a634c60a5ca74b26b0d1f679d0166d0c 1
eafca908701c410ab9488970fcb3d74e 8
544851f9b7954fd2962675f957e8836a 3
a634c60a5ca74b26b0d1f679d0166d0c 3
eafca908701c410ab9488970fcb3d74e 4
[ OK ] LeaderRebalancerTest.FunctionalTestForNotDivided (29302 ms)
[ RUN ] LeaderRebalancerTest.AddTserver
I20260129 02:58:41.184921 19276 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.18.211.62:34285
I20260129 02:58:41.186074 19276 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260129 02:58:41.191943 20314 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260129 02:58:41.192008 20313 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260129 02:58:41.192931 20316 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260129 02:58:41.193600 19276 server_base.cc:1060] running on GCE node
I20260129 02:58:41.194550 19276 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260129 02:58:41.194736 19276 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260129 02:58:41.194851 19276 hybrid_clock.cc:648] HybridClock initialized: now 1769655521194840 us; error 0 us; skew 500 ppm
I20260129 02:58:41.195338 19276 server_base.cc:860] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260129 02:58:41.197644 19276 webserver.cc:492] Webserver started at http://127.18.211.62:44479/ using document root <none> and password file <none>
I20260129 02:58:41.198067 19276 fs_manager.cc:362] Metadata directory not provided
I20260129 02:58:41.198223 19276 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260129 02:58:41.198434 19276 server_base.cc:908] This appears to be a new deployment of Kudu; creating new FS layout
I20260129 02:58:41.199544 19276 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskrTP_Jo/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1769655460890924-19276-0/minicluster-data/master-0-root/instance:
uuid: "a91e6696d0a44ee19bcf2489fd709c38"
format_stamp: "Formatted at 2026-01-29 02:58:41 on dist-test-slave-vngr"
I20260129 02:58:41.203856 19276 fs_manager.cc:696] Time spent creating directory manager: real 0.004s user 0.001s sys 0.002s
I20260129 02:58:41.207242 20321 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260129 02:58:41.208030 19276 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260129 02:58:41.208320 19276 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskrTP_Jo/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1769655460890924-19276-0/minicluster-data/master-0-root
uuid: "a91e6696d0a44ee19bcf2489fd709c38"
format_stamp: "Formatted at 2026-01-29 02:58:41 on dist-test-slave-vngr"
I20260129 02:58:41.208593 19276 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrTP_Jo/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1769655460890924-19276-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-taskrTP_Jo/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1769655460890924-19276-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-taskrTP_Jo/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1769655460890924-19276-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260129 02:58:41.222261 19276 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260129 02:58:41.223529 19276 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260129 02:58:41.263767 19276 rpc_server.cc:307] RPC server started. Bound to: 127.18.211.62:34285
I20260129 02:58:41.263950 20372 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.18.211.62:34285 every 8 connection(s)
I20260129 02:58:41.267694 20373 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260129 02:58:41.277424 20373 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a91e6696d0a44ee19bcf2489fd709c38: Bootstrap starting.
I20260129 02:58:41.282209 20373 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a91e6696d0a44ee19bcf2489fd709c38: Neither blocks nor log segments found. Creating new log.
I20260129 02:58:41.286769 20373 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a91e6696d0a44ee19bcf2489fd709c38: No bootstrap required, opened a new log
I20260129 02:58:41.288935 20373 raft_consensus.cc:359] T 00000000000000000000000000000000 P a91e6696d0a44ee19bcf2489fd709c38 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a91e6696d0a44ee19bcf2489fd709c38" member_type: VOTER }
I20260129 02:58:41.289342 20373 raft_consensus.cc:385] T 00000000000000000000000000000000 P a91e6696d0a44ee19bcf2489fd709c38 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260129 02:58:41.289598 20373 raft_consensus.cc:740] T 00000000000000000000000000000000 P a91e6696d0a44ee19bcf2489fd709c38 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a91e6696d0a44ee19bcf2489fd709c38, State: Initialized, Role: FOLLOWER
I20260129 02:58:41.290187 20373 consensus_queue.cc:260] T 00000000000000000000000000000000 P a91e6696d0a44ee19bcf2489fd709c38 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a91e6696d0a44ee19bcf2489fd709c38" member_type: VOTER }
I20260129 02:58:41.290630 20373 raft_consensus.cc:399] T 00000000000000000000000000000000 P a91e6696d0a44ee19bcf2489fd709c38 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260129 02:58:41.290865 20373 raft_consensus.cc:493] T 00000000000000000000000000000000 P a91e6696d0a44ee19bcf2489fd709c38 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260129 02:58:41.291137 20373 raft_consensus.cc:3060] T 00000000000000000000000000000000 P a91e6696d0a44ee19bcf2489fd709c38 [term 0 FOLLOWER]: Advancing to term 1
I20260129 02:58:41.295281 20373 raft_consensus.cc:515] T 00000000000000000000000000000000 P a91e6696d0a44ee19bcf2489fd709c38 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a91e6696d0a44ee19bcf2489fd709c38" member_type: VOTER }
I20260129 02:58:41.296085 20373 leader_election.cc:304] T 00000000000000000000000000000000 P a91e6696d0a44ee19bcf2489fd709c38 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a91e6696d0a44ee19bcf2489fd709c38; no voters:
I20260129 02:58:41.308090 20373 leader_election.cc:290] T 00000000000000000000000000000000 P a91e6696d0a44ee19bcf2489fd709c38 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260129 02:58:41.308471 20377 raft_consensus.cc:2804] T 00000000000000000000000000000000 P a91e6696d0a44ee19bcf2489fd709c38 [term 1 FOLLOWER]: Leader election won for term 1
I20260129 02:58:41.310390 20377 raft_consensus.cc:697] T 00000000000000000000000000000000 P a91e6696d0a44ee19bcf2489fd709c38 [term 1 LEADER]: Becoming Leader. State: Replica: a91e6696d0a44ee19bcf2489fd709c38, State: Running, Role: LEADER
I20260129 02:58:41.311352 20377 consensus_queue.cc:237] T 00000000000000000000000000000000 P a91e6696d0a44ee19bcf2489fd709c38 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a91e6696d0a44ee19bcf2489fd709c38" member_type: VOTER }
I20260129 02:58:41.312963 20373 sys_catalog.cc:565] T 00000000000000000000000000000000 P a91e6696d0a44ee19bcf2489fd709c38 [sys.catalog]: configured and running, proceeding with master startup.
I20260129 02:58:41.319916 20378 sys_catalog.cc:455] T 00000000000000000000000000000000 P a91e6696d0a44ee19bcf2489fd709c38 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a91e6696d0a44ee19bcf2489fd709c38" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a91e6696d0a44ee19bcf2489fd709c38" member_type: VOTER } }
I20260129 02:58:41.320034 20379 sys_catalog.cc:455] T 00000000000000000000000000000000 P a91e6696d0a44ee19bcf2489fd709c38 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a91e6696d0a44ee19bcf2489fd709c38. Latest consensus state: current_term: 1 leader_uuid: "a91e6696d0a44ee19bcf2489fd709c38" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a91e6696d0a44ee19bcf2489fd709c38" member_type: VOTER } }
I20260129 02:58:41.320812 20378 sys_catalog.cc:458] T 00000000000000000000000000000000 P a91e6696d0a44ee19bcf2489fd709c38 [sys.catalog]: This master's current role is: LEADER
I20260129 02:58:41.320900 20379 sys_catalog.cc:458] T 00000000000000000000000000000000 P a91e6696d0a44ee19bcf2489fd709c38 [sys.catalog]: This master's current role is: LEADER
I20260129 02:58:41.326749 20383 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20260129 02:58:41.331828 20383 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20260129 02:58:41.335361 19276 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20260129 02:58:41.341254 20383 catalog_manager.cc:1357] Generated new cluster ID: c265bc99dcf24f61856b5014ff09db26
I20260129 02:58:41.341539 20383 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20260129 02:58:41.365598 20383 catalog_manager.cc:1380] Generated new certificate authority record
I20260129 02:58:41.366864 20383 catalog_manager.cc:1514] Loading token signing keys...
I20260129 02:58:41.388324 20383 catalog_manager.cc:6027] T 00000000000000000000000000000000 P a91e6696d0a44ee19bcf2489fd709c38: Generated new TSK 0
I20260129 02:58:41.388980 20383 catalog_manager.cc:1524] Initializing in-progress tserver states...
I20260129 02:58:41.402580 19276 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260129 02:58:41.409446 20395 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260129 02:58:41.410331 20396 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260129 02:58:41.412279 19276 server_base.cc:1060] running on GCE node
W20260129 02:58:41.412849 20398 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260129 02:58:41.413862 19276 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260129 02:58:41.414103 19276 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260129 02:58:41.414357 19276 hybrid_clock.cc:648] HybridClock initialized: now 1769655521414338 us; error 0 us; skew 500 ppm
I20260129 02:58:41.414929 19276 server_base.cc:860] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260129 02:58:41.417498 19276 webserver.cc:492] Webserver started at http://127.18.211.1:45713/ using document root <none> and password file <none>
I20260129 02:58:41.418028 19276 fs_manager.cc:362] Metadata directory not provided
I20260129 02:58:41.418241 19276 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260129 02:58:41.418540 19276 server_base.cc:908] This appears to be a new deployment of Kudu; creating new FS layout
I20260129 02:58:41.419978 19276 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskrTP_Jo/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1769655460890924-19276-0/minicluster-data/ts-0-root/instance:
uuid: "197b880fa39d432a9b472845876bfaf0"
format_stamp: "Formatted at 2026-01-29 02:58:41 on dist-test-slave-vngr"
I20260129 02:58:41.424679 19276 fs_manager.cc:696] Time spent creating directory manager: real 0.004s user 0.000s sys 0.004s
I20260129 02:58:41.428241 20403 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260129 02:58:41.429203 19276 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260129 02:58:41.429461 19276 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskrTP_Jo/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1769655460890924-19276-0/minicluster-data/ts-0-root
uuid: "197b880fa39d432a9b472845876bfaf0"
format_stamp: "Formatted at 2026-01-29 02:58:41 on dist-test-slave-vngr"
I20260129 02:58:41.429721 19276 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrTP_Jo/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1769655460890924-19276-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-taskrTP_Jo/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1769655460890924-19276-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-taskrTP_Jo/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1769655460890924-19276-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260129 02:58:41.443185 19276 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260129 02:58:41.444509 19276 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260129 02:58:41.445972 19276 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260129 02:58:41.448427 19276 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260129 02:58:41.448686 19276 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260129 02:58:41.448937 19276 ts_tablet_manager.cc:616] Registered 0 tablets
I20260129 02:58:41.449119 19276 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260129 02:58:41.489929 19276 rpc_server.cc:307] RPC server started. Bound to: 127.18.211.1:42027
I20260129 02:58:41.490044 20465 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.18.211.1:42027 every 8 connection(s)
I20260129 02:58:41.494737 19276 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260129 02:58:41.502043 20470 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260129 02:58:41.502960 20471 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260129 02:58:41.506253 20473 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260129 02:58:41.506289 19276 server_base.cc:1060] running on GCE node
I20260129 02:58:41.508359 19276 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260129 02:58:41.508668 19276 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260129 02:58:41.508859 19276 hybrid_clock.cc:648] HybridClock initialized: now 1769655521508844 us; error 0 us; skew 500 ppm
I20260129 02:58:41.509462 20466 heartbeater.cc:344] Connected to a master server at 127.18.211.62:34285
I20260129 02:58:41.509562 19276 server_base.cc:860] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260129 02:58:41.509995 20466 heartbeater.cc:461] Registering TS with master...
I20260129 02:58:41.511020 20466 heartbeater.cc:507] Master 127.18.211.62:34285 requested a full tablet report, sending...
I20260129 02:58:41.512501 19276 webserver.cc:492] Webserver started at http://127.18.211.2:33085/ using document root <none> and password file <none>
I20260129 02:58:41.513084 19276 fs_manager.cc:362] Metadata directory not provided
I20260129 02:58:41.513276 19276 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260129 02:58:41.513393 20338 ts_manager.cc:194] Registered new tserver with Master: 197b880fa39d432a9b472845876bfaf0 (127.18.211.1:42027)
I20260129 02:58:41.513537 19276 server_base.cc:908] This appears to be a new deployment of Kudu; creating new FS layout
I20260129 02:58:41.514931 19276 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskrTP_Jo/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1769655460890924-19276-0/minicluster-data/ts-1-root/instance:
uuid: "974b802141bd4f13beffef2aa03f88f9"
format_stamp: "Formatted at 2026-01-29 02:58:41 on dist-test-slave-vngr"
I20260129 02:58:41.515375 20338 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:55348
I20260129 02:58:41.519992 19276 fs_manager.cc:696] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260129 02:58:41.523255 20478 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260129 02:58:41.524053 19276 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260129 02:58:41.524331 19276 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskrTP_Jo/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1769655460890924-19276-0/minicluster-data/ts-1-root
uuid: "974b802141bd4f13beffef2aa03f88f9"
format_stamp: "Formatted at 2026-01-29 02:58:41 on dist-test-slave-vngr"
I20260129 02:58:41.524617 19276 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrTP_Jo/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1769655460890924-19276-0/minicluster-data/ts-1-root
metadata directory: /tmp/dist-test-taskrTP_Jo/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1769655460890924-19276-0/minicluster-data/ts-1-root
1 data directories: /tmp/dist-test-taskrTP_Jo/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1769655460890924-19276-0/minicluster-data/ts-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260129 02:58:41.537539 19276 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260129 02:58:41.538709 19276 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260129 02:58:41.540177 19276 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260129 02:58:41.542306 19276 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260129 02:58:41.542526 19276 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260129 02:58:41.542760 19276 ts_tablet_manager.cc:616] Registered 0 tablets
I20260129 02:58:41.542937 19276 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260129 02:58:41.584218 19276 rpc_server.cc:307] RPC server started. Bound to: 127.18.211.2:34517
I20260129 02:58:41.584312 20540 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.18.211.2:34517 every 8 connection(s)
I20260129 02:58:41.589047 19276 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260129 02:58:41.596254 20544 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260129 02:58:41.598484 20545 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260129 02:58:41.599596 19276 server_base.cc:1060] running on GCE node
W20260129 02:58:41.600095 20547 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260129 02:58:41.601028 19276 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260129 02:58:41.601218 19276 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260129 02:58:41.601451 19276 hybrid_clock.cc:648] HybridClock initialized: now 1769655521601433 us; error 0 us; skew 500 ppm
I20260129 02:58:41.601990 19276 server_base.cc:860] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260129 02:58:41.604005 20541 heartbeater.cc:344] Connected to a master server at 127.18.211.62:34285
I20260129 02:58:41.604343 20541 heartbeater.cc:461] Registering TS with master...
I20260129 02:58:41.605046 20541 heartbeater.cc:507] Master 127.18.211.62:34285 requested a full tablet report, sending...
I20260129 02:58:41.606045 19276 webserver.cc:492] Webserver started at http://127.18.211.3:34599/ using document root <none> and password file <none>
I20260129 02:58:41.606592 19276 fs_manager.cc:362] Metadata directory not provided
I20260129 02:58:41.606806 19276 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260129 02:58:41.607074 19276 server_base.cc:908] This appears to be a new deployment of Kudu; creating new FS layout
I20260129 02:58:41.607072 20338 ts_manager.cc:194] Registered new tserver with Master: 974b802141bd4f13beffef2aa03f88f9 (127.18.211.2:34517)
I20260129 02:58:41.608625 19276 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskrTP_Jo/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1769655460890924-19276-0/minicluster-data/ts-2-root/instance:
uuid: "1c50de89a5a845668696dcb28c3badd6"
format_stamp: "Formatted at 2026-01-29 02:58:41 on dist-test-slave-vngr"
I20260129 02:58:41.608888 20338 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:55364
I20260129 02:58:41.613767 19276 fs_manager.cc:696] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260129 02:58:41.617201 20552 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260129 02:58:41.617924 19276 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260129 02:58:41.618213 19276 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskrTP_Jo/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1769655460890924-19276-0/minicluster-data/ts-2-root
uuid: "1c50de89a5a845668696dcb28c3badd6"
format_stamp: "Formatted at 2026-01-29 02:58:41 on dist-test-slave-vngr"
I20260129 02:58:41.618520 19276 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrTP_Jo/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1769655460890924-19276-0/minicluster-data/ts-2-root
metadata directory: /tmp/dist-test-taskrTP_Jo/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1769655460890924-19276-0/minicluster-data/ts-2-root
1 data directories: /tmp/dist-test-taskrTP_Jo/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1769655460890924-19276-0/minicluster-data/ts-2-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260129 02:58:41.654326 19276 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260129 02:58:41.655591 19276 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260129 02:58:41.657063 19276 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260129 02:58:41.659121 19276 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260129 02:58:41.659339 19276 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260129 02:58:41.659575 19276 ts_tablet_manager.cc:616] Registered 0 tablets
I20260129 02:58:41.659744 19276 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260129 02:58:41.698414 19276 rpc_server.cc:307] RPC server started. Bound to: 127.18.211.3:37735
I20260129 02:58:41.698511 20614 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.18.211.3:37735 every 8 connection(s)
I20260129 02:58:41.712144 20615 heartbeater.cc:344] Connected to a master server at 127.18.211.62:34285
I20260129 02:58:41.712522 20615 heartbeater.cc:461] Registering TS with master...
I20260129 02:58:41.713364 20615 heartbeater.cc:507] Master 127.18.211.62:34285 requested a full tablet report, sending...
I20260129 02:58:41.715225 20338 ts_manager.cc:194] Registered new tserver with Master: 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3:37735)
I20260129 02:58:41.715667 19276 internal_mini_cluster.cc:371] 3 TS(s) registered with all masters after 0.014238873s
I20260129 02:58:41.716596 20338 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:55376
I20260129 02:58:41.741149 20338 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55378:
name: "test-workload"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "\004\001\000\313c+\002\004\001\000\226\307V\004\004\001\000a+\202\006\004\001\000,\217\255\010\004\001\000\367\362\330\n\004\001\000\302V\004\r\004\001\000\215\272/\017\004\001\000X\036[\021\004\001\000#\202\206\023\004\001\000\356\345\261\025\004\001\000\271I\335\027\004\001\000\204\255\010\032\004\001\000O\0214\034\004\001\000\032u_\036\004\001\000\345\330\212 \004\001\000\260<\266\"\004\001\000{\240\341$\004\001\000F\004\r\'\004\001\000\021h8)\004\001\000\334\313c+\004\001\000\247/\217-\004\001\000r\223\272/\004\001\000=\367\3451\004\001\000\010[\0214\004\001\000\323\276<6\004\001\000\236\"h8\004\001\000i\206\223:\004\001\0004\352\276<\004\001\000\377M\352>\004\001\000\312\261\025A\004\001\000\225\025AC\004\001\000`ylE\004\001\000+\335\227G\004\001\000\366@\303I\004\001\000\301\244\356K\004\001\000\214\010\032N\004\001\000WlEP\004\001\000\"\320pR\004\001\000\3553\234T\004\001\000\270\227\307V\004\001\000\203\373\362X\004\001\000N_\036[\004\001\000\031\303I]\004\001\000\344&u_\004\001\000\257\212\240a\004\001\000z\356\313c\004\001\000ER\367e\004\001\000\020\266\"h\004\001\000\333\031Nj\004\001\000\246}yl\004\001\000q\341\244n\004\001\000<E\320p\004\001\000\007\251\373r\004\001\000\322\014\'u\004\001\000\235pRw\004\001\000h\324}y\004\001\00038\251{\004\001\000\376\233\324}""\004\001\000\313c+\002\004\001\000\226\307V\004\004\001\000a+\202\006\004\001\000,\217\255\010\004\001\000\367\362\330\n\004\001\000\302V\004\r\004\001\000\215\272/\017\004\001\000X\036[\021\004\001\000#\202\206\023\004\001\000\356\345\261\025\004\001\000\271I\335\027\004\001\000\204\255\010\032\004\001\000O\0214\034\004\001\000\032u_\036\004\001\000\345\330\212 \004\001\000\260<\266\"\004\001\000{\240\341$\004\001\000F\004\r\'\004\001\000\021h8)\004\001\000\334\313c+\004\001\000\247/\217-\004\001\000r\223\272/\004\001\000=\367\3451\004\001\000\010[\0214\004\001\000\323\276<6\004\001\000\236\"h8\004\001\000i\206\223:\004\001\0004\352\276<\004\001\000\377M\352>\004\001\000\312\261\025A\004\001\000\225\025AC\004\001\000`ylE\004\001\000+\335\227G\004\001\000\366@\303I\004\001\000\301\244\356K\004\001\000\214\010\032N\004\001\000WlEP\004\001\000\"\320pR\004\001\000\3553\234T\004\001\000\270\227\307V\004\001\000\203\373\362X\004\001\000N_\036[\004\001\000\031\303I]\004\001\000\344&u_\004\001\000\257\212\240a\004\001\000z\356\313c\004\001\000ER\367e\004\001\000\020\266\"h\004\001\000\333\031Nj\004\001\000\246}yl\004\001\000q\341\244n\004\001\000<E\320p\004\001\000\007\251\373r\004\001\000\322\014\'u\004\001\000\235pRw\004\001\000h\324}y\004\001\00038\251{\004\001\000\376\233\324}"
indirect_data: """"
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260129 02:58:41.747340 20338 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-workload in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260129 02:58:41.963199 20506 tablet_service.cc:1505] Processing CreateTablet for tablet a1a39dcd328e4268ae41b988c66b4a75 (DEFAULT_TABLE table=test-workload [id=de73611ac8e3485a98aecc5add247032]), partition=RANGE (key) PARTITION VALUES < 36398027
I20260129 02:58:41.964907 20506 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a1a39dcd328e4268ae41b988c66b4a75. 1 dirs total, 0 dirs full, 0 dirs failed
I20260129 02:58:41.965143 20579 tablet_service.cc:1505] Processing CreateTablet for tablet e7d83d4ccf684e76acb29a0dc844e1b8 (DEFAULT_TABLE table=test-workload [id=de73611ac8e3485a98aecc5add247032]), partition=RANGE (key) PARTITION 36398027 <= VALUES < 72796054
I20260129 02:58:41.966105 20576 tablet_service.cc:1505] Processing CreateTablet for tablet 3e2d15f8640b407195210c476e09df91 (DEFAULT_TABLE table=test-workload [id=de73611ac8e3485a98aecc5add247032]), partition=RANGE (key) PARTITION 145592108 <= VALUES < 181990135
I20260129 02:58:41.967936 20578 tablet_service.cc:1505] Processing CreateTablet for tablet 4b1876f9a70b49edb3aa74105e66d3b3 (DEFAULT_TABLE table=test-workload [id=de73611ac8e3485a98aecc5add247032]), partition=RANGE (key) PARTITION 72796054 <= VALUES < 109194081
I20260129 02:58:41.963225 20580 tablet_service.cc:1505] Processing CreateTablet for tablet a1a39dcd328e4268ae41b988c66b4a75 (DEFAULT_TABLE table=test-workload [id=de73611ac8e3485a98aecc5add247032]), partition=RANGE (key) PARTITION VALUES < 36398027
I20260129 02:58:41.969380 20575 tablet_service.cc:1505] Processing CreateTablet for tablet a5a05fbcbbcf4916bbc01fd1a5b5e90d (DEFAULT_TABLE table=test-workload [id=de73611ac8e3485a98aecc5add247032]), partition=RANGE (key) PARTITION 181990135 <= VALUES < 218388162
I20260129 02:58:41.970299 20577 tablet_service.cc:1505] Processing CreateTablet for tablet 87d5fe48a44a408aaf03e894a67ecb2f (DEFAULT_TABLE table=test-workload [id=de73611ac8e3485a98aecc5add247032]), partition=RANGE (key) PARTITION 109194081 <= VALUES < 145592108
I20260129 02:58:41.966634 20579 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e7d83d4ccf684e76acb29a0dc844e1b8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260129 02:58:41.972266 20574 tablet_service.cc:1505] Processing CreateTablet for tablet 2e58e9a5a8aa4cecafb464addb4f35ed (DEFAULT_TABLE table=test-workload [id=de73611ac8e3485a98aecc5add247032]), partition=RANGE (key) PARTITION 218388162 <= VALUES < 254786189
I20260129 02:58:41.973575 20574 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2e58e9a5a8aa4cecafb464addb4f35ed. 1 dirs total, 0 dirs full, 0 dirs failed
I20260129 02:58:41.974192 20578 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4b1876f9a70b49edb3aa74105e66d3b3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260129 02:58:41.974890 20576 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3e2d15f8640b407195210c476e09df91. 1 dirs total, 0 dirs full, 0 dirs failed
I20260129 02:58:41.977416 20505 tablet_service.cc:1505] Processing CreateTablet for tablet e7d83d4ccf684e76acb29a0dc844e1b8 (DEFAULT_TABLE table=test-workload [id=de73611ac8e3485a98aecc5add247032]), partition=RANGE (key) PARTITION 36398027 <= VALUES < 72796054
I20260129 02:58:41.978736 20505 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e7d83d4ccf684e76acb29a0dc844e1b8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260129 02:58:41.979537 20580 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a1a39dcd328e4268ae41b988c66b4a75. 1 dirs total, 0 dirs full, 0 dirs failed
I20260129 02:58:41.984259 20577 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 87d5fe48a44a408aaf03e894a67ecb2f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260129 02:58:41.985414 20504 tablet_service.cc:1505] Processing CreateTablet for tablet 4b1876f9a70b49edb3aa74105e66d3b3 (DEFAULT_TABLE table=test-workload [id=de73611ac8e3485a98aecc5add247032]), partition=RANGE (key) PARTITION 72796054 <= VALUES < 109194081
I20260129 02:58:41.986763 20504 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4b1876f9a70b49edb3aa74105e66d3b3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260129 02:58:41.988287 20575 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a5a05fbcbbcf4916bbc01fd1a5b5e90d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260129 02:58:41.991496 20503 tablet_service.cc:1505] Processing CreateTablet for tablet 87d5fe48a44a408aaf03e894a67ecb2f (DEFAULT_TABLE table=test-workload [id=de73611ac8e3485a98aecc5add247032]), partition=RANGE (key) PARTITION 109194081 <= VALUES < 145592108
I20260129 02:58:41.992909 20503 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 87d5fe48a44a408aaf03e894a67ecb2f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260129 02:58:41.997088 20502 tablet_service.cc:1505] Processing CreateTablet for tablet 3e2d15f8640b407195210c476e09df91 (DEFAULT_TABLE table=test-workload [id=de73611ac8e3485a98aecc5add247032]), partition=RANGE (key) PARTITION 145592108 <= VALUES < 181990135
I20260129 02:58:41.998515 20502 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3e2d15f8640b407195210c476e09df91. 1 dirs total, 0 dirs full, 0 dirs failed
I20260129 02:58:42.000281 20573 tablet_service.cc:1505] Processing CreateTablet for tablet 1299723a4476420ab02fd8bfd7aa0645 (DEFAULT_TABLE table=test-workload [id=de73611ac8e3485a98aecc5add247032]), partition=RANGE (key) PARTITION 254786189 <= VALUES < 291184216
I20260129 02:58:42.001042 20501 tablet_service.cc:1505] Processing CreateTablet for tablet a5a05fbcbbcf4916bbc01fd1a5b5e90d (DEFAULT_TABLE table=test-workload [id=de73611ac8e3485a98aecc5add247032]), partition=RANGE (key) PARTITION 181990135 <= VALUES < 218388162
I20260129 02:58:42.001677 20573 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1299723a4476420ab02fd8bfd7aa0645. 1 dirs total, 0 dirs full, 0 dirs failed
I20260129 02:58:42.002401 20501 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a5a05fbcbbcf4916bbc01fd1a5b5e90d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260129 02:58:42.005035 20500 tablet_service.cc:1505] Processing CreateTablet for tablet 2e58e9a5a8aa4cecafb464addb4f35ed (DEFAULT_TABLE table=test-workload [id=de73611ac8e3485a98aecc5add247032]), partition=RANGE (key) PARTITION 218388162 <= VALUES < 254786189
I20260129 02:58:42.006412 20500 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2e58e9a5a8aa4cecafb464addb4f35ed. 1 dirs total, 0 dirs full, 0 dirs failed
I20260129 02:58:42.009106 20499 tablet_service.cc:1505] Processing CreateTablet for tablet 1299723a4476420ab02fd8bfd7aa0645 (DEFAULT_TABLE table=test-workload [id=de73611ac8e3485a98aecc5add247032]), partition=RANGE (key) PARTITION 254786189 <= VALUES < 291184216
I20260129 02:58:42.010413 20499 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1299723a4476420ab02fd8bfd7aa0645. 1 dirs total, 0 dirs full, 0 dirs failed
I20260129 02:58:42.013222 20497 tablet_service.cc:1505] Processing CreateTablet for tablet 0f20af7fbe5d4db89a6f17948d98912c (DEFAULT_TABLE table=test-workload [id=de73611ac8e3485a98aecc5add247032]), partition=RANGE (key) PARTITION 327582243 <= VALUES < 363980270
I20260129 02:58:42.014139 20572 tablet_service.cc:1505] Processing CreateTablet for tablet 724a236524cb4e7ba3428880681d6cc2 (DEFAULT_TABLE table=test-workload [id=de73611ac8e3485a98aecc5add247032]), partition=RANGE (key) PARTITION 291184216 <= VALUES < 327582243
I20260129 02:58:42.015538 20572 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 724a236524cb4e7ba3428880681d6cc2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260129 02:58:42.016031 20498 tablet_service.cc:1505] Processing CreateTablet for tablet 724a236524cb4e7ba3428880681d6cc2 (DEFAULT_TABLE table=test-workload [id=de73611ac8e3485a98aecc5add247032]), partition=RANGE (key) PARTITION 291184216 <= VALUES < 327582243
I20260129 02:58:42.017066 20571 tablet_service.cc:1505] Processing CreateTablet for tablet 0f20af7fbe5d4db89a6f17948d98912c (DEFAULT_TABLE table=test-workload [id=de73611ac8e3485a98aecc5add247032]), partition=RANGE (key) PARTITION 327582243 <= VALUES < 363980270
I20260129 02:58:42.014603 20497 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0f20af7fbe5d4db89a6f17948d98912c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260129 02:58:42.023627 20498 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 724a236524cb4e7ba3428880681d6cc2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260129 02:58:42.025283 20430 tablet_service.cc:1505] Processing CreateTablet for tablet e7d83d4ccf684e76acb29a0dc844e1b8 (DEFAULT_TABLE table=test-workload [id=de73611ac8e3485a98aecc5add247032]), partition=RANGE (key) PARTITION 36398027 <= VALUES < 72796054
I20260129 02:58:42.026562 20430 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e7d83d4ccf684e76acb29a0dc844e1b8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260129 02:58:42.027520 20504 tablet_service.cc:1505] Processing CreateTablet for tablet 45cc5f3d2688486d9b6dbe7e666dedde (DEFAULT_TABLE table=test-workload [id=de73611ac8e3485a98aecc5add247032]), partition=RANGE (key) PARTITION 363980270 <= VALUES < 400378297
I20260129 02:58:42.028993 20571 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0f20af7fbe5d4db89a6f17948d98912c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260129 02:58:42.029841 20426 tablet_service.cc:1505] Processing CreateTablet for tablet a5a05fbcbbcf4916bbc01fd1a5b5e90d (DEFAULT_TABLE table=test-workload [id=de73611ac8e3485a98aecc5add247032]), partition=RANGE (key) PARTITION 181990135 <= VALUES < 218388162
I20260129 02:58:42.031132 20426 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a5a05fbcbbcf4916bbc01fd1a5b5e90d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260129 02:58:42.031062 20506 tablet_service.cc:1505] Processing CreateTablet for tablet 75f45538dc704d9a8ec4ff8ad9f3ea5b (DEFAULT_TABLE table=test-workload [id=de73611ac8e3485a98aecc5add247032]), partition=RANGE (key) PARTITION 400378297 <= VALUES < 436776324
I20260129 02:58:42.031445 20428 tablet_service.cc:1505] Processing CreateTablet for tablet 87d5fe48a44a408aaf03e894a67ecb2f (DEFAULT_TABLE table=test-workload [id=de73611ac8e3485a98aecc5add247032]), partition=RANGE (key) PARTITION 109194081 <= VALUES < 145592108
I20260129 02:58:42.032393 20506 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 75f45538dc704d9a8ec4ff8ad9f3ea5b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260129 02:58:42.032761 20428 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 87d5fe48a44a408aaf03e894a67ecb2f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260129 02:58:42.033025 20425 tablet_service.cc:1505] Processing CreateTablet for tablet 2e58e9a5a8aa4cecafb464addb4f35ed (DEFAULT_TABLE table=test-workload [id=de73611ac8e3485a98aecc5add247032]), partition=RANGE (key) PARTITION 218388162 <= VALUES < 254786189
I20260129 02:58:42.034327 20425 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2e58e9a5a8aa4cecafb464addb4f35ed. 1 dirs total, 0 dirs full, 0 dirs failed
I20260129 02:58:42.035000 20504 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 45cc5f3d2688486d9b6dbe7e666dedde. 1 dirs total, 0 dirs full, 0 dirs failed
I20260129 02:58:42.037087 20424 tablet_service.cc:1505] Processing CreateTablet for tablet 1299723a4476420ab02fd8bfd7aa0645 (DEFAULT_TABLE table=test-workload [id=de73611ac8e3485a98aecc5add247032]), partition=RANGE (key) PARTITION 254786189 <= VALUES < 291184216
I20260129 02:58:42.038409 20424 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1299723a4476420ab02fd8bfd7aa0645. 1 dirs total, 0 dirs full, 0 dirs failed
I20260129 02:58:42.041478 20423 tablet_service.cc:1505] Processing CreateTablet for tablet 724a236524cb4e7ba3428880681d6cc2 (DEFAULT_TABLE table=test-workload [id=de73611ac8e3485a98aecc5add247032]), partition=RANGE (key) PARTITION 291184216 <= VALUES < 327582243
I20260129 02:58:42.043042 20423 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 724a236524cb4e7ba3428880681d6cc2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260129 02:58:42.045045 20422 tablet_service.cc:1505] Processing CreateTablet for tablet 0f20af7fbe5d4db89a6f17948d98912c (DEFAULT_TABLE table=test-workload [id=de73611ac8e3485a98aecc5add247032]), partition=RANGE (key) PARTITION 327582243 <= VALUES < 363980270
I20260129 02:58:42.046329 20422 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0f20af7fbe5d4db89a6f17948d98912c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260129 02:58:42.049278 20429 tablet_service.cc:1505] Processing CreateTablet for tablet 4b1876f9a70b49edb3aa74105e66d3b3 (DEFAULT_TABLE table=test-workload [id=de73611ac8e3485a98aecc5add247032]), partition=RANGE (key) PARTITION 72796054 <= VALUES < 109194081
I20260129 02:58:42.061022 20431 tablet_service.cc:1505] Processing CreateTablet for tablet a1a39dcd328e4268ae41b988c66b4a75 (DEFAULT_TABLE table=test-workload [id=de73611ac8e3485a98aecc5add247032]), partition=RANGE (key) PARTITION VALUES < 36398027
I20260129 02:58:42.062448 20431 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a1a39dcd328e4268ae41b988c66b4a75. 1 dirs total, 0 dirs full, 0 dirs failed
I20260129 02:58:42.064288 20429 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4b1876f9a70b49edb3aa74105e66d3b3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260129 02:58:42.065058 20427 tablet_service.cc:1505] Processing CreateTablet for tablet 3e2d15f8640b407195210c476e09df91 (DEFAULT_TABLE table=test-workload [id=de73611ac8e3485a98aecc5add247032]), partition=RANGE (key) PARTITION 145592108 <= VALUES < 181990135
I20260129 02:58:42.066463 20427 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3e2d15f8640b407195210c476e09df91. 1 dirs total, 0 dirs full, 0 dirs failed
I20260129 02:58:42.075228 20580 tablet_service.cc:1505] Processing CreateTablet for tablet 75f45538dc704d9a8ec4ff8ad9f3ea5b (DEFAULT_TABLE table=test-workload [id=de73611ac8e3485a98aecc5add247032]), partition=RANGE (key) PARTITION 400378297 <= VALUES < 436776324
I20260129 02:58:42.094741 20580 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 75f45538dc704d9a8ec4ff8ad9f3ea5b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260129 02:58:42.093338 20575 tablet_service.cc:1505] Processing CreateTablet for tablet 45cc5f3d2688486d9b6dbe7e666dedde (DEFAULT_TABLE table=test-workload [id=de73611ac8e3485a98aecc5add247032]), partition=RANGE (key) PARTITION 363980270 <= VALUES < 400378297
I20260129 02:58:42.091892 20578 tablet_service.cc:1505] Processing CreateTablet for tablet 78aa39096868415d9afc2ffcb248e198 (DEFAULT_TABLE table=test-workload [id=de73611ac8e3485a98aecc5add247032]), partition=RANGE (key) PARTITION 436776324 <= VALUES < 473174351
I20260129 02:58:42.126327 20630 tablet_bootstrap.cc:492] T e7d83d4ccf684e76acb29a0dc844e1b8 P 974b802141bd4f13beffef2aa03f88f9: Bootstrap starting.
I20260129 02:58:42.133467 20574 tablet_service.cc:1505] Processing CreateTablet for tablet add8d4923d1a489e86882c3a25692b95 (DEFAULT_TABLE table=test-workload [id=de73611ac8e3485a98aecc5add247032]), partition=RANGE (key) PARTITION 509572378 <= VALUES < 545970405
I20260129 02:58:42.137008 20500 tablet_service.cc:1505] Processing CreateTablet for tablet b189f08e16a74a11980cc55d99359a0a (DEFAULT_TABLE table=test-workload [id=de73611ac8e3485a98aecc5add247032]), partition=RANGE (key) PARTITION 545970405 <= VALUES < 582368432
I20260129 02:58:42.135829 20503 tablet_service.cc:1505] Processing CreateTablet for tablet add8d4923d1a489e86882c3a25692b95 (DEFAULT_TABLE table=test-workload [id=de73611ac8e3485a98aecc5add247032]), partition=RANGE (key) PARTITION 509572378 <= VALUES < 545970405
I20260129 02:58:42.142511 20500 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b189f08e16a74a11980cc55d99359a0a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260129 02:58:42.144845 20630 tablet_bootstrap.cc:654] T e7d83d4ccf684e76acb29a0dc844e1b8 P 974b802141bd4f13beffef2aa03f88f9: Neither blocks nor log segments found. Creating new log.
I20260129 02:58:42.138573 20505 tablet_service.cc:1505] Processing CreateTablet for tablet 78aa39096868415d9afc2ffcb248e198 (DEFAULT_TABLE table=test-workload [id=de73611ac8e3485a98aecc5add247032]), partition=RANGE (key) PARTITION 436776324 <= VALUES < 473174351
I20260129 02:58:42.145368 20503 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet add8d4923d1a489e86882c3a25692b95. 1 dirs total, 0 dirs full, 0 dirs failed
I20260129 02:58:42.140385 20577 tablet_service.cc:1505] Processing CreateTablet for tablet b189f08e16a74a11980cc55d99359a0a (DEFAULT_TABLE table=test-workload [id=de73611ac8e3485a98aecc5add247032]), partition=RANGE (key) PARTITION 545970405 <= VALUES < 582368432
I20260129 02:58:42.146808 20505 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 78aa39096868415d9afc2ffcb248e198. 1 dirs total, 0 dirs full, 0 dirs failed
I20260129 02:58:42.153996 20504 tablet_service.cc:1505] Processing CreateTablet for tablet 2ac659d732314e1288849f08fe5e074e (DEFAULT_TABLE table=test-workload [id=de73611ac8e3485a98aecc5add247032]), partition=RANGE (key) PARTITION 618766459 <= VALUES < 655164486
I20260129 02:58:42.148236 20499 tablet_service.cc:1505] Processing CreateTablet for tablet a609ac02fae84725ab743f95f51d33ce (DEFAULT_TABLE table=test-workload [id=de73611ac8e3485a98aecc5add247032]), partition=RANGE (key) PARTITION 582368432 <= VALUES < 618766459
I20260129 02:58:42.149452 20497 tablet_service.cc:1505] Processing CreateTablet for tablet 17d6431cc77a491e8464d68db4081a51 (DEFAULT_TABLE table=test-workload [id=de73611ac8e3485a98aecc5add247032]), partition=RANGE (key) PARTITION 473174351 <= VALUES < 509572378
I20260129 02:58:42.159034 20576 tablet_service.cc:1505] Processing CreateTablet for tablet a609ac02fae84725ab743f95f51d33ce (DEFAULT_TABLE table=test-workload [id=de73611ac8e3485a98aecc5add247032]), partition=RANGE (key) PARTITION 582368432 <= VALUES < 618766459
I20260129 02:58:42.155299 20504 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2ac659d732314e1288849f08fe5e074e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260129 02:58:42.161659 20497 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 17d6431cc77a491e8464d68db4081a51. 1 dirs total, 0 dirs full, 0 dirs failed
I20260129 02:58:42.163506 20499 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a609ac02fae84725ab743f95f51d33ce. 1 dirs total, 0 dirs full, 0 dirs failed
I20260129 02:58:42.172092 20575 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 45cc5f3d2688486d9b6dbe7e666dedde. 1 dirs total, 0 dirs full, 0 dirs failed
I20260129 02:58:42.172549 20573 tablet_service.cc:1505] Processing CreateTablet for tablet 17d6431cc77a491e8464d68db4081a51 (DEFAULT_TABLE table=test-workload [id=de73611ac8e3485a98aecc5add247032]), partition=RANGE (key) PARTITION 473174351 <= VALUES < 509572378
I20260129 02:58:42.173870 20573 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 17d6431cc77a491e8464d68db4081a51. 1 dirs total, 0 dirs full, 0 dirs failed
I20260129 02:58:42.183458 20431 tablet_service.cc:1505] Processing CreateTablet for tablet 45cc5f3d2688486d9b6dbe7e666dedde (DEFAULT_TABLE table=test-workload [id=de73611ac8e3485a98aecc5add247032]), partition=RANGE (key) PARTITION 363980270 <= VALUES < 400378297
I20260129 02:58:42.186733 20431 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 45cc5f3d2688486d9b6dbe7e666dedde. 1 dirs total, 0 dirs full, 0 dirs failed
I20260129 02:58:42.190744 20579 tablet_service.cc:1505] Processing CreateTablet for tablet 2ac659d732314e1288849f08fe5e074e (DEFAULT_TABLE table=test-workload [id=de73611ac8e3485a98aecc5add247032]), partition=RANGE (key) PARTITION 618766459 <= VALUES < 655164486
I20260129 02:58:42.198488 20571 tablet_service.cc:1505] Processing CreateTablet for tablet 1ce4ddf5b10e4ae8b193b13b2e5936ee (DEFAULT_TABLE table=test-workload [id=de73611ac8e3485a98aecc5add247032]), partition=RANGE (key) PARTITION 655164486 <= VALUES < 691562513
I20260129 02:58:42.199745 20427 tablet_service.cc:1505] Processing CreateTablet for tablet 75f45538dc704d9a8ec4ff8ad9f3ea5b (DEFAULT_TABLE table=test-workload [id=de73611ac8e3485a98aecc5add247032]), partition=RANGE (key) PARTITION 400378297 <= VALUES < 436776324
I20260129 02:58:42.201395 20427 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 75f45538dc704d9a8ec4ff8ad9f3ea5b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260129 02:58:42.214602 20506 tablet_service.cc:1505] Processing CreateTablet for tablet 1ce4ddf5b10e4ae8b193b13b2e5936ee (DEFAULT_TABLE table=test-workload [id=de73611ac8e3485a98aecc5add247032]), partition=RANGE (key) PARTITION 655164486 <= VALUES < 691562513
I20260129 02:58:42.222126 20576 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a609ac02fae84725ab743f95f51d33ce. 1 dirs total, 0 dirs full, 0 dirs failed
I20260129 02:58:42.227248 20631 tablet_bootstrap.cc:492] T 87d5fe48a44a408aaf03e894a67ecb2f P 1c50de89a5a845668696dcb28c3badd6: Bootstrap starting.
I20260129 02:58:42.227655 20574 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet add8d4923d1a489e86882c3a25692b95. 1 dirs total, 0 dirs full, 0 dirs failed
I20260129 02:58:42.229925 20630 tablet_bootstrap.cc:492] T e7d83d4ccf684e76acb29a0dc844e1b8 P 974b802141bd4f13beffef2aa03f88f9: No bootstrap required, opened a new log
I20260129 02:58:42.230403 20630 ts_tablet_manager.cc:1403] T e7d83d4ccf684e76acb29a0dc844e1b8 P 974b802141bd4f13beffef2aa03f88f9: Time spent bootstrapping tablet: real 0.104s user 0.011s sys 0.008s
I20260129 02:58:42.231251 20577 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b189f08e16a74a11980cc55d99359a0a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260129 02:58:42.224393 20506 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1ce4ddf5b10e4ae8b193b13b2e5936ee. 1 dirs total, 0 dirs full, 0 dirs failed
I20260129 02:58:42.237212 20579 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2ac659d732314e1288849f08fe5e074e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260129 02:58:42.238109 20633 tablet_bootstrap.cc:492] T a1a39dcd328e4268ae41b988c66b4a75 P 197b880fa39d432a9b472845876bfaf0: Bootstrap starting.
I20260129 02:58:42.236267 20630 raft_consensus.cc:359] T e7d83d4ccf684e76acb29a0dc844e1b8 P 974b802141bd4f13beffef2aa03f88f9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:58:42.240159 20630 raft_consensus.cc:385] T e7d83d4ccf684e76acb29a0dc844e1b8 P 974b802141bd4f13beffef2aa03f88f9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260129 02:58:42.240577 20630 raft_consensus.cc:740] T e7d83d4ccf684e76acb29a0dc844e1b8 P 974b802141bd4f13beffef2aa03f88f9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 974b802141bd4f13beffef2aa03f88f9, State: Initialized, Role: FOLLOWER
I20260129 02:58:42.241439 20502 tablet_service.cc:1505] Processing CreateTablet for tablet 4e420f90884f4796997b5ea41e49e411 (DEFAULT_TABLE table=test-workload [id=de73611ac8e3485a98aecc5add247032]), partition=RANGE (key) PARTITION 691562513 <= VALUES < 727960540
I20260129 02:58:42.248214 20578 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 78aa39096868415d9afc2ffcb248e198. 1 dirs total, 0 dirs full, 0 dirs failed
I20260129 02:58:42.249078 20428 tablet_service.cc:1505] Processing CreateTablet for tablet 17d6431cc77a491e8464d68db4081a51 (DEFAULT_TABLE table=test-workload [id=de73611ac8e3485a98aecc5add247032]), partition=RANGE (key) PARTITION 473174351 <= VALUES < 509572378
I20260129 02:58:42.241454 20630 consensus_queue.cc:260] T e7d83d4ccf684e76acb29a0dc844e1b8 P 974b802141bd4f13beffef2aa03f88f9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:58:42.250756 20501 tablet_service.cc:1505] Processing CreateTablet for tablet e4d18c35b0a7493e85d0a4b92956a48f (DEFAULT_TABLE table=test-workload [id=de73611ac8e3485a98aecc5add247032]), partition=RANGE (key) PARTITION 727960540 <= VALUES < 764358567
I20260129 02:58:42.254716 20424 tablet_service.cc:1505] Processing CreateTablet for tablet add8d4923d1a489e86882c3a25692b95 (DEFAULT_TABLE table=test-workload [id=de73611ac8e3485a98aecc5add247032]), partition=RANGE (key) PARTITION 509572378 <= VALUES < 545970405
I20260129 02:58:42.250340 20428 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 17d6431cc77a491e8464d68db4081a51. 1 dirs total, 0 dirs full, 0 dirs failed
I20260129 02:58:42.259618 20571 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1ce4ddf5b10e4ae8b193b13b2e5936ee. 1 dirs total, 0 dirs full, 0 dirs failed
I20260129 02:58:42.270044 20501 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e4d18c35b0a7493e85d0a4b92956a48f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260129 02:58:42.274338 20427 tablet_service.cc:1505] Processing CreateTablet for tablet 78aa39096868415d9afc2ffcb248e198 (DEFAULT_TABLE table=test-workload [id=de73611ac8e3485a98aecc5add247032]), partition=RANGE (key) PARTITION 436776324 <= VALUES < 473174351
I20260129 02:58:42.275555 20424 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet add8d4923d1a489e86882c3a25692b95. 1 dirs total, 0 dirs full, 0 dirs failed
I20260129 02:58:42.282274 20631 tablet_bootstrap.cc:654] T 87d5fe48a44a408aaf03e894a67ecb2f P 1c50de89a5a845668696dcb28c3badd6: Neither blocks nor log segments found. Creating new log.
I20260129 02:58:42.287221 20633 tablet_bootstrap.cc:654] T a1a39dcd328e4268ae41b988c66b4a75 P 197b880fa39d432a9b472845876bfaf0: Neither blocks nor log segments found. Creating new log.
I20260129 02:58:42.297191 20502 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4e420f90884f4796997b5ea41e49e411. 1 dirs total, 0 dirs full, 0 dirs failed
I20260129 02:58:42.299623 20630 ts_tablet_manager.cc:1434] T e7d83d4ccf684e76acb29a0dc844e1b8 P 974b802141bd4f13beffef2aa03f88f9: Time spent starting tablet: real 0.069s user 0.006s sys 0.004s
I20260129 02:58:42.308813 20630 tablet_bootstrap.cc:492] T 4b1876f9a70b49edb3aa74105e66d3b3 P 974b802141bd4f13beffef2aa03f88f9: Bootstrap starting.
I20260129 02:58:42.312925 20425 tablet_service.cc:1505] Processing CreateTablet for tablet a609ac02fae84725ab743f95f51d33ce (DEFAULT_TABLE table=test-workload [id=de73611ac8e3485a98aecc5add247032]), partition=RANGE (key) PARTITION 582368432 <= VALUES < 618766459
I20260129 02:58:42.301446 20427 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 78aa39096868415d9afc2ffcb248e198. 1 dirs total, 0 dirs full, 0 dirs failed
I20260129 02:58:42.302666 20541 heartbeater.cc:499] Master 127.18.211.62:34285 was elected leader, sending a full tablet report...
I20260129 02:58:42.314260 20425 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a609ac02fae84725ab743f95f51d33ce. 1 dirs total, 0 dirs full, 0 dirs failed
I20260129 02:58:42.321141 20580 tablet_service.cc:1505] Processing CreateTablet for tablet 4e420f90884f4796997b5ea41e49e411 (DEFAULT_TABLE table=test-workload [id=de73611ac8e3485a98aecc5add247032]), partition=RANGE (key) PARTITION 691562513 <= VALUES < 727960540
I20260129 02:58:42.323248 20572 tablet_service.cc:1505] Processing CreateTablet for tablet e4d18c35b0a7493e85d0a4b92956a48f (DEFAULT_TABLE table=test-workload [id=de73611ac8e3485a98aecc5add247032]), partition=RANGE (key) PARTITION 727960540 <= VALUES < 764358567
I20260129 02:58:42.328475 20498 tablet_service.cc:1505] Processing CreateTablet for tablet febffe4a8b394368b69b1d164308fb9a (DEFAULT_TABLE table=test-workload [id=de73611ac8e3485a98aecc5add247032]), partition=RANGE (key) PARTITION 764358567 <= VALUES < 800756594
I20260129 02:58:42.306885 20429 tablet_service.cc:1505] Processing CreateTablet for tablet 2ac659d732314e1288849f08fe5e074e (DEFAULT_TABLE table=test-workload [id=de73611ac8e3485a98aecc5add247032]), partition=RANGE (key) PARTITION 618766459 <= VALUES < 655164486
I20260129 02:58:42.307155 20423 tablet_service.cc:1505] Processing CreateTablet for tablet 1ce4ddf5b10e4ae8b193b13b2e5936ee (DEFAULT_TABLE table=test-workload [id=de73611ac8e3485a98aecc5add247032]), partition=RANGE (key) PARTITION 655164486 <= VALUES < 691562513
I20260129 02:58:42.324915 20422 tablet_service.cc:1505] Processing CreateTablet for tablet 4e420f90884f4796997b5ea41e49e411 (DEFAULT_TABLE table=test-workload [id=de73611ac8e3485a98aecc5add247032]), partition=RANGE (key) PARTITION 691562513 <= VALUES < 727960540
I20260129 02:58:42.325119 20426 tablet_service.cc:1505] Processing CreateTablet for tablet e4d18c35b0a7493e85d0a4b92956a48f (DEFAULT_TABLE table=test-workload [id=de73611ac8e3485a98aecc5add247032]), partition=RANGE (key) PARTITION 727960540 <= VALUES < 764358567
I20260129 02:58:42.279512 20430 tablet_service.cc:1505] Processing CreateTablet for tablet b189f08e16a74a11980cc55d99359a0a (DEFAULT_TABLE table=test-workload [id=de73611ac8e3485a98aecc5add247032]), partition=RANGE (key) PARTITION 545970405 <= VALUES < 582368432
I20260129 02:58:42.330477 20498 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet febffe4a8b394368b69b1d164308fb9a. 1 dirs total, 0 dirs full, 0 dirs failed
W20260129 02:58:42.330889 20387 auto_rebalancer.cc:229] Could not retrieve cluster info: Service unavailable: Tablet not running
W20260129 02:58:42.338723 20542 tablet.cc:2378] T e7d83d4ccf684e76acb29a0dc844e1b8 P 974b802141bd4f13beffef2aa03f88f9: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260129 02:58:42.339566 20500 tablet_service.cc:1505] Processing CreateTablet for tablet fcddab6bca684adf8ea032289a00f90e (DEFAULT_TABLE table=test-workload [id=de73611ac8e3485a98aecc5add247032]), partition=RANGE (key) PARTITION 800756594 <= VALUES < 837154621
I20260129 02:58:42.369439 20500 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fcddab6bca684adf8ea032289a00f90e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260129 02:58:42.340794 20630 tablet_bootstrap.cc:654] T 4b1876f9a70b49edb3aa74105e66d3b3 P 974b802141bd4f13beffef2aa03f88f9: Neither blocks nor log segments found. Creating new log.
I20260129 02:58:42.341779 20633 tablet_bootstrap.cc:492] T a1a39dcd328e4268ae41b988c66b4a75 P 197b880fa39d432a9b472845876bfaf0: No bootstrap required, opened a new log
I20260129 02:58:42.342360 20505 tablet_service.cc:1505] Processing CreateTablet for tablet dc2f4cb752cf4d59b5f6bc689f5d8104 (DEFAULT_TABLE table=test-workload [id=de73611ac8e3485a98aecc5add247032]), partition=RANGE (key) PARTITION 837154621 <= VALUES < 873552648
I20260129 02:58:42.349035 20634 raft_consensus.cc:493] T e7d83d4ccf684e76acb29a0dc844e1b8 P 974b802141bd4f13beffef2aa03f88f9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260129 02:58:42.381366 20633 ts_tablet_manager.cc:1403] T a1a39dcd328e4268ae41b988c66b4a75 P 197b880fa39d432a9b472845876bfaf0: Time spent bootstrapping tablet: real 0.144s user 0.010s sys 0.006s
I20260129 02:58:42.381383 20634 raft_consensus.cc:515] T e7d83d4ccf684e76acb29a0dc844e1b8 P 974b802141bd4f13beffef2aa03f88f9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:58:42.384470 20633 raft_consensus.cc:359] T a1a39dcd328e4268ae41b988c66b4a75 P 197b880fa39d432a9b472845876bfaf0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } } peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } }
I20260129 02:58:42.385193 20633 raft_consensus.cc:385] T a1a39dcd328e4268ae41b988c66b4a75 P 197b880fa39d432a9b472845876bfaf0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260129 02:58:42.385488 20633 raft_consensus.cc:740] T a1a39dcd328e4268ae41b988c66b4a75 P 197b880fa39d432a9b472845876bfaf0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 197b880fa39d432a9b472845876bfaf0, State: Initialized, Role: FOLLOWER
I20260129 02:58:42.385780 20505 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet dc2f4cb752cf4d59b5f6bc689f5d8104. 1 dirs total, 0 dirs full, 0 dirs failed
I20260129 02:58:42.386158 20633 consensus_queue.cc:260] T a1a39dcd328e4268ae41b988c66b4a75 P 197b880fa39d432a9b472845876bfaf0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } } peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } }
I20260129 02:58:42.390652 20634 leader_election.cc:290] T e7d83d4ccf684e76acb29a0dc844e1b8 P 974b802141bd4f13beffef2aa03f88f9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 197b880fa39d432a9b472845876bfaf0 (127.18.211.1:42027), 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3:37735)
I20260129 02:58:42.395605 20573 tablet_service.cc:1505] Processing CreateTablet for tablet dc2f4cb752cf4d59b5f6bc689f5d8104 (DEFAULT_TABLE table=test-workload [id=de73611ac8e3485a98aecc5add247032]), partition=RANGE (key) PARTITION 837154621 <= VALUES < 873552648
I20260129 02:58:42.356669 20572 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e4d18c35b0a7493e85d0a4b92956a48f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260129 02:58:42.357578 20575 tablet_service.cc:1505] Processing CreateTablet for tablet febffe4a8b394368b69b1d164308fb9a (DEFAULT_TABLE table=test-workload [id=de73611ac8e3485a98aecc5add247032]), partition=RANGE (key) PARTITION 764358567 <= VALUES < 800756594
I20260129 02:58:42.359257 20576 tablet_service.cc:1505] Processing CreateTablet for tablet fcddab6bca684adf8ea032289a00f90e (DEFAULT_TABLE table=test-workload [id=de73611ac8e3485a98aecc5add247032]), partition=RANGE (key) PARTITION 800756594 <= VALUES < 837154621
I20260129 02:58:42.406606 20631 tablet_bootstrap.cc:492] T 87d5fe48a44a408aaf03e894a67ecb2f P 1c50de89a5a845668696dcb28c3badd6: No bootstrap required, opened a new log
I20260129 02:58:42.365322 20422 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4e420f90884f4796997b5ea41e49e411. 1 dirs total, 0 dirs full, 0 dirs failed
I20260129 02:58:42.407087 20631 ts_tablet_manager.cc:1403] T 87d5fe48a44a408aaf03e894a67ecb2f P 1c50de89a5a845668696dcb28c3badd6: Time spent bootstrapping tablet: real 0.185s user 0.004s sys 0.012s
I20260129 02:58:42.409057 20428 tablet_service.cc:1505] Processing CreateTablet for tablet febffe4a8b394368b69b1d164308fb9a (DEFAULT_TABLE table=test-workload [id=de73611ac8e3485a98aecc5add247032]), partition=RANGE (key) PARTITION 764358567 <= VALUES < 800756594
I20260129 02:58:42.409838 20423 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1ce4ddf5b10e4ae8b193b13b2e5936ee. 1 dirs total, 0 dirs full, 0 dirs failed
I20260129 02:58:42.409978 20631 raft_consensus.cc:359] T 87d5fe48a44a408aaf03e894a67ecb2f P 1c50de89a5a845668696dcb28c3badd6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:58:42.410725 20631 raft_consensus.cc:385] T 87d5fe48a44a408aaf03e894a67ecb2f P 1c50de89a5a845668696dcb28c3badd6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260129 02:58:42.410826 20428 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet febffe4a8b394368b69b1d164308fb9a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260129 02:58:42.411044 20631 raft_consensus.cc:740] T 87d5fe48a44a408aaf03e894a67ecb2f P 1c50de89a5a845668696dcb28c3badd6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1c50de89a5a845668696dcb28c3badd6, State: Initialized, Role: FOLLOWER
I20260129 02:58:42.413419 20429 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2ac659d732314e1288849f08fe5e074e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260129 02:58:42.420439 20580 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4e420f90884f4796997b5ea41e49e411. 1 dirs total, 0 dirs full, 0 dirs failed
I20260129 02:58:42.428349 20578 tablet_service.cc:1505] Processing CreateTablet for tablet 76f713c3331540d2a5b76164103e9eb1 (DEFAULT_TABLE table=test-workload [id=de73611ac8e3485a98aecc5add247032]), partition=RANGE (key) PARTITION 909950675 <= VALUES < 946348702
I20260129 02:58:42.429805 20578 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 76f713c3331540d2a5b76164103e9eb1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260129 02:58:42.430956 20431 tablet_service.cc:1505] Processing CreateTablet for tablet fcddab6bca684adf8ea032289a00f90e (DEFAULT_TABLE table=test-workload [id=de73611ac8e3485a98aecc5add247032]), partition=RANGE (key) PARTITION 800756594 <= VALUES < 837154621
I20260129 02:58:42.431787 20573 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet dc2f4cb752cf4d59b5f6bc689f5d8104. 1 dirs total, 0 dirs full, 0 dirs failed
I20260129 02:58:42.395617 20577 tablet_service.cc:1505] Processing CreateTablet for tablet 913fc7d751ed4b7388d889aac30a5f9b (DEFAULT_TABLE table=test-workload [id=de73611ac8e3485a98aecc5add247032]), partition=RANGE (key) PARTITION 873552648 <= VALUES < 909950675
I20260129 02:58:42.413339 20631 consensus_queue.cc:260] T 87d5fe48a44a408aaf03e894a67ecb2f P 1c50de89a5a845668696dcb28c3badd6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:58:42.441167 20630 tablet_bootstrap.cc:492] T 4b1876f9a70b49edb3aa74105e66d3b3 P 974b802141bd4f13beffef2aa03f88f9: No bootstrap required, opened a new log
I20260129 02:58:42.444152 20576 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fcddab6bca684adf8ea032289a00f90e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260129 02:58:42.444865 20575 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet febffe4a8b394368b69b1d164308fb9a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260129 02:58:42.444176 20630 ts_tablet_manager.cc:1403] T 4b1876f9a70b49edb3aa74105e66d3b3 P 974b802141bd4f13beffef2aa03f88f9: Time spent bootstrapping tablet: real 0.136s user 0.008s sys 0.007s
I20260129 02:58:42.445145 20579 tablet_service.cc:1505] Processing CreateTablet for tablet 4f8f3a38f6e24ba4857875181f447442 (DEFAULT_TABLE table=test-workload [id=de73611ac8e3485a98aecc5add247032]), partition=RANGE (key) PARTITION 946348702 <= VALUES < 982746729
I20260129 02:58:42.446460 20579 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4f8f3a38f6e24ba4857875181f447442. 1 dirs total, 0 dirs full, 0 dirs failed
I20260129 02:58:42.452504 20466 heartbeater.cc:499] Master 127.18.211.62:34285 was elected leader, sending a full tablet report...
I20260129 02:58:42.454635 20630 raft_consensus.cc:359] T 4b1876f9a70b49edb3aa74105e66d3b3 P 974b802141bd4f13beffef2aa03f88f9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:58:42.455421 20630 raft_consensus.cc:385] T 4b1876f9a70b49edb3aa74105e66d3b3 P 974b802141bd4f13beffef2aa03f88f9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260129 02:58:42.455751 20630 raft_consensus.cc:740] T 4b1876f9a70b49edb3aa74105e66d3b3 P 974b802141bd4f13beffef2aa03f88f9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 974b802141bd4f13beffef2aa03f88f9, State: Initialized, Role: FOLLOWER
I20260129 02:58:42.456496 20630 consensus_queue.cc:260] T 4b1876f9a70b49edb3aa74105e66d3b3 P 974b802141bd4f13beffef2aa03f88f9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:58:42.441648 20426 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e4d18c35b0a7493e85d0a4b92956a48f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260129 02:58:42.458671 20430 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b189f08e16a74a11980cc55d99359a0a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260129 02:58:42.475013 20577 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 913fc7d751ed4b7388d889aac30a5f9b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260129 02:58:42.476972 20574 tablet_service.cc:1505] Processing CreateTablet for tablet fd8311b779af4e1fa1402b2a02a602f2 (DEFAULT_TABLE table=test-workload [id=de73611ac8e3485a98aecc5add247032]), partition=RANGE (key) PARTITION 982746729 <= VALUES < 1019144756
I20260129 02:58:42.478317 20574 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fd8311b779af4e1fa1402b2a02a602f2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260129 02:58:42.481621 20631 ts_tablet_manager.cc:1434] T 87d5fe48a44a408aaf03e894a67ecb2f P 1c50de89a5a845668696dcb28c3badd6: Time spent starting tablet: real 0.074s user 0.006s sys 0.009s
I20260129 02:58:42.482378 20615 heartbeater.cc:499] Master 127.18.211.62:34285 was elected leader, sending a full tablet report...
I20260129 02:58:42.489521 20431 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fcddab6bca684adf8ea032289a00f90e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260129 02:58:42.497038 20631 tablet_bootstrap.cc:492] T a5a05fbcbbcf4916bbc01fd1a5b5e90d P 1c50de89a5a845668696dcb28c3badd6: Bootstrap starting.
I20260129 02:58:42.501024 20630 ts_tablet_manager.cc:1434] T 4b1876f9a70b49edb3aa74105e66d3b3 P 974b802141bd4f13beffef2aa03f88f9: Time spent starting tablet: real 0.055s user 0.006s sys 0.000s
I20260129 02:58:42.501235 20502 tablet_service.cc:1505] Processing CreateTablet for tablet 76f713c3331540d2a5b76164103e9eb1 (DEFAULT_TABLE table=test-workload [id=de73611ac8e3485a98aecc5add247032]), partition=RANGE (key) PARTITION 909950675 <= VALUES < 946348702
I20260129 02:58:42.502514 20502 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 76f713c3331540d2a5b76164103e9eb1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260129 02:58:42.503096 20631 tablet_bootstrap.cc:654] T a5a05fbcbbcf4916bbc01fd1a5b5e90d P 1c50de89a5a845668696dcb28c3badd6: Neither blocks nor log segments found. Creating new log.
I20260129 02:58:42.505311 20506 tablet_service.cc:1505] Processing CreateTablet for tablet 4f8f3a38f6e24ba4857875181f447442 (DEFAULT_TABLE table=test-workload [id=de73611ac8e3485a98aecc5add247032]), partition=RANGE (key) PARTITION 946348702 <= VALUES < 982746729
I20260129 02:58:42.506275 20441 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e7d83d4ccf684e76acb29a0dc844e1b8" candidate_uuid: "974b802141bd4f13beffef2aa03f88f9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "197b880fa39d432a9b472845876bfaf0" is_pre_election: true
I20260129 02:58:42.506672 20506 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4f8f3a38f6e24ba4857875181f447442. 1 dirs total, 0 dirs full, 0 dirs failed
I20260129 02:58:42.507797 20504 tablet_service.cc:1505] Processing CreateTablet for tablet fd8311b779af4e1fa1402b2a02a602f2 (DEFAULT_TABLE table=test-workload [id=de73611ac8e3485a98aecc5add247032]), partition=RANGE (key) PARTITION 982746729 <= VALUES < 1019144756
I20260129 02:58:42.509064 20504 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fd8311b779af4e1fa1402b2a02a602f2. 1 dirs total, 0 dirs full, 0 dirs failed
W20260129 02:58:42.514268 20479 leader_election.cc:343] T e7d83d4ccf684e76acb29a0dc844e1b8 P 974b802141bd4f13beffef2aa03f88f9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 197b880fa39d432a9b472845876bfaf0 (127.18.211.1:42027): Illegal state: must be running to vote when last-logged opid is not known
I20260129 02:58:42.519011 20633 ts_tablet_manager.cc:1434] T a1a39dcd328e4268ae41b988c66b4a75 P 197b880fa39d432a9b472845876bfaf0: Time spent starting tablet: real 0.137s user 0.015s sys 0.005s
I20260129 02:58:42.520414 20633 tablet_bootstrap.cc:492] T 3e2d15f8640b407195210c476e09df91 P 197b880fa39d432a9b472845876bfaf0: Bootstrap starting.
I20260129 02:58:42.520891 20427 tablet_service.cc:1505] Processing CreateTablet for tablet dc2f4cb752cf4d59b5f6bc689f5d8104 (DEFAULT_TABLE table=test-workload [id=de73611ac8e3485a98aecc5add247032]), partition=RANGE (key) PARTITION 837154621 <= VALUES < 873552648
I20260129 02:58:42.522178 20427 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet dc2f4cb752cf4d59b5f6bc689f5d8104. 1 dirs total, 0 dirs full, 0 dirs failed
I20260129 02:58:42.524755 20501 tablet_service.cc:1505] Processing CreateTablet for tablet 913fc7d751ed4b7388d889aac30a5f9b (DEFAULT_TABLE table=test-workload [id=de73611ac8e3485a98aecc5add247032]), partition=RANGE (key) PARTITION 873552648 <= VALUES < 909950675
I20260129 02:58:42.526343 20501 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 913fc7d751ed4b7388d889aac30a5f9b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260129 02:58:42.552127 20630 tablet_bootstrap.cc:492] T a1a39dcd328e4268ae41b988c66b4a75 P 974b802141bd4f13beffef2aa03f88f9: Bootstrap starting.
I20260129 02:58:42.567250 20641 raft_consensus.cc:493] T 87d5fe48a44a408aaf03e894a67ecb2f P 1c50de89a5a845668696dcb28c3badd6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260129 02:58:42.576969 20499 tablet_service.cc:1505] Processing CreateTablet for tablet 75b0b6e82ddc4f92b1fe1d94f9a0b73b (DEFAULT_TABLE table=test-workload [id=de73611ac8e3485a98aecc5add247032]), partition=RANGE (key) PARTITION 1019144756 <= VALUES < 1055542783
I20260129 02:58:42.578375 20499 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 75b0b6e82ddc4f92b1fe1d94f9a0b73b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260129 02:58:42.581352 20633 tablet_bootstrap.cc:654] T 3e2d15f8640b407195210c476e09df91 P 197b880fa39d432a9b472845876bfaf0: Neither blocks nor log segments found. Creating new log.
I20260129 02:58:42.600276 20576 tablet_service.cc:1505] Processing CreateTablet for tablet 75b0b6e82ddc4f92b1fe1d94f9a0b73b (DEFAULT_TABLE table=test-workload [id=de73611ac8e3485a98aecc5add247032]), partition=RANGE (key) PARTITION 1019144756 <= VALUES < 1055542783
I20260129 02:58:42.608042 20425 tablet_service.cc:1505] Processing CreateTablet for tablet 913fc7d751ed4b7388d889aac30a5f9b (DEFAULT_TABLE table=test-workload [id=de73611ac8e3485a98aecc5add247032]), partition=RANGE (key) PARTITION 873552648 <= VALUES < 909950675
I20260129 02:58:42.622061 20426 tablet_service.cc:1505] Processing CreateTablet for tablet 76f713c3331540d2a5b76164103e9eb1 (DEFAULT_TABLE table=test-workload [id=de73611ac8e3485a98aecc5add247032]), partition=RANGE (key) PARTITION 909950675 <= VALUES < 946348702
I20260129 02:58:42.603204 20641 raft_consensus.cc:515] T 87d5fe48a44a408aaf03e894a67ecb2f P 1c50de89a5a845668696dcb28c3badd6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:58:42.632838 20576 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 75b0b6e82ddc4f92b1fe1d94f9a0b73b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260129 02:58:42.639423 20630 tablet_bootstrap.cc:654] T a1a39dcd328e4268ae41b988c66b4a75 P 974b802141bd4f13beffef2aa03f88f9: Neither blocks nor log segments found. Creating new log.
I20260129 02:58:42.635401 20631 tablet_bootstrap.cc:492] T a5a05fbcbbcf4916bbc01fd1a5b5e90d P 1c50de89a5a845668696dcb28c3badd6: No bootstrap required, opened a new log
I20260129 02:58:42.645146 20631 ts_tablet_manager.cc:1403] T a5a05fbcbbcf4916bbc01fd1a5b5e90d P 1c50de89a5a845668696dcb28c3badd6: Time spent bootstrapping tablet: real 0.148s user 0.017s sys 0.010s
I20260129 02:58:42.652104 20631 raft_consensus.cc:359] T a5a05fbcbbcf4916bbc01fd1a5b5e90d P 1c50de89a5a845668696dcb28c3badd6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:58:42.652868 20631 raft_consensus.cc:385] T a5a05fbcbbcf4916bbc01fd1a5b5e90d P 1c50de89a5a845668696dcb28c3badd6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260129 02:58:42.653165 20631 raft_consensus.cc:740] T a5a05fbcbbcf4916bbc01fd1a5b5e90d P 1c50de89a5a845668696dcb28c3badd6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1c50de89a5a845668696dcb28c3badd6, State: Initialized, Role: FOLLOWER
I20260129 02:58:42.653399 20500 tablet_service.cc:1505] Processing CreateTablet for tablet 505580c93c694bbb8ea96fdf9b17ab46 (DEFAULT_TABLE table=test-workload [id=de73611ac8e3485a98aecc5add247032]), partition=RANGE (key) PARTITION 1055542783 <= VALUES < 1091940810
I20260129 02:58:42.654702 20631 consensus_queue.cc:260] T a5a05fbcbbcf4916bbc01fd1a5b5e90d P 1c50de89a5a845668696dcb28c3badd6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:58:42.655542 20500 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 505580c93c694bbb8ea96fdf9b17ab46. 1 dirs total, 0 dirs full, 0 dirs failed
I20260129 02:58:42.657023 20631 ts_tablet_manager.cc:1434] T a5a05fbcbbcf4916bbc01fd1a5b5e90d P 1c50de89a5a845668696dcb28c3badd6: Time spent starting tablet: real 0.012s user 0.006s sys 0.000s
I20260129 02:58:42.636150 20424 tablet_service.cc:1505] Processing CreateTablet for tablet 4f8f3a38f6e24ba4857875181f447442 (DEFAULT_TABLE table=test-workload [id=de73611ac8e3485a98aecc5add247032]), partition=RANGE (key) PARTITION 946348702 <= VALUES < 982746729
I20260129 02:58:42.697474 20425 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 913fc7d751ed4b7388d889aac30a5f9b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260129 02:58:42.665515 20641 leader_election.cc:290] T 87d5fe48a44a408aaf03e894a67ecb2f P 1c50de89a5a845668696dcb28c3badd6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 197b880fa39d432a9b472845876bfaf0 (127.18.211.1:42027), 974b802141bd4f13beffef2aa03f88f9 (127.18.211.2:34517)
I20260129 02:58:42.711959 20633 tablet_bootstrap.cc:492] T 3e2d15f8640b407195210c476e09df91 P 197b880fa39d432a9b472845876bfaf0: No bootstrap required, opened a new log
I20260129 02:58:42.712468 20633 ts_tablet_manager.cc:1403] T 3e2d15f8640b407195210c476e09df91 P 197b880fa39d432a9b472845876bfaf0: Time spent bootstrapping tablet: real 0.192s user 0.030s sys 0.049s
I20260129 02:58:42.703504 20630 tablet_bootstrap.cc:492] T a1a39dcd328e4268ae41b988c66b4a75 P 974b802141bd4f13beffef2aa03f88f9: No bootstrap required, opened a new log
I20260129 02:58:42.720209 20630 ts_tablet_manager.cc:1403] T a1a39dcd328e4268ae41b988c66b4a75 P 974b802141bd4f13beffef2aa03f88f9: Time spent bootstrapping tablet: real 0.168s user 0.017s sys 0.002s
I20260129 02:58:42.722970 20630 raft_consensus.cc:359] T a1a39dcd328e4268ae41b988c66b4a75 P 974b802141bd4f13beffef2aa03f88f9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } } peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } }
I20260129 02:58:42.713901 20422 tablet_service.cc:1505] Processing CreateTablet for tablet fd8311b779af4e1fa1402b2a02a602f2 (DEFAULT_TABLE table=test-workload [id=de73611ac8e3485a98aecc5add247032]), partition=RANGE (key) PARTITION 982746729 <= VALUES < 1019144756
I20260129 02:58:42.724222 20572 tablet_service.cc:1505] Processing CreateTablet for tablet 505580c93c694bbb8ea96fdf9b17ab46 (DEFAULT_TABLE table=test-workload [id=de73611ac8e3485a98aecc5add247032]), partition=RANGE (key) PARTITION 1055542783 <= VALUES < 1091940810
I20260129 02:58:42.717016 20631 tablet_bootstrap.cc:492] T a1a39dcd328e4268ae41b988c66b4a75 P 1c50de89a5a845668696dcb28c3badd6: Bootstrap starting.
I20260129 02:58:42.731470 20590 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e7d83d4ccf684e76acb29a0dc844e1b8" candidate_uuid: "974b802141bd4f13beffef2aa03f88f9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c50de89a5a845668696dcb28c3badd6" is_pre_election: true
I20260129 02:58:42.734048 20498 tablet_service.cc:1505] Processing CreateTablet for tablet 368953c9102643b2a21f99af36b51905 (DEFAULT_TABLE table=test-workload [id=de73611ac8e3485a98aecc5add247032]), partition=RANGE (key) PARTITION 1128338837 <= VALUES < 1164736864
I20260129 02:58:42.735419 20498 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 368953c9102643b2a21f99af36b51905. 1 dirs total, 0 dirs full, 0 dirs failed
I20260129 02:58:42.714694 20423 tablet_service.cc:1505] Processing CreateTablet for tablet 75b0b6e82ddc4f92b1fe1d94f9a0b73b (DEFAULT_TABLE table=test-workload [id=de73611ac8e3485a98aecc5add247032]), partition=RANGE (key) PARTITION 1019144756 <= VALUES < 1055542783
I20260129 02:58:42.737661 20572 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 505580c93c694bbb8ea96fdf9b17ab46. 1 dirs total, 0 dirs full, 0 dirs failed
I20260129 02:58:42.727499 20504 tablet_service.cc:1505] Processing CreateTablet for tablet 4408006eb5174de1896b037cf3f18457 (DEFAULT_TABLE table=test-workload [id=de73611ac8e3485a98aecc5add247032]), partition=RANGE (key) PARTITION 1091940810 <= VALUES < 1128338837
I20260129 02:58:42.744325 20504 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4408006eb5174de1896b037cf3f18457. 1 dirs total, 0 dirs full, 0 dirs failed
I20260129 02:58:42.747146 20426 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 76f713c3331540d2a5b76164103e9eb1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260129 02:58:42.747761 20424 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4f8f3a38f6e24ba4857875181f447442. 1 dirs total, 0 dirs full, 0 dirs failed
I20260129 02:58:42.723691 20630 raft_consensus.cc:385] T a1a39dcd328e4268ae41b988c66b4a75 P 974b802141bd4f13beffef2aa03f88f9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260129 02:58:42.728220 20573 tablet_service.cc:1505] Processing CreateTablet for tablet 4408006eb5174de1896b037cf3f18457 (DEFAULT_TABLE table=test-workload [id=de73611ac8e3485a98aecc5add247032]), partition=RANGE (key) PARTITION 1091940810 <= VALUES < 1128338837
I20260129 02:58:42.752068 20631 tablet_bootstrap.cc:654] T a1a39dcd328e4268ae41b988c66b4a75 P 1c50de89a5a845668696dcb28c3badd6: Neither blocks nor log segments found. Creating new log.
I20260129 02:58:42.752092 20630 raft_consensus.cc:740] T a1a39dcd328e4268ae41b988c66b4a75 P 974b802141bd4f13beffef2aa03f88f9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 974b802141bd4f13beffef2aa03f88f9, State: Initialized, Role: FOLLOWER
I20260129 02:58:42.753818 20630 consensus_queue.cc:260] T a1a39dcd328e4268ae41b988c66b4a75 P 974b802141bd4f13beffef2aa03f88f9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } } peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } }
I20260129 02:58:42.741189 20503 tablet_service.cc:1505] Processing CreateTablet for tablet c2633a6fc34c44579de348d208b0c01e (DEFAULT_TABLE table=test-workload [id=de73611ac8e3485a98aecc5add247032]), partition=RANGE (key) PARTITION 1164736864 <= VALUES < 1201134891
I20260129 02:58:42.755016 20423 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 75b0b6e82ddc4f92b1fe1d94f9a0b73b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260129 02:58:42.755400 20503 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c2633a6fc34c44579de348d208b0c01e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260129 02:58:42.742789 20575 tablet_service.cc:1505] Processing CreateTablet for tablet 368953c9102643b2a21f99af36b51905 (DEFAULT_TABLE table=test-workload [id=de73611ac8e3485a98aecc5add247032]), partition=RANGE (key) PARTITION 1128338837 <= VALUES < 1164736864
I20260129 02:58:42.726171 20431 tablet_service.cc:1505] Processing CreateTablet for tablet 505580c93c694bbb8ea96fdf9b17ab46 (DEFAULT_TABLE table=test-workload [id=de73611ac8e3485a98aecc5add247032]), partition=RANGE (key) PARTITION 1055542783 <= VALUES < 1091940810
W20260129 02:58:42.758965 20480 leader_election.cc:343] T e7d83d4ccf684e76acb29a0dc844e1b8 P 974b802141bd4f13beffef2aa03f88f9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3:37735): Illegal state: must be running to vote when last-logged opid is not known
I20260129 02:58:42.759501 20480 leader_election.cc:304] T e7d83d4ccf684e76acb29a0dc844e1b8 P 974b802141bd4f13beffef2aa03f88f9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 974b802141bd4f13beffef2aa03f88f9; no voters: 197b880fa39d432a9b472845876bfaf0, 1c50de89a5a845668696dcb28c3badd6
I20260129 02:58:42.761123 20638 raft_consensus.cc:493] T a1a39dcd328e4268ae41b988c66b4a75 P 197b880fa39d432a9b472845876bfaf0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260129 02:58:42.761615 20638 raft_consensus.cc:515] T a1a39dcd328e4268ae41b988c66b4a75 P 197b880fa39d432a9b472845876bfaf0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } } peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } }
I20260129 02:58:42.759361 20428 tablet_service.cc:1505] Processing CreateTablet for tablet 4408006eb5174de1896b037cf3f18457 (DEFAULT_TABLE table=test-workload [id=de73611ac8e3485a98aecc5add247032]), partition=RANGE (key) PARTITION 1091940810 <= VALUES < 1128338837
I20260129 02:58:42.759007 20422 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fd8311b779af4e1fa1402b2a02a602f2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260129 02:58:42.775374 20573 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4408006eb5174de1896b037cf3f18457. 1 dirs total, 0 dirs full, 0 dirs failed
I20260129 02:58:42.784180 20634 raft_consensus.cc:2749] T e7d83d4ccf684e76acb29a0dc844e1b8 P 974b802141bd4f13beffef2aa03f88f9 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260129 02:58:42.764444 20638 leader_election.cc:290] T a1a39dcd328e4268ae41b988c66b4a75 P 197b880fa39d432a9b472845876bfaf0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 974b802141bd4f13beffef2aa03f88f9 (127.18.211.2:34517), 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3:37735)
I20260129 02:58:42.715222 20633 raft_consensus.cc:359] T 3e2d15f8640b407195210c476e09df91 P 197b880fa39d432a9b472845876bfaf0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:58:42.794986 20441 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "87d5fe48a44a408aaf03e894a67ecb2f" candidate_uuid: "1c50de89a5a845668696dcb28c3badd6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "197b880fa39d432a9b472845876bfaf0" is_pre_election: true
I20260129 02:58:42.805441 20429 tablet_service.cc:1505] Processing CreateTablet for tablet 368953c9102643b2a21f99af36b51905 (DEFAULT_TABLE table=test-workload [id=de73611ac8e3485a98aecc5add247032]), partition=RANGE (key) PARTITION 1128338837 <= VALUES < 1164736864
W20260129 02:58:42.808406 20553 leader_election.cc:343] T 87d5fe48a44a408aaf03e894a67ecb2f P 1c50de89a5a845668696dcb28c3badd6 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 197b880fa39d432a9b472845876bfaf0 (127.18.211.1:42027): Illegal state: must be running to vote when last-logged opid is not known
I20260129 02:58:42.826203 20506 tablet_service.cc:1505] Processing CreateTablet for tablet 36ff33be666a4c82954731691dbcf327 (DEFAULT_TABLE table=test-workload [id=de73611ac8e3485a98aecc5add247032]), partition=RANGE (key) PARTITION 1201134891 <= VALUES < 1237532918
I20260129 02:58:42.811113 20427 tablet_service.cc:1505] Processing CreateTablet for tablet c2633a6fc34c44579de348d208b0c01e (DEFAULT_TABLE table=test-workload [id=de73611ac8e3485a98aecc5add247032]), partition=RANGE (key) PARTITION 1164736864 <= VALUES < 1201134891
I20260129 02:58:42.829742 20506 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 36ff33be666a4c82954731691dbcf327. 1 dirs total, 0 dirs full, 0 dirs failed
I20260129 02:58:42.829936 20577 tablet_service.cc:1505] Processing CreateTablet for tablet c2633a6fc34c44579de348d208b0c01e (DEFAULT_TABLE table=test-workload [id=de73611ac8e3485a98aecc5add247032]), partition=RANGE (key) PARTITION 1164736864 <= VALUES < 1201134891
I20260129 02:58:42.831993 20579 tablet_service.cc:1505] Processing CreateTablet for tablet 36ff33be666a4c82954731691dbcf327 (DEFAULT_TABLE table=test-workload [id=de73611ac8e3485a98aecc5add247032]), partition=RANGE (key) PARTITION 1201134891 <= VALUES < 1237532918
I20260129 02:58:42.832718 20431 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 505580c93c694bbb8ea96fdf9b17ab46. 1 dirs total, 0 dirs full, 0 dirs failed
I20260129 02:58:42.835052 20499 tablet_service.cc:1505] Processing CreateTablet for tablet 7a9a69cd82724b1faed1d10aa091fc4a (DEFAULT_TABLE table=test-workload [id=de73611ac8e3485a98aecc5add247032]), partition=RANGE (key) PARTITION 1237532918 <= VALUES < 1273930945
I20260129 02:58:42.845010 20499 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7a9a69cd82724b1faed1d10aa091fc4a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260129 02:58:42.837422 20580 tablet_service.cc:1505] Processing CreateTablet for tablet 7a9a69cd82724b1faed1d10aa091fc4a (DEFAULT_TABLE table=test-workload [id=de73611ac8e3485a98aecc5add247032]), partition=RANGE (key) PARTITION 1237532918 <= VALUES < 1273930945
I20260129 02:58:42.848212 20498 tablet_service.cc:1505] Processing CreateTablet for tablet 96609ac68bdf4993a12e1c23355ca013 (DEFAULT_TABLE table=test-workload [id=de73611ac8e3485a98aecc5add247032]), partition=RANGE (key) PARTITION 1273930945 <= VALUES < 1310328972
I20260129 02:58:42.804188 20575 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 368953c9102643b2a21f99af36b51905. 1 dirs total, 0 dirs full, 0 dirs failed
I20260129 02:58:42.849256 20630 ts_tablet_manager.cc:1434] T a1a39dcd328e4268ae41b988c66b4a75 P 974b802141bd4f13beffef2aa03f88f9: Time spent starting tablet: real 0.129s user 0.007s sys 0.001s
I20260129 02:58:42.849484 20498 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 96609ac68bdf4993a12e1c23355ca013. 1 dirs total, 0 dirs full, 0 dirs failed
I20260129 02:58:42.850704 20630 tablet_bootstrap.cc:492] T 2e58e9a5a8aa4cecafb464addb4f35ed P 974b802141bd4f13beffef2aa03f88f9: Bootstrap starting.
I20260129 02:58:42.858963 20571 tablet_service.cc:1505] Processing CreateTablet for tablet 96609ac68bdf4993a12e1c23355ca013 (DEFAULT_TABLE table=test-workload [id=de73611ac8e3485a98aecc5add247032]), partition=RANGE (key) PARTITION 1273930945 <= VALUES < 1310328972
I20260129 02:58:42.860435 20571 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 96609ac68bdf4993a12e1c23355ca013. 1 dirs total, 0 dirs full, 0 dirs failed
I20260129 02:58:42.796228 20633 raft_consensus.cc:385] T 3e2d15f8640b407195210c476e09df91 P 197b880fa39d432a9b472845876bfaf0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260129 02:58:42.886245 20633 raft_consensus.cc:740] T 3e2d15f8640b407195210c476e09df91 P 197b880fa39d432a9b472845876bfaf0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 197b880fa39d432a9b472845876bfaf0, State: Initialized, Role: FOLLOWER
I20260129 02:58:42.887303 20633 consensus_queue.cc:260] T 3e2d15f8640b407195210c476e09df91 P 197b880fa39d432a9b472845876bfaf0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:58:42.904978 20516 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "87d5fe48a44a408aaf03e894a67ecb2f" candidate_uuid: "1c50de89a5a845668696dcb28c3badd6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "974b802141bd4f13beffef2aa03f88f9" is_pre_election: true
W20260129 02:58:42.907104 20556 leader_election.cc:343] T 87d5fe48a44a408aaf03e894a67ecb2f P 1c50de89a5a845668696dcb28c3badd6 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 974b802141bd4f13beffef2aa03f88f9 (127.18.211.2:34517): Illegal state: must be running to vote when last-logged opid is not known
I20260129 02:58:42.907711 20556 leader_election.cc:304] T 87d5fe48a44a408aaf03e894a67ecb2f P 1c50de89a5a845668696dcb28c3badd6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1c50de89a5a845668696dcb28c3badd6; no voters: 197b880fa39d432a9b472845876bfaf0, 974b802141bd4f13beffef2aa03f88f9
I20260129 02:58:42.908739 20641 raft_consensus.cc:2749] T 87d5fe48a44a408aaf03e894a67ecb2f P 1c50de89a5a845668696dcb28c3badd6 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260129 02:58:42.915843 20429 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 368953c9102643b2a21f99af36b51905. 1 dirs total, 0 dirs full, 0 dirs failed
I20260129 02:58:42.916813 20579 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 36ff33be666a4c82954731691dbcf327. 1 dirs total, 0 dirs full, 0 dirs failed
I20260129 02:58:42.931493 20575 tablet_service.cc:1505] Processing CreateTablet for tablet cb4727386a3e48aca5d71cc6c611c197 (DEFAULT_TABLE table=test-workload [id=de73611ac8e3485a98aecc5add247032]), partition=RANGE (key) PARTITION 1310328972 <= VALUES < 1346726999
I20260129 02:58:42.931947 20430 tablet_service.cc:1505] Processing CreateTablet for tablet 36ff33be666a4c82954731691dbcf327 (DEFAULT_TABLE table=test-workload [id=de73611ac8e3485a98aecc5add247032]), partition=RANGE (key) PARTITION 1201134891 <= VALUES < 1237532918
W20260129 02:58:42.933594 20480 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.211.3:37735, user_credentials={real_user=slave}} blocked reactor thread for 175448us
I20260129 02:58:42.928745 20428 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4408006eb5174de1896b037cf3f18457. 1 dirs total, 0 dirs full, 0 dirs failed
I20260129 02:58:42.941278 20427 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c2633a6fc34c44579de348d208b0c01e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260129 02:58:42.941817 20577 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c2633a6fc34c44579de348d208b0c01e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260129 02:58:42.942919 20631 tablet_bootstrap.cc:492] T a1a39dcd328e4268ae41b988c66b4a75 P 1c50de89a5a845668696dcb28c3badd6: No bootstrap required, opened a new log
I20260129 02:58:42.943396 20631 ts_tablet_manager.cc:1403] T a1a39dcd328e4268ae41b988c66b4a75 P 1c50de89a5a845668696dcb28c3badd6: Time spent bootstrapping tablet: real 0.227s user 0.005s sys 0.018s
I20260129 02:58:42.944886 20503 tablet_service.cc:1505] Processing CreateTablet for tablet e33a417cf9c54cf5af73a4ee3aac4d3a (DEFAULT_TABLE table=test-workload [id=de73611ac8e3485a98aecc5add247032]), partition=RANGE (key) PARTITION 1346726999 <= VALUES < 1383125026
I20260129 02:58:42.946156 20503 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e33a417cf9c54cf5af73a4ee3aac4d3a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260129 02:58:42.946251 20631 raft_consensus.cc:359] T a1a39dcd328e4268ae41b988c66b4a75 P 1c50de89a5a845668696dcb28c3badd6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } } peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } }
I20260129 02:58:42.947019 20631 raft_consensus.cc:385] T a1a39dcd328e4268ae41b988c66b4a75 P 1c50de89a5a845668696dcb28c3badd6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260129 02:58:42.947296 20631 raft_consensus.cc:740] T a1a39dcd328e4268ae41b988c66b4a75 P 1c50de89a5a845668696dcb28c3badd6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1c50de89a5a845668696dcb28c3badd6, State: Initialized, Role: FOLLOWER
I20260129 02:58:42.952644 20575 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cb4727386a3e48aca5d71cc6c611c197. 1 dirs total, 0 dirs full, 0 dirs failed
I20260129 02:58:42.955492 20430 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 36ff33be666a4c82954731691dbcf327. 1 dirs total, 0 dirs full, 0 dirs failed
I20260129 02:58:42.956197 20580 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7a9a69cd82724b1faed1d10aa091fc4a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260129 02:58:42.966171 20630 tablet_bootstrap.cc:654] T 2e58e9a5a8aa4cecafb464addb4f35ed P 974b802141bd4f13beffef2aa03f88f9: Neither blocks nor log segments found. Creating new log.
I20260129 02:58:42.940133 20633 ts_tablet_manager.cc:1434] T 3e2d15f8640b407195210c476e09df91 P 197b880fa39d432a9b472845876bfaf0: Time spent starting tablet: real 0.227s user 0.011s sys 0.000s
I20260129 02:58:42.968597 20641 raft_consensus.cc:493] T a5a05fbcbbcf4916bbc01fd1a5b5e90d P 1c50de89a5a845668696dcb28c3badd6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260129 02:58:42.968814 20633 tablet_bootstrap.cc:492] T 75f45538dc704d9a8ec4ff8ad9f3ea5b P 197b880fa39d432a9b472845876bfaf0: Bootstrap starting.
I20260129 02:58:42.969107 20641 raft_consensus.cc:515] T a5a05fbcbbcf4916bbc01fd1a5b5e90d P 1c50de89a5a845668696dcb28c3badd6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:58:42.971061 20641 leader_election.cc:290] T a5a05fbcbbcf4916bbc01fd1a5b5e90d P 1c50de89a5a845668696dcb28c3badd6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 197b880fa39d432a9b472845876bfaf0 (127.18.211.1:42027), 974b802141bd4f13beffef2aa03f88f9 (127.18.211.2:34517)
I20260129 02:58:42.972672 20441 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a5a05fbcbbcf4916bbc01fd1a5b5e90d" candidate_uuid: "1c50de89a5a845668696dcb28c3badd6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "197b880fa39d432a9b472845876bfaf0" is_pre_election: true
I20260129 02:58:42.974591 20516 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a5a05fbcbbcf4916bbc01fd1a5b5e90d" candidate_uuid: "1c50de89a5a845668696dcb28c3badd6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "974b802141bd4f13beffef2aa03f88f9" is_pre_election: true
W20260129 02:58:42.974794 20553 leader_election.cc:343] T a5a05fbcbbcf4916bbc01fd1a5b5e90d P 1c50de89a5a845668696dcb28c3badd6 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 197b880fa39d432a9b472845876bfaf0 (127.18.211.1:42027): Illegal state: must be running to vote when last-logged opid is not known
W20260129 02:58:42.975950 20556 leader_election.cc:343] T a5a05fbcbbcf4916bbc01fd1a5b5e90d P 1c50de89a5a845668696dcb28c3badd6 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 974b802141bd4f13beffef2aa03f88f9 (127.18.211.2:34517): Illegal state: must be running to vote when last-logged opid is not known
I20260129 02:58:42.976359 20556 leader_election.cc:304] T a5a05fbcbbcf4916bbc01fd1a5b5e90d P 1c50de89a5a845668696dcb28c3badd6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1c50de89a5a845668696dcb28c3badd6; no voters: 197b880fa39d432a9b472845876bfaf0, 974b802141bd4f13beffef2aa03f88f9
I20260129 02:58:42.978108 20641 raft_consensus.cc:2749] T a5a05fbcbbcf4916bbc01fd1a5b5e90d P 1c50de89a5a845668696dcb28c3badd6 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260129 02:58:42.980916 20501 tablet_service.cc:1505] Processing CreateTablet for tablet cb4727386a3e48aca5d71cc6c611c197 (DEFAULT_TABLE table=test-workload [id=de73611ac8e3485a98aecc5add247032]), partition=RANGE (key) PARTITION 1310328972 <= VALUES < 1346726999
I20260129 02:58:42.982246 20501 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cb4727386a3e48aca5d71cc6c611c197. 1 dirs total, 0 dirs full, 0 dirs failed
I20260129 02:58:42.999835 20653 raft_consensus.cc:493] T 4b1876f9a70b49edb3aa74105e66d3b3 P 974b802141bd4f13beffef2aa03f88f9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260129 02:58:43.000559 20653 raft_consensus.cc:515] T 4b1876f9a70b49edb3aa74105e66d3b3 P 974b802141bd4f13beffef2aa03f88f9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:58:43.006160 20426 tablet_service.cc:1505] Processing CreateTablet for tablet 96609ac68bdf4993a12e1c23355ca013 (DEFAULT_TABLE table=test-workload [id=de73611ac8e3485a98aecc5add247032]), partition=RANGE (key) PARTITION 1273930945 <= VALUES < 1310328972
I20260129 02:58:43.007849 20426 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 96609ac68bdf4993a12e1c23355ca013. 1 dirs total, 0 dirs full, 0 dirs failed
I20260129 02:58:43.012284 20441 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4b1876f9a70b49edb3aa74105e66d3b3" candidate_uuid: "974b802141bd4f13beffef2aa03f88f9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "197b880fa39d432a9b472845876bfaf0" is_pre_election: true
I20260129 02:58:43.012125 20425 tablet_service.cc:1505] Processing CreateTablet for tablet cb4727386a3e48aca5d71cc6c611c197 (DEFAULT_TABLE table=test-workload [id=de73611ac8e3485a98aecc5add247032]), partition=RANGE (key) PARTITION 1310328972 <= VALUES < 1346726999
I20260129 02:58:43.013551 20425 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cb4727386a3e48aca5d71cc6c611c197. 1 dirs total, 0 dirs full, 0 dirs failed
I20260129 02:58:43.013432 20590 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4b1876f9a70b49edb3aa74105e66d3b3" candidate_uuid: "974b802141bd4f13beffef2aa03f88f9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c50de89a5a845668696dcb28c3badd6" is_pre_election: true
W20260129 02:58:43.013669 20479 leader_election.cc:343] T 4b1876f9a70b49edb3aa74105e66d3b3 P 974b802141bd4f13beffef2aa03f88f9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 197b880fa39d432a9b472845876bfaf0 (127.18.211.1:42027): Illegal state: must be running to vote when last-logged opid is not known
I20260129 02:58:43.012394 20653 leader_election.cc:290] T 4b1876f9a70b49edb3aa74105e66d3b3 P 974b802141bd4f13beffef2aa03f88f9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 197b880fa39d432a9b472845876bfaf0 (127.18.211.1:42027), 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3:37735)
I20260129 02:58:43.014845 20571 tablet_service.cc:1505] Processing CreateTablet for tablet e33a417cf9c54cf5af73a4ee3aac4d3a (DEFAULT_TABLE table=test-workload [id=de73611ac8e3485a98aecc5add247032]), partition=RANGE (key) PARTITION 1346726999 <= VALUES < 1383125026
W20260129 02:58:43.015169 20480 leader_election.cc:343] T 4b1876f9a70b49edb3aa74105e66d3b3 P 974b802141bd4f13beffef2aa03f88f9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3:37735): Illegal state: must be running to vote when last-logged opid is not known
I20260129 02:58:43.015652 20480 leader_election.cc:304] T 4b1876f9a70b49edb3aa74105e66d3b3 P 974b802141bd4f13beffef2aa03f88f9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 974b802141bd4f13beffef2aa03f88f9; no voters: 197b880fa39d432a9b472845876bfaf0, 1c50de89a5a845668696dcb28c3badd6
I20260129 02:58:43.016121 20571 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e33a417cf9c54cf5af73a4ee3aac4d3a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260129 02:58:43.016757 20653 raft_consensus.cc:2749] T 4b1876f9a70b49edb3aa74105e66d3b3 P 974b802141bd4f13beffef2aa03f88f9 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260129 02:58:43.024025 20631 consensus_queue.cc:260] T a1a39dcd328e4268ae41b988c66b4a75 P 1c50de89a5a845668696dcb28c3badd6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } } peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } }
I20260129 02:58:43.024925 20516 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a1a39dcd328e4268ae41b988c66b4a75" candidate_uuid: "197b880fa39d432a9b472845876bfaf0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "974b802141bd4f13beffef2aa03f88f9" is_pre_election: true
I20260129 02:58:43.025583 20516 raft_consensus.cc:2468] T a1a39dcd328e4268ae41b988c66b4a75 P 974b802141bd4f13beffef2aa03f88f9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 197b880fa39d432a9b472845876bfaf0 in term 0.
I20260129 02:58:43.026739 20407 leader_election.cc:304] T a1a39dcd328e4268ae41b988c66b4a75 P 197b880fa39d432a9b472845876bfaf0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 197b880fa39d432a9b472845876bfaf0, 974b802141bd4f13beffef2aa03f88f9; no voters:
I20260129 02:58:43.027532 20638 raft_consensus.cc:2804] T a1a39dcd328e4268ae41b988c66b4a75 P 197b880fa39d432a9b472845876bfaf0 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260129 02:58:43.027904 20638 raft_consensus.cc:493] T a1a39dcd328e4268ae41b988c66b4a75 P 197b880fa39d432a9b472845876bfaf0 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260129 02:58:43.028239 20638 raft_consensus.cc:3060] T a1a39dcd328e4268ae41b988c66b4a75 P 197b880fa39d432a9b472845876bfaf0 [term 0 FOLLOWER]: Advancing to term 1
I20260129 02:58:43.029515 20631 ts_tablet_manager.cc:1434] T a1a39dcd328e4268ae41b988c66b4a75 P 1c50de89a5a845668696dcb28c3badd6: Time spent starting tablet: real 0.086s user 0.008s sys 0.003s
I20260129 02:58:43.030565 20631 tablet_bootstrap.cc:492] T 4b1876f9a70b49edb3aa74105e66d3b3 P 1c50de89a5a845668696dcb28c3badd6: Bootstrap starting.
I20260129 02:58:43.035131 20638 raft_consensus.cc:515] T a1a39dcd328e4268ae41b988c66b4a75 P 197b880fa39d432a9b472845876bfaf0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } } peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } }
I20260129 02:58:43.067710 20427 tablet_service.cc:1505] Processing CreateTablet for tablet a0fcc58d6fb6473cafbd9d34d5195269 (DEFAULT_TABLE table=test-workload [id=de73611ac8e3485a98aecc5add247032]), partition=RANGE (key) PARTITION 1383125026 <= VALUES < 1419523053
I20260129 02:58:43.069371 20427 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a0fcc58d6fb6473cafbd9d34d5195269. 1 dirs total, 0 dirs full, 0 dirs failed
I20260129 02:58:43.069677 20638 leader_election.cc:290] T a1a39dcd328e4268ae41b988c66b4a75 P 197b880fa39d432a9b472845876bfaf0 [CANDIDATE]: Term 1 election: Requested vote from peers 974b802141bd4f13beffef2aa03f88f9 (127.18.211.2:34517), 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3:37735)
I20260129 02:58:43.045548 20590 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a1a39dcd328e4268ae41b988c66b4a75" candidate_uuid: "197b880fa39d432a9b472845876bfaf0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c50de89a5a845668696dcb28c3badd6" is_pre_election: true
I20260129 02:58:43.024037 20424 tablet_service.cc:1505] Processing CreateTablet for tablet e33a417cf9c54cf5af73a4ee3aac4d3a (DEFAULT_TABLE table=test-workload [id=de73611ac8e3485a98aecc5add247032]), partition=RANGE (key) PARTITION 1346726999 <= VALUES < 1383125026
I20260129 02:58:43.070487 20590 raft_consensus.cc:2468] T a1a39dcd328e4268ae41b988c66b4a75 P 1c50de89a5a845668696dcb28c3badd6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 197b880fa39d432a9b472845876bfaf0 in term 0.
I20260129 02:58:43.071244 20516 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a1a39dcd328e4268ae41b988c66b4a75" candidate_uuid: "197b880fa39d432a9b472845876bfaf0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "974b802141bd4f13beffef2aa03f88f9"
I20260129 02:58:43.071893 20516 raft_consensus.cc:3060] T a1a39dcd328e4268ae41b988c66b4a75 P 974b802141bd4f13beffef2aa03f88f9 [term 0 FOLLOWER]: Advancing to term 1
I20260129 02:58:43.071643 20501 tablet_service.cc:1505] Processing CreateTablet for tablet 0340229d521b432580ae926f6062e3f7 (DEFAULT_TABLE table=test-workload [id=de73611ac8e3485a98aecc5add247032]), partition=RANGE (key) PARTITION 1419523053 <= VALUES < 1455921080
I20260129 02:58:43.072952 20501 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0340229d521b432580ae926f6062e3f7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260129 02:58:43.073385 20590 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a1a39dcd328e4268ae41b988c66b4a75" candidate_uuid: "197b880fa39d432a9b472845876bfaf0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c50de89a5a845668696dcb28c3badd6"
I20260129 02:58:43.073971 20590 raft_consensus.cc:3060] T a1a39dcd328e4268ae41b988c66b4a75 P 1c50de89a5a845668696dcb28c3badd6 [term 0 FOLLOWER]: Advancing to term 1
I20260129 02:58:43.073688 20431 tablet_service.cc:1505] Processing CreateTablet for tablet 0340229d521b432580ae926f6062e3f7 (DEFAULT_TABLE table=test-workload [id=de73611ac8e3485a98aecc5add247032]), partition=RANGE (key) PARTITION 1419523053 <= VALUES < 1455921080
I20260129 02:58:43.048422 20630 tablet_bootstrap.cc:492] T 2e58e9a5a8aa4cecafb464addb4f35ed P 974b802141bd4f13beffef2aa03f88f9: No bootstrap required, opened a new log
I20260129 02:58:43.076213 20630 ts_tablet_manager.cc:1403] T 2e58e9a5a8aa4cecafb464addb4f35ed P 974b802141bd4f13beffef2aa03f88f9: Time spent bootstrapping tablet: real 0.226s user 0.010s sys 0.004s
I20260129 02:58:43.077179 20503 tablet_service.cc:1505] Processing CreateTablet for tablet a0fcc58d6fb6473cafbd9d34d5195269 (DEFAULT_TABLE table=test-workload [id=de73611ac8e3485a98aecc5add247032]), partition=RANGE (key) PARTITION 1383125026 <= VALUES < 1419523053
I20260129 02:58:43.077914 20580 tablet_service.cc:1505] Processing CreateTablet for tablet 0340229d521b432580ae926f6062e3f7 (DEFAULT_TABLE table=test-workload [id=de73611ac8e3485a98aecc5add247032]), partition=RANGE (key) PARTITION 1419523053 <= VALUES < 1455921080
I20260129 02:58:43.006325 20422 tablet_service.cc:1505] Processing CreateTablet for tablet 7a9a69cd82724b1faed1d10aa091fc4a (DEFAULT_TABLE table=test-workload [id=de73611ac8e3485a98aecc5add247032]), partition=RANGE (key) PARTITION 1237532918 <= VALUES < 1273930945
I20260129 02:58:43.060083 20633 tablet_bootstrap.cc:654] T 75f45538dc704d9a8ec4ff8ad9f3ea5b P 197b880fa39d432a9b472845876bfaf0: Neither blocks nor log segments found. Creating new log.
I20260129 02:58:43.079097 20580 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0340229d521b432580ae926f6062e3f7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260129 02:58:43.096263 20571 tablet_service.cc:1505] Processing CreateTablet for tablet a0fcc58d6fb6473cafbd9d34d5195269 (DEFAULT_TABLE table=test-workload [id=de73611ac8e3485a98aecc5add247032]), partition=RANGE (key) PARTITION 1383125026 <= VALUES < 1419523053
I20260129 02:58:43.097599 20571 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a0fcc58d6fb6473cafbd9d34d5195269. 1 dirs total, 0 dirs full, 0 dirs failed
I20260129 02:58:43.079698 20424 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e33a417cf9c54cf5af73a4ee3aac4d3a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260129 02:58:43.099647 20572 tablet_service.cc:1505] Processing CreateTablet for tablet 18f30ebaa4174fd28c958e72d0cc7e24 (DEFAULT_TABLE table=test-workload [id=de73611ac8e3485a98aecc5add247032]), partition=RANGE (key) PARTITION 1528717134 <= VALUES < 1565115161
I20260129 02:58:43.100957 20572 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 18f30ebaa4174fd28c958e72d0cc7e24. 1 dirs total, 0 dirs full, 0 dirs failed
I20260129 02:58:43.079699 20500 tablet_service.cc:1505] Processing CreateTablet for tablet a644c02b16af47679824ccd060ac10fd (DEFAULT_TABLE table=test-workload [id=de73611ac8e3485a98aecc5add247032]), partition=RANGE (key) PARTITION 1455921080 <= VALUES < 1492319107
I20260129 02:58:43.081406 20575 tablet_service.cc:1505] Processing CreateTablet for tablet a644c02b16af47679824ccd060ac10fd (DEFAULT_TABLE table=test-workload [id=de73611ac8e3485a98aecc5add247032]), partition=RANGE (key) PARTITION 1455921080 <= VALUES < 1492319107
I20260129 02:58:43.103359 20575 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a644c02b16af47679824ccd060ac10fd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260129 02:58:43.104159 20422 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7a9a69cd82724b1faed1d10aa091fc4a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260129 02:58:43.105053 20423 tablet_service.cc:1505] Processing CreateTablet for tablet a644c02b16af47679824ccd060ac10fd (DEFAULT_TABLE table=test-workload [id=de73611ac8e3485a98aecc5add247032]), partition=RANGE (key) PARTITION 1455921080 <= VALUES < 1492319107
I20260129 02:58:43.105535 20431 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0340229d521b432580ae926f6062e3f7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260129 02:58:43.083611 20506 tablet_service.cc:1505] Processing CreateTablet for tablet cedce331e1584ac8b636c9b8a72b2ba0 (DEFAULT_TABLE table=test-workload [id=de73611ac8e3485a98aecc5add247032]), partition=RANGE (key) PARTITION 1492319107 <= VALUES < 1528717134
I20260129 02:58:43.085803 20516 raft_consensus.cc:2468] T a1a39dcd328e4268ae41b988c66b4a75 P 974b802141bd4f13beffef2aa03f88f9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 197b880fa39d432a9b472845876bfaf0 in term 1.
I20260129 02:58:43.139147 20423 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a644c02b16af47679824ccd060ac10fd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260129 02:58:43.078351 20503 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a0fcc58d6fb6473cafbd9d34d5195269. 1 dirs total, 0 dirs full, 0 dirs failed
I20260129 02:58:43.139954 20407 leader_election.cc:304] T a1a39dcd328e4268ae41b988c66b4a75 P 197b880fa39d432a9b472845876bfaf0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 197b880fa39d432a9b472845876bfaf0, 974b802141bd4f13beffef2aa03f88f9; no voters:
I20260129 02:58:43.140877 20638 raft_consensus.cc:2804] T a1a39dcd328e4268ae41b988c66b4a75 P 197b880fa39d432a9b472845876bfaf0 [term 1 FOLLOWER]: Leader election won for term 1
I20260129 02:58:43.141887 20500 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a644c02b16af47679824ccd060ac10fd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260129 02:58:43.114907 20504 tablet_service.cc:1505] Processing CreateTablet for tablet cb0b3f259ebc4310927338a8740e87b8 (DEFAULT_TABLE table=test-workload [id=de73611ac8e3485a98aecc5add247032]), partition=RANGE (key) PARTITION 1565115161 <= VALUES < 1601513188
I20260129 02:58:43.144639 20504 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cb0b3f259ebc4310927338a8740e87b8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260129 02:58:43.087288 20579 tablet_service.cc:1505] Processing CreateTablet for tablet cedce331e1584ac8b636c9b8a72b2ba0 (DEFAULT_TABLE table=test-workload [id=de73611ac8e3485a98aecc5add247032]), partition=RANGE (key) PARTITION 1492319107 <= VALUES < 1528717134
I20260129 02:58:43.118314 20633 tablet_bootstrap.cc:492] T 75f45538dc704d9a8ec4ff8ad9f3ea5b P 197b880fa39d432a9b472845876bfaf0: No bootstrap required, opened a new log
I20260129 02:58:43.095765 20498 tablet_service.cc:1505] Processing CreateTablet for tablet 18f30ebaa4174fd28c958e72d0cc7e24 (DEFAULT_TABLE table=test-workload [id=de73611ac8e3485a98aecc5add247032]), partition=RANGE (key) PARTITION 1528717134 <= VALUES < 1565115161
I20260129 02:58:43.079061 20630 raft_consensus.cc:359] T 2e58e9a5a8aa4cecafb464addb4f35ed P 974b802141bd4f13beffef2aa03f88f9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:58:43.148494 20506 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cedce331e1584ac8b636c9b8a72b2ba0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260129 02:58:43.148377 20578 tablet_service.cc:1505] Processing CreateTablet for tablet 926a250fa5834e88b96ccecbefbf5086 (DEFAULT_TABLE table=test-workload [id=de73611ac8e3485a98aecc5add247032]), partition=RANGE (key) PARTITION 1601513188 <= VALUES < 1637911215
I20260129 02:58:43.149439 20498 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 18f30ebaa4174fd28c958e72d0cc7e24. 1 dirs total, 0 dirs full, 0 dirs failed
I20260129 02:58:43.149698 20578 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 926a250fa5834e88b96ccecbefbf5086. 1 dirs total, 0 dirs full, 0 dirs failed
I20260129 02:58:43.123020 20573 tablet_service.cc:1505] Processing CreateTablet for tablet cb0b3f259ebc4310927338a8740e87b8 (DEFAULT_TABLE table=test-workload [id=de73611ac8e3485a98aecc5add247032]), partition=RANGE (key) PARTITION 1565115161 <= VALUES < 1601513188
I20260129 02:58:43.151949 20579 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cedce331e1584ac8b636c9b8a72b2ba0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260129 02:58:43.152328 20633 ts_tablet_manager.cc:1403] T 75f45538dc704d9a8ec4ff8ad9f3ea5b P 197b880fa39d432a9b472845876bfaf0: Time spent bootstrapping tablet: real 0.185s user 0.006s sys 0.009s
I20260129 02:58:43.152609 20573 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cb0b3f259ebc4310927338a8740e87b8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260129 02:58:43.126076 20499 tablet_service.cc:1505] Processing CreateTablet for tablet 926a250fa5834e88b96ccecbefbf5086 (DEFAULT_TABLE table=test-workload [id=de73611ac8e3485a98aecc5add247032]), partition=RANGE (key) PARTITION 1601513188 <= VALUES < 1637911215
I20260129 02:58:43.155308 20499 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 926a250fa5834e88b96ccecbefbf5086. 1 dirs total, 0 dirs full, 0 dirs failed
I20260129 02:58:43.162206 20633 raft_consensus.cc:359] T 75f45538dc704d9a8ec4ff8ad9f3ea5b P 197b880fa39d432a9b472845876bfaf0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:58:43.168303 20633 raft_consensus.cc:385] T 75f45538dc704d9a8ec4ff8ad9f3ea5b P 197b880fa39d432a9b472845876bfaf0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260129 02:58:43.169078 20633 raft_consensus.cc:740] T 75f45538dc704d9a8ec4ff8ad9f3ea5b P 197b880fa39d432a9b472845876bfaf0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 197b880fa39d432a9b472845876bfaf0, State: Initialized, Role: FOLLOWER
I20260129 02:58:43.086776 20590 raft_consensus.cc:2468] T a1a39dcd328e4268ae41b988c66b4a75 P 1c50de89a5a845668696dcb28c3badd6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 197b880fa39d432a9b472845876bfaf0 in term 1.
I20260129 02:58:43.148783 20630 raft_consensus.cc:385] T 2e58e9a5a8aa4cecafb464addb4f35ed P 974b802141bd4f13beffef2aa03f88f9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260129 02:58:43.172071 20630 raft_consensus.cc:740] T 2e58e9a5a8aa4cecafb464addb4f35ed P 974b802141bd4f13beffef2aa03f88f9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 974b802141bd4f13beffef2aa03f88f9, State: Initialized, Role: FOLLOWER
I20260129 02:58:43.172827 20630 consensus_queue.cc:260] T 2e58e9a5a8aa4cecafb464addb4f35ed P 974b802141bd4f13beffef2aa03f88f9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:58:43.175230 20630 ts_tablet_manager.cc:1434] T 2e58e9a5a8aa4cecafb464addb4f35ed P 974b802141bd4f13beffef2aa03f88f9: Time spent starting tablet: real 0.099s user 0.005s sys 0.001s
I20260129 02:58:43.138278 20631 tablet_bootstrap.cc:654] T 4b1876f9a70b49edb3aa74105e66d3b3 P 1c50de89a5a845668696dcb28c3badd6: Neither blocks nor log segments found. Creating new log.
I20260129 02:58:43.179172 20630 tablet_bootstrap.cc:492] T 87d5fe48a44a408aaf03e894a67ecb2f P 974b802141bd4f13beffef2aa03f88f9: Bootstrap starting.
I20260129 02:58:43.137717 20428 tablet_service.cc:1505] Processing CreateTablet for tablet cedce331e1584ac8b636c9b8a72b2ba0 (DEFAULT_TABLE table=test-workload [id=de73611ac8e3485a98aecc5add247032]), partition=RANGE (key) PARTITION 1492319107 <= VALUES < 1528717134
I20260129 02:58:43.180619 20428 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cedce331e1584ac8b636c9b8a72b2ba0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260129 02:58:43.187965 20426 tablet_service.cc:1505] Processing CreateTablet for tablet cb0b3f259ebc4310927338a8740e87b8 (DEFAULT_TABLE table=test-workload [id=de73611ac8e3485a98aecc5add247032]), partition=RANGE (key) PARTITION 1565115161 <= VALUES < 1601513188
I20260129 02:58:43.189271 20426 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cb0b3f259ebc4310927338a8740e87b8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260129 02:58:43.192898 20429 tablet_service.cc:1505] Processing CreateTablet for tablet 18f30ebaa4174fd28c958e72d0cc7e24 (DEFAULT_TABLE table=test-workload [id=de73611ac8e3485a98aecc5add247032]), partition=RANGE (key) PARTITION 1528717134 <= VALUES < 1565115161
I20260129 02:58:43.192920 20425 tablet_service.cc:1505] Processing CreateTablet for tablet 926a250fa5834e88b96ccecbefbf5086 (DEFAULT_TABLE table=test-workload [id=de73611ac8e3485a98aecc5add247032]), partition=RANGE (key) PARTITION 1601513188 <= VALUES < 1637911215
I20260129 02:58:43.196885 20425 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 926a250fa5834e88b96ccecbefbf5086. 1 dirs total, 0 dirs full, 0 dirs failed
I20260129 02:58:43.202090 20423 tablet_service.cc:1505] Processing CreateTablet for tablet 42d27a28d89b409e8807ec0efd97af69 (DEFAULT_TABLE table=test-workload [id=de73611ac8e3485a98aecc5add247032]), partition=RANGE (key) PARTITION 1637911215 <= VALUES < 1674309242
I20260129 02:58:43.203413 20423 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 42d27a28d89b409e8807ec0efd97af69. 1 dirs total, 0 dirs full, 0 dirs failed
I20260129 02:58:43.207023 20505 tablet_service.cc:1505] Processing CreateTablet for tablet 42d27a28d89b409e8807ec0efd97af69 (DEFAULT_TABLE table=test-workload [id=de73611ac8e3485a98aecc5add247032]), partition=RANGE (key) PARTITION 1637911215 <= VALUES < 1674309242
I20260129 02:58:43.189097 20633 consensus_queue.cc:260] T 75f45538dc704d9a8ec4ff8ad9f3ea5b P 197b880fa39d432a9b472845876bfaf0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:58:43.208376 20505 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 42d27a28d89b409e8807ec0efd97af69. 1 dirs total, 0 dirs full, 0 dirs failed
I20260129 02:58:43.212354 20429 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 18f30ebaa4174fd28c958e72d0cc7e24. 1 dirs total, 0 dirs full, 0 dirs failed
I20260129 02:58:43.221493 20630 tablet_bootstrap.cc:654] T 87d5fe48a44a408aaf03e894a67ecb2f P 974b802141bd4f13beffef2aa03f88f9: Neither blocks nor log segments found. Creating new log.
I20260129 02:58:43.211944 20638 raft_consensus.cc:697] T a1a39dcd328e4268ae41b988c66b4a75 P 197b880fa39d432a9b472845876bfaf0 [term 1 LEADER]: Becoming Leader. State: Replica: 197b880fa39d432a9b472845876bfaf0, State: Running, Role: LEADER
I20260129 02:58:43.233457 20502 tablet_service.cc:1505] Processing CreateTablet for tablet 0678b8b468bd43cdbc3e485900844821 (DEFAULT_TABLE table=test-workload [id=de73611ac8e3485a98aecc5add247032]), partition=RANGE (key) PARTITION 1674309242 <= VALUES < 1710707269
I20260129 02:58:43.234972 20502 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0678b8b468bd43cdbc3e485900844821. 1 dirs total, 0 dirs full, 0 dirs failed
I20260129 02:58:43.236655 20575 tablet_service.cc:1505] Processing CreateTablet for tablet 42d27a28d89b409e8807ec0efd97af69 (DEFAULT_TABLE table=test-workload [id=de73611ac8e3485a98aecc5add247032]), partition=RANGE (key) PARTITION 1637911215 <= VALUES < 1674309242
I20260129 02:58:43.237951 20575 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 42d27a28d89b409e8807ec0efd97af69. 1 dirs total, 0 dirs full, 0 dirs failed
I20260129 02:58:43.225807 20431 tablet_service.cc:1505] Processing CreateTablet for tablet 0678b8b468bd43cdbc3e485900844821 (DEFAULT_TABLE table=test-workload [id=de73611ac8e3485a98aecc5add247032]), partition=RANGE (key) PARTITION 1674309242 <= VALUES < 1710707269
I20260129 02:58:43.240926 20431 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0678b8b468bd43cdbc3e485900844821. 1 dirs total, 0 dirs full, 0 dirs failed
I20260129 02:58:43.234975 20638 consensus_queue.cc:237] T a1a39dcd328e4268ae41b988c66b4a75 P 197b880fa39d432a9b472845876bfaf0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } } peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } }
I20260129 02:58:43.249400 20631 tablet_bootstrap.cc:492] T 4b1876f9a70b49edb3aa74105e66d3b3 P 1c50de89a5a845668696dcb28c3badd6: No bootstrap required, opened a new log
I20260129 02:58:43.249904 20631 ts_tablet_manager.cc:1403] T 4b1876f9a70b49edb3aa74105e66d3b3 P 1c50de89a5a845668696dcb28c3badd6: Time spent bootstrapping tablet: real 0.220s user 0.011s sys 0.002s
I20260129 02:58:43.251776 20658 raft_consensus.cc:493] T 3e2d15f8640b407195210c476e09df91 P 197b880fa39d432a9b472845876bfaf0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260129 02:58:43.252278 20658 raft_consensus.cc:515] T 3e2d15f8640b407195210c476e09df91 P 197b880fa39d432a9b472845876bfaf0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:58:43.253293 20572 tablet_service.cc:1505] Processing CreateTablet for tablet 0678b8b468bd43cdbc3e485900844821 (DEFAULT_TABLE table=test-workload [id=de73611ac8e3485a98aecc5add247032]), partition=RANGE (key) PARTITION 1674309242 <= VALUES < 1710707269
I20260129 02:58:43.259150 20516 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3e2d15f8640b407195210c476e09df91" candidate_uuid: "197b880fa39d432a9b472845876bfaf0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "974b802141bd4f13beffef2aa03f88f9" is_pre_election: true
W20260129 02:58:43.260974 20407 leader_election.cc:343] T 3e2d15f8640b407195210c476e09df91 P 197b880fa39d432a9b472845876bfaf0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 974b802141bd4f13beffef2aa03f88f9 (127.18.211.2:34517): Illegal state: must be running to vote when last-logged opid is not known
I20260129 02:58:43.262687 20499 tablet_service.cc:1505] Processing CreateTablet for tablet c934d31efd3f4e5195605ddc354d14e1 (DEFAULT_TABLE table=test-workload [id=de73611ac8e3485a98aecc5add247032]), partition=RANGE (key) PARTITION 1710707269 <= VALUES < 1747105296
I20260129 02:58:43.266366 20499 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c934d31efd3f4e5195605ddc354d14e1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260129 02:58:43.254289 20658 leader_election.cc:290] T 3e2d15f8640b407195210c476e09df91 P 197b880fa39d432a9b472845876bfaf0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 974b802141bd4f13beffef2aa03f88f9 (127.18.211.2:34517), 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3:37735)
I20260129 02:58:43.288132 20590 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3e2d15f8640b407195210c476e09df91" candidate_uuid: "197b880fa39d432a9b472845876bfaf0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c50de89a5a845668696dcb28c3badd6" is_pre_election: true
W20260129 02:58:43.300484 20405 leader_election.cc:343] T 3e2d15f8640b407195210c476e09df91 P 197b880fa39d432a9b472845876bfaf0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3:37735): Illegal state: must be running to vote when last-logged opid is not known
I20260129 02:58:43.300309 20573 tablet_service.cc:1505] Processing CreateTablet for tablet c934d31efd3f4e5195605ddc354d14e1 (DEFAULT_TABLE table=test-workload [id=de73611ac8e3485a98aecc5add247032]), partition=RANGE (key) PARTITION 1710707269 <= VALUES < 1747105296
I20260129 02:58:43.300951 20405 leader_election.cc:304] T 3e2d15f8640b407195210c476e09df91 P 197b880fa39d432a9b472845876bfaf0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 197b880fa39d432a9b472845876bfaf0; no voters: 1c50de89a5a845668696dcb28c3badd6, 974b802141bd4f13beffef2aa03f88f9
I20260129 02:58:43.301754 20658 raft_consensus.cc:2749] T 3e2d15f8640b407195210c476e09df91 P 197b880fa39d432a9b472845876bfaf0 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260129 02:58:43.308672 20424 tablet_service.cc:1505] Processing CreateTablet for tablet c891d2bff4ce4568861bbc992e91fbb0 (DEFAULT_TABLE table=test-workload [id=de73611ac8e3485a98aecc5add247032]), partition=RANGE (key) PARTITION 1747105296 <= VALUES < 1783503323
I20260129 02:58:43.308888 20504 tablet_service.cc:1505] Processing CreateTablet for tablet c891d2bff4ce4568861bbc992e91fbb0 (DEFAULT_TABLE table=test-workload [id=de73611ac8e3485a98aecc5add247032]), partition=RANGE (key) PARTITION 1747105296 <= VALUES < 1783503323
I20260129 02:58:43.310225 20504 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c891d2bff4ce4568861bbc992e91fbb0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260129 02:58:43.310176 20424 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c891d2bff4ce4568861bbc992e91fbb0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260129 02:58:43.318421 20572 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0678b8b468bd43cdbc3e485900844821. 1 dirs total, 0 dirs full, 0 dirs failed
I20260129 02:58:43.321482 20579 tablet_service.cc:1505] Processing CreateTablet for tablet c891d2bff4ce4568861bbc992e91fbb0 (DEFAULT_TABLE table=test-workload [id=de73611ac8e3485a98aecc5add247032]), partition=RANGE (key) PARTITION 1747105296 <= VALUES < 1783503323
I20260129 02:58:43.322819 20579 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c891d2bff4ce4568861bbc992e91fbb0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260129 02:58:43.323782 20573 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c934d31efd3f4e5195605ddc354d14e1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260129 02:58:43.323176 20337 catalog_manager.cc:5654] T a1a39dcd328e4268ae41b988c66b4a75 P 197b880fa39d432a9b472845876bfaf0 reported cstate change: term changed from 0 to 1, leader changed from <none> to 197b880fa39d432a9b472845876bfaf0 (127.18.211.1). New cstate: current_term: 1 leader_uuid: "197b880fa39d432a9b472845876bfaf0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } health_report { overall_health: HEALTHY } } }
I20260129 02:58:43.324440 20631 raft_consensus.cc:359] T 4b1876f9a70b49edb3aa74105e66d3b3 P 1c50de89a5a845668696dcb28c3badd6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:58:43.325181 20631 raft_consensus.cc:385] T 4b1876f9a70b49edb3aa74105e66d3b3 P 1c50de89a5a845668696dcb28c3badd6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260129 02:58:43.325476 20631 raft_consensus.cc:740] T 4b1876f9a70b49edb3aa74105e66d3b3 P 1c50de89a5a845668696dcb28c3badd6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1c50de89a5a845668696dcb28c3badd6, State: Initialized, Role: FOLLOWER
I20260129 02:58:43.326285 20631 consensus_queue.cc:260] T 4b1876f9a70b49edb3aa74105e66d3b3 P 1c50de89a5a845668696dcb28c3badd6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:58:43.320453 20633 ts_tablet_manager.cc:1434] T 75f45538dc704d9a8ec4ff8ad9f3ea5b P 197b880fa39d432a9b472845876bfaf0: Time spent starting tablet: real 0.168s user 0.011s sys 0.000s
I20260129 02:58:43.330204 20633 tablet_bootstrap.cc:492] T 87d5fe48a44a408aaf03e894a67ecb2f P 197b880fa39d432a9b472845876bfaf0: Bootstrap starting.
I20260129 02:58:43.331597 20631 ts_tablet_manager.cc:1434] T 4b1876f9a70b49edb3aa74105e66d3b3 P 1c50de89a5a845668696dcb28c3badd6: Time spent starting tablet: real 0.081s user 0.008s sys 0.000s
I20260129 02:58:43.333422 20426 tablet_service.cc:1505] Processing CreateTablet for tablet 75be4bd6bd45474795fd472eff2953ac (DEFAULT_TABLE table=test-workload [id=de73611ac8e3485a98aecc5add247032]), partition=RANGE (key) PARTITION 1783503323 <= VALUES < 1819901350
I20260129 02:58:43.337361 20503 tablet_service.cc:1505] Processing CreateTablet for tablet 75be4bd6bd45474795fd472eff2953ac (DEFAULT_TABLE table=test-workload [id=de73611ac8e3485a98aecc5add247032]), partition=RANGE (key) PARTITION 1783503323 <= VALUES < 1819901350
I20260129 02:58:43.340421 20630 tablet_bootstrap.cc:492] T 87d5fe48a44a408aaf03e894a67ecb2f P 974b802141bd4f13beffef2aa03f88f9: No bootstrap required, opened a new log
I20260129 02:58:43.352212 20630 ts_tablet_manager.cc:1403] T 87d5fe48a44a408aaf03e894a67ecb2f P 974b802141bd4f13beffef2aa03f88f9: Time spent bootstrapping tablet: real 0.173s user 0.022s sys 0.029s
I20260129 02:58:43.351949 20505 tablet_service.cc:1505] Processing CreateTablet for tablet 8055232cf0004d40a85f3bf983fe00a9 (DEFAULT_TABLE table=test-workload [id=de73611ac8e3485a98aecc5add247032]), partition=RANGE (key) PARTITION 1819901350 <= VALUES < 1856299377
I20260129 02:58:43.340744 20578 tablet_service.cc:1505] Processing CreateTablet for tablet 75be4bd6bd45474795fd472eff2953ac (DEFAULT_TABLE table=test-workload [id=de73611ac8e3485a98aecc5add247032]), partition=RANGE (key) PARTITION 1783503323 <= VALUES < 1819901350
I20260129 02:58:43.345820 20429 tablet_service.cc:1505] Processing CreateTablet for tablet 8055232cf0004d40a85f3bf983fe00a9 (DEFAULT_TABLE table=test-workload [id=de73611ac8e3485a98aecc5add247032]), partition=RANGE (key) PARTITION 1819901350 <= VALUES < 1856299377
I20260129 02:58:43.254289 20422 tablet_service.cc:1505] Processing CreateTablet for tablet c934d31efd3f4e5195605ddc354d14e1 (DEFAULT_TABLE table=test-workload [id=de73611ac8e3485a98aecc5add247032]), partition=RANGE (key) PARTITION 1710707269 <= VALUES < 1747105296
I20260129 02:58:43.349777 20503 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 75be4bd6bd45474795fd472eff2953ac. 1 dirs total, 0 dirs full, 0 dirs failed
I20260129 02:58:43.353835 20631 tablet_bootstrap.cc:492] T 1299723a4476420ab02fd8bfd7aa0645 P 1c50de89a5a845668696dcb28c3badd6: Bootstrap starting.
I20260129 02:58:43.354185 20578 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 75be4bd6bd45474795fd472eff2953ac. 1 dirs total, 0 dirs full, 0 dirs failed
I20260129 02:58:43.375430 20633 tablet_bootstrap.cc:654] T 87d5fe48a44a408aaf03e894a67ecb2f P 197b880fa39d432a9b472845876bfaf0: Neither blocks nor log segments found. Creating new log.
I20260129 02:58:43.376137 20505 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8055232cf0004d40a85f3bf983fe00a9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260129 02:58:43.380179 20499 tablet_service.cc:1505] Processing CreateTablet for tablet 00e79d8f25a34676b1d866eb2068a2b2 (DEFAULT_TABLE table=test-workload [id=de73611ac8e3485a98aecc5add247032]), partition=RANGE (key) PARTITION 1856299377 <= VALUES < 1892697404
I20260129 02:58:43.382210 20653 raft_consensus.cc:493] T 2e58e9a5a8aa4cecafb464addb4f35ed P 974b802141bd4f13beffef2aa03f88f9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260129 02:58:43.382695 20653 raft_consensus.cc:515] T 2e58e9a5a8aa4cecafb464addb4f35ed P 974b802141bd4f13beffef2aa03f88f9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:58:43.384632 20653 leader_election.cc:290] T 2e58e9a5a8aa4cecafb464addb4f35ed P 974b802141bd4f13beffef2aa03f88f9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 197b880fa39d432a9b472845876bfaf0 (127.18.211.1:42027), 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3:37735)
I20260129 02:58:43.385790 20590 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2e58e9a5a8aa4cecafb464addb4f35ed" candidate_uuid: "974b802141bd4f13beffef2aa03f88f9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c50de89a5a845668696dcb28c3badd6" is_pre_election: true
I20260129 02:58:43.354581 20426 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 75be4bd6bd45474795fd472eff2953ac. 1 dirs total, 0 dirs full, 0 dirs failed
I20260129 02:58:43.355357 20630 raft_consensus.cc:359] T 87d5fe48a44a408aaf03e894a67ecb2f P 974b802141bd4f13beffef2aa03f88f9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:58:43.386117 20441 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2e58e9a5a8aa4cecafb464addb4f35ed" candidate_uuid: "974b802141bd4f13beffef2aa03f88f9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "197b880fa39d432a9b472845876bfaf0" is_pre_election: true
I20260129 02:58:43.386521 20630 raft_consensus.cc:385] T 87d5fe48a44a408aaf03e894a67ecb2f P 974b802141bd4f13beffef2aa03f88f9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260129 02:58:43.386871 20630 raft_consensus.cc:740] T 87d5fe48a44a408aaf03e894a67ecb2f P 974b802141bd4f13beffef2aa03f88f9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 974b802141bd4f13beffef2aa03f88f9, State: Initialized, Role: FOLLOWER
I20260129 02:58:43.381762 20499 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00e79d8f25a34676b1d866eb2068a2b2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260129 02:58:43.365811 20425 tablet_service.cc:1505] Processing CreateTablet for tablet 00e79d8f25a34676b1d866eb2068a2b2 (DEFAULT_TABLE table=test-workload [id=de73611ac8e3485a98aecc5add247032]), partition=RANGE (key) PARTITION 1856299377 <= VALUES < 1892697404
W20260129 02:58:43.369576 20387 auto_rebalancer.cc:229] Could not retrieve cluster info: Service unavailable: Tablet not running
W20260129 02:58:43.387168 20480 leader_election.cc:343] T 2e58e9a5a8aa4cecafb464addb4f35ed P 974b802141bd4f13beffef2aa03f88f9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3:37735): Illegal state: must be running to vote when last-logged opid is not known
I20260129 02:58:43.359295 20571 tablet_service.cc:1505] Processing CreateTablet for tablet 8055232cf0004d40a85f3bf983fe00a9 (DEFAULT_TABLE table=test-workload [id=de73611ac8e3485a98aecc5add247032]), partition=RANGE (key) PARTITION 1819901350 <= VALUES < 1856299377
W20260129 02:58:43.388484 20479 leader_election.cc:343] T 2e58e9a5a8aa4cecafb464addb4f35ed P 974b802141bd4f13beffef2aa03f88f9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 197b880fa39d432a9b472845876bfaf0 (127.18.211.1:42027): Illegal state: must be running to vote when last-logged opid is not known
I20260129 02:58:43.389150 20571 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8055232cf0004d40a85f3bf983fe00a9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260129 02:58:43.390702 20479 leader_election.cc:304] T 2e58e9a5a8aa4cecafb464addb4f35ed P 974b802141bd4f13beffef2aa03f88f9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 974b802141bd4f13beffef2aa03f88f9; no voters: 197b880fa39d432a9b472845876bfaf0, 1c50de89a5a845668696dcb28c3badd6
I20260129 02:58:43.393079 20653 raft_consensus.cc:2749] T 2e58e9a5a8aa4cecafb464addb4f35ed P 974b802141bd4f13beffef2aa03f88f9 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260129 02:58:43.387552 20630 consensus_queue.cc:260] T 87d5fe48a44a408aaf03e894a67ecb2f P 974b802141bd4f13beffef2aa03f88f9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:58:43.395355 20630 ts_tablet_manager.cc:1434] T 87d5fe48a44a408aaf03e894a67ecb2f P 974b802141bd4f13beffef2aa03f88f9: Time spent starting tablet: real 0.043s user 0.000s sys 0.005s
I20260129 02:58:43.396346 20630 tablet_bootstrap.cc:492] T 0f20af7fbe5d4db89a6f17948d98912c P 974b802141bd4f13beffef2aa03f88f9: Bootstrap starting.
I20260129 02:58:43.403601 20423 tablet_service.cc:1505] Processing CreateTablet for tablet 3ccac82a01a2445fa5b0ce6c0fcf2295 (DEFAULT_TABLE table=test-workload [id=de73611ac8e3485a98aecc5add247032]), partition=RANGE (key) PARTITION 1892697404 <= VALUES < 1929095431
I20260129 02:58:43.404157 20504 tablet_service.cc:1505] Processing CreateTablet for tablet 3ccac82a01a2445fa5b0ce6c0fcf2295 (DEFAULT_TABLE table=test-workload [id=de73611ac8e3485a98aecc5add247032]), partition=RANGE (key) PARTITION 1892697404 <= VALUES < 1929095431
I20260129 02:58:43.406884 20422 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c934d31efd3f4e5195605ddc354d14e1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260129 02:58:43.407574 20423 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3ccac82a01a2445fa5b0ce6c0fcf2295. 1 dirs total, 0 dirs full, 0 dirs failed
I20260129 02:58:43.404932 20573 tablet_service.cc:1505] Processing CreateTablet for tablet 00e79d8f25a34676b1d866eb2068a2b2 (DEFAULT_TABLE table=test-workload [id=de73611ac8e3485a98aecc5add247032]), partition=RANGE (key) PARTITION 1856299377 <= VALUES < 1892697404
I20260129 02:58:43.409322 20573 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00e79d8f25a34676b1d866eb2068a2b2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260129 02:58:43.410125 20425 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00e79d8f25a34676b1d866eb2068a2b2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260129 02:58:43.415264 20429 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8055232cf0004d40a85f3bf983fe00a9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260129 02:58:43.415710 20504 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3ccac82a01a2445fa5b0ce6c0fcf2295. 1 dirs total, 0 dirs full, 0 dirs failed
I20260129 02:58:43.429170 20579 tablet_service.cc:1505] Processing CreateTablet for tablet 3ccac82a01a2445fa5b0ce6c0fcf2295 (DEFAULT_TABLE table=test-workload [id=de73611ac8e3485a98aecc5add247032]), partition=RANGE (key) PARTITION 1892697404 <= VALUES < 1929095431
I20260129 02:58:43.430534 20579 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3ccac82a01a2445fa5b0ce6c0fcf2295. 1 dirs total, 0 dirs full, 0 dirs failed
I20260129 02:58:43.434964 20630 tablet_bootstrap.cc:654] T 0f20af7fbe5d4db89a6f17948d98912c P 974b802141bd4f13beffef2aa03f88f9: Neither blocks nor log segments found. Creating new log.
I20260129 02:58:43.445127 20500 tablet_service.cc:1505] Processing CreateTablet for tablet 4e3805ffa70242e8ade66da260dddf66 (DEFAULT_TABLE table=test-workload [id=de73611ac8e3485a98aecc5add247032]), partition=RANGE (key) PARTITION 1929095431 <= VALUES < 1965493458
I20260129 02:58:43.446489 20500 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4e3805ffa70242e8ade66da260dddf66. 1 dirs total, 0 dirs full, 0 dirs failed
I20260129 02:58:43.449322 20431 tablet_service.cc:1505] Processing CreateTablet for tablet 4e3805ffa70242e8ade66da260dddf66 (DEFAULT_TABLE table=test-workload [id=de73611ac8e3485a98aecc5add247032]), partition=RANGE (key) PARTITION 1929095431 <= VALUES < 1965493458
I20260129 02:58:43.450093 20630 tablet_bootstrap.cc:492] T 0f20af7fbe5d4db89a6f17948d98912c P 974b802141bd4f13beffef2aa03f88f9: No bootstrap required, opened a new log
I20260129 02:58:43.450572 20630 ts_tablet_manager.cc:1403] T 0f20af7fbe5d4db89a6f17948d98912c P 974b802141bd4f13beffef2aa03f88f9: Time spent bootstrapping tablet: real 0.054s user 0.006s sys 0.004s
I20260129 02:58:43.450623 20431 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4e3805ffa70242e8ade66da260dddf66. 1 dirs total, 0 dirs full, 0 dirs failed
I20260129 02:58:43.462714 20424 tablet_service.cc:1505] Processing CreateTablet for tablet 7d9494f5584d4d5db8b9adf1c8fc888b (DEFAULT_TABLE table=test-workload [id=de73611ac8e3485a98aecc5add247032]), partition=RANGE (key) PARTITION 1965493458 <= VALUES < 2001891485
I20260129 02:58:43.464123 20424 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7d9494f5584d4d5db8b9adf1c8fc888b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260129 02:58:43.464855 20575 tablet_service.cc:1505] Processing CreateTablet for tablet 4e3805ffa70242e8ade66da260dddf66 (DEFAULT_TABLE table=test-workload [id=de73611ac8e3485a98aecc5add247032]), partition=RANGE (key) PARTITION 1929095431 <= VALUES < 1965493458
I20260129 02:58:43.465054 20630 raft_consensus.cc:359] T 0f20af7fbe5d4db89a6f17948d98912c P 974b802141bd4f13beffef2aa03f88f9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:58:43.465775 20630 raft_consensus.cc:385] T 0f20af7fbe5d4db89a6f17948d98912c P 974b802141bd4f13beffef2aa03f88f9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260129 02:58:43.466135 20575 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4e3805ffa70242e8ade66da260dddf66. 1 dirs total, 0 dirs full, 0 dirs failed
I20260129 02:58:43.466094 20630 raft_consensus.cc:740] T 0f20af7fbe5d4db89a6f17948d98912c P 974b802141bd4f13beffef2aa03f88f9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 974b802141bd4f13beffef2aa03f88f9, State: Initialized, Role: FOLLOWER
I20260129 02:58:43.466883 20630 consensus_queue.cc:260] T 0f20af7fbe5d4db89a6f17948d98912c P 974b802141bd4f13beffef2aa03f88f9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:58:43.472092 20653 raft_consensus.cc:493] T 87d5fe48a44a408aaf03e894a67ecb2f P 974b802141bd4f13beffef2aa03f88f9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260129 02:58:43.472556 20653 raft_consensus.cc:515] T 87d5fe48a44a408aaf03e894a67ecb2f P 974b802141bd4f13beffef2aa03f88f9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:58:43.474486 20653 leader_election.cc:290] T 87d5fe48a44a408aaf03e894a67ecb2f P 974b802141bd4f13beffef2aa03f88f9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 197b880fa39d432a9b472845876bfaf0 (127.18.211.1:42027), 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3:37735)
I20260129 02:58:43.476016 20590 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "87d5fe48a44a408aaf03e894a67ecb2f" candidate_uuid: "974b802141bd4f13beffef2aa03f88f9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c50de89a5a845668696dcb28c3badd6" is_pre_election: true
I20260129 02:58:43.476662 20590 raft_consensus.cc:2468] T 87d5fe48a44a408aaf03e894a67ecb2f P 1c50de89a5a845668696dcb28c3badd6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 974b802141bd4f13beffef2aa03f88f9 in term 0.
I20260129 02:58:43.476817 20441 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "87d5fe48a44a408aaf03e894a67ecb2f" candidate_uuid: "974b802141bd4f13beffef2aa03f88f9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "197b880fa39d432a9b472845876bfaf0" is_pre_election: true
I20260129 02:58:43.477877 20480 leader_election.cc:304] T 87d5fe48a44a408aaf03e894a67ecb2f P 974b802141bd4f13beffef2aa03f88f9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1c50de89a5a845668696dcb28c3badd6, 974b802141bd4f13beffef2aa03f88f9; no voters:
W20260129 02:58:43.478649 20479 leader_election.cc:343] T 87d5fe48a44a408aaf03e894a67ecb2f P 974b802141bd4f13beffef2aa03f88f9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 197b880fa39d432a9b472845876bfaf0 (127.18.211.1:42027): Illegal state: must be running to vote when last-logged opid is not known
I20260129 02:58:43.479563 20653 raft_consensus.cc:2804] T 87d5fe48a44a408aaf03e894a67ecb2f P 974b802141bd4f13beffef2aa03f88f9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260129 02:58:43.479955 20653 raft_consensus.cc:493] T 87d5fe48a44a408aaf03e894a67ecb2f P 974b802141bd4f13beffef2aa03f88f9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260129 02:58:43.480283 20653 raft_consensus.cc:3060] T 87d5fe48a44a408aaf03e894a67ecb2f P 974b802141bd4f13beffef2aa03f88f9 [term 0 FOLLOWER]: Advancing to term 1
I20260129 02:58:43.482141 20630 ts_tablet_manager.cc:1434] T 0f20af7fbe5d4db89a6f17948d98912c P 974b802141bd4f13beffef2aa03f88f9: Time spent starting tablet: real 0.031s user 0.006s sys 0.002s
I20260129 02:58:43.482663 20504 tablet_service.cc:1505] Processing CreateTablet for tablet 7d9494f5584d4d5db8b9adf1c8fc888b (DEFAULT_TABLE table=test-workload [id=de73611ac8e3485a98aecc5add247032]), partition=RANGE (key) PARTITION 1965493458 <= VALUES < 2001891485
I20260129 02:58:43.483841 20504 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7d9494f5584d4d5db8b9adf1c8fc888b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260129 02:58:43.483163 20630 tablet_bootstrap.cc:492] T 1299723a4476420ab02fd8bfd7aa0645 P 974b802141bd4f13beffef2aa03f88f9: Bootstrap starting.
I20260129 02:58:43.490089 20633 tablet_bootstrap.cc:492] T 87d5fe48a44a408aaf03e894a67ecb2f P 197b880fa39d432a9b472845876bfaf0: No bootstrap required, opened a new log
I20260129 02:58:43.490603 20633 ts_tablet_manager.cc:1403] T 87d5fe48a44a408aaf03e894a67ecb2f P 197b880fa39d432a9b472845876bfaf0: Time spent bootstrapping tablet: real 0.161s user 0.012s sys 0.013s
I20260129 02:58:43.491037 20578 tablet_service.cc:1505] Processing CreateTablet for tablet 7d9494f5584d4d5db8b9adf1c8fc888b (DEFAULT_TABLE table=test-workload [id=de73611ac8e3485a98aecc5add247032]), partition=RANGE (key) PARTITION 1965493458 <= VALUES < 2001891485
I20260129 02:58:43.492373 20578 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7d9494f5584d4d5db8b9adf1c8fc888b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260129 02:58:43.492154 20428 tablet_service.cc:1505] Processing CreateTablet for tablet 4f5a77e94fef4b16a2ba7b067c8b144a (DEFAULT_TABLE table=test-workload [id=de73611ac8e3485a98aecc5add247032]), partition=RANGE (key) PARTITION 2001891485 <= VALUES < 2038289512
I20260129 02:58:43.493379 20428 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4f5a77e94fef4b16a2ba7b067c8b144a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260129 02:58:43.493414 20633 raft_consensus.cc:359] T 87d5fe48a44a408aaf03e894a67ecb2f P 197b880fa39d432a9b472845876bfaf0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:58:43.494241 20633 raft_consensus.cc:385] T 87d5fe48a44a408aaf03e894a67ecb2f P 197b880fa39d432a9b472845876bfaf0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260129 02:58:43.494534 20633 raft_consensus.cc:740] T 87d5fe48a44a408aaf03e894a67ecb2f P 197b880fa39d432a9b472845876bfaf0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 197b880fa39d432a9b472845876bfaf0, State: Initialized, Role: FOLLOWER
I20260129 02:58:43.495281 20633 consensus_queue.cc:260] T 87d5fe48a44a408aaf03e894a67ecb2f P 197b880fa39d432a9b472845876bfaf0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:58:43.487336 20653 raft_consensus.cc:515] T 87d5fe48a44a408aaf03e894a67ecb2f P 974b802141bd4f13beffef2aa03f88f9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:58:43.498611 20441 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "87d5fe48a44a408aaf03e894a67ecb2f" candidate_uuid: "974b802141bd4f13beffef2aa03f88f9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "197b880fa39d432a9b472845876bfaf0"
I20260129 02:58:43.499176 20441 raft_consensus.cc:3060] T 87d5fe48a44a408aaf03e894a67ecb2f P 197b880fa39d432a9b472845876bfaf0 [term 0 FOLLOWER]: Advancing to term 1
I20260129 02:58:43.499666 20590 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "87d5fe48a44a408aaf03e894a67ecb2f" candidate_uuid: "974b802141bd4f13beffef2aa03f88f9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c50de89a5a845668696dcb28c3badd6"
I20260129 02:58:43.500265 20590 raft_consensus.cc:3060] T 87d5fe48a44a408aaf03e894a67ecb2f P 1c50de89a5a845668696dcb28c3badd6 [term 0 FOLLOWER]: Advancing to term 1
I20260129 02:58:43.504902 20499 tablet_service.cc:1505] Processing CreateTablet for tablet 4f5a77e94fef4b16a2ba7b067c8b144a (DEFAULT_TABLE table=test-workload [id=de73611ac8e3485a98aecc5add247032]), partition=RANGE (key) PARTITION 2001891485 <= VALUES < 2038289512
I20260129 02:58:43.506211 20499 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4f5a77e94fef4b16a2ba7b067c8b144a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260129 02:58:43.509472 20579 tablet_service.cc:1505] Processing CreateTablet for tablet 4f5a77e94fef4b16a2ba7b067c8b144a (DEFAULT_TABLE table=test-workload [id=de73611ac8e3485a98aecc5add247032]), partition=RANGE (key) PARTITION 2001891485 <= VALUES < 2038289512
I20260129 02:58:43.510880 20590 raft_consensus.cc:2468] T 87d5fe48a44a408aaf03e894a67ecb2f P 1c50de89a5a845668696dcb28c3badd6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 974b802141bd4f13beffef2aa03f88f9 in term 1.
I20260129 02:58:43.511231 20579 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4f5a77e94fef4b16a2ba7b067c8b144a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260129 02:58:43.512104 20480 leader_election.cc:304] T 87d5fe48a44a408aaf03e894a67ecb2f P 974b802141bd4f13beffef2aa03f88f9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1c50de89a5a845668696dcb28c3badd6, 974b802141bd4f13beffef2aa03f88f9; no voters:
I20260129 02:58:43.515702 20441 raft_consensus.cc:2468] T 87d5fe48a44a408aaf03e894a67ecb2f P 197b880fa39d432a9b472845876bfaf0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 974b802141bd4f13beffef2aa03f88f9 in term 1.
I20260129 02:58:43.487704 20631 tablet_bootstrap.cc:654] T 1299723a4476420ab02fd8bfd7aa0645 P 1c50de89a5a845668696dcb28c3badd6: Neither blocks nor log segments found. Creating new log.
I20260129 02:58:43.517302 20630 tablet_bootstrap.cc:654] T 1299723a4476420ab02fd8bfd7aa0645 P 974b802141bd4f13beffef2aa03f88f9: Neither blocks nor log segments found. Creating new log.
I20260129 02:58:43.519551 20653 leader_election.cc:290] T 87d5fe48a44a408aaf03e894a67ecb2f P 974b802141bd4f13beffef2aa03f88f9 [CANDIDATE]: Term 1 election: Requested vote from peers 197b880fa39d432a9b472845876bfaf0 (127.18.211.1:42027), 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3:37735)
I20260129 02:58:43.519856 20427 tablet_service.cc:1505] Processing CreateTablet for tablet a4c1226b987b468fa84dfc6ba0e2dbe1 (DEFAULT_TABLE table=test-workload [id=de73611ac8e3485a98aecc5add247032]), partition=RANGE (key) PARTITION 2038289512 <= VALUES < 2074687539
I20260129 02:58:43.521808 20427 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a4c1226b987b468fa84dfc6ba0e2dbe1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260129 02:58:43.524452 20641 raft_consensus.cc:493] T 4b1876f9a70b49edb3aa74105e66d3b3 P 1c50de89a5a845668696dcb28c3badd6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260129 02:58:43.528064 20641 raft_consensus.cc:515] T 4b1876f9a70b49edb3aa74105e66d3b3 P 1c50de89a5a845668696dcb28c3badd6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:58:43.531720 20424 tablet_service.cc:1505] Processing CreateTablet for tablet 3906b080a5b34a74b8b64128635e8997 (DEFAULT_TABLE table=test-workload [id=de73611ac8e3485a98aecc5add247032]), partition=RANGE (key) PARTITION 2074687539 <= VALUES < 2111085566
I20260129 02:58:43.533511 20424 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3906b080a5b34a74b8b64128635e8997. 1 dirs total, 0 dirs full, 0 dirs failed
I20260129 02:58:43.533378 20440 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4b1876f9a70b49edb3aa74105e66d3b3" candidate_uuid: "1c50de89a5a845668696dcb28c3badd6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "197b880fa39d432a9b472845876bfaf0" is_pre_election: true
I20260129 02:58:43.534498 20516 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4b1876f9a70b49edb3aa74105e66d3b3" candidate_uuid: "1c50de89a5a845668696dcb28c3badd6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "974b802141bd4f13beffef2aa03f88f9" is_pre_election: true
I20260129 02:58:43.535151 20516 raft_consensus.cc:2468] T 4b1876f9a70b49edb3aa74105e66d3b3 P 974b802141bd4f13beffef2aa03f88f9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1c50de89a5a845668696dcb28c3badd6 in term 0.
I20260129 02:58:43.536341 20556 leader_election.cc:304] T 4b1876f9a70b49edb3aa74105e66d3b3 P 1c50de89a5a845668696dcb28c3badd6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1c50de89a5a845668696dcb28c3badd6, 974b802141bd4f13beffef2aa03f88f9; no voters:
W20260129 02:58:43.537235 20553 leader_election.cc:343] T 4b1876f9a70b49edb3aa74105e66d3b3 P 1c50de89a5a845668696dcb28c3badd6 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 197b880fa39d432a9b472845876bfaf0 (127.18.211.1:42027): Illegal state: must be running to vote when last-logged opid is not known
I20260129 02:58:43.533375 20641 leader_election.cc:290] T 4b1876f9a70b49edb3aa74105e66d3b3 P 1c50de89a5a845668696dcb28c3badd6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 197b880fa39d432a9b472845876bfaf0 (127.18.211.1:42027), 974b802141bd4f13beffef2aa03f88f9 (127.18.211.2:34517)
I20260129 02:58:43.540464 20641 raft_consensus.cc:2804] T 4b1876f9a70b49edb3aa74105e66d3b3 P 1c50de89a5a845668696dcb28c3badd6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260129 02:58:43.540838 20641 raft_consensus.cc:493] T 4b1876f9a70b49edb3aa74105e66d3b3 P 1c50de89a5a845668696dcb28c3badd6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260129 02:58:43.540571 20503 tablet_service.cc:1505] Processing CreateTablet for tablet 3906b080a5b34a74b8b64128635e8997 (DEFAULT_TABLE table=test-workload [id=de73611ac8e3485a98aecc5add247032]), partition=RANGE (key) PARTITION 2074687539 <= VALUES < 2111085566
I20260129 02:58:43.541181 20641 raft_consensus.cc:3060] T 4b1876f9a70b49edb3aa74105e66d3b3 P 1c50de89a5a845668696dcb28c3badd6 [term 0 FOLLOWER]: Advancing to term 1
I20260129 02:58:43.541899 20503 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3906b080a5b34a74b8b64128635e8997. 1 dirs total, 0 dirs full, 0 dirs failed
I20260129 02:58:43.548144 20430 tablet_service.cc:1505] Processing CreateTablet for tablet a8479d1fe0bd41e5946186e3a6ca8c7a (DEFAULT_TABLE table=test-workload [id=de73611ac8e3485a98aecc5add247032]), partition=RANGE (key) PARTITION 2111085566 <= VALUES
I20260129 02:58:43.549408 20633 ts_tablet_manager.cc:1434] T 87d5fe48a44a408aaf03e894a67ecb2f P 197b880fa39d432a9b472845876bfaf0: Time spent starting tablet: real 0.058s user 0.008s sys 0.000s
I20260129 02:58:43.549458 20430 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a8479d1fe0bd41e5946186e3a6ca8c7a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260129 02:58:43.556942 20653 raft_consensus.cc:2804] T 87d5fe48a44a408aaf03e894a67ecb2f P 974b802141bd4f13beffef2aa03f88f9 [term 1 FOLLOWER]: Leader election won for term 1
I20260129 02:58:43.589046 20499 tablet_service.cc:1505] Processing CreateTablet for tablet a8479d1fe0bd41e5946186e3a6ca8c7a (DEFAULT_TABLE table=test-workload [id=de73611ac8e3485a98aecc5add247032]), partition=RANGE (key) PARTITION 2111085566 <= VALUES
I20260129 02:58:43.590106 20499 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a8479d1fe0bd41e5946186e3a6ca8c7a. 1 dirs total, 0 dirs full, 0 dirs failed
W20260129 02:58:43.565518 20480 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.211.3:37735, user_credentials={real_user=slave}} blocked reactor thread for 52418.8us
I20260129 02:58:43.579182 20653 raft_consensus.cc:697] T 87d5fe48a44a408aaf03e894a67ecb2f P 974b802141bd4f13beffef2aa03f88f9 [term 1 LEADER]: Becoming Leader. State: Replica: 974b802141bd4f13beffef2aa03f88f9, State: Running, Role: LEADER
I20260129 02:58:43.600819 20572 tablet_service.cc:1505] Processing CreateTablet for tablet a4c1226b987b468fa84dfc6ba0e2dbe1 (DEFAULT_TABLE table=test-workload [id=de73611ac8e3485a98aecc5add247032]), partition=RANGE (key) PARTITION 2038289512 <= VALUES < 2074687539
I20260129 02:58:43.602139 20572 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a4c1226b987b468fa84dfc6ba0e2dbe1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260129 02:58:43.593735 20633 tablet_bootstrap.cc:492] T 1299723a4476420ab02fd8bfd7aa0645 P 197b880fa39d432a9b472845876bfaf0: Bootstrap starting.
I20260129 02:58:43.574805 20500 tablet_service.cc:1505] Processing CreateTablet for tablet a4c1226b987b468fa84dfc6ba0e2dbe1 (DEFAULT_TABLE table=test-workload [id=de73611ac8e3485a98aecc5add247032]), partition=RANGE (key) PARTITION 2038289512 <= VALUES < 2074687539
I20260129 02:58:43.604146 20500 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a4c1226b987b468fa84dfc6ba0e2dbe1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260129 02:58:43.577648 20641 raft_consensus.cc:515] T 4b1876f9a70b49edb3aa74105e66d3b3 P 1c50de89a5a845668696dcb28c3badd6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:58:43.624030 20641 leader_election.cc:290] T 4b1876f9a70b49edb3aa74105e66d3b3 P 1c50de89a5a845668696dcb28c3badd6 [CANDIDATE]: Term 1 election: Requested vote from peers 197b880fa39d432a9b472845876bfaf0 (127.18.211.1:42027), 974b802141bd4f13beffef2aa03f88f9 (127.18.211.2:34517)
I20260129 02:58:43.624415 20633 tablet_bootstrap.cc:654] T 1299723a4476420ab02fd8bfd7aa0645 P 197b880fa39d432a9b472845876bfaf0: Neither blocks nor log segments found. Creating new log.
I20260129 02:58:43.624796 20630 tablet_bootstrap.cc:492] T 1299723a4476420ab02fd8bfd7aa0645 P 974b802141bd4f13beffef2aa03f88f9: No bootstrap required, opened a new log
I20260129 02:58:43.625327 20630 ts_tablet_manager.cc:1403] T 1299723a4476420ab02fd8bfd7aa0645 P 974b802141bd4f13beffef2aa03f88f9: Time spent bootstrapping tablet: real 0.142s user 0.013s sys 0.014s
I20260129 02:58:43.624886 20653 consensus_queue.cc:237] T 87d5fe48a44a408aaf03e894a67ecb2f P 974b802141bd4f13beffef2aa03f88f9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:58:43.625880 20441 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4b1876f9a70b49edb3aa74105e66d3b3" candidate_uuid: "1c50de89a5a845668696dcb28c3badd6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "197b880fa39d432a9b472845876bfaf0"
W20260129 02:58:43.593825 20556 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.211.2:34517, user_credentials={real_user=slave}} blocked reactor thread for 57572.3us
W20260129 02:58:43.627372 20553 leader_election.cc:343] T 4b1876f9a70b49edb3aa74105e66d3b3 P 1c50de89a5a845668696dcb28c3badd6 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 197b880fa39d432a9b472845876bfaf0 (127.18.211.1:42027): Illegal state: must be running to vote when last-logged opid is not known
I20260129 02:58:43.628317 20630 raft_consensus.cc:359] T 1299723a4476420ab02fd8bfd7aa0645 P 974b802141bd4f13beffef2aa03f88f9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:58:43.629137 20630 raft_consensus.cc:385] T 1299723a4476420ab02fd8bfd7aa0645 P 974b802141bd4f13beffef2aa03f88f9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260129 02:58:43.629467 20630 raft_consensus.cc:740] T 1299723a4476420ab02fd8bfd7aa0645 P 974b802141bd4f13beffef2aa03f88f9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 974b802141bd4f13beffef2aa03f88f9, State: Initialized, Role: FOLLOWER
I20260129 02:58:43.630214 20630 consensus_queue.cc:260] T 1299723a4476420ab02fd8bfd7aa0645 P 974b802141bd4f13beffef2aa03f88f9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:58:43.631618 20638 raft_consensus.cc:493] T 75f45538dc704d9a8ec4ff8ad9f3ea5b P 197b880fa39d432a9b472845876bfaf0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260129 02:58:43.632135 20638 raft_consensus.cc:515] T 75f45538dc704d9a8ec4ff8ad9f3ea5b P 197b880fa39d432a9b472845876bfaf0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:58:43.633452 20516 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4b1876f9a70b49edb3aa74105e66d3b3" candidate_uuid: "1c50de89a5a845668696dcb28c3badd6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "974b802141bd4f13beffef2aa03f88f9"
I20260129 02:58:43.634042 20516 raft_consensus.cc:3060] T 4b1876f9a70b49edb3aa74105e66d3b3 P 974b802141bd4f13beffef2aa03f88f9 [term 0 FOLLOWER]: Advancing to term 1
I20260129 02:58:43.634117 20638 leader_election.cc:290] T 75f45538dc704d9a8ec4ff8ad9f3ea5b P 197b880fa39d432a9b472845876bfaf0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 974b802141bd4f13beffef2aa03f88f9 (127.18.211.2:34517), 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3:37735)
I20260129 02:58:43.635839 20631 tablet_bootstrap.cc:492] T 1299723a4476420ab02fd8bfd7aa0645 P 1c50de89a5a845668696dcb28c3badd6: No bootstrap required, opened a new log
I20260129 02:58:43.636379 20515 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "75f45538dc704d9a8ec4ff8ad9f3ea5b" candidate_uuid: "197b880fa39d432a9b472845876bfaf0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "974b802141bd4f13beffef2aa03f88f9" is_pre_election: true
I20260129 02:58:43.636372 20631 ts_tablet_manager.cc:1403] T 1299723a4476420ab02fd8bfd7aa0645 P 1c50de89a5a845668696dcb28c3badd6: Time spent bootstrapping tablet: real 0.283s user 0.011s sys 0.015s
I20260129 02:58:43.641382 20516 raft_consensus.cc:2468] T 4b1876f9a70b49edb3aa74105e66d3b3 P 974b802141bd4f13beffef2aa03f88f9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1c50de89a5a845668696dcb28c3badd6 in term 1.
W20260129 02:58:43.644062 20407 leader_election.cc:343] T 75f45538dc704d9a8ec4ff8ad9f3ea5b P 197b880fa39d432a9b472845876bfaf0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 974b802141bd4f13beffef2aa03f88f9 (127.18.211.2:34517): Illegal state: must be running to vote when last-logged opid is not known
I20260129 02:58:43.645099 20556 leader_election.cc:304] T 4b1876f9a70b49edb3aa74105e66d3b3 P 1c50de89a5a845668696dcb28c3badd6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1c50de89a5a845668696dcb28c3badd6, 974b802141bd4f13beffef2aa03f88f9; no voters: 197b880fa39d432a9b472845876bfaf0
I20260129 02:58:43.646116 20579 tablet_service.cc:1505] Processing CreateTablet for tablet a8479d1fe0bd41e5946186e3a6ca8c7a (DEFAULT_TABLE table=test-workload [id=de73611ac8e3485a98aecc5add247032]), partition=RANGE (key) PARTITION 2111085566 <= VALUES
I20260129 02:58:43.647531 20641 raft_consensus.cc:2804] T 4b1876f9a70b49edb3aa74105e66d3b3 P 1c50de89a5a845668696dcb28c3badd6 [term 1 FOLLOWER]: Leader election won for term 1
I20260129 02:58:43.648270 20641 raft_consensus.cc:697] T 4b1876f9a70b49edb3aa74105e66d3b3 P 1c50de89a5a845668696dcb28c3badd6 [term 1 LEADER]: Becoming Leader. State: Replica: 1c50de89a5a845668696dcb28c3badd6, State: Running, Role: LEADER
I20260129 02:58:43.647749 20579 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a8479d1fe0bd41e5946186e3a6ca8c7a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260129 02:58:43.649173 20573 tablet_service.cc:1505] Processing CreateTablet for tablet 3906b080a5b34a74b8b64128635e8997 (DEFAULT_TABLE table=test-workload [id=de73611ac8e3485a98aecc5add247032]), partition=RANGE (key) PARTITION 2074687539 <= VALUES < 2111085566
I20260129 02:58:43.649123 20641 consensus_queue.cc:237] T 4b1876f9a70b49edb3aa74105e66d3b3 P 1c50de89a5a845668696dcb28c3badd6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:58:43.650516 20573 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3906b080a5b34a74b8b64128635e8997. 1 dirs total, 0 dirs full, 0 dirs failed
I20260129 02:58:43.653553 20590 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "75f45538dc704d9a8ec4ff8ad9f3ea5b" candidate_uuid: "197b880fa39d432a9b472845876bfaf0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c50de89a5a845668696dcb28c3badd6" is_pre_election: true
I20260129 02:58:43.661052 20631 raft_consensus.cc:359] T 1299723a4476420ab02fd8bfd7aa0645 P 1c50de89a5a845668696dcb28c3badd6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:58:43.661820 20631 raft_consensus.cc:385] T 1299723a4476420ab02fd8bfd7aa0645 P 1c50de89a5a845668696dcb28c3badd6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260129 02:58:43.662125 20631 raft_consensus.cc:740] T 1299723a4476420ab02fd8bfd7aa0645 P 1c50de89a5a845668696dcb28c3badd6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1c50de89a5a845668696dcb28c3badd6, State: Initialized, Role: FOLLOWER
W20260129 02:58:43.662315 20405 leader_election.cc:343] T 75f45538dc704d9a8ec4ff8ad9f3ea5b P 197b880fa39d432a9b472845876bfaf0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3:37735): Illegal state: must be running to vote when last-logged opid is not known
I20260129 02:58:43.662861 20405 leader_election.cc:304] T 75f45538dc704d9a8ec4ff8ad9f3ea5b P 197b880fa39d432a9b472845876bfaf0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 197b880fa39d432a9b472845876bfaf0; no voters: 1c50de89a5a845668696dcb28c3badd6, 974b802141bd4f13beffef2aa03f88f9
I20260129 02:58:43.664165 20638 raft_consensus.cc:2749] T 75f45538dc704d9a8ec4ff8ad9f3ea5b P 197b880fa39d432a9b472845876bfaf0 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260129 02:58:43.661388 20630 ts_tablet_manager.cc:1434] T 1299723a4476420ab02fd8bfd7aa0645 P 974b802141bd4f13beffef2aa03f88f9: Time spent starting tablet: real 0.036s user 0.007s sys 0.000s
I20260129 02:58:43.668079 20630 tablet_bootstrap.cc:492] T 45cc5f3d2688486d9b6dbe7e666dedde P 974b802141bd4f13beffef2aa03f88f9: Bootstrap starting.
I20260129 02:58:43.679442 20336 catalog_manager.cc:5654] T 4b1876f9a70b49edb3aa74105e66d3b3 P 1c50de89a5a845668696dcb28c3badd6 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3). New cstate: current_term: 1 leader_uuid: "1c50de89a5a845668696dcb28c3badd6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } health_report { overall_health: HEALTHY } } }
I20260129 02:58:43.668193 20631 consensus_queue.cc:260] T 1299723a4476420ab02fd8bfd7aa0645 P 1c50de89a5a845668696dcb28c3badd6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:58:43.681650 20338 catalog_manager.cc:5654] T 87d5fe48a44a408aaf03e894a67ecb2f P 974b802141bd4f13beffef2aa03f88f9 reported cstate change: term changed from 0 to 1, leader changed from <none> to 974b802141bd4f13beffef2aa03f88f9 (127.18.211.2). New cstate: current_term: 1 leader_uuid: "974b802141bd4f13beffef2aa03f88f9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } health_report { overall_health: UNKNOWN } } }
I20260129 02:58:43.683310 20631 ts_tablet_manager.cc:1434] T 1299723a4476420ab02fd8bfd7aa0645 P 1c50de89a5a845668696dcb28c3badd6: Time spent starting tablet: real 0.044s user 0.007s sys 0.000s
I20260129 02:58:43.684757 20631 tablet_bootstrap.cc:492] T 2e58e9a5a8aa4cecafb464addb4f35ed P 1c50de89a5a845668696dcb28c3badd6: Bootstrap starting.
I20260129 02:58:43.687711 20633 tablet_bootstrap.cc:492] T 1299723a4476420ab02fd8bfd7aa0645 P 197b880fa39d432a9b472845876bfaf0: No bootstrap required, opened a new log
I20260129 02:58:43.688233 20633 ts_tablet_manager.cc:1403] T 1299723a4476420ab02fd8bfd7aa0645 P 197b880fa39d432a9b472845876bfaf0: Time spent bootstrapping tablet: real 0.107s user 0.019s sys 0.014s
I20260129 02:58:43.691018 20633 raft_consensus.cc:359] T 1299723a4476420ab02fd8bfd7aa0645 P 197b880fa39d432a9b472845876bfaf0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:58:43.691785 20633 raft_consensus.cc:385] T 1299723a4476420ab02fd8bfd7aa0645 P 197b880fa39d432a9b472845876bfaf0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260129 02:58:43.692098 20633 raft_consensus.cc:740] T 1299723a4476420ab02fd8bfd7aa0645 P 197b880fa39d432a9b472845876bfaf0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 197b880fa39d432a9b472845876bfaf0, State: Initialized, Role: FOLLOWER
I20260129 02:58:43.692840 20633 consensus_queue.cc:260] T 1299723a4476420ab02fd8bfd7aa0645 P 197b880fa39d432a9b472845876bfaf0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:58:43.698827 20633 ts_tablet_manager.cc:1434] T 1299723a4476420ab02fd8bfd7aa0645 P 197b880fa39d432a9b472845876bfaf0: Time spent starting tablet: real 0.010s user 0.006s sys 0.000s
I20260129 02:58:43.698580 20638 consensus_queue.cc:1048] T a1a39dcd328e4268ae41b988c66b4a75 P 197b880fa39d432a9b472845876bfaf0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260129 02:58:43.699903 20633 tablet_bootstrap.cc:492] T e7d83d4ccf684e76acb29a0dc844e1b8 P 197b880fa39d432a9b472845876bfaf0: Bootstrap starting.
I20260129 02:58:43.706480 20633 tablet_bootstrap.cc:654] T e7d83d4ccf684e76acb29a0dc844e1b8 P 197b880fa39d432a9b472845876bfaf0: Neither blocks nor log segments found. Creating new log.
I20260129 02:58:43.710875 20630 tablet_bootstrap.cc:654] T 45cc5f3d2688486d9b6dbe7e666dedde P 974b802141bd4f13beffef2aa03f88f9: Neither blocks nor log segments found. Creating new log.
I20260129 02:58:43.721755 20631 tablet_bootstrap.cc:654] T 2e58e9a5a8aa4cecafb464addb4f35ed P 1c50de89a5a845668696dcb28c3badd6: Neither blocks nor log segments found. Creating new log.
I20260129 02:58:43.732200 20638 consensus_queue.cc:1048] T a1a39dcd328e4268ae41b988c66b4a75 P 197b880fa39d432a9b472845876bfaf0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260129 02:58:43.735344 20633 tablet_bootstrap.cc:492] T e7d83d4ccf684e76acb29a0dc844e1b8 P 197b880fa39d432a9b472845876bfaf0: No bootstrap required, opened a new log
I20260129 02:58:43.735951 20633 ts_tablet_manager.cc:1403] T e7d83d4ccf684e76acb29a0dc844e1b8 P 197b880fa39d432a9b472845876bfaf0: Time spent bootstrapping tablet: real 0.036s user 0.018s sys 0.012s
I20260129 02:58:43.738912 20633 raft_consensus.cc:359] T e7d83d4ccf684e76acb29a0dc844e1b8 P 197b880fa39d432a9b472845876bfaf0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:58:43.739673 20633 raft_consensus.cc:385] T e7d83d4ccf684e76acb29a0dc844e1b8 P 197b880fa39d432a9b472845876bfaf0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260129 02:58:43.740015 20633 raft_consensus.cc:740] T e7d83d4ccf684e76acb29a0dc844e1b8 P 197b880fa39d432a9b472845876bfaf0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 197b880fa39d432a9b472845876bfaf0, State: Initialized, Role: FOLLOWER
I20260129 02:58:43.740895 20633 consensus_queue.cc:260] T e7d83d4ccf684e76acb29a0dc844e1b8 P 197b880fa39d432a9b472845876bfaf0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:58:43.752404 20633 ts_tablet_manager.cc:1434] T e7d83d4ccf684e76acb29a0dc844e1b8 P 197b880fa39d432a9b472845876bfaf0: Time spent starting tablet: real 0.016s user 0.007s sys 0.000s
I20260129 02:58:43.761375 20633 tablet_bootstrap.cc:492] T 2e58e9a5a8aa4cecafb464addb4f35ed P 197b880fa39d432a9b472845876bfaf0: Bootstrap starting.
I20260129 02:58:43.768117 20633 tablet_bootstrap.cc:654] T 2e58e9a5a8aa4cecafb464addb4f35ed P 197b880fa39d432a9b472845876bfaf0: Neither blocks nor log segments found. Creating new log.
I20260129 02:58:43.805424 20631 tablet_bootstrap.cc:492] T 2e58e9a5a8aa4cecafb464addb4f35ed P 1c50de89a5a845668696dcb28c3badd6: No bootstrap required, opened a new log
I20260129 02:58:43.805928 20631 ts_tablet_manager.cc:1403] T 2e58e9a5a8aa4cecafb464addb4f35ed P 1c50de89a5a845668696dcb28c3badd6: Time spent bootstrapping tablet: real 0.121s user 0.026s sys 0.022s
I20260129 02:58:43.812070 20630 tablet_bootstrap.cc:492] T 45cc5f3d2688486d9b6dbe7e666dedde P 974b802141bd4f13beffef2aa03f88f9: No bootstrap required, opened a new log
I20260129 02:58:43.812702 20630 ts_tablet_manager.cc:1403] T 45cc5f3d2688486d9b6dbe7e666dedde P 974b802141bd4f13beffef2aa03f88f9: Time spent bootstrapping tablet: real 0.145s user 0.020s sys 0.032s
I20260129 02:58:43.813738 20653 raft_consensus.cc:493] T 0f20af7fbe5d4db89a6f17948d98912c P 974b802141bd4f13beffef2aa03f88f9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260129 02:58:43.814312 20653 raft_consensus.cc:515] T 0f20af7fbe5d4db89a6f17948d98912c P 974b802141bd4f13beffef2aa03f88f9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:58:43.815805 20630 raft_consensus.cc:359] T 45cc5f3d2688486d9b6dbe7e666dedde P 974b802141bd4f13beffef2aa03f88f9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:58:43.809237 20631 raft_consensus.cc:359] T 2e58e9a5a8aa4cecafb464addb4f35ed P 1c50de89a5a845668696dcb28c3badd6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:58:43.816784 20631 raft_consensus.cc:385] T 2e58e9a5a8aa4cecafb464addb4f35ed P 1c50de89a5a845668696dcb28c3badd6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260129 02:58:43.817150 20631 raft_consensus.cc:740] T 2e58e9a5a8aa4cecafb464addb4f35ed P 1c50de89a5a845668696dcb28c3badd6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1c50de89a5a845668696dcb28c3badd6, State: Initialized, Role: FOLLOWER
I20260129 02:58:43.818218 20631 consensus_queue.cc:260] T 2e58e9a5a8aa4cecafb464addb4f35ed P 1c50de89a5a845668696dcb28c3badd6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:58:43.816574 20630 raft_consensus.cc:385] T 45cc5f3d2688486d9b6dbe7e666dedde P 974b802141bd4f13beffef2aa03f88f9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260129 02:58:43.819733 20630 raft_consensus.cc:740] T 45cc5f3d2688486d9b6dbe7e666dedde P 974b802141bd4f13beffef2aa03f88f9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 974b802141bd4f13beffef2aa03f88f9, State: Initialized, Role: FOLLOWER
I20260129 02:58:43.821861 20441 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0f20af7fbe5d4db89a6f17948d98912c" candidate_uuid: "974b802141bd4f13beffef2aa03f88f9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "197b880fa39d432a9b472845876bfaf0" is_pre_election: true
W20260129 02:58:43.823299 20479 leader_election.cc:343] T 0f20af7fbe5d4db89a6f17948d98912c P 974b802141bd4f13beffef2aa03f88f9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 197b880fa39d432a9b472845876bfaf0 (127.18.211.1:42027): Illegal state: must be running to vote when last-logged opid is not known
I20260129 02:58:43.823745 20631 ts_tablet_manager.cc:1434] T 2e58e9a5a8aa4cecafb464addb4f35ed P 1c50de89a5a845668696dcb28c3badd6: Time spent starting tablet: real 0.018s user 0.008s sys 0.000s
I20260129 02:58:43.821897 20630 consensus_queue.cc:260] T 45cc5f3d2688486d9b6dbe7e666dedde P 974b802141bd4f13beffef2aa03f88f9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:58:43.824673 20653 leader_election.cc:290] T 0f20af7fbe5d4db89a6f17948d98912c P 974b802141bd4f13beffef2aa03f88f9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 197b880fa39d432a9b472845876bfaf0 (127.18.211.1:42027), 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3:37735)
I20260129 02:58:43.826390 20630 ts_tablet_manager.cc:1434] T 45cc5f3d2688486d9b6dbe7e666dedde P 974b802141bd4f13beffef2aa03f88f9: Time spent starting tablet: real 0.013s user 0.007s sys 0.000s
I20260129 02:58:43.827275 20630 tablet_bootstrap.cc:492] T 75f45538dc704d9a8ec4ff8ad9f3ea5b P 974b802141bd4f13beffef2aa03f88f9: Bootstrap starting.
I20260129 02:58:43.828876 20589 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0f20af7fbe5d4db89a6f17948d98912c" candidate_uuid: "974b802141bd4f13beffef2aa03f88f9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c50de89a5a845668696dcb28c3badd6" is_pre_election: true
I20260129 02:58:43.829304 20631 tablet_bootstrap.cc:492] T 3e2d15f8640b407195210c476e09df91 P 1c50de89a5a845668696dcb28c3badd6: Bootstrap starting.
W20260129 02:58:43.830605 20480 leader_election.cc:343] T 0f20af7fbe5d4db89a6f17948d98912c P 974b802141bd4f13beffef2aa03f88f9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3:37735): Illegal state: must be running to vote when last-logged opid is not known
I20260129 02:58:43.831184 20480 leader_election.cc:304] T 0f20af7fbe5d4db89a6f17948d98912c P 974b802141bd4f13beffef2aa03f88f9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 974b802141bd4f13beffef2aa03f88f9; no voters: 197b880fa39d432a9b472845876bfaf0, 1c50de89a5a845668696dcb28c3badd6
I20260129 02:58:43.832710 20653 raft_consensus.cc:2749] T 0f20af7fbe5d4db89a6f17948d98912c P 974b802141bd4f13beffef2aa03f88f9 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260129 02:58:43.833135 20667 raft_consensus.cc:493] T 1299723a4476420ab02fd8bfd7aa0645 P 974b802141bd4f13beffef2aa03f88f9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260129 02:58:43.833570 20667 raft_consensus.cc:515] T 1299723a4476420ab02fd8bfd7aa0645 P 974b802141bd4f13beffef2aa03f88f9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:58:43.835660 20667 leader_election.cc:290] T 1299723a4476420ab02fd8bfd7aa0645 P 974b802141bd4f13beffef2aa03f88f9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 197b880fa39d432a9b472845876bfaf0 (127.18.211.1:42027), 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3:37735)
I20260129 02:58:43.836170 20631 tablet_bootstrap.cc:654] T 3e2d15f8640b407195210c476e09df91 P 1c50de89a5a845668696dcb28c3badd6: Neither blocks nor log segments found. Creating new log.
I20260129 02:58:43.836505 20658 raft_consensus.cc:493] T e7d83d4ccf684e76acb29a0dc844e1b8 P 197b880fa39d432a9b472845876bfaf0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260129 02:58:43.836977 20441 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1299723a4476420ab02fd8bfd7aa0645" candidate_uuid: "974b802141bd4f13beffef2aa03f88f9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "197b880fa39d432a9b472845876bfaf0" is_pre_election: true
I20260129 02:58:43.839565 20441 raft_consensus.cc:2468] T 1299723a4476420ab02fd8bfd7aa0645 P 197b880fa39d432a9b472845876bfaf0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 974b802141bd4f13beffef2aa03f88f9 in term 0.
I20260129 02:58:43.837235 20590 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1299723a4476420ab02fd8bfd7aa0645" candidate_uuid: "974b802141bd4f13beffef2aa03f88f9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c50de89a5a845668696dcb28c3badd6" is_pre_election: true
I20260129 02:58:43.838454 20641 raft_consensus.cc:493] T 1299723a4476420ab02fd8bfd7aa0645 P 1c50de89a5a845668696dcb28c3badd6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260129 02:58:43.840555 20641 raft_consensus.cc:515] T 1299723a4476420ab02fd8bfd7aa0645 P 1c50de89a5a845668696dcb28c3badd6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:58:43.840749 20479 leader_election.cc:304] T 1299723a4476420ab02fd8bfd7aa0645 P 974b802141bd4f13beffef2aa03f88f9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 197b880fa39d432a9b472845876bfaf0, 974b802141bd4f13beffef2aa03f88f9; no voters:
I20260129 02:58:43.841951 20667 raft_consensus.cc:2804] T 1299723a4476420ab02fd8bfd7aa0645 P 974b802141bd4f13beffef2aa03f88f9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260129 02:58:43.842315 20667 raft_consensus.cc:493] T 1299723a4476420ab02fd8bfd7aa0645 P 974b802141bd4f13beffef2aa03f88f9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260129 02:58:43.842569 20641 leader_election.cc:290] T 1299723a4476420ab02fd8bfd7aa0645 P 1c50de89a5a845668696dcb28c3badd6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 197b880fa39d432a9b472845876bfaf0 (127.18.211.1:42027), 974b802141bd4f13beffef2aa03f88f9 (127.18.211.2:34517)
I20260129 02:58:43.842720 20667 raft_consensus.cc:3060] T 1299723a4476420ab02fd8bfd7aa0645 P 974b802141bd4f13beffef2aa03f88f9 [term 0 FOLLOWER]: Advancing to term 1
I20260129 02:58:43.843120 20630 tablet_bootstrap.cc:654] T 75f45538dc704d9a8ec4ff8ad9f3ea5b P 974b802141bd4f13beffef2aa03f88f9: Neither blocks nor log segments found. Creating new log.
I20260129 02:58:43.844631 20441 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1299723a4476420ab02fd8bfd7aa0645" candidate_uuid: "1c50de89a5a845668696dcb28c3badd6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "197b880fa39d432a9b472845876bfaf0" is_pre_election: true
I20260129 02:58:43.845253 20441 raft_consensus.cc:2468] T 1299723a4476420ab02fd8bfd7aa0645 P 197b880fa39d432a9b472845876bfaf0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1c50de89a5a845668696dcb28c3badd6 in term 0.
I20260129 02:58:43.846482 20553 leader_election.cc:304] T 1299723a4476420ab02fd8bfd7aa0645 P 1c50de89a5a845668696dcb28c3badd6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 197b880fa39d432a9b472845876bfaf0, 1c50de89a5a845668696dcb28c3badd6; no voters:
I20260129 02:58:43.847851 20590 raft_consensus.cc:2468] T 1299723a4476420ab02fd8bfd7aa0645 P 1c50de89a5a845668696dcb28c3badd6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 974b802141bd4f13beffef2aa03f88f9 in term 0.
I20260129 02:58:43.848783 20641 raft_consensus.cc:2804] T 1299723a4476420ab02fd8bfd7aa0645 P 1c50de89a5a845668696dcb28c3badd6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260129 02:58:43.849205 20641 raft_consensus.cc:493] T 1299723a4476420ab02fd8bfd7aa0645 P 1c50de89a5a845668696dcb28c3badd6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260129 02:58:43.849589 20641 raft_consensus.cc:3060] T 1299723a4476420ab02fd8bfd7aa0645 P 1c50de89a5a845668696dcb28c3badd6 [term 0 FOLLOWER]: Advancing to term 1
I20260129 02:58:43.851958 20667 raft_consensus.cc:515] T 1299723a4476420ab02fd8bfd7aa0645 P 974b802141bd4f13beffef2aa03f88f9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:58:43.854681 20630 tablet_bootstrap.cc:492] T 75f45538dc704d9a8ec4ff8ad9f3ea5b P 974b802141bd4f13beffef2aa03f88f9: No bootstrap required, opened a new log
I20260129 02:58:43.854784 20441 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1299723a4476420ab02fd8bfd7aa0645" candidate_uuid: "974b802141bd4f13beffef2aa03f88f9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "197b880fa39d432a9b472845876bfaf0"
I20260129 02:58:43.868183 20630 ts_tablet_manager.cc:1403] T 75f45538dc704d9a8ec4ff8ad9f3ea5b P 974b802141bd4f13beffef2aa03f88f9: Time spent bootstrapping tablet: real 0.041s user 0.013s sys 0.000s
I20260129 02:58:43.868167 20441 raft_consensus.cc:3060] T 1299723a4476420ab02fd8bfd7aa0645 P 197b880fa39d432a9b472845876bfaf0 [term 0 FOLLOWER]: Advancing to term 1
I20260129 02:58:43.870800 20630 raft_consensus.cc:359] T 75f45538dc704d9a8ec4ff8ad9f3ea5b P 974b802141bd4f13beffef2aa03f88f9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:58:43.871501 20630 raft_consensus.cc:385] T 75f45538dc704d9a8ec4ff8ad9f3ea5b P 974b802141bd4f13beffef2aa03f88f9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260129 02:58:43.855554 20515 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1299723a4476420ab02fd8bfd7aa0645" candidate_uuid: "1c50de89a5a845668696dcb28c3badd6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "974b802141bd4f13beffef2aa03f88f9" is_pre_election: true
I20260129 02:58:43.871798 20630 raft_consensus.cc:740] T 75f45538dc704d9a8ec4ff8ad9f3ea5b P 974b802141bd4f13beffef2aa03f88f9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 974b802141bd4f13beffef2aa03f88f9, State: Initialized, Role: FOLLOWER
I20260129 02:58:43.872361 20515 raft_consensus.cc:2393] T 1299723a4476420ab02fd8bfd7aa0645 P 974b802141bd4f13beffef2aa03f88f9 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1c50de89a5a845668696dcb28c3badd6 in current term 1: Already voted for candidate 974b802141bd4f13beffef2aa03f88f9 in this term.
I20260129 02:58:43.872557 20630 consensus_queue.cc:260] T 75f45538dc704d9a8ec4ff8ad9f3ea5b P 974b802141bd4f13beffef2aa03f88f9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:58:43.856456 20667 leader_election.cc:290] T 1299723a4476420ab02fd8bfd7aa0645 P 974b802141bd4f13beffef2aa03f88f9 [CANDIDATE]: Term 1 election: Requested vote from peers 197b880fa39d432a9b472845876bfaf0 (127.18.211.1:42027), 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3:37735)
I20260129 02:58:43.860293 20590 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1299723a4476420ab02fd8bfd7aa0645" candidate_uuid: "974b802141bd4f13beffef2aa03f88f9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c50de89a5a845668696dcb28c3badd6"
I20260129 02:58:43.860733 20658 raft_consensus.cc:515] T e7d83d4ccf684e76acb29a0dc844e1b8 P 197b880fa39d432a9b472845876bfaf0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:58:43.875190 20630 ts_tablet_manager.cc:1434] T 75f45538dc704d9a8ec4ff8ad9f3ea5b P 974b802141bd4f13beffef2aa03f88f9: Time spent starting tablet: real 0.007s user 0.005s sys 0.001s
I20260129 02:58:43.875514 20658 leader_election.cc:290] T e7d83d4ccf684e76acb29a0dc844e1b8 P 197b880fa39d432a9b472845876bfaf0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 974b802141bd4f13beffef2aa03f88f9 (127.18.211.2:34517), 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3:37735)
I20260129 02:58:43.862516 20631 tablet_bootstrap.cc:492] T 3e2d15f8640b407195210c476e09df91 P 1c50de89a5a845668696dcb28c3badd6: No bootstrap required, opened a new log
I20260129 02:58:43.872849 20641 raft_consensus.cc:515] T 1299723a4476420ab02fd8bfd7aa0645 P 1c50de89a5a845668696dcb28c3badd6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:58:43.876332 20589 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e7d83d4ccf684e76acb29a0dc844e1b8" candidate_uuid: "197b880fa39d432a9b472845876bfaf0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c50de89a5a845668696dcb28c3badd6" is_pre_election: true
I20260129 02:58:43.877262 20590 raft_consensus.cc:2393] T 1299723a4476420ab02fd8bfd7aa0645 P 1c50de89a5a845668696dcb28c3badd6 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 974b802141bd4f13beffef2aa03f88f9 in current term 1: Already voted for candidate 1c50de89a5a845668696dcb28c3badd6 in this term.
I20260129 02:58:43.877135 20441 raft_consensus.cc:2468] T 1299723a4476420ab02fd8bfd7aa0645 P 197b880fa39d432a9b472845876bfaf0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 974b802141bd4f13beffef2aa03f88f9 in term 1.
W20260129 02:58:43.877732 20405 leader_election.cc:343] T e7d83d4ccf684e76acb29a0dc844e1b8 P 197b880fa39d432a9b472845876bfaf0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3:37735): Illegal state: must be running to vote when last-logged opid is not known
I20260129 02:58:43.878470 20479 leader_election.cc:304] T 1299723a4476420ab02fd8bfd7aa0645 P 974b802141bd4f13beffef2aa03f88f9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 197b880fa39d432a9b472845876bfaf0, 974b802141bd4f13beffef2aa03f88f9; no voters:
I20260129 02:58:43.878575 20515 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e7d83d4ccf684e76acb29a0dc844e1b8" candidate_uuid: "197b880fa39d432a9b472845876bfaf0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "974b802141bd4f13beffef2aa03f88f9" is_pre_election: true
I20260129 02:58:43.879278 20515 raft_consensus.cc:2468] T e7d83d4ccf684e76acb29a0dc844e1b8 P 974b802141bd4f13beffef2aa03f88f9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 197b880fa39d432a9b472845876bfaf0 in term 0.
I20260129 02:58:43.879444 20653 raft_consensus.cc:2804] T 1299723a4476420ab02fd8bfd7aa0645 P 974b802141bd4f13beffef2aa03f88f9 [term 1 FOLLOWER]: Leader election won for term 1
I20260129 02:58:43.880376 20407 leader_election.cc:304] T e7d83d4ccf684e76acb29a0dc844e1b8 P 197b880fa39d432a9b472845876bfaf0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 197b880fa39d432a9b472845876bfaf0, 974b802141bd4f13beffef2aa03f88f9; no voters: 1c50de89a5a845668696dcb28c3badd6
I20260129 02:58:43.881150 20658 raft_consensus.cc:2804] T e7d83d4ccf684e76acb29a0dc844e1b8 P 197b880fa39d432a9b472845876bfaf0 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260129 02:58:43.881552 20658 raft_consensus.cc:493] T e7d83d4ccf684e76acb29a0dc844e1b8 P 197b880fa39d432a9b472845876bfaf0 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260129 02:58:43.876359 20631 ts_tablet_manager.cc:1403] T 3e2d15f8640b407195210c476e09df91 P 1c50de89a5a845668696dcb28c3badd6: Time spent bootstrapping tablet: real 0.047s user 0.006s sys 0.017s
I20260129 02:58:43.881973 20658 raft_consensus.cc:3060] T e7d83d4ccf684e76acb29a0dc844e1b8 P 197b880fa39d432a9b472845876bfaf0 [term 0 FOLLOWER]: Advancing to term 1
I20260129 02:58:43.882561 20630 tablet_bootstrap.cc:492] T 3e2d15f8640b407195210c476e09df91 P 974b802141bd4f13beffef2aa03f88f9: Bootstrap starting.
I20260129 02:58:43.888226 20667 raft_consensus.cc:493] T 45cc5f3d2688486d9b6dbe7e666dedde P 974b802141bd4f13beffef2aa03f88f9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260129 02:58:43.888762 20667 raft_consensus.cc:515] T 45cc5f3d2688486d9b6dbe7e666dedde P 974b802141bd4f13beffef2aa03f88f9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:58:43.889709 20630 tablet_bootstrap.cc:654] T 3e2d15f8640b407195210c476e09df91 P 974b802141bd4f13beffef2aa03f88f9: Neither blocks nor log segments found. Creating new log.
I20260129 02:58:43.891355 20515 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1299723a4476420ab02fd8bfd7aa0645" candidate_uuid: "1c50de89a5a845668696dcb28c3badd6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "974b802141bd4f13beffef2aa03f88f9"
I20260129 02:58:43.891789 20641 leader_election.cc:290] T 1299723a4476420ab02fd8bfd7aa0645 P 1c50de89a5a845668696dcb28c3badd6 [CANDIDATE]: Term 1 election: Requested vote from peers 197b880fa39d432a9b472845876bfaf0 (127.18.211.1:42027), 974b802141bd4f13beffef2aa03f88f9 (127.18.211.2:34517)
I20260129 02:58:43.892133 20440 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "45cc5f3d2688486d9b6dbe7e666dedde" candidate_uuid: "974b802141bd4f13beffef2aa03f88f9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "197b880fa39d432a9b472845876bfaf0" is_pre_election: true
I20260129 02:58:43.892853 20658 raft_consensus.cc:515] T e7d83d4ccf684e76acb29a0dc844e1b8 P 197b880fa39d432a9b472845876bfaf0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
W20260129 02:58:43.897220 20479 leader_election.cc:343] T 45cc5f3d2688486d9b6dbe7e666dedde P 974b802141bd4f13beffef2aa03f88f9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 197b880fa39d432a9b472845876bfaf0 (127.18.211.1:42027): Illegal state: must be running to vote when last-logged opid is not known
I20260129 02:58:43.893769 20667 leader_election.cc:290] T 45cc5f3d2688486d9b6dbe7e666dedde P 974b802141bd4f13beffef2aa03f88f9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 197b880fa39d432a9b472845876bfaf0 (127.18.211.1:42027), 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3:37735)
I20260129 02:58:43.894032 20590 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "45cc5f3d2688486d9b6dbe7e666dedde" candidate_uuid: "974b802141bd4f13beffef2aa03f88f9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c50de89a5a845668696dcb28c3badd6" is_pre_election: true
I20260129 02:58:43.898527 20633 tablet_bootstrap.cc:492] T 2e58e9a5a8aa4cecafb464addb4f35ed P 197b880fa39d432a9b472845876bfaf0: No bootstrap required, opened a new log
I20260129 02:58:43.894791 20631 raft_consensus.cc:359] T 3e2d15f8640b407195210c476e09df91 P 1c50de89a5a845668696dcb28c3badd6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:58:43.898377 20516 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e7d83d4ccf684e76acb29a0dc844e1b8" candidate_uuid: "197b880fa39d432a9b472845876bfaf0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "974b802141bd4f13beffef2aa03f88f9"
I20260129 02:58:43.894410 20441 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1299723a4476420ab02fd8bfd7aa0645" candidate_uuid: "1c50de89a5a845668696dcb28c3badd6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "197b880fa39d432a9b472845876bfaf0"
I20260129 02:58:43.899050 20633 ts_tablet_manager.cc:1403] T 2e58e9a5a8aa4cecafb464addb4f35ed P 197b880fa39d432a9b472845876bfaf0: Time spent bootstrapping tablet: real 0.138s user 0.020s sys 0.042s
I20260129 02:58:43.899137 20516 raft_consensus.cc:3060] T e7d83d4ccf684e76acb29a0dc844e1b8 P 974b802141bd4f13beffef2aa03f88f9 [term 0 FOLLOWER]: Advancing to term 1
I20260129 02:58:43.899650 20441 raft_consensus.cc:2393] T 1299723a4476420ab02fd8bfd7aa0645 P 197b880fa39d432a9b472845876bfaf0 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 1c50de89a5a845668696dcb28c3badd6 in current term 1: Already voted for candidate 974b802141bd4f13beffef2aa03f88f9 in this term.
W20260129 02:58:43.899138 20480 leader_election.cc:343] T 45cc5f3d2688486d9b6dbe7e666dedde P 974b802141bd4f13beffef2aa03f88f9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3:37735): Illegal state: must be running to vote when last-logged opid is not known
I20260129 02:58:43.900381 20480 leader_election.cc:304] T 45cc5f3d2688486d9b6dbe7e666dedde P 974b802141bd4f13beffef2aa03f88f9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 974b802141bd4f13beffef2aa03f88f9; no voters: 197b880fa39d432a9b472845876bfaf0, 1c50de89a5a845668696dcb28c3badd6
I20260129 02:58:43.900653 20658 leader_election.cc:290] T e7d83d4ccf684e76acb29a0dc844e1b8 P 197b880fa39d432a9b472845876bfaf0 [CANDIDATE]: Term 1 election: Requested vote from peers 974b802141bd4f13beffef2aa03f88f9 (127.18.211.2:34517), 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3:37735)
I20260129 02:58:43.902185 20633 raft_consensus.cc:359] T 2e58e9a5a8aa4cecafb464addb4f35ed P 197b880fa39d432a9b472845876bfaf0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:58:43.902801 20590 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e7d83d4ccf684e76acb29a0dc844e1b8" candidate_uuid: "197b880fa39d432a9b472845876bfaf0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c50de89a5a845668696dcb28c3badd6"
W20260129 02:58:43.904292 20405 leader_election.cc:343] T e7d83d4ccf684e76acb29a0dc844e1b8 P 197b880fa39d432a9b472845876bfaf0 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3:37735): Illegal state: must be running to vote when last-logged opid is not known
I20260129 02:58:43.902959 20633 raft_consensus.cc:385] T 2e58e9a5a8aa4cecafb464addb4f35ed P 197b880fa39d432a9b472845876bfaf0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260129 02:58:43.905161 20633 raft_consensus.cc:740] T 2e58e9a5a8aa4cecafb464addb4f35ed P 197b880fa39d432a9b472845876bfaf0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 197b880fa39d432a9b472845876bfaf0, State: Initialized, Role: FOLLOWER
I20260129 02:58:43.906015 20633 consensus_queue.cc:260] T 2e58e9a5a8aa4cecafb464addb4f35ed P 197b880fa39d432a9b472845876bfaf0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:58:43.908082 20667 raft_consensus.cc:2749] T 45cc5f3d2688486d9b6dbe7e666dedde P 974b802141bd4f13beffef2aa03f88f9 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260129 02:58:43.909296 20633 ts_tablet_manager.cc:1434] T 2e58e9a5a8aa4cecafb464addb4f35ed P 197b880fa39d432a9b472845876bfaf0: Time spent starting tablet: real 0.010s user 0.002s sys 0.006s
I20260129 02:58:43.910300 20631 raft_consensus.cc:385] T 3e2d15f8640b407195210c476e09df91 P 1c50de89a5a845668696dcb28c3badd6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260129 02:58:43.910431 20633 tablet_bootstrap.cc:492] T 724a236524cb4e7ba3428880681d6cc2 P 197b880fa39d432a9b472845876bfaf0: Bootstrap starting.
I20260129 02:58:43.911005 20631 raft_consensus.cc:740] T 3e2d15f8640b407195210c476e09df91 P 1c50de89a5a845668696dcb28c3badd6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1c50de89a5a845668696dcb28c3badd6, State: Initialized, Role: FOLLOWER
I20260129 02:58:43.910595 20516 raft_consensus.cc:2468] T e7d83d4ccf684e76acb29a0dc844e1b8 P 974b802141bd4f13beffef2aa03f88f9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 197b880fa39d432a9b472845876bfaf0 in term 1.
I20260129 02:58:43.912416 20407 leader_election.cc:304] T e7d83d4ccf684e76acb29a0dc844e1b8 P 197b880fa39d432a9b472845876bfaf0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 197b880fa39d432a9b472845876bfaf0, 974b802141bd4f13beffef2aa03f88f9; no voters: 1c50de89a5a845668696dcb28c3badd6
I20260129 02:58:43.912607 20631 consensus_queue.cc:260] T 3e2d15f8640b407195210c476e09df91 P 1c50de89a5a845668696dcb28c3badd6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:58:43.913621 20658 raft_consensus.cc:2804] T e7d83d4ccf684e76acb29a0dc844e1b8 P 197b880fa39d432a9b472845876bfaf0 [term 1 FOLLOWER]: Leader election won for term 1
I20260129 02:58:43.914119 20658 raft_consensus.cc:697] T e7d83d4ccf684e76acb29a0dc844e1b8 P 197b880fa39d432a9b472845876bfaf0 [term 1 LEADER]: Becoming Leader. State: Replica: 197b880fa39d432a9b472845876bfaf0, State: Running, Role: LEADER
I20260129 02:58:43.915009 20658 consensus_queue.cc:237] T e7d83d4ccf684e76acb29a0dc844e1b8 P 197b880fa39d432a9b472845876bfaf0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:58:43.917966 20633 tablet_bootstrap.cc:654] T 724a236524cb4e7ba3428880681d6cc2 P 197b880fa39d432a9b472845876bfaf0: Neither blocks nor log segments found. Creating new log.
I20260129 02:58:43.923285 20631 ts_tablet_manager.cc:1434] T 3e2d15f8640b407195210c476e09df91 P 1c50de89a5a845668696dcb28c3badd6: Time spent starting tablet: real 0.041s user 0.003s sys 0.003s
I20260129 02:58:43.924358 20631 tablet_bootstrap.cc:492] T e7d83d4ccf684e76acb29a0dc844e1b8 P 1c50de89a5a845668696dcb28c3badd6: Bootstrap starting.
I20260129 02:58:43.924638 20653 raft_consensus.cc:697] T 1299723a4476420ab02fd8bfd7aa0645 P 974b802141bd4f13beffef2aa03f88f9 [term 1 LEADER]: Becoming Leader. State: Replica: 974b802141bd4f13beffef2aa03f88f9, State: Running, Role: LEADER
I20260129 02:58:43.924805 20633 tablet_bootstrap.cc:492] T 724a236524cb4e7ba3428880681d6cc2 P 197b880fa39d432a9b472845876bfaf0: No bootstrap required, opened a new log
I20260129 02:58:43.925570 20653 consensus_queue.cc:237] T 1299723a4476420ab02fd8bfd7aa0645 P 974b802141bd4f13beffef2aa03f88f9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:58:43.928115 20630 tablet_bootstrap.cc:492] T 3e2d15f8640b407195210c476e09df91 P 974b802141bd4f13beffef2aa03f88f9: No bootstrap required, opened a new log
I20260129 02:58:43.928670 20630 ts_tablet_manager.cc:1403] T 3e2d15f8640b407195210c476e09df91 P 974b802141bd4f13beffef2aa03f88f9: Time spent bootstrapping tablet: real 0.046s user 0.014s sys 0.001s
I20260129 02:58:43.925561 20633 ts_tablet_manager.cc:1403] T 724a236524cb4e7ba3428880681d6cc2 P 197b880fa39d432a9b472845876bfaf0: Time spent bootstrapping tablet: real 0.015s user 0.005s sys 0.008s
I20260129 02:58:43.930894 20631 tablet_bootstrap.cc:654] T e7d83d4ccf684e76acb29a0dc844e1b8 P 1c50de89a5a845668696dcb28c3badd6: Neither blocks nor log segments found. Creating new log.
I20260129 02:58:43.932770 20685 raft_consensus.cc:493] T 75f45538dc704d9a8ec4ff8ad9f3ea5b P 974b802141bd4f13beffef2aa03f88f9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260129 02:58:43.933220 20685 raft_consensus.cc:515] T 75f45538dc704d9a8ec4ff8ad9f3ea5b P 974b802141bd4f13beffef2aa03f88f9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:58:43.935067 20338 catalog_manager.cc:5654] T e7d83d4ccf684e76acb29a0dc844e1b8 P 197b880fa39d432a9b472845876bfaf0 reported cstate change: term changed from 0 to 1, leader changed from <none> to 197b880fa39d432a9b472845876bfaf0 (127.18.211.1). New cstate: current_term: 1 leader_uuid: "197b880fa39d432a9b472845876bfaf0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } health_report { overall_health: UNKNOWN } } }
I20260129 02:58:43.931245 20630 raft_consensus.cc:359] T 3e2d15f8640b407195210c476e09df91 P 974b802141bd4f13beffef2aa03f88f9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:58:43.937557 20441 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "75f45538dc704d9a8ec4ff8ad9f3ea5b" candidate_uuid: "974b802141bd4f13beffef2aa03f88f9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "197b880fa39d432a9b472845876bfaf0" is_pre_election: true
I20260129 02:58:43.938174 20630 raft_consensus.cc:385] T 3e2d15f8640b407195210c476e09df91 P 974b802141bd4f13beffef2aa03f88f9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260129 02:58:43.938377 20441 raft_consensus.cc:2468] T 75f45538dc704d9a8ec4ff8ad9f3ea5b P 197b880fa39d432a9b472845876bfaf0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 974b802141bd4f13beffef2aa03f88f9 in term 0.
I20260129 02:58:43.938465 20630 raft_consensus.cc:740] T 3e2d15f8640b407195210c476e09df91 P 974b802141bd4f13beffef2aa03f88f9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 974b802141bd4f13beffef2aa03f88f9, State: Initialized, Role: FOLLOWER
I20260129 02:58:43.938952 20685 leader_election.cc:290] T 75f45538dc704d9a8ec4ff8ad9f3ea5b P 974b802141bd4f13beffef2aa03f88f9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 197b880fa39d432a9b472845876bfaf0 (127.18.211.1:42027), 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3:37735)
I20260129 02:58:43.939208 20630 consensus_queue.cc:260] T 3e2d15f8640b407195210c476e09df91 P 974b802141bd4f13beffef2aa03f88f9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:58:43.939637 20479 leader_election.cc:304] T 75f45538dc704d9a8ec4ff8ad9f3ea5b P 974b802141bd4f13beffef2aa03f88f9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 197b880fa39d432a9b472845876bfaf0, 974b802141bd4f13beffef2aa03f88f9; no voters:
I20260129 02:58:43.940512 20685 raft_consensus.cc:2804] T 75f45538dc704d9a8ec4ff8ad9f3ea5b P 974b802141bd4f13beffef2aa03f88f9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260129 02:58:43.940889 20685 raft_consensus.cc:493] T 75f45538dc704d9a8ec4ff8ad9f3ea5b P 974b802141bd4f13beffef2aa03f88f9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260129 02:58:43.941256 20630 ts_tablet_manager.cc:1434] T 3e2d15f8640b407195210c476e09df91 P 974b802141bd4f13beffef2aa03f88f9: Time spent starting tablet: real 0.012s user 0.006s sys 0.000s
I20260129 02:58:43.941253 20685 raft_consensus.cc:3060] T 75f45538dc704d9a8ec4ff8ad9f3ea5b P 974b802141bd4f13beffef2aa03f88f9 [term 0 FOLLOWER]: Advancing to term 1
I20260129 02:58:43.932193 20633 raft_consensus.cc:359] T 724a236524cb4e7ba3428880681d6cc2 P 197b880fa39d432a9b472845876bfaf0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:58:43.941959 20590 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "75f45538dc704d9a8ec4ff8ad9f3ea5b" candidate_uuid: "974b802141bd4f13beffef2aa03f88f9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c50de89a5a845668696dcb28c3badd6" is_pre_election: true
I20260129 02:58:43.942247 20633 raft_consensus.cc:385] T 724a236524cb4e7ba3428880681d6cc2 P 197b880fa39d432a9b472845876bfaf0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260129 02:58:43.942313 20630 tablet_bootstrap.cc:492] T a5a05fbcbbcf4916bbc01fd1a5b5e90d P 974b802141bd4f13beffef2aa03f88f9: Bootstrap starting.
W20260129 02:58:43.944034 20480 leader_election.cc:343] T 75f45538dc704d9a8ec4ff8ad9f3ea5b P 974b802141bd4f13beffef2aa03f88f9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3:37735): Illegal state: must be running to vote when last-logged opid is not known
I20260129 02:58:43.947981 20633 raft_consensus.cc:740] T 724a236524cb4e7ba3428880681d6cc2 P 197b880fa39d432a9b472845876bfaf0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 197b880fa39d432a9b472845876bfaf0, State: Initialized, Role: FOLLOWER
I20260129 02:58:43.950337 20631 tablet_bootstrap.cc:492] T e7d83d4ccf684e76acb29a0dc844e1b8 P 1c50de89a5a845668696dcb28c3badd6: No bootstrap required, opened a new log
I20260129 02:58:43.950824 20631 ts_tablet_manager.cc:1403] T e7d83d4ccf684e76acb29a0dc844e1b8 P 1c50de89a5a845668696dcb28c3badd6: Time spent bootstrapping tablet: real 0.027s user 0.007s sys 0.009s
I20260129 02:58:43.951622 20556 leader_election.cc:304] T 1299723a4476420ab02fd8bfd7aa0645 P 1c50de89a5a845668696dcb28c3badd6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1c50de89a5a845668696dcb28c3badd6; no voters: 197b880fa39d432a9b472845876bfaf0, 974b802141bd4f13beffef2aa03f88f9
I20260129 02:58:43.951655 20685 raft_consensus.cc:515] T 75f45538dc704d9a8ec4ff8ad9f3ea5b P 974b802141bd4f13beffef2aa03f88f9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:58:43.952400 20633 consensus_queue.cc:260] T 724a236524cb4e7ba3428880681d6cc2 P 197b880fa39d432a9b472845876bfaf0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:58:43.952921 20641 raft_consensus.cc:2749] T 1299723a4476420ab02fd8bfd7aa0645 P 1c50de89a5a845668696dcb28c3badd6 [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20260129 02:58:43.953465 20631 raft_consensus.cc:359] T e7d83d4ccf684e76acb29a0dc844e1b8 P 1c50de89a5a845668696dcb28c3badd6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:58:43.954046 20631 raft_consensus.cc:385] T e7d83d4ccf684e76acb29a0dc844e1b8 P 1c50de89a5a845668696dcb28c3badd6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260129 02:58:43.954270 20631 raft_consensus.cc:740] T e7d83d4ccf684e76acb29a0dc844e1b8 P 1c50de89a5a845668696dcb28c3badd6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1c50de89a5a845668696dcb28c3badd6, State: Initialized, Role: FOLLOWER
I20260129 02:58:43.954891 20631 consensus_queue.cc:260] T e7d83d4ccf684e76acb29a0dc844e1b8 P 1c50de89a5a845668696dcb28c3badd6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:58:43.955520 20633 ts_tablet_manager.cc:1434] T 724a236524cb4e7ba3428880681d6cc2 P 197b880fa39d432a9b472845876bfaf0: Time spent starting tablet: real 0.026s user 0.005s sys 0.000s
I20260129 02:58:43.956708 20630 tablet_bootstrap.cc:654] T a5a05fbcbbcf4916bbc01fd1a5b5e90d P 974b802141bd4f13beffef2aa03f88f9: Neither blocks nor log segments found. Creating new log.
I20260129 02:58:43.956925 20631 ts_tablet_manager.cc:1434] T e7d83d4ccf684e76acb29a0dc844e1b8 P 1c50de89a5a845668696dcb28c3badd6: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260129 02:58:43.959638 20631 tablet_bootstrap.cc:492] T 0f20af7fbe5d4db89a6f17948d98912c P 1c50de89a5a845668696dcb28c3badd6: Bootstrap starting.
I20260129 02:58:43.959482 20338 catalog_manager.cc:5654] T 1299723a4476420ab02fd8bfd7aa0645 P 974b802141bd4f13beffef2aa03f88f9 reported cstate change: term changed from 0 to 1, leader changed from <none> to 974b802141bd4f13beffef2aa03f88f9 (127.18.211.2). New cstate: current_term: 1 leader_uuid: "974b802141bd4f13beffef2aa03f88f9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } health_report { overall_health: UNKNOWN } } }
I20260129 02:58:43.965624 20631 tablet_bootstrap.cc:654] T 0f20af7fbe5d4db89a6f17948d98912c P 1c50de89a5a845668696dcb28c3badd6: Neither blocks nor log segments found. Creating new log.
I20260129 02:58:43.976955 20633 tablet_bootstrap.cc:492] T 4b1876f9a70b49edb3aa74105e66d3b3 P 197b880fa39d432a9b472845876bfaf0: Bootstrap starting.
I20260129 02:58:43.980034 20631 tablet_bootstrap.cc:492] T 0f20af7fbe5d4db89a6f17948d98912c P 1c50de89a5a845668696dcb28c3badd6: No bootstrap required, opened a new log
I20260129 02:58:43.980517 20631 ts_tablet_manager.cc:1403] T 0f20af7fbe5d4db89a6f17948d98912c P 1c50de89a5a845668696dcb28c3badd6: Time spent bootstrapping tablet: real 0.021s user 0.011s sys 0.005s
I20260129 02:58:43.982944 20631 raft_consensus.cc:359] T 0f20af7fbe5d4db89a6f17948d98912c P 1c50de89a5a845668696dcb28c3badd6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:58:43.983594 20631 raft_consensus.cc:385] T 0f20af7fbe5d4db89a6f17948d98912c P 1c50de89a5a845668696dcb28c3badd6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260129 02:58:43.983855 20631 raft_consensus.cc:740] T 0f20af7fbe5d4db89a6f17948d98912c P 1c50de89a5a845668696dcb28c3badd6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1c50de89a5a845668696dcb28c3badd6, State: Initialized, Role: FOLLOWER
I20260129 02:58:43.984555 20633 tablet_bootstrap.cc:654] T 4b1876f9a70b49edb3aa74105e66d3b3 P 197b880fa39d432a9b472845876bfaf0: Neither blocks nor log segments found. Creating new log.
I20260129 02:58:43.984761 20631 consensus_queue.cc:260] T 0f20af7fbe5d4db89a6f17948d98912c P 1c50de89a5a845668696dcb28c3badd6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:58:43.986918 20631 ts_tablet_manager.cc:1434] T 0f20af7fbe5d4db89a6f17948d98912c P 1c50de89a5a845668696dcb28c3badd6: Time spent starting tablet: real 0.006s user 0.004s sys 0.000s
I20260129 02:58:43.987841 20631 tablet_bootstrap.cc:492] T 75f45538dc704d9a8ec4ff8ad9f3ea5b P 1c50de89a5a845668696dcb28c3badd6: Bootstrap starting.
I20260129 02:58:43.992312 20441 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "75f45538dc704d9a8ec4ff8ad9f3ea5b" candidate_uuid: "974b802141bd4f13beffef2aa03f88f9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "197b880fa39d432a9b472845876bfaf0"
I20260129 02:58:43.992911 20441 raft_consensus.cc:3060] T 75f45538dc704d9a8ec4ff8ad9f3ea5b P 197b880fa39d432a9b472845876bfaf0 [term 0 FOLLOWER]: Advancing to term 1
I20260129 02:58:43.993628 20630 tablet_bootstrap.cc:492] T a5a05fbcbbcf4916bbc01fd1a5b5e90d P 974b802141bd4f13beffef2aa03f88f9: No bootstrap required, opened a new log
I20260129 02:58:43.993700 20631 tablet_bootstrap.cc:654] T 75f45538dc704d9a8ec4ff8ad9f3ea5b P 1c50de89a5a845668696dcb28c3badd6: Neither blocks nor log segments found. Creating new log.
I20260129 02:58:43.994110 20630 ts_tablet_manager.cc:1403] T a5a05fbcbbcf4916bbc01fd1a5b5e90d P 974b802141bd4f13beffef2aa03f88f9: Time spent bootstrapping tablet: real 0.052s user 0.022s sys 0.015s
I20260129 02:58:43.996856 20630 raft_consensus.cc:359] T a5a05fbcbbcf4916bbc01fd1a5b5e90d P 974b802141bd4f13beffef2aa03f88f9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:58:43.997521 20630 raft_consensus.cc:385] T a5a05fbcbbcf4916bbc01fd1a5b5e90d P 974b802141bd4f13beffef2aa03f88f9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260129 02:58:43.997799 20630 raft_consensus.cc:740] T a5a05fbcbbcf4916bbc01fd1a5b5e90d P 974b802141bd4f13beffef2aa03f88f9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 974b802141bd4f13beffef2aa03f88f9, State: Initialized, Role: FOLLOWER
I20260129 02:58:43.998486 20630 consensus_queue.cc:260] T a5a05fbcbbcf4916bbc01fd1a5b5e90d P 974b802141bd4f13beffef2aa03f88f9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:58:44.000849 20630 ts_tablet_manager.cc:1434] T a5a05fbcbbcf4916bbc01fd1a5b5e90d P 974b802141bd4f13beffef2aa03f88f9: Time spent starting tablet: real 0.006s user 0.006s sys 0.001s
I20260129 02:58:44.001286 20441 raft_consensus.cc:2468] T 75f45538dc704d9a8ec4ff8ad9f3ea5b P 197b880fa39d432a9b472845876bfaf0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 974b802141bd4f13beffef2aa03f88f9 in term 1.
I20260129 02:58:44.001782 20630 tablet_bootstrap.cc:492] T 724a236524cb4e7ba3428880681d6cc2 P 974b802141bd4f13beffef2aa03f88f9: Bootstrap starting.
I20260129 02:58:44.002391 20479 leader_election.cc:304] T 75f45538dc704d9a8ec4ff8ad9f3ea5b P 974b802141bd4f13beffef2aa03f88f9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 197b880fa39d432a9b472845876bfaf0, 974b802141bd4f13beffef2aa03f88f9; no voters:
I20260129 02:58:44.003237 20653 raft_consensus.cc:2804] T 75f45538dc704d9a8ec4ff8ad9f3ea5b P 974b802141bd4f13beffef2aa03f88f9 [term 1 FOLLOWER]: Leader election won for term 1
I20260129 02:58:44.003710 20653 raft_consensus.cc:697] T 75f45538dc704d9a8ec4ff8ad9f3ea5b P 974b802141bd4f13beffef2aa03f88f9 [term 1 LEADER]: Becoming Leader. State: Replica: 974b802141bd4f13beffef2aa03f88f9, State: Running, Role: LEADER
I20260129 02:58:44.004766 20641 raft_consensus.cc:493] T e7d83d4ccf684e76acb29a0dc844e1b8 P 1c50de89a5a845668696dcb28c3badd6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260129 02:58:44.004518 20653 consensus_queue.cc:237] T 75f45538dc704d9a8ec4ff8ad9f3ea5b P 974b802141bd4f13beffef2aa03f88f9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:58:44.004992 20633 tablet_bootstrap.cc:492] T 4b1876f9a70b49edb3aa74105e66d3b3 P 197b880fa39d432a9b472845876bfaf0: No bootstrap required, opened a new log
I20260129 02:58:44.005604 20633 ts_tablet_manager.cc:1403] T 4b1876f9a70b49edb3aa74105e66d3b3 P 197b880fa39d432a9b472845876bfaf0: Time spent bootstrapping tablet: real 0.029s user 0.010s sys 0.003s
I20260129 02:58:44.005234 20641 raft_consensus.cc:515] T e7d83d4ccf684e76acb29a0dc844e1b8 P 1c50de89a5a845668696dcb28c3badd6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:58:44.007267 20641 leader_election.cc:290] T e7d83d4ccf684e76acb29a0dc844e1b8 P 1c50de89a5a845668696dcb28c3badd6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 197b880fa39d432a9b472845876bfaf0 (127.18.211.1:42027), 974b802141bd4f13beffef2aa03f88f9 (127.18.211.2:34517)
I20260129 02:58:44.008302 20633 raft_consensus.cc:359] T 4b1876f9a70b49edb3aa74105e66d3b3 P 197b880fa39d432a9b472845876bfaf0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:58:44.008790 20515 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e7d83d4ccf684e76acb29a0dc844e1b8" candidate_uuid: "1c50de89a5a845668696dcb28c3badd6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "974b802141bd4f13beffef2aa03f88f9" is_pre_election: true
I20260129 02:58:44.009128 20633 raft_consensus.cc:385] T 4b1876f9a70b49edb3aa74105e66d3b3 P 197b880fa39d432a9b472845876bfaf0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260129 02:58:44.009519 20633 raft_consensus.cc:740] T 4b1876f9a70b49edb3aa74105e66d3b3 P 197b880fa39d432a9b472845876bfaf0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 197b880fa39d432a9b472845876bfaf0, State: Initialized, Role: FOLLOWER
I20260129 02:58:44.010078 20590 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "75f45538dc704d9a8ec4ff8ad9f3ea5b" candidate_uuid: "974b802141bd4f13beffef2aa03f88f9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c50de89a5a845668696dcb28c3badd6"
I20260129 02:58:44.009541 20515 raft_consensus.cc:2393] T e7d83d4ccf684e76acb29a0dc844e1b8 P 974b802141bd4f13beffef2aa03f88f9 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1c50de89a5a845668696dcb28c3badd6 in current term 1: Already voted for candidate 197b880fa39d432a9b472845876bfaf0 in this term.
I20260129 02:58:44.010663 20631 tablet_bootstrap.cc:492] T 75f45538dc704d9a8ec4ff8ad9f3ea5b P 1c50de89a5a845668696dcb28c3badd6: No bootstrap required, opened a new log
I20260129 02:58:44.011082 20631 ts_tablet_manager.cc:1403] T 75f45538dc704d9a8ec4ff8ad9f3ea5b P 1c50de89a5a845668696dcb28c3badd6: Time spent bootstrapping tablet: real 0.023s user 0.009s sys 0.007s
W20260129 02:58:44.011440 20480 leader_election.cc:343] T 75f45538dc704d9a8ec4ff8ad9f3ea5b P 974b802141bd4f13beffef2aa03f88f9 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3:37735): Illegal state: must be running to vote when last-logged opid is not known
I20260129 02:58:44.012182 20685 leader_election.cc:290] T 75f45538dc704d9a8ec4ff8ad9f3ea5b P 974b802141bd4f13beffef2aa03f88f9 [CANDIDATE]: Term 1 election: Requested vote from peers 197b880fa39d432a9b472845876bfaf0 (127.18.211.1:42027), 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3:37735)
I20260129 02:58:44.010277 20633 consensus_queue.cc:260] T 4b1876f9a70b49edb3aa74105e66d3b3 P 197b880fa39d432a9b472845876bfaf0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:58:44.013546 20631 raft_consensus.cc:359] T 75f45538dc704d9a8ec4ff8ad9f3ea5b P 1c50de89a5a845668696dcb28c3badd6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:58:44.014156 20631 raft_consensus.cc:385] T 75f45538dc704d9a8ec4ff8ad9f3ea5b P 1c50de89a5a845668696dcb28c3badd6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260129 02:58:44.015101 20441 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e7d83d4ccf684e76acb29a0dc844e1b8" candidate_uuid: "1c50de89a5a845668696dcb28c3badd6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "197b880fa39d432a9b472845876bfaf0" is_pre_election: true
I20260129 02:58:44.016559 20553 leader_election.cc:304] T e7d83d4ccf684e76acb29a0dc844e1b8 P 1c50de89a5a845668696dcb28c3badd6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1c50de89a5a845668696dcb28c3badd6; no voters: 197b880fa39d432a9b472845876bfaf0, 974b802141bd4f13beffef2aa03f88f9
I20260129 02:58:44.017725 20641 raft_consensus.cc:3060] T e7d83d4ccf684e76acb29a0dc844e1b8 P 1c50de89a5a845668696dcb28c3badd6 [term 0 FOLLOWER]: Advancing to term 1
I20260129 02:58:44.019806 20630 tablet_bootstrap.cc:654] T 724a236524cb4e7ba3428880681d6cc2 P 974b802141bd4f13beffef2aa03f88f9: Neither blocks nor log segments found. Creating new log.
I20260129 02:58:44.025061 20641 raft_consensus.cc:2749] T e7d83d4ccf684e76acb29a0dc844e1b8 P 1c50de89a5a845668696dcb28c3badd6 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260129 02:58:44.024216 20338 catalog_manager.cc:5654] T 75f45538dc704d9a8ec4ff8ad9f3ea5b P 974b802141bd4f13beffef2aa03f88f9 reported cstate change: term changed from 0 to 1, leader changed from <none> to 974b802141bd4f13beffef2aa03f88f9 (127.18.211.2). New cstate: current_term: 1 leader_uuid: "974b802141bd4f13beffef2aa03f88f9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } health_report { overall_health: UNKNOWN } } }
I20260129 02:58:44.014413 20631 raft_consensus.cc:740] T 75f45538dc704d9a8ec4ff8ad9f3ea5b P 1c50de89a5a845668696dcb28c3badd6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1c50de89a5a845668696dcb28c3badd6, State: Initialized, Role: FOLLOWER
I20260129 02:58:44.025611 20633 ts_tablet_manager.cc:1434] T 4b1876f9a70b49edb3aa74105e66d3b3 P 197b880fa39d432a9b472845876bfaf0: Time spent starting tablet: real 0.020s user 0.006s sys 0.000s
I20260129 02:58:44.026510 20631 consensus_queue.cc:260] T 75f45538dc704d9a8ec4ff8ad9f3ea5b P 1c50de89a5a845668696dcb28c3badd6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:58:44.030476 20633 tablet_bootstrap.cc:492] T 0f20af7fbe5d4db89a6f17948d98912c P 197b880fa39d432a9b472845876bfaf0: Bootstrap starting.
I20260129 02:58:44.032809 20631 ts_tablet_manager.cc:1434] T 75f45538dc704d9a8ec4ff8ad9f3ea5b P 1c50de89a5a845668696dcb28c3badd6: Time spent starting tablet: real 0.021s user 0.007s sys 0.000s
I20260129 02:58:44.033764 20631 tablet_bootstrap.cc:492] T 724a236524cb4e7ba3428880681d6cc2 P 1c50de89a5a845668696dcb28c3badd6: Bootstrap starting.
I20260129 02:58:44.039106 20633 tablet_bootstrap.cc:654] T 0f20af7fbe5d4db89a6f17948d98912c P 197b880fa39d432a9b472845876bfaf0: Neither blocks nor log segments found. Creating new log.
I20260129 02:58:44.039798 20631 tablet_bootstrap.cc:654] T 724a236524cb4e7ba3428880681d6cc2 P 1c50de89a5a845668696dcb28c3badd6: Neither blocks nor log segments found. Creating new log.
I20260129 02:58:44.053139 20631 tablet_bootstrap.cc:492] T 724a236524cb4e7ba3428880681d6cc2 P 1c50de89a5a845668696dcb28c3badd6: No bootstrap required, opened a new log
I20260129 02:58:44.053274 20633 tablet_bootstrap.cc:492] T 0f20af7fbe5d4db89a6f17948d98912c P 197b880fa39d432a9b472845876bfaf0: No bootstrap required, opened a new log
I20260129 02:58:44.053649 20631 ts_tablet_manager.cc:1403] T 724a236524cb4e7ba3428880681d6cc2 P 1c50de89a5a845668696dcb28c3badd6: Time spent bootstrapping tablet: real 0.020s user 0.010s sys 0.007s
I20260129 02:58:44.056043 20633 ts_tablet_manager.cc:1403] T 0f20af7fbe5d4db89a6f17948d98912c P 197b880fa39d432a9b472845876bfaf0: Time spent bootstrapping tablet: real 0.026s user 0.008s sys 0.010s
I20260129 02:58:44.056459 20630 tablet_bootstrap.cc:492] T 724a236524cb4e7ba3428880681d6cc2 P 974b802141bd4f13beffef2aa03f88f9: No bootstrap required, opened a new log
I20260129 02:58:44.056859 20630 ts_tablet_manager.cc:1403] T 724a236524cb4e7ba3428880681d6cc2 P 974b802141bd4f13beffef2aa03f88f9: Time spent bootstrapping tablet: real 0.055s user 0.022s sys 0.020s
I20260129 02:58:44.058212 20641 consensus_queue.cc:1048] T 4b1876f9a70b49edb3aa74105e66d3b3 P 1c50de89a5a845668696dcb28c3badd6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260129 02:58:44.058701 20630 raft_consensus.cc:359] T 724a236524cb4e7ba3428880681d6cc2 P 974b802141bd4f13beffef2aa03f88f9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:58:44.059309 20630 raft_consensus.cc:385] T 724a236524cb4e7ba3428880681d6cc2 P 974b802141bd4f13beffef2aa03f88f9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260129 02:58:44.059126 20633 raft_consensus.cc:359] T 0f20af7fbe5d4db89a6f17948d98912c P 197b880fa39d432a9b472845876bfaf0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:58:44.059598 20630 raft_consensus.cc:740] T 724a236524cb4e7ba3428880681d6cc2 P 974b802141bd4f13beffef2aa03f88f9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 974b802141bd4f13beffef2aa03f88f9, State: Initialized, Role: FOLLOWER
I20260129 02:58:44.059835 20633 raft_consensus.cc:385] T 0f20af7fbe5d4db89a6f17948d98912c P 197b880fa39d432a9b472845876bfaf0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260129 02:58:44.060175 20633 raft_consensus.cc:740] T 0f20af7fbe5d4db89a6f17948d98912c P 197b880fa39d432a9b472845876bfaf0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 197b880fa39d432a9b472845876bfaf0, State: Initialized, Role: FOLLOWER
I20260129 02:58:44.060784 20633 consensus_queue.cc:260] T 0f20af7fbe5d4db89a6f17948d98912c P 197b880fa39d432a9b472845876bfaf0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:58:44.060402 20630 consensus_queue.cc:260] T 724a236524cb4e7ba3428880681d6cc2 P 974b802141bd4f13beffef2aa03f88f9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:58:44.063933 20653 consensus_queue.cc:1048] T 87d5fe48a44a408aaf03e894a67ecb2f P 974b802141bd4f13beffef2aa03f88f9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260129 02:58:44.064851 20630 ts_tablet_manager.cc:1434] T 724a236524cb4e7ba3428880681d6cc2 P 974b802141bd4f13beffef2aa03f88f9: Time spent starting tablet: real 0.008s user 0.003s sys 0.002s
I20260129 02:58:44.066180 20630 tablet_bootstrap.cc:492] T b189f08e16a74a11980cc55d99359a0a P 974b802141bd4f13beffef2aa03f88f9: Bootstrap starting.
I20260129 02:58:44.067289 20631 raft_consensus.cc:359] T 724a236524cb4e7ba3428880681d6cc2 P 1c50de89a5a845668696dcb28c3badd6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:58:44.070695 20631 raft_consensus.cc:385] T 724a236524cb4e7ba3428880681d6cc2 P 1c50de89a5a845668696dcb28c3badd6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260129 02:58:44.070990 20631 raft_consensus.cc:740] T 724a236524cb4e7ba3428880681d6cc2 P 1c50de89a5a845668696dcb28c3badd6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1c50de89a5a845668696dcb28c3badd6, State: Initialized, Role: FOLLOWER
I20260129 02:58:44.071623 20631 consensus_queue.cc:260] T 724a236524cb4e7ba3428880681d6cc2 P 1c50de89a5a845668696dcb28c3badd6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:58:44.074703 20630 tablet_bootstrap.cc:654] T b189f08e16a74a11980cc55d99359a0a P 974b802141bd4f13beffef2aa03f88f9: Neither blocks nor log segments found. Creating new log.
I20260129 02:58:44.077315 20631 ts_tablet_manager.cc:1434] T 724a236524cb4e7ba3428880681d6cc2 P 1c50de89a5a845668696dcb28c3badd6: Time spent starting tablet: real 0.023s user 0.007s sys 0.002s
I20260129 02:58:44.078307 20631 tablet_bootstrap.cc:492] T 45cc5f3d2688486d9b6dbe7e666dedde P 1c50de89a5a845668696dcb28c3badd6: Bootstrap starting.
I20260129 02:58:44.089398 20630 tablet_bootstrap.cc:492] T b189f08e16a74a11980cc55d99359a0a P 974b802141bd4f13beffef2aa03f88f9: No bootstrap required, opened a new log
I20260129 02:58:44.089919 20630 ts_tablet_manager.cc:1403] T b189f08e16a74a11980cc55d99359a0a P 974b802141bd4f13beffef2aa03f88f9: Time spent bootstrapping tablet: real 0.024s user 0.008s sys 0.011s
I20260129 02:58:44.090878 20633 ts_tablet_manager.cc:1434] T 0f20af7fbe5d4db89a6f17948d98912c P 197b880fa39d432a9b472845876bfaf0: Time spent starting tablet: real 0.034s user 0.002s sys 0.004s
I20260129 02:58:44.091064 20653 consensus_queue.cc:1048] T 87d5fe48a44a408aaf03e894a67ecb2f P 974b802141bd4f13beffef2aa03f88f9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260129 02:58:44.091840 20633 tablet_bootstrap.cc:492] T a5a05fbcbbcf4916bbc01fd1a5b5e90d P 197b880fa39d432a9b472845876bfaf0: Bootstrap starting.
I20260129 02:58:44.092480 20630 raft_consensus.cc:359] T b189f08e16a74a11980cc55d99359a0a P 974b802141bd4f13beffef2aa03f88f9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:58:44.093142 20630 raft_consensus.cc:385] T b189f08e16a74a11980cc55d99359a0a P 974b802141bd4f13beffef2aa03f88f9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260129 02:58:44.093402 20630 raft_consensus.cc:740] T b189f08e16a74a11980cc55d99359a0a P 974b802141bd4f13beffef2aa03f88f9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 974b802141bd4f13beffef2aa03f88f9, State: Initialized, Role: FOLLOWER
I20260129 02:58:44.093878 20440 raft_consensus.cc:3060] T 4b1876f9a70b49edb3aa74105e66d3b3 P 197b880fa39d432a9b472845876bfaf0 [term 0 FOLLOWER]: Advancing to term 1
I20260129 02:58:44.096226 20630 consensus_queue.cc:260] T b189f08e16a74a11980cc55d99359a0a P 974b802141bd4f13beffef2aa03f88f9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:58:44.098618 20630 ts_tablet_manager.cc:1434] T b189f08e16a74a11980cc55d99359a0a P 974b802141bd4f13beffef2aa03f88f9: Time spent starting tablet: real 0.008s user 0.003s sys 0.003s
I20260129 02:58:44.099835 20630 tablet_bootstrap.cc:492] T 78aa39096868415d9afc2ffcb248e198 P 974b802141bd4f13beffef2aa03f88f9: Bootstrap starting.
I20260129 02:58:44.100250 20631 tablet_bootstrap.cc:654] T 45cc5f3d2688486d9b6dbe7e666dedde P 1c50de89a5a845668696dcb28c3badd6: Neither blocks nor log segments found. Creating new log.
I20260129 02:58:44.105228 20633 tablet_bootstrap.cc:654] T a5a05fbcbbcf4916bbc01fd1a5b5e90d P 197b880fa39d432a9b472845876bfaf0: Neither blocks nor log segments found. Creating new log.
I20260129 02:58:44.106598 20630 tablet_bootstrap.cc:654] T 78aa39096868415d9afc2ffcb248e198 P 974b802141bd4f13beffef2aa03f88f9: Neither blocks nor log segments found. Creating new log.
I20260129 02:58:44.113092 20631 tablet_bootstrap.cc:492] T 45cc5f3d2688486d9b6dbe7e666dedde P 1c50de89a5a845668696dcb28c3badd6: No bootstrap required, opened a new log
I20260129 02:58:44.113554 20631 ts_tablet_manager.cc:1403] T 45cc5f3d2688486d9b6dbe7e666dedde P 1c50de89a5a845668696dcb28c3badd6: Time spent bootstrapping tablet: real 0.035s user 0.010s sys 0.000s
I20260129 02:58:44.116827 20633 tablet_bootstrap.cc:492] T a5a05fbcbbcf4916bbc01fd1a5b5e90d P 197b880fa39d432a9b472845876bfaf0: No bootstrap required, opened a new log
I20260129 02:58:44.117300 20633 ts_tablet_manager.cc:1403] T a5a05fbcbbcf4916bbc01fd1a5b5e90d P 197b880fa39d432a9b472845876bfaf0: Time spent bootstrapping tablet: real 0.026s user 0.002s sys 0.010s
I20260129 02:58:44.118803 20669 consensus_queue.cc:1048] T 4b1876f9a70b49edb3aa74105e66d3b3 P 1c50de89a5a845668696dcb28c3badd6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260129 02:58:44.119798 20630 tablet_bootstrap.cc:492] T 78aa39096868415d9afc2ffcb248e198 P 974b802141bd4f13beffef2aa03f88f9: No bootstrap required, opened a new log
I20260129 02:58:44.120256 20630 ts_tablet_manager.cc:1403] T 78aa39096868415d9afc2ffcb248e198 P 974b802141bd4f13beffef2aa03f88f9: Time spent bootstrapping tablet: real 0.021s user 0.007s sys 0.010s
I20260129 02:58:44.122776 20630 raft_consensus.cc:359] T 78aa39096868415d9afc2ffcb248e198 P 974b802141bd4f13beffef2aa03f88f9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:58:44.123487 20630 raft_consensus.cc:385] T 78aa39096868415d9afc2ffcb248e198 P 974b802141bd4f13beffef2aa03f88f9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260129 02:58:44.123801 20630 raft_consensus.cc:740] T 78aa39096868415d9afc2ffcb248e198 P 974b802141bd4f13beffef2aa03f88f9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 974b802141bd4f13beffef2aa03f88f9, State: Initialized, Role: FOLLOWER
I20260129 02:58:44.124456 20630 consensus_queue.cc:260] T 78aa39096868415d9afc2ffcb248e198 P 974b802141bd4f13beffef2aa03f88f9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:58:44.126642 20630 ts_tablet_manager.cc:1434] T 78aa39096868415d9afc2ffcb248e198 P 974b802141bd4f13beffef2aa03f88f9: Time spent starting tablet: real 0.006s user 0.004s sys 0.000s
I20260129 02:58:44.127489 20630 tablet_bootstrap.cc:492] T 17d6431cc77a491e8464d68db4081a51 P 974b802141bd4f13beffef2aa03f88f9: Bootstrap starting.
I20260129 02:58:44.128297 20633 raft_consensus.cc:359] T a5a05fbcbbcf4916bbc01fd1a5b5e90d P 197b880fa39d432a9b472845876bfaf0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:58:44.131032 20633 raft_consensus.cc:385] T a5a05fbcbbcf4916bbc01fd1a5b5e90d P 197b880fa39d432a9b472845876bfaf0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260129 02:58:44.131493 20633 raft_consensus.cc:740] T a5a05fbcbbcf4916bbc01fd1a5b5e90d P 197b880fa39d432a9b472845876bfaf0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 197b880fa39d432a9b472845876bfaf0, State: Initialized, Role: FOLLOWER
I20260129 02:58:44.132213 20633 consensus_queue.cc:260] T a5a05fbcbbcf4916bbc01fd1a5b5e90d P 197b880fa39d432a9b472845876bfaf0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:58:44.130255 20631 raft_consensus.cc:359] T 45cc5f3d2688486d9b6dbe7e666dedde P 1c50de89a5a845668696dcb28c3badd6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:58:44.133244 20631 raft_consensus.cc:385] T 45cc5f3d2688486d9b6dbe7e666dedde P 1c50de89a5a845668696dcb28c3badd6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260129 02:58:44.133330 20630 tablet_bootstrap.cc:654] T 17d6431cc77a491e8464d68db4081a51 P 974b802141bd4f13beffef2aa03f88f9: Neither blocks nor log segments found. Creating new log.
I20260129 02:58:44.133649 20631 raft_consensus.cc:740] T 45cc5f3d2688486d9b6dbe7e666dedde P 1c50de89a5a845668696dcb28c3badd6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1c50de89a5a845668696dcb28c3badd6, State: Initialized, Role: FOLLOWER
I20260129 02:58:44.135488 20631 consensus_queue.cc:260] T 45cc5f3d2688486d9b6dbe7e666dedde P 1c50de89a5a845668696dcb28c3badd6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:58:44.137177 20633 ts_tablet_manager.cc:1434] T a5a05fbcbbcf4916bbc01fd1a5b5e90d P 197b880fa39d432a9b472845876bfaf0: Time spent starting tablet: real 0.020s user 0.007s sys 0.000s
I20260129 02:58:44.138561 20631 ts_tablet_manager.cc:1434] T 45cc5f3d2688486d9b6dbe7e666dedde P 1c50de89a5a845668696dcb28c3badd6: Time spent starting tablet: real 0.025s user 0.004s sys 0.003s
I20260129 02:58:44.139849 20631 tablet_bootstrap.cc:492] T a609ac02fae84725ab743f95f51d33ce P 1c50de89a5a845668696dcb28c3badd6: Bootstrap starting.
I20260129 02:58:44.147991 20631 tablet_bootstrap.cc:654] T a609ac02fae84725ab743f95f51d33ce P 1c50de89a5a845668696dcb28c3badd6: Neither blocks nor log segments found. Creating new log.
I20260129 02:58:44.152055 20633 tablet_bootstrap.cc:492] T 17d6431cc77a491e8464d68db4081a51 P 197b880fa39d432a9b472845876bfaf0: Bootstrap starting.
I20260129 02:58:44.153170 20653 raft_consensus.cc:493] T 724a236524cb4e7ba3428880681d6cc2 P 974b802141bd4f13beffef2aa03f88f9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260129 02:58:44.153645 20653 raft_consensus.cc:515] T 724a236524cb4e7ba3428880681d6cc2 P 974b802141bd4f13beffef2aa03f88f9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:58:44.155711 20653 leader_election.cc:290] T 724a236524cb4e7ba3428880681d6cc2 P 974b802141bd4f13beffef2aa03f88f9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 197b880fa39d432a9b472845876bfaf0 (127.18.211.1:42027), 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3:37735)
I20260129 02:58:44.157835 20633 tablet_bootstrap.cc:654] T 17d6431cc77a491e8464d68db4081a51 P 197b880fa39d432a9b472845876bfaf0: Neither blocks nor log segments found. Creating new log.
I20260129 02:58:44.158018 20441 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "724a236524cb4e7ba3428880681d6cc2" candidate_uuid: "974b802141bd4f13beffef2aa03f88f9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "197b880fa39d432a9b472845876bfaf0" is_pre_election: true
I20260129 02:58:44.158654 20441 raft_consensus.cc:2468] T 724a236524cb4e7ba3428880681d6cc2 P 197b880fa39d432a9b472845876bfaf0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 974b802141bd4f13beffef2aa03f88f9 in term 0.
I20260129 02:58:44.160468 20479 leader_election.cc:304] T 724a236524cb4e7ba3428880681d6cc2 P 974b802141bd4f13beffef2aa03f88f9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 197b880fa39d432a9b472845876bfaf0, 974b802141bd4f13beffef2aa03f88f9; no voters:
I20260129 02:58:44.159705 20590 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "724a236524cb4e7ba3428880681d6cc2" candidate_uuid: "974b802141bd4f13beffef2aa03f88f9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c50de89a5a845668696dcb28c3badd6" is_pre_election: true
I20260129 02:58:44.161705 20590 raft_consensus.cc:2468] T 724a236524cb4e7ba3428880681d6cc2 P 1c50de89a5a845668696dcb28c3badd6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 974b802141bd4f13beffef2aa03f88f9 in term 0.
I20260129 02:58:44.163362 20653 raft_consensus.cc:2804] T 724a236524cb4e7ba3428880681d6cc2 P 974b802141bd4f13beffef2aa03f88f9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260129 02:58:44.163735 20653 raft_consensus.cc:493] T 724a236524cb4e7ba3428880681d6cc2 P 974b802141bd4f13beffef2aa03f88f9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260129 02:58:44.164100 20653 raft_consensus.cc:3060] T 724a236524cb4e7ba3428880681d6cc2 P 974b802141bd4f13beffef2aa03f88f9 [term 0 FOLLOWER]: Advancing to term 1
I20260129 02:58:44.166872 20630 tablet_bootstrap.cc:492] T 17d6431cc77a491e8464d68db4081a51 P 974b802141bd4f13beffef2aa03f88f9: No bootstrap required, opened a new log
I20260129 02:58:44.167291 20630 ts_tablet_manager.cc:1403] T 17d6431cc77a491e8464d68db4081a51 P 974b802141bd4f13beffef2aa03f88f9: Time spent bootstrapping tablet: real 0.040s user 0.018s sys 0.019s
I20260129 02:58:44.168458 20631 tablet_bootstrap.cc:492] T a609ac02fae84725ab743f95f51d33ce P 1c50de89a5a845668696dcb28c3badd6: No bootstrap required, opened a new log
I20260129 02:58:44.168908 20631 ts_tablet_manager.cc:1403] T a609ac02fae84725ab743f95f51d33ce P 1c50de89a5a845668696dcb28c3badd6: Time spent bootstrapping tablet: real 0.029s user 0.011s sys 0.017s
I20260129 02:58:44.169989 20630 raft_consensus.cc:359] T 17d6431cc77a491e8464d68db4081a51 P 974b802141bd4f13beffef2aa03f88f9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:58:44.170511 20630 raft_consensus.cc:385] T 17d6431cc77a491e8464d68db4081a51 P 974b802141bd4f13beffef2aa03f88f9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260129 02:58:44.170744 20630 raft_consensus.cc:740] T 17d6431cc77a491e8464d68db4081a51 P 974b802141bd4f13beffef2aa03f88f9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 974b802141bd4f13beffef2aa03f88f9, State: Initialized, Role: FOLLOWER
I20260129 02:58:44.171310 20630 consensus_queue.cc:260] T 17d6431cc77a491e8464d68db4081a51 P 974b802141bd4f13beffef2aa03f88f9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:58:44.171381 20631 raft_consensus.cc:359] T a609ac02fae84725ab743f95f51d33ce P 1c50de89a5a845668696dcb28c3badd6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:58:44.172094 20631 raft_consensus.cc:385] T a609ac02fae84725ab743f95f51d33ce P 1c50de89a5a845668696dcb28c3badd6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260129 02:58:44.172365 20631 raft_consensus.cc:740] T a609ac02fae84725ab743f95f51d33ce P 1c50de89a5a845668696dcb28c3badd6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1c50de89a5a845668696dcb28c3badd6, State: Initialized, Role: FOLLOWER
I20260129 02:58:44.172972 20631 consensus_queue.cc:260] T a609ac02fae84725ab743f95f51d33ce P 1c50de89a5a845668696dcb28c3badd6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:58:44.173620 20630 ts_tablet_manager.cc:1434] T 17d6431cc77a491e8464d68db4081a51 P 974b802141bd4f13beffef2aa03f88f9: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260129 02:58:44.174109 20653 raft_consensus.cc:515] T 724a236524cb4e7ba3428880681d6cc2 P 974b802141bd4f13beffef2aa03f88f9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:58:44.175035 20630 tablet_bootstrap.cc:492] T 2ac659d732314e1288849f08fe5e074e P 974b802141bd4f13beffef2aa03f88f9: Bootstrap starting.
I20260129 02:58:44.175081 20631 ts_tablet_manager.cc:1434] T a609ac02fae84725ab743f95f51d33ce P 1c50de89a5a845668696dcb28c3badd6: Time spent starting tablet: real 0.006s user 0.002s sys 0.002s
I20260129 02:58:44.176107 20631 tablet_bootstrap.cc:492] T 17d6431cc77a491e8464d68db4081a51 P 1c50de89a5a845668696dcb28c3badd6: Bootstrap starting.
I20260129 02:58:44.176795 20441 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "724a236524cb4e7ba3428880681d6cc2" candidate_uuid: "974b802141bd4f13beffef2aa03f88f9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "197b880fa39d432a9b472845876bfaf0"
I20260129 02:58:44.177343 20441 raft_consensus.cc:3060] T 724a236524cb4e7ba3428880681d6cc2 P 197b880fa39d432a9b472845876bfaf0 [term 0 FOLLOWER]: Advancing to term 1
I20260129 02:58:44.181305 20630 tablet_bootstrap.cc:654] T 2ac659d732314e1288849f08fe5e074e P 974b802141bd4f13beffef2aa03f88f9: Neither blocks nor log segments found. Creating new log.
I20260129 02:58:44.181789 20631 tablet_bootstrap.cc:654] T 17d6431cc77a491e8464d68db4081a51 P 1c50de89a5a845668696dcb28c3badd6: Neither blocks nor log segments found. Creating new log.
I20260129 02:58:44.182220 20669 raft_consensus.cc:493] T 45cc5f3d2688486d9b6dbe7e666dedde P 1c50de89a5a845668696dcb28c3badd6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260129 02:58:44.182721 20669 raft_consensus.cc:515] T 45cc5f3d2688486d9b6dbe7e666dedde P 1c50de89a5a845668696dcb28c3badd6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:58:44.184160 20658 raft_consensus.cc:493] T 2e58e9a5a8aa4cecafb464addb4f35ed P 197b880fa39d432a9b472845876bfaf0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260129 02:58:44.185017 20669 leader_election.cc:290] T 45cc5f3d2688486d9b6dbe7e666dedde P 1c50de89a5a845668696dcb28c3badd6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 197b880fa39d432a9b472845876bfaf0 (127.18.211.1:42027), 974b802141bd4f13beffef2aa03f88f9 (127.18.211.2:34517)
I20260129 02:58:44.185437 20658 raft_consensus.cc:515] T 2e58e9a5a8aa4cecafb464addb4f35ed P 197b880fa39d432a9b472845876bfaf0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:58:44.187696 20515 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "45cc5f3d2688486d9b6dbe7e666dedde" candidate_uuid: "1c50de89a5a845668696dcb28c3badd6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "974b802141bd4f13beffef2aa03f88f9" is_pre_election: true
I20260129 02:58:44.188570 20515 raft_consensus.cc:2468] T 45cc5f3d2688486d9b6dbe7e666dedde P 974b802141bd4f13beffef2aa03f88f9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1c50de89a5a845668696dcb28c3badd6 in term 0.
I20260129 02:58:44.189249 20516 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2e58e9a5a8aa4cecafb464addb4f35ed" candidate_uuid: "197b880fa39d432a9b472845876bfaf0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "974b802141bd4f13beffef2aa03f88f9" is_pre_election: true
I20260129 02:58:44.190068 20516 raft_consensus.cc:2468] T 2e58e9a5a8aa4cecafb464addb4f35ed P 974b802141bd4f13beffef2aa03f88f9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 197b880fa39d432a9b472845876bfaf0 in term 0.
I20260129 02:58:44.191457 20556 leader_election.cc:304] T 45cc5f3d2688486d9b6dbe7e666dedde P 1c50de89a5a845668696dcb28c3badd6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1c50de89a5a845668696dcb28c3badd6, 974b802141bd4f13beffef2aa03f88f9; no voters:
I20260129 02:58:44.191571 20407 leader_election.cc:304] T 2e58e9a5a8aa4cecafb464addb4f35ed P 197b880fa39d432a9b472845876bfaf0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 197b880fa39d432a9b472845876bfaf0, 974b802141bd4f13beffef2aa03f88f9; no voters:
I20260129 02:58:44.192464 20669 raft_consensus.cc:2804] T 45cc5f3d2688486d9b6dbe7e666dedde P 1c50de89a5a845668696dcb28c3badd6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260129 02:58:44.192600 20638 raft_consensus.cc:2804] T 2e58e9a5a8aa4cecafb464addb4f35ed P 197b880fa39d432a9b472845876bfaf0 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260129 02:58:44.192906 20669 raft_consensus.cc:493] T 45cc5f3d2688486d9b6dbe7e666dedde P 1c50de89a5a845668696dcb28c3badd6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260129 02:58:44.193040 20638 raft_consensus.cc:493] T 2e58e9a5a8aa4cecafb464addb4f35ed P 197b880fa39d432a9b472845876bfaf0 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260129 02:58:44.193504 20669 raft_consensus.cc:3060] T 45cc5f3d2688486d9b6dbe7e666dedde P 1c50de89a5a845668696dcb28c3badd6 [term 0 FOLLOWER]: Advancing to term 1
I20260129 02:58:44.193526 20638 raft_consensus.cc:3060] T 2e58e9a5a8aa4cecafb464addb4f35ed P 197b880fa39d432a9b472845876bfaf0 [term 0 FOLLOWER]: Advancing to term 1
I20260129 02:58:44.194203 20630 tablet_bootstrap.cc:492] T 2ac659d732314e1288849f08fe5e074e P 974b802141bd4f13beffef2aa03f88f9: No bootstrap required, opened a new log
I20260129 02:58:44.194681 20630 ts_tablet_manager.cc:1403] T 2ac659d732314e1288849f08fe5e074e P 974b802141bd4f13beffef2aa03f88f9: Time spent bootstrapping tablet: real 0.020s user 0.015s sys 0.000s
I20260129 02:58:44.197199 20630 raft_consensus.cc:359] T 2ac659d732314e1288849f08fe5e074e P 974b802141bd4f13beffef2aa03f88f9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:58:44.197872 20630 raft_consensus.cc:385] T 2ac659d732314e1288849f08fe5e074e P 974b802141bd4f13beffef2aa03f88f9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260129 02:58:44.197855 20590 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2e58e9a5a8aa4cecafb464addb4f35ed" candidate_uuid: "197b880fa39d432a9b472845876bfaf0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c50de89a5a845668696dcb28c3badd6" is_pre_election: true
I20260129 02:58:44.198333 20630 raft_consensus.cc:740] T 2ac659d732314e1288849f08fe5e074e P 974b802141bd4f13beffef2aa03f88f9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 974b802141bd4f13beffef2aa03f88f9, State: Initialized, Role: FOLLOWER
I20260129 02:58:44.198567 20590 raft_consensus.cc:2468] T 2e58e9a5a8aa4cecafb464addb4f35ed P 1c50de89a5a845668696dcb28c3badd6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 197b880fa39d432a9b472845876bfaf0 in term 0.
I20260129 02:58:44.199090 20630 consensus_queue.cc:260] T 2ac659d732314e1288849f08fe5e074e P 974b802141bd4f13beffef2aa03f88f9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:58:44.190358 20440 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "45cc5f3d2688486d9b6dbe7e666dedde" candidate_uuid: "1c50de89a5a845668696dcb28c3badd6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "197b880fa39d432a9b472845876bfaf0" is_pre_election: true
I20260129 02:58:44.200742 20658 leader_election.cc:290] T 2e58e9a5a8aa4cecafb464addb4f35ed P 197b880fa39d432a9b472845876bfaf0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 974b802141bd4f13beffef2aa03f88f9 (127.18.211.2:34517), 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3:37735)
W20260129 02:58:44.201325 20553 leader_election.cc:343] T 45cc5f3d2688486d9b6dbe7e666dedde P 1c50de89a5a845668696dcb28c3badd6 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 197b880fa39d432a9b472845876bfaf0 (127.18.211.1:42027): Illegal state: must be running to vote when last-logged opid is not known
I20260129 02:58:44.201581 20630 ts_tablet_manager.cc:1434] T 2ac659d732314e1288849f08fe5e074e P 974b802141bd4f13beffef2aa03f88f9: Time spent starting tablet: real 0.007s user 0.003s sys 0.004s
I20260129 02:58:44.202530 20630 tablet_bootstrap.cc:492] T add8d4923d1a489e86882c3a25692b95 P 974b802141bd4f13beffef2aa03f88f9: Bootstrap starting.
I20260129 02:58:44.200820 20631 tablet_bootstrap.cc:492] T 17d6431cc77a491e8464d68db4081a51 P 1c50de89a5a845668696dcb28c3badd6: No bootstrap required, opened a new log
I20260129 02:58:44.203171 20631 ts_tablet_manager.cc:1403] T 17d6431cc77a491e8464d68db4081a51 P 1c50de89a5a845668696dcb28c3badd6: Time spent bootstrapping tablet: real 0.027s user 0.016s sys 0.001s
I20260129 02:58:44.205116 20441 raft_consensus.cc:2468] T 724a236524cb4e7ba3428880681d6cc2 P 197b880fa39d432a9b472845876bfaf0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 974b802141bd4f13beffef2aa03f88f9 in term 1.
I20260129 02:58:44.206125 20479 leader_election.cc:304] T 724a236524cb4e7ba3428880681d6cc2 P 974b802141bd4f13beffef2aa03f88f9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 197b880fa39d432a9b472845876bfaf0, 974b802141bd4f13beffef2aa03f88f9; no voters:
I20260129 02:58:44.206399 20669 raft_consensus.cc:515] T 45cc5f3d2688486d9b6dbe7e666dedde P 1c50de89a5a845668696dcb28c3badd6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:58:44.207036 20685 raft_consensus.cc:2804] T 724a236524cb4e7ba3428880681d6cc2 P 974b802141bd4f13beffef2aa03f88f9 [term 1 FOLLOWER]: Leader election won for term 1
I20260129 02:58:44.207528 20685 raft_consensus.cc:697] T 724a236524cb4e7ba3428880681d6cc2 P 974b802141bd4f13beffef2aa03f88f9 [term 1 LEADER]: Becoming Leader. State: Replica: 974b802141bd4f13beffef2aa03f88f9, State: Running, Role: LEADER
I20260129 02:58:44.208257 20631 raft_consensus.cc:359] T 17d6431cc77a491e8464d68db4081a51 P 1c50de89a5a845668696dcb28c3badd6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:58:44.208390 20685 consensus_queue.cc:237] T 724a236524cb4e7ba3428880681d6cc2 P 974b802141bd4f13beffef2aa03f88f9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:58:44.212633 20631 raft_consensus.cc:385] T 17d6431cc77a491e8464d68db4081a51 P 1c50de89a5a845668696dcb28c3badd6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260129 02:58:44.212953 20633 tablet_bootstrap.cc:492] T 17d6431cc77a491e8464d68db4081a51 P 197b880fa39d432a9b472845876bfaf0: No bootstrap required, opened a new log
I20260129 02:58:44.213068 20631 raft_consensus.cc:740] T 17d6431cc77a491e8464d68db4081a51 P 1c50de89a5a845668696dcb28c3badd6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1c50de89a5a845668696dcb28c3badd6, State: Initialized, Role: FOLLOWER
I20260129 02:58:44.213415 20633 ts_tablet_manager.cc:1403] T 17d6431cc77a491e8464d68db4081a51 P 197b880fa39d432a9b472845876bfaf0: Time spent bootstrapping tablet: real 0.075s user 0.009s sys 0.001s
I20260129 02:58:44.209372 20667 raft_consensus.cc:493] T a5a05fbcbbcf4916bbc01fd1a5b5e90d P 974b802141bd4f13beffef2aa03f88f9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260129 02:58:44.213876 20631 consensus_queue.cc:260] T 17d6431cc77a491e8464d68db4081a51 P 1c50de89a5a845668696dcb28c3badd6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:58:44.214229 20667 raft_consensus.cc:515] T a5a05fbcbbcf4916bbc01fd1a5b5e90d P 974b802141bd4f13beffef2aa03f88f9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:58:44.216285 20667 leader_election.cc:290] T a5a05fbcbbcf4916bbc01fd1a5b5e90d P 974b802141bd4f13beffef2aa03f88f9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 197b880fa39d432a9b472845876bfaf0 (127.18.211.1:42027), 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3:37735)
I20260129 02:58:44.217633 20667 raft_consensus.cc:493] T b189f08e16a74a11980cc55d99359a0a P 974b802141bd4f13beffef2aa03f88f9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260129 02:58:44.200748 20638 raft_consensus.cc:515] T 2e58e9a5a8aa4cecafb464addb4f35ed P 197b880fa39d432a9b472845876bfaf0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:58:44.218108 20440 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a5a05fbcbbcf4916bbc01fd1a5b5e90d" candidate_uuid: "974b802141bd4f13beffef2aa03f88f9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "197b880fa39d432a9b472845876bfaf0" is_pre_election: true
I20260129 02:58:44.218830 20440 raft_consensus.cc:2468] T a5a05fbcbbcf4916bbc01fd1a5b5e90d P 197b880fa39d432a9b472845876bfaf0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 974b802141bd4f13beffef2aa03f88f9 in term 0.
I20260129 02:58:44.210856 20441 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "45cc5f3d2688486d9b6dbe7e666dedde" candidate_uuid: "1c50de89a5a845668696dcb28c3badd6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "197b880fa39d432a9b472845876bfaf0"
I20260129 02:58:44.211970 20669 leader_election.cc:290] T 45cc5f3d2688486d9b6dbe7e666dedde P 1c50de89a5a845668696dcb28c3badd6 [CANDIDATE]: Term 1 election: Requested vote from peers 197b880fa39d432a9b472845876bfaf0 (127.18.211.1:42027), 974b802141bd4f13beffef2aa03f88f9 (127.18.211.2:34517)
I20260129 02:58:44.220064 20479 leader_election.cc:304] T a5a05fbcbbcf4916bbc01fd1a5b5e90d P 974b802141bd4f13beffef2aa03f88f9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 197b880fa39d432a9b472845876bfaf0, 974b802141bd4f13beffef2aa03f88f9; no voters:
I20260129 02:58:44.220439 20631 ts_tablet_manager.cc:1434] T 17d6431cc77a491e8464d68db4081a51 P 1c50de89a5a845668696dcb28c3badd6: Time spent starting tablet: real 0.017s user 0.002s sys 0.007s
W20260129 02:58:44.221187 20553 leader_election.cc:343] T 45cc5f3d2688486d9b6dbe7e666dedde P 1c50de89a5a845668696dcb28c3badd6 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 197b880fa39d432a9b472845876bfaf0 (127.18.211.1:42027): Illegal state: must be running to vote when last-logged opid is not known
I20260129 02:58:44.221976 20590 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a5a05fbcbbcf4916bbc01fd1a5b5e90d" candidate_uuid: "974b802141bd4f13beffef2aa03f88f9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c50de89a5a845668696dcb28c3badd6" is_pre_election: true
I20260129 02:58:44.222445 20516 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2e58e9a5a8aa4cecafb464addb4f35ed" candidate_uuid: "197b880fa39d432a9b472845876bfaf0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "974b802141bd4f13beffef2aa03f88f9"
I20260129 02:58:44.222599 20590 raft_consensus.cc:2468] T a5a05fbcbbcf4916bbc01fd1a5b5e90d P 1c50de89a5a845668696dcb28c3badd6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 974b802141bd4f13beffef2aa03f88f9 in term 0.
I20260129 02:58:44.222801 20638 leader_election.cc:290] T 2e58e9a5a8aa4cecafb464addb4f35ed P 197b880fa39d432a9b472845876bfaf0 [CANDIDATE]: Term 1 election: Requested vote from peers 974b802141bd4f13beffef2aa03f88f9 (127.18.211.2:34517), 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3:37735)
I20260129 02:58:44.222975 20516 raft_consensus.cc:3060] T 2e58e9a5a8aa4cecafb464addb4f35ed P 974b802141bd4f13beffef2aa03f88f9 [term 0 FOLLOWER]: Advancing to term 1
I20260129 02:58:44.222612 20589 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2e58e9a5a8aa4cecafb464addb4f35ed" candidate_uuid: "197b880fa39d432a9b472845876bfaf0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c50de89a5a845668696dcb28c3badd6"
I20260129 02:58:44.223769 20589 raft_consensus.cc:3060] T 2e58e9a5a8aa4cecafb464addb4f35ed P 1c50de89a5a845668696dcb28c3badd6 [term 0 FOLLOWER]: Advancing to term 1
I20260129 02:58:44.226049 20653 leader_election.cc:290] T 724a236524cb4e7ba3428880681d6cc2 P 974b802141bd4f13beffef2aa03f88f9 [CANDIDATE]: Term 1 election: Requested vote from peers 197b880fa39d432a9b472845876bfaf0 (127.18.211.1:42027), 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3:37735)
I20260129 02:58:44.226598 20653 raft_consensus.cc:2804] T a5a05fbcbbcf4916bbc01fd1a5b5e90d P 974b802141bd4f13beffef2aa03f88f9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260129 02:58:44.226929 20653 raft_consensus.cc:493] T a5a05fbcbbcf4916bbc01fd1a5b5e90d P 974b802141bd4f13beffef2aa03f88f9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260129 02:58:44.227252 20653 raft_consensus.cc:3060] T a5a05fbcbbcf4916bbc01fd1a5b5e90d P 974b802141bd4f13beffef2aa03f88f9 [term 0 FOLLOWER]: Advancing to term 1
I20260129 02:58:44.229588 20590 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "724a236524cb4e7ba3428880681d6cc2" candidate_uuid: "974b802141bd4f13beffef2aa03f88f9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c50de89a5a845668696dcb28c3badd6"
I20260129 02:58:44.230182 20590 raft_consensus.cc:3060] T 724a236524cb4e7ba3428880681d6cc2 P 1c50de89a5a845668696dcb28c3badd6 [term 0 FOLLOWER]: Advancing to term 1
I20260129 02:58:44.231046 20516 raft_consensus.cc:2468] T 2e58e9a5a8aa4cecafb464addb4f35ed P 974b802141bd4f13beffef2aa03f88f9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 197b880fa39d432a9b472845876bfaf0 in term 1.
I20260129 02:58:44.212208 20630 tablet_bootstrap.cc:654] T add8d4923d1a489e86882c3a25692b95 P 974b802141bd4f13beffef2aa03f88f9: Neither blocks nor log segments found. Creating new log.
I20260129 02:58:44.218113 20667 raft_consensus.cc:515] T b189f08e16a74a11980cc55d99359a0a P 974b802141bd4f13beffef2aa03f88f9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:58:44.232376 20407 leader_election.cc:304] T 2e58e9a5a8aa4cecafb464addb4f35ed P 197b880fa39d432a9b472845876bfaf0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 197b880fa39d432a9b472845876bfaf0, 974b802141bd4f13beffef2aa03f88f9; no voters:
I20260129 02:58:44.234887 20667 leader_election.cc:290] T b189f08e16a74a11980cc55d99359a0a P 974b802141bd4f13beffef2aa03f88f9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 197b880fa39d432a9b472845876bfaf0 (127.18.211.1:42027), 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3:37735)
I20260129 02:58:44.234835 20441 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b189f08e16a74a11980cc55d99359a0a" candidate_uuid: "974b802141bd4f13beffef2aa03f88f9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "197b880fa39d432a9b472845876bfaf0" is_pre_election: true
I20260129 02:58:44.235275 20588 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b189f08e16a74a11980cc55d99359a0a" candidate_uuid: "974b802141bd4f13beffef2aa03f88f9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c50de89a5a845668696dcb28c3badd6" is_pre_election: true
W20260129 02:58:44.236399 20479 leader_election.cc:343] T b189f08e16a74a11980cc55d99359a0a P 974b802141bd4f13beffef2aa03f88f9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 197b880fa39d432a9b472845876bfaf0 (127.18.211.1:42027): Illegal state: must be running to vote when last-logged opid is not known
W20260129 02:58:44.236924 20480 leader_election.cc:343] T b189f08e16a74a11980cc55d99359a0a P 974b802141bd4f13beffef2aa03f88f9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3:37735): Illegal state: must be running to vote when last-logged opid is not known
I20260129 02:58:44.237334 20480 leader_election.cc:304] T b189f08e16a74a11980cc55d99359a0a P 974b802141bd4f13beffef2aa03f88f9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 974b802141bd4f13beffef2aa03f88f9; no voters: 197b880fa39d432a9b472845876bfaf0, 1c50de89a5a845668696dcb28c3badd6
I20260129 02:58:44.233208 20638 raft_consensus.cc:2804] T 2e58e9a5a8aa4cecafb464addb4f35ed P 197b880fa39d432a9b472845876bfaf0 [term 1 FOLLOWER]: Leader election won for term 1
I20260129 02:58:44.238469 20667 raft_consensus.cc:2749] T b189f08e16a74a11980cc55d99359a0a P 974b802141bd4f13beffef2aa03f88f9 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260129 02:58:44.238569 20653 raft_consensus.cc:515] T a5a05fbcbbcf4916bbc01fd1a5b5e90d P 974b802141bd4f13beffef2aa03f88f9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:58:44.241331 20441 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a5a05fbcbbcf4916bbc01fd1a5b5e90d" candidate_uuid: "974b802141bd4f13beffef2aa03f88f9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "197b880fa39d432a9b472845876bfaf0"
I20260129 02:58:44.241614 20588 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a5a05fbcbbcf4916bbc01fd1a5b5e90d" candidate_uuid: "974b802141bd4f13beffef2aa03f88f9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c50de89a5a845668696dcb28c3badd6"
I20260129 02:58:44.241914 20441 raft_consensus.cc:3060] T a5a05fbcbbcf4916bbc01fd1a5b5e90d P 197b880fa39d432a9b472845876bfaf0 [term 0 FOLLOWER]: Advancing to term 1
I20260129 02:58:44.242274 20588 raft_consensus.cc:3060] T a5a05fbcbbcf4916bbc01fd1a5b5e90d P 1c50de89a5a845668696dcb28c3badd6 [term 0 FOLLOWER]: Advancing to term 1
I20260129 02:58:44.241729 20337 catalog_manager.cc:5654] T 724a236524cb4e7ba3428880681d6cc2 P 974b802141bd4f13beffef2aa03f88f9 reported cstate change: term changed from 0 to 1, leader changed from <none> to 974b802141bd4f13beffef2aa03f88f9 (127.18.211.2). New cstate: current_term: 1 leader_uuid: "974b802141bd4f13beffef2aa03f88f9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } health_report { overall_health: UNKNOWN } } }
I20260129 02:58:44.243690 20589 raft_consensus.cc:2468] T 2e58e9a5a8aa4cecafb464addb4f35ed P 1c50de89a5a845668696dcb28c3badd6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 197b880fa39d432a9b472845876bfaf0 in term 1.
I20260129 02:58:44.244968 20638 raft_consensus.cc:697] T 2e58e9a5a8aa4cecafb464addb4f35ed P 197b880fa39d432a9b472845876bfaf0 [term 1 LEADER]: Becoming Leader. State: Replica: 197b880fa39d432a9b472845876bfaf0, State: Running, Role: LEADER
I20260129 02:58:44.246296 20590 raft_consensus.cc:2468] T 724a236524cb4e7ba3428880681d6cc2 P 1c50de89a5a845668696dcb28c3badd6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 974b802141bd4f13beffef2aa03f88f9 in term 1.
I20260129 02:58:44.247654 20631 tablet_bootstrap.cc:492] T 2ac659d732314e1288849f08fe5e074e P 1c50de89a5a845668696dcb28c3badd6: Bootstrap starting.
I20260129 02:58:44.245882 20638 consensus_queue.cc:237] T 2e58e9a5a8aa4cecafb464addb4f35ed P 197b880fa39d432a9b472845876bfaf0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:58:44.249444 20588 raft_consensus.cc:2468] T a5a05fbcbbcf4916bbc01fd1a5b5e90d P 1c50de89a5a845668696dcb28c3badd6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 974b802141bd4f13beffef2aa03f88f9 in term 1.
I20260129 02:58:44.250540 20480 leader_election.cc:304] T a5a05fbcbbcf4916bbc01fd1a5b5e90d P 974b802141bd4f13beffef2aa03f88f9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1c50de89a5a845668696dcb28c3badd6, 974b802141bd4f13beffef2aa03f88f9; no voters:
I20260129 02:58:44.255045 20667 raft_consensus.cc:2804] T a5a05fbcbbcf4916bbc01fd1a5b5e90d P 974b802141bd4f13beffef2aa03f88f9 [term 1 FOLLOWER]: Leader election won for term 1
I20260129 02:58:44.255586 20669 raft_consensus.cc:493] T 3e2d15f8640b407195210c476e09df91 P 1c50de89a5a845668696dcb28c3badd6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260129 02:58:44.261217 20669 raft_consensus.cc:515] T 3e2d15f8640b407195210c476e09df91 P 1c50de89a5a845668696dcb28c3badd6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:58:44.263649 20440 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3e2d15f8640b407195210c476e09df91" candidate_uuid: "1c50de89a5a845668696dcb28c3badd6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "197b880fa39d432a9b472845876bfaf0" is_pre_election: true
I20260129 02:58:44.264010 20515 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3e2d15f8640b407195210c476e09df91" candidate_uuid: "1c50de89a5a845668696dcb28c3badd6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "974b802141bd4f13beffef2aa03f88f9" is_pre_election: true
I20260129 02:58:44.264349 20440 raft_consensus.cc:2468] T 3e2d15f8640b407195210c476e09df91 P 197b880fa39d432a9b472845876bfaf0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1c50de89a5a845668696dcb28c3badd6 in term 0.
I20260129 02:58:44.264606 20515 raft_consensus.cc:2468] T 3e2d15f8640b407195210c476e09df91 P 974b802141bd4f13beffef2aa03f88f9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1c50de89a5a845668696dcb28c3badd6 in term 0.
I20260129 02:58:44.264504 20633 raft_consensus.cc:359] T 17d6431cc77a491e8464d68db4081a51 P 197b880fa39d432a9b472845876bfaf0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:58:44.265656 20553 leader_election.cc:304] T 3e2d15f8640b407195210c476e09df91 P 1c50de89a5a845668696dcb28c3badd6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 197b880fa39d432a9b472845876bfaf0, 1c50de89a5a845668696dcb28c3badd6; no voters:
I20260129 02:58:44.266484 20641 raft_consensus.cc:2804] T 3e2d15f8640b407195210c476e09df91 P 1c50de89a5a845668696dcb28c3badd6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260129 02:58:44.266853 20641 raft_consensus.cc:493] T 3e2d15f8640b407195210c476e09df91 P 1c50de89a5a845668696dcb28c3badd6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260129 02:58:44.267168 20641 raft_consensus.cc:3060] T 3e2d15f8640b407195210c476e09df91 P 1c50de89a5a845668696dcb28c3badd6 [term 0 FOLLOWER]: Advancing to term 1
I20260129 02:58:44.271409 20669 leader_election.cc:290] T 3e2d15f8640b407195210c476e09df91 P 1c50de89a5a845668696dcb28c3badd6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 197b880fa39d432a9b472845876bfaf0 (127.18.211.1:42027), 974b802141bd4f13beffef2aa03f88f9 (127.18.211.2:34517)
I20260129 02:58:44.256026 20653 leader_election.cc:290] T a5a05fbcbbcf4916bbc01fd1a5b5e90d P 974b802141bd4f13beffef2aa03f88f9 [CANDIDATE]: Term 1 election: Requested vote from peers 197b880fa39d432a9b472845876bfaf0 (127.18.211.1:42027), 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3:37735)
I20260129 02:58:44.256434 20516 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "45cc5f3d2688486d9b6dbe7e666dedde" candidate_uuid: "1c50de89a5a845668696dcb28c3badd6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "974b802141bd4f13beffef2aa03f88f9"
I20260129 02:58:44.272652 20516 raft_consensus.cc:3060] T 45cc5f3d2688486d9b6dbe7e666dedde P 974b802141bd4f13beffef2aa03f88f9 [term 0 FOLLOWER]: Advancing to term 1
I20260129 02:58:44.265552 20633 raft_consensus.cc:385] T 17d6431cc77a491e8464d68db4081a51 P 197b880fa39d432a9b472845876bfaf0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260129 02:58:44.273782 20633 raft_consensus.cc:740] T 17d6431cc77a491e8464d68db4081a51 P 197b880fa39d432a9b472845876bfaf0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 197b880fa39d432a9b472845876bfaf0, State: Initialized, Role: FOLLOWER
I20260129 02:58:44.257901 20441 raft_consensus.cc:2468] T a5a05fbcbbcf4916bbc01fd1a5b5e90d P 197b880fa39d432a9b472845876bfaf0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 974b802141bd4f13beffef2aa03f88f9 in term 1.
I20260129 02:58:44.279487 20641 raft_consensus.cc:515] T 3e2d15f8640b407195210c476e09df91 P 1c50de89a5a845668696dcb28c3badd6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:58:44.281481 20667 raft_consensus.cc:697] T a5a05fbcbbcf4916bbc01fd1a5b5e90d P 974b802141bd4f13beffef2aa03f88f9 [term 1 LEADER]: Becoming Leader. State: Replica: 974b802141bd4f13beffef2aa03f88f9, State: Running, Role: LEADER
I20260129 02:58:44.282491 20667 consensus_queue.cc:237] T a5a05fbcbbcf4916bbc01fd1a5b5e90d P 974b802141bd4f13beffef2aa03f88f9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:58:44.283042 20441 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3e2d15f8640b407195210c476e09df91" candidate_uuid: "1c50de89a5a845668696dcb28c3badd6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "197b880fa39d432a9b472845876bfaf0"
I20260129 02:58:44.283797 20441 raft_consensus.cc:3060] T 3e2d15f8640b407195210c476e09df91 P 197b880fa39d432a9b472845876bfaf0 [term 0 FOLLOWER]: Advancing to term 1
I20260129 02:58:44.260854 20631 tablet_bootstrap.cc:654] T 2ac659d732314e1288849f08fe5e074e P 1c50de89a5a845668696dcb28c3badd6: Neither blocks nor log segments found. Creating new log.
I20260129 02:58:44.292961 20337 catalog_manager.cc:5654] T 2e58e9a5a8aa4cecafb464addb4f35ed P 197b880fa39d432a9b472845876bfaf0 reported cstate change: term changed from 0 to 1, leader changed from <none> to 197b880fa39d432a9b472845876bfaf0 (127.18.211.1). New cstate: current_term: 1 leader_uuid: "197b880fa39d432a9b472845876bfaf0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } health_report { overall_health: UNKNOWN } } }
I20260129 02:58:44.296411 20515 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3e2d15f8640b407195210c476e09df91" candidate_uuid: "1c50de89a5a845668696dcb28c3badd6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "974b802141bd4f13beffef2aa03f88f9"
I20260129 02:58:44.297246 20515 raft_consensus.cc:3060] T 3e2d15f8640b407195210c476e09df91 P 974b802141bd4f13beffef2aa03f88f9 [term 0 FOLLOWER]: Advancing to term 1
I20260129 02:58:44.308036 20641 leader_election.cc:290] T 3e2d15f8640b407195210c476e09df91 P 1c50de89a5a845668696dcb28c3badd6 [CANDIDATE]: Term 1 election: Requested vote from peers 197b880fa39d432a9b472845876bfaf0 (127.18.211.1:42027), 974b802141bd4f13beffef2aa03f88f9 (127.18.211.2:34517)
I20260129 02:58:44.312541 20441 raft_consensus.cc:2468] T 3e2d15f8640b407195210c476e09df91 P 197b880fa39d432a9b472845876bfaf0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1c50de89a5a845668696dcb28c3badd6 in term 1.
I20260129 02:58:44.279490 20516 raft_consensus.cc:2468] T 45cc5f3d2688486d9b6dbe7e666dedde P 974b802141bd4f13beffef2aa03f88f9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1c50de89a5a845668696dcb28c3badd6 in term 1.
I20260129 02:58:44.279489 20633 consensus_queue.cc:260] T 17d6431cc77a491e8464d68db4081a51 P 197b880fa39d432a9b472845876bfaf0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:58:44.314237 20553 leader_election.cc:304] T 3e2d15f8640b407195210c476e09df91 P 1c50de89a5a845668696dcb28c3badd6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 197b880fa39d432a9b472845876bfaf0, 1c50de89a5a845668696dcb28c3badd6; no voters:
I20260129 02:58:44.316030 20556 leader_election.cc:304] T 45cc5f3d2688486d9b6dbe7e666dedde P 1c50de89a5a845668696dcb28c3badd6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1c50de89a5a845668696dcb28c3badd6, 974b802141bd4f13beffef2aa03f88f9; no voters: 197b880fa39d432a9b472845876bfaf0
I20260129 02:58:44.317137 20669 raft_consensus.cc:2804] T 3e2d15f8640b407195210c476e09df91 P 1c50de89a5a845668696dcb28c3badd6 [term 1 FOLLOWER]: Leader election won for term 1
I20260129 02:58:44.318189 20641 raft_consensus.cc:2804] T 45cc5f3d2688486d9b6dbe7e666dedde P 1c50de89a5a845668696dcb28c3badd6 [term 1 FOLLOWER]: Leader election won for term 1
I20260129 02:58:44.312638 20515 raft_consensus.cc:2468] T 3e2d15f8640b407195210c476e09df91 P 974b802141bd4f13beffef2aa03f88f9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1c50de89a5a845668696dcb28c3badd6 in term 1.
I20260129 02:58:44.318823 20630 tablet_bootstrap.cc:492] T add8d4923d1a489e86882c3a25692b95 P 974b802141bd4f13beffef2aa03f88f9: No bootstrap required, opened a new log
I20260129 02:58:44.322937 20630 ts_tablet_manager.cc:1403] T add8d4923d1a489e86882c3a25692b95 P 974b802141bd4f13beffef2aa03f88f9: Time spent bootstrapping tablet: real 0.121s user 0.043s sys 0.031s
I20260129 02:58:44.325624 20630 raft_consensus.cc:359] T add8d4923d1a489e86882c3a25692b95 P 974b802141bd4f13beffef2aa03f88f9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:58:44.326472 20630 raft_consensus.cc:385] T add8d4923d1a489e86882c3a25692b95 P 974b802141bd4f13beffef2aa03f88f9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260129 02:58:44.326821 20630 raft_consensus.cc:740] T add8d4923d1a489e86882c3a25692b95 P 974b802141bd4f13beffef2aa03f88f9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 974b802141bd4f13beffef2aa03f88f9, State: Initialized, Role: FOLLOWER
I20260129 02:58:44.326941 20641 raft_consensus.cc:697] T 45cc5f3d2688486d9b6dbe7e666dedde P 1c50de89a5a845668696dcb28c3badd6 [term 1 LEADER]: Becoming Leader. State: Replica: 1c50de89a5a845668696dcb28c3badd6, State: Running, Role: LEADER
I20260129 02:58:44.328258 20638 raft_consensus.cc:493] T 0f20af7fbe5d4db89a6f17948d98912c P 197b880fa39d432a9b472845876bfaf0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260129 02:58:44.328761 20638 raft_consensus.cc:515] T 0f20af7fbe5d4db89a6f17948d98912c P 197b880fa39d432a9b472845876bfaf0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:58:44.322425 20633 ts_tablet_manager.cc:1434] T 17d6431cc77a491e8464d68db4081a51 P 197b880fa39d432a9b472845876bfaf0: Time spent starting tablet: real 0.109s user 0.005s sys 0.002s
I20260129 02:58:44.330852 20638 leader_election.cc:290] T 0f20af7fbe5d4db89a6f17948d98912c P 197b880fa39d432a9b472845876bfaf0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 974b802141bd4f13beffef2aa03f88f9 (127.18.211.2:34517), 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3:37735)
I20260129 02:58:44.331605 20633 tablet_bootstrap.cc:492] T 45cc5f3d2688486d9b6dbe7e666dedde P 197b880fa39d432a9b472845876bfaf0: Bootstrap starting.
I20260129 02:58:44.328241 20641 consensus_queue.cc:237] T 45cc5f3d2688486d9b6dbe7e666dedde P 1c50de89a5a845668696dcb28c3badd6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:58:44.331799 20515 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0f20af7fbe5d4db89a6f17948d98912c" candidate_uuid: "197b880fa39d432a9b472845876bfaf0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "974b802141bd4f13beffef2aa03f88f9" is_pre_election: true
I20260129 02:58:44.332604 20515 raft_consensus.cc:2468] T 0f20af7fbe5d4db89a6f17948d98912c P 974b802141bd4f13beffef2aa03f88f9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 197b880fa39d432a9b472845876bfaf0 in term 0.
I20260129 02:58:44.332986 20588 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0f20af7fbe5d4db89a6f17948d98912c" candidate_uuid: "197b880fa39d432a9b472845876bfaf0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c50de89a5a845668696dcb28c3badd6" is_pre_election: true
I20260129 02:58:44.333576 20588 raft_consensus.cc:2468] T 0f20af7fbe5d4db89a6f17948d98912c P 1c50de89a5a845668696dcb28c3badd6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 197b880fa39d432a9b472845876bfaf0 in term 0.
I20260129 02:58:44.333722 20407 leader_election.cc:304] T 0f20af7fbe5d4db89a6f17948d98912c P 197b880fa39d432a9b472845876bfaf0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 197b880fa39d432a9b472845876bfaf0, 974b802141bd4f13beffef2aa03f88f9; no voters:
I20260129 02:58:44.334719 20638 raft_consensus.cc:2804] T 0f20af7fbe5d4db89a6f17948d98912c P 197b880fa39d432a9b472845876bfaf0 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260129 02:58:44.335099 20638 raft_consensus.cc:493] T 0f20af7fbe5d4db89a6f17948d98912c P 197b880fa39d432a9b472845876bfaf0 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260129 02:58:44.335426 20638 raft_consensus.cc:3060] T 0f20af7fbe5d4db89a6f17948d98912c P 197b880fa39d432a9b472845876bfaf0 [term 0 FOLLOWER]: Advancing to term 1
I20260129 02:58:44.335945 20336 catalog_manager.cc:5654] T a5a05fbcbbcf4916bbc01fd1a5b5e90d P 974b802141bd4f13beffef2aa03f88f9 reported cstate change: term changed from 0 to 1, leader changed from <none> to 974b802141bd4f13beffef2aa03f88f9 (127.18.211.2). New cstate: current_term: 1 leader_uuid: "974b802141bd4f13beffef2aa03f88f9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } health_report { overall_health: UNKNOWN } } }
I20260129 02:58:44.337126 20631 tablet_bootstrap.cc:492] T 2ac659d732314e1288849f08fe5e074e P 1c50de89a5a845668696dcb28c3badd6: No bootstrap required, opened a new log
I20260129 02:58:44.337551 20631 ts_tablet_manager.cc:1403] T 2ac659d732314e1288849f08fe5e074e P 1c50de89a5a845668696dcb28c3badd6: Time spent bootstrapping tablet: real 0.090s user 0.006s sys 0.005s
I20260129 02:58:44.340090 20631 raft_consensus.cc:359] T 2ac659d732314e1288849f08fe5e074e P 1c50de89a5a845668696dcb28c3badd6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:58:44.340763 20631 raft_consensus.cc:385] T 2ac659d732314e1288849f08fe5e074e P 1c50de89a5a845668696dcb28c3badd6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260129 02:58:44.341060 20631 raft_consensus.cc:740] T 2ac659d732314e1288849f08fe5e074e P 1c50de89a5a845668696dcb28c3badd6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1c50de89a5a845668696dcb28c3badd6, State: Initialized, Role: FOLLOWER
I20260129 02:58:44.341678 20631 consensus_queue.cc:260] T 2ac659d732314e1288849f08fe5e074e P 1c50de89a5a845668696dcb28c3badd6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:58:44.342659 20669 raft_consensus.cc:697] T 3e2d15f8640b407195210c476e09df91 P 1c50de89a5a845668696dcb28c3badd6 [term 1 LEADER]: Becoming Leader. State: Replica: 1c50de89a5a845668696dcb28c3badd6, State: Running, Role: LEADER
I20260129 02:58:44.343575 20669 consensus_queue.cc:237] T 3e2d15f8640b407195210c476e09df91 P 1c50de89a5a845668696dcb28c3badd6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:58:44.327633 20630 consensus_queue.cc:260] T add8d4923d1a489e86882c3a25692b95 P 974b802141bd4f13beffef2aa03f88f9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:58:44.347035 20633 tablet_bootstrap.cc:654] T 45cc5f3d2688486d9b6dbe7e666dedde P 197b880fa39d432a9b472845876bfaf0: Neither blocks nor log segments found. Creating new log.
I20260129 02:58:44.349339 20630 ts_tablet_manager.cc:1434] T add8d4923d1a489e86882c3a25692b95 P 974b802141bd4f13beffef2aa03f88f9: Time spent starting tablet: real 0.026s user 0.006s sys 0.000s
I20260129 02:58:44.351012 20631 ts_tablet_manager.cc:1434] T 2ac659d732314e1288849f08fe5e074e P 1c50de89a5a845668696dcb28c3badd6: Time spent starting tablet: real 0.013s user 0.008s sys 0.000s
I20260129 02:58:44.351289 20653 raft_consensus.cc:493] T 17d6431cc77a491e8464d68db4081a51 P 974b802141bd4f13beffef2aa03f88f9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260129 02:58:44.351966 20631 tablet_bootstrap.cc:492] T b189f08e16a74a11980cc55d99359a0a P 1c50de89a5a845668696dcb28c3badd6: Bootstrap starting.
I20260129 02:58:44.351800 20653 raft_consensus.cc:515] T 17d6431cc77a491e8464d68db4081a51 P 974b802141bd4f13beffef2aa03f88f9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:58:44.353793 20653 leader_election.cc:290] T 17d6431cc77a491e8464d68db4081a51 P 974b802141bd4f13beffef2aa03f88f9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 197b880fa39d432a9b472845876bfaf0 (127.18.211.1:42027), 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3:37735)
I20260129 02:58:44.354045 20630 tablet_bootstrap.cc:492] T 4e420f90884f4796997b5ea41e49e411 P 974b802141bd4f13beffef2aa03f88f9: Bootstrap starting.
I20260129 02:58:44.354841 20588 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "17d6431cc77a491e8464d68db4081a51" candidate_uuid: "974b802141bd4f13beffef2aa03f88f9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c50de89a5a845668696dcb28c3badd6" is_pre_election: true
I20260129 02:58:44.355324 20441 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "17d6431cc77a491e8464d68db4081a51" candidate_uuid: "974b802141bd4f13beffef2aa03f88f9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "197b880fa39d432a9b472845876bfaf0" is_pre_election: true
I20260129 02:58:44.355446 20588 raft_consensus.cc:2468] T 17d6431cc77a491e8464d68db4081a51 P 1c50de89a5a845668696dcb28c3badd6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 974b802141bd4f13beffef2aa03f88f9 in term 0.
I20260129 02:58:44.355995 20441 raft_consensus.cc:2468] T 17d6431cc77a491e8464d68db4081a51 P 197b880fa39d432a9b472845876bfaf0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 974b802141bd4f13beffef2aa03f88f9 in term 0.
I20260129 02:58:44.356750 20480 leader_election.cc:304] T 17d6431cc77a491e8464d68db4081a51 P 974b802141bd4f13beffef2aa03f88f9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1c50de89a5a845668696dcb28c3badd6, 974b802141bd4f13beffef2aa03f88f9; no voters:
I20260129 02:58:44.358093 20653 raft_consensus.cc:2804] T 17d6431cc77a491e8464d68db4081a51 P 974b802141bd4f13beffef2aa03f88f9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260129 02:58:44.358500 20653 raft_consensus.cc:493] T 17d6431cc77a491e8464d68db4081a51 P 974b802141bd4f13beffef2aa03f88f9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260129 02:58:44.358948 20653 raft_consensus.cc:3060] T 17d6431cc77a491e8464d68db4081a51 P 974b802141bd4f13beffef2aa03f88f9 [term 0 FOLLOWER]: Advancing to term 1
I20260129 02:58:44.364490 20630 tablet_bootstrap.cc:654] T 4e420f90884f4796997b5ea41e49e411 P 974b802141bd4f13beffef2aa03f88f9: Neither blocks nor log segments found. Creating new log.
I20260129 02:58:44.350171 20721 consensus_queue.cc:1048] T e7d83d4ccf684e76acb29a0dc844e1b8 P 197b880fa39d432a9b472845876bfaf0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260129 02:58:44.347373 20638 raft_consensus.cc:515] T 0f20af7fbe5d4db89a6f17948d98912c P 197b880fa39d432a9b472845876bfaf0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:58:44.368413 20516 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0f20af7fbe5d4db89a6f17948d98912c" candidate_uuid: "197b880fa39d432a9b472845876bfaf0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "974b802141bd4f13beffef2aa03f88f9"
I20260129 02:58:44.368839 20588 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0f20af7fbe5d4db89a6f17948d98912c" candidate_uuid: "197b880fa39d432a9b472845876bfaf0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c50de89a5a845668696dcb28c3badd6"
I20260129 02:58:44.369036 20516 raft_consensus.cc:3060] T 0f20af7fbe5d4db89a6f17948d98912c P 974b802141bd4f13beffef2aa03f88f9 [term 0 FOLLOWER]: Advancing to term 1
I20260129 02:58:44.369383 20588 raft_consensus.cc:3060] T 0f20af7fbe5d4db89a6f17948d98912c P 1c50de89a5a845668696dcb28c3badd6 [term 0 FOLLOWER]: Advancing to term 1
I20260129 02:58:44.374282 20631 tablet_bootstrap.cc:654] T b189f08e16a74a11980cc55d99359a0a P 1c50de89a5a845668696dcb28c3badd6: Neither blocks nor log segments found. Creating new log.
I20260129 02:58:44.376034 20588 raft_consensus.cc:2468] T 0f20af7fbe5d4db89a6f17948d98912c P 1c50de89a5a845668696dcb28c3badd6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 197b880fa39d432a9b472845876bfaf0 in term 1.
I20260129 02:58:44.376255 20516 raft_consensus.cc:2468] T 0f20af7fbe5d4db89a6f17948d98912c P 974b802141bd4f13beffef2aa03f88f9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 197b880fa39d432a9b472845876bfaf0 in term 1.
I20260129 02:58:44.376355 20667 raft_consensus.cc:493] T 2ac659d732314e1288849f08fe5e074e P 974b802141bd4f13beffef2aa03f88f9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260129 02:58:44.377240 20405 leader_election.cc:304] T 0f20af7fbe5d4db89a6f17948d98912c P 197b880fa39d432a9b472845876bfaf0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 197b880fa39d432a9b472845876bfaf0, 1c50de89a5a845668696dcb28c3badd6; no voters:
I20260129 02:58:44.377046 20667 raft_consensus.cc:515] T 2ac659d732314e1288849f08fe5e074e P 974b802141bd4f13beffef2aa03f88f9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:58:44.378118 20721 raft_consensus.cc:2804] T 0f20af7fbe5d4db89a6f17948d98912c P 197b880fa39d432a9b472845876bfaf0 [term 1 FOLLOWER]: Leader election won for term 1
I20260129 02:58:44.378309 20638 leader_election.cc:290] T 0f20af7fbe5d4db89a6f17948d98912c P 197b880fa39d432a9b472845876bfaf0 [CANDIDATE]: Term 1 election: Requested vote from peers 974b802141bd4f13beffef2aa03f88f9 (127.18.211.2:34517), 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3:37735)
I20260129 02:58:44.379372 20721 raft_consensus.cc:697] T 0f20af7fbe5d4db89a6f17948d98912c P 197b880fa39d432a9b472845876bfaf0 [term 1 LEADER]: Becoming Leader. State: Replica: 197b880fa39d432a9b472845876bfaf0, State: Running, Role: LEADER
I20260129 02:58:44.379995 20441 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2ac659d732314e1288849f08fe5e074e" candidate_uuid: "974b802141bd4f13beffef2aa03f88f9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "197b880fa39d432a9b472845876bfaf0" is_pre_election: true
I20260129 02:58:44.380300 20721 consensus_queue.cc:237] T 0f20af7fbe5d4db89a6f17948d98912c P 197b880fa39d432a9b472845876bfaf0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:58:44.371174 20653 raft_consensus.cc:515] T 17d6431cc77a491e8464d68db4081a51 P 974b802141bd4f13beffef2aa03f88f9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:58:44.380879 20667 leader_election.cc:290] T 2ac659d732314e1288849f08fe5e074e P 974b802141bd4f13beffef2aa03f88f9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 197b880fa39d432a9b472845876bfaf0 (127.18.211.1:42027), 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3:37735)
W20260129 02:58:44.381381 20479 leader_election.cc:343] T 2ac659d732314e1288849f08fe5e074e P 974b802141bd4f13beffef2aa03f88f9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 197b880fa39d432a9b472845876bfaf0 (127.18.211.1:42027): Illegal state: must be running to vote when last-logged opid is not known
I20260129 02:58:44.383464 20441 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "17d6431cc77a491e8464d68db4081a51" candidate_uuid: "974b802141bd4f13beffef2aa03f88f9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "197b880fa39d432a9b472845876bfaf0"
I20260129 02:58:44.383811 20590 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "17d6431cc77a491e8464d68db4081a51" candidate_uuid: "974b802141bd4f13beffef2aa03f88f9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c50de89a5a845668696dcb28c3badd6"
I20260129 02:58:44.384063 20441 raft_consensus.cc:3060] T 17d6431cc77a491e8464d68db4081a51 P 197b880fa39d432a9b472845876bfaf0 [term 0 FOLLOWER]: Advancing to term 1
I20260129 02:58:44.384426 20590 raft_consensus.cc:3060] T 17d6431cc77a491e8464d68db4081a51 P 1c50de89a5a845668696dcb28c3badd6 [term 0 FOLLOWER]: Advancing to term 1
I20260129 02:58:44.391134 20667 raft_consensus.cc:493] T add8d4923d1a489e86882c3a25692b95 P 974b802141bd4f13beffef2aa03f88f9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260129 02:58:44.391690 20590 raft_consensus.cc:2468] T 17d6431cc77a491e8464d68db4081a51 P 1c50de89a5a845668696dcb28c3badd6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 974b802141bd4f13beffef2aa03f88f9 in term 1.
I20260129 02:58:44.391839 20667 raft_consensus.cc:515] T add8d4923d1a489e86882c3a25692b95 P 974b802141bd4f13beffef2aa03f88f9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:58:44.392948 20480 leader_election.cc:304] T 17d6431cc77a491e8464d68db4081a51 P 974b802141bd4f13beffef2aa03f88f9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1c50de89a5a845668696dcb28c3badd6, 974b802141bd4f13beffef2aa03f88f9; no voters:
I20260129 02:58:44.393798 20685 raft_consensus.cc:2804] T 17d6431cc77a491e8464d68db4081a51 P 974b802141bd4f13beffef2aa03f88f9 [term 1 FOLLOWER]: Leader election won for term 1
W20260129 02:58:44.394068 20387 auto_rebalancer.cc:229] Could not retrieve cluster info: Service unavailable: Tablet not running
I20260129 02:58:44.394160 20667 leader_election.cc:290] T add8d4923d1a489e86882c3a25692b95 P 974b802141bd4f13beffef2aa03f88f9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 197b880fa39d432a9b472845876bfaf0 (127.18.211.1:42027), 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3:37735)
I20260129 02:58:44.394261 20685 raft_consensus.cc:697] T 17d6431cc77a491e8464d68db4081a51 P 974b802141bd4f13beffef2aa03f88f9 [term 1 LEADER]: Becoming Leader. State: Replica: 974b802141bd4f13beffef2aa03f88f9, State: Running, Role: LEADER
I20260129 02:58:44.396134 20440 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "add8d4923d1a489e86882c3a25692b95" candidate_uuid: "974b802141bd4f13beffef2aa03f88f9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "197b880fa39d432a9b472845876bfaf0" is_pre_election: true
I20260129 02:58:44.395310 20685 consensus_queue.cc:237] T 17d6431cc77a491e8464d68db4081a51 P 974b802141bd4f13beffef2aa03f88f9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:58:44.397027 20589 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "add8d4923d1a489e86882c3a25692b95" candidate_uuid: "974b802141bd4f13beffef2aa03f88f9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c50de89a5a845668696dcb28c3badd6" is_pre_election: true
W20260129 02:58:44.397640 20479 leader_election.cc:343] T add8d4923d1a489e86882c3a25692b95 P 974b802141bd4f13beffef2aa03f88f9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 197b880fa39d432a9b472845876bfaf0 (127.18.211.1:42027): Illegal state: must be running to vote when last-logged opid is not known
W20260129 02:58:44.398597 20480 leader_election.cc:343] T add8d4923d1a489e86882c3a25692b95 P 974b802141bd4f13beffef2aa03f88f9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3:37735): Illegal state: must be running to vote when last-logged opid is not known
I20260129 02:58:44.399035 20480 leader_election.cc:304] T add8d4923d1a489e86882c3a25692b95 P 974b802141bd4f13beffef2aa03f88f9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 974b802141bd4f13beffef2aa03f88f9; no voters: 197b880fa39d432a9b472845876bfaf0, 1c50de89a5a845668696dcb28c3badd6
I20260129 02:58:44.400488 20712 raft_consensus.cc:2749] T add8d4923d1a489e86882c3a25692b95 P 974b802141bd4f13beffef2aa03f88f9 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260129 02:58:44.382202 20588 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2ac659d732314e1288849f08fe5e074e" candidate_uuid: "974b802141bd4f13beffef2aa03f88f9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c50de89a5a845668696dcb28c3badd6" is_pre_election: true
I20260129 02:58:44.402223 20588 raft_consensus.cc:2468] T 2ac659d732314e1288849f08fe5e074e P 1c50de89a5a845668696dcb28c3badd6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 974b802141bd4f13beffef2aa03f88f9 in term 0.
I20260129 02:58:44.405402 20441 raft_consensus.cc:2468] T 17d6431cc77a491e8464d68db4081a51 P 197b880fa39d432a9b472845876bfaf0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 974b802141bd4f13beffef2aa03f88f9 in term 1.
I20260129 02:58:44.409988 20712 consensus_queue.cc:1048] T 75f45538dc704d9a8ec4ff8ad9f3ea5b P 974b802141bd4f13beffef2aa03f88f9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260129 02:58:44.413286 20480 leader_election.cc:304] T 2ac659d732314e1288849f08fe5e074e P 974b802141bd4f13beffef2aa03f88f9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1c50de89a5a845668696dcb28c3badd6, 974b802141bd4f13beffef2aa03f88f9; no voters: 197b880fa39d432a9b472845876bfaf0
I20260129 02:58:44.416690 20712 raft_consensus.cc:2804] T 2ac659d732314e1288849f08fe5e074e P 974b802141bd4f13beffef2aa03f88f9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260129 02:58:44.417239 20712 raft_consensus.cc:493] T 2ac659d732314e1288849f08fe5e074e P 974b802141bd4f13beffef2aa03f88f9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260129 02:58:44.407205 20638 consensus_queue.cc:1048] T e7d83d4ccf684e76acb29a0dc844e1b8 P 197b880fa39d432a9b472845876bfaf0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.010s
I20260129 02:58:44.417728 20712 raft_consensus.cc:3060] T 2ac659d732314e1288849f08fe5e074e P 974b802141bd4f13beffef2aa03f88f9 [term 0 FOLLOWER]: Advancing to term 1
I20260129 02:58:44.420040 20641 raft_consensus.cc:493] T 75f45538dc704d9a8ec4ff8ad9f3ea5b P 1c50de89a5a845668696dcb28c3badd6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260129 02:58:44.420629 20641 raft_consensus.cc:515] T 75f45538dc704d9a8ec4ff8ad9f3ea5b P 1c50de89a5a845668696dcb28c3badd6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:58:44.421470 20633 tablet_bootstrap.cc:492] T 45cc5f3d2688486d9b6dbe7e666dedde P 197b880fa39d432a9b472845876bfaf0: No bootstrap required, opened a new log
I20260129 02:58:44.422075 20633 ts_tablet_manager.cc:1403] T 45cc5f3d2688486d9b6dbe7e666dedde P 197b880fa39d432a9b472845876bfaf0: Time spent bootstrapping tablet: real 0.091s user 0.009s sys 0.002s
I20260129 02:58:44.424152 20439 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "75f45538dc704d9a8ec4ff8ad9f3ea5b" candidate_uuid: "1c50de89a5a845668696dcb28c3badd6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "197b880fa39d432a9b472845876bfaf0" is_pre_election: true
I20260129 02:58:44.427574 20641 leader_election.cc:290] T 75f45538dc704d9a8ec4ff8ad9f3ea5b P 1c50de89a5a845668696dcb28c3badd6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 197b880fa39d432a9b472845876bfaf0 (127.18.211.1:42027), 974b802141bd4f13beffef2aa03f88f9 (127.18.211.2:34517)
I20260129 02:58:44.428435 20515 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "75f45538dc704d9a8ec4ff8ad9f3ea5b" candidate_uuid: "1c50de89a5a845668696dcb28c3badd6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "974b802141bd4f13beffef2aa03f88f9" is_pre_election: true
I20260129 02:58:44.430200 20556 leader_election.cc:304] T 75f45538dc704d9a8ec4ff8ad9f3ea5b P 1c50de89a5a845668696dcb28c3badd6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1c50de89a5a845668696dcb28c3badd6; no voters: 197b880fa39d432a9b472845876bfaf0, 974b802141bd4f13beffef2aa03f88f9
I20260129 02:58:44.431823 20641 raft_consensus.cc:2749] T 75f45538dc704d9a8ec4ff8ad9f3ea5b P 1c50de89a5a845668696dcb28c3badd6 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260129 02:58:44.436910 20667 consensus_queue.cc:1048] T 1299723a4476420ab02fd8bfd7aa0645 P 974b802141bd4f13beffef2aa03f88f9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260129 02:58:44.436604 20338 catalog_manager.cc:5654] T 17d6431cc77a491e8464d68db4081a51 P 974b802141bd4f13beffef2aa03f88f9 reported cstate change: term changed from 0 to 1, leader changed from <none> to 974b802141bd4f13beffef2aa03f88f9 (127.18.211.2). New cstate: current_term: 1 leader_uuid: "974b802141bd4f13beffef2aa03f88f9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } health_report { overall_health: UNKNOWN } } }
W20260129 02:58:44.466435 20405 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.211.3:37735, user_credentials={real_user=slave}} blocked reactor thread for 62379.7us
I20260129 02:58:44.474720 20685 raft_consensus.cc:493] T 78aa39096868415d9afc2ffcb248e198 P 974b802141bd4f13beffef2aa03f88f9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260129 02:58:44.476428 20653 leader_election.cc:290] T 17d6431cc77a491e8464d68db4081a51 P 974b802141bd4f13beffef2aa03f88f9 [CANDIDATE]: Term 1 election: Requested vote from peers 197b880fa39d432a9b472845876bfaf0 (127.18.211.1:42027), 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3:37735)
I20260129 02:58:44.491106 20633 raft_consensus.cc:359] T 45cc5f3d2688486d9b6dbe7e666dedde P 197b880fa39d432a9b472845876bfaf0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:58:44.492667 20633 raft_consensus.cc:385] T 45cc5f3d2688486d9b6dbe7e666dedde P 197b880fa39d432a9b472845876bfaf0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260129 02:58:44.493041 20633 raft_consensus.cc:740] T 45cc5f3d2688486d9b6dbe7e666dedde P 197b880fa39d432a9b472845876bfaf0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 197b880fa39d432a9b472845876bfaf0, State: Initialized, Role: FOLLOWER
I20260129 02:58:44.493868 20633 consensus_queue.cc:260] T 45cc5f3d2688486d9b6dbe7e666dedde P 197b880fa39d432a9b472845876bfaf0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:58:44.475430 20685 raft_consensus.cc:515] T 78aa39096868415d9afc2ffcb248e198 P 974b802141bd4f13beffef2aa03f88f9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:58:44.498289 20633 ts_tablet_manager.cc:1434] T 45cc5f3d2688486d9b6dbe7e666dedde P 197b880fa39d432a9b472845876bfaf0: Time spent starting tablet: real 0.076s user 0.005s sys 0.004s
I20260129 02:58:44.499482 20633 tablet_bootstrap.cc:492] T 78aa39096868415d9afc2ffcb248e198 P 197b880fa39d432a9b472845876bfaf0: Bootstrap starting.
I20260129 02:58:44.495396 20712 raft_consensus.cc:515] T 2ac659d732314e1288849f08fe5e074e P 974b802141bd4f13beffef2aa03f88f9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
W20260129 02:58:44.491814 20407 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.211.2:34517, user_credentials={real_user=slave}} blocked reactor thread for 85998.6us
I20260129 02:58:44.507777 20337 catalog_manager.cc:5654] T 0f20af7fbe5d4db89a6f17948d98912c P 197b880fa39d432a9b472845876bfaf0 reported cstate change: term changed from 0 to 1, leader changed from <none> to 197b880fa39d432a9b472845876bfaf0 (127.18.211.1). New cstate: current_term: 1 leader_uuid: "197b880fa39d432a9b472845876bfaf0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } health_report { overall_health: UNKNOWN } } }
I20260129 02:58:44.537503 20685 leader_election.cc:290] T 78aa39096868415d9afc2ffcb248e198 P 974b802141bd4f13beffef2aa03f88f9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 197b880fa39d432a9b472845876bfaf0 (127.18.211.1:42027), 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3:37735)
I20260129 02:58:44.540198 20338 catalog_manager.cc:5654] T 3e2d15f8640b407195210c476e09df91 P 1c50de89a5a845668696dcb28c3badd6 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3). New cstate: current_term: 1 leader_uuid: "1c50de89a5a845668696dcb28c3badd6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } health_report { overall_health: HEALTHY } } }
I20260129 02:58:44.541904 20338 catalog_manager.cc:5654] T 45cc5f3d2688486d9b6dbe7e666dedde P 1c50de89a5a845668696dcb28c3badd6 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3). New cstate: current_term: 1 leader_uuid: "1c50de89a5a845668696dcb28c3badd6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } health_report { overall_health: HEALTHY } } }
I20260129 02:58:44.544603 20439 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "78aa39096868415d9afc2ffcb248e198" candidate_uuid: "974b802141bd4f13beffef2aa03f88f9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "197b880fa39d432a9b472845876bfaf0" is_pre_election: true
W20260129 02:58:44.548763 20479 leader_election.cc:343] T 78aa39096868415d9afc2ffcb248e198 P 974b802141bd4f13beffef2aa03f88f9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 197b880fa39d432a9b472845876bfaf0 (127.18.211.1:42027): Illegal state: must be running to vote when last-logged opid is not known
I20260129 02:58:44.550582 20712 leader_election.cc:290] T 2ac659d732314e1288849f08fe5e074e P 974b802141bd4f13beffef2aa03f88f9 [CANDIDATE]: Term 1 election: Requested vote from peers 197b880fa39d432a9b472845876bfaf0 (127.18.211.1:42027), 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3:37735)
I20260129 02:58:44.550601 20439 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2ac659d732314e1288849f08fe5e074e" candidate_uuid: "974b802141bd4f13beffef2aa03f88f9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "197b880fa39d432a9b472845876bfaf0"
W20260129 02:58:44.552327 20479 leader_election.cc:343] T 2ac659d732314e1288849f08fe5e074e P 974b802141bd4f13beffef2aa03f88f9 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 197b880fa39d432a9b472845876bfaf0 (127.18.211.1:42027): Illegal state: must be running to vote when last-logged opid is not known
I20260129 02:58:44.549737 20685 consensus_queue.cc:1048] T 1299723a4476420ab02fd8bfd7aa0645 P 974b802141bd4f13beffef2aa03f88f9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.003s
I20260129 02:58:44.566886 20630 tablet_bootstrap.cc:492] T 4e420f90884f4796997b5ea41e49e411 P 974b802141bd4f13beffef2aa03f88f9: No bootstrap required, opened a new log
I20260129 02:58:44.567561 20630 ts_tablet_manager.cc:1403] T 4e420f90884f4796997b5ea41e49e411 P 974b802141bd4f13beffef2aa03f88f9: Time spent bootstrapping tablet: real 0.214s user 0.040s sys 0.021s
I20260129 02:58:44.563076 20669 raft_consensus.cc:493] T a609ac02fae84725ab743f95f51d33ce P 1c50de89a5a845668696dcb28c3badd6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260129 02:58:44.569274 20669 raft_consensus.cc:515] T a609ac02fae84725ab743f95f51d33ce P 1c50de89a5a845668696dcb28c3badd6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:58:44.570667 20630 raft_consensus.cc:359] T 4e420f90884f4796997b5ea41e49e411 P 974b802141bd4f13beffef2aa03f88f9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:58:44.571373 20630 raft_consensus.cc:385] T 4e420f90884f4796997b5ea41e49e411 P 974b802141bd4f13beffef2aa03f88f9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260129 02:58:44.571508 20669 leader_election.cc:290] T a609ac02fae84725ab743f95f51d33ce P 1c50de89a5a845668696dcb28c3badd6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 197b880fa39d432a9b472845876bfaf0 (127.18.211.1:42027), 974b802141bd4f13beffef2aa03f88f9 (127.18.211.2:34517)
I20260129 02:58:44.571623 20630 raft_consensus.cc:740] T 4e420f90884f4796997b5ea41e49e411 P 974b802141bd4f13beffef2aa03f88f9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 974b802141bd4f13beffef2aa03f88f9, State: Initialized, Role: FOLLOWER
I20260129 02:58:44.572530 20630 consensus_queue.cc:260] T 4e420f90884f4796997b5ea41e49e411 P 974b802141bd4f13beffef2aa03f88f9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:58:44.574311 20440 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a609ac02fae84725ab743f95f51d33ce" candidate_uuid: "1c50de89a5a845668696dcb28c3badd6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "197b880fa39d432a9b472845876bfaf0" is_pre_election: true
I20260129 02:58:44.574610 20630 ts_tablet_manager.cc:1434] T 4e420f90884f4796997b5ea41e49e411 P 974b802141bd4f13beffef2aa03f88f9: Time spent starting tablet: real 0.007s user 0.005s sys 0.000s
I20260129 02:58:44.576130 20630 tablet_bootstrap.cc:492] T 1ce4ddf5b10e4ae8b193b13b2e5936ee P 974b802141bd4f13beffef2aa03f88f9: Bootstrap starting.
I20260129 02:58:44.583670 20515 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a609ac02fae84725ab743f95f51d33ce" candidate_uuid: "1c50de89a5a845668696dcb28c3badd6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "974b802141bd4f13beffef2aa03f88f9" is_pre_election: true
W20260129 02:58:44.587037 20553 leader_election.cc:343] T a609ac02fae84725ab743f95f51d33ce P 1c50de89a5a845668696dcb28c3badd6 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 197b880fa39d432a9b472845876bfaf0 (127.18.211.1:42027): Illegal state: must be running to vote when last-logged opid is not known
W20260129 02:58:44.588013 20556 leader_election.cc:343] T a609ac02fae84725ab743f95f51d33ce P 1c50de89a5a845668696dcb28c3badd6 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 974b802141bd4f13beffef2aa03f88f9 (127.18.211.2:34517): Illegal state: must be running to vote when last-logged opid is not known
I20260129 02:58:44.588575 20556 leader_election.cc:304] T a609ac02fae84725ab743f95f51d33ce P 1c50de89a5a845668696dcb28c3badd6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1c50de89a5a845668696dcb28c3badd6; no voters: 197b880fa39d432a9b472845876bfaf0, 974b802141bd4f13beffef2aa03f88f9
I20260129 02:58:44.590123 20633 tablet_bootstrap.cc:654] T 78aa39096868415d9afc2ffcb248e198 P 197b880fa39d432a9b472845876bfaf0: Neither blocks nor log segments found. Creating new log.
I20260129 02:58:44.590310 20669 raft_consensus.cc:2749] T a609ac02fae84725ab743f95f51d33ce P 1c50de89a5a845668696dcb28c3badd6 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260129 02:58:44.600557 20631 tablet_bootstrap.cc:492] T b189f08e16a74a11980cc55d99359a0a P 1c50de89a5a845668696dcb28c3badd6: No bootstrap required, opened a new log
I20260129 02:58:44.601214 20631 ts_tablet_manager.cc:1403] T b189f08e16a74a11980cc55d99359a0a P 1c50de89a5a845668696dcb28c3badd6: Time spent bootstrapping tablet: real 0.249s user 0.013s sys 0.000s
I20260129 02:58:44.609238 20631 raft_consensus.cc:359] T b189f08e16a74a11980cc55d99359a0a P 1c50de89a5a845668696dcb28c3badd6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:58:44.609988 20631 raft_consensus.cc:385] T b189f08e16a74a11980cc55d99359a0a P 1c50de89a5a845668696dcb28c3badd6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260129 02:58:44.610702 20589 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "78aa39096868415d9afc2ffcb248e198" candidate_uuid: "974b802141bd4f13beffef2aa03f88f9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c50de89a5a845668696dcb28c3badd6" is_pre_election: true
I20260129 02:58:44.610267 20631 raft_consensus.cc:740] T b189f08e16a74a11980cc55d99359a0a P 1c50de89a5a845668696dcb28c3badd6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1c50de89a5a845668696dcb28c3badd6, State: Initialized, Role: FOLLOWER
I20260129 02:58:44.611584 20590 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2ac659d732314e1288849f08fe5e074e" candidate_uuid: "974b802141bd4f13beffef2aa03f88f9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c50de89a5a845668696dcb28c3badd6"
I20260129 02:58:44.613183 20631 consensus_queue.cc:260] T b189f08e16a74a11980cc55d99359a0a P 1c50de89a5a845668696dcb28c3badd6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:58:44.615106 20590 raft_consensus.cc:3060] T 2ac659d732314e1288849f08fe5e074e P 1c50de89a5a845668696dcb28c3badd6 [term 0 FOLLOWER]: Advancing to term 1
I20260129 02:58:44.617615 20589 raft_consensus.cc:3060] T 75f45538dc704d9a8ec4ff8ad9f3ea5b P 1c50de89a5a845668696dcb28c3badd6 [term 0 FOLLOWER]: Advancing to term 1
I20260129 02:58:44.623745 20631 ts_tablet_manager.cc:1434] T b189f08e16a74a11980cc55d99359a0a P 1c50de89a5a845668696dcb28c3badd6: Time spent starting tablet: real 0.022s user 0.006s sys 0.000s
I20260129 02:58:44.623675 20590 raft_consensus.cc:2468] T 2ac659d732314e1288849f08fe5e074e P 1c50de89a5a845668696dcb28c3badd6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 974b802141bd4f13beffef2aa03f88f9 in term 1.
I20260129 02:58:44.624900 20631 tablet_bootstrap.cc:492] T add8d4923d1a489e86882c3a25692b95 P 1c50de89a5a845668696dcb28c3badd6: Bootstrap starting.
W20260129 02:58:44.627131 20480 leader_election.cc:343] T 78aa39096868415d9afc2ffcb248e198 P 974b802141bd4f13beffef2aa03f88f9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3:37735): Illegal state: must be running to vote when last-logged opid is not known
I20260129 02:58:44.627597 20480 leader_election.cc:304] T 78aa39096868415d9afc2ffcb248e198 P 974b802141bd4f13beffef2aa03f88f9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 974b802141bd4f13beffef2aa03f88f9; no voters: 197b880fa39d432a9b472845876bfaf0, 1c50de89a5a845668696dcb28c3badd6
I20260129 02:58:44.629317 20741 raft_consensus.cc:2749] T 78aa39096868415d9afc2ffcb248e198 P 974b802141bd4f13beffef2aa03f88f9 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260129 02:58:44.631027 20480 leader_election.cc:304] T 2ac659d732314e1288849f08fe5e074e P 974b802141bd4f13beffef2aa03f88f9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1c50de89a5a845668696dcb28c3badd6, 974b802141bd4f13beffef2aa03f88f9; no voters: 197b880fa39d432a9b472845876bfaf0
I20260129 02:58:44.632442 20741 raft_consensus.cc:2804] T 2ac659d732314e1288849f08fe5e074e P 974b802141bd4f13beffef2aa03f88f9 [term 1 FOLLOWER]: Leader election won for term 1
I20260129 02:58:44.632915 20741 raft_consensus.cc:697] T 2ac659d732314e1288849f08fe5e074e P 974b802141bd4f13beffef2aa03f88f9 [term 1 LEADER]: Becoming Leader. State: Replica: 974b802141bd4f13beffef2aa03f88f9, State: Running, Role: LEADER
I20260129 02:58:44.633285 20740 consensus_queue.cc:1048] T 724a236524cb4e7ba3428880681d6cc2 P 974b802141bd4f13beffef2aa03f88f9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260129 02:58:44.633803 20741 consensus_queue.cc:237] T 2ac659d732314e1288849f08fe5e074e P 974b802141bd4f13beffef2aa03f88f9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:58:44.637300 20633 tablet_bootstrap.cc:492] T 78aa39096868415d9afc2ffcb248e198 P 197b880fa39d432a9b472845876bfaf0: No bootstrap required, opened a new log
I20260129 02:58:44.637775 20633 ts_tablet_manager.cc:1403] T 78aa39096868415d9afc2ffcb248e198 P 197b880fa39d432a9b472845876bfaf0: Time spent bootstrapping tablet: real 0.139s user 0.018s sys 0.010s
I20260129 02:58:44.638307 20739 consensus_queue.cc:1048] T 75f45538dc704d9a8ec4ff8ad9f3ea5b P 974b802141bd4f13beffef2aa03f88f9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260129 02:58:44.641149 20633 raft_consensus.cc:359] T 78aa39096868415d9afc2ffcb248e198 P 197b880fa39d432a9b472845876bfaf0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:58:44.642051 20633 raft_consensus.cc:385] T 78aa39096868415d9afc2ffcb248e198 P 197b880fa39d432a9b472845876bfaf0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260129 02:58:44.642381 20633 raft_consensus.cc:740] T 78aa39096868415d9afc2ffcb248e198 P 197b880fa39d432a9b472845876bfaf0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 197b880fa39d432a9b472845876bfaf0, State: Initialized, Role: FOLLOWER
I20260129 02:58:44.648217 20630 tablet_bootstrap.cc:654] T 1ce4ddf5b10e4ae8b193b13b2e5936ee P 974b802141bd4f13beffef2aa03f88f9: Neither blocks nor log segments found. Creating new log.
I20260129 02:58:44.649452 20631 tablet_bootstrap.cc:654] T add8d4923d1a489e86882c3a25692b95 P 1c50de89a5a845668696dcb28c3badd6: Neither blocks nor log segments found. Creating new log.
I20260129 02:58:44.651135 20730 consensus_queue.cc:1048] T 2e58e9a5a8aa4cecafb464addb4f35ed P 197b880fa39d432a9b472845876bfaf0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260129 02:58:44.652390 20633 consensus_queue.cc:260] T 78aa39096868415d9afc2ffcb248e198 P 197b880fa39d432a9b472845876bfaf0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:58:44.653604 20337 catalog_manager.cc:5654] T 2ac659d732314e1288849f08fe5e074e P 974b802141bd4f13beffef2aa03f88f9 reported cstate change: term changed from 0 to 1, leader changed from <none> to 974b802141bd4f13beffef2aa03f88f9 (127.18.211.2). New cstate: current_term: 1 leader_uuid: "974b802141bd4f13beffef2aa03f88f9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } health_report { overall_health: UNKNOWN } } }
I20260129 02:58:44.654537 20633 ts_tablet_manager.cc:1434] T 78aa39096868415d9afc2ffcb248e198 P 197b880fa39d432a9b472845876bfaf0: Time spent starting tablet: real 0.016s user 0.003s sys 0.004s
I20260129 02:58:44.655678 20633 tablet_bootstrap.cc:492] T a609ac02fae84725ab743f95f51d33ce P 197b880fa39d432a9b472845876bfaf0: Bootstrap starting.
I20260129 02:58:44.672254 20633 tablet_bootstrap.cc:654] T a609ac02fae84725ab743f95f51d33ce P 197b880fa39d432a9b472845876bfaf0: Neither blocks nor log segments found. Creating new log.
I20260129 02:58:44.696905 20630 tablet_bootstrap.cc:492] T 1ce4ddf5b10e4ae8b193b13b2e5936ee P 974b802141bd4f13beffef2aa03f88f9: No bootstrap required, opened a new log
I20260129 02:58:44.697379 20630 ts_tablet_manager.cc:1403] T 1ce4ddf5b10e4ae8b193b13b2e5936ee P 974b802141bd4f13beffef2aa03f88f9: Time spent bootstrapping tablet: real 0.122s user 0.020s sys 0.004s
I20260129 02:58:44.699374 20732 consensus_queue.cc:1048] T 724a236524cb4e7ba3428880681d6cc2 P 974b802141bd4f13beffef2aa03f88f9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260129 02:58:44.702169 20729 consensus_queue.cc:1048] T 2e58e9a5a8aa4cecafb464addb4f35ed P 197b880fa39d432a9b472845876bfaf0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260129 02:58:44.703140 20630 raft_consensus.cc:359] T 1ce4ddf5b10e4ae8b193b13b2e5936ee P 974b802141bd4f13beffef2aa03f88f9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:58:44.703815 20630 raft_consensus.cc:385] T 1ce4ddf5b10e4ae8b193b13b2e5936ee P 974b802141bd4f13beffef2aa03f88f9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260129 02:58:44.704103 20630 raft_consensus.cc:740] T 1ce4ddf5b10e4ae8b193b13b2e5936ee P 974b802141bd4f13beffef2aa03f88f9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 974b802141bd4f13beffef2aa03f88f9, State: Initialized, Role: FOLLOWER
I20260129 02:58:44.709008 20633 tablet_bootstrap.cc:492] T a609ac02fae84725ab743f95f51d33ce P 197b880fa39d432a9b472845876bfaf0: No bootstrap required, opened a new log
I20260129 02:58:44.712365 20633 ts_tablet_manager.cc:1403] T a609ac02fae84725ab743f95f51d33ce P 197b880fa39d432a9b472845876bfaf0: Time spent bootstrapping tablet: real 0.057s user 0.017s sys 0.015s
I20260129 02:58:44.712301 20630 consensus_queue.cc:260] T 1ce4ddf5b10e4ae8b193b13b2e5936ee P 974b802141bd4f13beffef2aa03f88f9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:58:44.714918 20630 ts_tablet_manager.cc:1434] T 1ce4ddf5b10e4ae8b193b13b2e5936ee P 974b802141bd4f13beffef2aa03f88f9: Time spent starting tablet: real 0.017s user 0.007s sys 0.000s
I20260129 02:58:44.715811 20630 tablet_bootstrap.cc:492] T e4d18c35b0a7493e85d0a4b92956a48f P 974b802141bd4f13beffef2aa03f88f9: Bootstrap starting.
I20260129 02:58:44.722827 20630 tablet_bootstrap.cc:654] T e4d18c35b0a7493e85d0a4b92956a48f P 974b802141bd4f13beffef2aa03f88f9: Neither blocks nor log segments found. Creating new log.
I20260129 02:58:44.715721 20633 raft_consensus.cc:359] T a609ac02fae84725ab743f95f51d33ce P 197b880fa39d432a9b472845876bfaf0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:58:44.724515 20633 raft_consensus.cc:385] T a609ac02fae84725ab743f95f51d33ce P 197b880fa39d432a9b472845876bfaf0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260129 02:58:44.724759 20633 raft_consensus.cc:740] T a609ac02fae84725ab743f95f51d33ce P 197b880fa39d432a9b472845876bfaf0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 197b880fa39d432a9b472845876bfaf0, State: Initialized, Role: FOLLOWER
I20260129 02:58:44.725661 20633 consensus_queue.cc:260] T a609ac02fae84725ab743f95f51d33ce P 197b880fa39d432a9b472845876bfaf0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:58:44.728479 20740 consensus_queue.cc:1048] T a5a05fbcbbcf4916bbc01fd1a5b5e90d P 974b802141bd4f13beffef2aa03f88f9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260129 02:58:44.730194 20633 ts_tablet_manager.cc:1434] T a609ac02fae84725ab743f95f51d33ce P 197b880fa39d432a9b472845876bfaf0: Time spent starting tablet: real 0.017s user 0.006s sys 0.000s
I20260129 02:58:44.730777 20631 tablet_bootstrap.cc:492] T add8d4923d1a489e86882c3a25692b95 P 1c50de89a5a845668696dcb28c3badd6: No bootstrap required, opened a new log
I20260129 02:58:44.731201 20631 ts_tablet_manager.cc:1403] T add8d4923d1a489e86882c3a25692b95 P 1c50de89a5a845668696dcb28c3badd6: Time spent bootstrapping tablet: real 0.106s user 0.008s sys 0.003s
I20260129 02:58:44.731173 20633 tablet_bootstrap.cc:492] T e4d18c35b0a7493e85d0a4b92956a48f P 197b880fa39d432a9b472845876bfaf0: Bootstrap starting.
I20260129 02:58:44.733757 20631 raft_consensus.cc:359] T add8d4923d1a489e86882c3a25692b95 P 1c50de89a5a845668696dcb28c3badd6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:58:44.734530 20631 raft_consensus.cc:385] T add8d4923d1a489e86882c3a25692b95 P 1c50de89a5a845668696dcb28c3badd6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260129 02:58:44.734905 20631 raft_consensus.cc:740] T add8d4923d1a489e86882c3a25692b95 P 1c50de89a5a845668696dcb28c3badd6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1c50de89a5a845668696dcb28c3badd6, State: Initialized, Role: FOLLOWER
I20260129 02:58:44.735738 20631 consensus_queue.cc:260] T add8d4923d1a489e86882c3a25692b95 P 1c50de89a5a845668696dcb28c3badd6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:58:44.737936 20631 ts_tablet_manager.cc:1434] T add8d4923d1a489e86882c3a25692b95 P 1c50de89a5a845668696dcb28c3badd6: Time spent starting tablet: real 0.006s user 0.004s sys 0.001s
I20260129 02:58:44.738991 20633 tablet_bootstrap.cc:654] T e4d18c35b0a7493e85d0a4b92956a48f P 197b880fa39d432a9b472845876bfaf0: Neither blocks nor log segments found. Creating new log.
I20260129 02:58:44.746379 20631 tablet_bootstrap.cc:492] T 78aa39096868415d9afc2ffcb248e198 P 1c50de89a5a845668696dcb28c3badd6: Bootstrap starting.
I20260129 02:58:44.753717 20630 tablet_bootstrap.cc:492] T e4d18c35b0a7493e85d0a4b92956a48f P 974b802141bd4f13beffef2aa03f88f9: No bootstrap required, opened a new log
I20260129 02:58:44.753770 20631 tablet_bootstrap.cc:654] T 78aa39096868415d9afc2ffcb248e198 P 1c50de89a5a845668696dcb28c3badd6: Neither blocks nor log segments found. Creating new log.
I20260129 02:58:44.754354 20630 ts_tablet_manager.cc:1403] T e4d18c35b0a7493e85d0a4b92956a48f P 974b802141bd4f13beffef2aa03f88f9: Time spent bootstrapping tablet: real 0.039s user 0.014s sys 0.001s
I20260129 02:58:44.757298 20630 raft_consensus.cc:359] T e4d18c35b0a7493e85d0a4b92956a48f P 974b802141bd4f13beffef2aa03f88f9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:58:44.757954 20630 raft_consensus.cc:385] T e4d18c35b0a7493e85d0a4b92956a48f P 974b802141bd4f13beffef2aa03f88f9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260129 02:58:44.757565 20734 consensus_queue.cc:1048] T a5a05fbcbbcf4916bbc01fd1a5b5e90d P 974b802141bd4f13beffef2aa03f88f9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260129 02:58:44.758205 20630 raft_consensus.cc:740] T e4d18c35b0a7493e85d0a4b92956a48f P 974b802141bd4f13beffef2aa03f88f9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 974b802141bd4f13beffef2aa03f88f9, State: Initialized, Role: FOLLOWER
I20260129 02:58:44.759135 20630 consensus_queue.cc:260] T e4d18c35b0a7493e85d0a4b92956a48f P 974b802141bd4f13beffef2aa03f88f9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:58:44.761230 20630 ts_tablet_manager.cc:1434] T e4d18c35b0a7493e85d0a4b92956a48f P 974b802141bd4f13beffef2aa03f88f9: Time spent starting tablet: real 0.007s user 0.007s sys 0.000s
I20260129 02:58:44.762048 20630 tablet_bootstrap.cc:492] T a609ac02fae84725ab743f95f51d33ce P 974b802141bd4f13beffef2aa03f88f9: Bootstrap starting.
I20260129 02:58:44.764628 20734 raft_consensus.cc:493] T 1ce4ddf5b10e4ae8b193b13b2e5936ee P 974b802141bd4f13beffef2aa03f88f9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260129 02:58:44.765199 20734 raft_consensus.cc:515] T 1ce4ddf5b10e4ae8b193b13b2e5936ee P 974b802141bd4f13beffef2aa03f88f9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:58:44.767835 20440 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1ce4ddf5b10e4ae8b193b13b2e5936ee" candidate_uuid: "974b802141bd4f13beffef2aa03f88f9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "197b880fa39d432a9b472845876bfaf0" is_pre_election: true
I20260129 02:58:44.768239 20590 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1ce4ddf5b10e4ae8b193b13b2e5936ee" candidate_uuid: "974b802141bd4f13beffef2aa03f88f9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c50de89a5a845668696dcb28c3badd6" is_pre_election: true
I20260129 02:58:44.768611 20633 tablet_bootstrap.cc:492] T e4d18c35b0a7493e85d0a4b92956a48f P 197b880fa39d432a9b472845876bfaf0: No bootstrap required, opened a new log
I20260129 02:58:44.769057 20633 ts_tablet_manager.cc:1403] T e4d18c35b0a7493e85d0a4b92956a48f P 197b880fa39d432a9b472845876bfaf0: Time spent bootstrapping tablet: real 0.038s user 0.013s sys 0.004s
W20260129 02:58:44.769326 20479 leader_election.cc:343] T 1ce4ddf5b10e4ae8b193b13b2e5936ee P 974b802141bd4f13beffef2aa03f88f9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 197b880fa39d432a9b472845876bfaf0 (127.18.211.1:42027): Illegal state: must be running to vote when last-logged opid is not known
I20260129 02:58:44.769670 20734 leader_election.cc:290] T 1ce4ddf5b10e4ae8b193b13b2e5936ee P 974b802141bd4f13beffef2aa03f88f9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 197b880fa39d432a9b472845876bfaf0 (127.18.211.1:42027), 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3:37735)
W20260129 02:58:44.769982 20480 leader_election.cc:343] T 1ce4ddf5b10e4ae8b193b13b2e5936ee P 974b802141bd4f13beffef2aa03f88f9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3:37735): Illegal state: must be running to vote when last-logged opid is not known
I20260129 02:58:44.770423 20480 leader_election.cc:304] T 1ce4ddf5b10e4ae8b193b13b2e5936ee P 974b802141bd4f13beffef2aa03f88f9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 974b802141bd4f13beffef2aa03f88f9; no voters: 197b880fa39d432a9b472845876bfaf0, 1c50de89a5a845668696dcb28c3badd6
I20260129 02:58:44.771657 20734 raft_consensus.cc:2749] T 1ce4ddf5b10e4ae8b193b13b2e5936ee P 974b802141bd4f13beffef2aa03f88f9 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260129 02:58:44.771664 20630 tablet_bootstrap.cc:654] T a609ac02fae84725ab743f95f51d33ce P 974b802141bd4f13beffef2aa03f88f9: Neither blocks nor log segments found. Creating new log.
I20260129 02:58:44.771530 20633 raft_consensus.cc:359] T e4d18c35b0a7493e85d0a4b92956a48f P 197b880fa39d432a9b472845876bfaf0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:58:44.777243 20633 raft_consensus.cc:385] T e4d18c35b0a7493e85d0a4b92956a48f P 197b880fa39d432a9b472845876bfaf0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260129 02:58:44.777565 20633 raft_consensus.cc:740] T e4d18c35b0a7493e85d0a4b92956a48f P 197b880fa39d432a9b472845876bfaf0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 197b880fa39d432a9b472845876bfaf0, State: Initialized, Role: FOLLOWER
I20260129 02:58:44.783025 20669 consensus_queue.cc:1048] T 45cc5f3d2688486d9b6dbe7e666dedde P 1c50de89a5a845668696dcb28c3badd6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260129 02:58:44.778411 20633 consensus_queue.cc:260] T e4d18c35b0a7493e85d0a4b92956a48f P 197b880fa39d432a9b472845876bfaf0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:58:44.792070 20633 ts_tablet_manager.cc:1434] T e4d18c35b0a7493e85d0a4b92956a48f P 197b880fa39d432a9b472845876bfaf0: Time spent starting tablet: real 0.023s user 0.008s sys 0.000s
I20260129 02:58:44.793035 20633 tablet_bootstrap.cc:492] T add8d4923d1a489e86882c3a25692b95 P 197b880fa39d432a9b472845876bfaf0: Bootstrap starting.
I20260129 02:58:44.793272 20734 consensus_queue.cc:1048] T 17d6431cc77a491e8464d68db4081a51 P 974b802141bd4f13beffef2aa03f88f9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260129 02:58:44.800927 20633 tablet_bootstrap.cc:654] T add8d4923d1a489e86882c3a25692b95 P 197b880fa39d432a9b472845876bfaf0: Neither blocks nor log segments found. Creating new log.
I20260129 02:58:44.802642 20669 consensus_queue.cc:1048] T 3e2d15f8640b407195210c476e09df91 P 1c50de89a5a845668696dcb28c3badd6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260129 02:58:44.804556 20631 tablet_bootstrap.cc:492] T 78aa39096868415d9afc2ffcb248e198 P 1c50de89a5a845668696dcb28c3badd6: No bootstrap required, opened a new log
I20260129 02:58:44.805176 20631 ts_tablet_manager.cc:1403] T 78aa39096868415d9afc2ffcb248e198 P 1c50de89a5a845668696dcb28c3badd6: Time spent bootstrapping tablet: real 0.067s user 0.010s sys 0.002s
I20260129 02:58:44.807380 20729 raft_consensus.cc:493] T e4d18c35b0a7493e85d0a4b92956a48f P 197b880fa39d432a9b472845876bfaf0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260129 02:58:44.808139 20734 raft_consensus.cc:493] T 4e420f90884f4796997b5ea41e49e411 P 974b802141bd4f13beffef2aa03f88f9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260129 02:58:44.808032 20729 raft_consensus.cc:515] T e4d18c35b0a7493e85d0a4b92956a48f P 197b880fa39d432a9b472845876bfaf0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:58:44.808367 20631 raft_consensus.cc:359] T 78aa39096868415d9afc2ffcb248e198 P 1c50de89a5a845668696dcb28c3badd6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:58:44.808602 20734 raft_consensus.cc:515] T 4e420f90884f4796997b5ea41e49e411 P 974b802141bd4f13beffef2aa03f88f9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:58:44.809360 20631 raft_consensus.cc:385] T 78aa39096868415d9afc2ffcb248e198 P 1c50de89a5a845668696dcb28c3badd6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260129 02:58:44.810791 20734 leader_election.cc:290] T 4e420f90884f4796997b5ea41e49e411 P 974b802141bd4f13beffef2aa03f88f9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 197b880fa39d432a9b472845876bfaf0 (127.18.211.1:42027), 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3:37735)
I20260129 02:58:44.811995 20588 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e4d18c35b0a7493e85d0a4b92956a48f" candidate_uuid: "197b880fa39d432a9b472845876bfaf0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c50de89a5a845668696dcb28c3badd6" is_pre_election: true
I20260129 02:58:44.812045 20516 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e4d18c35b0a7493e85d0a4b92956a48f" candidate_uuid: "197b880fa39d432a9b472845876bfaf0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "974b802141bd4f13beffef2aa03f88f9" is_pre_election: true
I20260129 02:58:44.812898 20516 raft_consensus.cc:2468] T e4d18c35b0a7493e85d0a4b92956a48f P 974b802141bd4f13beffef2aa03f88f9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 197b880fa39d432a9b472845876bfaf0 in term 0.
I20260129 02:58:44.813230 20590 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4e420f90884f4796997b5ea41e49e411" candidate_uuid: "974b802141bd4f13beffef2aa03f88f9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c50de89a5a845668696dcb28c3badd6" is_pre_election: true
W20260129 02:58:44.813369 20405 leader_election.cc:343] T e4d18c35b0a7493e85d0a4b92956a48f P 197b880fa39d432a9b472845876bfaf0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3:37735): Illegal state: must be running to vote when last-logged opid is not known
I20260129 02:58:44.814252 20438 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4e420f90884f4796997b5ea41e49e411" candidate_uuid: "974b802141bd4f13beffef2aa03f88f9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "197b880fa39d432a9b472845876bfaf0" is_pre_election: true
W20260129 02:58:44.814595 20480 leader_election.cc:343] T 4e420f90884f4796997b5ea41e49e411 P 974b802141bd4f13beffef2aa03f88f9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3:37735): Illegal state: must be running to vote when last-logged opid is not known
I20260129 02:58:44.809765 20631 raft_consensus.cc:740] T 78aa39096868415d9afc2ffcb248e198 P 1c50de89a5a845668696dcb28c3badd6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1c50de89a5a845668696dcb28c3badd6, State: Initialized, Role: FOLLOWER
I20260129 02:58:44.815984 20407 leader_election.cc:304] T e4d18c35b0a7493e85d0a4b92956a48f P 197b880fa39d432a9b472845876bfaf0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 197b880fa39d432a9b472845876bfaf0, 974b802141bd4f13beffef2aa03f88f9; no voters: 1c50de89a5a845668696dcb28c3badd6
I20260129 02:58:44.816509 20631 consensus_queue.cc:260] T 78aa39096868415d9afc2ffcb248e198 P 1c50de89a5a845668696dcb28c3badd6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
W20260129 02:58:44.817077 20479 leader_election.cc:343] T 4e420f90884f4796997b5ea41e49e411 P 974b802141bd4f13beffef2aa03f88f9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 197b880fa39d432a9b472845876bfaf0 (127.18.211.1:42027): Illegal state: must be running to vote when last-logged opid is not known
I20260129 02:58:44.817684 20729 leader_election.cc:290] T e4d18c35b0a7493e85d0a4b92956a48f P 197b880fa39d432a9b472845876bfaf0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 974b802141bd4f13beffef2aa03f88f9 (127.18.211.2:34517), 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3:37735)
I20260129 02:58:44.817591 20479 leader_election.cc:304] T 4e420f90884f4796997b5ea41e49e411 P 974b802141bd4f13beffef2aa03f88f9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 974b802141bd4f13beffef2aa03f88f9; no voters: 197b880fa39d432a9b472845876bfaf0, 1c50de89a5a845668696dcb28c3badd6
I20260129 02:58:44.817158 20658 raft_consensus.cc:2804] T e4d18c35b0a7493e85d0a4b92956a48f P 197b880fa39d432a9b472845876bfaf0 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260129 02:58:44.818912 20658 raft_consensus.cc:493] T e4d18c35b0a7493e85d0a4b92956a48f P 197b880fa39d432a9b472845876bfaf0 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260129 02:58:44.819077 20740 raft_consensus.cc:2749] T 4e420f90884f4796997b5ea41e49e411 P 974b802141bd4f13beffef2aa03f88f9 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260129 02:58:44.819283 20658 raft_consensus.cc:3060] T e4d18c35b0a7493e85d0a4b92956a48f P 197b880fa39d432a9b472845876bfaf0 [term 0 FOLLOWER]: Advancing to term 1
I20260129 02:58:44.819463 20631 ts_tablet_manager.cc:1434] T 78aa39096868415d9afc2ffcb248e198 P 1c50de89a5a845668696dcb28c3badd6: Time spent starting tablet: real 0.014s user 0.009s sys 0.000s
I20260129 02:58:44.820833 20631 tablet_bootstrap.cc:492] T fcddab6bca684adf8ea032289a00f90e P 1c50de89a5a845668696dcb28c3badd6: Bootstrap starting.
I20260129 02:58:44.827185 20719 consensus_queue.cc:1048] T 3e2d15f8640b407195210c476e09df91 P 1c50de89a5a845668696dcb28c3badd6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260129 02:58:44.828694 20631 tablet_bootstrap.cc:654] T fcddab6bca684adf8ea032289a00f90e P 1c50de89a5a845668696dcb28c3badd6: Neither blocks nor log segments found. Creating new log.
I20260129 02:58:44.828420 20658 raft_consensus.cc:515] T e4d18c35b0a7493e85d0a4b92956a48f P 197b880fa39d432a9b472845876bfaf0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:58:44.834070 20514 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e4d18c35b0a7493e85d0a4b92956a48f" candidate_uuid: "197b880fa39d432a9b472845876bfaf0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "974b802141bd4f13beffef2aa03f88f9"
I20260129 02:58:44.834695 20514 raft_consensus.cc:3060] T e4d18c35b0a7493e85d0a4b92956a48f P 974b802141bd4f13beffef2aa03f88f9 [term 0 FOLLOWER]: Advancing to term 1
I20260129 02:58:44.835037 20658 leader_election.cc:290] T e4d18c35b0a7493e85d0a4b92956a48f P 197b880fa39d432a9b472845876bfaf0 [CANDIDATE]: Term 1 election: Requested vote from peers 974b802141bd4f13beffef2aa03f88f9 (127.18.211.2:34517), 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3:37735)
I20260129 02:58:44.836611 20590 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e4d18c35b0a7493e85d0a4b92956a48f" candidate_uuid: "197b880fa39d432a9b472845876bfaf0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c50de89a5a845668696dcb28c3badd6"
W20260129 02:58:44.838481 20405 leader_election.cc:343] T e4d18c35b0a7493e85d0a4b92956a48f P 197b880fa39d432a9b472845876bfaf0 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3:37735): Illegal state: must be running to vote when last-logged opid is not known
I20260129 02:58:44.842098 20514 raft_consensus.cc:2468] T e4d18c35b0a7493e85d0a4b92956a48f P 974b802141bd4f13beffef2aa03f88f9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 197b880fa39d432a9b472845876bfaf0 in term 1.
I20260129 02:58:44.846089 20630 tablet_bootstrap.cc:492] T a609ac02fae84725ab743f95f51d33ce P 974b802141bd4f13beffef2aa03f88f9: No bootstrap required, opened a new log
W20260129 02:58:44.846364 20759 log.cc:927] Time spent T 45cc5f3d2688486d9b6dbe7e666dedde P 974b802141bd4f13beffef2aa03f88f9: Append to log took a long time: real 0.057s user 0.001s sys 0.000s
I20260129 02:58:44.846551 20630 ts_tablet_manager.cc:1403] T a609ac02fae84725ab743f95f51d33ce P 974b802141bd4f13beffef2aa03f88f9: Time spent bootstrapping tablet: real 0.085s user 0.012s sys 0.009s
I20260129 02:58:44.848045 20407 leader_election.cc:304] T e4d18c35b0a7493e85d0a4b92956a48f P 197b880fa39d432a9b472845876bfaf0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 197b880fa39d432a9b472845876bfaf0, 974b802141bd4f13beffef2aa03f88f9; no voters: 1c50de89a5a845668696dcb28c3badd6
I20260129 02:58:44.849236 20658 raft_consensus.cc:2804] T e4d18c35b0a7493e85d0a4b92956a48f P 197b880fa39d432a9b472845876bfaf0 [term 1 FOLLOWER]: Leader election won for term 1
I20260129 02:58:44.849716 20658 raft_consensus.cc:697] T e4d18c35b0a7493e85d0a4b92956a48f P 197b880fa39d432a9b472845876bfaf0 [term 1 LEADER]: Becoming Leader. State: Replica: 197b880fa39d432a9b472845876bfaf0, State: Running, Role: LEADER
I20260129 02:58:44.850586 20658 consensus_queue.cc:237] T e4d18c35b0a7493e85d0a4b92956a48f P 197b880fa39d432a9b472845876bfaf0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:58:44.857518 20734 consensus_queue.cc:1048] T 17d6431cc77a491e8464d68db4081a51 P 974b802141bd4f13beffef2aa03f88f9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260129 02:58:44.859892 20638 raft_consensus.cc:493] T 45cc5f3d2688486d9b6dbe7e666dedde P 197b880fa39d432a9b472845876bfaf0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260129 02:58:44.860376 20638 raft_consensus.cc:515] T 45cc5f3d2688486d9b6dbe7e666dedde P 197b880fa39d432a9b472845876bfaf0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:58:44.860268 20630 raft_consensus.cc:359] T a609ac02fae84725ab743f95f51d33ce P 974b802141bd4f13beffef2aa03f88f9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:58:44.861250 20630 raft_consensus.cc:385] T a609ac02fae84725ab743f95f51d33ce P 974b802141bd4f13beffef2aa03f88f9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260129 02:58:44.861608 20630 raft_consensus.cc:740] T a609ac02fae84725ab743f95f51d33ce P 974b802141bd4f13beffef2aa03f88f9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 974b802141bd4f13beffef2aa03f88f9, State: Initialized, Role: FOLLOWER
I20260129 02:58:44.862430 20638 leader_election.cc:290] T 45cc5f3d2688486d9b6dbe7e666dedde P 197b880fa39d432a9b472845876bfaf0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 974b802141bd4f13beffef2aa03f88f9 (127.18.211.2:34517), 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3:37735)
I20260129 02:58:44.864112 20440 raft_consensus.cc:3060] T 45cc5f3d2688486d9b6dbe7e666dedde P 197b880fa39d432a9b472845876bfaf0 [term 0 FOLLOWER]: Advancing to term 1
I20260129 02:58:44.864238 20630 consensus_queue.cc:260] T a609ac02fae84725ab743f95f51d33ce P 974b802141bd4f13beffef2aa03f88f9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:58:44.866207 20588 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "45cc5f3d2688486d9b6dbe7e666dedde" candidate_uuid: "197b880fa39d432a9b472845876bfaf0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c50de89a5a845668696dcb28c3badd6" is_pre_election: true
I20260129 02:58:44.867084 20630 ts_tablet_manager.cc:1434] T a609ac02fae84725ab743f95f51d33ce P 974b802141bd4f13beffef2aa03f88f9: Time spent starting tablet: real 0.020s user 0.007s sys 0.000s
I20260129 02:58:44.869903 20516 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "45cc5f3d2688486d9b6dbe7e666dedde" candidate_uuid: "197b880fa39d432a9b472845876bfaf0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "974b802141bd4f13beffef2aa03f88f9" is_pre_election: true
I20260129 02:58:44.871806 20407 leader_election.cc:304] T 45cc5f3d2688486d9b6dbe7e666dedde P 197b880fa39d432a9b472845876bfaf0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 197b880fa39d432a9b472845876bfaf0; no voters: 1c50de89a5a845668696dcb28c3badd6, 974b802141bd4f13beffef2aa03f88f9
I20260129 02:58:44.874146 20641 raft_consensus.cc:493] T 78aa39096868415d9afc2ffcb248e198 P 1c50de89a5a845668696dcb28c3badd6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260129 02:58:44.874620 20641 raft_consensus.cc:515] T 78aa39096868415d9afc2ffcb248e198 P 1c50de89a5a845668696dcb28c3badd6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:58:44.876060 20630 tablet_bootstrap.cc:492] T febffe4a8b394368b69b1d164308fb9a P 974b802141bd4f13beffef2aa03f88f9: Bootstrap starting.
I20260129 02:58:44.876605 20641 leader_election.cc:290] T 78aa39096868415d9afc2ffcb248e198 P 1c50de89a5a845668696dcb28c3badd6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 197b880fa39d432a9b472845876bfaf0 (127.18.211.1:42027), 974b802141bd4f13beffef2aa03f88f9 (127.18.211.2:34517)
I20260129 02:58:44.878172 20516 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "78aa39096868415d9afc2ffcb248e198" candidate_uuid: "1c50de89a5a845668696dcb28c3badd6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "974b802141bd4f13beffef2aa03f88f9" is_pre_election: true
I20260129 02:58:44.878779 20516 raft_consensus.cc:2468] T 78aa39096868415d9afc2ffcb248e198 P 974b802141bd4f13beffef2aa03f88f9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1c50de89a5a845668696dcb28c3badd6 in term 0.
I20260129 02:58:44.879913 20556 leader_election.cc:304] T 78aa39096868415d9afc2ffcb248e198 P 1c50de89a5a845668696dcb28c3badd6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1c50de89a5a845668696dcb28c3badd6, 974b802141bd4f13beffef2aa03f88f9; no voters:
I20260129 02:58:44.880743 20719 consensus_queue.cc:1048] T 45cc5f3d2688486d9b6dbe7e666dedde P 1c50de89a5a845668696dcb28c3badd6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260129 02:58:44.881035 20641 raft_consensus.cc:2804] T 78aa39096868415d9afc2ffcb248e198 P 1c50de89a5a845668696dcb28c3badd6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260129 02:58:44.881165 20440 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "78aa39096868415d9afc2ffcb248e198" candidate_uuid: "1c50de89a5a845668696dcb28c3badd6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "197b880fa39d432a9b472845876bfaf0" is_pre_election: true
I20260129 02:58:44.881415 20641 raft_consensus.cc:493] T 78aa39096868415d9afc2ffcb248e198 P 1c50de89a5a845668696dcb28c3badd6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260129 02:58:44.881735 20729 raft_consensus.cc:2749] T 45cc5f3d2688486d9b6dbe7e666dedde P 197b880fa39d432a9b472845876bfaf0 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260129 02:58:44.881805 20641 raft_consensus.cc:3060] T 78aa39096868415d9afc2ffcb248e198 P 1c50de89a5a845668696dcb28c3badd6 [term 0 FOLLOWER]: Advancing to term 1
I20260129 02:58:44.883217 20630 tablet_bootstrap.cc:654] T febffe4a8b394368b69b1d164308fb9a P 974b802141bd4f13beffef2aa03f88f9: Neither blocks nor log segments found. Creating new log.
I20260129 02:58:44.887918 20440 raft_consensus.cc:2468] T 78aa39096868415d9afc2ffcb248e198 P 197b880fa39d432a9b472845876bfaf0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1c50de89a5a845668696dcb28c3badd6 in term 0.
I20260129 02:58:44.898772 20631 tablet_bootstrap.cc:492] T fcddab6bca684adf8ea032289a00f90e P 1c50de89a5a845668696dcb28c3badd6: No bootstrap required, opened a new log
I20260129 02:58:44.899266 20631 ts_tablet_manager.cc:1403] T fcddab6bca684adf8ea032289a00f90e P 1c50de89a5a845668696dcb28c3badd6: Time spent bootstrapping tablet: real 0.079s user 0.010s sys 0.009s
I20260129 02:58:44.902237 20631 raft_consensus.cc:359] T fcddab6bca684adf8ea032289a00f90e P 1c50de89a5a845668696dcb28c3badd6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:58:44.903095 20641 raft_consensus.cc:515] T 78aa39096868415d9afc2ffcb248e198 P 1c50de89a5a845668696dcb28c3badd6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:58:44.903560 20631 raft_consensus.cc:385] T fcddab6bca684adf8ea032289a00f90e P 1c50de89a5a845668696dcb28c3badd6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260129 02:58:44.903376 20729 consensus_queue.cc:1048] T 0f20af7fbe5d4db89a6f17948d98912c P 197b880fa39d432a9b472845876bfaf0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260129 02:58:44.904160 20631 raft_consensus.cc:740] T fcddab6bca684adf8ea032289a00f90e P 1c50de89a5a845668696dcb28c3badd6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1c50de89a5a845668696dcb28c3badd6, State: Initialized, Role: FOLLOWER
I20260129 02:58:44.902802 20633 tablet_bootstrap.cc:492] T add8d4923d1a489e86882c3a25692b95 P 197b880fa39d432a9b472845876bfaf0: No bootstrap required, opened a new log
I20260129 02:58:44.905227 20631 consensus_queue.cc:260] T fcddab6bca684adf8ea032289a00f90e P 1c50de89a5a845668696dcb28c3badd6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:58:44.905848 20633 ts_tablet_manager.cc:1403] T add8d4923d1a489e86882c3a25692b95 P 197b880fa39d432a9b472845876bfaf0: Time spent bootstrapping tablet: real 0.113s user 0.004s sys 0.014s
I20260129 02:58:44.907541 20440 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "78aa39096868415d9afc2ffcb248e198" candidate_uuid: "1c50de89a5a845668696dcb28c3badd6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "197b880fa39d432a9b472845876bfaf0"
I20260129 02:58:44.908169 20440 raft_consensus.cc:3060] T 78aa39096868415d9afc2ffcb248e198 P 197b880fa39d432a9b472845876bfaf0 [term 0 FOLLOWER]: Advancing to term 1
I20260129 02:58:44.908284 20641 leader_election.cc:290] T 78aa39096868415d9afc2ffcb248e198 P 1c50de89a5a845668696dcb28c3badd6 [CANDIDATE]: Term 1 election: Requested vote from peers 197b880fa39d432a9b472845876bfaf0 (127.18.211.1:42027), 974b802141bd4f13beffef2aa03f88f9 (127.18.211.2:34517)
I20260129 02:58:44.907670 20336 catalog_manager.cc:5654] T e4d18c35b0a7493e85d0a4b92956a48f P 197b880fa39d432a9b472845876bfaf0 reported cstate change: term changed from 0 to 1, leader changed from <none> to 197b880fa39d432a9b472845876bfaf0 (127.18.211.1). New cstate: current_term: 1 leader_uuid: "197b880fa39d432a9b472845876bfaf0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } health_report { overall_health: UNKNOWN } } }
I20260129 02:58:44.911932 20516 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "78aa39096868415d9afc2ffcb248e198" candidate_uuid: "1c50de89a5a845668696dcb28c3badd6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "974b802141bd4f13beffef2aa03f88f9"
I20260129 02:58:44.912505 20516 raft_consensus.cc:3060] T 78aa39096868415d9afc2ffcb248e198 P 974b802141bd4f13beffef2aa03f88f9 [term 0 FOLLOWER]: Advancing to term 1
I20260129 02:58:44.914561 20631 ts_tablet_manager.cc:1434] T fcddab6bca684adf8ea032289a00f90e P 1c50de89a5a845668696dcb28c3badd6: Time spent starting tablet: real 0.015s user 0.005s sys 0.000s
I20260129 02:58:44.915318 20630 tablet_bootstrap.cc:492] T febffe4a8b394368b69b1d164308fb9a P 974b802141bd4f13beffef2aa03f88f9: No bootstrap required, opened a new log
I20260129 02:58:44.915462 20631 tablet_bootstrap.cc:492] T fd8311b779af4e1fa1402b2a02a602f2 P 1c50de89a5a845668696dcb28c3badd6: Bootstrap starting.
I20260129 02:58:44.915761 20630 ts_tablet_manager.cc:1403] T febffe4a8b394368b69b1d164308fb9a P 974b802141bd4f13beffef2aa03f88f9: Time spent bootstrapping tablet: real 0.040s user 0.014s sys 0.005s
I20260129 02:58:44.916975 20440 raft_consensus.cc:2468] T 78aa39096868415d9afc2ffcb248e198 P 197b880fa39d432a9b472845876bfaf0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1c50de89a5a845668696dcb28c3badd6 in term 1.
I20260129 02:58:44.919250 20553 leader_election.cc:304] T 78aa39096868415d9afc2ffcb248e198 P 1c50de89a5a845668696dcb28c3badd6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 197b880fa39d432a9b472845876bfaf0, 1c50de89a5a845668696dcb28c3badd6; no voters:
I20260129 02:58:44.922966 20633 raft_consensus.cc:359] T add8d4923d1a489e86882c3a25692b95 P 197b880fa39d432a9b472845876bfaf0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:58:44.923662 20633 raft_consensus.cc:385] T add8d4923d1a489e86882c3a25692b95 P 197b880fa39d432a9b472845876bfaf0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260129 02:58:44.923947 20633 raft_consensus.cc:740] T add8d4923d1a489e86882c3a25692b95 P 197b880fa39d432a9b472845876bfaf0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 197b880fa39d432a9b472845876bfaf0, State: Initialized, Role: FOLLOWER
I20260129 02:58:44.926563 20719 raft_consensus.cc:2804] T 78aa39096868415d9afc2ffcb248e198 P 1c50de89a5a845668696dcb28c3badd6 [term 1 FOLLOWER]: Leader election won for term 1
I20260129 02:58:44.928261 20630 raft_consensus.cc:359] T febffe4a8b394368b69b1d164308fb9a P 974b802141bd4f13beffef2aa03f88f9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:58:44.929009 20630 raft_consensus.cc:385] T febffe4a8b394368b69b1d164308fb9a P 974b802141bd4f13beffef2aa03f88f9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260129 02:58:44.929284 20630 raft_consensus.cc:740] T febffe4a8b394368b69b1d164308fb9a P 974b802141bd4f13beffef2aa03f88f9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 974b802141bd4f13beffef2aa03f88f9, State: Initialized, Role: FOLLOWER
I20260129 02:58:44.929947 20630 consensus_queue.cc:260] T febffe4a8b394368b69b1d164308fb9a P 974b802141bd4f13beffef2aa03f88f9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:58:44.924554 20633 consensus_queue.cc:260] T add8d4923d1a489e86882c3a25692b95 P 197b880fa39d432a9b472845876bfaf0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:58:44.942432 20633 ts_tablet_manager.cc:1434] T add8d4923d1a489e86882c3a25692b95 P 197b880fa39d432a9b472845876bfaf0: Time spent starting tablet: real 0.036s user 0.006s sys 0.006s
I20260129 02:58:44.943383 20633 tablet_bootstrap.cc:492] T 1ce4ddf5b10e4ae8b193b13b2e5936ee P 197b880fa39d432a9b472845876bfaf0: Bootstrap starting.
I20260129 02:58:44.944288 20630 ts_tablet_manager.cc:1434] T febffe4a8b394368b69b1d164308fb9a P 974b802141bd4f13beffef2aa03f88f9: Time spent starting tablet: real 0.028s user 0.006s sys 0.000s
I20260129 02:58:44.945262 20630 tablet_bootstrap.cc:492] T fcddab6bca684adf8ea032289a00f90e P 974b802141bd4f13beffef2aa03f88f9: Bootstrap starting.
I20260129 02:58:44.947974 20516 raft_consensus.cc:2468] T 78aa39096868415d9afc2ffcb248e198 P 974b802141bd4f13beffef2aa03f88f9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1c50de89a5a845668696dcb28c3badd6 in term 1.
I20260129 02:58:44.949010 20719 raft_consensus.cc:697] T 78aa39096868415d9afc2ffcb248e198 P 1c50de89a5a845668696dcb28c3badd6 [term 1 LEADER]: Becoming Leader. State: Replica: 1c50de89a5a845668696dcb28c3badd6, State: Running, Role: LEADER
I20260129 02:58:44.951581 20631 tablet_bootstrap.cc:654] T fd8311b779af4e1fa1402b2a02a602f2 P 1c50de89a5a845668696dcb28c3badd6: Neither blocks nor log segments found. Creating new log.
I20260129 02:58:44.949977 20719 consensus_queue.cc:237] T 78aa39096868415d9afc2ffcb248e198 P 1c50de89a5a845668696dcb28c3badd6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:58:44.958470 20630 tablet_bootstrap.cc:654] T fcddab6bca684adf8ea032289a00f90e P 974b802141bd4f13beffef2aa03f88f9: Neither blocks nor log segments found. Creating new log.
I20260129 02:58:44.967188 20633 tablet_bootstrap.cc:654] T 1ce4ddf5b10e4ae8b193b13b2e5936ee P 197b880fa39d432a9b472845876bfaf0: Neither blocks nor log segments found. Creating new log.
I20260129 02:58:44.963249 20721 consensus_queue.cc:1048] T 0f20af7fbe5d4db89a6f17948d98912c P 197b880fa39d432a9b472845876bfaf0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.006s
I20260129 02:58:44.987838 20336 catalog_manager.cc:5654] T 78aa39096868415d9afc2ffcb248e198 P 1c50de89a5a845668696dcb28c3badd6 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3). New cstate: current_term: 1 leader_uuid: "1c50de89a5a845668696dcb28c3badd6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } health_report { overall_health: HEALTHY } } }
I20260129 02:58:45.009099 20631 tablet_bootstrap.cc:492] T fd8311b779af4e1fa1402b2a02a602f2 P 1c50de89a5a845668696dcb28c3badd6: No bootstrap required, opened a new log
I20260129 02:58:45.009410 20630 tablet_bootstrap.cc:492] T fcddab6bca684adf8ea032289a00f90e P 974b802141bd4f13beffef2aa03f88f9: No bootstrap required, opened a new log
I20260129 02:58:45.009631 20633 tablet_bootstrap.cc:492] T 1ce4ddf5b10e4ae8b193b13b2e5936ee P 197b880fa39d432a9b472845876bfaf0: No bootstrap required, opened a new log
I20260129 02:58:45.009593 20631 ts_tablet_manager.cc:1403] T fd8311b779af4e1fa1402b2a02a602f2 P 1c50de89a5a845668696dcb28c3badd6: Time spent bootstrapping tablet: real 0.094s user 0.003s sys 0.012s
I20260129 02:58:45.009851 20630 ts_tablet_manager.cc:1403] T fcddab6bca684adf8ea032289a00f90e P 974b802141bd4f13beffef2aa03f88f9: Time spent bootstrapping tablet: real 0.065s user 0.010s sys 0.001s
I20260129 02:58:45.010202 20633 ts_tablet_manager.cc:1403] T 1ce4ddf5b10e4ae8b193b13b2e5936ee P 197b880fa39d432a9b472845876bfaf0: Time spent bootstrapping tablet: real 0.067s user 0.010s sys 0.002s
I20260129 02:58:45.011142 20719 raft_consensus.cc:493] T b189f08e16a74a11980cc55d99359a0a P 1c50de89a5a845668696dcb28c3badd6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260129 02:58:45.011693 20719 raft_consensus.cc:515] T b189f08e16a74a11980cc55d99359a0a P 1c50de89a5a845668696dcb28c3badd6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:58:45.012429 20631 raft_consensus.cc:359] T fd8311b779af4e1fa1402b2a02a602f2 P 1c50de89a5a845668696dcb28c3badd6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:58:45.013087 20631 raft_consensus.cc:385] T fd8311b779af4e1fa1402b2a02a602f2 P 1c50de89a5a845668696dcb28c3badd6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260129 02:58:45.013381 20631 raft_consensus.cc:740] T fd8311b779af4e1fa1402b2a02a602f2 P 1c50de89a5a845668696dcb28c3badd6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1c50de89a5a845668696dcb28c3badd6, State: Initialized, Role: FOLLOWER
I20260129 02:58:45.014003 20631 consensus_queue.cc:260] T fd8311b779af4e1fa1402b2a02a602f2 P 1c50de89a5a845668696dcb28c3badd6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:58:45.014077 20633 raft_consensus.cc:359] T 1ce4ddf5b10e4ae8b193b13b2e5936ee P 197b880fa39d432a9b472845876bfaf0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:58:45.012768 20630 raft_consensus.cc:359] T fcddab6bca684adf8ea032289a00f90e P 974b802141bd4f13beffef2aa03f88f9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:58:45.016415 20630 raft_consensus.cc:385] T fcddab6bca684adf8ea032289a00f90e P 974b802141bd4f13beffef2aa03f88f9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260129 02:58:45.014894 20440 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b189f08e16a74a11980cc55d99359a0a" candidate_uuid: "1c50de89a5a845668696dcb28c3badd6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "197b880fa39d432a9b472845876bfaf0" is_pre_election: true
I20260129 02:58:45.014777 20633 raft_consensus.cc:385] T 1ce4ddf5b10e4ae8b193b13b2e5936ee P 197b880fa39d432a9b472845876bfaf0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260129 02:58:45.016424 20515 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b189f08e16a74a11980cc55d99359a0a" candidate_uuid: "1c50de89a5a845668696dcb28c3badd6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "974b802141bd4f13beffef2aa03f88f9" is_pre_election: true
I20260129 02:58:45.016942 20633 raft_consensus.cc:740] T 1ce4ddf5b10e4ae8b193b13b2e5936ee P 197b880fa39d432a9b472845876bfaf0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 197b880fa39d432a9b472845876bfaf0, State: Initialized, Role: FOLLOWER
I20260129 02:58:45.016254 20631 ts_tablet_manager.cc:1434] T fd8311b779af4e1fa1402b2a02a602f2 P 1c50de89a5a845668696dcb28c3badd6: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260129 02:58:45.017237 20515 raft_consensus.cc:2468] T b189f08e16a74a11980cc55d99359a0a P 974b802141bd4f13beffef2aa03f88f9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1c50de89a5a845668696dcb28c3badd6 in term 0.
I20260129 02:58:45.017421 20630 raft_consensus.cc:740] T fcddab6bca684adf8ea032289a00f90e P 974b802141bd4f13beffef2aa03f88f9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 974b802141bd4f13beffef2aa03f88f9, State: Initialized, Role: FOLLOWER
I20260129 02:58:45.017554 20633 consensus_queue.cc:260] T 1ce4ddf5b10e4ae8b193b13b2e5936ee P 197b880fa39d432a9b472845876bfaf0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:58:45.018582 20631 tablet_bootstrap.cc:492] T 1ce4ddf5b10e4ae8b193b13b2e5936ee P 1c50de89a5a845668696dcb28c3badd6: Bootstrap starting.
I20260129 02:58:45.019575 20719 leader_election.cc:290] T b189f08e16a74a11980cc55d99359a0a P 1c50de89a5a845668696dcb28c3badd6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 197b880fa39d432a9b472845876bfaf0 (127.18.211.1:42027), 974b802141bd4f13beffef2aa03f88f9 (127.18.211.2:34517)
W20260129 02:58:45.018687 20553 leader_election.cc:343] T b189f08e16a74a11980cc55d99359a0a P 1c50de89a5a845668696dcb28c3badd6 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 197b880fa39d432a9b472845876bfaf0 (127.18.211.1:42027): Illegal state: must be running to vote when last-logged opid is not known
I20260129 02:58:45.020124 20633 ts_tablet_manager.cc:1434] T 1ce4ddf5b10e4ae8b193b13b2e5936ee P 197b880fa39d432a9b472845876bfaf0: Time spent starting tablet: real 0.010s user 0.004s sys 0.002s
I20260129 02:58:45.022122 20556 leader_election.cc:304] T b189f08e16a74a11980cc55d99359a0a P 1c50de89a5a845668696dcb28c3badd6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1c50de89a5a845668696dcb28c3badd6, 974b802141bd4f13beffef2aa03f88f9; no voters: 197b880fa39d432a9b472845876bfaf0
I20260129 02:58:45.020128 20630 consensus_queue.cc:260] T fcddab6bca684adf8ea032289a00f90e P 974b802141bd4f13beffef2aa03f88f9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:58:45.024238 20633 tablet_bootstrap.cc:492] T 4e420f90884f4796997b5ea41e49e411 P 197b880fa39d432a9b472845876bfaf0: Bootstrap starting.
I20260129 02:58:45.026605 20719 raft_consensus.cc:2804] T b189f08e16a74a11980cc55d99359a0a P 1c50de89a5a845668696dcb28c3badd6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260129 02:58:45.027009 20719 raft_consensus.cc:493] T b189f08e16a74a11980cc55d99359a0a P 1c50de89a5a845668696dcb28c3badd6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260129 02:58:45.027371 20719 raft_consensus.cc:3060] T b189f08e16a74a11980cc55d99359a0a P 1c50de89a5a845668696dcb28c3badd6 [term 0 FOLLOWER]: Advancing to term 1
I20260129 02:58:45.030771 20633 tablet_bootstrap.cc:654] T 4e420f90884f4796997b5ea41e49e411 P 197b880fa39d432a9b472845876bfaf0: Neither blocks nor log segments found. Creating new log.
I20260129 02:58:45.034658 20630 ts_tablet_manager.cc:1434] T fcddab6bca684adf8ea032289a00f90e P 974b802141bd4f13beffef2aa03f88f9: Time spent starting tablet: real 0.024s user 0.006s sys 0.000s
I20260129 02:58:45.035732 20631 tablet_bootstrap.cc:654] T 1ce4ddf5b10e4ae8b193b13b2e5936ee P 1c50de89a5a845668696dcb28c3badd6: Neither blocks nor log segments found. Creating new log.
I20260129 02:58:45.035845 20719 raft_consensus.cc:515] T b189f08e16a74a11980cc55d99359a0a P 1c50de89a5a845668696dcb28c3badd6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:58:45.039383 20440 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b189f08e16a74a11980cc55d99359a0a" candidate_uuid: "1c50de89a5a845668696dcb28c3badd6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "197b880fa39d432a9b472845876bfaf0"
I20260129 02:58:45.040119 20630 tablet_bootstrap.cc:492] T fd8311b779af4e1fa1402b2a02a602f2 P 974b802141bd4f13beffef2aa03f88f9: Bootstrap starting.
W20260129 02:58:45.040998 20553 leader_election.cc:343] T b189f08e16a74a11980cc55d99359a0a P 1c50de89a5a845668696dcb28c3badd6 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 197b880fa39d432a9b472845876bfaf0 (127.18.211.1:42027): Illegal state: must be running to vote when last-logged opid is not known
I20260129 02:58:45.043476 20515 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b189f08e16a74a11980cc55d99359a0a" candidate_uuid: "1c50de89a5a845668696dcb28c3badd6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "974b802141bd4f13beffef2aa03f88f9"
I20260129 02:58:45.044246 20515 raft_consensus.cc:3060] T b189f08e16a74a11980cc55d99359a0a P 974b802141bd4f13beffef2aa03f88f9 [term 0 FOLLOWER]: Advancing to term 1
I20260129 02:58:45.046079 20630 tablet_bootstrap.cc:654] T fd8311b779af4e1fa1402b2a02a602f2 P 974b802141bd4f13beffef2aa03f88f9: Neither blocks nor log segments found. Creating new log.
I20260129 02:58:45.051223 20633 tablet_bootstrap.cc:492] T 4e420f90884f4796997b5ea41e49e411 P 197b880fa39d432a9b472845876bfaf0: No bootstrap required, opened a new log
I20260129 02:58:45.051808 20633 ts_tablet_manager.cc:1403] T 4e420f90884f4796997b5ea41e49e411 P 197b880fa39d432a9b472845876bfaf0: Time spent bootstrapping tablet: real 0.028s user 0.013s sys 0.011s
I20260129 02:58:45.052217 20515 raft_consensus.cc:2468] T b189f08e16a74a11980cc55d99359a0a P 974b802141bd4f13beffef2aa03f88f9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1c50de89a5a845668696dcb28c3badd6 in term 1.
I20260129 02:58:45.053378 20556 leader_election.cc:304] T b189f08e16a74a11980cc55d99359a0a P 1c50de89a5a845668696dcb28c3badd6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1c50de89a5a845668696dcb28c3badd6, 974b802141bd4f13beffef2aa03f88f9; no voters: 197b880fa39d432a9b472845876bfaf0
I20260129 02:58:45.054296 20771 raft_consensus.cc:2804] T b189f08e16a74a11980cc55d99359a0a P 1c50de89a5a845668696dcb28c3badd6 [term 1 FOLLOWER]: Leader election won for term 1
I20260129 02:58:45.054811 20771 raft_consensus.cc:697] T b189f08e16a74a11980cc55d99359a0a P 1c50de89a5a845668696dcb28c3badd6 [term 1 LEADER]: Becoming Leader. State: Replica: 1c50de89a5a845668696dcb28c3badd6, State: Running, Role: LEADER
I20260129 02:58:45.055723 20633 raft_consensus.cc:359] T 4e420f90884f4796997b5ea41e49e411 P 197b880fa39d432a9b472845876bfaf0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:58:45.056644 20633 raft_consensus.cc:385] T 4e420f90884f4796997b5ea41e49e411 P 197b880fa39d432a9b472845876bfaf0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260129 02:58:45.055663 20771 consensus_queue.cc:237] T b189f08e16a74a11980cc55d99359a0a P 1c50de89a5a845668696dcb28c3badd6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:58:45.056586 20719 leader_election.cc:290] T b189f08e16a74a11980cc55d99359a0a P 1c50de89a5a845668696dcb28c3badd6 [CANDIDATE]: Term 1 election: Requested vote from peers 197b880fa39d432a9b472845876bfaf0 (127.18.211.1:42027), 974b802141bd4f13beffef2aa03f88f9 (127.18.211.2:34517)
I20260129 02:58:45.058514 20633 raft_consensus.cc:740] T 4e420f90884f4796997b5ea41e49e411 P 197b880fa39d432a9b472845876bfaf0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 197b880fa39d432a9b472845876bfaf0, State: Initialized, Role: FOLLOWER
I20260129 02:58:45.056318 20631 tablet_bootstrap.cc:492] T 1ce4ddf5b10e4ae8b193b13b2e5936ee P 1c50de89a5a845668696dcb28c3badd6: No bootstrap required, opened a new log
I20260129 02:58:45.059900 20631 ts_tablet_manager.cc:1403] T 1ce4ddf5b10e4ae8b193b13b2e5936ee P 1c50de89a5a845668696dcb28c3badd6: Time spent bootstrapping tablet: real 0.041s user 0.012s sys 0.005s
I20260129 02:58:45.059540 20633 consensus_queue.cc:260] T 4e420f90884f4796997b5ea41e49e411 P 197b880fa39d432a9b472845876bfaf0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:58:45.062319 20631 raft_consensus.cc:359] T 1ce4ddf5b10e4ae8b193b13b2e5936ee P 1c50de89a5a845668696dcb28c3badd6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:58:45.062927 20631 raft_consensus.cc:385] T 1ce4ddf5b10e4ae8b193b13b2e5936ee P 1c50de89a5a845668696dcb28c3badd6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260129 02:58:45.063212 20631 raft_consensus.cc:740] T 1ce4ddf5b10e4ae8b193b13b2e5936ee P 1c50de89a5a845668696dcb28c3badd6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1c50de89a5a845668696dcb28c3badd6, State: Initialized, Role: FOLLOWER
I20260129 02:58:45.064087 20631 consensus_queue.cc:260] T 1ce4ddf5b10e4ae8b193b13b2e5936ee P 1c50de89a5a845668696dcb28c3badd6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:58:45.065725 20633 ts_tablet_manager.cc:1434] T 4e420f90884f4796997b5ea41e49e411 P 197b880fa39d432a9b472845876bfaf0: Time spent starting tablet: real 0.014s user 0.006s sys 0.000s
I20260129 02:58:45.066974 20631 ts_tablet_manager.cc:1434] T 1ce4ddf5b10e4ae8b193b13b2e5936ee P 1c50de89a5a845668696dcb28c3badd6: Time spent starting tablet: real 0.007s user 0.005s sys 0.000s
I20260129 02:58:45.067087 20633 tablet_bootstrap.cc:492] T fcddab6bca684adf8ea032289a00f90e P 197b880fa39d432a9b472845876bfaf0: Bootstrap starting.
I20260129 02:58:45.067894 20631 tablet_bootstrap.cc:492] T 4f8f3a38f6e24ba4857875181f447442 P 1c50de89a5a845668696dcb28c3badd6: Bootstrap starting.
I20260129 02:58:45.071609 20630 tablet_bootstrap.cc:492] T fd8311b779af4e1fa1402b2a02a602f2 P 974b802141bd4f13beffef2aa03f88f9: No bootstrap required, opened a new log
I20260129 02:58:45.072139 20630 ts_tablet_manager.cc:1403] T fd8311b779af4e1fa1402b2a02a602f2 P 974b802141bd4f13beffef2aa03f88f9: Time spent bootstrapping tablet: real 0.032s user 0.010s sys 0.004s
I20260129 02:58:45.074555 20630 raft_consensus.cc:359] T fd8311b779af4e1fa1402b2a02a602f2 P 974b802141bd4f13beffef2aa03f88f9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:58:45.075193 20630 raft_consensus.cc:385] T fd8311b779af4e1fa1402b2a02a602f2 P 974b802141bd4f13beffef2aa03f88f9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260129 02:58:45.075450 20630 raft_consensus.cc:740] T fd8311b779af4e1fa1402b2a02a602f2 P 974b802141bd4f13beffef2aa03f88f9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 974b802141bd4f13beffef2aa03f88f9, State: Initialized, Role: FOLLOWER
I20260129 02:58:45.076155 20633 tablet_bootstrap.cc:654] T fcddab6bca684adf8ea032289a00f90e P 197b880fa39d432a9b472845876bfaf0: Neither blocks nor log segments found. Creating new log.
I20260129 02:58:45.077749 20630 consensus_queue.cc:260] T fd8311b779af4e1fa1402b2a02a602f2 P 974b802141bd4f13beffef2aa03f88f9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:58:45.080377 20630 ts_tablet_manager.cc:1434] T fd8311b779af4e1fa1402b2a02a602f2 P 974b802141bd4f13beffef2aa03f88f9: Time spent starting tablet: real 0.008s user 0.006s sys 0.000s
I20260129 02:58:45.081282 20630 tablet_bootstrap.cc:492] T dc2f4cb752cf4d59b5f6bc689f5d8104 P 974b802141bd4f13beffef2aa03f88f9: Bootstrap starting.
I20260129 02:58:45.087131 20630 tablet_bootstrap.cc:654] T dc2f4cb752cf4d59b5f6bc689f5d8104 P 974b802141bd4f13beffef2aa03f88f9: Neither blocks nor log segments found. Creating new log.
I20260129 02:58:45.092455 20633 tablet_bootstrap.cc:492] T fcddab6bca684adf8ea032289a00f90e P 197b880fa39d432a9b472845876bfaf0: No bootstrap required, opened a new log
I20260129 02:58:45.092868 20631 tablet_bootstrap.cc:654] T 4f8f3a38f6e24ba4857875181f447442 P 1c50de89a5a845668696dcb28c3badd6: Neither blocks nor log segments found. Creating new log.
I20260129 02:58:45.092971 20633 ts_tablet_manager.cc:1403] T fcddab6bca684adf8ea032289a00f90e P 197b880fa39d432a9b472845876bfaf0: Time spent bootstrapping tablet: real 0.026s user 0.009s sys 0.011s
I20260129 02:58:45.094362 20719 raft_consensus.cc:493] T add8d4923d1a489e86882c3a25692b95 P 1c50de89a5a845668696dcb28c3badd6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260129 02:58:45.094849 20719 raft_consensus.cc:515] T add8d4923d1a489e86882c3a25692b95 P 1c50de89a5a845668696dcb28c3badd6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:58:45.096354 20633 raft_consensus.cc:359] T fcddab6bca684adf8ea032289a00f90e P 197b880fa39d432a9b472845876bfaf0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:58:45.096773 20719 leader_election.cc:290] T add8d4923d1a489e86882c3a25692b95 P 1c50de89a5a845668696dcb28c3badd6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 197b880fa39d432a9b472845876bfaf0 (127.18.211.1:42027), 974b802141bd4f13beffef2aa03f88f9 (127.18.211.2:34517)
I20260129 02:58:45.097254 20633 raft_consensus.cc:385] T fcddab6bca684adf8ea032289a00f90e P 197b880fa39d432a9b472845876bfaf0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260129 02:58:45.098408 20515 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "add8d4923d1a489e86882c3a25692b95" candidate_uuid: "1c50de89a5a845668696dcb28c3badd6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "974b802141bd4f13beffef2aa03f88f9" is_pre_election: true
I20260129 02:58:45.099009 20515 raft_consensus.cc:2468] T add8d4923d1a489e86882c3a25692b95 P 974b802141bd4f13beffef2aa03f88f9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1c50de89a5a845668696dcb28c3badd6 in term 0.
I20260129 02:58:45.097566 20633 raft_consensus.cc:740] T fcddab6bca684adf8ea032289a00f90e P 197b880fa39d432a9b472845876bfaf0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 197b880fa39d432a9b472845876bfaf0, State: Initialized, Role: FOLLOWER
I20260129 02:58:45.099974 20440 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "add8d4923d1a489e86882c3a25692b95" candidate_uuid: "1c50de89a5a845668696dcb28c3badd6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "197b880fa39d432a9b472845876bfaf0" is_pre_election: true
I20260129 02:58:45.099895 20633 consensus_queue.cc:260] T fcddab6bca684adf8ea032289a00f90e P 197b880fa39d432a9b472845876bfaf0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:58:45.100554 20440 raft_consensus.cc:2468] T add8d4923d1a489e86882c3a25692b95 P 197b880fa39d432a9b472845876bfaf0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1c50de89a5a845668696dcb28c3badd6 in term 0.
I20260129 02:58:45.103199 20630 tablet_bootstrap.cc:492] T dc2f4cb752cf4d59b5f6bc689f5d8104 P 974b802141bd4f13beffef2aa03f88f9: No bootstrap required, opened a new log
I20260129 02:58:45.103250 20633 ts_tablet_manager.cc:1434] T fcddab6bca684adf8ea032289a00f90e P 197b880fa39d432a9b472845876bfaf0: Time spent starting tablet: real 0.010s user 0.007s sys 0.000s
I20260129 02:58:45.103763 20630 ts_tablet_manager.cc:1403] T dc2f4cb752cf4d59b5f6bc689f5d8104 P 974b802141bd4f13beffef2aa03f88f9: Time spent bootstrapping tablet: real 0.023s user 0.009s sys 0.001s
I20260129 02:58:45.104357 20633 tablet_bootstrap.cc:492] T febffe4a8b394368b69b1d164308fb9a P 197b880fa39d432a9b472845876bfaf0: Bootstrap starting.
I20260129 02:58:45.106081 20553 leader_election.cc:304] T add8d4923d1a489e86882c3a25692b95 P 1c50de89a5a845668696dcb28c3badd6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 197b880fa39d432a9b472845876bfaf0, 1c50de89a5a845668696dcb28c3badd6; no voters:
I20260129 02:58:45.106226 20630 raft_consensus.cc:359] T dc2f4cb752cf4d59b5f6bc689f5d8104 P 974b802141bd4f13beffef2aa03f88f9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:58:45.106997 20630 raft_consensus.cc:385] T dc2f4cb752cf4d59b5f6bc689f5d8104 P 974b802141bd4f13beffef2aa03f88f9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260129 02:58:45.107230 20771 raft_consensus.cc:2804] T add8d4923d1a489e86882c3a25692b95 P 1c50de89a5a845668696dcb28c3badd6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260129 02:58:45.107290 20630 raft_consensus.cc:740] T dc2f4cb752cf4d59b5f6bc689f5d8104 P 974b802141bd4f13beffef2aa03f88f9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 974b802141bd4f13beffef2aa03f88f9, State: Initialized, Role: FOLLOWER
I20260129 02:58:45.107751 20771 raft_consensus.cc:493] T add8d4923d1a489e86882c3a25692b95 P 1c50de89a5a845668696dcb28c3badd6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260129 02:58:45.109275 20771 raft_consensus.cc:3060] T add8d4923d1a489e86882c3a25692b95 P 1c50de89a5a845668696dcb28c3badd6 [term 0 FOLLOWER]: Advancing to term 1
I20260129 02:58:45.110158 20630 consensus_queue.cc:260] T dc2f4cb752cf4d59b5f6bc689f5d8104 P 974b802141bd4f13beffef2aa03f88f9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:58:45.111595 20633 tablet_bootstrap.cc:654] T febffe4a8b394368b69b1d164308fb9a P 197b880fa39d432a9b472845876bfaf0: Neither blocks nor log segments found. Creating new log.
I20260129 02:58:45.112445 20658 raft_consensus.cc:493] T 4e420f90884f4796997b5ea41e49e411 P 197b880fa39d432a9b472845876bfaf0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260129 02:58:45.112978 20658 raft_consensus.cc:515] T 4e420f90884f4796997b5ea41e49e411 P 197b880fa39d432a9b472845876bfaf0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:58:45.114970 20658 leader_election.cc:290] T 4e420f90884f4796997b5ea41e49e411 P 197b880fa39d432a9b472845876bfaf0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 974b802141bd4f13beffef2aa03f88f9 (127.18.211.2:34517), 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3:37735)
I20260129 02:58:45.116766 20771 raft_consensus.cc:515] T add8d4923d1a489e86882c3a25692b95 P 1c50de89a5a845668696dcb28c3badd6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:58:45.117018 20588 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4e420f90884f4796997b5ea41e49e411" candidate_uuid: "197b880fa39d432a9b472845876bfaf0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c50de89a5a845668696dcb28c3badd6" is_pre_election: true
I20260129 02:58:45.120071 20441 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "add8d4923d1a489e86882c3a25692b95" candidate_uuid: "1c50de89a5a845668696dcb28c3badd6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "197b880fa39d432a9b472845876bfaf0"
I20260129 02:58:45.120684 20441 raft_consensus.cc:3060] T add8d4923d1a489e86882c3a25692b95 P 197b880fa39d432a9b472845876bfaf0 [term 0 FOLLOWER]: Advancing to term 1
W20260129 02:58:45.120851 20405 leader_election.cc:343] T 4e420f90884f4796997b5ea41e49e411 P 197b880fa39d432a9b472845876bfaf0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3:37735): Illegal state: must be running to vote when last-logged opid is not known
I20260129 02:58:45.123576 20515 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4e420f90884f4796997b5ea41e49e411" candidate_uuid: "197b880fa39d432a9b472845876bfaf0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "974b802141bd4f13beffef2aa03f88f9" is_pre_election: true
I20260129 02:58:45.124434 20515 raft_consensus.cc:2468] T 4e420f90884f4796997b5ea41e49e411 P 974b802141bd4f13beffef2aa03f88f9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 197b880fa39d432a9b472845876bfaf0 in term 0.
I20260129 02:58:45.125754 20407 leader_election.cc:304] T 4e420f90884f4796997b5ea41e49e411 P 197b880fa39d432a9b472845876bfaf0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 197b880fa39d432a9b472845876bfaf0, 974b802141bd4f13beffef2aa03f88f9; no voters: 1c50de89a5a845668696dcb28c3badd6
I20260129 02:58:45.126914 20658 raft_consensus.cc:2804] T 4e420f90884f4796997b5ea41e49e411 P 197b880fa39d432a9b472845876bfaf0 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260129 02:58:45.127250 20658 raft_consensus.cc:493] T 4e420f90884f4796997b5ea41e49e411 P 197b880fa39d432a9b472845876bfaf0 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260129 02:58:45.127565 20658 raft_consensus.cc:3060] T 4e420f90884f4796997b5ea41e49e411 P 197b880fa39d432a9b472845876bfaf0 [term 0 FOLLOWER]: Advancing to term 1
I20260129 02:58:45.132218 20337 catalog_manager.cc:5654] T b189f08e16a74a11980cc55d99359a0a P 1c50de89a5a845668696dcb28c3badd6 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3). New cstate: current_term: 1 leader_uuid: "1c50de89a5a845668696dcb28c3badd6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } health_report { overall_health: HEALTHY } } }
I20260129 02:58:45.135617 20658 raft_consensus.cc:515] T 4e420f90884f4796997b5ea41e49e411 P 197b880fa39d432a9b472845876bfaf0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
W20260129 02:58:45.136358 20479 consensus_peers.cc:597] T 2ac659d732314e1288849f08fe5e074e P 974b802141bd4f13beffef2aa03f88f9 -> Peer 197b880fa39d432a9b472845876bfaf0 (127.18.211.1:42027): Couldn't send request to peer 197b880fa39d432a9b472845876bfaf0. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20260129 02:58:45.136673 20771 leader_election.cc:290] T add8d4923d1a489e86882c3a25692b95 P 1c50de89a5a845668696dcb28c3badd6 [CANDIDATE]: Term 1 election: Requested vote from peers 197b880fa39d432a9b472845876bfaf0 (127.18.211.1:42027), 974b802141bd4f13beffef2aa03f88f9 (127.18.211.2:34517)
I20260129 02:58:45.137739 20515 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "add8d4923d1a489e86882c3a25692b95" candidate_uuid: "1c50de89a5a845668696dcb28c3badd6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "974b802141bd4f13beffef2aa03f88f9"
I20260129 02:58:45.138304 20515 raft_consensus.cc:3060] T add8d4923d1a489e86882c3a25692b95 P 974b802141bd4f13beffef2aa03f88f9 [term 0 FOLLOWER]: Advancing to term 1
I20260129 02:58:45.138980 20630 ts_tablet_manager.cc:1434] T dc2f4cb752cf4d59b5f6bc689f5d8104 P 974b802141bd4f13beffef2aa03f88f9: Time spent starting tablet: real 0.035s user 0.007s sys 0.000s
I20260129 02:58:45.136299 20441 raft_consensus.cc:2468] T add8d4923d1a489e86882c3a25692b95 P 197b880fa39d432a9b472845876bfaf0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1c50de89a5a845668696dcb28c3badd6 in term 1.
I20260129 02:58:45.141825 20553 leader_election.cc:304] T add8d4923d1a489e86882c3a25692b95 P 1c50de89a5a845668696dcb28c3badd6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 197b880fa39d432a9b472845876bfaf0, 1c50de89a5a845668696dcb28c3badd6; no voters:
I20260129 02:58:45.142628 20771 raft_consensus.cc:2804] T add8d4923d1a489e86882c3a25692b95 P 1c50de89a5a845668696dcb28c3badd6 [term 1 FOLLOWER]: Leader election won for term 1
I20260129 02:58:45.143080 20771 raft_consensus.cc:697] T add8d4923d1a489e86882c3a25692b95 P 1c50de89a5a845668696dcb28c3badd6 [term 1 LEADER]: Becoming Leader. State: Replica: 1c50de89a5a845668696dcb28c3badd6, State: Running, Role: LEADER
I20260129 02:58:45.143852 20771 consensus_queue.cc:237] T add8d4923d1a489e86882c3a25692b95 P 1c50de89a5a845668696dcb28c3badd6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:58:45.150386 20631 tablet_bootstrap.cc:492] T 4f8f3a38f6e24ba4857875181f447442 P 1c50de89a5a845668696dcb28c3badd6: No bootstrap required, opened a new log
I20260129 02:58:45.150880 20631 ts_tablet_manager.cc:1403] T 4f8f3a38f6e24ba4857875181f447442 P 1c50de89a5a845668696dcb28c3badd6: Time spent bootstrapping tablet: real 0.083s user 0.015s sys 0.001s
I20260129 02:58:45.153451 20631 raft_consensus.cc:359] T 4f8f3a38f6e24ba4857875181f447442 P 1c50de89a5a845668696dcb28c3badd6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:58:45.154134 20631 raft_consensus.cc:385] T 4f8f3a38f6e24ba4857875181f447442 P 1c50de89a5a845668696dcb28c3badd6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260129 02:58:45.154394 20631 raft_consensus.cc:740] T 4f8f3a38f6e24ba4857875181f447442 P 1c50de89a5a845668696dcb28c3badd6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1c50de89a5a845668696dcb28c3badd6, State: Initialized, Role: FOLLOWER
I20260129 02:58:45.154966 20631 consensus_queue.cc:260] T 4f8f3a38f6e24ba4857875181f447442 P 1c50de89a5a845668696dcb28c3badd6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:58:45.158025 20630 tablet_bootstrap.cc:492] T 76f713c3331540d2a5b76164103e9eb1 P 974b802141bd4f13beffef2aa03f88f9: Bootstrap starting.
I20260129 02:58:45.158854 20729 raft_consensus.cc:493] T fcddab6bca684adf8ea032289a00f90e P 197b880fa39d432a9b472845876bfaf0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260129 02:58:45.159468 20729 raft_consensus.cc:515] T fcddab6bca684adf8ea032289a00f90e P 197b880fa39d432a9b472845876bfaf0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:58:45.162128 20588 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4e420f90884f4796997b5ea41e49e411" candidate_uuid: "197b880fa39d432a9b472845876bfaf0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c50de89a5a845668696dcb28c3badd6"
I20260129 02:58:45.162500 20515 raft_consensus.cc:2468] T add8d4923d1a489e86882c3a25692b95 P 974b802141bd4f13beffef2aa03f88f9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1c50de89a5a845668696dcb28c3badd6 in term 1.
I20260129 02:58:45.163355 20658 leader_election.cc:290] T 4e420f90884f4796997b5ea41e49e411 P 197b880fa39d432a9b472845876bfaf0 [CANDIDATE]: Term 1 election: Requested vote from peers 974b802141bd4f13beffef2aa03f88f9 (127.18.211.2:34517), 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3:37735)
I20260129 02:58:45.163847 20631 ts_tablet_manager.cc:1434] T 4f8f3a38f6e24ba4857875181f447442 P 1c50de89a5a845668696dcb28c3badd6: Time spent starting tablet: real 0.013s user 0.007s sys 0.000s
I20260129 02:58:45.164857 20631 tablet_bootstrap.cc:492] T 4e420f90884f4796997b5ea41e49e411 P 1c50de89a5a845668696dcb28c3badd6: Bootstrap starting.
W20260129 02:58:45.166116 20405 leader_election.cc:343] T 4e420f90884f4796997b5ea41e49e411 P 197b880fa39d432a9b472845876bfaf0 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3:37735): Illegal state: must be running to vote when last-logged opid is not known
I20260129 02:58:45.168308 20516 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fcddab6bca684adf8ea032289a00f90e" candidate_uuid: "197b880fa39d432a9b472845876bfaf0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "974b802141bd4f13beffef2aa03f88f9" is_pre_election: true
I20260129 02:58:45.168893 20729 leader_election.cc:290] T fcddab6bca684adf8ea032289a00f90e P 197b880fa39d432a9b472845876bfaf0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 974b802141bd4f13beffef2aa03f88f9 (127.18.211.2:34517), 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3:37735)
I20260129 02:58:45.168956 20516 raft_consensus.cc:2468] T fcddab6bca684adf8ea032289a00f90e P 974b802141bd4f13beffef2aa03f88f9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 197b880fa39d432a9b472845876bfaf0 in term 0.
I20260129 02:58:45.169471 20729 raft_consensus.cc:493] T a609ac02fae84725ab743f95f51d33ce P 197b880fa39d432a9b472845876bfaf0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260129 02:58:45.169876 20729 raft_consensus.cc:515] T a609ac02fae84725ab743f95f51d33ce P 197b880fa39d432a9b472845876bfaf0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:58:45.172371 20631 tablet_bootstrap.cc:654] T 4e420f90884f4796997b5ea41e49e411 P 1c50de89a5a845668696dcb28c3badd6: Neither blocks nor log segments found. Creating new log.
I20260129 02:58:45.172717 20515 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4e420f90884f4796997b5ea41e49e411" candidate_uuid: "197b880fa39d432a9b472845876bfaf0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "974b802141bd4f13beffef2aa03f88f9"
I20260129 02:58:45.172899 20588 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fcddab6bca684adf8ea032289a00f90e" candidate_uuid: "197b880fa39d432a9b472845876bfaf0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c50de89a5a845668696dcb28c3badd6" is_pre_election: true
I20260129 02:58:45.173281 20515 raft_consensus.cc:3060] T 4e420f90884f4796997b5ea41e49e411 P 974b802141bd4f13beffef2aa03f88f9 [term 0 FOLLOWER]: Advancing to term 1
I20260129 02:58:45.173465 20588 raft_consensus.cc:2468] T fcddab6bca684adf8ea032289a00f90e P 1c50de89a5a845668696dcb28c3badd6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 197b880fa39d432a9b472845876bfaf0 in term 0.
I20260129 02:58:45.174582 20407 leader_election.cc:304] T fcddab6bca684adf8ea032289a00f90e P 197b880fa39d432a9b472845876bfaf0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 197b880fa39d432a9b472845876bfaf0, 974b802141bd4f13beffef2aa03f88f9; no voters:
I20260129 02:58:45.175079 20516 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a609ac02fae84725ab743f95f51d33ce" candidate_uuid: "197b880fa39d432a9b472845876bfaf0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "974b802141bd4f13beffef2aa03f88f9" is_pre_election: true
I20260129 02:58:45.175674 20516 raft_consensus.cc:2468] T a609ac02fae84725ab743f95f51d33ce P 974b802141bd4f13beffef2aa03f88f9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 197b880fa39d432a9b472845876bfaf0 in term 0.
I20260129 02:58:45.175724 20658 raft_consensus.cc:2804] T fcddab6bca684adf8ea032289a00f90e P 197b880fa39d432a9b472845876bfaf0 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260129 02:58:45.176203 20658 raft_consensus.cc:493] T fcddab6bca684adf8ea032289a00f90e P 197b880fa39d432a9b472845876bfaf0 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260129 02:58:45.176550 20658 raft_consensus.cc:3060] T fcddab6bca684adf8ea032289a00f90e P 197b880fa39d432a9b472845876bfaf0 [term 0 FOLLOWER]: Advancing to term 1
I20260129 02:58:45.176784 20633 tablet_bootstrap.cc:492] T febffe4a8b394368b69b1d164308fb9a P 197b880fa39d432a9b472845876bfaf0: No bootstrap required, opened a new log
I20260129 02:58:45.177198 20633 ts_tablet_manager.cc:1403] T febffe4a8b394368b69b1d164308fb9a P 197b880fa39d432a9b472845876bfaf0: Time spent bootstrapping tablet: real 0.073s user 0.034s sys 0.026s
I20260129 02:58:45.179422 20407 leader_election.cc:304] T a609ac02fae84725ab743f95f51d33ce P 197b880fa39d432a9b472845876bfaf0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 197b880fa39d432a9b472845876bfaf0, 974b802141bd4f13beffef2aa03f88f9; no voters:
I20260129 02:58:45.182895 20630 tablet_bootstrap.cc:654] T 76f713c3331540d2a5b76164103e9eb1 P 974b802141bd4f13beffef2aa03f88f9: Neither blocks nor log segments found. Creating new log.
I20260129 02:58:45.183293 20721 raft_consensus.cc:2804] T a609ac02fae84725ab743f95f51d33ce P 197b880fa39d432a9b472845876bfaf0 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260129 02:58:45.172402 20729 leader_election.cc:290] T a609ac02fae84725ab743f95f51d33ce P 197b880fa39d432a9b472845876bfaf0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 974b802141bd4f13beffef2aa03f88f9 (127.18.211.2:34517), 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3:37735)
I20260129 02:58:45.180272 20588 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a609ac02fae84725ab743f95f51d33ce" candidate_uuid: "197b880fa39d432a9b472845876bfaf0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c50de89a5a845668696dcb28c3badd6" is_pre_election: true
I20260129 02:58:45.184180 20588 raft_consensus.cc:2468] T a609ac02fae84725ab743f95f51d33ce P 1c50de89a5a845668696dcb28c3badd6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 197b880fa39d432a9b472845876bfaf0 in term 0.
I20260129 02:58:45.183686 20721 raft_consensus.cc:493] T a609ac02fae84725ab743f95f51d33ce P 197b880fa39d432a9b472845876bfaf0 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260129 02:58:45.184940 20721 raft_consensus.cc:3060] T a609ac02fae84725ab743f95f51d33ce P 197b880fa39d432a9b472845876bfaf0 [term 0 FOLLOWER]: Advancing to term 1
I20260129 02:58:45.191238 20721 raft_consensus.cc:515] T a609ac02fae84725ab743f95f51d33ce P 197b880fa39d432a9b472845876bfaf0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:58:45.192140 20734 consensus_queue.cc:1048] T 2ac659d732314e1288849f08fe5e074e P 974b802141bd4f13beffef2aa03f88f9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260129 02:58:45.187445 20658 raft_consensus.cc:515] T fcddab6bca684adf8ea032289a00f90e P 197b880fa39d432a9b472845876bfaf0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:58:45.195485 20337 catalog_manager.cc:5654] T add8d4923d1a489e86882c3a25692b95 P 1c50de89a5a845668696dcb28c3badd6 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3). New cstate: current_term: 1 leader_uuid: "1c50de89a5a845668696dcb28c3badd6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } health_report { overall_health: HEALTHY } } }
I20260129 02:58:45.197660 20721 leader_election.cc:290] T a609ac02fae84725ab743f95f51d33ce P 197b880fa39d432a9b472845876bfaf0 [CANDIDATE]: Term 1 election: Requested vote from peers 974b802141bd4f13beffef2aa03f88f9 (127.18.211.2:34517), 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3:37735)
I20260129 02:58:45.179956 20633 raft_consensus.cc:359] T febffe4a8b394368b69b1d164308fb9a P 197b880fa39d432a9b472845876bfaf0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:58:45.197618 20516 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fcddab6bca684adf8ea032289a00f90e" candidate_uuid: "197b880fa39d432a9b472845876bfaf0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "974b802141bd4f13beffef2aa03f88f9"
I20260129 02:58:45.198343 20633 raft_consensus.cc:385] T febffe4a8b394368b69b1d164308fb9a P 197b880fa39d432a9b472845876bfaf0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260129 02:58:45.198491 20590 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fcddab6bca684adf8ea032289a00f90e" candidate_uuid: "197b880fa39d432a9b472845876bfaf0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c50de89a5a845668696dcb28c3badd6"
I20260129 02:58:45.198616 20516 raft_consensus.cc:3060] T fcddab6bca684adf8ea032289a00f90e P 974b802141bd4f13beffef2aa03f88f9 [term 0 FOLLOWER]: Advancing to term 1
I20260129 02:58:45.199118 20590 raft_consensus.cc:3060] T fcddab6bca684adf8ea032289a00f90e P 1c50de89a5a845668696dcb28c3badd6 [term 0 FOLLOWER]: Advancing to term 1
I20260129 02:58:45.198822 20633 raft_consensus.cc:740] T febffe4a8b394368b69b1d164308fb9a P 197b880fa39d432a9b472845876bfaf0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 197b880fa39d432a9b472845876bfaf0, State: Initialized, Role: FOLLOWER
I20260129 02:58:45.204698 20633 consensus_queue.cc:260] T febffe4a8b394368b69b1d164308fb9a P 197b880fa39d432a9b472845876bfaf0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:58:45.205180 20515 raft_consensus.cc:2468] T 4e420f90884f4796997b5ea41e49e411 P 974b802141bd4f13beffef2aa03f88f9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 197b880fa39d432a9b472845876bfaf0 in term 1.
I20260129 02:58:45.205483 20590 raft_consensus.cc:2468] T fcddab6bca684adf8ea032289a00f90e P 1c50de89a5a845668696dcb28c3badd6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 197b880fa39d432a9b472845876bfaf0 in term 1.
I20260129 02:58:45.207116 20405 leader_election.cc:304] T fcddab6bca684adf8ea032289a00f90e P 197b880fa39d432a9b472845876bfaf0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 197b880fa39d432a9b472845876bfaf0, 1c50de89a5a845668696dcb28c3badd6; no voters:
I20260129 02:58:45.207939 20729 raft_consensus.cc:2804] T fcddab6bca684adf8ea032289a00f90e P 197b880fa39d432a9b472845876bfaf0 [term 1 FOLLOWER]: Leader election won for term 1
I20260129 02:58:45.208374 20729 raft_consensus.cc:697] T fcddab6bca684adf8ea032289a00f90e P 197b880fa39d432a9b472845876bfaf0 [term 1 LEADER]: Becoming Leader. State: Replica: 197b880fa39d432a9b472845876bfaf0, State: Running, Role: LEADER
I20260129 02:58:45.209129 20729 consensus_queue.cc:237] T fcddab6bca684adf8ea032289a00f90e P 197b880fa39d432a9b472845876bfaf0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:58:45.213609 20407 leader_election.cc:304] T 4e420f90884f4796997b5ea41e49e411 P 197b880fa39d432a9b472845876bfaf0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 197b880fa39d432a9b472845876bfaf0, 974b802141bd4f13beffef2aa03f88f9; no voters: 1c50de89a5a845668696dcb28c3badd6
I20260129 02:58:45.214377 20515 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a609ac02fae84725ab743f95f51d33ce" candidate_uuid: "197b880fa39d432a9b472845876bfaf0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "974b802141bd4f13beffef2aa03f88f9"
I20260129 02:58:45.214406 20721 raft_consensus.cc:2804] T 4e420f90884f4796997b5ea41e49e411 P 197b880fa39d432a9b472845876bfaf0 [term 1 FOLLOWER]: Leader election won for term 1
I20260129 02:58:45.215713 20590 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a609ac02fae84725ab743f95f51d33ce" candidate_uuid: "197b880fa39d432a9b472845876bfaf0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c50de89a5a845668696dcb28c3badd6"
I20260129 02:58:45.216168 20515 raft_consensus.cc:3060] T a609ac02fae84725ab743f95f51d33ce P 974b802141bd4f13beffef2aa03f88f9 [term 0 FOLLOWER]: Advancing to term 1
I20260129 02:58:45.216379 20721 raft_consensus.cc:697] T 4e420f90884f4796997b5ea41e49e411 P 197b880fa39d432a9b472845876bfaf0 [term 1 LEADER]: Becoming Leader. State: Replica: 197b880fa39d432a9b472845876bfaf0, State: Running, Role: LEADER
I20260129 02:58:45.216320 20590 raft_consensus.cc:3060] T a609ac02fae84725ab743f95f51d33ce P 1c50de89a5a845668696dcb28c3badd6 [term 0 FOLLOWER]: Advancing to term 1
I20260129 02:58:45.217635 20721 consensus_queue.cc:237] T 4e420f90884f4796997b5ea41e49e411 P 197b880fa39d432a9b472845876bfaf0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:58:45.220220 20658 leader_election.cc:290] T fcddab6bca684adf8ea032289a00f90e P 197b880fa39d432a9b472845876bfaf0 [CANDIDATE]: Term 1 election: Requested vote from peers 974b802141bd4f13beffef2aa03f88f9 (127.18.211.2:34517), 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3:37735)
I20260129 02:58:45.207191 20516 raft_consensus.cc:2468] T fcddab6bca684adf8ea032289a00f90e P 974b802141bd4f13beffef2aa03f88f9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 197b880fa39d432a9b472845876bfaf0 in term 1.
I20260129 02:58:45.215823 20633 ts_tablet_manager.cc:1434] T febffe4a8b394368b69b1d164308fb9a P 197b880fa39d432a9b472845876bfaf0: Time spent starting tablet: real 0.038s user 0.004s sys 0.003s
I20260129 02:58:45.224594 20633 tablet_bootstrap.cc:492] T b189f08e16a74a11980cc55d99359a0a P 197b880fa39d432a9b472845876bfaf0: Bootstrap starting.
I20260129 02:58:45.228761 20739 raft_consensus.cc:493] T dc2f4cb752cf4d59b5f6bc689f5d8104 P 974b802141bd4f13beffef2aa03f88f9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260129 02:58:45.229342 20739 raft_consensus.cc:515] T dc2f4cb752cf4d59b5f6bc689f5d8104 P 974b802141bd4f13beffef2aa03f88f9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:58:45.236968 20441 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "dc2f4cb752cf4d59b5f6bc689f5d8104" candidate_uuid: "974b802141bd4f13beffef2aa03f88f9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "197b880fa39d432a9b472845876bfaf0" is_pre_election: true
I20260129 02:58:45.237092 20739 leader_election.cc:290] T dc2f4cb752cf4d59b5f6bc689f5d8104 P 974b802141bd4f13beffef2aa03f88f9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 197b880fa39d432a9b472845876bfaf0 (127.18.211.1:42027), 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3:37735)
W20260129 02:58:45.245447 20479 leader_election.cc:343] T dc2f4cb752cf4d59b5f6bc689f5d8104 P 974b802141bd4f13beffef2aa03f88f9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 197b880fa39d432a9b472845876bfaf0 (127.18.211.1:42027): Illegal state: must be running to vote when last-logged opid is not known
I20260129 02:58:45.245741 20590 raft_consensus.cc:2468] T a609ac02fae84725ab743f95f51d33ce P 1c50de89a5a845668696dcb28c3badd6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 197b880fa39d432a9b472845876bfaf0 in term 1.
I20260129 02:58:45.245993 20589 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "dc2f4cb752cf4d59b5f6bc689f5d8104" candidate_uuid: "974b802141bd4f13beffef2aa03f88f9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c50de89a5a845668696dcb28c3badd6" is_pre_election: true
I20260129 02:58:45.248538 20405 leader_election.cc:304] T a609ac02fae84725ab743f95f51d33ce P 197b880fa39d432a9b472845876bfaf0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 197b880fa39d432a9b472845876bfaf0, 1c50de89a5a845668696dcb28c3badd6; no voters:
I20260129 02:58:45.249344 20721 raft_consensus.cc:2804] T a609ac02fae84725ab743f95f51d33ce P 197b880fa39d432a9b472845876bfaf0 [term 1 FOLLOWER]: Leader election won for term 1
I20260129 02:58:45.249976 20721 raft_consensus.cc:697] T a609ac02fae84725ab743f95f51d33ce P 197b880fa39d432a9b472845876bfaf0 [term 1 LEADER]: Becoming Leader. State: Replica: 197b880fa39d432a9b472845876bfaf0, State: Running, Role: LEADER
I20260129 02:58:45.250519 20515 raft_consensus.cc:2468] T a609ac02fae84725ab743f95f51d33ce P 974b802141bd4f13beffef2aa03f88f9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 197b880fa39d432a9b472845876bfaf0 in term 1.
I20260129 02:58:45.250741 20721 consensus_queue.cc:237] T a609ac02fae84725ab743f95f51d33ce P 197b880fa39d432a9b472845876bfaf0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
W20260129 02:58:45.253185 20480 leader_election.cc:343] T dc2f4cb752cf4d59b5f6bc689f5d8104 P 974b802141bd4f13beffef2aa03f88f9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3:37735): Illegal state: must be running to vote when last-logged opid is not known
I20260129 02:58:45.254318 20480 leader_election.cc:304] T dc2f4cb752cf4d59b5f6bc689f5d8104 P 974b802141bd4f13beffef2aa03f88f9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 974b802141bd4f13beffef2aa03f88f9; no voters: 197b880fa39d432a9b472845876bfaf0, 1c50de89a5a845668696dcb28c3badd6
I20260129 02:58:45.255103 20739 raft_consensus.cc:2749] T dc2f4cb752cf4d59b5f6bc689f5d8104 P 974b802141bd4f13beffef2aa03f88f9 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260129 02:58:45.258972 20631 tablet_bootstrap.cc:492] T 4e420f90884f4796997b5ea41e49e411 P 1c50de89a5a845668696dcb28c3badd6: No bootstrap required, opened a new log
I20260129 02:58:45.259436 20631 ts_tablet_manager.cc:1403] T 4e420f90884f4796997b5ea41e49e411 P 1c50de89a5a845668696dcb28c3badd6: Time spent bootstrapping tablet: real 0.095s user 0.022s sys 0.017s
I20260129 02:58:45.259495 20633 tablet_bootstrap.cc:654] T b189f08e16a74a11980cc55d99359a0a P 197b880fa39d432a9b472845876bfaf0: Neither blocks nor log segments found. Creating new log.
I20260129 02:58:45.262584 20337 catalog_manager.cc:5654] T fcddab6bca684adf8ea032289a00f90e P 197b880fa39d432a9b472845876bfaf0 reported cstate change: term changed from 0 to 1, leader changed from <none> to 197b880fa39d432a9b472845876bfaf0 (127.18.211.1). New cstate: current_term: 1 leader_uuid: "197b880fa39d432a9b472845876bfaf0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } health_report { overall_health: UNKNOWN } } }
I20260129 02:58:45.263849 20631 raft_consensus.cc:359] T 4e420f90884f4796997b5ea41e49e411 P 1c50de89a5a845668696dcb28c3badd6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:58:45.264515 20631 raft_consensus.cc:385] T 4e420f90884f4796997b5ea41e49e411 P 1c50de89a5a845668696dcb28c3badd6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260129 02:58:45.264778 20631 raft_consensus.cc:740] T 4e420f90884f4796997b5ea41e49e411 P 1c50de89a5a845668696dcb28c3badd6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1c50de89a5a845668696dcb28c3badd6, State: Initialized, Role: FOLLOWER
I20260129 02:58:45.264240 20337 catalog_manager.cc:5654] T 4e420f90884f4796997b5ea41e49e411 P 197b880fa39d432a9b472845876bfaf0 reported cstate change: term changed from 0 to 1, leader changed from <none> to 197b880fa39d432a9b472845876bfaf0 (127.18.211.1). New cstate: current_term: 1 leader_uuid: "197b880fa39d432a9b472845876bfaf0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } health_report { overall_health: UNKNOWN } } }
I20260129 02:58:45.265443 20631 consensus_queue.cc:260] T 4e420f90884f4796997b5ea41e49e411 P 1c50de89a5a845668696dcb28c3badd6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:58:45.267079 20631 ts_tablet_manager.cc:1434] T 4e420f90884f4796997b5ea41e49e411 P 1c50de89a5a845668696dcb28c3badd6: Time spent starting tablet: real 0.007s user 0.005s sys 0.000s
I20260129 02:58:45.271454 20630 tablet_bootstrap.cc:492] T 76f713c3331540d2a5b76164103e9eb1 P 974b802141bd4f13beffef2aa03f88f9: No bootstrap required, opened a new log
I20260129 02:58:45.272033 20631 tablet_bootstrap.cc:492] T 913fc7d751ed4b7388d889aac30a5f9b P 1c50de89a5a845668696dcb28c3badd6: Bootstrap starting.
I20260129 02:58:45.272172 20630 ts_tablet_manager.cc:1403] T 76f713c3331540d2a5b76164103e9eb1 P 974b802141bd4f13beffef2aa03f88f9: Time spent bootstrapping tablet: real 0.114s user 0.011s sys 0.000s
I20260129 02:58:45.278638 20630 raft_consensus.cc:359] T 76f713c3331540d2a5b76164103e9eb1 P 974b802141bd4f13beffef2aa03f88f9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:58:45.279289 20630 raft_consensus.cc:385] T 76f713c3331540d2a5b76164103e9eb1 P 974b802141bd4f13beffef2aa03f88f9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260129 02:58:45.279551 20630 raft_consensus.cc:740] T 76f713c3331540d2a5b76164103e9eb1 P 974b802141bd4f13beffef2aa03f88f9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 974b802141bd4f13beffef2aa03f88f9, State: Initialized, Role: FOLLOWER
I20260129 02:58:45.280246 20630 consensus_queue.cc:260] T 76f713c3331540d2a5b76164103e9eb1 P 974b802141bd4f13beffef2aa03f88f9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:58:45.285595 20631 tablet_bootstrap.cc:654] T 913fc7d751ed4b7388d889aac30a5f9b P 1c50de89a5a845668696dcb28c3badd6: Neither blocks nor log segments found. Creating new log.
I20260129 02:58:45.285749 20630 ts_tablet_manager.cc:1434] T 76f713c3331540d2a5b76164103e9eb1 P 974b802141bd4f13beffef2aa03f88f9: Time spent starting tablet: real 0.013s user 0.006s sys 0.002s
I20260129 02:58:45.286664 20630 tablet_bootstrap.cc:492] T 368953c9102643b2a21f99af36b51905 P 974b802141bd4f13beffef2aa03f88f9: Bootstrap starting.
I20260129 02:58:45.291700 20633 tablet_bootstrap.cc:492] T b189f08e16a74a11980cc55d99359a0a P 197b880fa39d432a9b472845876bfaf0: No bootstrap required, opened a new log
I20260129 02:58:45.292270 20630 tablet_bootstrap.cc:654] T 368953c9102643b2a21f99af36b51905 P 974b802141bd4f13beffef2aa03f88f9: Neither blocks nor log segments found. Creating new log.
I20260129 02:58:45.292335 20633 ts_tablet_manager.cc:1403] T b189f08e16a74a11980cc55d99359a0a P 197b880fa39d432a9b472845876bfaf0: Time spent bootstrapping tablet: real 0.068s user 0.011s sys 0.003s
I20260129 02:58:45.295701 20633 raft_consensus.cc:359] T b189f08e16a74a11980cc55d99359a0a P 197b880fa39d432a9b472845876bfaf0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:58:45.296599 20633 raft_consensus.cc:385] T b189f08e16a74a11980cc55d99359a0a P 197b880fa39d432a9b472845876bfaf0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260129 02:58:45.296878 20633 raft_consensus.cc:740] T b189f08e16a74a11980cc55d99359a0a P 197b880fa39d432a9b472845876bfaf0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 197b880fa39d432a9b472845876bfaf0, State: Initialized, Role: FOLLOWER
I20260129 02:58:45.297572 20633 consensus_queue.cc:260] T b189f08e16a74a11980cc55d99359a0a P 197b880fa39d432a9b472845876bfaf0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:58:45.299409 20631 tablet_bootstrap.cc:492] T 913fc7d751ed4b7388d889aac30a5f9b P 1c50de89a5a845668696dcb28c3badd6: No bootstrap required, opened a new log
I20260129 02:58:45.300974 20631 ts_tablet_manager.cc:1403] T 913fc7d751ed4b7388d889aac30a5f9b P 1c50de89a5a845668696dcb28c3badd6: Time spent bootstrapping tablet: real 0.029s user 0.010s sys 0.006s
I20260129 02:58:45.302129 20633 ts_tablet_manager.cc:1434] T b189f08e16a74a11980cc55d99359a0a P 197b880fa39d432a9b472845876bfaf0: Time spent starting tablet: real 0.008s user 0.007s sys 0.000s
I20260129 02:58:45.303087 20633 tablet_bootstrap.cc:492] T 2ac659d732314e1288849f08fe5e074e P 197b880fa39d432a9b472845876bfaf0: Bootstrap starting.
I20260129 02:58:45.306026 20630 tablet_bootstrap.cc:492] T 368953c9102643b2a21f99af36b51905 P 974b802141bd4f13beffef2aa03f88f9: No bootstrap required, opened a new log
I20260129 02:58:45.304209 20631 raft_consensus.cc:359] T 913fc7d751ed4b7388d889aac30a5f9b P 1c50de89a5a845668696dcb28c3badd6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:58:45.306478 20630 ts_tablet_manager.cc:1403] T 368953c9102643b2a21f99af36b51905 P 974b802141bd4f13beffef2aa03f88f9: Time spent bootstrapping tablet: real 0.020s user 0.006s sys 0.007s
I20260129 02:58:45.306452 20631 raft_consensus.cc:385] T 913fc7d751ed4b7388d889aac30a5f9b P 1c50de89a5a845668696dcb28c3badd6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260129 02:58:45.308239 20631 raft_consensus.cc:740] T 913fc7d751ed4b7388d889aac30a5f9b P 1c50de89a5a845668696dcb28c3badd6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1c50de89a5a845668696dcb28c3badd6, State: Initialized, Role: FOLLOWER
I20260129 02:58:45.308790 20630 raft_consensus.cc:359] T 368953c9102643b2a21f99af36b51905 P 974b802141bd4f13beffef2aa03f88f9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:58:45.308881 20631 consensus_queue.cc:260] T 913fc7d751ed4b7388d889aac30a5f9b P 1c50de89a5a845668696dcb28c3badd6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:58:45.309473 20630 raft_consensus.cc:385] T 368953c9102643b2a21f99af36b51905 P 974b802141bd4f13beffef2aa03f88f9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260129 02:58:45.309779 20630 raft_consensus.cc:740] T 368953c9102643b2a21f99af36b51905 P 974b802141bd4f13beffef2aa03f88f9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 974b802141bd4f13beffef2aa03f88f9, State: Initialized, Role: FOLLOWER
I20260129 02:58:45.310448 20630 consensus_queue.cc:260] T 368953c9102643b2a21f99af36b51905 P 974b802141bd4f13beffef2aa03f88f9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:58:45.311511 20631 ts_tablet_manager.cc:1434] T 913fc7d751ed4b7388d889aac30a5f9b P 1c50de89a5a845668696dcb28c3badd6: Time spent starting tablet: real 0.010s user 0.005s sys 0.001s
I20260129 02:58:45.312484 20631 tablet_bootstrap.cc:492] T febffe4a8b394368b69b1d164308fb9a P 1c50de89a5a845668696dcb28c3badd6: Bootstrap starting.
I20260129 02:58:45.312762 20630 ts_tablet_manager.cc:1434] T 368953c9102643b2a21f99af36b51905 P 974b802141bd4f13beffef2aa03f88f9: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20260129 02:58:45.312800 20633 tablet_bootstrap.cc:654] T 2ac659d732314e1288849f08fe5e074e P 197b880fa39d432a9b472845876bfaf0: Neither blocks nor log segments found. Creating new log.
I20260129 02:58:45.313751 20630 tablet_bootstrap.cc:492] T 75b0b6e82ddc4f92b1fe1d94f9a0b73b P 974b802141bd4f13beffef2aa03f88f9: Bootstrap starting.
W20260129 02:58:45.315639 20405 consensus_peers.cc:597] T e4d18c35b0a7493e85d0a4b92956a48f P 197b880fa39d432a9b472845876bfaf0 -> Peer 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3:37735): Couldn't send request to peer 1c50de89a5a845668696dcb28c3badd6. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20260129 02:58:45.319507 20630 tablet_bootstrap.cc:654] T 75b0b6e82ddc4f92b1fe1d94f9a0b73b P 974b802141bd4f13beffef2aa03f88f9: Neither blocks nor log segments found. Creating new log.
I20260129 02:58:45.319644 20631 tablet_bootstrap.cc:654] T febffe4a8b394368b69b1d164308fb9a P 1c50de89a5a845668696dcb28c3badd6: Neither blocks nor log segments found. Creating new log.
I20260129 02:58:45.321349 20338 catalog_manager.cc:5654] T a609ac02fae84725ab743f95f51d33ce P 197b880fa39d432a9b472845876bfaf0 reported cstate change: term changed from 0 to 1, leader changed from <none> to 197b880fa39d432a9b472845876bfaf0 (127.18.211.1). New cstate: current_term: 1 leader_uuid: "197b880fa39d432a9b472845876bfaf0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } health_report { overall_health: UNKNOWN } } }
I20260129 02:58:45.326246 20633 tablet_bootstrap.cc:492] T 2ac659d732314e1288849f08fe5e074e P 197b880fa39d432a9b472845876bfaf0: No bootstrap required, opened a new log
I20260129 02:58:45.326715 20633 ts_tablet_manager.cc:1403] T 2ac659d732314e1288849f08fe5e074e P 197b880fa39d432a9b472845876bfaf0: Time spent bootstrapping tablet: real 0.024s user 0.011s sys 0.006s
I20260129 02:58:45.327353 20630 tablet_bootstrap.cc:492] T 75b0b6e82ddc4f92b1fe1d94f9a0b73b P 974b802141bd4f13beffef2aa03f88f9: No bootstrap required, opened a new log
I20260129 02:58:45.327642 20631 tablet_bootstrap.cc:492] T febffe4a8b394368b69b1d164308fb9a P 1c50de89a5a845668696dcb28c3badd6: No bootstrap required, opened a new log
I20260129 02:58:45.327782 20630 ts_tablet_manager.cc:1403] T 75b0b6e82ddc4f92b1fe1d94f9a0b73b P 974b802141bd4f13beffef2aa03f88f9: Time spent bootstrapping tablet: real 0.014s user 0.008s sys 0.003s
I20260129 02:58:45.329214 20633 raft_consensus.cc:359] T 2ac659d732314e1288849f08fe5e074e P 197b880fa39d432a9b472845876bfaf0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:58:45.329833 20633 raft_consensus.cc:385] T 2ac659d732314e1288849f08fe5e074e P 197b880fa39d432a9b472845876bfaf0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260129 02:58:45.328197 20631 ts_tablet_manager.cc:1403] T febffe4a8b394368b69b1d164308fb9a P 1c50de89a5a845668696dcb28c3badd6: Time spent bootstrapping tablet: real 0.016s user 0.006s sys 0.006s
I20260129 02:58:45.330080 20633 raft_consensus.cc:740] T 2ac659d732314e1288849f08fe5e074e P 197b880fa39d432a9b472845876bfaf0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 197b880fa39d432a9b472845876bfaf0, State: Initialized, Role: FOLLOWER
I20260129 02:58:45.330183 20630 raft_consensus.cc:359] T 75b0b6e82ddc4f92b1fe1d94f9a0b73b P 974b802141bd4f13beffef2aa03f88f9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:58:45.330916 20630 raft_consensus.cc:385] T 75b0b6e82ddc4f92b1fe1d94f9a0b73b P 974b802141bd4f13beffef2aa03f88f9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260129 02:58:45.330814 20633 consensus_queue.cc:260] T 2ac659d732314e1288849f08fe5e074e P 197b880fa39d432a9b472845876bfaf0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:58:45.331228 20630 raft_consensus.cc:740] T 75b0b6e82ddc4f92b1fe1d94f9a0b73b P 974b802141bd4f13beffef2aa03f88f9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 974b802141bd4f13beffef2aa03f88f9, State: Initialized, Role: FOLLOWER
I20260129 02:58:45.332139 20630 consensus_queue.cc:260] T 75b0b6e82ddc4f92b1fe1d94f9a0b73b P 974b802141bd4f13beffef2aa03f88f9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:58:45.333295 20633 ts_tablet_manager.cc:1434] T 2ac659d732314e1288849f08fe5e074e P 197b880fa39d432a9b472845876bfaf0: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260129 02:58:45.333174 20631 raft_consensus.cc:359] T febffe4a8b394368b69b1d164308fb9a P 1c50de89a5a845668696dcb28c3badd6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:58:45.333863 20631 raft_consensus.cc:385] T febffe4a8b394368b69b1d164308fb9a P 1c50de89a5a845668696dcb28c3badd6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260129 02:58:45.334151 20631 raft_consensus.cc:740] T febffe4a8b394368b69b1d164308fb9a P 1c50de89a5a845668696dcb28c3badd6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1c50de89a5a845668696dcb28c3badd6, State: Initialized, Role: FOLLOWER
I20260129 02:58:45.335506 20721 raft_consensus.cc:493] T febffe4a8b394368b69b1d164308fb9a P 197b880fa39d432a9b472845876bfaf0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260129 02:58:45.336004 20721 raft_consensus.cc:515] T febffe4a8b394368b69b1d164308fb9a P 197b880fa39d432a9b472845876bfaf0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:58:45.338092 20721 leader_election.cc:290] T febffe4a8b394368b69b1d164308fb9a P 197b880fa39d432a9b472845876bfaf0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 974b802141bd4f13beffef2aa03f88f9 (127.18.211.2:34517), 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3:37735)
I20260129 02:58:45.334883 20631 consensus_queue.cc:260] T febffe4a8b394368b69b1d164308fb9a P 1c50de89a5a845668696dcb28c3badd6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:58:45.339385 20515 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "febffe4a8b394368b69b1d164308fb9a" candidate_uuid: "197b880fa39d432a9b472845876bfaf0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "974b802141bd4f13beffef2aa03f88f9" is_pre_election: true
I20260129 02:58:45.340054 20515 raft_consensus.cc:2468] T febffe4a8b394368b69b1d164308fb9a P 974b802141bd4f13beffef2aa03f88f9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 197b880fa39d432a9b472845876bfaf0 in term 0.
I20260129 02:58:45.341667 20630 ts_tablet_manager.cc:1434] T 75b0b6e82ddc4f92b1fe1d94f9a0b73b P 974b802141bd4f13beffef2aa03f88f9: Time spent starting tablet: real 0.014s user 0.004s sys 0.001s
I20260129 02:58:45.341845 20407 leader_election.cc:304] T febffe4a8b394368b69b1d164308fb9a P 197b880fa39d432a9b472845876bfaf0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 197b880fa39d432a9b472845876bfaf0, 974b802141bd4f13beffef2aa03f88f9; no voters:
I20260129 02:58:45.342582 20638 raft_consensus.cc:2804] T febffe4a8b394368b69b1d164308fb9a P 197b880fa39d432a9b472845876bfaf0 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260129 02:58:45.342697 20771 raft_consensus.cc:493] T fd8311b779af4e1fa1402b2a02a602f2 P 1c50de89a5a845668696dcb28c3badd6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260129 02:58:45.340269 20631 ts_tablet_manager.cc:1434] T febffe4a8b394368b69b1d164308fb9a P 1c50de89a5a845668696dcb28c3badd6: Time spent starting tablet: real 0.010s user 0.006s sys 0.000s
I20260129 02:58:45.343199 20771 raft_consensus.cc:515] T fd8311b779af4e1fa1402b2a02a602f2 P 1c50de89a5a845668696dcb28c3badd6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:58:45.334172 20633 tablet_bootstrap.cc:492] T dc2f4cb752cf4d59b5f6bc689f5d8104 P 197b880fa39d432a9b472845876bfaf0: Bootstrap starting.
I20260129 02:58:45.340754 20588 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "febffe4a8b394368b69b1d164308fb9a" candidate_uuid: "197b880fa39d432a9b472845876bfaf0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c50de89a5a845668696dcb28c3badd6" is_pre_election: true
I20260129 02:58:45.342950 20638 raft_consensus.cc:493] T febffe4a8b394368b69b1d164308fb9a P 197b880fa39d432a9b472845876bfaf0 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260129 02:58:45.344169 20638 raft_consensus.cc:3060] T febffe4a8b394368b69b1d164308fb9a P 197b880fa39d432a9b472845876bfaf0 [term 0 FOLLOWER]: Advancing to term 1
I20260129 02:58:45.344237 20588 raft_consensus.cc:2468] T febffe4a8b394368b69b1d164308fb9a P 1c50de89a5a845668696dcb28c3badd6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 197b880fa39d432a9b472845876bfaf0 in term 0.
I20260129 02:58:45.345202 20771 leader_election.cc:290] T fd8311b779af4e1fa1402b2a02a602f2 P 1c50de89a5a845668696dcb28c3badd6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 197b880fa39d432a9b472845876bfaf0 (127.18.211.1:42027), 974b802141bd4f13beffef2aa03f88f9 (127.18.211.2:34517)
I20260129 02:58:45.346063 20631 tablet_bootstrap.cc:492] T dc2f4cb752cf4d59b5f6bc689f5d8104 P 1c50de89a5a845668696dcb28c3badd6: Bootstrap starting.
I20260129 02:58:45.346298 20441 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fd8311b779af4e1fa1402b2a02a602f2" candidate_uuid: "1c50de89a5a845668696dcb28c3badd6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "197b880fa39d432a9b472845876bfaf0" is_pre_election: true
I20260129 02:58:45.349896 20515 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fd8311b779af4e1fa1402b2a02a602f2" candidate_uuid: "1c50de89a5a845668696dcb28c3badd6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "974b802141bd4f13beffef2aa03f88f9" is_pre_election: true
W20260129 02:58:45.350108 20553 leader_election.cc:343] T fd8311b779af4e1fa1402b2a02a602f2 P 1c50de89a5a845668696dcb28c3badd6 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 197b880fa39d432a9b472845876bfaf0 (127.18.211.1:42027): Illegal state: must be running to vote when last-logged opid is not known
I20260129 02:58:45.350526 20515 raft_consensus.cc:2468] T fd8311b779af4e1fa1402b2a02a602f2 P 974b802141bd4f13beffef2aa03f88f9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1c50de89a5a845668696dcb28c3badd6 in term 0.
I20260129 02:58:45.351647 20556 leader_election.cc:304] T fd8311b779af4e1fa1402b2a02a602f2 P 1c50de89a5a845668696dcb28c3badd6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1c50de89a5a845668696dcb28c3badd6, 974b802141bd4f13beffef2aa03f88f9; no voters: 197b880fa39d432a9b472845876bfaf0
I20260129 02:58:45.352696 20771 raft_consensus.cc:2804] T fd8311b779af4e1fa1402b2a02a602f2 P 1c50de89a5a845668696dcb28c3badd6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260129 02:58:45.353058 20771 raft_consensus.cc:493] T fd8311b779af4e1fa1402b2a02a602f2 P 1c50de89a5a845668696dcb28c3badd6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260129 02:58:45.353407 20771 raft_consensus.cc:3060] T fd8311b779af4e1fa1402b2a02a602f2 P 1c50de89a5a845668696dcb28c3badd6 [term 0 FOLLOWER]: Advancing to term 1
I20260129 02:58:45.357539 20633 tablet_bootstrap.cc:654] T dc2f4cb752cf4d59b5f6bc689f5d8104 P 197b880fa39d432a9b472845876bfaf0: Neither blocks nor log segments found. Creating new log.
I20260129 02:58:45.358771 20631 tablet_bootstrap.cc:654] T dc2f4cb752cf4d59b5f6bc689f5d8104 P 1c50de89a5a845668696dcb28c3badd6: Neither blocks nor log segments found. Creating new log.
I20260129 02:58:45.362250 20638 raft_consensus.cc:515] T febffe4a8b394368b69b1d164308fb9a P 197b880fa39d432a9b472845876bfaf0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:58:45.342945 20630 tablet_bootstrap.cc:492] T 4f8f3a38f6e24ba4857875181f447442 P 974b802141bd4f13beffef2aa03f88f9: Bootstrap starting.
I20260129 02:58:45.367163 20633 tablet_bootstrap.cc:492] T dc2f4cb752cf4d59b5f6bc689f5d8104 P 197b880fa39d432a9b472845876bfaf0: No bootstrap required, opened a new log
I20260129 02:58:45.366986 20771 raft_consensus.cc:515] T fd8311b779af4e1fa1402b2a02a602f2 P 1c50de89a5a845668696dcb28c3badd6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:58:45.370968 20631 tablet_bootstrap.cc:492] T dc2f4cb752cf4d59b5f6bc689f5d8104 P 1c50de89a5a845668696dcb28c3badd6: No bootstrap required, opened a new log
I20260129 02:58:45.367720 20633 ts_tablet_manager.cc:1403] T dc2f4cb752cf4d59b5f6bc689f5d8104 P 197b880fa39d432a9b472845876bfaf0: Time spent bootstrapping tablet: real 0.034s user 0.007s sys 0.007s
I20260129 02:58:45.371429 20631 ts_tablet_manager.cc:1403] T dc2f4cb752cf4d59b5f6bc689f5d8104 P 1c50de89a5a845668696dcb28c3badd6: Time spent bootstrapping tablet: real 0.028s user 0.013s sys 0.003s
I20260129 02:58:45.373648 20771 leader_election.cc:290] T fd8311b779af4e1fa1402b2a02a602f2 P 1c50de89a5a845668696dcb28c3badd6 [CANDIDATE]: Term 1 election: Requested vote from peers 197b880fa39d432a9b472845876bfaf0 (127.18.211.1:42027), 974b802141bd4f13beffef2aa03f88f9 (127.18.211.2:34517)
I20260129 02:58:45.371807 20630 tablet_bootstrap.cc:654] T 4f8f3a38f6e24ba4857875181f447442 P 974b802141bd4f13beffef2aa03f88f9: Neither blocks nor log segments found. Creating new log.
I20260129 02:58:45.374678 20633 raft_consensus.cc:359] T dc2f4cb752cf4d59b5f6bc689f5d8104 P 197b880fa39d432a9b472845876bfaf0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:58:45.373786 20441 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fd8311b779af4e1fa1402b2a02a602f2" candidate_uuid: "1c50de89a5a845668696dcb28c3badd6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "197b880fa39d432a9b472845876bfaf0"
I20260129 02:58:45.377669 20514 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "febffe4a8b394368b69b1d164308fb9a" candidate_uuid: "197b880fa39d432a9b472845876bfaf0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "974b802141bd4f13beffef2aa03f88f9"
I20260129 02:58:45.377933 20633 raft_consensus.cc:385] T dc2f4cb752cf4d59b5f6bc689f5d8104 P 197b880fa39d432a9b472845876bfaf0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260129 02:58:45.374183 20638 leader_election.cc:290] T febffe4a8b394368b69b1d164308fb9a P 197b880fa39d432a9b472845876bfaf0 [CANDIDATE]: Term 1 election: Requested vote from peers 974b802141bd4f13beffef2aa03f88f9 (127.18.211.2:34517), 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3:37735)
I20260129 02:58:45.378206 20514 raft_consensus.cc:3060] T febffe4a8b394368b69b1d164308fb9a P 974b802141bd4f13beffef2aa03f88f9 [term 0 FOLLOWER]: Advancing to term 1
I20260129 02:58:45.376688 20515 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fd8311b779af4e1fa1402b2a02a602f2" candidate_uuid: "1c50de89a5a845668696dcb28c3badd6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "974b802141bd4f13beffef2aa03f88f9"
I20260129 02:58:45.378381 20633 raft_consensus.cc:740] T dc2f4cb752cf4d59b5f6bc689f5d8104 P 197b880fa39d432a9b472845876bfaf0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 197b880fa39d432a9b472845876bfaf0, State: Initialized, Role: FOLLOWER
I20260129 02:58:45.378901 20515 raft_consensus.cc:3060] T fd8311b779af4e1fa1402b2a02a602f2 P 974b802141bd4f13beffef2aa03f88f9 [term 0 FOLLOWER]: Advancing to term 1
I20260129 02:58:45.379462 20633 consensus_queue.cc:260] T dc2f4cb752cf4d59b5f6bc689f5d8104 P 197b880fa39d432a9b472845876bfaf0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:58:45.378374 20771 raft_consensus.cc:493] T 4f8f3a38f6e24ba4857875181f447442 P 1c50de89a5a845668696dcb28c3badd6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260129 02:58:45.380545 20771 raft_consensus.cc:515] T 4f8f3a38f6e24ba4857875181f447442 P 1c50de89a5a845668696dcb28c3badd6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:58:45.382865 20633 ts_tablet_manager.cc:1434] T dc2f4cb752cf4d59b5f6bc689f5d8104 P 197b880fa39d432a9b472845876bfaf0: Time spent starting tablet: real 0.011s user 0.006s sys 0.001s
I20260129 02:58:45.383330 20771 leader_election.cc:290] T 4f8f3a38f6e24ba4857875181f447442 P 1c50de89a5a845668696dcb28c3badd6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 197b880fa39d432a9b472845876bfaf0 (127.18.211.1:42027), 974b802141bd4f13beffef2aa03f88f9 (127.18.211.2:34517)
I20260129 02:58:45.375687 20631 raft_consensus.cc:359] T dc2f4cb752cf4d59b5f6bc689f5d8104 P 1c50de89a5a845668696dcb28c3badd6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:58:45.383893 20631 raft_consensus.cc:385] T dc2f4cb752cf4d59b5f6bc689f5d8104 P 1c50de89a5a845668696dcb28c3badd6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
W20260129 02:58:45.384155 20553 leader_election.cc:343] T fd8311b779af4e1fa1402b2a02a602f2 P 1c50de89a5a845668696dcb28c3badd6 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 197b880fa39d432a9b472845876bfaf0 (127.18.211.1:42027): Illegal state: must be running to vote when last-logged opid is not known
I20260129 02:58:45.384166 20631 raft_consensus.cc:740] T dc2f4cb752cf4d59b5f6bc689f5d8104 P 1c50de89a5a845668696dcb28c3badd6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1c50de89a5a845668696dcb28c3badd6, State: Initialized, Role: FOLLOWER
I20260129 02:58:45.384555 20516 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4f8f3a38f6e24ba4857875181f447442" candidate_uuid: "1c50de89a5a845668696dcb28c3badd6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "974b802141bd4f13beffef2aa03f88f9" is_pre_election: true
I20260129 02:58:45.385111 20721 raft_consensus.cc:493] T 1ce4ddf5b10e4ae8b193b13b2e5936ee P 197b880fa39d432a9b472845876bfaf0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260129 02:58:45.385008 20631 consensus_queue.cc:260] T dc2f4cb752cf4d59b5f6bc689f5d8104 P 1c50de89a5a845668696dcb28c3badd6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:58:45.385583 20721 raft_consensus.cc:515] T 1ce4ddf5b10e4ae8b193b13b2e5936ee P 197b880fa39d432a9b472845876bfaf0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
W20260129 02:58:45.386019 20556 leader_election.cc:343] T 4f8f3a38f6e24ba4857875181f447442 P 1c50de89a5a845668696dcb28c3badd6 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 974b802141bd4f13beffef2aa03f88f9 (127.18.211.2:34517): Illegal state: must be running to vote when last-logged opid is not known
I20260129 02:58:45.387099 20631 ts_tablet_manager.cc:1434] T dc2f4cb752cf4d59b5f6bc689f5d8104 P 1c50de89a5a845668696dcb28c3badd6: Time spent starting tablet: real 0.015s user 0.008s sys 0.000s
I20260129 02:58:45.387189 20633 tablet_bootstrap.cc:492] T 913fc7d751ed4b7388d889aac30a5f9b P 197b880fa39d432a9b472845876bfaf0: Bootstrap starting.
I20260129 02:58:45.387476 20721 leader_election.cc:290] T 1ce4ddf5b10e4ae8b193b13b2e5936ee P 197b880fa39d432a9b472845876bfaf0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 974b802141bd4f13beffef2aa03f88f9 (127.18.211.2:34517), 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3:37735)
I20260129 02:58:45.387995 20631 tablet_bootstrap.cc:492] T e4d18c35b0a7493e85d0a4b92956a48f P 1c50de89a5a845668696dcb28c3badd6: Bootstrap starting.
I20260129 02:58:45.376215 20588 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "febffe4a8b394368b69b1d164308fb9a" candidate_uuid: "197b880fa39d432a9b472845876bfaf0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c50de89a5a845668696dcb28c3badd6"
I20260129 02:58:45.388453 20588 raft_consensus.cc:3060] T febffe4a8b394368b69b1d164308fb9a P 1c50de89a5a845668696dcb28c3badd6 [term 0 FOLLOWER]: Advancing to term 1
I20260129 02:58:45.389859 20589 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1ce4ddf5b10e4ae8b193b13b2e5936ee" candidate_uuid: "197b880fa39d432a9b472845876bfaf0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c50de89a5a845668696dcb28c3badd6" is_pre_election: true
I20260129 02:58:45.390486 20589 raft_consensus.cc:2468] T 1ce4ddf5b10e4ae8b193b13b2e5936ee P 1c50de89a5a845668696dcb28c3badd6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 197b880fa39d432a9b472845876bfaf0 in term 0.
I20260129 02:58:45.392176 20405 leader_election.cc:304] T 1ce4ddf5b10e4ae8b193b13b2e5936ee P 197b880fa39d432a9b472845876bfaf0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 197b880fa39d432a9b472845876bfaf0, 1c50de89a5a845668696dcb28c3badd6; no voters:
I20260129 02:58:45.393627 20516 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1ce4ddf5b10e4ae8b193b13b2e5936ee" candidate_uuid: "197b880fa39d432a9b472845876bfaf0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "974b802141bd4f13beffef2aa03f88f9" is_pre_election: true
I20260129 02:58:45.393828 20658 raft_consensus.cc:2804] T 1ce4ddf5b10e4ae8b193b13b2e5936ee P 197b880fa39d432a9b472845876bfaf0 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260129 02:58:45.394207 20658 raft_consensus.cc:493] T 1ce4ddf5b10e4ae8b193b13b2e5936ee P 197b880fa39d432a9b472845876bfaf0 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260129 02:58:45.394238 20631 tablet_bootstrap.cc:654] T e4d18c35b0a7493e85d0a4b92956a48f P 1c50de89a5a845668696dcb28c3badd6: Neither blocks nor log segments found. Creating new log.
I20260129 02:58:45.394232 20516 raft_consensus.cc:2468] T 1ce4ddf5b10e4ae8b193b13b2e5936ee P 974b802141bd4f13beffef2aa03f88f9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 197b880fa39d432a9b472845876bfaf0 in term 0.
I20260129 02:58:45.410296 20658 raft_consensus.cc:3060] T 1ce4ddf5b10e4ae8b193b13b2e5936ee P 197b880fa39d432a9b472845876bfaf0 [term 0 FOLLOWER]: Advancing to term 1
I20260129 02:58:45.411808 20588 raft_consensus.cc:2468] T febffe4a8b394368b69b1d164308fb9a P 1c50de89a5a845668696dcb28c3badd6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 197b880fa39d432a9b472845876bfaf0 in term 1.
I20260129 02:58:45.413451 20405 leader_election.cc:304] T febffe4a8b394368b69b1d164308fb9a P 197b880fa39d432a9b472845876bfaf0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 197b880fa39d432a9b472845876bfaf0, 1c50de89a5a845668696dcb28c3badd6; no voters:
I20260129 02:58:45.412266 20515 raft_consensus.cc:2468] T fd8311b779af4e1fa1402b2a02a602f2 P 974b802141bd4f13beffef2aa03f88f9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1c50de89a5a845668696dcb28c3badd6 in term 1.
I20260129 02:58:45.415136 20721 raft_consensus.cc:2804] T febffe4a8b394368b69b1d164308fb9a P 197b880fa39d432a9b472845876bfaf0 [term 1 FOLLOWER]: Leader election won for term 1
I20260129 02:58:45.415374 20556 leader_election.cc:304] T fd8311b779af4e1fa1402b2a02a602f2 P 1c50de89a5a845668696dcb28c3badd6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1c50de89a5a845668696dcb28c3badd6, 974b802141bd4f13beffef2aa03f88f9; no voters: 197b880fa39d432a9b472845876bfaf0
I20260129 02:58:45.416106 20721 raft_consensus.cc:697] T febffe4a8b394368b69b1d164308fb9a P 197b880fa39d432a9b472845876bfaf0 [term 1 LEADER]: Becoming Leader. State: Replica: 197b880fa39d432a9b472845876bfaf0, State: Running, Role: LEADER
I20260129 02:58:45.416713 20771 raft_consensus.cc:2804] T fd8311b779af4e1fa1402b2a02a602f2 P 1c50de89a5a845668696dcb28c3badd6 [term 1 FOLLOWER]: Leader election won for term 1
I20260129 02:58:45.417208 20771 raft_consensus.cc:697] T fd8311b779af4e1fa1402b2a02a602f2 P 1c50de89a5a845668696dcb28c3badd6 [term 1 LEADER]: Becoming Leader. State: Replica: 1c50de89a5a845668696dcb28c3badd6, State: Running, Role: LEADER
I20260129 02:58:45.417457 20721 consensus_queue.cc:237] T febffe4a8b394368b69b1d164308fb9a P 197b880fa39d432a9b472845876bfaf0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:58:45.417990 20771 consensus_queue.cc:237] T fd8311b779af4e1fa1402b2a02a602f2 P 1c50de89a5a845668696dcb28c3badd6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:58:45.423724 20719 consensus_queue.cc:1048] T 78aa39096868415d9afc2ffcb248e198 P 1c50de89a5a845668696dcb28c3badd6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260129 02:58:45.394935 20441 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4f8f3a38f6e24ba4857875181f447442" candidate_uuid: "1c50de89a5a845668696dcb28c3badd6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "197b880fa39d432a9b472845876bfaf0" is_pre_election: true
I20260129 02:58:45.425562 20633 tablet_bootstrap.cc:654] T 913fc7d751ed4b7388d889aac30a5f9b P 197b880fa39d432a9b472845876bfaf0: Neither blocks nor log segments found. Creating new log.
W20260129 02:58:45.426481 20553 leader_election.cc:343] T 4f8f3a38f6e24ba4857875181f447442 P 1c50de89a5a845668696dcb28c3badd6 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 197b880fa39d432a9b472845876bfaf0 (127.18.211.1:42027): Illegal state: must be running to vote when last-logged opid is not known
I20260129 02:58:45.426987 20553 leader_election.cc:304] T 4f8f3a38f6e24ba4857875181f447442 P 1c50de89a5a845668696dcb28c3badd6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1c50de89a5a845668696dcb28c3badd6; no voters: 197b880fa39d432a9b472845876bfaf0, 974b802141bd4f13beffef2aa03f88f9
I20260129 02:58:45.425335 20641 consensus_queue.cc:1048] T 78aa39096868415d9afc2ffcb248e198 P 1c50de89a5a845668696dcb28c3badd6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260129 02:58:45.428671 20514 raft_consensus.cc:2468] T febffe4a8b394368b69b1d164308fb9a P 974b802141bd4f13beffef2aa03f88f9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 197b880fa39d432a9b472845876bfaf0 in term 1.
I20260129 02:58:45.429013 20641 raft_consensus.cc:2749] T 4f8f3a38f6e24ba4857875181f447442 P 1c50de89a5a845668696dcb28c3badd6 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260129 02:58:45.442538 20337 catalog_manager.cc:5654] T fd8311b779af4e1fa1402b2a02a602f2 P 1c50de89a5a845668696dcb28c3badd6 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3). New cstate: current_term: 1 leader_uuid: "1c50de89a5a845668696dcb28c3badd6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } health_report { overall_health: HEALTHY } } }
I20260129 02:58:45.445719 20631 tablet_bootstrap.cc:492] T e4d18c35b0a7493e85d0a4b92956a48f P 1c50de89a5a845668696dcb28c3badd6: No bootstrap required, opened a new log
I20260129 02:58:45.446197 20631 ts_tablet_manager.cc:1403] T e4d18c35b0a7493e85d0a4b92956a48f P 1c50de89a5a845668696dcb28c3badd6: Time spent bootstrapping tablet: real 0.058s user 0.012s sys 0.000s
I20260129 02:58:45.448721 20633 tablet_bootstrap.cc:492] T 913fc7d751ed4b7388d889aac30a5f9b P 197b880fa39d432a9b472845876bfaf0: No bootstrap required, opened a new log
I20260129 02:58:45.448705 20631 raft_consensus.cc:359] T e4d18c35b0a7493e85d0a4b92956a48f P 1c50de89a5a845668696dcb28c3badd6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:58:45.449317 20633 ts_tablet_manager.cc:1403] T 913fc7d751ed4b7388d889aac30a5f9b P 197b880fa39d432a9b472845876bfaf0: Time spent bootstrapping tablet: real 0.062s user 0.008s sys 0.003s
I20260129 02:58:45.449445 20631 raft_consensus.cc:385] T e4d18c35b0a7493e85d0a4b92956a48f P 1c50de89a5a845668696dcb28c3badd6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260129 02:58:45.449802 20631 raft_consensus.cc:740] T e4d18c35b0a7493e85d0a4b92956a48f P 1c50de89a5a845668696dcb28c3badd6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1c50de89a5a845668696dcb28c3badd6, State: Initialized, Role: FOLLOWER
I20260129 02:58:45.450603 20631 consensus_queue.cc:260] T e4d18c35b0a7493e85d0a4b92956a48f P 1c50de89a5a845668696dcb28c3badd6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:58:45.450948 20658 raft_consensus.cc:515] T 1ce4ddf5b10e4ae8b193b13b2e5936ee P 197b880fa39d432a9b472845876bfaf0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:58:45.452244 20633 raft_consensus.cc:359] T 913fc7d751ed4b7388d889aac30a5f9b P 197b880fa39d432a9b472845876bfaf0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:58:45.452922 20633 raft_consensus.cc:385] T 913fc7d751ed4b7388d889aac30a5f9b P 197b880fa39d432a9b472845876bfaf0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260129 02:58:45.453230 20633 raft_consensus.cc:740] T 913fc7d751ed4b7388d889aac30a5f9b P 197b880fa39d432a9b472845876bfaf0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 197b880fa39d432a9b472845876bfaf0, State: Initialized, Role: FOLLOWER
I20260129 02:58:45.453884 20633 consensus_queue.cc:260] T 913fc7d751ed4b7388d889aac30a5f9b P 197b880fa39d432a9b472845876bfaf0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:58:45.456319 20633 ts_tablet_manager.cc:1434] T 913fc7d751ed4b7388d889aac30a5f9b P 197b880fa39d432a9b472845876bfaf0: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260129 02:58:45.457023 20633 tablet_bootstrap.cc:492] T 76f713c3331540d2a5b76164103e9eb1 P 197b880fa39d432a9b472845876bfaf0: Bootstrap starting.
W20260129 02:58:45.443236 20387 auto_rebalancer.cc:229] Could not retrieve cluster info: Service unavailable: Tablet not running
I20260129 02:58:45.460427 20631 ts_tablet_manager.cc:1434] T e4d18c35b0a7493e85d0a4b92956a48f P 1c50de89a5a845668696dcb28c3badd6: Time spent starting tablet: real 0.014s user 0.007s sys 0.001s
I20260129 02:58:45.461294 20631 tablet_bootstrap.cc:492] T 75b0b6e82ddc4f92b1fe1d94f9a0b73b P 1c50de89a5a845668696dcb28c3badd6: Bootstrap starting.
I20260129 02:58:45.459825 20658 leader_election.cc:290] T 1ce4ddf5b10e4ae8b193b13b2e5936ee P 197b880fa39d432a9b472845876bfaf0 [CANDIDATE]: Term 1 election: Requested vote from peers 974b802141bd4f13beffef2aa03f88f9 (127.18.211.2:34517), 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3:37735)
I20260129 02:58:45.459818 20515 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1ce4ddf5b10e4ae8b193b13b2e5936ee" candidate_uuid: "197b880fa39d432a9b472845876bfaf0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "974b802141bd4f13beffef2aa03f88f9"
I20260129 02:58:45.463672 20515 raft_consensus.cc:3060] T 1ce4ddf5b10e4ae8b193b13b2e5936ee P 974b802141bd4f13beffef2aa03f88f9 [term 0 FOLLOWER]: Advancing to term 1
I20260129 02:58:45.464089 20588 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1ce4ddf5b10e4ae8b193b13b2e5936ee" candidate_uuid: "197b880fa39d432a9b472845876bfaf0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c50de89a5a845668696dcb28c3badd6"
I20260129 02:58:45.464663 20588 raft_consensus.cc:3060] T 1ce4ddf5b10e4ae8b193b13b2e5936ee P 1c50de89a5a845668696dcb28c3badd6 [term 0 FOLLOWER]: Advancing to term 1
I20260129 02:58:45.470228 20515 raft_consensus.cc:2468] T 1ce4ddf5b10e4ae8b193b13b2e5936ee P 974b802141bd4f13beffef2aa03f88f9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 197b880fa39d432a9b472845876bfaf0 in term 1.
I20260129 02:58:45.473220 20407 leader_election.cc:304] T 1ce4ddf5b10e4ae8b193b13b2e5936ee P 197b880fa39d432a9b472845876bfaf0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 197b880fa39d432a9b472845876bfaf0, 974b802141bd4f13beffef2aa03f88f9; no voters:
I20260129 02:58:45.475829 20658 raft_consensus.cc:2804] T 1ce4ddf5b10e4ae8b193b13b2e5936ee P 197b880fa39d432a9b472845876bfaf0 [term 1 FOLLOWER]: Leader election won for term 1
I20260129 02:58:45.474045 20638 consensus_queue.cc:1048] T e4d18c35b0a7493e85d0a4b92956a48f P 197b880fa39d432a9b472845876bfaf0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.002s
I20260129 02:58:45.476790 20658 raft_consensus.cc:697] T 1ce4ddf5b10e4ae8b193b13b2e5936ee P 197b880fa39d432a9b472845876bfaf0 [term 1 LEADER]: Becoming Leader. State: Replica: 197b880fa39d432a9b472845876bfaf0, State: Running, Role: LEADER
I20260129 02:58:45.478639 20658 consensus_queue.cc:237] T 1ce4ddf5b10e4ae8b193b13b2e5936ee P 197b880fa39d432a9b472845876bfaf0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:58:45.479516 20588 raft_consensus.cc:2468] T 1ce4ddf5b10e4ae8b193b13b2e5936ee P 1c50de89a5a845668696dcb28c3badd6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 197b880fa39d432a9b472845876bfaf0 in term 1.
I20260129 02:58:45.480543 20633 tablet_bootstrap.cc:654] T 76f713c3331540d2a5b76164103e9eb1 P 197b880fa39d432a9b472845876bfaf0: Neither blocks nor log segments found. Creating new log.
I20260129 02:58:45.481570 20638 raft_consensus.cc:493] T 2ac659d732314e1288849f08fe5e074e P 197b880fa39d432a9b472845876bfaf0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260129 02:58:45.482296 20638 raft_consensus.cc:515] T 2ac659d732314e1288849f08fe5e074e P 197b880fa39d432a9b472845876bfaf0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:58:45.484247 20638 leader_election.cc:290] T 2ac659d732314e1288849f08fe5e074e P 197b880fa39d432a9b472845876bfaf0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 974b802141bd4f13beffef2aa03f88f9 (127.18.211.2:34517), 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3:37735)
I20260129 02:58:45.485070 20630 tablet_bootstrap.cc:492] T 4f8f3a38f6e24ba4857875181f447442 P 974b802141bd4f13beffef2aa03f88f9: No bootstrap required, opened a new log
I20260129 02:58:45.485481 20630 ts_tablet_manager.cc:1403] T 4f8f3a38f6e24ba4857875181f447442 P 974b802141bd4f13beffef2aa03f88f9: Time spent bootstrapping tablet: real 0.143s user 0.016s sys 0.015s
I20260129 02:58:45.487290 20631 tablet_bootstrap.cc:654] T 75b0b6e82ddc4f92b1fe1d94f9a0b73b P 1c50de89a5a845668696dcb28c3badd6: Neither blocks nor log segments found. Creating new log.
I20260129 02:58:45.487267 20630 raft_consensus.cc:359] T 4f8f3a38f6e24ba4857875181f447442 P 974b802141bd4f13beffef2aa03f88f9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:58:45.487993 20630 raft_consensus.cc:385] T 4f8f3a38f6e24ba4857875181f447442 P 974b802141bd4f13beffef2aa03f88f9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260129 02:58:45.488224 20630 raft_consensus.cc:740] T 4f8f3a38f6e24ba4857875181f447442 P 974b802141bd4f13beffef2aa03f88f9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 974b802141bd4f13beffef2aa03f88f9, State: Initialized, Role: FOLLOWER
I20260129 02:58:45.488757 20630 consensus_queue.cc:260] T 4f8f3a38f6e24ba4857875181f447442 P 974b802141bd4f13beffef2aa03f88f9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:58:45.490521 20630 ts_tablet_manager.cc:1434] T 4f8f3a38f6e24ba4857875181f447442 P 974b802141bd4f13beffef2aa03f88f9: Time spent starting tablet: real 0.005s user 0.003s sys 0.000s
I20260129 02:58:45.490757 20514 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2ac659d732314e1288849f08fe5e074e" candidate_uuid: "197b880fa39d432a9b472845876bfaf0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "974b802141bd4f13beffef2aa03f88f9" is_pre_election: true
I20260129 02:58:45.491559 20630 tablet_bootstrap.cc:492] T 913fc7d751ed4b7388d889aac30a5f9b P 974b802141bd4f13beffef2aa03f88f9: Bootstrap starting.
I20260129 02:58:45.492404 20588 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2ac659d732314e1288849f08fe5e074e" candidate_uuid: "197b880fa39d432a9b472845876bfaf0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c50de89a5a845668696dcb28c3badd6" is_pre_election: true
I20260129 02:58:45.494005 20405 leader_election.cc:304] T 2ac659d732314e1288849f08fe5e074e P 197b880fa39d432a9b472845876bfaf0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 197b880fa39d432a9b472845876bfaf0; no voters: 1c50de89a5a845668696dcb28c3badd6, 974b802141bd4f13beffef2aa03f88f9
I20260129 02:58:45.490540 20337 catalog_manager.cc:5654] T febffe4a8b394368b69b1d164308fb9a P 197b880fa39d432a9b472845876bfaf0 reported cstate change: term changed from 0 to 1, leader changed from <none> to 197b880fa39d432a9b472845876bfaf0 (127.18.211.1). New cstate: current_term: 1 leader_uuid: "197b880fa39d432a9b472845876bfaf0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } health_report { overall_health: UNKNOWN } } }
I20260129 02:58:45.495575 20721 raft_consensus.cc:2749] T 2ac659d732314e1288849f08fe5e074e P 197b880fa39d432a9b472845876bfaf0 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260129 02:58:45.495911 20771 raft_consensus.cc:493] T 4e420f90884f4796997b5ea41e49e411 P 1c50de89a5a845668696dcb28c3badd6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260129 02:58:45.496342 20771 raft_consensus.cc:515] T 4e420f90884f4796997b5ea41e49e411 P 1c50de89a5a845668696dcb28c3badd6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:58:45.500029 20771 leader_election.cc:290] T 4e420f90884f4796997b5ea41e49e411 P 1c50de89a5a845668696dcb28c3badd6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 197b880fa39d432a9b472845876bfaf0 (127.18.211.1:42027), 974b802141bd4f13beffef2aa03f88f9 (127.18.211.2:34517)
I20260129 02:58:45.500331 20631 tablet_bootstrap.cc:492] T 75b0b6e82ddc4f92b1fe1d94f9a0b73b P 1c50de89a5a845668696dcb28c3badd6: No bootstrap required, opened a new log
I20260129 02:58:45.501050 20631 ts_tablet_manager.cc:1403] T 75b0b6e82ddc4f92b1fe1d94f9a0b73b P 1c50de89a5a845668696dcb28c3badd6: Time spent bootstrapping tablet: real 0.040s user 0.004s sys 0.005s
I20260129 02:58:45.502746 20440 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4e420f90884f4796997b5ea41e49e411" candidate_uuid: "1c50de89a5a845668696dcb28c3badd6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "197b880fa39d432a9b472845876bfaf0" is_pre_election: true
I20260129 02:58:45.502746 20514 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4e420f90884f4796997b5ea41e49e411" candidate_uuid: "1c50de89a5a845668696dcb28c3badd6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "974b802141bd4f13beffef2aa03f88f9" is_pre_election: true
I20260129 02:58:45.507571 20514 raft_consensus.cc:2393] T 4e420f90884f4796997b5ea41e49e411 P 974b802141bd4f13beffef2aa03f88f9 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1c50de89a5a845668696dcb28c3badd6 in current term 1: Already voted for candidate 197b880fa39d432a9b472845876bfaf0 in this term.
I20260129 02:58:45.515677 20553 leader_election.cc:304] T 4e420f90884f4796997b5ea41e49e411 P 1c50de89a5a845668696dcb28c3badd6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1c50de89a5a845668696dcb28c3badd6; no voters: 197b880fa39d432a9b472845876bfaf0, 974b802141bd4f13beffef2aa03f88f9
I20260129 02:58:45.516850 20771 raft_consensus.cc:3060] T 4e420f90884f4796997b5ea41e49e411 P 1c50de89a5a845668696dcb28c3badd6 [term 0 FOLLOWER]: Advancing to term 1
I20260129 02:58:45.524214 20734 raft_consensus.cc:493] T 75b0b6e82ddc4f92b1fe1d94f9a0b73b P 974b802141bd4f13beffef2aa03f88f9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260129 02:58:45.524730 20734 raft_consensus.cc:515] T 75b0b6e82ddc4f92b1fe1d94f9a0b73b P 974b802141bd4f13beffef2aa03f88f9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:58:45.526641 20734 leader_election.cc:290] T 75b0b6e82ddc4f92b1fe1d94f9a0b73b P 974b802141bd4f13beffef2aa03f88f9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 197b880fa39d432a9b472845876bfaf0 (127.18.211.1:42027), 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3:37735)
I20260129 02:58:45.527949 20440 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "75b0b6e82ddc4f92b1fe1d94f9a0b73b" candidate_uuid: "974b802141bd4f13beffef2aa03f88f9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "197b880fa39d432a9b472845876bfaf0" is_pre_election: true
I20260129 02:58:45.528148 20588 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "75b0b6e82ddc4f92b1fe1d94f9a0b73b" candidate_uuid: "974b802141bd4f13beffef2aa03f88f9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c50de89a5a845668696dcb28c3badd6" is_pre_election: true
W20260129 02:58:45.529682 20480 leader_election.cc:343] T 75b0b6e82ddc4f92b1fe1d94f9a0b73b P 974b802141bd4f13beffef2aa03f88f9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3:37735): Illegal state: must be running to vote when last-logged opid is not known
W20260129 02:58:45.530190 20479 leader_election.cc:343] T 75b0b6e82ddc4f92b1fe1d94f9a0b73b P 974b802141bd4f13beffef2aa03f88f9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 197b880fa39d432a9b472845876bfaf0 (127.18.211.1:42027): Illegal state: must be running to vote when last-logged opid is not known
I20260129 02:58:45.532271 20630 tablet_bootstrap.cc:654] T 913fc7d751ed4b7388d889aac30a5f9b P 974b802141bd4f13beffef2aa03f88f9: Neither blocks nor log segments found. Creating new log.
I20260129 02:58:45.531967 20479 leader_election.cc:304] T 75b0b6e82ddc4f92b1fe1d94f9a0b73b P 974b802141bd4f13beffef2aa03f88f9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 974b802141bd4f13beffef2aa03f88f9; no voters: 197b880fa39d432a9b472845876bfaf0, 1c50de89a5a845668696dcb28c3badd6
I20260129 02:58:45.532963 20734 raft_consensus.cc:2749] T 75b0b6e82ddc4f92b1fe1d94f9a0b73b P 974b802141bd4f13beffef2aa03f88f9 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260129 02:58:45.541474 20631 raft_consensus.cc:359] T 75b0b6e82ddc4f92b1fe1d94f9a0b73b P 1c50de89a5a845668696dcb28c3badd6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:58:45.542707 20631 raft_consensus.cc:385] T 75b0b6e82ddc4f92b1fe1d94f9a0b73b P 1c50de89a5a845668696dcb28c3badd6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260129 02:58:45.542979 20631 raft_consensus.cc:740] T 75b0b6e82ddc4f92b1fe1d94f9a0b73b P 1c50de89a5a845668696dcb28c3badd6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1c50de89a5a845668696dcb28c3badd6, State: Initialized, Role: FOLLOWER
I20260129 02:58:45.543567 20631 consensus_queue.cc:260] T 75b0b6e82ddc4f92b1fe1d94f9a0b73b P 1c50de89a5a845668696dcb28c3badd6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:58:45.553109 20631 ts_tablet_manager.cc:1434] T 75b0b6e82ddc4f92b1fe1d94f9a0b73b P 1c50de89a5a845668696dcb28c3badd6: Time spent starting tablet: real 0.052s user 0.010s sys 0.010s
I20260129 02:58:45.554121 20631 tablet_bootstrap.cc:492] T 76f713c3331540d2a5b76164103e9eb1 P 1c50de89a5a845668696dcb28c3badd6: Bootstrap starting.
I20260129 02:58:45.560693 20771 raft_consensus.cc:2749] T 4e420f90884f4796997b5ea41e49e411 P 1c50de89a5a845668696dcb28c3badd6 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260129 02:58:45.565603 20633 tablet_bootstrap.cc:492] T 76f713c3331540d2a5b76164103e9eb1 P 197b880fa39d432a9b472845876bfaf0: No bootstrap required, opened a new log
I20260129 02:58:45.566244 20633 ts_tablet_manager.cc:1403] T 76f713c3331540d2a5b76164103e9eb1 P 197b880fa39d432a9b472845876bfaf0: Time spent bootstrapping tablet: real 0.109s user 0.013s sys 0.000s
I20260129 02:58:45.570416 20633 raft_consensus.cc:359] T 76f713c3331540d2a5b76164103e9eb1 P 197b880fa39d432a9b472845876bfaf0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:58:45.572043 20633 raft_consensus.cc:385] T 76f713c3331540d2a5b76164103e9eb1 P 197b880fa39d432a9b472845876bfaf0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260129 02:58:45.572162 20630 tablet_bootstrap.cc:492] T 913fc7d751ed4b7388d889aac30a5f9b P 974b802141bd4f13beffef2aa03f88f9: No bootstrap required, opened a new log
I20260129 02:58:45.572352 20633 raft_consensus.cc:740] T 76f713c3331540d2a5b76164103e9eb1 P 197b880fa39d432a9b472845876bfaf0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 197b880fa39d432a9b472845876bfaf0, State: Initialized, Role: FOLLOWER
I20260129 02:58:45.572901 20630 ts_tablet_manager.cc:1403] T 913fc7d751ed4b7388d889aac30a5f9b P 974b802141bd4f13beffef2aa03f88f9: Time spent bootstrapping tablet: real 0.082s user 0.013s sys 0.002s
I20260129 02:58:45.575527 20631 tablet_bootstrap.cc:654] T 76f713c3331540d2a5b76164103e9eb1 P 1c50de89a5a845668696dcb28c3badd6: Neither blocks nor log segments found. Creating new log.
I20260129 02:58:45.575346 20630 raft_consensus.cc:359] T 913fc7d751ed4b7388d889aac30a5f9b P 974b802141bd4f13beffef2aa03f88f9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:58:45.576203 20630 raft_consensus.cc:385] T 913fc7d751ed4b7388d889aac30a5f9b P 974b802141bd4f13beffef2aa03f88f9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260129 02:58:45.576537 20630 raft_consensus.cc:740] T 913fc7d751ed4b7388d889aac30a5f9b P 974b802141bd4f13beffef2aa03f88f9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 974b802141bd4f13beffef2aa03f88f9, State: Initialized, Role: FOLLOWER
I20260129 02:58:45.576440 20633 consensus_queue.cc:260] T 76f713c3331540d2a5b76164103e9eb1 P 197b880fa39d432a9b472845876bfaf0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:58:45.577342 20630 consensus_queue.cc:260] T 913fc7d751ed4b7388d889aac30a5f9b P 974b802141bd4f13beffef2aa03f88f9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:58:45.578647 20633 ts_tablet_manager.cc:1434] T 76f713c3331540d2a5b76164103e9eb1 P 197b880fa39d432a9b472845876bfaf0: Time spent starting tablet: real 0.012s user 0.006s sys 0.001s
I20260129 02:58:45.578327 20337 catalog_manager.cc:5654] T 1ce4ddf5b10e4ae8b193b13b2e5936ee P 197b880fa39d432a9b472845876bfaf0 reported cstate change: term changed from 0 to 1, leader changed from <none> to 197b880fa39d432a9b472845876bfaf0 (127.18.211.1). New cstate: current_term: 1 leader_uuid: "197b880fa39d432a9b472845876bfaf0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } health_report { overall_health: UNKNOWN } } }
I20260129 02:58:45.579532 20633 tablet_bootstrap.cc:492] T fd8311b779af4e1fa1402b2a02a602f2 P 197b880fa39d432a9b472845876bfaf0: Bootstrap starting.
I20260129 02:58:45.579672 20630 ts_tablet_manager.cc:1434] T 913fc7d751ed4b7388d889aac30a5f9b P 974b802141bd4f13beffef2aa03f88f9: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260129 02:58:45.580922 20630 tablet_bootstrap.cc:492] T 7a9a69cd82724b1faed1d10aa091fc4a P 974b802141bd4f13beffef2aa03f88f9: Bootstrap starting.
I20260129 02:58:45.586604 20630 tablet_bootstrap.cc:654] T 7a9a69cd82724b1faed1d10aa091fc4a P 974b802141bd4f13beffef2aa03f88f9: Neither blocks nor log segments found. Creating new log.
I20260129 02:58:45.588567 20633 tablet_bootstrap.cc:654] T fd8311b779af4e1fa1402b2a02a602f2 P 197b880fa39d432a9b472845876bfaf0: Neither blocks nor log segments found. Creating new log.
I20260129 02:58:45.589087 20440 raft_consensus.cc:3060] T b189f08e16a74a11980cc55d99359a0a P 197b880fa39d432a9b472845876bfaf0 [term 0 FOLLOWER]: Advancing to term 1
I20260129 02:58:45.589758 20739 raft_consensus.cc:493] T 76f713c3331540d2a5b76164103e9eb1 P 974b802141bd4f13beffef2aa03f88f9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260129 02:58:45.590515 20739 raft_consensus.cc:515] T 76f713c3331540d2a5b76164103e9eb1 P 974b802141bd4f13beffef2aa03f88f9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:58:45.592331 20658 raft_consensus.cc:493] T 913fc7d751ed4b7388d889aac30a5f9b P 197b880fa39d432a9b472845876bfaf0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260129 02:58:45.594022 20739 leader_election.cc:290] T 76f713c3331540d2a5b76164103e9eb1 P 974b802141bd4f13beffef2aa03f88f9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 197b880fa39d432a9b472845876bfaf0 (127.18.211.1:42027), 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3:37735)
I20260129 02:58:45.596652 20588 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "76f713c3331540d2a5b76164103e9eb1" candidate_uuid: "974b802141bd4f13beffef2aa03f88f9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c50de89a5a845668696dcb28c3badd6" is_pre_election: true
W20260129 02:58:45.599126 20480 leader_election.cc:343] T 76f713c3331540d2a5b76164103e9eb1 P 974b802141bd4f13beffef2aa03f88f9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3:37735): Illegal state: must be running to vote when last-logged opid is not known
I20260129 02:58:45.604185 20441 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "76f713c3331540d2a5b76164103e9eb1" candidate_uuid: "974b802141bd4f13beffef2aa03f88f9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "197b880fa39d432a9b472845876bfaf0" is_pre_election: true
I20260129 02:58:45.604846 20441 raft_consensus.cc:2468] T 76f713c3331540d2a5b76164103e9eb1 P 197b880fa39d432a9b472845876bfaf0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 974b802141bd4f13beffef2aa03f88f9 in term 0.
I20260129 02:58:45.592938 20658 raft_consensus.cc:515] T 913fc7d751ed4b7388d889aac30a5f9b P 197b880fa39d432a9b472845876bfaf0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:58:45.606415 20479 leader_election.cc:304] T 76f713c3331540d2a5b76164103e9eb1 P 974b802141bd4f13beffef2aa03f88f9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 197b880fa39d432a9b472845876bfaf0, 974b802141bd4f13beffef2aa03f88f9; no voters: 1c50de89a5a845668696dcb28c3badd6
I20260129 02:58:45.607524 20734 raft_consensus.cc:2804] T 76f713c3331540d2a5b76164103e9eb1 P 974b802141bd4f13beffef2aa03f88f9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260129 02:58:45.607925 20734 raft_consensus.cc:493] T 76f713c3331540d2a5b76164103e9eb1 P 974b802141bd4f13beffef2aa03f88f9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260129 02:58:45.608258 20734 raft_consensus.cc:3060] T 76f713c3331540d2a5b76164103e9eb1 P 974b802141bd4f13beffef2aa03f88f9 [term 0 FOLLOWER]: Advancing to term 1
I20260129 02:58:45.608350 20514 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "913fc7d751ed4b7388d889aac30a5f9b" candidate_uuid: "197b880fa39d432a9b472845876bfaf0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "974b802141bd4f13beffef2aa03f88f9" is_pre_election: true
I20260129 02:58:45.608999 20514 raft_consensus.cc:2468] T 913fc7d751ed4b7388d889aac30a5f9b P 974b802141bd4f13beffef2aa03f88f9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 197b880fa39d432a9b472845876bfaf0 in term 0.
I20260129 02:58:45.610348 20407 leader_election.cc:304] T 913fc7d751ed4b7388d889aac30a5f9b P 197b880fa39d432a9b472845876bfaf0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 197b880fa39d432a9b472845876bfaf0, 974b802141bd4f13beffef2aa03f88f9; no voters:
I20260129 02:58:45.611163 20729 raft_consensus.cc:2804] T 913fc7d751ed4b7388d889aac30a5f9b P 197b880fa39d432a9b472845876bfaf0 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260129 02:58:45.611534 20729 raft_consensus.cc:493] T 913fc7d751ed4b7388d889aac30a5f9b P 197b880fa39d432a9b472845876bfaf0 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260129 02:58:45.611845 20729 raft_consensus.cc:3060] T 913fc7d751ed4b7388d889aac30a5f9b P 197b880fa39d432a9b472845876bfaf0 [term 0 FOLLOWER]: Advancing to term 1
I20260129 02:58:45.615636 20734 raft_consensus.cc:515] T 76f713c3331540d2a5b76164103e9eb1 P 974b802141bd4f13beffef2aa03f88f9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:58:45.618547 20439 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "76f713c3331540d2a5b76164103e9eb1" candidate_uuid: "974b802141bd4f13beffef2aa03f88f9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "197b880fa39d432a9b472845876bfaf0"
I20260129 02:58:45.619204 20439 raft_consensus.cc:3060] T 76f713c3331540d2a5b76164103e9eb1 P 197b880fa39d432a9b472845876bfaf0 [term 0 FOLLOWER]: Advancing to term 1
I20260129 02:58:45.623356 20588 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "913fc7d751ed4b7388d889aac30a5f9b" candidate_uuid: "197b880fa39d432a9b472845876bfaf0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c50de89a5a845668696dcb28c3badd6" is_pre_election: true
I20260129 02:58:45.624089 20588 raft_consensus.cc:2468] T 913fc7d751ed4b7388d889aac30a5f9b P 1c50de89a5a845668696dcb28c3badd6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 197b880fa39d432a9b472845876bfaf0 in term 0.
I20260129 02:58:45.624307 20734 leader_election.cc:290] T 76f713c3331540d2a5b76164103e9eb1 P 974b802141bd4f13beffef2aa03f88f9 [CANDIDATE]: Term 1 election: Requested vote from peers 197b880fa39d432a9b472845876bfaf0 (127.18.211.1:42027), 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3:37735)
I20260129 02:58:45.625666 20589 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "76f713c3331540d2a5b76164103e9eb1" candidate_uuid: "974b802141bd4f13beffef2aa03f88f9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c50de89a5a845668696dcb28c3badd6"
I20260129 02:58:45.626036 20658 leader_election.cc:290] T 913fc7d751ed4b7388d889aac30a5f9b P 197b880fa39d432a9b472845876bfaf0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 974b802141bd4f13beffef2aa03f88f9 (127.18.211.2:34517), 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3:37735)
I20260129 02:58:45.626518 20633 tablet_bootstrap.cc:492] T fd8311b779af4e1fa1402b2a02a602f2 P 197b880fa39d432a9b472845876bfaf0: No bootstrap required, opened a new log
I20260129 02:58:45.626940 20633 ts_tablet_manager.cc:1403] T fd8311b779af4e1fa1402b2a02a602f2 P 197b880fa39d432a9b472845876bfaf0: Time spent bootstrapping tablet: real 0.048s user 0.009s sys 0.001s
W20260129 02:58:45.627254 20480 leader_election.cc:343] T 76f713c3331540d2a5b76164103e9eb1 P 974b802141bd4f13beffef2aa03f88f9 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3:37735): Illegal state: must be running to vote when last-logged opid is not known
I20260129 02:58:45.629624 20729 raft_consensus.cc:515] T 913fc7d751ed4b7388d889aac30a5f9b P 197b880fa39d432a9b472845876bfaf0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:58:45.629844 20633 raft_consensus.cc:359] T fd8311b779af4e1fa1402b2a02a602f2 P 197b880fa39d432a9b472845876bfaf0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:58:45.630436 20633 raft_consensus.cc:385] T fd8311b779af4e1fa1402b2a02a602f2 P 197b880fa39d432a9b472845876bfaf0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260129 02:58:45.630800 20633 raft_consensus.cc:740] T fd8311b779af4e1fa1402b2a02a602f2 P 197b880fa39d432a9b472845876bfaf0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 197b880fa39d432a9b472845876bfaf0, State: Initialized, Role: FOLLOWER
I20260129 02:58:45.631423 20633 consensus_queue.cc:260] T fd8311b779af4e1fa1402b2a02a602f2 P 197b880fa39d432a9b472845876bfaf0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:58:45.633106 20439 raft_consensus.cc:2468] T 76f713c3331540d2a5b76164103e9eb1 P 197b880fa39d432a9b472845876bfaf0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 974b802141bd4f13beffef2aa03f88f9 in term 1.
I20260129 02:58:45.633564 20633 ts_tablet_manager.cc:1434] T fd8311b779af4e1fa1402b2a02a602f2 P 197b880fa39d432a9b472845876bfaf0: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20260129 02:58:45.634361 20771 consensus_queue.cc:1048] T b189f08e16a74a11980cc55d99359a0a P 1c50de89a5a845668696dcb28c3badd6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260129 02:58:45.634756 20633 tablet_bootstrap.cc:492] T 4f8f3a38f6e24ba4857875181f447442 P 197b880fa39d432a9b472845876bfaf0: Bootstrap starting.
I20260129 02:58:45.634903 20638 raft_consensus.cc:493] T fd8311b779af4e1fa1402b2a02a602f2 P 197b880fa39d432a9b472845876bfaf0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260129 02:58:45.635309 20638 raft_consensus.cc:515] T fd8311b779af4e1fa1402b2a02a602f2 P 197b880fa39d432a9b472845876bfaf0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:58:45.639947 20479 leader_election.cc:304] T 76f713c3331540d2a5b76164103e9eb1 P 974b802141bd4f13beffef2aa03f88f9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 197b880fa39d432a9b472845876bfaf0, 974b802141bd4f13beffef2aa03f88f9; no voters: 1c50de89a5a845668696dcb28c3badd6
I20260129 02:58:45.641942 20734 raft_consensus.cc:2804] T 76f713c3331540d2a5b76164103e9eb1 P 974b802141bd4f13beffef2aa03f88f9 [term 1 FOLLOWER]: Leader election won for term 1
I20260129 02:58:45.642426 20734 raft_consensus.cc:697] T 76f713c3331540d2a5b76164103e9eb1 P 974b802141bd4f13beffef2aa03f88f9 [term 1 LEADER]: Becoming Leader. State: Replica: 974b802141bd4f13beffef2aa03f88f9, State: Running, Role: LEADER
I20260129 02:58:45.643268 20734 consensus_queue.cc:237] T 76f713c3331540d2a5b76164103e9eb1 P 974b802141bd4f13beffef2aa03f88f9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:58:45.645223 20638 leader_election.cc:290] T fd8311b779af4e1fa1402b2a02a602f2 P 197b880fa39d432a9b472845876bfaf0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 974b802141bd4f13beffef2aa03f88f9 (127.18.211.2:34517), 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3:37735)
I20260129 02:58:45.646886 20515 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fd8311b779af4e1fa1402b2a02a602f2" candidate_uuid: "197b880fa39d432a9b472845876bfaf0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "974b802141bd4f13beffef2aa03f88f9" is_pre_election: true
I20260129 02:58:45.647686 20515 raft_consensus.cc:2393] T fd8311b779af4e1fa1402b2a02a602f2 P 974b802141bd4f13beffef2aa03f88f9 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 197b880fa39d432a9b472845876bfaf0 in current term 1: Already voted for candidate 1c50de89a5a845668696dcb28c3badd6 in this term.
I20260129 02:58:45.648574 20633 tablet_bootstrap.cc:654] T 4f8f3a38f6e24ba4857875181f447442 P 197b880fa39d432a9b472845876bfaf0: Neither blocks nor log segments found. Creating new log.
I20260129 02:58:45.650203 20589 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fd8311b779af4e1fa1402b2a02a602f2" candidate_uuid: "197b880fa39d432a9b472845876bfaf0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c50de89a5a845668696dcb28c3badd6" is_pre_election: true
I20260129 02:58:45.652472 20439 raft_consensus.cc:3060] T 2ac659d732314e1288849f08fe5e074e P 197b880fa39d432a9b472845876bfaf0 [term 0 FOLLOWER]: Advancing to term 1
I20260129 02:58:45.654558 20630 tablet_bootstrap.cc:492] T 7a9a69cd82724b1faed1d10aa091fc4a P 974b802141bd4f13beffef2aa03f88f9: No bootstrap required, opened a new log
I20260129 02:58:45.655141 20630 ts_tablet_manager.cc:1403] T 7a9a69cd82724b1faed1d10aa091fc4a P 974b802141bd4f13beffef2aa03f88f9: Time spent bootstrapping tablet: real 0.075s user 0.019s sys 0.016s
I20260129 02:58:45.656840 20405 leader_election.cc:304] T fd8311b779af4e1fa1402b2a02a602f2 P 197b880fa39d432a9b472845876bfaf0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 197b880fa39d432a9b472845876bfaf0; no voters: 1c50de89a5a845668696dcb28c3badd6, 974b802141bd4f13beffef2aa03f88f9
I20260129 02:58:45.660185 20638 raft_consensus.cc:3060] T fd8311b779af4e1fa1402b2a02a602f2 P 197b880fa39d432a9b472845876bfaf0 [term 0 FOLLOWER]: Advancing to term 1
I20260129 02:58:45.660971 20630 raft_consensus.cc:359] T 7a9a69cd82724b1faed1d10aa091fc4a P 974b802141bd4f13beffef2aa03f88f9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:58:45.661630 20630 raft_consensus.cc:385] T 7a9a69cd82724b1faed1d10aa091fc4a P 974b802141bd4f13beffef2aa03f88f9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260129 02:58:45.661868 20630 raft_consensus.cc:740] T 7a9a69cd82724b1faed1d10aa091fc4a P 974b802141bd4f13beffef2aa03f88f9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 974b802141bd4f13beffef2aa03f88f9, State: Initialized, Role: FOLLOWER
I20260129 02:58:45.664317 20820 consensus_queue.cc:1048] T fcddab6bca684adf8ea032289a00f90e P 197b880fa39d432a9b472845876bfaf0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260129 02:58:45.667485 20630 consensus_queue.cc:260] T 7a9a69cd82724b1faed1d10aa091fc4a P 974b802141bd4f13beffef2aa03f88f9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:58:45.672833 20633 tablet_bootstrap.cc:492] T 4f8f3a38f6e24ba4857875181f447442 P 197b880fa39d432a9b472845876bfaf0: No bootstrap required, opened a new log
I20260129 02:58:45.673475 20633 ts_tablet_manager.cc:1403] T 4f8f3a38f6e24ba4857875181f447442 P 197b880fa39d432a9b472845876bfaf0: Time spent bootstrapping tablet: real 0.039s user 0.010s sys 0.001s
I20260129 02:58:45.674211 20630 ts_tablet_manager.cc:1434] T 7a9a69cd82724b1faed1d10aa091fc4a P 974b802141bd4f13beffef2aa03f88f9: Time spent starting tablet: real 0.019s user 0.010s sys 0.000s
I20260129 02:58:45.675469 20630 tablet_bootstrap.cc:492] T c2633a6fc34c44579de348d208b0c01e P 974b802141bd4f13beffef2aa03f88f9: Bootstrap starting.
I20260129 02:58:45.677868 20739 raft_consensus.cc:493] T 4f8f3a38f6e24ba4857875181f447442 P 974b802141bd4f13beffef2aa03f88f9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260129 02:58:45.696331 20515 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "913fc7d751ed4b7388d889aac30a5f9b" candidate_uuid: "197b880fa39d432a9b472845876bfaf0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "974b802141bd4f13beffef2aa03f88f9"
I20260129 02:58:45.696950 20515 raft_consensus.cc:3060] T 913fc7d751ed4b7388d889aac30a5f9b P 974b802141bd4f13beffef2aa03f88f9 [term 0 FOLLOWER]: Advancing to term 1
I20260129 02:58:45.691032 20732 raft_consensus.cc:493] T 368953c9102643b2a21f99af36b51905 P 974b802141bd4f13beffef2aa03f88f9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260129 02:58:45.698262 20338 catalog_manager.cc:5654] T 76f713c3331540d2a5b76164103e9eb1 P 974b802141bd4f13beffef2aa03f88f9 reported cstate change: term changed from 0 to 1, leader changed from <none> to 974b802141bd4f13beffef2aa03f88f9 (127.18.211.2). New cstate: current_term: 1 leader_uuid: "974b802141bd4f13beffef2aa03f88f9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } health_report { overall_health: UNKNOWN } } }
I20260129 02:58:45.699100 20732 raft_consensus.cc:515] T 368953c9102643b2a21f99af36b51905 P 974b802141bd4f13beffef2aa03f88f9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:58:45.701845 20732 leader_election.cc:290] T 368953c9102643b2a21f99af36b51905 P 974b802141bd4f13beffef2aa03f88f9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 197b880fa39d432a9b472845876bfaf0 (127.18.211.1:42027), 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3:37735)
I20260129 02:58:45.693552 20739 raft_consensus.cc:515] T 4f8f3a38f6e24ba4857875181f447442 P 974b802141bd4f13beffef2aa03f88f9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:58:45.704062 20588 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "368953c9102643b2a21f99af36b51905" candidate_uuid: "974b802141bd4f13beffef2aa03f88f9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c50de89a5a845668696dcb28c3badd6" is_pre_election: true
I20260129 02:58:45.705217 20441 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "368953c9102643b2a21f99af36b51905" candidate_uuid: "974b802141bd4f13beffef2aa03f88f9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "197b880fa39d432a9b472845876bfaf0" is_pre_election: true
I20260129 02:58:45.705428 20638 raft_consensus.cc:2749] T fd8311b779af4e1fa1402b2a02a602f2 P 197b880fa39d432a9b472845876bfaf0 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260129 02:58:45.692696 20729 leader_election.cc:290] T 913fc7d751ed4b7388d889aac30a5f9b P 197b880fa39d432a9b472845876bfaf0 [CANDIDATE]: Term 1 election: Requested vote from peers 974b802141bd4f13beffef2aa03f88f9 (127.18.211.2:34517), 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3:37735)
W20260129 02:58:45.706877 20479 leader_election.cc:343] T 368953c9102643b2a21f99af36b51905 P 974b802141bd4f13beffef2aa03f88f9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 197b880fa39d432a9b472845876bfaf0 (127.18.211.1:42027): Illegal state: must be running to vote when last-logged opid is not known
W20260129 02:58:45.708052 20480 leader_election.cc:343] T 368953c9102643b2a21f99af36b51905 P 974b802141bd4f13beffef2aa03f88f9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3:37735): Illegal state: must be running to vote when last-logged opid is not known
I20260129 02:58:45.708202 20739 leader_election.cc:290] T 4f8f3a38f6e24ba4857875181f447442 P 974b802141bd4f13beffef2aa03f88f9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 197b880fa39d432a9b472845876bfaf0 (127.18.211.1:42027), 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3:37735)
I20260129 02:58:45.708483 20480 leader_election.cc:304] T 368953c9102643b2a21f99af36b51905 P 974b802141bd4f13beffef2aa03f88f9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 974b802141bd4f13beffef2aa03f88f9; no voters: 197b880fa39d432a9b472845876bfaf0, 1c50de89a5a845668696dcb28c3badd6
I20260129 02:58:45.712409 20721 raft_consensus.cc:493] T b189f08e16a74a11980cc55d99359a0a P 197b880fa39d432a9b472845876bfaf0 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260129 02:58:45.712760 20734 raft_consensus.cc:2749] T 368953c9102643b2a21f99af36b51905 P 974b802141bd4f13beffef2aa03f88f9 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260129 02:58:45.712916 20721 raft_consensus.cc:515] T b189f08e16a74a11980cc55d99359a0a P 197b880fa39d432a9b472845876bfaf0 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:58:45.713310 20771 consensus_queue.cc:1048] T b189f08e16a74a11980cc55d99359a0a P 1c50de89a5a845668696dcb28c3badd6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260129 02:58:45.716308 20638 consensus_queue.cc:1048] T 4e420f90884f4796997b5ea41e49e411 P 197b880fa39d432a9b472845876bfaf0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260129 02:58:45.717290 20590 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "913fc7d751ed4b7388d889aac30a5f9b" candidate_uuid: "197b880fa39d432a9b472845876bfaf0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c50de89a5a845668696dcb28c3badd6"
I20260129 02:58:45.717893 20590 raft_consensus.cc:3060] T 913fc7d751ed4b7388d889aac30a5f9b P 1c50de89a5a845668696dcb28c3badd6 [term 0 FOLLOWER]: Advancing to term 1
I20260129 02:58:45.718938 20513 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b189f08e16a74a11980cc55d99359a0a" candidate_uuid: "197b880fa39d432a9b472845876bfaf0" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "974b802141bd4f13beffef2aa03f88f9" is_pre_election: true
I20260129 02:58:45.721504 20721 leader_election.cc:290] T b189f08e16a74a11980cc55d99359a0a P 197b880fa39d432a9b472845876bfaf0 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 974b802141bd4f13beffef2aa03f88f9 (127.18.211.2:34517), 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3:37735)
I20260129 02:58:45.722086 20588 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4f8f3a38f6e24ba4857875181f447442" candidate_uuid: "974b802141bd4f13beffef2aa03f88f9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c50de89a5a845668696dcb28c3badd6" is_pre_election: true
I20260129 02:58:45.722729 20588 raft_consensus.cc:2468] T 4f8f3a38f6e24ba4857875181f447442 P 1c50de89a5a845668696dcb28c3badd6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 974b802141bd4f13beffef2aa03f88f9 in term 0.
I20260129 02:58:45.722891 20586 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b189f08e16a74a11980cc55d99359a0a" candidate_uuid: "197b880fa39d432a9b472845876bfaf0" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c50de89a5a845668696dcb28c3badd6" is_pre_election: true
I20260129 02:58:45.725946 20480 leader_election.cc:304] T 4f8f3a38f6e24ba4857875181f447442 P 974b802141bd4f13beffef2aa03f88f9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1c50de89a5a845668696dcb28c3badd6, 974b802141bd4f13beffef2aa03f88f9; no voters:
I20260129 02:58:45.732231 20633 raft_consensus.cc:359] T 4f8f3a38f6e24ba4857875181f447442 P 197b880fa39d432a9b472845876bfaf0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:58:45.732936 20633 raft_consensus.cc:385] T 4f8f3a38f6e24ba4857875181f447442 P 197b880fa39d432a9b472845876bfaf0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260129 02:58:45.733193 20633 raft_consensus.cc:740] T 4f8f3a38f6e24ba4857875181f447442 P 197b880fa39d432a9b472845876bfaf0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 197b880fa39d432a9b472845876bfaf0, State: Initialized, Role: FOLLOWER
I20260129 02:58:45.734335 20633 consensus_queue.cc:260] T 4f8f3a38f6e24ba4857875181f447442 P 197b880fa39d432a9b472845876bfaf0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:58:45.732264 20739 raft_consensus.cc:2804] T 4f8f3a38f6e24ba4857875181f447442 P 974b802141bd4f13beffef2aa03f88f9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260129 02:58:45.740008 20739 raft_consensus.cc:493] T 4f8f3a38f6e24ba4857875181f447442 P 974b802141bd4f13beffef2aa03f88f9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260129 02:58:45.741956 20739 raft_consensus.cc:3060] T 4f8f3a38f6e24ba4857875181f447442 P 974b802141bd4f13beffef2aa03f88f9 [term 0 FOLLOWER]: Advancing to term 1
I20260129 02:58:45.742080 20633 ts_tablet_manager.cc:1434] T 4f8f3a38f6e24ba4857875181f447442 P 197b880fa39d432a9b472845876bfaf0: Time spent starting tablet: real 0.068s user 0.006s sys 0.002s
I20260129 02:58:45.743942 20441 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4f8f3a38f6e24ba4857875181f447442" candidate_uuid: "974b802141bd4f13beffef2aa03f88f9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "197b880fa39d432a9b472845876bfaf0" is_pre_election: true
I20260129 02:58:45.744488 20719 consensus_queue.cc:1048] T add8d4923d1a489e86882c3a25692b95 P 1c50de89a5a845668696dcb28c3badd6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260129 02:58:45.744804 20441 raft_consensus.cc:2468] T 4f8f3a38f6e24ba4857875181f447442 P 197b880fa39d432a9b472845876bfaf0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 974b802141bd4f13beffef2aa03f88f9 in term 0.
I20260129 02:58:45.745358 20405 leader_election.cc:304] T b189f08e16a74a11980cc55d99359a0a P 197b880fa39d432a9b472845876bfaf0 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 197b880fa39d432a9b472845876bfaf0; no voters: 1c50de89a5a845668696dcb28c3badd6, 974b802141bd4f13beffef2aa03f88f9
I20260129 02:58:45.743144 20633 tablet_bootstrap.cc:492] T c2633a6fc34c44579de348d208b0c01e P 197b880fa39d432a9b472845876bfaf0: Bootstrap starting.
I20260129 02:58:45.731745 20729 consensus_queue.cc:1048] T fcddab6bca684adf8ea032289a00f90e P 197b880fa39d432a9b472845876bfaf0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.006s
I20260129 02:58:45.759902 20721 consensus_queue.cc:1048] T a609ac02fae84725ab743f95f51d33ce P 197b880fa39d432a9b472845876bfaf0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260129 02:58:45.763717 20590 raft_consensus.cc:2468] T 913fc7d751ed4b7388d889aac30a5f9b P 1c50de89a5a845668696dcb28c3badd6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 197b880fa39d432a9b472845876bfaf0 in term 1.
I20260129 02:58:45.790197 20719 consensus_queue.cc:1048] T add8d4923d1a489e86882c3a25692b95 P 1c50de89a5a845668696dcb28c3badd6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.004s
I20260129 02:58:45.848634 20771 raft_consensus.cc:493] T dc2f4cb752cf4d59b5f6bc689f5d8104 P 1c50de89a5a845668696dcb28c3badd6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260129 02:58:45.849192 20771 raft_consensus.cc:515] T dc2f4cb752cf4d59b5f6bc689f5d8104 P 1c50de89a5a845668696dcb28c3badd6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:58:45.849630 20739 raft_consensus.cc:515] T 4f8f3a38f6e24ba4857875181f447442 P 974b802141bd4f13beffef2aa03f88f9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:58:45.788156 20729 consensus_queue.cc:1048] T 4e420f90884f4796997b5ea41e49e411 P 197b880fa39d432a9b472845876bfaf0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.039s
I20260129 02:58:45.839748 20515 raft_consensus.cc:2468] T 913fc7d751ed4b7388d889aac30a5f9b P 974b802141bd4f13beffef2aa03f88f9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 197b880fa39d432a9b472845876bfaf0 in term 1.
I20260129 02:58:45.870685 20631 tablet_bootstrap.cc:492] T 76f713c3331540d2a5b76164103e9eb1 P 1c50de89a5a845668696dcb28c3badd6: No bootstrap required, opened a new log
I20260129 02:58:45.871636 20631 ts_tablet_manager.cc:1403] T 76f713c3331540d2a5b76164103e9eb1 P 1c50de89a5a845668696dcb28c3badd6: Time spent bootstrapping tablet: real 0.318s user 0.026s sys 0.083s
I20260129 02:58:45.866927 20638 raft_consensus.cc:493] T dc2f4cb752cf4d59b5f6bc689f5d8104 P 197b880fa39d432a9b472845876bfaf0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260129 02:58:45.887210 20771 leader_election.cc:290] T dc2f4cb752cf4d59b5f6bc689f5d8104 P 1c50de89a5a845668696dcb28c3badd6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 197b880fa39d432a9b472845876bfaf0 (127.18.211.1:42027), 974b802141bd4f13beffef2aa03f88f9 (127.18.211.2:34517)
I20260129 02:58:45.894068 20405 leader_election.cc:304] T 913fc7d751ed4b7388d889aac30a5f9b P 197b880fa39d432a9b472845876bfaf0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 197b880fa39d432a9b472845876bfaf0, 1c50de89a5a845668696dcb28c3badd6; no voters:
I20260129 02:58:45.879536 20638 raft_consensus.cc:515] T dc2f4cb752cf4d59b5f6bc689f5d8104 P 197b880fa39d432a9b472845876bfaf0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:58:45.895143 20721 raft_consensus.cc:2804] T 913fc7d751ed4b7388d889aac30a5f9b P 197b880fa39d432a9b472845876bfaf0 [term 1 FOLLOWER]: Leader election won for term 1
I20260129 02:58:45.879259 20820 raft_consensus.cc:2749] T b189f08e16a74a11980cc55d99359a0a P 197b880fa39d432a9b472845876bfaf0 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260129 02:58:45.901801 20438 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "dc2f4cb752cf4d59b5f6bc689f5d8104" candidate_uuid: "1c50de89a5a845668696dcb28c3badd6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "197b880fa39d432a9b472845876bfaf0" is_pre_election: true
I20260129 02:58:45.902503 20438 raft_consensus.cc:2468] T dc2f4cb752cf4d59b5f6bc689f5d8104 P 197b880fa39d432a9b472845876bfaf0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1c50de89a5a845668696dcb28c3badd6 in term 0.
I20260129 02:58:45.893236 20740 raft_consensus.cc:493] T 7a9a69cd82724b1faed1d10aa091fc4a P 974b802141bd4f13beffef2aa03f88f9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260129 02:58:45.895915 20721 raft_consensus.cc:697] T 913fc7d751ed4b7388d889aac30a5f9b P 197b880fa39d432a9b472845876bfaf0 [term 1 LEADER]: Becoming Leader. State: Replica: 197b880fa39d432a9b472845876bfaf0, State: Running, Role: LEADER
I20260129 02:58:45.899690 20739 leader_election.cc:290] T 4f8f3a38f6e24ba4857875181f447442 P 974b802141bd4f13beffef2aa03f88f9 [CANDIDATE]: Term 1 election: Requested vote from peers 197b880fa39d432a9b472845876bfaf0 (127.18.211.1:42027), 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3:37735)
I20260129 02:58:45.906349 20740 raft_consensus.cc:515] T 7a9a69cd82724b1faed1d10aa091fc4a P 974b802141bd4f13beffef2aa03f88f9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:58:45.914093 20586 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4f8f3a38f6e24ba4857875181f447442" candidate_uuid: "974b802141bd4f13beffef2aa03f88f9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c50de89a5a845668696dcb28c3badd6"
I20260129 02:58:45.916973 20638 leader_election.cc:290] T dc2f4cb752cf4d59b5f6bc689f5d8104 P 197b880fa39d432a9b472845876bfaf0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 974b802141bd4f13beffef2aa03f88f9 (127.18.211.2:34517), 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3:37735)
I20260129 02:58:45.931565 20438 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4f8f3a38f6e24ba4857875181f447442" candidate_uuid: "974b802141bd4f13beffef2aa03f88f9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "197b880fa39d432a9b472845876bfaf0"
I20260129 02:58:45.925596 20586 raft_consensus.cc:3060] T 4f8f3a38f6e24ba4857875181f447442 P 1c50de89a5a845668696dcb28c3badd6 [term 0 FOLLOWER]: Advancing to term 1
I20260129 02:58:45.937325 20633 tablet_bootstrap.cc:654] T c2633a6fc34c44579de348d208b0c01e P 197b880fa39d432a9b472845876bfaf0: Neither blocks nor log segments found. Creating new log.
I20260129 02:58:45.938133 20740 leader_election.cc:290] T 7a9a69cd82724b1faed1d10aa091fc4a P 974b802141bd4f13beffef2aa03f88f9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 197b880fa39d432a9b472845876bfaf0 (127.18.211.1:42027), 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3:37735)
I20260129 02:58:45.946242 20553 leader_election.cc:304] T dc2f4cb752cf4d59b5f6bc689f5d8104 P 1c50de89a5a845668696dcb28c3badd6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 197b880fa39d432a9b472845876bfaf0, 1c50de89a5a845668696dcb28c3badd6; no voters:
I20260129 02:58:45.939532 20438 raft_consensus.cc:3060] T 4f8f3a38f6e24ba4857875181f447442 P 197b880fa39d432a9b472845876bfaf0 [term 0 FOLLOWER]: Advancing to term 1
I20260129 02:58:45.948204 20436 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7a9a69cd82724b1faed1d10aa091fc4a" candidate_uuid: "974b802141bd4f13beffef2aa03f88f9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "197b880fa39d432a9b472845876bfaf0" is_pre_election: true
W20260129 02:58:45.951012 20823 log.cc:927] Time spent T 2ac659d732314e1288849f08fe5e074e P 197b880fa39d432a9b472845876bfaf0: Append to log took a long time: real 0.051s user 0.000s sys 0.000s
I20260129 02:58:45.951620 20589 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7a9a69cd82724b1faed1d10aa091fc4a" candidate_uuid: "974b802141bd4f13beffef2aa03f88f9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c50de89a5a845668696dcb28c3badd6" is_pre_election: true
I20260129 02:58:45.936712 20719 raft_consensus.cc:493] T e4d18c35b0a7493e85d0a4b92956a48f P 1c50de89a5a845668696dcb28c3badd6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260129 02:58:45.959074 20479 leader_election.cc:343] T 7a9a69cd82724b1faed1d10aa091fc4a P 974b802141bd4f13beffef2aa03f88f9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 197b880fa39d432a9b472845876bfaf0 (127.18.211.1:42027): Illegal state: must be running to vote when last-logged opid is not known
W20260129 02:58:45.965451 20825 log.cc:927] Time spent T a609ac02fae84725ab743f95f51d33ce P 974b802141bd4f13beffef2aa03f88f9: Append to log took a long time: real 0.069s user 0.000s sys 0.000s
I20260129 02:58:45.966019 20511 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "dc2f4cb752cf4d59b5f6bc689f5d8104" candidate_uuid: "197b880fa39d432a9b472845876bfaf0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "974b802141bd4f13beffef2aa03f88f9" is_pre_election: true
I20260129 02:58:45.954209 20669 raft_consensus.cc:2804] T dc2f4cb752cf4d59b5f6bc689f5d8104 P 1c50de89a5a845668696dcb28c3badd6 [term 0 FOLLOWER]: Leader pre-election won for term 1
W20260129 02:58:45.957700 20824 log.cc:927] Time spent T fcddab6bca684adf8ea032289a00f90e P 974b802141bd4f13beffef2aa03f88f9: Append to log took a long time: real 0.064s user 0.000s sys 0.001s
I20260129 02:58:45.971805 20630 tablet_bootstrap.cc:654] T c2633a6fc34c44579de348d208b0c01e P 974b802141bd4f13beffef2aa03f88f9: Neither blocks nor log segments found. Creating new log.
I20260129 02:58:45.956683 20719 raft_consensus.cc:515] T e4d18c35b0a7493e85d0a4b92956a48f P 1c50de89a5a845668696dcb28c3badd6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:58:45.987686 20719 leader_election.cc:290] T e4d18c35b0a7493e85d0a4b92956a48f P 1c50de89a5a845668696dcb28c3badd6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 197b880fa39d432a9b472845876bfaf0 (127.18.211.1:42027), 974b802141bd4f13beffef2aa03f88f9 (127.18.211.2:34517)
I20260129 02:58:45.940363 20721 consensus_queue.cc:237] T 913fc7d751ed4b7388d889aac30a5f9b P 197b880fa39d432a9b472845876bfaf0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:58:45.973124 20669 raft_consensus.cc:493] T dc2f4cb752cf4d59b5f6bc689f5d8104 P 1c50de89a5a845668696dcb28c3badd6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260129 02:58:45.972734 20515 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "dc2f4cb752cf4d59b5f6bc689f5d8104" candidate_uuid: "1c50de89a5a845668696dcb28c3badd6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "974b802141bd4f13beffef2aa03f88f9" is_pre_election: true
I20260129 02:58:46.017478 20586 raft_consensus.cc:2468] T 4f8f3a38f6e24ba4857875181f447442 P 1c50de89a5a845668696dcb28c3badd6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 974b802141bd4f13beffef2aa03f88f9 in term 1.
I20260129 02:58:45.966693 20511 raft_consensus.cc:2468] T dc2f4cb752cf4d59b5f6bc689f5d8104 P 974b802141bd4f13beffef2aa03f88f9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 197b880fa39d432a9b472845876bfaf0 in term 0.
I20260129 02:58:45.996029 20589 raft_consensus.cc:3060] T e4d18c35b0a7493e85d0a4b92956a48f P 1c50de89a5a845668696dcb28c3badd6 [term 0 FOLLOWER]: Advancing to term 1
W20260129 02:58:45.976589 20480 leader_election.cc:343] T 7a9a69cd82724b1faed1d10aa091fc4a P 974b802141bd4f13beffef2aa03f88f9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3:37735): Illegal state: must be running to vote when last-logged opid is not known
I20260129 02:58:46.020923 20631 raft_consensus.cc:359] T 76f713c3331540d2a5b76164103e9eb1 P 1c50de89a5a845668696dcb28c3badd6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:58:46.024564 20631 raft_consensus.cc:385] T 76f713c3331540d2a5b76164103e9eb1 P 1c50de89a5a845668696dcb28c3badd6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260129 02:58:46.024822 20631 raft_consensus.cc:740] T 76f713c3331540d2a5b76164103e9eb1 P 1c50de89a5a845668696dcb28c3badd6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1c50de89a5a845668696dcb28c3badd6, State: Initialized, Role: FOLLOWER
I20260129 02:58:46.025323 20631 consensus_queue.cc:260] T 76f713c3331540d2a5b76164103e9eb1 P 1c50de89a5a845668696dcb28c3badd6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:58:46.026798 20631 ts_tablet_manager.cc:1434] T 76f713c3331540d2a5b76164103e9eb1 P 1c50de89a5a845668696dcb28c3badd6: Time spent starting tablet: real 0.151s user 0.004s sys 0.011s
I20260129 02:58:46.027565 20631 tablet_bootstrap.cc:492] T 368953c9102643b2a21f99af36b51905 P 1c50de89a5a845668696dcb28c3badd6: Bootstrap starting.
W20260129 02:58:46.024236 20553 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.211.1:42027, user_credentials={real_user=slave}} blocked reactor thread for 68040.3us
I20260129 02:58:46.030746 20480 leader_election.cc:304] T 7a9a69cd82724b1faed1d10aa091fc4a P 974b802141bd4f13beffef2aa03f88f9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 974b802141bd4f13beffef2aa03f88f9; no voters: 197b880fa39d432a9b472845876bfaf0, 1c50de89a5a845668696dcb28c3badd6
I20260129 02:58:45.963485 20827 consensus_queue.cc:1048] T a609ac02fae84725ab743f95f51d33ce P 197b880fa39d432a9b472845876bfaf0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.060s
I20260129 02:58:46.019584 20669 raft_consensus.cc:3060] T dc2f4cb752cf4d59b5f6bc689f5d8104 P 1c50de89a5a845668696dcb28c3badd6 [term 0 FOLLOWER]: Advancing to term 1
I20260129 02:58:46.042078 20740 raft_consensus.cc:2749] T 7a9a69cd82724b1faed1d10aa091fc4a P 974b802141bd4f13beffef2aa03f88f9 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20260129 02:58:46.046387 20480 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.211.3:37735, user_credentials={real_user=slave}} blocked reactor thread for 73973.5us
I20260129 02:58:46.046089 20820 consensus_queue.cc:1048] T febffe4a8b394368b69b1d164308fb9a P 197b880fa39d432a9b472845876bfaf0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.018s
I20260129 02:58:46.063184 20438 raft_consensus.cc:2468] T 4f8f3a38f6e24ba4857875181f447442 P 197b880fa39d432a9b472845876bfaf0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 974b802141bd4f13beffef2aa03f88f9 in term 1.
I20260129 02:58:46.085784 20479 leader_election.cc:304] T 4f8f3a38f6e24ba4857875181f447442 P 974b802141bd4f13beffef2aa03f88f9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 197b880fa39d432a9b472845876bfaf0, 974b802141bd4f13beffef2aa03f88f9; no voters:
I20260129 02:58:46.088065 20739 raft_consensus.cc:2804] T 4f8f3a38f6e24ba4857875181f447442 P 974b802141bd4f13beffef2aa03f88f9 [term 1 FOLLOWER]: Leader election won for term 1
I20260129 02:58:46.088605 20739 raft_consensus.cc:697] T 4f8f3a38f6e24ba4857875181f447442 P 974b802141bd4f13beffef2aa03f88f9 [term 1 LEADER]: Becoming Leader. State: Replica: 974b802141bd4f13beffef2aa03f88f9, State: Running, Role: LEADER
I20260129 02:58:46.089426 20739 consensus_queue.cc:237] T 4f8f3a38f6e24ba4857875181f447442 P 974b802141bd4f13beffef2aa03f88f9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:58:46.090230 20633 tablet_bootstrap.cc:492] T c2633a6fc34c44579de348d208b0c01e P 197b880fa39d432a9b472845876bfaf0: No bootstrap required, opened a new log
I20260129 02:58:46.090701 20633 ts_tablet_manager.cc:1403] T c2633a6fc34c44579de348d208b0c01e P 197b880fa39d432a9b472845876bfaf0: Time spent bootstrapping tablet: real 0.348s user 0.012s sys 0.037s
I20260129 02:58:46.093319 20829 raft_consensus.cc:493] T 75b0b6e82ddc4f92b1fe1d94f9a0b73b P 1c50de89a5a845668696dcb28c3badd6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260129 02:58:46.093820 20829 raft_consensus.cc:515] T 75b0b6e82ddc4f92b1fe1d94f9a0b73b P 1c50de89a5a845668696dcb28c3badd6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:58:46.096961 20829 leader_election.cc:290] T 75b0b6e82ddc4f92b1fe1d94f9a0b73b P 1c50de89a5a845668696dcb28c3badd6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 197b880fa39d432a9b472845876bfaf0 (127.18.211.1:42027), 974b802141bd4f13beffef2aa03f88f9 (127.18.211.2:34517)
I20260129 02:58:46.098446 20439 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e4d18c35b0a7493e85d0a4b92956a48f" candidate_uuid: "1c50de89a5a845668696dcb28c3badd6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "197b880fa39d432a9b472845876bfaf0" is_pre_election: true
I20260129 02:58:46.104367 20514 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e4d18c35b0a7493e85d0a4b92956a48f" candidate_uuid: "1c50de89a5a845668696dcb28c3badd6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "974b802141bd4f13beffef2aa03f88f9" is_pre_election: true
I20260129 02:58:46.107676 20638 consensus_queue.cc:1048] T febffe4a8b394368b69b1d164308fb9a P 197b880fa39d432a9b472845876bfaf0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260129 02:58:46.085929 20587 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "dc2f4cb752cf4d59b5f6bc689f5d8104" candidate_uuid: "197b880fa39d432a9b472845876bfaf0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c50de89a5a845668696dcb28c3badd6" is_pre_election: true
I20260129 02:58:46.091732 20771 consensus_queue.cc:1048] T fd8311b779af4e1fa1402b2a02a602f2 P 1c50de89a5a845668696dcb28c3badd6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.023s
I20260129 02:58:46.101657 20407 leader_election.cc:304] T dc2f4cb752cf4d59b5f6bc689f5d8104 P 197b880fa39d432a9b472845876bfaf0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 197b880fa39d432a9b472845876bfaf0, 974b802141bd4f13beffef2aa03f88f9; no voters:
I20260129 02:58:46.121737 20828 raft_consensus.cc:2804] T dc2f4cb752cf4d59b5f6bc689f5d8104 P 197b880fa39d432a9b472845876bfaf0 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260129 02:58:46.122282 20828 raft_consensus.cc:493] T dc2f4cb752cf4d59b5f6bc689f5d8104 P 197b880fa39d432a9b472845876bfaf0 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260129 02:58:46.122722 20828 raft_consensus.cc:3060] T dc2f4cb752cf4d59b5f6bc689f5d8104 P 197b880fa39d432a9b472845876bfaf0 [term 0 FOLLOWER]: Advancing to term 1
I20260129 02:58:46.128870 20633 raft_consensus.cc:359] T c2633a6fc34c44579de348d208b0c01e P 197b880fa39d432a9b472845876bfaf0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:58:46.129630 20633 raft_consensus.cc:385] T c2633a6fc34c44579de348d208b0c01e P 197b880fa39d432a9b472845876bfaf0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260129 02:58:46.129897 20633 raft_consensus.cc:740] T c2633a6fc34c44579de348d208b0c01e P 197b880fa39d432a9b472845876bfaf0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 197b880fa39d432a9b472845876bfaf0, State: Initialized, Role: FOLLOWER
I20260129 02:58:46.130475 20631 tablet_bootstrap.cc:654] T 368953c9102643b2a21f99af36b51905 P 1c50de89a5a845668696dcb28c3badd6: Neither blocks nor log segments found. Creating new log.
I20260129 02:58:46.140553 20633 consensus_queue.cc:260] T c2633a6fc34c44579de348d208b0c01e P 197b880fa39d432a9b472845876bfaf0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:58:46.138880 20336 catalog_manager.cc:5654] T 913fc7d751ed4b7388d889aac30a5f9b P 197b880fa39d432a9b472845876bfaf0 reported cstate change: term changed from 0 to 1, leader changed from <none> to 197b880fa39d432a9b472845876bfaf0 (127.18.211.1). New cstate: current_term: 1 leader_uuid: "197b880fa39d432a9b472845876bfaf0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } health_report { overall_health: UNKNOWN } } }
I20260129 02:58:46.128911 20440 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "75b0b6e82ddc4f92b1fe1d94f9a0b73b" candidate_uuid: "1c50de89a5a845668696dcb28c3badd6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "197b880fa39d432a9b472845876bfaf0" is_pre_election: true
I20260129 02:58:46.148469 20514 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "75b0b6e82ddc4f92b1fe1d94f9a0b73b" candidate_uuid: "1c50de89a5a845668696dcb28c3badd6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "974b802141bd4f13beffef2aa03f88f9" is_pre_election: true
I20260129 02:58:46.149149 20514 raft_consensus.cc:2468] T 75b0b6e82ddc4f92b1fe1d94f9a0b73b P 974b802141bd4f13beffef2aa03f88f9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1c50de89a5a845668696dcb28c3badd6 in term 0.
I20260129 02:58:46.149597 20669 raft_consensus.cc:515] T dc2f4cb752cf4d59b5f6bc689f5d8104 P 1c50de89a5a845668696dcb28c3badd6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:58:46.153347 20587 raft_consensus.cc:2393] T dc2f4cb752cf4d59b5f6bc689f5d8104 P 1c50de89a5a845668696dcb28c3badd6 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 197b880fa39d432a9b472845876bfaf0 in current term 1: Already voted for candidate 1c50de89a5a845668696dcb28c3badd6 in this term.
I20260129 02:58:46.154680 20669 leader_election.cc:290] T dc2f4cb752cf4d59b5f6bc689f5d8104 P 1c50de89a5a845668696dcb28c3badd6 [CANDIDATE]: Term 1 election: Requested vote from peers 197b880fa39d432a9b472845876bfaf0 (127.18.211.1:42027), 974b802141bd4f13beffef2aa03f88f9 (127.18.211.2:34517)
I20260129 02:58:46.146382 20771 consensus_queue.cc:1048] T fd8311b779af4e1fa1402b2a02a602f2 P 1c50de89a5a845668696dcb28c3badd6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.012s
I20260129 02:58:46.159019 20658 consensus_queue.cc:1048] T 1ce4ddf5b10e4ae8b193b13b2e5936ee P 197b880fa39d432a9b472845876bfaf0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260129 02:58:46.162094 20658 consensus_queue.cc:1048] T 1ce4ddf5b10e4ae8b193b13b2e5936ee P 197b880fa39d432a9b472845876bfaf0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260129 02:58:46.166170 20337 catalog_manager.cc:5654] T 4f8f3a38f6e24ba4857875181f447442 P 974b802141bd4f13beffef2aa03f88f9 reported cstate change: term changed from 0 to 1, leader changed from <none> to 974b802141bd4f13beffef2aa03f88f9 (127.18.211.2). New cstate: current_term: 1 leader_uuid: "974b802141bd4f13beffef2aa03f88f9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } health_report { overall_health: UNKNOWN } } }
I20260129 02:58:46.172325 20633 ts_tablet_manager.cc:1434] T c2633a6fc34c44579de348d208b0c01e P 197b880fa39d432a9b472845876bfaf0: Time spent starting tablet: real 0.081s user 0.008s sys 0.000s
I20260129 02:58:46.174113 20633 tablet_bootstrap.cc:492] T 505580c93c694bbb8ea96fdf9b17ab46 P 197b880fa39d432a9b472845876bfaf0: Bootstrap starting.
I20260129 02:58:46.176680 20553 leader_election.cc:304] T e4d18c35b0a7493e85d0a4b92956a48f P 1c50de89a5a845668696dcb28c3badd6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1c50de89a5a845668696dcb28c3badd6; no voters: 197b880fa39d432a9b472845876bfaf0, 974b802141bd4f13beffef2aa03f88f9
I20260129 02:58:46.179719 20630 tablet_bootstrap.cc:492] T c2633a6fc34c44579de348d208b0c01e P 974b802141bd4f13beffef2aa03f88f9: No bootstrap required, opened a new log
I20260129 02:58:46.180212 20630 ts_tablet_manager.cc:1403] T c2633a6fc34c44579de348d208b0c01e P 974b802141bd4f13beffef2aa03f88f9: Time spent bootstrapping tablet: real 0.505s user 0.027s sys 0.020s
I20260129 02:58:46.181255 20515 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "dc2f4cb752cf4d59b5f6bc689f5d8104" candidate_uuid: "1c50de89a5a845668696dcb28c3badd6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "974b802141bd4f13beffef2aa03f88f9"
I20260129 02:58:46.181917 20515 raft_consensus.cc:3060] T dc2f4cb752cf4d59b5f6bc689f5d8104 P 974b802141bd4f13beffef2aa03f88f9 [term 0 FOLLOWER]: Advancing to term 1
I20260129 02:58:46.194018 20719 raft_consensus.cc:2749] T e4d18c35b0a7493e85d0a4b92956a48f P 1c50de89a5a845668696dcb28c3badd6 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20260129 02:58:46.184255 20553 leader_election.cc:343] T 75b0b6e82ddc4f92b1fe1d94f9a0b73b P 1c50de89a5a845668696dcb28c3badd6 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 197b880fa39d432a9b472845876bfaf0 (127.18.211.1:42027): Illegal state: must be running to vote when last-logged opid is not known
I20260129 02:58:46.218500 20590 raft_consensus.cc:3060] T 76f713c3331540d2a5b76164103e9eb1 P 1c50de89a5a845668696dcb28c3badd6 [term 0 FOLLOWER]: Advancing to term 1
I20260129 02:58:46.220644 20828 raft_consensus.cc:515] T dc2f4cb752cf4d59b5f6bc689f5d8104 P 197b880fa39d432a9b472845876bfaf0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
W20260129 02:58:46.221621 20838 log.cc:927] Time spent T a609ac02fae84725ab743f95f51d33ce P 1c50de89a5a845668696dcb28c3badd6: Append to log took a long time: real 0.052s user 0.000s sys 0.001s
I20260129 02:58:46.229893 20515 raft_consensus.cc:2468] T dc2f4cb752cf4d59b5f6bc689f5d8104 P 974b802141bd4f13beffef2aa03f88f9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1c50de89a5a845668696dcb28c3badd6 in term 1.
I20260129 02:58:46.230605 20739 consensus_queue.cc:1048] T 76f713c3331540d2a5b76164103e9eb1 P 974b802141bd4f13beffef2aa03f88f9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260129 02:58:46.231384 20740 consensus_queue.cc:1048] T 76f713c3331540d2a5b76164103e9eb1 P 974b802141bd4f13beffef2aa03f88f9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260129 02:58:46.233196 20435 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "dc2f4cb752cf4d59b5f6bc689f5d8104" candidate_uuid: "1c50de89a5a845668696dcb28c3badd6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "197b880fa39d432a9b472845876bfaf0"
I20260129 02:58:46.234010 20435 raft_consensus.cc:2393] T dc2f4cb752cf4d59b5f6bc689f5d8104 P 197b880fa39d432a9b472845876bfaf0 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 1c50de89a5a845668696dcb28c3badd6 in current term 1: Already voted for candidate 197b880fa39d432a9b472845876bfaf0 in this term.
I20260129 02:58:46.238189 20631 tablet_bootstrap.cc:492] T 368953c9102643b2a21f99af36b51905 P 1c50de89a5a845668696dcb28c3badd6: No bootstrap required, opened a new log
I20260129 02:58:46.240649 20828 leader_election.cc:290] T dc2f4cb752cf4d59b5f6bc689f5d8104 P 197b880fa39d432a9b472845876bfaf0 [CANDIDATE]: Term 1 election: Requested vote from peers 974b802141bd4f13beffef2aa03f88f9 (127.18.211.2:34517), 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3:37735)
I20260129 02:58:46.239148 20631 ts_tablet_manager.cc:1403] T 368953c9102643b2a21f99af36b51905 P 1c50de89a5a845668696dcb28c3badd6: Time spent bootstrapping tablet: real 0.212s user 0.013s sys 0.005s
I20260129 02:58:46.245766 20630 raft_consensus.cc:359] T c2633a6fc34c44579de348d208b0c01e P 974b802141bd4f13beffef2aa03f88f9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:58:46.246519 20630 raft_consensus.cc:385] T c2633a6fc34c44579de348d208b0c01e P 974b802141bd4f13beffef2aa03f88f9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260129 02:58:46.246789 20630 raft_consensus.cc:740] T c2633a6fc34c44579de348d208b0c01e P 974b802141bd4f13beffef2aa03f88f9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 974b802141bd4f13beffef2aa03f88f9, State: Initialized, Role: FOLLOWER
I20260129 02:58:46.250070 20633 tablet_bootstrap.cc:654] T 505580c93c694bbb8ea96fdf9b17ab46 P 197b880fa39d432a9b472845876bfaf0: Neither blocks nor log segments found. Creating new log.
I20260129 02:58:46.249855 20515 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "dc2f4cb752cf4d59b5f6bc689f5d8104" candidate_uuid: "197b880fa39d432a9b472845876bfaf0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "974b802141bd4f13beffef2aa03f88f9"
I20260129 02:58:46.250746 20515 raft_consensus.cc:2393] T dc2f4cb752cf4d59b5f6bc689f5d8104 P 974b802141bd4f13beffef2aa03f88f9 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 197b880fa39d432a9b472845876bfaf0 in current term 1: Already voted for candidate 1c50de89a5a845668696dcb28c3badd6 in this term.
I20260129 02:58:46.257373 20556 leader_election.cc:304] T 75b0b6e82ddc4f92b1fe1d94f9a0b73b P 1c50de89a5a845668696dcb28c3badd6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1c50de89a5a845668696dcb28c3badd6, 974b802141bd4f13beffef2aa03f88f9; no voters: 197b880fa39d432a9b472845876bfaf0
I20260129 02:58:46.247443 20630 consensus_queue.cc:260] T c2633a6fc34c44579de348d208b0c01e P 974b802141bd4f13beffef2aa03f88f9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:58:46.259073 20631 raft_consensus.cc:359] T 368953c9102643b2a21f99af36b51905 P 1c50de89a5a845668696dcb28c3badd6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:58:46.259815 20631 raft_consensus.cc:385] T 368953c9102643b2a21f99af36b51905 P 1c50de89a5a845668696dcb28c3badd6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260129 02:58:46.260131 20631 raft_consensus.cc:740] T 368953c9102643b2a21f99af36b51905 P 1c50de89a5a845668696dcb28c3badd6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1c50de89a5a845668696dcb28c3badd6, State: Initialized, Role: FOLLOWER
I20260129 02:58:46.263881 20631 consensus_queue.cc:260] T 368953c9102643b2a21f99af36b51905 P 1c50de89a5a845668696dcb28c3badd6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:58:46.267469 20669 raft_consensus.cc:2804] T 75b0b6e82ddc4f92b1fe1d94f9a0b73b P 1c50de89a5a845668696dcb28c3badd6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260129 02:58:46.271313 20669 raft_consensus.cc:493] T 75b0b6e82ddc4f92b1fe1d94f9a0b73b P 1c50de89a5a845668696dcb28c3badd6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260129 02:58:46.272348 20669 raft_consensus.cc:3060] T 75b0b6e82ddc4f92b1fe1d94f9a0b73b P 1c50de89a5a845668696dcb28c3badd6 [term 0 FOLLOWER]: Advancing to term 1
W20260129 02:58:46.268698 20556 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.211.2:34517, user_credentials={real_user=slave}} blocked reactor thread for 81942us
I20260129 02:58:46.279459 20587 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "dc2f4cb752cf4d59b5f6bc689f5d8104" candidate_uuid: "197b880fa39d432a9b472845876bfaf0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c50de89a5a845668696dcb28c3badd6"
I20260129 02:58:46.280323 20587 raft_consensus.cc:2393] T dc2f4cb752cf4d59b5f6bc689f5d8104 P 1c50de89a5a845668696dcb28c3badd6 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 197b880fa39d432a9b472845876bfaf0 in current term 1: Already voted for candidate 1c50de89a5a845668696dcb28c3badd6 in this term.
I20260129 02:58:46.286300 20631 ts_tablet_manager.cc:1434] T 368953c9102643b2a21f99af36b51905 P 1c50de89a5a845668696dcb28c3badd6: Time spent starting tablet: real 0.044s user 0.007s sys 0.000s
I20260129 02:58:46.288692 20630 ts_tablet_manager.cc:1434] T c2633a6fc34c44579de348d208b0c01e P 974b802141bd4f13beffef2aa03f88f9: Time spent starting tablet: real 0.108s user 0.008s sys 0.000s
I20260129 02:58:46.290187 20630 tablet_bootstrap.cc:492] T 4408006eb5174de1896b037cf3f18457 P 974b802141bd4f13beffef2aa03f88f9: Bootstrap starting.
I20260129 02:58:46.302387 20669 raft_consensus.cc:515] T 75b0b6e82ddc4f92b1fe1d94f9a0b73b P 1c50de89a5a845668696dcb28c3badd6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:58:46.304451 20669 leader_election.cc:290] T 75b0b6e82ddc4f92b1fe1d94f9a0b73b P 1c50de89a5a845668696dcb28c3badd6 [CANDIDATE]: Term 1 election: Requested vote from peers 197b880fa39d432a9b472845876bfaf0 (127.18.211.1:42027), 974b802141bd4f13beffef2aa03f88f9 (127.18.211.2:34517)
I20260129 02:58:46.296726 20631 tablet_bootstrap.cc:492] T 4408006eb5174de1896b037cf3f18457 P 1c50de89a5a845668696dcb28c3badd6: Bootstrap starting.
I20260129 02:58:46.308604 20556 leader_election.cc:304] T dc2f4cb752cf4d59b5f6bc689f5d8104 P 1c50de89a5a845668696dcb28c3badd6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1c50de89a5a845668696dcb28c3badd6, 974b802141bd4f13beffef2aa03f88f9; no voters: 197b880fa39d432a9b472845876bfaf0
I20260129 02:58:46.312041 20771 raft_consensus.cc:2804] T dc2f4cb752cf4d59b5f6bc689f5d8104 P 1c50de89a5a845668696dcb28c3badd6 [term 1 FOLLOWER]: Leader election won for term 1
I20260129 02:58:46.312764 20440 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "75b0b6e82ddc4f92b1fe1d94f9a0b73b" candidate_uuid: "1c50de89a5a845668696dcb28c3badd6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "197b880fa39d432a9b472845876bfaf0"
I20260129 02:58:46.313042 20771 raft_consensus.cc:697] T dc2f4cb752cf4d59b5f6bc689f5d8104 P 1c50de89a5a845668696dcb28c3badd6 [term 1 LEADER]: Becoming Leader. State: Replica: 1c50de89a5a845668696dcb28c3badd6, State: Running, Role: LEADER
I20260129 02:58:46.314127 20771 consensus_queue.cc:237] T dc2f4cb752cf4d59b5f6bc689f5d8104 P 1c50de89a5a845668696dcb28c3badd6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:58:46.319043 20405 leader_election.cc:304] T dc2f4cb752cf4d59b5f6bc689f5d8104 P 197b880fa39d432a9b472845876bfaf0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 197b880fa39d432a9b472845876bfaf0; no voters: 1c50de89a5a845668696dcb28c3badd6, 974b802141bd4f13beffef2aa03f88f9
I20260129 02:58:46.320076 20828 raft_consensus.cc:2749] T dc2f4cb752cf4d59b5f6bc689f5d8104 P 197b880fa39d432a9b472845876bfaf0 [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
W20260129 02:58:46.320380 20553 leader_election.cc:343] T 75b0b6e82ddc4f92b1fe1d94f9a0b73b P 1c50de89a5a845668696dcb28c3badd6 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 197b880fa39d432a9b472845876bfaf0 (127.18.211.1:42027): Illegal state: must be running to vote when last-logged opid is not known
I20260129 02:58:46.332396 20512 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "75b0b6e82ddc4f92b1fe1d94f9a0b73b" candidate_uuid: "1c50de89a5a845668696dcb28c3badd6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "974b802141bd4f13beffef2aa03f88f9"
I20260129 02:58:46.333076 20512 raft_consensus.cc:3060] T 75b0b6e82ddc4f92b1fe1d94f9a0b73b P 974b802141bd4f13beffef2aa03f88f9 [term 0 FOLLOWER]: Advancing to term 1
I20260129 02:58:46.345438 20630 tablet_bootstrap.cc:654] T 4408006eb5174de1896b037cf3f18457 P 974b802141bd4f13beffef2aa03f88f9: Neither blocks nor log segments found. Creating new log.
I20260129 02:58:46.362108 20633 tablet_bootstrap.cc:492] T 505580c93c694bbb8ea96fdf9b17ab46 P 197b880fa39d432a9b472845876bfaf0: No bootstrap required, opened a new log
I20260129 02:58:46.362711 20633 ts_tablet_manager.cc:1403] T 505580c93c694bbb8ea96fdf9b17ab46 P 197b880fa39d432a9b472845876bfaf0: Time spent bootstrapping tablet: real 0.190s user 0.004s sys 0.006s
I20260129 02:58:46.350826 20512 raft_consensus.cc:2468] T 75b0b6e82ddc4f92b1fe1d94f9a0b73b P 974b802141bd4f13beffef2aa03f88f9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1c50de89a5a845668696dcb28c3badd6 in term 1.
I20260129 02:58:46.365749 20633 raft_consensus.cc:359] T 505580c93c694bbb8ea96fdf9b17ab46 P 197b880fa39d432a9b472845876bfaf0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:58:46.366674 20633 raft_consensus.cc:385] T 505580c93c694bbb8ea96fdf9b17ab46 P 197b880fa39d432a9b472845876bfaf0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260129 02:58:46.367046 20633 raft_consensus.cc:740] T 505580c93c694bbb8ea96fdf9b17ab46 P 197b880fa39d432a9b472845876bfaf0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 197b880fa39d432a9b472845876bfaf0, State: Initialized, Role: FOLLOWER
I20260129 02:58:46.368140 20630 tablet_bootstrap.cc:492] T 4408006eb5174de1896b037cf3f18457 P 974b802141bd4f13beffef2aa03f88f9: No bootstrap required, opened a new log
I20260129 02:58:46.368628 20630 ts_tablet_manager.cc:1403] T 4408006eb5174de1896b037cf3f18457 P 974b802141bd4f13beffef2aa03f88f9: Time spent bootstrapping tablet: real 0.079s user 0.011s sys 0.000s
I20260129 02:58:46.371101 20630 raft_consensus.cc:359] T 4408006eb5174de1896b037cf3f18457 P 974b802141bd4f13beffef2aa03f88f9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:58:46.371806 20630 raft_consensus.cc:385] T 4408006eb5174de1896b037cf3f18457 P 974b802141bd4f13beffef2aa03f88f9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260129 02:58:46.372102 20630 raft_consensus.cc:740] T 4408006eb5174de1896b037cf3f18457 P 974b802141bd4f13beffef2aa03f88f9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 974b802141bd4f13beffef2aa03f88f9, State: Initialized, Role: FOLLOWER
I20260129 02:58:46.372215 20633 consensus_queue.cc:260] T 505580c93c694bbb8ea96fdf9b17ab46 P 197b880fa39d432a9b472845876bfaf0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:58:46.372822 20630 consensus_queue.cc:260] T 4408006eb5174de1896b037cf3f18457 P 974b802141bd4f13beffef2aa03f88f9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:58:46.376099 20631 tablet_bootstrap.cc:654] T 4408006eb5174de1896b037cf3f18457 P 1c50de89a5a845668696dcb28c3badd6: Neither blocks nor log segments found. Creating new log.
I20260129 02:58:46.380589 20633 ts_tablet_manager.cc:1434] T 505580c93c694bbb8ea96fdf9b17ab46 P 197b880fa39d432a9b472845876bfaf0: Time spent starting tablet: real 0.018s user 0.009s sys 0.000s
I20260129 02:58:46.381644 20633 tablet_bootstrap.cc:492] T 75b0b6e82ddc4f92b1fe1d94f9a0b73b P 197b880fa39d432a9b472845876bfaf0: Bootstrap starting.
I20260129 02:58:46.386958 20556 leader_election.cc:304] T 75b0b6e82ddc4f92b1fe1d94f9a0b73b P 1c50de89a5a845668696dcb28c3badd6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1c50de89a5a845668696dcb28c3badd6, 974b802141bd4f13beffef2aa03f88f9; no voters: 197b880fa39d432a9b472845876bfaf0
I20260129 02:58:46.388540 20630 ts_tablet_manager.cc:1434] T 4408006eb5174de1896b037cf3f18457 P 974b802141bd4f13beffef2aa03f88f9: Time spent starting tablet: real 0.020s user 0.007s sys 0.000s
I20260129 02:58:46.390179 20840 raft_consensus.cc:2804] T 75b0b6e82ddc4f92b1fe1d94f9a0b73b P 1c50de89a5a845668696dcb28c3badd6 [term 1 FOLLOWER]: Leader election won for term 1
I20260129 02:58:46.390738 20840 raft_consensus.cc:697] T 75b0b6e82ddc4f92b1fe1d94f9a0b73b P 1c50de89a5a845668696dcb28c3badd6 [term 1 LEADER]: Becoming Leader. State: Replica: 1c50de89a5a845668696dcb28c3badd6, State: Running, Role: LEADER
I20260129 02:58:46.391690 20633 tablet_bootstrap.cc:654] T 75b0b6e82ddc4f92b1fe1d94f9a0b73b P 197b880fa39d432a9b472845876bfaf0: Neither blocks nor log segments found. Creating new log.
I20260129 02:58:46.391623 20840 consensus_queue.cc:237] T 75b0b6e82ddc4f92b1fe1d94f9a0b73b P 1c50de89a5a845668696dcb28c3badd6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:58:46.397429 20719 raft_consensus.cc:493] T 368953c9102643b2a21f99af36b51905 P 1c50de89a5a845668696dcb28c3badd6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260129 02:58:46.397955 20719 raft_consensus.cc:515] T 368953c9102643b2a21f99af36b51905 P 1c50de89a5a845668696dcb28c3badd6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:58:46.398455 20630 tablet_bootstrap.cc:492] T 96609ac68bdf4993a12e1c23355ca013 P 974b802141bd4f13beffef2aa03f88f9: Bootstrap starting.
I20260129 02:58:46.399902 20719 leader_election.cc:290] T 368953c9102643b2a21f99af36b51905 P 1c50de89a5a845668696dcb28c3badd6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 197b880fa39d432a9b472845876bfaf0 (127.18.211.1:42027), 974b802141bd4f13beffef2aa03f88f9 (127.18.211.2:34517)
I20260129 02:58:46.401477 20440 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "368953c9102643b2a21f99af36b51905" candidate_uuid: "1c50de89a5a845668696dcb28c3badd6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "197b880fa39d432a9b472845876bfaf0" is_pre_election: true
I20260129 02:58:46.402737 20512 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "368953c9102643b2a21f99af36b51905" candidate_uuid: "1c50de89a5a845668696dcb28c3badd6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "974b802141bd4f13beffef2aa03f88f9" is_pre_election: true
I20260129 02:58:46.403355 20512 raft_consensus.cc:2468] T 368953c9102643b2a21f99af36b51905 P 974b802141bd4f13beffef2aa03f88f9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1c50de89a5a845668696dcb28c3badd6 in term 0.
W20260129 02:58:46.403386 20553 leader_election.cc:343] T 368953c9102643b2a21f99af36b51905 P 1c50de89a5a845668696dcb28c3badd6 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 197b880fa39d432a9b472845876bfaf0 (127.18.211.1:42027): Illegal state: must be running to vote when last-logged opid is not known
I20260129 02:58:46.404949 20556 leader_election.cc:304] T 368953c9102643b2a21f99af36b51905 P 1c50de89a5a845668696dcb28c3badd6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1c50de89a5a845668696dcb28c3badd6, 974b802141bd4f13beffef2aa03f88f9; no voters: 197b880fa39d432a9b472845876bfaf0
I20260129 02:58:46.405861 20719 raft_consensus.cc:2804] T 368953c9102643b2a21f99af36b51905 P 1c50de89a5a845668696dcb28c3badd6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260129 02:58:46.406227 20719 raft_consensus.cc:493] T 368953c9102643b2a21f99af36b51905 P 1c50de89a5a845668696dcb28c3badd6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260129 02:58:46.406625 20719 raft_consensus.cc:3060] T 368953c9102643b2a21f99af36b51905 P 1c50de89a5a845668696dcb28c3badd6 [term 0 FOLLOWER]: Advancing to term 1
I20260129 02:58:46.422951 20719 raft_consensus.cc:515] T 368953c9102643b2a21f99af36b51905 P 1c50de89a5a845668696dcb28c3badd6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:58:46.426803 20440 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "368953c9102643b2a21f99af36b51905" candidate_uuid: "1c50de89a5a845668696dcb28c3badd6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "197b880fa39d432a9b472845876bfaf0"
W20260129 02:58:46.428642 20553 leader_election.cc:343] T 368953c9102643b2a21f99af36b51905 P 1c50de89a5a845668696dcb28c3badd6 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 197b880fa39d432a9b472845876bfaf0 (127.18.211.1:42027): Illegal state: must be running to vote when last-logged opid is not known
I20260129 02:58:46.432636 20630 tablet_bootstrap.cc:654] T 96609ac68bdf4993a12e1c23355ca013 P 974b802141bd4f13beffef2aa03f88f9: Neither blocks nor log segments found. Creating new log.
I20260129 02:58:46.432134 20719 leader_election.cc:290] T 368953c9102643b2a21f99af36b51905 P 1c50de89a5a845668696dcb28c3badd6 [CANDIDATE]: Term 1 election: Requested vote from peers 197b880fa39d432a9b472845876bfaf0 (127.18.211.1:42027), 974b802141bd4f13beffef2aa03f88f9 (127.18.211.2:34517)
I20260129 02:58:46.433444 20512 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "368953c9102643b2a21f99af36b51905" candidate_uuid: "1c50de89a5a845668696dcb28c3badd6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "974b802141bd4f13beffef2aa03f88f9"
I20260129 02:58:46.434144 20512 raft_consensus.cc:3060] T 368953c9102643b2a21f99af36b51905 P 974b802141bd4f13beffef2aa03f88f9 [term 0 FOLLOWER]: Advancing to term 1
I20260129 02:58:46.435225 20633 tablet_bootstrap.cc:492] T 75b0b6e82ddc4f92b1fe1d94f9a0b73b P 197b880fa39d432a9b472845876bfaf0: No bootstrap required, opened a new log
I20260129 02:58:46.435623 20633 ts_tablet_manager.cc:1403] T 75b0b6e82ddc4f92b1fe1d94f9a0b73b P 197b880fa39d432a9b472845876bfaf0: Time spent bootstrapping tablet: real 0.054s user 0.012s sys 0.000s
I20260129 02:58:46.440426 20631 tablet_bootstrap.cc:492] T 4408006eb5174de1896b037cf3f18457 P 1c50de89a5a845668696dcb28c3badd6: No bootstrap required, opened a new log
I20260129 02:58:46.440958 20631 ts_tablet_manager.cc:1403] T 4408006eb5174de1896b037cf3f18457 P 1c50de89a5a845668696dcb28c3badd6: Time spent bootstrapping tablet: real 0.144s user 0.023s sys 0.022s
I20260129 02:58:46.441648 20512 raft_consensus.cc:2468] T 368953c9102643b2a21f99af36b51905 P 974b802141bd4f13beffef2aa03f88f9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1c50de89a5a845668696dcb28c3badd6 in term 1.
I20260129 02:58:46.442644 20556 leader_election.cc:304] T 368953c9102643b2a21f99af36b51905 P 1c50de89a5a845668696dcb28c3badd6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1c50de89a5a845668696dcb28c3badd6, 974b802141bd4f13beffef2aa03f88f9; no voters: 197b880fa39d432a9b472845876bfaf0
I20260129 02:58:46.442487 20633 raft_consensus.cc:359] T 75b0b6e82ddc4f92b1fe1d94f9a0b73b P 197b880fa39d432a9b472845876bfaf0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:58:46.443145 20633 raft_consensus.cc:385] T 75b0b6e82ddc4f92b1fe1d94f9a0b73b P 197b880fa39d432a9b472845876bfaf0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260129 02:58:46.443413 20633 raft_consensus.cc:740] T 75b0b6e82ddc4f92b1fe1d94f9a0b73b P 197b880fa39d432a9b472845876bfaf0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 197b880fa39d432a9b472845876bfaf0, State: Initialized, Role: FOLLOWER
I20260129 02:58:46.443447 20719 raft_consensus.cc:2804] T 368953c9102643b2a21f99af36b51905 P 1c50de89a5a845668696dcb28c3badd6 [term 1 FOLLOWER]: Leader election won for term 1
I20260129 02:58:46.443425 20631 raft_consensus.cc:359] T 4408006eb5174de1896b037cf3f18457 P 1c50de89a5a845668696dcb28c3badd6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:58:46.444397 20631 raft_consensus.cc:385] T 4408006eb5174de1896b037cf3f18457 P 1c50de89a5a845668696dcb28c3badd6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260129 02:58:46.444430 20719 raft_consensus.cc:697] T 368953c9102643b2a21f99af36b51905 P 1c50de89a5a845668696dcb28c3badd6 [term 1 LEADER]: Becoming Leader. State: Replica: 1c50de89a5a845668696dcb28c3badd6, State: Running, Role: LEADER
I20260129 02:58:46.444275 20633 consensus_queue.cc:260] T 75b0b6e82ddc4f92b1fe1d94f9a0b73b P 197b880fa39d432a9b472845876bfaf0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:58:46.444810 20631 raft_consensus.cc:740] T 4408006eb5174de1896b037cf3f18457 P 1c50de89a5a845668696dcb28c3badd6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1c50de89a5a845668696dcb28c3badd6, State: Initialized, Role: FOLLOWER
I20260129 02:58:46.445453 20631 consensus_queue.cc:260] T 4408006eb5174de1896b037cf3f18457 P 1c50de89a5a845668696dcb28c3badd6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:58:46.446712 20719 consensus_queue.cc:237] T 368953c9102643b2a21f99af36b51905 P 1c50de89a5a845668696dcb28c3badd6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:58:46.447793 20631 ts_tablet_manager.cc:1434] T 4408006eb5174de1896b037cf3f18457 P 1c50de89a5a845668696dcb28c3badd6: Time spent starting tablet: real 0.006s user 0.003s sys 0.000s
I20260129 02:58:46.448750 20631 tablet_bootstrap.cc:492] T 96609ac68bdf4993a12e1c23355ca013 P 1c50de89a5a845668696dcb28c3badd6: Bootstrap starting.
I20260129 02:58:46.448284 20337 catalog_manager.cc:5654] T dc2f4cb752cf4d59b5f6bc689f5d8104 P 1c50de89a5a845668696dcb28c3badd6 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3). New cstate: current_term: 1 leader_uuid: "1c50de89a5a845668696dcb28c3badd6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } health_report { overall_health: HEALTHY } } }
I20260129 02:58:46.452239 20633 ts_tablet_manager.cc:1434] T 75b0b6e82ddc4f92b1fe1d94f9a0b73b P 197b880fa39d432a9b472845876bfaf0: Time spent starting tablet: real 0.016s user 0.007s sys 0.000s
I20260129 02:58:46.453197 20633 tablet_bootstrap.cc:492] T 4408006eb5174de1896b037cf3f18457 P 197b880fa39d432a9b472845876bfaf0: Bootstrap starting.
I20260129 02:58:46.452718 20337 catalog_manager.cc:5654] T 75b0b6e82ddc4f92b1fe1d94f9a0b73b P 1c50de89a5a845668696dcb28c3badd6 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3). New cstate: current_term: 1 leader_uuid: "1c50de89a5a845668696dcb28c3badd6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } health_report { overall_health: HEALTHY } } }
I20260129 02:58:46.464155 20631 tablet_bootstrap.cc:654] T 96609ac68bdf4993a12e1c23355ca013 P 1c50de89a5a845668696dcb28c3badd6: Neither blocks nor log segments found. Creating new log.
I20260129 02:58:46.472945 20633 tablet_bootstrap.cc:654] T 4408006eb5174de1896b037cf3f18457 P 197b880fa39d432a9b472845876bfaf0: Neither blocks nor log segments found. Creating new log.
I20260129 02:58:46.481796 20630 tablet_bootstrap.cc:492] T 96609ac68bdf4993a12e1c23355ca013 P 974b802141bd4f13beffef2aa03f88f9: No bootstrap required, opened a new log
I20260129 02:58:46.482402 20630 ts_tablet_manager.cc:1403] T 96609ac68bdf4993a12e1c23355ca013 P 974b802141bd4f13beffef2aa03f88f9: Time spent bootstrapping tablet: real 0.084s user 0.023s sys 0.000s
I20260129 02:58:46.485286 20631 tablet_bootstrap.cc:492] T 96609ac68bdf4993a12e1c23355ca013 P 1c50de89a5a845668696dcb28c3badd6: No bootstrap required, opened a new log
I20260129 02:58:46.485683 20630 raft_consensus.cc:359] T 96609ac68bdf4993a12e1c23355ca013 P 974b802141bd4f13beffef2aa03f88f9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:58:46.486706 20630 raft_consensus.cc:385] T 96609ac68bdf4993a12e1c23355ca013 P 974b802141bd4f13beffef2aa03f88f9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260129 02:58:46.487047 20630 raft_consensus.cc:740] T 96609ac68bdf4993a12e1c23355ca013 P 974b802141bd4f13beffef2aa03f88f9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 974b802141bd4f13beffef2aa03f88f9, State: Initialized, Role: FOLLOWER
I20260129 02:58:46.485785 20631 ts_tablet_manager.cc:1403] T 96609ac68bdf4993a12e1c23355ca013 P 1c50de89a5a845668696dcb28c3badd6: Time spent bootstrapping tablet: real 0.037s user 0.008s sys 0.003s
I20260129 02:58:46.493252 20630 consensus_queue.cc:260] T 96609ac68bdf4993a12e1c23355ca013 P 974b802141bd4f13beffef2aa03f88f9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:58:46.496582 20630 ts_tablet_manager.cc:1434] T 96609ac68bdf4993a12e1c23355ca013 P 974b802141bd4f13beffef2aa03f88f9: Time spent starting tablet: real 0.014s user 0.010s sys 0.000s
I20260129 02:58:46.497268 20739 raft_consensus.cc:493] T 96609ac68bdf4993a12e1c23355ca013 P 974b802141bd4f13beffef2aa03f88f9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260129 02:58:46.497748 20739 raft_consensus.cc:515] T 96609ac68bdf4993a12e1c23355ca013 P 974b802141bd4f13beffef2aa03f88f9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:58:46.498375 20630 tablet_bootstrap.cc:492] T 505580c93c694bbb8ea96fdf9b17ab46 P 974b802141bd4f13beffef2aa03f88f9: Bootstrap starting.
I20260129 02:58:46.499697 20739 leader_election.cc:290] T 96609ac68bdf4993a12e1c23355ca013 P 974b802141bd4f13beffef2aa03f88f9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 197b880fa39d432a9b472845876bfaf0 (127.18.211.1:42027), 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3:37735)
I20260129 02:58:46.500782 20587 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "96609ac68bdf4993a12e1c23355ca013" candidate_uuid: "974b802141bd4f13beffef2aa03f88f9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c50de89a5a845668696dcb28c3badd6" is_pre_election: true
I20260129 02:58:46.501482 20633 tablet_bootstrap.cc:492] T 4408006eb5174de1896b037cf3f18457 P 197b880fa39d432a9b472845876bfaf0: No bootstrap required, opened a new log
I20260129 02:58:46.501536 20440 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "96609ac68bdf4993a12e1c23355ca013" candidate_uuid: "974b802141bd4f13beffef2aa03f88f9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "197b880fa39d432a9b472845876bfaf0" is_pre_election: true
I20260129 02:58:46.502035 20633 ts_tablet_manager.cc:1403] T 4408006eb5174de1896b037cf3f18457 P 197b880fa39d432a9b472845876bfaf0: Time spent bootstrapping tablet: real 0.049s user 0.019s sys 0.008s
W20260129 02:58:46.502350 20480 leader_election.cc:343] T 96609ac68bdf4993a12e1c23355ca013 P 974b802141bd4f13beffef2aa03f88f9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3:37735): Illegal state: must be running to vote when last-logged opid is not known
W20260129 02:58:46.502933 20479 leader_election.cc:343] T 96609ac68bdf4993a12e1c23355ca013 P 974b802141bd4f13beffef2aa03f88f9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 197b880fa39d432a9b472845876bfaf0 (127.18.211.1:42027): Illegal state: must be running to vote when last-logged opid is not known
I20260129 02:58:46.503343 20479 leader_election.cc:304] T 96609ac68bdf4993a12e1c23355ca013 P 974b802141bd4f13beffef2aa03f88f9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 974b802141bd4f13beffef2aa03f88f9; no voters: 197b880fa39d432a9b472845876bfaf0, 1c50de89a5a845668696dcb28c3badd6
I20260129 02:58:46.504426 20739 raft_consensus.cc:2749] T 96609ac68bdf4993a12e1c23355ca013 P 974b802141bd4f13beffef2aa03f88f9 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260129 02:58:46.504699 20633 raft_consensus.cc:359] T 4408006eb5174de1896b037cf3f18457 P 197b880fa39d432a9b472845876bfaf0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:58:46.505358 20633 raft_consensus.cc:385] T 4408006eb5174de1896b037cf3f18457 P 197b880fa39d432a9b472845876bfaf0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260129 02:58:46.505621 20633 raft_consensus.cc:740] T 4408006eb5174de1896b037cf3f18457 P 197b880fa39d432a9b472845876bfaf0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 197b880fa39d432a9b472845876bfaf0, State: Initialized, Role: FOLLOWER
I20260129 02:58:46.506306 20633 consensus_queue.cc:260] T 4408006eb5174de1896b037cf3f18457 P 197b880fa39d432a9b472845876bfaf0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:58:46.509096 20630 tablet_bootstrap.cc:654] T 505580c93c694bbb8ea96fdf9b17ab46 P 974b802141bd4f13beffef2aa03f88f9: Neither blocks nor log segments found. Creating new log.
I20260129 02:58:46.515264 20638 consensus_queue.cc:1048] T 913fc7d751ed4b7388d889aac30a5f9b P 197b880fa39d432a9b472845876bfaf0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260129 02:58:46.517237 20337 catalog_manager.cc:5654] T 368953c9102643b2a21f99af36b51905 P 1c50de89a5a845668696dcb28c3badd6 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3). New cstate: current_term: 1 leader_uuid: "1c50de89a5a845668696dcb28c3badd6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } health_report { overall_health: HEALTHY } } }
I20260129 02:58:46.530028 20633 ts_tablet_manager.cc:1434] T 4408006eb5174de1896b037cf3f18457 P 197b880fa39d432a9b472845876bfaf0: Time spent starting tablet: real 0.028s user 0.005s sys 0.001s
I20260129 02:58:46.531193 20638 raft_consensus.cc:493] T 75b0b6e82ddc4f92b1fe1d94f9a0b73b P 197b880fa39d432a9b472845876bfaf0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260129 02:58:46.532011 20638 raft_consensus.cc:515] T 75b0b6e82ddc4f92b1fe1d94f9a0b73b P 197b880fa39d432a9b472845876bfaf0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:58:46.533566 20633 tablet_bootstrap.cc:492] T 368953c9102643b2a21f99af36b51905 P 197b880fa39d432a9b472845876bfaf0: Bootstrap starting.
I20260129 02:58:46.536969 20512 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "75b0b6e82ddc4f92b1fe1d94f9a0b73b" candidate_uuid: "197b880fa39d432a9b472845876bfaf0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "974b802141bd4f13beffef2aa03f88f9" is_pre_election: true
I20260129 02:58:46.537766 20512 raft_consensus.cc:2393] T 75b0b6e82ddc4f92b1fe1d94f9a0b73b P 974b802141bd4f13beffef2aa03f88f9 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 197b880fa39d432a9b472845876bfaf0 in current term 1: Already voted for candidate 1c50de89a5a845668696dcb28c3badd6 in this term.
I20260129 02:58:46.527431 20631 raft_consensus.cc:359] T 96609ac68bdf4993a12e1c23355ca013 P 1c50de89a5a845668696dcb28c3badd6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:58:46.539229 20631 raft_consensus.cc:385] T 96609ac68bdf4993a12e1c23355ca013 P 1c50de89a5a845668696dcb28c3badd6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260129 02:58:46.539539 20631 raft_consensus.cc:740] T 96609ac68bdf4993a12e1c23355ca013 P 1c50de89a5a845668696dcb28c3badd6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1c50de89a5a845668696dcb28c3badd6, State: Initialized, Role: FOLLOWER
I20260129 02:58:46.540225 20631 consensus_queue.cc:260] T 96609ac68bdf4993a12e1c23355ca013 P 1c50de89a5a845668696dcb28c3badd6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:58:46.549120 20631 ts_tablet_manager.cc:1434] T 96609ac68bdf4993a12e1c23355ca013 P 1c50de89a5a845668696dcb28c3badd6: Time spent starting tablet: real 0.058s user 0.005s sys 0.002s
I20260129 02:58:46.550207 20631 tablet_bootstrap.cc:492] T 505580c93c694bbb8ea96fdf9b17ab46 P 1c50de89a5a845668696dcb28c3badd6: Bootstrap starting.
I20260129 02:58:46.546355 20638 leader_election.cc:290] T 75b0b6e82ddc4f92b1fe1d94f9a0b73b P 197b880fa39d432a9b472845876bfaf0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 974b802141bd4f13beffef2aa03f88f9 (127.18.211.2:34517), 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3:37735)
I20260129 02:58:46.556718 20587 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "75b0b6e82ddc4f92b1fe1d94f9a0b73b" candidate_uuid: "197b880fa39d432a9b472845876bfaf0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c50de89a5a845668696dcb28c3badd6" is_pre_election: true
I20260129 02:58:46.564970 20405 leader_election.cc:304] T 75b0b6e82ddc4f92b1fe1d94f9a0b73b P 197b880fa39d432a9b472845876bfaf0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 197b880fa39d432a9b472845876bfaf0; no voters: 1c50de89a5a845668696dcb28c3badd6, 974b802141bd4f13beffef2aa03f88f9
I20260129 02:58:46.567273 20721 raft_consensus.cc:3060] T 75b0b6e82ddc4f92b1fe1d94f9a0b73b P 197b880fa39d432a9b472845876bfaf0 [term 0 FOLLOWER]: Advancing to term 1
I20260129 02:58:46.567180 20638 consensus_queue.cc:1048] T 913fc7d751ed4b7388d889aac30a5f9b P 197b880fa39d432a9b472845876bfaf0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260129 02:58:46.570744 20630 tablet_bootstrap.cc:492] T 505580c93c694bbb8ea96fdf9b17ab46 P 974b802141bd4f13beffef2aa03f88f9: No bootstrap required, opened a new log
I20260129 02:58:46.572315 20630 ts_tablet_manager.cc:1403] T 505580c93c694bbb8ea96fdf9b17ab46 P 974b802141bd4f13beffef2aa03f88f9: Time spent bootstrapping tablet: real 0.074s user 0.019s sys 0.021s
I20260129 02:58:46.575139 20630 raft_consensus.cc:359] T 505580c93c694bbb8ea96fdf9b17ab46 P 974b802141bd4f13beffef2aa03f88f9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:58:46.575822 20630 raft_consensus.cc:385] T 505580c93c694bbb8ea96fdf9b17ab46 P 974b802141bd4f13beffef2aa03f88f9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260129 02:58:46.576118 20630 raft_consensus.cc:740] T 505580c93c694bbb8ea96fdf9b17ab46 P 974b802141bd4f13beffef2aa03f88f9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 974b802141bd4f13beffef2aa03f88f9, State: Initialized, Role: FOLLOWER
I20260129 02:58:46.576787 20630 consensus_queue.cc:260] T 505580c93c694bbb8ea96fdf9b17ab46 P 974b802141bd4f13beffef2aa03f88f9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:58:46.577688 20721 raft_consensus.cc:2749] T 75b0b6e82ddc4f92b1fe1d94f9a0b73b P 197b880fa39d432a9b472845876bfaf0 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260129 02:58:46.579080 20630 ts_tablet_manager.cc:1434] T 505580c93c694bbb8ea96fdf9b17ab46 P 974b802141bd4f13beffef2aa03f88f9: Time spent starting tablet: real 0.006s user 0.003s sys 0.001s
I20260129 02:58:46.580232 20630 tablet_bootstrap.cc:492] T 36ff33be666a4c82954731691dbcf327 P 974b802141bd4f13beffef2aa03f88f9: Bootstrap starting.
I20260129 02:58:46.580847 20732 raft_consensus.cc:493] T 4408006eb5174de1896b037cf3f18457 P 974b802141bd4f13beffef2aa03f88f9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260129 02:58:46.581286 20732 raft_consensus.cc:515] T 4408006eb5174de1896b037cf3f18457 P 974b802141bd4f13beffef2aa03f88f9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:58:46.584175 20732 leader_election.cc:290] T 4408006eb5174de1896b037cf3f18457 P 974b802141bd4f13beffef2aa03f88f9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 197b880fa39d432a9b472845876bfaf0 (127.18.211.1:42027), 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3:37735)
I20260129 02:58:46.586612 20440 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4408006eb5174de1896b037cf3f18457" candidate_uuid: "974b802141bd4f13beffef2aa03f88f9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "197b880fa39d432a9b472845876bfaf0" is_pre_election: true
I20260129 02:58:46.587261 20440 raft_consensus.cc:2468] T 4408006eb5174de1896b037cf3f18457 P 197b880fa39d432a9b472845876bfaf0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 974b802141bd4f13beffef2aa03f88f9 in term 0.
I20260129 02:58:46.587589 20633 tablet_bootstrap.cc:654] T 368953c9102643b2a21f99af36b51905 P 197b880fa39d432a9b472845876bfaf0: Neither blocks nor log segments found. Creating new log.
I20260129 02:58:46.588550 20479 leader_election.cc:304] T 4408006eb5174de1896b037cf3f18457 P 974b802141bd4f13beffef2aa03f88f9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 197b880fa39d432a9b472845876bfaf0, 974b802141bd4f13beffef2aa03f88f9; no voters:
I20260129 02:58:46.589687 20732 raft_consensus.cc:2804] T 4408006eb5174de1896b037cf3f18457 P 974b802141bd4f13beffef2aa03f88f9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260129 02:58:46.590050 20732 raft_consensus.cc:493] T 4408006eb5174de1896b037cf3f18457 P 974b802141bd4f13beffef2aa03f88f9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260129 02:58:46.590392 20732 raft_consensus.cc:3060] T 4408006eb5174de1896b037cf3f18457 P 974b802141bd4f13beffef2aa03f88f9 [term 0 FOLLOWER]: Advancing to term 1
I20260129 02:58:46.591472 20631 tablet_bootstrap.cc:654] T 505580c93c694bbb8ea96fdf9b17ab46 P 1c50de89a5a845668696dcb28c3badd6: Neither blocks nor log segments found. Creating new log.
W20260129 02:58:46.600159 20387 auto_rebalancer.cc:229] Could not retrieve cluster info: Service unavailable: Tablet not running
I20260129 02:58:46.600673 20732 raft_consensus.cc:515] T 4408006eb5174de1896b037cf3f18457 P 974b802141bd4f13beffef2aa03f88f9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:58:46.602423 20732 leader_election.cc:290] T 4408006eb5174de1896b037cf3f18457 P 974b802141bd4f13beffef2aa03f88f9 [CANDIDATE]: Term 1 election: Requested vote from peers 197b880fa39d432a9b472845876bfaf0 (127.18.211.1:42027), 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3:37735)
I20260129 02:58:46.602924 20587 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4408006eb5174de1896b037cf3f18457" candidate_uuid: "974b802141bd4f13beffef2aa03f88f9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c50de89a5a845668696dcb28c3badd6" is_pre_election: true
I20260129 02:58:46.603523 20589 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4408006eb5174de1896b037cf3f18457" candidate_uuid: "974b802141bd4f13beffef2aa03f88f9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c50de89a5a845668696dcb28c3badd6"
I20260129 02:58:46.603719 20587 raft_consensus.cc:2468] T 4408006eb5174de1896b037cf3f18457 P 1c50de89a5a845668696dcb28c3badd6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 974b802141bd4f13beffef2aa03f88f9 in term 0.
I20260129 02:58:46.605402 20440 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4408006eb5174de1896b037cf3f18457" candidate_uuid: "974b802141bd4f13beffef2aa03f88f9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "197b880fa39d432a9b472845876bfaf0"
I20260129 02:58:46.605990 20440 raft_consensus.cc:3060] T 4408006eb5174de1896b037cf3f18457 P 197b880fa39d432a9b472845876bfaf0 [term 0 FOLLOWER]: Advancing to term 1
I20260129 02:58:46.611264 20630 tablet_bootstrap.cc:654] T 36ff33be666a4c82954731691dbcf327 P 974b802141bd4f13beffef2aa03f88f9: Neither blocks nor log segments found. Creating new log.
I20260129 02:58:46.614060 20440 raft_consensus.cc:2468] T 4408006eb5174de1896b037cf3f18457 P 197b880fa39d432a9b472845876bfaf0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 974b802141bd4f13beffef2aa03f88f9 in term 1.
I20260129 02:58:46.614141 20721 raft_consensus.cc:493] T c2633a6fc34c44579de348d208b0c01e P 197b880fa39d432a9b472845876bfaf0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260129 02:58:46.614709 20739 consensus_queue.cc:1048] T 4f8f3a38f6e24ba4857875181f447442 P 974b802141bd4f13beffef2aa03f88f9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260129 02:58:46.615437 20721 raft_consensus.cc:515] T c2633a6fc34c44579de348d208b0c01e P 197b880fa39d432a9b472845876bfaf0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:58:46.617502 20479 leader_election.cc:304] T 4408006eb5174de1896b037cf3f18457 P 974b802141bd4f13beffef2aa03f88f9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 197b880fa39d432a9b472845876bfaf0, 974b802141bd4f13beffef2aa03f88f9; no voters: 1c50de89a5a845668696dcb28c3badd6
I20260129 02:58:46.618944 20739 raft_consensus.cc:2804] T 4408006eb5174de1896b037cf3f18457 P 974b802141bd4f13beffef2aa03f88f9 [term 1 FOLLOWER]: Leader election won for term 1
I20260129 02:58:46.619498 20739 raft_consensus.cc:697] T 4408006eb5174de1896b037cf3f18457 P 974b802141bd4f13beffef2aa03f88f9 [term 1 LEADER]: Becoming Leader. State: Replica: 974b802141bd4f13beffef2aa03f88f9, State: Running, Role: LEADER
I20260129 02:58:46.619124 20721 leader_election.cc:290] T c2633a6fc34c44579de348d208b0c01e P 197b880fa39d432a9b472845876bfaf0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 974b802141bd4f13beffef2aa03f88f9 (127.18.211.2:34517), 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3:37735)
I20260129 02:58:46.620410 20739 consensus_queue.cc:237] T 4408006eb5174de1896b037cf3f18457 P 974b802141bd4f13beffef2aa03f88f9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:58:46.620880 20512 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c2633a6fc34c44579de348d208b0c01e" candidate_uuid: "197b880fa39d432a9b472845876bfaf0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "974b802141bd4f13beffef2aa03f88f9" is_pre_election: true
I20260129 02:58:46.623762 20631 tablet_bootstrap.cc:492] T 505580c93c694bbb8ea96fdf9b17ab46 P 1c50de89a5a845668696dcb28c3badd6: No bootstrap required, opened a new log
I20260129 02:58:46.624233 20631 ts_tablet_manager.cc:1403] T 505580c93c694bbb8ea96fdf9b17ab46 P 1c50de89a5a845668696dcb28c3badd6: Time spent bootstrapping tablet: real 0.074s user 0.013s sys 0.003s
I20260129 02:58:46.624245 20633 tablet_bootstrap.cc:492] T 368953c9102643b2a21f99af36b51905 P 197b880fa39d432a9b472845876bfaf0: No bootstrap required, opened a new log
I20260129 02:58:46.624809 20633 ts_tablet_manager.cc:1403] T 368953c9102643b2a21f99af36b51905 P 197b880fa39d432a9b472845876bfaf0: Time spent bootstrapping tablet: real 0.091s user 0.011s sys 0.005s
I20260129 02:58:46.624977 20587 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c2633a6fc34c44579de348d208b0c01e" candidate_uuid: "197b880fa39d432a9b472845876bfaf0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c50de89a5a845668696dcb28c3badd6" is_pre_election: true
I20260129 02:58:46.621729 20512 raft_consensus.cc:2468] T c2633a6fc34c44579de348d208b0c01e P 974b802141bd4f13beffef2aa03f88f9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 197b880fa39d432a9b472845876bfaf0 in term 0.
W20260129 02:58:46.626614 20405 leader_election.cc:343] T c2633a6fc34c44579de348d208b0c01e P 197b880fa39d432a9b472845876bfaf0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3:37735): Illegal state: must be running to vote when last-logged opid is not known
I20260129 02:58:46.627573 20407 leader_election.cc:304] T c2633a6fc34c44579de348d208b0c01e P 197b880fa39d432a9b472845876bfaf0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 197b880fa39d432a9b472845876bfaf0, 974b802141bd4f13beffef2aa03f88f9; no voters: 1c50de89a5a845668696dcb28c3badd6
I20260129 02:58:46.627307 20633 raft_consensus.cc:359] T 368953c9102643b2a21f99af36b51905 P 197b880fa39d432a9b472845876bfaf0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:58:46.628295 20633 raft_consensus.cc:385] T 368953c9102643b2a21f99af36b51905 P 197b880fa39d432a9b472845876bfaf0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260129 02:58:46.628852 20721 raft_consensus.cc:2804] T c2633a6fc34c44579de348d208b0c01e P 197b880fa39d432a9b472845876bfaf0 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260129 02:58:46.629264 20721 raft_consensus.cc:493] T c2633a6fc34c44579de348d208b0c01e P 197b880fa39d432a9b472845876bfaf0 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260129 02:58:46.629721 20721 raft_consensus.cc:3060] T c2633a6fc34c44579de348d208b0c01e P 197b880fa39d432a9b472845876bfaf0 [term 0 FOLLOWER]: Advancing to term 1
I20260129 02:58:46.628683 20633 raft_consensus.cc:740] T 368953c9102643b2a21f99af36b51905 P 197b880fa39d432a9b472845876bfaf0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 197b880fa39d432a9b472845876bfaf0, State: Initialized, Role: FOLLOWER
I20260129 02:58:46.631700 20633 consensus_queue.cc:260] T 368953c9102643b2a21f99af36b51905 P 197b880fa39d432a9b472845876bfaf0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:58:46.634769 20633 ts_tablet_manager.cc:1434] T 368953c9102643b2a21f99af36b51905 P 197b880fa39d432a9b472845876bfaf0: Time spent starting tablet: real 0.010s user 0.005s sys 0.000s
I20260129 02:58:46.648335 20633 tablet_bootstrap.cc:492] T cb4727386a3e48aca5d71cc6c611c197 P 197b880fa39d432a9b472845876bfaf0: Bootstrap starting.
I20260129 02:58:46.649117 20721 raft_consensus.cc:515] T c2633a6fc34c44579de348d208b0c01e P 197b880fa39d432a9b472845876bfaf0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:58:46.636583 20631 raft_consensus.cc:359] T 505580c93c694bbb8ea96fdf9b17ab46 P 1c50de89a5a845668696dcb28c3badd6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:58:46.651461 20631 raft_consensus.cc:385] T 505580c93c694bbb8ea96fdf9b17ab46 P 1c50de89a5a845668696dcb28c3badd6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260129 02:58:46.651479 20734 consensus_queue.cc:1048] T 4f8f3a38f6e24ba4857875181f447442 P 974b802141bd4f13beffef2aa03f88f9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260129 02:58:46.652004 20512 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c2633a6fc34c44579de348d208b0c01e" candidate_uuid: "197b880fa39d432a9b472845876bfaf0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "974b802141bd4f13beffef2aa03f88f9"
I20260129 02:58:46.652412 20721 leader_election.cc:290] T c2633a6fc34c44579de348d208b0c01e P 197b880fa39d432a9b472845876bfaf0 [CANDIDATE]: Term 1 election: Requested vote from peers 974b802141bd4f13beffef2aa03f88f9 (127.18.211.2:34517), 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3:37735)
I20260129 02:58:46.651886 20631 raft_consensus.cc:740] T 505580c93c694bbb8ea96fdf9b17ab46 P 1c50de89a5a845668696dcb28c3badd6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1c50de89a5a845668696dcb28c3badd6, State: Initialized, Role: FOLLOWER
I20260129 02:58:46.652622 20512 raft_consensus.cc:3060] T c2633a6fc34c44579de348d208b0c01e P 974b802141bd4f13beffef2aa03f88f9 [term 0 FOLLOWER]: Advancing to term 1
I20260129 02:58:46.653226 20631 consensus_queue.cc:260] T 505580c93c694bbb8ea96fdf9b17ab46 P 1c50de89a5a845668696dcb28c3badd6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:58:46.647310 20630 tablet_bootstrap.cc:492] T 36ff33be666a4c82954731691dbcf327 P 974b802141bd4f13beffef2aa03f88f9: No bootstrap required, opened a new log
I20260129 02:58:46.654486 20630 ts_tablet_manager.cc:1403] T 36ff33be666a4c82954731691dbcf327 P 974b802141bd4f13beffef2aa03f88f9: Time spent bootstrapping tablet: real 0.075s user 0.011s sys 0.003s
I20260129 02:58:46.655556 20587 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c2633a6fc34c44579de348d208b0c01e" candidate_uuid: "197b880fa39d432a9b472845876bfaf0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c50de89a5a845668696dcb28c3badd6"
W20260129 02:58:46.656912 20405 leader_election.cc:343] T c2633a6fc34c44579de348d208b0c01e P 197b880fa39d432a9b472845876bfaf0 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3:37735): Illegal state: must be running to vote when last-logged opid is not known
I20260129 02:58:46.657681 20633 tablet_bootstrap.cc:654] T cb4727386a3e48aca5d71cc6c611c197 P 197b880fa39d432a9b472845876bfaf0: Neither blocks nor log segments found. Creating new log.
I20260129 02:58:46.667433 20630 raft_consensus.cc:359] T 36ff33be666a4c82954731691dbcf327 P 974b802141bd4f13beffef2aa03f88f9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:58:46.668187 20630 raft_consensus.cc:385] T 36ff33be666a4c82954731691dbcf327 P 974b802141bd4f13beffef2aa03f88f9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260129 02:58:46.668551 20630 raft_consensus.cc:740] T 36ff33be666a4c82954731691dbcf327 P 974b802141bd4f13beffef2aa03f88f9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 974b802141bd4f13beffef2aa03f88f9, State: Initialized, Role: FOLLOWER
I20260129 02:58:46.676004 20631 ts_tablet_manager.cc:1434] T 505580c93c694bbb8ea96fdf9b17ab46 P 1c50de89a5a845668696dcb28c3badd6: Time spent starting tablet: real 0.051s user 0.007s sys 0.000s
I20260129 02:58:46.677086 20631 tablet_bootstrap.cc:492] T 36ff33be666a4c82954731691dbcf327 P 1c50de89a5a845668696dcb28c3badd6: Bootstrap starting.
I20260129 02:58:46.680645 20630 consensus_queue.cc:260] T 36ff33be666a4c82954731691dbcf327 P 974b802141bd4f13beffef2aa03f88f9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:58:46.690798 20719 raft_consensus.cc:493] T 505580c93c694bbb8ea96fdf9b17ab46 P 1c50de89a5a845668696dcb28c3badd6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260129 02:58:46.691584 20719 raft_consensus.cc:515] T 505580c93c694bbb8ea96fdf9b17ab46 P 1c50de89a5a845668696dcb28c3badd6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:58:46.695451 20440 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "505580c93c694bbb8ea96fdf9b17ab46" candidate_uuid: "1c50de89a5a845668696dcb28c3badd6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "197b880fa39d432a9b472845876bfaf0" is_pre_election: true
I20260129 02:58:46.696177 20440 raft_consensus.cc:2468] T 505580c93c694bbb8ea96fdf9b17ab46 P 197b880fa39d432a9b472845876bfaf0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1c50de89a5a845668696dcb28c3badd6 in term 0.
I20260129 02:58:46.701224 20553 leader_election.cc:304] T 505580c93c694bbb8ea96fdf9b17ab46 P 1c50de89a5a845668696dcb28c3badd6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 197b880fa39d432a9b472845876bfaf0, 1c50de89a5a845668696dcb28c3badd6; no voters:
I20260129 02:58:46.709038 20514 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "505580c93c694bbb8ea96fdf9b17ab46" candidate_uuid: "1c50de89a5a845668696dcb28c3badd6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "974b802141bd4f13beffef2aa03f88f9" is_pre_election: true
I20260129 02:58:46.709807 20630 ts_tablet_manager.cc:1434] T 36ff33be666a4c82954731691dbcf327 P 974b802141bd4f13beffef2aa03f88f9: Time spent starting tablet: real 0.055s user 0.007s sys 0.000s
I20260129 02:58:46.710014 20514 raft_consensus.cc:2468] T 505580c93c694bbb8ea96fdf9b17ab46 P 974b802141bd4f13beffef2aa03f88f9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1c50de89a5a845668696dcb28c3badd6 in term 0.
I20260129 02:58:46.702251 20840 raft_consensus.cc:2804] T 505580c93c694bbb8ea96fdf9b17ab46 P 1c50de89a5a845668696dcb28c3badd6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260129 02:58:46.711405 20840 raft_consensus.cc:493] T 505580c93c694bbb8ea96fdf9b17ab46 P 1c50de89a5a845668696dcb28c3badd6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260129 02:58:46.710911 20630 tablet_bootstrap.cc:492] T cb4727386a3e48aca5d71cc6c611c197 P 974b802141bd4f13beffef2aa03f88f9: Bootstrap starting.
I20260129 02:58:46.712999 20840 raft_consensus.cc:3060] T 505580c93c694bbb8ea96fdf9b17ab46 P 1c50de89a5a845668696dcb28c3badd6 [term 0 FOLLOWER]: Advancing to term 1
I20260129 02:58:46.702534 20719 leader_election.cc:290] T 505580c93c694bbb8ea96fdf9b17ab46 P 1c50de89a5a845668696dcb28c3badd6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 197b880fa39d432a9b472845876bfaf0 (127.18.211.1:42027), 974b802141bd4f13beffef2aa03f88f9 (127.18.211.2:34517)
I20260129 02:58:46.727777 20512 raft_consensus.cc:2468] T c2633a6fc34c44579de348d208b0c01e P 974b802141bd4f13beffef2aa03f88f9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 197b880fa39d432a9b472845876bfaf0 in term 1.
I20260129 02:58:46.729168 20771 consensus_queue.cc:1048] T dc2f4cb752cf4d59b5f6bc689f5d8104 P 1c50de89a5a845668696dcb28c3badd6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260129 02:58:46.736799 20407 leader_election.cc:304] T c2633a6fc34c44579de348d208b0c01e P 197b880fa39d432a9b472845876bfaf0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 197b880fa39d432a9b472845876bfaf0, 974b802141bd4f13beffef2aa03f88f9; no voters: 1c50de89a5a845668696dcb28c3badd6
I20260129 02:58:46.738421 20721 raft_consensus.cc:2804] T c2633a6fc34c44579de348d208b0c01e P 197b880fa39d432a9b472845876bfaf0 [term 1 FOLLOWER]: Leader election won for term 1
I20260129 02:58:46.739066 20721 raft_consensus.cc:697] T c2633a6fc34c44579de348d208b0c01e P 197b880fa39d432a9b472845876bfaf0 [term 1 LEADER]: Becoming Leader. State: Replica: 197b880fa39d432a9b472845876bfaf0, State: Running, Role: LEADER
I20260129 02:58:46.740193 20721 consensus_queue.cc:237] T c2633a6fc34c44579de348d208b0c01e P 197b880fa39d432a9b472845876bfaf0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:58:46.752087 20633 tablet_bootstrap.cc:492] T cb4727386a3e48aca5d71cc6c611c197 P 197b880fa39d432a9b472845876bfaf0: No bootstrap required, opened a new log
I20260129 02:58:46.752734 20633 ts_tablet_manager.cc:1403] T cb4727386a3e48aca5d71cc6c611c197 P 197b880fa39d432a9b472845876bfaf0: Time spent bootstrapping tablet: real 0.105s user 0.037s sys 0.061s
I20260129 02:58:46.769578 20631 tablet_bootstrap.cc:654] T 36ff33be666a4c82954731691dbcf327 P 1c50de89a5a845668696dcb28c3badd6: Neither blocks nor log segments found. Creating new log.
I20260129 02:58:46.775056 20630 tablet_bootstrap.cc:654] T cb4727386a3e48aca5d71cc6c611c197 P 974b802141bd4f13beffef2aa03f88f9: Neither blocks nor log segments found. Creating new log.
I20260129 02:58:46.776705 20840 raft_consensus.cc:515] T 505580c93c694bbb8ea96fdf9b17ab46 P 1c50de89a5a845668696dcb28c3badd6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:58:46.788336 20633 raft_consensus.cc:359] T cb4727386a3e48aca5d71cc6c611c197 P 197b880fa39d432a9b472845876bfaf0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:58:46.789055 20633 raft_consensus.cc:385] T cb4727386a3e48aca5d71cc6c611c197 P 197b880fa39d432a9b472845876bfaf0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260129 02:58:46.789323 20633 raft_consensus.cc:740] T cb4727386a3e48aca5d71cc6c611c197 P 197b880fa39d432a9b472845876bfaf0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 197b880fa39d432a9b472845876bfaf0, State: Initialized, Role: FOLLOWER
I20260129 02:58:46.789505 20435 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "505580c93c694bbb8ea96fdf9b17ab46" candidate_uuid: "1c50de89a5a845668696dcb28c3badd6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "197b880fa39d432a9b472845876bfaf0"
I20260129 02:58:46.790127 20435 raft_consensus.cc:3060] T 505580c93c694bbb8ea96fdf9b17ab46 P 197b880fa39d432a9b472845876bfaf0 [term 0 FOLLOWER]: Advancing to term 1
I20260129 02:58:46.789919 20633 consensus_queue.cc:260] T cb4727386a3e48aca5d71cc6c611c197 P 197b880fa39d432a9b472845876bfaf0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:58:46.820274 20514 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "505580c93c694bbb8ea96fdf9b17ab46" candidate_uuid: "1c50de89a5a845668696dcb28c3badd6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "974b802141bd4f13beffef2aa03f88f9"
I20260129 02:58:46.821131 20514 raft_consensus.cc:3060] T 505580c93c694bbb8ea96fdf9b17ab46 P 974b802141bd4f13beffef2aa03f88f9 [term 0 FOLLOWER]: Advancing to term 1
I20260129 02:58:46.821753 20840 leader_election.cc:290] T 505580c93c694bbb8ea96fdf9b17ab46 P 1c50de89a5a845668696dcb28c3badd6 [CANDIDATE]: Term 1 election: Requested vote from peers 197b880fa39d432a9b472845876bfaf0 (127.18.211.1:42027), 974b802141bd4f13beffef2aa03f88f9 (127.18.211.2:34517)
I20260129 02:58:46.830842 20338 catalog_manager.cc:5654] T c2633a6fc34c44579de348d208b0c01e P 197b880fa39d432a9b472845876bfaf0 reported cstate change: term changed from 0 to 1, leader changed from <none> to 197b880fa39d432a9b472845876bfaf0 (127.18.211.1). New cstate: current_term: 1 leader_uuid: "197b880fa39d432a9b472845876bfaf0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } health_report { overall_health: UNKNOWN } } }
I20260129 02:58:46.835134 20840 raft_consensus.cc:493] T 96609ac68bdf4993a12e1c23355ca013 P 1c50de89a5a845668696dcb28c3badd6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260129 02:58:46.835824 20840 raft_consensus.cc:515] T 96609ac68bdf4993a12e1c23355ca013 P 1c50de89a5a845668696dcb28c3badd6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:58:46.839406 20441 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "96609ac68bdf4993a12e1c23355ca013" candidate_uuid: "1c50de89a5a845668696dcb28c3badd6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "197b880fa39d432a9b472845876bfaf0" is_pre_election: true
I20260129 02:58:46.840370 20633 ts_tablet_manager.cc:1434] T cb4727386a3e48aca5d71cc6c611c197 P 197b880fa39d432a9b472845876bfaf0: Time spent starting tablet: real 0.087s user 0.017s sys 0.021s
I20260129 02:58:46.841403 20633 tablet_bootstrap.cc:492] T 96609ac68bdf4993a12e1c23355ca013 P 197b880fa39d432a9b472845876bfaf0: Bootstrap starting.
I20260129 02:58:46.843077 20435 raft_consensus.cc:2468] T 505580c93c694bbb8ea96fdf9b17ab46 P 197b880fa39d432a9b472845876bfaf0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1c50de89a5a845668696dcb28c3badd6 in term 1.
I20260129 02:58:46.844521 20335 catalog_manager.cc:5654] T 4408006eb5174de1896b037cf3f18457 P 974b802141bd4f13beffef2aa03f88f9 reported cstate change: term changed from 0 to 1, leader changed from <none> to 974b802141bd4f13beffef2aa03f88f9 (127.18.211.2). New cstate: current_term: 1 leader_uuid: "974b802141bd4f13beffef2aa03f88f9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } health_report { overall_health: UNKNOWN } } }
I20260129 02:58:46.844540 20514 raft_consensus.cc:2468] T 505580c93c694bbb8ea96fdf9b17ab46 P 974b802141bd4f13beffef2aa03f88f9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1c50de89a5a845668696dcb28c3badd6 in term 1.
I20260129 02:58:46.850042 20556 leader_election.cc:304] T 505580c93c694bbb8ea96fdf9b17ab46 P 1c50de89a5a845668696dcb28c3badd6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1c50de89a5a845668696dcb28c3badd6, 974b802141bd4f13beffef2aa03f88f9; no voters:
I20260129 02:58:46.851433 20771 raft_consensus.cc:2804] T 505580c93c694bbb8ea96fdf9b17ab46 P 1c50de89a5a845668696dcb28c3badd6 [term 1 FOLLOWER]: Leader election won for term 1
W20260129 02:58:46.854672 20553 leader_election.cc:343] T 96609ac68bdf4993a12e1c23355ca013 P 1c50de89a5a845668696dcb28c3badd6 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 197b880fa39d432a9b472845876bfaf0 (127.18.211.1:42027): Illegal state: must be running to vote when last-logged opid is not known
I20260129 02:58:46.855016 20515 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "96609ac68bdf4993a12e1c23355ca013" candidate_uuid: "1c50de89a5a845668696dcb28c3badd6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "974b802141bd4f13beffef2aa03f88f9" is_pre_election: true
I20260129 02:58:46.853792 20771 raft_consensus.cc:697] T 505580c93c694bbb8ea96fdf9b17ab46 P 1c50de89a5a845668696dcb28c3badd6 [term 1 LEADER]: Becoming Leader. State: Replica: 1c50de89a5a845668696dcb28c3badd6, State: Running, Role: LEADER
I20260129 02:58:46.855646 20515 raft_consensus.cc:2468] T 96609ac68bdf4993a12e1c23355ca013 P 974b802141bd4f13beffef2aa03f88f9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1c50de89a5a845668696dcb28c3badd6 in term 0.
I20260129 02:58:46.856556 20631 tablet_bootstrap.cc:492] T 36ff33be666a4c82954731691dbcf327 P 1c50de89a5a845668696dcb28c3badd6: No bootstrap required, opened a new log
I20260129 02:58:46.856187 20771 consensus_queue.cc:237] T 505580c93c694bbb8ea96fdf9b17ab46 P 1c50de89a5a845668696dcb28c3badd6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:58:46.856766 20556 leader_election.cc:304] T 96609ac68bdf4993a12e1c23355ca013 P 1c50de89a5a845668696dcb28c3badd6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1c50de89a5a845668696dcb28c3badd6, 974b802141bd4f13beffef2aa03f88f9; no voters: 197b880fa39d432a9b472845876bfaf0
I20260129 02:58:46.856992 20631 ts_tablet_manager.cc:1403] T 36ff33be666a4c82954731691dbcf327 P 1c50de89a5a845668696dcb28c3badd6: Time spent bootstrapping tablet: real 0.180s user 0.010s sys 0.004s
I20260129 02:58:46.857543 20719 raft_consensus.cc:2804] T 96609ac68bdf4993a12e1c23355ca013 P 1c50de89a5a845668696dcb28c3badd6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260129 02:58:46.858914 20719 raft_consensus.cc:493] T 96609ac68bdf4993a12e1c23355ca013 P 1c50de89a5a845668696dcb28c3badd6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260129 02:58:46.859254 20719 raft_consensus.cc:3060] T 96609ac68bdf4993a12e1c23355ca013 P 1c50de89a5a845668696dcb28c3badd6 [term 0 FOLLOWER]: Advancing to term 1
I20260129 02:58:46.858058 20840 leader_election.cc:290] T 96609ac68bdf4993a12e1c23355ca013 P 1c50de89a5a845668696dcb28c3badd6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 197b880fa39d432a9b472845876bfaf0 (127.18.211.1:42027), 974b802141bd4f13beffef2aa03f88f9 (127.18.211.2:34517)
I20260129 02:58:46.894901 20633 tablet_bootstrap.cc:654] T 96609ac68bdf4993a12e1c23355ca013 P 197b880fa39d432a9b472845876bfaf0: Neither blocks nor log segments found. Creating new log.
I20260129 02:58:46.896392 20739 raft_consensus.cc:493] T 36ff33be666a4c82954731691dbcf327 P 974b802141bd4f13beffef2aa03f88f9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260129 02:58:46.897070 20739 raft_consensus.cc:515] T 36ff33be666a4c82954731691dbcf327 P 974b802141bd4f13beffef2aa03f88f9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:58:46.916471 20739 leader_election.cc:290] T 36ff33be666a4c82954731691dbcf327 P 974b802141bd4f13beffef2aa03f88f9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 197b880fa39d432a9b472845876bfaf0 (127.18.211.1:42027), 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3:37735)
I20260129 02:58:46.917374 20832 consensus_queue.cc:1048] T dc2f4cb752cf4d59b5f6bc689f5d8104 P 1c50de89a5a845668696dcb28c3badd6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.018s
I20260129 02:58:46.973147 20630 tablet_bootstrap.cc:492] T cb4727386a3e48aca5d71cc6c611c197 P 974b802141bd4f13beffef2aa03f88f9: No bootstrap required, opened a new log
I20260129 02:58:46.973680 20630 ts_tablet_manager.cc:1403] T cb4727386a3e48aca5d71cc6c611c197 P 974b802141bd4f13beffef2aa03f88f9: Time spent bootstrapping tablet: real 0.263s user 0.024s sys 0.003s
I20260129 02:58:46.976353 20630 raft_consensus.cc:359] T cb4727386a3e48aca5d71cc6c611c197 P 974b802141bd4f13beffef2aa03f88f9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:58:46.977248 20630 raft_consensus.cc:385] T cb4727386a3e48aca5d71cc6c611c197 P 974b802141bd4f13beffef2aa03f88f9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260129 02:58:46.977609 20630 raft_consensus.cc:740] T cb4727386a3e48aca5d71cc6c611c197 P 974b802141bd4f13beffef2aa03f88f9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 974b802141bd4f13beffef2aa03f88f9, State: Initialized, Role: FOLLOWER
I20260129 02:58:46.976326 20631 raft_consensus.cc:359] T 36ff33be666a4c82954731691dbcf327 P 1c50de89a5a845668696dcb28c3badd6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:58:46.978303 20630 consensus_queue.cc:260] T cb4727386a3e48aca5d71cc6c611c197 P 974b802141bd4f13beffef2aa03f88f9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:58:46.978793 20631 raft_consensus.cc:385] T 36ff33be666a4c82954731691dbcf327 P 1c50de89a5a845668696dcb28c3badd6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260129 02:58:46.979066 20631 raft_consensus.cc:740] T 36ff33be666a4c82954731691dbcf327 P 1c50de89a5a845668696dcb28c3badd6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1c50de89a5a845668696dcb28c3badd6, State: Initialized, Role: FOLLOWER
I20260129 02:58:46.980970 20631 consensus_queue.cc:260] T 36ff33be666a4c82954731691dbcf327 P 1c50de89a5a845668696dcb28c3badd6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:58:46.951423 20719 raft_consensus.cc:515] T 96609ac68bdf4993a12e1c23355ca013 P 1c50de89a5a845668696dcb28c3badd6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:58:46.989665 20589 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "36ff33be666a4c82954731691dbcf327" candidate_uuid: "974b802141bd4f13beffef2aa03f88f9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c50de89a5a845668696dcb28c3badd6" is_pre_election: true
I20260129 02:58:46.990345 20589 raft_consensus.cc:2468] T 36ff33be666a4c82954731691dbcf327 P 1c50de89a5a845668696dcb28c3badd6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 974b802141bd4f13beffef2aa03f88f9 in term 0.
I20260129 02:58:46.993458 20440 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "36ff33be666a4c82954731691dbcf327" candidate_uuid: "974b802141bd4f13beffef2aa03f88f9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "197b880fa39d432a9b472845876bfaf0" is_pre_election: true
I20260129 02:58:47.009186 20828 raft_consensus.cc:493] T 368953c9102643b2a21f99af36b51905 P 197b880fa39d432a9b472845876bfaf0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260129 02:58:47.009720 20828 raft_consensus.cc:515] T 368953c9102643b2a21f99af36b51905 P 197b880fa39d432a9b472845876bfaf0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:58:47.011653 20828 leader_election.cc:290] T 368953c9102643b2a21f99af36b51905 P 197b880fa39d432a9b472845876bfaf0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 974b802141bd4f13beffef2aa03f88f9 (127.18.211.2:34517), 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3:37735)
I20260129 02:58:47.016052 20630 ts_tablet_manager.cc:1434] T cb4727386a3e48aca5d71cc6c611c197 P 974b802141bd4f13beffef2aa03f88f9: Time spent starting tablet: real 0.042s user 0.006s sys 0.002s
I20260129 02:58:47.017371 20512 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "368953c9102643b2a21f99af36b51905" candidate_uuid: "197b880fa39d432a9b472845876bfaf0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "974b802141bd4f13beffef2aa03f88f9" is_pre_election: true
I20260129 02:58:47.018180 20512 raft_consensus.cc:2393] T 368953c9102643b2a21f99af36b51905 P 974b802141bd4f13beffef2aa03f88f9 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 197b880fa39d432a9b472845876bfaf0 in current term 1: Already voted for candidate 1c50de89a5a845668696dcb28c3badd6 in this term.
I20260129 02:58:47.017371 20630 tablet_bootstrap.cc:492] T e33a417cf9c54cf5af73a4ee3aac4d3a P 974b802141bd4f13beffef2aa03f88f9: Bootstrap starting.
I20260129 02:58:47.020529 20870 raft_consensus.cc:493] T cb4727386a3e48aca5d71cc6c611c197 P 974b802141bd4f13beffef2aa03f88f9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260129 02:58:47.020989 20870 raft_consensus.cc:515] T cb4727386a3e48aca5d71cc6c611c197 P 974b802141bd4f13beffef2aa03f88f9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:58:47.023536 20870 leader_election.cc:290] T cb4727386a3e48aca5d71cc6c611c197 P 974b802141bd4f13beffef2aa03f88f9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 197b880fa39d432a9b472845876bfaf0 (127.18.211.1:42027), 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3:37735)
I20260129 02:58:47.024298 20719 leader_election.cc:290] T 96609ac68bdf4993a12e1c23355ca013 P 1c50de89a5a845668696dcb28c3badd6 [CANDIDATE]: Term 1 election: Requested vote from peers 197b880fa39d432a9b472845876bfaf0 (127.18.211.1:42027), 974b802141bd4f13beffef2aa03f88f9 (127.18.211.2:34517)
I20260129 02:58:47.025231 20480 leader_election.cc:304] T 36ff33be666a4c82954731691dbcf327 P 974b802141bd4f13beffef2aa03f88f9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1c50de89a5a845668696dcb28c3badd6, 974b802141bd4f13beffef2aa03f88f9; no voters:
I20260129 02:58:47.026443 20870 raft_consensus.cc:2804] T 36ff33be666a4c82954731691dbcf327 P 974b802141bd4f13beffef2aa03f88f9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260129 02:58:47.026805 20870 raft_consensus.cc:493] T 36ff33be666a4c82954731691dbcf327 P 974b802141bd4f13beffef2aa03f88f9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260129 02:58:47.027132 20870 raft_consensus.cc:3060] T 36ff33be666a4c82954731691dbcf327 P 974b802141bd4f13beffef2aa03f88f9 [term 0 FOLLOWER]: Advancing to term 1
I20260129 02:58:47.031857 20439 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "cb4727386a3e48aca5d71cc6c611c197" candidate_uuid: "974b802141bd4f13beffef2aa03f88f9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "197b880fa39d432a9b472845876bfaf0" is_pre_election: true
I20260129 02:58:47.033103 20439 raft_consensus.cc:2468] T cb4727386a3e48aca5d71cc6c611c197 P 197b880fa39d432a9b472845876bfaf0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 974b802141bd4f13beffef2aa03f88f9 in term 0.
I20260129 02:58:47.033427 20630 tablet_bootstrap.cc:654] T e33a417cf9c54cf5af73a4ee3aac4d3a P 974b802141bd4f13beffef2aa03f88f9: Neither blocks nor log segments found. Creating new log.
W20260129 02:58:47.038722 20556 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.211.2:34517, user_credentials={real_user=slave}} blocked reactor thread for 56796.7us
I20260129 02:58:47.042243 20588 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "368953c9102643b2a21f99af36b51905" candidate_uuid: "197b880fa39d432a9b472845876bfaf0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c50de89a5a845668696dcb28c3badd6" is_pre_election: true
I20260129 02:58:47.046712 20511 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "96609ac68bdf4993a12e1c23355ca013" candidate_uuid: "1c50de89a5a845668696dcb28c3badd6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "974b802141bd4f13beffef2aa03f88f9"
I20260129 02:58:47.046974 20829 consensus_queue.cc:1048] T 368953c9102643b2a21f99af36b51905 P 1c50de89a5a845668696dcb28c3badd6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260129 02:58:47.047492 20511 raft_consensus.cc:3060] T 96609ac68bdf4993a12e1c23355ca013 P 974b802141bd4f13beffef2aa03f88f9 [term 0 FOLLOWER]: Advancing to term 1
I20260129 02:58:47.042773 20641 consensus_queue.cc:1048] T 75b0b6e82ddc4f92b1fe1d94f9a0b73b P 1c50de89a5a845668696dcb28c3badd6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260129 02:58:47.053356 20586 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "cb4727386a3e48aca5d71cc6c611c197" candidate_uuid: "974b802141bd4f13beffef2aa03f88f9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c50de89a5a845668696dcb28c3badd6" is_pre_election: true
I20260129 02:58:47.054343 20405 leader_election.cc:304] T 368953c9102643b2a21f99af36b51905 P 197b880fa39d432a9b472845876bfaf0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 197b880fa39d432a9b472845876bfaf0; no voters: 1c50de89a5a845668696dcb28c3badd6, 974b802141bd4f13beffef2aa03f88f9
W20260129 02:58:47.055161 20480 leader_election.cc:343] T cb4727386a3e48aca5d71cc6c611c197 P 974b802141bd4f13beffef2aa03f88f9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3:37735): Illegal state: must be running to vote when last-logged opid is not known
I20260129 02:58:47.055994 20828 raft_consensus.cc:3060] T 368953c9102643b2a21f99af36b51905 P 197b880fa39d432a9b472845876bfaf0 [term 0 FOLLOWER]: Advancing to term 1
W20260129 02:58:47.103231 20479 leader_election.cc:343] T 36ff33be666a4c82954731691dbcf327 P 974b802141bd4f13beffef2aa03f88f9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 197b880fa39d432a9b472845876bfaf0 (127.18.211.1:42027): Illegal state: must be running to vote when last-logged opid is not known
I20260129 02:58:47.106423 20479 leader_election.cc:304] T cb4727386a3e48aca5d71cc6c611c197 P 974b802141bd4f13beffef2aa03f88f9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 197b880fa39d432a9b472845876bfaf0, 974b802141bd4f13beffef2aa03f88f9; no voters: 1c50de89a5a845668696dcb28c3badd6
I20260129 02:58:47.109238 20631 ts_tablet_manager.cc:1434] T 36ff33be666a4c82954731691dbcf327 P 1c50de89a5a845668696dcb28c3badd6: Time spent starting tablet: real 0.252s user 0.001s sys 0.017s
I20260129 02:58:47.111896 20828 raft_consensus.cc:2749] T 368953c9102643b2a21f99af36b51905 P 197b880fa39d432a9b472845876bfaf0 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260129 02:58:47.113148 20631 tablet_bootstrap.cc:492] T cb4727386a3e48aca5d71cc6c611c197 P 1c50de89a5a845668696dcb28c3badd6: Bootstrap starting.
I20260129 02:58:47.155390 20511 raft_consensus.cc:2468] T 96609ac68bdf4993a12e1c23355ca013 P 974b802141bd4f13beffef2aa03f88f9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1c50de89a5a845668696dcb28c3badd6 in term 1.
I20260129 02:58:47.156706 20869 raft_consensus.cc:2804] T cb4727386a3e48aca5d71cc6c611c197 P 974b802141bd4f13beffef2aa03f88f9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260129 02:58:47.157161 20869 raft_consensus.cc:493] T cb4727386a3e48aca5d71cc6c611c197 P 974b802141bd4f13beffef2aa03f88f9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260129 02:58:47.157075 20556 leader_election.cc:304] T 96609ac68bdf4993a12e1c23355ca013 P 1c50de89a5a845668696dcb28c3badd6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1c50de89a5a845668696dcb28c3badd6, 974b802141bd4f13beffef2aa03f88f9; no voters:
I20260129 02:58:47.157692 20869 raft_consensus.cc:3060] T cb4727386a3e48aca5d71cc6c611c197 P 974b802141bd4f13beffef2aa03f88f9 [term 0 FOLLOWER]: Advancing to term 1
I20260129 02:58:47.158394 20840 raft_consensus.cc:2804] T 96609ac68bdf4993a12e1c23355ca013 P 1c50de89a5a845668696dcb28c3badd6 [term 1 FOLLOWER]: Leader election won for term 1
I20260129 02:58:47.159775 20630 tablet_bootstrap.cc:492] T e33a417cf9c54cf5af73a4ee3aac4d3a P 974b802141bd4f13beffef2aa03f88f9: No bootstrap required, opened a new log
I20260129 02:58:47.160831 20630 ts_tablet_manager.cc:1403] T e33a417cf9c54cf5af73a4ee3aac4d3a P 974b802141bd4f13beffef2aa03f88f9: Time spent bootstrapping tablet: real 0.144s user 0.016s sys 0.005s
I20260129 02:58:47.162822 20633 tablet_bootstrap.cc:492] T 96609ac68bdf4993a12e1c23355ca013 P 197b880fa39d432a9b472845876bfaf0: No bootstrap required, opened a new log
I20260129 02:58:47.163298 20633 ts_tablet_manager.cc:1403] T 96609ac68bdf4993a12e1c23355ca013 P 197b880fa39d432a9b472845876bfaf0: Time spent bootstrapping tablet: real 0.322s user 0.036s sys 0.045s
I20260129 02:58:47.161717 20840 raft_consensus.cc:697] T 96609ac68bdf4993a12e1c23355ca013 P 1c50de89a5a845668696dcb28c3badd6 [term 1 LEADER]: Becoming Leader. State: Replica: 1c50de89a5a845668696dcb28c3badd6, State: Running, Role: LEADER
I20260129 02:58:47.172731 20435 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "96609ac68bdf4993a12e1c23355ca013" candidate_uuid: "1c50de89a5a845668696dcb28c3badd6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "197b880fa39d432a9b472845876bfaf0"
I20260129 02:58:47.173682 20840 consensus_queue.cc:237] T 96609ac68bdf4993a12e1c23355ca013 P 1c50de89a5a845668696dcb28c3badd6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:58:47.171762 20870 raft_consensus.cc:515] T 36ff33be666a4c82954731691dbcf327 P 974b802141bd4f13beffef2aa03f88f9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:58:47.191365 20337 catalog_manager.cc:5654] T 505580c93c694bbb8ea96fdf9b17ab46 P 1c50de89a5a845668696dcb28c3badd6 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3). New cstate: current_term: 1 leader_uuid: "1c50de89a5a845668696dcb28c3badd6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } health_report { overall_health: HEALTHY } } }
I20260129 02:58:47.195364 20590 raft_consensus.cc:3060] T 4408006eb5174de1896b037cf3f18457 P 1c50de89a5a845668696dcb28c3badd6 [term 0 FOLLOWER]: Advancing to term 1
I20260129 02:58:47.201198 20633 raft_consensus.cc:359] T 96609ac68bdf4993a12e1c23355ca013 P 197b880fa39d432a9b472845876bfaf0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:58:47.201979 20633 raft_consensus.cc:385] T 96609ac68bdf4993a12e1c23355ca013 P 197b880fa39d432a9b472845876bfaf0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260129 02:58:47.202237 20633 raft_consensus.cc:740] T 96609ac68bdf4993a12e1c23355ca013 P 197b880fa39d432a9b472845876bfaf0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 197b880fa39d432a9b472845876bfaf0, State: Initialized, Role: FOLLOWER
I20260129 02:58:47.202841 20633 consensus_queue.cc:260] T 96609ac68bdf4993a12e1c23355ca013 P 197b880fa39d432a9b472845876bfaf0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:58:47.203183 20881 consensus_queue.cc:1048] T 75b0b6e82ddc4f92b1fe1d94f9a0b73b P 1c50de89a5a845668696dcb28c3badd6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260129 02:58:47.204875 20587 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "36ff33be666a4c82954731691dbcf327" candidate_uuid: "974b802141bd4f13beffef2aa03f88f9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c50de89a5a845668696dcb28c3badd6"
I20260129 02:58:47.205435 20587 raft_consensus.cc:3060] T 36ff33be666a4c82954731691dbcf327 P 1c50de89a5a845668696dcb28c3badd6 [term 0 FOLLOWER]: Advancing to term 1
I20260129 02:58:47.207129 20435 raft_consensus.cc:3060] T 96609ac68bdf4993a12e1c23355ca013 P 197b880fa39d432a9b472845876bfaf0 [term 0 FOLLOWER]: Advancing to term 1
I20260129 02:58:47.195652 20870 leader_election.cc:290] T 36ff33be666a4c82954731691dbcf327 P 974b802141bd4f13beffef2aa03f88f9 [CANDIDATE]: Term 1 election: Requested vote from peers 197b880fa39d432a9b472845876bfaf0 (127.18.211.1:42027), 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3:37735)
I20260129 02:58:47.216514 20633 ts_tablet_manager.cc:1434] T 96609ac68bdf4993a12e1c23355ca013 P 197b880fa39d432a9b472845876bfaf0: Time spent starting tablet: real 0.053s user 0.008s sys 0.000s
I20260129 02:58:47.217574 20633 tablet_bootstrap.cc:492] T 36ff33be666a4c82954731691dbcf327 P 197b880fa39d432a9b472845876bfaf0: Bootstrap starting.
W20260129 02:58:47.222573 20405 consensus_peers.cc:597] T c2633a6fc34c44579de348d208b0c01e P 197b880fa39d432a9b472845876bfaf0 -> Peer 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3:37735): Couldn't send request to peer 1c50de89a5a845668696dcb28c3badd6. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20260129 02:58:47.219722 20440 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "36ff33be666a4c82954731691dbcf327" candidate_uuid: "974b802141bd4f13beffef2aa03f88f9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "197b880fa39d432a9b472845876bfaf0"
I20260129 02:58:47.226953 20435 raft_consensus.cc:2468] T 96609ac68bdf4993a12e1c23355ca013 P 197b880fa39d432a9b472845876bfaf0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1c50de89a5a845668696dcb28c3badd6 in term 1.
I20260129 02:58:47.229719 20869 raft_consensus.cc:515] T cb4727386a3e48aca5d71cc6c611c197 P 974b802141bd4f13beffef2aa03f88f9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
W20260129 02:58:47.231810 20479 leader_election.cc:343] T 36ff33be666a4c82954731691dbcf327 P 974b802141bd4f13beffef2aa03f88f9 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 197b880fa39d432a9b472845876bfaf0 (127.18.211.1:42027): Illegal state: must be running to vote when last-logged opid is not known
I20260129 02:58:47.234081 20739 consensus_queue.cc:1048] T 4408006eb5174de1896b037cf3f18457 P 974b802141bd4f13beffef2aa03f88f9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260129 02:58:47.235715 20832 consensus_queue.cc:1048] T 368953c9102643b2a21f99af36b51905 P 1c50de89a5a845668696dcb28c3badd6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260129 02:58:47.236384 20590 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "cb4727386a3e48aca5d71cc6c611c197" candidate_uuid: "974b802141bd4f13beffef2aa03f88f9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c50de89a5a845668696dcb28c3badd6"
I20260129 02:58:47.237159 20869 leader_election.cc:290] T cb4727386a3e48aca5d71cc6c611c197 P 974b802141bd4f13beffef2aa03f88f9 [CANDIDATE]: Term 1 election: Requested vote from peers 197b880fa39d432a9b472845876bfaf0 (127.18.211.1:42027), 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3:37735)
W20260129 02:58:47.240624 20480 leader_election.cc:343] T cb4727386a3e48aca5d71cc6c611c197 P 974b802141bd4f13beffef2aa03f88f9 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3:37735): Illegal state: must be running to vote when last-logged opid is not known
I20260129 02:58:47.243955 20631 tablet_bootstrap.cc:654] T cb4727386a3e48aca5d71cc6c611c197 P 1c50de89a5a845668696dcb28c3badd6: Neither blocks nor log segments found. Creating new log.
I20260129 02:58:47.239802 20435 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "cb4727386a3e48aca5d71cc6c611c197" candidate_uuid: "974b802141bd4f13beffef2aa03f88f9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "197b880fa39d432a9b472845876bfaf0"
I20260129 02:58:47.251329 20435 raft_consensus.cc:3060] T cb4727386a3e48aca5d71cc6c611c197 P 197b880fa39d432a9b472845876bfaf0 [term 0 FOLLOWER]: Advancing to term 1
I20260129 02:58:47.251183 20587 raft_consensus.cc:2468] T 36ff33be666a4c82954731691dbcf327 P 1c50de89a5a845668696dcb28c3badd6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 974b802141bd4f13beffef2aa03f88f9 in term 1.
I20260129 02:58:47.264066 20480 leader_election.cc:304] T 36ff33be666a4c82954731691dbcf327 P 974b802141bd4f13beffef2aa03f88f9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1c50de89a5a845668696dcb28c3badd6, 974b802141bd4f13beffef2aa03f88f9; no voters: 197b880fa39d432a9b472845876bfaf0
I20260129 02:58:47.269608 20630 raft_consensus.cc:359] T e33a417cf9c54cf5af73a4ee3aac4d3a P 974b802141bd4f13beffef2aa03f88f9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:58:47.269989 20869 raft_consensus.cc:2804] T 36ff33be666a4c82954731691dbcf327 P 974b802141bd4f13beffef2aa03f88f9 [term 1 FOLLOWER]: Leader election won for term 1
I20260129 02:58:47.270387 20630 raft_consensus.cc:385] T e33a417cf9c54cf5af73a4ee3aac4d3a P 974b802141bd4f13beffef2aa03f88f9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260129 02:58:47.285617 20630 raft_consensus.cc:740] T e33a417cf9c54cf5af73a4ee3aac4d3a P 974b802141bd4f13beffef2aa03f88f9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 974b802141bd4f13beffef2aa03f88f9, State: Initialized, Role: FOLLOWER
I20260129 02:58:47.286609 20630 consensus_queue.cc:260] T e33a417cf9c54cf5af73a4ee3aac4d3a P 974b802141bd4f13beffef2aa03f88f9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:58:47.293676 20633 tablet_bootstrap.cc:654] T 36ff33be666a4c82954731691dbcf327 P 197b880fa39d432a9b472845876bfaf0: Neither blocks nor log segments found. Creating new log.
I20260129 02:58:47.313190 20869 raft_consensus.cc:697] T 36ff33be666a4c82954731691dbcf327 P 974b802141bd4f13beffef2aa03f88f9 [term 1 LEADER]: Becoming Leader. State: Replica: 974b802141bd4f13beffef2aa03f88f9, State: Running, Role: LEADER
I20260129 02:58:47.318451 20630 ts_tablet_manager.cc:1434] T e33a417cf9c54cf5af73a4ee3aac4d3a P 974b802141bd4f13beffef2aa03f88f9: Time spent starting tablet: real 0.157s user 0.003s sys 0.005s
I20260129 02:58:47.319690 20630 tablet_bootstrap.cc:492] T 0340229d521b432580ae926f6062e3f7 P 974b802141bd4f13beffef2aa03f88f9: Bootstrap starting.
I20260129 02:58:47.325026 20828 consensus_queue.cc:1048] T c2633a6fc34c44579de348d208b0c01e P 197b880fa39d432a9b472845876bfaf0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260129 02:58:47.327996 20869 consensus_queue.cc:237] T 36ff33be666a4c82954731691dbcf327 P 974b802141bd4f13beffef2aa03f88f9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:58:47.335991 20337 catalog_manager.cc:5654] T 96609ac68bdf4993a12e1c23355ca013 P 1c50de89a5a845668696dcb28c3badd6 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3). New cstate: current_term: 1 leader_uuid: "1c50de89a5a845668696dcb28c3badd6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } health_report { overall_health: HEALTHY } } }
I20260129 02:58:47.344478 20633 tablet_bootstrap.cc:492] T 36ff33be666a4c82954731691dbcf327 P 197b880fa39d432a9b472845876bfaf0: No bootstrap required, opened a new log
I20260129 02:58:47.345036 20633 ts_tablet_manager.cc:1403] T 36ff33be666a4c82954731691dbcf327 P 197b880fa39d432a9b472845876bfaf0: Time spent bootstrapping tablet: real 0.128s user 0.012s sys 0.000s
I20260129 02:58:47.369151 20891 consensus_queue.cc:1048] T 4408006eb5174de1896b037cf3f18457 P 974b802141bd4f13beffef2aa03f88f9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.006s
I20260129 02:58:47.375028 20869 raft_consensus.cc:493] T e33a417cf9c54cf5af73a4ee3aac4d3a P 974b802141bd4f13beffef2aa03f88f9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260129 02:58:47.375604 20869 raft_consensus.cc:515] T e33a417cf9c54cf5af73a4ee3aac4d3a P 974b802141bd4f13beffef2aa03f88f9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:58:47.378741 20869 leader_election.cc:290] T e33a417cf9c54cf5af73a4ee3aac4d3a P 974b802141bd4f13beffef2aa03f88f9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 197b880fa39d432a9b472845876bfaf0 (127.18.211.1:42027), 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3:37735)
I20260129 02:58:47.379648 20631 tablet_bootstrap.cc:492] T cb4727386a3e48aca5d71cc6c611c197 P 1c50de89a5a845668696dcb28c3badd6: No bootstrap required, opened a new log
I20260129 02:58:47.380112 20631 ts_tablet_manager.cc:1403] T cb4727386a3e48aca5d71cc6c611c197 P 1c50de89a5a845668696dcb28c3badd6: Time spent bootstrapping tablet: real 0.270s user 0.014s sys 0.010s
I20260129 02:58:47.379904 20633 raft_consensus.cc:359] T 36ff33be666a4c82954731691dbcf327 P 197b880fa39d432a9b472845876bfaf0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:58:47.380589 20633 raft_consensus.cc:385] T 36ff33be666a4c82954731691dbcf327 P 197b880fa39d432a9b472845876bfaf0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260129 02:58:47.380856 20633 raft_consensus.cc:740] T 36ff33be666a4c82954731691dbcf327 P 197b880fa39d432a9b472845876bfaf0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 197b880fa39d432a9b472845876bfaf0, State: Initialized, Role: FOLLOWER
I20260129 02:58:47.381441 20633 consensus_queue.cc:260] T 36ff33be666a4c82954731691dbcf327 P 197b880fa39d432a9b472845876bfaf0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:58:47.383453 20435 raft_consensus.cc:2468] T cb4727386a3e48aca5d71cc6c611c197 P 197b880fa39d432a9b472845876bfaf0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 974b802141bd4f13beffef2aa03f88f9 in term 1.
I20260129 02:58:47.385073 20631 raft_consensus.cc:359] T cb4727386a3e48aca5d71cc6c611c197 P 1c50de89a5a845668696dcb28c3badd6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:58:47.386940 20631 raft_consensus.cc:385] T cb4727386a3e48aca5d71cc6c611c197 P 1c50de89a5a845668696dcb28c3badd6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260129 02:58:47.387148 20631 raft_consensus.cc:740] T cb4727386a3e48aca5d71cc6c611c197 P 1c50de89a5a845668696dcb28c3badd6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1c50de89a5a845668696dcb28c3badd6, State: Initialized, Role: FOLLOWER
I20260129 02:58:47.388386 20587 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e33a417cf9c54cf5af73a4ee3aac4d3a" candidate_uuid: "974b802141bd4f13beffef2aa03f88f9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c50de89a5a845668696dcb28c3badd6" is_pre_election: true
I20260129 02:58:47.388849 20631 consensus_queue.cc:260] T cb4727386a3e48aca5d71cc6c611c197 P 1c50de89a5a845668696dcb28c3badd6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
W20260129 02:58:47.389703 20480 leader_election.cc:343] T e33a417cf9c54cf5af73a4ee3aac4d3a P 974b802141bd4f13beffef2aa03f88f9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3:37735): Illegal state: must be running to vote when last-logged opid is not known
I20260129 02:58:47.390285 20440 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e33a417cf9c54cf5af73a4ee3aac4d3a" candidate_uuid: "974b802141bd4f13beffef2aa03f88f9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "197b880fa39d432a9b472845876bfaf0" is_pre_election: true
I20260129 02:58:47.390925 20631 ts_tablet_manager.cc:1434] T cb4727386a3e48aca5d71cc6c611c197 P 1c50de89a5a845668696dcb28c3badd6: Time spent starting tablet: real 0.011s user 0.002s sys 0.003s
W20260129 02:58:47.394460 20479 leader_election.cc:343] T e33a417cf9c54cf5af73a4ee3aac4d3a P 974b802141bd4f13beffef2aa03f88f9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 197b880fa39d432a9b472845876bfaf0 (127.18.211.1:42027): Illegal state: must be running to vote when last-logged opid is not known
I20260129 02:58:47.394898 20479 leader_election.cc:304] T e33a417cf9c54cf5af73a4ee3aac4d3a P 974b802141bd4f13beffef2aa03f88f9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 974b802141bd4f13beffef2aa03f88f9; no voters: 197b880fa39d432a9b472845876bfaf0, 1c50de89a5a845668696dcb28c3badd6
I20260129 02:58:47.395205 20633 ts_tablet_manager.cc:1434] T 36ff33be666a4c82954731691dbcf327 P 197b880fa39d432a9b472845876bfaf0: Time spent starting tablet: real 0.050s user 0.006s sys 0.000s
I20260129 02:58:47.397238 20631 tablet_bootstrap.cc:492] T 7a9a69cd82724b1faed1d10aa091fc4a P 1c50de89a5a845668696dcb28c3badd6: Bootstrap starting.
I20260129 02:58:47.397359 20633 tablet_bootstrap.cc:492] T 0340229d521b432580ae926f6062e3f7 P 197b880fa39d432a9b472845876bfaf0: Bootstrap starting.
I20260129 02:58:47.399948 20479 leader_election.cc:304] T cb4727386a3e48aca5d71cc6c611c197 P 974b802141bd4f13beffef2aa03f88f9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 197b880fa39d432a9b472845876bfaf0, 974b802141bd4f13beffef2aa03f88f9; no voters: 1c50de89a5a845668696dcb28c3badd6
I20260129 02:58:47.396612 20891 raft_consensus.cc:2749] T e33a417cf9c54cf5af73a4ee3aac4d3a P 974b802141bd4f13beffef2aa03f88f9 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260129 02:58:47.403280 20740 raft_consensus.cc:2804] T cb4727386a3e48aca5d71cc6c611c197 P 974b802141bd4f13beffef2aa03f88f9 [term 1 FOLLOWER]: Leader election won for term 1
I20260129 02:58:47.405146 20740 raft_consensus.cc:697] T cb4727386a3e48aca5d71cc6c611c197 P 974b802141bd4f13beffef2aa03f88f9 [term 1 LEADER]: Becoming Leader. State: Replica: 974b802141bd4f13beffef2aa03f88f9, State: Running, Role: LEADER
I20260129 02:58:47.406728 20630 tablet_bootstrap.cc:654] T 0340229d521b432580ae926f6062e3f7 P 974b802141bd4f13beffef2aa03f88f9: Neither blocks nor log segments found. Creating new log.
I20260129 02:58:47.405877 20740 consensus_queue.cc:237] T cb4727386a3e48aca5d71cc6c611c197 P 974b802141bd4f13beffef2aa03f88f9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
W20260129 02:58:47.413024 20886 log.cc:927] Time spent T 4408006eb5174de1896b037cf3f18457 P 1c50de89a5a845668696dcb28c3badd6: Append to log took a long time: real 0.066s user 0.003s sys 0.000s
I20260129 02:58:47.421350 20337 catalog_manager.cc:5654] T 36ff33be666a4c82954731691dbcf327 P 974b802141bd4f13beffef2aa03f88f9 reported cstate change: term changed from 0 to 1, leader changed from <none> to 974b802141bd4f13beffef2aa03f88f9 (127.18.211.2). New cstate: current_term: 1 leader_uuid: "974b802141bd4f13beffef2aa03f88f9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } health_report { overall_health: UNKNOWN } } }
I20260129 02:58:47.448009 20631 tablet_bootstrap.cc:654] T 7a9a69cd82724b1faed1d10aa091fc4a P 1c50de89a5a845668696dcb28c3badd6: Neither blocks nor log segments found. Creating new log.
I20260129 02:58:47.446923 20633 tablet_bootstrap.cc:654] T 0340229d521b432580ae926f6062e3f7 P 197b880fa39d432a9b472845876bfaf0: Neither blocks nor log segments found. Creating new log.
I20260129 02:58:47.465380 20335 catalog_manager.cc:5654] T cb4727386a3e48aca5d71cc6c611c197 P 974b802141bd4f13beffef2aa03f88f9 reported cstate change: term changed from 0 to 1, leader changed from <none> to 974b802141bd4f13beffef2aa03f88f9 (127.18.211.2). New cstate: current_term: 1 leader_uuid: "974b802141bd4f13beffef2aa03f88f9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } health_report { overall_health: UNKNOWN } } }
I20260129 02:58:47.478130 20641 consensus_queue.cc:1048] T 505580c93c694bbb8ea96fdf9b17ab46 P 1c50de89a5a845668696dcb28c3badd6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260129 02:58:47.478827 20633 tablet_bootstrap.cc:492] T 0340229d521b432580ae926f6062e3f7 P 197b880fa39d432a9b472845876bfaf0: No bootstrap required, opened a new log
I20260129 02:58:47.479288 20633 ts_tablet_manager.cc:1403] T 0340229d521b432580ae926f6062e3f7 P 197b880fa39d432a9b472845876bfaf0: Time spent bootstrapping tablet: real 0.082s user 0.009s sys 0.008s
I20260129 02:58:47.482206 20633 raft_consensus.cc:359] T 0340229d521b432580ae926f6062e3f7 P 197b880fa39d432a9b472845876bfaf0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:58:47.482910 20633 raft_consensus.cc:385] T 0340229d521b432580ae926f6062e3f7 P 197b880fa39d432a9b472845876bfaf0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260129 02:58:47.483175 20633 raft_consensus.cc:740] T 0340229d521b432580ae926f6062e3f7 P 197b880fa39d432a9b472845876bfaf0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 197b880fa39d432a9b472845876bfaf0, State: Initialized, Role: FOLLOWER
I20260129 02:58:47.483942 20633 consensus_queue.cc:260] T 0340229d521b432580ae926f6062e3f7 P 197b880fa39d432a9b472845876bfaf0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:58:47.484679 20630 tablet_bootstrap.cc:492] T 0340229d521b432580ae926f6062e3f7 P 974b802141bd4f13beffef2aa03f88f9: No bootstrap required, opened a new log
I20260129 02:58:47.484867 20631 tablet_bootstrap.cc:492] T 7a9a69cd82724b1faed1d10aa091fc4a P 1c50de89a5a845668696dcb28c3badd6: No bootstrap required, opened a new log
I20260129 02:58:47.485136 20630 ts_tablet_manager.cc:1403] T 0340229d521b432580ae926f6062e3f7 P 974b802141bd4f13beffef2aa03f88f9: Time spent bootstrapping tablet: real 0.166s user 0.016s sys 0.025s
I20260129 02:58:47.485319 20631 ts_tablet_manager.cc:1403] T 7a9a69cd82724b1faed1d10aa091fc4a P 1c50de89a5a845668696dcb28c3badd6: Time spent bootstrapping tablet: real 0.088s user 0.008s sys 0.003s
I20260129 02:58:47.485342 20641 raft_consensus.cc:493] T cb4727386a3e48aca5d71cc6c611c197 P 1c50de89a5a845668696dcb28c3badd6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260129 02:58:47.485968 20641 raft_consensus.cc:515] T cb4727386a3e48aca5d71cc6c611c197 P 1c50de89a5a845668696dcb28c3badd6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:58:47.487529 20631 raft_consensus.cc:359] T 7a9a69cd82724b1faed1d10aa091fc4a P 1c50de89a5a845668696dcb28c3badd6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:58:47.487694 20630 raft_consensus.cc:359] T 0340229d521b432580ae926f6062e3f7 P 974b802141bd4f13beffef2aa03f88f9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:58:47.488520 20630 raft_consensus.cc:385] T 0340229d521b432580ae926f6062e3f7 P 974b802141bd4f13beffef2aa03f88f9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260129 02:58:47.488840 20630 raft_consensus.cc:740] T 0340229d521b432580ae926f6062e3f7 P 974b802141bd4f13beffef2aa03f88f9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 974b802141bd4f13beffef2aa03f88f9, State: Initialized, Role: FOLLOWER
I20260129 02:58:47.488989 20441 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "cb4727386a3e48aca5d71cc6c611c197" candidate_uuid: "1c50de89a5a845668696dcb28c3badd6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "197b880fa39d432a9b472845876bfaf0" is_pre_election: true
I20260129 02:58:47.489899 20441 raft_consensus.cc:2393] T cb4727386a3e48aca5d71cc6c611c197 P 197b880fa39d432a9b472845876bfaf0 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1c50de89a5a845668696dcb28c3badd6 in current term 1: Already voted for candidate 974b802141bd4f13beffef2aa03f88f9 in this term.
I20260129 02:58:47.489712 20630 consensus_queue.cc:260] T 0340229d521b432580ae926f6062e3f7 P 974b802141bd4f13beffef2aa03f88f9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:58:47.489988 20516 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "cb4727386a3e48aca5d71cc6c611c197" candidate_uuid: "1c50de89a5a845668696dcb28c3badd6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "974b802141bd4f13beffef2aa03f88f9" is_pre_election: true
I20260129 02:58:47.492120 20630 ts_tablet_manager.cc:1434] T 0340229d521b432580ae926f6062e3f7 P 974b802141bd4f13beffef2aa03f88f9: Time spent starting tablet: real 0.007s user 0.007s sys 0.000s
I20260129 02:58:47.493278 20641 leader_election.cc:290] T cb4727386a3e48aca5d71cc6c611c197 P 1c50de89a5a845668696dcb28c3badd6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 197b880fa39d432a9b472845876bfaf0 (127.18.211.1:42027), 974b802141bd4f13beffef2aa03f88f9 (127.18.211.2:34517)
I20260129 02:58:47.488209 20631 raft_consensus.cc:385] T 7a9a69cd82724b1faed1d10aa091fc4a P 1c50de89a5a845668696dcb28c3badd6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260129 02:58:47.494597 20631 raft_consensus.cc:740] T 7a9a69cd82724b1faed1d10aa091fc4a P 1c50de89a5a845668696dcb28c3badd6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1c50de89a5a845668696dcb28c3badd6, State: Initialized, Role: FOLLOWER
I20260129 02:58:47.495258 20631 consensus_queue.cc:260] T 7a9a69cd82724b1faed1d10aa091fc4a P 1c50de89a5a845668696dcb28c3badd6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:58:47.497354 20631 ts_tablet_manager.cc:1434] T 7a9a69cd82724b1faed1d10aa091fc4a P 1c50de89a5a845668696dcb28c3badd6: Time spent starting tablet: real 0.012s user 0.003s sys 0.002s
I20260129 02:58:47.493373 20556 leader_election.cc:304] T cb4727386a3e48aca5d71cc6c611c197 P 1c50de89a5a845668696dcb28c3badd6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1c50de89a5a845668696dcb28c3badd6; no voters: 197b880fa39d432a9b472845876bfaf0, 974b802141bd4f13beffef2aa03f88f9
I20260129 02:58:47.498440 20631 tablet_bootstrap.cc:492] T e33a417cf9c54cf5af73a4ee3aac4d3a P 1c50de89a5a845668696dcb28c3badd6: Bootstrap starting.
I20260129 02:58:47.498749 20829 raft_consensus.cc:3060] T cb4727386a3e48aca5d71cc6c611c197 P 1c50de89a5a845668696dcb28c3badd6 [term 0 FOLLOWER]: Advancing to term 1
I20260129 02:58:47.498842 20633 ts_tablet_manager.cc:1434] T 0340229d521b432580ae926f6062e3f7 P 197b880fa39d432a9b472845876bfaf0: Time spent starting tablet: real 0.019s user 0.007s sys 0.000s
I20260129 02:58:47.504261 20633 tablet_bootstrap.cc:492] T a644c02b16af47679824ccd060ac10fd P 197b880fa39d432a9b472845876bfaf0: Bootstrap starting.
I20260129 02:58:47.508648 20641 consensus_queue.cc:1048] T 505580c93c694bbb8ea96fdf9b17ab46 P 1c50de89a5a845668696dcb28c3badd6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260129 02:58:47.509918 20633 tablet_bootstrap.cc:654] T a644c02b16af47679824ccd060ac10fd P 197b880fa39d432a9b472845876bfaf0: Neither blocks nor log segments found. Creating new log.
I20260129 02:58:47.515573 20829 raft_consensus.cc:2749] T cb4727386a3e48aca5d71cc6c611c197 P 1c50de89a5a845668696dcb28c3badd6 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260129 02:58:47.516067 20631 tablet_bootstrap.cc:654] T e33a417cf9c54cf5af73a4ee3aac4d3a P 1c50de89a5a845668696dcb28c3badd6: Neither blocks nor log segments found. Creating new log.
I20260129 02:58:47.517325 20630 tablet_bootstrap.cc:492] T cb0b3f259ebc4310927338a8740e87b8 P 974b802141bd4f13beffef2aa03f88f9: Bootstrap starting.
I20260129 02:58:47.529770 20633 tablet_bootstrap.cc:492] T a644c02b16af47679824ccd060ac10fd P 197b880fa39d432a9b472845876bfaf0: No bootstrap required, opened a new log
I20260129 02:58:47.530227 20633 ts_tablet_manager.cc:1403] T a644c02b16af47679824ccd060ac10fd P 197b880fa39d432a9b472845876bfaf0: Time spent bootstrapping tablet: real 0.026s user 0.015s sys 0.006s
I20260129 02:58:47.533470 20630 tablet_bootstrap.cc:654] T cb0b3f259ebc4310927338a8740e87b8 P 974b802141bd4f13beffef2aa03f88f9: Neither blocks nor log segments found. Creating new log.
I20260129 02:58:47.537189 20633 raft_consensus.cc:359] T a644c02b16af47679824ccd060ac10fd P 197b880fa39d432a9b472845876bfaf0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:58:47.540745 20633 raft_consensus.cc:385] T a644c02b16af47679824ccd060ac10fd P 197b880fa39d432a9b472845876bfaf0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260129 02:58:47.541247 20633 raft_consensus.cc:740] T a644c02b16af47679824ccd060ac10fd P 197b880fa39d432a9b472845876bfaf0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 197b880fa39d432a9b472845876bfaf0, State: Initialized, Role: FOLLOWER
I20260129 02:58:47.541505 20828 raft_consensus.cc:493] T 0340229d521b432580ae926f6062e3f7 P 197b880fa39d432a9b472845876bfaf0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260129 02:58:47.541980 20828 raft_consensus.cc:515] T 0340229d521b432580ae926f6062e3f7 P 197b880fa39d432a9b472845876bfaf0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:58:47.542935 20633 consensus_queue.cc:260] T a644c02b16af47679824ccd060ac10fd P 197b880fa39d432a9b472845876bfaf0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:58:47.543839 20828 leader_election.cc:290] T 0340229d521b432580ae926f6062e3f7 P 197b880fa39d432a9b472845876bfaf0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 974b802141bd4f13beffef2aa03f88f9 (127.18.211.2:34517), 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3:37735)
I20260129 02:58:47.545554 20589 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0340229d521b432580ae926f6062e3f7" candidate_uuid: "197b880fa39d432a9b472845876bfaf0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c50de89a5a845668696dcb28c3badd6" is_pre_election: true
I20260129 02:58:47.546190 20633 ts_tablet_manager.cc:1434] T a644c02b16af47679824ccd060ac10fd P 197b880fa39d432a9b472845876bfaf0: Time spent starting tablet: real 0.016s user 0.007s sys 0.002s
W20260129 02:58:47.546900 20405 leader_election.cc:343] T 0340229d521b432580ae926f6062e3f7 P 197b880fa39d432a9b472845876bfaf0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3:37735): Illegal state: must be running to vote when last-logged opid is not known
I20260129 02:58:47.547700 20514 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0340229d521b432580ae926f6062e3f7" candidate_uuid: "197b880fa39d432a9b472845876bfaf0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "974b802141bd4f13beffef2aa03f88f9" is_pre_election: true
I20260129 02:58:47.548357 20514 raft_consensus.cc:2468] T 0340229d521b432580ae926f6062e3f7 P 974b802141bd4f13beffef2aa03f88f9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 197b880fa39d432a9b472845876bfaf0 in term 0.
I20260129 02:58:47.548871 20633 tablet_bootstrap.cc:492] T a0fcc58d6fb6473cafbd9d34d5195269 P 197b880fa39d432a9b472845876bfaf0: Bootstrap starting.
I20260129 02:58:47.549468 20407 leader_election.cc:304] T 0340229d521b432580ae926f6062e3f7 P 197b880fa39d432a9b472845876bfaf0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 197b880fa39d432a9b472845876bfaf0, 974b802141bd4f13beffef2aa03f88f9; no voters: 1c50de89a5a845668696dcb28c3badd6
I20260129 02:58:47.550467 20828 raft_consensus.cc:2804] T 0340229d521b432580ae926f6062e3f7 P 197b880fa39d432a9b472845876bfaf0 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260129 02:58:47.550832 20828 raft_consensus.cc:493] T 0340229d521b432580ae926f6062e3f7 P 197b880fa39d432a9b472845876bfaf0 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260129 02:58:47.551159 20828 raft_consensus.cc:3060] T 0340229d521b432580ae926f6062e3f7 P 197b880fa39d432a9b472845876bfaf0 [term 0 FOLLOWER]: Advancing to term 1
I20260129 02:58:47.557199 20633 tablet_bootstrap.cc:654] T a0fcc58d6fb6473cafbd9d34d5195269 P 197b880fa39d432a9b472845876bfaf0: Neither blocks nor log segments found. Creating new log.
I20260129 02:58:47.560144 20630 tablet_bootstrap.cc:492] T cb0b3f259ebc4310927338a8740e87b8 P 974b802141bd4f13beffef2aa03f88f9: No bootstrap required, opened a new log
I20260129 02:58:47.560637 20630 ts_tablet_manager.cc:1403] T cb0b3f259ebc4310927338a8740e87b8 P 974b802141bd4f13beffef2aa03f88f9: Time spent bootstrapping tablet: real 0.044s user 0.016s sys 0.001s
I20260129 02:58:47.562130 20631 tablet_bootstrap.cc:492] T e33a417cf9c54cf5af73a4ee3aac4d3a P 1c50de89a5a845668696dcb28c3badd6: No bootstrap required, opened a new log
I20260129 02:58:47.563110 20631 ts_tablet_manager.cc:1403] T e33a417cf9c54cf5af73a4ee3aac4d3a P 1c50de89a5a845668696dcb28c3badd6: Time spent bootstrapping tablet: real 0.065s user 0.018s sys 0.015s
I20260129 02:58:47.563886 20828 raft_consensus.cc:515] T 0340229d521b432580ae926f6062e3f7 P 197b880fa39d432a9b472845876bfaf0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:58:47.563119 20630 raft_consensus.cc:359] T cb0b3f259ebc4310927338a8740e87b8 P 974b802141bd4f13beffef2aa03f88f9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:58:47.565698 20630 raft_consensus.cc:385] T cb0b3f259ebc4310927338a8740e87b8 P 974b802141bd4f13beffef2aa03f88f9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260129 02:58:47.566112 20630 raft_consensus.cc:740] T cb0b3f259ebc4310927338a8740e87b8 P 974b802141bd4f13beffef2aa03f88f9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 974b802141bd4f13beffef2aa03f88f9, State: Initialized, Role: FOLLOWER
I20260129 02:58:47.566680 20514 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0340229d521b432580ae926f6062e3f7" candidate_uuid: "197b880fa39d432a9b472845876bfaf0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "974b802141bd4f13beffef2aa03f88f9"
I20260129 02:58:47.566766 20631 raft_consensus.cc:359] T e33a417cf9c54cf5af73a4ee3aac4d3a P 1c50de89a5a845668696dcb28c3badd6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:58:47.567332 20514 raft_consensus.cc:3060] T 0340229d521b432580ae926f6062e3f7 P 974b802141bd4f13beffef2aa03f88f9 [term 0 FOLLOWER]: Advancing to term 1
I20260129 02:58:47.567462 20631 raft_consensus.cc:385] T e33a417cf9c54cf5af73a4ee3aac4d3a P 1c50de89a5a845668696dcb28c3badd6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260129 02:58:47.567762 20631 raft_consensus.cc:740] T e33a417cf9c54cf5af73a4ee3aac4d3a P 1c50de89a5a845668696dcb28c3badd6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1c50de89a5a845668696dcb28c3badd6, State: Initialized, Role: FOLLOWER
I20260129 02:58:47.567013 20630 consensus_queue.cc:260] T cb0b3f259ebc4310927338a8740e87b8 P 974b802141bd4f13beffef2aa03f88f9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:58:47.568444 20631 consensus_queue.cc:260] T e33a417cf9c54cf5af73a4ee3aac4d3a P 1c50de89a5a845668696dcb28c3badd6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:58:47.570652 20631 ts_tablet_manager.cc:1434] T e33a417cf9c54cf5af73a4ee3aac4d3a P 1c50de89a5a845668696dcb28c3badd6: Time spent starting tablet: real 0.007s user 0.001s sys 0.002s
I20260129 02:58:47.571548 20631 tablet_bootstrap.cc:492] T a644c02b16af47679824ccd060ac10fd P 1c50de89a5a845668696dcb28c3badd6: Bootstrap starting.
I20260129 02:58:47.582115 20630 ts_tablet_manager.cc:1434] T cb0b3f259ebc4310927338a8740e87b8 P 974b802141bd4f13beffef2aa03f88f9: Time spent starting tablet: real 0.021s user 0.006s sys 0.001s
I20260129 02:58:47.591341 20631 tablet_bootstrap.cc:654] T a644c02b16af47679824ccd060ac10fd P 1c50de89a5a845668696dcb28c3badd6: Neither blocks nor log segments found. Creating new log.
I20260129 02:58:47.588291 20514 raft_consensus.cc:2468] T 0340229d521b432580ae926f6062e3f7 P 974b802141bd4f13beffef2aa03f88f9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 197b880fa39d432a9b472845876bfaf0 in term 1.
I20260129 02:58:47.594287 20633 tablet_bootstrap.cc:492] T a0fcc58d6fb6473cafbd9d34d5195269 P 197b880fa39d432a9b472845876bfaf0: No bootstrap required, opened a new log
I20260129 02:58:47.594738 20633 ts_tablet_manager.cc:1403] T a0fcc58d6fb6473cafbd9d34d5195269 P 197b880fa39d432a9b472845876bfaf0: Time spent bootstrapping tablet: real 0.046s user 0.016s sys 0.022s
I20260129 02:58:47.596472 20630 tablet_bootstrap.cc:492] T 926a250fa5834e88b96ccecbefbf5086 P 974b802141bd4f13beffef2aa03f88f9: Bootstrap starting.
I20260129 02:58:47.597234 20633 raft_consensus.cc:359] T a0fcc58d6fb6473cafbd9d34d5195269 P 197b880fa39d432a9b472845876bfaf0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:58:47.597904 20633 raft_consensus.cc:385] T a0fcc58d6fb6473cafbd9d34d5195269 P 197b880fa39d432a9b472845876bfaf0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260129 02:58:47.598160 20633 raft_consensus.cc:740] T a0fcc58d6fb6473cafbd9d34d5195269 P 197b880fa39d432a9b472845876bfaf0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 197b880fa39d432a9b472845876bfaf0, State: Initialized, Role: FOLLOWER
I20260129 02:58:47.598760 20633 consensus_queue.cc:260] T a0fcc58d6fb6473cafbd9d34d5195269 P 197b880fa39d432a9b472845876bfaf0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:58:47.601121 20633 ts_tablet_manager.cc:1434] T a0fcc58d6fb6473cafbd9d34d5195269 P 197b880fa39d432a9b472845876bfaf0: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260129 02:58:47.602051 20633 tablet_bootstrap.cc:492] T 7a9a69cd82724b1faed1d10aa091fc4a P 197b880fa39d432a9b472845876bfaf0: Bootstrap starting.
I20260129 02:58:47.603305 20630 tablet_bootstrap.cc:654] T 926a250fa5834e88b96ccecbefbf5086 P 974b802141bd4f13beffef2aa03f88f9: Neither blocks nor log segments found. Creating new log.
I20260129 02:58:47.613344 20829 consensus_queue.cc:1048] T 96609ac68bdf4993a12e1c23355ca013 P 1c50de89a5a845668696dcb28c3badd6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260129 02:58:47.620849 20633 tablet_bootstrap.cc:654] T 7a9a69cd82724b1faed1d10aa091fc4a P 197b880fa39d432a9b472845876bfaf0: Neither blocks nor log segments found. Creating new log.
I20260129 02:58:47.623689 20631 tablet_bootstrap.cc:492] T a644c02b16af47679824ccd060ac10fd P 1c50de89a5a845668696dcb28c3badd6: No bootstrap required, opened a new log
I20260129 02:58:47.625579 20631 ts_tablet_manager.cc:1403] T a644c02b16af47679824ccd060ac10fd P 1c50de89a5a845668696dcb28c3badd6: Time spent bootstrapping tablet: real 0.054s user 0.014s sys 0.008s
I20260129 02:58:47.629900 20630 tablet_bootstrap.cc:492] T 926a250fa5834e88b96ccecbefbf5086 P 974b802141bd4f13beffef2aa03f88f9: No bootstrap required, opened a new log
I20260129 02:58:47.630532 20630 ts_tablet_manager.cc:1403] T 926a250fa5834e88b96ccecbefbf5086 P 974b802141bd4f13beffef2aa03f88f9: Time spent bootstrapping tablet: real 0.034s user 0.006s sys 0.021s
I20260129 02:58:47.631408 20631 raft_consensus.cc:359] T a644c02b16af47679824ccd060ac10fd P 1c50de89a5a845668696dcb28c3badd6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:58:47.632599 20631 raft_consensus.cc:385] T a644c02b16af47679824ccd060ac10fd P 1c50de89a5a845668696dcb28c3badd6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260129 02:58:47.632879 20631 raft_consensus.cc:740] T a644c02b16af47679824ccd060ac10fd P 1c50de89a5a845668696dcb28c3badd6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1c50de89a5a845668696dcb28c3badd6, State: Initialized, Role: FOLLOWER
I20260129 02:58:47.634785 20631 consensus_queue.cc:260] T a644c02b16af47679824ccd060ac10fd P 1c50de89a5a845668696dcb28c3badd6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:58:47.634860 20407 leader_election.cc:304] T 0340229d521b432580ae926f6062e3f7 P 197b880fa39d432a9b472845876bfaf0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 197b880fa39d432a9b472845876bfaf0, 974b802141bd4f13beffef2aa03f88f9; no voters:
I20260129 02:58:47.635885 20721 raft_consensus.cc:2804] T 0340229d521b432580ae926f6062e3f7 P 197b880fa39d432a9b472845876bfaf0 [term 1 FOLLOWER]: Leader election won for term 1
I20260129 02:58:47.636375 20721 raft_consensus.cc:697] T 0340229d521b432580ae926f6062e3f7 P 197b880fa39d432a9b472845876bfaf0 [term 1 LEADER]: Becoming Leader. State: Replica: 197b880fa39d432a9b472845876bfaf0, State: Running, Role: LEADER
I20260129 02:58:47.639420 20630 raft_consensus.cc:359] T 926a250fa5834e88b96ccecbefbf5086 P 974b802141bd4f13beffef2aa03f88f9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:58:47.639356 20828 leader_election.cc:290] T 0340229d521b432580ae926f6062e3f7 P 197b880fa39d432a9b472845876bfaf0 [CANDIDATE]: Term 1 election: Requested vote from peers 974b802141bd4f13beffef2aa03f88f9 (127.18.211.2:34517), 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3:37735)
I20260129 02:58:47.640414 20630 raft_consensus.cc:385] T 926a250fa5834e88b96ccecbefbf5086 P 974b802141bd4f13beffef2aa03f88f9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260129 02:58:47.640880 20630 raft_consensus.cc:740] T 926a250fa5834e88b96ccecbefbf5086 P 974b802141bd4f13beffef2aa03f88f9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 974b802141bd4f13beffef2aa03f88f9, State: Initialized, Role: FOLLOWER
I20260129 02:58:47.641829 20630 consensus_queue.cc:260] T 926a250fa5834e88b96ccecbefbf5086 P 974b802141bd4f13beffef2aa03f88f9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:58:47.648736 20829 raft_consensus.cc:493] T e33a417cf9c54cf5af73a4ee3aac4d3a P 1c50de89a5a845668696dcb28c3badd6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260129 02:58:47.649240 20829 raft_consensus.cc:515] T e33a417cf9c54cf5af73a4ee3aac4d3a P 1c50de89a5a845668696dcb28c3badd6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:58:47.651223 20829 leader_election.cc:290] T e33a417cf9c54cf5af73a4ee3aac4d3a P 1c50de89a5a845668696dcb28c3badd6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 197b880fa39d432a9b472845876bfaf0 (127.18.211.1:42027), 974b802141bd4f13beffef2aa03f88f9 (127.18.211.2:34517)
I20260129 02:58:47.652921 20438 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e33a417cf9c54cf5af73a4ee3aac4d3a" candidate_uuid: "1c50de89a5a845668696dcb28c3badd6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "197b880fa39d432a9b472845876bfaf0" is_pre_election: true
W20260129 02:58:47.654376 20553 leader_election.cc:343] T e33a417cf9c54cf5af73a4ee3aac4d3a P 1c50de89a5a845668696dcb28c3badd6 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 197b880fa39d432a9b472845876bfaf0 (127.18.211.1:42027): Illegal state: must be running to vote when last-logged opid is not known
W20260129 02:58:47.655959 20407 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.211.2:34517, user_credentials={real_user=slave}} blocked reactor thread for 63119.6us
I20260129 02:58:47.656111 20589 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0340229d521b432580ae926f6062e3f7" candidate_uuid: "197b880fa39d432a9b472845876bfaf0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c50de89a5a845668696dcb28c3badd6"
I20260129 02:58:47.658851 20511 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e33a417cf9c54cf5af73a4ee3aac4d3a" candidate_uuid: "1c50de89a5a845668696dcb28c3badd6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "974b802141bd4f13beffef2aa03f88f9" is_pre_election: true
I20260129 02:58:47.659471 20511 raft_consensus.cc:2468] T e33a417cf9c54cf5af73a4ee3aac4d3a P 974b802141bd4f13beffef2aa03f88f9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1c50de89a5a845668696dcb28c3badd6 in term 0.
I20260129 02:58:47.664099 20630 ts_tablet_manager.cc:1434] T 926a250fa5834e88b96ccecbefbf5086 P 974b802141bd4f13beffef2aa03f88f9: Time spent starting tablet: real 0.033s user 0.008s sys 0.001s
W20260129 02:58:47.664433 20405 leader_election.cc:343] T 0340229d521b432580ae926f6062e3f7 P 197b880fa39d432a9b472845876bfaf0 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3:37735): Illegal state: must be running to vote when last-logged opid is not known
I20260129 02:58:47.665086 20630 tablet_bootstrap.cc:492] T cedce331e1584ac8b636c9b8a72b2ba0 P 974b802141bd4f13beffef2aa03f88f9: Bootstrap starting.
I20260129 02:58:47.665757 20840 consensus_queue.cc:1048] T 96609ac68bdf4993a12e1c23355ca013 P 1c50de89a5a845668696dcb28c3badd6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260129 02:58:47.665943 20556 leader_election.cc:304] T e33a417cf9c54cf5af73a4ee3aac4d3a P 1c50de89a5a845668696dcb28c3badd6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1c50de89a5a845668696dcb28c3badd6, 974b802141bd4f13beffef2aa03f88f9; no voters: 197b880fa39d432a9b472845876bfaf0
I20260129 02:58:47.666045 20721 consensus_queue.cc:237] T 0340229d521b432580ae926f6062e3f7 P 197b880fa39d432a9b472845876bfaf0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:58:47.667768 20631 ts_tablet_manager.cc:1434] T a644c02b16af47679824ccd060ac10fd P 1c50de89a5a845668696dcb28c3badd6: Time spent starting tablet: real 0.042s user 0.006s sys 0.000s
I20260129 02:58:47.674641 20832 raft_consensus.cc:2804] T e33a417cf9c54cf5af73a4ee3aac4d3a P 1c50de89a5a845668696dcb28c3badd6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260129 02:58:47.675017 20832 raft_consensus.cc:493] T e33a417cf9c54cf5af73a4ee3aac4d3a P 1c50de89a5a845668696dcb28c3badd6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260129 02:58:47.675326 20832 raft_consensus.cc:3060] T e33a417cf9c54cf5af73a4ee3aac4d3a P 1c50de89a5a845668696dcb28c3badd6 [term 0 FOLLOWER]: Advancing to term 1
I20260129 02:58:47.677237 20631 tablet_bootstrap.cc:492] T 18f30ebaa4174fd28c958e72d0cc7e24 P 1c50de89a5a845668696dcb28c3badd6: Bootstrap starting.
I20260129 02:58:47.686390 20633 tablet_bootstrap.cc:492] T 7a9a69cd82724b1faed1d10aa091fc4a P 197b880fa39d432a9b472845876bfaf0: No bootstrap required, opened a new log
I20260129 02:58:47.686951 20633 ts_tablet_manager.cc:1403] T 7a9a69cd82724b1faed1d10aa091fc4a P 197b880fa39d432a9b472845876bfaf0: Time spent bootstrapping tablet: real 0.085s user 0.026s sys 0.025s
I20260129 02:58:47.702984 20633 raft_consensus.cc:359] T 7a9a69cd82724b1faed1d10aa091fc4a P 197b880fa39d432a9b472845876bfaf0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:58:47.703768 20633 raft_consensus.cc:385] T 7a9a69cd82724b1faed1d10aa091fc4a P 197b880fa39d432a9b472845876bfaf0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260129 02:58:47.704106 20633 raft_consensus.cc:740] T 7a9a69cd82724b1faed1d10aa091fc4a P 197b880fa39d432a9b472845876bfaf0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 197b880fa39d432a9b472845876bfaf0, State: Initialized, Role: FOLLOWER
I20260129 02:58:47.704735 20633 consensus_queue.cc:260] T 7a9a69cd82724b1faed1d10aa091fc4a P 197b880fa39d432a9b472845876bfaf0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:58:47.710847 20631 tablet_bootstrap.cc:654] T 18f30ebaa4174fd28c958e72d0cc7e24 P 1c50de89a5a845668696dcb28c3badd6: Neither blocks nor log segments found. Creating new log.
I20260129 02:58:47.713881 20828 raft_consensus.cc:493] T 7a9a69cd82724b1faed1d10aa091fc4a P 197b880fa39d432a9b472845876bfaf0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260129 02:58:47.712831 20633 ts_tablet_manager.cc:1434] T 7a9a69cd82724b1faed1d10aa091fc4a P 197b880fa39d432a9b472845876bfaf0: Time spent starting tablet: real 0.026s user 0.006s sys 0.005s
I20260129 02:58:47.714448 20828 raft_consensus.cc:515] T 7a9a69cd82724b1faed1d10aa091fc4a P 197b880fa39d432a9b472845876bfaf0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:58:47.715265 20633 tablet_bootstrap.cc:492] T e33a417cf9c54cf5af73a4ee3aac4d3a P 197b880fa39d432a9b472845876bfaf0: Bootstrap starting.
W20260129 02:58:47.715847 20387 auto_rebalancer.cc:229] Could not retrieve cluster info: Service unavailable: Tablet not running
I20260129 02:58:47.716614 20828 leader_election.cc:290] T 7a9a69cd82724b1faed1d10aa091fc4a P 197b880fa39d432a9b472845876bfaf0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 974b802141bd4f13beffef2aa03f88f9 (127.18.211.2:34517), 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3:37735)
I20260129 02:58:47.723356 20633 tablet_bootstrap.cc:654] T e33a417cf9c54cf5af73a4ee3aac4d3a P 197b880fa39d432a9b472845876bfaf0: Neither blocks nor log segments found. Creating new log.
I20260129 02:58:47.725261 20589 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7a9a69cd82724b1faed1d10aa091fc4a" candidate_uuid: "197b880fa39d432a9b472845876bfaf0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c50de89a5a845668696dcb28c3badd6" is_pre_election: true
I20260129 02:58:47.727877 20589 raft_consensus.cc:2468] T 7a9a69cd82724b1faed1d10aa091fc4a P 1c50de89a5a845668696dcb28c3badd6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 197b880fa39d432a9b472845876bfaf0 in term 0.
I20260129 02:58:47.729598 20832 raft_consensus.cc:515] T e33a417cf9c54cf5af73a4ee3aac4d3a P 1c50de89a5a845668696dcb28c3badd6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:58:47.732014 20832 leader_election.cc:290] T e33a417cf9c54cf5af73a4ee3aac4d3a P 1c50de89a5a845668696dcb28c3badd6 [CANDIDATE]: Term 1 election: Requested vote from peers 197b880fa39d432a9b472845876bfaf0 (127.18.211.1:42027), 974b802141bd4f13beffef2aa03f88f9 (127.18.211.2:34517)
I20260129 02:58:47.735826 20633 tablet_bootstrap.cc:492] T e33a417cf9c54cf5af73a4ee3aac4d3a P 197b880fa39d432a9b472845876bfaf0: No bootstrap required, opened a new log
I20260129 02:58:47.736375 20633 ts_tablet_manager.cc:1403] T e33a417cf9c54cf5af73a4ee3aac4d3a P 197b880fa39d432a9b472845876bfaf0: Time spent bootstrapping tablet: real 0.021s user 0.013s sys 0.006s
I20260129 02:58:47.736547 20630 tablet_bootstrap.cc:654] T cedce331e1584ac8b636c9b8a72b2ba0 P 974b802141bd4f13beffef2aa03f88f9: Neither blocks nor log segments found. Creating new log.
I20260129 02:58:47.738112 20405 leader_election.cc:304] T 7a9a69cd82724b1faed1d10aa091fc4a P 197b880fa39d432a9b472845876bfaf0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 197b880fa39d432a9b472845876bfaf0, 1c50de89a5a845668696dcb28c3badd6; no voters:
I20260129 02:58:47.739780 20633 raft_consensus.cc:359] T e33a417cf9c54cf5af73a4ee3aac4d3a P 197b880fa39d432a9b472845876bfaf0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:58:47.740689 20633 raft_consensus.cc:385] T e33a417cf9c54cf5af73a4ee3aac4d3a P 197b880fa39d432a9b472845876bfaf0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260129 02:58:47.741024 20633 raft_consensus.cc:740] T e33a417cf9c54cf5af73a4ee3aac4d3a P 197b880fa39d432a9b472845876bfaf0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 197b880fa39d432a9b472845876bfaf0, State: Initialized, Role: FOLLOWER
I20260129 02:58:47.741849 20633 consensus_queue.cc:260] T e33a417cf9c54cf5af73a4ee3aac4d3a P 197b880fa39d432a9b472845876bfaf0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:58:47.743189 20828 raft_consensus.cc:2804] T 7a9a69cd82724b1faed1d10aa091fc4a P 197b880fa39d432a9b472845876bfaf0 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260129 02:58:47.743664 20828 raft_consensus.cc:493] T 7a9a69cd82724b1faed1d10aa091fc4a P 197b880fa39d432a9b472845876bfaf0 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260129 02:58:47.744117 20828 raft_consensus.cc:3060] T 7a9a69cd82724b1faed1d10aa091fc4a P 197b880fa39d432a9b472845876bfaf0 [term 0 FOLLOWER]: Advancing to term 1
I20260129 02:58:47.746199 20516 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e33a417cf9c54cf5af73a4ee3aac4d3a" candidate_uuid: "1c50de89a5a845668696dcb28c3badd6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "974b802141bd4f13beffef2aa03f88f9"
I20260129 02:58:47.746816 20516 raft_consensus.cc:3060] T e33a417cf9c54cf5af73a4ee3aac4d3a P 974b802141bd4f13beffef2aa03f88f9 [term 0 FOLLOWER]: Advancing to term 1
I20260129 02:58:47.749418 20630 tablet_bootstrap.cc:492] T cedce331e1584ac8b636c9b8a72b2ba0 P 974b802141bd4f13beffef2aa03f88f9: No bootstrap required, opened a new log
I20260129 02:58:47.749890 20630 ts_tablet_manager.cc:1403] T cedce331e1584ac8b636c9b8a72b2ba0 P 974b802141bd4f13beffef2aa03f88f9: Time spent bootstrapping tablet: real 0.085s user 0.009s sys 0.000s
I20260129 02:58:47.752384 20630 raft_consensus.cc:359] T cedce331e1584ac8b636c9b8a72b2ba0 P 974b802141bd4f13beffef2aa03f88f9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:58:47.753008 20630 raft_consensus.cc:385] T cedce331e1584ac8b636c9b8a72b2ba0 P 974b802141bd4f13beffef2aa03f88f9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260129 02:58:47.753278 20630 raft_consensus.cc:740] T cedce331e1584ac8b636c9b8a72b2ba0 P 974b802141bd4f13beffef2aa03f88f9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 974b802141bd4f13beffef2aa03f88f9, State: Initialized, Role: FOLLOWER
I20260129 02:58:47.753893 20828 raft_consensus.cc:515] T 7a9a69cd82724b1faed1d10aa091fc4a P 197b880fa39d432a9b472845876bfaf0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:58:47.755888 20438 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e33a417cf9c54cf5af73a4ee3aac4d3a" candidate_uuid: "1c50de89a5a845668696dcb28c3badd6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "197b880fa39d432a9b472845876bfaf0"
I20260129 02:58:47.756142 20828 leader_election.cc:290] T 7a9a69cd82724b1faed1d10aa091fc4a P 197b880fa39d432a9b472845876bfaf0 [CANDIDATE]: Term 1 election: Requested vote from peers 974b802141bd4f13beffef2aa03f88f9 (127.18.211.2:34517), 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3:37735)
I20260129 02:58:47.756469 20438 raft_consensus.cc:3060] T e33a417cf9c54cf5af73a4ee3aac4d3a P 197b880fa39d432a9b472845876bfaf0 [term 0 FOLLOWER]: Advancing to term 1
I20260129 02:58:47.757916 20633 ts_tablet_manager.cc:1434] T e33a417cf9c54cf5af73a4ee3aac4d3a P 197b880fa39d432a9b472845876bfaf0: Time spent starting tablet: real 0.021s user 0.006s sys 0.000s
I20260129 02:58:47.758800 20633 tablet_bootstrap.cc:492] T cedce331e1584ac8b636c9b8a72b2ba0 P 197b880fa39d432a9b472845876bfaf0: Bootstrap starting.
I20260129 02:58:47.762745 20891 consensus_queue.cc:1048] T 36ff33be666a4c82954731691dbcf327 P 974b802141bd4f13beffef2aa03f88f9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260129 02:58:47.771552 20633 tablet_bootstrap.cc:654] T cedce331e1584ac8b636c9b8a72b2ba0 P 197b880fa39d432a9b472845876bfaf0: Neither blocks nor log segments found. Creating new log.
I20260129 02:58:47.778039 20633 tablet_bootstrap.cc:492] T cedce331e1584ac8b636c9b8a72b2ba0 P 197b880fa39d432a9b472845876bfaf0: No bootstrap required, opened a new log
I20260129 02:58:47.778481 20633 ts_tablet_manager.cc:1403] T cedce331e1584ac8b636c9b8a72b2ba0 P 197b880fa39d432a9b472845876bfaf0: Time spent bootstrapping tablet: real 0.020s user 0.010s sys 0.001s
I20260129 02:58:47.781608 20438 raft_consensus.cc:2468] T e33a417cf9c54cf5af73a4ee3aac4d3a P 197b880fa39d432a9b472845876bfaf0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1c50de89a5a845668696dcb28c3badd6 in term 1.
I20260129 02:58:47.781641 20633 raft_consensus.cc:359] T cedce331e1584ac8b636c9b8a72b2ba0 P 197b880fa39d432a9b472845876bfaf0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:58:47.753891 20630 consensus_queue.cc:260] T cedce331e1584ac8b636c9b8a72b2ba0 P 974b802141bd4f13beffef2aa03f88f9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:58:47.785269 20553 leader_election.cc:304] T e33a417cf9c54cf5af73a4ee3aac4d3a P 1c50de89a5a845668696dcb28c3badd6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 197b880fa39d432a9b472845876bfaf0, 1c50de89a5a845668696dcb28c3badd6; no voters:
I20260129 02:58:47.786204 20832 raft_consensus.cc:2804] T e33a417cf9c54cf5af73a4ee3aac4d3a P 1c50de89a5a845668696dcb28c3badd6 [term 1 FOLLOWER]: Leader election won for term 1
I20260129 02:58:47.786610 20633 raft_consensus.cc:385] T cedce331e1584ac8b636c9b8a72b2ba0 P 197b880fa39d432a9b472845876bfaf0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260129 02:58:47.787187 20633 raft_consensus.cc:740] T cedce331e1584ac8b636c9b8a72b2ba0 P 197b880fa39d432a9b472845876bfaf0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 197b880fa39d432a9b472845876bfaf0, State: Initialized, Role: FOLLOWER
I20260129 02:58:47.787467 20588 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7a9a69cd82724b1faed1d10aa091fc4a" candidate_uuid: "197b880fa39d432a9b472845876bfaf0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c50de89a5a845668696dcb28c3badd6"
I20260129 02:58:47.788089 20588 raft_consensus.cc:3060] T 7a9a69cd82724b1faed1d10aa091fc4a P 1c50de89a5a845668696dcb28c3badd6 [term 0 FOLLOWER]: Advancing to term 1
I20260129 02:58:47.787943 20633 consensus_queue.cc:260] T cedce331e1584ac8b636c9b8a72b2ba0 P 197b880fa39d432a9b472845876bfaf0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:58:47.788643 20516 raft_consensus.cc:2468] T e33a417cf9c54cf5af73a4ee3aac4d3a P 974b802141bd4f13beffef2aa03f88f9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1c50de89a5a845668696dcb28c3badd6 in term 1.
I20260129 02:58:47.791270 20633 ts_tablet_manager.cc:1434] T cedce331e1584ac8b636c9b8a72b2ba0 P 197b880fa39d432a9b472845876bfaf0: Time spent starting tablet: real 0.012s user 0.009s sys 0.000s
I20260129 02:58:47.791404 20516 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7a9a69cd82724b1faed1d10aa091fc4a" candidate_uuid: "197b880fa39d432a9b472845876bfaf0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "974b802141bd4f13beffef2aa03f88f9" is_pre_election: true
I20260129 02:58:47.786675 20832 raft_consensus.cc:697] T e33a417cf9c54cf5af73a4ee3aac4d3a P 1c50de89a5a845668696dcb28c3badd6 [term 1 LEADER]: Becoming Leader. State: Replica: 1c50de89a5a845668696dcb28c3badd6, State: Running, Role: LEADER
I20260129 02:58:47.792215 20633 tablet_bootstrap.cc:492] T cb0b3f259ebc4310927338a8740e87b8 P 197b880fa39d432a9b472845876bfaf0: Bootstrap starting.
I20260129 02:58:47.792470 20832 consensus_queue.cc:237] T e33a417cf9c54cf5af73a4ee3aac4d3a P 1c50de89a5a845668696dcb28c3badd6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:58:47.794328 20441 raft_consensus.cc:3060] T 36ff33be666a4c82954731691dbcf327 P 197b880fa39d432a9b472845876bfaf0 [term 0 FOLLOWER]: Advancing to term 1
I20260129 02:58:47.795137 20588 raft_consensus.cc:2468] T 7a9a69cd82724b1faed1d10aa091fc4a P 1c50de89a5a845668696dcb28c3badd6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 197b880fa39d432a9b472845876bfaf0 in term 1.
I20260129 02:58:47.785240 20630 ts_tablet_manager.cc:1434] T cedce331e1584ac8b636c9b8a72b2ba0 P 974b802141bd4f13beffef2aa03f88f9: Time spent starting tablet: real 0.035s user 0.007s sys 0.001s
I20260129 02:58:47.797585 20630 tablet_bootstrap.cc:492] T 18f30ebaa4174fd28c958e72d0cc7e24 P 974b802141bd4f13beffef2aa03f88f9: Bootstrap starting.
I20260129 02:58:47.799219 20633 tablet_bootstrap.cc:654] T cb0b3f259ebc4310927338a8740e87b8 P 197b880fa39d432a9b472845876bfaf0: Neither blocks nor log segments found. Creating new log.
I20260129 02:58:47.800788 20828 raft_consensus.cc:493] T 36ff33be666a4c82954731691dbcf327 P 197b880fa39d432a9b472845876bfaf0 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260129 02:58:47.801520 20516 raft_consensus.cc:2468] T 7a9a69cd82724b1faed1d10aa091fc4a P 974b802141bd4f13beffef2aa03f88f9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 197b880fa39d432a9b472845876bfaf0 in term 0.
I20260129 02:58:47.801273 20828 raft_consensus.cc:515] T 36ff33be666a4c82954731691dbcf327 P 197b880fa39d432a9b472845876bfaf0 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:58:47.803120 20869 consensus_queue.cc:1048] T 36ff33be666a4c82954731691dbcf327 P 974b802141bd4f13beffef2aa03f88f9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260129 02:58:47.804934 20630 tablet_bootstrap.cc:654] T 18f30ebaa4174fd28c958e72d0cc7e24 P 974b802141bd4f13beffef2aa03f88f9: Neither blocks nor log segments found. Creating new log.
I20260129 02:58:47.809620 20828 leader_election.cc:290] T 36ff33be666a4c82954731691dbcf327 P 197b880fa39d432a9b472845876bfaf0 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 974b802141bd4f13beffef2aa03f88f9 (127.18.211.2:34517), 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3:37735)
I20260129 02:58:47.810988 20633 tablet_bootstrap.cc:492] T cb0b3f259ebc4310927338a8740e87b8 P 197b880fa39d432a9b472845876bfaf0: No bootstrap required, opened a new log
I20260129 02:58:47.811512 20633 ts_tablet_manager.cc:1403] T cb0b3f259ebc4310927338a8740e87b8 P 197b880fa39d432a9b472845876bfaf0: Time spent bootstrapping tablet: real 0.019s user 0.016s sys 0.000s
I20260129 02:58:47.817186 20869 consensus_queue.cc:1048] T cb4727386a3e48aca5d71cc6c611c197 P 974b802141bd4f13beffef2aa03f88f9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260129 02:58:47.818993 20633 raft_consensus.cc:359] T cb0b3f259ebc4310927338a8740e87b8 P 197b880fa39d432a9b472845876bfaf0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:58:47.819845 20633 raft_consensus.cc:385] T cb0b3f259ebc4310927338a8740e87b8 P 197b880fa39d432a9b472845876bfaf0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260129 02:58:47.820206 20633 raft_consensus.cc:740] T cb0b3f259ebc4310927338a8740e87b8 P 197b880fa39d432a9b472845876bfaf0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 197b880fa39d432a9b472845876bfaf0, State: Initialized, Role: FOLLOWER
I20260129 02:58:47.821250 20633 consensus_queue.cc:260] T cb0b3f259ebc4310927338a8740e87b8 P 197b880fa39d432a9b472845876bfaf0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:58:47.823853 20514 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7a9a69cd82724b1faed1d10aa091fc4a" candidate_uuid: "197b880fa39d432a9b472845876bfaf0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "974b802141bd4f13beffef2aa03f88f9"
I20260129 02:58:47.824493 20514 raft_consensus.cc:3060] T 7a9a69cd82724b1faed1d10aa091fc4a P 974b802141bd4f13beffef2aa03f88f9 [term 0 FOLLOWER]: Advancing to term 1
I20260129 02:58:47.824846 20740 consensus_queue.cc:1048] T cb4727386a3e48aca5d71cc6c611c197 P 974b802141bd4f13beffef2aa03f88f9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260129 02:58:47.824059 20633 ts_tablet_manager.cc:1434] T cb0b3f259ebc4310927338a8740e87b8 P 197b880fa39d432a9b472845876bfaf0: Time spent starting tablet: real 0.012s user 0.010s sys 0.000s
I20260129 02:58:47.835237 20633 tablet_bootstrap.cc:492] T 18f30ebaa4174fd28c958e72d0cc7e24 P 197b880fa39d432a9b472845876bfaf0: Bootstrap starting.
I20260129 02:58:47.837607 20739 raft_consensus.cc:493] T cedce331e1584ac8b636c9b8a72b2ba0 P 974b802141bd4f13beffef2aa03f88f9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260129 02:58:47.838280 20739 raft_consensus.cc:515] T cedce331e1584ac8b636c9b8a72b2ba0 P 974b802141bd4f13beffef2aa03f88f9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:58:47.841384 20739 leader_election.cc:290] T cedce331e1584ac8b636c9b8a72b2ba0 P 974b802141bd4f13beffef2aa03f88f9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 197b880fa39d432a9b472845876bfaf0 (127.18.211.1:42027), 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3:37735)
I20260129 02:58:47.842329 20589 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "cedce331e1584ac8b636c9b8a72b2ba0" candidate_uuid: "974b802141bd4f13beffef2aa03f88f9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c50de89a5a845668696dcb28c3badd6" is_pre_election: true
I20260129 02:58:47.843176 20514 raft_consensus.cc:2468] T 7a9a69cd82724b1faed1d10aa091fc4a P 974b802141bd4f13beffef2aa03f88f9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 197b880fa39d432a9b472845876bfaf0 in term 1.
W20260129 02:58:47.844076 20480 leader_election.cc:343] T cedce331e1584ac8b636c9b8a72b2ba0 P 974b802141bd4f13beffef2aa03f88f9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3:37735): Illegal state: must be running to vote when last-logged opid is not known
I20260129 02:58:47.850775 20511 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "36ff33be666a4c82954731691dbcf327" candidate_uuid: "197b880fa39d432a9b472845876bfaf0" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "974b802141bd4f13beffef2aa03f88f9" is_pre_election: true
I20260129 02:58:47.851819 20633 tablet_bootstrap.cc:654] T 18f30ebaa4174fd28c958e72d0cc7e24 P 197b880fa39d432a9b472845876bfaf0: Neither blocks nor log segments found. Creating new log.
I20260129 02:58:47.855484 20740 raft_consensus.cc:493] T cb0b3f259ebc4310927338a8740e87b8 P 974b802141bd4f13beffef2aa03f88f9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260129 02:58:47.856007 20740 raft_consensus.cc:515] T cb0b3f259ebc4310927338a8740e87b8 P 974b802141bd4f13beffef2aa03f88f9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:58:47.856828 20587 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "36ff33be666a4c82954731691dbcf327" candidate_uuid: "197b880fa39d432a9b472845876bfaf0" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c50de89a5a845668696dcb28c3badd6" is_pre_election: true
I20260129 02:58:47.858059 20740 leader_election.cc:290] T cb0b3f259ebc4310927338a8740e87b8 P 974b802141bd4f13beffef2aa03f88f9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 197b880fa39d432a9b472845876bfaf0 (127.18.211.1:42027), 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3:37735)
I20260129 02:58:47.866457 20587 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "cb0b3f259ebc4310927338a8740e87b8" candidate_uuid: "974b802141bd4f13beffef2aa03f88f9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c50de89a5a845668696dcb28c3badd6" is_pre_election: true
W20260129 02:58:47.867754 20480 leader_election.cc:343] T cb0b3f259ebc4310927338a8740e87b8 P 974b802141bd4f13beffef2aa03f88f9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3:37735): Illegal state: must be running to vote when last-logged opid is not known
I20260129 02:58:47.872438 20405 leader_election.cc:304] T 7a9a69cd82724b1faed1d10aa091fc4a P 197b880fa39d432a9b472845876bfaf0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 197b880fa39d432a9b472845876bfaf0, 1c50de89a5a845668696dcb28c3badd6; no voters:
I20260129 02:58:47.874853 20828 raft_consensus.cc:2804] T 7a9a69cd82724b1faed1d10aa091fc4a P 197b880fa39d432a9b472845876bfaf0 [term 1 FOLLOWER]: Leader election won for term 1
I20260129 02:58:47.875349 20828 raft_consensus.cc:697] T 7a9a69cd82724b1faed1d10aa091fc4a P 197b880fa39d432a9b472845876bfaf0 [term 1 LEADER]: Becoming Leader. State: Replica: 197b880fa39d432a9b472845876bfaf0, State: Running, Role: LEADER
I20260129 02:58:47.876207 20828 consensus_queue.cc:237] T 7a9a69cd82724b1faed1d10aa091fc4a P 197b880fa39d432a9b472845876bfaf0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:58:47.881543 20405 leader_election.cc:304] T 36ff33be666a4c82954731691dbcf327 P 197b880fa39d432a9b472845876bfaf0 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 197b880fa39d432a9b472845876bfaf0; no voters: 1c50de89a5a845668696dcb28c3badd6, 974b802141bd4f13beffef2aa03f88f9
I20260129 02:58:47.883924 20827 raft_consensus.cc:2749] T 36ff33be666a4c82954731691dbcf327 P 197b880fa39d432a9b472845876bfaf0 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260129 02:58:47.885241 20337 catalog_manager.cc:5654] T 0340229d521b432580ae926f6062e3f7 P 197b880fa39d432a9b472845876bfaf0 reported cstate change: term changed from 0 to 1, leader changed from <none> to 197b880fa39d432a9b472845876bfaf0 (127.18.211.1). New cstate: current_term: 1 leader_uuid: "197b880fa39d432a9b472845876bfaf0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } health_report { overall_health: UNKNOWN } } }
I20260129 02:58:47.892426 20439 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "cedce331e1584ac8b636c9b8a72b2ba0" candidate_uuid: "974b802141bd4f13beffef2aa03f88f9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "197b880fa39d432a9b472845876bfaf0" is_pre_election: true
I20260129 02:58:47.893157 20439 raft_consensus.cc:2468] T cedce331e1584ac8b636c9b8a72b2ba0 P 197b880fa39d432a9b472845876bfaf0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 974b802141bd4f13beffef2aa03f88f9 in term 0.
I20260129 02:58:47.894546 20439 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "cb0b3f259ebc4310927338a8740e87b8" candidate_uuid: "974b802141bd4f13beffef2aa03f88f9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "197b880fa39d432a9b472845876bfaf0" is_pre_election: true
I20260129 02:58:47.895129 20439 raft_consensus.cc:2468] T cb0b3f259ebc4310927338a8740e87b8 P 197b880fa39d432a9b472845876bfaf0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 974b802141bd4f13beffef2aa03f88f9 in term 0.
I20260129 02:58:47.901587 20479 leader_election.cc:304] T cedce331e1584ac8b636c9b8a72b2ba0 P 974b802141bd4f13beffef2aa03f88f9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 197b880fa39d432a9b472845876bfaf0, 974b802141bd4f13beffef2aa03f88f9; no voters: 1c50de89a5a845668696dcb28c3badd6
I20260129 02:58:47.902735 20740 raft_consensus.cc:2804] T cedce331e1584ac8b636c9b8a72b2ba0 P 974b802141bd4f13beffef2aa03f88f9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260129 02:58:47.903120 20740 raft_consensus.cc:493] T cedce331e1584ac8b636c9b8a72b2ba0 P 974b802141bd4f13beffef2aa03f88f9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260129 02:58:47.903515 20740 raft_consensus.cc:3060] T cedce331e1584ac8b636c9b8a72b2ba0 P 974b802141bd4f13beffef2aa03f88f9 [term 0 FOLLOWER]: Advancing to term 1
I20260129 02:58:47.909427 20479 leader_election.cc:304] T cb0b3f259ebc4310927338a8740e87b8 P 974b802141bd4f13beffef2aa03f88f9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 197b880fa39d432a9b472845876bfaf0, 974b802141bd4f13beffef2aa03f88f9; no voters: 1c50de89a5a845668696dcb28c3badd6
I20260129 02:58:47.910374 20739 raft_consensus.cc:2804] T cb0b3f259ebc4310927338a8740e87b8 P 974b802141bd4f13beffef2aa03f88f9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260129 02:58:47.910744 20739 raft_consensus.cc:493] T cb0b3f259ebc4310927338a8740e87b8 P 974b802141bd4f13beffef2aa03f88f9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260129 02:58:47.911191 20739 raft_consensus.cc:3060] T cb0b3f259ebc4310927338a8740e87b8 P 974b802141bd4f13beffef2aa03f88f9 [term 0 FOLLOWER]: Advancing to term 1
I20260129 02:58:47.917282 20721 raft_consensus.cc:493] T a0fcc58d6fb6473cafbd9d34d5195269 P 197b880fa39d432a9b472845876bfaf0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260129 02:58:47.917779 20721 raft_consensus.cc:515] T a0fcc58d6fb6473cafbd9d34d5195269 P 197b880fa39d432a9b472845876bfaf0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:58:47.919704 20721 leader_election.cc:290] T a0fcc58d6fb6473cafbd9d34d5195269 P 197b880fa39d432a9b472845876bfaf0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 974b802141bd4f13beffef2aa03f88f9 (127.18.211.2:34517), 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3:37735)
I20260129 02:58:47.933744 20739 raft_consensus.cc:515] T cb0b3f259ebc4310927338a8740e87b8 P 974b802141bd4f13beffef2aa03f88f9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:58:47.935189 20840 raft_consensus.cc:493] T a644c02b16af47679824ccd060ac10fd P 1c50de89a5a845668696dcb28c3badd6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260129 02:58:47.935688 20840 raft_consensus.cc:515] T a644c02b16af47679824ccd060ac10fd P 1c50de89a5a845668696dcb28c3badd6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:58:47.937557 20840 leader_election.cc:290] T a644c02b16af47679824ccd060ac10fd P 1c50de89a5a845668696dcb28c3badd6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 197b880fa39d432a9b472845876bfaf0 (127.18.211.1:42027), 974b802141bd4f13beffef2aa03f88f9 (127.18.211.2:34517)
W20260129 02:58:47.936012 20407 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.211.2:34517, user_credentials={real_user=slave}} blocked reactor thread for 62938.8us
I20260129 02:58:47.947306 20516 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a644c02b16af47679824ccd060ac10fd" candidate_uuid: "1c50de89a5a845668696dcb28c3badd6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "974b802141bd4f13beffef2aa03f88f9" is_pre_election: true
I20260129 02:58:47.954272 20516 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a0fcc58d6fb6473cafbd9d34d5195269" candidate_uuid: "197b880fa39d432a9b472845876bfaf0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "974b802141bd4f13beffef2aa03f88f9" is_pre_election: true
I20260129 02:58:47.956182 20869 raft_consensus.cc:493] T 926a250fa5834e88b96ccecbefbf5086 P 974b802141bd4f13beffef2aa03f88f9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260129 02:58:47.956641 20869 raft_consensus.cc:515] T 926a250fa5834e88b96ccecbefbf5086 P 974b802141bd4f13beffef2aa03f88f9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:58:47.959311 20869 leader_election.cc:290] T 926a250fa5834e88b96ccecbefbf5086 P 974b802141bd4f13beffef2aa03f88f9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 197b880fa39d432a9b472845876bfaf0 (127.18.211.1:42027), 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3:37735)
I20260129 02:58:47.959494 20587 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "926a250fa5834e88b96ccecbefbf5086" candidate_uuid: "974b802141bd4f13beffef2aa03f88f9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c50de89a5a845668696dcb28c3badd6" is_pre_election: true
W20260129 02:58:47.960711 20480 leader_election.cc:343] T 926a250fa5834e88b96ccecbefbf5086 P 974b802141bd4f13beffef2aa03f88f9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3:37735): Illegal state: must be running to vote when last-logged opid is not known
W20260129 02:58:47.970914 20407 leader_election.cc:343] T a0fcc58d6fb6473cafbd9d34d5195269 P 197b880fa39d432a9b472845876bfaf0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 974b802141bd4f13beffef2aa03f88f9 (127.18.211.2:34517): Illegal state: must be running to vote when last-logged opid is not known
I20260129 02:58:47.976562 20740 raft_consensus.cc:515] T cedce331e1584ac8b636c9b8a72b2ba0 P 974b802141bd4f13beffef2aa03f88f9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:58:47.981822 20336 catalog_manager.cc:5654] T 7a9a69cd82724b1faed1d10aa091fc4a P 197b880fa39d432a9b472845876bfaf0 reported cstate change: term changed from 0 to 1, leader changed from <none> to 197b880fa39d432a9b472845876bfaf0 (127.18.211.1). New cstate: current_term: 1 leader_uuid: "197b880fa39d432a9b472845876bfaf0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } health_report { overall_health: UNKNOWN } } }
I20260129 02:58:47.983265 20439 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a644c02b16af47679824ccd060ac10fd" candidate_uuid: "1c50de89a5a845668696dcb28c3badd6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "197b880fa39d432a9b472845876bfaf0" is_pre_election: true
I20260129 02:58:47.983947 20439 raft_consensus.cc:2468] T a644c02b16af47679824ccd060ac10fd P 197b880fa39d432a9b472845876bfaf0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1c50de89a5a845668696dcb28c3badd6 in term 0.
I20260129 02:58:47.991406 20739 leader_election.cc:290] T cb0b3f259ebc4310927338a8740e87b8 P 974b802141bd4f13beffef2aa03f88f9 [CANDIDATE]: Term 1 election: Requested vote from peers 197b880fa39d432a9b472845876bfaf0 (127.18.211.1:42027), 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3:37735)
I20260129 02:58:47.992164 20587 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "cb0b3f259ebc4310927338a8740e87b8" candidate_uuid: "974b802141bd4f13beffef2aa03f88f9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c50de89a5a845668696dcb28c3badd6"
W20260129 02:58:48.019063 20480 leader_election.cc:343] T cb0b3f259ebc4310927338a8740e87b8 P 974b802141bd4f13beffef2aa03f88f9 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3:37735): Illegal state: must be running to vote when last-logged opid is not known
W20260129 02:58:48.030376 20556 leader_election.cc:343] T a644c02b16af47679824ccd060ac10fd P 1c50de89a5a845668696dcb28c3badd6 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 974b802141bd4f13beffef2aa03f88f9 (127.18.211.2:34517): Illegal state: must be running to vote when last-logged opid is not known
I20260129 02:58:48.031342 20587 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a0fcc58d6fb6473cafbd9d34d5195269" candidate_uuid: "197b880fa39d432a9b472845876bfaf0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c50de89a5a845668696dcb28c3badd6" is_pre_election: true
I20260129 02:58:48.037549 20553 leader_election.cc:304] T a644c02b16af47679824ccd060ac10fd P 1c50de89a5a845668696dcb28c3badd6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 197b880fa39d432a9b472845876bfaf0, 1c50de89a5a845668696dcb28c3badd6; no voters: 974b802141bd4f13beffef2aa03f88f9
I20260129 02:58:48.037891 20336 catalog_manager.cc:5654] T e33a417cf9c54cf5af73a4ee3aac4d3a P 1c50de89a5a845668696dcb28c3badd6 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3). New cstate: current_term: 1 leader_uuid: "1c50de89a5a845668696dcb28c3badd6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } health_report { overall_health: HEALTHY } } }
I20260129 02:58:48.038709 20840 raft_consensus.cc:2804] T a644c02b16af47679824ccd060ac10fd P 1c50de89a5a845668696dcb28c3badd6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260129 02:58:48.039076 20840 raft_consensus.cc:493] T a644c02b16af47679824ccd060ac10fd P 1c50de89a5a845668696dcb28c3badd6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260129 02:58:48.039397 20840 raft_consensus.cc:3060] T a644c02b16af47679824ccd060ac10fd P 1c50de89a5a845668696dcb28c3badd6 [term 0 FOLLOWER]: Advancing to term 1
I20260129 02:58:48.047127 20840 raft_consensus.cc:515] T a644c02b16af47679824ccd060ac10fd P 1c50de89a5a845668696dcb28c3badd6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:58:48.055770 20840 leader_election.cc:290] T a644c02b16af47679824ccd060ac10fd P 1c50de89a5a845668696dcb28c3badd6 [CANDIDATE]: Term 1 election: Requested vote from peers 197b880fa39d432a9b472845876bfaf0 (127.18.211.1:42027), 974b802141bd4f13beffef2aa03f88f9 (127.18.211.2:34517)
I20260129 02:58:48.077179 20439 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "cb0b3f259ebc4310927338a8740e87b8" candidate_uuid: "974b802141bd4f13beffef2aa03f88f9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "197b880fa39d432a9b472845876bfaf0"
I20260129 02:58:48.080204 20439 raft_consensus.cc:3060] T cb0b3f259ebc4310927338a8740e87b8 P 197b880fa39d432a9b472845876bfaf0 [term 0 FOLLOWER]: Advancing to term 1
I20260129 02:58:48.094165 20631 tablet_bootstrap.cc:492] T 18f30ebaa4174fd28c958e72d0cc7e24 P 1c50de89a5a845668696dcb28c3badd6: No bootstrap required, opened a new log
I20260129 02:58:48.094863 20631 ts_tablet_manager.cc:1403] T 18f30ebaa4174fd28c958e72d0cc7e24 P 1c50de89a5a845668696dcb28c3badd6: Time spent bootstrapping tablet: real 0.418s user 0.108s sys 0.150s
I20260129 02:58:48.096855 20441 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "926a250fa5834e88b96ccecbefbf5086" candidate_uuid: "974b802141bd4f13beffef2aa03f88f9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "197b880fa39d432a9b472845876bfaf0" is_pre_election: true
I20260129 02:58:48.098340 20631 raft_consensus.cc:359] T 18f30ebaa4174fd28c958e72d0cc7e24 P 1c50de89a5a845668696dcb28c3badd6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:58:48.098889 20441 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "cedce331e1584ac8b636c9b8a72b2ba0" candidate_uuid: "974b802141bd4f13beffef2aa03f88f9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "197b880fa39d432a9b472845876bfaf0"
I20260129 02:58:48.099270 20631 raft_consensus.cc:385] T 18f30ebaa4174fd28c958e72d0cc7e24 P 1c50de89a5a845668696dcb28c3badd6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260129 02:58:48.099648 20441 raft_consensus.cc:3060] T cedce331e1584ac8b636c9b8a72b2ba0 P 197b880fa39d432a9b472845876bfaf0 [term 0 FOLLOWER]: Advancing to term 1
I20260129 02:58:48.099673 20631 raft_consensus.cc:740] T 18f30ebaa4174fd28c958e72d0cc7e24 P 1c50de89a5a845668696dcb28c3badd6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1c50de89a5a845668696dcb28c3badd6, State: Initialized, Role: FOLLOWER
I20260129 02:58:48.102432 20630 tablet_bootstrap.cc:492] T 18f30ebaa4174fd28c958e72d0cc7e24 P 974b802141bd4f13beffef2aa03f88f9: No bootstrap required, opened a new log
I20260129 02:58:48.102891 20630 ts_tablet_manager.cc:1403] T 18f30ebaa4174fd28c958e72d0cc7e24 P 974b802141bd4f13beffef2aa03f88f9: Time spent bootstrapping tablet: real 0.306s user 0.066s sys 0.060s
I20260129 02:58:48.103657 20633 tablet_bootstrap.cc:492] T 18f30ebaa4174fd28c958e72d0cc7e24 P 197b880fa39d432a9b472845876bfaf0: No bootstrap required, opened a new log
W20260129 02:58:48.103574 20405 leader_election.cc:343] T a0fcc58d6fb6473cafbd9d34d5195269 P 197b880fa39d432a9b472845876bfaf0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3:37735): Illegal state: must be running to vote when last-logged opid is not known
I20260129 02:58:48.104242 20633 ts_tablet_manager.cc:1403] T 18f30ebaa4174fd28c958e72d0cc7e24 P 197b880fa39d432a9b472845876bfaf0: Time spent bootstrapping tablet: real 0.269s user 0.089s sys 0.163s
I20260129 02:58:48.104218 20405 leader_election.cc:304] T a0fcc58d6fb6473cafbd9d34d5195269 P 197b880fa39d432a9b472845876bfaf0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 197b880fa39d432a9b472845876bfaf0; no voters: 1c50de89a5a845668696dcb28c3badd6, 974b802141bd4f13beffef2aa03f88f9
I20260129 02:58:48.105038 20721 raft_consensus.cc:2749] T a0fcc58d6fb6473cafbd9d34d5195269 P 197b880fa39d432a9b472845876bfaf0 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260129 02:58:48.106066 20441 raft_consensus.cc:2468] T cedce331e1584ac8b636c9b8a72b2ba0 P 197b880fa39d432a9b472845876bfaf0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 974b802141bd4f13beffef2aa03f88f9 in term 1.
I20260129 02:58:48.106838 20516 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a644c02b16af47679824ccd060ac10fd" candidate_uuid: "1c50de89a5a845668696dcb28c3badd6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "974b802141bd4f13beffef2aa03f88f9"
I20260129 02:58:48.106716 20633 raft_consensus.cc:359] T 18f30ebaa4174fd28c958e72d0cc7e24 P 197b880fa39d432a9b472845876bfaf0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:58:48.102806 20631 consensus_queue.cc:260] T 18f30ebaa4174fd28c958e72d0cc7e24 P 1c50de89a5a845668696dcb28c3badd6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:58:48.108896 20630 raft_consensus.cc:359] T 18f30ebaa4174fd28c958e72d0cc7e24 P 974b802141bd4f13beffef2aa03f88f9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:58:48.109262 20587 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "cedce331e1584ac8b636c9b8a72b2ba0" candidate_uuid: "974b802141bd4f13beffef2aa03f88f9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c50de89a5a845668696dcb28c3badd6"
I20260129 02:58:48.109547 20630 raft_consensus.cc:385] T 18f30ebaa4174fd28c958e72d0cc7e24 P 974b802141bd4f13beffef2aa03f88f9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260129 02:58:48.109936 20630 raft_consensus.cc:740] T 18f30ebaa4174fd28c958e72d0cc7e24 P 974b802141bd4f13beffef2aa03f88f9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 974b802141bd4f13beffef2aa03f88f9, State: Initialized, Role: FOLLOWER
I20260129 02:58:48.110217 20631 ts_tablet_manager.cc:1434] T 18f30ebaa4174fd28c958e72d0cc7e24 P 1c50de89a5a845668696dcb28c3badd6: Time spent starting tablet: real 0.015s user 0.006s sys 0.001s
W20260129 02:58:48.110682 20480 leader_election.cc:343] T cedce331e1584ac8b636c9b8a72b2ba0 P 974b802141bd4f13beffef2aa03f88f9 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3:37735): Illegal state: must be running to vote when last-logged opid is not known
I20260129 02:58:48.107488 20633 raft_consensus.cc:385] T 18f30ebaa4174fd28c958e72d0cc7e24 P 197b880fa39d432a9b472845876bfaf0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260129 02:58:48.110777 20630 consensus_queue.cc:260] T 18f30ebaa4174fd28c958e72d0cc7e24 P 974b802141bd4f13beffef2aa03f88f9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:58:48.108194 20740 leader_election.cc:290] T cedce331e1584ac8b636c9b8a72b2ba0 P 974b802141bd4f13beffef2aa03f88f9 [CANDIDATE]: Term 1 election: Requested vote from peers 197b880fa39d432a9b472845876bfaf0 (127.18.211.1:42027), 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3:37735)
I20260129 02:58:48.111382 20631 tablet_bootstrap.cc:492] T c2633a6fc34c44579de348d208b0c01e P 1c50de89a5a845668696dcb28c3badd6: Bootstrap starting.
I20260129 02:58:48.111400 20633 raft_consensus.cc:740] T 18f30ebaa4174fd28c958e72d0cc7e24 P 197b880fa39d432a9b472845876bfaf0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 197b880fa39d432a9b472845876bfaf0, State: Initialized, Role: FOLLOWER
I20260129 02:58:48.112505 20630 ts_tablet_manager.cc:1434] T 18f30ebaa4174fd28c958e72d0cc7e24 P 974b802141bd4f13beffef2aa03f88f9: Time spent starting tablet: real 0.009s user 0.005s sys 0.002s
I20260129 02:58:48.112722 20633 consensus_queue.cc:260] T 18f30ebaa4174fd28c958e72d0cc7e24 P 197b880fa39d432a9b472845876bfaf0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:58:48.113444 20630 tablet_bootstrap.cc:492] T 0678b8b468bd43cdbc3e485900844821 P 974b802141bd4f13beffef2aa03f88f9: Bootstrap starting.
I20260129 02:58:48.115787 20633 ts_tablet_manager.cc:1434] T 18f30ebaa4174fd28c958e72d0cc7e24 P 197b880fa39d432a9b472845876bfaf0: Time spent starting tablet: real 0.011s user 0.006s sys 0.000s
I20260129 02:58:48.119553 20631 tablet_bootstrap.cc:654] T c2633a6fc34c44579de348d208b0c01e P 1c50de89a5a845668696dcb28c3badd6: Neither blocks nor log segments found. Creating new log.
I20260129 02:58:48.121852 20633 tablet_bootstrap.cc:492] T 42d27a28d89b409e8807ec0efd97af69 P 197b880fa39d432a9b472845876bfaf0: Bootstrap starting.
I20260129 02:58:48.125447 20630 tablet_bootstrap.cc:654] T 0678b8b468bd43cdbc3e485900844821 P 974b802141bd4f13beffef2aa03f88f9: Neither blocks nor log segments found. Creating new log.
I20260129 02:58:48.128499 20633 tablet_bootstrap.cc:654] T 42d27a28d89b409e8807ec0efd97af69 P 197b880fa39d432a9b472845876bfaf0: Neither blocks nor log segments found. Creating new log.
I20260129 02:58:48.139257 20633 tablet_bootstrap.cc:492] T 42d27a28d89b409e8807ec0efd97af69 P 197b880fa39d432a9b472845876bfaf0: No bootstrap required, opened a new log
I20260129 02:58:48.139137 20630 tablet_bootstrap.cc:492] T 0678b8b468bd43cdbc3e485900844821 P 974b802141bd4f13beffef2aa03f88f9: No bootstrap required, opened a new log
I20260129 02:58:48.139842 20633 ts_tablet_manager.cc:1403] T 42d27a28d89b409e8807ec0efd97af69 P 197b880fa39d432a9b472845876bfaf0: Time spent bootstrapping tablet: real 0.018s user 0.013s sys 0.002s
I20260129 02:58:48.139984 20630 ts_tablet_manager.cc:1403] T 0678b8b468bd43cdbc3e485900844821 P 974b802141bd4f13beffef2aa03f88f9: Time spent bootstrapping tablet: real 0.027s user 0.017s sys 0.002s
I20260129 02:58:48.142365 20633 raft_consensus.cc:359] T 42d27a28d89b409e8807ec0efd97af69 P 197b880fa39d432a9b472845876bfaf0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:58:48.143092 20633 raft_consensus.cc:385] T 42d27a28d89b409e8807ec0efd97af69 P 197b880fa39d432a9b472845876bfaf0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260129 02:58:48.143386 20633 raft_consensus.cc:740] T 42d27a28d89b409e8807ec0efd97af69 P 197b880fa39d432a9b472845876bfaf0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 197b880fa39d432a9b472845876bfaf0, State: Initialized, Role: FOLLOWER
I20260129 02:58:48.143399 20630 raft_consensus.cc:359] T 0678b8b468bd43cdbc3e485900844821 P 974b802141bd4f13beffef2aa03f88f9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:58:48.144418 20630 raft_consensus.cc:385] T 0678b8b468bd43cdbc3e485900844821 P 974b802141bd4f13beffef2aa03f88f9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260129 02:58:48.144134 20633 consensus_queue.cc:260] T 42d27a28d89b409e8807ec0efd97af69 P 197b880fa39d432a9b472845876bfaf0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:58:48.144766 20630 raft_consensus.cc:740] T 0678b8b468bd43cdbc3e485900844821 P 974b802141bd4f13beffef2aa03f88f9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 974b802141bd4f13beffef2aa03f88f9, State: Initialized, Role: FOLLOWER
I20260129 02:58:48.145723 20630 consensus_queue.cc:260] T 0678b8b468bd43cdbc3e485900844821 P 974b802141bd4f13beffef2aa03f88f9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:58:48.146333 20633 ts_tablet_manager.cc:1434] T 42d27a28d89b409e8807ec0efd97af69 P 197b880fa39d432a9b472845876bfaf0: Time spent starting tablet: real 0.006s user 0.002s sys 0.002s
I20260129 02:58:48.147573 20633 tablet_bootstrap.cc:492] T 926a250fa5834e88b96ccecbefbf5086 P 197b880fa39d432a9b472845876bfaf0: Bootstrap starting.
I20260129 02:58:48.148279 20630 ts_tablet_manager.cc:1434] T 0678b8b468bd43cdbc3e485900844821 P 974b802141bd4f13beffef2aa03f88f9: Time spent starting tablet: real 0.008s user 0.007s sys 0.001s
I20260129 02:58:48.150420 20630 tablet_bootstrap.cc:492] T 42d27a28d89b409e8807ec0efd97af69 P 974b802141bd4f13beffef2aa03f88f9: Bootstrap starting.
I20260129 02:58:48.153294 20633 tablet_bootstrap.cc:654] T 926a250fa5834e88b96ccecbefbf5086 P 197b880fa39d432a9b472845876bfaf0: Neither blocks nor log segments found. Creating new log.
W20260129 02:58:48.159900 20556 leader_election.cc:343] T a644c02b16af47679824ccd060ac10fd P 1c50de89a5a845668696dcb28c3badd6 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 974b802141bd4f13beffef2aa03f88f9 (127.18.211.2:34517): Illegal state: must be running to vote when last-logged opid is not known
I20260129 02:58:48.164222 20435 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a644c02b16af47679824ccd060ac10fd" candidate_uuid: "1c50de89a5a845668696dcb28c3badd6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "197b880fa39d432a9b472845876bfaf0"
I20260129 02:58:48.164395 20633 tablet_bootstrap.cc:492] T 926a250fa5834e88b96ccecbefbf5086 P 197b880fa39d432a9b472845876bfaf0: No bootstrap required, opened a new log
I20260129 02:58:48.170531 20633 ts_tablet_manager.cc:1403] T 926a250fa5834e88b96ccecbefbf5086 P 197b880fa39d432a9b472845876bfaf0: Time spent bootstrapping tablet: real 0.023s user 0.005s sys 0.011s
I20260129 02:58:48.173169 20633 raft_consensus.cc:359] T 926a250fa5834e88b96ccecbefbf5086 P 197b880fa39d432a9b472845876bfaf0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:58:48.173846 20633 raft_consensus.cc:385] T 926a250fa5834e88b96ccecbefbf5086 P 197b880fa39d432a9b472845876bfaf0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260129 02:58:48.174122 20633 raft_consensus.cc:740] T 926a250fa5834e88b96ccecbefbf5086 P 197b880fa39d432a9b472845876bfaf0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 197b880fa39d432a9b472845876bfaf0, State: Initialized, Role: FOLLOWER
I20260129 02:58:48.174787 20633 consensus_queue.cc:260] T 926a250fa5834e88b96ccecbefbf5086 P 197b880fa39d432a9b472845876bfaf0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:58:48.176709 20633 ts_tablet_manager.cc:1434] T 926a250fa5834e88b96ccecbefbf5086 P 197b880fa39d432a9b472845876bfaf0: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20260129 02:58:48.177361 20435 raft_consensus.cc:3060] T a644c02b16af47679824ccd060ac10fd P 197b880fa39d432a9b472845876bfaf0 [term 0 FOLLOWER]: Advancing to term 1
I20260129 02:58:48.178606 20630 tablet_bootstrap.cc:654] T 42d27a28d89b409e8807ec0efd97af69 P 974b802141bd4f13beffef2aa03f88f9: Neither blocks nor log segments found. Creating new log.
I20260129 02:58:48.187458 20435 raft_consensus.cc:2468] T a644c02b16af47679824ccd060ac10fd P 197b880fa39d432a9b472845876bfaf0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1c50de89a5a845668696dcb28c3badd6 in term 1.
I20260129 02:58:48.189560 20633 tablet_bootstrap.cc:492] T c891d2bff4ce4568861bbc992e91fbb0 P 197b880fa39d432a9b472845876bfaf0: Bootstrap starting.
I20260129 02:58:48.196574 20633 tablet_bootstrap.cc:654] T c891d2bff4ce4568861bbc992e91fbb0 P 197b880fa39d432a9b472845876bfaf0: Neither blocks nor log segments found. Creating new log.
I20260129 02:58:48.198856 20439 raft_consensus.cc:2468] T cb0b3f259ebc4310927338a8740e87b8 P 197b880fa39d432a9b472845876bfaf0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 974b802141bd4f13beffef2aa03f88f9 in term 1.
W20260129 02:58:48.207190 20479 leader_election.cc:343] T 926a250fa5834e88b96ccecbefbf5086 P 974b802141bd4f13beffef2aa03f88f9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 197b880fa39d432a9b472845876bfaf0 (127.18.211.1:42027): Illegal state: must be running to vote when last-logged opid is not known
I20260129 02:58:48.207836 20479 leader_election.cc:304] T 926a250fa5834e88b96ccecbefbf5086 P 974b802141bd4f13beffef2aa03f88f9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 974b802141bd4f13beffef2aa03f88f9; no voters: 197b880fa39d432a9b472845876bfaf0, 1c50de89a5a845668696dcb28c3badd6
I20260129 02:58:48.209148 20739 raft_consensus.cc:2749] T 926a250fa5834e88b96ccecbefbf5086 P 974b802141bd4f13beffef2aa03f88f9 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260129 02:58:48.211679 20633 tablet_bootstrap.cc:492] T c891d2bff4ce4568861bbc992e91fbb0 P 197b880fa39d432a9b472845876bfaf0: No bootstrap required, opened a new log
I20260129 02:58:48.212239 20633 ts_tablet_manager.cc:1403] T c891d2bff4ce4568861bbc992e91fbb0 P 197b880fa39d432a9b472845876bfaf0: Time spent bootstrapping tablet: real 0.023s user 0.013s sys 0.005s
I20260129 02:58:48.214785 20633 raft_consensus.cc:359] T c891d2bff4ce4568861bbc992e91fbb0 P 197b880fa39d432a9b472845876bfaf0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:58:48.215495 20633 raft_consensus.cc:385] T c891d2bff4ce4568861bbc992e91fbb0 P 197b880fa39d432a9b472845876bfaf0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260129 02:58:48.215816 20633 raft_consensus.cc:740] T c891d2bff4ce4568861bbc992e91fbb0 P 197b880fa39d432a9b472845876bfaf0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 197b880fa39d432a9b472845876bfaf0, State: Initialized, Role: FOLLOWER
I20260129 02:58:48.216571 20633 consensus_queue.cc:260] T c891d2bff4ce4568861bbc992e91fbb0 P 197b880fa39d432a9b472845876bfaf0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:58:48.217571 20479 leader_election.cc:304] T cedce331e1584ac8b636c9b8a72b2ba0 P 974b802141bd4f13beffef2aa03f88f9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 197b880fa39d432a9b472845876bfaf0, 974b802141bd4f13beffef2aa03f88f9; no voters: 1c50de89a5a845668696dcb28c3badd6
I20260129 02:58:48.218366 20739 raft_consensus.cc:2804] T cedce331e1584ac8b636c9b8a72b2ba0 P 974b802141bd4f13beffef2aa03f88f9 [term 1 FOLLOWER]: Leader election won for term 1
I20260129 02:58:48.218816 20739 raft_consensus.cc:697] T cedce331e1584ac8b636c9b8a72b2ba0 P 974b802141bd4f13beffef2aa03f88f9 [term 1 LEADER]: Becoming Leader. State: Replica: 974b802141bd4f13beffef2aa03f88f9, State: Running, Role: LEADER
I20260129 02:58:48.219623 20739 consensus_queue.cc:237] T cedce331e1584ac8b636c9b8a72b2ba0 P 974b802141bd4f13beffef2aa03f88f9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:58:48.224967 20633 ts_tablet_manager.cc:1434] T c891d2bff4ce4568861bbc992e91fbb0 P 197b880fa39d432a9b472845876bfaf0: Time spent starting tablet: real 0.012s user 0.005s sys 0.000s
I20260129 02:58:48.226733 20633 tablet_bootstrap.cc:492] T 0678b8b468bd43cdbc3e485900844821 P 197b880fa39d432a9b472845876bfaf0: Bootstrap starting.
I20260129 02:58:48.229405 20553 leader_election.cc:304] T a644c02b16af47679824ccd060ac10fd P 1c50de89a5a845668696dcb28c3badd6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 197b880fa39d432a9b472845876bfaf0, 1c50de89a5a845668696dcb28c3badd6; no voters: 974b802141bd4f13beffef2aa03f88f9
I20260129 02:58:48.231439 20630 tablet_bootstrap.cc:492] T 42d27a28d89b409e8807ec0efd97af69 P 974b802141bd4f13beffef2aa03f88f9: No bootstrap required, opened a new log
I20260129 02:58:48.232173 20479 leader_election.cc:304] T cb0b3f259ebc4310927338a8740e87b8 P 974b802141bd4f13beffef2aa03f88f9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 197b880fa39d432a9b472845876bfaf0, 974b802141bd4f13beffef2aa03f88f9; no voters: 1c50de89a5a845668696dcb28c3badd6
I20260129 02:58:48.234344 20630 ts_tablet_manager.cc:1403] T 42d27a28d89b409e8807ec0efd97af69 P 974b802141bd4f13beffef2aa03f88f9: Time spent bootstrapping tablet: real 0.084s user 0.010s sys 0.004s
I20260129 02:58:48.234586 20869 raft_consensus.cc:2804] T cb0b3f259ebc4310927338a8740e87b8 P 974b802141bd4f13beffef2aa03f88f9 [term 1 FOLLOWER]: Leader election won for term 1
I20260129 02:58:48.235100 20869 raft_consensus.cc:697] T cb0b3f259ebc4310927338a8740e87b8 P 974b802141bd4f13beffef2aa03f88f9 [term 1 LEADER]: Becoming Leader. State: Replica: 974b802141bd4f13beffef2aa03f88f9, State: Running, Role: LEADER
I20260129 02:58:48.236106 20869 consensus_queue.cc:237] T cb0b3f259ebc4310927338a8740e87b8 P 974b802141bd4f13beffef2aa03f88f9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:58:48.237015 20630 raft_consensus.cc:359] T 42d27a28d89b409e8807ec0efd97af69 P 974b802141bd4f13beffef2aa03f88f9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:58:48.237778 20630 raft_consensus.cc:385] T 42d27a28d89b409e8807ec0efd97af69 P 974b802141bd4f13beffef2aa03f88f9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260129 02:58:48.238046 20630 raft_consensus.cc:740] T 42d27a28d89b409e8807ec0efd97af69 P 974b802141bd4f13beffef2aa03f88f9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 974b802141bd4f13beffef2aa03f88f9, State: Initialized, Role: FOLLOWER
I20260129 02:58:48.238265 20721 consensus_queue.cc:1048] T 0340229d521b432580ae926f6062e3f7 P 197b880fa39d432a9b472845876bfaf0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260129 02:58:48.238741 20630 consensus_queue.cc:260] T 42d27a28d89b409e8807ec0efd97af69 P 974b802141bd4f13beffef2aa03f88f9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:58:48.241258 20630 ts_tablet_manager.cc:1434] T 42d27a28d89b409e8807ec0efd97af69 P 974b802141bd4f13beffef2aa03f88f9: Time spent starting tablet: real 0.007s user 0.005s sys 0.000s
I20260129 02:58:48.242158 20630 tablet_bootstrap.cc:492] T a0fcc58d6fb6473cafbd9d34d5195269 P 974b802141bd4f13beffef2aa03f88f9: Bootstrap starting.
I20260129 02:58:48.250743 20633 tablet_bootstrap.cc:654] T 0678b8b468bd43cdbc3e485900844821 P 197b880fa39d432a9b472845876bfaf0: Neither blocks nor log segments found. Creating new log.
I20260129 02:58:48.253332 20630 tablet_bootstrap.cc:654] T a0fcc58d6fb6473cafbd9d34d5195269 P 974b802141bd4f13beffef2aa03f88f9: Neither blocks nor log segments found. Creating new log.
I20260129 02:58:48.262559 20832 raft_consensus.cc:2804] T a644c02b16af47679824ccd060ac10fd P 1c50de89a5a845668696dcb28c3badd6 [term 1 FOLLOWER]: Leader election won for term 1
I20260129 02:58:48.263096 20832 raft_consensus.cc:697] T a644c02b16af47679824ccd060ac10fd P 1c50de89a5a845668696dcb28c3badd6 [term 1 LEADER]: Becoming Leader. State: Replica: 1c50de89a5a845668696dcb28c3badd6, State: Running, Role: LEADER
I20260129 02:58:48.264286 20832 consensus_queue.cc:237] T a644c02b16af47679824ccd060ac10fd P 1c50de89a5a845668696dcb28c3badd6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:58:48.271479 20631 tablet_bootstrap.cc:492] T c2633a6fc34c44579de348d208b0c01e P 1c50de89a5a845668696dcb28c3badd6: No bootstrap required, opened a new log
I20260129 02:58:48.272023 20631 ts_tablet_manager.cc:1403] T c2633a6fc34c44579de348d208b0c01e P 1c50de89a5a845668696dcb28c3badd6: Time spent bootstrapping tablet: real 0.161s user 0.068s sys 0.047s
I20260129 02:58:48.275437 20631 raft_consensus.cc:359] T c2633a6fc34c44579de348d208b0c01e P 1c50de89a5a845668696dcb28c3badd6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:58:48.276149 20631 raft_consensus.cc:385] T c2633a6fc34c44579de348d208b0c01e P 1c50de89a5a845668696dcb28c3badd6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260129 02:58:48.276427 20631 raft_consensus.cc:740] T c2633a6fc34c44579de348d208b0c01e P 1c50de89a5a845668696dcb28c3badd6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1c50de89a5a845668696dcb28c3badd6, State: Initialized, Role: FOLLOWER
I20260129 02:58:48.277148 20631 consensus_queue.cc:260] T c2633a6fc34c44579de348d208b0c01e P 1c50de89a5a845668696dcb28c3badd6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:58:48.279260 20631 ts_tablet_manager.cc:1434] T c2633a6fc34c44579de348d208b0c01e P 1c50de89a5a845668696dcb28c3badd6: Time spent starting tablet: real 0.007s user 0.004s sys 0.000s
I20260129 02:58:48.280113 20631 tablet_bootstrap.cc:492] T a0fcc58d6fb6473cafbd9d34d5195269 P 1c50de89a5a845668696dcb28c3badd6: Bootstrap starting.
I20260129 02:58:48.285000 20587 raft_consensus.cc:3060] T c2633a6fc34c44579de348d208b0c01e P 1c50de89a5a845668696dcb28c3badd6 [term 0 FOLLOWER]: Advancing to term 1
I20260129 02:58:48.288843 20631 tablet_bootstrap.cc:654] T a0fcc58d6fb6473cafbd9d34d5195269 P 1c50de89a5a845668696dcb28c3badd6: Neither blocks nor log segments found. Creating new log.
I20260129 02:58:48.289278 20638 raft_consensus.cc:493] T 42d27a28d89b409e8807ec0efd97af69 P 197b880fa39d432a9b472845876bfaf0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260129 02:58:48.289752 20638 raft_consensus.cc:515] T 42d27a28d89b409e8807ec0efd97af69 P 197b880fa39d432a9b472845876bfaf0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:58:48.295964 20630 tablet_bootstrap.cc:492] T a0fcc58d6fb6473cafbd9d34d5195269 P 974b802141bd4f13beffef2aa03f88f9: No bootstrap required, opened a new log
I20260129 02:58:48.296454 20630 ts_tablet_manager.cc:1403] T a0fcc58d6fb6473cafbd9d34d5195269 P 974b802141bd4f13beffef2aa03f88f9: Time spent bootstrapping tablet: real 0.054s user 0.010s sys 0.007s
I20260129 02:58:48.291765 20638 leader_election.cc:290] T 42d27a28d89b409e8807ec0efd97af69 P 197b880fa39d432a9b472845876bfaf0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 974b802141bd4f13beffef2aa03f88f9 (127.18.211.2:34517), 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3:37735)
I20260129 02:58:48.299654 20516 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "42d27a28d89b409e8807ec0efd97af69" candidate_uuid: "197b880fa39d432a9b472845876bfaf0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "974b802141bd4f13beffef2aa03f88f9" is_pre_election: true
I20260129 02:58:48.300443 20516 raft_consensus.cc:2468] T 42d27a28d89b409e8807ec0efd97af69 P 974b802141bd4f13beffef2aa03f88f9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 197b880fa39d432a9b472845876bfaf0 in term 0.
I20260129 02:58:48.301371 20630 raft_consensus.cc:359] T a0fcc58d6fb6473cafbd9d34d5195269 P 974b802141bd4f13beffef2aa03f88f9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:58:48.301625 20407 leader_election.cc:304] T 42d27a28d89b409e8807ec0efd97af69 P 197b880fa39d432a9b472845876bfaf0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 197b880fa39d432a9b472845876bfaf0, 974b802141bd4f13beffef2aa03f88f9; no voters:
W20260129 02:58:48.303290 20405 consensus_peers.cc:597] T 0340229d521b432580ae926f6062e3f7 P 197b880fa39d432a9b472845876bfaf0 -> Peer 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3:37735): Couldn't send request to peer 1c50de89a5a845668696dcb28c3badd6. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20260129 02:58:48.304131 20638 raft_consensus.cc:2804] T 42d27a28d89b409e8807ec0efd97af69 P 197b880fa39d432a9b472845876bfaf0 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260129 02:58:48.304492 20638 raft_consensus.cc:493] T 42d27a28d89b409e8807ec0efd97af69 P 197b880fa39d432a9b472845876bfaf0 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260129 02:58:48.304814 20638 raft_consensus.cc:3060] T 42d27a28d89b409e8807ec0efd97af69 P 197b880fa39d432a9b472845876bfaf0 [term 0 FOLLOWER]: Advancing to term 1
I20260129 02:58:48.304908 20630 raft_consensus.cc:385] T a0fcc58d6fb6473cafbd9d34d5195269 P 974b802141bd4f13beffef2aa03f88f9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260129 02:58:48.305315 20630 raft_consensus.cc:740] T a0fcc58d6fb6473cafbd9d34d5195269 P 974b802141bd4f13beffef2aa03f88f9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 974b802141bd4f13beffef2aa03f88f9, State: Initialized, Role: FOLLOWER
I20260129 02:58:48.306092 20630 consensus_queue.cc:260] T a0fcc58d6fb6473cafbd9d34d5195269 P 974b802141bd4f13beffef2aa03f88f9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:58:48.307096 20588 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "42d27a28d89b409e8807ec0efd97af69" candidate_uuid: "197b880fa39d432a9b472845876bfaf0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c50de89a5a845668696dcb28c3badd6" is_pre_election: true
W20260129 02:58:48.309166 20405 leader_election.cc:343] T 42d27a28d89b409e8807ec0efd97af69 P 197b880fa39d432a9b472845876bfaf0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3:37735): Illegal state: must be running to vote when last-logged opid is not known
I20260129 02:58:48.307893 20337 catalog_manager.cc:5654] T cedce331e1584ac8b636c9b8a72b2ba0 P 974b802141bd4f13beffef2aa03f88f9 reported cstate change: term changed from 0 to 1, leader changed from <none> to 974b802141bd4f13beffef2aa03f88f9 (127.18.211.2). New cstate: current_term: 1 leader_uuid: "974b802141bd4f13beffef2aa03f88f9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } health_report { overall_health: UNKNOWN } } }
I20260129 02:58:48.313905 20630 ts_tablet_manager.cc:1434] T a0fcc58d6fb6473cafbd9d34d5195269 P 974b802141bd4f13beffef2aa03f88f9: Time spent starting tablet: real 0.017s user 0.005s sys 0.001s
I20260129 02:58:48.315047 20630 tablet_bootstrap.cc:492] T c934d31efd3f4e5195605ddc354d14e1 P 974b802141bd4f13beffef2aa03f88f9: Bootstrap starting.
I20260129 02:58:48.317903 20721 raft_consensus.cc:493] T 18f30ebaa4174fd28c958e72d0cc7e24 P 197b880fa39d432a9b472845876bfaf0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260129 02:58:48.320642 20721 raft_consensus.cc:515] T 18f30ebaa4174fd28c958e72d0cc7e24 P 197b880fa39d432a9b472845876bfaf0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:58:48.322479 20638 raft_consensus.cc:515] T 42d27a28d89b409e8807ec0efd97af69 P 197b880fa39d432a9b472845876bfaf0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:58:48.323084 20721 leader_election.cc:290] T 18f30ebaa4174fd28c958e72d0cc7e24 P 197b880fa39d432a9b472845876bfaf0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 974b802141bd4f13beffef2aa03f88f9 (127.18.211.2:34517), 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3:37735)
I20260129 02:58:48.324476 20638 leader_election.cc:290] T 42d27a28d89b409e8807ec0efd97af69 P 197b880fa39d432a9b472845876bfaf0 [CANDIDATE]: Term 1 election: Requested vote from peers 974b802141bd4f13beffef2aa03f88f9 (127.18.211.2:34517), 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3:37735)
I20260129 02:58:48.327112 20516 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "18f30ebaa4174fd28c958e72d0cc7e24" candidate_uuid: "197b880fa39d432a9b472845876bfaf0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "974b802141bd4f13beffef2aa03f88f9" is_pre_election: true
I20260129 02:58:48.327710 20516 raft_consensus.cc:2468] T 18f30ebaa4174fd28c958e72d0cc7e24 P 974b802141bd4f13beffef2aa03f88f9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 197b880fa39d432a9b472845876bfaf0 in term 0.
I20260129 02:58:48.326643 20588 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "18f30ebaa4174fd28c958e72d0cc7e24" candidate_uuid: "197b880fa39d432a9b472845876bfaf0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c50de89a5a845668696dcb28c3badd6" is_pre_election: true
I20260129 02:58:48.328624 20590 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "42d27a28d89b409e8807ec0efd97af69" candidate_uuid: "197b880fa39d432a9b472845876bfaf0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c50de89a5a845668696dcb28c3badd6"
I20260129 02:58:48.328837 20588 raft_consensus.cc:2468] T 18f30ebaa4174fd28c958e72d0cc7e24 P 1c50de89a5a845668696dcb28c3badd6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 197b880fa39d432a9b472845876bfaf0 in term 0.
W20260129 02:58:48.329742 20405 leader_election.cc:343] T 42d27a28d89b409e8807ec0efd97af69 P 197b880fa39d432a9b472845876bfaf0 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3:37735): Illegal state: must be running to vote when last-logged opid is not known
I20260129 02:58:48.330487 20516 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "42d27a28d89b409e8807ec0efd97af69" candidate_uuid: "197b880fa39d432a9b472845876bfaf0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "974b802141bd4f13beffef2aa03f88f9"
I20260129 02:58:48.330992 20516 raft_consensus.cc:3060] T 42d27a28d89b409e8807ec0efd97af69 P 974b802141bd4f13beffef2aa03f88f9 [term 0 FOLLOWER]: Advancing to term 1
I20260129 02:58:48.332576 20407 leader_election.cc:304] T 18f30ebaa4174fd28c958e72d0cc7e24 P 197b880fa39d432a9b472845876bfaf0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 197b880fa39d432a9b472845876bfaf0, 974b802141bd4f13beffef2aa03f88f9; no voters:
I20260129 02:58:48.334117 20658 raft_consensus.cc:2804] T 18f30ebaa4174fd28c958e72d0cc7e24 P 197b880fa39d432a9b472845876bfaf0 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260129 02:58:48.334540 20658 raft_consensus.cc:493] T 18f30ebaa4174fd28c958e72d0cc7e24 P 197b880fa39d432a9b472845876bfaf0 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260129 02:58:48.334863 20658 raft_consensus.cc:3060] T 18f30ebaa4174fd28c958e72d0cc7e24 P 197b880fa39d432a9b472845876bfaf0 [term 0 FOLLOWER]: Advancing to term 1
I20260129 02:58:48.335424 20721 consensus_queue.cc:1048] T 7a9a69cd82724b1faed1d10aa091fc4a P 197b880fa39d432a9b472845876bfaf0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260129 02:58:48.340586 20630 tablet_bootstrap.cc:654] T c934d31efd3f4e5195605ddc354d14e1 P 974b802141bd4f13beffef2aa03f88f9: Neither blocks nor log segments found. Creating new log.
I20260129 02:58:48.342412 20516 raft_consensus.cc:2468] T 42d27a28d89b409e8807ec0efd97af69 P 974b802141bd4f13beffef2aa03f88f9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 197b880fa39d432a9b472845876bfaf0 in term 1.
I20260129 02:58:48.346922 20407 leader_election.cc:304] T 42d27a28d89b409e8807ec0efd97af69 P 197b880fa39d432a9b472845876bfaf0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 197b880fa39d432a9b472845876bfaf0, 974b802141bd4f13beffef2aa03f88f9; no voters: 1c50de89a5a845668696dcb28c3badd6
I20260129 02:58:48.348006 20721 consensus_queue.cc:1048] T 7a9a69cd82724b1faed1d10aa091fc4a P 197b880fa39d432a9b472845876bfaf0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260129 02:58:48.351373 20638 raft_consensus.cc:2804] T 42d27a28d89b409e8807ec0efd97af69 P 197b880fa39d432a9b472845876bfaf0 [term 1 FOLLOWER]: Leader election won for term 1
I20260129 02:58:48.352012 20638 raft_consensus.cc:697] T 42d27a28d89b409e8807ec0efd97af69 P 197b880fa39d432a9b472845876bfaf0 [term 1 LEADER]: Becoming Leader. State: Replica: 197b880fa39d432a9b472845876bfaf0, State: Running, Role: LEADER
I20260129 02:58:48.352821 20638 consensus_queue.cc:237] T 42d27a28d89b409e8807ec0efd97af69 P 197b880fa39d432a9b472845876bfaf0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:58:48.353602 20337 catalog_manager.cc:5654] T a644c02b16af47679824ccd060ac10fd P 1c50de89a5a845668696dcb28c3badd6 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3). New cstate: current_term: 1 leader_uuid: "1c50de89a5a845668696dcb28c3badd6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } health_report { overall_health: HEALTHY } } }
I20260129 02:58:48.355118 20336 catalog_manager.cc:5654] T cb0b3f259ebc4310927338a8740e87b8 P 974b802141bd4f13beffef2aa03f88f9 reported cstate change: term changed from 0 to 1, leader changed from <none> to 974b802141bd4f13beffef2aa03f88f9 (127.18.211.2). New cstate: current_term: 1 leader_uuid: "974b802141bd4f13beffef2aa03f88f9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } health_report { overall_health: UNKNOWN } } }
I20260129 02:58:48.368806 20658 raft_consensus.cc:515] T 18f30ebaa4174fd28c958e72d0cc7e24 P 197b880fa39d432a9b472845876bfaf0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:58:48.376159 20658 leader_election.cc:290] T 18f30ebaa4174fd28c958e72d0cc7e24 P 197b880fa39d432a9b472845876bfaf0 [CANDIDATE]: Term 1 election: Requested vote from peers 974b802141bd4f13beffef2aa03f88f9 (127.18.211.2:34517), 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3:37735)
I20260129 02:58:48.381359 20588 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "18f30ebaa4174fd28c958e72d0cc7e24" candidate_uuid: "197b880fa39d432a9b472845876bfaf0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c50de89a5a845668696dcb28c3badd6"
I20260129 02:58:48.382203 20588 raft_consensus.cc:3060] T 18f30ebaa4174fd28c958e72d0cc7e24 P 1c50de89a5a845668696dcb28c3badd6 [term 0 FOLLOWER]: Advancing to term 1
I20260129 02:58:48.384020 20832 consensus_queue.cc:1048] T e33a417cf9c54cf5af73a4ee3aac4d3a P 1c50de89a5a845668696dcb28c3badd6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260129 02:58:48.389166 20631 tablet_bootstrap.cc:492] T a0fcc58d6fb6473cafbd9d34d5195269 P 1c50de89a5a845668696dcb28c3badd6: No bootstrap required, opened a new log
I20260129 02:58:48.389689 20631 ts_tablet_manager.cc:1403] T a0fcc58d6fb6473cafbd9d34d5195269 P 1c50de89a5a845668696dcb28c3badd6: Time spent bootstrapping tablet: real 0.110s user 0.017s sys 0.006s
I20260129 02:58:48.392793 20633 tablet_bootstrap.cc:492] T 0678b8b468bd43cdbc3e485900844821 P 197b880fa39d432a9b472845876bfaf0: No bootstrap required, opened a new log
I20260129 02:58:48.393262 20633 ts_tablet_manager.cc:1403] T 0678b8b468bd43cdbc3e485900844821 P 197b880fa39d432a9b472845876bfaf0: Time spent bootstrapping tablet: real 0.167s user 0.013s sys 0.012s
I20260129 02:58:48.405248 20631 raft_consensus.cc:359] T a0fcc58d6fb6473cafbd9d34d5195269 P 1c50de89a5a845668696dcb28c3badd6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:58:48.406764 20631 raft_consensus.cc:385] T a0fcc58d6fb6473cafbd9d34d5195269 P 1c50de89a5a845668696dcb28c3badd6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260129 02:58:48.407099 20631 raft_consensus.cc:740] T a0fcc58d6fb6473cafbd9d34d5195269 P 1c50de89a5a845668696dcb28c3badd6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1c50de89a5a845668696dcb28c3badd6, State: Initialized, Role: FOLLOWER
I20260129 02:58:48.407805 20631 consensus_queue.cc:260] T a0fcc58d6fb6473cafbd9d34d5195269 P 1c50de89a5a845668696dcb28c3badd6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:58:48.408601 20516 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "18f30ebaa4174fd28c958e72d0cc7e24" candidate_uuid: "197b880fa39d432a9b472845876bfaf0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "974b802141bd4f13beffef2aa03f88f9"
I20260129 02:58:48.409060 20516 raft_consensus.cc:3060] T 18f30ebaa4174fd28c958e72d0cc7e24 P 974b802141bd4f13beffef2aa03f88f9 [term 0 FOLLOWER]: Advancing to term 1
I20260129 02:58:48.411343 20336 catalog_manager.cc:5654] T 42d27a28d89b409e8807ec0efd97af69 P 197b880fa39d432a9b472845876bfaf0 reported cstate change: term changed from 0 to 1, leader changed from <none> to 197b880fa39d432a9b472845876bfaf0 (127.18.211.1). New cstate: current_term: 1 leader_uuid: "197b880fa39d432a9b472845876bfaf0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } health_report { overall_health: UNKNOWN } } }
I20260129 02:58:48.417335 20633 raft_consensus.cc:359] T 0678b8b468bd43cdbc3e485900844821 P 197b880fa39d432a9b472845876bfaf0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:58:48.419363 20633 raft_consensus.cc:385] T 0678b8b468bd43cdbc3e485900844821 P 197b880fa39d432a9b472845876bfaf0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260129 02:58:48.419612 20633 raft_consensus.cc:740] T 0678b8b468bd43cdbc3e485900844821 P 197b880fa39d432a9b472845876bfaf0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 197b880fa39d432a9b472845876bfaf0, State: Initialized, Role: FOLLOWER
I20260129 02:58:48.420291 20633 consensus_queue.cc:260] T 0678b8b468bd43cdbc3e485900844821 P 197b880fa39d432a9b472845876bfaf0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:58:48.431988 20633 ts_tablet_manager.cc:1434] T 0678b8b468bd43cdbc3e485900844821 P 197b880fa39d432a9b472845876bfaf0: Time spent starting tablet: real 0.038s user 0.002s sys 0.006s
I20260129 02:58:48.433250 20633 tablet_bootstrap.cc:492] T 7d9494f5584d4d5db8b9adf1c8fc888b P 197b880fa39d432a9b472845876bfaf0: Bootstrap starting.
I20260129 02:58:48.440864 20631 ts_tablet_manager.cc:1434] T a0fcc58d6fb6473cafbd9d34d5195269 P 1c50de89a5a845668696dcb28c3badd6: Time spent starting tablet: real 0.051s user 0.006s sys 0.001s
I20260129 02:58:48.441010 20516 raft_consensus.cc:2468] T 18f30ebaa4174fd28c958e72d0cc7e24 P 974b802141bd4f13beffef2aa03f88f9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 197b880fa39d432a9b472845876bfaf0 in term 1.
I20260129 02:58:48.441771 20631 tablet_bootstrap.cc:492] T cedce331e1584ac8b636c9b8a72b2ba0 P 1c50de89a5a845668696dcb28c3badd6: Bootstrap starting.
I20260129 02:58:48.440282 20869 raft_consensus.cc:493] T a0fcc58d6fb6473cafbd9d34d5195269 P 974b802141bd4f13beffef2aa03f88f9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260129 02:58:48.442983 20869 raft_consensus.cc:515] T a0fcc58d6fb6473cafbd9d34d5195269 P 974b802141bd4f13beffef2aa03f88f9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:58:48.446605 20407 leader_election.cc:304] T 18f30ebaa4174fd28c958e72d0cc7e24 P 197b880fa39d432a9b472845876bfaf0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 197b880fa39d432a9b472845876bfaf0, 974b802141bd4f13beffef2aa03f88f9; no voters:
I20260129 02:58:48.447780 20658 raft_consensus.cc:2804] T 18f30ebaa4174fd28c958e72d0cc7e24 P 197b880fa39d432a9b472845876bfaf0 [term 1 FOLLOWER]: Leader election won for term 1
I20260129 02:58:48.448230 20658 raft_consensus.cc:697] T 18f30ebaa4174fd28c958e72d0cc7e24 P 197b880fa39d432a9b472845876bfaf0 [term 1 LEADER]: Becoming Leader. State: Replica: 197b880fa39d432a9b472845876bfaf0, State: Running, Role: LEADER
I20260129 02:58:48.448890 20869 leader_election.cc:290] T a0fcc58d6fb6473cafbd9d34d5195269 P 974b802141bd4f13beffef2aa03f88f9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 197b880fa39d432a9b472845876bfaf0 (127.18.211.1:42027), 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3:37735)
I20260129 02:58:48.448989 20658 consensus_queue.cc:237] T 18f30ebaa4174fd28c958e72d0cc7e24 P 197b880fa39d432a9b472845876bfaf0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:58:48.452708 20588 raft_consensus.cc:2468] T 18f30ebaa4174fd28c958e72d0cc7e24 P 1c50de89a5a845668696dcb28c3badd6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 197b880fa39d432a9b472845876bfaf0 in term 1.
I20260129 02:58:48.453339 20589 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a0fcc58d6fb6473cafbd9d34d5195269" candidate_uuid: "974b802141bd4f13beffef2aa03f88f9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c50de89a5a845668696dcb28c3badd6" is_pre_election: true
I20260129 02:58:48.453944 20589 raft_consensus.cc:2468] T a0fcc58d6fb6473cafbd9d34d5195269 P 1c50de89a5a845668696dcb28c3badd6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 974b802141bd4f13beffef2aa03f88f9 in term 0.
I20260129 02:58:48.454963 20438 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a0fcc58d6fb6473cafbd9d34d5195269" candidate_uuid: "974b802141bd4f13beffef2aa03f88f9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "197b880fa39d432a9b472845876bfaf0" is_pre_election: true
I20260129 02:58:48.455724 20438 raft_consensus.cc:2468] T a0fcc58d6fb6473cafbd9d34d5195269 P 197b880fa39d432a9b472845876bfaf0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 974b802141bd4f13beffef2aa03f88f9 in term 0.
I20260129 02:58:48.462344 20479 leader_election.cc:304] T a0fcc58d6fb6473cafbd9d34d5195269 P 974b802141bd4f13beffef2aa03f88f9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 197b880fa39d432a9b472845876bfaf0, 974b802141bd4f13beffef2aa03f88f9; no voters:
I20260129 02:58:48.464897 20832 consensus_queue.cc:1048] T e33a417cf9c54cf5af73a4ee3aac4d3a P 1c50de89a5a845668696dcb28c3badd6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260129 02:58:48.464866 20739 raft_consensus.cc:2804] T a0fcc58d6fb6473cafbd9d34d5195269 P 974b802141bd4f13beffef2aa03f88f9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260129 02:58:48.466945 20630 tablet_bootstrap.cc:492] T c934d31efd3f4e5195605ddc354d14e1 P 974b802141bd4f13beffef2aa03f88f9: No bootstrap required, opened a new log
I20260129 02:58:48.467725 20630 ts_tablet_manager.cc:1403] T c934d31efd3f4e5195605ddc354d14e1 P 974b802141bd4f13beffef2aa03f88f9: Time spent bootstrapping tablet: real 0.153s user 0.011s sys 0.013s
I20260129 02:58:48.467317 20739 raft_consensus.cc:493] T a0fcc58d6fb6473cafbd9d34d5195269 P 974b802141bd4f13beffef2aa03f88f9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260129 02:58:48.468323 20739 raft_consensus.cc:3060] T a0fcc58d6fb6473cafbd9d34d5195269 P 974b802141bd4f13beffef2aa03f88f9 [term 0 FOLLOWER]: Advancing to term 1
I20260129 02:58:48.473330 20633 tablet_bootstrap.cc:654] T 7d9494f5584d4d5db8b9adf1c8fc888b P 197b880fa39d432a9b472845876bfaf0: Neither blocks nor log segments found. Creating new log.
I20260129 02:58:48.484767 20739 raft_consensus.cc:515] T a0fcc58d6fb6473cafbd9d34d5195269 P 974b802141bd4f13beffef2aa03f88f9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:58:48.486961 20739 leader_election.cc:290] T a0fcc58d6fb6473cafbd9d34d5195269 P 974b802141bd4f13beffef2aa03f88f9 [CANDIDATE]: Term 1 election: Requested vote from peers 197b880fa39d432a9b472845876bfaf0 (127.18.211.1:42027), 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3:37735)
I20260129 02:58:48.488142 20589 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a0fcc58d6fb6473cafbd9d34d5195269" candidate_uuid: "974b802141bd4f13beffef2aa03f88f9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c50de89a5a845668696dcb28c3badd6"
I20260129 02:58:48.489476 20589 raft_consensus.cc:3060] T a0fcc58d6fb6473cafbd9d34d5195269 P 1c50de89a5a845668696dcb28c3badd6 [term 0 FOLLOWER]: Advancing to term 1
I20260129 02:58:48.481050 20335 catalog_manager.cc:5654] T 18f30ebaa4174fd28c958e72d0cc7e24 P 197b880fa39d432a9b472845876bfaf0 reported cstate change: term changed from 0 to 1, leader changed from <none> to 197b880fa39d432a9b472845876bfaf0 (127.18.211.1). New cstate: current_term: 1 leader_uuid: "197b880fa39d432a9b472845876bfaf0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } health_report { overall_health: UNKNOWN } } }
I20260129 02:58:48.496810 20630 raft_consensus.cc:359] T c934d31efd3f4e5195605ddc354d14e1 P 974b802141bd4f13beffef2aa03f88f9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:58:48.489794 20438 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a0fcc58d6fb6473cafbd9d34d5195269" candidate_uuid: "974b802141bd4f13beffef2aa03f88f9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "197b880fa39d432a9b472845876bfaf0"
I20260129 02:58:48.497498 20630 raft_consensus.cc:385] T c934d31efd3f4e5195605ddc354d14e1 P 974b802141bd4f13beffef2aa03f88f9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260129 02:58:48.497786 20630 raft_consensus.cc:740] T c934d31efd3f4e5195605ddc354d14e1 P 974b802141bd4f13beffef2aa03f88f9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 974b802141bd4f13beffef2aa03f88f9, State: Initialized, Role: FOLLOWER
I20260129 02:58:48.498375 20631 tablet_bootstrap.cc:654] T cedce331e1584ac8b636c9b8a72b2ba0 P 1c50de89a5a845668696dcb28c3badd6: Neither blocks nor log segments found. Creating new log.
I20260129 02:58:48.498464 20630 consensus_queue.cc:260] T c934d31efd3f4e5195605ddc354d14e1 P 974b802141bd4f13beffef2aa03f88f9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:58:48.500831 20630 ts_tablet_manager.cc:1434] T c934d31efd3f4e5195605ddc354d14e1 P 974b802141bd4f13beffef2aa03f88f9: Time spent starting tablet: real 0.033s user 0.006s sys 0.001s
I20260129 02:58:48.501798 20630 tablet_bootstrap.cc:492] T a644c02b16af47679824ccd060ac10fd P 974b802141bd4f13beffef2aa03f88f9: Bootstrap starting.
I20260129 02:58:48.501883 20658 raft_consensus.cc:493] T 0678b8b468bd43cdbc3e485900844821 P 197b880fa39d432a9b472845876bfaf0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260129 02:58:48.501012 20438 raft_consensus.cc:3060] T a0fcc58d6fb6473cafbd9d34d5195269 P 197b880fa39d432a9b472845876bfaf0 [term 0 FOLLOWER]: Advancing to term 1
I20260129 02:58:48.502396 20658 raft_consensus.cc:515] T 0678b8b468bd43cdbc3e485900844821 P 197b880fa39d432a9b472845876bfaf0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:58:48.509183 20438 raft_consensus.cc:2468] T a0fcc58d6fb6473cafbd9d34d5195269 P 197b880fa39d432a9b472845876bfaf0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 974b802141bd4f13beffef2aa03f88f9 in term 1.
I20260129 02:58:48.510615 20514 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0678b8b468bd43cdbc3e485900844821" candidate_uuid: "197b880fa39d432a9b472845876bfaf0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "974b802141bd4f13beffef2aa03f88f9" is_pre_election: true
I20260129 02:58:48.510782 20479 leader_election.cc:304] T a0fcc58d6fb6473cafbd9d34d5195269 P 974b802141bd4f13beffef2aa03f88f9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 197b880fa39d432a9b472845876bfaf0, 974b802141bd4f13beffef2aa03f88f9; no voters:
I20260129 02:58:48.511274 20514 raft_consensus.cc:2468] T 0678b8b468bd43cdbc3e485900844821 P 974b802141bd4f13beffef2aa03f88f9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 197b880fa39d432a9b472845876bfaf0 in term 0.
I20260129 02:58:48.511850 20740 raft_consensus.cc:2804] T a0fcc58d6fb6473cafbd9d34d5195269 P 974b802141bd4f13beffef2aa03f88f9 [term 1 FOLLOWER]: Leader election won for term 1
I20260129 02:58:48.512362 20658 leader_election.cc:290] T 0678b8b468bd43cdbc3e485900844821 P 197b880fa39d432a9b472845876bfaf0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 974b802141bd4f13beffef2aa03f88f9 (127.18.211.2:34517), 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3:37735)
I20260129 02:58:48.512408 20740 raft_consensus.cc:697] T a0fcc58d6fb6473cafbd9d34d5195269 P 974b802141bd4f13beffef2aa03f88f9 [term 1 LEADER]: Becoming Leader. State: Replica: 974b802141bd4f13beffef2aa03f88f9, State: Running, Role: LEADER
I20260129 02:58:48.513306 20740 consensus_queue.cc:237] T a0fcc58d6fb6473cafbd9d34d5195269 P 974b802141bd4f13beffef2aa03f88f9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:58:48.514060 20407 leader_election.cc:304] T 0678b8b468bd43cdbc3e485900844821 P 197b880fa39d432a9b472845876bfaf0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 197b880fa39d432a9b472845876bfaf0, 974b802141bd4f13beffef2aa03f88f9; no voters:
I20260129 02:58:48.514844 20658 raft_consensus.cc:2804] T 0678b8b468bd43cdbc3e485900844821 P 197b880fa39d432a9b472845876bfaf0 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260129 02:58:48.515218 20658 raft_consensus.cc:493] T 0678b8b468bd43cdbc3e485900844821 P 197b880fa39d432a9b472845876bfaf0 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260129 02:58:48.516431 20590 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0678b8b468bd43cdbc3e485900844821" candidate_uuid: "197b880fa39d432a9b472845876bfaf0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c50de89a5a845668696dcb28c3badd6" is_pre_election: true
I20260129 02:58:48.515534 20658 raft_consensus.cc:3060] T 0678b8b468bd43cdbc3e485900844821 P 197b880fa39d432a9b472845876bfaf0 [term 0 FOLLOWER]: Advancing to term 1
W20260129 02:58:48.517863 20405 leader_election.cc:343] T 0678b8b468bd43cdbc3e485900844821 P 197b880fa39d432a9b472845876bfaf0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3:37735): Illegal state: must be running to vote when last-logged opid is not known
I20260129 02:58:48.523363 20589 raft_consensus.cc:2468] T a0fcc58d6fb6473cafbd9d34d5195269 P 1c50de89a5a845668696dcb28c3badd6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 974b802141bd4f13beffef2aa03f88f9 in term 1.
I20260129 02:58:48.523263 20658 raft_consensus.cc:515] T 0678b8b468bd43cdbc3e485900844821 P 197b880fa39d432a9b472845876bfaf0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:58:48.531136 20658 leader_election.cc:290] T 0678b8b468bd43cdbc3e485900844821 P 197b880fa39d432a9b472845876bfaf0 [CANDIDATE]: Term 1 election: Requested vote from peers 974b802141bd4f13beffef2aa03f88f9 (127.18.211.2:34517), 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3:37735)
I20260129 02:58:48.532059 20630 tablet_bootstrap.cc:654] T a644c02b16af47679824ccd060ac10fd P 974b802141bd4f13beffef2aa03f88f9: Neither blocks nor log segments found. Creating new log.
I20260129 02:58:48.535367 20589 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0678b8b468bd43cdbc3e485900844821" candidate_uuid: "197b880fa39d432a9b472845876bfaf0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c50de89a5a845668696dcb28c3badd6"
W20260129 02:58:48.537308 20405 leader_election.cc:343] T 0678b8b468bd43cdbc3e485900844821 P 197b880fa39d432a9b472845876bfaf0 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3:37735): Illegal state: must be running to vote when last-logged opid is not known
I20260129 02:58:48.537609 20516 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0678b8b468bd43cdbc3e485900844821" candidate_uuid: "197b880fa39d432a9b472845876bfaf0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "974b802141bd4f13beffef2aa03f88f9"
I20260129 02:58:48.538149 20516 raft_consensus.cc:3060] T 0678b8b468bd43cdbc3e485900844821 P 974b802141bd4f13beffef2aa03f88f9 [term 0 FOLLOWER]: Advancing to term 1
I20260129 02:58:48.538655 20631 tablet_bootstrap.cc:492] T cedce331e1584ac8b636c9b8a72b2ba0 P 1c50de89a5a845668696dcb28c3badd6: No bootstrap required, opened a new log
I20260129 02:58:48.539110 20631 ts_tablet_manager.cc:1403] T cedce331e1584ac8b636c9b8a72b2ba0 P 1c50de89a5a845668696dcb28c3badd6: Time spent bootstrapping tablet: real 0.097s user 0.011s sys 0.001s
I20260129 02:58:48.540227 20630 tablet_bootstrap.cc:492] T a644c02b16af47679824ccd060ac10fd P 974b802141bd4f13beffef2aa03f88f9: No bootstrap required, opened a new log
I20260129 02:58:48.540900 20630 ts_tablet_manager.cc:1403] T a644c02b16af47679824ccd060ac10fd P 974b802141bd4f13beffef2aa03f88f9: Time spent bootstrapping tablet: real 0.039s user 0.004s sys 0.009s
I20260129 02:58:48.542977 20631 raft_consensus.cc:359] T cedce331e1584ac8b636c9b8a72b2ba0 P 1c50de89a5a845668696dcb28c3badd6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:58:48.543673 20631 raft_consensus.cc:385] T cedce331e1584ac8b636c9b8a72b2ba0 P 1c50de89a5a845668696dcb28c3badd6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260129 02:58:48.543413 20630 raft_consensus.cc:359] T a644c02b16af47679824ccd060ac10fd P 974b802141bd4f13beffef2aa03f88f9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:58:48.544005 20631 raft_consensus.cc:740] T cedce331e1584ac8b636c9b8a72b2ba0 P 1c50de89a5a845668696dcb28c3badd6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1c50de89a5a845668696dcb28c3badd6, State: Initialized, Role: FOLLOWER
I20260129 02:58:48.544130 20630 raft_consensus.cc:385] T a644c02b16af47679824ccd060ac10fd P 974b802141bd4f13beffef2aa03f88f9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260129 02:58:48.544479 20630 raft_consensus.cc:740] T a644c02b16af47679824ccd060ac10fd P 974b802141bd4f13beffef2aa03f88f9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 974b802141bd4f13beffef2aa03f88f9, State: Initialized, Role: FOLLOWER
I20260129 02:58:48.544688 20631 consensus_queue.cc:260] T cedce331e1584ac8b636c9b8a72b2ba0 P 1c50de89a5a845668696dcb28c3badd6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:58:48.545120 20630 consensus_queue.cc:260] T a644c02b16af47679824ccd060ac10fd P 974b802141bd4f13beffef2aa03f88f9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:58:48.546530 20633 tablet_bootstrap.cc:492] T 7d9494f5584d4d5db8b9adf1c8fc888b P 197b880fa39d432a9b472845876bfaf0: No bootstrap required, opened a new log
I20260129 02:58:48.547928 20633 ts_tablet_manager.cc:1403] T 7d9494f5584d4d5db8b9adf1c8fc888b P 197b880fa39d432a9b472845876bfaf0: Time spent bootstrapping tablet: real 0.115s user 0.016s sys 0.006s
I20260129 02:58:48.548148 20630 ts_tablet_manager.cc:1434] T a644c02b16af47679824ccd060ac10fd P 974b802141bd4f13beffef2aa03f88f9: Time spent starting tablet: real 0.007s user 0.007s sys 0.000s
I20260129 02:58:48.549083 20630 tablet_bootstrap.cc:492] T c891d2bff4ce4568861bbc992e91fbb0 P 974b802141bd4f13beffef2aa03f88f9: Bootstrap starting.
I20260129 02:58:48.550585 20633 raft_consensus.cc:359] T 7d9494f5584d4d5db8b9adf1c8fc888b P 197b880fa39d432a9b472845876bfaf0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:58:48.551539 20633 raft_consensus.cc:385] T 7d9494f5584d4d5db8b9adf1c8fc888b P 197b880fa39d432a9b472845876bfaf0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260129 02:58:48.551913 20633 raft_consensus.cc:740] T 7d9494f5584d4d5db8b9adf1c8fc888b P 197b880fa39d432a9b472845876bfaf0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 197b880fa39d432a9b472845876bfaf0, State: Initialized, Role: FOLLOWER
I20260129 02:58:48.552685 20633 consensus_queue.cc:260] T 7d9494f5584d4d5db8b9adf1c8fc888b P 197b880fa39d432a9b472845876bfaf0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:58:48.554796 20630 tablet_bootstrap.cc:654] T c891d2bff4ce4568861bbc992e91fbb0 P 974b802141bd4f13beffef2aa03f88f9: Neither blocks nor log segments found. Creating new log.
I20260129 02:58:48.555238 20633 ts_tablet_manager.cc:1434] T 7d9494f5584d4d5db8b9adf1c8fc888b P 197b880fa39d432a9b472845876bfaf0: Time spent starting tablet: real 0.007s user 0.006s sys 0.000s
I20260129 02:58:48.556232 20633 tablet_bootstrap.cc:492] T 8055232cf0004d40a85f3bf983fe00a9 P 197b880fa39d432a9b472845876bfaf0: Bootstrap starting.
I20260129 02:58:48.563839 20633 tablet_bootstrap.cc:654] T 8055232cf0004d40a85f3bf983fe00a9 P 197b880fa39d432a9b472845876bfaf0: Neither blocks nor log segments found. Creating new log.
I20260129 02:58:48.568918 20630 tablet_bootstrap.cc:492] T c891d2bff4ce4568861bbc992e91fbb0 P 974b802141bd4f13beffef2aa03f88f9: No bootstrap required, opened a new log
I20260129 02:58:48.569486 20630 ts_tablet_manager.cc:1403] T c891d2bff4ce4568861bbc992e91fbb0 P 974b802141bd4f13beffef2aa03f88f9: Time spent bootstrapping tablet: real 0.021s user 0.007s sys 0.009s
I20260129 02:58:48.572892 20630 raft_consensus.cc:359] T c891d2bff4ce4568861bbc992e91fbb0 P 974b802141bd4f13beffef2aa03f88f9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:58:48.573644 20630 raft_consensus.cc:385] T c891d2bff4ce4568861bbc992e91fbb0 P 974b802141bd4f13beffef2aa03f88f9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260129 02:58:48.573930 20630 raft_consensus.cc:740] T c891d2bff4ce4568861bbc992e91fbb0 P 974b802141bd4f13beffef2aa03f88f9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 974b802141bd4f13beffef2aa03f88f9, State: Initialized, Role: FOLLOWER
I20260129 02:58:48.576184 20516 raft_consensus.cc:2468] T 0678b8b468bd43cdbc3e485900844821 P 974b802141bd4f13beffef2aa03f88f9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 197b880fa39d432a9b472845876bfaf0 in term 1.
I20260129 02:58:48.574586 20630 consensus_queue.cc:260] T c891d2bff4ce4568861bbc992e91fbb0 P 974b802141bd4f13beffef2aa03f88f9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:58:48.577260 20633 tablet_bootstrap.cc:492] T 8055232cf0004d40a85f3bf983fe00a9 P 197b880fa39d432a9b472845876bfaf0: No bootstrap required, opened a new log
I20260129 02:58:48.577934 20633 ts_tablet_manager.cc:1403] T 8055232cf0004d40a85f3bf983fe00a9 P 197b880fa39d432a9b472845876bfaf0: Time spent bootstrapping tablet: real 0.022s user 0.007s sys 0.011s
I20260129 02:58:48.578250 20407 leader_election.cc:304] T 0678b8b468bd43cdbc3e485900844821 P 197b880fa39d432a9b472845876bfaf0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 197b880fa39d432a9b472845876bfaf0, 974b802141bd4f13beffef2aa03f88f9; no voters: 1c50de89a5a845668696dcb28c3badd6
I20260129 02:58:48.577529 20338 catalog_manager.cc:5654] T a0fcc58d6fb6473cafbd9d34d5195269 P 974b802141bd4f13beffef2aa03f88f9 reported cstate change: term changed from 0 to 1, leader changed from <none> to 974b802141bd4f13beffef2aa03f88f9 (127.18.211.2). New cstate: current_term: 1 leader_uuid: "974b802141bd4f13beffef2aa03f88f9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } health_report { overall_health: UNKNOWN } } }
I20260129 02:58:48.579667 20638 raft_consensus.cc:2804] T 0678b8b468bd43cdbc3e485900844821 P 197b880fa39d432a9b472845876bfaf0 [term 1 FOLLOWER]: Leader election won for term 1
I20260129 02:58:48.579829 20630 ts_tablet_manager.cc:1434] T c891d2bff4ce4568861bbc992e91fbb0 P 974b802141bd4f13beffef2aa03f88f9: Time spent starting tablet: real 0.010s user 0.005s sys 0.003s
I20260129 02:58:48.580129 20638 raft_consensus.cc:697] T 0678b8b468bd43cdbc3e485900844821 P 197b880fa39d432a9b472845876bfaf0 [term 1 LEADER]: Becoming Leader. State: Replica: 197b880fa39d432a9b472845876bfaf0, State: Running, Role: LEADER
I20260129 02:58:48.580794 20630 tablet_bootstrap.cc:492] T 00e79d8f25a34676b1d866eb2068a2b2 P 974b802141bd4f13beffef2aa03f88f9: Bootstrap starting.
I20260129 02:58:48.580868 20638 consensus_queue.cc:237] T 0678b8b468bd43cdbc3e485900844821 P 197b880fa39d432a9b472845876bfaf0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:58:48.580922 20633 raft_consensus.cc:359] T 8055232cf0004d40a85f3bf983fe00a9 P 197b880fa39d432a9b472845876bfaf0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:58:48.581881 20633 raft_consensus.cc:385] T 8055232cf0004d40a85f3bf983fe00a9 P 197b880fa39d432a9b472845876bfaf0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260129 02:58:48.582195 20633 raft_consensus.cc:740] T 8055232cf0004d40a85f3bf983fe00a9 P 197b880fa39d432a9b472845876bfaf0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 197b880fa39d432a9b472845876bfaf0, State: Initialized, Role: FOLLOWER
I20260129 02:58:48.547381 20631 ts_tablet_manager.cc:1434] T cedce331e1584ac8b636c9b8a72b2ba0 P 1c50de89a5a845668696dcb28c3badd6: Time spent starting tablet: real 0.008s user 0.007s sys 0.000s
I20260129 02:58:48.583562 20631 tablet_bootstrap.cc:492] T cb0b3f259ebc4310927338a8740e87b8 P 1c50de89a5a845668696dcb28c3badd6: Bootstrap starting.
I20260129 02:58:48.587321 20630 tablet_bootstrap.cc:654] T 00e79d8f25a34676b1d866eb2068a2b2 P 974b802141bd4f13beffef2aa03f88f9: Neither blocks nor log segments found. Creating new log.
I20260129 02:58:48.589433 20631 tablet_bootstrap.cc:654] T cb0b3f259ebc4310927338a8740e87b8 P 1c50de89a5a845668696dcb28c3badd6: Neither blocks nor log segments found. Creating new log.
I20260129 02:58:48.591048 20658 raft_consensus.cc:493] T c891d2bff4ce4568861bbc992e91fbb0 P 197b880fa39d432a9b472845876bfaf0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260129 02:58:48.591567 20658 raft_consensus.cc:515] T c891d2bff4ce4568861bbc992e91fbb0 P 197b880fa39d432a9b472845876bfaf0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:58:48.593385 20828 raft_consensus.cc:493] T 926a250fa5834e88b96ccecbefbf5086 P 197b880fa39d432a9b472845876bfaf0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260129 02:58:48.593657 20658 leader_election.cc:290] T c891d2bff4ce4568861bbc992e91fbb0 P 197b880fa39d432a9b472845876bfaf0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 974b802141bd4f13beffef2aa03f88f9 (127.18.211.2:34517), 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3:37735)
I20260129 02:58:48.594041 20828 raft_consensus.cc:515] T 926a250fa5834e88b96ccecbefbf5086 P 197b880fa39d432a9b472845876bfaf0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:58:48.598886 20828 leader_election.cc:290] T 926a250fa5834e88b96ccecbefbf5086 P 197b880fa39d432a9b472845876bfaf0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 974b802141bd4f13beffef2aa03f88f9 (127.18.211.2:34517), 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3:37735)
I20260129 02:58:48.600420 20514 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c891d2bff4ce4568861bbc992e91fbb0" candidate_uuid: "197b880fa39d432a9b472845876bfaf0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "974b802141bd4f13beffef2aa03f88f9" is_pre_election: true
I20260129 02:58:48.601075 20514 raft_consensus.cc:2468] T c891d2bff4ce4568861bbc992e91fbb0 P 974b802141bd4f13beffef2aa03f88f9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 197b880fa39d432a9b472845876bfaf0 in term 0.
I20260129 02:58:48.601814 20633 consensus_queue.cc:260] T 8055232cf0004d40a85f3bf983fe00a9 P 197b880fa39d432a9b472845876bfaf0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:58:48.603626 20514 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "926a250fa5834e88b96ccecbefbf5086" candidate_uuid: "197b880fa39d432a9b472845876bfaf0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "974b802141bd4f13beffef2aa03f88f9" is_pre_election: true
I20260129 02:58:48.604260 20514 raft_consensus.cc:2468] T 926a250fa5834e88b96ccecbefbf5086 P 974b802141bd4f13beffef2aa03f88f9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 197b880fa39d432a9b472845876bfaf0 in term 0.
I20260129 02:58:48.607527 20590 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "926a250fa5834e88b96ccecbefbf5086" candidate_uuid: "197b880fa39d432a9b472845876bfaf0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c50de89a5a845668696dcb28c3badd6" is_pre_election: true
I20260129 02:58:48.608348 20633 ts_tablet_manager.cc:1434] T 8055232cf0004d40a85f3bf983fe00a9 P 197b880fa39d432a9b472845876bfaf0: Time spent starting tablet: real 0.030s user 0.006s sys 0.000s
W20260129 02:58:48.608806 20405 leader_election.cc:343] T 926a250fa5834e88b96ccecbefbf5086 P 197b880fa39d432a9b472845876bfaf0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3:37735): Illegal state: must be running to vote when last-logged opid is not known
I20260129 02:58:48.609781 20633 tablet_bootstrap.cc:492] T 75be4bd6bd45474795fd472eff2953ac P 197b880fa39d432a9b472845876bfaf0: Bootstrap starting.
I20260129 02:58:48.608253 20589 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c891d2bff4ce4568861bbc992e91fbb0" candidate_uuid: "197b880fa39d432a9b472845876bfaf0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c50de89a5a845668696dcb28c3badd6" is_pre_election: true
I20260129 02:58:48.609787 20407 leader_election.cc:304] T c891d2bff4ce4568861bbc992e91fbb0 P 197b880fa39d432a9b472845876bfaf0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 197b880fa39d432a9b472845876bfaf0, 974b802141bd4f13beffef2aa03f88f9; no voters:
I20260129 02:58:48.613838 20638 raft_consensus.cc:2804] T c891d2bff4ce4568861bbc992e91fbb0 P 197b880fa39d432a9b472845876bfaf0 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260129 02:58:48.614454 20638 raft_consensus.cc:493] T c891d2bff4ce4568861bbc992e91fbb0 P 197b880fa39d432a9b472845876bfaf0 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260129 02:58:48.615051 20638 raft_consensus.cc:3060] T c891d2bff4ce4568861bbc992e91fbb0 P 197b880fa39d432a9b472845876bfaf0 [term 0 FOLLOWER]: Advancing to term 1
W20260129 02:58:48.616511 20405 leader_election.cc:343] T c891d2bff4ce4568861bbc992e91fbb0 P 197b880fa39d432a9b472845876bfaf0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3:37735): Illegal state: must be running to vote when last-logged opid is not known
W20260129 02:58:48.618376 20480 consensus_peers.cc:597] T cb0b3f259ebc4310927338a8740e87b8 P 974b802141bd4f13beffef2aa03f88f9 -> Peer 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3:37735): Couldn't send request to peer 1c50de89a5a845668696dcb28c3badd6. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 1: this message will repeat every 5th retry.
I20260129 02:58:48.624630 20407 leader_election.cc:304] T 926a250fa5834e88b96ccecbefbf5086 P 197b880fa39d432a9b472845876bfaf0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 197b880fa39d432a9b472845876bfaf0, 974b802141bd4f13beffef2aa03f88f9; no voters: 1c50de89a5a845668696dcb28c3badd6
I20260129 02:58:48.625856 20828 raft_consensus.cc:2804] T 926a250fa5834e88b96ccecbefbf5086 P 197b880fa39d432a9b472845876bfaf0 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260129 02:58:48.626340 20828 raft_consensus.cc:493] T 926a250fa5834e88b96ccecbefbf5086 P 197b880fa39d432a9b472845876bfaf0 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260129 02:58:48.626744 20828 raft_consensus.cc:3060] T 926a250fa5834e88b96ccecbefbf5086 P 197b880fa39d432a9b472845876bfaf0 [term 0 FOLLOWER]: Advancing to term 1
I20260129 02:58:48.628049 20631 tablet_bootstrap.cc:492] T cb0b3f259ebc4310927338a8740e87b8 P 1c50de89a5a845668696dcb28c3badd6: No bootstrap required, opened a new log
I20260129 02:58:48.628525 20631 ts_tablet_manager.cc:1403] T cb0b3f259ebc4310927338a8740e87b8 P 1c50de89a5a845668696dcb28c3badd6: Time spent bootstrapping tablet: real 0.045s user 0.013s sys 0.003s
I20260129 02:58:48.635076 20631 raft_consensus.cc:359] T cb0b3f259ebc4310927338a8740e87b8 P 1c50de89a5a845668696dcb28c3badd6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:58:48.635675 20631 raft_consensus.cc:385] T cb0b3f259ebc4310927338a8740e87b8 P 1c50de89a5a845668696dcb28c3badd6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260129 02:58:48.635938 20631 raft_consensus.cc:740] T cb0b3f259ebc4310927338a8740e87b8 P 1c50de89a5a845668696dcb28c3badd6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1c50de89a5a845668696dcb28c3badd6, State: Initialized, Role: FOLLOWER
I20260129 02:58:48.636519 20631 consensus_queue.cc:260] T cb0b3f259ebc4310927338a8740e87b8 P 1c50de89a5a845668696dcb28c3badd6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:58:48.639125 20631 ts_tablet_manager.cc:1434] T cb0b3f259ebc4310927338a8740e87b8 P 1c50de89a5a845668696dcb28c3badd6: Time spent starting tablet: real 0.010s user 0.006s sys 0.000s
I20260129 02:58:48.640269 20638 raft_consensus.cc:515] T c891d2bff4ce4568861bbc992e91fbb0 P 197b880fa39d432a9b472845876bfaf0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:58:48.642338 20631 tablet_bootstrap.cc:492] T 926a250fa5834e88b96ccecbefbf5086 P 1c50de89a5a845668696dcb28c3badd6: Bootstrap starting.
I20260129 02:58:48.645539 20740 raft_consensus.cc:493] T c934d31efd3f4e5195605ddc354d14e1 P 974b802141bd4f13beffef2aa03f88f9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260129 02:58:48.646319 20633 tablet_bootstrap.cc:654] T 75be4bd6bd45474795fd472eff2953ac P 197b880fa39d432a9b472845876bfaf0: Neither blocks nor log segments found. Creating new log.
I20260129 02:58:48.649142 20338 catalog_manager.cc:5654] T 0678b8b468bd43cdbc3e485900844821 P 197b880fa39d432a9b472845876bfaf0 reported cstate change: term changed from 0 to 1, leader changed from <none> to 197b880fa39d432a9b472845876bfaf0 (127.18.211.1). New cstate: current_term: 1 leader_uuid: "197b880fa39d432a9b472845876bfaf0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } health_report { overall_health: UNKNOWN } } }
I20260129 02:58:48.651727 20638 leader_election.cc:290] T c891d2bff4ce4568861bbc992e91fbb0 P 197b880fa39d432a9b472845876bfaf0 [CANDIDATE]: Term 1 election: Requested vote from peers 974b802141bd4f13beffef2aa03f88f9 (127.18.211.2:34517), 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3:37735)
I20260129 02:58:48.646734 20740 raft_consensus.cc:515] T c934d31efd3f4e5195605ddc354d14e1 P 974b802141bd4f13beffef2aa03f88f9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:58:48.656266 20590 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c891d2bff4ce4568861bbc992e91fbb0" candidate_uuid: "197b880fa39d432a9b472845876bfaf0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c50de89a5a845668696dcb28c3badd6"
I20260129 02:58:48.660578 20740 leader_election.cc:290] T c934d31efd3f4e5195605ddc354d14e1 P 974b802141bd4f13beffef2aa03f88f9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 197b880fa39d432a9b472845876bfaf0 (127.18.211.1:42027), 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3:37735)
I20260129 02:58:48.663475 20590 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c934d31efd3f4e5195605ddc354d14e1" candidate_uuid: "974b802141bd4f13beffef2aa03f88f9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c50de89a5a845668696dcb28c3badd6" is_pre_election: true
I20260129 02:58:48.663947 20435 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c934d31efd3f4e5195605ddc354d14e1" candidate_uuid: "974b802141bd4f13beffef2aa03f88f9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "197b880fa39d432a9b472845876bfaf0" is_pre_election: true
W20260129 02:58:48.664379 20405 leader_election.cc:343] T c891d2bff4ce4568861bbc992e91fbb0 P 197b880fa39d432a9b472845876bfaf0 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3:37735): Illegal state: must be running to vote when last-logged opid is not known
W20260129 02:58:48.664824 20480 leader_election.cc:343] T c934d31efd3f4e5195605ddc354d14e1 P 974b802141bd4f13beffef2aa03f88f9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3:37735): Illegal state: must be running to vote when last-logged opid is not known
W20260129 02:58:48.667117 20479 leader_election.cc:343] T c934d31efd3f4e5195605ddc354d14e1 P 974b802141bd4f13beffef2aa03f88f9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 197b880fa39d432a9b472845876bfaf0 (127.18.211.1:42027): Illegal state: must be running to vote when last-logged opid is not known
I20260129 02:58:48.667512 20479 leader_election.cc:304] T c934d31efd3f4e5195605ddc354d14e1 P 974b802141bd4f13beffef2aa03f88f9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 974b802141bd4f13beffef2aa03f88f9; no voters: 197b880fa39d432a9b472845876bfaf0, 1c50de89a5a845668696dcb28c3badd6
I20260129 02:58:48.668504 20740 raft_consensus.cc:2749] T c934d31efd3f4e5195605ddc354d14e1 P 974b802141bd4f13beffef2aa03f88f9 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260129 02:58:48.671294 20630 tablet_bootstrap.cc:492] T 00e79d8f25a34676b1d866eb2068a2b2 P 974b802141bd4f13beffef2aa03f88f9: No bootstrap required, opened a new log
I20260129 02:58:48.671016 20828 raft_consensus.cc:515] T 926a250fa5834e88b96ccecbefbf5086 P 197b880fa39d432a9b472845876bfaf0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:58:48.671803 20630 ts_tablet_manager.cc:1403] T 00e79d8f25a34676b1d866eb2068a2b2 P 974b802141bd4f13beffef2aa03f88f9: Time spent bootstrapping tablet: real 0.091s user 0.030s sys 0.036s
I20260129 02:58:48.673230 20516 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c891d2bff4ce4568861bbc992e91fbb0" candidate_uuid: "197b880fa39d432a9b472845876bfaf0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "974b802141bd4f13beffef2aa03f88f9"
I20260129 02:58:48.674252 20516 raft_consensus.cc:3060] T c891d2bff4ce4568861bbc992e91fbb0 P 974b802141bd4f13beffef2aa03f88f9 [term 0 FOLLOWER]: Advancing to term 1
I20260129 02:58:48.674403 20630 raft_consensus.cc:359] T 00e79d8f25a34676b1d866eb2068a2b2 P 974b802141bd4f13beffef2aa03f88f9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:58:48.673875 20631 tablet_bootstrap.cc:654] T 926a250fa5834e88b96ccecbefbf5086 P 1c50de89a5a845668696dcb28c3badd6: Neither blocks nor log segments found. Creating new log.
I20260129 02:58:48.675058 20630 raft_consensus.cc:385] T 00e79d8f25a34676b1d866eb2068a2b2 P 974b802141bd4f13beffef2aa03f88f9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260129 02:58:48.675356 20630 raft_consensus.cc:740] T 00e79d8f25a34676b1d866eb2068a2b2 P 974b802141bd4f13beffef2aa03f88f9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 974b802141bd4f13beffef2aa03f88f9, State: Initialized, Role: FOLLOWER
I20260129 02:58:48.676013 20630 consensus_queue.cc:260] T 00e79d8f25a34676b1d866eb2068a2b2 P 974b802141bd4f13beffef2aa03f88f9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:58:48.678256 20630 ts_tablet_manager.cc:1434] T 00e79d8f25a34676b1d866eb2068a2b2 P 974b802141bd4f13beffef2aa03f88f9: Time spent starting tablet: real 0.006s user 0.004s sys 0.000s
I20260129 02:58:48.679440 20829 raft_consensus.cc:493] T cedce331e1584ac8b636c9b8a72b2ba0 P 1c50de89a5a845668696dcb28c3badd6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260129 02:58:48.679625 20630 tablet_bootstrap.cc:492] T 3ccac82a01a2445fa5b0ce6c0fcf2295 P 974b802141bd4f13beffef2aa03f88f9: Bootstrap starting.
I20260129 02:58:48.679936 20829 raft_consensus.cc:515] T cedce331e1584ac8b636c9b8a72b2ba0 P 1c50de89a5a845668696dcb28c3badd6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:58:48.682771 20829 leader_election.cc:290] T cedce331e1584ac8b636c9b8a72b2ba0 P 1c50de89a5a845668696dcb28c3badd6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 197b880fa39d432a9b472845876bfaf0 (127.18.211.1:42027), 974b802141bd4f13beffef2aa03f88f9 (127.18.211.2:34517)
I20260129 02:58:48.685513 20511 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "cedce331e1584ac8b636c9b8a72b2ba0" candidate_uuid: "1c50de89a5a845668696dcb28c3badd6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "974b802141bd4f13beffef2aa03f88f9" is_pre_election: true
I20260129 02:58:48.685804 20630 tablet_bootstrap.cc:654] T 3ccac82a01a2445fa5b0ce6c0fcf2295 P 974b802141bd4f13beffef2aa03f88f9: Neither blocks nor log segments found. Creating new log.
I20260129 02:58:48.689723 20633 tablet_bootstrap.cc:492] T 75be4bd6bd45474795fd472eff2953ac P 197b880fa39d432a9b472845876bfaf0: No bootstrap required, opened a new log
I20260129 02:58:48.690165 20633 ts_tablet_manager.cc:1403] T 75be4bd6bd45474795fd472eff2953ac P 197b880fa39d432a9b472845876bfaf0: Time spent bootstrapping tablet: real 0.081s user 0.018s sys 0.004s
I20260129 02:58:48.691272 20828 leader_election.cc:290] T 926a250fa5834e88b96ccecbefbf5086 P 197b880fa39d432a9b472845876bfaf0 [CANDIDATE]: Term 1 election: Requested vote from peers 974b802141bd4f13beffef2aa03f88f9 (127.18.211.2:34517), 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3:37735)
I20260129 02:58:48.692098 20435 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "cedce331e1584ac8b636c9b8a72b2ba0" candidate_uuid: "1c50de89a5a845668696dcb28c3badd6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "197b880fa39d432a9b472845876bfaf0" is_pre_election: true
I20260129 02:58:48.692754 20435 raft_consensus.cc:2393] T cedce331e1584ac8b636c9b8a72b2ba0 P 197b880fa39d432a9b472845876bfaf0 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1c50de89a5a845668696dcb28c3badd6 in current term 1: Already voted for candidate 974b802141bd4f13beffef2aa03f88f9 in this term.
I20260129 02:58:48.692679 20633 raft_consensus.cc:359] T 75be4bd6bd45474795fd472eff2953ac P 197b880fa39d432a9b472845876bfaf0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:58:48.693393 20633 raft_consensus.cc:385] T 75be4bd6bd45474795fd472eff2953ac P 197b880fa39d432a9b472845876bfaf0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260129 02:58:48.693648 20633 raft_consensus.cc:740] T 75be4bd6bd45474795fd472eff2953ac P 197b880fa39d432a9b472845876bfaf0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 197b880fa39d432a9b472845876bfaf0, State: Initialized, Role: FOLLOWER
I20260129 02:58:48.694231 20633 consensus_queue.cc:260] T 75be4bd6bd45474795fd472eff2953ac P 197b880fa39d432a9b472845876bfaf0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:58:48.696944 20553 leader_election.cc:304] T cedce331e1584ac8b636c9b8a72b2ba0 P 1c50de89a5a845668696dcb28c3badd6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1c50de89a5a845668696dcb28c3badd6; no voters: 197b880fa39d432a9b472845876bfaf0, 974b802141bd4f13beffef2aa03f88f9
I20260129 02:58:48.697891 20633 ts_tablet_manager.cc:1434] T 75be4bd6bd45474795fd472eff2953ac P 197b880fa39d432a9b472845876bfaf0: Time spent starting tablet: real 0.007s user 0.005s sys 0.001s
I20260129 02:58:48.697808 20590 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "926a250fa5834e88b96ccecbefbf5086" candidate_uuid: "197b880fa39d432a9b472845876bfaf0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c50de89a5a845668696dcb28c3badd6"
I20260129 02:58:48.698081 20881 raft_consensus.cc:3060] T cedce331e1584ac8b636c9b8a72b2ba0 P 1c50de89a5a845668696dcb28c3badd6 [term 0 FOLLOWER]: Advancing to term 1
I20260129 02:58:48.698875 20633 tablet_bootstrap.cc:492] T c934d31efd3f4e5195605ddc354d14e1 P 197b880fa39d432a9b472845876bfaf0: Bootstrap starting.
W20260129 02:58:48.699573 20405 leader_election.cc:343] T 926a250fa5834e88b96ccecbefbf5086 P 197b880fa39d432a9b472845876bfaf0 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3:37735): Illegal state: must be running to vote when last-logged opid is not known
I20260129 02:58:48.702633 20511 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "926a250fa5834e88b96ccecbefbf5086" candidate_uuid: "197b880fa39d432a9b472845876bfaf0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "974b802141bd4f13beffef2aa03f88f9"
I20260129 02:58:48.703276 20511 raft_consensus.cc:3060] T 926a250fa5834e88b96ccecbefbf5086 P 974b802141bd4f13beffef2aa03f88f9 [term 0 FOLLOWER]: Advancing to term 1
I20260129 02:58:48.705536 20516 raft_consensus.cc:2468] T c891d2bff4ce4568861bbc992e91fbb0 P 974b802141bd4f13beffef2aa03f88f9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 197b880fa39d432a9b472845876bfaf0 in term 1.
I20260129 02:58:48.706305 20630 tablet_bootstrap.cc:492] T 3ccac82a01a2445fa5b0ce6c0fcf2295 P 974b802141bd4f13beffef2aa03f88f9: No bootstrap required, opened a new log
I20260129 02:58:48.706755 20630 ts_tablet_manager.cc:1403] T 3ccac82a01a2445fa5b0ce6c0fcf2295 P 974b802141bd4f13beffef2aa03f88f9: Time spent bootstrapping tablet: real 0.027s user 0.006s sys 0.003s
I20260129 02:58:48.707041 20407 leader_election.cc:304] T c891d2bff4ce4568861bbc992e91fbb0 P 197b880fa39d432a9b472845876bfaf0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 197b880fa39d432a9b472845876bfaf0, 974b802141bd4f13beffef2aa03f88f9; no voters: 1c50de89a5a845668696dcb28c3badd6
I20260129 02:58:48.707849 20828 raft_consensus.cc:2804] T c891d2bff4ce4568861bbc992e91fbb0 P 197b880fa39d432a9b472845876bfaf0 [term 1 FOLLOWER]: Leader election won for term 1
I20260129 02:58:48.708321 20828 raft_consensus.cc:697] T c891d2bff4ce4568861bbc992e91fbb0 P 197b880fa39d432a9b472845876bfaf0 [term 1 LEADER]: Becoming Leader. State: Replica: 197b880fa39d432a9b472845876bfaf0, State: Running, Role: LEADER
I20260129 02:58:48.709110 20828 consensus_queue.cc:237] T c891d2bff4ce4568861bbc992e91fbb0 P 197b880fa39d432a9b472845876bfaf0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:58:48.704617 20631 tablet_bootstrap.cc:492] T 926a250fa5834e88b96ccecbefbf5086 P 1c50de89a5a845668696dcb28c3badd6: No bootstrap required, opened a new log
I20260129 02:58:48.710283 20631 ts_tablet_manager.cc:1403] T 926a250fa5834e88b96ccecbefbf5086 P 1c50de89a5a845668696dcb28c3badd6: Time spent bootstrapping tablet: real 0.068s user 0.008s sys 0.003s
I20260129 02:58:48.711522 20633 tablet_bootstrap.cc:654] T c934d31efd3f4e5195605ddc354d14e1 P 197b880fa39d432a9b472845876bfaf0: Neither blocks nor log segments found. Creating new log.
I20260129 02:58:48.712747 20631 raft_consensus.cc:359] T 926a250fa5834e88b96ccecbefbf5086 P 1c50de89a5a845668696dcb28c3badd6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:58:48.712965 20740 consensus_queue.cc:1048] T cedce331e1584ac8b636c9b8a72b2ba0 P 974b802141bd4f13beffef2aa03f88f9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260129 02:58:48.713420 20631 raft_consensus.cc:385] T 926a250fa5834e88b96ccecbefbf5086 P 1c50de89a5a845668696dcb28c3badd6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260129 02:58:48.713717 20631 raft_consensus.cc:740] T 926a250fa5834e88b96ccecbefbf5086 P 1c50de89a5a845668696dcb28c3badd6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1c50de89a5a845668696dcb28c3badd6, State: Initialized, Role: FOLLOWER
I20260129 02:58:48.709390 20630 raft_consensus.cc:359] T 3ccac82a01a2445fa5b0ce6c0fcf2295 P 974b802141bd4f13beffef2aa03f88f9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:58:48.714551 20630 raft_consensus.cc:385] T 3ccac82a01a2445fa5b0ce6c0fcf2295 P 974b802141bd4f13beffef2aa03f88f9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260129 02:58:48.714483 20631 consensus_queue.cc:260] T 926a250fa5834e88b96ccecbefbf5086 P 1c50de89a5a845668696dcb28c3badd6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:58:48.715000 20630 raft_consensus.cc:740] T 3ccac82a01a2445fa5b0ce6c0fcf2295 P 974b802141bd4f13beffef2aa03f88f9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 974b802141bd4f13beffef2aa03f88f9, State: Initialized, Role: FOLLOWER
I20260129 02:58:48.715664 20630 consensus_queue.cc:260] T 3ccac82a01a2445fa5b0ce6c0fcf2295 P 974b802141bd4f13beffef2aa03f88f9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:58:48.716892 20631 ts_tablet_manager.cc:1434] T 926a250fa5834e88b96ccecbefbf5086 P 1c50de89a5a845668696dcb28c3badd6: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260129 02:58:48.717938 20630 ts_tablet_manager.cc:1434] T 3ccac82a01a2445fa5b0ce6c0fcf2295 P 974b802141bd4f13beffef2aa03f88f9: Time spent starting tablet: real 0.011s user 0.007s sys 0.000s
I20260129 02:58:48.718814 20630 tablet_bootstrap.cc:492] T 75be4bd6bd45474795fd472eff2953ac P 974b802141bd4f13beffef2aa03f88f9: Bootstrap starting.
I20260129 02:58:48.725883 20630 tablet_bootstrap.cc:654] T 75be4bd6bd45474795fd472eff2953ac P 974b802141bd4f13beffef2aa03f88f9: Neither blocks nor log segments found. Creating new log.
I20260129 02:58:48.726478 20633 tablet_bootstrap.cc:492] T c934d31efd3f4e5195605ddc354d14e1 P 197b880fa39d432a9b472845876bfaf0: No bootstrap required, opened a new log
I20260129 02:58:48.728189 20633 ts_tablet_manager.cc:1403] T c934d31efd3f4e5195605ddc354d14e1 P 197b880fa39d432a9b472845876bfaf0: Time spent bootstrapping tablet: real 0.030s user 0.011s sys 0.000s
I20260129 02:58:48.729324 20740 raft_consensus.cc:493] T 3ccac82a01a2445fa5b0ce6c0fcf2295 P 974b802141bd4f13beffef2aa03f88f9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260129 02:58:48.730002 20740 raft_consensus.cc:515] T 3ccac82a01a2445fa5b0ce6c0fcf2295 P 974b802141bd4f13beffef2aa03f88f9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:58:48.730679 20633 raft_consensus.cc:359] T c934d31efd3f4e5195605ddc354d14e1 P 197b880fa39d432a9b472845876bfaf0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:58:48.731364 20633 raft_consensus.cc:385] T c934d31efd3f4e5195605ddc354d14e1 P 197b880fa39d432a9b472845876bfaf0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260129 02:58:48.731626 20633 raft_consensus.cc:740] T c934d31efd3f4e5195605ddc354d14e1 P 197b880fa39d432a9b472845876bfaf0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 197b880fa39d432a9b472845876bfaf0, State: Initialized, Role: FOLLOWER
I20260129 02:58:48.732223 20633 consensus_queue.cc:260] T c934d31efd3f4e5195605ddc354d14e1 P 197b880fa39d432a9b472845876bfaf0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:58:48.733959 20881 raft_consensus.cc:2749] T cedce331e1584ac8b636c9b8a72b2ba0 P 1c50de89a5a845668696dcb28c3badd6 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260129 02:58:48.741792 20511 raft_consensus.cc:2468] T 926a250fa5834e88b96ccecbefbf5086 P 974b802141bd4f13beffef2aa03f88f9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 197b880fa39d432a9b472845876bfaf0 in term 1.
I20260129 02:58:48.743232 20407 leader_election.cc:304] T 926a250fa5834e88b96ccecbefbf5086 P 197b880fa39d432a9b472845876bfaf0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 197b880fa39d432a9b472845876bfaf0, 974b802141bd4f13beffef2aa03f88f9; no voters: 1c50de89a5a845668696dcb28c3badd6
I20260129 02:58:48.734802 20633 ts_tablet_manager.cc:1434] T c934d31efd3f4e5195605ddc354d14e1 P 197b880fa39d432a9b472845876bfaf0: Time spent starting tablet: real 0.006s user 0.003s sys 0.002s
I20260129 02:58:48.735564 20740 leader_election.cc:290] T 3ccac82a01a2445fa5b0ce6c0fcf2295 P 974b802141bd4f13beffef2aa03f88f9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 197b880fa39d432a9b472845876bfaf0 (127.18.211.1:42027), 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3:37735)
I20260129 02:58:48.736563 20438 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3ccac82a01a2445fa5b0ce6c0fcf2295" candidate_uuid: "974b802141bd4f13beffef2aa03f88f9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "197b880fa39d432a9b472845876bfaf0" is_pre_election: true
I20260129 02:58:48.744287 20828 raft_consensus.cc:2804] T 926a250fa5834e88b96ccecbefbf5086 P 197b880fa39d432a9b472845876bfaf0 [term 1 FOLLOWER]: Leader election won for term 1
I20260129 02:58:48.744771 20828 raft_consensus.cc:697] T 926a250fa5834e88b96ccecbefbf5086 P 197b880fa39d432a9b472845876bfaf0 [term 1 LEADER]: Becoming Leader. State: Replica: 197b880fa39d432a9b472845876bfaf0, State: Running, Role: LEADER
I20260129 02:58:48.745635 20828 consensus_queue.cc:237] T 926a250fa5834e88b96ccecbefbf5086 P 197b880fa39d432a9b472845876bfaf0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:58:48.750377 20590 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3ccac82a01a2445fa5b0ce6c0fcf2295" candidate_uuid: "974b802141bd4f13beffef2aa03f88f9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c50de89a5a845668696dcb28c3badd6" is_pre_election: true
W20260129 02:58:48.752727 20480 leader_election.cc:343] T 3ccac82a01a2445fa5b0ce6c0fcf2295 P 974b802141bd4f13beffef2aa03f88f9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3:37735): Illegal state: must be running to vote when last-logged opid is not known
W20260129 02:58:48.754513 20479 leader_election.cc:343] T 3ccac82a01a2445fa5b0ce6c0fcf2295 P 974b802141bd4f13beffef2aa03f88f9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 197b880fa39d432a9b472845876bfaf0 (127.18.211.1:42027): Illegal state: must be running to vote when last-logged opid is not known
I20260129 02:58:48.755066 20479 leader_election.cc:304] T 3ccac82a01a2445fa5b0ce6c0fcf2295 P 974b802141bd4f13beffef2aa03f88f9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 974b802141bd4f13beffef2aa03f88f9; no voters: 197b880fa39d432a9b472845876bfaf0, 1c50de89a5a845668696dcb28c3badd6
I20260129 02:58:48.756752 20740 raft_consensus.cc:2749] T 3ccac82a01a2445fa5b0ce6c0fcf2295 P 974b802141bd4f13beffef2aa03f88f9 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260129 02:58:48.762820 20633 tablet_bootstrap.cc:492] T 4f5a77e94fef4b16a2ba7b067c8b144a P 197b880fa39d432a9b472845876bfaf0: Bootstrap starting.
I20260129 02:58:48.763991 20631 tablet_bootstrap.cc:492] T 0340229d521b432580ae926f6062e3f7 P 1c50de89a5a845668696dcb28c3badd6: Bootstrap starting.
I20260129 02:58:48.768803 20869 consensus_queue.cc:1048] T cedce331e1584ac8b636c9b8a72b2ba0 P 974b802141bd4f13beffef2aa03f88f9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260129 02:58:48.784484 20338 catalog_manager.cc:5654] T c891d2bff4ce4568861bbc992e91fbb0 P 197b880fa39d432a9b472845876bfaf0 reported cstate change: term changed from 0 to 1, leader changed from <none> to 197b880fa39d432a9b472845876bfaf0 (127.18.211.1). New cstate: current_term: 1 leader_uuid: "197b880fa39d432a9b472845876bfaf0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } health_report { overall_health: UNKNOWN } } }
I20260129 02:58:48.809535 20740 raft_consensus.cc:493] T 00e79d8f25a34676b1d866eb2068a2b2 P 974b802141bd4f13beffef2aa03f88f9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260129 02:58:48.829638 20740 raft_consensus.cc:515] T 00e79d8f25a34676b1d866eb2068a2b2 P 974b802141bd4f13beffef2aa03f88f9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:58:48.812448 20721 raft_consensus.cc:493] T 7d9494f5584d4d5db8b9adf1c8fc888b P 197b880fa39d432a9b472845876bfaf0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260129 02:58:48.832154 20721 raft_consensus.cc:515] T 7d9494f5584d4d5db8b9adf1c8fc888b P 197b880fa39d432a9b472845876bfaf0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:58:48.820367 20739 raft_consensus.cc:493] T a644c02b16af47679824ccd060ac10fd P 974b802141bd4f13beffef2aa03f88f9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260129 02:58:48.834697 20739 raft_consensus.cc:515] T a644c02b16af47679824ccd060ac10fd P 974b802141bd4f13beffef2aa03f88f9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:58:48.837407 20511 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7d9494f5584d4d5db8b9adf1c8fc888b" candidate_uuid: "197b880fa39d432a9b472845876bfaf0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "974b802141bd4f13beffef2aa03f88f9" is_pre_election: true
W20260129 02:58:48.838940 20407 leader_election.cc:343] T 7d9494f5584d4d5db8b9adf1c8fc888b P 197b880fa39d432a9b472845876bfaf0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 974b802141bd4f13beffef2aa03f88f9 (127.18.211.2:34517): Illegal state: must be running to vote when last-logged opid is not known
I20260129 02:58:48.864347 20721 leader_election.cc:290] T 7d9494f5584d4d5db8b9adf1c8fc888b P 197b880fa39d432a9b472845876bfaf0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 974b802141bd4f13beffef2aa03f88f9 (127.18.211.2:34517), 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3:37735)
I20260129 02:58:48.873549 20590 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7d9494f5584d4d5db8b9adf1c8fc888b" candidate_uuid: "197b880fa39d432a9b472845876bfaf0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c50de89a5a845668696dcb28c3badd6" is_pre_election: true
I20260129 02:58:48.874181 20881 consensus_queue.cc:1048] T a644c02b16af47679824ccd060ac10fd P 1c50de89a5a845668696dcb28c3badd6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.006s
I20260129 02:58:48.883617 20633 tablet_bootstrap.cc:654] T 4f5a77e94fef4b16a2ba7b067c8b144a P 197b880fa39d432a9b472845876bfaf0: Neither blocks nor log segments found. Creating new log.
W20260129 02:58:48.886790 20405 leader_election.cc:343] T 7d9494f5584d4d5db8b9adf1c8fc888b P 197b880fa39d432a9b472845876bfaf0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3:37735): Illegal state: must be running to vote when last-logged opid is not known
I20260129 02:58:48.889052 20405 leader_election.cc:304] T 7d9494f5584d4d5db8b9adf1c8fc888b P 197b880fa39d432a9b472845876bfaf0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 197b880fa39d432a9b472845876bfaf0; no voters: 1c50de89a5a845668696dcb28c3badd6, 974b802141bd4f13beffef2aa03f88f9
I20260129 02:58:48.890538 20828 raft_consensus.cc:2749] T 7d9494f5584d4d5db8b9adf1c8fc888b P 197b880fa39d432a9b472845876bfaf0 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260129 02:58:48.891083 20828 consensus_queue.cc:1048] T 18f30ebaa4174fd28c958e72d0cc7e24 P 197b880fa39d432a9b472845876bfaf0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260129 02:58:48.892709 20829 raft_consensus.cc:493] T 926a250fa5834e88b96ccecbefbf5086 P 1c50de89a5a845668696dcb28c3badd6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260129 02:58:48.893174 20829 raft_consensus.cc:515] T 926a250fa5834e88b96ccecbefbf5086 P 1c50de89a5a845668696dcb28c3badd6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:58:48.895046 20829 leader_election.cc:290] T 926a250fa5834e88b96ccecbefbf5086 P 1c50de89a5a845668696dcb28c3badd6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 197b880fa39d432a9b472845876bfaf0 (127.18.211.1:42027), 974b802141bd4f13beffef2aa03f88f9 (127.18.211.2:34517)
I20260129 02:58:48.896107 20739 leader_election.cc:290] T a644c02b16af47679824ccd060ac10fd P 974b802141bd4f13beffef2aa03f88f9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 197b880fa39d432a9b472845876bfaf0 (127.18.211.1:42027), 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3:37735)
W20260129 02:58:48.896733 20387 auto_rebalancer.cc:229] Could not retrieve cluster info: Service unavailable: Tablet not running
I20260129 02:58:48.897125 20588 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a644c02b16af47679824ccd060ac10fd" candidate_uuid: "974b802141bd4f13beffef2aa03f88f9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c50de89a5a845668696dcb28c3badd6" is_pre_election: true
I20260129 02:58:48.898918 20740 leader_election.cc:290] T 00e79d8f25a34676b1d866eb2068a2b2 P 974b802141bd4f13beffef2aa03f88f9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 197b880fa39d432a9b472845876bfaf0 (127.18.211.1:42027), 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3:37735)
W20260129 02:58:48.899818 20479 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.211.1:42027, user_credentials={real_user=slave}} blocked reactor thread for 78734.3us
I20260129 02:58:48.905639 20441 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "926a250fa5834e88b96ccecbefbf5086" candidate_uuid: "1c50de89a5a845668696dcb28c3badd6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "197b880fa39d432a9b472845876bfaf0" is_pre_election: true
I20260129 02:58:48.907447 20338 catalog_manager.cc:5654] T 926a250fa5834e88b96ccecbefbf5086 P 197b880fa39d432a9b472845876bfaf0 reported cstate change: term changed from 0 to 1, leader changed from <none> to 197b880fa39d432a9b472845876bfaf0 (127.18.211.1). New cstate: current_term: 1 leader_uuid: "197b880fa39d432a9b472845876bfaf0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } health_report { overall_health: UNKNOWN } } }
I20260129 02:58:48.910257 20441 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00e79d8f25a34676b1d866eb2068a2b2" candidate_uuid: "974b802141bd4f13beffef2aa03f88f9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "197b880fa39d432a9b472845876bfaf0" is_pre_election: true
I20260129 02:58:48.912509 20441 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a644c02b16af47679824ccd060ac10fd" candidate_uuid: "974b802141bd4f13beffef2aa03f88f9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "197b880fa39d432a9b472845876bfaf0" is_pre_election: true
I20260129 02:58:48.913252 20441 raft_consensus.cc:2393] T a644c02b16af47679824ccd060ac10fd P 197b880fa39d432a9b472845876bfaf0 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 974b802141bd4f13beffef2aa03f88f9 in current term 1: Already voted for candidate 1c50de89a5a845668696dcb28c3badd6 in this term.
W20260129 02:58:48.914546 20479 leader_election.cc:343] T 00e79d8f25a34676b1d866eb2068a2b2 P 974b802141bd4f13beffef2aa03f88f9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 197b880fa39d432a9b472845876bfaf0 (127.18.211.1:42027): Illegal state: must be running to vote when last-logged opid is not known
I20260129 02:58:48.915405 20479 leader_election.cc:304] T a644c02b16af47679824ccd060ac10fd P 974b802141bd4f13beffef2aa03f88f9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 974b802141bd4f13beffef2aa03f88f9; no voters: 197b880fa39d432a9b472845876bfaf0, 1c50de89a5a845668696dcb28c3badd6
I20260129 02:58:48.916870 20963 raft_consensus.cc:3060] T a644c02b16af47679824ccd060ac10fd P 974b802141bd4f13beffef2aa03f88f9 [term 0 FOLLOWER]: Advancing to term 1
I20260129 02:58:48.920167 20630 tablet_bootstrap.cc:492] T 75be4bd6bd45474795fd472eff2953ac P 974b802141bd4f13beffef2aa03f88f9: No bootstrap required, opened a new log
I20260129 02:58:48.920652 20630 ts_tablet_manager.cc:1403] T 75be4bd6bd45474795fd472eff2953ac P 974b802141bd4f13beffef2aa03f88f9: Time spent bootstrapping tablet: real 0.202s user 0.039s sys 0.039s
I20260129 02:58:48.920812 20638 consensus_queue.cc:1048] T 42d27a28d89b409e8807ec0efd97af69 P 197b880fa39d432a9b472845876bfaf0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260129 02:58:48.924000 20630 raft_consensus.cc:359] T 75be4bd6bd45474795fd472eff2953ac P 974b802141bd4f13beffef2aa03f88f9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:58:48.924697 20630 raft_consensus.cc:385] T 75be4bd6bd45474795fd472eff2953ac P 974b802141bd4f13beffef2aa03f88f9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260129 02:58:48.924971 20630 raft_consensus.cc:740] T 75be4bd6bd45474795fd472eff2953ac P 974b802141bd4f13beffef2aa03f88f9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 974b802141bd4f13beffef2aa03f88f9, State: Initialized, Role: FOLLOWER
I20260129 02:58:48.926265 20630 consensus_queue.cc:260] T 75be4bd6bd45474795fd472eff2953ac P 974b802141bd4f13beffef2aa03f88f9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:58:48.928514 20630 ts_tablet_manager.cc:1434] T 75be4bd6bd45474795fd472eff2953ac P 974b802141bd4f13beffef2aa03f88f9: Time spent starting tablet: real 0.008s user 0.007s sys 0.000s
I20260129 02:58:48.931735 20963 raft_consensus.cc:2749] T a644c02b16af47679824ccd060ac10fd P 974b802141bd4f13beffef2aa03f88f9 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260129 02:58:48.934190 20630 tablet_bootstrap.cc:492] T 4e3805ffa70242e8ade66da260dddf66 P 974b802141bd4f13beffef2aa03f88f9: Bootstrap starting.
I20260129 02:58:48.934556 20631 tablet_bootstrap.cc:654] T 0340229d521b432580ae926f6062e3f7 P 1c50de89a5a845668696dcb28c3badd6: Neither blocks nor log segments found. Creating new log.
I20260129 02:58:48.938575 20739 consensus_queue.cc:1048] T cb0b3f259ebc4310927338a8740e87b8 P 974b802141bd4f13beffef2aa03f88f9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260129 02:58:48.941700 20588 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00e79d8f25a34676b1d866eb2068a2b2" candidate_uuid: "974b802141bd4f13beffef2aa03f88f9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c50de89a5a845668696dcb28c3badd6" is_pre_election: true
I20260129 02:58:48.960201 20638 raft_consensus.cc:493] T c934d31efd3f4e5195605ddc354d14e1 P 197b880fa39d432a9b472845876bfaf0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260129 02:58:48.960839 20638 raft_consensus.cc:515] T c934d31efd3f4e5195605ddc354d14e1 P 197b880fa39d432a9b472845876bfaf0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:58:48.963213 20638 leader_election.cc:290] T c934d31efd3f4e5195605ddc354d14e1 P 197b880fa39d432a9b472845876bfaf0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 974b802141bd4f13beffef2aa03f88f9 (127.18.211.2:34517), 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3:37735)
I20260129 02:58:48.964905 20516 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c934d31efd3f4e5195605ddc354d14e1" candidate_uuid: "197b880fa39d432a9b472845876bfaf0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "974b802141bd4f13beffef2aa03f88f9" is_pre_election: true
I20260129 02:58:48.965531 20516 raft_consensus.cc:2468] T c934d31efd3f4e5195605ddc354d14e1 P 974b802141bd4f13beffef2aa03f88f9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 197b880fa39d432a9b472845876bfaf0 in term 0.
I20260129 02:58:48.967567 20407 leader_election.cc:304] T c934d31efd3f4e5195605ddc354d14e1 P 197b880fa39d432a9b472845876bfaf0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 197b880fa39d432a9b472845876bfaf0, 974b802141bd4f13beffef2aa03f88f9; no voters:
I20260129 02:58:48.968490 20638 raft_consensus.cc:2804] T c934d31efd3f4e5195605ddc354d14e1 P 197b880fa39d432a9b472845876bfaf0 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260129 02:58:48.968811 20638 raft_consensus.cc:493] T c934d31efd3f4e5195605ddc354d14e1 P 197b880fa39d432a9b472845876bfaf0 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260129 02:58:48.969087 20638 raft_consensus.cc:3060] T c934d31efd3f4e5195605ddc354d14e1 P 197b880fa39d432a9b472845876bfaf0 [term 0 FOLLOWER]: Advancing to term 1
I20260129 02:58:48.969594 20630 tablet_bootstrap.cc:654] T 4e3805ffa70242e8ade66da260dddf66 P 974b802141bd4f13beffef2aa03f88f9: Neither blocks nor log segments found. Creating new log.
I20260129 02:58:48.955714 20828 consensus_queue.cc:1048] T 18f30ebaa4174fd28c958e72d0cc7e24 P 197b880fa39d432a9b472845876bfaf0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.018s
W20260129 02:58:48.975804 20480 leader_election.cc:343] T 00e79d8f25a34676b1d866eb2068a2b2 P 974b802141bd4f13beffef2aa03f88f9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3:37735): Illegal state: must be running to vote when last-logged opid is not known
I20260129 02:58:48.978574 20638 raft_consensus.cc:515] T c934d31efd3f4e5195605ddc354d14e1 P 197b880fa39d432a9b472845876bfaf0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:58:48.981405 20638 leader_election.cc:290] T c934d31efd3f4e5195605ddc354d14e1 P 197b880fa39d432a9b472845876bfaf0 [CANDIDATE]: Term 1 election: Requested vote from peers 974b802141bd4f13beffef2aa03f88f9 (127.18.211.2:34517), 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3:37735)
I20260129 02:58:48.998852 20513 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "926a250fa5834e88b96ccecbefbf5086" candidate_uuid: "1c50de89a5a845668696dcb28c3badd6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "974b802141bd4f13beffef2aa03f88f9" is_pre_election: true
I20260129 02:58:49.002599 20965 consensus_queue.cc:1048] T a0fcc58d6fb6473cafbd9d34d5195269 P 974b802141bd4f13beffef2aa03f88f9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260129 02:58:48.976339 20480 leader_election.cc:304] T 00e79d8f25a34676b1d866eb2068a2b2 P 974b802141bd4f13beffef2aa03f88f9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 974b802141bd4f13beffef2aa03f88f9; no voters: 197b880fa39d432a9b472845876bfaf0, 1c50de89a5a845668696dcb28c3badd6
I20260129 02:58:49.007040 20638 consensus_queue.cc:1048] T 0678b8b468bd43cdbc3e485900844821 P 197b880fa39d432a9b472845876bfaf0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260129 02:58:49.007392 20633 tablet_bootstrap.cc:492] T 4f5a77e94fef4b16a2ba7b067c8b144a P 197b880fa39d432a9b472845876bfaf0: No bootstrap required, opened a new log
I20260129 02:58:49.007930 20633 ts_tablet_manager.cc:1403] T 4f5a77e94fef4b16a2ba7b067c8b144a P 197b880fa39d432a9b472845876bfaf0: Time spent bootstrapping tablet: real 0.245s user 0.041s sys 0.010s
I20260129 02:58:49.006690 20513 raft_consensus.cc:2393] T 926a250fa5834e88b96ccecbefbf5086 P 974b802141bd4f13beffef2aa03f88f9 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1c50de89a5a845668696dcb28c3badd6 in current term 1: Already voted for candidate 197b880fa39d432a9b472845876bfaf0 in this term.
W20260129 02:58:49.038897 20556 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.211.2:34517, user_credentials={real_user=slave}} blocked reactor thread for 53077us
I20260129 02:58:48.991758 20881 consensus_queue.cc:1048] T a644c02b16af47679824ccd060ac10fd P 1c50de89a5a845668696dcb28c3badd6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.031s
I20260129 02:58:49.045112 20515 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c934d31efd3f4e5195605ddc354d14e1" candidate_uuid: "197b880fa39d432a9b472845876bfaf0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "974b802141bd4f13beffef2aa03f88f9"
I20260129 02:58:49.045960 20515 raft_consensus.cc:3060] T c934d31efd3f4e5195605ddc354d14e1 P 974b802141bd4f13beffef2aa03f88f9 [term 0 FOLLOWER]: Advancing to term 1
I20260129 02:58:49.021682 20740 raft_consensus.cc:2749] T 00e79d8f25a34676b1d866eb2068a2b2 P 974b802141bd4f13beffef2aa03f88f9 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260129 02:58:49.055557 20638 raft_consensus.cc:493] T 8055232cf0004d40a85f3bf983fe00a9 P 197b880fa39d432a9b472845876bfaf0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260129 02:58:49.056068 20638 raft_consensus.cc:515] T 8055232cf0004d40a85f3bf983fe00a9 P 197b880fa39d432a9b472845876bfaf0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:58:49.058560 20638 leader_election.cc:290] T 8055232cf0004d40a85f3bf983fe00a9 P 197b880fa39d432a9b472845876bfaf0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 974b802141bd4f13beffef2aa03f88f9 (127.18.211.2:34517), 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3:37735)
W20260129 02:58:49.052482 20553 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.211.1:42027, user_credentials={real_user=slave}} blocked reactor thread for 48184.9us
I20260129 02:58:49.060879 20633 raft_consensus.cc:359] T 4f5a77e94fef4b16a2ba7b067c8b144a P 197b880fa39d432a9b472845876bfaf0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:58:49.070744 20633 raft_consensus.cc:385] T 4f5a77e94fef4b16a2ba7b067c8b144a P 197b880fa39d432a9b472845876bfaf0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260129 02:58:49.071197 20633 raft_consensus.cc:740] T 4f5a77e94fef4b16a2ba7b067c8b144a P 197b880fa39d432a9b472845876bfaf0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 197b880fa39d432a9b472845876bfaf0, State: Initialized, Role: FOLLOWER
I20260129 02:58:49.083326 20556 leader_election.cc:304] T 926a250fa5834e88b96ccecbefbf5086 P 1c50de89a5a845668696dcb28c3badd6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1c50de89a5a845668696dcb28c3badd6; no voters: 197b880fa39d432a9b472845876bfaf0, 974b802141bd4f13beffef2aa03f88f9
I20260129 02:58:49.101408 20590 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c934d31efd3f4e5195605ddc354d14e1" candidate_uuid: "197b880fa39d432a9b472845876bfaf0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c50de89a5a845668696dcb28c3badd6" is_pre_election: true
I20260129 02:58:49.105739 20638 raft_consensus.cc:493] T 75be4bd6bd45474795fd472eff2953ac P 197b880fa39d432a9b472845876bfaf0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260129 02:58:49.106227 20638 raft_consensus.cc:515] T 75be4bd6bd45474795fd472eff2953ac P 197b880fa39d432a9b472845876bfaf0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:58:49.107372 20633 consensus_queue.cc:260] T 4f5a77e94fef4b16a2ba7b067c8b144a P 197b880fa39d432a9b472845876bfaf0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:58:49.114568 20638 leader_election.cc:290] T 75be4bd6bd45474795fd472eff2953ac P 197b880fa39d432a9b472845876bfaf0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 974b802141bd4f13beffef2aa03f88f9 (127.18.211.2:34517), 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3:37735)
I20260129 02:58:49.134200 20978 raft_consensus.cc:493] T cb0b3f259ebc4310927338a8740e87b8 P 1c50de89a5a845668696dcb28c3badd6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260129 02:58:49.134727 20978 raft_consensus.cc:515] T cb0b3f259ebc4310927338a8740e87b8 P 1c50de89a5a845668696dcb28c3badd6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:58:49.134905 20590 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c934d31efd3f4e5195605ddc354d14e1" candidate_uuid: "197b880fa39d432a9b472845876bfaf0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c50de89a5a845668696dcb28c3badd6"
I20260129 02:58:49.136765 20978 leader_election.cc:290] T cb0b3f259ebc4310927338a8740e87b8 P 1c50de89a5a845668696dcb28c3badd6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 197b880fa39d432a9b472845876bfaf0 (127.18.211.1:42027), 974b802141bd4f13beffef2aa03f88f9 (127.18.211.2:34517)
I20260129 02:58:49.109143 20829 raft_consensus.cc:3060] T 926a250fa5834e88b96ccecbefbf5086 P 1c50de89a5a845668696dcb28c3badd6 [term 0 FOLLOWER]: Advancing to term 1
I20260129 02:58:49.146076 20515 raft_consensus.cc:2468] T c934d31efd3f4e5195605ddc354d14e1 P 974b802141bd4f13beffef2aa03f88f9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 197b880fa39d432a9b472845876bfaf0 in term 1.
I20260129 02:58:49.150684 20633 ts_tablet_manager.cc:1434] T 4f5a77e94fef4b16a2ba7b067c8b144a P 197b880fa39d432a9b472845876bfaf0: Time spent starting tablet: real 0.142s user 0.010s sys 0.003s
I20260129 02:58:49.168864 20407 leader_election.cc:304] T c934d31efd3f4e5195605ddc354d14e1 P 197b880fa39d432a9b472845876bfaf0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 197b880fa39d432a9b472845876bfaf0, 974b802141bd4f13beffef2aa03f88f9; no voters:
I20260129 02:58:49.170809 20658 raft_consensus.cc:2804] T c934d31efd3f4e5195605ddc354d14e1 P 197b880fa39d432a9b472845876bfaf0 [term 1 FOLLOWER]: Leader election won for term 1
I20260129 02:58:49.171401 20658 raft_consensus.cc:697] T c934d31efd3f4e5195605ddc354d14e1 P 197b880fa39d432a9b472845876bfaf0 [term 1 LEADER]: Becoming Leader. State: Replica: 197b880fa39d432a9b472845876bfaf0, State: Running, Role: LEADER
I20260129 02:58:49.172360 20658 consensus_queue.cc:237] T c934d31efd3f4e5195605ddc354d14e1 P 197b880fa39d432a9b472845876bfaf0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:58:49.174039 20633 tablet_bootstrap.cc:492] T 3ccac82a01a2445fa5b0ce6c0fcf2295 P 197b880fa39d432a9b472845876bfaf0: Bootstrap starting.
W20260129 02:58:49.133424 20405 consensus_peers.cc:597] T 42d27a28d89b409e8807ec0efd97af69 P 197b880fa39d432a9b472845876bfaf0 -> Peer 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3:37735): Couldn't send request to peer 1c50de89a5a845668696dcb28c3badd6. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
W20260129 02:58:49.188719 20969 log.cc:927] Time spent T 18f30ebaa4174fd28c958e72d0cc7e24 P 1c50de89a5a845668696dcb28c3badd6: Append to log took a long time: real 0.097s user 0.001s sys 0.000s
I20260129 02:58:49.190766 20440 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "cb0b3f259ebc4310927338a8740e87b8" candidate_uuid: "1c50de89a5a845668696dcb28c3badd6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "197b880fa39d432a9b472845876bfaf0" is_pre_election: true
I20260129 02:58:49.178293 20587 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "8055232cf0004d40a85f3bf983fe00a9" candidate_uuid: "197b880fa39d432a9b472845876bfaf0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c50de89a5a845668696dcb28c3badd6" is_pre_election: true
W20260129 02:58:49.181442 20405 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.211.3:37735, user_credentials={real_user=slave}} blocked reactor thread for 68703.2us
I20260129 02:58:49.207041 20631 tablet_bootstrap.cc:492] T 0340229d521b432580ae926f6062e3f7 P 1c50de89a5a845668696dcb28c3badd6: No bootstrap required, opened a new log
I20260129 02:58:49.207648 20631 ts_tablet_manager.cc:1403] T 0340229d521b432580ae926f6062e3f7 P 1c50de89a5a845668696dcb28c3badd6: Time spent bootstrapping tablet: real 0.444s user 0.008s sys 0.023s
I20260129 02:58:49.184414 20507 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "8055232cf0004d40a85f3bf983fe00a9" candidate_uuid: "197b880fa39d432a9b472845876bfaf0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "974b802141bd4f13beffef2aa03f88f9" is_pre_election: true
I20260129 02:58:49.212877 20631 raft_consensus.cc:359] T 0340229d521b432580ae926f6062e3f7 P 1c50de89a5a845668696dcb28c3badd6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:58:49.213887 20631 raft_consensus.cc:385] T 0340229d521b432580ae926f6062e3f7 P 1c50de89a5a845668696dcb28c3badd6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260129 02:58:49.214224 20631 raft_consensus.cc:740] T 0340229d521b432580ae926f6062e3f7 P 1c50de89a5a845668696dcb28c3badd6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1c50de89a5a845668696dcb28c3badd6, State: Initialized, Role: FOLLOWER
I20260129 02:58:49.214983 20631 consensus_queue.cc:260] T 0340229d521b432580ae926f6062e3f7 P 1c50de89a5a845668696dcb28c3badd6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:58:49.219995 20631 ts_tablet_manager.cc:1434] T 0340229d521b432580ae926f6062e3f7 P 1c50de89a5a845668696dcb28c3badd6: Time spent starting tablet: real 0.012s user 0.007s sys 0.000s
I20260129 02:58:49.220738 20510 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "75be4bd6bd45474795fd472eff2953ac" candidate_uuid: "197b880fa39d432a9b472845876bfaf0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "974b802141bd4f13beffef2aa03f88f9" is_pre_election: true
I20260129 02:58:49.221355 20510 raft_consensus.cc:2468] T 75be4bd6bd45474795fd472eff2953ac P 974b802141bd4f13beffef2aa03f88f9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 197b880fa39d432a9b472845876bfaf0 in term 0.
W20260129 02:58:49.218840 20405 leader_election.cc:343] T c934d31efd3f4e5195605ddc354d14e1 P 197b880fa39d432a9b472845876bfaf0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3:37735): Illegal state: must be running to vote when last-logged opid is not known
I20260129 02:58:49.248050 20630 tablet_bootstrap.cc:492] T 4e3805ffa70242e8ade66da260dddf66 P 974b802141bd4f13beffef2aa03f88f9: No bootstrap required, opened a new log
I20260129 02:58:49.248744 20630 ts_tablet_manager.cc:1403] T 4e3805ffa70242e8ade66da260dddf66 P 974b802141bd4f13beffef2aa03f88f9: Time spent bootstrapping tablet: real 0.315s user 0.041s sys 0.064s
W20260129 02:58:49.251102 20556 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.211.2:34517, user_credentials={real_user=slave}} blocked reactor thread for 59044.6us
W20260129 02:58:49.246198 20405 leader_election.cc:343] T c934d31efd3f4e5195605ddc354d14e1 P 197b880fa39d432a9b472845876bfaf0 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3:37735): Illegal state: must be running to vote when last-logged opid is not known
W20260129 02:58:49.269455 20407 leader_election.cc:343] T 8055232cf0004d40a85f3bf983fe00a9 P 197b880fa39d432a9b472845876bfaf0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 974b802141bd4f13beffef2aa03f88f9 (127.18.211.2:34517): Illegal state: must be running to vote when last-logged opid is not known
I20260129 02:58:49.270591 20407 leader_election.cc:304] T 75be4bd6bd45474795fd472eff2953ac P 197b880fa39d432a9b472845876bfaf0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 197b880fa39d432a9b472845876bfaf0, 974b802141bd4f13beffef2aa03f88f9; no voters:
I20260129 02:58:49.275406 20658 raft_consensus.cc:2804] T 75be4bd6bd45474795fd472eff2953ac P 197b880fa39d432a9b472845876bfaf0 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260129 02:58:49.275817 20658 raft_consensus.cc:493] T 75be4bd6bd45474795fd472eff2953ac P 197b880fa39d432a9b472845876bfaf0 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260129 02:58:49.276317 20658 raft_consensus.cc:3060] T 75be4bd6bd45474795fd472eff2953ac P 197b880fa39d432a9b472845876bfaf0 [term 0 FOLLOWER]: Advancing to term 1
I20260129 02:58:49.254738 20983 raft_consensus.cc:493] T 75be4bd6bd45474795fd472eff2953ac P 974b802141bd4f13beffef2aa03f88f9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260129 02:58:49.265224 20507 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "cb0b3f259ebc4310927338a8740e87b8" candidate_uuid: "1c50de89a5a845668696dcb28c3badd6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "974b802141bd4f13beffef2aa03f88f9" is_pre_election: true
I20260129 02:58:49.282196 20630 raft_consensus.cc:359] T 4e3805ffa70242e8ade66da260dddf66 P 974b802141bd4f13beffef2aa03f88f9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:58:49.284463 20630 raft_consensus.cc:385] T 4e3805ffa70242e8ade66da260dddf66 P 974b802141bd4f13beffef2aa03f88f9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260129 02:58:49.284839 20630 raft_consensus.cc:740] T 4e3805ffa70242e8ade66da260dddf66 P 974b802141bd4f13beffef2aa03f88f9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 974b802141bd4f13beffef2aa03f88f9, State: Initialized, Role: FOLLOWER
I20260129 02:58:49.285619 20630 consensus_queue.cc:260] T 4e3805ffa70242e8ade66da260dddf66 P 974b802141bd4f13beffef2aa03f88f9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:58:49.338487 20588 raft_consensus.cc:3060] T cb0b3f259ebc4310927338a8740e87b8 P 1c50de89a5a845668696dcb28c3badd6 [term 0 FOLLOWER]: Advancing to term 1
I20260129 02:58:49.344329 20630 ts_tablet_manager.cc:1434] T 4e3805ffa70242e8ade66da260dddf66 P 974b802141bd4f13beffef2aa03f88f9: Time spent starting tablet: real 0.095s user 0.006s sys 0.002s
W20260129 02:58:49.346118 20479 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.211.1:42027, user_credentials={real_user=slave}} blocked reactor thread for 82832.8us
W20260129 02:58:49.331672 20405 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.211.3:37735, user_credentials={real_user=slave}} blocked reactor thread for 85511.8us
W20260129 02:58:49.356396 20405 consensus_peers.cc:597] T 0678b8b468bd43cdbc3e485900844821 P 197b880fa39d432a9b472845876bfaf0 -> Peer 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3:37735): Couldn't send request to peer 1c50de89a5a845668696dcb28c3badd6. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
W20260129 02:58:49.358834 20405 leader_election.cc:343] T 8055232cf0004d40a85f3bf983fe00a9 P 197b880fa39d432a9b472845876bfaf0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3:37735): Illegal state: must be running to vote when last-logged opid is not known
I20260129 02:58:49.359324 20556 leader_election.cc:304] T cb0b3f259ebc4310927338a8740e87b8 P 1c50de89a5a845668696dcb28c3badd6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1c50de89a5a845668696dcb28c3badd6; no voters: 197b880fa39d432a9b472845876bfaf0, 974b802141bd4f13beffef2aa03f88f9
I20260129 02:58:49.364301 20829 raft_consensus.cc:2749] T 926a250fa5834e88b96ccecbefbf5086 P 1c50de89a5a845668696dcb28c3badd6 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20260129 02:58:49.364801 20976 log.cc:927] Time spent T a0fcc58d6fb6473cafbd9d34d5195269 P 197b880fa39d432a9b472845876bfaf0: Append to log took a long time: real 0.114s user 0.001s sys 0.002s
I20260129 02:58:49.365193 20829 raft_consensus.cc:493] T 0340229d521b432580ae926f6062e3f7 P 1c50de89a5a845668696dcb28c3badd6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260129 02:58:49.365796 20829 raft_consensus.cc:515] T 0340229d521b432580ae926f6062e3f7 P 1c50de89a5a845668696dcb28c3badd6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:58:49.369390 20981 consensus_queue.cc:1048] T a0fcc58d6fb6473cafbd9d34d5195269 P 974b802141bd4f13beffef2aa03f88f9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.002s
I20260129 02:58:49.372916 20631 tablet_bootstrap.cc:492] T c891d2bff4ce4568861bbc992e91fbb0 P 1c50de89a5a845668696dcb28c3badd6: Bootstrap starting.
I20260129 02:58:49.353860 20633 tablet_bootstrap.cc:654] T 3ccac82a01a2445fa5b0ce6c0fcf2295 P 197b880fa39d432a9b472845876bfaf0: Neither blocks nor log segments found. Creating new log.
I20260129 02:58:49.283783 20983 raft_consensus.cc:515] T 75be4bd6bd45474795fd472eff2953ac P 974b802141bd4f13beffef2aa03f88f9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:58:49.352629 20658 raft_consensus.cc:515] T 75be4bd6bd45474795fd472eff2953ac P 197b880fa39d432a9b472845876bfaf0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:58:49.380110 20630 tablet_bootstrap.cc:492] T 8055232cf0004d40a85f3bf983fe00a9 P 974b802141bd4f13beffef2aa03f88f9: Bootstrap starting.
I20260129 02:58:49.388543 20658 leader_election.cc:290] T 75be4bd6bd45474795fd472eff2953ac P 197b880fa39d432a9b472845876bfaf0 [CANDIDATE]: Term 1 election: Requested vote from peers 974b802141bd4f13beffef2aa03f88f9 (127.18.211.2:34517), 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3:37735)
I20260129 02:58:49.389639 20829 leader_election.cc:290] T 0340229d521b432580ae926f6062e3f7 P 1c50de89a5a845668696dcb28c3badd6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 197b880fa39d432a9b472845876bfaf0 (127.18.211.1:42027), 974b802141bd4f13beffef2aa03f88f9 (127.18.211.2:34517)
I20260129 02:58:49.359324 20405 leader_election.cc:304] T 8055232cf0004d40a85f3bf983fe00a9 P 197b880fa39d432a9b472845876bfaf0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 197b880fa39d432a9b472845876bfaf0; no voters: 1c50de89a5a845668696dcb28c3badd6, 974b802141bd4f13beffef2aa03f88f9
I20260129 02:58:49.410000 20638 raft_consensus.cc:2749] T 8055232cf0004d40a85f3bf983fe00a9 P 197b880fa39d432a9b472845876bfaf0 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20260129 02:58:49.411105 20405 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.211.3:37735, user_credentials={real_user=slave}} blocked reactor thread for 52303.5us
I20260129 02:58:49.411672 20631 tablet_bootstrap.cc:654] T c891d2bff4ce4568861bbc992e91fbb0 P 1c50de89a5a845668696dcb28c3badd6: Neither blocks nor log segments found. Creating new log.
I20260129 02:58:49.416921 20658 raft_consensus.cc:493] T 4f5a77e94fef4b16a2ba7b067c8b144a P 197b880fa39d432a9b472845876bfaf0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260129 02:58:49.417572 20658 raft_consensus.cc:515] T 4f5a77e94fef4b16a2ba7b067c8b144a P 197b880fa39d432a9b472845876bfaf0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:58:49.423856 20638 consensus_queue.cc:1048] T c891d2bff4ce4568861bbc992e91fbb0 P 197b880fa39d432a9b472845876bfaf0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260129 02:58:49.430212 20658 leader_election.cc:290] T 4f5a77e94fef4b16a2ba7b067c8b144a P 197b880fa39d432a9b472845876bfaf0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 974b802141bd4f13beffef2aa03f88f9 (127.18.211.2:34517), 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3:37735)
I20260129 02:58:49.428531 20440 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "75be4bd6bd45474795fd472eff2953ac" candidate_uuid: "974b802141bd4f13beffef2aa03f88f9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "197b880fa39d432a9b472845876bfaf0" is_pre_election: true
I20260129 02:58:49.442227 20516 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "75be4bd6bd45474795fd472eff2953ac" candidate_uuid: "197b880fa39d432a9b472845876bfaf0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "974b802141bd4f13beffef2aa03f88f9"
I20260129 02:58:49.443497 20516 raft_consensus.cc:3060] T 75be4bd6bd45474795fd472eff2953ac P 974b802141bd4f13beffef2aa03f88f9 [term 0 FOLLOWER]: Advancing to term 1
I20260129 02:58:49.457716 20515 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0340229d521b432580ae926f6062e3f7" candidate_uuid: "1c50de89a5a845668696dcb28c3badd6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "974b802141bd4f13beffef2aa03f88f9" is_pre_election: true
I20260129 02:58:49.462020 20586 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "75be4bd6bd45474795fd472eff2953ac" candidate_uuid: "197b880fa39d432a9b472845876bfaf0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c50de89a5a845668696dcb28c3badd6" is_pre_election: true
I20260129 02:58:49.468029 20440 raft_consensus.cc:2393] T 75be4bd6bd45474795fd472eff2953ac P 197b880fa39d432a9b472845876bfaf0 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 974b802141bd4f13beffef2aa03f88f9 in current term 1: Already voted for candidate 197b880fa39d432a9b472845876bfaf0 in this term.
I20260129 02:58:49.449245 20983 leader_election.cc:290] T 75be4bd6bd45474795fd472eff2953ac P 974b802141bd4f13beffef2aa03f88f9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 197b880fa39d432a9b472845876bfaf0 (127.18.211.1:42027), 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3:37735)
I20260129 02:58:49.492681 20586 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "75be4bd6bd45474795fd472eff2953ac" candidate_uuid: "974b802141bd4f13beffef2aa03f88f9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c50de89a5a845668696dcb28c3badd6" is_pre_election: true
I20260129 02:58:49.495762 20630 tablet_bootstrap.cc:654] T 8055232cf0004d40a85f3bf983fe00a9 P 974b802141bd4f13beffef2aa03f88f9: Neither blocks nor log segments found. Creating new log.
I20260129 02:58:49.507805 20516 raft_consensus.cc:2468] T 75be4bd6bd45474795fd472eff2953ac P 974b802141bd4f13beffef2aa03f88f9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 197b880fa39d432a9b472845876bfaf0 in term 1.
I20260129 02:58:49.501598 20869 raft_consensus.cc:493] T 4e3805ffa70242e8ade66da260dddf66 P 974b802141bd4f13beffef2aa03f88f9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260129 02:58:49.509025 20438 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0340229d521b432580ae926f6062e3f7" candidate_uuid: "1c50de89a5a845668696dcb28c3badd6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "197b880fa39d432a9b472845876bfaf0" is_pre_election: true
I20260129 02:58:49.550362 20989 raft_consensus.cc:2749] T cb0b3f259ebc4310927338a8740e87b8 P 1c50de89a5a845668696dcb28c3badd6 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260129 02:58:49.530916 20869 raft_consensus.cc:515] T 4e3805ffa70242e8ade66da260dddf66 P 974b802141bd4f13beffef2aa03f88f9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:58:49.548260 20585 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "75be4bd6bd45474795fd472eff2953ac" candidate_uuid: "197b880fa39d432a9b472845876bfaf0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c50de89a5a845668696dcb28c3badd6"
W20260129 02:58:49.555792 20992 log.cc:927] Time spent T cb0b3f259ebc4310927338a8740e87b8 P 974b802141bd4f13beffef2aa03f88f9: Append to log took a long time: real 0.080s user 0.001s sys 0.000s
I20260129 02:58:49.559427 20869 leader_election.cc:290] T 4e3805ffa70242e8ade66da260dddf66 P 974b802141bd4f13beffef2aa03f88f9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 197b880fa39d432a9b472845876bfaf0 (127.18.211.1:42027), 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3:37735)
I20260129 02:58:49.561594 20631 tablet_bootstrap.cc:492] T c891d2bff4ce4568861bbc992e91fbb0 P 1c50de89a5a845668696dcb28c3badd6: No bootstrap required, opened a new log
I20260129 02:58:49.562006 20631 ts_tablet_manager.cc:1403] T c891d2bff4ce4568861bbc992e91fbb0 P 1c50de89a5a845668696dcb28c3badd6: Time spent bootstrapping tablet: real 0.189s user 0.015s sys 0.002s
W20260129 02:58:49.552309 20480 leader_election.cc:343] T 75be4bd6bd45474795fd472eff2953ac P 974b802141bd4f13beffef2aa03f88f9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3:37735): Illegal state: must be running to vote when last-logged opid is not known
I20260129 02:58:49.565869 20633 tablet_bootstrap.cc:492] T 3ccac82a01a2445fa5b0ce6c0fcf2295 P 197b880fa39d432a9b472845876bfaf0: No bootstrap required, opened a new log
I20260129 02:58:49.566344 20633 ts_tablet_manager.cc:1403] T 3ccac82a01a2445fa5b0ce6c0fcf2295 P 197b880fa39d432a9b472845876bfaf0: Time spent bootstrapping tablet: real 0.405s user 0.026s sys 0.059s
I20260129 02:58:49.574892 20438 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4e3805ffa70242e8ade66da260dddf66" candidate_uuid: "974b802141bd4f13beffef2aa03f88f9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "197b880fa39d432a9b472845876bfaf0" is_pre_election: true
I20260129 02:58:49.575639 20508 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4f5a77e94fef4b16a2ba7b067c8b144a" candidate_uuid: "197b880fa39d432a9b472845876bfaf0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "974b802141bd4f13beffef2aa03f88f9" is_pre_election: true
W20260129 02:58:49.576427 20479 leader_election.cc:343] T 4e3805ffa70242e8ade66da260dddf66 P 974b802141bd4f13beffef2aa03f88f9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 197b880fa39d432a9b472845876bfaf0 (127.18.211.1:42027): Illegal state: must be running to vote when last-logged opid is not known
I20260129 02:58:49.576594 20480 leader_election.cc:304] T 75be4bd6bd45474795fd472eff2953ac P 974b802141bd4f13beffef2aa03f88f9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 974b802141bd4f13beffef2aa03f88f9; no voters: 197b880fa39d432a9b472845876bfaf0, 1c50de89a5a845668696dcb28c3badd6
I20260129 02:58:49.577603 20590 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4f5a77e94fef4b16a2ba7b067c8b144a" candidate_uuid: "197b880fa39d432a9b472845876bfaf0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c50de89a5a845668696dcb28c3badd6" is_pre_election: true
I20260129 02:58:49.579990 20553 leader_election.cc:304] T 0340229d521b432580ae926f6062e3f7 P 1c50de89a5a845668696dcb28c3badd6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1c50de89a5a845668696dcb28c3badd6; no voters: 197b880fa39d432a9b472845876bfaf0, 974b802141bd4f13beffef2aa03f88f9
I20260129 02:58:49.580718 20631 raft_consensus.cc:359] T c891d2bff4ce4568861bbc992e91fbb0 P 1c50de89a5a845668696dcb28c3badd6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:58:49.581401 20631 raft_consensus.cc:385] T c891d2bff4ce4568861bbc992e91fbb0 P 1c50de89a5a845668696dcb28c3badd6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260129 02:58:49.581661 20631 raft_consensus.cc:740] T c891d2bff4ce4568861bbc992e91fbb0 P 1c50de89a5a845668696dcb28c3badd6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1c50de89a5a845668696dcb28c3badd6, State: Initialized, Role: FOLLOWER
I20260129 02:58:49.582286 20631 consensus_queue.cc:260] T c891d2bff4ce4568861bbc992e91fbb0 P 1c50de89a5a845668696dcb28c3badd6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
W20260129 02:58:49.577778 20405 leader_election.cc:343] T 75be4bd6bd45474795fd472eff2953ac P 197b880fa39d432a9b472845876bfaf0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3:37735): Illegal state: must be running to vote when last-logged opid is not known
I20260129 02:58:49.586411 20633 raft_consensus.cc:359] T 3ccac82a01a2445fa5b0ce6c0fcf2295 P 197b880fa39d432a9b472845876bfaf0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:58:49.587080 20633 raft_consensus.cc:385] T 3ccac82a01a2445fa5b0ce6c0fcf2295 P 197b880fa39d432a9b472845876bfaf0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260129 02:58:49.587358 20633 raft_consensus.cc:740] T 3ccac82a01a2445fa5b0ce6c0fcf2295 P 197b880fa39d432a9b472845876bfaf0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 197b880fa39d432a9b472845876bfaf0, State: Initialized, Role: FOLLOWER
I20260129 02:58:49.586864 20338 catalog_manager.cc:5654] T c934d31efd3f4e5195605ddc354d14e1 P 197b880fa39d432a9b472845876bfaf0 reported cstate change: term changed from 0 to 1, leader changed from <none> to 197b880fa39d432a9b472845876bfaf0 (127.18.211.1). New cstate: current_term: 1 leader_uuid: "197b880fa39d432a9b472845876bfaf0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } health_report { overall_health: UNKNOWN } } }
I20260129 02:58:49.588040 20633 consensus_queue.cc:260] T 3ccac82a01a2445fa5b0ce6c0fcf2295 P 197b880fa39d432a9b472845876bfaf0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:58:49.587487 20869 raft_consensus.cc:2749] T 75be4bd6bd45474795fd472eff2953ac P 974b802141bd4f13beffef2aa03f88f9 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260129 02:58:49.609014 20407 leader_election.cc:304] T 75be4bd6bd45474795fd472eff2953ac P 197b880fa39d432a9b472845876bfaf0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 197b880fa39d432a9b472845876bfaf0, 974b802141bd4f13beffef2aa03f88f9; no voters:
I20260129 02:58:49.593727 20978 raft_consensus.cc:2749] T 0340229d521b432580ae926f6062e3f7 P 1c50de89a5a845668696dcb28c3badd6 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260129 02:58:49.624085 20631 ts_tablet_manager.cc:1434] T c891d2bff4ce4568861bbc992e91fbb0 P 1c50de89a5a845668696dcb28c3badd6: Time spent starting tablet: real 0.062s user 0.002s sys 0.005s
W20260129 02:58:49.595913 20405 consensus_peers.cc:597] T c891d2bff4ce4568861bbc992e91fbb0 P 197b880fa39d432a9b472845876bfaf0 -> Peer 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3:37735): Couldn't send request to peer 1c50de89a5a845668696dcb28c3badd6. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 1: this message will repeat every 5th retry.
I20260129 02:58:49.609910 20586 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4e3805ffa70242e8ade66da260dddf66" candidate_uuid: "974b802141bd4f13beffef2aa03f88f9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c50de89a5a845668696dcb28c3badd6" is_pre_election: true
I20260129 02:58:49.628729 20631 tablet_bootstrap.cc:492] T c934d31efd3f4e5195605ddc354d14e1 P 1c50de89a5a845668696dcb28c3badd6: Bootstrap starting.
I20260129 02:58:49.596585 20827 consensus_queue.cc:1048] T 926a250fa5834e88b96ccecbefbf5086 P 197b880fa39d432a9b472845876bfaf0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.032s
I20260129 02:58:49.637121 20633 ts_tablet_manager.cc:1434] T 3ccac82a01a2445fa5b0ce6c0fcf2295 P 197b880fa39d432a9b472845876bfaf0: Time spent starting tablet: real 0.071s user 0.005s sys 0.001s
I20260129 02:58:49.622854 20828 raft_consensus.cc:2804] T 75be4bd6bd45474795fd472eff2953ac P 197b880fa39d432a9b472845876bfaf0 [term 1 FOLLOWER]: Leader election won for term 1
I20260129 02:58:49.641202 20633 tablet_bootstrap.cc:492] T 00e79d8f25a34676b1d866eb2068a2b2 P 197b880fa39d432a9b472845876bfaf0: Bootstrap starting.
I20260129 02:58:49.643271 20658 consensus_queue.cc:1048] T 926a250fa5834e88b96ccecbefbf5086 P 197b880fa39d432a9b472845876bfaf0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
W20260129 02:58:49.658032 20407 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.211.2:34517, user_credentials={real_user=slave}} blocked reactor thread for 47976.4us
I20260129 02:58:49.653116 20828 raft_consensus.cc:697] T 75be4bd6bd45474795fd472eff2953ac P 197b880fa39d432a9b472845876bfaf0 [term 1 LEADER]: Becoming Leader. State: Replica: 197b880fa39d432a9b472845876bfaf0, State: Running, Role: LEADER
W20260129 02:58:49.665259 20407 leader_election.cc:343] T 4f5a77e94fef4b16a2ba7b067c8b144a P 197b880fa39d432a9b472845876bfaf0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 974b802141bd4f13beffef2aa03f88f9 (127.18.211.2:34517): Illegal state: must be running to vote when last-logged opid is not known
W20260129 02:58:49.658123 20480 leader_election.cc:343] T 4e3805ffa70242e8ade66da260dddf66 P 974b802141bd4f13beffef2aa03f88f9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3:37735): Illegal state: must be running to vote when last-logged opid is not known
I20260129 02:58:49.670336 20828 consensus_queue.cc:237] T 75be4bd6bd45474795fd472eff2953ac P 197b880fa39d432a9b472845876bfaf0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:58:49.679447 20630 tablet_bootstrap.cc:492] T 8055232cf0004d40a85f3bf983fe00a9 P 974b802141bd4f13beffef2aa03f88f9: No bootstrap required, opened a new log
I20260129 02:58:49.680074 20630 ts_tablet_manager.cc:1403] T 8055232cf0004d40a85f3bf983fe00a9 P 974b802141bd4f13beffef2aa03f88f9: Time spent bootstrapping tablet: real 0.300s user 0.019s sys 0.031s
I20260129 02:58:49.675609 20480 leader_election.cc:304] T 4e3805ffa70242e8ade66da260dddf66 P 974b802141bd4f13beffef2aa03f88f9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 974b802141bd4f13beffef2aa03f88f9; no voters: 197b880fa39d432a9b472845876bfaf0, 1c50de89a5a845668696dcb28c3badd6
I20260129 02:58:49.683235 20630 raft_consensus.cc:359] T 8055232cf0004d40a85f3bf983fe00a9 P 974b802141bd4f13beffef2aa03f88f9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:58:49.683895 20630 raft_consensus.cc:385] T 8055232cf0004d40a85f3bf983fe00a9 P 974b802141bd4f13beffef2aa03f88f9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260129 02:58:49.683929 20983 raft_consensus.cc:2749] T 4e3805ffa70242e8ade66da260dddf66 P 974b802141bd4f13beffef2aa03f88f9 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260129 02:58:49.684265 20630 raft_consensus.cc:740] T 8055232cf0004d40a85f3bf983fe00a9 P 974b802141bd4f13beffef2aa03f88f9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 974b802141bd4f13beffef2aa03f88f9, State: Initialized, Role: FOLLOWER
I20260129 02:58:49.684864 20630 consensus_queue.cc:260] T 8055232cf0004d40a85f3bf983fe00a9 P 974b802141bd4f13beffef2aa03f88f9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:58:49.686205 20631 tablet_bootstrap.cc:654] T c934d31efd3f4e5195605ddc354d14e1 P 1c50de89a5a845668696dcb28c3badd6: Neither blocks nor log segments found. Creating new log.
I20260129 02:58:49.696861 20630 ts_tablet_manager.cc:1434] T 8055232cf0004d40a85f3bf983fe00a9 P 974b802141bd4f13beffef2aa03f88f9: Time spent starting tablet: real 0.016s user 0.006s sys 0.000s
I20260129 02:58:49.698078 20630 tablet_bootstrap.cc:492] T 7d9494f5584d4d5db8b9adf1c8fc888b P 974b802141bd4f13beffef2aa03f88f9: Bootstrap starting.
I20260129 02:58:49.723695 20638 consensus_queue.cc:1048] T c934d31efd3f4e5195605ddc354d14e1 P 197b880fa39d432a9b472845876bfaf0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.004s
I20260129 02:58:49.738027 20630 tablet_bootstrap.cc:654] T 7d9494f5584d4d5db8b9adf1c8fc888b P 974b802141bd4f13beffef2aa03f88f9: Neither blocks nor log segments found. Creating new log.
W20260129 02:58:49.703933 20405 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.211.3:37735, user_credentials={real_user=slave}} blocked reactor thread for 60907us
I20260129 02:58:49.751024 20633 tablet_bootstrap.cc:654] T 00e79d8f25a34676b1d866eb2068a2b2 P 197b880fa39d432a9b472845876bfaf0: Neither blocks nor log segments found. Creating new log.
I20260129 02:58:49.770780 20631 tablet_bootstrap.cc:492] T c934d31efd3f4e5195605ddc354d14e1 P 1c50de89a5a845668696dcb28c3badd6: No bootstrap required, opened a new log
I20260129 02:58:49.771565 20631 ts_tablet_manager.cc:1403] T c934d31efd3f4e5195605ddc354d14e1 P 1c50de89a5a845668696dcb28c3badd6: Time spent bootstrapping tablet: real 0.143s user 0.013s sys 0.008s
I20260129 02:58:49.779235 20633 tablet_bootstrap.cc:492] T 00e79d8f25a34676b1d866eb2068a2b2 P 197b880fa39d432a9b472845876bfaf0: No bootstrap required, opened a new log
I20260129 02:58:49.779738 20633 ts_tablet_manager.cc:1403] T 00e79d8f25a34676b1d866eb2068a2b2 P 197b880fa39d432a9b472845876bfaf0: Time spent bootstrapping tablet: real 0.139s user 0.012s sys 0.008s
W20260129 02:58:49.775669 20405 leader_election.cc:343] T 75be4bd6bd45474795fd472eff2953ac P 197b880fa39d432a9b472845876bfaf0 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3:37735): Illegal state: must be running to vote when last-logged opid is not known
W20260129 02:58:49.784415 20405 leader_election.cc:343] T 4f5a77e94fef4b16a2ba7b067c8b144a P 197b880fa39d432a9b472845876bfaf0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3:37735): Illegal state: must be running to vote when last-logged opid is not known
I20260129 02:58:49.784592 20631 raft_consensus.cc:359] T c934d31efd3f4e5195605ddc354d14e1 P 1c50de89a5a845668696dcb28c3badd6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:58:49.785022 20405 leader_election.cc:304] T 4f5a77e94fef4b16a2ba7b067c8b144a P 197b880fa39d432a9b472845876bfaf0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 197b880fa39d432a9b472845876bfaf0; no voters: 1c50de89a5a845668696dcb28c3badd6, 974b802141bd4f13beffef2aa03f88f9
I20260129 02:58:49.785517 20631 raft_consensus.cc:385] T c934d31efd3f4e5195605ddc354d14e1 P 1c50de89a5a845668696dcb28c3badd6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260129 02:58:49.786104 20631 raft_consensus.cc:740] T c934d31efd3f4e5195605ddc354d14e1 P 1c50de89a5a845668696dcb28c3badd6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1c50de89a5a845668696dcb28c3badd6, State: Initialized, Role: FOLLOWER
I20260129 02:58:49.786806 20631 consensus_queue.cc:260] T c934d31efd3f4e5195605ddc354d14e1 P 1c50de89a5a845668696dcb28c3badd6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:58:49.787223 20658 raft_consensus.cc:2749] T 4f5a77e94fef4b16a2ba7b067c8b144a P 197b880fa39d432a9b472845876bfaf0 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260129 02:58:49.802990 20631 ts_tablet_manager.cc:1434] T c934d31efd3f4e5195605ddc354d14e1 P 1c50de89a5a845668696dcb28c3badd6: Time spent starting tablet: real 0.031s user 0.002s sys 0.004s
I20260129 02:58:49.804044 20631 tablet_bootstrap.cc:492] T 42d27a28d89b409e8807ec0efd97af69 P 1c50de89a5a845668696dcb28c3badd6: Bootstrap starting.
I20260129 02:58:49.809371 20633 raft_consensus.cc:359] T 00e79d8f25a34676b1d866eb2068a2b2 P 197b880fa39d432a9b472845876bfaf0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:58:49.811611 20630 tablet_bootstrap.cc:492] T 7d9494f5584d4d5db8b9adf1c8fc888b P 974b802141bd4f13beffef2aa03f88f9: No bootstrap required, opened a new log
I20260129 02:58:49.812103 20630 ts_tablet_manager.cc:1403] T 7d9494f5584d4d5db8b9adf1c8fc888b P 974b802141bd4f13beffef2aa03f88f9: Time spent bootstrapping tablet: real 0.114s user 0.015s sys 0.009s
I20260129 02:58:49.810050 20633 raft_consensus.cc:385] T 00e79d8f25a34676b1d866eb2068a2b2 P 197b880fa39d432a9b472845876bfaf0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260129 02:58:49.812892 20633 raft_consensus.cc:740] T 00e79d8f25a34676b1d866eb2068a2b2 P 197b880fa39d432a9b472845876bfaf0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 197b880fa39d432a9b472845876bfaf0, State: Initialized, Role: FOLLOWER
I20260129 02:58:49.815233 20633 consensus_queue.cc:260] T 00e79d8f25a34676b1d866eb2068a2b2 P 197b880fa39d432a9b472845876bfaf0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:58:49.816030 20630 raft_consensus.cc:359] T 7d9494f5584d4d5db8b9adf1c8fc888b P 974b802141bd4f13beffef2aa03f88f9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:58:49.816722 20630 raft_consensus.cc:385] T 7d9494f5584d4d5db8b9adf1c8fc888b P 974b802141bd4f13beffef2aa03f88f9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260129 02:58:49.816955 20630 raft_consensus.cc:740] T 7d9494f5584d4d5db8b9adf1c8fc888b P 974b802141bd4f13beffef2aa03f88f9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 974b802141bd4f13beffef2aa03f88f9, State: Initialized, Role: FOLLOWER
I20260129 02:58:49.817655 20630 consensus_queue.cc:260] T 7d9494f5584d4d5db8b9adf1c8fc888b P 974b802141bd4f13beffef2aa03f88f9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:58:49.819659 20585 raft_consensus.cc:3060] T 0340229d521b432580ae926f6062e3f7 P 1c50de89a5a845668696dcb28c3badd6 [term 0 FOLLOWER]: Advancing to term 1
I20260129 02:58:49.821096 20588 raft_consensus.cc:3060] T c891d2bff4ce4568861bbc992e91fbb0 P 1c50de89a5a845668696dcb28c3badd6 [term 0 FOLLOWER]: Advancing to term 1
I20260129 02:58:49.822449 20583 raft_consensus.cc:3060] T c934d31efd3f4e5195605ddc354d14e1 P 1c50de89a5a845668696dcb28c3badd6 [term 0 FOLLOWER]: Advancing to term 1
I20260129 02:58:49.823770 20630 ts_tablet_manager.cc:1434] T 7d9494f5584d4d5db8b9adf1c8fc888b P 974b802141bd4f13beffef2aa03f88f9: Time spent starting tablet: real 0.011s user 0.006s sys 0.000s
I20260129 02:58:49.824739 20630 tablet_bootstrap.cc:492] T 4f5a77e94fef4b16a2ba7b067c8b144a P 974b802141bd4f13beffef2aa03f88f9: Bootstrap starting.
I20260129 02:58:49.837479 20336 catalog_manager.cc:5654] T 75be4bd6bd45474795fd472eff2953ac P 197b880fa39d432a9b472845876bfaf0 reported cstate change: term changed from 0 to 1, leader changed from <none> to 197b880fa39d432a9b472845876bfaf0 (127.18.211.1). New cstate: current_term: 1 leader_uuid: "197b880fa39d432a9b472845876bfaf0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } health_report { overall_health: UNKNOWN } } }
I20260129 02:58:49.840378 20633 ts_tablet_manager.cc:1434] T 00e79d8f25a34676b1d866eb2068a2b2 P 197b880fa39d432a9b472845876bfaf0: Time spent starting tablet: real 0.060s user 0.007s sys 0.000s
I20260129 02:58:49.848758 20633 tablet_bootstrap.cc:492] T a4c1226b987b468fa84dfc6ba0e2dbe1 P 197b880fa39d432a9b472845876bfaf0: Bootstrap starting.
I20260129 02:58:49.863106 20630 tablet_bootstrap.cc:654] T 4f5a77e94fef4b16a2ba7b067c8b144a P 974b802141bd4f13beffef2aa03f88f9: Neither blocks nor log segments found. Creating new log.
I20260129 02:58:49.864076 20631 tablet_bootstrap.cc:654] T 42d27a28d89b409e8807ec0efd97af69 P 1c50de89a5a845668696dcb28c3badd6: Neither blocks nor log segments found. Creating new log.
I20260129 02:58:49.875169 20630 tablet_bootstrap.cc:492] T 4f5a77e94fef4b16a2ba7b067c8b144a P 974b802141bd4f13beffef2aa03f88f9: No bootstrap required, opened a new log
I20260129 02:58:49.875726 20630 ts_tablet_manager.cc:1403] T 4f5a77e94fef4b16a2ba7b067c8b144a P 974b802141bd4f13beffef2aa03f88f9: Time spent bootstrapping tablet: real 0.051s user 0.014s sys 0.015s
I20260129 02:58:49.878355 20630 raft_consensus.cc:359] T 4f5a77e94fef4b16a2ba7b067c8b144a P 974b802141bd4f13beffef2aa03f88f9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:58:49.879197 20630 raft_consensus.cc:385] T 4f5a77e94fef4b16a2ba7b067c8b144a P 974b802141bd4f13beffef2aa03f88f9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260129 02:58:49.879539 20630 raft_consensus.cc:740] T 4f5a77e94fef4b16a2ba7b067c8b144a P 974b802141bd4f13beffef2aa03f88f9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 974b802141bd4f13beffef2aa03f88f9, State: Initialized, Role: FOLLOWER
I20260129 02:58:49.880326 20630 consensus_queue.cc:260] T 4f5a77e94fef4b16a2ba7b067c8b144a P 974b802141bd4f13beffef2aa03f88f9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:58:49.882195 20630 ts_tablet_manager.cc:1434] T 4f5a77e94fef4b16a2ba7b067c8b144a P 974b802141bd4f13beffef2aa03f88f9: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20260129 02:58:49.883549 20984 raft_consensus.cc:493] T 8055232cf0004d40a85f3bf983fe00a9 P 974b802141bd4f13beffef2aa03f88f9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260129 02:58:49.884196 20630 tablet_bootstrap.cc:492] T a8479d1fe0bd41e5946186e3a6ca8c7a P 974b802141bd4f13beffef2aa03f88f9: Bootstrap starting.
I20260129 02:58:49.885993 20658 consensus_queue.cc:1048] T c934d31efd3f4e5195605ddc354d14e1 P 197b880fa39d432a9b472845876bfaf0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260129 02:58:49.884253 20984 raft_consensus.cc:515] T 8055232cf0004d40a85f3bf983fe00a9 P 974b802141bd4f13beffef2aa03f88f9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:58:49.892632 20984 leader_election.cc:290] T 8055232cf0004d40a85f3bf983fe00a9 P 974b802141bd4f13beffef2aa03f88f9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 197b880fa39d432a9b472845876bfaf0 (127.18.211.1:42027), 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3:37735)
I20260129 02:58:49.893898 20438 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "8055232cf0004d40a85f3bf983fe00a9" candidate_uuid: "974b802141bd4f13beffef2aa03f88f9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "197b880fa39d432a9b472845876bfaf0" is_pre_election: true
I20260129 02:58:49.894794 20438 raft_consensus.cc:2468] T 8055232cf0004d40a85f3bf983fe00a9 P 197b880fa39d432a9b472845876bfaf0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 974b802141bd4f13beffef2aa03f88f9 in term 0.
I20260129 02:58:49.895927 20633 tablet_bootstrap.cc:654] T a4c1226b987b468fa84dfc6ba0e2dbe1 P 197b880fa39d432a9b472845876bfaf0: Neither blocks nor log segments found. Creating new log.
I20260129 02:58:49.900332 20630 tablet_bootstrap.cc:654] T a8479d1fe0bd41e5946186e3a6ca8c7a P 974b802141bd4f13beffef2aa03f88f9: Neither blocks nor log segments found. Creating new log.
I20260129 02:58:49.901203 20479 leader_election.cc:304] T 8055232cf0004d40a85f3bf983fe00a9 P 974b802141bd4f13beffef2aa03f88f9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 197b880fa39d432a9b472845876bfaf0, 974b802141bd4f13beffef2aa03f88f9; no voters:
I20260129 02:58:49.908187 20988 raft_consensus.cc:2804] T 8055232cf0004d40a85f3bf983fe00a9 P 974b802141bd4f13beffef2aa03f88f9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260129 02:58:49.908582 20988 raft_consensus.cc:493] T 8055232cf0004d40a85f3bf983fe00a9 P 974b802141bd4f13beffef2aa03f88f9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260129 02:58:49.908900 20988 raft_consensus.cc:3060] T 8055232cf0004d40a85f3bf983fe00a9 P 974b802141bd4f13beffef2aa03f88f9 [term 0 FOLLOWER]: Advancing to term 1
I20260129 02:58:49.946900 20658 raft_consensus.cc:493] T 3ccac82a01a2445fa5b0ce6c0fcf2295 P 197b880fa39d432a9b472845876bfaf0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260129 02:58:49.948197 20658 raft_consensus.cc:515] T 3ccac82a01a2445fa5b0ce6c0fcf2295 P 197b880fa39d432a9b472845876bfaf0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:58:49.952405 20516 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3ccac82a01a2445fa5b0ce6c0fcf2295" candidate_uuid: "197b880fa39d432a9b472845876bfaf0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "974b802141bd4f13beffef2aa03f88f9" is_pre_election: true
I20260129 02:58:49.953495 20516 raft_consensus.cc:2468] T 3ccac82a01a2445fa5b0ce6c0fcf2295 P 974b802141bd4f13beffef2aa03f88f9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 197b880fa39d432a9b472845876bfaf0 in term 0.
I20260129 02:58:49.954897 20407 leader_election.cc:304] T 3ccac82a01a2445fa5b0ce6c0fcf2295 P 197b880fa39d432a9b472845876bfaf0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 197b880fa39d432a9b472845876bfaf0, 974b802141bd4f13beffef2aa03f88f9; no voters:
I20260129 02:58:49.958673 20658 leader_election.cc:290] T 3ccac82a01a2445fa5b0ce6c0fcf2295 P 197b880fa39d432a9b472845876bfaf0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 974b802141bd4f13beffef2aa03f88f9 (127.18.211.2:34517), 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3:37735)
W20260129 02:58:50.107954 20407 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.211.2:34517, user_credentials={real_user=slave}} blocked reactor thread for 153109us
I20260129 02:58:49.949692 20585 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "8055232cf0004d40a85f3bf983fe00a9" candidate_uuid: "974b802141bd4f13beffef2aa03f88f9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c50de89a5a845668696dcb28c3badd6" is_pre_election: true
I20260129 02:58:50.124980 20658 raft_consensus.cc:2804] T 3ccac82a01a2445fa5b0ce6c0fcf2295 P 197b880fa39d432a9b472845876bfaf0 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260129 02:58:50.125429 20658 raft_consensus.cc:493] T 3ccac82a01a2445fa5b0ce6c0fcf2295 P 197b880fa39d432a9b472845876bfaf0 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260129 02:58:50.125717 20658 raft_consensus.cc:3060] T 3ccac82a01a2445fa5b0ce6c0fcf2295 P 197b880fa39d432a9b472845876bfaf0 [term 0 FOLLOWER]: Advancing to term 1
I20260129 02:58:50.137825 20631 tablet_bootstrap.cc:492] T 42d27a28d89b409e8807ec0efd97af69 P 1c50de89a5a845668696dcb28c3badd6: No bootstrap required, opened a new log
I20260129 02:58:50.138365 20631 ts_tablet_manager.cc:1403] T 42d27a28d89b409e8807ec0efd97af69 P 1c50de89a5a845668696dcb28c3badd6: Time spent bootstrapping tablet: real 0.335s user 0.014s sys 0.027s
I20260129 02:58:50.138537 20988 raft_consensus.cc:515] T 8055232cf0004d40a85f3bf983fe00a9 P 974b802141bd4f13beffef2aa03f88f9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:58:50.141466 20587 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3ccac82a01a2445fa5b0ce6c0fcf2295" candidate_uuid: "197b880fa39d432a9b472845876bfaf0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c50de89a5a845668696dcb28c3badd6" is_pre_election: true
W20260129 02:58:50.145700 20480 leader_election.cc:343] T 8055232cf0004d40a85f3bf983fe00a9 P 974b802141bd4f13beffef2aa03f88f9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3:37735): Illegal state: must be running to vote when last-logged opid is not known
I20260129 02:58:50.146144 20633 tablet_bootstrap.cc:492] T a4c1226b987b468fa84dfc6ba0e2dbe1 P 197b880fa39d432a9b472845876bfaf0: No bootstrap required, opened a new log
I20260129 02:58:50.146621 20633 ts_tablet_manager.cc:1403] T a4c1226b987b468fa84dfc6ba0e2dbe1 P 197b880fa39d432a9b472845876bfaf0: Time spent bootstrapping tablet: real 0.298s user 0.013s sys 0.012s
I20260129 02:58:50.191524 20631 raft_consensus.cc:359] T 42d27a28d89b409e8807ec0efd97af69 P 1c50de89a5a845668696dcb28c3badd6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:58:50.192513 20631 raft_consensus.cc:385] T 42d27a28d89b409e8807ec0efd97af69 P 1c50de89a5a845668696dcb28c3badd6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260129 02:58:50.192832 20631 raft_consensus.cc:740] T 42d27a28d89b409e8807ec0efd97af69 P 1c50de89a5a845668696dcb28c3badd6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1c50de89a5a845668696dcb28c3badd6, State: Initialized, Role: FOLLOWER
I20260129 02:58:50.193578 20631 consensus_queue.cc:260] T 42d27a28d89b409e8807ec0efd97af69 P 1c50de89a5a845668696dcb28c3badd6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:58:50.204382 20988 leader_election.cc:290] T 8055232cf0004d40a85f3bf983fe00a9 P 974b802141bd4f13beffef2aa03f88f9 [CANDIDATE]: Term 1 election: Requested vote from peers 197b880fa39d432a9b472845876bfaf0 (127.18.211.1:42027), 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3:37735)
I20260129 02:58:50.202013 20633 raft_consensus.cc:359] T a4c1226b987b468fa84dfc6ba0e2dbe1 P 197b880fa39d432a9b472845876bfaf0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:58:50.206732 20633 raft_consensus.cc:385] T a4c1226b987b468fa84dfc6ba0e2dbe1 P 197b880fa39d432a9b472845876bfaf0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260129 02:58:50.207072 20633 raft_consensus.cc:740] T a4c1226b987b468fa84dfc6ba0e2dbe1 P 197b880fa39d432a9b472845876bfaf0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 197b880fa39d432a9b472845876bfaf0, State: Initialized, Role: FOLLOWER
I20260129 02:58:50.211702 20633 consensus_queue.cc:260] T a4c1226b987b468fa84dfc6ba0e2dbe1 P 197b880fa39d432a9b472845876bfaf0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:58:50.240463 20633 ts_tablet_manager.cc:1434] T a4c1226b987b468fa84dfc6ba0e2dbe1 P 197b880fa39d432a9b472845876bfaf0: Time spent starting tablet: real 0.094s user 0.007s sys 0.002s
I20260129 02:58:50.243880 20633 tablet_bootstrap.cc:492] T 3906b080a5b34a74b8b64128635e8997 P 197b880fa39d432a9b472845876bfaf0: Bootstrap starting.
I20260129 02:58:50.237720 20630 tablet_bootstrap.cc:492] T a8479d1fe0bd41e5946186e3a6ca8c7a P 974b802141bd4f13beffef2aa03f88f9: No bootstrap required, opened a new log
I20260129 02:58:50.244647 20630 ts_tablet_manager.cc:1403] T a8479d1fe0bd41e5946186e3a6ca8c7a P 974b802141bd4f13beffef2aa03f88f9: Time spent bootstrapping tablet: real 0.361s user 0.053s sys 0.051s
I20260129 02:58:50.247721 20630 raft_consensus.cc:359] T a8479d1fe0bd41e5946186e3a6ca8c7a P 974b802141bd4f13beffef2aa03f88f9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:58:50.256342 20630 raft_consensus.cc:385] T a8479d1fe0bd41e5946186e3a6ca8c7a P 974b802141bd4f13beffef2aa03f88f9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260129 02:58:50.257295 20630 raft_consensus.cc:740] T a8479d1fe0bd41e5946186e3a6ca8c7a P 974b802141bd4f13beffef2aa03f88f9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 974b802141bd4f13beffef2aa03f88f9, State: Initialized, Role: FOLLOWER
I20260129 02:58:50.264590 20630 consensus_queue.cc:260] T a8479d1fe0bd41e5946186e3a6ca8c7a P 974b802141bd4f13beffef2aa03f88f9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:58:50.268937 20630 ts_tablet_manager.cc:1434] T a8479d1fe0bd41e5946186e3a6ca8c7a P 974b802141bd4f13beffef2aa03f88f9: Time spent starting tablet: real 0.024s user 0.006s sys 0.000s
I20260129 02:58:50.270598 20965 raft_consensus.cc:493] T 7d9494f5584d4d5db8b9adf1c8fc888b P 974b802141bd4f13beffef2aa03f88f9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260129 02:58:50.266533 20658 raft_consensus.cc:515] T 3ccac82a01a2445fa5b0ce6c0fcf2295 P 197b880fa39d432a9b472845876bfaf0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:58:50.271050 20965 raft_consensus.cc:515] T 7d9494f5584d4d5db8b9adf1c8fc888b P 974b802141bd4f13beffef2aa03f88f9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
W20260129 02:58:50.263739 20387 auto_rebalancer.cc:229] Could not retrieve cluster info: Service unavailable: Tablet not running
I20260129 02:58:50.272056 20438 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "8055232cf0004d40a85f3bf983fe00a9" candidate_uuid: "974b802141bd4f13beffef2aa03f88f9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "197b880fa39d432a9b472845876bfaf0"
I20260129 02:58:50.272950 20658 leader_election.cc:290] T 3ccac82a01a2445fa5b0ce6c0fcf2295 P 197b880fa39d432a9b472845876bfaf0 [CANDIDATE]: Term 1 election: Requested vote from peers 974b802141bd4f13beffef2aa03f88f9 (127.18.211.2:34517), 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3:37735)
I20260129 02:58:50.272991 20438 raft_consensus.cc:3060] T 8055232cf0004d40a85f3bf983fe00a9 P 197b880fa39d432a9b472845876bfaf0 [term 0 FOLLOWER]: Advancing to term 1
I20260129 02:58:50.273934 20630 tablet_bootstrap.cc:492] T a4c1226b987b468fa84dfc6ba0e2dbe1 P 974b802141bd4f13beffef2aa03f88f9: Bootstrap starting.
W20260129 02:58:50.285425 20405 leader_election.cc:343] T 3ccac82a01a2445fa5b0ce6c0fcf2295 P 197b880fa39d432a9b472845876bfaf0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3:37735): Illegal state: must be running to vote when last-logged opid is not known
I20260129 02:58:50.288079 20588 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "8055232cf0004d40a85f3bf983fe00a9" candidate_uuid: "974b802141bd4f13beffef2aa03f88f9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c50de89a5a845668696dcb28c3badd6"
I20260129 02:58:50.281414 20438 raft_consensus.cc:2468] T 8055232cf0004d40a85f3bf983fe00a9 P 197b880fa39d432a9b472845876bfaf0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 974b802141bd4f13beffef2aa03f88f9 in term 1.
I20260129 02:58:50.273423 20965 leader_election.cc:290] T 7d9494f5584d4d5db8b9adf1c8fc888b P 974b802141bd4f13beffef2aa03f88f9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 197b880fa39d432a9b472845876bfaf0 (127.18.211.1:42027), 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3:37735)
I20260129 02:58:50.290933 20584 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7d9494f5584d4d5db8b9adf1c8fc888b" candidate_uuid: "974b802141bd4f13beffef2aa03f88f9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c50de89a5a845668696dcb28c3badd6" is_pre_election: true
I20260129 02:58:50.292639 20441 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7d9494f5584d4d5db8b9adf1c8fc888b" candidate_uuid: "974b802141bd4f13beffef2aa03f88f9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "197b880fa39d432a9b472845876bfaf0" is_pre_election: true
I20260129 02:58:50.293169 20441 raft_consensus.cc:2468] T 7d9494f5584d4d5db8b9adf1c8fc888b P 197b880fa39d432a9b472845876bfaf0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 974b802141bd4f13beffef2aa03f88f9 in term 0.
W20260129 02:58:50.296160 20480 leader_election.cc:343] T 8055232cf0004d40a85f3bf983fe00a9 P 974b802141bd4f13beffef2aa03f88f9 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3:37735): Illegal state: must be running to vote when last-logged opid is not known
I20260129 02:58:50.298821 20965 raft_consensus.cc:493] T 4f5a77e94fef4b16a2ba7b067c8b144a P 974b802141bd4f13beffef2aa03f88f9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260129 02:58:50.299515 20480 leader_election.cc:343] T 7d9494f5584d4d5db8b9adf1c8fc888b P 974b802141bd4f13beffef2aa03f88f9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3:37735): Illegal state: must be running to vote when last-logged opid is not known
I20260129 02:58:50.299286 20965 raft_consensus.cc:515] T 4f5a77e94fef4b16a2ba7b067c8b144a P 974b802141bd4f13beffef2aa03f88f9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:58:50.301748 20633 tablet_bootstrap.cc:654] T 3906b080a5b34a74b8b64128635e8997 P 197b880fa39d432a9b472845876bfaf0: Neither blocks nor log segments found. Creating new log.
I20260129 02:58:50.303588 21024 raft_consensus.cc:493] T 00e79d8f25a34676b1d866eb2068a2b2 P 197b880fa39d432a9b472845876bfaf0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260129 02:58:50.303808 20479 leader_election.cc:304] T 8055232cf0004d40a85f3bf983fe00a9 P 974b802141bd4f13beffef2aa03f88f9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 197b880fa39d432a9b472845876bfaf0, 974b802141bd4f13beffef2aa03f88f9; no voters: 1c50de89a5a845668696dcb28c3badd6
I20260129 02:58:50.304103 21024 raft_consensus.cc:515] T 00e79d8f25a34676b1d866eb2068a2b2 P 197b880fa39d432a9b472845876bfaf0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:58:50.305111 20965 leader_election.cc:290] T 4f5a77e94fef4b16a2ba7b067c8b144a P 974b802141bd4f13beffef2aa03f88f9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 197b880fa39d432a9b472845876bfaf0 (127.18.211.1:42027), 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3:37735)
I20260129 02:58:50.304581 20983 raft_consensus.cc:2804] T 8055232cf0004d40a85f3bf983fe00a9 P 974b802141bd4f13beffef2aa03f88f9 [term 1 FOLLOWER]: Leader election won for term 1
I20260129 02:58:50.305753 20983 raft_consensus.cc:697] T 8055232cf0004d40a85f3bf983fe00a9 P 974b802141bd4f13beffef2aa03f88f9 [term 1 LEADER]: Becoming Leader. State: Replica: 974b802141bd4f13beffef2aa03f88f9, State: Running, Role: LEADER
I20260129 02:58:50.306078 20479 leader_election.cc:304] T 7d9494f5584d4d5db8b9adf1c8fc888b P 974b802141bd4f13beffef2aa03f88f9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 197b880fa39d432a9b472845876bfaf0, 974b802141bd4f13beffef2aa03f88f9; no voters: 1c50de89a5a845668696dcb28c3badd6
I20260129 02:58:50.306381 21024 leader_election.cc:290] T 00e79d8f25a34676b1d866eb2068a2b2 P 197b880fa39d432a9b472845876bfaf0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 974b802141bd4f13beffef2aa03f88f9 (127.18.211.2:34517), 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3:37735)
I20260129 02:58:50.306628 20983 consensus_queue.cc:237] T 8055232cf0004d40a85f3bf983fe00a9 P 974b802141bd4f13beffef2aa03f88f9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:58:50.307991 21030 raft_consensus.cc:2804] T 7d9494f5584d4d5db8b9adf1c8fc888b P 974b802141bd4f13beffef2aa03f88f9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260129 02:58:50.308362 21030 raft_consensus.cc:493] T 7d9494f5584d4d5db8b9adf1c8fc888b P 974b802141bd4f13beffef2aa03f88f9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260129 02:58:50.308674 21030 raft_consensus.cc:3060] T 7d9494f5584d4d5db8b9adf1c8fc888b P 974b802141bd4f13beffef2aa03f88f9 [term 0 FOLLOWER]: Advancing to term 1
I20260129 02:58:50.311146 20441 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4f5a77e94fef4b16a2ba7b067c8b144a" candidate_uuid: "974b802141bd4f13beffef2aa03f88f9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "197b880fa39d432a9b472845876bfaf0" is_pre_election: true
I20260129 02:58:50.311714 20441 raft_consensus.cc:2468] T 4f5a77e94fef4b16a2ba7b067c8b144a P 197b880fa39d432a9b472845876bfaf0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 974b802141bd4f13beffef2aa03f88f9 in term 0.
I20260129 02:58:50.312801 20479 leader_election.cc:304] T 4f5a77e94fef4b16a2ba7b067c8b144a P 974b802141bd4f13beffef2aa03f88f9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 197b880fa39d432a9b472845876bfaf0, 974b802141bd4f13beffef2aa03f88f9; no voters:
I20260129 02:58:50.314848 20965 raft_consensus.cc:2804] T 4f5a77e94fef4b16a2ba7b067c8b144a P 974b802141bd4f13beffef2aa03f88f9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260129 02:58:50.315207 20965 raft_consensus.cc:493] T 4f5a77e94fef4b16a2ba7b067c8b144a P 974b802141bd4f13beffef2aa03f88f9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260129 02:58:50.315542 20965 raft_consensus.cc:3060] T 4f5a77e94fef4b16a2ba7b067c8b144a P 974b802141bd4f13beffef2aa03f88f9 [term 0 FOLLOWER]: Advancing to term 1
I20260129 02:58:50.317220 20630 tablet_bootstrap.cc:654] T a4c1226b987b468fa84dfc6ba0e2dbe1 P 974b802141bd4f13beffef2aa03f88f9: Neither blocks nor log segments found. Creating new log.
I20260129 02:58:50.319113 20588 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4f5a77e94fef4b16a2ba7b067c8b144a" candidate_uuid: "974b802141bd4f13beffef2aa03f88f9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c50de89a5a845668696dcb28c3badd6" is_pre_election: true
I20260129 02:58:50.317998 21030 raft_consensus.cc:515] T 7d9494f5584d4d5db8b9adf1c8fc888b P 974b802141bd4f13beffef2aa03f88f9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:58:50.321588 20438 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7d9494f5584d4d5db8b9adf1c8fc888b" candidate_uuid: "974b802141bd4f13beffef2aa03f88f9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "197b880fa39d432a9b472845876bfaf0"
I20260129 02:58:50.327081 20965 raft_consensus.cc:515] T 4f5a77e94fef4b16a2ba7b067c8b144a P 974b802141bd4f13beffef2aa03f88f9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:58:50.328027 20631 ts_tablet_manager.cc:1434] T 42d27a28d89b409e8807ec0efd97af69 P 1c50de89a5a845668696dcb28c3badd6: Time spent starting tablet: real 0.189s user 0.007s sys 0.017s
W20260129 02:58:50.331212 20480 leader_election.cc:343] T 4f5a77e94fef4b16a2ba7b067c8b144a P 974b802141bd4f13beffef2aa03f88f9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3:37735): Illegal state: must be running to vote when last-logged opid is not known
I20260129 02:58:50.332386 20631 tablet_bootstrap.cc:492] T 75be4bd6bd45474795fd472eff2953ac P 1c50de89a5a845668696dcb28c3badd6: Bootstrap starting.
I20260129 02:58:50.333348 20516 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3ccac82a01a2445fa5b0ce6c0fcf2295" candidate_uuid: "197b880fa39d432a9b472845876bfaf0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "974b802141bd4f13beffef2aa03f88f9"
I20260129 02:58:50.333869 20516 raft_consensus.cc:3060] T 3ccac82a01a2445fa5b0ce6c0fcf2295 P 974b802141bd4f13beffef2aa03f88f9 [term 0 FOLLOWER]: Advancing to term 1
I20260129 02:58:50.328439 21030 leader_election.cc:290] T 7d9494f5584d4d5db8b9adf1c8fc888b P 974b802141bd4f13beffef2aa03f88f9 [CANDIDATE]: Term 1 election: Requested vote from peers 197b880fa39d432a9b472845876bfaf0 (127.18.211.1:42027), 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3:37735)
I20260129 02:58:50.333110 20438 raft_consensus.cc:3060] T 7d9494f5584d4d5db8b9adf1c8fc888b P 197b880fa39d432a9b472845876bfaf0 [term 0 FOLLOWER]: Advancing to term 1
I20260129 02:58:50.339015 20510 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00e79d8f25a34676b1d866eb2068a2b2" candidate_uuid: "197b880fa39d432a9b472845876bfaf0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "974b802141bd4f13beffef2aa03f88f9" is_pre_election: true
I20260129 02:58:50.339654 20510 raft_consensus.cc:2468] T 00e79d8f25a34676b1d866eb2068a2b2 P 974b802141bd4f13beffef2aa03f88f9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 197b880fa39d432a9b472845876bfaf0 in term 0.
I20260129 02:58:50.342077 20633 tablet_bootstrap.cc:492] T 3906b080a5b34a74b8b64128635e8997 P 197b880fa39d432a9b472845876bfaf0: No bootstrap required, opened a new log
I20260129 02:58:50.342518 20633 ts_tablet_manager.cc:1403] T 3906b080a5b34a74b8b64128635e8997 P 197b880fa39d432a9b472845876bfaf0: Time spent bootstrapping tablet: real 0.099s user 0.013s sys 0.000s
I20260129 02:58:50.344360 20965 leader_election.cc:290] T 4f5a77e94fef4b16a2ba7b067c8b144a P 974b802141bd4f13beffef2aa03f88f9 [CANDIDATE]: Term 1 election: Requested vote from peers 197b880fa39d432a9b472845876bfaf0 (127.18.211.1:42027), 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3:37735)
I20260129 02:58:50.345432 20441 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4f5a77e94fef4b16a2ba7b067c8b144a" candidate_uuid: "974b802141bd4f13beffef2aa03f88f9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "197b880fa39d432a9b472845876bfaf0"
I20260129 02:58:50.345866 20441 raft_consensus.cc:3060] T 4f5a77e94fef4b16a2ba7b067c8b144a P 197b880fa39d432a9b472845876bfaf0 [term 0 FOLLOWER]: Advancing to term 1
I20260129 02:58:50.342532 20583 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3ccac82a01a2445fa5b0ce6c0fcf2295" candidate_uuid: "197b880fa39d432a9b472845876bfaf0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c50de89a5a845668696dcb28c3badd6"
I20260129 02:58:50.391721 20441 raft_consensus.cc:2468] T 4f5a77e94fef4b16a2ba7b067c8b144a P 197b880fa39d432a9b472845876bfaf0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 974b802141bd4f13beffef2aa03f88f9 in term 1.
I20260129 02:58:50.393035 20479 leader_election.cc:304] T 4f5a77e94fef4b16a2ba7b067c8b144a P 974b802141bd4f13beffef2aa03f88f9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 197b880fa39d432a9b472845876bfaf0, 974b802141bd4f13beffef2aa03f88f9; no voters:
I20260129 02:58:50.395545 20407 leader_election.cc:304] T 00e79d8f25a34676b1d866eb2068a2b2 P 197b880fa39d432a9b472845876bfaf0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 197b880fa39d432a9b472845876bfaf0, 974b802141bd4f13beffef2aa03f88f9; no voters:
I20260129 02:58:50.396306 20658 raft_consensus.cc:2804] T 00e79d8f25a34676b1d866eb2068a2b2 P 197b880fa39d432a9b472845876bfaf0 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260129 02:58:50.396643 20658 raft_consensus.cc:493] T 00e79d8f25a34676b1d866eb2068a2b2 P 197b880fa39d432a9b472845876bfaf0 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260129 02:58:50.396945 20658 raft_consensus.cc:3060] T 00e79d8f25a34676b1d866eb2068a2b2 P 197b880fa39d432a9b472845876bfaf0 [term 0 FOLLOWER]: Advancing to term 1
W20260129 02:58:50.399240 20405 consensus_peers.cc:597] T 75be4bd6bd45474795fd472eff2953ac P 197b880fa39d432a9b472845876bfaf0 -> Peer 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3:37735): Couldn't send request to peer 1c50de89a5a845668696dcb28c3badd6. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 1: this message will repeat every 5th retry.
I20260129 02:58:50.403367 20658 raft_consensus.cc:515] T 00e79d8f25a34676b1d866eb2068a2b2 P 197b880fa39d432a9b472845876bfaf0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:58:50.406831 20658 leader_election.cc:290] T 00e79d8f25a34676b1d866eb2068a2b2 P 197b880fa39d432a9b472845876bfaf0 [CANDIDATE]: Term 1 election: Requested vote from peers 974b802141bd4f13beffef2aa03f88f9 (127.18.211.2:34517), 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3:37735)
I20260129 02:58:50.398567 20983 raft_consensus.cc:2804] T 4f5a77e94fef4b16a2ba7b067c8b144a P 974b802141bd4f13beffef2aa03f88f9 [term 1 FOLLOWER]: Leader election won for term 1
I20260129 02:58:50.417622 20630 tablet_bootstrap.cc:492] T a4c1226b987b468fa84dfc6ba0e2dbe1 P 974b802141bd4f13beffef2aa03f88f9: No bootstrap required, opened a new log
I20260129 02:58:50.419831 20630 ts_tablet_manager.cc:1403] T a4c1226b987b468fa84dfc6ba0e2dbe1 P 974b802141bd4f13beffef2aa03f88f9: Time spent bootstrapping tablet: real 0.146s user 0.014s sys 0.000s
I20260129 02:58:50.424984 21025 consensus_queue.cc:1048] T 75be4bd6bd45474795fd472eff2953ac P 197b880fa39d432a9b472845876bfaf0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260129 02:58:50.425442 20582 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4f5a77e94fef4b16a2ba7b067c8b144a" candidate_uuid: "974b802141bd4f13beffef2aa03f88f9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c50de89a5a845668696dcb28c3badd6"
W20260129 02:58:50.426698 20480 leader_election.cc:343] T 4f5a77e94fef4b16a2ba7b067c8b144a P 974b802141bd4f13beffef2aa03f88f9 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3:37735): Illegal state: must be running to vote when last-logged opid is not known
I20260129 02:58:50.429636 20516 raft_consensus.cc:2468] T 3ccac82a01a2445fa5b0ce6c0fcf2295 P 974b802141bd4f13beffef2aa03f88f9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 197b880fa39d432a9b472845876bfaf0 in term 1.
I20260129 02:58:50.423800 20983 raft_consensus.cc:697] T 4f5a77e94fef4b16a2ba7b067c8b144a P 974b802141bd4f13beffef2aa03f88f9 [term 1 LEADER]: Becoming Leader. State: Replica: 974b802141bd4f13beffef2aa03f88f9, State: Running, Role: LEADER
W20260129 02:58:50.432924 20405 leader_election.cc:343] T 3ccac82a01a2445fa5b0ce6c0fcf2295 P 197b880fa39d432a9b472845876bfaf0 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3:37735): Illegal state: must be running to vote when last-logged opid is not known
I20260129 02:58:50.433764 20983 consensus_queue.cc:237] T 4f5a77e94fef4b16a2ba7b067c8b144a P 974b802141bd4f13beffef2aa03f88f9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:58:50.438542 20588 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00e79d8f25a34676b1d866eb2068a2b2" candidate_uuid: "197b880fa39d432a9b472845876bfaf0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c50de89a5a845668696dcb28c3badd6"
I20260129 02:58:50.438750 20631 tablet_bootstrap.cc:654] T 75be4bd6bd45474795fd472eff2953ac P 1c50de89a5a845668696dcb28c3badd6: Neither blocks nor log segments found. Creating new log.
W20260129 02:58:50.440078 20405 leader_election.cc:343] T 00e79d8f25a34676b1d866eb2068a2b2 P 197b880fa39d432a9b472845876bfaf0 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3:37735): Illegal state: must be running to vote when last-logged opid is not known
I20260129 02:58:50.441948 20982 raft_consensus.cc:493] T 42d27a28d89b409e8807ec0efd97af69 P 1c50de89a5a845668696dcb28c3badd6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260129 02:58:50.443717 20633 raft_consensus.cc:359] T 3906b080a5b34a74b8b64128635e8997 P 197b880fa39d432a9b472845876bfaf0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:58:50.444414 20633 raft_consensus.cc:385] T 3906b080a5b34a74b8b64128635e8997 P 197b880fa39d432a9b472845876bfaf0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260129 02:58:50.444679 20633 raft_consensus.cc:740] T 3906b080a5b34a74b8b64128635e8997 P 197b880fa39d432a9b472845876bfaf0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 197b880fa39d432a9b472845876bfaf0, State: Initialized, Role: FOLLOWER
I20260129 02:58:50.445237 20982 raft_consensus.cc:515] T 42d27a28d89b409e8807ec0efd97af69 P 1c50de89a5a845668696dcb28c3badd6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:58:50.446657 20633 consensus_queue.cc:260] T 3906b080a5b34a74b8b64128635e8997 P 197b880fa39d432a9b472845876bfaf0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:58:50.449219 20630 raft_consensus.cc:359] T a4c1226b987b468fa84dfc6ba0e2dbe1 P 974b802141bd4f13beffef2aa03f88f9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:58:50.449890 20630 raft_consensus.cc:385] T a4c1226b987b468fa84dfc6ba0e2dbe1 P 974b802141bd4f13beffef2aa03f88f9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260129 02:58:50.450132 20630 raft_consensus.cc:740] T a4c1226b987b468fa84dfc6ba0e2dbe1 P 974b802141bd4f13beffef2aa03f88f9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 974b802141bd4f13beffef2aa03f88f9, State: Initialized, Role: FOLLOWER
I20260129 02:58:50.450680 20630 consensus_queue.cc:260] T a4c1226b987b468fa84dfc6ba0e2dbe1 P 974b802141bd4f13beffef2aa03f88f9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:58:50.450986 20441 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "42d27a28d89b409e8807ec0efd97af69" candidate_uuid: "1c50de89a5a845668696dcb28c3badd6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "197b880fa39d432a9b472845876bfaf0" is_pre_election: true
I20260129 02:58:50.452744 20982 leader_election.cc:290] T 42d27a28d89b409e8807ec0efd97af69 P 1c50de89a5a845668696dcb28c3badd6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 197b880fa39d432a9b472845876bfaf0 (127.18.211.1:42027), 974b802141bd4f13beffef2aa03f88f9 (127.18.211.2:34517)
I20260129 02:58:50.452914 20630 ts_tablet_manager.cc:1434] T a4c1226b987b468fa84dfc6ba0e2dbe1 P 974b802141bd4f13beffef2aa03f88f9: Time spent starting tablet: real 0.033s user 0.002s sys 0.004s
I20260129 02:58:50.453882 20630 tablet_bootstrap.cc:492] T 3906b080a5b34a74b8b64128635e8997 P 974b802141bd4f13beffef2aa03f88f9: Bootstrap starting.
I20260129 02:58:50.454372 20586 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00e79d8f25a34676b1d866eb2068a2b2" candidate_uuid: "197b880fa39d432a9b472845876bfaf0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c50de89a5a845668696dcb28c3badd6" is_pre_election: true
I20260129 02:58:50.454833 20438 raft_consensus.cc:2468] T 7d9494f5584d4d5db8b9adf1c8fc888b P 197b880fa39d432a9b472845876bfaf0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 974b802141bd4f13beffef2aa03f88f9 in term 1.
W20260129 02:58:50.455650 20405 leader_election.cc:343] T 00e79d8f25a34676b1d866eb2068a2b2 P 197b880fa39d432a9b472845876bfaf0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3:37735): Illegal state: must be running to vote when last-logged opid is not known
I20260129 02:58:50.455960 20479 leader_election.cc:304] T 7d9494f5584d4d5db8b9adf1c8fc888b P 974b802141bd4f13beffef2aa03f88f9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 197b880fa39d432a9b472845876bfaf0, 974b802141bd4f13beffef2aa03f88f9; no voters:
I20260129 02:58:50.456715 20963 raft_consensus.cc:2804] T 7d9494f5584d4d5db8b9adf1c8fc888b P 974b802141bd4f13beffef2aa03f88f9 [term 1 FOLLOWER]: Leader election won for term 1
I20260129 02:58:50.457176 20963 raft_consensus.cc:697] T 7d9494f5584d4d5db8b9adf1c8fc888b P 974b802141bd4f13beffef2aa03f88f9 [term 1 LEADER]: Becoming Leader. State: Replica: 974b802141bd4f13beffef2aa03f88f9, State: Running, Role: LEADER
I20260129 02:58:50.457967 20963 consensus_queue.cc:237] T 7d9494f5584d4d5db8b9adf1c8fc888b P 974b802141bd4f13beffef2aa03f88f9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:58:50.459353 20587 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7d9494f5584d4d5db8b9adf1c8fc888b" candidate_uuid: "974b802141bd4f13beffef2aa03f88f9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c50de89a5a845668696dcb28c3badd6"
I20260129 02:58:50.459884 21026 raft_consensus.cc:493] T a8479d1fe0bd41e5946186e3a6ca8c7a P 974b802141bd4f13beffef2aa03f88f9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260129 02:58:50.460357 21026 raft_consensus.cc:515] T a8479d1fe0bd41e5946186e3a6ca8c7a P 974b802141bd4f13beffef2aa03f88f9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:58:50.463140 21026 leader_election.cc:290] T a8479d1fe0bd41e5946186e3a6ca8c7a P 974b802141bd4f13beffef2aa03f88f9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 197b880fa39d432a9b472845876bfaf0 (127.18.211.1:42027), 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3:37735)
I20260129 02:58:50.463939 20585 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a8479d1fe0bd41e5946186e3a6ca8c7a" candidate_uuid: "974b802141bd4f13beffef2aa03f88f9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c50de89a5a845668696dcb28c3badd6" is_pre_election: true
W20260129 02:58:50.465198 20480 leader_election.cc:343] T a8479d1fe0bd41e5946186e3a6ca8c7a P 974b802141bd4f13beffef2aa03f88f9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3:37735): Illegal state: must be running to vote when last-logged opid is not known
I20260129 02:58:50.466838 20630 tablet_bootstrap.cc:654] T 3906b080a5b34a74b8b64128635e8997 P 974b802141bd4f13beffef2aa03f88f9: Neither blocks nor log segments found. Creating new log.
I20260129 02:58:50.468483 20441 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a8479d1fe0bd41e5946186e3a6ca8c7a" candidate_uuid: "974b802141bd4f13beffef2aa03f88f9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "197b880fa39d432a9b472845876bfaf0" is_pre_election: true
I20260129 02:58:50.469846 20511 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "42d27a28d89b409e8807ec0efd97af69" candidate_uuid: "1c50de89a5a845668696dcb28c3badd6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "974b802141bd4f13beffef2aa03f88f9" is_pre_election: true
W20260129 02:58:50.470110 20480 leader_election.cc:343] T 7d9494f5584d4d5db8b9adf1c8fc888b P 974b802141bd4f13beffef2aa03f88f9 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3:37735): Illegal state: must be running to vote when last-logged opid is not known
W20260129 02:58:50.471549 20479 leader_election.cc:343] T a8479d1fe0bd41e5946186e3a6ca8c7a P 974b802141bd4f13beffef2aa03f88f9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 197b880fa39d432a9b472845876bfaf0 (127.18.211.1:42027): Illegal state: must be running to vote when last-logged opid is not known
I20260129 02:58:50.471993 20479 leader_election.cc:304] T a8479d1fe0bd41e5946186e3a6ca8c7a P 974b802141bd4f13beffef2aa03f88f9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 974b802141bd4f13beffef2aa03f88f9; no voters: 197b880fa39d432a9b472845876bfaf0, 1c50de89a5a845668696dcb28c3badd6
I20260129 02:58:50.474053 20965 raft_consensus.cc:2749] T a8479d1fe0bd41e5946186e3a6ca8c7a P 974b802141bd4f13beffef2aa03f88f9 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260129 02:58:50.476082 20631 tablet_bootstrap.cc:492] T 75be4bd6bd45474795fd472eff2953ac P 1c50de89a5a845668696dcb28c3badd6: No bootstrap required, opened a new log
I20260129 02:58:50.476681 20631 ts_tablet_manager.cc:1403] T 75be4bd6bd45474795fd472eff2953ac P 1c50de89a5a845668696dcb28c3badd6: Time spent bootstrapping tablet: real 0.146s user 0.010s sys 0.006s
I20260129 02:58:50.477289 20407 leader_election.cc:304] T 3ccac82a01a2445fa5b0ce6c0fcf2295 P 197b880fa39d432a9b472845876bfaf0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 197b880fa39d432a9b472845876bfaf0, 974b802141bd4f13beffef2aa03f88f9; no voters: 1c50de89a5a845668696dcb28c3badd6
I20260129 02:58:50.479106 20511 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00e79d8f25a34676b1d866eb2068a2b2" candidate_uuid: "197b880fa39d432a9b472845876bfaf0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "974b802141bd4f13beffef2aa03f88f9"
I20260129 02:58:50.479264 21024 raft_consensus.cc:2804] T 3ccac82a01a2445fa5b0ce6c0fcf2295 P 197b880fa39d432a9b472845876bfaf0 [term 1 FOLLOWER]: Leader election won for term 1
I20260129 02:58:50.479666 20511 raft_consensus.cc:3060] T 00e79d8f25a34676b1d866eb2068a2b2 P 974b802141bd4f13beffef2aa03f88f9 [term 0 FOLLOWER]: Advancing to term 1
I20260129 02:58:50.479751 21024 raft_consensus.cc:697] T 3ccac82a01a2445fa5b0ce6c0fcf2295 P 197b880fa39d432a9b472845876bfaf0 [term 1 LEADER]: Becoming Leader. State: Replica: 197b880fa39d432a9b472845876bfaf0, State: Running, Role: LEADER
I20260129 02:58:50.479707 20631 raft_consensus.cc:359] T 75be4bd6bd45474795fd472eff2953ac P 1c50de89a5a845668696dcb28c3badd6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:58:50.480558 20631 raft_consensus.cc:385] T 75be4bd6bd45474795fd472eff2953ac P 1c50de89a5a845668696dcb28c3badd6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260129 02:58:50.480841 20631 raft_consensus.cc:740] T 75be4bd6bd45474795fd472eff2953ac P 1c50de89a5a845668696dcb28c3badd6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1c50de89a5a845668696dcb28c3badd6, State: Initialized, Role: FOLLOWER
I20260129 02:58:50.480770 21024 consensus_queue.cc:237] T 3ccac82a01a2445fa5b0ce6c0fcf2295 P 197b880fa39d432a9b472845876bfaf0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:58:50.481712 20631 consensus_queue.cc:260] T 75be4bd6bd45474795fd472eff2953ac P 1c50de89a5a845668696dcb28c3badd6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:58:50.483286 20631 ts_tablet_manager.cc:1434] T 75be4bd6bd45474795fd472eff2953ac P 1c50de89a5a845668696dcb28c3badd6: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20260129 02:58:50.484139 20631 tablet_bootstrap.cc:492] T 3ccac82a01a2445fa5b0ce6c0fcf2295 P 1c50de89a5a845668696dcb28c3badd6: Bootstrap starting.
I20260129 02:58:50.488171 20511 raft_consensus.cc:2468] T 00e79d8f25a34676b1d866eb2068a2b2 P 974b802141bd4f13beffef2aa03f88f9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 197b880fa39d432a9b472845876bfaf0 in term 1.
I20260129 02:58:50.489517 20631 tablet_bootstrap.cc:654] T 3ccac82a01a2445fa5b0ce6c0fcf2295 P 1c50de89a5a845668696dcb28c3badd6: Neither blocks nor log segments found. Creating new log.
I20260129 02:58:50.489621 20556 leader_election.cc:304] T 42d27a28d89b409e8807ec0efd97af69 P 1c50de89a5a845668696dcb28c3badd6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1c50de89a5a845668696dcb28c3badd6; no voters: 197b880fa39d432a9b472845876bfaf0, 974b802141bd4f13beffef2aa03f88f9
I20260129 02:58:50.489874 20338 catalog_manager.cc:5654] T 8055232cf0004d40a85f3bf983fe00a9 P 974b802141bd4f13beffef2aa03f88f9 reported cstate change: term changed from 0 to 1, leader changed from <none> to 974b802141bd4f13beffef2aa03f88f9 (127.18.211.2). New cstate: current_term: 1 leader_uuid: "974b802141bd4f13beffef2aa03f88f9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } health_report { overall_health: UNKNOWN } } }
I20260129 02:58:50.492105 21028 raft_consensus.cc:2749] T 42d27a28d89b409e8807ec0efd97af69 P 1c50de89a5a845668696dcb28c3badd6 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260129 02:58:50.501111 20631 tablet_bootstrap.cc:492] T 3ccac82a01a2445fa5b0ce6c0fcf2295 P 1c50de89a5a845668696dcb28c3badd6: No bootstrap required, opened a new log
I20260129 02:58:50.501554 20631 ts_tablet_manager.cc:1403] T 3ccac82a01a2445fa5b0ce6c0fcf2295 P 1c50de89a5a845668696dcb28c3badd6: Time spent bootstrapping tablet: real 0.018s user 0.015s sys 0.000s
I20260129 02:58:50.501889 20633 ts_tablet_manager.cc:1434] T 3906b080a5b34a74b8b64128635e8997 P 197b880fa39d432a9b472845876bfaf0: Time spent starting tablet: real 0.159s user 0.006s sys 0.000s
I20260129 02:58:50.502452 21031 raft_consensus.cc:493] T 75be4bd6bd45474795fd472eff2953ac P 1c50de89a5a845668696dcb28c3badd6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260129 02:58:50.502923 21031 raft_consensus.cc:515] T 75be4bd6bd45474795fd472eff2953ac P 1c50de89a5a845668696dcb28c3badd6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:58:50.503993 20631 raft_consensus.cc:359] T 3ccac82a01a2445fa5b0ce6c0fcf2295 P 1c50de89a5a845668696dcb28c3badd6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:58:50.504663 20631 raft_consensus.cc:385] T 3ccac82a01a2445fa5b0ce6c0fcf2295 P 1c50de89a5a845668696dcb28c3badd6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260129 02:58:50.504941 20631 raft_consensus.cc:740] T 3ccac82a01a2445fa5b0ce6c0fcf2295 P 1c50de89a5a845668696dcb28c3badd6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1c50de89a5a845668696dcb28c3badd6, State: Initialized, Role: FOLLOWER
I20260129 02:58:50.505728 20631 consensus_queue.cc:260] T 3ccac82a01a2445fa5b0ce6c0fcf2295 P 1c50de89a5a845668696dcb28c3badd6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:58:50.506500 21031 leader_election.cc:290] T 75be4bd6bd45474795fd472eff2953ac P 1c50de89a5a845668696dcb28c3badd6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 197b880fa39d432a9b472845876bfaf0 (127.18.211.1:42027), 974b802141bd4f13beffef2aa03f88f9 (127.18.211.2:34517)
I20260129 02:58:50.506266 20441 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "75be4bd6bd45474795fd472eff2953ac" candidate_uuid: "1c50de89a5a845668696dcb28c3badd6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "197b880fa39d432a9b472845876bfaf0" is_pre_election: true
I20260129 02:58:50.507973 20631 ts_tablet_manager.cc:1434] T 3ccac82a01a2445fa5b0ce6c0fcf2295 P 1c50de89a5a845668696dcb28c3badd6: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260129 02:58:50.502888 20633 tablet_bootstrap.cc:492] T a8479d1fe0bd41e5946186e3a6ca8c7a P 197b880fa39d432a9b472845876bfaf0: Bootstrap starting.
I20260129 02:58:50.511009 20631 tablet_bootstrap.cc:492] T 8055232cf0004d40a85f3bf983fe00a9 P 1c50de89a5a845668696dcb28c3badd6: Bootstrap starting.
I20260129 02:58:50.517050 20514 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "75be4bd6bd45474795fd472eff2953ac" candidate_uuid: "1c50de89a5a845668696dcb28c3badd6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "974b802141bd4f13beffef2aa03f88f9" is_pre_election: true
I20260129 02:58:50.518419 20556 leader_election.cc:304] T 75be4bd6bd45474795fd472eff2953ac P 1c50de89a5a845668696dcb28c3badd6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1c50de89a5a845668696dcb28c3badd6; no voters: 197b880fa39d432a9b472845876bfaf0, 974b802141bd4f13beffef2aa03f88f9
I20260129 02:58:50.519367 20633 tablet_bootstrap.cc:654] T a8479d1fe0bd41e5946186e3a6ca8c7a P 197b880fa39d432a9b472845876bfaf0: Neither blocks nor log segments found. Creating new log.
I20260129 02:58:50.527585 20631 tablet_bootstrap.cc:654] T 8055232cf0004d40a85f3bf983fe00a9 P 1c50de89a5a845668696dcb28c3badd6: Neither blocks nor log segments found. Creating new log.
I20260129 02:58:50.528257 20407 leader_election.cc:304] T 00e79d8f25a34676b1d866eb2068a2b2 P 197b880fa39d432a9b472845876bfaf0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 197b880fa39d432a9b472845876bfaf0, 974b802141bd4f13beffef2aa03f88f9; no voters: 1c50de89a5a845668696dcb28c3badd6
I20260129 02:58:50.529564 21024 raft_consensus.cc:2804] T 00e79d8f25a34676b1d866eb2068a2b2 P 197b880fa39d432a9b472845876bfaf0 [term 1 FOLLOWER]: Leader election won for term 1
I20260129 02:58:50.533721 21024 raft_consensus.cc:697] T 00e79d8f25a34676b1d866eb2068a2b2 P 197b880fa39d432a9b472845876bfaf0 [term 1 LEADER]: Becoming Leader. State: Replica: 197b880fa39d432a9b472845876bfaf0, State: Running, Role: LEADER
I20260129 02:58:50.534875 21024 consensus_queue.cc:237] T 00e79d8f25a34676b1d866eb2068a2b2 P 197b880fa39d432a9b472845876bfaf0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:58:50.519430 21028 raft_consensus.cc:2749] T 75be4bd6bd45474795fd472eff2953ac P 1c50de89a5a845668696dcb28c3badd6 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260129 02:58:50.569664 20631 tablet_bootstrap.cc:492] T 8055232cf0004d40a85f3bf983fe00a9 P 1c50de89a5a845668696dcb28c3badd6: No bootstrap required, opened a new log
I20260129 02:58:50.570123 20631 ts_tablet_manager.cc:1403] T 8055232cf0004d40a85f3bf983fe00a9 P 1c50de89a5a845668696dcb28c3badd6: Time spent bootstrapping tablet: real 0.059s user 0.022s sys 0.004s
I20260129 02:58:50.572780 20631 raft_consensus.cc:359] T 8055232cf0004d40a85f3bf983fe00a9 P 1c50de89a5a845668696dcb28c3badd6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:58:50.573426 20631 raft_consensus.cc:385] T 8055232cf0004d40a85f3bf983fe00a9 P 1c50de89a5a845668696dcb28c3badd6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260129 02:58:50.573709 20631 raft_consensus.cc:740] T 8055232cf0004d40a85f3bf983fe00a9 P 1c50de89a5a845668696dcb28c3badd6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1c50de89a5a845668696dcb28c3badd6, State: Initialized, Role: FOLLOWER
I20260129 02:58:50.574409 20631 consensus_queue.cc:260] T 8055232cf0004d40a85f3bf983fe00a9 P 1c50de89a5a845668696dcb28c3badd6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:58:50.575765 20633 tablet_bootstrap.cc:492] T a8479d1fe0bd41e5946186e3a6ca8c7a P 197b880fa39d432a9b472845876bfaf0: No bootstrap required, opened a new log
I20260129 02:58:50.576320 20633 ts_tablet_manager.cc:1403] T a8479d1fe0bd41e5946186e3a6ca8c7a P 197b880fa39d432a9b472845876bfaf0: Time spent bootstrapping tablet: real 0.074s user 0.009s sys 0.010s
I20260129 02:58:50.578922 20633 raft_consensus.cc:359] T a8479d1fe0bd41e5946186e3a6ca8c7a P 197b880fa39d432a9b472845876bfaf0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:58:50.579622 20633 raft_consensus.cc:385] T a8479d1fe0bd41e5946186e3a6ca8c7a P 197b880fa39d432a9b472845876bfaf0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260129 02:58:50.579943 20633 raft_consensus.cc:740] T a8479d1fe0bd41e5946186e3a6ca8c7a P 197b880fa39d432a9b472845876bfaf0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 197b880fa39d432a9b472845876bfaf0, State: Initialized, Role: FOLLOWER
I20260129 02:58:50.580691 20633 consensus_queue.cc:260] T a8479d1fe0bd41e5946186e3a6ca8c7a P 197b880fa39d432a9b472845876bfaf0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:58:50.576000 20336 catalog_manager.cc:5654] T 3ccac82a01a2445fa5b0ce6c0fcf2295 P 197b880fa39d432a9b472845876bfaf0 reported cstate change: term changed from 0 to 1, leader changed from <none> to 197b880fa39d432a9b472845876bfaf0 (127.18.211.1). New cstate: current_term: 1 leader_uuid: "197b880fa39d432a9b472845876bfaf0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } health_report { overall_health: UNKNOWN } } }
I20260129 02:58:50.583880 20631 ts_tablet_manager.cc:1434] T 8055232cf0004d40a85f3bf983fe00a9 P 1c50de89a5a845668696dcb28c3badd6: Time spent starting tablet: real 0.013s user 0.008s sys 0.000s
I20260129 02:58:50.584868 20631 tablet_bootstrap.cc:492] T 0678b8b468bd43cdbc3e485900844821 P 1c50de89a5a845668696dcb28c3badd6: Bootstrap starting.
I20260129 02:58:50.583441 20633 ts_tablet_manager.cc:1434] T a8479d1fe0bd41e5946186e3a6ca8c7a P 197b880fa39d432a9b472845876bfaf0: Time spent starting tablet: real 0.007s user 0.003s sys 0.001s
I20260129 02:58:50.586201 20633 tablet_bootstrap.cc:492] T 4e3805ffa70242e8ade66da260dddf66 P 197b880fa39d432a9b472845876bfaf0: Bootstrap starting.
I20260129 02:58:50.593700 20633 tablet_bootstrap.cc:654] T 4e3805ffa70242e8ade66da260dddf66 P 197b880fa39d432a9b472845876bfaf0: Neither blocks nor log segments found. Creating new log.
I20260129 02:58:50.600251 20983 raft_consensus.cc:493] T a4c1226b987b468fa84dfc6ba0e2dbe1 P 974b802141bd4f13beffef2aa03f88f9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260129 02:58:50.600962 20983 raft_consensus.cc:515] T a4c1226b987b468fa84dfc6ba0e2dbe1 P 974b802141bd4f13beffef2aa03f88f9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:58:50.604693 20983 leader_election.cc:290] T a4c1226b987b468fa84dfc6ba0e2dbe1 P 974b802141bd4f13beffef2aa03f88f9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 197b880fa39d432a9b472845876bfaf0 (127.18.211.1:42027), 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3:37735)
I20260129 02:58:50.606606 20584 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a4c1226b987b468fa84dfc6ba0e2dbe1" candidate_uuid: "974b802141bd4f13beffef2aa03f88f9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c50de89a5a845668696dcb28c3badd6" is_pre_election: true
I20260129 02:58:50.609580 20438 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a4c1226b987b468fa84dfc6ba0e2dbe1" candidate_uuid: "974b802141bd4f13beffef2aa03f88f9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "197b880fa39d432a9b472845876bfaf0" is_pre_election: true
I20260129 02:58:50.609171 20338 catalog_manager.cc:5654] T 7d9494f5584d4d5db8b9adf1c8fc888b P 974b802141bd4f13beffef2aa03f88f9 reported cstate change: term changed from 0 to 1, leader changed from <none> to 974b802141bd4f13beffef2aa03f88f9 (127.18.211.2). New cstate: current_term: 1 leader_uuid: "974b802141bd4f13beffef2aa03f88f9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } health_report { overall_health: UNKNOWN } } }
I20260129 02:58:50.610420 20438 raft_consensus.cc:2468] T a4c1226b987b468fa84dfc6ba0e2dbe1 P 197b880fa39d432a9b472845876bfaf0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 974b802141bd4f13beffef2aa03f88f9 in term 0.
W20260129 02:58:50.613054 20480 leader_election.cc:343] T a4c1226b987b468fa84dfc6ba0e2dbe1 P 974b802141bd4f13beffef2aa03f88f9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3:37735): Illegal state: must be running to vote when last-logged opid is not known
I20260129 02:58:50.614559 20479 leader_election.cc:304] T a4c1226b987b468fa84dfc6ba0e2dbe1 P 974b802141bd4f13beffef2aa03f88f9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 197b880fa39d432a9b472845876bfaf0, 974b802141bd4f13beffef2aa03f88f9; no voters: 1c50de89a5a845668696dcb28c3badd6
I20260129 02:58:50.616086 20338 catalog_manager.cc:5654] T 4f5a77e94fef4b16a2ba7b067c8b144a P 974b802141bd4f13beffef2aa03f88f9 reported cstate change: term changed from 0 to 1, leader changed from <none> to 974b802141bd4f13beffef2aa03f88f9 (127.18.211.2). New cstate: current_term: 1 leader_uuid: "974b802141bd4f13beffef2aa03f88f9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } health_report { overall_health: UNKNOWN } } }
I20260129 02:58:50.618281 20584 raft_consensus.cc:3060] T 75be4bd6bd45474795fd472eff2953ac P 1c50de89a5a845668696dcb28c3badd6 [term 0 FOLLOWER]: Advancing to term 1
I20260129 02:58:50.620689 20965 raft_consensus.cc:2804] T a4c1226b987b468fa84dfc6ba0e2dbe1 P 974b802141bd4f13beffef2aa03f88f9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260129 02:58:50.623816 20965 raft_consensus.cc:493] T a4c1226b987b468fa84dfc6ba0e2dbe1 P 974b802141bd4f13beffef2aa03f88f9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260129 02:58:50.624280 20965 raft_consensus.cc:3060] T a4c1226b987b468fa84dfc6ba0e2dbe1 P 974b802141bd4f13beffef2aa03f88f9 [term 0 FOLLOWER]: Advancing to term 1
I20260129 02:58:50.635892 20965 raft_consensus.cc:515] T a4c1226b987b468fa84dfc6ba0e2dbe1 P 974b802141bd4f13beffef2aa03f88f9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:58:50.638247 20631 tablet_bootstrap.cc:654] T 0678b8b468bd43cdbc3e485900844821 P 1c50de89a5a845668696dcb28c3badd6: Neither blocks nor log segments found. Creating new log.
I20260129 02:58:50.639515 20965 leader_election.cc:290] T a4c1226b987b468fa84dfc6ba0e2dbe1 P 974b802141bd4f13beffef2aa03f88f9 [CANDIDATE]: Term 1 election: Requested vote from peers 197b880fa39d432a9b472845876bfaf0 (127.18.211.1:42027), 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3:37735)
I20260129 02:58:50.640022 20587 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a4c1226b987b468fa84dfc6ba0e2dbe1" candidate_uuid: "974b802141bd4f13beffef2aa03f88f9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c50de89a5a845668696dcb28c3badd6"
I20260129 02:58:50.640753 20438 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a4c1226b987b468fa84dfc6ba0e2dbe1" candidate_uuid: "974b802141bd4f13beffef2aa03f88f9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "197b880fa39d432a9b472845876bfaf0"
I20260129 02:58:50.642139 20438 raft_consensus.cc:3060] T a4c1226b987b468fa84dfc6ba0e2dbe1 P 197b880fa39d432a9b472845876bfaf0 [term 0 FOLLOWER]: Advancing to term 1
W20260129 02:58:50.642599 20480 leader_election.cc:343] T a4c1226b987b468fa84dfc6ba0e2dbe1 P 974b802141bd4f13beffef2aa03f88f9 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3:37735): Illegal state: must be running to vote when last-logged opid is not known
I20260129 02:58:50.646786 20336 catalog_manager.cc:5654] T 00e79d8f25a34676b1d866eb2068a2b2 P 197b880fa39d432a9b472845876bfaf0 reported cstate change: term changed from 0 to 1, leader changed from <none> to 197b880fa39d432a9b472845876bfaf0 (127.18.211.1). New cstate: current_term: 1 leader_uuid: "197b880fa39d432a9b472845876bfaf0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } health_report { overall_health: UNKNOWN } } }
I20260129 02:58:50.641333 20630 tablet_bootstrap.cc:492] T 3906b080a5b34a74b8b64128635e8997 P 974b802141bd4f13beffef2aa03f88f9: No bootstrap required, opened a new log
I20260129 02:58:50.648463 20630 ts_tablet_manager.cc:1403] T 3906b080a5b34a74b8b64128635e8997 P 974b802141bd4f13beffef2aa03f88f9: Time spent bootstrapping tablet: real 0.195s user 0.038s sys 0.039s
I20260129 02:58:50.658176 20438 raft_consensus.cc:2468] T a4c1226b987b468fa84dfc6ba0e2dbe1 P 197b880fa39d432a9b472845876bfaf0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 974b802141bd4f13beffef2aa03f88f9 in term 1.
I20260129 02:58:50.659199 20630 raft_consensus.cc:359] T 3906b080a5b34a74b8b64128635e8997 P 974b802141bd4f13beffef2aa03f88f9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:58:50.660110 20630 raft_consensus.cc:385] T 3906b080a5b34a74b8b64128635e8997 P 974b802141bd4f13beffef2aa03f88f9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260129 02:58:50.660445 20630 raft_consensus.cc:740] T 3906b080a5b34a74b8b64128635e8997 P 974b802141bd4f13beffef2aa03f88f9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 974b802141bd4f13beffef2aa03f88f9, State: Initialized, Role: FOLLOWER
I20260129 02:58:50.660756 20479 leader_election.cc:304] T a4c1226b987b468fa84dfc6ba0e2dbe1 P 974b802141bd4f13beffef2aa03f88f9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 197b880fa39d432a9b472845876bfaf0, 974b802141bd4f13beffef2aa03f88f9; no voters: 1c50de89a5a845668696dcb28c3badd6
I20260129 02:58:50.661110 20630 consensus_queue.cc:260] T 3906b080a5b34a74b8b64128635e8997 P 974b802141bd4f13beffef2aa03f88f9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:58:50.662029 21026 raft_consensus.cc:2804] T a4c1226b987b468fa84dfc6ba0e2dbe1 P 974b802141bd4f13beffef2aa03f88f9 [term 1 FOLLOWER]: Leader election won for term 1
I20260129 02:58:50.662560 21026 raft_consensus.cc:697] T a4c1226b987b468fa84dfc6ba0e2dbe1 P 974b802141bd4f13beffef2aa03f88f9 [term 1 LEADER]: Becoming Leader. State: Replica: 974b802141bd4f13beffef2aa03f88f9, State: Running, Role: LEADER
I20260129 02:58:50.663332 21026 consensus_queue.cc:237] T a4c1226b987b468fa84dfc6ba0e2dbe1 P 974b802141bd4f13beffef2aa03f88f9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:58:50.669811 20630 ts_tablet_manager.cc:1434] T 3906b080a5b34a74b8b64128635e8997 P 974b802141bd4f13beffef2aa03f88f9: Time spent starting tablet: real 0.021s user 0.004s sys 0.004s
I20260129 02:58:50.678637 20633 tablet_bootstrap.cc:492] T 4e3805ffa70242e8ade66da260dddf66 P 197b880fa39d432a9b472845876bfaf0: No bootstrap required, opened a new log
I20260129 02:58:50.679075 20633 ts_tablet_manager.cc:1403] T 4e3805ffa70242e8ade66da260dddf66 P 197b880fa39d432a9b472845876bfaf0: Time spent bootstrapping tablet: real 0.093s user 0.014s sys 0.012s
I20260129 02:58:50.681867 20633 raft_consensus.cc:359] T 4e3805ffa70242e8ade66da260dddf66 P 197b880fa39d432a9b472845876bfaf0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:58:50.682767 20633 raft_consensus.cc:385] T 4e3805ffa70242e8ade66da260dddf66 P 197b880fa39d432a9b472845876bfaf0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260129 02:58:50.683115 20633 raft_consensus.cc:740] T 4e3805ffa70242e8ade66da260dddf66 P 197b880fa39d432a9b472845876bfaf0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 197b880fa39d432a9b472845876bfaf0, State: Initialized, Role: FOLLOWER
I20260129 02:58:50.684005 20633 consensus_queue.cc:260] T 4e3805ffa70242e8ade66da260dddf66 P 197b880fa39d432a9b472845876bfaf0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:58:50.691470 20335 catalog_manager.cc:5654] T a4c1226b987b468fa84dfc6ba0e2dbe1 P 974b802141bd4f13beffef2aa03f88f9 reported cstate change: term changed from 0 to 1, leader changed from <none> to 974b802141bd4f13beffef2aa03f88f9 (127.18.211.2). New cstate: current_term: 1 leader_uuid: "974b802141bd4f13beffef2aa03f88f9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } health_report { overall_health: UNKNOWN } } }
I20260129 02:58:50.696278 20631 tablet_bootstrap.cc:492] T 0678b8b468bd43cdbc3e485900844821 P 1c50de89a5a845668696dcb28c3badd6: No bootstrap required, opened a new log
I20260129 02:58:50.697221 20631 ts_tablet_manager.cc:1403] T 0678b8b468bd43cdbc3e485900844821 P 1c50de89a5a845668696dcb28c3badd6: Time spent bootstrapping tablet: real 0.113s user 0.013s sys 0.021s
I20260129 02:58:50.700790 20631 raft_consensus.cc:359] T 0678b8b468bd43cdbc3e485900844821 P 1c50de89a5a845668696dcb28c3badd6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:58:50.701514 20631 raft_consensus.cc:385] T 0678b8b468bd43cdbc3e485900844821 P 1c50de89a5a845668696dcb28c3badd6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260129 02:58:50.701762 20631 raft_consensus.cc:740] T 0678b8b468bd43cdbc3e485900844821 P 1c50de89a5a845668696dcb28c3badd6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1c50de89a5a845668696dcb28c3badd6, State: Initialized, Role: FOLLOWER
I20260129 02:58:50.702363 20631 consensus_queue.cc:260] T 0678b8b468bd43cdbc3e485900844821 P 1c50de89a5a845668696dcb28c3badd6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:58:50.706980 20633 ts_tablet_manager.cc:1434] T 4e3805ffa70242e8ade66da260dddf66 P 197b880fa39d432a9b472845876bfaf0: Time spent starting tablet: real 0.028s user 0.008s sys 0.000s
I20260129 02:58:50.706336 20631 ts_tablet_manager.cc:1434] T 0678b8b468bd43cdbc3e485900844821 P 1c50de89a5a845668696dcb28c3badd6: Time spent starting tablet: real 0.009s user 0.006s sys 0.000s
I20260129 02:58:50.708933 20631 tablet_bootstrap.cc:492] T 4e3805ffa70242e8ade66da260dddf66 P 1c50de89a5a845668696dcb28c3badd6: Bootstrap starting.
I20260129 02:58:50.716673 20631 tablet_bootstrap.cc:654] T 4e3805ffa70242e8ade66da260dddf66 P 1c50de89a5a845668696dcb28c3badd6: Neither blocks nor log segments found. Creating new log.
I20260129 02:58:50.725095 20631 tablet_bootstrap.cc:492] T 4e3805ffa70242e8ade66da260dddf66 P 1c50de89a5a845668696dcb28c3badd6: No bootstrap required, opened a new log
I20260129 02:58:50.725544 20631 ts_tablet_manager.cc:1403] T 4e3805ffa70242e8ade66da260dddf66 P 1c50de89a5a845668696dcb28c3badd6: Time spent bootstrapping tablet: real 0.017s user 0.005s sys 0.005s
I20260129 02:58:50.727953 20631 raft_consensus.cc:359] T 4e3805ffa70242e8ade66da260dddf66 P 1c50de89a5a845668696dcb28c3badd6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:58:50.728618 20631 raft_consensus.cc:385] T 4e3805ffa70242e8ade66da260dddf66 P 1c50de89a5a845668696dcb28c3badd6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260129 02:58:50.728940 20631 raft_consensus.cc:740] T 4e3805ffa70242e8ade66da260dddf66 P 1c50de89a5a845668696dcb28c3badd6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1c50de89a5a845668696dcb28c3badd6, State: Initialized, Role: FOLLOWER
I20260129 02:58:50.729555 20631 consensus_queue.cc:260] T 4e3805ffa70242e8ade66da260dddf66 P 1c50de89a5a845668696dcb28c3badd6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:58:50.731623 20631 ts_tablet_manager.cc:1434] T 4e3805ffa70242e8ade66da260dddf66 P 1c50de89a5a845668696dcb28c3badd6: Time spent starting tablet: real 0.006s user 0.006s sys 0.001s
I20260129 02:58:50.732477 20631 tablet_bootstrap.cc:492] T 7d9494f5584d4d5db8b9adf1c8fc888b P 1c50de89a5a845668696dcb28c3badd6: Bootstrap starting.
I20260129 02:58:50.736981 20584 raft_consensus.cc:3060] T 42d27a28d89b409e8807ec0efd97af69 P 1c50de89a5a845668696dcb28c3badd6 [term 0 FOLLOWER]: Advancing to term 1
I20260129 02:58:50.737972 20631 tablet_bootstrap.cc:654] T 7d9494f5584d4d5db8b9adf1c8fc888b P 1c50de89a5a845668696dcb28c3badd6: Neither blocks nor log segments found. Creating new log.
I20260129 02:58:50.743649 21031 raft_consensus.cc:493] T 3ccac82a01a2445fa5b0ce6c0fcf2295 P 1c50de89a5a845668696dcb28c3badd6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260129 02:58:50.744360 21031 raft_consensus.cc:515] T 3ccac82a01a2445fa5b0ce6c0fcf2295 P 1c50de89a5a845668696dcb28c3badd6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:58:50.746599 20631 tablet_bootstrap.cc:492] T 7d9494f5584d4d5db8b9adf1c8fc888b P 1c50de89a5a845668696dcb28c3badd6: No bootstrap required, opened a new log
I20260129 02:58:50.746856 21031 leader_election.cc:290] T 3ccac82a01a2445fa5b0ce6c0fcf2295 P 1c50de89a5a845668696dcb28c3badd6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 197b880fa39d432a9b472845876bfaf0 (127.18.211.1:42027), 974b802141bd4f13beffef2aa03f88f9 (127.18.211.2:34517)
I20260129 02:58:50.747087 20631 ts_tablet_manager.cc:1403] T 7d9494f5584d4d5db8b9adf1c8fc888b P 1c50de89a5a845668696dcb28c3badd6: Time spent bootstrapping tablet: real 0.015s user 0.010s sys 0.004s
I20260129 02:58:50.747462 20438 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3ccac82a01a2445fa5b0ce6c0fcf2295" candidate_uuid: "1c50de89a5a845668696dcb28c3badd6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "197b880fa39d432a9b472845876bfaf0" is_pre_election: true
I20260129 02:58:50.748488 20514 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3ccac82a01a2445fa5b0ce6c0fcf2295" candidate_uuid: "1c50de89a5a845668696dcb28c3badd6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "974b802141bd4f13beffef2aa03f88f9" is_pre_election: true
I20260129 02:58:50.749293 20514 raft_consensus.cc:2393] T 3ccac82a01a2445fa5b0ce6c0fcf2295 P 974b802141bd4f13beffef2aa03f88f9 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1c50de89a5a845668696dcb28c3badd6 in current term 1: Already voted for candidate 197b880fa39d432a9b472845876bfaf0 in this term.
I20260129 02:58:50.749984 20631 raft_consensus.cc:359] T 7d9494f5584d4d5db8b9adf1c8fc888b P 1c50de89a5a845668696dcb28c3badd6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:58:50.750622 20631 raft_consensus.cc:385] T 7d9494f5584d4d5db8b9adf1c8fc888b P 1c50de89a5a845668696dcb28c3badd6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260129 02:58:50.750857 20631 raft_consensus.cc:740] T 7d9494f5584d4d5db8b9adf1c8fc888b P 1c50de89a5a845668696dcb28c3badd6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1c50de89a5a845668696dcb28c3badd6, State: Initialized, Role: FOLLOWER
I20260129 02:58:50.751499 20631 consensus_queue.cc:260] T 7d9494f5584d4d5db8b9adf1c8fc888b P 1c50de89a5a845668696dcb28c3badd6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:58:50.752655 20553 leader_election.cc:304] T 3ccac82a01a2445fa5b0ce6c0fcf2295 P 1c50de89a5a845668696dcb28c3badd6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1c50de89a5a845668696dcb28c3badd6; no voters: 197b880fa39d432a9b472845876bfaf0, 974b802141bd4f13beffef2aa03f88f9
I20260129 02:58:50.753777 20631 ts_tablet_manager.cc:1434] T 7d9494f5584d4d5db8b9adf1c8fc888b P 1c50de89a5a845668696dcb28c3badd6: Time spent starting tablet: real 0.006s user 0.004s sys 0.000s
I20260129 02:58:50.753926 20832 raft_consensus.cc:3060] T 3ccac82a01a2445fa5b0ce6c0fcf2295 P 1c50de89a5a845668696dcb28c3badd6 [term 0 FOLLOWER]: Advancing to term 1
I20260129 02:58:50.754870 20631 tablet_bootstrap.cc:492] T 4f5a77e94fef4b16a2ba7b067c8b144a P 1c50de89a5a845668696dcb28c3badd6: Bootstrap starting.
I20260129 02:58:50.761251 20631 tablet_bootstrap.cc:654] T 4f5a77e94fef4b16a2ba7b067c8b144a P 1c50de89a5a845668696dcb28c3badd6: Neither blocks nor log segments found. Creating new log.
I20260129 02:58:50.762634 20584 raft_consensus.cc:3060] T 8055232cf0004d40a85f3bf983fe00a9 P 1c50de89a5a845668696dcb28c3badd6 [term 0 FOLLOWER]: Advancing to term 1
I20260129 02:58:50.769830 20587 raft_consensus.cc:3060] T 0678b8b468bd43cdbc3e485900844821 P 1c50de89a5a845668696dcb28c3badd6 [term 0 FOLLOWER]: Advancing to term 1
I20260129 02:58:50.774395 20983 consensus_queue.cc:1048] T 8055232cf0004d40a85f3bf983fe00a9 P 974b802141bd4f13beffef2aa03f88f9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260129 02:58:50.776782 20983 consensus_queue.cc:1048] T 8055232cf0004d40a85f3bf983fe00a9 P 974b802141bd4f13beffef2aa03f88f9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.005s
I20260129 02:58:50.783309 21032 raft_consensus.cc:493] T 8055232cf0004d40a85f3bf983fe00a9 P 1c50de89a5a845668696dcb28c3badd6 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260129 02:58:50.783978 21032 raft_consensus.cc:515] T 8055232cf0004d40a85f3bf983fe00a9 P 1c50de89a5a845668696dcb28c3badd6 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:58:50.789932 20516 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "8055232cf0004d40a85f3bf983fe00a9" candidate_uuid: "1c50de89a5a845668696dcb28c3badd6" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "974b802141bd4f13beffef2aa03f88f9" is_pre_election: true
I20260129 02:58:50.787834 21032 leader_election.cc:290] T 8055232cf0004d40a85f3bf983fe00a9 P 1c50de89a5a845668696dcb28c3badd6 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 197b880fa39d432a9b472845876bfaf0 (127.18.211.1:42027), 974b802141bd4f13beffef2aa03f88f9 (127.18.211.2:34517)
I20260129 02:58:50.795620 20440 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "8055232cf0004d40a85f3bf983fe00a9" candidate_uuid: "1c50de89a5a845668696dcb28c3badd6" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "197b880fa39d432a9b472845876bfaf0" is_pre_election: true
I20260129 02:58:50.799297 20553 leader_election.cc:304] T 8055232cf0004d40a85f3bf983fe00a9 P 1c50de89a5a845668696dcb28c3badd6 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1c50de89a5a845668696dcb28c3badd6; no voters: 197b880fa39d432a9b472845876bfaf0, 974b802141bd4f13beffef2aa03f88f9
I20260129 02:58:50.816380 20832 raft_consensus.cc:2749] T 3ccac82a01a2445fa5b0ce6c0fcf2295 P 1c50de89a5a845668696dcb28c3badd6 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260129 02:58:50.821817 20631 tablet_bootstrap.cc:492] T 4f5a77e94fef4b16a2ba7b067c8b144a P 1c50de89a5a845668696dcb28c3badd6: No bootstrap required, opened a new log
I20260129 02:58:50.822292 20631 ts_tablet_manager.cc:1403] T 4f5a77e94fef4b16a2ba7b067c8b144a P 1c50de89a5a845668696dcb28c3badd6: Time spent bootstrapping tablet: real 0.068s user 0.027s sys 0.031s
I20260129 02:58:50.824688 20631 raft_consensus.cc:359] T 4f5a77e94fef4b16a2ba7b067c8b144a P 1c50de89a5a845668696dcb28c3badd6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:58:50.825315 20631 raft_consensus.cc:385] T 4f5a77e94fef4b16a2ba7b067c8b144a P 1c50de89a5a845668696dcb28c3badd6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260129 02:58:50.825753 20631 raft_consensus.cc:740] T 4f5a77e94fef4b16a2ba7b067c8b144a P 1c50de89a5a845668696dcb28c3badd6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1c50de89a5a845668696dcb28c3badd6, State: Initialized, Role: FOLLOWER
I20260129 02:58:50.826323 20631 consensus_queue.cc:260] T 4f5a77e94fef4b16a2ba7b067c8b144a P 1c50de89a5a845668696dcb28c3badd6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:58:50.828567 20631 ts_tablet_manager.cc:1434] T 4f5a77e94fef4b16a2ba7b067c8b144a P 1c50de89a5a845668696dcb28c3badd6: Time spent starting tablet: real 0.006s user 0.006s sys 0.001s
I20260129 02:58:50.829583 20631 tablet_bootstrap.cc:492] T 00e79d8f25a34676b1d866eb2068a2b2 P 1c50de89a5a845668696dcb28c3badd6: Bootstrap starting.
I20260129 02:58:50.838740 20631 tablet_bootstrap.cc:654] T 00e79d8f25a34676b1d866eb2068a2b2 P 1c50de89a5a845668696dcb28c3badd6: Neither blocks nor log segments found. Creating new log.
I20260129 02:58:50.840063 21031 raft_consensus.cc:2749] T 8055232cf0004d40a85f3bf983fe00a9 P 1c50de89a5a845668696dcb28c3badd6 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260129 02:58:50.846226 20631 tablet_bootstrap.cc:492] T 00e79d8f25a34676b1d866eb2068a2b2 P 1c50de89a5a845668696dcb28c3badd6: No bootstrap required, opened a new log
I20260129 02:58:50.846728 20631 ts_tablet_manager.cc:1403] T 00e79d8f25a34676b1d866eb2068a2b2 P 1c50de89a5a845668696dcb28c3badd6: Time spent bootstrapping tablet: real 0.017s user 0.007s sys 0.005s
I20260129 02:58:50.849315 20631 raft_consensus.cc:359] T 00e79d8f25a34676b1d866eb2068a2b2 P 1c50de89a5a845668696dcb28c3badd6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:58:50.850003 20631 raft_consensus.cc:385] T 00e79d8f25a34676b1d866eb2068a2b2 P 1c50de89a5a845668696dcb28c3badd6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260129 02:58:50.850282 20631 raft_consensus.cc:740] T 00e79d8f25a34676b1d866eb2068a2b2 P 1c50de89a5a845668696dcb28c3badd6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1c50de89a5a845668696dcb28c3badd6, State: Initialized, Role: FOLLOWER
I20260129 02:58:50.850899 20631 consensus_queue.cc:260] T 00e79d8f25a34676b1d866eb2068a2b2 P 1c50de89a5a845668696dcb28c3badd6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:58:50.853073 20631 ts_tablet_manager.cc:1434] T 00e79d8f25a34676b1d866eb2068a2b2 P 1c50de89a5a845668696dcb28c3badd6: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20260129 02:58:50.853953 20631 tablet_bootstrap.cc:492] T 3906b080a5b34a74b8b64128635e8997 P 1c50de89a5a845668696dcb28c3badd6: Bootstrap starting.
I20260129 02:58:50.856292 20965 consensus_queue.cc:1048] T 4f5a77e94fef4b16a2ba7b067c8b144a P 974b802141bd4f13beffef2aa03f88f9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260129 02:58:50.859920 20631 tablet_bootstrap.cc:654] T 3906b080a5b34a74b8b64128635e8997 P 1c50de89a5a845668696dcb28c3badd6: Neither blocks nor log segments found. Creating new log.
I20260129 02:58:50.868764 20631 tablet_bootstrap.cc:492] T 3906b080a5b34a74b8b64128635e8997 P 1c50de89a5a845668696dcb28c3badd6: No bootstrap required, opened a new log
I20260129 02:58:50.869167 20631 ts_tablet_manager.cc:1403] T 3906b080a5b34a74b8b64128635e8997 P 1c50de89a5a845668696dcb28c3badd6: Time spent bootstrapping tablet: real 0.015s user 0.008s sys 0.003s
I20260129 02:58:50.872864 20631 raft_consensus.cc:359] T 3906b080a5b34a74b8b64128635e8997 P 1c50de89a5a845668696dcb28c3badd6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:58:50.873530 20631 raft_consensus.cc:385] T 3906b080a5b34a74b8b64128635e8997 P 1c50de89a5a845668696dcb28c3badd6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260129 02:58:50.873809 20631 raft_consensus.cc:740] T 3906b080a5b34a74b8b64128635e8997 P 1c50de89a5a845668696dcb28c3badd6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1c50de89a5a845668696dcb28c3badd6, State: Initialized, Role: FOLLOWER
I20260129 02:58:50.875056 20585 raft_consensus.cc:3060] T 4f5a77e94fef4b16a2ba7b067c8b144a P 1c50de89a5a845668696dcb28c3badd6 [term 0 FOLLOWER]: Advancing to term 1
I20260129 02:58:50.883220 20631 consensus_queue.cc:260] T 3906b080a5b34a74b8b64128635e8997 P 1c50de89a5a845668696dcb28c3badd6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:58:50.885638 20631 ts_tablet_manager.cc:1434] T 3906b080a5b34a74b8b64128635e8997 P 1c50de89a5a845668696dcb28c3badd6: Time spent starting tablet: real 0.016s user 0.006s sys 0.000s
I20260129 02:58:50.886200 21026 consensus_queue.cc:1048] T 4f5a77e94fef4b16a2ba7b067c8b144a P 974b802141bd4f13beffef2aa03f88f9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260129 02:58:50.886540 20631 tablet_bootstrap.cc:492] T a4c1226b987b468fa84dfc6ba0e2dbe1 P 1c50de89a5a845668696dcb28c3badd6: Bootstrap starting.
I20260129 02:58:50.899237 21005 raft_consensus.cc:493] T 3906b080a5b34a74b8b64128635e8997 P 197b880fa39d432a9b472845876bfaf0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260129 02:58:50.899837 21005 raft_consensus.cc:515] T 3906b080a5b34a74b8b64128635e8997 P 197b880fa39d432a9b472845876bfaf0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:58:50.900550 20631 tablet_bootstrap.cc:654] T a4c1226b987b468fa84dfc6ba0e2dbe1 P 1c50de89a5a845668696dcb28c3badd6: Neither blocks nor log segments found. Creating new log.
I20260129 02:58:50.902946 20516 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3906b080a5b34a74b8b64128635e8997" candidate_uuid: "197b880fa39d432a9b472845876bfaf0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "974b802141bd4f13beffef2aa03f88f9" is_pre_election: true
I20260129 02:58:50.903056 21005 leader_election.cc:290] T 3906b080a5b34a74b8b64128635e8997 P 197b880fa39d432a9b472845876bfaf0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 974b802141bd4f13beffef2aa03f88f9 (127.18.211.2:34517), 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3:37735)
I20260129 02:58:50.903560 20516 raft_consensus.cc:2468] T 3906b080a5b34a74b8b64128635e8997 P 974b802141bd4f13beffef2aa03f88f9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 197b880fa39d432a9b472845876bfaf0 in term 0.
I20260129 02:58:50.904258 20587 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3906b080a5b34a74b8b64128635e8997" candidate_uuid: "197b880fa39d432a9b472845876bfaf0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c50de89a5a845668696dcb28c3badd6" is_pre_election: true
I20260129 02:58:50.904791 20407 leader_election.cc:304] T 3906b080a5b34a74b8b64128635e8997 P 197b880fa39d432a9b472845876bfaf0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 197b880fa39d432a9b472845876bfaf0, 974b802141bd4f13beffef2aa03f88f9; no voters:
I20260129 02:58:50.904958 20587 raft_consensus.cc:2468] T 3906b080a5b34a74b8b64128635e8997 P 1c50de89a5a845668696dcb28c3badd6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 197b880fa39d432a9b472845876bfaf0 in term 0.
I20260129 02:58:50.906396 21027 raft_consensus.cc:2804] T 3906b080a5b34a74b8b64128635e8997 P 197b880fa39d432a9b472845876bfaf0 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260129 02:58:50.909752 21027 raft_consensus.cc:493] T 3906b080a5b34a74b8b64128635e8997 P 197b880fa39d432a9b472845876bfaf0 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260129 02:58:50.910113 21027 raft_consensus.cc:3060] T 3906b080a5b34a74b8b64128635e8997 P 197b880fa39d432a9b472845876bfaf0 [term 0 FOLLOWER]: Advancing to term 1
I20260129 02:58:50.911437 20631 tablet_bootstrap.cc:492] T a4c1226b987b468fa84dfc6ba0e2dbe1 P 1c50de89a5a845668696dcb28c3badd6: No bootstrap required, opened a new log
I20260129 02:58:50.911768 20631 ts_tablet_manager.cc:1403] T a4c1226b987b468fa84dfc6ba0e2dbe1 P 1c50de89a5a845668696dcb28c3badd6: Time spent bootstrapping tablet: real 0.025s user 0.009s sys 0.000s
I20260129 02:58:50.914291 20631 raft_consensus.cc:359] T a4c1226b987b468fa84dfc6ba0e2dbe1 P 1c50de89a5a845668696dcb28c3badd6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:58:50.914872 20631 raft_consensus.cc:385] T a4c1226b987b468fa84dfc6ba0e2dbe1 P 1c50de89a5a845668696dcb28c3badd6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260129 02:58:50.915059 20631 raft_consensus.cc:740] T a4c1226b987b468fa84dfc6ba0e2dbe1 P 1c50de89a5a845668696dcb28c3badd6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1c50de89a5a845668696dcb28c3badd6, State: Initialized, Role: FOLLOWER
I20260129 02:58:50.915504 20631 consensus_queue.cc:260] T a4c1226b987b468fa84dfc6ba0e2dbe1 P 1c50de89a5a845668696dcb28c3badd6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:58:50.917452 20631 ts_tablet_manager.cc:1434] T a4c1226b987b468fa84dfc6ba0e2dbe1 P 1c50de89a5a845668696dcb28c3badd6: Time spent starting tablet: real 0.005s user 0.002s sys 0.002s
I20260129 02:58:50.917078 21027 raft_consensus.cc:515] T 3906b080a5b34a74b8b64128635e8997 P 197b880fa39d432a9b472845876bfaf0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:58:50.918398 20631 tablet_bootstrap.cc:492] T a8479d1fe0bd41e5946186e3a6ca8c7a P 1c50de89a5a845668696dcb28c3badd6: Bootstrap starting.
I20260129 02:58:50.920593 21027 leader_election.cc:290] T 3906b080a5b34a74b8b64128635e8997 P 197b880fa39d432a9b472845876bfaf0 [CANDIDATE]: Term 1 election: Requested vote from peers 974b802141bd4f13beffef2aa03f88f9 (127.18.211.2:34517), 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3:37735)
I20260129 02:58:50.921329 20516 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3906b080a5b34a74b8b64128635e8997" candidate_uuid: "197b880fa39d432a9b472845876bfaf0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "974b802141bd4f13beffef2aa03f88f9"
I20260129 02:58:50.921826 20516 raft_consensus.cc:3060] T 3906b080a5b34a74b8b64128635e8997 P 974b802141bd4f13beffef2aa03f88f9 [term 0 FOLLOWER]: Advancing to term 1
I20260129 02:58:50.924157 20585 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3906b080a5b34a74b8b64128635e8997" candidate_uuid: "197b880fa39d432a9b472845876bfaf0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c50de89a5a845668696dcb28c3badd6"
I20260129 02:58:50.924634 20587 raft_consensus.cc:3060] T 00e79d8f25a34676b1d866eb2068a2b2 P 1c50de89a5a845668696dcb28c3badd6 [term 0 FOLLOWER]: Advancing to term 1
I20260129 02:58:50.924721 20585 raft_consensus.cc:3060] T 3906b080a5b34a74b8b64128635e8997 P 1c50de89a5a845668696dcb28c3badd6 [term 0 FOLLOWER]: Advancing to term 1
I20260129 02:58:50.930341 20516 raft_consensus.cc:2468] T 3906b080a5b34a74b8b64128635e8997 P 974b802141bd4f13beffef2aa03f88f9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 197b880fa39d432a9b472845876bfaf0 in term 1.
I20260129 02:58:50.930498 20631 tablet_bootstrap.cc:654] T a8479d1fe0bd41e5946186e3a6ca8c7a P 1c50de89a5a845668696dcb28c3badd6: Neither blocks nor log segments found. Creating new log.
I20260129 02:58:50.931615 20407 leader_election.cc:304] T 3906b080a5b34a74b8b64128635e8997 P 197b880fa39d432a9b472845876bfaf0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 197b880fa39d432a9b472845876bfaf0, 974b802141bd4f13beffef2aa03f88f9; no voters:
I20260129 02:58:50.931705 20585 raft_consensus.cc:2468] T 3906b080a5b34a74b8b64128635e8997 P 1c50de89a5a845668696dcb28c3badd6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 197b880fa39d432a9b472845876bfaf0 in term 1.
I20260129 02:58:50.933892 21027 raft_consensus.cc:2804] T 3906b080a5b34a74b8b64128635e8997 P 197b880fa39d432a9b472845876bfaf0 [term 1 FOLLOWER]: Leader election won for term 1
I20260129 02:58:50.934403 21004 consensus_queue.cc:1048] T 00e79d8f25a34676b1d866eb2068a2b2 P 197b880fa39d432a9b472845876bfaf0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260129 02:58:50.936151 21027 raft_consensus.cc:697] T 3906b080a5b34a74b8b64128635e8997 P 197b880fa39d432a9b472845876bfaf0 [term 1 LEADER]: Becoming Leader. State: Replica: 197b880fa39d432a9b472845876bfaf0, State: Running, Role: LEADER
I20260129 02:58:50.936933 21027 consensus_queue.cc:237] T 3906b080a5b34a74b8b64128635e8997 P 197b880fa39d432a9b472845876bfaf0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:58:50.942041 20631 tablet_bootstrap.cc:492] T a8479d1fe0bd41e5946186e3a6ca8c7a P 1c50de89a5a845668696dcb28c3badd6: No bootstrap required, opened a new log
I20260129 02:58:50.942510 20631 ts_tablet_manager.cc:1403] T a8479d1fe0bd41e5946186e3a6ca8c7a P 1c50de89a5a845668696dcb28c3badd6: Time spent bootstrapping tablet: real 0.024s user 0.009s sys 0.001s
I20260129 02:58:50.945128 20631 raft_consensus.cc:359] T a8479d1fe0bd41e5946186e3a6ca8c7a P 1c50de89a5a845668696dcb28c3badd6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:58:50.945613 20631 raft_consensus.cc:385] T a8479d1fe0bd41e5946186e3a6ca8c7a P 1c50de89a5a845668696dcb28c3badd6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260129 02:58:50.945834 20631 raft_consensus.cc:740] T a8479d1fe0bd41e5946186e3a6ca8c7a P 1c50de89a5a845668696dcb28c3badd6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1c50de89a5a845668696dcb28c3badd6, State: Initialized, Role: FOLLOWER
I20260129 02:58:50.946419 20631 consensus_queue.cc:260] T a8479d1fe0bd41e5946186e3a6ca8c7a P 1c50de89a5a845668696dcb28c3badd6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:58:50.948570 20631 ts_tablet_manager.cc:1434] T a8479d1fe0bd41e5946186e3a6ca8c7a P 1c50de89a5a845668696dcb28c3badd6: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260129 02:58:50.949043 20586 raft_consensus.cc:3060] T 7d9494f5584d4d5db8b9adf1c8fc888b P 1c50de89a5a845668696dcb28c3badd6 [term 0 FOLLOWER]: Advancing to term 1
I20260129 02:58:50.956950 21004 raft_consensus.cc:493] T 4e3805ffa70242e8ade66da260dddf66 P 197b880fa39d432a9b472845876bfaf0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260129 02:58:50.957329 21030 consensus_queue.cc:1048] T 7d9494f5584d4d5db8b9adf1c8fc888b P 974b802141bd4f13beffef2aa03f88f9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260129 02:58:50.957625 21004 raft_consensus.cc:515] T 4e3805ffa70242e8ade66da260dddf66 P 197b880fa39d432a9b472845876bfaf0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:58:50.967320 21005 raft_consensus.cc:493] T a8479d1fe0bd41e5946186e3a6ca8c7a P 197b880fa39d432a9b472845876bfaf0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260129 02:58:50.967942 21005 raft_consensus.cc:515] T a8479d1fe0bd41e5946186e3a6ca8c7a P 197b880fa39d432a9b472845876bfaf0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:58:50.971045 20516 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4e3805ffa70242e8ade66da260dddf66" candidate_uuid: "197b880fa39d432a9b472845876bfaf0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "974b802141bd4f13beffef2aa03f88f9" is_pre_election: true
I20260129 02:58:50.971655 20516 raft_consensus.cc:2468] T 4e3805ffa70242e8ade66da260dddf66 P 974b802141bd4f13beffef2aa03f88f9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 197b880fa39d432a9b472845876bfaf0 in term 0.
I20260129 02:58:50.972791 20407 leader_election.cc:304] T 4e3805ffa70242e8ade66da260dddf66 P 197b880fa39d432a9b472845876bfaf0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 197b880fa39d432a9b472845876bfaf0, 974b802141bd4f13beffef2aa03f88f9; no voters:
I20260129 02:58:50.973678 21024 raft_consensus.cc:2804] T 4e3805ffa70242e8ade66da260dddf66 P 197b880fa39d432a9b472845876bfaf0 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260129 02:58:50.974020 21024 raft_consensus.cc:493] T 4e3805ffa70242e8ade66da260dddf66 P 197b880fa39d432a9b472845876bfaf0 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260129 02:58:50.974328 21024 raft_consensus.cc:3060] T 4e3805ffa70242e8ade66da260dddf66 P 197b880fa39d432a9b472845876bfaf0 [term 0 FOLLOWER]: Advancing to term 1
I20260129 02:58:50.977622 20516 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a8479d1fe0bd41e5946186e3a6ca8c7a" candidate_uuid: "197b880fa39d432a9b472845876bfaf0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "974b802141bd4f13beffef2aa03f88f9" is_pre_election: true
I20260129 02:58:50.977972 21005 leader_election.cc:290] T a8479d1fe0bd41e5946186e3a6ca8c7a P 197b880fa39d432a9b472845876bfaf0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 974b802141bd4f13beffef2aa03f88f9 (127.18.211.2:34517), 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3:37735)
I20260129 02:58:50.978664 20516 raft_consensus.cc:2468] T a8479d1fe0bd41e5946186e3a6ca8c7a P 974b802141bd4f13beffef2aa03f88f9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 197b880fa39d432a9b472845876bfaf0 in term 0.
I20260129 02:58:50.979117 21032 raft_consensus.cc:493] T a8479d1fe0bd41e5946186e3a6ca8c7a P 1c50de89a5a845668696dcb28c3badd6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260129 02:58:50.979523 20407 leader_election.cc:304] T a8479d1fe0bd41e5946186e3a6ca8c7a P 197b880fa39d432a9b472845876bfaf0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 197b880fa39d432a9b472845876bfaf0, 974b802141bd4f13beffef2aa03f88f9; no voters:
I20260129 02:58:50.980199 20338 catalog_manager.cc:5654] T 3906b080a5b34a74b8b64128635e8997 P 197b880fa39d432a9b472845876bfaf0 reported cstate change: term changed from 0 to 1, leader changed from <none> to 197b880fa39d432a9b472845876bfaf0 (127.18.211.1). New cstate: current_term: 1 leader_uuid: "197b880fa39d432a9b472845876bfaf0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } health_report { overall_health: UNKNOWN } } }
I20260129 02:58:50.980201 20587 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a8479d1fe0bd41e5946186e3a6ca8c7a" candidate_uuid: "197b880fa39d432a9b472845876bfaf0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c50de89a5a845668696dcb28c3badd6" is_pre_election: true
I20260129 02:58:50.981068 21032 raft_consensus.cc:515] T a8479d1fe0bd41e5946186e3a6ca8c7a P 1c50de89a5a845668696dcb28c3badd6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:58:50.984088 21032 leader_election.cc:290] T a8479d1fe0bd41e5946186e3a6ca8c7a P 1c50de89a5a845668696dcb28c3badd6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 197b880fa39d432a9b472845876bfaf0 (127.18.211.1:42027), 974b802141bd4f13beffef2aa03f88f9 (127.18.211.2:34517)
I20260129 02:58:50.981709 21027 raft_consensus.cc:2804] T a8479d1fe0bd41e5946186e3a6ca8c7a P 197b880fa39d432a9b472845876bfaf0 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260129 02:58:50.984328 20587 raft_consensus.cc:2468] T a8479d1fe0bd41e5946186e3a6ca8c7a P 1c50de89a5a845668696dcb28c3badd6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 197b880fa39d432a9b472845876bfaf0 in term 0.
I20260129 02:58:50.985440 21027 raft_consensus.cc:493] T a8479d1fe0bd41e5946186e3a6ca8c7a P 197b880fa39d432a9b472845876bfaf0 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260129 02:58:50.985992 21027 raft_consensus.cc:3060] T a8479d1fe0bd41e5946186e3a6ca8c7a P 197b880fa39d432a9b472845876bfaf0 [term 0 FOLLOWER]: Advancing to term 1
I20260129 02:58:50.985483 20438 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a8479d1fe0bd41e5946186e3a6ca8c7a" candidate_uuid: "1c50de89a5a845668696dcb28c3badd6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "197b880fa39d432a9b472845876bfaf0" is_pre_election: true
I20260129 02:58:50.984812 21004 leader_election.cc:290] T 4e3805ffa70242e8ade66da260dddf66 P 197b880fa39d432a9b472845876bfaf0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 974b802141bd4f13beffef2aa03f88f9 (127.18.211.2:34517), 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3:37735)
I20260129 02:58:50.993055 21004 consensus_queue.cc:1048] T 00e79d8f25a34676b1d866eb2068a2b2 P 197b880fa39d432a9b472845876bfaf0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260129 02:58:50.995302 21027 raft_consensus.cc:515] T a8479d1fe0bd41e5946186e3a6ca8c7a P 197b880fa39d432a9b472845876bfaf0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:58:50.999977 21027 leader_election.cc:290] T a8479d1fe0bd41e5946186e3a6ca8c7a P 197b880fa39d432a9b472845876bfaf0 [CANDIDATE]: Term 1 election: Requested vote from peers 974b802141bd4f13beffef2aa03f88f9 (127.18.211.2:34517), 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3:37735)
I20260129 02:58:51.000232 20438 raft_consensus.cc:2393] T a8479d1fe0bd41e5946186e3a6ca8c7a P 197b880fa39d432a9b472845876bfaf0 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1c50de89a5a845668696dcb28c3badd6 in current term 1: Already voted for candidate 197b880fa39d432a9b472845876bfaf0 in this term.
I20260129 02:58:51.001319 20510 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a8479d1fe0bd41e5946186e3a6ca8c7a" candidate_uuid: "197b880fa39d432a9b472845876bfaf0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "974b802141bd4f13beffef2aa03f88f9"
I20260129 02:58:51.001926 20510 raft_consensus.cc:3060] T a8479d1fe0bd41e5946186e3a6ca8c7a P 974b802141bd4f13beffef2aa03f88f9 [term 0 FOLLOWER]: Advancing to term 1
I20260129 02:58:50.986840 20516 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a8479d1fe0bd41e5946186e3a6ca8c7a" candidate_uuid: "1c50de89a5a845668696dcb28c3badd6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "974b802141bd4f13beffef2aa03f88f9" is_pre_election: true
I20260129 02:58:51.012458 20983 consensus_queue.cc:1048] T 7d9494f5584d4d5db8b9adf1c8fc888b P 974b802141bd4f13beffef2aa03f88f9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260129 02:58:51.015524 21024 raft_consensus.cc:515] T 4e3805ffa70242e8ade66da260dddf66 P 197b880fa39d432a9b472845876bfaf0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:58:51.015827 21005 consensus_queue.cc:1048] T 3ccac82a01a2445fa5b0ce6c0fcf2295 P 197b880fa39d432a9b472845876bfaf0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260129 02:58:51.016919 20584 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a8479d1fe0bd41e5946186e3a6ca8c7a" candidate_uuid: "197b880fa39d432a9b472845876bfaf0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c50de89a5a845668696dcb28c3badd6"
I20260129 02:58:51.017417 20510 raft_consensus.cc:2468] T a8479d1fe0bd41e5946186e3a6ca8c7a P 974b802141bd4f13beffef2aa03f88f9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 197b880fa39d432a9b472845876bfaf0 in term 1.
I20260129 02:58:51.017495 20584 raft_consensus.cc:3060] T a8479d1fe0bd41e5946186e3a6ca8c7a P 1c50de89a5a845668696dcb28c3badd6 [term 0 FOLLOWER]: Advancing to term 1
I20260129 02:58:51.019469 20556 leader_election.cc:304] T a8479d1fe0bd41e5946186e3a6ca8c7a P 1c50de89a5a845668696dcb28c3badd6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1c50de89a5a845668696dcb28c3badd6; no voters: 197b880fa39d432a9b472845876bfaf0, 974b802141bd4f13beffef2aa03f88f9
I20260129 02:58:51.022015 20585 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4e3805ffa70242e8ade66da260dddf66" candidate_uuid: "197b880fa39d432a9b472845876bfaf0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c50de89a5a845668696dcb28c3badd6" is_pre_election: true
I20260129 02:58:51.022672 21027 consensus_queue.cc:1048] T 3ccac82a01a2445fa5b0ce6c0fcf2295 P 197b880fa39d432a9b472845876bfaf0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260129 02:58:51.024992 20584 raft_consensus.cc:2468] T a8479d1fe0bd41e5946186e3a6ca8c7a P 1c50de89a5a845668696dcb28c3badd6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 197b880fa39d432a9b472845876bfaf0 in term 1.
I20260129 02:58:51.026425 20585 raft_consensus.cc:2468] T 4e3805ffa70242e8ade66da260dddf66 P 1c50de89a5a845668696dcb28c3badd6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 197b880fa39d432a9b472845876bfaf0 in term 0.
I20260129 02:58:51.026779 21032 raft_consensus.cc:2749] T a8479d1fe0bd41e5946186e3a6ca8c7a P 1c50de89a5a845668696dcb28c3badd6 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260129 02:58:51.027596 20407 leader_election.cc:304] T a8479d1fe0bd41e5946186e3a6ca8c7a P 197b880fa39d432a9b472845876bfaf0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 197b880fa39d432a9b472845876bfaf0, 974b802141bd4f13beffef2aa03f88f9; no voters:
I20260129 02:58:51.028810 20585 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4e3805ffa70242e8ade66da260dddf66" candidate_uuid: "197b880fa39d432a9b472845876bfaf0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c50de89a5a845668696dcb28c3badd6"
I20260129 02:58:51.029345 20585 raft_consensus.cc:3060] T 4e3805ffa70242e8ade66da260dddf66 P 1c50de89a5a845668696dcb28c3badd6 [term 0 FOLLOWER]: Advancing to term 1
I20260129 02:58:51.029969 20658 raft_consensus.cc:2804] T a8479d1fe0bd41e5946186e3a6ca8c7a P 197b880fa39d432a9b472845876bfaf0 [term 1 FOLLOWER]: Leader election won for term 1
I20260129 02:58:51.030411 20658 raft_consensus.cc:697] T a8479d1fe0bd41e5946186e3a6ca8c7a P 197b880fa39d432a9b472845876bfaf0 [term 1 LEADER]: Becoming Leader. State: Replica: 197b880fa39d432a9b472845876bfaf0, State: Running, Role: LEADER
I20260129 02:58:51.030571 20510 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4e3805ffa70242e8ade66da260dddf66" candidate_uuid: "197b880fa39d432a9b472845876bfaf0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "974b802141bd4f13beffef2aa03f88f9"
I20260129 02:58:51.031144 20510 raft_consensus.cc:3060] T 4e3805ffa70242e8ade66da260dddf66 P 974b802141bd4f13beffef2aa03f88f9 [term 0 FOLLOWER]: Advancing to term 1
I20260129 02:58:51.031113 20658 consensus_queue.cc:237] T a8479d1fe0bd41e5946186e3a6ca8c7a P 197b880fa39d432a9b472845876bfaf0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:58:51.025861 21024 leader_election.cc:290] T 4e3805ffa70242e8ade66da260dddf66 P 197b880fa39d432a9b472845876bfaf0 [CANDIDATE]: Term 1 election: Requested vote from peers 974b802141bd4f13beffef2aa03f88f9 (127.18.211.2:34517), 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3:37735)
I20260129 02:58:51.047286 20585 raft_consensus.cc:2468] T 4e3805ffa70242e8ade66da260dddf66 P 1c50de89a5a845668696dcb28c3badd6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 197b880fa39d432a9b472845876bfaf0 in term 1.
I20260129 02:58:51.048695 20405 leader_election.cc:304] T 4e3805ffa70242e8ade66da260dddf66 P 197b880fa39d432a9b472845876bfaf0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 197b880fa39d432a9b472845876bfaf0, 1c50de89a5a845668696dcb28c3badd6; no voters:
I20260129 02:58:51.050809 21027 raft_consensus.cc:2804] T 4e3805ffa70242e8ade66da260dddf66 P 197b880fa39d432a9b472845876bfaf0 [term 1 FOLLOWER]: Leader election won for term 1
I20260129 02:58:51.051322 21027 raft_consensus.cc:697] T 4e3805ffa70242e8ade66da260dddf66 P 197b880fa39d432a9b472845876bfaf0 [term 1 LEADER]: Becoming Leader. State: Replica: 197b880fa39d432a9b472845876bfaf0, State: Running, Role: LEADER
I20260129 02:58:51.052129 21027 consensus_queue.cc:237] T 4e3805ffa70242e8ade66da260dddf66 P 197b880fa39d432a9b472845876bfaf0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:58:51.108176 20336 catalog_manager.cc:5654] T a8479d1fe0bd41e5946186e3a6ca8c7a P 197b880fa39d432a9b472845876bfaf0 reported cstate change: term changed from 0 to 1, leader changed from <none> to 197b880fa39d432a9b472845876bfaf0 (127.18.211.1). New cstate: current_term: 1 leader_uuid: "197b880fa39d432a9b472845876bfaf0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } health_report { overall_health: UNKNOWN } } }
I20260129 02:58:51.109963 20336 catalog_manager.cc:5654] T 4e3805ffa70242e8ade66da260dddf66 P 197b880fa39d432a9b472845876bfaf0 reported cstate change: term changed from 0 to 1, leader changed from <none> to 197b880fa39d432a9b472845876bfaf0 (127.18.211.1). New cstate: current_term: 1 leader_uuid: "197b880fa39d432a9b472845876bfaf0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } health_report { overall_health: UNKNOWN } } }
I20260129 02:58:51.113447 20585 raft_consensus.cc:3060] T a4c1226b987b468fa84dfc6ba0e2dbe1 P 1c50de89a5a845668696dcb28c3badd6 [term 0 FOLLOWER]: Advancing to term 1
I20260129 02:58:51.118415 20510 raft_consensus.cc:2468] T 4e3805ffa70242e8ade66da260dddf66 P 974b802141bd4f13beffef2aa03f88f9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 197b880fa39d432a9b472845876bfaf0 in term 1.
I20260129 02:58:51.140841 21030 consensus_queue.cc:1048] T a4c1226b987b468fa84dfc6ba0e2dbe1 P 974b802141bd4f13beffef2aa03f88f9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260129 02:58:51.156203 20963 consensus_queue.cc:1048] T a4c1226b987b468fa84dfc6ba0e2dbe1 P 974b802141bd4f13beffef2aa03f88f9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260129 02:58:51.467753 21024 consensus_queue.cc:1048] T 3906b080a5b34a74b8b64128635e8997 P 197b880fa39d432a9b472845876bfaf0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260129 02:58:51.533047 21027 consensus_queue.cc:1048] T 3906b080a5b34a74b8b64128635e8997 P 197b880fa39d432a9b472845876bfaf0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260129 02:58:51.594004 21027 consensus_queue.cc:1048] T 4e3805ffa70242e8ade66da260dddf66 P 197b880fa39d432a9b472845876bfaf0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260129 02:58:51.608844 21024 consensus_queue.cc:1048] T 4e3805ffa70242e8ade66da260dddf66 P 197b880fa39d432a9b472845876bfaf0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260129 02:58:51.645046 21024 consensus_queue.cc:1048] T a8479d1fe0bd41e5946186e3a6ca8c7a P 197b880fa39d432a9b472845876bfaf0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260129 02:58:51.660246 21027 consensus_queue.cc:1048] T a8479d1fe0bd41e5946186e3a6ca8c7a P 197b880fa39d432a9b472845876bfaf0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260129 02:58:52.118126 19276 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260129 02:58:52.146188 21090 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260129 02:58:52.146294 21091 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260129 02:58:52.146248 21093 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260129 02:58:52.148595 19276 server_base.cc:1060] running on GCE node
I20260129 02:58:52.151248 19276 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260129 02:58:52.151549 19276 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260129 02:58:52.151749 19276 hybrid_clock.cc:648] HybridClock initialized: now 1769655532151733 us; error 0 us; skew 500 ppm
I20260129 02:58:52.152593 19276 server_base.cc:860] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260129 02:58:52.156087 19276 webserver.cc:492] Webserver started at http://127.18.211.4:43195/ using document root <none> and password file <none>
I20260129 02:58:52.156829 19276 fs_manager.cc:362] Metadata directory not provided
I20260129 02:58:52.157124 19276 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260129 02:58:52.157577 19276 server_base.cc:908] This appears to be a new deployment of Kudu; creating new FS layout
I20260129 02:58:52.159564 19276 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskrTP_Jo/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1769655460890924-19276-0/minicluster-data/ts-3-root/instance:
uuid: "7a9daf5d42a84615a6383cfec9a10608"
format_stamp: "Formatted at 2026-01-29 02:58:52 on dist-test-slave-vngr"
I20260129 02:58:52.164906 19276 fs_manager.cc:696] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260129 02:58:52.169185 21098 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260129 02:58:52.170047 19276 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.004s sys 0.001s
I20260129 02:58:52.170356 19276 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskrTP_Jo/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1769655460890924-19276-0/minicluster-data/ts-3-root
uuid: "7a9daf5d42a84615a6383cfec9a10608"
format_stamp: "Formatted at 2026-01-29 02:58:52 on dist-test-slave-vngr"
I20260129 02:58:52.170694 19276 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrTP_Jo/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1769655460890924-19276-0/minicluster-data/ts-3-root
metadata directory: /tmp/dist-test-taskrTP_Jo/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1769655460890924-19276-0/minicluster-data/ts-3-root
1 data directories: /tmp/dist-test-taskrTP_Jo/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1769655460890924-19276-0/minicluster-data/ts-3-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260129 02:58:52.188055 19276 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260129 02:58:52.189404 19276 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260129 02:58:52.190901 19276 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260129 02:58:52.193277 19276 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260129 02:58:52.193533 19276 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260129 02:58:52.193781 19276 ts_tablet_manager.cc:616] Registered 0 tablets
I20260129 02:58:52.193967 19276 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260129 02:58:52.249735 19276 rpc_server.cc:307] RPC server started. Bound to: 127.18.211.4:40439
I20260129 02:58:52.249863 21160 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.18.211.4:40439 every 8 connection(s)
I20260129 02:58:52.254954 19276 auto_leader_rebalancer-test.cc:417] add a tserver: 7a9daf5d42a84615a6383cfec9a10608
I20260129 02:58:52.292358 21161 heartbeater.cc:344] Connected to a master server at 127.18.211.62:34285
I20260129 02:58:52.292878 21161 heartbeater.cc:461] Registering TS with master...
I20260129 02:58:52.293789 21161 heartbeater.cc:507] Master 127.18.211.62:34285 requested a full tablet report, sending...
I20260129 02:58:52.295673 20335 ts_manager.cc:194] Registered new tserver with Master: 7a9daf5d42a84615a6383cfec9a10608 (127.18.211.4:40439)
I20260129 02:58:52.297008 20335 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:33836
I20260129 02:58:52.399219 20586 consensus_queue.cc:237] T a644c02b16af47679824ccd060ac10fd P 1c50de89a5a845668696dcb28c3badd6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } attrs { replace: true } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: NON_VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: true } }
I20260129 02:58:52.410109 20510 raft_consensus.cc:1275] T a644c02b16af47679824ccd060ac10fd P 974b802141bd4f13beffef2aa03f88f9 [term 1 FOLLOWER]: Refusing update from remote peer 1c50de89a5a845668696dcb28c3badd6: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260129 02:58:52.416338 20832 consensus_queue.cc:1048] T a644c02b16af47679824ccd060ac10fd P 1c50de89a5a845668696dcb28c3badd6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260129 02:58:52.423658 20435 raft_consensus.cc:1275] T a644c02b16af47679824ccd060ac10fd P 197b880fa39d432a9b472845876bfaf0 [term 1 FOLLOWER]: Refusing update from remote peer 1c50de89a5a845668696dcb28c3badd6: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260129 02:58:52.428858 20982 consensus_queue.cc:1048] T a644c02b16af47679824ccd060ac10fd P 1c50de89a5a845668696dcb28c3badd6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260129 02:58:52.428769 20832 raft_consensus.cc:2955] T a644c02b16af47679824ccd060ac10fd P 1c50de89a5a845668696dcb28c3badd6 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 7a9daf5d42a84615a6383cfec9a10608 (127.18.211.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } attrs { replace: true } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: NON_VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: true } } }
I20260129 02:58:52.437179 20510 raft_consensus.cc:2955] T a644c02b16af47679824ccd060ac10fd P 974b802141bd4f13beffef2aa03f88f9 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 7a9daf5d42a84615a6383cfec9a10608 (127.18.211.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } attrs { replace: true } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: NON_VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: true } } }
W20260129 02:58:52.440349 20554 consensus_peers.cc:597] T a644c02b16af47679824ccd060ac10fd P 1c50de89a5a845668696dcb28c3badd6 -> Peer 7a9daf5d42a84615a6383cfec9a10608 (127.18.211.4:40439): Couldn't send request to peer 7a9daf5d42a84615a6383cfec9a10608. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: a644c02b16af47679824ccd060ac10fd. This is attempt 1: this message will repeat every 5th retry.
I20260129 02:58:52.457830 20435 raft_consensus.cc:2955] T a644c02b16af47679824ccd060ac10fd P 197b880fa39d432a9b472845876bfaf0 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 7a9daf5d42a84615a6383cfec9a10608 (127.18.211.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } attrs { replace: true } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: NON_VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: true } } }
I20260129 02:58:52.470167 20335 catalog_manager.cc:5654] T a644c02b16af47679824ccd060ac10fd P 974b802141bd4f13beffef2aa03f88f9 reported cstate change: config changed from index -1 to 2, NON_VOTER 7a9daf5d42a84615a6383cfec9a10608 (127.18.211.4) added. New cstate: current_term: 1 leader_uuid: "1c50de89a5a845668696dcb28c3badd6" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } attrs { replace: true } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: NON_VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: true } } }
I20260129 02:58:52.486091 20438 consensus_queue.cc:237] T 42d27a28d89b409e8807ec0efd97af69 P 197b880fa39d432a9b472845876bfaf0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } attrs { replace: true } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: NON_VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: true } }
I20260129 02:58:52.502220 20586 raft_consensus.cc:1275] T 42d27a28d89b409e8807ec0efd97af69 P 1c50de89a5a845668696dcb28c3badd6 [term 1 FOLLOWER]: Refusing update from remote peer 197b880fa39d432a9b472845876bfaf0: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260129 02:58:52.502059 20510 raft_consensus.cc:1275] T 42d27a28d89b409e8807ec0efd97af69 P 974b802141bd4f13beffef2aa03f88f9 [term 1 FOLLOWER]: Refusing update from remote peer 197b880fa39d432a9b472845876bfaf0: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260129 02:58:52.504575 21004 consensus_queue.cc:1048] T 42d27a28d89b409e8807ec0efd97af69 P 197b880fa39d432a9b472845876bfaf0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260129 02:58:52.508059 21004 consensus_queue.cc:1048] T 42d27a28d89b409e8807ec0efd97af69 P 197b880fa39d432a9b472845876bfaf0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260129 02:58:52.521498 21024 raft_consensus.cc:2955] T 42d27a28d89b409e8807ec0efd97af69 P 197b880fa39d432a9b472845876bfaf0 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 7a9daf5d42a84615a6383cfec9a10608 (127.18.211.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } attrs { replace: true } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: NON_VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: true } } }
I20260129 02:58:52.528055 20586 raft_consensus.cc:2955] T 42d27a28d89b409e8807ec0efd97af69 P 1c50de89a5a845668696dcb28c3badd6 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 7a9daf5d42a84615a6383cfec9a10608 (127.18.211.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } attrs { replace: true } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: NON_VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: true } } }
W20260129 02:58:52.551194 20405 consensus_peers.cc:597] T 42d27a28d89b409e8807ec0efd97af69 P 197b880fa39d432a9b472845876bfaf0 -> Peer 7a9daf5d42a84615a6383cfec9a10608 (127.18.211.4:40439): Couldn't send request to peer 7a9daf5d42a84615a6383cfec9a10608. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 42d27a28d89b409e8807ec0efd97af69. This is attempt 1: this message will repeat every 5th retry.
I20260129 02:58:52.553943 20338 catalog_manager.cc:5654] T 42d27a28d89b409e8807ec0efd97af69 P 197b880fa39d432a9b472845876bfaf0 reported cstate change: config changed from index -1 to 2, NON_VOTER 7a9daf5d42a84615a6383cfec9a10608 (127.18.211.4) added. New cstate: current_term: 1 leader_uuid: "197b880fa39d432a9b472845876bfaf0" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: NON_VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20260129 02:58:52.555227 20438 consensus_queue.cc:237] T febffe4a8b394368b69b1d164308fb9a P 197b880fa39d432a9b472845876bfaf0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } attrs { replace: true } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: NON_VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: true } }
I20260129 02:58:52.558090 20514 raft_consensus.cc:2955] T 42d27a28d89b409e8807ec0efd97af69 P 974b802141bd4f13beffef2aa03f88f9 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 7a9daf5d42a84615a6383cfec9a10608 (127.18.211.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } attrs { replace: true } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: NON_VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: true } } }
I20260129 02:58:52.587996 20514 raft_consensus.cc:1275] T febffe4a8b394368b69b1d164308fb9a P 974b802141bd4f13beffef2aa03f88f9 [term 1 FOLLOWER]: Refusing update from remote peer 197b880fa39d432a9b472845876bfaf0: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20260129 02:58:52.588217 20405 consensus_peers.cc:597] T febffe4a8b394368b69b1d164308fb9a P 197b880fa39d432a9b472845876bfaf0 -> Peer 7a9daf5d42a84615a6383cfec9a10608 (127.18.211.4:40439): Couldn't send request to peer 7a9daf5d42a84615a6383cfec9a10608. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: febffe4a8b394368b69b1d164308fb9a. This is attempt 1: this message will repeat every 5th retry.
I20260129 02:58:52.589763 21185 consensus_queue.cc:1048] T febffe4a8b394368b69b1d164308fb9a P 197b880fa39d432a9b472845876bfaf0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260129 02:58:52.592413 20586 raft_consensus.cc:1275] T febffe4a8b394368b69b1d164308fb9a P 1c50de89a5a845668696dcb28c3badd6 [term 1 FOLLOWER]: Refusing update from remote peer 197b880fa39d432a9b472845876bfaf0: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260129 02:58:52.593569 21185 consensus_queue.cc:1048] T febffe4a8b394368b69b1d164308fb9a P 197b880fa39d432a9b472845876bfaf0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260129 02:58:52.602459 21004 raft_consensus.cc:2955] T febffe4a8b394368b69b1d164308fb9a P 197b880fa39d432a9b472845876bfaf0 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 7a9daf5d42a84615a6383cfec9a10608 (127.18.211.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } attrs { replace: true } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: NON_VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: true } } }
I20260129 02:58:52.604563 20586 raft_consensus.cc:2955] T febffe4a8b394368b69b1d164308fb9a P 1c50de89a5a845668696dcb28c3badd6 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 7a9daf5d42a84615a6383cfec9a10608 (127.18.211.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } attrs { replace: true } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: NON_VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: true } } }
I20260129 02:58:52.607378 20516 raft_consensus.cc:2955] T febffe4a8b394368b69b1d164308fb9a P 974b802141bd4f13beffef2aa03f88f9 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 7a9daf5d42a84615a6383cfec9a10608 (127.18.211.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } attrs { replace: true } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: NON_VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: true } } }
I20260129 02:58:52.619602 20586 consensus_queue.cc:237] T fd8311b779af4e1fa1402b2a02a602f2 P 1c50de89a5a845668696dcb28c3badd6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } attrs { replace: true } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: NON_VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: true } }
I20260129 02:58:52.620667 20335 catalog_manager.cc:5654] T febffe4a8b394368b69b1d164308fb9a P 1c50de89a5a845668696dcb28c3badd6 reported cstate change: config changed from index -1 to 2, NON_VOTER 7a9daf5d42a84615a6383cfec9a10608 (127.18.211.4) added. New cstate: current_term: 1 leader_uuid: "197b880fa39d432a9b472845876bfaf0" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } attrs { replace: true } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: NON_VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: true } } }
W20260129 02:58:52.629676 20554 consensus_peers.cc:597] T fd8311b779af4e1fa1402b2a02a602f2 P 1c50de89a5a845668696dcb28c3badd6 -> Peer 7a9daf5d42a84615a6383cfec9a10608 (127.18.211.4:40439): Couldn't send request to peer 7a9daf5d42a84615a6383cfec9a10608. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: fd8311b779af4e1fa1402b2a02a602f2. This is attempt 1: this message will repeat every 5th retry.
I20260129 02:58:52.631714 20516 raft_consensus.cc:1275] T fd8311b779af4e1fa1402b2a02a602f2 P 974b802141bd4f13beffef2aa03f88f9 [term 1 FOLLOWER]: Refusing update from remote peer 1c50de89a5a845668696dcb28c3badd6: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260129 02:58:52.631805 20438 raft_consensus.cc:1275] T fd8311b779af4e1fa1402b2a02a602f2 P 197b880fa39d432a9b472845876bfaf0 [term 1 FOLLOWER]: Refusing update from remote peer 1c50de89a5a845668696dcb28c3badd6: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260129 02:58:52.633268 21168 consensus_queue.cc:1048] T fd8311b779af4e1fa1402b2a02a602f2 P 1c50de89a5a845668696dcb28c3badd6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260129 02:58:52.634160 20832 consensus_queue.cc:1048] T fd8311b779af4e1fa1402b2a02a602f2 P 1c50de89a5a845668696dcb28c3badd6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260129 02:58:52.647895 20832 raft_consensus.cc:2955] T fd8311b779af4e1fa1402b2a02a602f2 P 1c50de89a5a845668696dcb28c3badd6 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 7a9daf5d42a84615a6383cfec9a10608 (127.18.211.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } attrs { replace: true } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: NON_VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: true } } }
I20260129 02:58:52.653172 20438 raft_consensus.cc:2955] T fd8311b779af4e1fa1402b2a02a602f2 P 197b880fa39d432a9b472845876bfaf0 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 7a9daf5d42a84615a6383cfec9a10608 (127.18.211.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } attrs { replace: true } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: NON_VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: true } } }
I20260129 02:58:52.656239 20516 raft_consensus.cc:2955] T fd8311b779af4e1fa1402b2a02a602f2 P 974b802141bd4f13beffef2aa03f88f9 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 7a9daf5d42a84615a6383cfec9a10608 (127.18.211.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } attrs { replace: true } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: NON_VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: true } } }
I20260129 02:58:52.665402 20338 catalog_manager.cc:5654] T fd8311b779af4e1fa1402b2a02a602f2 P 1c50de89a5a845668696dcb28c3badd6 reported cstate change: config changed from index -1 to 2, NON_VOTER 7a9daf5d42a84615a6383cfec9a10608 (127.18.211.4) added. New cstate: current_term: 1 leader_uuid: "1c50de89a5a845668696dcb28c3badd6" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: NON_VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20260129 02:58:52.981698 21193 ts_tablet_manager.cc:933] T a644c02b16af47679824ccd060ac10fd P 7a9daf5d42a84615a6383cfec9a10608: Initiating tablet copy from peer 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3:37735)
I20260129 02:58:52.989033 21193 tablet_copy_client.cc:323] T a644c02b16af47679824ccd060ac10fd P 7a9daf5d42a84615a6383cfec9a10608: tablet copy: Beginning tablet copy session from remote peer at address 127.18.211.3:37735
I20260129 02:58:53.001711 20600 tablet_copy_service.cc:140] P 1c50de89a5a845668696dcb28c3badd6: Received BeginTabletCopySession request for tablet a644c02b16af47679824ccd060ac10fd from peer 7a9daf5d42a84615a6383cfec9a10608 ({username='slave'} at 127.0.0.1:48526)
I20260129 02:58:53.002266 20600 tablet_copy_service.cc:161] P 1c50de89a5a845668696dcb28c3badd6: Beginning new tablet copy session on tablet a644c02b16af47679824ccd060ac10fd from peer 7a9daf5d42a84615a6383cfec9a10608 at {username='slave'} at 127.0.0.1:48526: session id = 7a9daf5d42a84615a6383cfec9a10608-a644c02b16af47679824ccd060ac10fd
I20260129 02:58:53.009188 20600 tablet_copy_source_session.cc:215] T a644c02b16af47679824ccd060ac10fd P 1c50de89a5a845668696dcb28c3badd6: Tablet Copy: opened 0 blocks and 1 log segments
I20260129 02:58:53.013017 21193 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a644c02b16af47679824ccd060ac10fd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260129 02:58:53.027328 21193 tablet_copy_client.cc:806] T a644c02b16af47679824ccd060ac10fd P 7a9daf5d42a84615a6383cfec9a10608: tablet copy: Starting download of 0 data blocks...
I20260129 02:58:53.028049 21193 tablet_copy_client.cc:670] T a644c02b16af47679824ccd060ac10fd P 7a9daf5d42a84615a6383cfec9a10608: tablet copy: Starting download of 1 WAL segments...
I20260129 02:58:53.031073 21197 ts_tablet_manager.cc:933] T 42d27a28d89b409e8807ec0efd97af69 P 7a9daf5d42a84615a6383cfec9a10608: Initiating tablet copy from peer 197b880fa39d432a9b472845876bfaf0 (127.18.211.1:42027)
I20260129 02:58:53.034572 21193 tablet_copy_client.cc:538] T a644c02b16af47679824ccd060ac10fd P 7a9daf5d42a84615a6383cfec9a10608: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260129 02:58:53.039410 21197 tablet_copy_client.cc:323] T 42d27a28d89b409e8807ec0efd97af69 P 7a9daf5d42a84615a6383cfec9a10608: tablet copy: Beginning tablet copy session from remote peer at address 127.18.211.1:42027
I20260129 02:58:53.043916 21193 tablet_bootstrap.cc:492] T a644c02b16af47679824ccd060ac10fd P 7a9daf5d42a84615a6383cfec9a10608: Bootstrap starting.
I20260129 02:58:53.077461 20451 tablet_copy_service.cc:140] P 197b880fa39d432a9b472845876bfaf0: Received BeginTabletCopySession request for tablet 42d27a28d89b409e8807ec0efd97af69 from peer 7a9daf5d42a84615a6383cfec9a10608 ({username='slave'} at 127.0.0.1:48356)
I20260129 02:58:53.078081 20451 tablet_copy_service.cc:161] P 197b880fa39d432a9b472845876bfaf0: Beginning new tablet copy session on tablet 42d27a28d89b409e8807ec0efd97af69 from peer 7a9daf5d42a84615a6383cfec9a10608 at {username='slave'} at 127.0.0.1:48356: session id = 7a9daf5d42a84615a6383cfec9a10608-42d27a28d89b409e8807ec0efd97af69
I20260129 02:58:53.081110 21193 tablet_bootstrap.cc:492] T a644c02b16af47679824ccd060ac10fd P 7a9daf5d42a84615a6383cfec9a10608: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260129 02:58:53.082141 21193 tablet_bootstrap.cc:492] T a644c02b16af47679824ccd060ac10fd P 7a9daf5d42a84615a6383cfec9a10608: Bootstrap complete.
I20260129 02:58:53.082818 21193 ts_tablet_manager.cc:1403] T a644c02b16af47679824ccd060ac10fd P 7a9daf5d42a84615a6383cfec9a10608: Time spent bootstrapping tablet: real 0.039s user 0.028s sys 0.009s
I20260129 02:58:53.085793 21193 raft_consensus.cc:359] T a644c02b16af47679824ccd060ac10fd P 7a9daf5d42a84615a6383cfec9a10608 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } attrs { replace: true } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: NON_VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: true } }
I20260129 02:58:53.086627 21193 raft_consensus.cc:740] T a644c02b16af47679824ccd060ac10fd P 7a9daf5d42a84615a6383cfec9a10608 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 7a9daf5d42a84615a6383cfec9a10608, State: Initialized, Role: LEARNER
I20260129 02:58:53.086910 20451 tablet_copy_source_session.cc:215] T 42d27a28d89b409e8807ec0efd97af69 P 197b880fa39d432a9b472845876bfaf0: Tablet Copy: opened 0 blocks and 1 log segments
I20260129 02:58:53.087299 21193 consensus_queue.cc:260] T a644c02b16af47679824ccd060ac10fd P 7a9daf5d42a84615a6383cfec9a10608 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } attrs { replace: true } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: NON_VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: true } }
I20260129 02:58:53.089920 21161 heartbeater.cc:499] Master 127.18.211.62:34285 was elected leader, sending a full tablet report...
I20260129 02:58:53.090339 21197 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 42d27a28d89b409e8807ec0efd97af69. 1 dirs total, 0 dirs full, 0 dirs failed
I20260129 02:58:53.090677 21193 ts_tablet_manager.cc:1434] T a644c02b16af47679824ccd060ac10fd P 7a9daf5d42a84615a6383cfec9a10608: Time spent starting tablet: real 0.008s user 0.002s sys 0.007s
I20260129 02:58:53.093214 20600 tablet_copy_service.cc:342] P 1c50de89a5a845668696dcb28c3badd6: Request end of tablet copy session 7a9daf5d42a84615a6383cfec9a10608-a644c02b16af47679824ccd060ac10fd received from {username='slave'} at 127.0.0.1:48526
I20260129 02:58:53.093736 20600 tablet_copy_service.cc:434] P 1c50de89a5a845668696dcb28c3badd6: ending tablet copy session 7a9daf5d42a84615a6383cfec9a10608-a644c02b16af47679824ccd060ac10fd on tablet a644c02b16af47679824ccd060ac10fd with peer 7a9daf5d42a84615a6383cfec9a10608
I20260129 02:58:53.113371 21197 tablet_copy_client.cc:806] T 42d27a28d89b409e8807ec0efd97af69 P 7a9daf5d42a84615a6383cfec9a10608: tablet copy: Starting download of 0 data blocks...
I20260129 02:58:53.113986 21197 tablet_copy_client.cc:670] T 42d27a28d89b409e8807ec0efd97af69 P 7a9daf5d42a84615a6383cfec9a10608: tablet copy: Starting download of 1 WAL segments...
I20260129 02:58:53.124410 21197 tablet_copy_client.cc:538] T 42d27a28d89b409e8807ec0efd97af69 P 7a9daf5d42a84615a6383cfec9a10608: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260129 02:58:53.135295 21197 tablet_bootstrap.cc:492] T 42d27a28d89b409e8807ec0efd97af69 P 7a9daf5d42a84615a6383cfec9a10608: Bootstrap starting.
I20260129 02:58:53.155088 21197 tablet_bootstrap.cc:492] T 42d27a28d89b409e8807ec0efd97af69 P 7a9daf5d42a84615a6383cfec9a10608: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260129 02:58:53.155815 21197 tablet_bootstrap.cc:492] T 42d27a28d89b409e8807ec0efd97af69 P 7a9daf5d42a84615a6383cfec9a10608: Bootstrap complete.
I20260129 02:58:53.156523 21197 ts_tablet_manager.cc:1403] T 42d27a28d89b409e8807ec0efd97af69 P 7a9daf5d42a84615a6383cfec9a10608: Time spent bootstrapping tablet: real 0.021s user 0.009s sys 0.012s
I20260129 02:58:53.159168 21197 raft_consensus.cc:359] T 42d27a28d89b409e8807ec0efd97af69 P 7a9daf5d42a84615a6383cfec9a10608 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } attrs { replace: true } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: NON_VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: true } }
I20260129 02:58:53.159999 21197 raft_consensus.cc:740] T 42d27a28d89b409e8807ec0efd97af69 P 7a9daf5d42a84615a6383cfec9a10608 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 7a9daf5d42a84615a6383cfec9a10608, State: Initialized, Role: LEARNER
I20260129 02:58:53.160657 21197 consensus_queue.cc:260] T 42d27a28d89b409e8807ec0efd97af69 P 7a9daf5d42a84615a6383cfec9a10608 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } attrs { replace: true } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: NON_VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: true } }
I20260129 02:58:53.163012 21197 ts_tablet_manager.cc:1434] T 42d27a28d89b409e8807ec0efd97af69 P 7a9daf5d42a84615a6383cfec9a10608: Time spent starting tablet: real 0.006s user 0.004s sys 0.000s
I20260129 02:58:53.167201 20451 tablet_copy_service.cc:342] P 197b880fa39d432a9b472845876bfaf0: Request end of tablet copy session 7a9daf5d42a84615a6383cfec9a10608-42d27a28d89b409e8807ec0efd97af69 received from {username='slave'} at 127.0.0.1:48356
I20260129 02:58:53.167629 20451 tablet_copy_service.cc:434] P 197b880fa39d432a9b472845876bfaf0: ending tablet copy session 7a9daf5d42a84615a6383cfec9a10608-42d27a28d89b409e8807ec0efd97af69 on tablet 42d27a28d89b409e8807ec0efd97af69 with peer 7a9daf5d42a84615a6383cfec9a10608
I20260129 02:58:53.181679 21197 ts_tablet_manager.cc:933] T febffe4a8b394368b69b1d164308fb9a P 7a9daf5d42a84615a6383cfec9a10608: Initiating tablet copy from peer 197b880fa39d432a9b472845876bfaf0 (127.18.211.1:42027)
I20260129 02:58:53.183264 21197 tablet_copy_client.cc:323] T febffe4a8b394368b69b1d164308fb9a P 7a9daf5d42a84615a6383cfec9a10608: tablet copy: Beginning tablet copy session from remote peer at address 127.18.211.1:42027
I20260129 02:58:53.184710 20451 tablet_copy_service.cc:140] P 197b880fa39d432a9b472845876bfaf0: Received BeginTabletCopySession request for tablet febffe4a8b394368b69b1d164308fb9a from peer 7a9daf5d42a84615a6383cfec9a10608 ({username='slave'} at 127.0.0.1:48356)
I20260129 02:58:53.185194 20451 tablet_copy_service.cc:161] P 197b880fa39d432a9b472845876bfaf0: Beginning new tablet copy session on tablet febffe4a8b394368b69b1d164308fb9a from peer 7a9daf5d42a84615a6383cfec9a10608 at {username='slave'} at 127.0.0.1:48356: session id = 7a9daf5d42a84615a6383cfec9a10608-febffe4a8b394368b69b1d164308fb9a
I20260129 02:58:53.188144 21193 ts_tablet_manager.cc:933] T fd8311b779af4e1fa1402b2a02a602f2 P 7a9daf5d42a84615a6383cfec9a10608: Initiating tablet copy from peer 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3:37735)
I20260129 02:58:53.191720 21193 tablet_copy_client.cc:323] T fd8311b779af4e1fa1402b2a02a602f2 P 7a9daf5d42a84615a6383cfec9a10608: tablet copy: Beginning tablet copy session from remote peer at address 127.18.211.3:37735
I20260129 02:58:53.193362 20600 tablet_copy_service.cc:140] P 1c50de89a5a845668696dcb28c3badd6: Received BeginTabletCopySession request for tablet fd8311b779af4e1fa1402b2a02a602f2 from peer 7a9daf5d42a84615a6383cfec9a10608 ({username='slave'} at 127.0.0.1:48526)
I20260129 02:58:53.193847 20600 tablet_copy_service.cc:161] P 1c50de89a5a845668696dcb28c3badd6: Beginning new tablet copy session on tablet fd8311b779af4e1fa1402b2a02a602f2 from peer 7a9daf5d42a84615a6383cfec9a10608 at {username='slave'} at 127.0.0.1:48526: session id = 7a9daf5d42a84615a6383cfec9a10608-fd8311b779af4e1fa1402b2a02a602f2
I20260129 02:58:53.193925 20451 tablet_copy_source_session.cc:215] T febffe4a8b394368b69b1d164308fb9a P 197b880fa39d432a9b472845876bfaf0: Tablet Copy: opened 0 blocks and 1 log segments
I20260129 02:58:53.200505 20600 tablet_copy_source_session.cc:215] T fd8311b779af4e1fa1402b2a02a602f2 P 1c50de89a5a845668696dcb28c3badd6: Tablet Copy: opened 0 blocks and 1 log segments
I20260129 02:58:53.201563 21197 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet febffe4a8b394368b69b1d164308fb9a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260129 02:58:53.203254 21193 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fd8311b779af4e1fa1402b2a02a602f2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260129 02:58:53.218391 21193 tablet_copy_client.cc:806] T fd8311b779af4e1fa1402b2a02a602f2 P 7a9daf5d42a84615a6383cfec9a10608: tablet copy: Starting download of 0 data blocks...
I20260129 02:58:53.218417 21197 tablet_copy_client.cc:806] T febffe4a8b394368b69b1d164308fb9a P 7a9daf5d42a84615a6383cfec9a10608: tablet copy: Starting download of 0 data blocks...
I20260129 02:58:53.219148 21193 tablet_copy_client.cc:670] T fd8311b779af4e1fa1402b2a02a602f2 P 7a9daf5d42a84615a6383cfec9a10608: tablet copy: Starting download of 1 WAL segments...
I20260129 02:58:53.220741 21197 tablet_copy_client.cc:670] T febffe4a8b394368b69b1d164308fb9a P 7a9daf5d42a84615a6383cfec9a10608: tablet copy: Starting download of 1 WAL segments...
I20260129 02:58:53.226996 21197 tablet_copy_client.cc:538] T febffe4a8b394368b69b1d164308fb9a P 7a9daf5d42a84615a6383cfec9a10608: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260129 02:58:53.230113 21193 tablet_copy_client.cc:538] T fd8311b779af4e1fa1402b2a02a602f2 P 7a9daf5d42a84615a6383cfec9a10608: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260129 02:58:53.236609 21197 tablet_bootstrap.cc:492] T febffe4a8b394368b69b1d164308fb9a P 7a9daf5d42a84615a6383cfec9a10608: Bootstrap starting.
I20260129 02:58:53.237960 21193 tablet_bootstrap.cc:492] T fd8311b779af4e1fa1402b2a02a602f2 P 7a9daf5d42a84615a6383cfec9a10608: Bootstrap starting.
I20260129 02:58:53.258756 21197 tablet_bootstrap.cc:492] T febffe4a8b394368b69b1d164308fb9a P 7a9daf5d42a84615a6383cfec9a10608: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260129 02:58:53.259748 21197 tablet_bootstrap.cc:492] T febffe4a8b394368b69b1d164308fb9a P 7a9daf5d42a84615a6383cfec9a10608: Bootstrap complete.
I20260129 02:58:53.260526 21197 ts_tablet_manager.cc:1403] T febffe4a8b394368b69b1d164308fb9a P 7a9daf5d42a84615a6383cfec9a10608: Time spent bootstrapping tablet: real 0.024s user 0.013s sys 0.010s
I20260129 02:58:53.261705 21193 tablet_bootstrap.cc:492] T fd8311b779af4e1fa1402b2a02a602f2 P 7a9daf5d42a84615a6383cfec9a10608: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260129 02:58:53.262620 21193 tablet_bootstrap.cc:492] T fd8311b779af4e1fa1402b2a02a602f2 P 7a9daf5d42a84615a6383cfec9a10608: Bootstrap complete.
I20260129 02:58:53.263314 21193 ts_tablet_manager.cc:1403] T fd8311b779af4e1fa1402b2a02a602f2 P 7a9daf5d42a84615a6383cfec9a10608: Time spent bootstrapping tablet: real 0.026s user 0.019s sys 0.004s
I20260129 02:58:53.263597 21197 raft_consensus.cc:359] T febffe4a8b394368b69b1d164308fb9a P 7a9daf5d42a84615a6383cfec9a10608 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } attrs { replace: true } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: NON_VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: true } }
I20260129 02:58:53.264575 21197 raft_consensus.cc:740] T febffe4a8b394368b69b1d164308fb9a P 7a9daf5d42a84615a6383cfec9a10608 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 7a9daf5d42a84615a6383cfec9a10608, State: Initialized, Role: LEARNER
I20260129 02:58:53.265272 21197 consensus_queue.cc:260] T febffe4a8b394368b69b1d164308fb9a P 7a9daf5d42a84615a6383cfec9a10608 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } attrs { replace: true } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: NON_VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: true } }
I20260129 02:58:53.266086 21193 raft_consensus.cc:359] T fd8311b779af4e1fa1402b2a02a602f2 P 7a9daf5d42a84615a6383cfec9a10608 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } attrs { replace: true } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: NON_VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: true } }
I20260129 02:58:53.266955 21193 raft_consensus.cc:740] T fd8311b779af4e1fa1402b2a02a602f2 P 7a9daf5d42a84615a6383cfec9a10608 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 7a9daf5d42a84615a6383cfec9a10608, State: Initialized, Role: LEARNER
I20260129 02:58:53.267652 21193 consensus_queue.cc:260] T fd8311b779af4e1fa1402b2a02a602f2 P 7a9daf5d42a84615a6383cfec9a10608 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } attrs { replace: true } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: NON_VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: true } }
I20260129 02:58:53.269562 21197 ts_tablet_manager.cc:1434] T febffe4a8b394368b69b1d164308fb9a P 7a9daf5d42a84615a6383cfec9a10608: Time spent starting tablet: real 0.009s user 0.005s sys 0.000s
I20260129 02:58:53.270787 21193 ts_tablet_manager.cc:1434] T fd8311b779af4e1fa1402b2a02a602f2 P 7a9daf5d42a84615a6383cfec9a10608: Time spent starting tablet: real 0.007s user 0.005s sys 0.000s
I20260129 02:58:53.272519 20600 tablet_copy_service.cc:342] P 1c50de89a5a845668696dcb28c3badd6: Request end of tablet copy session 7a9daf5d42a84615a6383cfec9a10608-fd8311b779af4e1fa1402b2a02a602f2 received from {username='slave'} at 127.0.0.1:48526
I20260129 02:58:53.272927 20600 tablet_copy_service.cc:434] P 1c50de89a5a845668696dcb28c3badd6: ending tablet copy session 7a9daf5d42a84615a6383cfec9a10608-fd8311b779af4e1fa1402b2a02a602f2 on tablet fd8311b779af4e1fa1402b2a02a602f2 with peer 7a9daf5d42a84615a6383cfec9a10608
I20260129 02:58:53.277355 20451 tablet_copy_service.cc:342] P 197b880fa39d432a9b472845876bfaf0: Request end of tablet copy session 7a9daf5d42a84615a6383cfec9a10608-febffe4a8b394368b69b1d164308fb9a received from {username='slave'} at 127.0.0.1:48356
I20260129 02:58:53.277740 20451 tablet_copy_service.cc:434] P 197b880fa39d432a9b472845876bfaf0: ending tablet copy session 7a9daf5d42a84615a6383cfec9a10608-febffe4a8b394368b69b1d164308fb9a on tablet febffe4a8b394368b69b1d164308fb9a with peer 7a9daf5d42a84615a6383cfec9a10608
I20260129 02:58:53.464504 21136 raft_consensus.cc:1217] T 42d27a28d89b409e8807ec0efd97af69 P 7a9daf5d42a84615a6383cfec9a10608 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20260129 02:58:53.558490 21136 raft_consensus.cc:1217] T a644c02b16af47679824ccd060ac10fd P 7a9daf5d42a84615a6383cfec9a10608 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20260129 02:58:53.653195 21136 raft_consensus.cc:1217] T fd8311b779af4e1fa1402b2a02a602f2 P 7a9daf5d42a84615a6383cfec9a10608 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20260129 02:58:53.734546 21136 raft_consensus.cc:1217] T febffe4a8b394368b69b1d164308fb9a P 7a9daf5d42a84615a6383cfec9a10608 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20260129 02:58:54.050159 21024 raft_consensus.cc:1064] T 42d27a28d89b409e8807ec0efd97af69 P 197b880fa39d432a9b472845876bfaf0: attempting to promote NON_VOTER 7a9daf5d42a84615a6383cfec9a10608 to VOTER
I20260129 02:58:54.052776 21024 consensus_queue.cc:237] T 42d27a28d89b409e8807ec0efd97af69 P 197b880fa39d432a9b472845876bfaf0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } attrs { replace: true } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } }
I20260129 02:58:54.064337 21209 raft_consensus.cc:1064] T a644c02b16af47679824ccd060ac10fd P 1c50de89a5a845668696dcb28c3badd6: attempting to promote NON_VOTER 7a9daf5d42a84615a6383cfec9a10608 to VOTER
I20260129 02:58:54.065477 21135 raft_consensus.cc:1275] T 42d27a28d89b409e8807ec0efd97af69 P 7a9daf5d42a84615a6383cfec9a10608 [term 1 LEARNER]: Refusing update from remote peer 197b880fa39d432a9b472845876bfaf0: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260129 02:58:54.066957 20514 raft_consensus.cc:1275] T 42d27a28d89b409e8807ec0efd97af69 P 974b802141bd4f13beffef2aa03f88f9 [term 1 FOLLOWER]: Refusing update from remote peer 197b880fa39d432a9b472845876bfaf0: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260129 02:58:54.066993 21209 consensus_queue.cc:237] T a644c02b16af47679824ccd060ac10fd P 1c50de89a5a845668696dcb28c3badd6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } attrs { replace: true } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } }
I20260129 02:58:54.068500 21004 consensus_queue.cc:1048] T 42d27a28d89b409e8807ec0efd97af69 P 197b880fa39d432a9b472845876bfaf0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260129 02:58:54.071277 21024 consensus_queue.cc:1048] T 42d27a28d89b409e8807ec0efd97af69 P 197b880fa39d432a9b472845876bfaf0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260129 02:58:54.071713 21208 raft_consensus.cc:1064] T fd8311b779af4e1fa1402b2a02a602f2 P 1c50de89a5a845668696dcb28c3badd6: attempting to promote NON_VOTER 7a9daf5d42a84615a6383cfec9a10608 to VOTER
I20260129 02:58:54.072178 20587 raft_consensus.cc:1275] T 42d27a28d89b409e8807ec0efd97af69 P 1c50de89a5a845668696dcb28c3badd6 [term 1 FOLLOWER]: Refusing update from remote peer 197b880fa39d432a9b472845876bfaf0: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260129 02:58:54.073921 21208 consensus_queue.cc:237] T fd8311b779af4e1fa1402b2a02a602f2 P 1c50de89a5a845668696dcb28c3badd6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } attrs { replace: true } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } }
I20260129 02:58:54.081137 21004 consensus_queue.cc:1048] T 42d27a28d89b409e8807ec0efd97af69 P 197b880fa39d432a9b472845876bfaf0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260129 02:58:54.091476 21136 raft_consensus.cc:1275] T fd8311b779af4e1fa1402b2a02a602f2 P 7a9daf5d42a84615a6383cfec9a10608 [term 1 LEARNER]: Refusing update from remote peer 1c50de89a5a845668696dcb28c3badd6: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260129 02:58:54.093847 21136 raft_consensus.cc:1275] T a644c02b16af47679824ccd060ac10fd P 7a9daf5d42a84615a6383cfec9a10608 [term 1 LEARNER]: Refusing update from remote peer 1c50de89a5a845668696dcb28c3badd6: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260129 02:58:54.096087 20435 raft_consensus.cc:1275] T fd8311b779af4e1fa1402b2a02a602f2 P 197b880fa39d432a9b472845876bfaf0 [term 1 FOLLOWER]: Refusing update from remote peer 1c50de89a5a845668696dcb28c3badd6: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260129 02:58:54.099103 20982 consensus_queue.cc:1048] T fd8311b779af4e1fa1402b2a02a602f2 P 1c50de89a5a845668696dcb28c3badd6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260129 02:58:54.100312 21217 consensus_queue.cc:1048] T a644c02b16af47679824ccd060ac10fd P 1c50de89a5a845668696dcb28c3badd6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260129 02:58:54.101449 20440 raft_consensus.cc:1275] T a644c02b16af47679824ccd060ac10fd P 197b880fa39d432a9b472845876bfaf0 [term 1 FOLLOWER]: Refusing update from remote peer 1c50de89a5a845668696dcb28c3badd6: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260129 02:58:54.108043 20982 consensus_queue.cc:1048] T fd8311b779af4e1fa1402b2a02a602f2 P 1c50de89a5a845668696dcb28c3badd6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260129 02:58:54.109390 21208 consensus_queue.cc:1048] T a644c02b16af47679824ccd060ac10fd P 1c50de89a5a845668696dcb28c3badd6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260129 02:58:54.111850 20516 raft_consensus.cc:1275] T a644c02b16af47679824ccd060ac10fd P 974b802141bd4f13beffef2aa03f88f9 [term 1 FOLLOWER]: Refusing update from remote peer 1c50de89a5a845668696dcb28c3badd6: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260129 02:58:54.116984 21024 raft_consensus.cc:2955] T 42d27a28d89b409e8807ec0efd97af69 P 197b880fa39d432a9b472845876bfaf0 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 7a9daf5d42a84615a6383cfec9a10608 (127.18.211.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } attrs { replace: true } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } } }
I20260129 02:58:54.118463 20514 raft_consensus.cc:1275] T fd8311b779af4e1fa1402b2a02a602f2 P 974b802141bd4f13beffef2aa03f88f9 [term 1 FOLLOWER]: Refusing update from remote peer 1c50de89a5a845668696dcb28c3badd6: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260129 02:58:54.118115 20516 raft_consensus.cc:2955] T 42d27a28d89b409e8807ec0efd97af69 P 974b802141bd4f13beffef2aa03f88f9 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 7a9daf5d42a84615a6383cfec9a10608 (127.18.211.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } attrs { replace: true } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } } }
I20260129 02:58:54.120342 20982 consensus_queue.cc:1048] T a644c02b16af47679824ccd060ac10fd P 1c50de89a5a845668696dcb28c3badd6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260129 02:58:54.133831 20587 raft_consensus.cc:2955] T 42d27a28d89b409e8807ec0efd97af69 P 1c50de89a5a845668696dcb28c3badd6 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 7a9daf5d42a84615a6383cfec9a10608 (127.18.211.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } attrs { replace: true } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } } }
I20260129 02:58:54.140839 21217 consensus_queue.cc:1048] T fd8311b779af4e1fa1402b2a02a602f2 P 1c50de89a5a845668696dcb28c3badd6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260129 02:58:54.152172 20335 catalog_manager.cc:5654] T 42d27a28d89b409e8807ec0efd97af69 P 1c50de89a5a845668696dcb28c3badd6 reported cstate change: config changed from index 2 to 3, 7a9daf5d42a84615a6383cfec9a10608 (127.18.211.4) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "197b880fa39d432a9b472845876bfaf0" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } attrs { replace: true } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } } }
I20260129 02:58:54.155656 21217 raft_consensus.cc:2955] T fd8311b779af4e1fa1402b2a02a602f2 P 1c50de89a5a845668696dcb28c3badd6 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 7a9daf5d42a84615a6383cfec9a10608 (127.18.211.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } attrs { replace: true } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } } }
I20260129 02:58:54.167505 20435 raft_consensus.cc:2955] T fd8311b779af4e1fa1402b2a02a602f2 P 197b880fa39d432a9b472845876bfaf0 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 7a9daf5d42a84615a6383cfec9a10608 (127.18.211.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } attrs { replace: true } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } } }
I20260129 02:58:54.167845 20516 raft_consensus.cc:2955] T fd8311b779af4e1fa1402b2a02a602f2 P 974b802141bd4f13beffef2aa03f88f9 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 7a9daf5d42a84615a6383cfec9a10608 (127.18.211.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } attrs { replace: true } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } } }
I20260129 02:58:54.185732 21136 raft_consensus.cc:2955] T fd8311b779af4e1fa1402b2a02a602f2 P 7a9daf5d42a84615a6383cfec9a10608 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 7a9daf5d42a84615a6383cfec9a10608 (127.18.211.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } attrs { replace: true } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } } }
W20260129 02:58:54.195190 20405 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.211.3:37735, user_credentials={real_user=slave}} blocked reactor thread for 56598.4us
I20260129 02:58:54.201107 21134 raft_consensus.cc:2955] T 42d27a28d89b409e8807ec0efd97af69 P 7a9daf5d42a84615a6383cfec9a10608 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 7a9daf5d42a84615a6383cfec9a10608 (127.18.211.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } attrs { replace: true } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } } }
I20260129 02:58:54.203671 20337 catalog_manager.cc:5654] T fd8311b779af4e1fa1402b2a02a602f2 P 7a9daf5d42a84615a6383cfec9a10608 reported cstate change: config changed from index 2 to 3, 7a9daf5d42a84615a6383cfec9a10608 (127.18.211.4) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "1c50de89a5a845668696dcb28c3badd6" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } attrs { replace: true } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } } }
I20260129 02:58:54.208833 20982 raft_consensus.cc:2955] T a644c02b16af47679824ccd060ac10fd P 1c50de89a5a845668696dcb28c3badd6 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 7a9daf5d42a84615a6383cfec9a10608 (127.18.211.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } attrs { replace: true } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } } }
I20260129 02:58:54.212630 20510 raft_consensus.cc:2955] T a644c02b16af47679824ccd060ac10fd P 974b802141bd4f13beffef2aa03f88f9 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 7a9daf5d42a84615a6383cfec9a10608 (127.18.211.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } attrs { replace: true } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } } }
I20260129 02:58:54.215319 21135 raft_consensus.cc:2955] T a644c02b16af47679824ccd060ac10fd P 7a9daf5d42a84615a6383cfec9a10608 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 7a9daf5d42a84615a6383cfec9a10608 (127.18.211.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } attrs { replace: true } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } } }
I20260129 02:58:54.220508 20435 raft_consensus.cc:2955] T a644c02b16af47679824ccd060ac10fd P 197b880fa39d432a9b472845876bfaf0 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 7a9daf5d42a84615a6383cfec9a10608 (127.18.211.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } attrs { replace: true } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } } }
I20260129 02:58:54.222169 21188 raft_consensus.cc:1064] T febffe4a8b394368b69b1d164308fb9a P 197b880fa39d432a9b472845876bfaf0: attempting to promote NON_VOTER 7a9daf5d42a84615a6383cfec9a10608 to VOTER
I20260129 02:58:54.224817 21188 consensus_queue.cc:237] T febffe4a8b394368b69b1d164308fb9a P 197b880fa39d432a9b472845876bfaf0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } attrs { replace: true } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } }
I20260129 02:58:54.274780 20584 raft_consensus.cc:1275] T febffe4a8b394368b69b1d164308fb9a P 1c50de89a5a845668696dcb28c3badd6 [term 1 FOLLOWER]: Refusing update from remote peer 197b880fa39d432a9b472845876bfaf0: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260129 02:58:54.275274 20516 raft_consensus.cc:1275] T febffe4a8b394368b69b1d164308fb9a P 974b802141bd4f13beffef2aa03f88f9 [term 1 FOLLOWER]: Refusing update from remote peer 197b880fa39d432a9b472845876bfaf0: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260129 02:58:54.279045 20438 consensus_queue.cc:237] T 42d27a28d89b409e8807ec0efd97af69 P 197b880fa39d432a9b472845876bfaf0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } }
I20260129 02:58:54.316075 21134 raft_consensus.cc:1275] T febffe4a8b394368b69b1d164308fb9a P 7a9daf5d42a84615a6383cfec9a10608 [term 1 LEARNER]: Refusing update from remote peer 197b880fa39d432a9b472845876bfaf0: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260129 02:58:54.279079 21188 consensus_queue.cc:1048] T febffe4a8b394368b69b1d164308fb9a P 197b880fa39d432a9b472845876bfaf0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.034s
I20260129 02:58:54.328001 21225 consensus_queue.cc:1048] T febffe4a8b394368b69b1d164308fb9a P 197b880fa39d432a9b472845876bfaf0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260129 02:58:54.330327 21225 consensus_queue.cc:1048] T febffe4a8b394368b69b1d164308fb9a P 197b880fa39d432a9b472845876bfaf0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260129 02:58:54.336865 21134 raft_consensus.cc:1275] T 42d27a28d89b409e8807ec0efd97af69 P 7a9daf5d42a84615a6383cfec9a10608 [term 1 FOLLOWER]: Refusing update from remote peer 197b880fa39d432a9b472845876bfaf0: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260129 02:58:54.340736 20337 catalog_manager.cc:5654] T a644c02b16af47679824ccd060ac10fd P 7a9daf5d42a84615a6383cfec9a10608 reported cstate change: config changed from index 2 to 3, 7a9daf5d42a84615a6383cfec9a10608 (127.18.211.4) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "1c50de89a5a845668696dcb28c3badd6" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } attrs { replace: true } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } } }
I20260129 02:58:54.342037 20586 raft_consensus.cc:1275] T 42d27a28d89b409e8807ec0efd97af69 P 1c50de89a5a845668696dcb28c3badd6 [term 1 FOLLOWER]: Refusing update from remote peer 197b880fa39d432a9b472845876bfaf0: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260129 02:58:54.358771 21024 consensus_queue.cc:1048] T 42d27a28d89b409e8807ec0efd97af69 P 197b880fa39d432a9b472845876bfaf0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260129 02:58:54.360361 21024 consensus_queue.cc:1048] T 42d27a28d89b409e8807ec0efd97af69 P 197b880fa39d432a9b472845876bfaf0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260129 02:58:54.373199 21224 raft_consensus.cc:2955] T 42d27a28d89b409e8807ec0efd97af69 P 197b880fa39d432a9b472845876bfaf0 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 974b802141bd4f13beffef2aa03f88f9 (127.18.211.2) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } } }
I20260129 02:58:54.374925 21135 raft_consensus.cc:2955] T 42d27a28d89b409e8807ec0efd97af69 P 7a9daf5d42a84615a6383cfec9a10608 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 974b802141bd4f13beffef2aa03f88f9 (127.18.211.2) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } } }
I20260129 02:58:54.385617 20322 catalog_manager.cc:5167] ChangeConfig:REMOVE_PEER RPC for tablet 42d27a28d89b409e8807ec0efd97af69 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260129 02:58:54.390003 20587 consensus_queue.cc:237] T a644c02b16af47679824ccd060ac10fd P 1c50de89a5a845668696dcb28c3badd6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } }
I20260129 02:58:54.390324 21225 raft_consensus.cc:2955] T febffe4a8b394368b69b1d164308fb9a P 197b880fa39d432a9b472845876bfaf0 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 7a9daf5d42a84615a6383cfec9a10608 (127.18.211.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } attrs { replace: true } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } } }
I20260129 02:58:54.394270 20586 raft_consensus.cc:2955] T 42d27a28d89b409e8807ec0efd97af69 P 1c50de89a5a845668696dcb28c3badd6 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 974b802141bd4f13beffef2aa03f88f9 (127.18.211.2) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } } }
I20260129 02:58:54.397766 20585 raft_consensus.cc:2955] T febffe4a8b394368b69b1d164308fb9a P 1c50de89a5a845668696dcb28c3badd6 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 7a9daf5d42a84615a6383cfec9a10608 (127.18.211.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } attrs { replace: true } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } } }
I20260129 02:58:54.406746 20510 raft_consensus.cc:1275] T a644c02b16af47679824ccd060ac10fd P 974b802141bd4f13beffef2aa03f88f9 [term 1 FOLLOWER]: Refusing update from remote peer 1c50de89a5a845668696dcb28c3badd6: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260129 02:58:54.409446 21134 raft_consensus.cc:1275] T a644c02b16af47679824ccd060ac10fd P 7a9daf5d42a84615a6383cfec9a10608 [term 1 FOLLOWER]: Refusing update from remote peer 1c50de89a5a845668696dcb28c3badd6: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260129 02:58:54.403288 20336 catalog_manager.cc:5654] T 42d27a28d89b409e8807ec0efd97af69 P 7a9daf5d42a84615a6383cfec9a10608 reported cstate change: config changed from index 3 to 4, VOTER 974b802141bd4f13beffef2aa03f88f9 (127.18.211.2) evicted. New cstate: current_term: 1 leader_uuid: "197b880fa39d432a9b472845876bfaf0" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } } }
I20260129 02:58:54.407248 20584 consensus_queue.cc:237] T fd8311b779af4e1fa1402b2a02a602f2 P 1c50de89a5a845668696dcb28c3badd6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } }
I20260129 02:58:54.412887 21218 consensus_queue.cc:1048] T a644c02b16af47679824ccd060ac10fd P 1c50de89a5a845668696dcb28c3badd6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260129 02:58:54.419902 21218 consensus_queue.cc:1048] T a644c02b16af47679824ccd060ac10fd P 1c50de89a5a845668696dcb28c3badd6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260129 02:58:54.420548 21136 raft_consensus.cc:1275] T fd8311b779af4e1fa1402b2a02a602f2 P 7a9daf5d42a84615a6383cfec9a10608 [term 1 FOLLOWER]: Refusing update from remote peer 1c50de89a5a845668696dcb28c3badd6: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260129 02:58:54.422223 20510 raft_consensus.cc:2955] T febffe4a8b394368b69b1d164308fb9a P 974b802141bd4f13beffef2aa03f88f9 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 7a9daf5d42a84615a6383cfec9a10608 (127.18.211.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } attrs { replace: true } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } } }
I20260129 02:58:54.423511 20438 raft_consensus.cc:1275] T fd8311b779af4e1fa1402b2a02a602f2 P 197b880fa39d432a9b472845876bfaf0 [term 1 FOLLOWER]: Refusing update from remote peer 1c50de89a5a845668696dcb28c3badd6: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260129 02:58:54.424906 20982 consensus_queue.cc:1048] T fd8311b779af4e1fa1402b2a02a602f2 P 1c50de89a5a845668696dcb28c3badd6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260129 02:58:54.432662 21208 consensus_queue.cc:1048] T fd8311b779af4e1fa1402b2a02a602f2 P 1c50de89a5a845668696dcb28c3badd6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260129 02:58:54.432808 21209 raft_consensus.cc:2955] T a644c02b16af47679824ccd060ac10fd P 1c50de89a5a845668696dcb28c3badd6 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 197b880fa39d432a9b472845876bfaf0 (127.18.211.1) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } } }
I20260129 02:58:54.435642 20511 raft_consensus.cc:2955] T a644c02b16af47679824ccd060ac10fd P 974b802141bd4f13beffef2aa03f88f9 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 197b880fa39d432a9b472845876bfaf0 (127.18.211.1) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } } }
I20260129 02:58:54.436472 21208 raft_consensus.cc:2955] T fd8311b779af4e1fa1402b2a02a602f2 P 1c50de89a5a845668696dcb28c3badd6 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 974b802141bd4f13beffef2aa03f88f9 (127.18.211.2) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } } }
I20260129 02:58:54.440812 20438 raft_consensus.cc:2955] T fd8311b779af4e1fa1402b2a02a602f2 P 197b880fa39d432a9b472845876bfaf0 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 974b802141bd4f13beffef2aa03f88f9 (127.18.211.2) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } } }
I20260129 02:58:54.454181 20500 tablet_service.cc:1552] Processing DeleteTablet for tablet 42d27a28d89b409e8807ec0efd97af69 with delete_type TABLET_DATA_TOMBSTONED (TS 974b802141bd4f13beffef2aa03f88f9 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:57580
I20260129 02:58:54.454303 21136 raft_consensus.cc:2955] T a644c02b16af47679824ccd060ac10fd P 7a9daf5d42a84615a6383cfec9a10608 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 197b880fa39d432a9b472845876bfaf0 (127.18.211.1) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } } }
I20260129 02:58:54.456801 21133 raft_consensus.cc:2955] T fd8311b779af4e1fa1402b2a02a602f2 P 7a9daf5d42a84615a6383cfec9a10608 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 974b802141bd4f13beffef2aa03f88f9 (127.18.211.2) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } } }
I20260129 02:58:54.431119 21135 raft_consensus.cc:2955] T febffe4a8b394368b69b1d164308fb9a P 7a9daf5d42a84615a6383cfec9a10608 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 7a9daf5d42a84615a6383cfec9a10608 (127.18.211.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } attrs { replace: true } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } } }
I20260129 02:58:54.470037 20323 catalog_manager.cc:5167] ChangeConfig:REMOVE_PEER RPC for tablet a644c02b16af47679824ccd060ac10fd with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260129 02:58:54.466959 20338 catalog_manager.cc:5654] T febffe4a8b394368b69b1d164308fb9a P 974b802141bd4f13beffef2aa03f88f9 reported cstate change: config changed from index 2 to 3, 7a9daf5d42a84615a6383cfec9a10608 (127.18.211.4) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "197b880fa39d432a9b472845876bfaf0" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } attrs { replace: true } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } } }
I20260129 02:58:54.484576 21236 tablet_replica.cc:333] T 42d27a28d89b409e8807ec0efd97af69 P 974b802141bd4f13beffef2aa03f88f9: stopping tablet replica
I20260129 02:58:54.489792 20323 catalog_manager.cc:5167] ChangeConfig:REMOVE_PEER RPC for tablet fd8311b779af4e1fa1402b2a02a602f2 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260129 02:58:54.493294 21236 raft_consensus.cc:2243] T 42d27a28d89b409e8807ec0efd97af69 P 974b802141bd4f13beffef2aa03f88f9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260129 02:58:54.496903 21236 raft_consensus.cc:2272] T 42d27a28d89b409e8807ec0efd97af69 P 974b802141bd4f13beffef2aa03f88f9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260129 02:58:54.505236 21236 ts_tablet_manager.cc:1916] T 42d27a28d89b409e8807ec0efd97af69 P 974b802141bd4f13beffef2aa03f88f9: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260129 02:58:54.507573 20335 catalog_manager.cc:5654] T a644c02b16af47679824ccd060ac10fd P 7a9daf5d42a84615a6383cfec9a10608 reported cstate change: config changed from index 3 to 4, VOTER 197b880fa39d432a9b472845876bfaf0 (127.18.211.1) evicted. New cstate: current_term: 1 leader_uuid: "1c50de89a5a845668696dcb28c3badd6" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } } }
I20260129 02:58:54.513288 20335 catalog_manager.cc:5654] T fd8311b779af4e1fa1402b2a02a602f2 P 7a9daf5d42a84615a6383cfec9a10608 reported cstate change: config changed from index 3 to 4, VOTER 974b802141bd4f13beffef2aa03f88f9 (127.18.211.2) evicted. New cstate: current_term: 1 leader_uuid: "1c50de89a5a845668696dcb28c3badd6" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } } }
I20260129 02:58:54.514014 20438 consensus_queue.cc:237] T febffe4a8b394368b69b1d164308fb9a P 197b880fa39d432a9b472845876bfaf0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } }
I20260129 02:58:54.522281 20511 raft_consensus.cc:1275] T febffe4a8b394368b69b1d164308fb9a P 974b802141bd4f13beffef2aa03f88f9 [term 1 FOLLOWER]: Refusing update from remote peer 197b880fa39d432a9b472845876bfaf0: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260129 02:58:54.523023 21133 raft_consensus.cc:1275] T febffe4a8b394368b69b1d164308fb9a P 7a9daf5d42a84615a6383cfec9a10608 [term 1 FOLLOWER]: Refusing update from remote peer 197b880fa39d432a9b472845876bfaf0: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260129 02:58:54.523782 21210 consensus_queue.cc:1048] T febffe4a8b394368b69b1d164308fb9a P 197b880fa39d432a9b472845876bfaf0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260129 02:58:54.527839 21210 consensus_queue.cc:1048] T febffe4a8b394368b69b1d164308fb9a P 197b880fa39d432a9b472845876bfaf0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260129 02:58:54.531548 21024 raft_consensus.cc:2955] T febffe4a8b394368b69b1d164308fb9a P 197b880fa39d432a9b472845876bfaf0 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } } }
I20260129 02:58:54.538205 20431 tablet_service.cc:1552] Processing DeleteTablet for tablet a644c02b16af47679824ccd060ac10fd with delete_type TABLET_DATA_TOMBSTONED (TS 197b880fa39d432a9b472845876bfaf0 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:55666
I20260129 02:58:54.539124 20500 tablet_service.cc:1552] Processing DeleteTablet for tablet fd8311b779af4e1fa1402b2a02a602f2 with delete_type TABLET_DATA_TOMBSTONED (TS 974b802141bd4f13beffef2aa03f88f9 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:57580
I20260129 02:58:54.539613 21236 ts_tablet_manager.cc:1929] T 42d27a28d89b409e8807ec0efd97af69 P 974b802141bd4f13beffef2aa03f88f9: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20260129 02:58:54.540127 21236 log.cc:1199] T 42d27a28d89b409e8807ec0efd97af69 P 974b802141bd4f13beffef2aa03f88f9: Deleting WAL directory at /tmp/dist-test-taskrTP_Jo/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1769655460890924-19276-0/minicluster-data/ts-1-root/wals/42d27a28d89b409e8807ec0efd97af69
I20260129 02:58:54.540921 21133 raft_consensus.cc:2955] T febffe4a8b394368b69b1d164308fb9a P 7a9daf5d42a84615a6383cfec9a10608 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } } }
I20260129 02:58:54.541949 20325 catalog_manager.cc:4985] TS 974b802141bd4f13beffef2aa03f88f9 (127.18.211.2:34517): tablet 42d27a28d89b409e8807ec0efd97af69 (table test-workload [id=de73611ac8e3485a98aecc5add247032]) successfully deleted
I20260129 02:58:54.544441 21236 tablet_replica.cc:333] T fd8311b779af4e1fa1402b2a02a602f2 P 974b802141bd4f13beffef2aa03f88f9: stopping tablet replica
I20260129 02:58:54.546010 21236 raft_consensus.cc:2243] T fd8311b779af4e1fa1402b2a02a602f2 P 974b802141bd4f13beffef2aa03f88f9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260129 02:58:54.546778 21236 raft_consensus.cc:2272] T fd8311b779af4e1fa1402b2a02a602f2 P 974b802141bd4f13beffef2aa03f88f9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260129 02:58:54.550544 21236 ts_tablet_manager.cc:1916] T fd8311b779af4e1fa1402b2a02a602f2 P 974b802141bd4f13beffef2aa03f88f9: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260129 02:58:54.549787 20511 raft_consensus.cc:2955] T febffe4a8b394368b69b1d164308fb9a P 974b802141bd4f13beffef2aa03f88f9 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } } }
I20260129 02:58:54.552963 21237 tablet_replica.cc:333] T a644c02b16af47679824ccd060ac10fd P 197b880fa39d432a9b472845876bfaf0: stopping tablet replica
I20260129 02:58:54.553907 21237 raft_consensus.cc:2243] T a644c02b16af47679824ccd060ac10fd P 197b880fa39d432a9b472845876bfaf0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260129 02:58:54.554648 21237 raft_consensus.cc:2272] T a644c02b16af47679824ccd060ac10fd P 197b880fa39d432a9b472845876bfaf0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260129 02:58:54.553992 20335 catalog_manager.cc:5654] T febffe4a8b394368b69b1d164308fb9a P 7a9daf5d42a84615a6383cfec9a10608 reported cstate change: config changed from index 3 to 4, VOTER 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3) evicted. New cstate: current_term: 1 leader_uuid: "197b880fa39d432a9b472845876bfaf0" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } } }
I20260129 02:58:54.557433 20322 catalog_manager.cc:5167] ChangeConfig:REMOVE_PEER RPC for tablet febffe4a8b394368b69b1d164308fb9a with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260129 02:58:54.565574 21237 ts_tablet_manager.cc:1916] T a644c02b16af47679824ccd060ac10fd P 197b880fa39d432a9b472845876bfaf0: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260129 02:58:54.581527 21237 ts_tablet_manager.cc:1929] T a644c02b16af47679824ccd060ac10fd P 197b880fa39d432a9b472845876bfaf0: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20260129 02:58:54.583712 21237 log.cc:1199] T a644c02b16af47679824ccd060ac10fd P 197b880fa39d432a9b472845876bfaf0: Deleting WAL directory at /tmp/dist-test-taskrTP_Jo/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1769655460890924-19276-0/minicluster-data/ts-0-root/wals/a644c02b16af47679824ccd060ac10fd
I20260129 02:58:54.588135 20579 tablet_service.cc:1552] Processing DeleteTablet for tablet febffe4a8b394368b69b1d164308fb9a with delete_type TABLET_DATA_TOMBSTONED (TS 1c50de89a5a845668696dcb28c3badd6 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:35744
I20260129 02:58:54.596464 20322 catalog_manager.cc:4985] TS 197b880fa39d432a9b472845876bfaf0 (127.18.211.1:42027): tablet a644c02b16af47679824ccd060ac10fd (table test-workload [id=de73611ac8e3485a98aecc5add247032]) successfully deleted
I20260129 02:58:54.597069 21238 tablet_replica.cc:333] T febffe4a8b394368b69b1d164308fb9a P 1c50de89a5a845668696dcb28c3badd6: stopping tablet replica
I20260129 02:58:54.613085 21236 ts_tablet_manager.cc:1929] T fd8311b779af4e1fa1402b2a02a602f2 P 974b802141bd4f13beffef2aa03f88f9: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20260129 02:58:54.613567 21236 log.cc:1199] T fd8311b779af4e1fa1402b2a02a602f2 P 974b802141bd4f13beffef2aa03f88f9: Deleting WAL directory at /tmp/dist-test-taskrTP_Jo/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1769655460890924-19276-0/minicluster-data/ts-1-root/wals/fd8311b779af4e1fa1402b2a02a602f2
I20260129 02:58:54.615267 20325 catalog_manager.cc:4985] TS 974b802141bd4f13beffef2aa03f88f9 (127.18.211.2:34517): tablet fd8311b779af4e1fa1402b2a02a602f2 (table test-workload [id=de73611ac8e3485a98aecc5add247032]) successfully deleted
I20260129 02:58:54.616837 21238 raft_consensus.cc:2243] T febffe4a8b394368b69b1d164308fb9a P 1c50de89a5a845668696dcb28c3badd6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260129 02:58:54.617448 21238 raft_consensus.cc:2272] T febffe4a8b394368b69b1d164308fb9a P 1c50de89a5a845668696dcb28c3badd6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260129 02:58:54.620174 21238 ts_tablet_manager.cc:1916] T febffe4a8b394368b69b1d164308fb9a P 1c50de89a5a845668696dcb28c3badd6: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260129 02:58:54.631089 21238 ts_tablet_manager.cc:1929] T febffe4a8b394368b69b1d164308fb9a P 1c50de89a5a845668696dcb28c3badd6: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20260129 02:58:54.631390 21238 log.cc:1199] T febffe4a8b394368b69b1d164308fb9a P 1c50de89a5a845668696dcb28c3badd6: Deleting WAL directory at /tmp/dist-test-taskrTP_Jo/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1769655460890924-19276-0/minicluster-data/ts-2-root/wals/febffe4a8b394368b69b1d164308fb9a
I20260129 02:58:54.632735 20323 catalog_manager.cc:4985] TS 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3:37735): tablet febffe4a8b394368b69b1d164308fb9a (table test-workload [id=de73611ac8e3485a98aecc5add247032]) successfully deleted
I20260129 02:58:55.627650 20586 consensus_queue.cc:237] T 3e2d15f8640b407195210c476e09df91 P 1c50de89a5a845668696dcb28c3badd6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } attrs { replace: true } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: NON_VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: true } }
I20260129 02:58:55.638933 20435 raft_consensus.cc:1275] T 3e2d15f8640b407195210c476e09df91 P 197b880fa39d432a9b472845876bfaf0 [term 1 FOLLOWER]: Refusing update from remote peer 1c50de89a5a845668696dcb28c3badd6: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260129 02:58:55.641291 21218 consensus_queue.cc:1048] T 3e2d15f8640b407195210c476e09df91 P 1c50de89a5a845668696dcb28c3badd6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260129 02:58:55.642573 20511 raft_consensus.cc:1275] T 3e2d15f8640b407195210c476e09df91 P 974b802141bd4f13beffef2aa03f88f9 [term 1 FOLLOWER]: Refusing update from remote peer 1c50de89a5a845668696dcb28c3badd6: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20260129 02:58:55.643735 20554 consensus_peers.cc:597] T 3e2d15f8640b407195210c476e09df91 P 1c50de89a5a845668696dcb28c3badd6 -> Peer 7a9daf5d42a84615a6383cfec9a10608 (127.18.211.4:40439): Couldn't send request to peer 7a9daf5d42a84615a6383cfec9a10608. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 3e2d15f8640b407195210c476e09df91. This is attempt 1: this message will repeat every 5th retry.
I20260129 02:58:55.648443 21208 consensus_queue.cc:1048] T 3e2d15f8640b407195210c476e09df91 P 1c50de89a5a845668696dcb28c3badd6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260129 02:58:55.674505 21218 raft_consensus.cc:2955] T 3e2d15f8640b407195210c476e09df91 P 1c50de89a5a845668696dcb28c3badd6 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 7a9daf5d42a84615a6383cfec9a10608 (127.18.211.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } attrs { replace: true } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: NON_VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: true } } }
I20260129 02:58:55.679095 20435 raft_consensus.cc:2955] T 3e2d15f8640b407195210c476e09df91 P 197b880fa39d432a9b472845876bfaf0 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 7a9daf5d42a84615a6383cfec9a10608 (127.18.211.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } attrs { replace: true } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: NON_VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: true } } }
W20260129 02:58:55.743000 20556 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.211.2:34517, user_credentials={real_user=slave}} blocked reactor thread for 59966.1us
I20260129 02:58:55.754623 20514 raft_consensus.cc:2955] T 3e2d15f8640b407195210c476e09df91 P 974b802141bd4f13beffef2aa03f88f9 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 7a9daf5d42a84615a6383cfec9a10608 (127.18.211.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } attrs { replace: true } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: NON_VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: true } } }
I20260129 02:58:55.768574 20335 catalog_manager.cc:5654] T 3e2d15f8640b407195210c476e09df91 P 1c50de89a5a845668696dcb28c3badd6 reported cstate change: config changed from index -1 to 2, NON_VOTER 7a9daf5d42a84615a6383cfec9a10608 (127.18.211.4) added. New cstate: current_term: 1 leader_uuid: "1c50de89a5a845668696dcb28c3badd6" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: NON_VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20260129 02:58:55.786209 20514 consensus_queue.cc:237] T 87d5fe48a44a408aaf03e894a67ecb2f P 974b802141bd4f13beffef2aa03f88f9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } attrs { replace: true } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: NON_VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: true } }
I20260129 02:58:55.795408 20438 raft_consensus.cc:1275] T 87d5fe48a44a408aaf03e894a67ecb2f P 197b880fa39d432a9b472845876bfaf0 [term 1 FOLLOWER]: Refusing update from remote peer 974b802141bd4f13beffef2aa03f88f9: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260129 02:58:55.797434 21026 consensus_queue.cc:1048] T 87d5fe48a44a408aaf03e894a67ecb2f P 974b802141bd4f13beffef2aa03f88f9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260129 02:58:55.805440 20586 raft_consensus.cc:1275] T 87d5fe48a44a408aaf03e894a67ecb2f P 1c50de89a5a845668696dcb28c3badd6 [term 1 FOLLOWER]: Refusing update from remote peer 974b802141bd4f13beffef2aa03f88f9: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260129 02:58:55.809435 21239 consensus_queue.cc:1048] T 87d5fe48a44a408aaf03e894a67ecb2f P 974b802141bd4f13beffef2aa03f88f9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260129 02:58:55.821403 20435 raft_consensus.cc:2955] T 87d5fe48a44a408aaf03e894a67ecb2f P 197b880fa39d432a9b472845876bfaf0 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 7a9daf5d42a84615a6383cfec9a10608 (127.18.211.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } attrs { replace: true } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: NON_VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: true } } }
W20260129 02:58:55.823467 20480 consensus_peers.cc:597] T 87d5fe48a44a408aaf03e894a67ecb2f P 974b802141bd4f13beffef2aa03f88f9 -> Peer 7a9daf5d42a84615a6383cfec9a10608 (127.18.211.4:40439): Couldn't send request to peer 7a9daf5d42a84615a6383cfec9a10608. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 87d5fe48a44a408aaf03e894a67ecb2f. This is attempt 1: this message will repeat every 5th retry.
I20260129 02:58:55.819893 21239 raft_consensus.cc:2955] T 87d5fe48a44a408aaf03e894a67ecb2f P 974b802141bd4f13beffef2aa03f88f9 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 7a9daf5d42a84615a6383cfec9a10608 (127.18.211.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } attrs { replace: true } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: NON_VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: true } } }
I20260129 02:58:55.826967 20586 raft_consensus.cc:2955] T 87d5fe48a44a408aaf03e894a67ecb2f P 1c50de89a5a845668696dcb28c3badd6 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 7a9daf5d42a84615a6383cfec9a10608 (127.18.211.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } attrs { replace: true } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: NON_VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: true } } }
I20260129 02:58:55.836310 20438 consensus_queue.cc:237] T a609ac02fae84725ab743f95f51d33ce P 197b880fa39d432a9b472845876bfaf0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } attrs { replace: true } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: NON_VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: true } }
I20260129 02:58:55.840415 20338 catalog_manager.cc:5654] T 87d5fe48a44a408aaf03e894a67ecb2f P 197b880fa39d432a9b472845876bfaf0 reported cstate change: config changed from index -1 to 2, NON_VOTER 7a9daf5d42a84615a6383cfec9a10608 (127.18.211.4) added. New cstate: current_term: 1 leader_uuid: "974b802141bd4f13beffef2aa03f88f9" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } attrs { replace: true } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: NON_VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: true } } }
I20260129 02:58:55.845656 20586 raft_consensus.cc:1275] T a609ac02fae84725ab743f95f51d33ce P 1c50de89a5a845668696dcb28c3badd6 [term 1 FOLLOWER]: Refusing update from remote peer 197b880fa39d432a9b472845876bfaf0: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260129 02:58:55.846546 20514 raft_consensus.cc:1275] T a609ac02fae84725ab743f95f51d33ce P 974b802141bd4f13beffef2aa03f88f9 [term 1 FOLLOWER]: Refusing update from remote peer 197b880fa39d432a9b472845876bfaf0: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20260129 02:58:55.845641 20405 consensus_peers.cc:597] T a609ac02fae84725ab743f95f51d33ce P 197b880fa39d432a9b472845876bfaf0 -> Peer 7a9daf5d42a84615a6383cfec9a10608 (127.18.211.4:40439): Couldn't send request to peer 7a9daf5d42a84615a6383cfec9a10608. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: a609ac02fae84725ab743f95f51d33ce. This is attempt 1: this message will repeat every 5th retry.
I20260129 02:58:55.848104 21246 consensus_queue.cc:1048] T a609ac02fae84725ab743f95f51d33ce P 197b880fa39d432a9b472845876bfaf0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260129 02:58:55.849792 21024 consensus_queue.cc:1048] T a609ac02fae84725ab743f95f51d33ce P 197b880fa39d432a9b472845876bfaf0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260129 02:58:55.857403 21245 raft_consensus.cc:2955] T a609ac02fae84725ab743f95f51d33ce P 197b880fa39d432a9b472845876bfaf0 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 7a9daf5d42a84615a6383cfec9a10608 (127.18.211.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } attrs { replace: true } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: NON_VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: true } } }
I20260129 02:58:55.859200 20586 raft_consensus.cc:2955] T a609ac02fae84725ab743f95f51d33ce P 1c50de89a5a845668696dcb28c3badd6 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 7a9daf5d42a84615a6383cfec9a10608 (127.18.211.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } attrs { replace: true } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: NON_VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: true } } }
I20260129 02:58:55.874491 20514 raft_consensus.cc:2955] T a609ac02fae84725ab743f95f51d33ce P 974b802141bd4f13beffef2aa03f88f9 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 7a9daf5d42a84615a6383cfec9a10608 (127.18.211.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } attrs { replace: true } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: NON_VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: true } } }
I20260129 02:58:55.888159 20514 consensus_queue.cc:237] T 4f5a77e94fef4b16a2ba7b067c8b144a P 974b802141bd4f13beffef2aa03f88f9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } attrs { replace: true } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: NON_VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: true } }
I20260129 02:58:55.883483 20335 catalog_manager.cc:5654] T a609ac02fae84725ab743f95f51d33ce P 1c50de89a5a845668696dcb28c3badd6 reported cstate change: config changed from index -1 to 2, NON_VOTER 7a9daf5d42a84615a6383cfec9a10608 (127.18.211.4) added. New cstate: current_term: 1 leader_uuid: "197b880fa39d432a9b472845876bfaf0" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } attrs { replace: true } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: NON_VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: true } } }
I20260129 02:58:55.898880 20585 raft_consensus.cc:1275] T 4f5a77e94fef4b16a2ba7b067c8b144a P 1c50de89a5a845668696dcb28c3badd6 [term 1 FOLLOWER]: Refusing update from remote peer 974b802141bd4f13beffef2aa03f88f9: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260129 02:58:55.899398 20435 raft_consensus.cc:1275] T 4f5a77e94fef4b16a2ba7b067c8b144a P 197b880fa39d432a9b472845876bfaf0 [term 1 FOLLOWER]: Refusing update from remote peer 974b802141bd4f13beffef2aa03f88f9: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260129 02:58:55.900787 21026 consensus_queue.cc:1048] T 4f5a77e94fef4b16a2ba7b067c8b144a P 974b802141bd4f13beffef2aa03f88f9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
W20260129 02:58:55.905455 20480 consensus_peers.cc:597] T 4f5a77e94fef4b16a2ba7b067c8b144a P 974b802141bd4f13beffef2aa03f88f9 -> Peer 7a9daf5d42a84615a6383cfec9a10608 (127.18.211.4:40439): Couldn't send request to peer 7a9daf5d42a84615a6383cfec9a10608. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 4f5a77e94fef4b16a2ba7b067c8b144a. This is attempt 1: this message will repeat every 5th retry.
I20260129 02:58:55.906069 21026 consensus_queue.cc:1048] T 4f5a77e94fef4b16a2ba7b067c8b144a P 974b802141bd4f13beffef2aa03f88f9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260129 02:58:55.912288 21239 raft_consensus.cc:2955] T 4f5a77e94fef4b16a2ba7b067c8b144a P 974b802141bd4f13beffef2aa03f88f9 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 7a9daf5d42a84615a6383cfec9a10608 (127.18.211.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } attrs { replace: true } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: NON_VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: true } } }
I20260129 02:58:55.917554 20435 raft_consensus.cc:2955] T 4f5a77e94fef4b16a2ba7b067c8b144a P 197b880fa39d432a9b472845876bfaf0 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 7a9daf5d42a84615a6383cfec9a10608 (127.18.211.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } attrs { replace: true } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: NON_VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: true } } }
I20260129 02:58:55.919077 20586 raft_consensus.cc:2955] T 4f5a77e94fef4b16a2ba7b067c8b144a P 1c50de89a5a845668696dcb28c3badd6 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 7a9daf5d42a84615a6383cfec9a10608 (127.18.211.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } attrs { replace: true } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: NON_VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: true } } }
I20260129 02:58:55.952719 20336 catalog_manager.cc:5654] T 4f5a77e94fef4b16a2ba7b067c8b144a P 197b880fa39d432a9b472845876bfaf0 reported cstate change: config changed from index -1 to 2, NON_VOTER 7a9daf5d42a84615a6383cfec9a10608 (127.18.211.4) added. New cstate: current_term: 1 leader_uuid: "974b802141bd4f13beffef2aa03f88f9" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } attrs { replace: true } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: NON_VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: true } } }
I20260129 02:58:56.206947 21265 ts_tablet_manager.cc:933] T 87d5fe48a44a408aaf03e894a67ecb2f P 7a9daf5d42a84615a6383cfec9a10608: Initiating tablet copy from peer 974b802141bd4f13beffef2aa03f88f9 (127.18.211.2:34517)
I20260129 02:58:56.210172 21265 tablet_copy_client.cc:323] T 87d5fe48a44a408aaf03e894a67ecb2f P 7a9daf5d42a84615a6383cfec9a10608: tablet copy: Beginning tablet copy session from remote peer at address 127.18.211.2:34517
I20260129 02:58:56.233363 21267 ts_tablet_manager.cc:933] T 3e2d15f8640b407195210c476e09df91 P 7a9daf5d42a84615a6383cfec9a10608: Initiating tablet copy from peer 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3:37735)
I20260129 02:58:56.244753 20526 tablet_copy_service.cc:140] P 974b802141bd4f13beffef2aa03f88f9: Received BeginTabletCopySession request for tablet 87d5fe48a44a408aaf03e894a67ecb2f from peer 7a9daf5d42a84615a6383cfec9a10608 ({username='slave'} at 127.0.0.1:48544)
I20260129 02:58:56.245420 20526 tablet_copy_service.cc:161] P 974b802141bd4f13beffef2aa03f88f9: Beginning new tablet copy session on tablet 87d5fe48a44a408aaf03e894a67ecb2f from peer 7a9daf5d42a84615a6383cfec9a10608 at {username='slave'} at 127.0.0.1:48544: session id = 7a9daf5d42a84615a6383cfec9a10608-87d5fe48a44a408aaf03e894a67ecb2f
I20260129 02:58:56.252398 21267 tablet_copy_client.cc:323] T 3e2d15f8640b407195210c476e09df91 P 7a9daf5d42a84615a6383cfec9a10608: tablet copy: Beginning tablet copy session from remote peer at address 127.18.211.3:37735
I20260129 02:58:56.252333 20526 tablet_copy_source_session.cc:215] T 87d5fe48a44a408aaf03e894a67ecb2f P 974b802141bd4f13beffef2aa03f88f9: Tablet Copy: opened 0 blocks and 1 log segments
I20260129 02:58:56.254048 20600 tablet_copy_service.cc:140] P 1c50de89a5a845668696dcb28c3badd6: Received BeginTabletCopySession request for tablet 3e2d15f8640b407195210c476e09df91 from peer 7a9daf5d42a84615a6383cfec9a10608 ({username='slave'} at 127.0.0.1:48526)
I20260129 02:58:56.254582 20600 tablet_copy_service.cc:161] P 1c50de89a5a845668696dcb28c3badd6: Beginning new tablet copy session on tablet 3e2d15f8640b407195210c476e09df91 from peer 7a9daf5d42a84615a6383cfec9a10608 at {username='slave'} at 127.0.0.1:48526: session id = 7a9daf5d42a84615a6383cfec9a10608-3e2d15f8640b407195210c476e09df91
I20260129 02:58:56.256808 21265 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 87d5fe48a44a408aaf03e894a67ecb2f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260129 02:58:56.265483 20600 tablet_copy_source_session.cc:215] T 3e2d15f8640b407195210c476e09df91 P 1c50de89a5a845668696dcb28c3badd6: Tablet Copy: opened 0 blocks and 1 log segments
I20260129 02:58:56.269086 21267 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3e2d15f8640b407195210c476e09df91. 1 dirs total, 0 dirs full, 0 dirs failed
I20260129 02:58:56.278714 21265 tablet_copy_client.cc:806] T 87d5fe48a44a408aaf03e894a67ecb2f P 7a9daf5d42a84615a6383cfec9a10608: tablet copy: Starting download of 0 data blocks...
I20260129 02:58:56.279304 21265 tablet_copy_client.cc:670] T 87d5fe48a44a408aaf03e894a67ecb2f P 7a9daf5d42a84615a6383cfec9a10608: tablet copy: Starting download of 1 WAL segments...
I20260129 02:58:56.283010 21265 tablet_copy_client.cc:538] T 87d5fe48a44a408aaf03e894a67ecb2f P 7a9daf5d42a84615a6383cfec9a10608: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260129 02:58:56.285485 21267 tablet_copy_client.cc:806] T 3e2d15f8640b407195210c476e09df91 P 7a9daf5d42a84615a6383cfec9a10608: tablet copy: Starting download of 0 data blocks...
I20260129 02:58:56.286131 21267 tablet_copy_client.cc:670] T 3e2d15f8640b407195210c476e09df91 P 7a9daf5d42a84615a6383cfec9a10608: tablet copy: Starting download of 1 WAL segments...
I20260129 02:58:56.289798 21267 tablet_copy_client.cc:538] T 3e2d15f8640b407195210c476e09df91 P 7a9daf5d42a84615a6383cfec9a10608: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260129 02:58:56.293445 21265 tablet_bootstrap.cc:492] T 87d5fe48a44a408aaf03e894a67ecb2f P 7a9daf5d42a84615a6383cfec9a10608: Bootstrap starting.
I20260129 02:58:56.299773 21267 tablet_bootstrap.cc:492] T 3e2d15f8640b407195210c476e09df91 P 7a9daf5d42a84615a6383cfec9a10608: Bootstrap starting.
I20260129 02:58:56.325309 21265 tablet_bootstrap.cc:492] T 87d5fe48a44a408aaf03e894a67ecb2f P 7a9daf5d42a84615a6383cfec9a10608: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260129 02:58:56.326319 21265 tablet_bootstrap.cc:492] T 87d5fe48a44a408aaf03e894a67ecb2f P 7a9daf5d42a84615a6383cfec9a10608: Bootstrap complete.
I20260129 02:58:56.327035 21265 ts_tablet_manager.cc:1403] T 87d5fe48a44a408aaf03e894a67ecb2f P 7a9daf5d42a84615a6383cfec9a10608: Time spent bootstrapping tablet: real 0.034s user 0.029s sys 0.004s
I20260129 02:58:56.329960 21265 raft_consensus.cc:359] T 87d5fe48a44a408aaf03e894a67ecb2f P 7a9daf5d42a84615a6383cfec9a10608 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } attrs { replace: true } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: NON_VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: true } }
I20260129 02:58:56.330837 21265 raft_consensus.cc:740] T 87d5fe48a44a408aaf03e894a67ecb2f P 7a9daf5d42a84615a6383cfec9a10608 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 7a9daf5d42a84615a6383cfec9a10608, State: Initialized, Role: LEARNER
I20260129 02:58:56.331557 21265 consensus_queue.cc:260] T 87d5fe48a44a408aaf03e894a67ecb2f P 7a9daf5d42a84615a6383cfec9a10608 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } attrs { replace: true } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: NON_VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: true } }
I20260129 02:58:56.332229 21267 tablet_bootstrap.cc:492] T 3e2d15f8640b407195210c476e09df91 P 7a9daf5d42a84615a6383cfec9a10608: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260129 02:58:56.333300 21267 tablet_bootstrap.cc:492] T 3e2d15f8640b407195210c476e09df91 P 7a9daf5d42a84615a6383cfec9a10608: Bootstrap complete.
I20260129 02:58:56.334064 21267 ts_tablet_manager.cc:1403] T 3e2d15f8640b407195210c476e09df91 P 7a9daf5d42a84615a6383cfec9a10608: Time spent bootstrapping tablet: real 0.035s user 0.025s sys 0.008s
I20260129 02:58:56.334856 21265 ts_tablet_manager.cc:1434] T 87d5fe48a44a408aaf03e894a67ecb2f P 7a9daf5d42a84615a6383cfec9a10608: Time spent starting tablet: real 0.007s user 0.006s sys 0.000s
I20260129 02:58:56.336735 21267 raft_consensus.cc:359] T 3e2d15f8640b407195210c476e09df91 P 7a9daf5d42a84615a6383cfec9a10608 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } attrs { replace: true } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: NON_VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: true } }
I20260129 02:58:56.338001 21267 raft_consensus.cc:740] T 3e2d15f8640b407195210c476e09df91 P 7a9daf5d42a84615a6383cfec9a10608 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 7a9daf5d42a84615a6383cfec9a10608, State: Initialized, Role: LEARNER
I20260129 02:58:56.338795 20526 tablet_copy_service.cc:342] P 974b802141bd4f13beffef2aa03f88f9: Request end of tablet copy session 7a9daf5d42a84615a6383cfec9a10608-87d5fe48a44a408aaf03e894a67ecb2f received from {username='slave'} at 127.0.0.1:48544
I20260129 02:58:56.347234 20526 tablet_copy_service.cc:434] P 974b802141bd4f13beffef2aa03f88f9: ending tablet copy session 7a9daf5d42a84615a6383cfec9a10608-87d5fe48a44a408aaf03e894a67ecb2f on tablet 87d5fe48a44a408aaf03e894a67ecb2f with peer 7a9daf5d42a84615a6383cfec9a10608
I20260129 02:58:56.338762 21267 consensus_queue.cc:260] T 3e2d15f8640b407195210c476e09df91 P 7a9daf5d42a84615a6383cfec9a10608 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } attrs { replace: true } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: NON_VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: true } }
I20260129 02:58:56.349196 21267 ts_tablet_manager.cc:1434] T 3e2d15f8640b407195210c476e09df91 P 7a9daf5d42a84615a6383cfec9a10608: Time spent starting tablet: real 0.015s user 0.006s sys 0.008s
I20260129 02:58:56.350781 20600 tablet_copy_service.cc:342] P 1c50de89a5a845668696dcb28c3badd6: Request end of tablet copy session 7a9daf5d42a84615a6383cfec9a10608-3e2d15f8640b407195210c476e09df91 received from {username='slave'} at 127.0.0.1:48526
I20260129 02:58:56.351189 20600 tablet_copy_service.cc:434] P 1c50de89a5a845668696dcb28c3badd6: ending tablet copy session 7a9daf5d42a84615a6383cfec9a10608-3e2d15f8640b407195210c476e09df91 on tablet 3e2d15f8640b407195210c476e09df91 with peer 7a9daf5d42a84615a6383cfec9a10608
I20260129 02:58:56.450941 21265 ts_tablet_manager.cc:933] T a609ac02fae84725ab743f95f51d33ce P 7a9daf5d42a84615a6383cfec9a10608: Initiating tablet copy from peer 197b880fa39d432a9b472845876bfaf0 (127.18.211.1:42027)
I20260129 02:58:56.453277 21265 tablet_copy_client.cc:323] T a609ac02fae84725ab743f95f51d33ce P 7a9daf5d42a84615a6383cfec9a10608: tablet copy: Beginning tablet copy session from remote peer at address 127.18.211.1:42027
I20260129 02:58:56.454897 20451 tablet_copy_service.cc:140] P 197b880fa39d432a9b472845876bfaf0: Received BeginTabletCopySession request for tablet a609ac02fae84725ab743f95f51d33ce from peer 7a9daf5d42a84615a6383cfec9a10608 ({username='slave'} at 127.0.0.1:48356)
I20260129 02:58:56.455478 20451 tablet_copy_service.cc:161] P 197b880fa39d432a9b472845876bfaf0: Beginning new tablet copy session on tablet a609ac02fae84725ab743f95f51d33ce from peer 7a9daf5d42a84615a6383cfec9a10608 at {username='slave'} at 127.0.0.1:48356: session id = 7a9daf5d42a84615a6383cfec9a10608-a609ac02fae84725ab743f95f51d33ce
I20260129 02:58:56.462095 20451 tablet_copy_source_session.cc:215] T a609ac02fae84725ab743f95f51d33ce P 197b880fa39d432a9b472845876bfaf0: Tablet Copy: opened 0 blocks and 1 log segments
I20260129 02:58:56.465462 21265 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a609ac02fae84725ab743f95f51d33ce. 1 dirs total, 0 dirs full, 0 dirs failed
I20260129 02:58:56.479344 21265 tablet_copy_client.cc:806] T a609ac02fae84725ab743f95f51d33ce P 7a9daf5d42a84615a6383cfec9a10608: tablet copy: Starting download of 0 data blocks...
I20260129 02:58:56.479939 21265 tablet_copy_client.cc:670] T a609ac02fae84725ab743f95f51d33ce P 7a9daf5d42a84615a6383cfec9a10608: tablet copy: Starting download of 1 WAL segments...
I20260129 02:58:56.483388 21265 tablet_copy_client.cc:538] T a609ac02fae84725ab743f95f51d33ce P 7a9daf5d42a84615a6383cfec9a10608: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260129 02:58:56.491974 21265 tablet_bootstrap.cc:492] T a609ac02fae84725ab743f95f51d33ce P 7a9daf5d42a84615a6383cfec9a10608: Bootstrap starting.
I20260129 02:58:56.506512 21267 ts_tablet_manager.cc:933] T 4f5a77e94fef4b16a2ba7b067c8b144a P 7a9daf5d42a84615a6383cfec9a10608: Initiating tablet copy from peer 974b802141bd4f13beffef2aa03f88f9 (127.18.211.2:34517)
I20260129 02:58:56.517377 21265 tablet_bootstrap.cc:492] T a609ac02fae84725ab743f95f51d33ce P 7a9daf5d42a84615a6383cfec9a10608: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260129 02:58:56.518229 21265 tablet_bootstrap.cc:492] T a609ac02fae84725ab743f95f51d33ce P 7a9daf5d42a84615a6383cfec9a10608: Bootstrap complete.
I20260129 02:58:56.518879 21265 ts_tablet_manager.cc:1403] T a609ac02fae84725ab743f95f51d33ce P 7a9daf5d42a84615a6383cfec9a10608: Time spent bootstrapping tablet: real 0.027s user 0.019s sys 0.004s
I20260129 02:58:56.520632 21267 tablet_copy_client.cc:323] T 4f5a77e94fef4b16a2ba7b067c8b144a P 7a9daf5d42a84615a6383cfec9a10608: tablet copy: Beginning tablet copy session from remote peer at address 127.18.211.2:34517
I20260129 02:58:56.522353 20526 tablet_copy_service.cc:140] P 974b802141bd4f13beffef2aa03f88f9: Received BeginTabletCopySession request for tablet 4f5a77e94fef4b16a2ba7b067c8b144a from peer 7a9daf5d42a84615a6383cfec9a10608 ({username='slave'} at 127.0.0.1:48544)
I20260129 02:58:56.522858 20526 tablet_copy_service.cc:161] P 974b802141bd4f13beffef2aa03f88f9: Beginning new tablet copy session on tablet 4f5a77e94fef4b16a2ba7b067c8b144a from peer 7a9daf5d42a84615a6383cfec9a10608 at {username='slave'} at 127.0.0.1:48544: session id = 7a9daf5d42a84615a6383cfec9a10608-4f5a77e94fef4b16a2ba7b067c8b144a
I20260129 02:58:56.526625 21265 raft_consensus.cc:359] T a609ac02fae84725ab743f95f51d33ce P 7a9daf5d42a84615a6383cfec9a10608 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } attrs { replace: true } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: NON_VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: true } }
I20260129 02:58:56.527518 21265 raft_consensus.cc:740] T a609ac02fae84725ab743f95f51d33ce P 7a9daf5d42a84615a6383cfec9a10608 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 7a9daf5d42a84615a6383cfec9a10608, State: Initialized, Role: LEARNER
I20260129 02:58:56.528226 21265 consensus_queue.cc:260] T a609ac02fae84725ab743f95f51d33ce P 7a9daf5d42a84615a6383cfec9a10608 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } attrs { replace: true } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: NON_VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: true } }
I20260129 02:58:56.530612 21265 ts_tablet_manager.cc:1434] T a609ac02fae84725ab743f95f51d33ce P 7a9daf5d42a84615a6383cfec9a10608: Time spent starting tablet: real 0.011s user 0.004s sys 0.000s
I20260129 02:58:56.533607 20451 tablet_copy_service.cc:342] P 197b880fa39d432a9b472845876bfaf0: Request end of tablet copy session 7a9daf5d42a84615a6383cfec9a10608-a609ac02fae84725ab743f95f51d33ce received from {username='slave'} at 127.0.0.1:48356
I20260129 02:58:56.534017 20451 tablet_copy_service.cc:434] P 197b880fa39d432a9b472845876bfaf0: ending tablet copy session 7a9daf5d42a84615a6383cfec9a10608-a609ac02fae84725ab743f95f51d33ce on tablet a609ac02fae84725ab743f95f51d33ce with peer 7a9daf5d42a84615a6383cfec9a10608
I20260129 02:58:56.535745 20526 tablet_copy_source_session.cc:215] T 4f5a77e94fef4b16a2ba7b067c8b144a P 974b802141bd4f13beffef2aa03f88f9: Tablet Copy: opened 0 blocks and 1 log segments
I20260129 02:58:56.543378 21267 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4f5a77e94fef4b16a2ba7b067c8b144a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260129 02:58:56.557384 21267 tablet_copy_client.cc:806] T 4f5a77e94fef4b16a2ba7b067c8b144a P 7a9daf5d42a84615a6383cfec9a10608: tablet copy: Starting download of 0 data blocks...
I20260129 02:58:56.558025 21267 tablet_copy_client.cc:670] T 4f5a77e94fef4b16a2ba7b067c8b144a P 7a9daf5d42a84615a6383cfec9a10608: tablet copy: Starting download of 1 WAL segments...
I20260129 02:58:56.561401 21267 tablet_copy_client.cc:538] T 4f5a77e94fef4b16a2ba7b067c8b144a P 7a9daf5d42a84615a6383cfec9a10608: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260129 02:58:56.569568 21267 tablet_bootstrap.cc:492] T 4f5a77e94fef4b16a2ba7b067c8b144a P 7a9daf5d42a84615a6383cfec9a10608: Bootstrap starting.
I20260129 02:58:56.589972 21133 raft_consensus.cc:1217] T 3e2d15f8640b407195210c476e09df91 P 7a9daf5d42a84615a6383cfec9a10608 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20260129 02:58:56.591120 21267 tablet_bootstrap.cc:492] T 4f5a77e94fef4b16a2ba7b067c8b144a P 7a9daf5d42a84615a6383cfec9a10608: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260129 02:58:56.592108 21267 tablet_bootstrap.cc:492] T 4f5a77e94fef4b16a2ba7b067c8b144a P 7a9daf5d42a84615a6383cfec9a10608: Bootstrap complete.
I20260129 02:58:56.592818 21267 ts_tablet_manager.cc:1403] T 4f5a77e94fef4b16a2ba7b067c8b144a P 7a9daf5d42a84615a6383cfec9a10608: Time spent bootstrapping tablet: real 0.023s user 0.020s sys 0.001s
I20260129 02:58:56.597182 21267 raft_consensus.cc:359] T 4f5a77e94fef4b16a2ba7b067c8b144a P 7a9daf5d42a84615a6383cfec9a10608 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } attrs { replace: true } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: NON_VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: true } }
I20260129 02:58:56.604113 21267 raft_consensus.cc:740] T 4f5a77e94fef4b16a2ba7b067c8b144a P 7a9daf5d42a84615a6383cfec9a10608 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 7a9daf5d42a84615a6383cfec9a10608, State: Initialized, Role: LEARNER
I20260129 02:58:56.604982 21267 consensus_queue.cc:260] T 4f5a77e94fef4b16a2ba7b067c8b144a P 7a9daf5d42a84615a6383cfec9a10608 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } attrs { replace: true } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: NON_VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: true } }
I20260129 02:58:56.607627 21267 ts_tablet_manager.cc:1434] T 4f5a77e94fef4b16a2ba7b067c8b144a P 7a9daf5d42a84615a6383cfec9a10608: Time spent starting tablet: real 0.014s user 0.006s sys 0.000s
I20260129 02:58:56.609301 20526 tablet_copy_service.cc:342] P 974b802141bd4f13beffef2aa03f88f9: Request end of tablet copy session 7a9daf5d42a84615a6383cfec9a10608-4f5a77e94fef4b16a2ba7b067c8b144a received from {username='slave'} at 127.0.0.1:48544
I20260129 02:58:56.609712 20526 tablet_copy_service.cc:434] P 974b802141bd4f13beffef2aa03f88f9: ending tablet copy session 7a9daf5d42a84615a6383cfec9a10608-4f5a77e94fef4b16a2ba7b067c8b144a on tablet 4f5a77e94fef4b16a2ba7b067c8b144a with peer 7a9daf5d42a84615a6383cfec9a10608
I20260129 02:58:56.633991 21133 raft_consensus.cc:1217] T 87d5fe48a44a408aaf03e894a67ecb2f P 7a9daf5d42a84615a6383cfec9a10608 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20260129 02:58:56.974098 21133 raft_consensus.cc:1217] T a609ac02fae84725ab743f95f51d33ce P 7a9daf5d42a84615a6383cfec9a10608 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20260129 02:58:57.052525 21133 raft_consensus.cc:1217] T 4f5a77e94fef4b16a2ba7b067c8b144a P 7a9daf5d42a84615a6383cfec9a10608 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20260129 02:58:57.098619 21218 raft_consensus.cc:1064] T 3e2d15f8640b407195210c476e09df91 P 1c50de89a5a845668696dcb28c3badd6: attempting to promote NON_VOTER 7a9daf5d42a84615a6383cfec9a10608 to VOTER
I20260129 02:58:57.101121 21258 raft_consensus.cc:1064] T 87d5fe48a44a408aaf03e894a67ecb2f P 974b802141bd4f13beffef2aa03f88f9: attempting to promote NON_VOTER 7a9daf5d42a84615a6383cfec9a10608 to VOTER
I20260129 02:58:57.101082 21218 consensus_queue.cc:237] T 3e2d15f8640b407195210c476e09df91 P 1c50de89a5a845668696dcb28c3badd6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } attrs { replace: true } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } }
I20260129 02:58:57.103317 21258 consensus_queue.cc:237] T 87d5fe48a44a408aaf03e894a67ecb2f P 974b802141bd4f13beffef2aa03f88f9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } attrs { replace: true } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } }
I20260129 02:58:57.108860 20510 raft_consensus.cc:1275] T 3e2d15f8640b407195210c476e09df91 P 974b802141bd4f13beffef2aa03f88f9 [term 1 FOLLOWER]: Refusing update from remote peer 1c50de89a5a845668696dcb28c3badd6: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260129 02:58:57.109099 21133 raft_consensus.cc:1275] T 3e2d15f8640b407195210c476e09df91 P 7a9daf5d42a84615a6383cfec9a10608 [term 1 LEARNER]: Refusing update from remote peer 1c50de89a5a845668696dcb28c3badd6: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260129 02:58:57.110143 20438 raft_consensus.cc:1275] T 3e2d15f8640b407195210c476e09df91 P 197b880fa39d432a9b472845876bfaf0 [term 1 FOLLOWER]: Refusing update from remote peer 1c50de89a5a845668696dcb28c3badd6: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260129 02:58:57.110595 21280 consensus_queue.cc:1048] T 3e2d15f8640b407195210c476e09df91 P 1c50de89a5a845668696dcb28c3badd6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260129 02:58:57.113938 21133 raft_consensus.cc:1275] T 87d5fe48a44a408aaf03e894a67ecb2f P 7a9daf5d42a84615a6383cfec9a10608 [term 1 LEARNER]: Refusing update from remote peer 974b802141bd4f13beffef2aa03f88f9: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260129 02:58:57.115226 21239 consensus_queue.cc:1048] T 87d5fe48a44a408aaf03e894a67ecb2f P 974b802141bd4f13beffef2aa03f88f9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260129 02:58:57.116183 21241 consensus_queue.cc:1048] T 3e2d15f8640b407195210c476e09df91 P 1c50de89a5a845668696dcb28c3badd6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260129 02:58:57.119372 21279 consensus_queue.cc:1048] T 3e2d15f8640b407195210c476e09df91 P 1c50de89a5a845668696dcb28c3badd6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260129 02:58:57.122002 20435 raft_consensus.cc:1275] T 87d5fe48a44a408aaf03e894a67ecb2f P 197b880fa39d432a9b472845876bfaf0 [term 1 FOLLOWER]: Refusing update from remote peer 974b802141bd4f13beffef2aa03f88f9: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260129 02:58:57.124843 21239 consensus_queue.cc:1048] T 87d5fe48a44a408aaf03e894a67ecb2f P 974b802141bd4f13beffef2aa03f88f9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260129 02:58:57.132644 20586 raft_consensus.cc:1275] T 87d5fe48a44a408aaf03e894a67ecb2f P 1c50de89a5a845668696dcb28c3badd6 [term 1 FOLLOWER]: Refusing update from remote peer 974b802141bd4f13beffef2aa03f88f9: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260129 02:58:57.134824 21239 consensus_queue.cc:1048] T 87d5fe48a44a408aaf03e894a67ecb2f P 974b802141bd4f13beffef2aa03f88f9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260129 02:58:57.171828 21279 raft_consensus.cc:2955] T 3e2d15f8640b407195210c476e09df91 P 1c50de89a5a845668696dcb28c3badd6 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 7a9daf5d42a84615a6383cfec9a10608 (127.18.211.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } attrs { replace: true } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } } }
I20260129 02:58:57.173810 20516 raft_consensus.cc:2955] T 3e2d15f8640b407195210c476e09df91 P 974b802141bd4f13beffef2aa03f88f9 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 7a9daf5d42a84615a6383cfec9a10608 (127.18.211.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } attrs { replace: true } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } } }
I20260129 02:58:57.186409 21239 raft_consensus.cc:2955] T 87d5fe48a44a408aaf03e894a67ecb2f P 974b802141bd4f13beffef2aa03f88f9 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 7a9daf5d42a84615a6383cfec9a10608 (127.18.211.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } attrs { replace: true } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } } }
I20260129 02:58:57.193936 21136 raft_consensus.cc:2955] T 3e2d15f8640b407195210c476e09df91 P 7a9daf5d42a84615a6383cfec9a10608 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 7a9daf5d42a84615a6383cfec9a10608 (127.18.211.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } attrs { replace: true } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } } }
I20260129 02:58:57.203481 20586 raft_consensus.cc:2955] T 87d5fe48a44a408aaf03e894a67ecb2f P 1c50de89a5a845668696dcb28c3badd6 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 7a9daf5d42a84615a6383cfec9a10608 (127.18.211.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } attrs { replace: true } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } } }
I20260129 02:58:57.222389 20338 catalog_manager.cc:5654] T 87d5fe48a44a408aaf03e894a67ecb2f P 974b802141bd4f13beffef2aa03f88f9 reported cstate change: config changed from index 2 to 3, 7a9daf5d42a84615a6383cfec9a10608 (127.18.211.4) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "974b802141bd4f13beffef2aa03f88f9" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20260129 02:58:57.250036 20435 raft_consensus.cc:2955] T 87d5fe48a44a408aaf03e894a67ecb2f P 197b880fa39d432a9b472845876bfaf0 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 7a9daf5d42a84615a6383cfec9a10608 (127.18.211.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } attrs { replace: true } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } } }
I20260129 02:58:57.251755 20335 catalog_manager.cc:5654] T 3e2d15f8640b407195210c476e09df91 P 1c50de89a5a845668696dcb28c3badd6 reported cstate change: config changed from index 2 to 3, 7a9daf5d42a84615a6383cfec9a10608 (127.18.211.4) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "1c50de89a5a845668696dcb28c3badd6" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260129 02:58:57.266089 20438 raft_consensus.cc:2955] T 3e2d15f8640b407195210c476e09df91 P 197b880fa39d432a9b472845876bfaf0 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 7a9daf5d42a84615a6383cfec9a10608 (127.18.211.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } attrs { replace: true } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } } }
I20260129 02:58:57.342408 20586 consensus_queue.cc:237] T 3e2d15f8640b407195210c476e09df91 P 1c50de89a5a845668696dcb28c3badd6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } }
I20260129 02:58:57.373835 21136 raft_consensus.cc:1275] T 3e2d15f8640b407195210c476e09df91 P 7a9daf5d42a84615a6383cfec9a10608 [term 1 FOLLOWER]: Refusing update from remote peer 1c50de89a5a845668696dcb28c3badd6: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260129 02:58:57.375337 21279 consensus_queue.cc:1048] T 3e2d15f8640b407195210c476e09df91 P 1c50de89a5a845668696dcb28c3badd6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260129 02:58:57.375710 21133 raft_consensus.cc:2955] T 87d5fe48a44a408aaf03e894a67ecb2f P 7a9daf5d42a84615a6383cfec9a10608 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 7a9daf5d42a84615a6383cfec9a10608 (127.18.211.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } attrs { replace: true } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } } }
I20260129 02:58:57.383265 21241 raft_consensus.cc:2955] T 3e2d15f8640b407195210c476e09df91 P 1c50de89a5a845668696dcb28c3badd6 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 197b880fa39d432a9b472845876bfaf0 (127.18.211.1) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } } }
I20260129 02:58:57.392689 20511 raft_consensus.cc:1275] T 3e2d15f8640b407195210c476e09df91 P 974b802141bd4f13beffef2aa03f88f9 [term 1 FOLLOWER]: Refusing update from remote peer 1c50de89a5a845668696dcb28c3badd6: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260129 02:58:57.389163 21135 raft_consensus.cc:2955] T 3e2d15f8640b407195210c476e09df91 P 7a9daf5d42a84615a6383cfec9a10608 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 197b880fa39d432a9b472845876bfaf0 (127.18.211.1) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } } }
I20260129 02:58:57.413347 20516 consensus_queue.cc:237] T 87d5fe48a44a408aaf03e894a67ecb2f P 974b802141bd4f13beffef2aa03f88f9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } }
I20260129 02:58:57.421445 20323 catalog_manager.cc:5167] ChangeConfig:REMOVE_PEER RPC for tablet 3e2d15f8640b407195210c476e09df91 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260129 02:58:57.422169 21279 consensus_queue.cc:1048] T 3e2d15f8640b407195210c476e09df91 P 1c50de89a5a845668696dcb28c3badd6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260129 02:58:57.426193 21135 raft_consensus.cc:1275] T 87d5fe48a44a408aaf03e894a67ecb2f P 7a9daf5d42a84615a6383cfec9a10608 [term 1 FOLLOWER]: Refusing update from remote peer 974b802141bd4f13beffef2aa03f88f9: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260129 02:58:57.428711 20511 raft_consensus.cc:2955] T 3e2d15f8640b407195210c476e09df91 P 974b802141bd4f13beffef2aa03f88f9 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 197b880fa39d432a9b472845876bfaf0 (127.18.211.1) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } } }
I20260129 02:58:57.431119 20438 raft_consensus.cc:1275] T 87d5fe48a44a408aaf03e894a67ecb2f P 197b880fa39d432a9b472845876bfaf0 [term 1 FOLLOWER]: Refusing update from remote peer 974b802141bd4f13beffef2aa03f88f9: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260129 02:58:57.431622 21026 consensus_queue.cc:1048] T 87d5fe48a44a408aaf03e894a67ecb2f P 974b802141bd4f13beffef2aa03f88f9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260129 02:58:57.432822 20338 catalog_manager.cc:5654] T 3e2d15f8640b407195210c476e09df91 P 7a9daf5d42a84615a6383cfec9a10608 reported cstate change: config changed from index 3 to 4, VOTER 197b880fa39d432a9b472845876bfaf0 (127.18.211.1) evicted. New cstate: current_term: 1 leader_uuid: "1c50de89a5a845668696dcb28c3badd6" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } } }
I20260129 02:58:57.434783 21258 consensus_queue.cc:1048] T 87d5fe48a44a408aaf03e894a67ecb2f P 974b802141bd4f13beffef2aa03f88f9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260129 02:58:57.438694 21024 raft_consensus.cc:1064] T a609ac02fae84725ab743f95f51d33ce P 197b880fa39d432a9b472845876bfaf0: attempting to promote NON_VOTER 7a9daf5d42a84615a6383cfec9a10608 to VOTER
I20260129 02:58:57.440338 21239 raft_consensus.cc:1064] T 4f5a77e94fef4b16a2ba7b067c8b144a P 974b802141bd4f13beffef2aa03f88f9: attempting to promote NON_VOTER 7a9daf5d42a84615a6383cfec9a10608 to VOTER
I20260129 02:58:57.443017 21239 consensus_queue.cc:237] T 4f5a77e94fef4b16a2ba7b067c8b144a P 974b802141bd4f13beffef2aa03f88f9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } attrs { replace: true } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } }
I20260129 02:58:57.444056 21026 raft_consensus.cc:2955] T 87d5fe48a44a408aaf03e894a67ecb2f P 974b802141bd4f13beffef2aa03f88f9 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } } }
I20260129 02:58:57.445225 21024 consensus_queue.cc:237] T a609ac02fae84725ab743f95f51d33ce P 197b880fa39d432a9b472845876bfaf0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } attrs { replace: true } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } }
I20260129 02:58:57.444058 21136 raft_consensus.cc:2955] T 87d5fe48a44a408aaf03e894a67ecb2f P 7a9daf5d42a84615a6383cfec9a10608 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } } }
I20260129 02:58:57.455406 20431 tablet_service.cc:1552] Processing DeleteTablet for tablet 3e2d15f8640b407195210c476e09df91 with delete_type TABLET_DATA_TOMBSTONED (TS 197b880fa39d432a9b472845876bfaf0 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:55666
I20260129 02:58:57.473842 21297 tablet_replica.cc:333] T 3e2d15f8640b407195210c476e09df91 P 197b880fa39d432a9b472845876bfaf0: stopping tablet replica
I20260129 02:58:57.479707 21297 raft_consensus.cc:2243] T 3e2d15f8640b407195210c476e09df91 P 197b880fa39d432a9b472845876bfaf0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260129 02:58:57.481209 21297 raft_consensus.cc:2272] T 3e2d15f8640b407195210c476e09df91 P 197b880fa39d432a9b472845876bfaf0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260129 02:58:57.492525 20514 raft_consensus.cc:1275] T a609ac02fae84725ab743f95f51d33ce P 974b802141bd4f13beffef2aa03f88f9 [term 1 FOLLOWER]: Refusing update from remote peer 197b880fa39d432a9b472845876bfaf0: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
W20260129 02:58:57.492859 20479 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.211.1:42027, user_credentials={real_user=slave}} blocked reactor thread for 46358.3us
I20260129 02:58:57.492950 20586 raft_consensus.cc:1275] T a609ac02fae84725ab743f95f51d33ce P 1c50de89a5a845668696dcb28c3badd6 [term 1 FOLLOWER]: Refusing update from remote peer 197b880fa39d432a9b472845876bfaf0: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260129 02:58:57.494455 21135 raft_consensus.cc:1275] T a609ac02fae84725ab743f95f51d33ce P 7a9daf5d42a84615a6383cfec9a10608 [term 1 LEARNER]: Refusing update from remote peer 197b880fa39d432a9b472845876bfaf0: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260129 02:58:57.496723 20438 raft_consensus.cc:1275] T 4f5a77e94fef4b16a2ba7b067c8b144a P 197b880fa39d432a9b472845876bfaf0 [term 1 FOLLOWER]: Refusing update from remote peer 974b802141bd4f13beffef2aa03f88f9: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260129 02:58:57.498657 21024 consensus_queue.cc:1048] T a609ac02fae84725ab743f95f51d33ce P 197b880fa39d432a9b472845876bfaf0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260129 02:58:57.499636 21289 consensus_queue.cc:1048] T a609ac02fae84725ab743f95f51d33ce P 197b880fa39d432a9b472845876bfaf0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260129 02:58:57.501386 21278 consensus_queue.cc:1048] T a609ac02fae84725ab743f95f51d33ce P 197b880fa39d432a9b472845876bfaf0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260129 02:58:57.503185 20325 catalog_manager.cc:5167] ChangeConfig:REMOVE_PEER RPC for tablet 87d5fe48a44a408aaf03e894a67ecb2f with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260129 02:58:57.504704 20438 raft_consensus.cc:2955] T 87d5fe48a44a408aaf03e894a67ecb2f P 197b880fa39d432a9b472845876bfaf0 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } } }
I20260129 02:58:57.508153 20335 catalog_manager.cc:5654] T 87d5fe48a44a408aaf03e894a67ecb2f P 974b802141bd4f13beffef2aa03f88f9 reported cstate change: config changed from index 3 to 4, VOTER 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3) evicted. New cstate: current_term: 1 leader_uuid: "974b802141bd4f13beffef2aa03f88f9" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260129 02:58:57.512018 21239 consensus_queue.cc:1048] T 4f5a77e94fef4b16a2ba7b067c8b144a P 974b802141bd4f13beffef2aa03f88f9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260129 02:58:57.528750 20579 tablet_service.cc:1552] Processing DeleteTablet for tablet 87d5fe48a44a408aaf03e894a67ecb2f with delete_type TABLET_DATA_TOMBSTONED (TS 1c50de89a5a845668696dcb28c3badd6 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:35744
I20260129 02:58:57.528908 21297 ts_tablet_manager.cc:1916] T 3e2d15f8640b407195210c476e09df91 P 197b880fa39d432a9b472845876bfaf0: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260129 02:58:57.545883 21136 raft_consensus.cc:1275] T 4f5a77e94fef4b16a2ba7b067c8b144a P 7a9daf5d42a84615a6383cfec9a10608 [term 1 LEARNER]: Refusing update from remote peer 974b802141bd4f13beffef2aa03f88f9: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260129 02:58:57.548684 20587 raft_consensus.cc:1275] T 4f5a77e94fef4b16a2ba7b067c8b144a P 1c50de89a5a845668696dcb28c3badd6 [term 1 FOLLOWER]: Refusing update from remote peer 974b802141bd4f13beffef2aa03f88f9: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260129 02:58:57.563349 21297 ts_tablet_manager.cc:1929] T 3e2d15f8640b407195210c476e09df91 P 197b880fa39d432a9b472845876bfaf0: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20260129 02:58:57.563813 21297 log.cc:1199] T 3e2d15f8640b407195210c476e09df91 P 197b880fa39d432a9b472845876bfaf0: Deleting WAL directory at /tmp/dist-test-taskrTP_Jo/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1769655460890924-19276-0/minicluster-data/ts-0-root/wals/3e2d15f8640b407195210c476e09df91
I20260129 02:58:57.566465 20322 catalog_manager.cc:4985] TS 197b880fa39d432a9b472845876bfaf0 (127.18.211.1:42027): tablet 3e2d15f8640b407195210c476e09df91 (table test-workload [id=de73611ac8e3485a98aecc5add247032]) successfully deleted
I20260129 02:58:57.569280 21258 consensus_queue.cc:1048] T 4f5a77e94fef4b16a2ba7b067c8b144a P 974b802141bd4f13beffef2aa03f88f9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260129 02:58:57.571069 21258 consensus_queue.cc:1048] T 4f5a77e94fef4b16a2ba7b067c8b144a P 974b802141bd4f13beffef2aa03f88f9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260129 02:58:57.578299 21304 tablet_replica.cc:333] T 87d5fe48a44a408aaf03e894a67ecb2f P 1c50de89a5a845668696dcb28c3badd6: stopping tablet replica
I20260129 02:58:57.582887 21304 raft_consensus.cc:2243] T 87d5fe48a44a408aaf03e894a67ecb2f P 1c50de89a5a845668696dcb28c3badd6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260129 02:58:57.583681 21304 raft_consensus.cc:2272] T 87d5fe48a44a408aaf03e894a67ecb2f P 1c50de89a5a845668696dcb28c3badd6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260129 02:58:57.589090 21239 raft_consensus.cc:2955] T 4f5a77e94fef4b16a2ba7b067c8b144a P 974b802141bd4f13beffef2aa03f88f9 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 7a9daf5d42a84615a6383cfec9a10608 (127.18.211.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } attrs { replace: true } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } } }
I20260129 02:58:57.593197 21304 ts_tablet_manager.cc:1916] T 87d5fe48a44a408aaf03e894a67ecb2f P 1c50de89a5a845668696dcb28c3badd6: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260129 02:58:57.596225 20440 raft_consensus.cc:2955] T 4f5a77e94fef4b16a2ba7b067c8b144a P 197b880fa39d432a9b472845876bfaf0 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 7a9daf5d42a84615a6383cfec9a10608 (127.18.211.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } attrs { replace: true } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } } }
I20260129 02:58:57.597869 21136 raft_consensus.cc:2955] T 4f5a77e94fef4b16a2ba7b067c8b144a P 7a9daf5d42a84615a6383cfec9a10608 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 7a9daf5d42a84615a6383cfec9a10608 (127.18.211.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } attrs { replace: true } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } } }
I20260129 02:58:57.602320 20585 raft_consensus.cc:2955] T 4f5a77e94fef4b16a2ba7b067c8b144a P 1c50de89a5a845668696dcb28c3badd6 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 7a9daf5d42a84615a6383cfec9a10608 (127.18.211.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } attrs { replace: true } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } } }
I20260129 02:58:57.618314 20338 catalog_manager.cc:5654] T 4f5a77e94fef4b16a2ba7b067c8b144a P 197b880fa39d432a9b472845876bfaf0 reported cstate change: config changed from index 2 to 3, 7a9daf5d42a84615a6383cfec9a10608 (127.18.211.4) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "974b802141bd4f13beffef2aa03f88f9" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } attrs { replace: true } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } } }
I20260129 02:58:57.640312 21289 raft_consensus.cc:2955] T a609ac02fae84725ab743f95f51d33ce P 197b880fa39d432a9b472845876bfaf0 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 7a9daf5d42a84615a6383cfec9a10608 (127.18.211.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } attrs { replace: true } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } } }
I20260129 02:58:57.643370 20585 raft_consensus.cc:2955] T a609ac02fae84725ab743f95f51d33ce P 1c50de89a5a845668696dcb28c3badd6 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 7a9daf5d42a84615a6383cfec9a10608 (127.18.211.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } attrs { replace: true } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } } }
I20260129 02:58:57.673326 21304 ts_tablet_manager.cc:1929] T 87d5fe48a44a408aaf03e894a67ecb2f P 1c50de89a5a845668696dcb28c3badd6: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20260129 02:58:57.673825 21304 log.cc:1199] T 87d5fe48a44a408aaf03e894a67ecb2f P 1c50de89a5a845668696dcb28c3badd6: Deleting WAL directory at /tmp/dist-test-taskrTP_Jo/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1769655460890924-19276-0/minicluster-data/ts-2-root/wals/87d5fe48a44a408aaf03e894a67ecb2f
I20260129 02:58:57.673497 20514 consensus_queue.cc:237] T 4f5a77e94fef4b16a2ba7b067c8b144a P 974b802141bd4f13beffef2aa03f88f9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } }
I20260129 02:58:57.675643 20323 catalog_manager.cc:4985] TS 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3:37735): tablet 87d5fe48a44a408aaf03e894a67ecb2f (table test-workload [id=de73611ac8e3485a98aecc5add247032]) successfully deleted
I20260129 02:58:57.685853 20336 catalog_manager.cc:5654] T a609ac02fae84725ab743f95f51d33ce P 1c50de89a5a845668696dcb28c3badd6 reported cstate change: config changed from index 2 to 3, 7a9daf5d42a84615a6383cfec9a10608 (127.18.211.4) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "197b880fa39d432a9b472845876bfaf0" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } attrs { replace: true } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } } }
I20260129 02:58:57.691843 21133 raft_consensus.cc:2955] T a609ac02fae84725ab743f95f51d33ce P 7a9daf5d42a84615a6383cfec9a10608 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 7a9daf5d42a84615a6383cfec9a10608 (127.18.211.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } attrs { replace: true } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } } }
I20260129 02:58:57.698377 21135 raft_consensus.cc:1275] T 4f5a77e94fef4b16a2ba7b067c8b144a P 7a9daf5d42a84615a6383cfec9a10608 [term 1 FOLLOWER]: Refusing update from remote peer 974b802141bd4f13beffef2aa03f88f9: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260129 02:58:57.699693 20585 raft_consensus.cc:1275] T 4f5a77e94fef4b16a2ba7b067c8b144a P 1c50de89a5a845668696dcb28c3badd6 [term 1 FOLLOWER]: Refusing update from remote peer 974b802141bd4f13beffef2aa03f88f9: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260129 02:58:57.701484 21300 consensus_queue.cc:1048] T 4f5a77e94fef4b16a2ba7b067c8b144a P 974b802141bd4f13beffef2aa03f88f9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260129 02:58:57.701546 20514 raft_consensus.cc:2955] T a609ac02fae84725ab743f95f51d33ce P 974b802141bd4f13beffef2aa03f88f9 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 7a9daf5d42a84615a6383cfec9a10608 (127.18.211.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } attrs { replace: true } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } } }
I20260129 02:58:57.703377 21300 consensus_queue.cc:1048] T 4f5a77e94fef4b16a2ba7b067c8b144a P 974b802141bd4f13beffef2aa03f88f9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260129 02:58:57.715822 20438 consensus_queue.cc:237] T a609ac02fae84725ab743f95f51d33ce P 197b880fa39d432a9b472845876bfaf0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } }
I20260129 02:58:57.717043 21296 raft_consensus.cc:2955] T 4f5a77e94fef4b16a2ba7b067c8b144a P 974b802141bd4f13beffef2aa03f88f9 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 197b880fa39d432a9b472845876bfaf0 (127.18.211.1) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } } }
I20260129 02:58:57.722329 20516 raft_consensus.cc:1275] T a609ac02fae84725ab743f95f51d33ce P 974b802141bd4f13beffef2aa03f88f9 [term 1 FOLLOWER]: Refusing update from remote peer 197b880fa39d432a9b472845876bfaf0: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260129 02:58:57.723641 21133 raft_consensus.cc:1275] T a609ac02fae84725ab743f95f51d33ce P 7a9daf5d42a84615a6383cfec9a10608 [term 1 FOLLOWER]: Refusing update from remote peer 197b880fa39d432a9b472845876bfaf0: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260129 02:58:57.724336 21277 consensus_queue.cc:1048] T a609ac02fae84725ab743f95f51d33ce P 197b880fa39d432a9b472845876bfaf0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260129 02:58:57.725389 20587 raft_consensus.cc:2955] T 4f5a77e94fef4b16a2ba7b067c8b144a P 1c50de89a5a845668696dcb28c3badd6 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 197b880fa39d432a9b472845876bfaf0 (127.18.211.1) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } } }
I20260129 02:58:57.727996 21135 raft_consensus.cc:2955] T 4f5a77e94fef4b16a2ba7b067c8b144a P 7a9daf5d42a84615a6383cfec9a10608 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 197b880fa39d432a9b472845876bfaf0 (127.18.211.1) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } } }
I20260129 02:58:57.736145 20325 catalog_manager.cc:5167] ChangeConfig:REMOVE_PEER RPC for tablet 4f5a77e94fef4b16a2ba7b067c8b144a with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260129 02:58:57.740360 21277 consensus_queue.cc:1048] T a609ac02fae84725ab743f95f51d33ce P 197b880fa39d432a9b472845876bfaf0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260129 02:58:57.750429 20338 catalog_manager.cc:5654] T 4f5a77e94fef4b16a2ba7b067c8b144a P 1c50de89a5a845668696dcb28c3badd6 reported cstate change: config changed from index 3 to 4, VOTER 197b880fa39d432a9b472845876bfaf0 (127.18.211.1) evicted. New cstate: current_term: 1 leader_uuid: "974b802141bd4f13beffef2aa03f88f9" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } } }
I20260129 02:58:57.757131 21278 raft_consensus.cc:2955] T a609ac02fae84725ab743f95f51d33ce P 197b880fa39d432a9b472845876bfaf0 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } } }
I20260129 02:58:57.759148 21133 raft_consensus.cc:2955] T a609ac02fae84725ab743f95f51d33ce P 7a9daf5d42a84615a6383cfec9a10608 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } } }
I20260129 02:58:57.763059 20431 tablet_service.cc:1552] Processing DeleteTablet for tablet 4f5a77e94fef4b16a2ba7b067c8b144a with delete_type TABLET_DATA_TOMBSTONED (TS 197b880fa39d432a9b472845876bfaf0 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:55666
I20260129 02:58:57.764456 21297 tablet_replica.cc:333] T 4f5a77e94fef4b16a2ba7b067c8b144a P 197b880fa39d432a9b472845876bfaf0: stopping tablet replica
I20260129 02:58:57.765385 21297 raft_consensus.cc:2243] T 4f5a77e94fef4b16a2ba7b067c8b144a P 197b880fa39d432a9b472845876bfaf0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260129 02:58:57.766067 21297 raft_consensus.cc:2272] T 4f5a77e94fef4b16a2ba7b067c8b144a P 197b880fa39d432a9b472845876bfaf0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260129 02:58:57.773386 21297 ts_tablet_manager.cc:1916] T 4f5a77e94fef4b16a2ba7b067c8b144a P 197b880fa39d432a9b472845876bfaf0: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260129 02:58:57.777405 20514 raft_consensus.cc:2955] T a609ac02fae84725ab743f95f51d33ce P 974b802141bd4f13beffef2aa03f88f9 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } } }
I20260129 02:58:57.780308 20322 catalog_manager.cc:5167] ChangeConfig:REMOVE_PEER RPC for tablet a609ac02fae84725ab743f95f51d33ce with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260129 02:58:57.780403 20335 catalog_manager.cc:5654] T a609ac02fae84725ab743f95f51d33ce P 7a9daf5d42a84615a6383cfec9a10608 reported cstate change: config changed from index 3 to 4, VOTER 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3) evicted. New cstate: current_term: 1 leader_uuid: "197b880fa39d432a9b472845876bfaf0" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } } }
I20260129 02:58:57.792299 20579 tablet_service.cc:1552] Processing DeleteTablet for tablet a609ac02fae84725ab743f95f51d33ce with delete_type TABLET_DATA_TOMBSTONED (TS 1c50de89a5a845668696dcb28c3badd6 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:35744
I20260129 02:58:57.793170 21304 tablet_replica.cc:333] T a609ac02fae84725ab743f95f51d33ce P 1c50de89a5a845668696dcb28c3badd6: stopping tablet replica
I20260129 02:58:57.794087 21304 raft_consensus.cc:2243] T a609ac02fae84725ab743f95f51d33ce P 1c50de89a5a845668696dcb28c3badd6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260129 02:58:57.794893 21304 raft_consensus.cc:2272] T a609ac02fae84725ab743f95f51d33ce P 1c50de89a5a845668696dcb28c3badd6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260129 02:58:57.798346 21304 ts_tablet_manager.cc:1916] T a609ac02fae84725ab743f95f51d33ce P 1c50de89a5a845668696dcb28c3badd6: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260129 02:58:57.801193 21297 ts_tablet_manager.cc:1929] T 4f5a77e94fef4b16a2ba7b067c8b144a P 197b880fa39d432a9b472845876bfaf0: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20260129 02:58:57.801666 21297 log.cc:1199] T 4f5a77e94fef4b16a2ba7b067c8b144a P 197b880fa39d432a9b472845876bfaf0: Deleting WAL directory at /tmp/dist-test-taskrTP_Jo/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1769655460890924-19276-0/minicluster-data/ts-0-root/wals/4f5a77e94fef4b16a2ba7b067c8b144a
I20260129 02:58:57.803186 20322 catalog_manager.cc:4985] TS 197b880fa39d432a9b472845876bfaf0 (127.18.211.1:42027): tablet 4f5a77e94fef4b16a2ba7b067c8b144a (table test-workload [id=de73611ac8e3485a98aecc5add247032]) successfully deleted
I20260129 02:58:57.812770 21304 ts_tablet_manager.cc:1929] T a609ac02fae84725ab743f95f51d33ce P 1c50de89a5a845668696dcb28c3badd6: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20260129 02:58:57.813215 21304 log.cc:1199] T a609ac02fae84725ab743f95f51d33ce P 1c50de89a5a845668696dcb28c3badd6: Deleting WAL directory at /tmp/dist-test-taskrTP_Jo/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1769655460890924-19276-0/minicluster-data/ts-2-root/wals/a609ac02fae84725ab743f95f51d33ce
I20260129 02:58:57.814750 20323 catalog_manager.cc:4985] TS 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3:37735): tablet a609ac02fae84725ab743f95f51d33ce (table test-workload [id=de73611ac8e3485a98aecc5add247032]) successfully deleted
I20260129 02:58:58.851831 20440 consensus_queue.cc:237] T 2e58e9a5a8aa4cecafb464addb4f35ed P 197b880fa39d432a9b472845876bfaf0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } attrs { replace: true } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: NON_VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: true } }
W20260129 02:58:58.859028 20405 consensus_peers.cc:597] T 2e58e9a5a8aa4cecafb464addb4f35ed P 197b880fa39d432a9b472845876bfaf0 -> Peer 7a9daf5d42a84615a6383cfec9a10608 (127.18.211.4:40439): Couldn't send request to peer 7a9daf5d42a84615a6383cfec9a10608. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 2e58e9a5a8aa4cecafb464addb4f35ed. This is attempt 1: this message will repeat every 5th retry.
I20260129 02:58:58.859851 20585 raft_consensus.cc:1275] T 2e58e9a5a8aa4cecafb464addb4f35ed P 1c50de89a5a845668696dcb28c3badd6 [term 1 FOLLOWER]: Refusing update from remote peer 197b880fa39d432a9b472845876bfaf0: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260129 02:58:58.860162 20511 raft_consensus.cc:1275] T 2e58e9a5a8aa4cecafb464addb4f35ed P 974b802141bd4f13beffef2aa03f88f9 [term 1 FOLLOWER]: Refusing update from remote peer 197b880fa39d432a9b472845876bfaf0: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260129 02:58:58.861191 21277 consensus_queue.cc:1048] T 2e58e9a5a8aa4cecafb464addb4f35ed P 197b880fa39d432a9b472845876bfaf0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260129 02:58:58.861913 21024 consensus_queue.cc:1048] T 2e58e9a5a8aa4cecafb464addb4f35ed P 197b880fa39d432a9b472845876bfaf0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260129 02:58:58.868750 21310 raft_consensus.cc:2955] T 2e58e9a5a8aa4cecafb464addb4f35ed P 197b880fa39d432a9b472845876bfaf0 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 7a9daf5d42a84615a6383cfec9a10608 (127.18.211.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } attrs { replace: true } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: NON_VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: true } } }
I20260129 02:58:58.870716 20511 raft_consensus.cc:2955] T 2e58e9a5a8aa4cecafb464addb4f35ed P 974b802141bd4f13beffef2aa03f88f9 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 7a9daf5d42a84615a6383cfec9a10608 (127.18.211.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } attrs { replace: true } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: NON_VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: true } } }
I20260129 02:58:58.870716 20585 raft_consensus.cc:2955] T 2e58e9a5a8aa4cecafb464addb4f35ed P 1c50de89a5a845668696dcb28c3badd6 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 7a9daf5d42a84615a6383cfec9a10608 (127.18.211.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } attrs { replace: true } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: NON_VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: true } } }
I20260129 02:58:58.887081 20511 consensus_queue.cc:237] T cb0b3f259ebc4310927338a8740e87b8 P 974b802141bd4f13beffef2aa03f88f9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } attrs { replace: true } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: NON_VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: true } }
I20260129 02:58:58.886837 20336 catalog_manager.cc:5654] T 2e58e9a5a8aa4cecafb464addb4f35ed P 197b880fa39d432a9b472845876bfaf0 reported cstate change: config changed from index -1 to 2, NON_VOTER 7a9daf5d42a84615a6383cfec9a10608 (127.18.211.4) added. New cstate: current_term: 1 leader_uuid: "197b880fa39d432a9b472845876bfaf0" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: NON_VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20260129 02:58:58.897938 20585 raft_consensus.cc:1275] T cb0b3f259ebc4310927338a8740e87b8 P 1c50de89a5a845668696dcb28c3badd6 [term 1 FOLLOWER]: Refusing update from remote peer 974b802141bd4f13beffef2aa03f88f9: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20260129 02:58:58.898589 20480 consensus_peers.cc:597] T cb0b3f259ebc4310927338a8740e87b8 P 974b802141bd4f13beffef2aa03f88f9 -> Peer 7a9daf5d42a84615a6383cfec9a10608 (127.18.211.4:40439): Couldn't send request to peer 7a9daf5d42a84615a6383cfec9a10608. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: cb0b3f259ebc4310927338a8740e87b8. This is attempt 1: this message will repeat every 5th retry.
I20260129 02:58:58.898602 20440 raft_consensus.cc:1275] T cb0b3f259ebc4310927338a8740e87b8 P 197b880fa39d432a9b472845876bfaf0 [term 1 FOLLOWER]: Refusing update from remote peer 974b802141bd4f13beffef2aa03f88f9: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260129 02:58:58.899555 21295 consensus_queue.cc:1048] T cb0b3f259ebc4310927338a8740e87b8 P 974b802141bd4f13beffef2aa03f88f9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260129 02:58:58.900406 21300 consensus_queue.cc:1048] T cb0b3f259ebc4310927338a8740e87b8 P 974b802141bd4f13beffef2aa03f88f9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260129 02:58:58.912724 21300 raft_consensus.cc:2955] T cb0b3f259ebc4310927338a8740e87b8 P 974b802141bd4f13beffef2aa03f88f9 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 7a9daf5d42a84615a6383cfec9a10608 (127.18.211.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } attrs { replace: true } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: NON_VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: true } } }
I20260129 02:58:58.914341 20440 raft_consensus.cc:2955] T cb0b3f259ebc4310927338a8740e87b8 P 197b880fa39d432a9b472845876bfaf0 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 7a9daf5d42a84615a6383cfec9a10608 (127.18.211.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } attrs { replace: true } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: NON_VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: true } } }
I20260129 02:58:58.915788 20585 raft_consensus.cc:2955] T cb0b3f259ebc4310927338a8740e87b8 P 1c50de89a5a845668696dcb28c3badd6 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 7a9daf5d42a84615a6383cfec9a10608 (127.18.211.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } attrs { replace: true } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: NON_VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: true } } }
I20260129 02:58:58.938401 20336 catalog_manager.cc:5654] T cb0b3f259ebc4310927338a8740e87b8 P 197b880fa39d432a9b472845876bfaf0 reported cstate change: config changed from index -1 to 2, NON_VOTER 7a9daf5d42a84615a6383cfec9a10608 (127.18.211.4) added. New cstate: current_term: 1 leader_uuid: "974b802141bd4f13beffef2aa03f88f9" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } attrs { replace: true } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: NON_VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: true } } }
I20260129 02:58:58.941391 20440 consensus_queue.cc:237] T 18f30ebaa4174fd28c958e72d0cc7e24 P 197b880fa39d432a9b472845876bfaf0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } attrs { replace: true } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: NON_VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: true } }
I20260129 02:58:58.951754 20511 raft_consensus.cc:1275] T 18f30ebaa4174fd28c958e72d0cc7e24 P 974b802141bd4f13beffef2aa03f88f9 [term 1 FOLLOWER]: Refusing update from remote peer 197b880fa39d432a9b472845876bfaf0: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260129 02:58:58.953696 20585 raft_consensus.cc:1275] T 18f30ebaa4174fd28c958e72d0cc7e24 P 1c50de89a5a845668696dcb28c3badd6 [term 1 FOLLOWER]: Refusing update from remote peer 197b880fa39d432a9b472845876bfaf0: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20260129 02:58:58.954553 20405 consensus_peers.cc:597] T 18f30ebaa4174fd28c958e72d0cc7e24 P 197b880fa39d432a9b472845876bfaf0 -> Peer 7a9daf5d42a84615a6383cfec9a10608 (127.18.211.4:40439): Couldn't send request to peer 7a9daf5d42a84615a6383cfec9a10608. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 18f30ebaa4174fd28c958e72d0cc7e24. This is attempt 1: this message will repeat every 5th retry.
I20260129 02:58:58.954592 21024 consensus_queue.cc:1048] T 18f30ebaa4174fd28c958e72d0cc7e24 P 197b880fa39d432a9b472845876bfaf0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260129 02:58:58.956005 21277 consensus_queue.cc:1048] T 18f30ebaa4174fd28c958e72d0cc7e24 P 197b880fa39d432a9b472845876bfaf0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260129 02:58:58.962725 21277 raft_consensus.cc:2955] T 18f30ebaa4174fd28c958e72d0cc7e24 P 197b880fa39d432a9b472845876bfaf0 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 7a9daf5d42a84615a6383cfec9a10608 (127.18.211.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } attrs { replace: true } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: NON_VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: true } } }
I20260129 02:58:58.971670 20511 raft_consensus.cc:2955] T 18f30ebaa4174fd28c958e72d0cc7e24 P 974b802141bd4f13beffef2aa03f88f9 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 7a9daf5d42a84615a6383cfec9a10608 (127.18.211.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } attrs { replace: true } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: NON_VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: true } } }
I20260129 02:58:58.979360 20587 raft_consensus.cc:2955] T 18f30ebaa4174fd28c958e72d0cc7e24 P 1c50de89a5a845668696dcb28c3badd6 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 7a9daf5d42a84615a6383cfec9a10608 (127.18.211.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } attrs { replace: true } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: NON_VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: true } } }
I20260129 02:58:58.982342 20335 catalog_manager.cc:5654] T 18f30ebaa4174fd28c958e72d0cc7e24 P 197b880fa39d432a9b472845876bfaf0 reported cstate change: config changed from index -1 to 2, NON_VOTER 7a9daf5d42a84615a6383cfec9a10608 (127.18.211.4) added. New cstate: current_term: 1 leader_uuid: "197b880fa39d432a9b472845876bfaf0" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: NON_VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20260129 02:58:58.983223 20440 consensus_queue.cc:237] T 00e79d8f25a34676b1d866eb2068a2b2 P 197b880fa39d432a9b472845876bfaf0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } attrs { replace: true } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: NON_VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: true } }
W20260129 02:58:59.015062 20405 consensus_peers.cc:597] T 00e79d8f25a34676b1d866eb2068a2b2 P 197b880fa39d432a9b472845876bfaf0 -> Peer 7a9daf5d42a84615a6383cfec9a10608 (127.18.211.4:40439): Couldn't send request to peer 7a9daf5d42a84615a6383cfec9a10608. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 00e79d8f25a34676b1d866eb2068a2b2. This is attempt 1: this message will repeat every 5th retry.
I20260129 02:58:59.015684 20511 raft_consensus.cc:1275] T 00e79d8f25a34676b1d866eb2068a2b2 P 974b802141bd4f13beffef2aa03f88f9 [term 1 FOLLOWER]: Refusing update from remote peer 197b880fa39d432a9b472845876bfaf0: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260129 02:58:59.016965 20585 raft_consensus.cc:1275] T 00e79d8f25a34676b1d866eb2068a2b2 P 1c50de89a5a845668696dcb28c3badd6 [term 1 FOLLOWER]: Refusing update from remote peer 197b880fa39d432a9b472845876bfaf0: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260129 02:58:59.019027 21277 consensus_queue.cc:1048] T 00e79d8f25a34676b1d866eb2068a2b2 P 197b880fa39d432a9b472845876bfaf0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260129 02:58:59.020089 21310 consensus_queue.cc:1048] T 00e79d8f25a34676b1d866eb2068a2b2 P 197b880fa39d432a9b472845876bfaf0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260129 02:58:59.036921 21277 raft_consensus.cc:2955] T 00e79d8f25a34676b1d866eb2068a2b2 P 197b880fa39d432a9b472845876bfaf0 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 7a9daf5d42a84615a6383cfec9a10608 (127.18.211.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } attrs { replace: true } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: NON_VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: true } } }
I20260129 02:58:59.044659 20587 raft_consensus.cc:2955] T 00e79d8f25a34676b1d866eb2068a2b2 P 1c50de89a5a845668696dcb28c3badd6 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 7a9daf5d42a84615a6383cfec9a10608 (127.18.211.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } attrs { replace: true } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: NON_VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: true } } }
I20260129 02:58:59.051107 20511 raft_consensus.cc:2955] T 00e79d8f25a34676b1d866eb2068a2b2 P 974b802141bd4f13beffef2aa03f88f9 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 7a9daf5d42a84615a6383cfec9a10608 (127.18.211.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } attrs { replace: true } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: NON_VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: true } } }
I20260129 02:58:59.066594 20338 catalog_manager.cc:5654] T 00e79d8f25a34676b1d866eb2068a2b2 P 1c50de89a5a845668696dcb28c3badd6 reported cstate change: config changed from index -1 to 2, NON_VOTER 7a9daf5d42a84615a6383cfec9a10608 (127.18.211.4) added. New cstate: current_term: 1 leader_uuid: "197b880fa39d432a9b472845876bfaf0" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } attrs { replace: true } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: NON_VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: true } } }
I20260129 02:58:59.331269 21329 ts_tablet_manager.cc:933] T 2e58e9a5a8aa4cecafb464addb4f35ed P 7a9daf5d42a84615a6383cfec9a10608: Initiating tablet copy from peer 197b880fa39d432a9b472845876bfaf0 (127.18.211.1:42027)
I20260129 02:58:59.333163 21329 tablet_copy_client.cc:323] T 2e58e9a5a8aa4cecafb464addb4f35ed P 7a9daf5d42a84615a6383cfec9a10608: tablet copy: Beginning tablet copy session from remote peer at address 127.18.211.1:42027
I20260129 02:58:59.334832 20451 tablet_copy_service.cc:140] P 197b880fa39d432a9b472845876bfaf0: Received BeginTabletCopySession request for tablet 2e58e9a5a8aa4cecafb464addb4f35ed from peer 7a9daf5d42a84615a6383cfec9a10608 ({username='slave'} at 127.0.0.1:48356)
I20260129 02:58:59.335578 20451 tablet_copy_service.cc:161] P 197b880fa39d432a9b472845876bfaf0: Beginning new tablet copy session on tablet 2e58e9a5a8aa4cecafb464addb4f35ed from peer 7a9daf5d42a84615a6383cfec9a10608 at {username='slave'} at 127.0.0.1:48356: session id = 7a9daf5d42a84615a6383cfec9a10608-2e58e9a5a8aa4cecafb464addb4f35ed
I20260129 02:58:59.342315 20451 tablet_copy_source_session.cc:215] T 2e58e9a5a8aa4cecafb464addb4f35ed P 197b880fa39d432a9b472845876bfaf0: Tablet Copy: opened 0 blocks and 1 log segments
I20260129 02:58:59.345187 21329 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2e58e9a5a8aa4cecafb464addb4f35ed. 1 dirs total, 0 dirs full, 0 dirs failed
I20260129 02:58:59.361465 21329 tablet_copy_client.cc:806] T 2e58e9a5a8aa4cecafb464addb4f35ed P 7a9daf5d42a84615a6383cfec9a10608: tablet copy: Starting download of 0 data blocks...
I20260129 02:58:59.362054 21329 tablet_copy_client.cc:670] T 2e58e9a5a8aa4cecafb464addb4f35ed P 7a9daf5d42a84615a6383cfec9a10608: tablet copy: Starting download of 1 WAL segments...
I20260129 02:58:59.366542 21329 tablet_copy_client.cc:538] T 2e58e9a5a8aa4cecafb464addb4f35ed P 7a9daf5d42a84615a6383cfec9a10608: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260129 02:58:59.374372 21329 tablet_bootstrap.cc:492] T 2e58e9a5a8aa4cecafb464addb4f35ed P 7a9daf5d42a84615a6383cfec9a10608: Bootstrap starting.
I20260129 02:58:59.391789 21329 tablet_bootstrap.cc:492] T 2e58e9a5a8aa4cecafb464addb4f35ed P 7a9daf5d42a84615a6383cfec9a10608: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260129 02:58:59.392619 21329 tablet_bootstrap.cc:492] T 2e58e9a5a8aa4cecafb464addb4f35ed P 7a9daf5d42a84615a6383cfec9a10608: Bootstrap complete.
I20260129 02:58:59.393311 21329 ts_tablet_manager.cc:1403] T 2e58e9a5a8aa4cecafb464addb4f35ed P 7a9daf5d42a84615a6383cfec9a10608: Time spent bootstrapping tablet: real 0.019s user 0.015s sys 0.005s
I20260129 02:58:59.396019 21329 raft_consensus.cc:359] T 2e58e9a5a8aa4cecafb464addb4f35ed P 7a9daf5d42a84615a6383cfec9a10608 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } attrs { replace: true } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: NON_VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: true } }
I20260129 02:58:59.396826 21329 raft_consensus.cc:740] T 2e58e9a5a8aa4cecafb464addb4f35ed P 7a9daf5d42a84615a6383cfec9a10608 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 7a9daf5d42a84615a6383cfec9a10608, State: Initialized, Role: LEARNER
I20260129 02:58:59.397410 21329 consensus_queue.cc:260] T 2e58e9a5a8aa4cecafb464addb4f35ed P 7a9daf5d42a84615a6383cfec9a10608 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } attrs { replace: true } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: NON_VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: true } }
I20260129 02:58:59.401012 21329 ts_tablet_manager.cc:1434] T 2e58e9a5a8aa4cecafb464addb4f35ed P 7a9daf5d42a84615a6383cfec9a10608: Time spent starting tablet: real 0.007s user 0.003s sys 0.005s
I20260129 02:58:59.403744 20451 tablet_copy_service.cc:342] P 197b880fa39d432a9b472845876bfaf0: Request end of tablet copy session 7a9daf5d42a84615a6383cfec9a10608-2e58e9a5a8aa4cecafb464addb4f35ed received from {username='slave'} at 127.0.0.1:48356
I20260129 02:58:59.404210 20451 tablet_copy_service.cc:434] P 197b880fa39d432a9b472845876bfaf0: ending tablet copy session 7a9daf5d42a84615a6383cfec9a10608-2e58e9a5a8aa4cecafb464addb4f35ed on tablet 2e58e9a5a8aa4cecafb464addb4f35ed with peer 7a9daf5d42a84615a6383cfec9a10608
I20260129 02:58:59.426030 21329 ts_tablet_manager.cc:933] T cb0b3f259ebc4310927338a8740e87b8 P 7a9daf5d42a84615a6383cfec9a10608: Initiating tablet copy from peer 974b802141bd4f13beffef2aa03f88f9 (127.18.211.2:34517)
I20260129 02:58:59.427639 21329 tablet_copy_client.cc:323] T cb0b3f259ebc4310927338a8740e87b8 P 7a9daf5d42a84615a6383cfec9a10608: tablet copy: Beginning tablet copy session from remote peer at address 127.18.211.2:34517
I20260129 02:58:59.429343 20526 tablet_copy_service.cc:140] P 974b802141bd4f13beffef2aa03f88f9: Received BeginTabletCopySession request for tablet cb0b3f259ebc4310927338a8740e87b8 from peer 7a9daf5d42a84615a6383cfec9a10608 ({username='slave'} at 127.0.0.1:48544)
I20260129 02:58:59.429848 20526 tablet_copy_service.cc:161] P 974b802141bd4f13beffef2aa03f88f9: Beginning new tablet copy session on tablet cb0b3f259ebc4310927338a8740e87b8 from peer 7a9daf5d42a84615a6383cfec9a10608 at {username='slave'} at 127.0.0.1:48544: session id = 7a9daf5d42a84615a6383cfec9a10608-cb0b3f259ebc4310927338a8740e87b8
I20260129 02:58:59.437345 20526 tablet_copy_source_session.cc:215] T cb0b3f259ebc4310927338a8740e87b8 P 974b802141bd4f13beffef2aa03f88f9: Tablet Copy: opened 0 blocks and 1 log segments
I20260129 02:58:59.440059 21329 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cb0b3f259ebc4310927338a8740e87b8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260129 02:58:59.456847 21329 tablet_copy_client.cc:806] T cb0b3f259ebc4310927338a8740e87b8 P 7a9daf5d42a84615a6383cfec9a10608: tablet copy: Starting download of 0 data blocks...
I20260129 02:58:59.465111 21329 tablet_copy_client.cc:670] T cb0b3f259ebc4310927338a8740e87b8 P 7a9daf5d42a84615a6383cfec9a10608: tablet copy: Starting download of 1 WAL segments...
I20260129 02:58:59.467725 21334 ts_tablet_manager.cc:933] T 00e79d8f25a34676b1d866eb2068a2b2 P 7a9daf5d42a84615a6383cfec9a10608: Initiating tablet copy from peer 197b880fa39d432a9b472845876bfaf0 (127.18.211.1:42027)
I20260129 02:58:59.468905 21329 tablet_copy_client.cc:538] T cb0b3f259ebc4310927338a8740e87b8 P 7a9daf5d42a84615a6383cfec9a10608: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260129 02:58:59.479240 21329 tablet_bootstrap.cc:492] T cb0b3f259ebc4310927338a8740e87b8 P 7a9daf5d42a84615a6383cfec9a10608: Bootstrap starting.
I20260129 02:58:59.481419 21334 tablet_copy_client.cc:323] T 00e79d8f25a34676b1d866eb2068a2b2 P 7a9daf5d42a84615a6383cfec9a10608: tablet copy: Beginning tablet copy session from remote peer at address 127.18.211.1:42027
I20260129 02:58:59.481554 21335 ts_tablet_manager.cc:933] T 18f30ebaa4174fd28c958e72d0cc7e24 P 7a9daf5d42a84615a6383cfec9a10608: Initiating tablet copy from peer 197b880fa39d432a9b472845876bfaf0 (127.18.211.1:42027)
I20260129 02:58:59.483170 20451 tablet_copy_service.cc:140] P 197b880fa39d432a9b472845876bfaf0: Received BeginTabletCopySession request for tablet 00e79d8f25a34676b1d866eb2068a2b2 from peer 7a9daf5d42a84615a6383cfec9a10608 ({username='slave'} at 127.0.0.1:48356)
I20260129 02:58:59.483700 20451 tablet_copy_service.cc:161] P 197b880fa39d432a9b472845876bfaf0: Beginning new tablet copy session on tablet 00e79d8f25a34676b1d866eb2068a2b2 from peer 7a9daf5d42a84615a6383cfec9a10608 at {username='slave'} at 127.0.0.1:48356: session id = 7a9daf5d42a84615a6383cfec9a10608-00e79d8f25a34676b1d866eb2068a2b2
I20260129 02:58:59.489629 21335 tablet_copy_client.cc:323] T 18f30ebaa4174fd28c958e72d0cc7e24 P 7a9daf5d42a84615a6383cfec9a10608: tablet copy: Beginning tablet copy session from remote peer at address 127.18.211.1:42027
I20260129 02:58:59.491734 20450 tablet_copy_service.cc:140] P 197b880fa39d432a9b472845876bfaf0: Received BeginTabletCopySession request for tablet 18f30ebaa4174fd28c958e72d0cc7e24 from peer 7a9daf5d42a84615a6383cfec9a10608 ({username='slave'} at 127.0.0.1:48356)
I20260129 02:58:59.492331 20450 tablet_copy_service.cc:161] P 197b880fa39d432a9b472845876bfaf0: Beginning new tablet copy session on tablet 18f30ebaa4174fd28c958e72d0cc7e24 from peer 7a9daf5d42a84615a6383cfec9a10608 at {username='slave'} at 127.0.0.1:48356: session id = 7a9daf5d42a84615a6383cfec9a10608-18f30ebaa4174fd28c958e72d0cc7e24
I20260129 02:58:59.498270 20450 tablet_copy_source_session.cc:215] T 18f30ebaa4174fd28c958e72d0cc7e24 P 197b880fa39d432a9b472845876bfaf0: Tablet Copy: opened 0 blocks and 1 log segments
I20260129 02:58:59.502305 21335 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 18f30ebaa4174fd28c958e72d0cc7e24. 1 dirs total, 0 dirs full, 0 dirs failed
I20260129 02:58:59.497669 20451 tablet_copy_source_session.cc:215] T 00e79d8f25a34676b1d866eb2068a2b2 P 197b880fa39d432a9b472845876bfaf0: Tablet Copy: opened 0 blocks and 1 log segments
I20260129 02:58:59.505954 21334 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00e79d8f25a34676b1d866eb2068a2b2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260129 02:58:59.506453 21329 tablet_bootstrap.cc:492] T cb0b3f259ebc4310927338a8740e87b8 P 7a9daf5d42a84615a6383cfec9a10608: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260129 02:58:59.507333 21329 tablet_bootstrap.cc:492] T cb0b3f259ebc4310927338a8740e87b8 P 7a9daf5d42a84615a6383cfec9a10608: Bootstrap complete.
I20260129 02:58:59.508028 21329 ts_tablet_manager.cc:1403] T cb0b3f259ebc4310927338a8740e87b8 P 7a9daf5d42a84615a6383cfec9a10608: Time spent bootstrapping tablet: real 0.029s user 0.021s sys 0.004s
I20260129 02:58:59.510823 21329 raft_consensus.cc:359] T cb0b3f259ebc4310927338a8740e87b8 P 7a9daf5d42a84615a6383cfec9a10608 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } attrs { replace: true } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: NON_VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: true } }
I20260129 02:58:59.511655 21329 raft_consensus.cc:740] T cb0b3f259ebc4310927338a8740e87b8 P 7a9daf5d42a84615a6383cfec9a10608 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 7a9daf5d42a84615a6383cfec9a10608, State: Initialized, Role: LEARNER
I20260129 02:58:59.512279 21329 consensus_queue.cc:260] T cb0b3f259ebc4310927338a8740e87b8 P 7a9daf5d42a84615a6383cfec9a10608 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } attrs { replace: true } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: NON_VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: true } }
I20260129 02:58:59.515272 21329 ts_tablet_manager.cc:1434] T cb0b3f259ebc4310927338a8740e87b8 P 7a9daf5d42a84615a6383cfec9a10608: Time spent starting tablet: real 0.007s user 0.007s sys 0.000s
I20260129 02:58:59.520015 20526 tablet_copy_service.cc:342] P 974b802141bd4f13beffef2aa03f88f9: Request end of tablet copy session 7a9daf5d42a84615a6383cfec9a10608-cb0b3f259ebc4310927338a8740e87b8 received from {username='slave'} at 127.0.0.1:48544
I20260129 02:58:59.520392 20526 tablet_copy_service.cc:434] P 974b802141bd4f13beffef2aa03f88f9: ending tablet copy session 7a9daf5d42a84615a6383cfec9a10608-cb0b3f259ebc4310927338a8740e87b8 on tablet cb0b3f259ebc4310927338a8740e87b8 with peer 7a9daf5d42a84615a6383cfec9a10608
I20260129 02:58:59.532141 21335 tablet_copy_client.cc:806] T 18f30ebaa4174fd28c958e72d0cc7e24 P 7a9daf5d42a84615a6383cfec9a10608: tablet copy: Starting download of 0 data blocks...
I20260129 02:58:59.532784 21334 tablet_copy_client.cc:806] T 00e79d8f25a34676b1d866eb2068a2b2 P 7a9daf5d42a84615a6383cfec9a10608: tablet copy: Starting download of 0 data blocks...
I20260129 02:58:59.532781 21335 tablet_copy_client.cc:670] T 18f30ebaa4174fd28c958e72d0cc7e24 P 7a9daf5d42a84615a6383cfec9a10608: tablet copy: Starting download of 1 WAL segments...
I20260129 02:58:59.534070 21334 tablet_copy_client.cc:670] T 00e79d8f25a34676b1d866eb2068a2b2 P 7a9daf5d42a84615a6383cfec9a10608: tablet copy: Starting download of 1 WAL segments...
I20260129 02:58:59.538548 21335 tablet_copy_client.cc:538] T 18f30ebaa4174fd28c958e72d0cc7e24 P 7a9daf5d42a84615a6383cfec9a10608: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260129 02:58:59.538547 21334 tablet_copy_client.cc:538] T 00e79d8f25a34676b1d866eb2068a2b2 P 7a9daf5d42a84615a6383cfec9a10608: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260129 02:58:59.547020 21335 tablet_bootstrap.cc:492] T 18f30ebaa4174fd28c958e72d0cc7e24 P 7a9daf5d42a84615a6383cfec9a10608: Bootstrap starting.
I20260129 02:58:59.549137 21334 tablet_bootstrap.cc:492] T 00e79d8f25a34676b1d866eb2068a2b2 P 7a9daf5d42a84615a6383cfec9a10608: Bootstrap starting.
I20260129 02:58:59.585601 21335 tablet_bootstrap.cc:492] T 18f30ebaa4174fd28c958e72d0cc7e24 P 7a9daf5d42a84615a6383cfec9a10608: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260129 02:58:59.586485 21335 tablet_bootstrap.cc:492] T 18f30ebaa4174fd28c958e72d0cc7e24 P 7a9daf5d42a84615a6383cfec9a10608: Bootstrap complete.
I20260129 02:58:59.587157 21335 ts_tablet_manager.cc:1403] T 18f30ebaa4174fd28c958e72d0cc7e24 P 7a9daf5d42a84615a6383cfec9a10608: Time spent bootstrapping tablet: real 0.040s user 0.018s sys 0.011s
I20260129 02:58:59.590019 21335 raft_consensus.cc:359] T 18f30ebaa4174fd28c958e72d0cc7e24 P 7a9daf5d42a84615a6383cfec9a10608 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } attrs { replace: true } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: NON_VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: true } }
I20260129 02:58:59.590809 21335 raft_consensus.cc:740] T 18f30ebaa4174fd28c958e72d0cc7e24 P 7a9daf5d42a84615a6383cfec9a10608 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 7a9daf5d42a84615a6383cfec9a10608, State: Initialized, Role: LEARNER
I20260129 02:58:59.591432 21335 consensus_queue.cc:260] T 18f30ebaa4174fd28c958e72d0cc7e24 P 7a9daf5d42a84615a6383cfec9a10608 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } attrs { replace: true } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: NON_VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: true } }
I20260129 02:58:59.601097 21335 ts_tablet_manager.cc:1434] T 18f30ebaa4174fd28c958e72d0cc7e24 P 7a9daf5d42a84615a6383cfec9a10608: Time spent starting tablet: real 0.014s user 0.009s sys 0.000s
I20260129 02:58:59.619366 20450 tablet_copy_service.cc:342] P 197b880fa39d432a9b472845876bfaf0: Request end of tablet copy session 7a9daf5d42a84615a6383cfec9a10608-18f30ebaa4174fd28c958e72d0cc7e24 received from {username='slave'} at 127.0.0.1:48356
I20260129 02:58:59.619853 20450 tablet_copy_service.cc:434] P 197b880fa39d432a9b472845876bfaf0: ending tablet copy session 7a9daf5d42a84615a6383cfec9a10608-18f30ebaa4174fd28c958e72d0cc7e24 on tablet 18f30ebaa4174fd28c958e72d0cc7e24 with peer 7a9daf5d42a84615a6383cfec9a10608
I20260129 02:58:59.646860 21334 tablet_bootstrap.cc:492] T 00e79d8f25a34676b1d866eb2068a2b2 P 7a9daf5d42a84615a6383cfec9a10608: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260129 02:58:59.647795 21334 tablet_bootstrap.cc:492] T 00e79d8f25a34676b1d866eb2068a2b2 P 7a9daf5d42a84615a6383cfec9a10608: Bootstrap complete.
I20260129 02:58:59.648504 21334 ts_tablet_manager.cc:1403] T 00e79d8f25a34676b1d866eb2068a2b2 P 7a9daf5d42a84615a6383cfec9a10608: Time spent bootstrapping tablet: real 0.100s user 0.030s sys 0.036s
I20260129 02:58:59.651172 21334 raft_consensus.cc:359] T 00e79d8f25a34676b1d866eb2068a2b2 P 7a9daf5d42a84615a6383cfec9a10608 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } attrs { replace: true } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: NON_VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: true } }
I20260129 02:58:59.652020 21334 raft_consensus.cc:740] T 00e79d8f25a34676b1d866eb2068a2b2 P 7a9daf5d42a84615a6383cfec9a10608 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 7a9daf5d42a84615a6383cfec9a10608, State: Initialized, Role: LEARNER
I20260129 02:58:59.652575 21334 consensus_queue.cc:260] T 00e79d8f25a34676b1d866eb2068a2b2 P 7a9daf5d42a84615a6383cfec9a10608 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } attrs { replace: true } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: NON_VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: true } }
I20260129 02:58:59.657244 21334 ts_tablet_manager.cc:1434] T 00e79d8f25a34676b1d866eb2068a2b2 P 7a9daf5d42a84615a6383cfec9a10608: Time spent starting tablet: real 0.008s user 0.006s sys 0.000s
I20260129 02:58:59.658684 20450 tablet_copy_service.cc:342] P 197b880fa39d432a9b472845876bfaf0: Request end of tablet copy session 7a9daf5d42a84615a6383cfec9a10608-00e79d8f25a34676b1d866eb2068a2b2 received from {username='slave'} at 127.0.0.1:48356
I20260129 02:58:59.659108 20450 tablet_copy_service.cc:434] P 197b880fa39d432a9b472845876bfaf0: ending tablet copy session 7a9daf5d42a84615a6383cfec9a10608-00e79d8f25a34676b1d866eb2068a2b2 on tablet 00e79d8f25a34676b1d866eb2068a2b2 with peer 7a9daf5d42a84615a6383cfec9a10608
I20260129 02:58:59.859231 21135 raft_consensus.cc:1217] T 00e79d8f25a34676b1d866eb2068a2b2 P 7a9daf5d42a84615a6383cfec9a10608 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20260129 02:58:59.900175 21135 raft_consensus.cc:1217] T 2e58e9a5a8aa4cecafb464addb4f35ed P 7a9daf5d42a84615a6383cfec9a10608 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20260129 02:59:00.002000 21135 raft_consensus.cc:1217] T cb0b3f259ebc4310927338a8740e87b8 P 7a9daf5d42a84615a6383cfec9a10608 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20260129 02:59:00.081597 21135 raft_consensus.cc:1217] T 18f30ebaa4174fd28c958e72d0cc7e24 P 7a9daf5d42a84615a6383cfec9a10608 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20260129 02:59:00.425997 21328 raft_consensus.cc:1064] T 00e79d8f25a34676b1d866eb2068a2b2 P 197b880fa39d432a9b472845876bfaf0: attempting to promote NON_VOTER 7a9daf5d42a84615a6383cfec9a10608 to VOTER
I20260129 02:59:00.428445 21328 consensus_queue.cc:237] T 00e79d8f25a34676b1d866eb2068a2b2 P 197b880fa39d432a9b472845876bfaf0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } attrs { replace: true } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } }
I20260129 02:59:00.449191 21135 raft_consensus.cc:1275] T 00e79d8f25a34676b1d866eb2068a2b2 P 7a9daf5d42a84615a6383cfec9a10608 [term 1 LEARNER]: Refusing update from remote peer 197b880fa39d432a9b472845876bfaf0: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260129 02:59:00.449223 20586 raft_consensus.cc:1275] T 00e79d8f25a34676b1d866eb2068a2b2 P 1c50de89a5a845668696dcb28c3badd6 [term 1 FOLLOWER]: Refusing update from remote peer 197b880fa39d432a9b472845876bfaf0: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260129 02:59:00.450681 20511 raft_consensus.cc:1275] T 00e79d8f25a34676b1d866eb2068a2b2 P 974b802141bd4f13beffef2aa03f88f9 [term 1 FOLLOWER]: Refusing update from remote peer 197b880fa39d432a9b472845876bfaf0: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260129 02:59:00.451152 21342 consensus_queue.cc:1048] T 00e79d8f25a34676b1d866eb2068a2b2 P 197b880fa39d432a9b472845876bfaf0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260129 02:59:00.452149 21310 consensus_queue.cc:1048] T 00e79d8f25a34676b1d866eb2068a2b2 P 197b880fa39d432a9b472845876bfaf0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260129 02:59:00.454802 21328 consensus_queue.cc:1048] T 00e79d8f25a34676b1d866eb2068a2b2 P 197b880fa39d432a9b472845876bfaf0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260129 02:59:00.470783 21342 raft_consensus.cc:2955] T 00e79d8f25a34676b1d866eb2068a2b2 P 197b880fa39d432a9b472845876bfaf0 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 7a9daf5d42a84615a6383cfec9a10608 (127.18.211.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } attrs { replace: true } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } } }
I20260129 02:59:00.472447 20587 raft_consensus.cc:2955] T 00e79d8f25a34676b1d866eb2068a2b2 P 1c50de89a5a845668696dcb28c3badd6 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 7a9daf5d42a84615a6383cfec9a10608 (127.18.211.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } attrs { replace: true } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } } }
I20260129 02:59:00.474934 21135 raft_consensus.cc:2955] T 00e79d8f25a34676b1d866eb2068a2b2 P 7a9daf5d42a84615a6383cfec9a10608 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 7a9daf5d42a84615a6383cfec9a10608 (127.18.211.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } attrs { replace: true } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } } }
I20260129 02:59:00.478698 21321 raft_consensus.cc:1064] T cb0b3f259ebc4310927338a8740e87b8 P 974b802141bd4f13beffef2aa03f88f9: attempting to promote NON_VOTER 7a9daf5d42a84615a6383cfec9a10608 to VOTER
I20260129 02:59:00.488992 20335 catalog_manager.cc:5654] T 00e79d8f25a34676b1d866eb2068a2b2 P 7a9daf5d42a84615a6383cfec9a10608 reported cstate change: config changed from index 2 to 3, 7a9daf5d42a84615a6383cfec9a10608 (127.18.211.4) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "197b880fa39d432a9b472845876bfaf0" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } attrs { replace: true } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } } }
I20260129 02:59:00.488957 21321 consensus_queue.cc:237] T cb0b3f259ebc4310927338a8740e87b8 P 974b802141bd4f13beffef2aa03f88f9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } attrs { replace: true } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } }
I20260129 02:59:00.500694 20511 raft_consensus.cc:2955] T 00e79d8f25a34676b1d866eb2068a2b2 P 974b802141bd4f13beffef2aa03f88f9 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 7a9daf5d42a84615a6383cfec9a10608 (127.18.211.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } attrs { replace: true } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } } }
I20260129 02:59:00.507129 21340 raft_consensus.cc:1064] T 2e58e9a5a8aa4cecafb464addb4f35ed P 197b880fa39d432a9b472845876bfaf0: attempting to promote NON_VOTER 7a9daf5d42a84615a6383cfec9a10608 to VOTER
I20260129 02:59:00.509824 21340 consensus_queue.cc:237] T 2e58e9a5a8aa4cecafb464addb4f35ed P 197b880fa39d432a9b472845876bfaf0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } attrs { replace: true } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } }
I20260129 02:59:00.524228 21133 raft_consensus.cc:1275] T cb0b3f259ebc4310927338a8740e87b8 P 7a9daf5d42a84615a6383cfec9a10608 [term 1 LEARNER]: Refusing update from remote peer 974b802141bd4f13beffef2aa03f88f9: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260129 02:59:00.526878 20437 raft_consensus.cc:1275] T cb0b3f259ebc4310927338a8740e87b8 P 197b880fa39d432a9b472845876bfaf0 [term 1 FOLLOWER]: Refusing update from remote peer 974b802141bd4f13beffef2aa03f88f9: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260129 02:59:00.529696 21343 consensus_queue.cc:1048] T cb0b3f259ebc4310927338a8740e87b8 P 974b802141bd4f13beffef2aa03f88f9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260129 02:59:00.532234 21320 consensus_queue.cc:1048] T cb0b3f259ebc4310927338a8740e87b8 P 974b802141bd4f13beffef2aa03f88f9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260129 02:59:00.534754 20587 raft_consensus.cc:1275] T cb0b3f259ebc4310927338a8740e87b8 P 1c50de89a5a845668696dcb28c3badd6 [term 1 FOLLOWER]: Refusing update from remote peer 974b802141bd4f13beffef2aa03f88f9: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260129 02:59:00.540671 21328 raft_consensus.cc:1064] T 18f30ebaa4174fd28c958e72d0cc7e24 P 197b880fa39d432a9b472845876bfaf0: attempting to promote NON_VOTER 7a9daf5d42a84615a6383cfec9a10608 to VOTER
I20260129 02:59:00.544984 20587 raft_consensus.cc:1275] T 2e58e9a5a8aa4cecafb464addb4f35ed P 1c50de89a5a845668696dcb28c3badd6 [term 1 FOLLOWER]: Refusing update from remote peer 197b880fa39d432a9b472845876bfaf0: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260129 02:59:00.544857 21328 consensus_queue.cc:237] T 18f30ebaa4174fd28c958e72d0cc7e24 P 197b880fa39d432a9b472845876bfaf0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } attrs { replace: true } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } }
I20260129 02:59:00.546649 21310 consensus_queue.cc:1048] T 2e58e9a5a8aa4cecafb464addb4f35ed P 197b880fa39d432a9b472845876bfaf0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260129 02:59:00.552623 20511 raft_consensus.cc:1275] T 2e58e9a5a8aa4cecafb464addb4f35ed P 974b802141bd4f13beffef2aa03f88f9 [term 1 FOLLOWER]: Refusing update from remote peer 197b880fa39d432a9b472845876bfaf0: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260129 02:59:00.551169 21133 raft_consensus.cc:1275] T 2e58e9a5a8aa4cecafb464addb4f35ed P 7a9daf5d42a84615a6383cfec9a10608 [term 1 LEARNER]: Refusing update from remote peer 197b880fa39d432a9b472845876bfaf0: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260129 02:59:00.554175 21310 consensus_queue.cc:1048] T 2e58e9a5a8aa4cecafb464addb4f35ed P 197b880fa39d432a9b472845876bfaf0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260129 02:59:00.558815 21320 consensus_queue.cc:1048] T cb0b3f259ebc4310927338a8740e87b8 P 974b802141bd4f13beffef2aa03f88f9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260129 02:59:00.558923 21342 consensus_queue.cc:1048] T 2e58e9a5a8aa4cecafb464addb4f35ed P 197b880fa39d432a9b472845876bfaf0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
W20260129 02:59:00.581561 20479 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.211.1:42027, user_credentials={real_user=slave}} blocked reactor thread for 52505.3us
I20260129 02:59:00.651049 21352 raft_consensus.cc:2955] T cb0b3f259ebc4310927338a8740e87b8 P 974b802141bd4f13beffef2aa03f88f9 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 7a9daf5d42a84615a6383cfec9a10608 (127.18.211.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } attrs { replace: true } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } } }
I20260129 02:59:00.660281 20586 raft_consensus.cc:1275] T 18f30ebaa4174fd28c958e72d0cc7e24 P 1c50de89a5a845668696dcb28c3badd6 [term 1 FOLLOWER]: Refusing update from remote peer 197b880fa39d432a9b472845876bfaf0: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260129 02:59:00.662438 21340 consensus_queue.cc:1048] T 18f30ebaa4174fd28c958e72d0cc7e24 P 197b880fa39d432a9b472845876bfaf0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260129 02:59:00.666741 20516 raft_consensus.cc:1275] T 18f30ebaa4174fd28c958e72d0cc7e24 P 974b802141bd4f13beffef2aa03f88f9 [term 1 FOLLOWER]: Refusing update from remote peer 197b880fa39d432a9b472845876bfaf0: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260129 02:59:00.669613 21362 consensus_queue.cc:1048] T 18f30ebaa4174fd28c958e72d0cc7e24 P 197b880fa39d432a9b472845876bfaf0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260129 02:59:00.670881 21133 raft_consensus.cc:1275] T 18f30ebaa4174fd28c958e72d0cc7e24 P 7a9daf5d42a84615a6383cfec9a10608 [term 1 LEARNER]: Refusing update from remote peer 197b880fa39d432a9b472845876bfaf0: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260129 02:59:00.667649 21350 raft_consensus.cc:2955] T 2e58e9a5a8aa4cecafb464addb4f35ed P 197b880fa39d432a9b472845876bfaf0 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 7a9daf5d42a84615a6383cfec9a10608 (127.18.211.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } attrs { replace: true } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } } }
I20260129 02:59:00.678071 20585 raft_consensus.cc:2955] T cb0b3f259ebc4310927338a8740e87b8 P 1c50de89a5a845668696dcb28c3badd6 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 7a9daf5d42a84615a6383cfec9a10608 (127.18.211.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } attrs { replace: true } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } } }
I20260129 02:59:00.678994 21362 consensus_queue.cc:1048] T 18f30ebaa4174fd28c958e72d0cc7e24 P 197b880fa39d432a9b472845876bfaf0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260129 02:59:00.678592 20587 raft_consensus.cc:2955] T 2e58e9a5a8aa4cecafb464addb4f35ed P 1c50de89a5a845668696dcb28c3badd6 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 7a9daf5d42a84615a6383cfec9a10608 (127.18.211.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } attrs { replace: true } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } } }
I20260129 02:59:00.679069 21133 raft_consensus.cc:2955] T 2e58e9a5a8aa4cecafb464addb4f35ed P 7a9daf5d42a84615a6383cfec9a10608 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 7a9daf5d42a84615a6383cfec9a10608 (127.18.211.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } attrs { replace: true } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } } }
I20260129 02:59:00.732359 21136 raft_consensus.cc:2955] T cb0b3f259ebc4310927338a8740e87b8 P 7a9daf5d42a84615a6383cfec9a10608 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 7a9daf5d42a84615a6383cfec9a10608 (127.18.211.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } attrs { replace: true } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } } }
W20260129 02:59:00.790805 21372 log.cc:927] Time spent T cb0b3f259ebc4310927338a8740e87b8 P 197b880fa39d432a9b472845876bfaf0: Append to log took a long time: real 0.071s user 0.000s sys 0.000s
W20260129 02:59:00.801879 20479 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.211.1:42027, user_credentials={real_user=slave}} blocked reactor thread for 86332us
I20260129 02:59:00.880044 20440 consensus_queue.cc:237] T 00e79d8f25a34676b1d866eb2068a2b2 P 197b880fa39d432a9b472845876bfaf0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } }
W20260129 02:59:00.868439 21370 log.cc:927] Time spent T 18f30ebaa4174fd28c958e72d0cc7e24 P 974b802141bd4f13beffef2aa03f88f9: Append to log took a long time: real 0.144s user 0.007s sys 0.000s
I20260129 02:59:00.920928 20337 catalog_manager.cc:5654] T cb0b3f259ebc4310927338a8740e87b8 P 974b802141bd4f13beffef2aa03f88f9 reported cstate change: config changed from index 2 to 3, 7a9daf5d42a84615a6383cfec9a10608 (127.18.211.4) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "974b802141bd4f13beffef2aa03f88f9" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
W20260129 02:59:00.925616 20479 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.211.1:42027, user_credentials={real_user=slave}} blocked reactor thread for 83006.3us
I20260129 02:59:00.932402 20516 raft_consensus.cc:1275] T 00e79d8f25a34676b1d866eb2068a2b2 P 974b802141bd4f13beffef2aa03f88f9 [term 1 FOLLOWER]: Refusing update from remote peer 197b880fa39d432a9b472845876bfaf0: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260129 02:59:00.957247 20516 raft_consensus.cc:2955] T 2e58e9a5a8aa4cecafb464addb4f35ed P 974b802141bd4f13beffef2aa03f88f9 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 7a9daf5d42a84615a6383cfec9a10608 (127.18.211.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } attrs { replace: true } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } } }
W20260129 02:59:00.960422 21371 log.cc:927] Time spent T 18f30ebaa4174fd28c958e72d0cc7e24 P 1c50de89a5a845668696dcb28c3badd6: Append to log took a long time: real 0.100s user 0.000s sys 0.002s
I20260129 02:59:00.963513 21365 consensus_queue.cc:1048] T 00e79d8f25a34676b1d866eb2068a2b2 P 197b880fa39d432a9b472845876bfaf0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.021s
W20260129 02:59:00.984664 20556 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.211.2:34517, user_credentials={real_user=slave}} blocked reactor thread for 56536.5us
I20260129 02:59:00.986680 21133 raft_consensus.cc:1275] T 00e79d8f25a34676b1d866eb2068a2b2 P 7a9daf5d42a84615a6383cfec9a10608 [term 1 FOLLOWER]: Refusing update from remote peer 197b880fa39d432a9b472845876bfaf0: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260129 02:59:00.994174 21342 raft_consensus.cc:2955] T 18f30ebaa4174fd28c958e72d0cc7e24 P 197b880fa39d432a9b472845876bfaf0 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 7a9daf5d42a84615a6383cfec9a10608 (127.18.211.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } attrs { replace: true } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } } }
I20260129 02:59:00.997429 21365 consensus_queue.cc:1048] T 00e79d8f25a34676b1d866eb2068a2b2 P 197b880fa39d432a9b472845876bfaf0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260129 02:59:00.996667 21135 raft_consensus.cc:2955] T 18f30ebaa4174fd28c958e72d0cc7e24 P 7a9daf5d42a84615a6383cfec9a10608 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 7a9daf5d42a84615a6383cfec9a10608 (127.18.211.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } attrs { replace: true } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } } }
I20260129 02:59:01.011590 20338 catalog_manager.cc:5654] T 2e58e9a5a8aa4cecafb464addb4f35ed P 197b880fa39d432a9b472845876bfaf0 reported cstate change: config changed from index 2 to 3, 7a9daf5d42a84615a6383cfec9a10608 (127.18.211.4) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "197b880fa39d432a9b472845876bfaf0" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
W20260129 02:59:01.018311 20480 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.211.3:37735, user_credentials={real_user=slave}} blocked reactor thread for 157949us
I20260129 02:59:01.031607 20585 raft_consensus.cc:2955] T 18f30ebaa4174fd28c958e72d0cc7e24 P 1c50de89a5a845668696dcb28c3badd6 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 7a9daf5d42a84615a6383cfec9a10608 (127.18.211.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } attrs { replace: true } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } } }
I20260129 02:59:01.037139 20437 consensus_queue.cc:237] T 2e58e9a5a8aa4cecafb464addb4f35ed P 197b880fa39d432a9b472845876bfaf0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } }
I20260129 02:59:01.043666 21350 raft_consensus.cc:2955] T 00e79d8f25a34676b1d866eb2068a2b2 P 197b880fa39d432a9b472845876bfaf0 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } } }
I20260129 02:59:01.048619 20435 raft_consensus.cc:2955] T cb0b3f259ebc4310927338a8740e87b8 P 197b880fa39d432a9b472845876bfaf0 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 7a9daf5d42a84615a6383cfec9a10608 (127.18.211.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } attrs { replace: true } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } } }
I20260129 02:59:01.053740 20583 raft_consensus.cc:1275] T 2e58e9a5a8aa4cecafb464addb4f35ed P 1c50de89a5a845668696dcb28c3badd6 [term 1 FOLLOWER]: Refusing update from remote peer 197b880fa39d432a9b472845876bfaf0: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260129 02:59:01.054313 20510 raft_consensus.cc:2955] T 18f30ebaa4174fd28c958e72d0cc7e24 P 974b802141bd4f13beffef2aa03f88f9 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 7a9daf5d42a84615a6383cfec9a10608 (127.18.211.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } attrs { replace: true } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } } }
I20260129 02:59:01.056069 21136 raft_consensus.cc:1275] T 2e58e9a5a8aa4cecafb464addb4f35ed P 7a9daf5d42a84615a6383cfec9a10608 [term 1 FOLLOWER]: Refusing update from remote peer 197b880fa39d432a9b472845876bfaf0: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260129 02:59:01.059278 21368 consensus_queue.cc:1048] T 2e58e9a5a8aa4cecafb464addb4f35ed P 197b880fa39d432a9b472845876bfaf0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260129 02:59:01.061813 21340 consensus_queue.cc:1048] T 2e58e9a5a8aa4cecafb464addb4f35ed P 197b880fa39d432a9b472845876bfaf0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260129 02:59:01.065526 20514 raft_consensus.cc:2955] T 00e79d8f25a34676b1d866eb2068a2b2 P 974b802141bd4f13beffef2aa03f88f9 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } } }
I20260129 02:59:01.084630 20322 catalog_manager.cc:5167] ChangeConfig:REMOVE_PEER RPC for tablet 00e79d8f25a34676b1d866eb2068a2b2 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260129 02:59:01.086601 21133 raft_consensus.cc:2955] T 00e79d8f25a34676b1d866eb2068a2b2 P 7a9daf5d42a84615a6383cfec9a10608 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } } }
I20260129 02:59:01.086927 21364 raft_consensus.cc:2955] T 2e58e9a5a8aa4cecafb464addb4f35ed P 197b880fa39d432a9b472845876bfaf0 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 974b802141bd4f13beffef2aa03f88f9 (127.18.211.2) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } } }
I20260129 02:59:01.088487 20507 consensus_queue.cc:237] T cb0b3f259ebc4310927338a8740e87b8 P 974b802141bd4f13beffef2aa03f88f9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } }
I20260129 02:59:01.089143 21135 raft_consensus.cc:2955] T 2e58e9a5a8aa4cecafb464addb4f35ed P 7a9daf5d42a84615a6383cfec9a10608 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 974b802141bd4f13beffef2aa03f88f9 (127.18.211.2) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } } }
I20260129 02:59:01.113171 21136 raft_consensus.cc:1275] T cb0b3f259ebc4310927338a8740e87b8 P 7a9daf5d42a84615a6383cfec9a10608 [term 1 FOLLOWER]: Refusing update from remote peer 974b802141bd4f13beffef2aa03f88f9: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260129 02:59:01.113989 20585 raft_consensus.cc:2955] T 2e58e9a5a8aa4cecafb464addb4f35ed P 1c50de89a5a845668696dcb28c3badd6 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 974b802141bd4f13beffef2aa03f88f9 (127.18.211.2) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } } }
I20260129 02:59:01.116410 21377 consensus_queue.cc:1048] T cb0b3f259ebc4310927338a8740e87b8 P 974b802141bd4f13beffef2aa03f88f9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260129 02:59:01.117074 20584 raft_consensus.cc:1275] T cb0b3f259ebc4310927338a8740e87b8 P 1c50de89a5a845668696dcb28c3badd6 [term 1 FOLLOWER]: Refusing update from remote peer 974b802141bd4f13beffef2aa03f88f9: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260129 02:59:01.123754 21389 consensus_queue.cc:1048] T cb0b3f259ebc4310927338a8740e87b8 P 974b802141bd4f13beffef2aa03f88f9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260129 02:59:01.129459 21320 raft_consensus.cc:2955] T cb0b3f259ebc4310927338a8740e87b8 P 974b802141bd4f13beffef2aa03f88f9 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 197b880fa39d432a9b472845876bfaf0 (127.18.211.1) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } } }
I20260129 02:59:01.131486 21136 raft_consensus.cc:2955] T cb0b3f259ebc4310927338a8740e87b8 P 7a9daf5d42a84615a6383cfec9a10608 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 197b880fa39d432a9b472845876bfaf0 (127.18.211.1) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } } }
I20260129 02:59:01.147104 20325 catalog_manager.cc:5167] ChangeConfig:REMOVE_PEER RPC for tablet cb0b3f259ebc4310927338a8740e87b8 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260129 02:59:01.148459 20322 catalog_manager.cc:5167] ChangeConfig:REMOVE_PEER RPC for tablet 2e58e9a5a8aa4cecafb464addb4f35ed with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260129 02:59:01.152493 20585 raft_consensus.cc:2955] T cb0b3f259ebc4310927338a8740e87b8 P 1c50de89a5a845668696dcb28c3badd6 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 197b880fa39d432a9b472845876bfaf0 (127.18.211.1) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } } }
I20260129 02:59:01.157558 20338 catalog_manager.cc:5654] T 18f30ebaa4174fd28c958e72d0cc7e24 P 7a9daf5d42a84615a6383cfec9a10608 reported cstate change: config changed from index 2 to 3, 7a9daf5d42a84615a6383cfec9a10608 (127.18.211.4) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "197b880fa39d432a9b472845876bfaf0" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } attrs { replace: true } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } } }
I20260129 02:59:01.167228 20335 catalog_manager.cc:5654] T 2e58e9a5a8aa4cecafb464addb4f35ed P 197b880fa39d432a9b472845876bfaf0 reported cstate change: config changed from index 3 to 4, VOTER 974b802141bd4f13beffef2aa03f88f9 (127.18.211.2) evicted. New cstate: current_term: 1 leader_uuid: "197b880fa39d432a9b472845876bfaf0" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260129 02:59:01.180173 20500 tablet_service.cc:1552] Processing DeleteTablet for tablet 2e58e9a5a8aa4cecafb464addb4f35ed with delete_type TABLET_DATA_TOMBSTONED (TS 974b802141bd4f13beffef2aa03f88f9 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:57580
I20260129 02:59:01.182319 20337 catalog_manager.cc:5654] T 00e79d8f25a34676b1d866eb2068a2b2 P 974b802141bd4f13beffef2aa03f88f9 reported cstate change: config changed from index 3 to 4, VOTER 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3) evicted. New cstate: current_term: 1 leader_uuid: "197b880fa39d432a9b472845876bfaf0" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } } }
I20260129 02:59:01.185017 21390 tablet_replica.cc:333] T 2e58e9a5a8aa4cecafb464addb4f35ed P 974b802141bd4f13beffef2aa03f88f9: stopping tablet replica
I20260129 02:59:01.186168 20437 consensus_queue.cc:237] T 18f30ebaa4174fd28c958e72d0cc7e24 P 197b880fa39d432a9b472845876bfaf0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } }
I20260129 02:59:01.192620 20585 raft_consensus.cc:1275] T 18f30ebaa4174fd28c958e72d0cc7e24 P 1c50de89a5a845668696dcb28c3badd6 [term 1 FOLLOWER]: Refusing update from remote peer 197b880fa39d432a9b472845876bfaf0: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260129 02:59:01.192826 21136 raft_consensus.cc:1275] T 18f30ebaa4174fd28c958e72d0cc7e24 P 7a9daf5d42a84615a6383cfec9a10608 [term 1 FOLLOWER]: Refusing update from remote peer 197b880fa39d432a9b472845876bfaf0: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260129 02:59:01.194200 21361 consensus_queue.cc:1048] T 18f30ebaa4174fd28c958e72d0cc7e24 P 197b880fa39d432a9b472845876bfaf0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260129 02:59:01.196393 21361 consensus_queue.cc:1048] T 18f30ebaa4174fd28c958e72d0cc7e24 P 197b880fa39d432a9b472845876bfaf0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260129 02:59:01.206523 21386 raft_consensus.cc:2955] T 18f30ebaa4174fd28c958e72d0cc7e24 P 197b880fa39d432a9b472845876bfaf0 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 974b802141bd4f13beffef2aa03f88f9 (127.18.211.2) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } } }
I20260129 02:59:01.208387 21390 raft_consensus.cc:2243] T 2e58e9a5a8aa4cecafb464addb4f35ed P 974b802141bd4f13beffef2aa03f88f9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260129 02:59:01.208143 20585 raft_consensus.cc:2955] T 18f30ebaa4174fd28c958e72d0cc7e24 P 1c50de89a5a845668696dcb28c3badd6 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 974b802141bd4f13beffef2aa03f88f9 (127.18.211.2) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } } }
I20260129 02:59:01.209285 21390 raft_consensus.cc:2272] T 2e58e9a5a8aa4cecafb464addb4f35ed P 974b802141bd4f13beffef2aa03f88f9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260129 02:59:01.212430 20579 tablet_service.cc:1552] Processing DeleteTablet for tablet 00e79d8f25a34676b1d866eb2068a2b2 with delete_type TABLET_DATA_TOMBSTONED (TS 1c50de89a5a845668696dcb28c3badd6 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:35744
I20260129 02:59:01.213195 21390 ts_tablet_manager.cc:1916] T 2e58e9a5a8aa4cecafb464addb4f35ed P 974b802141bd4f13beffef2aa03f88f9: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260129 02:59:01.228101 20322 catalog_manager.cc:5167] ChangeConfig:REMOVE_PEER RPC for tablet 18f30ebaa4174fd28c958e72d0cc7e24 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260129 02:59:01.236150 21391 tablet_replica.cc:333] T 00e79d8f25a34676b1d866eb2068a2b2 P 1c50de89a5a845668696dcb28c3badd6: stopping tablet replica
I20260129 02:59:01.239521 21391 raft_consensus.cc:2243] T 00e79d8f25a34676b1d866eb2068a2b2 P 1c50de89a5a845668696dcb28c3badd6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260129 02:59:01.240949 21136 raft_consensus.cc:2955] T 18f30ebaa4174fd28c958e72d0cc7e24 P 7a9daf5d42a84615a6383cfec9a10608 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 974b802141bd4f13beffef2aa03f88f9 (127.18.211.2) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } } }
I20260129 02:59:01.242089 21391 raft_consensus.cc:2272] T 00e79d8f25a34676b1d866eb2068a2b2 P 1c50de89a5a845668696dcb28c3badd6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260129 02:59:01.247958 21390 ts_tablet_manager.cc:1929] T 2e58e9a5a8aa4cecafb464addb4f35ed P 974b802141bd4f13beffef2aa03f88f9: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20260129 02:59:01.248512 21390 log.cc:1199] T 2e58e9a5a8aa4cecafb464addb4f35ed P 974b802141bd4f13beffef2aa03f88f9: Deleting WAL directory at /tmp/dist-test-taskrTP_Jo/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1769655460890924-19276-0/minicluster-data/ts-1-root/wals/2e58e9a5a8aa4cecafb464addb4f35ed
I20260129 02:59:01.250247 20325 catalog_manager.cc:4985] TS 974b802141bd4f13beffef2aa03f88f9 (127.18.211.2:34517): tablet 2e58e9a5a8aa4cecafb464addb4f35ed (table test-workload [id=de73611ac8e3485a98aecc5add247032]) successfully deleted
I20260129 02:59:01.239413 20337 catalog_manager.cc:5654] T cb0b3f259ebc4310927338a8740e87b8 P 974b802141bd4f13beffef2aa03f88f9 reported cstate change: config changed from index 3 to 4, VOTER 197b880fa39d432a9b472845876bfaf0 (127.18.211.1) evicted. New cstate: current_term: 1 leader_uuid: "974b802141bd4f13beffef2aa03f88f9" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260129 02:59:01.250514 20334 catalog_manager.cc:5654] T 18f30ebaa4174fd28c958e72d0cc7e24 P 197b880fa39d432a9b472845876bfaf0 reported cstate change: config changed from index 3 to 4, VOTER 974b802141bd4f13beffef2aa03f88f9 (127.18.211.2) evicted. New cstate: current_term: 1 leader_uuid: "197b880fa39d432a9b472845876bfaf0" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260129 02:59:01.254354 21391 ts_tablet_manager.cc:1916] T 00e79d8f25a34676b1d866eb2068a2b2 P 1c50de89a5a845668696dcb28c3badd6: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260129 02:59:01.263293 20431 tablet_service.cc:1552] Processing DeleteTablet for tablet cb0b3f259ebc4310927338a8740e87b8 with delete_type TABLET_DATA_TOMBSTONED (TS 197b880fa39d432a9b472845876bfaf0 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:55666
I20260129 02:59:01.273926 20500 tablet_service.cc:1552] Processing DeleteTablet for tablet 18f30ebaa4174fd28c958e72d0cc7e24 with delete_type TABLET_DATA_TOMBSTONED (TS 974b802141bd4f13beffef2aa03f88f9 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:57580
I20260129 02:59:01.274946 21390 tablet_replica.cc:333] T 18f30ebaa4174fd28c958e72d0cc7e24 P 974b802141bd4f13beffef2aa03f88f9: stopping tablet replica
I20260129 02:59:01.275148 21391 ts_tablet_manager.cc:1929] T 00e79d8f25a34676b1d866eb2068a2b2 P 1c50de89a5a845668696dcb28c3badd6: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20260129 02:59:01.275594 21391 log.cc:1199] T 00e79d8f25a34676b1d866eb2068a2b2 P 1c50de89a5a845668696dcb28c3badd6: Deleting WAL directory at /tmp/dist-test-taskrTP_Jo/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1769655460890924-19276-0/minicluster-data/ts-2-root/wals/00e79d8f25a34676b1d866eb2068a2b2
I20260129 02:59:01.276149 21390 raft_consensus.cc:2243] T 18f30ebaa4174fd28c958e72d0cc7e24 P 974b802141bd4f13beffef2aa03f88f9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260129 02:59:01.276949 21390 raft_consensus.cc:2272] T 18f30ebaa4174fd28c958e72d0cc7e24 P 974b802141bd4f13beffef2aa03f88f9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260129 02:59:01.278476 20323 catalog_manager.cc:4985] TS 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3:37735): tablet 00e79d8f25a34676b1d866eb2068a2b2 (table test-workload [id=de73611ac8e3485a98aecc5add247032]) successfully deleted
I20260129 02:59:01.284499 21390 ts_tablet_manager.cc:1916] T 18f30ebaa4174fd28c958e72d0cc7e24 P 974b802141bd4f13beffef2aa03f88f9: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260129 02:59:01.291640 21392 tablet_replica.cc:333] T cb0b3f259ebc4310927338a8740e87b8 P 197b880fa39d432a9b472845876bfaf0: stopping tablet replica
I20260129 02:59:01.292936 21392 raft_consensus.cc:2243] T cb0b3f259ebc4310927338a8740e87b8 P 197b880fa39d432a9b472845876bfaf0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260129 02:59:01.293607 21392 raft_consensus.cc:2272] T cb0b3f259ebc4310927338a8740e87b8 P 197b880fa39d432a9b472845876bfaf0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260129 02:59:01.296432 21392 ts_tablet_manager.cc:1916] T cb0b3f259ebc4310927338a8740e87b8 P 197b880fa39d432a9b472845876bfaf0: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260129 02:59:01.301725 21390 ts_tablet_manager.cc:1929] T 18f30ebaa4174fd28c958e72d0cc7e24 P 974b802141bd4f13beffef2aa03f88f9: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20260129 02:59:01.302213 21390 log.cc:1199] T 18f30ebaa4174fd28c958e72d0cc7e24 P 974b802141bd4f13beffef2aa03f88f9: Deleting WAL directory at /tmp/dist-test-taskrTP_Jo/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1769655460890924-19276-0/minicluster-data/ts-1-root/wals/18f30ebaa4174fd28c958e72d0cc7e24
I20260129 02:59:01.303793 20325 catalog_manager.cc:4985] TS 974b802141bd4f13beffef2aa03f88f9 (127.18.211.2:34517): tablet 18f30ebaa4174fd28c958e72d0cc7e24 (table test-workload [id=de73611ac8e3485a98aecc5add247032]) successfully deleted
I20260129 02:59:01.309919 21392 ts_tablet_manager.cc:1929] T cb0b3f259ebc4310927338a8740e87b8 P 197b880fa39d432a9b472845876bfaf0: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20260129 02:59:01.310402 21392 log.cc:1199] T cb0b3f259ebc4310927338a8740e87b8 P 197b880fa39d432a9b472845876bfaf0: Deleting WAL directory at /tmp/dist-test-taskrTP_Jo/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1769655460890924-19276-0/minicluster-data/ts-0-root/wals/cb0b3f259ebc4310927338a8740e87b8
I20260129 02:59:01.312037 20322 catalog_manager.cc:4985] TS 197b880fa39d432a9b472845876bfaf0 (127.18.211.1:42027): tablet cb0b3f259ebc4310927338a8740e87b8 (table test-workload [id=de73611ac8e3485a98aecc5add247032]) successfully deleted
W20260129 02:59:03.076404 20482 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.519s user 0.003s sys 0.024s
W20260129 02:59:03.077039 20482 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.519s user 0.003s sys 0.024s
W20260129 02:59:03.092094 20480 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.533s user 0.011s sys 0.033s
W20260129 02:59:03.092509 20480 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.534s user 0.011s sys 0.035s
W20260129 02:59:03.092772 20480 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.211.3:37735, user_credentials={real_user=slave}} blocked reactor thread for 534401us
W20260129 02:59:03.102828 20404 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.502s user 0.011s sys 0.005s
W20260129 02:59:03.103960 20404 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.503s user 0.011s sys 0.005s
I20260129 02:59:03.197279 21398 raft_consensus.cc:493] T 0f20af7fbe5d4db89a6f17948d98912c P 974b802141bd4f13beffef2aa03f88f9 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 197b880fa39d432a9b472845876bfaf0)
I20260129 02:59:03.198474 21398 raft_consensus.cc:515] T 0f20af7fbe5d4db89a6f17948d98912c P 974b802141bd4f13beffef2aa03f88f9 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:59:03.202370 21397 raft_consensus.cc:493] T e4d18c35b0a7493e85d0a4b92956a48f P 974b802141bd4f13beffef2aa03f88f9 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 197b880fa39d432a9b472845876bfaf0)
I20260129 02:59:03.203111 21397 raft_consensus.cc:515] T e4d18c35b0a7493e85d0a4b92956a48f P 974b802141bd4f13beffef2aa03f88f9 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:59:03.206177 21397 leader_election.cc:290] T e4d18c35b0a7493e85d0a4b92956a48f P 974b802141bd4f13beffef2aa03f88f9 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 197b880fa39d432a9b472845876bfaf0 (127.18.211.1:42027), 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3:37735)
W20260129 02:59:03.261709 20406 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.560s user 0.010s sys 0.010s
W20260129 02:59:03.262229 20406 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.561s user 0.010s sys 0.010s
I20260129 02:59:03.278726 21397 raft_consensus.cc:493] T 75be4bd6bd45474795fd472eff2953ac P 974b802141bd4f13beffef2aa03f88f9 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 197b880fa39d432a9b472845876bfaf0)
I20260129 02:59:03.279742 21397 raft_consensus.cc:515] T 75be4bd6bd45474795fd472eff2953ac P 974b802141bd4f13beffef2aa03f88f9 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:59:03.287542 21397 leader_election.cc:290] T 75be4bd6bd45474795fd472eff2953ac P 974b802141bd4f13beffef2aa03f88f9 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 197b880fa39d432a9b472845876bfaf0 (127.18.211.1:42027), 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3:37735)
I20260129 02:59:03.331725 21398 leader_election.cc:290] T 0f20af7fbe5d4db89a6f17948d98912c P 974b802141bd4f13beffef2aa03f88f9 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 197b880fa39d432a9b472845876bfaf0 (127.18.211.1:42027), 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3:37735)
I20260129 02:59:03.336053 21408 raft_consensus.cc:493] T 3ccac82a01a2445fa5b0ce6c0fcf2295 P 974b802141bd4f13beffef2aa03f88f9 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 197b880fa39d432a9b472845876bfaf0)
I20260129 02:59:03.336396 21385 raft_consensus.cc:493] T a4c1226b987b468fa84dfc6ba0e2dbe1 P 197b880fa39d432a9b472845876bfaf0 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 974b802141bd4f13beffef2aa03f88f9)
I20260129 02:59:03.338742 21396 raft_consensus.cc:493] T 87d5fe48a44a408aaf03e894a67ecb2f P 197b880fa39d432a9b472845876bfaf0 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 974b802141bd4f13beffef2aa03f88f9)
I20260129 02:59:03.340673 21411 raft_consensus.cc:493] T e7d83d4ccf684e76acb29a0dc844e1b8 P 974b802141bd4f13beffef2aa03f88f9 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 197b880fa39d432a9b472845876bfaf0)
I20260129 02:59:03.342657 21401 raft_consensus.cc:493] T 45cc5f3d2688486d9b6dbe7e666dedde P 197b880fa39d432a9b472845876bfaf0 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 1c50de89a5a845668696dcb28c3badd6)
I20260129 02:59:03.344298 21407 raft_consensus.cc:493] T cedce331e1584ac8b636c9b8a72b2ba0 P 1c50de89a5a845668696dcb28c3badd6 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 974b802141bd4f13beffef2aa03f88f9)
I20260129 02:59:03.344385 21405 raft_consensus.cc:493] T 4e420f90884f4796997b5ea41e49e411 P 1c50de89a5a845668696dcb28c3badd6 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 197b880fa39d432a9b472845876bfaf0)
I20260129 02:59:03.346549 21402 raft_consensus.cc:493] T cb0b3f259ebc4310927338a8740e87b8 P 1c50de89a5a845668696dcb28c3badd6 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 974b802141bd4f13beffef2aa03f88f9)
I20260129 02:59:03.352954 21399 raft_consensus.cc:493] T dc2f4cb752cf4d59b5f6bc689f5d8104 P 197b880fa39d432a9b472845876bfaf0 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 1c50de89a5a845668696dcb28c3badd6)
I20260129 02:59:03.387955 21398 raft_consensus.cc:493] T 00e79d8f25a34676b1d866eb2068a2b2 P 974b802141bd4f13beffef2aa03f88f9 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 197b880fa39d432a9b472845876bfaf0)
I20260129 02:59:03.400485 21400 raft_consensus.cc:493] T cb4727386a3e48aca5d71cc6c611c197 P 197b880fa39d432a9b472845876bfaf0 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 974b802141bd4f13beffef2aa03f88f9)
I20260129 02:59:03.387676 21385 raft_consensus.cc:515] T a4c1226b987b468fa84dfc6ba0e2dbe1 P 197b880fa39d432a9b472845876bfaf0 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:59:03.404568 21394 raft_consensus.cc:493] T 4408006eb5174de1896b037cf3f18457 P 197b880fa39d432a9b472845876bfaf0 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 974b802141bd4f13beffef2aa03f88f9)
I20260129 02:59:03.404567 21400 raft_consensus.cc:515] T cb4727386a3e48aca5d71cc6c611c197 P 197b880fa39d432a9b472845876bfaf0 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:59:03.410238 21406 raft_consensus.cc:493] T 724a236524cb4e7ba3428880681d6cc2 P 197b880fa39d432a9b472845876bfaf0 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 974b802141bd4f13beffef2aa03f88f9)
I20260129 02:59:03.397544 21405 raft_consensus.cc:515] T 4e420f90884f4796997b5ea41e49e411 P 1c50de89a5a845668696dcb28c3badd6 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:59:03.402153 21401 raft_consensus.cc:515] T 45cc5f3d2688486d9b6dbe7e666dedde P 197b880fa39d432a9b472845876bfaf0 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:59:03.394501 21407 raft_consensus.cc:515] T cedce331e1584ac8b636c9b8a72b2ba0 P 1c50de89a5a845668696dcb28c3badd6 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:59:03.398483 21399 raft_consensus.cc:515] T dc2f4cb752cf4d59b5f6bc689f5d8104 P 197b880fa39d432a9b472845876bfaf0 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:59:03.420233 21409 raft_consensus.cc:493] T c2633a6fc34c44579de348d208b0c01e P 974b802141bd4f13beffef2aa03f88f9 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 197b880fa39d432a9b472845876bfaf0)
I20260129 02:59:03.391913 21396 raft_consensus.cc:515] T 87d5fe48a44a408aaf03e894a67ecb2f P 197b880fa39d432a9b472845876bfaf0 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } }
I20260129 02:59:03.437614 21412 raft_consensus.cc:493] T 76f713c3331540d2a5b76164103e9eb1 P 1c50de89a5a845668696dcb28c3badd6 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 974b802141bd4f13beffef2aa03f88f9)
I20260129 02:59:03.423339 21398 raft_consensus.cc:515] T 00e79d8f25a34676b1d866eb2068a2b2 P 974b802141bd4f13beffef2aa03f88f9 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } }
I20260129 02:59:03.440838 21374 raft_consensus.cc:493] T a609ac02fae84725ab743f95f51d33ce P 974b802141bd4f13beffef2aa03f88f9 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 197b880fa39d432a9b472845876bfaf0)
W20260129 02:59:03.440824 20556 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.211.2:34517, user_credentials={real_user=slave}} blocked reactor thread for 43640.8us
I20260129 02:59:03.396521 21402 raft_consensus.cc:515] T cb0b3f259ebc4310927338a8740e87b8 P 1c50de89a5a845668696dcb28c3badd6 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } }
I20260129 02:59:03.396678 21408 raft_consensus.cc:515] T 3ccac82a01a2445fa5b0ce6c0fcf2295 P 974b802141bd4f13beffef2aa03f88f9 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:59:03.394043 21411 raft_consensus.cc:515] T e7d83d4ccf684e76acb29a0dc844e1b8 P 974b802141bd4f13beffef2aa03f88f9 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:59:03.432756 21397 raft_consensus.cc:493] T febffe4a8b394368b69b1d164308fb9a P 974b802141bd4f13beffef2aa03f88f9 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 197b880fa39d432a9b472845876bfaf0)
I20260129 02:59:03.459028 21412 raft_consensus.cc:515] T 76f713c3331540d2a5b76164103e9eb1 P 1c50de89a5a845668696dcb28c3badd6 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:59:03.439957 21410 raft_consensus.cc:493] T 7a9a69cd82724b1faed1d10aa091fc4a P 974b802141bd4f13beffef2aa03f88f9 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 197b880fa39d432a9b472845876bfaf0)
I20260129 02:59:03.442052 21396 leader_election.cc:290] T 87d5fe48a44a408aaf03e894a67ecb2f P 197b880fa39d432a9b472845876bfaf0 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 974b802141bd4f13beffef2aa03f88f9 (127.18.211.2:34517), 7a9daf5d42a84615a6383cfec9a10608 (127.18.211.4:40439)
I20260129 02:59:03.441550 21374 raft_consensus.cc:515] T a609ac02fae84725ab743f95f51d33ce P 974b802141bd4f13beffef2aa03f88f9 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } }
I20260129 02:59:03.475725 21405 leader_election.cc:290] T 4e420f90884f4796997b5ea41e49e411 P 1c50de89a5a845668696dcb28c3badd6 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 197b880fa39d432a9b472845876bfaf0 (127.18.211.1:42027), 974b802141bd4f13beffef2aa03f88f9 (127.18.211.2:34517)
I20260129 02:59:03.476352 21405 raft_consensus.cc:493] T a5a05fbcbbcf4916bbc01fd1a5b5e90d P 1c50de89a5a845668696dcb28c3badd6 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 974b802141bd4f13beffef2aa03f88f9)
I20260129 02:59:03.476387 21374 leader_election.cc:290] T a609ac02fae84725ab743f95f51d33ce P 974b802141bd4f13beffef2aa03f88f9 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 197b880fa39d432a9b472845876bfaf0 (127.18.211.1:42027), 7a9daf5d42a84615a6383cfec9a10608 (127.18.211.4:40439)
I20260129 02:59:03.476850 21405 raft_consensus.cc:515] T a5a05fbcbbcf4916bbc01fd1a5b5e90d P 1c50de89a5a845668696dcb28c3badd6 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:59:03.478832 21405 leader_election.cc:290] T a5a05fbcbbcf4916bbc01fd1a5b5e90d P 1c50de89a5a845668696dcb28c3badd6 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 197b880fa39d432a9b472845876bfaf0 (127.18.211.1:42027), 974b802141bd4f13beffef2aa03f88f9 (127.18.211.2:34517)
I20260129 02:59:03.431631 21394 raft_consensus.cc:515] T 4408006eb5174de1896b037cf3f18457 P 197b880fa39d432a9b472845876bfaf0 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:59:03.488981 21404 raft_consensus.cc:493] T c934d31efd3f4e5195605ddc354d14e1 P 1c50de89a5a845668696dcb28c3badd6 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 197b880fa39d432a9b472845876bfaf0)
I20260129 02:59:03.497860 21413 raft_consensus.cc:493] T c934d31efd3f4e5195605ddc354d14e1 P 974b802141bd4f13beffef2aa03f88f9 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 197b880fa39d432a9b472845876bfaf0)
I20260129 02:59:03.498625 21413 raft_consensus.cc:515] T c934d31efd3f4e5195605ddc354d14e1 P 974b802141bd4f13beffef2aa03f88f9 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:59:03.501302 21413 leader_election.cc:290] T c934d31efd3f4e5195605ddc354d14e1 P 974b802141bd4f13beffef2aa03f88f9 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 197b880fa39d432a9b472845876bfaf0 (127.18.211.1:42027), 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3:37735)
I20260129 02:59:03.505764 21408 leader_election.cc:290] T 3ccac82a01a2445fa5b0ce6c0fcf2295 P 974b802141bd4f13beffef2aa03f88f9 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 197b880fa39d432a9b472845876bfaf0 (127.18.211.1:42027), 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3:37735)
I20260129 02:59:03.437994 21406 raft_consensus.cc:515] T 724a236524cb4e7ba3428880681d6cc2 P 197b880fa39d432a9b472845876bfaf0 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:59:03.506016 20436 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "75be4bd6bd45474795fd472eff2953ac" candidate_uuid: "974b802141bd4f13beffef2aa03f88f9" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "197b880fa39d432a9b472845876bfaf0" is_pre_election: true
I20260129 02:59:03.505873 20439 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e4d18c35b0a7493e85d0a4b92956a48f" candidate_uuid: "974b802141bd4f13beffef2aa03f88f9" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "197b880fa39d432a9b472845876bfaf0" is_pre_election: true
W20260129 02:59:03.507748 20556 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.211.2:34517, user_credentials={real_user=slave}} blocked reactor thread for 44253.8us
I20260129 02:59:03.478854 21403 raft_consensus.cc:493] T 7a9a69cd82724b1faed1d10aa091fc4a P 1c50de89a5a845668696dcb28c3badd6 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 197b880fa39d432a9b472845876bfaf0)
I20260129 02:59:03.487792 21396 raft_consensus.cc:493] T 1299723a4476420ab02fd8bfd7aa0645 P 197b880fa39d432a9b472845876bfaf0 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 974b802141bd4f13beffef2aa03f88f9)
I20260129 02:59:03.467912 21397 raft_consensus.cc:515] T febffe4a8b394368b69b1d164308fb9a P 974b802141bd4f13beffef2aa03f88f9 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } }
I20260129 02:59:03.463431 21409 raft_consensus.cc:515] T c2633a6fc34c44579de348d208b0c01e P 974b802141bd4f13beffef2aa03f88f9 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:59:03.519140 21404 raft_consensus.cc:515] T c934d31efd3f4e5195605ddc354d14e1 P 1c50de89a5a845668696dcb28c3badd6 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:59:03.523319 20439 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4e420f90884f4796997b5ea41e49e411" candidate_uuid: "1c50de89a5a845668696dcb28c3badd6" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "197b880fa39d432a9b472845876bfaf0" is_pre_election: true
I20260129 02:59:03.523561 21402 leader_election.cc:290] T cb0b3f259ebc4310927338a8740e87b8 P 1c50de89a5a845668696dcb28c3badd6 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 974b802141bd4f13beffef2aa03f88f9 (127.18.211.2:34517), 7a9daf5d42a84615a6383cfec9a10608 (127.18.211.4:40439)
I20260129 02:59:03.523850 20435 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "cedce331e1584ac8b636c9b8a72b2ba0" candidate_uuid: "1c50de89a5a845668696dcb28c3badd6" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "197b880fa39d432a9b472845876bfaf0" is_pre_election: true
I20260129 02:59:03.524171 21404 leader_election.cc:290] T c934d31efd3f4e5195605ddc354d14e1 P 1c50de89a5a845668696dcb28c3badd6 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 197b880fa39d432a9b472845876bfaf0 (127.18.211.1:42027), 974b802141bd4f13beffef2aa03f88f9 (127.18.211.2:34517)
I20260129 02:59:03.524706 20439 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a5a05fbcbbcf4916bbc01fd1a5b5e90d" candidate_uuid: "1c50de89a5a845668696dcb28c3badd6" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "197b880fa39d432a9b472845876bfaf0" is_pre_election: true
I20260129 02:59:03.519574 21401 leader_election.cc:290] T 45cc5f3d2688486d9b6dbe7e666dedde P 197b880fa39d432a9b472845876bfaf0 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 974b802141bd4f13beffef2aa03f88f9 (127.18.211.2:34517), 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3:37735)
I20260129 02:59:03.526050 21401 raft_consensus.cc:493] T e33a417cf9c54cf5af73a4ee3aac4d3a P 197b880fa39d432a9b472845876bfaf0 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 1c50de89a5a845668696dcb28c3badd6)
I20260129 02:59:03.526584 21401 raft_consensus.cc:515] T e33a417cf9c54cf5af73a4ee3aac4d3a P 197b880fa39d432a9b472845876bfaf0 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:59:03.527782 21398 leader_election.cc:290] T 00e79d8f25a34676b1d866eb2068a2b2 P 974b802141bd4f13beffef2aa03f88f9 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 197b880fa39d432a9b472845876bfaf0 (127.18.211.1:42027), 7a9daf5d42a84615a6383cfec9a10608 (127.18.211.4:40439)
I20260129 02:59:03.528865 21412 leader_election.cc:290] T 76f713c3331540d2a5b76164103e9eb1 P 1c50de89a5a845668696dcb28c3badd6 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 197b880fa39d432a9b472845876bfaf0 (127.18.211.1:42027), 974b802141bd4f13beffef2aa03f88f9 (127.18.211.2:34517)
I20260129 02:59:03.529199 21406 leader_election.cc:290] T 724a236524cb4e7ba3428880681d6cc2 P 197b880fa39d432a9b472845876bfaf0 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 974b802141bd4f13beffef2aa03f88f9 (127.18.211.2:34517), 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3:37735)
I20260129 02:59:03.520102 21407 leader_election.cc:290] T cedce331e1584ac8b636c9b8a72b2ba0 P 1c50de89a5a845668696dcb28c3badd6 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 197b880fa39d432a9b472845876bfaf0 (127.18.211.1:42027), 974b802141bd4f13beffef2aa03f88f9 (127.18.211.2:34517)
I20260129 02:59:03.531778 21396 raft_consensus.cc:515] T 1299723a4476420ab02fd8bfd7aa0645 P 197b880fa39d432a9b472845876bfaf0 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:59:03.532389 21409 leader_election.cc:290] T c2633a6fc34c44579de348d208b0c01e P 974b802141bd4f13beffef2aa03f88f9 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 197b880fa39d432a9b472845876bfaf0 (127.18.211.1:42027), 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3:37735)
I20260129 02:59:03.532972 21416 raft_consensus.cc:493] T 4f8f3a38f6e24ba4857875181f447442 P 197b880fa39d432a9b472845876bfaf0 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 974b802141bd4f13beffef2aa03f88f9)
I20260129 02:59:03.533087 21399 leader_election.cc:290] T dc2f4cb752cf4d59b5f6bc689f5d8104 P 197b880fa39d432a9b472845876bfaf0 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 974b802141bd4f13beffef2aa03f88f9 (127.18.211.2:34517), 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3:37735)
I20260129 02:59:03.528811 21401 leader_election.cc:290] T e33a417cf9c54cf5af73a4ee3aac4d3a P 197b880fa39d432a9b472845876bfaf0 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 974b802141bd4f13beffef2aa03f88f9 (127.18.211.2:34517), 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3:37735)
I20260129 02:59:03.524714 21400 leader_election.cc:290] T cb4727386a3e48aca5d71cc6c611c197 P 197b880fa39d432a9b472845876bfaf0 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 974b802141bd4f13beffef2aa03f88f9 (127.18.211.2:34517), 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3:37735)
I20260129 02:59:03.538285 21405 raft_consensus.cc:493] T 1299723a4476420ab02fd8bfd7aa0645 P 1c50de89a5a845668696dcb28c3badd6 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 974b802141bd4f13beffef2aa03f88f9)
I20260129 02:59:03.512312 20432 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0f20af7fbe5d4db89a6f17948d98912c" candidate_uuid: "974b802141bd4f13beffef2aa03f88f9" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "197b880fa39d432a9b472845876bfaf0" is_pre_election: true
I20260129 02:59:03.543001 21411 leader_election.cc:290] T e7d83d4ccf684e76acb29a0dc844e1b8 P 974b802141bd4f13beffef2aa03f88f9 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 197b880fa39d432a9b472845876bfaf0 (127.18.211.1:42027), 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3:37735)
I20260129 02:59:03.533851 21396 leader_election.cc:290] T 1299723a4476420ab02fd8bfd7aa0645 P 197b880fa39d432a9b472845876bfaf0 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 974b802141bd4f13beffef2aa03f88f9 (127.18.211.2:34517), 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3:37735)
I20260129 02:59:03.535653 21406 raft_consensus.cc:493] T 368953c9102643b2a21f99af36b51905 P 197b880fa39d432a9b472845876bfaf0 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 1c50de89a5a845668696dcb28c3badd6)
I20260129 02:59:03.539842 21402 raft_consensus.cc:493] T fcddab6bca684adf8ea032289a00f90e P 1c50de89a5a845668696dcb28c3badd6 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 197b880fa39d432a9b472845876bfaf0)
I20260129 02:59:03.549191 21415 raft_consensus.cc:493] T 2ac659d732314e1288849f08fe5e074e P 197b880fa39d432a9b472845876bfaf0 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 974b802141bd4f13beffef2aa03f88f9)
I20260129 02:59:03.550117 21402 raft_consensus.cc:515] T fcddab6bca684adf8ea032289a00f90e P 1c50de89a5a845668696dcb28c3badd6 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:59:03.540026 21404 raft_consensus.cc:493] T 1ce4ddf5b10e4ae8b193b13b2e5936ee P 1c50de89a5a845668696dcb28c3badd6 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 197b880fa39d432a9b472845876bfaf0)
I20260129 02:59:03.490090 21410 raft_consensus.cc:515] T 7a9a69cd82724b1faed1d10aa091fc4a P 974b802141bd4f13beffef2aa03f88f9 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:59:03.550457 21415 raft_consensus.cc:515] T 2ac659d732314e1288849f08fe5e074e P 197b880fa39d432a9b472845876bfaf0 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:59:03.555881 21420 raft_consensus.cc:493] T 3906b080a5b34a74b8b64128635e8997 P 1c50de89a5a845668696dcb28c3badd6 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 197b880fa39d432a9b472845876bfaf0)
I20260129 02:59:03.550711 21394 leader_election.cc:290] T 4408006eb5174de1896b037cf3f18457 P 197b880fa39d432a9b472845876bfaf0 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 974b802141bd4f13beffef2aa03f88f9 (127.18.211.2:34517), 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3:37735)
I20260129 02:59:03.533614 21416 raft_consensus.cc:515] T 4f8f3a38f6e24ba4857875181f447442 P 197b880fa39d432a9b472845876bfaf0 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:59:03.547781 21401 raft_consensus.cc:493] T fd8311b779af4e1fa1402b2a02a602f2 P 197b880fa39d432a9b472845876bfaf0 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 1c50de89a5a845668696dcb28c3badd6)
I20260129 02:59:03.548409 20436 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a609ac02fae84725ab743f95f51d33ce" candidate_uuid: "974b802141bd4f13beffef2aa03f88f9" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "197b880fa39d432a9b472845876bfaf0" is_pre_election: true
I20260129 02:59:03.560235 21417 raft_consensus.cc:493] T add8d4923d1a489e86882c3a25692b95 P 197b880fa39d432a9b472845876bfaf0 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 1c50de89a5a845668696dcb28c3badd6)
I20260129 02:59:03.529640 21403 raft_consensus.cc:515] T 7a9a69cd82724b1faed1d10aa091fc4a P 1c50de89a5a845668696dcb28c3badd6 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:59:03.576387 21403 leader_election.cc:290] T 7a9a69cd82724b1faed1d10aa091fc4a P 1c50de89a5a845668696dcb28c3badd6 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 197b880fa39d432a9b472845876bfaf0 (127.18.211.1:42027), 974b802141bd4f13beffef2aa03f88f9 (127.18.211.2:34517)
I20260129 02:59:03.549664 21405 raft_consensus.cc:515] T 1299723a4476420ab02fd8bfd7aa0645 P 1c50de89a5a845668696dcb28c3badd6 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
W20260129 02:59:03.586138 20556 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.211.2:34517, user_credentials={real_user=slave}} blocked reactor thread for 65195.7us
I20260129 02:59:03.557577 21406 raft_consensus.cc:515] T 368953c9102643b2a21f99af36b51905 P 197b880fa39d432a9b472845876bfaf0 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:59:03.559136 21404 raft_consensus.cc:515] T 1ce4ddf5b10e4ae8b193b13b2e5936ee P 1c50de89a5a845668696dcb28c3badd6 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:59:03.584095 21401 raft_consensus.cc:515] T fd8311b779af4e1fa1402b2a02a602f2 P 197b880fa39d432a9b472845876bfaf0 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } }
I20260129 02:59:03.580399 21415 leader_election.cc:290] T 2ac659d732314e1288849f08fe5e074e P 197b880fa39d432a9b472845876bfaf0 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 974b802141bd4f13beffef2aa03f88f9 (127.18.211.2:34517), 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3:37735)
I20260129 02:59:03.628638 21419 raft_consensus.cc:493] T 4e3805ffa70242e8ade66da260dddf66 P 1c50de89a5a845668696dcb28c3badd6 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 197b880fa39d432a9b472845876bfaf0)
I20260129 02:59:03.629150 21418 raft_consensus.cc:493] T c891d2bff4ce4568861bbc992e91fbb0 P 1c50de89a5a845668696dcb28c3badd6 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 197b880fa39d432a9b472845876bfaf0)
I20260129 02:59:03.720772 20439 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c934d31efd3f4e5195605ddc354d14e1" candidate_uuid: "1c50de89a5a845668696dcb28c3badd6" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "197b880fa39d432a9b472845876bfaf0" is_pre_election: true
I20260129 02:59:03.716547 21423 raft_consensus.cc:493] T c891d2bff4ce4568861bbc992e91fbb0 P 974b802141bd4f13beffef2aa03f88f9 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 197b880fa39d432a9b472845876bfaf0)
I20260129 02:59:03.720958 20436 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7a9a69cd82724b1faed1d10aa091fc4a" candidate_uuid: "1c50de89a5a845668696dcb28c3badd6" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "197b880fa39d432a9b472845876bfaf0" is_pre_election: true
I20260129 02:59:03.585163 21420 raft_consensus.cc:515] T 3906b080a5b34a74b8b64128635e8997 P 1c50de89a5a845668696dcb28c3badd6 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:59:03.717310 21397 leader_election.cc:290] T febffe4a8b394368b69b1d164308fb9a P 974b802141bd4f13beffef2aa03f88f9 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 197b880fa39d432a9b472845876bfaf0 (127.18.211.1:42027), 7a9daf5d42a84615a6383cfec9a10608 (127.18.211.4:40439)
I20260129 02:59:03.718734 21409 raft_consensus.cc:493] T 926a250fa5834e88b96ccecbefbf5086 P 974b802141bd4f13beffef2aa03f88f9 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 197b880fa39d432a9b472845876bfaf0)
I20260129 02:59:03.732348 21424 raft_consensus.cc:493] T cb0b3f259ebc4310927338a8740e87b8 P 7a9daf5d42a84615a6383cfec9a10608 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 974b802141bd4f13beffef2aa03f88f9)
I20260129 02:59:03.732740 21405 leader_election.cc:290] T 1299723a4476420ab02fd8bfd7aa0645 P 1c50de89a5a845668696dcb28c3badd6 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 197b880fa39d432a9b472845876bfaf0 (127.18.211.1:42027), 974b802141bd4f13beffef2aa03f88f9 (127.18.211.2:34517)
I20260129 02:59:03.719126 21416 leader_election.cc:290] T 4f8f3a38f6e24ba4857875181f447442 P 197b880fa39d432a9b472845876bfaf0 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 974b802141bd4f13beffef2aa03f88f9 (127.18.211.2:34517), 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3:37735)
I20260129 02:59:03.720626 21418 raft_consensus.cc:515] T c891d2bff4ce4568861bbc992e91fbb0 P 1c50de89a5a845668696dcb28c3badd6 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:59:03.749461 21405 raft_consensus.cc:493] T 0678b8b468bd43cdbc3e485900844821 P 1c50de89a5a845668696dcb28c3badd6 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 197b880fa39d432a9b472845876bfaf0)
I20260129 02:59:03.727296 20432 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "76f713c3331540d2a5b76164103e9eb1" candidate_uuid: "1c50de89a5a845668696dcb28c3badd6" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "197b880fa39d432a9b472845876bfaf0" is_pre_election: true
I20260129 02:59:03.750866 21405 raft_consensus.cc:515] T 0678b8b468bd43cdbc3e485900844821 P 1c50de89a5a845668696dcb28c3badd6 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:59:03.585853 21417 raft_consensus.cc:515] T add8d4923d1a489e86882c3a25692b95 P 197b880fa39d432a9b472845876bfaf0 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:59:03.760424 21405 leader_election.cc:290] T 0678b8b468bd43cdbc3e485900844821 P 1c50de89a5a845668696dcb28c3badd6 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 197b880fa39d432a9b472845876bfaf0 (127.18.211.1:42027), 974b802141bd4f13beffef2aa03f88f9 (127.18.211.2:34517)
I20260129 02:59:03.752967 21414 raft_consensus.cc:493] T 0f20af7fbe5d4db89a6f17948d98912c P 1c50de89a5a845668696dcb28c3badd6 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 197b880fa39d432a9b472845876bfaf0)
I20260129 02:59:03.736809 21403 raft_consensus.cc:493] T 2ac659d732314e1288849f08fe5e074e P 1c50de89a5a845668696dcb28c3badd6 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 974b802141bd4f13beffef2aa03f88f9)
I20260129 02:59:03.767930 21426 raft_consensus.cc:493] T 3ccac82a01a2445fa5b0ce6c0fcf2295 P 1c50de89a5a845668696dcb28c3badd6 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 197b880fa39d432a9b472845876bfaf0)
I20260129 02:59:03.768139 21403 raft_consensus.cc:515] T 2ac659d732314e1288849f08fe5e074e P 1c50de89a5a845668696dcb28c3badd6 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:59:03.768535 21426 raft_consensus.cc:515] T 3ccac82a01a2445fa5b0ce6c0fcf2295 P 1c50de89a5a845668696dcb28c3badd6 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:59:03.770290 21403 leader_election.cc:290] T 2ac659d732314e1288849f08fe5e074e P 1c50de89a5a845668696dcb28c3badd6 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 197b880fa39d432a9b472845876bfaf0 (127.18.211.1:42027), 974b802141bd4f13beffef2aa03f88f9 (127.18.211.2:34517)
I20260129 02:59:03.726424 21419 raft_consensus.cc:515] T 4e3805ffa70242e8ade66da260dddf66 P 1c50de89a5a845668696dcb28c3badd6 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:59:03.770651 21426 leader_election.cc:290] T 3ccac82a01a2445fa5b0ce6c0fcf2295 P 1c50de89a5a845668696dcb28c3badd6 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 197b880fa39d432a9b472845876bfaf0 (127.18.211.1:42027), 974b802141bd4f13beffef2aa03f88f9 (127.18.211.2:34517)
I20260129 02:59:03.740906 21423 raft_consensus.cc:515] T c891d2bff4ce4568861bbc992e91fbb0 P 974b802141bd4f13beffef2aa03f88f9 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:59:03.761340 21414 raft_consensus.cc:515] T 0f20af7fbe5d4db89a6f17948d98912c P 1c50de89a5a845668696dcb28c3badd6 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:59:03.749715 21409 raft_consensus.cc:515] T 926a250fa5834e88b96ccecbefbf5086 P 974b802141bd4f13beffef2aa03f88f9 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:59:03.767009 21406 leader_election.cc:290] T 368953c9102643b2a21f99af36b51905 P 197b880fa39d432a9b472845876bfaf0 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 974b802141bd4f13beffef2aa03f88f9 (127.18.211.2:34517), 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3:37735)
I20260129 02:59:03.772071 21402 leader_election.cc:290] T fcddab6bca684adf8ea032289a00f90e P 1c50de89a5a845668696dcb28c3badd6 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 197b880fa39d432a9b472845876bfaf0 (127.18.211.1:42027), 974b802141bd4f13beffef2aa03f88f9 (127.18.211.2:34517)
W20260129 02:59:03.783885 20556 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.211.2:34517, user_credentials={real_user=slave}} blocked reactor thread for 61521.6us
I20260129 02:59:03.774629 21401 leader_election.cc:290] T fd8311b779af4e1fa1402b2a02a602f2 P 197b880fa39d432a9b472845876bfaf0 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3:37735), 7a9daf5d42a84615a6383cfec9a10608 (127.18.211.4:40439)
I20260129 02:59:03.732997 21424 raft_consensus.cc:515] T cb0b3f259ebc4310927338a8740e87b8 P 7a9daf5d42a84615a6383cfec9a10608 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } }
I20260129 02:59:03.779776 21404 leader_election.cc:290] T 1ce4ddf5b10e4ae8b193b13b2e5936ee P 1c50de89a5a845668696dcb28c3badd6 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 197b880fa39d432a9b472845876bfaf0 (127.18.211.1:42027), 974b802141bd4f13beffef2aa03f88f9 (127.18.211.2:34517)
I20260129 02:59:03.775030 21410 leader_election.cc:290] T 7a9a69cd82724b1faed1d10aa091fc4a P 974b802141bd4f13beffef2aa03f88f9 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 197b880fa39d432a9b472845876bfaf0 (127.18.211.1:42027), 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3:37735)
I20260129 02:59:03.788117 21428 raft_consensus.cc:493] T febffe4a8b394368b69b1d164308fb9a P 7a9daf5d42a84615a6383cfec9a10608 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 197b880fa39d432a9b472845876bfaf0)
I20260129 02:59:03.785274 21426 raft_consensus.cc:493] T c2633a6fc34c44579de348d208b0c01e P 1c50de89a5a845668696dcb28c3badd6 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 197b880fa39d432a9b472845876bfaf0)
I20260129 02:59:03.798746 21430 raft_consensus.cc:493] T 4f5a77e94fef4b16a2ba7b067c8b144a P 7a9daf5d42a84615a6383cfec9a10608 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 974b802141bd4f13beffef2aa03f88f9)
I20260129 02:59:03.792385 21405 raft_consensus.cc:493] T a1a39dcd328e4268ae41b988c66b4a75 P 1c50de89a5a845668696dcb28c3badd6 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 197b880fa39d432a9b472845876bfaf0)
W20260129 02:59:03.798907 20369 debug-util.cc:398] Leaking SignalData structure 0x7b0800034520 after lost signal to thread 19279
I20260129 02:59:03.802877 21432 raft_consensus.cc:493] T 0340229d521b432580ae926f6062e3f7 P 1c50de89a5a845668696dcb28c3badd6 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 197b880fa39d432a9b472845876bfaf0)
I20260129 02:59:03.802752 21405 raft_consensus.cc:515] T a1a39dcd328e4268ae41b988c66b4a75 P 1c50de89a5a845668696dcb28c3badd6 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } } peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } }
I20260129 02:59:03.803226 21414 leader_election.cc:290] T 0f20af7fbe5d4db89a6f17948d98912c P 1c50de89a5a845668696dcb28c3badd6 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 197b880fa39d432a9b472845876bfaf0 (127.18.211.1:42027), 974b802141bd4f13beffef2aa03f88f9 (127.18.211.2:34517)
I20260129 02:59:03.796216 21418 leader_election.cc:290] T c891d2bff4ce4568861bbc992e91fbb0 P 1c50de89a5a845668696dcb28c3badd6 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 197b880fa39d432a9b472845876bfaf0 (127.18.211.1:42027), 974b802141bd4f13beffef2aa03f88f9 (127.18.211.2:34517)
I20260129 02:59:03.799126 20508 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "87d5fe48a44a408aaf03e894a67ecb2f" candidate_uuid: "197b880fa39d432a9b472845876bfaf0" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "974b802141bd4f13beffef2aa03f88f9" is_pre_election: true
W20260129 02:59:03.807176 20479 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.211.1:42027, user_credentials={real_user=slave}} blocked reactor thread for 251234us
I20260129 02:59:03.809677 21405 leader_election.cc:290] T a1a39dcd328e4268ae41b988c66b4a75 P 1c50de89a5a845668696dcb28c3badd6 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 974b802141bd4f13beffef2aa03f88f9 (127.18.211.2:34517), 197b880fa39d432a9b472845876bfaf0 (127.18.211.1:42027)
I20260129 02:59:03.811796 21409 leader_election.cc:290] T 926a250fa5834e88b96ccecbefbf5086 P 974b802141bd4f13beffef2aa03f88f9 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 197b880fa39d432a9b472845876bfaf0 (127.18.211.1:42027), 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3:37735)
I20260129 02:59:03.809707 21417 leader_election.cc:290] T add8d4923d1a489e86882c3a25692b95 P 197b880fa39d432a9b472845876bfaf0 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 974b802141bd4f13beffef2aa03f88f9 (127.18.211.2:34517), 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3:37735)
I20260129 02:59:03.807693 21420 leader_election.cc:290] T 3906b080a5b34a74b8b64128635e8997 P 1c50de89a5a845668696dcb28c3badd6 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 197b880fa39d432a9b472845876bfaf0 (127.18.211.1:42027), 974b802141bd4f13beffef2aa03f88f9 (127.18.211.2:34517)
W20260129 02:59:03.806439 20369 debug-util.cc:398] Leaking SignalData structure 0x7b0800034bc0 after lost signal to thread 20372
I20260129 02:59:03.800261 21426 raft_consensus.cc:515] T c2633a6fc34c44579de348d208b0c01e P 1c50de89a5a845668696dcb28c3badd6 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:59:03.812340 20513 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "dc2f4cb752cf4d59b5f6bc689f5d8104" candidate_uuid: "197b880fa39d432a9b472845876bfaf0" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "974b802141bd4f13beffef2aa03f88f9" is_pre_election: true
I20260129 02:59:03.803534 21432 raft_consensus.cc:515] T 0340229d521b432580ae926f6062e3f7 P 1c50de89a5a845668696dcb28c3badd6 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:59:03.808704 21418 raft_consensus.cc:493] T 75be4bd6bd45474795fd472eff2953ac P 1c50de89a5a845668696dcb28c3badd6 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 197b880fa39d432a9b472845876bfaf0)
I20260129 02:59:03.822863 20582 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "75be4bd6bd45474795fd472eff2953ac" candidate_uuid: "974b802141bd4f13beffef2aa03f88f9" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1c50de89a5a845668696dcb28c3badd6" is_pre_election: true
I20260129 02:59:03.823271 20432 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c934d31efd3f4e5195605ddc354d14e1" candidate_uuid: "974b802141bd4f13beffef2aa03f88f9" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "197b880fa39d432a9b472845876bfaf0" is_pre_election: true
I20260129 02:59:03.823138 21418 raft_consensus.cc:515] T 75be4bd6bd45474795fd472eff2953ac P 1c50de89a5a845668696dcb28c3badd6 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:59:03.825192 21418 leader_election.cc:290] T 75be4bd6bd45474795fd472eff2953ac P 1c50de89a5a845668696dcb28c3badd6 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 197b880fa39d432a9b472845876bfaf0 (127.18.211.1:42027), 974b802141bd4f13beffef2aa03f88f9 (127.18.211.2:34517)
I20260129 02:59:03.810592 21423 leader_election.cc:290] T c891d2bff4ce4568861bbc992e91fbb0 P 974b802141bd4f13beffef2aa03f88f9 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 197b880fa39d432a9b472845876bfaf0 (127.18.211.1:42027), 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3:37735)
W20260129 02:59:03.828274 20369 debug-util.cc:398] Leaking SignalData structure 0x7b08002a3a60 after lost signal to thread 20465
I20260129 02:59:03.829444 21431 raft_consensus.cc:493] T 3e2d15f8640b407195210c476e09df91 P 7a9daf5d42a84615a6383cfec9a10608 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 1c50de89a5a845668696dcb28c3badd6)
I20260129 02:59:03.792316 21428 raft_consensus.cc:515] T febffe4a8b394368b69b1d164308fb9a P 7a9daf5d42a84615a6383cfec9a10608 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } }
I20260129 02:59:03.825289 20432 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c2633a6fc34c44579de348d208b0c01e" candidate_uuid: "974b802141bd4f13beffef2aa03f88f9" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "197b880fa39d432a9b472845876bfaf0" is_pre_election: true
I20260129 02:59:03.825965 20436 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3ccac82a01a2445fa5b0ce6c0fcf2295" candidate_uuid: "974b802141bd4f13beffef2aa03f88f9" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "197b880fa39d432a9b472845876bfaf0" is_pre_election: true
I20260129 02:59:03.804311 21430 raft_consensus.cc:515] T 4f5a77e94fef4b16a2ba7b067c8b144a P 7a9daf5d42a84615a6383cfec9a10608 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } }
I20260129 02:59:03.826682 20439 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00e79d8f25a34676b1d866eb2068a2b2" candidate_uuid: "974b802141bd4f13beffef2aa03f88f9" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "197b880fa39d432a9b472845876bfaf0" is_pre_election: true
I20260129 02:59:03.825542 20582 raft_consensus.cc:2468] T 75be4bd6bd45474795fd472eff2953ac P 1c50de89a5a845668696dcb28c3badd6 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 974b802141bd4f13beffef2aa03f88f9 in term 1.
I20260129 02:59:03.830024 21431 raft_consensus.cc:515] T 3e2d15f8640b407195210c476e09df91 P 7a9daf5d42a84615a6383cfec9a10608 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } }
I20260129 02:59:03.833320 20590 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0f20af7fbe5d4db89a6f17948d98912c" candidate_uuid: "974b802141bd4f13beffef2aa03f88f9" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1c50de89a5a845668696dcb28c3badd6" is_pre_election: true
I20260129 02:59:03.834091 20441 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e7d83d4ccf684e76acb29a0dc844e1b8" candidate_uuid: "974b802141bd4f13beffef2aa03f88f9" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "197b880fa39d432a9b472845876bfaf0" is_pre_election: true
I20260129 02:59:03.811816 21424 leader_election.cc:290] T cb0b3f259ebc4310927338a8740e87b8 P 7a9daf5d42a84615a6383cfec9a10608 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 974b802141bd4f13beffef2aa03f88f9 (127.18.211.2:34517), 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3:37735)
I20260129 02:59:03.839031 20507 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "45cc5f3d2688486d9b6dbe7e666dedde" candidate_uuid: "197b880fa39d432a9b472845876bfaf0" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "974b802141bd4f13beffef2aa03f88f9" is_pre_election: true
I20260129 02:59:03.839347 20432 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7a9a69cd82724b1faed1d10aa091fc4a" candidate_uuid: "974b802141bd4f13beffef2aa03f88f9" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "197b880fa39d432a9b472845876bfaf0" is_pre_election: true
I20260129 02:59:03.840268 20590 raft_consensus.cc:2468] T 0f20af7fbe5d4db89a6f17948d98912c P 1c50de89a5a845668696dcb28c3badd6 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 974b802141bd4f13beffef2aa03f88f9 in term 1.
I20260129 02:59:03.846477 20441 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c891d2bff4ce4568861bbc992e91fbb0" candidate_uuid: "974b802141bd4f13beffef2aa03f88f9" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "197b880fa39d432a9b472845876bfaf0" is_pre_election: true
I20260129 02:59:03.850049 21430 leader_election.cc:290] T 4f5a77e94fef4b16a2ba7b067c8b144a P 7a9daf5d42a84615a6383cfec9a10608 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 974b802141bd4f13beffef2aa03f88f9 (127.18.211.2:34517), 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3:37735)
I20260129 02:59:03.850322 20436 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "926a250fa5834e88b96ccecbefbf5086" candidate_uuid: "974b802141bd4f13beffef2aa03f88f9" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "197b880fa39d432a9b472845876bfaf0" is_pre_election: true
I20260129 02:59:03.851308 20513 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "724a236524cb4e7ba3428880681d6cc2" candidate_uuid: "197b880fa39d432a9b472845876bfaf0" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "974b802141bd4f13beffef2aa03f88f9" is_pre_election: true
I20260129 02:59:03.851481 21428 leader_election.cc:290] T febffe4a8b394368b69b1d164308fb9a P 7a9daf5d42a84615a6383cfec9a10608 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 197b880fa39d432a9b472845876bfaf0 (127.18.211.1:42027), 974b802141bd4f13beffef2aa03f88f9 (127.18.211.2:34517)
I20260129 02:59:03.835994 20433 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "febffe4a8b394368b69b1d164308fb9a" candidate_uuid: "974b802141bd4f13beffef2aa03f88f9" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "197b880fa39d432a9b472845876bfaf0" is_pre_election: true
I20260129 02:59:03.860268 20590 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7a9a69cd82724b1faed1d10aa091fc4a" candidate_uuid: "974b802141bd4f13beffef2aa03f88f9" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1c50de89a5a845668696dcb28c3badd6" is_pre_election: true
I20260129 02:59:03.860939 20590 raft_consensus.cc:2468] T 7a9a69cd82724b1faed1d10aa091fc4a P 1c50de89a5a845668696dcb28c3badd6 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 974b802141bd4f13beffef2aa03f88f9 in term 1.
I20260129 02:59:03.863190 20584 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4f5a77e94fef4b16a2ba7b067c8b144a" candidate_uuid: "7a9daf5d42a84615a6383cfec9a10608" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "1c50de89a5a845668696dcb28c3badd6" is_pre_election: true
I20260129 02:59:03.864511 20584 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c891d2bff4ce4568861bbc992e91fbb0" candidate_uuid: "974b802141bd4f13beffef2aa03f88f9" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1c50de89a5a845668696dcb28c3badd6" is_pre_election: true
I20260129 02:59:03.865051 20584 raft_consensus.cc:2468] T c891d2bff4ce4568861bbc992e91fbb0 P 1c50de89a5a845668696dcb28c3badd6 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 974b802141bd4f13beffef2aa03f88f9 in term 1.
I20260129 02:59:03.865772 20584 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e7d83d4ccf684e76acb29a0dc844e1b8" candidate_uuid: "974b802141bd4f13beffef2aa03f88f9" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1c50de89a5a845668696dcb28c3badd6" is_pre_election: true
I20260129 02:59:03.866250 20584 raft_consensus.cc:2468] T e7d83d4ccf684e76acb29a0dc844e1b8 P 1c50de89a5a845668696dcb28c3badd6 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 974b802141bd4f13beffef2aa03f88f9 in term 1.
I20260129 02:59:03.866952 20584 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "926a250fa5834e88b96ccecbefbf5086" candidate_uuid: "974b802141bd4f13beffef2aa03f88f9" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1c50de89a5a845668696dcb28c3badd6" is_pre_election: true
I20260129 02:59:03.867465 20584 raft_consensus.cc:2468] T 926a250fa5834e88b96ccecbefbf5086 P 1c50de89a5a845668696dcb28c3badd6 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 974b802141bd4f13beffef2aa03f88f9 in term 1.
W20260129 02:59:03.856956 20369 debug-util.cc:398] Leaking SignalData structure 0x7b0800323620 after lost signal to thread 20540
I20260129 02:59:03.861775 20590 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "cb0b3f259ebc4310927338a8740e87b8" candidate_uuid: "7a9daf5d42a84615a6383cfec9a10608" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "1c50de89a5a845668696dcb28c3badd6" is_pre_election: true
I20260129 02:59:03.869040 21428 raft_consensus.cc:493] T 87d5fe48a44a408aaf03e894a67ecb2f P 7a9daf5d42a84615a6383cfec9a10608 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 974b802141bd4f13beffef2aa03f88f9)
I20260129 02:59:03.869695 20588 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c2633a6fc34c44579de348d208b0c01e" candidate_uuid: "974b802141bd4f13beffef2aa03f88f9" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1c50de89a5a845668696dcb28c3badd6" is_pre_election: true
I20260129 02:59:03.869771 21428 raft_consensus.cc:515] T 87d5fe48a44a408aaf03e894a67ecb2f P 7a9daf5d42a84615a6383cfec9a10608 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } }
I20260129 02:59:03.874866 21385 leader_election.cc:290] T a4c1226b987b468fa84dfc6ba0e2dbe1 P 197b880fa39d432a9b472845876bfaf0 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 974b802141bd4f13beffef2aa03f88f9 (127.18.211.2:34517), 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3:37735)
I20260129 02:59:03.875222 20586 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e4d18c35b0a7493e85d0a4b92956a48f" candidate_uuid: "974b802141bd4f13beffef2aa03f88f9" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1c50de89a5a845668696dcb28c3badd6" is_pre_election: true
I20260129 02:59:03.848192 21424 raft_consensus.cc:493] T a609ac02fae84725ab743f95f51d33ce P 7a9daf5d42a84615a6383cfec9a10608 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 197b880fa39d432a9b472845876bfaf0)
I20260129 02:59:03.878135 20510 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "cb4727386a3e48aca5d71cc6c611c197" candidate_uuid: "197b880fa39d432a9b472845876bfaf0" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "974b802141bd4f13beffef2aa03f88f9" is_pre_election: true
I20260129 02:59:03.877980 20581 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3ccac82a01a2445fa5b0ce6c0fcf2295" candidate_uuid: "974b802141bd4f13beffef2aa03f88f9" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1c50de89a5a845668696dcb28c3badd6" is_pre_election: true
I20260129 02:59:03.883100 20507 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e33a417cf9c54cf5af73a4ee3aac4d3a" candidate_uuid: "197b880fa39d432a9b472845876bfaf0" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "974b802141bd4f13beffef2aa03f88f9" is_pre_election: true
I20260129 02:59:03.885547 21426 leader_election.cc:290] T c2633a6fc34c44579de348d208b0c01e P 1c50de89a5a845668696dcb28c3badd6 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 197b880fa39d432a9b472845876bfaf0 (127.18.211.1:42027), 974b802141bd4f13beffef2aa03f88f9 (127.18.211.2:34517)
I20260129 02:59:03.894961 20438 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "febffe4a8b394368b69b1d164308fb9a" candidate_uuid: "7a9daf5d42a84615a6383cfec9a10608" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "197b880fa39d432a9b472845876bfaf0" is_pre_election: true
I20260129 02:59:03.895226 21431 leader_election.cc:290] T 3e2d15f8640b407195210c476e09df91 P 7a9daf5d42a84615a6383cfec9a10608 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 974b802141bd4f13beffef2aa03f88f9 (127.18.211.2:34517), 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3:37735)
I20260129 02:59:03.896649 21429 raft_consensus.cc:493] T a8479d1fe0bd41e5946186e3a6ca8c7a P 1c50de89a5a845668696dcb28c3badd6 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 197b880fa39d432a9b472845876bfaf0)
I20260129 02:59:03.895645 21424 raft_consensus.cc:515] T a609ac02fae84725ab743f95f51d33ce P 7a9daf5d42a84615a6383cfec9a10608 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } }
I20260129 02:59:03.904174 20589 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c934d31efd3f4e5195605ddc354d14e1" candidate_uuid: "974b802141bd4f13beffef2aa03f88f9" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1c50de89a5a845668696dcb28c3badd6" is_pre_election: true
I20260129 02:59:03.905223 20585 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3e2d15f8640b407195210c476e09df91" candidate_uuid: "7a9daf5d42a84615a6383cfec9a10608" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "1c50de89a5a845668696dcb28c3badd6" is_pre_election: true
I20260129 02:59:03.907599 21424 leader_election.cc:290] T a609ac02fae84725ab743f95f51d33ce P 7a9daf5d42a84615a6383cfec9a10608 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 197b880fa39d432a9b472845876bfaf0 (127.18.211.1:42027), 974b802141bd4f13beffef2aa03f88f9 (127.18.211.2:34517)
W20260129 02:59:03.909430 20369 debug-util.cc:398] Leaking SignalData structure 0x7b080025c020 after lost signal to thread 20614
I20260129 02:59:03.909896 21428 leader_election.cc:290] T 87d5fe48a44a408aaf03e894a67ecb2f P 7a9daf5d42a84615a6383cfec9a10608 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 197b880fa39d432a9b472845876bfaf0 (127.18.211.1:42027), 974b802141bd4f13beffef2aa03f88f9 (127.18.211.2:34517)
I20260129 02:59:03.911212 20516 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1299723a4476420ab02fd8bfd7aa0645" candidate_uuid: "197b880fa39d432a9b472845876bfaf0" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "974b802141bd4f13beffef2aa03f88f9" is_pre_election: true
I20260129 02:59:03.911383 20437 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "87d5fe48a44a408aaf03e894a67ecb2f" candidate_uuid: "7a9daf5d42a84615a6383cfec9a10608" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "197b880fa39d432a9b472845876bfaf0" is_pre_election: true
I20260129 02:59:03.897742 21432 leader_election.cc:290] T 0340229d521b432580ae926f6062e3f7 P 1c50de89a5a845668696dcb28c3badd6 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 197b880fa39d432a9b472845876bfaf0 (127.18.211.1:42027), 974b802141bd4f13beffef2aa03f88f9 (127.18.211.2:34517)
I20260129 02:59:03.911249 20434 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a609ac02fae84725ab743f95f51d33ce" candidate_uuid: "7a9daf5d42a84615a6383cfec9a10608" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "197b880fa39d432a9b472845876bfaf0" is_pre_election: true
I20260129 02:59:03.897226 21429 raft_consensus.cc:515] T a8479d1fe0bd41e5946186e3a6ca8c7a P 1c50de89a5a845668696dcb28c3badd6 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:59:03.896425 20480 leader_election.cc:304] T 0f20af7fbe5d4db89a6f17948d98912c P 974b802141bd4f13beffef2aa03f88f9 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1c50de89a5a845668696dcb28c3badd6, 974b802141bd4f13beffef2aa03f88f9; no voters: 197b880fa39d432a9b472845876bfaf0
W20260129 02:59:03.913892 20553 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.211.1:42027, user_credentials={real_user=slave}} blocked reactor thread for 141565us
I20260129 02:59:03.919133 21411 raft_consensus.cc:2804] T 0f20af7fbe5d4db89a6f17948d98912c P 974b802141bd4f13beffef2aa03f88f9 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260129 02:59:03.919554 21411 raft_consensus.cc:493] T 0f20af7fbe5d4db89a6f17948d98912c P 974b802141bd4f13beffef2aa03f88f9 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 197b880fa39d432a9b472845876bfaf0)
I20260129 02:59:03.920542 21411 raft_consensus.cc:3060] T 0f20af7fbe5d4db89a6f17948d98912c P 974b802141bd4f13beffef2aa03f88f9 [term 1 FOLLOWER]: Advancing to term 2
I20260129 02:59:03.923194 21134 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "cb0b3f259ebc4310927338a8740e87b8" candidate_uuid: "1c50de89a5a845668696dcb28c3badd6" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "7a9daf5d42a84615a6383cfec9a10608" is_pre_election: true
I20260129 02:59:03.923897 21134 raft_consensus.cc:2468] T cb0b3f259ebc4310927338a8740e87b8 P 7a9daf5d42a84615a6383cfec9a10608 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1c50de89a5a845668696dcb28c3badd6 in term 1.
I20260129 02:59:03.918268 21419 leader_election.cc:290] T 4e3805ffa70242e8ade66da260dddf66 P 1c50de89a5a845668696dcb28c3badd6 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 197b880fa39d432a9b472845876bfaf0 (127.18.211.1:42027), 974b802141bd4f13beffef2aa03f88f9 (127.18.211.2:34517)
I20260129 02:59:03.924901 20587 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "cb4727386a3e48aca5d71cc6c611c197" candidate_uuid: "197b880fa39d432a9b472845876bfaf0" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1c50de89a5a845668696dcb28c3badd6" is_pre_election: true
I20260129 02:59:03.925071 20589 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e33a417cf9c54cf5af73a4ee3aac4d3a" candidate_uuid: "197b880fa39d432a9b472845876bfaf0" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1c50de89a5a845668696dcb28c3badd6" is_pre_election: true
I20260129 02:59:03.927248 20582 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "dc2f4cb752cf4d59b5f6bc689f5d8104" candidate_uuid: "197b880fa39d432a9b472845876bfaf0" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1c50de89a5a845668696dcb28c3badd6" is_pre_election: true
W20260129 02:59:03.914784 20369 debug-util.cc:398] Leaking SignalData structure 0x7b080020a660 after lost signal to thread 21160
I20260129 02:59:03.925705 20585 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "724a236524cb4e7ba3428880681d6cc2" candidate_uuid: "197b880fa39d432a9b472845876bfaf0" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1c50de89a5a845668696dcb28c3badd6" is_pre_election: true
I20260129 02:59:03.922573 20480 leader_election.cc:304] T 7a9a69cd82724b1faed1d10aa091fc4a P 974b802141bd4f13beffef2aa03f88f9 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1c50de89a5a845668696dcb28c3badd6, 974b802141bd4f13beffef2aa03f88f9; no voters: 197b880fa39d432a9b472845876bfaf0
I20260129 02:59:03.926784 20581 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1299723a4476420ab02fd8bfd7aa0645" candidate_uuid: "197b880fa39d432a9b472845876bfaf0" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1c50de89a5a845668696dcb28c3badd6" is_pre_election: true
I20260129 02:59:03.926875 20583 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "45cc5f3d2688486d9b6dbe7e666dedde" candidate_uuid: "197b880fa39d432a9b472845876bfaf0" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1c50de89a5a845668696dcb28c3badd6" is_pre_election: true
I20260129 02:59:03.928875 20511 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4408006eb5174de1896b037cf3f18457" candidate_uuid: "197b880fa39d432a9b472845876bfaf0" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "974b802141bd4f13beffef2aa03f88f9" is_pre_election: true
I20260129 02:59:03.934049 21411 raft_consensus.cc:515] T 0f20af7fbe5d4db89a6f17948d98912c P 974b802141bd4f13beffef2aa03f88f9 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:59:03.936514 20511 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2ac659d732314e1288849f08fe5e074e" candidate_uuid: "197b880fa39d432a9b472845876bfaf0" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "974b802141bd4f13beffef2aa03f88f9" is_pre_election: true
I20260129 02:59:03.927418 21429 leader_election.cc:290] T a8479d1fe0bd41e5946186e3a6ca8c7a P 1c50de89a5a845668696dcb28c3badd6 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 197b880fa39d432a9b472845876bfaf0 (127.18.211.1:42027), 974b802141bd4f13beffef2aa03f88f9 (127.18.211.2:34517)
I20260129 02:59:03.937721 20514 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "368953c9102643b2a21f99af36b51905" candidate_uuid: "197b880fa39d432a9b472845876bfaf0" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "974b802141bd4f13beffef2aa03f88f9" is_pre_election: true
I20260129 02:59:03.937852 20480 leader_election.cc:304] T c891d2bff4ce4568861bbc992e91fbb0 P 974b802141bd4f13beffef2aa03f88f9 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1c50de89a5a845668696dcb28c3badd6, 974b802141bd4f13beffef2aa03f88f9; no voters: 197b880fa39d432a9b472845876bfaf0
I20260129 02:59:03.938181 20589 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4408006eb5174de1896b037cf3f18457" candidate_uuid: "197b880fa39d432a9b472845876bfaf0" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1c50de89a5a845668696dcb28c3badd6" is_pre_election: true
I20260129 02:59:03.936826 20516 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4f8f3a38f6e24ba4857875181f447442" candidate_uuid: "197b880fa39d432a9b472845876bfaf0" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "974b802141bd4f13beffef2aa03f88f9" is_pre_election: true
I20260129 02:59:03.939320 21409 raft_consensus.cc:2804] T 7a9a69cd82724b1faed1d10aa091fc4a P 974b802141bd4f13beffef2aa03f88f9 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260129 02:59:03.939628 20587 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2ac659d732314e1288849f08fe5e074e" candidate_uuid: "197b880fa39d432a9b472845876bfaf0" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1c50de89a5a845668696dcb28c3badd6" is_pre_election: true
I20260129 02:59:03.940307 20434 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1299723a4476420ab02fd8bfd7aa0645" candidate_uuid: "1c50de89a5a845668696dcb28c3badd6" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "197b880fa39d432a9b472845876bfaf0" is_pre_election: true
I20260129 02:59:03.940927 20480 leader_election.cc:304] T e7d83d4ccf684e76acb29a0dc844e1b8 P 974b802141bd4f13beffef2aa03f88f9 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1c50de89a5a845668696dcb28c3badd6, 974b802141bd4f13beffef2aa03f88f9; no voters: 197b880fa39d432a9b472845876bfaf0
I20260129 02:59:03.940351 21398 raft_consensus.cc:2804] T c891d2bff4ce4568861bbc992e91fbb0 P 974b802141bd4f13beffef2aa03f88f9 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260129 02:59:03.941607 21398 raft_consensus.cc:493] T c891d2bff4ce4568861bbc992e91fbb0 P 974b802141bd4f13beffef2aa03f88f9 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 197b880fa39d432a9b472845876bfaf0)
I20260129 02:59:03.942035 21433 raft_consensus.cc:2804] T e7d83d4ccf684e76acb29a0dc844e1b8 P 974b802141bd4f13beffef2aa03f88f9 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260129 02:59:03.942080 20480 leader_election.cc:304] T c2633a6fc34c44579de348d208b0c01e P 974b802141bd4f13beffef2aa03f88f9 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 974b802141bd4f13beffef2aa03f88f9; no voters: 197b880fa39d432a9b472845876bfaf0, 1c50de89a5a845668696dcb28c3badd6
I20260129 02:59:03.936400 21411 leader_election.cc:290] T 0f20af7fbe5d4db89a6f17948d98912c P 974b802141bd4f13beffef2aa03f88f9 [CANDIDATE]: Term 2 election: Requested vote from peers 197b880fa39d432a9b472845876bfaf0 (127.18.211.1:42027), 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3:37735)
I20260129 02:59:03.939744 21409 raft_consensus.cc:493] T 7a9a69cd82724b1faed1d10aa091fc4a P 974b802141bd4f13beffef2aa03f88f9 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 197b880fa39d432a9b472845876bfaf0)
I20260129 02:59:03.942054 21398 raft_consensus.cc:3060] T c891d2bff4ce4568861bbc992e91fbb0 P 974b802141bd4f13beffef2aa03f88f9 [term 1 FOLLOWER]: Advancing to term 2
I20260129 02:59:03.942698 21433 raft_consensus.cc:493] T e7d83d4ccf684e76acb29a0dc844e1b8 P 974b802141bd4f13beffef2aa03f88f9 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 197b880fa39d432a9b472845876bfaf0)
I20260129 02:59:03.943046 21409 raft_consensus.cc:3060] T 7a9a69cd82724b1faed1d10aa091fc4a P 974b802141bd4f13beffef2aa03f88f9 [term 1 FOLLOWER]: Advancing to term 2
I20260129 02:59:03.943909 21397 raft_consensus.cc:2749] T c2633a6fc34c44579de348d208b0c01e P 974b802141bd4f13beffef2aa03f88f9 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260129 02:59:03.944641 21136 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a609ac02fae84725ab743f95f51d33ce" candidate_uuid: "974b802141bd4f13beffef2aa03f88f9" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "7a9daf5d42a84615a6383cfec9a10608" is_pre_election: true
I20260129 02:59:03.945116 20437 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fcddab6bca684adf8ea032289a00f90e" candidate_uuid: "1c50de89a5a845668696dcb28c3badd6" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "197b880fa39d432a9b472845876bfaf0" is_pre_election: true
I20260129 02:59:03.945758 20587 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fd8311b779af4e1fa1402b2a02a602f2" candidate_uuid: "197b880fa39d432a9b472845876bfaf0" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "1c50de89a5a845668696dcb28c3badd6" is_pre_election: true
I20260129 02:59:03.945933 20434 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0f20af7fbe5d4db89a6f17948d98912c" candidate_uuid: "974b802141bd4f13beffef2aa03f88f9" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "197b880fa39d432a9b472845876bfaf0"
I20260129 02:59:03.943446 21433 raft_consensus.cc:3060] T e7d83d4ccf684e76acb29a0dc844e1b8 P 974b802141bd4f13beffef2aa03f88f9 [term 1 FOLLOWER]: Advancing to term 2
I20260129 02:59:03.945456 21136 raft_consensus.cc:2468] T a609ac02fae84725ab743f95f51d33ce P 7a9daf5d42a84615a6383cfec9a10608 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 974b802141bd4f13beffef2aa03f88f9 in term 1.
I20260129 02:59:03.978890 20437 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1ce4ddf5b10e4ae8b193b13b2e5936ee" candidate_uuid: "1c50de89a5a845668696dcb28c3badd6" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "197b880fa39d432a9b472845876bfaf0" is_pre_election: true
I20260129 02:59:03.947326 20480 leader_election.cc:304] T 926a250fa5834e88b96ccecbefbf5086 P 974b802141bd4f13beffef2aa03f88f9 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1c50de89a5a845668696dcb28c3badd6, 974b802141bd4f13beffef2aa03f88f9; no voters: 197b880fa39d432a9b472845876bfaf0
I20260129 02:59:03.972350 20438 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2ac659d732314e1288849f08fe5e074e" candidate_uuid: "1c50de89a5a845668696dcb28c3badd6" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "197b880fa39d432a9b472845876bfaf0" is_pre_election: true
I20260129 02:59:03.948443 20435 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0678b8b468bd43cdbc3e485900844821" candidate_uuid: "1c50de89a5a845668696dcb28c3badd6" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "197b880fa39d432a9b472845876bfaf0" is_pre_election: true
I20260129 02:59:03.984175 21134 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00e79d8f25a34676b1d866eb2068a2b2" candidate_uuid: "974b802141bd4f13beffef2aa03f88f9" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "7a9daf5d42a84615a6383cfec9a10608" is_pre_election: true
I20260129 02:59:03.989593 20437 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c891d2bff4ce4568861bbc992e91fbb0" candidate_uuid: "1c50de89a5a845668696dcb28c3badd6" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "197b880fa39d432a9b472845876bfaf0" is_pre_election: true
I20260129 02:59:03.990116 20587 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "368953c9102643b2a21f99af36b51905" candidate_uuid: "197b880fa39d432a9b472845876bfaf0" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1c50de89a5a845668696dcb28c3badd6" is_pre_election: true
I20260129 02:59:03.948160 20516 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "add8d4923d1a489e86882c3a25692b95" candidate_uuid: "197b880fa39d432a9b472845876bfaf0" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "974b802141bd4f13beffef2aa03f88f9" is_pre_election: true
I20260129 02:59:03.986881 20480 leader_election.cc:304] T e4d18c35b0a7493e85d0a4b92956a48f P 974b802141bd4f13beffef2aa03f88f9 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 974b802141bd4f13beffef2aa03f88f9; no voters: 197b880fa39d432a9b472845876bfaf0, 1c50de89a5a845668696dcb28c3badd6
I20260129 02:59:03.991236 20514 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a4c1226b987b468fa84dfc6ba0e2dbe1" candidate_uuid: "197b880fa39d432a9b472845876bfaf0" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "974b802141bd4f13beffef2aa03f88f9" is_pre_election: true
I20260129 02:59:03.991385 21135 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "febffe4a8b394368b69b1d164308fb9a" candidate_uuid: "974b802141bd4f13beffef2aa03f88f9" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "7a9daf5d42a84615a6383cfec9a10608" is_pre_election: true
I20260129 02:59:03.992553 20583 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "add8d4923d1a489e86882c3a25692b95" candidate_uuid: "197b880fa39d432a9b472845876bfaf0" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1c50de89a5a845668696dcb28c3badd6" is_pre_election: true
I20260129 02:59:03.986742 21397 raft_consensus.cc:2804] T 926a250fa5834e88b96ccecbefbf5086 P 974b802141bd4f13beffef2aa03f88f9 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260129 02:59:03.995373 21397 raft_consensus.cc:493] T 926a250fa5834e88b96ccecbefbf5086 P 974b802141bd4f13beffef2aa03f88f9 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 197b880fa39d432a9b472845876bfaf0)
I20260129 02:59:03.995761 21397 raft_consensus.cc:3060] T 926a250fa5834e88b96ccecbefbf5086 P 974b802141bd4f13beffef2aa03f88f9 [term 1 FOLLOWER]: Advancing to term 2
I20260129 02:59:03.945477 20589 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4f8f3a38f6e24ba4857875181f447442" candidate_uuid: "197b880fa39d432a9b472845876bfaf0" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1c50de89a5a845668696dcb28c3badd6" is_pre_election: true
I20260129 02:59:03.998288 20480 leader_election.cc:304] T 3ccac82a01a2445fa5b0ce6c0fcf2295 P 974b802141bd4f13beffef2aa03f88f9 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 974b802141bd4f13beffef2aa03f88f9; no voters: 197b880fa39d432a9b472845876bfaf0, 1c50de89a5a845668696dcb28c3badd6
I20260129 02:59:03.998622 20510 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "cedce331e1584ac8b636c9b8a72b2ba0" candidate_uuid: "1c50de89a5a845668696dcb28c3badd6" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "974b802141bd4f13beffef2aa03f88f9" is_pre_election: true
I20260129 02:59:03.978612 21435 raft_consensus.cc:493] T 4f8f3a38f6e24ba4857875181f447442 P 1c50de89a5a845668696dcb28c3badd6 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 974b802141bd4f13beffef2aa03f88f9)
I20260129 02:59:03.997727 21411 raft_consensus.cc:2749] T e4d18c35b0a7493e85d0a4b92956a48f P 974b802141bd4f13beffef2aa03f88f9 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260129 02:59:04.006806 20438 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3ccac82a01a2445fa5b0ce6c0fcf2295" candidate_uuid: "1c50de89a5a845668696dcb28c3badd6" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "197b880fa39d432a9b472845876bfaf0" is_pre_election: true
I20260129 02:59:04.007385 20435 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3906b080a5b34a74b8b64128635e8997" candidate_uuid: "1c50de89a5a845668696dcb28c3badd6" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "197b880fa39d432a9b472845876bfaf0" is_pre_election: true
I20260129 02:59:04.007520 20514 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4e420f90884f4796997b5ea41e49e411" candidate_uuid: "1c50de89a5a845668696dcb28c3badd6" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "974b802141bd4f13beffef2aa03f88f9" is_pre_election: true
I20260129 02:59:04.005720 21130 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fd8311b779af4e1fa1402b2a02a602f2" candidate_uuid: "197b880fa39d432a9b472845876bfaf0" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "7a9daf5d42a84615a6383cfec9a10608" is_pre_election: true
I20260129 02:59:04.006275 20516 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c934d31efd3f4e5195605ddc354d14e1" candidate_uuid: "1c50de89a5a845668696dcb28c3badd6" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "974b802141bd4f13beffef2aa03f88f9" is_pre_election: true
I20260129 02:59:04.008877 20511 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a5a05fbcbbcf4916bbc01fd1a5b5e90d" candidate_uuid: "1c50de89a5a845668696dcb28c3badd6" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "974b802141bd4f13beffef2aa03f88f9" is_pre_election: true
I20260129 02:59:04.010627 20507 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "cb0b3f259ebc4310927338a8740e87b8" candidate_uuid: "1c50de89a5a845668696dcb28c3badd6" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "974b802141bd4f13beffef2aa03f88f9" is_pre_election: true
I20260129 02:59:04.003648 21411 raft_consensus.cc:2749] T 3ccac82a01a2445fa5b0ce6c0fcf2295 P 974b802141bd4f13beffef2aa03f88f9 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260129 02:59:04.014292 20583 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a4c1226b987b468fa84dfc6ba0e2dbe1" candidate_uuid: "197b880fa39d432a9b472845876bfaf0" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1c50de89a5a845668696dcb28c3badd6" is_pre_election: true
I20260129 02:59:04.017059 20437 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0f20af7fbe5d4db89a6f17948d98912c" candidate_uuid: "1c50de89a5a845668696dcb28c3badd6" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "197b880fa39d432a9b472845876bfaf0" is_pre_election: true
I20260129 02:59:04.020331 21131 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "87d5fe48a44a408aaf03e894a67ecb2f" candidate_uuid: "197b880fa39d432a9b472845876bfaf0" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "7a9daf5d42a84615a6383cfec9a10608" is_pre_election: true
I20260129 02:59:04.022560 20480 leader_election.cc:304] T c934d31efd3f4e5195605ddc354d14e1 P 974b802141bd4f13beffef2aa03f88f9 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 974b802141bd4f13beffef2aa03f88f9; no voters: 197b880fa39d432a9b472845876bfaf0, 1c50de89a5a845668696dcb28c3badd6
I20260129 02:59:04.001176 21435 raft_consensus.cc:515] T 4f8f3a38f6e24ba4857875181f447442 P 1c50de89a5a845668696dcb28c3badd6 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:59:04.043035 20507 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7a9a69cd82724b1faed1d10aa091fc4a" candidate_uuid: "1c50de89a5a845668696dcb28c3badd6" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "974b802141bd4f13beffef2aa03f88f9" is_pre_election: true
I20260129 02:59:04.030552 20587 consensus_queue.cc:237] T 78aa39096868415d9afc2ffcb248e198 P 1c50de89a5a845668696dcb28c3badd6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } attrs { replace: true } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: NON_VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: true } }
I20260129 02:59:04.031756 20438 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a1a39dcd328e4268ae41b988c66b4a75" candidate_uuid: "1c50de89a5a845668696dcb28c3badd6" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "197b880fa39d432a9b472845876bfaf0" is_pre_election: true
I20260129 02:59:04.024044 20437 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "75be4bd6bd45474795fd472eff2953ac" candidate_uuid: "1c50de89a5a845668696dcb28c3badd6" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "197b880fa39d432a9b472845876bfaf0" is_pre_election: true
I20260129 02:59:04.038395 20554 leader_election.cc:304] T cb0b3f259ebc4310927338a8740e87b8 P 1c50de89a5a845668696dcb28c3badd6 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1c50de89a5a845668696dcb28c3badd6, 7a9daf5d42a84615a6383cfec9a10608; no voters:
I20260129 02:59:04.038235 20405 leader_election.cc:304] T 724a236524cb4e7ba3428880681d6cc2 P 197b880fa39d432a9b472845876bfaf0 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 197b880fa39d432a9b472845876bfaf0; no voters: 1c50de89a5a845668696dcb28c3badd6, 974b802141bd4f13beffef2aa03f88f9
I20260129 02:59:04.045992 20508 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fcddab6bca684adf8ea032289a00f90e" candidate_uuid: "1c50de89a5a845668696dcb28c3badd6" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "974b802141bd4f13beffef2aa03f88f9" is_pre_election: true
I20260129 02:59:04.046473 21419 raft_consensus.cc:2804] T cb0b3f259ebc4310927338a8740e87b8 P 1c50de89a5a845668696dcb28c3badd6 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260129 02:59:04.046875 21419 raft_consensus.cc:493] T cb0b3f259ebc4310927338a8740e87b8 P 1c50de89a5a845668696dcb28c3badd6 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 974b802141bd4f13beffef2aa03f88f9)
I20260129 02:59:04.047077 21399 raft_consensus.cc:2749] T 724a236524cb4e7ba3428880681d6cc2 P 197b880fa39d432a9b472845876bfaf0 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260129 02:59:04.047255 21419 raft_consensus.cc:3060] T cb0b3f259ebc4310927338a8740e87b8 P 1c50de89a5a845668696dcb28c3badd6 [term 1 FOLLOWER]: Advancing to term 2
I20260129 02:59:04.048902 20514 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "76f713c3331540d2a5b76164103e9eb1" candidate_uuid: "1c50de89a5a845668696dcb28c3badd6" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "974b802141bd4f13beffef2aa03f88f9" is_pre_election: true
I20260129 02:59:04.049098 20405 leader_election.cc:304] T dc2f4cb752cf4d59b5f6bc689f5d8104 P 197b880fa39d432a9b472845876bfaf0 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 197b880fa39d432a9b472845876bfaf0; no voters: 1c50de89a5a845668696dcb28c3badd6, 974b802141bd4f13beffef2aa03f88f9
I20260129 02:59:04.038535 21398 raft_consensus.cc:515] T c891d2bff4ce4568861bbc992e91fbb0 P 974b802141bd4f13beffef2aa03f88f9 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:59:04.041394 20510 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1299723a4476420ab02fd8bfd7aa0645" candidate_uuid: "1c50de89a5a845668696dcb28c3badd6" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "974b802141bd4f13beffef2aa03f88f9" is_pre_election: true
I20260129 02:59:04.041739 20513 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0678b8b468bd43cdbc3e485900844821" candidate_uuid: "1c50de89a5a845668696dcb28c3badd6" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "974b802141bd4f13beffef2aa03f88f9" is_pre_election: true
I20260129 02:59:04.042251 21409 raft_consensus.cc:515] T 7a9a69cd82724b1faed1d10aa091fc4a P 974b802141bd4f13beffef2aa03f88f9 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:59:04.051280 20437 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0340229d521b432580ae926f6062e3f7" candidate_uuid: "1c50de89a5a845668696dcb28c3badd6" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "197b880fa39d432a9b472845876bfaf0" is_pre_election: true
I20260129 02:59:04.051626 20480 leader_election.cc:304] T 75be4bd6bd45474795fd472eff2953ac P 974b802141bd4f13beffef2aa03f88f9 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1c50de89a5a845668696dcb28c3badd6, 974b802141bd4f13beffef2aa03f88f9; no voters: 197b880fa39d432a9b472845876bfaf0
I20260129 02:59:04.051668 20405 leader_election.cc:304] T 1299723a4476420ab02fd8bfd7aa0645 P 197b880fa39d432a9b472845876bfaf0 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 197b880fa39d432a9b472845876bfaf0; no voters: 1c50de89a5a845668696dcb28c3badd6, 974b802141bd4f13beffef2aa03f88f9
I20260129 02:59:04.050655 21394 raft_consensus.cc:2749] T dc2f4cb752cf4d59b5f6bc689f5d8104 P 197b880fa39d432a9b472845876bfaf0 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260129 02:59:04.054054 20435 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c2633a6fc34c44579de348d208b0c01e" candidate_uuid: "1c50de89a5a845668696dcb28c3badd6" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "197b880fa39d432a9b472845876bfaf0" is_pre_election: true
I20260129 02:59:04.054225 20510 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2ac659d732314e1288849f08fe5e074e" candidate_uuid: "1c50de89a5a845668696dcb28c3badd6" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "974b802141bd4f13beffef2aa03f88f9" is_pre_election: true
I20260129 02:59:04.054999 20434 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a8479d1fe0bd41e5946186e3a6ca8c7a" candidate_uuid: "1c50de89a5a845668696dcb28c3badd6" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "197b880fa39d432a9b472845876bfaf0" is_pre_election: true
I20260129 02:59:04.060189 20513 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3ccac82a01a2445fa5b0ce6c0fcf2295" candidate_uuid: "1c50de89a5a845668696dcb28c3badd6" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "974b802141bd4f13beffef2aa03f88f9" is_pre_election: true
I20260129 02:59:04.060832 21399 raft_consensus.cc:2749] T 1299723a4476420ab02fd8bfd7aa0645 P 197b880fa39d432a9b472845876bfaf0 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260129 02:59:04.061578 21374 raft_consensus.cc:2804] T 75be4bd6bd45474795fd472eff2953ac P 974b802141bd4f13beffef2aa03f88f9 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260129 02:59:04.061970 21374 raft_consensus.cc:493] T 75be4bd6bd45474795fd472eff2953ac P 974b802141bd4f13beffef2aa03f88f9 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 197b880fa39d432a9b472845876bfaf0)
I20260129 02:59:04.062323 21374 raft_consensus.cc:3060] T 75be4bd6bd45474795fd472eff2953ac P 974b802141bd4f13beffef2aa03f88f9 [term 1 FOLLOWER]: Advancing to term 2
I20260129 02:59:04.063390 20514 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0f20af7fbe5d4db89a6f17948d98912c" candidate_uuid: "1c50de89a5a845668696dcb28c3badd6" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "974b802141bd4f13beffef2aa03f88f9" is_pre_election: true
I20260129 02:59:04.063866 20510 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c891d2bff4ce4568861bbc992e91fbb0" candidate_uuid: "1c50de89a5a845668696dcb28c3badd6" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "974b802141bd4f13beffef2aa03f88f9" is_pre_election: true
I20260129 02:59:04.055081 20438 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4e3805ffa70242e8ade66da260dddf66" candidate_uuid: "1c50de89a5a845668696dcb28c3badd6" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "197b880fa39d432a9b472845876bfaf0" is_pre_election: true
I20260129 02:59:04.053539 21410 raft_consensus.cc:2749] T c934d31efd3f4e5195605ddc354d14e1 P 974b802141bd4f13beffef2aa03f88f9 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260129 02:59:04.065024 20513 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1ce4ddf5b10e4ae8b193b13b2e5936ee" candidate_uuid: "1c50de89a5a845668696dcb28c3badd6" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "974b802141bd4f13beffef2aa03f88f9" is_pre_election: true
I20260129 02:59:04.065408 20405 leader_election.cc:304] T 45cc5f3d2688486d9b6dbe7e666dedde P 197b880fa39d432a9b472845876bfaf0 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 197b880fa39d432a9b472845876bfaf0; no voters: 1c50de89a5a845668696dcb28c3badd6, 974b802141bd4f13beffef2aa03f88f9
I20260129 02:59:04.075567 21399 raft_consensus.cc:2749] T 45cc5f3d2688486d9b6dbe7e666dedde P 197b880fa39d432a9b472845876bfaf0 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260129 02:59:04.066130 20508 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3906b080a5b34a74b8b64128635e8997" candidate_uuid: "1c50de89a5a845668696dcb28c3badd6" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "974b802141bd4f13beffef2aa03f88f9" is_pre_election: true
I20260129 02:59:04.077411 20405 leader_election.cc:304] T cb4727386a3e48aca5d71cc6c611c197 P 197b880fa39d432a9b472845876bfaf0 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 197b880fa39d432a9b472845876bfaf0; no voters: 1c50de89a5a845668696dcb28c3badd6, 974b802141bd4f13beffef2aa03f88f9
I20260129 02:59:04.078507 21394 raft_consensus.cc:2749] T cb4727386a3e48aca5d71cc6c611c197 P 197b880fa39d432a9b472845876bfaf0 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260129 02:59:04.079334 20508 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a1a39dcd328e4268ae41b988c66b4a75" candidate_uuid: "1c50de89a5a845668696dcb28c3badd6" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "974b802141bd4f13beffef2aa03f88f9" is_pre_election: true
I20260129 02:59:04.079841 20510 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c2633a6fc34c44579de348d208b0c01e" candidate_uuid: "1c50de89a5a845668696dcb28c3badd6" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "974b802141bd4f13beffef2aa03f88f9" is_pre_election: true
I20260129 02:59:04.087713 21433 raft_consensus.cc:515] T e7d83d4ccf684e76acb29a0dc844e1b8 P 974b802141bd4f13beffef2aa03f88f9 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:59:04.079886 20507 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0340229d521b432580ae926f6062e3f7" candidate_uuid: "1c50de89a5a845668696dcb28c3badd6" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "974b802141bd4f13beffef2aa03f88f9" is_pre_election: true
I20260129 02:59:04.080304 21419 raft_consensus.cc:515] T cb0b3f259ebc4310927338a8740e87b8 P 1c50de89a5a845668696dcb28c3badd6 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } }
I20260129 02:59:04.080690 20514 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "75be4bd6bd45474795fd472eff2953ac" candidate_uuid: "1c50de89a5a845668696dcb28c3badd6" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "974b802141bd4f13beffef2aa03f88f9" is_pre_election: true
I20260129 02:59:04.080732 20405 leader_election.cc:304] T e33a417cf9c54cf5af73a4ee3aac4d3a P 197b880fa39d432a9b472845876bfaf0 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 197b880fa39d432a9b472845876bfaf0; no voters: 1c50de89a5a845668696dcb28c3badd6, 974b802141bd4f13beffef2aa03f88f9
I20260129 02:59:04.082032 21435 leader_election.cc:290] T 4f8f3a38f6e24ba4857875181f447442 P 1c50de89a5a845668696dcb28c3badd6 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 197b880fa39d432a9b472845876bfaf0 (127.18.211.1:42027), 974b802141bd4f13beffef2aa03f88f9 (127.18.211.2:34517)
I20260129 02:59:04.090189 21399 raft_consensus.cc:2749] T e33a417cf9c54cf5af73a4ee3aac4d3a P 197b880fa39d432a9b472845876bfaf0 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260129 02:59:04.059638 21397 raft_consensus.cc:515] T 926a250fa5834e88b96ccecbefbf5086 P 974b802141bd4f13beffef2aa03f88f9 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:59:04.087277 21374 raft_consensus.cc:515] T 75be4bd6bd45474795fd472eff2953ac P 974b802141bd4f13beffef2aa03f88f9 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:59:04.093721 21433 leader_election.cc:290] T e7d83d4ccf684e76acb29a0dc844e1b8 P 974b802141bd4f13beffef2aa03f88f9 [CANDIDATE]: Term 2 election: Requested vote from peers 197b880fa39d432a9b472845876bfaf0 (127.18.211.1:42027), 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3:37735)
W20260129 02:59:04.094499 20554 consensus_peers.cc:597] T 78aa39096868415d9afc2ffcb248e198 P 1c50de89a5a845668696dcb28c3badd6 -> Peer 7a9daf5d42a84615a6383cfec9a10608 (127.18.211.4:40439): Couldn't send request to peer 7a9daf5d42a84615a6383cfec9a10608. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 78aa39096868415d9afc2ffcb248e198. This is attempt 1: this message will repeat every 5th retry.
I20260129 02:59:04.085688 21409 leader_election.cc:290] T 7a9a69cd82724b1faed1d10aa091fc4a P 974b802141bd4f13beffef2aa03f88f9 [CANDIDATE]: Term 2 election: Requested vote from peers 197b880fa39d432a9b472845876bfaf0 (127.18.211.1:42027), 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3:37735)
I20260129 02:59:04.086362 21398 leader_election.cc:290] T c891d2bff4ce4568861bbc992e91fbb0 P 974b802141bd4f13beffef2aa03f88f9 [CANDIDATE]: Term 2 election: Requested vote from peers 197b880fa39d432a9b472845876bfaf0 (127.18.211.1:42027), 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3:37735)
I20260129 02:59:04.095100 20507 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a8479d1fe0bd41e5946186e3a6ca8c7a" candidate_uuid: "1c50de89a5a845668696dcb28c3badd6" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "974b802141bd4f13beffef2aa03f88f9" is_pre_election: true
I20260129 02:59:04.096143 21374 leader_election.cc:290] T 75be4bd6bd45474795fd472eff2953ac P 974b802141bd4f13beffef2aa03f88f9 [CANDIDATE]: Term 2 election: Requested vote from peers 197b880fa39d432a9b472845876bfaf0 (127.18.211.1:42027), 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3:37735)
I20260129 02:59:04.096539 21419 leader_election.cc:290] T cb0b3f259ebc4310927338a8740e87b8 P 1c50de89a5a845668696dcb28c3badd6 [CANDIDATE]: Term 2 election: Requested vote from peers 974b802141bd4f13beffef2aa03f88f9 (127.18.211.2:34517), 7a9daf5d42a84615a6383cfec9a10608 (127.18.211.4:40439)
I20260129 02:59:04.097561 21131 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "cb0b3f259ebc4310927338a8740e87b8" candidate_uuid: "1c50de89a5a845668696dcb28c3badd6" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "7a9daf5d42a84615a6383cfec9a10608"
I20260129 02:59:04.098287 21397 leader_election.cc:290] T 926a250fa5834e88b96ccecbefbf5086 P 974b802141bd4f13beffef2aa03f88f9 [CANDIDATE]: Term 2 election: Requested vote from peers 197b880fa39d432a9b472845876bfaf0 (127.18.211.1:42027), 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3:37735)
I20260129 02:59:04.103289 20511 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4f8f3a38f6e24ba4857875181f447442" candidate_uuid: "1c50de89a5a845668696dcb28c3badd6" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "974b802141bd4f13beffef2aa03f88f9" is_pre_election: true
I20260129 02:59:04.105161 20511 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "cb0b3f259ebc4310927338a8740e87b8" candidate_uuid: "1c50de89a5a845668696dcb28c3badd6" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "974b802141bd4f13beffef2aa03f88f9"
I20260129 02:59:04.110426 20556 leader_election.cc:304] T cedce331e1584ac8b636c9b8a72b2ba0 P 1c50de89a5a845668696dcb28c3badd6 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1c50de89a5a845668696dcb28c3badd6; no voters: 197b880fa39d432a9b472845876bfaf0, 974b802141bd4f13beffef2aa03f88f9
I20260129 02:59:04.112043 21435 raft_consensus.cc:2749] T cedce331e1584ac8b636c9b8a72b2ba0 P 1c50de89a5a845668696dcb28c3badd6 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260129 02:59:04.111624 20556 leader_election.cc:304] T c934d31efd3f4e5195605ddc354d14e1 P 1c50de89a5a845668696dcb28c3badd6 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1c50de89a5a845668696dcb28c3badd6; no voters: 197b880fa39d432a9b472845876bfaf0, 974b802141bd4f13beffef2aa03f88f9
I20260129 02:59:04.091970 20405 leader_election.cc:304] T 2ac659d732314e1288849f08fe5e074e P 197b880fa39d432a9b472845876bfaf0 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 197b880fa39d432a9b472845876bfaf0; no voters: 1c50de89a5a845668696dcb28c3badd6, 974b802141bd4f13beffef2aa03f88f9
I20260129 02:59:04.093079 20514 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4e3805ffa70242e8ade66da260dddf66" candidate_uuid: "1c50de89a5a845668696dcb28c3badd6" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "974b802141bd4f13beffef2aa03f88f9" is_pre_election: true
I20260129 02:59:04.113263 21429 raft_consensus.cc:2749] T c934d31efd3f4e5195605ddc354d14e1 P 1c50de89a5a845668696dcb28c3badd6 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260129 02:59:04.112627 20511 raft_consensus.cc:1275] T 78aa39096868415d9afc2ffcb248e198 P 974b802141bd4f13beffef2aa03f88f9 [term 1 FOLLOWER]: Refusing update from remote peer 1c50de89a5a845668696dcb28c3badd6: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260129 02:59:04.113544 21394 raft_consensus.cc:2749] T 2ac659d732314e1288849f08fe5e074e P 197b880fa39d432a9b472845876bfaf0 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260129 02:59:04.113866 20405 leader_election.cc:304] T 4408006eb5174de1896b037cf3f18457 P 197b880fa39d432a9b472845876bfaf0 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 197b880fa39d432a9b472845876bfaf0; no voters: 1c50de89a5a845668696dcb28c3badd6, 974b802141bd4f13beffef2aa03f88f9
I20260129 02:59:04.114698 21394 raft_consensus.cc:2749] T 4408006eb5174de1896b037cf3f18457 P 197b880fa39d432a9b472845876bfaf0 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260129 02:59:04.114952 20556 leader_election.cc:304] T a5a05fbcbbcf4916bbc01fd1a5b5e90d P 1c50de89a5a845668696dcb28c3badd6 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1c50de89a5a845668696dcb28c3badd6; no voters: 197b880fa39d432a9b472845876bfaf0, 974b802141bd4f13beffef2aa03f88f9
I20260129 02:59:04.115422 20405 leader_election.cc:304] T 368953c9102643b2a21f99af36b51905 P 197b880fa39d432a9b472845876bfaf0 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 197b880fa39d432a9b472845876bfaf0; no voters: 1c50de89a5a845668696dcb28c3badd6, 974b802141bd4f13beffef2aa03f88f9
I20260129 02:59:04.115706 21429 raft_consensus.cc:2749] T a5a05fbcbbcf4916bbc01fd1a5b5e90d P 1c50de89a5a845668696dcb28c3badd6 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260129 02:59:04.116264 21394 raft_consensus.cc:2749] T 368953c9102643b2a21f99af36b51905 P 197b880fa39d432a9b472845876bfaf0 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260129 02:59:04.116868 20556 leader_election.cc:304] T 4e420f90884f4796997b5ea41e49e411 P 1c50de89a5a845668696dcb28c3badd6 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1c50de89a5a845668696dcb28c3badd6; no voters: 197b880fa39d432a9b472845876bfaf0, 974b802141bd4f13beffef2aa03f88f9
I20260129 02:59:04.117513 21429 raft_consensus.cc:2749] T 4e420f90884f4796997b5ea41e49e411 P 1c50de89a5a845668696dcb28c3badd6 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260129 02:59:04.120297 20556 leader_election.cc:304] T 7a9a69cd82724b1faed1d10aa091fc4a P 1c50de89a5a845668696dcb28c3badd6 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1c50de89a5a845668696dcb28c3badd6; no voters: 197b880fa39d432a9b472845876bfaf0, 974b802141bd4f13beffef2aa03f88f9
I20260129 02:59:04.120947 21429 raft_consensus.cc:2749] T 7a9a69cd82724b1faed1d10aa091fc4a P 1c50de89a5a845668696dcb28c3badd6 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260129 02:59:04.121975 20556 leader_election.cc:304] T fcddab6bca684adf8ea032289a00f90e P 1c50de89a5a845668696dcb28c3badd6 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1c50de89a5a845668696dcb28c3badd6; no voters: 197b880fa39d432a9b472845876bfaf0, 974b802141bd4f13beffef2aa03f88f9
I20260129 02:59:04.122623 21429 raft_consensus.cc:2749] T fcddab6bca684adf8ea032289a00f90e P 1c50de89a5a845668696dcb28c3badd6 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260129 02:59:04.123826 20556 leader_election.cc:304] T 76f713c3331540d2a5b76164103e9eb1 P 1c50de89a5a845668696dcb28c3badd6 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1c50de89a5a845668696dcb28c3badd6; no voters: 197b880fa39d432a9b472845876bfaf0, 974b802141bd4f13beffef2aa03f88f9
I20260129 02:59:04.124534 21419 raft_consensus.cc:2749] T 76f713c3331540d2a5b76164103e9eb1 P 1c50de89a5a845668696dcb28c3badd6 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260129 02:59:04.123766 20405 leader_election.cc:304] T fd8311b779af4e1fa1402b2a02a602f2 P 197b880fa39d432a9b472845876bfaf0 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 197b880fa39d432a9b472845876bfaf0; no voters: 1c50de89a5a845668696dcb28c3badd6, 7a9daf5d42a84615a6383cfec9a10608
I20260129 02:59:04.125646 20556 leader_election.cc:304] T 1299723a4476420ab02fd8bfd7aa0645 P 1c50de89a5a845668696dcb28c3badd6 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1c50de89a5a845668696dcb28c3badd6; no voters: 197b880fa39d432a9b472845876bfaf0, 974b802141bd4f13beffef2aa03f88f9
I20260129 02:59:04.125896 21399 raft_consensus.cc:2749] T fd8311b779af4e1fa1402b2a02a602f2 P 197b880fa39d432a9b472845876bfaf0 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260129 02:59:04.126343 21419 raft_consensus.cc:2749] T 1299723a4476420ab02fd8bfd7aa0645 P 1c50de89a5a845668696dcb28c3badd6 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260129 02:59:04.128988 21374 raft_consensus.cc:493] T 45cc5f3d2688486d9b6dbe7e666dedde P 974b802141bd4f13beffef2aa03f88f9 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 1c50de89a5a845668696dcb28c3badd6)
I20260129 02:59:04.129628 20583 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0f20af7fbe5d4db89a6f17948d98912c" candidate_uuid: "974b802141bd4f13beffef2aa03f88f9" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1c50de89a5a845668696dcb28c3badd6"
I20260129 02:59:04.129594 21374 raft_consensus.cc:515] T 45cc5f3d2688486d9b6dbe7e666dedde P 974b802141bd4f13beffef2aa03f88f9 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:59:04.131512 20405 leader_election.cc:304] T 87d5fe48a44a408aaf03e894a67ecb2f P 197b880fa39d432a9b472845876bfaf0 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 197b880fa39d432a9b472845876bfaf0; no voters: 7a9daf5d42a84615a6383cfec9a10608, 974b802141bd4f13beffef2aa03f88f9
I20260129 02:59:04.131803 21374 leader_election.cc:290] T 45cc5f3d2688486d9b6dbe7e666dedde P 974b802141bd4f13beffef2aa03f88f9 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 197b880fa39d432a9b472845876bfaf0 (127.18.211.1:42027), 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3:37735)
I20260129 02:59:04.132293 21394 raft_consensus.cc:2749] T 87d5fe48a44a408aaf03e894a67ecb2f P 197b880fa39d432a9b472845876bfaf0 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260129 02:59:04.132642 20588 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c891d2bff4ce4568861bbc992e91fbb0" candidate_uuid: "974b802141bd4f13beffef2aa03f88f9" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1c50de89a5a845668696dcb28c3badd6"
I20260129 02:59:04.133148 20585 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7a9a69cd82724b1faed1d10aa091fc4a" candidate_uuid: "974b802141bd4f13beffef2aa03f88f9" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1c50de89a5a845668696dcb28c3badd6"
I20260129 02:59:04.049719 20589 raft_consensus.cc:2468] T 4f8f3a38f6e24ba4857875181f447442 P 1c50de89a5a845668696dcb28c3badd6 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 197b880fa39d432a9b472845876bfaf0 in term 1.
I20260129 02:59:04.139566 20441 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4f8f3a38f6e24ba4857875181f447442" candidate_uuid: "1c50de89a5a845668696dcb28c3badd6" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "197b880fa39d432a9b472845876bfaf0" is_pre_election: true
I20260129 02:59:04.139928 20553 leader_election.cc:304] T 1ce4ddf5b10e4ae8b193b13b2e5936ee P 1c50de89a5a845668696dcb28c3badd6 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1c50de89a5a845668696dcb28c3badd6; no voters: 197b880fa39d432a9b472845876bfaf0, 974b802141bd4f13beffef2aa03f88f9
I20260129 02:59:04.141184 20553 leader_election.cc:304] T 2ac659d732314e1288849f08fe5e074e P 1c50de89a5a845668696dcb28c3badd6 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1c50de89a5a845668696dcb28c3badd6; no voters: 197b880fa39d432a9b472845876bfaf0, 974b802141bd4f13beffef2aa03f88f9
I20260129 02:59:04.142215 20553 leader_election.cc:304] T 0678b8b468bd43cdbc3e485900844821 P 1c50de89a5a845668696dcb28c3badd6 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1c50de89a5a845668696dcb28c3badd6; no voters: 197b880fa39d432a9b472845876bfaf0, 974b802141bd4f13beffef2aa03f88f9
I20260129 02:59:04.153146 20553 leader_election.cc:304] T c891d2bff4ce4568861bbc992e91fbb0 P 1c50de89a5a845668696dcb28c3badd6 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1c50de89a5a845668696dcb28c3badd6; no voters: 197b880fa39d432a9b472845876bfaf0, 974b802141bd4f13beffef2aa03f88f9
I20260129 02:59:04.142743 21429 raft_consensus.cc:2749] T 1ce4ddf5b10e4ae8b193b13b2e5936ee P 1c50de89a5a845668696dcb28c3badd6 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260129 02:59:04.143622 20480 leader_election.cc:304] T a609ac02fae84725ab743f95f51d33ce P 974b802141bd4f13beffef2aa03f88f9 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 7a9daf5d42a84615a6383cfec9a10608, 974b802141bd4f13beffef2aa03f88f9; no voters: 197b880fa39d432a9b472845876bfaf0
I20260129 02:59:04.154285 21426 raft_consensus.cc:2749] T 0678b8b468bd43cdbc3e485900844821 P 1c50de89a5a845668696dcb28c3badd6 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260129 02:59:04.144013 21432 raft_consensus.cc:2749] T 2ac659d732314e1288849f08fe5e074e P 1c50de89a5a845668696dcb28c3badd6 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260129 02:59:04.154930 21426 raft_consensus.cc:2749] T c891d2bff4ce4568861bbc992e91fbb0 P 1c50de89a5a845668696dcb28c3badd6 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260129 02:59:04.148862 20407 leader_election.cc:304] T a4c1226b987b468fa84dfc6ba0e2dbe1 P 197b880fa39d432a9b472845876bfaf0 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 197b880fa39d432a9b472845876bfaf0; no voters: 1c50de89a5a845668696dcb28c3badd6, 974b802141bd4f13beffef2aa03f88f9
I20260129 02:59:04.155113 20480 leader_election.cc:304] T febffe4a8b394368b69b1d164308fb9a P 974b802141bd4f13beffef2aa03f88f9 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 974b802141bd4f13beffef2aa03f88f9; no voters: 197b880fa39d432a9b472845876bfaf0, 7a9daf5d42a84615a6383cfec9a10608
I20260129 02:59:04.155788 21399 raft_consensus.cc:2749] T a4c1226b987b468fa84dfc6ba0e2dbe1 P 197b880fa39d432a9b472845876bfaf0 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260129 02:59:04.155618 21436 raft_consensus.cc:2804] T a609ac02fae84725ab743f95f51d33ce P 974b802141bd4f13beffef2aa03f88f9 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260129 02:59:04.150667 20511 raft_consensus.cc:1240] T 0f20af7fbe5d4db89a6f17948d98912c P 974b802141bd4f13beffef2aa03f88f9 [term 2 FOLLOWER]: Rejecting Update request from peer 197b880fa39d432a9b472845876bfaf0 for earlier term 1. Current term is 2. Ops: []
I20260129 02:59:04.156571 21374 raft_consensus.cc:2749] T febffe4a8b394368b69b1d164308fb9a P 974b802141bd4f13beffef2aa03f88f9 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260129 02:59:04.156579 20407 leader_election.cc:304] T add8d4923d1a489e86882c3a25692b95 P 197b880fa39d432a9b472845876bfaf0 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 197b880fa39d432a9b472845876bfaf0; no voters: 1c50de89a5a845668696dcb28c3badd6, 974b802141bd4f13beffef2aa03f88f9
I20260129 02:59:04.158524 21394 raft_consensus.cc:2749] T add8d4923d1a489e86882c3a25692b95 P 197b880fa39d432a9b472845876bfaf0 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260129 02:59:04.158615 20553 leader_election.cc:304] T 0f20af7fbe5d4db89a6f17948d98912c P 1c50de89a5a845668696dcb28c3badd6 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1c50de89a5a845668696dcb28c3badd6; no voters: 197b880fa39d432a9b472845876bfaf0, 974b802141bd4f13beffef2aa03f88f9
I20260129 02:59:04.159518 21435 raft_consensus.cc:2749] T 0f20af7fbe5d4db89a6f17948d98912c P 1c50de89a5a845668696dcb28c3badd6 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260129 02:59:04.156589 21436 raft_consensus.cc:493] T a609ac02fae84725ab743f95f51d33ce P 974b802141bd4f13beffef2aa03f88f9 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 197b880fa39d432a9b472845876bfaf0)
I20260129 02:59:04.160236 21436 raft_consensus.cc:3060] T a609ac02fae84725ab743f95f51d33ce P 974b802141bd4f13beffef2aa03f88f9 [term 1 FOLLOWER]: Advancing to term 2
I20260129 02:59:04.160069 20553 leader_election.cc:304] T 3ccac82a01a2445fa5b0ce6c0fcf2295 P 1c50de89a5a845668696dcb28c3badd6 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1c50de89a5a845668696dcb28c3badd6; no voters: 197b880fa39d432a9b472845876bfaf0, 974b802141bd4f13beffef2aa03f88f9
I20260129 02:59:04.161396 21435 raft_consensus.cc:2749] T 3ccac82a01a2445fa5b0ce6c0fcf2295 P 1c50de89a5a845668696dcb28c3badd6 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260129 02:59:04.161487 20553 leader_election.cc:304] T 3906b080a5b34a74b8b64128635e8997 P 1c50de89a5a845668696dcb28c3badd6 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1c50de89a5a845668696dcb28c3badd6; no voters: 197b880fa39d432a9b472845876bfaf0, 974b802141bd4f13beffef2aa03f88f9
I20260129 02:59:04.162932 20553 leader_election.cc:304] T 75be4bd6bd45474795fd472eff2953ac P 1c50de89a5a845668696dcb28c3badd6 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1c50de89a5a845668696dcb28c3badd6; no voters: 197b880fa39d432a9b472845876bfaf0, 974b802141bd4f13beffef2aa03f88f9
I20260129 02:59:04.165266 21399 consensus_queue.cc:1059] T 0f20af7fbe5d4db89a6f17948d98912c P 197b880fa39d432a9b472845876bfaf0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260129 02:59:04.166432 21403 raft_consensus.cc:2749] T 75be4bd6bd45474795fd472eff2953ac P 1c50de89a5a845668696dcb28c3badd6 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260129 02:59:04.169081 20553 leader_election.cc:304] T a1a39dcd328e4268ae41b988c66b4a75 P 1c50de89a5a845668696dcb28c3badd6 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1c50de89a5a845668696dcb28c3badd6; no voters: 197b880fa39d432a9b472845876bfaf0, 974b802141bd4f13beffef2aa03f88f9
I20260129 02:59:04.163180 21435 raft_consensus.cc:2749] T 3906b080a5b34a74b8b64128635e8997 P 1c50de89a5a845668696dcb28c3badd6 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260129 02:59:04.166489 21399 raft_consensus.cc:3055] T 0f20af7fbe5d4db89a6f17948d98912c P 197b880fa39d432a9b472845876bfaf0 [term 1 LEADER]: Stepping down as leader of term 1
I20260129 02:59:04.170729 20438 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c891d2bff4ce4568861bbc992e91fbb0" candidate_uuid: "974b802141bd4f13beffef2aa03f88f9" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "197b880fa39d432a9b472845876bfaf0"
I20260129 02:59:04.171576 21399 raft_consensus.cc:740] T 0f20af7fbe5d4db89a6f17948d98912c P 197b880fa39d432a9b472845876bfaf0 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 197b880fa39d432a9b472845876bfaf0, State: Running, Role: LEADER
I20260129 02:59:04.171952 21435 raft_consensus.cc:2749] T a1a39dcd328e4268ae41b988c66b4a75 P 1c50de89a5a845668696dcb28c3badd6 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260129 02:59:04.172559 21399 consensus_queue.cc:260] T 0f20af7fbe5d4db89a6f17948d98912c P 197b880fa39d432a9b472845876bfaf0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:59:04.172966 20553 leader_election.cc:304] T a8479d1fe0bd41e5946186e3a6ca8c7a P 1c50de89a5a845668696dcb28c3badd6 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1c50de89a5a845668696dcb28c3badd6; no voters: 197b880fa39d432a9b472845876bfaf0, 974b802141bd4f13beffef2aa03f88f9
I20260129 02:59:04.173913 21435 raft_consensus.cc:2749] T a8479d1fe0bd41e5946186e3a6ca8c7a P 1c50de89a5a845668696dcb28c3badd6 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260129 02:59:04.174017 21399 raft_consensus.cc:3060] T 0f20af7fbe5d4db89a6f17948d98912c P 197b880fa39d432a9b472845876bfaf0 [term 1 FOLLOWER]: Advancing to term 2
I20260129 02:59:04.174412 20553 leader_election.cc:304] T 0340229d521b432580ae926f6062e3f7 P 1c50de89a5a845668696dcb28c3badd6 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1c50de89a5a845668696dcb28c3badd6; no voters: 197b880fa39d432a9b472845876bfaf0, 974b802141bd4f13beffef2aa03f88f9
I20260129 02:59:04.174901 20438 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7a9a69cd82724b1faed1d10aa091fc4a" candidate_uuid: "974b802141bd4f13beffef2aa03f88f9" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "197b880fa39d432a9b472845876bfaf0"
I20260129 02:59:04.176204 20441 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e7d83d4ccf684e76acb29a0dc844e1b8" candidate_uuid: "974b802141bd4f13beffef2aa03f88f9" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "197b880fa39d432a9b472845876bfaf0"
I20260129 02:59:04.177390 21435 raft_consensus.cc:2749] T 0340229d521b432580ae926f6062e3f7 P 1c50de89a5a845668696dcb28c3badd6 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260129 02:59:04.157166 20480 leader_election.cc:304] T 00e79d8f25a34676b1d866eb2068a2b2 P 974b802141bd4f13beffef2aa03f88f9 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 974b802141bd4f13beffef2aa03f88f9; no voters: 197b880fa39d432a9b472845876bfaf0, 7a9daf5d42a84615a6383cfec9a10608
I20260129 02:59:04.178295 21374 raft_consensus.cc:2749] T 00e79d8f25a34676b1d866eb2068a2b2 P 974b802141bd4f13beffef2aa03f88f9 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260129 02:59:04.179445 20556 leader_election.cc:304] T cb0b3f259ebc4310927338a8740e87b8 P 1c50de89a5a845668696dcb28c3badd6 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1c50de89a5a845668696dcb28c3badd6; no voters: 7a9daf5d42a84615a6383cfec9a10608, 974b802141bd4f13beffef2aa03f88f9
I20260129 02:59:04.179780 20511 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "cb0b3f259ebc4310927338a8740e87b8" candidate_uuid: "7a9daf5d42a84615a6383cfec9a10608" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "974b802141bd4f13beffef2aa03f88f9" is_pre_election: true
I20260129 02:59:04.180784 20514 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4f5a77e94fef4b16a2ba7b067c8b144a" candidate_uuid: "7a9daf5d42a84615a6383cfec9a10608" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "974b802141bd4f13beffef2aa03f88f9" is_pre_election: true
I20260129 02:59:04.181886 20513 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "febffe4a8b394368b69b1d164308fb9a" candidate_uuid: "7a9daf5d42a84615a6383cfec9a10608" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "974b802141bd4f13beffef2aa03f88f9" is_pre_election: true
I20260129 02:59:04.198975 20441 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "926a250fa5834e88b96ccecbefbf5086" candidate_uuid: "974b802141bd4f13beffef2aa03f88f9" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "197b880fa39d432a9b472845876bfaf0"
I20260129 02:59:04.182879 20508 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3e2d15f8640b407195210c476e09df91" candidate_uuid: "7a9daf5d42a84615a6383cfec9a10608" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "974b802141bd4f13beffef2aa03f88f9" is_pre_election: true
I20260129 02:59:04.183600 20510 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "87d5fe48a44a408aaf03e894a67ecb2f" candidate_uuid: "7a9daf5d42a84615a6383cfec9a10608" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "974b802141bd4f13beffef2aa03f88f9" is_pre_election: true
I20260129 02:59:04.184217 20507 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a609ac02fae84725ab743f95f51d33ce" candidate_uuid: "7a9daf5d42a84615a6383cfec9a10608" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "974b802141bd4f13beffef2aa03f88f9" is_pre_election: true
I20260129 02:59:04.185876 20589 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "75be4bd6bd45474795fd472eff2953ac" candidate_uuid: "974b802141bd4f13beffef2aa03f88f9" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1c50de89a5a845668696dcb28c3badd6"
I20260129 02:59:04.187021 20585 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "926a250fa5834e88b96ccecbefbf5086" candidate_uuid: "974b802141bd4f13beffef2aa03f88f9" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1c50de89a5a845668696dcb28c3badd6"
I20260129 02:59:04.187703 20582 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "45cc5f3d2688486d9b6dbe7e666dedde" candidate_uuid: "974b802141bd4f13beffef2aa03f88f9" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1c50de89a5a845668696dcb28c3badd6" is_pre_election: true
I20260129 02:59:04.203460 21419 raft_consensus.cc:2749] T cb0b3f259ebc4310927338a8740e87b8 P 1c50de89a5a845668696dcb28c3badd6 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260129 02:59:04.206230 20441 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "45cc5f3d2688486d9b6dbe7e666dedde" candidate_uuid: "974b802141bd4f13beffef2aa03f88f9" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "197b880fa39d432a9b472845876bfaf0" is_pre_election: true
I20260129 02:59:04.190982 21436 raft_consensus.cc:515] T a609ac02fae84725ab743f95f51d33ce P 974b802141bd4f13beffef2aa03f88f9 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } }
I20260129 02:59:04.177388 20553 leader_election.cc:304] T c2633a6fc34c44579de348d208b0c01e P 1c50de89a5a845668696dcb28c3badd6 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1c50de89a5a845668696dcb28c3badd6; no voters: 197b880fa39d432a9b472845876bfaf0, 974b802141bd4f13beffef2aa03f88f9
I20260129 02:59:04.196779 20480 leader_election.cc:304] T 0f20af7fbe5d4db89a6f17948d98912c P 974b802141bd4f13beffef2aa03f88f9 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 974b802141bd4f13beffef2aa03f88f9; no voters: 197b880fa39d432a9b472845876bfaf0, 1c50de89a5a845668696dcb28c3badd6
I20260129 02:59:04.191005 20587 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e7d83d4ccf684e76acb29a0dc844e1b8" candidate_uuid: "974b802141bd4f13beffef2aa03f88f9" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1c50de89a5a845668696dcb28c3badd6"
I20260129 02:59:04.209184 21374 raft_consensus.cc:2749] T 0f20af7fbe5d4db89a6f17948d98912c P 974b802141bd4f13beffef2aa03f88f9 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260129 02:59:04.197701 20438 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "75be4bd6bd45474795fd472eff2953ac" candidate_uuid: "974b802141bd4f13beffef2aa03f88f9" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "197b880fa39d432a9b472845876bfaf0"
I20260129 02:59:04.211990 20507 raft_consensus.cc:1240] T e7d83d4ccf684e76acb29a0dc844e1b8 P 974b802141bd4f13beffef2aa03f88f9 [term 2 FOLLOWER]: Rejecting Update request from peer 197b880fa39d432a9b472845876bfaf0 for earlier term 1. Current term is 2. Ops: []
I20260129 02:59:04.214125 21419 consensus_queue.cc:1048] T 78aa39096868415d9afc2ffcb248e198 P 1c50de89a5a845668696dcb28c3badd6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260129 02:59:04.217670 20507 raft_consensus.cc:1240] T 7a9a69cd82724b1faed1d10aa091fc4a P 974b802141bd4f13beffef2aa03f88f9 [term 2 FOLLOWER]: Rejecting Update request from peer 197b880fa39d432a9b472845876bfaf0 for earlier term 1. Current term is 2. Ops: []
I20260129 02:59:04.214322 21416 consensus_queue.cc:1059] T e7d83d4ccf684e76acb29a0dc844e1b8 P 197b880fa39d432a9b472845876bfaf0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260129 02:59:04.215600 21102 leader_election.cc:304] T cb0b3f259ebc4310927338a8740e87b8 P 7a9daf5d42a84615a6383cfec9a10608 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7a9daf5d42a84615a6383cfec9a10608; no voters: 1c50de89a5a845668696dcb28c3badd6, 974b802141bd4f13beffef2aa03f88f9
I20260129 02:59:04.218581 21416 raft_consensus.cc:3055] T e7d83d4ccf684e76acb29a0dc844e1b8 P 197b880fa39d432a9b472845876bfaf0 [term 1 LEADER]: Stepping down as leader of term 1
I20260129 02:59:04.218942 21416 raft_consensus.cc:740] T e7d83d4ccf684e76acb29a0dc844e1b8 P 197b880fa39d432a9b472845876bfaf0 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 197b880fa39d432a9b472845876bfaf0, State: Running, Role: LEADER
I20260129 02:59:04.219074 21102 leader_election.cc:304] T 4f5a77e94fef4b16a2ba7b067c8b144a P 7a9daf5d42a84615a6383cfec9a10608 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7a9daf5d42a84615a6383cfec9a10608; no voters: 1c50de89a5a845668696dcb28c3badd6, 974b802141bd4f13beffef2aa03f88f9
I20260129 02:59:04.219233 21386 consensus_queue.cc:1059] T 7a9a69cd82724b1faed1d10aa091fc4a P 197b880fa39d432a9b472845876bfaf0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260129 02:59:04.220063 21428 raft_consensus.cc:2749] T cb0b3f259ebc4310927338a8740e87b8 P 7a9daf5d42a84615a6383cfec9a10608 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260129 02:59:04.219887 21416 consensus_queue.cc:260] T e7d83d4ccf684e76acb29a0dc844e1b8 P 197b880fa39d432a9b472845876bfaf0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:59:04.220391 21386 raft_consensus.cc:3055] T 7a9a69cd82724b1faed1d10aa091fc4a P 197b880fa39d432a9b472845876bfaf0 [term 1 LEADER]: Stepping down as leader of term 1
I20260129 02:59:04.220697 21428 raft_consensus.cc:2749] T 4f5a77e94fef4b16a2ba7b067c8b144a P 7a9daf5d42a84615a6383cfec9a10608 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260129 02:59:04.220839 21386 raft_consensus.cc:740] T 7a9a69cd82724b1faed1d10aa091fc4a P 197b880fa39d432a9b472845876bfaf0 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 197b880fa39d432a9b472845876bfaf0, State: Running, Role: LEADER
I20260129 02:59:04.216198 21403 raft_consensus.cc:2749] T c2633a6fc34c44579de348d208b0c01e P 1c50de89a5a845668696dcb28c3badd6 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260129 02:59:04.221328 21416 raft_consensus.cc:3060] T e7d83d4ccf684e76acb29a0dc844e1b8 P 197b880fa39d432a9b472845876bfaf0 [term 1 FOLLOWER]: Advancing to term 2
I20260129 02:59:04.222213 21102 leader_election.cc:304] T febffe4a8b394368b69b1d164308fb9a P 7a9daf5d42a84615a6383cfec9a10608 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7a9daf5d42a84615a6383cfec9a10608; no voters: 197b880fa39d432a9b472845876bfaf0, 974b802141bd4f13beffef2aa03f88f9
I20260129 02:59:04.221865 21386 consensus_queue.cc:260] T 7a9a69cd82724b1faed1d10aa091fc4a P 197b880fa39d432a9b472845876bfaf0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:59:04.223074 21424 raft_consensus.cc:2749] T febffe4a8b394368b69b1d164308fb9a P 7a9daf5d42a84615a6383cfec9a10608 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260129 02:59:04.223312 21386 raft_consensus.cc:3060] T 7a9a69cd82724b1faed1d10aa091fc4a P 197b880fa39d432a9b472845876bfaf0 [term 1 FOLLOWER]: Advancing to term 2
I20260129 02:59:04.224380 21102 leader_election.cc:304] T 3e2d15f8640b407195210c476e09df91 P 7a9daf5d42a84615a6383cfec9a10608 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7a9daf5d42a84615a6383cfec9a10608; no voters: 1c50de89a5a845668696dcb28c3badd6, 974b802141bd4f13beffef2aa03f88f9
I20260129 02:59:04.225111 21424 raft_consensus.cc:2749] T 3e2d15f8640b407195210c476e09df91 P 7a9daf5d42a84615a6383cfec9a10608 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260129 02:59:04.226279 21102 leader_election.cc:304] T 87d5fe48a44a408aaf03e894a67ecb2f P 7a9daf5d42a84615a6383cfec9a10608 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7a9daf5d42a84615a6383cfec9a10608; no voters: 197b880fa39d432a9b472845876bfaf0, 974b802141bd4f13beffef2aa03f88f9
I20260129 02:59:04.226279 20553 leader_election.cc:304] T 4e3805ffa70242e8ade66da260dddf66 P 1c50de89a5a845668696dcb28c3badd6 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1c50de89a5a845668696dcb28c3badd6; no voters: 197b880fa39d432a9b472845876bfaf0, 974b802141bd4f13beffef2aa03f88f9
I20260129 02:59:04.227581 21426 raft_consensus.cc:2749] T 4e3805ffa70242e8ade66da260dddf66 P 1c50de89a5a845668696dcb28c3badd6 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260129 02:59:04.230785 20405 leader_election.cc:304] T 4f8f3a38f6e24ba4857875181f447442 P 197b880fa39d432a9b472845876bfaf0 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 197b880fa39d432a9b472845876bfaf0, 1c50de89a5a845668696dcb28c3badd6; no voters: 974b802141bd4f13beffef2aa03f88f9
I20260129 02:59:04.232362 21424 raft_consensus.cc:2749] T 87d5fe48a44a408aaf03e894a67ecb2f P 7a9daf5d42a84615a6383cfec9a10608 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260129 02:59:04.230749 20553 leader_election.cc:304] T 4f8f3a38f6e24ba4857875181f447442 P 1c50de89a5a845668696dcb28c3badd6 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1c50de89a5a845668696dcb28c3badd6; no voters: 197b880fa39d432a9b472845876bfaf0, 974b802141bd4f13beffef2aa03f88f9
I20260129 02:59:04.234978 21419 raft_consensus.cc:2749] T 4f8f3a38f6e24ba4857875181f447442 P 1c50de89a5a845668696dcb28c3badd6 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260129 02:59:04.235021 21406 raft_consensus.cc:2804] T 4f8f3a38f6e24ba4857875181f447442 P 197b880fa39d432a9b472845876bfaf0 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260129 02:59:04.235539 21406 raft_consensus.cc:493] T 4f8f3a38f6e24ba4857875181f447442 P 197b880fa39d432a9b472845876bfaf0 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 974b802141bd4f13beffef2aa03f88f9)
I20260129 02:59:04.235954 21406 raft_consensus.cc:3060] T 4f8f3a38f6e24ba4857875181f447442 P 197b880fa39d432a9b472845876bfaf0 [term 1 FOLLOWER]: Advancing to term 2
I20260129 02:59:04.236878 21102 leader_election.cc:304] T a609ac02fae84725ab743f95f51d33ce P 7a9daf5d42a84615a6383cfec9a10608 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7a9daf5d42a84615a6383cfec9a10608; no voters: 197b880fa39d432a9b472845876bfaf0, 974b802141bd4f13beffef2aa03f88f9
I20260129 02:59:04.237640 21424 raft_consensus.cc:2749] T a609ac02fae84725ab743f95f51d33ce P 7a9daf5d42a84615a6383cfec9a10608 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260129 02:59:04.244720 21406 raft_consensus.cc:515] T 4f8f3a38f6e24ba4857875181f447442 P 197b880fa39d432a9b472845876bfaf0 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:59:04.252467 20587 raft_consensus.cc:1240] T cb0b3f259ebc4310927338a8740e87b8 P 1c50de89a5a845668696dcb28c3badd6 [term 2 FOLLOWER]: Rejecting Update request from peer 974b802141bd4f13beffef2aa03f88f9 for earlier term 1. Current term is 2. Ops: []
I20260129 02:59:04.250897 21436 leader_election.cc:290] T a609ac02fae84725ab743f95f51d33ce P 974b802141bd4f13beffef2aa03f88f9 [CANDIDATE]: Term 2 election: Requested vote from peers 197b880fa39d432a9b472845876bfaf0 (127.18.211.1:42027), 7a9daf5d42a84615a6383cfec9a10608 (127.18.211.4:40439)
I20260129 02:59:04.258195 20479 leader_election.cc:304] T c891d2bff4ce4568861bbc992e91fbb0 P 974b802141bd4f13beffef2aa03f88f9 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 974b802141bd4f13beffef2aa03f88f9; no voters: 197b880fa39d432a9b472845876bfaf0, 1c50de89a5a845668696dcb28c3badd6
I20260129 02:59:04.258983 21436 consensus_queue.cc:1059] T cb0b3f259ebc4310927338a8740e87b8 P 974b802141bd4f13beffef2aa03f88f9 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260129 02:59:04.260178 21436 raft_consensus.cc:2749] T c891d2bff4ce4568861bbc992e91fbb0 P 974b802141bd4f13beffef2aa03f88f9 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260129 02:59:04.260295 21398 raft_consensus.cc:3055] T cb0b3f259ebc4310927338a8740e87b8 P 974b802141bd4f13beffef2aa03f88f9 [term 1 LEADER]: Stepping down as leader of term 1
I20260129 02:59:04.260722 21398 raft_consensus.cc:740] T cb0b3f259ebc4310927338a8740e87b8 P 974b802141bd4f13beffef2aa03f88f9 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 974b802141bd4f13beffef2aa03f88f9, State: Running, Role: LEADER
I20260129 02:59:04.260762 20510 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4f8f3a38f6e24ba4857875181f447442" candidate_uuid: "197b880fa39d432a9b472845876bfaf0" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "974b802141bd4f13beffef2aa03f88f9"
I20260129 02:59:04.260964 21406 leader_election.cc:290] T 4f8f3a38f6e24ba4857875181f447442 P 197b880fa39d432a9b472845876bfaf0 [CANDIDATE]: Term 2 election: Requested vote from peers 974b802141bd4f13beffef2aa03f88f9 (127.18.211.2:34517), 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3:37735)
I20260129 02:59:04.261535 21398 consensus_queue.cc:260] T cb0b3f259ebc4310927338a8740e87b8 P 974b802141bd4f13beffef2aa03f88f9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } }
I20260129 02:59:04.262928 21398 raft_consensus.cc:3060] T cb0b3f259ebc4310927338a8740e87b8 P 974b802141bd4f13beffef2aa03f88f9 [term 1 FOLLOWER]: Advancing to term 2
I20260129 02:59:04.259233 21131 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a609ac02fae84725ab743f95f51d33ce" candidate_uuid: "974b802141bd4f13beffef2aa03f88f9" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "7a9daf5d42a84615a6383cfec9a10608"
I20260129 02:59:04.265551 20510 raft_consensus.cc:1240] T 926a250fa5834e88b96ccecbefbf5086 P 974b802141bd4f13beffef2aa03f88f9 [term 2 FOLLOWER]: Rejecting Update request from peer 197b880fa39d432a9b472845876bfaf0 for earlier term 1. Current term is 2. Ops: []
I20260129 02:59:04.267061 21399 consensus_queue.cc:1059] T 926a250fa5834e88b96ccecbefbf5086 P 197b880fa39d432a9b472845876bfaf0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260129 02:59:04.268244 21394 raft_consensus.cc:3055] T 926a250fa5834e88b96ccecbefbf5086 P 197b880fa39d432a9b472845876bfaf0 [term 1 LEADER]: Stepping down as leader of term 1
I20260129 02:59:04.268601 21394 raft_consensus.cc:740] T 926a250fa5834e88b96ccecbefbf5086 P 197b880fa39d432a9b472845876bfaf0 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 197b880fa39d432a9b472845876bfaf0, State: Running, Role: LEADER
I20260129 02:59:04.269371 21394 consensus_queue.cc:260] T 926a250fa5834e88b96ccecbefbf5086 P 197b880fa39d432a9b472845876bfaf0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:59:04.270561 21394 raft_consensus.cc:3060] T 926a250fa5834e88b96ccecbefbf5086 P 197b880fa39d432a9b472845876bfaf0 [term 1 FOLLOWER]: Advancing to term 2
I20260129 02:59:04.275810 20479 leader_election.cc:304] T e7d83d4ccf684e76acb29a0dc844e1b8 P 974b802141bd4f13beffef2aa03f88f9 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 974b802141bd4f13beffef2aa03f88f9; no voters: 197b880fa39d432a9b472845876bfaf0, 1c50de89a5a845668696dcb28c3badd6
I20260129 02:59:04.280632 21374 raft_consensus.cc:2749] T e7d83d4ccf684e76acb29a0dc844e1b8 P 974b802141bd4f13beffef2aa03f88f9 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260129 02:59:04.285110 20582 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4f8f3a38f6e24ba4857875181f447442" candidate_uuid: "197b880fa39d432a9b472845876bfaf0" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1c50de89a5a845668696dcb28c3badd6"
I20260129 02:59:04.286370 20438 raft_consensus.cc:1275] T 78aa39096868415d9afc2ffcb248e198 P 197b880fa39d432a9b472845876bfaf0 [term 1 FOLLOWER]: Refusing update from remote peer 1c50de89a5a845668696dcb28c3badd6: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260129 02:59:04.289114 20405 leader_election.cc:304] T 4f8f3a38f6e24ba4857875181f447442 P 197b880fa39d432a9b472845876bfaf0 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 197b880fa39d432a9b472845876bfaf0; no voters: 1c50de89a5a845668696dcb28c3badd6, 974b802141bd4f13beffef2aa03f88f9
I20260129 02:59:04.291805 21361 raft_consensus.cc:2749] T 4f8f3a38f6e24ba4857875181f447442 P 197b880fa39d432a9b472845876bfaf0 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260129 02:59:04.290294 20479 leader_election.cc:304] T 7a9a69cd82724b1faed1d10aa091fc4a P 974b802141bd4f13beffef2aa03f88f9 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 974b802141bd4f13beffef2aa03f88f9; no voters: 197b880fa39d432a9b472845876bfaf0, 1c50de89a5a845668696dcb28c3badd6
I20260129 02:59:04.294917 21397 raft_consensus.cc:2749] T 7a9a69cd82724b1faed1d10aa091fc4a P 974b802141bd4f13beffef2aa03f88f9 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260129 02:59:04.296343 20479 leader_election.cc:304] T 926a250fa5834e88b96ccecbefbf5086 P 974b802141bd4f13beffef2aa03f88f9 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 974b802141bd4f13beffef2aa03f88f9; no voters: 197b880fa39d432a9b472845876bfaf0, 1c50de89a5a845668696dcb28c3badd6
I20260129 02:59:04.299185 21419 consensus_queue.cc:1048] T 78aa39096868415d9afc2ffcb248e198 P 1c50de89a5a845668696dcb28c3badd6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260129 02:59:04.293588 20438 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a609ac02fae84725ab743f95f51d33ce" candidate_uuid: "974b802141bd4f13beffef2aa03f88f9" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "197b880fa39d432a9b472845876bfaf0"
I20260129 02:59:04.302328 20479 leader_election.cc:304] T 45cc5f3d2688486d9b6dbe7e666dedde P 974b802141bd4f13beffef2aa03f88f9 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 974b802141bd4f13beffef2aa03f88f9; no voters: 197b880fa39d432a9b472845876bfaf0, 1c50de89a5a845668696dcb28c3badd6
I20260129 02:59:04.304039 21374 raft_consensus.cc:2749] T 926a250fa5834e88b96ccecbefbf5086 P 974b802141bd4f13beffef2aa03f88f9 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260129 02:59:04.304574 21426 raft_consensus.cc:2955] T 78aa39096868415d9afc2ffcb248e198 P 1c50de89a5a845668696dcb28c3badd6 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 7a9daf5d42a84615a6383cfec9a10608 (127.18.211.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } attrs { replace: true } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: NON_VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: true } } }
I20260129 02:59:04.313421 20507 raft_consensus.cc:2955] T 78aa39096868415d9afc2ffcb248e198 P 974b802141bd4f13beffef2aa03f88f9 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 7a9daf5d42a84615a6383cfec9a10608 (127.18.211.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } attrs { replace: true } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: NON_VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: true } } }
I20260129 02:59:04.316267 21436 raft_consensus.cc:2749] T 45cc5f3d2688486d9b6dbe7e666dedde P 974b802141bd4f13beffef2aa03f88f9 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260129 02:59:04.317544 20479 leader_election.cc:304] T 75be4bd6bd45474795fd472eff2953ac P 974b802141bd4f13beffef2aa03f88f9 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 974b802141bd4f13beffef2aa03f88f9; no voters: 197b880fa39d432a9b472845876bfaf0, 1c50de89a5a845668696dcb28c3badd6
I20260129 02:59:04.322135 21374 raft_consensus.cc:2749] T 75be4bd6bd45474795fd472eff2953ac P 974b802141bd4f13beffef2aa03f88f9 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260129 02:59:04.321915 20479 leader_election.cc:304] T a609ac02fae84725ab743f95f51d33ce P 974b802141bd4f13beffef2aa03f88f9 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 974b802141bd4f13beffef2aa03f88f9; no voters: 197b880fa39d432a9b472845876bfaf0, 7a9daf5d42a84615a6383cfec9a10608
I20260129 02:59:04.334954 20507 raft_consensus.cc:1240] T a609ac02fae84725ab743f95f51d33ce P 974b802141bd4f13beffef2aa03f88f9 [term 2 FOLLOWER]: Rejecting Update request from peer 197b880fa39d432a9b472845876bfaf0 for earlier term 1. Current term is 2. Ops: []
I20260129 02:59:04.336597 21394 consensus_queue.cc:1059] T a609ac02fae84725ab743f95f51d33ce P 197b880fa39d432a9b472845876bfaf0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260129 02:59:04.337667 21394 raft_consensus.cc:3055] T a609ac02fae84725ab743f95f51d33ce P 197b880fa39d432a9b472845876bfaf0 [term 1 LEADER]: Stepping down as leader of term 1
I20260129 02:59:04.337986 21394 raft_consensus.cc:740] T a609ac02fae84725ab743f95f51d33ce P 197b880fa39d432a9b472845876bfaf0 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 197b880fa39d432a9b472845876bfaf0, State: Running, Role: LEADER
I20260129 02:59:04.338673 20441 consensus_queue.cc:237] T 0678b8b468bd43cdbc3e485900844821 P 197b880fa39d432a9b472845876bfaf0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } attrs { replace: true } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: NON_VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: true } }
I20260129 02:59:04.338769 21394 consensus_queue.cc:260] T a609ac02fae84725ab743f95f51d33ce P 197b880fa39d432a9b472845876bfaf0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } }
I20260129 02:59:04.340684 21394 raft_consensus.cc:3060] T a609ac02fae84725ab743f95f51d33ce P 197b880fa39d432a9b472845876bfaf0 [term 1 FOLLOWER]: Advancing to term 2
I20260129 02:59:04.342623 21398 raft_consensus.cc:2749] T a609ac02fae84725ab743f95f51d33ce P 974b802141bd4f13beffef2aa03f88f9 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260129 02:59:04.344461 20435 raft_consensus.cc:1240] T 4f8f3a38f6e24ba4857875181f447442 P 197b880fa39d432a9b472845876bfaf0 [term 2 FOLLOWER]: Rejecting Update request from peer 974b802141bd4f13beffef2aa03f88f9 for earlier term 1. Current term is 2. Ops: []
W20260129 02:59:04.350919 20405 consensus_peers.cc:597] T 0678b8b468bd43cdbc3e485900844821 P 197b880fa39d432a9b472845876bfaf0 -> Peer 7a9daf5d42a84615a6383cfec9a10608 (127.18.211.4:40439): Couldn't send request to peer 7a9daf5d42a84615a6383cfec9a10608. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 0678b8b468bd43cdbc3e485900844821. This is attempt 1: this message will repeat every 5th retry.
I20260129 02:59:04.339251 20338 catalog_manager.cc:5654] T 78aa39096868415d9afc2ffcb248e198 P 974b802141bd4f13beffef2aa03f88f9 reported cstate change: config changed from index -1 to 2, NON_VOTER 7a9daf5d42a84615a6383cfec9a10608 (127.18.211.4) added. New cstate: current_term: 1 leader_uuid: "1c50de89a5a845668696dcb28c3badd6" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } attrs { replace: true } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: NON_VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: true } } }
I20260129 02:59:04.353834 21397 consensus_queue.cc:1059] T 4f8f3a38f6e24ba4857875181f447442 P 974b802141bd4f13beffef2aa03f88f9 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260129 02:59:04.357191 21436 raft_consensus.cc:3055] T 4f8f3a38f6e24ba4857875181f447442 P 974b802141bd4f13beffef2aa03f88f9 [term 1 LEADER]: Stepping down as leader of term 1
I20260129 02:59:04.356482 20438 raft_consensus.cc:2955] T 78aa39096868415d9afc2ffcb248e198 P 197b880fa39d432a9b472845876bfaf0 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 7a9daf5d42a84615a6383cfec9a10608 (127.18.211.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } attrs { replace: true } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: NON_VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: true } } }
I20260129 02:59:04.357578 21436 raft_consensus.cc:740] T 4f8f3a38f6e24ba4857875181f447442 P 974b802141bd4f13beffef2aa03f88f9 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 974b802141bd4f13beffef2aa03f88f9, State: Running, Role: LEADER
I20260129 02:59:04.359189 21436 consensus_queue.cc:260] T 4f8f3a38f6e24ba4857875181f447442 P 974b802141bd4f13beffef2aa03f88f9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:59:04.360877 21436 raft_consensus.cc:3060] T 4f8f3a38f6e24ba4857875181f447442 P 974b802141bd4f13beffef2aa03f88f9 [term 1 FOLLOWER]: Advancing to term 2
I20260129 02:59:04.362262 20507 raft_consensus.cc:1275] T 0678b8b468bd43cdbc3e485900844821 P 974b802141bd4f13beffef2aa03f88f9 [term 1 FOLLOWER]: Refusing update from remote peer 197b880fa39d432a9b472845876bfaf0: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260129 02:59:04.363835 21396 consensus_queue.cc:1048] T 0678b8b468bd43cdbc3e485900844821 P 197b880fa39d432a9b472845876bfaf0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260129 02:59:04.372828 20587 raft_consensus.cc:1275] T 0678b8b468bd43cdbc3e485900844821 P 1c50de89a5a845668696dcb28c3badd6 [term 1 FOLLOWER]: Refusing update from remote peer 197b880fa39d432a9b472845876bfaf0: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260129 02:59:04.377450 21416 consensus_queue.cc:1048] T 0678b8b468bd43cdbc3e485900844821 P 197b880fa39d432a9b472845876bfaf0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260129 02:59:04.393195 20507 raft_consensus.cc:1240] T 75be4bd6bd45474795fd472eff2953ac P 974b802141bd4f13beffef2aa03f88f9 [term 2 FOLLOWER]: Rejecting Update request from peer 197b880fa39d432a9b472845876bfaf0 for earlier term 1. Current term is 2. Ops: []
I20260129 02:59:04.394634 21416 consensus_queue.cc:1059] T 75be4bd6bd45474795fd472eff2953ac P 197b880fa39d432a9b472845876bfaf0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260129 02:59:04.396275 21406 raft_consensus.cc:3055] T 75be4bd6bd45474795fd472eff2953ac P 197b880fa39d432a9b472845876bfaf0 [term 1 LEADER]: Stepping down as leader of term 1
I20260129 02:59:04.396626 21406 raft_consensus.cc:740] T 75be4bd6bd45474795fd472eff2953ac P 197b880fa39d432a9b472845876bfaf0 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 197b880fa39d432a9b472845876bfaf0, State: Running, Role: LEADER
I20260129 02:59:04.397387 21406 consensus_queue.cc:260] T 75be4bd6bd45474795fd472eff2953ac P 197b880fa39d432a9b472845876bfaf0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:59:04.398473 21406 raft_consensus.cc:3060] T 75be4bd6bd45474795fd472eff2953ac P 197b880fa39d432a9b472845876bfaf0 [term 1 FOLLOWER]: Advancing to term 2
I20260129 02:59:04.405383 21394 raft_consensus.cc:2955] T 0678b8b468bd43cdbc3e485900844821 P 197b880fa39d432a9b472845876bfaf0 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 7a9daf5d42a84615a6383cfec9a10608 (127.18.211.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } attrs { replace: true } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: NON_VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: true } } }
I20260129 02:59:04.411105 20587 raft_consensus.cc:2955] T 0678b8b468bd43cdbc3e485900844821 P 1c50de89a5a845668696dcb28c3badd6 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 7a9daf5d42a84615a6383cfec9a10608 (127.18.211.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } attrs { replace: true } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: NON_VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: true } } }
I20260129 02:59:04.420341 20507 raft_consensus.cc:2955] T 0678b8b468bd43cdbc3e485900844821 P 974b802141bd4f13beffef2aa03f88f9 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 7a9daf5d42a84615a6383cfec9a10608 (127.18.211.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } attrs { replace: true } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: NON_VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: true } } }
I20260129 02:59:04.438381 20510 consensus_queue.cc:237] T 75f45538dc704d9a8ec4ff8ad9f3ea5b P 974b802141bd4f13beffef2aa03f88f9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } attrs { replace: true } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: NON_VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: true } }
I20260129 02:59:04.462114 20508 raft_consensus.cc:1240] T c891d2bff4ce4568861bbc992e91fbb0 P 974b802141bd4f13beffef2aa03f88f9 [term 2 FOLLOWER]: Rejecting Update request from peer 197b880fa39d432a9b472845876bfaf0 for earlier term 1. Current term is 2. Ops: []
W20260129 02:59:04.464706 20480 consensus_peers.cc:597] T 75f45538dc704d9a8ec4ff8ad9f3ea5b P 974b802141bd4f13beffef2aa03f88f9 -> Peer 7a9daf5d42a84615a6383cfec9a10608 (127.18.211.4:40439): Couldn't send request to peer 7a9daf5d42a84615a6383cfec9a10608. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 75f45538dc704d9a8ec4ff8ad9f3ea5b. This is attempt 1: this message will repeat every 5th retry.
I20260129 02:59:04.465965 21386 consensus_queue.cc:1059] T c891d2bff4ce4568861bbc992e91fbb0 P 197b880fa39d432a9b472845876bfaf0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260129 02:59:04.466313 20582 raft_consensus.cc:1275] T 75f45538dc704d9a8ec4ff8ad9f3ea5b P 1c50de89a5a845668696dcb28c3badd6 [term 1 FOLLOWER]: Refusing update from remote peer 974b802141bd4f13beffef2aa03f88f9: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260129 02:59:04.467031 21386 raft_consensus.cc:3055] T c891d2bff4ce4568861bbc992e91fbb0 P 197b880fa39d432a9b472845876bfaf0 [term 1 LEADER]: Stepping down as leader of term 1
I20260129 02:59:04.467398 21386 raft_consensus.cc:740] T c891d2bff4ce4568861bbc992e91fbb0 P 197b880fa39d432a9b472845876bfaf0 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 197b880fa39d432a9b472845876bfaf0, State: Running, Role: LEADER
I20260129 02:59:04.468710 21436 consensus_queue.cc:1048] T 75f45538dc704d9a8ec4ff8ad9f3ea5b P 974b802141bd4f13beffef2aa03f88f9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260129 02:59:04.468248 21386 consensus_queue.cc:260] T c891d2bff4ce4568861bbc992e91fbb0 P 197b880fa39d432a9b472845876bfaf0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:59:04.472257 21386 raft_consensus.cc:3060] T c891d2bff4ce4568861bbc992e91fbb0 P 197b880fa39d432a9b472845876bfaf0 [term 1 FOLLOWER]: Advancing to term 2
I20260129 02:59:04.482101 20441 raft_consensus.cc:1275] T 75f45538dc704d9a8ec4ff8ad9f3ea5b P 197b880fa39d432a9b472845876bfaf0 [term 1 FOLLOWER]: Refusing update from remote peer 974b802141bd4f13beffef2aa03f88f9: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260129 02:59:04.483525 21397 consensus_queue.cc:1048] T 75f45538dc704d9a8ec4ff8ad9f3ea5b P 974b802141bd4f13beffef2aa03f88f9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260129 02:59:04.485673 21449 ts_tablet_manager.cc:933] T 78aa39096868415d9afc2ffcb248e198 P 7a9daf5d42a84615a6383cfec9a10608: Initiating tablet copy from peer 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3:37735)
I20260129 02:59:04.495121 21397 raft_consensus.cc:2955] T 75f45538dc704d9a8ec4ff8ad9f3ea5b P 974b802141bd4f13beffef2aa03f88f9 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 7a9daf5d42a84615a6383cfec9a10608 (127.18.211.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } attrs { replace: true } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: NON_VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: true } } }
I20260129 02:59:04.501976 20441 raft_consensus.cc:2955] T 75f45538dc704d9a8ec4ff8ad9f3ea5b P 197b880fa39d432a9b472845876bfaf0 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 7a9daf5d42a84615a6383cfec9a10608 (127.18.211.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } attrs { replace: true } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: NON_VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: true } } }
I20260129 02:59:04.518033 20585 raft_consensus.cc:2955] T 75f45538dc704d9a8ec4ff8ad9f3ea5b P 1c50de89a5a845668696dcb28c3badd6 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 7a9daf5d42a84615a6383cfec9a10608 (127.18.211.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } attrs { replace: true } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: NON_VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: true } } }
W20260129 02:59:04.536881 20387 auto_rebalancer.cc:256] failed to send replica move request: Illegal state: Replica 974b802141bd4f13beffef2aa03f88f9 is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 2 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } } }
I20260129 02:59:04.539799 20334 catalog_manager.cc:5654] T 0678b8b468bd43cdbc3e485900844821 P 974b802141bd4f13beffef2aa03f88f9 reported cstate change: config changed from index -1 to 2, NON_VOTER 7a9daf5d42a84615a6383cfec9a10608 (127.18.211.4) added. New cstate: current_term: 1 leader_uuid: "197b880fa39d432a9b472845876bfaf0" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } attrs { replace: true } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: NON_VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: true } } }
I20260129 02:59:04.564668 21449 tablet_copy_client.cc:323] T 78aa39096868415d9afc2ffcb248e198 P 7a9daf5d42a84615a6383cfec9a10608: tablet copy: Beginning tablet copy session from remote peer at address 127.18.211.3:37735
I20260129 02:59:04.566499 20600 tablet_copy_service.cc:140] P 1c50de89a5a845668696dcb28c3badd6: Received BeginTabletCopySession request for tablet 78aa39096868415d9afc2ffcb248e198 from peer 7a9daf5d42a84615a6383cfec9a10608 ({username='slave'} at 127.0.0.1:48526)
I20260129 02:59:04.567168 20600 tablet_copy_service.cc:161] P 1c50de89a5a845668696dcb28c3badd6: Beginning new tablet copy session on tablet 78aa39096868415d9afc2ffcb248e198 from peer 7a9daf5d42a84615a6383cfec9a10608 at {username='slave'} at 127.0.0.1:48526: session id = 7a9daf5d42a84615a6383cfec9a10608-78aa39096868415d9afc2ffcb248e198
W20260129 02:59:04.588100 20387 auto_rebalancer.cc:726] Could not move replica: Incomplete: tablet 4f8f3a38f6e24ba4857875181f447442, TS 1c50de89a5a845668696dcb28c3badd6 -> TS 7a9daf5d42a84615a6383cfec9a10608 move failed, destination replica disappeared from tablet's Raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
W20260129 02:59:04.588537 20387 auto_rebalancer.cc:266] scheduled replica move failed to complete: Incomplete: tablet 4f8f3a38f6e24ba4857875181f447442, TS 1c50de89a5a845668696dcb28c3badd6 -> TS 7a9daf5d42a84615a6383cfec9a10608 move failed, destination replica disappeared from tablet's Raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:59:04.593662 20600 tablet_copy_source_session.cc:215] T 78aa39096868415d9afc2ffcb248e198 P 1c50de89a5a845668696dcb28c3badd6: Tablet Copy: opened 0 blocks and 1 log segments
I20260129 02:59:04.584159 20335 catalog_manager.cc:5654] T 75f45538dc704d9a8ec4ff8ad9f3ea5b P 197b880fa39d432a9b472845876bfaf0 reported cstate change: config changed from index -1 to 2, NON_VOTER 7a9daf5d42a84615a6383cfec9a10608 (127.18.211.4) added. New cstate: current_term: 1 leader_uuid: "974b802141bd4f13beffef2aa03f88f9" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } attrs { replace: true } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: NON_VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: true } } }
I20260129 02:59:04.598048 21449 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 78aa39096868415d9afc2ffcb248e198. 1 dirs total, 0 dirs full, 0 dirs failed
I20260129 02:59:04.636158 21449 tablet_copy_client.cc:806] T 78aa39096868415d9afc2ffcb248e198 P 7a9daf5d42a84615a6383cfec9a10608: tablet copy: Starting download of 0 data blocks...
I20260129 02:59:04.638533 21449 tablet_copy_client.cc:670] T 78aa39096868415d9afc2ffcb248e198 P 7a9daf5d42a84615a6383cfec9a10608: tablet copy: Starting download of 1 WAL segments...
I20260129 02:59:04.643272 21449 tablet_copy_client.cc:538] T 78aa39096868415d9afc2ffcb248e198 P 7a9daf5d42a84615a6383cfec9a10608: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260129 02:59:04.653373 21449 tablet_bootstrap.cc:492] T 78aa39096868415d9afc2ffcb248e198 P 7a9daf5d42a84615a6383cfec9a10608: Bootstrap starting.
I20260129 02:59:04.675993 21449 tablet_bootstrap.cc:492] T 78aa39096868415d9afc2ffcb248e198 P 7a9daf5d42a84615a6383cfec9a10608: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260129 02:59:04.676890 21449 tablet_bootstrap.cc:492] T 78aa39096868415d9afc2ffcb248e198 P 7a9daf5d42a84615a6383cfec9a10608: Bootstrap complete.
I20260129 02:59:04.677609 21449 ts_tablet_manager.cc:1403] T 78aa39096868415d9afc2ffcb248e198 P 7a9daf5d42a84615a6383cfec9a10608: Time spent bootstrapping tablet: real 0.024s user 0.015s sys 0.007s
I20260129 02:59:04.680577 21449 raft_consensus.cc:359] T 78aa39096868415d9afc2ffcb248e198 P 7a9daf5d42a84615a6383cfec9a10608 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } attrs { replace: true } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: NON_VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: true } }
I20260129 02:59:04.681389 21449 raft_consensus.cc:740] T 78aa39096868415d9afc2ffcb248e198 P 7a9daf5d42a84615a6383cfec9a10608 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 7a9daf5d42a84615a6383cfec9a10608, State: Initialized, Role: LEARNER
I20260129 02:59:04.681996 21449 consensus_queue.cc:260] T 78aa39096868415d9afc2ffcb248e198 P 7a9daf5d42a84615a6383cfec9a10608 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } attrs { replace: true } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: NON_VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: true } }
I20260129 02:59:04.686436 21449 ts_tablet_manager.cc:1434] T 78aa39096868415d9afc2ffcb248e198 P 7a9daf5d42a84615a6383cfec9a10608: Time spent starting tablet: real 0.009s user 0.002s sys 0.005s
I20260129 02:59:04.691516 20600 tablet_copy_service.cc:342] P 1c50de89a5a845668696dcb28c3badd6: Request end of tablet copy session 7a9daf5d42a84615a6383cfec9a10608-78aa39096868415d9afc2ffcb248e198 received from {username='slave'} at 127.0.0.1:48526
I20260129 02:59:04.692123 20600 tablet_copy_service.cc:434] P 1c50de89a5a845668696dcb28c3badd6: ending tablet copy session 7a9daf5d42a84615a6383cfec9a10608-78aa39096868415d9afc2ffcb248e198 on tablet 78aa39096868415d9afc2ffcb248e198 with peer 7a9daf5d42a84615a6383cfec9a10608
I20260129 02:59:04.750998 21449 ts_tablet_manager.cc:933] T 0678b8b468bd43cdbc3e485900844821 P 7a9daf5d42a84615a6383cfec9a10608: Initiating tablet copy from peer 197b880fa39d432a9b472845876bfaf0 (127.18.211.1:42027)
I20260129 02:59:04.757987 21449 tablet_copy_client.cc:323] T 0678b8b468bd43cdbc3e485900844821 P 7a9daf5d42a84615a6383cfec9a10608: tablet copy: Beginning tablet copy session from remote peer at address 127.18.211.1:42027
I20260129 02:59:04.759549 20450 tablet_copy_service.cc:140] P 197b880fa39d432a9b472845876bfaf0: Received BeginTabletCopySession request for tablet 0678b8b468bd43cdbc3e485900844821 from peer 7a9daf5d42a84615a6383cfec9a10608 ({username='slave'} at 127.0.0.1:48356)
I20260129 02:59:04.760159 20450 tablet_copy_service.cc:161] P 197b880fa39d432a9b472845876bfaf0: Beginning new tablet copy session on tablet 0678b8b468bd43cdbc3e485900844821 from peer 7a9daf5d42a84615a6383cfec9a10608 at {username='slave'} at 127.0.0.1:48356: session id = 7a9daf5d42a84615a6383cfec9a10608-0678b8b468bd43cdbc3e485900844821
I20260129 02:59:04.767472 20450 tablet_copy_source_session.cc:215] T 0678b8b468bd43cdbc3e485900844821 P 197b880fa39d432a9b472845876bfaf0: Tablet Copy: opened 0 blocks and 1 log segments
I20260129 02:59:04.772007 21449 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0678b8b468bd43cdbc3e485900844821. 1 dirs total, 0 dirs full, 0 dirs failed
I20260129 02:59:04.793486 21449 tablet_copy_client.cc:806] T 0678b8b468bd43cdbc3e485900844821 P 7a9daf5d42a84615a6383cfec9a10608: tablet copy: Starting download of 0 data blocks...
I20260129 02:59:04.794152 21449 tablet_copy_client.cc:670] T 0678b8b468bd43cdbc3e485900844821 P 7a9daf5d42a84615a6383cfec9a10608: tablet copy: Starting download of 1 WAL segments...
I20260129 02:59:04.801963 21449 tablet_copy_client.cc:538] T 0678b8b468bd43cdbc3e485900844821 P 7a9daf5d42a84615a6383cfec9a10608: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260129 02:59:04.814539 21449 tablet_bootstrap.cc:492] T 0678b8b468bd43cdbc3e485900844821 P 7a9daf5d42a84615a6383cfec9a10608: Bootstrap starting.
I20260129 02:59:04.841890 21449 tablet_bootstrap.cc:492] T 0678b8b468bd43cdbc3e485900844821 P 7a9daf5d42a84615a6383cfec9a10608: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260129 02:59:04.842757 21449 tablet_bootstrap.cc:492] T 0678b8b468bd43cdbc3e485900844821 P 7a9daf5d42a84615a6383cfec9a10608: Bootstrap complete.
I20260129 02:59:04.843431 21449 ts_tablet_manager.cc:1403] T 0678b8b468bd43cdbc3e485900844821 P 7a9daf5d42a84615a6383cfec9a10608: Time spent bootstrapping tablet: real 0.032s user 0.020s sys 0.004s
I20260129 02:59:04.846135 21449 raft_consensus.cc:359] T 0678b8b468bd43cdbc3e485900844821 P 7a9daf5d42a84615a6383cfec9a10608 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } attrs { replace: true } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: NON_VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: true } }
I20260129 02:59:04.846902 21449 raft_consensus.cc:740] T 0678b8b468bd43cdbc3e485900844821 P 7a9daf5d42a84615a6383cfec9a10608 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 7a9daf5d42a84615a6383cfec9a10608, State: Initialized, Role: LEARNER
I20260129 02:59:04.847518 21449 consensus_queue.cc:260] T 0678b8b468bd43cdbc3e485900844821 P 7a9daf5d42a84615a6383cfec9a10608 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } attrs { replace: true } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: NON_VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: true } }
I20260129 02:59:04.849634 21449 ts_tablet_manager.cc:1434] T 0678b8b468bd43cdbc3e485900844821 P 7a9daf5d42a84615a6383cfec9a10608: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260129 02:59:04.851104 20450 tablet_copy_service.cc:342] P 197b880fa39d432a9b472845876bfaf0: Request end of tablet copy session 7a9daf5d42a84615a6383cfec9a10608-0678b8b468bd43cdbc3e485900844821 received from {username='slave'} at 127.0.0.1:48356
I20260129 02:59:04.851522 20450 tablet_copy_service.cc:434] P 197b880fa39d432a9b472845876bfaf0: ending tablet copy session 7a9daf5d42a84615a6383cfec9a10608-0678b8b468bd43cdbc3e485900844821 on tablet 0678b8b468bd43cdbc3e485900844821 with peer 7a9daf5d42a84615a6383cfec9a10608
I20260129 02:59:04.923364 21449 ts_tablet_manager.cc:933] T 75f45538dc704d9a8ec4ff8ad9f3ea5b P 7a9daf5d42a84615a6383cfec9a10608: Initiating tablet copy from peer 974b802141bd4f13beffef2aa03f88f9 (127.18.211.2:34517)
I20260129 02:59:04.934139 21449 tablet_copy_client.cc:323] T 75f45538dc704d9a8ec4ff8ad9f3ea5b P 7a9daf5d42a84615a6383cfec9a10608: tablet copy: Beginning tablet copy session from remote peer at address 127.18.211.2:34517
I20260129 02:59:04.936090 20526 tablet_copy_service.cc:140] P 974b802141bd4f13beffef2aa03f88f9: Received BeginTabletCopySession request for tablet 75f45538dc704d9a8ec4ff8ad9f3ea5b from peer 7a9daf5d42a84615a6383cfec9a10608 ({username='slave'} at 127.0.0.1:48544)
I20260129 02:59:04.936690 20526 tablet_copy_service.cc:161] P 974b802141bd4f13beffef2aa03f88f9: Beginning new tablet copy session on tablet 75f45538dc704d9a8ec4ff8ad9f3ea5b from peer 7a9daf5d42a84615a6383cfec9a10608 at {username='slave'} at 127.0.0.1:48544: session id = 7a9daf5d42a84615a6383cfec9a10608-75f45538dc704d9a8ec4ff8ad9f3ea5b
I20260129 02:59:04.947043 20526 tablet_copy_source_session.cc:215] T 75f45538dc704d9a8ec4ff8ad9f3ea5b P 974b802141bd4f13beffef2aa03f88f9: Tablet Copy: opened 0 blocks and 1 log segments
I20260129 02:59:04.950444 21449 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 75f45538dc704d9a8ec4ff8ad9f3ea5b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260129 02:59:04.965811 21131 raft_consensus.cc:1217] T 78aa39096868415d9afc2ffcb248e198 P 7a9daf5d42a84615a6383cfec9a10608 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20260129 02:59:04.973467 21449 tablet_copy_client.cc:806] T 75f45538dc704d9a8ec4ff8ad9f3ea5b P 7a9daf5d42a84615a6383cfec9a10608: tablet copy: Starting download of 0 data blocks...
I20260129 02:59:04.974689 21449 tablet_copy_client.cc:670] T 75f45538dc704d9a8ec4ff8ad9f3ea5b P 7a9daf5d42a84615a6383cfec9a10608: tablet copy: Starting download of 1 WAL segments...
I20260129 02:59:04.978785 21449 tablet_copy_client.cc:538] T 75f45538dc704d9a8ec4ff8ad9f3ea5b P 7a9daf5d42a84615a6383cfec9a10608: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260129 02:59:04.987636 21449 tablet_bootstrap.cc:492] T 75f45538dc704d9a8ec4ff8ad9f3ea5b P 7a9daf5d42a84615a6383cfec9a10608: Bootstrap starting.
I20260129 02:59:05.018484 21449 tablet_bootstrap.cc:492] T 75f45538dc704d9a8ec4ff8ad9f3ea5b P 7a9daf5d42a84615a6383cfec9a10608: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260129 02:59:05.019413 21449 tablet_bootstrap.cc:492] T 75f45538dc704d9a8ec4ff8ad9f3ea5b P 7a9daf5d42a84615a6383cfec9a10608: Bootstrap complete.
I20260129 02:59:05.020118 21449 ts_tablet_manager.cc:1403] T 75f45538dc704d9a8ec4ff8ad9f3ea5b P 7a9daf5d42a84615a6383cfec9a10608: Time spent bootstrapping tablet: real 0.033s user 0.024s sys 0.011s
I20260129 02:59:05.022792 21449 raft_consensus.cc:359] T 75f45538dc704d9a8ec4ff8ad9f3ea5b P 7a9daf5d42a84615a6383cfec9a10608 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } attrs { replace: true } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: NON_VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: true } }
I20260129 02:59:05.023682 21449 raft_consensus.cc:740] T 75f45538dc704d9a8ec4ff8ad9f3ea5b P 7a9daf5d42a84615a6383cfec9a10608 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 7a9daf5d42a84615a6383cfec9a10608, State: Initialized, Role: LEARNER
I20260129 02:59:05.024303 21449 consensus_queue.cc:260] T 75f45538dc704d9a8ec4ff8ad9f3ea5b P 7a9daf5d42a84615a6383cfec9a10608 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } attrs { replace: true } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: NON_VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: true } }
I20260129 02:59:05.033141 21449 ts_tablet_manager.cc:1434] T 75f45538dc704d9a8ec4ff8ad9f3ea5b P 7a9daf5d42a84615a6383cfec9a10608: Time spent starting tablet: real 0.013s user 0.005s sys 0.001s
I20260129 02:59:05.034864 20526 tablet_copy_service.cc:342] P 974b802141bd4f13beffef2aa03f88f9: Request end of tablet copy session 7a9daf5d42a84615a6383cfec9a10608-75f45538dc704d9a8ec4ff8ad9f3ea5b received from {username='slave'} at 127.0.0.1:48544
I20260129 02:59:05.035300 20526 tablet_copy_service.cc:434] P 974b802141bd4f13beffef2aa03f88f9: ending tablet copy session 7a9daf5d42a84615a6383cfec9a10608-75f45538dc704d9a8ec4ff8ad9f3ea5b on tablet 75f45538dc704d9a8ec4ff8ad9f3ea5b with peer 7a9daf5d42a84615a6383cfec9a10608
I20260129 02:59:05.226480 21131 raft_consensus.cc:1217] T 0678b8b468bd43cdbc3e485900844821 P 7a9daf5d42a84615a6383cfec9a10608 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20260129 02:59:05.409842 21131 raft_consensus.cc:1217] T 75f45538dc704d9a8ec4ff8ad9f3ea5b P 7a9daf5d42a84615a6383cfec9a10608 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20260129 02:59:05.423213 21426 raft_consensus.cc:493] T 926a250fa5834e88b96ccecbefbf5086 P 1c50de89a5a845668696dcb28c3badd6 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 197b880fa39d432a9b472845876bfaf0)
I20260129 02:59:05.423844 21426 raft_consensus.cc:515] T 926a250fa5834e88b96ccecbefbf5086 P 1c50de89a5a845668696dcb28c3badd6 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:59:05.425594 21426 leader_election.cc:290] T 926a250fa5834e88b96ccecbefbf5086 P 1c50de89a5a845668696dcb28c3badd6 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 197b880fa39d432a9b472845876bfaf0 (127.18.211.1:42027), 974b802141bd4f13beffef2aa03f88f9 (127.18.211.2:34517)
I20260129 02:59:05.426469 21403 raft_consensus.cc:493] T e7d83d4ccf684e76acb29a0dc844e1b8 P 1c50de89a5a845668696dcb28c3badd6 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 197b880fa39d432a9b472845876bfaf0)
I20260129 02:59:05.427331 21403 raft_consensus.cc:515] T e7d83d4ccf684e76acb29a0dc844e1b8 P 1c50de89a5a845668696dcb28c3badd6 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:59:05.429427 21403 leader_election.cc:290] T e7d83d4ccf684e76acb29a0dc844e1b8 P 1c50de89a5a845668696dcb28c3badd6 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 197b880fa39d432a9b472845876bfaf0 (127.18.211.1:42027), 974b802141bd4f13beffef2aa03f88f9 (127.18.211.2:34517)
I20260129 02:59:05.426656 20438 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "926a250fa5834e88b96ccecbefbf5086" candidate_uuid: "1c50de89a5a845668696dcb28c3badd6" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "197b880fa39d432a9b472845876bfaf0" is_pre_election: true
I20260129 02:59:05.430289 20507 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e7d83d4ccf684e76acb29a0dc844e1b8" candidate_uuid: "1c50de89a5a845668696dcb28c3badd6" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "974b802141bd4f13beffef2aa03f88f9" is_pre_election: true
I20260129 02:59:05.430760 20438 raft_consensus.cc:2468] T 926a250fa5834e88b96ccecbefbf5086 P 197b880fa39d432a9b472845876bfaf0 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1c50de89a5a845668696dcb28c3badd6 in term 2.
I20260129 02:59:05.431128 20507 raft_consensus.cc:2393] T e7d83d4ccf684e76acb29a0dc844e1b8 P 974b802141bd4f13beffef2aa03f88f9 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1c50de89a5a845668696dcb28c3badd6 in current term 2: Already voted for candidate 974b802141bd4f13beffef2aa03f88f9 in this term.
I20260129 02:59:05.431419 20441 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e7d83d4ccf684e76acb29a0dc844e1b8" candidate_uuid: "1c50de89a5a845668696dcb28c3badd6" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "197b880fa39d432a9b472845876bfaf0" is_pre_election: true
I20260129 02:59:05.432039 20441 raft_consensus.cc:2468] T e7d83d4ccf684e76acb29a0dc844e1b8 P 197b880fa39d432a9b472845876bfaf0 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1c50de89a5a845668696dcb28c3badd6 in term 2.
I20260129 02:59:05.433362 20553 leader_election.cc:304] T 926a250fa5834e88b96ccecbefbf5086 P 1c50de89a5a845668696dcb28c3badd6 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 197b880fa39d432a9b472845876bfaf0, 1c50de89a5a845668696dcb28c3badd6; no voters:
I20260129 02:59:05.434142 20553 leader_election.cc:304] T e7d83d4ccf684e76acb29a0dc844e1b8 P 1c50de89a5a845668696dcb28c3badd6 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 197b880fa39d432a9b472845876bfaf0, 1c50de89a5a845668696dcb28c3badd6; no voters: 974b802141bd4f13beffef2aa03f88f9
I20260129 02:59:05.434777 21426 raft_consensus.cc:2804] T 926a250fa5834e88b96ccecbefbf5086 P 1c50de89a5a845668696dcb28c3badd6 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260129 02:59:05.435195 21426 raft_consensus.cc:493] T 926a250fa5834e88b96ccecbefbf5086 P 1c50de89a5a845668696dcb28c3badd6 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 197b880fa39d432a9b472845876bfaf0)
I20260129 02:59:05.436930 21426 raft_consensus.cc:3060] T 926a250fa5834e88b96ccecbefbf5086 P 1c50de89a5a845668696dcb28c3badd6 [term 1 FOLLOWER]: Advancing to term 2
I20260129 02:59:05.435217 21403 raft_consensus.cc:2804] T e7d83d4ccf684e76acb29a0dc844e1b8 P 1c50de89a5a845668696dcb28c3badd6 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260129 02:59:05.437819 20508 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "926a250fa5834e88b96ccecbefbf5086" candidate_uuid: "1c50de89a5a845668696dcb28c3badd6" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "974b802141bd4f13beffef2aa03f88f9" is_pre_election: true
I20260129 02:59:05.437964 21403 raft_consensus.cc:493] T e7d83d4ccf684e76acb29a0dc844e1b8 P 1c50de89a5a845668696dcb28c3badd6 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 197b880fa39d432a9b472845876bfaf0)
I20260129 02:59:05.438772 21403 raft_consensus.cc:3060] T e7d83d4ccf684e76acb29a0dc844e1b8 P 1c50de89a5a845668696dcb28c3badd6 [term 1 FOLLOWER]: Advancing to term 2
I20260129 02:59:05.440300 20508 raft_consensus.cc:2393] T 926a250fa5834e88b96ccecbefbf5086 P 974b802141bd4f13beffef2aa03f88f9 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1c50de89a5a845668696dcb28c3badd6 in current term 2: Already voted for candidate 974b802141bd4f13beffef2aa03f88f9 in this term.
I20260129 02:59:05.451319 21426 raft_consensus.cc:515] T 926a250fa5834e88b96ccecbefbf5086 P 1c50de89a5a845668696dcb28c3badd6 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:59:05.454131 20438 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "926a250fa5834e88b96ccecbefbf5086" candidate_uuid: "1c50de89a5a845668696dcb28c3badd6" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "197b880fa39d432a9b472845876bfaf0"
I20260129 02:59:05.454752 20508 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "926a250fa5834e88b96ccecbefbf5086" candidate_uuid: "1c50de89a5a845668696dcb28c3badd6" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "974b802141bd4f13beffef2aa03f88f9"
I20260129 02:59:05.455188 21403 raft_consensus.cc:515] T e7d83d4ccf684e76acb29a0dc844e1b8 P 1c50de89a5a845668696dcb28c3badd6 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:59:05.455654 20508 raft_consensus.cc:2393] T 926a250fa5834e88b96ccecbefbf5086 P 974b802141bd4f13beffef2aa03f88f9 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 1c50de89a5a845668696dcb28c3badd6 in current term 2: Already voted for candidate 974b802141bd4f13beffef2aa03f88f9 in this term.
I20260129 02:59:05.457024 21426 leader_election.cc:290] T 926a250fa5834e88b96ccecbefbf5086 P 1c50de89a5a845668696dcb28c3badd6 [CANDIDATE]: Term 2 election: Requested vote from peers 197b880fa39d432a9b472845876bfaf0 (127.18.211.1:42027), 974b802141bd4f13beffef2aa03f88f9 (127.18.211.2:34517)
I20260129 02:59:05.457789 21403 leader_election.cc:290] T e7d83d4ccf684e76acb29a0dc844e1b8 P 1c50de89a5a845668696dcb28c3badd6 [CANDIDATE]: Term 2 election: Requested vote from peers 197b880fa39d432a9b472845876bfaf0 (127.18.211.1:42027), 974b802141bd4f13beffef2aa03f88f9 (127.18.211.2:34517)
I20260129 02:59:05.458406 20508 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e7d83d4ccf684e76acb29a0dc844e1b8" candidate_uuid: "1c50de89a5a845668696dcb28c3badd6" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "974b802141bd4f13beffef2aa03f88f9"
I20260129 02:59:05.459357 20508 raft_consensus.cc:2393] T e7d83d4ccf684e76acb29a0dc844e1b8 P 974b802141bd4f13beffef2aa03f88f9 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 1c50de89a5a845668696dcb28c3badd6 in current term 2: Already voted for candidate 974b802141bd4f13beffef2aa03f88f9 in this term.
I20260129 02:59:05.468900 20435 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e7d83d4ccf684e76acb29a0dc844e1b8" candidate_uuid: "1c50de89a5a845668696dcb28c3badd6" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "197b880fa39d432a9b472845876bfaf0"
I20260129 02:59:05.469278 20438 raft_consensus.cc:2468] T 926a250fa5834e88b96ccecbefbf5086 P 197b880fa39d432a9b472845876bfaf0 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1c50de89a5a845668696dcb28c3badd6 in term 2.
I20260129 02:59:05.472242 20553 leader_election.cc:304] T 926a250fa5834e88b96ccecbefbf5086 P 1c50de89a5a845668696dcb28c3badd6 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 197b880fa39d432a9b472845876bfaf0, 1c50de89a5a845668696dcb28c3badd6; no voters: 974b802141bd4f13beffef2aa03f88f9
I20260129 02:59:05.476969 20435 raft_consensus.cc:2468] T e7d83d4ccf684e76acb29a0dc844e1b8 P 197b880fa39d432a9b472845876bfaf0 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1c50de89a5a845668696dcb28c3badd6 in term 2.
I20260129 02:59:05.477849 21403 raft_consensus.cc:2804] T 926a250fa5834e88b96ccecbefbf5086 P 1c50de89a5a845668696dcb28c3badd6 [term 2 FOLLOWER]: Leader election won for term 2
I20260129 02:59:05.477967 20553 leader_election.cc:304] T e7d83d4ccf684e76acb29a0dc844e1b8 P 1c50de89a5a845668696dcb28c3badd6 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 197b880fa39d432a9b472845876bfaf0, 1c50de89a5a845668696dcb28c3badd6; no voters: 974b802141bd4f13beffef2aa03f88f9
I20260129 02:59:05.478574 21403 raft_consensus.cc:697] T 926a250fa5834e88b96ccecbefbf5086 P 1c50de89a5a845668696dcb28c3badd6 [term 2 LEADER]: Becoming Leader. State: Replica: 1c50de89a5a845668696dcb28c3badd6, State: Running, Role: LEADER
I20260129 02:59:05.478991 21429 raft_consensus.cc:2804] T e7d83d4ccf684e76acb29a0dc844e1b8 P 1c50de89a5a845668696dcb28c3badd6 [term 2 FOLLOWER]: Leader election won for term 2
I20260129 02:59:05.479420 21429 raft_consensus.cc:697] T e7d83d4ccf684e76acb29a0dc844e1b8 P 1c50de89a5a845668696dcb28c3badd6 [term 2 LEADER]: Becoming Leader. State: Replica: 1c50de89a5a845668696dcb28c3badd6, State: Running, Role: LEADER
I20260129 02:59:05.479897 21403 consensus_queue.cc:237] T 926a250fa5834e88b96ccecbefbf5086 P 1c50de89a5a845668696dcb28c3badd6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:59:05.480142 21429 consensus_queue.cc:237] T e7d83d4ccf684e76acb29a0dc844e1b8 P 1c50de89a5a845668696dcb28c3badd6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:59:05.505682 20334 catalog_manager.cc:5654] T 926a250fa5834e88b96ccecbefbf5086 P 1c50de89a5a845668696dcb28c3badd6 reported cstate change: term changed from 1 to 2, leader changed from 197b880fa39d432a9b472845876bfaf0 (127.18.211.1) to 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3). New cstate: current_term: 2 leader_uuid: "1c50de89a5a845668696dcb28c3badd6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } health_report { overall_health: HEALTHY } } }
I20260129 02:59:05.528781 21426 raft_consensus.cc:1064] T 78aa39096868415d9afc2ffcb248e198 P 1c50de89a5a845668696dcb28c3badd6: attempting to promote NON_VOTER 7a9daf5d42a84615a6383cfec9a10608 to VOTER
I20260129 02:59:05.531435 21426 consensus_queue.cc:237] T 78aa39096868415d9afc2ffcb248e198 P 1c50de89a5a845668696dcb28c3badd6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } attrs { replace: true } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } }
I20260129 02:59:05.546046 21131 raft_consensus.cc:1275] T 78aa39096868415d9afc2ffcb248e198 P 7a9daf5d42a84615a6383cfec9a10608 [term 1 LEARNER]: Refusing update from remote peer 1c50de89a5a845668696dcb28c3badd6: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260129 02:59:05.547641 21462 consensus_queue.cc:1048] T 78aa39096868415d9afc2ffcb248e198 P 1c50de89a5a845668696dcb28c3badd6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260129 02:59:05.548532 20507 raft_consensus.cc:1275] T 78aa39096868415d9afc2ffcb248e198 P 974b802141bd4f13beffef2aa03f88f9 [term 1 FOLLOWER]: Refusing update from remote peer 1c50de89a5a845668696dcb28c3badd6: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260129 02:59:05.566694 21462 consensus_queue.cc:1048] T 78aa39096868415d9afc2ffcb248e198 P 1c50de89a5a845668696dcb28c3badd6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260129 02:59:05.568228 20438 raft_consensus.cc:1275] T 78aa39096868415d9afc2ffcb248e198 P 197b880fa39d432a9b472845876bfaf0 [term 1 FOLLOWER]: Refusing update from remote peer 1c50de89a5a845668696dcb28c3badd6: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260129 02:59:05.569712 21403 consensus_queue.cc:1048] T 78aa39096868415d9afc2ffcb248e198 P 1c50de89a5a845668696dcb28c3badd6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260129 02:59:05.594247 21462 raft_consensus.cc:2955] T 78aa39096868415d9afc2ffcb248e198 P 1c50de89a5a845668696dcb28c3badd6 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 7a9daf5d42a84615a6383cfec9a10608 (127.18.211.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } attrs { replace: true } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } } }
I20260129 02:59:05.596109 20507 raft_consensus.cc:2955] T 78aa39096868415d9afc2ffcb248e198 P 974b802141bd4f13beffef2aa03f88f9 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 7a9daf5d42a84615a6383cfec9a10608 (127.18.211.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } attrs { replace: true } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } } }
I20260129 02:59:05.599246 20435 raft_consensus.cc:2955] T 78aa39096868415d9afc2ffcb248e198 P 197b880fa39d432a9b472845876bfaf0 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 7a9daf5d42a84615a6383cfec9a10608 (127.18.211.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } attrs { replace: true } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } } }
I20260129 02:59:05.607769 20334 catalog_manager.cc:5654] T e7d83d4ccf684e76acb29a0dc844e1b8 P 1c50de89a5a845668696dcb28c3badd6 reported cstate change: term changed from 1 to 2, leader changed from 197b880fa39d432a9b472845876bfaf0 (127.18.211.1) to 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3). New cstate: current_term: 2 leader_uuid: "1c50de89a5a845668696dcb28c3badd6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } health_report { overall_health: HEALTHY } } }
I20260129 02:59:05.618376 20335 catalog_manager.cc:5654] T 78aa39096868415d9afc2ffcb248e198 P 197b880fa39d432a9b472845876bfaf0 reported cstate change: config changed from index 2 to 3, 7a9daf5d42a84615a6383cfec9a10608 (127.18.211.4) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "1c50de89a5a845668696dcb28c3badd6" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } attrs { replace: true } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } } }
I20260129 02:59:05.624202 21131 raft_consensus.cc:2955] T 78aa39096868415d9afc2ffcb248e198 P 7a9daf5d42a84615a6383cfec9a10608 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 7a9daf5d42a84615a6383cfec9a10608 (127.18.211.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } attrs { replace: true } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } } }
I20260129 02:59:05.630975 21416 raft_consensus.cc:493] T 7a9a69cd82724b1faed1d10aa091fc4a P 197b880fa39d432a9b472845876bfaf0 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260129 02:59:05.631518 21416 raft_consensus.cc:515] T 7a9a69cd82724b1faed1d10aa091fc4a P 197b880fa39d432a9b472845876bfaf0 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:59:05.633379 21416 leader_election.cc:290] T 7a9a69cd82724b1faed1d10aa091fc4a P 197b880fa39d432a9b472845876bfaf0 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 974b802141bd4f13beffef2aa03f88f9 (127.18.211.2:34517), 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3:37735)
I20260129 02:59:05.638165 20507 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7a9a69cd82724b1faed1d10aa091fc4a" candidate_uuid: "197b880fa39d432a9b472845876bfaf0" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "974b802141bd4f13beffef2aa03f88f9" is_pre_election: true
I20260129 02:59:05.638826 20507 raft_consensus.cc:2468] T 7a9a69cd82724b1faed1d10aa091fc4a P 974b802141bd4f13beffef2aa03f88f9 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 197b880fa39d432a9b472845876bfaf0 in term 2.
I20260129 02:59:05.639849 20407 leader_election.cc:304] T 7a9a69cd82724b1faed1d10aa091fc4a P 197b880fa39d432a9b472845876bfaf0 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 197b880fa39d432a9b472845876bfaf0, 974b802141bd4f13beffef2aa03f88f9; no voters:
I20260129 02:59:05.640759 21406 raft_consensus.cc:2804] T 7a9a69cd82724b1faed1d10aa091fc4a P 197b880fa39d432a9b472845876bfaf0 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260129 02:59:05.641108 21406 raft_consensus.cc:493] T 7a9a69cd82724b1faed1d10aa091fc4a P 197b880fa39d432a9b472845876bfaf0 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260129 02:59:05.641422 21406 raft_consensus.cc:3060] T 7a9a69cd82724b1faed1d10aa091fc4a P 197b880fa39d432a9b472845876bfaf0 [term 2 FOLLOWER]: Advancing to term 3
I20260129 02:59:05.651547 21406 raft_consensus.cc:515] T 7a9a69cd82724b1faed1d10aa091fc4a P 197b880fa39d432a9b472845876bfaf0 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:59:05.667034 20582 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7a9a69cd82724b1faed1d10aa091fc4a" candidate_uuid: "197b880fa39d432a9b472845876bfaf0" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1c50de89a5a845668696dcb28c3badd6" is_pre_election: true
I20260129 02:59:05.673039 20582 raft_consensus.cc:2468] T 7a9a69cd82724b1faed1d10aa091fc4a P 1c50de89a5a845668696dcb28c3badd6 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 197b880fa39d432a9b472845876bfaf0 in term 1.
I20260129 02:59:05.674669 20508 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7a9a69cd82724b1faed1d10aa091fc4a" candidate_uuid: "197b880fa39d432a9b472845876bfaf0" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "974b802141bd4f13beffef2aa03f88f9"
I20260129 02:59:05.675305 20508 raft_consensus.cc:3060] T 7a9a69cd82724b1faed1d10aa091fc4a P 974b802141bd4f13beffef2aa03f88f9 [term 2 FOLLOWER]: Advancing to term 3
I20260129 02:59:05.683375 21406 leader_election.cc:290] T 7a9a69cd82724b1faed1d10aa091fc4a P 197b880fa39d432a9b472845876bfaf0 [CANDIDATE]: Term 3 election: Requested vote from peers 974b802141bd4f13beffef2aa03f88f9 (127.18.211.2:34517), 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3:37735)
I20260129 02:59:05.684478 20582 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7a9a69cd82724b1faed1d10aa091fc4a" candidate_uuid: "197b880fa39d432a9b472845876bfaf0" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1c50de89a5a845668696dcb28c3badd6"
I20260129 02:59:05.685081 20582 raft_consensus.cc:3060] T 7a9a69cd82724b1faed1d10aa091fc4a P 1c50de89a5a845668696dcb28c3badd6 [term 1 FOLLOWER]: Advancing to term 3
I20260129 02:59:05.689165 21409 raft_consensus.cc:493] T 4f8f3a38f6e24ba4857875181f447442 P 974b802141bd4f13beffef2aa03f88f9 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260129 02:59:05.689648 21409 raft_consensus.cc:515] T 4f8f3a38f6e24ba4857875181f447442 P 974b802141bd4f13beffef2aa03f88f9 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:59:05.690652 20585 consensus_queue.cc:237] T 78aa39096868415d9afc2ffcb248e198 P 1c50de89a5a845668696dcb28c3badd6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } }
I20260129 02:59:05.692183 20435 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4f8f3a38f6e24ba4857875181f447442" candidate_uuid: "974b802141bd4f13beffef2aa03f88f9" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "197b880fa39d432a9b472845876bfaf0" is_pre_election: true
I20260129 02:59:05.692801 20435 raft_consensus.cc:2468] T 4f8f3a38f6e24ba4857875181f447442 P 197b880fa39d432a9b472845876bfaf0 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 974b802141bd4f13beffef2aa03f88f9 in term 2.
I20260129 02:59:05.693099 20587 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4f8f3a38f6e24ba4857875181f447442" candidate_uuid: "974b802141bd4f13beffef2aa03f88f9" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1c50de89a5a845668696dcb28c3badd6" is_pre_election: true
I20260129 02:59:05.693650 20587 raft_consensus.cc:2468] T 4f8f3a38f6e24ba4857875181f447442 P 1c50de89a5a845668696dcb28c3badd6 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 974b802141bd4f13beffef2aa03f88f9 in term 1.
I20260129 02:59:05.694717 20479 leader_election.cc:304] T 4f8f3a38f6e24ba4857875181f447442 P 974b802141bd4f13beffef2aa03f88f9 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 197b880fa39d432a9b472845876bfaf0, 974b802141bd4f13beffef2aa03f88f9; no voters:
I20260129 02:59:05.695488 21398 raft_consensus.cc:2804] T 4f8f3a38f6e24ba4857875181f447442 P 974b802141bd4f13beffef2aa03f88f9 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260129 02:59:05.695839 21398 raft_consensus.cc:493] T 4f8f3a38f6e24ba4857875181f447442 P 974b802141bd4f13beffef2aa03f88f9 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260129 02:59:05.695995 21409 leader_election.cc:290] T 4f8f3a38f6e24ba4857875181f447442 P 974b802141bd4f13beffef2aa03f88f9 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 197b880fa39d432a9b472845876bfaf0 (127.18.211.1:42027), 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3:37735)
I20260129 02:59:05.696237 21398 raft_consensus.cc:3060] T 4f8f3a38f6e24ba4857875181f447442 P 974b802141bd4f13beffef2aa03f88f9 [term 2 FOLLOWER]: Advancing to term 3
I20260129 02:59:05.700663 20582 raft_consensus.cc:2468] T 7a9a69cd82724b1faed1d10aa091fc4a P 1c50de89a5a845668696dcb28c3badd6 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 197b880fa39d432a9b472845876bfaf0 in term 3.
I20260129 02:59:05.701869 21131 raft_consensus.cc:1275] T 78aa39096868415d9afc2ffcb248e198 P 7a9daf5d42a84615a6383cfec9a10608 [term 1 FOLLOWER]: Refusing update from remote peer 1c50de89a5a845668696dcb28c3badd6: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260129 02:59:05.703137 21463 consensus_queue.cc:1048] T 78aa39096868415d9afc2ffcb248e198 P 1c50de89a5a845668696dcb28c3badd6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260129 02:59:05.691090 20508 raft_consensus.cc:2468] T 7a9a69cd82724b1faed1d10aa091fc4a P 974b802141bd4f13beffef2aa03f88f9 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 197b880fa39d432a9b472845876bfaf0 in term 3.
I20260129 02:59:05.705433 20435 raft_consensus.cc:1275] T 78aa39096868415d9afc2ffcb248e198 P 197b880fa39d432a9b472845876bfaf0 [term 1 FOLLOWER]: Refusing update from remote peer 1c50de89a5a845668696dcb28c3badd6: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260129 02:59:05.706511 20407 leader_election.cc:304] T 7a9a69cd82724b1faed1d10aa091fc4a P 197b880fa39d432a9b472845876bfaf0 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 197b880fa39d432a9b472845876bfaf0, 974b802141bd4f13beffef2aa03f88f9; no voters:
I20260129 02:59:05.708879 21416 raft_consensus.cc:2804] T 7a9a69cd82724b1faed1d10aa091fc4a P 197b880fa39d432a9b472845876bfaf0 [term 3 FOLLOWER]: Leader election won for term 3
I20260129 02:59:05.709580 21416 raft_consensus.cc:697] T 7a9a69cd82724b1faed1d10aa091fc4a P 197b880fa39d432a9b472845876bfaf0 [term 3 LEADER]: Becoming Leader. State: Replica: 197b880fa39d432a9b472845876bfaf0, State: Running, Role: LEADER
I20260129 02:59:05.711436 21463 consensus_queue.cc:1048] T 78aa39096868415d9afc2ffcb248e198 P 1c50de89a5a845668696dcb28c3badd6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260129 02:59:05.712407 21416 consensus_queue.cc:237] T 7a9a69cd82724b1faed1d10aa091fc4a P 197b880fa39d432a9b472845876bfaf0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:59:05.714221 21462 raft_consensus.cc:2955] T 78aa39096868415d9afc2ffcb248e198 P 1c50de89a5a845668696dcb28c3badd6 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 974b802141bd4f13beffef2aa03f88f9 (127.18.211.2) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } } }
I20260129 02:59:05.721879 20435 raft_consensus.cc:2955] T 78aa39096868415d9afc2ffcb248e198 P 197b880fa39d432a9b472845876bfaf0 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 974b802141bd4f13beffef2aa03f88f9 (127.18.211.2) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } } }
I20260129 02:59:05.702920 21398 raft_consensus.cc:515] T 4f8f3a38f6e24ba4857875181f447442 P 974b802141bd4f13beffef2aa03f88f9 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:59:05.729458 20440 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4f8f3a38f6e24ba4857875181f447442" candidate_uuid: "974b802141bd4f13beffef2aa03f88f9" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "197b880fa39d432a9b472845876bfaf0"
I20260129 02:59:05.730084 20440 raft_consensus.cc:3060] T 4f8f3a38f6e24ba4857875181f447442 P 197b880fa39d432a9b472845876bfaf0 [term 2 FOLLOWER]: Advancing to term 3
I20260129 02:59:05.730948 20323 catalog_manager.cc:5167] ChangeConfig:REMOVE_PEER RPC for tablet 78aa39096868415d9afc2ffcb248e198 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260129 02:59:05.727661 21398 leader_election.cc:290] T 4f8f3a38f6e24ba4857875181f447442 P 974b802141bd4f13beffef2aa03f88f9 [CANDIDATE]: Term 3 election: Requested vote from peers 197b880fa39d432a9b472845876bfaf0 (127.18.211.1:42027), 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3:37735)
I20260129 02:59:05.735127 21424 raft_consensus.cc:493] T cb0b3f259ebc4310927338a8740e87b8 P 7a9daf5d42a84615a6383cfec9a10608 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 974b802141bd4f13beffef2aa03f88f9)
I20260129 02:59:05.735688 21424 raft_consensus.cc:515] T cb0b3f259ebc4310927338a8740e87b8 P 7a9daf5d42a84615a6383cfec9a10608 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } }
I20260129 02:59:05.737532 21424 leader_election.cc:290] T cb0b3f259ebc4310927338a8740e87b8 P 7a9daf5d42a84615a6383cfec9a10608 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 974b802141bd4f13beffef2aa03f88f9 (127.18.211.2:34517), 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3:37735)
I20260129 02:59:05.739959 20508 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "cb0b3f259ebc4310927338a8740e87b8" candidate_uuid: "7a9daf5d42a84615a6383cfec9a10608" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "974b802141bd4f13beffef2aa03f88f9" is_pre_election: true
I20260129 02:59:05.739557 21131 raft_consensus.cc:2955] T 78aa39096868415d9afc2ffcb248e198 P 7a9daf5d42a84615a6383cfec9a10608 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 974b802141bd4f13beffef2aa03f88f9 (127.18.211.2) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } } }
I20260129 02:59:05.740597 20508 raft_consensus.cc:2468] T cb0b3f259ebc4310927338a8740e87b8 P 974b802141bd4f13beffef2aa03f88f9 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7a9daf5d42a84615a6383cfec9a10608 in term 2.
I20260129 02:59:05.742795 21102 leader_election.cc:304] T cb0b3f259ebc4310927338a8740e87b8 P 7a9daf5d42a84615a6383cfec9a10608 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7a9daf5d42a84615a6383cfec9a10608, 974b802141bd4f13beffef2aa03f88f9; no voters:
I20260129 02:59:05.743570 21424 raft_consensus.cc:2804] T cb0b3f259ebc4310927338a8740e87b8 P 7a9daf5d42a84615a6383cfec9a10608 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260129 02:59:05.747144 20440 raft_consensus.cc:2468] T 4f8f3a38f6e24ba4857875181f447442 P 197b880fa39d432a9b472845876bfaf0 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 974b802141bd4f13beffef2aa03f88f9 in term 3.
I20260129 02:59:05.748265 20479 leader_election.cc:304] T 4f8f3a38f6e24ba4857875181f447442 P 974b802141bd4f13beffef2aa03f88f9 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 197b880fa39d432a9b472845876bfaf0, 974b802141bd4f13beffef2aa03f88f9; no voters:
I20260129 02:59:05.748745 21386 raft_consensus.cc:1064] T 0678b8b468bd43cdbc3e485900844821 P 197b880fa39d432a9b472845876bfaf0: attempting to promote NON_VOTER 7a9daf5d42a84615a6383cfec9a10608 to VOTER
I20260129 02:59:05.749006 20589 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4f8f3a38f6e24ba4857875181f447442" candidate_uuid: "974b802141bd4f13beffef2aa03f88f9" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1c50de89a5a845668696dcb28c3badd6"
I20260129 02:59:05.749644 20589 raft_consensus.cc:3060] T 4f8f3a38f6e24ba4857875181f447442 P 1c50de89a5a845668696dcb28c3badd6 [term 1 FOLLOWER]: Advancing to term 3
I20260129 02:59:05.750176 21409 raft_consensus.cc:2804] T 4f8f3a38f6e24ba4857875181f447442 P 974b802141bd4f13beffef2aa03f88f9 [term 3 FOLLOWER]: Leader election won for term 3
I20260129 02:59:05.750679 21409 raft_consensus.cc:697] T 4f8f3a38f6e24ba4857875181f447442 P 974b802141bd4f13beffef2aa03f88f9 [term 3 LEADER]: Becoming Leader. State: Replica: 974b802141bd4f13beffef2aa03f88f9, State: Running, Role: LEADER
I20260129 02:59:05.751295 21386 consensus_queue.cc:237] T 0678b8b468bd43cdbc3e485900844821 P 197b880fa39d432a9b472845876bfaf0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } attrs { replace: true } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } }
I20260129 02:59:05.751475 21409 consensus_queue.cc:237] T 4f8f3a38f6e24ba4857875181f447442 P 974b802141bd4f13beffef2aa03f88f9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:59:05.743979 21424 raft_consensus.cc:493] T cb0b3f259ebc4310927338a8740e87b8 P 7a9daf5d42a84615a6383cfec9a10608 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 974b802141bd4f13beffef2aa03f88f9)
I20260129 02:59:05.758152 21424 raft_consensus.cc:3060] T cb0b3f259ebc4310927338a8740e87b8 P 7a9daf5d42a84615a6383cfec9a10608 [term 1 FOLLOWER]: Advancing to term 2
I20260129 02:59:05.747609 20337 catalog_manager.cc:5654] T 78aa39096868415d9afc2ffcb248e198 P 1c50de89a5a845668696dcb28c3badd6 reported cstate change: config changed from index 3 to 4, VOTER 974b802141bd4f13beffef2aa03f88f9 (127.18.211.2) evicted. New cstate: current_term: 1 leader_uuid: "1c50de89a5a845668696dcb28c3badd6" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260129 02:59:05.765154 21436 raft_consensus.cc:493] T 0f20af7fbe5d4db89a6f17948d98912c P 974b802141bd4f13beffef2aa03f88f9 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260129 02:59:05.765694 21436 raft_consensus.cc:515] T 0f20af7fbe5d4db89a6f17948d98912c P 974b802141bd4f13beffef2aa03f88f9 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:59:05.771705 21436 leader_election.cc:290] T 0f20af7fbe5d4db89a6f17948d98912c P 974b802141bd4f13beffef2aa03f88f9 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 197b880fa39d432a9b472845876bfaf0 (127.18.211.1:42027), 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3:37735)
I20260129 02:59:05.776043 21469 raft_consensus.cc:493] T a609ac02fae84725ab743f95f51d33ce P 197b880fa39d432a9b472845876bfaf0 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260129 02:59:05.776165 20435 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0f20af7fbe5d4db89a6f17948d98912c" candidate_uuid: "974b802141bd4f13beffef2aa03f88f9" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "197b880fa39d432a9b472845876bfaf0" is_pre_election: true
I20260129 02:59:05.776806 20435 raft_consensus.cc:2468] T 0f20af7fbe5d4db89a6f17948d98912c P 197b880fa39d432a9b472845876bfaf0 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 974b802141bd4f13beffef2aa03f88f9 in term 2.
I20260129 02:59:05.776640 21469 raft_consensus.cc:515] T a609ac02fae84725ab743f95f51d33ce P 197b880fa39d432a9b472845876bfaf0 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } }
I20260129 02:59:05.777983 20479 leader_election.cc:304] T 0f20af7fbe5d4db89a6f17948d98912c P 974b802141bd4f13beffef2aa03f88f9 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 197b880fa39d432a9b472845876bfaf0, 974b802141bd4f13beffef2aa03f88f9; no voters:
I20260129 02:59:05.778873 21469 leader_election.cc:290] T a609ac02fae84725ab743f95f51d33ce P 197b880fa39d432a9b472845876bfaf0 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 974b802141bd4f13beffef2aa03f88f9 (127.18.211.2:34517), 7a9daf5d42a84615a6383cfec9a10608 (127.18.211.4:40439)
I20260129 02:59:05.779508 20500 tablet_service.cc:1552] Processing DeleteTablet for tablet 78aa39096868415d9afc2ffcb248e198 with delete_type TABLET_DATA_TOMBSTONED (TS 974b802141bd4f13beffef2aa03f88f9 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:57580
I20260129 02:59:05.780479 21134 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a609ac02fae84725ab743f95f51d33ce" candidate_uuid: "197b880fa39d432a9b472845876bfaf0" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "7a9daf5d42a84615a6383cfec9a10608" is_pre_election: true
I20260129 02:59:05.781098 21134 raft_consensus.cc:2468] T a609ac02fae84725ab743f95f51d33ce P 7a9daf5d42a84615a6383cfec9a10608 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 197b880fa39d432a9b472845876bfaf0 in term 1.
I20260129 02:59:05.781441 20589 raft_consensus.cc:2468] T 4f8f3a38f6e24ba4857875181f447442 P 1c50de89a5a845668696dcb28c3badd6 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 974b802141bd4f13beffef2aa03f88f9 in term 3.
I20260129 02:59:05.781694 20508 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a609ac02fae84725ab743f95f51d33ce" candidate_uuid: "197b880fa39d432a9b472845876bfaf0" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "974b802141bd4f13beffef2aa03f88f9" is_pre_election: true
I20260129 02:59:05.782286 20508 raft_consensus.cc:2468] T a609ac02fae84725ab743f95f51d33ce P 974b802141bd4f13beffef2aa03f88f9 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 197b880fa39d432a9b472845876bfaf0 in term 2.
I20260129 02:59:05.782299 20405 leader_election.cc:304] T a609ac02fae84725ab743f95f51d33ce P 197b880fa39d432a9b472845876bfaf0 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 197b880fa39d432a9b472845876bfaf0, 7a9daf5d42a84615a6383cfec9a10608; no voters:
I20260129 02:59:05.784216 21416 raft_consensus.cc:2804] T a609ac02fae84725ab743f95f51d33ce P 197b880fa39d432a9b472845876bfaf0 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260129 02:59:05.784567 21416 raft_consensus.cc:493] T a609ac02fae84725ab743f95f51d33ce P 197b880fa39d432a9b472845876bfaf0 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260129 02:59:05.784883 21416 raft_consensus.cc:3060] T a609ac02fae84725ab743f95f51d33ce P 197b880fa39d432a9b472845876bfaf0 [term 2 FOLLOWER]: Advancing to term 3
I20260129 02:59:05.805569 21475 raft_consensus.cc:493] T c891d2bff4ce4568861bbc992e91fbb0 P 197b880fa39d432a9b472845876bfaf0 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260129 02:59:05.807438 21475 raft_consensus.cc:515] T c891d2bff4ce4568861bbc992e91fbb0 P 197b880fa39d432a9b472845876bfaf0 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:59:05.811487 21475 leader_election.cc:290] T c891d2bff4ce4568861bbc992e91fbb0 P 197b880fa39d432a9b472845876bfaf0 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 974b802141bd4f13beffef2aa03f88f9 (127.18.211.2:34517), 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3:37735)
I20260129 02:59:05.814095 20508 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c891d2bff4ce4568861bbc992e91fbb0" candidate_uuid: "197b880fa39d432a9b472845876bfaf0" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "974b802141bd4f13beffef2aa03f88f9" is_pre_election: true
I20260129 02:59:05.819279 20508 raft_consensus.cc:2468] T c891d2bff4ce4568861bbc992e91fbb0 P 974b802141bd4f13beffef2aa03f88f9 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 197b880fa39d432a9b472845876bfaf0 in term 2.
I20260129 02:59:05.802181 21436 raft_consensus.cc:2804] T 0f20af7fbe5d4db89a6f17948d98912c P 974b802141bd4f13beffef2aa03f88f9 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260129 02:59:05.823199 21424 raft_consensus.cc:515] T cb0b3f259ebc4310927338a8740e87b8 P 7a9daf5d42a84615a6383cfec9a10608 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } }
I20260129 02:59:05.824282 21436 raft_consensus.cc:493] T 0f20af7fbe5d4db89a6f17948d98912c P 974b802141bd4f13beffef2aa03f88f9 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260129 02:59:05.824816 21436 raft_consensus.cc:3060] T 0f20af7fbe5d4db89a6f17948d98912c P 974b802141bd4f13beffef2aa03f88f9 [term 2 FOLLOWER]: Advancing to term 3
I20260129 02:59:05.825985 20589 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0f20af7fbe5d4db89a6f17948d98912c" candidate_uuid: "974b802141bd4f13beffef2aa03f88f9" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1c50de89a5a845668696dcb28c3badd6" is_pre_election: true
I20260129 02:59:05.824972 21416 raft_consensus.cc:515] T a609ac02fae84725ab743f95f51d33ce P 197b880fa39d432a9b472845876bfaf0 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } }
I20260129 02:59:05.826784 20589 raft_consensus.cc:2468] T 0f20af7fbe5d4db89a6f17948d98912c P 1c50de89a5a845668696dcb28c3badd6 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 974b802141bd4f13beffef2aa03f88f9 in term 1.
I20260129 02:59:05.827749 20582 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "cb0b3f259ebc4310927338a8740e87b8" candidate_uuid: "7a9daf5d42a84615a6383cfec9a10608" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "1c50de89a5a845668696dcb28c3badd6" is_pre_election: true
I20260129 02:59:05.828536 21416 leader_election.cc:290] T a609ac02fae84725ab743f95f51d33ce P 197b880fa39d432a9b472845876bfaf0 [CANDIDATE]: Term 3 election: Requested vote from peers 974b802141bd4f13beffef2aa03f88f9 (127.18.211.2:34517), 7a9daf5d42a84615a6383cfec9a10608 (127.18.211.4:40439)
I20260129 02:59:05.828790 20582 raft_consensus.cc:2393] T cb0b3f259ebc4310927338a8740e87b8 P 1c50de89a5a845668696dcb28c3badd6 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 7a9daf5d42a84615a6383cfec9a10608 in current term 2: Already voted for candidate 1c50de89a5a845668696dcb28c3badd6 in this term.
I20260129 02:59:05.833400 20582 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "cb0b3f259ebc4310927338a8740e87b8" candidate_uuid: "7a9daf5d42a84615a6383cfec9a10608" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "1c50de89a5a845668696dcb28c3badd6"
I20260129 02:59:05.834156 20582 raft_consensus.cc:2393] T cb0b3f259ebc4310927338a8740e87b8 P 1c50de89a5a845668696dcb28c3badd6 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 7a9daf5d42a84615a6383cfec9a10608 in current term 2: Already voted for candidate 1c50de89a5a845668696dcb28c3badd6 in this term.
I20260129 02:59:05.835636 21424 leader_election.cc:290] T cb0b3f259ebc4310927338a8740e87b8 P 7a9daf5d42a84615a6383cfec9a10608 [CANDIDATE]: Term 2 election: Requested vote from peers 974b802141bd4f13beffef2aa03f88f9 (127.18.211.2:34517), 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3:37735)
I20260129 02:59:05.816839 20337 catalog_manager.cc:5654] T 7a9a69cd82724b1faed1d10aa091fc4a P 197b880fa39d432a9b472845876bfaf0 reported cstate change: term changed from 1 to 3. New cstate: current_term: 3 leader_uuid: "197b880fa39d432a9b472845876bfaf0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } health_report { overall_health: UNKNOWN } } }
I20260129 02:59:05.837256 21409 raft_consensus.cc:493] T 75be4bd6bd45474795fd472eff2953ac P 974b802141bd4f13beffef2aa03f88f9 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260129 02:59:05.837752 21409 raft_consensus.cc:515] T 75be4bd6bd45474795fd472eff2953ac P 974b802141bd4f13beffef2aa03f88f9 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:59:05.841356 21473 tablet_replica.cc:333] T 78aa39096868415d9afc2ffcb248e198 P 974b802141bd4f13beffef2aa03f88f9: stopping tablet replica
I20260129 02:59:05.842434 21473 raft_consensus.cc:2243] T 78aa39096868415d9afc2ffcb248e198 P 974b802141bd4f13beffef2aa03f88f9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260129 02:59:05.839670 21409 leader_election.cc:290] T 75be4bd6bd45474795fd472eff2953ac P 974b802141bd4f13beffef2aa03f88f9 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 197b880fa39d432a9b472845876bfaf0 (127.18.211.1:42027), 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3:37735)
I20260129 02:59:05.843232 21473 raft_consensus.cc:2272] T 78aa39096868415d9afc2ffcb248e198 P 974b802141bd4f13beffef2aa03f88f9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260129 02:59:05.846743 20407 leader_election.cc:304] T c891d2bff4ce4568861bbc992e91fbb0 P 197b880fa39d432a9b472845876bfaf0 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 197b880fa39d432a9b472845876bfaf0, 974b802141bd4f13beffef2aa03f88f9; no voters:
I20260129 02:59:05.848721 20508 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a609ac02fae84725ab743f95f51d33ce" candidate_uuid: "197b880fa39d432a9b472845876bfaf0" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "974b802141bd4f13beffef2aa03f88f9"
I20260129 02:59:05.848959 21469 raft_consensus.cc:2804] T c891d2bff4ce4568861bbc992e91fbb0 P 197b880fa39d432a9b472845876bfaf0 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260129 02:59:05.849329 20508 raft_consensus.cc:3060] T a609ac02fae84725ab743f95f51d33ce P 974b802141bd4f13beffef2aa03f88f9 [term 2 FOLLOWER]: Advancing to term 3
I20260129 02:59:05.849404 21469 raft_consensus.cc:493] T c891d2bff4ce4568861bbc992e91fbb0 P 197b880fa39d432a9b472845876bfaf0 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260129 02:59:05.849889 21469 raft_consensus.cc:3060] T c891d2bff4ce4568861bbc992e91fbb0 P 197b880fa39d432a9b472845876bfaf0 [term 2 FOLLOWER]: Advancing to term 3
I20260129 02:59:05.856530 21473 ts_tablet_manager.cc:1916] T 78aa39096868415d9afc2ffcb248e198 P 974b802141bd4f13beffef2aa03f88f9: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260129 02:59:05.857460 21469 raft_consensus.cc:515] T c891d2bff4ce4568861bbc992e91fbb0 P 197b880fa39d432a9b472845876bfaf0 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:59:05.859695 21469 leader_election.cc:290] T c891d2bff4ce4568861bbc992e91fbb0 P 197b880fa39d432a9b472845876bfaf0 [CANDIDATE]: Term 3 election: Requested vote from peers 974b802141bd4f13beffef2aa03f88f9 (127.18.211.2:34517), 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3:37735)
I20260129 02:59:05.860232 21436 raft_consensus.cc:515] T 0f20af7fbe5d4db89a6f17948d98912c P 974b802141bd4f13beffef2aa03f88f9 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:59:05.862105 21436 leader_election.cc:290] T 0f20af7fbe5d4db89a6f17948d98912c P 974b802141bd4f13beffef2aa03f88f9 [CANDIDATE]: Term 3 election: Requested vote from peers 197b880fa39d432a9b472845876bfaf0 (127.18.211.1:42027), 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3:37735)
I20260129 02:59:05.864691 20440 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "75be4bd6bd45474795fd472eff2953ac" candidate_uuid: "974b802141bd4f13beffef2aa03f88f9" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "197b880fa39d432a9b472845876bfaf0" is_pre_election: true
I20260129 02:59:05.865504 20440 raft_consensus.cc:2468] T 75be4bd6bd45474795fd472eff2953ac P 197b880fa39d432a9b472845876bfaf0 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 974b802141bd4f13beffef2aa03f88f9 in term 2.
I20260129 02:59:05.866883 20336 catalog_manager.cc:5654] T 4f8f3a38f6e24ba4857875181f447442 P 974b802141bd4f13beffef2aa03f88f9 reported cstate change: term changed from 1 to 3. New cstate: current_term: 3 leader_uuid: "974b802141bd4f13beffef2aa03f88f9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } health_report { overall_health: UNKNOWN } } }
I20260129 02:59:05.873879 20440 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0f20af7fbe5d4db89a6f17948d98912c" candidate_uuid: "974b802141bd4f13beffef2aa03f88f9" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "197b880fa39d432a9b472845876bfaf0"
I20260129 02:59:05.874570 20440 raft_consensus.cc:3060] T 0f20af7fbe5d4db89a6f17948d98912c P 197b880fa39d432a9b472845876bfaf0 [term 2 FOLLOWER]: Advancing to term 3
I20260129 02:59:05.857625 20508 raft_consensus.cc:2468] T a609ac02fae84725ab743f95f51d33ce P 974b802141bd4f13beffef2aa03f88f9 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 197b880fa39d432a9b472845876bfaf0 in term 3.
I20260129 02:59:05.882426 20435 raft_consensus.cc:1275] T 926a250fa5834e88b96ccecbefbf5086 P 197b880fa39d432a9b472845876bfaf0 [term 2 FOLLOWER]: Refusing update from remote peer 1c50de89a5a845668696dcb28c3badd6: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260129 02:59:05.883162 20479 leader_election.cc:304] T 75be4bd6bd45474795fd472eff2953ac P 974b802141bd4f13beffef2aa03f88f9 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 197b880fa39d432a9b472845876bfaf0, 974b802141bd4f13beffef2aa03f88f9; no voters:
I20260129 02:59:05.883237 20407 leader_election.cc:304] T a609ac02fae84725ab743f95f51d33ce P 197b880fa39d432a9b472845876bfaf0 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 197b880fa39d432a9b472845876bfaf0, 974b802141bd4f13beffef2aa03f88f9; no voters:
I20260129 02:59:05.883746 20507 raft_consensus.cc:1275] T 0678b8b468bd43cdbc3e485900844821 P 974b802141bd4f13beffef2aa03f88f9 [term 1 FOLLOWER]: Refusing update from remote peer 197b880fa39d432a9b472845876bfaf0: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260129 02:59:05.885308 21436 raft_consensus.cc:2804] T 75be4bd6bd45474795fd472eff2953ac P 974b802141bd4f13beffef2aa03f88f9 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260129 02:59:05.886698 21416 raft_consensus.cc:2804] T a609ac02fae84725ab743f95f51d33ce P 197b880fa39d432a9b472845876bfaf0 [term 3 FOLLOWER]: Leader election won for term 3
I20260129 02:59:05.887632 21462 consensus_queue.cc:1048] T 926a250fa5834e88b96ccecbefbf5086 P 1c50de89a5a845668696dcb28c3badd6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260129 02:59:05.890748 20508 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c891d2bff4ce4568861bbc992e91fbb0" candidate_uuid: "197b880fa39d432a9b472845876bfaf0" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "974b802141bd4f13beffef2aa03f88f9"
I20260129 02:59:05.887562 21416 raft_consensus.cc:697] T a609ac02fae84725ab743f95f51d33ce P 197b880fa39d432a9b472845876bfaf0 [term 3 LEADER]: Becoming Leader. State: Replica: 197b880fa39d432a9b472845876bfaf0, State: Running, Role: LEADER
I20260129 02:59:05.891418 20508 raft_consensus.cc:3060] T c891d2bff4ce4568861bbc992e91fbb0 P 974b802141bd4f13beffef2aa03f88f9 [term 2 FOLLOWER]: Advancing to term 3
I20260129 02:59:05.892262 21476 consensus_queue.cc:1048] T 0678b8b468bd43cdbc3e485900844821 P 197b880fa39d432a9b472845876bfaf0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260129 02:59:05.892052 21416 consensus_queue.cc:237] T a609ac02fae84725ab743f95f51d33ce P 197b880fa39d432a9b472845876bfaf0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } }
I20260129 02:59:05.896152 20440 raft_consensus.cc:2468] T 0f20af7fbe5d4db89a6f17948d98912c P 197b880fa39d432a9b472845876bfaf0 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 974b802141bd4f13beffef2aa03f88f9 in term 3.
I20260129 02:59:05.885712 21436 raft_consensus.cc:493] T 75be4bd6bd45474795fd472eff2953ac P 974b802141bd4f13beffef2aa03f88f9 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260129 02:59:05.899811 21436 raft_consensus.cc:3060] T 75be4bd6bd45474795fd472eff2953ac P 974b802141bd4f13beffef2aa03f88f9 [term 2 FOLLOWER]: Advancing to term 3
I20260129 02:59:05.902709 20479 leader_election.cc:304] T 0f20af7fbe5d4db89a6f17948d98912c P 974b802141bd4f13beffef2aa03f88f9 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 197b880fa39d432a9b472845876bfaf0, 974b802141bd4f13beffef2aa03f88f9; no voters:
I20260129 02:59:05.903728 21398 raft_consensus.cc:2804] T 0f20af7fbe5d4db89a6f17948d98912c P 974b802141bd4f13beffef2aa03f88f9 [term 3 FOLLOWER]: Leader election won for term 3
I20260129 02:59:05.904232 21398 raft_consensus.cc:697] T 0f20af7fbe5d4db89a6f17948d98912c P 974b802141bd4f13beffef2aa03f88f9 [term 3 LEADER]: Becoming Leader. State: Replica: 974b802141bd4f13beffef2aa03f88f9, State: Running, Role: LEADER
I20260129 02:59:05.905079 21398 consensus_queue.cc:237] T 0f20af7fbe5d4db89a6f17948d98912c P 974b802141bd4f13beffef2aa03f88f9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:59:05.906708 21473 ts_tablet_manager.cc:1929] T 78aa39096868415d9afc2ffcb248e198 P 974b802141bd4f13beffef2aa03f88f9: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20260129 02:59:05.907126 21473 log.cc:1199] T 78aa39096868415d9afc2ffcb248e198 P 974b802141bd4f13beffef2aa03f88f9: Deleting WAL directory at /tmp/dist-test-taskrTP_Jo/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1769655460890924-19276-0/minicluster-data/ts-1-root/wals/78aa39096868415d9afc2ffcb248e198
I20260129 02:59:05.912554 20325 catalog_manager.cc:4985] TS 974b802141bd4f13beffef2aa03f88f9 (127.18.211.2:34517): tablet 78aa39096868415d9afc2ffcb248e198 (table test-workload [id=de73611ac8e3485a98aecc5add247032]) successfully deleted
I20260129 02:59:05.916684 20582 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c891d2bff4ce4568861bbc992e91fbb0" candidate_uuid: "197b880fa39d432a9b472845876bfaf0" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1c50de89a5a845668696dcb28c3badd6" is_pre_election: true
I20260129 02:59:05.922937 21131 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a609ac02fae84725ab743f95f51d33ce" candidate_uuid: "197b880fa39d432a9b472845876bfaf0" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "7a9daf5d42a84615a6383cfec9a10608"
I20260129 02:59:05.923557 21131 raft_consensus.cc:3060] T a609ac02fae84725ab743f95f51d33ce P 7a9daf5d42a84615a6383cfec9a10608 [term 1 FOLLOWER]: Advancing to term 3
W20260129 02:59:05.929173 20480 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.211.4:40439, user_credentials={real_user=slave}} blocked reactor thread for 138653us
I20260129 02:59:05.930974 20582 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "75be4bd6bd45474795fd472eff2953ac" candidate_uuid: "974b802141bd4f13beffef2aa03f88f9" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1c50de89a5a845668696dcb28c3badd6" is_pre_election: true
I20260129 02:59:05.931582 21131 raft_consensus.cc:2468] T a609ac02fae84725ab743f95f51d33ce P 7a9daf5d42a84615a6383cfec9a10608 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 197b880fa39d432a9b472845876bfaf0 in term 3.
I20260129 02:59:05.931766 20582 raft_consensus.cc:2468] T 75be4bd6bd45474795fd472eff2953ac P 1c50de89a5a845668696dcb28c3badd6 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 974b802141bd4f13beffef2aa03f88f9 in term 1.
I20260129 02:59:05.933979 21463 raft_consensus.cc:493] T c891d2bff4ce4568861bbc992e91fbb0 P 1c50de89a5a845668696dcb28c3badd6 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 197b880fa39d432a9b472845876bfaf0)
I20260129 02:59:05.934383 20582 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0f20af7fbe5d4db89a6f17948d98912c" candidate_uuid: "974b802141bd4f13beffef2aa03f88f9" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1c50de89a5a845668696dcb28c3badd6"
I20260129 02:59:05.934511 21463 raft_consensus.cc:515] T c891d2bff4ce4568861bbc992e91fbb0 P 1c50de89a5a845668696dcb28c3badd6 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:59:05.935258 20582 raft_consensus.cc:3060] T 0f20af7fbe5d4db89a6f17948d98912c P 1c50de89a5a845668696dcb28c3badd6 [term 1 FOLLOWER]: Advancing to term 3
I20260129 02:59:05.936899 21463 leader_election.cc:290] T c891d2bff4ce4568861bbc992e91fbb0 P 1c50de89a5a845668696dcb28c3badd6 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 197b880fa39d432a9b472845876bfaf0 (127.18.211.1:42027), 974b802141bd4f13beffef2aa03f88f9 (127.18.211.2:34517)
I20260129 02:59:05.938670 20513 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c891d2bff4ce4568861bbc992e91fbb0" candidate_uuid: "1c50de89a5a845668696dcb28c3badd6" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "974b802141bd4f13beffef2aa03f88f9" is_pre_election: true
I20260129 02:59:05.940285 20440 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c891d2bff4ce4568861bbc992e91fbb0" candidate_uuid: "1c50de89a5a845668696dcb28c3badd6" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "197b880fa39d432a9b472845876bfaf0" is_pre_election: true
I20260129 02:59:05.942322 20440 raft_consensus.cc:2368] T c891d2bff4ce4568861bbc992e91fbb0 P 197b880fa39d432a9b472845876bfaf0 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1c50de89a5a845668696dcb28c3badd6 for earlier term 2. Current term is 3.
I20260129 02:59:05.945910 20553 leader_election.cc:400] T c891d2bff4ce4568861bbc992e91fbb0 P 1c50de89a5a845668696dcb28c3badd6 [CANDIDATE]: Term 2 pre-election: Vote denied by peer 197b880fa39d432a9b472845876bfaf0 (127.18.211.1:42027) with higher term. Message: Invalid argument: T c891d2bff4ce4568861bbc992e91fbb0 P 197b880fa39d432a9b472845876bfaf0 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1c50de89a5a845668696dcb28c3badd6 for earlier term 2. Current term is 3.
I20260129 02:59:05.946199 20553 leader_election.cc:403] T c891d2bff4ce4568861bbc992e91fbb0 P 1c50de89a5a845668696dcb28c3badd6 [CANDIDATE]: Term 2 pre-election: Cancelling election due to peer responding with higher term
I20260129 02:59:05.947556 21463 raft_consensus.cc:3060] T c891d2bff4ce4568861bbc992e91fbb0 P 1c50de89a5a845668696dcb28c3badd6 [term 1 FOLLOWER]: Advancing to term 3
I20260129 02:59:05.950412 20513 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "cb0b3f259ebc4310927338a8740e87b8" candidate_uuid: "7a9daf5d42a84615a6383cfec9a10608" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "974b802141bd4f13beffef2aa03f88f9"
I20260129 02:59:05.952589 20589 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c891d2bff4ce4568861bbc992e91fbb0" candidate_uuid: "197b880fa39d432a9b472845876bfaf0" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1c50de89a5a845668696dcb28c3badd6"
I20260129 02:59:05.958092 21131 raft_consensus.cc:1275] T 0678b8b468bd43cdbc3e485900844821 P 7a9daf5d42a84615a6383cfec9a10608 [term 1 LEARNER]: Refusing update from remote peer 197b880fa39d432a9b472845876bfaf0: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260129 02:59:05.958299 20513 raft_consensus.cc:2468] T cb0b3f259ebc4310927338a8740e87b8 P 974b802141bd4f13beffef2aa03f88f9 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7a9daf5d42a84615a6383cfec9a10608 in term 2.
I20260129 02:59:05.960532 20582 raft_consensus.cc:2468] T 0f20af7fbe5d4db89a6f17948d98912c P 1c50de89a5a845668696dcb28c3badd6 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 974b802141bd4f13beffef2aa03f88f9 in term 3.
I20260129 02:59:05.961243 20587 raft_consensus.cc:1275] T 0678b8b468bd43cdbc3e485900844821 P 1c50de89a5a845668696dcb28c3badd6 [term 1 FOLLOWER]: Refusing update from remote peer 197b880fa39d432a9b472845876bfaf0: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260129 02:59:05.963454 21102 leader_election.cc:304] T cb0b3f259ebc4310927338a8740e87b8 P 7a9daf5d42a84615a6383cfec9a10608 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 7a9daf5d42a84615a6383cfec9a10608, 974b802141bd4f13beffef2aa03f88f9; no voters: 1c50de89a5a845668696dcb28c3badd6
I20260129 02:59:05.965458 21424 raft_consensus.cc:2804] T cb0b3f259ebc4310927338a8740e87b8 P 7a9daf5d42a84615a6383cfec9a10608 [term 2 FOLLOWER]: Leader election won for term 2
I20260129 02:59:05.985515 20513 raft_consensus.cc:1275] T 926a250fa5834e88b96ccecbefbf5086 P 974b802141bd4f13beffef2aa03f88f9 [term 2 FOLLOWER]: Refusing update from remote peer 1c50de89a5a845668696dcb28c3badd6: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260129 02:59:05.989102 20440 raft_consensus.cc:1275] T e7d83d4ccf684e76acb29a0dc844e1b8 P 197b880fa39d432a9b472845876bfaf0 [term 2 FOLLOWER]: Refusing update from remote peer 1c50de89a5a845668696dcb28c3badd6: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260129 02:59:05.993579 21409 raft_consensus.cc:1064] T 75f45538dc704d9a8ec4ff8ad9f3ea5b P 974b802141bd4f13beffef2aa03f88f9: attempting to promote NON_VOTER 7a9daf5d42a84615a6383cfec9a10608 to VOTER
I20260129 02:59:05.993425 21429 consensus_queue.cc:1048] T e7d83d4ccf684e76acb29a0dc844e1b8 P 1c50de89a5a845668696dcb28c3badd6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260129 02:59:05.993141 21462 consensus_queue.cc:1048] T 926a250fa5834e88b96ccecbefbf5086 P 1c50de89a5a845668696dcb28c3badd6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.003s
I20260129 02:59:05.999176 21409 consensus_queue.cc:237] T 75f45538dc704d9a8ec4ff8ad9f3ea5b P 974b802141bd4f13beffef2aa03f88f9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } attrs { replace: true } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } }
I20260129 02:59:06.001890 20507 raft_consensus.cc:1275] T e7d83d4ccf684e76acb29a0dc844e1b8 P 974b802141bd4f13beffef2aa03f88f9 [term 2 FOLLOWER]: Refusing update from remote peer 1c50de89a5a845668696dcb28c3badd6: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260129 02:59:06.003903 21462 consensus_queue.cc:1048] T e7d83d4ccf684e76acb29a0dc844e1b8 P 1c50de89a5a845668696dcb28c3badd6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260129 02:59:06.009577 21476 consensus_queue.cc:1048] T 0678b8b468bd43cdbc3e485900844821 P 197b880fa39d432a9b472845876bfaf0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260129 02:59:06.021777 21386 consensus_queue.cc:1048] T 0678b8b468bd43cdbc3e485900844821 P 197b880fa39d432a9b472845876bfaf0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260129 02:59:06.023190 21436 raft_consensus.cc:515] T 75be4bd6bd45474795fd472eff2953ac P 974b802141bd4f13beffef2aa03f88f9 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:59:06.035194 21424 raft_consensus.cc:697] T cb0b3f259ebc4310927338a8740e87b8 P 7a9daf5d42a84615a6383cfec9a10608 [term 2 LEADER]: Becoming Leader. State: Replica: 7a9daf5d42a84615a6383cfec9a10608, State: Running, Role: LEADER
I20260129 02:59:06.057776 20508 raft_consensus.cc:2468] T c891d2bff4ce4568861bbc992e91fbb0 P 974b802141bd4f13beffef2aa03f88f9 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 197b880fa39d432a9b472845876bfaf0 in term 3.
I20260129 02:59:06.058427 20337 catalog_manager.cc:5654] T a609ac02fae84725ab743f95f51d33ce P 197b880fa39d432a9b472845876bfaf0 reported cstate change: term changed from 1 to 3. New cstate: current_term: 3 leader_uuid: "197b880fa39d432a9b472845876bfaf0" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20260129 02:59:06.096649 21424 consensus_queue.cc:237] T cb0b3f259ebc4310927338a8740e87b8 P 7a9daf5d42a84615a6383cfec9a10608 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } }
I20260129 02:59:06.066689 21463 raft_consensus.cc:2749] T c891d2bff4ce4568861bbc992e91fbb0 P 1c50de89a5a845668696dcb28c3badd6 [term 3 FOLLOWER]: Leader pre-election lost for term 2. Reason: Vote denied by peer 197b880fa39d432a9b472845876bfaf0 (127.18.211.1:42027) with higher term. Message: Invalid argument: T c891d2bff4ce4568861bbc992e91fbb0 P 197b880fa39d432a9b472845876bfaf0 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1c50de89a5a845668696dcb28c3badd6 for earlier term 2. Current term is 3.
I20260129 02:59:06.077308 21131 raft_consensus.cc:2955] T 0678b8b468bd43cdbc3e485900844821 P 7a9daf5d42a84615a6383cfec9a10608 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 7a9daf5d42a84615a6383cfec9a10608 (127.18.211.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } attrs { replace: true } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } } }
I20260129 02:59:06.103523 20407 leader_election.cc:304] T c891d2bff4ce4568861bbc992e91fbb0 P 197b880fa39d432a9b472845876bfaf0 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 197b880fa39d432a9b472845876bfaf0, 974b802141bd4f13beffef2aa03f88f9; no voters:
I20260129 02:59:06.104343 21416 raft_consensus.cc:2804] T c891d2bff4ce4568861bbc992e91fbb0 P 197b880fa39d432a9b472845876bfaf0 [term 3 FOLLOWER]: Leader election won for term 3
I20260129 02:59:06.104815 21416 raft_consensus.cc:697] T c891d2bff4ce4568861bbc992e91fbb0 P 197b880fa39d432a9b472845876bfaf0 [term 3 LEADER]: Becoming Leader. State: Replica: 197b880fa39d432a9b472845876bfaf0, State: Running, Role: LEADER
I20260129 02:59:06.105903 20589 raft_consensus.cc:2468] T c891d2bff4ce4568861bbc992e91fbb0 P 1c50de89a5a845668696dcb28c3badd6 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 197b880fa39d432a9b472845876bfaf0 in term 3.
I20260129 02:59:06.085011 21134 raft_consensus.cc:1275] T 75f45538dc704d9a8ec4ff8ad9f3ea5b P 7a9daf5d42a84615a6383cfec9a10608 [term 1 LEARNER]: Refusing update from remote peer 974b802141bd4f13beffef2aa03f88f9: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260129 02:59:06.087615 20582 raft_consensus.cc:1275] T 75f45538dc704d9a8ec4ff8ad9f3ea5b P 1c50de89a5a845668696dcb28c3badd6 [term 1 FOLLOWER]: Refusing update from remote peer 974b802141bd4f13beffef2aa03f88f9: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260129 02:59:06.090965 20438 raft_consensus.cc:1275] T 75f45538dc704d9a8ec4ff8ad9f3ea5b P 197b880fa39d432a9b472845876bfaf0 [term 1 FOLLOWER]: Refusing update from remote peer 974b802141bd4f13beffef2aa03f88f9: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260129 02:59:06.112913 21409 consensus_queue.cc:1048] T 75f45538dc704d9a8ec4ff8ad9f3ea5b P 974b802141bd4f13beffef2aa03f88f9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260129 02:59:06.116847 21474 consensus_queue.cc:1048] T 75f45538dc704d9a8ec4ff8ad9f3ea5b P 974b802141bd4f13beffef2aa03f88f9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260129 02:59:06.117802 21409 consensus_queue.cc:1048] T 75f45538dc704d9a8ec4ff8ad9f3ea5b P 974b802141bd4f13beffef2aa03f88f9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.002s
I20260129 02:59:06.048245 21476 raft_consensus.cc:2955] T 0678b8b468bd43cdbc3e485900844821 P 197b880fa39d432a9b472845876bfaf0 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 7a9daf5d42a84615a6383cfec9a10608 (127.18.211.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } attrs { replace: true } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } } }
I20260129 02:59:06.105638 21416 consensus_queue.cc:237] T c891d2bff4ce4568861bbc992e91fbb0 P 197b880fa39d432a9b472845876bfaf0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:59:06.094419 20583 raft_consensus.cc:2955] T 0678b8b468bd43cdbc3e485900844821 P 1c50de89a5a845668696dcb28c3badd6 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 7a9daf5d42a84615a6383cfec9a10608 (127.18.211.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } attrs { replace: true } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } } }
I20260129 02:59:06.170248 21474 raft_consensus.cc:2955] T 75f45538dc704d9a8ec4ff8ad9f3ea5b P 974b802141bd4f13beffef2aa03f88f9 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 7a9daf5d42a84615a6383cfec9a10608 (127.18.211.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } attrs { replace: true } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } } }
I20260129 02:59:06.175138 21436 leader_election.cc:290] T 75be4bd6bd45474795fd472eff2953ac P 974b802141bd4f13beffef2aa03f88f9 [CANDIDATE]: Term 3 election: Requested vote from peers 197b880fa39d432a9b472845876bfaf0 (127.18.211.1:42027), 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3:37735)
I20260129 02:59:06.176781 20582 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "75be4bd6bd45474795fd472eff2953ac" candidate_uuid: "974b802141bd4f13beffef2aa03f88f9" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1c50de89a5a845668696dcb28c3badd6"
I20260129 02:59:06.177531 20582 raft_consensus.cc:3060] T 75be4bd6bd45474795fd472eff2953ac P 1c50de89a5a845668696dcb28c3badd6 [term 1 FOLLOWER]: Advancing to term 3
I20260129 02:59:06.191829 20435 raft_consensus.cc:2955] T 75f45538dc704d9a8ec4ff8ad9f3ea5b P 197b880fa39d432a9b472845876bfaf0 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 7a9daf5d42a84615a6383cfec9a10608 (127.18.211.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } attrs { replace: true } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } } }
I20260129 02:59:06.194936 20441 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "75be4bd6bd45474795fd472eff2953ac" candidate_uuid: "974b802141bd4f13beffef2aa03f88f9" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "197b880fa39d432a9b472845876bfaf0"
I20260129 02:59:06.197353 20441 raft_consensus.cc:3060] T 75be4bd6bd45474795fd472eff2953ac P 197b880fa39d432a9b472845876bfaf0 [term 2 FOLLOWER]: Advancing to term 3
I20260129 02:59:06.198510 20582 raft_consensus.cc:2468] T 75be4bd6bd45474795fd472eff2953ac P 1c50de89a5a845668696dcb28c3badd6 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 974b802141bd4f13beffef2aa03f88f9 in term 3.
I20260129 02:59:06.190871 21131 raft_consensus.cc:2955] T 75f45538dc704d9a8ec4ff8ad9f3ea5b P 7a9daf5d42a84615a6383cfec9a10608 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 7a9daf5d42a84615a6383cfec9a10608 (127.18.211.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } attrs { replace: true } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } } }
I20260129 02:59:06.199760 20480 leader_election.cc:304] T 75be4bd6bd45474795fd472eff2953ac P 974b802141bd4f13beffef2aa03f88f9 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1c50de89a5a845668696dcb28c3badd6, 974b802141bd4f13beffef2aa03f88f9; no voters:
I20260129 02:59:06.201722 20337 catalog_manager.cc:5654] T 0f20af7fbe5d4db89a6f17948d98912c P 974b802141bd4f13beffef2aa03f88f9 reported cstate change: term changed from 1 to 3, leader changed from 197b880fa39d432a9b472845876bfaf0 (127.18.211.1) to 974b802141bd4f13beffef2aa03f88f9 (127.18.211.2). New cstate: current_term: 3 leader_uuid: "974b802141bd4f13beffef2aa03f88f9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } health_report { overall_health: UNKNOWN } } }
I20260129 02:59:06.203099 21479 raft_consensus.cc:2804] T 75be4bd6bd45474795fd472eff2953ac P 974b802141bd4f13beffef2aa03f88f9 [term 3 FOLLOWER]: Leader election won for term 3
I20260129 02:59:06.203665 21479 raft_consensus.cc:697] T 75be4bd6bd45474795fd472eff2953ac P 974b802141bd4f13beffef2aa03f88f9 [term 3 LEADER]: Becoming Leader. State: Replica: 974b802141bd4f13beffef2aa03f88f9, State: Running, Role: LEADER
I20260129 02:59:06.204635 21479 consensus_queue.cc:237] T 75be4bd6bd45474795fd472eff2953ac P 974b802141bd4f13beffef2aa03f88f9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:59:06.220191 20510 raft_consensus.cc:2955] T 0678b8b468bd43cdbc3e485900844821 P 974b802141bd4f13beffef2aa03f88f9 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 7a9daf5d42a84615a6383cfec9a10608 (127.18.211.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } attrs { replace: true } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } } }
I20260129 02:59:06.212368 20582 raft_consensus.cc:2955] T 75f45538dc704d9a8ec4ff8ad9f3ea5b P 1c50de89a5a845668696dcb28c3badd6 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 7a9daf5d42a84615a6383cfec9a10608 (127.18.211.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } attrs { replace: true } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } } }
I20260129 02:59:06.224534 20507 raft_consensus.cc:1275] T 7a9a69cd82724b1faed1d10aa091fc4a P 974b802141bd4f13beffef2aa03f88f9 [term 3 FOLLOWER]: Refusing update from remote peer 197b880fa39d432a9b472845876bfaf0: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20260129 02:59:06.229707 21386 consensus_queue.cc:1048] T 7a9a69cd82724b1faed1d10aa091fc4a P 197b880fa39d432a9b472845876bfaf0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260129 02:59:06.233686 20441 raft_consensus.cc:2468] T 75be4bd6bd45474795fd472eff2953ac P 197b880fa39d432a9b472845876bfaf0 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 974b802141bd4f13beffef2aa03f88f9 in term 3.
W20260129 02:59:06.237637 20556 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.211.2:34517, user_credentials={real_user=slave}} blocked reactor thread for 66143.1us
I20260129 02:59:06.244553 20438 raft_consensus.cc:1275] T 4f8f3a38f6e24ba4857875181f447442 P 197b880fa39d432a9b472845876bfaf0 [term 3 FOLLOWER]: Refusing update from remote peer 974b802141bd4f13beffef2aa03f88f9: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20260129 02:59:06.261590 21398 consensus_queue.cc:1048] T 4f8f3a38f6e24ba4857875181f447442 P 974b802141bd4f13beffef2aa03f88f9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260129 02:59:06.262445 20336 catalog_manager.cc:5654] T c891d2bff4ce4568861bbc992e91fbb0 P 197b880fa39d432a9b472845876bfaf0 reported cstate change: term changed from 1 to 3. New cstate: current_term: 3 leader_uuid: "197b880fa39d432a9b472845876bfaf0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } health_report { overall_health: UNKNOWN } } }
I20260129 02:59:06.284122 21131 raft_consensus.cc:1275] T a609ac02fae84725ab743f95f51d33ce P 7a9daf5d42a84615a6383cfec9a10608 [term 3 FOLLOWER]: Refusing update from remote peer 197b880fa39d432a9b472845876bfaf0: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 3 index: 5. (index mismatch)
I20260129 02:59:06.294443 21477 consensus_queue.cc:1048] T a609ac02fae84725ab743f95f51d33ce P 197b880fa39d432a9b472845876bfaf0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20260129 02:59:06.299767 20336 catalog_manager.cc:5654] T cb0b3f259ebc4310927338a8740e87b8 P 7a9daf5d42a84615a6383cfec9a10608 reported cstate change: term changed from 1 to 2, leader changed from 974b802141bd4f13beffef2aa03f88f9 (127.18.211.2) to 7a9daf5d42a84615a6383cfec9a10608 (127.18.211.4). New cstate: current_term: 2 leader_uuid: "7a9daf5d42a84615a6383cfec9a10608" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260129 02:59:06.320612 20338 catalog_manager.cc:5654] T 0678b8b468bd43cdbc3e485900844821 P 197b880fa39d432a9b472845876bfaf0 reported cstate change: config changed from index 2 to 3, 7a9daf5d42a84615a6383cfec9a10608 (127.18.211.4) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "197b880fa39d432a9b472845876bfaf0" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260129 02:59:06.322705 20338 catalog_manager.cc:5654] T 75f45538dc704d9a8ec4ff8ad9f3ea5b P 197b880fa39d432a9b472845876bfaf0 reported cstate change: config changed from index 2 to 3, 7a9daf5d42a84615a6383cfec9a10608 (127.18.211.4) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "974b802141bd4f13beffef2aa03f88f9" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } attrs { replace: true } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } } }
I20260129 02:59:06.334285 20507 raft_consensus.cc:1275] T a609ac02fae84725ab743f95f51d33ce P 974b802141bd4f13beffef2aa03f88f9 [term 3 FOLLOWER]: Refusing update from remote peer 197b880fa39d432a9b472845876bfaf0: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 3 index: 5. (index mismatch)
I20260129 02:59:06.334595 20588 raft_consensus.cc:1275] T 7a9a69cd82724b1faed1d10aa091fc4a P 1c50de89a5a845668696dcb28c3badd6 [term 3 FOLLOWER]: Refusing update from remote peer 197b880fa39d432a9b472845876bfaf0: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20260129 02:59:06.337648 20590 raft_consensus.cc:1275] T 4f8f3a38f6e24ba4857875181f447442 P 1c50de89a5a845668696dcb28c3badd6 [term 3 FOLLOWER]: Refusing update from remote peer 974b802141bd4f13beffef2aa03f88f9: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20260129 02:59:06.341492 21478 consensus_queue.cc:1048] T a609ac02fae84725ab743f95f51d33ce P 197b880fa39d432a9b472845876bfaf0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.002s
I20260129 02:59:06.346581 21436 consensus_queue.cc:1048] T 4f8f3a38f6e24ba4857875181f447442 P 974b802141bd4f13beffef2aa03f88f9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260129 02:59:06.359597 21478 consensus_queue.cc:1048] T 7a9a69cd82724b1faed1d10aa091fc4a P 197b880fa39d432a9b472845876bfaf0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260129 02:59:06.371951 20335 catalog_manager.cc:5654] T 75be4bd6bd45474795fd472eff2953ac P 974b802141bd4f13beffef2aa03f88f9 reported cstate change: term changed from 1 to 3, leader changed from 197b880fa39d432a9b472845876bfaf0 (127.18.211.1) to 974b802141bd4f13beffef2aa03f88f9 (127.18.211.2). New cstate: current_term: 3 leader_uuid: "974b802141bd4f13beffef2aa03f88f9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } health_report { overall_health: UNKNOWN } } }
I20260129 02:59:06.379184 21429 raft_consensus.cc:493] T cb0b3f259ebc4310927338a8740e87b8 P 1c50de89a5a845668696dcb28c3badd6 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260129 02:59:06.379752 21429 raft_consensus.cc:515] T cb0b3f259ebc4310927338a8740e87b8 P 1c50de89a5a845668696dcb28c3badd6 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } }
I20260129 02:59:06.381767 21429 leader_election.cc:290] T cb0b3f259ebc4310927338a8740e87b8 P 1c50de89a5a845668696dcb28c3badd6 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 974b802141bd4f13beffef2aa03f88f9 (127.18.211.2:34517), 7a9daf5d42a84615a6383cfec9a10608 (127.18.211.4:40439)
I20260129 02:59:06.382741 21135 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "cb0b3f259ebc4310927338a8740e87b8" candidate_uuid: "1c50de89a5a845668696dcb28c3badd6" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "7a9daf5d42a84615a6383cfec9a10608" is_pre_election: true
I20260129 02:59:06.400445 20507 consensus_queue.cc:237] T 75f45538dc704d9a8ec4ff8ad9f3ea5b P 974b802141bd4f13beffef2aa03f88f9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } }
I20260129 02:59:06.417598 20437 raft_consensus.cc:1275] T 0f20af7fbe5d4db89a6f17948d98912c P 197b880fa39d432a9b472845876bfaf0 [term 3 FOLLOWER]: Refusing update from remote peer 974b802141bd4f13beffef2aa03f88f9: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20260129 02:59:06.418608 20441 consensus_queue.cc:237] T 0678b8b468bd43cdbc3e485900844821 P 197b880fa39d432a9b472845876bfaf0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } }
I20260129 02:59:06.419610 21398 consensus_queue.cc:1048] T 0f20af7fbe5d4db89a6f17948d98912c P 974b802141bd4f13beffef2aa03f88f9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260129 02:59:06.428719 21135 raft_consensus.cc:1275] T 0678b8b468bd43cdbc3e485900844821 P 7a9daf5d42a84615a6383cfec9a10608 [term 1 FOLLOWER]: Refusing update from remote peer 197b880fa39d432a9b472845876bfaf0: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260129 02:59:06.429494 20508 raft_consensus.cc:1275] T 0678b8b468bd43cdbc3e485900844821 P 974b802141bd4f13beffef2aa03f88f9 [term 1 FOLLOWER]: Refusing update from remote peer 197b880fa39d432a9b472845876bfaf0: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260129 02:59:06.430629 21478 consensus_queue.cc:1048] T 0678b8b468bd43cdbc3e485900844821 P 197b880fa39d432a9b472845876bfaf0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260129 02:59:06.431581 21386 consensus_queue.cc:1048] T 0678b8b468bd43cdbc3e485900844821 P 197b880fa39d432a9b472845876bfaf0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260129 02:59:06.448997 21135 raft_consensus.cc:1275] T 75f45538dc704d9a8ec4ff8ad9f3ea5b P 7a9daf5d42a84615a6383cfec9a10608 [term 1 FOLLOWER]: Refusing update from remote peer 974b802141bd4f13beffef2aa03f88f9: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260129 02:59:06.452131 21474 consensus_queue.cc:1048] T 75f45538dc704d9a8ec4ff8ad9f3ea5b P 974b802141bd4f13beffef2aa03f88f9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260129 02:59:06.456977 20507 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "cb0b3f259ebc4310927338a8740e87b8" candidate_uuid: "1c50de89a5a845668696dcb28c3badd6" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "974b802141bd4f13beffef2aa03f88f9" is_pre_election: true
I20260129 02:59:06.457679 20507 raft_consensus.cc:2468] T cb0b3f259ebc4310927338a8740e87b8 P 974b802141bd4f13beffef2aa03f88f9 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1c50de89a5a845668696dcb28c3badd6 in term 2.
I20260129 02:59:06.476598 20585 raft_consensus.cc:1275] T 0f20af7fbe5d4db89a6f17948d98912c P 1c50de89a5a845668696dcb28c3badd6 [term 3 FOLLOWER]: Refusing update from remote peer 974b802141bd4f13beffef2aa03f88f9: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20260129 02:59:06.479912 20556 leader_election.cc:304] T cb0b3f259ebc4310927338a8740e87b8 P 1c50de89a5a845668696dcb28c3badd6 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1c50de89a5a845668696dcb28c3badd6, 974b802141bd4f13beffef2aa03f88f9; no voters: 7a9daf5d42a84615a6383cfec9a10608
I20260129 02:59:06.457837 20513 raft_consensus.cc:2955] T 0678b8b468bd43cdbc3e485900844821 P 974b802141bd4f13beffef2aa03f88f9 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } } }
I20260129 02:59:06.486370 20583 raft_consensus.cc:1275] T 75f45538dc704d9a8ec4ff8ad9f3ea5b P 1c50de89a5a845668696dcb28c3badd6 [term 1 FOLLOWER]: Refusing update from remote peer 974b802141bd4f13beffef2aa03f88f9: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260129 02:59:06.488214 21436 consensus_queue.cc:1048] T 75f45538dc704d9a8ec4ff8ad9f3ea5b P 974b802141bd4f13beffef2aa03f88f9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260129 02:59:06.488870 21429 raft_consensus.cc:2804] T cb0b3f259ebc4310927338a8740e87b8 P 1c50de89a5a845668696dcb28c3badd6 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260129 02:59:06.489259 21429 raft_consensus.cc:493] T cb0b3f259ebc4310927338a8740e87b8 P 1c50de89a5a845668696dcb28c3badd6 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260129 02:59:06.489578 21429 raft_consensus.cc:3060] T cb0b3f259ebc4310927338a8740e87b8 P 1c50de89a5a845668696dcb28c3badd6 [term 2 FOLLOWER]: Advancing to term 3
I20260129 02:59:06.452903 21478 raft_consensus.cc:2955] T 0678b8b468bd43cdbc3e485900844821 P 197b880fa39d432a9b472845876bfaf0 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } } }
I20260129 02:59:06.480381 21398 consensus_queue.cc:1048] T 0f20af7fbe5d4db89a6f17948d98912c P 974b802141bd4f13beffef2aa03f88f9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.019s
I20260129 02:59:06.524783 20508 raft_consensus.cc:1275] T c891d2bff4ce4568861bbc992e91fbb0 P 974b802141bd4f13beffef2aa03f88f9 [term 3 FOLLOWER]: Refusing update from remote peer 197b880fa39d432a9b472845876bfaf0: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20260129 02:59:06.525105 21429 raft_consensus.cc:515] T cb0b3f259ebc4310927338a8740e87b8 P 1c50de89a5a845668696dcb28c3badd6 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } }
I20260129 02:59:06.526813 21409 raft_consensus.cc:2955] T 75f45538dc704d9a8ec4ff8ad9f3ea5b P 974b802141bd4f13beffef2aa03f88f9 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 197b880fa39d432a9b472845876bfaf0 (127.18.211.1) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } } }
I20260129 02:59:06.544224 21429 leader_election.cc:290] T cb0b3f259ebc4310927338a8740e87b8 P 1c50de89a5a845668696dcb28c3badd6 [CANDIDATE]: Term 3 election: Requested vote from peers 974b802141bd4f13beffef2aa03f88f9 (127.18.211.2:34517), 7a9daf5d42a84615a6383cfec9a10608 (127.18.211.4:40439)
I20260129 02:59:06.528790 21477 consensus_queue.cc:1048] T c891d2bff4ce4568861bbc992e91fbb0 P 197b880fa39d432a9b472845876bfaf0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.021s
W20260129 02:59:06.593832 20405 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.211.3:37735, user_credentials={real_user=slave}} blocked reactor thread for 101050us
I20260129 02:59:06.599056 20325 catalog_manager.cc:5167] ChangeConfig:REMOVE_PEER RPC for tablet 75f45538dc704d9a8ec4ff8ad9f3ea5b with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260129 02:59:06.570674 21134 raft_consensus.cc:2955] T 75f45538dc704d9a8ec4ff8ad9f3ea5b P 7a9daf5d42a84615a6383cfec9a10608 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 197b880fa39d432a9b472845876bfaf0 (127.18.211.1) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } } }
I20260129 02:59:06.634243 20322 catalog_manager.cc:5167] ChangeConfig:REMOVE_PEER RPC for tablet 0678b8b468bd43cdbc3e485900844821 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260129 02:59:06.647418 20510 raft_consensus.cc:1275] T cb0b3f259ebc4310927338a8740e87b8 P 974b802141bd4f13beffef2aa03f88f9 [term 2 FOLLOWER]: Refusing update from remote peer 7a9daf5d42a84615a6383cfec9a10608: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260129 02:59:06.656678 21131 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "cb0b3f259ebc4310927338a8740e87b8" candidate_uuid: "1c50de89a5a845668696dcb28c3badd6" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "7a9daf5d42a84615a6383cfec9a10608"
I20260129 02:59:06.666105 21486 consensus_queue.cc:1048] T cb0b3f259ebc4310927338a8740e87b8 P 7a9daf5d42a84615a6383cfec9a10608 [LEADER]: Connected to new peer: Peer: permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260129 02:59:06.670778 20437 raft_consensus.cc:1275] T 75be4bd6bd45474795fd472eff2953ac P 197b880fa39d432a9b472845876bfaf0 [term 3 FOLLOWER]: Refusing update from remote peer 974b802141bd4f13beffef2aa03f88f9: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20260129 02:59:06.674211 21479 consensus_queue.cc:1048] T 75be4bd6bd45474795fd472eff2953ac P 974b802141bd4f13beffef2aa03f88f9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260129 02:59:06.685284 20511 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "cb0b3f259ebc4310927338a8740e87b8" candidate_uuid: "1c50de89a5a845668696dcb28c3badd6" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "974b802141bd4f13beffef2aa03f88f9"
I20260129 02:59:06.686762 20585 raft_consensus.cc:1240] T cb0b3f259ebc4310927338a8740e87b8 P 1c50de89a5a845668696dcb28c3badd6 [term 3 FOLLOWER]: Rejecting Update request from peer 7a9daf5d42a84615a6383cfec9a10608 for earlier term 2. Current term is 3. Ops: []
I20260129 02:59:06.688117 20587 raft_consensus.cc:1275] T 75be4bd6bd45474795fd472eff2953ac P 1c50de89a5a845668696dcb28c3badd6 [term 3 FOLLOWER]: Refusing update from remote peer 974b802141bd4f13beffef2aa03f88f9: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20260129 02:59:06.696295 20337 catalog_manager.cc:5654] T 0678b8b468bd43cdbc3e485900844821 P 197b880fa39d432a9b472845876bfaf0 reported cstate change: config changed from index 3 to 4, VOTER 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3) evicted. New cstate: current_term: 1 leader_uuid: "197b880fa39d432a9b472845876bfaf0" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20260129 02:59:06.701076 20338 catalog_manager.cc:5654] T 75f45538dc704d9a8ec4ff8ad9f3ea5b P 7a9daf5d42a84615a6383cfec9a10608 reported cstate change: config changed from index 3 to 4, VOTER 197b880fa39d432a9b472845876bfaf0 (127.18.211.1) evicted. New cstate: current_term: 1 leader_uuid: "974b802141bd4f13beffef2aa03f88f9" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } } }
I20260129 02:59:06.694545 21486 consensus_queue.cc:1059] T cb0b3f259ebc4310927338a8740e87b8 P 7a9daf5d42a84615a6383cfec9a10608 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 }, Status: INVALID_TERM, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.005s
I20260129 02:59:06.707787 20556 leader_election.cc:304] T cb0b3f259ebc4310927338a8740e87b8 P 1c50de89a5a845668696dcb28c3badd6 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1c50de89a5a845668696dcb28c3badd6; no voters: 7a9daf5d42a84615a6383cfec9a10608, 974b802141bd4f13beffef2aa03f88f9
I20260129 02:59:06.714836 21403 raft_consensus.cc:2749] T cb0b3f259ebc4310927338a8740e87b8 P 1c50de89a5a845668696dcb28c3badd6 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260129 02:59:06.715008 21424 raft_consensus.cc:3055] T cb0b3f259ebc4310927338a8740e87b8 P 7a9daf5d42a84615a6383cfec9a10608 [term 2 LEADER]: Stepping down as leader of term 2
I20260129 02:59:06.715466 21424 raft_consensus.cc:740] T cb0b3f259ebc4310927338a8740e87b8 P 7a9daf5d42a84615a6383cfec9a10608 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 7a9daf5d42a84615a6383cfec9a10608, State: Running, Role: LEADER
I20260129 02:59:06.694522 21409 consensus_queue.cc:1048] T 75be4bd6bd45474795fd472eff2953ac P 974b802141bd4f13beffef2aa03f88f9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.014s
I20260129 02:59:06.716276 21424 consensus_queue.cc:260] T cb0b3f259ebc4310927338a8740e87b8 P 7a9daf5d42a84615a6383cfec9a10608 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 2.5, Last appended by leader: 5, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } }
I20260129 02:59:06.717391 21424 raft_consensus.cc:3060] T cb0b3f259ebc4310927338a8740e87b8 P 7a9daf5d42a84615a6383cfec9a10608 [term 2 FOLLOWER]: Advancing to term 3
I20260129 02:59:06.779235 20589 raft_consensus.cc:2955] T 75f45538dc704d9a8ec4ff8ad9f3ea5b P 1c50de89a5a845668696dcb28c3badd6 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 197b880fa39d432a9b472845876bfaf0 (127.18.211.1) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } } }
W20260129 02:59:06.887691 20556 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.211.2:34517, user_credentials={real_user=slave}} blocked reactor thread for 179983us
W20260129 02:59:06.901429 20407 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.211.2:34517, user_credentials={real_user=slave}} blocked reactor thread for 192415us
I20260129 02:59:06.913340 21134 raft_consensus.cc:2955] T 0678b8b468bd43cdbc3e485900844821 P 7a9daf5d42a84615a6383cfec9a10608 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } } }
I20260129 02:59:06.919476 20431 tablet_service.cc:1552] Processing DeleteTablet for tablet 75f45538dc704d9a8ec4ff8ad9f3ea5b with delete_type TABLET_DATA_TOMBSTONED (TS 197b880fa39d432a9b472845876bfaf0 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:55666
I20260129 02:59:06.939762 20579 tablet_service.cc:1552] Processing DeleteTablet for tablet 0678b8b468bd43cdbc3e485900844821 with delete_type TABLET_DATA_TOMBSTONED (TS 1c50de89a5a845668696dcb28c3badd6 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:35744
I20260129 02:59:06.952334 21524 tablet_replica.cc:333] T 0678b8b468bd43cdbc3e485900844821 P 1c50de89a5a845668696dcb28c3badd6: stopping tablet replica
I20260129 02:59:06.953541 21524 raft_consensus.cc:2243] T 0678b8b468bd43cdbc3e485900844821 P 1c50de89a5a845668696dcb28c3badd6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260129 02:59:06.955305 21524 raft_consensus.cc:2272] T 0678b8b468bd43cdbc3e485900844821 P 1c50de89a5a845668696dcb28c3badd6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260129 02:59:06.965934 21524 ts_tablet_manager.cc:1916] T 0678b8b468bd43cdbc3e485900844821 P 1c50de89a5a845668696dcb28c3badd6: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260129 02:59:06.978977 21523 tablet_replica.cc:333] T 75f45538dc704d9a8ec4ff8ad9f3ea5b P 197b880fa39d432a9b472845876bfaf0: stopping tablet replica
I20260129 02:59:06.985790 21523 raft_consensus.cc:2243] T 75f45538dc704d9a8ec4ff8ad9f3ea5b P 197b880fa39d432a9b472845876bfaf0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260129 02:59:06.986563 21523 raft_consensus.cc:2272] T 75f45538dc704d9a8ec4ff8ad9f3ea5b P 197b880fa39d432a9b472845876bfaf0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260129 02:59:06.990448 21523 ts_tablet_manager.cc:1916] T 75f45538dc704d9a8ec4ff8ad9f3ea5b P 197b880fa39d432a9b472845876bfaf0: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260129 02:59:06.997998 21524 ts_tablet_manager.cc:1929] T 0678b8b468bd43cdbc3e485900844821 P 1c50de89a5a845668696dcb28c3badd6: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20260129 02:59:06.998596 21524 log.cc:1199] T 0678b8b468bd43cdbc3e485900844821 P 1c50de89a5a845668696dcb28c3badd6: Deleting WAL directory at /tmp/dist-test-taskrTP_Jo/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1769655460890924-19276-0/minicluster-data/ts-2-root/wals/0678b8b468bd43cdbc3e485900844821
I20260129 02:59:07.006197 20590 raft_consensus.cc:1275] T c891d2bff4ce4568861bbc992e91fbb0 P 1c50de89a5a845668696dcb28c3badd6 [term 3 FOLLOWER]: Refusing update from remote peer 197b880fa39d432a9b472845876bfaf0: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20260129 02:59:07.018788 21523 ts_tablet_manager.cc:1929] T 75f45538dc704d9a8ec4ff8ad9f3ea5b P 197b880fa39d432a9b472845876bfaf0: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20260129 02:59:07.019351 21523 log.cc:1199] T 75f45538dc704d9a8ec4ff8ad9f3ea5b P 197b880fa39d432a9b472845876bfaf0: Deleting WAL directory at /tmp/dist-test-taskrTP_Jo/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1769655460890924-19276-0/minicluster-data/ts-0-root/wals/75f45538dc704d9a8ec4ff8ad9f3ea5b
I20260129 02:59:07.020655 21386 consensus_queue.cc:1048] T c891d2bff4ce4568861bbc992e91fbb0 P 197b880fa39d432a9b472845876bfaf0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260129 02:59:07.022010 20322 catalog_manager.cc:4985] TS 197b880fa39d432a9b472845876bfaf0 (127.18.211.1:42027): tablet 75f45538dc704d9a8ec4ff8ad9f3ea5b (table test-workload [id=de73611ac8e3485a98aecc5add247032]) successfully deleted
I20260129 02:59:07.023617 20323 catalog_manager.cc:4985] TS 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3:37735): tablet 0678b8b468bd43cdbc3e485900844821 (table test-workload [id=de73611ac8e3485a98aecc5add247032]) successfully deleted
I20260129 02:59:07.747381 20437 consensus_queue.cc:237] T 4e3805ffa70242e8ade66da260dddf66 P 197b880fa39d432a9b472845876bfaf0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } attrs { replace: true } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: NON_VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: true } }
I20260129 02:59:07.754146 20589 raft_consensus.cc:1275] T 4e3805ffa70242e8ade66da260dddf66 P 1c50de89a5a845668696dcb28c3badd6 [term 1 FOLLOWER]: Refusing update from remote peer 197b880fa39d432a9b472845876bfaf0: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20260129 02:59:07.754235 20405 consensus_peers.cc:597] T 4e3805ffa70242e8ade66da260dddf66 P 197b880fa39d432a9b472845876bfaf0 -> Peer 7a9daf5d42a84615a6383cfec9a10608 (127.18.211.4:40439): Couldn't send request to peer 7a9daf5d42a84615a6383cfec9a10608. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 4e3805ffa70242e8ade66da260dddf66. This is attempt 1: this message will repeat every 5th retry.
I20260129 02:59:07.754371 20507 raft_consensus.cc:1275] T 4e3805ffa70242e8ade66da260dddf66 P 974b802141bd4f13beffef2aa03f88f9 [term 1 FOLLOWER]: Refusing update from remote peer 197b880fa39d432a9b472845876bfaf0: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260129 02:59:07.755507 21519 consensus_queue.cc:1048] T 4e3805ffa70242e8ade66da260dddf66 P 197b880fa39d432a9b472845876bfaf0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260129 02:59:07.756470 21510 consensus_queue.cc:1048] T 4e3805ffa70242e8ade66da260dddf66 P 197b880fa39d432a9b472845876bfaf0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260129 02:59:07.765564 21386 raft_consensus.cc:2955] T 4e3805ffa70242e8ade66da260dddf66 P 197b880fa39d432a9b472845876bfaf0 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 7a9daf5d42a84615a6383cfec9a10608 (127.18.211.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } attrs { replace: true } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: NON_VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: true } } }
I20260129 02:59:07.781991 20510 consensus_queue.cc:237] T 2ac659d732314e1288849f08fe5e074e P 974b802141bd4f13beffef2aa03f88f9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } attrs { replace: true } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: NON_VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: true } }
I20260129 02:59:07.789206 20507 raft_consensus.cc:2955] T 4e3805ffa70242e8ade66da260dddf66 P 974b802141bd4f13beffef2aa03f88f9 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 7a9daf5d42a84615a6383cfec9a10608 (127.18.211.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } attrs { replace: true } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: NON_VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: true } } }
I20260129 02:59:07.810544 20335 catalog_manager.cc:5654] T 4e3805ffa70242e8ade66da260dddf66 P 197b880fa39d432a9b472845876bfaf0 reported cstate change: config changed from index -1 to 2, NON_VOTER 7a9daf5d42a84615a6383cfec9a10608 (127.18.211.4) added. New cstate: current_term: 1 leader_uuid: "197b880fa39d432a9b472845876bfaf0" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: NON_VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20260129 02:59:07.825766 20590 raft_consensus.cc:2955] T 4e3805ffa70242e8ade66da260dddf66 P 1c50de89a5a845668696dcb28c3badd6 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 7a9daf5d42a84615a6383cfec9a10608 (127.18.211.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } attrs { replace: true } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: NON_VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: true } } }
I20260129 02:59:07.828981 20589 raft_consensus.cc:1275] T 2ac659d732314e1288849f08fe5e074e P 1c50de89a5a845668696dcb28c3badd6 [term 1 FOLLOWER]: Refusing update from remote peer 974b802141bd4f13beffef2aa03f88f9: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260129 02:59:07.829474 20437 raft_consensus.cc:1275] T 2ac659d732314e1288849f08fe5e074e P 197b880fa39d432a9b472845876bfaf0 [term 1 FOLLOWER]: Refusing update from remote peer 974b802141bd4f13beffef2aa03f88f9: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260129 02:59:07.831074 21474 consensus_queue.cc:1048] T 2ac659d732314e1288849f08fe5e074e P 974b802141bd4f13beffef2aa03f88f9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
W20260129 02:59:07.832749 20480 consensus_peers.cc:597] T 2ac659d732314e1288849f08fe5e074e P 974b802141bd4f13beffef2aa03f88f9 -> Peer 7a9daf5d42a84615a6383cfec9a10608 (127.18.211.4:40439): Couldn't send request to peer 7a9daf5d42a84615a6383cfec9a10608. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 2ac659d732314e1288849f08fe5e074e. This is attempt 1: this message will repeat every 5th retry.
I20260129 02:59:07.832818 21474 consensus_queue.cc:1048] T 2ac659d732314e1288849f08fe5e074e P 974b802141bd4f13beffef2aa03f88f9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260129 02:59:07.864616 21513 raft_consensus.cc:2955] T 2ac659d732314e1288849f08fe5e074e P 974b802141bd4f13beffef2aa03f88f9 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 7a9daf5d42a84615a6383cfec9a10608 (127.18.211.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } attrs { replace: true } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: NON_VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: true } } }
I20260129 02:59:07.875579 20581 raft_consensus.cc:2955] T 2ac659d732314e1288849f08fe5e074e P 1c50de89a5a845668696dcb28c3badd6 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 7a9daf5d42a84615a6383cfec9a10608 (127.18.211.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } attrs { replace: true } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: NON_VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: true } } }
I20260129 02:59:07.882941 20437 raft_consensus.cc:2955] T 2ac659d732314e1288849f08fe5e074e P 197b880fa39d432a9b472845876bfaf0 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 7a9daf5d42a84615a6383cfec9a10608 (127.18.211.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } attrs { replace: true } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: NON_VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: true } } }
I20260129 02:59:07.895269 20581 consensus_queue.cc:237] T 75b0b6e82ddc4f92b1fe1d94f9a0b73b P 1c50de89a5a845668696dcb28c3badd6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } attrs { replace: true } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: NON_VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: true } }
I20260129 02:59:07.915748 20510 raft_consensus.cc:1275] T 75b0b6e82ddc4f92b1fe1d94f9a0b73b P 974b802141bd4f13beffef2aa03f88f9 [term 1 FOLLOWER]: Refusing update from remote peer 1c50de89a5a845668696dcb28c3badd6: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20260129 02:59:07.916877 20554 consensus_peers.cc:597] T 75b0b6e82ddc4f92b1fe1d94f9a0b73b P 1c50de89a5a845668696dcb28c3badd6 -> Peer 7a9daf5d42a84615a6383cfec9a10608 (127.18.211.4:40439): Couldn't send request to peer 7a9daf5d42a84615a6383cfec9a10608. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 75b0b6e82ddc4f92b1fe1d94f9a0b73b. This is attempt 1: this message will repeat every 5th retry.
I20260129 02:59:07.917985 21429 consensus_queue.cc:1048] T 75b0b6e82ddc4f92b1fe1d94f9a0b73b P 1c50de89a5a845668696dcb28c3badd6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260129 02:59:07.927183 20437 raft_consensus.cc:1275] T 75b0b6e82ddc4f92b1fe1d94f9a0b73b P 197b880fa39d432a9b472845876bfaf0 [term 1 FOLLOWER]: Refusing update from remote peer 1c50de89a5a845668696dcb28c3badd6: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260129 02:59:07.929416 21511 consensus_queue.cc:1048] T 75b0b6e82ddc4f92b1fe1d94f9a0b73b P 1c50de89a5a845668696dcb28c3badd6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.023s
I20260129 02:59:07.951272 20336 catalog_manager.cc:5654] T 2ac659d732314e1288849f08fe5e074e P 197b880fa39d432a9b472845876bfaf0 reported cstate change: config changed from index -1 to 2, NON_VOTER 7a9daf5d42a84615a6383cfec9a10608 (127.18.211.4) added. New cstate: current_term: 1 leader_uuid: "974b802141bd4f13beffef2aa03f88f9" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } attrs { replace: true } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: NON_VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: true } } }
I20260129 02:59:07.983547 21429 raft_consensus.cc:2955] T 75b0b6e82ddc4f92b1fe1d94f9a0b73b P 1c50de89a5a845668696dcb28c3badd6 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 7a9daf5d42a84615a6383cfec9a10608 (127.18.211.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } attrs { replace: true } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: NON_VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: true } } }
I20260129 02:59:07.998551 20441 consensus_queue.cc:237] T 4e420f90884f4796997b5ea41e49e411 P 197b880fa39d432a9b472845876bfaf0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } attrs { replace: true } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: NON_VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: true } }
I20260129 02:59:08.021113 20435 raft_consensus.cc:2955] T 75b0b6e82ddc4f92b1fe1d94f9a0b73b P 197b880fa39d432a9b472845876bfaf0 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 7a9daf5d42a84615a6383cfec9a10608 (127.18.211.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } attrs { replace: true } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: NON_VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: true } } }
I20260129 02:59:08.029361 20507 raft_consensus.cc:1275] T 4e420f90884f4796997b5ea41e49e411 P 974b802141bd4f13beffef2aa03f88f9 [term 1 FOLLOWER]: Refusing update from remote peer 197b880fa39d432a9b472845876bfaf0: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20260129 02:59:08.030165 20405 consensus_peers.cc:597] T 4e420f90884f4796997b5ea41e49e411 P 197b880fa39d432a9b472845876bfaf0 -> Peer 7a9daf5d42a84615a6383cfec9a10608 (127.18.211.4:40439): Couldn't send request to peer 7a9daf5d42a84615a6383cfec9a10608. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 4e420f90884f4796997b5ea41e49e411. This is attempt 1: this message will repeat every 5th retry.
I20260129 02:59:08.031126 21519 consensus_queue.cc:1048] T 4e420f90884f4796997b5ea41e49e411 P 197b880fa39d432a9b472845876bfaf0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260129 02:59:08.038615 20511 raft_consensus.cc:2955] T 75b0b6e82ddc4f92b1fe1d94f9a0b73b P 974b802141bd4f13beffef2aa03f88f9 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 7a9daf5d42a84615a6383cfec9a10608 (127.18.211.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } attrs { replace: true } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: NON_VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: true } } }
I20260129 02:59:08.048622 20587 raft_consensus.cc:1275] T 4e420f90884f4796997b5ea41e49e411 P 1c50de89a5a845668696dcb28c3badd6 [term 1 FOLLOWER]: Refusing update from remote peer 197b880fa39d432a9b472845876bfaf0: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260129 02:59:08.048735 20337 catalog_manager.cc:5654] T 75b0b6e82ddc4f92b1fe1d94f9a0b73b P 1c50de89a5a845668696dcb28c3badd6 reported cstate change: config changed from index -1 to 2, NON_VOTER 7a9daf5d42a84615a6383cfec9a10608 (127.18.211.4) added. New cstate: current_term: 1 leader_uuid: "1c50de89a5a845668696dcb28c3badd6" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: NON_VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20260129 02:59:08.062641 21519 consensus_queue.cc:1048] T 4e420f90884f4796997b5ea41e49e411 P 197b880fa39d432a9b472845876bfaf0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260129 02:59:08.064977 21510 raft_consensus.cc:2955] T 4e420f90884f4796997b5ea41e49e411 P 197b880fa39d432a9b472845876bfaf0 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 7a9daf5d42a84615a6383cfec9a10608 (127.18.211.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } attrs { replace: true } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: NON_VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: true } } }
I20260129 02:59:08.069296 20510 raft_consensus.cc:2955] T 4e420f90884f4796997b5ea41e49e411 P 974b802141bd4f13beffef2aa03f88f9 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 7a9daf5d42a84615a6383cfec9a10608 (127.18.211.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } attrs { replace: true } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: NON_VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: true } } }
I20260129 02:59:08.090384 20335 catalog_manager.cc:5654] T 4e420f90884f4796997b5ea41e49e411 P 197b880fa39d432a9b472845876bfaf0 reported cstate change: config changed from index -1 to 2, NON_VOTER 7a9daf5d42a84615a6383cfec9a10608 (127.18.211.4) added. New cstate: current_term: 1 leader_uuid: "197b880fa39d432a9b472845876bfaf0" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: NON_VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20260129 02:59:08.114779 20590 raft_consensus.cc:2955] T 4e420f90884f4796997b5ea41e49e411 P 1c50de89a5a845668696dcb28c3badd6 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 7a9daf5d42a84615a6383cfec9a10608 (127.18.211.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } attrs { replace: true } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: NON_VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: true } } }
I20260129 02:59:08.166571 21546 ts_tablet_manager.cc:933] T 4e3805ffa70242e8ade66da260dddf66 P 7a9daf5d42a84615a6383cfec9a10608: Initiating tablet copy from peer 197b880fa39d432a9b472845876bfaf0 (127.18.211.1:42027)
I20260129 02:59:08.170296 21546 tablet_copy_client.cc:323] T 4e3805ffa70242e8ade66da260dddf66 P 7a9daf5d42a84615a6383cfec9a10608: tablet copy: Beginning tablet copy session from remote peer at address 127.18.211.1:42027
I20260129 02:59:08.172461 20450 tablet_copy_service.cc:140] P 197b880fa39d432a9b472845876bfaf0: Received BeginTabletCopySession request for tablet 4e3805ffa70242e8ade66da260dddf66 from peer 7a9daf5d42a84615a6383cfec9a10608 ({username='slave'} at 127.0.0.1:48356)
I20260129 02:59:08.173005 20450 tablet_copy_service.cc:161] P 197b880fa39d432a9b472845876bfaf0: Beginning new tablet copy session on tablet 4e3805ffa70242e8ade66da260dddf66 from peer 7a9daf5d42a84615a6383cfec9a10608 at {username='slave'} at 127.0.0.1:48356: session id = 7a9daf5d42a84615a6383cfec9a10608-4e3805ffa70242e8ade66da260dddf66
I20260129 02:59:08.183522 20450 tablet_copy_source_session.cc:215] T 4e3805ffa70242e8ade66da260dddf66 P 197b880fa39d432a9b472845876bfaf0: Tablet Copy: opened 0 blocks and 1 log segments
I20260129 02:59:08.185587 21513 raft_consensus.cc:493] T cb0b3f259ebc4310927338a8740e87b8 P 974b802141bd4f13beffef2aa03f88f9 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 7a9daf5d42a84615a6383cfec9a10608)
I20260129 02:59:08.186229 21513 raft_consensus.cc:515] T cb0b3f259ebc4310927338a8740e87b8 P 974b802141bd4f13beffef2aa03f88f9 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } }
I20260129 02:59:08.188481 21513 leader_election.cc:290] T cb0b3f259ebc4310927338a8740e87b8 P 974b802141bd4f13beffef2aa03f88f9 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3:37735), 7a9daf5d42a84615a6383cfec9a10608 (127.18.211.4:40439)
I20260129 02:59:08.189240 20590 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "cb0b3f259ebc4310927338a8740e87b8" candidate_uuid: "974b802141bd4f13beffef2aa03f88f9" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "1c50de89a5a845668696dcb28c3badd6" is_pre_election: true
I20260129 02:59:08.192142 20590 raft_consensus.cc:2393] T cb0b3f259ebc4310927338a8740e87b8 P 1c50de89a5a845668696dcb28c3badd6 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 974b802141bd4f13beffef2aa03f88f9 in current term 3: Already voted for candidate 1c50de89a5a845668696dcb28c3badd6 in this term.
I20260129 02:59:08.192916 21546 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4e3805ffa70242e8ade66da260dddf66. 1 dirs total, 0 dirs full, 0 dirs failed
I20260129 02:59:08.190374 21134 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "cb0b3f259ebc4310927338a8740e87b8" candidate_uuid: "974b802141bd4f13beffef2aa03f88f9" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "7a9daf5d42a84615a6383cfec9a10608" is_pre_election: true
I20260129 02:59:08.193900 21134 raft_consensus.cc:2468] T cb0b3f259ebc4310927338a8740e87b8 P 7a9daf5d42a84615a6383cfec9a10608 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 974b802141bd4f13beffef2aa03f88f9 in term 3.
I20260129 02:59:08.195271 20480 leader_election.cc:304] T cb0b3f259ebc4310927338a8740e87b8 P 974b802141bd4f13beffef2aa03f88f9 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 7a9daf5d42a84615a6383cfec9a10608, 974b802141bd4f13beffef2aa03f88f9; no voters: 1c50de89a5a845668696dcb28c3badd6
I20260129 02:59:08.198206 21513 raft_consensus.cc:2804] T cb0b3f259ebc4310927338a8740e87b8 P 974b802141bd4f13beffef2aa03f88f9 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260129 02:59:08.198546 21513 raft_consensus.cc:493] T cb0b3f259ebc4310927338a8740e87b8 P 974b802141bd4f13beffef2aa03f88f9 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 7a9daf5d42a84615a6383cfec9a10608)
I20260129 02:59:08.198911 21513 raft_consensus.cc:3060] T cb0b3f259ebc4310927338a8740e87b8 P 974b802141bd4f13beffef2aa03f88f9 [term 2 FOLLOWER]: Advancing to term 3
I20260129 02:59:08.211479 21546 tablet_copy_client.cc:806] T 4e3805ffa70242e8ade66da260dddf66 P 7a9daf5d42a84615a6383cfec9a10608: tablet copy: Starting download of 0 data blocks...
I20260129 02:59:08.213438 21546 tablet_copy_client.cc:670] T 4e3805ffa70242e8ade66da260dddf66 P 7a9daf5d42a84615a6383cfec9a10608: tablet copy: Starting download of 1 WAL segments...
I20260129 02:59:08.221253 21513 raft_consensus.cc:515] T cb0b3f259ebc4310927338a8740e87b8 P 974b802141bd4f13beffef2aa03f88f9 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } }
I20260129 02:59:08.220769 21546 tablet_copy_client.cc:538] T 4e3805ffa70242e8ade66da260dddf66 P 7a9daf5d42a84615a6383cfec9a10608: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260129 02:59:08.224014 20582 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "cb0b3f259ebc4310927338a8740e87b8" candidate_uuid: "974b802141bd4f13beffef2aa03f88f9" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "1c50de89a5a845668696dcb28c3badd6"
I20260129 02:59:08.224749 20582 raft_consensus.cc:2393] T cb0b3f259ebc4310927338a8740e87b8 P 1c50de89a5a845668696dcb28c3badd6 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 974b802141bd4f13beffef2aa03f88f9 in current term 3: Already voted for candidate 1c50de89a5a845668696dcb28c3badd6 in this term.
I20260129 02:59:08.227312 21134 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "cb0b3f259ebc4310927338a8740e87b8" candidate_uuid: "974b802141bd4f13beffef2aa03f88f9" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "7a9daf5d42a84615a6383cfec9a10608"
I20260129 02:59:08.232796 21546 tablet_bootstrap.cc:492] T 4e3805ffa70242e8ade66da260dddf66 P 7a9daf5d42a84615a6383cfec9a10608: Bootstrap starting.
I20260129 02:59:08.243062 21134 raft_consensus.cc:2468] T cb0b3f259ebc4310927338a8740e87b8 P 7a9daf5d42a84615a6383cfec9a10608 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 974b802141bd4f13beffef2aa03f88f9 in term 3.
I20260129 02:59:08.245594 20480 leader_election.cc:304] T cb0b3f259ebc4310927338a8740e87b8 P 974b802141bd4f13beffef2aa03f88f9 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 7a9daf5d42a84615a6383cfec9a10608, 974b802141bd4f13beffef2aa03f88f9; no voters: 1c50de89a5a845668696dcb28c3badd6
I20260129 02:59:08.246462 21533 raft_consensus.cc:2804] T cb0b3f259ebc4310927338a8740e87b8 P 974b802141bd4f13beffef2aa03f88f9 [term 3 FOLLOWER]: Leader election won for term 3
I20260129 02:59:08.246927 21533 raft_consensus.cc:697] T cb0b3f259ebc4310927338a8740e87b8 P 974b802141bd4f13beffef2aa03f88f9 [term 3 LEADER]: Becoming Leader. State: Replica: 974b802141bd4f13beffef2aa03f88f9, State: Running, Role: LEADER
I20260129 02:59:08.247736 21533 consensus_queue.cc:237] T cb0b3f259ebc4310927338a8740e87b8 P 974b802141bd4f13beffef2aa03f88f9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 2.5, Last appended by leader: 5, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } }
I20260129 02:59:08.256065 21513 leader_election.cc:290] T cb0b3f259ebc4310927338a8740e87b8 P 974b802141bd4f13beffef2aa03f88f9 [CANDIDATE]: Term 3 election: Requested vote from peers 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3:37735), 7a9daf5d42a84615a6383cfec9a10608 (127.18.211.4:40439)
I20260129 02:59:08.265954 21546 tablet_bootstrap.cc:492] T 4e3805ffa70242e8ade66da260dddf66 P 7a9daf5d42a84615a6383cfec9a10608: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260129 02:59:08.269845 20336 catalog_manager.cc:5654] T cb0b3f259ebc4310927338a8740e87b8 P 974b802141bd4f13beffef2aa03f88f9 reported cstate change: term changed from 2 to 3, leader changed from 7a9daf5d42a84615a6383cfec9a10608 (127.18.211.4) to 974b802141bd4f13beffef2aa03f88f9 (127.18.211.2). New cstate: current_term: 3 leader_uuid: "974b802141bd4f13beffef2aa03f88f9" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20260129 02:59:08.275231 21546 tablet_bootstrap.cc:492] T 4e3805ffa70242e8ade66da260dddf66 P 7a9daf5d42a84615a6383cfec9a10608: Bootstrap complete.
I20260129 02:59:08.276041 21546 ts_tablet_manager.cc:1403] T 4e3805ffa70242e8ade66da260dddf66 P 7a9daf5d42a84615a6383cfec9a10608: Time spent bootstrapping tablet: real 0.043s user 0.022s sys 0.004s
I20260129 02:59:08.278934 21546 raft_consensus.cc:359] T 4e3805ffa70242e8ade66da260dddf66 P 7a9daf5d42a84615a6383cfec9a10608 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } attrs { replace: true } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: NON_VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: true } }
I20260129 02:59:08.279750 21546 raft_consensus.cc:740] T 4e3805ffa70242e8ade66da260dddf66 P 7a9daf5d42a84615a6383cfec9a10608 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 7a9daf5d42a84615a6383cfec9a10608, State: Initialized, Role: LEARNER
I20260129 02:59:08.280403 21546 consensus_queue.cc:260] T 4e3805ffa70242e8ade66da260dddf66 P 7a9daf5d42a84615a6383cfec9a10608 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } attrs { replace: true } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: NON_VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: true } }
I20260129 02:59:08.300984 21546 ts_tablet_manager.cc:1434] T 4e3805ffa70242e8ade66da260dddf66 P 7a9daf5d42a84615a6383cfec9a10608: Time spent starting tablet: real 0.025s user 0.010s sys 0.012s
I20260129 02:59:08.303318 20450 tablet_copy_service.cc:342] P 197b880fa39d432a9b472845876bfaf0: Request end of tablet copy session 7a9daf5d42a84615a6383cfec9a10608-4e3805ffa70242e8ade66da260dddf66 received from {username='slave'} at 127.0.0.1:48356
I20260129 02:59:08.303763 20450 tablet_copy_service.cc:434] P 197b880fa39d432a9b472845876bfaf0: ending tablet copy session 7a9daf5d42a84615a6383cfec9a10608-4e3805ffa70242e8ade66da260dddf66 on tablet 4e3805ffa70242e8ade66da260dddf66 with peer 7a9daf5d42a84615a6383cfec9a10608
I20260129 02:59:08.307796 21511 raft_consensus.cc:493] T cb0b3f259ebc4310927338a8740e87b8 P 1c50de89a5a845668696dcb28c3badd6 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260129 02:59:08.308317 21511 raft_consensus.cc:515] T cb0b3f259ebc4310927338a8740e87b8 P 1c50de89a5a845668696dcb28c3badd6 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } }
I20260129 02:59:08.311201 20510 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "cb0b3f259ebc4310927338a8740e87b8" candidate_uuid: "1c50de89a5a845668696dcb28c3badd6" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "974b802141bd4f13beffef2aa03f88f9" is_pre_election: true
I20260129 02:59:08.313125 21511 leader_election.cc:290] T cb0b3f259ebc4310927338a8740e87b8 P 1c50de89a5a845668696dcb28c3badd6 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 974b802141bd4f13beffef2aa03f88f9 (127.18.211.2:34517), 7a9daf5d42a84615a6383cfec9a10608 (127.18.211.4:40439)
I20260129 02:59:08.314407 21134 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "cb0b3f259ebc4310927338a8740e87b8" candidate_uuid: "1c50de89a5a845668696dcb28c3badd6" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "7a9daf5d42a84615a6383cfec9a10608" is_pre_election: true
I20260129 02:59:08.315622 21134 raft_consensus.cc:2410] T cb0b3f259ebc4310927338a8740e87b8 P 7a9daf5d42a84615a6383cfec9a10608 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1c50de89a5a845668696dcb28c3badd6 for term 4 because replica has last-logged OpId of term: 2 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 4.
I20260129 02:59:08.317016 20554 leader_election.cc:304] T cb0b3f259ebc4310927338a8740e87b8 P 1c50de89a5a845668696dcb28c3badd6 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1c50de89a5a845668696dcb28c3badd6; no voters: 7a9daf5d42a84615a6383cfec9a10608, 974b802141bd4f13beffef2aa03f88f9
I20260129 02:59:08.318109 21511 raft_consensus.cc:2749] T cb0b3f259ebc4310927338a8740e87b8 P 1c50de89a5a845668696dcb28c3badd6 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260129 02:59:08.388734 21546 ts_tablet_manager.cc:933] T 75b0b6e82ddc4f92b1fe1d94f9a0b73b P 7a9daf5d42a84615a6383cfec9a10608: Initiating tablet copy from peer 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3:37735)
I20260129 02:59:08.390573 21546 tablet_copy_client.cc:323] T 75b0b6e82ddc4f92b1fe1d94f9a0b73b P 7a9daf5d42a84615a6383cfec9a10608: tablet copy: Beginning tablet copy session from remote peer at address 127.18.211.3:37735
I20260129 02:59:08.392021 20600 tablet_copy_service.cc:140] P 1c50de89a5a845668696dcb28c3badd6: Received BeginTabletCopySession request for tablet 75b0b6e82ddc4f92b1fe1d94f9a0b73b from peer 7a9daf5d42a84615a6383cfec9a10608 ({username='slave'} at 127.0.0.1:48526)
I20260129 02:59:08.392484 20600 tablet_copy_service.cc:161] P 1c50de89a5a845668696dcb28c3badd6: Beginning new tablet copy session on tablet 75b0b6e82ddc4f92b1fe1d94f9a0b73b from peer 7a9daf5d42a84615a6383cfec9a10608 at {username='slave'} at 127.0.0.1:48526: session id = 7a9daf5d42a84615a6383cfec9a10608-75b0b6e82ddc4f92b1fe1d94f9a0b73b
I20260129 02:59:08.400198 20600 tablet_copy_source_session.cc:215] T 75b0b6e82ddc4f92b1fe1d94f9a0b73b P 1c50de89a5a845668696dcb28c3badd6: Tablet Copy: opened 0 blocks and 1 log segments
I20260129 02:59:08.404420 21546 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 75b0b6e82ddc4f92b1fe1d94f9a0b73b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260129 02:59:08.425287 21546 tablet_copy_client.cc:806] T 75b0b6e82ddc4f92b1fe1d94f9a0b73b P 7a9daf5d42a84615a6383cfec9a10608: tablet copy: Starting download of 0 data blocks...
I20260129 02:59:08.425805 21546 tablet_copy_client.cc:670] T 75b0b6e82ddc4f92b1fe1d94f9a0b73b P 7a9daf5d42a84615a6383cfec9a10608: tablet copy: Starting download of 1 WAL segments...
I20260129 02:59:08.429719 21546 tablet_copy_client.cc:538] T 75b0b6e82ddc4f92b1fe1d94f9a0b73b P 7a9daf5d42a84615a6383cfec9a10608: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260129 02:59:08.438087 21546 tablet_bootstrap.cc:492] T 75b0b6e82ddc4f92b1fe1d94f9a0b73b P 7a9daf5d42a84615a6383cfec9a10608: Bootstrap starting.
I20260129 02:59:08.469313 21552 ts_tablet_manager.cc:933] T 2ac659d732314e1288849f08fe5e074e P 7a9daf5d42a84615a6383cfec9a10608: Initiating tablet copy from peer 974b802141bd4f13beffef2aa03f88f9 (127.18.211.2:34517)
I20260129 02:59:08.479797 21552 tablet_copy_client.cc:323] T 2ac659d732314e1288849f08fe5e074e P 7a9daf5d42a84615a6383cfec9a10608: tablet copy: Beginning tablet copy session from remote peer at address 127.18.211.2:34517
I20260129 02:59:08.482209 20526 tablet_copy_service.cc:140] P 974b802141bd4f13beffef2aa03f88f9: Received BeginTabletCopySession request for tablet 2ac659d732314e1288849f08fe5e074e from peer 7a9daf5d42a84615a6383cfec9a10608 ({username='slave'} at 127.0.0.1:48544)
I20260129 02:59:08.482982 20526 tablet_copy_service.cc:161] P 974b802141bd4f13beffef2aa03f88f9: Beginning new tablet copy session on tablet 2ac659d732314e1288849f08fe5e074e from peer 7a9daf5d42a84615a6383cfec9a10608 at {username='slave'} at 127.0.0.1:48544: session id = 7a9daf5d42a84615a6383cfec9a10608-2ac659d732314e1288849f08fe5e074e
I20260129 02:59:08.504647 20526 tablet_copy_source_session.cc:215] T 2ac659d732314e1288849f08fe5e074e P 974b802141bd4f13beffef2aa03f88f9: Tablet Copy: opened 0 blocks and 1 log segments
I20260129 02:59:08.524935 21546 tablet_bootstrap.cc:492] T 75b0b6e82ddc4f92b1fe1d94f9a0b73b P 7a9daf5d42a84615a6383cfec9a10608: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260129 02:59:08.531095 21546 tablet_bootstrap.cc:492] T 75b0b6e82ddc4f92b1fe1d94f9a0b73b P 7a9daf5d42a84615a6383cfec9a10608: Bootstrap complete.
I20260129 02:59:08.531904 21546 ts_tablet_manager.cc:1403] T 75b0b6e82ddc4f92b1fe1d94f9a0b73b P 7a9daf5d42a84615a6383cfec9a10608: Time spent bootstrapping tablet: real 0.094s user 0.035s sys 0.024s
I20260129 02:59:08.533990 21552 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2ac659d732314e1288849f08fe5e074e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260129 02:59:08.535293 21555 ts_tablet_manager.cc:933] T 4e420f90884f4796997b5ea41e49e411 P 7a9daf5d42a84615a6383cfec9a10608: Initiating tablet copy from peer 197b880fa39d432a9b472845876bfaf0 (127.18.211.1:42027)
I20260129 02:59:08.544844 21555 tablet_copy_client.cc:323] T 4e420f90884f4796997b5ea41e49e411 P 7a9daf5d42a84615a6383cfec9a10608: tablet copy: Beginning tablet copy session from remote peer at address 127.18.211.1:42027
I20260129 02:59:08.547159 20450 tablet_copy_service.cc:140] P 197b880fa39d432a9b472845876bfaf0: Received BeginTabletCopySession request for tablet 4e420f90884f4796997b5ea41e49e411 from peer 7a9daf5d42a84615a6383cfec9a10608 ({username='slave'} at 127.0.0.1:48356)
I20260129 02:59:08.547703 20450 tablet_copy_service.cc:161] P 197b880fa39d432a9b472845876bfaf0: Beginning new tablet copy session on tablet 4e420f90884f4796997b5ea41e49e411 from peer 7a9daf5d42a84615a6383cfec9a10608 at {username='slave'} at 127.0.0.1:48356: session id = 7a9daf5d42a84615a6383cfec9a10608-4e420f90884f4796997b5ea41e49e411
I20260129 02:59:08.548589 21546 raft_consensus.cc:359] T 75b0b6e82ddc4f92b1fe1d94f9a0b73b P 7a9daf5d42a84615a6383cfec9a10608 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } attrs { replace: true } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: NON_VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: true } }
I20260129 02:59:08.549511 21546 raft_consensus.cc:740] T 75b0b6e82ddc4f92b1fe1d94f9a0b73b P 7a9daf5d42a84615a6383cfec9a10608 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 7a9daf5d42a84615a6383cfec9a10608, State: Initialized, Role: LEARNER
I20260129 02:59:08.550217 21546 consensus_queue.cc:260] T 75b0b6e82ddc4f92b1fe1d94f9a0b73b P 7a9daf5d42a84615a6383cfec9a10608 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } attrs { replace: true } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: NON_VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: true } }
I20260129 02:59:08.553315 21546 ts_tablet_manager.cc:1434] T 75b0b6e82ddc4f92b1fe1d94f9a0b73b P 7a9daf5d42a84615a6383cfec9a10608: Time spent starting tablet: real 0.021s user 0.006s sys 0.000s
I20260129 02:59:08.556149 20600 tablet_copy_service.cc:342] P 1c50de89a5a845668696dcb28c3badd6: Request end of tablet copy session 7a9daf5d42a84615a6383cfec9a10608-75b0b6e82ddc4f92b1fe1d94f9a0b73b received from {username='slave'} at 127.0.0.1:48526
I20260129 02:59:08.556586 20600 tablet_copy_service.cc:434] P 1c50de89a5a845668696dcb28c3badd6: ending tablet copy session 7a9daf5d42a84615a6383cfec9a10608-75b0b6e82ddc4f92b1fe1d94f9a0b73b on tablet 75b0b6e82ddc4f92b1fe1d94f9a0b73b with peer 7a9daf5d42a84615a6383cfec9a10608
I20260129 02:59:08.562682 20450 tablet_copy_source_session.cc:215] T 4e420f90884f4796997b5ea41e49e411 P 197b880fa39d432a9b472845876bfaf0: Tablet Copy: opened 0 blocks and 1 log segments
I20260129 02:59:08.566267 21555 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4e420f90884f4796997b5ea41e49e411. 1 dirs total, 0 dirs full, 0 dirs failed
I20260129 02:59:08.568908 21552 tablet_copy_client.cc:806] T 2ac659d732314e1288849f08fe5e074e P 7a9daf5d42a84615a6383cfec9a10608: tablet copy: Starting download of 0 data blocks...
I20260129 02:59:08.571909 21552 tablet_copy_client.cc:670] T 2ac659d732314e1288849f08fe5e074e P 7a9daf5d42a84615a6383cfec9a10608: tablet copy: Starting download of 1 WAL segments...
I20260129 02:59:08.580169 21552 tablet_copy_client.cc:538] T 2ac659d732314e1288849f08fe5e074e P 7a9daf5d42a84615a6383cfec9a10608: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260129 02:59:08.583770 21555 tablet_copy_client.cc:806] T 4e420f90884f4796997b5ea41e49e411 P 7a9daf5d42a84615a6383cfec9a10608: tablet copy: Starting download of 0 data blocks...
I20260129 02:59:08.584453 21555 tablet_copy_client.cc:670] T 4e420f90884f4796997b5ea41e49e411 P 7a9daf5d42a84615a6383cfec9a10608: tablet copy: Starting download of 1 WAL segments...
I20260129 02:59:08.590646 21552 tablet_bootstrap.cc:492] T 2ac659d732314e1288849f08fe5e074e P 7a9daf5d42a84615a6383cfec9a10608: Bootstrap starting.
I20260129 02:59:08.595034 21555 tablet_copy_client.cc:538] T 4e420f90884f4796997b5ea41e49e411 P 7a9daf5d42a84615a6383cfec9a10608: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260129 02:59:08.613039 21555 tablet_bootstrap.cc:492] T 4e420f90884f4796997b5ea41e49e411 P 7a9daf5d42a84615a6383cfec9a10608: Bootstrap starting.
I20260129 02:59:08.628770 21552 tablet_bootstrap.cc:492] T 2ac659d732314e1288849f08fe5e074e P 7a9daf5d42a84615a6383cfec9a10608: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260129 02:59:08.648175 21552 tablet_bootstrap.cc:492] T 2ac659d732314e1288849f08fe5e074e P 7a9daf5d42a84615a6383cfec9a10608: Bootstrap complete.
I20260129 02:59:08.649115 21552 ts_tablet_manager.cc:1403] T 2ac659d732314e1288849f08fe5e074e P 7a9daf5d42a84615a6383cfec9a10608: Time spent bootstrapping tablet: real 0.059s user 0.028s sys 0.004s
I20260129 02:59:08.655586 21552 raft_consensus.cc:359] T 2ac659d732314e1288849f08fe5e074e P 7a9daf5d42a84615a6383cfec9a10608 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } attrs { replace: true } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: NON_VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: true } }
I20260129 02:59:08.658697 21552 raft_consensus.cc:740] T 2ac659d732314e1288849f08fe5e074e P 7a9daf5d42a84615a6383cfec9a10608 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 7a9daf5d42a84615a6383cfec9a10608, State: Initialized, Role: LEARNER
I20260129 02:59:08.659469 21552 consensus_queue.cc:260] T 2ac659d732314e1288849f08fe5e074e P 7a9daf5d42a84615a6383cfec9a10608 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } attrs { replace: true } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: NON_VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: true } }
I20260129 02:59:08.663877 21555 tablet_bootstrap.cc:492] T 4e420f90884f4796997b5ea41e49e411 P 7a9daf5d42a84615a6383cfec9a10608: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260129 02:59:08.664824 21555 tablet_bootstrap.cc:492] T 4e420f90884f4796997b5ea41e49e411 P 7a9daf5d42a84615a6383cfec9a10608: Bootstrap complete.
I20260129 02:59:08.665582 21555 ts_tablet_manager.cc:1403] T 4e420f90884f4796997b5ea41e49e411 P 7a9daf5d42a84615a6383cfec9a10608: Time spent bootstrapping tablet: real 0.053s user 0.021s sys 0.008s
I20260129 02:59:08.669332 21555 raft_consensus.cc:359] T 4e420f90884f4796997b5ea41e49e411 P 7a9daf5d42a84615a6383cfec9a10608 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } attrs { replace: true } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: NON_VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: true } }
I20260129 02:59:08.670233 21555 raft_consensus.cc:740] T 4e420f90884f4796997b5ea41e49e411 P 7a9daf5d42a84615a6383cfec9a10608 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 7a9daf5d42a84615a6383cfec9a10608, State: Initialized, Role: LEARNER
I20260129 02:59:08.671013 21555 consensus_queue.cc:260] T 4e420f90884f4796997b5ea41e49e411 P 7a9daf5d42a84615a6383cfec9a10608 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } attrs { replace: true } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: NON_VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: true } }
I20260129 02:59:08.673421 21552 ts_tablet_manager.cc:1434] T 2ac659d732314e1288849f08fe5e074e P 7a9daf5d42a84615a6383cfec9a10608: Time spent starting tablet: real 0.024s user 0.003s sys 0.004s
I20260129 02:59:08.676740 21555 ts_tablet_manager.cc:1434] T 4e420f90884f4796997b5ea41e49e411 P 7a9daf5d42a84615a6383cfec9a10608: Time spent starting tablet: real 0.010s user 0.006s sys 0.000s
I20260129 02:59:08.678290 20526 tablet_copy_service.cc:342] P 974b802141bd4f13beffef2aa03f88f9: Request end of tablet copy session 7a9daf5d42a84615a6383cfec9a10608-2ac659d732314e1288849f08fe5e074e received from {username='slave'} at 127.0.0.1:48544
I20260129 02:59:08.678561 20450 tablet_copy_service.cc:342] P 197b880fa39d432a9b472845876bfaf0: Request end of tablet copy session 7a9daf5d42a84615a6383cfec9a10608-4e420f90884f4796997b5ea41e49e411 received from {username='slave'} at 127.0.0.1:48356
I20260129 02:59:08.678761 20526 tablet_copy_service.cc:434] P 974b802141bd4f13beffef2aa03f88f9: ending tablet copy session 7a9daf5d42a84615a6383cfec9a10608-2ac659d732314e1288849f08fe5e074e on tablet 2ac659d732314e1288849f08fe5e074e with peer 7a9daf5d42a84615a6383cfec9a10608
I20260129 02:59:08.679013 20450 tablet_copy_service.cc:434] P 197b880fa39d432a9b472845876bfaf0: ending tablet copy session 7a9daf5d42a84615a6383cfec9a10608-4e420f90884f4796997b5ea41e49e411 on tablet 4e420f90884f4796997b5ea41e49e411 with peer 7a9daf5d42a84615a6383cfec9a10608
I20260129 02:59:08.716642 21134 raft_consensus.cc:1275] T cb0b3f259ebc4310927338a8740e87b8 P 7a9daf5d42a84615a6383cfec9a10608 [term 3 FOLLOWER]: Refusing update from remote peer 974b802141bd4f13beffef2aa03f88f9: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 3 index: 6. (index mismatch)
I20260129 02:59:08.717988 21513 consensus_queue.cc:1048] T cb0b3f259ebc4310927338a8740e87b8 P 974b802141bd4f13beffef2aa03f88f9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 4, Time since last communication: 0.000s
I20260129 02:59:08.760372 21134 raft_consensus.cc:1217] T 4e3805ffa70242e8ade66da260dddf66 P 7a9daf5d42a84615a6383cfec9a10608 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20260129 02:59:08.764643 20590 raft_consensus.cc:1275] T cb0b3f259ebc4310927338a8740e87b8 P 1c50de89a5a845668696dcb28c3badd6 [term 3 FOLLOWER]: Refusing update from remote peer 974b802141bd4f13beffef2aa03f88f9: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 3 index: 6. (index mismatch)
I20260129 02:59:08.767483 21513 consensus_queue.cc:1048] T cb0b3f259ebc4310927338a8740e87b8 P 974b802141bd4f13beffef2aa03f88f9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 4, Time since last communication: 0.000s
I20260129 02:59:08.918949 21134 raft_consensus.cc:1217] T 75b0b6e82ddc4f92b1fe1d94f9a0b73b P 7a9daf5d42a84615a6383cfec9a10608 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20260129 02:59:08.929000 21134 raft_consensus.cc:1217] T 2ac659d732314e1288849f08fe5e074e P 7a9daf5d42a84615a6383cfec9a10608 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20260129 02:59:09.075326 21134 raft_consensus.cc:1217] T 4e420f90884f4796997b5ea41e49e411 P 7a9daf5d42a84615a6383cfec9a10608 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20260129 02:59:09.249320 21519 raft_consensus.cc:1064] T 4e3805ffa70242e8ade66da260dddf66 P 197b880fa39d432a9b472845876bfaf0: attempting to promote NON_VOTER 7a9daf5d42a84615a6383cfec9a10608 to VOTER
I20260129 02:59:09.251940 21519 consensus_queue.cc:237] T 4e3805ffa70242e8ade66da260dddf66 P 197b880fa39d432a9b472845876bfaf0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } attrs { replace: true } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } }
I20260129 02:59:09.258539 20587 raft_consensus.cc:1275] T 4e3805ffa70242e8ade66da260dddf66 P 1c50de89a5a845668696dcb28c3badd6 [term 1 FOLLOWER]: Refusing update from remote peer 197b880fa39d432a9b472845876bfaf0: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260129 02:59:09.258813 21134 raft_consensus.cc:1275] T 4e3805ffa70242e8ade66da260dddf66 P 7a9daf5d42a84615a6383cfec9a10608 [term 1 LEARNER]: Refusing update from remote peer 197b880fa39d432a9b472845876bfaf0: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260129 02:59:09.259943 21529 consensus_queue.cc:1048] T 4e3805ffa70242e8ade66da260dddf66 P 197b880fa39d432a9b472845876bfaf0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260129 02:59:09.262228 21519 consensus_queue.cc:1048] T 4e3805ffa70242e8ade66da260dddf66 P 197b880fa39d432a9b472845876bfaf0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260129 02:59:09.263968 20510 raft_consensus.cc:1275] T 4e3805ffa70242e8ade66da260dddf66 P 974b802141bd4f13beffef2aa03f88f9 [term 1 FOLLOWER]: Refusing update from remote peer 197b880fa39d432a9b472845876bfaf0: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260129 02:59:09.269176 21531 consensus_queue.cc:1048] T 4e3805ffa70242e8ade66da260dddf66 P 197b880fa39d432a9b472845876bfaf0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260129 02:59:09.333504 20511 raft_consensus.cc:2955] T 4e3805ffa70242e8ade66da260dddf66 P 974b802141bd4f13beffef2aa03f88f9 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 7a9daf5d42a84615a6383cfec9a10608 (127.18.211.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } attrs { replace: true } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } } }
I20260129 02:59:09.333760 21565 raft_consensus.cc:2955] T 4e3805ffa70242e8ade66da260dddf66 P 197b880fa39d432a9b472845876bfaf0 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 7a9daf5d42a84615a6383cfec9a10608 (127.18.211.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } attrs { replace: true } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } } }
I20260129 02:59:09.348225 20590 raft_consensus.cc:2955] T 4e3805ffa70242e8ade66da260dddf66 P 1c50de89a5a845668696dcb28c3badd6 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 7a9daf5d42a84615a6383cfec9a10608 (127.18.211.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } attrs { replace: true } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } } }
I20260129 02:59:09.362064 20336 catalog_manager.cc:5654] T 4e3805ffa70242e8ade66da260dddf66 P 974b802141bd4f13beffef2aa03f88f9 reported cstate change: config changed from index 2 to 3, 7a9daf5d42a84615a6383cfec9a10608 (127.18.211.4) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "197b880fa39d432a9b472845876bfaf0" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } attrs { replace: true } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } } }
I20260129 02:59:09.363004 21134 raft_consensus.cc:2955] T 4e3805ffa70242e8ade66da260dddf66 P 7a9daf5d42a84615a6383cfec9a10608 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 7a9daf5d42a84615a6383cfec9a10608 (127.18.211.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } attrs { replace: true } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } } }
I20260129 02:59:09.426740 20437 consensus_queue.cc:237] T 4e3805ffa70242e8ade66da260dddf66 P 197b880fa39d432a9b472845876bfaf0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } }
I20260129 02:59:09.437538 21134 raft_consensus.cc:1275] T 4e3805ffa70242e8ade66da260dddf66 P 7a9daf5d42a84615a6383cfec9a10608 [term 1 FOLLOWER]: Refusing update from remote peer 197b880fa39d432a9b472845876bfaf0: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260129 02:59:09.438510 20587 raft_consensus.cc:1275] T 4e3805ffa70242e8ade66da260dddf66 P 1c50de89a5a845668696dcb28c3badd6 [term 1 FOLLOWER]: Refusing update from remote peer 197b880fa39d432a9b472845876bfaf0: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260129 02:59:09.438846 21529 consensus_queue.cc:1048] T 4e3805ffa70242e8ade66da260dddf66 P 197b880fa39d432a9b472845876bfaf0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260129 02:59:09.441836 21531 consensus_queue.cc:1048] T 4e3805ffa70242e8ade66da260dddf66 P 197b880fa39d432a9b472845876bfaf0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260129 02:59:09.452266 21529 raft_consensus.cc:2955] T 4e3805ffa70242e8ade66da260dddf66 P 197b880fa39d432a9b472845876bfaf0 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 974b802141bd4f13beffef2aa03f88f9 (127.18.211.2) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } } }
I20260129 02:59:09.461870 21134 raft_consensus.cc:2955] T 4e3805ffa70242e8ade66da260dddf66 P 7a9daf5d42a84615a6383cfec9a10608 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 974b802141bd4f13beffef2aa03f88f9 (127.18.211.2) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } } }
I20260129 02:59:09.465162 20582 raft_consensus.cc:2955] T 4e3805ffa70242e8ade66da260dddf66 P 1c50de89a5a845668696dcb28c3badd6 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 974b802141bd4f13beffef2aa03f88f9 (127.18.211.2) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } } }
I20260129 02:59:09.478238 21519 raft_consensus.cc:1064] T 4e420f90884f4796997b5ea41e49e411 P 197b880fa39d432a9b472845876bfaf0: attempting to promote NON_VOTER 7a9daf5d42a84615a6383cfec9a10608 to VOTER
I20260129 02:59:09.480938 21519 consensus_queue.cc:237] T 4e420f90884f4796997b5ea41e49e411 P 197b880fa39d432a9b472845876bfaf0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } attrs { replace: true } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } }
I20260129 02:59:09.486029 20338 catalog_manager.cc:5654] T 4e3805ffa70242e8ade66da260dddf66 P 1c50de89a5a845668696dcb28c3badd6 reported cstate change: config changed from index 3 to 4, VOTER 974b802141bd4f13beffef2aa03f88f9 (127.18.211.2) evicted. New cstate: current_term: 1 leader_uuid: "197b880fa39d432a9b472845876bfaf0" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } } }
I20260129 02:59:09.488530 21511 raft_consensus.cc:1064] T 75b0b6e82ddc4f92b1fe1d94f9a0b73b P 1c50de89a5a845668696dcb28c3badd6: attempting to promote NON_VOTER 7a9daf5d42a84615a6383cfec9a10608 to VOTER
I20260129 02:59:09.500068 20322 catalog_manager.cc:5167] ChangeConfig:REMOVE_PEER RPC for tablet 4e3805ffa70242e8ade66da260dddf66 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260129 02:59:09.491037 21511 consensus_queue.cc:237] T 75b0b6e82ddc4f92b1fe1d94f9a0b73b P 1c50de89a5a845668696dcb28c3badd6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } attrs { replace: true } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } }
I20260129 02:59:09.516141 20511 raft_consensus.cc:1275] T 4e420f90884f4796997b5ea41e49e411 P 974b802141bd4f13beffef2aa03f88f9 [term 1 FOLLOWER]: Refusing update from remote peer 197b880fa39d432a9b472845876bfaf0: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260129 02:59:09.519583 20587 raft_consensus.cc:1275] T 4e420f90884f4796997b5ea41e49e411 P 1c50de89a5a845668696dcb28c3badd6 [term 1 FOLLOWER]: Refusing update from remote peer 197b880fa39d432a9b472845876bfaf0: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260129 02:59:09.522149 21519 consensus_queue.cc:1048] T 4e420f90884f4796997b5ea41e49e411 P 197b880fa39d432a9b472845876bfaf0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260129 02:59:09.522492 21134 raft_consensus.cc:1275] T 4e420f90884f4796997b5ea41e49e411 P 7a9daf5d42a84615a6383cfec9a10608 [term 1 LEARNER]: Refusing update from remote peer 197b880fa39d432a9b472845876bfaf0: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260129 02:59:09.526506 21131 raft_consensus.cc:1275] T 75b0b6e82ddc4f92b1fe1d94f9a0b73b P 7a9daf5d42a84615a6383cfec9a10608 [term 1 LEARNER]: Refusing update from remote peer 1c50de89a5a845668696dcb28c3badd6: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260129 02:59:09.530838 20441 raft_consensus.cc:1275] T 75b0b6e82ddc4f92b1fe1d94f9a0b73b P 197b880fa39d432a9b472845876bfaf0 [term 1 FOLLOWER]: Refusing update from remote peer 1c50de89a5a845668696dcb28c3badd6: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260129 02:59:09.537866 20511 raft_consensus.cc:1275] T 75b0b6e82ddc4f92b1fe1d94f9a0b73b P 974b802141bd4f13beffef2aa03f88f9 [term 1 FOLLOWER]: Refusing update from remote peer 1c50de89a5a845668696dcb28c3badd6: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260129 02:59:09.542447 21518 consensus_queue.cc:1048] T 75b0b6e82ddc4f92b1fe1d94f9a0b73b P 1c50de89a5a845668696dcb28c3badd6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260129 02:59:09.544297 20500 tablet_service.cc:1552] Processing DeleteTablet for tablet 4e3805ffa70242e8ade66da260dddf66 with delete_type TABLET_DATA_TOMBSTONED (TS 974b802141bd4f13beffef2aa03f88f9 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:57580
I20260129 02:59:09.540303 21565 consensus_queue.cc:1048] T 4e420f90884f4796997b5ea41e49e411 P 197b880fa39d432a9b472845876bfaf0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260129 02:59:09.544315 21518 consensus_queue.cc:1048] T 75b0b6e82ddc4f92b1fe1d94f9a0b73b P 1c50de89a5a845668696dcb28c3badd6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.004s
I20260129 02:59:09.549669 21518 consensus_queue.cc:1048] T 75b0b6e82ddc4f92b1fe1d94f9a0b73b P 1c50de89a5a845668696dcb28c3badd6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260129 02:59:09.553054 21567 consensus_queue.cc:1048] T 4e420f90884f4796997b5ea41e49e411 P 197b880fa39d432a9b472845876bfaf0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260129 02:59:09.557401 21578 tablet_replica.cc:333] T 4e3805ffa70242e8ade66da260dddf66 P 974b802141bd4f13beffef2aa03f88f9: stopping tablet replica
I20260129 02:59:09.558455 21578 raft_consensus.cc:2243] T 4e3805ffa70242e8ade66da260dddf66 P 974b802141bd4f13beffef2aa03f88f9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260129 02:59:09.559333 21578 raft_consensus.cc:2272] T 4e3805ffa70242e8ade66da260dddf66 P 974b802141bd4f13beffef2aa03f88f9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260129 02:59:09.568619 21578 ts_tablet_manager.cc:1916] T 4e3805ffa70242e8ade66da260dddf66 P 974b802141bd4f13beffef2aa03f88f9: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260129 02:59:09.572234 21474 raft_consensus.cc:1064] T 2ac659d732314e1288849f08fe5e074e P 974b802141bd4f13beffef2aa03f88f9: attempting to promote NON_VOTER 7a9daf5d42a84615a6383cfec9a10608 to VOTER
I20260129 02:59:09.575162 21474 consensus_queue.cc:237] T 2ac659d732314e1288849f08fe5e074e P 974b802141bd4f13beffef2aa03f88f9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } attrs { replace: true } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } }
I20260129 02:59:09.613662 21511 raft_consensus.cc:2955] T 75b0b6e82ddc4f92b1fe1d94f9a0b73b P 1c50de89a5a845668696dcb28c3badd6 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 7a9daf5d42a84615a6383cfec9a10608 (127.18.211.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } attrs { replace: true } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } } }
I20260129 02:59:09.615443 20435 raft_consensus.cc:2955] T 75b0b6e82ddc4f92b1fe1d94f9a0b73b P 197b880fa39d432a9b472845876bfaf0 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 7a9daf5d42a84615a6383cfec9a10608 (127.18.211.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } attrs { replace: true } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } } }
I20260129 02:59:09.618450 21565 raft_consensus.cc:2955] T 4e420f90884f4796997b5ea41e49e411 P 197b880fa39d432a9b472845876bfaf0 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 7a9daf5d42a84615a6383cfec9a10608 (127.18.211.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } attrs { replace: true } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } } }
I20260129 02:59:09.627643 20582 raft_consensus.cc:1275] T 2ac659d732314e1288849f08fe5e074e P 1c50de89a5a845668696dcb28c3badd6 [term 1 FOLLOWER]: Refusing update from remote peer 974b802141bd4f13beffef2aa03f88f9: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260129 02:59:09.627223 20510 raft_consensus.cc:2955] T 4e420f90884f4796997b5ea41e49e411 P 974b802141bd4f13beffef2aa03f88f9 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 7a9daf5d42a84615a6383cfec9a10608 (127.18.211.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } attrs { replace: true } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } } }
I20260129 02:59:09.631050 21513 consensus_queue.cc:1048] T 2ac659d732314e1288849f08fe5e074e P 974b802141bd4f13beffef2aa03f88f9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260129 02:59:09.632501 21134 raft_consensus.cc:1275] T 2ac659d732314e1288849f08fe5e074e P 7a9daf5d42a84615a6383cfec9a10608 [term 1 LEARNER]: Refusing update from remote peer 974b802141bd4f13beffef2aa03f88f9: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260129 02:59:09.633955 21513 consensus_queue.cc:1048] T 2ac659d732314e1288849f08fe5e074e P 974b802141bd4f13beffef2aa03f88f9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260129 02:59:09.637485 20437 raft_consensus.cc:1275] T 2ac659d732314e1288849f08fe5e074e P 197b880fa39d432a9b472845876bfaf0 [term 1 FOLLOWER]: Refusing update from remote peer 974b802141bd4f13beffef2aa03f88f9: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260129 02:59:09.639222 21513 consensus_queue.cc:1048] T 2ac659d732314e1288849f08fe5e074e P 974b802141bd4f13beffef2aa03f88f9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260129 02:59:09.645046 21130 raft_consensus.cc:2955] T 4e420f90884f4796997b5ea41e49e411 P 7a9daf5d42a84615a6383cfec9a10608 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 7a9daf5d42a84615a6383cfec9a10608 (127.18.211.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } attrs { replace: true } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } } }
I20260129 02:59:09.659401 20338 catalog_manager.cc:5654] T 75b0b6e82ddc4f92b1fe1d94f9a0b73b P 197b880fa39d432a9b472845876bfaf0 reported cstate change: config changed from index 2 to 3, 7a9daf5d42a84615a6383cfec9a10608 (127.18.211.4) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "1c50de89a5a845668696dcb28c3badd6" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } attrs { replace: true } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } } }
I20260129 02:59:09.677592 20338 catalog_manager.cc:5654] T 4e420f90884f4796997b5ea41e49e411 P 7a9daf5d42a84615a6383cfec9a10608 reported cstate change: config changed from index 2 to 3, 7a9daf5d42a84615a6383cfec9a10608 (127.18.211.4) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "197b880fa39d432a9b472845876bfaf0" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } attrs { replace: true } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } } }
W20260129 02:59:09.682221 20556 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.211.2:34517, user_credentials={real_user=slave}} blocked reactor thread for 47219.4us
I20260129 02:59:09.690907 21135 raft_consensus.cc:2955] T 75b0b6e82ddc4f92b1fe1d94f9a0b73b P 7a9daf5d42a84615a6383cfec9a10608 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 7a9daf5d42a84615a6383cfec9a10608 (127.18.211.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } attrs { replace: true } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } } }
I20260129 02:59:09.708582 20587 raft_consensus.cc:2955] T 4e420f90884f4796997b5ea41e49e411 P 1c50de89a5a845668696dcb28c3badd6 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 7a9daf5d42a84615a6383cfec9a10608 (127.18.211.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } attrs { replace: true } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } } }
I20260129 02:59:09.730655 20435 consensus_queue.cc:237] T 4e420f90884f4796997b5ea41e49e411 P 197b880fa39d432a9b472845876bfaf0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } }
I20260129 02:59:09.759517 21513 raft_consensus.cc:2955] T 2ac659d732314e1288849f08fe5e074e P 974b802141bd4f13beffef2aa03f88f9 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 7a9daf5d42a84615a6383cfec9a10608 (127.18.211.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } attrs { replace: true } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } } }
I20260129 02:59:09.772435 21135 raft_consensus.cc:1275] T 4e420f90884f4796997b5ea41e49e411 P 7a9daf5d42a84615a6383cfec9a10608 [term 1 FOLLOWER]: Refusing update from remote peer 197b880fa39d432a9b472845876bfaf0: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260129 02:59:09.776024 20507 raft_consensus.cc:2955] T 75b0b6e82ddc4f92b1fe1d94f9a0b73b P 974b802141bd4f13beffef2aa03f88f9 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 7a9daf5d42a84615a6383cfec9a10608 (127.18.211.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } attrs { replace: true } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } } }
I20260129 02:59:09.783125 21565 consensus_queue.cc:1048] T 4e420f90884f4796997b5ea41e49e411 P 197b880fa39d432a9b472845876bfaf0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260129 02:59:09.784341 20510 raft_consensus.cc:1275] T 4e420f90884f4796997b5ea41e49e411 P 974b802141bd4f13beffef2aa03f88f9 [term 1 FOLLOWER]: Refusing update from remote peer 197b880fa39d432a9b472845876bfaf0: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260129 02:59:09.787128 20587 raft_consensus.cc:2955] T 2ac659d732314e1288849f08fe5e074e P 1c50de89a5a845668696dcb28c3badd6 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 7a9daf5d42a84615a6383cfec9a10608 (127.18.211.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } attrs { replace: true } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } } }
I20260129 02:59:09.796420 21565 consensus_queue.cc:1048] T 4e420f90884f4796997b5ea41e49e411 P 197b880fa39d432a9b472845876bfaf0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260129 02:59:09.800225 21134 raft_consensus.cc:2955] T 2ac659d732314e1288849f08fe5e074e P 7a9daf5d42a84615a6383cfec9a10608 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 7a9daf5d42a84615a6383cfec9a10608 (127.18.211.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } attrs { replace: true } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } } }
I20260129 02:59:09.800802 21531 raft_consensus.cc:2955] T 4e420f90884f4796997b5ea41e49e411 P 197b880fa39d432a9b472845876bfaf0 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } } }
I20260129 02:59:09.809599 21131 raft_consensus.cc:2955] T 4e420f90884f4796997b5ea41e49e411 P 7a9daf5d42a84615a6383cfec9a10608 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } } }
I20260129 02:59:09.814723 20582 consensus_queue.cc:237] T 75b0b6e82ddc4f92b1fe1d94f9a0b73b P 1c50de89a5a845668696dcb28c3badd6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } }
I20260129 02:59:09.816324 20337 catalog_manager.cc:5654] T 2ac659d732314e1288849f08fe5e074e P 974b802141bd4f13beffef2aa03f88f9 reported cstate change: config changed from index 2 to 3, 7a9daf5d42a84615a6383cfec9a10608 (127.18.211.4) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "974b802141bd4f13beffef2aa03f88f9" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260129 02:59:09.819510 21578 ts_tablet_manager.cc:1929] T 4e3805ffa70242e8ade66da260dddf66 P 974b802141bd4f13beffef2aa03f88f9: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20260129 02:59:09.820075 21578 log.cc:1199] T 4e3805ffa70242e8ade66da260dddf66 P 974b802141bd4f13beffef2aa03f88f9: Deleting WAL directory at /tmp/dist-test-taskrTP_Jo/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1769655460890924-19276-0/minicluster-data/ts-1-root/wals/4e3805ffa70242e8ade66da260dddf66
W20260129 02:59:09.828045 20479 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.211.1:42027, user_credentials={real_user=slave}} blocked reactor thread for 75812.6us
I20260129 02:59:09.833017 20322 catalog_manager.cc:5167] ChangeConfig:REMOVE_PEER RPC for tablet 4e420f90884f4796997b5ea41e49e411 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260129 02:59:09.835940 20325 catalog_manager.cc:4985] TS 974b802141bd4f13beffef2aa03f88f9 (127.18.211.2:34517): tablet 4e3805ffa70242e8ade66da260dddf66 (table test-workload [id=de73611ac8e3485a98aecc5add247032]) successfully deleted
I20260129 02:59:09.837263 20335 catalog_manager.cc:5654] T 4e420f90884f4796997b5ea41e49e411 P 197b880fa39d432a9b472845876bfaf0 reported cstate change: config changed from index 3 to 4, VOTER 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3) evicted. New cstate: current_term: 1 leader_uuid: "197b880fa39d432a9b472845876bfaf0" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260129 02:59:09.838315 20435 raft_consensus.cc:2955] T 2ac659d732314e1288849f08fe5e074e P 197b880fa39d432a9b472845876bfaf0 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 7a9daf5d42a84615a6383cfec9a10608 (127.18.211.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } attrs { replace: true } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } } }
I20260129 02:59:09.858457 20579 tablet_service.cc:1552] Processing DeleteTablet for tablet 4e420f90884f4796997b5ea41e49e411 with delete_type TABLET_DATA_TOMBSTONED (TS 1c50de89a5a845668696dcb28c3badd6 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:35744
I20260129 02:59:09.858551 20507 raft_consensus.cc:2955] T 4e420f90884f4796997b5ea41e49e411 P 974b802141bd4f13beffef2aa03f88f9 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } } }
I20260129 02:59:09.878485 21131 raft_consensus.cc:1275] T 75b0b6e82ddc4f92b1fe1d94f9a0b73b P 7a9daf5d42a84615a6383cfec9a10608 [term 1 FOLLOWER]: Refusing update from remote peer 1c50de89a5a845668696dcb28c3badd6: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260129 02:59:09.879994 21511 consensus_queue.cc:1048] T 75b0b6e82ddc4f92b1fe1d94f9a0b73b P 1c50de89a5a845668696dcb28c3badd6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260129 02:59:09.887288 20510 raft_consensus.cc:1275] T 75b0b6e82ddc4f92b1fe1d94f9a0b73b P 974b802141bd4f13beffef2aa03f88f9 [term 1 FOLLOWER]: Refusing update from remote peer 1c50de89a5a845668696dcb28c3badd6: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260129 02:59:09.889725 21518 consensus_queue.cc:1048] T 75b0b6e82ddc4f92b1fe1d94f9a0b73b P 1c50de89a5a845668696dcb28c3badd6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260129 02:59:09.892681 21511 raft_consensus.cc:2955] T 75b0b6e82ddc4f92b1fe1d94f9a0b73b P 1c50de89a5a845668696dcb28c3badd6 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 197b880fa39d432a9b472845876bfaf0 (127.18.211.1) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } } }
I20260129 02:59:09.895102 21131 raft_consensus.cc:2955] T 75b0b6e82ddc4f92b1fe1d94f9a0b73b P 7a9daf5d42a84615a6383cfec9a10608 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 197b880fa39d432a9b472845876bfaf0 (127.18.211.1) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } } }
I20260129 02:59:09.906980 20510 consensus_queue.cc:237] T 2ac659d732314e1288849f08fe5e074e P 974b802141bd4f13beffef2aa03f88f9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } }
I20260129 02:59:09.911669 21134 raft_consensus.cc:1275] T 2ac659d732314e1288849f08fe5e074e P 7a9daf5d42a84615a6383cfec9a10608 [term 1 FOLLOWER]: Refusing update from remote peer 974b802141bd4f13beffef2aa03f88f9: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260129 02:59:09.913465 21533 consensus_queue.cc:1048] T 2ac659d732314e1288849f08fe5e074e P 974b802141bd4f13beffef2aa03f88f9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260129 02:59:09.922827 21596 tablet_replica.cc:333] T 4e420f90884f4796997b5ea41e49e411 P 1c50de89a5a845668696dcb28c3badd6: stopping tablet replica
W20260129 02:59:09.926369 20407 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.211.2:34517, user_credentials={real_user=slave}} blocked reactor thread for 47570.3us
I20260129 02:59:09.932765 21592 raft_consensus.cc:2955] T 2ac659d732314e1288849f08fe5e074e P 974b802141bd4f13beffef2aa03f88f9 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } } }
I20260129 02:59:09.928001 20441 raft_consensus.cc:1275] T 2ac659d732314e1288849f08fe5e074e P 197b880fa39d432a9b472845876bfaf0 [term 1 FOLLOWER]: Refusing update from remote peer 974b802141bd4f13beffef2aa03f88f9: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260129 02:59:09.940593 21596 raft_consensus.cc:2243] T 4e420f90884f4796997b5ea41e49e411 P 1c50de89a5a845668696dcb28c3badd6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260129 02:59:09.941458 21596 raft_consensus.cc:2272] T 4e420f90884f4796997b5ea41e49e411 P 1c50de89a5a845668696dcb28c3badd6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260129 02:59:09.949807 21596 ts_tablet_manager.cc:1916] T 4e420f90884f4796997b5ea41e49e411 P 1c50de89a5a845668696dcb28c3badd6: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260129 02:59:09.949930 20510 raft_consensus.cc:2955] T 75b0b6e82ddc4f92b1fe1d94f9a0b73b P 974b802141bd4f13beffef2aa03f88f9 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 197b880fa39d432a9b472845876bfaf0 (127.18.211.1) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } } }
I20260129 02:59:09.955461 20323 catalog_manager.cc:5167] ChangeConfig:REMOVE_PEER RPC for tablet 75b0b6e82ddc4f92b1fe1d94f9a0b73b with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260129 02:59:09.966509 21134 raft_consensus.cc:2955] T 2ac659d732314e1288849f08fe5e074e P 7a9daf5d42a84615a6383cfec9a10608 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } } }
I20260129 02:59:09.970921 20335 catalog_manager.cc:5654] T 75b0b6e82ddc4f92b1fe1d94f9a0b73b P 7a9daf5d42a84615a6383cfec9a10608 reported cstate change: config changed from index 3 to 4, VOTER 197b880fa39d432a9b472845876bfaf0 (127.18.211.1) evicted. New cstate: current_term: 1 leader_uuid: "1c50de89a5a845668696dcb28c3badd6" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } } }
I20260129 02:59:09.977671 21593 consensus_queue.cc:1048] T 2ac659d732314e1288849f08fe5e074e P 974b802141bd4f13beffef2aa03f88f9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260129 02:59:09.984330 20325 catalog_manager.cc:5167] ChangeConfig:REMOVE_PEER RPC for tablet 2ac659d732314e1288849f08fe5e074e with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260129 02:59:09.987915 21596 ts_tablet_manager.cc:1929] T 4e420f90884f4796997b5ea41e49e411 P 1c50de89a5a845668696dcb28c3badd6: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20260129 02:59:09.988451 21596 log.cc:1199] T 4e420f90884f4796997b5ea41e49e411 P 1c50de89a5a845668696dcb28c3badd6: Deleting WAL directory at /tmp/dist-test-taskrTP_Jo/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1769655460890924-19276-0/minicluster-data/ts-2-root/wals/4e420f90884f4796997b5ea41e49e411
I20260129 02:59:09.999106 20441 raft_consensus.cc:2955] T 2ac659d732314e1288849f08fe5e074e P 197b880fa39d432a9b472845876bfaf0 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } } }
I20260129 02:59:09.992923 20323 catalog_manager.cc:4985] TS 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3:37735): tablet 4e420f90884f4796997b5ea41e49e411 (table test-workload [id=de73611ac8e3485a98aecc5add247032]) successfully deleted
I20260129 02:59:10.018157 20338 catalog_manager.cc:5654] T 2ac659d732314e1288849f08fe5e074e P 974b802141bd4f13beffef2aa03f88f9 reported cstate change: config changed from index 3 to 4, VOTER 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3) evicted. New cstate: current_term: 1 leader_uuid: "974b802141bd4f13beffef2aa03f88f9" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260129 02:59:10.077422 20579 tablet_service.cc:1552] Processing DeleteTablet for tablet 2ac659d732314e1288849f08fe5e074e with delete_type TABLET_DATA_TOMBSTONED (TS 1c50de89a5a845668696dcb28c3badd6 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:35744
I20260129 02:59:10.077584 20431 tablet_service.cc:1552] Processing DeleteTablet for tablet 75b0b6e82ddc4f92b1fe1d94f9a0b73b with delete_type TABLET_DATA_TOMBSTONED (TS 197b880fa39d432a9b472845876bfaf0 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:55666
I20260129 02:59:10.078487 21596 tablet_replica.cc:333] T 2ac659d732314e1288849f08fe5e074e P 1c50de89a5a845668696dcb28c3badd6: stopping tablet replica
I20260129 02:59:10.079591 21596 raft_consensus.cc:2243] T 2ac659d732314e1288849f08fe5e074e P 1c50de89a5a845668696dcb28c3badd6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260129 02:59:10.080435 21596 raft_consensus.cc:2272] T 2ac659d732314e1288849f08fe5e074e P 1c50de89a5a845668696dcb28c3badd6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260129 02:59:10.086210 21596 ts_tablet_manager.cc:1916] T 2ac659d732314e1288849f08fe5e074e P 1c50de89a5a845668696dcb28c3badd6: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260129 02:59:10.104524 21597 tablet_replica.cc:333] T 75b0b6e82ddc4f92b1fe1d94f9a0b73b P 197b880fa39d432a9b472845876bfaf0: stopping tablet replica
I20260129 02:59:10.105559 21597 raft_consensus.cc:2243] T 75b0b6e82ddc4f92b1fe1d94f9a0b73b P 197b880fa39d432a9b472845876bfaf0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260129 02:59:10.106242 21597 raft_consensus.cc:2272] T 75b0b6e82ddc4f92b1fe1d94f9a0b73b P 197b880fa39d432a9b472845876bfaf0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260129 02:59:10.109656 21597 ts_tablet_manager.cc:1916] T 75b0b6e82ddc4f92b1fe1d94f9a0b73b P 197b880fa39d432a9b472845876bfaf0: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260129 02:59:10.126730 21597 ts_tablet_manager.cc:1929] T 75b0b6e82ddc4f92b1fe1d94f9a0b73b P 197b880fa39d432a9b472845876bfaf0: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20260129 02:59:10.127220 21597 log.cc:1199] T 75b0b6e82ddc4f92b1fe1d94f9a0b73b P 197b880fa39d432a9b472845876bfaf0: Deleting WAL directory at /tmp/dist-test-taskrTP_Jo/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1769655460890924-19276-0/minicluster-data/ts-0-root/wals/75b0b6e82ddc4f92b1fe1d94f9a0b73b
I20260129 02:59:10.129611 20322 catalog_manager.cc:4985] TS 197b880fa39d432a9b472845876bfaf0 (127.18.211.1:42027): tablet 75b0b6e82ddc4f92b1fe1d94f9a0b73b (table test-workload [id=de73611ac8e3485a98aecc5add247032]) successfully deleted
I20260129 02:59:10.131989 21596 ts_tablet_manager.cc:1929] T 2ac659d732314e1288849f08fe5e074e P 1c50de89a5a845668696dcb28c3badd6: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20260129 02:59:10.132433 21596 log.cc:1199] T 2ac659d732314e1288849f08fe5e074e P 1c50de89a5a845668696dcb28c3badd6: Deleting WAL directory at /tmp/dist-test-taskrTP_Jo/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1769655460890924-19276-0/minicluster-data/ts-2-root/wals/2ac659d732314e1288849f08fe5e074e
I20260129 02:59:10.133898 20323 catalog_manager.cc:4985] TS 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3:37735): tablet 2ac659d732314e1288849f08fe5e074e (table test-workload [id=de73611ac8e3485a98aecc5add247032]) successfully deleted
I20260129 02:59:11.051822 20582 consensus_queue.cc:237] T dc2f4cb752cf4d59b5f6bc689f5d8104 P 1c50de89a5a845668696dcb28c3badd6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } attrs { replace: true } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: NON_VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: true } }
I20260129 02:59:11.056954 20507 raft_consensus.cc:1275] T dc2f4cb752cf4d59b5f6bc689f5d8104 P 974b802141bd4f13beffef2aa03f88f9 [term 1 FOLLOWER]: Refusing update from remote peer 1c50de89a5a845668696dcb28c3badd6: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260129 02:59:11.057380 20435 raft_consensus.cc:1275] T dc2f4cb752cf4d59b5f6bc689f5d8104 P 197b880fa39d432a9b472845876bfaf0 [term 1 FOLLOWER]: Refusing update from remote peer 1c50de89a5a845668696dcb28c3badd6: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20260129 02:59:11.057785 20554 consensus_peers.cc:597] T dc2f4cb752cf4d59b5f6bc689f5d8104 P 1c50de89a5a845668696dcb28c3badd6 -> Peer 7a9daf5d42a84615a6383cfec9a10608 (127.18.211.4:40439): Couldn't send request to peer 7a9daf5d42a84615a6383cfec9a10608. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: dc2f4cb752cf4d59b5f6bc689f5d8104. This is attempt 1: this message will repeat every 5th retry.
I20260129 02:59:11.058380 21599 consensus_queue.cc:1048] T dc2f4cb752cf4d59b5f6bc689f5d8104 P 1c50de89a5a845668696dcb28c3badd6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260129 02:59:11.059306 21594 consensus_queue.cc:1048] T dc2f4cb752cf4d59b5f6bc689f5d8104 P 1c50de89a5a845668696dcb28c3badd6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260129 02:59:11.066475 21594 raft_consensus.cc:2955] T dc2f4cb752cf4d59b5f6bc689f5d8104 P 1c50de89a5a845668696dcb28c3badd6 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 7a9daf5d42a84615a6383cfec9a10608 (127.18.211.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } attrs { replace: true } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: NON_VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: true } } }
I20260129 02:59:11.068876 20435 raft_consensus.cc:2955] T dc2f4cb752cf4d59b5f6bc689f5d8104 P 197b880fa39d432a9b472845876bfaf0 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 7a9daf5d42a84615a6383cfec9a10608 (127.18.211.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } attrs { replace: true } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: NON_VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: true } } }
I20260129 02:59:11.071713 20507 raft_consensus.cc:2955] T dc2f4cb752cf4d59b5f6bc689f5d8104 P 974b802141bd4f13beffef2aa03f88f9 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 7a9daf5d42a84615a6383cfec9a10608 (127.18.211.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } attrs { replace: true } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: NON_VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: true } } }
I20260129 02:59:11.080852 20582 consensus_queue.cc:237] T 45cc5f3d2688486d9b6dbe7e666dedde P 1c50de89a5a845668696dcb28c3badd6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } attrs { replace: true } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: NON_VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: true } }
I20260129 02:59:11.087168 20338 catalog_manager.cc:5654] T dc2f4cb752cf4d59b5f6bc689f5d8104 P 1c50de89a5a845668696dcb28c3badd6 reported cstate change: config changed from index -1 to 2, NON_VOTER 7a9daf5d42a84615a6383cfec9a10608 (127.18.211.4) added. New cstate: current_term: 1 leader_uuid: "1c50de89a5a845668696dcb28c3badd6" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: NON_VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20260129 02:59:11.100855 20510 raft_consensus.cc:1275] T 45cc5f3d2688486d9b6dbe7e666dedde P 974b802141bd4f13beffef2aa03f88f9 [term 1 FOLLOWER]: Refusing update from remote peer 1c50de89a5a845668696dcb28c3badd6: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20260129 02:59:11.101864 20554 consensus_peers.cc:597] T 45cc5f3d2688486d9b6dbe7e666dedde P 1c50de89a5a845668696dcb28c3badd6 -> Peer 7a9daf5d42a84615a6383cfec9a10608 (127.18.211.4:40439): Couldn't send request to peer 7a9daf5d42a84615a6383cfec9a10608. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 45cc5f3d2688486d9b6dbe7e666dedde. This is attempt 1: this message will repeat every 5th retry.
I20260129 02:59:11.105502 20435 raft_consensus.cc:1275] T 45cc5f3d2688486d9b6dbe7e666dedde P 197b880fa39d432a9b472845876bfaf0 [term 1 FOLLOWER]: Refusing update from remote peer 1c50de89a5a845668696dcb28c3badd6: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260129 02:59:11.107033 21604 consensus_queue.cc:1048] T 45cc5f3d2688486d9b6dbe7e666dedde P 1c50de89a5a845668696dcb28c3badd6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260129 02:59:11.107926 21511 consensus_queue.cc:1048] T 45cc5f3d2688486d9b6dbe7e666dedde P 1c50de89a5a845668696dcb28c3badd6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260129 02:59:11.144922 21511 raft_consensus.cc:2955] T 45cc5f3d2688486d9b6dbe7e666dedde P 1c50de89a5a845668696dcb28c3badd6 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 7a9daf5d42a84615a6383cfec9a10608 (127.18.211.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } attrs { replace: true } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: NON_VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: true } } }
I20260129 02:59:11.157163 20511 raft_consensus.cc:2955] T 45cc5f3d2688486d9b6dbe7e666dedde P 974b802141bd4f13beffef2aa03f88f9 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 7a9daf5d42a84615a6383cfec9a10608 (127.18.211.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } attrs { replace: true } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: NON_VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: true } } }
I20260129 02:59:11.161168 20437 raft_consensus.cc:2955] T 45cc5f3d2688486d9b6dbe7e666dedde P 197b880fa39d432a9b472845876bfaf0 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 7a9daf5d42a84615a6383cfec9a10608 (127.18.211.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } attrs { replace: true } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: NON_VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: true } } }
I20260129 02:59:11.172896 20581 consensus_queue.cc:237] T 505580c93c694bbb8ea96fdf9b17ab46 P 1c50de89a5a845668696dcb28c3badd6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } attrs { replace: true } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: NON_VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: true } }
I20260129 02:59:11.181377 20511 raft_consensus.cc:1275] T 505580c93c694bbb8ea96fdf9b17ab46 P 974b802141bd4f13beffef2aa03f88f9 [term 1 FOLLOWER]: Refusing update from remote peer 1c50de89a5a845668696dcb28c3badd6: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260129 02:59:11.182730 21511 consensus_queue.cc:1048] T 505580c93c694bbb8ea96fdf9b17ab46 P 1c50de89a5a845668696dcb28c3badd6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
W20260129 02:59:11.184639 20554 consensus_peers.cc:597] T 505580c93c694bbb8ea96fdf9b17ab46 P 1c50de89a5a845668696dcb28c3badd6 -> Peer 7a9daf5d42a84615a6383cfec9a10608 (127.18.211.4:40439): Couldn't send request to peer 7a9daf5d42a84615a6383cfec9a10608. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 505580c93c694bbb8ea96fdf9b17ab46. This is attempt 1: this message will repeat every 5th retry.
I20260129 02:59:11.186100 20437 raft_consensus.cc:1275] T 505580c93c694bbb8ea96fdf9b17ab46 P 197b880fa39d432a9b472845876bfaf0 [term 1 FOLLOWER]: Refusing update from remote peer 1c50de89a5a845668696dcb28c3badd6: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260129 02:59:11.189718 21604 consensus_queue.cc:1048] T 505580c93c694bbb8ea96fdf9b17ab46 P 1c50de89a5a845668696dcb28c3badd6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260129 02:59:11.188951 20336 catalog_manager.cc:5654] T 45cc5f3d2688486d9b6dbe7e666dedde P 974b802141bd4f13beffef2aa03f88f9 reported cstate change: config changed from index -1 to 2, NON_VOTER 7a9daf5d42a84615a6383cfec9a10608 (127.18.211.4) added. New cstate: current_term: 1 leader_uuid: "1c50de89a5a845668696dcb28c3badd6" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } attrs { replace: true } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: NON_VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: true } } }
I20260129 02:59:11.204696 21594 raft_consensus.cc:2955] T 505580c93c694bbb8ea96fdf9b17ab46 P 1c50de89a5a845668696dcb28c3badd6 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 7a9daf5d42a84615a6383cfec9a10608 (127.18.211.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } attrs { replace: true } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: NON_VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: true } } }
I20260129 02:59:11.208525 20435 raft_consensus.cc:2955] T 505580c93c694bbb8ea96fdf9b17ab46 P 197b880fa39d432a9b472845876bfaf0 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 7a9daf5d42a84615a6383cfec9a10608 (127.18.211.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } attrs { replace: true } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: NON_VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: true } } }
I20260129 02:59:11.210258 20511 raft_consensus.cc:2955] T 505580c93c694bbb8ea96fdf9b17ab46 P 974b802141bd4f13beffef2aa03f88f9 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 7a9daf5d42a84615a6383cfec9a10608 (127.18.211.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } attrs { replace: true } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: NON_VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: true } } }
I20260129 02:59:11.227833 20336 catalog_manager.cc:5654] T 505580c93c694bbb8ea96fdf9b17ab46 P 1c50de89a5a845668696dcb28c3badd6 reported cstate change: config changed from index -1 to 2, NON_VOTER 7a9daf5d42a84615a6383cfec9a10608 (127.18.211.4) added. New cstate: current_term: 1 leader_uuid: "1c50de89a5a845668696dcb28c3badd6" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: NON_VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20260129 02:59:11.227558 20437 consensus_queue.cc:237] T c934d31efd3f4e5195605ddc354d14e1 P 197b880fa39d432a9b472845876bfaf0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } attrs { replace: true } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: NON_VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: true } }
W20260129 02:59:11.243695 20405 consensus_peers.cc:597] T c934d31efd3f4e5195605ddc354d14e1 P 197b880fa39d432a9b472845876bfaf0 -> Peer 7a9daf5d42a84615a6383cfec9a10608 (127.18.211.4:40439): Couldn't send request to peer 7a9daf5d42a84615a6383cfec9a10608. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: c934d31efd3f4e5195605ddc354d14e1. This is attempt 1: this message will repeat every 5th retry.
I20260129 02:59:11.245787 20582 raft_consensus.cc:1275] T c934d31efd3f4e5195605ddc354d14e1 P 1c50de89a5a845668696dcb28c3badd6 [term 1 FOLLOWER]: Refusing update from remote peer 197b880fa39d432a9b472845876bfaf0: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260129 02:59:11.245862 20511 raft_consensus.cc:1275] T c934d31efd3f4e5195605ddc354d14e1 P 974b802141bd4f13beffef2aa03f88f9 [term 1 FOLLOWER]: Refusing update from remote peer 197b880fa39d432a9b472845876bfaf0: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260129 02:59:11.248100 21614 consensus_queue.cc:1048] T c934d31efd3f4e5195605ddc354d14e1 P 197b880fa39d432a9b472845876bfaf0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260129 02:59:11.248946 21519 consensus_queue.cc:1048] T c934d31efd3f4e5195605ddc354d14e1 P 197b880fa39d432a9b472845876bfaf0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260129 02:59:11.258056 21565 raft_consensus.cc:2955] T c934d31efd3f4e5195605ddc354d14e1 P 197b880fa39d432a9b472845876bfaf0 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 7a9daf5d42a84615a6383cfec9a10608 (127.18.211.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } attrs { replace: true } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: NON_VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: true } } }
I20260129 02:59:11.260543 20511 raft_consensus.cc:2955] T c934d31efd3f4e5195605ddc354d14e1 P 974b802141bd4f13beffef2aa03f88f9 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 7a9daf5d42a84615a6383cfec9a10608 (127.18.211.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } attrs { replace: true } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: NON_VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: true } } }
I20260129 02:59:11.263394 20581 raft_consensus.cc:2955] T c934d31efd3f4e5195605ddc354d14e1 P 1c50de89a5a845668696dcb28c3badd6 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 7a9daf5d42a84615a6383cfec9a10608 (127.18.211.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } attrs { replace: true } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: NON_VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: true } } }
I20260129 02:59:11.279196 20335 catalog_manager.cc:5654] T c934d31efd3f4e5195605ddc354d14e1 P 197b880fa39d432a9b472845876bfaf0 reported cstate change: config changed from index -1 to 2, NON_VOTER 7a9daf5d42a84615a6383cfec9a10608 (127.18.211.4) added. New cstate: current_term: 1 leader_uuid: "197b880fa39d432a9b472845876bfaf0" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: NON_VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20260129 02:59:11.533416 21618 ts_tablet_manager.cc:933] T 45cc5f3d2688486d9b6dbe7e666dedde P 7a9daf5d42a84615a6383cfec9a10608: Initiating tablet copy from peer 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3:37735)
I20260129 02:59:11.536542 21618 tablet_copy_client.cc:323] T 45cc5f3d2688486d9b6dbe7e666dedde P 7a9daf5d42a84615a6383cfec9a10608: tablet copy: Beginning tablet copy session from remote peer at address 127.18.211.3:37735
I20260129 02:59:11.539232 20600 tablet_copy_service.cc:140] P 1c50de89a5a845668696dcb28c3badd6: Received BeginTabletCopySession request for tablet 45cc5f3d2688486d9b6dbe7e666dedde from peer 7a9daf5d42a84615a6383cfec9a10608 ({username='slave'} at 127.0.0.1:48526)
I20260129 02:59:11.540060 20600 tablet_copy_service.cc:161] P 1c50de89a5a845668696dcb28c3badd6: Beginning new tablet copy session on tablet 45cc5f3d2688486d9b6dbe7e666dedde from peer 7a9daf5d42a84615a6383cfec9a10608 at {username='slave'} at 127.0.0.1:48526: session id = 7a9daf5d42a84615a6383cfec9a10608-45cc5f3d2688486d9b6dbe7e666dedde
I20260129 02:59:11.582142 20600 tablet_copy_source_session.cc:215] T 45cc5f3d2688486d9b6dbe7e666dedde P 1c50de89a5a845668696dcb28c3badd6: Tablet Copy: opened 0 blocks and 1 log segments
I20260129 02:59:11.587661 21618 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 45cc5f3d2688486d9b6dbe7e666dedde. 1 dirs total, 0 dirs full, 0 dirs failed
I20260129 02:59:11.615829 21618 tablet_copy_client.cc:806] T 45cc5f3d2688486d9b6dbe7e666dedde P 7a9daf5d42a84615a6383cfec9a10608: tablet copy: Starting download of 0 data blocks...
I20260129 02:59:11.617244 21618 tablet_copy_client.cc:670] T 45cc5f3d2688486d9b6dbe7e666dedde P 7a9daf5d42a84615a6383cfec9a10608: tablet copy: Starting download of 1 WAL segments...
I20260129 02:59:11.621582 21618 tablet_copy_client.cc:538] T 45cc5f3d2688486d9b6dbe7e666dedde P 7a9daf5d42a84615a6383cfec9a10608: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260129 02:59:11.630322 21618 tablet_bootstrap.cc:492] T 45cc5f3d2688486d9b6dbe7e666dedde P 7a9daf5d42a84615a6383cfec9a10608: Bootstrap starting.
I20260129 02:59:11.653223 21621 ts_tablet_manager.cc:933] T dc2f4cb752cf4d59b5f6bc689f5d8104 P 7a9daf5d42a84615a6383cfec9a10608: Initiating tablet copy from peer 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3:37735)
I20260129 02:59:11.696914 21623 ts_tablet_manager.cc:933] T 505580c93c694bbb8ea96fdf9b17ab46 P 7a9daf5d42a84615a6383cfec9a10608: Initiating tablet copy from peer 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3:37735)
I20260129 02:59:11.689886 21618 tablet_bootstrap.cc:492] T 45cc5f3d2688486d9b6dbe7e666dedde P 7a9daf5d42a84615a6383cfec9a10608: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260129 02:59:11.707553 21621 tablet_copy_client.cc:323] T dc2f4cb752cf4d59b5f6bc689f5d8104 P 7a9daf5d42a84615a6383cfec9a10608: tablet copy: Beginning tablet copy session from remote peer at address 127.18.211.3:37735
I20260129 02:59:11.709038 21618 tablet_bootstrap.cc:492] T 45cc5f3d2688486d9b6dbe7e666dedde P 7a9daf5d42a84615a6383cfec9a10608: Bootstrap complete.
I20260129 02:59:11.711743 21618 ts_tablet_manager.cc:1403] T 45cc5f3d2688486d9b6dbe7e666dedde P 7a9daf5d42a84615a6383cfec9a10608: Time spent bootstrapping tablet: real 0.082s user 0.030s sys 0.022s
I20260129 02:59:11.712033 20600 tablet_copy_service.cc:140] P 1c50de89a5a845668696dcb28c3badd6: Received BeginTabletCopySession request for tablet dc2f4cb752cf4d59b5f6bc689f5d8104 from peer 7a9daf5d42a84615a6383cfec9a10608 ({username='slave'} at 127.0.0.1:48526)
I20260129 02:59:11.712739 20600 tablet_copy_service.cc:161] P 1c50de89a5a845668696dcb28c3badd6: Beginning new tablet copy session on tablet dc2f4cb752cf4d59b5f6bc689f5d8104 from peer 7a9daf5d42a84615a6383cfec9a10608 at {username='slave'} at 127.0.0.1:48526: session id = 7a9daf5d42a84615a6383cfec9a10608-dc2f4cb752cf4d59b5f6bc689f5d8104
I20260129 02:59:11.719422 21623 tablet_copy_client.cc:323] T 505580c93c694bbb8ea96fdf9b17ab46 P 7a9daf5d42a84615a6383cfec9a10608: tablet copy: Beginning tablet copy session from remote peer at address 127.18.211.3:37735
I20260129 02:59:11.719061 21618 raft_consensus.cc:359] T 45cc5f3d2688486d9b6dbe7e666dedde P 7a9daf5d42a84615a6383cfec9a10608 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } attrs { replace: true } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: NON_VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: true } }
I20260129 02:59:11.721935 21618 raft_consensus.cc:740] T 45cc5f3d2688486d9b6dbe7e666dedde P 7a9daf5d42a84615a6383cfec9a10608 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 7a9daf5d42a84615a6383cfec9a10608, State: Initialized, Role: LEARNER
I20260129 02:59:11.722888 20599 tablet_copy_service.cc:140] P 1c50de89a5a845668696dcb28c3badd6: Received BeginTabletCopySession request for tablet 505580c93c694bbb8ea96fdf9b17ab46 from peer 7a9daf5d42a84615a6383cfec9a10608 ({username='slave'} at 127.0.0.1:48526)
I20260129 02:59:11.723606 20599 tablet_copy_service.cc:161] P 1c50de89a5a845668696dcb28c3badd6: Beginning new tablet copy session on tablet 505580c93c694bbb8ea96fdf9b17ab46 from peer 7a9daf5d42a84615a6383cfec9a10608 at {username='slave'} at 127.0.0.1:48526: session id = 7a9daf5d42a84615a6383cfec9a10608-505580c93c694bbb8ea96fdf9b17ab46
I20260129 02:59:11.736536 21618 consensus_queue.cc:260] T 45cc5f3d2688486d9b6dbe7e666dedde P 7a9daf5d42a84615a6383cfec9a10608 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } attrs { replace: true } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: NON_VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: true } }
I20260129 02:59:11.738587 20599 tablet_copy_source_session.cc:215] T 505580c93c694bbb8ea96fdf9b17ab46 P 1c50de89a5a845668696dcb28c3badd6: Tablet Copy: opened 0 blocks and 1 log segments
I20260129 02:59:11.742508 21623 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 505580c93c694bbb8ea96fdf9b17ab46. 1 dirs total, 0 dirs full, 0 dirs failed
I20260129 02:59:11.751679 21618 ts_tablet_manager.cc:1434] T 45cc5f3d2688486d9b6dbe7e666dedde P 7a9daf5d42a84615a6383cfec9a10608: Time spent starting tablet: real 0.039s user 0.005s sys 0.011s
I20260129 02:59:11.763970 20599 tablet_copy_service.cc:342] P 1c50de89a5a845668696dcb28c3badd6: Request end of tablet copy session 7a9daf5d42a84615a6383cfec9a10608-45cc5f3d2688486d9b6dbe7e666dedde received from {username='slave'} at 127.0.0.1:48526
I20260129 02:59:11.764530 20599 tablet_copy_service.cc:434] P 1c50de89a5a845668696dcb28c3badd6: ending tablet copy session 7a9daf5d42a84615a6383cfec9a10608-45cc5f3d2688486d9b6dbe7e666dedde on tablet 45cc5f3d2688486d9b6dbe7e666dedde with peer 7a9daf5d42a84615a6383cfec9a10608
I20260129 02:59:11.769883 20600 tablet_copy_source_session.cc:215] T dc2f4cb752cf4d59b5f6bc689f5d8104 P 1c50de89a5a845668696dcb28c3badd6: Tablet Copy: opened 0 blocks and 1 log segments
I20260129 02:59:11.776942 21621 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet dc2f4cb752cf4d59b5f6bc689f5d8104. 1 dirs total, 0 dirs full, 0 dirs failed
I20260129 02:59:11.789179 21618 ts_tablet_manager.cc:933] T c934d31efd3f4e5195605ddc354d14e1 P 7a9daf5d42a84615a6383cfec9a10608: Initiating tablet copy from peer 197b880fa39d432a9b472845876bfaf0 (127.18.211.1:42027)
I20260129 02:59:11.791549 21623 tablet_copy_client.cc:806] T 505580c93c694bbb8ea96fdf9b17ab46 P 7a9daf5d42a84615a6383cfec9a10608: tablet copy: Starting download of 0 data blocks...
I20260129 02:59:11.792264 21623 tablet_copy_client.cc:670] T 505580c93c694bbb8ea96fdf9b17ab46 P 7a9daf5d42a84615a6383cfec9a10608: tablet copy: Starting download of 1 WAL segments...
I20260129 02:59:11.796187 21623 tablet_copy_client.cc:538] T 505580c93c694bbb8ea96fdf9b17ab46 P 7a9daf5d42a84615a6383cfec9a10608: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260129 02:59:11.799366 21621 tablet_copy_client.cc:806] T dc2f4cb752cf4d59b5f6bc689f5d8104 P 7a9daf5d42a84615a6383cfec9a10608: tablet copy: Starting download of 0 data blocks...
I20260129 02:59:11.799988 21621 tablet_copy_client.cc:670] T dc2f4cb752cf4d59b5f6bc689f5d8104 P 7a9daf5d42a84615a6383cfec9a10608: tablet copy: Starting download of 1 WAL segments...
I20260129 02:59:11.804519 21621 tablet_copy_client.cc:538] T dc2f4cb752cf4d59b5f6bc689f5d8104 P 7a9daf5d42a84615a6383cfec9a10608: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260129 02:59:11.807835 21618 tablet_copy_client.cc:323] T c934d31efd3f4e5195605ddc354d14e1 P 7a9daf5d42a84615a6383cfec9a10608: tablet copy: Beginning tablet copy session from remote peer at address 127.18.211.1:42027
I20260129 02:59:11.810019 21623 tablet_bootstrap.cc:492] T 505580c93c694bbb8ea96fdf9b17ab46 P 7a9daf5d42a84615a6383cfec9a10608: Bootstrap starting.
I20260129 02:59:11.809950 20450 tablet_copy_service.cc:140] P 197b880fa39d432a9b472845876bfaf0: Received BeginTabletCopySession request for tablet c934d31efd3f4e5195605ddc354d14e1 from peer 7a9daf5d42a84615a6383cfec9a10608 ({username='slave'} at 127.0.0.1:48356)
I20260129 02:59:11.810756 20450 tablet_copy_service.cc:161] P 197b880fa39d432a9b472845876bfaf0: Beginning new tablet copy session on tablet c934d31efd3f4e5195605ddc354d14e1 from peer 7a9daf5d42a84615a6383cfec9a10608 at {username='slave'} at 127.0.0.1:48356: session id = 7a9daf5d42a84615a6383cfec9a10608-c934d31efd3f4e5195605ddc354d14e1
I20260129 02:59:11.815004 21621 tablet_bootstrap.cc:492] T dc2f4cb752cf4d59b5f6bc689f5d8104 P 7a9daf5d42a84615a6383cfec9a10608: Bootstrap starting.
I20260129 02:59:11.819607 20450 tablet_copy_source_session.cc:215] T c934d31efd3f4e5195605ddc354d14e1 P 197b880fa39d432a9b472845876bfaf0: Tablet Copy: opened 0 blocks and 1 log segments
I20260129 02:59:11.825124 21618 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c934d31efd3f4e5195605ddc354d14e1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260129 02:59:11.853586 21618 tablet_copy_client.cc:806] T c934d31efd3f4e5195605ddc354d14e1 P 7a9daf5d42a84615a6383cfec9a10608: tablet copy: Starting download of 0 data blocks...
I20260129 02:59:11.854297 21618 tablet_copy_client.cc:670] T c934d31efd3f4e5195605ddc354d14e1 P 7a9daf5d42a84615a6383cfec9a10608: tablet copy: Starting download of 1 WAL segments...
I20260129 02:59:11.858840 21618 tablet_copy_client.cc:538] T c934d31efd3f4e5195605ddc354d14e1 P 7a9daf5d42a84615a6383cfec9a10608: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260129 02:59:11.861050 21623 tablet_bootstrap.cc:492] T 505580c93c694bbb8ea96fdf9b17ab46 P 7a9daf5d42a84615a6383cfec9a10608: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260129 02:59:11.861994 21623 tablet_bootstrap.cc:492] T 505580c93c694bbb8ea96fdf9b17ab46 P 7a9daf5d42a84615a6383cfec9a10608: Bootstrap complete.
I20260129 02:59:11.862668 21623 ts_tablet_manager.cc:1403] T 505580c93c694bbb8ea96fdf9b17ab46 P 7a9daf5d42a84615a6383cfec9a10608: Time spent bootstrapping tablet: real 0.053s user 0.025s sys 0.006s
I20260129 02:59:11.870069 21618 tablet_bootstrap.cc:492] T c934d31efd3f4e5195605ddc354d14e1 P 7a9daf5d42a84615a6383cfec9a10608: Bootstrap starting.
I20260129 02:59:11.869987 21623 raft_consensus.cc:359] T 505580c93c694bbb8ea96fdf9b17ab46 P 7a9daf5d42a84615a6383cfec9a10608 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } attrs { replace: true } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: NON_VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: true } }
I20260129 02:59:11.870951 21623 raft_consensus.cc:740] T 505580c93c694bbb8ea96fdf9b17ab46 P 7a9daf5d42a84615a6383cfec9a10608 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 7a9daf5d42a84615a6383cfec9a10608, State: Initialized, Role: LEARNER
I20260129 02:59:11.872736 21623 consensus_queue.cc:260] T 505580c93c694bbb8ea96fdf9b17ab46 P 7a9daf5d42a84615a6383cfec9a10608 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } attrs { replace: true } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: NON_VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: true } }
I20260129 02:59:11.877017 21623 ts_tablet_manager.cc:1434] T 505580c93c694bbb8ea96fdf9b17ab46 P 7a9daf5d42a84615a6383cfec9a10608: Time spent starting tablet: real 0.014s user 0.010s sys 0.000s
I20260129 02:59:11.888146 20600 tablet_copy_service.cc:342] P 1c50de89a5a845668696dcb28c3badd6: Request end of tablet copy session 7a9daf5d42a84615a6383cfec9a10608-505580c93c694bbb8ea96fdf9b17ab46 received from {username='slave'} at 127.0.0.1:48526
I20260129 02:59:11.888808 20600 tablet_copy_service.cc:434] P 1c50de89a5a845668696dcb28c3badd6: ending tablet copy session 7a9daf5d42a84615a6383cfec9a10608-505580c93c694bbb8ea96fdf9b17ab46 on tablet 505580c93c694bbb8ea96fdf9b17ab46 with peer 7a9daf5d42a84615a6383cfec9a10608
I20260129 02:59:11.892444 21621 tablet_bootstrap.cc:492] T dc2f4cb752cf4d59b5f6bc689f5d8104 P 7a9daf5d42a84615a6383cfec9a10608: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260129 02:59:11.893488 21621 tablet_bootstrap.cc:492] T dc2f4cb752cf4d59b5f6bc689f5d8104 P 7a9daf5d42a84615a6383cfec9a10608: Bootstrap complete.
I20260129 02:59:11.894207 21621 ts_tablet_manager.cc:1403] T dc2f4cb752cf4d59b5f6bc689f5d8104 P 7a9daf5d42a84615a6383cfec9a10608: Time spent bootstrapping tablet: real 0.079s user 0.025s sys 0.007s
I20260129 02:59:11.897292 21621 raft_consensus.cc:359] T dc2f4cb752cf4d59b5f6bc689f5d8104 P 7a9daf5d42a84615a6383cfec9a10608 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } attrs { replace: true } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: NON_VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: true } }
I20260129 02:59:11.898125 21621 raft_consensus.cc:740] T dc2f4cb752cf4d59b5f6bc689f5d8104 P 7a9daf5d42a84615a6383cfec9a10608 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 7a9daf5d42a84615a6383cfec9a10608, State: Initialized, Role: LEARNER
I20260129 02:59:11.910199 21621 consensus_queue.cc:260] T dc2f4cb752cf4d59b5f6bc689f5d8104 P 7a9daf5d42a84615a6383cfec9a10608 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } attrs { replace: true } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: NON_VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: true } }
I20260129 02:59:11.913048 21621 ts_tablet_manager.cc:1434] T dc2f4cb752cf4d59b5f6bc689f5d8104 P 7a9daf5d42a84615a6383cfec9a10608: Time spent starting tablet: real 0.019s user 0.002s sys 0.006s
I20260129 02:59:11.914752 20600 tablet_copy_service.cc:342] P 1c50de89a5a845668696dcb28c3badd6: Request end of tablet copy session 7a9daf5d42a84615a6383cfec9a10608-dc2f4cb752cf4d59b5f6bc689f5d8104 received from {username='slave'} at 127.0.0.1:48526
I20260129 02:59:11.915184 20600 tablet_copy_service.cc:434] P 1c50de89a5a845668696dcb28c3badd6: ending tablet copy session 7a9daf5d42a84615a6383cfec9a10608-dc2f4cb752cf4d59b5f6bc689f5d8104 on tablet dc2f4cb752cf4d59b5f6bc689f5d8104 with peer 7a9daf5d42a84615a6383cfec9a10608
I20260129 02:59:11.918483 21618 tablet_bootstrap.cc:492] T c934d31efd3f4e5195605ddc354d14e1 P 7a9daf5d42a84615a6383cfec9a10608: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260129 02:59:11.919417 21618 tablet_bootstrap.cc:492] T c934d31efd3f4e5195605ddc354d14e1 P 7a9daf5d42a84615a6383cfec9a10608: Bootstrap complete.
I20260129 02:59:11.920112 21618 ts_tablet_manager.cc:1403] T c934d31efd3f4e5195605ddc354d14e1 P 7a9daf5d42a84615a6383cfec9a10608: Time spent bootstrapping tablet: real 0.050s user 0.018s sys 0.029s
I20260129 02:59:11.923287 21618 raft_consensus.cc:359] T c934d31efd3f4e5195605ddc354d14e1 P 7a9daf5d42a84615a6383cfec9a10608 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } attrs { replace: true } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: NON_VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: true } }
I20260129 02:59:11.924124 21618 raft_consensus.cc:740] T c934d31efd3f4e5195605ddc354d14e1 P 7a9daf5d42a84615a6383cfec9a10608 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 7a9daf5d42a84615a6383cfec9a10608, State: Initialized, Role: LEARNER
I20260129 02:59:11.924772 21618 consensus_queue.cc:260] T c934d31efd3f4e5195605ddc354d14e1 P 7a9daf5d42a84615a6383cfec9a10608 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } attrs { replace: true } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: NON_VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: true } }
I20260129 02:59:11.927248 21618 ts_tablet_manager.cc:1434] T c934d31efd3f4e5195605ddc354d14e1 P 7a9daf5d42a84615a6383cfec9a10608: Time spent starting tablet: real 0.007s user 0.005s sys 0.001s
I20260129 02:59:11.928804 20450 tablet_copy_service.cc:342] P 197b880fa39d432a9b472845876bfaf0: Request end of tablet copy session 7a9daf5d42a84615a6383cfec9a10608-c934d31efd3f4e5195605ddc354d14e1 received from {username='slave'} at 127.0.0.1:48356
I20260129 02:59:11.929245 20450 tablet_copy_service.cc:434] P 197b880fa39d432a9b472845876bfaf0: ending tablet copy session 7a9daf5d42a84615a6383cfec9a10608-c934d31efd3f4e5195605ddc354d14e1 on tablet c934d31efd3f4e5195605ddc354d14e1 with peer 7a9daf5d42a84615a6383cfec9a10608
I20260129 02:59:11.943568 21131 raft_consensus.cc:1217] T 45cc5f3d2688486d9b6dbe7e666dedde P 7a9daf5d42a84615a6383cfec9a10608 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20260129 02:59:12.194944 21134 raft_consensus.cc:1217] T c934d31efd3f4e5195605ddc354d14e1 P 7a9daf5d42a84615a6383cfec9a10608 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20260129 02:59:12.226024 21131 raft_consensus.cc:1217] T 505580c93c694bbb8ea96fdf9b17ab46 P 7a9daf5d42a84615a6383cfec9a10608 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20260129 02:59:12.263971 21131 raft_consensus.cc:1217] T dc2f4cb752cf4d59b5f6bc689f5d8104 P 7a9daf5d42a84615a6383cfec9a10608 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20260129 02:59:12.340759 19276 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260129 02:59:12.418373 20441 tablet_service.cc:2066] Received LeaderStepDown RPC: tablet_id: "4e3805ffa70242e8ade66da260dddf66"
dest_uuid: "197b880fa39d432a9b472845876bfaf0"
mode: GRACEFUL
new_leader_uuid: "7a9daf5d42a84615a6383cfec9a10608"
from {username='slave'} at 127.0.0.1:57776
I20260129 02:59:12.419018 20441 raft_consensus.cc:606] T 4e3805ffa70242e8ade66da260dddf66 P 197b880fa39d432a9b472845876bfaf0 [term 1 LEADER]: Received request to transfer leadership to TS 7a9daf5d42a84615a6383cfec9a10608
I20260129 02:59:12.431514 20510 tablet_service.cc:2066] Received LeaderStepDown RPC: tablet_id: "75be4bd6bd45474795fd472eff2953ac"
dest_uuid: "974b802141bd4f13beffef2aa03f88f9"
mode: GRACEFUL
new_leader_uuid: "197b880fa39d432a9b472845876bfaf0"
from {username='slave'} at 127.0.0.1:52008
I20260129 02:59:12.432045 20510 raft_consensus.cc:606] T 75be4bd6bd45474795fd472eff2953ac P 974b802141bd4f13beffef2aa03f88f9 [term 3 LEADER]: Received request to transfer leadership to TS 197b880fa39d432a9b472845876bfaf0
I20260129 02:59:12.434598 20441 tablet_service.cc:2066] Received LeaderStepDown RPC: tablet_id: "a1a39dcd328e4268ae41b988c66b4a75"
dest_uuid: "197b880fa39d432a9b472845876bfaf0"
mode: GRACEFUL
new_leader_uuid: "1c50de89a5a845668696dcb28c3badd6"
from {username='slave'} at 127.0.0.1:57776
I20260129 02:59:12.435117 20441 raft_consensus.cc:606] T a1a39dcd328e4268ae41b988c66b4a75 P 197b880fa39d432a9b472845876bfaf0 [term 1 LEADER]: Received request to transfer leadership to TS 1c50de89a5a845668696dcb28c3badd6
I20260129 02:59:12.437685 20441 tablet_service.cc:2066] Received LeaderStepDown RPC: tablet_id: "a609ac02fae84725ab743f95f51d33ce"
dest_uuid: "197b880fa39d432a9b472845876bfaf0"
mode: GRACEFUL
new_leader_uuid: "7a9daf5d42a84615a6383cfec9a10608"
from {username='slave'} at 127.0.0.1:57776
I20260129 02:59:12.438165 20441 raft_consensus.cc:606] T a609ac02fae84725ab743f95f51d33ce P 197b880fa39d432a9b472845876bfaf0 [term 3 LEADER]: Received request to transfer leadership to TS 7a9daf5d42a84615a6383cfec9a10608
I20260129 02:59:12.451722 20581 tablet_service.cc:2066] Received LeaderStepDown RPC: tablet_id: "b189f08e16a74a11980cc55d99359a0a"
dest_uuid: "1c50de89a5a845668696dcb28c3badd6"
mode: GRACEFUL
new_leader_uuid: "197b880fa39d432a9b472845876bfaf0"
from {username='slave'} at 127.0.0.1:54860
I20260129 02:59:12.452304 20581 raft_consensus.cc:606] T b189f08e16a74a11980cc55d99359a0a P 1c50de89a5a845668696dcb28c3badd6 [term 1 LEADER]: Received request to transfer leadership to TS 197b880fa39d432a9b472845876bfaf0
I20260129 02:59:12.454689 20441 tablet_service.cc:2066] Received LeaderStepDown RPC: tablet_id: "c891d2bff4ce4568861bbc992e91fbb0"
dest_uuid: "197b880fa39d432a9b472845876bfaf0"
mode: GRACEFUL
new_leader_uuid: "1c50de89a5a845668696dcb28c3badd6"
from {username='slave'} at 127.0.0.1:57776
I20260129 02:59:12.455226 20441 raft_consensus.cc:606] T c891d2bff4ce4568861bbc992e91fbb0 P 197b880fa39d432a9b472845876bfaf0 [term 3 LEADER]: Received request to transfer leadership to TS 1c50de89a5a845668696dcb28c3badd6
I20260129 02:59:12.457655 20440 tablet_service.cc:2066] Received LeaderStepDown RPC: tablet_id: "fcddab6bca684adf8ea032289a00f90e"
dest_uuid: "197b880fa39d432a9b472845876bfaf0"
mode: GRACEFUL
new_leader_uuid: "1c50de89a5a845668696dcb28c3badd6"
from {username='slave'} at 127.0.0.1:57776
I20260129 02:59:12.458134 20440 raft_consensus.cc:606] T fcddab6bca684adf8ea032289a00f90e P 197b880fa39d432a9b472845876bfaf0 [term 1 LEADER]: Received request to transfer leadership to TS 1c50de89a5a845668696dcb28c3badd6
I20260129 02:59:12.459141 19276 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 7
I20260129 02:59:12.460484 19276 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20260129 02:59:12.461272 19276 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260129 02:59:12.550956 21594 raft_consensus.cc:1064] T 45cc5f3d2688486d9b6dbe7e666dedde P 1c50de89a5a845668696dcb28c3badd6: attempting to promote NON_VOTER 7a9daf5d42a84615a6383cfec9a10608 to VOTER
I20260129 02:59:12.553346 21594 consensus_queue.cc:237] T 45cc5f3d2688486d9b6dbe7e666dedde P 1c50de89a5a845668696dcb28c3badd6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } attrs { replace: true } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } }
I20260129 02:59:12.559815 21131 raft_consensus.cc:1275] T 45cc5f3d2688486d9b6dbe7e666dedde P 7a9daf5d42a84615a6383cfec9a10608 [term 1 LEARNER]: Refusing update from remote peer 1c50de89a5a845668696dcb28c3badd6: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260129 02:59:12.561384 20440 raft_consensus.cc:1275] T 45cc5f3d2688486d9b6dbe7e666dedde P 197b880fa39d432a9b472845876bfaf0 [term 1 FOLLOWER]: Refusing update from remote peer 1c50de89a5a845668696dcb28c3badd6: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260129 02:59:12.561533 20510 raft_consensus.cc:1275] T 45cc5f3d2688486d9b6dbe7e666dedde P 974b802141bd4f13beffef2aa03f88f9 [term 1 FOLLOWER]: Refusing update from remote peer 1c50de89a5a845668696dcb28c3badd6: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260129 02:59:12.562812 21638 consensus_queue.cc:1048] T 45cc5f3d2688486d9b6dbe7e666dedde P 1c50de89a5a845668696dcb28c3badd6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260129 02:59:12.563741 21594 consensus_queue.cc:1048] T 45cc5f3d2688486d9b6dbe7e666dedde P 1c50de89a5a845668696dcb28c3badd6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260129 02:59:12.564808 21636 consensus_queue.cc:1048] T 45cc5f3d2688486d9b6dbe7e666dedde P 1c50de89a5a845668696dcb28c3badd6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260129 02:59:12.586736 21614 raft_consensus.cc:993] T 4e3805ffa70242e8ade66da260dddf66 P 197b880fa39d432a9b472845876bfaf0: : Instructing follower 7a9daf5d42a84615a6383cfec9a10608 to start an election
I20260129 02:59:12.587213 21614 raft_consensus.cc:1081] T 4e3805ffa70242e8ade66da260dddf66 P 197b880fa39d432a9b472845876bfaf0 [term 1 LEADER]: Signalling peer 7a9daf5d42a84615a6383cfec9a10608 to start an election
I20260129 02:59:12.589028 21134 tablet_service.cc:2038] Received Run Leader Election RPC: tablet_id: "4e3805ffa70242e8ade66da260dddf66"
dest_uuid: "7a9daf5d42a84615a6383cfec9a10608"
from {username='slave'} at 127.0.0.1:34060
I20260129 02:59:12.589663 21134 raft_consensus.cc:493] T 4e3805ffa70242e8ade66da260dddf66 P 7a9daf5d42a84615a6383cfec9a10608 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260129 02:59:12.590015 21134 raft_consensus.cc:3060] T 4e3805ffa70242e8ade66da260dddf66 P 7a9daf5d42a84615a6383cfec9a10608 [term 1 FOLLOWER]: Advancing to term 2
I20260129 02:59:12.589274 21636 raft_consensus.cc:2955] T 45cc5f3d2688486d9b6dbe7e666dedde P 1c50de89a5a845668696dcb28c3badd6 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 7a9daf5d42a84615a6383cfec9a10608 (127.18.211.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } attrs { replace: true } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } } }
I20260129 02:59:12.593364 20511 raft_consensus.cc:2955] T 45cc5f3d2688486d9b6dbe7e666dedde P 974b802141bd4f13beffef2aa03f88f9 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 7a9daf5d42a84615a6383cfec9a10608 (127.18.211.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } attrs { replace: true } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } } }
I20260129 02:59:12.607417 21614 raft_consensus.cc:993] T a609ac02fae84725ab743f95f51d33ce P 197b880fa39d432a9b472845876bfaf0: : Instructing follower 7a9daf5d42a84615a6383cfec9a10608 to start an election
I20260129 02:59:12.608534 21624 raft_consensus.cc:1081] T a609ac02fae84725ab743f95f51d33ce P 197b880fa39d432a9b472845876bfaf0 [term 3 LEADER]: Signalling peer 7a9daf5d42a84615a6383cfec9a10608 to start an election
I20260129 02:59:12.610460 21135 tablet_service.cc:2038] Received Run Leader Election RPC: tablet_id: "a609ac02fae84725ab743f95f51d33ce"
dest_uuid: "7a9daf5d42a84615a6383cfec9a10608"
from {username='slave'} at 127.0.0.1:34060
I20260129 02:59:12.611006 21135 raft_consensus.cc:493] T a609ac02fae84725ab743f95f51d33ce P 7a9daf5d42a84615a6383cfec9a10608 [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20260129 02:59:12.611318 21135 raft_consensus.cc:3060] T a609ac02fae84725ab743f95f51d33ce P 7a9daf5d42a84615a6383cfec9a10608 [term 3 FOLLOWER]: Advancing to term 4
I20260129 02:59:12.615980 21134 raft_consensus.cc:515] T 4e3805ffa70242e8ade66da260dddf66 P 7a9daf5d42a84615a6383cfec9a10608 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } }
I20260129 02:59:12.626832 21135 raft_consensus.cc:515] T a609ac02fae84725ab743f95f51d33ce P 7a9daf5d42a84615a6383cfec9a10608 [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } }
I20260129 02:59:12.629977 20336 catalog_manager.cc:5654] T 45cc5f3d2688486d9b6dbe7e666dedde P 974b802141bd4f13beffef2aa03f88f9 reported cstate change: config changed from index 2 to 3, 7a9daf5d42a84615a6383cfec9a10608 (127.18.211.4) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "1c50de89a5a845668696dcb28c3badd6" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } attrs { replace: true } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } } }
I20260129 02:59:12.631307 20440 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a609ac02fae84725ab743f95f51d33ce" candidate_uuid: "7a9daf5d42a84615a6383cfec9a10608" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: true dest_uuid: "197b880fa39d432a9b472845876bfaf0"
I20260129 02:59:12.645011 20440 raft_consensus.cc:3055] T a609ac02fae84725ab743f95f51d33ce P 197b880fa39d432a9b472845876bfaf0 [term 3 LEADER]: Stepping down as leader of term 3
I20260129 02:59:12.645334 20440 raft_consensus.cc:740] T a609ac02fae84725ab743f95f51d33ce P 197b880fa39d432a9b472845876bfaf0 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 197b880fa39d432a9b472845876bfaf0, State: Running, Role: LEADER
I20260129 02:59:12.646087 20440 consensus_queue.cc:260] T a609ac02fae84725ab743f95f51d33ce P 197b880fa39d432a9b472845876bfaf0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.5, Last appended by leader: 5, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } }
I20260129 02:59:12.647228 20440 raft_consensus.cc:3060] T a609ac02fae84725ab743f95f51d33ce P 197b880fa39d432a9b472845876bfaf0 [term 3 FOLLOWER]: Advancing to term 4
I20260129 02:59:12.646561 21131 raft_consensus.cc:2955] T 45cc5f3d2688486d9b6dbe7e666dedde P 7a9daf5d42a84615a6383cfec9a10608 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 7a9daf5d42a84615a6383cfec9a10608 (127.18.211.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } attrs { replace: true } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } } }
I20260129 02:59:12.633342 20441 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4e3805ffa70242e8ade66da260dddf66" candidate_uuid: "7a9daf5d42a84615a6383cfec9a10608" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "197b880fa39d432a9b472845876bfaf0"
I20260129 02:59:12.648337 20441 raft_consensus.cc:3055] T 4e3805ffa70242e8ade66da260dddf66 P 197b880fa39d432a9b472845876bfaf0 [term 1 LEADER]: Stepping down as leader of term 1
I20260129 02:59:12.648626 20441 raft_consensus.cc:740] T 4e3805ffa70242e8ade66da260dddf66 P 197b880fa39d432a9b472845876bfaf0 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 197b880fa39d432a9b472845876bfaf0, State: Running, Role: LEADER
I20260129 02:59:12.649410 20441 consensus_queue.cc:260] T 4e3805ffa70242e8ade66da260dddf66 P 197b880fa39d432a9b472845876bfaf0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } }
I20260129 02:59:12.649892 21599 raft_consensus.cc:993] T b189f08e16a74a11980cc55d99359a0a P 1c50de89a5a845668696dcb28c3badd6: : Instructing follower 197b880fa39d432a9b472845876bfaf0 to start an election
I20260129 02:59:12.650408 21599 raft_consensus.cc:1081] T b189f08e16a74a11980cc55d99359a0a P 1c50de89a5a845668696dcb28c3badd6 [term 1 LEADER]: Signalling peer 197b880fa39d432a9b472845876bfaf0 to start an election
I20260129 02:59:12.650532 20441 raft_consensus.cc:3060] T 4e3805ffa70242e8ade66da260dddf66 P 197b880fa39d432a9b472845876bfaf0 [term 1 FOLLOWER]: Advancing to term 2
I20260129 02:59:12.651185 21633 raft_consensus.cc:1064] T c934d31efd3f4e5195605ddc354d14e1 P 197b880fa39d432a9b472845876bfaf0: attempting to promote NON_VOTER 7a9daf5d42a84615a6383cfec9a10608 to VOTER
I20260129 02:59:12.632818 21134 leader_election.cc:290] T 4e3805ffa70242e8ade66da260dddf66 P 7a9daf5d42a84615a6383cfec9a10608 [CANDIDATE]: Term 2 election: Requested vote from peers 197b880fa39d432a9b472845876bfaf0 (127.18.211.1:42027), 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3:37735)
I20260129 02:59:12.652555 20437 tablet_service.cc:2038] Received Run Leader Election RPC: tablet_id: "b189f08e16a74a11980cc55d99359a0a"
dest_uuid: "197b880fa39d432a9b472845876bfaf0"
from {username='slave'} at 127.0.0.1:55676
I20260129 02:59:12.653055 20437 raft_consensus.cc:493] T b189f08e16a74a11980cc55d99359a0a P 197b880fa39d432a9b472845876bfaf0 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260129 02:59:12.653580 20437 raft_consensus.cc:3060] T b189f08e16a74a11980cc55d99359a0a P 197b880fa39d432a9b472845876bfaf0 [term 1 FOLLOWER]: Advancing to term 2
I20260129 02:59:12.662501 21638 raft_consensus.cc:1064] T 505580c93c694bbb8ea96fdf9b17ab46 P 1c50de89a5a845668696dcb28c3badd6: attempting to promote NON_VOTER 7a9daf5d42a84615a6383cfec9a10608 to VOTER
I20260129 02:59:12.664342 21633 consensus_queue.cc:237] T c934d31efd3f4e5195605ddc354d14e1 P 197b880fa39d432a9b472845876bfaf0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } attrs { replace: true } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } }
I20260129 02:59:12.665278 21638 consensus_queue.cc:237] T 505580c93c694bbb8ea96fdf9b17ab46 P 1c50de89a5a845668696dcb28c3badd6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } attrs { replace: true } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } }
I20260129 02:59:12.640035 21135 leader_election.cc:290] T a609ac02fae84725ab743f95f51d33ce P 7a9daf5d42a84615a6383cfec9a10608 [CANDIDATE]: Term 4 election: Requested vote from peers 197b880fa39d432a9b472845876bfaf0 (127.18.211.1:42027), 974b802141bd4f13beffef2aa03f88f9 (127.18.211.2:34517)
I20260129 02:59:12.666131 20437 raft_consensus.cc:515] T b189f08e16a74a11980cc55d99359a0a P 197b880fa39d432a9b472845876bfaf0 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:59:12.640659 20511 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a609ac02fae84725ab743f95f51d33ce" candidate_uuid: "7a9daf5d42a84615a6383cfec9a10608" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: true dest_uuid: "974b802141bd4f13beffef2aa03f88f9"
I20260129 02:59:12.644312 20438 raft_consensus.cc:2955] T 45cc5f3d2688486d9b6dbe7e666dedde P 197b880fa39d432a9b472845876bfaf0 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 7a9daf5d42a84615a6383cfec9a10608 (127.18.211.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } attrs { replace: true } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } } }
I20260129 02:59:12.633342 20581 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4e3805ffa70242e8ade66da260dddf66" candidate_uuid: "7a9daf5d42a84615a6383cfec9a10608" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "1c50de89a5a845668696dcb28c3badd6"
I20260129 02:59:12.684103 20507 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b189f08e16a74a11980cc55d99359a0a" candidate_uuid: "197b880fa39d432a9b472845876bfaf0" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "974b802141bd4f13beffef2aa03f88f9"
I20260129 02:59:12.694635 20507 raft_consensus.cc:3060] T b189f08e16a74a11980cc55d99359a0a P 974b802141bd4f13beffef2aa03f88f9 [term 1 FOLLOWER]: Advancing to term 2
I20260129 02:59:12.697767 21135 raft_consensus.cc:1275] T c934d31efd3f4e5195605ddc354d14e1 P 7a9daf5d42a84615a6383cfec9a10608 [term 1 LEARNER]: Refusing update from remote peer 197b880fa39d432a9b472845876bfaf0: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260129 02:59:12.699105 20587 raft_consensus.cc:1275] T c934d31efd3f4e5195605ddc354d14e1 P 1c50de89a5a845668696dcb28c3badd6 [term 1 FOLLOWER]: Refusing update from remote peer 197b880fa39d432a9b472845876bfaf0: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260129 02:59:12.685909 21131 raft_consensus.cc:1275] T 505580c93c694bbb8ea96fdf9b17ab46 P 7a9daf5d42a84615a6383cfec9a10608 [term 1 LEARNER]: Refusing update from remote peer 1c50de89a5a845668696dcb28c3badd6: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260129 02:59:12.701188 21519 consensus_queue.cc:1048] T c934d31efd3f4e5195605ddc354d14e1 P 197b880fa39d432a9b472845876bfaf0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260129 02:59:12.702185 21614 consensus_queue.cc:1048] T c934d31efd3f4e5195605ddc354d14e1 P 197b880fa39d432a9b472845876bfaf0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260129 02:59:12.703104 21599 consensus_queue.cc:1048] T 505580c93c694bbb8ea96fdf9b17ab46 P 1c50de89a5a845668696dcb28c3badd6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260129 02:59:12.704697 20507 raft_consensus.cc:2468] T b189f08e16a74a11980cc55d99359a0a P 974b802141bd4f13beffef2aa03f88f9 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 197b880fa39d432a9b472845876bfaf0 in term 2.
I20260129 02:59:12.686137 20581 raft_consensus.cc:3060] T 4e3805ffa70242e8ade66da260dddf66 P 1c50de89a5a845668696dcb28c3badd6 [term 1 FOLLOWER]: Advancing to term 2
I20260129 02:59:12.706076 20407 leader_election.cc:304] T b189f08e16a74a11980cc55d99359a0a P 197b880fa39d432a9b472845876bfaf0 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 197b880fa39d432a9b472845876bfaf0, 974b802141bd4f13beffef2aa03f88f9; no voters:
I20260129 02:59:12.708132 21624 raft_consensus.cc:2804] T b189f08e16a74a11980cc55d99359a0a P 197b880fa39d432a9b472845876bfaf0 [term 2 FOLLOWER]: Leader election won for term 2
I20260129 02:59:12.708603 21624 raft_consensus.cc:697] T b189f08e16a74a11980cc55d99359a0a P 197b880fa39d432a9b472845876bfaf0 [term 2 LEADER]: Becoming Leader. State: Replica: 197b880fa39d432a9b472845876bfaf0, State: Running, Role: LEADER
I20260129 02:59:12.709440 21624 consensus_queue.cc:237] T b189f08e16a74a11980cc55d99359a0a P 197b880fa39d432a9b472845876bfaf0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:59:12.686286 20511 raft_consensus.cc:3060] T a609ac02fae84725ab743f95f51d33ce P 974b802141bd4f13beffef2aa03f88f9 [term 3 FOLLOWER]: Advancing to term 4
I20260129 02:59:12.714985 20513 raft_consensus.cc:1275] T c934d31efd3f4e5195605ddc354d14e1 P 974b802141bd4f13beffef2aa03f88f9 [term 1 FOLLOWER]: Refusing update from remote peer 197b880fa39d432a9b472845876bfaf0: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260129 02:59:12.686920 20440 raft_consensus.cc:2468] T a609ac02fae84725ab743f95f51d33ce P 197b880fa39d432a9b472845876bfaf0 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7a9daf5d42a84615a6383cfec9a10608 in term 4.
I20260129 02:59:12.717919 21565 consensus_queue.cc:1048] T c934d31efd3f4e5195605ddc354d14e1 P 197b880fa39d432a9b472845876bfaf0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260129 02:59:12.719424 21099 leader_election.cc:304] T a609ac02fae84725ab743f95f51d33ce P 7a9daf5d42a84615a6383cfec9a10608 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 197b880fa39d432a9b472845876bfaf0, 7a9daf5d42a84615a6383cfec9a10608; no voters:
I20260129 02:59:12.720571 21626 raft_consensus.cc:2804] T a609ac02fae84725ab743f95f51d33ce P 7a9daf5d42a84615a6383cfec9a10608 [term 4 FOLLOWER]: Leader election won for term 4
I20260129 02:59:12.683769 20437 leader_election.cc:290] T b189f08e16a74a11980cc55d99359a0a P 197b880fa39d432a9b472845876bfaf0 [CANDIDATE]: Term 2 election: Requested vote from peers 974b802141bd4f13beffef2aa03f88f9 (127.18.211.2:34517), 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3:37735)
I20260129 02:59:12.688841 20590 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b189f08e16a74a11980cc55d99359a0a" candidate_uuid: "197b880fa39d432a9b472845876bfaf0" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "1c50de89a5a845668696dcb28c3badd6"
I20260129 02:59:12.732221 20590 raft_consensus.cc:3055] T b189f08e16a74a11980cc55d99359a0a P 1c50de89a5a845668696dcb28c3badd6 [term 1 LEADER]: Stepping down as leader of term 1
I20260129 02:59:12.732527 20590 raft_consensus.cc:740] T b189f08e16a74a11980cc55d99359a0a P 1c50de89a5a845668696dcb28c3badd6 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 1c50de89a5a845668696dcb28c3badd6, State: Running, Role: LEADER
I20260129 02:59:12.691681 20510 raft_consensus.cc:1275] T 505580c93c694bbb8ea96fdf9b17ab46 P 974b802141bd4f13beffef2aa03f88f9 [term 1 FOLLOWER]: Refusing update from remote peer 1c50de89a5a845668696dcb28c3badd6: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260129 02:59:12.734093 20590 consensus_queue.cc:260] T b189f08e16a74a11980cc55d99359a0a P 1c50de89a5a845668696dcb28c3badd6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:59:12.735548 20590 raft_consensus.cc:3060] T b189f08e16a74a11980cc55d99359a0a P 1c50de89a5a845668696dcb28c3badd6 [term 1 FOLLOWER]: Advancing to term 2
I20260129 02:59:12.742547 21638 consensus_queue.cc:1048] T 505580c93c694bbb8ea96fdf9b17ab46 P 1c50de89a5a845668696dcb28c3badd6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260129 02:59:12.687769 20441 raft_consensus.cc:2468] T 4e3805ffa70242e8ade66da260dddf66 P 197b880fa39d432a9b472845876bfaf0 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7a9daf5d42a84615a6383cfec9a10608 in term 2.
I20260129 02:59:12.689523 20438 raft_consensus.cc:1275] T 505580c93c694bbb8ea96fdf9b17ab46 P 197b880fa39d432a9b472845876bfaf0 [term 1 FOLLOWER]: Refusing update from remote peer 1c50de89a5a845668696dcb28c3badd6: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260129 02:59:12.745712 21099 leader_election.cc:304] T 4e3805ffa70242e8ade66da260dddf66 P 7a9daf5d42a84615a6383cfec9a10608 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 197b880fa39d432a9b472845876bfaf0, 7a9daf5d42a84615a6383cfec9a10608; no voters:
I20260129 02:59:12.746667 21638 consensus_queue.cc:1048] T 505580c93c694bbb8ea96fdf9b17ab46 P 1c50de89a5a845668696dcb28c3badd6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260129 02:59:12.763495 20590 raft_consensus.cc:2468] T b189f08e16a74a11980cc55d99359a0a P 1c50de89a5a845668696dcb28c3badd6 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 197b880fa39d432a9b472845876bfaf0 in term 2.
I20260129 02:59:12.764695 20581 raft_consensus.cc:2468] T 4e3805ffa70242e8ade66da260dddf66 P 1c50de89a5a845668696dcb28c3badd6 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7a9daf5d42a84615a6383cfec9a10608 in term 2.
I20260129 02:59:12.768919 20511 raft_consensus.cc:2468] T a609ac02fae84725ab743f95f51d33ce P 974b802141bd4f13beffef2aa03f88f9 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7a9daf5d42a84615a6383cfec9a10608 in term 4.
I20260129 02:59:12.816586 21655 raft_consensus.cc:2804] T 4e3805ffa70242e8ade66da260dddf66 P 7a9daf5d42a84615a6383cfec9a10608 [term 2 FOLLOWER]: Leader election won for term 2
I20260129 02:59:12.817471 21655 raft_consensus.cc:697] T 4e3805ffa70242e8ade66da260dddf66 P 7a9daf5d42a84615a6383cfec9a10608 [term 2 LEADER]: Becoming Leader. State: Replica: 7a9daf5d42a84615a6383cfec9a10608, State: Running, Role: LEADER
W20260129 02:59:12.825245 21099 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.211.1:42027, user_credentials={real_user=slave}} blocked reactor thread for 79608.4us
I20260129 02:59:12.842546 21626 raft_consensus.cc:697] T a609ac02fae84725ab743f95f51d33ce P 7a9daf5d42a84615a6383cfec9a10608 [term 4 LEADER]: Becoming Leader. State: Replica: 7a9daf5d42a84615a6383cfec9a10608, State: Running, Role: LEADER
I20260129 02:59:12.844237 21626 consensus_queue.cc:237] T a609ac02fae84725ab743f95f51d33ce P 7a9daf5d42a84615a6383cfec9a10608 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.5, Last appended by leader: 5, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } }
W20260129 02:59:12.847018 21653 log.cc:927] Time spent T b189f08e16a74a11980cc55d99359a0a P 197b880fa39d432a9b472845876bfaf0: Append to log took a long time: real 0.102s user 0.001s sys 0.000s
I20260129 02:59:12.858767 21599 raft_consensus.cc:1064] T dc2f4cb752cf4d59b5f6bc689f5d8104 P 1c50de89a5a845668696dcb28c3badd6: attempting to promote NON_VOTER 7a9daf5d42a84615a6383cfec9a10608 to VOTER
I20260129 02:59:12.861284 21633 raft_consensus.cc:993] T fcddab6bca684adf8ea032289a00f90e P 197b880fa39d432a9b472845876bfaf0: : Instructing follower 1c50de89a5a845668696dcb28c3badd6 to start an election
I20260129 02:59:12.862061 21633 raft_consensus.cc:1081] T fcddab6bca684adf8ea032289a00f90e P 197b880fa39d432a9b472845876bfaf0 [term 1 LEADER]: Signalling peer 1c50de89a5a845668696dcb28c3badd6 to start an election
W20260129 02:59:12.865113 21658 log.cc:927] Time spent T c934d31efd3f4e5195605ddc354d14e1 P 974b802141bd4f13beffef2aa03f88f9: Append to log took a long time: real 0.070s user 0.000s sys 0.005s
I20260129 02:59:12.865621 20590 tablet_service.cc:2038] Received Run Leader Election RPC: tablet_id: "fcddab6bca684adf8ea032289a00f90e"
dest_uuid: "1c50de89a5a845668696dcb28c3badd6"
from {username='slave'} at 127.0.0.1:35754
I20260129 02:59:12.866858 20590 raft_consensus.cc:493] T fcddab6bca684adf8ea032289a00f90e P 1c50de89a5a845668696dcb28c3badd6 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260129 02:59:12.867301 20590 raft_consensus.cc:3060] T fcddab6bca684adf8ea032289a00f90e P 1c50de89a5a845668696dcb28c3badd6 [term 1 FOLLOWER]: Advancing to term 2
I20260129 02:59:12.869340 21663 raft_consensus.cc:993] T c891d2bff4ce4568861bbc992e91fbb0 P 197b880fa39d432a9b472845876bfaf0: : Instructing follower 1c50de89a5a845668696dcb28c3badd6 to start an election
I20260129 02:59:12.869096 21655 consensus_queue.cc:237] T 4e3805ffa70242e8ade66da260dddf66 P 7a9daf5d42a84615a6383cfec9a10608 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } }
I20260129 02:59:12.870369 21599 consensus_queue.cc:237] T dc2f4cb752cf4d59b5f6bc689f5d8104 P 1c50de89a5a845668696dcb28c3badd6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } attrs { replace: true } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } }
W20260129 02:59:12.866000 21651 log.cc:927] Time spent T 505580c93c694bbb8ea96fdf9b17ab46 P 7a9daf5d42a84615a6383cfec9a10608: Append to log took a long time: real 0.082s user 0.003s sys 0.004s
I20260129 02:59:12.874747 20590 raft_consensus.cc:515] T fcddab6bca684adf8ea032289a00f90e P 1c50de89a5a845668696dcb28c3badd6 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
W20260129 02:59:12.865759 21650 log.cc:927] Time spent T c934d31efd3f4e5195605ddc354d14e1 P 1c50de89a5a845668696dcb28c3badd6: Append to log took a long time: real 0.084s user 0.000s sys 0.008s
I20260129 02:59:12.869807 21663 raft_consensus.cc:1081] T c891d2bff4ce4568861bbc992e91fbb0 P 197b880fa39d432a9b472845876bfaf0 [term 3 LEADER]: Signalling peer 1c50de89a5a845668696dcb28c3badd6 to start an election
W20260129 02:59:12.865732 21657 log.cc:927] Time spent T 505580c93c694bbb8ea96fdf9b17ab46 P 974b802141bd4f13beffef2aa03f88f9: Append to log took a long time: real 0.074s user 0.004s sys 0.000s
I20260129 02:59:12.882704 20440 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fcddab6bca684adf8ea032289a00f90e" candidate_uuid: "1c50de89a5a845668696dcb28c3badd6" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "197b880fa39d432a9b472845876bfaf0"
I20260129 02:59:12.883477 20440 raft_consensus.cc:3055] T fcddab6bca684adf8ea032289a00f90e P 197b880fa39d432a9b472845876bfaf0 [term 1 LEADER]: Stepping down as leader of term 1
I20260129 02:59:12.883886 20440 raft_consensus.cc:740] T fcddab6bca684adf8ea032289a00f90e P 197b880fa39d432a9b472845876bfaf0 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 197b880fa39d432a9b472845876bfaf0, State: Running, Role: LEADER
I20260129 02:59:12.884615 20440 consensus_queue.cc:260] T fcddab6bca684adf8ea032289a00f90e P 197b880fa39d432a9b472845876bfaf0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:59:12.887673 20590 leader_election.cc:290] T fcddab6bca684adf8ea032289a00f90e P 1c50de89a5a845668696dcb28c3badd6 [CANDIDATE]: Term 2 election: Requested vote from peers 197b880fa39d432a9b472845876bfaf0 (127.18.211.1:42027), 974b802141bd4f13beffef2aa03f88f9 (127.18.211.2:34517)
I20260129 02:59:12.889169 20511 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fcddab6bca684adf8ea032289a00f90e" candidate_uuid: "1c50de89a5a845668696dcb28c3badd6" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "974b802141bd4f13beffef2aa03f88f9"
I20260129 02:59:12.889959 20511 raft_consensus.cc:3060] T fcddab6bca684adf8ea032289a00f90e P 974b802141bd4f13beffef2aa03f88f9 [term 1 FOLLOWER]: Advancing to term 2
I20260129 02:59:12.900671 20440 raft_consensus.cc:3060] T fcddab6bca684adf8ea032289a00f90e P 197b880fa39d432a9b472845876bfaf0 [term 1 FOLLOWER]: Advancing to term 2
I20260129 02:59:12.902760 21593 raft_consensus.cc:993] T 75be4bd6bd45474795fd472eff2953ac P 974b802141bd4f13beffef2aa03f88f9: : Instructing follower 197b880fa39d432a9b472845876bfaf0 to start an election
I20260129 02:59:12.903244 21593 raft_consensus.cc:1081] T 75be4bd6bd45474795fd472eff2953ac P 974b802141bd4f13beffef2aa03f88f9 [term 3 LEADER]: Signalling peer 197b880fa39d432a9b472845876bfaf0 to start an election
I20260129 02:59:12.912431 20438 tablet_service.cc:2038] Received Run Leader Election RPC: tablet_id: "75be4bd6bd45474795fd472eff2953ac"
dest_uuid: "197b880fa39d432a9b472845876bfaf0"
from {username='slave'} at 127.0.0.1:55672
I20260129 02:59:12.912957 20438 raft_consensus.cc:493] T 75be4bd6bd45474795fd472eff2953ac P 197b880fa39d432a9b472845876bfaf0 [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20260129 02:59:12.913295 20438 raft_consensus.cc:3060] T 75be4bd6bd45474795fd472eff2953ac P 197b880fa39d432a9b472845876bfaf0 [term 3 FOLLOWER]: Advancing to term 4
I20260129 02:59:12.913107 20511 raft_consensus.cc:2468] T fcddab6bca684adf8ea032289a00f90e P 974b802141bd4f13beffef2aa03f88f9 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1c50de89a5a845668696dcb28c3badd6 in term 2.
I20260129 02:59:12.916246 20440 raft_consensus.cc:2468] T fcddab6bca684adf8ea032289a00f90e P 197b880fa39d432a9b472845876bfaf0 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1c50de89a5a845668696dcb28c3badd6 in term 2.
I20260129 02:59:12.933090 20438 raft_consensus.cc:515] T 75be4bd6bd45474795fd472eff2953ac P 197b880fa39d432a9b472845876bfaf0 [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:59:12.936889 20553 leader_election.cc:304] T fcddab6bca684adf8ea032289a00f90e P 1c50de89a5a845668696dcb28c3badd6 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 197b880fa39d432a9b472845876bfaf0, 1c50de89a5a845668696dcb28c3badd6; no voters:
I20260129 02:59:12.938252 21634 raft_consensus.cc:2804] T fcddab6bca684adf8ea032289a00f90e P 1c50de89a5a845668696dcb28c3badd6 [term 2 FOLLOWER]: Leader election won for term 2
I20260129 02:59:12.939029 21634 raft_consensus.cc:697] T fcddab6bca684adf8ea032289a00f90e P 1c50de89a5a845668696dcb28c3badd6 [term 2 LEADER]: Becoming Leader. State: Replica: 1c50de89a5a845668696dcb28c3badd6, State: Running, Role: LEADER
I20260129 02:59:12.940223 21634 consensus_queue.cc:237] T fcddab6bca684adf8ea032289a00f90e P 1c50de89a5a845668696dcb28c3badd6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:59:12.959916 20438 leader_election.cc:290] T 75be4bd6bd45474795fd472eff2953ac P 197b880fa39d432a9b472845876bfaf0 [CANDIDATE]: Term 4 election: Requested vote from peers 974b802141bd4f13beffef2aa03f88f9 (127.18.211.2:34517), 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3:37735)
I20260129 02:59:12.975682 21669 raft_consensus.cc:993] T a1a39dcd328e4268ae41b988c66b4a75 P 197b880fa39d432a9b472845876bfaf0: : Instructing follower 1c50de89a5a845668696dcb28c3badd6 to start an election
I20260129 02:59:12.993124 20511 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "75be4bd6bd45474795fd472eff2953ac" candidate_uuid: "197b880fa39d432a9b472845876bfaf0" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: true dest_uuid: "974b802141bd4f13beffef2aa03f88f9"
I20260129 02:59:12.992022 21665 raft_consensus.cc:2955] T c934d31efd3f4e5195605ddc354d14e1 P 197b880fa39d432a9b472845876bfaf0 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 7a9daf5d42a84615a6383cfec9a10608 (127.18.211.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } attrs { replace: true } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } } }
I20260129 02:59:12.993710 20511 raft_consensus.cc:3055] T 75be4bd6bd45474795fd472eff2953ac P 974b802141bd4f13beffef2aa03f88f9 [term 3 LEADER]: Stepping down as leader of term 3
I20260129 02:59:12.994017 20511 raft_consensus.cc:740] T 75be4bd6bd45474795fd472eff2953ac P 974b802141bd4f13beffef2aa03f88f9 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 974b802141bd4f13beffef2aa03f88f9, State: Running, Role: LEADER
I20260129 02:59:12.994684 20511 consensus_queue.cc:260] T 75be4bd6bd45474795fd472eff2953ac P 974b802141bd4f13beffef2aa03f88f9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:59:12.995711 20511 raft_consensus.cc:3060] T 75be4bd6bd45474795fd472eff2953ac P 974b802141bd4f13beffef2aa03f88f9 [term 3 FOLLOWER]: Advancing to term 4
I20260129 02:59:12.998342 20510 raft_consensus.cc:2955] T c934d31efd3f4e5195605ddc354d14e1 P 974b802141bd4f13beffef2aa03f88f9 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 7a9daf5d42a84615a6383cfec9a10608 (127.18.211.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } attrs { replace: true } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } } }
I20260129 02:59:12.989308 21624 raft_consensus.cc:1081] T a1a39dcd328e4268ae41b988c66b4a75 P 197b880fa39d432a9b472845876bfaf0 [term 1 LEADER]: Signalling peer 1c50de89a5a845668696dcb28c3badd6 to start an election
I20260129 02:59:13.002655 20511 raft_consensus.cc:2468] T 75be4bd6bd45474795fd472eff2953ac P 974b802141bd4f13beffef2aa03f88f9 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 197b880fa39d432a9b472845876bfaf0 in term 4.
I20260129 02:59:13.003805 20407 leader_election.cc:304] T 75be4bd6bd45474795fd472eff2953ac P 197b880fa39d432a9b472845876bfaf0 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 197b880fa39d432a9b472845876bfaf0, 974b802141bd4f13beffef2aa03f88f9; no voters:
I20260129 02:59:13.005913 21662 raft_consensus.cc:2804] T 75be4bd6bd45474795fd472eff2953ac P 197b880fa39d432a9b472845876bfaf0 [term 4 FOLLOWER]: Leader election won for term 4
I20260129 02:59:13.006400 21662 raft_consensus.cc:697] T 75be4bd6bd45474795fd472eff2953ac P 197b880fa39d432a9b472845876bfaf0 [term 4 LEADER]: Becoming Leader. State: Replica: 197b880fa39d432a9b472845876bfaf0, State: Running, Role: LEADER
I20260129 02:59:13.006542 21668 raft_consensus.cc:2955] T 505580c93c694bbb8ea96fdf9b17ab46 P 1c50de89a5a845668696dcb28c3badd6 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 7a9daf5d42a84615a6383cfec9a10608 (127.18.211.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } attrs { replace: true } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } } }
I20260129 02:59:13.007256 21662 consensus_queue.cc:237] T 75be4bd6bd45474795fd472eff2953ac P 197b880fa39d432a9b472845876bfaf0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:59:13.013782 20435 raft_consensus.cc:2955] T 505580c93c694bbb8ea96fdf9b17ab46 P 197b880fa39d432a9b472845876bfaf0 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 7a9daf5d42a84615a6383cfec9a10608 (127.18.211.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } attrs { replace: true } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } } }
I20260129 02:59:13.018188 20590 tablet_service.cc:2038] Received Run Leader Election RPC: tablet_id: "c891d2bff4ce4568861bbc992e91fbb0"
dest_uuid: "1c50de89a5a845668696dcb28c3badd6"
from {username='slave'} at 127.0.0.1:35754
I20260129 02:59:13.018761 20590 raft_consensus.cc:493] T c891d2bff4ce4568861bbc992e91fbb0 P 1c50de89a5a845668696dcb28c3badd6 [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20260129 02:59:13.019069 20590 raft_consensus.cc:3060] T c891d2bff4ce4568861bbc992e91fbb0 P 1c50de89a5a845668696dcb28c3badd6 [term 3 FOLLOWER]: Advancing to term 4
I20260129 02:59:13.024761 20582 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "75be4bd6bd45474795fd472eff2953ac" candidate_uuid: "197b880fa39d432a9b472845876bfaf0" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: true dest_uuid: "1c50de89a5a845668696dcb28c3badd6"
I20260129 02:59:13.025305 20582 raft_consensus.cc:3060] T 75be4bd6bd45474795fd472eff2953ac P 1c50de89a5a845668696dcb28c3badd6 [term 3 FOLLOWER]: Advancing to term 4
I20260129 02:59:13.025386 20590 raft_consensus.cc:515] T c891d2bff4ce4568861bbc992e91fbb0 P 1c50de89a5a845668696dcb28c3badd6 [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:59:13.025794 20581 raft_consensus.cc:2955] T c934d31efd3f4e5195605ddc354d14e1 P 1c50de89a5a845668696dcb28c3badd6 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 7a9daf5d42a84615a6383cfec9a10608 (127.18.211.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } attrs { replace: true } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } } }
I20260129 02:59:13.027769 20440 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c891d2bff4ce4568861bbc992e91fbb0" candidate_uuid: "1c50de89a5a845668696dcb28c3badd6" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: true dest_uuid: "197b880fa39d432a9b472845876bfaf0"
I20260129 02:59:13.028323 20440 raft_consensus.cc:3055] T c891d2bff4ce4568861bbc992e91fbb0 P 197b880fa39d432a9b472845876bfaf0 [term 3 LEADER]: Stepping down as leader of term 3
I20260129 02:59:13.028597 20440 raft_consensus.cc:740] T c891d2bff4ce4568861bbc992e91fbb0 P 197b880fa39d432a9b472845876bfaf0 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 197b880fa39d432a9b472845876bfaf0, State: Running, Role: LEADER
I20260129 02:59:13.021638 21135 raft_consensus.cc:2955] T c934d31efd3f4e5195605ddc354d14e1 P 7a9daf5d42a84615a6383cfec9a10608 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 7a9daf5d42a84615a6383cfec9a10608 (127.18.211.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } attrs { replace: true } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } } }
I20260129 02:59:13.029249 20440 consensus_queue.cc:260] T c891d2bff4ce4568861bbc992e91fbb0 P 197b880fa39d432a9b472845876bfaf0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:59:13.029975 20587 tablet_service.cc:2038] Received Run Leader Election RPC: tablet_id: "a1a39dcd328e4268ae41b988c66b4a75"
dest_uuid: "1c50de89a5a845668696dcb28c3badd6"
from {username='slave'} at 127.0.0.1:35754
I20260129 02:59:13.030252 20440 raft_consensus.cc:3060] T c891d2bff4ce4568861bbc992e91fbb0 P 197b880fa39d432a9b472845876bfaf0 [term 3 FOLLOWER]: Advancing to term 4
I20260129 02:59:13.030529 20587 raft_consensus.cc:493] T a1a39dcd328e4268ae41b988c66b4a75 P 1c50de89a5a845668696dcb28c3badd6 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260129 02:59:13.030915 20587 raft_consensus.cc:3060] T a1a39dcd328e4268ae41b988c66b4a75 P 1c50de89a5a845668696dcb28c3badd6 [term 1 FOLLOWER]: Advancing to term 2
I20260129 02:59:13.037971 20587 raft_consensus.cc:515] T a1a39dcd328e4268ae41b988c66b4a75 P 1c50de89a5a845668696dcb28c3badd6 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } } peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } }
I20260129 02:59:13.039572 20587 leader_election.cc:290] T a1a39dcd328e4268ae41b988c66b4a75 P 1c50de89a5a845668696dcb28c3badd6 [CANDIDATE]: Term 2 election: Requested vote from peers 974b802141bd4f13beffef2aa03f88f9 (127.18.211.2:34517), 197b880fa39d432a9b472845876bfaf0 (127.18.211.1:42027)
I20260129 02:59:13.040982 20438 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a1a39dcd328e4268ae41b988c66b4a75" candidate_uuid: "1c50de89a5a845668696dcb28c3badd6" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "197b880fa39d432a9b472845876bfaf0"
I20260129 02:59:13.041539 20438 raft_consensus.cc:3055] T a1a39dcd328e4268ae41b988c66b4a75 P 197b880fa39d432a9b472845876bfaf0 [term 1 LEADER]: Stepping down as leader of term 1
I20260129 02:59:13.041831 20438 raft_consensus.cc:740] T a1a39dcd328e4268ae41b988c66b4a75 P 197b880fa39d432a9b472845876bfaf0 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 197b880fa39d432a9b472845876bfaf0, State: Running, Role: LEADER
I20260129 02:59:13.042805 20438 consensus_queue.cc:260] T a1a39dcd328e4268ae41b988c66b4a75 P 197b880fa39d432a9b472845876bfaf0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } } peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } }
I20260129 02:59:13.044153 20438 raft_consensus.cc:3060] T a1a39dcd328e4268ae41b988c66b4a75 P 197b880fa39d432a9b472845876bfaf0 [term 1 FOLLOWER]: Advancing to term 2
I20260129 02:59:13.045440 20582 raft_consensus.cc:2468] T 75be4bd6bd45474795fd472eff2953ac P 1c50de89a5a845668696dcb28c3badd6 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 197b880fa39d432a9b472845876bfaf0 in term 4.
I20260129 02:59:13.049811 20590 leader_election.cc:290] T c891d2bff4ce4568861bbc992e91fbb0 P 1c50de89a5a845668696dcb28c3badd6 [CANDIDATE]: Term 4 election: Requested vote from peers 197b880fa39d432a9b472845876bfaf0 (127.18.211.1:42027), 974b802141bd4f13beffef2aa03f88f9 (127.18.211.2:34517)
I20260129 02:59:13.051335 20513 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a1a39dcd328e4268ae41b988c66b4a75" candidate_uuid: "1c50de89a5a845668696dcb28c3badd6" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "974b802141bd4f13beffef2aa03f88f9"
I20260129 02:59:13.058938 20513 raft_consensus.cc:3060] T a1a39dcd328e4268ae41b988c66b4a75 P 974b802141bd4f13beffef2aa03f88f9 [term 1 FOLLOWER]: Advancing to term 2
I20260129 02:59:13.065780 20513 raft_consensus.cc:2468] T a1a39dcd328e4268ae41b988c66b4a75 P 974b802141bd4f13beffef2aa03f88f9 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1c50de89a5a845668696dcb28c3badd6 in term 2.
I20260129 02:59:13.069789 20438 raft_consensus.cc:2468] T a1a39dcd328e4268ae41b988c66b4a75 P 197b880fa39d432a9b472845876bfaf0 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1c50de89a5a845668696dcb28c3badd6 in term 2.
I20260129 02:59:13.058346 20440 raft_consensus.cc:2468] T c891d2bff4ce4568861bbc992e91fbb0 P 197b880fa39d432a9b472845876bfaf0 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1c50de89a5a845668696dcb28c3badd6 in term 4.
I20260129 02:59:13.070544 20556 leader_election.cc:304] T a1a39dcd328e4268ae41b988c66b4a75 P 1c50de89a5a845668696dcb28c3badd6 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1c50de89a5a845668696dcb28c3badd6, 974b802141bd4f13beffef2aa03f88f9; no voters:
I20260129 02:59:13.071364 21599 raft_consensus.cc:2804] T a1a39dcd328e4268ae41b988c66b4a75 P 1c50de89a5a845668696dcb28c3badd6 [term 2 FOLLOWER]: Leader election won for term 2
I20260129 02:59:13.071854 21599 raft_consensus.cc:697] T a1a39dcd328e4268ae41b988c66b4a75 P 1c50de89a5a845668696dcb28c3badd6 [term 2 LEADER]: Becoming Leader. State: Replica: 1c50de89a5a845668696dcb28c3badd6, State: Running, Role: LEADER
I20260129 02:59:13.072849 21599 consensus_queue.cc:237] T a1a39dcd328e4268ae41b988c66b4a75 P 1c50de89a5a845668696dcb28c3badd6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } } peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } }
I20260129 02:59:13.075635 20508 raft_consensus.cc:1275] T dc2f4cb752cf4d59b5f6bc689f5d8104 P 974b802141bd4f13beffef2aa03f88f9 [term 1 FOLLOWER]: Refusing update from remote peer 1c50de89a5a845668696dcb28c3badd6: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260129 02:59:13.081233 20553 leader_election.cc:304] T c891d2bff4ce4568861bbc992e91fbb0 P 1c50de89a5a845668696dcb28c3badd6 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 197b880fa39d432a9b472845876bfaf0, 1c50de89a5a845668696dcb28c3badd6; no voters:
I20260129 02:59:13.082222 21636 raft_consensus.cc:2804] T c891d2bff4ce4568861bbc992e91fbb0 P 1c50de89a5a845668696dcb28c3badd6 [term 4 FOLLOWER]: Leader election won for term 4
I20260129 02:59:13.082748 21636 raft_consensus.cc:697] T c891d2bff4ce4568861bbc992e91fbb0 P 1c50de89a5a845668696dcb28c3badd6 [term 4 LEADER]: Becoming Leader. State: Replica: 1c50de89a5a845668696dcb28c3badd6, State: Running, Role: LEADER
I20260129 02:59:13.082983 21673 consensus_queue.cc:1048] T dc2f4cb752cf4d59b5f6bc689f5d8104 P 1c50de89a5a845668696dcb28c3badd6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260129 02:59:13.083689 21636 consensus_queue.cc:237] T c891d2bff4ce4568861bbc992e91fbb0 P 1c50de89a5a845668696dcb28c3badd6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:59:13.075780 20337 catalog_manager.cc:5654] T a609ac02fae84725ab743f95f51d33ce P 7a9daf5d42a84615a6383cfec9a10608 reported cstate change: term changed from 3 to 4, leader changed from 197b880fa39d432a9b472845876bfaf0 (127.18.211.1) to 7a9daf5d42a84615a6383cfec9a10608 (127.18.211.4). New cstate: current_term: 4 leader_uuid: "7a9daf5d42a84615a6383cfec9a10608" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260129 02:59:13.044216 20511 raft_consensus.cc:2955] T 505580c93c694bbb8ea96fdf9b17ab46 P 974b802141bd4f13beffef2aa03f88f9 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 7a9daf5d42a84615a6383cfec9a10608 (127.18.211.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } attrs { replace: true } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } } }
I20260129 02:59:13.102480 20440 raft_consensus.cc:1275] T dc2f4cb752cf4d59b5f6bc689f5d8104 P 197b880fa39d432a9b472845876bfaf0 [term 1 FOLLOWER]: Refusing update from remote peer 1c50de89a5a845668696dcb28c3badd6: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260129 02:59:13.104936 20510 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c891d2bff4ce4568861bbc992e91fbb0" candidate_uuid: "1c50de89a5a845668696dcb28c3badd6" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: true dest_uuid: "974b802141bd4f13beffef2aa03f88f9"
I20260129 02:59:13.105096 21135 raft_consensus.cc:1275] T dc2f4cb752cf4d59b5f6bc689f5d8104 P 7a9daf5d42a84615a6383cfec9a10608 [term 1 LEARNER]: Refusing update from remote peer 1c50de89a5a845668696dcb28c3badd6: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260129 02:59:13.105553 20510 raft_consensus.cc:3060] T c891d2bff4ce4568861bbc992e91fbb0 P 974b802141bd4f13beffef2aa03f88f9 [term 3 FOLLOWER]: Advancing to term 4
I20260129 02:59:13.103922 20337 catalog_manager.cc:5654] T 4e3805ffa70242e8ade66da260dddf66 P 7a9daf5d42a84615a6383cfec9a10608 reported cstate change: term changed from 1 to 2, leader changed from 197b880fa39d432a9b472845876bfaf0 (127.18.211.1) to 7a9daf5d42a84615a6383cfec9a10608 (127.18.211.4). New cstate: current_term: 2 leader_uuid: "7a9daf5d42a84615a6383cfec9a10608" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260129 02:59:13.108300 21634 consensus_queue.cc:1048] T dc2f4cb752cf4d59b5f6bc689f5d8104 P 1c50de89a5a845668696dcb28c3badd6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260129 02:59:13.110579 21135 raft_consensus.cc:2955] T 505580c93c694bbb8ea96fdf9b17ab46 P 7a9daf5d42a84615a6383cfec9a10608 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 7a9daf5d42a84615a6383cfec9a10608 (127.18.211.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } attrs { replace: true } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } } }
I20260129 02:59:13.115893 20337 catalog_manager.cc:5654] T c934d31efd3f4e5195605ddc354d14e1 P 7a9daf5d42a84615a6383cfec9a10608 reported cstate change: config changed from index 2 to 3, 7a9daf5d42a84615a6383cfec9a10608 (127.18.211.4) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "197b880fa39d432a9b472845876bfaf0" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } attrs { replace: true } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } } }
I20260129 02:59:13.149158 21594 consensus_queue.cc:1048] T dc2f4cb752cf4d59b5f6bc689f5d8104 P 1c50de89a5a845668696dcb28c3badd6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260129 02:59:13.153622 20587 raft_consensus.cc:1275] T b189f08e16a74a11980cc55d99359a0a P 1c50de89a5a845668696dcb28c3badd6 [term 2 FOLLOWER]: Refusing update from remote peer 197b880fa39d432a9b472845876bfaf0: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260129 02:59:13.155311 21661 consensus_queue.cc:1048] T b189f08e16a74a11980cc55d99359a0a P 197b880fa39d432a9b472845876bfaf0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260129 02:59:13.166729 20510 raft_consensus.cc:2468] T c891d2bff4ce4568861bbc992e91fbb0 P 974b802141bd4f13beffef2aa03f88f9 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1c50de89a5a845668696dcb28c3badd6 in term 4.
I20260129 02:59:13.183264 20510 raft_consensus.cc:1275] T b189f08e16a74a11980cc55d99359a0a P 974b802141bd4f13beffef2aa03f88f9 [term 2 FOLLOWER]: Refusing update from remote peer 197b880fa39d432a9b472845876bfaf0: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260129 02:59:13.193996 21665 consensus_queue.cc:1048] T b189f08e16a74a11980cc55d99359a0a P 197b880fa39d432a9b472845876bfaf0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.005s
I20260129 02:59:13.204242 20338 catalog_manager.cc:5654] T fcddab6bca684adf8ea032289a00f90e P 1c50de89a5a845668696dcb28c3badd6 reported cstate change: term changed from 1 to 2, leader changed from 197b880fa39d432a9b472845876bfaf0 (127.18.211.1) to 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3). New cstate: current_term: 2 leader_uuid: "1c50de89a5a845668696dcb28c3badd6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } health_report { overall_health: HEALTHY } } }
I20260129 02:59:13.229178 20336 catalog_manager.cc:5654] T 75be4bd6bd45474795fd472eff2953ac P 197b880fa39d432a9b472845876bfaf0 reported cstate change: term changed from 3 to 4, leader changed from 974b802141bd4f13beffef2aa03f88f9 (127.18.211.2) to 197b880fa39d432a9b472845876bfaf0 (127.18.211.1). New cstate: current_term: 4 leader_uuid: "197b880fa39d432a9b472845876bfaf0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } health_report { overall_health: UNKNOWN } } }
W20260129 02:59:13.232643 20611 debug-util.cc:398] Leaking SignalData structure 0x7b08001cdaa0 after lost signal to thread 19279
I20260129 02:59:13.236979 20336 catalog_manager.cc:5654] T 505580c93c694bbb8ea96fdf9b17ab46 P 197b880fa39d432a9b472845876bfaf0 reported cstate change: config changed from index 2 to 3, 7a9daf5d42a84615a6383cfec9a10608 (127.18.211.4) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "1c50de89a5a845668696dcb28c3badd6" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } attrs { replace: true } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } } }
W20260129 02:59:13.238680 20611 debug-util.cc:398] Leaking SignalData structure 0x7b0800016480 after lost signal to thread 20372
I20260129 02:59:13.238534 20336 catalog_manager.cc:5654] T b189f08e16a74a11980cc55d99359a0a P 197b880fa39d432a9b472845876bfaf0 reported cstate change: term changed from 1 to 2, leader changed from 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3) to 197b880fa39d432a9b472845876bfaf0 (127.18.211.1). New cstate: current_term: 2 leader_uuid: "197b880fa39d432a9b472845876bfaf0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } health_report { overall_health: UNKNOWN } } }
W20260129 02:59:13.240460 20611 debug-util.cc:398] Leaking SignalData structure 0x7b08002ef0e0 after lost signal to thread 20465
W20260129 02:59:13.241976 20611 debug-util.cc:398] Leaking SignalData structure 0x7b0800347540 after lost signal to thread 20540
W20260129 02:59:13.247702 20611 debug-util.cc:398] Leaking SignalData structure 0x7b0800148060 after lost signal to thread 20614
W20260129 02:59:13.248871 20611 debug-util.cc:398] Leaking SignalData structure 0x7b08000b6a40 after lost signal to thread 21160
I20260129 02:59:13.249830 20587 consensus_queue.cc:237] T 45cc5f3d2688486d9b6dbe7e666dedde P 1c50de89a5a845668696dcb28c3badd6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } }
I20260129 02:59:13.258334 21131 raft_consensus.cc:1275] T 45cc5f3d2688486d9b6dbe7e666dedde P 7a9daf5d42a84615a6383cfec9a10608 [term 1 FOLLOWER]: Refusing update from remote peer 1c50de89a5a845668696dcb28c3badd6: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260129 02:59:13.259560 21668 consensus_queue.cc:1048] T 45cc5f3d2688486d9b6dbe7e666dedde P 1c50de89a5a845668696dcb28c3badd6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260129 02:59:13.262414 20438 consensus_queue.cc:237] T c934d31efd3f4e5195605ddc354d14e1 P 197b880fa39d432a9b472845876bfaf0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } }
I20260129 02:59:13.263186 20510 raft_consensus.cc:1275] T 45cc5f3d2688486d9b6dbe7e666dedde P 974b802141bd4f13beffef2aa03f88f9 [term 1 FOLLOWER]: Refusing update from remote peer 1c50de89a5a845668696dcb28c3badd6: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260129 02:59:13.264952 20435 raft_consensus.cc:1275] T a609ac02fae84725ab743f95f51d33ce P 197b880fa39d432a9b472845876bfaf0 [term 4 FOLLOWER]: Refusing update from remote peer 7a9daf5d42a84615a6383cfec9a10608: Log matching property violated. Preceding OpId in replica: term: 3 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20260129 02:59:13.270148 21655 consensus_queue.cc:1048] T a609ac02fae84725ab743f95f51d33ce P 7a9daf5d42a84615a6383cfec9a10608 [LEADER]: Connected to new peer: Peer: permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260129 02:59:13.269615 21594 raft_consensus.cc:2955] T 45cc5f3d2688486d9b6dbe7e666dedde P 1c50de89a5a845668696dcb28c3badd6 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 197b880fa39d432a9b472845876bfaf0 (127.18.211.1) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } } }
I20260129 02:59:13.271468 21673 consensus_queue.cc:1048] T 45cc5f3d2688486d9b6dbe7e666dedde P 1c50de89a5a845668696dcb28c3badd6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260129 02:59:13.274377 21131 raft_consensus.cc:2955] T 45cc5f3d2688486d9b6dbe7e666dedde P 7a9daf5d42a84615a6383cfec9a10608 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 197b880fa39d432a9b472845876bfaf0 (127.18.211.1) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } } }
I20260129 02:59:13.278561 21673 raft_consensus.cc:2955] T dc2f4cb752cf4d59b5f6bc689f5d8104 P 1c50de89a5a845668696dcb28c3badd6 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 7a9daf5d42a84615a6383cfec9a10608 (127.18.211.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } attrs { replace: true } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } } }
I20260129 02:59:13.282694 21135 raft_consensus.cc:1275] T c934d31efd3f4e5195605ddc354d14e1 P 7a9daf5d42a84615a6383cfec9a10608 [term 1 FOLLOWER]: Refusing update from remote peer 197b880fa39d432a9b472845876bfaf0: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260129 02:59:13.286002 20510 raft_consensus.cc:1275] T c934d31efd3f4e5195605ddc354d14e1 P 974b802141bd4f13beffef2aa03f88f9 [term 1 FOLLOWER]: Refusing update from remote peer 197b880fa39d432a9b472845876bfaf0: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260129 02:59:13.284538 20508 raft_consensus.cc:2955] T 45cc5f3d2688486d9b6dbe7e666dedde P 974b802141bd4f13beffef2aa03f88f9 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 197b880fa39d432a9b472845876bfaf0 (127.18.211.1) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } } }
I20260129 02:59:13.289510 21665 consensus_queue.cc:1048] T c934d31efd3f4e5195605ddc354d14e1 P 197b880fa39d432a9b472845876bfaf0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.003s
I20260129 02:59:13.293397 21135 raft_consensus.cc:2955] T dc2f4cb752cf4d59b5f6bc689f5d8104 P 7a9daf5d42a84615a6383cfec9a10608 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 7a9daf5d42a84615a6383cfec9a10608 (127.18.211.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } attrs { replace: true } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } } }
I20260129 02:59:13.300490 21665 consensus_queue.cc:1048] T c934d31efd3f4e5195605ddc354d14e1 P 197b880fa39d432a9b472845876bfaf0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260129 02:59:13.311977 21666 raft_consensus.cc:2955] T c934d31efd3f4e5195605ddc354d14e1 P 197b880fa39d432a9b472845876bfaf0 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } } }
I20260129 02:59:13.317178 20435 raft_consensus.cc:2955] T dc2f4cb752cf4d59b5f6bc689f5d8104 P 197b880fa39d432a9b472845876bfaf0 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 7a9daf5d42a84615a6383cfec9a10608 (127.18.211.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } attrs { replace: true } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } } }
I20260129 02:59:13.330907 21134 raft_consensus.cc:2955] T c934d31efd3f4e5195605ddc354d14e1 P 7a9daf5d42a84615a6383cfec9a10608 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } } }
I20260129 02:59:13.329358 20510 raft_consensus.cc:2955] T c934d31efd3f4e5195605ddc354d14e1 P 974b802141bd4f13beffef2aa03f88f9 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } } }
I20260129 02:59:13.299867 20337 catalog_manager.cc:5654] T c891d2bff4ce4568861bbc992e91fbb0 P 1c50de89a5a845668696dcb28c3badd6 reported cstate change: term changed from 3 to 4, leader changed from 197b880fa39d432a9b472845876bfaf0 (127.18.211.1) to 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3). New cstate: current_term: 4 leader_uuid: "1c50de89a5a845668696dcb28c3badd6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } health_report { overall_health: HEALTHY } } }
I20260129 02:59:13.345165 20337 catalog_manager.cc:5654] T a1a39dcd328e4268ae41b988c66b4a75 P 1c50de89a5a845668696dcb28c3badd6 reported cstate change: term changed from 1 to 2, leader changed from 197b880fa39d432a9b472845876bfaf0 (127.18.211.1) to 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3). New cstate: current_term: 2 leader_uuid: "1c50de89a5a845668696dcb28c3badd6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } health_report { overall_health: UNKNOWN } } }
I20260129 02:59:13.355738 20322 catalog_manager.cc:5167] ChangeConfig:REMOVE_PEER RPC for tablet c934d31efd3f4e5195605ddc354d14e1 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260129 02:59:13.362329 20511 raft_consensus.cc:2955] T dc2f4cb752cf4d59b5f6bc689f5d8104 P 974b802141bd4f13beffef2aa03f88f9 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 7a9daf5d42a84615a6383cfec9a10608 (127.18.211.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } attrs { replace: true } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } } }
I20260129 02:59:13.365787 20323 catalog_manager.cc:5167] ChangeConfig:REMOVE_PEER RPC for tablet 45cc5f3d2688486d9b6dbe7e666dedde with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260129 02:59:13.365518 20336 catalog_manager.cc:5654] T dc2f4cb752cf4d59b5f6bc689f5d8104 P 7a9daf5d42a84615a6383cfec9a10608 reported cstate change: config changed from index 2 to 3, 7a9daf5d42a84615a6383cfec9a10608 (127.18.211.4) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "1c50de89a5a845668696dcb28c3badd6" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } attrs { replace: true } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } } }
I20260129 02:59:13.388698 20336 catalog_manager.cc:5654] T 45cc5f3d2688486d9b6dbe7e666dedde P 7a9daf5d42a84615a6383cfec9a10608 reported cstate change: config changed from index 3 to 4, VOTER 197b880fa39d432a9b472845876bfaf0 (127.18.211.1) evicted. New cstate: current_term: 1 leader_uuid: "1c50de89a5a845668696dcb28c3badd6" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } } }
I20260129 02:59:13.407284 20336 catalog_manager.cc:5654] T c934d31efd3f4e5195605ddc354d14e1 P 7a9daf5d42a84615a6383cfec9a10608 reported cstate change: config changed from index 3 to 4, VOTER 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3) evicted. New cstate: current_term: 1 leader_uuid: "197b880fa39d432a9b472845876bfaf0" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } } }
I20260129 02:59:13.409044 20440 raft_consensus.cc:1275] T 4e3805ffa70242e8ade66da260dddf66 P 197b880fa39d432a9b472845876bfaf0 [term 2 FOLLOWER]: Refusing update from remote peer 7a9daf5d42a84615a6383cfec9a10608: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260129 02:59:13.410692 20581 consensus_queue.cc:237] T 505580c93c694bbb8ea96fdf9b17ab46 P 1c50de89a5a845668696dcb28c3badd6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } }
I20260129 02:59:13.411022 21626 consensus_queue.cc:1048] T 4e3805ffa70242e8ade66da260dddf66 P 7a9daf5d42a84615a6383cfec9a10608 [LEADER]: Connected to new peer: Peer: permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260129 02:59:13.418843 21131 raft_consensus.cc:1275] T 505580c93c694bbb8ea96fdf9b17ab46 P 7a9daf5d42a84615a6383cfec9a10608 [term 1 FOLLOWER]: Refusing update from remote peer 1c50de89a5a845668696dcb28c3badd6: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260129 02:59:13.422421 21684 consensus_queue.cc:1048] T 505580c93c694bbb8ea96fdf9b17ab46 P 1c50de89a5a845668696dcb28c3badd6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260129 02:59:13.429208 20508 raft_consensus.cc:1275] T 75be4bd6bd45474795fd472eff2953ac P 974b802141bd4f13beffef2aa03f88f9 [term 4 FOLLOWER]: Refusing update from remote peer 197b880fa39d432a9b472845876bfaf0: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20260129 02:59:13.429560 20431 tablet_service.cc:1552] Processing DeleteTablet for tablet 45cc5f3d2688486d9b6dbe7e666dedde with delete_type TABLET_DATA_TOMBSTONED (TS 197b880fa39d432a9b472845876bfaf0 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:55666
I20260129 02:59:13.429162 20511 raft_consensus.cc:1275] T 505580c93c694bbb8ea96fdf9b17ab46 P 974b802141bd4f13beffef2aa03f88f9 [term 1 FOLLOWER]: Refusing update from remote peer 1c50de89a5a845668696dcb28c3badd6: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260129 02:59:13.432215 21599 raft_consensus.cc:2955] T 505580c93c694bbb8ea96fdf9b17ab46 P 1c50de89a5a845668696dcb28c3badd6 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 197b880fa39d432a9b472845876bfaf0 (127.18.211.1) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } } }
I20260129 02:59:13.441231 20579 tablet_service.cc:1552] Processing DeleteTablet for tablet c934d31efd3f4e5195605ddc354d14e1 with delete_type TABLET_DATA_TOMBSTONED (TS 1c50de89a5a845668696dcb28c3badd6 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:35744
I20260129 02:59:13.444882 21134 raft_consensus.cc:2955] T 505580c93c694bbb8ea96fdf9b17ab46 P 7a9daf5d42a84615a6383cfec9a10608 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 197b880fa39d432a9b472845876bfaf0 (127.18.211.1) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } } }
I20260129 02:59:13.448045 21694 tablet_replica.cc:333] T 45cc5f3d2688486d9b6dbe7e666dedde P 197b880fa39d432a9b472845876bfaf0: stopping tablet replica
I20260129 02:59:13.448120 21662 consensus_queue.cc:1048] T 75be4bd6bd45474795fd472eff2953ac P 197b880fa39d432a9b472845876bfaf0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260129 02:59:13.449849 21696 tablet_replica.cc:333] T c934d31efd3f4e5195605ddc354d14e1 P 1c50de89a5a845668696dcb28c3badd6: stopping tablet replica
I20260129 02:59:13.451193 21636 consensus_queue.cc:1048] T 505580c93c694bbb8ea96fdf9b17ab46 P 1c50de89a5a845668696dcb28c3badd6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260129 02:59:13.455468 20435 raft_consensus.cc:1275] T fcddab6bca684adf8ea032289a00f90e P 197b880fa39d432a9b472845876bfaf0 [term 2 FOLLOWER]: Refusing update from remote peer 1c50de89a5a845668696dcb28c3badd6: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260129 02:59:13.470084 21636 consensus_queue.cc:1048] T fcddab6bca684adf8ea032289a00f90e P 1c50de89a5a845668696dcb28c3badd6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260129 02:59:13.451265 21696 raft_consensus.cc:2243] T c934d31efd3f4e5195605ddc354d14e1 P 1c50de89a5a845668696dcb28c3badd6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260129 02:59:13.471616 21696 raft_consensus.cc:2272] T c934d31efd3f4e5195605ddc354d14e1 P 1c50de89a5a845668696dcb28c3badd6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260129 02:59:13.474418 20511 raft_consensus.cc:2955] T 505580c93c694bbb8ea96fdf9b17ab46 P 974b802141bd4f13beffef2aa03f88f9 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 197b880fa39d432a9b472845876bfaf0 (127.18.211.1) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } } }
I20260129 02:59:13.487598 20508 raft_consensus.cc:1275] T a609ac02fae84725ab743f95f51d33ce P 974b802141bd4f13beffef2aa03f88f9 [term 4 FOLLOWER]: Refusing update from remote peer 7a9daf5d42a84615a6383cfec9a10608: Log matching property violated. Preceding OpId in replica: term: 3 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20260129 02:59:13.496781 21626 consensus_queue.cc:1048] T a609ac02fae84725ab743f95f51d33ce P 7a9daf5d42a84615a6383cfec9a10608 [LEADER]: Connected to new peer: Peer: permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260129 02:59:13.500797 21696 ts_tablet_manager.cc:1916] T c934d31efd3f4e5195605ddc354d14e1 P 1c50de89a5a845668696dcb28c3badd6: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260129 02:59:13.502491 20582 raft_consensus.cc:1275] T 4e3805ffa70242e8ade66da260dddf66 P 1c50de89a5a845668696dcb28c3badd6 [term 2 FOLLOWER]: Refusing update from remote peer 7a9daf5d42a84615a6383cfec9a10608: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260129 02:59:13.509956 20587 raft_consensus.cc:1275] T 75be4bd6bd45474795fd472eff2953ac P 1c50de89a5a845668696dcb28c3badd6 [term 4 FOLLOWER]: Refusing update from remote peer 197b880fa39d432a9b472845876bfaf0: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20260129 02:59:13.509438 19276 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260129 02:59:13.517659 21655 consensus_queue.cc:1048] T 4e3805ffa70242e8ade66da260dddf66 P 7a9daf5d42a84615a6383cfec9a10608 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260129 02:59:13.534866 20323 catalog_manager.cc:5167] ChangeConfig:REMOVE_PEER RPC for tablet 505580c93c694bbb8ea96fdf9b17ab46 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260129 02:59:13.535218 21663 consensus_queue.cc:1048] T 75be4bd6bd45474795fd472eff2953ac P 197b880fa39d432a9b472845876bfaf0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.010s
I20260129 02:59:13.554827 20513 raft_consensus.cc:1275] T fcddab6bca684adf8ea032289a00f90e P 974b802141bd4f13beffef2aa03f88f9 [term 2 FOLLOWER]: Refusing update from remote peer 1c50de89a5a845668696dcb28c3badd6: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260129 02:59:13.556770 21694 raft_consensus.cc:2243] T 45cc5f3d2688486d9b6dbe7e666dedde P 197b880fa39d432a9b472845876bfaf0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260129 02:59:13.560855 21684 consensus_queue.cc:1048] T fcddab6bca684adf8ea032289a00f90e P 1c50de89a5a845668696dcb28c3badd6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260129 02:59:13.562665 21694 raft_consensus.cc:2272] T 45cc5f3d2688486d9b6dbe7e666dedde P 197b880fa39d432a9b472845876bfaf0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260129 02:59:13.580255 20513 raft_consensus.cc:1275] T c891d2bff4ce4568861bbc992e91fbb0 P 974b802141bd4f13beffef2aa03f88f9 [term 4 FOLLOWER]: Refusing update from remote peer 1c50de89a5a845668696dcb28c3badd6: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20260129 02:59:13.582973 21694 ts_tablet_manager.cc:1916] T 45cc5f3d2688486d9b6dbe7e666dedde P 197b880fa39d432a9b472845876bfaf0: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260129 02:59:13.603168 21668 consensus_queue.cc:1048] T c891d2bff4ce4568861bbc992e91fbb0 P 1c50de89a5a845668696dcb28c3badd6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260129 02:59:13.606832 20513 tablet_service.cc:2066] Received LeaderStepDown RPC: tablet_id: "1299723a4476420ab02fd8bfd7aa0645"
dest_uuid: "974b802141bd4f13beffef2aa03f88f9"
mode: GRACEFUL
new_leader_uuid: "197b880fa39d432a9b472845876bfaf0"
from {username='slave'} at 127.0.0.1:52008
I20260129 02:59:13.607687 20513 raft_consensus.cc:606] T 1299723a4476420ab02fd8bfd7aa0645 P 974b802141bd4f13beffef2aa03f88f9 [term 1 LEADER]: Received request to transfer leadership to TS 197b880fa39d432a9b472845876bfaf0
I20260129 02:59:13.608142 20338 catalog_manager.cc:5654] T 505580c93c694bbb8ea96fdf9b17ab46 P 7a9daf5d42a84615a6383cfec9a10608 reported cstate change: config changed from index 3 to 4, VOTER 197b880fa39d432a9b472845876bfaf0 (127.18.211.1) evicted. New cstate: current_term: 1 leader_uuid: "1c50de89a5a845668696dcb28c3badd6" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } } }
I20260129 02:59:13.609084 20438 raft_consensus.cc:1275] T a1a39dcd328e4268ae41b988c66b4a75 P 197b880fa39d432a9b472845876bfaf0 [term 2 FOLLOWER]: Refusing update from remote peer 1c50de89a5a845668696dcb28c3badd6: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260129 02:59:13.608144 20435 raft_consensus.cc:1275] T c891d2bff4ce4568861bbc992e91fbb0 P 197b880fa39d432a9b472845876bfaf0 [term 4 FOLLOWER]: Refusing update from remote peer 1c50de89a5a845668696dcb28c3badd6: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20260129 02:59:13.610809 21634 consensus_queue.cc:1048] T a1a39dcd328e4268ae41b988c66b4a75 P 1c50de89a5a845668696dcb28c3badd6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260129 02:59:13.617339 21634 consensus_queue.cc:1048] T c891d2bff4ce4568861bbc992e91fbb0 P 1c50de89a5a845668696dcb28c3badd6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260129 02:59:13.633916 20438 tablet_service.cc:2066] Received LeaderStepDown RPC: tablet_id: "18f30ebaa4174fd28c958e72d0cc7e24"
dest_uuid: "197b880fa39d432a9b472845876bfaf0"
mode: GRACEFUL
new_leader_uuid: "7a9daf5d42a84615a6383cfec9a10608"
from {username='slave'} at 127.0.0.1:57776
I20260129 02:59:13.634505 20438 raft_consensus.cc:606] T 18f30ebaa4174fd28c958e72d0cc7e24 P 197b880fa39d432a9b472845876bfaf0 [term 1 LEADER]: Received request to transfer leadership to TS 7a9daf5d42a84615a6383cfec9a10608
I20260129 02:59:13.641371 20581 consensus_queue.cc:237] T dc2f4cb752cf4d59b5f6bc689f5d8104 P 1c50de89a5a845668696dcb28c3badd6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } }
I20260129 02:59:13.642333 21513 raft_consensus.cc:993] T 1299723a4476420ab02fd8bfd7aa0645 P 974b802141bd4f13beffef2aa03f88f9: : Instructing follower 197b880fa39d432a9b472845876bfaf0 to start an election
I20260129 02:59:13.642973 21592 raft_consensus.cc:1081] T 1299723a4476420ab02fd8bfd7aa0645 P 974b802141bd4f13beffef2aa03f88f9 [term 1 LEADER]: Signalling peer 197b880fa39d432a9b472845876bfaf0 to start an election
I20260129 02:59:13.645896 20438 tablet_service.cc:2038] Received Run Leader Election RPC: tablet_id: "1299723a4476420ab02fd8bfd7aa0645"
dest_uuid: "197b880fa39d432a9b472845876bfaf0"
from {username='slave'} at 127.0.0.1:55672
I20260129 02:59:13.646427 20438 raft_consensus.cc:493] T 1299723a4476420ab02fd8bfd7aa0645 P 197b880fa39d432a9b472845876bfaf0 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260129 02:59:13.646751 20438 raft_consensus.cc:3060] T 1299723a4476420ab02fd8bfd7aa0645 P 197b880fa39d432a9b472845876bfaf0 [term 1 FOLLOWER]: Advancing to term 2
I20260129 02:59:13.655645 20437 tablet_service.cc:2066] Received LeaderStepDown RPC: tablet_id: "2e58e9a5a8aa4cecafb464addb4f35ed"
dest_uuid: "197b880fa39d432a9b472845876bfaf0"
mode: GRACEFUL
new_leader_uuid: "7a9daf5d42a84615a6383cfec9a10608"
from {username='slave'} at 127.0.0.1:57776
I20260129 02:59:13.656270 20437 raft_consensus.cc:606] T 2e58e9a5a8aa4cecafb464addb4f35ed P 197b880fa39d432a9b472845876bfaf0 [term 1 LEADER]: Received request to transfer leadership to TS 7a9daf5d42a84615a6383cfec9a10608
I20260129 02:59:13.671663 20434 tablet_service.cc:2066] Received LeaderStepDown RPC: tablet_id: "42d27a28d89b409e8807ec0efd97af69"
dest_uuid: "197b880fa39d432a9b472845876bfaf0"
mode: GRACEFUL
new_leader_uuid: "7a9daf5d42a84615a6383cfec9a10608"
from {username='slave'} at 127.0.0.1:57776
I20260129 02:59:13.672338 20434 raft_consensus.cc:606] T 42d27a28d89b409e8807ec0efd97af69 P 197b880fa39d432a9b472845876bfaf0 [term 1 LEADER]: Received request to transfer leadership to TS 7a9daf5d42a84615a6383cfec9a10608
I20260129 02:59:13.673383 20431 tablet_service.cc:1552] Processing DeleteTablet for tablet 505580c93c694bbb8ea96fdf9b17ab46 with delete_type TABLET_DATA_TOMBSTONED (TS 197b880fa39d432a9b472845876bfaf0 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:55666
I20260129 02:59:13.674834 20434 tablet_service.cc:2066] Received LeaderStepDown RPC: tablet_id: "c934d31efd3f4e5195605ddc354d14e1"
dest_uuid: "197b880fa39d432a9b472845876bfaf0"
mode: GRACEFUL
new_leader_uuid: "7a9daf5d42a84615a6383cfec9a10608"
from {username='slave'} at 127.0.0.1:57776
I20260129 02:59:13.675391 20434 raft_consensus.cc:606] T c934d31efd3f4e5195605ddc354d14e1 P 197b880fa39d432a9b472845876bfaf0 [term 1 LEADER]: Received request to transfer leadership to TS 7a9daf5d42a84615a6383cfec9a10608
I20260129 02:59:13.679260 21131 raft_consensus.cc:1275] T dc2f4cb752cf4d59b5f6bc689f5d8104 P 7a9daf5d42a84615a6383cfec9a10608 [term 1 FOLLOWER]: Refusing update from remote peer 1c50de89a5a845668696dcb28c3badd6: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260129 02:59:13.686388 20438 raft_consensus.cc:515] T 1299723a4476420ab02fd8bfd7aa0645 P 197b880fa39d432a9b472845876bfaf0 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:59:13.692878 20508 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1299723a4476420ab02fd8bfd7aa0645" candidate_uuid: "197b880fa39d432a9b472845876bfaf0" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "974b802141bd4f13beffef2aa03f88f9"
I20260129 02:59:13.693462 20508 raft_consensus.cc:3055] T 1299723a4476420ab02fd8bfd7aa0645 P 974b802141bd4f13beffef2aa03f88f9 [term 1 LEADER]: Stepping down as leader of term 1
I20260129 02:59:13.693754 20508 raft_consensus.cc:740] T 1299723a4476420ab02fd8bfd7aa0645 P 974b802141bd4f13beffef2aa03f88f9 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 974b802141bd4f13beffef2aa03f88f9, State: Running, Role: LEADER
I20260129 02:59:13.694449 20508 consensus_queue.cc:260] T 1299723a4476420ab02fd8bfd7aa0645 P 974b802141bd4f13beffef2aa03f88f9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:59:13.695497 20508 raft_consensus.cc:3060] T 1299723a4476420ab02fd8bfd7aa0645 P 974b802141bd4f13beffef2aa03f88f9 [term 1 FOLLOWER]: Advancing to term 2
I20260129 02:59:13.697692 21673 consensus_queue.cc:1048] T dc2f4cb752cf4d59b5f6bc689f5d8104 P 1c50de89a5a845668696dcb28c3badd6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260129 02:59:13.702577 21696 ts_tablet_manager.cc:1929] T c934d31efd3f4e5195605ddc354d14e1 P 1c50de89a5a845668696dcb28c3badd6: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20260129 02:59:13.703158 21696 log.cc:1199] T c934d31efd3f4e5195605ddc354d14e1 P 1c50de89a5a845668696dcb28c3badd6: Deleting WAL directory at /tmp/dist-test-taskrTP_Jo/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1769655460890924-19276-0/minicluster-data/ts-2-root/wals/c934d31efd3f4e5195605ddc354d14e1
W20260129 02:59:13.706681 20556 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.211.2:34517, user_credentials={real_user=slave}} blocked reactor thread for 57671.8us
I20260129 02:59:13.709863 20510 raft_consensus.cc:1275] T a1a39dcd328e4268ae41b988c66b4a75 P 974b802141bd4f13beffef2aa03f88f9 [term 2 FOLLOWER]: Refusing update from remote peer 1c50de89a5a845668696dcb28c3badd6: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260129 02:59:13.713641 21694 ts_tablet_manager.cc:1929] T 45cc5f3d2688486d9b6dbe7e666dedde P 197b880fa39d432a9b472845876bfaf0: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20260129 02:59:13.714144 21694 log.cc:1199] T 45cc5f3d2688486d9b6dbe7e666dedde P 197b880fa39d432a9b472845876bfaf0: Deleting WAL directory at /tmp/dist-test-taskrTP_Jo/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1769655460890924-19276-0/minicluster-data/ts-0-root/wals/45cc5f3d2688486d9b6dbe7e666dedde
I20260129 02:59:13.714059 20323 catalog_manager.cc:4985] TS 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3:37735): tablet c934d31efd3f4e5195605ddc354d14e1 (table test-workload [id=de73611ac8e3485a98aecc5add247032]) successfully deleted
I20260129 02:59:13.714779 21684 raft_consensus.cc:2955] T dc2f4cb752cf4d59b5f6bc689f5d8104 P 1c50de89a5a845668696dcb28c3badd6 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 974b802141bd4f13beffef2aa03f88f9 (127.18.211.2) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } } }
I20260129 02:59:13.718101 20590 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1299723a4476420ab02fd8bfd7aa0645" candidate_uuid: "197b880fa39d432a9b472845876bfaf0" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "1c50de89a5a845668696dcb28c3badd6"
I20260129 02:59:13.718700 20590 raft_consensus.cc:3060] T 1299723a4476420ab02fd8bfd7aa0645 P 1c50de89a5a845668696dcb28c3badd6 [term 1 FOLLOWER]: Advancing to term 2
I20260129 02:59:13.719745 20438 leader_election.cc:290] T 1299723a4476420ab02fd8bfd7aa0645 P 197b880fa39d432a9b472845876bfaf0 [CANDIDATE]: Term 2 election: Requested vote from peers 974b802141bd4f13beffef2aa03f88f9 (127.18.211.2:34517), 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3:37735)
I20260129 02:59:13.706842 20508 raft_consensus.cc:2468] T 1299723a4476420ab02fd8bfd7aa0645 P 974b802141bd4f13beffef2aa03f88f9 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 197b880fa39d432a9b472845876bfaf0 in term 2.
I20260129 02:59:13.722496 20407 leader_election.cc:304] T 1299723a4476420ab02fd8bfd7aa0645 P 197b880fa39d432a9b472845876bfaf0 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 197b880fa39d432a9b472845876bfaf0, 974b802141bd4f13beffef2aa03f88f9; no voters:
I20260129 02:59:13.723292 21624 raft_consensus.cc:2804] T 1299723a4476420ab02fd8bfd7aa0645 P 197b880fa39d432a9b472845876bfaf0 [term 2 FOLLOWER]: Leader election won for term 2
I20260129 02:59:13.723784 21624 raft_consensus.cc:697] T 1299723a4476420ab02fd8bfd7aa0645 P 197b880fa39d432a9b472845876bfaf0 [term 2 LEADER]: Becoming Leader. State: Replica: 197b880fa39d432a9b472845876bfaf0, State: Running, Role: LEADER
I20260129 02:59:13.724644 21624 consensus_queue.cc:237] T 1299723a4476420ab02fd8bfd7aa0645 P 197b880fa39d432a9b472845876bfaf0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:59:13.724439 21131 raft_consensus.cc:2955] T dc2f4cb752cf4d59b5f6bc689f5d8104 P 7a9daf5d42a84615a6383cfec9a10608 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 974b802141bd4f13beffef2aa03f88f9 (127.18.211.2) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } } }
I20260129 02:59:13.725386 20590 raft_consensus.cc:2468] T 1299723a4476420ab02fd8bfd7aa0645 P 1c50de89a5a845668696dcb28c3badd6 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 197b880fa39d432a9b472845876bfaf0 in term 2.
I20260129 02:59:13.739390 21694 tablet_replica.cc:333] T 505580c93c694bbb8ea96fdf9b17ab46 P 197b880fa39d432a9b472845876bfaf0: stopping tablet replica
I20260129 02:59:13.740125 20322 catalog_manager.cc:4985] TS 197b880fa39d432a9b472845876bfaf0 (127.18.211.1:42027): tablet 45cc5f3d2688486d9b6dbe7e666dedde (table test-workload [id=de73611ac8e3485a98aecc5add247032]) successfully deleted
I20260129 02:59:13.740473 21694 raft_consensus.cc:2243] T 505580c93c694bbb8ea96fdf9b17ab46 P 197b880fa39d432a9b472845876bfaf0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260129 02:59:13.741384 21694 raft_consensus.cc:2272] T 505580c93c694bbb8ea96fdf9b17ab46 P 197b880fa39d432a9b472845876bfaf0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260129 02:59:13.756613 20590 tablet_service.cc:2066] Received LeaderStepDown RPC: tablet_id: "fcddab6bca684adf8ea032289a00f90e"
dest_uuid: "1c50de89a5a845668696dcb28c3badd6"
mode: GRACEFUL
new_leader_uuid: "197b880fa39d432a9b472845876bfaf0"
from {username='slave'} at 127.0.0.1:54860
I20260129 02:59:13.759687 20435 raft_consensus.cc:1275] T dc2f4cb752cf4d59b5f6bc689f5d8104 P 197b880fa39d432a9b472845876bfaf0 [term 1 FOLLOWER]: Refusing update from remote peer 1c50de89a5a845668696dcb28c3badd6: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260129 02:59:13.771668 21694 ts_tablet_manager.cc:1916] T 505580c93c694bbb8ea96fdf9b17ab46 P 197b880fa39d432a9b472845876bfaf0: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260129 02:59:13.774607 21708 consensus_queue.cc:1048] T dc2f4cb752cf4d59b5f6bc689f5d8104 P 1c50de89a5a845668696dcb28c3badd6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.003s
I20260129 02:59:13.782265 20338 catalog_manager.cc:5654] T dc2f4cb752cf4d59b5f6bc689f5d8104 P 7a9daf5d42a84615a6383cfec9a10608 reported cstate change: config changed from index 3 to 4, VOTER 974b802141bd4f13beffef2aa03f88f9 (127.18.211.2) evicted. New cstate: current_term: 1 leader_uuid: "1c50de89a5a845668696dcb28c3badd6" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } } }
I20260129 02:59:13.783730 21673 consensus_queue.cc:1048] T a1a39dcd328e4268ae41b988c66b4a75 P 1c50de89a5a845668696dcb28c3badd6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260129 02:59:13.786578 20435 raft_consensus.cc:2955] T dc2f4cb752cf4d59b5f6bc689f5d8104 P 197b880fa39d432a9b472845876bfaf0 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 974b802141bd4f13beffef2aa03f88f9 (127.18.211.2) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } } }
I20260129 02:59:13.790493 20323 catalog_manager.cc:5167] ChangeConfig:REMOVE_PEER RPC for tablet dc2f4cb752cf4d59b5f6bc689f5d8104 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260129 02:59:13.822237 21662 raft_consensus.cc:993] T 42d27a28d89b409e8807ec0efd97af69 P 197b880fa39d432a9b472845876bfaf0: : Instructing follower 7a9daf5d42a84615a6383cfec9a10608 to start an election
I20260129 02:59:13.822928 21662 raft_consensus.cc:1081] T 42d27a28d89b409e8807ec0efd97af69 P 197b880fa39d432a9b472845876bfaf0 [term 1 LEADER]: Signalling peer 7a9daf5d42a84615a6383cfec9a10608 to start an election
I20260129 02:59:13.825733 21131 tablet_service.cc:2038] Received Run Leader Election RPC: tablet_id: "42d27a28d89b409e8807ec0efd97af69"
dest_uuid: "7a9daf5d42a84615a6383cfec9a10608"
from {username='slave'} at 127.0.0.1:34060
I20260129 02:59:13.826419 21131 raft_consensus.cc:493] T 42d27a28d89b409e8807ec0efd97af69 P 7a9daf5d42a84615a6383cfec9a10608 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260129 02:59:13.826848 21131 raft_consensus.cc:3060] T 42d27a28d89b409e8807ec0efd97af69 P 7a9daf5d42a84615a6383cfec9a10608 [term 1 FOLLOWER]: Advancing to term 2
I20260129 02:59:13.828924 20500 tablet_service.cc:1552] Processing DeleteTablet for tablet dc2f4cb752cf4d59b5f6bc689f5d8104 with delete_type TABLET_DATA_TOMBSTONED (TS 974b802141bd4f13beffef2aa03f88f9 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:57580
I20260129 02:59:13.839166 20590 raft_consensus.cc:606] T fcddab6bca684adf8ea032289a00f90e P 1c50de89a5a845668696dcb28c3badd6 [term 2 LEADER]: Received request to transfer leadership to TS 197b880fa39d432a9b472845876bfaf0
I20260129 02:59:13.881534 21131 raft_consensus.cc:515] T 42d27a28d89b409e8807ec0efd97af69 P 7a9daf5d42a84615a6383cfec9a10608 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } }
I20260129 02:59:13.891475 19276 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 6
I20260129 02:59:13.893286 19276 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20260129 02:59:13.894052 19276 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260129 02:59:13.898718 21716 tablet_replica.cc:333] T dc2f4cb752cf4d59b5f6bc689f5d8104 P 974b802141bd4f13beffef2aa03f88f9: stopping tablet replica
I20260129 02:59:13.900231 21716 raft_consensus.cc:2243] T dc2f4cb752cf4d59b5f6bc689f5d8104 P 974b802141bd4f13beffef2aa03f88f9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260129 02:59:13.905809 21716 raft_consensus.cc:2272] T dc2f4cb752cf4d59b5f6bc689f5d8104 P 974b802141bd4f13beffef2aa03f88f9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260129 02:59:13.922688 20437 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "42d27a28d89b409e8807ec0efd97af69" candidate_uuid: "7a9daf5d42a84615a6383cfec9a10608" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "197b880fa39d432a9b472845876bfaf0"
I20260129 02:59:13.923528 20437 raft_consensus.cc:3055] T 42d27a28d89b409e8807ec0efd97af69 P 197b880fa39d432a9b472845876bfaf0 [term 1 LEADER]: Stepping down as leader of term 1
I20260129 02:59:13.923985 20437 raft_consensus.cc:740] T 42d27a28d89b409e8807ec0efd97af69 P 197b880fa39d432a9b472845876bfaf0 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 197b880fa39d432a9b472845876bfaf0, State: Running, Role: LEADER
I20260129 02:59:13.924755 20437 consensus_queue.cc:260] T 42d27a28d89b409e8807ec0efd97af69 P 197b880fa39d432a9b472845876bfaf0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } }
I20260129 02:59:13.926019 20437 raft_consensus.cc:3060] T 42d27a28d89b409e8807ec0efd97af69 P 197b880fa39d432a9b472845876bfaf0 [term 1 FOLLOWER]: Advancing to term 2
I20260129 02:59:13.928627 21131 leader_election.cc:290] T 42d27a28d89b409e8807ec0efd97af69 P 7a9daf5d42a84615a6383cfec9a10608 [CANDIDATE]: Term 2 election: Requested vote from peers 197b880fa39d432a9b472845876bfaf0 (127.18.211.1:42027), 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3:37735)
I20260129 02:59:13.932894 20582 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "42d27a28d89b409e8807ec0efd97af69" candidate_uuid: "7a9daf5d42a84615a6383cfec9a10608" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "1c50de89a5a845668696dcb28c3badd6"
I20260129 02:59:13.936175 21662 raft_consensus.cc:993] T 18f30ebaa4174fd28c958e72d0cc7e24 P 197b880fa39d432a9b472845876bfaf0: : Instructing follower 7a9daf5d42a84615a6383cfec9a10608 to start an election
I20260129 02:59:13.936713 21662 raft_consensus.cc:1081] T 18f30ebaa4174fd28c958e72d0cc7e24 P 197b880fa39d432a9b472845876bfaf0 [term 1 LEADER]: Signalling peer 7a9daf5d42a84615a6383cfec9a10608 to start an election
I20260129 02:59:13.939666 21624 raft_consensus.cc:993] T c934d31efd3f4e5195605ddc354d14e1 P 197b880fa39d432a9b472845876bfaf0: : Instructing follower 7a9daf5d42a84615a6383cfec9a10608 to start an election
I20260129 02:59:13.940150 21624 raft_consensus.cc:1081] T c934d31efd3f4e5195605ddc354d14e1 P 197b880fa39d432a9b472845876bfaf0 [term 1 LEADER]: Signalling peer 7a9daf5d42a84615a6383cfec9a10608 to start an election
I20260129 02:59:13.936720 20338 catalog_manager.cc:5654] T 1299723a4476420ab02fd8bfd7aa0645 P 197b880fa39d432a9b472845876bfaf0 reported cstate change: term changed from 1 to 2, leader changed from 974b802141bd4f13beffef2aa03f88f9 (127.18.211.2) to 197b880fa39d432a9b472845876bfaf0 (127.18.211.1). New cstate: current_term: 2 leader_uuid: "197b880fa39d432a9b472845876bfaf0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } health_report { overall_health: UNKNOWN } } }
I20260129 02:59:13.943686 21716 ts_tablet_manager.cc:1916] T dc2f4cb752cf4d59b5f6bc689f5d8104 P 974b802141bd4f13beffef2aa03f88f9: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260129 02:59:13.954725 21134 tablet_service.cc:2038] Received Run Leader Election RPC: tablet_id: "18f30ebaa4174fd28c958e72d0cc7e24"
dest_uuid: "7a9daf5d42a84615a6383cfec9a10608"
from {username='slave'} at 127.0.0.1:34060
I20260129 02:59:13.955355 21134 raft_consensus.cc:493] T 18f30ebaa4174fd28c958e72d0cc7e24 P 7a9daf5d42a84615a6383cfec9a10608 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260129 02:59:13.955421 20437 raft_consensus.cc:2468] T 42d27a28d89b409e8807ec0efd97af69 P 197b880fa39d432a9b472845876bfaf0 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7a9daf5d42a84615a6383cfec9a10608 in term 2.
I20260129 02:59:13.955790 21134 raft_consensus.cc:3060] T 18f30ebaa4174fd28c958e72d0cc7e24 P 7a9daf5d42a84615a6383cfec9a10608 [term 1 FOLLOWER]: Advancing to term 2
I20260129 02:59:13.956768 21099 leader_election.cc:304] T 42d27a28d89b409e8807ec0efd97af69 P 7a9daf5d42a84615a6383cfec9a10608 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 197b880fa39d432a9b472845876bfaf0, 7a9daf5d42a84615a6383cfec9a10608; no voters:
I20260129 02:59:13.957711 21700 raft_consensus.cc:2804] T 42d27a28d89b409e8807ec0efd97af69 P 7a9daf5d42a84615a6383cfec9a10608 [term 2 FOLLOWER]: Leader election won for term 2
I20260129 02:59:13.958164 21700 raft_consensus.cc:697] T 42d27a28d89b409e8807ec0efd97af69 P 7a9daf5d42a84615a6383cfec9a10608 [term 2 LEADER]: Becoming Leader. State: Replica: 7a9daf5d42a84615a6383cfec9a10608, State: Running, Role: LEADER
I20260129 02:59:13.959020 21700 consensus_queue.cc:237] T 42d27a28d89b409e8807ec0efd97af69 P 7a9daf5d42a84615a6383cfec9a10608 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } }
I20260129 02:59:13.966984 21131 tablet_service.cc:2038] Received Run Leader Election RPC: tablet_id: "c934d31efd3f4e5195605ddc354d14e1"
dest_uuid: "7a9daf5d42a84615a6383cfec9a10608"
from {username='slave'} at 127.0.0.1:34060
I20260129 02:59:13.967571 21131 raft_consensus.cc:493] T c934d31efd3f4e5195605ddc354d14e1 P 7a9daf5d42a84615a6383cfec9a10608 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260129 02:59:13.967900 21716 ts_tablet_manager.cc:1929] T dc2f4cb752cf4d59b5f6bc689f5d8104 P 974b802141bd4f13beffef2aa03f88f9: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20260129 02:59:13.967968 21131 raft_consensus.cc:3060] T c934d31efd3f4e5195605ddc354d14e1 P 7a9daf5d42a84615a6383cfec9a10608 [term 1 FOLLOWER]: Advancing to term 2
I20260129 02:59:13.968446 21134 raft_consensus.cc:515] T 18f30ebaa4174fd28c958e72d0cc7e24 P 7a9daf5d42a84615a6383cfec9a10608 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } }
I20260129 02:59:13.970963 20437 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "18f30ebaa4174fd28c958e72d0cc7e24" candidate_uuid: "7a9daf5d42a84615a6383cfec9a10608" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "197b880fa39d432a9b472845876bfaf0"
I20260129 02:59:13.971498 20437 raft_consensus.cc:3055] T 18f30ebaa4174fd28c958e72d0cc7e24 P 197b880fa39d432a9b472845876bfaf0 [term 1 LEADER]: Stepping down as leader of term 1
I20260129 02:59:13.971769 20437 raft_consensus.cc:740] T 18f30ebaa4174fd28c958e72d0cc7e24 P 197b880fa39d432a9b472845876bfaf0 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 197b880fa39d432a9b472845876bfaf0, State: Running, Role: LEADER
I20260129 02:59:13.972488 20437 consensus_queue.cc:260] T 18f30ebaa4174fd28c958e72d0cc7e24 P 197b880fa39d432a9b472845876bfaf0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } }
I20260129 02:59:13.973549 20437 raft_consensus.cc:3060] T 18f30ebaa4174fd28c958e72d0cc7e24 P 197b880fa39d432a9b472845876bfaf0 [term 1 FOLLOWER]: Advancing to term 2
W20260129 02:59:13.973486 20553 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.211.1:42027, user_credentials={real_user=slave}} blocked reactor thread for 66806.9us
I20260129 02:59:13.968451 21716 log.cc:1199] T dc2f4cb752cf4d59b5f6bc689f5d8104 P 974b802141bd4f13beffef2aa03f88f9: Deleting WAL directory at /tmp/dist-test-taskrTP_Jo/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1769655460890924-19276-0/minicluster-data/ts-1-root/wals/dc2f4cb752cf4d59b5f6bc689f5d8104
I20260129 02:59:13.979213 20335 catalog_manager.cc:5654] T 42d27a28d89b409e8807ec0efd97af69 P 7a9daf5d42a84615a6383cfec9a10608 reported cstate change: term changed from 1 to 2, leader changed from 197b880fa39d432a9b472845876bfaf0 (127.18.211.1) to 7a9daf5d42a84615a6383cfec9a10608 (127.18.211.4). New cstate: current_term: 2 leader_uuid: "7a9daf5d42a84615a6383cfec9a10608" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260129 02:59:13.981240 20581 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "18f30ebaa4174fd28c958e72d0cc7e24" candidate_uuid: "7a9daf5d42a84615a6383cfec9a10608" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "1c50de89a5a845668696dcb28c3badd6"
I20260129 02:59:13.981863 20581 raft_consensus.cc:3060] T 18f30ebaa4174fd28c958e72d0cc7e24 P 1c50de89a5a845668696dcb28c3badd6 [term 1 FOLLOWER]: Advancing to term 2
I20260129 02:59:13.983013 21134 leader_election.cc:290] T 18f30ebaa4174fd28c958e72d0cc7e24 P 7a9daf5d42a84615a6383cfec9a10608 [CANDIDATE]: Term 2 election: Requested vote from peers 197b880fa39d432a9b472845876bfaf0 (127.18.211.1:42027), 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3:37735)
I20260129 02:59:13.943317 20582 raft_consensus.cc:3060] T 42d27a28d89b409e8807ec0efd97af69 P 1c50de89a5a845668696dcb28c3badd6 [term 1 FOLLOWER]: Advancing to term 2
I20260129 02:59:13.988511 20581 raft_consensus.cc:2468] T 18f30ebaa4174fd28c958e72d0cc7e24 P 1c50de89a5a845668696dcb28c3badd6 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7a9daf5d42a84615a6383cfec9a10608 in term 2.
I20260129 02:59:13.989809 21100 leader_election.cc:304] T 18f30ebaa4174fd28c958e72d0cc7e24 P 7a9daf5d42a84615a6383cfec9a10608 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1c50de89a5a845668696dcb28c3badd6, 7a9daf5d42a84615a6383cfec9a10608; no voters:
I20260129 02:59:13.990649 21700 raft_consensus.cc:2804] T 18f30ebaa4174fd28c958e72d0cc7e24 P 7a9daf5d42a84615a6383cfec9a10608 [term 2 FOLLOWER]: Leader election won for term 2
I20260129 02:59:13.991358 21700 raft_consensus.cc:697] T 18f30ebaa4174fd28c958e72d0cc7e24 P 7a9daf5d42a84615a6383cfec9a10608 [term 2 LEADER]: Becoming Leader. State: Replica: 7a9daf5d42a84615a6383cfec9a10608, State: Running, Role: LEADER
I20260129 02:59:13.991792 20325 catalog_manager.cc:4985] TS 974b802141bd4f13beffef2aa03f88f9 (127.18.211.2:34517): tablet dc2f4cb752cf4d59b5f6bc689f5d8104 (table test-workload [id=de73611ac8e3485a98aecc5add247032]) successfully deleted
I20260129 02:59:13.992239 21700 consensus_queue.cc:237] T 18f30ebaa4174fd28c958e72d0cc7e24 P 7a9daf5d42a84615a6383cfec9a10608 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } }
I20260129 02:59:13.996338 20437 raft_consensus.cc:2468] T 18f30ebaa4174fd28c958e72d0cc7e24 P 197b880fa39d432a9b472845876bfaf0 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7a9daf5d42a84615a6383cfec9a10608 in term 2.
I20260129 02:59:14.003101 21131 raft_consensus.cc:515] T c934d31efd3f4e5195605ddc354d14e1 P 7a9daf5d42a84615a6383cfec9a10608 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } }
I20260129 02:59:14.006299 20437 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c934d31efd3f4e5195605ddc354d14e1" candidate_uuid: "7a9daf5d42a84615a6383cfec9a10608" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "197b880fa39d432a9b472845876bfaf0"
I20260129 02:59:14.006551 20511 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c934d31efd3f4e5195605ddc354d14e1" candidate_uuid: "7a9daf5d42a84615a6383cfec9a10608" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "974b802141bd4f13beffef2aa03f88f9"
I20260129 02:59:14.007109 20437 raft_consensus.cc:3055] T c934d31efd3f4e5195605ddc354d14e1 P 197b880fa39d432a9b472845876bfaf0 [term 1 LEADER]: Stepping down as leader of term 1
I20260129 02:59:14.007647 20437 raft_consensus.cc:740] T c934d31efd3f4e5195605ddc354d14e1 P 197b880fa39d432a9b472845876bfaf0 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 197b880fa39d432a9b472845876bfaf0, State: Running, Role: LEADER
I20260129 02:59:14.007155 20511 raft_consensus.cc:3060] T c934d31efd3f4e5195605ddc354d14e1 P 974b802141bd4f13beffef2aa03f88f9 [term 1 FOLLOWER]: Advancing to term 2
I20260129 02:59:14.010805 20437 consensus_queue.cc:260] T c934d31efd3f4e5195605ddc354d14e1 P 197b880fa39d432a9b472845876bfaf0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } }
I20260129 02:59:14.016644 20437 raft_consensus.cc:3060] T c934d31efd3f4e5195605ddc354d14e1 P 197b880fa39d432a9b472845876bfaf0 [term 1 FOLLOWER]: Advancing to term 2
I20260129 02:59:14.034996 21708 raft_consensus.cc:993] T fcddab6bca684adf8ea032289a00f90e P 1c50de89a5a845668696dcb28c3badd6: : Instructing follower 197b880fa39d432a9b472845876bfaf0 to start an election
I20260129 02:59:14.035554 21708 raft_consensus.cc:1081] T fcddab6bca684adf8ea032289a00f90e P 1c50de89a5a845668696dcb28c3badd6 [term 2 LEADER]: Signalling peer 197b880fa39d432a9b472845876bfaf0 to start an election
I20260129 02:59:14.042822 21663 raft_consensus.cc:993] T 2e58e9a5a8aa4cecafb464addb4f35ed P 197b880fa39d432a9b472845876bfaf0: : Instructing follower 7a9daf5d42a84615a6383cfec9a10608 to start an election
I20260129 02:59:14.043310 21663 raft_consensus.cc:1081] T 2e58e9a5a8aa4cecafb464addb4f35ed P 197b880fa39d432a9b472845876bfaf0 [term 1 LEADER]: Signalling peer 7a9daf5d42a84615a6383cfec9a10608 to start an election
I20260129 02:59:14.044929 21134 tablet_service.cc:2038] Received Run Leader Election RPC: tablet_id: "2e58e9a5a8aa4cecafb464addb4f35ed"
dest_uuid: "7a9daf5d42a84615a6383cfec9a10608"
from {username='slave'} at 127.0.0.1:34060
I20260129 02:59:14.045464 21134 raft_consensus.cc:493] T 2e58e9a5a8aa4cecafb464addb4f35ed P 7a9daf5d42a84615a6383cfec9a10608 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260129 02:59:14.045799 21134 raft_consensus.cc:3060] T 2e58e9a5a8aa4cecafb464addb4f35ed P 7a9daf5d42a84615a6383cfec9a10608 [term 1 FOLLOWER]: Advancing to term 2
I20260129 02:59:14.051579 21134 raft_consensus.cc:515] T 2e58e9a5a8aa4cecafb464addb4f35ed P 7a9daf5d42a84615a6383cfec9a10608 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } }
I20260129 02:59:14.053485 21134 leader_election.cc:290] T 2e58e9a5a8aa4cecafb464addb4f35ed P 7a9daf5d42a84615a6383cfec9a10608 [CANDIDATE]: Term 2 election: Requested vote from peers 197b880fa39d432a9b472845876bfaf0 (127.18.211.1:42027), 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3:37735)
I20260129 02:59:14.054255 20440 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2e58e9a5a8aa4cecafb464addb4f35ed" candidate_uuid: "7a9daf5d42a84615a6383cfec9a10608" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "197b880fa39d432a9b472845876bfaf0"
I20260129 02:59:14.054843 20440 raft_consensus.cc:3055] T 2e58e9a5a8aa4cecafb464addb4f35ed P 197b880fa39d432a9b472845876bfaf0 [term 1 LEADER]: Stepping down as leader of term 1
I20260129 02:59:14.054857 20581 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2e58e9a5a8aa4cecafb464addb4f35ed" candidate_uuid: "7a9daf5d42a84615a6383cfec9a10608" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "1c50de89a5a845668696dcb28c3badd6"
I20260129 02:59:14.055263 20440 raft_consensus.cc:740] T 2e58e9a5a8aa4cecafb464addb4f35ed P 197b880fa39d432a9b472845876bfaf0 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 197b880fa39d432a9b472845876bfaf0, State: Running, Role: LEADER
I20260129 02:59:14.055567 20581 raft_consensus.cc:3060] T 2e58e9a5a8aa4cecafb464addb4f35ed P 1c50de89a5a845668696dcb28c3badd6 [term 1 FOLLOWER]: Advancing to term 2
I20260129 02:59:14.056032 20440 consensus_queue.cc:260] T 2e58e9a5a8aa4cecafb464addb4f35ed P 197b880fa39d432a9b472845876bfaf0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } }
I20260129 02:59:14.058676 20440 raft_consensus.cc:3060] T 2e58e9a5a8aa4cecafb464addb4f35ed P 197b880fa39d432a9b472845876bfaf0 [term 1 FOLLOWER]: Advancing to term 2
I20260129 02:59:14.062191 20581 raft_consensus.cc:2468] T 2e58e9a5a8aa4cecafb464addb4f35ed P 1c50de89a5a845668696dcb28c3badd6 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7a9daf5d42a84615a6383cfec9a10608 in term 2.
I20260129 02:59:14.062779 21131 leader_election.cc:290] T c934d31efd3f4e5195605ddc354d14e1 P 7a9daf5d42a84615a6383cfec9a10608 [CANDIDATE]: Term 2 election: Requested vote from peers 197b880fa39d432a9b472845876bfaf0 (127.18.211.1:42027), 974b802141bd4f13beffef2aa03f88f9 (127.18.211.2:34517)
I20260129 02:59:14.069046 20582 raft_consensus.cc:2468] T 42d27a28d89b409e8807ec0efd97af69 P 1c50de89a5a845668696dcb28c3badd6 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7a9daf5d42a84615a6383cfec9a10608 in term 2.
I20260129 02:59:14.069092 20511 raft_consensus.cc:2468] T c934d31efd3f4e5195605ddc354d14e1 P 974b802141bd4f13beffef2aa03f88f9 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7a9daf5d42a84615a6383cfec9a10608 in term 2.
I20260129 02:59:14.070840 21102 leader_election.cc:304] T c934d31efd3f4e5195605ddc354d14e1 P 7a9daf5d42a84615a6383cfec9a10608 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7a9daf5d42a84615a6383cfec9a10608, 974b802141bd4f13beffef2aa03f88f9; no voters:
I20260129 02:59:14.071784 21700 raft_consensus.cc:2804] T c934d31efd3f4e5195605ddc354d14e1 P 7a9daf5d42a84615a6383cfec9a10608 [term 2 FOLLOWER]: Leader election won for term 2
I20260129 02:59:14.072618 21100 leader_election.cc:304] T 2e58e9a5a8aa4cecafb464addb4f35ed P 7a9daf5d42a84615a6383cfec9a10608 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1c50de89a5a845668696dcb28c3badd6, 7a9daf5d42a84615a6383cfec9a10608; no voters:
I20260129 02:59:14.072767 21700 raft_consensus.cc:697] T c934d31efd3f4e5195605ddc354d14e1 P 7a9daf5d42a84615a6383cfec9a10608 [term 2 LEADER]: Becoming Leader. State: Replica: 7a9daf5d42a84615a6383cfec9a10608, State: Running, Role: LEADER
I20260129 02:59:14.073021 21694 ts_tablet_manager.cc:1929] T 505580c93c694bbb8ea96fdf9b17ab46 P 197b880fa39d432a9b472845876bfaf0: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20260129 02:59:14.073467 21694 log.cc:1199] T 505580c93c694bbb8ea96fdf9b17ab46 P 197b880fa39d432a9b472845876bfaf0: Deleting WAL directory at /tmp/dist-test-taskrTP_Jo/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1769655460890924-19276-0/minicluster-data/ts-0-root/wals/505580c93c694bbb8ea96fdf9b17ab46
I20260129 02:59:14.073691 21626 raft_consensus.cc:2804] T 2e58e9a5a8aa4cecafb464addb4f35ed P 7a9daf5d42a84615a6383cfec9a10608 [term 2 FOLLOWER]: Leader election won for term 2
I20260129 02:59:14.074362 21626 raft_consensus.cc:697] T 2e58e9a5a8aa4cecafb464addb4f35ed P 7a9daf5d42a84615a6383cfec9a10608 [term 2 LEADER]: Becoming Leader. State: Replica: 7a9daf5d42a84615a6383cfec9a10608, State: Running, Role: LEADER
I20260129 02:59:14.073737 21700 consensus_queue.cc:237] T c934d31efd3f4e5195605ddc354d14e1 P 7a9daf5d42a84615a6383cfec9a10608 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } }
I20260129 02:59:14.075147 21626 consensus_queue.cc:237] T 2e58e9a5a8aa4cecafb464addb4f35ed P 7a9daf5d42a84615a6383cfec9a10608 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } }
I20260129 02:59:14.080411 20440 raft_consensus.cc:2468] T 2e58e9a5a8aa4cecafb464addb4f35ed P 197b880fa39d432a9b472845876bfaf0 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7a9daf5d42a84615a6383cfec9a10608 in term 2.
I20260129 02:59:14.077054 20438 tablet_service.cc:2038] Received Run Leader Election RPC: tablet_id: "fcddab6bca684adf8ea032289a00f90e"
dest_uuid: "197b880fa39d432a9b472845876bfaf0"
from {username='slave'} at 127.0.0.1:55676
I20260129 02:59:14.081943 20438 raft_consensus.cc:493] T fcddab6bca684adf8ea032289a00f90e P 197b880fa39d432a9b472845876bfaf0 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20260129 02:59:14.082316 20438 raft_consensus.cc:3060] T fcddab6bca684adf8ea032289a00f90e P 197b880fa39d432a9b472845876bfaf0 [term 2 FOLLOWER]: Advancing to term 3
I20260129 02:59:14.086860 20437 raft_consensus.cc:2468] T c934d31efd3f4e5195605ddc354d14e1 P 197b880fa39d432a9b472845876bfaf0 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7a9daf5d42a84615a6383cfec9a10608 in term 2.
I20260129 02:59:14.090610 20438 raft_consensus.cc:515] T fcddab6bca684adf8ea032289a00f90e P 197b880fa39d432a9b472845876bfaf0 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:59:14.094120 20582 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fcddab6bca684adf8ea032289a00f90e" candidate_uuid: "197b880fa39d432a9b472845876bfaf0" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "1c50de89a5a845668696dcb28c3badd6"
I20260129 02:59:14.094704 20582 raft_consensus.cc:3055] T fcddab6bca684adf8ea032289a00f90e P 1c50de89a5a845668696dcb28c3badd6 [term 2 LEADER]: Stepping down as leader of term 2
I20260129 02:59:14.094987 20582 raft_consensus.cc:740] T fcddab6bca684adf8ea032289a00f90e P 1c50de89a5a845668696dcb28c3badd6 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 1c50de89a5a845668696dcb28c3badd6, State: Running, Role: LEADER
I20260129 02:59:14.095745 20582 consensus_queue.cc:260] T fcddab6bca684adf8ea032289a00f90e P 1c50de89a5a845668696dcb28c3badd6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:59:14.095988 20508 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fcddab6bca684adf8ea032289a00f90e" candidate_uuid: "197b880fa39d432a9b472845876bfaf0" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "974b802141bd4f13beffef2aa03f88f9"
I20260129 02:59:14.096632 20508 raft_consensus.cc:3060] T fcddab6bca684adf8ea032289a00f90e P 974b802141bd4f13beffef2aa03f88f9 [term 2 FOLLOWER]: Advancing to term 3
I20260129 02:59:14.084724 20322 catalog_manager.cc:4985] TS 197b880fa39d432a9b472845876bfaf0 (127.18.211.1:42027): tablet 505580c93c694bbb8ea96fdf9b17ab46 (table test-workload [id=de73611ac8e3485a98aecc5add247032]) successfully deleted
I20260129 02:59:14.098583 20336 catalog_manager.cc:5654] T 18f30ebaa4174fd28c958e72d0cc7e24 P 7a9daf5d42a84615a6383cfec9a10608 reported cstate change: term changed from 1 to 2, leader changed from 197b880fa39d432a9b472845876bfaf0 (127.18.211.1) to 7a9daf5d42a84615a6383cfec9a10608 (127.18.211.4). New cstate: current_term: 2 leader_uuid: "7a9daf5d42a84615a6383cfec9a10608" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260129 02:59:14.103391 20508 raft_consensus.cc:2468] T fcddab6bca684adf8ea032289a00f90e P 974b802141bd4f13beffef2aa03f88f9 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 197b880fa39d432a9b472845876bfaf0 in term 3.
I20260129 02:59:14.104414 20438 leader_election.cc:290] T fcddab6bca684adf8ea032289a00f90e P 197b880fa39d432a9b472845876bfaf0 [CANDIDATE]: Term 3 election: Requested vote from peers 974b802141bd4f13beffef2aa03f88f9 (127.18.211.2:34517), 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3:37735)
I20260129 02:59:14.105615 20582 raft_consensus.cc:3060] T fcddab6bca684adf8ea032289a00f90e P 1c50de89a5a845668696dcb28c3badd6 [term 2 FOLLOWER]: Advancing to term 3
I20260129 02:59:14.112509 20407 leader_election.cc:304] T fcddab6bca684adf8ea032289a00f90e P 197b880fa39d432a9b472845876bfaf0 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 197b880fa39d432a9b472845876bfaf0, 974b802141bd4f13beffef2aa03f88f9; no voters:
I20260129 02:59:14.112973 20582 raft_consensus.cc:2468] T fcddab6bca684adf8ea032289a00f90e P 1c50de89a5a845668696dcb28c3badd6 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 197b880fa39d432a9b472845876bfaf0 in term 3.
I20260129 02:59:14.113250 21624 raft_consensus.cc:2804] T fcddab6bca684adf8ea032289a00f90e P 197b880fa39d432a9b472845876bfaf0 [term 3 FOLLOWER]: Leader election won for term 3
I20260129 02:59:14.113806 21624 raft_consensus.cc:697] T fcddab6bca684adf8ea032289a00f90e P 197b880fa39d432a9b472845876bfaf0 [term 3 LEADER]: Becoming Leader. State: Replica: 197b880fa39d432a9b472845876bfaf0, State: Running, Role: LEADER
I20260129 02:59:14.114650 21624 consensus_queue.cc:237] T fcddab6bca684adf8ea032289a00f90e P 197b880fa39d432a9b472845876bfaf0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:59:14.125289 20336 catalog_manager.cc:5654] T c934d31efd3f4e5195605ddc354d14e1 P 7a9daf5d42a84615a6383cfec9a10608 reported cstate change: term changed from 1 to 2, leader changed from 197b880fa39d432a9b472845876bfaf0 (127.18.211.1) to 7a9daf5d42a84615a6383cfec9a10608 (127.18.211.4). New cstate: current_term: 2 leader_uuid: "7a9daf5d42a84615a6383cfec9a10608" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260129 02:59:14.126940 20336 catalog_manager.cc:5654] T 2e58e9a5a8aa4cecafb464addb4f35ed P 7a9daf5d42a84615a6383cfec9a10608 reported cstate change: term changed from 1 to 2, leader changed from 197b880fa39d432a9b472845876bfaf0 (127.18.211.1) to 7a9daf5d42a84615a6383cfec9a10608 (127.18.211.4). New cstate: current_term: 2 leader_uuid: "7a9daf5d42a84615a6383cfec9a10608" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260129 02:59:14.128604 20335 catalog_manager.cc:5654] T fcddab6bca684adf8ea032289a00f90e P 197b880fa39d432a9b472845876bfaf0 reported cstate change: term changed from 2 to 3, leader changed from 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3) to 197b880fa39d432a9b472845876bfaf0 (127.18.211.1). New cstate: current_term: 3 leader_uuid: "197b880fa39d432a9b472845876bfaf0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } health_report { overall_health: UNKNOWN } } }
I20260129 02:59:14.154709 20581 raft_consensus.cc:1275] T 1299723a4476420ab02fd8bfd7aa0645 P 1c50de89a5a845668696dcb28c3badd6 [term 2 FOLLOWER]: Refusing update from remote peer 197b880fa39d432a9b472845876bfaf0: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260129 02:59:14.156193 21624 consensus_queue.cc:1048] T 1299723a4476420ab02fd8bfd7aa0645 P 197b880fa39d432a9b472845876bfaf0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260129 02:59:14.176576 20508 raft_consensus.cc:1275] T 1299723a4476420ab02fd8bfd7aa0645 P 974b802141bd4f13beffef2aa03f88f9 [term 2 FOLLOWER]: Refusing update from remote peer 197b880fa39d432a9b472845876bfaf0: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260129 02:59:14.187817 21663 consensus_queue.cc:1048] T 1299723a4476420ab02fd8bfd7aa0645 P 197b880fa39d432a9b472845876bfaf0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260129 02:59:14.364889 20435 raft_consensus.cc:1275] T 42d27a28d89b409e8807ec0efd97af69 P 197b880fa39d432a9b472845876bfaf0 [term 2 FOLLOWER]: Refusing update from remote peer 7a9daf5d42a84615a6383cfec9a10608: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260129 02:59:14.366209 21655 consensus_queue.cc:1048] T 42d27a28d89b409e8807ec0efd97af69 P 7a9daf5d42a84615a6383cfec9a10608 [LEADER]: Connected to new peer: Peer: permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260129 02:59:14.380721 20581 raft_consensus.cc:1275] T 42d27a28d89b409e8807ec0efd97af69 P 1c50de89a5a845668696dcb28c3badd6 [term 2 FOLLOWER]: Refusing update from remote peer 7a9daf5d42a84615a6383cfec9a10608: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260129 02:59:14.381990 21655 consensus_queue.cc:1048] T 42d27a28d89b409e8807ec0efd97af69 P 7a9daf5d42a84615a6383cfec9a10608 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260129 02:59:14.467720 20435 raft_consensus.cc:1275] T 2e58e9a5a8aa4cecafb464addb4f35ed P 197b880fa39d432a9b472845876bfaf0 [term 2 FOLLOWER]: Refusing update from remote peer 7a9daf5d42a84615a6383cfec9a10608: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260129 02:59:14.469758 21655 consensus_queue.cc:1048] T 2e58e9a5a8aa4cecafb464addb4f35ed P 7a9daf5d42a84615a6383cfec9a10608 [LEADER]: Connected to new peer: Peer: permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260129 02:59:14.492350 20582 raft_consensus.cc:1275] T 2e58e9a5a8aa4cecafb464addb4f35ed P 1c50de89a5a845668696dcb28c3badd6 [term 2 FOLLOWER]: Refusing update from remote peer 7a9daf5d42a84615a6383cfec9a10608: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260129 02:59:14.493856 21700 consensus_queue.cc:1048] T 2e58e9a5a8aa4cecafb464addb4f35ed P 7a9daf5d42a84615a6383cfec9a10608 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260129 02:59:14.502278 20581 raft_consensus.cc:1275] T 18f30ebaa4174fd28c958e72d0cc7e24 P 1c50de89a5a845668696dcb28c3badd6 [term 2 FOLLOWER]: Refusing update from remote peer 7a9daf5d42a84615a6383cfec9a10608: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260129 02:59:14.504894 20581 raft_consensus.cc:1275] T fcddab6bca684adf8ea032289a00f90e P 1c50de89a5a845668696dcb28c3badd6 [term 3 FOLLOWER]: Refusing update from remote peer 197b880fa39d432a9b472845876bfaf0: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260129 02:59:14.506558 21666 consensus_queue.cc:1048] T fcddab6bca684adf8ea032289a00f90e P 197b880fa39d432a9b472845876bfaf0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260129 02:59:14.507453 21700 consensus_queue.cc:1048] T 18f30ebaa4174fd28c958e72d0cc7e24 P 7a9daf5d42a84615a6383cfec9a10608 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260129 02:59:14.519179 20510 raft_consensus.cc:1275] T fcddab6bca684adf8ea032289a00f90e P 974b802141bd4f13beffef2aa03f88f9 [term 3 FOLLOWER]: Refusing update from remote peer 197b880fa39d432a9b472845876bfaf0: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260129 02:59:14.522090 21663 consensus_queue.cc:1048] T fcddab6bca684adf8ea032289a00f90e P 197b880fa39d432a9b472845876bfaf0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260129 02:59:14.546106 20435 raft_consensus.cc:1275] T 18f30ebaa4174fd28c958e72d0cc7e24 P 197b880fa39d432a9b472845876bfaf0 [term 2 FOLLOWER]: Refusing update from remote peer 7a9daf5d42a84615a6383cfec9a10608: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260129 02:59:14.547945 21701 consensus_queue.cc:1048] T 18f30ebaa4174fd28c958e72d0cc7e24 P 7a9daf5d42a84615a6383cfec9a10608 [LEADER]: Connected to new peer: Peer: permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260129 02:59:14.598708 20435 raft_consensus.cc:1275] T c934d31efd3f4e5195605ddc354d14e1 P 197b880fa39d432a9b472845876bfaf0 [term 2 FOLLOWER]: Refusing update from remote peer 7a9daf5d42a84615a6383cfec9a10608: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260129 02:59:14.600769 21701 consensus_queue.cc:1048] T c934d31efd3f4e5195605ddc354d14e1 P 7a9daf5d42a84615a6383cfec9a10608 [LEADER]: Connected to new peer: Peer: permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260129 02:59:14.650167 20510 raft_consensus.cc:1275] T c934d31efd3f4e5195605ddc354d14e1 P 974b802141bd4f13beffef2aa03f88f9 [term 2 FOLLOWER]: Refusing update from remote peer 7a9daf5d42a84615a6383cfec9a10608: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260129 02:59:14.652696 21700 consensus_queue.cc:1048] T c934d31efd3f4e5195605ddc354d14e1 P 7a9daf5d42a84615a6383cfec9a10608 [LEADER]: Connected to new peer: Peer: permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260129 02:59:14.884815 20438 consensus_queue.cc:237] T 1299723a4476420ab02fd8bfd7aa0645 P 197b880fa39d432a9b472845876bfaf0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } attrs { replace: true } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: NON_VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: true } }
W20260129 02:59:14.891176 20405 consensus_peers.cc:597] T 1299723a4476420ab02fd8bfd7aa0645 P 197b880fa39d432a9b472845876bfaf0 -> Peer 7a9daf5d42a84615a6383cfec9a10608 (127.18.211.4:40439): Couldn't send request to peer 7a9daf5d42a84615a6383cfec9a10608. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 1299723a4476420ab02fd8bfd7aa0645. This is attempt 1: this message will repeat every 5th retry.
I20260129 02:59:14.892032 20513 raft_consensus.cc:1275] T 1299723a4476420ab02fd8bfd7aa0645 P 974b802141bd4f13beffef2aa03f88f9 [term 2 FOLLOWER]: Refusing update from remote peer 197b880fa39d432a9b472845876bfaf0: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20260129 02:59:14.892964 20582 raft_consensus.cc:1275] T 1299723a4476420ab02fd8bfd7aa0645 P 1c50de89a5a845668696dcb28c3badd6 [term 2 FOLLOWER]: Refusing update from remote peer 197b880fa39d432a9b472845876bfaf0: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20260129 02:59:14.897629 21663 consensus_queue.cc:1048] T 1299723a4476420ab02fd8bfd7aa0645 P 197b880fa39d432a9b472845876bfaf0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260129 02:59:14.899120 21666 consensus_queue.cc:1048] T 1299723a4476420ab02fd8bfd7aa0645 P 197b880fa39d432a9b472845876bfaf0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260129 02:59:14.906426 21662 raft_consensus.cc:2955] T 1299723a4476420ab02fd8bfd7aa0645 P 197b880fa39d432a9b472845876bfaf0 [term 2 LEADER]: Committing config change with OpId 2.3: config changed from index -1 to 3, NON_VOTER 7a9daf5d42a84615a6383cfec9a10608 (127.18.211.4) added. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } attrs { replace: true } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: NON_VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: true } } }
I20260129 02:59:14.912948 20513 raft_consensus.cc:2955] T 1299723a4476420ab02fd8bfd7aa0645 P 974b802141bd4f13beffef2aa03f88f9 [term 2 FOLLOWER]: Committing config change with OpId 2.3: config changed from index -1 to 3, NON_VOTER 7a9daf5d42a84615a6383cfec9a10608 (127.18.211.4) added. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } attrs { replace: true } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: NON_VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: true } } }
I20260129 02:59:14.920567 20582 raft_consensus.cc:2955] T 1299723a4476420ab02fd8bfd7aa0645 P 1c50de89a5a845668696dcb28c3badd6 [term 2 FOLLOWER]: Committing config change with OpId 2.3: config changed from index -1 to 3, NON_VOTER 7a9daf5d42a84615a6383cfec9a10608 (127.18.211.4) added. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } attrs { replace: true } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: NON_VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: true } } }
I20260129 02:59:14.931747 20581 consensus_queue.cc:237] T add8d4923d1a489e86882c3a25692b95 P 1c50de89a5a845668696dcb28c3badd6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } attrs { replace: true } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: NON_VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: true } }
I20260129 02:59:14.945458 20337 catalog_manager.cc:5654] T 1299723a4476420ab02fd8bfd7aa0645 P 197b880fa39d432a9b472845876bfaf0 reported cstate change: config changed from index -1 to 3, NON_VOTER 7a9daf5d42a84615a6383cfec9a10608 (127.18.211.4) added. New cstate: current_term: 2 leader_uuid: "197b880fa39d432a9b472845876bfaf0" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: NON_VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
W20260129 02:59:14.992831 20554 consensus_peers.cc:597] T add8d4923d1a489e86882c3a25692b95 P 1c50de89a5a845668696dcb28c3badd6 -> Peer 7a9daf5d42a84615a6383cfec9a10608 (127.18.211.4:40439): Couldn't send request to peer 7a9daf5d42a84615a6383cfec9a10608. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: add8d4923d1a489e86882c3a25692b95. This is attempt 1: this message will repeat every 5th retry.
I20260129 02:59:14.996588 20513 raft_consensus.cc:1275] T add8d4923d1a489e86882c3a25692b95 P 974b802141bd4f13beffef2aa03f88f9 [term 1 FOLLOWER]: Refusing update from remote peer 1c50de89a5a845668696dcb28c3badd6: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260129 02:59:14.996624 20438 raft_consensus.cc:1275] T add8d4923d1a489e86882c3a25692b95 P 197b880fa39d432a9b472845876bfaf0 [term 1 FOLLOWER]: Refusing update from remote peer 1c50de89a5a845668696dcb28c3badd6: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260129 02:59:14.999918 21638 consensus_queue.cc:1048] T add8d4923d1a489e86882c3a25692b95 P 1c50de89a5a845668696dcb28c3badd6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260129 02:59:15.001899 21638 consensus_queue.cc:1048] T add8d4923d1a489e86882c3a25692b95 P 1c50de89a5a845668696dcb28c3badd6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260129 02:59:15.029887 21684 raft_consensus.cc:2955] T add8d4923d1a489e86882c3a25692b95 P 1c50de89a5a845668696dcb28c3badd6 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 7a9daf5d42a84615a6383cfec9a10608 (127.18.211.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } attrs { replace: true } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: NON_VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: true } } }
I20260129 02:59:15.048122 20437 consensus_queue.cc:237] T 3ccac82a01a2445fa5b0ce6c0fcf2295 P 197b880fa39d432a9b472845876bfaf0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } attrs { replace: true } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: NON_VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: true } }
I20260129 02:59:15.049324 19276 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
W20260129 02:59:15.061533 20405 consensus_peers.cc:597] T 3ccac82a01a2445fa5b0ce6c0fcf2295 P 197b880fa39d432a9b472845876bfaf0 -> Peer 7a9daf5d42a84615a6383cfec9a10608 (127.18.211.4:40439): Couldn't send request to peer 7a9daf5d42a84615a6383cfec9a10608. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 3ccac82a01a2445fa5b0ce6c0fcf2295. This is attempt 1: this message will repeat every 5th retry.
I20260129 02:59:15.065855 20508 raft_consensus.cc:1275] T 3ccac82a01a2445fa5b0ce6c0fcf2295 P 974b802141bd4f13beffef2aa03f88f9 [term 1 FOLLOWER]: Refusing update from remote peer 197b880fa39d432a9b472845876bfaf0: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260129 02:59:15.070490 21624 consensus_queue.cc:1048] T 3ccac82a01a2445fa5b0ce6c0fcf2295 P 197b880fa39d432a9b472845876bfaf0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260129 02:59:15.071588 20438 raft_consensus.cc:2955] T add8d4923d1a489e86882c3a25692b95 P 197b880fa39d432a9b472845876bfaf0 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 7a9daf5d42a84615a6383cfec9a10608 (127.18.211.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } attrs { replace: true } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: NON_VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: true } } }
I20260129 02:59:15.034897 20513 raft_consensus.cc:2955] T add8d4923d1a489e86882c3a25692b95 P 974b802141bd4f13beffef2aa03f88f9 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 7a9daf5d42a84615a6383cfec9a10608 (127.18.211.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } attrs { replace: true } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: NON_VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: true } } }
I20260129 02:59:15.089784 21624 raft_consensus.cc:2955] T 3ccac82a01a2445fa5b0ce6c0fcf2295 P 197b880fa39d432a9b472845876bfaf0 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 7a9daf5d42a84615a6383cfec9a10608 (127.18.211.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } attrs { replace: true } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: NON_VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: true } } }
I20260129 02:59:15.092833 20582 raft_consensus.cc:1275] T 3ccac82a01a2445fa5b0ce6c0fcf2295 P 1c50de89a5a845668696dcb28c3badd6 [term 1 FOLLOWER]: Refusing update from remote peer 197b880fa39d432a9b472845876bfaf0: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260129 02:59:15.093355 20511 raft_consensus.cc:2955] T 3ccac82a01a2445fa5b0ce6c0fcf2295 P 974b802141bd4f13beffef2aa03f88f9 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 7a9daf5d42a84615a6383cfec9a10608 (127.18.211.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } attrs { replace: true } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: NON_VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: true } } }
I20260129 02:59:15.112675 20437 tablet_service.cc:2066] Received LeaderStepDown RPC: tablet_id: "00e79d8f25a34676b1d866eb2068a2b2"
dest_uuid: "197b880fa39d432a9b472845876bfaf0"
mode: GRACEFUL
new_leader_uuid: "7a9daf5d42a84615a6383cfec9a10608"
from {username='slave'} at 127.0.0.1:57776
I20260129 02:59:15.113274 20437 raft_consensus.cc:606] T 00e79d8f25a34676b1d866eb2068a2b2 P 197b880fa39d432a9b472845876bfaf0 [term 1 LEADER]: Received request to transfer leadership to TS 7a9daf5d42a84615a6383cfec9a10608
I20260129 02:59:15.115731 21666 consensus_queue.cc:1048] T 3ccac82a01a2445fa5b0ce6c0fcf2295 P 197b880fa39d432a9b472845876bfaf0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260129 02:59:15.128134 20437 tablet_service.cc:2066] Received LeaderStepDown RPC: tablet_id: "0340229d521b432580ae926f6062e3f7"
dest_uuid: "197b880fa39d432a9b472845876bfaf0"
mode: GRACEFUL
new_leader_uuid: "1c50de89a5a845668696dcb28c3badd6"
from {username='slave'} at 127.0.0.1:57776
I20260129 02:59:15.133204 20437 raft_consensus.cc:606] T 0340229d521b432580ae926f6062e3f7 P 197b880fa39d432a9b472845876bfaf0 [term 1 LEADER]: Received request to transfer leadership to TS 1c50de89a5a845668696dcb28c3badd6
I20260129 02:59:15.132488 20582 raft_consensus.cc:2955] T 3ccac82a01a2445fa5b0ce6c0fcf2295 P 1c50de89a5a845668696dcb28c3badd6 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 7a9daf5d42a84615a6383cfec9a10608 (127.18.211.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } attrs { replace: true } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: NON_VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: true } } }
I20260129 02:59:15.137135 20590 tablet_service.cc:2066] Received LeaderStepDown RPC: tablet_id: "4b1876f9a70b49edb3aa74105e66d3b3"
dest_uuid: "1c50de89a5a845668696dcb28c3badd6"
mode: GRACEFUL
new_leader_uuid: "197b880fa39d432a9b472845876bfaf0"
from {username='slave'} at 127.0.0.1:54860
I20260129 02:59:15.137735 20590 raft_consensus.cc:606] T 4b1876f9a70b49edb3aa74105e66d3b3 P 1c50de89a5a845668696dcb28c3badd6 [term 1 LEADER]: Received request to transfer leadership to TS 197b880fa39d432a9b472845876bfaf0
I20260129 02:59:15.141577 20508 tablet_service.cc:2066] Received LeaderStepDown RPC: tablet_id: "a5a05fbcbbcf4916bbc01fd1a5b5e90d"
dest_uuid: "974b802141bd4f13beffef2aa03f88f9"
mode: GRACEFUL
new_leader_uuid: "1c50de89a5a845668696dcb28c3badd6"
from {username='slave'} at 127.0.0.1:52008
I20260129 02:59:15.142169 20508 raft_consensus.cc:606] T a5a05fbcbbcf4916bbc01fd1a5b5e90d P 974b802141bd4f13beffef2aa03f88f9 [term 1 LEADER]: Received request to transfer leadership to TS 1c50de89a5a845668696dcb28c3badd6
I20260129 02:59:15.147465 20590 tablet_service.cc:2066] Received LeaderStepDown RPC: tablet_id: "c891d2bff4ce4568861bbc992e91fbb0"
dest_uuid: "1c50de89a5a845668696dcb28c3badd6"
mode: GRACEFUL
new_leader_uuid: "197b880fa39d432a9b472845876bfaf0"
from {username='slave'} at 127.0.0.1:54860
I20260129 02:59:15.148064 20590 raft_consensus.cc:606] T c891d2bff4ce4568861bbc992e91fbb0 P 1c50de89a5a845668696dcb28c3badd6 [term 4 LEADER]: Received request to transfer leadership to TS 197b880fa39d432a9b472845876bfaf0
I20260129 02:59:15.153530 19276 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 5
I20260129 02:59:15.155380 19276 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20260129 02:59:15.156203 19276 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260129 02:59:15.119477 20335 catalog_manager.cc:5654] T add8d4923d1a489e86882c3a25692b95 P 1c50de89a5a845668696dcb28c3badd6 reported cstate change: config changed from index -1 to 2, NON_VOTER 7a9daf5d42a84615a6383cfec9a10608 (127.18.211.4) added. New cstate: current_term: 1 leader_uuid: "1c50de89a5a845668696dcb28c3badd6" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: NON_VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20260129 02:59:15.171027 20337 catalog_manager.cc:5654] T 3ccac82a01a2445fa5b0ce6c0fcf2295 P 974b802141bd4f13beffef2aa03f88f9 reported cstate change: config changed from index -1 to 2, NON_VOTER 7a9daf5d42a84615a6383cfec9a10608 (127.18.211.4) added. New cstate: current_term: 1 leader_uuid: "197b880fa39d432a9b472845876bfaf0" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } attrs { replace: true } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: NON_VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: true } } }
I20260129 02:59:15.173415 21594 raft_consensus.cc:993] T 4b1876f9a70b49edb3aa74105e66d3b3 P 1c50de89a5a845668696dcb28c3badd6: : Instructing follower 197b880fa39d432a9b472845876bfaf0 to start an election
I20260129 02:59:15.173943 21638 raft_consensus.cc:1081] T 4b1876f9a70b49edb3aa74105e66d3b3 P 1c50de89a5a845668696dcb28c3badd6 [term 1 LEADER]: Signalling peer 197b880fa39d432a9b472845876bfaf0 to start an election
I20260129 02:59:15.176857 20435 tablet_service.cc:2038] Received Run Leader Election RPC: tablet_id: "4b1876f9a70b49edb3aa74105e66d3b3"
dest_uuid: "197b880fa39d432a9b472845876bfaf0"
from {username='slave'} at 127.0.0.1:55676
I20260129 02:59:15.177378 20435 raft_consensus.cc:493] T 4b1876f9a70b49edb3aa74105e66d3b3 P 197b880fa39d432a9b472845876bfaf0 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260129 02:59:15.177738 20435 raft_consensus.cc:3060] T 4b1876f9a70b49edb3aa74105e66d3b3 P 197b880fa39d432a9b472845876bfaf0 [term 1 FOLLOWER]: Advancing to term 2
I20260129 02:59:15.185931 21729 raft_consensus.cc:993] T a5a05fbcbbcf4916bbc01fd1a5b5e90d P 974b802141bd4f13beffef2aa03f88f9: : Instructing follower 1c50de89a5a845668696dcb28c3badd6 to start an election
I20260129 02:59:15.189747 21513 raft_consensus.cc:1081] T a5a05fbcbbcf4916bbc01fd1a5b5e90d P 974b802141bd4f13beffef2aa03f88f9 [term 1 LEADER]: Signalling peer 1c50de89a5a845668696dcb28c3badd6 to start an election
I20260129 02:59:15.191998 20581 consensus_queue.cc:237] T 96609ac68bdf4993a12e1c23355ca013 P 1c50de89a5a845668696dcb28c3badd6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } attrs { replace: true } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: NON_VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: true } }
I20260129 02:59:15.213449 20435 raft_consensus.cc:515] T 4b1876f9a70b49edb3aa74105e66d3b3 P 197b880fa39d432a9b472845876bfaf0 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:59:15.216267 20590 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4b1876f9a70b49edb3aa74105e66d3b3" candidate_uuid: "197b880fa39d432a9b472845876bfaf0" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "1c50de89a5a845668696dcb28c3badd6"
I20260129 02:59:15.216826 20590 raft_consensus.cc:3055] T 4b1876f9a70b49edb3aa74105e66d3b3 P 1c50de89a5a845668696dcb28c3badd6 [term 1 LEADER]: Stepping down as leader of term 1
I20260129 02:59:15.217096 20590 raft_consensus.cc:740] T 4b1876f9a70b49edb3aa74105e66d3b3 P 1c50de89a5a845668696dcb28c3badd6 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 1c50de89a5a845668696dcb28c3badd6, State: Running, Role: LEADER
I20260129 02:59:15.217849 20590 consensus_queue.cc:260] T 4b1876f9a70b49edb3aa74105e66d3b3 P 1c50de89a5a845668696dcb28c3badd6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:59:15.218947 20590 raft_consensus.cc:3060] T 4b1876f9a70b49edb3aa74105e66d3b3 P 1c50de89a5a845668696dcb28c3badd6 [term 1 FOLLOWER]: Advancing to term 2
I20260129 02:59:15.227551 21684 raft_consensus.cc:993] T c891d2bff4ce4568861bbc992e91fbb0 P 1c50de89a5a845668696dcb28c3badd6: : Instructing follower 197b880fa39d432a9b472845876bfaf0 to start an election
I20260129 02:59:15.228828 20435 leader_election.cc:290] T 4b1876f9a70b49edb3aa74105e66d3b3 P 197b880fa39d432a9b472845876bfaf0 [CANDIDATE]: Term 2 election: Requested vote from peers 974b802141bd4f13beffef2aa03f88f9 (127.18.211.2:34517), 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3:37735)
I20260129 02:59:15.229274 21684 raft_consensus.cc:1081] T c891d2bff4ce4568861bbc992e91fbb0 P 1c50de89a5a845668696dcb28c3badd6 [term 4 LEADER]: Signalling peer 197b880fa39d432a9b472845876bfaf0 to start an election
I20260129 02:59:15.227772 20511 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4b1876f9a70b49edb3aa74105e66d3b3" candidate_uuid: "197b880fa39d432a9b472845876bfaf0" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "974b802141bd4f13beffef2aa03f88f9"
I20260129 02:59:15.230307 20511 raft_consensus.cc:3060] T 4b1876f9a70b49edb3aa74105e66d3b3 P 974b802141bd4f13beffef2aa03f88f9 [term 1 FOLLOWER]: Advancing to term 2
I20260129 02:59:15.231070 20438 tablet_service.cc:2038] Received Run Leader Election RPC: tablet_id: "c891d2bff4ce4568861bbc992e91fbb0"
dest_uuid: "197b880fa39d432a9b472845876bfaf0"
from {username='slave'} at 127.0.0.1:55676
I20260129 02:59:15.231611 20438 raft_consensus.cc:493] T c891d2bff4ce4568861bbc992e91fbb0 P 197b880fa39d432a9b472845876bfaf0 [term 4 FOLLOWER]: Starting forced leader election (received explicit request)
I20260129 02:59:15.231971 20438 raft_consensus.cc:3060] T c891d2bff4ce4568861bbc992e91fbb0 P 197b880fa39d432a9b472845876bfaf0 [term 4 FOLLOWER]: Advancing to term 5
I20260129 02:59:15.240958 21662 raft_consensus.cc:993] T 00e79d8f25a34676b1d866eb2068a2b2 P 197b880fa39d432a9b472845876bfaf0: : Instructing follower 7a9daf5d42a84615a6383cfec9a10608 to start an election
I20260129 02:59:15.241469 21663 raft_consensus.cc:1081] T 00e79d8f25a34676b1d866eb2068a2b2 P 197b880fa39d432a9b472845876bfaf0 [term 1 LEADER]: Signalling peer 7a9daf5d42a84615a6383cfec9a10608 to start an election
I20260129 02:59:15.243048 21134 tablet_service.cc:2038] Received Run Leader Election RPC: tablet_id: "00e79d8f25a34676b1d866eb2068a2b2"
dest_uuid: "7a9daf5d42a84615a6383cfec9a10608"
from {username='slave'} at 127.0.0.1:34060
I20260129 02:59:15.243572 21134 raft_consensus.cc:493] T 00e79d8f25a34676b1d866eb2068a2b2 P 7a9daf5d42a84615a6383cfec9a10608 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260129 02:59:15.243937 21134 raft_consensus.cc:3060] T 00e79d8f25a34676b1d866eb2068a2b2 P 7a9daf5d42a84615a6383cfec9a10608 [term 1 FOLLOWER]: Advancing to term 2
I20260129 02:59:15.228391 20582 tablet_service.cc:2038] Received Run Leader Election RPC: tablet_id: "a5a05fbcbbcf4916bbc01fd1a5b5e90d"
dest_uuid: "1c50de89a5a845668696dcb28c3badd6"
from {username='slave'} at 127.0.0.1:35752
I20260129 02:59:15.251618 21134 raft_consensus.cc:515] T 00e79d8f25a34676b1d866eb2068a2b2 P 7a9daf5d42a84615a6383cfec9a10608 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } }
W20260129 02:59:15.262310 20554 consensus_peers.cc:597] T 96609ac68bdf4993a12e1c23355ca013 P 1c50de89a5a845668696dcb28c3badd6 -> Peer 7a9daf5d42a84615a6383cfec9a10608 (127.18.211.4:40439): Couldn't send request to peer 7a9daf5d42a84615a6383cfec9a10608. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 96609ac68bdf4993a12e1c23355ca013. This is attempt 1: this message will repeat every 5th retry.
I20260129 02:59:15.262576 21134 leader_election.cc:290] T 00e79d8f25a34676b1d866eb2068a2b2 P 7a9daf5d42a84615a6383cfec9a10608 [CANDIDATE]: Term 2 election: Requested vote from peers 197b880fa39d432a9b472845876bfaf0 (127.18.211.1:42027), 974b802141bd4f13beffef2aa03f88f9 (127.18.211.2:34517)
I20260129 02:59:15.260624 20510 raft_consensus.cc:1275] T 96609ac68bdf4993a12e1c23355ca013 P 974b802141bd4f13beffef2aa03f88f9 [term 1 FOLLOWER]: Refusing update from remote peer 1c50de89a5a845668696dcb28c3badd6: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260129 02:59:15.262835 20434 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00e79d8f25a34676b1d866eb2068a2b2" candidate_uuid: "7a9daf5d42a84615a6383cfec9a10608" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "197b880fa39d432a9b472845876bfaf0"
I20260129 02:59:15.246191 20511 raft_consensus.cc:2468] T 4b1876f9a70b49edb3aa74105e66d3b3 P 974b802141bd4f13beffef2aa03f88f9 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 197b880fa39d432a9b472845876bfaf0 in term 2.
I20260129 02:59:15.263645 20434 raft_consensus.cc:3055] T 00e79d8f25a34676b1d866eb2068a2b2 P 197b880fa39d432a9b472845876bfaf0 [term 1 LEADER]: Stepping down as leader of term 1
I20260129 02:59:15.263974 20513 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00e79d8f25a34676b1d866eb2068a2b2" candidate_uuid: "7a9daf5d42a84615a6383cfec9a10608" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "974b802141bd4f13beffef2aa03f88f9"
I20260129 02:59:15.264586 20513 raft_consensus.cc:3060] T 00e79d8f25a34676b1d866eb2068a2b2 P 974b802141bd4f13beffef2aa03f88f9 [term 1 FOLLOWER]: Advancing to term 2
I20260129 02:59:15.265020 20407 leader_election.cc:304] T 4b1876f9a70b49edb3aa74105e66d3b3 P 197b880fa39d432a9b472845876bfaf0 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 197b880fa39d432a9b472845876bfaf0, 974b802141bd4f13beffef2aa03f88f9; no voters:
I20260129 02:59:15.264030 20434 raft_consensus.cc:740] T 00e79d8f25a34676b1d866eb2068a2b2 P 197b880fa39d432a9b472845876bfaf0 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 197b880fa39d432a9b472845876bfaf0, State: Running, Role: LEADER
I20260129 02:59:15.253034 20582 raft_consensus.cc:493] T a5a05fbcbbcf4916bbc01fd1a5b5e90d P 1c50de89a5a845668696dcb28c3badd6 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260129 02:59:15.266155 20582 raft_consensus.cc:3060] T a5a05fbcbbcf4916bbc01fd1a5b5e90d P 1c50de89a5a845668696dcb28c3badd6 [term 1 FOLLOWER]: Advancing to term 2
I20260129 02:59:15.266333 20434 consensus_queue.cc:260] T 00e79d8f25a34676b1d866eb2068a2b2 P 197b880fa39d432a9b472845876bfaf0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } }
I20260129 02:59:15.267488 20434 raft_consensus.cc:3060] T 00e79d8f25a34676b1d866eb2068a2b2 P 197b880fa39d432a9b472845876bfaf0 [term 1 FOLLOWER]: Advancing to term 2
I20260129 02:59:15.269012 21594 consensus_queue.cc:1048] T 96609ac68bdf4993a12e1c23355ca013 P 1c50de89a5a845668696dcb28c3badd6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260129 02:59:15.270881 21662 raft_consensus.cc:2804] T 4b1876f9a70b49edb3aa74105e66d3b3 P 197b880fa39d432a9b472845876bfaf0 [term 2 FOLLOWER]: Leader election won for term 2
I20260129 02:59:15.260682 20437 raft_consensus.cc:1275] T 96609ac68bdf4993a12e1c23355ca013 P 197b880fa39d432a9b472845876bfaf0 [term 1 FOLLOWER]: Refusing update from remote peer 1c50de89a5a845668696dcb28c3badd6: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260129 02:59:15.272907 21594 consensus_queue.cc:1048] T 96609ac68bdf4993a12e1c23355ca013 P 1c50de89a5a845668696dcb28c3badd6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260129 02:59:15.273809 20513 raft_consensus.cc:2468] T 00e79d8f25a34676b1d866eb2068a2b2 P 974b802141bd4f13beffef2aa03f88f9 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7a9daf5d42a84615a6383cfec9a10608 in term 2.
I20260129 02:59:15.274927 21102 leader_election.cc:304] T 00e79d8f25a34676b1d866eb2068a2b2 P 7a9daf5d42a84615a6383cfec9a10608 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7a9daf5d42a84615a6383cfec9a10608, 974b802141bd4f13beffef2aa03f88f9; no voters:
I20260129 02:59:15.279101 21662 raft_consensus.cc:697] T 4b1876f9a70b49edb3aa74105e66d3b3 P 197b880fa39d432a9b472845876bfaf0 [term 2 LEADER]: Becoming Leader. State: Replica: 197b880fa39d432a9b472845876bfaf0, State: Running, Role: LEADER
I20260129 02:59:15.279275 21700 raft_consensus.cc:2804] T 00e79d8f25a34676b1d866eb2068a2b2 P 7a9daf5d42a84615a6383cfec9a10608 [term 2 FOLLOWER]: Leader election won for term 2
I20260129 02:59:15.279853 21700 raft_consensus.cc:697] T 00e79d8f25a34676b1d866eb2068a2b2 P 7a9daf5d42a84615a6383cfec9a10608 [term 2 LEADER]: Becoming Leader. State: Replica: 7a9daf5d42a84615a6383cfec9a10608, State: Running, Role: LEADER
I20260129 02:59:15.254381 20590 raft_consensus.cc:2468] T 4b1876f9a70b49edb3aa74105e66d3b3 P 1c50de89a5a845668696dcb28c3badd6 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 197b880fa39d432a9b472845876bfaf0 in term 2.
I20260129 02:59:15.280108 21662 consensus_queue.cc:237] T 4b1876f9a70b49edb3aa74105e66d3b3 P 197b880fa39d432a9b472845876bfaf0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:59:15.280692 21700 consensus_queue.cc:237] T 00e79d8f25a34676b1d866eb2068a2b2 P 7a9daf5d42a84615a6383cfec9a10608 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } }
I20260129 02:59:15.302078 20582 raft_consensus.cc:515] T a5a05fbcbbcf4916bbc01fd1a5b5e90d P 1c50de89a5a845668696dcb28c3badd6 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:59:15.275633 20434 raft_consensus.cc:2468] T 00e79d8f25a34676b1d866eb2068a2b2 P 197b880fa39d432a9b472845876bfaf0 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7a9daf5d42a84615a6383cfec9a10608 in term 2.
I20260129 02:59:15.305756 20435 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a5a05fbcbbcf4916bbc01fd1a5b5e90d" candidate_uuid: "1c50de89a5a845668696dcb28c3badd6" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "197b880fa39d432a9b472845876bfaf0"
I20260129 02:59:15.306321 20435 raft_consensus.cc:3060] T a5a05fbcbbcf4916bbc01fd1a5b5e90d P 197b880fa39d432a9b472845876bfaf0 [term 1 FOLLOWER]: Advancing to term 2
I20260129 02:59:15.254033 20438 raft_consensus.cc:515] T c891d2bff4ce4568861bbc992e91fbb0 P 197b880fa39d432a9b472845876bfaf0 [term 5 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:59:15.310177 20438 leader_election.cc:290] T c891d2bff4ce4568861bbc992e91fbb0 P 197b880fa39d432a9b472845876bfaf0 [CANDIDATE]: Term 5 election: Requested vote from peers 974b802141bd4f13beffef2aa03f88f9 (127.18.211.2:34517), 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3:37735)
I20260129 02:59:15.312167 20590 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c891d2bff4ce4568861bbc992e91fbb0" candidate_uuid: "197b880fa39d432a9b472845876bfaf0" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: true dest_uuid: "1c50de89a5a845668696dcb28c3badd6"
I20260129 02:59:15.312726 20590 raft_consensus.cc:3055] T c891d2bff4ce4568861bbc992e91fbb0 P 1c50de89a5a845668696dcb28c3badd6 [term 4 LEADER]: Stepping down as leader of term 4
I20260129 02:59:15.313006 20590 raft_consensus.cc:740] T c891d2bff4ce4568861bbc992e91fbb0 P 1c50de89a5a845668696dcb28c3badd6 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 1c50de89a5a845668696dcb28c3badd6, State: Running, Role: LEADER
I20260129 02:59:15.313757 20590 consensus_queue.cc:260] T c891d2bff4ce4568861bbc992e91fbb0 P 1c50de89a5a845668696dcb28c3badd6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:59:15.315245 20590 raft_consensus.cc:3060] T c891d2bff4ce4568861bbc992e91fbb0 P 1c50de89a5a845668696dcb28c3badd6 [term 4 FOLLOWER]: Advancing to term 5
I20260129 02:59:15.316076 20510 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a5a05fbcbbcf4916bbc01fd1a5b5e90d" candidate_uuid: "1c50de89a5a845668696dcb28c3badd6" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "974b802141bd4f13beffef2aa03f88f9"
I20260129 02:59:15.316753 20510 raft_consensus.cc:3055] T a5a05fbcbbcf4916bbc01fd1a5b5e90d P 974b802141bd4f13beffef2aa03f88f9 [term 1 LEADER]: Stepping down as leader of term 1
I20260129 02:59:15.316020 21594 raft_consensus.cc:2955] T 96609ac68bdf4993a12e1c23355ca013 P 1c50de89a5a845668696dcb28c3badd6 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 7a9daf5d42a84615a6383cfec9a10608 (127.18.211.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } attrs { replace: true } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: NON_VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: true } } }
I20260129 02:59:15.313620 20337 catalog_manager.cc:5654] T 00e79d8f25a34676b1d866eb2068a2b2 P 7a9daf5d42a84615a6383cfec9a10608 reported cstate change: term changed from 1 to 2, leader changed from 197b880fa39d432a9b472845876bfaf0 (127.18.211.1) to 7a9daf5d42a84615a6383cfec9a10608 (127.18.211.4). New cstate: current_term: 2 leader_uuid: "7a9daf5d42a84615a6383cfec9a10608" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260129 02:59:15.318269 20437 raft_consensus.cc:2955] T 96609ac68bdf4993a12e1c23355ca013 P 197b880fa39d432a9b472845876bfaf0 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 7a9daf5d42a84615a6383cfec9a10608 (127.18.211.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } attrs { replace: true } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: NON_VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: true } } }
I20260129 02:59:15.315186 20513 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c891d2bff4ce4568861bbc992e91fbb0" candidate_uuid: "197b880fa39d432a9b472845876bfaf0" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: true dest_uuid: "974b802141bd4f13beffef2aa03f88f9"
I20260129 02:59:15.319990 20513 raft_consensus.cc:3060] T c891d2bff4ce4568861bbc992e91fbb0 P 974b802141bd4f13beffef2aa03f88f9 [term 4 FOLLOWER]: Advancing to term 5
I20260129 02:59:15.317054 20510 raft_consensus.cc:740] T a5a05fbcbbcf4916bbc01fd1a5b5e90d P 974b802141bd4f13beffef2aa03f88f9 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 974b802141bd4f13beffef2aa03f88f9, State: Running, Role: LEADER
I20260129 02:59:15.324462 20510 consensus_queue.cc:260] T a5a05fbcbbcf4916bbc01fd1a5b5e90d P 974b802141bd4f13beffef2aa03f88f9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:59:15.325568 20510 raft_consensus.cc:3060] T a5a05fbcbbcf4916bbc01fd1a5b5e90d P 974b802141bd4f13beffef2aa03f88f9 [term 1 FOLLOWER]: Advancing to term 2
I20260129 02:59:15.327744 20582 leader_election.cc:290] T a5a05fbcbbcf4916bbc01fd1a5b5e90d P 1c50de89a5a845668696dcb28c3badd6 [CANDIDATE]: Term 2 election: Requested vote from peers 197b880fa39d432a9b472845876bfaf0 (127.18.211.1:42027), 974b802141bd4f13beffef2aa03f88f9 (127.18.211.2:34517)
I20260129 02:59:15.334582 20435 raft_consensus.cc:2468] T a5a05fbcbbcf4916bbc01fd1a5b5e90d P 197b880fa39d432a9b472845876bfaf0 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1c50de89a5a845668696dcb28c3badd6 in term 2.
I20260129 02:59:15.334925 20590 raft_consensus.cc:2468] T c891d2bff4ce4568861bbc992e91fbb0 P 1c50de89a5a845668696dcb28c3badd6 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 197b880fa39d432a9b472845876bfaf0 in term 5.
I20260129 02:59:15.337996 20553 leader_election.cc:304] T a5a05fbcbbcf4916bbc01fd1a5b5e90d P 1c50de89a5a845668696dcb28c3badd6 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 197b880fa39d432a9b472845876bfaf0, 1c50de89a5a845668696dcb28c3badd6; no voters:
I20260129 02:59:15.341218 21748 raft_consensus.cc:2804] T a5a05fbcbbcf4916bbc01fd1a5b5e90d P 1c50de89a5a845668696dcb28c3badd6 [term 2 FOLLOWER]: Leader election won for term 2
I20260129 02:59:15.341750 21748 raft_consensus.cc:697] T a5a05fbcbbcf4916bbc01fd1a5b5e90d P 1c50de89a5a845668696dcb28c3badd6 [term 2 LEADER]: Becoming Leader. State: Replica: 1c50de89a5a845668696dcb28c3badd6, State: Running, Role: LEADER
I20260129 02:59:15.342334 20405 leader_election.cc:304] T c891d2bff4ce4568861bbc992e91fbb0 P 197b880fa39d432a9b472845876bfaf0 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 197b880fa39d432a9b472845876bfaf0, 1c50de89a5a845668696dcb28c3badd6; no voters:
I20260129 02:59:15.342689 21748 consensus_queue.cc:237] T a5a05fbcbbcf4916bbc01fd1a5b5e90d P 1c50de89a5a845668696dcb28c3badd6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:59:15.343240 21665 raft_consensus.cc:2804] T c891d2bff4ce4568861bbc992e91fbb0 P 197b880fa39d432a9b472845876bfaf0 [term 5 FOLLOWER]: Leader election won for term 5
I20260129 02:59:15.346964 21665 raft_consensus.cc:697] T c891d2bff4ce4568861bbc992e91fbb0 P 197b880fa39d432a9b472845876bfaf0 [term 5 LEADER]: Becoming Leader. State: Replica: 197b880fa39d432a9b472845876bfaf0, State: Running, Role: LEADER
I20260129 02:59:15.348428 21665 consensus_queue.cc:237] T c891d2bff4ce4568861bbc992e91fbb0 P 197b880fa39d432a9b472845876bfaf0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:59:15.347062 20513 raft_consensus.cc:2468] T c891d2bff4ce4568861bbc992e91fbb0 P 974b802141bd4f13beffef2aa03f88f9 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 197b880fa39d432a9b472845876bfaf0 in term 5.
I20260129 02:59:15.371979 20513 raft_consensus.cc:2955] T 96609ac68bdf4993a12e1c23355ca013 P 974b802141bd4f13beffef2aa03f88f9 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 7a9daf5d42a84615a6383cfec9a10608 (127.18.211.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } attrs { replace: true } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: NON_VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: true } } }
I20260129 02:59:15.397446 20510 raft_consensus.cc:2468] T a5a05fbcbbcf4916bbc01fd1a5b5e90d P 974b802141bd4f13beffef2aa03f88f9 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1c50de89a5a845668696dcb28c3badd6 in term 2.
I20260129 02:59:15.407670 20337 catalog_manager.cc:5654] T 4b1876f9a70b49edb3aa74105e66d3b3 P 197b880fa39d432a9b472845876bfaf0 reported cstate change: term changed from 1 to 2, leader changed from 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3) to 197b880fa39d432a9b472845876bfaf0 (127.18.211.1). New cstate: current_term: 2 leader_uuid: "197b880fa39d432a9b472845876bfaf0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } health_report { overall_health: UNKNOWN } } }
I20260129 02:59:15.446724 21662 raft_consensus.cc:993] T 0340229d521b432580ae926f6062e3f7 P 197b880fa39d432a9b472845876bfaf0: : Instructing follower 1c50de89a5a845668696dcb28c3badd6 to start an election
I20260129 02:59:15.447737 20335 catalog_manager.cc:5654] T 96609ac68bdf4993a12e1c23355ca013 P 974b802141bd4f13beffef2aa03f88f9 reported cstate change: config changed from index -1 to 2, NON_VOTER 7a9daf5d42a84615a6383cfec9a10608 (127.18.211.4) added. New cstate: current_term: 1 leader_uuid: "1c50de89a5a845668696dcb28c3badd6" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } attrs { replace: true } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: NON_VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: true } } }
I20260129 02:59:15.457046 21662 raft_consensus.cc:1081] T 0340229d521b432580ae926f6062e3f7 P 197b880fa39d432a9b472845876bfaf0 [term 1 LEADER]: Signalling peer 1c50de89a5a845668696dcb28c3badd6 to start an election
I20260129 02:59:15.460085 20581 tablet_service.cc:2038] Received Run Leader Election RPC: tablet_id: "0340229d521b432580ae926f6062e3f7"
dest_uuid: "1c50de89a5a845668696dcb28c3badd6"
from {username='slave'} at 127.0.0.1:35754
I20260129 02:59:15.460626 20581 raft_consensus.cc:493] T 0340229d521b432580ae926f6062e3f7 P 1c50de89a5a845668696dcb28c3badd6 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260129 02:59:15.460958 20581 raft_consensus.cc:3060] T 0340229d521b432580ae926f6062e3f7 P 1c50de89a5a845668696dcb28c3badd6 [term 1 FOLLOWER]: Advancing to term 2
I20260129 02:59:15.463543 21754 ts_tablet_manager.cc:933] T 1299723a4476420ab02fd8bfd7aa0645 P 7a9daf5d42a84615a6383cfec9a10608: Initiating tablet copy from peer 197b880fa39d432a9b472845876bfaf0 (127.18.211.1:42027)
I20260129 02:59:15.468587 20581 raft_consensus.cc:515] T 0340229d521b432580ae926f6062e3f7 P 1c50de89a5a845668696dcb28c3badd6 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:59:15.469967 20338 catalog_manager.cc:5654] T c891d2bff4ce4568861bbc992e91fbb0 P 197b880fa39d432a9b472845876bfaf0 reported cstate change: term changed from 4 to 5, leader changed from 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3) to 197b880fa39d432a9b472845876bfaf0 (127.18.211.1). New cstate: current_term: 5 leader_uuid: "197b880fa39d432a9b472845876bfaf0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } health_report { overall_health: UNKNOWN } } }
I20260129 02:59:15.471138 20435 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0340229d521b432580ae926f6062e3f7" candidate_uuid: "1c50de89a5a845668696dcb28c3badd6" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "197b880fa39d432a9b472845876bfaf0"
I20260129 02:59:15.471819 20435 raft_consensus.cc:3055] T 0340229d521b432580ae926f6062e3f7 P 197b880fa39d432a9b472845876bfaf0 [term 1 LEADER]: Stepping down as leader of term 1
I20260129 02:59:15.472342 20435 raft_consensus.cc:740] T 0340229d521b432580ae926f6062e3f7 P 197b880fa39d432a9b472845876bfaf0 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 197b880fa39d432a9b472845876bfaf0, State: Running, Role: LEADER
I20260129 02:59:15.472730 20513 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0340229d521b432580ae926f6062e3f7" candidate_uuid: "1c50de89a5a845668696dcb28c3badd6" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "974b802141bd4f13beffef2aa03f88f9"
I20260129 02:59:15.473436 20581 leader_election.cc:290] T 0340229d521b432580ae926f6062e3f7 P 1c50de89a5a845668696dcb28c3badd6 [CANDIDATE]: Term 2 election: Requested vote from peers 197b880fa39d432a9b472845876bfaf0 (127.18.211.1:42027), 974b802141bd4f13beffef2aa03f88f9 (127.18.211.2:34517)
I20260129 02:59:15.473258 20435 consensus_queue.cc:260] T 0340229d521b432580ae926f6062e3f7 P 197b880fa39d432a9b472845876bfaf0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:59:15.475006 20435 raft_consensus.cc:3060] T 0340229d521b432580ae926f6062e3f7 P 197b880fa39d432a9b472845876bfaf0 [term 1 FOLLOWER]: Advancing to term 2
I20260129 02:59:15.473284 20513 raft_consensus.cc:3060] T 0340229d521b432580ae926f6062e3f7 P 974b802141bd4f13beffef2aa03f88f9 [term 1 FOLLOWER]: Advancing to term 2
I20260129 02:59:15.477187 21754 tablet_copy_client.cc:323] T 1299723a4476420ab02fd8bfd7aa0645 P 7a9daf5d42a84615a6383cfec9a10608: tablet copy: Beginning tablet copy session from remote peer at address 127.18.211.1:42027
I20260129 02:59:15.482921 20513 raft_consensus.cc:2468] T 0340229d521b432580ae926f6062e3f7 P 974b802141bd4f13beffef2aa03f88f9 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1c50de89a5a845668696dcb28c3badd6 in term 2.
I20260129 02:59:15.483994 20556 leader_election.cc:304] T 0340229d521b432580ae926f6062e3f7 P 1c50de89a5a845668696dcb28c3badd6 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1c50de89a5a845668696dcb28c3badd6, 974b802141bd4f13beffef2aa03f88f9; no voters:
I20260129 02:59:15.484515 20435 raft_consensus.cc:2468] T 0340229d521b432580ae926f6062e3f7 P 197b880fa39d432a9b472845876bfaf0 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1c50de89a5a845668696dcb28c3badd6 in term 2.
I20260129 02:59:15.484800 21748 raft_consensus.cc:2804] T 0340229d521b432580ae926f6062e3f7 P 1c50de89a5a845668696dcb28c3badd6 [term 2 FOLLOWER]: Leader election won for term 2
I20260129 02:59:15.485334 21748 raft_consensus.cc:697] T 0340229d521b432580ae926f6062e3f7 P 1c50de89a5a845668696dcb28c3badd6 [term 2 LEADER]: Becoming Leader. State: Replica: 1c50de89a5a845668696dcb28c3badd6, State: Running, Role: LEADER
I20260129 02:59:15.486151 21748 consensus_queue.cc:237] T 0340229d521b432580ae926f6062e3f7 P 1c50de89a5a845668696dcb28c3badd6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:59:15.490891 20450 tablet_copy_service.cc:140] P 197b880fa39d432a9b472845876bfaf0: Received BeginTabletCopySession request for tablet 1299723a4476420ab02fd8bfd7aa0645 from peer 7a9daf5d42a84615a6383cfec9a10608 ({username='slave'} at 127.0.0.1:48356)
I20260129 02:59:15.491429 20450 tablet_copy_service.cc:161] P 197b880fa39d432a9b472845876bfaf0: Beginning new tablet copy session on tablet 1299723a4476420ab02fd8bfd7aa0645 from peer 7a9daf5d42a84615a6383cfec9a10608 at {username='slave'} at 127.0.0.1:48356: session id = 7a9daf5d42a84615a6383cfec9a10608-1299723a4476420ab02fd8bfd7aa0645
I20260129 02:59:15.498911 20450 tablet_copy_source_session.cc:215] T 1299723a4476420ab02fd8bfd7aa0645 P 197b880fa39d432a9b472845876bfaf0: Tablet Copy: opened 0 blocks and 1 log segments
I20260129 02:59:15.508119 21754 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1299723a4476420ab02fd8bfd7aa0645. 1 dirs total, 0 dirs full, 0 dirs failed
I20260129 02:59:15.534186 20337 catalog_manager.cc:5654] T a5a05fbcbbcf4916bbc01fd1a5b5e90d P 1c50de89a5a845668696dcb28c3badd6 reported cstate change: term changed from 1 to 2, leader changed from 974b802141bd4f13beffef2aa03f88f9 (127.18.211.2) to 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3). New cstate: current_term: 2 leader_uuid: "1c50de89a5a845668696dcb28c3badd6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } health_report { overall_health: HEALTHY } } }
I20260129 02:59:15.535898 20337 catalog_manager.cc:5654] T 0340229d521b432580ae926f6062e3f7 P 1c50de89a5a845668696dcb28c3badd6 reported cstate change: term changed from 1 to 2, leader changed from 197b880fa39d432a9b472845876bfaf0 (127.18.211.1) to 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3). New cstate: current_term: 2 leader_uuid: "1c50de89a5a845668696dcb28c3badd6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } health_report { overall_health: HEALTHY } } }
I20260129 02:59:15.549839 21757 ts_tablet_manager.cc:933] T add8d4923d1a489e86882c3a25692b95 P 7a9daf5d42a84615a6383cfec9a10608: Initiating tablet copy from peer 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3:37735)
I20260129 02:59:15.577615 21757 tablet_copy_client.cc:323] T add8d4923d1a489e86882c3a25692b95 P 7a9daf5d42a84615a6383cfec9a10608: tablet copy: Beginning tablet copy session from remote peer at address 127.18.211.3:37735
I20260129 02:59:15.579557 20600 tablet_copy_service.cc:140] P 1c50de89a5a845668696dcb28c3badd6: Received BeginTabletCopySession request for tablet add8d4923d1a489e86882c3a25692b95 from peer 7a9daf5d42a84615a6383cfec9a10608 ({username='slave'} at 127.0.0.1:48526)
I20260129 02:59:15.580195 20600 tablet_copy_service.cc:161] P 1c50de89a5a845668696dcb28c3badd6: Beginning new tablet copy session on tablet add8d4923d1a489e86882c3a25692b95 from peer 7a9daf5d42a84615a6383cfec9a10608 at {username='slave'} at 127.0.0.1:48526: session id = 7a9daf5d42a84615a6383cfec9a10608-add8d4923d1a489e86882c3a25692b95
I20260129 02:59:15.582083 21754 tablet_copy_client.cc:806] T 1299723a4476420ab02fd8bfd7aa0645 P 7a9daf5d42a84615a6383cfec9a10608: tablet copy: Starting download of 0 data blocks...
I20260129 02:59:15.582737 21754 tablet_copy_client.cc:670] T 1299723a4476420ab02fd8bfd7aa0645 P 7a9daf5d42a84615a6383cfec9a10608: tablet copy: Starting download of 1 WAL segments...
I20260129 02:59:15.591094 21754 tablet_copy_client.cc:538] T 1299723a4476420ab02fd8bfd7aa0645 P 7a9daf5d42a84615a6383cfec9a10608: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260129 02:59:15.601873 21754 tablet_bootstrap.cc:492] T 1299723a4476420ab02fd8bfd7aa0645 P 7a9daf5d42a84615a6383cfec9a10608: Bootstrap starting.
I20260129 02:59:15.604966 20600 tablet_copy_source_session.cc:215] T add8d4923d1a489e86882c3a25692b95 P 1c50de89a5a845668696dcb28c3badd6: Tablet Copy: opened 0 blocks and 1 log segments
I20260129 02:59:15.608635 21757 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet add8d4923d1a489e86882c3a25692b95. 1 dirs total, 0 dirs full, 0 dirs failed
I20260129 02:59:15.655340 21757 tablet_copy_client.cc:806] T add8d4923d1a489e86882c3a25692b95 P 7a9daf5d42a84615a6383cfec9a10608: tablet copy: Starting download of 0 data blocks...
I20260129 02:59:15.655922 21757 tablet_copy_client.cc:670] T add8d4923d1a489e86882c3a25692b95 P 7a9daf5d42a84615a6383cfec9a10608: tablet copy: Starting download of 1 WAL segments...
I20260129 02:59:15.661022 21754 tablet_bootstrap.cc:492] T 1299723a4476420ab02fd8bfd7aa0645 P 7a9daf5d42a84615a6383cfec9a10608: Bootstrap replayed 1/1 log segments. Stats: ops{read=3 overwritten=0 applied=3 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260129 02:59:15.661978 21754 tablet_bootstrap.cc:492] T 1299723a4476420ab02fd8bfd7aa0645 P 7a9daf5d42a84615a6383cfec9a10608: Bootstrap complete.
I20260129 02:59:15.662670 21754 ts_tablet_manager.cc:1403] T 1299723a4476420ab02fd8bfd7aa0645 P 7a9daf5d42a84615a6383cfec9a10608: Time spent bootstrapping tablet: real 0.061s user 0.016s sys 0.015s
I20260129 02:59:15.665712 21754 raft_consensus.cc:359] T 1299723a4476420ab02fd8bfd7aa0645 P 7a9daf5d42a84615a6383cfec9a10608 [term 2 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } attrs { replace: true } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: NON_VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: true } }
I20260129 02:59:15.666288 21757 tablet_copy_client.cc:538] T add8d4923d1a489e86882c3a25692b95 P 7a9daf5d42a84615a6383cfec9a10608: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260129 02:59:15.666558 21754 raft_consensus.cc:740] T 1299723a4476420ab02fd8bfd7aa0645 P 7a9daf5d42a84615a6383cfec9a10608 [term 2 LEARNER]: Becoming Follower/Learner. State: Replica: 7a9daf5d42a84615a6383cfec9a10608, State: Initialized, Role: LEARNER
I20260129 02:59:15.667245 21754 consensus_queue.cc:260] T 1299723a4476420ab02fd8bfd7aa0645 P 7a9daf5d42a84615a6383cfec9a10608 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 3, Last appended: 2.3, Last appended by leader: 3, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } attrs { replace: true } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: NON_VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: true } }
I20260129 02:59:15.676501 21757 tablet_bootstrap.cc:492] T add8d4923d1a489e86882c3a25692b95 P 7a9daf5d42a84615a6383cfec9a10608: Bootstrap starting.
I20260129 02:59:15.681900 21754 ts_tablet_manager.cc:1434] T 1299723a4476420ab02fd8bfd7aa0645 P 7a9daf5d42a84615a6383cfec9a10608: Time spent starting tablet: real 0.019s user 0.008s sys 0.000s
I20260129 02:59:15.683100 21760 ts_tablet_manager.cc:933] T 3ccac82a01a2445fa5b0ce6c0fcf2295 P 7a9daf5d42a84615a6383cfec9a10608: Initiating tablet copy from peer 197b880fa39d432a9b472845876bfaf0 (127.18.211.1:42027)
I20260129 02:59:15.692533 20450 tablet_copy_service.cc:342] P 197b880fa39d432a9b472845876bfaf0: Request end of tablet copy session 7a9daf5d42a84615a6383cfec9a10608-1299723a4476420ab02fd8bfd7aa0645 received from {username='slave'} at 127.0.0.1:48356
I20260129 02:59:15.693055 20450 tablet_copy_service.cc:434] P 197b880fa39d432a9b472845876bfaf0: ending tablet copy session 7a9daf5d42a84615a6383cfec9a10608-1299723a4476420ab02fd8bfd7aa0645 on tablet 1299723a4476420ab02fd8bfd7aa0645 with peer 7a9daf5d42a84615a6383cfec9a10608
I20260129 02:59:15.699798 21757 tablet_bootstrap.cc:492] T add8d4923d1a489e86882c3a25692b95 P 7a9daf5d42a84615a6383cfec9a10608: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260129 02:59:15.700781 21757 tablet_bootstrap.cc:492] T add8d4923d1a489e86882c3a25692b95 P 7a9daf5d42a84615a6383cfec9a10608: Bootstrap complete.
I20260129 02:59:15.701440 21757 ts_tablet_manager.cc:1403] T add8d4923d1a489e86882c3a25692b95 P 7a9daf5d42a84615a6383cfec9a10608: Time spent bootstrapping tablet: real 0.025s user 0.012s sys 0.010s
I20260129 02:59:15.704316 21757 raft_consensus.cc:359] T add8d4923d1a489e86882c3a25692b95 P 7a9daf5d42a84615a6383cfec9a10608 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } attrs { replace: true } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: NON_VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: true } }
I20260129 02:59:15.705103 21757 raft_consensus.cc:740] T add8d4923d1a489e86882c3a25692b95 P 7a9daf5d42a84615a6383cfec9a10608 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 7a9daf5d42a84615a6383cfec9a10608, State: Initialized, Role: LEARNER
I20260129 02:59:15.705729 21757 consensus_queue.cc:260] T add8d4923d1a489e86882c3a25692b95 P 7a9daf5d42a84615a6383cfec9a10608 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } attrs { replace: true } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: NON_VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: true } }
I20260129 02:59:15.711283 21757 ts_tablet_manager.cc:1434] T add8d4923d1a489e86882c3a25692b95 P 7a9daf5d42a84615a6383cfec9a10608: Time spent starting tablet: real 0.010s user 0.005s sys 0.003s
I20260129 02:59:15.712958 20600 tablet_copy_service.cc:342] P 1c50de89a5a845668696dcb28c3badd6: Request end of tablet copy session 7a9daf5d42a84615a6383cfec9a10608-add8d4923d1a489e86882c3a25692b95 received from {username='slave'} at 127.0.0.1:48526
I20260129 02:59:15.713392 20600 tablet_copy_service.cc:434] P 1c50de89a5a845668696dcb28c3badd6: ending tablet copy session 7a9daf5d42a84615a6383cfec9a10608-add8d4923d1a489e86882c3a25692b95 on tablet add8d4923d1a489e86882c3a25692b95 with peer 7a9daf5d42a84615a6383cfec9a10608
I20260129 02:59:15.714299 21760 tablet_copy_client.cc:323] T 3ccac82a01a2445fa5b0ce6c0fcf2295 P 7a9daf5d42a84615a6383cfec9a10608: tablet copy: Beginning tablet copy session from remote peer at address 127.18.211.1:42027
I20260129 02:59:15.716142 20513 raft_consensus.cc:1275] T 00e79d8f25a34676b1d866eb2068a2b2 P 974b802141bd4f13beffef2aa03f88f9 [term 2 FOLLOWER]: Refusing update from remote peer 7a9daf5d42a84615a6383cfec9a10608: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260129 02:59:15.716305 20450 tablet_copy_service.cc:140] P 197b880fa39d432a9b472845876bfaf0: Received BeginTabletCopySession request for tablet 3ccac82a01a2445fa5b0ce6c0fcf2295 from peer 7a9daf5d42a84615a6383cfec9a10608 ({username='slave'} at 127.0.0.1:48356)
I20260129 02:59:15.716883 20450 tablet_copy_service.cc:161] P 197b880fa39d432a9b472845876bfaf0: Beginning new tablet copy session on tablet 3ccac82a01a2445fa5b0ce6c0fcf2295 from peer 7a9daf5d42a84615a6383cfec9a10608 at {username='slave'} at 127.0.0.1:48356: session id = 7a9daf5d42a84615a6383cfec9a10608-3ccac82a01a2445fa5b0ce6c0fcf2295
I20260129 02:59:15.717846 21700 consensus_queue.cc:1048] T 00e79d8f25a34676b1d866eb2068a2b2 P 7a9daf5d42a84615a6383cfec9a10608 [LEADER]: Connected to new peer: Peer: permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260129 02:59:15.727777 20450 tablet_copy_source_session.cc:215] T 3ccac82a01a2445fa5b0ce6c0fcf2295 P 197b880fa39d432a9b472845876bfaf0: Tablet Copy: opened 0 blocks and 1 log segments
I20260129 02:59:15.731802 21760 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3ccac82a01a2445fa5b0ce6c0fcf2295. 1 dirs total, 0 dirs full, 0 dirs failed
I20260129 02:59:15.767462 20438 raft_consensus.cc:1275] T 00e79d8f25a34676b1d866eb2068a2b2 P 197b880fa39d432a9b472845876bfaf0 [term 2 FOLLOWER]: Refusing update from remote peer 7a9daf5d42a84615a6383cfec9a10608: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260129 02:59:15.775069 21700 consensus_queue.cc:1048] T 00e79d8f25a34676b1d866eb2068a2b2 P 7a9daf5d42a84615a6383cfec9a10608 [LEADER]: Connected to new peer: Peer: permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260129 02:59:15.781519 21760 tablet_copy_client.cc:806] T 3ccac82a01a2445fa5b0ce6c0fcf2295 P 7a9daf5d42a84615a6383cfec9a10608: tablet copy: Starting download of 0 data blocks...
I20260129 02:59:15.782157 21760 tablet_copy_client.cc:670] T 3ccac82a01a2445fa5b0ce6c0fcf2295 P 7a9daf5d42a84615a6383cfec9a10608: tablet copy: Starting download of 1 WAL segments...
I20260129 02:59:15.787815 21760 tablet_copy_client.cc:538] T 3ccac82a01a2445fa5b0ce6c0fcf2295 P 7a9daf5d42a84615a6383cfec9a10608: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260129 02:59:15.798039 21760 tablet_bootstrap.cc:492] T 3ccac82a01a2445fa5b0ce6c0fcf2295 P 7a9daf5d42a84615a6383cfec9a10608: Bootstrap starting.
I20260129 02:59:15.805868 20513 raft_consensus.cc:1275] T c891d2bff4ce4568861bbc992e91fbb0 P 974b802141bd4f13beffef2aa03f88f9 [term 5 FOLLOWER]: Refusing update from remote peer 197b880fa39d432a9b472845876bfaf0: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 5 index: 4. (index mismatch)
I20260129 02:59:15.813371 20513 raft_consensus.cc:1275] T a5a05fbcbbcf4916bbc01fd1a5b5e90d P 974b802141bd4f13beffef2aa03f88f9 [term 2 FOLLOWER]: Refusing update from remote peer 1c50de89a5a845668696dcb28c3badd6: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260129 02:59:15.813699 21662 consensus_queue.cc:1048] T c891d2bff4ce4568861bbc992e91fbb0 P 197b880fa39d432a9b472845876bfaf0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.006s
I20260129 02:59:15.814692 21748 consensus_queue.cc:1048] T a5a05fbcbbcf4916bbc01fd1a5b5e90d P 1c50de89a5a845668696dcb28c3badd6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260129 02:59:15.823501 21760 tablet_bootstrap.cc:492] T 3ccac82a01a2445fa5b0ce6c0fcf2295 P 7a9daf5d42a84615a6383cfec9a10608: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260129 02:59:15.824512 21760 tablet_bootstrap.cc:492] T 3ccac82a01a2445fa5b0ce6c0fcf2295 P 7a9daf5d42a84615a6383cfec9a10608: Bootstrap complete.
I20260129 02:59:15.825201 21760 ts_tablet_manager.cc:1403] T 3ccac82a01a2445fa5b0ce6c0fcf2295 P 7a9daf5d42a84615a6383cfec9a10608: Time spent bootstrapping tablet: real 0.027s user 0.018s sys 0.005s
I20260129 02:59:15.830045 21757 ts_tablet_manager.cc:933] T 96609ac68bdf4993a12e1c23355ca013 P 7a9daf5d42a84615a6383cfec9a10608: Initiating tablet copy from peer 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3:37735)
I20260129 02:59:15.836254 21760 raft_consensus.cc:359] T 3ccac82a01a2445fa5b0ce6c0fcf2295 P 7a9daf5d42a84615a6383cfec9a10608 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } attrs { replace: true } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: NON_VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: true } }
I20260129 02:59:15.840957 21760 raft_consensus.cc:740] T 3ccac82a01a2445fa5b0ce6c0fcf2295 P 7a9daf5d42a84615a6383cfec9a10608 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 7a9daf5d42a84615a6383cfec9a10608, State: Initialized, Role: LEARNER
I20260129 02:59:15.843936 21760 consensus_queue.cc:260] T 3ccac82a01a2445fa5b0ce6c0fcf2295 P 7a9daf5d42a84615a6383cfec9a10608 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } attrs { replace: true } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: NON_VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: true } }
I20260129 02:59:15.865597 20510 raft_consensus.cc:1275] T 4b1876f9a70b49edb3aa74105e66d3b3 P 974b802141bd4f13beffef2aa03f88f9 [term 2 FOLLOWER]: Refusing update from remote peer 197b880fa39d432a9b472845876bfaf0: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260129 02:59:15.867432 21662 consensus_queue.cc:1048] T 4b1876f9a70b49edb3aa74105e66d3b3 P 197b880fa39d432a9b472845876bfaf0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260129 02:59:15.868623 21757 tablet_copy_client.cc:323] T 96609ac68bdf4993a12e1c23355ca013 P 7a9daf5d42a84615a6383cfec9a10608: tablet copy: Beginning tablet copy session from remote peer at address 127.18.211.3:37735
I20260129 02:59:15.870294 20600 tablet_copy_service.cc:140] P 1c50de89a5a845668696dcb28c3badd6: Received BeginTabletCopySession request for tablet 96609ac68bdf4993a12e1c23355ca013 from peer 7a9daf5d42a84615a6383cfec9a10608 ({username='slave'} at 127.0.0.1:48526)
I20260129 02:59:15.870867 20600 tablet_copy_service.cc:161] P 1c50de89a5a845668696dcb28c3badd6: Beginning new tablet copy session on tablet 96609ac68bdf4993a12e1c23355ca013 from peer 7a9daf5d42a84615a6383cfec9a10608 at {username='slave'} at 127.0.0.1:48526: session id = 7a9daf5d42a84615a6383cfec9a10608-96609ac68bdf4993a12e1c23355ca013
I20260129 02:59:15.876389 21134 raft_consensus.cc:1217] T 1299723a4476420ab02fd8bfd7aa0645 P 7a9daf5d42a84615a6383cfec9a10608 [term 2 LEARNER]: Deduplicated request from leader. Original: 2.2->[2.3-2.3] Dedup: 2.3->[]
I20260129 02:59:15.888264 20590 raft_consensus.cc:1275] T c891d2bff4ce4568861bbc992e91fbb0 P 1c50de89a5a845668696dcb28c3badd6 [term 5 FOLLOWER]: Refusing update from remote peer 197b880fa39d432a9b472845876bfaf0: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 5 index: 4. (index mismatch)
I20260129 02:59:15.889498 20437 raft_consensus.cc:1275] T a5a05fbcbbcf4916bbc01fd1a5b5e90d P 197b880fa39d432a9b472845876bfaf0 [term 2 FOLLOWER]: Refusing update from remote peer 1c50de89a5a845668696dcb28c3badd6: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260129 02:59:15.891120 21748 consensus_queue.cc:1048] T a5a05fbcbbcf4916bbc01fd1a5b5e90d P 1c50de89a5a845668696dcb28c3badd6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260129 02:59:15.891574 21666 consensus_queue.cc:1048] T c891d2bff4ce4568861bbc992e91fbb0 P 197b880fa39d432a9b472845876bfaf0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260129 02:59:15.899253 20590 raft_consensus.cc:1275] T 4b1876f9a70b49edb3aa74105e66d3b3 P 1c50de89a5a845668696dcb28c3badd6 [term 2 FOLLOWER]: Refusing update from remote peer 197b880fa39d432a9b472845876bfaf0: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260129 02:59:15.900444 21665 consensus_queue.cc:1048] T 4b1876f9a70b49edb3aa74105e66d3b3 P 197b880fa39d432a9b472845876bfaf0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260129 02:59:15.900349 20600 tablet_copy_source_session.cc:215] T 96609ac68bdf4993a12e1c23355ca013 P 1c50de89a5a845668696dcb28c3badd6: Tablet Copy: opened 0 blocks and 1 log segments
I20260129 02:59:15.905735 21757 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 96609ac68bdf4993a12e1c23355ca013. 1 dirs total, 0 dirs full, 0 dirs failed
I20260129 02:59:15.911099 21760 ts_tablet_manager.cc:1434] T 3ccac82a01a2445fa5b0ce6c0fcf2295 P 7a9daf5d42a84615a6383cfec9a10608: Time spent starting tablet: real 0.086s user 0.009s sys 0.000s
I20260129 02:59:15.916339 21134 raft_consensus.cc:1217] T add8d4923d1a489e86882c3a25692b95 P 7a9daf5d42a84615a6383cfec9a10608 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
W20260129 02:59:16.036887 20407 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.211.2:34517, user_credentials={real_user=slave}} blocked reactor thread for 82143.6us
W20260129 02:59:16.108078 20405 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.211.3:37735, user_credentials={real_user=slave}} blocked reactor thread for 61329us
I20260129 02:59:16.100536 20450 tablet_copy_service.cc:342] P 197b880fa39d432a9b472845876bfaf0: Request end of tablet copy session 7a9daf5d42a84615a6383cfec9a10608-3ccac82a01a2445fa5b0ce6c0fcf2295 received from {username='slave'} at 127.0.0.1:48356
I20260129 02:59:16.255321 20450 tablet_copy_service.cc:434] P 197b880fa39d432a9b472845876bfaf0: ending tablet copy session 7a9daf5d42a84615a6383cfec9a10608-3ccac82a01a2445fa5b0ce6c0fcf2295 on tablet 3ccac82a01a2445fa5b0ce6c0fcf2295 with peer 7a9daf5d42a84615a6383cfec9a10608
I20260129 02:59:16.409584 19276 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
W20260129 02:59:16.455045 20405 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.211.3:37735, user_credentials={real_user=slave}} blocked reactor thread for 343135us
W20260129 02:59:16.462769 20480 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.211.3:37735, user_credentials={real_user=slave}} blocked reactor thread for 294994us
W20260129 02:59:16.487958 21776 log.cc:927] Time spent T 4b1876f9a70b49edb3aa74105e66d3b3 P 1c50de89a5a845668696dcb28c3badd6: Append to log took a long time: real 0.057s user 0.000s sys 0.000s
W20260129 02:59:16.561784 20407 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.211.2:34517, user_credentials={real_user=slave}} blocked reactor thread for 140150us
W20260129 02:59:16.520025 21775 log.cc:927] Time spent T c891d2bff4ce4568861bbc992e91fbb0 P 1c50de89a5a845668696dcb28c3badd6: Append to log took a long time: real 0.162s user 0.001s sys 0.000s
W20260129 02:59:16.582355 20554 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.211.4:40439, user_credentials={real_user=slave}} blocked reactor thread for 49543.3us
W20260129 02:59:16.624081 21100 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.211.3:37735, user_credentials={real_user=slave}} blocked reactor thread for 443572us
W20260129 02:59:16.672732 21774 log.cc:927] Time spent T a5a05fbcbbcf4916bbc01fd1a5b5e90d P 197b880fa39d432a9b472845876bfaf0: Append to log took a long time: real 0.192s user 0.000s sys 0.001s
W20260129 02:59:16.678791 20556 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.547s user 0.013s sys 0.010s
W20260129 02:59:16.698976 20556 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.567s user 0.013s sys 0.010s
W20260129 02:59:16.736330 20479 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.211.1:42027, user_credentials={real_user=slave}} blocked reactor thread for 114767us
I20260129 02:59:16.866967 21757 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.StartTabletCopy from 127.0.0.1:34046 (request call id 243) took 1037 ms. Trace:
I20260129 02:59:16.867762 21757 rpcz_store.cc:276] 0129 02:59:15.829318 (+ 0us) service_pool.cc:168] Inserting onto call queue
0129 02:59:15.829461 (+ 143us) service_pool.cc:225] Handling call
0129 02:59:15.830448 (+ 987us) ts_tablet_manager.cc:934] T 96609ac68bdf4993a12e1c23355ca013 P 7a9daf5d42a84615a6383cfec9a10608: Initiating tablet copy from peer 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3:37735)
0129 02:59:15.904478 (+ 74030us) tablet_copy_client.cc:368] Tablet copy session begun
0129 02:59:16.492360 (+587882us) tablet_metadata.cc:641] Metadata flushed
0129 02:59:16.502337 (+ 9977us) tablet_copy_client.cc:497] Wrote new tablet metadata
0129 02:59:16.863689 (+361352us) tablet_copy_client.cc:418] Wrote new consensus metadata
0129 02:59:16.866336 (+ 2647us) tablet_replica.cc:206] Creating consensus instance
0129 02:59:16.866774 (+ 438us) ts_tablet_manager.cc:989] Replica registered.
0129 02:59:16.866861 (+ 87us) inbound_call.cc:173] Queueing success response
Metrics: {"dns_us":229,"tablet-copy.queue_time_us":143,"thread_start_us":37357,"threads_started":1}
I20260129 02:59:16.921906 21757 tablet_copy_client.cc:806] T 96609ac68bdf4993a12e1c23355ca013 P 7a9daf5d42a84615a6383cfec9a10608: tablet copy: Starting download of 0 data blocks...
I20260129 02:59:16.934926 20590 tablet_service.cc:2066] Received LeaderStepDown RPC: tablet_id: "0340229d521b432580ae926f6062e3f7"
dest_uuid: "1c50de89a5a845668696dcb28c3badd6"
mode: GRACEFUL
new_leader_uuid: "197b880fa39d432a9b472845876bfaf0"
from {username='slave'} at 127.0.0.1:54860
I20260129 02:59:16.936138 20511 raft_consensus.cc:1275] T 0340229d521b432580ae926f6062e3f7 P 974b802141bd4f13beffef2aa03f88f9 [term 2 FOLLOWER]: Refusing update from remote peer 1c50de89a5a845668696dcb28c3badd6: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260129 02:59:16.936302 20590 raft_consensus.cc:606] T 0340229d521b432580ae926f6062e3f7 P 1c50de89a5a845668696dcb28c3badd6 [term 2 LEADER]: Received request to transfer leadership to TS 197b880fa39d432a9b472845876bfaf0
I20260129 02:59:16.931043 21757 tablet_copy_client.cc:670] T 96609ac68bdf4993a12e1c23355ca013 P 7a9daf5d42a84615a6383cfec9a10608: tablet copy: Starting download of 1 WAL segments...
I20260129 02:59:16.956590 21793 raft_consensus.cc:1064] T add8d4923d1a489e86882c3a25692b95 P 1c50de89a5a845668696dcb28c3badd6: attempting to promote NON_VOTER 7a9daf5d42a84615a6383cfec9a10608 to VOTER
I20260129 02:59:16.959007 21777 consensus_queue.cc:1048] T 0340229d521b432580ae926f6062e3f7 P 1c50de89a5a845668696dcb28c3badd6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.004s
I20260129 02:59:16.978266 21700 raft_consensus.cc:493] T 87d5fe48a44a408aaf03e894a67ecb2f P 7a9daf5d42a84615a6383cfec9a10608 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 974b802141bd4f13beffef2aa03f88f9)
I20260129 02:59:16.999424 21700 raft_consensus.cc:515] T 87d5fe48a44a408aaf03e894a67ecb2f P 7a9daf5d42a84615a6383cfec9a10608 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } }
I20260129 02:59:17.006189 20516 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "87d5fe48a44a408aaf03e894a67ecb2f" candidate_uuid: "7a9daf5d42a84615a6383cfec9a10608" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "974b802141bd4f13beffef2aa03f88f9" is_pre_election: true
I20260129 02:59:17.006378 21700 leader_election.cc:290] T 87d5fe48a44a408aaf03e894a67ecb2f P 7a9daf5d42a84615a6383cfec9a10608 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 197b880fa39d432a9b472845876bfaf0 (127.18.211.1:42027), 974b802141bd4f13beffef2aa03f88f9 (127.18.211.2:34517)
I20260129 02:59:17.006987 21814 raft_consensus.cc:493] T 505580c93c694bbb8ea96fdf9b17ab46 P 974b802141bd4f13beffef2aa03f88f9 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 1c50de89a5a845668696dcb28c3badd6)
I20260129 02:59:17.013357 21757 tablet_copy_client.cc:538] T 96609ac68bdf4993a12e1c23355ca013 P 7a9daf5d42a84615a6383cfec9a10608: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260129 02:59:17.030831 21638 raft_consensus.cc:493] T cb0b3f259ebc4310927338a8740e87b8 P 1c50de89a5a845668696dcb28c3badd6 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 974b802141bd4f13beffef2aa03f88f9)
I20260129 02:59:17.031507 21638 raft_consensus.cc:515] T cb0b3f259ebc4310927338a8740e87b8 P 1c50de89a5a845668696dcb28c3badd6 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } }
I20260129 02:59:17.034346 21638 leader_election.cc:290] T cb0b3f259ebc4310927338a8740e87b8 P 1c50de89a5a845668696dcb28c3badd6 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 974b802141bd4f13beffef2aa03f88f9 (127.18.211.2:34517), 7a9daf5d42a84615a6383cfec9a10608 (127.18.211.4:40439)
I20260129 02:59:17.036577 20516 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "cb0b3f259ebc4310927338a8740e87b8" candidate_uuid: "1c50de89a5a845668696dcb28c3badd6" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "974b802141bd4f13beffef2aa03f88f9" is_pre_election: true
I20260129 02:59:17.008764 21814 raft_consensus.cc:515] T 505580c93c694bbb8ea96fdf9b17ab46 P 974b802141bd4f13beffef2aa03f88f9 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } }
I20260129 02:59:17.046461 21814 leader_election.cc:290] T 505580c93c694bbb8ea96fdf9b17ab46 P 974b802141bd4f13beffef2aa03f88f9 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3:37735), 7a9daf5d42a84615a6383cfec9a10608 (127.18.211.4:40439)
I20260129 02:59:17.048460 21134 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "505580c93c694bbb8ea96fdf9b17ab46" candidate_uuid: "974b802141bd4f13beffef2aa03f88f9" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "7a9daf5d42a84615a6383cfec9a10608" is_pre_election: true
I20260129 02:59:17.070837 20582 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "505580c93c694bbb8ea96fdf9b17ab46" candidate_uuid: "974b802141bd4f13beffef2aa03f88f9" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "1c50de89a5a845668696dcb28c3badd6" is_pre_election: true
I20260129 02:59:17.076268 21131 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "cb0b3f259ebc4310927338a8740e87b8" candidate_uuid: "1c50de89a5a845668696dcb28c3badd6" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "7a9daf5d42a84615a6383cfec9a10608" is_pre_election: true
I20260129 02:59:17.079313 21814 raft_consensus.cc:493] T 926a250fa5834e88b96ccecbefbf5086 P 974b802141bd4f13beffef2aa03f88f9 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 1c50de89a5a845668696dcb28c3badd6)
I20260129 02:59:17.079898 21814 raft_consensus.cc:515] T 926a250fa5834e88b96ccecbefbf5086 P 974b802141bd4f13beffef2aa03f88f9 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:59:17.082114 21814 leader_election.cc:290] T 926a250fa5834e88b96ccecbefbf5086 P 974b802141bd4f13beffef2aa03f88f9 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 197b880fa39d432a9b472845876bfaf0 (127.18.211.1:42027), 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3:37735)
I20260129 02:59:17.083670 20480 leader_election.cc:304] T 505580c93c694bbb8ea96fdf9b17ab46 P 974b802141bd4f13beffef2aa03f88f9 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 974b802141bd4f13beffef2aa03f88f9; no voters: 1c50de89a5a845668696dcb28c3badd6, 7a9daf5d42a84615a6383cfec9a10608
I20260129 02:59:17.084722 21799 raft_consensus.cc:2749] T 505580c93c694bbb8ea96fdf9b17ab46 P 974b802141bd4f13beffef2aa03f88f9 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260129 02:59:17.096263 21757 tablet_bootstrap.cc:492] T 96609ac68bdf4993a12e1c23355ca013 P 7a9daf5d42a84615a6383cfec9a10608: Bootstrap starting.
I20260129 02:59:17.096896 21788 raft_consensus.cc:493] T dc2f4cb752cf4d59b5f6bc689f5d8104 P 197b880fa39d432a9b472845876bfaf0 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 1c50de89a5a845668696dcb28c3badd6)
I20260129 02:59:17.097748 21789 raft_consensus.cc:493] T 36ff33be666a4c82954731691dbcf327 P 197b880fa39d432a9b472845876bfaf0 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 974b802141bd4f13beffef2aa03f88f9)
I20260129 02:59:17.098240 21789 raft_consensus.cc:515] T 36ff33be666a4c82954731691dbcf327 P 197b880fa39d432a9b472845876bfaf0 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:59:17.100286 21789 leader_election.cc:290] T 36ff33be666a4c82954731691dbcf327 P 197b880fa39d432a9b472845876bfaf0 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 974b802141bd4f13beffef2aa03f88f9 (127.18.211.2:34517), 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3:37735)
I20260129 02:59:17.097416 21788 raft_consensus.cc:515] T dc2f4cb752cf4d59b5f6bc689f5d8104 P 197b880fa39d432a9b472845876bfaf0 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } }
I20260129 02:59:17.102885 21788 leader_election.cc:290] T dc2f4cb752cf4d59b5f6bc689f5d8104 P 197b880fa39d432a9b472845876bfaf0 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3:37735), 7a9daf5d42a84615a6383cfec9a10608 (127.18.211.4:40439)
I20260129 02:59:17.105763 20582 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "36ff33be666a4c82954731691dbcf327" candidate_uuid: "197b880fa39d432a9b472845876bfaf0" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1c50de89a5a845668696dcb28c3badd6" is_pre_election: true
I20260129 02:59:17.116274 21700 raft_consensus.cc:493] T cb0b3f259ebc4310927338a8740e87b8 P 7a9daf5d42a84615a6383cfec9a10608 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 974b802141bd4f13beffef2aa03f88f9)
I20260129 02:59:17.120085 21700 raft_consensus.cc:515] T cb0b3f259ebc4310927338a8740e87b8 P 7a9daf5d42a84615a6383cfec9a10608 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } }
I20260129 02:59:17.122531 21700 leader_election.cc:290] T cb0b3f259ebc4310927338a8740e87b8 P 7a9daf5d42a84615a6383cfec9a10608 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 974b802141bd4f13beffef2aa03f88f9 (127.18.211.2:34517), 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3:37735)
I20260129 02:59:17.209666 21749 raft_consensus.cc:493] T cb4727386a3e48aca5d71cc6c611c197 P 197b880fa39d432a9b472845876bfaf0 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 974b802141bd4f13beffef2aa03f88f9)
I20260129 02:59:17.210681 21749 raft_consensus.cc:515] T cb4727386a3e48aca5d71cc6c611c197 P 197b880fa39d432a9b472845876bfaf0 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:59:17.213666 21749 leader_election.cc:290] T cb4727386a3e48aca5d71cc6c611c197 P 197b880fa39d432a9b472845876bfaf0 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 974b802141bd4f13beffef2aa03f88f9 (127.18.211.2:34517), 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3:37735)
I20260129 02:59:17.442013 21749 raft_consensus.cc:493] T 87d5fe48a44a408aaf03e894a67ecb2f P 197b880fa39d432a9b472845876bfaf0 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 974b802141bd4f13beffef2aa03f88f9)
I20260129 02:59:17.442829 21749 raft_consensus.cc:515] T 87d5fe48a44a408aaf03e894a67ecb2f P 197b880fa39d432a9b472845876bfaf0 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } }
I20260129 02:59:17.445680 21749 leader_election.cc:290] T 87d5fe48a44a408aaf03e894a67ecb2f P 197b880fa39d432a9b472845876bfaf0 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 974b802141bd4f13beffef2aa03f88f9 (127.18.211.2:34517), 7a9daf5d42a84615a6383cfec9a10608 (127.18.211.4:40439)
I20260129 02:59:17.537405 21821 raft_consensus.cc:493] T a0fcc58d6fb6473cafbd9d34d5195269 P 197b880fa39d432a9b472845876bfaf0 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 974b802141bd4f13beffef2aa03f88f9)
I20260129 02:59:17.537874 21821 raft_consensus.cc:515] T a0fcc58d6fb6473cafbd9d34d5195269 P 197b880fa39d432a9b472845876bfaf0 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:59:17.539544 21821 leader_election.cc:290] T a0fcc58d6fb6473cafbd9d34d5195269 P 197b880fa39d432a9b472845876bfaf0 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 974b802141bd4f13beffef2aa03f88f9 (127.18.211.2:34517), 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3:37735)
I20260129 02:59:17.540536 21821 raft_consensus.cc:493] T 00e79d8f25a34676b1d866eb2068a2b2 P 197b880fa39d432a9b472845876bfaf0 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 7a9daf5d42a84615a6383cfec9a10608)
I20260129 02:59:17.541008 21821 raft_consensus.cc:515] T 00e79d8f25a34676b1d866eb2068a2b2 P 197b880fa39d432a9b472845876bfaf0 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } }
I20260129 02:59:17.543056 21821 leader_election.cc:290] T 00e79d8f25a34676b1d866eb2068a2b2 P 197b880fa39d432a9b472845876bfaf0 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 974b802141bd4f13beffef2aa03f88f9 (127.18.211.2:34517), 7a9daf5d42a84615a6383cfec9a10608 (127.18.211.4:40439)
I20260129 02:59:17.543568 21821 raft_consensus.cc:493] T 2ac659d732314e1288849f08fe5e074e P 197b880fa39d432a9b472845876bfaf0 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 974b802141bd4f13beffef2aa03f88f9)
I20260129 02:59:17.544133 21821 raft_consensus.cc:515] T 2ac659d732314e1288849f08fe5e074e P 197b880fa39d432a9b472845876bfaf0 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } }
I20260129 02:59:17.545883 21821 leader_election.cc:290] T 2ac659d732314e1288849f08fe5e074e P 197b880fa39d432a9b472845876bfaf0 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 974b802141bd4f13beffef2aa03f88f9 (127.18.211.2:34517), 7a9daf5d42a84615a6383cfec9a10608 (127.18.211.4:40439)
I20260129 02:59:17.546432 21821 raft_consensus.cc:493] T a5a05fbcbbcf4916bbc01fd1a5b5e90d P 197b880fa39d432a9b472845876bfaf0 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 1c50de89a5a845668696dcb28c3badd6)
I20260129 02:59:17.546917 21821 raft_consensus.cc:515] T a5a05fbcbbcf4916bbc01fd1a5b5e90d P 197b880fa39d432a9b472845876bfaf0 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:59:17.553702 21821 leader_election.cc:290] T a5a05fbcbbcf4916bbc01fd1a5b5e90d P 197b880fa39d432a9b472845876bfaf0 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 974b802141bd4f13beffef2aa03f88f9 (127.18.211.2:34517), 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3:37735)
I20260129 02:59:17.554352 21821 raft_consensus.cc:493] T 96609ac68bdf4993a12e1c23355ca013 P 197b880fa39d432a9b472845876bfaf0 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 1c50de89a5a845668696dcb28c3badd6)
I20260129 02:59:17.554879 21821 raft_consensus.cc:515] T 96609ac68bdf4993a12e1c23355ca013 P 197b880fa39d432a9b472845876bfaf0 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } attrs { replace: true } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: NON_VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: true } }
I20260129 02:59:17.557132 21821 leader_election.cc:290] T 96609ac68bdf4993a12e1c23355ca013 P 197b880fa39d432a9b472845876bfaf0 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 974b802141bd4f13beffef2aa03f88f9 (127.18.211.2:34517), 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3:37735)
I20260129 02:59:17.601154 21752 raft_consensus.cc:493] T 7d9494f5584d4d5db8b9adf1c8fc888b P 197b880fa39d432a9b472845876bfaf0 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 974b802141bd4f13beffef2aa03f88f9)
I20260129 02:59:17.601794 21752 raft_consensus.cc:515] T 7d9494f5584d4d5db8b9adf1c8fc888b P 197b880fa39d432a9b472845876bfaf0 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:59:17.603946 21752 leader_election.cc:290] T 7d9494f5584d4d5db8b9adf1c8fc888b P 197b880fa39d432a9b472845876bfaf0 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 974b802141bd4f13beffef2aa03f88f9 (127.18.211.2:34517), 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3:37735)
I20260129 02:59:17.678436 21822 raft_consensus.cc:493] T 4f8f3a38f6e24ba4857875181f447442 P 197b880fa39d432a9b472845876bfaf0 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 974b802141bd4f13beffef2aa03f88f9)
I20260129 02:59:17.679126 21822 raft_consensus.cc:515] T 4f8f3a38f6e24ba4857875181f447442 P 197b880fa39d432a9b472845876bfaf0 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:59:17.681612 21822 leader_election.cc:290] T 4f8f3a38f6e24ba4857875181f447442 P 197b880fa39d432a9b472845876bfaf0 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 974b802141bd4f13beffef2aa03f88f9 (127.18.211.2:34517), 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3:37735)
I20260129 02:59:17.686975 21822 raft_consensus.cc:493] T 0340229d521b432580ae926f6062e3f7 P 197b880fa39d432a9b472845876bfaf0 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260129 02:59:17.686870 20441 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "87d5fe48a44a408aaf03e894a67ecb2f" candidate_uuid: "7a9daf5d42a84615a6383cfec9a10608" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "197b880fa39d432a9b472845876bfaf0" is_pre_election: true
I20260129 02:59:17.687785 20441 raft_consensus.cc:2468] T 87d5fe48a44a408aaf03e894a67ecb2f P 197b880fa39d432a9b472845876bfaf0 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7a9daf5d42a84615a6383cfec9a10608 in term 1.
I20260129 02:59:17.687623 21822 raft_consensus.cc:515] T 0340229d521b432580ae926f6062e3f7 P 197b880fa39d432a9b472845876bfaf0 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:59:17.689771 21822 leader_election.cc:290] T 0340229d521b432580ae926f6062e3f7 P 197b880fa39d432a9b472845876bfaf0 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 974b802141bd4f13beffef2aa03f88f9 (127.18.211.2:34517), 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3:37735)
I20260129 02:59:17.725262 21729 raft_consensus.cc:493] T 0678b8b468bd43cdbc3e485900844821 P 974b802141bd4f13beffef2aa03f88f9 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 197b880fa39d432a9b472845876bfaf0)
I20260129 02:59:17.725709 21802 raft_consensus.cc:493] T 75b0b6e82ddc4f92b1fe1d94f9a0b73b P 974b802141bd4f13beffef2aa03f88f9 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 1c50de89a5a845668696dcb28c3badd6)
I20260129 02:59:17.725901 21729 raft_consensus.cc:515] T 0678b8b468bd43cdbc3e485900844821 P 974b802141bd4f13beffef2aa03f88f9 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } }
I20260129 02:59:17.726362 21802 raft_consensus.cc:515] T 75b0b6e82ddc4f92b1fe1d94f9a0b73b P 974b802141bd4f13beffef2aa03f88f9 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } }
I20260129 02:59:17.728403 21729 leader_election.cc:290] T 0678b8b468bd43cdbc3e485900844821 P 974b802141bd4f13beffef2aa03f88f9 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 197b880fa39d432a9b472845876bfaf0 (127.18.211.1:42027), 7a9daf5d42a84615a6383cfec9a10608 (127.18.211.4:40439)
I20260129 02:59:17.729140 21802 leader_election.cc:290] T 75b0b6e82ddc4f92b1fe1d94f9a0b73b P 974b802141bd4f13beffef2aa03f88f9 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3:37735), 7a9daf5d42a84615a6383cfec9a10608 (127.18.211.4:40439)
I20260129 02:59:17.729158 21729 raft_consensus.cc:493] T b189f08e16a74a11980cc55d99359a0a P 974b802141bd4f13beffef2aa03f88f9 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 197b880fa39d432a9b472845876bfaf0)
I20260129 02:59:17.730332 21729 raft_consensus.cc:515] T b189f08e16a74a11980cc55d99359a0a P 974b802141bd4f13beffef2aa03f88f9 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:59:17.731618 20507 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "cb0b3f259ebc4310927338a8740e87b8" candidate_uuid: "7a9daf5d42a84615a6383cfec9a10608" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "974b802141bd4f13beffef2aa03f88f9" is_pre_election: true
I20260129 02:59:17.731983 21729 leader_election.cc:290] T b189f08e16a74a11980cc55d99359a0a P 974b802141bd4f13beffef2aa03f88f9 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 197b880fa39d432a9b472845876bfaf0 (127.18.211.1:42027), 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3:37735)
I20260129 02:59:17.775806 21757 tablet_bootstrap.cc:492] T 96609ac68bdf4993a12e1c23355ca013 P 7a9daf5d42a84615a6383cfec9a10608: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260129 02:59:17.776717 20432 raft_consensus.cc:1275] T 0340229d521b432580ae926f6062e3f7 P 197b880fa39d432a9b472845876bfaf0 [term 2 FOLLOWER]: Refusing update from remote peer 1c50de89a5a845668696dcb28c3badd6: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260129 02:59:17.776973 21757 tablet_bootstrap.cc:492] T 96609ac68bdf4993a12e1c23355ca013 P 7a9daf5d42a84615a6383cfec9a10608: Bootstrap complete.
I20260129 02:59:17.778393 21757 ts_tablet_manager.cc:1403] T 96609ac68bdf4993a12e1c23355ca013 P 7a9daf5d42a84615a6383cfec9a10608: Time spent bootstrapping tablet: real 0.682s user 0.017s sys 0.009s
I20260129 02:59:17.779606 20439 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "926a250fa5834e88b96ccecbefbf5086" candidate_uuid: "974b802141bd4f13beffef2aa03f88f9" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "197b880fa39d432a9b472845876bfaf0" is_pre_election: true
I20260129 02:59:17.782411 21757 raft_consensus.cc:359] T 96609ac68bdf4993a12e1c23355ca013 P 7a9daf5d42a84615a6383cfec9a10608 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } attrs { replace: true } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: NON_VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: true } }
I20260129 02:59:17.783432 21757 raft_consensus.cc:740] T 96609ac68bdf4993a12e1c23355ca013 P 7a9daf5d42a84615a6383cfec9a10608 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 7a9daf5d42a84615a6383cfec9a10608, State: Initialized, Role: LEARNER
I20260129 02:59:17.778187 20433 tablet_service.cc:2066] Received LeaderStepDown RPC: tablet_id: "0678b8b468bd43cdbc3e485900844821"
dest_uuid: "197b880fa39d432a9b472845876bfaf0"
mode: GRACEFUL
new_leader_uuid: "7a9daf5d42a84615a6383cfec9a10608"
from {username='slave'} at 127.0.0.1:57776
I20260129 02:59:17.784104 20433 raft_consensus.cc:606] T 0678b8b468bd43cdbc3e485900844821 P 197b880fa39d432a9b472845876bfaf0 [term 1 LEADER]: Received request to transfer leadership to TS 7a9daf5d42a84615a6383cfec9a10608
I20260129 02:59:17.784212 21757 consensus_queue.cc:260] T 96609ac68bdf4993a12e1c23355ca013 P 7a9daf5d42a84615a6383cfec9a10608 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } attrs { replace: true } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: NON_VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: true } }
I20260129 02:59:17.787199 21757 ts_tablet_manager.cc:1434] T 96609ac68bdf4993a12e1c23355ca013 P 7a9daf5d42a84615a6383cfec9a10608: Time spent starting tablet: real 0.008s user 0.006s sys 0.000s
I20260129 02:59:17.792883 20590 tablet_service.cc:2066] Received LeaderStepDown RPC: tablet_id: "368953c9102643b2a21f99af36b51905"
dest_uuid: "1c50de89a5a845668696dcb28c3badd6"
mode: GRACEFUL
new_leader_uuid: "197b880fa39d432a9b472845876bfaf0"
from {username='slave'} at 127.0.0.1:54860
I20260129 02:59:17.792802 21829 consensus_queue.cc:1048] T 0340229d521b432580ae926f6062e3f7 P 1c50de89a5a845668696dcb28c3badd6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260129 02:59:17.793587 20590 raft_consensus.cc:606] T 368953c9102643b2a21f99af36b51905 P 1c50de89a5a845668696dcb28c3badd6 [term 1 LEADER]: Received request to transfer leadership to TS 197b880fa39d432a9b472845876bfaf0
I20260129 02:59:17.809552 20554 leader_election.cc:304] T cb0b3f259ebc4310927338a8740e87b8 P 1c50de89a5a845668696dcb28c3badd6 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1c50de89a5a845668696dcb28c3badd6; no voters: 7a9daf5d42a84615a6383cfec9a10608, 974b802141bd4f13beffef2aa03f88f9
I20260129 02:59:17.809644 20587 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "926a250fa5834e88b96ccecbefbf5086" candidate_uuid: "974b802141bd4f13beffef2aa03f88f9" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "1c50de89a5a845668696dcb28c3badd6" is_pre_election: true
I20260129 02:59:17.811385 21829 raft_consensus.cc:2749] T cb0b3f259ebc4310927338a8740e87b8 P 1c50de89a5a845668696dcb28c3badd6 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260129 02:59:17.821076 20581 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "75b0b6e82ddc4f92b1fe1d94f9a0b73b" candidate_uuid: "974b802141bd4f13beffef2aa03f88f9" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "1c50de89a5a845668696dcb28c3badd6" is_pre_election: true
I20260129 02:59:17.825042 21684 raft_consensus.cc:993] T 368953c9102643b2a21f99af36b51905 P 1c50de89a5a845668696dcb28c3badd6: : Instructing follower 197b880fa39d432a9b472845876bfaf0 to start an election
I20260129 02:59:17.826117 21811 raft_consensus.cc:1081] T 368953c9102643b2a21f99af36b51905 P 1c50de89a5a845668696dcb28c3badd6 [term 1 LEADER]: Signalling peer 197b880fa39d432a9b472845876bfaf0 to start an election
I20260129 02:59:17.823609 20581 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b189f08e16a74a11980cc55d99359a0a" candidate_uuid: "974b802141bd4f13beffef2aa03f88f9" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "1c50de89a5a845668696dcb28c3badd6" is_pre_election: true
I20260129 02:59:17.828421 20433 tablet_service.cc:2038] Received Run Leader Election RPC: tablet_id: "368953c9102643b2a21f99af36b51905"
dest_uuid: "197b880fa39d432a9b472845876bfaf0"
from {username='slave'} at 127.0.0.1:55676
I20260129 02:59:17.829077 20433 raft_consensus.cc:493] T 368953c9102643b2a21f99af36b51905 P 197b880fa39d432a9b472845876bfaf0 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260129 02:59:17.829496 20433 raft_consensus.cc:3060] T 368953c9102643b2a21f99af36b51905 P 197b880fa39d432a9b472845876bfaf0 [term 1 FOLLOWER]: Advancing to term 2
I20260129 02:59:17.836843 21796 raft_consensus.cc:493] T a8479d1fe0bd41e5946186e3a6ca8c7a P 974b802141bd4f13beffef2aa03f88f9 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 197b880fa39d432a9b472845876bfaf0)
I20260129 02:59:17.837466 21796 raft_consensus.cc:515] T a8479d1fe0bd41e5946186e3a6ca8c7a P 974b802141bd4f13beffef2aa03f88f9 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:59:17.839975 20514 tablet_service.cc:2066] Received LeaderStepDown RPC: tablet_id: "cedce331e1584ac8b636c9b8a72b2ba0"
dest_uuid: "974b802141bd4f13beffef2aa03f88f9"
mode: GRACEFUL
new_leader_uuid: "1c50de89a5a845668696dcb28c3badd6"
from {username='slave'} at 127.0.0.1:52008
I20260129 02:59:17.840606 20514 raft_consensus.cc:606] T cedce331e1584ac8b636c9b8a72b2ba0 P 974b802141bd4f13beffef2aa03f88f9 [term 1 LEADER]: Received request to transfer leadership to TS 1c50de89a5a845668696dcb28c3badd6
I20260129 02:59:17.841701 20513 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "36ff33be666a4c82954731691dbcf327" candidate_uuid: "197b880fa39d432a9b472845876bfaf0" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "974b802141bd4f13beffef2aa03f88f9" is_pre_election: true
I20260129 02:59:17.860071 21811 raft_consensus.cc:993] T 0340229d521b432580ae926f6062e3f7 P 1c50de89a5a845668696dcb28c3badd6: : Instructing follower 197b880fa39d432a9b472845876bfaf0 to start an election
I20260129 02:59:17.860630 21829 raft_consensus.cc:1081] T 0340229d521b432580ae926f6062e3f7 P 1c50de89a5a845668696dcb28c3badd6 [term 2 LEADER]: Signalling peer 197b880fa39d432a9b472845876bfaf0 to start an election
I20260129 02:59:17.862340 20432 tablet_service.cc:2038] Received Run Leader Election RPC: tablet_id: "0340229d521b432580ae926f6062e3f7"
dest_uuid: "197b880fa39d432a9b472845876bfaf0"
from {username='slave'} at 127.0.0.1:55676
I20260129 02:59:17.862908 20432 raft_consensus.cc:493] T 0340229d521b432580ae926f6062e3f7 P 197b880fa39d432a9b472845876bfaf0 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20260129 02:59:17.863261 20432 raft_consensus.cc:3060] T 0340229d521b432580ae926f6062e3f7 P 197b880fa39d432a9b472845876bfaf0 [term 2 FOLLOWER]: Advancing to term 3
I20260129 02:59:17.865478 20581 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "cb0b3f259ebc4310927338a8740e87b8" candidate_uuid: "7a9daf5d42a84615a6383cfec9a10608" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "1c50de89a5a845668696dcb28c3badd6" is_pre_election: true
I20260129 02:59:17.869917 20590 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a8479d1fe0bd41e5946186e3a6ca8c7a" candidate_uuid: "974b802141bd4f13beffef2aa03f88f9" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1c50de89a5a845668696dcb28c3badd6" is_pre_election: true
I20260129 02:59:17.873143 21136 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "dc2f4cb752cf4d59b5f6bc689f5d8104" candidate_uuid: "197b880fa39d432a9b472845876bfaf0" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "7a9daf5d42a84615a6383cfec9a10608" is_pre_election: true
I20260129 02:59:17.874831 20511 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "cb4727386a3e48aca5d71cc6c611c197" candidate_uuid: "197b880fa39d432a9b472845876bfaf0" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "974b802141bd4f13beffef2aa03f88f9" is_pre_election: true
I20260129 02:59:17.876121 20514 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2ac659d732314e1288849f08fe5e074e" candidate_uuid: "197b880fa39d432a9b472845876bfaf0" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "974b802141bd4f13beffef2aa03f88f9" is_pre_election: true
I20260129 02:59:17.876559 20512 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "87d5fe48a44a408aaf03e894a67ecb2f" candidate_uuid: "197b880fa39d432a9b472845876bfaf0" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "974b802141bd4f13beffef2aa03f88f9" is_pre_election: true
I20260129 02:59:17.876139 20513 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00e79d8f25a34676b1d866eb2068a2b2" candidate_uuid: "197b880fa39d432a9b472845876bfaf0" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "974b802141bd4f13beffef2aa03f88f9" is_pre_election: true
I20260129 02:59:17.839587 21796 leader_election.cc:290] T a8479d1fe0bd41e5946186e3a6ca8c7a P 974b802141bd4f13beffef2aa03f88f9 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 197b880fa39d432a9b472845876bfaf0 (127.18.211.1:42027), 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3:37735)
I20260129 02:59:17.877817 20509 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a0fcc58d6fb6473cafbd9d34d5195269" candidate_uuid: "197b880fa39d432a9b472845876bfaf0" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "974b802141bd4f13beffef2aa03f88f9" is_pre_election: true
I20260129 02:59:17.878130 20514 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a5a05fbcbbcf4916bbc01fd1a5b5e90d" candidate_uuid: "197b880fa39d432a9b472845876bfaf0" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "974b802141bd4f13beffef2aa03f88f9" is_pre_election: true
I20260129 02:59:17.879066 20512 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "96609ac68bdf4993a12e1c23355ca013" candidate_uuid: "197b880fa39d432a9b472845876bfaf0" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "974b802141bd4f13beffef2aa03f88f9" is_pre_election: true
I20260129 02:59:17.879658 20512 raft_consensus.cc:2468] T 96609ac68bdf4993a12e1c23355ca013 P 974b802141bd4f13beffef2aa03f88f9 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 197b880fa39d432a9b472845876bfaf0 in term 1.
I20260129 02:59:17.880196 20510 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7d9494f5584d4d5db8b9adf1c8fc888b" candidate_uuid: "197b880fa39d432a9b472845876bfaf0" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "974b802141bd4f13beffef2aa03f88f9" is_pre_election: true
I20260129 02:59:17.881580 20510 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4f8f3a38f6e24ba4857875181f447442" candidate_uuid: "197b880fa39d432a9b472845876bfaf0" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "974b802141bd4f13beffef2aa03f88f9" is_pre_election: true
I20260129 02:59:17.884164 20510 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0340229d521b432580ae926f6062e3f7" candidate_uuid: "197b880fa39d432a9b472845876bfaf0" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "974b802141bd4f13beffef2aa03f88f9" is_pre_election: true
I20260129 02:59:17.885749 20407 leader_election.cc:304] T 36ff33be666a4c82954731691dbcf327 P 197b880fa39d432a9b472845876bfaf0 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 197b880fa39d432a9b472845876bfaf0; no voters: 1c50de89a5a845668696dcb28c3badd6, 974b802141bd4f13beffef2aa03f88f9
I20260129 02:59:17.887154 20437 tablet_service.cc:2066] Received LeaderStepDown RPC: tablet_id: "fcddab6bca684adf8ea032289a00f90e"
dest_uuid: "197b880fa39d432a9b472845876bfaf0"
mode: GRACEFUL
new_leader_uuid: "1c50de89a5a845668696dcb28c3badd6"
from {username='slave'} at 127.0.0.1:57776
I20260129 02:59:17.887766 20437 raft_consensus.cc:606] T fcddab6bca684adf8ea032289a00f90e P 197b880fa39d432a9b472845876bfaf0 [term 3 LEADER]: Received request to transfer leadership to TS 1c50de89a5a845668696dcb28c3badd6
I20260129 02:59:17.888847 21816 raft_consensus.cc:2749] T 36ff33be666a4c82954731691dbcf327 P 197b880fa39d432a9b472845876bfaf0 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260129 02:59:17.897775 19276 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 5
I20260129 02:59:17.905431 20407 leader_election.cc:304] T 96609ac68bdf4993a12e1c23355ca013 P 197b880fa39d432a9b472845876bfaf0 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 197b880fa39d432a9b472845876bfaf0, 974b802141bd4f13beffef2aa03f88f9; no voters:
I20260129 02:59:17.905468 20433 raft_consensus.cc:515] T 368953c9102643b2a21f99af36b51905 P 197b880fa39d432a9b472845876bfaf0 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:59:17.906679 21788 raft_consensus.cc:2804] T 96609ac68bdf4993a12e1c23355ca013 P 197b880fa39d432a9b472845876bfaf0 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260129 02:59:17.910176 21788 raft_consensus.cc:493] T 96609ac68bdf4993a12e1c23355ca013 P 197b880fa39d432a9b472845876bfaf0 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 1c50de89a5a845668696dcb28c3badd6)
I20260129 02:59:17.910542 20433 leader_election.cc:290] T 368953c9102643b2a21f99af36b51905 P 197b880fa39d432a9b472845876bfaf0 [CANDIDATE]: Term 2 election: Requested vote from peers 974b802141bd4f13beffef2aa03f88f9 (127.18.211.2:34517), 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3:37735)
I20260129 02:59:17.910601 21788 raft_consensus.cc:3060] T 96609ac68bdf4993a12e1c23355ca013 P 197b880fa39d432a9b472845876bfaf0 [term 1 FOLLOWER]: Advancing to term 2
I20260129 02:59:17.914937 21793 consensus_queue.cc:237] T add8d4923d1a489e86882c3a25692b95 P 1c50de89a5a845668696dcb28c3badd6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } attrs { replace: true } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } }
I20260129 02:59:17.923884 19276 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20260129 02:59:17.924438 20439 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b189f08e16a74a11980cc55d99359a0a" candidate_uuid: "974b802141bd4f13beffef2aa03f88f9" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "197b880fa39d432a9b472845876bfaf0" is_pre_election: true
I20260129 02:59:17.924758 19276 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260129 02:59:17.925875 20435 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a8479d1fe0bd41e5946186e3a6ca8c7a" candidate_uuid: "974b802141bd4f13beffef2aa03f88f9" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "197b880fa39d432a9b472845876bfaf0" is_pre_election: true
I20260129 02:59:17.895133 20432 raft_consensus.cc:515] T 0340229d521b432580ae926f6062e3f7 P 197b880fa39d432a9b472845876bfaf0 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:59:17.934223 20437 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0678b8b468bd43cdbc3e485900844821" candidate_uuid: "974b802141bd4f13beffef2aa03f88f9" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "197b880fa39d432a9b472845876bfaf0" is_pre_election: true
I20260129 02:59:17.944481 20432 leader_election.cc:290] T 0340229d521b432580ae926f6062e3f7 P 197b880fa39d432a9b472845876bfaf0 [CANDIDATE]: Term 3 election: Requested vote from peers 974b802141bd4f13beffef2aa03f88f9 (127.18.211.2:34517), 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3:37735)
I20260129 02:59:17.945940 20508 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0340229d521b432580ae926f6062e3f7" candidate_uuid: "197b880fa39d432a9b472845876bfaf0" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "974b802141bd4f13beffef2aa03f88f9"
I20260129 02:59:17.946499 20508 raft_consensus.cc:3060] T 0340229d521b432580ae926f6062e3f7 P 974b802141bd4f13beffef2aa03f88f9 [term 2 FOLLOWER]: Advancing to term 3
I20260129 02:59:17.947988 20515 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "368953c9102643b2a21f99af36b51905" candidate_uuid: "197b880fa39d432a9b472845876bfaf0" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "974b802141bd4f13beffef2aa03f88f9"
I20260129 02:59:17.948555 20515 raft_consensus.cc:3060] T 368953c9102643b2a21f99af36b51905 P 974b802141bd4f13beffef2aa03f88f9 [term 1 FOLLOWER]: Advancing to term 2
I20260129 02:59:17.959782 20508 raft_consensus.cc:2468] T 0340229d521b432580ae926f6062e3f7 P 974b802141bd4f13beffef2aa03f88f9 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 197b880fa39d432a9b472845876bfaf0 in term 3.
I20260129 02:59:17.961234 20407 leader_election.cc:304] T 0340229d521b432580ae926f6062e3f7 P 197b880fa39d432a9b472845876bfaf0 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 197b880fa39d432a9b472845876bfaf0, 974b802141bd4f13beffef2aa03f88f9; no voters:
I20260129 02:59:17.965428 21816 raft_consensus.cc:2804] T 0340229d521b432580ae926f6062e3f7 P 197b880fa39d432a9b472845876bfaf0 [term 3 FOLLOWER]: Leader election won for term 3
I20260129 02:59:17.965932 21816 raft_consensus.cc:697] T 0340229d521b432580ae926f6062e3f7 P 197b880fa39d432a9b472845876bfaf0 [term 3 LEADER]: Becoming Leader. State: Replica: 197b880fa39d432a9b472845876bfaf0, State: Running, Role: LEADER
I20260129 02:59:17.966737 21816 consensus_queue.cc:237] T 0340229d521b432580ae926f6062e3f7 P 197b880fa39d432a9b472845876bfaf0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:59:17.969192 20582 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "dc2f4cb752cf4d59b5f6bc689f5d8104" candidate_uuid: "197b880fa39d432a9b472845876bfaf0" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "1c50de89a5a845668696dcb28c3badd6" is_pre_election: true
I20260129 02:59:17.982071 21777 raft_consensus.cc:493] T c891d2bff4ce4568861bbc992e91fbb0 P 1c50de89a5a845668696dcb28c3badd6 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 197b880fa39d432a9b472845876bfaf0)
I20260129 02:59:17.983328 20515 raft_consensus.cc:2468] T 368953c9102643b2a21f99af36b51905 P 974b802141bd4f13beffef2aa03f88f9 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 197b880fa39d432a9b472845876bfaf0 in term 2.
I20260129 02:59:17.985265 20407 leader_election.cc:304] T 368953c9102643b2a21f99af36b51905 P 197b880fa39d432a9b472845876bfaf0 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 197b880fa39d432a9b472845876bfaf0, 974b802141bd4f13beffef2aa03f88f9; no voters:
I20260129 02:59:18.008692 20508 raft_consensus.cc:1240] T 0340229d521b432580ae926f6062e3f7 P 974b802141bd4f13beffef2aa03f88f9 [term 3 FOLLOWER]: Rejecting Update request from peer 1c50de89a5a845668696dcb28c3badd6 for earlier term 2. Current term is 3. Ops: []
W20260129 02:59:18.009826 20479 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.211.1:42027, user_credentials={real_user=slave}} blocked reactor thread for 115110us
I20260129 02:59:17.987087 21638 raft_consensus.cc:493] T fcddab6bca684adf8ea032289a00f90e P 1c50de89a5a845668696dcb28c3badd6 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 197b880fa39d432a9b472845876bfaf0)
I20260129 02:59:18.015512 21638 raft_consensus.cc:515] T fcddab6bca684adf8ea032289a00f90e P 1c50de89a5a845668696dcb28c3badd6 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:59:18.017122 21099 leader_election.cc:304] T 87d5fe48a44a408aaf03e894a67ecb2f P 7a9daf5d42a84615a6383cfec9a10608 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 197b880fa39d432a9b472845876bfaf0, 7a9daf5d42a84615a6383cfec9a10608; no voters: 974b802141bd4f13beffef2aa03f88f9
I20260129 02:59:18.017827 21638 leader_election.cc:290] T fcddab6bca684adf8ea032289a00f90e P 1c50de89a5a845668696dcb28c3badd6 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 197b880fa39d432a9b472845876bfaf0 (127.18.211.1:42027), 974b802141bd4f13beffef2aa03f88f9 (127.18.211.2:34517)
I20260129 02:59:18.019403 21836 raft_consensus.cc:2804] T 87d5fe48a44a408aaf03e894a67ecb2f P 7a9daf5d42a84615a6383cfec9a10608 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260129 02:59:18.019833 21836 raft_consensus.cc:493] T 87d5fe48a44a408aaf03e894a67ecb2f P 7a9daf5d42a84615a6383cfec9a10608 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 974b802141bd4f13beffef2aa03f88f9)
I20260129 02:59:18.020191 21836 raft_consensus.cc:3060] T 87d5fe48a44a408aaf03e894a67ecb2f P 7a9daf5d42a84615a6383cfec9a10608 [term 1 FOLLOWER]: Advancing to term 2
I20260129 02:59:17.982625 21777 raft_consensus.cc:515] T c891d2bff4ce4568861bbc992e91fbb0 P 1c50de89a5a845668696dcb28c3badd6 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:59:18.023099 21777 leader_election.cc:290] T c891d2bff4ce4568861bbc992e91fbb0 P 1c50de89a5a845668696dcb28c3badd6 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 197b880fa39d432a9b472845876bfaf0 (127.18.211.1:42027), 974b802141bd4f13beffef2aa03f88f9 (127.18.211.2:34517)
I20260129 02:59:18.034029 20437 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c891d2bff4ce4568861bbc992e91fbb0" candidate_uuid: "1c50de89a5a845668696dcb28c3badd6" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "197b880fa39d432a9b472845876bfaf0" is_pre_election: true
I20260129 02:59:18.049415 21785 raft_consensus.cc:993] T cedce331e1584ac8b636c9b8a72b2ba0 P 974b802141bd4f13beffef2aa03f88f9: : Instructing follower 1c50de89a5a845668696dcb28c3badd6 to start an election
I20260129 02:59:18.121666 21785 raft_consensus.cc:1081] T cedce331e1584ac8b636c9b8a72b2ba0 P 974b802141bd4f13beffef2aa03f88f9 [term 1 LEADER]: Signalling peer 1c50de89a5a845668696dcb28c3badd6 to start an election
I20260129 02:59:18.049736 20432 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fcddab6bca684adf8ea032289a00f90e" candidate_uuid: "1c50de89a5a845668696dcb28c3badd6" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "197b880fa39d432a9b472845876bfaf0" is_pre_election: true
I20260129 02:59:18.123821 20588 tablet_service.cc:2038] Received Run Leader Election RPC: tablet_id: "cedce331e1584ac8b636c9b8a72b2ba0"
dest_uuid: "1c50de89a5a845668696dcb28c3badd6"
from {username='slave'} at 127.0.0.1:35752
I20260129 02:59:18.124502 21135 raft_consensus.cc:1275] T add8d4923d1a489e86882c3a25692b95 P 7a9daf5d42a84615a6383cfec9a10608 [term 1 LEARNER]: Refusing update from remote peer 1c50de89a5a845668696dcb28c3badd6: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260129 02:59:18.124511 20588 raft_consensus.cc:493] T cedce331e1584ac8b636c9b8a72b2ba0 P 1c50de89a5a845668696dcb28c3badd6 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260129 02:59:18.125305 20588 raft_consensus.cc:3060] T cedce331e1584ac8b636c9b8a72b2ba0 P 1c50de89a5a845668696dcb28c3badd6 [term 1 FOLLOWER]: Advancing to term 2
I20260129 02:59:18.128652 21134 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "75b0b6e82ddc4f92b1fe1d94f9a0b73b" candidate_uuid: "974b802141bd4f13beffef2aa03f88f9" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "7a9daf5d42a84615a6383cfec9a10608" is_pre_election: true
I20260129 02:59:18.131368 21788 raft_consensus.cc:515] T 96609ac68bdf4993a12e1c23355ca013 P 197b880fa39d432a9b472845876bfaf0 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } attrs { replace: true } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: NON_VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: true } }
I20260129 02:59:18.132521 21835 raft_consensus.cc:493] T febffe4a8b394368b69b1d164308fb9a P 7a9daf5d42a84615a6383cfec9a10608 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 197b880fa39d432a9b472845876bfaf0)
I20260129 02:59:18.132651 21100 leader_election.cc:304] T cb0b3f259ebc4310927338a8740e87b8 P 7a9daf5d42a84615a6383cfec9a10608 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7a9daf5d42a84615a6383cfec9a10608; no voters: 1c50de89a5a845668696dcb28c3badd6, 974b802141bd4f13beffef2aa03f88f9
W20260129 02:59:18.108492 20407 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.211.2:34517, user_credentials={real_user=slave}} blocked reactor thread for 83969us
I20260129 02:59:18.048051 21778 raft_consensus.cc:1027] T add8d4923d1a489e86882c3a25692b95 P 1c50de89a5a845668696dcb28c3badd6 [term 1 LEADER]: attempt to promote peer 7a9daf5d42a84615a6383cfec9a10608: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20260129 02:59:18.108621 20479 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.211.1:42027, user_credentials={real_user=slave}} blocked reactor thread for 83759.5us
I20260129 02:59:18.134184 21134 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0678b8b468bd43cdbc3e485900844821" candidate_uuid: "974b802141bd4f13beffef2aa03f88f9" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "7a9daf5d42a84615a6383cfec9a10608" is_pre_election: true
I20260129 02:59:18.134717 21838 raft_consensus.cc:2749] T cb0b3f259ebc4310927338a8740e87b8 P 7a9daf5d42a84615a6383cfec9a10608 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260129 02:59:18.137049 20479 leader_election.cc:304] T 926a250fa5834e88b96ccecbefbf5086 P 974b802141bd4f13beffef2aa03f88f9 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 974b802141bd4f13beffef2aa03f88f9; no voters: 197b880fa39d432a9b472845876bfaf0, 1c50de89a5a845668696dcb28c3badd6
I20260129 02:59:18.141714 21824 raft_consensus.cc:2749] T 926a250fa5834e88b96ccecbefbf5086 P 974b802141bd4f13beffef2aa03f88f9 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260129 02:59:18.144901 20479 leader_election.cc:304] T b189f08e16a74a11980cc55d99359a0a P 974b802141bd4f13beffef2aa03f88f9 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 974b802141bd4f13beffef2aa03f88f9; no voters: 197b880fa39d432a9b472845876bfaf0, 1c50de89a5a845668696dcb28c3badd6
I20260129 02:59:18.145851 21729 raft_consensus.cc:2749] T b189f08e16a74a11980cc55d99359a0a P 974b802141bd4f13beffef2aa03f88f9 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260129 02:59:18.146311 20479 leader_election.cc:304] T a8479d1fe0bd41e5946186e3a6ca8c7a P 974b802141bd4f13beffef2aa03f88f9 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 974b802141bd4f13beffef2aa03f88f9; no voters: 197b880fa39d432a9b472845876bfaf0, 1c50de89a5a845668696dcb28c3badd6
I20260129 02:59:18.147249 21729 raft_consensus.cc:2749] T a8479d1fe0bd41e5946186e3a6ca8c7a P 974b802141bd4f13beffef2aa03f88f9 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20260129 02:59:18.112142 20553 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.211.1:42027, user_credentials={real_user=slave}} blocked reactor thread for 64638us
I20260129 02:59:18.156062 20432 raft_consensus.cc:1275] T add8d4923d1a489e86882c3a25692b95 P 197b880fa39d432a9b472845876bfaf0 [term 1 FOLLOWER]: Refusing update from remote peer 1c50de89a5a845668696dcb28c3badd6: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260129 02:59:18.160651 21133 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00e79d8f25a34676b1d866eb2068a2b2" candidate_uuid: "197b880fa39d432a9b472845876bfaf0" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "7a9daf5d42a84615a6383cfec9a10608" is_pre_election: true
I20260129 02:59:18.160808 21135 raft_consensus.cc:1217] T 3ccac82a01a2445fa5b0ce6c0fcf2295 P 7a9daf5d42a84615a6383cfec9a10608 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20260129 02:59:18.161079 21638 consensus_queue.cc:1048] T add8d4923d1a489e86882c3a25692b95 P 1c50de89a5a845668696dcb28c3badd6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260129 02:59:18.162161 21134 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "87d5fe48a44a408aaf03e894a67ecb2f" candidate_uuid: "197b880fa39d432a9b472845876bfaf0" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "7a9daf5d42a84615a6383cfec9a10608" is_pre_election: true
I20260129 02:59:18.164857 21130 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2ac659d732314e1288849f08fe5e074e" candidate_uuid: "197b880fa39d432a9b472845876bfaf0" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "7a9daf5d42a84615a6383cfec9a10608" is_pre_election: true
I20260129 02:59:18.174064 20480 leader_election.cc:304] T 75b0b6e82ddc4f92b1fe1d94f9a0b73b P 974b802141bd4f13beffef2aa03f88f9 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 974b802141bd4f13beffef2aa03f88f9; no voters: 1c50de89a5a845668696dcb28c3badd6, 7a9daf5d42a84615a6383cfec9a10608
I20260129 02:59:18.175454 21837 raft_consensus.cc:2749] T 75b0b6e82ddc4f92b1fe1d94f9a0b73b P 974b802141bd4f13beffef2aa03f88f9 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260129 02:59:18.176702 20480 leader_election.cc:304] T 0678b8b468bd43cdbc3e485900844821 P 974b802141bd4f13beffef2aa03f88f9 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 974b802141bd4f13beffef2aa03f88f9; no voters: 197b880fa39d432a9b472845876bfaf0, 7a9daf5d42a84615a6383cfec9a10608
I20260129 02:59:18.115065 20600 tablet_copy_service.cc:342] P 1c50de89a5a845668696dcb28c3badd6: Request end of tablet copy session 7a9daf5d42a84615a6383cfec9a10608-96609ac68bdf4993a12e1c23355ca013 received from {username='slave'} at 127.0.0.1:48526
W20260129 02:59:18.118470 20556 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.211.2:34517, user_credentials={real_user=slave}} blocked reactor thread for 80910us
I20260129 02:59:18.178148 20600 tablet_copy_service.cc:434] P 1c50de89a5a845668696dcb28c3badd6: ending tablet copy session 7a9daf5d42a84615a6383cfec9a10608-96609ac68bdf4993a12e1c23355ca013 on tablet 96609ac68bdf4993a12e1c23355ca013 with peer 7a9daf5d42a84615a6383cfec9a10608
I20260129 02:59:18.178658 21837 raft_consensus.cc:2749] T 0678b8b468bd43cdbc3e485900844821 P 974b802141bd4f13beffef2aa03f88f9 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260129 02:59:18.179215 21684 consensus_queue.cc:1059] T 0340229d521b432580ae926f6062e3f7 P 1c50de89a5a845668696dcb28c3badd6 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 }, Status: INVALID_TERM, Last received: 2.2, Next index: 3, Last known committed idx: 1, Time since last communication: 0.000s
I20260129 02:59:18.183782 20583 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a0fcc58d6fb6473cafbd9d34d5195269" candidate_uuid: "197b880fa39d432a9b472845876bfaf0" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1c50de89a5a845668696dcb28c3badd6" is_pre_election: true
I20260129 02:59:18.184176 21748 raft_consensus.cc:3055] T 0340229d521b432580ae926f6062e3f7 P 1c50de89a5a845668696dcb28c3badd6 [term 2 LEADER]: Stepping down as leader of term 2
I20260129 02:59:18.184618 21748 raft_consensus.cc:740] T 0340229d521b432580ae926f6062e3f7 P 1c50de89a5a845668696dcb28c3badd6 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 1c50de89a5a845668696dcb28c3badd6, State: Running, Role: LEADER
I20260129 02:59:18.133545 21788 leader_election.cc:290] T 96609ac68bdf4993a12e1c23355ca013 P 197b880fa39d432a9b472845876bfaf0 [CANDIDATE]: Term 2 election: Requested vote from peers 974b802141bd4f13beffef2aa03f88f9 (127.18.211.2:34517), 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3:37735)
I20260129 02:59:18.133169 21835 raft_consensus.cc:515] T febffe4a8b394368b69b1d164308fb9a P 7a9daf5d42a84615a6383cfec9a10608 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } }
I20260129 02:59:18.176733 21778 consensus_queue.cc:1048] T add8d4923d1a489e86882c3a25692b95 P 1c50de89a5a845668696dcb28c3badd6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260129 02:59:18.185460 21748 consensus_queue.cc:260] T 0340229d521b432580ae926f6062e3f7 P 1c50de89a5a845668696dcb28c3badd6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:59:18.187250 21748 raft_consensus.cc:3060] T 0340229d521b432580ae926f6062e3f7 P 1c50de89a5a845668696dcb28c3badd6 [term 2 FOLLOWER]: Advancing to term 3
I20260129 02:59:18.187803 21835 leader_election.cc:290] T febffe4a8b394368b69b1d164308fb9a P 7a9daf5d42a84615a6383cfec9a10608 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 197b880fa39d432a9b472845876bfaf0 (127.18.211.1:42027), 974b802141bd4f13beffef2aa03f88f9 (127.18.211.2:34517)
I20260129 02:59:18.188959 20512 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "febffe4a8b394368b69b1d164308fb9a" candidate_uuid: "7a9daf5d42a84615a6383cfec9a10608" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "974b802141bd4f13beffef2aa03f88f9" is_pre_election: true
I20260129 02:59:18.181015 20590 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "cb4727386a3e48aca5d71cc6c611c197" candidate_uuid: "197b880fa39d432a9b472845876bfaf0" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1c50de89a5a845668696dcb28c3badd6" is_pre_election: true
I20260129 02:59:18.119544 21136 raft_consensus.cc:1217] T 96609ac68bdf4993a12e1c23355ca013 P 7a9daf5d42a84615a6383cfec9a10608 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20260129 02:59:18.196588 20437 raft_consensus.cc:1240] T 96609ac68bdf4993a12e1c23355ca013 P 197b880fa39d432a9b472845876bfaf0 [term 2 FOLLOWER]: Rejecting Update request from peer 1c50de89a5a845668696dcb28c3badd6 for earlier term 1. Current term is 2. Ops: []
I20260129 02:59:18.182720 20585 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a5a05fbcbbcf4916bbc01fd1a5b5e90d" candidate_uuid: "197b880fa39d432a9b472845876bfaf0" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "1c50de89a5a845668696dcb28c3badd6" is_pre_election: true
I20260129 02:59:18.183001 21816 raft_consensus.cc:1064] T 1299723a4476420ab02fd8bfd7aa0645 P 197b880fa39d432a9b472845876bfaf0: attempting to promote NON_VOTER 7a9daf5d42a84615a6383cfec9a10608 to VOTER
I20260129 02:59:18.200513 21816 consensus_queue.cc:237] T 1299723a4476420ab02fd8bfd7aa0645 P 197b880fa39d432a9b472845876bfaf0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 2.3, Last appended by leader: 1, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } attrs { replace: true } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } }
I20260129 02:59:18.201841 20509 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "96609ac68bdf4993a12e1c23355ca013" candidate_uuid: "197b880fa39d432a9b472845876bfaf0" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "974b802141bd4f13beffef2aa03f88f9"
I20260129 02:59:18.032140 21666 raft_consensus.cc:2804] T 368953c9102643b2a21f99af36b51905 P 197b880fa39d432a9b472845876bfaf0 [term 2 FOLLOWER]: Leader election won for term 2
I20260129 02:59:18.203943 21666 raft_consensus.cc:697] T 368953c9102643b2a21f99af36b51905 P 197b880fa39d432a9b472845876bfaf0 [term 2 LEADER]: Becoming Leader. State: Replica: 197b880fa39d432a9b472845876bfaf0, State: Running, Role: LEADER
I20260129 02:59:18.204468 20583 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "96609ac68bdf4993a12e1c23355ca013" candidate_uuid: "197b880fa39d432a9b472845876bfaf0" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "1c50de89a5a845668696dcb28c3badd6" is_pre_election: true
I20260129 02:59:18.204852 21666 consensus_queue.cc:237] T 368953c9102643b2a21f99af36b51905 P 197b880fa39d432a9b472845876bfaf0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:59:18.184072 20405 leader_election.cc:304] T 00e79d8f25a34676b1d866eb2068a2b2 P 197b880fa39d432a9b472845876bfaf0 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 197b880fa39d432a9b472845876bfaf0; no voters: 7a9daf5d42a84615a6383cfec9a10608, 974b802141bd4f13beffef2aa03f88f9
I20260129 02:59:18.208894 20405 leader_election.cc:304] T 87d5fe48a44a408aaf03e894a67ecb2f P 197b880fa39d432a9b472845876bfaf0 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 197b880fa39d432a9b472845876bfaf0; no voters: 7a9daf5d42a84615a6383cfec9a10608, 974b802141bd4f13beffef2aa03f88f9
I20260129 02:59:18.112135 21836 raft_consensus.cc:515] T 87d5fe48a44a408aaf03e894a67ecb2f P 7a9daf5d42a84615a6383cfec9a10608 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } }
I20260129 02:59:18.211833 20589 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7d9494f5584d4d5db8b9adf1c8fc888b" candidate_uuid: "197b880fa39d432a9b472845876bfaf0" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1c50de89a5a845668696dcb28c3badd6" is_pre_election: true
I20260129 02:59:18.212917 21786 raft_consensus.cc:2749] T 87d5fe48a44a408aaf03e894a67ecb2f P 197b880fa39d432a9b472845876bfaf0 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260129 02:59:18.210966 21788 raft_consensus.cc:2749] T 00e79d8f25a34676b1d866eb2068a2b2 P 197b880fa39d432a9b472845876bfaf0 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260129 02:59:18.219574 20405 leader_election.cc:304] T 2ac659d732314e1288849f08fe5e074e P 197b880fa39d432a9b472845876bfaf0 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 197b880fa39d432a9b472845876bfaf0; no voters: 7a9daf5d42a84615a6383cfec9a10608, 974b802141bd4f13beffef2aa03f88f9
I20260129 02:59:18.217321 21684 consensus_queue.cc:1059] T 96609ac68bdf4993a12e1c23355ca013 P 1c50de89a5a845668696dcb28c3badd6 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 }, Status: INVALID_TERM, Last received: 1.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.004s
I20260129 02:59:18.221272 21684 raft_consensus.cc:3055] T 96609ac68bdf4993a12e1c23355ca013 P 1c50de89a5a845668696dcb28c3badd6 [term 1 LEADER]: Stepping down as leader of term 1
I20260129 02:59:18.221626 21684 raft_consensus.cc:740] T 96609ac68bdf4993a12e1c23355ca013 P 1c50de89a5a845668696dcb28c3badd6 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 1c50de89a5a845668696dcb28c3badd6, State: Running, Role: LEADER
I20260129 02:59:18.222767 21788 raft_consensus.cc:2749] T 2ac659d732314e1288849f08fe5e074e P 197b880fa39d432a9b472845876bfaf0 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260129 02:59:18.222419 21684 consensus_queue.cc:260] T 96609ac68bdf4993a12e1c23355ca013 P 1c50de89a5a845668696dcb28c3badd6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } attrs { replace: true } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: NON_VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: true } }
W20260129 02:59:18.223994 21777 consensus_queue.cc:1175] T 96609ac68bdf4993a12e1c23355ca013 P 1c50de89a5a845668696dcb28c3badd6 [NON_LEADER]: Queue is closed or peer was untracked, disregarding peer response. Response: responder_uuid: "7a9daf5d42a84615a6383cfec9a10608" responder_term: 1 status { last_received { term: 1 index: 2 } last_committed_idx: 2 last_received_current_leader { term: 1 index: 2 } }
I20260129 02:59:18.222035 20335 catalog_manager.cc:5654] T 0340229d521b432580ae926f6062e3f7 P 197b880fa39d432a9b472845876bfaf0 reported cstate change: term changed from 2 to 3, leader changed from 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3) to 197b880fa39d432a9b472845876bfaf0 (127.18.211.1). New cstate: current_term: 3 leader_uuid: "197b880fa39d432a9b472845876bfaf0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } health_report { overall_health: UNKNOWN } } }
I20260129 02:59:18.225610 21789 raft_consensus.cc:993] T 0678b8b468bd43cdbc3e485900844821 P 197b880fa39d432a9b472845876bfaf0: : Instructing follower 7a9daf5d42a84615a6383cfec9a10608 to start an election
I20260129 02:59:18.226056 21786 raft_consensus.cc:1081] T 0678b8b468bd43cdbc3e485900844821 P 197b880fa39d432a9b472845876bfaf0 [term 1 LEADER]: Signalling peer 7a9daf5d42a84615a6383cfec9a10608 to start an election
I20260129 02:59:18.231684 20581 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4f8f3a38f6e24ba4857875181f447442" candidate_uuid: "197b880fa39d432a9b472845876bfaf0" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "1c50de89a5a845668696dcb28c3badd6" is_pre_election: true
I20260129 02:59:18.234915 20585 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "368953c9102643b2a21f99af36b51905" candidate_uuid: "197b880fa39d432a9b472845876bfaf0" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "1c50de89a5a845668696dcb28c3badd6"
I20260129 02:59:18.235492 20585 raft_consensus.cc:3055] T 368953c9102643b2a21f99af36b51905 P 1c50de89a5a845668696dcb28c3badd6 [term 1 LEADER]: Stepping down as leader of term 1
I20260129 02:59:18.235773 20585 raft_consensus.cc:740] T 368953c9102643b2a21f99af36b51905 P 1c50de89a5a845668696dcb28c3badd6 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 1c50de89a5a845668696dcb28c3badd6, State: Running, Role: LEADER
I20260129 02:59:18.236039 21132 tablet_service.cc:2038] Received Run Leader Election RPC: tablet_id: "0678b8b468bd43cdbc3e485900844821"
dest_uuid: "7a9daf5d42a84615a6383cfec9a10608"
from {username='slave'} at 127.0.0.1:34060
I20260129 02:59:18.237097 20581 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0340229d521b432580ae926f6062e3f7" candidate_uuid: "197b880fa39d432a9b472845876bfaf0" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1c50de89a5a845668696dcb28c3badd6" is_pre_election: true
I20260129 02:59:18.239509 21131 raft_consensus.cc:1240] T 87d5fe48a44a408aaf03e894a67ecb2f P 7a9daf5d42a84615a6383cfec9a10608 [term 2 FOLLOWER]: Rejecting Update request from peer 974b802141bd4f13beffef2aa03f88f9 for earlier term 1. Current term is 2. Ops: []
W20260129 02:59:18.218233 21100 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.211.3:37735, user_credentials={real_user=slave}} blocked reactor thread for 85663.6us
I20260129 02:59:18.240602 20587 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0340229d521b432580ae926f6062e3f7" candidate_uuid: "197b880fa39d432a9b472845876bfaf0" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "1c50de89a5a845668696dcb28c3badd6"
I20260129 02:59:18.236573 20585 consensus_queue.cc:260] T 368953c9102643b2a21f99af36b51905 P 1c50de89a5a845668696dcb28c3badd6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:59:18.242347 20585 raft_consensus.cc:3060] T 368953c9102643b2a21f99af36b51905 P 1c50de89a5a845668696dcb28c3badd6 [term 1 FOLLOWER]: Advancing to term 2
I20260129 02:59:18.243840 20405 leader_election.cc:304] T dc2f4cb752cf4d59b5f6bc689f5d8104 P 197b880fa39d432a9b472845876bfaf0 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 197b880fa39d432a9b472845876bfaf0; no voters: 1c50de89a5a845668696dcb28c3badd6, 7a9daf5d42a84615a6383cfec9a10608
I20260129 02:59:18.244774 20589 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "96609ac68bdf4993a12e1c23355ca013" candidate_uuid: "197b880fa39d432a9b472845876bfaf0" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "1c50de89a5a845668696dcb28c3badd6"
I20260129 02:59:18.236635 21132 raft_consensus.cc:493] T 0678b8b468bd43cdbc3e485900844821 P 7a9daf5d42a84615a6383cfec9a10608 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260129 02:59:18.246129 21132 raft_consensus.cc:3060] T 0678b8b468bd43cdbc3e485900844821 P 7a9daf5d42a84615a6383cfec9a10608 [term 1 FOLLOWER]: Advancing to term 2
I20260129 02:59:18.247448 20588 raft_consensus.cc:515] T cedce331e1584ac8b636c9b8a72b2ba0 P 1c50de89a5a845668696dcb28c3badd6 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:59:18.252586 20515 raft_consensus.cc:1275] T add8d4923d1a489e86882c3a25692b95 P 974b802141bd4f13beffef2aa03f88f9 [term 1 FOLLOWER]: Refusing update from remote peer 1c50de89a5a845668696dcb28c3badd6: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260129 02:59:18.253046 20588 leader_election.cc:290] T cedce331e1584ac8b636c9b8a72b2ba0 P 1c50de89a5a845668696dcb28c3badd6 [CANDIDATE]: Term 2 election: Requested vote from peers 197b880fa39d432a9b472845876bfaf0 (127.18.211.1:42027), 974b802141bd4f13beffef2aa03f88f9 (127.18.211.2:34517)
I20260129 02:59:18.260602 20437 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "cedce331e1584ac8b636c9b8a72b2ba0" candidate_uuid: "1c50de89a5a845668696dcb28c3badd6" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "197b880fa39d432a9b472845876bfaf0"
I20260129 02:59:18.261575 20437 raft_consensus.cc:3060] T cedce331e1584ac8b636c9b8a72b2ba0 P 197b880fa39d432a9b472845876bfaf0 [term 1 FOLLOWER]: Advancing to term 2
I20260129 02:59:18.265961 21788 raft_consensus.cc:2749] T dc2f4cb752cf4d59b5f6bc689f5d8104 P 197b880fa39d432a9b472845876bfaf0 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260129 02:59:18.268100 20405 leader_election.cc:304] T a0fcc58d6fb6473cafbd9d34d5195269 P 197b880fa39d432a9b472845876bfaf0 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 197b880fa39d432a9b472845876bfaf0; no voters: 1c50de89a5a845668696dcb28c3badd6, 974b802141bd4f13beffef2aa03f88f9
I20260129 02:59:18.266405 20514 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "cedce331e1584ac8b636c9b8a72b2ba0" candidate_uuid: "1c50de89a5a845668696dcb28c3badd6" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "974b802141bd4f13beffef2aa03f88f9"
I20260129 02:59:18.268956 20514 raft_consensus.cc:3055] T cedce331e1584ac8b636c9b8a72b2ba0 P 974b802141bd4f13beffef2aa03f88f9 [term 1 LEADER]: Stepping down as leader of term 1
I20260129 02:59:18.269361 20514 raft_consensus.cc:740] T cedce331e1584ac8b636c9b8a72b2ba0 P 974b802141bd4f13beffef2aa03f88f9 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 974b802141bd4f13beffef2aa03f88f9, State: Running, Role: LEADER
I20260129 02:59:18.270619 20510 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c891d2bff4ce4568861bbc992e91fbb0" candidate_uuid: "1c50de89a5a845668696dcb28c3badd6" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "974b802141bd4f13beffef2aa03f88f9" is_pre_election: true
I20260129 02:59:18.272101 20556 leader_election.cc:304] T c891d2bff4ce4568861bbc992e91fbb0 P 1c50de89a5a845668696dcb28c3badd6 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1c50de89a5a845668696dcb28c3badd6; no voters: 197b880fa39d432a9b472845876bfaf0, 974b802141bd4f13beffef2aa03f88f9
I20260129 02:59:18.274192 20405 leader_election.cc:304] T cb4727386a3e48aca5d71cc6c611c197 P 197b880fa39d432a9b472845876bfaf0 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 197b880fa39d432a9b472845876bfaf0; no voters: 1c50de89a5a845668696dcb28c3badd6, 974b802141bd4f13beffef2aa03f88f9
I20260129 02:59:18.271133 20514 consensus_queue.cc:260] T cedce331e1584ac8b636c9b8a72b2ba0 P 974b802141bd4f13beffef2aa03f88f9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:59:18.247012 20512 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fcddab6bca684adf8ea032289a00f90e" candidate_uuid: "1c50de89a5a845668696dcb28c3badd6" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "974b802141bd4f13beffef2aa03f88f9" is_pre_election: true
I20260129 02:59:18.251753 21684 raft_consensus.cc:3060] T 96609ac68bdf4993a12e1c23355ca013 P 1c50de89a5a845668696dcb28c3badd6 [term 1 FOLLOWER]: Advancing to term 2
I20260129 02:59:18.293501 20437 raft_consensus.cc:2468] T cedce331e1584ac8b636c9b8a72b2ba0 P 197b880fa39d432a9b472845876bfaf0 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1c50de89a5a845668696dcb28c3badd6 in term 2.
I20260129 02:59:18.294836 20514 raft_consensus.cc:3060] T cedce331e1584ac8b636c9b8a72b2ba0 P 974b802141bd4f13beffef2aa03f88f9 [term 1 FOLLOWER]: Advancing to term 2
I20260129 02:59:18.295356 20553 leader_election.cc:304] T cedce331e1584ac8b636c9b8a72b2ba0 P 1c50de89a5a845668696dcb28c3badd6 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 197b880fa39d432a9b472845876bfaf0, 1c50de89a5a845668696dcb28c3badd6; no voters:
I20260129 02:59:18.267376 21824 consensus_queue.cc:1059] T 87d5fe48a44a408aaf03e894a67ecb2f P 974b802141bd4f13beffef2aa03f88f9 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.020s
I20260129 02:59:18.279524 21807 raft_consensus.cc:2749] T a0fcc58d6fb6473cafbd9d34d5195269 P 197b880fa39d432a9b472845876bfaf0 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260129 02:59:18.282552 21788 raft_consensus.cc:2749] T cb4727386a3e48aca5d71cc6c611c197 P 197b880fa39d432a9b472845876bfaf0 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260129 02:59:18.301476 21813 raft_consensus.cc:2749] T c891d2bff4ce4568861bbc992e91fbb0 P 1c50de89a5a845668696dcb28c3badd6 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260129 02:59:18.303581 20405 leader_election.cc:304] T a5a05fbcbbcf4916bbc01fd1a5b5e90d P 197b880fa39d432a9b472845876bfaf0 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 197b880fa39d432a9b472845876bfaf0; no voters: 1c50de89a5a845668696dcb28c3badd6, 974b802141bd4f13beffef2aa03f88f9
I20260129 02:59:18.306828 21836 leader_election.cc:290] T 87d5fe48a44a408aaf03e894a67ecb2f P 7a9daf5d42a84615a6383cfec9a10608 [CANDIDATE]: Term 2 election: Requested vote from peers 197b880fa39d432a9b472845876bfaf0 (127.18.211.1:42027), 974b802141bd4f13beffef2aa03f88f9 (127.18.211.2:34517)
I20260129 02:59:18.308295 21824 raft_consensus.cc:3055] T 87d5fe48a44a408aaf03e894a67ecb2f P 974b802141bd4f13beffef2aa03f88f9 [term 1 LEADER]: Stepping down as leader of term 1
I20260129 02:59:18.308712 21824 raft_consensus.cc:740] T 87d5fe48a44a408aaf03e894a67ecb2f P 974b802141bd4f13beffef2aa03f88f9 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 974b802141bd4f13beffef2aa03f88f9, State: Running, Role: LEADER
I20260129 02:59:18.309527 21824 consensus_queue.cc:260] T 87d5fe48a44a408aaf03e894a67ecb2f P 974b802141bd4f13beffef2aa03f88f9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } }
I20260129 02:59:18.312431 20556 leader_election.cc:304] T fcddab6bca684adf8ea032289a00f90e P 1c50de89a5a845668696dcb28c3badd6 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1c50de89a5a845668696dcb28c3badd6; no voters: 197b880fa39d432a9b472845876bfaf0, 974b802141bd4f13beffef2aa03f88f9
I20260129 02:59:18.314206 21811 raft_consensus.cc:2749] T fcddab6bca684adf8ea032289a00f90e P 1c50de89a5a845668696dcb28c3badd6 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260129 02:59:18.314502 20585 raft_consensus.cc:2468] T 368953c9102643b2a21f99af36b51905 P 1c50de89a5a845668696dcb28c3badd6 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 197b880fa39d432a9b472845876bfaf0 in term 2.
I20260129 02:59:18.274237 21790 consensus_queue.cc:1048] T add8d4923d1a489e86882c3a25692b95 P 1c50de89a5a845668696dcb28c3badd6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.025s
I20260129 02:59:18.315227 20514 raft_consensus.cc:2468] T cedce331e1584ac8b636c9b8a72b2ba0 P 974b802141bd4f13beffef2aa03f88f9 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1c50de89a5a845668696dcb28c3badd6 in term 2.
I20260129 02:59:18.320070 20515 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "87d5fe48a44a408aaf03e894a67ecb2f" candidate_uuid: "7a9daf5d42a84615a6383cfec9a10608" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "974b802141bd4f13beffef2aa03f88f9"
W20260129 02:59:18.322497 20407 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.211.2:34517, user_credentials={real_user=slave}} blocked reactor thread for 48939.9us
I20260129 02:59:18.325371 21788 raft_consensus.cc:993] T fcddab6bca684adf8ea032289a00f90e P 197b880fa39d432a9b472845876bfaf0: : Instructing follower 1c50de89a5a845668696dcb28c3badd6 to start an election
I20260129 02:59:18.315773 21829 raft_consensus.cc:2804] T cedce331e1584ac8b636c9b8a72b2ba0 P 1c50de89a5a845668696dcb28c3badd6 [term 2 FOLLOWER]: Leader election won for term 2
I20260129 02:59:18.342079 20581 raft_consensus.cc:2410] T 0340229d521b432580ae926f6062e3f7 P 1c50de89a5a845668696dcb28c3badd6 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 197b880fa39d432a9b472845876bfaf0 for term 3 because replica has last-logged OpId of term: 2 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 1.
I20260129 02:59:18.340615 21789 raft_consensus.cc:1081] T fcddab6bca684adf8ea032289a00f90e P 197b880fa39d432a9b472845876bfaf0 [term 3 LEADER]: Signalling peer 1c50de89a5a845668696dcb28c3badd6 to start an election
I20260129 02:59:18.348610 21824 raft_consensus.cc:3060] T 87d5fe48a44a408aaf03e894a67ecb2f P 974b802141bd4f13beffef2aa03f88f9 [term 1 FOLLOWER]: Advancing to term 2
I20260129 02:59:18.351037 21829 raft_consensus.cc:697] T cedce331e1584ac8b636c9b8a72b2ba0 P 1c50de89a5a845668696dcb28c3badd6 [term 2 LEADER]: Becoming Leader. State: Replica: 1c50de89a5a845668696dcb28c3badd6, State: Running, Role: LEADER
I20260129 02:59:18.356148 20589 raft_consensus.cc:2468] T 96609ac68bdf4993a12e1c23355ca013 P 1c50de89a5a845668696dcb28c3badd6 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 197b880fa39d432a9b472845876bfaf0 in term 2.
W20260129 02:59:18.360690 20405 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.211.3:37735, user_credentials={real_user=slave}} blocked reactor thread for 57299.7us
I20260129 02:59:18.359159 21829 consensus_queue.cc:237] T cedce331e1584ac8b636c9b8a72b2ba0 P 1c50de89a5a845668696dcb28c3badd6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:59:18.326977 21132 raft_consensus.cc:515] T 0678b8b468bd43cdbc3e485900844821 P 7a9daf5d42a84615a6383cfec9a10608 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } }
I20260129 02:59:18.371248 20405 leader_election.cc:304] T 7d9494f5584d4d5db8b9adf1c8fc888b P 197b880fa39d432a9b472845876bfaf0 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 197b880fa39d432a9b472845876bfaf0; no voters: 1c50de89a5a845668696dcb28c3badd6, 974b802141bd4f13beffef2aa03f88f9
I20260129 02:59:18.368830 21816 raft_consensus.cc:2749] T a5a05fbcbbcf4916bbc01fd1a5b5e90d P 197b880fa39d432a9b472845876bfaf0 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20260129 02:59:18.374377 20407 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260129 02:59:18.375447 20514 raft_consensus.cc:1275] T 1299723a4476420ab02fd8bfd7aa0645 P 974b802141bd4f13beffef2aa03f88f9 [term 2 FOLLOWER]: Refusing update from remote peer 197b880fa39d432a9b472845876bfaf0: Log matching property violated. Preceding OpId in replica: term: 2 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20260129 02:59:18.376005 20437 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "87d5fe48a44a408aaf03e894a67ecb2f" candidate_uuid: "7a9daf5d42a84615a6383cfec9a10608" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "197b880fa39d432a9b472845876bfaf0"
I20260129 02:59:18.373545 21789 raft_consensus.cc:2749] T 7d9494f5584d4d5db8b9adf1c8fc888b P 197b880fa39d432a9b472845876bfaf0 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260129 02:59:18.377887 20432 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "febffe4a8b394368b69b1d164308fb9a" candidate_uuid: "7a9daf5d42a84615a6383cfec9a10608" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "197b880fa39d432a9b472845876bfaf0" is_pre_election: true
I20260129 02:59:18.367471 21791 raft_consensus.cc:2955] T add8d4923d1a489e86882c3a25692b95 P 1c50de89a5a845668696dcb28c3badd6 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 7a9daf5d42a84615a6383cfec9a10608 (127.18.211.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } attrs { replace: true } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } } }
I20260129 02:59:18.384604 21099 leader_election.cc:304] T febffe4a8b394368b69b1d164308fb9a P 7a9daf5d42a84615a6383cfec9a10608 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7a9daf5d42a84615a6383cfec9a10608; no voters: 197b880fa39d432a9b472845876bfaf0, 974b802141bd4f13beffef2aa03f88f9
I20260129 02:59:18.383730 21136 raft_consensus.cc:2955] T add8d4923d1a489e86882c3a25692b95 P 7a9daf5d42a84615a6383cfec9a10608 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 7a9daf5d42a84615a6383cfec9a10608 (127.18.211.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } attrs { replace: true } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } } }
I20260129 02:59:18.384833 21846 consensus_queue.cc:1048] T 1299723a4476420ab02fd8bfd7aa0645 P 197b880fa39d432a9b472845876bfaf0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260129 02:59:18.385838 21836 raft_consensus.cc:2749] T febffe4a8b394368b69b1d164308fb9a P 7a9daf5d42a84615a6383cfec9a10608 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260129 02:59:18.388150 21132 leader_election.cc:290] T 0678b8b468bd43cdbc3e485900844821 P 7a9daf5d42a84615a6383cfec9a10608 [CANDIDATE]: Term 2 election: Requested vote from peers 197b880fa39d432a9b472845876bfaf0 (127.18.211.1:42027), 974b802141bd4f13beffef2aa03f88f9 (127.18.211.2:34517)
I20260129 02:59:18.388994 20405 leader_election.cc:304] T 4f8f3a38f6e24ba4857875181f447442 P 197b880fa39d432a9b472845876bfaf0 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 197b880fa39d432a9b472845876bfaf0; no voters: 1c50de89a5a845668696dcb28c3badd6, 974b802141bd4f13beffef2aa03f88f9
I20260129 02:59:18.387355 20512 raft_consensus.cc:2955] T add8d4923d1a489e86882c3a25692b95 P 974b802141bd4f13beffef2aa03f88f9 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 7a9daf5d42a84615a6383cfec9a10608 (127.18.211.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } attrs { replace: true } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } } }
I20260129 02:59:18.392253 20514 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0678b8b468bd43cdbc3e485900844821" candidate_uuid: "7a9daf5d42a84615a6383cfec9a10608" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "974b802141bd4f13beffef2aa03f88f9"
I20260129 02:59:18.392805 20514 raft_consensus.cc:3060] T 0678b8b468bd43cdbc3e485900844821 P 974b802141bd4f13beffef2aa03f88f9 [term 1 FOLLOWER]: Advancing to term 2
I20260129 02:59:18.398739 20405 leader_election.cc:304] T 0340229d521b432580ae926f6062e3f7 P 197b880fa39d432a9b472845876bfaf0 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 197b880fa39d432a9b472845876bfaf0; no voters: 1c50de89a5a845668696dcb28c3badd6, 974b802141bd4f13beffef2aa03f88f9
I20260129 02:59:18.400480 21816 raft_consensus.cc:2749] T 4f8f3a38f6e24ba4857875181f447442 P 197b880fa39d432a9b472845876bfaf0 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260129 02:59:18.401486 21798 raft_consensus.cc:2749] T 0340229d521b432580ae926f6062e3f7 P 197b880fa39d432a9b472845876bfaf0 [term 3 LEADER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260129 02:59:18.399807 20405 leader_election.cc:304] T 96609ac68bdf4993a12e1c23355ca013 P 197b880fa39d432a9b472845876bfaf0 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 197b880fa39d432a9b472845876bfaf0, 1c50de89a5a845668696dcb28c3badd6; no voters: 974b802141bd4f13beffef2aa03f88f9
I20260129 02:59:18.402374 20514 raft_consensus.cc:2468] T 0678b8b468bd43cdbc3e485900844821 P 974b802141bd4f13beffef2aa03f88f9 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7a9daf5d42a84615a6383cfec9a10608 in term 2.
I20260129 02:59:18.403880 21789 raft_consensus.cc:2804] T 96609ac68bdf4993a12e1c23355ca013 P 197b880fa39d432a9b472845876bfaf0 [term 2 FOLLOWER]: Leader election won for term 2
I20260129 02:59:18.403935 21102 leader_election.cc:304] T 0678b8b468bd43cdbc3e485900844821 P 7a9daf5d42a84615a6383cfec9a10608 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7a9daf5d42a84615a6383cfec9a10608, 974b802141bd4f13beffef2aa03f88f9; no voters:
I20260129 02:59:18.404513 21789 raft_consensus.cc:697] T 96609ac68bdf4993a12e1c23355ca013 P 197b880fa39d432a9b472845876bfaf0 [term 2 LEADER]: Becoming Leader. State: Replica: 197b880fa39d432a9b472845876bfaf0, State: Running, Role: LEADER
I20260129 02:59:18.405427 21789 consensus_queue.cc:237] T 96609ac68bdf4993a12e1c23355ca013 P 197b880fa39d432a9b472845876bfaf0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } attrs { replace: true } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: NON_VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: true } }
I20260129 02:59:18.410313 21836 raft_consensus.cc:2804] T 0678b8b468bd43cdbc3e485900844821 P 7a9daf5d42a84615a6383cfec9a10608 [term 2 FOLLOWER]: Leader election won for term 2
I20260129 02:59:18.410830 21836 raft_consensus.cc:697] T 0678b8b468bd43cdbc3e485900844821 P 7a9daf5d42a84615a6383cfec9a10608 [term 2 LEADER]: Becoming Leader. State: Replica: 7a9daf5d42a84615a6383cfec9a10608, State: Running, Role: LEADER
I20260129 02:59:18.404088 20432 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0678b8b468bd43cdbc3e485900844821" candidate_uuid: "7a9daf5d42a84615a6383cfec9a10608" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "197b880fa39d432a9b472845876bfaf0"
I20260129 02:59:18.411989 20432 raft_consensus.cc:3055] T 0678b8b468bd43cdbc3e485900844821 P 197b880fa39d432a9b472845876bfaf0 [term 1 LEADER]: Stepping down as leader of term 1
I20260129 02:59:18.412304 20432 raft_consensus.cc:740] T 0678b8b468bd43cdbc3e485900844821 P 197b880fa39d432a9b472845876bfaf0 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 197b880fa39d432a9b472845876bfaf0, State: Running, Role: LEADER
I20260129 02:59:18.413192 20432 consensus_queue.cc:260] T 0678b8b468bd43cdbc3e485900844821 P 197b880fa39d432a9b472845876bfaf0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } }
I20260129 02:59:18.414403 20432 raft_consensus.cc:3060] T 0678b8b468bd43cdbc3e485900844821 P 197b880fa39d432a9b472845876bfaf0 [term 1 FOLLOWER]: Advancing to term 2
I20260129 02:59:18.415243 20515 raft_consensus.cc:2468] T 87d5fe48a44a408aaf03e894a67ecb2f P 974b802141bd4f13beffef2aa03f88f9 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7a9daf5d42a84615a6383cfec9a10608 in term 2.
I20260129 02:59:18.411695 21836 consensus_queue.cc:237] T 0678b8b468bd43cdbc3e485900844821 P 7a9daf5d42a84615a6383cfec9a10608 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } }
I20260129 02:59:18.417189 21102 leader_election.cc:304] T 87d5fe48a44a408aaf03e894a67ecb2f P 7a9daf5d42a84615a6383cfec9a10608 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 7a9daf5d42a84615a6383cfec9a10608, 974b802141bd4f13beffef2aa03f88f9; no voters: 197b880fa39d432a9b472845876bfaf0
I20260129 02:59:18.420410 20581 raft_consensus.cc:1275] T 1299723a4476420ab02fd8bfd7aa0645 P 1c50de89a5a845668696dcb28c3badd6 [term 2 FOLLOWER]: Refusing update from remote peer 197b880fa39d432a9b472845876bfaf0: Log matching property violated. Preceding OpId in replica: term: 2 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20260129 02:59:18.420823 21834 raft_consensus.cc:2804] T 87d5fe48a44a408aaf03e894a67ecb2f P 7a9daf5d42a84615a6383cfec9a10608 [term 2 FOLLOWER]: Leader election won for term 2
I20260129 02:59:18.421314 21834 raft_consensus.cc:697] T 87d5fe48a44a408aaf03e894a67ecb2f P 7a9daf5d42a84615a6383cfec9a10608 [term 2 LEADER]: Becoming Leader. State: Replica: 7a9daf5d42a84615a6383cfec9a10608, State: Running, Role: LEADER
I20260129 02:59:18.422752 20581 tablet_service.cc:2038] Received Run Leader Election RPC: tablet_id: "fcddab6bca684adf8ea032289a00f90e"
dest_uuid: "1c50de89a5a845668696dcb28c3badd6"
from {username='slave'} at 127.0.0.1:35754
I20260129 02:59:18.423319 20581 raft_consensus.cc:493] T fcddab6bca684adf8ea032289a00f90e P 1c50de89a5a845668696dcb28c3badd6 [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20260129 02:59:18.423676 20581 raft_consensus.cc:3060] T fcddab6bca684adf8ea032289a00f90e P 1c50de89a5a845668696dcb28c3badd6 [term 3 FOLLOWER]: Advancing to term 4
I20260129 02:59:18.423121 21834 consensus_queue.cc:237] T 87d5fe48a44a408aaf03e894a67ecb2f P 7a9daf5d42a84615a6383cfec9a10608 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } }
I20260129 02:59:18.427263 21132 raft_consensus.cc:1275] T 1299723a4476420ab02fd8bfd7aa0645 P 7a9daf5d42a84615a6383cfec9a10608 [term 2 LEARNER]: Refusing update from remote peer 197b880fa39d432a9b472845876bfaf0: Log matching property violated. Preceding OpId in replica: term: 2 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20260129 02:59:18.435217 21846 consensus_queue.cc:1048] T 1299723a4476420ab02fd8bfd7aa0645 P 197b880fa39d432a9b472845876bfaf0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260129 02:59:18.436561 20515 raft_consensus.cc:1275] T 0340229d521b432580ae926f6062e3f7 P 974b802141bd4f13beffef2aa03f88f9 [term 3 FOLLOWER]: Refusing update from remote peer 197b880fa39d432a9b472845876bfaf0: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260129 02:59:18.436977 20581 raft_consensus.cc:515] T fcddab6bca684adf8ea032289a00f90e P 1c50de89a5a845668696dcb28c3badd6 [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:59:18.438485 21798 consensus_queue.cc:1048] T 0340229d521b432580ae926f6062e3f7 P 197b880fa39d432a9b472845876bfaf0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 1, Time since last communication: 0.000s
I20260129 02:59:18.443537 21798 consensus_queue.cc:1048] T 1299723a4476420ab02fd8bfd7aa0645 P 197b880fa39d432a9b472845876bfaf0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260129 02:59:18.447543 20581 leader_election.cc:290] T fcddab6bca684adf8ea032289a00f90e P 1c50de89a5a845668696dcb28c3badd6 [CANDIDATE]: Term 4 election: Requested vote from peers 197b880fa39d432a9b472845876bfaf0 (127.18.211.1:42027), 974b802141bd4f13beffef2aa03f88f9 (127.18.211.2:34517)
I20260129 02:59:18.447417 20435 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fcddab6bca684adf8ea032289a00f90e" candidate_uuid: "1c50de89a5a845668696dcb28c3badd6" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: true dest_uuid: "197b880fa39d432a9b472845876bfaf0"
I20260129 02:59:18.448102 20435 raft_consensus.cc:3055] T fcddab6bca684adf8ea032289a00f90e P 197b880fa39d432a9b472845876bfaf0 [term 3 LEADER]: Stepping down as leader of term 3
I20260129 02:59:18.448396 20435 raft_consensus.cc:740] T fcddab6bca684adf8ea032289a00f90e P 197b880fa39d432a9b472845876bfaf0 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 197b880fa39d432a9b472845876bfaf0, State: Running, Role: LEADER
I20260129 02:59:18.449219 20514 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fcddab6bca684adf8ea032289a00f90e" candidate_uuid: "1c50de89a5a845668696dcb28c3badd6" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: true dest_uuid: "974b802141bd4f13beffef2aa03f88f9"
I20260129 02:59:18.449105 20435 consensus_queue.cc:260] T fcddab6bca684adf8ea032289a00f90e P 197b880fa39d432a9b472845876bfaf0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:59:18.449719 20514 raft_consensus.cc:3060] T fcddab6bca684adf8ea032289a00f90e P 974b802141bd4f13beffef2aa03f88f9 [term 3 FOLLOWER]: Advancing to term 4
I20260129 02:59:18.450321 20435 raft_consensus.cc:3060] T fcddab6bca684adf8ea032289a00f90e P 197b880fa39d432a9b472845876bfaf0 [term 3 FOLLOWER]: Advancing to term 4
I20260129 02:59:18.461637 20435 raft_consensus.cc:2468] T fcddab6bca684adf8ea032289a00f90e P 197b880fa39d432a9b472845876bfaf0 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1c50de89a5a845668696dcb28c3badd6 in term 4.
I20260129 02:59:18.473246 20514 raft_consensus.cc:2468] T fcddab6bca684adf8ea032289a00f90e P 974b802141bd4f13beffef2aa03f88f9 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1c50de89a5a845668696dcb28c3badd6 in term 4.
I20260129 02:59:18.479900 20556 leader_election.cc:304] T fcddab6bca684adf8ea032289a00f90e P 1c50de89a5a845668696dcb28c3badd6 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1c50de89a5a845668696dcb28c3badd6, 974b802141bd4f13beffef2aa03f88f9; no voters:
I20260129 02:59:18.488307 21813 raft_consensus.cc:2804] T fcddab6bca684adf8ea032289a00f90e P 1c50de89a5a845668696dcb28c3badd6 [term 4 FOLLOWER]: Leader election won for term 4
I20260129 02:59:18.508430 21813 raft_consensus.cc:697] T fcddab6bca684adf8ea032289a00f90e P 1c50de89a5a845668696dcb28c3badd6 [term 4 LEADER]: Becoming Leader. State: Replica: 1c50de89a5a845668696dcb28c3badd6, State: Running, Role: LEADER
I20260129 02:59:18.509606 21813 consensus_queue.cc:237] T fcddab6bca684adf8ea032289a00f90e P 1c50de89a5a845668696dcb28c3badd6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:59:18.520468 20589 raft_consensus.cc:1275] T 0340229d521b432580ae926f6062e3f7 P 1c50de89a5a845668696dcb28c3badd6 [term 3 FOLLOWER]: Refusing update from remote peer 197b880fa39d432a9b472845876bfaf0: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260129 02:59:18.528057 21798 consensus_queue.cc:1048] T 0340229d521b432580ae926f6062e3f7 P 197b880fa39d432a9b472845876bfaf0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
W20260129 02:59:18.529523 20462 debug-util.cc:398] Leaking SignalData structure 0x7b08000b3be0 after lost signal to thread 19279
W20260129 02:59:18.530663 20462 debug-util.cc:398] Leaking SignalData structure 0x7b08000da7c0 after lost signal to thread 20372
W20260129 02:59:18.552321 20462 debug-util.cc:398] Leaking SignalData structure 0x7b080028e500 after lost signal to thread 20465
I20260129 02:59:18.613355 20337 catalog_manager.cc:5654] T cedce331e1584ac8b636c9b8a72b2ba0 P 1c50de89a5a845668696dcb28c3badd6 reported cstate change: term changed from 1 to 2, leader changed from 974b802141bd4f13beffef2aa03f88f9 (127.18.211.2) to 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3). New cstate: current_term: 2 leader_uuid: "1c50de89a5a845668696dcb28c3badd6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } health_report { overall_health: HEALTHY } } }
I20260129 02:59:18.645274 20338 catalog_manager.cc:5654] T add8d4923d1a489e86882c3a25692b95 P 974b802141bd4f13beffef2aa03f88f9 reported cstate change: config changed from index 2 to 3, 7a9daf5d42a84615a6383cfec9a10608 (127.18.211.4) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "1c50de89a5a845668696dcb28c3badd6" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } attrs { replace: true } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } } }
W20260129 02:59:18.657177 21099 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.211.1:42027, user_credentials={real_user=slave}} blocked reactor thread for 99219.2us
W20260129 02:59:18.659612 20462 debug-util.cc:398] Leaking SignalData structure 0x7b08001e8a60 after lost signal to thread 20540
W20260129 02:59:18.663020 21833 log.cc:927] Time spent T 0340229d521b432580ae926f6062e3f7 P 1c50de89a5a845668696dcb28c3badd6: Append to log took a long time: real 0.081s user 0.000s sys 0.000s
W20260129 02:59:18.679208 20462 debug-util.cc:398] Leaking SignalData structure 0x7b08001ad040 after lost signal to thread 20614
W20260129 02:59:18.680788 20462 debug-util.cc:398] Leaking SignalData structure 0x7b0800366b40 after lost signal to thread 21160
W20260129 02:59:18.681214 20462 debug-util.cc:398] Leaking SignalData structure 0x7b08002ab6c0 after lost signal to thread 21655
W20260129 02:59:18.681608 20462 debug-util.cc:398] Leaking SignalData structure 0x7b08002f49a0 after lost signal to thread 21701
W20260129 02:59:18.681895 20462 debug-util.cc:398] Leaking SignalData structure 0x7b08002f4a40 after lost signal to thread 21730
W20260129 02:59:18.682186 20462 debug-util.cc:398] Leaking SignalData structure 0x7b08002f4880 after lost signal to thread 21744
W20260129 02:59:18.682515 20462 debug-util.cc:398] Leaking SignalData structure 0x7b08002aef60 after lost signal to thread 21750
W20260129 02:59:18.682781 20462 debug-util.cc:398] Leaking SignalData structure 0x7b08002af160 after lost signal to thread 21751
W20260129 02:59:18.683094 20462 debug-util.cc:398] Leaking SignalData structure 0x7b08002b90c0 after lost signal to thread 21760
I20260129 02:59:18.713452 20432 raft_consensus.cc:2468] T 0678b8b468bd43cdbc3e485900844821 P 197b880fa39d432a9b472845876bfaf0 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7a9daf5d42a84615a6383cfec9a10608 in term 2.
W20260129 02:59:18.708449 20462 debug-util.cc:398] Leaking SignalData structure 0x7b080005c460 after lost signal to thread 21781
I20260129 02:59:18.770766 21798 raft_consensus.cc:2955] T 1299723a4476420ab02fd8bfd7aa0645 P 197b880fa39d432a9b472845876bfaf0 [term 2 LEADER]: Committing config change with OpId 2.4: config changed from index 3 to 4, 7a9daf5d42a84615a6383cfec9a10608 (127.18.211.4) changed from NON_VOTER to VOTER. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } attrs { replace: true } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } } }
W20260129 02:59:18.776407 20407 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.211.2:34517, user_credentials={real_user=slave}} blocked reactor thread for 48539.4us
W20260129 02:59:18.778862 20405 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.211.3:37735, user_credentials={real_user=slave}} blocked reactor thread for 48029.5us
W20260129 02:59:18.726858 20462 debug-util.cc:398] Leaking SignalData structure 0x7b0800326140 after lost signal to thread 21784
W20260129 02:59:18.798650 20462 debug-util.cc:398] Leaking SignalData structure 0x7b080036a360 after lost signal to thread 21797
W20260129 02:59:18.799147 20462 debug-util.cc:398] Leaking SignalData structure 0x7b08003272a0 after lost signal to thread 21806
W20260129 02:59:18.799506 20462 debug-util.cc:398] Leaking SignalData structure 0x7b08002cee20 after lost signal to thread 21810
I20260129 02:59:18.803759 20585 raft_consensus.cc:2955] T 1299723a4476420ab02fd8bfd7aa0645 P 1c50de89a5a845668696dcb28c3badd6 [term 2 FOLLOWER]: Committing config change with OpId 2.4: config changed from index 3 to 4, 7a9daf5d42a84615a6383cfec9a10608 (127.18.211.4) changed from NON_VOTER to VOTER. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } attrs { replace: true } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } } }
I20260129 02:59:18.799546 20337 catalog_manager.cc:5654] T fcddab6bca684adf8ea032289a00f90e P 1c50de89a5a845668696dcb28c3badd6 reported cstate change: term changed from 3 to 4, leader changed from 197b880fa39d432a9b472845876bfaf0 (127.18.211.1) to 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3). New cstate: current_term: 4 leader_uuid: "1c50de89a5a845668696dcb28c3badd6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } health_report { overall_health: HEALTHY } } }
I20260129 02:59:18.870977 20435 raft_consensus.cc:2955] T add8d4923d1a489e86882c3a25692b95 P 197b880fa39d432a9b472845876bfaf0 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 7a9daf5d42a84615a6383cfec9a10608 (127.18.211.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } attrs { replace: true } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } } }
I20260129 02:59:18.887342 20581 raft_consensus.cc:1275] T 96609ac68bdf4993a12e1c23355ca013 P 1c50de89a5a845668696dcb28c3badd6 [term 2 FOLLOWER]: Refusing update from remote peer 197b880fa39d432a9b472845876bfaf0: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20260129 02:59:18.887605 20585 raft_consensus.cc:1275] T 368953c9102643b2a21f99af36b51905 P 1c50de89a5a845668696dcb28c3badd6 [term 2 FOLLOWER]: Refusing update from remote peer 197b880fa39d432a9b472845876bfaf0: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260129 02:59:18.899148 21132 raft_consensus.cc:3060] T 96609ac68bdf4993a12e1c23355ca013 P 7a9daf5d42a84615a6383cfec9a10608 [term 1 LEARNER]: Advancing to term 2
I20260129 02:59:18.902379 20515 raft_consensus.cc:1275] T cedce331e1584ac8b636c9b8a72b2ba0 P 974b802141bd4f13beffef2aa03f88f9 [term 2 FOLLOWER]: Refusing update from remote peer 1c50de89a5a845668696dcb28c3badd6: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260129 02:59:18.905238 21807 consensus_queue.cc:1048] T 96609ac68bdf4993a12e1c23355ca013 P 197b880fa39d432a9b472845876bfaf0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260129 02:59:18.905792 21788 consensus_queue.cc:1048] T 368953c9102643b2a21f99af36b51905 P 197b880fa39d432a9b472845876bfaf0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260129 02:59:18.910034 21815 consensus_queue.cc:1048] T cedce331e1584ac8b636c9b8a72b2ba0 P 1c50de89a5a845668696dcb28c3badd6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.003s
I20260129 02:59:18.927477 20585 consensus_queue.cc:237] T add8d4923d1a489e86882c3a25692b95 P 1c50de89a5a845668696dcb28c3badd6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } }
I20260129 02:59:18.934535 20514 raft_consensus.cc:1275] T add8d4923d1a489e86882c3a25692b95 P 974b802141bd4f13beffef2aa03f88f9 [term 1 FOLLOWER]: Refusing update from remote peer 1c50de89a5a845668696dcb28c3badd6: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260129 02:59:18.941787 21131 raft_consensus.cc:1275] T add8d4923d1a489e86882c3a25692b95 P 7a9daf5d42a84615a6383cfec9a10608 [term 1 FOLLOWER]: Refusing update from remote peer 1c50de89a5a845668696dcb28c3badd6: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260129 02:59:18.942438 20432 raft_consensus.cc:1275] T 0678b8b468bd43cdbc3e485900844821 P 197b880fa39d432a9b472845876bfaf0 [term 2 FOLLOWER]: Refusing update from remote peer 7a9daf5d42a84615a6383cfec9a10608: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260129 02:59:18.946067 21131 raft_consensus.cc:2955] T 1299723a4476420ab02fd8bfd7aa0645 P 7a9daf5d42a84615a6383cfec9a10608 [term 2 FOLLOWER]: Committing config change with OpId 2.4: config changed from index 3 to 4, 7a9daf5d42a84615a6383cfec9a10608 (127.18.211.4) changed from NON_VOTER to VOTER. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } attrs { replace: true } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } } }
I20260129 02:59:18.952510 20509 raft_consensus.cc:2955] T 1299723a4476420ab02fd8bfd7aa0645 P 974b802141bd4f13beffef2aa03f88f9 [term 2 FOLLOWER]: Committing config change with OpId 2.4: config changed from index 3 to 4, 7a9daf5d42a84615a6383cfec9a10608 (127.18.211.4) changed from NON_VOTER to VOTER. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } attrs { replace: true } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } } }
I20260129 02:59:18.955034 20513 raft_consensus.cc:3060] T 96609ac68bdf4993a12e1c23355ca013 P 974b802141bd4f13beffef2aa03f88f9 [term 1 FOLLOWER]: Advancing to term 2
I20260129 02:59:18.932119 21801 raft_consensus.cc:1064] T 3ccac82a01a2445fa5b0ce6c0fcf2295 P 197b880fa39d432a9b472845876bfaf0: attempting to promote NON_VOTER 7a9daf5d42a84615a6383cfec9a10608 to VOTER
I20260129 02:59:18.972635 20433 raft_consensus.cc:3060] T 87d5fe48a44a408aaf03e894a67ecb2f P 197b880fa39d432a9b472845876bfaf0 [term 1 FOLLOWER]: Advancing to term 2
I20260129 02:59:18.975560 20335 catalog_manager.cc:5654] T 0678b8b468bd43cdbc3e485900844821 P 7a9daf5d42a84615a6383cfec9a10608 reported cstate change: term changed from 1 to 2, leader changed from 197b880fa39d432a9b472845876bfaf0 (127.18.211.1) to 7a9daf5d42a84615a6383cfec9a10608 (127.18.211.4). New cstate: current_term: 2 leader_uuid: "7a9daf5d42a84615a6383cfec9a10608" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260129 02:59:18.974130 21829 consensus_queue.cc:1048] T add8d4923d1a489e86882c3a25692b95 P 1c50de89a5a845668696dcb28c3badd6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260129 02:59:18.982347 20335 catalog_manager.cc:5654] T 87d5fe48a44a408aaf03e894a67ecb2f P 7a9daf5d42a84615a6383cfec9a10608 reported cstate change: term changed from 1 to 2, leader changed from 974b802141bd4f13beffef2aa03f88f9 (127.18.211.2) to 7a9daf5d42a84615a6383cfec9a10608 (127.18.211.4). New cstate: current_term: 2 leader_uuid: "7a9daf5d42a84615a6383cfec9a10608" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260129 02:59:18.962114 20512 raft_consensus.cc:1275] T 368953c9102643b2a21f99af36b51905 P 974b802141bd4f13beffef2aa03f88f9 [term 2 FOLLOWER]: Refusing update from remote peer 197b880fa39d432a9b472845876bfaf0: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260129 02:59:18.978365 20513 raft_consensus.cc:1275] T 96609ac68bdf4993a12e1c23355ca013 P 974b802141bd4f13beffef2aa03f88f9 [term 2 FOLLOWER]: Refusing update from remote peer 197b880fa39d432a9b472845876bfaf0: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20260129 02:59:18.918898 21132 raft_consensus.cc:1275] T 96609ac68bdf4993a12e1c23355ca013 P 7a9daf5d42a84615a6383cfec9a10608 [term 2 LEARNER]: Refusing update from remote peer 197b880fa39d432a9b472845876bfaf0: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20260129 02:59:18.974822 21786 raft_consensus.cc:1064] T 3ccac82a01a2445fa5b0ce6c0fcf2295 P 197b880fa39d432a9b472845876bfaf0: attempting to promote NON_VOTER 7a9daf5d42a84615a6383cfec9a10608 to VOTER
I20260129 02:59:19.000229 21787 consensus_queue.cc:1048] T 368953c9102643b2a21f99af36b51905 P 197b880fa39d432a9b472845876bfaf0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.005s
I20260129 02:59:19.004428 20432 raft_consensus.cc:1275] T cedce331e1584ac8b636c9b8a72b2ba0 P 197b880fa39d432a9b472845876bfaf0 [term 2 FOLLOWER]: Refusing update from remote peer 1c50de89a5a845668696dcb28c3badd6: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260129 02:59:19.004285 21847 consensus_queue.cc:1048] T 96609ac68bdf4993a12e1c23355ca013 P 197b880fa39d432a9b472845876bfaf0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.008s
I20260129 02:59:19.015542 21787 consensus_queue.cc:1048] T 96609ac68bdf4993a12e1c23355ca013 P 197b880fa39d432a9b472845876bfaf0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: NON_VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260129 02:59:19.024057 20511 raft_consensus.cc:1275] T fcddab6bca684adf8ea032289a00f90e P 974b802141bd4f13beffef2aa03f88f9 [term 4 FOLLOWER]: Refusing update from remote peer 1c50de89a5a845668696dcb28c3badd6: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20260129 02:59:18.994164 21638 consensus_queue.cc:1048] T add8d4923d1a489e86882c3a25692b95 P 1c50de89a5a845668696dcb28c3badd6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.031s
I20260129 02:59:19.032112 21815 consensus_queue.cc:1048] T fcddab6bca684adf8ea032289a00f90e P 1c50de89a5a845668696dcb28c3badd6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260129 02:59:19.023995 21791 consensus_queue.cc:1048] T cedce331e1584ac8b636c9b8a72b2ba0 P 1c50de89a5a845668696dcb28c3badd6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.011s
I20260129 02:59:19.038631 19276 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260129 02:59:19.052665 20433 raft_consensus.cc:1275] T 87d5fe48a44a408aaf03e894a67ecb2f P 197b880fa39d432a9b472845876bfaf0 [term 2 FOLLOWER]: Refusing update from remote peer 7a9daf5d42a84615a6383cfec9a10608: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260129 02:59:19.054996 21801 consensus_queue.cc:237] T 3ccac82a01a2445fa5b0ce6c0fcf2295 P 197b880fa39d432a9b472845876bfaf0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } attrs { replace: true } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } }
I20260129 02:59:19.060206 21821 raft_consensus.cc:1064] T 96609ac68bdf4993a12e1c23355ca013 P 197b880fa39d432a9b472845876bfaf0: attempting to promote NON_VOTER 7a9daf5d42a84615a6383cfec9a10608 to VOTER
W20260129 02:59:19.060689 21821 raft_consensus.cc:1068] T 96609ac68bdf4993a12e1c23355ca013 P 197b880fa39d432a9b472845876bfaf0: Unable to promote non-voter 7a9daf5d42a84615a6383cfec9a10608: Illegal state: Leader has not yet committed an operation in its own term
I20260129 02:59:19.066996 21836 consensus_queue.cc:1048] T 87d5fe48a44a408aaf03e894a67ecb2f P 7a9daf5d42a84615a6383cfec9a10608 [LEADER]: Connected to new peer: Peer: permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260129 02:59:19.064267 21134 raft_consensus.cc:2955] T add8d4923d1a489e86882c3a25692b95 P 7a9daf5d42a84615a6383cfec9a10608 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 197b880fa39d432a9b472845876bfaf0 (127.18.211.1) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } } }
W20260129 02:59:19.117035 20479 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.211.1:42027, user_credentials={real_user=slave}} blocked reactor thread for 47732.6us
W20260129 02:59:19.119484 20553 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.211.1:42027, user_credentials={real_user=slave}} blocked reactor thread for 49987.6us
I20260129 02:59:19.123309 20514 raft_consensus.cc:2955] T add8d4923d1a489e86882c3a25692b95 P 974b802141bd4f13beffef2aa03f88f9 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 197b880fa39d432a9b472845876bfaf0 (127.18.211.1) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } } }
I20260129 02:59:19.137850 20515 raft_consensus.cc:1275] T 87d5fe48a44a408aaf03e894a67ecb2f P 974b802141bd4f13beffef2aa03f88f9 [term 2 FOLLOWER]: Refusing update from remote peer 7a9daf5d42a84615a6383cfec9a10608: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260129 02:59:19.138929 21862 consensus_queue.cc:1048] T 0678b8b468bd43cdbc3e485900844821 P 7a9daf5d42a84615a6383cfec9a10608 [LEADER]: Connected to new peer: Peer: permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.165s
I20260129 02:59:19.262317 20335 catalog_manager.cc:5654] T 96609ac68bdf4993a12e1c23355ca013 P 1c50de89a5a845668696dcb28c3badd6 reported cstate change: term changed from 1 to 2, leader changed from 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3) to 197b880fa39d432a9b472845876bfaf0 (127.18.211.1). New cstate: current_term: 2 leader_uuid: "197b880fa39d432a9b472845876bfaf0" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } attrs { replace: true } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: NON_VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: true } } }
W20260129 02:59:19.285784 20479 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.211.1:42027, user_credentials={real_user=slave}} blocked reactor thread for 66464us
I20260129 02:59:19.269086 21834 consensus_queue.cc:1048] T 87d5fe48a44a408aaf03e894a67ecb2f P 7a9daf5d42a84615a6383cfec9a10608 [LEADER]: Connected to new peer: Peer: permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.033s
I20260129 02:59:19.277997 20335 catalog_manager.cc:5654] T 1299723a4476420ab02fd8bfd7aa0645 P 1c50de89a5a845668696dcb28c3badd6 reported cstate change: config changed from index 3 to 4, 7a9daf5d42a84615a6383cfec9a10608 (127.18.211.4) changed from NON_VOTER to VOTER. New cstate: current_term: 2 leader_uuid: "197b880fa39d432a9b472845876bfaf0" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } attrs { replace: true } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } } }
I20260129 02:59:19.320207 20335 catalog_manager.cc:5654] T 368953c9102643b2a21f99af36b51905 P 1c50de89a5a845668696dcb28c3badd6 reported cstate change: term changed from 1 to 2, leader changed from 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3) to 197b880fa39d432a9b472845876bfaf0 (127.18.211.1). New cstate: current_term: 2 leader_uuid: "197b880fa39d432a9b472845876bfaf0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } } }
I20260129 02:59:19.418958 20581 tablet_service.cc:2066] Received LeaderStepDown RPC: tablet_id: "505580c93c694bbb8ea96fdf9b17ab46"
dest_uuid: "1c50de89a5a845668696dcb28c3badd6"
mode: GRACEFUL
new_leader_uuid: "974b802141bd4f13beffef2aa03f88f9"
from {username='slave'} at 127.0.0.1:54860
I20260129 02:59:19.419683 20581 raft_consensus.cc:606] T 505580c93c694bbb8ea96fdf9b17ab46 P 1c50de89a5a845668696dcb28c3badd6 [term 1 LEADER]: Received request to transfer leadership to TS 974b802141bd4f13beffef2aa03f88f9
I20260129 02:59:19.137902 21811 raft_consensus.cc:2955] T add8d4923d1a489e86882c3a25692b95 P 1c50de89a5a845668696dcb28c3badd6 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 197b880fa39d432a9b472845876bfaf0 (127.18.211.1) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } } }
W20260129 02:59:19.418632 21848 log.cc:927] Time spent T 368953c9102643b2a21f99af36b51905 P 197b880fa39d432a9b472845876bfaf0: Append to log took a long time: real 0.063s user 0.000s sys 0.000s
I20260129 02:59:19.437633 20432 raft_consensus.cc:1275] T fcddab6bca684adf8ea032289a00f90e P 197b880fa39d432a9b472845876bfaf0 [term 4 FOLLOWER]: Refusing update from remote peer 1c50de89a5a845668696dcb28c3badd6: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
W20260129 02:59:19.448398 21870 log.cc:927] Time spent T 96609ac68bdf4993a12e1c23355ca013 P 1c50de89a5a845668696dcb28c3badd6: Append to log took a long time: real 0.242s user 0.000s sys 0.001s
W20260129 02:59:19.475708 21871 log.cc:927] Time spent T 368953c9102643b2a21f99af36b51905 P 1c50de89a5a845668696dcb28c3badd6: Append to log took a long time: real 0.284s user 0.001s sys 0.001s
W20260129 02:59:19.497103 21878 log.cc:927] Time spent T fcddab6bca684adf8ea032289a00f90e P 974b802141bd4f13beffef2aa03f88f9: Append to log took a long time: real 0.089s user 0.001s sys 0.000s
W20260129 02:59:19.490978 21876 log.cc:927] Time spent T cedce331e1584ac8b636c9b8a72b2ba0 P 197b880fa39d432a9b472845876bfaf0: Append to log took a long time: real 0.201s user 0.000s sys 0.001s
W20260129 02:59:19.514815 21798 consensus_meta.cc:287] T 1299723a4476420ab02fd8bfd7aa0645 P 197b880fa39d432a9b472845876bfaf0: Time spent flushing consensus metadata: real 0.741s user 0.001s sys 0.011s
I20260129 02:59:19.547683 20513 raft_consensus.cc:1275] T 0678b8b468bd43cdbc3e485900844821 P 974b802141bd4f13beffef2aa03f88f9 [term 2 FOLLOWER]: Refusing update from remote peer 7a9daf5d42a84615a6383cfec9a10608: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260129 02:59:19.572435 21862 consensus_queue.cc:1048] T 0678b8b468bd43cdbc3e485900844821 P 7a9daf5d42a84615a6383cfec9a10608 [LEADER]: Connected to new peer: Peer: permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
W20260129 02:59:19.579584 20407 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.211.2:34517, user_credentials={real_user=slave}} blocked reactor thread for 226391us
W20260129 02:59:19.609699 21801 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.541s user 0.072s sys 0.121s
W20260129 02:59:19.622749 21801 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.554s user 0.072s sys 0.122s
I20260129 02:59:19.684566 20335 catalog_manager.cc:5654] T add8d4923d1a489e86882c3a25692b95 P 7a9daf5d42a84615a6383cfec9a10608 reported cstate change: config changed from index 3 to 4, VOTER 197b880fa39d432a9b472845876bfaf0 (127.18.211.1) evicted. New cstate: current_term: 1 leader_uuid: "1c50de89a5a845668696dcb28c3badd6" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } } }
W20260129 02:59:19.705425 21099 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.211.1:42027, user_credentials={real_user=slave}} blocked reactor thread for 346186us
I20260129 02:59:19.712129 20510 tablet_service.cc:2066] Received LeaderStepDown RPC: tablet_id: "75f45538dc704d9a8ec4ff8ad9f3ea5b"
dest_uuid: "974b802141bd4f13beffef2aa03f88f9"
mode: GRACEFUL
new_leader_uuid: "1c50de89a5a845668696dcb28c3badd6"
from {username='slave'} at 127.0.0.1:52008
W20260129 02:59:19.711050 20479 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.211.1:42027, user_credentials={real_user=slave}} blocked reactor thread for 354398us
I20260129 02:59:19.712903 20510 raft_consensus.cc:606] T 75f45538dc704d9a8ec4ff8ad9f3ea5b P 974b802141bd4f13beffef2aa03f88f9 [term 1 LEADER]: Received request to transfer leadership to TS 1c50de89a5a845668696dcb28c3badd6
I20260129 02:59:19.762763 21135 raft_consensus.cc:1275] T 3ccac82a01a2445fa5b0ce6c0fcf2295 P 7a9daf5d42a84615a6383cfec9a10608 [term 1 LEARNER]: Refusing update from remote peer 197b880fa39d432a9b472845876bfaf0: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
W20260129 02:59:19.790518 21786 raft_consensus.cc:1068] T 3ccac82a01a2445fa5b0ce6c0fcf2295 P 197b880fa39d432a9b472845876bfaf0: Unable to promote non-voter 7a9daf5d42a84615a6383cfec9a10608: Illegal state: RaftConfig change currently pending. Only one is allowed at a time.
Committed config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } attrs { replace: true } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: NON_VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: true } }.
Pending config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } attrs { replace: true } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } }
W20260129 02:59:19.791318 21900 log.cc:927] Time spent T 87d5fe48a44a408aaf03e894a67ecb2f P 974b802141bd4f13beffef2aa03f88f9: Append to log took a long time: real 0.051s user 0.001s sys 0.000s
W20260129 02:59:19.842926 20407 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.211.2:34517, user_credentials={real_user=slave}} blocked reactor thread for 58755.3us
W20260129 02:59:19.843842 20554 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.211.4:40439, user_credentials={real_user=slave}} blocked reactor thread for 54326.6us
I20260129 02:59:19.772689 21847 consensus_queue.cc:1048] T 3ccac82a01a2445fa5b0ce6c0fcf2295 P 197b880fa39d432a9b472845876bfaf0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.012s
W20260129 02:59:19.910319 20407 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.211.2:34517, user_credentials={real_user=slave}} blocked reactor thread for 57643.5us
W20260129 02:59:19.788938 21100 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.211.3:37735, user_credentials={real_user=slave}} blocked reactor thread for 380193us
I20260129 02:59:20.039780 21790 consensus_queue.cc:1048] T fcddab6bca684adf8ea032289a00f90e P 1c50de89a5a845668696dcb28c3badd6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
W20260129 02:59:20.302208 20556 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.211.2:34517, user_credentials={real_user=slave}} blocked reactor thread for 260456us
I20260129 02:59:20.355015 21811 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.ChangeConfig from 127.0.0.1:35744 (request call id 701) took 1376 ms. Trace:
I20260129 02:59:20.637972 21811 rpcz_store.cc:276] 0129 02:59:18.924993 (+ 0us) service_pool.cc:168] Inserting onto call queue
0129 02:59:18.925362 (+ 369us) service_pool.cc:225] Handling call
0129 02:59:18.930287 (+ 4925us) log.cc:844] Serialized 494 byte log entry
0129 02:59:20.301470 (+1371183us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":193,"raft.run_cpu_time_us":981,"raft.run_wall_time_us":980}
W20260129 02:59:20.792398 20556 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.211.2:34517, user_credentials={real_user=slave}} blocked reactor thread for 489417us
I20260129 02:59:20.920964 21891 raft_consensus.cc:493] T fcddab6bca684adf8ea032289a00f90e P 974b802141bd4f13beffef2aa03f88f9 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 1c50de89a5a845668696dcb28c3badd6)
I20260129 02:59:20.921133 21905 raft_consensus.cc:493] T 0340229d521b432580ae926f6062e3f7 P 974b802141bd4f13beffef2aa03f88f9 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 197b880fa39d432a9b472845876bfaf0)
I20260129 02:59:20.921759 21891 raft_consensus.cc:515] T fcddab6bca684adf8ea032289a00f90e P 974b802141bd4f13beffef2aa03f88f9 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:59:20.921900 21905 raft_consensus.cc:515] T 0340229d521b432580ae926f6062e3f7 P 974b802141bd4f13beffef2aa03f88f9 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:59:20.923395 21891 leader_election.cc:290] T fcddab6bca684adf8ea032289a00f90e P 974b802141bd4f13beffef2aa03f88f9 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 197b880fa39d432a9b472845876bfaf0 (127.18.211.1:42027), 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3:37735)
I20260129 02:59:20.924072 21905 leader_election.cc:290] T 0340229d521b432580ae926f6062e3f7 P 974b802141bd4f13beffef2aa03f88f9 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 197b880fa39d432a9b472845876bfaf0 (127.18.211.1:42027), 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3:37735)
W20260129 02:59:20.947369 20480 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 1.019s user 0.005s sys 0.015s
W20260129 02:59:20.948069 20480 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 1.019s user 0.005s sys 0.015s
W20260129 02:59:20.948508 20480 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.211.3:37735, user_credentials={real_user=slave}} blocked reactor thread for 1.02001e+06us
I20260129 02:59:20.954479 21905 raft_consensus.cc:493] T 926a250fa5834e88b96ccecbefbf5086 P 974b802141bd4f13beffef2aa03f88f9 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 1c50de89a5a845668696dcb28c3badd6)
I20260129 02:59:20.955178 21891 raft_consensus.cc:493] T e33a417cf9c54cf5af73a4ee3aac4d3a P 974b802141bd4f13beffef2aa03f88f9 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 1c50de89a5a845668696dcb28c3badd6)
I20260129 02:59:20.955055 21905 raft_consensus.cc:515] T 926a250fa5834e88b96ccecbefbf5086 P 974b802141bd4f13beffef2aa03f88f9 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:59:20.955679 21891 raft_consensus.cc:515] T e33a417cf9c54cf5af73a4ee3aac4d3a P 974b802141bd4f13beffef2aa03f88f9 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:59:20.957283 21891 leader_election.cc:290] T e33a417cf9c54cf5af73a4ee3aac4d3a P 974b802141bd4f13beffef2aa03f88f9 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 197b880fa39d432a9b472845876bfaf0 (127.18.211.1:42027), 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3:37735)
I20260129 02:59:20.957288 21905 leader_election.cc:290] T 926a250fa5834e88b96ccecbefbf5086 P 974b802141bd4f13beffef2aa03f88f9 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 197b880fa39d432a9b472845876bfaf0 (127.18.211.1:42027), 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3:37735)
W20260129 02:59:20.970865 20479 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 1.053s user 0.032s sys 0.095s
W20260129 02:59:20.971244 20479 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 1.053s user 0.032s sys 0.095s
W20260129 02:59:20.971488 20479 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.211.1:42027, user_credentials={real_user=slave}} blocked reactor thread for 1.05348e+06us
I20260129 02:59:20.980854 21913 raft_consensus.cc:493] T 45cc5f3d2688486d9b6dbe7e666dedde P 974b802141bd4f13beffef2aa03f88f9 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 1c50de89a5a845668696dcb28c3badd6)
I20260129 02:59:20.981513 21913 raft_consensus.cc:515] T 45cc5f3d2688486d9b6dbe7e666dedde P 974b802141bd4f13beffef2aa03f88f9 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } }
I20260129 02:59:20.985289 21913 leader_election.cc:290] T 45cc5f3d2688486d9b6dbe7e666dedde P 974b802141bd4f13beffef2aa03f88f9 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3:37735), 7a9daf5d42a84615a6383cfec9a10608 (127.18.211.4:40439)
I20260129 02:59:21.022540 21914 raft_consensus.cc:493] T e4d18c35b0a7493e85d0a4b92956a48f P 974b802141bd4f13beffef2aa03f88f9 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 197b880fa39d432a9b472845876bfaf0)
I20260129 02:59:21.024957 21914 raft_consensus.cc:515] T e4d18c35b0a7493e85d0a4b92956a48f P 974b802141bd4f13beffef2aa03f88f9 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:59:21.027611 21914 leader_election.cc:290] T e4d18c35b0a7493e85d0a4b92956a48f P 974b802141bd4f13beffef2aa03f88f9 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 197b880fa39d432a9b472845876bfaf0 (127.18.211.1:42027), 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3:37735)
I20260129 02:59:21.036872 21839 raft_consensus.cc:493] T e7d83d4ccf684e76acb29a0dc844e1b8 P 974b802141bd4f13beffef2aa03f88f9 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 1c50de89a5a845668696dcb28c3badd6)
I20260129 02:59:21.037467 21839 raft_consensus.cc:515] T e7d83d4ccf684e76acb29a0dc844e1b8 P 974b802141bd4f13beffef2aa03f88f9 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:59:21.039718 21839 leader_election.cc:290] T e7d83d4ccf684e76acb29a0dc844e1b8 P 974b802141bd4f13beffef2aa03f88f9 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 197b880fa39d432a9b472845876bfaf0 (127.18.211.1:42027), 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3:37735)
I20260129 02:59:21.040302 21914 raft_consensus.cc:493] T 3e2d15f8640b407195210c476e09df91 P 974b802141bd4f13beffef2aa03f88f9 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 1c50de89a5a845668696dcb28c3badd6)
I20260129 02:59:21.040910 21914 raft_consensus.cc:515] T 3e2d15f8640b407195210c476e09df91 P 974b802141bd4f13beffef2aa03f88f9 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } }
I20260129 02:59:21.043493 20515 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:48544 (request call id 63) took 1441 ms. Trace:
I20260129 02:59:21.044317 21135 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3e2d15f8640b407195210c476e09df91" candidate_uuid: "974b802141bd4f13beffef2aa03f88f9" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "7a9daf5d42a84615a6383cfec9a10608" is_pre_election: true
I20260129 02:59:21.046499 21914 leader_election.cc:290] T 3e2d15f8640b407195210c476e09df91 P 974b802141bd4f13beffef2aa03f88f9 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3:37735), 7a9daf5d42a84615a6383cfec9a10608 (127.18.211.4:40439)
I20260129 02:59:21.047715 21914 raft_consensus.cc:493] T 00e79d8f25a34676b1d866eb2068a2b2 P 974b802141bd4f13beffef2aa03f88f9 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 7a9daf5d42a84615a6383cfec9a10608)
I20260129 02:59:21.047976 20587 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35754 (request call id 1388) took 1011 ms. Trace:
I20260129 02:59:21.048209 21914 raft_consensus.cc:515] T 00e79d8f25a34676b1d866eb2068a2b2 P 974b802141bd4f13beffef2aa03f88f9 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } }
I20260129 02:59:21.048382 20587 rpcz_store.cc:276] 0129 02:59:20.036416 (+ 0us) service_pool.cc:168] Inserting onto call queue
0129 02:59:20.036523 (+ 107us) service_pool.cc:225] Handling call
0129 02:59:21.047495 (+1010972us) raft_consensus.cc:1471] Updating replica for 0 ops
0129 02:59:21.047658 (+ 163us) raft_consensus.cc:1534] Early marking committed up to index 0
0129 02:59:21.047682 (+ 24us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0129 02:59:21.047722 (+ 40us) raft_consensus.cc:1645] Marking committed up to 1
0129 02:59:21.047753 (+ 31us) raft_consensus.cc:1697] Filling consensus response to leader.
0129 02:59:21.047807 (+ 54us) raft_consensus.cc:1691] UpdateReplicas() finished
0129 02:59:21.047926 (+ 119us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260129 02:59:21.051910 21912 raft_consensus.cc:493] T a609ac02fae84725ab743f95f51d33ce P 974b802141bd4f13beffef2aa03f88f9 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 7a9daf5d42a84615a6383cfec9a10608)
I20260129 02:59:21.052507 21912 raft_consensus.cc:515] T a609ac02fae84725ab743f95f51d33ce P 974b802141bd4f13beffef2aa03f88f9 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } }
I20260129 02:59:21.052997 21914 leader_election.cc:290] T 00e79d8f25a34676b1d866eb2068a2b2 P 974b802141bd4f13beffef2aa03f88f9 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 197b880fa39d432a9b472845876bfaf0 (127.18.211.1:42027), 7a9daf5d42a84615a6383cfec9a10608 (127.18.211.4:40439)
I20260129 02:59:21.053975 21133 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00e79d8f25a34676b1d866eb2068a2b2" candidate_uuid: "974b802141bd4f13beffef2aa03f88f9" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "7a9daf5d42a84615a6383cfec9a10608" is_pre_election: true
I20260129 02:59:21.054626 21912 leader_election.cc:290] T a609ac02fae84725ab743f95f51d33ce P 974b802141bd4f13beffef2aa03f88f9 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 197b880fa39d432a9b472845876bfaf0 (127.18.211.1:42027), 7a9daf5d42a84615a6383cfec9a10608 (127.18.211.4:40439)
I20260129 02:59:21.043895 20515 rpcz_store.cc:276] 0129 02:59:19.602588 (+ 0us) service_pool.cc:168] Inserting onto call queue
0129 02:59:19.606476 (+ 3888us) service_pool.cc:225] Handling call
0129 02:59:19.664424 (+ 57948us) spinlock_profiling.cc:243] Waited 2.29 ms on lock 0x7b5800254d98. stack: 00007f8fe044e373 00007f8fe044e270 00007f8fdfda0574 00007f8fe9a5c6b4 00007f8fe9a5c649 00007f8fe02c08b1 00007f8fe0442db1 00007f8fe0442c62 00007f8fe92eddd9 00007f8fe846d260 00007f8fe8535dbb 00007f8fe8535e41 00007f8fe84f7af0 00007f8fe84e2459 00007f8fe84e1f02 00007f8fe1a6ed34
0129 02:59:19.664702 (+ 278us) raft_consensus.cc:1471] Updating replica for 1 ops
0129 02:59:19.675974 (+ 11272us) raft_consensus.cc:1534] Early marking committed up to index 0
0129 02:59:19.676022 (+ 48us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0129 02:59:19.676581 (+ 559us) log.cc:844] Serialized 26 byte log entry
0129 02:59:20.039418 (+362837us) raft_consensus.cc:1645] Marking committed up to 4
0129 02:59:20.039459 (+ 41us) raft_consensus.cc:1697] Filling consensus response to leader.
0129 02:59:20.039525 (+ 66us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0129 02:59:21.043184 (+1003659us) raft_consensus.cc:1685] finished
0129 02:59:21.043206 (+ 22us) raft_consensus.cc:1691] UpdateReplicas() finished
0129 02:59:21.043409 (+ 203us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":10751,"raft.queue_time_us":11691,"raft.run_cpu_time_us":132,"raft.run_wall_time_us":9626,"spinlock_wait_cycles":5259136,"thread_start_us":309146,"threads_started":1,"wal-append.queue_time_us":1094164}
I20260129 02:59:21.056437 21805 raft_consensus.cc:493] T 87d5fe48a44a408aaf03e894a67ecb2f P 974b802141bd4f13beffef2aa03f88f9 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 7a9daf5d42a84615a6383cfec9a10608)
I20260129 02:59:21.056919 21805 raft_consensus.cc:515] T 87d5fe48a44a408aaf03e894a67ecb2f P 974b802141bd4f13beffef2aa03f88f9 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } }
W20260129 02:59:21.057622 20407 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.211.2:34517, user_credentials={real_user=slave}} blocked reactor thread for 1.0657e+06us
I20260129 02:59:21.058102 21133 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a609ac02fae84725ab743f95f51d33ce" candidate_uuid: "974b802141bd4f13beffef2aa03f88f9" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "7a9daf5d42a84615a6383cfec9a10608" is_pre_election: true
I20260129 02:59:21.058962 21805 leader_election.cc:290] T 87d5fe48a44a408aaf03e894a67ecb2f P 974b802141bd4f13beffef2aa03f88f9 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 197b880fa39d432a9b472845876bfaf0 (127.18.211.1:42027), 7a9daf5d42a84615a6383cfec9a10608 (127.18.211.4:40439)
I20260129 02:59:21.061738 20435 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.GetConsensusState from 127.0.0.1:55666 (request call id 980) took 2126 ms. Trace:
I20260129 02:59:21.061774 21133 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "87d5fe48a44a408aaf03e894a67ecb2f" candidate_uuid: "974b802141bd4f13beffef2aa03f88f9" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "7a9daf5d42a84615a6383cfec9a10608" is_pre_election: true
I20260129 02:59:21.062237 20435 rpcz_store.cc:276] 0129 02:59:18.935782 (+ 0us) service_pool.cc:168] Inserting onto call queue
0129 02:59:18.935983 (+ 201us) service_pool.cc:225] Handling call
0129 02:59:21.059587 (+2123604us) spinlock_profiling.cc:243] Waited 950 ms on lock 0x7b54003511d4. stack: 00007f8fe044e373 00007f8fe044e270 00007f8fdfda0574 00007f8fe9a5c6b4 00007f8fe9a5c649 00007f8fe34980c0 00007f8fe3536bba 00007f8fe84e758e 00007f8fe1a86437 00007f8fe1a863c2 00007f8fe1a862f1 00007f8fe1a8626c 00007f8fe1a85522 00007f8fe186d34c 00007f8fe186c786 00007f8fe186c12f
0129 02:59:21.061658 (+ 2071us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2185936768}
I20260129 02:59:21.072069 21912 raft_consensus.cc:493] T 75b0b6e82ddc4f92b1fe1d94f9a0b73b P 974b802141bd4f13beffef2aa03f88f9 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 1c50de89a5a845668696dcb28c3badd6)
I20260129 02:59:21.072904 21912 raft_consensus.cc:515] T 75b0b6e82ddc4f92b1fe1d94f9a0b73b P 974b802141bd4f13beffef2aa03f88f9 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } }
I20260129 02:59:21.079732 21912 leader_election.cc:290] T 75b0b6e82ddc4f92b1fe1d94f9a0b73b P 974b802141bd4f13beffef2aa03f88f9 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3:37735), 7a9daf5d42a84615a6383cfec9a10608 (127.18.211.4:40439)
I20260129 02:59:21.094197 21912 raft_consensus.cc:493] T febffe4a8b394368b69b1d164308fb9a P 974b802141bd4f13beffef2aa03f88f9 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 197b880fa39d432a9b472845876bfaf0)
I20260129 02:59:21.121999 21915 raft_consensus.cc:493] T 4e420f90884f4796997b5ea41e49e411 P 974b802141bd4f13beffef2aa03f88f9 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 197b880fa39d432a9b472845876bfaf0)
I20260129 02:59:21.130956 21915 raft_consensus.cc:515] T 4e420f90884f4796997b5ea41e49e411 P 974b802141bd4f13beffef2aa03f88f9 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } }
I20260129 02:59:21.135277 21915 leader_election.cc:290] T 4e420f90884f4796997b5ea41e49e411 P 974b802141bd4f13beffef2aa03f88f9 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 197b880fa39d432a9b472845876bfaf0 (127.18.211.1:42027), 7a9daf5d42a84615a6383cfec9a10608 (127.18.211.4:40439)
I20260129 02:59:21.135663 21132 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4e420f90884f4796997b5ea41e49e411" candidate_uuid: "974b802141bd4f13beffef2aa03f88f9" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "7a9daf5d42a84615a6383cfec9a10608" is_pre_election: true
I20260129 02:59:21.136574 21132 raft_consensus.cc:2468] T 4e420f90884f4796997b5ea41e49e411 P 7a9daf5d42a84615a6383cfec9a10608 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 974b802141bd4f13beffef2aa03f88f9 in term 1.
I20260129 02:59:21.138012 20480 leader_election.cc:304] T 4e420f90884f4796997b5ea41e49e411 P 974b802141bd4f13beffef2aa03f88f9 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7a9daf5d42a84615a6383cfec9a10608, 974b802141bd4f13beffef2aa03f88f9; no voters:
I20260129 02:59:21.139021 21839 raft_consensus.cc:493] T 96609ac68bdf4993a12e1c23355ca013 P 974b802141bd4f13beffef2aa03f88f9 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 197b880fa39d432a9b472845876bfaf0)
I20260129 02:59:21.137219 21912 raft_consensus.cc:515] T febffe4a8b394368b69b1d164308fb9a P 974b802141bd4f13beffef2aa03f88f9 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } }
I20260129 02:59:21.139734 21839 raft_consensus.cc:515] T 96609ac68bdf4993a12e1c23355ca013 P 974b802141bd4f13beffef2aa03f88f9 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } attrs { replace: true } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: NON_VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: true } }
I20260129 02:59:21.080258 21133 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "75b0b6e82ddc4f92b1fe1d94f9a0b73b" candidate_uuid: "974b802141bd4f13beffef2aa03f88f9" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "7a9daf5d42a84615a6383cfec9a10608" is_pre_election: true
I20260129 02:59:21.142325 21839 leader_election.cc:290] T 96609ac68bdf4993a12e1c23355ca013 P 974b802141bd4f13beffef2aa03f88f9 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 197b880fa39d432a9b472845876bfaf0 (127.18.211.1:42027), 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3:37735)
I20260129 02:59:21.142947 21839 raft_consensus.cc:2804] T 4e420f90884f4796997b5ea41e49e411 P 974b802141bd4f13beffef2aa03f88f9 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260129 02:59:21.143345 21839 raft_consensus.cc:493] T 4e420f90884f4796997b5ea41e49e411 P 974b802141bd4f13beffef2aa03f88f9 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 197b880fa39d432a9b472845876bfaf0)
I20260129 02:59:21.147850 21912 leader_election.cc:290] T febffe4a8b394368b69b1d164308fb9a P 974b802141bd4f13beffef2aa03f88f9 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 197b880fa39d432a9b472845876bfaf0 (127.18.211.1:42027), 7a9daf5d42a84615a6383cfec9a10608 (127.18.211.4:40439)
I20260129 02:59:21.143792 21839 raft_consensus.cc:3060] T 4e420f90884f4796997b5ea41e49e411 P 974b802141bd4f13beffef2aa03f88f9 [term 1 FOLLOWER]: Advancing to term 2
I20260129 02:59:21.162328 21912 raft_consensus.cc:493] T 3906b080a5b34a74b8b64128635e8997 P 974b802141bd4f13beffef2aa03f88f9 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 197b880fa39d432a9b472845876bfaf0)
I20260129 02:59:21.162955 21912 raft_consensus.cc:515] T 3906b080a5b34a74b8b64128635e8997 P 974b802141bd4f13beffef2aa03f88f9 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:59:21.165221 21912 leader_election.cc:290] T 3906b080a5b34a74b8b64128635e8997 P 974b802141bd4f13beffef2aa03f88f9 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 197b880fa39d432a9b472845876bfaf0 (127.18.211.1:42027), 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3:37735)
I20260129 02:59:21.166170 20587 tablet_service.cc:2066] Received LeaderStepDown RPC: tablet_id: "e33a417cf9c54cf5af73a4ee3aac4d3a"
dest_uuid: "1c50de89a5a845668696dcb28c3badd6"
mode: GRACEFUL
new_leader_uuid: "197b880fa39d432a9b472845876bfaf0"
from {username='slave'} at 127.0.0.1:54860
I20260129 02:59:21.166980 20587 raft_consensus.cc:606] T e33a417cf9c54cf5af73a4ee3aac4d3a P 1c50de89a5a845668696dcb28c3badd6 [term 1 LEADER]: Received request to transfer leadership to TS 197b880fa39d432a9b472845876bfaf0
I20260129 02:59:21.165324 21839 raft_consensus.cc:515] T 4e420f90884f4796997b5ea41e49e411 P 974b802141bd4f13beffef2aa03f88f9 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } }
I20260129 02:59:21.169592 21839 leader_election.cc:290] T 4e420f90884f4796997b5ea41e49e411 P 974b802141bd4f13beffef2aa03f88f9 [CANDIDATE]: Term 2 election: Requested vote from peers 197b880fa39d432a9b472845876bfaf0 (127.18.211.1:42027), 7a9daf5d42a84615a6383cfec9a10608 (127.18.211.4:40439)
I20260129 02:59:21.172597 21133 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "febffe4a8b394368b69b1d164308fb9a" candidate_uuid: "974b802141bd4f13beffef2aa03f88f9" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "7a9daf5d42a84615a6383cfec9a10608" is_pre_election: true
I20260129 02:59:21.173930 21132 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4e420f90884f4796997b5ea41e49e411" candidate_uuid: "974b802141bd4f13beffef2aa03f88f9" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "7a9daf5d42a84615a6383cfec9a10608"
I20260129 02:59:21.174538 21132 raft_consensus.cc:3060] T 4e420f90884f4796997b5ea41e49e411 P 7a9daf5d42a84615a6383cfec9a10608 [term 1 FOLLOWER]: Advancing to term 2
I20260129 02:59:21.192325 21132 raft_consensus.cc:2468] T 4e420f90884f4796997b5ea41e49e411 P 7a9daf5d42a84615a6383cfec9a10608 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 974b802141bd4f13beffef2aa03f88f9 in term 2.
I20260129 02:59:21.193691 20480 leader_election.cc:304] T 4e420f90884f4796997b5ea41e49e411 P 974b802141bd4f13beffef2aa03f88f9 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7a9daf5d42a84615a6383cfec9a10608, 974b802141bd4f13beffef2aa03f88f9; no voters:
I20260129 02:59:21.194790 21912 raft_consensus.cc:2804] T 4e420f90884f4796997b5ea41e49e411 P 974b802141bd4f13beffef2aa03f88f9 [term 2 FOLLOWER]: Leader election won for term 2
I20260129 02:59:21.195307 21912 raft_consensus.cc:697] T 4e420f90884f4796997b5ea41e49e411 P 974b802141bd4f13beffef2aa03f88f9 [term 2 LEADER]: Becoming Leader. State: Replica: 974b802141bd4f13beffef2aa03f88f9, State: Running, Role: LEADER
I20260129 02:59:21.196235 21912 consensus_queue.cc:237] T 4e420f90884f4796997b5ea41e49e411 P 974b802141bd4f13beffef2aa03f88f9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } }
I20260129 02:59:21.197408 21839 raft_consensus.cc:493] T 913fc7d751ed4b7388d889aac30a5f9b P 974b802141bd4f13beffef2aa03f88f9 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 197b880fa39d432a9b472845876bfaf0)
I20260129 02:59:21.197927 21839 raft_consensus.cc:515] T 913fc7d751ed4b7388d889aac30a5f9b P 974b802141bd4f13beffef2aa03f88f9 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:59:21.200764 21839 leader_election.cc:290] T 913fc7d751ed4b7388d889aac30a5f9b P 974b802141bd4f13beffef2aa03f88f9 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 197b880fa39d432a9b472845876bfaf0 (127.18.211.1:42027), 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3:37735)
W20260129 02:59:21.203101 20581 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 1.161s user 0.004s sys 0.016s
W20260129 02:59:21.203501 20581 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 1.161s user 0.004s sys 0.016s
I20260129 02:59:21.205552 21891 raft_consensus.cc:493] T 1ce4ddf5b10e4ae8b193b13b2e5936ee P 974b802141bd4f13beffef2aa03f88f9 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 197b880fa39d432a9b472845876bfaf0)
I20260129 02:59:21.206132 21891 raft_consensus.cc:515] T 1ce4ddf5b10e4ae8b193b13b2e5936ee P 974b802141bd4f13beffef2aa03f88f9 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:59:21.213826 21884 raft_consensus.cc:493] T 926a250fa5834e88b96ccecbefbf5086 P 197b880fa39d432a9b472845876bfaf0 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 1c50de89a5a845668696dcb28c3badd6)
I20260129 02:59:21.214421 21884 raft_consensus.cc:515] T 926a250fa5834e88b96ccecbefbf5086 P 197b880fa39d432a9b472845876bfaf0 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:59:21.217592 21893 raft_consensus.cc:493] T fcddab6bca684adf8ea032289a00f90e P 197b880fa39d432a9b472845876bfaf0 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260129 02:59:21.218124 21893 raft_consensus.cc:515] T fcddab6bca684adf8ea032289a00f90e P 197b880fa39d432a9b472845876bfaf0 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
W20260129 02:59:21.220786 20556 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.211.2:34517, user_credentials={real_user=slave}} blocked reactor thread for 427664us
I20260129 02:59:21.222329 21839 raft_consensus.cc:493] T 0678b8b468bd43cdbc3e485900844821 P 974b802141bd4f13beffef2aa03f88f9 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 7a9daf5d42a84615a6383cfec9a10608)
I20260129 02:59:21.222910 21839 raft_consensus.cc:515] T 0678b8b468bd43cdbc3e485900844821 P 974b802141bd4f13beffef2aa03f88f9 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } }
I20260129 02:59:21.225102 21839 leader_election.cc:290] T 0678b8b468bd43cdbc3e485900844821 P 974b802141bd4f13beffef2aa03f88f9 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 197b880fa39d432a9b472845876bfaf0 (127.18.211.1:42027), 7a9daf5d42a84615a6383cfec9a10608 (127.18.211.4:40439)
I20260129 02:59:21.226339 21778 raft_consensus.cc:493] T 7d9494f5584d4d5db8b9adf1c8fc888b P 1c50de89a5a845668696dcb28c3badd6 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 974b802141bd4f13beffef2aa03f88f9)
I20260129 02:59:21.226658 21788 raft_consensus.cc:493] T a5a05fbcbbcf4916bbc01fd1a5b5e90d P 197b880fa39d432a9b472845876bfaf0 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 1c50de89a5a845668696dcb28c3badd6)
I20260129 02:59:21.226930 21778 raft_consensus.cc:515] T 7d9494f5584d4d5db8b9adf1c8fc888b P 1c50de89a5a845668696dcb28c3badd6 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:59:21.227267 21788 raft_consensus.cc:515] T a5a05fbcbbcf4916bbc01fd1a5b5e90d P 197b880fa39d432a9b472845876bfaf0 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:59:21.229235 21662 raft_consensus.cc:493] T cedce331e1584ac8b636c9b8a72b2ba0 P 197b880fa39d432a9b472845876bfaf0 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 1c50de89a5a845668696dcb28c3badd6)
I20260129 02:59:21.229804 21662 raft_consensus.cc:515] T cedce331e1584ac8b636c9b8a72b2ba0 P 197b880fa39d432a9b472845876bfaf0 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:59:21.231828 21666 raft_consensus.cc:493] T fd8311b779af4e1fa1402b2a02a602f2 P 197b880fa39d432a9b472845876bfaf0 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 1c50de89a5a845668696dcb28c3badd6)
I20260129 02:59:21.232400 21666 raft_consensus.cc:515] T fd8311b779af4e1fa1402b2a02a602f2 P 197b880fa39d432a9b472845876bfaf0 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } }
I20260129 02:59:21.234256 21786 raft_consensus.cc:493] T add8d4923d1a489e86882c3a25692b95 P 197b880fa39d432a9b472845876bfaf0 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 1c50de89a5a845668696dcb28c3badd6)
I20260129 02:59:21.234362 21920 raft_consensus.cc:493] T 368953c9102643b2a21f99af36b51905 P 1c50de89a5a845668696dcb28c3badd6 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 197b880fa39d432a9b472845876bfaf0)
I20260129 02:59:21.235188 21920 raft_consensus.cc:515] T 368953c9102643b2a21f99af36b51905 P 1c50de89a5a845668696dcb28c3badd6 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:59:21.235791 21886 raft_consensus.cc:493] T 7d9494f5584d4d5db8b9adf1c8fc888b P 197b880fa39d432a9b472845876bfaf0 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 974b802141bd4f13beffef2aa03f88f9)
I20260129 02:59:21.236536 21886 raft_consensus.cc:515] T 7d9494f5584d4d5db8b9adf1c8fc888b P 197b880fa39d432a9b472845876bfaf0 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:59:21.237890 21920 leader_election.cc:290] T 368953c9102643b2a21f99af36b51905 P 1c50de89a5a845668696dcb28c3badd6 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 197b880fa39d432a9b472845876bfaf0 (127.18.211.1:42027), 974b802141bd4f13beffef2aa03f88f9 (127.18.211.2:34517)
I20260129 02:59:21.239059 21846 raft_consensus.cc:493] T 0f20af7fbe5d4db89a6f17948d98912c P 197b880fa39d432a9b472845876bfaf0 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 974b802141bd4f13beffef2aa03f88f9)
I20260129 02:59:21.240979 21829 raft_consensus.cc:493] T a4c1226b987b468fa84dfc6ba0e2dbe1 P 1c50de89a5a845668696dcb28c3badd6 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 974b802141bd4f13beffef2aa03f88f9)
I20260129 02:59:21.241626 21790 raft_consensus.cc:493] T a0fcc58d6fb6473cafbd9d34d5195269 P 1c50de89a5a845668696dcb28c3badd6 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 974b802141bd4f13beffef2aa03f88f9)
I20260129 02:59:21.241524 21829 raft_consensus.cc:515] T a4c1226b987b468fa84dfc6ba0e2dbe1 P 1c50de89a5a845668696dcb28c3badd6 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:59:21.242141 21790 raft_consensus.cc:515] T a0fcc58d6fb6473cafbd9d34d5195269 P 1c50de89a5a845668696dcb28c3badd6 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:59:21.243512 21829 leader_election.cc:290] T a4c1226b987b468fa84dfc6ba0e2dbe1 P 1c50de89a5a845668696dcb28c3badd6 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 197b880fa39d432a9b472845876bfaf0 (127.18.211.1:42027), 974b802141bd4f13beffef2aa03f88f9 (127.18.211.2:34517)
I20260129 02:59:21.244038 21829 raft_consensus.cc:493] T 4e3805ffa70242e8ade66da260dddf66 P 1c50de89a5a845668696dcb28c3badd6 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 7a9daf5d42a84615a6383cfec9a10608)
I20260129 02:59:21.244398 21896 raft_consensus.cc:493] T 76f713c3331540d2a5b76164103e9eb1 P 1c50de89a5a845668696dcb28c3badd6 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 974b802141bd4f13beffef2aa03f88f9)
I20260129 02:59:21.244506 21829 raft_consensus.cc:515] T 4e3805ffa70242e8ade66da260dddf66 P 1c50de89a5a845668696dcb28c3badd6 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } }
I20260129 02:59:21.244920 21896 raft_consensus.cc:515] T 76f713c3331540d2a5b76164103e9eb1 P 1c50de89a5a845668696dcb28c3badd6 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:59:21.247061 21880 raft_consensus.cc:493] T 4e3805ffa70242e8ade66da260dddf66 P 197b880fa39d432a9b472845876bfaf0 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 7a9daf5d42a84615a6383cfec9a10608)
I20260129 02:59:21.234375 21666 leader_election.cc:290] T fd8311b779af4e1fa1402b2a02a602f2 P 197b880fa39d432a9b472845876bfaf0 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3:37735), 7a9daf5d42a84615a6383cfec9a10608 (127.18.211.4:40439)
I20260129 02:59:21.248869 21133 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0678b8b468bd43cdbc3e485900844821" candidate_uuid: "974b802141bd4f13beffef2aa03f88f9" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "7a9daf5d42a84615a6383cfec9a10608" is_pre_election: true
I20260129 02:59:21.248013 21787 raft_consensus.cc:493] T 4f8f3a38f6e24ba4857875181f447442 P 197b880fa39d432a9b472845876bfaf0 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 974b802141bd4f13beffef2aa03f88f9)
W20260129 02:59:21.250274 20407 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.211.2:34517, user_credentials={real_user=slave}} blocked reactor thread for 50616.6us
I20260129 02:59:21.246500 21829 leader_election.cc:290] T 4e3805ffa70242e8ade66da260dddf66 P 1c50de89a5a845668696dcb28c3badd6 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 197b880fa39d432a9b472845876bfaf0 (127.18.211.1:42027), 7a9daf5d42a84615a6383cfec9a10608 (127.18.211.4:40439)
I20260129 02:59:21.252241 21790 leader_election.cc:290] T a0fcc58d6fb6473cafbd9d34d5195269 P 1c50de89a5a845668696dcb28c3badd6 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 197b880fa39d432a9b472845876bfaf0 (127.18.211.1:42027), 974b802141bd4f13beffef2aa03f88f9 (127.18.211.2:34517)
I20260129 02:59:21.247581 21880 raft_consensus.cc:515] T 4e3805ffa70242e8ade66da260dddf66 P 197b880fa39d432a9b472845876bfaf0 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } }
I20260129 02:59:21.260099 21884 leader_election.cc:290] T 926a250fa5834e88b96ccecbefbf5086 P 197b880fa39d432a9b472845876bfaf0 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 974b802141bd4f13beffef2aa03f88f9 (127.18.211.2:34517), 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3:37735)
I20260129 02:59:21.276932 21896 leader_election.cc:290] T 76f713c3331540d2a5b76164103e9eb1 P 1c50de89a5a845668696dcb28c3badd6 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 197b880fa39d432a9b472845876bfaf0 (127.18.211.1:42027), 974b802141bd4f13beffef2aa03f88f9 (127.18.211.2:34517)
I20260129 02:59:21.277369 21913 raft_consensus.cc:493] T a1a39dcd328e4268ae41b988c66b4a75 P 974b802141bd4f13beffef2aa03f88f9 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 1c50de89a5a845668696dcb28c3badd6)
I20260129 02:59:21.278079 21913 raft_consensus.cc:515] T a1a39dcd328e4268ae41b988c66b4a75 P 974b802141bd4f13beffef2aa03f88f9 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } } peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } }
I20260129 02:59:21.281843 21839 raft_consensus.cc:493] T cedce331e1584ac8b636c9b8a72b2ba0 P 974b802141bd4f13beffef2aa03f88f9 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 1c50de89a5a845668696dcb28c3badd6)
I20260129 02:59:21.282522 21839 raft_consensus.cc:515] T cedce331e1584ac8b636c9b8a72b2ba0 P 974b802141bd4f13beffef2aa03f88f9 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:59:21.284587 21839 leader_election.cc:290] T cedce331e1584ac8b636c9b8a72b2ba0 P 974b802141bd4f13beffef2aa03f88f9 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 197b880fa39d432a9b472845876bfaf0 (127.18.211.1:42027), 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3:37735)
I20260129 02:59:21.285001 21788 leader_election.cc:290] T a5a05fbcbbcf4916bbc01fd1a5b5e90d P 197b880fa39d432a9b472845876bfaf0 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 974b802141bd4f13beffef2aa03f88f9 (127.18.211.2:34517), 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3:37735)
I20260129 02:59:21.285095 21839 raft_consensus.cc:493] T 505580c93c694bbb8ea96fdf9b17ab46 P 974b802141bd4f13beffef2aa03f88f9 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 1c50de89a5a845668696dcb28c3badd6)
I20260129 02:59:21.285229 21914 raft_consensus.cc:493] T add8d4923d1a489e86882c3a25692b95 P 974b802141bd4f13beffef2aa03f88f9 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 1c50de89a5a845668696dcb28c3badd6)
I20260129 02:59:21.285712 21920 raft_consensus.cc:493] T 4408006eb5174de1896b037cf3f18457 P 1c50de89a5a845668696dcb28c3badd6 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 974b802141bd4f13beffef2aa03f88f9)
I20260129 02:59:21.285983 21914 raft_consensus.cc:515] T add8d4923d1a489e86882c3a25692b95 P 974b802141bd4f13beffef2aa03f88f9 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } }
I20260129 02:59:21.286273 21920 raft_consensus.cc:515] T 4408006eb5174de1896b037cf3f18457 P 1c50de89a5a845668696dcb28c3badd6 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:59:21.288120 21880 leader_election.cc:290] T 4e3805ffa70242e8ade66da260dddf66 P 197b880fa39d432a9b472845876bfaf0 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3:37735), 7a9daf5d42a84615a6383cfec9a10608 (127.18.211.4:40439)
I20260129 02:59:21.288511 21914 leader_election.cc:290] T add8d4923d1a489e86882c3a25692b95 P 974b802141bd4f13beffef2aa03f88f9 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3:37735), 7a9daf5d42a84615a6383cfec9a10608 (127.18.211.4:40439)
I20260129 02:59:21.239779 21846 raft_consensus.cc:515] T 0f20af7fbe5d4db89a6f17948d98912c P 197b880fa39d432a9b472845876bfaf0 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:59:21.235188 21786 raft_consensus.cc:515] T add8d4923d1a489e86882c3a25692b95 P 197b880fa39d432a9b472845876bfaf0 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } attrs { replace: true } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } }
I20260129 02:59:21.289029 21839 raft_consensus.cc:515] T 505580c93c694bbb8ea96fdf9b17ab46 P 974b802141bd4f13beffef2aa03f88f9 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } }
I20260129 02:59:21.249787 21787 raft_consensus.cc:515] T 4f8f3a38f6e24ba4857875181f447442 P 197b880fa39d432a9b472845876bfaf0 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:59:21.293879 21915 raft_consensus.cc:493] T a644c02b16af47679824ccd060ac10fd P 974b802141bd4f13beffef2aa03f88f9 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 1c50de89a5a845668696dcb28c3badd6)
W20260129 02:59:21.298760 20556 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.211.2:34517, user_credentials={real_user=slave}} blocked reactor thread for 77098.1us
I20260129 02:59:21.303938 20513 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "926a250fa5834e88b96ccecbefbf5086" candidate_uuid: "197b880fa39d432a9b472845876bfaf0" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "974b802141bd4f13beffef2aa03f88f9" is_pre_election: true
I20260129 02:59:21.304816 20513 raft_consensus.cc:2468] T 926a250fa5834e88b96ccecbefbf5086 P 974b802141bd4f13beffef2aa03f88f9 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 197b880fa39d432a9b472845876bfaf0 in term 2.
I20260129 02:59:21.297365 21815 raft_consensus.cc:493] T e4d18c35b0a7493e85d0a4b92956a48f P 1c50de89a5a845668696dcb28c3badd6 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 197b880fa39d432a9b472845876bfaf0)
I20260129 02:59:21.331157 21815 raft_consensus.cc:515] T e4d18c35b0a7493e85d0a4b92956a48f P 1c50de89a5a845668696dcb28c3badd6 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:59:21.331516 21896 raft_consensus.cc:993] T 505580c93c694bbb8ea96fdf9b17ab46 P 1c50de89a5a845668696dcb28c3badd6: : Instructing follower 974b802141bd4f13beffef2aa03f88f9 to start an election
I20260129 02:59:21.332535 21899 raft_consensus.cc:1081] T 505580c93c694bbb8ea96fdf9b17ab46 P 1c50de89a5a845668696dcb28c3badd6 [term 1 LEADER]: Signalling peer 974b802141bd4f13beffef2aa03f88f9 to start an election
I20260129 02:59:21.285657 21829 raft_consensus.cc:493] T 0f20af7fbe5d4db89a6f17948d98912c P 1c50de89a5a845668696dcb28c3badd6 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 974b802141bd4f13beffef2aa03f88f9)
I20260129 02:59:21.331492 20581 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35754 (request call id 1397) took 1267 ms. Trace:
I20260129 02:59:21.281929 21913 leader_election.cc:290] T a1a39dcd328e4268ae41b988c66b4a75 P 974b802141bd4f13beffef2aa03f88f9 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3:37735), 197b880fa39d432a9b472845876bfaf0 (127.18.211.1:42027)
I20260129 02:59:21.296124 21789 raft_consensus.cc:493] T 2e58e9a5a8aa4cecafb464addb4f35ed P 197b880fa39d432a9b472845876bfaf0 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 7a9daf5d42a84615a6383cfec9a10608)
I20260129 02:59:21.296830 21790 raft_consensus.cc:493] T cb0b3f259ebc4310927338a8740e87b8 P 1c50de89a5a845668696dcb28c3badd6 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 974b802141bd4f13beffef2aa03f88f9)
I20260129 02:59:21.296482 21666 raft_consensus.cc:493] T cb4727386a3e48aca5d71cc6c611c197 P 197b880fa39d432a9b472845876bfaf0 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 974b802141bd4f13beffef2aa03f88f9)
I20260129 02:59:21.335188 20589 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e33a417cf9c54cf5af73a4ee3aac4d3a" candidate_uuid: "974b802141bd4f13beffef2aa03f88f9" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1c50de89a5a845668696dcb28c3badd6" is_pre_election: true
I20260129 02:59:21.435586 21805 raft_consensus.cc:493] T a5a05fbcbbcf4916bbc01fd1a5b5e90d P 974b802141bd4f13beffef2aa03f88f9 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 1c50de89a5a845668696dcb28c3badd6)
I20260129 02:59:21.333478 20583 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fcddab6bca684adf8ea032289a00f90e" candidate_uuid: "974b802141bd4f13beffef2aa03f88f9" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "1c50de89a5a845668696dcb28c3badd6" is_pre_election: true
I20260129 02:59:21.351974 21883 raft_consensus.cc:476] T 0340229d521b432580ae926f6062e3f7 P 197b880fa39d432a9b472845876bfaf0 [term 3 LEADER]: Not starting pre-election -- already a leader
I20260129 02:59:21.458537 21805 raft_consensus.cc:515] T a5a05fbcbbcf4916bbc01fd1a5b5e90d P 974b802141bd4f13beffef2aa03f88f9 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:59:21.460283 21666 raft_consensus.cc:515] T cb4727386a3e48aca5d71cc6c611c197 P 197b880fa39d432a9b472845876bfaf0 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:59:21.352319 20585 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "926a250fa5834e88b96ccecbefbf5086" candidate_uuid: "974b802141bd4f13beffef2aa03f88f9" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "1c50de89a5a845668696dcb28c3badd6" is_pre_election: true
I20260129 02:59:21.461820 21883 raft_consensus.cc:1064] T 96609ac68bdf4993a12e1c23355ca013 P 197b880fa39d432a9b472845876bfaf0: attempting to promote NON_VOTER 7a9daf5d42a84615a6383cfec9a10608 to VOTER
I20260129 02:59:21.462376 21666 leader_election.cc:290] T cb4727386a3e48aca5d71cc6c611c197 P 197b880fa39d432a9b472845876bfaf0 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 974b802141bd4f13beffef2aa03f88f9 (127.18.211.2:34517), 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3:37735)
I20260129 02:59:21.463140 21666 raft_consensus.cc:493] T 18f30ebaa4174fd28c958e72d0cc7e24 P 197b880fa39d432a9b472845876bfaf0 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 7a9daf5d42a84615a6383cfec9a10608)
I20260129 02:59:21.440160 21791 raft_consensus.cc:493] T 18f30ebaa4174fd28c958e72d0cc7e24 P 1c50de89a5a845668696dcb28c3badd6 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 7a9daf5d42a84615a6383cfec9a10608)
I20260129 02:59:21.463515 21805 leader_election.cc:290] T a5a05fbcbbcf4916bbc01fd1a5b5e90d P 974b802141bd4f13beffef2aa03f88f9 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 197b880fa39d432a9b472845876bfaf0 (127.18.211.1:42027), 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3:37735)
I20260129 02:59:21.481307 21805 raft_consensus.cc:493] T b189f08e16a74a11980cc55d99359a0a P 974b802141bd4f13beffef2aa03f88f9 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 197b880fa39d432a9b472845876bfaf0)
I20260129 02:59:21.396148 21839 leader_election.cc:290] T 505580c93c694bbb8ea96fdf9b17ab46 P 974b802141bd4f13beffef2aa03f88f9 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3:37735), 7a9daf5d42a84615a6383cfec9a10608 (127.18.211.4:40439)
I20260129 02:59:21.483050 20588 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "913fc7d751ed4b7388d889aac30a5f9b" candidate_uuid: "974b802141bd4f13beffef2aa03f88f9" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1c50de89a5a845668696dcb28c3badd6" is_pre_election: true
I20260129 02:59:21.484134 20588 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3e2d15f8640b407195210c476e09df91" candidate_uuid: "974b802141bd4f13beffef2aa03f88f9" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "1c50de89a5a845668696dcb28c3badd6" is_pre_election: true
I20260129 02:59:21.485039 20588 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e7d83d4ccf684e76acb29a0dc844e1b8" candidate_uuid: "974b802141bd4f13beffef2aa03f88f9" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "1c50de89a5a845668696dcb28c3badd6" is_pre_election: true
I20260129 02:59:21.485949 20588 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a1a39dcd328e4268ae41b988c66b4a75" candidate_uuid: "974b802141bd4f13beffef2aa03f88f9" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "1c50de89a5a845668696dcb28c3badd6" is_pre_election: true
I20260129 02:59:21.382023 21787 leader_election.cc:290] T 4f8f3a38f6e24ba4857875181f447442 P 197b880fa39d432a9b472845876bfaf0 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 974b802141bd4f13beffef2aa03f88f9 (127.18.211.2:34517), 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3:37735)
I20260129 02:59:21.335229 20515 raft_consensus.cc:1240] T 4e420f90884f4796997b5ea41e49e411 P 974b802141bd4f13beffef2aa03f88f9 [term 2 LEADER]: Rejecting Update request from peer 197b880fa39d432a9b472845876bfaf0 for earlier term 1. Current term is 2. Ops: []
I20260129 02:59:21.486881 20588 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3906b080a5b34a74b8b64128635e8997" candidate_uuid: "974b802141bd4f13beffef2aa03f88f9" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1c50de89a5a845668696dcb28c3badd6" is_pre_election: true
I20260129 02:59:21.356691 20511 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0f20af7fbe5d4db89a6f17948d98912c" candidate_uuid: "197b880fa39d432a9b472845876bfaf0" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "974b802141bd4f13beffef2aa03f88f9" is_pre_election: true
I20260129 02:59:21.466500 21899 raft_consensus.cc:493] T cb4727386a3e48aca5d71cc6c611c197 P 1c50de89a5a845668696dcb28c3badd6 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 974b802141bd4f13beffef2aa03f88f9)
I20260129 02:59:21.348558 21815 leader_election.cc:290] T e4d18c35b0a7493e85d0a4b92956a48f P 1c50de89a5a845668696dcb28c3badd6 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 197b880fa39d432a9b472845876bfaf0 (127.18.211.1:42027), 974b802141bd4f13beffef2aa03f88f9 (127.18.211.2:34517)
I20260129 02:59:21.333541 21915 raft_consensus.cc:515] T a644c02b16af47679824ccd060ac10fd P 974b802141bd4f13beffef2aa03f88f9 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } }
I20260129 02:59:21.355058 21846 leader_election.cc:290] T 0f20af7fbe5d4db89a6f17948d98912c P 197b880fa39d432a9b472845876bfaf0 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 974b802141bd4f13beffef2aa03f88f9 (127.18.211.2:34517), 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3:37735)
I20260129 02:59:21.356786 20407 leader_election.cc:304] T 926a250fa5834e88b96ccecbefbf5086 P 197b880fa39d432a9b472845876bfaf0 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 197b880fa39d432a9b472845876bfaf0, 974b802141bd4f13beffef2aa03f88f9; no voters:
I20260129 02:59:21.383204 21786 leader_election.cc:290] T add8d4923d1a489e86882c3a25692b95 P 197b880fa39d432a9b472845876bfaf0 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 974b802141bd4f13beffef2aa03f88f9 (127.18.211.2:34517), 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3:37735), 7a9daf5d42a84615a6383cfec9a10608 (127.18.211.4:40439)
I20260129 02:59:21.489022 21786 raft_consensus.cc:2804] T 926a250fa5834e88b96ccecbefbf5086 P 197b880fa39d432a9b472845876bfaf0 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260129 02:59:21.489185 20511 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "368953c9102643b2a21f99af36b51905" candidate_uuid: "1c50de89a5a845668696dcb28c3badd6" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "974b802141bd4f13beffef2aa03f88f9" is_pre_election: true
I20260129 02:59:21.489483 21786 raft_consensus.cc:493] T 926a250fa5834e88b96ccecbefbf5086 P 197b880fa39d432a9b472845876bfaf0 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 1c50de89a5a845668696dcb28c3badd6)
I20260129 02:59:21.489805 20511 raft_consensus.cc:2468] T 368953c9102643b2a21f99af36b51905 P 974b802141bd4f13beffef2aa03f88f9 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1c50de89a5a845668696dcb28c3badd6 in term 2.
I20260129 02:59:21.489888 21786 raft_consensus.cc:3060] T 926a250fa5834e88b96ccecbefbf5086 P 197b880fa39d432a9b472845876bfaf0 [term 2 FOLLOWER]: Advancing to term 3
I20260129 02:59:21.490600 21915 leader_election.cc:290] T a644c02b16af47679824ccd060ac10fd P 974b802141bd4f13beffef2aa03f88f9 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3:37735), 7a9daf5d42a84615a6383cfec9a10608 (127.18.211.4:40439)
I20260129 02:59:21.299141 21891 leader_election.cc:290] T 1ce4ddf5b10e4ae8b193b13b2e5936ee P 974b802141bd4f13beffef2aa03f88f9 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 197b880fa39d432a9b472845876bfaf0 (127.18.211.1:42027), 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3:37735)
I20260129 02:59:21.299372 21880 raft_consensus.cc:493] T a0fcc58d6fb6473cafbd9d34d5195269 P 197b880fa39d432a9b472845876bfaf0 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 974b802141bd4f13beffef2aa03f88f9)
I20260129 02:59:21.360150 21923 raft_consensus.cc:493] T e33a417cf9c54cf5af73a4ee3aac4d3a P 197b880fa39d432a9b472845876bfaf0 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 1c50de89a5a845668696dcb28c3badd6)
I20260129 02:59:21.356276 20514 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7d9494f5584d4d5db8b9adf1c8fc888b" candidate_uuid: "197b880fa39d432a9b472845876bfaf0" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "974b802141bd4f13beffef2aa03f88f9" is_pre_election: true
I20260129 02:59:21.395134 21778 leader_election.cc:290] T 7d9494f5584d4d5db8b9adf1c8fc888b P 1c50de89a5a845668696dcb28c3badd6 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 197b880fa39d432a9b472845876bfaf0 (127.18.211.1:42027), 974b802141bd4f13beffef2aa03f88f9 (127.18.211.2:34517)
I20260129 02:59:21.468350 20508 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fcddab6bca684adf8ea032289a00f90e" candidate_uuid: "197b880fa39d432a9b472845876bfaf0" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "974b802141bd4f13beffef2aa03f88f9" is_pre_election: true
I20260129 02:59:21.468461 20323 catalog_manager.cc:5167] ChangeConfig:REMOVE_PEER RPC for tablet add8d4923d1a489e86882c3a25692b95 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260129 02:59:21.468701 20586 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e4d18c35b0a7493e85d0a4b92956a48f" candidate_uuid: "974b802141bd4f13beffef2aa03f88f9" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1c50de89a5a845668696dcb28c3badd6" is_pre_election: true
I20260129 02:59:21.508718 20508 raft_consensus.cc:2410] T fcddab6bca684adf8ea032289a00f90e P 974b802141bd4f13beffef2aa03f88f9 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 197b880fa39d432a9b472845876bfaf0 for term 5 because replica has last-logged OpId of term: 4 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 3.
I20260129 02:59:21.508798 20586 raft_consensus.cc:2468] T e4d18c35b0a7493e85d0a4b92956a48f P 1c50de89a5a845668696dcb28c3badd6 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 974b802141bd4f13beffef2aa03f88f9 in term 1.
I20260129 02:59:21.509879 20586 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a5a05fbcbbcf4916bbc01fd1a5b5e90d" candidate_uuid: "974b802141bd4f13beffef2aa03f88f9" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "1c50de89a5a845668696dcb28c3badd6" is_pre_election: true
I20260129 02:59:21.330174 21788 raft_consensus.cc:493] T 78aa39096868415d9afc2ffcb248e198 P 197b880fa39d432a9b472845876bfaf0 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 1c50de89a5a845668696dcb28c3badd6)
I20260129 02:59:21.468854 20587 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0340229d521b432580ae926f6062e3f7" candidate_uuid: "974b802141bd4f13beffef2aa03f88f9" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "1c50de89a5a845668696dcb28c3badd6" is_pre_election: true
I20260129 02:59:21.352787 20590 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "45cc5f3d2688486d9b6dbe7e666dedde" candidate_uuid: "974b802141bd4f13beffef2aa03f88f9" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "1c50de89a5a845668696dcb28c3badd6" is_pre_election: true
I20260129 02:59:21.469703 21913 raft_consensus.cc:493] T a8479d1fe0bd41e5946186e3a6ca8c7a P 974b802141bd4f13beffef2aa03f88f9 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 197b880fa39d432a9b472845876bfaf0)
I20260129 02:59:21.469348 20583 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "96609ac68bdf4993a12e1c23355ca013" candidate_uuid: "974b802141bd4f13beffef2aa03f88f9" candidate_term: 3 candidate_status { last_received { term: 2 index: 3 } } ignore_live_leader: false dest_uuid: "1c50de89a5a845668696dcb28c3badd6" is_pre_election: true
I20260129 02:59:21.496657 21923 raft_consensus.cc:515] T e33a417cf9c54cf5af73a4ee3aac4d3a P 197b880fa39d432a9b472845876bfaf0 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:59:21.336843 21920 leader_election.cc:290] T 4408006eb5174de1896b037cf3f18457 P 1c50de89a5a845668696dcb28c3badd6 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 197b880fa39d432a9b472845876bfaf0 (127.18.211.1:42027), 974b802141bd4f13beffef2aa03f88f9 (127.18.211.2:34517)
I20260129 02:59:21.349005 20512 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a5a05fbcbbcf4916bbc01fd1a5b5e90d" candidate_uuid: "197b880fa39d432a9b472845876bfaf0" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "974b802141bd4f13beffef2aa03f88f9" is_pre_election: true
I20260129 02:59:21.459352 21789 raft_consensus.cc:515] T 2e58e9a5a8aa4cecafb464addb4f35ed P 197b880fa39d432a9b472845876bfaf0 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } }
I20260129 02:59:21.356666 20507 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "add8d4923d1a489e86882c3a25692b95" candidate_uuid: "197b880fa39d432a9b472845876bfaf0" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "974b802141bd4f13beffef2aa03f88f9" is_pre_election: true
I20260129 02:59:21.303452 21916 raft_consensus.cc:493] T dc2f4cb752cf4d59b5f6bc689f5d8104 P 197b880fa39d432a9b472845876bfaf0 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 1c50de89a5a845668696dcb28c3badd6)
I20260129 02:59:21.470785 21829 raft_consensus.cc:515] T 0f20af7fbe5d4db89a6f17948d98912c P 1c50de89a5a845668696dcb28c3badd6 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:59:21.478525 21666 raft_consensus.cc:515] T 18f30ebaa4174fd28c958e72d0cc7e24 P 197b880fa39d432a9b472845876bfaf0 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } }
I20260129 02:59:21.478977 21893 leader_election.cc:290] T fcddab6bca684adf8ea032289a00f90e P 197b880fa39d432a9b472845876bfaf0 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 974b802141bd4f13beffef2aa03f88f9 (127.18.211.2:34517), 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3:37735)
I20260129 02:59:21.479959 21791 raft_consensus.cc:515] T 18f30ebaa4174fd28c958e72d0cc7e24 P 1c50de89a5a845668696dcb28c3badd6 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } }
I20260129 02:59:21.524050 21666 leader_election.cc:290] T 18f30ebaa4174fd28c958e72d0cc7e24 P 197b880fa39d432a9b472845876bfaf0 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3:37735), 7a9daf5d42a84615a6383cfec9a10608 (127.18.211.4:40439)
I20260129 02:59:21.459414 21790 raft_consensus.cc:515] T cb0b3f259ebc4310927338a8740e87b8 P 1c50de89a5a845668696dcb28c3badd6 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } }
I20260129 02:59:21.480599 21883 consensus_queue.cc:237] T 96609ac68bdf4993a12e1c23355ca013 P 197b880fa39d432a9b472845876bfaf0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 2.3, Last appended by leader: 2, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } attrs { replace: true } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } }
W20260129 02:59:21.488902 20407 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.211.2:34517, user_credentials={real_user=slave}} blocked reactor thread for 132223us
I20260129 02:59:21.487833 20588 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "cedce331e1584ac8b636c9b8a72b2ba0" candidate_uuid: "974b802141bd4f13beffef2aa03f88f9" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "1c50de89a5a845668696dcb28c3badd6" is_pre_election: true
I20260129 02:59:21.495808 21880 raft_consensus.cc:515] T a0fcc58d6fb6473cafbd9d34d5195269 P 197b880fa39d432a9b472845876bfaf0 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:59:21.497231 20514 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a4c1226b987b468fa84dfc6ba0e2dbe1" candidate_uuid: "1c50de89a5a845668696dcb28c3badd6" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "974b802141bd4f13beffef2aa03f88f9" is_pre_election: true
I20260129 02:59:21.517289 21846 raft_consensus.cc:493] T 0678b8b468bd43cdbc3e485900844821 P 197b880fa39d432a9b472845876bfaf0 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 7a9daf5d42a84615a6383cfec9a10608)
I20260129 02:59:21.493722 21805 raft_consensus.cc:515] T b189f08e16a74a11980cc55d99359a0a P 974b802141bd4f13beffef2aa03f88f9 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:59:21.499637 21896 raft_consensus.cc:493] T b189f08e16a74a11980cc55d99359a0a P 1c50de89a5a845668696dcb28c3badd6 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 197b880fa39d432a9b472845876bfaf0)
I20260129 02:59:21.501073 21921 raft_consensus.cc:493] T 42d27a28d89b409e8807ec0efd97af69 P 1c50de89a5a845668696dcb28c3badd6 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 7a9daf5d42a84615a6383cfec9a10608)
I20260129 02:59:21.509915 20508 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "76f713c3331540d2a5b76164103e9eb1" candidate_uuid: "1c50de89a5a845668696dcb28c3badd6" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "974b802141bd4f13beffef2aa03f88f9" is_pre_election: true
I20260129 02:59:21.463663 20581 rpcz_store.cc:276] 0129 02:59:20.041538 (+ 0us) service_pool.cc:168] Inserting onto call queue
0129 02:59:20.041781 (+ 243us) service_pool.cc:225] Handling call
0129 02:59:20.042010 (+ 229us) raft_consensus.cc:1471] Updating replica for 0 ops
0129 02:59:20.042179 (+ 169us) raft_consensus.cc:1534] Early marking committed up to index 3
0129 02:59:21.203832 (+1161653us) log.cc:844] Serialized 14 byte log entry
0129 02:59:21.308250 (+104418us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0129 02:59:21.308297 (+ 47us) raft_consensus.cc:1645] Marking committed up to 3
0129 02:59:21.308340 (+ 43us) raft_consensus.cc:1697] Filling consensus response to leader.
0129 02:59:21.308400 (+ 60us) raft_consensus.cc:1691] UpdateReplicas() finished
0129 02:59:21.308511 (+ 111us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":13116,"prepare.queue_time_us":620254,"prepare.run_cpu_time_us":76,"prepare.run_wall_time_us":1614,"raft.queue_time_us":35010,"raft.run_wall_time_us":14432,"spinlock_wait_cycles":2004736,"thread_start_us":1252056,"threads_started":2,"wal-append.queue_time_us":129760}
I20260129 02:59:21.519042 20507 raft_consensus.cc:1813] T add8d4923d1a489e86882c3a25692b95 P 974b802141bd4f13beffef2aa03f88f9 [term 1 FOLLOWER]: Handling vote request from an unknown peer 197b880fa39d432a9b472845876bfaf0
I20260129 02:59:21.539544 20507 raft_consensus.cc:2410] T add8d4923d1a489e86882c3a25692b95 P 974b802141bd4f13beffef2aa03f88f9 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 197b880fa39d432a9b472845876bfaf0 for term 2 because replica has last-logged OpId of term: 1 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 3.
I20260129 02:59:21.540377 20507 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4f8f3a38f6e24ba4857875181f447442" candidate_uuid: "197b880fa39d432a9b472845876bfaf0" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "974b802141bd4f13beffef2aa03f88f9" is_pre_election: true
I20260129 02:59:21.541199 20507 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e4d18c35b0a7493e85d0a4b92956a48f" candidate_uuid: "1c50de89a5a845668696dcb28c3badd6" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "974b802141bd4f13beffef2aa03f88f9" is_pre_election: true
W20260129 02:59:21.517500 20405 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.211.3:37735, user_credentials={real_user=slave}} blocked reactor thread for 46607.1us
I20260129 02:59:21.491086 21915 raft_consensus.cc:493] T 368953c9102643b2a21f99af36b51905 P 974b802141bd4f13beffef2aa03f88f9 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 197b880fa39d432a9b472845876bfaf0)
I20260129 02:59:21.519032 20587 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "505580c93c694bbb8ea96fdf9b17ab46" candidate_uuid: "974b802141bd4f13beffef2aa03f88f9" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "1c50de89a5a845668696dcb28c3badd6" is_pre_election: true
I20260129 02:59:21.521457 21895 raft_consensus.cc:493] T 4f8f3a38f6e24ba4857875181f447442 P 1c50de89a5a845668696dcb28c3badd6 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 974b802141bd4f13beffef2aa03f88f9)
I20260129 02:59:21.522480 21789 leader_election.cc:290] T 2e58e9a5a8aa4cecafb464addb4f35ed P 197b880fa39d432a9b472845876bfaf0 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3:37735), 7a9daf5d42a84615a6383cfec9a10608 (127.18.211.4:40439)
I20260129 02:59:21.518208 21913 raft_consensus.cc:515] T a8479d1fe0bd41e5946186e3a6ca8c7a P 974b802141bd4f13beffef2aa03f88f9 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:59:21.526273 20590 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1ce4ddf5b10e4ae8b193b13b2e5936ee" candidate_uuid: "974b802141bd4f13beffef2aa03f88f9" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1c50de89a5a845668696dcb28c3badd6" is_pre_election: true
I20260129 02:59:21.522480 21829 leader_election.cc:290] T 0f20af7fbe5d4db89a6f17948d98912c P 1c50de89a5a845668696dcb28c3badd6 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 197b880fa39d432a9b472845876bfaf0 (127.18.211.1:42027), 974b802141bd4f13beffef2aa03f88f9 (127.18.211.2:34517)
I20260129 02:59:21.497315 20511 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a0fcc58d6fb6473cafbd9d34d5195269" candidate_uuid: "1c50de89a5a845668696dcb28c3badd6" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "974b802141bd4f13beffef2aa03f88f9" is_pre_election: true
I20260129 02:59:21.527175 20512 raft_consensus.cc:2468] T a5a05fbcbbcf4916bbc01fd1a5b5e90d P 974b802141bd4f13beffef2aa03f88f9 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 197b880fa39d432a9b472845876bfaf0 in term 2.
I20260129 02:59:21.531176 20514 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4408006eb5174de1896b037cf3f18457" candidate_uuid: "1c50de89a5a845668696dcb28c3badd6" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "974b802141bd4f13beffef2aa03f88f9" is_pre_election: true
I20260129 02:59:21.547546 20507 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "cb4727386a3e48aca5d71cc6c611c197" candidate_uuid: "197b880fa39d432a9b472845876bfaf0" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "974b802141bd4f13beffef2aa03f88f9" is_pre_election: true
I20260129 02:59:21.550689 20480 leader_election.cc:304] T 3e2d15f8640b407195210c476e09df91 P 974b802141bd4f13beffef2aa03f88f9 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 974b802141bd4f13beffef2aa03f88f9; no voters: 1c50de89a5a845668696dcb28c3badd6, 7a9daf5d42a84615a6383cfec9a10608
I20260129 02:59:21.517834 21788 raft_consensus.cc:515] T 78aa39096868415d9afc2ffcb248e198 P 197b880fa39d432a9b472845876bfaf0 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } }
I20260129 02:59:21.553609 21922 raft_consensus.cc:493] T 0340229d521b432580ae926f6062e3f7 P 1c50de89a5a845668696dcb28c3badd6 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 197b880fa39d432a9b472845876bfaf0)
I20260129 02:59:21.490957 21899 raft_consensus.cc:515] T cb4727386a3e48aca5d71cc6c611c197 P 1c50de89a5a845668696dcb28c3badd6 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:59:21.555886 21131 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "45cc5f3d2688486d9b6dbe7e666dedde" candidate_uuid: "974b802141bd4f13beffef2aa03f88f9" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "7a9daf5d42a84615a6383cfec9a10608" is_pre_election: true
I20260129 02:59:21.526543 21791 leader_election.cc:290] T 18f30ebaa4174fd28c958e72d0cc7e24 P 1c50de89a5a845668696dcb28c3badd6 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 197b880fa39d432a9b472845876bfaf0 (127.18.211.1:42027), 7a9daf5d42a84615a6383cfec9a10608 (127.18.211.4:40439)
I20260129 02:59:21.554395 21921 raft_consensus.cc:515] T 42d27a28d89b409e8807ec0efd97af69 P 1c50de89a5a845668696dcb28c3badd6 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } }
I20260129 02:59:21.555943 20590 raft_consensus.cc:2468] T 1ce4ddf5b10e4ae8b193b13b2e5936ee P 1c50de89a5a845668696dcb28c3badd6 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 974b802141bd4f13beffef2aa03f88f9 in term 1.
I20260129 02:59:21.534039 21920 raft_consensus.cc:493] T 3ccac82a01a2445fa5b0ce6c0fcf2295 P 1c50de89a5a845668696dcb28c3badd6 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 197b880fa39d432a9b472845876bfaf0)
I20260129 02:59:21.565215 21131 raft_consensus.cc:2468] T 45cc5f3d2688486d9b6dbe7e666dedde P 7a9daf5d42a84615a6383cfec9a10608 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 974b802141bd4f13beffef2aa03f88f9 in term 1.
I20260129 02:59:21.544689 21666 raft_consensus.cc:493] T 76f713c3331540d2a5b76164103e9eb1 P 197b880fa39d432a9b472845876bfaf0 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 974b802141bd4f13beffef2aa03f88f9)
I20260129 02:59:21.537629 21896 raft_consensus.cc:515] T b189f08e16a74a11980cc55d99359a0a P 1c50de89a5a845668696dcb28c3badd6 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:59:21.574055 20507 tablet_service.cc:2038] Received Run Leader Election RPC: tablet_id: "505580c93c694bbb8ea96fdf9b17ab46"
dest_uuid: "974b802141bd4f13beffef2aa03f88f9"
from {username='slave'} at 127.0.0.1:57584
I20260129 02:59:21.583928 20507 raft_consensus.cc:493] T 505580c93c694bbb8ea96fdf9b17ab46 P 974b802141bd4f13beffef2aa03f88f9 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260129 02:59:21.584283 20507 raft_consensus.cc:3060] T 505580c93c694bbb8ea96fdf9b17ab46 P 974b802141bd4f13beffef2aa03f88f9 [term 1 FOLLOWER]: Advancing to term 2
I20260129 02:59:21.584718 20407 leader_election.cc:304] T a5a05fbcbbcf4916bbc01fd1a5b5e90d P 197b880fa39d432a9b472845876bfaf0 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 197b880fa39d432a9b472845876bfaf0, 974b802141bd4f13beffef2aa03f88f9; no voters:
I20260129 02:59:21.567606 21934 raft_consensus.cc:493] T c891d2bff4ce4568861bbc992e91fbb0 P 974b802141bd4f13beffef2aa03f88f9 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 197b880fa39d432a9b472845876bfaf0)
I20260129 02:59:21.527043 21916 raft_consensus.cc:515] T dc2f4cb752cf4d59b5f6bc689f5d8104 P 197b880fa39d432a9b472845876bfaf0 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } }
I20260129 02:59:21.563731 21805 leader_election.cc:290] T b189f08e16a74a11980cc55d99359a0a P 974b802141bd4f13beffef2aa03f88f9 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 197b880fa39d432a9b472845876bfaf0 (127.18.211.1:42027), 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3:37735)
I20260129 02:59:21.563539 21829 raft_consensus.cc:493] T 4b1876f9a70b49edb3aa74105e66d3b3 P 1c50de89a5a845668696dcb28c3badd6 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 197b880fa39d432a9b472845876bfaf0)
I20260129 02:59:21.536839 21846 raft_consensus.cc:515] T 0678b8b468bd43cdbc3e485900844821 P 197b880fa39d432a9b472845876bfaf0 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } }
I20260129 02:59:21.585958 21778 raft_consensus.cc:493] T 96609ac68bdf4993a12e1c23355ca013 P 1c50de89a5a845668696dcb28c3badd6 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 197b880fa39d432a9b472845876bfaf0)
I20260129 02:59:21.556878 21915 raft_consensus.cc:515] T 368953c9102643b2a21f99af36b51905 P 974b802141bd4f13beffef2aa03f88f9 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:59:21.581326 21791 raft_consensus.cc:493] T 1299723a4476420ab02fd8bfd7aa0645 P 1c50de89a5a845668696dcb28c3badd6 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 197b880fa39d432a9b472845876bfaf0)
I20260129 02:59:21.591378 21666 raft_consensus.cc:515] T 76f713c3331540d2a5b76164103e9eb1 P 197b880fa39d432a9b472845876bfaf0 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:59:21.598321 21829 raft_consensus.cc:515] T 4b1876f9a70b49edb3aa74105e66d3b3 P 1c50de89a5a845668696dcb28c3badd6 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:59:21.585692 21920 raft_consensus.cc:515] T 3ccac82a01a2445fa5b0ce6c0fcf2295 P 1c50de89a5a845668696dcb28c3badd6 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } attrs { replace: true } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: NON_VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: true } }
I20260129 02:59:21.608680 20507 raft_consensus.cc:515] T 505580c93c694bbb8ea96fdf9b17ab46 P 974b802141bd4f13beffef2aa03f88f9 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } }
W20260129 02:59:21.615000 20480 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.211.3:37735, user_credentials={real_user=slave}} blocked reactor thread for 64391us
I20260129 02:59:21.617190 20480 leader_election.cc:304] T e4d18c35b0a7493e85d0a4b92956a48f P 974b802141bd4f13beffef2aa03f88f9 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1c50de89a5a845668696dcb28c3badd6, 974b802141bd4f13beffef2aa03f88f9; no voters:
I20260129 02:59:21.624969 21815 raft_consensus.cc:493] T c891d2bff4ce4568861bbc992e91fbb0 P 1c50de89a5a845668696dcb28c3badd6 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 197b880fa39d432a9b472845876bfaf0)
I20260129 02:59:21.599756 21787 raft_consensus.cc:2804] T a5a05fbcbbcf4916bbc01fd1a5b5e90d P 197b880fa39d432a9b472845876bfaf0 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260129 02:59:21.629347 20581 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a5a05fbcbbcf4916bbc01fd1a5b5e90d" candidate_uuid: "197b880fa39d432a9b472845876bfaf0" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "1c50de89a5a845668696dcb28c3badd6" is_pre_election: true
I20260129 02:59:21.630617 20581 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "add8d4923d1a489e86882c3a25692b95" candidate_uuid: "197b880fa39d432a9b472845876bfaf0" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "1c50de89a5a845668696dcb28c3badd6" is_pre_election: true
I20260129 02:59:21.604200 21666 leader_election.cc:290] T 76f713c3331540d2a5b76164103e9eb1 P 197b880fa39d432a9b472845876bfaf0 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 974b802141bd4f13beffef2aa03f88f9 (127.18.211.2:34517), 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3:37735)
I20260129 02:59:21.631593 20581 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0f20af7fbe5d4db89a6f17948d98912c" candidate_uuid: "197b880fa39d432a9b472845876bfaf0" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "1c50de89a5a845668696dcb28c3badd6" is_pre_election: true
I20260129 02:59:21.631985 20556 leader_election.cc:304] T 368953c9102643b2a21f99af36b51905 P 1c50de89a5a845668696dcb28c3badd6 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1c50de89a5a845668696dcb28c3badd6, 974b802141bd4f13beffef2aa03f88f9; no voters:
I20260129 02:59:21.603834 21923 leader_election.cc:290] T e33a417cf9c54cf5af73a4ee3aac4d3a P 197b880fa39d432a9b472845876bfaf0 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 974b802141bd4f13beffef2aa03f88f9 (127.18.211.2:34517), 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3:37735)
I20260129 02:59:21.606247 21829 leader_election.cc:290] T 4b1876f9a70b49edb3aa74105e66d3b3 P 1c50de89a5a845668696dcb28c3badd6 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 197b880fa39d432a9b472845876bfaf0 (127.18.211.1:42027), 974b802141bd4f13beffef2aa03f88f9 (127.18.211.2:34517)
I20260129 02:59:21.634404 20581 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4f8f3a38f6e24ba4857875181f447442" candidate_uuid: "197b880fa39d432a9b472845876bfaf0" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "1c50de89a5a845668696dcb28c3badd6" is_pre_election: true
I20260129 02:59:21.635305 20587 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fd8311b779af4e1fa1402b2a02a602f2" candidate_uuid: "197b880fa39d432a9b472845876bfaf0" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "1c50de89a5a845668696dcb28c3badd6" is_pre_election: true
I20260129 02:59:21.602197 21791 raft_consensus.cc:515] T 1299723a4476420ab02fd8bfd7aa0645 P 1c50de89a5a845668696dcb28c3badd6 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } attrs { replace: true } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } }
I20260129 02:59:21.579334 21895 raft_consensus.cc:515] T 4f8f3a38f6e24ba4857875181f447442 P 1c50de89a5a845668696dcb28c3badd6 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:59:21.629523 21787 raft_consensus.cc:493] T a5a05fbcbbcf4916bbc01fd1a5b5e90d P 197b880fa39d432a9b472845876bfaf0 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 1c50de89a5a845668696dcb28c3badd6)
I20260129 02:59:21.632587 21879 raft_consensus.cc:2804] T e4d18c35b0a7493e85d0a4b92956a48f P 974b802141bd4f13beffef2aa03f88f9 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260129 02:59:21.602319 21778 raft_consensus.cc:515] T 96609ac68bdf4993a12e1c23355ca013 P 1c50de89a5a845668696dcb28c3badd6 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } attrs { replace: true } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: NON_VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: true } }
I20260129 02:59:21.649653 21926 raft_consensus.cc:2749] T 3e2d15f8640b407195210c476e09df91 P 974b802141bd4f13beffef2aa03f88f9 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260129 02:59:21.651158 21879 raft_consensus.cc:493] T e4d18c35b0a7493e85d0a4b92956a48f P 974b802141bd4f13beffef2aa03f88f9 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 197b880fa39d432a9b472845876bfaf0)
I20260129 02:59:21.629069 21815 raft_consensus.cc:515] T c891d2bff4ce4568861bbc992e91fbb0 P 1c50de89a5a845668696dcb28c3badd6 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:59:21.651638 21921 leader_election.cc:290] T 42d27a28d89b409e8807ec0efd97af69 P 1c50de89a5a845668696dcb28c3badd6 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 197b880fa39d432a9b472845876bfaf0 (127.18.211.1:42027), 7a9daf5d42a84615a6383cfec9a10608 (127.18.211.4:40439)
I20260129 02:59:21.651654 21879 raft_consensus.cc:3060] T e4d18c35b0a7493e85d0a4b92956a48f P 974b802141bd4f13beffef2aa03f88f9 [term 1 FOLLOWER]: Advancing to term 2
I20260129 02:59:21.651795 21931 raft_consensus.cc:493] T 724a236524cb4e7ba3428880681d6cc2 P 1c50de89a5a845668696dcb28c3badd6 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 974b802141bd4f13beffef2aa03f88f9)
I20260129 02:59:21.651631 21899 leader_election.cc:290] T cb4727386a3e48aca5d71cc6c611c197 P 1c50de89a5a845668696dcb28c3badd6 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 197b880fa39d432a9b472845876bfaf0 (127.18.211.1:42027), 974b802141bd4f13beffef2aa03f88f9 (127.18.211.2:34517)
I20260129 02:59:21.654045 21931 raft_consensus.cc:515] T 724a236524cb4e7ba3428880681d6cc2 P 1c50de89a5a845668696dcb28c3badd6 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:59:21.648500 21787 raft_consensus.cc:3060] T a5a05fbcbbcf4916bbc01fd1a5b5e90d P 197b880fa39d432a9b472845876bfaf0 [term 2 FOLLOWER]: Advancing to term 3
I20260129 02:59:21.587014 21922 raft_consensus.cc:515] T 0340229d521b432580ae926f6062e3f7 P 1c50de89a5a845668696dcb28c3badd6 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:59:21.656304 21788 leader_election.cc:290] T 78aa39096868415d9afc2ffcb248e198 P 197b880fa39d432a9b472845876bfaf0 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3:37735), 7a9daf5d42a84615a6383cfec9a10608 (127.18.211.4:40439)
I20260129 02:59:21.653267 21880 leader_election.cc:290] T a0fcc58d6fb6473cafbd9d34d5195269 P 197b880fa39d432a9b472845876bfaf0 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 974b802141bd4f13beffef2aa03f88f9 (127.18.211.2:34517), 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3:37735)
I20260129 02:59:21.654373 21815 leader_election.cc:290] T c891d2bff4ce4568861bbc992e91fbb0 P 1c50de89a5a845668696dcb28c3badd6 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 197b880fa39d432a9b472845876bfaf0 (127.18.211.1:42027), 974b802141bd4f13beffef2aa03f88f9 (127.18.211.2:34517)
I20260129 02:59:21.660288 21778 leader_election.cc:290] T 96609ac68bdf4993a12e1c23355ca013 P 1c50de89a5a845668696dcb28c3badd6 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 197b880fa39d432a9b472845876bfaf0 (127.18.211.1:42027), 974b802141bd4f13beffef2aa03f88f9 (127.18.211.2:34517)
I20260129 02:59:21.660297 21896 leader_election.cc:290] T b189f08e16a74a11980cc55d99359a0a P 1c50de89a5a845668696dcb28c3badd6 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 197b880fa39d432a9b472845876bfaf0 (127.18.211.1:42027), 974b802141bd4f13beffef2aa03f88f9 (127.18.211.2:34517)
I20260129 02:59:21.662358 21791 leader_election.cc:290] T 1299723a4476420ab02fd8bfd7aa0645 P 1c50de89a5a845668696dcb28c3badd6 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 197b880fa39d432a9b472845876bfaf0 (127.18.211.1:42027), 974b802141bd4f13beffef2aa03f88f9 (127.18.211.2:34517), 7a9daf5d42a84615a6383cfec9a10608 (127.18.211.4:40439)
I20260129 02:59:21.602065 21934 raft_consensus.cc:515] T c891d2bff4ce4568861bbc992e91fbb0 P 974b802141bd4f13beffef2aa03f88f9 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:59:21.659050 21913 leader_election.cc:290] T a8479d1fe0bd41e5946186e3a6ca8c7a P 974b802141bd4f13beffef2aa03f88f9 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 197b880fa39d432a9b472845876bfaf0 (127.18.211.1:42027), 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3:37735)
I20260129 02:59:21.665298 21920 leader_election.cc:290] T 3ccac82a01a2445fa5b0ce6c0fcf2295 P 1c50de89a5a845668696dcb28c3badd6 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 197b880fa39d432a9b472845876bfaf0 (127.18.211.1:42027), 974b802141bd4f13beffef2aa03f88f9 (127.18.211.2:34517)
I20260129 02:59:21.665601 21846 leader_election.cc:290] T 0678b8b468bd43cdbc3e485900844821 P 197b880fa39d432a9b472845876bfaf0 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 974b802141bd4f13beffef2aa03f88f9 (127.18.211.2:34517), 7a9daf5d42a84615a6383cfec9a10608 (127.18.211.4:40439)
I20260129 02:59:21.665788 21915 leader_election.cc:290] T 368953c9102643b2a21f99af36b51905 P 974b802141bd4f13beffef2aa03f88f9 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 197b880fa39d432a9b472845876bfaf0 (127.18.211.1:42027), 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3:37735)
I20260129 02:59:21.704169 21922 leader_election.cc:290] T 0340229d521b432580ae926f6062e3f7 P 1c50de89a5a845668696dcb28c3badd6 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 197b880fa39d432a9b472845876bfaf0 (127.18.211.1:42027), 974b802141bd4f13beffef2aa03f88f9 (127.18.211.2:34517)
I20260129 02:59:21.729839 21895 leader_election.cc:290] T 4f8f3a38f6e24ba4857875181f447442 P 1c50de89a5a845668696dcb28c3badd6 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 197b880fa39d432a9b472845876bfaf0 (127.18.211.1:42027), 974b802141bd4f13beffef2aa03f88f9 (127.18.211.2:34517)
I20260129 02:59:21.736898 21934 leader_election.cc:290] T c891d2bff4ce4568861bbc992e91fbb0 P 974b802141bd4f13beffef2aa03f88f9 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 197b880fa39d432a9b472845876bfaf0 (127.18.211.1:42027), 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3:37735)
I20260129 02:59:21.723172 21916 leader_election.cc:290] T dc2f4cb752cf4d59b5f6bc689f5d8104 P 197b880fa39d432a9b472845876bfaf0 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3:37735), 7a9daf5d42a84615a6383cfec9a10608 (127.18.211.4:40439)
I20260129 02:59:21.744438 21931 leader_election.cc:290] T 724a236524cb4e7ba3428880681d6cc2 P 1c50de89a5a845668696dcb28c3badd6 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 197b880fa39d432a9b472845876bfaf0 (127.18.211.1:42027), 974b802141bd4f13beffef2aa03f88f9 (127.18.211.2:34517)
I20260129 02:59:21.705261 21939 raft_consensus.cc:493] T 75be4bd6bd45474795fd472eff2953ac P 1c50de89a5a845668696dcb28c3badd6 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 197b880fa39d432a9b472845876bfaf0)
I20260129 02:59:21.745877 21939 raft_consensus.cc:515] T 75be4bd6bd45474795fd472eff2953ac P 1c50de89a5a845668696dcb28c3badd6 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:59:21.748873 21939 leader_election.cc:290] T 75be4bd6bd45474795fd472eff2953ac P 1c50de89a5a845668696dcb28c3badd6 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 197b880fa39d432a9b472845876bfaf0 (127.18.211.1:42027), 974b802141bd4f13beffef2aa03f88f9 (127.18.211.2:34517)
W20260129 02:59:21.782070 20405 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.211.3:37735, user_credentials={real_user=slave}} blocked reactor thread for 226084us
W20260129 02:59:21.796207 20406 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260129 02:59:21.796593 20407 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.211.2:34517, user_credentials={real_user=slave}} blocked reactor thread for 211837us
I20260129 02:59:21.811976 20480 leader_election.cc:304] T 1ce4ddf5b10e4ae8b193b13b2e5936ee P 974b802141bd4f13beffef2aa03f88f9 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1c50de89a5a845668696dcb28c3badd6, 974b802141bd4f13beffef2aa03f88f9; no voters:
I20260129 02:59:21.821949 20507 leader_election.cc:290] T 505580c93c694bbb8ea96fdf9b17ab46 P 974b802141bd4f13beffef2aa03f88f9 [CANDIDATE]: Term 2 election: Requested vote from peers 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3:37735), 7a9daf5d42a84615a6383cfec9a10608 (127.18.211.4:40439)
I20260129 02:59:21.885545 20511 raft_consensus.cc:1240] T 505580c93c694bbb8ea96fdf9b17ab46 P 974b802141bd4f13beffef2aa03f88f9 [term 2 FOLLOWER]: Rejecting Update request from peer 1c50de89a5a845668696dcb28c3badd6 for earlier term 1. Current term is 2. Ops: []
I20260129 02:59:21.648872 20587 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4e3805ffa70242e8ade66da260dddf66" candidate_uuid: "197b880fa39d432a9b472845876bfaf0" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "1c50de89a5a845668696dcb28c3badd6" is_pre_election: true
I20260129 02:59:21.822404 20515 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0f20af7fbe5d4db89a6f17948d98912c" candidate_uuid: "1c50de89a5a845668696dcb28c3badd6" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "974b802141bd4f13beffef2aa03f88f9" is_pre_election: true
I20260129 02:59:21.632643 20583 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "926a250fa5834e88b96ccecbefbf5086" candidate_uuid: "197b880fa39d432a9b472845876bfaf0" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "1c50de89a5a845668696dcb28c3badd6" is_pre_election: true
I20260129 02:59:21.961372 20581 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "cb4727386a3e48aca5d71cc6c611c197" candidate_uuid: "197b880fa39d432a9b472845876bfaf0" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1c50de89a5a845668696dcb28c3badd6" is_pre_election: true
I20260129 02:59:21.964406 21913 raft_consensus.cc:2804] T 1ce4ddf5b10e4ae8b193b13b2e5936ee P 974b802141bd4f13beffef2aa03f88f9 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260129 02:59:21.946805 20509 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4b1876f9a70b49edb3aa74105e66d3b3" candidate_uuid: "1c50de89a5a845668696dcb28c3badd6" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "974b802141bd4f13beffef2aa03f88f9" is_pre_election: true
I20260129 02:59:21.964510 21926 raft_consensus.cc:493] T 75be4bd6bd45474795fd472eff2953ac P 974b802141bd4f13beffef2aa03f88f9 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 197b880fa39d432a9b472845876bfaf0)
I20260129 02:59:21.972350 21916 raft_consensus.cc:493] T c934d31efd3f4e5195605ddc354d14e1 P 197b880fa39d432a9b472845876bfaf0 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 7a9daf5d42a84615a6383cfec9a10608)
I20260129 02:59:21.973517 21916 raft_consensus.cc:515] T c934d31efd3f4e5195605ddc354d14e1 P 197b880fa39d432a9b472845876bfaf0 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } }
I20260129 02:59:21.974985 21662 leader_election.cc:290] T cedce331e1584ac8b636c9b8a72b2ba0 P 197b880fa39d432a9b472845876bfaf0 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 974b802141bd4f13beffef2aa03f88f9 (127.18.211.2:34517), 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3:37735)
I20260129 02:59:21.981884 21886 leader_election.cc:290] T 7d9494f5584d4d5db8b9adf1c8fc888b P 197b880fa39d432a9b472845876bfaf0 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 974b802141bd4f13beffef2aa03f88f9 (127.18.211.2:34517), 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3:37735)
I20260129 02:59:21.984414 20507 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "76f713c3331540d2a5b76164103e9eb1" candidate_uuid: "197b880fa39d432a9b472845876bfaf0" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "974b802141bd4f13beffef2aa03f88f9" is_pre_election: true
I20260129 02:59:21.999980 21943 raft_consensus.cc:493] T 4408006eb5174de1896b037cf3f18457 P 197b880fa39d432a9b472845876bfaf0 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 974b802141bd4f13beffef2aa03f88f9)
I20260129 02:59:21.966502 21913 raft_consensus.cc:493] T 1ce4ddf5b10e4ae8b193b13b2e5936ee P 974b802141bd4f13beffef2aa03f88f9 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 197b880fa39d432a9b472845876bfaf0)
I20260129 02:59:22.000981 21943 raft_consensus.cc:515] T 4408006eb5174de1896b037cf3f18457 P 197b880fa39d432a9b472845876bfaf0 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:59:22.006327 20515 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e33a417cf9c54cf5af73a4ee3aac4d3a" candidate_uuid: "197b880fa39d432a9b472845876bfaf0" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "974b802141bd4f13beffef2aa03f88f9" is_pre_election: true
I20260129 02:59:22.006991 20512 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a0fcc58d6fb6473cafbd9d34d5195269" candidate_uuid: "197b880fa39d432a9b472845876bfaf0" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "974b802141bd4f13beffef2aa03f88f9" is_pre_election: true
I20260129 02:59:22.007174 20515 raft_consensus.cc:2468] T e33a417cf9c54cf5af73a4ee3aac4d3a P 974b802141bd4f13beffef2aa03f88f9 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 197b880fa39d432a9b472845876bfaf0 in term 1.
I20260129 02:59:22.014107 21862 raft_consensus.cc:493] T add8d4923d1a489e86882c3a25692b95 P 7a9daf5d42a84615a6383cfec9a10608 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 1c50de89a5a845668696dcb28c3badd6)
I20260129 02:59:22.025756 21811 raft_consensus.cc:493] T a8479d1fe0bd41e5946186e3a6ca8c7a P 1c50de89a5a845668696dcb28c3badd6 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 197b880fa39d432a9b472845876bfaf0)
I20260129 02:59:22.008426 21913 raft_consensus.cc:3060] T 1ce4ddf5b10e4ae8b193b13b2e5936ee P 974b802141bd4f13beffef2aa03f88f9 [term 1 FOLLOWER]: Advancing to term 2
I20260129 02:59:22.026167 21926 raft_consensus.cc:515] T 75be4bd6bd45474795fd472eff2953ac P 974b802141bd4f13beffef2aa03f88f9 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:59:22.029811 20507 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0678b8b468bd43cdbc3e485900844821" candidate_uuid: "197b880fa39d432a9b472845876bfaf0" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "974b802141bd4f13beffef2aa03f88f9" is_pre_election: true
I20260129 02:59:21.996992 21927 consensus_queue.cc:1059] T 4e420f90884f4796997b5ea41e49e411 P 197b880fa39d432a9b472845876bfaf0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.069s
I20260129 02:59:22.027834 21811 raft_consensus.cc:515] T a8479d1fe0bd41e5946186e3a6ca8c7a P 1c50de89a5a845668696dcb28c3badd6 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:59:22.066366 21862 raft_consensus.cc:515] T add8d4923d1a489e86882c3a25692b95 P 7a9daf5d42a84615a6383cfec9a10608 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } }
I20260129 02:59:22.072752 20587 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a644c02b16af47679824ccd060ac10fd" candidate_uuid: "974b802141bd4f13beffef2aa03f88f9" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "1c50de89a5a845668696dcb28c3badd6" is_pre_election: true
I20260129 02:59:22.076521 20507 raft_consensus.cc:2468] T 0678b8b468bd43cdbc3e485900844821 P 974b802141bd4f13beffef2aa03f88f9 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 197b880fa39d432a9b472845876bfaf0 in term 2.
I20260129 02:59:22.080353 21954 raft_consensus.cc:493] T dc2f4cb752cf4d59b5f6bc689f5d8104 P 7a9daf5d42a84615a6383cfec9a10608 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 1c50de89a5a845668696dcb28c3badd6)
I20260129 02:59:22.084619 21916 leader_election.cc:290] T c934d31efd3f4e5195605ddc354d14e1 P 197b880fa39d432a9b472845876bfaf0 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 974b802141bd4f13beffef2aa03f88f9 (127.18.211.2:34517), 7a9daf5d42a84615a6383cfec9a10608 (127.18.211.4:40439)
I20260129 02:59:22.091815 21836 raft_consensus.cc:493] T febffe4a8b394368b69b1d164308fb9a P 7a9daf5d42a84615a6383cfec9a10608 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 197b880fa39d432a9b472845876bfaf0)
W20260129 02:59:22.092352 20556 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.211.2:34517, user_credentials={real_user=slave}} blocked reactor thread for 440451us
I20260129 02:59:22.092590 21836 raft_consensus.cc:515] T febffe4a8b394368b69b1d164308fb9a P 7a9daf5d42a84615a6383cfec9a10608 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } }
I20260129 02:59:22.093824 20512 raft_consensus.cc:1275] T 96609ac68bdf4993a12e1c23355ca013 P 974b802141bd4f13beffef2aa03f88f9 [term 2 FOLLOWER]: Refusing update from remote peer 197b880fa39d432a9b472845876bfaf0: Log matching property violated. Preceding OpId in replica: term: 2 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20260129 02:59:22.099843 21829 raft_consensus.cc:2804] T 368953c9102643b2a21f99af36b51905 P 1c50de89a5a845668696dcb28c3badd6 [term 2 FOLLOWER]: Leader pre-election won for term 3
W20260129 02:59:22.102391 21100 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.211.3:37735, user_credentials={real_user=slave}} blocked reactor thread for 57962us
I20260129 02:59:22.104435 20480 leader_election.cc:304] T 45cc5f3d2688486d9b6dbe7e666dedde P 974b802141bd4f13beffef2aa03f88f9 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 7a9daf5d42a84615a6383cfec9a10608, 974b802141bd4f13beffef2aa03f88f9; no voters: 1c50de89a5a845668696dcb28c3badd6
W20260129 02:59:22.110587 21786 consensus_meta.cc:287] T 926a250fa5834e88b96ccecbefbf5086 P 197b880fa39d432a9b472845876bfaf0: Time spent flushing consensus metadata: real 0.620s user 0.001s sys 0.016s
I20260129 02:59:22.111238 21786 raft_consensus.cc:515] T 926a250fa5834e88b96ccecbefbf5086 P 197b880fa39d432a9b472845876bfaf0 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:59:22.083856 21835 raft_consensus.cc:493] T 78aa39096868415d9afc2ffcb248e198 P 7a9daf5d42a84615a6383cfec9a10608 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 1c50de89a5a845668696dcb28c3badd6)
I20260129 02:59:22.112565 21835 raft_consensus.cc:515] T 78aa39096868415d9afc2ffcb248e198 P 7a9daf5d42a84615a6383cfec9a10608 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } }
I20260129 02:59:22.113694 21786 leader_election.cc:290] T 926a250fa5834e88b96ccecbefbf5086 P 197b880fa39d432a9b472845876bfaf0 [CANDIDATE]: Term 3 election: Requested vote from peers 974b802141bd4f13beffef2aa03f88f9 (127.18.211.2:34517), 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3:37735)
I20260129 02:59:22.094740 21943 leader_election.cc:290] T 4408006eb5174de1896b037cf3f18457 P 197b880fa39d432a9b472845876bfaf0 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 974b802141bd4f13beffef2aa03f88f9 (127.18.211.2:34517), 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3:37735)
I20260129 02:59:22.120510 21788 raft_consensus.cc:493] T 36ff33be666a4c82954731691dbcf327 P 197b880fa39d432a9b472845876bfaf0 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 974b802141bd4f13beffef2aa03f88f9)
I20260129 02:59:22.120962 20589 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "add8d4923d1a489e86882c3a25692b95" candidate_uuid: "974b802141bd4f13beffef2aa03f88f9" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "1c50de89a5a845668696dcb28c3badd6" is_pre_election: true
I20260129 02:59:22.121577 21788 raft_consensus.cc:515] T 36ff33be666a4c82954731691dbcf327 P 197b880fa39d432a9b472845876bfaf0 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:59:22.142473 21666 raft_consensus.cc:493] T 87d5fe48a44a408aaf03e894a67ecb2f P 197b880fa39d432a9b472845876bfaf0 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 7a9daf5d42a84615a6383cfec9a10608)
I20260129 02:59:22.143214 21666 raft_consensus.cc:515] T 87d5fe48a44a408aaf03e894a67ecb2f P 197b880fa39d432a9b472845876bfaf0 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } }
I20260129 02:59:22.145792 21666 leader_election.cc:290] T 87d5fe48a44a408aaf03e894a67ecb2f P 197b880fa39d432a9b472845876bfaf0 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 974b802141bd4f13beffef2aa03f88f9 (127.18.211.2:34517), 7a9daf5d42a84615a6383cfec9a10608 (127.18.211.4:40439)
I20260129 02:59:22.148402 20508 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7d9494f5584d4d5db8b9adf1c8fc888b" candidate_uuid: "1c50de89a5a845668696dcb28c3badd6" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "974b802141bd4f13beffef2aa03f88f9" is_pre_election: true
I20260129 02:59:22.139772 21926 leader_election.cc:290] T 75be4bd6bd45474795fd472eff2953ac P 974b802141bd4f13beffef2aa03f88f9 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 197b880fa39d432a9b472845876bfaf0 (127.18.211.1:42027), 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3:37735)
I20260129 02:59:22.090906 21927 consensus_queue.cc:1243] T 4e420f90884f4796997b5ea41e49e411 P 197b880fa39d432a9b472845876bfaf0 [LEADER]: Peer 974b802141bd4f13beffef2aa03f88f9 log is divergent from this leader: its last log entry 2.5 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 4
I20260129 02:59:22.115710 21835 leader_election.cc:290] T 78aa39096868415d9afc2ffcb248e198 P 7a9daf5d42a84615a6383cfec9a10608 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 197b880fa39d432a9b472845876bfaf0 (127.18.211.1:42027), 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3:37735)
I20260129 02:59:22.189632 21937 raft_consensus.cc:493] T 00e79d8f25a34676b1d866eb2068a2b2 P 197b880fa39d432a9b472845876bfaf0 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 7a9daf5d42a84615a6383cfec9a10608)
I20260129 02:59:22.189100 21786 raft_consensus.cc:493] T 2ac659d732314e1288849f08fe5e074e P 197b880fa39d432a9b472845876bfaf0 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 974b802141bd4f13beffef2aa03f88f9)
I20260129 02:59:22.152105 21836 leader_election.cc:290] T febffe4a8b394368b69b1d164308fb9a P 7a9daf5d42a84615a6383cfec9a10608 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 197b880fa39d432a9b472845876bfaf0 (127.18.211.1:42027), 974b802141bd4f13beffef2aa03f88f9 (127.18.211.2:34517)
I20260129 02:59:22.121802 21916 raft_consensus.cc:493] T a4c1226b987b468fa84dfc6ba0e2dbe1 P 197b880fa39d432a9b472845876bfaf0 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 974b802141bd4f13beffef2aa03f88f9)
I20260129 02:59:22.216672 21862 leader_election.cc:290] T add8d4923d1a489e86882c3a25692b95 P 7a9daf5d42a84615a6383cfec9a10608 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 974b802141bd4f13beffef2aa03f88f9 (127.18.211.2:34517), 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3:37735)
I20260129 02:59:22.141692 21954 raft_consensus.cc:515] T dc2f4cb752cf4d59b5f6bc689f5d8104 P 7a9daf5d42a84615a6383cfec9a10608 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } }
W20260129 02:59:22.200145 21787 consensus_meta.cc:287] T a5a05fbcbbcf4916bbc01fd1a5b5e90d P 197b880fa39d432a9b472845876bfaf0: Time spent flushing consensus metadata: real 0.527s user 0.010s sys 0.003s
I20260129 02:59:22.234177 21943 raft_consensus.cc:493] T 724a236524cb4e7ba3428880681d6cc2 P 197b880fa39d432a9b472845876bfaf0 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 974b802141bd4f13beffef2aa03f88f9)
I20260129 02:59:22.243772 21662 raft_consensus.cc:493] T 8055232cf0004d40a85f3bf983fe00a9 P 197b880fa39d432a9b472845876bfaf0 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 974b802141bd4f13beffef2aa03f88f9)
I20260129 02:59:22.252035 21662 raft_consensus.cc:515] T 8055232cf0004d40a85f3bf983fe00a9 P 197b880fa39d432a9b472845876bfaf0 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:59:22.252352 21954 leader_election.cc:290] T dc2f4cb752cf4d59b5f6bc689f5d8104 P 7a9daf5d42a84615a6383cfec9a10608 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 197b880fa39d432a9b472845876bfaf0 (127.18.211.1:42027), 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3:37735)
I20260129 02:59:22.254307 21662 leader_election.cc:290] T 8055232cf0004d40a85f3bf983fe00a9 P 197b880fa39d432a9b472845876bfaf0 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 974b802141bd4f13beffef2aa03f88f9 (127.18.211.2:34517), 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3:37735)
I20260129 02:59:22.265839 20507 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4408006eb5174de1896b037cf3f18457" candidate_uuid: "197b880fa39d432a9b472845876bfaf0" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "974b802141bd4f13beffef2aa03f88f9" is_pre_election: true
I20260129 02:59:22.238448 21787 raft_consensus.cc:515] T a5a05fbcbbcf4916bbc01fd1a5b5e90d P 197b880fa39d432a9b472845876bfaf0 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:59:22.255791 20583 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b189f08e16a74a11980cc55d99359a0a" candidate_uuid: "974b802141bd4f13beffef2aa03f88f9" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "1c50de89a5a845668696dcb28c3badd6" is_pre_election: true
I20260129 02:59:22.256999 21786 raft_consensus.cc:515] T 2ac659d732314e1288849f08fe5e074e P 197b880fa39d432a9b472845876bfaf0 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } }
I20260129 02:59:22.275146 21839 raft_consensus.cc:2804] T 45cc5f3d2688486d9b6dbe7e666dedde P 974b802141bd4f13beffef2aa03f88f9 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260129 02:59:22.269560 21916 raft_consensus.cc:515] T a4c1226b987b468fa84dfc6ba0e2dbe1 P 197b880fa39d432a9b472845876bfaf0 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:59:22.257365 21811 leader_election.cc:290] T a8479d1fe0bd41e5946186e3a6ca8c7a P 1c50de89a5a845668696dcb28c3badd6 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 197b880fa39d432a9b472845876bfaf0 (127.18.211.1:42027), 974b802141bd4f13beffef2aa03f88f9 (127.18.211.2:34517)
I20260129 02:59:22.249471 21937 raft_consensus.cc:515] T 00e79d8f25a34676b1d866eb2068a2b2 P 197b880fa39d432a9b472845876bfaf0 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } }
I20260129 02:59:22.285621 21131 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4e3805ffa70242e8ade66da260dddf66" candidate_uuid: "1c50de89a5a845668696dcb28c3badd6" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "7a9daf5d42a84615a6383cfec9a10608" is_pre_election: true
I20260129 02:59:22.286481 21788 leader_election.cc:290] T 36ff33be666a4c82954731691dbcf327 P 197b880fa39d432a9b472845876bfaf0 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 974b802141bd4f13beffef2aa03f88f9 (127.18.211.2:34517), 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3:37735)
I20260129 02:59:22.294250 20587 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a8479d1fe0bd41e5946186e3a6ca8c7a" candidate_uuid: "974b802141bd4f13beffef2aa03f88f9" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1c50de89a5a845668696dcb28c3badd6" is_pre_election: true
I20260129 02:59:22.279227 20512 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c934d31efd3f4e5195605ddc354d14e1" candidate_uuid: "197b880fa39d432a9b472845876bfaf0" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "974b802141bd4f13beffef2aa03f88f9" is_pre_election: true
I20260129 02:59:22.281739 21839 raft_consensus.cc:493] T 45cc5f3d2688486d9b6dbe7e666dedde P 974b802141bd4f13beffef2aa03f88f9 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 1c50de89a5a845668696dcb28c3badd6)
I20260129 02:59:22.295053 20587 raft_consensus.cc:2468] T a8479d1fe0bd41e5946186e3a6ca8c7a P 1c50de89a5a845668696dcb28c3badd6 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 974b802141bd4f13beffef2aa03f88f9 in term 1.
I20260129 02:59:22.298202 21839 raft_consensus.cc:3060] T 45cc5f3d2688486d9b6dbe7e666dedde P 974b802141bd4f13beffef2aa03f88f9 [term 1 FOLLOWER]: Advancing to term 2
I20260129 02:59:22.299856 21790 leader_election.cc:290] T cb0b3f259ebc4310927338a8740e87b8 P 1c50de89a5a845668696dcb28c3badd6 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 974b802141bd4f13beffef2aa03f88f9 (127.18.211.2:34517), 7a9daf5d42a84615a6383cfec9a10608 (127.18.211.4:40439)
I20260129 02:59:22.282195 21807 raft_consensus.cc:493] T 17d6431cc77a491e8464d68db4081a51 P 197b880fa39d432a9b472845876bfaf0 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 974b802141bd4f13beffef2aa03f88f9)
I20260129 02:59:22.302363 21807 raft_consensus.cc:515] T 17d6431cc77a491e8464d68db4081a51 P 197b880fa39d432a9b472845876bfaf0 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:59:22.304387 20581 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c891d2bff4ce4568861bbc992e91fbb0" candidate_uuid: "974b802141bd4f13beffef2aa03f88f9" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "1c50de89a5a845668696dcb28c3badd6" is_pre_election: true
I20260129 02:59:22.305195 20581 raft_consensus.cc:2468] T c891d2bff4ce4568861bbc992e91fbb0 P 1c50de89a5a845668696dcb28c3badd6 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 974b802141bd4f13beffef2aa03f88f9 in term 5.
I20260129 02:59:22.306062 21944 consensus_queue.cc:1059] T 505580c93c694bbb8ea96fdf9b17ab46 P 1c50de89a5a845668696dcb28c3badd6 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
W20260129 02:59:22.308096 21157 debug-util.cc:398] Leaking SignalData structure 0x7b08002de8a0 after lost signal to thread 19279
W20260129 02:59:22.309361 21157 debug-util.cc:398] Leaking SignalData structure 0x7b08002def80 after lost signal to thread 20372
I20260129 02:59:22.309136 20582 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "368953c9102643b2a21f99af36b51905" candidate_uuid: "974b802141bd4f13beffef2aa03f88f9" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "1c50de89a5a845668696dcb28c3badd6" is_pre_election: true
I20260129 02:59:22.311461 21945 raft_consensus.cc:3055] T 505580c93c694bbb8ea96fdf9b17ab46 P 1c50de89a5a845668696dcb28c3badd6 [term 1 LEADER]: Stepping down as leader of term 1
I20260129 02:59:22.311919 21945 raft_consensus.cc:740] T 505580c93c694bbb8ea96fdf9b17ab46 P 1c50de89a5a845668696dcb28c3badd6 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 1c50de89a5a845668696dcb28c3badd6, State: Running, Role: LEADER
I20260129 02:59:22.312856 19276 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 3
I20260129 02:59:22.313118 21945 consensus_queue.cc:260] T 505580c93c694bbb8ea96fdf9b17ab46 P 1c50de89a5a845668696dcb28c3badd6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } }
I20260129 02:59:22.314589 20584 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "75be4bd6bd45474795fd472eff2953ac" candidate_uuid: "974b802141bd4f13beffef2aa03f88f9" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "1c50de89a5a845668696dcb28c3badd6" is_pre_election: true
I20260129 02:59:22.314842 21945 raft_consensus.cc:3060] T 505580c93c694bbb8ea96fdf9b17ab46 P 1c50de89a5a845668696dcb28c3badd6 [term 1 FOLLOWER]: Advancing to term 2
I20260129 02:59:22.311510 21924 raft_consensus.cc:493] T e7d83d4ccf684e76acb29a0dc844e1b8 P 197b880fa39d432a9b472845876bfaf0 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 1c50de89a5a845668696dcb28c3badd6)
I20260129 02:59:22.316854 21924 raft_consensus.cc:515] T e7d83d4ccf684e76acb29a0dc844e1b8 P 197b880fa39d432a9b472845876bfaf0 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:59:22.319303 21924 leader_election.cc:290] T e7d83d4ccf684e76acb29a0dc844e1b8 P 197b880fa39d432a9b472845876bfaf0 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 974b802141bd4f13beffef2aa03f88f9 (127.18.211.2:34517), 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3:37735)
I20260129 02:59:22.315333 20584 raft_consensus.cc:2468] T 75be4bd6bd45474795fd472eff2953ac P 1c50de89a5a845668696dcb28c3badd6 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 974b802141bd4f13beffef2aa03f88f9 in term 4.
I20260129 02:59:22.321053 21916 leader_election.cc:290] T a4c1226b987b468fa84dfc6ba0e2dbe1 P 197b880fa39d432a9b472845876bfaf0 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 974b802141bd4f13beffef2aa03f88f9 (127.18.211.2:34517), 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3:37735)
I20260129 02:59:22.322499 20513 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "cedce331e1584ac8b636c9b8a72b2ba0" candidate_uuid: "197b880fa39d432a9b472845876bfaf0" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "974b802141bd4f13beffef2aa03f88f9" is_pre_election: true
I20260129 02:59:22.322896 21132 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "18f30ebaa4174fd28c958e72d0cc7e24" candidate_uuid: "1c50de89a5a845668696dcb28c3badd6" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "7a9daf5d42a84615a6383cfec9a10608" is_pre_election: true
I20260129 02:59:22.323345 20513 raft_consensus.cc:2468] T cedce331e1584ac8b636c9b8a72b2ba0 P 974b802141bd4f13beffef2aa03f88f9 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 197b880fa39d432a9b472845876bfaf0 in term 2.
I20260129 02:59:22.338871 20435 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fcddab6bca684adf8ea032289a00f90e" candidate_uuid: "974b802141bd4f13beffef2aa03f88f9" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "197b880fa39d432a9b472845876bfaf0" is_pre_election: true
I20260129 02:59:22.340027 20435 raft_consensus.cc:2468] T fcddab6bca684adf8ea032289a00f90e P 197b880fa39d432a9b472845876bfaf0 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 974b802141bd4f13beffef2aa03f88f9 in term 4.
W20260129 02:59:22.341440 21879 consensus_meta.cc:287] T e4d18c35b0a7493e85d0a4b92956a48f P 974b802141bd4f13beffef2aa03f88f9: Time spent flushing consensus metadata: real 0.624s user 0.009s sys 0.014s
I20260129 02:59:22.341887 21879 raft_consensus.cc:515] T e4d18c35b0a7493e85d0a4b92956a48f P 974b802141bd4f13beffef2aa03f88f9 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:59:22.342329 21787 leader_election.cc:290] T a5a05fbcbbcf4916bbc01fd1a5b5e90d P 197b880fa39d432a9b472845876bfaf0 [CANDIDATE]: Term 3 election: Requested vote from peers 974b802141bd4f13beffef2aa03f88f9 (127.18.211.2:34517), 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3:37735)
I20260129 02:59:22.342923 20509 raft_consensus.cc:1240] T e4d18c35b0a7493e85d0a4b92956a48f P 974b802141bd4f13beffef2aa03f88f9 [term 2 FOLLOWER]: Rejecting Update request from peer 197b880fa39d432a9b472845876bfaf0 for earlier term 1. Current term is 2. Ops: []
I20260129 02:59:22.343151 20480 leader_election.cc:304] T c891d2bff4ce4568861bbc992e91fbb0 P 974b802141bd4f13beffef2aa03f88f9 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1c50de89a5a845668696dcb28c3badd6, 974b802141bd4f13beffef2aa03f88f9; no voters:
I20260129 02:59:22.344193 21947 raft_consensus.cc:2804] T c891d2bff4ce4568861bbc992e91fbb0 P 974b802141bd4f13beffef2aa03f88f9 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260129 02:59:22.344260 20585 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "75b0b6e82ddc4f92b1fe1d94f9a0b73b" candidate_uuid: "974b802141bd4f13beffef2aa03f88f9" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "1c50de89a5a845668696dcb28c3badd6" is_pre_election: true
I20260129 02:59:22.344614 21947 raft_consensus.cc:493] T c891d2bff4ce4568861bbc992e91fbb0 P 974b802141bd4f13beffef2aa03f88f9 [term 5 FOLLOWER]: Starting leader election (detected failure of leader 197b880fa39d432a9b472845876bfaf0)
I20260129 02:59:22.345093 21947 raft_consensus.cc:3060] T c891d2bff4ce4568861bbc992e91fbb0 P 974b802141bd4f13beffef2aa03f88f9 [term 5 FOLLOWER]: Advancing to term 6
I20260129 02:59:22.345270 21879 leader_election.cc:290] T e4d18c35b0a7493e85d0a4b92956a48f P 974b802141bd4f13beffef2aa03f88f9 [CANDIDATE]: Term 2 election: Requested vote from peers 197b880fa39d432a9b472845876bfaf0 (127.18.211.1:42027), 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3:37735)
I20260129 02:59:22.347535 21130 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "42d27a28d89b409e8807ec0efd97af69" candidate_uuid: "1c50de89a5a845668696dcb28c3badd6" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "7a9daf5d42a84615a6383cfec9a10608" is_pre_election: true
I20260129 02:59:22.350934 21942 raft_consensus.cc:3055] T 4e420f90884f4796997b5ea41e49e411 P 197b880fa39d432a9b472845876bfaf0 [term 1 LEADER]: Stepping down as leader of term 1
I20260129 02:59:22.351372 21942 raft_consensus.cc:740] T 4e420f90884f4796997b5ea41e49e411 P 197b880fa39d432a9b472845876bfaf0 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 197b880fa39d432a9b472845876bfaf0, State: Running, Role: LEADER
I20260129 02:59:22.354676 20480 leader_election.cc:304] T a8479d1fe0bd41e5946186e3a6ca8c7a P 974b802141bd4f13beffef2aa03f88f9 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1c50de89a5a845668696dcb28c3badd6, 974b802141bd4f13beffef2aa03f88f9; no voters:
I20260129 02:59:22.356036 21929 raft_consensus.cc:2804] T a8479d1fe0bd41e5946186e3a6ca8c7a P 974b802141bd4f13beffef2aa03f88f9 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260129 02:59:22.290215 21884 raft_consensus.cc:493] T a1a39dcd328e4268ae41b988c66b4a75 P 197b880fa39d432a9b472845876bfaf0 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 1c50de89a5a845668696dcb28c3badd6)
I20260129 02:59:22.357504 20585 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e4d18c35b0a7493e85d0a4b92956a48f" candidate_uuid: "974b802141bd4f13beffef2aa03f88f9" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1c50de89a5a845668696dcb28c3badd6"
I20260129 02:59:22.360561 20585 raft_consensus.cc:3060] T e4d18c35b0a7493e85d0a4b92956a48f P 1c50de89a5a845668696dcb28c3badd6 [term 1 FOLLOWER]: Advancing to term 2
I20260129 02:59:22.362584 21786 leader_election.cc:290] T 2ac659d732314e1288849f08fe5e074e P 197b880fa39d432a9b472845876bfaf0 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 974b802141bd4f13beffef2aa03f88f9 (127.18.211.2:34517), 7a9daf5d42a84615a6383cfec9a10608 (127.18.211.4:40439)
I20260129 02:59:22.363947 21829 raft_consensus.cc:493] T 368953c9102643b2a21f99af36b51905 P 1c50de89a5a845668696dcb28c3badd6 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 197b880fa39d432a9b472845876bfaf0)
I20260129 02:59:22.364424 21829 raft_consensus.cc:3060] T 368953c9102643b2a21f99af36b51905 P 1c50de89a5a845668696dcb28c3badd6 [term 2 FOLLOWER]: Advancing to term 3
I20260129 02:59:22.364542 20407 leader_election.cc:304] T e33a417cf9c54cf5af73a4ee3aac4d3a P 197b880fa39d432a9b472845876bfaf0 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 197b880fa39d432a9b472845876bfaf0, 974b802141bd4f13beffef2aa03f88f9; no voters:
I20260129 02:59:22.356446 21929 raft_consensus.cc:493] T a8479d1fe0bd41e5946186e3a6ca8c7a P 974b802141bd4f13beffef2aa03f88f9 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 197b880fa39d432a9b472845876bfaf0)
I20260129 02:59:22.367945 21929 raft_consensus.cc:3060] T a8479d1fe0bd41e5946186e3a6ca8c7a P 974b802141bd4f13beffef2aa03f88f9 [term 1 FOLLOWER]: Advancing to term 2
I20260129 02:59:22.367854 20435 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0340229d521b432580ae926f6062e3f7" candidate_uuid: "974b802141bd4f13beffef2aa03f88f9" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "197b880fa39d432a9b472845876bfaf0" is_pre_election: true
I20260129 02:59:22.370591 20338 rpcz_store.cc:275] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:55364 (request call id 250) took 2692 ms. Trace:
I20260129 02:59:22.369146 20480 leader_election.cc:304] T 75be4bd6bd45474795fd472eff2953ac P 974b802141bd4f13beffef2aa03f88f9 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1c50de89a5a845668696dcb28c3badd6, 974b802141bd4f13beffef2aa03f88f9; no voters:
I20260129 02:59:22.371007 20338 rpcz_store.cc:276] 0129 02:59:19.678381 (+ 0us) service_pool.cc:168] Inserting onto call queue
0129 02:59:19.678552 (+ 171us) service_pool.cc:225] Handling call
0129 02:59:22.370375 (+2691823us) inbound_call.cc:173] Queueing success response
Metrics: {"reported_tablets":3}
I20260129 02:59:22.373462 21884 raft_consensus.cc:515] T a1a39dcd328e4268ae41b988c66b4a75 P 197b880fa39d432a9b472845876bfaf0 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } } peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } }
I20260129 02:59:22.377571 20335 rpcz_store.cc:275] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:33836 (request call id 143) took 2768 ms. Trace:
I20260129 02:59:22.292315 21943 raft_consensus.cc:515] T 724a236524cb4e7ba3428880681d6cc2 P 197b880fa39d432a9b472845876bfaf0 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
W20260129 02:59:22.382905 21100 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.211.3:37735, user_credentials={real_user=slave}} blocked reactor thread for 279560us
I20260129 02:59:22.380853 21786 raft_consensus.cc:2804] T e33a417cf9c54cf5af73a4ee3aac4d3a P 197b880fa39d432a9b472845876bfaf0 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260129 02:59:22.379648 20585 raft_consensus.cc:2468] T e4d18c35b0a7493e85d0a4b92956a48f P 1c50de89a5a845668696dcb28c3badd6 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 974b802141bd4f13beffef2aa03f88f9 in term 2.
I20260129 02:59:22.379329 19276 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20260129 02:59:22.389043 21937 leader_election.cc:290] T 00e79d8f25a34676b1d866eb2068a2b2 P 197b880fa39d432a9b472845876bfaf0 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 974b802141bd4f13beffef2aa03f88f9 (127.18.211.2:34517), 7a9daf5d42a84615a6383cfec9a10608 (127.18.211.4:40439)
I20260129 02:59:22.390971 20407 leader_election.cc:304] T 0678b8b468bd43cdbc3e485900844821 P 197b880fa39d432a9b472845876bfaf0 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 197b880fa39d432a9b472845876bfaf0, 974b802141bd4f13beffef2aa03f88f9; no voters:
I20260129 02:59:22.392114 20589 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "dc2f4cb752cf4d59b5f6bc689f5d8104" candidate_uuid: "7a9daf5d42a84615a6383cfec9a10608" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "1c50de89a5a845668696dcb28c3badd6" is_pre_election: true
W20260129 02:59:22.379483 21157 debug-util.cc:398] Leaking SignalData structure 0x7b08002f48c0 after lost signal to thread 20465
I20260129 02:59:22.394972 20585 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "add8d4923d1a489e86882c3a25692b95" candidate_uuid: "7a9daf5d42a84615a6383cfec9a10608" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "1c50de89a5a845668696dcb28c3badd6" is_pre_election: true
I20260129 02:59:22.395881 21894 raft_consensus.cc:493] T fd8311b779af4e1fa1402b2a02a602f2 P 7a9daf5d42a84615a6383cfec9a10608 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 1c50de89a5a845668696dcb28c3badd6)
I20260129 02:59:22.396080 21930 raft_consensus.cc:2804] T 75be4bd6bd45474795fd472eff2953ac P 974b802141bd4f13beffef2aa03f88f9 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260129 02:59:22.396579 20513 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "cb4727386a3e48aca5d71cc6c611c197" candidate_uuid: "1c50de89a5a845668696dcb28c3badd6" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "974b802141bd4f13beffef2aa03f88f9" is_pre_election: true
I20260129 02:59:22.398654 19276 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260129 02:59:22.400136 20586 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:48526 (request call id 80) took 1067 ms. Trace:
I20260129 02:59:22.400576 20586 rpcz_store.cc:276] 0129 02:59:21.332403 (+ 0us) service_pool.cc:168] Inserting onto call queue
0129 02:59:21.520365 (+187962us) service_pool.cc:225] Handling call
0129 02:59:21.529163 (+ 8798us) raft_consensus.cc:1471] Updating replica for 0 ops
0129 02:59:21.588604 (+ 59441us) raft_consensus.cc:1534] Early marking committed up to index 0
0129 02:59:21.588636 (+ 32us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0129 02:59:21.588696 (+ 60us) raft_consensus.cc:1645] Marking committed up to 5
0129 02:59:21.588742 (+ 46us) raft_consensus.cc:1697] Filling consensus response to leader.
0129 02:59:22.399632 (+810890us) spinlock_profiling.cc:243] Waited 59.2 ms on lock 0x7b5400134f54. stack: 00007f8fe044e373 00007f8fe044e270 00007f8fdfda0574 00007f8fe9a5c6b4 00007f8fe9a5c649 00007f8fe34975a1 00007f8fe352ea81 00007f8fe352d4be 00007f8fe84e1f49 00007f8fe1a6ed34 00007f8fe1a6ecc2 00007f8fe1a6ebf1 00007f8fe1a6eb6c 00007f8fe1a6de22 00007f8fe186d34c 00007f8fe186c786
0129 02:59:22.399712 (+ 80us) raft_consensus.cc:1691] UpdateReplicas() finished
0129 02:59:22.399939 (+ 227us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":136257408}
I20260129 02:59:22.401932 21835 raft_consensus.cc:493] T 1299723a4476420ab02fd8bfd7aa0645 P 7a9daf5d42a84615a6383cfec9a10608 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 197b880fa39d432a9b472845876bfaf0)
I20260129 02:59:22.402206 21839 raft_consensus.cc:515] T 45cc5f3d2688486d9b6dbe7e666dedde P 974b802141bd4f13beffef2aa03f88f9 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } }
I20260129 02:59:22.402581 21835 raft_consensus.cc:515] T 1299723a4476420ab02fd8bfd7aa0645 P 7a9daf5d42a84615a6383cfec9a10608 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } attrs { replace: true } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } }
I20260129 02:59:22.364849 21942 consensus_queue.cc:260] T 4e420f90884f4796997b5ea41e49e411 P 197b880fa39d432a9b472845876bfaf0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } }
I20260129 02:59:22.390945 21829 raft_consensus.cc:515] T 368953c9102643b2a21f99af36b51905 P 1c50de89a5a845668696dcb28c3badd6 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:59:22.408713 21835 leader_election.cc:290] T 1299723a4476420ab02fd8bfd7aa0645 P 7a9daf5d42a84615a6383cfec9a10608 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 197b880fa39d432a9b472845876bfaf0 (127.18.211.1:42027), 974b802141bd4f13beffef2aa03f88f9 (127.18.211.2:34517), 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3:37735)
I20260129 02:59:22.365592 20583 raft_consensus.cc:2468] T b189f08e16a74a11980cc55d99359a0a P 1c50de89a5a845668696dcb28c3badd6 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 974b802141bd4f13beffef2aa03f88f9 in term 2.
I20260129 02:59:22.415226 21869 raft_consensus.cc:2804] T 0678b8b468bd43cdbc3e485900844821 P 197b880fa39d432a9b472845876bfaf0 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260129 02:59:22.415791 21869 raft_consensus.cc:493] T 0678b8b468bd43cdbc3e485900844821 P 197b880fa39d432a9b472845876bfaf0 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 7a9daf5d42a84615a6383cfec9a10608)
I20260129 02:59:22.416205 21869 raft_consensus.cc:3060] T 0678b8b468bd43cdbc3e485900844821 P 197b880fa39d432a9b472845876bfaf0 [term 2 FOLLOWER]: Advancing to term 3
I20260129 02:59:22.386811 21943 leader_election.cc:290] T 724a236524cb4e7ba3428880681d6cc2 P 197b880fa39d432a9b472845876bfaf0 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 974b802141bd4f13beffef2aa03f88f9 (127.18.211.2:34517), 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3:37735)
I20260129 02:59:22.396549 21894 raft_consensus.cc:515] T fd8311b779af4e1fa1402b2a02a602f2 P 7a9daf5d42a84615a6383cfec9a10608 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } }
I20260129 02:59:22.412066 21135 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fd8311b779af4e1fa1402b2a02a602f2" candidate_uuid: "197b880fa39d432a9b472845876bfaf0" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "7a9daf5d42a84615a6383cfec9a10608" is_pre_election: true
I20260129 02:59:22.412890 21133 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4e3805ffa70242e8ade66da260dddf66" candidate_uuid: "197b880fa39d432a9b472845876bfaf0" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "7a9daf5d42a84615a6383cfec9a10608" is_pre_election: true
I20260129 02:59:22.413321 20513 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c891d2bff4ce4568861bbc992e91fbb0" candidate_uuid: "1c50de89a5a845668696dcb28c3badd6" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "974b802141bd4f13beffef2aa03f88f9" is_pre_election: true
I20260129 02:59:22.396541 21930 raft_consensus.cc:493] T 75be4bd6bd45474795fd472eff2953ac P 974b802141bd4f13beffef2aa03f88f9 [term 4 FOLLOWER]: Starting leader election (detected failure of leader 197b880fa39d432a9b472845876bfaf0)
I20260129 02:59:22.386898 21786 raft_consensus.cc:493] T e33a417cf9c54cf5af73a4ee3aac4d3a P 197b880fa39d432a9b472845876bfaf0 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 1c50de89a5a845668696dcb28c3badd6)
I20260129 02:59:22.418838 20582 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "78aa39096868415d9afc2ffcb248e198" candidate_uuid: "7a9daf5d42a84615a6383cfec9a10608" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "1c50de89a5a845668696dcb28c3badd6" is_pre_election: true
I20260129 02:59:22.416847 21788 consensus_queue.cc:1048] T 96609ac68bdf4993a12e1c23355ca013 P 197b880fa39d432a9b472845876bfaf0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260129 02:59:22.423803 20480 leader_election.cc:304] T 75b0b6e82ddc4f92b1fe1d94f9a0b73b P 974b802141bd4f13beffef2aa03f88f9 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 974b802141bd4f13beffef2aa03f88f9; no voters: 1c50de89a5a845668696dcb28c3badd6, 7a9daf5d42a84615a6383cfec9a10608
I20260129 02:59:22.428273 20435 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "926a250fa5834e88b96ccecbefbf5086" candidate_uuid: "974b802141bd4f13beffef2aa03f88f9" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "197b880fa39d432a9b472845876bfaf0" is_pre_election: true
I20260129 02:59:22.429848 20435 raft_consensus.cc:2393] T 926a250fa5834e88b96ccecbefbf5086 P 197b880fa39d432a9b472845876bfaf0 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 974b802141bd4f13beffef2aa03f88f9 in current term 3: Already voted for candidate 197b880fa39d432a9b472845876bfaf0 in this term.
I20260129 02:59:22.433254 20432 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e33a417cf9c54cf5af73a4ee3aac4d3a" candidate_uuid: "974b802141bd4f13beffef2aa03f88f9" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "197b880fa39d432a9b472845876bfaf0" is_pre_election: true
I20260129 02:59:22.434250 20590 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "505580c93c694bbb8ea96fdf9b17ab46" candidate_uuid: "974b802141bd4f13beffef2aa03f88f9" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "1c50de89a5a845668696dcb28c3badd6"
I20260129 02:59:22.435221 21947 raft_consensus.cc:515] T c891d2bff4ce4568861bbc992e91fbb0 P 974b802141bd4f13beffef2aa03f88f9 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:59:22.430075 21930 raft_consensus.cc:3060] T 75be4bd6bd45474795fd472eff2953ac P 974b802141bd4f13beffef2aa03f88f9 [term 4 FOLLOWER]: Advancing to term 5
I20260129 02:59:22.417529 21879 raft_consensus.cc:493] T 1299723a4476420ab02fd8bfd7aa0645 P 974b802141bd4f13beffef2aa03f88f9 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 197b880fa39d432a9b472845876bfaf0)
I20260129 02:59:22.431375 20407 leader_election.cc:304] T cedce331e1584ac8b636c9b8a72b2ba0 P 197b880fa39d432a9b472845876bfaf0 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 197b880fa39d432a9b472845876bfaf0, 974b802141bd4f13beffef2aa03f88f9; no voters:
I20260129 02:59:22.440459 21135 raft_consensus.cc:2468] T fd8311b779af4e1fa1402b2a02a602f2 P 7a9daf5d42a84615a6383cfec9a10608 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 197b880fa39d432a9b472845876bfaf0 in term 1.
I20260129 02:59:22.415947 21839 leader_election.cc:290] T 45cc5f3d2688486d9b6dbe7e666dedde P 974b802141bd4f13beffef2aa03f88f9 [CANDIDATE]: Term 2 election: Requested vote from peers 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3:37735), 7a9daf5d42a84615a6383cfec9a10608 (127.18.211.4:40439)
I20260129 02:59:22.417997 21884 leader_election.cc:290] T a1a39dcd328e4268ae41b988c66b4a75 P 197b880fa39d432a9b472845876bfaf0 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 974b802141bd4f13beffef2aa03f88f9 (127.18.211.2:34517), 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3:37735)
I20260129 02:59:22.444363 20513 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "96609ac68bdf4993a12e1c23355ca013" candidate_uuid: "1c50de89a5a845668696dcb28c3badd6" candidate_term: 3 candidate_status { last_received { term: 2 index: 3 } } ignore_live_leader: false dest_uuid: "974b802141bd4f13beffef2aa03f88f9" is_pre_election: true
I20260129 02:59:22.444984 20513 raft_consensus.cc:2468] T 96609ac68bdf4993a12e1c23355ca013 P 974b802141bd4f13beffef2aa03f88f9 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1c50de89a5a845668696dcb28c3badd6 in term 2.
I20260129 02:59:22.446060 21890 raft_consensus.cc:2749] T 75b0b6e82ddc4f92b1fe1d94f9a0b73b P 974b802141bd4f13beffef2aa03f88f9 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260129 02:59:22.446475 21133 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "add8d4923d1a489e86882c3a25692b95" candidate_uuid: "197b880fa39d432a9b472845876bfaf0" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "7a9daf5d42a84615a6383cfec9a10608" is_pre_election: true
I20260129 02:59:22.447127 20516 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1299723a4476420ab02fd8bfd7aa0645" candidate_uuid: "1c50de89a5a845668696dcb28c3badd6" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "974b802141bd4f13beffef2aa03f88f9" is_pre_election: true
I20260129 02:59:22.448302 20508 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b189f08e16a74a11980cc55d99359a0a" candidate_uuid: "1c50de89a5a845668696dcb28c3badd6" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "974b802141bd4f13beffef2aa03f88f9" is_pre_election: true
I20260129 02:59:22.429661 21807 leader_election.cc:290] T 17d6431cc77a491e8464d68db4081a51 P 197b880fa39d432a9b472845876bfaf0 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 974b802141bd4f13beffef2aa03f88f9 (127.18.211.2:34517), 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3:37735)
I20260129 02:59:22.450223 21829 leader_election.cc:290] T 368953c9102643b2a21f99af36b51905 P 1c50de89a5a845668696dcb28c3badd6 [CANDIDATE]: Term 3 election: Requested vote from peers 197b880fa39d432a9b472845876bfaf0 (127.18.211.1:42027), 974b802141bd4f13beffef2aa03f88f9 (127.18.211.2:34517)
I20260129 02:59:22.451480 21132 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1299723a4476420ab02fd8bfd7aa0645" candidate_uuid: "1c50de89a5a845668696dcb28c3badd6" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "7a9daf5d42a84615a6383cfec9a10608" is_pre_election: true
I20260129 02:59:22.453478 21894 leader_election.cc:290] T fd8311b779af4e1fa1402b2a02a602f2 P 7a9daf5d42a84615a6383cfec9a10608 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 197b880fa39d432a9b472845876bfaf0 (127.18.211.1:42027), 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3:37735)
I20260129 02:59:22.454166 21929 raft_consensus.cc:515] T a8479d1fe0bd41e5946186e3a6ca8c7a P 974b802141bd4f13beffef2aa03f88f9 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:59:22.449149 20480 leader_election.cc:304] T e4d18c35b0a7493e85d0a4b92956a48f P 974b802141bd4f13beffef2aa03f88f9 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1c50de89a5a845668696dcb28c3badd6, 974b802141bd4f13beffef2aa03f88f9; no voters:
I20260129 02:59:22.430627 21786 raft_consensus.cc:3060] T e33a417cf9c54cf5af73a4ee3aac4d3a P 197b880fa39d432a9b472845876bfaf0 [term 1 FOLLOWER]: Advancing to term 2
I20260129 02:59:22.447153 21879 raft_consensus.cc:515] T 1299723a4476420ab02fd8bfd7aa0645 P 974b802141bd4f13beffef2aa03f88f9 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } attrs { replace: true } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } }
I20260129 02:59:22.460216 20516 raft_consensus.cc:2468] T 1299723a4476420ab02fd8bfd7aa0645 P 974b802141bd4f13beffef2aa03f88f9 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1c50de89a5a845668696dcb28c3badd6 in term 2.
I20260129 02:59:22.377987 20335 rpcz_store.cc:276] 0129 02:59:19.608938 (+ 0us) service_pool.cc:168] Inserting onto call queue
0129 02:59:19.609108 (+ 170us) service_pool.cc:225] Handling call
0129 02:59:22.377363 (+2768255us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0129 02:59:19.738539 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0129 02:59:19.947197 (+208658us) write_op.cc:432] Acquiring schema lock in shared mode
0129 02:59:19.947239 (+ 42us) write_op.cc:435] Acquired schema lock
0129 02:59:19.947253 (+ 14us) tablet.cc:660] Decoding operations
0129 02:59:19.947620 (+ 367us) write_op.cc:620] Acquiring the partition lock for write op
0129 02:59:19.947678 (+ 58us) write_op.cc:641] Partition lock acquired for write op
0129 02:59:19.947703 (+ 25us) tablet.cc:683] Acquiring locks for 1 operations
0129 02:59:19.947899 (+ 196us) tablet.cc:699] Row locks acquired
0129 02:59:19.947915 (+ 16us) write_op.cc:260] PREPARE: finished
0129 02:59:19.948081 (+ 166us) write_op.cc:270] Start()
0129 02:59:19.948206 (+ 125us) write_op.cc:276] Timestamp: P: 1769655559948027 usec, L: 0
0129 02:59:19.948221 (+ 15us) op_driver.cc:348] REPLICATION: starting
0129 02:59:19.963150 (+ 14929us) log.cc:844] Serialized 533 byte log entry
0129 02:59:20.050947 (+ 87797us) op_driver.cc:464] REPLICATION: finished
0129 02:59:20.793932 (+742985us) write_op.cc:301] APPLY: starting
0129 02:59:20.794024 (+ 92us) tablet.cc:1365] starting BulkCheckPresence
0129 02:59:20.794150 (+ 126us) tablet.cc:1368] finished BulkCheckPresence
0129 02:59:20.794304 (+ 154us) tablet.cc:1370] starting ApplyRowOperation cycle
0129 02:59:20.794489 (+ 185us) tablet.cc:1381] finished ApplyRowOperation cycle
0129 02:59:20.794604 (+ 115us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0129 02:59:20.794632 (+ 28us) write_op.cc:312] APPLY: finished
0129 02:59:20.794778 (+ 146us) log.cc:844] Serialized 23 byte log entry
0129 02:59:20.794983 (+ 205us) write_op.cc:489] Releasing partition, row and schema locks
0129 02:59:20.795077 (+ 94us) write_op.cc:454] Released schema lock
0129 02:59:20.795125 (+ 48us) write_op.cc:341] FINISH: Updating metrics
0129 02:59:20.795484 (+ 359us) write_op.cc:489] Releasing partition, row and schema locks
0129 02:59:20.795517 (+ 33us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":52309,"reported_tablets":10,"child_traces":[["op",{"apply.queue_time_us":492312,"apply.run_cpu_time_us":1573,"apply.run_wall_time_us":1572,"num_ops":1,"prepare.queue_time_us":20386,"prepare.run_cpu_time_us":6968,"prepare.run_wall_time_us":234794,"replication_time_us":99018,"thread_start_us":439818,"threads_started":1}]]}
I20260129 02:59:22.463786 20582 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fd8311b779af4e1fa1402b2a02a602f2" candidate_uuid: "7a9daf5d42a84615a6383cfec9a10608" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "1c50de89a5a845668696dcb28c3badd6" is_pre_election: true
I20260129 02:59:22.462368 21131 raft_consensus.cc:1240] T 4e420f90884f4796997b5ea41e49e411 P 7a9daf5d42a84615a6383cfec9a10608 [term 2 FOLLOWER]: Rejecting Update request from peer 197b880fa39d432a9b472845876bfaf0 for earlier term 1. Current term is 2. Ops: []
I20260129 02:59:22.469219 20508 raft_consensus.cc:1240] T a8479d1fe0bd41e5946186e3a6ca8c7a P 974b802141bd4f13beffef2aa03f88f9 [term 2 FOLLOWER]: Rejecting Update request from peer 197b880fa39d432a9b472845876bfaf0 for earlier term 1. Current term is 2. Ops: []
I20260129 02:59:22.466761 20480 leader_election.cc:304] T b189f08e16a74a11980cc55d99359a0a P 974b802141bd4f13beffef2aa03f88f9 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1c50de89a5a845668696dcb28c3badd6, 974b802141bd4f13beffef2aa03f88f9; no voters:
I20260129 02:59:22.466570 20583 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1299723a4476420ab02fd8bfd7aa0645" candidate_uuid: "7a9daf5d42a84615a6383cfec9a10608" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "1c50de89a5a845668696dcb28c3badd6" is_pre_election: true
I20260129 02:59:22.460820 21913 raft_consensus.cc:515] T 1ce4ddf5b10e4ae8b193b13b2e5936ee P 974b802141bd4f13beffef2aa03f88f9 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:59:22.475955 21131 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2e58e9a5a8aa4cecafb464addb4f35ed" candidate_uuid: "197b880fa39d432a9b472845876bfaf0" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "7a9daf5d42a84615a6383cfec9a10608" is_pre_election: true
I20260129 02:59:22.645542 20513 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3ccac82a01a2445fa5b0ce6c0fcf2295" candidate_uuid: "1c50de89a5a845668696dcb28c3badd6" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "974b802141bd4f13beffef2aa03f88f9" is_pre_election: true
W20260129 02:59:22.536010 20405 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.211.3:37735, user_credentials={real_user=slave}} blocked reactor thread for 149819us
W20260129 02:59:22.515290 20554 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260129 02:59:22.511444 20515 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "8055232cf0004d40a85f3bf983fe00a9" candidate_uuid: "197b880fa39d432a9b472845876bfaf0" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "974b802141bd4f13beffef2aa03f88f9" is_pre_election: true
I20260129 02:59:22.475955 20507 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "87d5fe48a44a408aaf03e894a67ecb2f" candidate_uuid: "197b880fa39d432a9b472845876bfaf0" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "974b802141bd4f13beffef2aa03f88f9" is_pre_election: true
W20260129 02:59:22.535676 20480 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.211.3:37735, user_credentials={real_user=slave}} blocked reactor thread for 74044.5us
I20260129 02:59:22.658469 20508 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "724a236524cb4e7ba3428880681d6cc2" candidate_uuid: "1c50de89a5a845668696dcb28c3badd6" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "974b802141bd4f13beffef2aa03f88f9" is_pre_election: true
I20260129 02:59:22.663650 20589 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "45cc5f3d2688486d9b6dbe7e666dedde" candidate_uuid: "974b802141bd4f13beffef2aa03f88f9" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "1c50de89a5a845668696dcb28c3badd6"
W20260129 02:59:22.646443 20479 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.18.211.1:42027 timed out after 1.633s (SENT)
I20260129 02:59:22.498704 20511 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0340229d521b432580ae926f6062e3f7" candidate_uuid: "1c50de89a5a845668696dcb28c3badd6" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "974b802141bd4f13beffef2aa03f88f9" is_pre_election: true
I20260129 02:59:22.565500 21133 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "78aa39096868415d9afc2ffcb248e198" candidate_uuid: "197b880fa39d432a9b472845876bfaf0" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "7a9daf5d42a84615a6383cfec9a10608" is_pre_election: true
I20260129 02:59:22.465592 21885 raft_consensus.cc:2804] T e4d18c35b0a7493e85d0a4b92956a48f P 974b802141bd4f13beffef2aa03f88f9 [term 2 FOLLOWER]: Leader election won for term 2
I20260129 02:59:22.665795 20590 raft_consensus.cc:2468] T 505580c93c694bbb8ea96fdf9b17ab46 P 1c50de89a5a845668696dcb28c3badd6 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 974b802141bd4f13beffef2aa03f88f9 in term 2.
I20260129 02:59:22.513824 20514 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4f8f3a38f6e24ba4857875181f447442" candidate_uuid: "1c50de89a5a845668696dcb28c3badd6" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "974b802141bd4f13beffef2aa03f88f9" is_pre_election: true
I20260129 02:59:22.666499 20516 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "75be4bd6bd45474795fd472eff2953ac" candidate_uuid: "1c50de89a5a845668696dcb28c3badd6" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "974b802141bd4f13beffef2aa03f88f9" is_pre_election: true
I20260129 02:59:22.468888 21942 raft_consensus.cc:3060] T 4e420f90884f4796997b5ea41e49e411 P 197b880fa39d432a9b472845876bfaf0 [term 1 FOLLOWER]: Advancing to term 2
I20260129 02:59:22.591590 21134 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "dc2f4cb752cf4d59b5f6bc689f5d8104" candidate_uuid: "197b880fa39d432a9b472845876bfaf0" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "7a9daf5d42a84615a6383cfec9a10608" is_pre_election: true
I20260129 02:59:22.594028 20437 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00e79d8f25a34676b1d866eb2068a2b2" candidate_uuid: "974b802141bd4f13beffef2aa03f88f9" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "197b880fa39d432a9b472845876bfaf0" is_pre_election: true
I20260129 02:59:22.671059 20587 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a8479d1fe0bd41e5946186e3a6ca8c7a" candidate_uuid: "974b802141bd4f13beffef2aa03f88f9" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1c50de89a5a845668696dcb28c3badd6"
I20260129 02:59:22.671660 20587 raft_consensus.cc:3060] T a8479d1fe0bd41e5946186e3a6ca8c7a P 1c50de89a5a845668696dcb28c3badd6 [term 1 FOLLOWER]: Advancing to term 2
I20260129 02:59:22.543887 21839 raft_consensus.cc:2804] T b189f08e16a74a11980cc55d99359a0a P 974b802141bd4f13beffef2aa03f88f9 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260129 02:59:22.670243 20437 raft_consensus.cc:2468] T 00e79d8f25a34676b1d866eb2068a2b2 P 197b880fa39d432a9b472845876bfaf0 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 974b802141bd4f13beffef2aa03f88f9 in term 2.
I20260129 02:59:22.470129 20509 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "926a250fa5834e88b96ccecbefbf5086" candidate_uuid: "197b880fa39d432a9b472845876bfaf0" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "974b802141bd4f13beffef2aa03f88f9"
W20260129 02:59:22.673501 21102 outbound_call.cc:321] RPC callback for RPC call kudu.master.MasterService.TSHeartbeat -> {remote=127.18.211.62:34285, user_credentials={real_user=slave}} blocked reactor thread for 67815.6us
I20260129 02:59:22.673681 20433 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e4d18c35b0a7493e85d0a4b92956a48f" candidate_uuid: "974b802141bd4f13beffef2aa03f88f9" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "197b880fa39d432a9b472845876bfaf0" is_pre_election: true
I20260129 02:59:22.632846 21130 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0678b8b468bd43cdbc3e485900844821" candidate_uuid: "197b880fa39d432a9b472845876bfaf0" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "7a9daf5d42a84615a6383cfec9a10608" is_pre_election: true
I20260129 02:59:22.674537 20585 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1299723a4476420ab02fd8bfd7aa0645" candidate_uuid: "974b802141bd4f13beffef2aa03f88f9" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "1c50de89a5a845668696dcb28c3badd6" is_pre_election: true
I20260129 02:59:22.590832 21136 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c934d31efd3f4e5195605ddc354d14e1" candidate_uuid: "197b880fa39d432a9b472845876bfaf0" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "7a9daf5d42a84615a6383cfec9a10608" is_pre_election: true
I20260129 02:59:22.562305 21947 leader_election.cc:290] T c891d2bff4ce4568861bbc992e91fbb0 P 974b802141bd4f13beffef2aa03f88f9 [CANDIDATE]: Term 6 election: Requested vote from peers 197b880fa39d432a9b472845876bfaf0 (127.18.211.1:42027), 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3:37735)
I20260129 02:59:22.656108 21786 raft_consensus.cc:515] T e33a417cf9c54cf5af73a4ee3aac4d3a P 197b880fa39d432a9b472845876bfaf0 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:59:22.593743 20434 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a609ac02fae84725ab743f95f51d33ce" candidate_uuid: "974b802141bd4f13beffef2aa03f88f9" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "197b880fa39d432a9b472845876bfaf0" is_pre_election: true
I20260129 02:59:22.559891 21929 leader_election.cc:290] T a8479d1fe0bd41e5946186e3a6ca8c7a P 974b802141bd4f13beffef2aa03f88f9 [CANDIDATE]: Term 2 election: Requested vote from peers 197b880fa39d432a9b472845876bfaf0 (127.18.211.1:42027), 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3:37735)
I20260129 02:59:22.644562 21880 raft_consensus.cc:2804] T cedce331e1584ac8b636c9b8a72b2ba0 P 197b880fa39d432a9b472845876bfaf0 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260129 02:59:22.647040 21915 raft_consensus.cc:493] T c934d31efd3f4e5195605ddc354d14e1 P 974b802141bd4f13beffef2aa03f88f9 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 7a9daf5d42a84615a6383cfec9a10608)
I20260129 02:59:22.675720 20507 raft_consensus.cc:2468] T 87d5fe48a44a408aaf03e894a67ecb2f P 974b802141bd4f13beffef2aa03f88f9 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 197b880fa39d432a9b472845876bfaf0 in term 2.
I20260129 02:59:22.476009 21879 leader_election.cc:290] T 1299723a4476420ab02fd8bfd7aa0645 P 974b802141bd4f13beffef2aa03f88f9 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 197b880fa39d432a9b472845876bfaf0 (127.18.211.1:42027), 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3:37735), 7a9daf5d42a84615a6383cfec9a10608 (127.18.211.4:40439)
I20260129 02:59:22.664040 21132 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "18f30ebaa4174fd28c958e72d0cc7e24" candidate_uuid: "197b880fa39d432a9b472845876bfaf0" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "7a9daf5d42a84615a6383cfec9a10608" is_pre_election: true
I20260129 02:59:22.689908 20510 raft_consensus.cc:1240] T 1ce4ddf5b10e4ae8b193b13b2e5936ee P 974b802141bd4f13beffef2aa03f88f9 [term 2 FOLLOWER]: Rejecting Update request from peer 197b880fa39d432a9b472845876bfaf0 for earlier term 1. Current term is 2. Ops: []
I20260129 02:59:22.558946 21129 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "87d5fe48a44a408aaf03e894a67ecb2f" candidate_uuid: "197b880fa39d432a9b472845876bfaf0" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "7a9daf5d42a84615a6383cfec9a10608" is_pre_election: true
I20260129 02:59:22.542773 21128 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "add8d4923d1a489e86882c3a25692b95" candidate_uuid: "974b802141bd4f13beffef2aa03f88f9" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "7a9daf5d42a84615a6383cfec9a10608" is_pre_election: true
I20260129 02:59:22.693181 21131 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a644c02b16af47679824ccd060ac10fd" candidate_uuid: "974b802141bd4f13beffef2aa03f88f9" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "7a9daf5d42a84615a6383cfec9a10608" is_pre_election: true
I20260129 02:59:22.693137 20405 leader_election.cc:304] T add8d4923d1a489e86882c3a25692b95 P 197b880fa39d432a9b472845876bfaf0 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 197b880fa39d432a9b472845876bfaf0; no voters: 1c50de89a5a845668696dcb28c3badd6, 974b802141bd4f13beffef2aa03f88f9
I20260129 02:59:22.675405 20589 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c891d2bff4ce4568861bbc992e91fbb0" candidate_uuid: "974b802141bd4f13beffef2aa03f88f9" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "1c50de89a5a845668696dcb28c3badd6"
I20260129 02:59:22.676112 21839 raft_consensus.cc:493] T b189f08e16a74a11980cc55d99359a0a P 974b802141bd4f13beffef2aa03f88f9 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 197b880fa39d432a9b472845876bfaf0)
I20260129 02:59:22.676177 21885 raft_consensus.cc:697] T e4d18c35b0a7493e85d0a4b92956a48f P 974b802141bd4f13beffef2aa03f88f9 [term 2 LEADER]: Becoming Leader. State: Replica: 974b802141bd4f13beffef2aa03f88f9, State: Running, Role: LEADER
I20260129 02:59:22.700202 20514 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "cb0b3f259ebc4310927338a8740e87b8" candidate_uuid: "1c50de89a5a845668696dcb28c3badd6" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "974b802141bd4f13beffef2aa03f88f9" is_pre_election: true
I20260129 02:59:22.704547 20511 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a8479d1fe0bd41e5946186e3a6ca8c7a" candidate_uuid: "1c50de89a5a845668696dcb28c3badd6" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "974b802141bd4f13beffef2aa03f88f9" is_pre_election: true
I20260129 02:59:22.706568 20589 raft_consensus.cc:3060] T c891d2bff4ce4568861bbc992e91fbb0 P 1c50de89a5a845668696dcb28c3badd6 [term 5 FOLLOWER]: Advancing to term 6
I20260129 02:59:22.708268 20513 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "add8d4923d1a489e86882c3a25692b95" candidate_uuid: "7a9daf5d42a84615a6383cfec9a10608" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "974b802141bd4f13beffef2aa03f88f9" is_pre_election: true
I20260129 02:59:22.708922 20511 raft_consensus.cc:2393] T a8479d1fe0bd41e5946186e3a6ca8c7a P 974b802141bd4f13beffef2aa03f88f9 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1c50de89a5a845668696dcb28c3badd6 in current term 2: Already voted for candidate 974b802141bd4f13beffef2aa03f88f9 in this term.
I20260129 02:59:22.681537 20583 raft_consensus.cc:2468] T 1299723a4476420ab02fd8bfd7aa0645 P 1c50de89a5a845668696dcb28c3badd6 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7a9daf5d42a84615a6383cfec9a10608 in term 2.
W20260129 02:59:22.706025 21157 debug-util.cc:398] Leaking SignalData structure 0x7b08003690e0 after lost signal to thread 20540
I20260129 02:59:22.707584 21944 raft_consensus.cc:493] T 3906b080a5b34a74b8b64128635e8997 P 1c50de89a5a845668696dcb28c3badd6 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 197b880fa39d432a9b472845876bfaf0)
I20260129 02:59:22.716519 20439 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "febffe4a8b394368b69b1d164308fb9a" candidate_uuid: "974b802141bd4f13beffef2aa03f88f9" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "197b880fa39d432a9b472845876bfaf0" is_pre_election: true
I20260129 02:59:22.717319 20432 raft_consensus.cc:2393] T e33a417cf9c54cf5af73a4ee3aac4d3a P 197b880fa39d432a9b472845876bfaf0 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 974b802141bd4f13beffef2aa03f88f9 in current term 2: Already voted for candidate 197b880fa39d432a9b472845876bfaf0 in this term.
I20260129 02:59:22.717587 20440 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e7d83d4ccf684e76acb29a0dc844e1b8" candidate_uuid: "974b802141bd4f13beffef2aa03f88f9" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "197b880fa39d432a9b472845876bfaf0" is_pre_election: true
I20260129 02:59:22.717855 20508 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "febffe4a8b394368b69b1d164308fb9a" candidate_uuid: "7a9daf5d42a84615a6383cfec9a10608" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "974b802141bd4f13beffef2aa03f88f9" is_pre_election: true
I20260129 02:59:22.717589 21127 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "505580c93c694bbb8ea96fdf9b17ab46" candidate_uuid: "974b802141bd4f13beffef2aa03f88f9" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "7a9daf5d42a84615a6383cfec9a10608" is_pre_election: true
I20260129 02:59:22.718241 20441 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "87d5fe48a44a408aaf03e894a67ecb2f" candidate_uuid: "974b802141bd4f13beffef2aa03f88f9" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "197b880fa39d432a9b472845876bfaf0" is_pre_election: true
I20260129 02:59:22.722489 20440 raft_consensus.cc:2468] T e7d83d4ccf684e76acb29a0dc844e1b8 P 197b880fa39d432a9b472845876bfaf0 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 974b802141bd4f13beffef2aa03f88f9 in term 2.
I20260129 02:59:22.721143 20405 leader_election.cc:304] T 0f20af7fbe5d4db89a6f17948d98912c P 197b880fa39d432a9b472845876bfaf0 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 197b880fa39d432a9b472845876bfaf0; no voters: 1c50de89a5a845668696dcb28c3badd6, 974b802141bd4f13beffef2aa03f88f9
I20260129 02:59:22.697324 20436 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4e420f90884f4796997b5ea41e49e411" candidate_uuid: "974b802141bd4f13beffef2aa03f88f9" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "197b880fa39d432a9b472845876bfaf0" is_pre_election: true
I20260129 02:59:22.703504 21880 raft_consensus.cc:493] T cedce331e1584ac8b636c9b8a72b2ba0 P 197b880fa39d432a9b472845876bfaf0 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 1c50de89a5a845668696dcb28c3badd6)
I20260129 02:59:22.710938 21839 raft_consensus.cc:3060] T b189f08e16a74a11980cc55d99359a0a P 974b802141bd4f13beffef2aa03f88f9 [term 2 FOLLOWER]: Advancing to term 3
I20260129 02:59:22.719978 20510 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57600 (request call id 1488) took 1358 ms. Trace:
I20260129 02:59:22.719481 21884 raft_consensus.cc:2749] T add8d4923d1a489e86882c3a25692b95 P 197b880fa39d432a9b472845876bfaf0 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260129 02:59:22.727851 20437 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "78aa39096868415d9afc2ffcb248e198" candidate_uuid: "7a9daf5d42a84615a6383cfec9a10608" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "197b880fa39d432a9b472845876bfaf0" is_pre_election: true
I20260129 02:59:22.712249 21937 raft_consensus.cc:493] T 42d27a28d89b409e8807ec0efd97af69 P 197b880fa39d432a9b472845876bfaf0 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 7a9daf5d42a84615a6383cfec9a10608)
I20260129 02:59:22.723636 21136 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "505580c93c694bbb8ea96fdf9b17ab46" candidate_uuid: "974b802141bd4f13beffef2aa03f88f9" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "7a9daf5d42a84615a6383cfec9a10608"
I20260129 02:59:22.727051 20441 raft_consensus.cc:2468] T 87d5fe48a44a408aaf03e894a67ecb2f P 197b880fa39d432a9b472845876bfaf0 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 974b802141bd4f13beffef2aa03f88f9 in term 2.
I20260129 02:59:22.728528 20510 rpcz_store.cc:276] 0129 02:59:21.355083 (+ 0us) service_pool.cc:168] Inserting onto call queue
0129 02:59:21.364233 (+ 9150us) service_pool.cc:225] Handling call
0129 02:59:22.442884 (+1078651us) spinlock_profiling.cc:243] Waited 22.5 ms on lock 0x7b5800254d98. stack: 00007f8fe044e373 00007f8fe044e270 00007f8fdfda0574 00007f8fe9a5c6b4 00007f8fe9a5c649 00007f8fe02c08b1 00007f8fe0442db1 00007f8fe0442c62 00007f8fe92eddd9 00007f8fe846d260 00007f8fe8535dbb 00007f8fe8535e41 00007f8fe84f7af0 00007f8fe84e2459 00007f8fe84e1f02 00007f8fe1a6ed34
0129 02:59:22.446030 (+ 3146us) raft_consensus.cc:1471] Updating replica for 0 ops
0129 02:59:22.707669 (+261639us) raft_consensus.cc:1697] Filling consensus response to leader.
0129 02:59:22.712920 (+ 5251us) spinlock_profiling.cc:243] Waited 226 ms on lock 0x7b5400167954. stack: 00007f8fe044e373 00007f8fe044e270 00007f8fdfda0574 00007f8fe9a5c6b4 00007f8fe9a5c649 00007f8fe34975a1 00007f8fe352ea81 00007f8fe352d4be 00007f8fe84e1f49 00007f8fe1a6ed34 00007f8fe1a6ecc2 00007f8fe1a6ebf1 00007f8fe1a6eb6c 00007f8fe1a6de22 00007f8fe186d34c 00007f8fe186c786
0129 02:59:22.713020 (+ 100us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":572441216}
I20260129 02:59:22.731221 21136 raft_consensus.cc:3060] T 505580c93c694bbb8ea96fdf9b17ab46 P 7a9daf5d42a84615a6383cfec9a10608 [term 1 FOLLOWER]: Advancing to term 2
I20260129 02:59:22.736527 20438 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "febffe4a8b394368b69b1d164308fb9a" candidate_uuid: "7a9daf5d42a84615a6383cfec9a10608" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "197b880fa39d432a9b472845876bfaf0" is_pre_election: true
I20260129 02:59:22.736006 20337 rpcz_store.cc:275] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:55348 (request call id 249) took 1417 ms. Trace:
I20260129 02:59:22.737200 21883 raft_consensus.cc:2749] T 0f20af7fbe5d4db89a6f17948d98912c P 197b880fa39d432a9b472845876bfaf0 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260129 02:59:22.726678 20433 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "96609ac68bdf4993a12e1c23355ca013" candidate_uuid: "974b802141bd4f13beffef2aa03f88f9" candidate_term: 3 candidate_status { last_received { term: 2 index: 3 } } ignore_live_leader: false dest_uuid: "197b880fa39d432a9b472845876bfaf0" is_pre_election: true
I20260129 02:59:22.740684 20434 raft_consensus.cc:2468] T a609ac02fae84725ab743f95f51d33ce P 197b880fa39d432a9b472845876bfaf0 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 974b802141bd4f13beffef2aa03f88f9 in term 4.
I20260129 02:59:22.726917 21880 raft_consensus.cc:3060] T cedce331e1584ac8b636c9b8a72b2ba0 P 197b880fa39d432a9b472845876bfaf0 [term 2 FOLLOWER]: Advancing to term 3
I20260129 02:59:22.737521 20337 rpcz_store.cc:276] 0129 02:59:21.297737 (+ 0us) service_pool.cc:168] Inserting onto call queue
0129 02:59:21.297885 (+ 148us) service_pool.cc:225] Handling call
0129 02:59:22.710925 (+1413040us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":942789,"reported_tablets":10}
I20260129 02:59:22.742306 20440 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "dc2f4cb752cf4d59b5f6bc689f5d8104" candidate_uuid: "7a9daf5d42a84615a6383cfec9a10608" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "197b880fa39d432a9b472845876bfaf0" is_pre_election: true
W20260129 02:59:22.766363 20479 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.211.1:42027, user_credentials={real_user=slave}} blocked reactor thread for 120078us
I20260129 02:59:22.768513 20587 raft_consensus.cc:2468] T a8479d1fe0bd41e5946186e3a6ca8c7a P 1c50de89a5a845668696dcb28c3badd6 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 974b802141bd4f13beffef2aa03f88f9 in term 2.
I20260129 02:59:22.769002 20589 raft_consensus.cc:2468] T c891d2bff4ce4568861bbc992e91fbb0 P 1c50de89a5a845668696dcb28c3badd6 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 974b802141bd4f13beffef2aa03f88f9 in term 6.
I20260129 02:59:22.769586 21136 raft_consensus.cc:2468] T 505580c93c694bbb8ea96fdf9b17ab46 P 7a9daf5d42a84615a6383cfec9a10608 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 974b802141bd4f13beffef2aa03f88f9 in term 2.
I20260129 02:59:22.772959 20440 raft_consensus.cc:2468] T dc2f4cb752cf4d59b5f6bc689f5d8104 P 197b880fa39d432a9b472845876bfaf0 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7a9daf5d42a84615a6383cfec9a10608 in term 1.
I20260129 02:59:22.773046 21937 raft_consensus.cc:515] T 42d27a28d89b409e8807ec0efd97af69 P 197b880fa39d432a9b472845876bfaf0 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } }
I20260129 02:59:22.769798 20434 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3906b080a5b34a74b8b64128635e8997" candidate_uuid: "974b802141bd4f13beffef2aa03f88f9" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "197b880fa39d432a9b472845876bfaf0" is_pre_election: true
I20260129 02:59:22.774803 20438 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1299723a4476420ab02fd8bfd7aa0645" candidate_uuid: "7a9daf5d42a84615a6383cfec9a10608" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "197b880fa39d432a9b472845876bfaf0" is_pre_election: true
I20260129 02:59:22.775421 21937 leader_election.cc:290] T 42d27a28d89b409e8807ec0efd97af69 P 197b880fa39d432a9b472845876bfaf0 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3:37735), 7a9daf5d42a84615a6383cfec9a10608 (127.18.211.4:40439)
I20260129 02:59:22.776579 21786 leader_election.cc:290] T e33a417cf9c54cf5af73a4ee3aac4d3a P 197b880fa39d432a9b472845876bfaf0 [CANDIDATE]: Term 2 election: Requested vote from peers 974b802141bd4f13beffef2aa03f88f9 (127.18.211.2:34517), 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3:37735)
I20260129 02:59:22.777715 20510 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1299723a4476420ab02fd8bfd7aa0645" candidate_uuid: "7a9daf5d42a84615a6383cfec9a10608" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "974b802141bd4f13beffef2aa03f88f9" is_pre_election: true
I20260129 02:59:22.781780 20510 raft_consensus.cc:2468] T 1299723a4476420ab02fd8bfd7aa0645 P 974b802141bd4f13beffef2aa03f88f9 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7a9daf5d42a84615a6383cfec9a10608 in term 2.
I20260129 02:59:22.724637 21915 raft_consensus.cc:515] T c934d31efd3f4e5195605ddc354d14e1 P 974b802141bd4f13beffef2aa03f88f9 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } }
I20260129 02:59:22.778245 20440 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1ce4ddf5b10e4ae8b193b13b2e5936ee" candidate_uuid: "974b802141bd4f13beffef2aa03f88f9" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "197b880fa39d432a9b472845876bfaf0" is_pre_election: true
I20260129 02:59:22.778661 20434 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "913fc7d751ed4b7388d889aac30a5f9b" candidate_uuid: "974b802141bd4f13beffef2aa03f88f9" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "197b880fa39d432a9b472845876bfaf0" is_pre_election: true
I20260129 02:59:22.781792 20438 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4e420f90884f4796997b5ea41e49e411" candidate_uuid: "974b802141bd4f13beffef2aa03f88f9" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "197b880fa39d432a9b472845876bfaf0"
I20260129 02:59:22.784267 21885 consensus_queue.cc:237] T e4d18c35b0a7493e85d0a4b92956a48f P 974b802141bd4f13beffef2aa03f88f9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:59:22.784705 21869 raft_consensus.cc:515] T 0678b8b468bd43cdbc3e485900844821 P 197b880fa39d432a9b472845876bfaf0 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } }
I20260129 02:59:22.785370 20433 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0678b8b468bd43cdbc3e485900844821" candidate_uuid: "974b802141bd4f13beffef2aa03f88f9" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "197b880fa39d432a9b472845876bfaf0" is_pre_election: true
W20260129 02:59:22.775460 21157 debug-util.cc:398] Leaking SignalData structure 0x7b080032bc20 after lost signal to thread 20614
I20260129 02:59:22.740671 21954 raft_consensus.cc:493] T 2ac659d732314e1288849f08fe5e074e P 7a9daf5d42a84615a6383cfec9a10608 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 974b802141bd4f13beffef2aa03f88f9)
I20260129 02:59:22.786778 20437 raft_consensus.cc:2468] T 78aa39096868415d9afc2ffcb248e198 P 197b880fa39d432a9b472845876bfaf0 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7a9daf5d42a84615a6383cfec9a10608 in term 1.
W20260129 02:59:22.788161 20405 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.211.3:37735, user_credentials={real_user=slave}} blocked reactor thread for 72882.3us
I20260129 02:59:22.790666 20441 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fd8311b779af4e1fa1402b2a02a602f2" candidate_uuid: "7a9daf5d42a84615a6383cfec9a10608" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "197b880fa39d432a9b472845876bfaf0" is_pre_election: true
I20260129 02:59:22.799646 20438 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a1a39dcd328e4268ae41b988c66b4a75" candidate_uuid: "974b802141bd4f13beffef2aa03f88f9" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "197b880fa39d432a9b472845876bfaf0" is_pre_election: true
I20260129 02:59:22.800346 20438 raft_consensus.cc:2468] T a1a39dcd328e4268ae41b988c66b4a75 P 197b880fa39d432a9b472845876bfaf0 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 974b802141bd4f13beffef2aa03f88f9 in term 2.
I20260129 02:59:22.805554 20480 leader_election.cc:304] T 505580c93c694bbb8ea96fdf9b17ab46 P 974b802141bd4f13beffef2aa03f88f9 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1c50de89a5a845668696dcb28c3badd6, 974b802141bd4f13beffef2aa03f88f9; no voters:
I20260129 02:59:22.768942 21944 raft_consensus.cc:515] T 3906b080a5b34a74b8b64128635e8997 P 1c50de89a5a845668696dcb28c3badd6 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:59:22.806288 20435 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "cedce331e1584ac8b636c9b8a72b2ba0" candidate_uuid: "974b802141bd4f13beffef2aa03f88f9" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "197b880fa39d432a9b472845876bfaf0" is_pre_election: true
I20260129 02:59:22.789561 21915 leader_election.cc:290] T c934d31efd3f4e5195605ddc354d14e1 P 974b802141bd4f13beffef2aa03f88f9 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 197b880fa39d432a9b472845876bfaf0 (127.18.211.1:42027), 7a9daf5d42a84615a6383cfec9a10608 (127.18.211.4:40439)
I20260129 02:59:22.807935 20441 raft_consensus.cc:2468] T fd8311b779af4e1fa1402b2a02a602f2 P 197b880fa39d432a9b472845876bfaf0 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7a9daf5d42a84615a6383cfec9a10608 in term 1.
I20260129 02:59:22.801558 21828 raft_consensus.cc:493] T 913fc7d751ed4b7388d889aac30a5f9b P 1c50de89a5a845668696dcb28c3badd6 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 197b880fa39d432a9b472845876bfaf0)
I20260129 02:59:22.807713 21945 raft_consensus.cc:493] T 7a9a69cd82724b1faed1d10aa091fc4a P 1c50de89a5a845668696dcb28c3badd6 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 197b880fa39d432a9b472845876bfaf0)
I20260129 02:59:22.826049 21945 raft_consensus.cc:515] T 7a9a69cd82724b1faed1d10aa091fc4a P 1c50de89a5a845668696dcb28c3badd6 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:59:22.811451 21102 leader_election.cc:304] T add8d4923d1a489e86882c3a25692b95 P 7a9daf5d42a84615a6383cfec9a10608 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7a9daf5d42a84615a6383cfec9a10608; no voters: 1c50de89a5a845668696dcb28c3badd6, 974b802141bd4f13beffef2aa03f88f9
I20260129 02:59:22.828425 21945 leader_election.cc:290] T 7a9a69cd82724b1faed1d10aa091fc4a P 1c50de89a5a845668696dcb28c3badd6 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 197b880fa39d432a9b472845876bfaf0 (127.18.211.1:42027), 974b802141bd4f13beffef2aa03f88f9 (127.18.211.2:34517)
I20260129 02:59:22.815234 21946 raft_consensus.cc:2804] T 505580c93c694bbb8ea96fdf9b17ab46 P 974b802141bd4f13beffef2aa03f88f9 [term 2 FOLLOWER]: Leader election won for term 2
I20260129 02:59:22.829200 21946 raft_consensus.cc:697] T 505580c93c694bbb8ea96fdf9b17ab46 P 974b802141bd4f13beffef2aa03f88f9 [term 2 LEADER]: Becoming Leader. State: Replica: 974b802141bd4f13beffef2aa03f88f9, State: Running, Role: LEADER
I20260129 02:59:22.829933 20438 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a5a05fbcbbcf4916bbc01fd1a5b5e90d" candidate_uuid: "974b802141bd4f13beffef2aa03f88f9" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "197b880fa39d432a9b472845876bfaf0" is_pre_election: true
I20260129 02:59:22.830791 20438 raft_consensus.cc:2393] T a5a05fbcbbcf4916bbc01fd1a5b5e90d P 197b880fa39d432a9b472845876bfaf0 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 974b802141bd4f13beffef2aa03f88f9 in current term 3: Already voted for candidate 197b880fa39d432a9b472845876bfaf0 in this term.
I20260129 02:59:22.830261 21946 consensus_queue.cc:237] T 505580c93c694bbb8ea96fdf9b17ab46 P 974b802141bd4f13beffef2aa03f88f9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } }
W20260129 02:59:22.831032 21157 debug-util.cc:398] Leaking SignalData structure 0x7b08000f61c0 after lost signal to thread 21160
I20260129 02:59:22.804913 21928 raft_consensus.cc:493] T 3ccac82a01a2445fa5b0ce6c0fcf2295 P 974b802141bd4f13beffef2aa03f88f9 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 197b880fa39d432a9b472845876bfaf0)
I20260129 02:59:22.806087 21954 raft_consensus.cc:515] T 2ac659d732314e1288849f08fe5e074e P 7a9daf5d42a84615a6383cfec9a10608 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } }
I20260129 02:59:22.832208 21928 raft_consensus.cc:515] T 3ccac82a01a2445fa5b0ce6c0fcf2295 P 974b802141bd4f13beffef2aa03f88f9 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } attrs { replace: true } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } }
I20260129 02:59:22.833494 21954 leader_election.cc:290] T 2ac659d732314e1288849f08fe5e074e P 7a9daf5d42a84615a6383cfec9a10608 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 197b880fa39d432a9b472845876bfaf0 (127.18.211.1:42027), 974b802141bd4f13beffef2aa03f88f9 (127.18.211.2:34517)
I20260129 02:59:22.822976 20433 raft_consensus.cc:2393] T 0678b8b468bd43cdbc3e485900844821 P 197b880fa39d432a9b472845876bfaf0 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 974b802141bd4f13beffef2aa03f88f9 in current term 3: Already voted for candidate 197b880fa39d432a9b472845876bfaf0 in this term.
I20260129 02:59:22.824395 20516 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "36ff33be666a4c82954731691dbcf327" candidate_uuid: "197b880fa39d432a9b472845876bfaf0" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "974b802141bd4f13beffef2aa03f88f9" is_pre_election: true
I20260129 02:59:22.835150 21928 leader_election.cc:290] T 3ccac82a01a2445fa5b0ce6c0fcf2295 P 974b802141bd4f13beffef2aa03f88f9 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 197b880fa39d432a9b472845876bfaf0 (127.18.211.1:42027), 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3:37735), 7a9daf5d42a84615a6383cfec9a10608 (127.18.211.4:40439)
I20260129 02:59:22.837448 20589 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2e58e9a5a8aa4cecafb464addb4f35ed" candidate_uuid: "197b880fa39d432a9b472845876bfaf0" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "1c50de89a5a845668696dcb28c3badd6" is_pre_election: true
I20260129 02:59:22.838614 20515 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "724a236524cb4e7ba3428880681d6cc2" candidate_uuid: "197b880fa39d432a9b472845876bfaf0" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "974b802141bd4f13beffef2aa03f88f9" is_pre_election: true
I20260129 02:59:22.840337 20587 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fcddab6bca684adf8ea032289a00f90e" candidate_uuid: "197b880fa39d432a9b472845876bfaf0" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "1c50de89a5a845668696dcb28c3badd6" is_pre_election: true
I20260129 02:59:22.812001 21913 leader_election.cc:290] T 1ce4ddf5b10e4ae8b193b13b2e5936ee P 974b802141bd4f13beffef2aa03f88f9 [CANDIDATE]: Term 2 election: Requested vote from peers 197b880fa39d432a9b472845876bfaf0 (127.18.211.1:42027), 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3:37735)
I20260129 02:59:22.849802 20405 leader_election.cc:304] T 4f8f3a38f6e24ba4857875181f447442 P 197b880fa39d432a9b472845876bfaf0 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 197b880fa39d432a9b472845876bfaf0; no voters: 1c50de89a5a845668696dcb28c3badd6, 974b802141bd4f13beffef2aa03f88f9
I20260129 02:59:22.817682 21127 raft_consensus.cc:1275] T 4e420f90884f4796997b5ea41e49e411 P 7a9daf5d42a84615a6383cfec9a10608 [term 2 FOLLOWER]: Refusing update from remote peer 974b802141bd4f13beffef2aa03f88f9: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260129 02:59:22.856662 20514 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2ac659d732314e1288849f08fe5e074e" candidate_uuid: "197b880fa39d432a9b472845876bfaf0" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "974b802141bd4f13beffef2aa03f88f9" is_pre_election: true
I20260129 02:59:22.859464 21828 raft_consensus.cc:515] T 913fc7d751ed4b7388d889aac30a5f9b P 1c50de89a5a845668696dcb28c3badd6 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:59:22.859843 21930 raft_consensus.cc:515] T 75be4bd6bd45474795fd472eff2953ac P 974b802141bd4f13beffef2aa03f88f9 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:59:22.862419 20513 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e7d83d4ccf684e76acb29a0dc844e1b8" candidate_uuid: "197b880fa39d432a9b472845876bfaf0" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "974b802141bd4f13beffef2aa03f88f9" is_pre_election: true
I20260129 02:59:22.853781 20337 catalog_manager.cc:5654] T 4e420f90884f4796997b5ea41e49e411 P 974b802141bd4f13beffef2aa03f88f9 reported cstate change: term changed from 1 to 2, leader changed from 197b880fa39d432a9b472845876bfaf0 (127.18.211.1) to 974b802141bd4f13beffef2aa03f88f9 (127.18.211.2). New cstate: current_term: 2 leader_uuid: "974b802141bd4f13beffef2aa03f88f9" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20260129 02:59:22.860416 20582 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "18f30ebaa4174fd28c958e72d0cc7e24" candidate_uuid: "197b880fa39d432a9b472845876bfaf0" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "1c50de89a5a845668696dcb28c3badd6" is_pre_election: true
I20260129 02:59:22.865947 20480 leader_election.cc:304] T c891d2bff4ce4568861bbc992e91fbb0 P 974b802141bd4f13beffef2aa03f88f9 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1c50de89a5a845668696dcb28c3badd6, 974b802141bd4f13beffef2aa03f88f9; no voters:
I20260129 02:59:22.867250 20507 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a1a39dcd328e4268ae41b988c66b4a75" candidate_uuid: "197b880fa39d432a9b472845876bfaf0" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "974b802141bd4f13beffef2aa03f88f9" is_pre_election: true
I20260129 02:59:22.867697 20480 leader_election.cc:304] T a8479d1fe0bd41e5946186e3a6ca8c7a P 974b802141bd4f13beffef2aa03f88f9 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1c50de89a5a845668696dcb28c3badd6, 974b802141bd4f13beffef2aa03f88f9; no voters:
I20260129 02:59:22.855734 21836 raft_consensus.cc:2749] T add8d4923d1a489e86882c3a25692b95 P 7a9daf5d42a84615a6383cfec9a10608 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260129 02:59:22.873037 21869 leader_election.cc:290] T 0678b8b468bd43cdbc3e485900844821 P 197b880fa39d432a9b472845876bfaf0 [CANDIDATE]: Term 3 election: Requested vote from peers 974b802141bd4f13beffef2aa03f88f9 (127.18.211.2:34517), 7a9daf5d42a84615a6383cfec9a10608 (127.18.211.4:40439)
I20260129 02:59:22.873965 21944 leader_election.cc:290] T 3906b080a5b34a74b8b64128635e8997 P 1c50de89a5a845668696dcb28c3badd6 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 197b880fa39d432a9b472845876bfaf0 (127.18.211.1:42027), 974b802141bd4f13beffef2aa03f88f9 (127.18.211.2:34517)
W20260129 02:59:22.876636 20405 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.211.3:37735, user_credentials={real_user=slave}} blocked reactor thread for 54192.8us
I20260129 02:59:22.877928 20405 leader_election.cc:304] T cb4727386a3e48aca5d71cc6c611c197 P 197b880fa39d432a9b472845876bfaf0 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 197b880fa39d432a9b472845876bfaf0; no voters: 1c50de89a5a845668696dcb28c3badd6, 974b802141bd4f13beffef2aa03f88f9
I20260129 02:59:22.878890 20515 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00e79d8f25a34676b1d866eb2068a2b2" candidate_uuid: "197b880fa39d432a9b472845876bfaf0" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "974b802141bd4f13beffef2aa03f88f9" is_pre_election: true
I20260129 02:59:22.879554 20515 raft_consensus.cc:2468] T 00e79d8f25a34676b1d866eb2068a2b2 P 974b802141bd4f13beffef2aa03f88f9 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 197b880fa39d432a9b472845876bfaf0 in term 2.
W20260129 02:59:22.881558 21102 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.211.2:34517, user_credentials={real_user=slave}} blocked reactor thread for 76950.4us
I20260129 02:59:22.881784 20509 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a4c1226b987b468fa84dfc6ba0e2dbe1" candidate_uuid: "197b880fa39d432a9b472845876bfaf0" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "974b802141bd4f13beffef2aa03f88f9" is_pre_election: true
I20260129 02:59:22.890055 20589 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a0fcc58d6fb6473cafbd9d34d5195269" candidate_uuid: "197b880fa39d432a9b472845876bfaf0" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1c50de89a5a845668696dcb28c3badd6" is_pre_election: true
I20260129 02:59:22.890512 20588 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "cedce331e1584ac8b636c9b8a72b2ba0" candidate_uuid: "197b880fa39d432a9b472845876bfaf0" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "1c50de89a5a845668696dcb28c3badd6" is_pre_election: true
I20260129 02:59:22.891662 20588 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7d9494f5584d4d5db8b9adf1c8fc888b" candidate_uuid: "197b880fa39d432a9b472845876bfaf0" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1c50de89a5a845668696dcb28c3badd6" is_pre_election: true
I20260129 02:59:22.894122 20581 raft_consensus.cc:1240] T a8479d1fe0bd41e5946186e3a6ca8c7a P 1c50de89a5a845668696dcb28c3badd6 [term 2 FOLLOWER]: Rejecting Update request from peer 197b880fa39d432a9b472845876bfaf0 for earlier term 1. Current term is 2. Ops: []
I20260129 02:59:22.894975 20584 raft_consensus.cc:1240] T e4d18c35b0a7493e85d0a4b92956a48f P 1c50de89a5a845668696dcb28c3badd6 [term 2 FOLLOWER]: Rejecting Update request from peer 197b880fa39d432a9b472845876bfaf0 for earlier term 1. Current term is 2. Ops: []
I20260129 02:59:22.896319 21099 leader_election.cc:304] T dc2f4cb752cf4d59b5f6bc689f5d8104 P 7a9daf5d42a84615a6383cfec9a10608 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 197b880fa39d432a9b472845876bfaf0, 7a9daf5d42a84615a6383cfec9a10608; no voters: 1c50de89a5a845668696dcb28c3badd6
I20260129 02:59:22.896435 20441 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a8479d1fe0bd41e5946186e3a6ca8c7a" candidate_uuid: "974b802141bd4f13beffef2aa03f88f9" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "197b880fa39d432a9b472845876bfaf0" is_pre_election: true
I20260129 02:59:22.895898 20438 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b189f08e16a74a11980cc55d99359a0a" candidate_uuid: "974b802141bd4f13beffef2aa03f88f9" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "197b880fa39d432a9b472845876bfaf0" is_pre_election: true
I20260129 02:59:22.898305 21099 leader_election.cc:304] T 78aa39096868415d9afc2ffcb248e198 P 7a9daf5d42a84615a6383cfec9a10608 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 197b880fa39d432a9b472845876bfaf0, 7a9daf5d42a84615a6383cfec9a10608; no voters: 1c50de89a5a845668696dcb28c3badd6
I20260129 02:59:22.898557 21136 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c934d31efd3f4e5195605ddc354d14e1" candidate_uuid: "974b802141bd4f13beffef2aa03f88f9" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "7a9daf5d42a84615a6383cfec9a10608" is_pre_election: true
I20260129 02:59:22.899027 21102 leader_election.cc:304] T febffe4a8b394368b69b1d164308fb9a P 7a9daf5d42a84615a6383cfec9a10608 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7a9daf5d42a84615a6383cfec9a10608; no voters: 197b880fa39d432a9b472845876bfaf0, 974b802141bd4f13beffef2aa03f88f9
I20260129 02:59:22.899245 21954 raft_consensus.cc:2804] T dc2f4cb752cf4d59b5f6bc689f5d8104 P 7a9daf5d42a84615a6383cfec9a10608 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260129 02:59:22.899668 21954 raft_consensus.cc:493] T dc2f4cb752cf4d59b5f6bc689f5d8104 P 7a9daf5d42a84615a6383cfec9a10608 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 1c50de89a5a845668696dcb28c3badd6)
I20260129 02:59:22.867655 20511 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a5a05fbcbbcf4916bbc01fd1a5b5e90d" candidate_uuid: "197b880fa39d432a9b472845876bfaf0" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "974b802141bd4f13beffef2aa03f88f9"
I20260129 02:59:22.899762 20585 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "78aa39096868415d9afc2ffcb248e198" candidate_uuid: "197b880fa39d432a9b472845876bfaf0" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "1c50de89a5a845668696dcb28c3badd6" is_pre_election: true
I20260129 02:59:22.900100 21954 raft_consensus.cc:3060] T dc2f4cb752cf4d59b5f6bc689f5d8104 P 7a9daf5d42a84615a6383cfec9a10608 [term 1 FOLLOWER]: Advancing to term 2
I20260129 02:59:22.882019 20514 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e33a417cf9c54cf5af73a4ee3aac4d3a" candidate_uuid: "197b880fa39d432a9b472845876bfaf0" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "974b802141bd4f13beffef2aa03f88f9"
I20260129 02:59:22.901016 20437 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "368953c9102643b2a21f99af36b51905" candidate_uuid: "974b802141bd4f13beffef2aa03f88f9" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "197b880fa39d432a9b472845876bfaf0" is_pre_election: true
I20260129 02:59:22.861783 21828 leader_election.cc:290] T 913fc7d751ed4b7388d889aac30a5f9b P 1c50de89a5a845668696dcb28c3badd6 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 197b880fa39d432a9b472845876bfaf0 (127.18.211.1:42027), 974b802141bd4f13beffef2aa03f88f9 (127.18.211.2:34517)
I20260129 02:59:22.888156 20507 raft_consensus.cc:1240] T c891d2bff4ce4568861bbc992e91fbb0 P 974b802141bd4f13beffef2aa03f88f9 [term 6 FOLLOWER]: Rejecting Update request from peer 197b880fa39d432a9b472845876bfaf0 for earlier term 5. Current term is 6. Ops: []
I20260129 02:59:22.888024 21786 raft_consensus.cc:2749] T 4f8f3a38f6e24ba4857875181f447442 P 197b880fa39d432a9b472845876bfaf0 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260129 02:59:22.896409 20434 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c891d2bff4ce4568861bbc992e91fbb0" candidate_uuid: "974b802141bd4f13beffef2aa03f88f9" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "197b880fa39d432a9b472845876bfaf0" is_pre_election: true
I20260129 02:59:22.882722 20512 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "17d6431cc77a491e8464d68db4081a51" candidate_uuid: "197b880fa39d432a9b472845876bfaf0" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "974b802141bd4f13beffef2aa03f88f9" is_pre_election: true
I20260129 02:59:22.903352 20585 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4408006eb5174de1896b037cf3f18457" candidate_uuid: "197b880fa39d432a9b472845876bfaf0" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1c50de89a5a845668696dcb28c3badd6" is_pre_election: true
I20260129 02:59:22.908996 20587 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e33a417cf9c54cf5af73a4ee3aac4d3a" candidate_uuid: "197b880fa39d432a9b472845876bfaf0" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1c50de89a5a845668696dcb28c3badd6" is_pre_election: true
I20260129 02:59:22.902619 21102 leader_election.cc:304] T 1299723a4476420ab02fd8bfd7aa0645 P 7a9daf5d42a84615a6383cfec9a10608 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 4 responses out of 4 voters: 3 yes votes; 1 no votes. yes voters: 1c50de89a5a845668696dcb28c3badd6, 7a9daf5d42a84615a6383cfec9a10608, 974b802141bd4f13beffef2aa03f88f9; no voters: 197b880fa39d432a9b472845876bfaf0
W20260129 02:59:22.917393 20480 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.211.3:37735, user_credentials={real_user=slave}} blocked reactor thread for 49807.3us
I20260129 02:59:22.924376 20590 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "76f713c3331540d2a5b76164103e9eb1" candidate_uuid: "197b880fa39d432a9b472845876bfaf0" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1c50de89a5a845668696dcb28c3badd6" is_pre_election: true
I20260129 02:59:22.917901 20586 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "dc2f4cb752cf4d59b5f6bc689f5d8104" candidate_uuid: "197b880fa39d432a9b472845876bfaf0" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "1c50de89a5a845668696dcb28c3badd6" is_pre_election: true
I20260129 02:59:22.907927 21955 raft_consensus.cc:2749] T febffe4a8b394368b69b1d164308fb9a P 7a9daf5d42a84615a6383cfec9a10608 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260129 02:59:22.900164 21897 raft_consensus.cc:2804] T 78aa39096868415d9afc2ffcb248e198 P 7a9daf5d42a84615a6383cfec9a10608 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260129 02:59:22.930644 21134 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1299723a4476420ab02fd8bfd7aa0645" candidate_uuid: "974b802141bd4f13beffef2aa03f88f9" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "7a9daf5d42a84615a6383cfec9a10608" is_pre_election: true
W20260129 02:59:22.932045 20405 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.211.3:37735, user_credentials={real_user=slave}} blocked reactor thread for 54334.3us
I20260129 02:59:22.917820 21926 raft_consensus.cc:2804] T c891d2bff4ce4568861bbc992e91fbb0 P 974b802141bd4f13beffef2aa03f88f9 [term 6 FOLLOWER]: Leader election won for term 6
W20260129 02:59:22.953035 21102 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.211.2:34517, user_credentials={real_user=slave}} blocked reactor thread for 50483.5us
I20260129 02:59:22.952061 21136 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "cb0b3f259ebc4310927338a8740e87b8" candidate_uuid: "1c50de89a5a845668696dcb28c3badd6" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "7a9daf5d42a84615a6383cfec9a10608" is_pre_election: true
I20260129 02:59:22.957566 20587 raft_consensus.cc:1240] T c891d2bff4ce4568861bbc992e91fbb0 P 1c50de89a5a845668696dcb28c3badd6 [term 6 FOLLOWER]: Rejecting Update request from peer 197b880fa39d432a9b472845876bfaf0 for earlier term 5. Current term is 6. Ops: []
I20260129 02:59:22.929055 21928 raft_consensus.cc:2804] T a8479d1fe0bd41e5946186e3a6ca8c7a P 974b802141bd4f13beffef2aa03f88f9 [term 2 FOLLOWER]: Leader election won for term 2
I20260129 02:59:22.968999 20438 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "75be4bd6bd45474795fd472eff2953ac" candidate_uuid: "974b802141bd4f13beffef2aa03f88f9" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "197b880fa39d432a9b472845876bfaf0" is_pre_election: true
I20260129 02:59:22.970166 21864 raft_consensus.cc:2804] T 1299723a4476420ab02fd8bfd7aa0645 P 7a9daf5d42a84615a6383cfec9a10608 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260129 02:59:22.970604 21864 raft_consensus.cc:493] T 1299723a4476420ab02fd8bfd7aa0645 P 7a9daf5d42a84615a6383cfec9a10608 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 197b880fa39d432a9b472845876bfaf0)
I20260129 02:59:22.970969 21864 raft_consensus.cc:3060] T 1299723a4476420ab02fd8bfd7aa0645 P 7a9daf5d42a84615a6383cfec9a10608 [term 2 FOLLOWER]: Advancing to term 3
I20260129 02:59:22.971205 20432 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1299723a4476420ab02fd8bfd7aa0645" candidate_uuid: "974b802141bd4f13beffef2aa03f88f9" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "197b880fa39d432a9b472845876bfaf0" is_pre_election: true
I20260129 02:59:22.954044 21869 raft_consensus.cc:2749] T cb4727386a3e48aca5d71cc6c611c197 P 197b880fa39d432a9b472845876bfaf0 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260129 02:59:22.933250 21897 raft_consensus.cc:493] T 78aa39096868415d9afc2ffcb248e198 P 7a9daf5d42a84615a6383cfec9a10608 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 1c50de89a5a845668696dcb28c3badd6)
I20260129 02:59:23.021631 20441 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c891d2bff4ce4568861bbc992e91fbb0" candidate_uuid: "974b802141bd4f13beffef2aa03f88f9" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "197b880fa39d432a9b472845876bfaf0"
I20260129 02:59:22.983690 20407 leader_election.cc:304] T 87d5fe48a44a408aaf03e894a67ecb2f P 197b880fa39d432a9b472845876bfaf0 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 197b880fa39d432a9b472845876bfaf0, 974b802141bd4f13beffef2aa03f88f9; no voters:
W20260129 02:59:23.026774 20479 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.211.1:42027, user_credentials={real_user=slave}} blocked reactor thread for 51428.5us
I20260129 02:59:22.975133 21930 leader_election.cc:290] T 75be4bd6bd45474795fd472eff2953ac P 974b802141bd4f13beffef2aa03f88f9 [CANDIDATE]: Term 5 election: Requested vote from peers 197b880fa39d432a9b472845876bfaf0 (127.18.211.1:42027), 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3:37735)
I20260129 02:59:22.966269 21127 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "45cc5f3d2688486d9b6dbe7e666dedde" candidate_uuid: "974b802141bd4f13beffef2aa03f88f9" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "7a9daf5d42a84615a6383cfec9a10608"
I20260129 02:59:22.983143 21897 raft_consensus.cc:3060] T 78aa39096868415d9afc2ffcb248e198 P 7a9daf5d42a84615a6383cfec9a10608 [term 1 FOLLOWER]: Advancing to term 2
I20260129 02:59:23.042069 21136 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2ac659d732314e1288849f08fe5e074e" candidate_uuid: "197b880fa39d432a9b472845876bfaf0" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "7a9daf5d42a84615a6383cfec9a10608" is_pre_election: true
I20260129 02:59:22.977070 21788 consensus_queue.cc:1059] T e4d18c35b0a7493e85d0a4b92956a48f P 197b880fa39d432a9b472845876bfaf0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.055s
I20260129 02:59:22.981770 21928 raft_consensus.cc:697] T a8479d1fe0bd41e5946186e3a6ca8c7a P 974b802141bd4f13beffef2aa03f88f9 [term 2 LEADER]: Becoming Leader. State: Replica: 974b802141bd4f13beffef2aa03f88f9, State: Running, Role: LEADER
I20260129 02:59:23.029877 20436 raft_consensus.cc:2468] T 4e420f90884f4796997b5ea41e49e411 P 197b880fa39d432a9b472845876bfaf0 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 974b802141bd4f13beffef2aa03f88f9 in term 2.
I20260129 02:59:23.051868 20440 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e4d18c35b0a7493e85d0a4b92956a48f" candidate_uuid: "974b802141bd4f13beffef2aa03f88f9" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "197b880fa39d432a9b472845876bfaf0" is_pre_election: true
I20260129 02:59:23.031183 20432 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a609ac02fae84725ab743f95f51d33ce" candidate_uuid: "974b802141bd4f13beffef2aa03f88f9" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "197b880fa39d432a9b472845876bfaf0" is_pre_election: true
I20260129 02:59:23.055024 21131 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0678b8b468bd43cdbc3e485900844821" candidate_uuid: "197b880fa39d432a9b472845876bfaf0" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "7a9daf5d42a84615a6383cfec9a10608"
W20260129 02:59:23.044929 20405 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260129 02:59:22.981613 20431 tablet_service.cc:1552] Processing DeleteTablet for tablet add8d4923d1a489e86882c3a25692b95 with delete_type TABLET_DATA_TOMBSTONED (TS 197b880fa39d432a9b472845876bfaf0 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:55666
I20260129 02:59:23.058151 20440 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0340229d521b432580ae926f6062e3f7" candidate_uuid: "974b802141bd4f13beffef2aa03f88f9" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "197b880fa39d432a9b472845876bfaf0" is_pre_election: true
I20260129 02:59:23.062350 20438 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "926a250fa5834e88b96ccecbefbf5086" candidate_uuid: "974b802141bd4f13beffef2aa03f88f9" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "197b880fa39d432a9b472845876bfaf0" is_pre_election: true
I20260129 02:59:23.064116 20441 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4e420f90884f4796997b5ea41e49e411" candidate_uuid: "974b802141bd4f13beffef2aa03f88f9" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "197b880fa39d432a9b472845876bfaf0" is_pre_election: true
I20260129 02:59:23.065483 20587 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "926a250fa5834e88b96ccecbefbf5086" candidate_uuid: "197b880fa39d432a9b472845876bfaf0" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "1c50de89a5a845668696dcb28c3badd6"
I20260129 02:59:23.066185 21129 raft_consensus.cc:1275] T 96609ac68bdf4993a12e1c23355ca013 P 7a9daf5d42a84615a6383cfec9a10608 [term 2 LEARNER]: Refusing update from remote peer 197b880fa39d432a9b472845876bfaf0: Log matching property violated. Preceding OpId in replica: term: 2 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20260129 02:59:23.068104 21128 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "42d27a28d89b409e8807ec0efd97af69" candidate_uuid: "197b880fa39d432a9b472845876bfaf0" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "7a9daf5d42a84615a6383cfec9a10608" is_pre_election: true
I20260129 02:59:23.067966 21789 consensus_queue.cc:1059] T a8479d1fe0bd41e5946186e3a6ca8c7a P 197b880fa39d432a9b472845876bfaf0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.043s
I20260129 02:59:23.070468 20438 raft_consensus.cc:2393] T 926a250fa5834e88b96ccecbefbf5086 P 197b880fa39d432a9b472845876bfaf0 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 974b802141bd4f13beffef2aa03f88f9 in current term 3: Already voted for candidate 197b880fa39d432a9b472845876bfaf0 in this term.
I20260129 02:59:23.051841 20434 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e4d18c35b0a7493e85d0a4b92956a48f" candidate_uuid: "974b802141bd4f13beffef2aa03f88f9" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "197b880fa39d432a9b472845876bfaf0"
W20260129 02:59:23.077633 20407 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.211.2:34517, user_credentials={real_user=slave}} blocked reactor thread for 94019.3us
I20260129 02:59:23.080572 20441 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00e79d8f25a34676b1d866eb2068a2b2" candidate_uuid: "974b802141bd4f13beffef2aa03f88f9" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "197b880fa39d432a9b472845876bfaf0" is_pre_election: true
I20260129 02:59:23.084115 20438 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1ce4ddf5b10e4ae8b193b13b2e5936ee" candidate_uuid: "974b802141bd4f13beffef2aa03f88f9" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "197b880fa39d432a9b472845876bfaf0"
W20260129 02:59:23.086279 20554 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.211.4:40439, user_credentials={real_user=slave}} blocked reactor thread for 199355us
I20260129 02:59:23.085795 20440 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3ccac82a01a2445fa5b0ce6c0fcf2295" candidate_uuid: "974b802141bd4f13beffef2aa03f88f9" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "197b880fa39d432a9b472845876bfaf0" is_pre_election: true
I20260129 02:59:23.089833 20587 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "8055232cf0004d40a85f3bf983fe00a9" candidate_uuid: "197b880fa39d432a9b472845876bfaf0" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1c50de89a5a845668696dcb28c3badd6" is_pre_election: true
I20260129 02:59:23.073369 21132 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00e79d8f25a34676b1d866eb2068a2b2" candidate_uuid: "197b880fa39d432a9b472845876bfaf0" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "7a9daf5d42a84615a6383cfec9a10608" is_pre_election: true
I20260129 02:59:23.074867 20439 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fcddab6bca684adf8ea032289a00f90e" candidate_uuid: "974b802141bd4f13beffef2aa03f88f9" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "197b880fa39d432a9b472845876bfaf0" is_pre_election: true
I20260129 02:59:23.096207 20438 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a8479d1fe0bd41e5946186e3a6ca8c7a" candidate_uuid: "974b802141bd4f13beffef2aa03f88f9" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "197b880fa39d432a9b472845876bfaf0"
I20260129 02:59:23.096611 20441 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a4c1226b987b468fa84dfc6ba0e2dbe1" candidate_uuid: "1c50de89a5a845668696dcb28c3badd6" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "197b880fa39d432a9b472845876bfaf0" is_pre_election: true
I20260129 02:59:23.070595 21954 raft_consensus.cc:515] T dc2f4cb752cf4d59b5f6bc689f5d8104 P 7a9daf5d42a84615a6383cfec9a10608 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } }
I20260129 02:59:23.100857 20439 raft_consensus.cc:2468] T fcddab6bca684adf8ea032289a00f90e P 197b880fa39d432a9b472845876bfaf0 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 974b802141bd4f13beffef2aa03f88f9 in term 4.
I20260129 02:59:23.101541 20441 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "368953c9102643b2a21f99af36b51905" candidate_uuid: "1c50de89a5a845668696dcb28c3badd6" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "197b880fa39d432a9b472845876bfaf0" is_pre_election: true
I20260129 02:59:23.103134 20439 raft_consensus.cc:1240] T a5a05fbcbbcf4916bbc01fd1a5b5e90d P 197b880fa39d432a9b472845876bfaf0 [term 3 FOLLOWER]: Rejecting Update request from peer 1c50de89a5a845668696dcb28c3badd6 for earlier term 2. Current term is 3. Ops: []
I20260129 02:59:23.103667 20441 raft_consensus.cc:1240] T 926a250fa5834e88b96ccecbefbf5086 P 197b880fa39d432a9b472845876bfaf0 [term 3 FOLLOWER]: Rejecting Update request from peer 1c50de89a5a845668696dcb28c3badd6 for earlier term 2. Current term is 3. Ops: []
I20260129 02:59:23.101675 21954 leader_election.cc:290] T dc2f4cb752cf4d59b5f6bc689f5d8104 P 7a9daf5d42a84615a6383cfec9a10608 [CANDIDATE]: Term 2 election: Requested vote from peers 197b880fa39d432a9b472845876bfaf0 (127.18.211.1:42027), 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3:37735)
I20260129 02:59:23.102139 20438 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a0fcc58d6fb6473cafbd9d34d5195269" candidate_uuid: "1c50de89a5a845668696dcb28c3badd6" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "197b880fa39d432a9b472845876bfaf0" is_pre_election: true
I20260129 02:59:23.107120 20440 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "75be4bd6bd45474795fd472eff2953ac" candidate_uuid: "974b802141bd4f13beffef2aa03f88f9" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "197b880fa39d432a9b472845876bfaf0"
I20260129 02:59:23.111017 20587 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "36ff33be666a4c82954731691dbcf327" candidate_uuid: "197b880fa39d432a9b472845876bfaf0" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1c50de89a5a845668696dcb28c3badd6" is_pre_election: true
I20260129 02:59:23.106760 21787 raft_consensus.cc:2804] T 87d5fe48a44a408aaf03e894a67ecb2f P 197b880fa39d432a9b472845876bfaf0 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260129 02:59:23.111840 21787 raft_consensus.cc:493] T 87d5fe48a44a408aaf03e894a67ecb2f P 197b880fa39d432a9b472845876bfaf0 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 7a9daf5d42a84615a6383cfec9a10608)
I20260129 02:59:23.112246 21787 raft_consensus.cc:3060] T 87d5fe48a44a408aaf03e894a67ecb2f P 197b880fa39d432a9b472845876bfaf0 [term 2 FOLLOWER]: Advancing to term 3
I20260129 02:59:23.139448 20441 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4e3805ffa70242e8ade66da260dddf66" candidate_uuid: "1c50de89a5a845668696dcb28c3badd6" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "197b880fa39d432a9b472845876bfaf0" is_pre_election: true
I20260129 02:59:23.143399 20405 leader_election.cc:304] T 4e3805ffa70242e8ade66da260dddf66 P 197b880fa39d432a9b472845876bfaf0 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 197b880fa39d432a9b472845876bfaf0; no voters: 1c50de89a5a845668696dcb28c3badd6, 7a9daf5d42a84615a6383cfec9a10608
I20260129 02:59:23.130920 20437 consensus_queue.cc:237] T 1299723a4476420ab02fd8bfd7aa0645 P 197b880fa39d432a9b472845876bfaf0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 2.4, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } }
W20260129 02:59:23.139832 20553 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.211.1:42027, user_credentials={real_user=slave}} blocked reactor thread for 174835us
W20260129 02:59:23.159457 21965 log.cc:927] Time spent T 0678b8b468bd43cdbc3e485900844821 P 974b802141bd4f13beffef2aa03f88f9: Append to log took a long time: real 0.092s user 0.001s sys 0.000s
I20260129 02:59:23.160413 20432 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c934d31efd3f4e5195605ddc354d14e1" candidate_uuid: "974b802141bd4f13beffef2aa03f88f9" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "197b880fa39d432a9b472845876bfaf0" is_pre_election: true
I20260129 02:59:23.161777 21839 raft_consensus.cc:515] T b189f08e16a74a11980cc55d99359a0a P 974b802141bd4f13beffef2aa03f88f9 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:59:23.162993 21883 raft_consensus.cc:3055] T a8479d1fe0bd41e5946186e3a6ca8c7a P 197b880fa39d432a9b472845876bfaf0 [term 1 LEADER]: Stepping down as leader of term 1
I20260129 02:59:23.163431 21883 raft_consensus.cc:740] T a8479d1fe0bd41e5946186e3a6ca8c7a P 197b880fa39d432a9b472845876bfaf0 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 197b880fa39d432a9b472845876bfaf0, State: Running, Role: LEADER
I20260129 02:59:23.136648 21928 consensus_queue.cc:237] T a8479d1fe0bd41e5946186e3a6ca8c7a P 974b802141bd4f13beffef2aa03f88f9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:59:23.130122 21789 consensus_queue.cc:1059] T 1ce4ddf5b10e4ae8b193b13b2e5936ee P 197b880fa39d432a9b472845876bfaf0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.025s
I20260129 02:59:23.168713 21789 raft_consensus.cc:3055] T 1ce4ddf5b10e4ae8b193b13b2e5936ee P 197b880fa39d432a9b472845876bfaf0 [term 1 LEADER]: Stepping down as leader of term 1
I20260129 02:59:23.169060 21789 raft_consensus.cc:740] T 1ce4ddf5b10e4ae8b193b13b2e5936ee P 197b880fa39d432a9b472845876bfaf0 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 197b880fa39d432a9b472845876bfaf0, State: Running, Role: LEADER
I20260129 02:59:23.169873 21789 consensus_queue.cc:260] T 1ce4ddf5b10e4ae8b193b13b2e5936ee P 197b880fa39d432a9b472845876bfaf0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:59:23.157891 21942 raft_consensus.cc:2749] T 4e3805ffa70242e8ade66da260dddf66 P 197b880fa39d432a9b472845876bfaf0 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260129 02:59:23.167786 21864 raft_consensus.cc:515] T 1299723a4476420ab02fd8bfd7aa0645 P 7a9daf5d42a84615a6383cfec9a10608 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } attrs { replace: true } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } }
I20260129 02:59:23.170965 21786 raft_consensus.cc:3055] T e4d18c35b0a7493e85d0a4b92956a48f P 197b880fa39d432a9b472845876bfaf0 [term 1 LEADER]: Stepping down as leader of term 1
I20260129 02:59:23.177467 21786 raft_consensus.cc:740] T e4d18c35b0a7493e85d0a4b92956a48f P 197b880fa39d432a9b472845876bfaf0 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 197b880fa39d432a9b472845876bfaf0, State: Running, Role: LEADER
I20260129 02:59:23.178406 21786 consensus_queue.cc:260] T e4d18c35b0a7493e85d0a4b92956a48f P 197b880fa39d432a9b472845876bfaf0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:59:23.179680 21839 leader_election.cc:290] T b189f08e16a74a11980cc55d99359a0a P 974b802141bd4f13beffef2aa03f88f9 [CANDIDATE]: Term 3 election: Requested vote from peers 197b880fa39d432a9b472845876bfaf0 (127.18.211.1:42027), 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3:37735)
I20260129 02:59:23.181123 20433 raft_consensus.cc:1275] T 4e420f90884f4796997b5ea41e49e411 P 197b880fa39d432a9b472845876bfaf0 [term 2 FOLLOWER]: Refusing update from remote peer 974b802141bd4f13beffef2aa03f88f9: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260129 02:59:23.183346 20582 raft_consensus.cc:1275] T 96609ac68bdf4993a12e1c23355ca013 P 1c50de89a5a845668696dcb28c3badd6 [term 2 FOLLOWER]: Refusing update from remote peer 197b880fa39d432a9b472845876bfaf0: Log matching property violated. Preceding OpId in replica: term: 2 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20260129 02:59:23.179627 21786 raft_consensus.cc:3060] T e4d18c35b0a7493e85d0a4b92956a48f P 197b880fa39d432a9b472845876bfaf0 [term 1 FOLLOWER]: Advancing to term 2
I20260129 02:59:23.181775 20590 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a4c1226b987b468fa84dfc6ba0e2dbe1" candidate_uuid: "197b880fa39d432a9b472845876bfaf0" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1c50de89a5a845668696dcb28c3badd6" is_pre_election: true
I20260129 02:59:23.181949 20585 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a5a05fbcbbcf4916bbc01fd1a5b5e90d" candidate_uuid: "197b880fa39d432a9b472845876bfaf0" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "1c50de89a5a845668696dcb28c3badd6"
I20260129 02:59:23.187031 20586 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e7d83d4ccf684e76acb29a0dc844e1b8" candidate_uuid: "197b880fa39d432a9b472845876bfaf0" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "1c50de89a5a845668696dcb28c3badd6" is_pre_election: true
I20260129 02:59:23.188088 21864 leader_election.cc:290] T 1299723a4476420ab02fd8bfd7aa0645 P 7a9daf5d42a84615a6383cfec9a10608 [CANDIDATE]: Term 3 election: Requested vote from peers 197b880fa39d432a9b472845876bfaf0 (127.18.211.1:42027), 974b802141bd4f13beffef2aa03f88f9 (127.18.211.2:34517), 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3:37735)
I20260129 02:59:23.175750 21787 raft_consensus.cc:515] T 87d5fe48a44a408aaf03e894a67ecb2f P 197b880fa39d432a9b472845876bfaf0 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } }
I20260129 02:59:23.171049 21789 raft_consensus.cc:3060] T 1ce4ddf5b10e4ae8b193b13b2e5936ee P 197b880fa39d432a9b472845876bfaf0 [term 1 FOLLOWER]: Advancing to term 2
I20260129 02:59:23.155025 21926 raft_consensus.cc:697] T c891d2bff4ce4568861bbc992e91fbb0 P 974b802141bd4f13beffef2aa03f88f9 [term 6 LEADER]: Becoming Leader. State: Replica: 974b802141bd4f13beffef2aa03f88f9, State: Running, Role: LEADER
W20260129 02:59:23.202448 20554 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.211.4:40439, user_credentials={real_user=slave}} blocked reactor thread for 50427.2us
I20260129 02:59:23.205452 20407 leader_election.cc:304] T 00e79d8f25a34676b1d866eb2068a2b2 P 197b880fa39d432a9b472845876bfaf0 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 197b880fa39d432a9b472845876bfaf0, 974b802141bd4f13beffef2aa03f88f9; no voters:
I20260129 02:59:23.122238 21880 raft_consensus.cc:515] T cedce331e1584ac8b636c9b8a72b2ba0 P 197b880fa39d432a9b472845876bfaf0 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:59:23.211582 20586 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1299723a4476420ab02fd8bfd7aa0645" candidate_uuid: "7a9daf5d42a84615a6383cfec9a10608" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "1c50de89a5a845668696dcb28c3badd6"
W20260129 02:59:23.212329 20405 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.211.4:40439, user_credentials={real_user=slave}} blocked reactor thread for 69048.1us
I20260129 02:59:23.213773 20433 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "76f713c3331540d2a5b76164103e9eb1" candidate_uuid: "1c50de89a5a845668696dcb28c3badd6" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "197b880fa39d432a9b472845876bfaf0" is_pre_election: true
I20260129 02:59:23.213897 20583 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "dc2f4cb752cf4d59b5f6bc689f5d8104" candidate_uuid: "7a9daf5d42a84615a6383cfec9a10608" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "1c50de89a5a845668696dcb28c3badd6"
I20260129 02:59:23.214754 21942 raft_consensus.cc:2804] T 00e79d8f25a34676b1d866eb2068a2b2 P 197b880fa39d432a9b472845876bfaf0 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260129 02:59:23.215204 21942 raft_consensus.cc:493] T 00e79d8f25a34676b1d866eb2068a2b2 P 197b880fa39d432a9b472845876bfaf0 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 7a9daf5d42a84615a6383cfec9a10608)
I20260129 02:59:23.215581 21942 raft_consensus.cc:3060] T 00e79d8f25a34676b1d866eb2068a2b2 P 197b880fa39d432a9b472845876bfaf0 [term 2 FOLLOWER]: Advancing to term 3
I20260129 02:59:23.216274 20588 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "724a236524cb4e7ba3428880681d6cc2" candidate_uuid: "197b880fa39d432a9b472845876bfaf0" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1c50de89a5a845668696dcb28c3badd6" is_pre_election: true
I20260129 02:59:23.217233 20584 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "17d6431cc77a491e8464d68db4081a51" candidate_uuid: "197b880fa39d432a9b472845876bfaf0" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1c50de89a5a845668696dcb28c3badd6" is_pre_election: true
I20260129 02:59:23.217787 21976 tablet_replica.cc:333] T add8d4923d1a489e86882c3a25692b95 P 197b880fa39d432a9b472845876bfaf0: stopping tablet replica
I20260129 02:59:23.218909 21976 raft_consensus.cc:2243] T add8d4923d1a489e86882c3a25692b95 P 197b880fa39d432a9b472845876bfaf0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260129 02:59:23.219632 21976 raft_consensus.cc:2272] T add8d4923d1a489e86882c3a25692b95 P 197b880fa39d432a9b472845876bfaf0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260129 02:59:23.220085 21883 consensus_queue.cc:260] T a8479d1fe0bd41e5946186e3a6ca8c7a P 197b880fa39d432a9b472845876bfaf0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:59:23.230614 20508 raft_consensus.cc:1240] T 45cc5f3d2688486d9b6dbe7e666dedde P 974b802141bd4f13beffef2aa03f88f9 [term 2 FOLLOWER]: Rejecting Update request from peer 1c50de89a5a845668696dcb28c3badd6 for earlier term 1. Current term is 2. Ops: []
I20260129 02:59:23.227970 20511 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7a9a69cd82724b1faed1d10aa091fc4a" candidate_uuid: "1c50de89a5a845668696dcb28c3badd6" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "974b802141bd4f13beffef2aa03f88f9" is_pre_election: true
I20260129 02:59:23.230811 21883 raft_consensus.cc:3060] T a8479d1fe0bd41e5946186e3a6ca8c7a P 197b880fa39d432a9b472845876bfaf0 [term 1 FOLLOWER]: Advancing to term 2
I20260129 02:59:23.256381 20507 raft_consensus.cc:1240] T 75be4bd6bd45474795fd472eff2953ac P 974b802141bd4f13beffef2aa03f88f9 [term 5 FOLLOWER]: Rejecting Update request from peer 197b880fa39d432a9b472845876bfaf0 for earlier term 4. Current term is 5. Ops: []
I20260129 02:59:23.229453 20588 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "42d27a28d89b409e8807ec0efd97af69" candidate_uuid: "197b880fa39d432a9b472845876bfaf0" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "1c50de89a5a845668696dcb28c3badd6" is_pre_election: true
I20260129 02:59:23.232800 21880 leader_election.cc:290] T cedce331e1584ac8b636c9b8a72b2ba0 P 197b880fa39d432a9b472845876bfaf0 [CANDIDATE]: Term 3 election: Requested vote from peers 974b802141bd4f13beffef2aa03f88f9 (127.18.211.2:34517), 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3:37735)
I20260129 02:59:23.264693 20588 raft_consensus.cc:2468] T 42d27a28d89b409e8807ec0efd97af69 P 1c50de89a5a845668696dcb28c3badd6 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 197b880fa39d432a9b472845876bfaf0 in term 2.
I20260129 02:59:23.265827 20507 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "cedce331e1584ac8b636c9b8a72b2ba0" candidate_uuid: "197b880fa39d432a9b472845876bfaf0" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "974b802141bd4f13beffef2aa03f88f9"
I20260129 02:59:23.232296 20433 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4408006eb5174de1896b037cf3f18457" candidate_uuid: "1c50de89a5a845668696dcb28c3badd6" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "197b880fa39d432a9b472845876bfaf0" is_pre_election: true
I20260129 02:59:23.234267 21926 consensus_queue.cc:237] T c891d2bff4ce4568861bbc992e91fbb0 P 974b802141bd4f13beffef2aa03f88f9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 5.4, Last appended by leader: 4, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:59:23.235914 20510 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "368953c9102643b2a21f99af36b51905" candidate_uuid: "1c50de89a5a845668696dcb28c3badd6" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "974b802141bd4f13beffef2aa03f88f9"
I20260129 02:59:23.231920 20435 raft_consensus.cc:2393] T cedce331e1584ac8b636c9b8a72b2ba0 P 197b880fa39d432a9b472845876bfaf0 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 974b802141bd4f13beffef2aa03f88f9 in current term 3: Already voted for candidate 197b880fa39d432a9b472845876bfaf0 in this term.
I20260129 02:59:23.238797 21788 consensus_queue.cc:1059] T c891d2bff4ce4568861bbc992e91fbb0 P 197b880fa39d432a9b472845876bfaf0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 }, Status: INVALID_TERM, Last received: 5.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20260129 02:59:23.241392 21976 ts_tablet_manager.cc:1916] T add8d4923d1a489e86882c3a25692b95 P 197b880fa39d432a9b472845876bfaf0: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260129 02:59:23.241645 21942 raft_consensus.cc:515] T 00e79d8f25a34676b1d866eb2068a2b2 P 197b880fa39d432a9b472845876bfaf0 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } }
I20260129 02:59:23.242591 20436 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e4d18c35b0a7493e85d0a4b92956a48f" candidate_uuid: "1c50de89a5a845668696dcb28c3badd6" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "197b880fa39d432a9b472845876bfaf0" is_pre_election: true
I20260129 02:59:23.242976 20441 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7d9494f5584d4d5db8b9adf1c8fc888b" candidate_uuid: "1c50de89a5a845668696dcb28c3badd6" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "197b880fa39d432a9b472845876bfaf0" is_pre_election: true
I20260129 02:59:23.243404 20511 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "913fc7d751ed4b7388d889aac30a5f9b" candidate_uuid: "1c50de89a5a845668696dcb28c3badd6" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "974b802141bd4f13beffef2aa03f88f9" is_pre_election: true
I20260129 02:59:23.243672 20508 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3906b080a5b34a74b8b64128635e8997" candidate_uuid: "1c50de89a5a845668696dcb28c3badd6" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "974b802141bd4f13beffef2aa03f88f9" is_pre_election: true
I20260129 02:59:23.224900 20583 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a1a39dcd328e4268ae41b988c66b4a75" candidate_uuid: "197b880fa39d432a9b472845876bfaf0" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "1c50de89a5a845668696dcb28c3badd6" is_pre_election: true
I20260129 02:59:23.243965 20432 raft_consensus.cc:1240] T e33a417cf9c54cf5af73a4ee3aac4d3a P 197b880fa39d432a9b472845876bfaf0 [term 2 FOLLOWER]: Rejecting Update request from peer 1c50de89a5a845668696dcb28c3badd6 for earlier term 1. Current term is 2. Ops: []
I20260129 02:59:23.280169 20436 raft_consensus.cc:2468] T e4d18c35b0a7493e85d0a4b92956a48f P 197b880fa39d432a9b472845876bfaf0 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1c50de89a5a845668696dcb28c3badd6 in term 2.
I20260129 02:59:23.280258 20508 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "87d5fe48a44a408aaf03e894a67ecb2f" candidate_uuid: "197b880fa39d432a9b472845876bfaf0" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "974b802141bd4f13beffef2aa03f88f9"
I20260129 02:59:23.244064 21099 leader_election.cc:304] T fd8311b779af4e1fa1402b2a02a602f2 P 7a9daf5d42a84615a6383cfec9a10608 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 197b880fa39d432a9b472845876bfaf0, 7a9daf5d42a84615a6383cfec9a10608; no voters: 1c50de89a5a845668696dcb28c3badd6
I20260129 02:59:23.280973 20432 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1299723a4476420ab02fd8bfd7aa0645" candidate_uuid: "7a9daf5d42a84615a6383cfec9a10608" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "197b880fa39d432a9b472845876bfaf0"
I20260129 02:59:23.281281 20436 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "dc2f4cb752cf4d59b5f6bc689f5d8104" candidate_uuid: "7a9daf5d42a84615a6383cfec9a10608" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "197b880fa39d432a9b472845876bfaf0"
I20260129 02:59:23.247802 20440 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0f20af7fbe5d4db89a6f17948d98912c" candidate_uuid: "1c50de89a5a845668696dcb28c3badd6" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "197b880fa39d432a9b472845876bfaf0" is_pre_election: true
I20260129 02:59:23.282874 21954 raft_consensus.cc:2804] T fd8311b779af4e1fa1402b2a02a602f2 P 7a9daf5d42a84615a6383cfec9a10608 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260129 02:59:23.283299 21954 raft_consensus.cc:493] T fd8311b779af4e1fa1402b2a02a602f2 P 7a9daf5d42a84615a6383cfec9a10608 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 1c50de89a5a845668696dcb28c3badd6)
I20260129 02:59:23.283679 21954 raft_consensus.cc:3060] T fd8311b779af4e1fa1402b2a02a602f2 P 7a9daf5d42a84615a6383cfec9a10608 [term 1 FOLLOWER]: Advancing to term 2
I20260129 02:59:23.248821 20586 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e33a417cf9c54cf5af73a4ee3aac4d3a" candidate_uuid: "197b880fa39d432a9b472845876bfaf0" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1c50de89a5a845668696dcb28c3badd6"
I20260129 02:59:23.252700 20512 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0678b8b468bd43cdbc3e485900844821" candidate_uuid: "197b880fa39d432a9b472845876bfaf0" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "974b802141bd4f13beffef2aa03f88f9"
I20260129 02:59:23.285436 21942 leader_election.cc:290] T 00e79d8f25a34676b1d866eb2068a2b2 P 197b880fa39d432a9b472845876bfaf0 [CANDIDATE]: Term 3 election: Requested vote from peers 974b802141bd4f13beffef2aa03f88f9 (127.18.211.2:34517), 7a9daf5d42a84615a6383cfec9a10608 (127.18.211.4:40439)
I20260129 02:59:23.271162 21788 raft_consensus.cc:3055] T c891d2bff4ce4568861bbc992e91fbb0 P 197b880fa39d432a9b472845876bfaf0 [term 5 LEADER]: Stepping down as leader of term 5
I20260129 02:59:23.286402 21788 raft_consensus.cc:740] T c891d2bff4ce4568861bbc992e91fbb0 P 197b880fa39d432a9b472845876bfaf0 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 197b880fa39d432a9b472845876bfaf0, State: Running, Role: LEADER
I20260129 02:59:23.287375 21788 consensus_queue.cc:260] T c891d2bff4ce4568861bbc992e91fbb0 P 197b880fa39d432a9b472845876bfaf0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 5.4, Last appended by leader: 4, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:59:23.272578 21787 leader_election.cc:290] T 87d5fe48a44a408aaf03e894a67ecb2f P 197b880fa39d432a9b472845876bfaf0 [CANDIDATE]: Term 3 election: Requested vote from peers 974b802141bd4f13beffef2aa03f88f9 (127.18.211.2:34517), 7a9daf5d42a84615a6383cfec9a10608 (127.18.211.4:40439)
I20260129 02:59:23.273741 20438 raft_consensus.cc:1240] T 0678b8b468bd43cdbc3e485900844821 P 197b880fa39d432a9b472845876bfaf0 [term 3 FOLLOWER]: Rejecting Update request from peer 7a9daf5d42a84615a6383cfec9a10608 for earlier term 2. Current term is 3. Ops: []
I20260129 02:59:23.283679 20433 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2ac659d732314e1288849f08fe5e074e" candidate_uuid: "7a9daf5d42a84615a6383cfec9a10608" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "197b880fa39d432a9b472845876bfaf0" is_pre_election: true
W20260129 02:59:23.289846 20405 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.211.4:40439, user_credentials={real_user=slave}} blocked reactor thread for 63634.3us
I20260129 02:59:23.297860 20441 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "18f30ebaa4174fd28c958e72d0cc7e24" candidate_uuid: "1c50de89a5a845668696dcb28c3badd6" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "197b880fa39d432a9b472845876bfaf0" is_pre_election: true
W20260129 02:59:23.324829 20480 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.211.3:37735, user_credentials={real_user=slave}} blocked reactor thread for 359102us
I20260129 02:59:23.292560 21786 consensus_queue.cc:1059] T 75be4bd6bd45474795fd472eff2953ac P 197b880fa39d432a9b472845876bfaf0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 }, Status: INVALID_TERM, Last received: 4.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
W20260129 02:59:23.373661 20556 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.211.2:34517, user_credentials={real_user=slave}} blocked reactor thread for 82180.2us
W20260129 02:59:23.366874 20407 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.211.2:34517, user_credentials={real_user=slave}} blocked reactor thread for 73608.9us
I20260129 02:59:23.374754 20439 raft_consensus.cc:1240] T cedce331e1584ac8b636c9b8a72b2ba0 P 197b880fa39d432a9b472845876bfaf0 [term 3 FOLLOWER]: Rejecting Update request from peer 1c50de89a5a845668696dcb28c3badd6 for earlier term 2. Current term is 3. Ops: []
I20260129 02:59:23.269515 21897 raft_consensus.cc:515] T 78aa39096868415d9afc2ffcb248e198 P 7a9daf5d42a84615a6383cfec9a10608 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } }
I20260129 02:59:23.381738 21786 raft_consensus.cc:3055] T 75be4bd6bd45474795fd472eff2953ac P 197b880fa39d432a9b472845876bfaf0 [term 4 LEADER]: Stepping down as leader of term 4
I20260129 02:59:23.381987 20587 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1ce4ddf5b10e4ae8b193b13b2e5936ee" candidate_uuid: "974b802141bd4f13beffef2aa03f88f9" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1c50de89a5a845668696dcb28c3badd6"
I20260129 02:59:23.382350 21786 raft_consensus.cc:740] T 75be4bd6bd45474795fd472eff2953ac P 197b880fa39d432a9b472845876bfaf0 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 197b880fa39d432a9b472845876bfaf0, State: Running, Role: LEADER
I20260129 02:59:23.381767 21134 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3ccac82a01a2445fa5b0ce6c0fcf2295" candidate_uuid: "974b802141bd4f13beffef2aa03f88f9" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "7a9daf5d42a84615a6383cfec9a10608" is_pre_election: true
I20260129 02:59:23.384161 21959 raft_consensus.cc:493] T 4e420f90884f4796997b5ea41e49e411 P 7a9daf5d42a84615a6383cfec9a10608 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260129 02:59:23.384789 21959 raft_consensus.cc:515] T 4e420f90884f4796997b5ea41e49e411 P 7a9daf5d42a84615a6383cfec9a10608 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } }
I20260129 02:59:23.385494 20512 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00e79d8f25a34676b1d866eb2068a2b2" candidate_uuid: "197b880fa39d432a9b472845876bfaf0" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "974b802141bd4f13beffef2aa03f88f9"
I20260129 02:59:23.386135 20512 raft_consensus.cc:3060] T 00e79d8f25a34676b1d866eb2068a2b2 P 974b802141bd4f13beffef2aa03f88f9 [term 2 FOLLOWER]: Advancing to term 3
I20260129 02:59:23.387091 21959 leader_election.cc:290] T 4e420f90884f4796997b5ea41e49e411 P 7a9daf5d42a84615a6383cfec9a10608 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 197b880fa39d432a9b472845876bfaf0 (127.18.211.1:42027), 974b802141bd4f13beffef2aa03f88f9 (127.18.211.2:34517)
I20260129 02:59:23.374903 20433 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4b1876f9a70b49edb3aa74105e66d3b3" candidate_uuid: "1c50de89a5a845668696dcb28c3badd6" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "197b880fa39d432a9b472845876bfaf0" is_pre_election: true
I20260129 02:59:23.391026 20587 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3ccac82a01a2445fa5b0ce6c0fcf2295" candidate_uuid: "974b802141bd4f13beffef2aa03f88f9" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "1c50de89a5a845668696dcb28c3badd6" is_pre_election: true
I20260129 02:59:23.396158 20433 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "cb4727386a3e48aca5d71cc6c611c197" candidate_uuid: "1c50de89a5a845668696dcb28c3badd6" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "197b880fa39d432a9b472845876bfaf0" is_pre_election: true
I20260129 02:59:23.396365 21132 raft_consensus.cc:1240] T 78aa39096868415d9afc2ffcb248e198 P 7a9daf5d42a84615a6383cfec9a10608 [term 2 FOLLOWER]: Rejecting Update request from peer 1c50de89a5a845668696dcb28c3badd6 for earlier term 1. Current term is 2. Ops: []
I20260129 02:59:23.383441 21786 consensus_queue.cc:260] T 75be4bd6bd45474795fd472eff2953ac P 197b880fa39d432a9b472845876bfaf0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:59:23.398241 20438 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1299723a4476420ab02fd8bfd7aa0645" candidate_uuid: "1c50de89a5a845668696dcb28c3badd6" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "197b880fa39d432a9b472845876bfaf0" is_pre_election: true
I20260129 02:59:23.398818 20432 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c891d2bff4ce4568861bbc992e91fbb0" candidate_uuid: "1c50de89a5a845668696dcb28c3badd6" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "197b880fa39d432a9b472845876bfaf0" is_pre_election: true
I20260129 02:59:23.398929 20441 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4e420f90884f4796997b5ea41e49e411" candidate_uuid: "7a9daf5d42a84615a6383cfec9a10608" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "197b880fa39d432a9b472845876bfaf0" is_pre_election: true
I20260129 02:59:23.399417 20434 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b189f08e16a74a11980cc55d99359a0a" candidate_uuid: "1c50de89a5a845668696dcb28c3badd6" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "197b880fa39d432a9b472845876bfaf0" is_pre_election: true
I20260129 02:59:23.397625 20433 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "42d27a28d89b409e8807ec0efd97af69" candidate_uuid: "1c50de89a5a845668696dcb28c3badd6" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "197b880fa39d432a9b472845876bfaf0" is_pre_election: true
I20260129 02:59:23.399633 20441 raft_consensus.cc:2468] T 4e420f90884f4796997b5ea41e49e411 P 197b880fa39d432a9b472845876bfaf0 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7a9daf5d42a84615a6383cfec9a10608 in term 2.
I20260129 02:59:23.397624 21786 raft_consensus.cc:3060] T 75be4bd6bd45474795fd472eff2953ac P 197b880fa39d432a9b472845876bfaf0 [term 4 FOLLOWER]: Advancing to term 5
I20260129 02:59:23.395998 21954 raft_consensus.cc:515] T fd8311b779af4e1fa1402b2a02a602f2 P 7a9daf5d42a84615a6383cfec9a10608 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } }
I20260129 02:59:23.412932 20586 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "78aa39096868415d9afc2ffcb248e198" candidate_uuid: "7a9daf5d42a84615a6383cfec9a10608" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "1c50de89a5a845668696dcb28c3badd6"
I20260129 02:59:23.414510 21954 leader_election.cc:290] T fd8311b779af4e1fa1402b2a02a602f2 P 7a9daf5d42a84615a6383cfec9a10608 [CANDIDATE]: Term 2 election: Requested vote from peers 197b880fa39d432a9b472845876bfaf0 (127.18.211.1:42027), 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3:37735)
I20260129 02:59:23.403939 20405 leader_election.cc:304] T fd8311b779af4e1fa1402b2a02a602f2 P 197b880fa39d432a9b472845876bfaf0 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 197b880fa39d432a9b472845876bfaf0, 7a9daf5d42a84615a6383cfec9a10608; no voters: 1c50de89a5a845668696dcb28c3badd6
I20260129 02:59:23.408510 20583 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "75be4bd6bd45474795fd472eff2953ac" candidate_uuid: "974b802141bd4f13beffef2aa03f88f9" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "1c50de89a5a845668696dcb28c3badd6"
I20260129 02:59:23.409260 20440 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "96609ac68bdf4993a12e1c23355ca013" candidate_uuid: "1c50de89a5a845668696dcb28c3badd6" candidate_term: 3 candidate_status { last_received { term: 2 index: 3 } } ignore_live_leader: false dest_uuid: "197b880fa39d432a9b472845876bfaf0" is_pre_election: true
I20260129 02:59:23.411978 21897 leader_election.cc:290] T 78aa39096868415d9afc2ffcb248e198 P 7a9daf5d42a84615a6383cfec9a10608 [CANDIDATE]: Term 2 election: Requested vote from peers 197b880fa39d432a9b472845876bfaf0 (127.18.211.1:42027), 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3:37735)
I20260129 02:59:23.412693 21099 leader_election.cc:304] T 1299723a4476420ab02fd8bfd7aa0645 P 7a9daf5d42a84615a6383cfec9a10608 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 7a9daf5d42a84615a6383cfec9a10608; no voters: 197b880fa39d432a9b472845876bfaf0, 1c50de89a5a845668696dcb28c3badd6
I20260129 02:59:23.415580 20586 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fd8311b779af4e1fa1402b2a02a602f2" candidate_uuid: "7a9daf5d42a84615a6383cfec9a10608" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "1c50de89a5a845668696dcb28c3badd6"
I20260129 02:59:23.398161 20435 raft_consensus.cc:1240] T 00e79d8f25a34676b1d866eb2068a2b2 P 197b880fa39d432a9b472845876bfaf0 [term 3 FOLLOWER]: Rejecting Update request from peer 7a9daf5d42a84615a6383cfec9a10608 for earlier term 2. Current term is 3. Ops: []
I20260129 02:59:23.375636 21788 raft_consensus.cc:3060] T c891d2bff4ce4568861bbc992e91fbb0 P 197b880fa39d432a9b472845876bfaf0 [term 5 FOLLOWER]: Advancing to term 6
I20260129 02:59:23.421739 20585 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b189f08e16a74a11980cc55d99359a0a" candidate_uuid: "974b802141bd4f13beffef2aa03f88f9" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "1c50de89a5a845668696dcb28c3badd6"
I20260129 02:59:23.427328 20440 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3ccac82a01a2445fa5b0ce6c0fcf2295" candidate_uuid: "1c50de89a5a845668696dcb28c3badd6" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "197b880fa39d432a9b472845876bfaf0" is_pre_election: true
I20260129 02:59:23.433379 20435 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0340229d521b432580ae926f6062e3f7" candidate_uuid: "1c50de89a5a845668696dcb28c3badd6" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "197b880fa39d432a9b472845876bfaf0" is_pre_election: true
I20260129 02:59:23.436187 20511 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1299723a4476420ab02fd8bfd7aa0645" candidate_uuid: "7a9daf5d42a84615a6383cfec9a10608" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "974b802141bd4f13beffef2aa03f88f9"
I20260129 02:59:23.440666 20508 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2ac659d732314e1288849f08fe5e074e" candidate_uuid: "7a9daf5d42a84615a6383cfec9a10608" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "974b802141bd4f13beffef2aa03f88f9" is_pre_election: true
I20260129 02:59:23.435091 20437 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4f8f3a38f6e24ba4857875181f447442" candidate_uuid: "1c50de89a5a845668696dcb28c3badd6" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "197b880fa39d432a9b472845876bfaf0" is_pre_election: true
I20260129 02:59:23.424939 21880 raft_consensus.cc:2804] T fd8311b779af4e1fa1402b2a02a602f2 P 197b880fa39d432a9b472845876bfaf0 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260129 02:59:23.434720 20556 leader_election.cc:304] T 96609ac68bdf4993a12e1c23355ca013 P 1c50de89a5a845668696dcb28c3badd6 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1c50de89a5a845668696dcb28c3badd6, 974b802141bd4f13beffef2aa03f88f9; no voters:
I20260129 02:59:23.448283 20512 raft_consensus.cc:2468] T 00e79d8f25a34676b1d866eb2068a2b2 P 974b802141bd4f13beffef2aa03f88f9 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 197b880fa39d432a9b472845876bfaf0 in term 3.
I20260129 02:59:23.448239 21880 raft_consensus.cc:493] T fd8311b779af4e1fa1402b2a02a602f2 P 197b880fa39d432a9b472845876bfaf0 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 1c50de89a5a845668696dcb28c3badd6)
I20260129 02:59:23.453193 21880 raft_consensus.cc:3060] T fd8311b779af4e1fa1402b2a02a602f2 P 197b880fa39d432a9b472845876bfaf0 [term 1 FOLLOWER]: Advancing to term 2
I20260129 02:59:23.447681 20439 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "75be4bd6bd45474795fd472eff2953ac" candidate_uuid: "1c50de89a5a845668696dcb28c3badd6" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "197b880fa39d432a9b472845876bfaf0" is_pre_election: true
I20260129 02:59:23.449086 21944 raft_consensus.cc:2804] T 96609ac68bdf4993a12e1c23355ca013 P 1c50de89a5a845668696dcb28c3badd6 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260129 02:59:23.454882 21944 raft_consensus.cc:493] T 96609ac68bdf4993a12e1c23355ca013 P 1c50de89a5a845668696dcb28c3badd6 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 197b880fa39d432a9b472845876bfaf0)
I20260129 02:59:23.454974 20439 raft_consensus.cc:2468] T 75be4bd6bd45474795fd472eff2953ac P 197b880fa39d432a9b472845876bfaf0 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1c50de89a5a845668696dcb28c3badd6 in term 5.
I20260129 02:59:23.455286 21944 raft_consensus.cc:3060] T 96609ac68bdf4993a12e1c23355ca013 P 1c50de89a5a845668696dcb28c3badd6 [term 2 FOLLOWER]: Advancing to term 3
I20260129 02:59:23.457399 20440 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "724a236524cb4e7ba3428880681d6cc2" candidate_uuid: "1c50de89a5a845668696dcb28c3badd6" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "197b880fa39d432a9b472845876bfaf0" is_pre_election: true
W20260129 02:59:23.458851 20405 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.211.4:40439, user_credentials={real_user=slave}} blocked reactor thread for 55024.5us
I20260129 02:59:23.460223 20405 leader_election.cc:304] T c934d31efd3f4e5195605ddc354d14e1 P 197b880fa39d432a9b472845876bfaf0 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 197b880fa39d432a9b472845876bfaf0; no voters: 7a9daf5d42a84615a6383cfec9a10608, 974b802141bd4f13beffef2aa03f88f9
I20260129 02:59:23.462682 21942 raft_consensus.cc:2749] T c934d31efd3f4e5195605ddc354d14e1 P 197b880fa39d432a9b472845876bfaf0 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260129 02:59:23.462301 21944 raft_consensus.cc:515] T 96609ac68bdf4993a12e1c23355ca013 P 1c50de89a5a845668696dcb28c3badd6 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } attrs { replace: true } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: NON_VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: true } }
I20260129 02:59:23.464458 21944 leader_election.cc:290] T 96609ac68bdf4993a12e1c23355ca013 P 1c50de89a5a845668696dcb28c3badd6 [CANDIDATE]: Term 3 election: Requested vote from peers 197b880fa39d432a9b472845876bfaf0 (127.18.211.1:42027), 974b802141bd4f13beffef2aa03f88f9 (127.18.211.2:34517)
I20260129 02:59:23.465047 21976 ts_tablet_manager.cc:1929] T add8d4923d1a489e86882c3a25692b95 P 197b880fa39d432a9b472845876bfaf0: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20260129 02:59:23.465754 21976 log.cc:1199] T add8d4923d1a489e86882c3a25692b95 P 197b880fa39d432a9b472845876bfaf0: Deleting WAL directory at /tmp/dist-test-taskrTP_Jo/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1769655460890924-19276-0/minicluster-data/ts-0-root/wals/add8d4923d1a489e86882c3a25692b95
I20260129 02:59:23.467769 21099 leader_election.cc:304] T dc2f4cb752cf4d59b5f6bc689f5d8104 P 7a9daf5d42a84615a6383cfec9a10608 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7a9daf5d42a84615a6383cfec9a10608; no voters: 197b880fa39d432a9b472845876bfaf0, 1c50de89a5a845668696dcb28c3badd6
I20260129 02:59:23.462365 20405 leader_election.cc:304] T 2ac659d732314e1288849f08fe5e074e P 197b880fa39d432a9b472845876bfaf0 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 197b880fa39d432a9b472845876bfaf0; no voters: 7a9daf5d42a84615a6383cfec9a10608, 974b802141bd4f13beffef2aa03f88f9
I20260129 02:59:23.472985 20405 leader_election.cc:304] T 0678b8b468bd43cdbc3e485900844821 P 197b880fa39d432a9b472845876bfaf0 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 197b880fa39d432a9b472845876bfaf0; no voters: 7a9daf5d42a84615a6383cfec9a10608, 974b802141bd4f13beffef2aa03f88f9
I20260129 02:59:23.474907 21786 raft_consensus.cc:2749] T 0678b8b468bd43cdbc3e485900844821 P 197b880fa39d432a9b472845876bfaf0 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260129 02:59:23.475914 21864 raft_consensus.cc:2749] T dc2f4cb752cf4d59b5f6bc689f5d8104 P 7a9daf5d42a84615a6383cfec9a10608 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260129 02:59:23.474723 20322 catalog_manager.cc:4985] TS 197b880fa39d432a9b472845876bfaf0 (127.18.211.1:42027): tablet add8d4923d1a489e86882c3a25692b95 (table test-workload [id=de73611ac8e3485a98aecc5add247032]) successfully deleted
I20260129 02:59:23.478475 21880 raft_consensus.cc:515] T fd8311b779af4e1fa1402b2a02a602f2 P 197b880fa39d432a9b472845876bfaf0 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } }
I20260129 02:59:23.479179 21864 consensus_queue.cc:1059] T 0678b8b468bd43cdbc3e485900844821 P 7a9daf5d42a84615a6383cfec9a10608 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260129 02:59:23.480046 21863 raft_consensus.cc:3055] T 0678b8b468bd43cdbc3e485900844821 P 7a9daf5d42a84615a6383cfec9a10608 [term 2 LEADER]: Stepping down as leader of term 2
I20260129 02:59:23.480393 21863 raft_consensus.cc:740] T 0678b8b468bd43cdbc3e485900844821 P 7a9daf5d42a84615a6383cfec9a10608 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 7a9daf5d42a84615a6383cfec9a10608, State: Running, Role: LEADER
I20260129 02:59:23.480767 20508 raft_consensus.cc:1240] T 00e79d8f25a34676b1d866eb2068a2b2 P 974b802141bd4f13beffef2aa03f88f9 [term 3 FOLLOWER]: Rejecting Update request from peer 7a9daf5d42a84615a6383cfec9a10608 for earlier term 2. Current term is 3. Ops: []
I20260129 02:59:23.481173 21863 consensus_queue.cc:260] T 0678b8b468bd43cdbc3e485900844821 P 7a9daf5d42a84615a6383cfec9a10608 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } }
I20260129 02:59:23.482179 21099 leader_election.cc:304] T 4e420f90884f4796997b5ea41e49e411 P 7a9daf5d42a84615a6383cfec9a10608 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 197b880fa39d432a9b472845876bfaf0, 7a9daf5d42a84615a6383cfec9a10608; no voters:
I20260129 02:59:23.482407 21863 raft_consensus.cc:3060] T 0678b8b468bd43cdbc3e485900844821 P 7a9daf5d42a84615a6383cfec9a10608 [term 2 FOLLOWER]: Advancing to term 3
I20260129 02:59:23.482991 21836 raft_consensus.cc:2804] T 4e420f90884f4796997b5ea41e49e411 P 7a9daf5d42a84615a6383cfec9a10608 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260129 02:59:23.482923 20508 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4e420f90884f4796997b5ea41e49e411" candidate_uuid: "7a9daf5d42a84615a6383cfec9a10608" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "974b802141bd4f13beffef2aa03f88f9" is_pre_election: true
I20260129 02:59:23.429563 21897 raft_consensus.cc:2749] T 1299723a4476420ab02fd8bfd7aa0645 P 7a9daf5d42a84615a6383cfec9a10608 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260129 02:59:23.483994 20480 leader_election.cc:304] T a644c02b16af47679824ccd060ac10fd P 974b802141bd4f13beffef2aa03f88f9 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 974b802141bd4f13beffef2aa03f88f9; no voters: 1c50de89a5a845668696dcb28c3badd6, 7a9daf5d42a84615a6383cfec9a10608
I20260129 02:59:23.485571 21942 raft_consensus.cc:2749] T 2ac659d732314e1288849f08fe5e074e P 197b880fa39d432a9b472845876bfaf0 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260129 02:59:23.483417 21836 raft_consensus.cc:493] T 4e420f90884f4796997b5ea41e49e411 P 7a9daf5d42a84615a6383cfec9a10608 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260129 02:59:23.488216 21836 raft_consensus.cc:3060] T 4e420f90884f4796997b5ea41e49e411 P 7a9daf5d42a84615a6383cfec9a10608 [term 2 FOLLOWER]: Advancing to term 3
I20260129 02:59:23.488425 20407 leader_election.cc:304] T 00e79d8f25a34676b1d866eb2068a2b2 P 197b880fa39d432a9b472845876bfaf0 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 197b880fa39d432a9b472845876bfaf0, 974b802141bd4f13beffef2aa03f88f9; no voters:
I20260129 02:59:23.489924 21942 raft_consensus.cc:2804] T 00e79d8f25a34676b1d866eb2068a2b2 P 197b880fa39d432a9b472845876bfaf0 [term 3 FOLLOWER]: Leader election won for term 3
I20260129 02:59:23.490635 21942 raft_consensus.cc:697] T 00e79d8f25a34676b1d866eb2068a2b2 P 197b880fa39d432a9b472845876bfaf0 [term 3 LEADER]: Becoming Leader. State: Replica: 197b880fa39d432a9b472845876bfaf0, State: Running, Role: LEADER
I20260129 02:59:23.491531 21942 consensus_queue.cc:237] T 00e79d8f25a34676b1d866eb2068a2b2 P 197b880fa39d432a9b472845876bfaf0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } }
I20260129 02:59:23.495679 20480 leader_election.cc:304] T add8d4923d1a489e86882c3a25692b95 P 974b802141bd4f13beffef2aa03f88f9 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 974b802141bd4f13beffef2aa03f88f9; no voters: 1c50de89a5a845668696dcb28c3badd6, 7a9daf5d42a84615a6383cfec9a10608
I20260129 02:59:23.497504 21913 raft_consensus.cc:2749] T a644c02b16af47679824ccd060ac10fd P 974b802141bd4f13beffef2aa03f88f9 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260129 02:59:23.498952 21132 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "87d5fe48a44a408aaf03e894a67ecb2f" candidate_uuid: "197b880fa39d432a9b472845876bfaf0" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "7a9daf5d42a84615a6383cfec9a10608"
I20260129 02:59:23.498317 21789 raft_consensus.cc:2955] T 3ccac82a01a2445fa5b0ce6c0fcf2295 P 197b880fa39d432a9b472845876bfaf0 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 7a9daf5d42a84615a6383cfec9a10608 (127.18.211.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } attrs { replace: true } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } } }
I20260129 02:59:23.499584 21134 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00e79d8f25a34676b1d866eb2068a2b2" candidate_uuid: "197b880fa39d432a9b472845876bfaf0" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "7a9daf5d42a84615a6383cfec9a10608"
I20260129 02:59:23.438263 21945 consensus_queue.cc:1059] T 78aa39096868415d9afc2ffcb248e198 P 1c50de89a5a845668696dcb28c3badd6 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.063s
I20260129 02:59:23.501966 20439 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a8479d1fe0bd41e5946186e3a6ca8c7a" candidate_uuid: "1c50de89a5a845668696dcb28c3badd6" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "197b880fa39d432a9b472845876bfaf0" is_pre_election: true
I20260129 02:59:23.503073 20439 raft_consensus.cc:2468] T a8479d1fe0bd41e5946186e3a6ca8c7a P 197b880fa39d432a9b472845876bfaf0 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1c50de89a5a845668696dcb28c3badd6 in term 2.
I20260129 02:59:23.502130 21134 raft_consensus.cc:1240] T 1299723a4476420ab02fd8bfd7aa0645 P 7a9daf5d42a84615a6383cfec9a10608 [term 3 FOLLOWER]: Rejecting Update request from peer 197b880fa39d432a9b472845876bfaf0 for earlier term 2. Current term is 3. Ops: []
I20260129 02:59:23.505045 20405 leader_election.cc:304] T 2e58e9a5a8aa4cecafb464addb4f35ed P 197b880fa39d432a9b472845876bfaf0 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 197b880fa39d432a9b472845876bfaf0; no voters: 1c50de89a5a845668696dcb28c3badd6, 7a9daf5d42a84615a6383cfec9a10608
I20260129 02:59:23.505862 21883 raft_consensus.cc:2749] T 2e58e9a5a8aa4cecafb464addb4f35ed P 197b880fa39d432a9b472845876bfaf0 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260129 02:59:23.505805 20441 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "368953c9102643b2a21f99af36b51905" candidate_uuid: "1c50de89a5a845668696dcb28c3badd6" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "197b880fa39d432a9b472845876bfaf0"
I20260129 02:59:23.507206 20405 leader_election.cc:304] T fcddab6bca684adf8ea032289a00f90e P 197b880fa39d432a9b472845876bfaf0 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 197b880fa39d432a9b472845876bfaf0; no voters: 1c50de89a5a845668696dcb28c3badd6, 974b802141bd4f13beffef2aa03f88f9
I20260129 02:59:23.508298 21937 raft_consensus.cc:2749] T fcddab6bca684adf8ea032289a00f90e P 197b880fa39d432a9b472845876bfaf0 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260129 02:59:23.509454 20441 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7a9a69cd82724b1faed1d10aa091fc4a" candidate_uuid: "1c50de89a5a845668696dcb28c3badd6" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "197b880fa39d432a9b472845876bfaf0" is_pre_election: true
I20260129 02:59:23.510571 20440 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "913fc7d751ed4b7388d889aac30a5f9b" candidate_uuid: "1c50de89a5a845668696dcb28c3badd6" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "197b880fa39d432a9b472845876bfaf0" is_pre_election: true
I20260129 02:59:23.509461 20405 leader_election.cc:304] T a0fcc58d6fb6473cafbd9d34d5195269 P 197b880fa39d432a9b472845876bfaf0 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 197b880fa39d432a9b472845876bfaf0; no voters: 1c50de89a5a845668696dcb28c3badd6, 974b802141bd4f13beffef2aa03f88f9
I20260129 02:59:23.512576 21937 raft_consensus.cc:2749] T a0fcc58d6fb6473cafbd9d34d5195269 P 197b880fa39d432a9b472845876bfaf0 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260129 02:59:23.513265 20439 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3906b080a5b34a74b8b64128635e8997" candidate_uuid: "1c50de89a5a845668696dcb28c3badd6" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "197b880fa39d432a9b472845876bfaf0" is_pre_election: true
I20260129 02:59:23.505157 21945 raft_consensus.cc:3055] T 78aa39096868415d9afc2ffcb248e198 P 1c50de89a5a845668696dcb28c3badd6 [term 1 LEADER]: Stepping down as leader of term 1
I20260129 02:59:23.514411 21945 raft_consensus.cc:740] T 78aa39096868415d9afc2ffcb248e198 P 1c50de89a5a845668696dcb28c3badd6 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 1c50de89a5a845668696dcb28c3badd6, State: Running, Role: LEADER
I20260129 02:59:23.515125 21836 raft_consensus.cc:515] T 4e420f90884f4796997b5ea41e49e411 P 7a9daf5d42a84615a6383cfec9a10608 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } }
I20260129 02:59:23.516498 20405 leader_election.cc:304] T 7d9494f5584d4d5db8b9adf1c8fc888b P 197b880fa39d432a9b472845876bfaf0 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 197b880fa39d432a9b472845876bfaf0; no voters: 1c50de89a5a845668696dcb28c3badd6, 974b802141bd4f13beffef2aa03f88f9
I20260129 02:59:23.516590 20439 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4e3805ffa70242e8ade66da260dddf66" candidate_uuid: "1c50de89a5a845668696dcb28c3badd6" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "197b880fa39d432a9b472845876bfaf0" is_pre_election: true
I20260129 02:59:23.518522 20440 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a4c1226b987b468fa84dfc6ba0e2dbe1" candidate_uuid: "1c50de89a5a845668696dcb28c3badd6" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "197b880fa39d432a9b472845876bfaf0" is_pre_election: true
I20260129 02:59:23.494994 21894 consensus_queue.cc:1059] T 00e79d8f25a34676b1d866eb2068a2b2 P 7a9daf5d42a84615a6383cfec9a10608 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.026s
I20260129 02:59:23.520506 21937 raft_consensus.cc:2749] T 7d9494f5584d4d5db8b9adf1c8fc888b P 197b880fa39d432a9b472845876bfaf0 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260129 02:59:23.521549 21880 leader_election.cc:290] T fd8311b779af4e1fa1402b2a02a602f2 P 197b880fa39d432a9b472845876bfaf0 [CANDIDATE]: Term 2 election: Requested vote from peers 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3:37735), 7a9daf5d42a84615a6383cfec9a10608 (127.18.211.4:40439)
I20260129 02:59:23.522158 21864 raft_consensus.cc:3055] T 00e79d8f25a34676b1d866eb2068a2b2 P 7a9daf5d42a84615a6383cfec9a10608 [term 2 LEADER]: Stepping down as leader of term 2
I20260129 02:59:23.522203 20405 leader_election.cc:304] T 18f30ebaa4174fd28c958e72d0cc7e24 P 197b880fa39d432a9b472845876bfaf0 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 197b880fa39d432a9b472845876bfaf0; no voters: 1c50de89a5a845668696dcb28c3badd6, 7a9daf5d42a84615a6383cfec9a10608
I20260129 02:59:23.522671 21864 raft_consensus.cc:740] T 00e79d8f25a34676b1d866eb2068a2b2 P 7a9daf5d42a84615a6383cfec9a10608 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 7a9daf5d42a84615a6383cfec9a10608, State: Running, Role: LEADER
I20260129 02:59:23.524257 21880 raft_consensus.cc:2749] T 18f30ebaa4174fd28c958e72d0cc7e24 P 197b880fa39d432a9b472845876bfaf0 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260129 02:59:23.515566 21945 consensus_queue.cc:260] T 78aa39096868415d9afc2ffcb248e198 P 1c50de89a5a845668696dcb28c3badd6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } }
I20260129 02:59:23.527154 20405 leader_election.cc:304] T 78aa39096868415d9afc2ffcb248e198 P 197b880fa39d432a9b472845876bfaf0 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 197b880fa39d432a9b472845876bfaf0; no voters: 1c50de89a5a845668696dcb28c3badd6, 7a9daf5d42a84615a6383cfec9a10608
I20260129 02:59:23.524448 21864 consensus_queue.cc:260] T 00e79d8f25a34676b1d866eb2068a2b2 P 7a9daf5d42a84615a6383cfec9a10608 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } }
W20260129 02:59:23.527813 20556 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.211.2:34517, user_credentials={real_user=slave}} blocked reactor thread for 76462.4us
I20260129 02:59:23.528654 20556 leader_election.cc:304] T cb0b3f259ebc4310927338a8740e87b8 P 1c50de89a5a845668696dcb28c3badd6 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1c50de89a5a845668696dcb28c3badd6; no voters: 7a9daf5d42a84615a6383cfec9a10608, 974b802141bd4f13beffef2aa03f88f9
I20260129 02:59:23.525921 21926 raft_consensus.cc:2749] T add8d4923d1a489e86882c3a25692b95 P 974b802141bd4f13beffef2aa03f88f9 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260129 02:59:23.529394 21864 raft_consensus.cc:3060] T 00e79d8f25a34676b1d866eb2068a2b2 P 7a9daf5d42a84615a6383cfec9a10608 [term 2 FOLLOWER]: Advancing to term 3
I20260129 02:59:23.536182 21828 raft_consensus.cc:2749] T cb0b3f259ebc4310927338a8740e87b8 P 1c50de89a5a845668696dcb28c3badd6 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260129 02:59:23.497062 21787 consensus_queue.cc:1048] T 96609ac68bdf4993a12e1c23355ca013 P 197b880fa39d432a9b472845876bfaf0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.032s
I20260129 02:59:23.538847 20405 leader_election.cc:304] T 4408006eb5174de1896b037cf3f18457 P 197b880fa39d432a9b472845876bfaf0 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 197b880fa39d432a9b472845876bfaf0; no voters: 1c50de89a5a845668696dcb28c3badd6, 974b802141bd4f13beffef2aa03f88f9
I20260129 02:59:23.539412 20480 leader_election.cc:304] T 505580c93c694bbb8ea96fdf9b17ab46 P 974b802141bd4f13beffef2aa03f88f9 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 974b802141bd4f13beffef2aa03f88f9; no voters: 1c50de89a5a845668696dcb28c3badd6, 7a9daf5d42a84615a6383cfec9a10608
I20260129 02:59:23.540222 21926 raft_consensus.cc:2749] T 505580c93c694bbb8ea96fdf9b17ab46 P 974b802141bd4f13beffef2aa03f88f9 [term 2 LEADER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260129 02:59:23.541303 20440 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0f20af7fbe5d4db89a6f17948d98912c" candidate_uuid: "1c50de89a5a845668696dcb28c3badd6" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "197b880fa39d432a9b472845876bfaf0" is_pre_election: true
I20260129 02:59:23.542533 20441 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "76f713c3331540d2a5b76164103e9eb1" candidate_uuid: "1c50de89a5a845668696dcb28c3badd6" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "197b880fa39d432a9b472845876bfaf0" is_pre_election: true
I20260129 02:59:23.544100 20441 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "78aa39096868415d9afc2ffcb248e198" candidate_uuid: "7a9daf5d42a84615a6383cfec9a10608" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "197b880fa39d432a9b472845876bfaf0"
I20260129 02:59:23.538585 20439 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a0fcc58d6fb6473cafbd9d34d5195269" candidate_uuid: "1c50de89a5a845668696dcb28c3badd6" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "197b880fa39d432a9b472845876bfaf0" is_pre_election: true
I20260129 02:59:23.536320 21937 raft_consensus.cc:2749] T 78aa39096868415d9afc2ffcb248e198 P 197b880fa39d432a9b472845876bfaf0 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260129 02:59:23.547331 20440 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4408006eb5174de1896b037cf3f18457" candidate_uuid: "1c50de89a5a845668696dcb28c3badd6" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "197b880fa39d432a9b472845876bfaf0" is_pre_election: true
I20260129 02:59:23.538484 21945 raft_consensus.cc:3060] T 78aa39096868415d9afc2ffcb248e198 P 1c50de89a5a845668696dcb28c3badd6 [term 1 FOLLOWER]: Advancing to term 2
I20260129 02:59:23.557309 21928 consensus_queue.cc:1048] T 4e420f90884f4796997b5ea41e49e411 P 974b802141bd4f13beffef2aa03f88f9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20260129 02:59:23.559734 20405 leader_election.cc:304] T 76f713c3331540d2a5b76164103e9eb1 P 197b880fa39d432a9b472845876bfaf0 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 197b880fa39d432a9b472845876bfaf0; no voters: 1c50de89a5a845668696dcb28c3badd6, 974b802141bd4f13beffef2aa03f88f9
I20260129 02:59:23.560137 20440 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fd8311b779af4e1fa1402b2a02a602f2" candidate_uuid: "7a9daf5d42a84615a6383cfec9a10608" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "197b880fa39d432a9b472845876bfaf0"
I20260129 02:59:23.556411 20480 leader_election.cc:304] T 1299723a4476420ab02fd8bfd7aa0645 P 974b802141bd4f13beffef2aa03f88f9 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 974b802141bd4f13beffef2aa03f88f9; no voters: 1c50de89a5a845668696dcb28c3badd6, 7a9daf5d42a84615a6383cfec9a10608
I20260129 02:59:23.549093 21942 raft_consensus.cc:2749] T 4408006eb5174de1896b037cf3f18457 P 197b880fa39d432a9b472845876bfaf0 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260129 02:59:23.562812 20480 leader_election.cc:304] T 45cc5f3d2688486d9b6dbe7e666dedde P 974b802141bd4f13beffef2aa03f88f9 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 974b802141bd4f13beffef2aa03f88f9; no voters: 1c50de89a5a845668696dcb28c3badd6, 7a9daf5d42a84615a6383cfec9a10608
I20260129 02:59:23.555516 20432 raft_consensus.cc:2468] T c891d2bff4ce4568861bbc992e91fbb0 P 197b880fa39d432a9b472845876bfaf0 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1c50de89a5a845668696dcb28c3badd6 in term 6.
I20260129 02:59:23.566264 20440 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "368953c9102643b2a21f99af36b51905" candidate_uuid: "1c50de89a5a845668696dcb28c3badd6" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "197b880fa39d432a9b472845876bfaf0" is_pre_election: true
I20260129 02:59:23.567708 21971 consensus_queue.cc:1059] T 45cc5f3d2688486d9b6dbe7e666dedde P 1c50de89a5a845668696dcb28c3badd6 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260129 02:59:23.565884 21946 raft_consensus.cc:2749] T 45cc5f3d2688486d9b6dbe7e666dedde P 974b802141bd4f13beffef2aa03f88f9 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260129 02:59:23.565044 21928 raft_consensus.cc:2749] T 1299723a4476420ab02fd8bfd7aa0645 P 974b802141bd4f13beffef2aa03f88f9 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260129 02:59:23.568863 21984 raft_consensus.cc:3055] T 45cc5f3d2688486d9b6dbe7e666dedde P 1c50de89a5a845668696dcb28c3badd6 [term 1 LEADER]: Stepping down as leader of term 1
I20260129 02:59:23.572029 20512 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "96609ac68bdf4993a12e1c23355ca013" candidate_uuid: "1c50de89a5a845668696dcb28c3badd6" candidate_term: 3 candidate_status { last_received { term: 2 index: 3 } } ignore_live_leader: false dest_uuid: "974b802141bd4f13beffef2aa03f88f9"
I20260129 02:59:23.574050 20480 leader_election.cc:304] T 3ccac82a01a2445fa5b0ce6c0fcf2295 P 974b802141bd4f13beffef2aa03f88f9 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 974b802141bd4f13beffef2aa03f88f9; no voters: 1c50de89a5a845668696dcb28c3badd6, 7a9daf5d42a84615a6383cfec9a10608
I20260129 02:59:23.574481 20405 leader_election.cc:304] T dc2f4cb752cf4d59b5f6bc689f5d8104 P 197b880fa39d432a9b472845876bfaf0 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 197b880fa39d432a9b472845876bfaf0; no voters: 1c50de89a5a845668696dcb28c3badd6, 7a9daf5d42a84615a6383cfec9a10608
I20260129 02:59:23.573125 21984 raft_consensus.cc:740] T 45cc5f3d2688486d9b6dbe7e666dedde P 1c50de89a5a845668696dcb28c3badd6 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 1c50de89a5a845668696dcb28c3badd6, State: Running, Role: LEADER
I20260129 02:59:23.575678 21836 leader_election.cc:290] T 4e420f90884f4796997b5ea41e49e411 P 7a9daf5d42a84615a6383cfec9a10608 [CANDIDATE]: Term 3 election: Requested vote from peers 197b880fa39d432a9b472845876bfaf0 (127.18.211.1:42027), 974b802141bd4f13beffef2aa03f88f9 (127.18.211.2:34517)
I20260129 02:59:23.575731 20583 raft_consensus.cc:1275] T 1299723a4476420ab02fd8bfd7aa0645 P 1c50de89a5a845668696dcb28c3badd6 [term 2 FOLLOWER]: Refusing update from remote peer 197b880fa39d432a9b472845876bfaf0: Log matching property violated. Preceding OpId in replica: term: 2 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260129 02:59:23.576400 21984 consensus_queue.cc:260] T 45cc5f3d2688486d9b6dbe7e666dedde P 1c50de89a5a845668696dcb28c3badd6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } }
I20260129 02:59:23.576965 20584 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "cedce331e1584ac8b636c9b8a72b2ba0" candidate_uuid: "197b880fa39d432a9b472845876bfaf0" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "1c50de89a5a845668696dcb28c3badd6"
I20260129 02:59:23.573851 21787 raft_consensus.cc:2749] T 76f713c3331540d2a5b76164103e9eb1 P 197b880fa39d432a9b472845876bfaf0 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260129 02:59:23.578868 20432 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e4d18c35b0a7493e85d0a4b92956a48f" candidate_uuid: "1c50de89a5a845668696dcb28c3badd6" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "197b880fa39d432a9b472845876bfaf0" is_pre_election: true
I20260129 02:59:23.579579 20432 raft_consensus.cc:2468] T e4d18c35b0a7493e85d0a4b92956a48f P 197b880fa39d432a9b472845876bfaf0 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1c50de89a5a845668696dcb28c3badd6 in term 2.
I20260129 02:59:23.582418 20432 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "96609ac68bdf4993a12e1c23355ca013" candidate_uuid: "1c50de89a5a845668696dcb28c3badd6" candidate_term: 3 candidate_status { last_received { term: 2 index: 3 } } ignore_live_leader: false dest_uuid: "197b880fa39d432a9b472845876bfaf0"
I20260129 02:59:23.585088 20432 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4b1876f9a70b49edb3aa74105e66d3b3" candidate_uuid: "1c50de89a5a845668696dcb28c3badd6" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "197b880fa39d432a9b472845876bfaf0" is_pre_election: true
I20260129 02:59:23.588340 21942 raft_consensus.cc:2749] T dc2f4cb752cf4d59b5f6bc689f5d8104 P 197b880fa39d432a9b472845876bfaf0 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260129 02:59:23.577935 21984 raft_consensus.cc:3060] T 45cc5f3d2688486d9b6dbe7e666dedde P 1c50de89a5a845668696dcb28c3badd6 [term 1 FOLLOWER]: Advancing to term 2
I20260129 02:59:23.593504 20512 raft_consensus.cc:2955] T 3ccac82a01a2445fa5b0ce6c0fcf2295 P 974b802141bd4f13beffef2aa03f88f9 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 7a9daf5d42a84615a6383cfec9a10608 (127.18.211.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } attrs { replace: true } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } } }
W20260129 02:59:23.598251 20556 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260129 02:59:23.601737 21134 raft_consensus.cc:1240] T fd8311b779af4e1fa1402b2a02a602f2 P 7a9daf5d42a84615a6383cfec9a10608 [term 2 FOLLOWER]: Rejecting Update request from peer 1c50de89a5a845668696dcb28c3badd6 for earlier term 1. Current term is 2. Ops: []
I20260129 02:59:23.593734 20405 leader_election.cc:304] T 926a250fa5834e88b96ccecbefbf5086 P 197b880fa39d432a9b472845876bfaf0 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 197b880fa39d432a9b472845876bfaf0; no voters: 1c50de89a5a845668696dcb28c3badd6, 974b802141bd4f13beffef2aa03f88f9
I20260129 02:59:23.604799 21988 consensus_queue.cc:1059] T fd8311b779af4e1fa1402b2a02a602f2 P 1c50de89a5a845668696dcb28c3badd6 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20260129 02:59:23.605840 21988 raft_consensus.cc:3055] T fd8311b779af4e1fa1402b2a02a602f2 P 1c50de89a5a845668696dcb28c3badd6 [term 1 LEADER]: Stepping down as leader of term 1
I20260129 02:59:23.606187 21988 raft_consensus.cc:740] T fd8311b779af4e1fa1402b2a02a602f2 P 1c50de89a5a845668696dcb28c3badd6 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 1c50de89a5a845668696dcb28c3badd6, State: Running, Role: LEADER
I20260129 02:59:23.609896 21988 consensus_queue.cc:260] T fd8311b779af4e1fa1402b2a02a602f2 P 1c50de89a5a845668696dcb28c3badd6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } }
I20260129 02:59:23.611138 21988 raft_consensus.cc:3060] T fd8311b779af4e1fa1402b2a02a602f2 P 1c50de89a5a845668696dcb28c3badd6 [term 1 FOLLOWER]: Advancing to term 2
I20260129 02:59:23.613001 21099 leader_election.cc:304] T 78aa39096868415d9afc2ffcb248e198 P 7a9daf5d42a84615a6383cfec9a10608 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7a9daf5d42a84615a6383cfec9a10608; no voters: 197b880fa39d432a9b472845876bfaf0, 1c50de89a5a845668696dcb28c3badd6
I20260129 02:59:23.613821 21864 raft_consensus.cc:2749] T 78aa39096868415d9afc2ffcb248e198 P 7a9daf5d42a84615a6383cfec9a10608 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260129 02:59:23.614259 21099 leader_election.cc:304] T fd8311b779af4e1fa1402b2a02a602f2 P 7a9daf5d42a84615a6383cfec9a10608 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7a9daf5d42a84615a6383cfec9a10608; no voters: 197b880fa39d432a9b472845876bfaf0, 1c50de89a5a845668696dcb28c3badd6
I20260129 02:59:23.617730 20432 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4e420f90884f4796997b5ea41e49e411" candidate_uuid: "7a9daf5d42a84615a6383cfec9a10608" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "197b880fa39d432a9b472845876bfaf0"
I20260129 02:59:23.621176 20432 raft_consensus.cc:3060] T 4e420f90884f4796997b5ea41e49e411 P 197b880fa39d432a9b472845876bfaf0 [term 2 FOLLOWER]: Advancing to term 3
I20260129 02:59:23.623732 20479 leader_election.cc:304] T 0340229d521b432580ae926f6062e3f7 P 974b802141bd4f13beffef2aa03f88f9 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 974b802141bd4f13beffef2aa03f88f9; no voters: 197b880fa39d432a9b472845876bfaf0, 1c50de89a5a845668696dcb28c3badd6
W20260129 02:59:23.618101 20480 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.211.3:37735, user_credentials={real_user=slave}} blocked reactor thread for 44171.8us
I20260129 02:59:23.624949 21928 raft_consensus.cc:2749] T 0340229d521b432580ae926f6062e3f7 P 974b802141bd4f13beffef2aa03f88f9 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260129 02:59:23.618835 21789 raft_consensus.cc:2749] T 926a250fa5834e88b96ccecbefbf5086 P 197b880fa39d432a9b472845876bfaf0 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260129 02:59:23.619639 21984 consensus_queue.cc:1059] T a5a05fbcbbcf4916bbc01fd1a5b5e90d P 1c50de89a5a845668696dcb28c3badd6 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 }, Status: INVALID_TERM, Last received: 2.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260129 02:59:23.628404 21945 raft_consensus.cc:3055] T a5a05fbcbbcf4916bbc01fd1a5b5e90d P 1c50de89a5a845668696dcb28c3badd6 [term 2 LEADER]: Stepping down as leader of term 2
I20260129 02:59:23.628832 21945 raft_consensus.cc:740] T a5a05fbcbbcf4916bbc01fd1a5b5e90d P 1c50de89a5a845668696dcb28c3badd6 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 1c50de89a5a845668696dcb28c3badd6, State: Running, Role: LEADER
I20260129 02:59:23.629676 21945 consensus_queue.cc:260] T a5a05fbcbbcf4916bbc01fd1a5b5e90d P 1c50de89a5a845668696dcb28c3badd6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:59:23.631172 21945 raft_consensus.cc:3060] T a5a05fbcbbcf4916bbc01fd1a5b5e90d P 1c50de89a5a845668696dcb28c3badd6 [term 2 FOLLOWER]: Advancing to term 3
I20260129 02:59:23.632853 20432 raft_consensus.cc:2468] T 4e420f90884f4796997b5ea41e49e411 P 197b880fa39d432a9b472845876bfaf0 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7a9daf5d42a84615a6383cfec9a10608 in term 3.
I20260129 02:59:23.620414 20405 leader_election.cc:304] T 8055232cf0004d40a85f3bf983fe00a9 P 197b880fa39d432a9b472845876bfaf0 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 197b880fa39d432a9b472845876bfaf0; no voters: 1c50de89a5a845668696dcb28c3badd6, 974b802141bd4f13beffef2aa03f88f9
I20260129 02:59:23.641328 20405 leader_election.cc:304] T 36ff33be666a4c82954731691dbcf327 P 197b880fa39d432a9b472845876bfaf0 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 197b880fa39d432a9b472845876bfaf0; no voters: 1c50de89a5a845668696dcb28c3badd6, 974b802141bd4f13beffef2aa03f88f9
I20260129 02:59:23.642135 21942 raft_consensus.cc:2749] T 8055232cf0004d40a85f3bf983fe00a9 P 197b880fa39d432a9b472845876bfaf0 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260129 02:59:23.642791 21942 raft_consensus.cc:2749] T 36ff33be666a4c82954731691dbcf327 P 197b880fa39d432a9b472845876bfaf0 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260129 02:59:23.645170 21942 consensus_queue.cc:1048] T 96609ac68bdf4993a12e1c23355ca013 P 197b880fa39d432a9b472845876bfaf0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260129 02:59:23.649042 20405 leader_election.cc:304] T a4c1226b987b468fa84dfc6ba0e2dbe1 P 197b880fa39d432a9b472845876bfaf0 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 197b880fa39d432a9b472845876bfaf0; no voters: 1c50de89a5a845668696dcb28c3badd6, 974b802141bd4f13beffef2aa03f88f9
I20260129 02:59:23.636452 20479 leader_election.cc:304] T 926a250fa5834e88b96ccecbefbf5086 P 974b802141bd4f13beffef2aa03f88f9 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 974b802141bd4f13beffef2aa03f88f9; no voters: 197b880fa39d432a9b472845876bfaf0, 1c50de89a5a845668696dcb28c3badd6
I20260129 02:59:23.639803 21828 consensus_queue.cc:1059] T 926a250fa5834e88b96ccecbefbf5086 P 1c50de89a5a845668696dcb28c3badd6 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 }, Status: INVALID_TERM, Last received: 2.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260129 02:59:23.628296 21864 raft_consensus.cc:2749] T fd8311b779af4e1fa1402b2a02a602f2 P 7a9daf5d42a84615a6383cfec9a10608 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260129 02:59:23.650671 21945 raft_consensus.cc:3055] T 926a250fa5834e88b96ccecbefbf5086 P 1c50de89a5a845668696dcb28c3badd6 [term 2 LEADER]: Stepping down as leader of term 2
I20260129 02:59:23.651790 21928 raft_consensus.cc:3060] T 926a250fa5834e88b96ccecbefbf5086 P 974b802141bd4f13beffef2aa03f88f9 [term 2 FOLLOWER]: Advancing to term 3
I20260129 02:59:23.652468 20479 leader_election.cc:304] T a609ac02fae84725ab743f95f51d33ce P 974b802141bd4f13beffef2aa03f88f9 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 974b802141bd4f13beffef2aa03f88f9; no voters: 197b880fa39d432a9b472845876bfaf0, 7a9daf5d42a84615a6383cfec9a10608
I20260129 02:59:23.653715 20584 raft_consensus.cc:1275] T e4d18c35b0a7493e85d0a4b92956a48f P 1c50de89a5a845668696dcb28c3badd6 [term 2 FOLLOWER]: Refusing update from remote peer 974b802141bd4f13beffef2aa03f88f9: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260129 02:59:23.672020 20553 leader_election.cc:304] T 18f30ebaa4174fd28c958e72d0cc7e24 P 1c50de89a5a845668696dcb28c3badd6 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1c50de89a5a845668696dcb28c3badd6; no voters: 197b880fa39d432a9b472845876bfaf0, 7a9daf5d42a84615a6383cfec9a10608
I20260129 02:59:23.672916 21991 raft_consensus.cc:2749] T 18f30ebaa4174fd28c958e72d0cc7e24 P 1c50de89a5a845668696dcb28c3badd6 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260129 02:59:23.674933 20553 leader_election.cc:304] T cb4727386a3e48aca5d71cc6c611c197 P 1c50de89a5a845668696dcb28c3badd6 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1c50de89a5a845668696dcb28c3badd6; no voters: 197b880fa39d432a9b472845876bfaf0, 974b802141bd4f13beffef2aa03f88f9
I20260129 02:59:23.675724 21991 raft_consensus.cc:2749] T cb4727386a3e48aca5d71cc6c611c197 P 1c50de89a5a845668696dcb28c3badd6 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260129 02:59:23.677488 20553 leader_election.cc:304] T 1299723a4476420ab02fd8bfd7aa0645 P 1c50de89a5a845668696dcb28c3badd6 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 1c50de89a5a845668696dcb28c3badd6, 974b802141bd4f13beffef2aa03f88f9; no voters: 197b880fa39d432a9b472845876bfaf0, 7a9daf5d42a84615a6383cfec9a10608
I20260129 02:59:23.671198 21988 consensus_queue.cc:1059] T e33a417cf9c54cf5af73a4ee3aac4d3a P 1c50de89a5a845668696dcb28c3badd6 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.008s
I20260129 02:59:23.653712 20405 leader_election.cc:304] T a5a05fbcbbcf4916bbc01fd1a5b5e90d P 197b880fa39d432a9b472845876bfaf0 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 197b880fa39d432a9b472845876bfaf0; no voters: 1c50de89a5a845668696dcb28c3badd6, 974b802141bd4f13beffef2aa03f88f9
I20260129 02:59:23.654906 21789 raft_consensus.cc:2749] T a4c1226b987b468fa84dfc6ba0e2dbe1 P 197b880fa39d432a9b472845876bfaf0 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260129 02:59:23.663568 21134 raft_consensus.cc:1240] T 4e420f90884f4796997b5ea41e49e411 P 7a9daf5d42a84615a6383cfec9a10608 [term 3 FOLLOWER]: Rejecting Update request from peer 974b802141bd4f13beffef2aa03f88f9 for earlier term 2. Current term is 3. Ops: [2.5-2.5]
I20260129 02:59:23.663988 21102 leader_election.cc:304] T 2ac659d732314e1288849f08fe5e074e P 7a9daf5d42a84615a6383cfec9a10608 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7a9daf5d42a84615a6383cfec9a10608; no voters: 197b880fa39d432a9b472845876bfaf0, 974b802141bd4f13beffef2aa03f88f9
I20260129 02:59:23.665216 20508 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4e420f90884f4796997b5ea41e49e411" candidate_uuid: "7a9daf5d42a84615a6383cfec9a10608" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "974b802141bd4f13beffef2aa03f88f9"
I20260129 02:59:23.665215 21132 raft_consensus.cc:1275] T 505580c93c694bbb8ea96fdf9b17ab46 P 7a9daf5d42a84615a6383cfec9a10608 [term 2 FOLLOWER]: Refusing update from remote peer 974b802141bd4f13beffef2aa03f88f9: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260129 02:59:23.678685 21928 raft_consensus.cc:2749] T 926a250fa5834e88b96ccecbefbf5086 P 974b802141bd4f13beffef2aa03f88f9 [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260129 02:59:23.678862 21828 raft_consensus.cc:2749] T 1299723a4476420ab02fd8bfd7aa0645 P 1c50de89a5a845668696dcb28c3badd6 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20260129 02:59:23.695922 20404 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260129 02:59:23.693225 21942 raft_consensus.cc:2749] T a5a05fbcbbcf4916bbc01fd1a5b5e90d P 197b880fa39d432a9b472845876bfaf0 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260129 02:59:23.680003 20479 leader_election.cc:304] T 00e79d8f25a34676b1d866eb2068a2b2 P 974b802141bd4f13beffef2aa03f88f9 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 974b802141bd4f13beffef2aa03f88f9; no voters: 197b880fa39d432a9b472845876bfaf0, 7a9daf5d42a84615a6383cfec9a10608
I20260129 02:59:23.680346 21913 raft_consensus.cc:2749] T a609ac02fae84725ab743f95f51d33ce P 974b802141bd4f13beffef2aa03f88f9 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260129 02:59:23.690629 21864 raft_consensus.cc:2749] T 2ac659d732314e1288849f08fe5e074e P 7a9daf5d42a84615a6383cfec9a10608 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260129 02:59:23.680701 21945 raft_consensus.cc:740] T 926a250fa5834e88b96ccecbefbf5086 P 1c50de89a5a845668696dcb28c3badd6 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 1c50de89a5a845668696dcb28c3badd6, State: Running, Role: LEADER
I20260129 02:59:23.692982 21926 consensus_queue.cc:1048] T e4d18c35b0a7493e85d0a4b92956a48f P 974b802141bd4f13beffef2aa03f88f9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.015s
I20260129 02:59:23.708019 21945 consensus_queue.cc:260] T 926a250fa5834e88b96ccecbefbf5086 P 1c50de89a5a845668696dcb28c3badd6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:59:23.682835 20553 leader_election.cc:304] T b189f08e16a74a11980cc55d99359a0a P 1c50de89a5a845668696dcb28c3badd6 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1c50de89a5a845668696dcb28c3badd6; no voters: 197b880fa39d432a9b472845876bfaf0, 974b802141bd4f13beffef2aa03f88f9
I20260129 02:59:23.712009 21975 consensus_queue.cc:1059] T 4e420f90884f4796997b5ea41e49e411 P 974b802141bd4f13beffef2aa03f88f9 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260129 02:59:23.712522 20553 leader_election.cc:304] T 42d27a28d89b409e8807ec0efd97af69 P 1c50de89a5a845668696dcb28c3badd6 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1c50de89a5a845668696dcb28c3badd6; no voters: 197b880fa39d432a9b472845876bfaf0, 7a9daf5d42a84615a6383cfec9a10608
I20260129 02:59:23.713574 20440 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "febffe4a8b394368b69b1d164308fb9a" candidate_uuid: "974b802141bd4f13beffef2aa03f88f9" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "197b880fa39d432a9b472845876bfaf0" is_pre_election: true
I20260129 02:59:23.714114 20553 leader_election.cc:304] T 3ccac82a01a2445fa5b0ce6c0fcf2295 P 1c50de89a5a845668696dcb28c3badd6 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1c50de89a5a845668696dcb28c3badd6; no voters: 197b880fa39d432a9b472845876bfaf0, 974b802141bd4f13beffef2aa03f88f9
I20260129 02:59:23.714609 20440 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "368953c9102643b2a21f99af36b51905" candidate_uuid: "974b802141bd4f13beffef2aa03f88f9" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "197b880fa39d432a9b472845876bfaf0" is_pre_election: true
I20260129 02:59:23.715497 21828 raft_consensus.cc:2749] T b189f08e16a74a11980cc55d99359a0a P 1c50de89a5a845668696dcb28c3badd6 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260129 02:59:23.716228 20439 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b189f08e16a74a11980cc55d99359a0a" candidate_uuid: "974b802141bd4f13beffef2aa03f88f9" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "197b880fa39d432a9b472845876bfaf0"
I20260129 02:59:23.716683 21828 raft_consensus.cc:2749] T 42d27a28d89b409e8807ec0efd97af69 P 1c50de89a5a845668696dcb28c3badd6 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260129 02:59:23.716813 21944 raft_consensus.cc:2749] T 3ccac82a01a2445fa5b0ce6c0fcf2295 P 1c50de89a5a845668696dcb28c3badd6 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260129 02:59:23.692540 20405 leader_election.cc:304] T e7d83d4ccf684e76acb29a0dc844e1b8 P 197b880fa39d432a9b472845876bfaf0 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 197b880fa39d432a9b472845876bfaf0; no voters: 1c50de89a5a845668696dcb28c3badd6, 974b802141bd4f13beffef2aa03f88f9
I20260129 02:59:23.709407 20433 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "96609ac68bdf4993a12e1c23355ca013" candidate_uuid: "974b802141bd4f13beffef2aa03f88f9" candidate_term: 3 candidate_status { last_received { term: 2 index: 3 } } ignore_live_leader: false dest_uuid: "197b880fa39d432a9b472845876bfaf0" is_pre_election: true
I20260129 02:59:23.717375 20439 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "cedce331e1584ac8b636c9b8a72b2ba0" candidate_uuid: "974b802141bd4f13beffef2aa03f88f9" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "197b880fa39d432a9b472845876bfaf0" is_pre_election: true
I20260129 02:59:23.717432 21828 consensus_queue.cc:1059] T cedce331e1584ac8b636c9b8a72b2ba0 P 1c50de89a5a845668696dcb28c3badd6 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 }, Status: INVALID_TERM, Last received: 2.2, Next index: 3, Last known committed idx: 1, Time since last communication: 0.000s
I20260129 02:59:23.718266 20439 raft_consensus.cc:2393] T cedce331e1584ac8b636c9b8a72b2ba0 P 197b880fa39d432a9b472845876bfaf0 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 974b802141bd4f13beffef2aa03f88f9 in current term 3: Already voted for candidate 197b880fa39d432a9b472845876bfaf0 in this term.
I20260129 02:59:23.718497 20433 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e7d83d4ccf684e76acb29a0dc844e1b8" candidate_uuid: "974b802141bd4f13beffef2aa03f88f9" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "197b880fa39d432a9b472845876bfaf0" is_pre_election: true
I20260129 02:59:23.718727 21913 raft_consensus.cc:3055] T 4e420f90884f4796997b5ea41e49e411 P 974b802141bd4f13beffef2aa03f88f9 [term 2 LEADER]: Stepping down as leader of term 2
I20260129 02:59:23.719064 21913 raft_consensus.cc:740] T 4e420f90884f4796997b5ea41e49e411 P 974b802141bd4f13beffef2aa03f88f9 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 974b802141bd4f13beffef2aa03f88f9, State: Running, Role: LEADER
I20260129 02:59:23.719022 20439 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a1a39dcd328e4268ae41b988c66b4a75" candidate_uuid: "974b802141bd4f13beffef2aa03f88f9" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "197b880fa39d432a9b472845876bfaf0" is_pre_election: true
I20260129 02:59:23.709872 20441 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "87d5fe48a44a408aaf03e894a67ecb2f" candidate_uuid: "974b802141bd4f13beffef2aa03f88f9" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "197b880fa39d432a9b472845876bfaf0" is_pre_election: true
I20260129 02:59:23.720180 20435 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e33a417cf9c54cf5af73a4ee3aac4d3a" candidate_uuid: "974b802141bd4f13beffef2aa03f88f9" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "197b880fa39d432a9b472845876bfaf0" is_pre_election: true
I20260129 02:59:23.719973 21885 raft_consensus.cc:2749] T 00e79d8f25a34676b1d866eb2068a2b2 P 974b802141bd4f13beffef2aa03f88f9 [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260129 02:59:23.720989 20435 raft_consensus.cc:2393] T e33a417cf9c54cf5af73a4ee3aac4d3a P 197b880fa39d432a9b472845876bfaf0 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 974b802141bd4f13beffef2aa03f88f9 in current term 2: Already voted for candidate 197b880fa39d432a9b472845876bfaf0 in this term.
I20260129 02:59:23.721042 20441 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1ce4ddf5b10e4ae8b193b13b2e5936ee" candidate_uuid: "974b802141bd4f13beffef2aa03f88f9" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "197b880fa39d432a9b472845876bfaf0" is_pre_election: true
I20260129 02:59:23.721699 20441 raft_consensus.cc:2468] T 1ce4ddf5b10e4ae8b193b13b2e5936ee P 197b880fa39d432a9b472845876bfaf0 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 974b802141bd4f13beffef2aa03f88f9 in term 2.
I20260129 02:59:23.716658 20432 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b189f08e16a74a11980cc55d99359a0a" candidate_uuid: "974b802141bd4f13beffef2aa03f88f9" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "197b880fa39d432a9b472845876bfaf0" is_pre_election: true
I20260129 02:59:23.710230 20434 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "913fc7d751ed4b7388d889aac30a5f9b" candidate_uuid: "974b802141bd4f13beffef2aa03f88f9" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "197b880fa39d432a9b472845876bfaf0" is_pre_election: true
I20260129 02:59:23.710943 20438 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4e420f90884f4796997b5ea41e49e411" candidate_uuid: "974b802141bd4f13beffef2aa03f88f9" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "197b880fa39d432a9b472845876bfaf0"
I20260129 02:59:23.723678 20438 raft_consensus.cc:2368] T 4e420f90884f4796997b5ea41e49e411 P 197b880fa39d432a9b472845876bfaf0 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 974b802141bd4f13beffef2aa03f88f9 for earlier term 2. Current term is 3.
I20260129 02:59:23.710965 20436 raft_consensus.cc:1275] T e4d18c35b0a7493e85d0a4b92956a48f P 197b880fa39d432a9b472845876bfaf0 [term 2 FOLLOWER]: Refusing update from remote peer 974b802141bd4f13beffef2aa03f88f9: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260129 02:59:23.656842 21099 leader_election.cc:304] T 4e420f90884f4796997b5ea41e49e411 P 7a9daf5d42a84615a6383cfec9a10608 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 197b880fa39d432a9b472845876bfaf0, 7a9daf5d42a84615a6383cfec9a10608; no voters:
W20260129 02:59:23.725116 21099 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.211.1:42027, user_credentials={real_user=slave}} blocked reactor thread for 68376.3us
I20260129 02:59:23.725958 20437 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0678b8b468bd43cdbc3e485900844821" candidate_uuid: "974b802141bd4f13beffef2aa03f88f9" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "197b880fa39d432a9b472845876bfaf0" is_pre_election: true
I20260129 02:59:23.718103 20479 leader_election.cc:304] T 1ce4ddf5b10e4ae8b193b13b2e5936ee P 974b802141bd4f13beffef2aa03f88f9 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 974b802141bd4f13beffef2aa03f88f9; no voters: 197b880fa39d432a9b472845876bfaf0, 1c50de89a5a845668696dcb28c3badd6
I20260129 02:59:23.726740 20437 raft_consensus.cc:2393] T 0678b8b468bd43cdbc3e485900844821 P 197b880fa39d432a9b472845876bfaf0 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 974b802141bd4f13beffef2aa03f88f9 in current term 3: Already voted for candidate 197b880fa39d432a9b472845876bfaf0 in this term.
I20260129 02:59:23.726658 21863 raft_consensus.cc:2804] T 4e420f90884f4796997b5ea41e49e411 P 7a9daf5d42a84615a6383cfec9a10608 [term 3 FOLLOWER]: Leader election won for term 3
I20260129 02:59:23.727435 21863 raft_consensus.cc:697] T 4e420f90884f4796997b5ea41e49e411 P 7a9daf5d42a84615a6383cfec9a10608 [term 3 LEADER]: Becoming Leader. State: Replica: 7a9daf5d42a84615a6383cfec9a10608, State: Running, Role: LEADER
I20260129 02:59:23.715575 20440 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a5a05fbcbbcf4916bbc01fd1a5b5e90d" candidate_uuid: "974b802141bd4f13beffef2aa03f88f9" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "197b880fa39d432a9b472845876bfaf0" is_pre_election: true
I20260129 02:59:23.729912 20507 raft_consensus.cc:1240] T b189f08e16a74a11980cc55d99359a0a P 974b802141bd4f13beffef2aa03f88f9 [term 3 FOLLOWER]: Rejecting Update request from peer 197b880fa39d432a9b472845876bfaf0 for earlier term 2. Current term is 3. Ops: []
I20260129 02:59:23.731135 21863 consensus_queue.cc:237] T 4e420f90884f4796997b5ea41e49e411 P 7a9daf5d42a84615a6383cfec9a10608 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } }
I20260129 02:59:23.720045 21913 consensus_queue.cc:260] T 4e420f90884f4796997b5ea41e49e411 P 974b802141bd4f13beffef2aa03f88f9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 2.5, Last appended by leader: 5, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } }
I20260129 02:59:23.720214 21944 raft_consensus.cc:3055] T cedce331e1584ac8b636c9b8a72b2ba0 P 1c50de89a5a845668696dcb28c3badd6 [term 2 LEADER]: Stepping down as leader of term 2
I20260129 02:59:23.736737 21944 raft_consensus.cc:740] T cedce331e1584ac8b636c9b8a72b2ba0 P 1c50de89a5a845668696dcb28c3badd6 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 1c50de89a5a845668696dcb28c3badd6, State: Running, Role: LEADER
I20260129 02:59:23.737828 20432 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0340229d521b432580ae926f6062e3f7" candidate_uuid: "1c50de89a5a845668696dcb28c3badd6" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "197b880fa39d432a9b472845876bfaf0" is_pre_election: true
I20260129 02:59:23.739300 20434 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "724a236524cb4e7ba3428880681d6cc2" candidate_uuid: "1c50de89a5a845668696dcb28c3badd6" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "197b880fa39d432a9b472845876bfaf0" is_pre_election: true
I20260129 02:59:23.739430 20432 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7d9494f5584d4d5db8b9adf1c8fc888b" candidate_uuid: "1c50de89a5a845668696dcb28c3badd6" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "197b880fa39d432a9b472845876bfaf0" is_pre_election: true
I20260129 02:59:23.739418 21926 raft_consensus.cc:2749] T 1ce4ddf5b10e4ae8b193b13b2e5936ee P 974b802141bd4f13beffef2aa03f88f9 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260129 02:59:23.737587 21944 consensus_queue.cc:260] T cedce331e1584ac8b636c9b8a72b2ba0 P 1c50de89a5a845668696dcb28c3badd6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:59:23.736377 20437 raft_consensus.cc:1240] T 87d5fe48a44a408aaf03e894a67ecb2f P 197b880fa39d432a9b472845876bfaf0 [term 3 FOLLOWER]: Rejecting Update request from peer 7a9daf5d42a84615a6383cfec9a10608 for earlier term 2. Current term is 3. Ops: []
I20260129 02:59:23.744158 20479 leader_election.cc:304] T fcddab6bca684adf8ea032289a00f90e P 974b802141bd4f13beffef2aa03f88f9 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 197b880fa39d432a9b472845876bfaf0, 974b802141bd4f13beffef2aa03f88f9; no voters: 1c50de89a5a845668696dcb28c3badd6
I20260129 02:59:23.719573 20433 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3906b080a5b34a74b8b64128635e8997" candidate_uuid: "974b802141bd4f13beffef2aa03f88f9" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "197b880fa39d432a9b472845876bfaf0" is_pre_election: true
I20260129 02:59:23.744995 21928 raft_consensus.cc:2804] T fcddab6bca684adf8ea032289a00f90e P 974b802141bd4f13beffef2aa03f88f9 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260129 02:59:23.745383 21928 raft_consensus.cc:493] T fcddab6bca684adf8ea032289a00f90e P 974b802141bd4f13beffef2aa03f88f9 [term 4 FOLLOWER]: Starting leader election (detected failure of leader 1c50de89a5a845668696dcb28c3badd6)
W20260129 02:59:23.745738 20405 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.211.3:37735, user_credentials={real_user=slave}} blocked reactor thread for 53221.2us
I20260129 02:59:23.745752 21928 raft_consensus.cc:3060] T fcddab6bca684adf8ea032289a00f90e P 974b802141bd4f13beffef2aa03f88f9 [term 4 FOLLOWER]: Advancing to term 5
I20260129 02:59:23.745886 20479 leader_election.cc:304] T 75be4bd6bd45474795fd472eff2953ac P 974b802141bd4f13beffef2aa03f88f9 [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 974b802141bd4f13beffef2aa03f88f9; no voters: 197b880fa39d432a9b472845876bfaf0, 1c50de89a5a845668696dcb28c3badd6
I20260129 02:59:23.745904 20440 raft_consensus.cc:2393] T a5a05fbcbbcf4916bbc01fd1a5b5e90d P 197b880fa39d432a9b472845876bfaf0 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 974b802141bd4f13beffef2aa03f88f9 in current term 3: Already voted for candidate 197b880fa39d432a9b472845876bfaf0 in this term.
I20260129 02:59:23.747628 20405 leader_election.cc:304] T 724a236524cb4e7ba3428880681d6cc2 P 197b880fa39d432a9b472845876bfaf0 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 197b880fa39d432a9b472845876bfaf0; no voters: 1c50de89a5a845668696dcb28c3badd6, 974b802141bd4f13beffef2aa03f88f9
I20260129 02:59:23.747529 21946 raft_consensus.cc:2749] T 3ccac82a01a2445fa5b0ce6c0fcf2295 P 974b802141bd4f13beffef2aa03f88f9 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260129 02:59:23.748517 21926 raft_consensus.cc:2749] T 75be4bd6bd45474795fd472eff2953ac P 974b802141bd4f13beffef2aa03f88f9 [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20260129 02:59:23.748100 20479 leader_election.cc:304] T c934d31efd3f4e5195605ddc354d14e1 P 974b802141bd4f13beffef2aa03f88f9 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 974b802141bd4f13beffef2aa03f88f9; no voters: 197b880fa39d432a9b472845876bfaf0, 7a9daf5d42a84615a6383cfec9a10608
I20260129 02:59:23.750409 21946 raft_consensus.cc:2749] T c934d31efd3f4e5195605ddc354d14e1 P 974b802141bd4f13beffef2aa03f88f9 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260129 02:59:23.745767 21789 raft_consensus.cc:2749] T e7d83d4ccf684e76acb29a0dc844e1b8 P 197b880fa39d432a9b472845876bfaf0 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260129 02:59:23.751359 20479 leader_election.cc:304] T febffe4a8b394368b69b1d164308fb9a P 974b802141bd4f13beffef2aa03f88f9 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 974b802141bd4f13beffef2aa03f88f9; no voters: 197b880fa39d432a9b472845876bfaf0, 7a9daf5d42a84615a6383cfec9a10608
I20260129 02:59:23.751940 21883 consensus_queue.cc:1059] T b189f08e16a74a11980cc55d99359a0a P 197b880fa39d432a9b472845876bfaf0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 }, Status: INVALID_TERM, Last received: 2.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260129 02:59:23.753072 21880 raft_consensus.cc:3055] T b189f08e16a74a11980cc55d99359a0a P 197b880fa39d432a9b472845876bfaf0 [term 2 LEADER]: Stepping down as leader of term 2
I20260129 02:59:23.753336 21984 raft_consensus.cc:3055] T e33a417cf9c54cf5af73a4ee3aac4d3a P 1c50de89a5a845668696dcb28c3badd6 [term 1 LEADER]: Stepping down as leader of term 1
I20260129 02:59:23.753438 21880 raft_consensus.cc:740] T b189f08e16a74a11980cc55d99359a0a P 197b880fa39d432a9b472845876bfaf0 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 197b880fa39d432a9b472845876bfaf0, State: Running, Role: LEADER
I20260129 02:59:23.753768 21984 raft_consensus.cc:740] T e33a417cf9c54cf5af73a4ee3aac4d3a P 1c50de89a5a845668696dcb28c3badd6 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 1c50de89a5a845668696dcb28c3badd6, State: Running, Role: LEADER
I20260129 02:59:23.754343 21880 consensus_queue.cc:260] T b189f08e16a74a11980cc55d99359a0a P 197b880fa39d432a9b472845876bfaf0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:59:23.754674 21984 consensus_queue.cc:260] T e33a417cf9c54cf5af73a4ee3aac4d3a P 1c50de89a5a845668696dcb28c3badd6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:59:23.755707 21836 consensus_queue.cc:1059] T 87d5fe48a44a408aaf03e894a67ecb2f P 7a9daf5d42a84615a6383cfec9a10608 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260129 02:59:23.755818 21984 raft_consensus.cc:3060] T e33a417cf9c54cf5af73a4ee3aac4d3a P 1c50de89a5a845668696dcb28c3badd6 [term 1 FOLLOWER]: Advancing to term 2
I20260129 02:59:23.756721 21836 raft_consensus.cc:3055] T 87d5fe48a44a408aaf03e894a67ecb2f P 7a9daf5d42a84615a6383cfec9a10608 [term 2 LEADER]: Stepping down as leader of term 2
I20260129 02:59:23.757062 21836 raft_consensus.cc:740] T 87d5fe48a44a408aaf03e894a67ecb2f P 7a9daf5d42a84615a6383cfec9a10608 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 7a9daf5d42a84615a6383cfec9a10608, State: Running, Role: LEADER
I20260129 02:59:23.757822 21836 consensus_queue.cc:260] T 87d5fe48a44a408aaf03e894a67ecb2f P 7a9daf5d42a84615a6383cfec9a10608 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } }
I20260129 02:59:23.758890 21836 raft_consensus.cc:3060] T 87d5fe48a44a408aaf03e894a67ecb2f P 7a9daf5d42a84615a6383cfec9a10608 [term 2 FOLLOWER]: Advancing to term 3
I20260129 02:59:23.751246 21946 consensus_queue.cc:1048] T 4e420f90884f4796997b5ea41e49e411 P 974b802141bd4f13beffef2aa03f88f9 [NON_LEADER]: Connected to new peer: Peer: permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
W20260129 02:59:23.768229 20480 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.211.4:40439, user_credentials={real_user=slave}} blocked reactor thread for 78431.8us
I20260129 02:59:23.770159 20479 leader_election.cc:304] T 368953c9102643b2a21f99af36b51905 P 974b802141bd4f13beffef2aa03f88f9 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 974b802141bd4f13beffef2aa03f88f9; no voters: 197b880fa39d432a9b472845876bfaf0, 1c50de89a5a845668696dcb28c3badd6
W20260129 02:59:23.773130 20553 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.211.1:42027, user_credentials={real_user=slave}} blocked reactor thread for 57993.5us
I20260129 02:59:23.774158 20553 leader_election.cc:304] T 4f8f3a38f6e24ba4857875181f447442 P 1c50de89a5a845668696dcb28c3badd6 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1c50de89a5a845668696dcb28c3badd6; no voters: 197b880fa39d432a9b472845876bfaf0, 974b802141bd4f13beffef2aa03f88f9
I20260129 02:59:23.774515 20405 leader_election.cc:304] T 17d6431cc77a491e8464d68db4081a51 P 197b880fa39d432a9b472845876bfaf0 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 197b880fa39d432a9b472845876bfaf0; no voters: 1c50de89a5a845668696dcb28c3badd6, 974b802141bd4f13beffef2aa03f88f9
I20260129 02:59:23.776777 20405 leader_election.cc:304] T 42d27a28d89b409e8807ec0efd97af69 P 197b880fa39d432a9b472845876bfaf0 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 197b880fa39d432a9b472845876bfaf0, 1c50de89a5a845668696dcb28c3badd6; no voters: 7a9daf5d42a84615a6383cfec9a10608
I20260129 02:59:23.775373 21942 raft_consensus.cc:2749] T 17d6431cc77a491e8464d68db4081a51 P 197b880fa39d432a9b472845876bfaf0 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260129 02:59:23.779915 21885 raft_consensus.cc:2749] T 368953c9102643b2a21f99af36b51905 P 974b802141bd4f13beffef2aa03f88f9 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260129 02:59:23.772296 21787 raft_consensus.cc:2749] T 724a236524cb4e7ba3428880681d6cc2 P 197b880fa39d432a9b472845876bfaf0 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260129 02:59:23.768103 21926 raft_consensus.cc:2749] T febffe4a8b394368b69b1d164308fb9a P 974b802141bd4f13beffef2aa03f88f9 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260129 02:59:23.782536 20405 leader_election.cc:304] T a1a39dcd328e4268ae41b988c66b4a75 P 197b880fa39d432a9b472845876bfaf0 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 197b880fa39d432a9b472845876bfaf0; no voters: 1c50de89a5a845668696dcb28c3badd6, 974b802141bd4f13beffef2aa03f88f9
I20260129 02:59:23.784827 20479 leader_election.cc:304] T b189f08e16a74a11980cc55d99359a0a P 974b802141bd4f13beffef2aa03f88f9 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 974b802141bd4f13beffef2aa03f88f9; no voters: 197b880fa39d432a9b472845876bfaf0, 1c50de89a5a845668696dcb28c3badd6
I20260129 02:59:23.786422 20405 leader_election.cc:304] T e33a417cf9c54cf5af73a4ee3aac4d3a P 197b880fa39d432a9b472845876bfaf0 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 197b880fa39d432a9b472845876bfaf0; no voters: 1c50de89a5a845668696dcb28c3badd6, 974b802141bd4f13beffef2aa03f88f9
I20260129 02:59:23.788429 21937 raft_consensus.cc:2749] T e33a417cf9c54cf5af73a4ee3aac4d3a P 197b880fa39d432a9b472845876bfaf0 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260129 02:59:23.789173 21942 raft_consensus.cc:2749] T a1a39dcd328e4268ae41b988c66b4a75 P 197b880fa39d432a9b472845876bfaf0 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260129 02:59:23.789901 21926 raft_consensus.cc:2749] T b189f08e16a74a11980cc55d99359a0a P 974b802141bd4f13beffef2aa03f88f9 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260129 02:59:23.790302 20479 leader_election.cc:304] T 96609ac68bdf4993a12e1c23355ca013 P 974b802141bd4f13beffef2aa03f88f9 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 974b802141bd4f13beffef2aa03f88f9; no voters: 197b880fa39d432a9b472845876bfaf0, 1c50de89a5a845668696dcb28c3badd6
I20260129 02:59:23.791316 21974 raft_consensus.cc:2749] T 96609ac68bdf4993a12e1c23355ca013 P 974b802141bd4f13beffef2aa03f88f9 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260129 02:59:23.783224 21991 raft_consensus.cc:2749] T 4f8f3a38f6e24ba4857875181f447442 P 1c50de89a5a845668696dcb28c3badd6 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260129 02:59:23.791811 20479 leader_election.cc:304] T cedce331e1584ac8b636c9b8a72b2ba0 P 974b802141bd4f13beffef2aa03f88f9 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 974b802141bd4f13beffef2aa03f88f9; no voters: 197b880fa39d432a9b472845876bfaf0, 1c50de89a5a845668696dcb28c3badd6
I20260129 02:59:23.796753 20479 leader_election.cc:304] T e7d83d4ccf684e76acb29a0dc844e1b8 P 974b802141bd4f13beffef2aa03f88f9 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 974b802141bd4f13beffef2aa03f88f9; no voters: 197b880fa39d432a9b472845876bfaf0, 1c50de89a5a845668696dcb28c3badd6
I20260129 02:59:23.783149 21788 raft_consensus.cc:2804] T 42d27a28d89b409e8807ec0efd97af69 P 197b880fa39d432a9b472845876bfaf0 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260129 02:59:23.797854 21788 raft_consensus.cc:493] T 42d27a28d89b409e8807ec0efd97af69 P 197b880fa39d432a9b472845876bfaf0 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 7a9daf5d42a84615a6383cfec9a10608)
I20260129 02:59:23.798224 21788 raft_consensus.cc:3060] T 42d27a28d89b409e8807ec0efd97af69 P 197b880fa39d432a9b472845876bfaf0 [term 2 FOLLOWER]: Advancing to term 3
I20260129 02:59:23.799690 21994 raft_consensus.cc:3060] T cedce331e1584ac8b636c9b8a72b2ba0 P 974b802141bd4f13beffef2aa03f88f9 [term 2 FOLLOWER]: Advancing to term 3
I20260129 02:59:23.801566 20553 leader_election.cc:304] T 75be4bd6bd45474795fd472eff2953ac P 1c50de89a5a845668696dcb28c3badd6 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 197b880fa39d432a9b472845876bfaf0, 1c50de89a5a845668696dcb28c3badd6; no voters: 974b802141bd4f13beffef2aa03f88f9
I20260129 02:59:23.804517 20405 leader_election.cc:304] T cedce331e1584ac8b636c9b8a72b2ba0 P 197b880fa39d432a9b472845876bfaf0 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 197b880fa39d432a9b472845876bfaf0; no voters: 1c50de89a5a845668696dcb28c3badd6, 974b802141bd4f13beffef2aa03f88f9
I20260129 02:59:23.808830 21937 raft_consensus.cc:2749] T cedce331e1584ac8b636c9b8a72b2ba0 P 197b880fa39d432a9b472845876bfaf0 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260129 02:59:23.810040 20479 leader_election.cc:304] T 87d5fe48a44a408aaf03e894a67ecb2f P 974b802141bd4f13beffef2aa03f88f9 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 974b802141bd4f13beffef2aa03f88f9; no voters: 197b880fa39d432a9b472845876bfaf0, 7a9daf5d42a84615a6383cfec9a10608
I20260129 02:59:23.802305 21974 raft_consensus.cc:2749] T e7d83d4ccf684e76acb29a0dc844e1b8 P 974b802141bd4f13beffef2aa03f88f9 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260129 02:59:23.818230 21913 raft_consensus.cc:3060] T 4e420f90884f4796997b5ea41e49e411 P 974b802141bd4f13beffef2aa03f88f9 [term 2 FOLLOWER]: Advancing to term 3
I20260129 02:59:23.818765 21880 raft_consensus.cc:3060] T b189f08e16a74a11980cc55d99359a0a P 197b880fa39d432a9b472845876bfaf0 [term 2 FOLLOWER]: Advancing to term 3
I20260129 02:59:23.805476 21945 raft_consensus.cc:3060] T 926a250fa5834e88b96ccecbefbf5086 P 1c50de89a5a845668696dcb28c3badd6 [term 2 FOLLOWER]: Advancing to term 3
I20260129 02:59:23.821502 20479 leader_election.cc:304] T e33a417cf9c54cf5af73a4ee3aac4d3a P 974b802141bd4f13beffef2aa03f88f9 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 974b802141bd4f13beffef2aa03f88f9; no voters: 197b880fa39d432a9b472845876bfaf0, 1c50de89a5a845668696dcb28c3badd6
I20260129 02:59:23.821768 21926 raft_consensus.cc:2749] T 87d5fe48a44a408aaf03e894a67ecb2f P 974b802141bd4f13beffef2aa03f88f9 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260129 02:59:23.822427 21926 raft_consensus.cc:3060] T e33a417cf9c54cf5af73a4ee3aac4d3a P 974b802141bd4f13beffef2aa03f88f9 [term 1 FOLLOWER]: Advancing to term 2
I20260129 02:59:23.823796 20479 leader_election.cc:304] T 913fc7d751ed4b7388d889aac30a5f9b P 974b802141bd4f13beffef2aa03f88f9 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 974b802141bd4f13beffef2aa03f88f9; no voters: 197b880fa39d432a9b472845876bfaf0, 1c50de89a5a845668696dcb28c3badd6
I20260129 02:59:23.824983 21885 raft_consensus.cc:2749] T 913fc7d751ed4b7388d889aac30a5f9b P 974b802141bd4f13beffef2aa03f88f9 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260129 02:59:23.826428 20479 leader_election.cc:400] T 4e420f90884f4796997b5ea41e49e411 P 974b802141bd4f13beffef2aa03f88f9 [CANDIDATE]: Term 2 election: Vote denied by peer 197b880fa39d432a9b472845876bfaf0 (127.18.211.1:42027) with higher term. Message: Invalid argument: T 4e420f90884f4796997b5ea41e49e411 P 197b880fa39d432a9b472845876bfaf0 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 974b802141bd4f13beffef2aa03f88f9 for earlier term 2. Current term is 3.
I20260129 02:59:23.805688 21942 consensus_queue.cc:1048] T 1299723a4476420ab02fd8bfd7aa0645 P 197b880fa39d432a9b472845876bfaf0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.022s
I20260129 02:59:23.832370 20338 catalog_manager.cc:5654] T 4e420f90884f4796997b5ea41e49e411 P 7a9daf5d42a84615a6383cfec9a10608 reported cstate change: term changed from 2 to 3, leader changed from 974b802141bd4f13beffef2aa03f88f9 (127.18.211.2) to 7a9daf5d42a84615a6383cfec9a10608 (127.18.211.4). New cstate: current_term: 3 leader_uuid: "7a9daf5d42a84615a6383cfec9a10608" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260129 02:59:23.838680 21994 raft_consensus.cc:2749] T cedce331e1584ac8b636c9b8a72b2ba0 P 974b802141bd4f13beffef2aa03f88f9 [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260129 02:59:23.843684 20479 leader_election.cc:304] T 0678b8b468bd43cdbc3e485900844821 P 974b802141bd4f13beffef2aa03f88f9 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 974b802141bd4f13beffef2aa03f88f9; no voters: 197b880fa39d432a9b472845876bfaf0, 7a9daf5d42a84615a6383cfec9a10608
I20260129 02:59:23.841409 21974 consensus_queue.cc:1048] T e4d18c35b0a7493e85d0a4b92956a48f P 974b802141bd4f13beffef2aa03f88f9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260129 02:59:23.844571 21885 consensus_queue.cc:1048] T 505580c93c694bbb8ea96fdf9b17ab46 P 974b802141bd4f13beffef2aa03f88f9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260129 02:59:23.845240 21988 raft_consensus.cc:2804] T 75be4bd6bd45474795fd472eff2953ac P 1c50de89a5a845668696dcb28c3badd6 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260129 02:59:23.846182 20585 raft_consensus.cc:1275] T c891d2bff4ce4568861bbc992e91fbb0 P 1c50de89a5a845668696dcb28c3badd6 [term 6 FOLLOWER]: Refusing update from remote peer 974b802141bd4f13beffef2aa03f88f9: Log matching property violated. Preceding OpId in replica: term: 5 index: 4. Preceding OpId from leader: term: 6 index: 5. (index mismatch)
I20260129 02:59:23.846446 21944 raft_consensus.cc:3060] T cedce331e1584ac8b636c9b8a72b2ba0 P 1c50de89a5a845668696dcb28c3badd6 [term 2 FOLLOWER]: Advancing to term 3
I20260129 02:59:23.847121 20587 raft_consensus.cc:1275] T 505580c93c694bbb8ea96fdf9b17ab46 P 1c50de89a5a845668696dcb28c3badd6 [term 2 FOLLOWER]: Refusing update from remote peer 974b802141bd4f13beffef2aa03f88f9: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260129 02:59:23.847610 21134 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fd8311b779af4e1fa1402b2a02a602f2" candidate_uuid: "197b880fa39d432a9b472845876bfaf0" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "7a9daf5d42a84615a6383cfec9a10608"
I20260129 02:59:23.848094 21132 raft_consensus.cc:2955] T 3ccac82a01a2445fa5b0ce6c0fcf2295 P 7a9daf5d42a84615a6383cfec9a10608 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 7a9daf5d42a84615a6383cfec9a10608 (127.18.211.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } attrs { replace: true } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } } }
W20260129 02:59:23.850260 20555 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260129 02:59:23.852685 20590 raft_consensus.cc:1240] T 96609ac68bdf4993a12e1c23355ca013 P 1c50de89a5a845668696dcb28c3badd6 [term 3 FOLLOWER]: Rejecting Update request from peer 197b880fa39d432a9b472845876bfaf0 for earlier term 2. Current term is 3. Ops: [2.4-2.4]
I20260129 02:59:23.854130 20588 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fd8311b779af4e1fa1402b2a02a602f2" candidate_uuid: "197b880fa39d432a9b472845876bfaf0" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "1c50de89a5a845668696dcb28c3badd6"
I20260129 02:59:23.857141 20336 catalog_manager.cc:5654] T 3ccac82a01a2445fa5b0ce6c0fcf2295 P 197b880fa39d432a9b472845876bfaf0 reported cstate change: config changed from index 2 to 3, 7a9daf5d42a84615a6383cfec9a10608 (127.18.211.4) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "197b880fa39d432a9b472845876bfaf0" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260129 02:59:23.858332 20335 catalog_manager.cc:5654] T e4d18c35b0a7493e85d0a4b92956a48f P 1c50de89a5a845668696dcb28c3badd6 reported cstate change: term changed from 1 to 2, leader changed from 197b880fa39d432a9b472845876bfaf0 (127.18.211.1) to 974b802141bd4f13beffef2aa03f88f9 (127.18.211.2). New cstate: current_term: 2 leader_uuid: "974b802141bd4f13beffef2aa03f88f9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } } }
I20260129 02:59:23.859411 20405 leader_election.cc:304] T 87d5fe48a44a408aaf03e894a67ecb2f P 197b880fa39d432a9b472845876bfaf0 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 197b880fa39d432a9b472845876bfaf0; no voters: 7a9daf5d42a84615a6383cfec9a10608, 974b802141bd4f13beffef2aa03f88f9
I20260129 02:59:23.783913 21928 raft_consensus.cc:515] T fcddab6bca684adf8ea032289a00f90e P 974b802141bd4f13beffef2aa03f88f9 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:59:23.859915 20581 raft_consensus.cc:2955] T 3ccac82a01a2445fa5b0ce6c0fcf2295 P 1c50de89a5a845668696dcb28c3badd6 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 7a9daf5d42a84615a6383cfec9a10608 (127.18.211.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } attrs { replace: true } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } } }
I20260129 02:59:23.835343 21788 raft_consensus.cc:515] T 42d27a28d89b409e8807ec0efd97af69 P 197b880fa39d432a9b472845876bfaf0 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } }
I20260129 02:59:23.863957 20553 leader_election.cc:304] T a8479d1fe0bd41e5946186e3a6ca8c7a P 1c50de89a5a845668696dcb28c3badd6 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 197b880fa39d432a9b472845876bfaf0, 1c50de89a5a845668696dcb28c3badd6; no voters: 974b802141bd4f13beffef2aa03f88f9
I20260129 02:59:23.881068 21883 consensus_queue.cc:1059] T 1299723a4476420ab02fd8bfd7aa0645 P 197b880fa39d432a9b472845876bfaf0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false }, Status: INVALID_TERM, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20260129 02:59:23.882534 21928 leader_election.cc:290] T fcddab6bca684adf8ea032289a00f90e P 974b802141bd4f13beffef2aa03f88f9 [CANDIDATE]: Term 5 election: Requested vote from peers 197b880fa39d432a9b472845876bfaf0 (127.18.211.1:42027), 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3:37735)
I20260129 02:59:23.883111 21788 leader_election.cc:290] T 42d27a28d89b409e8807ec0efd97af69 P 197b880fa39d432a9b472845876bfaf0 [CANDIDATE]: Term 3 election: Requested vote from peers 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3:37735), 7a9daf5d42a84615a6383cfec9a10608 (127.18.211.4:40439)
I20260129 02:59:23.883344 21928 raft_consensus.cc:3060] T 0678b8b468bd43cdbc3e485900844821 P 974b802141bd4f13beffef2aa03f88f9 [term 2 FOLLOWER]: Advancing to term 3
I20260129 02:59:23.873822 20336 catalog_manager.cc:5654] T 00e79d8f25a34676b1d866eb2068a2b2 P 197b880fa39d432a9b472845876bfaf0 reported cstate change: term changed from 2 to 3, leader changed from 7a9daf5d42a84615a6383cfec9a10608 (127.18.211.4) to 197b880fa39d432a9b472845876bfaf0 (127.18.211.1). New cstate: current_term: 3 leader_uuid: "197b880fa39d432a9b472845876bfaf0" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20260129 02:59:23.878363 21942 raft_consensus.cc:2749] T 87d5fe48a44a408aaf03e894a67ecb2f P 197b880fa39d432a9b472845876bfaf0 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260129 02:59:23.897358 20553 leader_election.cc:304] T 368953c9102643b2a21f99af36b51905 P 1c50de89a5a845668696dcb28c3badd6 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1c50de89a5a845668696dcb28c3badd6; no voters: 197b880fa39d432a9b472845876bfaf0, 974b802141bd4f13beffef2aa03f88f9
I20260129 02:59:23.898630 21134 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "42d27a28d89b409e8807ec0efd97af69" candidate_uuid: "197b880fa39d432a9b472845876bfaf0" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "7a9daf5d42a84615a6383cfec9a10608"
I20260129 02:59:23.863390 21988 raft_consensus.cc:493] T 75be4bd6bd45474795fd472eff2953ac P 1c50de89a5a845668696dcb28c3badd6 [term 4 FOLLOWER]: Starting leader election (detected failure of leader 197b880fa39d432a9b472845876bfaf0)
I20260129 02:59:23.899436 20588 raft_consensus.cc:2468] T fd8311b779af4e1fa1402b2a02a602f2 P 1c50de89a5a845668696dcb28c3badd6 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 197b880fa39d432a9b472845876bfaf0 in term 2.
I20260129 02:59:23.899269 20584 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "42d27a28d89b409e8807ec0efd97af69" candidate_uuid: "197b880fa39d432a9b472845876bfaf0" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "1c50de89a5a845668696dcb28c3badd6"
I20260129 02:59:23.902248 20553 leader_election.cc:304] T 7a9a69cd82724b1faed1d10aa091fc4a P 1c50de89a5a845668696dcb28c3badd6 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1c50de89a5a845668696dcb28c3badd6; no voters: 197b880fa39d432a9b472845876bfaf0, 974b802141bd4f13beffef2aa03f88f9
I20260129 02:59:23.905150 21828 raft_consensus.cc:2749] T 368953c9102643b2a21f99af36b51905 P 1c50de89a5a845668696dcb28c3badd6 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260129 02:59:23.916427 20553 leader_election.cc:304] T 913fc7d751ed4b7388d889aac30a5f9b P 1c50de89a5a845668696dcb28c3badd6 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1c50de89a5a845668696dcb28c3badd6; no voters: 197b880fa39d432a9b472845876bfaf0, 974b802141bd4f13beffef2aa03f88f9
W20260129 02:59:23.915259 20480 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.211.3:37735, user_credentials={real_user=slave}} blocked reactor thread for 49136.6us
I20260129 02:59:23.915968 21988 raft_consensus.cc:3060] T 75be4bd6bd45474795fd472eff2953ac P 1c50de89a5a845668696dcb28c3badd6 [term 4 FOLLOWER]: Advancing to term 5
I20260129 02:59:23.921871 21937 consensus_queue.cc:1059] T 96609ac68bdf4993a12e1c23355ca013 P 197b880fa39d432a9b472845876bfaf0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 }, Status: INVALID_TERM, Last received: 2.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260129 02:59:23.896677 21995 raft_consensus.cc:2764] T a8479d1fe0bd41e5946186e3a6ca8c7a P 1c50de89a5a845668696dcb28c3badd6 [term 2 FOLLOWER]: Leader pre-election decision vote started in defunct term 1: won
I20260129 02:59:23.925526 21984 raft_consensus.cc:2749] T 913fc7d751ed4b7388d889aac30a5f9b P 1c50de89a5a845668696dcb28c3badd6 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260129 02:59:23.926154 21883 raft_consensus.cc:3055] T 1299723a4476420ab02fd8bfd7aa0645 P 197b880fa39d432a9b472845876bfaf0 [term 2 LEADER]: Stepping down as leader of term 2
I20260129 02:59:23.926551 21883 raft_consensus.cc:740] T 1299723a4476420ab02fd8bfd7aa0645 P 197b880fa39d432a9b472845876bfaf0 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 197b880fa39d432a9b472845876bfaf0, State: Running, Role: LEADER
I20260129 02:59:23.927326 21883 consensus_queue.cc:260] T 1299723a4476420ab02fd8bfd7aa0645 P 197b880fa39d432a9b472845876bfaf0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 2.5, Last appended by leader: 5, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } }
I20260129 02:59:23.928508 21883 raft_consensus.cc:3060] T 1299723a4476420ab02fd8bfd7aa0645 P 197b880fa39d432a9b472845876bfaf0 [term 2 FOLLOWER]: Advancing to term 3
I20260129 02:59:23.928750 21946 consensus_queue.cc:1048] T c891d2bff4ce4568861bbc992e91fbb0 P 974b802141bd4f13beffef2aa03f88f9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260129 02:59:23.930907 21946 consensus_queue.cc:1048] T 505580c93c694bbb8ea96fdf9b17ab46 P 974b802141bd4f13beffef2aa03f88f9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260129 02:59:23.934798 20584 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fcddab6bca684adf8ea032289a00f90e" candidate_uuid: "974b802141bd4f13beffef2aa03f88f9" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "1c50de89a5a845668696dcb28c3badd6"
I20260129 02:59:23.938563 20553 leader_election.cc:304] T 3906b080a5b34a74b8b64128635e8997 P 1c50de89a5a845668696dcb28c3badd6 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1c50de89a5a845668696dcb28c3badd6; no voters: 197b880fa39d432a9b472845876bfaf0, 974b802141bd4f13beffef2aa03f88f9
I20260129 02:59:23.922766 21945 raft_consensus.cc:2749] T 7a9a69cd82724b1faed1d10aa091fc4a P 1c50de89a5a845668696dcb28c3badd6 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260129 02:59:23.949354 20553 leader_election.cc:304] T a4c1226b987b468fa84dfc6ba0e2dbe1 P 1c50de89a5a845668696dcb28c3badd6 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1c50de89a5a845668696dcb28c3badd6; no voters: 197b880fa39d432a9b472845876bfaf0, 974b802141bd4f13beffef2aa03f88f9
I20260129 02:59:23.954229 21828 raft_consensus.cc:2749] T 3906b080a5b34a74b8b64128635e8997 P 1c50de89a5a845668696dcb28c3badd6 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260129 02:59:23.959956 21828 raft_consensus.cc:2749] T a4c1226b987b468fa84dfc6ba0e2dbe1 P 1c50de89a5a845668696dcb28c3badd6 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20260129 02:59:23.976938 20405 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.211.3:37735, user_credentials={real_user=slave}} blocked reactor thread for 59677.5us
I20260129 02:59:23.982151 20405 leader_election.cc:304] T fd8311b779af4e1fa1402b2a02a602f2 P 197b880fa39d432a9b472845876bfaf0 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 197b880fa39d432a9b472845876bfaf0, 1c50de89a5a845668696dcb28c3badd6; no voters: 7a9daf5d42a84615a6383cfec9a10608
I20260129 02:59:23.984758 21787 raft_consensus.cc:2804] T fd8311b779af4e1fa1402b2a02a602f2 P 197b880fa39d432a9b472845876bfaf0 [term 2 FOLLOWER]: Leader election won for term 2
I20260129 02:59:23.985189 21926 raft_consensus.cc:2749] T e33a417cf9c54cf5af73a4ee3aac4d3a P 974b802141bd4f13beffef2aa03f88f9 [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260129 02:59:23.985491 21787 raft_consensus.cc:697] T fd8311b779af4e1fa1402b2a02a602f2 P 197b880fa39d432a9b472845876bfaf0 [term 2 LEADER]: Becoming Leader. State: Replica: 197b880fa39d432a9b472845876bfaf0, State: Running, Role: LEADER
I20260129 02:59:23.986740 21787 consensus_queue.cc:237] T fd8311b779af4e1fa1402b2a02a602f2 P 197b880fa39d432a9b472845876bfaf0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } }
I20260129 02:59:24.069809 21928 raft_consensus.cc:2749] T 0678b8b468bd43cdbc3e485900844821 P 974b802141bd4f13beffef2aa03f88f9 [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260129 02:59:24.076915 21788 raft_consensus.cc:3055] T 96609ac68bdf4993a12e1c23355ca013 P 197b880fa39d432a9b472845876bfaf0 [term 2 LEADER]: Stepping down as leader of term 2
I20260129 02:59:24.077473 21788 raft_consensus.cc:740] T 96609ac68bdf4993a12e1c23355ca013 P 197b880fa39d432a9b472845876bfaf0 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 197b880fa39d432a9b472845876bfaf0, State: Running, Role: LEADER
I20260129 02:59:24.078433 21788 consensus_queue.cc:260] T 96609ac68bdf4993a12e1c23355ca013 P 197b880fa39d432a9b472845876bfaf0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 2.4, Last appended by leader: 4, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } attrs { replace: true } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } }
I20260129 02:59:24.081000 21788 raft_consensus.cc:3060] T 96609ac68bdf4993a12e1c23355ca013 P 197b880fa39d432a9b472845876bfaf0 [term 2 FOLLOWER]: Advancing to term 3
I20260129 02:59:24.090735 20439 consensus_queue.cc:237] T 3ccac82a01a2445fa5b0ce6c0fcf2295 P 197b880fa39d432a9b472845876bfaf0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } }
I20260129 02:59:24.097599 21988 raft_consensus.cc:515] T 75be4bd6bd45474795fd472eff2953ac P 1c50de89a5a845668696dcb28c3badd6 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:59:24.134191 20440 raft_consensus.cc:1240] T 42d27a28d89b409e8807ec0efd97af69 P 197b880fa39d432a9b472845876bfaf0 [term 3 FOLLOWER]: Rejecting Update request from peer 7a9daf5d42a84615a6383cfec9a10608 for earlier term 2. Current term is 3. Ops: []
W20260129 02:59:24.136829 20479 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.211.1:42027, user_credentials={real_user=slave}} blocked reactor thread for 293185us
I20260129 02:59:24.139335 20479 leader_election.cc:304] T 3906b080a5b34a74b8b64128635e8997 P 974b802141bd4f13beffef2aa03f88f9 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 974b802141bd4f13beffef2aa03f88f9; no voters: 197b880fa39d432a9b472845876bfaf0, 1c50de89a5a845668696dcb28c3badd6
W20260129 02:59:24.147634 20405 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.211.3:37735, user_credentials={real_user=slave}} blocked reactor thread for 165604us
I20260129 02:59:24.147361 21975 raft_consensus.cc:2749] T 3906b080a5b34a74b8b64128635e8997 P 974b802141bd4f13beffef2aa03f88f9 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260129 02:59:24.151600 21134 raft_consensus.cc:1275] T 00e79d8f25a34676b1d866eb2068a2b2 P 7a9daf5d42a84615a6383cfec9a10608 [term 3 FOLLOWER]: Refusing update from remote peer 197b880fa39d432a9b472845876bfaf0: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 3 index: 6. (index mismatch)
I20260129 02:59:24.142728 20512 raft_consensus.cc:1275] T 3ccac82a01a2445fa5b0ce6c0fcf2295 P 974b802141bd4f13beffef2aa03f88f9 [term 1 FOLLOWER]: Refusing update from remote peer 197b880fa39d432a9b472845876bfaf0: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260129 02:59:24.153872 21988 leader_election.cc:290] T 75be4bd6bd45474795fd472eff2953ac P 1c50de89a5a845668696dcb28c3badd6 [CANDIDATE]: Term 5 election: Requested vote from peers 197b880fa39d432a9b472845876bfaf0 (127.18.211.1:42027), 974b802141bd4f13beffef2aa03f88f9 (127.18.211.2:34517)
I20260129 02:59:24.154048 21127 raft_consensus.cc:1275] T 3ccac82a01a2445fa5b0ce6c0fcf2295 P 7a9daf5d42a84615a6383cfec9a10608 [term 1 FOLLOWER]: Refusing update from remote peer 197b880fa39d432a9b472845876bfaf0: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260129 02:59:24.156463 20582 raft_consensus.cc:1240] T 368953c9102643b2a21f99af36b51905 P 1c50de89a5a845668696dcb28c3badd6 [term 3 FOLLOWER]: Rejecting Update request from peer 197b880fa39d432a9b472845876bfaf0 for earlier term 2. Current term is 3. Ops: []
I20260129 02:59:24.156278 21942 consensus_queue.cc:1048] T 3ccac82a01a2445fa5b0ce6c0fcf2295 P 197b880fa39d432a9b472845876bfaf0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260129 02:59:24.156173 20405 leader_election.cc:304] T 42d27a28d89b409e8807ec0efd97af69 P 197b880fa39d432a9b472845876bfaf0 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 197b880fa39d432a9b472845876bfaf0; no voters: 1c50de89a5a845668696dcb28c3badd6, 7a9daf5d42a84615a6383cfec9a10608
I20260129 02:59:24.160104 20479 leader_election.cc:304] T a5a05fbcbbcf4916bbc01fd1a5b5e90d P 974b802141bd4f13beffef2aa03f88f9 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 974b802141bd4f13beffef2aa03f88f9; no voters: 197b880fa39d432a9b472845876bfaf0, 1c50de89a5a845668696dcb28c3badd6
I20260129 02:59:24.162186 20511 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "75be4bd6bd45474795fd472eff2953ac" candidate_uuid: "1c50de89a5a845668696dcb28c3badd6" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "974b802141bd4f13beffef2aa03f88f9"
I20260129 02:59:24.141855 21894 consensus_queue.cc:1059] T 42d27a28d89b409e8807ec0efd97af69 P 7a9daf5d42a84615a6383cfec9a10608 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.010s
I20260129 02:59:24.164403 21894 raft_consensus.cc:3055] T 42d27a28d89b409e8807ec0efd97af69 P 7a9daf5d42a84615a6383cfec9a10608 [term 2 LEADER]: Stepping down as leader of term 2
I20260129 02:59:24.164788 21894 raft_consensus.cc:740] T 42d27a28d89b409e8807ec0efd97af69 P 7a9daf5d42a84615a6383cfec9a10608 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 7a9daf5d42a84615a6383cfec9a10608, State: Running, Role: LEADER
I20260129 02:59:24.165676 21894 consensus_queue.cc:260] T 42d27a28d89b409e8807ec0efd97af69 P 7a9daf5d42a84615a6383cfec9a10608 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } }
I20260129 02:59:24.166917 20511 raft_consensus.cc:2393] T 75be4bd6bd45474795fd472eff2953ac P 974b802141bd4f13beffef2aa03f88f9 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 1c50de89a5a845668696dcb28c3badd6 in current term 5: Already voted for candidate 974b802141bd4f13beffef2aa03f88f9 in this term.
I20260129 02:59:24.154621 20336 catalog_manager.cc:5654] T c891d2bff4ce4568861bbc992e91fbb0 P 974b802141bd4f13beffef2aa03f88f9 reported cstate change: term changed from 5 to 6, leader changed from 197b880fa39d432a9b472845876bfaf0 (127.18.211.1) to 974b802141bd4f13beffef2aa03f88f9 (127.18.211.2). New cstate: current_term: 6 leader_uuid: "974b802141bd4f13beffef2aa03f88f9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } health_report { overall_health: UNKNOWN } } }
W20260129 02:59:24.163245 20553 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.211.1:42027, user_credentials={real_user=slave}} blocked reactor thread for 214693us
I20260129 02:59:24.190147 20336 catalog_manager.cc:5654] T 505580c93c694bbb8ea96fdf9b17ab46 P 974b802141bd4f13beffef2aa03f88f9 reported cstate change: term changed from 1 to 2, leader changed from 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3) to 974b802141bd4f13beffef2aa03f88f9 (127.18.211.2). New cstate: current_term: 2 leader_uuid: "974b802141bd4f13beffef2aa03f88f9" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20260129 02:59:24.191275 20553 leader_election.cc:304] T 4e3805ffa70242e8ade66da260dddf66 P 1c50de89a5a845668696dcb28c3badd6 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1c50de89a5a845668696dcb28c3badd6; no voters: 197b880fa39d432a9b472845876bfaf0, 7a9daf5d42a84615a6383cfec9a10608
I20260129 02:59:24.192343 21975 raft_consensus.cc:3060] T a5a05fbcbbcf4916bbc01fd1a5b5e90d P 974b802141bd4f13beffef2aa03f88f9 [term 2 FOLLOWER]: Advancing to term 3
I20260129 02:59:24.192107 20336 catalog_manager.cc:5654] T a8479d1fe0bd41e5946186e3a6ca8c7a P 974b802141bd4f13beffef2aa03f88f9 reported cstate change: term changed from 1 to 2, leader changed from 197b880fa39d432a9b472845876bfaf0 (127.18.211.1) to 974b802141bd4f13beffef2aa03f88f9 (127.18.211.2). New cstate: current_term: 2 leader_uuid: "974b802141bd4f13beffef2aa03f88f9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } health_report { overall_health: UNKNOWN } } }
I20260129 02:59:24.194566 21786 consensus_queue.cc:1048] T 3ccac82a01a2445fa5b0ce6c0fcf2295 P 197b880fa39d432a9b472845876bfaf0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260129 02:59:24.194628 21937 consensus_queue.cc:1048] T 00e79d8f25a34676b1d866eb2068a2b2 P 197b880fa39d432a9b472845876bfaf0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
I20260129 02:59:24.205050 20553 leader_election.cc:304] T 0f20af7fbe5d4db89a6f17948d98912c P 1c50de89a5a845668696dcb28c3badd6 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1c50de89a5a845668696dcb28c3badd6; no voters: 197b880fa39d432a9b472845876bfaf0, 974b802141bd4f13beffef2aa03f88f9
I20260129 02:59:24.207360 21942 consensus_queue.cc:1059] T 368953c9102643b2a21f99af36b51905 P 197b880fa39d432a9b472845876bfaf0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 }, Status: INVALID_TERM, Last received: 2.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260129 02:59:24.203672 22004 raft_consensus.cc:2749] T 4e3805ffa70242e8ade66da260dddf66 P 1c50de89a5a845668696dcb28c3badd6 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260129 02:59:24.197481 21787 raft_consensus.cc:2749] T 42d27a28d89b409e8807ec0efd97af69 P 197b880fa39d432a9b472845876bfaf0 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260129 02:59:24.217901 21942 raft_consensus.cc:3055] T 368953c9102643b2a21f99af36b51905 P 197b880fa39d432a9b472845876bfaf0 [term 2 LEADER]: Stepping down as leader of term 2
I20260129 02:59:24.220806 20553 leader_election.cc:304] T 76f713c3331540d2a5b76164103e9eb1 P 1c50de89a5a845668696dcb28c3badd6 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1c50de89a5a845668696dcb28c3badd6; no voters: 197b880fa39d432a9b472845876bfaf0, 974b802141bd4f13beffef2aa03f88f9
W20260129 02:59:24.228415 20479 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.211.1:42027, user_credentials={real_user=slave}} blocked reactor thread for 68392.3us
I20260129 02:59:24.229144 20479 leader_election.cc:304] T a1a39dcd328e4268ae41b988c66b4a75 P 974b802141bd4f13beffef2aa03f88f9 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 974b802141bd4f13beffef2aa03f88f9; no voters: 197b880fa39d432a9b472845876bfaf0, 1c50de89a5a845668696dcb28c3badd6
I20260129 02:59:24.224571 22004 raft_consensus.cc:2749] T 0f20af7fbe5d4db89a6f17948d98912c P 1c50de89a5a845668696dcb28c3badd6 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260129 02:59:24.238530 20511 raft_consensus.cc:1275] T 00e79d8f25a34676b1d866eb2068a2b2 P 974b802141bd4f13beffef2aa03f88f9 [term 3 FOLLOWER]: Refusing update from remote peer 197b880fa39d432a9b472845876bfaf0: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 3 index: 6. (index mismatch)
I20260129 02:59:24.239630 21945 raft_consensus.cc:2749] T 76f713c3331540d2a5b76164103e9eb1 P 1c50de89a5a845668696dcb28c3badd6 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260129 02:59:24.219928 21894 raft_consensus.cc:3060] T 42d27a28d89b409e8807ec0efd97af69 P 7a9daf5d42a84615a6383cfec9a10608 [term 2 FOLLOWER]: Advancing to term 3
I20260129 02:59:24.221091 21942 raft_consensus.cc:740] T 368953c9102643b2a21f99af36b51905 P 197b880fa39d432a9b472845876bfaf0 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 197b880fa39d432a9b472845876bfaf0, State: Running, Role: LEADER
I20260129 02:59:24.246137 20553 leader_election.cc:304] T a0fcc58d6fb6473cafbd9d34d5195269 P 1c50de89a5a845668696dcb28c3badd6 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1c50de89a5a845668696dcb28c3badd6; no voters: 197b880fa39d432a9b472845876bfaf0, 974b802141bd4f13beffef2aa03f88f9
I20260129 02:59:24.255900 20584 raft_consensus.cc:1275] T a8479d1fe0bd41e5946186e3a6ca8c7a P 1c50de89a5a845668696dcb28c3badd6 [term 2 FOLLOWER]: Refusing update from remote peer 974b802141bd4f13beffef2aa03f88f9: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260129 02:59:24.257400 21787 raft_consensus.cc:2955] T 3ccac82a01a2445fa5b0ce6c0fcf2295 P 197b880fa39d432a9b472845876bfaf0 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } } }
I20260129 02:59:24.259918 21913 consensus_queue.cc:1048] T a8479d1fe0bd41e5946186e3a6ca8c7a P 974b802141bd4f13beffef2aa03f88f9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260129 02:59:24.260327 20438 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fcddab6bca684adf8ea032289a00f90e" candidate_uuid: "974b802141bd4f13beffef2aa03f88f9" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "197b880fa39d432a9b472845876bfaf0"
I20260129 02:59:24.262285 20433 raft_consensus.cc:1275] T a8479d1fe0bd41e5946186e3a6ca8c7a P 197b880fa39d432a9b472845876bfaf0 [term 2 FOLLOWER]: Refusing update from remote peer 974b802141bd4f13beffef2aa03f88f9: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260129 02:59:24.243827 21926 raft_consensus.cc:2749] T a1a39dcd328e4268ae41b988c66b4a75 P 974b802141bd4f13beffef2aa03f88f9 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260129 02:59:24.260569 20432 raft_consensus.cc:1275] T c891d2bff4ce4568861bbc992e91fbb0 P 197b880fa39d432a9b472845876bfaf0 [term 6 FOLLOWER]: Refusing update from remote peer 974b802141bd4f13beffef2aa03f88f9: Log matching property violated. Preceding OpId in replica: term: 5 index: 4. Preceding OpId from leader: term: 6 index: 5. (index mismatch)
I20260129 02:59:24.268805 21942 consensus_queue.cc:260] T 368953c9102643b2a21f99af36b51905 P 197b880fa39d432a9b472845876bfaf0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:59:24.270251 21942 raft_consensus.cc:3060] T 368953c9102643b2a21f99af36b51905 P 197b880fa39d432a9b472845876bfaf0 [term 2 FOLLOWER]: Advancing to term 3
I20260129 02:59:24.269143 20432 raft_consensus.cc:1275] T 4e420f90884f4796997b5ea41e49e411 P 197b880fa39d432a9b472845876bfaf0 [term 3 FOLLOWER]: Refusing update from remote peer 7a9daf5d42a84615a6383cfec9a10608: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 3 index: 5. (index mismatch)
I20260129 02:59:24.251400 21786 consensus_queue.cc:1048] T 00e79d8f25a34676b1d866eb2068a2b2 P 197b880fa39d432a9b472845876bfaf0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.028s
I20260129 02:59:24.296936 20508 raft_consensus.cc:1275] T 4e420f90884f4796997b5ea41e49e411 P 974b802141bd4f13beffef2aa03f88f9 [term 3 FOLLOWER]: Refusing update from remote peer 7a9daf5d42a84615a6383cfec9a10608: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 3 index: 5. (term mismatch)
I20260129 02:59:24.297348 20508 pending_rounds.cc:85] T 4e420f90884f4796997b5ea41e49e411 P 974b802141bd4f13beffef2aa03f88f9: Aborting all ops after (but not including) 4
I20260129 02:59:24.297596 20508 pending_rounds.cc:107] T 4e420f90884f4796997b5ea41e49e411 P 974b802141bd4f13beffef2aa03f88f9: Aborting uncommitted NO_OP operation due to leader change: 2.5
I20260129 02:59:24.297824 20508 raft_consensus.cc:2889] T 4e420f90884f4796997b5ea41e49e411 P 974b802141bd4f13beffef2aa03f88f9 [term 3 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260129 02:59:24.296944 21129 raft_consensus.cc:2955] T 3ccac82a01a2445fa5b0ce6c0fcf2295 P 7a9daf5d42a84615a6383cfec9a10608 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } } }
I20260129 02:59:24.301765 21975 raft_consensus.cc:2749] T a5a05fbcbbcf4916bbc01fd1a5b5e90d P 974b802141bd4f13beffef2aa03f88f9 [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260129 02:59:24.305780 21836 consensus_queue.cc:1048] T 4e420f90884f4796997b5ea41e49e411 P 7a9daf5d42a84615a6383cfec9a10608 [LEADER]: Connected to new peer: Peer: permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20260129 02:59:24.308462 20511 raft_consensus.cc:2955] T 3ccac82a01a2445fa5b0ce6c0fcf2295 P 974b802141bd4f13beffef2aa03f88f9 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } } }
W20260129 02:59:24.324249 22014 log.cc:927] Time spent T 505580c93c694bbb8ea96fdf9b17ab46 P 7a9daf5d42a84615a6383cfec9a10608: Append to log took a long time: real 0.097s user 0.001s sys 0.000s
W20260129 02:59:24.343478 20553 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.211.1:42027, user_credentials={real_user=slave}} blocked reactor thread for 97415.3us
I20260129 02:59:24.344796 20553 leader_election.cc:304] T 4408006eb5174de1896b037cf3f18457 P 1c50de89a5a845668696dcb28c3badd6 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1c50de89a5a845668696dcb28c3badd6; no voters: 197b880fa39d432a9b472845876bfaf0, 974b802141bd4f13beffef2aa03f88f9
I20260129 02:59:24.342445 20479 leader_election.cc:304] T fcddab6bca684adf8ea032289a00f90e P 974b802141bd4f13beffef2aa03f88f9 [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 974b802141bd4f13beffef2aa03f88f9; no voters: 197b880fa39d432a9b472845876bfaf0, 1c50de89a5a845668696dcb28c3badd6
W20260129 02:59:24.348822 20556 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.211.2:34517, user_credentials={real_user=slave}} blocked reactor thread for 76672us
W20260129 02:59:24.350682 20554 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.211.4:40439, user_credentials={real_user=slave}} blocked reactor thread for 56508.9us
I20260129 02:59:24.345660 21988 raft_consensus.cc:2749] T a0fcc58d6fb6473cafbd9d34d5195269 P 1c50de89a5a845668696dcb28c3badd6 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260129 02:59:24.362668 21946 raft_consensus.cc:2749] T fcddab6bca684adf8ea032289a00f90e P 974b802141bd4f13beffef2aa03f88f9 [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20260129 02:59:24.374845 20336 catalog_manager.cc:5654] T fd8311b779af4e1fa1402b2a02a602f2 P 197b880fa39d432a9b472845876bfaf0 reported cstate change: term changed from 1 to 2, leader changed from 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3) to 197b880fa39d432a9b472845876bfaf0 (127.18.211.1). New cstate: current_term: 2 leader_uuid: "197b880fa39d432a9b472845876bfaf0" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20260129 02:59:24.373909 21945 raft_consensus.cc:2749] T 4408006eb5174de1896b037cf3f18457 P 1c50de89a5a845668696dcb28c3badd6 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260129 02:59:24.390133 20553 leader_election.cc:304] T c891d2bff4ce4568861bbc992e91fbb0 P 1c50de89a5a845668696dcb28c3badd6 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 197b880fa39d432a9b472845876bfaf0, 1c50de89a5a845668696dcb28c3badd6; no voters: 974b802141bd4f13beffef2aa03f88f9
I20260129 02:59:24.412467 20322 catalog_manager.cc:5167] ChangeConfig:REMOVE_PEER RPC for tablet 3ccac82a01a2445fa5b0ce6c0fcf2295 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260129 02:59:24.371387 22002 consensus_queue.cc:1048] T a8479d1fe0bd41e5946186e3a6ca8c7a P 974b802141bd4f13beffef2aa03f88f9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.041s
W20260129 02:59:24.414327 22012 log.cc:927] Time spent T 505580c93c694bbb8ea96fdf9b17ab46 P 1c50de89a5a845668696dcb28c3badd6: Append to log took a long time: real 0.211s user 0.000s sys 0.001s
I20260129 02:59:24.385737 21975 consensus_queue.cc:1048] T c891d2bff4ce4568861bbc992e91fbb0 P 974b802141bd4f13beffef2aa03f88f9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.031s
I20260129 02:59:24.431490 20553 leader_election.cc:304] T e4d18c35b0a7493e85d0a4b92956a48f P 1c50de89a5a845668696dcb28c3badd6 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 197b880fa39d432a9b472845876bfaf0, 1c50de89a5a845668696dcb28c3badd6; no voters: 974b802141bd4f13beffef2aa03f88f9
W20260129 02:59:24.406072 21099 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.211.1:42027, user_credentials={real_user=slave}} blocked reactor thread for 131498us
I20260129 02:59:24.416103 21984 raft_consensus.cc:2764] T c891d2bff4ce4568861bbc992e91fbb0 P 1c50de89a5a845668696dcb28c3badd6 [term 6 FOLLOWER]: Leader pre-election decision vote started in defunct term 5: won
I20260129 02:59:24.436179 20553 leader_election.cc:304] T 96609ac68bdf4993a12e1c23355ca013 P 1c50de89a5a845668696dcb28c3badd6 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1c50de89a5a845668696dcb28c3badd6; no voters: 197b880fa39d432a9b472845876bfaf0, 974b802141bd4f13beffef2aa03f88f9
I20260129 02:59:24.411931 21988 raft_consensus.cc:493] T 1ce4ddf5b10e4ae8b193b13b2e5936ee P 1c50de89a5a845668696dcb28c3badd6 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 197b880fa39d432a9b472845876bfaf0)
I20260129 02:59:24.464556 21836 consensus_queue.cc:1048] T 4e420f90884f4796997b5ea41e49e411 P 7a9daf5d42a84615a6383cfec9a10608 [LEADER]: Connected to new peer: Peer: permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20260129 02:59:24.466054 21988 raft_consensus.cc:515] T 1ce4ddf5b10e4ae8b193b13b2e5936ee P 1c50de89a5a845668696dcb28c3badd6 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:59:24.468436 21988 leader_election.cc:290] T 1ce4ddf5b10e4ae8b193b13b2e5936ee P 1c50de89a5a845668696dcb28c3badd6 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 197b880fa39d432a9b472845876bfaf0 (127.18.211.1:42027), 974b802141bd4f13beffef2aa03f88f9 (127.18.211.2:34517)
I20260129 02:59:24.473968 20553 leader_election.cc:304] T 4b1876f9a70b49edb3aa74105e66d3b3 P 1c50de89a5a845668696dcb28c3badd6 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1c50de89a5a845668696dcb28c3badd6; no voters: 197b880fa39d432a9b472845876bfaf0, 974b802141bd4f13beffef2aa03f88f9
I20260129 02:59:24.436393 21945 raft_consensus.cc:2764] T e4d18c35b0a7493e85d0a4b92956a48f P 1c50de89a5a845668696dcb28c3badd6 [term 2 FOLLOWER]: Leader pre-election decision vote started in defunct term 1: won
I20260129 02:59:24.477630 21828 raft_consensus.cc:2749] T 96609ac68bdf4993a12e1c23355ca013 P 1c50de89a5a845668696dcb28c3badd6 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260129 02:59:24.487190 20512 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1ce4ddf5b10e4ae8b193b13b2e5936ee" candidate_uuid: "1c50de89a5a845668696dcb28c3badd6" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "974b802141bd4f13beffef2aa03f88f9" is_pre_election: true
I20260129 02:59:24.488024 20512 raft_consensus.cc:2393] T 1ce4ddf5b10e4ae8b193b13b2e5936ee P 974b802141bd4f13beffef2aa03f88f9 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1c50de89a5a845668696dcb28c3badd6 in current term 2: Already voted for candidate 974b802141bd4f13beffef2aa03f88f9 in this term.
I20260129 02:59:24.488500 20553 leader_election.cc:304] T 0340229d521b432580ae926f6062e3f7 P 1c50de89a5a845668696dcb28c3badd6 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1c50de89a5a845668696dcb28c3badd6; no voters: 197b880fa39d432a9b472845876bfaf0, 974b802141bd4f13beffef2aa03f88f9
I20260129 02:59:24.497100 21828 raft_consensus.cc:2749] T 0340229d521b432580ae926f6062e3f7 P 1c50de89a5a845668696dcb28c3badd6 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260129 02:59:24.476262 19276 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260129 02:59:24.490888 21995 raft_consensus.cc:2749] T 4b1876f9a70b49edb3aa74105e66d3b3 P 1c50de89a5a845668696dcb28c3badd6 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20260129 02:59:24.503788 22021 log.cc:927] Time spent T a8479d1fe0bd41e5946186e3a6ca8c7a P 1c50de89a5a845668696dcb28c3badd6: Append to log took a long time: real 0.076s user 0.000s sys 0.000s
I20260129 02:59:24.511637 20553 leader_election.cc:304] T 7d9494f5584d4d5db8b9adf1c8fc888b P 1c50de89a5a845668696dcb28c3badd6 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1c50de89a5a845668696dcb28c3badd6; no voters: 197b880fa39d432a9b472845876bfaf0, 974b802141bd4f13beffef2aa03f88f9
I20260129 02:59:24.514900 20336 catalog_manager.cc:5654] T 3ccac82a01a2445fa5b0ce6c0fcf2295 P 197b880fa39d432a9b472845876bfaf0 reported cstate change: config changed from index 3 to 4, VOTER 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3) evicted. New cstate: current_term: 1 leader_uuid: "197b880fa39d432a9b472845876bfaf0" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260129 02:59:24.530323 20553 leader_election.cc:304] T 724a236524cb4e7ba3428880681d6cc2 P 1c50de89a5a845668696dcb28c3badd6 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1c50de89a5a845668696dcb28c3badd6; no voters: 197b880fa39d432a9b472845876bfaf0, 974b802141bd4f13beffef2aa03f88f9
I20260129 02:59:24.529599 21995 raft_consensus.cc:2749] T 7d9494f5584d4d5db8b9adf1c8fc888b P 1c50de89a5a845668696dcb28c3badd6 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20260129 02:59:24.542969 20479 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.211.1:42027, user_credentials={real_user=slave}} blocked reactor thread for 153587us
I20260129 02:59:24.536887 21945 raft_consensus.cc:2749] T 724a236524cb4e7ba3428880681d6cc2 P 1c50de89a5a845668696dcb28c3badd6 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260129 02:59:24.550045 20438 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "75be4bd6bd45474795fd472eff2953ac" candidate_uuid: "1c50de89a5a845668696dcb28c3badd6" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "197b880fa39d432a9b472845876bfaf0"
I20260129 02:59:24.556123 20435 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1ce4ddf5b10e4ae8b193b13b2e5936ee" candidate_uuid: "1c50de89a5a845668696dcb28c3badd6" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "197b880fa39d432a9b472845876bfaf0" is_pre_election: true
I20260129 02:59:24.556779 20435 raft_consensus.cc:2468] T 1ce4ddf5b10e4ae8b193b13b2e5936ee P 197b880fa39d432a9b472845876bfaf0 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1c50de89a5a845668696dcb28c3badd6 in term 2.
I20260129 02:59:24.563606 20438 raft_consensus.cc:2468] T 75be4bd6bd45474795fd472eff2953ac P 197b880fa39d432a9b472845876bfaf0 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1c50de89a5a845668696dcb28c3badd6 in term 5.
I20260129 02:59:24.589560 20579 tablet_service.cc:1552] Processing DeleteTablet for tablet 3ccac82a01a2445fa5b0ce6c0fcf2295 with delete_type TABLET_DATA_TOMBSTONED (TS 1c50de89a5a845668696dcb28c3badd6 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:35744
I20260129 02:59:24.598146 21132 raft_consensus.cc:1240] T dc2f4cb752cf4d59b5f6bc689f5d8104 P 7a9daf5d42a84615a6383cfec9a10608 [term 2 FOLLOWER]: Rejecting Update request from peer 1c50de89a5a845668696dcb28c3badd6 for earlier term 1. Current term is 2. Ops: []
I20260129 02:59:24.654451 20583 raft_consensus.cc:1275] T fd8311b779af4e1fa1402b2a02a602f2 P 1c50de89a5a845668696dcb28c3badd6 [term 2 FOLLOWER]: Refusing update from remote peer 197b880fa39d432a9b472845876bfaf0: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260129 02:59:24.660753 21942 consensus_queue.cc:1048] T fd8311b779af4e1fa1402b2a02a602f2 P 197b880fa39d432a9b472845876bfaf0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
W20260129 02:59:24.687122 20480 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.211.4:40439, user_credentials={real_user=slave}} blocked reactor thread for 58336.1us
I20260129 02:59:24.641104 21971 consensus_queue.cc:1059] T dc2f4cb752cf4d59b5f6bc689f5d8104 P 1c50de89a5a845668696dcb28c3badd6 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.028s
I20260129 02:59:24.824903 22033 raft_consensus.cc:493] T 45cc5f3d2688486d9b6dbe7e666dedde P 7a9daf5d42a84615a6383cfec9a10608 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 1c50de89a5a845668696dcb28c3badd6)
I20260129 02:59:24.825649 21971 raft_consensus.cc:3055] T dc2f4cb752cf4d59b5f6bc689f5d8104 P 1c50de89a5a845668696dcb28c3badd6 [term 1 LEADER]: Stepping down as leader of term 1
I20260129 02:59:24.826185 21971 raft_consensus.cc:740] T dc2f4cb752cf4d59b5f6bc689f5d8104 P 1c50de89a5a845668696dcb28c3badd6 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 1c50de89a5a845668696dcb28c3badd6, State: Running, Role: LEADER
I20260129 02:59:24.828171 21971 consensus_queue.cc:260] T dc2f4cb752cf4d59b5f6bc689f5d8104 P 1c50de89a5a845668696dcb28c3badd6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } }
I20260129 02:59:24.829509 21971 raft_consensus.cc:3060] T dc2f4cb752cf4d59b5f6bc689f5d8104 P 1c50de89a5a845668696dcb28c3badd6 [term 1 FOLLOWER]: Advancing to term 2
I20260129 02:59:24.830039 21134 raft_consensus.cc:1275] T fd8311b779af4e1fa1402b2a02a602f2 P 7a9daf5d42a84615a6383cfec9a10608 [term 2 FOLLOWER]: Refusing update from remote peer 197b880fa39d432a9b472845876bfaf0: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260129 02:59:24.836621 21944 raft_consensus.cc:493] T 78aa39096868415d9afc2ffcb248e198 P 1c50de89a5a845668696dcb28c3badd6 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260129 02:59:24.893616 21944 raft_consensus.cc:515] T 78aa39096868415d9afc2ffcb248e198 P 1c50de89a5a845668696dcb28c3badd6 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } }
I20260129 02:59:24.843815 20432 tablet_service.cc:2066] Received LeaderStepDown RPC: tablet_id: "4b1876f9a70b49edb3aa74105e66d3b3"
dest_uuid: "197b880fa39d432a9b472845876bfaf0"
mode: GRACEFUL
new_leader_uuid: "1c50de89a5a845668696dcb28c3badd6"
from {username='slave'} at 127.0.0.1:57776
I20260129 02:59:24.896173 20432 raft_consensus.cc:606] T 4b1876f9a70b49edb3aa74105e66d3b3 P 197b880fa39d432a9b472845876bfaf0 [term 2 LEADER]: Received request to transfer leadership to TS 1c50de89a5a845668696dcb28c3badd6
I20260129 02:59:24.844615 22031 tablet_replica.cc:333] T 3ccac82a01a2445fa5b0ce6c0fcf2295 P 1c50de89a5a845668696dcb28c3badd6: stopping tablet replica
I20260129 02:59:24.856688 20508 raft_consensus.cc:1240] T fcddab6bca684adf8ea032289a00f90e P 974b802141bd4f13beffef2aa03f88f9 [term 5 FOLLOWER]: Rejecting Update request from peer 1c50de89a5a845668696dcb28c3badd6 for earlier term 4. Current term is 5. Ops: []
I20260129 02:59:24.898725 21944 leader_election.cc:290] T 78aa39096868415d9afc2ffcb248e198 P 1c50de89a5a845668696dcb28c3badd6 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 197b880fa39d432a9b472845876bfaf0 (127.18.211.1:42027), 7a9daf5d42a84615a6383cfec9a10608 (127.18.211.4:40439)
I20260129 02:59:24.900327 20508 tablet_service.cc:2066] Received LeaderStepDown RPC: tablet_id: "76f713c3331540d2a5b76164103e9eb1"
dest_uuid: "974b802141bd4f13beffef2aa03f88f9"
mode: GRACEFUL
new_leader_uuid: "1c50de89a5a845668696dcb28c3badd6"
from {username='slave'} at 127.0.0.1:52008
I20260129 02:59:24.900873 21788 consensus_queue.cc:1048] T fd8311b779af4e1fa1402b2a02a602f2 P 197b880fa39d432a9b472845876bfaf0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260129 02:59:24.900926 20508 raft_consensus.cc:606] T 76f713c3331540d2a5b76164103e9eb1 P 974b802141bd4f13beffef2aa03f88f9 [term 1 LEADER]: Received request to transfer leadership to TS 1c50de89a5a845668696dcb28c3badd6
I20260129 02:59:24.902853 22023 raft_consensus.cc:493] T 0678b8b468bd43cdbc3e485900844821 P 7a9daf5d42a84615a6383cfec9a10608 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260129 02:59:24.903330 22023 raft_consensus.cc:515] T 0678b8b468bd43cdbc3e485900844821 P 7a9daf5d42a84615a6383cfec9a10608 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } }
I20260129 02:59:24.905009 22004 raft_consensus.cc:493] T cedce331e1584ac8b636c9b8a72b2ba0 P 1c50de89a5a845668696dcb28c3badd6 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260129 02:59:24.905629 22023 leader_election.cc:290] T 0678b8b468bd43cdbc3e485900844821 P 7a9daf5d42a84615a6383cfec9a10608 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 197b880fa39d432a9b472845876bfaf0 (127.18.211.1:42027), 974b802141bd4f13beffef2aa03f88f9 (127.18.211.2:34517)
I20260129 02:59:24.905563 22004 raft_consensus.cc:515] T cedce331e1584ac8b636c9b8a72b2ba0 P 1c50de89a5a845668696dcb28c3badd6 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
W20260129 02:59:24.909750 20480 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.211.3:37735, user_credentials={real_user=slave}} blocked reactor thread for 208480us
I20260129 02:59:24.910200 20511 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "cedce331e1584ac8b636c9b8a72b2ba0" candidate_uuid: "1c50de89a5a845668696dcb28c3badd6" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "974b802141bd4f13beffef2aa03f88f9" is_pre_election: true
W20260129 02:59:24.860621 20553 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.211.1:42027, user_credentials={real_user=slave}} blocked reactor thread for 210585us
I20260129 02:59:24.879057 21991 raft_consensus.cc:493] T 45cc5f3d2688486d9b6dbe7e666dedde P 1c50de89a5a845668696dcb28c3badd6 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260129 02:59:24.825697 22033 raft_consensus.cc:515] T 45cc5f3d2688486d9b6dbe7e666dedde P 7a9daf5d42a84615a6383cfec9a10608 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } }
I20260129 02:59:24.910885 20508 tablet_service.cc:2066] Received LeaderStepDown RPC: tablet_id: "7d9494f5584d4d5db8b9adf1c8fc888b"
dest_uuid: "974b802141bd4f13beffef2aa03f88f9"
mode: GRACEFUL
new_leader_uuid: "1c50de89a5a845668696dcb28c3badd6"
from {username='slave'} at 127.0.0.1:52008
I20260129 02:59:24.907673 22004 leader_election.cc:290] T cedce331e1584ac8b636c9b8a72b2ba0 P 1c50de89a5a845668696dcb28c3badd6 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 197b880fa39d432a9b472845876bfaf0 (127.18.211.1:42027), 974b802141bd4f13beffef2aa03f88f9 (127.18.211.2:34517)
I20260129 02:59:24.911432 20553 leader_election.cc:304] T 1ce4ddf5b10e4ae8b193b13b2e5936ee P 1c50de89a5a845668696dcb28c3badd6 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 197b880fa39d432a9b472845876bfaf0, 1c50de89a5a845668696dcb28c3badd6; no voters: 974b802141bd4f13beffef2aa03f88f9
I20260129 02:59:24.917673 22033 leader_election.cc:290] T 45cc5f3d2688486d9b6dbe7e666dedde P 7a9daf5d42a84615a6383cfec9a10608 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 974b802141bd4f13beffef2aa03f88f9 (127.18.211.2:34517), 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3:37735)
I20260129 02:59:24.917788 21991 raft_consensus.cc:515] T 45cc5f3d2688486d9b6dbe7e666dedde P 1c50de89a5a845668696dcb28c3badd6 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } }
I20260129 02:59:24.918588 21984 consensus_queue.cc:1059] T fcddab6bca684adf8ea032289a00f90e P 1c50de89a5a845668696dcb28c3badd6 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 }, Status: INVALID_TERM, Last received: 4.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.003s
I20260129 02:59:24.919768 20588 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "45cc5f3d2688486d9b6dbe7e666dedde" candidate_uuid: "7a9daf5d42a84615a6383cfec9a10608" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "1c50de89a5a845668696dcb28c3badd6" is_pre_election: true
I20260129 02:59:24.914139 20508 raft_consensus.cc:606] T 7d9494f5584d4d5db8b9adf1c8fc888b P 974b802141bd4f13beffef2aa03f88f9 [term 1 LEADER]: Received request to transfer leadership to TS 1c50de89a5a845668696dcb28c3badd6
I20260129 02:59:24.924669 21132 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "78aa39096868415d9afc2ffcb248e198" candidate_uuid: "1c50de89a5a845668696dcb28c3badd6" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "7a9daf5d42a84615a6383cfec9a10608" is_pre_election: true
I20260129 02:59:24.934747 20553 leader_election.cc:304] T 75be4bd6bd45474795fd472eff2953ac P 1c50de89a5a845668696dcb28c3badd6 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 197b880fa39d432a9b472845876bfaf0, 1c50de89a5a845668696dcb28c3badd6; no voters: 974b802141bd4f13beffef2aa03f88f9
I20260129 02:59:24.935520 20588 raft_consensus.cc:2468] T 45cc5f3d2688486d9b6dbe7e666dedde P 1c50de89a5a845668696dcb28c3badd6 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7a9daf5d42a84615a6383cfec9a10608 in term 2.
I20260129 02:59:24.937225 20432 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0678b8b468bd43cdbc3e485900844821" candidate_uuid: "7a9daf5d42a84615a6383cfec9a10608" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "197b880fa39d432a9b472845876bfaf0" is_pre_election: true
I20260129 02:59:24.937845 20432 raft_consensus.cc:2468] T 0678b8b468bd43cdbc3e485900844821 P 197b880fa39d432a9b472845876bfaf0 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7a9daf5d42a84615a6383cfec9a10608 in term 3.
I20260129 02:59:24.938076 21991 leader_election.cc:290] T 45cc5f3d2688486d9b6dbe7e666dedde P 1c50de89a5a845668696dcb28c3badd6 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 974b802141bd4f13beffef2aa03f88f9 (127.18.211.2:34517), 7a9daf5d42a84615a6383cfec9a10608 (127.18.211.4:40439)
I20260129 02:59:24.939034 21100 leader_election.cc:304] T 45cc5f3d2688486d9b6dbe7e666dedde P 7a9daf5d42a84615a6383cfec9a10608 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1c50de89a5a845668696dcb28c3badd6, 7a9daf5d42a84615a6383cfec9a10608; no voters:
I20260129 02:59:24.939488 21132 raft_consensus.cc:2468] T 78aa39096868415d9afc2ffcb248e198 P 7a9daf5d42a84615a6383cfec9a10608 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1c50de89a5a845668696dcb28c3badd6 in term 2.
I20260129 02:59:24.940181 22033 raft_consensus.cc:2804] T 45cc5f3d2688486d9b6dbe7e666dedde P 7a9daf5d42a84615a6383cfec9a10608 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260129 02:59:24.940600 22033 raft_consensus.cc:493] T 45cc5f3d2688486d9b6dbe7e666dedde P 7a9daf5d42a84615a6383cfec9a10608 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 1c50de89a5a845668696dcb28c3badd6)
I20260129 02:59:24.940826 22004 raft_consensus.cc:3055] T fcddab6bca684adf8ea032289a00f90e P 1c50de89a5a845668696dcb28c3badd6 [term 4 LEADER]: Stepping down as leader of term 4
I20260129 02:59:24.940958 22033 raft_consensus.cc:3060] T 45cc5f3d2688486d9b6dbe7e666dedde P 7a9daf5d42a84615a6383cfec9a10608 [term 1 FOLLOWER]: Advancing to term 2
I20260129 02:59:24.933962 21971 raft_consensus.cc:2804] T 1ce4ddf5b10e4ae8b193b13b2e5936ee P 1c50de89a5a845668696dcb28c3badd6 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260129 02:59:24.943012 20507 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "45cc5f3d2688486d9b6dbe7e666dedde" candidate_uuid: "7a9daf5d42a84615a6383cfec9a10608" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "974b802141bd4f13beffef2aa03f88f9" is_pre_election: true
I20260129 02:59:24.944504 20507 raft_consensus.cc:2393] T 45cc5f3d2688486d9b6dbe7e666dedde P 974b802141bd4f13beffef2aa03f88f9 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 7a9daf5d42a84615a6383cfec9a10608 in current term 2: Already voted for candidate 974b802141bd4f13beffef2aa03f88f9 in this term.
I20260129 02:59:24.935528 20512 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0678b8b468bd43cdbc3e485900844821" candidate_uuid: "7a9daf5d42a84615a6383cfec9a10608" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "974b802141bd4f13beffef2aa03f88f9" is_pre_election: true
I20260129 02:59:24.945232 20512 raft_consensus.cc:2468] T 0678b8b468bd43cdbc3e485900844821 P 974b802141bd4f13beffef2aa03f88f9 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7a9daf5d42a84615a6383cfec9a10608 in term 3.
I20260129 02:59:24.943135 21863 raft_consensus.cc:493] T 87d5fe48a44a408aaf03e894a67ecb2f P 7a9daf5d42a84615a6383cfec9a10608 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260129 02:59:24.946048 21863 raft_consensus.cc:515] T 87d5fe48a44a408aaf03e894a67ecb2f P 7a9daf5d42a84615a6383cfec9a10608 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } }
I20260129 02:59:24.948094 21863 leader_election.cc:290] T 87d5fe48a44a408aaf03e894a67ecb2f P 7a9daf5d42a84615a6383cfec9a10608 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 197b880fa39d432a9b472845876bfaf0 (127.18.211.1:42027), 974b802141bd4f13beffef2aa03f88f9 (127.18.211.2:34517)
I20260129 02:59:24.948695 21971 raft_consensus.cc:493] T 1ce4ddf5b10e4ae8b193b13b2e5936ee P 1c50de89a5a845668696dcb28c3badd6 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 197b880fa39d432a9b472845876bfaf0)
I20260129 02:59:24.949361 21971 raft_consensus.cc:3060] T 1ce4ddf5b10e4ae8b193b13b2e5936ee P 1c50de89a5a845668696dcb28c3badd6 [term 1 FOLLOWER]: Advancing to term 2
I20260129 02:59:24.950487 20432 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "87d5fe48a44a408aaf03e894a67ecb2f" candidate_uuid: "7a9daf5d42a84615a6383cfec9a10608" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "197b880fa39d432a9b472845876bfaf0" is_pre_election: true
I20260129 02:59:24.951241 20432 raft_consensus.cc:2468] T 87d5fe48a44a408aaf03e894a67ecb2f P 197b880fa39d432a9b472845876bfaf0 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7a9daf5d42a84615a6383cfec9a10608 in term 3.
I20260129 02:59:24.951911 22033 raft_consensus.cc:515] T 45cc5f3d2688486d9b6dbe7e666dedde P 7a9daf5d42a84615a6383cfec9a10608 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } }
I20260129 02:59:24.954101 21991 raft_consensus.cc:2804] T 75be4bd6bd45474795fd472eff2953ac P 1c50de89a5a845668696dcb28c3badd6 [term 5 FOLLOWER]: Leader election won for term 5
I20260129 02:59:24.954663 21991 raft_consensus.cc:697] T 75be4bd6bd45474795fd472eff2953ac P 1c50de89a5a845668696dcb28c3badd6 [term 5 LEADER]: Becoming Leader. State: Replica: 1c50de89a5a845668696dcb28c3badd6, State: Running, Role: LEADER
I20260129 02:59:24.955571 21991 consensus_queue.cc:237] T 75be4bd6bd45474795fd472eff2953ac P 1c50de89a5a845668696dcb28c3badd6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:59:24.960139 22004 raft_consensus.cc:740] T fcddab6bca684adf8ea032289a00f90e P 1c50de89a5a845668696dcb28c3badd6 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 1c50de89a5a845668696dcb28c3badd6, State: Running, Role: LEADER
I20260129 02:59:24.960459 21971 raft_consensus.cc:515] T 1ce4ddf5b10e4ae8b193b13b2e5936ee P 1c50de89a5a845668696dcb28c3badd6 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:59:24.961131 22004 consensus_queue.cc:260] T fcddab6bca684adf8ea032289a00f90e P 1c50de89a5a845668696dcb28c3badd6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 4.4, Last appended by leader: 4, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:59:24.962247 20508 tablet_service.cc:2066] Received LeaderStepDown RPC: tablet_id: "a0fcc58d6fb6473cafbd9d34d5195269"
dest_uuid: "974b802141bd4f13beffef2aa03f88f9"
mode: GRACEFUL
new_leader_uuid: "197b880fa39d432a9b472845876bfaf0"
from {username='slave'} at 127.0.0.1:52008
I20260129 02:59:24.962896 20508 raft_consensus.cc:606] T a0fcc58d6fb6473cafbd9d34d5195269 P 974b802141bd4f13beffef2aa03f88f9 [term 1 LEADER]: Received request to transfer leadership to TS 197b880fa39d432a9b472845876bfaf0
I20260129 02:59:24.967340 20438 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "78aa39096868415d9afc2ffcb248e198" candidate_uuid: "1c50de89a5a845668696dcb28c3badd6" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "197b880fa39d432a9b472845876bfaf0" is_pre_election: true
I20260129 02:59:24.967952 20438 raft_consensus.cc:2468] T 78aa39096868415d9afc2ffcb248e198 P 197b880fa39d432a9b472845876bfaf0 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1c50de89a5a845668696dcb28c3badd6 in term 1.
I20260129 02:59:24.965301 20507 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "45cc5f3d2688486d9b6dbe7e666dedde" candidate_uuid: "1c50de89a5a845668696dcb28c3badd6" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "974b802141bd4f13beffef2aa03f88f9" is_pre_election: true
I20260129 02:59:24.968868 20507 raft_consensus.cc:2468] T 45cc5f3d2688486d9b6dbe7e666dedde P 974b802141bd4f13beffef2aa03f88f9 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1c50de89a5a845668696dcb28c3badd6 in term 2.
I20260129 02:59:24.968533 21944 raft_consensus.cc:493] T a5a05fbcbbcf4916bbc01fd1a5b5e90d P 1c50de89a5a845668696dcb28c3badd6 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260129 02:59:24.971108 20556 leader_election.cc:304] T 45cc5f3d2688486d9b6dbe7e666dedde P 1c50de89a5a845668696dcb28c3badd6 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1c50de89a5a845668696dcb28c3badd6, 974b802141bd4f13beffef2aa03f88f9; no voters:
I20260129 02:59:24.971782 20553 leader_election.cc:304] T 78aa39096868415d9afc2ffcb248e198 P 1c50de89a5a845668696dcb28c3badd6 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 197b880fa39d432a9b472845876bfaf0, 1c50de89a5a845668696dcb28c3badd6; no voters:
I20260129 02:59:24.962340 22004 raft_consensus.cc:3060] T fcddab6bca684adf8ea032289a00f90e P 1c50de89a5a845668696dcb28c3badd6 [term 4 FOLLOWER]: Advancing to term 5
I20260129 02:59:24.975180 20439 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "cedce331e1584ac8b636c9b8a72b2ba0" candidate_uuid: "1c50de89a5a845668696dcb28c3badd6" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "197b880fa39d432a9b472845876bfaf0" is_pre_election: true
I20260129 02:59:24.974381 20440 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1ce4ddf5b10e4ae8b193b13b2e5936ee" candidate_uuid: "1c50de89a5a845668696dcb28c3badd6" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "197b880fa39d432a9b472845876bfaf0"
I20260129 02:59:24.976380 20439 raft_consensus.cc:2468] T cedce331e1584ac8b636c9b8a72b2ba0 P 197b880fa39d432a9b472845876bfaf0 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1c50de89a5a845668696dcb28c3badd6 in term 3.
I20260129 02:59:24.949383 21099 leader_election.cc:304] T 0678b8b468bd43cdbc3e485900844821 P 7a9daf5d42a84615a6383cfec9a10608 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 197b880fa39d432a9b472845876bfaf0, 7a9daf5d42a84615a6383cfec9a10608; no voters:
I20260129 02:59:24.978714 20553 leader_election.cc:304] T cedce331e1584ac8b636c9b8a72b2ba0 P 1c50de89a5a845668696dcb28c3badd6 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 197b880fa39d432a9b472845876bfaf0, 1c50de89a5a845668696dcb28c3badd6; no voters: 974b802141bd4f13beffef2aa03f88f9
I20260129 02:59:24.976341 21995 raft_consensus.cc:2804] T 78aa39096868415d9afc2ffcb248e198 P 1c50de89a5a845668696dcb28c3badd6 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260129 02:59:24.983716 21995 raft_consensus.cc:493] T 78aa39096868415d9afc2ffcb248e198 P 1c50de89a5a845668696dcb28c3badd6 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260129 02:59:24.984107 21995 raft_consensus.cc:3060] T 78aa39096868415d9afc2ffcb248e198 P 1c50de89a5a845668696dcb28c3badd6 [term 2 FOLLOWER]: Advancing to term 3
I20260129 02:59:24.987346 22033 leader_election.cc:290] T 45cc5f3d2688486d9b6dbe7e666dedde P 7a9daf5d42a84615a6383cfec9a10608 [CANDIDATE]: Term 2 election: Requested vote from peers 974b802141bd4f13beffef2aa03f88f9 (127.18.211.2:34517), 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3:37735)
I20260129 02:59:24.987569 20583 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "45cc5f3d2688486d9b6dbe7e666dedde" candidate_uuid: "7a9daf5d42a84615a6383cfec9a10608" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "1c50de89a5a845668696dcb28c3badd6"
I20260129 02:59:24.988186 21988 raft_consensus.cc:2804] T cedce331e1584ac8b636c9b8a72b2ba0 P 1c50de89a5a845668696dcb28c3badd6 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260129 02:59:24.988642 21988 raft_consensus.cc:493] T cedce331e1584ac8b636c9b8a72b2ba0 P 1c50de89a5a845668696dcb28c3badd6 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260129 02:59:24.988972 21988 raft_consensus.cc:3060] T cedce331e1584ac8b636c9b8a72b2ba0 P 1c50de89a5a845668696dcb28c3badd6 [term 3 FOLLOWER]: Advancing to term 4
I20260129 02:59:24.974484 21984 raft_consensus.cc:2804] T 45cc5f3d2688486d9b6dbe7e666dedde P 1c50de89a5a845668696dcb28c3badd6 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260129 02:59:24.997720 21984 raft_consensus.cc:493] T 45cc5f3d2688486d9b6dbe7e666dedde P 1c50de89a5a845668696dcb28c3badd6 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260129 02:59:24.998099 21984 raft_consensus.cc:3060] T 45cc5f3d2688486d9b6dbe7e666dedde P 1c50de89a5a845668696dcb28c3badd6 [term 2 FOLLOWER]: Advancing to term 3
I20260129 02:59:24.998140 20508 tablet_service.cc:2066] Received LeaderStepDown RPC: tablet_id: "a4c1226b987b468fa84dfc6ba0e2dbe1"
dest_uuid: "974b802141bd4f13beffef2aa03f88f9"
mode: GRACEFUL
new_leader_uuid: "197b880fa39d432a9b472845876bfaf0"
from {username='slave'} at 127.0.0.1:52008
I20260129 02:59:24.998804 20508 raft_consensus.cc:606] T a4c1226b987b468fa84dfc6ba0e2dbe1 P 974b802141bd4f13beffef2aa03f88f9 [term 1 LEADER]: Received request to transfer leadership to TS 197b880fa39d432a9b472845876bfaf0
I20260129 02:59:25.000196 22003 raft_consensus.cc:993] T a0fcc58d6fb6473cafbd9d34d5195269 P 974b802141bd4f13beffef2aa03f88f9: : Instructing follower 197b880fa39d432a9b472845876bfaf0 to start an election
I20260129 02:59:25.000659 22003 raft_consensus.cc:1081] T a0fcc58d6fb6473cafbd9d34d5195269 P 974b802141bd4f13beffef2aa03f88f9 [term 1 LEADER]: Signalling peer 197b880fa39d432a9b472845876bfaf0 to start an election
I20260129 02:59:25.001211 19276 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 5
I20260129 02:59:25.002908 19276 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20260129 02:59:25.004814 20432 tablet_service.cc:2038] Received Run Leader Election RPC: tablet_id: "a0fcc58d6fb6473cafbd9d34d5195269"
dest_uuid: "197b880fa39d432a9b472845876bfaf0"
from {username='slave'} at 127.0.0.1:55672
I20260129 02:59:25.005654 20432 raft_consensus.cc:493] T a0fcc58d6fb6473cafbd9d34d5195269 P 197b880fa39d432a9b472845876bfaf0 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260129 02:59:25.006150 20432 raft_consensus.cc:3060] T a0fcc58d6fb6473cafbd9d34d5195269 P 197b880fa39d432a9b472845876bfaf0 [term 1 FOLLOWER]: Advancing to term 2
I20260129 02:59:25.007438 19276 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260129 02:59:25.015070 22037 raft_consensus.cc:493] T 926a250fa5834e88b96ccecbefbf5086 P 1c50de89a5a845668696dcb28c3badd6 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260129 02:59:25.015632 22037 raft_consensus.cc:515] T 926a250fa5834e88b96ccecbefbf5086 P 1c50de89a5a845668696dcb28c3badd6 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:59:25.016870 22031 raft_consensus.cc:2243] T 3ccac82a01a2445fa5b0ce6c0fcf2295 P 1c50de89a5a845668696dcb28c3badd6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260129 02:59:25.017683 22031 raft_consensus.cc:2272] T 3ccac82a01a2445fa5b0ce6c0fcf2295 P 1c50de89a5a845668696dcb28c3badd6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260129 02:59:25.017886 22037 leader_election.cc:290] T 926a250fa5834e88b96ccecbefbf5086 P 1c50de89a5a845668696dcb28c3badd6 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 197b880fa39d432a9b472845876bfaf0 (127.18.211.1:42027), 974b802141bd4f13beffef2aa03f88f9 (127.18.211.2:34517)
I20260129 02:59:25.019048 20434 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "926a250fa5834e88b96ccecbefbf5086" candidate_uuid: "1c50de89a5a845668696dcb28c3badd6" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "197b880fa39d432a9b472845876bfaf0" is_pre_election: true
I20260129 02:59:25.020687 20508 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "926a250fa5834e88b96ccecbefbf5086" candidate_uuid: "1c50de89a5a845668696dcb28c3badd6" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "974b802141bd4f13beffef2aa03f88f9" is_pre_election: true
I20260129 02:59:25.021319 20508 raft_consensus.cc:2468] T 926a250fa5834e88b96ccecbefbf5086 P 974b802141bd4f13beffef2aa03f88f9 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1c50de89a5a845668696dcb28c3badd6 in term 3.
I20260129 02:59:24.970763 21944 raft_consensus.cc:515] T a5a05fbcbbcf4916bbc01fd1a5b5e90d P 1c50de89a5a845668696dcb28c3badd6 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:59:25.023136 21944 leader_election.cc:290] T a5a05fbcbbcf4916bbc01fd1a5b5e90d P 1c50de89a5a845668696dcb28c3badd6 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 197b880fa39d432a9b472845876bfaf0 (127.18.211.1:42027), 974b802141bd4f13beffef2aa03f88f9 (127.18.211.2:34517)
I20260129 02:59:25.023725 21129 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "45cc5f3d2688486d9b6dbe7e666dedde" candidate_uuid: "1c50de89a5a845668696dcb28c3badd6" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "7a9daf5d42a84615a6383cfec9a10608" is_pre_election: true
I20260129 02:59:25.023823 20433 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a5a05fbcbbcf4916bbc01fd1a5b5e90d" candidate_uuid: "1c50de89a5a845668696dcb28c3badd6" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "197b880fa39d432a9b472845876bfaf0" is_pre_election: true
I20260129 02:59:25.024433 21129 raft_consensus.cc:2468] T 45cc5f3d2688486d9b6dbe7e666dedde P 7a9daf5d42a84615a6383cfec9a10608 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1c50de89a5a845668696dcb28c3badd6 in term 2.
I20260129 02:59:25.024552 20433 raft_consensus.cc:2468] T a5a05fbcbbcf4916bbc01fd1a5b5e90d P 197b880fa39d432a9b472845876bfaf0 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1c50de89a5a845668696dcb28c3badd6 in term 3.
I20260129 02:59:25.024643 20556 leader_election.cc:304] T 926a250fa5834e88b96ccecbefbf5086 P 1c50de89a5a845668696dcb28c3badd6 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1c50de89a5a845668696dcb28c3badd6, 974b802141bd4f13beffef2aa03f88f9; no voters:
I20260129 02:59:25.025889 20553 leader_election.cc:304] T a5a05fbcbbcf4916bbc01fd1a5b5e90d P 1c50de89a5a845668696dcb28c3badd6 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 197b880fa39d432a9b472845876bfaf0, 1c50de89a5a845668696dcb28c3badd6; no voters:
I20260129 02:59:25.026019 22035 raft_consensus.cc:2804] T 926a250fa5834e88b96ccecbefbf5086 P 1c50de89a5a845668696dcb28c3badd6 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260129 02:59:25.026463 22035 raft_consensus.cc:493] T 926a250fa5834e88b96ccecbefbf5086 P 1c50de89a5a845668696dcb28c3badd6 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260129 02:59:25.026767 22005 raft_consensus.cc:2804] T a5a05fbcbbcf4916bbc01fd1a5b5e90d P 1c50de89a5a845668696dcb28c3badd6 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260129 02:59:25.027230 22005 raft_consensus.cc:493] T a5a05fbcbbcf4916bbc01fd1a5b5e90d P 1c50de89a5a845668696dcb28c3badd6 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260129 02:59:25.027319 20508 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a5a05fbcbbcf4916bbc01fd1a5b5e90d" candidate_uuid: "1c50de89a5a845668696dcb28c3badd6" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "974b802141bd4f13beffef2aa03f88f9" is_pre_election: true
I20260129 02:59:25.027587 22005 raft_consensus.cc:3060] T a5a05fbcbbcf4916bbc01fd1a5b5e90d P 1c50de89a5a845668696dcb28c3badd6 [term 3 FOLLOWER]: Advancing to term 4
I20260129 02:59:25.027961 20508 raft_consensus.cc:2468] T a5a05fbcbbcf4916bbc01fd1a5b5e90d P 974b802141bd4f13beffef2aa03f88f9 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1c50de89a5a845668696dcb28c3badd6 in term 3.
I20260129 02:59:25.029820 20440 raft_consensus.cc:2468] T 1ce4ddf5b10e4ae8b193b13b2e5936ee P 197b880fa39d432a9b472845876bfaf0 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1c50de89a5a845668696dcb28c3badd6 in term 2.
I20260129 02:59:25.019752 20434 raft_consensus.cc:2468] T 926a250fa5834e88b96ccecbefbf5086 P 197b880fa39d432a9b472845876bfaf0 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1c50de89a5a845668696dcb28c3badd6 in term 3.
I20260129 02:59:25.026774 22035 raft_consensus.cc:3060] T 926a250fa5834e88b96ccecbefbf5086 P 1c50de89a5a845668696dcb28c3badd6 [term 3 FOLLOWER]: Advancing to term 4
I20260129 02:59:25.030676 21863 raft_consensus.cc:2804] T 0678b8b468bd43cdbc3e485900844821 P 7a9daf5d42a84615a6383cfec9a10608 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260129 02:59:25.032296 20553 leader_election.cc:304] T 1ce4ddf5b10e4ae8b193b13b2e5936ee P 1c50de89a5a845668696dcb28c3badd6 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 197b880fa39d432a9b472845876bfaf0, 1c50de89a5a845668696dcb28c3badd6; no voters:
I20260129 02:59:25.033171 21863 raft_consensus.cc:493] T 0678b8b468bd43cdbc3e485900844821 P 7a9daf5d42a84615a6383cfec9a10608 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260129 02:59:25.034485 21863 raft_consensus.cc:3060] T 0678b8b468bd43cdbc3e485900844821 P 7a9daf5d42a84615a6383cfec9a10608 [term 3 FOLLOWER]: Advancing to term 4
I20260129 02:59:25.050740 20508 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1ce4ddf5b10e4ae8b193b13b2e5936ee" candidate_uuid: "1c50de89a5a845668696dcb28c3badd6" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "974b802141bd4f13beffef2aa03f88f9"
I20260129 02:59:25.050930 20512 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "87d5fe48a44a408aaf03e894a67ecb2f" candidate_uuid: "7a9daf5d42a84615a6383cfec9a10608" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "974b802141bd4f13beffef2aa03f88f9" is_pre_election: true
I20260129 02:59:25.051542 20508 raft_consensus.cc:2393] T 1ce4ddf5b10e4ae8b193b13b2e5936ee P 974b802141bd4f13beffef2aa03f88f9 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 1c50de89a5a845668696dcb28c3badd6 in current term 2: Already voted for candidate 974b802141bd4f13beffef2aa03f88f9 in this term.
I20260129 02:59:25.055516 21883 raft_consensus.cc:493] T b189f08e16a74a11980cc55d99359a0a P 197b880fa39d432a9b472845876bfaf0 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260129 02:59:25.066602 21883 raft_consensus.cc:515] T b189f08e16a74a11980cc55d99359a0a P 197b880fa39d432a9b472845876bfaf0 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
W20260129 02:59:25.030129 21099 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.211.1:42027, user_credentials={real_user=slave}} blocked reactor thread for 80055.6us
I20260129 02:59:25.059290 21984 raft_consensus.cc:515] T 45cc5f3d2688486d9b6dbe7e666dedde P 1c50de89a5a845668696dcb28c3badd6 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } }
I20260129 02:59:25.072042 20508 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "45cc5f3d2688486d9b6dbe7e666dedde" candidate_uuid: "7a9daf5d42a84615a6383cfec9a10608" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "974b802141bd4f13beffef2aa03f88f9"
I20260129 02:59:25.072855 20508 raft_consensus.cc:2393] T 45cc5f3d2688486d9b6dbe7e666dedde P 974b802141bd4f13beffef2aa03f88f9 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 7a9daf5d42a84615a6383cfec9a10608 in current term 2: Already voted for candidate 974b802141bd4f13beffef2aa03f88f9 in this term.
I20260129 02:59:25.043368 21991 raft_consensus.cc:2804] T 1ce4ddf5b10e4ae8b193b13b2e5936ee P 1c50de89a5a845668696dcb28c3badd6 [term 2 FOLLOWER]: Leader election won for term 2
I20260129 02:59:25.089083 21863 raft_consensus.cc:515] T 0678b8b468bd43cdbc3e485900844821 P 7a9daf5d42a84615a6383cfec9a10608 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } }
I20260129 02:59:25.093132 20432 raft_consensus.cc:515] T a0fcc58d6fb6473cafbd9d34d5195269 P 197b880fa39d432a9b472845876bfaf0 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:59:25.053188 21971 leader_election.cc:290] T 1ce4ddf5b10e4ae8b193b13b2e5936ee P 1c50de89a5a845668696dcb28c3badd6 [CANDIDATE]: Term 2 election: Requested vote from peers 197b880fa39d432a9b472845876bfaf0 (127.18.211.1:42027), 974b802141bd4f13beffef2aa03f88f9 (127.18.211.2:34517)
I20260129 02:59:25.033861 21988 raft_consensus.cc:515] T cedce331e1584ac8b636c9b8a72b2ba0 P 1c50de89a5a845668696dcb28c3badd6 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:59:25.103490 22031 ts_tablet_manager.cc:1916] T 3ccac82a01a2445fa5b0ce6c0fcf2295 P 1c50de89a5a845668696dcb28c3badd6: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260129 02:59:25.094372 21863 leader_election.cc:290] T 0678b8b468bd43cdbc3e485900844821 P 7a9daf5d42a84615a6383cfec9a10608 [CANDIDATE]: Term 4 election: Requested vote from peers 197b880fa39d432a9b472845876bfaf0 (127.18.211.1:42027), 974b802141bd4f13beffef2aa03f88f9 (127.18.211.2:34517)
I20260129 02:59:25.097317 21991 raft_consensus.cc:697] T 1ce4ddf5b10e4ae8b193b13b2e5936ee P 1c50de89a5a845668696dcb28c3badd6 [term 2 LEADER]: Becoming Leader. State: Replica: 1c50de89a5a845668696dcb28c3badd6, State: Running, Role: LEADER
I20260129 02:59:25.121913 20512 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b189f08e16a74a11980cc55d99359a0a" candidate_uuid: "197b880fa39d432a9b472845876bfaf0" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "974b802141bd4f13beffef2aa03f88f9" is_pre_election: true
I20260129 02:59:25.125177 20512 raft_consensus.cc:2468] T b189f08e16a74a11980cc55d99359a0a P 974b802141bd4f13beffef2aa03f88f9 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 197b880fa39d432a9b472845876bfaf0 in term 3.
I20260129 02:59:25.123925 20583 raft_consensus.cc:2368] T 45cc5f3d2688486d9b6dbe7e666dedde P 1c50de89a5a845668696dcb28c3badd6 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 7a9daf5d42a84615a6383cfec9a10608 for earlier term 2. Current term is 3.
I20260129 02:59:25.127012 20407 leader_election.cc:304] T b189f08e16a74a11980cc55d99359a0a P 197b880fa39d432a9b472845876bfaf0 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 197b880fa39d432a9b472845876bfaf0, 974b802141bd4f13beffef2aa03f88f9; no voters:
I20260129 02:59:25.098248 22035 raft_consensus.cc:515] T 926a250fa5834e88b96ccecbefbf5086 P 1c50de89a5a845668696dcb28c3badd6 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:59:25.135398 20586 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b189f08e16a74a11980cc55d99359a0a" candidate_uuid: "197b880fa39d432a9b472845876bfaf0" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "1c50de89a5a845668696dcb28c3badd6" is_pre_election: true
I20260129 02:59:25.136507 20586 raft_consensus.cc:2468] T b189f08e16a74a11980cc55d99359a0a P 1c50de89a5a845668696dcb28c3badd6 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 197b880fa39d432a9b472845876bfaf0 in term 2.
I20260129 02:59:25.137630 21099 leader_election.cc:304] T 87d5fe48a44a408aaf03e894a67ecb2f P 7a9daf5d42a84615a6383cfec9a10608 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 197b880fa39d432a9b472845876bfaf0, 7a9daf5d42a84615a6383cfec9a10608; no voters: 974b802141bd4f13beffef2aa03f88f9
I20260129 02:59:25.130157 22042 raft_consensus.cc:2804] T b189f08e16a74a11980cc55d99359a0a P 197b880fa39d432a9b472845876bfaf0 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260129 02:59:25.145125 22023 raft_consensus.cc:2804] T 87d5fe48a44a408aaf03e894a67ecb2f P 7a9daf5d42a84615a6383cfec9a10608 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260129 02:59:25.145514 22042 raft_consensus.cc:493] T b189f08e16a74a11980cc55d99359a0a P 197b880fa39d432a9b472845876bfaf0 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260129 02:59:25.145596 22023 raft_consensus.cc:493] T 87d5fe48a44a408aaf03e894a67ecb2f P 7a9daf5d42a84615a6383cfec9a10608 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260129 02:59:25.144949 21787 raft_consensus.cc:993] T 4b1876f9a70b49edb3aa74105e66d3b3 P 197b880fa39d432a9b472845876bfaf0: : Instructing follower 1c50de89a5a845668696dcb28c3badd6 to start an election
I20260129 02:59:25.146143 22023 raft_consensus.cc:3060] T 87d5fe48a44a408aaf03e894a67ecb2f P 7a9daf5d42a84615a6383cfec9a10608 [term 3 FOLLOWER]: Advancing to term 4
I20260129 02:59:25.146766 21787 raft_consensus.cc:1081] T 4b1876f9a70b49edb3aa74105e66d3b3 P 197b880fa39d432a9b472845876bfaf0 [term 2 LEADER]: Signalling peer 1c50de89a5a845668696dcb28c3badd6 to start an election
I20260129 02:59:25.144670 21991 consensus_queue.cc:237] T 1ce4ddf5b10e4ae8b193b13b2e5936ee P 1c50de89a5a845668696dcb28c3badd6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:59:25.146091 22042 raft_consensus.cc:3060] T b189f08e16a74a11980cc55d99359a0a P 197b880fa39d432a9b472845876bfaf0 [term 3 FOLLOWER]: Advancing to term 4
I20260129 02:59:25.119513 22005 raft_consensus.cc:515] T a5a05fbcbbcf4916bbc01fd1a5b5e90d P 1c50de89a5a845668696dcb28c3badd6 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:59:25.133342 21883 leader_election.cc:290] T b189f08e16a74a11980cc55d99359a0a P 197b880fa39d432a9b472845876bfaf0 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 974b802141bd4f13beffef2aa03f88f9 (127.18.211.2:34517), 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3:37735)
I20260129 02:59:25.151933 22002 raft_consensus.cc:993] T a4c1226b987b468fa84dfc6ba0e2dbe1 P 974b802141bd4f13beffef2aa03f88f9: : Instructing follower 197b880fa39d432a9b472845876bfaf0 to start an election
I20260129 02:59:25.152067 20586 tablet_service.cc:2038] Received Run Leader Election RPC: tablet_id: "4b1876f9a70b49edb3aa74105e66d3b3"
dest_uuid: "1c50de89a5a845668696dcb28c3badd6"
from {username='slave'} at 127.0.0.1:35754
I20260129 02:59:25.164742 20586 raft_consensus.cc:493] T 4b1876f9a70b49edb3aa74105e66d3b3 P 1c50de89a5a845668696dcb28c3badd6 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20260129 02:59:25.167567 20586 raft_consensus.cc:3060] T 4b1876f9a70b49edb3aa74105e66d3b3 P 1c50de89a5a845668696dcb28c3badd6 [term 2 FOLLOWER]: Advancing to term 3
I20260129 02:59:25.156806 21100 leader_election.cc:400] T 45cc5f3d2688486d9b6dbe7e666dedde P 7a9daf5d42a84615a6383cfec9a10608 [CANDIDATE]: Term 2 election: Vote denied by peer 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3:37735) with higher term. Message: Invalid argument: T 45cc5f3d2688486d9b6dbe7e666dedde P 1c50de89a5a845668696dcb28c3badd6 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 7a9daf5d42a84615a6383cfec9a10608 for earlier term 2. Current term is 3.
I20260129 02:59:25.158335 21788 raft_consensus.cc:493] T 1299723a4476420ab02fd8bfd7aa0645 P 197b880fa39d432a9b472845876bfaf0 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260129 02:59:25.153137 21998 raft_consensus.cc:993] T 76f713c3331540d2a5b76164103e9eb1 P 974b802141bd4f13beffef2aa03f88f9: : Instructing follower 1c50de89a5a845668696dcb28c3badd6 to start an election
I20260129 02:59:25.171061 21100 leader_election.cc:403] T 45cc5f3d2688486d9b6dbe7e666dedde P 7a9daf5d42a84615a6383cfec9a10608 [CANDIDATE]: Term 2 election: Cancelling election due to peer responding with higher term
I20260129 02:59:25.175541 22002 raft_consensus.cc:1081] T a4c1226b987b468fa84dfc6ba0e2dbe1 P 974b802141bd4f13beffef2aa03f88f9 [term 1 LEADER]: Signalling peer 197b880fa39d432a9b472845876bfaf0 to start an election
I20260129 02:59:25.176782 21788 raft_consensus.cc:515] T 1299723a4476420ab02fd8bfd7aa0645 P 197b880fa39d432a9b472845876bfaf0 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } }
I20260129 02:59:25.178877 21788 leader_election.cc:290] T 1299723a4476420ab02fd8bfd7aa0645 P 197b880fa39d432a9b472845876bfaf0 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3:37735), 7a9daf5d42a84615a6383cfec9a10608 (127.18.211.4:40439)
I20260129 02:59:25.180356 21994 raft_consensus.cc:1081] T 76f713c3331540d2a5b76164103e9eb1 P 974b802141bd4f13beffef2aa03f88f9 [term 1 LEADER]: Signalling peer 1c50de89a5a845668696dcb28c3badd6 to start an election
I20260129 02:59:25.183100 21129 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1299723a4476420ab02fd8bfd7aa0645" candidate_uuid: "197b880fa39d432a9b472845876bfaf0" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "7a9daf5d42a84615a6383cfec9a10608" is_pre_election: true
I20260129 02:59:25.182498 20335 catalog_manager.cc:5654] T 75be4bd6bd45474795fd472eff2953ac P 1c50de89a5a845668696dcb28c3badd6 reported cstate change: term changed from 4 to 5, leader changed from 197b880fa39d432a9b472845876bfaf0 (127.18.211.1) to 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3). New cstate: current_term: 5 leader_uuid: "1c50de89a5a845668696dcb28c3badd6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } health_report { overall_health: HEALTHY } } }
I20260129 02:59:25.184558 21129 raft_consensus.cc:2468] T 1299723a4476420ab02fd8bfd7aa0645 P 7a9daf5d42a84615a6383cfec9a10608 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 197b880fa39d432a9b472845876bfaf0 in term 3.
I20260129 02:59:25.184911 21984 leader_election.cc:290] T 45cc5f3d2688486d9b6dbe7e666dedde P 1c50de89a5a845668696dcb28c3badd6 [CANDIDATE]: Term 3 election: Requested vote from peers 974b802141bd4f13beffef2aa03f88f9 (127.18.211.2:34517), 7a9daf5d42a84615a6383cfec9a10608 (127.18.211.4:40439)
I20260129 02:59:25.185616 22033 raft_consensus.cc:3060] T 45cc5f3d2688486d9b6dbe7e666dedde P 7a9daf5d42a84615a6383cfec9a10608 [term 2 FOLLOWER]: Advancing to term 3
I20260129 02:59:25.187564 21134 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "45cc5f3d2688486d9b6dbe7e666dedde" candidate_uuid: "1c50de89a5a845668696dcb28c3badd6" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "7a9daf5d42a84615a6383cfec9a10608"
I20260129 02:59:25.190723 20405 leader_election.cc:304] T 1299723a4476420ab02fd8bfd7aa0645 P 197b880fa39d432a9b472845876bfaf0 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 197b880fa39d432a9b472845876bfaf0, 7a9daf5d42a84615a6383cfec9a10608; no voters:
I20260129 02:59:25.183938 20512 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "45cc5f3d2688486d9b6dbe7e666dedde" candidate_uuid: "1c50de89a5a845668696dcb28c3badd6" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "974b802141bd4f13beffef2aa03f88f9"
I20260129 02:59:25.191577 20507 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0678b8b468bd43cdbc3e485900844821" candidate_uuid: "7a9daf5d42a84615a6383cfec9a10608" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "974b802141bd4f13beffef2aa03f88f9"
I20260129 02:59:25.192233 20507 raft_consensus.cc:3060] T 0678b8b468bd43cdbc3e485900844821 P 974b802141bd4f13beffef2aa03f88f9 [term 3 FOLLOWER]: Advancing to term 4
I20260129 02:59:25.193222 20584 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1299723a4476420ab02fd8bfd7aa0645" candidate_uuid: "197b880fa39d432a9b472845876bfaf0" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "1c50de89a5a845668696dcb28c3badd6" is_pre_election: true
I20260129 02:59:25.191898 20512 raft_consensus.cc:3060] T 45cc5f3d2688486d9b6dbe7e666dedde P 974b802141bd4f13beffef2aa03f88f9 [term 2 FOLLOWER]: Advancing to term 3
I20260129 02:59:25.197777 20581 tablet_service.cc:2038] Received Run Leader Election RPC: tablet_id: "76f713c3331540d2a5b76164103e9eb1"
dest_uuid: "1c50de89a5a845668696dcb28c3badd6"
from {username='slave'} at 127.0.0.1:35752
I20260129 02:59:25.198290 20581 raft_consensus.cc:493] T 76f713c3331540d2a5b76164103e9eb1 P 1c50de89a5a845668696dcb28c3badd6 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260129 02:59:25.198606 20581 raft_consensus.cc:3060] T 76f713c3331540d2a5b76164103e9eb1 P 1c50de89a5a845668696dcb28c3badd6 [term 1 FOLLOWER]: Advancing to term 2
I20260129 02:59:25.209120 20511 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a0fcc58d6fb6473cafbd9d34d5195269" candidate_uuid: "197b880fa39d432a9b472845876bfaf0" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "974b802141bd4f13beffef2aa03f88f9"
I20260129 02:59:25.209761 20511 raft_consensus.cc:3055] T a0fcc58d6fb6473cafbd9d34d5195269 P 974b802141bd4f13beffef2aa03f88f9 [term 1 LEADER]: Stepping down as leader of term 1
I20260129 02:59:25.210168 20511 raft_consensus.cc:740] T a0fcc58d6fb6473cafbd9d34d5195269 P 974b802141bd4f13beffef2aa03f88f9 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 974b802141bd4f13beffef2aa03f88f9, State: Running, Role: LEADER
I20260129 02:59:25.205427 20586 raft_consensus.cc:515] T 4b1876f9a70b49edb3aa74105e66d3b3 P 1c50de89a5a845668696dcb28c3badd6 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:59:25.211072 20511 consensus_queue.cc:260] T a0fcc58d6fb6473cafbd9d34d5195269 P 974b802141bd4f13beffef2aa03f88f9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:59:25.124718 21995 raft_consensus.cc:515] T 78aa39096868415d9afc2ffcb248e198 P 1c50de89a5a845668696dcb28c3badd6 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } }
I20260129 02:59:25.217487 20586 leader_election.cc:290] T 4b1876f9a70b49edb3aa74105e66d3b3 P 1c50de89a5a845668696dcb28c3badd6 [CANDIDATE]: Term 3 election: Requested vote from peers 197b880fa39d432a9b472845876bfaf0 (127.18.211.1:42027), 974b802141bd4f13beffef2aa03f88f9 (127.18.211.2:34517)
I20260129 02:59:25.220764 20440 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "cedce331e1584ac8b636c9b8a72b2ba0" candidate_uuid: "1c50de89a5a845668696dcb28c3badd6" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "197b880fa39d432a9b472845876bfaf0"
I20260129 02:59:25.221587 20440 raft_consensus.cc:3060] T cedce331e1584ac8b636c9b8a72b2ba0 P 197b880fa39d432a9b472845876bfaf0 [term 3 FOLLOWER]: Advancing to term 4
I20260129 02:59:25.224444 20511 raft_consensus.cc:3060] T a0fcc58d6fb6473cafbd9d34d5195269 P 974b802141bd4f13beffef2aa03f88f9 [term 1 FOLLOWER]: Advancing to term 2
I20260129 02:59:25.187474 20433 tablet_service.cc:2038] Received Run Leader Election RPC: tablet_id: "a4c1226b987b468fa84dfc6ba0e2dbe1"
dest_uuid: "197b880fa39d432a9b472845876bfaf0"
from {username='slave'} at 127.0.0.1:55672
I20260129 02:59:25.212045 21883 raft_consensus.cc:2804] T 1299723a4476420ab02fd8bfd7aa0645 P 197b880fa39d432a9b472845876bfaf0 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260129 02:59:25.230683 20581 raft_consensus.cc:515] T 76f713c3331540d2a5b76164103e9eb1 P 1c50de89a5a845668696dcb28c3badd6 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:59:25.229261 20508 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4b1876f9a70b49edb3aa74105e66d3b3" candidate_uuid: "1c50de89a5a845668696dcb28c3badd6" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "974b802141bd4f13beffef2aa03f88f9"
I20260129 02:59:25.233069 20508 raft_consensus.cc:3060] T 4b1876f9a70b49edb3aa74105e66d3b3 P 974b802141bd4f13beffef2aa03f88f9 [term 2 FOLLOWER]: Advancing to term 3
I20260129 02:59:25.235540 21944 raft_consensus.cc:493] T 42d27a28d89b409e8807ec0efd97af69 P 1c50de89a5a845668696dcb28c3badd6 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 7a9daf5d42a84615a6383cfec9a10608)
I20260129 02:59:25.235862 20432 leader_election.cc:290] T a0fcc58d6fb6473cafbd9d34d5195269 P 197b880fa39d432a9b472845876bfaf0 [CANDIDATE]: Term 2 election: Requested vote from peers 974b802141bd4f13beffef2aa03f88f9 (127.18.211.2:34517), 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3:37735)
I20260129 02:59:25.237306 21944 raft_consensus.cc:515] T 42d27a28d89b409e8807ec0efd97af69 P 1c50de89a5a845668696dcb28c3badd6 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } }
I20260129 02:59:25.239408 21944 leader_election.cc:290] T 42d27a28d89b409e8807ec0efd97af69 P 1c50de89a5a845668696dcb28c3badd6 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 197b880fa39d432a9b472845876bfaf0 (127.18.211.1:42027), 7a9daf5d42a84615a6383cfec9a10608 (127.18.211.4:40439)
W20260129 02:59:25.210465 21099 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.211.1:42027, user_credentials={real_user=slave}} blocked reactor thread for 65281.9us
I20260129 02:59:25.242395 20581 leader_election.cc:290] T 76f713c3331540d2a5b76164103e9eb1 P 1c50de89a5a845668696dcb28c3badd6 [CANDIDATE]: Term 2 election: Requested vote from peers 197b880fa39d432a9b472845876bfaf0 (127.18.211.1:42027), 974b802141bd4f13beffef2aa03f88f9 (127.18.211.2:34517)
I20260129 02:59:25.250658 20583 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a0fcc58d6fb6473cafbd9d34d5195269" candidate_uuid: "197b880fa39d432a9b472845876bfaf0" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "1c50de89a5a845668696dcb28c3badd6"
I20260129 02:59:25.251379 20583 raft_consensus.cc:3060] T a0fcc58d6fb6473cafbd9d34d5195269 P 1c50de89a5a845668696dcb28c3badd6 [term 1 FOLLOWER]: Advancing to term 2
I20260129 02:59:25.243337 20510 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "76f713c3331540d2a5b76164103e9eb1" candidate_uuid: "1c50de89a5a845668696dcb28c3badd6" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "974b802141bd4f13beffef2aa03f88f9"
I20260129 02:59:25.256371 20510 raft_consensus.cc:3055] T 76f713c3331540d2a5b76164103e9eb1 P 974b802141bd4f13beffef2aa03f88f9 [term 1 LEADER]: Stepping down as leader of term 1
I20260129 02:59:25.256778 20510 raft_consensus.cc:740] T 76f713c3331540d2a5b76164103e9eb1 P 974b802141bd4f13beffef2aa03f88f9 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 974b802141bd4f13beffef2aa03f88f9, State: Running, Role: LEADER
I20260129 02:59:25.257692 20510 consensus_queue.cc:260] T 76f713c3331540d2a5b76164103e9eb1 P 974b802141bd4f13beffef2aa03f88f9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:59:25.259311 20510 raft_consensus.cc:3060] T 76f713c3331540d2a5b76164103e9eb1 P 974b802141bd4f13beffef2aa03f88f9 [term 1 FOLLOWER]: Advancing to term 2
I20260129 02:59:25.260478 20432 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "42d27a28d89b409e8807ec0efd97af69" candidate_uuid: "1c50de89a5a845668696dcb28c3badd6" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "197b880fa39d432a9b472845876bfaf0" is_pre_election: true
I20260129 02:59:25.261579 20432 raft_consensus.cc:2393] T 42d27a28d89b409e8807ec0efd97af69 P 197b880fa39d432a9b472845876bfaf0 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1c50de89a5a845668696dcb28c3badd6 in current term 3: Already voted for candidate 197b880fa39d432a9b472845876bfaf0 in this term.
I20260129 02:59:25.263449 22004 raft_consensus.cc:493] T e33a417cf9c54cf5af73a4ee3aac4d3a P 1c50de89a5a845668696dcb28c3badd6 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260129 02:59:25.264012 22004 raft_consensus.cc:515] T e33a417cf9c54cf5af73a4ee3aac4d3a P 1c50de89a5a845668696dcb28c3badd6 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:59:25.245712 21995 leader_election.cc:290] T 78aa39096868415d9afc2ffcb248e198 P 1c50de89a5a845668696dcb28c3badd6 [CANDIDATE]: Term 3 election: Requested vote from peers 197b880fa39d432a9b472845876bfaf0 (127.18.211.1:42027), 7a9daf5d42a84615a6383cfec9a10608 (127.18.211.4:40439)
I20260129 02:59:25.246565 21974 raft_consensus.cc:993] T 7d9494f5584d4d5db8b9adf1c8fc888b P 974b802141bd4f13beffef2aa03f88f9: : Instructing follower 1c50de89a5a845668696dcb28c3badd6 to start an election
I20260129 02:59:25.247396 21988 leader_election.cc:290] T cedce331e1584ac8b636c9b8a72b2ba0 P 1c50de89a5a845668696dcb28c3badd6 [CANDIDATE]: Term 4 election: Requested vote from peers 197b880fa39d432a9b472845876bfaf0 (127.18.211.1:42027), 974b802141bd4f13beffef2aa03f88f9 (127.18.211.2:34517)
I20260129 02:59:25.249135 21129 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "78aa39096868415d9afc2ffcb248e198" candidate_uuid: "1c50de89a5a845668696dcb28c3badd6" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "7a9daf5d42a84615a6383cfec9a10608"
I20260129 02:59:25.249640 20507 raft_consensus.cc:2468] T 0678b8b468bd43cdbc3e485900844821 P 974b802141bd4f13beffef2aa03f88f9 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7a9daf5d42a84615a6383cfec9a10608 in term 4.
I20260129 02:59:25.249987 21132 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "42d27a28d89b409e8807ec0efd97af69" candidate_uuid: "1c50de89a5a845668696dcb28c3badd6" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "7a9daf5d42a84615a6383cfec9a10608" is_pre_election: true
I20260129 02:59:25.282253 21102 leader_election.cc:304] T 0678b8b468bd43cdbc3e485900844821 P 7a9daf5d42a84615a6383cfec9a10608 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7a9daf5d42a84615a6383cfec9a10608, 974b802141bd4f13beffef2aa03f88f9; no voters:
I20260129 02:59:25.268514 20508 raft_consensus.cc:2468] T 4b1876f9a70b49edb3aa74105e66d3b3 P 974b802141bd4f13beffef2aa03f88f9 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1c50de89a5a845668696dcb28c3badd6 in term 3.
I20260129 02:59:25.269707 20516 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "cedce331e1584ac8b636c9b8a72b2ba0" candidate_uuid: "1c50de89a5a845668696dcb28c3badd6" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "974b802141bd4f13beffef2aa03f88f9"
I20260129 02:59:25.272755 20440 raft_consensus.cc:2468] T cedce331e1584ac8b636c9b8a72b2ba0 P 197b880fa39d432a9b472845876bfaf0 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1c50de89a5a845668696dcb28c3badd6 in term 4.
I20260129 02:59:25.228869 20433 raft_consensus.cc:493] T a4c1226b987b468fa84dfc6ba0e2dbe1 P 197b880fa39d432a9b472845876bfaf0 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260129 02:59:25.289575 21129 raft_consensus.cc:3060] T 78aa39096868415d9afc2ffcb248e198 P 7a9daf5d42a84615a6383cfec9a10608 [term 2 FOLLOWER]: Advancing to term 3
I20260129 02:59:25.290123 20439 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "76f713c3331540d2a5b76164103e9eb1" candidate_uuid: "1c50de89a5a845668696dcb28c3badd6" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "197b880fa39d432a9b472845876bfaf0"
I20260129 02:59:25.290253 22042 raft_consensus.cc:515] T b189f08e16a74a11980cc55d99359a0a P 197b880fa39d432a9b472845876bfaf0 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:59:25.300616 20439 raft_consensus.cc:3060] T 76f713c3331540d2a5b76164103e9eb1 P 197b880fa39d432a9b472845876bfaf0 [term 1 FOLLOWER]: Advancing to term 2
I20260129 02:59:25.290594 20512 raft_consensus.cc:2468] T 45cc5f3d2688486d9b6dbe7e666dedde P 974b802141bd4f13beffef2aa03f88f9 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1c50de89a5a845668696dcb28c3badd6 in term 3.
I20260129 02:59:25.294759 20516 raft_consensus.cc:3060] T cedce331e1584ac8b636c9b8a72b2ba0 P 974b802141bd4f13beffef2aa03f88f9 [term 3 FOLLOWER]: Advancing to term 4
I20260129 02:59:25.295632 20556 leader_election.cc:304] T 4b1876f9a70b49edb3aa74105e66d3b3 P 1c50de89a5a845668696dcb28c3badd6 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1c50de89a5a845668696dcb28c3badd6, 974b802141bd4f13beffef2aa03f88f9; no voters:
I20260129 02:59:25.294423 20436 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "78aa39096868415d9afc2ffcb248e198" candidate_uuid: "1c50de89a5a845668696dcb28c3badd6" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "197b880fa39d432a9b472845876bfaf0"
I20260129 02:59:25.295632 20553 leader_election.cc:304] T cedce331e1584ac8b636c9b8a72b2ba0 P 1c50de89a5a845668696dcb28c3badd6 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 197b880fa39d432a9b472845876bfaf0, 1c50de89a5a845668696dcb28c3badd6; no voters:
I20260129 02:59:25.294018 20437 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "926a250fa5834e88b96ccecbefbf5086" candidate_uuid: "1c50de89a5a845668696dcb28c3badd6" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "197b880fa39d432a9b472845876bfaf0"
I20260129 02:59:25.280773 21913 raft_consensus.cc:1081] T 7d9494f5584d4d5db8b9adf1c8fc888b P 974b802141bd4f13beffef2aa03f88f9 [term 1 LEADER]: Signalling peer 1c50de89a5a845668696dcb28c3badd6 to start an election
I20260129 02:59:25.303718 22037 raft_consensus.cc:2804] T 4b1876f9a70b49edb3aa74105e66d3b3 P 1c50de89a5a845668696dcb28c3badd6 [term 3 FOLLOWER]: Leader election won for term 3
I20260129 02:59:25.304142 22049 raft_consensus.cc:493] T 96609ac68bdf4993a12e1c23355ca013 P 7a9daf5d42a84615a6383cfec9a10608 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 197b880fa39d432a9b472845876bfaf0)
I20260129 02:59:25.305848 22042 leader_election.cc:290] T b189f08e16a74a11980cc55d99359a0a P 197b880fa39d432a9b472845876bfaf0 [CANDIDATE]: Term 4 election: Requested vote from peers 974b802141bd4f13beffef2aa03f88f9 (127.18.211.2:34517), 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3:37735)
I20260129 02:59:25.321012 20511 raft_consensus.cc:2468] T a0fcc58d6fb6473cafbd9d34d5195269 P 974b802141bd4f13beffef2aa03f88f9 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 197b880fa39d432a9b472845876bfaf0 in term 2.
I20260129 02:59:25.306201 20433 raft_consensus.cc:3060] T a4c1226b987b468fa84dfc6ba0e2dbe1 P 197b880fa39d432a9b472845876bfaf0 [term 1 FOLLOWER]: Advancing to term 2
I20260129 02:59:25.307160 20556 leader_election.cc:304] T 45cc5f3d2688486d9b6dbe7e666dedde P 1c50de89a5a845668696dcb28c3badd6 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1c50de89a5a845668696dcb28c3badd6, 974b802141bd4f13beffef2aa03f88f9; no voters:
I20260129 02:59:25.320931 22049 raft_consensus.cc:515] T 96609ac68bdf4993a12e1c23355ca013 P 7a9daf5d42a84615a6383cfec9a10608 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } attrs { replace: true } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } }
I20260129 02:59:25.322036 21988 raft_consensus.cc:2804] T 45cc5f3d2688486d9b6dbe7e666dedde P 1c50de89a5a845668696dcb28c3badd6 [term 3 FOLLOWER]: Leader election won for term 3
I20260129 02:59:25.322523 21988 raft_consensus.cc:697] T 45cc5f3d2688486d9b6dbe7e666dedde P 1c50de89a5a845668696dcb28c3badd6 [term 3 LEADER]: Becoming Leader. State: Replica: 1c50de89a5a845668696dcb28c3badd6, State: Running, Role: LEADER
I20260129 02:59:25.323429 21988 consensus_queue.cc:237] T 45cc5f3d2688486d9b6dbe7e666dedde P 1c50de89a5a845668696dcb28c3badd6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } }
I20260129 02:59:25.318459 21984 raft_consensus.cc:2804] T cedce331e1584ac8b636c9b8a72b2ba0 P 1c50de89a5a845668696dcb28c3badd6 [term 4 FOLLOWER]: Leader election won for term 4
I20260129 02:59:25.307194 20434 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4b1876f9a70b49edb3aa74105e66d3b3" candidate_uuid: "1c50de89a5a845668696dcb28c3badd6" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "197b880fa39d432a9b472845876bfaf0"
I20260129 02:59:25.312489 21132 raft_consensus.cc:2468] T 42d27a28d89b409e8807ec0efd97af69 P 7a9daf5d42a84615a6383cfec9a10608 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1c50de89a5a845668696dcb28c3badd6 in term 3.
I20260129 02:59:25.325852 20434 raft_consensus.cc:3055] T 4b1876f9a70b49edb3aa74105e66d3b3 P 197b880fa39d432a9b472845876bfaf0 [term 2 LEADER]: Stepping down as leader of term 2
I20260129 02:59:25.326301 20434 raft_consensus.cc:740] T 4b1876f9a70b49edb3aa74105e66d3b3 P 197b880fa39d432a9b472845876bfaf0 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 197b880fa39d432a9b472845876bfaf0, State: Running, Role: LEADER
I20260129 02:59:25.326578 22037 raft_consensus.cc:697] T 4b1876f9a70b49edb3aa74105e66d3b3 P 1c50de89a5a845668696dcb28c3badd6 [term 3 LEADER]: Becoming Leader. State: Replica: 1c50de89a5a845668696dcb28c3badd6, State: Running, Role: LEADER
I20260129 02:59:25.327302 20407 leader_election.cc:304] T a0fcc58d6fb6473cafbd9d34d5195269 P 197b880fa39d432a9b472845876bfaf0 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 197b880fa39d432a9b472845876bfaf0, 974b802141bd4f13beffef2aa03f88f9; no voters:
I20260129 02:59:25.315519 20510 raft_consensus.cc:2468] T 76f713c3331540d2a5b76164103e9eb1 P 974b802141bd4f13beffef2aa03f88f9 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1c50de89a5a845668696dcb28c3badd6 in term 2.
I20260129 02:59:25.327590 22037 consensus_queue.cc:237] T 4b1876f9a70b49edb3aa74105e66d3b3 P 1c50de89a5a845668696dcb28c3badd6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:59:25.328155 22011 raft_consensus.cc:493] T 96609ac68bdf4993a12e1c23355ca013 P 197b880fa39d432a9b472845876bfaf0 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260129 02:59:25.328845 20581 tablet_service.cc:2038] Received Run Leader Election RPC: tablet_id: "7d9494f5584d4d5db8b9adf1c8fc888b"
dest_uuid: "1c50de89a5a845668696dcb28c3badd6"
from {username='slave'} at 127.0.0.1:35752
I20260129 02:59:25.329491 20581 raft_consensus.cc:493] T 7d9494f5584d4d5db8b9adf1c8fc888b P 1c50de89a5a845668696dcb28c3badd6 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260129 02:59:25.329867 20581 raft_consensus.cc:3060] T 7d9494f5584d4d5db8b9adf1c8fc888b P 1c50de89a5a845668696dcb28c3badd6 [term 1 FOLLOWER]: Advancing to term 2
I20260129 02:59:25.317081 20437 raft_consensus.cc:3060] T 926a250fa5834e88b96ccecbefbf5086 P 197b880fa39d432a9b472845876bfaf0 [term 3 FOLLOWER]: Advancing to term 4
I20260129 02:59:25.332316 20441 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a5a05fbcbbcf4916bbc01fd1a5b5e90d" candidate_uuid: "1c50de89a5a845668696dcb28c3badd6" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "197b880fa39d432a9b472845876bfaf0"
I20260129 02:59:25.330451 20436 raft_consensus.cc:3060] T 78aa39096868415d9afc2ffcb248e198 P 197b880fa39d432a9b472845876bfaf0 [term 1 FOLLOWER]: Advancing to term 3
I20260129 02:59:25.332913 20441 raft_consensus.cc:3060] T a5a05fbcbbcf4916bbc01fd1a5b5e90d P 197b880fa39d432a9b472845876bfaf0 [term 3 FOLLOWER]: Advancing to term 4
I20260129 02:59:25.303617 22050 raft_consensus.cc:2804] T 0678b8b468bd43cdbc3e485900844821 P 7a9daf5d42a84615a6383cfec9a10608 [term 4 FOLLOWER]: Leader election won for term 4
I20260129 02:59:25.325275 21984 raft_consensus.cc:697] T cedce331e1584ac8b636c9b8a72b2ba0 P 1c50de89a5a845668696dcb28c3badd6 [term 4 LEADER]: Becoming Leader. State: Replica: 1c50de89a5a845668696dcb28c3badd6, State: Running, Role: LEADER
W20260129 02:59:25.274075 21100 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.211.3:37735, user_credentials={real_user=slave}} blocked reactor thread for 97113.5us
I20260129 02:59:25.332866 20556 leader_election.cc:304] T 76f713c3331540d2a5b76164103e9eb1 P 1c50de89a5a845668696dcb28c3badd6 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1c50de89a5a845668696dcb28c3badd6, 974b802141bd4f13beffef2aa03f88f9; no voters:
I20260129 02:59:25.320013 20516 raft_consensus.cc:2468] T cedce331e1584ac8b636c9b8a72b2ba0 P 974b802141bd4f13beffef2aa03f88f9 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1c50de89a5a845668696dcb28c3badd6 in term 4.
I20260129 02:59:25.334246 20514 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b189f08e16a74a11980cc55d99359a0a" candidate_uuid: "197b880fa39d432a9b472845876bfaf0" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "974b802141bd4f13beffef2aa03f88f9"
I20260129 02:59:25.343137 20514 raft_consensus.cc:3060] T b189f08e16a74a11980cc55d99359a0a P 974b802141bd4f13beffef2aa03f88f9 [term 3 FOLLOWER]: Advancing to term 4
I20260129 02:59:25.334275 20511 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "96609ac68bdf4993a12e1c23355ca013" candidate_uuid: "7a9daf5d42a84615a6383cfec9a10608" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "974b802141bd4f13beffef2aa03f88f9" is_pre_election: true
I20260129 02:59:25.313762 21883 raft_consensus.cc:493] T 1299723a4476420ab02fd8bfd7aa0645 P 197b880fa39d432a9b472845876bfaf0 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260129 02:59:25.336741 20586 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b189f08e16a74a11980cc55d99359a0a" candidate_uuid: "197b880fa39d432a9b472845876bfaf0" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "1c50de89a5a845668696dcb28c3badd6"
I20260129 02:59:25.327136 20434 consensus_queue.cc:260] T 4b1876f9a70b49edb3aa74105e66d3b3 P 197b880fa39d432a9b472845876bfaf0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:59:25.295575 20432 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0678b8b468bd43cdbc3e485900844821" candidate_uuid: "7a9daf5d42a84615a6383cfec9a10608" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "197b880fa39d432a9b472845876bfaf0"
I20260129 02:59:25.299386 20583 raft_consensus.cc:2468] T a0fcc58d6fb6473cafbd9d34d5195269 P 1c50de89a5a845668696dcb28c3badd6 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 197b880fa39d432a9b472845876bfaf0 in term 2.
I20260129 02:59:25.340030 22002 raft_consensus.cc:493] T 87d5fe48a44a408aaf03e894a67ecb2f P 974b802141bd4f13beffef2aa03f88f9 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 7a9daf5d42a84615a6383cfec9a10608)
I20260129 02:59:25.341782 21995 raft_consensus.cc:2804] T 76f713c3331540d2a5b76164103e9eb1 P 1c50de89a5a845668696dcb28c3badd6 [term 2 FOLLOWER]: Leader election won for term 2
I20260129 02:59:25.340057 22050 raft_consensus.cc:697] T 0678b8b468bd43cdbc3e485900844821 P 7a9daf5d42a84615a6383cfec9a10608 [term 4 LEADER]: Becoming Leader. State: Replica: 7a9daf5d42a84615a6383cfec9a10608, State: Running, Role: LEADER
I20260129 02:59:25.348495 21883 raft_consensus.cc:3060] T 1299723a4476420ab02fd8bfd7aa0645 P 197b880fa39d432a9b472845876bfaf0 [term 3 FOLLOWER]: Advancing to term 4
I20260129 02:59:25.353410 20434 raft_consensus.cc:3060] T 4b1876f9a70b49edb3aa74105e66d3b3 P 197b880fa39d432a9b472845876bfaf0 [term 2 FOLLOWER]: Advancing to term 3
I20260129 02:59:25.356532 22033 raft_consensus.cc:2749] T 45cc5f3d2688486d9b6dbe7e666dedde P 7a9daf5d42a84615a6383cfec9a10608 [term 3 FOLLOWER]: Leader election lost for term 2. Reason: Vote denied by peer 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3:37735) with higher term. Message: Invalid argument: T 45cc5f3d2688486d9b6dbe7e666dedde P 1c50de89a5a845668696dcb28c3badd6 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 7a9daf5d42a84615a6383cfec9a10608 for earlier term 2. Current term is 3.
I20260129 02:59:25.328601 22011 raft_consensus.cc:515] T 96609ac68bdf4993a12e1c23355ca013 P 197b880fa39d432a9b472845876bfaf0 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } attrs { replace: true } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } }
I20260129 02:59:25.353215 20432 raft_consensus.cc:3060] T 0678b8b468bd43cdbc3e485900844821 P 197b880fa39d432a9b472845876bfaf0 [term 3 FOLLOWER]: Advancing to term 4
I20260129 02:59:25.363472 22050 consensus_queue.cc:237] T 0678b8b468bd43cdbc3e485900844821 P 7a9daf5d42a84615a6383cfec9a10608 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } }
I20260129 02:59:25.365869 20441 raft_consensus.cc:2468] T a5a05fbcbbcf4916bbc01fd1a5b5e90d P 197b880fa39d432a9b472845876bfaf0 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1c50de89a5a845668696dcb28c3badd6 in term 4.
I20260129 02:59:25.330098 21129 raft_consensus.cc:2468] T 78aa39096868415d9afc2ffcb248e198 P 7a9daf5d42a84615a6383cfec9a10608 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1c50de89a5a845668696dcb28c3badd6 in term 3.
I20260129 02:59:25.323721 22049 leader_election.cc:290] T 96609ac68bdf4993a12e1c23355ca013 P 7a9daf5d42a84615a6383cfec9a10608 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 197b880fa39d432a9b472845876bfaf0 (127.18.211.1:42027), 974b802141bd4f13beffef2aa03f88f9 (127.18.211.2:34517), 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3:37735)
I20260129 02:59:25.379518 20432 raft_consensus.cc:2468] T 0678b8b468bd43cdbc3e485900844821 P 197b880fa39d432a9b472845876bfaf0 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7a9daf5d42a84615a6383cfec9a10608 in term 4.
I20260129 02:59:25.345916 20581 raft_consensus.cc:515] T 7d9494f5584d4d5db8b9adf1c8fc888b P 1c50de89a5a845668696dcb28c3badd6 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:59:25.333695 21787 raft_consensus.cc:2804] T a0fcc58d6fb6473cafbd9d34d5195269 P 197b880fa39d432a9b472845876bfaf0 [term 2 FOLLOWER]: Leader election won for term 2
I20260129 02:59:25.366449 21944 raft_consensus.cc:493] T 1299723a4476420ab02fd8bfd7aa0645 P 1c50de89a5a845668696dcb28c3badd6 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 197b880fa39d432a9b472845876bfaf0)
I20260129 02:59:25.367120 22002 raft_consensus.cc:515] T 87d5fe48a44a408aaf03e894a67ecb2f P 974b802141bd4f13beffef2aa03f88f9 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } }
I20260129 02:59:25.346918 20586 raft_consensus.cc:3060] T b189f08e16a74a11980cc55d99359a0a P 1c50de89a5a845668696dcb28c3badd6 [term 2 FOLLOWER]: Advancing to term 4
I20260129 02:59:25.369441 20553 leader_election.cc:304] T a5a05fbcbbcf4916bbc01fd1a5b5e90d P 1c50de89a5a845668696dcb28c3badd6 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 197b880fa39d432a9b472845876bfaf0, 1c50de89a5a845668696dcb28c3badd6; no voters:
I20260129 02:59:25.348873 22035 leader_election.cc:290] T 926a250fa5834e88b96ccecbefbf5086 P 1c50de89a5a845668696dcb28c3badd6 [CANDIDATE]: Term 4 election: Requested vote from peers 197b880fa39d432a9b472845876bfaf0 (127.18.211.1:42027), 974b802141bd4f13beffef2aa03f88f9 (127.18.211.2:34517)
I20260129 02:59:25.380422 20584 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "96609ac68bdf4993a12e1c23355ca013" candidate_uuid: "197b880fa39d432a9b472845876bfaf0" candidate_term: 4 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "1c50de89a5a845668696dcb28c3badd6" is_pre_election: true
I20260129 02:59:25.388008 20584 raft_consensus.cc:2468] T 96609ac68bdf4993a12e1c23355ca013 P 1c50de89a5a845668696dcb28c3badd6 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 197b880fa39d432a9b472845876bfaf0 in term 3.
I20260129 02:59:25.388810 20510 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a5a05fbcbbcf4916bbc01fd1a5b5e90d" candidate_uuid: "1c50de89a5a845668696dcb28c3badd6" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "974b802141bd4f13beffef2aa03f88f9"
I20260129 02:59:25.389531 20510 raft_consensus.cc:3060] T a5a05fbcbbcf4916bbc01fd1a5b5e90d P 974b802141bd4f13beffef2aa03f88f9 [term 3 FOLLOWER]: Advancing to term 4
I20260129 02:59:25.387171 20434 raft_consensus.cc:2468] T 4b1876f9a70b49edb3aa74105e66d3b3 P 197b880fa39d432a9b472845876bfaf0 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1c50de89a5a845668696dcb28c3badd6 in term 3.
I20260129 02:59:25.396198 22035 raft_consensus.cc:2804] T a5a05fbcbbcf4916bbc01fd1a5b5e90d P 1c50de89a5a845668696dcb28c3badd6 [term 4 FOLLOWER]: Leader election won for term 4
I20260129 02:59:25.396672 22035 raft_consensus.cc:697] T a5a05fbcbbcf4916bbc01fd1a5b5e90d P 1c50de89a5a845668696dcb28c3badd6 [term 4 LEADER]: Becoming Leader. State: Replica: 1c50de89a5a845668696dcb28c3badd6, State: Running, Role: LEADER
I20260129 02:59:25.397482 22035 consensus_queue.cc:237] T a5a05fbcbbcf4916bbc01fd1a5b5e90d P 1c50de89a5a845668696dcb28c3badd6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:59:25.398411 20510 raft_consensus.cc:2468] T a5a05fbcbbcf4916bbc01fd1a5b5e90d P 974b802141bd4f13beffef2aa03f88f9 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1c50de89a5a845668696dcb28c3badd6 in term 4.
I20260129 02:59:25.402228 20584 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "96609ac68bdf4993a12e1c23355ca013" candidate_uuid: "7a9daf5d42a84615a6383cfec9a10608" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "1c50de89a5a845668696dcb28c3badd6" is_pre_election: true
I20260129 02:59:25.404528 20584 raft_consensus.cc:2393] T 96609ac68bdf4993a12e1c23355ca013 P 1c50de89a5a845668696dcb28c3badd6 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 7a9daf5d42a84615a6383cfec9a10608 in current term 3: Already voted for candidate 1c50de89a5a845668696dcb28c3badd6 in this term.
I20260129 02:59:25.404515 20512 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "96609ac68bdf4993a12e1c23355ca013" candidate_uuid: "197b880fa39d432a9b472845876bfaf0" candidate_term: 4 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "974b802141bd4f13beffef2aa03f88f9" is_pre_election: true
I20260129 02:59:25.404430 20554 leader_election.cc:304] T 42d27a28d89b409e8807ec0efd97af69 P 1c50de89a5a845668696dcb28c3badd6 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1c50de89a5a845668696dcb28c3badd6, 7a9daf5d42a84615a6383cfec9a10608; no voters: 197b880fa39d432a9b472845876bfaf0
I20260129 02:59:25.381891 20581 leader_election.cc:290] T 7d9494f5584d4d5db8b9adf1c8fc888b P 1c50de89a5a845668696dcb28c3badd6 [CANDIDATE]: Term 2 election: Requested vote from peers 197b880fa39d432a9b472845876bfaf0 (127.18.211.1:42027), 974b802141bd4f13beffef2aa03f88f9 (127.18.211.2:34517)
I20260129 02:59:25.348191 20516 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "926a250fa5834e88b96ccecbefbf5086" candidate_uuid: "1c50de89a5a845668696dcb28c3badd6" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "974b802141bd4f13beffef2aa03f88f9"
I20260129 02:59:25.382371 21134 raft_consensus.cc:2468] T 45cc5f3d2688486d9b6dbe7e666dedde P 7a9daf5d42a84615a6383cfec9a10608 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1c50de89a5a845668696dcb28c3badd6 in term 3.
I20260129 02:59:25.386380 20508 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7d9494f5584d4d5db8b9adf1c8fc888b" candidate_uuid: "1c50de89a5a845668696dcb28c3badd6" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "974b802141bd4f13beffef2aa03f88f9"
I20260129 02:59:25.377236 21995 raft_consensus.cc:697] T 76f713c3331540d2a5b76164103e9eb1 P 1c50de89a5a845668696dcb28c3badd6 [term 2 LEADER]: Becoming Leader. State: Replica: 1c50de89a5a845668696dcb28c3badd6, State: Running, Role: LEADER
I20260129 02:59:25.407580 20516 raft_consensus.cc:3060] T 926a250fa5834e88b96ccecbefbf5086 P 974b802141bd4f13beffef2aa03f88f9 [term 3 FOLLOWER]: Advancing to term 4
I20260129 02:59:25.407645 20508 raft_consensus.cc:3055] T 7d9494f5584d4d5db8b9adf1c8fc888b P 974b802141bd4f13beffef2aa03f88f9 [term 1 LEADER]: Stepping down as leader of term 1
I20260129 02:59:25.410468 20508 raft_consensus.cc:740] T 7d9494f5584d4d5db8b9adf1c8fc888b P 974b802141bd4f13beffef2aa03f88f9 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 974b802141bd4f13beffef2aa03f88f9, State: Running, Role: LEADER
I20260129 02:59:25.403381 20586 raft_consensus.cc:2468] T b189f08e16a74a11980cc55d99359a0a P 1c50de89a5a845668696dcb28c3badd6 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 197b880fa39d432a9b472845876bfaf0 in term 4.
I20260129 02:59:25.384613 22005 leader_election.cc:290] T a5a05fbcbbcf4916bbc01fd1a5b5e90d P 1c50de89a5a845668696dcb28c3badd6 [CANDIDATE]: Term 4 election: Requested vote from peers 197b880fa39d432a9b472845876bfaf0 (127.18.211.1:42027), 974b802141bd4f13beffef2aa03f88f9 (127.18.211.2:34517)
W20260129 02:59:25.418320 20553 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.211.1:42027, user_credentials={real_user=slave}} blocked reactor thread for 48980.6us
I20260129 02:59:25.377120 22011 leader_election.cc:290] T 96609ac68bdf4993a12e1c23355ca013 P 197b880fa39d432a9b472845876bfaf0 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 974b802141bd4f13beffef2aa03f88f9 (127.18.211.2:34517), 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3:37735), 7a9daf5d42a84615a6383cfec9a10608 (127.18.211.4:40439)
I20260129 02:59:25.411504 20508 consensus_queue.cc:260] T 7d9494f5584d4d5db8b9adf1c8fc888b P 974b802141bd4f13beffef2aa03f88f9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:59:25.351429 22023 raft_consensus.cc:515] T 87d5fe48a44a408aaf03e894a67ecb2f P 7a9daf5d42a84615a6383cfec9a10608 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } }
I20260129 02:59:25.427934 20405 leader_election.cc:304] T b189f08e16a74a11980cc55d99359a0a P 197b880fa39d432a9b472845876bfaf0 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 197b880fa39d432a9b472845876bfaf0, 1c50de89a5a845668696dcb28c3badd6; no voters:
I20260129 02:59:25.428789 22011 raft_consensus.cc:2804] T b189f08e16a74a11980cc55d99359a0a P 197b880fa39d432a9b472845876bfaf0 [term 4 FOLLOWER]: Leader election won for term 4
I20260129 02:59:25.429309 22011 raft_consensus.cc:697] T b189f08e16a74a11980cc55d99359a0a P 197b880fa39d432a9b472845876bfaf0 [term 4 LEADER]: Becoming Leader. State: Replica: 197b880fa39d432a9b472845876bfaf0, State: Running, Role: LEADER
I20260129 02:59:25.429723 22004 leader_election.cc:290] T e33a417cf9c54cf5af73a4ee3aac4d3a P 1c50de89a5a845668696dcb28c3badd6 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 197b880fa39d432a9b472845876bfaf0 (127.18.211.1:42027), 974b802141bd4f13beffef2aa03f88f9 (127.18.211.2:34517)
I20260129 02:59:25.409188 21787 raft_consensus.cc:697] T a0fcc58d6fb6473cafbd9d34d5195269 P 197b880fa39d432a9b472845876bfaf0 [term 2 LEADER]: Becoming Leader. State: Replica: 197b880fa39d432a9b472845876bfaf0, State: Running, Role: LEADER
I20260129 02:59:25.430227 22011 consensus_queue.cc:237] T b189f08e16a74a11980cc55d99359a0a P 197b880fa39d432a9b472845876bfaf0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:59:25.431541 21787 consensus_queue.cc:237] T a0fcc58d6fb6473cafbd9d34d5195269 P 197b880fa39d432a9b472845876bfaf0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:59:25.425947 22043 raft_consensus.cc:493] T 96609ac68bdf4993a12e1c23355ca013 P 974b802141bd4f13beffef2aa03f88f9 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 197b880fa39d432a9b472845876bfaf0)
I20260129 02:59:25.434924 20512 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "87d5fe48a44a408aaf03e894a67ecb2f" candidate_uuid: "7a9daf5d42a84615a6383cfec9a10608" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "974b802141bd4f13beffef2aa03f88f9"
I20260129 02:59:25.435075 22043 raft_consensus.cc:515] T 96609ac68bdf4993a12e1c23355ca013 P 974b802141bd4f13beffef2aa03f88f9 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } attrs { replace: true } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } }
I20260129 02:59:25.436084 22041 raft_consensus.cc:493] T 75be4bd6bd45474795fd472eff2953ac P 974b802141bd4f13beffef2aa03f88f9 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260129 02:59:25.436564 22041 raft_consensus.cc:515] T 75be4bd6bd45474795fd472eff2953ac P 974b802141bd4f13beffef2aa03f88f9 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:59:25.437765 22043 leader_election.cc:290] T 96609ac68bdf4993a12e1c23355ca013 P 974b802141bd4f13beffef2aa03f88f9 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 197b880fa39d432a9b472845876bfaf0 (127.18.211.1:42027), 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3:37735), 7a9daf5d42a84615a6383cfec9a10608 (127.18.211.4:40439)
I20260129 02:59:25.435527 20512 raft_consensus.cc:3060] T 87d5fe48a44a408aaf03e894a67ecb2f P 974b802141bd4f13beffef2aa03f88f9 [term 2 FOLLOWER]: Advancing to term 4
I20260129 02:59:25.439407 20510 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e33a417cf9c54cf5af73a4ee3aac4d3a" candidate_uuid: "1c50de89a5a845668696dcb28c3badd6" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "974b802141bd4f13beffef2aa03f88f9" is_pre_election: true
I20260129 02:59:25.440040 20510 raft_consensus.cc:2468] T e33a417cf9c54cf5af73a4ee3aac4d3a P 974b802141bd4f13beffef2aa03f88f9 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1c50de89a5a845668696dcb28c3badd6 in term 2.
I20260129 02:59:25.441149 20556 leader_election.cc:304] T e33a417cf9c54cf5af73a4ee3aac4d3a P 1c50de89a5a845668696dcb28c3badd6 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1c50de89a5a845668696dcb28c3badd6, 974b802141bd4f13beffef2aa03f88f9; no voters:
I20260129 02:59:25.441939 22004 raft_consensus.cc:2804] T e33a417cf9c54cf5af73a4ee3aac4d3a P 1c50de89a5a845668696dcb28c3badd6 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260129 02:59:25.441361 21995 consensus_queue.cc:237] T 76f713c3331540d2a5b76164103e9eb1 P 1c50de89a5a845668696dcb28c3badd6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:59:25.442302 22004 raft_consensus.cc:493] T e33a417cf9c54cf5af73a4ee3aac4d3a P 1c50de89a5a845668696dcb28c3badd6 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260129 02:59:25.442665 22004 raft_consensus.cc:3060] T e33a417cf9c54cf5af73a4ee3aac4d3a P 1c50de89a5a845668696dcb28c3badd6 [term 2 FOLLOWER]: Advancing to term 3
I20260129 02:59:25.445484 20432 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "96609ac68bdf4993a12e1c23355ca013" candidate_uuid: "974b802141bd4f13beffef2aa03f88f9" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "197b880fa39d432a9b472845876bfaf0" is_pre_election: true
I20260129 02:59:25.446122 20432 raft_consensus.cc:2468] T 96609ac68bdf4993a12e1c23355ca013 P 197b880fa39d432a9b472845876bfaf0 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 974b802141bd4f13beffef2aa03f88f9 in term 3.
W20260129 02:59:25.385677 21099 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.211.1:42027, user_credentials={real_user=slave}} blocked reactor thread for 93201.8us
I20260129 02:59:25.422981 21971 raft_consensus.cc:2804] T 42d27a28d89b409e8807ec0efd97af69 P 1c50de89a5a845668696dcb28c3badd6 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260129 02:59:25.450582 21971 raft_consensus.cc:493] T 42d27a28d89b409e8807ec0efd97af69 P 1c50de89a5a845668696dcb28c3badd6 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 7a9daf5d42a84615a6383cfec9a10608)
I20260129 02:59:25.451066 21971 raft_consensus.cc:3060] T 42d27a28d89b409e8807ec0efd97af69 P 1c50de89a5a845668696dcb28c3badd6 [term 2 FOLLOWER]: Advancing to term 3
I20260129 02:59:25.453382 20438 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "87d5fe48a44a408aaf03e894a67ecb2f" candidate_uuid: "7a9daf5d42a84615a6383cfec9a10608" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "197b880fa39d432a9b472845876bfaf0"
I20260129 02:59:25.429327 22023 leader_election.cc:290] T 87d5fe48a44a408aaf03e894a67ecb2f P 7a9daf5d42a84615a6383cfec9a10608 [CANDIDATE]: Term 4 election: Requested vote from peers 197b880fa39d432a9b472845876bfaf0 (127.18.211.1:42027), 974b802141bd4f13beffef2aa03f88f9 (127.18.211.2:34517)
I20260129 02:59:25.454000 20438 raft_consensus.cc:3060] T 87d5fe48a44a408aaf03e894a67ecb2f P 197b880fa39d432a9b472845876bfaf0 [term 3 FOLLOWER]: Advancing to term 4
I20260129 02:59:25.459954 20432 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "96609ac68bdf4993a12e1c23355ca013" candidate_uuid: "7a9daf5d42a84615a6383cfec9a10608" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "197b880fa39d432a9b472845876bfaf0" is_pre_election: true
I20260129 02:59:25.460604 20432 raft_consensus.cc:2468] T 96609ac68bdf4993a12e1c23355ca013 P 197b880fa39d432a9b472845876bfaf0 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7a9daf5d42a84615a6383cfec9a10608 in term 3.
I20260129 02:59:25.462181 22041 leader_election.cc:290] T 75be4bd6bd45474795fd472eff2953ac P 974b802141bd4f13beffef2aa03f88f9 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 197b880fa39d432a9b472845876bfaf0 (127.18.211.1:42027), 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3:37735)
I20260129 02:59:25.404207 21944 raft_consensus.cc:515] T 1299723a4476420ab02fd8bfd7aa0645 P 1c50de89a5a845668696dcb28c3badd6 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } }
I20260129 02:59:25.467401 20438 raft_consensus.cc:2468] T 87d5fe48a44a408aaf03e894a67ecb2f P 197b880fa39d432a9b472845876bfaf0 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7a9daf5d42a84615a6383cfec9a10608 in term 4.
I20260129 02:59:25.468312 20432 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "75be4bd6bd45474795fd472eff2953ac" candidate_uuid: "974b802141bd4f13beffef2aa03f88f9" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "197b880fa39d432a9b472845876bfaf0" is_pre_election: true
I20260129 02:59:25.469115 20432 raft_consensus.cc:2468] T 75be4bd6bd45474795fd472eff2953ac P 197b880fa39d432a9b472845876bfaf0 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 974b802141bd4f13beffef2aa03f88f9 in term 5.
I20260129 02:59:25.475241 20512 raft_consensus.cc:2468] T 87d5fe48a44a408aaf03e894a67ecb2f P 974b802141bd4f13beffef2aa03f88f9 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7a9daf5d42a84615a6383cfec9a10608 in term 4.
I20260129 02:59:25.476903 21102 leader_election.cc:304] T 87d5fe48a44a408aaf03e894a67ecb2f P 7a9daf5d42a84615a6383cfec9a10608 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7a9daf5d42a84615a6383cfec9a10608, 974b802141bd4f13beffef2aa03f88f9; no voters:
I20260129 02:59:25.492002 20516 raft_consensus.cc:2468] T 926a250fa5834e88b96ccecbefbf5086 P 974b802141bd4f13beffef2aa03f88f9 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1c50de89a5a845668696dcb28c3badd6 in term 4.
I20260129 02:59:25.493487 20556 leader_election.cc:304] T 926a250fa5834e88b96ccecbefbf5086 P 1c50de89a5a845668696dcb28c3badd6 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1c50de89a5a845668696dcb28c3badd6, 974b802141bd4f13beffef2aa03f88f9; no voters:
I20260129 02:59:25.493633 20479 leader_election.cc:304] T 75be4bd6bd45474795fd472eff2953ac P 974b802141bd4f13beffef2aa03f88f9 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 197b880fa39d432a9b472845876bfaf0, 974b802141bd4f13beffef2aa03f88f9; no voters:
I20260129 02:59:25.494427 20508 raft_consensus.cc:3060] T 7d9494f5584d4d5db8b9adf1c8fc888b P 974b802141bd4f13beffef2aa03f88f9 [term 1 FOLLOWER]: Advancing to term 2
I20260129 02:59:25.495709 20439 raft_consensus.cc:2468] T 76f713c3331540d2a5b76164103e9eb1 P 197b880fa39d432a9b472845876bfaf0 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1c50de89a5a845668696dcb28c3badd6 in term 2.
I20260129 02:59:25.497314 20437 raft_consensus.cc:2468] T 926a250fa5834e88b96ccecbefbf5086 P 197b880fa39d432a9b472845876bfaf0 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1c50de89a5a845668696dcb28c3badd6 in term 4.
I20260129 02:59:25.499238 20439 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e33a417cf9c54cf5af73a4ee3aac4d3a" candidate_uuid: "1c50de89a5a845668696dcb28c3badd6" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "197b880fa39d432a9b472845876bfaf0" is_pre_election: true
I20260129 02:59:25.499745 20586 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "96609ac68bdf4993a12e1c23355ca013" candidate_uuid: "974b802141bd4f13beffef2aa03f88f9" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "1c50de89a5a845668696dcb28c3badd6" is_pre_election: true
I20260129 02:59:25.500463 20439 raft_consensus.cc:2468] T e33a417cf9c54cf5af73a4ee3aac4d3a P 197b880fa39d432a9b472845876bfaf0 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1c50de89a5a845668696dcb28c3badd6 in term 2.
I20260129 02:59:25.500769 20586 raft_consensus.cc:2393] T 96609ac68bdf4993a12e1c23355ca013 P 1c50de89a5a845668696dcb28c3badd6 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 974b802141bd4f13beffef2aa03f88f9 in current term 3: Already voted for candidate 1c50de89a5a845668696dcb28c3badd6 in this term.
I20260129 02:59:25.498574 20584 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "75be4bd6bd45474795fd472eff2953ac" candidate_uuid: "974b802141bd4f13beffef2aa03f88f9" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "1c50de89a5a845668696dcb28c3badd6" is_pre_election: true
I20260129 02:59:25.506283 22041 raft_consensus.cc:493] T 1ce4ddf5b10e4ae8b193b13b2e5936ee P 974b802141bd4f13beffef2aa03f88f9 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260129 02:59:25.510725 22041 raft_consensus.cc:515] T 1ce4ddf5b10e4ae8b193b13b2e5936ee P 974b802141bd4f13beffef2aa03f88f9 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:59:25.515066 22041 leader_election.cc:290] T 1ce4ddf5b10e4ae8b193b13b2e5936ee P 974b802141bd4f13beffef2aa03f88f9 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 197b880fa39d432a9b472845876bfaf0 (127.18.211.1:42027), 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3:37735)
I20260129 02:59:25.515142 20437 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1ce4ddf5b10e4ae8b193b13b2e5936ee" candidate_uuid: "974b802141bd4f13beffef2aa03f88f9" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "197b880fa39d432a9b472845876bfaf0" is_pre_election: true
I20260129 02:59:25.516404 20437 raft_consensus.cc:2468] T 1ce4ddf5b10e4ae8b193b13b2e5936ee P 197b880fa39d432a9b472845876bfaf0 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 974b802141bd4f13beffef2aa03f88f9 in term 2.
I20260129 02:59:25.524427 20508 raft_consensus.cc:2468] T 7d9494f5584d4d5db8b9adf1c8fc888b P 974b802141bd4f13beffef2aa03f88f9 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1c50de89a5a845668696dcb28c3badd6 in term 2.
I20260129 02:59:25.527820 20479 leader_election.cc:304] T 1ce4ddf5b10e4ae8b193b13b2e5936ee P 974b802141bd4f13beffef2aa03f88f9 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 197b880fa39d432a9b472845876bfaf0, 974b802141bd4f13beffef2aa03f88f9; no voters:
I20260129 02:59:25.507126 22028 raft_consensus.cc:2804] T 75be4bd6bd45474795fd472eff2953ac P 974b802141bd4f13beffef2aa03f88f9 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260129 02:59:25.506211 20434 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7d9494f5584d4d5db8b9adf1c8fc888b" candidate_uuid: "1c50de89a5a845668696dcb28c3badd6" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "197b880fa39d432a9b472845876bfaf0"
W20260129 02:59:25.525521 21099 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.211.1:42027, user_credentials={real_user=slave}} blocked reactor thread for 75965us
I20260129 02:59:25.516736 22053 raft_consensus.cc:2804] T 87d5fe48a44a408aaf03e894a67ecb2f P 7a9daf5d42a84615a6383cfec9a10608 [term 4 FOLLOWER]: Leader election won for term 4
I20260129 02:59:25.542198 20584 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1ce4ddf5b10e4ae8b193b13b2e5936ee" candidate_uuid: "974b802141bd4f13beffef2aa03f88f9" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1c50de89a5a845668696dcb28c3badd6" is_pre_election: true
I20260129 02:59:25.417596 21984 consensus_queue.cc:237] T cedce331e1584ac8b636c9b8a72b2ba0 P 1c50de89a5a845668696dcb28c3badd6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:59:25.533767 21974 raft_consensus.cc:2804] T 1ce4ddf5b10e4ae8b193b13b2e5936ee P 974b802141bd4f13beffef2aa03f88f9 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260129 02:59:25.532445 22028 raft_consensus.cc:493] T 75be4bd6bd45474795fd472eff2953ac P 974b802141bd4f13beffef2aa03f88f9 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260129 02:59:25.548837 20556 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.211.2:34517, user_credentials={real_user=slave}} blocked reactor thread for 55435us
I20260129 02:59:25.552307 20556 leader_election.cc:304] T 7d9494f5584d4d5db8b9adf1c8fc888b P 1c50de89a5a845668696dcb28c3badd6 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1c50de89a5a845668696dcb28c3badd6, 974b802141bd4f13beffef2aa03f88f9; no voters:
I20260129 02:59:25.545156 20434 raft_consensus.cc:3060] T 7d9494f5584d4d5db8b9adf1c8fc888b P 197b880fa39d432a9b472845876bfaf0 [term 1 FOLLOWER]: Advancing to term 2
I20260129 02:59:25.554679 21991 raft_consensus.cc:2804] T 7d9494f5584d4d5db8b9adf1c8fc888b P 1c50de89a5a845668696dcb28c3badd6 [term 2 FOLLOWER]: Leader election won for term 2
I20260129 02:59:25.555176 21991 raft_consensus.cc:697] T 7d9494f5584d4d5db8b9adf1c8fc888b P 1c50de89a5a845668696dcb28c3badd6 [term 2 LEADER]: Becoming Leader. State: Replica: 1c50de89a5a845668696dcb28c3badd6, State: Running, Role: LEADER
I20260129 02:59:25.557139 22028 raft_consensus.cc:3060] T 75be4bd6bd45474795fd472eff2953ac P 974b802141bd4f13beffef2aa03f88f9 [term 5 FOLLOWER]: Advancing to term 6
I20260129 02:59:25.560120 21913 raft_consensus.cc:493] T 368953c9102643b2a21f99af36b51905 P 974b802141bd4f13beffef2aa03f88f9 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 197b880fa39d432a9b472845876bfaf0)
I20260129 02:59:25.560648 21913 raft_consensus.cc:515] T 368953c9102643b2a21f99af36b51905 P 974b802141bd4f13beffef2aa03f88f9 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:59:25.562621 21913 leader_election.cc:290] T 368953c9102643b2a21f99af36b51905 P 974b802141bd4f13beffef2aa03f88f9 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 197b880fa39d432a9b472845876bfaf0 (127.18.211.1:42027), 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3:37735)
I20260129 02:59:25.564791 20437 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "368953c9102643b2a21f99af36b51905" candidate_uuid: "974b802141bd4f13beffef2aa03f88f9" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "197b880fa39d432a9b472845876bfaf0" is_pre_election: true
I20260129 02:59:25.564595 21991 consensus_queue.cc:237] T 7d9494f5584d4d5db8b9adf1c8fc888b P 1c50de89a5a845668696dcb28c3badd6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:59:25.565358 20437 raft_consensus.cc:2468] T 368953c9102643b2a21f99af36b51905 P 197b880fa39d432a9b472845876bfaf0 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 974b802141bd4f13beffef2aa03f88f9 in term 3.
I20260129 02:59:25.566592 20479 leader_election.cc:304] T 368953c9102643b2a21f99af36b51905 P 974b802141bd4f13beffef2aa03f88f9 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 197b880fa39d432a9b472845876bfaf0, 974b802141bd4f13beffef2aa03f88f9; no voters:
I20260129 02:59:25.567718 22036 raft_consensus.cc:2804] T 368953c9102643b2a21f99af36b51905 P 974b802141bd4f13beffef2aa03f88f9 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260129 02:59:25.568145 22036 raft_consensus.cc:493] T 368953c9102643b2a21f99af36b51905 P 974b802141bd4f13beffef2aa03f88f9 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 197b880fa39d432a9b472845876bfaf0)
I20260129 02:59:25.508147 21100 leader_election.cc:304] T 96609ac68bdf4993a12e1c23355ca013 P 7a9daf5d42a84615a6383cfec9a10608 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 7a9daf5d42a84615a6383cfec9a10608; no voters: 1c50de89a5a845668696dcb28c3badd6, 974b802141bd4f13beffef2aa03f88f9
I20260129 02:59:25.538576 21988 raft_consensus.cc:2804] T 926a250fa5834e88b96ccecbefbf5086 P 1c50de89a5a845668696dcb28c3badd6 [term 4 FOLLOWER]: Leader election won for term 4
I20260129 02:59:25.571354 20583 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "368953c9102643b2a21f99af36b51905" candidate_uuid: "974b802141bd4f13beffef2aa03f88f9" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "1c50de89a5a845668696dcb28c3badd6" is_pre_election: true
I20260129 02:59:25.574216 20583 raft_consensus.cc:2393] T 368953c9102643b2a21f99af36b51905 P 1c50de89a5a845668696dcb28c3badd6 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 974b802141bd4f13beffef2aa03f88f9 in current term 3: Already voted for candidate 1c50de89a5a845668696dcb28c3badd6 in this term.
I20260129 02:59:25.576141 20437 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "87d5fe48a44a408aaf03e894a67ecb2f" candidate_uuid: "974b802141bd4f13beffef2aa03f88f9" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "197b880fa39d432a9b472845876bfaf0" is_pre_election: true
I20260129 02:59:25.577140 20437 raft_consensus.cc:2368] T 87d5fe48a44a408aaf03e894a67ecb2f P 197b880fa39d432a9b472845876bfaf0 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 974b802141bd4f13beffef2aa03f88f9 for earlier term 3. Current term is 4.
I20260129 02:59:25.579685 20479 leader_election.cc:400] T 87d5fe48a44a408aaf03e894a67ecb2f P 974b802141bd4f13beffef2aa03f88f9 [CANDIDATE]: Term 3 pre-election: Vote denied by peer 197b880fa39d432a9b472845876bfaf0 (127.18.211.1:42027) with higher term. Message: Invalid argument: T 87d5fe48a44a408aaf03e894a67ecb2f P 197b880fa39d432a9b472845876bfaf0 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 974b802141bd4f13beffef2aa03f88f9 for earlier term 3. Current term is 4.
I20260129 02:59:25.580112 20479 leader_election.cc:403] T 87d5fe48a44a408aaf03e894a67ecb2f P 974b802141bd4f13beffef2aa03f88f9 [CANDIDATE]: Term 3 pre-election: Cancelling election due to peer responding with higher term
I20260129 02:59:25.568554 22036 raft_consensus.cc:3060] T 368953c9102643b2a21f99af36b51905 P 974b802141bd4f13beffef2aa03f88f9 [term 2 FOLLOWER]: Advancing to term 3
I20260129 02:59:25.549467 21974 raft_consensus.cc:493] T 1ce4ddf5b10e4ae8b193b13b2e5936ee P 974b802141bd4f13beffef2aa03f88f9 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260129 02:59:25.549249 22053 raft_consensus.cc:697] T 87d5fe48a44a408aaf03e894a67ecb2f P 7a9daf5d42a84615a6383cfec9a10608 [term 4 LEADER]: Becoming Leader. State: Replica: 7a9daf5d42a84615a6383cfec9a10608, State: Running, Role: LEADER
W20260129 02:59:25.630929 20480 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.211.3:37735, user_credentials={real_user=slave}} blocked reactor thread for 45200us
I20260129 02:59:25.583158 22033 raft_consensus.cc:493] T dc2f4cb752cf4d59b5f6bc689f5d8104 P 7a9daf5d42a84615a6383cfec9a10608 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260129 02:59:25.631803 22059 raft_consensus.cc:3060] T 96609ac68bdf4993a12e1c23355ca013 P 7a9daf5d42a84615a6383cfec9a10608 [term 2 FOLLOWER]: Advancing to term 3
I20260129 02:59:25.632800 21988 raft_consensus.cc:697] T 926a250fa5834e88b96ccecbefbf5086 P 1c50de89a5a845668696dcb28c3badd6 [term 4 LEADER]: Becoming Leader. State: Replica: 1c50de89a5a845668696dcb28c3badd6, State: Running, Role: LEADER
W20260129 02:59:25.629242 21100 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.211.3:37735, user_credentials={real_user=slave}} blocked reactor thread for 131803us
I20260129 02:59:25.658185 21974 raft_consensus.cc:3060] T 1ce4ddf5b10e4ae8b193b13b2e5936ee P 974b802141bd4f13beffef2aa03f88f9 [term 2 FOLLOWER]: Advancing to term 3
I20260129 02:59:25.632900 21913 raft_consensus.cc:2749] T 87d5fe48a44a408aaf03e894a67ecb2f P 974b802141bd4f13beffef2aa03f88f9 [term 4 FOLLOWER]: Leader pre-election lost for term 3. Reason: Vote denied by peer 197b880fa39d432a9b472845876bfaf0 (127.18.211.1:42027) with higher term. Message: Invalid argument: T 87d5fe48a44a408aaf03e894a67ecb2f P 197b880fa39d432a9b472845876bfaf0 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 974b802141bd4f13beffef2aa03f88f9 for earlier term 3. Current term is 4.
I20260129 02:59:25.722296 20437 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1299723a4476420ab02fd8bfd7aa0645" candidate_uuid: "1c50de89a5a845668696dcb28c3badd6" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "197b880fa39d432a9b472845876bfaf0" is_pre_election: true
I20260129 02:59:25.662454 21944 leader_election.cc:290] T 1299723a4476420ab02fd8bfd7aa0645 P 1c50de89a5a845668696dcb28c3badd6 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 197b880fa39d432a9b472845876bfaf0 (127.18.211.1:42027), 7a9daf5d42a84615a6383cfec9a10608 (127.18.211.4:40439)
I20260129 02:59:25.704442 22002 leader_election.cc:290] T 87d5fe48a44a408aaf03e894a67ecb2f P 974b802141bd4f13beffef2aa03f88f9 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 197b880fa39d432a9b472845876bfaf0 (127.18.211.1:42027), 7a9daf5d42a84615a6383cfec9a10608 (127.18.211.4:40439)
I20260129 02:59:25.633882 21883 raft_consensus.cc:515] T 1299723a4476420ab02fd8bfd7aa0645 P 197b880fa39d432a9b472845876bfaf0 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } }
I20260129 02:59:25.747064 22053 consensus_queue.cc:237] T 87d5fe48a44a408aaf03e894a67ecb2f P 7a9daf5d42a84615a6383cfec9a10608 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } }
I20260129 02:59:25.760290 20554 leader_election.cc:304] T 78aa39096868415d9afc2ffcb248e198 P 1c50de89a5a845668696dcb28c3badd6 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1c50de89a5a845668696dcb28c3badd6, 7a9daf5d42a84615a6383cfec9a10608; no voters:
I20260129 02:59:25.663017 20433 raft_consensus.cc:515] T a4c1226b987b468fa84dfc6ba0e2dbe1 P 197b880fa39d432a9b472845876bfaf0 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:59:25.762955 20514 raft_consensus.cc:2468] T b189f08e16a74a11980cc55d99359a0a P 974b802141bd4f13beffef2aa03f88f9 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 197b880fa39d432a9b472845876bfaf0 in term 4.
W20260129 02:59:25.737538 21099 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.211.1:42027, user_credentials={real_user=slave}} blocked reactor thread for 107300us
I20260129 02:59:25.769758 20437 raft_consensus.cc:2368] T 1299723a4476420ab02fd8bfd7aa0645 P 197b880fa39d432a9b472845876bfaf0 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1c50de89a5a845668696dcb28c3badd6 for earlier term 3. Current term is 4.
I20260129 02:59:25.706334 22033 raft_consensus.cc:515] T dc2f4cb752cf4d59b5f6bc689f5d8104 P 7a9daf5d42a84615a6383cfec9a10608 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } }
I20260129 02:59:25.743211 20434 raft_consensus.cc:2468] T 7d9494f5584d4d5db8b9adf1c8fc888b P 197b880fa39d432a9b472845876bfaf0 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1c50de89a5a845668696dcb28c3badd6 in term 2.
I20260129 02:59:25.771503 21128 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "96609ac68bdf4993a12e1c23355ca013" candidate_uuid: "974b802141bd4f13beffef2aa03f88f9" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "7a9daf5d42a84615a6383cfec9a10608" is_pre_election: true
I20260129 02:59:25.779196 22044 raft_consensus.cc:493] T 1299723a4476420ab02fd8bfd7aa0645 P 974b802141bd4f13beffef2aa03f88f9 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 197b880fa39d432a9b472845876bfaf0)
I20260129 02:59:25.788029 20439 raft_consensus.cc:1240] T a4c1226b987b468fa84dfc6ba0e2dbe1 P 197b880fa39d432a9b472845876bfaf0 [term 2 FOLLOWER]: Rejecting Update request from peer 974b802141bd4f13beffef2aa03f88f9 for earlier term 1. Current term is 2. Ops: []
I20260129 02:59:25.789369 21131 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1299723a4476420ab02fd8bfd7aa0645" candidate_uuid: "1c50de89a5a845668696dcb28c3badd6" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "7a9daf5d42a84615a6383cfec9a10608" is_pre_election: true
I20260129 02:59:25.790405 21131 raft_consensus.cc:2393] T 1299723a4476420ab02fd8bfd7aa0645 P 7a9daf5d42a84615a6383cfec9a10608 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1c50de89a5a845668696dcb28c3badd6 in current term 3: Already voted for candidate 7a9daf5d42a84615a6383cfec9a10608 in this term.
I20260129 02:59:25.744658 21971 raft_consensus.cc:515] T 42d27a28d89b409e8807ec0efd97af69 P 1c50de89a5a845668696dcb28c3badd6 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } }
I20260129 02:59:25.763263 21913 raft_consensus.cc:476] T 505580c93c694bbb8ea96fdf9b17ab46 P 974b802141bd4f13beffef2aa03f88f9 [term 2 LEADER]: Not starting pre-election -- already a leader
I20260129 02:59:25.753793 22004 raft_consensus.cc:515] T e33a417cf9c54cf5af73a4ee3aac4d3a P 1c50de89a5a845668696dcb28c3badd6 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:59:25.809844 21131 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "87d5fe48a44a408aaf03e894a67ecb2f" candidate_uuid: "974b802141bd4f13beffef2aa03f88f9" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "7a9daf5d42a84615a6383cfec9a10608" is_pre_election: true
I20260129 02:59:25.773931 22066 raft_consensus.cc:2804] T 78aa39096868415d9afc2ffcb248e198 P 1c50de89a5a845668696dcb28c3badd6 [term 3 FOLLOWER]: Leader election won for term 3
I20260129 02:59:25.821733 21971 leader_election.cc:290] T 42d27a28d89b409e8807ec0efd97af69 P 1c50de89a5a845668696dcb28c3badd6 [CANDIDATE]: Term 3 election: Requested vote from peers 197b880fa39d432a9b472845876bfaf0 (127.18.211.1:42027), 7a9daf5d42a84615a6383cfec9a10608 (127.18.211.4:40439)
I20260129 02:59:25.827124 20432 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "42d27a28d89b409e8807ec0efd97af69" candidate_uuid: "1c50de89a5a845668696dcb28c3badd6" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "197b880fa39d432a9b472845876bfaf0"
I20260129 02:59:25.827610 20438 raft_consensus.cc:1275] T 1ce4ddf5b10e4ae8b193b13b2e5936ee P 197b880fa39d432a9b472845876bfaf0 [term 2 FOLLOWER]: Refusing update from remote peer 1c50de89a5a845668696dcb28c3badd6: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260129 02:59:25.827916 20432 raft_consensus.cc:2393] T 42d27a28d89b409e8807ec0efd97af69 P 197b880fa39d432a9b472845876bfaf0 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 1c50de89a5a845668696dcb28c3badd6 in current term 3: Already voted for candidate 197b880fa39d432a9b472845876bfaf0 in this term.
I20260129 02:59:25.840179 20434 raft_consensus.cc:1275] T 75be4bd6bd45474795fd472eff2953ac P 197b880fa39d432a9b472845876bfaf0 [term 5 FOLLOWER]: Refusing update from remote peer 1c50de89a5a845668696dcb28c3badd6: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 5 index: 4. (index mismatch)
I20260129 02:59:25.844053 22064 consensus_queue.cc:1048] T 1ce4ddf5b10e4ae8b193b13b2e5936ee P 1c50de89a5a845668696dcb28c3badd6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260129 02:59:25.844883 20433 leader_election.cc:290] T a4c1226b987b468fa84dfc6ba0e2dbe1 P 197b880fa39d432a9b472845876bfaf0 [CANDIDATE]: Term 2 election: Requested vote from peers 974b802141bd4f13beffef2aa03f88f9 (127.18.211.2:34517), 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3:37735)
I20260129 02:59:25.845767 22066 raft_consensus.cc:697] T 78aa39096868415d9afc2ffcb248e198 P 1c50de89a5a845668696dcb28c3badd6 [term 3 LEADER]: Becoming Leader. State: Replica: 1c50de89a5a845668696dcb28c3badd6, State: Running, Role: LEADER
W20260129 02:59:25.846419 22055 log.cc:927] Time spent T 0678b8b468bd43cdbc3e485900844821 P 7a9daf5d42a84615a6383cfec9a10608: Append to log took a long time: real 0.134s user 0.000s sys 0.001s
I20260129 02:59:25.847393 22004 leader_election.cc:290] T e33a417cf9c54cf5af73a4ee3aac4d3a P 1c50de89a5a845668696dcb28c3badd6 [CANDIDATE]: Term 3 election: Requested vote from peers 197b880fa39d432a9b472845876bfaf0 (127.18.211.1:42027), 974b802141bd4f13beffef2aa03f88f9 (127.18.211.2:34517)
I20260129 02:59:25.847028 22066 consensus_queue.cc:237] T 78aa39096868415d9afc2ffcb248e198 P 1c50de89a5a845668696dcb28c3badd6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } }
I20260129 02:59:25.847718 22033 leader_election.cc:290] T dc2f4cb752cf4d59b5f6bc689f5d8104 P 7a9daf5d42a84615a6383cfec9a10608 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 197b880fa39d432a9b472845876bfaf0 (127.18.211.1:42027), 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3:37735)
W20260129 02:59:25.833874 22058 log.cc:927] Time spent T 45cc5f3d2688486d9b6dbe7e666dedde P 1c50de89a5a845668696dcb28c3badd6: Append to log took a long time: real 0.156s user 0.000s sys 0.001s
I20260129 02:59:25.849637 20583 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a4c1226b987b468fa84dfc6ba0e2dbe1" candidate_uuid: "197b880fa39d432a9b472845876bfaf0" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "1c50de89a5a845668696dcb28c3badd6"
I20260129 02:59:25.850246 20583 raft_consensus.cc:3060] T a4c1226b987b468fa84dfc6ba0e2dbe1 P 1c50de89a5a845668696dcb28c3badd6 [term 1 FOLLOWER]: Advancing to term 2
W20260129 02:59:25.837260 21099 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.211.1:42027, user_credentials={real_user=slave}} blocked reactor thread for 45863.2us
I20260129 02:59:25.853844 20514 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e33a417cf9c54cf5af73a4ee3aac4d3a" candidate_uuid: "1c50de89a5a845668696dcb28c3badd6" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "974b802141bd4f13beffef2aa03f88f9"
I20260129 02:59:25.854557 20514 raft_consensus.cc:3060] T e33a417cf9c54cf5af73a4ee3aac4d3a P 974b802141bd4f13beffef2aa03f88f9 [term 2 FOLLOWER]: Advancing to term 3
W20260129 02:59:25.863322 22057 log.cc:927] Time spent T a0fcc58d6fb6473cafbd9d34d5195269 P 197b880fa39d432a9b472845876bfaf0: Append to log took a long time: real 0.310s user 0.000s sys 0.001s
I20260129 02:59:25.864809 21883 leader_election.cc:290] T 1299723a4476420ab02fd8bfd7aa0645 P 197b880fa39d432a9b472845876bfaf0 [CANDIDATE]: Term 4 election: Requested vote from peers 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3:37735), 7a9daf5d42a84615a6383cfec9a10608 (127.18.211.4:40439)
I20260129 02:59:25.865780 20584 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1299723a4476420ab02fd8bfd7aa0645" candidate_uuid: "197b880fa39d432a9b472845876bfaf0" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "1c50de89a5a845668696dcb28c3badd6"
I20260129 02:59:25.865917 21136 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1299723a4476420ab02fd8bfd7aa0645" candidate_uuid: "197b880fa39d432a9b472845876bfaf0" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "7a9daf5d42a84615a6383cfec9a10608"
I20260129 02:59:25.866369 20584 raft_consensus.cc:3060] T 1299723a4476420ab02fd8bfd7aa0645 P 1c50de89a5a845668696dcb28c3badd6 [term 2 FOLLOWER]: Advancing to term 4
I20260129 02:59:25.866518 21136 raft_consensus.cc:3060] T 1299723a4476420ab02fd8bfd7aa0645 P 7a9daf5d42a84615a6383cfec9a10608 [term 3 FOLLOWER]: Advancing to term 4
I20260129 02:59:25.797626 22044 raft_consensus.cc:515] T 1299723a4476420ab02fd8bfd7aa0645 P 974b802141bd4f13beffef2aa03f88f9 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } attrs { replace: true } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } }
I20260129 02:59:25.795225 21988 consensus_queue.cc:237] T 926a250fa5834e88b96ccecbefbf5086 P 1c50de89a5a845668696dcb28c3badd6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
W20260129 02:59:25.882953 20436 consensus_meta.cc:287] T 78aa39096868415d9afc2ffcb248e198 P 197b880fa39d432a9b472845876bfaf0: Time spent flushing consensus metadata: real 0.550s user 0.009s sys 0.001s
I20260129 02:59:25.883421 20436 raft_consensus.cc:2468] T 78aa39096868415d9afc2ffcb248e198 P 197b880fa39d432a9b472845876bfaf0 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1c50de89a5a845668696dcb28c3badd6 in term 3.
I20260129 02:59:25.885614 20440 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1299723a4476420ab02fd8bfd7aa0645" candidate_uuid: "974b802141bd4f13beffef2aa03f88f9" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "197b880fa39d432a9b472845876bfaf0" is_pre_election: true
I20260129 02:59:25.886265 20440 raft_consensus.cc:1813] T 1299723a4476420ab02fd8bfd7aa0645 P 197b880fa39d432a9b472845876bfaf0 [term 4 FOLLOWER]: Handling vote request from an unknown peer 974b802141bd4f13beffef2aa03f88f9
I20260129 02:59:25.886747 20440 raft_consensus.cc:2368] T 1299723a4476420ab02fd8bfd7aa0645 P 197b880fa39d432a9b472845876bfaf0 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 974b802141bd4f13beffef2aa03f88f9 for earlier term 3. Current term is 4.
W20260129 02:59:25.887037 22060 log.cc:927] Time spent T 1ce4ddf5b10e4ae8b193b13b2e5936ee P 1c50de89a5a845668696dcb28c3badd6: Append to log took a long time: real 0.333s user 0.001s sys 0.000s
I20260129 02:59:25.890689 20508 raft_consensus.cc:1275] T 45cc5f3d2688486d9b6dbe7e666dedde P 974b802141bd4f13beffef2aa03f88f9 [term 3 FOLLOWER]: Refusing update from remote peer 1c50de89a5a845668696dcb28c3badd6: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 3 index: 5. (index mismatch)
I20260129 02:59:25.891687 20439 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e33a417cf9c54cf5af73a4ee3aac4d3a" candidate_uuid: "1c50de89a5a845668696dcb28c3badd6" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "197b880fa39d432a9b472845876bfaf0"
I20260129 02:59:25.892349 20553 leader_election.cc:400] T 1299723a4476420ab02fd8bfd7aa0645 P 1c50de89a5a845668696dcb28c3badd6 [CANDIDATE]: Term 3 pre-election: Vote denied by peer 197b880fa39d432a9b472845876bfaf0 (127.18.211.1:42027) with higher term. Message: Invalid argument: T 1299723a4476420ab02fd8bfd7aa0645 P 197b880fa39d432a9b472845876bfaf0 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1c50de89a5a845668696dcb28c3badd6 for earlier term 3. Current term is 4.
I20260129 02:59:25.892647 20553 leader_election.cc:403] T 1299723a4476420ab02fd8bfd7aa0645 P 1c50de89a5a845668696dcb28c3badd6 [CANDIDATE]: Term 3 pre-election: Cancelling election due to peer responding with higher term
I20260129 02:59:25.893144 22061 consensus_queue.cc:1048] T 75be4bd6bd45474795fd472eff2953ac P 1c50de89a5a845668696dcb28c3badd6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260129 02:59:25.893249 21991 consensus_queue.cc:1048] T 45cc5f3d2688486d9b6dbe7e666dedde P 1c50de89a5a845668696dcb28c3badd6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260129 02:59:25.892269 20439 raft_consensus.cc:3060] T e33a417cf9c54cf5af73a4ee3aac4d3a P 197b880fa39d432a9b472845876bfaf0 [term 2 FOLLOWER]: Advancing to term 3
I20260129 02:59:25.897034 20514 raft_consensus.cc:2468] T e33a417cf9c54cf5af73a4ee3aac4d3a P 974b802141bd4f13beffef2aa03f88f9 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1c50de89a5a845668696dcb28c3badd6 in term 3.
I20260129 02:59:25.824771 22028 raft_consensus.cc:515] T 75be4bd6bd45474795fd472eff2953ac P 974b802141bd4f13beffef2aa03f88f9 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:59:25.892184 20433 raft_consensus.cc:1275] T 76f713c3331540d2a5b76164103e9eb1 P 197b880fa39d432a9b472845876bfaf0 [term 2 FOLLOWER]: Refusing update from remote peer 1c50de89a5a845668696dcb28c3badd6: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260129 02:59:25.906195 21974 raft_consensus.cc:515] T 1ce4ddf5b10e4ae8b193b13b2e5936ee P 974b802141bd4f13beffef2aa03f88f9 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:59:25.908005 20479 leader_election.cc:400] T 1299723a4476420ab02fd8bfd7aa0645 P 974b802141bd4f13beffef2aa03f88f9 [CANDIDATE]: Term 3 pre-election: Vote denied by peer 197b880fa39d432a9b472845876bfaf0 (127.18.211.1:42027) with higher term. Message: Invalid argument: T 1299723a4476420ab02fd8bfd7aa0645 P 197b880fa39d432a9b472845876bfaf0 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 974b802141bd4f13beffef2aa03f88f9 for earlier term 3. Current term is 4.
I20260129 02:59:25.908340 20479 leader_election.cc:403] T 1299723a4476420ab02fd8bfd7aa0645 P 974b802141bd4f13beffef2aa03f88f9 [CANDIDATE]: Term 3 pre-election: Cancelling election due to peer responding with higher term
I20260129 02:59:25.909396 21994 consensus_queue.cc:1059] T a4c1226b987b468fa84dfc6ba0e2dbe1 P 974b802141bd4f13beffef2aa03f88f9 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260129 02:59:25.910127 21971 consensus_queue.cc:1048] T 76f713c3331540d2a5b76164103e9eb1 P 1c50de89a5a845668696dcb28c3badd6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260129 02:59:25.910921 21994 raft_consensus.cc:3055] T a4c1226b987b468fa84dfc6ba0e2dbe1 P 974b802141bd4f13beffef2aa03f88f9 [term 1 LEADER]: Stepping down as leader of term 1
I20260129 02:59:25.910429 22022 raft_consensus.cc:3060] T 1299723a4476420ab02fd8bfd7aa0645 P 974b802141bd4f13beffef2aa03f88f9 [term 2 FOLLOWER]: Advancing to term 4
I20260129 02:59:25.911350 21994 raft_consensus.cc:740] T a4c1226b987b468fa84dfc6ba0e2dbe1 P 974b802141bd4f13beffef2aa03f88f9 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 974b802141bd4f13beffef2aa03f88f9, State: Running, Role: LEADER
I20260129 02:59:25.913411 20583 raft_consensus.cc:2468] T a4c1226b987b468fa84dfc6ba0e2dbe1 P 1c50de89a5a845668696dcb28c3badd6 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 197b880fa39d432a9b472845876bfaf0 in term 2.
I20260129 02:59:25.915449 20405 leader_election.cc:304] T a4c1226b987b468fa84dfc6ba0e2dbe1 P 197b880fa39d432a9b472845876bfaf0 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 197b880fa39d432a9b472845876bfaf0, 1c50de89a5a845668696dcb28c3badd6; no voters:
I20260129 02:59:25.916458 21787 raft_consensus.cc:2804] T a4c1226b987b468fa84dfc6ba0e2dbe1 P 197b880fa39d432a9b472845876bfaf0 [term 2 FOLLOWER]: Leader election won for term 2
I20260129 02:59:25.916930 21787 raft_consensus.cc:697] T a4c1226b987b468fa84dfc6ba0e2dbe1 P 197b880fa39d432a9b472845876bfaf0 [term 2 LEADER]: Becoming Leader. State: Replica: 197b880fa39d432a9b472845876bfaf0, State: Running, Role: LEADER
I20260129 02:59:25.917794 21787 consensus_queue.cc:237] T a4c1226b987b468fa84dfc6ba0e2dbe1 P 197b880fa39d432a9b472845876bfaf0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:59:25.918367 20556 leader_election.cc:304] T e33a417cf9c54cf5af73a4ee3aac4d3a P 1c50de89a5a845668696dcb28c3badd6 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1c50de89a5a845668696dcb28c3badd6, 974b802141bd4f13beffef2aa03f88f9; no voters:
I20260129 02:59:25.919454 22062 raft_consensus.cc:2804] T e33a417cf9c54cf5af73a4ee3aac4d3a P 1c50de89a5a845668696dcb28c3badd6 [term 3 FOLLOWER]: Leader election won for term 3
I20260129 02:59:25.919672 20508 raft_consensus.cc:1240] T 1ce4ddf5b10e4ae8b193b13b2e5936ee P 974b802141bd4f13beffef2aa03f88f9 [term 3 FOLLOWER]: Rejecting Update request from peer 1c50de89a5a845668696dcb28c3badd6 for earlier term 2. Current term is 3. Ops: []
I20260129 02:59:25.919965 22062 raft_consensus.cc:697] T e33a417cf9c54cf5af73a4ee3aac4d3a P 1c50de89a5a845668696dcb28c3badd6 [term 3 LEADER]: Becoming Leader. State: Replica: 1c50de89a5a845668696dcb28c3badd6, State: Running, Role: LEADER
I20260129 02:59:25.920327 21132 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1299723a4476420ab02fd8bfd7aa0645" candidate_uuid: "974b802141bd4f13beffef2aa03f88f9" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "7a9daf5d42a84615a6383cfec9a10608" is_pre_election: true
I20260129 02:59:25.920843 22062 consensus_queue.cc:237] T e33a417cf9c54cf5af73a4ee3aac4d3a P 1c50de89a5a845668696dcb28c3badd6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:59:25.912446 22044 leader_election.cc:290] T 1299723a4476420ab02fd8bfd7aa0645 P 974b802141bd4f13beffef2aa03f88f9 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 197b880fa39d432a9b472845876bfaf0 (127.18.211.1:42027), 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3:37735), 7a9daf5d42a84615a6383cfec9a10608 (127.18.211.4:40439)
I20260129 02:59:25.926610 20516 raft_consensus.cc:1240] T 75be4bd6bd45474795fd472eff2953ac P 974b802141bd4f13beffef2aa03f88f9 [term 6 FOLLOWER]: Rejecting Update request from peer 1c50de89a5a845668696dcb28c3badd6 for earlier term 5. Current term is 6. Ops: []
I20260129 02:59:25.928596 20584 raft_consensus.cc:2468] T 1299723a4476420ab02fd8bfd7aa0645 P 1c50de89a5a845668696dcb28c3badd6 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 197b880fa39d432a9b472845876bfaf0 in term 4.
I20260129 02:59:25.927611 22028 leader_election.cc:290] T 75be4bd6bd45474795fd472eff2953ac P 974b802141bd4f13beffef2aa03f88f9 [CANDIDATE]: Term 6 election: Requested vote from peers 197b880fa39d432a9b472845876bfaf0 (127.18.211.1:42027), 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3:37735)
I20260129 02:59:25.934329 20405 leader_election.cc:304] T 1299723a4476420ab02fd8bfd7aa0645 P 197b880fa39d432a9b472845876bfaf0 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 197b880fa39d432a9b472845876bfaf0, 1c50de89a5a845668696dcb28c3badd6; no voters:
I20260129 02:59:25.932880 21136 raft_consensus.cc:2468] T 1299723a4476420ab02fd8bfd7aa0645 P 7a9daf5d42a84615a6383cfec9a10608 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 197b880fa39d432a9b472845876bfaf0 in term 4.
I20260129 02:59:25.935395 20440 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1ce4ddf5b10e4ae8b193b13b2e5936ee" candidate_uuid: "974b802141bd4f13beffef2aa03f88f9" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "197b880fa39d432a9b472845876bfaf0"
I20260129 02:59:25.946278 20439 raft_consensus.cc:2468] T e33a417cf9c54cf5af73a4ee3aac4d3a P 197b880fa39d432a9b472845876bfaf0 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1c50de89a5a845668696dcb28c3badd6 in term 3.
I20260129 02:59:25.971298 20508 raft_consensus.cc:1275] T 0678b8b468bd43cdbc3e485900844821 P 974b802141bd4f13beffef2aa03f88f9 [term 4 FOLLOWER]: Refusing update from remote peer 7a9daf5d42a84615a6383cfec9a10608: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
W20260129 02:59:25.973490 21100 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.211.3:37735, user_credentials={real_user=slave}} blocked reactor thread for 149482us
I20260129 02:59:25.940516 22004 raft_consensus.cc:2749] T 1299723a4476420ab02fd8bfd7aa0645 P 1c50de89a5a845668696dcb28c3badd6 [term 4 FOLLOWER]: Leader pre-election lost for term 3. Reason: Vote denied by peer 197b880fa39d432a9b472845876bfaf0 (127.18.211.1:42027) with higher term. Message: Invalid argument: T 1299723a4476420ab02fd8bfd7aa0645 P 197b880fa39d432a9b472845876bfaf0 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1c50de89a5a845668696dcb28c3badd6 for earlier term 3. Current term is 4.
I20260129 02:59:25.972579 22042 raft_consensus.cc:2804] T 1299723a4476420ab02fd8bfd7aa0645 P 197b880fa39d432a9b472845876bfaf0 [term 4 FOLLOWER]: Leader election won for term 4
I20260129 02:59:25.933431 21971 consensus_queue.cc:1059] T 1ce4ddf5b10e4ae8b193b13b2e5936ee P 1c50de89a5a845668696dcb28c3badd6 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 }, Status: INVALID_TERM, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.056s
W20260129 02:59:26.028820 20554 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.211.4:40439, user_credentials={real_user=slave}} blocked reactor thread for 237901us
I20260129 02:59:26.024888 20584 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "dc2f4cb752cf4d59b5f6bc689f5d8104" candidate_uuid: "7a9daf5d42a84615a6383cfec9a10608" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "1c50de89a5a845668696dcb28c3badd6" is_pre_election: true
W20260129 02:59:26.030189 20407 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.211.2:34517, user_credentials={real_user=slave}} blocked reactor thread for 242952us
I20260129 02:59:26.030560 20584 raft_consensus.cc:2468] T dc2f4cb752cf4d59b5f6bc689f5d8104 P 1c50de89a5a845668696dcb28c3badd6 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7a9daf5d42a84615a6383cfec9a10608 in term 2.
I20260129 02:59:26.035534 20512 raft_consensus.cc:1275] T b189f08e16a74a11980cc55d99359a0a P 974b802141bd4f13beffef2aa03f88f9 [term 4 FOLLOWER]: Refusing update from remote peer 197b880fa39d432a9b472845876bfaf0: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20260129 02:59:25.981559 20433 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "75be4bd6bd45474795fd472eff2953ac" candidate_uuid: "974b802141bd4f13beffef2aa03f88f9" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "197b880fa39d432a9b472845876bfaf0"
I20260129 02:59:25.914191 21994 consensus_queue.cc:260] T a4c1226b987b468fa84dfc6ba0e2dbe1 P 974b802141bd4f13beffef2aa03f88f9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:59:26.064492 20508 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a4c1226b987b468fa84dfc6ba0e2dbe1" candidate_uuid: "197b880fa39d432a9b472845876bfaf0" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "974b802141bd4f13beffef2aa03f88f9"
I20260129 02:59:26.021489 21974 leader_election.cc:290] T 1ce4ddf5b10e4ae8b193b13b2e5936ee P 974b802141bd4f13beffef2aa03f88f9 [CANDIDATE]: Term 3 election: Requested vote from peers 197b880fa39d432a9b472845876bfaf0 (127.18.211.1:42027), 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3:37735)
I20260129 02:59:25.980935 22049 consensus_queue.cc:1048] T 0678b8b468bd43cdbc3e485900844821 P 7a9daf5d42a84615a6383cfec9a10608 [LEADER]: Connected to new peer: Peer: permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.062s
I20260129 02:59:26.022006 22061 consensus_queue.cc:1059] T 75be4bd6bd45474795fd472eff2953ac P 1c50de89a5a845668696dcb28c3badd6 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 }, Status: INVALID_TERM, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.053s
I20260129 02:59:26.059813 21984 raft_consensus.cc:3055] T 1ce4ddf5b10e4ae8b193b13b2e5936ee P 1c50de89a5a845668696dcb28c3badd6 [term 2 LEADER]: Stepping down as leader of term 2
I20260129 02:59:26.192885 20510 raft_consensus.cc:1275] T 7d9494f5584d4d5db8b9adf1c8fc888b P 974b802141bd4f13beffef2aa03f88f9 [term 2 FOLLOWER]: Refusing update from remote peer 1c50de89a5a845668696dcb28c3badd6: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260129 02:59:26.142719 21944 raft_consensus.cc:493] T 368953c9102643b2a21f99af36b51905 P 1c50de89a5a845668696dcb28c3badd6 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260129 02:59:26.142155 21984 raft_consensus.cc:740] T 1ce4ddf5b10e4ae8b193b13b2e5936ee P 1c50de89a5a845668696dcb28c3badd6 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 1c50de89a5a845668696dcb28c3badd6, State: Running, Role: LEADER
I20260129 02:59:26.193217 22037 raft_consensus.cc:3055] T 75be4bd6bd45474795fd472eff2953ac P 1c50de89a5a845668696dcb28c3badd6 [term 5 LEADER]: Stepping down as leader of term 5
W20260129 02:59:26.196244 21100 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.211.3:37735, user_credentials={real_user=slave}} blocked reactor thread for 50081.7us
I20260129 02:59:26.203894 22075 raft_consensus.cc:493] T fcddab6bca684adf8ea032289a00f90e P 974b802141bd4f13beffef2aa03f88f9 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260129 02:59:26.230572 20584 raft_consensus.cc:1275] T a0fcc58d6fb6473cafbd9d34d5195269 P 1c50de89a5a845668696dcb28c3badd6 [term 2 FOLLOWER]: Refusing update from remote peer 197b880fa39d432a9b472845876bfaf0: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260129 02:59:26.235357 20335 rpcz_store.cc:275] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:55376 (request call id 241) took 1071 ms. Trace:
I20260129 02:59:26.145128 19276 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260129 02:59:26.250877 20439 raft_consensus.cc:1275] T a5a05fbcbbcf4916bbc01fd1a5b5e90d P 197b880fa39d432a9b472845876bfaf0 [term 4 FOLLOWER]: Refusing update from remote peer 1c50de89a5a845668696dcb28c3badd6: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20260129 02:59:26.245106 20335 rpcz_store.cc:276] 0129 02:59:25.163956 (+ 0us) service_pool.cc:168] Inserting onto call queue
0129 02:59:25.164082 (+ 126us) service_pool.cc:225] Handling call
0129 02:59:26.235276 (+1071194us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0129 02:59:25.242865 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0129 02:59:25.243414 (+ 549us) write_op.cc:432] Acquiring schema lock in shared mode
0129 02:59:25.243451 (+ 37us) write_op.cc:435] Acquired schema lock
0129 02:59:25.243467 (+ 16us) tablet.cc:660] Decoding operations
0129 02:59:25.243793 (+ 326us) write_op.cc:620] Acquiring the partition lock for write op
0129 02:59:25.243858 (+ 65us) write_op.cc:641] Partition lock acquired for write op
0129 02:59:25.243918 (+ 60us) tablet.cc:683] Acquiring locks for 1 operations
0129 02:59:25.244098 (+ 180us) tablet.cc:699] Row locks acquired
0129 02:59:25.244122 (+ 24us) write_op.cc:260] PREPARE: finished
0129 02:59:25.244260 (+ 138us) write_op.cc:270] Start()
0129 02:59:25.244366 (+ 106us) write_op.cc:276] Timestamp: P: 1769655565244210 usec, L: 0
0129 02:59:25.244385 (+ 19us) op_driver.cc:348] REPLICATION: starting
0129 02:59:25.244989 (+ 604us) log.cc:844] Serialized 538 byte log entry
0129 02:59:25.416722 (+171733us) op_driver.cc:464] REPLICATION: finished
0129 02:59:25.745524 (+328802us) write_op.cc:301] APPLY: starting
0129 02:59:25.752946 (+ 7422us) tablet.cc:1365] starting BulkCheckPresence
0129 02:59:25.765179 (+ 12233us) tablet.cc:1368] finished BulkCheckPresence
0129 02:59:25.787660 (+ 22481us) tablet.cc:1370] starting ApplyRowOperation cycle
0129 02:59:25.810676 (+ 23016us) tablet.cc:1381] finished ApplyRowOperation cycle
0129 02:59:25.839587 (+ 28911us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0129 02:59:25.839635 (+ 48us) write_op.cc:312] APPLY: finished
0129 02:59:25.839838 (+ 203us) log.cc:844] Serialized 23 byte log entry
0129 02:59:25.840027 (+ 189us) write_op.cc:489] Releasing partition, row and schema locks
0129 02:59:25.840108 (+ 81us) write_op.cc:454] Released schema lock
0129 02:59:25.840146 (+ 38us) write_op.cc:341] FINISH: Updating metrics
0129 02:59:25.840502 (+ 356us) write_op.cc:489] Releasing partition, row and schema locks
0129 02:59:25.840529 (+ 27us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":1960,"reported_tablets":2,"child_traces":[["op",{"apply.queue_time_us":286627,"apply.run_cpu_time_us":2078,"apply.run_wall_time_us":97776,"num_ops":1,"prepare.queue_time_us":45069,"prepare.run_cpu_time_us":2410,"prepare.run_wall_time_us":2411,"replication_time_us":143728,"thread_start_us":108185,"threads_started":2}]]}
I20260129 02:59:26.294664 20510 raft_consensus.cc:1275] T a5a05fbcbbcf4916bbc01fd1a5b5e90d P 974b802141bd4f13beffef2aa03f88f9 [term 4 FOLLOWER]: Refusing update from remote peer 1c50de89a5a845668696dcb28c3badd6: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
W20260129 02:59:26.230195 22036 consensus_meta.cc:287] T 368953c9102643b2a21f99af36b51905 P 974b802141bd4f13beffef2aa03f88f9: Time spent flushing consensus metadata: real 0.569s user 0.004s sys 0.007s
I20260129 02:59:26.238248 20338 catalog_manager.cc:5654] T 0678b8b468bd43cdbc3e485900844821 P 7a9daf5d42a84615a6383cfec9a10608 reported cstate change: term changed from 2 to 4. New cstate: current_term: 4 leader_uuid: "7a9daf5d42a84615a6383cfec9a10608" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260129 02:59:26.226884 22037 raft_consensus.cc:740] T 75be4bd6bd45474795fd472eff2953ac P 1c50de89a5a845668696dcb28c3badd6 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 1c50de89a5a845668696dcb28c3badd6, State: Running, Role: LEADER
I20260129 02:59:26.205209 22061 consensus_queue.cc:1048] T 7d9494f5584d4d5db8b9adf1c8fc888b P 1c50de89a5a845668696dcb28c3badd6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.045s
I20260129 02:59:26.308676 20583 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1299723a4476420ab02fd8bfd7aa0645" candidate_uuid: "974b802141bd4f13beffef2aa03f88f9" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "1c50de89a5a845668696dcb28c3badd6" is_pre_election: true
I20260129 02:59:26.447089 20583 raft_consensus.cc:1813] T 1299723a4476420ab02fd8bfd7aa0645 P 1c50de89a5a845668696dcb28c3badd6 [term 4 FOLLOWER]: Handling vote request from an unknown peer 974b802141bd4f13beffef2aa03f88f9
I20260129 02:59:26.449515 20584 raft_consensus.cc:1275] T b189f08e16a74a11980cc55d99359a0a P 1c50de89a5a845668696dcb28c3badd6 [term 4 FOLLOWER]: Refusing update from remote peer 197b880fa39d432a9b472845876bfaf0: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20260129 02:59:26.451050 20583 raft_consensus.cc:2368] T 1299723a4476420ab02fd8bfd7aa0645 P 1c50de89a5a845668696dcb28c3badd6 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 974b802141bd4f13beffef2aa03f88f9 for earlier term 3. Current term is 4.
I20260129 02:59:26.217316 21944 raft_consensus.cc:515] T 368953c9102643b2a21f99af36b51905 P 1c50de89a5a845668696dcb28c3badd6 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
W20260129 02:59:26.453764 20434 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.514s user 0.000s sys 0.007s
W20260129 02:59:26.466408 20479 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.211.1:42027, user_credentials={real_user=slave}} blocked reactor thread for 158398us
I20260129 02:59:26.487102 20586 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "75be4bd6bd45474795fd472eff2953ac" candidate_uuid: "974b802141bd4f13beffef2aa03f88f9" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "1c50de89a5a845668696dcb28c3badd6"
I20260129 02:59:26.489488 21944 leader_election.cc:290] T 368953c9102643b2a21f99af36b51905 P 1c50de89a5a845668696dcb28c3badd6 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 197b880fa39d432a9b472845876bfaf0 (127.18.211.1:42027), 974b802141bd4f13beffef2aa03f88f9 (127.18.211.2:34517)
I20260129 02:59:26.490327 21994 raft_consensus.cc:3060] T a4c1226b987b468fa84dfc6ba0e2dbe1 P 974b802141bd4f13beffef2aa03f88f9 [term 1 FOLLOWER]: Advancing to term 2
I20260129 02:59:26.491369 20583 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1ce4ddf5b10e4ae8b193b13b2e5936ee" candidate_uuid: "974b802141bd4f13beffef2aa03f88f9" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1c50de89a5a845668696dcb28c3badd6"
I20260129 02:59:26.493348 22042 raft_consensus.cc:697] T 1299723a4476420ab02fd8bfd7aa0645 P 197b880fa39d432a9b472845876bfaf0 [term 4 LEADER]: Becoming Leader. State: Replica: 197b880fa39d432a9b472845876bfaf0, State: Running, Role: LEADER
I20260129 02:59:26.494704 22042 consensus_queue.cc:237] T 1299723a4476420ab02fd8bfd7aa0645 P 197b880fa39d432a9b472845876bfaf0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 2.5, Last appended by leader: 5, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } }
W20260129 02:59:26.497786 20556 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.211.2:34517, user_credentials={real_user=slave}} blocked reactor thread for 193672us
I20260129 02:59:26.504115 20511 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "368953c9102643b2a21f99af36b51905" candidate_uuid: "1c50de89a5a845668696dcb28c3badd6" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "974b802141bd4f13beffef2aa03f88f9" is_pre_election: true
I20260129 02:59:26.505231 20439 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "dc2f4cb752cf4d59b5f6bc689f5d8104" candidate_uuid: "7a9daf5d42a84615a6383cfec9a10608" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "197b880fa39d432a9b472845876bfaf0" is_pre_election: true
I20260129 02:59:26.509593 22004 raft_consensus.cc:493] T fcddab6bca684adf8ea032289a00f90e P 1c50de89a5a845668696dcb28c3badd6 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260129 02:59:26.510196 22004 raft_consensus.cc:515] T fcddab6bca684adf8ea032289a00f90e P 1c50de89a5a845668696dcb28c3badd6 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
W20260129 02:59:26.516639 20405 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.211.3:37735, user_credentials={real_user=slave}} blocked reactor thread for 67847us
W20260129 02:59:26.475044 22059 consensus_meta.cc:287] T 96609ac68bdf4993a12e1c23355ca013 P 7a9daf5d42a84615a6383cfec9a10608: Time spent flushing consensus metadata: real 0.731s user 0.012s sys 0.007s
W20260129 02:59:26.531235 20514 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.506s user 0.100s sys 0.147s
W20260129 02:59:26.540680 20514 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.515s user 0.100s sys 0.148s
I20260129 02:59:26.547514 20510 raft_consensus.cc:1275] T 87d5fe48a44a408aaf03e894a67ecb2f P 974b802141bd4f13beffef2aa03f88f9 [term 4 FOLLOWER]: Refusing update from remote peer 7a9daf5d42a84615a6383cfec9a10608: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20260129 02:59:26.449208 21100 leader_election.cc:304] T dc2f4cb752cf4d59b5f6bc689f5d8104 P 7a9daf5d42a84615a6383cfec9a10608 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1c50de89a5a845668696dcb28c3badd6, 7a9daf5d42a84615a6383cfec9a10608; no voters:
I20260129 02:59:26.304075 22066 consensus_queue.cc:1048] T a5a05fbcbbcf4916bbc01fd1a5b5e90d P 1c50de89a5a845668696dcb28c3badd6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.219s
I20260129 02:59:26.297760 22075 raft_consensus.cc:515] T fcddab6bca684adf8ea032289a00f90e P 974b802141bd4f13beffef2aa03f88f9 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:59:26.544639 22031 ts_tablet_manager.cc:1929] T 3ccac82a01a2445fa5b0ce6c0fcf2295 P 1c50de89a5a845668696dcb28c3badd6: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20260129 02:59:26.448309 21984 consensus_queue.cc:260] T 1ce4ddf5b10e4ae8b193b13b2e5936ee P 1c50de89a5a845668696dcb28c3badd6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:59:26.549858 22059 raft_consensus.cc:2749] T 96609ac68bdf4993a12e1c23355ca013 P 7a9daf5d42a84615a6383cfec9a10608 [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260129 02:59:26.587641 20507 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fcddab6bca684adf8ea032289a00f90e" candidate_uuid: "1c50de89a5a845668696dcb28c3badd6" candidate_term: 6 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "974b802141bd4f13beffef2aa03f88f9" is_pre_election: true
I20260129 02:59:26.306951 22036 raft_consensus.cc:515] T 368953c9102643b2a21f99af36b51905 P 974b802141bd4f13beffef2aa03f88f9 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:59:26.591626 20516 raft_consensus.cc:1275] T a0fcc58d6fb6473cafbd9d34d5195269 P 974b802141bd4f13beffef2aa03f88f9 [term 2 FOLLOWER]: Refusing update from remote peer 197b880fa39d432a9b472845876bfaf0: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260129 02:59:26.578728 22004 leader_election.cc:290] T fcddab6bca684adf8ea032289a00f90e P 1c50de89a5a845668696dcb28c3badd6 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 197b880fa39d432a9b472845876bfaf0 (127.18.211.1:42027), 974b802141bd4f13beffef2aa03f88f9 (127.18.211.2:34517)
W20260129 02:59:26.593333 22084 log.cc:927] Time spent T 4b1876f9a70b49edb3aa74105e66d3b3 P 1c50de89a5a845668696dcb28c3badd6: Append to log took a long time: real 0.060s user 0.000s sys 0.000s
I20260129 02:59:26.594657 22105 raft_consensus.cc:493] T 42d27a28d89b409e8807ec0efd97af69 P 197b880fa39d432a9b472845876bfaf0 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260129 02:59:26.595391 22077 raft_consensus.cc:493] T dc2f4cb752cf4d59b5f6bc689f5d8104 P 197b880fa39d432a9b472845876bfaf0 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 1c50de89a5a845668696dcb28c3badd6)
W20260129 02:59:26.606669 21099 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.211.1:42027, user_credentials={real_user=slave}} blocked reactor thread for 87710.1us
I20260129 02:59:26.612977 20507 raft_consensus.cc:2468] T fcddab6bca684adf8ea032289a00f90e P 974b802141bd4f13beffef2aa03f88f9 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1c50de89a5a845668696dcb28c3badd6 in term 5.
I20260129 02:59:26.619323 20513 raft_consensus.cc:1275] T cedce331e1584ac8b636c9b8a72b2ba0 P 974b802141bd4f13beffef2aa03f88f9 [term 4 FOLLOWER]: Refusing update from remote peer 1c50de89a5a845668696dcb28c3badd6: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20260129 02:59:26.600494 21883 raft_consensus.cc:493] T 4e3805ffa70242e8ade66da260dddf66 P 197b880fa39d432a9b472845876bfaf0 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 7a9daf5d42a84615a6383cfec9a10608)
I20260129 02:59:26.548676 22037 consensus_queue.cc:260] T 75be4bd6bd45474795fd472eff2953ac P 1c50de89a5a845668696dcb28c3badd6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.4, Last appended by leader: 4, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:59:26.645592 20437 raft_consensus.cc:1275] T cedce331e1584ac8b636c9b8a72b2ba0 P 197b880fa39d432a9b472845876bfaf0 [term 4 FOLLOWER]: Refusing update from remote peer 1c50de89a5a845668696dcb28c3badd6: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20260129 02:59:26.610190 22031 log.cc:1199] T 3ccac82a01a2445fa5b0ce6c0fcf2295 P 1c50de89a5a845668696dcb28c3badd6: Deleting WAL directory at /tmp/dist-test-taskrTP_Jo/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1769655460890924-19276-0/minicluster-data/ts-2-root/wals/3ccac82a01a2445fa5b0ce6c0fcf2295
I20260129 02:59:26.623812 20511 raft_consensus.cc:2468] T 368953c9102643b2a21f99af36b51905 P 974b802141bd4f13beffef2aa03f88f9 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1c50de89a5a845668696dcb28c3badd6 in term 3.
I20260129 02:59:26.640962 20440 raft_consensus.cc:1275] T e33a417cf9c54cf5af73a4ee3aac4d3a P 197b880fa39d432a9b472845876bfaf0 [term 3 FOLLOWER]: Refusing update from remote peer 1c50de89a5a845668696dcb28c3badd6: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20260129 02:59:26.608776 22049 raft_consensus.cc:2804] T dc2f4cb752cf4d59b5f6bc689f5d8104 P 7a9daf5d42a84615a6383cfec9a10608 [term 2 FOLLOWER]: Leader pre-election won for term 3
W20260129 02:59:26.658792 20480 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.211.3:37735, user_credentials={real_user=slave}} blocked reactor thread for 126950us
I20260129 02:59:26.660097 20440 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fcddab6bca684adf8ea032289a00f90e" candidate_uuid: "1c50de89a5a845668696dcb28c3badd6" candidate_term: 6 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "197b880fa39d432a9b472845876bfaf0" is_pre_election: true
W20260129 02:59:26.603404 21100 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.211.3:37735, user_credentials={real_user=slave}} blocked reactor thread for 302722us
I20260129 02:59:26.622258 21787 raft_consensus.cc:493] T fcddab6bca684adf8ea032289a00f90e P 197b880fa39d432a9b472845876bfaf0 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 1c50de89a5a845668696dcb28c3badd6)
I20260129 02:59:26.677388 20513 tablet_service.cc:2066] Received LeaderStepDown RPC: tablet_id: "36ff33be666a4c82954731691dbcf327"
dest_uuid: "974b802141bd4f13beffef2aa03f88f9"
mode: GRACEFUL
new_leader_uuid: "1c50de89a5a845668696dcb28c3badd6"
from {username='slave'} at 127.0.0.1:52008
I20260129 02:59:26.683673 20513 raft_consensus.cc:606] T 36ff33be666a4c82954731691dbcf327 P 974b802141bd4f13beffef2aa03f88f9 [term 1 LEADER]: Received request to transfer leadership to TS 1c50de89a5a845668696dcb28c3badd6
I20260129 02:59:26.675980 21883 raft_consensus.cc:515] T 4e3805ffa70242e8ade66da260dddf66 P 197b880fa39d432a9b472845876bfaf0 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } }
I20260129 02:59:26.613946 21944 consensus_queue.cc:1048] T a5a05fbcbbcf4916bbc01fd1a5b5e90d P 1c50de89a5a845668696dcb28c3badd6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.051s
I20260129 02:59:26.622335 21128 raft_consensus.cc:2468] T 96609ac68bdf4993a12e1c23355ca013 P 7a9daf5d42a84615a6383cfec9a10608 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 974b802141bd4f13beffef2aa03f88f9 in term 3.
I20260129 02:59:26.653298 22049 raft_consensus.cc:493] T dc2f4cb752cf4d59b5f6bc689f5d8104 P 7a9daf5d42a84615a6383cfec9a10608 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260129 02:59:26.696301 22081 log.cc:927] Time spent T 78aa39096868415d9afc2ffcb248e198 P 1c50de89a5a845668696dcb28c3badd6: Append to log took a long time: real 0.154s user 0.000s sys 0.002s
I20260129 02:59:26.685616 21787 raft_consensus.cc:515] T fcddab6bca684adf8ea032289a00f90e P 197b880fa39d432a9b472845876bfaf0 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
W20260129 02:59:26.684227 22096 log.cc:927] Time spent T 926a250fa5834e88b96ccecbefbf5086 P 1c50de89a5a845668696dcb28c3badd6: Append to log took a long time: real 0.133s user 0.000s sys 0.000s
I20260129 02:59:26.624267 21130 raft_consensus.cc:1275] T 45cc5f3d2688486d9b6dbe7e666dedde P 7a9daf5d42a84615a6383cfec9a10608 [term 3 FOLLOWER]: Refusing update from remote peer 1c50de89a5a845668696dcb28c3badd6: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 3 index: 5. (index mismatch)
I20260129 02:59:26.708746 20480 leader_election.cc:400] T 1299723a4476420ab02fd8bfd7aa0645 P 974b802141bd4f13beffef2aa03f88f9 [CANDIDATE]: Term 3 pre-election: Vote denied by peer 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3:37735) with higher term. Message: Invalid argument: T 1299723a4476420ab02fd8bfd7aa0645 P 1c50de89a5a845668696dcb28c3badd6 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 974b802141bd4f13beffef2aa03f88f9 for earlier term 3. Current term is 4.
I20260129 02:59:26.710729 20440 raft_consensus.cc:2468] T fcddab6bca684adf8ea032289a00f90e P 197b880fa39d432a9b472845876bfaf0 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1c50de89a5a845668696dcb28c3badd6 in term 4.
I20260129 02:59:26.713753 22092 raft_consensus.cc:493] T c934d31efd3f4e5195605ddc354d14e1 P 197b880fa39d432a9b472845876bfaf0 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 7a9daf5d42a84615a6383cfec9a10608)
W20260129 02:59:26.714030 22095 log.cc:927] Time spent T 76f713c3331540d2a5b76164103e9eb1 P 197b880fa39d432a9b472845876bfaf0: Append to log took a long time: real 0.112s user 0.000s sys 0.000s
I20260129 02:59:26.616819 22070 consensus_queue.cc:1048] T a0fcc58d6fb6473cafbd9d34d5195269 P 197b880fa39d432a9b472845876bfaf0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.103s
I20260129 02:59:26.714361 22092 raft_consensus.cc:515] T c934d31efd3f4e5195605ddc354d14e1 P 197b880fa39d432a9b472845876bfaf0 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } }
I20260129 02:59:26.709528 21787 leader_election.cc:290] T fcddab6bca684adf8ea032289a00f90e P 197b880fa39d432a9b472845876bfaf0 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 974b802141bd4f13beffef2aa03f88f9 (127.18.211.2:34517), 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3:37735)
I20260129 02:59:26.687340 21883 leader_election.cc:290] T 4e3805ffa70242e8ade66da260dddf66 P 197b880fa39d432a9b472845876bfaf0 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3:37735), 7a9daf5d42a84615a6383cfec9a10608 (127.18.211.4:40439)
I20260129 02:59:26.721086 21136 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "42d27a28d89b409e8807ec0efd97af69" candidate_uuid: "1c50de89a5a845668696dcb28c3badd6" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "7a9daf5d42a84615a6383cfec9a10608"
I20260129 02:59:26.701851 22049 raft_consensus.cc:3060] T dc2f4cb752cf4d59b5f6bc689f5d8104 P 7a9daf5d42a84615a6383cfec9a10608 [term 2 FOLLOWER]: Advancing to term 3
I20260129 02:59:26.618331 22105 raft_consensus.cc:515] T 42d27a28d89b409e8807ec0efd97af69 P 197b880fa39d432a9b472845876bfaf0 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } }
I20260129 02:59:26.640009 22077 raft_consensus.cc:515] T dc2f4cb752cf4d59b5f6bc689f5d8104 P 197b880fa39d432a9b472845876bfaf0 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } }
I20260129 02:59:26.722867 20433 raft_consensus.cc:1275] T 7d9494f5584d4d5db8b9adf1c8fc888b P 197b880fa39d432a9b472845876bfaf0 [term 2 FOLLOWER]: Refusing update from remote peer 1c50de89a5a845668696dcb28c3badd6: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260129 02:59:26.730091 22031 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35744 (request call id 703) took 2130 ms. Trace:
I20260129 02:59:26.752775 22109 consensus_queue.cc:1048] T 45cc5f3d2688486d9b6dbe7e666dedde P 1c50de89a5a845668696dcb28c3badd6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.003s
W20260129 02:59:26.743829 22022 consensus_meta.cc:287] T 1299723a4476420ab02fd8bfd7aa0645 P 974b802141bd4f13beffef2aa03f88f9: Time spent flushing consensus metadata: real 0.832s user 0.010s sys 0.002s
I20260129 02:59:26.764927 22075 leader_election.cc:290] T fcddab6bca684adf8ea032289a00f90e P 974b802141bd4f13beffef2aa03f88f9 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 197b880fa39d432a9b472845876bfaf0 (127.18.211.1:42027), 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3:37735)
I20260129 02:59:26.783031 21136 raft_consensus.cc:2468] T 42d27a28d89b409e8807ec0efd97af69 P 7a9daf5d42a84615a6383cfec9a10608 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1c50de89a5a845668696dcb28c3badd6 in term 3.
W20260129 02:59:26.784307 22098 log.cc:927] Time spent T 0678b8b468bd43cdbc3e485900844821 P 974b802141bd4f13beffef2aa03f88f9: Append to log took a long time: real 0.163s user 0.000s sys 0.000s
I20260129 02:59:26.784454 21129 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "96609ac68bdf4993a12e1c23355ca013" candidate_uuid: "197b880fa39d432a9b472845876bfaf0" candidate_term: 4 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "7a9daf5d42a84615a6383cfec9a10608" is_pre_election: true
W20260129 02:59:26.785445 21129 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34060 (request call id 402) took 1375 ms (client timeout 1593 ms). Trace:
I20260129 02:59:26.772843 20439 raft_consensus.cc:2468] T dc2f4cb752cf4d59b5f6bc689f5d8104 P 197b880fa39d432a9b472845876bfaf0 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7a9daf5d42a84615a6383cfec9a10608 in term 1.
W20260129 02:59:26.785851 21129 rpcz_store.cc:269] 0129 02:59:25.409914 (+ 0us) service_pool.cc:168] Inserting onto call queue
0129 02:59:25.769192 (+359278us) service_pool.cc:225] Handling call
0129 02:59:26.784254 (+1015062us) spinlock_profiling.cc:243] Waited 4 ms on lock 0x7b5800111f98. stack: 00007f8fe044e373 00007f8fe044e270 00007f8fdfda0574 00007f8fe9a5c6b4 00007f8fe9a5c649 00007f8fe02c08b1 00007f8fe0442db1 00007f8fe0442c62 00007f8fe92eddd9 00007f8fe846d260 00007f8fe8535dbb 00007f8fe8535e41 00007f8fe84e3bd0 00007f8fe84e34b4 00007f8fe1a73834 00007f8fe1a737c2
0129 02:59:26.785361 (+ 1107us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":9202048}
I20260129 02:59:26.810276 22105 leader_election.cc:290] T 42d27a28d89b409e8807ec0efd97af69 P 197b880fa39d432a9b472845876bfaf0 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3:37735), 7a9daf5d42a84615a6383cfec9a10608 (127.18.211.4:40439)
I20260129 02:59:26.773334 22022 raft_consensus.cc:2749] T 1299723a4476420ab02fd8bfd7aa0645 P 974b802141bd4f13beffef2aa03f88f9 [term 4 FOLLOWER]: Leader pre-election lost for term 3. Reason: Vote denied by peer 197b880fa39d432a9b472845876bfaf0 (127.18.211.1:42027) with higher term. Message: Invalid argument: T 1299723a4476420ab02fd8bfd7aa0645 P 197b880fa39d432a9b472845876bfaf0 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 974b802141bd4f13beffef2aa03f88f9 for earlier term 3. Current term is 4.
I20260129 02:59:26.813715 20554 leader_election.cc:304] T 42d27a28d89b409e8807ec0efd97af69 P 1c50de89a5a845668696dcb28c3badd6 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1c50de89a5a845668696dcb28c3badd6, 7a9daf5d42a84615a6383cfec9a10608; no voters: 197b880fa39d432a9b472845876bfaf0
W20260129 02:59:26.814466 22074 log.cc:927] Time spent T b189f08e16a74a11980cc55d99359a0a P 197b880fa39d432a9b472845876bfaf0: Append to log took a long time: real 0.674s user 0.000s sys 0.007s
I20260129 02:59:26.814740 22077 leader_election.cc:290] T dc2f4cb752cf4d59b5f6bc689f5d8104 P 197b880fa39d432a9b472845876bfaf0 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3:37735), 7a9daf5d42a84615a6383cfec9a10608 (127.18.211.4:40439)
I20260129 02:59:26.819082 22092 leader_election.cc:290] T c934d31efd3f4e5195605ddc354d14e1 P 197b880fa39d432a9b472845876bfaf0 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 974b802141bd4f13beffef2aa03f88f9 (127.18.211.2:34517), 7a9daf5d42a84615a6383cfec9a10608 (127.18.211.4:40439)
I20260129 02:59:26.820142 22023 raft_consensus.cc:493] T 3ccac82a01a2445fa5b0ce6c0fcf2295 P 7a9daf5d42a84615a6383cfec9a10608 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 197b880fa39d432a9b472845876bfaf0)
I20260129 02:59:26.828869 21129 raft_consensus.cc:1275] T 78aa39096868415d9afc2ffcb248e198 P 7a9daf5d42a84615a6383cfec9a10608 [term 3 FOLLOWER]: Refusing update from remote peer 1c50de89a5a845668696dcb28c3badd6: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 3 index: 5. (index mismatch)
I20260129 02:59:26.820711 22023 raft_consensus.cc:515] T 3ccac82a01a2445fa5b0ce6c0fcf2295 P 7a9daf5d42a84615a6383cfec9a10608 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } }
I20260129 02:59:26.831287 22049 raft_consensus.cc:515] T dc2f4cb752cf4d59b5f6bc689f5d8104 P 7a9daf5d42a84615a6383cfec9a10608 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } }
I20260129 02:59:26.833012 22023 leader_election.cc:290] T 3ccac82a01a2445fa5b0ce6c0fcf2295 P 7a9daf5d42a84615a6383cfec9a10608 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 197b880fa39d432a9b472845876bfaf0 (127.18.211.1:42027), 974b802141bd4f13beffef2aa03f88f9 (127.18.211.2:34517)
I20260129 02:59:26.833185 22049 leader_election.cc:290] T dc2f4cb752cf4d59b5f6bc689f5d8104 P 7a9daf5d42a84615a6383cfec9a10608 [CANDIDATE]: Term 3 election: Requested vote from peers 197b880fa39d432a9b472845876bfaf0 (127.18.211.1:42027), 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3:37735)
I20260129 02:59:26.821800 22036 leader_election.cc:290] T 368953c9102643b2a21f99af36b51905 P 974b802141bd4f13beffef2aa03f88f9 [CANDIDATE]: Term 3 election: Requested vote from peers 197b880fa39d432a9b472845876bfaf0 (127.18.211.1:42027), 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3:37735)
I20260129 02:59:26.846666 20508 raft_consensus.cc:2468] T a4c1226b987b468fa84dfc6ba0e2dbe1 P 974b802141bd4f13beffef2aa03f88f9 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 197b880fa39d432a9b472845876bfaf0 in term 2.
I20260129 02:59:26.848137 22103 raft_consensus.cc:2804] T 42d27a28d89b409e8807ec0efd97af69 P 1c50de89a5a845668696dcb28c3badd6 [term 3 FOLLOWER]: Leader election won for term 3
W20260129 02:59:26.853341 20479 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.211.1:42027, user_credentials={real_user=slave}} blocked reactor thread for 306055us
I20260129 02:59:26.859266 22087 raft_consensus.cc:493] T a8479d1fe0bd41e5946186e3a6ca8c7a P 197b880fa39d432a9b472845876bfaf0 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 974b802141bd4f13beffef2aa03f88f9)
I20260129 02:59:26.859819 22087 raft_consensus.cc:515] T a8479d1fe0bd41e5946186e3a6ca8c7a P 197b880fa39d432a9b472845876bfaf0 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:59:26.868036 20336 rpcz_store.cc:275] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:55348 (request call id 256) took 1472 ms. Trace:
I20260129 02:59:26.868583 20336 rpcz_store.cc:276] 0129 02:59:25.395835 (+ 0us) service_pool.cc:168] Inserting onto call queue
0129 02:59:25.395991 (+ 156us) service_pool.cc:225] Handling call
0129 02:59:25.873122 (+477131us) spinlock_profiling.cc:243] Waited 234 ms on lock 0x7b54003ac854. stack: 00007f8fe044e373 00007f8fe044e270 00007f8fdfda0574 00007f8fe9a5c6b4 00007f8fe9a5c649 00007f8fe34980c0 00007f8fe3536bba 00007f8fe92b7eac 00007f8fe93b210e 00007f8fe78ad4a4 00007f8fe78ad432 00007f8fe78ad361 00007f8fe78ad2dc 00007f8fe78ac592 00007f8fe186d34c 00007f8fe186c786
0129 02:59:26.867952 (+994830us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":124868,"reported_tablets":1,"spinlock_wait_cycles":537462528}
W20260129 02:59:26.873728 21099 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.211.1:42027, user_credentials={real_user=slave}} blocked reactor thread for 214871us
I20260129 02:59:26.878203 22101 raft_consensus.cc:493] T 7a9a69cd82724b1faed1d10aa091fc4a P 1c50de89a5a845668696dcb28c3badd6 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 197b880fa39d432a9b472845876bfaf0)
I20260129 02:59:26.879009 22101 raft_consensus.cc:515] T 7a9a69cd82724b1faed1d10aa091fc4a P 1c50de89a5a845668696dcb28c3badd6 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:59:26.884016 20435 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fcddab6bca684adf8ea032289a00f90e" candidate_uuid: "974b802141bd4f13beffef2aa03f88f9" candidate_term: 6 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "197b880fa39d432a9b472845876bfaf0" is_pre_election: true
I20260129 02:59:26.891831 21984 raft_consensus.cc:3060] T 1ce4ddf5b10e4ae8b193b13b2e5936ee P 1c50de89a5a845668696dcb28c3badd6 [term 2 FOLLOWER]: Advancing to term 3
I20260129 02:59:26.891903 20441 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "368953c9102643b2a21f99af36b51905" candidate_uuid: "974b802141bd4f13beffef2aa03f88f9" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "197b880fa39d432a9b472845876bfaf0"
I20260129 02:59:26.857868 22103 raft_consensus.cc:697] T 42d27a28d89b409e8807ec0efd97af69 P 1c50de89a5a845668696dcb28c3badd6 [term 3 LEADER]: Becoming Leader. State: Replica: 1c50de89a5a845668696dcb28c3badd6, State: Running, Role: LEADER
I20260129 02:59:26.755270 22031 rpcz_store.cc:276] 0129 02:59:24.589083 (+ 0us) service_pool.cc:168] Inserting onto call queue
0129 02:59:24.589277 (+ 194us) service_pool.cc:225] Handling call
0129 02:59:24.844256 (+254979us) ts_tablet_manager.cc:1151] Deleting tablet 3ccac82a01a2445fa5b0ce6c0fcf2295
0129 02:59:24.844315 (+ 59us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
0129 02:59:25.517789 (+673474us) tablet_metadata.cc:641] Metadata flushed
0129 02:59:26.487183 (+969394us) tablet_metadata.cc:641] Metadata flushed
0129 02:59:26.716318 (+229135us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":113824,"tablet-delete.queue_time_us":254065,"thread_start_us":95822,"threads_started":1}
W20260129 02:59:26.913987 20556 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.211.2:34517, user_credentials={real_user=slave}} blocked reactor thread for 297665us
I20260129 02:59:26.940482 20507 raft_consensus.cc:1275] T e33a417cf9c54cf5af73a4ee3aac4d3a P 974b802141bd4f13beffef2aa03f88f9 [term 3 FOLLOWER]: Refusing update from remote peer 1c50de89a5a845668696dcb28c3badd6: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
W20260129 02:59:26.942257 20553 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.211.1:42027, user_credentials={real_user=slave}} blocked reactor thread for 300544us
I20260129 02:59:26.948701 20435 raft_consensus.cc:1275] T 926a250fa5834e88b96ccecbefbf5086 P 197b880fa39d432a9b472845876bfaf0 [term 4 FOLLOWER]: Refusing update from remote peer 1c50de89a5a845668696dcb28c3badd6: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20260129 02:59:26.949306 20507 raft_consensus.cc:1275] T 76f713c3331540d2a5b76164103e9eb1 P 974b802141bd4f13beffef2aa03f88f9 [term 2 FOLLOWER]: Refusing update from remote peer 1c50de89a5a845668696dcb28c3badd6: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260129 02:59:26.952158 22103 consensus_queue.cc:237] T 42d27a28d89b409e8807ec0efd97af69 P 1c50de89a5a845668696dcb28c3badd6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } }
I20260129 02:59:26.956306 20441 raft_consensus.cc:2468] T 368953c9102643b2a21f99af36b51905 P 197b880fa39d432a9b472845876bfaf0 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 974b802141bd4f13beffef2aa03f88f9 in term 3.
I20260129 02:59:26.960515 20479 leader_election.cc:304] T 368953c9102643b2a21f99af36b51905 P 974b802141bd4f13beffef2aa03f88f9 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 197b880fa39d432a9b472845876bfaf0, 974b802141bd4f13beffef2aa03f88f9; no voters:
I20260129 02:59:26.906538 22108 consensus_queue.cc:1048] T 78aa39096868415d9afc2ffcb248e198 P 1c50de89a5a845668696dcb28c3badd6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.100s
I20260129 02:59:26.949353 22087 leader_election.cc:290] T a8479d1fe0bd41e5946186e3a6ca8c7a P 197b880fa39d432a9b472845876bfaf0 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 974b802141bd4f13beffef2aa03f88f9 (127.18.211.2:34517), 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3:37735)
W20260129 02:59:26.967626 20405 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.211.3:37735, user_credentials={real_user=slave}} blocked reactor thread for 450069us
I20260129 02:59:26.971712 20588 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "dc2f4cb752cf4d59b5f6bc689f5d8104" candidate_uuid: "7a9daf5d42a84615a6383cfec9a10608" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "1c50de89a5a845668696dcb28c3badd6"
I20260129 02:59:26.972349 20588 raft_consensus.cc:3060] T dc2f4cb752cf4d59b5f6bc689f5d8104 P 1c50de89a5a845668696dcb28c3badd6 [term 2 FOLLOWER]: Advancing to term 3
I20260129 02:59:26.964128 22075 raft_consensus.cc:2804] T 368953c9102643b2a21f99af36b51905 P 974b802141bd4f13beffef2aa03f88f9 [term 3 FOLLOWER]: Leader election won for term 3
I20260129 02:59:26.975615 20556 leader_election.cc:304] T fcddab6bca684adf8ea032289a00f90e P 1c50de89a5a845668696dcb28c3badd6 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1c50de89a5a845668696dcb28c3badd6, 974b802141bd4f13beffef2aa03f88f9; no voters:
I20260129 02:59:26.963181 21995 consensus_queue.cc:1048] T e33a417cf9c54cf5af73a4ee3aac4d3a P 1c50de89a5a845668696dcb28c3badd6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.010s
I20260129 02:59:26.984094 20588 raft_consensus.cc:2468] T dc2f4cb752cf4d59b5f6bc689f5d8104 P 1c50de89a5a845668696dcb28c3badd6 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7a9daf5d42a84615a6383cfec9a10608 in term 3.
I20260129 02:59:26.970919 22101 leader_election.cc:290] T 7a9a69cd82724b1faed1d10aa091fc4a P 1c50de89a5a845668696dcb28c3badd6 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 197b880fa39d432a9b472845876bfaf0 (127.18.211.1:42027), 974b802141bd4f13beffef2aa03f88f9 (127.18.211.2:34517)
I20260129 02:59:26.973413 21998 raft_consensus.cc:493] T a0fcc58d6fb6473cafbd9d34d5195269 P 974b802141bd4f13beffef2aa03f88f9 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260129 02:59:26.996147 20507 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fcddab6bca684adf8ea032289a00f90e" candidate_uuid: "197b880fa39d432a9b472845876bfaf0" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "974b802141bd4f13beffef2aa03f88f9" is_pre_election: true
I20260129 02:59:26.996927 20507 raft_consensus.cc:2393] T fcddab6bca684adf8ea032289a00f90e P 974b802141bd4f13beffef2aa03f88f9 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 197b880fa39d432a9b472845876bfaf0 in current term 5: Already voted for candidate 974b802141bd4f13beffef2aa03f88f9 in this term.
I20260129 02:59:26.998694 20435 raft_consensus.cc:1275] T 0678b8b468bd43cdbc3e485900844821 P 197b880fa39d432a9b472845876bfaf0 [term 4 FOLLOWER]: Refusing update from remote peer 7a9daf5d42a84615a6383cfec9a10608: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20260129 02:59:26.991473 22079 consensus_queue.cc:1048] T 7d9494f5584d4d5db8b9adf1c8fc888b P 1c50de89a5a845668696dcb28c3badd6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.004s
I20260129 02:59:26.991782 20405 leader_election.cc:304] T 96609ac68bdf4993a12e1c23355ca013 P 197b880fa39d432a9b472845876bfaf0 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 197b880fa39d432a9b472845876bfaf0, 1c50de89a5a845668696dcb28c3badd6; no voters: 7a9daf5d42a84615a6383cfec9a10608, 974b802141bd4f13beffef2aa03f88f9
I20260129 02:59:26.975632 20433 raft_consensus.cc:1275] T 78aa39096868415d9afc2ffcb248e198 P 197b880fa39d432a9b472845876bfaf0 [term 3 FOLLOWER]: Refusing update from remote peer 1c50de89a5a845668696dcb28c3badd6: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 3 index: 5. (index mismatch)
W20260129 02:59:27.001780 20480 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.211.3:37735, user_credentials={real_user=slave}} blocked reactor thread for 270096us
I20260129 02:59:27.008399 20440 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3ccac82a01a2445fa5b0ce6c0fcf2295" candidate_uuid: "7a9daf5d42a84615a6383cfec9a10608" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "197b880fa39d432a9b472845876bfaf0" is_pre_election: true
I20260129 02:59:26.978302 22075 raft_consensus.cc:697] T 368953c9102643b2a21f99af36b51905 P 974b802141bd4f13beffef2aa03f88f9 [term 3 LEADER]: Becoming Leader. State: Replica: 974b802141bd4f13beffef2aa03f88f9, State: Running, Role: LEADER
I20260129 02:59:27.014707 20435 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "dc2f4cb752cf4d59b5f6bc689f5d8104" candidate_uuid: "7a9daf5d42a84615a6383cfec9a10608" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "197b880fa39d432a9b472845876bfaf0"
I20260129 02:59:27.017166 21134 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c934d31efd3f4e5195605ddc354d14e1" candidate_uuid: "197b880fa39d432a9b472845876bfaf0" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "7a9daf5d42a84615a6383cfec9a10608" is_pre_election: true
I20260129 02:59:27.020929 20507 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a8479d1fe0bd41e5946186e3a6ca8c7a" candidate_uuid: "197b880fa39d432a9b472845876bfaf0" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "974b802141bd4f13beffef2aa03f88f9" is_pre_election: true
I20260129 02:59:27.020953 20581 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fcddab6bca684adf8ea032289a00f90e" candidate_uuid: "974b802141bd4f13beffef2aa03f88f9" candidate_term: 6 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "1c50de89a5a845668696dcb28c3badd6" is_pre_election: true
I20260129 02:59:27.025090 20441 raft_consensus.cc:1275] T 87d5fe48a44a408aaf03e894a67ecb2f P 197b880fa39d432a9b472845876bfaf0 [term 4 FOLLOWER]: Refusing update from remote peer 7a9daf5d42a84615a6383cfec9a10608: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
W20260129 02:59:27.079787 20553 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.211.1:42027, user_credentials={real_user=slave}} blocked reactor thread for 100238us
I20260129 02:59:27.082320 20435 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7a9a69cd82724b1faed1d10aa091fc4a" candidate_uuid: "1c50de89a5a845668696dcb28c3badd6" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "197b880fa39d432a9b472845876bfaf0" is_pre_election: true
I20260129 02:59:26.992182 22077 consensus_queue.cc:1048] T a0fcc58d6fb6473cafbd9d34d5195269 P 197b880fa39d432a9b472845876bfaf0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.013s
I20260129 02:59:27.088239 20510 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c934d31efd3f4e5195605ddc354d14e1" candidate_uuid: "197b880fa39d432a9b472845876bfaf0" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "974b802141bd4f13beffef2aa03f88f9" is_pre_election: true
I20260129 02:59:27.088551 22106 raft_consensus.cc:2749] T 96609ac68bdf4993a12e1c23355ca013 P 197b880fa39d432a9b472845876bfaf0 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260129 02:59:27.088871 20510 raft_consensus.cc:2468] T c934d31efd3f4e5195605ddc354d14e1 P 974b802141bd4f13beffef2aa03f88f9 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 197b880fa39d432a9b472845876bfaf0 in term 2.
W20260129 02:59:27.089429 20480 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.211.4:40439, user_credentials={real_user=slave}} blocked reactor thread for 65161.9us
I20260129 02:59:27.087704 20584 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "368953c9102643b2a21f99af36b51905" candidate_uuid: "974b802141bd4f13beffef2aa03f88f9" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "1c50de89a5a845668696dcb28c3badd6"
I20260129 02:59:27.000873 22101 raft_consensus.cc:2804] T fcddab6bca684adf8ea032289a00f90e P 1c50de89a5a845668696dcb28c3badd6 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260129 02:59:27.092029 20584 raft_consensus.cc:2393] T 368953c9102643b2a21f99af36b51905 P 1c50de89a5a845668696dcb28c3badd6 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 974b802141bd4f13beffef2aa03f88f9 in current term 3: Already voted for candidate 1c50de89a5a845668696dcb28c3badd6 in this term.
W20260129 02:59:27.113428 20556 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.211.2:34517, user_credentials={real_user=slave}} blocked reactor thread for 94172.7us
I20260129 02:59:26.970952 22069 consensus_queue.cc:1048] T b189f08e16a74a11980cc55d99359a0a P 197b880fa39d432a9b472845876bfaf0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.042s
I20260129 02:59:27.132124 20588 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fcddab6bca684adf8ea032289a00f90e" candidate_uuid: "197b880fa39d432a9b472845876bfaf0" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "1c50de89a5a845668696dcb28c3badd6" is_pre_election: true
W20260129 02:59:27.134239 20407 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.211.2:34517, user_credentials={real_user=slave}} blocked reactor thread for 110699us
I20260129 02:59:27.090500 21946 raft_consensus.cc:493] T c934d31efd3f4e5195605ddc354d14e1 P 974b802141bd4f13beffef2aa03f88f9 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 7a9daf5d42a84615a6383cfec9a10608)
I20260129 02:59:27.163347 20585 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "42d27a28d89b409e8807ec0efd97af69" candidate_uuid: "197b880fa39d432a9b472845876bfaf0" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "1c50de89a5a845668696dcb28c3badd6" is_pre_election: true
I20260129 02:59:27.105930 22101 raft_consensus.cc:493] T fcddab6bca684adf8ea032289a00f90e P 1c50de89a5a845668696dcb28c3badd6 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260129 02:59:27.164994 20585 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a8479d1fe0bd41e5946186e3a6ca8c7a" candidate_uuid: "197b880fa39d432a9b472845876bfaf0" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "1c50de89a5a845668696dcb28c3badd6" is_pre_election: true
I20260129 02:59:26.999923 21998 raft_consensus.cc:515] T a0fcc58d6fb6473cafbd9d34d5195269 P 974b802141bd4f13beffef2aa03f88f9 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:59:27.169423 20510 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3ccac82a01a2445fa5b0ce6c0fcf2295" candidate_uuid: "7a9daf5d42a84615a6383cfec9a10608" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "974b802141bd4f13beffef2aa03f88f9" is_pre_election: true
I20260129 02:59:27.171919 20589 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "dc2f4cb752cf4d59b5f6bc689f5d8104" candidate_uuid: "197b880fa39d432a9b472845876bfaf0" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "1c50de89a5a845668696dcb28c3badd6" is_pre_election: true
I20260129 02:59:27.173448 20589 raft_consensus.cc:2368] T dc2f4cb752cf4d59b5f6bc689f5d8104 P 1c50de89a5a845668696dcb28c3badd6 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 197b880fa39d432a9b472845876bfaf0 for earlier term 2. Current term is 3.
I20260129 02:59:27.172147 20587 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4e3805ffa70242e8ade66da260dddf66" candidate_uuid: "197b880fa39d432a9b472845876bfaf0" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "1c50de89a5a845668696dcb28c3badd6" is_pre_election: true
I20260129 02:59:27.091856 22088 consensus_queue.cc:1048] T cedce331e1584ac8b636c9b8a72b2ba0 P 1c50de89a5a845668696dcb28c3badd6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 1, Time since last communication: 0.183s
I20260129 02:59:27.094291 22053 consensus_queue.cc:1048] T 0678b8b468bd43cdbc3e485900844821 P 7a9daf5d42a84615a6383cfec9a10608 [LEADER]: Connected to new peer: Peer: permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.052s
I20260129 02:59:27.175385 22056 raft_consensus.cc:493] T fd8311b779af4e1fa1402b2a02a602f2 P 1c50de89a5a845668696dcb28c3badd6 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 197b880fa39d432a9b472845876bfaf0)
I20260129 02:59:27.175968 22056 raft_consensus.cc:515] T fd8311b779af4e1fa1402b2a02a602f2 P 1c50de89a5a845668696dcb28c3badd6 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } }
I20260129 02:59:27.178236 22056 leader_election.cc:290] T fd8311b779af4e1fa1402b2a02a602f2 P 1c50de89a5a845668696dcb28c3badd6 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 197b880fa39d432a9b472845876bfaf0 (127.18.211.1:42027), 7a9daf5d42a84615a6383cfec9a10608 (127.18.211.4:40439)
W20260129 02:59:27.178148 20554 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.211.4:40439, user_credentials={real_user=slave}} blocked reactor thread for 67171.8us
W20260129 02:59:27.180011 20554 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260129 02:59:27.180580 21134 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fd8311b779af4e1fa1402b2a02a602f2" candidate_uuid: "1c50de89a5a845668696dcb28c3badd6" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "7a9daf5d42a84615a6383cfec9a10608" is_pre_election: true
W20260129 02:59:27.187036 20407 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.211.2:34517, user_credentials={real_user=slave}} blocked reactor thread for 49158.3us
I20260129 02:59:27.092917 21100 leader_election.cc:304] T dc2f4cb752cf4d59b5f6bc689f5d8104 P 7a9daf5d42a84615a6383cfec9a10608 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1c50de89a5a845668696dcb28c3badd6, 7a9daf5d42a84615a6383cfec9a10608; no voters:
I20260129 02:59:27.105495 21971 consensus_queue.cc:1048] T 926a250fa5834e88b96ccecbefbf5086 P 1c50de89a5a845668696dcb28c3badd6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.065s
I20260129 02:59:27.198318 20407 leader_election.cc:304] T fcddab6bca684adf8ea032289a00f90e P 197b880fa39d432a9b472845876bfaf0 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 197b880fa39d432a9b472845876bfaf0; no voters: 1c50de89a5a845668696dcb28c3badd6, 974b802141bd4f13beffef2aa03f88f9
I20260129 02:59:27.201123 20407 leader_election.cc:304] T c934d31efd3f4e5195605ddc354d14e1 P 197b880fa39d432a9b472845876bfaf0 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 197b880fa39d432a9b472845876bfaf0, 974b802141bd4f13beffef2aa03f88f9; no voters: 7a9daf5d42a84615a6383cfec9a10608
I20260129 02:59:27.139757 22118 consensus_queue.cc:1048] T 87d5fe48a44a408aaf03e894a67ecb2f P 7a9daf5d42a84615a6383cfec9a10608 [LEADER]: Connected to new peer: Peer: permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.050s
I20260129 02:59:27.192205 20323 catalog_manager.cc:4985] TS 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3:37735): tablet 3ccac82a01a2445fa5b0ce6c0fcf2295 (table test-workload [id=de73611ac8e3485a98aecc5add247032]) successfully deleted
I20260129 02:59:27.192453 22119 raft_consensus.cc:493] T 87d5fe48a44a408aaf03e894a67ecb2f P 197b880fa39d432a9b472845876bfaf0 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260129 02:59:27.181876 21988 consensus_queue.cc:1048] T cedce331e1584ac8b636c9b8a72b2ba0 P 1c50de89a5a845668696dcb28c3badd6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.012s
I20260129 02:59:27.223284 20510 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7a9a69cd82724b1faed1d10aa091fc4a" candidate_uuid: "1c50de89a5a845668696dcb28c3badd6" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "974b802141bd4f13beffef2aa03f88f9" is_pre_election: true
W20260129 02:59:27.208559 20323 outbound_call.cc:321] RPC callback for RPC call kudu.tserver.TabletServerAdminService.DeleteTablet -> {remote=127.18.211.3:37735, user_credentials={real_user=slave}} blocked reactor thread for 91305.1us
I20260129 02:59:27.172647 22101 raft_consensus.cc:3060] T fcddab6bca684adf8ea032289a00f90e P 1c50de89a5a845668696dcb28c3badd6 [term 5 FOLLOWER]: Advancing to term 6
I20260129 02:59:27.241791 20405 leader_election.cc:400] T dc2f4cb752cf4d59b5f6bc689f5d8104 P 197b880fa39d432a9b472845876bfaf0 [CANDIDATE]: Term 2 pre-election: Vote denied by peer 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3:37735) with higher term. Message: Invalid argument: T dc2f4cb752cf4d59b5f6bc689f5d8104 P 1c50de89a5a845668696dcb28c3badd6 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 197b880fa39d432a9b472845876bfaf0 for earlier term 2. Current term is 3.
I20260129 02:59:27.242282 20405 leader_election.cc:403] T dc2f4cb752cf4d59b5f6bc689f5d8104 P 197b880fa39d432a9b472845876bfaf0 [CANDIDATE]: Term 2 pre-election: Cancelling election due to peer responding with higher term
I20260129 02:59:27.092517 22115 consensus_queue.cc:1048] T e33a417cf9c54cf5af73a4ee3aac4d3a P 1c50de89a5a845668696dcb28c3badd6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.195s
I20260129 02:59:27.179468 21946 raft_consensus.cc:515] T c934d31efd3f4e5195605ddc354d14e1 P 974b802141bd4f13beffef2aa03f88f9 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } }
I20260129 02:59:27.229112 22053 raft_consensus.cc:2804] T dc2f4cb752cf4d59b5f6bc689f5d8104 P 7a9daf5d42a84615a6383cfec9a10608 [term 3 FOLLOWER]: Leader election won for term 3
W20260129 02:59:27.220108 21100 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.211.3:37735, user_credentials={real_user=slave}} blocked reactor thread for 207323us
I20260129 02:59:27.188181 22087 consensus_queue.cc:1048] T b189f08e16a74a11980cc55d99359a0a P 197b880fa39d432a9b472845876bfaf0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.051s
I20260129 02:59:27.179518 22075 consensus_queue.cc:237] T 368953c9102643b2a21f99af36b51905 P 974b802141bd4f13beffef2aa03f88f9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:59:27.253655 22105 raft_consensus.cc:3060] T dc2f4cb752cf4d59b5f6bc689f5d8104 P 197b880fa39d432a9b472845876bfaf0 [term 1 FOLLOWER]: Advancing to term 3
I20260129 02:59:27.235602 22119 raft_consensus.cc:515] T 87d5fe48a44a408aaf03e894a67ecb2f P 197b880fa39d432a9b472845876bfaf0 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } }
I20260129 02:59:27.270483 22126 consensus_queue.cc:1048] T 76f713c3331540d2a5b76164103e9eb1 P 1c50de89a5a845668696dcb28c3badd6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.087s
I20260129 02:59:27.250101 22003 raft_consensus.cc:493] T 75b0b6e82ddc4f92b1fe1d94f9a0b73b P 974b802141bd4f13beffef2aa03f88f9 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 1c50de89a5a845668696dcb28c3badd6)
I20260129 02:59:27.259505 22053 raft_consensus.cc:697] T dc2f4cb752cf4d59b5f6bc689f5d8104 P 7a9daf5d42a84615a6383cfec9a10608 [term 3 LEADER]: Becoming Leader. State: Replica: 7a9daf5d42a84615a6383cfec9a10608, State: Running, Role: LEADER
W20260129 02:59:27.292081 20407 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.211.2:34517, user_credentials={real_user=slave}} blocked reactor thread for 87214.4us
I20260129 02:59:27.293525 20407 leader_election.cc:304] T a8479d1fe0bd41e5946186e3a6ca8c7a P 197b880fa39d432a9b472845876bfaf0 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 197b880fa39d432a9b472845876bfaf0; no voters: 1c50de89a5a845668696dcb28c3badd6, 974b802141bd4f13beffef2aa03f88f9
I20260129 02:59:27.301957 21099 leader_election.cc:304] T 3ccac82a01a2445fa5b0ce6c0fcf2295 P 7a9daf5d42a84615a6383cfec9a10608 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7a9daf5d42a84615a6383cfec9a10608; no voters: 197b880fa39d432a9b472845876bfaf0, 974b802141bd4f13beffef2aa03f88f9
I20260129 02:59:27.304626 21946 leader_election.cc:290] T c934d31efd3f4e5195605ddc354d14e1 P 974b802141bd4f13beffef2aa03f88f9 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 197b880fa39d432a9b472845876bfaf0 (127.18.211.1:42027), 7a9daf5d42a84615a6383cfec9a10608 (127.18.211.4:40439)
I20260129 02:59:27.307714 22059 raft_consensus.cc:2749] T 3ccac82a01a2445fa5b0ce6c0fcf2295 P 7a9daf5d42a84615a6383cfec9a10608 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260129 02:59:27.309022 22049 consensus_queue.cc:1048] T 87d5fe48a44a408aaf03e894a67ecb2f P 7a9daf5d42a84615a6383cfec9a10608 [LEADER]: Connected to new peer: Peer: permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260129 02:59:27.317029 22120 raft_consensus.cc:2749] T a8479d1fe0bd41e5946186e3a6ca8c7a P 197b880fa39d432a9b472845876bfaf0 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260129 02:59:27.293459 22070 raft_consensus.cc:2804] T c934d31efd3f4e5195605ddc354d14e1 P 197b880fa39d432a9b472845876bfaf0 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260129 02:59:27.333236 22119 leader_election.cc:290] T 87d5fe48a44a408aaf03e894a67ecb2f P 197b880fa39d432a9b472845876bfaf0 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 974b802141bd4f13beffef2aa03f88f9 (127.18.211.2:34517), 7a9daf5d42a84615a6383cfec9a10608 (127.18.211.4:40439)
I20260129 02:59:27.329995 22070 raft_consensus.cc:493] T c934d31efd3f4e5195605ddc354d14e1 P 197b880fa39d432a9b472845876bfaf0 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 7a9daf5d42a84615a6383cfec9a10608)
I20260129 02:59:27.285842 22003 raft_consensus.cc:515] T 75b0b6e82ddc4f92b1fe1d94f9a0b73b P 974b802141bd4f13beffef2aa03f88f9 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } }
I20260129 02:59:27.340770 22037 raft_consensus.cc:3060] T 75be4bd6bd45474795fd472eff2953ac P 1c50de89a5a845668696dcb28c3badd6 [term 5 FOLLOWER]: Advancing to term 6
I20260129 02:59:27.343012 21127 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "42d27a28d89b409e8807ec0efd97af69" candidate_uuid: "197b880fa39d432a9b472845876bfaf0" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "7a9daf5d42a84615a6383cfec9a10608" is_pre_election: true
I20260129 02:59:27.322420 22089 raft_consensus.cc:3060] T fcddab6bca684adf8ea032289a00f90e P 197b880fa39d432a9b472845876bfaf0 [term 4 FOLLOWER]: Advancing to term 5
I20260129 02:59:27.347100 22070 raft_consensus.cc:3060] T c934d31efd3f4e5195605ddc354d14e1 P 197b880fa39d432a9b472845876bfaf0 [term 2 FOLLOWER]: Advancing to term 3
I20260129 02:59:27.346714 21998 leader_election.cc:290] T a0fcc58d6fb6473cafbd9d34d5195269 P 974b802141bd4f13beffef2aa03f88f9 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 197b880fa39d432a9b472845876bfaf0 (127.18.211.1:42027), 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3:37735)
I20260129 02:59:27.355787 20434 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "368953c9102643b2a21f99af36b51905" candidate_uuid: "1c50de89a5a845668696dcb28c3badd6" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "197b880fa39d432a9b472845876bfaf0" is_pre_election: true
I20260129 02:59:27.360823 21127 raft_consensus.cc:2468] T 42d27a28d89b409e8807ec0efd97af69 P 7a9daf5d42a84615a6383cfec9a10608 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 197b880fa39d432a9b472845876bfaf0 in term 3.
I20260129 02:59:27.344677 22053 consensus_queue.cc:237] T dc2f4cb752cf4d59b5f6bc689f5d8104 P 7a9daf5d42a84615a6383cfec9a10608 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } }
I20260129 02:59:27.414817 22091 raft_consensus.cc:493] T a1a39dcd328e4268ae41b988c66b4a75 P 197b880fa39d432a9b472845876bfaf0 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 1c50de89a5a845668696dcb28c3badd6)
I20260129 02:59:27.423202 22092 raft_consensus.cc:493] T 76f713c3331540d2a5b76164103e9eb1 P 197b880fa39d432a9b472845876bfaf0 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 1c50de89a5a845668696dcb28c3badd6)
I20260129 02:59:27.384186 20434 raft_consensus.cc:2468] T 368953c9102643b2a21f99af36b51905 P 197b880fa39d432a9b472845876bfaf0 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1c50de89a5a845668696dcb28c3badd6 in term 3.
I20260129 02:59:27.395991 22082 raft_consensus.cc:493] T 36ff33be666a4c82954731691dbcf327 P 197b880fa39d432a9b472845876bfaf0 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 974b802141bd4f13beffef2aa03f88f9)
I20260129 02:59:27.414786 22011 raft_consensus.cc:493] T 1ce4ddf5b10e4ae8b193b13b2e5936ee P 197b880fa39d432a9b472845876bfaf0 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 1c50de89a5a845668696dcb28c3badd6)
I20260129 02:59:27.403759 22050 raft_consensus.cc:493] T 505580c93c694bbb8ea96fdf9b17ab46 P 7a9daf5d42a84615a6383cfec9a10608 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 974b802141bd4f13beffef2aa03f88f9)
I20260129 02:59:27.446455 21998 raft_consensus.cc:493] T 45cc5f3d2688486d9b6dbe7e666dedde P 974b802141bd4f13beffef2aa03f88f9 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 1c50de89a5a845668696dcb28c3badd6)
I20260129 02:59:27.430193 22092 raft_consensus.cc:515] T 76f713c3331540d2a5b76164103e9eb1 P 197b880fa39d432a9b472845876bfaf0 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:59:27.423647 22091 raft_consensus.cc:515] T a1a39dcd328e4268ae41b988c66b4a75 P 197b880fa39d432a9b472845876bfaf0 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } } peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } }
I20260129 02:59:27.490064 20432 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c934d31efd3f4e5195605ddc354d14e1" candidate_uuid: "974b802141bd4f13beffef2aa03f88f9" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "197b880fa39d432a9b472845876bfaf0" is_pre_election: true
I20260129 02:59:27.460875 22011 raft_consensus.cc:515] T 1ce4ddf5b10e4ae8b193b13b2e5936ee P 197b880fa39d432a9b472845876bfaf0 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:59:27.508145 20439 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a0fcc58d6fb6473cafbd9d34d5195269" candidate_uuid: "974b802141bd4f13beffef2aa03f88f9" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "197b880fa39d432a9b472845876bfaf0" is_pre_election: true
I20260129 02:59:27.470484 22050 raft_consensus.cc:515] T 505580c93c694bbb8ea96fdf9b17ab46 P 7a9daf5d42a84615a6383cfec9a10608 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } }
I20260129 02:59:27.479725 21998 raft_consensus.cc:515] T 45cc5f3d2688486d9b6dbe7e666dedde P 974b802141bd4f13beffef2aa03f88f9 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } }
I20260129 02:59:27.470842 22082 raft_consensus.cc:515] T 36ff33be666a4c82954731691dbcf327 P 197b880fa39d432a9b472845876bfaf0 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:59:27.526381 20432 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fd8311b779af4e1fa1402b2a02a602f2" candidate_uuid: "1c50de89a5a845668696dcb28c3badd6" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "197b880fa39d432a9b472845876bfaf0" is_pre_election: true
I20260129 02:59:27.511126 22086 raft_consensus.cc:493] T 75be4bd6bd45474795fd472eff2953ac P 197b880fa39d432a9b472845876bfaf0 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 1c50de89a5a845668696dcb28c3badd6)
I20260129 02:59:27.507683 22003 leader_election.cc:290] T 75b0b6e82ddc4f92b1fe1d94f9a0b73b P 974b802141bd4f13beffef2aa03f88f9 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3:37735), 7a9daf5d42a84615a6383cfec9a10608 (127.18.211.4:40439)
W20260129 02:59:27.545852 20405 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.211.3:37735, user_credentials={real_user=slave}} blocked reactor thread for 304228us
I20260129 02:59:27.536154 22076 raft_consensus.cc:493] T 3e2d15f8640b407195210c476e09df91 P 7a9daf5d42a84615a6383cfec9a10608 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 1c50de89a5a845668696dcb28c3badd6)
I20260129 02:59:27.551770 22082 leader_election.cc:290] T 36ff33be666a4c82954731691dbcf327 P 197b880fa39d432a9b472845876bfaf0 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 974b802141bd4f13beffef2aa03f88f9 (127.18.211.2:34517), 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3:37735)
I20260129 02:59:27.556805 22076 raft_consensus.cc:515] T 3e2d15f8640b407195210c476e09df91 P 7a9daf5d42a84615a6383cfec9a10608 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } }
I20260129 02:59:27.558015 21130 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "87d5fe48a44a408aaf03e894a67ecb2f" candidate_uuid: "197b880fa39d432a9b472845876bfaf0" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "7a9daf5d42a84615a6383cfec9a10608" is_pre_election: true
I20260129 02:59:27.559099 22076 leader_election.cc:290] T 3e2d15f8640b407195210c476e09df91 P 7a9daf5d42a84615a6383cfec9a10608 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 974b802141bd4f13beffef2aa03f88f9 (127.18.211.2:34517), 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3:37735)
W20260129 02:59:27.568490 22130 log.cc:927] Time spent T 7d9494f5584d4d5db8b9adf1c8fc888b P 974b802141bd4f13beffef2aa03f88f9: Append to log took a long time: real 0.054s user 0.001s sys 0.000s
W20260129 02:59:27.569941 22134 log.cc:927] Time spent T 42d27a28d89b409e8807ec0efd97af69 P 1c50de89a5a845668696dcb28c3badd6: Append to log took a long time: real 0.075s user 0.001s sys 0.000s
I20260129 02:59:27.570623 22011 leader_election.cc:290] T 1ce4ddf5b10e4ae8b193b13b2e5936ee P 197b880fa39d432a9b472845876bfaf0 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 974b802141bd4f13beffef2aa03f88f9 (127.18.211.2:34517), 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3:37735)
I20260129 02:59:27.565070 22113 raft_consensus.cc:493] T c2633a6fc34c44579de348d208b0c01e P 974b802141bd4f13beffef2aa03f88f9 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 197b880fa39d432a9b472845876bfaf0)
I20260129 02:59:27.573865 20512 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "87d5fe48a44a408aaf03e894a67ecb2f" candidate_uuid: "197b880fa39d432a9b472845876bfaf0" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "974b802141bd4f13beffef2aa03f88f9" is_pre_election: true
I20260129 02:59:27.574144 20405 leader_election.cc:304] T 42d27a28d89b409e8807ec0efd97af69 P 197b880fa39d432a9b472845876bfaf0 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 197b880fa39d432a9b472845876bfaf0, 7a9daf5d42a84615a6383cfec9a10608; no voters: 1c50de89a5a845668696dcb28c3badd6
I20260129 02:59:27.578992 20589 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a0fcc58d6fb6473cafbd9d34d5195269" candidate_uuid: "974b802141bd4f13beffef2aa03f88f9" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1c50de89a5a845668696dcb28c3badd6" is_pre_election: true
I20260129 02:59:27.579027 20553 leader_election.cc:304] T 368953c9102643b2a21f99af36b51905 P 1c50de89a5a845668696dcb28c3badd6 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 197b880fa39d432a9b472845876bfaf0, 1c50de89a5a845668696dcb28c3badd6; no voters:
I20260129 02:59:27.580159 20585 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "75b0b6e82ddc4f92b1fe1d94f9a0b73b" candidate_uuid: "974b802141bd4f13beffef2aa03f88f9" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "1c50de89a5a845668696dcb28c3badd6" is_pre_election: true
I20260129 02:59:27.581032 21130 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c934d31efd3f4e5195605ddc354d14e1" candidate_uuid: "974b802141bd4f13beffef2aa03f88f9" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "7a9daf5d42a84615a6383cfec9a10608" is_pre_election: true
I20260129 02:59:27.580950 22062 consensus_queue.cc:1048] T 78aa39096868415d9afc2ffcb248e198 P 1c50de89a5a845668696dcb28c3badd6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20260129 02:59:27.561096 21998 leader_election.cc:290] T 45cc5f3d2688486d9b6dbe7e666dedde P 974b802141bd4f13beffef2aa03f88f9 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3:37735), 7a9daf5d42a84615a6383cfec9a10608 (127.18.211.4:40439)
I20260129 02:59:27.572393 22022 raft_consensus.cc:493] T 87d5fe48a44a408aaf03e894a67ecb2f P 974b802141bd4f13beffef2aa03f88f9 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260129 02:59:27.584822 20553 leader_election.cc:304] T fd8311b779af4e1fa1402b2a02a602f2 P 1c50de89a5a845668696dcb28c3badd6 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1c50de89a5a845668696dcb28c3badd6; no voters: 197b880fa39d432a9b472845876bfaf0, 7a9daf5d42a84615a6383cfec9a10608
I20260129 02:59:27.566468 22003 raft_consensus.cc:493] T a5a05fbcbbcf4916bbc01fd1a5b5e90d P 974b802141bd4f13beffef2aa03f88f9 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260129 02:59:27.600782 22091 leader_election.cc:290] T a1a39dcd328e4268ae41b988c66b4a75 P 197b880fa39d432a9b472845876bfaf0 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 974b802141bd4f13beffef2aa03f88f9 (127.18.211.2:34517), 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3:37735)
W20260129 02:59:27.577037 21984 consensus_meta.cc:287] T 1ce4ddf5b10e4ae8b193b13b2e5936ee P 1c50de89a5a845668696dcb28c3badd6: Time spent flushing consensus metadata: real 0.685s user 0.007s sys 0.006s
I20260129 02:59:27.595930 21136 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "75b0b6e82ddc4f92b1fe1d94f9a0b73b" candidate_uuid: "974b802141bd4f13beffef2aa03f88f9" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "7a9daf5d42a84615a6383cfec9a10608" is_pre_election: true
I20260129 02:59:27.597990 20515 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1ce4ddf5b10e4ae8b193b13b2e5936ee" candidate_uuid: "197b880fa39d432a9b472845876bfaf0" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "974b802141bd4f13beffef2aa03f88f9" is_pre_election: true
I20260129 02:59:27.608244 20511 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "36ff33be666a4c82954731691dbcf327" candidate_uuid: "197b880fa39d432a9b472845876bfaf0" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "974b802141bd4f13beffef2aa03f88f9" is_pre_election: true
I20260129 02:59:27.657879 20515 raft_consensus.cc:2393] T 1ce4ddf5b10e4ae8b193b13b2e5936ee P 974b802141bd4f13beffef2aa03f88f9 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 197b880fa39d432a9b472845876bfaf0 in current term 3: Already voted for candidate 974b802141bd4f13beffef2aa03f88f9 in this term.
I20260129 02:59:27.592785 22107 raft_consensus.cc:2804] T 368953c9102643b2a21f99af36b51905 P 1c50de89a5a845668696dcb28c3badd6 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260129 02:59:27.565482 22101 raft_consensus.cc:515] T fcddab6bca684adf8ea032289a00f90e P 1c50de89a5a845668696dcb28c3badd6 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:59:27.549162 22086 raft_consensus.cc:515] T 75be4bd6bd45474795fd472eff2953ac P 197b880fa39d432a9b472845876bfaf0 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:59:27.664414 20507 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "76f713c3331540d2a5b76164103e9eb1" candidate_uuid: "197b880fa39d432a9b472845876bfaf0" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "974b802141bd4f13beffef2aa03f88f9" is_pre_election: true
I20260129 02:59:27.664592 20511 tablet_service.cc:2066] Received LeaderStepDown RPC: tablet_id: "75f45538dc704d9a8ec4ff8ad9f3ea5b"
dest_uuid: "974b802141bd4f13beffef2aa03f88f9"
mode: GRACEFUL
new_leader_uuid: "1c50de89a5a845668696dcb28c3badd6"
from {username='slave'} at 127.0.0.1:52008
I20260129 02:59:27.673676 20511 raft_consensus.cc:606] T 75f45538dc704d9a8ec4ff8ad9f3ea5b P 974b802141bd4f13beffef2aa03f88f9 [term 1 LEADER]: Received request to transfer leadership to TS 1c50de89a5a845668696dcb28c3badd6
I20260129 02:59:27.662868 20515 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a1a39dcd328e4268ae41b988c66b4a75" candidate_uuid: "197b880fa39d432a9b472845876bfaf0" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "974b802141bd4f13beffef2aa03f88f9" is_pre_election: true
W20260129 02:59:27.666081 20407 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.211.2:34517, user_credentials={real_user=slave}} blocked reactor thread for 60650.5us
W20260129 02:59:27.666175 20405 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.211.4:40439, user_credentials={real_user=slave}} blocked reactor thread for 71050.4us
I20260129 02:59:27.590493 22113 raft_consensus.cc:515] T c2633a6fc34c44579de348d208b0c01e P 974b802141bd4f13beffef2aa03f88f9 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:59:27.677183 20587 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3e2d15f8640b407195210c476e09df91" candidate_uuid: "7a9daf5d42a84615a6383cfec9a10608" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "1c50de89a5a845668696dcb28c3badd6" is_pre_election: true
I20260129 02:59:27.661041 22028 raft_consensus.cc:493] T 0678b8b468bd43cdbc3e485900844821 P 974b802141bd4f13beffef2aa03f88f9 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 7a9daf5d42a84615a6383cfec9a10608)
I20260129 02:59:27.672363 20583 raft_consensus.cc:2410] T 1ce4ddf5b10e4ae8b193b13b2e5936ee P 1c50de89a5a845668696dcb28c3badd6 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 974b802141bd4f13beffef2aa03f88f9 for term 3 because replica has last-logged OpId of term: 2 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 1.
I20260129 02:59:27.676231 20507 raft_consensus.cc:2468] T 76f713c3331540d2a5b76164103e9eb1 P 974b802141bd4f13beffef2aa03f88f9 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 197b880fa39d432a9b472845876bfaf0 in term 2.
I20260129 02:59:27.687664 20407 leader_election.cc:304] T 76f713c3331540d2a5b76164103e9eb1 P 197b880fa39d432a9b472845876bfaf0 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 197b880fa39d432a9b472845876bfaf0, 974b802141bd4f13beffef2aa03f88f9; no voters:
I20260129 02:59:27.657383 22137 raft_consensus.cc:493] T cedce331e1584ac8b636c9b8a72b2ba0 P 974b802141bd4f13beffef2aa03f88f9 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260129 02:59:27.661901 22092 leader_election.cc:290] T 76f713c3331540d2a5b76164103e9eb1 P 197b880fa39d432a9b472845876bfaf0 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 974b802141bd4f13beffef2aa03f88f9 (127.18.211.2:34517), 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3:37735)
I20260129 02:59:27.692982 20581 raft_consensus.cc:2393] T fcddab6bca684adf8ea032289a00f90e P 1c50de89a5a845668696dcb28c3badd6 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 974b802141bd4f13beffef2aa03f88f9 in current term 6: Already voted for candidate 1c50de89a5a845668696dcb28c3badd6 in this term.
W20260129 02:59:27.695338 20407 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.18.211.2:34517 timed out after 1.663s (SENT) [suppressed 33 similar messages]
I20260129 02:59:27.609817 22003 raft_consensus.cc:515] T a5a05fbcbbcf4916bbc01fd1a5b5e90d P 974b802141bd4f13beffef2aa03f88f9 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:59:27.686844 22115 raft_consensus.cc:493] T 1ce4ddf5b10e4ae8b193b13b2e5936ee P 1c50de89a5a845668696dcb28c3badd6 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260129 02:59:27.677166 22107 raft_consensus.cc:493] T 368953c9102643b2a21f99af36b51905 P 1c50de89a5a845668696dcb28c3badd6 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260129 02:59:27.603482 22022 raft_consensus.cc:515] T 87d5fe48a44a408aaf03e894a67ecb2f P 974b802141bd4f13beffef2aa03f88f9 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } }
I20260129 02:59:27.766664 22050 leader_election.cc:290] T 505580c93c694bbb8ea96fdf9b17ab46 P 7a9daf5d42a84615a6383cfec9a10608 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 974b802141bd4f13beffef2aa03f88f9 (127.18.211.2:34517), 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3:37735)
I20260129 02:59:27.692313 22066 raft_consensus.cc:493] T 1299723a4476420ab02fd8bfd7aa0645 P 1c50de89a5a845668696dcb28c3badd6 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260129 02:59:27.794528 22133 log.cc:927] Time spent T 78aa39096868415d9afc2ffcb248e198 P 7a9daf5d42a84615a6383cfec9a10608: Append to log took a long time: real 0.255s user 0.000s sys 0.001s
I20260129 02:59:27.728700 22079 raft_consensus.cc:493] T a4c1226b987b468fa84dfc6ba0e2dbe1 P 1c50de89a5a845668696dcb28c3badd6 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260129 02:59:27.799559 22086 leader_election.cc:290] T 75be4bd6bd45474795fd472eff2953ac P 197b880fa39d432a9b472845876bfaf0 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 974b802141bd4f13beffef2aa03f88f9 (127.18.211.2:34517), 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3:37735)
I20260129 02:59:27.801141 20507 tablet_service.cc:2066] Received LeaderStepDown RPC: tablet_id: "7d9494f5584d4d5db8b9adf1c8fc888b"
dest_uuid: "974b802141bd4f13beffef2aa03f88f9"
mode: GRACEFUL
new_leader_uuid: "197b880fa39d432a9b472845876bfaf0"
from {username='slave'} at 127.0.0.1:52008
I20260129 02:59:27.804244 20507 raft_consensus.cc:606] T 7d9494f5584d4d5db8b9adf1c8fc888b P 974b802141bd4f13beffef2aa03f88f9 [term 2 FOLLOWER]: Received request to transfer leadership to TS 197b880fa39d432a9b472845876bfaf0
I20260129 02:59:27.804698 20507 raft_consensus.cc:614] T 7d9494f5584d4d5db8b9adf1c8fc888b P 974b802141bd4f13beffef2aa03f88f9 [term 2 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260129 02:59:27.690639 22028 raft_consensus.cc:515] T 0678b8b468bd43cdbc3e485900844821 P 974b802141bd4f13beffef2aa03f88f9 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } }
I20260129 02:59:27.806998 20512 raft_consensus.cc:2468] T 87d5fe48a44a408aaf03e894a67ecb2f P 974b802141bd4f13beffef2aa03f88f9 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 197b880fa39d432a9b472845876bfaf0 in term 4.
I20260129 02:59:27.768347 22107 raft_consensus.cc:3060] T 368953c9102643b2a21f99af36b51905 P 1c50de89a5a845668696dcb28c3badd6 [term 3 FOLLOWER]: Advancing to term 4
I20260129 02:59:27.805524 22140 raft_consensus.cc:2749] T fd8311b779af4e1fa1402b2a02a602f2 P 1c50de89a5a845668696dcb28c3badd6 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260129 02:59:27.815937 22141 raft_consensus.cc:2804] T 42d27a28d89b409e8807ec0efd97af69 P 197b880fa39d432a9b472845876bfaf0 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260129 02:59:27.816566 22141 raft_consensus.cc:493] T 42d27a28d89b409e8807ec0efd97af69 P 197b880fa39d432a9b472845876bfaf0 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260129 02:59:27.816908 22141 raft_consensus.cc:3060] T 42d27a28d89b409e8807ec0efd97af69 P 197b880fa39d432a9b472845876bfaf0 [term 3 FOLLOWER]: Advancing to term 4
I20260129 02:59:27.797210 22137 raft_consensus.cc:515] T cedce331e1584ac8b636c9b8a72b2ba0 P 974b802141bd4f13beffef2aa03f88f9 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:59:27.812453 22101 leader_election.cc:290] T fcddab6bca684adf8ea032289a00f90e P 1c50de89a5a845668696dcb28c3badd6 [CANDIDATE]: Term 6 election: Requested vote from peers 197b880fa39d432a9b472845876bfaf0 (127.18.211.1:42027), 974b802141bd4f13beffef2aa03f88f9 (127.18.211.2:34517)
I20260129 02:59:27.819517 20586 raft_consensus.cc:2410] T 75be4bd6bd45474795fd472eff2953ac P 1c50de89a5a845668696dcb28c3badd6 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 974b802141bd4f13beffef2aa03f88f9 for term 6 because replica has last-logged OpId of term: 5 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 4 index: 3.
I20260129 02:59:27.803263 22062 raft_consensus.cc:493] T 3906b080a5b34a74b8b64128635e8997 P 1c50de89a5a845668696dcb28c3badd6 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 197b880fa39d432a9b472845876bfaf0)
I20260129 02:59:27.839804 20507 tablet_service.cc:2066] Received LeaderStepDown RPC: tablet_id: "8055232cf0004d40a85f3bf983fe00a9"
dest_uuid: "974b802141bd4f13beffef2aa03f88f9"
mode: GRACEFUL
new_leader_uuid: "197b880fa39d432a9b472845876bfaf0"
from {username='slave'} at 127.0.0.1:52008
I20260129 02:59:27.840790 20507 raft_consensus.cc:606] T 8055232cf0004d40a85f3bf983fe00a9 P 974b802141bd4f13beffef2aa03f88f9 [term 1 LEADER]: Received request to transfer leadership to TS 197b880fa39d432a9b472845876bfaf0
I20260129 02:59:27.820379 22126 raft_consensus.cc:493] T c891d2bff4ce4568861bbc992e91fbb0 P 1c50de89a5a845668696dcb28c3badd6 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 974b802141bd4f13beffef2aa03f88f9)
I20260129 02:59:27.801117 22066 raft_consensus.cc:515] T 1299723a4476420ab02fd8bfd7aa0645 P 1c50de89a5a845668696dcb28c3badd6 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } }
I20260129 02:59:27.847738 20587 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "505580c93c694bbb8ea96fdf9b17ab46" candidate_uuid: "7a9daf5d42a84615a6383cfec9a10608" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "1c50de89a5a845668696dcb28c3badd6" is_pre_election: true
I20260129 02:59:27.847777 20514 raft_consensus.cc:1275] T a4c1226b987b468fa84dfc6ba0e2dbe1 P 974b802141bd4f13beffef2aa03f88f9 [term 2 FOLLOWER]: Refusing update from remote peer 197b880fa39d432a9b472845876bfaf0: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
W20260129 02:59:27.851912 20508 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:57600 (request call id 1587) took 1814 ms (client timeout 1663 ms). Trace:
W20260129 02:59:27.852355 20508 rpcz_store.cc:269] 0129 02:59:26.032758 (+ 0us) service_pool.cc:168] Inserting onto call queue
0129 02:59:26.032949 (+ 191us) service_pool.cc:225] Handling call
0129 02:59:27.846439 (+1813490us) spinlock_profiling.cc:243] Waited 770 ms on lock 0x7b540012a054. stack: 00007f8fe044e373 00007f8fe044e270 00007f8fdfda0574 00007f8fe9a5c6b4 00007f8fe9a5c649 00007f8fe34975a1 00007f8fe3531f7a 00007f8fe84e3647 00007f8fe1a73834 00007f8fe1a737c2 00007f8fe1a736f1 00007f8fe1a7366c 00007f8fe1a72922 00007f8fe186d34c 00007f8fe186c786 00007f8fe186c12f
0129 02:59:27.846600 (+ 161us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1770054400}
I20260129 02:59:27.857419 19276 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 4
I20260129 02:59:27.800664 22115 raft_consensus.cc:515] T 1ce4ddf5b10e4ae8b193b13b2e5936ee P 1c50de89a5a845668696dcb28c3badd6 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:59:27.859634 19276 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20260129 02:59:27.838924 22062 raft_consensus.cc:515] T 3906b080a5b34a74b8b64128635e8997 P 1c50de89a5a845668696dcb28c3badd6 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
W20260129 02:59:27.897359 20553 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260129 02:59:27.874096 20556 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.625s user 0.007s sys 0.027s
I20260129 02:59:27.805859 22079 raft_consensus.cc:515] T a4c1226b987b468fa84dfc6ba0e2dbe1 P 1c50de89a5a845668696dcb28c3badd6 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
W20260129 02:59:27.914494 20556 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.710s user 0.008s sys 0.027s
W20260129 02:59:27.901504 22075 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.588s user 0.016s sys 0.026s
I20260129 02:59:27.874687 22140 raft_consensus.cc:493] T a8479d1fe0bd41e5946186e3a6ca8c7a P 1c50de89a5a845668696dcb28c3badd6 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 974b802141bd4f13beffef2aa03f88f9)
I20260129 02:59:27.977391 20556 leader_election.cc:304] T 7a9a69cd82724b1faed1d10aa091fc4a P 1c50de89a5a845668696dcb28c3badd6 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1c50de89a5a845668696dcb28c3badd6; no voters: 197b880fa39d432a9b472845876bfaf0, 974b802141bd4f13beffef2aa03f88f9
W20260129 02:59:27.988865 22148 log.cc:927] Time spent T 1299723a4476420ab02fd8bfd7aa0645 P 197b880fa39d432a9b472845876bfaf0: Append to log took a long time: real 0.212s user 0.001s sys 0.000s
I20260129 02:59:27.988996 22028 leader_election.cc:290] T 0678b8b468bd43cdbc3e485900844821 P 974b802141bd4f13beffef2aa03f88f9 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 197b880fa39d432a9b472845876bfaf0 (127.18.211.1:42027), 7a9daf5d42a84615a6383cfec9a10608 (127.18.211.4:40439)
I20260129 02:59:27.884354 22113 leader_election.cc:290] T c2633a6fc34c44579de348d208b0c01e P 974b802141bd4f13beffef2aa03f88f9 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 197b880fa39d432a9b472845876bfaf0 (127.18.211.1:42027), 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3:37735)
I20260129 02:59:27.974809 22022 leader_election.cc:290] T 87d5fe48a44a408aaf03e894a67ecb2f P 974b802141bd4f13beffef2aa03f88f9 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 197b880fa39d432a9b472845876bfaf0 (127.18.211.1:42027), 7a9daf5d42a84615a6383cfec9a10608 (127.18.211.4:40439)
W20260129 02:59:27.977066 22075 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.664s user 0.016s sys 0.026s
I20260129 02:59:28.001781 22143 raft_consensus.cc:493] T 8055232cf0004d40a85f3bf983fe00a9 P 197b880fa39d432a9b472845876bfaf0 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 974b802141bd4f13beffef2aa03f88f9)
I20260129 02:59:28.004722 22143 raft_consensus.cc:515] T 8055232cf0004d40a85f3bf983fe00a9 P 197b880fa39d432a9b472845876bfaf0 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:59:27.977494 22003 leader_election.cc:290] T a5a05fbcbbcf4916bbc01fd1a5b5e90d P 974b802141bd4f13beffef2aa03f88f9 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 197b880fa39d432a9b472845876bfaf0 (127.18.211.1:42027), 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3:37735)
I20260129 02:59:27.861711 22126 raft_consensus.cc:515] T c891d2bff4ce4568861bbc992e91fbb0 P 1c50de89a5a845668696dcb28c3badd6 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:59:27.996544 22064 raft_consensus.cc:2749] T 7a9a69cd82724b1faed1d10aa091fc4a P 1c50de89a5a845668696dcb28c3badd6 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260129 02:59:28.018743 21134 raft_consensus.cc:1275] T 1299723a4476420ab02fd8bfd7aa0645 P 7a9daf5d42a84615a6383cfec9a10608 [term 4 FOLLOWER]: Refusing update from remote peer 197b880fa39d432a9b472845876bfaf0: Log matching property violated. Preceding OpId in replica: term: 2 index: 4. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
W20260129 02:59:28.023715 22105 consensus_meta.cc:287] T dc2f4cb752cf4d59b5f6bc689f5d8104 P 197b880fa39d432a9b472845876bfaf0: Time spent flushing consensus metadata: real 0.743s user 0.009s sys 0.004s
I20260129 02:59:28.016517 22091 raft_consensus.cc:493] T a609ac02fae84725ab743f95f51d33ce P 197b880fa39d432a9b472845876bfaf0 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 7a9daf5d42a84615a6383cfec9a10608)
I20260129 02:59:28.077129 22137 leader_election.cc:290] T cedce331e1584ac8b636c9b8a72b2ba0 P 974b802141bd4f13beffef2aa03f88f9 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 197b880fa39d432a9b472845876bfaf0 (127.18.211.1:42027), 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3:37735)
I20260129 02:59:28.163080 22126 leader_election.cc:290] T c891d2bff4ce4568861bbc992e91fbb0 P 1c50de89a5a845668696dcb28c3badd6 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 197b880fa39d432a9b472845876bfaf0 (127.18.211.1:42027), 974b802141bd4f13beffef2aa03f88f9 (127.18.211.2:34517)
I20260129 02:59:28.028378 22062 leader_election.cc:290] T 3906b080a5b34a74b8b64128635e8997 P 1c50de89a5a845668696dcb28c3badd6 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 197b880fa39d432a9b472845876bfaf0 (127.18.211.1:42027), 974b802141bd4f13beffef2aa03f88f9 (127.18.211.2:34517)
I20260129 02:59:28.143952 22053 raft_consensus.cc:493] T 75f45538dc704d9a8ec4ff8ad9f3ea5b P 7a9daf5d42a84615a6383cfec9a10608 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 974b802141bd4f13beffef2aa03f88f9)
I20260129 02:59:28.167431 22053 raft_consensus.cc:515] T 75f45538dc704d9a8ec4ff8ad9f3ea5b P 7a9daf5d42a84615a6383cfec9a10608 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } }
I20260129 02:59:28.146730 22105 raft_consensus.cc:2749] T dc2f4cb752cf4d59b5f6bc689f5d8104 P 197b880fa39d432a9b472845876bfaf0 [term 3 FOLLOWER]: Leader pre-election lost for term 2. Reason: Vote denied by peer 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3:37735) with higher term. Message: Invalid argument: T dc2f4cb752cf4d59b5f6bc689f5d8104 P 1c50de89a5a845668696dcb28c3badd6 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 197b880fa39d432a9b472845876bfaf0 for earlier term 2. Current term is 3.
I20260129 02:59:28.091815 22011 consensus_queue.cc:1048] T 1299723a4476420ab02fd8bfd7aa0645 P 197b880fa39d432a9b472845876bfaf0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 4, Time since last communication: 0.078s
I20260129 02:59:28.008211 22140 raft_consensus.cc:515] T a8479d1fe0bd41e5946186e3a6ca8c7a P 1c50de89a5a845668696dcb28c3badd6 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:59:28.183358 22062 raft_consensus.cc:493] T 75be4bd6bd45474795fd472eff2953ac P 1c50de89a5a845668696dcb28c3badd6 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260129 02:59:28.136579 22079 leader_election.cc:290] T a4c1226b987b468fa84dfc6ba0e2dbe1 P 1c50de89a5a845668696dcb28c3badd6 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 197b880fa39d432a9b472845876bfaf0 (127.18.211.1:42027), 974b802141bd4f13beffef2aa03f88f9 (127.18.211.2:34517)
I20260129 02:59:28.258387 20516 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:48544 (request call id 106) took 1119 ms. Trace:
I20260129 02:59:28.101814 22100 raft_consensus.cc:493] T 0f20af7fbe5d4db89a6f17948d98912c P 197b880fa39d432a9b472845876bfaf0 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 974b802141bd4f13beffef2aa03f88f9)
W20260129 02:59:28.275200 21128 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:47844 (request call id 239) took 2519 ms (client timeout 1983 ms). Trace:
I20260129 02:59:28.278784 20437 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55676 (request call id 1042) took 1617 ms. Trace:
I20260129 02:59:28.283674 20581 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:35752 (request call id 1419) took 1263 ms. Trace:
I20260129 02:59:28.289522 20510 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fcddab6bca684adf8ea032289a00f90e" candidate_uuid: "1c50de89a5a845668696dcb28c3badd6" candidate_term: 6 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "974b802141bd4f13beffef2aa03f88f9"
I20260129 02:59:28.268859 22137 raft_consensus.cc:493] T a1a39dcd328e4268ae41b988c66b4a75 P 974b802141bd4f13beffef2aa03f88f9 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 1c50de89a5a845668696dcb28c3badd6)
I20260129 02:59:28.293952 20438 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55672 (request call id 1352) took 1435 ms. Trace:
I20260129 02:59:28.269518 22062 raft_consensus.cc:515] T 75be4bd6bd45474795fd472eff2953ac P 1c50de89a5a845668696dcb28c3badd6 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:59:28.187780 22066 leader_election.cc:290] T 1299723a4476420ab02fd8bfd7aa0645 P 1c50de89a5a845668696dcb28c3badd6 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 197b880fa39d432a9b472845876bfaf0 (127.18.211.1:42027), 7a9daf5d42a84615a6383cfec9a10608 (127.18.211.4:40439)
I20260129 02:59:28.265570 20516 rpcz_store.cc:276] 0129 02:59:27.139002 (+ 0us) service_pool.cc:168] Inserting onto call queue
0129 02:59:27.139180 (+ 178us) service_pool.cc:225] Handling call
0129 02:59:27.182081 (+ 42901us) raft_consensus.cc:1471] Updating replica for 0 ops
0129 02:59:27.260471 (+ 78390us) raft_consensus.cc:1534] Early marking committed up to index 0
0129 02:59:27.260502 (+ 31us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0129 02:59:27.260544 (+ 42us) raft_consensus.cc:1645] Marking committed up to 5
0129 02:59:27.260573 (+ 29us) raft_consensus.cc:1697] Filling consensus response to leader.
0129 02:59:28.258022 (+997449us) spinlock_profiling.cc:243] Waited 70 ms on lock 0x7b54002ef754. stack: 00007f8fe044e373 00007f8fe044e270 00007f8fdfda0574 00007f8fe9a5c6b4 00007f8fe9a5c649 00007f8fe34975a1 00007f8fe352ea81 00007f8fe352d4be 00007f8fe84e1f49 00007f8fe1a6ed34 00007f8fe1a6ecc2 00007f8fe1a6ebf1 00007f8fe1a6eb6c 00007f8fe1a6de22 00007f8fe186d34c 00007f8fe186c786
0129 02:59:28.258093 (+ 71us) raft_consensus.cc:1691] UpdateReplicas() finished
0129 02:59:28.258301 (+ 208us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":160934272}
I20260129 02:59:28.306855 20510 raft_consensus.cc:3060] T fcddab6bca684adf8ea032289a00f90e P 974b802141bd4f13beffef2aa03f88f9 [term 5 FOLLOWER]: Advancing to term 6
I20260129 02:59:28.306641 20581 rpcz_store.cc:276] 0129 02:59:27.019968 (+ 0us) service_pool.cc:168] Inserting onto call queue
0129 02:59:27.020691 (+ 723us) service_pool.cc:225] Handling call
0129 02:59:28.283395 (+1262704us) spinlock_profiling.cc:243] Waited 669 ms on lock 0x7b540003edd4. stack: 00007f8fe044e373 00007f8fe044e270 00007f8fdfda0574 00007f8fe9a5c6b4 00007f8fe9a5c649 00007f8fe34975a1 00007f8fe3531f7a 00007f8fe84e3647 00007f8fe1a73834 00007f8fe1a737c2 00007f8fe1a736f1 00007f8fe1a7366c 00007f8fe1a72922 00007f8fe186d34c 00007f8fe186c786 00007f8fe186c12f
0129 02:59:28.283578 (+ 183us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1538149632}
I20260129 02:59:28.307284 20437 rpcz_store.cc:276] 0129 02:59:26.661729 (+ 0us) service_pool.cc:168] Inserting onto call queue
0129 02:59:26.662686 (+ 957us) service_pool.cc:225] Handling call
0129 02:59:26.863319 (+200633us) spinlock_profiling.cc:243] Waited 9.94 ms on lock 0x7b5800000f98. stack: 00007f8fe044e373 00007f8fe044e270 00007f8fdfda0574 00007f8fe9a5c6b4 00007f8fe9a5c649 00007f8fe02c08b1 00007f8fe0442db1 00007f8fe0442c62 00007f8fe92eddd9 00007f8fe846d260 00007f8fe8535dbb 00007f8fe8535e41 00007f8fe84f7af0 00007f8fe84e2459 00007f8fe84e1f02 00007f8fe1a6ed34
0129 02:59:28.277763 (+1414444us) spinlock_profiling.cc:243] Waited 7.14 ms on lock 0x7b5800000f98. stack: 00007f8fe044e373 00007f8fe044e270 00007f8fdfda0574 00007f8fe9a5c6b4 00007f8fe9a5c649 00007f8fe02c08b1 00007f8fe04430dc 00007f8fe0442c9b 00007f8fe92eddd9 00007f8fe846d260 00007f8fe8535dbb 00007f8fe8535e41 00007f8fe84f7af0 00007f8fe84e2459 00007f8fe84e1f02 00007f8fe1a6ed34
0129 02:59:28.278014 (+ 251us) raft_consensus.cc:1471] Updating replica for 0 ops
0129 02:59:28.278244 (+ 230us) raft_consensus.cc:1534] Early marking committed up to index 0
0129 02:59:28.278281 (+ 37us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0129 02:59:28.278437 (+ 156us) raft_consensus.cc:1645] Marking committed up to 2
0129 02:59:28.278484 (+ 47us) raft_consensus.cc:1697] Filling consensus response to leader.
0129 02:59:28.278561 (+ 77us) raft_consensus.cc:1691] UpdateReplicas() finished
0129 02:59:28.278689 (+ 128us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":39282944}
I20260129 02:59:28.312330 20438 rpcz_store.cc:276] 0129 02:59:26.858800 (+ 0us) service_pool.cc:168] Inserting onto call queue
0129 02:59:26.863131 (+ 4331us) service_pool.cc:225] Handling call
0129 02:59:27.532832 (+669701us) spinlock_profiling.cc:243] Waited 6.6 ms on lock 0x7b5800000f98. stack: 00007f8fe044e373 00007f8fe044e270 00007f8fdfda0574 00007f8fe9a5c6b4 00007f8fe9a5c649 00007f8fe02c08b1 00007f8fe0442db1 00007f8fe0442c62 00007f8fe92eddd9 00007f8fe846d260 00007f8fe8535dbb 00007f8fe8535e41 00007f8fe84f7af0 00007f8fe84e2459 00007f8fe84e1f02 00007f8fe1a6ed34
0129 02:59:27.533018 (+ 186us) raft_consensus.cc:1471] Updating replica for 0 ops
0129 02:59:27.545088 (+ 12070us) raft_consensus.cc:1534] Early marking committed up to index 0
0129 02:59:27.545135 (+ 47us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0129 02:59:27.545199 (+ 64us) raft_consensus.cc:1645] Marking committed up to 1
0129 02:59:27.545246 (+ 47us) raft_consensus.cc:1697] Filling consensus response to leader.
0129 02:59:28.293648 (+748402us) spinlock_profiling.cc:243] Waited 11.8 ms on lock 0x7b54001423d4. stack: 00007f8fe044e373 00007f8fe044e270 00007f8fdfda0574 00007f8fe9a5c6b4 00007f8fe9a5c649 00007f8fe34975a1 00007f8fe352ea81 00007f8fe352d4be 00007f8fe84e1f49 00007f8fe1a6ed34 00007f8fe1a6ecc2 00007f8fe1a6ebf1 00007f8fe1a6eb6c 00007f8fe1a6de22 00007f8fe186d34c 00007f8fe186c786
0129 02:59:28.293713 (+ 65us) raft_consensus.cc:1691] UpdateReplicas() finished
0129 02:59:28.293869 (+ 156us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":42370688}
I20260129 02:59:28.147734 22091 raft_consensus.cc:515] T a609ac02fae84725ab743f95f51d33ce P 197b880fa39d432a9b472845876bfaf0 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } }
I20260129 02:59:28.189323 22143 leader_election.cc:290] T 8055232cf0004d40a85f3bf983fe00a9 P 197b880fa39d432a9b472845876bfaf0 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 974b802141bd4f13beffef2aa03f88f9 (127.18.211.2:34517), 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3:37735)
W20260129 02:59:28.302523 21128 rpcz_store.cc:269] 0129 02:59:25.756016 (+ 0us) service_pool.cc:168] Inserting onto call queue
0129 02:59:25.767970 (+ 11954us) service_pool.cc:225] Handling call
0129 02:59:28.274927 (+2506957us) spinlock_profiling.cc:243] Waited 826 ms on lock 0x7b54000b27d4. stack: 00007f8fe044e373 00007f8fe044e270 00007f8fdfda0574 00007f8fe9a5c6b4 00007f8fe9a5c649 00007f8fe34975a1 00007f8fe3531f7a 00007f8fe84e3647 00007f8fe1a73834 00007f8fe1a737c2 00007f8fe1a736f1 00007f8fe1a7366c 00007f8fe1a72922 00007f8fe186d34c 00007f8fe186c786 00007f8fe186c12f
0129 02:59:28.275090 (+ 163us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1899276544}
W20260129 02:59:28.331724 20439 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.770s user 0.028s sys 0.047s
W20260129 02:59:28.363138 20439 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.802s user 0.028s sys 0.047s
I20260129 02:59:28.337765 21131 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "dc2f4cb752cf4d59b5f6bc689f5d8104" candidate_uuid: "197b880fa39d432a9b472845876bfaf0" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "7a9daf5d42a84615a6383cfec9a10608" is_pre_election: true
I20260129 02:59:28.319979 22137 raft_consensus.cc:515] T a1a39dcd328e4268ae41b988c66b4a75 P 974b802141bd4f13beffef2aa03f88f9 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } } peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } }
I20260129 02:59:28.301116 22141 raft_consensus.cc:515] T 42d27a28d89b409e8807ec0efd97af69 P 197b880fa39d432a9b472845876bfaf0 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } }
I20260129 02:59:28.370886 20510 raft_consensus.cc:2468] T fcddab6bca684adf8ea032289a00f90e P 974b802141bd4f13beffef2aa03f88f9 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1c50de89a5a845668696dcb28c3badd6 in term 6.
I20260129 02:59:28.342692 22053 leader_election.cc:290] T 75f45538dc704d9a8ec4ff8ad9f3ea5b P 7a9daf5d42a84615a6383cfec9a10608 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 974b802141bd4f13beffef2aa03f88f9 (127.18.211.2:34517), 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3:37735)
I20260129 02:59:28.339246 22100 raft_consensus.cc:515] T 0f20af7fbe5d4db89a6f17948d98912c P 197b880fa39d432a9b472845876bfaf0 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
W20260129 02:59:28.387171 21131 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34060 (request call id 413) took 1365 ms (client timeout 1803 ms). Trace:
I20260129 02:59:28.373981 20437 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c2633a6fc34c44579de348d208b0c01e" candidate_uuid: "974b802141bd4f13beffef2aa03f88f9" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "197b880fa39d432a9b472845876bfaf0" is_pre_election: true
W20260129 02:59:28.377328 22089 consensus_meta.cc:287] T fcddab6bca684adf8ea032289a00f90e P 197b880fa39d432a9b472845876bfaf0: Time spent flushing consensus metadata: real 1.012s user 0.013s sys 0.000s
I20260129 02:59:28.380452 22140 leader_election.cc:290] T a8479d1fe0bd41e5946186e3a6ca8c7a P 1c50de89a5a845668696dcb28c3badd6 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 197b880fa39d432a9b472845876bfaf0 (127.18.211.1:42027), 974b802141bd4f13beffef2aa03f88f9 (127.18.211.2:34517)
W20260129 02:59:28.398490 22116 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.824s user 0.026s sys 0.059s
W20260129 02:59:28.398885 22116 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.918s user 0.026s sys 0.060s
I20260129 02:59:28.400235 20441 raft_consensus.cc:1240] T 42d27a28d89b409e8807ec0efd97af69 P 197b880fa39d432a9b472845876bfaf0 [term 4 FOLLOWER]: Rejecting Update request from peer 1c50de89a5a845668696dcb28c3badd6 for earlier term 3. Current term is 4. Ops: []
W20260129 02:59:28.400873 20432 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.821s user 0.019s sys 0.086s
W20260129 02:59:28.402383 20432 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.823s user 0.019s sys 0.086s
I20260129 02:59:28.373981 21132 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4e3805ffa70242e8ade66da260dddf66" candidate_uuid: "197b880fa39d432a9b472845876bfaf0" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "7a9daf5d42a84615a6383cfec9a10608" is_pre_election: true
W20260129 02:59:28.388505 21131 rpcz_store.cc:269] 0129 02:59:27.013857 (+ 0us) service_pool.cc:168] Inserting onto call queue
0129 02:59:27.014012 (+ 155us) service_pool.cc:225] Handling call
0129 02:59:28.337545 (+1323533us) spinlock_profiling.cc:243] Waited 4.83 ms on lock 0x7b5800111f98. stack: 00007f8fe044e373 00007f8fe044e270 00007f8fdfda0574 00007f8fe9a5c6b4 00007f8fe9a5c649 00007f8fe02c08b1 00007f8fe0442db1 00007f8fe0442c62 00007f8fe92eddd9 00007f8fe846d260 00007f8fe8535dbb 00007f8fe8535e41 00007f8fe84e3bd0 00007f8fe84e34b4 00007f8fe1a73834 00007f8fe1a737c2
0129 02:59:28.379229 (+ 41684us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":11112320}
I20260129 02:59:28.397687 22115 leader_election.cc:290] T 1ce4ddf5b10e4ae8b193b13b2e5936ee P 1c50de89a5a845668696dcb28c3badd6 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 197b880fa39d432a9b472845876bfaf0 (127.18.211.1:42027), 974b802141bd4f13beffef2aa03f88f9 (127.18.211.2:34517)
W20260129 02:59:28.401438 20435 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.826s user 0.077s sys 0.151s
W20260129 02:59:28.407460 20435 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.841s user 0.077s sys 0.152s
W20260129 02:59:28.408277 20586 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:35752 (request call id 1413) took 1926 ms (client timeout 2081 ms). Trace:
W20260129 02:59:28.408808 20586 rpcz_store.cc:269] 0129 02:59:26.484659 (+ 0us) service_pool.cc:168] Inserting onto call queue
0129 02:59:26.486772 (+ 2113us) service_pool.cc:225] Handling call
0129 02:59:28.408036 (+1921264us) spinlock_profiling.cc:243] Waited 1.33 s on lock 0x7b540018bfd4. stack: 00007f8fe044e373 00007f8fe044e270 00007f8fdfda0574 00007f8fe9a5c6b4 00007f8fe9a5c649 00007f8fe34975a1 00007f8fe3531f7a 00007f8fe84e3647 00007f8fe1a73834 00007f8fe1a737c2 00007f8fe1a736f1 00007f8fe1a7366c 00007f8fe1a72922 00007f8fe186d34c 00007f8fe186c786 00007f8fe186c12f
0129 02:59:28.408205 (+ 169us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":3060378496}
I20260129 02:59:28.422140 21133 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:47844 (request call id 248) took 1390 ms. Trace:
I20260129 02:59:28.422269 21132 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34060 (request call id 410) took 1431 ms. Trace:
I20260129 02:59:28.401480 22089 raft_consensus.cc:2749] T fcddab6bca684adf8ea032289a00f90e P 197b880fa39d432a9b472845876bfaf0 [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260129 02:59:28.422693 22100 leader_election.cc:290] T 0f20af7fbe5d4db89a6f17948d98912c P 197b880fa39d432a9b472845876bfaf0 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 974b802141bd4f13beffef2aa03f88f9 (127.18.211.2:34517), 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3:37735)
I20260129 02:59:28.422806 21132 rpcz_store.cc:276] 0129 02:59:26.990419 (+ 0us) service_pool.cc:168] Inserting onto call queue
0129 02:59:26.992582 (+ 2163us) service_pool.cc:225] Handling call
0129 02:59:28.354906 (+1362324us) spinlock_profiling.cc:243] Waited 66.7 ms on lock 0x7b5800111f98. stack: 00007f8fe044e373 00007f8fe044e270 00007f8fdfda0574 00007f8fe9a5c6b4 00007f8fe9a5c649 00007f8fe02c08b1 00007f8fe0442db1 00007f8fe0442c62 00007f8fe92eddd9 00007f8fe846d260 00007f8fe8535dbb 00007f8fe8535e41 00007f8fe84e3bd0 00007f8fe84e34b4 00007f8fe1a73834 00007f8fe1a737c2
0129 02:59:28.422196 (+ 67290us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":153416960}
I20260129 02:59:28.423830 22091 leader_election.cc:290] T a609ac02fae84725ab743f95f51d33ce P 197b880fa39d432a9b472845876bfaf0 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 974b802141bd4f13beffef2aa03f88f9 (127.18.211.2:34517), 7a9daf5d42a84615a6383cfec9a10608 (127.18.211.4:40439)
I20260129 02:59:28.403054 20336 catalog_manager.cc:5654] T 45cc5f3d2688486d9b6dbe7e666dedde P 974b802141bd4f13beffef2aa03f88f9 reported cstate change: term changed from 1 to 3. New cstate: current_term: 3 leader_uuid: "1c50de89a5a845668696dcb28c3badd6" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } } }
I20260129 02:59:28.422684 21133 rpcz_store.cc:276] 0129 02:59:27.016703 (+ 0us) service_pool.cc:168] Inserting onto call queue
0129 02:59:27.016828 (+ 125us) service_pool.cc:225] Handling call
0129 02:59:27.482406 (+465578us) spinlock_profiling.cc:243] Waited 3.58 ms on lock 0x7b5800111f98. stack: 00007f8fe044e373 00007f8fe044e270 00007f8fdfda0574 00007f8fe9a5c6b4 00007f8fe9a5c649 00007f8fe02c08b1 00007f8fe0442db1 00007f8fe0442c62 00007f8fe92eddd9 00007f8fe846d260 00007f8fe8535dbb 00007f8fe8535e41 00007f8fe84f7af0 00007f8fe84e2459 00007f8fe84e1f02 00007f8fe1a6ed34
0129 02:59:27.488716 (+ 6310us) raft_consensus.cc:1471] Updating replica for 0 ops
0129 02:59:27.534589 (+ 45873us) raft_consensus.cc:1534] Early marking committed up to index 0
0129 02:59:27.534635 (+ 46us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0129 02:59:27.534691 (+ 56us) raft_consensus.cc:1645] Marking committed up to 5
0129 02:59:27.534734 (+ 43us) raft_consensus.cc:1697] Filling consensus response to leader.
0129 02:59:28.406940 (+872206us) spinlock_profiling.cc:243] Waited 45.6 ms on lock 0x7b540073ee54. stack: 00007f8fe044e373 00007f8fe044e270 00007f8fdfda0574 00007f8fe9a5c6b4 00007f8fe9a5c649 00007f8fe34975a1 00007f8fe352ea81 00007f8fe352d4be 00007f8fe84e1f49 00007f8fe1a6ed34 00007f8fe1a6ecc2 00007f8fe1a6ebf1 00007f8fe1a6eb6c 00007f8fe1a6de22 00007f8fe186d34c 00007f8fe186c786
0129 02:59:28.406992 (+ 52us) raft_consensus.cc:1691] UpdateReplicas() finished
0129 02:59:28.407128 (+ 136us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":113128448}
I20260129 02:59:28.432327 22141 leader_election.cc:290] T 42d27a28d89b409e8807ec0efd97af69 P 197b880fa39d432a9b472845876bfaf0 [CANDIDATE]: Term 4 election: Requested vote from peers 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3:37735), 7a9daf5d42a84615a6383cfec9a10608 (127.18.211.4:40439)
I20260129 02:59:28.383175 20438 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fcddab6bca684adf8ea032289a00f90e" candidate_uuid: "1c50de89a5a845668696dcb28c3badd6" candidate_term: 6 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "197b880fa39d432a9b472845876bfaf0"
I20260129 02:59:28.439006 20438 raft_consensus.cc:3060] T fcddab6bca684adf8ea032289a00f90e P 197b880fa39d432a9b472845876bfaf0 [term 5 FOLLOWER]: Advancing to term 6
I20260129 02:59:28.428894 22137 leader_election.cc:290] T a1a39dcd328e4268ae41b988c66b4a75 P 974b802141bd4f13beffef2aa03f88f9 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3:37735), 197b880fa39d432a9b472845876bfaf0 (127.18.211.1:42027)
I20260129 02:59:28.391618 20437 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a4c1226b987b468fa84dfc6ba0e2dbe1" candidate_uuid: "1c50de89a5a845668696dcb28c3badd6" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "197b880fa39d432a9b472845876bfaf0" is_pre_election: true
I20260129 02:59:28.490887 20438 raft_consensus.cc:2468] T fcddab6bca684adf8ea032289a00f90e P 197b880fa39d432a9b472845876bfaf0 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1c50de89a5a845668696dcb28c3badd6 in term 6.
I20260129 02:59:28.459206 21974 raft_consensus.cc:493] T a4c1226b987b468fa84dfc6ba0e2dbe1 P 974b802141bd4f13beffef2aa03f88f9 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260129 02:59:28.460243 22043 raft_consensus.cc:493] T 7d9494f5584d4d5db8b9adf1c8fc888b P 974b802141bd4f13beffef2aa03f88f9 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 1c50de89a5a845668696dcb28c3badd6)
W20260129 02:59:28.496186 22070 consensus_meta.cc:287] T c934d31efd3f4e5195605ddc354d14e1 P 197b880fa39d432a9b472845876bfaf0: Time spent flushing consensus metadata: real 1.126s user 0.010s sys 0.008s
I20260129 02:59:28.650527 20438 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a8479d1fe0bd41e5946186e3a6ca8c7a" candidate_uuid: "1c50de89a5a845668696dcb28c3badd6" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "197b880fa39d432a9b472845876bfaf0" is_pre_election: true
I20260129 02:59:28.464931 22062 leader_election.cc:290] T 75be4bd6bd45474795fd472eff2953ac P 1c50de89a5a845668696dcb28c3badd6 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 197b880fa39d432a9b472845876bfaf0 (127.18.211.1:42027), 974b802141bd4f13beffef2aa03f88f9 (127.18.211.2:34517)
I20260129 02:59:28.504287 20437 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "87d5fe48a44a408aaf03e894a67ecb2f" candidate_uuid: "974b802141bd4f13beffef2aa03f88f9" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "197b880fa39d432a9b472845876bfaf0" is_pre_election: true
W20260129 02:59:28.506469 22049 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.904s user 0.010s sys 0.044s
W20260129 02:59:28.502486 22118 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.829s user 0.008s sys 0.026s
W20260129 02:59:28.685957 22049 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 1.084s user 0.010s sys 0.044s
W20260129 02:59:28.686148 22118 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 1.014s user 0.008s sys 0.027s
I20260129 02:59:28.501716 22141 raft_consensus.cc:493] T 18f30ebaa4174fd28c958e72d0cc7e24 P 197b880fa39d432a9b472845876bfaf0 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 7a9daf5d42a84615a6383cfec9a10608)
I20260129 02:59:28.525859 22122 raft_consensus.cc:493] T 1299723a4476420ab02fd8bfd7aa0645 P 7a9daf5d42a84615a6383cfec9a10608 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260129 02:59:28.533581 22107 consensus_meta.cc:287] T 368953c9102643b2a21f99af36b51905 P 1c50de89a5a845668696dcb28c3badd6: Time spent flushing consensus metadata: real 0.702s user 0.007s sys 0.007s
I20260129 02:59:28.534693 20440 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.GetConsensusState from 127.0.0.1:55666 (request call id 1046) took 1519 ms. Trace:
W20260129 02:59:28.539485 20513 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.853s user 0.000s sys 0.014s
W20260129 02:59:28.604208 22139 thread.cc:641] apply [worker] (thread pool) Time spent creating pthread: real 1.019s user 0.059s sys 0.096s
I20260129 02:59:28.699846 20440 rpcz_store.cc:276] 0129 02:59:27.015332 (+ 0us) service_pool.cc:168] Inserting onto call queue
0129 02:59:27.017238 (+ 1906us) service_pool.cc:225] Handling call
0129 02:59:28.533861 (+1516623us) spinlock_profiling.cc:243] Waited 515 ms on lock 0x7b54003511d4. stack: 00007f8fe044e373 00007f8fe044e270 00007f8fdfda0574 00007f8fe9a5c6b4 00007f8fe9a5c649 00007f8fe34980c0 00007f8fe3536bba 00007f8fe84e758e 00007f8fe1a86437 00007f8fe1a863c2 00007f8fe1a862f1 00007f8fe1a8626c 00007f8fe1a85522 00007f8fe186d34c 00007f8fe186c786 00007f8fe186c12f
0129 02:59:28.534583 (+ 722us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1183470976}
W20260129 02:59:28.701360 20513 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 1.024s user 0.001s sys 0.014s
W20260129 02:59:28.610091 20480 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.952s user 0.008s sys 0.006s
W20260129 02:59:28.610617 20556 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.616s user 0.003s sys 0.010s
W20260129 02:59:28.610417 20407 thread.cc:641] dns-resolver [worker] (thread pool) Time spent creating pthread: real 0.870s user 0.002s sys 0.013s
W20260129 02:59:28.608690 20436 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.912s user 0.017s sys 0.035s
W20260129 02:59:28.611800 22035 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.751s user 0.009s sys 0.004s
W20260129 02:59:28.711055 22035 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.850s user 0.010s sys 0.004s
W20260129 02:59:28.620205 20555 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.622s user 0.010s sys 0.013s
W20260129 02:59:28.626206 20405 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.599s user 0.013s sys 0.036s
I20260129 02:59:28.631239 20553 leader_election.cc:304] T fcddab6bca684adf8ea032289a00f90e P 1c50de89a5a845668696dcb28c3badd6 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 197b880fa39d432a9b472845876bfaf0, 1c50de89a5a845668696dcb28c3badd6; no voters:
W20260129 02:59:28.647279 20554 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.211.4:40439, user_credentials={real_user=slave}} blocked reactor thread for 482558us
I20260129 02:59:28.618685 22091 raft_consensus.cc:493] T c891d2bff4ce4568861bbc992e91fbb0 P 197b880fa39d432a9b472845876bfaf0 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 974b802141bd4f13beffef2aa03f88f9)
I20260129 02:59:28.653180 20438 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1ce4ddf5b10e4ae8b193b13b2e5936ee" candidate_uuid: "1c50de89a5a845668696dcb28c3badd6" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "197b880fa39d432a9b472845876bfaf0" is_pre_election: true
I20260129 02:59:28.713708 20438 raft_consensus.cc:2468] T 1ce4ddf5b10e4ae8b193b13b2e5936ee P 197b880fa39d432a9b472845876bfaf0 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1c50de89a5a845668696dcb28c3badd6 in term 2.
W20260129 02:59:28.703573 22139 thread.cc:608] apply [worker] (thread pool) Time spent starting thread: real 1.120s user 0.059s sys 0.096s
I20260129 02:59:28.648049 22028 raft_consensus.cc:493] T 4b1876f9a70b49edb3aa74105e66d3b3 P 974b802141bd4f13beffef2aa03f88f9 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 1c50de89a5a845668696dcb28c3badd6)
W20260129 02:59:28.671926 20509 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.644s user 0.012s sys 0.021s
W20260129 02:59:28.707856 20556 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.713s user 0.004s sys 0.010s
I20260129 02:59:28.704818 22122 raft_consensus.cc:515] T 1299723a4476420ab02fd8bfd7aa0645 P 7a9daf5d42a84615a6383cfec9a10608 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } attrs { replace: true } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } }
I20260129 02:59:28.649909 21974 raft_consensus.cc:515] T a4c1226b987b468fa84dfc6ba0e2dbe1 P 974b802141bd4f13beffef2aa03f88f9 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
W20260129 02:59:28.711673 20555 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.714s user 0.010s sys 0.013s
W20260129 02:59:28.708026 20407 thread.cc:608] dns-resolver [worker] (thread pool) Time spent starting thread: real 0.968s user 0.002s sys 0.013s
W20260129 02:59:28.730830 20407 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.211.2:34517, user_credentials={real_user=slave}} blocked reactor thread for 1.03547e+06us
I20260129 02:59:28.709043 20437 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c891d2bff4ce4568861bbc992e91fbb0" candidate_uuid: "1c50de89a5a845668696dcb28c3badd6" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "197b880fa39d432a9b472845876bfaf0" is_pre_election: true
I20260129 02:59:28.661026 22043 raft_consensus.cc:515] T 7d9494f5584d4d5db8b9adf1c8fc888b P 974b802141bd4f13beffef2aa03f88f9 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:59:28.708534 20440 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a1a39dcd328e4268ae41b988c66b4a75" candidate_uuid: "974b802141bd4f13beffef2aa03f88f9" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "197b880fa39d432a9b472845876bfaf0" is_pre_election: true
W20260129 02:59:28.711823 20405 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.685s user 0.013s sys 0.037s
W20260129 02:59:28.751842 20405 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.211.4:40439, user_credentials={real_user=slave}} blocked reactor thread for 724752us
W20260129 02:59:28.708905 20436 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 1.017s user 0.017s sys 0.035s
I20260129 02:59:28.761497 20586 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1ce4ddf5b10e4ae8b193b13b2e5936ee" candidate_uuid: "197b880fa39d432a9b472845876bfaf0" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "1c50de89a5a845668696dcb28c3badd6" is_pre_election: true
I20260129 02:59:28.709342 20439 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55676 (request call id 1047) took 1187 ms. Trace:
W20260129 02:59:28.710758 20480 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 1.053s user 0.009s sys 0.006s
I20260129 02:59:28.724473 20438 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0678b8b468bd43cdbc3e485900844821" candidate_uuid: "974b802141bd4f13beffef2aa03f88f9" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "197b880fa39d432a9b472845876bfaf0" is_pre_election: true
I20260129 02:59:28.711716 22092 raft_consensus.cc:2804] T 76f713c3331540d2a5b76164103e9eb1 P 197b880fa39d432a9b472845876bfaf0 [term 2 FOLLOWER]: Leader pre-election won for term 3
W20260129 02:59:28.716804 20553 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.211.1:42027, user_credentials={real_user=slave}} blocked reactor thread for 85699.3us
I20260129 02:59:28.758368 21133 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1299723a4476420ab02fd8bfd7aa0645" candidate_uuid: "1c50de89a5a845668696dcb28c3badd6" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "7a9daf5d42a84615a6383cfec9a10608" is_pre_election: true
I20260129 02:59:28.772109 20440 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1299723a4476420ab02fd8bfd7aa0645" candidate_uuid: "1c50de89a5a845668696dcb28c3badd6" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "197b880fa39d432a9b472845876bfaf0" is_pre_election: true
I20260129 02:59:28.778220 20438 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3906b080a5b34a74b8b64128635e8997" candidate_uuid: "1c50de89a5a845668696dcb28c3badd6" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "197b880fa39d432a9b472845876bfaf0" is_pre_election: true
I20260129 02:59:28.772575 20581 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "76f713c3331540d2a5b76164103e9eb1" candidate_uuid: "197b880fa39d432a9b472845876bfaf0" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "1c50de89a5a845668696dcb28c3badd6" is_pre_election: true
I20260129 02:59:28.772843 20589 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "75be4bd6bd45474795fd472eff2953ac" candidate_uuid: "197b880fa39d432a9b472845876bfaf0" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "1c50de89a5a845668696dcb28c3badd6" is_pre_election: true
I20260129 02:59:28.779322 20589 raft_consensus.cc:2468] T 75be4bd6bd45474795fd472eff2953ac P 1c50de89a5a845668696dcb28c3badd6 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 197b880fa39d432a9b472845876bfaf0 in term 6.
I20260129 02:59:28.780682 20585 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "42d27a28d89b409e8807ec0efd97af69" candidate_uuid: "197b880fa39d432a9b472845876bfaf0" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "1c50de89a5a845668696dcb28c3badd6"
I20260129 02:59:28.780998 20589 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0f20af7fbe5d4db89a6f17948d98912c" candidate_uuid: "197b880fa39d432a9b472845876bfaf0" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "1c50de89a5a845668696dcb28c3badd6" is_pre_election: true
I20260129 02:59:28.707267 22156 raft_consensus.cc:493] T add8d4923d1a489e86882c3a25692b95 P 7a9daf5d42a84615a6383cfec9a10608 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 1c50de89a5a845668696dcb28c3badd6)
I20260129 02:59:28.790777 20438 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a5a05fbcbbcf4916bbc01fd1a5b5e90d" candidate_uuid: "974b802141bd4f13beffef2aa03f88f9" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "197b880fa39d432a9b472845876bfaf0" is_pre_election: true
I20260129 02:59:28.791599 20438 raft_consensus.cc:2468] T a5a05fbcbbcf4916bbc01fd1a5b5e90d P 197b880fa39d432a9b472845876bfaf0 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 974b802141bd4f13beffef2aa03f88f9 in term 4.
I20260129 02:59:28.764720 20439 rpcz_store.cc:276] 0129 02:59:27.522199 (+ 0us) service_pool.cc:168] Inserting onto call queue
0129 02:59:27.522356 (+ 157us) service_pool.cc:225] Handling call
0129 02:59:27.536615 (+ 14259us) raft_consensus.cc:1471] Updating replica for 0 ops
0129 02:59:27.536802 (+ 187us) raft_consensus.cc:1534] Early marking committed up to index 2
0129 02:59:28.363767 (+826965us) log.cc:844] Serialized 14 byte log entry
0129 02:59:28.706520 (+342753us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0129 02:59:28.706590 (+ 70us) raft_consensus.cc:1645] Marking committed up to 2
0129 02:59:28.708124 (+ 1534us) raft_consensus.cc:1697] Filling consensus response to leader.
0129 02:59:28.709089 (+ 965us) raft_consensus.cc:1691] UpdateReplicas() finished
0129 02:59:28.709271 (+ 182us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":1068333,"prepare.run_cpu_time_us":65,"prepare.run_wall_time_us":1730,"raft.queue_time_us":2305,"raft.run_cpu_time_us":123,"raft.run_wall_time_us":2413,"thread_start_us":1141240,"threads_started":2}
W20260129 02:59:28.725001 20509 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.689s user 0.013s sys 0.021s
I20260129 02:59:28.701709 22141 raft_consensus.cc:515] T 18f30ebaa4174fd28c958e72d0cc7e24 P 197b880fa39d432a9b472845876bfaf0 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } }
I20260129 02:59:28.701762 22107 raft_consensus.cc:515] T 368953c9102643b2a21f99af36b51905 P 1c50de89a5a845668696dcb28c3badd6 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:59:28.688652 22041 raft_consensus.cc:493] T e7d83d4ccf684e76acb29a0dc844e1b8 P 974b802141bd4f13beffef2aa03f88f9 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 1c50de89a5a845668696dcb28c3badd6)
I20260129 02:59:28.774464 20588 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a1a39dcd328e4268ae41b988c66b4a75" candidate_uuid: "197b880fa39d432a9b472845876bfaf0" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "1c50de89a5a845668696dcb28c3badd6" is_pre_election: true
I20260129 02:59:28.782351 21133 raft_consensus.cc:2468] T 1299723a4476420ab02fd8bfd7aa0645 P 7a9daf5d42a84615a6383cfec9a10608 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1c50de89a5a845668696dcb28c3badd6 in term 4.
I20260129 02:59:28.787281 20587 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "36ff33be666a4c82954731691dbcf327" candidate_uuid: "197b880fa39d432a9b472845876bfaf0" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1c50de89a5a845668696dcb28c3badd6" is_pre_election: true
I20260129 02:59:28.809509 20440 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "75be4bd6bd45474795fd472eff2953ac" candidate_uuid: "1c50de89a5a845668696dcb28c3badd6" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "197b880fa39d432a9b472845876bfaf0" is_pre_election: true
I20260129 02:59:28.832134 20440 raft_consensus.cc:2468] T 75be4bd6bd45474795fd472eff2953ac P 197b880fa39d432a9b472845876bfaf0 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1c50de89a5a845668696dcb28c3badd6 in term 5.
I20260129 02:59:28.771036 22102 raft_consensus.cc:2804] T fcddab6bca684adf8ea032289a00f90e P 1c50de89a5a845668696dcb28c3badd6 [term 6 FOLLOWER]: Leader election won for term 6
I20260129 02:59:28.777256 22115 raft_consensus.cc:493] T dc2f4cb752cf4d59b5f6bc689f5d8104 P 1c50de89a5a845668696dcb28c3badd6 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260129 02:59:28.661842 22070 raft_consensus.cc:515] T c934d31efd3f4e5195605ddc354d14e1 P 197b880fa39d432a9b472845876bfaf0 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } }
I20260129 02:59:28.845407 20405 leader_election.cc:304] T 4e3805ffa70242e8ade66da260dddf66 P 197b880fa39d432a9b472845876bfaf0 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 197b880fa39d432a9b472845876bfaf0; no voters: 1c50de89a5a845668696dcb28c3badd6, 7a9daf5d42a84615a6383cfec9a10608
I20260129 02:59:28.725346 22028 raft_consensus.cc:515] T 4b1876f9a70b49edb3aa74105e66d3b3 P 974b802141bd4f13beffef2aa03f88f9 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:59:28.855823 20479 leader_election.cc:304] T a5a05fbcbbcf4916bbc01fd1a5b5e90d P 974b802141bd4f13beffef2aa03f88f9 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 197b880fa39d432a9b472845876bfaf0, 974b802141bd4f13beffef2aa03f88f9; no voters:
I20260129 02:59:28.812263 20590 raft_consensus.cc:1275] T a4c1226b987b468fa84dfc6ba0e2dbe1 P 1c50de89a5a845668696dcb28c3badd6 [term 2 FOLLOWER]: Refusing update from remote peer 197b880fa39d432a9b472845876bfaf0: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260129 02:59:28.836901 21134 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a609ac02fae84725ab743f95f51d33ce" candidate_uuid: "197b880fa39d432a9b472845876bfaf0" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "7a9daf5d42a84615a6383cfec9a10608" is_pre_election: true
I20260129 02:59:28.781258 21984 raft_consensus.cc:493] T 96609ac68bdf4993a12e1c23355ca013 P 1c50de89a5a845668696dcb28c3badd6 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260129 02:59:28.770915 22062 raft_consensus.cc:493] T c2633a6fc34c44579de348d208b0c01e P 1c50de89a5a845668696dcb28c3badd6 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 197b880fa39d432a9b472845876bfaf0)
I20260129 02:59:28.829126 22168 raft_consensus.cc:493] T cb0b3f259ebc4310927338a8740e87b8 P 7a9daf5d42a84615a6383cfec9a10608 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 974b802141bd4f13beffef2aa03f88f9)
I20260129 02:59:28.849938 20516 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3906b080a5b34a74b8b64128635e8997" candidate_uuid: "1c50de89a5a845668696dcb28c3badd6" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "974b802141bd4f13beffef2aa03f88f9" is_pre_election: true
I20260129 02:59:28.849592 20438 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "cedce331e1584ac8b636c9b8a72b2ba0" candidate_uuid: "974b802141bd4f13beffef2aa03f88f9" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "197b880fa39d432a9b472845876bfaf0" is_pre_election: true
I20260129 02:59:28.851406 20586 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "8055232cf0004d40a85f3bf983fe00a9" candidate_uuid: "197b880fa39d432a9b472845876bfaf0" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1c50de89a5a845668696dcb28c3badd6" is_pre_election: true
I20260129 02:59:28.725561 22091 raft_consensus.cc:515] T c891d2bff4ce4568861bbc992e91fbb0 P 197b880fa39d432a9b472845876bfaf0 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:59:28.826647 22077 raft_consensus.cc:493] T 78aa39096868415d9afc2ffcb248e198 P 197b880fa39d432a9b472845876bfaf0 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260129 02:59:28.813376 22072 raft_consensus.cc:493] T a644c02b16af47679824ccd060ac10fd P 974b802141bd4f13beffef2aa03f88f9 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 1c50de89a5a845668696dcb28c3badd6)
I20260129 02:59:28.811192 22116 raft_consensus.cc:493] T 505580c93c694bbb8ea96fdf9b17ab46 P 1c50de89a5a845668696dcb28c3badd6 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 974b802141bd4f13beffef2aa03f88f9)
I20260129 02:59:28.869944 20553 leader_election.cc:304] T 1ce4ddf5b10e4ae8b193b13b2e5936ee P 1c50de89a5a845668696dcb28c3badd6 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 197b880fa39d432a9b472845876bfaf0, 1c50de89a5a845668696dcb28c3badd6; no voters:
I20260129 02:59:28.890173 20586 raft_consensus.cc:2468] T 8055232cf0004d40a85f3bf983fe00a9 P 1c50de89a5a845668696dcb28c3badd6 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 197b880fa39d432a9b472845876bfaf0 in term 1.
I20260129 02:59:28.888866 20516 raft_consensus.cc:2468] T 3906b080a5b34a74b8b64128635e8997 P 974b802141bd4f13beffef2aa03f88f9 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1c50de89a5a845668696dcb28c3badd6 in term 1.
I20260129 02:59:28.833598 22041 raft_consensus.cc:515] T e7d83d4ccf684e76acb29a0dc844e1b8 P 974b802141bd4f13beffef2aa03f88f9 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:59:28.834131 22100 raft_consensus.cc:493] T 4f8f3a38f6e24ba4857875181f447442 P 197b880fa39d432a9b472845876bfaf0 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 974b802141bd4f13beffef2aa03f88f9)
I20260129 02:59:28.856540 22143 raft_consensus.cc:2749] T 4e3805ffa70242e8ade66da260dddf66 P 197b880fa39d432a9b472845876bfaf0 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260129 02:59:28.908737 20516 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a4c1226b987b468fa84dfc6ba0e2dbe1" candidate_uuid: "1c50de89a5a845668696dcb28c3badd6" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "974b802141bd4f13beffef2aa03f88f9" is_pre_election: true
I20260129 02:59:28.909530 20516 raft_consensus.cc:2468] T a4c1226b987b468fa84dfc6ba0e2dbe1 P 974b802141bd4f13beffef2aa03f88f9 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1c50de89a5a845668696dcb28c3badd6 in term 2.
I20260129 02:59:28.856014 22115 raft_consensus.cc:515] T dc2f4cb752cf4d59b5f6bc689f5d8104 P 1c50de89a5a845668696dcb28c3badd6 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } }
I20260129 02:59:28.893362 22062 raft_consensus.cc:515] T c2633a6fc34c44579de348d208b0c01e P 1c50de89a5a845668696dcb28c3badd6 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:59:28.906931 22116 raft_consensus.cc:515] T 505580c93c694bbb8ea96fdf9b17ab46 P 1c50de89a5a845668696dcb28c3badd6 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } }
I20260129 02:59:28.935387 22072 raft_consensus.cc:515] T a644c02b16af47679824ccd060ac10fd P 974b802141bd4f13beffef2aa03f88f9 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } }
I20260129 02:59:28.896129 22168 raft_consensus.cc:515] T cb0b3f259ebc4310927338a8740e87b8 P 7a9daf5d42a84615a6383cfec9a10608 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } }
I20260129 02:59:28.893800 22103 raft_consensus.cc:493] T cb0b3f259ebc4310927338a8740e87b8 P 1c50de89a5a845668696dcb28c3badd6 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 974b802141bd4f13beffef2aa03f88f9)
I20260129 02:59:28.950130 20516 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "8055232cf0004d40a85f3bf983fe00a9" candidate_uuid: "197b880fa39d432a9b472845876bfaf0" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "974b802141bd4f13beffef2aa03f88f9" is_pre_election: true
I20260129 02:59:28.916589 20437 raft_consensus.cc:2468] T c891d2bff4ce4568861bbc992e91fbb0 P 197b880fa39d432a9b472845876bfaf0 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1c50de89a5a845668696dcb28c3badd6 in term 6.
I20260129 02:59:28.974474 20553 leader_election.cc:304] T 75be4bd6bd45474795fd472eff2953ac P 1c50de89a5a845668696dcb28c3badd6 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 197b880fa39d432a9b472845876bfaf0, 1c50de89a5a845668696dcb28c3badd6; no voters:
I20260129 02:59:28.891404 22102 raft_consensus.cc:697] T fcddab6bca684adf8ea032289a00f90e P 1c50de89a5a845668696dcb28c3badd6 [term 6 LEADER]: Becoming Leader. State: Replica: 1c50de89a5a845668696dcb28c3badd6, State: Running, Role: LEADER
I20260129 02:59:28.818816 22156 raft_consensus.cc:515] T add8d4923d1a489e86882c3a25692b95 P 7a9daf5d42a84615a6383cfec9a10608 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } }
I20260129 02:59:28.991770 20587 raft_consensus.cc:1275] T dc2f4cb752cf4d59b5f6bc689f5d8104 P 1c50de89a5a845668696dcb28c3badd6 [term 3 FOLLOWER]: Refusing update from remote peer 7a9daf5d42a84615a6383cfec9a10608: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 3 index: 5. (index mismatch)
I20260129 02:59:28.984360 20588 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "75f45538dc704d9a8ec4ff8ad9f3ea5b" candidate_uuid: "7a9daf5d42a84615a6383cfec9a10608" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "1c50de89a5a845668696dcb28c3badd6" is_pre_election: true
W20260129 02:59:28.986660 22154 log.cc:927] Time spent T 926a250fa5834e88b96ccecbefbf5086 P 197b880fa39d432a9b472845876bfaf0: Append to log took a long time: real 0.326s user 0.000s sys 0.004s
W20260129 02:59:28.999959 20553 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.211.1:42027, user_credentials={real_user=slave}} blocked reactor thread for 80611.8us
I20260129 02:59:28.939284 20433 raft_consensus.cc:1240] T c934d31efd3f4e5195605ddc354d14e1 P 197b880fa39d432a9b472845876bfaf0 [term 3 FOLLOWER]: Rejecting Update request from peer 7a9daf5d42a84615a6383cfec9a10608 for earlier term 2. Current term is 3. Ops: []
I20260129 02:59:28.953198 21130 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "42d27a28d89b409e8807ec0efd97af69" candidate_uuid: "197b880fa39d432a9b472845876bfaf0" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "7a9daf5d42a84615a6383cfec9a10608"
I20260129 02:59:28.916311 22140 raft_consensus.cc:493] T 8055232cf0004d40a85f3bf983fe00a9 P 1c50de89a5a845668696dcb28c3badd6 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 974b802141bd4f13beffef2aa03f88f9)
I20260129 02:59:28.970104 22164 raft_consensus.cc:493] T 4f8f3a38f6e24ba4857875181f447442 P 1c50de89a5a845668696dcb28c3badd6 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 974b802141bd4f13beffef2aa03f88f9)
I20260129 02:59:29.000972 20516 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a8479d1fe0bd41e5946186e3a6ca8c7a" candidate_uuid: "1c50de89a5a845668696dcb28c3badd6" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "974b802141bd4f13beffef2aa03f88f9" is_pre_election: true
I20260129 02:59:28.892280 21984 raft_consensus.cc:515] T 96609ac68bdf4993a12e1c23355ca013 P 1c50de89a5a845668696dcb28c3badd6 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } attrs { replace: true } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: NON_VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: true } }
I20260129 02:59:28.970188 22118 raft_consensus.cc:493] T 42d27a28d89b409e8807ec0efd97af69 P 7a9daf5d42a84615a6383cfec9a10608 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260129 02:59:29.022084 21974 leader_election.cc:290] T a4c1226b987b468fa84dfc6ba0e2dbe1 P 974b802141bd4f13beffef2aa03f88f9 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 197b880fa39d432a9b472845876bfaf0 (127.18.211.1:42027), 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3:37735)
I20260129 02:59:28.940944 22100 raft_consensus.cc:515] T 4f8f3a38f6e24ba4857875181f447442 P 197b880fa39d432a9b472845876bfaf0 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:59:28.993206 20588 raft_consensus.cc:2468] T 75f45538dc704d9a8ec4ff8ad9f3ea5b P 1c50de89a5a845668696dcb28c3badd6 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7a9daf5d42a84615a6383cfec9a10608 in term 1.
I20260129 02:59:28.990878 21944 raft_consensus.cc:493] T 4408006eb5174de1896b037cf3f18457 P 1c50de89a5a845668696dcb28c3badd6 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 974b802141bd4f13beffef2aa03f88f9)
I20260129 02:59:29.005043 21988 raft_consensus.cc:2804] T 1ce4ddf5b10e4ae8b193b13b2e5936ee P 1c50de89a5a845668696dcb28c3badd6 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260129 02:59:29.021999 22043 leader_election.cc:290] T 7d9494f5584d4d5db8b9adf1c8fc888b P 974b802141bd4f13beffef2aa03f88f9 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 197b880fa39d432a9b472845876bfaf0 (127.18.211.1:42027), 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3:37735)
I20260129 02:59:28.903831 22077 raft_consensus.cc:515] T 78aa39096868415d9afc2ffcb248e198 P 197b880fa39d432a9b472845876bfaf0 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } }
I20260129 02:59:29.010452 22120 raft_consensus.cc:493] T 2e58e9a5a8aa4cecafb464addb4f35ed P 197b880fa39d432a9b472845876bfaf0 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 7a9daf5d42a84615a6383cfec9a10608)
I20260129 02:59:29.019969 20516 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1ce4ddf5b10e4ae8b193b13b2e5936ee" candidate_uuid: "1c50de89a5a845668696dcb28c3badd6" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "974b802141bd4f13beffef2aa03f88f9" is_pre_election: true
I20260129 02:59:29.157969 20516 raft_consensus.cc:2468] T 1ce4ddf5b10e4ae8b193b13b2e5936ee P 974b802141bd4f13beffef2aa03f88f9 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1c50de89a5a845668696dcb28c3badd6 in term 3.
I20260129 02:59:29.049288 20432 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55676 (request call id 1050) took 1518 ms. Trace:
I20260129 02:59:29.141629 20514 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57600 (request call id 1596) took 2141 ms. Trace:
I20260129 02:59:29.023429 22141 leader_election.cc:290] T 18f30ebaa4174fd28c958e72d0cc7e24 P 197b880fa39d432a9b472845876bfaf0 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3:37735), 7a9daf5d42a84615a6383cfec9a10608 (127.18.211.4:40439)
I20260129 02:59:29.166002 22156 leader_election.cc:290] T add8d4923d1a489e86882c3a25692b95 P 7a9daf5d42a84615a6383cfec9a10608 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 974b802141bd4f13beffef2aa03f88f9 (127.18.211.2:34517), 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3:37735)
W20260129 02:59:29.127275 22167 log.cc:927] Time spent T 4b1876f9a70b49edb3aa74105e66d3b3 P 974b802141bd4f13beffef2aa03f88f9: Append to log took a long time: real 0.145s user 0.000s sys 0.001s
W20260129 02:59:29.167969 22163 log.cc:927] Time spent T 78aa39096868415d9afc2ffcb248e198 P 1c50de89a5a845668696dcb28c3badd6: Append to log took a long time: real 0.386s user 0.001s sys 0.000s
I20260129 02:59:29.168648 20508 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57584 (request call id 1122) took 1146 ms. Trace:
I20260129 02:59:29.168923 22122 leader_election.cc:290] T 1299723a4476420ab02fd8bfd7aa0645 P 7a9daf5d42a84615a6383cfec9a10608 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 197b880fa39d432a9b472845876bfaf0 (127.18.211.1:42027), 974b802141bd4f13beffef2aa03f88f9 (127.18.211.2:34517), 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3:37735)
I20260129 02:59:29.169507 19276 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260129 02:59:29.171178 22041 leader_election.cc:290] T e7d83d4ccf684e76acb29a0dc844e1b8 P 974b802141bd4f13beffef2aa03f88f9 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 197b880fa39d432a9b472845876bfaf0 (127.18.211.1:42027), 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3:37735)
I20260129 02:59:29.169061 20508 rpcz_store.cc:276] 0129 02:59:28.022648 (+ 0us) service_pool.cc:168] Inserting onto call queue
0129 02:59:28.022875 (+ 227us) service_pool.cc:225] Handling call
0129 02:59:28.023318 (+ 443us) raft_consensus.cc:1471] Updating replica for 1 ops
0129 02:59:28.023588 (+ 270us) raft_consensus.cc:1534] Early marking committed up to index 0
0129 02:59:28.023624 (+ 36us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0129 02:59:28.026772 (+ 3148us) log.cc:844] Serialized 26 byte log entry
0129 02:59:28.632538 (+605766us) raft_consensus.cc:1645] Marking committed up to 2
0129 02:59:28.632612 (+ 74us) raft_consensus.cc:1697] Filling consensus response to leader.
0129 02:59:28.632712 (+ 100us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0129 02:59:29.168426 (+535714us) raft_consensus.cc:1685] finished
0129 02:59:29.168448 (+ 22us) raft_consensus.cc:1691] UpdateReplicas() finished
0129 02:59:29.168589 (+ 141us) inbound_call.cc:173] Queueing success response
Metrics: {"thread_start_us":481701,"threads_started":1,"wal-append.queue_time_us":823405}
I20260129 02:59:29.163381 20515 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57584 (request call id 1124) took 1123 ms. Trace:
I20260129 02:59:29.164266 20514 rpcz_store.cc:276] 0129 02:59:26.995609 (+ 0us) service_pool.cc:168] Inserting onto call queue
0129 02:59:26.995764 (+ 155us) service_pool.cc:225] Handling call
0129 02:59:27.847348 (+851584us) raft_consensus.cc:1471] Updating replica for 0 ops
0129 02:59:27.848603 (+ 1255us) raft_consensus.cc:1697] Filling consensus response to leader.
0129 02:59:29.137223 (+1288620us) spinlock_profiling.cc:243] Waited 835 ms on lock 0x7b540012a050. stack: 00007f8fe044e373 00007f8fe044e270 00007f8fdfda0574 00007f8fe9a5c6b4 00007f8fe9a5c649 00007f8fe34975a1 00007f8fe352d52d 00007f8fe84e1f49 00007f8fe1a6ed34 00007f8fe1a6ecc2 00007f8fe1a6ebf1 00007f8fe1a6eb6c 00007f8fe1a6de22 00007f8fe186d34c 00007f8fe186c786 00007f8fe186c12f
0129 02:59:29.137424 (+ 201us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1919951744}
I20260129 02:59:29.180302 20516 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0f20af7fbe5d4db89a6f17948d98912c" candidate_uuid: "197b880fa39d432a9b472845876bfaf0" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "974b802141bd4f13beffef2aa03f88f9" is_pre_election: true
I20260129 02:59:28.991744 22143 raft_consensus.cc:493] T 4e420f90884f4796997b5ea41e49e411 P 197b880fa39d432a9b472845876bfaf0 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 7a9daf5d42a84615a6383cfec9a10608)
I20260129 02:59:29.188089 20508 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c891d2bff4ce4568861bbc992e91fbb0" candidate_uuid: "1c50de89a5a845668696dcb28c3badd6" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "974b802141bd4f13beffef2aa03f88f9" is_pre_election: true
I20260129 02:59:29.050124 21135 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "505580c93c694bbb8ea96fdf9b17ab46" candidate_uuid: "1c50de89a5a845668696dcb28c3badd6" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "7a9daf5d42a84615a6383cfec9a10608" is_pre_election: true
I20260129 02:59:29.189632 21135 raft_consensus.cc:2468] T 505580c93c694bbb8ea96fdf9b17ab46 P 7a9daf5d42a84615a6383cfec9a10608 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1c50de89a5a845668696dcb28c3badd6 in term 2.
W20260129 02:59:29.050632 22177 log.cc:927] Time spent T a5a05fbcbbcf4916bbc01fd1a5b5e90d P 974b802141bd4f13beffef2aa03f88f9: Append to log took a long time: real 0.135s user 0.000s sys 0.000s
W20260129 02:59:29.152096 22153 log.cc:927] Time spent T 0678b8b468bd43cdbc3e485900844821 P 197b880fa39d432a9b472845876bfaf0: Append to log took a long time: real 0.399s user 0.000s sys 0.001s
I20260129 02:59:29.173341 21988 raft_consensus.cc:493] T 1ce4ddf5b10e4ae8b193b13b2e5936ee P 1c50de89a5a845668696dcb28c3badd6 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260129 02:59:29.160923 20432 rpcz_store.cc:276] 0129 02:59:27.531019 (+ 0us) service_pool.cc:168] Inserting onto call queue
0129 02:59:27.531152 (+ 133us) service_pool.cc:225] Handling call
0129 02:59:27.531521 (+ 369us) raft_consensus.cc:1471] Updating replica for 1 ops
0129 02:59:27.532169 (+ 648us) raft_consensus.cc:1534] Early marking committed up to index 0
0129 02:59:27.532199 (+ 30us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0129 02:59:27.538333 (+ 6134us) log.cc:844] Serialized 26 byte log entry
0129 02:59:28.404003 (+865670us) raft_consensus.cc:1645] Marking committed up to 2
0129 02:59:28.404059 (+ 56us) raft_consensus.cc:1697] Filling consensus response to leader.
0129 02:59:28.404157 (+ 98us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0129 02:59:29.048938 (+644781us) raft_consensus.cc:1685] finished
0129 02:59:29.048977 (+ 39us) raft_consensus.cc:1691] UpdateReplicas() finished
0129 02:59:29.049202 (+ 225us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":38,"raft.queue_time_us":4577,"raft.run_cpu_time_us":179,"raft.run_wall_time_us":3920,"thread_start_us":824491,"threads_started":1,"wal-append.queue_time_us":1060471}
I20260129 02:59:29.260962 20508 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a609ac02fae84725ab743f95f51d33ce" candidate_uuid: "197b880fa39d432a9b472845876bfaf0" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "974b802141bd4f13beffef2aa03f88f9" is_pre_election: true
W20260129 02:59:29.197103 20583 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:35752 (request call id 1414) took 2698 ms (client timeout 2067 ms). Trace:
I20260129 02:59:29.263742 20516 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "75be4bd6bd45474795fd472eff2953ac" candidate_uuid: "197b880fa39d432a9b472845876bfaf0" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "974b802141bd4f13beffef2aa03f88f9" is_pre_election: true
I20260129 02:59:29.176571 20515 rpcz_store.cc:276] 0129 02:59:28.040456 (+ 0us) service_pool.cc:168] Inserting onto call queue
0129 02:59:28.040639 (+ 183us) service_pool.cc:225] Handling call
0129 02:59:28.140960 (+100321us) raft_consensus.cc:1471] Updating replica for 1 ops
0129 02:59:28.148813 (+ 7853us) raft_consensus.cc:1534] Early marking committed up to index 0
0129 02:59:28.148858 (+ 45us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0129 02:59:28.165547 (+ 16689us) log.cc:844] Serialized 26 byte log entry
0129 02:59:28.789970 (+624423us) raft_consensus.cc:1645] Marking committed up to 1
0129 02:59:28.790015 (+ 45us) raft_consensus.cc:1697] Filling consensus response to leader.
0129 02:59:28.790086 (+ 71us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0129 02:59:29.023557 (+233471us) raft_consensus.cc:1685] finished
0129 02:59:29.023585 (+ 28us) raft_consensus.cc:1691] UpdateReplicas() finished
0129 02:59:29.163309 (+139724us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":2114,"raft.queue_time_us":7501,"raft.run_cpu_time_us":198,"raft.run_wall_time_us":15374,"thread_start_us":482962,"threads_started":1,"wal-append.queue_time_us":814128}
I20260129 02:59:29.117851 21942 raft_consensus.cc:493] T e4d18c35b0a7493e85d0a4b92956a48f P 197b880fa39d432a9b472845876bfaf0 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 974b802141bd4f13beffef2aa03f88f9)
W20260129 02:59:29.049535 22159 log.cc:927] Time spent T a0fcc58d6fb6473cafbd9d34d5195269 P 974b802141bd4f13beffef2aa03f88f9: Append to log took a long time: real 0.072s user 0.000s sys 0.000s
W20260129 02:59:29.050406 22182 log.cc:927] Time spent T 87d5fe48a44a408aaf03e894a67ecb2f P 7a9daf5d42a84615a6383cfec9a10608: Append to log took a long time: real 0.061s user 0.000s sys 0.000s
I20260129 02:59:29.271411 20516 raft_consensus.cc:2393] T 75be4bd6bd45474795fd472eff2953ac P 974b802141bd4f13beffef2aa03f88f9 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 197b880fa39d432a9b472845876bfaf0 in current term 6: Already voted for candidate 974b802141bd4f13beffef2aa03f88f9 in this term.
I20260129 02:59:29.048404 22116 leader_election.cc:290] T 505580c93c694bbb8ea96fdf9b17ab46 P 1c50de89a5a845668696dcb28c3badd6 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 974b802141bd4f13beffef2aa03f88f9 (127.18.211.2:34517), 7a9daf5d42a84615a6383cfec9a10608 (127.18.211.4:40439)
I20260129 02:59:29.166406 22100 leader_election.cc:290] T 4f8f3a38f6e24ba4857875181f447442 P 197b880fa39d432a9b472845876bfaf0 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 974b802141bd4f13beffef2aa03f88f9 (127.18.211.2:34517), 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3:37735)
I20260129 02:59:29.152840 22107 leader_election.cc:290] T 368953c9102643b2a21f99af36b51905 P 1c50de89a5a845668696dcb28c3badd6 [CANDIDATE]: Term 4 election: Requested vote from peers 197b880fa39d432a9b472845876bfaf0 (127.18.211.1:42027), 974b802141bd4f13beffef2aa03f88f9 (127.18.211.2:34517)
I20260129 02:59:29.274003 20432 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55672 (request call id 1372) took 1125 ms. Trace:
I20260129 02:59:29.276281 20514 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "75be4bd6bd45474795fd472eff2953ac" candidate_uuid: "1c50de89a5a845668696dcb28c3badd6" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "974b802141bd4f13beffef2aa03f88f9" is_pre_election: true
I20260129 02:59:29.198966 22050 raft_consensus.cc:493] T 4f5a77e94fef4b16a2ba7b067c8b144a P 7a9daf5d42a84615a6383cfec9a10608 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 974b802141bd4f13beffef2aa03f88f9)
I20260129 02:59:28.992265 22103 raft_consensus.cc:515] T cb0b3f259ebc4310927338a8740e87b8 P 1c50de89a5a845668696dcb28c3badd6 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } }
I20260129 02:59:29.320402 20514 raft_consensus.cc:2468] T 75be4bd6bd45474795fd472eff2953ac P 974b802141bd4f13beffef2aa03f88f9 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1c50de89a5a845668696dcb28c3badd6 in term 6.
I20260129 02:59:29.323396 22131 raft_consensus.cc:493] T e4d18c35b0a7493e85d0a4b92956a48f P 1c50de89a5a845668696dcb28c3badd6 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 974b802141bd4f13beffef2aa03f88f9)
I20260129 02:59:29.323704 22144 raft_consensus.cc:493] T 75f45538dc704d9a8ec4ff8ad9f3ea5b P 1c50de89a5a845668696dcb28c3badd6 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 974b802141bd4f13beffef2aa03f88f9)
W20260129 02:59:29.263105 20583 rpcz_store.cc:269] 0129 02:59:26.488841 (+ 0us) service_pool.cc:168] Inserting onto call queue
0129 02:59:26.491129 (+ 2288us) service_pool.cc:225] Handling call
0129 02:59:29.186528 (+2695399us) spinlock_profiling.cc:243] Waited 1.17 s on lock 0x7b54000c8854. stack: 00007f8fe044e373 00007f8fe044e270 00007f8fdfda0574 00007f8fe9a5c6b4 00007f8fe9a5c649 00007f8fe34975a1 00007f8fe3531f7a 00007f8fe84e3647 00007f8fe1a73834 00007f8fe1a737c2 00007f8fe1a736f1 00007f8fe1a7366c 00007f8fe1a72922 00007f8fe186d34c 00007f8fe186c786 00007f8fe186c12f
0129 02:59:29.186680 (+ 152us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2682655488}
I20260129 02:59:29.124989 22004 raft_consensus.cc:2804] T 75be4bd6bd45474795fd472eff2953ac P 1c50de89a5a845668696dcb28c3badd6 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260129 02:59:29.191270 21974 raft_consensus.cc:2804] T a5a05fbcbbcf4916bbc01fd1a5b5e90d P 974b802141bd4f13beffef2aa03f88f9 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260129 02:59:29.024654 22140 raft_consensus.cc:515] T 8055232cf0004d40a85f3bf983fe00a9 P 1c50de89a5a845668696dcb28c3badd6 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:59:29.321709 20432 rpcz_store.cc:276] 0129 02:59:28.146726 (+ 0us) service_pool.cc:168] Inserting onto call queue
0129 02:59:29.261826 (+1115100us) service_pool.cc:225] Handling call
0129 02:59:29.262272 (+ 446us) raft_consensus.cc:1471] Updating replica for 0 ops
0129 02:59:29.267225 (+ 4953us) raft_consensus.cc:1534] Early marking committed up to index 0
0129 02:59:29.267272 (+ 47us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0129 02:59:29.267328 (+ 56us) raft_consensus.cc:1645] Marking committed up to 1
0129 02:59:29.267373 (+ 45us) raft_consensus.cc:1697] Filling consensus response to leader.
0129 02:59:29.267457 (+ 84us) raft_consensus.cc:1691] UpdateReplicas() finished
0129 02:59:29.271782 (+ 4325us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260129 02:59:29.319213 22116 raft_consensus.cc:493] T 724a236524cb4e7ba3428880681d6cc2 P 1c50de89a5a845668696dcb28c3badd6 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 974b802141bd4f13beffef2aa03f88f9)
I20260129 02:59:29.205539 22033 raft_consensus.cc:493] T 2ac659d732314e1288849f08fe5e074e P 7a9daf5d42a84615a6383cfec9a10608 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 974b802141bd4f13beffef2aa03f88f9)
I20260129 02:59:29.190781 22070 leader_election.cc:290] T c934d31efd3f4e5195605ddc354d14e1 P 197b880fa39d432a9b472845876bfaf0 [CANDIDATE]: Term 3 election: Requested vote from peers 974b802141bd4f13beffef2aa03f88f9 (127.18.211.2:34517), 7a9daf5d42a84615a6383cfec9a10608 (127.18.211.4:40439)
I20260129 02:59:29.187433 22062 leader_election.cc:290] T c2633a6fc34c44579de348d208b0c01e P 1c50de89a5a845668696dcb28c3badd6 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 197b880fa39d432a9b472845876bfaf0 (127.18.211.1:42027), 974b802141bd4f13beffef2aa03f88f9 (127.18.211.2:34517)
I20260129 02:59:29.179237 21944 raft_consensus.cc:515] T 4408006eb5174de1896b037cf3f18457 P 1c50de89a5a845668696dcb28c3badd6 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:59:29.324298 22144 raft_consensus.cc:515] T 75f45538dc704d9a8ec4ff8ad9f3ea5b P 1c50de89a5a845668696dcb28c3badd6 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } }
I20260129 02:59:29.323414 22050 raft_consensus.cc:515] T 4f5a77e94fef4b16a2ba7b067c8b144a P 7a9daf5d42a84615a6383cfec9a10608 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } }
I20260129 02:59:29.335109 21991 raft_consensus.cc:493] T 4e3805ffa70242e8ade66da260dddf66 P 1c50de89a5a845668696dcb28c3badd6 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 7a9daf5d42a84615a6383cfec9a10608)
I20260129 02:59:29.328709 22004 raft_consensus.cc:493] T 75be4bd6bd45474795fd472eff2953ac P 1c50de89a5a845668696dcb28c3badd6 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260129 02:59:29.202313 21984 leader_election.cc:290] T 96609ac68bdf4993a12e1c23355ca013 P 1c50de89a5a845668696dcb28c3badd6 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 197b880fa39d432a9b472845876bfaf0 (127.18.211.1:42027), 974b802141bd4f13beffef2aa03f88f9 (127.18.211.2:34517)
I20260129 02:59:29.265513 21988 raft_consensus.cc:3060] T 1ce4ddf5b10e4ae8b193b13b2e5936ee P 1c50de89a5a845668696dcb28c3badd6 [term 3 FOLLOWER]: Advancing to term 4
I20260129 02:59:29.199227 22053 raft_consensus.cc:493] T febffe4a8b394368b69b1d164308fb9a P 7a9daf5d42a84615a6383cfec9a10608 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 197b880fa39d432a9b472845876bfaf0)
I20260129 02:59:29.185644 22118 raft_consensus.cc:515] T 42d27a28d89b409e8807ec0efd97af69 P 7a9daf5d42a84615a6383cfec9a10608 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } }
I20260129 02:59:29.084647 22164 raft_consensus.cc:515] T 4f8f3a38f6e24ba4857875181f447442 P 1c50de89a5a845668696dcb28c3badd6 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:59:29.390061 22053 raft_consensus.cc:515] T febffe4a8b394368b69b1d164308fb9a P 7a9daf5d42a84615a6383cfec9a10608 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } }
I20260129 02:59:29.302196 22191 raft_consensus.cc:493] T 3906b080a5b34a74b8b64128635e8997 P 974b802141bd4f13beffef2aa03f88f9 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 197b880fa39d432a9b472845876bfaf0)
I20260129 02:59:29.324047 22131 raft_consensus.cc:515] T e4d18c35b0a7493e85d0a4b92956a48f P 1c50de89a5a845668696dcb28c3badd6 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:59:29.050674 22183 raft_consensus.cc:493] T 4e420f90884f4796997b5ea41e49e411 P 974b802141bd4f13beffef2aa03f88f9 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 7a9daf5d42a84615a6383cfec9a10608)
I20260129 02:59:29.147053 22185 raft_consensus.cc:493] T a609ac02fae84725ab743f95f51d33ce P 974b802141bd4f13beffef2aa03f88f9 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 7a9daf5d42a84615a6383cfec9a10608)
I20260129 02:59:29.336205 21974 raft_consensus.cc:493] T a5a05fbcbbcf4916bbc01fd1a5b5e90d P 974b802141bd4f13beffef2aa03f88f9 [term 4 FOLLOWER]: Starting leader election (detected failure of leader 1c50de89a5a845668696dcb28c3badd6)
W20260129 02:59:29.362972 22175 log.cc:927] Time spent T 0678b8b468bd43cdbc3e485900844821 P 7a9daf5d42a84615a6383cfec9a10608: Append to log took a long time: real 0.069s user 0.000s sys 0.000s
I20260129 02:59:29.398605 21944 leader_election.cc:290] T 4408006eb5174de1896b037cf3f18457 P 1c50de89a5a845668696dcb28c3badd6 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 197b880fa39d432a9b472845876bfaf0 (127.18.211.1:42027), 974b802141bd4f13beffef2aa03f88f9 (127.18.211.2:34517)
I20260129 02:59:29.399426 22059 raft_consensus.cc:493] T 96609ac68bdf4993a12e1c23355ca013 P 7a9daf5d42a84615a6383cfec9a10608 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260129 02:59:29.363023 20507 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57584 (request call id 1126) took 1252 ms. Trace:
I20260129 02:59:29.365549 20510 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57584 (request call id 1128) took 1218 ms. Trace:
I20260129 02:59:29.323444 22107 raft_consensus.cc:493] T 0f20af7fbe5d4db89a6f17948d98912c P 1c50de89a5a845668696dcb28c3badd6 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 974b802141bd4f13beffef2aa03f88f9)
I20260129 02:59:29.381635 22103 leader_election.cc:290] T cb0b3f259ebc4310927338a8740e87b8 P 1c50de89a5a845668696dcb28c3badd6 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 974b802141bd4f13beffef2aa03f88f9 (127.18.211.2:34517), 7a9daf5d42a84615a6383cfec9a10608 (127.18.211.4:40439)
I20260129 02:59:29.401684 20510 rpcz_store.cc:276] 0129 02:59:28.147243 (+ 0us) service_pool.cc:168] Inserting onto call queue
0129 02:59:28.383769 (+236526us) service_pool.cc:225] Handling call
0129 02:59:28.384210 (+ 441us) raft_consensus.cc:1471] Updating replica for 1 ops
0129 02:59:28.384635 (+ 425us) raft_consensus.cc:1534] Early marking committed up to index 0
0129 02:59:28.384675 (+ 40us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0129 02:59:28.459141 (+ 74466us) log.cc:844] Serialized 26 byte log entry
0129 02:59:28.712232 (+253091us) raft_consensus.cc:1645] Marking committed up to 2
0129 02:59:28.712292 (+ 60us) raft_consensus.cc:1697] Filling consensus response to leader.
0129 02:59:28.712375 (+ 83us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0129 02:59:29.365318 (+652943us) raft_consensus.cc:1685] finished
0129 02:59:29.365341 (+ 23us) raft_consensus.cc:1691] UpdateReplicas() finished
0129 02:59:29.365489 (+ 148us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":2267,"raft.run_cpu_time_us":143,"raft.run_wall_time_us":146,"thread_start_us":208240,"threads_started":1,"wal-append.queue_time_us":345258}
I20260129 02:59:29.384111 22072 leader_election.cc:290] T a644c02b16af47679824ccd060ac10fd P 974b802141bd4f13beffef2aa03f88f9 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3:37735), 7a9daf5d42a84615a6383cfec9a10608 (127.18.211.4:40439)
I20260129 02:59:29.384790 22168 leader_election.cc:290] T cb0b3f259ebc4310927338a8740e87b8 P 7a9daf5d42a84615a6383cfec9a10608 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 974b802141bd4f13beffef2aa03f88f9 (127.18.211.2:34517), 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3:37735)
I20260129 02:59:29.384884 22092 raft_consensus.cc:493] T 76f713c3331540d2a5b76164103e9eb1 P 197b880fa39d432a9b472845876bfaf0 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 1c50de89a5a845668696dcb28c3badd6)
I20260129 02:59:29.410423 22092 raft_consensus.cc:3060] T 76f713c3331540d2a5b76164103e9eb1 P 197b880fa39d432a9b472845876bfaf0 [term 2 FOLLOWER]: Advancing to term 3
I20260129 02:59:29.385128 22050 leader_election.cc:290] T 4f5a77e94fef4b16a2ba7b067c8b144a P 7a9daf5d42a84615a6383cfec9a10608 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 974b802141bd4f13beffef2aa03f88f9 (127.18.211.2:34517), 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3:37735)
W20260129 02:59:29.413770 20512 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:57600 (request call id 1601) took 1840 ms (client timeout 2243 ms). Trace:
W20260129 02:59:29.414258 20512 rpcz_store.cc:269] 0129 02:59:27.573439 (+ 0us) service_pool.cc:168] Inserting onto call queue
0129 02:59:27.573611 (+ 172us) service_pool.cc:225] Handling call
0129 02:59:29.413593 (+1839982us) spinlock_profiling.cc:243] Waited 232 ms on lock 0x7b54003ef654. stack: 00007f8fe044e373 00007f8fe044e270 00007f8fdfda0574 00007f8fe9a5c6b4 00007f8fe9a5c649 00007f8fe34975a1 00007f8fe3531f7a 00007f8fe84e3647 00007f8fe1a73834 00007f8fe1a737c2 00007f8fe1a736f1 00007f8fe1a7366c 00007f8fe1a72922 00007f8fe186d34c 00007f8fe186c786 00007f8fe186c12f
0129 02:59:29.413707 (+ 114us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":534522240}
I20260129 02:59:29.373325 22115 leader_election.cc:290] T dc2f4cb752cf4d59b5f6bc689f5d8104 P 1c50de89a5a845668696dcb28c3badd6 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 197b880fa39d432a9b472845876bfaf0 (127.18.211.1:42027), 7a9daf5d42a84615a6383cfec9a10608 (127.18.211.4:40439)
I20260129 02:59:29.338934 22091 leader_election.cc:290] T c891d2bff4ce4568861bbc992e91fbb0 P 197b880fa39d432a9b472845876bfaf0 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 974b802141bd4f13beffef2aa03f88f9 (127.18.211.2:34517), 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3:37735)
I20260129 02:59:29.385900 21991 raft_consensus.cc:515] T 4e3805ffa70242e8ade66da260dddf66 P 1c50de89a5a845668696dcb28c3badd6 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } }
I20260129 02:59:29.428411 22118 leader_election.cc:290] T 42d27a28d89b409e8807ec0efd97af69 P 7a9daf5d42a84615a6383cfec9a10608 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 197b880fa39d432a9b472845876bfaf0 (127.18.211.1:42027), 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3:37735)
I20260129 02:59:29.429937 21991 leader_election.cc:290] T 4e3805ffa70242e8ade66da260dddf66 P 1c50de89a5a845668696dcb28c3badd6 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 197b880fa39d432a9b472845876bfaf0 (127.18.211.1:42027), 7a9daf5d42a84615a6383cfec9a10608 (127.18.211.4:40439)
I20260129 02:59:29.387359 20513 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57600 (request call id 1600) took 1820 ms. Trace:
I20260129 02:59:29.383363 22004 raft_consensus.cc:3060] T 75be4bd6bd45474795fd472eff2953ac P 1c50de89a5a845668696dcb28c3badd6 [term 6 FOLLOWER]: Advancing to term 7
I20260129 02:59:29.430686 20513 rpcz_store.cc:276] 0129 02:59:27.566647 (+ 0us) service_pool.cc:168] Inserting onto call queue
0129 02:59:27.566842 (+ 195us) service_pool.cc:225] Handling call
0129 02:59:27.582418 (+ 15576us) raft_consensus.cc:1471] Updating replica for 1 ops
0129 02:59:27.596304 (+ 13886us) raft_consensus.cc:1534] Early marking committed up to index 0
0129 02:59:27.596333 (+ 29us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0129 02:59:27.602840 (+ 6507us) log.cc:844] Serialized 26 byte log entry
0129 02:59:28.721443 (+1118603us) raft_consensus.cc:1645] Marking committed up to 1
0129 02:59:28.721503 (+ 60us) raft_consensus.cc:1697] Filling consensus response to leader.
0129 02:59:28.721598 (+ 95us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0129 02:59:29.387096 (+665498us) raft_consensus.cc:1685] finished
0129 02:59:29.387127 (+ 31us) raft_consensus.cc:1691] UpdateReplicas() finished
0129 02:59:29.387286 (+ 159us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":13538,"raft.queue_time_us":14047,"raft.run_cpu_time_us":143,"raft.run_wall_time_us":8485,"thread_start_us":1032875,"threads_started":1,"wal-append.queue_time_us":1205663}
I20260129 02:59:29.402630 21974 raft_consensus.cc:3060] T a5a05fbcbbcf4916bbc01fd1a5b5e90d P 974b802141bd4f13beffef2aa03f88f9 [term 4 FOLLOWER]: Advancing to term 5
I20260129 02:59:29.259333 22143 raft_consensus.cc:515] T 4e420f90884f4796997b5ea41e49e411 P 197b880fa39d432a9b472845876bfaf0 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } }
I20260129 02:59:29.394603 20435 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:48356 (request call id 193) took 2078 ms. Trace:
I20260129 02:59:29.299230 21942 raft_consensus.cc:515] T e4d18c35b0a7493e85d0a4b92956a48f P 197b880fa39d432a9b472845876bfaf0 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:59:29.263002 22120 raft_consensus.cc:515] T 2e58e9a5a8aa4cecafb464addb4f35ed P 197b880fa39d432a9b472845876bfaf0 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } }
I20260129 02:59:29.398092 22077 leader_election.cc:290] T 78aa39096868415d9afc2ffcb248e198 P 197b880fa39d432a9b472845876bfaf0 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3:37735), 7a9daf5d42a84615a6383cfec9a10608 (127.18.211.4:40439)
I20260129 02:59:29.358661 22116 raft_consensus.cc:515] T 724a236524cb4e7ba3428880681d6cc2 P 1c50de89a5a845668696dcb28c3badd6 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:59:29.400180 20507 rpcz_store.cc:276] 0129 02:59:28.125894 (+ 0us) service_pool.cc:168] Inserting onto call queue
0129 02:59:28.145252 (+ 19358us) service_pool.cc:225] Handling call
0129 02:59:28.173724 (+ 28472us) raft_consensus.cc:1471] Updating replica for 1 ops
0129 02:59:28.178306 (+ 4582us) raft_consensus.cc:1534] Early marking committed up to index 0
0129 02:59:28.178352 (+ 46us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0129 02:59:28.215431 (+ 37079us) log.cc:844] Serialized 26 byte log entry
0129 02:59:28.661131 (+445700us) raft_consensus.cc:1645] Marking committed up to 2
0129 02:59:28.661192 (+ 61us) raft_consensus.cc:1697] Filling consensus response to leader.
0129 02:59:28.661293 (+ 101us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0129 02:59:29.362792 (+701499us) raft_consensus.cc:1685] finished
0129 02:59:29.362820 (+ 28us) raft_consensus.cc:1691] UpdateReplicas() finished
0129 02:59:29.362961 (+ 141us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":5839,"raft.run_cpu_time_us":209,"raft.run_wall_time_us":75470,"thread_start_us":345140,"threads_started":1,"wal-append.queue_time_us":874958}
I20260129 02:59:29.453104 20441 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55676 (request call id 1053) took 1765 ms. Trace:
I20260129 02:59:29.453666 20441 rpcz_store.cc:276] 0129 02:59:27.687428 (+ 0us) service_pool.cc:168] Inserting onto call queue
0129 02:59:27.820624 (+133196us) service_pool.cc:225] Handling call
0129 02:59:27.836035 (+ 15411us) raft_consensus.cc:1471] Updating replica for 0 ops
0129 02:59:28.400635 (+564600us) raft_consensus.cc:1697] Filling consensus response to leader.
0129 02:59:29.452731 (+1052096us) spinlock_profiling.cc:243] Waited 564 ms on lock 0x7b5400230854. stack: 00007f8fe044e373 00007f8fe044e270 00007f8fdfda0574 00007f8fe9a5c6b4 00007f8fe9a5c649 00007f8fe34975a1 00007f8fe352ea81 00007f8fe352d4be 00007f8fe84e1f49 00007f8fe1a6ed34 00007f8fe1a6ecc2 00007f8fe1a6ebf1 00007f8fe1a6eb6c 00007f8fe1a6de22 00007f8fe186d34c 00007f8fe186c786
0129 02:59:29.453003 (+ 272us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1297093888}
I20260129 02:59:29.375367 22033 raft_consensus.cc:515] T 2ac659d732314e1288849f08fe5e074e P 7a9daf5d42a84615a6383cfec9a10608 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } }
I20260129 02:59:29.459218 20441 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55672 (request call id 1374) took 1191 ms. Trace:
I20260129 02:59:29.460166 20441 rpcz_store.cc:276] 0129 02:59:28.268169 (+ 0us) service_pool.cc:168] Inserting onto call queue
0129 02:59:29.454973 (+1186804us) service_pool.cc:225] Handling call
0129 02:59:29.458579 (+ 3606us) raft_consensus.cc:1471] Updating replica for 0 ops
0129 02:59:29.458809 (+ 230us) raft_consensus.cc:1534] Early marking committed up to index 0
0129 02:59:29.458844 (+ 35us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0129 02:59:29.458897 (+ 53us) raft_consensus.cc:1645] Marking committed up to 4
0129 02:59:29.458938 (+ 41us) raft_consensus.cc:1697] Filling consensus response to leader.
0129 02:59:29.459007 (+ 69us) raft_consensus.cc:1691] UpdateReplicas() finished
0129 02:59:29.459147 (+ 140us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260129 02:59:29.171809 22102 consensus_queue.cc:237] T fcddab6bca684adf8ea032289a00f90e P 1c50de89a5a845668696dcb28c3badd6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 4.4, Last appended by leader: 4, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:59:29.404598 22059 raft_consensus.cc:515] T 96609ac68bdf4993a12e1c23355ca013 P 7a9daf5d42a84615a6383cfec9a10608 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } attrs { replace: true } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } }
I20260129 02:59:29.361181 22187 raft_consensus.cc:493] T fd8311b779af4e1fa1402b2a02a602f2 P 7a9daf5d42a84615a6383cfec9a10608 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 197b880fa39d432a9b472845876bfaf0)
I20260129 02:59:29.421422 21130 raft_consensus.cc:3060] T 42d27a28d89b409e8807ec0efd97af69 P 7a9daf5d42a84615a6383cfec9a10608 [term 3 FOLLOWER]: Advancing to term 4
I20260129 02:59:29.405743 22107 raft_consensus.cc:515] T 0f20af7fbe5d4db89a6f17948d98912c P 1c50de89a5a845668696dcb28c3badd6 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:59:29.415807 20512 raft_consensus.cc:1275] T 926a250fa5834e88b96ccecbefbf5086 P 974b802141bd4f13beffef2aa03f88f9 [term 4 FOLLOWER]: Refusing update from remote peer 1c50de89a5a845668696dcb28c3badd6: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20260129 02:59:29.435681 20435 rpcz_store.cc:276] 0129 02:59:27.315769 (+ 0us) service_pool.cc:168] Inserting onto call queue
0129 02:59:27.316062 (+ 293us) service_pool.cc:225] Handling call
0129 02:59:27.316418 (+ 356us) raft_consensus.cc:1471] Updating replica for 1 ops
0129 02:59:27.522085 (+205667us) raft_consensus.cc:1534] Early marking committed up to index 0
0129 02:59:27.522128 (+ 43us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0129 02:59:27.532201 (+ 10073us) log.cc:844] Serialized 26 byte log entry
0129 02:59:28.408040 (+875839us) raft_consensus.cc:1645] Marking committed up to 5
0129 02:59:28.408101 (+ 61us) raft_consensus.cc:1697] Filling consensus response to leader.
0129 02:59:28.408199 (+ 98us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0129 02:59:29.394325 (+986126us) raft_consensus.cc:1685] finished
0129 02:59:29.394356 (+ 31us) raft_consensus.cc:1691] UpdateReplicas() finished
0129 02:59:29.394531 (+ 175us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":2268,"raft.run_cpu_time_us":125,"raft.run_wall_time_us":125,"thread_start_us":1046229,"threads_started":2,"wal-append.queue_time_us":1113315}
I20260129 02:59:29.479241 20435 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55676 (request call id 1062) took 1158 ms. Trace:
I20260129 02:59:29.479704 20435 rpcz_store.cc:276] 0129 02:59:28.321237 (+ 0us) service_pool.cc:168] Inserting onto call queue
0129 02:59:29.478287 (+1157050us) service_pool.cc:225] Handling call
0129 02:59:29.478600 (+ 313us) raft_consensus.cc:1471] Updating replica for 0 ops
0129 02:59:29.478814 (+ 214us) raft_consensus.cc:1534] Early marking committed up to index 0
0129 02:59:29.478853 (+ 39us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0129 02:59:29.478919 (+ 66us) raft_consensus.cc:1645] Marking committed up to 2
0129 02:59:29.478961 (+ 42us) raft_consensus.cc:1697] Filling consensus response to leader.
0129 02:59:29.479028 (+ 67us) raft_consensus.cc:1691] UpdateReplicas() finished
0129 02:59:29.479173 (+ 145us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260129 02:59:29.422238 22191 raft_consensus.cc:515] T 3906b080a5b34a74b8b64128635e8997 P 974b802141bd4f13beffef2aa03f88f9 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:59:29.483521 20441 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55672 (request call id 1375) took 1211 ms. Trace:
I20260129 02:59:29.483914 20441 rpcz_store.cc:276] 0129 02:59:28.272052 (+ 0us) service_pool.cc:168] Inserting onto call queue
0129 02:59:29.461461 (+1189409us) service_pool.cc:225] Handling call
0129 02:59:29.482648 (+ 21187us) raft_consensus.cc:1471] Updating replica for 0 ops
0129 02:59:29.482843 (+ 195us) raft_consensus.cc:1534] Early marking committed up to index 0
0129 02:59:29.482897 (+ 54us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0129 02:59:29.482950 (+ 53us) raft_consensus.cc:1645] Marking committed up to 1
0129 02:59:29.482990 (+ 40us) raft_consensus.cc:1697] Filling consensus response to leader.
0129 02:59:29.483197 (+ 207us) raft_consensus.cc:1691] UpdateReplicas() finished
0129 02:59:29.483468 (+ 271us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260129 02:59:29.485996 22140 leader_election.cc:290] T 8055232cf0004d40a85f3bf983fe00a9 P 1c50de89a5a845668696dcb28c3badd6 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 197b880fa39d432a9b472845876bfaf0 (127.18.211.1:42027), 974b802141bd4f13beffef2aa03f88f9 (127.18.211.2:34517)
I20260129 02:59:29.486454 20433 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:48356 (request call id 194) took 2170 ms. Trace:
I20260129 02:59:29.486734 22059 leader_election.cc:290] T 96609ac68bdf4993a12e1c23355ca013 P 7a9daf5d42a84615a6383cfec9a10608 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 197b880fa39d432a9b472845876bfaf0 (127.18.211.1:42027), 974b802141bd4f13beffef2aa03f88f9 (127.18.211.2:34517), 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3:37735)
I20260129 02:59:29.486932 20433 rpcz_store.cc:276] 0129 02:59:27.316153 (+ 0us) service_pool.cc:168] Inserting onto call queue
0129 02:59:27.319548 (+ 3395us) service_pool.cc:225] Handling call
0129 02:59:27.328275 (+ 8727us) raft_consensus.cc:1471] Updating replica for 0 ops
0129 02:59:29.011836 (+1683561us) raft_consensus.cc:1697] Filling consensus response to leader.
0129 02:59:29.486180 (+474344us) spinlock_profiling.cc:243] Waited 1.57 s on lock 0x7b54002e2054. stack: 00007f8fe044e373 00007f8fe044e270 00007f8fdfda0574 00007f8fe9a5c6b4 00007f8fe9a5c649 00007f8fe34975a1 00007f8fe352ea81 00007f8fe352d4be 00007f8fe84e1f49 00007f8fe1a6ed34 00007f8fe1a6ecc2 00007f8fe1a6ebf1 00007f8fe1a6eb6c 00007f8fe1a6de22 00007f8fe186d34c 00007f8fe186c786
0129 02:59:29.486368 (+ 188us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":3604384640}
I20260129 02:59:29.494619 22004 raft_consensus.cc:515] T 75be4bd6bd45474795fd472eff2953ac P 1c50de89a5a845668696dcb28c3badd6 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:59:29.497056 22131 leader_election.cc:290] T e4d18c35b0a7493e85d0a4b92956a48f P 1c50de89a5a845668696dcb28c3badd6 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 197b880fa39d432a9b472845876bfaf0 (127.18.211.1:42027), 974b802141bd4f13beffef2aa03f88f9 (127.18.211.2:34517)
I20260129 02:59:29.525359 22164 leader_election.cc:290] T 4f8f3a38f6e24ba4857875181f447442 P 1c50de89a5a845668696dcb28c3badd6 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 197b880fa39d432a9b472845876bfaf0 (127.18.211.1:42027), 974b802141bd4f13beffef2aa03f88f9 (127.18.211.2:34517)
I20260129 02:59:29.400079 22183 raft_consensus.cc:515] T 4e420f90884f4796997b5ea41e49e411 P 974b802141bd4f13beffef2aa03f88f9 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } }
I20260129 02:59:29.505265 22033 leader_election.cc:290] T 2ac659d732314e1288849f08fe5e074e P 7a9daf5d42a84615a6383cfec9a10608 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 197b880fa39d432a9b472845876bfaf0 (127.18.211.1:42027), 974b802141bd4f13beffef2aa03f88f9 (127.18.211.2:34517)
W20260129 02:59:29.578984 22155 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.555s user 0.001s sys 0.001s
W20260129 02:59:29.607821 20481 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.636s user 0.005s sys 0.003s
I20260129 02:59:29.612612 20554 leader_election.cc:304] T 505580c93c694bbb8ea96fdf9b17ab46 P 1c50de89a5a845668696dcb28c3badd6 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1c50de89a5a845668696dcb28c3badd6, 7a9daf5d42a84615a6383cfec9a10608; no voters:
I20260129 02:59:29.616117 21134 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "cb0b3f259ebc4310927338a8740e87b8" candidate_uuid: "1c50de89a5a845668696dcb28c3badd6" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "7a9daf5d42a84615a6383cfec9a10608" is_pre_election: true
I20260129 02:59:29.407305 22185 raft_consensus.cc:515] T a609ac02fae84725ab743f95f51d33ce P 974b802141bd4f13beffef2aa03f88f9 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } }
W20260129 02:59:29.617229 20584 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.639s user 0.000s sys 0.010s
W20260129 02:59:29.617786 20407 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.770s user 0.036s sys 0.071s
W20260129 02:59:29.647449 20404 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.764s user 0.008s sys 0.029s
W20260129 02:59:29.628666 20481 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.657s user 0.005s sys 0.003s
W20260129 02:59:29.665550 20404 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.800s user 0.008s sys 0.029s
I20260129 02:59:29.629823 21134 raft_consensus.cc:2468] T cb0b3f259ebc4310927338a8740e87b8 P 7a9daf5d42a84615a6383cfec9a10608 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1c50de89a5a845668696dcb28c3badd6 in term 3.
W20260129 02:59:29.630962 20440 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.764s user 0.025s sys 0.027s
W20260129 02:59:29.637519 20511 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.708s user 0.021s sys 0.021s
I20260129 02:59:29.564527 22059 raft_consensus.cc:493] T 75b0b6e82ddc4f92b1fe1d94f9a0b73b P 7a9daf5d42a84615a6383cfec9a10608 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 1c50de89a5a845668696dcb28c3badd6)
W20260129 02:59:29.669132 20511 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.756s user 0.024s sys 0.021s
W20260129 02:59:29.647701 20407 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.800s user 0.037s sys 0.071s
W20260129 02:59:29.647637 20584 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.670s user 0.000s sys 0.010s
I20260129 02:59:29.664728 21131 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "dc2f4cb752cf4d59b5f6bc689f5d8104" candidate_uuid: "1c50de89a5a845668696dcb28c3badd6" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "7a9daf5d42a84615a6383cfec9a10608" is_pre_election: true
W20260129 02:59:29.671135 20407 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.211.2:34517, user_credentials={real_user=slave}} blocked reactor thread for 823699us
W20260129 02:59:29.668622 20440 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.802s user 0.025s sys 0.027s
W20260129 02:59:29.617985 20436 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.714s user 0.016s sys 0.025s
W20260129 02:59:29.646708 22155 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.630s user 0.001s sys 0.001s
I20260129 02:59:29.677666 20554 leader_election.cc:304] T cb0b3f259ebc4310927338a8740e87b8 P 1c50de89a5a845668696dcb28c3badd6 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1c50de89a5a845668696dcb28c3badd6, 7a9daf5d42a84615a6383cfec9a10608; no voters:
W20260129 02:59:29.678239 20581 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.690s user 0.001s sys 0.012s
I20260129 02:59:29.677897 20508 raft_consensus.cc:2468] T a609ac02fae84725ab743f95f51d33ce P 974b802141bd4f13beffef2aa03f88f9 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 197b880fa39d432a9b472845876bfaf0 in term 4.
W20260129 02:59:29.678961 20581 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.705s user 0.001s sys 0.013s
I20260129 02:59:29.680980 21135 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4e3805ffa70242e8ade66da260dddf66" candidate_uuid: "1c50de89a5a845668696dcb28c3badd6" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "7a9daf5d42a84615a6383cfec9a10608" is_pre_election: true
I20260129 02:59:29.487329 22187 raft_consensus.cc:515] T fd8311b779af4e1fa1402b2a02a602f2 P 7a9daf5d42a84615a6383cfec9a10608 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } }
I20260129 02:59:29.662770 22131 raft_consensus.cc:2804] T 505580c93c694bbb8ea96fdf9b17ab46 P 1c50de89a5a845668696dcb28c3badd6 [term 2 FOLLOWER]: Leader pre-election won for term 3
W20260129 02:59:29.694116 20479 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.802s user 0.009s sys 0.017s
W20260129 02:59:29.672570 20434 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.681s user 0.009s sys 0.007s
W20260129 02:59:29.679387 22035 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.670s user 0.006s sys 0.011s
W20260129 02:59:29.679042 20436 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.774s user 0.016s sys 0.028s
I20260129 02:59:29.682088 22053 leader_election.cc:290] T febffe4a8b394368b69b1d164308fb9a P 7a9daf5d42a84615a6383cfec9a10608 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 197b880fa39d432a9b472845876bfaf0 (127.18.211.1:42027), 974b802141bd4f13beffef2aa03f88f9 (127.18.211.2:34517)
I20260129 02:59:29.672132 22079 raft_consensus.cc:493] T b189f08e16a74a11980cc55d99359a0a P 1c50de89a5a845668696dcb28c3badd6 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 197b880fa39d432a9b472845876bfaf0)
W20260129 02:59:29.706228 20556 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.719s user 0.003s sys 0.019s
W20260129 02:59:29.724234 20556 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.737s user 0.004s sys 0.019s
I20260129 02:59:29.674417 22077 raft_consensus.cc:493] T 724a236524cb4e7ba3428880681d6cc2 P 197b880fa39d432a9b472845876bfaf0 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 974b802141bd4f13beffef2aa03f88f9)
I20260129 02:59:29.674252 22193 raft_consensus.cc:493] T 0340229d521b432580ae926f6062e3f7 P 974b802141bd4f13beffef2aa03f88f9 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 197b880fa39d432a9b472845876bfaf0)
I20260129 02:59:29.678468 22191 leader_election.cc:290] T 3906b080a5b34a74b8b64128635e8997 P 974b802141bd4f13beffef2aa03f88f9 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 197b880fa39d432a9b472845876bfaf0 (127.18.211.1:42027), 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3:37735)
I20260129 02:59:29.679210 22201 raft_consensus.cc:493] T add8d4923d1a489e86882c3a25692b95 P 974b802141bd4f13beffef2aa03f88f9 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 1c50de89a5a845668696dcb28c3badd6)
W20260129 02:59:29.699522 20479 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.807s user 0.009s sys 0.017s
I20260129 02:59:29.696656 22103 raft_consensus.cc:2804] T cb0b3f259ebc4310927338a8740e87b8 P 1c50de89a5a845668696dcb28c3badd6 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260129 02:59:29.679948 21991 consensus_queue.cc:1059] T 42d27a28d89b409e8807ec0efd97af69 P 1c50de89a5a845668696dcb28c3badd6 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 }, Status: INVALID_TERM, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.042s
I20260129 02:59:29.694531 22131 raft_consensus.cc:493] T 505580c93c694bbb8ea96fdf9b17ab46 P 1c50de89a5a845668696dcb28c3badd6 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 974b802141bd4f13beffef2aa03f88f9)
W20260129 02:59:29.710228 22114 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 1.696s user 0.014s sys 0.025s
W20260129 02:59:29.707635 22035 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.709s user 0.006s sys 0.011s
W20260129 02:59:29.705273 20434 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.718s user 0.010s sys 0.008s
I20260129 02:59:29.674045 22059 raft_consensus.cc:515] T 75b0b6e82ddc4f92b1fe1d94f9a0b73b P 7a9daf5d42a84615a6383cfec9a10608 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } }
I20260129 02:59:29.716140 21942 leader_election.cc:290] T e4d18c35b0a7493e85d0a4b92956a48f P 197b880fa39d432a9b472845876bfaf0 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 974b802141bd4f13beffef2aa03f88f9 (127.18.211.2:34517), 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3:37735)
I20260129 02:59:29.701124 22156 consensus_queue.cc:1059] T c934d31efd3f4e5195605ddc354d14e1 P 7a9daf5d42a84615a6383cfec9a10608 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.032s
W20260129 02:59:29.738112 22181 log.cc:927] Time spent T a0fcc58d6fb6473cafbd9d34d5195269 P 197b880fa39d432a9b472845876bfaf0: Append to log took a long time: real 0.095s user 0.001s sys 0.000s
W20260129 02:59:29.727540 21100 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.585s user 0.011s sys 0.030s
I20260129 02:59:29.732355 22120 leader_election.cc:290] T 2e58e9a5a8aa4cecafb464addb4f35ed P 197b880fa39d432a9b472845876bfaf0 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3:37735), 7a9daf5d42a84615a6383cfec9a10608 (127.18.211.4:40439)
W20260129 02:59:29.810767 21100 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.669s user 0.011s sys 0.030s
I20260129 02:59:29.739904 22116 leader_election.cc:290] T 724a236524cb4e7ba3428880681d6cc2 P 1c50de89a5a845668696dcb28c3badd6 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 197b880fa39d432a9b472845876bfaf0 (127.18.211.1:42027), 974b802141bd4f13beffef2aa03f88f9 (127.18.211.2:34517)
W20260129 02:59:29.794672 22114 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 1.783s user 0.014s sys 0.026s
W20260129 02:59:29.812214 20405 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.965s user 0.013s sys 0.032s
I20260129 02:59:29.737614 22103 raft_consensus.cc:493] T cb0b3f259ebc4310927338a8740e87b8 P 1c50de89a5a845668696dcb28c3badd6 [term 3 FOLLOWER]: Starting leader election (detected failure of leader 974b802141bd4f13beffef2aa03f88f9)
I20260129 02:59:29.794394 22131 raft_consensus.cc:3060] T 505580c93c694bbb8ea96fdf9b17ab46 P 1c50de89a5a845668696dcb28c3badd6 [term 2 FOLLOWER]: Advancing to term 3
I20260129 02:59:29.727319 22079 raft_consensus.cc:515] T b189f08e16a74a11980cc55d99359a0a P 1c50de89a5a845668696dcb28c3badd6 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:59:29.733183 22193 raft_consensus.cc:515] T 0340229d521b432580ae926f6062e3f7 P 974b802141bd4f13beffef2aa03f88f9 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:59:29.795536 22107 leader_election.cc:290] T 0f20af7fbe5d4db89a6f17948d98912c P 1c50de89a5a845668696dcb28c3badd6 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 197b880fa39d432a9b472845876bfaf0 (127.18.211.1:42027), 974b802141bd4f13beffef2aa03f88f9 (127.18.211.2:34517)
W20260129 02:59:29.814822 22170 log.cc:927] Time spent T 76f713c3331540d2a5b76164103e9eb1 P 974b802141bd4f13beffef2aa03f88f9: Append to log took a long time: real 0.378s user 0.001s sys 0.001s
I20260129 02:59:29.732837 22092 raft_consensus.cc:515] T 76f713c3331540d2a5b76164103e9eb1 P 197b880fa39d432a9b472845876bfaf0 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
W20260129 02:59:29.846153 20439 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.978s user 0.012s sys 0.080s
I20260129 02:59:29.729868 22077 raft_consensus.cc:515] T 724a236524cb4e7ba3428880681d6cc2 P 197b880fa39d432a9b472845876bfaf0 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:59:29.812125 22004 leader_election.cc:290] T 75be4bd6bd45474795fd472eff2953ac P 1c50de89a5a845668696dcb28c3badd6 [CANDIDATE]: Term 7 election: Requested vote from peers 197b880fa39d432a9b472845876bfaf0 (127.18.211.1:42027), 974b802141bd4f13beffef2aa03f88f9 (127.18.211.2:34517)
I20260129 02:59:29.804841 22203 raft_consensus.cc:493] T 17d6431cc77a491e8464d68db4081a51 P 1c50de89a5a845668696dcb28c3badd6 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 974b802141bd4f13beffef2aa03f88f9)
W20260129 02:59:29.844091 20405 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.997s user 0.013s sys 0.032s
W20260129 02:59:29.912554 20405 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.211.4:40439, user_credentials={real_user=slave}} blocked reactor thread for 1.06568e+06us
W20260129 02:59:29.853320 20480 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 1.009s user 0.021s sys 0.024s
W20260129 02:59:29.853404 20482 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.670s user 0.004s sys 0.015s
W20260129 02:59:29.858553 21102 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.211.2:34517, user_credentials={real_user=slave}} blocked reactor thread for 516565us
W20260129 02:59:29.812252 20509 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.920s user 0.018s sys 0.024s
I20260129 02:59:29.847188 22103 raft_consensus.cc:3060] T cb0b3f259ebc4310927338a8740e87b8 P 1c50de89a5a845668696dcb28c3badd6 [term 3 FOLLOWER]: Advancing to term 4
W20260129 02:59:29.895670 20439 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 1.047s user 0.013s sys 0.080s
I20260129 02:59:29.739389 22201 raft_consensus.cc:515] T add8d4923d1a489e86882c3a25692b95 P 974b802141bd4f13beffef2aa03f88f9 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } }
W20260129 02:59:29.844991 21100 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.211.3:37735, user_credentials={real_user=slave}} blocked reactor thread for 825573us
I20260129 02:59:29.846944 22090 raft_consensus.cc:493] T 4f5a77e94fef4b16a2ba7b067c8b144a P 1c50de89a5a845668696dcb28c3badd6 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 974b802141bd4f13beffef2aa03f88f9)
I20260129 02:59:29.904476 22157 raft_consensus.cc:493] T e33a417cf9c54cf5af73a4ee3aac4d3a P 974b802141bd4f13beffef2aa03f88f9 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 1c50de89a5a845668696dcb28c3badd6)
W20260129 02:59:29.979931 20482 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.974s user 0.009s sys 0.016s
I20260129 02:59:29.965644 21135 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "18f30ebaa4174fd28c958e72d0cc7e24" candidate_uuid: "197b880fa39d432a9b472845876bfaf0" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "7a9daf5d42a84615a6383cfec9a10608" is_pre_election: true
W20260129 02:59:29.946000 22209 log.cc:927] Time spent T a0fcc58d6fb6473cafbd9d34d5195269 P 1c50de89a5a845668696dcb28c3badd6: Append to log took a long time: real 0.065s user 0.000s sys 0.000s
W20260129 02:59:29.954912 20514 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.547s user 0.011s sys 0.037s
I20260129 02:59:29.849584 22191 raft_consensus.cc:493] T 76f713c3331540d2a5b76164103e9eb1 P 974b802141bd4f13beffef2aa03f88f9 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 1c50de89a5a845668696dcb28c3badd6)
I20260129 02:59:29.850528 22197 raft_consensus.cc:493] T 1299723a4476420ab02fd8bfd7aa0645 P 974b802141bd4f13beffef2aa03f88f9 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260129 02:59:29.949621 22120 raft_consensus.cc:493] T 368953c9102643b2a21f99af36b51905 P 197b880fa39d432a9b472845876bfaf0 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260129 02:59:29.985323 20514 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.578s user 0.012s sys 0.037s
I20260129 02:59:29.986789 20556 leader_election.cc:304] T 3906b080a5b34a74b8b64128635e8997 P 1c50de89a5a845668696dcb28c3badd6 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1c50de89a5a845668696dcb28c3badd6, 974b802141bd4f13beffef2aa03f88f9; no voters: 197b880fa39d432a9b472845876bfaf0
W20260129 02:59:29.987293 20480 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 1.143s user 0.025s sys 0.035s
I20260129 02:59:29.989593 21131 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c934d31efd3f4e5195605ddc354d14e1" candidate_uuid: "197b880fa39d432a9b472845876bfaf0" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "7a9daf5d42a84615a6383cfec9a10608"
I20260129 02:59:29.974843 22116 raft_consensus.cc:493] T fd8311b779af4e1fa1402b2a02a602f2 P 1c50de89a5a845668696dcb28c3badd6 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 197b880fa39d432a9b472845876bfaf0)
I20260129 02:59:30.076676 20405 leader_election.cc:304] T 1ce4ddf5b10e4ae8b193b13b2e5936ee P 197b880fa39d432a9b472845876bfaf0 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 197b880fa39d432a9b472845876bfaf0; no voters: 1c50de89a5a845668696dcb28c3badd6, 974b802141bd4f13beffef2aa03f88f9
I20260129 02:59:30.126471 21133 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34046 (request call id 362) took 1412 ms. Trace:
I20260129 02:59:30.127285 21133 rpcz_store.cc:276] 0129 02:59:28.714330 (+ 0us) service_pool.cc:168] Inserting onto call queue
0129 02:59:28.714506 (+ 176us) service_pool.cc:225] Handling call
0129 02:59:30.126070 (+1411564us) spinlock_profiling.cc:243] Waited 3.88 ms on lock 0x7b5400730354. stack: 00007f8fe044e373 00007f8fe044e270 00007f8fdfda0574 00007f8fe9a5c6b4 00007f8fe9a5c649 00007f8fe34975a1 00007f8fe3531f7a 00007f8fe84e3647 00007f8fe1a73834 00007f8fe1a737c2 00007f8fe1a736f1 00007f8fe1a7366c 00007f8fe1a72922 00007f8fe186d34c 00007f8fe186c786 00007f8fe186c12f
0129 02:59:30.126375 (+ 305us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":8915328}
I20260129 02:59:29.976687 22059 leader_election.cc:290] T 75b0b6e82ddc4f92b1fe1d94f9a0b73b P 7a9daf5d42a84615a6383cfec9a10608 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 974b802141bd4f13beffef2aa03f88f9 (127.18.211.2:34517), 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3:37735)
I20260129 02:59:30.097626 20510 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4f8f3a38f6e24ba4857875181f447442" candidate_uuid: "197b880fa39d432a9b472845876bfaf0" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "974b802141bd4f13beffef2aa03f88f9" is_pre_election: true
I20260129 02:59:30.133790 20510 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c891d2bff4ce4568861bbc992e91fbb0" candidate_uuid: "197b880fa39d432a9b472845876bfaf0" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "974b802141bd4f13beffef2aa03f88f9" is_pre_election: true
I20260129 02:59:30.135448 20510 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3e2d15f8640b407195210c476e09df91" candidate_uuid: "7a9daf5d42a84615a6383cfec9a10608" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "974b802141bd4f13beffef2aa03f88f9" is_pre_election: true
I20260129 02:59:30.104897 21136 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "78aa39096868415d9afc2ffcb248e198" candidate_uuid: "197b880fa39d432a9b472845876bfaf0" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "7a9daf5d42a84615a6383cfec9a10608" is_pre_election: true
I20260129 02:59:30.030774 20512 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a4c1226b987b468fa84dfc6ba0e2dbe1" candidate_uuid: "197b880fa39d432a9b472845876bfaf0" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "974b802141bd4f13beffef2aa03f88f9"
W20260129 02:59:29.984861 20509 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 1.098s user 0.018s sys 0.028s
W20260129 02:59:30.117799 20556 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.211.2:34517, user_credentials={real_user=slave}} blocked reactor thread for 131116us
I20260129 02:59:30.138782 20556 leader_election.cc:304] T a4c1226b987b468fa84dfc6ba0e2dbe1 P 1c50de89a5a845668696dcb28c3badd6 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1c50de89a5a845668696dcb28c3badd6, 974b802141bd4f13beffef2aa03f88f9; no voters: 197b880fa39d432a9b472845876bfaf0
I20260129 02:59:30.139976 20556 leader_election.cc:304] T a8479d1fe0bd41e5946186e3a6ca8c7a P 1c50de89a5a845668696dcb28c3badd6 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1c50de89a5a845668696dcb28c3badd6; no voters: 197b880fa39d432a9b472845876bfaf0, 974b802141bd4f13beffef2aa03f88f9
I20260129 02:59:30.123437 20581 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "add8d4923d1a489e86882c3a25692b95" candidate_uuid: "7a9daf5d42a84615a6383cfec9a10608" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "1c50de89a5a845668696dcb28c3badd6" is_pre_election: true
I20260129 02:59:30.123965 20513 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c934d31efd3f4e5195605ddc354d14e1" candidate_uuid: "197b880fa39d432a9b472845876bfaf0" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "974b802141bd4f13beffef2aa03f88f9"
I20260129 02:59:30.128713 20554 leader_election.cc:304] T 1299723a4476420ab02fd8bfd7aa0645 P 1c50de89a5a845668696dcb28c3badd6 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1c50de89a5a845668696dcb28c3badd6, 7a9daf5d42a84615a6383cfec9a10608; no voters: 197b880fa39d432a9b472845876bfaf0
I20260129 02:59:29.982810 22185 leader_election.cc:290] T a609ac02fae84725ab743f95f51d33ce P 974b802141bd4f13beffef2aa03f88f9 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 197b880fa39d432a9b472845876bfaf0 (127.18.211.1:42027), 7a9daf5d42a84615a6383cfec9a10608 (127.18.211.4:40439)
I20260129 02:59:30.143393 21135 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2e58e9a5a8aa4cecafb464addb4f35ed" candidate_uuid: "197b880fa39d432a9b472845876bfaf0" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "7a9daf5d42a84615a6383cfec9a10608" is_pre_election: true
I20260129 02:59:30.128839 20511 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57584 (request call id 1125) took 2043 ms. Trace:
I20260129 02:59:29.950217 22203 raft_consensus.cc:515] T 17d6431cc77a491e8464d68db4081a51 P 1c50de89a5a845668696dcb28c3badd6 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:59:30.136987 20512 raft_consensus.cc:2376] T a4c1226b987b468fa84dfc6ba0e2dbe1 P 974b802141bd4f13beffef2aa03f88f9 [term 2 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 197b880fa39d432a9b472845876bfaf0 in term 2. Re-sending same reply.
W20260129 02:59:30.073925 20433 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.527s user 0.008s sys 0.053s
I20260129 02:59:30.141551 20513 raft_consensus.cc:3060] T c934d31efd3f4e5195605ddc354d14e1 P 974b802141bd4f13beffef2aa03f88f9 [term 2 FOLLOWER]: Advancing to term 3
W20260129 02:59:30.154482 20405 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.211.3:37735, user_credentials={real_user=slave}} blocked reactor thread for 77945.8us
I20260129 02:59:30.156419 20512 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e4d18c35b0a7493e85d0a4b92956a48f" candidate_uuid: "197b880fa39d432a9b472845876bfaf0" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "974b802141bd4f13beffef2aa03f88f9" is_pre_election: true
I20260129 02:59:30.146425 20511 rpcz_store.cc:276] 0129 02:59:28.085501 (+ 0us) service_pool.cc:168] Inserting onto call queue
0129 02:59:28.085655 (+ 154us) service_pool.cc:225] Handling call
0129 02:59:28.132407 (+ 46752us) raft_consensus.cc:1471] Updating replica for 1 ops
0129 02:59:28.140244 (+ 7837us) raft_consensus.cc:1534] Early marking committed up to index 0
0129 02:59:28.140280 (+ 36us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0129 02:59:28.170048 (+ 29768us) log.cc:844] Serialized 26 byte log entry
0129 02:59:28.654220 (+484172us) raft_consensus.cc:1645] Marking committed up to 2
0129 02:59:28.912784 (+258564us) log.cc:844] Serialized 14 byte log entry
0129 02:59:29.669956 (+757172us) raft_consensus.cc:1697] Filling consensus response to leader.
0129 02:59:29.670031 (+ 75us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0129 02:59:30.128574 (+458543us) raft_consensus.cc:1685] finished
0129 02:59:30.128603 (+ 29us) raft_consensus.cc:1691] UpdateReplicas() finished
0129 02:59:30.128763 (+ 160us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":35226,"prepare.queue_time_us":191410,"prepare.run_cpu_time_us":67,"prepare.run_wall_time_us":6512,"raft.queue_time_us":313642,"raft.run_cpu_time_us":394,"raft.run_wall_time_us":20163,"thread_start_us":1348134,"threads_started":3,"wal-append.queue_time_us":1222486}
I20260129 02:59:30.136466 20510 raft_consensus.cc:2468] T 3e2d15f8640b407195210c476e09df91 P 974b802141bd4f13beffef2aa03f88f9 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7a9daf5d42a84615a6383cfec9a10608 in term 1.
I20260129 02:59:30.028671 22090 raft_consensus.cc:515] T 4f5a77e94fef4b16a2ba7b067c8b144a P 1c50de89a5a845668696dcb28c3badd6 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } }
I20260129 02:59:30.113622 22071 raft_consensus.cc:493] T 913fc7d751ed4b7388d889aac30a5f9b P 974b802141bd4f13beffef2aa03f88f9 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 197b880fa39d432a9b472845876bfaf0)
W20260129 02:59:30.223300 20556 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.211.2:34517, user_credentials={real_user=slave}} blocked reactor thread for 71579.6us
I20260129 02:59:30.328392 22071 raft_consensus.cc:515] T 913fc7d751ed4b7388d889aac30a5f9b P 974b802141bd4f13beffef2aa03f88f9 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:59:30.329538 22090 leader_election.cc:290] T 4f5a77e94fef4b16a2ba7b067c8b144a P 1c50de89a5a845668696dcb28c3badd6 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 974b802141bd4f13beffef2aa03f88f9 (127.18.211.2:34517), 7a9daf5d42a84615a6383cfec9a10608 (127.18.211.4:40439)
I20260129 02:59:30.330453 22071 leader_election.cc:290] T 913fc7d751ed4b7388d889aac30a5f9b P 974b802141bd4f13beffef2aa03f88f9 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 197b880fa39d432a9b472845876bfaf0 (127.18.211.1:42027), 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3:37735)
I20260129 02:59:30.334832 20513 raft_consensus.cc:2468] T c934d31efd3f4e5195605ddc354d14e1 P 974b802141bd4f13beffef2aa03f88f9 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 197b880fa39d432a9b472845876bfaf0 in term 3.
I20260129 02:59:30.137820 22187 leader_election.cc:290] T fd8311b779af4e1fa1402b2a02a602f2 P 7a9daf5d42a84615a6383cfec9a10608 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 197b880fa39d432a9b472845876bfaf0 (127.18.211.1:42027), 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3:37735)
I20260129 02:59:29.984413 22157 raft_consensus.cc:515] T e33a417cf9c54cf5af73a4ee3aac4d3a P 974b802141bd4f13beffef2aa03f88f9 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:59:30.129736 22114 raft_consensus.cc:2804] T 3906b080a5b34a74b8b64128635e8997 P 1c50de89a5a845668696dcb28c3badd6 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260129 02:59:30.230698 20581 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1299723a4476420ab02fd8bfd7aa0645" candidate_uuid: "7a9daf5d42a84615a6383cfec9a10608" candidate_term: 5 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "1c50de89a5a845668696dcb28c3badd6" is_pre_election: true
I20260129 02:59:30.000041 22199 raft_consensus.cc:493] T 7d9494f5584d4d5db8b9adf1c8fc888b P 197b880fa39d432a9b472845876bfaf0 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 1c50de89a5a845668696dcb28c3badd6)
I20260129 02:59:30.034739 22160 raft_consensus.cc:493] T 00e79d8f25a34676b1d866eb2068a2b2 P 974b802141bd4f13beffef2aa03f88f9 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 197b880fa39d432a9b472845876bfaf0)
W20260129 02:59:30.321810 22202 log.cc:927] Time spent T b189f08e16a74a11980cc55d99359a0a P 1c50de89a5a845668696dcb28c3badd6: Append to log took a long time: real 0.148s user 0.000s sys 0.000s
I20260129 02:59:30.322878 22188 raft_consensus.cc:493] T 78aa39096868415d9afc2ffcb248e198 P 7a9daf5d42a84615a6383cfec9a10608 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 1c50de89a5a845668696dcb28c3badd6)
W20260129 02:59:30.325922 21130 consensus_meta.cc:287] T 42d27a28d89b409e8807ec0efd97af69 P 7a9daf5d42a84615a6383cfec9a10608: Time spent flushing consensus metadata: real 0.835s user 0.007s sys 0.000s
I20260129 02:59:30.316744 20405 leader_election.cc:304] T 75be4bd6bd45474795fd472eff2953ac P 197b880fa39d432a9b472845876bfaf0 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 197b880fa39d432a9b472845876bfaf0, 1c50de89a5a845668696dcb28c3badd6; no voters:
W20260129 02:59:30.576382 20407 thread.cc:641] dns-resolver [worker] (thread pool) Time spent creating pthread: real 0.627s user 0.055s sys 0.127s
W20260129 02:59:30.576735 20407 thread.cc:608] dns-resolver [worker] (thread pool) Time spent starting thread: real 0.628s user 0.055s sys 0.127s
I20260129 02:59:30.031098 22191 raft_consensus.cc:515] T 76f713c3331540d2a5b76164103e9eb1 P 974b802141bd4f13beffef2aa03f88f9 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:59:30.029397 22183 leader_election.cc:290] T 4e420f90884f4796997b5ea41e49e411 P 974b802141bd4f13beffef2aa03f88f9 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 197b880fa39d432a9b472845876bfaf0 (127.18.211.1:42027), 7a9daf5d42a84615a6383cfec9a10608 (127.18.211.4:40439)
W20260129 02:59:30.356773 20553 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.211.1:42027, user_credentials={real_user=slave}} blocked reactor thread for 377553us
I20260129 02:59:30.155571 22132 raft_consensus.cc:2749] T a8479d1fe0bd41e5946186e3a6ca8c7a P 1c50de89a5a845668696dcb28c3badd6 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260129 02:59:30.374697 20513 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "75f45538dc704d9a8ec4ff8ad9f3ea5b" candidate_uuid: "7a9daf5d42a84615a6383cfec9a10608" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "974b802141bd4f13beffef2aa03f88f9" is_pre_election: true
I20260129 02:59:30.158761 22126 raft_consensus.cc:2804] T a4c1226b987b468fa84dfc6ba0e2dbe1 P 1c50de89a5a845668696dcb28c3badd6 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260129 02:59:30.475090 22035 raft_consensus.cc:493] T 7a9a69cd82724b1faed1d10aa091fc4a P 1c50de89a5a845668696dcb28c3badd6 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 197b880fa39d432a9b472845876bfaf0)
I20260129 02:59:30.580147 21130 raft_consensus.cc:2468] T 42d27a28d89b409e8807ec0efd97af69 P 7a9daf5d42a84615a6383cfec9a10608 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 197b880fa39d432a9b472845876bfaf0 in term 4.
I20260129 02:59:30.616689 22126 raft_consensus.cc:493] T a4c1226b987b468fa84dfc6ba0e2dbe1 P 1c50de89a5a845668696dcb28c3badd6 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260129 02:59:30.617265 22126 raft_consensus.cc:3060] T a4c1226b987b468fa84dfc6ba0e2dbe1 P 1c50de89a5a845668696dcb28c3badd6 [term 2 FOLLOWER]: Advancing to term 3
I20260129 02:59:30.616973 22035 raft_consensus.cc:515] T 7a9a69cd82724b1faed1d10aa091fc4a P 1c50de89a5a845668696dcb28c3badd6 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:59:30.619221 22132 raft_consensus.cc:493] T 18f30ebaa4174fd28c958e72d0cc7e24 P 1c50de89a5a845668696dcb28c3badd6 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 7a9daf5d42a84615a6383cfec9a10608)
I20260129 02:59:30.318046 22011 raft_consensus.cc:3060] T 1ce4ddf5b10e4ae8b193b13b2e5936ee P 197b880fa39d432a9b472845876bfaf0 [term 2 FOLLOWER]: Advancing to term 3
W20260129 02:59:30.353907 20433 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.769s user 0.013s sys 0.058s
I20260129 02:59:30.619784 22132 raft_consensus.cc:515] T 18f30ebaa4174fd28c958e72d0cc7e24 P 1c50de89a5a845668696dcb28c3badd6 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } }
W20260129 02:59:30.570453 20556 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.211.2:34517, user_credentials={real_user=slave}} blocked reactor thread for 241730us
I20260129 02:59:30.572867 20511 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "505580c93c694bbb8ea96fdf9b17ab46" candidate_uuid: "7a9daf5d42a84615a6383cfec9a10608" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "974b802141bd4f13beffef2aa03f88f9" is_pre_election: true
W20260129 02:59:30.576609 20405 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.211.3:37735, user_credentials={real_user=slave}} blocked reactor thread for 259962us
I20260129 02:59:30.327088 22033 raft_consensus.cc:493] T 00e79d8f25a34676b1d866eb2068a2b2 P 7a9daf5d42a84615a6383cfec9a10608 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 197b880fa39d432a9b472845876bfaf0)
I20260129 02:59:30.139349 22116 raft_consensus.cc:515] T fd8311b779af4e1fa1402b2a02a602f2 P 1c50de89a5a845668696dcb28c3badd6 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } }
I20260129 02:59:30.479770 22114 raft_consensus.cc:493] T 3906b080a5b34a74b8b64128635e8997 P 1c50de89a5a845668696dcb28c3badd6 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 197b880fa39d432a9b472845876bfaf0)
I20260129 02:59:30.316977 22115 raft_consensus.cc:2804] T 1299723a4476420ab02fd8bfd7aa0645 P 1c50de89a5a845668696dcb28c3badd6 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260129 02:59:30.129964 22120 raft_consensus.cc:515] T 368953c9102643b2a21f99af36b51905 P 197b880fa39d432a9b472845876bfaf0 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:59:30.034648 22197 raft_consensus.cc:515] T 1299723a4476420ab02fd8bfd7aa0645 P 974b802141bd4f13beffef2aa03f88f9 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } attrs { replace: true } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } }
I20260129 02:59:30.585103 22195 raft_consensus.cc:2804] T 75be4bd6bd45474795fd472eff2953ac P 197b880fa39d432a9b472845876bfaf0 [term 5 FOLLOWER]: Leader pre-election won for term 6
W20260129 02:59:30.577267 20407 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.211.2:34517, user_credentials={real_user=slave}} blocked reactor thread for 628383us
I20260129 02:59:30.641225 20583 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4f5a77e94fef4b16a2ba7b067c8b144a" candidate_uuid: "7a9daf5d42a84615a6383cfec9a10608" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "1c50de89a5a845668696dcb28c3badd6" is_pre_election: true
I20260129 02:59:30.578634 20509 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57584 (request call id 1121) took 2552 ms. Trace:
W20260129 02:59:30.621601 22205 log.cc:927] Time spent T 7d9494f5584d4d5db8b9adf1c8fc888b P 197b880fa39d432a9b472845876bfaf0: Append to log took a long time: real 0.778s user 0.001s sys 0.000s
I20260129 02:59:30.583642 22028 leader_election.cc:290] T 4b1876f9a70b49edb3aa74105e66d3b3 P 974b802141bd4f13beffef2aa03f88f9 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 197b880fa39d432a9b472845876bfaf0 (127.18.211.1:42027), 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3:37735)
I20260129 02:59:30.627792 21131 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4f5a77e94fef4b16a2ba7b067c8b144a" candidate_uuid: "1c50de89a5a845668696dcb28c3badd6" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "7a9daf5d42a84615a6383cfec9a10608" is_pre_election: true
I20260129 02:59:30.633894 20587 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "cb0b3f259ebc4310927338a8740e87b8" candidate_uuid: "7a9daf5d42a84615a6383cfec9a10608" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "1c50de89a5a845668696dcb28c3badd6" is_pre_election: true
I20260129 02:59:30.641921 20583 raft_consensus.cc:2468] T 4f5a77e94fef4b16a2ba7b067c8b144a P 1c50de89a5a845668696dcb28c3badd6 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7a9daf5d42a84615a6383cfec9a10608 in term 1.
I20260129 02:59:30.621030 22059 raft_consensus.cc:493] T 3ccac82a01a2445fa5b0ce6c0fcf2295 P 7a9daf5d42a84615a6383cfec9a10608 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 197b880fa39d432a9b472845876bfaf0)
I20260129 02:59:30.625368 22193 leader_election.cc:290] T 0340229d521b432580ae926f6062e3f7 P 974b802141bd4f13beffef2aa03f88f9 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 197b880fa39d432a9b472845876bfaf0 (127.18.211.1:42027), 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3:37735)
I20260129 02:59:30.635100 22114 raft_consensus.cc:3060] T 3906b080a5b34a74b8b64128635e8997 P 1c50de89a5a845668696dcb28c3badd6 [term 1 FOLLOWER]: Advancing to term 2
W20260129 02:59:30.663839 22211 log.cc:927] Time spent T 78aa39096868415d9afc2ffcb248e198 P 7a9daf5d42a84615a6383cfec9a10608: Append to log took a long time: real 1.066s user 0.006s sys 0.008s
I20260129 02:59:30.654413 21131 raft_consensus.cc:2468] T 4f5a77e94fef4b16a2ba7b067c8b144a P 7a9daf5d42a84615a6383cfec9a10608 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1c50de89a5a845668696dcb28c3badd6 in term 1.
I20260129 02:59:30.661926 21883 raft_consensus.cc:493] T a8479d1fe0bd41e5946186e3a6ca8c7a P 197b880fa39d432a9b472845876bfaf0 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 974b802141bd4f13beffef2aa03f88f9)
I20260129 02:59:30.662899 22035 leader_election.cc:290] T 7a9a69cd82724b1faed1d10aa091fc4a P 1c50de89a5a845668696dcb28c3badd6 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 197b880fa39d432a9b472845876bfaf0 (127.18.211.1:42027), 974b802141bd4f13beffef2aa03f88f9 (127.18.211.2:34517)
I20260129 02:59:30.635349 22115 raft_consensus.cc:493] T 1299723a4476420ab02fd8bfd7aa0645 P 1c50de89a5a845668696dcb28c3badd6 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260129 02:59:30.663558 22157 leader_election.cc:290] T e33a417cf9c54cf5af73a4ee3aac4d3a P 974b802141bd4f13beffef2aa03f88f9 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 197b880fa39d432a9b472845876bfaf0 (127.18.211.1:42027), 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3:37735)
I20260129 02:59:30.637611 22004 consensus_queue.cc:1048] T 926a250fa5834e88b96ccecbefbf5086 P 1c50de89a5a845668696dcb28c3badd6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.030s
I20260129 02:59:30.649849 22201 leader_election.cc:290] T add8d4923d1a489e86882c3a25692b95 P 974b802141bd4f13beffef2aa03f88f9 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3:37735), 7a9daf5d42a84615a6383cfec9a10608 (127.18.211.4:40439)
I20260129 02:59:30.670424 22011 raft_consensus.cc:2749] T 1ce4ddf5b10e4ae8b193b13b2e5936ee P 197b880fa39d432a9b472845876bfaf0 [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260129 02:59:30.641629 22195 raft_consensus.cc:493] T 75be4bd6bd45474795fd472eff2953ac P 197b880fa39d432a9b472845876bfaf0 [term 5 FOLLOWER]: Starting leader election (detected failure of leader 1c50de89a5a845668696dcb28c3badd6)
I20260129 02:59:30.580273 22188 raft_consensus.cc:515] T 78aa39096868415d9afc2ffcb248e198 P 7a9daf5d42a84615a6383cfec9a10608 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } }
I20260129 02:59:30.713215 20439 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55676 (request call id 1054) took 3021 ms. Trace:
I20260129 02:59:30.672384 22059 raft_consensus.cc:515] T 3ccac82a01a2445fa5b0ce6c0fcf2295 P 7a9daf5d42a84615a6383cfec9a10608 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } }
W20260129 02:59:30.682881 22214 log.cc:927] Time spent T e33a417cf9c54cf5af73a4ee3aac4d3a P 197b880fa39d432a9b472845876bfaf0: Append to log took a long time: real 0.448s user 0.001s sys 0.000s
W20260129 02:59:30.683701 22216 log.cc:927] Time spent T b189f08e16a74a11980cc55d99359a0a P 974b802141bd4f13beffef2aa03f88f9: Append to log took a long time: real 0.102s user 0.000s sys 0.000s
I20260129 02:59:30.685076 21883 raft_consensus.cc:515] T a8479d1fe0bd41e5946186e3a6ca8c7a P 197b880fa39d432a9b472845876bfaf0 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:59:30.577978 22199 raft_consensus.cc:515] T 7d9494f5584d4d5db8b9adf1c8fc888b P 197b880fa39d432a9b472845876bfaf0 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:59:30.728950 20583 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "42d27a28d89b409e8807ec0efd97af69" candidate_uuid: "7a9daf5d42a84615a6383cfec9a10608" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "1c50de89a5a845668696dcb28c3badd6" is_pre_election: true
W20260129 02:59:30.685632 22131 consensus_meta.cc:287] T 505580c93c694bbb8ea96fdf9b17ab46 P 1c50de89a5a845668696dcb28c3badd6: Time spent flushing consensus metadata: real 0.825s user 0.006s sys 0.003s
I20260129 02:59:30.649210 22092 leader_election.cc:290] T 76f713c3331540d2a5b76164103e9eb1 P 197b880fa39d432a9b472845876bfaf0 [CANDIDATE]: Term 3 election: Requested vote from peers 974b802141bd4f13beffef2aa03f88f9 (127.18.211.2:34517), 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3:37735)
I20260129 02:59:30.702659 20513 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "76f713c3331540d2a5b76164103e9eb1" candidate_uuid: "197b880fa39d432a9b472845876bfaf0" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "974b802141bd4f13beffef2aa03f88f9"
I20260129 02:59:30.734422 20513 raft_consensus.cc:3060] T 76f713c3331540d2a5b76164103e9eb1 P 974b802141bd4f13beffef2aa03f88f9 [term 2 FOLLOWER]: Advancing to term 3
I20260129 02:59:30.736068 21883 leader_election.cc:290] T a8479d1fe0bd41e5946186e3a6ca8c7a P 197b880fa39d432a9b472845876bfaf0 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 974b802141bd4f13beffef2aa03f88f9 (127.18.211.2:34517), 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3:37735)
I20260129 02:59:30.737155 22199 leader_election.cc:290] T 7d9494f5584d4d5db8b9adf1c8fc888b P 197b880fa39d432a9b472845876bfaf0 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 974b802141bd4f13beffef2aa03f88f9 (127.18.211.2:34517), 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3:37735)
I20260129 02:59:30.582661 22160 raft_consensus.cc:515] T 00e79d8f25a34676b1d866eb2068a2b2 P 974b802141bd4f13beffef2aa03f88f9 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } }
I20260129 02:59:30.740772 20514 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57600 (request call id 1617) took 1847 ms. Trace:
I20260129 02:59:30.714690 22195 raft_consensus.cc:3060] T 75be4bd6bd45474795fd472eff2953ac P 197b880fa39d432a9b472845876bfaf0 [term 5 FOLLOWER]: Advancing to term 6
I20260129 02:59:30.752545 20435 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55676 (request call id 1063) took 2428 ms. Trace:
I20260129 02:59:30.753371 22188 leader_election.cc:290] T 78aa39096868415d9afc2ffcb248e198 P 7a9daf5d42a84615a6383cfec9a10608 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 197b880fa39d432a9b472845876bfaf0 (127.18.211.1:42027), 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3:37735)
I20260129 02:59:30.754665 22132 leader_election.cc:290] T 18f30ebaa4174fd28c958e72d0cc7e24 P 1c50de89a5a845668696dcb28c3badd6 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 197b880fa39d432a9b472845876bfaf0 (127.18.211.1:42027), 7a9daf5d42a84615a6383cfec9a10608 (127.18.211.4:40439)
I20260129 02:59:30.755436 20513 raft_consensus.cc:2468] T 76f713c3331540d2a5b76164103e9eb1 P 974b802141bd4f13beffef2aa03f88f9 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 197b880fa39d432a9b472845876bfaf0 in term 3.
I20260129 02:59:30.755774 22203 leader_election.cc:290] T 17d6431cc77a491e8464d68db4081a51 P 1c50de89a5a845668696dcb28c3badd6 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 197b880fa39d432a9b472845876bfaf0 (127.18.211.1:42027), 974b802141bd4f13beffef2aa03f88f9 (127.18.211.2:34517)
I20260129 02:59:30.718653 20434 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55676 (request call id 1051) took 3185 ms. Trace:
I20260129 02:59:30.756407 22191 leader_election.cc:290] T 76f713c3331540d2a5b76164103e9eb1 P 974b802141bd4f13beffef2aa03f88f9 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 197b880fa39d432a9b472845876bfaf0 (127.18.211.1:42027), 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3:37735)
I20260129 02:59:30.757622 22120 leader_election.cc:290] T 368953c9102643b2a21f99af36b51905 P 197b880fa39d432a9b472845876bfaf0 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 974b802141bd4f13beffef2aa03f88f9 (127.18.211.2:34517), 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3:37735)
I20260129 02:59:30.756543 20434 rpcz_store.cc:276] 0129 02:59:27.532904 (+ 0us) service_pool.cc:168] Inserting onto call queue
0129 02:59:27.574953 (+ 42049us) service_pool.cc:225] Handling call
0129 02:59:27.602279 (+ 27326us) raft_consensus.cc:1471] Updating replica for 1 ops
0129 02:59:27.695072 (+ 92793us) raft_consensus.cc:1534] Early marking committed up to index 0
0129 02:59:27.729224 (+ 34152us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0129 02:59:28.895269 (+1166045us) spinlock_profiling.cc:243] Waited 1.55 ms on lock 0x7b2c000014c8. stack: 00007f8fe044e373 00007f8fe044e270 00007f8fdfda0574 00007f8fe9a5c6b4 00007f8fe9a5c649 00007f8fe1e2c9b1 00007f8fe1e3bfe2 00007f8fe3569851 00007f8fe352e0ba 00007f8fe352d4be 00007f8fe84e1f49 00007f8fe1a6ed34 00007f8fe1a6ecc2 00007f8fe1a6ebf1 00007f8fe1a6eb6c 00007f8fe1a6de22
0129 02:59:28.937555 (+ 42286us) log.cc:844] Serialized 26 byte log entry
0129 02:59:29.812450 (+874895us) raft_consensus.cc:1645] Marking committed up to 1
0129 02:59:29.845674 (+ 33224us) raft_consensus.cc:1697] Filling consensus response to leader.
0129 02:59:29.849110 (+ 3436us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0129 02:59:30.690651 (+841541us) raft_consensus.cc:1685] finished
0129 02:59:30.690680 (+ 29us) raft_consensus.cc:1691] UpdateReplicas() finished
0129 02:59:30.716850 (+ 26170us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":4635,"raft.queue_time_us":9942,"raft.run_cpu_time_us":177,"raft.run_wall_time_us":32605,"spinlock_wait_cycles":3574144,"thread_start_us":814357,"threads_started":1,"wal-append.queue_time_us":741010}
I20260129 02:59:30.759428 22197 leader_election.cc:290] T 1299723a4476420ab02fd8bfd7aa0645 P 974b802141bd4f13beffef2aa03f88f9 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 197b880fa39d432a9b472845876bfaf0 (127.18.211.1:42027), 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3:37735), 7a9daf5d42a84615a6383cfec9a10608 (127.18.211.4:40439)
I20260129 02:59:30.713936 20584 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35754 (request call id 1518) took 1925 ms. Trace:
I20260129 02:59:30.714128 20439 rpcz_store.cc:276] 0129 02:59:27.691567 (+ 0us) service_pool.cc:168] Inserting onto call queue
0129 02:59:28.826015 (+1134448us) service_pool.cc:225] Handling call
0129 02:59:28.837354 (+ 11339us) raft_consensus.cc:1471] Updating replica for 1 ops
0129 02:59:29.992113 (+1154759us) raft_consensus.cc:1534] Early marking committed up to index 0
0129 02:59:29.992148 (+ 35us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0129 02:59:29.992633 (+ 485us) log.cc:844] Serialized 26 byte log entry
0129 02:59:30.625663 (+633030us) raft_consensus.cc:1645] Marking committed up to 2
0129 02:59:30.625704 (+ 41us) raft_consensus.cc:1697] Filling consensus response to leader.
0129 02:59:30.625786 (+ 82us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0129 02:59:30.713020 (+ 87234us) raft_consensus.cc:1685] finished
0129 02:59:30.713038 (+ 18us) raft_consensus.cc:1691] UpdateReplicas() finished
0129 02:59:30.713163 (+ 125us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":9694,"raft.queue_time_us":133521,"raft.run_cpu_time_us":178,"raft.run_wall_time_us":13770,"thread_start_us":1620633,"threads_started":2,"wal-append.queue_time_us":668334}
I20260129 02:59:30.691047 22115 raft_consensus.cc:3060] T 1299723a4476420ab02fd8bfd7aa0645 P 1c50de89a5a845668696dcb28c3badd6 [term 4 FOLLOWER]: Advancing to term 5
I20260129 02:59:30.752957 20435 rpcz_store.cc:276] 0129 02:59:28.324464 (+ 0us) service_pool.cc:168] Inserting onto call queue
0129 02:59:29.480785 (+1156321us) service_pool.cc:225] Handling call
0129 02:59:29.481166 (+ 381us) raft_consensus.cc:1471] Updating replica for 1 ops
0129 02:59:29.488953 (+ 7787us) raft_consensus.cc:1534] Early marking committed up to index 0
0129 02:59:29.488980 (+ 27us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0129 02:59:29.491096 (+ 2116us) log.cc:844] Serialized 26 byte log entry
0129 02:59:29.844602 (+353506us) raft_consensus.cc:1645] Marking committed up to 1
0129 02:59:29.844667 (+ 65us) raft_consensus.cc:1697] Filling consensus response to leader.
0129 02:59:29.844768 (+ 101us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0129 02:59:30.752340 (+907572us) raft_consensus.cc:1685] finished
0129 02:59:30.752365 (+ 25us) raft_consensus.cc:1691] UpdateReplicas() finished
0129 02:59:30.752490 (+ 125us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":717,"raft.run_cpu_time_us":201,"raft.run_wall_time_us":13744,"thread_start_us":274925,"threads_started":1,"wal-append.queue_time_us":445911}
I20260129 02:59:30.762457 20435 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4408006eb5174de1896b037cf3f18457" candidate_uuid: "1c50de89a5a845668696dcb28c3badd6" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "197b880fa39d432a9b472845876bfaf0" is_pre_election: true
I20260129 02:59:30.760556 20439 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4e3805ffa70242e8ade66da260dddf66" candidate_uuid: "1c50de89a5a845668696dcb28c3badd6" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "197b880fa39d432a9b472845876bfaf0" is_pre_election: true
I20260129 02:59:30.763473 20435 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55676 (request call id 1072) took 1069 ms. Trace:
I20260129 02:59:30.763630 20513 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "add8d4923d1a489e86882c3a25692b95" candidate_uuid: "7a9daf5d42a84615a6383cfec9a10608" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "974b802141bd4f13beffef2aa03f88f9" is_pre_election: true
I20260129 02:59:30.741096 20588 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "75b0b6e82ddc4f92b1fe1d94f9a0b73b" candidate_uuid: "7a9daf5d42a84615a6383cfec9a10608" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "1c50de89a5a845668696dcb28c3badd6" is_pre_election: true
I20260129 02:59:30.629603 22033 raft_consensus.cc:515] T 00e79d8f25a34676b1d866eb2068a2b2 P 7a9daf5d42a84615a6383cfec9a10608 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } }
I20260129 02:59:30.741223 20514 rpcz_store.cc:276] 0129 02:59:28.893809 (+ 0us) service_pool.cc:168] Inserting onto call queue
0129 02:59:29.360712 (+466903us) service_pool.cc:225] Handling call
0129 02:59:29.367700 (+ 6988us) raft_consensus.cc:1471] Updating replica for 1 ops
0129 02:59:29.368183 (+ 483us) raft_consensus.cc:1534] Early marking committed up to index 0
0129 02:59:29.368369 (+ 186us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0129 02:59:29.403251 (+ 34882us) log.cc:844] Serialized 26 byte log entry
0129 02:59:30.001449 (+598198us) raft_consensus.cc:1645] Marking committed up to 2
0129 02:59:30.001505 (+ 56us) raft_consensus.cc:1697] Filling consensus response to leader.
0129 02:59:30.001601 (+ 96us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0129 02:59:30.740559 (+738958us) raft_consensus.cc:1685] finished
0129 02:59:30.740585 (+ 26us) raft_consensus.cc:1691] UpdateReplicas() finished
0129 02:59:30.740715 (+ 130us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":4375,"raft.run_cpu_time_us":155,"raft.run_wall_time_us":5812,"thread_start_us":593507,"threads_started":1,"wal-append.queue_time_us":735992}
I20260129 02:59:30.742560 20583 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "96609ac68bdf4993a12e1c23355ca013" candidate_uuid: "7a9daf5d42a84615a6383cfec9a10608" candidate_term: 4 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "1c50de89a5a845668696dcb28c3badd6" is_pre_election: true
I20260129 02:59:30.770767 20583 raft_consensus.cc:2468] T 96609ac68bdf4993a12e1c23355ca013 P 1c50de89a5a845668696dcb28c3badd6 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7a9daf5d42a84615a6383cfec9a10608 in term 3.
I20260129 02:59:30.745419 22160 leader_election.cc:290] T 00e79d8f25a34676b1d866eb2068a2b2 P 974b802141bd4f13beffef2aa03f88f9 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 197b880fa39d432a9b472845876bfaf0 (127.18.211.1:42027), 7a9daf5d42a84615a6383cfec9a10608 (127.18.211.4:40439)
I20260129 02:59:30.781176 20514 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "add8d4923d1a489e86882c3a25692b95" candidate_uuid: "7a9daf5d42a84615a6383cfec9a10608" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "974b802141bd4f13beffef2aa03f88f9" is_pre_election: true
I20260129 02:59:30.740480 22131 raft_consensus.cc:515] T 505580c93c694bbb8ea96fdf9b17ab46 P 1c50de89a5a845668696dcb28c3badd6 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } }
I20260129 02:59:30.749135 22143 leader_election.cc:290] T 4e420f90884f4796997b5ea41e49e411 P 197b880fa39d432a9b472845876bfaf0 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 974b802141bd4f13beffef2aa03f88f9 (127.18.211.2:34517), 7a9daf5d42a84615a6383cfec9a10608 (127.18.211.4:40439)
I20260129 02:59:30.763684 20439 raft_consensus.cc:2468] T 4e3805ffa70242e8ade66da260dddf66 P 197b880fa39d432a9b472845876bfaf0 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1c50de89a5a845668696dcb28c3badd6 in term 2.
I20260129 02:59:30.733844 22059 leader_election.cc:290] T 3ccac82a01a2445fa5b0ce6c0fcf2295 P 7a9daf5d42a84615a6383cfec9a10608 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 197b880fa39d432a9b472845876bfaf0 (127.18.211.1:42027), 974b802141bd4f13beffef2aa03f88f9 (127.18.211.2:34517)
I20260129 02:59:30.760067 20584 rpcz_store.cc:276] 0129 02:59:28.780676 (+ 0us) service_pool.cc:168] Inserting onto call queue
0129 02:59:28.783661 (+ 2985us) service_pool.cc:225] Handling call
0129 02:59:28.830840 (+ 47179us) raft_consensus.cc:1471] Updating replica for 1 ops
0129 02:59:28.887676 (+ 56836us) raft_consensus.cc:1534] Early marking committed up to index 0
0129 02:59:28.887723 (+ 47us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0129 02:59:28.896998 (+ 9275us) log.cc:844] Serialized 26 byte log entry
0129 02:59:29.671430 (+774432us) raft_consensus.cc:1645] Marking committed up to 2
0129 02:59:29.671480 (+ 50us) raft_consensus.cc:1697] Filling consensus response to leader.
0129 02:59:29.671561 (+ 81us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0129 02:59:30.705678 (+1034117us) raft_consensus.cc:1685] finished
0129 02:59:30.705699 (+ 21us) raft_consensus.cc:1691] UpdateReplicas() finished
0129 02:59:30.705859 (+ 160us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":56387,"raft.queue_time_us":60499,"raft.run_cpu_time_us":179,"raft.run_wall_time_us":26857,"thread_start_us":693629,"threads_started":1,"wal-append.queue_time_us":785666}
I20260129 02:59:30.763854 20435 rpcz_store.cc:276] 0129 02:59:29.694653 (+ 0us) service_pool.cc:168] Inserting onto call queue
0129 02:59:30.762230 (+1067577us) service_pool.cc:225] Handling call
0129 02:59:30.763421 (+ 1191us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260129 02:59:30.828302 20435 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4f8f3a38f6e24ba4857875181f447442" candidate_uuid: "1c50de89a5a845668696dcb28c3badd6" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "197b880fa39d432a9b472845876bfaf0" is_pre_election: true
I20260129 02:59:30.828908 20514 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1299723a4476420ab02fd8bfd7aa0645" candidate_uuid: "7a9daf5d42a84615a6383cfec9a10608" candidate_term: 5 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "974b802141bd4f13beffef2aa03f88f9" is_pre_election: true
W20260129 02:59:30.767686 20554 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.211.4:40439, user_credentials={real_user=slave}} blocked reactor thread for 143214us
I20260129 02:59:30.829741 20514 raft_consensus.cc:2468] T 1299723a4476420ab02fd8bfd7aa0645 P 974b802141bd4f13beffef2aa03f88f9 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7a9daf5d42a84615a6383cfec9a10608 in term 4.
I20260129 02:59:30.831633 20554 leader_election.cc:304] T 4f5a77e94fef4b16a2ba7b067c8b144a P 1c50de89a5a845668696dcb28c3badd6 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1c50de89a5a845668696dcb28c3badd6, 7a9daf5d42a84615a6383cfec9a10608; no voters:
I20260129 02:59:30.829198 20435 raft_consensus.cc:2468] T 4f8f3a38f6e24ba4857875181f447442 P 197b880fa39d432a9b472845876bfaf0 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1c50de89a5a845668696dcb28c3badd6 in term 3.
I20260129 02:59:30.833221 20435 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55676 (request call id 1074) took 1138 ms. Trace:
W20260129 02:59:30.834144 20437 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55676 (request call id 1060) took 2511 ms (client timeout 1821 ms). Trace:
I20260129 02:59:30.774821 22033 leader_election.cc:290] T 00e79d8f25a34676b1d866eb2068a2b2 P 7a9daf5d42a84615a6383cfec9a10608 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 197b880fa39d432a9b472845876bfaf0 (127.18.211.1:42027), 974b802141bd4f13beffef2aa03f88f9 (127.18.211.2:34517)
I20260129 02:59:30.774914 22079 leader_election.cc:290] T b189f08e16a74a11980cc55d99359a0a P 1c50de89a5a845668696dcb28c3badd6 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 197b880fa39d432a9b472845876bfaf0 (127.18.211.1:42027), 974b802141bd4f13beffef2aa03f88f9 (127.18.211.2:34517)
I20260129 02:59:30.772318 20513 raft_consensus.cc:2468] T add8d4923d1a489e86882c3a25692b95 P 974b802141bd4f13beffef2aa03f88f9 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7a9daf5d42a84615a6383cfec9a10608 in term 1.
I20260129 02:59:30.751288 22126 raft_consensus.cc:515] T a4c1226b987b468fa84dfc6ba0e2dbe1 P 1c50de89a5a845668696dcb28c3badd6 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
W20260129 02:59:30.834626 20437 rpcz_store.cc:269] 0129 02:59:28.317729 (+ 0us) service_pool.cc:168] Inserting onto call queue
0129 02:59:28.704551 (+386822us) service_pool.cc:225] Handling call
0129 02:59:30.827371 (+2122820us) spinlock_profiling.cc:243] Waited 178 ms on lock 0x7b54003d1654. stack: 00007f8fe044e373 00007f8fe044e270 00007f8fdfda0574 00007f8fe9a5c6b4 00007f8fe9a5c649 00007f8fe34975a1 00007f8fe3531f7a 00007f8fe84e3647 00007f8fe1a73834 00007f8fe1a737c2 00007f8fe1a736f1 00007f8fe1a7366c 00007f8fe1a72922 00007f8fe186d34c 00007f8fe186c786 00007f8fe186c12f
0129 02:59:30.828538 (+ 1167us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":408758656}
I20260129 02:59:30.835696 20437 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0f20af7fbe5d4db89a6f17948d98912c" candidate_uuid: "1c50de89a5a845668696dcb28c3badd6" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "197b880fa39d432a9b472845876bfaf0" is_pre_election: true
I20260129 02:59:30.835805 20513 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3e2d15f8640b407195210c476e09df91" candidate_uuid: "7a9daf5d42a84615a6383cfec9a10608" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "974b802141bd4f13beffef2aa03f88f9" is_pre_election: true
I20260129 02:59:30.836303 20437 raft_consensus.cc:2468] T 0f20af7fbe5d4db89a6f17948d98912c P 197b880fa39d432a9b472845876bfaf0 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1c50de89a5a845668696dcb28c3badd6 in term 3.
I20260129 02:59:30.792090 20439 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55676 (request call id 1073) took 1097 ms. Trace:
I20260129 02:59:30.833968 20514 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "febffe4a8b394368b69b1d164308fb9a" candidate_uuid: "7a9daf5d42a84615a6383cfec9a10608" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "974b802141bd4f13beffef2aa03f88f9" is_pre_election: true
I20260129 02:59:30.836902 20437 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55676 (request call id 1078) took 1126 ms. Trace:
I20260129 02:59:30.836833 22126 leader_election.cc:290] T a4c1226b987b468fa84dfc6ba0e2dbe1 P 1c50de89a5a845668696dcb28c3badd6 [CANDIDATE]: Term 3 election: Requested vote from peers 197b880fa39d432a9b472845876bfaf0 (127.18.211.1:42027), 974b802141bd4f13beffef2aa03f88f9 (127.18.211.2:34517)
I20260129 02:59:30.759438 20434 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1299723a4476420ab02fd8bfd7aa0645" candidate_uuid: "7a9daf5d42a84615a6383cfec9a10608" candidate_term: 5 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "197b880fa39d432a9b472845876bfaf0" is_pre_election: true
I20260129 02:59:30.837436 20437 rpcz_store.cc:276] 0129 02:59:29.710304 (+ 0us) service_pool.cc:168] Inserting onto call queue
0129 02:59:30.835472 (+1125168us) service_pool.cc:225] Handling call
0129 02:59:30.836846 (+ 1374us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260129 02:59:30.837584 20514 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3ccac82a01a2445fa5b0ce6c0fcf2295" candidate_uuid: "7a9daf5d42a84615a6383cfec9a10608" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "974b802141bd4f13beffef2aa03f88f9" is_pre_election: true
I20260129 02:59:30.836848 20439 rpcz_store.cc:276] 0129 02:59:29.694975 (+ 0us) service_pool.cc:168] Inserting onto call queue
0129 02:59:30.760325 (+1065350us) service_pool.cc:225] Handling call
0129 02:59:30.792008 (+ 31683us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260129 02:59:30.838469 20437 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e4d18c35b0a7493e85d0a4b92956a48f" candidate_uuid: "1c50de89a5a845668696dcb28c3badd6" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "197b880fa39d432a9b472845876bfaf0" is_pre_election: true
I20260129 02:59:30.839015 20439 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "75be4bd6bd45474795fd472eff2953ac" candidate_uuid: "1c50de89a5a845668696dcb28c3badd6" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "197b880fa39d432a9b472845876bfaf0"
I20260129 02:59:30.839184 20514 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "75f45538dc704d9a8ec4ff8ad9f3ea5b" candidate_uuid: "7a9daf5d42a84615a6383cfec9a10608" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "974b802141bd4f13beffef2aa03f88f9" is_pre_election: true
I20260129 02:59:30.839128 20437 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55676 (request call id 1076) took 1143 ms. Trace:
I20260129 02:59:30.840553 20514 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2ac659d732314e1288849f08fe5e074e" candidate_uuid: "7a9daf5d42a84615a6383cfec9a10608" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "974b802141bd4f13beffef2aa03f88f9" is_pre_election: true
I20260129 02:59:30.841056 20434 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:48356 (request call id 204) took 1134 ms. Trace:
I20260129 02:59:30.840528 20437 rpcz_store.cc:276] 0129 02:59:29.695730 (+ 0us) service_pool.cc:168] Inserting onto call queue
0129 02:59:30.838247 (+1142517us) service_pool.cc:225] Handling call
0129 02:59:30.839081 (+ 834us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260129 02:59:30.841542 20434 rpcz_store.cc:276] 0129 02:59:29.706554 (+ 0us) service_pool.cc:168] Inserting onto call queue
0129 02:59:30.759140 (+1052586us) service_pool.cc:225] Handling call
0129 02:59:30.840974 (+ 81834us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260129 02:59:30.842128 20514 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "75b0b6e82ddc4f92b1fe1d94f9a0b73b" candidate_uuid: "7a9daf5d42a84615a6383cfec9a10608" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "974b802141bd4f13beffef2aa03f88f9" is_pre_election: true
I20260129 02:59:30.842512 20437 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "724a236524cb4e7ba3428880681d6cc2" candidate_uuid: "1c50de89a5a845668696dcb28c3badd6" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "197b880fa39d432a9b472845876bfaf0" is_pre_election: true
I20260129 02:59:30.842792 20434 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a4c1226b987b468fa84dfc6ba0e2dbe1" candidate_uuid: "974b802141bd4f13beffef2aa03f88f9" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "197b880fa39d432a9b472845876bfaf0" is_pre_election: true
I20260129 02:59:30.833654 20435 rpcz_store.cc:276] 0129 02:59:29.695243 (+ 0us) service_pool.cc:168] Inserting onto call queue
0129 02:59:30.827974 (+1132731us) service_pool.cc:225] Handling call
0129 02:59:30.833164 (+ 5190us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260129 02:59:30.844038 20435 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "febffe4a8b394368b69b1d164308fb9a" candidate_uuid: "7a9daf5d42a84615a6383cfec9a10608" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "197b880fa39d432a9b472845876bfaf0" is_pre_election: true
I20260129 02:59:30.844411 20434 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "42d27a28d89b409e8807ec0efd97af69" candidate_uuid: "7a9daf5d42a84615a6383cfec9a10608" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "197b880fa39d432a9b472845876bfaf0" is_pre_election: true
I20260129 02:59:30.844942 20435 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:48356 (request call id 210) took 1052 ms. Trace:
I20260129 02:59:30.845625 20434 raft_consensus.cc:2393] T 42d27a28d89b409e8807ec0efd97af69 P 197b880fa39d432a9b472845876bfaf0 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 7a9daf5d42a84615a6383cfec9a10608 in current term 4: Already voted for candidate 197b880fa39d432a9b472845876bfaf0 in this term.
I20260129 02:59:30.847878 20434 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:48356 (request call id 207) took 1140 ms. Trace:
W20260129 02:59:30.779373 21974 consensus_meta.cc:287] T a5a05fbcbbcf4916bbc01fd1a5b5e90d P 974b802141bd4f13beffef2aa03f88f9: Time spent flushing consensus metadata: real 1.346s user 0.005s sys 0.005s
I20260129 02:59:30.857581 21974 raft_consensus.cc:515] T a5a05fbcbbcf4916bbc01fd1a5b5e90d P 974b802141bd4f13beffef2aa03f88f9 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:59:30.859961 21883 raft_consensus.cc:493] T cb4727386a3e48aca5d71cc6c611c197 P 197b880fa39d432a9b472845876bfaf0 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 974b802141bd4f13beffef2aa03f88f9)
I20260129 02:59:30.860738 21883 raft_consensus.cc:515] T cb4727386a3e48aca5d71cc6c611c197 P 197b880fa39d432a9b472845876bfaf0 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:59:30.863415 21883 leader_election.cc:290] T cb4727386a3e48aca5d71cc6c611c197 P 197b880fa39d432a9b472845876bfaf0 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 974b802141bd4f13beffef2aa03f88f9 (127.18.211.2:34517), 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3:37735)
I20260129 02:59:30.865516 21131 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "18f30ebaa4174fd28c958e72d0cc7e24" candidate_uuid: "1c50de89a5a845668696dcb28c3badd6" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "7a9daf5d42a84615a6383cfec9a10608" is_pre_election: true
I20260129 02:59:30.876812 20480 leader_election.cc:304] T c934d31efd3f4e5195605ddc354d14e1 P 974b802141bd4f13beffef2aa03f88f9 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 974b802141bd4f13beffef2aa03f88f9; no voters: 197b880fa39d432a9b472845876bfaf0, 7a9daf5d42a84615a6383cfec9a10608
I20260129 02:59:30.877885 22160 raft_consensus.cc:2749] T c934d31efd3f4e5195605ddc354d14e1 P 974b802141bd4f13beffef2aa03f88f9 [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260129 02:59:30.878345 20480 leader_election.cc:304] T 96609ac68bdf4993a12e1c23355ca013 P 974b802141bd4f13beffef2aa03f88f9 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 4 responses out of 4 voters: 3 yes votes; 1 no votes. yes voters: 197b880fa39d432a9b472845876bfaf0, 7a9daf5d42a84615a6383cfec9a10608, 974b802141bd4f13beffef2aa03f88f9; no voters: 1c50de89a5a845668696dcb28c3badd6
I20260129 02:59:30.879287 22160 raft_consensus.cc:2804] T 96609ac68bdf4993a12e1c23355ca013 P 974b802141bd4f13beffef2aa03f88f9 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260129 02:59:30.879648 22160 raft_consensus.cc:493] T 96609ac68bdf4993a12e1c23355ca013 P 974b802141bd4f13beffef2aa03f88f9 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 197b880fa39d432a9b472845876bfaf0)
I20260129 02:59:30.880038 22160 raft_consensus.cc:3060] T 96609ac68bdf4993a12e1c23355ca013 P 974b802141bd4f13beffef2aa03f88f9 [term 2 FOLLOWER]: Advancing to term 3
I20260129 02:59:30.880671 21136 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4e420f90884f4796997b5ea41e49e411" candidate_uuid: "197b880fa39d432a9b472845876bfaf0" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "7a9daf5d42a84615a6383cfec9a10608" is_pre_election: true
I20260129 02:59:30.888767 22160 raft_consensus.cc:515] T 96609ac68bdf4993a12e1c23355ca013 P 974b802141bd4f13beffef2aa03f88f9 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } attrs { replace: true } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } }
I20260129 02:59:30.891752 22160 leader_election.cc:290] T 96609ac68bdf4993a12e1c23355ca013 P 974b802141bd4f13beffef2aa03f88f9 [CANDIDATE]: Term 3 election: Requested vote from peers 197b880fa39d432a9b472845876bfaf0 (127.18.211.1:42027), 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3:37735), 7a9daf5d42a84615a6383cfec9a10608 (127.18.211.4:40439)
I20260129 02:59:30.786501 22082 raft_consensus.cc:493] T 17d6431cc77a491e8464d68db4081a51 P 197b880fa39d432a9b472845876bfaf0 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 974b802141bd4f13beffef2aa03f88f9)
I20260129 02:59:30.836366 20513 raft_consensus.cc:2468] T 3e2d15f8640b407195210c476e09df91 P 974b802141bd4f13beffef2aa03f88f9 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7a9daf5d42a84615a6383cfec9a10608 in term 1.
I20260129 02:59:30.843132 20437 raft_consensus.cc:2468] T 724a236524cb4e7ba3428880681d6cc2 P 197b880fa39d432a9b472845876bfaf0 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1c50de89a5a845668696dcb28c3badd6 in term 1.
I20260129 02:59:30.847352 20435 rpcz_store.cc:276] 0129 02:59:29.793083 (+ 0us) service_pool.cc:168] Inserting onto call queue
0129 02:59:30.843739 (+1050656us) service_pool.cc:225] Handling call
0129 02:59:30.844877 (+ 1138us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260129 02:59:30.900997 20437 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55676 (request call id 1077) took 1205 ms. Trace:
I20260129 02:59:30.842741 20514 raft_consensus.cc:2468] T 75b0b6e82ddc4f92b1fe1d94f9a0b73b P 974b802141bd4f13beffef2aa03f88f9 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7a9daf5d42a84615a6383cfec9a10608 in term 1.
I20260129 02:59:30.901333 20435 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7d9494f5584d4d5db8b9adf1c8fc888b" candidate_uuid: "974b802141bd4f13beffef2aa03f88f9" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "197b880fa39d432a9b472845876bfaf0" is_pre_election: true
I20260129 02:59:30.826655 22104 raft_consensus.cc:493] T 96609ac68bdf4993a12e1c23355ca013 P 197b880fa39d432a9b472845876bfaf0 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260129 02:59:30.902103 20435 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55672 (request call id 1380) took 1100 ms. Trace:
I20260129 02:59:30.827158 22120 raft_consensus.cc:493] T 926a250fa5834e88b96ccecbefbf5086 P 197b880fa39d432a9b472845876bfaf0 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 1c50de89a5a845668696dcb28c3badd6)
W20260129 02:59:30.856926 20556 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.211.2:34517, user_credentials={real_user=slave}} blocked reactor thread for 225978us
I20260129 02:59:30.901358 20437 rpcz_store.cc:276] 0129 02:59:29.696016 (+ 0us) service_pool.cc:168] Inserting onto call queue
0129 02:59:30.842281 (+1146265us) service_pool.cc:225] Handling call
0129 02:59:30.900940 (+ 58659us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260129 02:59:30.906014 20437 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "dc2f4cb752cf4d59b5f6bc689f5d8104" candidate_uuid: "1c50de89a5a845668696dcb28c3badd6" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "197b880fa39d432a9b472845876bfaf0" is_pre_election: true
I20260129 02:59:30.901937 20514 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4f5a77e94fef4b16a2ba7b067c8b144a" candidate_uuid: "7a9daf5d42a84615a6383cfec9a10608" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "974b802141bd4f13beffef2aa03f88f9" is_pre_election: true
I20260129 02:59:30.907318 20514 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "96609ac68bdf4993a12e1c23355ca013" candidate_uuid: "7a9daf5d42a84615a6383cfec9a10608" candidate_term: 4 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "974b802141bd4f13beffef2aa03f88f9" is_pre_election: true
I20260129 02:59:30.902479 20435 rpcz_store.cc:276] 0129 02:59:29.802228 (+ 0us) service_pool.cc:168] Inserting onto call queue
0129 02:59:30.901081 (+1098853us) service_pool.cc:225] Handling call
0129 02:59:30.902048 (+ 967us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260129 02:59:30.908627 20435 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2ac659d732314e1288849f08fe5e074e" candidate_uuid: "7a9daf5d42a84615a6383cfec9a10608" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "197b880fa39d432a9b472845876bfaf0" is_pre_election: true
I20260129 02:59:30.909322 20435 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:48356 (request call id 208) took 1187 ms. Trace:
I20260129 02:59:30.909672 20435 rpcz_store.cc:276] 0129 02:59:29.726170 (+ 0us) service_pool.cc:168] Inserting onto call queue
0129 02:59:30.908399 (+1182229us) service_pool.cc:225] Handling call
0129 02:59:30.909274 (+ 875us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260129 02:59:30.910597 20435 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c2633a6fc34c44579de348d208b0c01e" candidate_uuid: "1c50de89a5a845668696dcb28c3badd6" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "197b880fa39d432a9b472845876bfaf0" is_pre_election: true
I20260129 02:59:30.918524 20407 leader_election.cc:304] T 0f20af7fbe5d4db89a6f17948d98912c P 197b880fa39d432a9b472845876bfaf0 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 197b880fa39d432a9b472845876bfaf0; no voters: 1c50de89a5a845668696dcb28c3badd6, 974b802141bd4f13beffef2aa03f88f9
I20260129 02:59:30.849754 22213 raft_consensus.cc:2804] T 4f5a77e94fef4b16a2ba7b067c8b144a P 1c50de89a5a845668696dcb28c3badd6 [term 1 FOLLOWER]: Leader pre-election won for term 2
W20260129 02:59:30.927208 20480 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.211.4:40439, user_credentials={real_user=slave}} blocked reactor thread for 48940.8us
I20260129 02:59:30.929795 20437 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55676 (request call id 1070) took 1256 ms. Trace:
I20260129 02:59:30.930171 20437 rpcz_store.cc:276] 0129 02:59:29.673849 (+ 0us) service_pool.cc:168] Inserting onto call queue
0129 02:59:30.905777 (+1231928us) service_pool.cc:225] Handling call
0129 02:59:30.929727 (+ 23950us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260129 02:59:30.930991 20437 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "96609ac68bdf4993a12e1c23355ca013" candidate_uuid: "7a9daf5d42a84615a6383cfec9a10608" candidate_term: 4 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "197b880fa39d432a9b472845876bfaf0" is_pre_election: true
I20260129 02:59:30.921916 20435 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55676 (request call id 1068) took 1248 ms. Trace:
I20260129 02:59:30.925360 20513 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "505580c93c694bbb8ea96fdf9b17ab46" candidate_uuid: "7a9daf5d42a84615a6383cfec9a10608" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "974b802141bd4f13beffef2aa03f88f9" is_pre_election: true
I20260129 02:59:30.932857 20435 rpcz_store.cc:276] 0129 02:59:29.673240 (+ 0us) service_pool.cc:168] Inserting onto call queue
0129 02:59:30.910367 (+1237127us) service_pool.cc:225] Handling call
0129 02:59:30.921849 (+ 11482us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260129 02:59:30.934124 20435 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4e420f90884f4796997b5ea41e49e411" candidate_uuid: "974b802141bd4f13beffef2aa03f88f9" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "197b880fa39d432a9b472845876bfaf0" is_pre_election: true
I20260129 02:59:30.907946 20514 raft_consensus.cc:2468] T 96609ac68bdf4993a12e1c23355ca013 P 974b802141bd4f13beffef2aa03f88f9 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7a9daf5d42a84615a6383cfec9a10608 in term 3.
I20260129 02:59:30.938925 20514 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "cb0b3f259ebc4310927338a8740e87b8" candidate_uuid: "7a9daf5d42a84615a6383cfec9a10608" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "974b802141bd4f13beffef2aa03f88f9" is_pre_election: true
I20260129 02:59:30.859752 21974 leader_election.cc:290] T a5a05fbcbbcf4916bbc01fd1a5b5e90d P 974b802141bd4f13beffef2aa03f88f9 [CANDIDATE]: Term 5 election: Requested vote from peers 197b880fa39d432a9b472845876bfaf0 (127.18.211.1:42027), 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3:37735)
I20260129 02:59:30.948347 21136 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fd8311b779af4e1fa1402b2a02a602f2" candidate_uuid: "1c50de89a5a845668696dcb28c3badd6" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "7a9daf5d42a84615a6383cfec9a10608" is_pre_election: true
I20260129 02:59:30.952623 20514 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c2633a6fc34c44579de348d208b0c01e" candidate_uuid: "1c50de89a5a845668696dcb28c3badd6" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "974b802141bd4f13beffef2aa03f88f9" is_pre_election: true
I20260129 02:59:30.956558 20514 raft_consensus.cc:2468] T c2633a6fc34c44579de348d208b0c01e P 974b802141bd4f13beffef2aa03f88f9 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1c50de89a5a845668696dcb28c3badd6 in term 1.
I20260129 02:59:30.957687 20514 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "368953c9102643b2a21f99af36b51905" candidate_uuid: "1c50de89a5a845668696dcb28c3badd6" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "974b802141bd4f13beffef2aa03f88f9"
I20260129 02:59:30.959096 20514 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "96609ac68bdf4993a12e1c23355ca013" candidate_uuid: "1c50de89a5a845668696dcb28c3badd6" candidate_term: 4 candidate_status { last_received { term: 2 index: 3 } } ignore_live_leader: false dest_uuid: "974b802141bd4f13beffef2aa03f88f9" is_pre_election: true
I20260129 02:59:30.960086 20514 raft_consensus.cc:2410] T 96609ac68bdf4993a12e1c23355ca013 P 974b802141bd4f13beffef2aa03f88f9 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1c50de89a5a845668696dcb28c3badd6 for term 4 because replica has last-logged OpId of term: 2 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 3.
I20260129 02:59:30.950537 20435 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "8055232cf0004d40a85f3bf983fe00a9" candidate_uuid: "1c50de89a5a845668696dcb28c3badd6" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "197b880fa39d432a9b472845876bfaf0" is_pre_election: true
I20260129 02:59:30.971937 20435 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55676 (request call id 1075) took 1276 ms. Trace:
I20260129 02:59:30.972332 20435 rpcz_store.cc:276] 0129 02:59:29.695485 (+ 0us) service_pool.cc:168] Inserting onto call queue
0129 02:59:30.950206 (+1254721us) service_pool.cc:225] Handling call
0129 02:59:30.971841 (+ 21635us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260129 02:59:30.975526 20513 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "505580c93c694bbb8ea96fdf9b17ab46" candidate_uuid: "1c50de89a5a845668696dcb28c3badd6" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "974b802141bd4f13beffef2aa03f88f9" is_pre_election: true
I20260129 02:59:30.978402 20435 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "368953c9102643b2a21f99af36b51905" candidate_uuid: "1c50de89a5a845668696dcb28c3badd6" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "197b880fa39d432a9b472845876bfaf0"
I20260129 02:59:30.945276 22160 raft_consensus.cc:493] T 7a9a69cd82724b1faed1d10aa091fc4a P 974b802141bd4f13beffef2aa03f88f9 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 197b880fa39d432a9b472845876bfaf0)
I20260129 02:59:30.980715 22114 raft_consensus.cc:515] T 3906b080a5b34a74b8b64128635e8997 P 1c50de89a5a845668696dcb28c3badd6 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:59:30.980860 22160 raft_consensus.cc:515] T 7a9a69cd82724b1faed1d10aa091fc4a P 974b802141bd4f13beffef2aa03f88f9 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:59:30.982487 22227 raft_consensus.cc:493] T 0340229d521b432580ae926f6062e3f7 P 1c50de89a5a845668696dcb28c3badd6 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 197b880fa39d432a9b472845876bfaf0)
I20260129 02:59:30.982942 22114 leader_election.cc:290] T 3906b080a5b34a74b8b64128635e8997 P 1c50de89a5a845668696dcb28c3badd6 [CANDIDATE]: Term 2 election: Requested vote from peers 197b880fa39d432a9b472845876bfaf0 (127.18.211.1:42027), 974b802141bd4f13beffef2aa03f88f9 (127.18.211.2:34517)
I20260129 02:59:30.983260 20516 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57600 (request call id 1612) took 2134 ms. Trace:
I20260129 02:59:30.983824 20516 rpcz_store.cc:276] 0129 02:59:28.848597 (+ 0us) service_pool.cc:168] Inserting onto call queue
0129 02:59:29.325406 (+476809us) service_pool.cc:225] Handling call
0129 02:59:29.326437 (+ 1031us) raft_consensus.cc:1471] Updating replica for 0 ops
0129 02:59:29.484012 (+157575us) raft_consensus.cc:1534] Early marking committed up to index 0
0129 02:59:29.484044 (+ 32us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0129 02:59:29.484080 (+ 36us) raft_consensus.cc:1645] Marking committed up to 1
0129 02:59:29.484109 (+ 29us) raft_consensus.cc:1697] Filling consensus response to leader.
0129 02:59:30.983010 (+1498901us) spinlock_profiling.cc:243] Waited 157 ms on lock 0x7b540026f054. stack: 00007f8fe044e373 00007f8fe044e270 00007f8fdfda0574 00007f8fe9a5c6b4 00007f8fe9a5c649 00007f8fe34975a1 00007f8fe352ea81 00007f8fe352d4be 00007f8fe84e1f49 00007f8fe1a6ed34 00007f8fe1a6ecc2 00007f8fe1a6ebf1 00007f8fe1a6eb6c 00007f8fe1a6de22 00007f8fe186d34c 00007f8fe186c786
0129 02:59:30.983086 (+ 76us) raft_consensus.cc:1691] UpdateReplicas() finished
0129 02:59:30.983202 (+ 116us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":361942784}
I20260129 02:59:30.984757 20553 leader_election.cc:304] T 0f20af7fbe5d4db89a6f17948d98912c P 1c50de89a5a845668696dcb28c3badd6 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 197b880fa39d432a9b472845876bfaf0, 1c50de89a5a845668696dcb28c3badd6; no voters:
I20260129 02:59:30.985096 22135 raft_consensus.cc:993] T 36ff33be666a4c82954731691dbcf327 P 974b802141bd4f13beffef2aa03f88f9: : Instructing follower 1c50de89a5a845668696dcb28c3badd6 to start an election
I20260129 02:59:30.985230 20516 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4408006eb5174de1896b037cf3f18457" candidate_uuid: "1c50de89a5a845668696dcb28c3badd6" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "974b802141bd4f13beffef2aa03f88f9" is_pre_election: true
I20260129 02:59:30.985611 22135 raft_consensus.cc:1081] T 36ff33be666a4c82954731691dbcf327 P 974b802141bd4f13beffef2aa03f88f9 [term 1 LEADER]: Signalling peer 1c50de89a5a845668696dcb28c3badd6 to start an election
I20260129 02:59:30.985723 20553 leader_election.cc:304] T 4e3805ffa70242e8ade66da260dddf66 P 1c50de89a5a845668696dcb28c3badd6 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 197b880fa39d432a9b472845876bfaf0, 1c50de89a5a845668696dcb28c3badd6; no voters: 7a9daf5d42a84615a6383cfec9a10608
I20260129 02:59:30.986351 20516 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0f20af7fbe5d4db89a6f17948d98912c" candidate_uuid: "1c50de89a5a845668696dcb28c3badd6" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "974b802141bd4f13beffef2aa03f88f9" is_pre_election: true
I20260129 02:59:30.974467 20514 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "cb0b3f259ebc4310927338a8740e87b8" candidate_uuid: "1c50de89a5a845668696dcb28c3badd6" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "974b802141bd4f13beffef2aa03f88f9" is_pre_election: true
I20260129 02:59:30.983086 22227 raft_consensus.cc:515] T 0340229d521b432580ae926f6062e3f7 P 1c50de89a5a845668696dcb28c3badd6 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:59:30.987177 20553 leader_election.cc:304] T 4f8f3a38f6e24ba4857875181f447442 P 1c50de89a5a845668696dcb28c3badd6 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 197b880fa39d432a9b472845876bfaf0, 1c50de89a5a845668696dcb28c3badd6; no voters:
I20260129 02:59:30.987373 20516 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "75be4bd6bd45474795fd472eff2953ac" candidate_uuid: "1c50de89a5a845668696dcb28c3badd6" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "974b802141bd4f13beffef2aa03f88f9"
I20260129 02:59:30.979911 20480 leader_election.cc:304] T 75b0b6e82ddc4f92b1fe1d94f9a0b73b P 974b802141bd4f13beffef2aa03f88f9 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 974b802141bd4f13beffef2aa03f88f9; no voters: 1c50de89a5a845668696dcb28c3badd6, 7a9daf5d42a84615a6383cfec9a10608
I20260129 02:59:30.983345 22160 leader_election.cc:290] T 7a9a69cd82724b1faed1d10aa091fc4a P 974b802141bd4f13beffef2aa03f88f9 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 197b880fa39d432a9b472845876bfaf0 (127.18.211.1:42027), 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3:37735)
I20260129 02:59:30.926611 22190 raft_consensus.cc:2749] T 0f20af7fbe5d4db89a6f17948d98912c P 197b880fa39d432a9b472845876bfaf0 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260129 02:59:30.989637 20435 raft_consensus.cc:3060] T 368953c9102643b2a21f99af36b51905 P 197b880fa39d432a9b472845876bfaf0 [term 3 FOLLOWER]: Advancing to term 4
I20260129 02:59:30.989415 20480 leader_election.cc:304] T a0fcc58d6fb6473cafbd9d34d5195269 P 974b802141bd4f13beffef2aa03f88f9 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 974b802141bd4f13beffef2aa03f88f9; no voters: 197b880fa39d432a9b472845876bfaf0, 1c50de89a5a845668696dcb28c3badd6
I20260129 02:59:30.852195 20434 rpcz_store.cc:276] 0129 02:59:29.707281 (+ 0us) service_pool.cc:168] Inserting onto call queue
0129 02:59:30.844085 (+1136804us) service_pool.cc:225] Handling call
0129 02:59:30.847777 (+ 3692us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260129 02:59:30.990233 20553 leader_election.cc:304] T 724a236524cb4e7ba3428880681d6cc2 P 1c50de89a5a845668696dcb28c3badd6 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 197b880fa39d432a9b472845876bfaf0, 1c50de89a5a845668696dcb28c3badd6; no voters:
I20260129 02:59:30.926102 22213 raft_consensus.cc:493] T 4f5a77e94fef4b16a2ba7b067c8b144a P 1c50de89a5a845668696dcb28c3badd6 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 974b802141bd4f13beffef2aa03f88f9)
I20260129 02:59:30.953119 20433 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:48356 (request call id 203) took 2098 ms. Trace:
I20260129 02:59:30.906601 22082 raft_consensus.cc:515] T 17d6431cc77a491e8464d68db4081a51 P 197b880fa39d432a9b472845876bfaf0 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:59:30.993134 21994 raft_consensus.cc:2749] T 75b0b6e82ddc4f92b1fe1d94f9a0b73b P 974b802141bd4f13beffef2aa03f88f9 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260129 02:59:30.935504 22116 leader_election.cc:290] T fd8311b779af4e1fa1402b2a02a602f2 P 1c50de89a5a845668696dcb28c3badd6 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 197b880fa39d432a9b472845876bfaf0 (127.18.211.1:42027), 7a9daf5d42a84615a6383cfec9a10608 (127.18.211.4:40439)
I20260129 02:59:31.034267 20516 raft_consensus.cc:3060] T 75be4bd6bd45474795fd472eff2953ac P 974b802141bd4f13beffef2aa03f88f9 [term 6 FOLLOWER]: Advancing to term 7
I20260129 02:59:30.976929 22102 raft_consensus.cc:493] T a0fcc58d6fb6473cafbd9d34d5195269 P 1c50de89a5a845668696dcb28c3badd6 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 197b880fa39d432a9b472845876bfaf0)
I20260129 02:59:30.925350 22143 consensus_queue.cc:1048] T a4c1226b987b468fa84dfc6ba0e2dbe1 P 197b880fa39d432a9b472845876bfaf0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.070s
I20260129 02:59:30.921967 21100 leader_election.cc:304] T 75f45538dc704d9a8ec4ff8ad9f3ea5b P 7a9daf5d42a84615a6383cfec9a10608 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1c50de89a5a845668696dcb28c3badd6, 7a9daf5d42a84615a6383cfec9a10608; no voters:
I20260129 02:59:30.929281 22120 raft_consensus.cc:515] T 926a250fa5834e88b96ccecbefbf5086 P 197b880fa39d432a9b472845876bfaf0 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
W20260129 02:59:31.052234 21988 consensus_meta.cc:287] T 1ce4ddf5b10e4ae8b193b13b2e5936ee P 1c50de89a5a845668696dcb28c3badd6: Time spent flushing consensus metadata: real 1.663s user 0.014s sys 0.000s
I20260129 02:59:31.153076 22213 raft_consensus.cc:3060] T 4f5a77e94fef4b16a2ba7b067c8b144a P 1c50de89a5a845668696dcb28c3badd6 [term 1 FOLLOWER]: Advancing to term 2
W20260129 02:59:31.193351 22221 log.cc:927] Time spent T 1299723a4476420ab02fd8bfd7aa0645 P 7a9daf5d42a84615a6383cfec9a10608: Append to log took a long time: real 0.241s user 0.000s sys 0.000s
W20260129 02:59:31.199333 20407 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.211.2:34517, user_credentials={real_user=slave}} blocked reactor thread for 252624us
I20260129 02:59:31.200203 20516 raft_consensus.cc:2468] T 75be4bd6bd45474795fd472eff2953ac P 974b802141bd4f13beffef2aa03f88f9 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1c50de89a5a845668696dcb28c3badd6 in term 7.
I20260129 02:59:30.925576 22104 raft_consensus.cc:515] T 96609ac68bdf4993a12e1c23355ca013 P 197b880fa39d432a9b472845876bfaf0 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } attrs { replace: true } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } }
I20260129 02:59:31.310026 20511 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "724a236524cb4e7ba3428880681d6cc2" candidate_uuid: "1c50de89a5a845668696dcb28c3badd6" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "974b802141bd4f13beffef2aa03f88f9" is_pre_election: true
W20260129 02:59:31.310874 20480 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.211.3:37735, user_credentials={real_user=slave}} blocked reactor thread for 321526us
I20260129 02:59:31.310433 20514 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e4d18c35b0a7493e85d0a4b92956a48f" candidate_uuid: "1c50de89a5a845668696dcb28c3badd6" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "974b802141bd4f13beffef2aa03f88f9" is_pre_election: true
I20260129 02:59:31.110010 22072 raft_consensus.cc:2749] T a0fcc58d6fb6473cafbd9d34d5195269 P 974b802141bd4f13beffef2aa03f88f9 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260129 02:59:31.152715 22190 raft_consensus.cc:493] T 2ac659d732314e1288849f08fe5e074e P 197b880fa39d432a9b472845876bfaf0 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 974b802141bd4f13beffef2aa03f88f9)
I20260129 02:59:31.337935 20516 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b189f08e16a74a11980cc55d99359a0a" candidate_uuid: "1c50de89a5a845668696dcb28c3badd6" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "974b802141bd4f13beffef2aa03f88f9" is_pre_election: true
I20260129 02:59:31.334113 20407 leader_election.cc:304] T c934d31efd3f4e5195605ddc354d14e1 P 197b880fa39d432a9b472845876bfaf0 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 197b880fa39d432a9b472845876bfaf0, 974b802141bd4f13beffef2aa03f88f9; no voters: 7a9daf5d42a84615a6383cfec9a10608
I20260129 02:59:31.334563 20513 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4f8f3a38f6e24ba4857875181f447442" candidate_uuid: "1c50de89a5a845668696dcb28c3badd6" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "974b802141bd4f13beffef2aa03f88f9" is_pre_election: true
I20260129 02:59:31.267041 22198 consensus_queue.cc:1048] T a4c1226b987b468fa84dfc6ba0e2dbe1 P 197b880fa39d432a9b472845876bfaf0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.069s
I20260129 02:59:31.348223 20407 leader_election.cc:304] T 76f713c3331540d2a5b76164103e9eb1 P 197b880fa39d432a9b472845876bfaf0 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 197b880fa39d432a9b472845876bfaf0, 974b802141bd4f13beffef2aa03f88f9; no voters:
I20260129 02:59:31.157671 22079 raft_consensus.cc:2804] T 0f20af7fbe5d4db89a6f17948d98912c P 1c50de89a5a845668696dcb28c3badd6 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260129 02:59:31.184648 21942 raft_consensus.cc:493] T 0678b8b468bd43cdbc3e485900844821 P 197b880fa39d432a9b472845876bfaf0 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 7a9daf5d42a84615a6383cfec9a10608)
I20260129 02:59:31.340322 20511 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4f5a77e94fef4b16a2ba7b067c8b144a" candidate_uuid: "1c50de89a5a845668696dcb28c3badd6" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "974b802141bd4f13beffef2aa03f88f9" is_pre_election: true
I20260129 02:59:31.339461 20516 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7a9a69cd82724b1faed1d10aa091fc4a" candidate_uuid: "1c50de89a5a845668696dcb28c3badd6" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "974b802141bd4f13beffef2aa03f88f9" is_pre_election: true
I20260129 02:59:31.419761 20516 raft_consensus.cc:2468] T 7a9a69cd82724b1faed1d10aa091fc4a P 974b802141bd4f13beffef2aa03f88f9 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1c50de89a5a845668696dcb28c3badd6 in term 3.
I20260129 02:59:31.420774 20516 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "17d6431cc77a491e8464d68db4081a51" candidate_uuid: "1c50de89a5a845668696dcb28c3badd6" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "974b802141bd4f13beffef2aa03f88f9" is_pre_election: true
I20260129 02:59:31.421597 20583 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "18f30ebaa4174fd28c958e72d0cc7e24" candidate_uuid: "197b880fa39d432a9b472845876bfaf0" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "1c50de89a5a845668696dcb28c3badd6" is_pre_election: true
W20260129 02:59:31.414968 20554 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.211.4:40439, user_credentials={real_user=slave}} blocked reactor thread for 95342.8us
I20260129 02:59:31.337440 20480 leader_election.cc:304] T fcddab6bca684adf8ea032289a00f90e P 974b802141bd4f13beffef2aa03f88f9 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 974b802141bd4f13beffef2aa03f88f9; no voters: 197b880fa39d432a9b472845876bfaf0, 1c50de89a5a845668696dcb28c3badd6
I20260129 02:59:31.423043 20583 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4f8f3a38f6e24ba4857875181f447442" candidate_uuid: "197b880fa39d432a9b472845876bfaf0" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "1c50de89a5a845668696dcb28c3badd6" is_pre_election: true
I20260129 02:59:31.310587 22227 leader_election.cc:290] T 0340229d521b432580ae926f6062e3f7 P 1c50de89a5a845668696dcb28c3badd6 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 197b880fa39d432a9b472845876bfaf0 (127.18.211.1:42027), 974b802141bd4f13beffef2aa03f88f9 (127.18.211.2:34517)
I20260129 02:59:31.218722 22082 leader_election.cc:290] T 17d6431cc77a491e8464d68db4081a51 P 197b880fa39d432a9b472845876bfaf0 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 974b802141bd4f13beffef2aa03f88f9 (127.18.211.2:34517), 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3:37735)
I20260129 02:59:31.216631 20434 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e7d83d4ccf684e76acb29a0dc844e1b8" candidate_uuid: "974b802141bd4f13beffef2aa03f88f9" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "197b880fa39d432a9b472845876bfaf0" is_pre_election: true
I20260129 02:59:31.320506 21991 raft_consensus.cc:2804] T 4e3805ffa70242e8ade66da260dddf66 P 1c50de89a5a845668696dcb28c3badd6 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260129 02:59:31.332187 22090 raft_consensus.cc:493] T 36ff33be666a4c82954731691dbcf327 P 1c50de89a5a845668696dcb28c3badd6 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 974b802141bd4f13beffef2aa03f88f9)
W20260129 02:59:31.491886 20480 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.211.3:37735, user_credentials={real_user=slave}} blocked reactor thread for 154505us
I20260129 02:59:31.271531 22102 raft_consensus.cc:515] T a0fcc58d6fb6473cafbd9d34d5195269 P 1c50de89a5a845668696dcb28c3badd6 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:59:31.503158 20480 leader_election.cc:304] T 75be4bd6bd45474795fd472eff2953ac P 974b802141bd4f13beffef2aa03f88f9 [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 974b802141bd4f13beffef2aa03f88f9; no voters: 197b880fa39d432a9b472845876bfaf0, 1c50de89a5a845668696dcb28c3badd6
I20260129 02:59:31.504787 20480 leader_election.cc:304] T 1ce4ddf5b10e4ae8b193b13b2e5936ee P 974b802141bd4f13beffef2aa03f88f9 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 974b802141bd4f13beffef2aa03f88f9; no voters: 197b880fa39d432a9b472845876bfaf0, 1c50de89a5a845668696dcb28c3badd6
I20260129 02:59:31.492030 20514 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a4c1226b987b468fa84dfc6ba0e2dbe1" candidate_uuid: "1c50de89a5a845668696dcb28c3badd6" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "974b802141bd4f13beffef2aa03f88f9"
I20260129 02:59:31.332592 22126 raft_consensus.cc:2804] T 4f8f3a38f6e24ba4857875181f447442 P 1c50de89a5a845668696dcb28c3badd6 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260129 02:59:30.644737 20509 rpcz_store.cc:276] 0129 02:59:28.021659 (+ 0us) service_pool.cc:168] Inserting onto call queue
0129 02:59:28.024045 (+ 2386us) service_pool.cc:225] Handling call
0129 02:59:28.026279 (+ 2234us) raft_consensus.cc:1471] Updating replica for 0 ops
0129 02:59:28.028012 (+ 1733us) raft_consensus.cc:1534] Early marking committed up to index 5
0129 02:59:28.834315 (+806303us) log.cc:844] Serialized 14 byte log entry
0129 02:59:30.154669 (+1320354us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0129 02:59:30.228251 (+ 73582us) raft_consensus.cc:1645] Marking committed up to 5
0129 02:59:30.234589 (+ 6338us) raft_consensus.cc:1697] Filling consensus response to leader.
0129 02:59:30.357157 (+122568us) raft_consensus.cc:1691] UpdateReplicas() finished
0129 02:59:30.573990 (+216833us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":742817,"prepare.run_cpu_time_us":42,"prepare.run_wall_time_us":43,"raft.queue_time_us":12013,"raft.run_cpu_time_us":157,"raft.run_wall_time_us":8655,"spinlock_wait_cycles":228288512,"thread_start_us":2055514,"threads_started":2,"wal-append.queue_time_us":1127535}
I20260129 02:59:31.506430 20514 raft_consensus.cc:3060] T a4c1226b987b468fa84dfc6ba0e2dbe1 P 974b802141bd4f13beffef2aa03f88f9 [term 2 FOLLOWER]: Advancing to term 3
I20260129 02:59:31.434072 20437 raft_consensus.cc:2468] T 96609ac68bdf4993a12e1c23355ca013 P 197b880fa39d432a9b472845876bfaf0 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7a9daf5d42a84615a6383cfec9a10608 in term 3.
I20260129 02:59:31.219728 22195 raft_consensus.cc:515] T 75be4bd6bd45474795fd472eff2953ac P 197b880fa39d432a9b472845876bfaf0 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:59:31.415475 22079 raft_consensus.cc:493] T 0f20af7fbe5d4db89a6f17948d98912c P 1c50de89a5a845668696dcb28c3badd6 [term 3 FOLLOWER]: Starting leader election (detected failure of leader 974b802141bd4f13beffef2aa03f88f9)
I20260129 02:59:31.321966 21988 raft_consensus.cc:515] T 1ce4ddf5b10e4ae8b193b13b2e5936ee P 1c50de89a5a845668696dcb28c3badd6 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:59:31.418112 22222 raft_consensus.cc:2804] T 75f45538dc704d9a8ec4ff8ad9f3ea5b P 7a9daf5d42a84615a6383cfec9a10608 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260129 02:59:31.529862 20588 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c891d2bff4ce4568861bbc992e91fbb0" candidate_uuid: "197b880fa39d432a9b472845876bfaf0" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "1c50de89a5a845668696dcb28c3badd6" is_pre_election: true
I20260129 02:59:31.530534 20588 raft_consensus.cc:2468] T c891d2bff4ce4568861bbc992e91fbb0 P 1c50de89a5a845668696dcb28c3badd6 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 197b880fa39d432a9b472845876bfaf0 in term 6.
I20260129 02:59:31.530814 20511 raft_consensus.cc:1240] T a5a05fbcbbcf4916bbc01fd1a5b5e90d P 974b802141bd4f13beffef2aa03f88f9 [term 5 FOLLOWER]: Rejecting Update request from peer 1c50de89a5a845668696dcb28c3badd6 for earlier term 4. Current term is 5. Ops: []
W20260129 02:59:31.356863 21100 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.211.3:37735, user_credentials={real_user=slave}} blocked reactor thread for 458647us
I20260129 02:59:31.526803 20589 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "76f713c3331540d2a5b76164103e9eb1" candidate_uuid: "197b880fa39d432a9b472845876bfaf0" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "1c50de89a5a845668696dcb28c3badd6"
I20260129 02:59:31.535923 20585 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e4d18c35b0a7493e85d0a4b92956a48f" candidate_uuid: "197b880fa39d432a9b472845876bfaf0" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "1c50de89a5a845668696dcb28c3badd6" is_pre_election: true
I20260129 02:59:31.493414 22131 leader_election.cc:290] T 505580c93c694bbb8ea96fdf9b17ab46 P 1c50de89a5a845668696dcb28c3badd6 [CANDIDATE]: Term 3 election: Requested vote from peers 974b802141bd4f13beffef2aa03f88f9 (127.18.211.2:34517), 7a9daf5d42a84615a6383cfec9a10608 (127.18.211.4:40439)
I20260129 02:59:31.525365 20586 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2e58e9a5a8aa4cecafb464addb4f35ed" candidate_uuid: "197b880fa39d432a9b472845876bfaf0" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "1c50de89a5a845668696dcb28c3badd6" is_pre_election: true
I20260129 02:59:31.435704 22164 raft_consensus.cc:2804] T 724a236524cb4e7ba3428880681d6cc2 P 1c50de89a5a845668696dcb28c3badd6 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260129 02:59:31.504335 21974 raft_consensus.cc:2749] T fcddab6bca684adf8ea032289a00f90e P 974b802141bd4f13beffef2aa03f88f9 [term 6 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260129 02:59:31.536583 20585 raft_consensus.cc:2468] T e4d18c35b0a7493e85d0a4b92956a48f P 1c50de89a5a845668696dcb28c3badd6 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 197b880fa39d432a9b472845876bfaf0 in term 2.
I20260129 02:59:31.547148 20590 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "78aa39096868415d9afc2ffcb248e198" candidate_uuid: "197b880fa39d432a9b472845876bfaf0" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "1c50de89a5a845668696dcb28c3badd6" is_pre_election: true
I20260129 02:59:31.414863 22190 raft_consensus.cc:515] T 2ac659d732314e1288849f08fe5e074e P 197b880fa39d432a9b472845876bfaf0 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } }
I20260129 02:59:31.522795 22126 raft_consensus.cc:493] T 4f8f3a38f6e24ba4857875181f447442 P 1c50de89a5a845668696dcb28c3badd6 [term 3 FOLLOWER]: Starting leader election (detected failure of leader 974b802141bd4f13beffef2aa03f88f9)
I20260129 02:59:31.543311 20586 raft_consensus.cc:2468] T 2e58e9a5a8aa4cecafb464addb4f35ed P 1c50de89a5a845668696dcb28c3badd6 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 197b880fa39d432a9b472845876bfaf0 in term 2.
I20260129 02:59:31.502221 22082 raft_consensus.cc:2804] T c934d31efd3f4e5195605ddc354d14e1 P 197b880fa39d432a9b472845876bfaf0 [term 3 FOLLOWER]: Leader election won for term 3
I20260129 02:59:31.524504 22090 raft_consensus.cc:515] T 36ff33be666a4c82954731691dbcf327 P 1c50de89a5a845668696dcb28c3badd6 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:59:31.502353 21991 raft_consensus.cc:493] T 4e3805ffa70242e8ade66da260dddf66 P 1c50de89a5a845668696dcb28c3badd6 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 7a9daf5d42a84615a6383cfec9a10608)
I20260129 02:59:31.521970 20513 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "8055232cf0004d40a85f3bf983fe00a9" candidate_uuid: "1c50de89a5a845668696dcb28c3badd6" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "974b802141bd4f13beffef2aa03f88f9" is_pre_election: true
I20260129 02:59:31.512787 22157 raft_consensus.cc:2749] T 75be4bd6bd45474795fd472eff2953ac P 974b802141bd4f13beffef2aa03f88f9 [term 7 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20260129 02:59:31.545888 20439 raft_consensus.cc:3060] T 75be4bd6bd45474795fd472eff2953ac P 197b880fa39d432a9b472845876bfaf0 [term 6 FOLLOWER]: Advancing to term 7
I20260129 02:59:31.534523 22079 raft_consensus.cc:3060] T 0f20af7fbe5d4db89a6f17948d98912c P 1c50de89a5a845668696dcb28c3badd6 [term 3 FOLLOWER]: Advancing to term 4
I20260129 02:59:31.437849 21942 raft_consensus.cc:515] T 0678b8b468bd43cdbc3e485900844821 P 197b880fa39d432a9b472845876bfaf0 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } }
I20260129 02:59:31.523393 20434 raft_consensus.cc:2468] T e7d83d4ccf684e76acb29a0dc844e1b8 P 197b880fa39d432a9b472845876bfaf0 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 974b802141bd4f13beffef2aa03f88f9 in term 2.
I20260129 02:59:31.529959 22222 raft_consensus.cc:493] T 75f45538dc704d9a8ec4ff8ad9f3ea5b P 7a9daf5d42a84615a6383cfec9a10608 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 974b802141bd4f13beffef2aa03f88f9)
I20260129 02:59:31.559602 21991 raft_consensus.cc:3060] T 4e3805ffa70242e8ade66da260dddf66 P 1c50de89a5a845668696dcb28c3badd6 [term 2 FOLLOWER]: Advancing to term 3
I20260129 02:59:31.592355 20586 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7d9494f5584d4d5db8b9adf1c8fc888b" candidate_uuid: "197b880fa39d432a9b472845876bfaf0" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "1c50de89a5a845668696dcb28c3badd6" is_pre_election: true
I20260129 02:59:31.632463 20582 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "8055232cf0004d40a85f3bf983fe00a9" candidate_uuid: "197b880fa39d432a9b472845876bfaf0" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1c50de89a5a845668696dcb28c3badd6" is_pre_election: true
I20260129 02:59:31.545485 22164 raft_consensus.cc:493] T 724a236524cb4e7ba3428880681d6cc2 P 1c50de89a5a845668696dcb28c3badd6 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 974b802141bd4f13beffef2aa03f88f9)
I20260129 02:59:31.612155 20588 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "36ff33be666a4c82954731691dbcf327" candidate_uuid: "197b880fa39d432a9b472845876bfaf0" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1c50de89a5a845668696dcb28c3badd6" is_pre_election: true
I20260129 02:59:31.599645 20589 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a1a39dcd328e4268ae41b988c66b4a75" candidate_uuid: "197b880fa39d432a9b472845876bfaf0" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "1c50de89a5a845668696dcb28c3badd6" is_pre_election: true
I20260129 02:59:31.545960 22196 raft_consensus.cc:2749] T 1ce4ddf5b10e4ae8b193b13b2e5936ee P 974b802141bd4f13beffef2aa03f88f9 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260129 02:59:31.629844 20585 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "368953c9102643b2a21f99af36b51905" candidate_uuid: "197b880fa39d432a9b472845876bfaf0" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "1c50de89a5a845668696dcb28c3badd6" is_pre_election: true
I20260129 02:59:31.631429 20514 raft_consensus.cc:2468] T a4c1226b987b468fa84dfc6ba0e2dbe1 P 974b802141bd4f13beffef2aa03f88f9 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1c50de89a5a845668696dcb28c3badd6 in term 3.
I20260129 02:59:31.656610 20585 raft_consensus.cc:2393] T 368953c9102643b2a21f99af36b51905 P 1c50de89a5a845668696dcb28c3badd6 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 197b880fa39d432a9b472845876bfaf0 in current term 4: Already voted for candidate 1c50de89a5a845668696dcb28c3badd6 in this term.
I20260129 02:59:31.556411 22131 raft_consensus.cc:493] T 913fc7d751ed4b7388d889aac30a5f9b P 1c50de89a5a845668696dcb28c3badd6 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 197b880fa39d432a9b472845876bfaf0)
I20260129 02:59:31.642724 20439 raft_consensus.cc:2468] T 75be4bd6bd45474795fd472eff2953ac P 197b880fa39d432a9b472845876bfaf0 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1c50de89a5a845668696dcb28c3badd6 in term 7.
I20260129 02:59:31.648398 20588 raft_consensus.cc:2468] T 36ff33be666a4c82954731691dbcf327 P 1c50de89a5a845668696dcb28c3badd6 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 197b880fa39d432a9b472845876bfaf0 in term 1.
I20260129 02:59:31.556772 22126 raft_consensus.cc:3060] T 4f8f3a38f6e24ba4857875181f447442 P 1c50de89a5a845668696dcb28c3badd6 [term 3 FOLLOWER]: Advancing to term 4
I20260129 02:59:31.647927 20590 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a8479d1fe0bd41e5946186e3a6ca8c7a" candidate_uuid: "197b880fa39d432a9b472845876bfaf0" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "1c50de89a5a845668696dcb28c3badd6" is_pre_election: true
W20260129 02:59:31.597415 22004 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.636s user 0.052s sys 0.056s
I20260129 02:59:31.663071 20590 raft_consensus.cc:2468] T a8479d1fe0bd41e5946186e3a6ca8c7a P 1c50de89a5a845668696dcb28c3badd6 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 197b880fa39d432a9b472845876bfaf0 in term 2.
I20260129 02:59:31.551651 22069 raft_consensus.cc:2804] T 76f713c3331540d2a5b76164103e9eb1 P 197b880fa39d432a9b472845876bfaf0 [term 3 FOLLOWER]: Leader election won for term 3
I20260129 02:59:31.652872 20586 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "cb4727386a3e48aca5d71cc6c611c197" candidate_uuid: "197b880fa39d432a9b472845876bfaf0" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1c50de89a5a845668696dcb28c3badd6" is_pre_election: true
W20260129 02:59:31.641386 20553 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.211.1:42027, user_credentials={real_user=slave}} blocked reactor thread for 651139us
I20260129 02:59:31.633355 20582 raft_consensus.cc:2468] T 8055232cf0004d40a85f3bf983fe00a9 P 1c50de89a5a845668696dcb28c3badd6 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 197b880fa39d432a9b472845876bfaf0 in term 1.
I20260129 02:59:31.589820 22082 raft_consensus.cc:697] T c934d31efd3f4e5195605ddc354d14e1 P 197b880fa39d432a9b472845876bfaf0 [term 3 LEADER]: Becoming Leader. State: Replica: 197b880fa39d432a9b472845876bfaf0, State: Running, Role: LEADER
I20260129 02:59:31.664526 20586 raft_consensus.cc:2468] T cb4727386a3e48aca5d71cc6c611c197 P 1c50de89a5a845668696dcb28c3badd6 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 197b880fa39d432a9b472845876bfaf0 in term 1.
I20260129 02:59:31.666349 20553 leader_election.cc:304] T dc2f4cb752cf4d59b5f6bc689f5d8104 P 1c50de89a5a845668696dcb28c3badd6 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1c50de89a5a845668696dcb28c3badd6; no voters: 197b880fa39d432a9b472845876bfaf0, 7a9daf5d42a84615a6383cfec9a10608
I20260129 02:59:31.672811 20586 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "17d6431cc77a491e8464d68db4081a51" candidate_uuid: "197b880fa39d432a9b472845876bfaf0" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1c50de89a5a845668696dcb28c3badd6" is_pre_election: true
I20260129 02:59:31.673710 20586 raft_consensus.cc:2468] T 17d6431cc77a491e8464d68db4081a51 P 1c50de89a5a845668696dcb28c3badd6 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 197b880fa39d432a9b472845876bfaf0 in term 1.
I20260129 02:59:31.677733 20514 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3906b080a5b34a74b8b64128635e8997" candidate_uuid: "1c50de89a5a845668696dcb28c3badd6" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "974b802141bd4f13beffef2aa03f88f9"
I20260129 02:59:31.678838 20514 raft_consensus.cc:3060] T 3906b080a5b34a74b8b64128635e8997 P 974b802141bd4f13beffef2aa03f88f9 [term 1 FOLLOWER]: Advancing to term 2
W20260129 02:59:31.665939 22004 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.714s user 0.052s sys 0.057s
I20260129 02:59:31.658695 22164 raft_consensus.cc:3060] T 724a236524cb4e7ba3428880681d6cc2 P 1c50de89a5a845668696dcb28c3badd6 [term 1 FOLLOWER]: Advancing to term 2
I20260129 02:59:31.678026 20585 raft_consensus.cc:1240] T 3906b080a5b34a74b8b64128635e8997 P 1c50de89a5a845668696dcb28c3badd6 [term 2 FOLLOWER]: Rejecting Update request from peer 197b880fa39d432a9b472845876bfaf0 for earlier term 1. Current term is 2. Ops: []
I20260129 02:59:31.650681 22222 raft_consensus.cc:3060] T 75f45538dc704d9a8ec4ff8ad9f3ea5b P 7a9daf5d42a84615a6383cfec9a10608 [term 1 FOLLOWER]: Advancing to term 2
W20260129 02:59:31.655349 20434 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55672 (request call id 1382) took 1794 ms (client timeout 2191 ms). Trace:
W20260129 02:59:31.703760 22234 log.cc:927] Time spent T a5a05fbcbbcf4916bbc01fd1a5b5e90d P 1c50de89a5a845668696dcb28c3badd6: Append to log took a long time: real 0.109s user 0.000s sys 0.000s
I20260129 02:59:31.715150 22122 consensus_queue.cc:1048] T dc2f4cb752cf4d59b5f6bc689f5d8104 P 7a9daf5d42a84615a6383cfec9a10608 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20260129 02:59:31.718262 21128 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34046 (request call id 371) took 2102 ms. Trace:
I20260129 02:59:31.724159 20514 raft_consensus.cc:2468] T 3906b080a5b34a74b8b64128635e8997 P 974b802141bd4f13beffef2aa03f88f9 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1c50de89a5a845668696dcb28c3badd6 in term 2.
I20260129 02:59:31.737926 20432 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55672 (request call id 1373) took 3581 ms. Trace:
I20260129 02:59:31.743017 20432 rpcz_store.cc:276] 0129 02:59:28.156415 (+ 0us) service_pool.cc:168] Inserting onto call queue
0129 02:59:29.351873 (+1195458us) service_pool.cc:225] Handling call
0129 02:59:29.352330 (+ 457us) raft_consensus.cc:1471] Updating replica for 0 ops
0129 02:59:29.475176 (+122846us) raft_consensus.cc:1534] Early marking committed up to index 0
0129 02:59:29.475223 (+ 47us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0129 02:59:29.475282 (+ 59us) raft_consensus.cc:1645] Marking committed up to 2
0129 02:59:29.475325 (+ 43us) raft_consensus.cc:1697] Filling consensus response to leader.
0129 02:59:31.737547 (+2262222us) spinlock_profiling.cc:243] Waited 123 ms on lock 0x7b54002cfbd4. stack: 00007f8fe044e373 00007f8fe044e270 00007f8fdfda0574 00007f8fe9a5c6b4 00007f8fe9a5c649 00007f8fe34975a1 00007f8fe352ea81 00007f8fe352d4be 00007f8fe84e1f49 00007f8fe1a6ed34 00007f8fe1a6ecc2 00007f8fe1a6ebf1 00007f8fe1a6eb6c 00007f8fe1a6de22 00007f8fe186d34c 00007f8fe186c786
0129 02:59:31.737617 (+ 70us) raft_consensus.cc:1691] UpdateReplicas() finished
0129 02:59:31.737830 (+ 213us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":281937280}
I20260129 02:59:31.746517 20514 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "505580c93c694bbb8ea96fdf9b17ab46" candidate_uuid: "1c50de89a5a845668696dcb28c3badd6" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "974b802141bd4f13beffef2aa03f88f9"
W20260129 02:59:31.704350 22236 log.cc:927] Time spent T e33a417cf9c54cf5af73a4ee3aac4d3a P 1c50de89a5a845668696dcb28c3badd6: Append to log took a long time: real 0.073s user 0.000s sys 0.000s
I20260129 02:59:31.739318 20441 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55672 (request call id 1376) took 3412 ms. Trace:
I20260129 02:59:31.796262 20441 rpcz_store.cc:276] 0129 02:59:28.326679 (+ 0us) service_pool.cc:168] Inserting onto call queue
0129 02:59:29.484873 (+1158194us) service_pool.cc:225] Handling call
0129 02:59:30.827494 (+1342621us) raft_consensus.cc:1471] Updating replica for 0 ops
0129 02:59:30.827691 (+ 197us) raft_consensus.cc:1534] Early marking committed up to index 0
0129 02:59:30.827724 (+ 33us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0129 02:59:30.827764 (+ 40us) raft_consensus.cc:1645] Marking committed up to 5
0129 02:59:30.827807 (+ 43us) raft_consensus.cc:1697] Filling consensus response to leader.
0129 02:59:30.827921 (+ 114us) raft_consensus.cc:1691] UpdateReplicas() finished
0129 02:59:31.739052 (+911131us) spinlock_profiling.cc:243] Waited 1.34 s on lock 0x7b54003d1650. stack: 00007f8fe044e373 00007f8fe044e270 00007f8fdfda0574 00007f8fe9a5c6b4 00007f8fe9a5c649 00007f8fe34975a1 00007f8fe352d52d 00007f8fe84e1f49 00007f8fe1a6ed34 00007f8fe1a6ecc2 00007f8fe1a6ebf1 00007f8fe1a6eb6c 00007f8fe1a6de22 00007f8fe186d34c 00007f8fe186c786 00007f8fe186c12f
0129 02:59:31.739234 (+ 182us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":3085401856}
W20260129 02:59:31.812211 21102 leader_election.cc:336] T 3e2d15f8640b407195210c476e09df91 P 7a9daf5d42a84615a6383cfec9a10608 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 974b802141bd4f13beffef2aa03f88f9 (127.18.211.2:34517): Timed out: RequestConsensusVote RPC to 127.18.211.2:34517 timed out after 1.667s (SENT)
I20260129 02:59:31.668941 22131 raft_consensus.cc:515] T 913fc7d751ed4b7388d889aac30a5f9b P 1c50de89a5a845668696dcb28c3badd6 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:59:31.812647 20432 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3906b080a5b34a74b8b64128635e8997" candidate_uuid: "974b802141bd4f13beffef2aa03f88f9" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "197b880fa39d432a9b472845876bfaf0" is_pre_election: true
I20260129 02:59:31.718861 21128 rpcz_store.cc:276] 0129 02:59:29.615982 (+ 0us) service_pool.cc:168] Inserting onto call queue
0129 02:59:29.628347 (+ 12365us) service_pool.cc:225] Handling call
0129 02:59:29.703061 (+ 74714us) raft_consensus.cc:1471] Updating replica for 0 ops
0129 02:59:29.849742 (+146681us) raft_consensus.cc:1534] Early marking committed up to index 0
0129 02:59:29.851557 (+ 1815us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0129 02:59:29.853740 (+ 2183us) raft_consensus.cc:1645] Marking committed up to 4
0129 02:59:29.861612 (+ 7872us) raft_consensus.cc:1697] Filling consensus response to leader.
0129 02:59:31.717862 (+1856250us) spinlock_profiling.cc:243] Waited 111 ms on lock 0x7b54005cd5d4. stack: 00007f8fe044e373 00007f8fe044e270 00007f8fdfda0574 00007f8fe9a5c6b4 00007f8fe9a5c649 00007f8fe34975a1 00007f8fe352ea81 00007f8fe352d4be 00007f8fe84e1f49 00007f8fe1a6ed34 00007f8fe1a6ecc2 00007f8fe1a6ebf1 00007f8fe1a6eb6c 00007f8fe1a6de22 00007f8fe186d34c 00007f8fe186c786
0129 02:59:31.717931 (+ 69us) raft_consensus.cc:1691] UpdateReplicas() finished
0129 02:59:31.718151 (+ 220us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":255624448}
W20260129 02:59:31.820871 20432 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55672 (request call id 1385) took 1714 ms (client timeout 2026 ms). Trace:
W20260129 02:59:31.821439 20432 rpcz_store.cc:269] 0129 02:59:30.106815 (+ 0us) service_pool.cc:168] Inserting onto call queue
0129 02:59:31.807399 (+1700584us) service_pool.cc:225] Handling call
0129 02:59:31.820792 (+ 13393us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20260129 02:59:31.820684 20407 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.211.2:34517, user_credentials={real_user=slave}} blocked reactor thread for 471271us
I20260129 02:59:31.817446 21102 leader_election.cc:304] T 3e2d15f8640b407195210c476e09df91 P 7a9daf5d42a84615a6383cfec9a10608 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7a9daf5d42a84615a6383cfec9a10608; no voters: 1c50de89a5a845668696dcb28c3badd6, 974b802141bd4f13beffef2aa03f88f9
W20260129 02:59:31.830577 20480 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.211.3:37735, user_credentials={real_user=slave}} blocked reactor thread for 325861us
I20260129 02:59:31.703274 22069 raft_consensus.cc:697] T 76f713c3331540d2a5b76164103e9eb1 P 197b880fa39d432a9b472845876bfaf0 [term 3 LEADER]: Becoming Leader. State: Replica: 197b880fa39d432a9b472845876bfaf0, State: Running, Role: LEADER
I20260129 02:59:31.832237 20432 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a609ac02fae84725ab743f95f51d33ce" candidate_uuid: "974b802141bd4f13beffef2aa03f88f9" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "197b880fa39d432a9b472845876bfaf0" is_pre_election: true
I20260129 02:59:31.832600 20514 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0340229d521b432580ae926f6062e3f7" candidate_uuid: "1c50de89a5a845668696dcb28c3badd6" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "974b802141bd4f13beffef2aa03f88f9" is_pre_election: true
I20260129 02:59:31.837227 20514 raft_consensus.cc:2468] T 0340229d521b432580ae926f6062e3f7 P 974b802141bd4f13beffef2aa03f88f9 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1c50de89a5a845668696dcb28c3badd6 in term 3.
I20260129 02:59:31.836594 20432 raft_consensus.cc:2468] T a609ac02fae84725ab743f95f51d33ce P 197b880fa39d432a9b472845876bfaf0 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 974b802141bd4f13beffef2aa03f88f9 in term 4.
I20260129 02:59:31.842150 20432 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55672 (request call id 1387) took 1359 ms. Trace:
I20260129 02:59:31.842549 20432 rpcz_store.cc:276] 0129 02:59:30.483266 (+ 0us) service_pool.cc:168] Inserting onto call queue
0129 02:59:31.831956 (+1348690us) service_pool.cc:225] Handling call
0129 02:59:31.842075 (+ 10119us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260129 02:59:31.843783 20432 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "913fc7d751ed4b7388d889aac30a5f9b" candidate_uuid: "974b802141bd4f13beffef2aa03f88f9" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "197b880fa39d432a9b472845876bfaf0" is_pre_election: true
I20260129 02:59:31.844753 20432 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55672 (request call id 1392) took 1208 ms. Trace:
I20260129 02:59:31.845145 20432 rpcz_store.cc:276] 0129 02:59:30.636571 (+ 0us) service_pool.cc:168] Inserting onto call queue
0129 02:59:31.843471 (+1206900us) service_pool.cc:225] Handling call
0129 02:59:31.844691 (+ 1220us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260129 02:59:31.846176 20432 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0340229d521b432580ae926f6062e3f7" candidate_uuid: "974b802141bd4f13beffef2aa03f88f9" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "197b880fa39d432a9b472845876bfaf0" is_pre_election: true
I20260129 02:59:31.820205 22227 raft_consensus.cc:2749] T dc2f4cb752cf4d59b5f6bc689f5d8104 P 1c50de89a5a845668696dcb28c3badd6 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
W20260129 02:59:31.856602 20405 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.211.4:40439, user_credentials={real_user=slave}} blocked reactor thread for 356095us
I20260129 02:59:31.849840 21136 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0678b8b468bd43cdbc3e485900844821" candidate_uuid: "974b802141bd4f13beffef2aa03f88f9" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "7a9daf5d42a84615a6383cfec9a10608" is_pre_election: true
I20260129 02:59:31.858834 20432 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55672 (request call id 1390) took 1272 ms. Trace:
I20260129 02:59:31.859491 20432 rpcz_store.cc:276] 0129 02:59:30.586174 (+ 0us) service_pool.cc:168] Inserting onto call queue
0129 02:59:31.845877 (+1259703us) service_pool.cc:225] Handling call
0129 02:59:31.858751 (+ 12874us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260129 02:59:31.861714 21133 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "45cc5f3d2688486d9b6dbe7e666dedde" candidate_uuid: "974b802141bd4f13beffef2aa03f88f9" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "7a9daf5d42a84615a6383cfec9a10608" is_pre_election: true
I20260129 02:59:31.862424 20405 leader_election.cc:304] T 18f30ebaa4174fd28c958e72d0cc7e24 P 197b880fa39d432a9b472845876bfaf0 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 197b880fa39d432a9b472845876bfaf0; no voters: 1c50de89a5a845668696dcb28c3badd6, 7a9daf5d42a84615a6383cfec9a10608
I20260129 02:59:31.815877 22144 leader_election.cc:290] T 75f45538dc704d9a8ec4ff8ad9f3ea5b P 1c50de89a5a845668696dcb28c3badd6 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 974b802141bd4f13beffef2aa03f88f9 (127.18.211.2:34517), 7a9daf5d42a84615a6383cfec9a10608 (127.18.211.4:40439)
I20260129 02:59:31.863950 21136 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "add8d4923d1a489e86882c3a25692b95" candidate_uuid: "974b802141bd4f13beffef2aa03f88f9" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "7a9daf5d42a84615a6383cfec9a10608" is_pre_election: true
I20260129 02:59:31.849344 22165 raft_consensus.cc:493] T e33a417cf9c54cf5af73a4ee3aac4d3a P 197b880fa39d432a9b472845876bfaf0 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 1c50de89a5a845668696dcb28c3badd6)
W20260129 02:59:31.841598 20407 connection.cc:232] RPC call timeout handler was delayed by -1.1043478089999326s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.221s for a response
W20260129 02:59:31.865797 20554 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.211.4:40439, user_credentials={real_user=slave}} blocked reactor thread for 442672us
I20260129 02:59:31.866506 20514 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7d9494f5584d4d5db8b9adf1c8fc888b" candidate_uuid: "197b880fa39d432a9b472845876bfaf0" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "974b802141bd4f13beffef2aa03f88f9" is_pre_election: true
I20260129 02:59:31.865549 22233 raft_consensus.cc:493] T a5a05fbcbbcf4916bbc01fd1a5b5e90d P 197b880fa39d432a9b472845876bfaf0 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 1c50de89a5a845668696dcb28c3badd6)
W20260129 02:59:31.856319 22101 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.512s user 0.013s sys 0.032s
I20260129 02:59:31.869088 21135 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "87d5fe48a44a408aaf03e894a67ecb2f" candidate_uuid: "974b802141bd4f13beffef2aa03f88f9" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "7a9daf5d42a84615a6383cfec9a10608" is_pre_election: true
W20260129 02:59:31.872279 20435 consensus_meta.cc:287] T 368953c9102643b2a21f99af36b51905 P 197b880fa39d432a9b472845876bfaf0: Time spent flushing consensus metadata: real 0.863s user 0.012s sys 0.000s
I20260129 02:59:31.927451 20435 raft_consensus.cc:2468] T 368953c9102643b2a21f99af36b51905 P 197b880fa39d432a9b472845876bfaf0 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1c50de89a5a845668696dcb28c3badd6 in term 4.
I20260129 02:59:31.155388 20433 rpcz_store.cc:276] 0129 02:59:28.847121 (+ 0us) service_pool.cc:168] Inserting onto call queue
0129 02:59:29.545957 (+698836us) service_pool.cc:225] Handling call
0129 02:59:29.546230 (+ 273us) raft_consensus.cc:1471] Updating replica for 0 ops
0129 02:59:29.546435 (+ 205us) raft_consensus.cc:1534] Early marking committed up to index 6
0129 02:59:30.620305 (+1073870us) log.cc:844] Serialized 14 byte log entry
0129 02:59:30.849683 (+229378us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0129 02:59:30.894477 (+ 44794us) raft_consensus.cc:1645] Marking committed up to 6
0129 02:59:30.899043 (+ 4566us) raft_consensus.cc:1697] Filling consensus response to leader.
0129 02:59:30.922102 (+ 23059us) raft_consensus.cc:1691] UpdateReplicas() finished
0129 02:59:30.945538 (+ 23436us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":6033,"prepare.queue_time_us":1119383,"prepare.run_cpu_time_us":81,"prepare.run_wall_time_us":6615,"raft.queue_time_us":20482,"raft.run_cpu_time_us":200,"raft.run_wall_time_us":5663,"thread_start_us":1239379,"threads_started":2,"wal-append.queue_time_us":325777}
I20260129 02:59:31.896973 20432 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b189f08e16a74a11980cc55d99359a0a" candidate_uuid: "1c50de89a5a845668696dcb28c3badd6" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "197b880fa39d432a9b472845876bfaf0" is_pre_election: true
I20260129 02:59:31.898700 20480 leader_election.cc:304] T 0678b8b468bd43cdbc3e485900844821 P 974b802141bd4f13beffef2aa03f88f9 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 974b802141bd4f13beffef2aa03f88f9; no voters: 197b880fa39d432a9b472845876bfaf0, 7a9daf5d42a84615a6383cfec9a10608
I20260129 02:59:31.872557 22195 leader_election.cc:290] T 75be4bd6bd45474795fd472eff2953ac P 197b880fa39d432a9b472845876bfaf0 [CANDIDATE]: Term 6 election: Requested vote from peers 974b802141bd4f13beffef2aa03f88f9 (127.18.211.2:34517), 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3:37735)
I20260129 02:59:31.908572 21136 raft_consensus.cc:2468] T add8d4923d1a489e86882c3a25692b95 P 7a9daf5d42a84615a6383cfec9a10608 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 974b802141bd4f13beffef2aa03f88f9 in term 1.
I20260129 02:59:31.857781 22102 leader_election.cc:290] T a0fcc58d6fb6473cafbd9d34d5195269 P 1c50de89a5a845668696dcb28c3badd6 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 197b880fa39d432a9b472845876bfaf0 (127.18.211.1:42027), 974b802141bd4f13beffef2aa03f88f9 (127.18.211.2:34517)
I20260129 02:59:31.860642 20441 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e33a417cf9c54cf5af73a4ee3aac4d3a" candidate_uuid: "974b802141bd4f13beffef2aa03f88f9" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "197b880fa39d432a9b472845876bfaf0" is_pre_election: true
I20260129 02:59:31.871019 22160 raft_consensus.cc:493] T febffe4a8b394368b69b1d164308fb9a P 974b802141bd4f13beffef2aa03f88f9 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 197b880fa39d432a9b472845876bfaf0)
I20260129 02:59:31.913900 20516 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00e79d8f25a34676b1d866eb2068a2b2" candidate_uuid: "7a9daf5d42a84615a6383cfec9a10608" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "974b802141bd4f13beffef2aa03f88f9" is_pre_election: true
I20260129 02:59:31.960428 20516 raft_consensus.cc:2468] T 00e79d8f25a34676b1d866eb2068a2b2 P 974b802141bd4f13beffef2aa03f88f9 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7a9daf5d42a84615a6383cfec9a10608 in term 3.
I20260129 02:59:31.961330 20516 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "febffe4a8b394368b69b1d164308fb9a" candidate_uuid: "7a9daf5d42a84615a6383cfec9a10608" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "974b802141bd4f13beffef2aa03f88f9" is_pre_election: true
I20260129 02:59:31.925864 20507 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57600 (request call id 1621) took 1895 ms. Trace:
W20260129 02:59:31.706566 20434 rpcz_store.cc:269] 0129 02:59:29.860923 (+ 0us) service_pool.cc:168] Inserting onto call queue
0129 02:59:31.158669 (+1297746us) service_pool.cc:225] Handling call
0129 02:59:31.649003 (+490334us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260129 02:59:31.868621 21988 leader_election.cc:290] T 1ce4ddf5b10e4ae8b193b13b2e5936ee P 1c50de89a5a845668696dcb28c3badd6 [CANDIDATE]: Term 4 election: Requested vote from peers 197b880fa39d432a9b472845876bfaf0 (127.18.211.1:42027), 974b802141bd4f13beffef2aa03f88f9 (127.18.211.2:34517)
W20260129 02:59:31.928256 20435 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55676 (request call id 1067) took 2255 ms (client timeout 2256 ms). Trace:
W20260129 02:59:31.862161 21883 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.593s user 0.034s sys 0.041s
I20260129 02:59:31.945847 20432 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55676 (request call id 1089) took 1001 ms. Trace:
I20260129 02:59:31.946866 20480 leader_election.cc:304] T 87d5fe48a44a408aaf03e894a67ecb2f P 974b802141bd4f13beffef2aa03f88f9 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 974b802141bd4f13beffef2aa03f88f9; no voters: 197b880fa39d432a9b472845876bfaf0, 7a9daf5d42a84615a6383cfec9a10608
W20260129 02:59:31.970211 20435 rpcz_store.cc:269] 0129 02:59:29.672891 (+ 0us) service_pool.cc:168] Inserting onto call queue
0129 02:59:30.973056 (+1300165us) service_pool.cc:225] Handling call
0129 02:59:31.928192 (+955136us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20260129 02:59:31.971278 20554 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.211.4:40439, user_credentials={real_user=slave}} blocked reactor thread for 57746.1us
I20260129 02:59:31.972316 20480 leader_election.cc:304] T add8d4923d1a489e86882c3a25692b95 P 974b802141bd4f13beffef2aa03f88f9 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7a9daf5d42a84615a6383cfec9a10608, 974b802141bd4f13beffef2aa03f88f9; no voters:
I20260129 02:59:31.871223 22237 raft_consensus.cc:493] T dc2f4cb752cf4d59b5f6bc689f5d8104 P 197b880fa39d432a9b472845876bfaf0 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 7a9daf5d42a84615a6383cfec9a10608)
I20260129 02:59:31.956000 21133 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00e79d8f25a34676b1d866eb2068a2b2" candidate_uuid: "974b802141bd4f13beffef2aa03f88f9" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "7a9daf5d42a84615a6383cfec9a10608" is_pre_election: true
W20260129 02:59:31.955334 21102 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.211.2:34517, user_credentials={real_user=slave}} blocked reactor thread for 119948us
I20260129 02:59:31.877348 22157 raft_consensus.cc:493] T 3ccac82a01a2445fa5b0ce6c0fcf2295 P 974b802141bd4f13beffef2aa03f88f9 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 197b880fa39d432a9b472845876bfaf0)
I20260129 02:59:31.979988 20582 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a5a05fbcbbcf4916bbc01fd1a5b5e90d" candidate_uuid: "974b802141bd4f13beffef2aa03f88f9" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "1c50de89a5a845668696dcb28c3badd6" is_pre_election: true
I20260129 02:59:31.926412 22190 leader_election.cc:290] T 2ac659d732314e1288849f08fe5e074e P 197b880fa39d432a9b472845876bfaf0 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 974b802141bd4f13beffef2aa03f88f9 (127.18.211.2:34517), 7a9daf5d42a84615a6383cfec9a10608 (127.18.211.4:40439)
I20260129 02:59:31.962373 21136 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1299723a4476420ab02fd8bfd7aa0645" candidate_uuid: "974b802141bd4f13beffef2aa03f88f9" candidate_term: 5 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "7a9daf5d42a84615a6383cfec9a10608" is_pre_election: true
I20260129 02:59:31.993115 21136 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "75f45538dc704d9a8ec4ff8ad9f3ea5b" candidate_uuid: "1c50de89a5a845668696dcb28c3badd6" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "7a9daf5d42a84615a6383cfec9a10608" is_pre_election: true
I20260129 02:59:31.908711 21129 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a609ac02fae84725ab743f95f51d33ce" candidate_uuid: "974b802141bd4f13beffef2aa03f88f9" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "7a9daf5d42a84615a6383cfec9a10608" is_pre_election: true
I20260129 02:59:31.962004 20507 rpcz_store.cc:276] 0129 02:59:30.030379 (+ 0us) service_pool.cc:168] Inserting onto call queue
0129 02:59:30.030531 (+ 152us) service_pool.cc:225] Handling call
0129 02:59:30.131720 (+101189us) raft_consensus.cc:1471] Updating replica for 0 ops
0129 02:59:30.337517 (+205797us) raft_consensus.cc:1534] Early marking committed up to index 0
0129 02:59:30.337550 (+ 33us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0129 02:59:30.337600 (+ 50us) raft_consensus.cc:1645] Marking committed up to 1
0129 02:59:30.337630 (+ 30us) raft_consensus.cc:1697] Filling consensus response to leader.
0129 02:59:31.925597 (+1587967us) spinlock_profiling.cc:243] Waited 206 ms on lock 0x7b5400229554. stack: 00007f8fe044e373 00007f8fe044e270 00007f8fdfda0574 00007f8fe9a5c6b4 00007f8fe9a5c649 00007f8fe34975a1 00007f8fe352ea81 00007f8fe352d4be 00007f8fe84e1f49 00007f8fe1a6ed34 00007f8fe1a6ecc2 00007f8fe1a6ebf1 00007f8fe1a6eb6c 00007f8fe1a6de22 00007f8fe186d34c 00007f8fe186c786
0129 02:59:31.925648 (+ 51us) raft_consensus.cc:1691] UpdateReplicas() finished
0129 02:59:31.925803 (+ 155us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":472765184}
I20260129 02:59:31.948212 22201 raft_consensus.cc:2749] T 0678b8b468bd43cdbc3e485900844821 P 974b802141bd4f13beffef2aa03f88f9 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260129 02:59:31.970441 20514 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1299723a4476420ab02fd8bfd7aa0645" candidate_uuid: "7a9daf5d42a84615a6383cfec9a10608" candidate_term: 5 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "974b802141bd4f13beffef2aa03f88f9" is_pre_election: true
I20260129 02:59:32.003473 21135 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "96609ac68bdf4993a12e1c23355ca013" candidate_uuid: "974b802141bd4f13beffef2aa03f88f9" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "7a9daf5d42a84615a6383cfec9a10608"
I20260129 02:59:32.005471 20514 raft_consensus.cc:2468] T 1299723a4476420ab02fd8bfd7aa0645 P 974b802141bd4f13beffef2aa03f88f9 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7a9daf5d42a84615a6383cfec9a10608 in term 4.
I20260129 02:59:31.970448 20432 rpcz_store.cc:276] 0129 02:59:30.944662 (+ 0us) service_pool.cc:168] Inserting onto call queue
0129 02:59:31.896553 (+951891us) service_pool.cc:225] Handling call
0129 02:59:31.945762 (+ 49209us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20260129 02:59:31.974396 21883 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.748s user 0.034s sys 0.041s
I20260129 02:59:31.974375 20589 raft_consensus.cc:1240] T 505580c93c694bbb8ea96fdf9b17ab46 P 1c50de89a5a845668696dcb28c3badd6 [term 3 FOLLOWER]: Rejecting Update request from peer 974b802141bd4f13beffef2aa03f88f9 for earlier term 2. Current term is 3. Ops: []
I20260129 02:59:31.974772 22036 raft_consensus.cc:2804] T add8d4923d1a489e86882c3a25692b95 P 974b802141bd4f13beffef2aa03f88f9 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260129 02:59:31.976217 22104 leader_election.cc:290] T 96609ac68bdf4993a12e1c23355ca013 P 197b880fa39d432a9b472845876bfaf0 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 974b802141bd4f13beffef2aa03f88f9 (127.18.211.2:34517), 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3:37735), 7a9daf5d42a84615a6383cfec9a10608 (127.18.211.4:40439)
I20260129 02:59:31.971331 22160 raft_consensus.cc:515] T febffe4a8b394368b69b1d164308fb9a P 974b802141bd4f13beffef2aa03f88f9 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } }
I20260129 02:59:31.977766 20590 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c2633a6fc34c44579de348d208b0c01e" candidate_uuid: "974b802141bd4f13beffef2aa03f88f9" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1c50de89a5a845668696dcb28c3badd6" is_pre_election: true
I20260129 02:59:31.980451 20586 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "cedce331e1584ac8b636c9b8a72b2ba0" candidate_uuid: "974b802141bd4f13beffef2aa03f88f9" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "1c50de89a5a845668696dcb28c3badd6" is_pre_election: true
I20260129 02:59:31.982211 21133 raft_consensus.cc:2468] T 00e79d8f25a34676b1d866eb2068a2b2 P 7a9daf5d42a84615a6383cfec9a10608 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 974b802141bd4f13beffef2aa03f88f9 in term 3.
W20260129 02:59:32.022317 20479 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.211.1:42027, user_credentials={real_user=slave}} blocked reactor thread for 238306us
I20260129 02:59:32.023027 20480 leader_election.cc:304] T 00e79d8f25a34676b1d866eb2068a2b2 P 974b802141bd4f13beffef2aa03f88f9 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7a9daf5d42a84615a6383cfec9a10608, 974b802141bd4f13beffef2aa03f88f9; no voters:
I20260129 02:59:31.973567 20585 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "45cc5f3d2688486d9b6dbe7e666dedde" candidate_uuid: "974b802141bd4f13beffef2aa03f88f9" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "1c50de89a5a845668696dcb28c3badd6" is_pre_election: true
I20260129 02:59:31.957641 22239 raft_consensus.cc:2749] T 18f30ebaa4174fd28c958e72d0cc7e24 P 197b880fa39d432a9b472845876bfaf0 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20260129 02:59:31.926836 22101 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.594s user 0.014s sys 0.033s
I20260129 02:59:32.000831 20582 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a4c1226b987b468fa84dfc6ba0e2dbe1" candidate_uuid: "974b802141bd4f13beffef2aa03f88f9" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1c50de89a5a845668696dcb28c3badd6" is_pre_election: true
I20260129 02:59:31.935443 22233 raft_consensus.cc:515] T a5a05fbcbbcf4916bbc01fd1a5b5e90d P 197b880fa39d432a9b472845876bfaf0 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:59:31.866786 22082 consensus_queue.cc:237] T c934d31efd3f4e5195605ddc354d14e1 P 197b880fa39d432a9b472845876bfaf0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } }
I20260129 02:59:31.960626 21128 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a644c02b16af47679824ccd060ac10fd" candidate_uuid: "974b802141bd4f13beffef2aa03f88f9" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "7a9daf5d42a84615a6383cfec9a10608" is_pre_election: true
I20260129 02:59:32.011559 20589 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a1a39dcd328e4268ae41b988c66b4a75" candidate_uuid: "974b802141bd4f13beffef2aa03f88f9" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "1c50de89a5a845668696dcb28c3badd6" is_pre_election: true
I20260129 02:59:31.963534 21942 leader_election.cc:290] T 0678b8b468bd43cdbc3e485900844821 P 197b880fa39d432a9b472845876bfaf0 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 974b802141bd4f13beffef2aa03f88f9 (127.18.211.2:34517), 7a9daf5d42a84615a6383cfec9a10608 (127.18.211.4:40439)
I20260129 02:59:31.927179 22165 raft_consensus.cc:515] T e33a417cf9c54cf5af73a4ee3aac4d3a P 197b880fa39d432a9b472845876bfaf0 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:59:32.016870 20513 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "75f45538dc704d9a8ec4ff8ad9f3ea5b" candidate_uuid: "1c50de89a5a845668696dcb28c3badd6" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "974b802141bd4f13beffef2aa03f88f9" is_pre_election: true
I20260129 02:59:32.021376 20507 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4e420f90884f4796997b5ea41e49e411" candidate_uuid: "197b880fa39d432a9b472845876bfaf0" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "974b802141bd4f13beffef2aa03f88f9" is_pre_election: true
I20260129 02:59:32.025152 20514 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "75be4bd6bd45474795fd472eff2953ac" candidate_uuid: "197b880fa39d432a9b472845876bfaf0" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "974b802141bd4f13beffef2aa03f88f9"
I20260129 02:59:32.056802 20507 raft_consensus.cc:2468] T 4e420f90884f4796997b5ea41e49e411 P 974b802141bd4f13beffef2aa03f88f9 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 197b880fa39d432a9b472845876bfaf0 in term 3.
I20260129 02:59:32.057073 20514 raft_consensus.cc:2368] T 75be4bd6bd45474795fd472eff2953ac P 974b802141bd4f13beffef2aa03f88f9 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 197b880fa39d432a9b472845876bfaf0 for earlier term 6. Current term is 7.
I20260129 02:59:32.025571 20586 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e7d83d4ccf684e76acb29a0dc844e1b8" candidate_uuid: "974b802141bd4f13beffef2aa03f88f9" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "1c50de89a5a845668696dcb28c3badd6" is_pre_election: true
W20260129 02:59:32.026249 20405 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.211.3:37735, user_credentials={real_user=slave}} blocked reactor thread for 128828us
I20260129 02:59:31.944799 22090 leader_election.cc:290] T 36ff33be666a4c82954731691dbcf327 P 1c50de89a5a845668696dcb28c3badd6 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 197b880fa39d432a9b472845876bfaf0 (127.18.211.1:42027), 974b802141bd4f13beffef2aa03f88f9 (127.18.211.2:34517)
I20260129 02:59:32.028349 20582 raft_consensus.cc:2393] T a4c1226b987b468fa84dfc6ba0e2dbe1 P 1c50de89a5a845668696dcb28c3badd6 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 974b802141bd4f13beffef2aa03f88f9 in current term 3: Already voted for candidate 1c50de89a5a845668696dcb28c3badd6 in this term.
I20260129 02:59:32.029243 20585 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7d9494f5584d4d5db8b9adf1c8fc888b" candidate_uuid: "974b802141bd4f13beffef2aa03f88f9" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "1c50de89a5a845668696dcb28c3badd6" is_pre_election: true
I20260129 02:59:32.003013 20433 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "76f713c3331540d2a5b76164103e9eb1" candidate_uuid: "974b802141bd4f13beffef2aa03f88f9" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "197b880fa39d432a9b472845876bfaf0" is_pre_election: true
I20260129 02:59:32.031742 20556 leader_election.cc:304] T c2633a6fc34c44579de348d208b0c01e P 1c50de89a5a845668696dcb28c3badd6 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1c50de89a5a845668696dcb28c3badd6, 974b802141bd4f13beffef2aa03f88f9; no voters: 197b880fa39d432a9b472845876bfaf0
I20260129 02:59:32.023559 22190 raft_consensus.cc:493] T 4408006eb5174de1896b037cf3f18457 P 197b880fa39d432a9b472845876bfaf0 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 974b802141bd4f13beffef2aa03f88f9)
I20260129 02:59:32.031461 22201 raft_consensus.cc:2804] T 00e79d8f25a34676b1d866eb2068a2b2 P 974b802141bd4f13beffef2aa03f88f9 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260129 02:59:32.082847 21883 raft_consensus.cc:493] T fcddab6bca684adf8ea032289a00f90e P 197b880fa39d432a9b472845876bfaf0 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260129 02:59:32.083622 21883 raft_consensus.cc:515] T fcddab6bca684adf8ea032289a00f90e P 197b880fa39d432a9b472845876bfaf0 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
W20260129 02:59:32.040592 20553 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.211.1:42027, user_credentials={real_user=slave}} blocked reactor thread for 193585us
I20260129 02:59:32.050283 20516 raft_consensus.cc:2468] T febffe4a8b394368b69b1d164308fb9a P 974b802141bd4f13beffef2aa03f88f9 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7a9daf5d42a84615a6383cfec9a10608 in term 1.
I20260129 02:59:32.053491 21133 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "505580c93c694bbb8ea96fdf9b17ab46" candidate_uuid: "1c50de89a5a845668696dcb28c3badd6" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "7a9daf5d42a84615a6383cfec9a10608"
I20260129 02:59:32.089480 21133 raft_consensus.cc:3060] T 505580c93c694bbb8ea96fdf9b17ab46 P 7a9daf5d42a84615a6383cfec9a10608 [term 2 FOLLOWER]: Advancing to term 3
W20260129 02:59:32.010555 21100 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.211.3:37735, user_credentials={real_user=slave}} blocked reactor thread for 459021us
I20260129 02:59:31.996181 22157 raft_consensus.cc:515] T 3ccac82a01a2445fa5b0ce6c0fcf2295 P 974b802141bd4f13beffef2aa03f88f9 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } }
I20260129 02:59:32.020521 22036 raft_consensus.cc:493] T add8d4923d1a489e86882c3a25692b95 P 974b802141bd4f13beffef2aa03f88f9 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 1c50de89a5a845668696dcb28c3badd6)
W20260129 02:59:32.031229 22210 log.cc:927] Time spent T 78aa39096868415d9afc2ffcb248e198 P 197b880fa39d432a9b472845876bfaf0: Append to log took a long time: real 2.431s user 0.003s sys 0.026s
I20260129 02:59:32.093871 22036 raft_consensus.cc:3060] T add8d4923d1a489e86882c3a25692b95 P 974b802141bd4f13beffef2aa03f88f9 [term 1 FOLLOWER]: Advancing to term 2
I20260129 02:59:32.095223 20438 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55676 (request call id 1056) took 4318 ms. Trace:
I20260129 02:59:32.095749 20438 rpcz_store.cc:276] 0129 02:59:27.776773 (+ 0us) service_pool.cc:168] Inserting onto call queue
0129 02:59:28.897855 (+1121082us) service_pool.cc:225] Handling call
0129 02:59:28.898253 (+ 398us) raft_consensus.cc:1471] Updating replica for 1 ops
0129 02:59:29.263713 (+365460us) raft_consensus.cc:1534] Early marking committed up to index 0
0129 02:59:29.263745 (+ 32us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0129 02:59:29.267907 (+ 4162us) log.cc:844] Serialized 26 byte log entry
0129 02:59:29.599734 (+331827us) raft_consensus.cc:1645] Marking committed up to 4
0129 02:59:29.599795 (+ 61us) raft_consensus.cc:1697] Filling consensus response to leader.
0129 02:59:31.720430 (+2120635us) spinlock_profiling.cc:243] Waited 365 ms on lock 0x7b540035fa54. stack: 00007f8fe044e373 00007f8fe044e270 00007f8fdfda0574 00007f8fe9a5c6b4 00007f8fe9a5c649 00007f8fe34975a1 00007f8fe352ea81 00007f8fe352d4be 00007f8fe84e1f49 00007f8fe1a6ed34 00007f8fe1a6ecc2 00007f8fe1a6ebf1 00007f8fe1a6eb6c 00007f8fe1a6de22 00007f8fe186d34c 00007f8fe186c786
0129 02:59:31.720505 (+ 75us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0129 02:59:31.821064 (+100559us) spinlock_profiling.cc:243] Waited 29.9 ms on lock 0x7b440002304c. stack: 00007f8fe044e373 00007f8fe044e270 00007f8fdfda0574 00007f8fe9a5c6b4 00007f8fe9a5c649 00007f8fe02c08b1 00007f8fe038f005 00007f8fe038fa22 00007f8fe30d957b 00007f8fe30d4e4f 00007f8fe30c8c0e 00007f8fe30c7f56 00007f8fe30c7b78 00007f8fe30d15c1 00007f8fe30d1579 00007f8fe30d1509
0129 02:59:32.094921 (+273857us) raft_consensus.cc:1685] finished
0129 02:59:32.094952 (+ 31us) raft_consensus.cc:1691] UpdateReplicas() finished
0129 02:59:32.095133 (+ 181us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":515,"raft.run_cpu_time_us":166,"raft.run_wall_time_us":9973,"spinlock_wait_cycles":908471424,"thread_start_us":322286,"threads_started":1,"wal-append.queue_time_us":322704}
I20260129 02:59:32.098268 20438 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7a9a69cd82724b1faed1d10aa091fc4a" candidate_uuid: "974b802141bd4f13beffef2aa03f88f9" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "197b880fa39d432a9b472845876bfaf0" is_pre_election: true
I20260129 02:59:32.099296 20438 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55672 (request call id 1402) took 1112 ms. Trace:
I20260129 02:59:32.019909 20434 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4b1876f9a70b49edb3aa74105e66d3b3" candidate_uuid: "974b802141bd4f13beffef2aa03f88f9" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "197b880fa39d432a9b472845876bfaf0" is_pre_election: true
W20260129 02:59:32.101069 20434 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55672 (request call id 1389) took 1517 ms (client timeout 2009 ms). Trace:
W20260129 02:59:32.101598 20434 rpcz_store.cc:269] 0129 02:59:30.583989 (+ 0us) service_pool.cc:168] Inserting onto call queue
0129 02:59:31.988853 (+1404864us) service_pool.cc:225] Handling call
0129 02:59:32.100985 (+112132us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260129 02:59:32.102864 20434 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a4c1226b987b468fa84dfc6ba0e2dbe1" candidate_uuid: "1c50de89a5a845668696dcb28c3badd6" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "197b880fa39d432a9b472845876bfaf0"
I20260129 02:59:32.103847 20434 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55676 (request call id 1091) took 1128 ms. Trace:
I20260129 02:59:32.105608 20434 rpcz_store.cc:276] 0129 02:59:30.975661 (+ 0us) service_pool.cc:168] Inserting onto call queue
0129 02:59:32.102509 (+1126848us) service_pool.cc:225] Handling call
0129 02:59:32.103777 (+ 1268us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260129 02:59:32.107219 20434 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00e79d8f25a34676b1d866eb2068a2b2" candidate_uuid: "974b802141bd4f13beffef2aa03f88f9" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "197b880fa39d432a9b472845876bfaf0" is_pre_election: true
I20260129 02:59:31.996848 22237 raft_consensus.cc:515] T dc2f4cb752cf4d59b5f6bc689f5d8104 P 197b880fa39d432a9b472845876bfaf0 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } }
I20260129 02:59:32.064360 21128 raft_consensus.cc:2468] T a644c02b16af47679824ccd060ac10fd P 7a9daf5d42a84615a6383cfec9a10608 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 974b802141bd4f13beffef2aa03f88f9 in term 1.
I20260129 02:59:32.078269 22201 raft_consensus.cc:493] T 00e79d8f25a34676b1d866eb2068a2b2 P 974b802141bd4f13beffef2aa03f88f9 [term 3 FOLLOWER]: Starting leader election (detected failure of leader 197b880fa39d432a9b472845876bfaf0)
W20260129 02:59:32.240978 20433 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55672 (request call id 1398) took 1484 ms (client timeout 1723 ms). Trace:
W20260129 02:59:32.246409 20433 rpcz_store.cc:269] 0129 02:59:30.756771 (+ 0us) service_pool.cc:168] Inserting onto call queue
0129 02:59:31.960917 (+1204146us) service_pool.cc:225] Handling call
0129 02:59:32.240637 (+279720us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260129 02:59:32.153345 20480 leader_election.cc:304] T a644c02b16af47679824ccd060ac10fd P 974b802141bd4f13beffef2aa03f88f9 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7a9daf5d42a84615a6383cfec9a10608, 974b802141bd4f13beffef2aa03f88f9; no voters:
W20260129 02:59:32.250093 20480 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.211.4:40439, user_credentials={real_user=slave}} blocked reactor thread for 96681.1us
W20260129 02:59:32.245149 20405 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.211.3:37735, user_credentials={real_user=slave}} blocked reactor thread for 184997us
I20260129 02:59:32.123581 20441 raft_consensus.cc:2468] T e33a417cf9c54cf5af73a4ee3aac4d3a P 197b880fa39d432a9b472845876bfaf0 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 974b802141bd4f13beffef2aa03f88f9 in term 3.
I20260129 02:59:32.145478 20507 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "368953c9102643b2a21f99af36b51905" candidate_uuid: "197b880fa39d432a9b472845876bfaf0" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "974b802141bd4f13beffef2aa03f88f9" is_pre_election: true
I20260129 02:59:32.256222 20405 leader_election.cc:304] T e4d18c35b0a7493e85d0a4b92956a48f P 197b880fa39d432a9b472845876bfaf0 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 197b880fa39d432a9b472845876bfaf0, 1c50de89a5a845668696dcb28c3badd6; no voters: 974b802141bd4f13beffef2aa03f88f9
I20260129 02:59:32.258853 20405 leader_election.cc:304] T 2e58e9a5a8aa4cecafb464addb4f35ed P 197b880fa39d432a9b472845876bfaf0 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 197b880fa39d432a9b472845876bfaf0, 1c50de89a5a845668696dcb28c3badd6; no voters: 7a9daf5d42a84615a6383cfec9a10608
I20260129 02:59:32.180548 20582 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "913fc7d751ed4b7388d889aac30a5f9b" candidate_uuid: "974b802141bd4f13beffef2aa03f88f9" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1c50de89a5a845668696dcb28c3badd6" is_pre_election: true
I20260129 02:59:32.168249 21133 raft_consensus.cc:2468] T 505580c93c694bbb8ea96fdf9b17ab46 P 7a9daf5d42a84615a6383cfec9a10608 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1c50de89a5a845668696dcb28c3badd6 in term 3.
W20260129 02:59:32.143625 20556 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.211.2:34517, user_credentials={real_user=slave}} blocked reactor thread for 111947us
I20260129 02:59:32.264318 20582 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "add8d4923d1a489e86882c3a25692b95" candidate_uuid: "974b802141bd4f13beffef2aa03f88f9" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "1c50de89a5a845668696dcb28c3badd6" is_pre_election: true
I20260129 02:59:32.265190 21128 raft_consensus.cc:1240] T 505580c93c694bbb8ea96fdf9b17ab46 P 7a9daf5d42a84615a6383cfec9a10608 [term 3 FOLLOWER]: Rejecting Update request from peer 974b802141bd4f13beffef2aa03f88f9 for earlier term 2. Current term is 3. Ops: []
I20260129 02:59:32.266396 20582 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0340229d521b432580ae926f6062e3f7" candidate_uuid: "974b802141bd4f13beffef2aa03f88f9" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "1c50de89a5a845668696dcb28c3badd6" is_pre_election: true
I20260129 02:59:32.183136 21883 leader_election.cc:290] T fcddab6bca684adf8ea032289a00f90e P 197b880fa39d432a9b472845876bfaf0 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 974b802141bd4f13beffef2aa03f88f9 (127.18.211.2:34517), 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3:37735)
I20260129 02:59:32.268297 20582 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4b1876f9a70b49edb3aa74105e66d3b3" candidate_uuid: "974b802141bd4f13beffef2aa03f88f9" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "1c50de89a5a845668696dcb28c3badd6" is_pre_election: true
I20260129 02:59:32.151468 20586 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3906b080a5b34a74b8b64128635e8997" candidate_uuid: "974b802141bd4f13beffef2aa03f88f9" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1c50de89a5a845668696dcb28c3badd6" is_pre_election: true
W20260129 02:59:32.155987 21102 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.211.2:34517, user_credentials={real_user=slave}} blocked reactor thread for 172537us
I20260129 02:59:32.256407 20507 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "17d6431cc77a491e8464d68db4081a51" candidate_uuid: "197b880fa39d432a9b472845876bfaf0" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "974b802141bd4f13beffef2aa03f88f9" is_pre_election: true
W20260129 02:59:32.156387 20515 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.643s user 0.000s sys 0.007s
W20260129 02:59:32.272608 20515 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.761s user 0.000s sys 0.008s
I20260129 02:59:32.192188 22036 raft_consensus.cc:515] T add8d4923d1a489e86882c3a25692b95 P 974b802141bd4f13beffef2aa03f88f9 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } }
I20260129 02:59:32.274982 22082 raft_consensus.cc:2804] T e4d18c35b0a7493e85d0a4b92956a48f P 197b880fa39d432a9b472845876bfaf0 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260129 02:59:32.275285 21129 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "febffe4a8b394368b69b1d164308fb9a" candidate_uuid: "974b802141bd4f13beffef2aa03f88f9" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "7a9daf5d42a84615a6383cfec9a10608" is_pre_election: true
W20260129 02:59:32.245247 20407 service_pool.cc:137] UpdateConsensus request on kudu.consensus.ConsensusService from 127.0.0.1:48356 dropped due to backpressure: service queue is full with 50 items
I20260129 02:59:32.275620 22082 raft_consensus.cc:493] T e4d18c35b0a7493e85d0a4b92956a48f P 197b880fa39d432a9b472845876bfaf0 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 974b802141bd4f13beffef2aa03f88f9)
I20260129 02:59:32.500505 20516 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a8479d1fe0bd41e5946186e3a6ca8c7a" candidate_uuid: "197b880fa39d432a9b472845876bfaf0" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "974b802141bd4f13beffef2aa03f88f9" is_pre_election: true
I20260129 02:59:32.279029 20586 raft_consensus.cc:2393] T 3906b080a5b34a74b8b64128635e8997 P 1c50de89a5a845668696dcb28c3badd6 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 974b802141bd4f13beffef2aa03f88f9 in current term 2: Already voted for candidate 1c50de89a5a845668696dcb28c3badd6 in this term.
I20260129 02:59:32.279248 21133 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3ccac82a01a2445fa5b0ce6c0fcf2295" candidate_uuid: "974b802141bd4f13beffef2aa03f88f9" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "7a9daf5d42a84615a6383cfec9a10608" is_pre_election: true
I20260129 02:59:32.501363 21129 raft_consensus.cc:2468] T febffe4a8b394368b69b1d164308fb9a P 7a9daf5d42a84615a6383cfec9a10608 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 974b802141bd4f13beffef2aa03f88f9 in term 1.
I20260129 02:59:32.462603 21135 raft_consensus.cc:2468] T 96609ac68bdf4993a12e1c23355ca013 P 7a9daf5d42a84615a6383cfec9a10608 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 974b802141bd4f13beffef2aa03f88f9 in term 3.
I20260129 02:59:32.108950 20434 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55672 (request call id 1397) took 1359 ms. Trace:
I20260129 02:59:32.455685 22112 raft_consensus.cc:2804] T a644c02b16af47679824ccd060ac10fd P 974b802141bd4f13beffef2aa03f88f9 [term 1 FOLLOWER]: Leader pre-election won for term 2
W20260129 02:59:32.188865 21991 consensus_meta.cc:287] T 4e3805ffa70242e8ade66da260dddf66 P 1c50de89a5a845668696dcb28c3badd6: Time spent flushing consensus metadata: real 0.523s user 0.012s sys 0.000s
I20260129 02:59:32.278455 21127 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4e420f90884f4796997b5ea41e49e411" candidate_uuid: "974b802141bd4f13beffef2aa03f88f9" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "7a9daf5d42a84615a6383cfec9a10608" is_pre_election: true
W20260129 02:59:32.521032 20508 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:57600 (request call id 1615) took 3662 ms (client timeout 2210 ms). Trace:
I20260129 02:59:32.188684 22165 leader_election.cc:290] T e33a417cf9c54cf5af73a4ee3aac4d3a P 197b880fa39d432a9b472845876bfaf0 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 974b802141bd4f13beffef2aa03f88f9 (127.18.211.2:34517), 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3:37735)
I20260129 02:59:32.521116 21991 raft_consensus.cc:515] T 4e3805ffa70242e8ade66da260dddf66 P 1c50de89a5a845668696dcb28c3badd6 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } }
W20260129 02:59:32.521682 20508 rpcz_store.cc:269] 0129 02:59:28.858184 (+ 0us) service_pool.cc:168] Inserting onto call queue
0129 02:59:29.260635 (+402451us) service_pool.cc:225] Handling call
0129 02:59:32.520570 (+3259935us) spinlock_profiling.cc:243] Waited 416 ms on lock 0x7b54002ee354. stack: 00007f8fe044e373 00007f8fe044e270 00007f8fdfda0574 00007f8fe9a5c6b4 00007f8fe9a5c649 00007f8fe34975a1 00007f8fe3531f7a 00007f8fe84e3647 00007f8fe1a73834 00007f8fe1a737c2 00007f8fe1a736f1 00007f8fe1a7366c 00007f8fe1a72922 00007f8fe186d34c 00007f8fe186c786 00007f8fe186c12f
0129 02:59:32.520784 (+ 214us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":956001152}
I20260129 02:59:32.524955 22036 leader_election.cc:290] T add8d4923d1a489e86882c3a25692b95 P 974b802141bd4f13beffef2aa03f88f9 [CANDIDATE]: Term 2 election: Requested vote from peers 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3:37735), 7a9daf5d42a84615a6383cfec9a10608 (127.18.211.4:40439)
I20260129 02:59:32.519701 20556 leader_election.cc:304] T 4408006eb5174de1896b037cf3f18457 P 1c50de89a5a845668696dcb28c3badd6 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1c50de89a5a845668696dcb28c3badd6; no voters: 197b880fa39d432a9b472845876bfaf0, 974b802141bd4f13beffef2aa03f88f9
I20260129 02:59:32.526482 22112 raft_consensus.cc:493] T a644c02b16af47679824ccd060ac10fd P 974b802141bd4f13beffef2aa03f88f9 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 1c50de89a5a845668696dcb28c3badd6)
I20260129 02:59:32.527227 22112 raft_consensus.cc:3060] T a644c02b16af47679824ccd060ac10fd P 974b802141bd4f13beffef2aa03f88f9 [term 1 FOLLOWER]: Advancing to term 2
I20260129 02:59:32.163152 22157 leader_election.cc:290] T 3ccac82a01a2445fa5b0ce6c0fcf2295 P 974b802141bd4f13beffef2aa03f88f9 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 197b880fa39d432a9b472845876bfaf0 (127.18.211.1:42027), 7a9daf5d42a84615a6383cfec9a10608 (127.18.211.4:40439)
I20260129 02:59:32.119987 22102 raft_consensus.cc:2804] T c2633a6fc34c44579de348d208b0c01e P 1c50de89a5a845668696dcb28c3badd6 [term 1 FOLLOWER]: Leader pre-election won for term 2
W20260129 02:59:32.201197 22115 consensus_meta.cc:287] T 1299723a4476420ab02fd8bfd7aa0645 P 1c50de89a5a845668696dcb28c3badd6: Time spent flushing consensus metadata: real 1.431s user 0.013s sys 0.000s
I20260129 02:59:32.160014 22131 leader_election.cc:290] T 913fc7d751ed4b7388d889aac30a5f9b P 1c50de89a5a845668696dcb28c3badd6 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 197b880fa39d432a9b472845876bfaf0 (127.18.211.1:42027), 974b802141bd4f13beffef2aa03f88f9 (127.18.211.2:34517)
W20260129 02:59:32.527137 20556 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.211.2:34517, user_credentials={real_user=slave}} blocked reactor thread for 260007us
W20260129 02:59:32.143219 22241 log.cc:927] Time spent T b189f08e16a74a11980cc55d99359a0a P 197b880fa39d432a9b472845876bfaf0: Append to log took a long time: real 0.065s user 0.000s sys 0.000s
I20260129 02:59:32.530845 21133 raft_consensus.cc:2468] T 3ccac82a01a2445fa5b0ce6c0fcf2295 P 7a9daf5d42a84615a6383cfec9a10608 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 974b802141bd4f13beffef2aa03f88f9 in term 1.
W20260129 02:59:32.201285 22103 consensus_meta.cc:287] T cb0b3f259ebc4310927338a8740e87b8 P 1c50de89a5a845668696dcb28c3badd6: Time spent flushing consensus metadata: real 2.204s user 0.014s sys 0.001s
I20260129 02:59:32.509851 22082 raft_consensus.cc:3060] T e4d18c35b0a7493e85d0a4b92956a48f P 197b880fa39d432a9b472845876bfaf0 [term 2 FOLLOWER]: Advancing to term 3
W20260129 02:59:32.140872 22238 log.cc:927] Time spent T cedce331e1584ac8b636c9b8a72b2ba0 P 1c50de89a5a845668696dcb28c3badd6: Append to log took a long time: real 0.180s user 0.001s sys 0.000s
I20260129 02:59:32.145181 20590 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a644c02b16af47679824ccd060ac10fd" candidate_uuid: "974b802141bd4f13beffef2aa03f88f9" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "1c50de89a5a845668696dcb28c3badd6" is_pre_election: true
W20260129 02:59:32.191761 22213 consensus_meta.cc:287] T 4f5a77e94fef4b16a2ba7b067c8b144a P 1c50de89a5a845668696dcb28c3badd6: Time spent flushing consensus metadata: real 0.849s user 0.004s sys 0.008s
W20260129 02:59:32.207417 22126 consensus_meta.cc:287] T 4f8f3a38f6e24ba4857875181f447442 P 1c50de89a5a845668696dcb28c3badd6: Time spent flushing consensus metadata: real 0.534s user 0.005s sys 0.003s
I20260129 02:59:32.099737 20438 rpcz_store.cc:276] 0129 02:59:30.987046 (+ 0us) service_pool.cc:168] Inserting onto call queue
0129 02:59:32.097903 (+1110857us) service_pool.cc:225] Handling call
0129 02:59:32.099222 (+ 1319us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260129 02:59:32.527806 22247 raft_consensus.cc:2749] T 4408006eb5174de1896b037cf3f18457 P 1c50de89a5a845668696dcb28c3badd6 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260129 02:59:32.110122 22184 raft_consensus.cc:493] T b189f08e16a74a11980cc55d99359a0a P 974b802141bd4f13beffef2aa03f88f9 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 197b880fa39d432a9b472845876bfaf0)
W20260129 02:59:32.544842 20511 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.510s user 0.006s sys 0.026s
I20260129 02:59:32.135075 20513 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "87d5fe48a44a408aaf03e894a67ecb2f" candidate_uuid: "197b880fa39d432a9b472845876bfaf0" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "974b802141bd4f13beffef2aa03f88f9" is_pre_election: true
I20260129 02:59:32.159994 22201 raft_consensus.cc:3060] T 00e79d8f25a34676b1d866eb2068a2b2 P 974b802141bd4f13beffef2aa03f88f9 [term 3 FOLLOWER]: Advancing to term 4
I20260129 02:59:32.147352 20514 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "cb4727386a3e48aca5d71cc6c611c197" candidate_uuid: "197b880fa39d432a9b472845876bfaf0" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "974b802141bd4f13beffef2aa03f88f9" is_pre_election: true
I20260129 02:59:32.528817 22102 raft_consensus.cc:493] T c2633a6fc34c44579de348d208b0c01e P 1c50de89a5a845668696dcb28c3badd6 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 197b880fa39d432a9b472845876bfaf0)
I20260129 02:59:32.546064 22143 raft_consensus.cc:2804] T 2e58e9a5a8aa4cecafb464addb4f35ed P 197b880fa39d432a9b472845876bfaf0 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260129 02:59:32.154446 22257 raft_consensus.cc:493] T a644c02b16af47679824ccd060ac10fd P 7a9daf5d42a84615a6383cfec9a10608 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 1c50de89a5a845668696dcb28c3badd6)
W20260129 02:59:32.530936 21099 leader_election.cc:336] T 1299723a4476420ab02fd8bfd7aa0645 P 7a9daf5d42a84615a6383cfec9a10608 [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer 197b880fa39d432a9b472845876bfaf0 (127.18.211.1:42027): Timed out: RequestConsensusVote RPC to 127.18.211.1:42027 timed out after 1.616s (SENT)
W20260129 02:59:32.554751 22264 log.cc:927] Time spent T cedce331e1584ac8b636c9b8a72b2ba0 P 974b802141bd4f13beffef2aa03f88f9: Append to log took a long time: real 0.340s user 0.000s sys 0.002s
I20260129 02:59:32.561565 20511 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57584 (request call id 1161) took 1505 ms. Trace:
I20260129 02:59:32.562924 21132 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34060 (request call id 426) took 2606 ms. Trace:
I20260129 02:59:32.565969 20516 raft_consensus.cc:1240] T 76f713c3331540d2a5b76164103e9eb1 P 974b802141bd4f13beffef2aa03f88f9 [term 3 FOLLOWER]: Rejecting Update request from peer 1c50de89a5a845668696dcb28c3badd6 for earlier term 2. Current term is 3. Ops: []
W20260129 02:59:32.559502 22222 consensus_meta.cc:287] T 75f45538dc704d9a8ec4ff8ad9f3ea5b P 7a9daf5d42a84615a6383cfec9a10608: Time spent flushing consensus metadata: real 0.806s user 0.009s sys 0.012s
I20260129 02:59:32.546375 22197 raft_consensus.cc:2749] T 87d5fe48a44a408aaf03e894a67ecb2f P 974b802141bd4f13beffef2aa03f88f9 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260129 02:59:32.569490 21102 leader_election.cc:304] T add8d4923d1a489e86882c3a25692b95 P 7a9daf5d42a84615a6383cfec9a10608 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7a9daf5d42a84615a6383cfec9a10608; no voters: 1c50de89a5a845668696dcb28c3badd6, 974b802141bd4f13beffef2aa03f88f9
I20260129 02:59:32.570945 22255 raft_consensus.cc:2749] T add8d4923d1a489e86882c3a25692b95 P 7a9daf5d42a84615a6383cfec9a10608 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260129 02:59:32.571070 20590 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e33a417cf9c54cf5af73a4ee3aac4d3a" candidate_uuid: "974b802141bd4f13beffef2aa03f88f9" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "1c50de89a5a845668696dcb28c3badd6" is_pre_election: true
I20260129 02:59:32.566387 21132 rpcz_store.cc:276] 0129 02:59:29.956050 (+ 0us) service_pool.cc:168] Inserting onto call queue
0129 02:59:29.957837 (+ 1787us) service_pool.cc:225] Handling call
0129 02:59:30.129630 (+171793us) raft_consensus.cc:1471] Updating replica for 2 ops
0129 02:59:30.227254 (+ 97624us) raft_consensus.cc:1534] Early marking committed up to index 0
0129 02:59:30.227291 (+ 37us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0129 02:59:30.322479 (+ 95188us) log.cc:844] Serialized 520 byte log entry
0129 02:59:30.728109 (+405630us) raft_consensus.cc:1645] Marking committed up to 4
0129 02:59:30.741203 (+ 13094us) raft_consensus.cc:1697] Filling consensus response to leader.
0129 02:59:30.744313 (+ 3110us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0129 02:59:31.552785 (+808472us) raft_consensus.cc:1685] finished
0129 02:59:31.552822 (+ 37us) raft_consensus.cc:1691] UpdateReplicas() finished
0129 02:59:32.562313 (+1009491us) spinlock_profiling.cc:243] Waited 147 ms on lock 0x7b5400730350. stack: 00007f8fe044e373 00007f8fe044e270 00007f8fdfda0574 00007f8fe9a5c6b4 00007f8fe9a5c649 00007f8fe34975a1 00007f8fe352d52d 00007f8fe84e1f49 00007f8fe1a6ed34 00007f8fe1a6ecc2 00007f8fe1a6ebf1 00007f8fe1a6eb6c 00007f8fe1a6de22 00007f8fe186d34c 00007f8fe186c786 00007f8fe186c12f
0129 02:59:32.562680 (+ 367us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":106988,"raft.queue_time_us":596,"raft.run_cpu_time_us":209,"raft.run_wall_time_us":134266,"spinlock_wait_cycles":339004672,"thread_start_us":395455,"threads_started":1,"wal-append.queue_time_us":526424}
I20260129 02:59:32.572413 20582 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "76f713c3331540d2a5b76164103e9eb1" candidate_uuid: "974b802141bd4f13beffef2aa03f88f9" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "1c50de89a5a845668696dcb28c3badd6" is_pre_election: true
I20260129 02:59:32.552234 22143 raft_consensus.cc:493] T 2e58e9a5a8aa4cecafb464addb4f35ed P 197b880fa39d432a9b472845876bfaf0 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 7a9daf5d42a84615a6383cfec9a10608)
I20260129 02:59:32.573406 22143 raft_consensus.cc:3060] T 2e58e9a5a8aa4cecafb464addb4f35ed P 197b880fa39d432a9b472845876bfaf0 [term 2 FOLLOWER]: Advancing to term 3
I20260129 02:59:32.574159 20582 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7a9a69cd82724b1faed1d10aa091fc4a" candidate_uuid: "974b802141bd4f13beffef2aa03f88f9" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "1c50de89a5a845668696dcb28c3badd6" is_pre_election: true
I20260129 02:59:32.174307 22237 leader_election.cc:290] T dc2f4cb752cf4d59b5f6bc689f5d8104 P 197b880fa39d432a9b472845876bfaf0 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3:37735), 7a9daf5d42a84615a6383cfec9a10608 (127.18.211.4:40439)
I20260129 02:59:32.577523 20590 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "96609ac68bdf4993a12e1c23355ca013" candidate_uuid: "974b802141bd4f13beffef2aa03f88f9" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "1c50de89a5a845668696dcb28c3badd6"
I20260129 02:59:32.575654 20582 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a5a05fbcbbcf4916bbc01fd1a5b5e90d" candidate_uuid: "974b802141bd4f13beffef2aa03f88f9" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "1c50de89a5a845668696dcb28c3badd6"
I20260129 02:59:32.580677 20590 raft_consensus.cc:2393] T 96609ac68bdf4993a12e1c23355ca013 P 1c50de89a5a845668696dcb28c3badd6 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 974b802141bd4f13beffef2aa03f88f9 in current term 3: Already voted for candidate 1c50de89a5a845668696dcb28c3badd6 in this term.
I20260129 02:59:32.557536 22184 raft_consensus.cc:515] T b189f08e16a74a11980cc55d99359a0a P 974b802141bd4f13beffef2aa03f88f9 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:59:32.583211 20510 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:48544 (request call id 108) took 2465 ms. Trace:
I20260129 02:59:32.583719 20510 rpcz_store.cc:276] 0129 02:59:30.116744 (+ 0us) service_pool.cc:168] Inserting onto call queue
0129 02:59:30.316442 (+199698us) service_pool.cc:225] Handling call
0129 02:59:30.316837 (+ 395us) raft_consensus.cc:1471] Updating replica for 1 ops
0129 02:59:30.356294 (+ 39457us) raft_consensus.cc:1534] Early marking committed up to index 0
0129 02:59:30.356339 (+ 45us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0129 02:59:30.356839 (+ 500us) log.cc:844] Serialized 26 byte log entry
0129 02:59:32.262891 (+1906052us) spinlock_profiling.cc:243] Waited 15.8 ms on lock 0x7b18001c92a0. stack: 00007f8fe044e373 00007f8fe044e270 00007f8fdfda0574 00007f8fe9a5c6b4 00007f8fe9a5c649 00007f8fe92d4c91 00007f8fe92e1671 00007f8fe0462c67 00007f8fe924b709 00007f8fe047f57b 00007f8fe047d81f 00007f8fe047fae2 00007f8fe30c75ab 00007f8fe30cd0c3 00007f8fe30cd471 00007f8fe34e973f
0129 02:59:32.544906 (+282015us) raft_consensus.cc:1645] Marking committed up to 5
0129 02:59:32.544985 (+ 79us) raft_consensus.cc:1697] Filling consensus response to leader.
0129 02:59:32.545121 (+ 136us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0129 02:59:32.578790 (+ 33669us) raft_consensus.cc:1685] finished
0129 02:59:32.578824 (+ 34us) raft_consensus.cc:1691] UpdateReplicas() finished
0129 02:59:32.582387 (+ 3563us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":38952,"raft.queue_time_us":39081,"raft.run_cpu_time_us":195,"raft.run_wall_time_us":109086,"spinlock_wait_cycles":36238080,"thread_start_us":224680,"threads_started":1,"wal-append.queue_time_us":2219047}
W20260129 02:59:32.598941 20556 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.211.2:34517, user_credentials={real_user=slave}} blocked reactor thread for 65704us
I20260129 02:59:32.608464 20582 tablet_service.cc:2038] Received Run Leader Election RPC: tablet_id: "36ff33be666a4c82954731691dbcf327"
dest_uuid: "1c50de89a5a845668696dcb28c3badd6"
from {username='slave'} at 127.0.0.1:35752
W20260129 02:59:32.581601 21099 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.211.1:42027, user_credentials={real_user=slave}} blocked reactor thread for 50874us
I20260129 02:59:32.099752 21100 leader_election.cc:304] T 4f5a77e94fef4b16a2ba7b067c8b144a P 7a9daf5d42a84615a6383cfec9a10608 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1c50de89a5a845668696dcb28c3badd6, 7a9daf5d42a84615a6383cfec9a10608; no voters:
I20260129 02:59:32.612701 21102 leader_election.cc:304] T 75b0b6e82ddc4f92b1fe1d94f9a0b73b P 7a9daf5d42a84615a6383cfec9a10608 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7a9daf5d42a84615a6383cfec9a10608, 974b802141bd4f13beffef2aa03f88f9; no voters:
I20260129 02:59:32.604494 21131 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34060 (request call id 431) took 1724 ms. Trace:
I20260129 02:59:33.130435 21131 rpcz_store.cc:276] 0129 02:59:30.879515 (+ 0us) service_pool.cc:168] Inserting onto call queue
0129 02:59:30.896863 (+ 17348us) service_pool.cc:225] Handling call
0129 02:59:32.581488 (+1684625us) spinlock_profiling.cc:243] Waited 6.15 ms on lock 0x7b5800111f98. stack: 00007f8fe044e373 00007f8fe044e270 00007f8fdfda0574 00007f8fe9a5c6b4 00007f8fe9a5c649 00007f8fe02c08b1 00007f8fe0442db1 00007f8fe0442c62 00007f8fe92eddd9 00007f8fe846d260 00007f8fe8535dbb 00007f8fe8535e41 00007f8fe84f7af0 00007f8fe84e2459 00007f8fe84e1f02 00007f8fe1a6ed34
0129 02:59:32.603427 (+ 21939us) raft_consensus.cc:1471] Updating replica for 0 ops
0129 02:59:32.603942 (+ 515us) raft_consensus.cc:1534] Early marking committed up to index 0
0129 02:59:32.603968 (+ 26us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0129 02:59:32.604027 (+ 59us) raft_consensus.cc:1645] Marking committed up to 6
0129 02:59:32.604088 (+ 61us) raft_consensus.cc:1697] Filling consensus response to leader.
0129 02:59:32.604162 (+ 74us) raft_consensus.cc:1691] UpdateReplicas() finished
0129 02:59:32.604307 (+ 145us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":14153088}
I20260129 02:59:32.595916 21991 leader_election.cc:290] T 4e3805ffa70242e8ade66da260dddf66 P 1c50de89a5a845668696dcb28c3badd6 [CANDIDATE]: Term 3 election: Requested vote from peers 197b880fa39d432a9b472845876bfaf0 (127.18.211.1:42027), 7a9daf5d42a84615a6383cfec9a10608 (127.18.211.4:40439)
I20260129 02:59:32.119997 22069 consensus_queue.cc:237] T 76f713c3331540d2a5b76164103e9eb1 P 197b880fa39d432a9b472845876bfaf0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
W20260129 02:59:32.621070 22079 consensus_meta.cc:287] T 0f20af7fbe5d4db89a6f17948d98912c P 1c50de89a5a845668696dcb28c3badd6: Time spent flushing consensus metadata: real 0.992s user 0.011s sys 0.016s
I20260129 02:59:33.157529 22079 raft_consensus.cc:515] T 0f20af7fbe5d4db89a6f17948d98912c P 1c50de89a5a845668696dcb28c3badd6 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:59:32.628468 22082 raft_consensus.cc:515] T e4d18c35b0a7493e85d0a4b92956a48f P 197b880fa39d432a9b472845876bfaf0 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:59:33.162523 22082 leader_election.cc:290] T e4d18c35b0a7493e85d0a4b92956a48f P 197b880fa39d432a9b472845876bfaf0 [CANDIDATE]: Term 3 election: Requested vote from peers 974b802141bd4f13beffef2aa03f88f9 (127.18.211.2:34517), 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3:37735)
I20260129 02:59:32.567092 22102 raft_consensus.cc:3060] T c2633a6fc34c44579de348d208b0c01e P 1c50de89a5a845668696dcb28c3badd6 [term 1 FOLLOWER]: Advancing to term 2
I20260129 02:59:32.596715 22103 raft_consensus.cc:515] T cb0b3f259ebc4310927338a8740e87b8 P 1c50de89a5a845668696dcb28c3badd6 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } }
I20260129 02:59:33.166658 22103 leader_election.cc:290] T cb0b3f259ebc4310927338a8740e87b8 P 1c50de89a5a845668696dcb28c3badd6 [CANDIDATE]: Term 4 election: Requested vote from peers 974b802141bd4f13beffef2aa03f88f9 (127.18.211.2:34517), 7a9daf5d42a84615a6383cfec9a10608 (127.18.211.4:40439)
I20260129 02:59:32.626178 22112 raft_consensus.cc:515] T a644c02b16af47679824ccd060ac10fd P 974b802141bd4f13beffef2aa03f88f9 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } }
I20260129 02:59:33.169661 22112 leader_election.cc:290] T a644c02b16af47679824ccd060ac10fd P 974b802141bd4f13beffef2aa03f88f9 [CANDIDATE]: Term 2 election: Requested vote from peers 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3:37735), 7a9daf5d42a84615a6383cfec9a10608 (127.18.211.4:40439)
I20260129 02:59:33.170423 22112 raft_consensus.cc:493] T a0fcc58d6fb6473cafbd9d34d5195269 P 974b802141bd4f13beffef2aa03f88f9 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 197b880fa39d432a9b472845876bfaf0)
I20260129 02:59:32.196866 22160 leader_election.cc:290] T febffe4a8b394368b69b1d164308fb9a P 974b802141bd4f13beffef2aa03f88f9 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 197b880fa39d432a9b472845876bfaf0 (127.18.211.1:42027), 7a9daf5d42a84615a6383cfec9a10608 (127.18.211.4:40439)
I20260129 02:59:32.585605 22184 leader_election.cc:290] T b189f08e16a74a11980cc55d99359a0a P 974b802141bd4f13beffef2aa03f88f9 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 197b880fa39d432a9b472845876bfaf0 (127.18.211.1:42027), 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3:37735)
W20260129 02:59:32.600598 22192 log.cc:927] Time spent T cedce331e1584ac8b636c9b8a72b2ba0 P 197b880fa39d432a9b472845876bfaf0: Append to log took a long time: real 3.023s user 0.025s sys 0.011s
I20260129 02:59:32.529742 22115 raft_consensus.cc:515] T 1299723a4476420ab02fd8bfd7aa0645 P 1c50de89a5a845668696dcb28c3badd6 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } }
I20260129 02:59:33.180521 20436 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55676 (request call id 1048) took 5657 ms. Trace:
I20260129 02:59:33.180976 20436 rpcz_store.cc:276] 0129 02:59:27.523040 (+ 0us) service_pool.cc:168] Inserting onto call queue
0129 02:59:27.525496 (+ 2456us) service_pool.cc:225] Handling call
0129 02:59:27.552883 (+ 27387us) raft_consensus.cc:1471] Updating replica for 1 ops
0129 02:59:27.611006 (+ 58123us) raft_consensus.cc:1534] Early marking committed up to index 2
0129 02:59:28.788289 (+1177283us) log.cc:844] Serialized 14 byte log entry
0129 02:59:29.729059 (+940770us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0129 02:59:29.814711 (+ 85652us) log.cc:844] Serialized 26 byte log entry
0129 02:59:29.850161 (+ 35450us) raft_consensus.cc:1645] Marking committed up to 2
0129 02:59:29.852014 (+ 1853us) raft_consensus.cc:1697] Filling consensus response to leader.
0129 02:59:29.855939 (+ 3925us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0129 02:59:32.599529 (+2743590us) spinlock_profiling.cc:243] Waited 92.8 ms on lock 0x7b440002304c. stack: 00007f8fe044e373 00007f8fe044e270 00007f8fdfda0574 00007f8fe9a5c6b4 00007f8fe9a5c649 00007f8fe02c08b1 00007f8fe038f005 00007f8fe038fa22 00007f8fe30d957b 00007f8fe30d4e4f 00007f8fe30c8c0e 00007f8fe30c7f56 00007f8fe30c7b78 00007f8fe30d15c1 00007f8fe30d1579 00007f8fe30d1509
0129 02:59:33.179955 (+580426us) raft_consensus.cc:1685] finished
0129 02:59:33.180098 (+ 143us) raft_consensus.cc:1691] UpdateReplicas() finished
0129 02:59:33.180342 (+ 244us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":31856,"prepare.queue_time_us":959288,"prepare.run_cpu_time_us":35,"prepare.run_wall_time_us":36,"raft.queue_time_us":21694,"raft.run_cpu_time_us":9369,"raft.run_wall_time_us":69379,"spinlock_wait_cycles":213475072,"thread_start_us":1890323,"threads_started":2,"wal-append.queue_time_us":741869}
W20260129 02:59:32.604302 22200 log.cc:927] Time spent T 45cc5f3d2688486d9b6dbe7e666dedde P 974b802141bd4f13beffef2aa03f88f9: Append to log took a long time: real 2.499s user 0.008s sys 0.012s
I20260129 02:59:32.597761 22126 raft_consensus.cc:515] T 4f8f3a38f6e24ba4857875181f447442 P 1c50de89a5a845668696dcb28c3badd6 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:59:32.080008 22190 raft_consensus.cc:515] T 4408006eb5174de1896b037cf3f18457 P 197b880fa39d432a9b472845876bfaf0 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:59:32.211589 22164 raft_consensus.cc:515] T 724a236524cb4e7ba3428880681d6cc2 P 1c50de89a5a845668696dcb28c3badd6 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:59:32.637475 22213 raft_consensus.cc:515] T 4f5a77e94fef4b16a2ba7b067c8b144a P 1c50de89a5a845668696dcb28c3badd6 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } }
I20260129 02:59:32.582512 22222 raft_consensus.cc:515] T 75f45538dc704d9a8ec4ff8ad9f3ea5b P 7a9daf5d42a84615a6383cfec9a10608 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } }
I20260129 02:59:32.196406 22233 leader_election.cc:290] T a5a05fbcbbcf4916bbc01fd1a5b5e90d P 197b880fa39d432a9b472845876bfaf0 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 974b802141bd4f13beffef2aa03f88f9 (127.18.211.2:34517), 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3:37735)
I20260129 02:59:32.624009 22257 raft_consensus.cc:515] T a644c02b16af47679824ccd060ac10fd P 7a9daf5d42a84615a6383cfec9a10608 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } }
I20260129 02:59:32.154685 22254 raft_consensus.cc:2749] T 3e2d15f8640b407195210c476e09df91 P 7a9daf5d42a84615a6383cfec9a10608 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20260129 02:59:32.736740 22206 log.cc:927] Time spent T 00000000000000000000000000000000 P a91e6696d0a44ee19bcf2489fd709c38: Append to log took a long time: real 3.132s user 0.000s sys 0.042s
I20260129 02:59:32.748085 20433 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c891d2bff4ce4568861bbc992e91fbb0" candidate_uuid: "1c50de89a5a845668696dcb28c3badd6" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "197b880fa39d432a9b472845876bfaf0" is_pre_election: true
I20260129 02:59:32.814493 21127 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4e3805ffa70242e8ade66da260dddf66" candidate_uuid: "1c50de89a5a845668696dcb28c3badd6" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "7a9daf5d42a84615a6383cfec9a10608"
I20260129 02:59:32.814929 20554 leader_election.cc:304] T 505580c93c694bbb8ea96fdf9b17ab46 P 1c50de89a5a845668696dcb28c3badd6 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1c50de89a5a845668696dcb28c3badd6, 7a9daf5d42a84615a6383cfec9a10608; no voters:
I20260129 02:59:33.186946 22126 leader_election.cc:290] T 4f8f3a38f6e24ba4857875181f447442 P 1c50de89a5a845668696dcb28c3badd6 [CANDIDATE]: Term 4 election: Requested vote from peers 197b880fa39d432a9b472845876bfaf0 (127.18.211.1:42027), 974b802141bd4f13beffef2aa03f88f9 (127.18.211.2:34517)
W20260129 02:59:33.202826 20554 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.211.4:40439, user_credentials={real_user=slave}} blocked reactor thread for 388043us
I20260129 02:59:33.203281 22126 raft_consensus.cc:2804] T 505580c93c694bbb8ea96fdf9b17ab46 P 1c50de89a5a845668696dcb28c3badd6 [term 3 FOLLOWER]: Leader election won for term 3
I20260129 02:59:33.203876 22126 raft_consensus.cc:697] T 505580c93c694bbb8ea96fdf9b17ab46 P 1c50de89a5a845668696dcb28c3badd6 [term 3 LEADER]: Becoming Leader. State: Replica: 1c50de89a5a845668696dcb28c3badd6, State: Running, Role: LEADER
I20260129 02:59:33.189013 20583 raft_consensus.cc:2393] T 4f8f3a38f6e24ba4857875181f447442 P 1c50de89a5a845668696dcb28c3badd6 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 197b880fa39d432a9b472845876bfaf0 in current term 4: Already voted for candidate 1c50de89a5a845668696dcb28c3badd6 in this term.
I20260129 02:59:32.846871 20509 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57584 (request call id 1160) took 1577 ms. Trace:
I20260129 02:59:32.862170 22201 raft_consensus.cc:515] T 00e79d8f25a34676b1d866eb2068a2b2 P 974b802141bd4f13beffef2aa03f88f9 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } }
I20260129 02:59:33.204950 22126 consensus_queue.cc:237] T 505580c93c694bbb8ea96fdf9b17ab46 P 1c50de89a5a845668696dcb28c3badd6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } }
I20260129 02:59:33.206650 22201 leader_election.cc:290] T 00e79d8f25a34676b1d866eb2068a2b2 P 974b802141bd4f13beffef2aa03f88f9 [CANDIDATE]: Term 4 election: Requested vote from peers 197b880fa39d432a9b472845876bfaf0 (127.18.211.1:42027), 7a9daf5d42a84615a6383cfec9a10608 (127.18.211.4:40439)
I20260129 02:59:32.319217 20407 service_pool.cc:138] UpdateConsensus request on kudu.consensus.ConsensusService from 127.0.0.1:48356 dropped due to backpressure: service queue is full with 50 items Contents of service queue:
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55672 (request call id 1400)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:48356 (request call id 213)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55672 (request call id 1401)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55676 (request call id 1088)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55676 (request call id 1087)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55676 (request call id 1084)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55676 (request call id 1092)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55676 (request call id 1095)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55676 (request call id 1096)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55676 (request call id 1093)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:48356 (request call id 220)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55676 (request call id 1099)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:48356 (request call id 218)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55672 (request call id 1410)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55676 (request call id 1100)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:48356 (request call id 216)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:48356 (request call id 219)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55672 (request call id 1407)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55672 (request call id 1406)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55676 (request call id 1098)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55672 (request call id 1408)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55672 (request call id 1409)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55672 (request call id 1399)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55676 (request call id 1071)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:48356 (request call id 205)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:48356 (request call id 206)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55672 (request call id 1383)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55672 (request call id 1384)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55672 (request call id 1388)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55672 (request call id 1391)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55672 (request call id 1393)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:48356 (request call id 211)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55672 (request call id 1394)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:48356 (request call id 212)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55672 (request call id 1395)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:48356 (request call id 214)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55676 (request call id 1080)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55676 (request call id 1081)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55676 (request call id 1082)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55676 (request call id 1083)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55676 (request call id 1085)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55672 (request call id 1403)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55676 (request call id 1094)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55676 (request call id 1097)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55672 (request call id 1404)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55672 (request call id 1405)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:48356 (request call id 215)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:48356 (request call id 217)
I20260129 02:59:33.207808 20508 raft_consensus.cc:1240] T 00e79d8f25a34676b1d866eb2068a2b2 P 974b802141bd4f13beffef2aa03f88f9 [term 4 FOLLOWER]: Rejecting Update request from peer 197b880fa39d432a9b472845876bfaf0 for earlier term 3. Current term is 4. Ops: []
I20260129 02:59:33.208503 22257 leader_election.cc:290] T a644c02b16af47679824ccd060ac10fd P 7a9daf5d42a84615a6383cfec9a10608 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 974b802141bd4f13beffef2aa03f88f9 (127.18.211.2:34517), 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3:37735)
I20260129 02:59:33.209120 20508 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57600 (request call id 1632) took 1336 ms. Trace:
I20260129 02:59:33.209506 20508 rpcz_store.cc:276] 0129 02:59:31.873080 (+ 0us) service_pool.cc:168] Inserting onto call queue
0129 02:59:32.523262 (+650182us) service_pool.cc:225] Handling call
0129 02:59:32.537577 (+ 14315us) raft_consensus.cc:1471] Updating replica for 0 ops
0129 02:59:33.208323 (+670746us) raft_consensus.cc:1697] Filling consensus response to leader.
0129 02:59:33.208857 (+ 534us) spinlock_profiling.cc:243] Waited 670 ms on lock 0x7b54002efc54. stack: 00007f8fe044e373 00007f8fe044e270 00007f8fdfda0574 00007f8fe9a5c6b4 00007f8fe9a5c649 00007f8fe34975a1 00007f8fe352ea81 00007f8fe352d4be 00007f8fe84e1f49 00007f8fe1a6ed34 00007f8fe1a6ecc2 00007f8fe1a6ebf1 00007f8fe1a6eb6c 00007f8fe1a6de22 00007f8fe186d34c 00007f8fe186c786
0129 02:59:33.209048 (+ 191us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1540858624}
I20260129 02:59:32.244603 20404 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.GetConsensusState from 127.0.0.1:55666 (request call id 1047) took 2538 ms. Trace:
I20260129 02:59:33.212030 20404 rpcz_store.cc:276] 0129 02:59:29.706237 (+ 0us) service_pool.cc:168] Inserting onto call queue
0129 02:59:32.244393 (+2538156us) inbound_call.cc:173] Queueing failure response
Metrics: {}
I20260129 02:59:32.905269 20515 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57584 (request call id 1134) took 3796 ms. Trace:
W20260129 02:59:32.908046 22194 log.cc:927] Time spent T 4b1876f9a70b49edb3aa74105e66d3b3 P 197b880fa39d432a9b472845876bfaf0: Append to log took a long time: real 3.332s user 0.009s sys 0.031s
W20260129 02:59:33.213769 20583 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:35754 (request call id 1521) took 1791 ms (client timeout 2084 ms). Trace:
W20260129 02:59:32.946205 20439 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55676 (request call id 1079) took 3220 ms (client timeout 1652 ms). Trace:
W20260129 02:59:32.948027 20435 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55672 (request call id 1400) took 2083 ms (client timeout 1980 ms). Trace:
I20260129 02:59:32.974262 20514 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:57600 (request call id 1633) took 1098 ms. Trace:
I20260129 02:59:32.562198 20511 rpcz_store.cc:276] 0129 02:59:31.055882 (+ 0us) service_pool.cc:168] Inserting onto call queue
0129 02:59:31.531833 (+475951us) service_pool.cc:225] Handling call
0129 02:59:31.537025 (+ 5192us) raft_consensus.cc:1471] Updating replica for 0 ops
0129 02:59:31.537228 (+ 203us) raft_consensus.cc:1534] Early marking committed up to index 3
0129 02:59:32.002336 (+465108us) log.cc:844] Serialized 14 byte log entry
0129 02:59:32.560768 (+558432us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0129 02:59:32.560906 (+ 138us) raft_consensus.cc:1645] Marking committed up to 3
0129 02:59:32.561046 (+ 140us) raft_consensus.cc:1697] Filling consensus response to leader.
0129 02:59:32.561194 (+ 148us) raft_consensus.cc:1691] UpdateReplicas() finished
0129 02:59:32.561454 (+ 260us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":17472,"prepare.run_cpu_time_us":38,"prepare.run_wall_time_us":38,"thread_start_us":989788,"threads_started":2,"wal-append.queue_time_us":211842}
I20260129 02:59:32.998853 20432 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "96609ac68bdf4993a12e1c23355ca013" candidate_uuid: "1c50de89a5a845668696dcb28c3badd6" candidate_term: 4 candidate_status { last_received { term: 2 index: 3 } } ignore_live_leader: false dest_uuid: "197b880fa39d432a9b472845876bfaf0" is_pre_election: true
W20260129 02:59:33.002552 20553 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.18.211.1:42027 timed out after 1.701s (SENT) [suppressed 31 similar messages]
W20260129 02:59:33.003518 20437 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:48356 (request call id 209) took 3273 ms (client timeout 2178 ms). Trace:
I20260129 02:59:33.011185 21128 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:47844 (request call id 266) took 1137 ms. Trace:
W20260129 02:59:33.030191 21130 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34060 (request call id 423) took 4193 ms (client timeout 2709 ms). Trace:
I20260129 02:59:33.218580 20585 raft_consensus.cc:1240] T 4f8f3a38f6e24ba4857875181f447442 P 1c50de89a5a845668696dcb28c3badd6 [term 4 FOLLOWER]: Rejecting Update request from peer 974b802141bd4f13beffef2aa03f88f9 for earlier term 3. Current term is 4. Ops: []
W20260129 02:59:33.218370 21130 rpcz_store.cc:269] 0129 02:59:28.837203 (+ 0us) service_pool.cc:168] Inserting onto call queue
0129 02:59:28.838669 (+ 1466us) service_pool.cc:225] Handling call
0129 02:59:33.029699 (+4191030us) spinlock_profiling.cc:243] Waited 410 ms on lock 0x7b54003b3dd4. stack: 00007f8fe044e373 00007f8fe044e270 00007f8fdfda0574 00007f8fe9a5c6b4 00007f8fe9a5c649 00007f8fe34975a1 00007f8fe3531f7a 00007f8fe84e3647 00007f8fe1a73834 00007f8fe1a737c2 00007f8fe1a736f1 00007f8fe1a7366c 00007f8fe1a72922 00007f8fe186d34c 00007f8fe186c786 00007f8fe186c12f
0129 02:59:33.029988 (+ 289us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":941934464}
I20260129 02:59:33.199487 20589 raft_consensus.cc:1240] T 4f5a77e94fef4b16a2ba7b067c8b144a P 1c50de89a5a845668696dcb28c3badd6 [term 2 FOLLOWER]: Rejecting Update request from peer 974b802141bd4f13beffef2aa03f88f9 for earlier term 1. Current term is 2. Ops: []
I20260129 02:59:33.031309 21134 raft_consensus.cc:1240] T 42d27a28d89b409e8807ec0efd97af69 P 7a9daf5d42a84615a6383cfec9a10608 [term 4 FOLLOWER]: Rejecting Update request from peer 1c50de89a5a845668696dcb28c3badd6 for earlier term 3. Current term is 4. Ops: []
I20260129 02:59:33.041115 22143 raft_consensus.cc:515] T 2e58e9a5a8aa4cecafb464addb4f35ed P 197b880fa39d432a9b472845876bfaf0 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } }
I20260129 02:59:33.199970 21136 raft_consensus.cc:2393] T 75f45538dc704d9a8ec4ff8ad9f3ea5b P 7a9daf5d42a84615a6383cfec9a10608 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1c50de89a5a845668696dcb28c3badd6 in current term 2: Already voted for candidate 7a9daf5d42a84615a6383cfec9a10608 in this term.
W20260129 02:59:33.042309 22225 log.cc:927] Time spent T fcddab6bca684adf8ea032289a00f90e P 1c50de89a5a845668696dcb28c3badd6: Append to log took a long time: real 1.732s user 0.000s sys 0.005s
I20260129 02:59:33.201476 20433 raft_consensus.cc:2468] T c891d2bff4ce4568861bbc992e91fbb0 P 197b880fa39d432a9b472845876bfaf0 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1c50de89a5a845668696dcb28c3badd6 in term 6.
W20260129 02:59:33.221282 20433 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55676 (request call id 1086) took 2300 ms (client timeout 1821 ms). Trace:
I20260129 02:59:32.618080 20590 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1299723a4476420ab02fd8bfd7aa0645" candidate_uuid: "974b802141bd4f13beffef2aa03f88f9" candidate_term: 5 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "1c50de89a5a845668696dcb28c3badd6" is_pre_election: true
W20260129 02:59:33.221707 20433 rpcz_store.cc:269] 0129 02:59:30.921292 (+ 0us) service_pool.cc:168] Inserting onto call queue
0129 02:59:32.247839 (+1326547us) service_pool.cc:225] Handling call
0129 02:59:33.221182 (+973343us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260129 02:59:33.222388 20432 raft_consensus.cc:2410] T 96609ac68bdf4993a12e1c23355ca013 P 197b880fa39d432a9b472845876bfaf0 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1c50de89a5a845668696dcb28c3badd6 for term 4 because replica has last-logged OpId of term: 2 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 3.
W20260129 02:59:33.214586 20439 rpcz_store.cc:269] 0129 02:59:29.725384 (+ 0us) service_pool.cc:168] Inserting onto call queue
0129 02:59:30.838727 (+1113343us) service_pool.cc:225] Handling call
0129 02:59:32.945862 (+2107135us) spinlock_profiling.cc:243] Waited 706 ms on lock 0x7b5400331154. stack: 00007f8fe044e373 00007f8fe044e270 00007f8fdfda0574 00007f8fe9a5c6b4 00007f8fe9a5c649 00007f8fe34975a1 00007f8fe3531f7a 00007f8fe84e3647 00007f8fe1a73834 00007f8fe1a737c2 00007f8fe1a736f1 00007f8fe1a7366c 00007f8fe1a72922 00007f8fe186d34c 00007f8fe186c786 00007f8fe186c12f
0129 02:59:32.946028 (+ 166us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1623768320}
W20260129 02:59:33.222846 20432 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55676 (request call id 1069) took 3549 ms (client timeout 2924 ms). Trace:
I20260129 02:59:33.222994 20590 raft_consensus.cc:1240] T 0f20af7fbe5d4db89a6f17948d98912c P 1c50de89a5a845668696dcb28c3badd6 [term 4 FOLLOWER]: Rejecting Update request from peer 974b802141bd4f13beffef2aa03f88f9 for earlier term 3. Current term is 4. Ops: []
W20260129 02:59:33.223222 20432 rpcz_store.cc:269] 0129 02:59:29.673545 (+ 0us) service_pool.cc:168] Inserting onto call queue
0129 02:59:32.015408 (+2341863us) service_pool.cc:225] Handling call
0129 02:59:32.998323 (+982915us) spinlock_profiling.cc:243] Waited 4.32 ms on lock 0x7b5800000f98. stack: 00007f8fe044e373 00007f8fe044e270 00007f8fdfda0574 00007f8fe9a5c6b4 00007f8fe9a5c649 00007f8fe02c08b1 00007f8fe0442db1 00007f8fe0442c62 00007f8fe92eddd9 00007f8fe846d260 00007f8fe8535dbb 00007f8fe8535e41 00007f8fe84e3bd0 00007f8fe84e34b4 00007f8fe1a73834 00007f8fe1a737c2
0129 02:59:33.222786 (+224463us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":9938304}
W20260129 02:59:33.059989 20438 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55676 (request call id 1090) took 2107 ms (client timeout 1791 ms). Trace:
I20260129 02:59:33.220304 20585 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35752 (request call id 1446) took 1183 ms. Trace:
W20260129 02:59:33.224035 20438 rpcz_store.cc:269] 0129 02:59:30.952435 (+ 0us) service_pool.cc:168] Inserting onto call queue
0129 02:59:33.059605 (+2107170us) service_pool.cc:212] Skipping call since client already timed out
0129 02:59:33.059786 (+ 181us) inbound_call.cc:173] Queueing failure response
Metrics: {}
I20260129 02:59:33.199156 22222 leader_election.cc:290] T 75f45538dc704d9a8ec4ff8ad9f3ea5b P 7a9daf5d42a84615a6383cfec9a10608 [CANDIDATE]: Term 2 election: Requested vote from peers 974b802141bd4f13beffef2aa03f88f9 (127.18.211.2:34517), 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3:37735)
W20260129 02:59:33.217262 20437 rpcz_store.cc:269] 0129 02:59:29.730003 (+ 0us) service_pool.cc:168] Inserting onto call queue
0129 02:59:30.930716 (+1200713us) service_pool.cc:225] Handling call
0129 02:59:33.003207 (+2072491us) spinlock_profiling.cc:243] Waited 502 ms on lock 0x7b5400330254. stack: 00007f8fe044e373 00007f8fe044e270 00007f8fdfda0574 00007f8fe9a5c6b4 00007f8fe9a5c649 00007f8fe34975a1 00007f8fe3531f7a 00007f8fe84e3647 00007f8fe1a73834 00007f8fe1a737c2 00007f8fe1a736f1 00007f8fe1a7366c 00007f8fe1a72922 00007f8fe186d34c 00007f8fe186c786 00007f8fe186c12f
0129 02:59:33.003371 (+ 164us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1155477888}
I20260129 02:59:33.159634 22079 leader_election.cc:290] T 0f20af7fbe5d4db89a6f17948d98912c P 1c50de89a5a845668696dcb28c3badd6 [CANDIDATE]: Term 4 election: Requested vote from peers 197b880fa39d432a9b472845876bfaf0 (127.18.211.1:42027), 974b802141bd4f13beffef2aa03f88f9 (127.18.211.2:34517)
I20260129 02:59:33.063514 22092 raft_consensus.cc:493] T 4b1876f9a70b49edb3aa74105e66d3b3 P 197b880fa39d432a9b472845876bfaf0 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 1c50de89a5a845668696dcb28c3badd6)
I20260129 02:59:33.066704 20556 leader_election.cc:304] T 75be4bd6bd45474795fd472eff2953ac P 1c50de89a5a845668696dcb28c3badd6 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1c50de89a5a845668696dcb28c3badd6, 974b802141bd4f13beffef2aa03f88f9; no voters:
I20260129 02:59:33.225510 20589 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35752 (request call id 1443) took 1218 ms. Trace:
W20260129 02:59:33.183846 20436 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:48356 (request call id 213) took 2527 ms (client timeout 2242 ms). Trace:
I20260129 02:59:33.062237 20434 rpcz_store.cc:276] 0129 02:59:30.749198 (+ 0us) service_pool.cc:168] Inserting onto call queue
0129 02:59:32.106669 (+1357471us) service_pool.cc:225] Handling call
0129 02:59:32.108834 (+ 2165us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20260129 02:59:33.225975 20556 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.211.2:34517, user_credentials={real_user=slave}} blocked reactor thread for 159441us
I20260129 02:59:33.069222 20582 raft_consensus.cc:493] T 36ff33be666a4c82954731691dbcf327 P 1c50de89a5a845668696dcb28c3badd6 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260129 02:59:33.226742 20582 raft_consensus.cc:3060] T 36ff33be666a4c82954731691dbcf327 P 1c50de89a5a845668696dcb28c3badd6 [term 1 FOLLOWER]: Advancing to term 2
I20260129 02:59:33.227026 20556 leader_election.cc:304] T e4d18c35b0a7493e85d0a4b92956a48f P 1c50de89a5a845668696dcb28c3badd6 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1c50de89a5a845668696dcb28c3badd6; no voters: 197b880fa39d432a9b472845876bfaf0, 974b802141bd4f13beffef2aa03f88f9
I20260129 02:59:33.225876 22092 raft_consensus.cc:515] T 4b1876f9a70b49edb3aa74105e66d3b3 P 197b880fa39d432a9b472845876bfaf0 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:59:33.224169 20585 rpcz_store.cc:276] 0129 02:59:32.037064 (+ 0us) service_pool.cc:168] Inserting onto call queue
0129 02:59:32.063051 (+ 25987us) service_pool.cc:225] Handling call
0129 02:59:33.218250 (+1155199us) raft_consensus.cc:1471] Updating replica for 0 ops
0129 02:59:33.219156 (+ 906us) raft_consensus.cc:1697] Filling consensus response to leader.
0129 02:59:33.220072 (+ 916us) spinlock_profiling.cc:243] Waited 1.15 s on lock 0x7b54003b1850. stack: 00007f8fe044e373 00007f8fe044e270 00007f8fdfda0574 00007f8fe9a5c6b4 00007f8fe9a5c649 00007f8fe34975a1 00007f8fe352d52d 00007f8fe84e1f49 00007f8fe1a6ed34 00007f8fe1a6ecc2 00007f8fe1a6ebf1 00007f8fe1a6eb6c 00007f8fe1a6de22 00007f8fe186d34c 00007f8fe186c786 00007f8fe186c12f
0129 02:59:33.220216 (+ 144us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2648961792}
I20260129 02:59:33.063454 20440 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55676 (request call id 1055) took 5295 ms. Trace:
I20260129 02:59:33.086155 22077 leader_election.cc:290] T 724a236524cb4e7ba3428880681d6cc2 P 197b880fa39d432a9b472845876bfaf0 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 974b802141bd4f13beffef2aa03f88f9 (127.18.211.2:34517), 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3:37735)
I20260129 02:59:33.098266 22212 raft_consensus.cc:3055] T c934d31efd3f4e5195605ddc354d14e1 P 7a9daf5d42a84615a6383cfec9a10608 [term 2 LEADER]: Stepping down as leader of term 2
I20260129 02:59:33.230556 22212 raft_consensus.cc:740] T c934d31efd3f4e5195605ddc354d14e1 P 7a9daf5d42a84615a6383cfec9a10608 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 7a9daf5d42a84615a6383cfec9a10608, State: Running, Role: LEADER
W20260129 02:59:33.226049 20436 rpcz_store.cc:269] 0129 02:59:30.656788 (+ 0us) service_pool.cc:168] Inserting onto call queue
0129 02:59:33.183594 (+2526806us) service_pool.cc:212] Skipping call since client already timed out
0129 02:59:33.183764 (+ 170us) inbound_call.cc:173] Queueing failure response
Metrics: {}
W20260129 02:59:33.111769 22235 log.cc:927] Time spent T 0678b8b468bd43cdbc3e485900844821 P 974b802141bd4f13beffef2aa03f88f9: Append to log took a long time: real 1.518s user 0.005s sys 0.000s
I20260129 02:59:33.230334 20440 rpcz_store.cc:276] 0129 02:59:27.767769 (+ 0us) service_pool.cc:168] Inserting onto call queue
0129 02:59:28.832754 (+1064985us) service_pool.cc:225] Handling call
0129 02:59:28.834515 (+ 1761us) raft_consensus.cc:1471] Updating replica for 0 ops
0129 02:59:28.834717 (+ 202us) raft_consensus.cc:1534] Early marking committed up to index 3
0129 02:59:28.866862 (+ 32145us) log.cc:844] Serialized 14 byte log entry
0129 02:59:32.906888 (+4040026us) spinlock_profiling.cc:243] Waited 53.7 ms on lock 0x7b440002304c. stack: 00007f8fe044e373 00007f8fe044e270 00007f8fdfda0574 00007f8fe9a5c6b4 00007f8fe9a5c649 00007f8fe02c08b1 00007f8fe038f005 00007f8fe038fa22 00007f8fe30d957b 00007f8fe30d4e4f 00007f8fe30c8c0e 00007f8fe30c7f56 00007f8fe30c7b78 00007f8fe30d15c1 00007f8fe30d1579 00007f8fe30d1509
0129 02:59:33.061771 (+154883us) spinlock_profiling.cc:243] Waited 1.36 ms on lock 0x7b1800336340. stack: 00007f8fe044e373 00007f8fe044e270 00007f8fdfda0574 00007f8fe9a5c6b4 00007f8fe9a5c649 00007f8fe92d4c91 00007f8fe92e1671 00007f8fe30d585c 00007f8fe04632eb 00007f8fe924b709 00007f8fe047f57b 00007f8fe047d81f 00007f8fe047fae2 00007f8fe30c75ab 00007f8fe30cd0c3 00007f8fe30cd5da
0129 02:59:33.062788 (+ 1017us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0129 02:59:33.062865 (+ 77us) raft_consensus.cc:1645] Marking committed up to 3
0129 02:59:33.062982 (+ 117us) raft_consensus.cc:1697] Filling consensus response to leader.
0129 02:59:33.063097 (+ 115us) raft_consensus.cc:1691] UpdateReplicas() finished
0129 02:59:33.063255 (+ 158us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":50744,"prepare.run_cpu_time_us":62,"prepare.run_wall_time_us":47,"raft.queue_time_us":1633,"raft.run_cpu_time_us":123,"raft.run_wall_time_us":124,"spinlock_wait_cycles":126588544,"thread_start_us":805426,"threads_started":1,"wal-append.queue_time_us":708912}
I20260129 02:59:33.232219 22061 raft_consensus.cc:2749] T e4d18c35b0a7493e85d0a4b92956a48f P 1c50de89a5a845668696dcb28c3badd6 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260129 02:59:33.114244 22157 raft_consensus.cc:493] T c934d31efd3f4e5195605ddc354d14e1 P 974b802141bd4f13beffef2aa03f88f9 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260129 02:59:33.137001 22255 raft_consensus.cc:2804] T 75b0b6e82ddc4f92b1fe1d94f9a0b73b P 7a9daf5d42a84615a6383cfec9a10608 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260129 02:59:33.233095 22255 raft_consensus.cc:493] T 75b0b6e82ddc4f92b1fe1d94f9a0b73b P 7a9daf5d42a84615a6383cfec9a10608 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 1c50de89a5a845668696dcb28c3badd6)
I20260129 02:59:33.233561 22255 raft_consensus.cc:3060] T 75b0b6e82ddc4f92b1fe1d94f9a0b73b P 7a9daf5d42a84615a6383cfec9a10608 [term 1 FOLLOWER]: Advancing to term 2
I20260129 02:59:33.233209 22157 raft_consensus.cc:515] T c934d31efd3f4e5195605ddc354d14e1 P 974b802141bd4f13beffef2aa03f88f9 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } }
W20260129 02:59:33.136425 21102 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.211.2:34517, user_credentials={real_user=slave}} blocked reactor thread for 557741us
I20260129 02:59:33.137243 22122 raft_consensus.cc:2804] T 4f5a77e94fef4b16a2ba7b067c8b144a P 7a9daf5d42a84615a6383cfec9a10608 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260129 02:59:33.236088 22122 raft_consensus.cc:493] T 4f5a77e94fef4b16a2ba7b067c8b144a P 7a9daf5d42a84615a6383cfec9a10608 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 974b802141bd4f13beffef2aa03f88f9)
I20260129 02:59:33.236367 22092 leader_election.cc:290] T 4b1876f9a70b49edb3aa74105e66d3b3 P 197b880fa39d432a9b472845876bfaf0 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 974b802141bd4f13beffef2aa03f88f9 (127.18.211.2:34517), 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3:37735)
I20260129 02:59:33.236578 22122 raft_consensus.cc:3060] T 4f5a77e94fef4b16a2ba7b067c8b144a P 7a9daf5d42a84615a6383cfec9a10608 [term 1 FOLLOWER]: Advancing to term 2
I20260129 02:59:33.236768 21102 leader_election.cc:304] T 505580c93c694bbb8ea96fdf9b17ab46 P 7a9daf5d42a84615a6383cfec9a10608 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7a9daf5d42a84615a6383cfec9a10608; no voters: 1c50de89a5a845668696dcb28c3badd6, 974b802141bd4f13beffef2aa03f88f9
I20260129 02:59:33.219921 21128 rpcz_store.cc:276] 0129 02:59:31.873525 (+ 0us) service_pool.cc:168] Inserting onto call queue
0129 02:59:32.152200 (+278675us) service_pool.cc:225] Handling call
0129 02:59:32.264531 (+112331us) raft_consensus.cc:1471] Updating replica for 0 ops
0129 02:59:32.265715 (+ 1184us) raft_consensus.cc:1697] Filling consensus response to leader.
0129 02:59:33.010883 (+745168us) spinlock_profiling.cc:243] Waited 112 ms on lock 0x7b540073ee50. stack: 00007f8fe044e373 00007f8fe044e270 00007f8fdfda0574 00007f8fe9a5c6b4 00007f8fe9a5c649 00007f8fe34975a1 00007f8fe352d52d 00007f8fe84e1f49 00007f8fe1a6ed34 00007f8fe1a6ecc2 00007f8fe1a6ebf1 00007f8fe1a6eb6c 00007f8fe1a6de22 00007f8fe186d34c 00007f8fe186c786 00007f8fe186c12f
0129 02:59:33.011095 (+ 212us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":257194240}
I20260129 02:59:33.240150 22222 raft_consensus.cc:2749] T 505580c93c694bbb8ea96fdf9b17ab46 P 7a9daf5d42a84615a6383cfec9a10608 [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260129 02:59:33.204774 20509 rpcz_store.cc:276] 0129 02:59:30.989310 (+ 0us) service_pool.cc:168] Inserting onto call queue
0129 02:59:31.536348 (+547038us) service_pool.cc:225] Handling call
0129 02:59:31.655669 (+119321us) raft_consensus.cc:1471] Updating replica for 1 ops
0129 02:59:31.699054 (+ 43385us) raft_consensus.cc:1534] Early marking committed up to index 0
0129 02:59:31.702650 (+ 3596us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0129 02:59:31.866132 (+163482us) log.cc:844] Serialized 26 byte log entry
0129 02:59:32.260717 (+394585us) raft_consensus.cc:1645] Marking committed up to 3
0129 02:59:32.261571 (+ 854us) log.cc:844] Serialized 14 byte log entry
0129 02:59:32.506967 (+245396us) raft_consensus.cc:1697] Filling consensus response to leader.
0129 02:59:32.538768 (+ 31801us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0129 02:59:32.541455 (+ 2687us) raft_consensus.cc:1685] finished
0129 02:59:32.541470 (+ 15us) raft_consensus.cc:1691] UpdateReplicas() finished
0129 02:59:32.566850 (+ 25380us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":79610,"prepare.run_cpu_time_us":93,"prepare.run_wall_time_us":5041,"raft.queue_time_us":280540,"raft.run_cpu_time_us":481,"raft.run_wall_time_us":11216,"thread_start_us":508790,"threads_started":2,"wal-append.queue_time_us":288202}
W20260129 02:59:33.240674 21136 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34046 (request call id 385) took 1261 ms (client timeout 1674 ms). Trace:
I20260129 02:59:33.239555 21102 leader_election.cc:304] T 00e79d8f25a34676b1d866eb2068a2b2 P 7a9daf5d42a84615a6383cfec9a10608 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7a9daf5d42a84615a6383cfec9a10608, 974b802141bd4f13beffef2aa03f88f9; no voters:
W20260129 02:59:33.241168 21136 rpcz_store.cc:269] 0129 02:59:31.979122 (+ 0us) service_pool.cc:168] Inserting onto call queue
0129 02:59:31.992845 (+ 13723us) service_pool.cc:225] Handling call
0129 02:59:33.240435 (+1247590us) spinlock_profiling.cc:243] Waited 1.21 s on lock 0x7b540068fbd4. stack: 00007f8fe044e373 00007f8fe044e270 00007f8fdfda0574 00007f8fe9a5c6b4 00007f8fe9a5c649 00007f8fe34975a1 00007f8fe3531f7a 00007f8fe84e3647 00007f8fe1a73834 00007f8fe1a737c2 00007f8fe1a736f1 00007f8fe1a7366c 00007f8fe1a72922 00007f8fe186d34c 00007f8fe186c786 00007f8fe186c12f
0129 02:59:33.240598 (+ 163us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2772892928}
I20260129 02:59:33.196877 22213 leader_election.cc:290] T 4f5a77e94fef4b16a2ba7b067c8b144a P 1c50de89a5a845668696dcb28c3badd6 [CANDIDATE]: Term 2 election: Requested vote from peers 974b802141bd4f13beffef2aa03f88f9 (127.18.211.2:34517), 7a9daf5d42a84615a6383cfec9a10608 (127.18.211.4:40439)
I20260129 02:59:33.244096 22257 raft_consensus.cc:2804] T 00e79d8f25a34676b1d866eb2068a2b2 P 7a9daf5d42a84615a6383cfec9a10608 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260129 02:59:33.194554 22164 leader_election.cc:290] T 724a236524cb4e7ba3428880681d6cc2 P 1c50de89a5a845668696dcb28c3badd6 [CANDIDATE]: Term 2 election: Requested vote from peers 197b880fa39d432a9b472845876bfaf0 (127.18.211.1:42027), 974b802141bd4f13beffef2aa03f88f9 (127.18.211.2:34517)
I20260129 02:59:33.244776 22222 raft_consensus.cc:493] T 1299723a4476420ab02fd8bfd7aa0645 P 7a9daf5d42a84615a6383cfec9a10608 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 197b880fa39d432a9b472845876bfaf0)
I20260129 02:59:33.245626 22222 raft_consensus.cc:515] T 1299723a4476420ab02fd8bfd7aa0645 P 7a9daf5d42a84615a6383cfec9a10608 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } }
I20260129 02:59:33.189688 22115 leader_election.cc:290] T 1299723a4476420ab02fd8bfd7aa0645 P 1c50de89a5a845668696dcb28c3badd6 [CANDIDATE]: Term 5 election: Requested vote from peers 197b880fa39d432a9b472845876bfaf0 (127.18.211.1:42027), 7a9daf5d42a84615a6383cfec9a10608 (127.18.211.4:40439)
W20260129 02:59:33.144400 20405 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.211.3:37735, user_credentials={real_user=slave}} blocked reactor thread for 885767us
I20260129 02:59:33.248562 21130 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4f5a77e94fef4b16a2ba7b067c8b144a" candidate_uuid: "1c50de89a5a845668696dcb28c3badd6" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "7a9daf5d42a84615a6383cfec9a10608"
I20260129 02:59:33.248714 20405 leader_election.cc:304] T 78aa39096868415d9afc2ffcb248e198 P 197b880fa39d432a9b472845876bfaf0 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 197b880fa39d432a9b472845876bfaf0; no voters: 1c50de89a5a845668696dcb28c3badd6, 7a9daf5d42a84615a6383cfec9a10608
I20260129 02:59:33.248873 20556 leader_election.cc:304] T 7a9a69cd82724b1faed1d10aa091fc4a P 1c50de89a5a845668696dcb28c3badd6 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1c50de89a5a845668696dcb28c3badd6, 974b802141bd4f13beffef2aa03f88f9; no voters:
I20260129 02:59:33.148831 22140 raft_consensus.cc:3055] T 42d27a28d89b409e8807ec0efd97af69 P 1c50de89a5a845668696dcb28c3badd6 [term 3 LEADER]: Stepping down as leader of term 3
I20260129 02:59:33.249197 22102 raft_consensus.cc:515] T c2633a6fc34c44579de348d208b0c01e P 1c50de89a5a845668696dcb28c3badd6 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:59:33.250432 21128 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1299723a4476420ab02fd8bfd7aa0645" candidate_uuid: "1c50de89a5a845668696dcb28c3badd6" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "7a9daf5d42a84615a6383cfec9a10608"
I20260129 02:59:33.250659 22140 raft_consensus.cc:740] T 42d27a28d89b409e8807ec0efd97af69 P 1c50de89a5a845668696dcb28c3badd6 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 1c50de89a5a845668696dcb28c3badd6, State: Running, Role: LEADER
I20260129 02:59:33.251228 21128 raft_consensus.cc:3060] T 1299723a4476420ab02fd8bfd7aa0645 P 7a9daf5d42a84615a6383cfec9a10608 [term 4 FOLLOWER]: Advancing to term 5
I20260129 02:59:33.230979 22079 raft_consensus.cc:2804] T 75be4bd6bd45474795fd472eff2953ac P 1c50de89a5a845668696dcb28c3badd6 [term 7 FOLLOWER]: Leader election won for term 7
I20260129 02:59:33.252525 22157 leader_election.cc:290] T c934d31efd3f4e5195605ddc354d14e1 P 974b802141bd4f13beffef2aa03f88f9 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 197b880fa39d432a9b472845876bfaf0 (127.18.211.1:42027), 7a9daf5d42a84615a6383cfec9a10608 (127.18.211.4:40439)
I20260129 02:59:33.252158 22140 consensus_queue.cc:260] T 42d27a28d89b409e8807ec0efd97af69 P 1c50de89a5a845668696dcb28c3badd6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.6, Last appended by leader: 6, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } }
I20260129 02:59:33.253567 22140 raft_consensus.cc:3060] T 42d27a28d89b409e8807ec0efd97af69 P 1c50de89a5a845668696dcb28c3badd6 [term 3 FOLLOWER]: Advancing to term 4
I20260129 02:59:33.151569 22137 raft_consensus.cc:493] T 75b0b6e82ddc4f92b1fe1d94f9a0b73b P 974b802141bd4f13beffef2aa03f88f9 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 1c50de89a5a845668696dcb28c3badd6)
I20260129 02:59:33.172431 20587 raft_consensus.cc:2393] T cb0b3f259ebc4310927338a8740e87b8 P 1c50de89a5a845668696dcb28c3badd6 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 7a9daf5d42a84615a6383cfec9a10608 in current term 4: Already voted for candidate 1c50de89a5a845668696dcb28c3badd6 in this term.
I20260129 02:59:33.171248 22112 raft_consensus.cc:515] T a0fcc58d6fb6473cafbd9d34d5195269 P 974b802141bd4f13beffef2aa03f88f9 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:59:33.257805 22061 raft_consensus.cc:2804] T 7a9a69cd82724b1faed1d10aa091fc4a P 1c50de89a5a845668696dcb28c3badd6 [term 3 FOLLOWER]: Leader pre-election won for term 4
W20260129 02:59:33.258282 20433 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55676 (request call id 1088) took 2322 ms (client timeout 2095 ms). Trace:
W20260129 02:59:33.258687 20433 rpcz_store.cc:269] 0129 02:59:30.935924 (+ 0us) service_pool.cc:168] Inserting onto call queue
0129 02:59:33.258040 (+2322116us) service_pool.cc:212] Skipping call since client already timed out
0129 02:59:33.258203 (+ 163us) inbound_call.cc:173] Queueing failure response
Metrics: {}
I20260129 02:59:33.258296 22061 raft_consensus.cc:493] T 7a9a69cd82724b1faed1d10aa091fc4a P 1c50de89a5a845668696dcb28c3badd6 [term 3 FOLLOWER]: Starting leader election (detected failure of leader 197b880fa39d432a9b472845876bfaf0)
I20260129 02:59:33.256776 21136 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "cb0b3f259ebc4310927338a8740e87b8" candidate_uuid: "1c50de89a5a845668696dcb28c3badd6" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "7a9daf5d42a84615a6383cfec9a10608"
I20260129 02:59:33.259976 22061 raft_consensus.cc:3060] T 7a9a69cd82724b1faed1d10aa091fc4a P 1c50de89a5a845668696dcb28c3badd6 [term 3 FOLLOWER]: Advancing to term 4
I20260129 02:59:33.185338 20581 raft_consensus.cc:2393] T 1299723a4476420ab02fd8bfd7aa0645 P 1c50de89a5a845668696dcb28c3badd6 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 7a9daf5d42a84615a6383cfec9a10608 in current term 5: Already voted for candidate 1c50de89a5a845668696dcb28c3badd6 in this term.
W20260129 02:59:33.260426 20439 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55676 (request call id 1084) took 2363 ms (client timeout 2216 ms). Trace:
I20260129 02:59:32.933914 20586 raft_consensus.cc:1240] T 368953c9102643b2a21f99af36b51905 P 1c50de89a5a845668696dcb28c3badd6 [term 4 FOLLOWER]: Rejecting Update request from peer 974b802141bd4f13beffef2aa03f88f9 for earlier term 3. Current term is 4. Ops: []
W20260129 02:59:33.189965 20441 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55672 (request call id 1396) took 2471 ms (client timeout 1519 ms). Trace:
W20260129 02:59:33.268028 21099 leader_election.cc:336] T febffe4a8b394368b69b1d164308fb9a P 7a9daf5d42a84615a6383cfec9a10608 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 197b880fa39d432a9b472845876bfaf0 (127.18.211.1:42027): Timed out: RequestConsensusVote RPC to 127.18.211.1:42027 timed out after 1.657s (SENT)
W20260129 02:59:33.217742 20583 rpcz_store.cc:269] 0129 02:59:31.422725 (+ 0us) service_pool.cc:168] Inserting onto call queue
0129 02:59:31.422828 (+ 103us) service_pool.cc:225] Handling call
0129 02:59:33.213493 (+1790665us) spinlock_profiling.cc:243] Waited 1.76 s on lock 0x7b54003b1854. stack: 00007f8fe044e373 00007f8fe044e270 00007f8fdfda0574 00007f8fe9a5c6b4 00007f8fe9a5c649 00007f8fe34975a1 00007f8fe3531f7a 00007f8fe84e3647 00007f8fe1a73834 00007f8fe1a737c2 00007f8fe1a736f1 00007f8fe1a7366c 00007f8fe1a72922 00007f8fe186d34c 00007f8fe186c786 00007f8fe186c12f
0129 02:59:33.213691 (+ 198us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":4059039616}
I20260129 02:59:33.221427 21134 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34046 (request call id 373) took 3551 ms. Trace:
I20260129 02:59:33.213284 20515 rpcz_store.cc:276] 0129 02:59:28.786500 (+ 0us) service_pool.cc:168] Inserting onto call queue
0129 02:59:29.285016 (+498516us) service_pool.cc:225] Handling call
0129 02:59:29.326935 (+ 41919us) raft_consensus.cc:1471] Updating replica for 0 ops
0129 02:59:29.327302 (+ 367us) raft_consensus.cc:1534] Early marking committed up to index 2
0129 02:59:29.354136 (+ 26834us) log.cc:844] Serialized 14 byte log entry
0129 02:59:31.511153 (+2157017us) spinlock_profiling.cc:243] Waited 4.1 ms on lock 0x7b1800256ba0. stack: 00007f8fe044e373 00007f8fe044e270 00007f8fdfda0574 00007f8fe9a5c6b4 00007f8fe9a5c649 00007f8fe92d4c91 00007f8fe92e1671 00007f8fe0462c67 00007f8fe924b709 00007f8fe047f57b 00007f8fe047d81f 00007f8fe047fae2 00007f8fe30c75ab 00007f8fe30cd0c3 00007f8fe30cd5da 00007f8fe35386aa
0129 02:59:32.534082 (+1022929us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0129 02:59:32.541574 (+ 7492us) raft_consensus.cc:1645] Marking committed up to 2
0129 02:59:32.547213 (+ 5639us) raft_consensus.cc:1697] Filling consensus response to leader.
0129 02:59:32.550056 (+ 2843us) raft_consensus.cc:1691] UpdateReplicas() finished
0129 02:59:32.582712 (+ 32656us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":24610,"prepare.run_cpu_time_us":43,"prepare.run_wall_time_us":44,"raft.queue_time_us":255151,"raft.run_cpu_time_us":172,"raft.run_wall_time_us":172,"spinlock_wait_cycles":10649216,"thread_start_us":1001765,"threads_started":2,"wal-append.queue_time_us":2844457,"wal-append.run_cpu_time_us":1335,"wal-append.run_wall_time_us":1009944}
I20260129 02:59:33.062602 22120 leader_election.cc:290] T 926a250fa5834e88b96ccecbefbf5086 P 197b880fa39d432a9b472845876bfaf0 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 974b802141bd4f13beffef2aa03f88f9 (127.18.211.2:34517), 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3:37735)
I20260129 02:59:33.224880 22143 leader_election.cc:290] T 2e58e9a5a8aa4cecafb464addb4f35ed P 197b880fa39d432a9b472845876bfaf0 [CANDIDATE]: Term 3 election: Requested vote from peers 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3:37735), 7a9daf5d42a84615a6383cfec9a10608 (127.18.211.4:40439)
I20260129 02:59:33.070392 20512 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:48544 (request call id 113) took 2485 ms. Trace:
I20260129 02:59:33.269207 20556 leader_election.cc:304] T 8055232cf0004d40a85f3bf983fe00a9 P 1c50de89a5a845668696dcb28c3badd6 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1c50de89a5a845668696dcb28c3badd6; no voters: 197b880fa39d432a9b472845876bfaf0, 974b802141bd4f13beffef2aa03f88f9
I20260129 02:59:33.273917 22092 raft_consensus.cc:2749] T 78aa39096868415d9afc2ffcb248e198 P 197b880fa39d432a9b472845876bfaf0 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260129 02:59:33.273841 20512 rpcz_store.cc:276] 0129 02:59:30.584922 (+ 0us) service_pool.cc:168] Inserting onto call queue
0129 02:59:30.585566 (+ 644us) service_pool.cc:225] Handling call
0129 02:59:32.524133 (+1938567us) raft_consensus.cc:1471] Updating replica for 0 ops
0129 02:59:32.524650 (+ 517us) raft_consensus.cc:1534] Early marking committed up to index 0
0129 02:59:32.524679 (+ 29us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0129 02:59:32.524747 (+ 68us) raft_consensus.cc:1645] Marking committed up to 6
0129 02:59:32.524790 (+ 43us) raft_consensus.cc:1697] Filling consensus response to leader.
0129 02:59:32.524888 (+ 98us) raft_consensus.cc:1691] UpdateReplicas() finished
0129 02:59:33.070057 (+545169us) spinlock_profiling.cc:243] Waited 1.94 s on lock 0x7b54002ee350. stack: 00007f8fe044e373 00007f8fe044e270 00007f8fdfda0574 00007f8fe9a5c6b4 00007f8fe9a5c649 00007f8fe34975a1 00007f8fe352d52d 00007f8fe84e1f49 00007f8fe1a6ed34 00007f8fe1a6ecc2 00007f8fe1a6ebf1 00007f8fe1a6eb6c 00007f8fe1a6de22 00007f8fe186d34c 00007f8fe186c786 00007f8fe186c12f
0129 02:59:33.070201 (+ 144us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":4457167744}
I20260129 02:59:33.072584 20338 rpcz_store.cc:275] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:33836 (request call id 151) took 7521 ms. Trace:
I20260129 02:59:33.275074 20556 leader_election.cc:304] T a4c1226b987b468fa84dfc6ba0e2dbe1 P 1c50de89a5a845668696dcb28c3badd6 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1c50de89a5a845668696dcb28c3badd6, 974b802141bd4f13beffef2aa03f88f9; no voters:
W20260129 02:59:33.107559 20480 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.211.4:40439, user_credentials={real_user=slave}} blocked reactor thread for 223486us
W20260129 02:59:33.138139 21100 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.211.3:37735, user_credentials={real_user=slave}} blocked reactor thread for 1.03792e+06us
I20260129 02:59:33.192163 22190 leader_election.cc:290] T 4408006eb5174de1896b037cf3f18457 P 197b880fa39d432a9b472845876bfaf0 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 974b802141bd4f13beffef2aa03f88f9 (127.18.211.2:34517), 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3:37735)
I20260129 02:59:33.256177 22222 leader_election.cc:290] T 1299723a4476420ab02fd8bfd7aa0645 P 7a9daf5d42a84615a6383cfec9a10608 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 197b880fa39d432a9b472845876bfaf0 (127.18.211.1:42027), 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3:37735)
I20260129 02:59:33.256234 22137 raft_consensus.cc:515] T 75b0b6e82ddc4f92b1fe1d94f9a0b73b P 974b802141bd4f13beffef2aa03f88f9 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } }
I20260129 02:59:33.284806 22137 leader_election.cc:290] T 75b0b6e82ddc4f92b1fe1d94f9a0b73b P 974b802141bd4f13beffef2aa03f88f9 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3:37735), 7a9daf5d42a84615a6383cfec9a10608 (127.18.211.4:40439)
I20260129 02:59:33.256706 22079 raft_consensus.cc:697] T 75be4bd6bd45474795fd472eff2953ac P 1c50de89a5a845668696dcb28c3badd6 [term 7 LEADER]: Becoming Leader. State: Replica: 1c50de89a5a845668696dcb28c3badd6, State: Running, Role: LEADER
I20260129 02:59:33.285991 22079 consensus_queue.cc:237] T 75be4bd6bd45474795fd472eff2953ac P 1c50de89a5a845668696dcb28c3badd6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.4, Last appended by leader: 4, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:59:33.287060 20582 raft_consensus.cc:515] T 36ff33be666a4c82954731691dbcf327 P 1c50de89a5a845668696dcb28c3badd6 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:59:33.288187 21128 raft_consensus.cc:2410] T 1299723a4476420ab02fd8bfd7aa0645 P 7a9daf5d42a84615a6383cfec9a10608 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 1c50de89a5a845668696dcb28c3badd6 for term 5 because replica has last-logged OpId of term: 4 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 5.
I20260129 02:59:33.288699 20436 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00e79d8f25a34676b1d866eb2068a2b2" candidate_uuid: "7a9daf5d42a84615a6383cfec9a10608" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "197b880fa39d432a9b472845876bfaf0" is_pre_election: true
I20260129 02:59:33.258867 22112 leader_election.cc:290] T a0fcc58d6fb6473cafbd9d34d5195269 P 974b802141bd4f13beffef2aa03f88f9 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 197b880fa39d432a9b472845876bfaf0 (127.18.211.1:42027), 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3:37735)
I20260129 02:59:33.259264 22103 consensus_queue.cc:1059] T a5a05fbcbbcf4916bbc01fd1a5b5e90d P 1c50de89a5a845668696dcb28c3badd6 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 }, Status: INVALID_TERM, Last received: 4.3, Next index: 4, Last known committed idx: 2, Time since last communication: 0.000s
W20260129 02:59:33.260860 20439 rpcz_store.cc:269] 0129 02:59:30.897155 (+ 0us) service_pool.cc:168] Inserting onto call queue
0129 02:59:33.260220 (+2363065us) service_pool.cc:212] Skipping call since client already timed out
0129 02:59:33.260361 (+ 141us) inbound_call.cc:173] Queueing failure response
Metrics: {}
I20260129 02:59:33.290936 20439 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "913fc7d751ed4b7388d889aac30a5f9b" candidate_uuid: "1c50de89a5a845668696dcb28c3badd6" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "197b880fa39d432a9b472845876bfaf0" is_pre_election: true
I20260129 02:59:33.212880 20404 service_pool.cc:138] GetConsensusState request on kudu.consensus.ConsensusService from 127.0.0.1:55666 dropped due to backpressure: service queue is full with 50 items Contents of service queue:
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55672 (request call id 1401)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55676 (request call id 1088)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55676 (request call id 1087)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55676 (request call id 1084)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55676 (request call id 1092)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55676 (request call id 1095)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55676 (request call id 1096)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55676 (request call id 1093)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:48356 (request call id 220)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55676 (request call id 1099)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:48356 (request call id 218)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55672 (request call id 1410)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55676 (request call id 1100)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:48356 (request call id 216)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:48356 (request call id 219)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55672 (request call id 1407)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55672 (request call id 1406)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55676 (request call id 1098)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55672 (request call id 1408)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55672 (request call id 1409)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:48356 (request call id 223)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:48356 (request call id 222)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:48356 (request call id 224)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55672 (request call id 1399)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:48356 (request call id 225)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:48356 (request call id 226)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55676 (request call id 1071)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:48356 (request call id 205)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:48356 (request call id 206)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55672 (request call id 1383)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55672 (request call id 1384)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55672 (request call id 1388)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55672 (request call id 1391)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55672 (request call id 1393)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:48356 (request call id 211)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55672 (request call id 1394)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:48356 (request call id 212)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55672 (request call id 1395)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:48356 (request call id 214)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55676 (request call id 1080)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55676 (request call id 1081)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55676 (request call id 1082)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55676 (request call id 1083)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55676 (request call id 1085)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55672 (request call id 1403)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55676 (request call id 1094)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55676 (request call id 1097)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55672 (request call id 1404)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55672 (request call id 1405)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:48356 (request call id 215)
I20260129 02:59:33.292081 20439 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55676 (request call id 1099) took 1048 ms. Trace:
I20260129 02:59:33.292541 20439 rpcz_store.cc:276] 0129 02:59:32.243543 (+ 0us) service_pool.cc:168] Inserting onto call queue
0129 02:59:33.290505 (+1046962us) service_pool.cc:225] Handling call
0129 02:59:33.291997 (+ 1492us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260129 02:59:33.293506 20439 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1299723a4476420ab02fd8bfd7aa0645" candidate_uuid: "7a9daf5d42a84615a6383cfec9a10608" candidate_term: 5 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "197b880fa39d432a9b472845876bfaf0" is_pre_election: true
I20260129 02:59:33.210374 20407 service_pool.cc:138] UpdateConsensus request on kudu.consensus.ConsensusService from 127.0.0.1:48356 dropped due to backpressure: service queue is full with 50 items Contents of service queue:
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55672 (request call id 1401)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55676 (request call id 1088)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55676 (request call id 1087)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55676 (request call id 1084)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55676 (request call id 1092)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55676 (request call id 1095)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55676 (request call id 1096)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55676 (request call id 1093)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:48356 (request call id 220)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55676 (request call id 1099)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:48356 (request call id 218)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55672 (request call id 1410)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55676 (request call id 1100)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:48356 (request call id 216)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:48356 (request call id 219)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55672 (request call id 1407)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55672 (request call id 1406)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55676 (request call id 1098)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55672 (request call id 1408)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55672 (request call id 1409)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:48356 (request call id 223)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:48356 (request call id 222)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:48356 (request call id 224)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55672 (request call id 1399)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:48356 (request call id 225)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:48356 (request call id 226)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55676 (request call id 1071)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:48356 (request call id 205)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:48356 (request call id 206)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55672 (request call id 1383)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55672 (request call id 1384)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55672 (request call id 1388)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55672 (request call id 1391)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55672 (request call id 1393)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:48356 (request call id 211)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55672 (request call id 1394)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:48356 (request call id 212)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55672 (request call id 1395)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:48356 (request call id 214)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55676 (request call id 1080)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55676 (request call id 1081)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55676 (request call id 1082)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55676 (request call id 1083)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55676 (request call id 1085)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55672 (request call id 1403)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55676 (request call id 1094)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55676 (request call id 1097)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55672 (request call id 1404)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55672 (request call id 1405)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:48356 (request call id 215)
I20260129 02:59:33.296238 20405 leader_election.cc:304] T a1a39dcd328e4268ae41b988c66b4a75 P 197b880fa39d432a9b472845876bfaf0 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 197b880fa39d432a9b472845876bfaf0; no voters: 1c50de89a5a845668696dcb28c3badd6, 974b802141bd4f13beffef2aa03f88f9
W20260129 02:59:33.263736 20581 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:48526 (request call id 132) took 3033 ms (client timeout 1616 ms). Trace:
I20260129 02:59:33.264504 20584 raft_consensus.cc:1240] T 1299723a4476420ab02fd8bfd7aa0645 P 1c50de89a5a845668696dcb28c3badd6 [term 5 FOLLOWER]: Rejecting Update request from peer 197b880fa39d432a9b472845876bfaf0 for earlier term 4. Current term is 5. Ops: []
W20260129 02:59:33.261780 20437 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55676 (request call id 1092) took 1579 ms (client timeout 1518 ms). Trace:
W20260129 02:59:33.261925 20438 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55676 (request call id 1087) took 2337 ms (client timeout 2135 ms). Trace:
W20260129 02:59:33.299407 20437 rpcz_store.cc:269] 0129 02:59:31.682413 (+ 0us) service_pool.cc:168] Inserting onto call queue
0129 02:59:33.261559 (+1579146us) service_pool.cc:212] Skipping call since client already timed out
0129 02:59:33.261705 (+ 146us) inbound_call.cc:173] Queueing failure response
Metrics: {}
W20260129 02:59:33.215468 20435 rpcz_store.cc:269] 0129 02:59:30.865022 (+ 0us) service_pool.cc:168] Inserting onto call queue
0129 02:59:32.947652 (+2082630us) service_pool.cc:212] Skipping call since client already timed out
0129 02:59:32.947797 (+ 145us) inbound_call.cc:173] Queueing failure response
Metrics: {}
W20260129 02:59:33.300009 20553 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.211.1:42027, user_credentials={real_user=slave}} blocked reactor thread for 297606us
W20260129 02:59:33.300657 20553 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260129 02:59:33.300793 20435 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3ccac82a01a2445fa5b0ce6c0fcf2295" candidate_uuid: "974b802141bd4f13beffef2aa03f88f9" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "197b880fa39d432a9b472845876bfaf0" is_pre_election: true
I20260129 02:59:33.301694 20435 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55672 (request call id 1410) took 1058 ms. Trace:
I20260129 02:59:33.302038 20435 rpcz_store.cc:276] 0129 02:59:32.242824 (+ 0us) service_pool.cc:168] Inserting onto call queue
0129 02:59:33.300486 (+1057662us) service_pool.cc:225] Handling call
0129 02:59:33.301625 (+ 1139us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20260129 02:59:33.302425 20553 leader_election.cc:336] T c891d2bff4ce4568861bbc992e91fbb0 P 1c50de89a5a845668696dcb28c3badd6 [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer 197b880fa39d432a9b472845876bfaf0 (127.18.211.1:42027): Timed out: RequestConsensusVote RPC to 127.18.211.1:42027 timed out after 1.821s (SENT)
I20260129 02:59:33.302866 20553 leader_election.cc:304] T c891d2bff4ce4568861bbc992e91fbb0 P 1c50de89a5a845668696dcb28c3badd6 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1c50de89a5a845668696dcb28c3badd6; no voters: 197b880fa39d432a9b472845876bfaf0, 974b802141bd4f13beffef2aa03f88f9
I20260129 02:59:33.302983 20435 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "75be4bd6bd45474795fd472eff2953ac" candidate_uuid: "1c50de89a5a845668696dcb28c3badd6" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "197b880fa39d432a9b472845876bfaf0"
I20260129 02:59:33.276196 20556 leader_election.cc:304] T 3906b080a5b34a74b8b64128635e8997 P 1c50de89a5a845668696dcb28c3badd6 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1c50de89a5a845668696dcb28c3badd6, 974b802141bd4f13beffef2aa03f88f9; no voters:
I20260129 02:59:33.215934 20514 rpcz_store.cc:276] 0129 02:59:31.875700 (+ 0us) service_pool.cc:168] Inserting onto call queue
0129 02:59:32.075663 (+199963us) service_pool.cc:225] Handling call
0129 02:59:32.974006 (+898343us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260129 02:59:33.269322 21134 rpcz_store.cc:276] 0129 02:59:29.670349 (+ 0us) service_pool.cc:168] Inserting onto call queue
0129 02:59:29.670473 (+ 124us) service_pool.cc:225] Handling call
0129 02:59:33.030813 (+3360340us) raft_consensus.cc:1471] Updating replica for 0 ops
0129 02:59:33.220441 (+189628us) raft_consensus.cc:1697] Filling consensus response to leader.
0129 02:59:33.221208 (+ 767us) spinlock_profiling.cc:243] Waited 3.36 s on lock 0x7b54003b3dd0. stack: 00007f8fe044e373 00007f8fe044e270 00007f8fdfda0574 00007f8fe9a5c6b4 00007f8fe9a5c649 00007f8fe34975a1 00007f8fe352d52d 00007f8fe84e1f49 00007f8fe1a6ed34 00007f8fe1a6ecc2 00007f8fe1a6ebf1 00007f8fe1a6eb6c 00007f8fe1a6de22 00007f8fe186d34c 00007f8fe186c786 00007f8fe186c12f
0129 02:59:33.221363 (+ 155us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":7727017344}
I20260129 02:59:33.226015 20589 rpcz_store.cc:276] 0129 02:59:32.006726 (+ 0us) service_pool.cc:168] Inserting onto call queue
0129 02:59:32.052864 (+ 46138us) service_pool.cc:225] Handling call
0129 02:59:32.053325 (+ 461us) raft_consensus.cc:1471] Updating replica for 0 ops
0129 02:59:33.219640 (+1166315us) raft_consensus.cc:1697] Filling consensus response to leader.
0129 02:59:33.225283 (+ 5643us) spinlock_profiling.cc:243] Waited 1.15 s on lock 0x7b54003aff54. stack: 00007f8fe044e373 00007f8fe044e270 00007f8fdfda0574 00007f8fe9a5c6b4 00007f8fe9a5c649 00007f8fe34975a1 00007f8fe352ea81 00007f8fe352d4be 00007f8fe84e1f49 00007f8fe1a6ed34 00007f8fe1a6ecc2 00007f8fe1a6ebf1 00007f8fe1a6eb6c 00007f8fe1a6de22 00007f8fe186d34c 00007f8fe186c786
0129 02:59:33.225459 (+ 176us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2635018496}
I20260129 02:59:33.275172 22213 raft_consensus.cc:2749] T 8055232cf0004d40a85f3bf983fe00a9 P 1c50de89a5a845668696dcb28c3badd6 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260129 02:59:33.231786 22212 consensus_queue.cc:260] T c934d31efd3f4e5195605ddc354d14e1 P 7a9daf5d42a84615a6383cfec9a10608 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } }
I20260129 02:59:33.306327 22212 raft_consensus.cc:3060] T c934d31efd3f4e5195605ddc354d14e1 P 7a9daf5d42a84615a6383cfec9a10608 [term 2 FOLLOWER]: Advancing to term 3
I20260129 02:59:33.306486 20584 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35754 (request call id 1519) took 1885 ms. Trace:
I20260129 02:59:33.306950 20584 rpcz_store.cc:276] 0129 02:59:31.420829 (+ 0us) service_pool.cc:168] Inserting onto call queue
0129 02:59:31.420957 (+ 128us) service_pool.cc:225] Handling call
0129 02:59:33.264178 (+1843221us) raft_consensus.cc:1471] Updating replica for 0 ops
0129 02:59:33.305667 (+ 41489us) raft_consensus.cc:1697] Filling consensus response to leader.
0129 02:59:33.306287 (+ 620us) spinlock_profiling.cc:243] Waited 1.76 s on lock 0x7b5400108450. stack: 00007f8fe044e373 00007f8fe044e270 00007f8fdfda0574 00007f8fe9a5c6b4 00007f8fe9a5c649 00007f8fe34975a1 00007f8fe352d52d 00007f8fe84e1f49 00007f8fe1a6ed34 00007f8fe1a6ecc2 00007f8fe1a6ebf1 00007f8fe1a6eb6c 00007f8fe1a6de22 00007f8fe186d34c 00007f8fe186c786 00007f8fe186c12f
0129 02:59:33.306430 (+ 143us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":4039079680}
W20260129 02:59:33.298830 20581 rpcz_store.cc:269] 0129 02:59:30.230146 (+ 0us) service_pool.cc:168] Inserting onto call queue
0129 02:59:30.230355 (+ 209us) service_pool.cc:225] Handling call
0129 02:59:33.262253 (+3031898us) spinlock_profiling.cc:243] Waited 2.71 s on lock 0x7b5400108454. stack: 00007f8fe044e373 00007f8fe044e270 00007f8fdfda0574 00007f8fe9a5c6b4 00007f8fe9a5c649 00007f8fe34975a1 00007f8fe3531f7a 00007f8fe84e3647 00007f8fe1a73834 00007f8fe1a737c2 00007f8fe1a736f1 00007f8fe1a7366c 00007f8fe1a72922 00007f8fe186d34c 00007f8fe186c786 00007f8fe186c12f
0129 02:59:33.263633 (+ 1380us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":6237662592}
I20260129 02:59:33.309104 22213 raft_consensus.cc:3055] T a5a05fbcbbcf4916bbc01fd1a5b5e90d P 1c50de89a5a845668696dcb28c3badd6 [term 4 LEADER]: Stepping down as leader of term 4
I20260129 02:59:33.309512 22213 raft_consensus.cc:740] T a5a05fbcbbcf4916bbc01fd1a5b5e90d P 1c50de89a5a845668696dcb28c3badd6 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 1c50de89a5a845668696dcb28c3badd6, State: Running, Role: LEADER
W20260129 02:59:33.276588 20440 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55672 (request call id 1401) took 2374 ms (client timeout 2071 ms). Trace:
I20260129 02:59:33.294584 20439 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:48356 (request call id 218) took 1051 ms. Trace:
W20260129 02:59:33.310215 20440 rpcz_store.cc:269] 0129 02:59:30.896279 (+ 0us) service_pool.cc:168] Inserting onto call queue
0129 02:59:33.263410 (+2367131us) service_pool.cc:212] Skipping call since client already timed out
0129 02:59:33.270315 (+ 6905us) inbound_call.cc:173] Queueing failure response
Metrics: {}
W20260129 02:59:33.272686 20587 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:48526 (request call id 133) took 2686 ms (client timeout 2233 ms). Trace:
W20260129 02:59:33.278679 21099 leader_election.cc:336] T 42d27a28d89b409e8807ec0efd97af69 P 7a9daf5d42a84615a6383cfec9a10608 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 197b880fa39d432a9b472845876bfaf0 (127.18.211.1:42027): Timed out: RequestConsensusVote RPC to 127.18.211.1:42027 timed out after 1.744s (SENT)
I20260129 02:59:33.310272 20439 rpcz_store.cc:276] 0129 02:59:32.243096 (+ 0us) service_pool.cc:168] Inserting onto call queue
0129 02:59:33.293231 (+1050135us) service_pool.cc:225] Handling call
0129 02:59:33.294508 (+ 1277us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260129 02:59:33.278889 20588 raft_consensus.cc:1240] T cb0b3f259ebc4310927338a8740e87b8 P 1c50de89a5a845668696dcb28c3badd6 [term 4 FOLLOWER]: Rejecting Update request from peer 974b802141bd4f13beffef2aa03f88f9 for earlier term 3. Current term is 4. Ops: []
I20260129 02:59:33.280229 22102 leader_election.cc:290] T c2633a6fc34c44579de348d208b0c01e P 1c50de89a5a845668696dcb28c3badd6 [CANDIDATE]: Term 2 election: Requested vote from peers 197b880fa39d432a9b472845876bfaf0 (127.18.211.1:42027), 974b802141bd4f13beffef2aa03f88f9 (127.18.211.2:34517)
I20260129 02:59:33.280483 20434 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1ce4ddf5b10e4ae8b193b13b2e5936ee" candidate_uuid: "1c50de89a5a845668696dcb28c3badd6" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "197b880fa39d432a9b472845876bfaf0"
I20260129 02:59:33.281692 20480 leader_election.cc:304] T 4e420f90884f4796997b5ea41e49e411 P 974b802141bd4f13beffef2aa03f88f9 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 974b802141bd4f13beffef2aa03f88f9; no voters: 197b880fa39d432a9b472845876bfaf0, 7a9daf5d42a84615a6383cfec9a10608
I20260129 02:59:33.275421 20338 rpcz_store.cc:276] 0129 02:59:25.553296 (+ 0us) service_pool.cc:168] Inserting onto call queue
0129 02:59:25.553506 (+ 210us) service_pool.cc:225] Handling call
0129 02:59:25.865461 (+311955us) spinlock_profiling.cc:243] Waited 76.4 ms on lock 0x7b54003ac854. stack: 00007f8fe044e373 00007f8fe044e270 00007f8fdfda0574 00007f8fe9a5c6b4 00007f8fe9a5c649 00007f8fe34980c0 00007f8fe3536bba 00007f8fe92b7eac 00007f8fe93b210e 00007f8fe78ad4a4 00007f8fe78ad432 00007f8fe78ad361 00007f8fe78ad2dc 00007f8fe78ac592 00007f8fe186d34c 00007f8fe186c786
0129 02:59:33.072275 (+7206814us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0129 02:59:27.082094 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0129 02:59:27.082833 (+ 739us) write_op.cc:432] Acquiring schema lock in shared mode
0129 02:59:27.082891 (+ 58us) write_op.cc:435] Acquired schema lock
0129 02:59:27.082919 (+ 28us) tablet.cc:660] Decoding operations
0129 02:59:27.083390 (+ 471us) write_op.cc:620] Acquiring the partition lock for write op
0129 02:59:27.083498 (+ 108us) write_op.cc:641] Partition lock acquired for write op
0129 02:59:27.083551 (+ 53us) tablet.cc:683] Acquiring locks for 1 operations
0129 02:59:27.083790 (+ 239us) tablet.cc:699] Row locks acquired
0129 02:59:27.083818 (+ 28us) write_op.cc:260] PREPARE: finished
0129 02:59:27.083997 (+ 179us) write_op.cc:270] Start()
0129 02:59:27.084106 (+ 109us) write_op.cc:276] Timestamp: P: 1769655567083951 usec, L: 0
0129 02:59:27.084129 (+ 23us) op_driver.cc:348] REPLICATION: starting
0129 02:59:27.084964 (+ 835us) log.cc:844] Serialized 533 byte log entry
0129 02:59:27.560147 (+475183us) op_driver.cc:464] REPLICATION: finished
0129 02:59:28.758117 (+1197970us) write_op.cc:301] APPLY: starting
0129 02:59:28.772168 (+ 14051us) tablet.cc:1365] starting BulkCheckPresence
0129 02:59:28.814994 (+ 42826us) tablet.cc:1368] finished BulkCheckPresence
0129 02:59:28.842556 (+ 27562us) tablet.cc:1370] starting ApplyRowOperation cycle
0129 02:59:28.895703 (+ 53147us) tablet.cc:1381] finished ApplyRowOperation cycle
0129 02:59:28.909794 (+ 14091us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0129 02:59:28.909888 (+ 94us) write_op.cc:312] APPLY: finished
0129 02:59:28.932036 (+ 22148us) log.cc:844] Serialized 23 byte log entry
0129 02:59:29.706474 (+774438us) write_op.cc:489] Releasing partition, row and schema locks
0129 02:59:29.723412 (+ 16938us) write_op.cc:454] Released schema lock
0129 02:59:29.726088 (+ 2676us) write_op.cc:341] FINISH: Updating metrics
0129 02:59:29.843511 (+117423us) write_op.cc:489] Releasing partition, row and schema locks
0129 02:59:29.845506 (+ 1995us) write_op.cc:454] Released schema lock
0129 02:59:32.735717 (+2890211us) spinlock_profiling.cc:243] Waited 25.5 ms on lock 0x7b440002304c. stack: 00007f8fe044e373 00007f8fe044e270 00007f8fdfda0574 00007f8fe9a5c6b4 00007f8fe9a5c649 00007f8fe02c08b1 00007f8fe038f005 00007f8fe038fa22 00007f8fe30d957b 00007f8fe30d4e4f 00007f8fe30c8c0e 00007f8fe30c7f56 00007f8fe30c7b78 00007f8fe30d15c1 00007f8fe30d1579 00007f8fe30d1509
Metrics: {"mutex_wait_us":939334,"reported_tablets":1,"spinlock_wait_cycles":175688448,"child_traces":[["op",{"apply.queue_time_us":1146962,"apply.run_cpu_time_us":25013,"apply.run_wall_time_us":1087725,"num_ops":1,"prepare.queue_time_us":212041,"prepare.run_cpu_time_us":23041,"prepare.run_wall_time_us":127875,"raft.queue_time_us":307111,"raft.run_cpu_time_us":156728,"raft.run_wall_time_us":1166839,"replication_time_us":475856,"spinlock_wait_cycles":58682624,"thread_start_us":2385601,"threads_started":6,"wal-append.queue_time_us":725415,"wal-append.run_cpu_time_us":16935,"wal-append.run_wall_time_us":1368355}]]}
I20260129 02:59:33.244596 22257 raft_consensus.cc:493] T 00e79d8f25a34676b1d866eb2068a2b2 P 7a9daf5d42a84615a6383cfec9a10608 [term 3 FOLLOWER]: Starting leader election (detected failure of leader 197b880fa39d432a9b472845876bfaf0)
I20260129 02:59:33.282310 22247 raft_consensus.cc:2804] T a4c1226b987b468fa84dfc6ba0e2dbe1 P 1c50de89a5a845668696dcb28c3badd6 [term 3 FOLLOWER]: Leader election won for term 3
I20260129 02:59:33.289660 20436 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:48356 (request call id 220) took 1045 ms. Trace:
W20260129 02:59:33.299496 20438 rpcz_store.cc:269] 0129 02:59:30.925561 (+ 0us) service_pool.cc:168] Inserting onto call queue
0129 02:59:33.261739 (+2336178us) service_pool.cc:212] Skipping call since client already timed out
0129 02:59:33.261861 (+ 122us) inbound_call.cc:173] Queueing failure response
Metrics: {}
W20260129 02:59:33.268102 20441 rpcz_store.cc:269] 0129 02:59:30.718834 (+ 0us) service_pool.cc:168] Inserting onto call queue
0129 02:59:31.798065 (+1079231us) service_pool.cc:225] Handling call
0129 02:59:33.189596 (+1391531us) spinlock_profiling.cc:243] Waited 168 ms on lock 0x7b540026e654. stack: 00007f8fe044e373 00007f8fe044e270 00007f8fdfda0574 00007f8fe9a5c6b4 00007f8fe9a5c649 00007f8fe34975a1 00007f8fe3531f7a 00007f8fe84e3647 00007f8fe1a73834 00007f8fe1a737c2 00007f8fe1a736f1 00007f8fe1a7366c 00007f8fe1a72922 00007f8fe186d34c 00007f8fe186c786 00007f8fe186c12f
0129 02:59:33.189740 (+ 144us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":385677056}
I20260129 02:59:33.311988 20582 leader_election.cc:290] T 36ff33be666a4c82954731691dbcf327 P 1c50de89a5a845668696dcb28c3badd6 [CANDIDATE]: Term 2 election: Requested vote from peers 197b880fa39d432a9b472845876bfaf0 (127.18.211.1:42027), 974b802141bd4f13beffef2aa03f88f9 (127.18.211.2:34517)
I20260129 02:59:33.312539 22243 raft_consensus.cc:2749] T c891d2bff4ce4568861bbc992e91fbb0 P 1c50de89a5a845668696dcb28c3badd6 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260129 02:59:33.312855 22213 consensus_queue.cc:260] T a5a05fbcbbcf4916bbc01fd1a5b5e90d P 1c50de89a5a845668696dcb28c3badd6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:59:33.314050 20514 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "96609ac68bdf4993a12e1c23355ca013" candidate_uuid: "197b880fa39d432a9b472845876bfaf0" candidate_term: 4 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "974b802141bd4f13beffef2aa03f88f9" is_pre_election: true
I20260129 02:59:33.314311 21100 leader_election.cc:304] T 42d27a28d89b409e8807ec0efd97af69 P 7a9daf5d42a84615a6383cfec9a10608 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7a9daf5d42a84615a6383cfec9a10608; no voters: 197b880fa39d432a9b472845876bfaf0, 1c50de89a5a845668696dcb28c3badd6
I20260129 02:59:33.315069 20515 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0678b8b468bd43cdbc3e485900844821" candidate_uuid: "197b880fa39d432a9b472845876bfaf0" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "974b802141bd4f13beffef2aa03f88f9" is_pre_election: true
I20260129 02:59:33.315434 20512 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2ac659d732314e1288849f08fe5e074e" candidate_uuid: "197b880fa39d432a9b472845876bfaf0" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "974b802141bd4f13beffef2aa03f88f9" is_pre_election: true
I20260129 02:59:33.317431 20508 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fcddab6bca684adf8ea032289a00f90e" candidate_uuid: "197b880fa39d432a9b472845876bfaf0" candidate_term: 7 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "974b802141bd4f13beffef2aa03f88f9" is_pre_election: true
I20260129 02:59:33.317559 22190 raft_consensus.cc:2749] T a1a39dcd328e4268ae41b988c66b4a75 P 197b880fa39d432a9b472845876bfaf0 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260129 02:59:33.317464 20588 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35752 (request call id 1438) took 1334 ms. Trace:
I20260129 02:59:33.352443 20588 rpcz_store.cc:276] 0129 02:59:31.982977 (+ 0us) service_pool.cc:168] Inserting onto call queue
0129 02:59:31.983146 (+ 169us) service_pool.cc:225] Handling call
0129 02:59:33.278542 (+1295396us) raft_consensus.cc:1471] Updating replica for 0 ops
0129 02:59:33.316009 (+ 37467us) raft_consensus.cc:1697] Filling consensus response to leader.
0129 02:59:33.317272 (+ 1263us) spinlock_profiling.cc:243] Waited 1.29 s on lock 0x7b5400109850. stack: 00007f8fe044e373 00007f8fe044e270 00007f8fdfda0574 00007f8fe9a5c6b4 00007f8fe9a5c649 00007f8fe34975a1 00007f8fe352d52d 00007f8fe84e1f49 00007f8fe1a6ed34 00007f8fe1a6ecc2 00007f8fe1a6ebf1 00007f8fe1a6eb6c 00007f8fe1a6de22 00007f8fe186d34c 00007f8fe186c786 00007f8fe186c12f
0129 02:59:33.317408 (+ 136us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2961334016}
I20260129 02:59:33.318113 20511 raft_consensus.cc:1240] T a4c1226b987b468fa84dfc6ba0e2dbe1 P 974b802141bd4f13beffef2aa03f88f9 [term 3 FOLLOWER]: Rejecting Update request from peer 197b880fa39d432a9b472845876bfaf0 for earlier term 2. Current term is 3. Ops: [2.2-2.2]
I20260129 02:59:33.354511 20511 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "724a236524cb4e7ba3428880681d6cc2" candidate_uuid: "197b880fa39d432a9b472845876bfaf0" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "974b802141bd4f13beffef2aa03f88f9" is_pre_election: true
I20260129 02:59:33.355170 22222 raft_consensus.cc:2749] T 42d27a28d89b409e8807ec0efd97af69 P 7a9daf5d42a84615a6383cfec9a10608 [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
W20260129 02:59:33.355581 21100 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.211.3:37735, user_credentials={real_user=slave}} blocked reactor thread for 76246.3us
I20260129 02:59:33.318231 20432 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a0fcc58d6fb6473cafbd9d34d5195269" candidate_uuid: "1c50de89a5a845668696dcb28c3badd6" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "197b880fa39d432a9b472845876bfaf0" is_pre_election: true
W20260129 02:59:33.357710 20432 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55676 (request call id 1095) took 1644 ms (client timeout 1676 ms). Trace:
W20260129 02:59:33.358198 20432 rpcz_store.cc:269] 0129 02:59:31.713208 (+ 0us) service_pool.cc:168] Inserting onto call queue
0129 02:59:33.304895 (+1591687us) service_pool.cc:225] Handling call
0129 02:59:33.357617 (+ 52722us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260129 02:59:33.358584 20508 raft_consensus.cc:2468] T fcddab6bca684adf8ea032289a00f90e P 974b802141bd4f13beffef2aa03f88f9 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 197b880fa39d432a9b472845876bfaf0 in term 6.
I20260129 02:59:33.359371 20432 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e7d83d4ccf684e76acb29a0dc844e1b8" candidate_uuid: "974b802141bd4f13beffef2aa03f88f9" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "197b880fa39d432a9b472845876bfaf0" is_pre_election: true
I20260129 02:59:33.318785 20510 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e33a417cf9c54cf5af73a4ee3aac4d3a" candidate_uuid: "197b880fa39d432a9b472845876bfaf0" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "974b802141bd4f13beffef2aa03f88f9" is_pre_election: true
I20260129 02:59:33.312554 20439 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "78aa39096868415d9afc2ffcb248e198" candidate_uuid: "7a9daf5d42a84615a6383cfec9a10608" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "197b880fa39d432a9b472845876bfaf0" is_pre_election: true
I20260129 02:59:33.343322 22255 raft_consensus.cc:515] T 75b0b6e82ddc4f92b1fe1d94f9a0b73b P 7a9daf5d42a84615a6383cfec9a10608 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } }
I20260129 02:59:33.360492 20510 raft_consensus.cc:2468] T e33a417cf9c54cf5af73a4ee3aac4d3a P 974b802141bd4f13beffef2aa03f88f9 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 197b880fa39d432a9b472845876bfaf0 in term 3.
I20260129 02:59:33.360591 20439 raft_consensus.cc:2468] T 78aa39096868415d9afc2ffcb248e198 P 197b880fa39d432a9b472845876bfaf0 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7a9daf5d42a84615a6383cfec9a10608 in term 3.
I20260129 02:59:33.361068 20439 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:48356 (request call id 219) took 1117 ms. Trace:
I20260129 02:59:33.324909 22257 raft_consensus.cc:3060] T 00e79d8f25a34676b1d866eb2068a2b2 P 7a9daf5d42a84615a6383cfec9a10608 [term 3 FOLLOWER]: Advancing to term 4
I20260129 02:59:33.357712 21100 leader_election.cc:304] T 96609ac68bdf4993a12e1c23355ca013 P 7a9daf5d42a84615a6383cfec9a10608 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 1c50de89a5a845668696dcb28c3badd6, 7a9daf5d42a84615a6383cfec9a10608, 974b802141bd4f13beffef2aa03f88f9; no voters:
I20260129 02:59:33.325217 20480 leader_election.cc:304] T febffe4a8b394368b69b1d164308fb9a P 974b802141bd4f13beffef2aa03f88f9 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7a9daf5d42a84615a6383cfec9a10608, 974b802141bd4f13beffef2aa03f88f9; no voters:
I20260129 02:59:33.325251 20436 rpcz_store.cc:276] 0129 02:59:32.243772 (+ 0us) service_pool.cc:168] Inserting onto call queue
0129 02:59:33.288364 (+1044592us) service_pool.cc:225] Handling call
0129 02:59:33.289562 (+ 1198us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260129 02:59:33.325850 20434 raft_consensus.cc:3060] T 1ce4ddf5b10e4ae8b193b13b2e5936ee P 197b880fa39d432a9b472845876bfaf0 [term 3 FOLLOWER]: Advancing to term 4
I20260129 02:59:33.326124 22164 raft_consensus.cc:2804] T 3906b080a5b34a74b8b64128635e8997 P 1c50de89a5a845668696dcb28c3badd6 [term 2 FOLLOWER]: Leader election won for term 2
I20260129 02:59:33.326745 20441 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "36ff33be666a4c82954731691dbcf327" candidate_uuid: "1c50de89a5a845668696dcb28c3badd6" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "197b880fa39d432a9b472845876bfaf0" is_pre_election: true
I20260129 02:59:33.326963 22112 raft_consensus.cc:2749] T 4e420f90884f4796997b5ea41e49e411 P 974b802141bd4f13beffef2aa03f88f9 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260129 02:59:33.327342 22122 raft_consensus.cc:515] T 4f5a77e94fef4b16a2ba7b067c8b144a P 7a9daf5d42a84615a6383cfec9a10608 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } }
I20260129 02:59:33.329439 20513 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a5a05fbcbbcf4916bbc01fd1a5b5e90d" candidate_uuid: "197b880fa39d432a9b472845876bfaf0" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "974b802141bd4f13beffef2aa03f88f9" is_pre_election: true
I20260129 02:59:33.330035 20516 raft_consensus.cc:1240] T 3906b080a5b34a74b8b64128635e8997 P 974b802141bd4f13beffef2aa03f88f9 [term 2 FOLLOWER]: Rejecting Update request from peer 197b880fa39d432a9b472845876bfaf0 for earlier term 1. Current term is 2. Ops: []
W20260129 02:59:33.323174 20556 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.211.2:34517, user_credentials={real_user=slave}} blocked reactor thread for 47201.5us
I20260129 02:59:33.336450 20507 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "926a250fa5834e88b96ccecbefbf5086" candidate_uuid: "197b880fa39d432a9b472845876bfaf0" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "974b802141bd4f13beffef2aa03f88f9" is_pre_election: true
I20260129 02:59:33.337123 20440 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a609ac02fae84725ab743f95f51d33ce" candidate_uuid: "974b802141bd4f13beffef2aa03f88f9" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "197b880fa39d432a9b472845876bfaf0" is_pre_election: true
I20260129 02:59:33.337563 22120 raft_consensus.cc:493] T e7d83d4ccf684e76acb29a0dc844e1b8 P 197b880fa39d432a9b472845876bfaf0 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 1c50de89a5a845668696dcb28c3badd6)
I20260129 02:59:33.371385 22213 raft_consensus.cc:3060] T a5a05fbcbbcf4916bbc01fd1a5b5e90d P 1c50de89a5a845668696dcb28c3badd6 [term 4 FOLLOWER]: Advancing to term 5
I20260129 02:59:33.371699 22120 raft_consensus.cc:515] T e7d83d4ccf684e76acb29a0dc844e1b8 P 197b880fa39d432a9b472845876bfaf0 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:59:33.374202 22120 leader_election.cc:290] T e7d83d4ccf684e76acb29a0dc844e1b8 P 197b880fa39d432a9b472845876bfaf0 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 974b802141bd4f13beffef2aa03f88f9 (127.18.211.2:34517), 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3:37735)
I20260129 02:59:33.376312 20513 raft_consensus.cc:2393] T a5a05fbcbbcf4916bbc01fd1a5b5e90d P 974b802141bd4f13beffef2aa03f88f9 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 197b880fa39d432a9b472845876bfaf0 in current term 5: Already voted for candidate 974b802141bd4f13beffef2aa03f88f9 in this term.
I20260129 02:59:33.337834 20433 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0340229d521b432580ae926f6062e3f7" candidate_uuid: "1c50de89a5a845668696dcb28c3badd6" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "197b880fa39d432a9b472845876bfaf0" is_pre_election: true
I20260129 02:59:33.338444 20405 leader_election.cc:304] T 36ff33be666a4c82954731691dbcf327 P 197b880fa39d432a9b472845876bfaf0 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 197b880fa39d432a9b472845876bfaf0, 1c50de89a5a845668696dcb28c3badd6; no voters: 974b802141bd4f13beffef2aa03f88f9
W20260129 02:59:33.377406 20433 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55676 (request call id 1093) took 1694 ms (client timeout 2076 ms). Trace:
I20260129 02:59:33.378060 20405 leader_election.cc:304] T a8479d1fe0bd41e5946186e3a6ca8c7a P 197b880fa39d432a9b472845876bfaf0 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 197b880fa39d432a9b472845876bfaf0, 1c50de89a5a845668696dcb28c3badd6; no voters:
I20260129 02:59:33.378291 20432 raft_consensus.cc:2468] T e7d83d4ccf684e76acb29a0dc844e1b8 P 197b880fa39d432a9b472845876bfaf0 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 974b802141bd4f13beffef2aa03f88f9 in term 2.
I20260129 02:59:33.379094 20405 leader_election.cc:304] T 8055232cf0004d40a85f3bf983fe00a9 P 197b880fa39d432a9b472845876bfaf0 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 197b880fa39d432a9b472845876bfaf0, 1c50de89a5a845668696dcb28c3badd6; no voters: 974b802141bd4f13beffef2aa03f88f9
I20260129 02:59:33.379381 20432 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55672 (request call id 1408) took 1137 ms. Trace:
I20260129 02:59:33.379959 22092 raft_consensus.cc:2804] T 36ff33be666a4c82954731691dbcf327 P 197b880fa39d432a9b472845876bfaf0 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260129 02:59:33.380110 22190 raft_consensus.cc:2804] T a8479d1fe0bd41e5946186e3a6ca8c7a P 197b880fa39d432a9b472845876bfaf0 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260129 02:59:33.380427 22092 raft_consensus.cc:493] T 36ff33be666a4c82954731691dbcf327 P 197b880fa39d432a9b472845876bfaf0 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 974b802141bd4f13beffef2aa03f88f9)
I20260129 02:59:33.330652 20437 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3ccac82a01a2445fa5b0ce6c0fcf2295" candidate_uuid: "7a9daf5d42a84615a6383cfec9a10608" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "197b880fa39d432a9b472845876bfaf0" is_pre_election: true
I20260129 02:59:33.380764 20510 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e4d18c35b0a7493e85d0a4b92956a48f" candidate_uuid: "197b880fa39d432a9b472845876bfaf0" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "974b802141bd4f13beffef2aa03f88f9"
I20260129 02:59:33.380960 22222 raft_consensus.cc:2804] T 96609ac68bdf4993a12e1c23355ca013 P 7a9daf5d42a84615a6383cfec9a10608 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260129 02:59:33.343990 20514 raft_consensus.cc:2468] T 96609ac68bdf4993a12e1c23355ca013 P 974b802141bd4f13beffef2aa03f88f9 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 197b880fa39d432a9b472845876bfaf0 in term 3.
I20260129 02:59:33.381541 22222 raft_consensus.cc:493] T 96609ac68bdf4993a12e1c23355ca013 P 7a9daf5d42a84615a6383cfec9a10608 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260129 02:59:33.348194 20515 raft_consensus.cc:2468] T 0678b8b468bd43cdbc3e485900844821 P 974b802141bd4f13beffef2aa03f88f9 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 197b880fa39d432a9b472845876bfaf0 in term 4.
I20260129 02:59:33.381909 22222 raft_consensus.cc:3060] T 96609ac68bdf4993a12e1c23355ca013 P 7a9daf5d42a84615a6383cfec9a10608 [term 3 FOLLOWER]: Advancing to term 4
W20260129 02:59:33.310910 20587 rpcz_store.cc:269] 0129 02:59:30.586438 (+ 0us) service_pool.cc:168] Inserting onto call queue
0129 02:59:30.586776 (+ 338us) service_pool.cc:225] Handling call
0129 02:59:33.262919 (+2676143us) spinlock_profiling.cc:243] Waited 2.52 s on lock 0x7b5400109854. stack: 00007f8fe044e373 00007f8fe044e270 00007f8fdfda0574 00007f8fe9a5c6b4 00007f8fe9a5c649 00007f8fe34975a1 00007f8fe3531f7a 00007f8fe84e3647 00007f8fe1a73834 00007f8fe1a737c2 00007f8fe1a736f1 00007f8fe1a7366c 00007f8fe1a72922 00007f8fe186d34c 00007f8fe186c786 00007f8fe186c12f
0129 02:59:33.272597 (+ 9678us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":5787864320}
I20260129 02:59:33.324115 20438 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3906b080a5b34a74b8b64128635e8997" candidate_uuid: "974b802141bd4f13beffef2aa03f88f9" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "197b880fa39d432a9b472845876bfaf0" is_pre_election: true
I20260129 02:59:33.382395 22164 raft_consensus.cc:697] T 3906b080a5b34a74b8b64128635e8997 P 1c50de89a5a845668696dcb28c3badd6 [term 2 LEADER]: Becoming Leader. State: Replica: 1c50de89a5a845668696dcb28c3badd6, State: Running, Role: LEADER
I20260129 02:59:33.382952 20438 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55672 (request call id 1406) took 1142 ms. Trace:
I20260129 02:59:33.383329 20438 rpcz_store.cc:276] 0129 02:59:32.240876 (+ 0us) service_pool.cc:168] Inserting onto call queue
0129 02:59:33.323777 (+1082901us) service_pool.cc:225] Handling call
0129 02:59:33.382893 (+ 59116us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260129 02:59:33.339211 22247 raft_consensus.cc:697] T a4c1226b987b468fa84dfc6ba0e2dbe1 P 1c50de89a5a845668696dcb28c3badd6 [term 3 LEADER]: Becoming Leader. State: Replica: 1c50de89a5a845668696dcb28c3badd6, State: Running, Role: LEADER
I20260129 02:59:33.384289 20438 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b189f08e16a74a11980cc55d99359a0a" candidate_uuid: "1c50de89a5a845668696dcb28c3badd6" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "197b880fa39d432a9b472845876bfaf0" is_pre_election: true
I20260129 02:59:33.383927 22164 consensus_queue.cc:237] T 3906b080a5b34a74b8b64128635e8997 P 1c50de89a5a845668696dcb28c3badd6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:59:33.384588 22247 consensus_queue.cc:237] T a4c1226b987b468fa84dfc6ba0e2dbe1 P 1c50de89a5a845668696dcb28c3badd6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:59:33.385556 20438 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1299723a4476420ab02fd8bfd7aa0645" candidate_uuid: "7a9daf5d42a84615a6383cfec9a10608" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "197b880fa39d432a9b472845876bfaf0" is_pre_election: true
I20260129 02:59:33.386582 20438 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00e79d8f25a34676b1d866eb2068a2b2" candidate_uuid: "7a9daf5d42a84615a6383cfec9a10608" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "197b880fa39d432a9b472845876bfaf0" is_pre_election: true
I20260129 02:59:33.363288 20439 rpcz_store.cc:276] 0129 02:59:32.243450 (+ 0us) service_pool.cc:168] Inserting onto call queue
0129 02:59:33.311188 (+1067738us) service_pool.cc:225] Handling call
0129 02:59:33.361015 (+ 49827us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260129 02:59:33.367285 21128 raft_consensus.cc:1240] T 75b0b6e82ddc4f92b1fe1d94f9a0b73b P 7a9daf5d42a84615a6383cfec9a10608 [term 2 FOLLOWER]: Rejecting Update request from peer 1c50de89a5a845668696dcb28c3badd6 for earlier term 1. Current term is 2. Ops: []
I20260129 02:59:33.381395 20437 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:48356 (request call id 216) took 1139 ms. Trace:
I20260129 02:59:33.379734 20432 rpcz_store.cc:276] 0129 02:59:32.242162 (+ 0us) service_pool.cc:168] Inserting onto call queue
0129 02:59:33.359033 (+1116871us) service_pool.cc:225] Handling call
0129 02:59:33.379195 (+ 20162us) spinlock_profiling.cc:243] Waited 18 ms on lock 0x7b54003106d4. stack: 00007f8fe044e373 00007f8fe044e270 00007f8fdfda0574 00007f8fe9a5c6b4 00007f8fe9a5c649 00007f8fe34975a1 00007f8fe3531f7a 00007f8fe84e3647 00007f8fe1a73834 00007f8fe1a737c2 00007f8fe1a736f1 00007f8fe1a7366c 00007f8fe1a72922 00007f8fe186d34c 00007f8fe186c786 00007f8fe186c12f
0129 02:59:33.379313 (+ 118us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":41499520}
I20260129 02:59:33.370940 20556 leader_election.cc:304] T 0340229d521b432580ae926f6062e3f7 P 1c50de89a5a845668696dcb28c3badd6 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1c50de89a5a845668696dcb28c3badd6, 974b802141bd4f13beffef2aa03f88f9; no voters:
I20260129 02:59:33.394438 20437 rpcz_store.cc:276] 0129 02:59:32.242373 (+ 0us) service_pool.cc:168] Inserting onto call queue
0129 02:59:33.305093 (+1062720us) service_pool.cc:225] Handling call
0129 02:59:33.381334 (+ 76241us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20260129 02:59:33.394658 20554 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.211.4:40439, user_credentials={real_user=slave}} blocked reactor thread for 53272.2us
W20260129 02:59:33.377792 20433 rpcz_store.cc:269] 0129 02:59:31.683183 (+ 0us) service_pool.cc:168] Inserting onto call queue
0129 02:59:33.299259 (+1616076us) service_pool.cc:225] Handling call
0129 02:59:33.377327 (+ 78068us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260129 02:59:33.381054 22092 raft_consensus.cc:3060] T 36ff33be666a4c82954731691dbcf327 P 197b880fa39d432a9b472845876bfaf0 [term 1 FOLLOWER]: Advancing to term 2
I20260129 02:59:33.388787 20439 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4e3805ffa70242e8ade66da260dddf66" candidate_uuid: "1c50de89a5a845668696dcb28c3badd6" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "197b880fa39d432a9b472845876bfaf0"
I20260129 02:59:33.370388 20507 raft_consensus.cc:2468] T 926a250fa5834e88b96ccecbefbf5086 P 974b802141bd4f13beffef2aa03f88f9 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 197b880fa39d432a9b472845876bfaf0 in term 4.
I20260129 02:59:33.339766 20435 raft_consensus.cc:2376] T 75be4bd6bd45474795fd472eff2953ac P 197b880fa39d432a9b472845876bfaf0 [term 7 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 1c50de89a5a845668696dcb28c3badd6 in term 7. Re-sending same reply.
I20260129 02:59:33.396893 20435 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55676 (request call id 1100) took 1153 ms. Trace:
I20260129 02:59:33.397342 20435 rpcz_store.cc:276] 0129 02:59:32.243840 (+ 0us) service_pool.cc:168] Inserting onto call queue
0129 02:59:33.302693 (+1058853us) service_pool.cc:225] Handling call
0129 02:59:33.396815 (+ 94122us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260129 02:59:33.387566 20438 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "febffe4a8b394368b69b1d164308fb9a" candidate_uuid: "7a9daf5d42a84615a6383cfec9a10608" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "197b880fa39d432a9b472845876bfaf0" is_pre_election: true
I20260129 02:59:33.371136 20440 raft_consensus.cc:2468] T a609ac02fae84725ab743f95f51d33ce P 197b880fa39d432a9b472845876bfaf0 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 974b802141bd4f13beffef2aa03f88f9 in term 4.
I20260129 02:59:33.399163 20440 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55672 (request call id 1407) took 1157 ms. Trace:
I20260129 02:59:33.399668 20440 rpcz_store.cc:276] 0129 02:59:32.241715 (+ 0us) service_pool.cc:168] Inserting onto call queue
0129 02:59:33.321996 (+1080281us) service_pool.cc:225] Handling call
0129 02:59:33.399083 (+ 77087us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260129 02:59:33.363574 22255 leader_election.cc:290] T 75b0b6e82ddc4f92b1fe1d94f9a0b73b P 7a9daf5d42a84615a6383cfec9a10608 [CANDIDATE]: Term 2 election: Requested vote from peers 974b802141bd4f13beffef2aa03f88f9 (127.18.211.2:34517), 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3:37735)
I20260129 02:59:33.406173 20432 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "42d27a28d89b409e8807ec0efd97af69" candidate_uuid: "7a9daf5d42a84615a6383cfec9a10608" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "197b880fa39d432a9b472845876bfaf0" is_pre_election: true
I20260129 02:59:33.388013 22137 raft_consensus.cc:2804] T febffe4a8b394368b69b1d164308fb9a P 974b802141bd4f13beffef2aa03f88f9 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260129 02:59:33.380606 22190 raft_consensus.cc:493] T a8479d1fe0bd41e5946186e3a6ca8c7a P 197b880fa39d432a9b472845876bfaf0 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 974b802141bd4f13beffef2aa03f88f9)
I20260129 02:59:33.411044 20435 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2ac659d732314e1288849f08fe5e074e" candidate_uuid: "7a9daf5d42a84615a6383cfec9a10608" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "197b880fa39d432a9b472845876bfaf0" is_pre_election: true
I20260129 02:59:33.394320 22122 leader_election.cc:290] T 4f5a77e94fef4b16a2ba7b067c8b144a P 7a9daf5d42a84615a6383cfec9a10608 [CANDIDATE]: Term 2 election: Requested vote from peers 974b802141bd4f13beffef2aa03f88f9 (127.18.211.2:34517), 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3:37735)
I20260129 02:59:33.412746 20480 leader_election.cc:304] T 3ccac82a01a2445fa5b0ce6c0fcf2295 P 974b802141bd4f13beffef2aa03f88f9 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7a9daf5d42a84615a6383cfec9a10608, 974b802141bd4f13beffef2aa03f88f9; no voters:
W20260129 02:59:33.416118 20407 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.211.2:34517, user_credentials={real_user=slave}} blocked reactor thread for 77974.1us
I20260129 02:59:33.413201 20433 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "78aa39096868415d9afc2ffcb248e198" candidate_uuid: "7a9daf5d42a84615a6383cfec9a10608" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "197b880fa39d432a9b472845876bfaf0" is_pre_election: true
I20260129 02:59:33.414405 20437 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3ccac82a01a2445fa5b0ce6c0fcf2295" candidate_uuid: "7a9daf5d42a84615a6383cfec9a10608" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "197b880fa39d432a9b472845876bfaf0" is_pre_election: true
I20260129 02:59:33.416404 20439 raft_consensus.cc:3060] T 4e3805ffa70242e8ade66da260dddf66 P 197b880fa39d432a9b472845876bfaf0 [term 2 FOLLOWER]: Advancing to term 3
I20260129 02:59:33.416252 22243 raft_consensus.cc:2804] T 0340229d521b432580ae926f6062e3f7 P 1c50de89a5a845668696dcb28c3badd6 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260129 02:59:33.416939 20433 raft_consensus.cc:2468] T 78aa39096868415d9afc2ffcb248e198 P 197b880fa39d432a9b472845876bfaf0 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7a9daf5d42a84615a6383cfec9a10608 in term 3.
I20260129 02:59:33.419368 20507 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4408006eb5174de1896b037cf3f18457" candidate_uuid: "197b880fa39d432a9b472845876bfaf0" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "974b802141bd4f13beffef2aa03f88f9" is_pre_election: true
I20260129 02:59:33.421921 20440 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1299723a4476420ab02fd8bfd7aa0645" candidate_uuid: "974b802141bd4f13beffef2aa03f88f9" candidate_term: 5 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "197b880fa39d432a9b472845876bfaf0" is_pre_election: true
I20260129 02:59:33.423687 20405 leader_election.cc:304] T cb4727386a3e48aca5d71cc6c611c197 P 197b880fa39d432a9b472845876bfaf0 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 197b880fa39d432a9b472845876bfaf0, 1c50de89a5a845668696dcb28c3badd6; no voters:
I20260129 02:59:33.380492 21100 leader_election.cc:304] T 1299723a4476420ab02fd8bfd7aa0645 P 7a9daf5d42a84615a6383cfec9a10608 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 7a9daf5d42a84615a6383cfec9a10608, 974b802141bd4f13beffef2aa03f88f9; no voters: 197b880fa39d432a9b472845876bfaf0, 1c50de89a5a845668696dcb28c3badd6
I20260129 02:59:33.428923 20440 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55672 (request call id 1399) took 2669 ms. Trace:
I20260129 02:59:33.430467 20440 rpcz_store.cc:276] 0129 02:59:30.759704 (+ 0us) service_pool.cc:168] Inserting onto call queue
0129 02:59:33.403137 (+2643433us) service_pool.cc:225] Handling call
0129 02:59:33.428863 (+ 25726us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260129 02:59:33.429629 20438 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fd8311b779af4e1fa1402b2a02a602f2" candidate_uuid: "7a9daf5d42a84615a6383cfec9a10608" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "197b880fa39d432a9b472845876bfaf0" is_pre_election: true
I20260129 02:59:33.389351 22061 raft_consensus.cc:515] T 7a9a69cd82724b1faed1d10aa091fc4a P 1c50de89a5a845668696dcb28c3badd6 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:59:33.442634 22061 leader_election.cc:290] T 7a9a69cd82724b1faed1d10aa091fc4a P 1c50de89a5a845668696dcb28c3badd6 [CANDIDATE]: Term 4 election: Requested vote from peers 197b880fa39d432a9b472845876bfaf0 (127.18.211.1:42027), 974b802141bd4f13beffef2aa03f88f9 (127.18.211.2:34517)
I20260129 02:59:33.409017 22137 raft_consensus.cc:493] T febffe4a8b394368b69b1d164308fb9a P 974b802141bd4f13beffef2aa03f88f9 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 197b880fa39d432a9b472845876bfaf0)
I20260129 02:59:33.443454 22137 raft_consensus.cc:3060] T febffe4a8b394368b69b1d164308fb9a P 974b802141bd4f13beffef2aa03f88f9 [term 1 FOLLOWER]: Advancing to term 2
I20260129 02:59:33.417781 22190 raft_consensus.cc:3060] T a8479d1fe0bd41e5946186e3a6ca8c7a P 197b880fa39d432a9b472845876bfaf0 [term 2 FOLLOWER]: Advancing to term 3
I20260129 02:59:33.417342 20436 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "febffe4a8b394368b69b1d164308fb9a" candidate_uuid: "974b802141bd4f13beffef2aa03f88f9" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "197b880fa39d432a9b472845876bfaf0" is_pre_election: true
I20260129 02:59:33.433847 22069 raft_consensus.cc:2804] T cb4727386a3e48aca5d71cc6c611c197 P 197b880fa39d432a9b472845876bfaf0 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260129 02:59:33.434970 21130 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "96609ac68bdf4993a12e1c23355ca013" candidate_uuid: "197b880fa39d432a9b472845876bfaf0" candidate_term: 4 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "7a9daf5d42a84615a6383cfec9a10608" is_pre_election: true
I20260129 02:59:33.435405 20438 raft_consensus.cc:1240] T 76f713c3331540d2a5b76164103e9eb1 P 197b880fa39d432a9b472845876bfaf0 [term 3 LEADER]: Rejecting Update request from peer 1c50de89a5a845668696dcb28c3badd6 for earlier term 2. Current term is 3. Ops: []
I20260129 02:59:33.479017 20438 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55676 (request call id 1071) took 3799 ms. Trace:
I20260129 02:59:33.479355 20438 rpcz_store.cc:276] 0129 02:59:29.680406 (+ 0us) service_pool.cc:168] Inserting onto call queue
0129 02:59:33.434835 (+3754429us) service_pool.cc:225] Handling call
0129 02:59:33.435211 (+ 376us) raft_consensus.cc:1471] Updating replica for 0 ops
0129 02:59:33.478746 (+ 43535us) raft_consensus.cc:1697] Filling consensus response to leader.
0129 02:59:33.478940 (+ 194us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260129 02:59:33.480337 20438 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b189f08e16a74a11980cc55d99359a0a" candidate_uuid: "974b802141bd4f13beffef2aa03f88f9" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "197b880fa39d432a9b472845876bfaf0" is_pre_election: true
I20260129 02:59:33.481431 20438 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e33a417cf9c54cf5af73a4ee3aac4d3a" candidate_uuid: "974b802141bd4f13beffef2aa03f88f9" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "197b880fa39d432a9b472845876bfaf0" is_pre_election: true
I20260129 02:59:33.436271 20437 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "96609ac68bdf4993a12e1c23355ca013" candidate_uuid: "7a9daf5d42a84615a6383cfec9a10608" candidate_term: 4 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "197b880fa39d432a9b472845876bfaf0" is_pre_election: true
I20260129 02:59:33.437382 22122 raft_consensus.cc:2749] T 1299723a4476420ab02fd8bfd7aa0645 P 7a9daf5d42a84615a6383cfec9a10608 [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260129 02:59:33.482307 20437 raft_consensus.cc:2468] T 96609ac68bdf4993a12e1c23355ca013 P 197b880fa39d432a9b472845876bfaf0 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7a9daf5d42a84615a6383cfec9a10608 in term 3.
I20260129 02:59:33.483242 20437 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7a9a69cd82724b1faed1d10aa091fc4a" candidate_uuid: "974b802141bd4f13beffef2aa03f88f9" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "197b880fa39d432a9b472845876bfaf0" is_pre_election: true
I20260129 02:59:33.484871 20440 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "96609ac68bdf4993a12e1c23355ca013" candidate_uuid: "1c50de89a5a845668696dcb28c3badd6" candidate_term: 4 candidate_status { last_received { term: 2 index: 3 } } ignore_live_leader: false dest_uuid: "197b880fa39d432a9b472845876bfaf0" is_pre_election: true
W20260129 02:59:33.437781 21100 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.211.3:37735, user_credentials={real_user=slave}} blocked reactor thread for 57407.5us
I20260129 02:59:33.433845 22157 raft_consensus.cc:2804] T 3ccac82a01a2445fa5b0ce6c0fcf2295 P 974b802141bd4f13beffef2aa03f88f9 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260129 02:59:33.488957 22157 raft_consensus.cc:493] T 3ccac82a01a2445fa5b0ce6c0fcf2295 P 974b802141bd4f13beffef2aa03f88f9 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 197b880fa39d432a9b472845876bfaf0)
I20260129 02:59:33.489364 22157 raft_consensus.cc:3060] T 3ccac82a01a2445fa5b0ce6c0fcf2295 P 974b802141bd4f13beffef2aa03f88f9 [term 1 FOLLOWER]: Advancing to term 2
I20260129 02:59:33.445057 21133 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "42d27a28d89b409e8807ec0efd97af69" candidate_uuid: "197b880fa39d432a9b472845876bfaf0" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "7a9daf5d42a84615a6383cfec9a10608"
I20260129 02:59:33.443475 20511 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4b1876f9a70b49edb3aa74105e66d3b3" candidate_uuid: "197b880fa39d432a9b472845876bfaf0" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "974b802141bd4f13beffef2aa03f88f9" is_pre_election: true
I20260129 02:59:33.490928 20440 raft_consensus.cc:2410] T 96609ac68bdf4993a12e1c23355ca013 P 197b880fa39d432a9b472845876bfaf0 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1c50de89a5a845668696dcb28c3badd6 for term 4 because replica has last-logged OpId of term: 2 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 3.
I20260129 02:59:33.444778 20556 leader_election.cc:304] T 75f45538dc704d9a8ec4ff8ad9f3ea5b P 1c50de89a5a845668696dcb28c3badd6 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1c50de89a5a845668696dcb28c3badd6; no voters: 7a9daf5d42a84615a6383cfec9a10608, 974b802141bd4f13beffef2aa03f88f9
I20260129 02:59:33.460824 21136 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2ac659d732314e1288849f08fe5e074e" candidate_uuid: "197b880fa39d432a9b472845876bfaf0" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "7a9daf5d42a84615a6383cfec9a10608" is_pre_election: true
I20260129 02:59:33.462150 20433 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "368953c9102643b2a21f99af36b51905" candidate_uuid: "1c50de89a5a845668696dcb28c3badd6" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "197b880fa39d432a9b472845876bfaf0"
I20260129 02:59:33.461083 21134 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0678b8b468bd43cdbc3e485900844821" candidate_uuid: "197b880fa39d432a9b472845876bfaf0" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "7a9daf5d42a84615a6383cfec9a10608" is_pre_election: true
I20260129 02:59:33.468432 20432 raft_consensus.cc:2393] T 42d27a28d89b409e8807ec0efd97af69 P 197b880fa39d432a9b472845876bfaf0 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 7a9daf5d42a84615a6383cfec9a10608 in current term 4: Already voted for candidate 197b880fa39d432a9b472845876bfaf0 in this term.
I20260129 02:59:33.471001 20435 raft_consensus.cc:2468] T 2ac659d732314e1288849f08fe5e074e P 197b880fa39d432a9b472845876bfaf0 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7a9daf5d42a84615a6383cfec9a10608 in term 1.
I20260129 02:59:33.476397 20436 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55672 (request call id 1409) took 1233 ms. Trace:
I20260129 02:59:33.421269 22243 raft_consensus.cc:493] T 0340229d521b432580ae926f6062e3f7 P 1c50de89a5a845668696dcb28c3badd6 [term 3 FOLLOWER]: Starting leader election (detected failure of leader 197b880fa39d432a9b472845876bfaf0)
I20260129 02:59:33.444214 21127 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "dc2f4cb752cf4d59b5f6bc689f5d8104" candidate_uuid: "197b880fa39d432a9b472845876bfaf0" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "7a9daf5d42a84615a6383cfec9a10608" is_pre_election: true
I20260129 02:59:33.477043 22112 consensus_queue.cc:1059] T 505580c93c694bbb8ea96fdf9b17ab46 P 974b802141bd4f13beffef2aa03f88f9 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260129 02:59:33.500748 20435 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a5a05fbcbbcf4916bbc01fd1a5b5e90d" candidate_uuid: "974b802141bd4f13beffef2aa03f88f9" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "197b880fa39d432a9b472845876bfaf0"
I20260129 02:59:33.501057 22112 raft_consensus.cc:3055] T 505580c93c694bbb8ea96fdf9b17ab46 P 974b802141bd4f13beffef2aa03f88f9 [term 2 LEADER]: Stepping down as leader of term 2
I20260129 02:59:33.501379 20435 raft_consensus.cc:3060] T a5a05fbcbbcf4916bbc01fd1a5b5e90d P 197b880fa39d432a9b472845876bfaf0 [term 4 FOLLOWER]: Advancing to term 5
I20260129 02:59:33.443851 22222 raft_consensus.cc:515] T 96609ac68bdf4993a12e1c23355ca013 P 7a9daf5d42a84615a6383cfec9a10608 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } attrs { replace: true } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } }
I20260129 02:59:33.433858 22120 raft_consensus.cc:2804] T 8055232cf0004d40a85f3bf983fe00a9 P 197b880fa39d432a9b472845876bfaf0 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260129 02:59:33.477948 22069 raft_consensus.cc:493] T cb4727386a3e48aca5d71cc6c611c197 P 197b880fa39d432a9b472845876bfaf0 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 974b802141bd4f13beffef2aa03f88f9)
I20260129 02:59:33.516973 20582 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "add8d4923d1a489e86882c3a25692b95" candidate_uuid: "974b802141bd4f13beffef2aa03f88f9" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "1c50de89a5a845668696dcb28c3badd6"
I20260129 02:59:33.517218 22069 raft_consensus.cc:3060] T cb4727386a3e48aca5d71cc6c611c197 P 197b880fa39d432a9b472845876bfaf0 [term 1 FOLLOWER]: Advancing to term 2
I20260129 02:59:33.517910 21127 raft_consensus.cc:1240] T 75f45538dc704d9a8ec4ff8ad9f3ea5b P 7a9daf5d42a84615a6383cfec9a10608 [term 2 FOLLOWER]: Rejecting Update request from peer 974b802141bd4f13beffef2aa03f88f9 for earlier term 1. Current term is 2. Ops: []
W20260129 02:59:33.460762 20322 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.GetConsensusState -> {remote=127.18.211.1:42027, user_credentials={real_user=slave}} blocked reactor thread for 83282.4us
I20260129 02:59:33.487463 20437 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "76f713c3331540d2a5b76164103e9eb1" candidate_uuid: "974b802141bd4f13beffef2aa03f88f9" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "197b880fa39d432a9b472845876bfaf0" is_pre_election: true
I20260129 02:59:33.488437 21128 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "add8d4923d1a489e86882c3a25692b95" candidate_uuid: "974b802141bd4f13beffef2aa03f88f9" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "7a9daf5d42a84615a6383cfec9a10608"
I20260129 02:59:33.439913 22092 raft_consensus.cc:515] T 36ff33be666a4c82954731691dbcf327 P 197b880fa39d432a9b472845876bfaf0 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:59:33.529290 22231 raft_consensus.cc:2955] T 1299723a4476420ab02fd8bfd7aa0645 P 197b880fa39d432a9b472845876bfaf0 [term 4 LEADER]: Committing config change with OpId 2.5: config changed from index 4 to 5, VOTER 974b802141bd4f13beffef2aa03f88f9 (127.18.211.2) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } } }
I20260129 02:59:33.530712 20441 raft_consensus.cc:2393] T 36ff33be666a4c82954731691dbcf327 P 197b880fa39d432a9b472845876bfaf0 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1c50de89a5a845668696dcb28c3badd6 in current term 2: Already voted for candidate 197b880fa39d432a9b472845876bfaf0 in this term.
I20260129 02:59:33.531198 22092 leader_election.cc:290] T 36ff33be666a4c82954731691dbcf327 P 197b880fa39d432a9b472845876bfaf0 [CANDIDATE]: Term 2 election: Requested vote from peers 974b802141bd4f13beffef2aa03f88f9 (127.18.211.2:34517), 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3:37735)
I20260129 02:59:33.532161 20441 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55676 (request call id 1098) took 1289 ms. Trace:
I20260129 02:59:33.486132 20438 raft_consensus.cc:2468] T e33a417cf9c54cf5af73a4ee3aac4d3a P 197b880fa39d432a9b472845876bfaf0 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 974b802141bd4f13beffef2aa03f88f9 in term 3.
I20260129 02:59:33.485064 21102 leader_election.cc:304] T febffe4a8b394368b69b1d164308fb9a P 7a9daf5d42a84615a6383cfec9a10608 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 7a9daf5d42a84615a6383cfec9a10608, 974b802141bd4f13beffef2aa03f88f9; no voters: 197b880fa39d432a9b472845876bfaf0
W20260129 02:59:33.502709 20553 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.211.1:42027, user_credentials={real_user=slave}} blocked reactor thread for 46905.7us
I20260129 02:59:33.490816 21133 raft_consensus.cc:2376] T 42d27a28d89b409e8807ec0efd97af69 P 7a9daf5d42a84615a6383cfec9a10608 [term 4 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 197b880fa39d432a9b472845876bfaf0 in term 4. Re-sending same reply.
I20260129 02:59:33.501113 20511 raft_consensus.cc:2468] T 4b1876f9a70b49edb3aa74105e66d3b3 P 974b802141bd4f13beffef2aa03f88f9 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 197b880fa39d432a9b472845876bfaf0 in term 3.
I20260129 02:59:33.501935 21136 raft_consensus.cc:2468] T 2ac659d732314e1288849f08fe5e074e P 7a9daf5d42a84615a6383cfec9a10608 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 197b880fa39d432a9b472845876bfaf0 in term 1.
I20260129 02:59:33.532662 20441 rpcz_store.cc:276] 0129 02:59:32.243216 (+ 0us) service_pool.cc:168] Inserting onto call queue
0129 02:59:33.326475 (+1083259us) service_pool.cc:225] Handling call
0129 02:59:33.531843 (+205368us) spinlock_profiling.cc:243] Waited 142 ms on lock 0x7b54001423d4. stack: 00007f8fe044e373 00007f8fe044e270 00007f8fdfda0574 00007f8fe9a5c6b4 00007f8fe9a5c649 00007f8fe34975a1 00007f8fe3531f7a 00007f8fe84e3647 00007f8fe1a73834 00007f8fe1a737c2 00007f8fe1a736f1 00007f8fe1a7366c 00007f8fe1a72922 00007f8fe186d34c 00007f8fe186c786 00007f8fe186c12f
0129 02:59:33.532059 (+ 216us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":327461760}
I20260129 02:59:33.514353 20439 raft_consensus.cc:2468] T 4e3805ffa70242e8ade66da260dddf66 P 197b880fa39d432a9b472845876bfaf0 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1c50de89a5a845668696dcb28c3badd6 in term 3.
I20260129 02:59:33.501650 22112 raft_consensus.cc:740] T 505580c93c694bbb8ea96fdf9b17ab46 P 974b802141bd4f13beffef2aa03f88f9 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 974b802141bd4f13beffef2aa03f88f9, State: Running, Role: LEADER
W20260129 02:59:33.516270 21099 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260129 02:59:33.516481 20556 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.211.2:34517, user_credentials={real_user=slave}} blocked reactor thread for 71797.8us
I20260129 02:59:33.515738 20440 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00e79d8f25a34676b1d866eb2068a2b2" candidate_uuid: "974b802141bd4f13beffef2aa03f88f9" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "197b880fa39d432a9b472845876bfaf0" is_pre_election: true
I20260129 02:59:33.513221 20587 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fd8311b779af4e1fa1402b2a02a602f2" candidate_uuid: "7a9daf5d42a84615a6383cfec9a10608" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "1c50de89a5a845668696dcb28c3badd6" is_pre_election: true
I20260129 02:59:33.525137 21134 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a644c02b16af47679824ccd060ac10fd" candidate_uuid: "974b802141bd4f13beffef2aa03f88f9" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "7a9daf5d42a84615a6383cfec9a10608"
I20260129 02:59:33.525209 20432 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4b1876f9a70b49edb3aa74105e66d3b3" candidate_uuid: "974b802141bd4f13beffef2aa03f88f9" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "197b880fa39d432a9b472845876bfaf0" is_pre_election: true
I20260129 02:59:33.526439 21130 raft_consensus.cc:2393] T 96609ac68bdf4993a12e1c23355ca013 P 7a9daf5d42a84615a6383cfec9a10608 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 197b880fa39d432a9b472845876bfaf0 in current term 4: Already voted for candidate 7a9daf5d42a84615a6383cfec9a10608 in this term.
I20260129 02:59:33.527155 22082 raft_consensus.cc:493] T 0f20af7fbe5d4db89a6f17948d98912c P 197b880fa39d432a9b472845876bfaf0 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 974b802141bd4f13beffef2aa03f88f9)
I20260129 02:59:33.529414 20433 raft_consensus.cc:2376] T 368953c9102643b2a21f99af36b51905 P 197b880fa39d432a9b472845876bfaf0 [term 4 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 1c50de89a5a845668696dcb28c3badd6 in term 4. Re-sending same reply.
I20260129 02:59:33.539676 20438 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "96609ac68bdf4993a12e1c23355ca013" candidate_uuid: "974b802141bd4f13beffef2aa03f88f9" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "197b880fa39d432a9b472845876bfaf0"
I20260129 02:59:33.505389 22243 raft_consensus.cc:3060] T 0340229d521b432580ae926f6062e3f7 P 1c50de89a5a845668696dcb28c3badd6 [term 3 FOLLOWER]: Advancing to term 4
I20260129 02:59:33.542040 20407 leader_election.cc:304] T 7d9494f5584d4d5db8b9adf1c8fc888b P 197b880fa39d432a9b472845876bfaf0 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 197b880fa39d432a9b472845876bfaf0; no voters: 1c50de89a5a845668696dcb28c3badd6, 974b802141bd4f13beffef2aa03f88f9
I20260129 02:59:33.543517 20432 raft_consensus.cc:2468] T 4b1876f9a70b49edb3aa74105e66d3b3 P 197b880fa39d432a9b472845876bfaf0 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 974b802141bd4f13beffef2aa03f88f9 in term 3.
I20260129 02:59:33.545114 20582 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b189f08e16a74a11980cc55d99359a0a" candidate_uuid: "974b802141bd4f13beffef2aa03f88f9" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "1c50de89a5a845668696dcb28c3badd6" is_pre_election: true
I20260129 02:59:33.519337 22120 raft_consensus.cc:493] T 8055232cf0004d40a85f3bf983fe00a9 P 197b880fa39d432a9b472845876bfaf0 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 974b802141bd4f13beffef2aa03f88f9)
I20260129 02:59:33.492225 22247 consensus_queue.cc:1059] T 75b0b6e82ddc4f92b1fe1d94f9a0b73b P 1c50de89a5a845668696dcb28c3badd6 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.048s
I20260129 02:59:33.666440 21134 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c934d31efd3f4e5195605ddc354d14e1" candidate_uuid: "974b802141bd4f13beffef2aa03f88f9" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "7a9daf5d42a84615a6383cfec9a10608" is_pre_election: true
I20260129 02:59:33.569922 20507 raft_consensus.cc:1240] T c934d31efd3f4e5195605ddc354d14e1 P 974b802141bd4f13beffef2aa03f88f9 [term 3 FOLLOWER]: Rejecting Update request from peer 7a9daf5d42a84615a6383cfec9a10608 for earlier term 2. Current term is 3. Ops: []
W20260129 02:59:33.556783 21102 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.211.2:34517, user_credentials={real_user=slave}} blocked reactor thread for 76623.2us
I20260129 02:59:33.547811 20439 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00e79d8f25a34676b1d866eb2068a2b2" candidate_uuid: "974b802141bd4f13beffef2aa03f88f9" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "197b880fa39d432a9b472845876bfaf0"
W20260129 02:59:33.614187 21099 leader_election.cc:336] T 2ac659d732314e1288849f08fe5e074e P 7a9daf5d42a84615a6383cfec9a10608 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 197b880fa39d432a9b472845876bfaf0 (127.18.211.1:42027): Timed out: RequestConsensusVote RPC to 127.18.211.1:42027 timed out after 1.962s (SENT)
I20260129 02:59:33.570312 20511 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2ac659d732314e1288849f08fe5e074e" candidate_uuid: "7a9daf5d42a84615a6383cfec9a10608" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "974b802141bd4f13beffef2aa03f88f9" is_pre_election: true
I20260129 02:59:33.667876 20441 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c934d31efd3f4e5195605ddc354d14e1" candidate_uuid: "974b802141bd4f13beffef2aa03f88f9" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "197b880fa39d432a9b472845876bfaf0" is_pre_election: true
I20260129 02:59:33.613551 20583 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a644c02b16af47679824ccd060ac10fd" candidate_uuid: "974b802141bd4f13beffef2aa03f88f9" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "1c50de89a5a845668696dcb28c3badd6"
I20260129 02:59:33.671950 21134 raft_consensus.cc:2468] T c934d31efd3f4e5195605ddc354d14e1 P 7a9daf5d42a84615a6383cfec9a10608 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 974b802141bd4f13beffef2aa03f88f9 in term 3.
W20260129 02:59:33.498601 21100 connection.cc:232] RPC call timeout handler was delayed by -1.5048724430000675s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.22330000000000003s for a response
I20260129 02:59:33.568965 20584 raft_consensus.cc:1240] T 724a236524cb4e7ba3428880681d6cc2 P 1c50de89a5a845668696dcb28c3badd6 [term 2 FOLLOWER]: Rejecting Update request from peer 974b802141bd4f13beffef2aa03f88f9 for earlier term 1. Current term is 2. Ops: []
I20260129 02:59:33.610473 20510 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "75f45538dc704d9a8ec4ff8ad9f3ea5b" candidate_uuid: "7a9daf5d42a84615a6383cfec9a10608" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "974b802141bd4f13beffef2aa03f88f9"
I20260129 02:59:33.620332 20433 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a0fcc58d6fb6473cafbd9d34d5195269" candidate_uuid: "974b802141bd4f13beffef2aa03f88f9" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "197b880fa39d432a9b472845876bfaf0" is_pre_election: true
I20260129 02:59:33.617667 20514 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "75b0b6e82ddc4f92b1fe1d94f9a0b73b" candidate_uuid: "7a9daf5d42a84615a6383cfec9a10608" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "974b802141bd4f13beffef2aa03f88f9"
I20260129 02:59:33.680337 20439 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4f8f3a38f6e24ba4857875181f447442" candidate_uuid: "1c50de89a5a845668696dcb28c3badd6" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "197b880fa39d432a9b472845876bfaf0"
I20260129 02:59:33.680717 20514 raft_consensus.cc:3060] T 75b0b6e82ddc4f92b1fe1d94f9a0b73b P 974b802141bd4f13beffef2aa03f88f9 [term 1 FOLLOWER]: Advancing to term 2
I20260129 02:59:33.550194 20437 raft_consensus.cc:1240] T 4e3805ffa70242e8ade66da260dddf66 P 197b880fa39d432a9b472845876bfaf0 [term 3 FOLLOWER]: Rejecting Update request from peer 7a9daf5d42a84615a6383cfec9a10608 for earlier term 2. Current term is 3. Ops: []
I20260129 02:59:33.619542 20515 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a644c02b16af47679824ccd060ac10fd" candidate_uuid: "7a9daf5d42a84615a6383cfec9a10608" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "974b802141bd4f13beffef2aa03f88f9" is_pre_election: true
I20260129 02:59:33.499958 20436 rpcz_store.cc:276] 0129 02:59:32.242502 (+ 0us) service_pool.cc:168] Inserting onto call queue
0129 02:59:33.367942 (+1125440us) service_pool.cc:225] Handling call
0129 02:59:33.476320 (+108378us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260129 02:59:33.677614 20513 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4f5a77e94fef4b16a2ba7b067c8b144a" candidate_uuid: "7a9daf5d42a84615a6383cfec9a10608" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "974b802141bd4f13beffef2aa03f88f9"
I20260129 02:59:33.541340 22103 raft_consensus.cc:3060] T 75f45538dc704d9a8ec4ff8ad9f3ea5b P 1c50de89a5a845668696dcb28c3badd6 [term 1 FOLLOWER]: Advancing to term 2
I20260129 02:59:33.679980 20581 raft_consensus.cc:1240] T 36ff33be666a4c82954731691dbcf327 P 1c50de89a5a845668696dcb28c3badd6 [term 2 FOLLOWER]: Rejecting Update request from peer 974b802141bd4f13beffef2aa03f88f9 for earlier term 1. Current term is 2. Ops: []
I20260129 02:59:33.620813 20479 leader_election.cc:304] T e7d83d4ccf684e76acb29a0dc844e1b8 P 974b802141bd4f13beffef2aa03f88f9 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 197b880fa39d432a9b472845876bfaf0, 974b802141bd4f13beffef2aa03f88f9; no voters:
I20260129 02:59:33.673756 21131 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "75b0b6e82ddc4f92b1fe1d94f9a0b73b" candidate_uuid: "974b802141bd4f13beffef2aa03f88f9" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "7a9daf5d42a84615a6383cfec9a10608" is_pre_election: true
I20260129 02:59:33.681945 20585 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a0fcc58d6fb6473cafbd9d34d5195269" candidate_uuid: "974b802141bd4f13beffef2aa03f88f9" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "1c50de89a5a845668696dcb28c3badd6" is_pre_election: true
I20260129 02:59:33.546914 22212 raft_consensus.cc:2804] T febffe4a8b394368b69b1d164308fb9a P 7a9daf5d42a84615a6383cfec9a10608 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260129 02:59:33.720837 21135 raft_consensus.cc:1240] T 1299723a4476420ab02fd8bfd7aa0645 P 7a9daf5d42a84615a6383cfec9a10608 [term 5 FOLLOWER]: Rejecting Update request from peer 197b880fa39d432a9b472845876bfaf0 for earlier term 4. Current term is 5. Ops: []
I20260129 02:59:33.715615 20440 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:48356 (request call id 206) took 4008 ms. Trace:
I20260129 02:59:33.549270 20589 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "78aa39096868415d9afc2ffcb248e198" candidate_uuid: "7a9daf5d42a84615a6383cfec9a10608" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "1c50de89a5a845668696dcb28c3badd6" is_pre_election: true
W20260129 02:59:33.701884 20407 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.211.2:34517, user_credentials={real_user=slave}} blocked reactor thread for 159920us
I20260129 02:59:33.710868 20582 raft_consensus.cc:2468] T b189f08e16a74a11980cc55d99359a0a P 1c50de89a5a845668696dcb28c3badd6 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 974b802141bd4f13beffef2aa03f88f9 in term 4.
I20260129 02:59:33.714731 20437 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:48356 (request call id 205) took 3998 ms. Trace:
I20260129 02:59:33.753480 20553 leader_election.cc:304] T b189f08e16a74a11980cc55d99359a0a P 1c50de89a5a845668696dcb28c3badd6 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1c50de89a5a845668696dcb28c3badd6; no voters: 197b880fa39d432a9b472845876bfaf0, 974b802141bd4f13beffef2aa03f88f9
I20260129 02:59:33.753901 20437 rpcz_store.cc:276] 0129 02:59:29.706793 (+ 0us) service_pool.cc:168] Inserting onto call queue
0129 02:59:33.549610 (+3842817us) service_pool.cc:225] Handling call
0129 02:59:33.550012 (+ 402us) raft_consensus.cc:1471] Updating replica for 0 ops
0129 02:59:33.704475 (+154463us) raft_consensus.cc:1697] Filling consensus response to leader.
0129 02:59:33.704966 (+ 491us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260129 02:59:33.614447 22157 raft_consensus.cc:515] T 3ccac82a01a2445fa5b0ce6c0fcf2295 P 974b802141bd4f13beffef2aa03f88f9 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } }
I20260129 02:59:33.755038 20437 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7a9a69cd82724b1faed1d10aa091fc4a" candidate_uuid: "1c50de89a5a845668696dcb28c3badd6" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "197b880fa39d432a9b472845876bfaf0"
I20260129 02:59:33.756114 20436 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fd8311b779af4e1fa1402b2a02a602f2" candidate_uuid: "1c50de89a5a845668696dcb28c3badd6" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "197b880fa39d432a9b472845876bfaf0" is_pre_election: true
I20260129 02:59:33.757305 22157 leader_election.cc:290] T 3ccac82a01a2445fa5b0ce6c0fcf2295 P 974b802141bd4f13beffef2aa03f88f9 [CANDIDATE]: Term 2 election: Requested vote from peers 197b880fa39d432a9b472845876bfaf0 (127.18.211.1:42027), 7a9daf5d42a84615a6383cfec9a10608 (127.18.211.4:40439)
I20260129 02:59:33.758524 20436 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "36ff33be666a4c82954731691dbcf327" candidate_uuid: "1c50de89a5a845668696dcb28c3badd6" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "197b880fa39d432a9b472845876bfaf0"
I20260129 02:59:33.759343 20436 raft_consensus.cc:2393] T 36ff33be666a4c82954731691dbcf327 P 197b880fa39d432a9b472845876bfaf0 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 1c50de89a5a845668696dcb28c3badd6 in current term 2: Already voted for candidate 197b880fa39d432a9b472845876bfaf0 in this term.
I20260129 02:59:33.714838 21133 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2e58e9a5a8aa4cecafb464addb4f35ed" candidate_uuid: "197b880fa39d432a9b472845876bfaf0" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "7a9daf5d42a84615a6383cfec9a10608"
I20260129 02:59:33.760174 20436 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "17d6431cc77a491e8464d68db4081a51" candidate_uuid: "1c50de89a5a845668696dcb28c3badd6" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "197b880fa39d432a9b472845876bfaf0" is_pre_election: true
I20260129 02:59:33.619076 22198 raft_consensus.cc:2749] T 7d9494f5584d4d5db8b9adf1c8fc888b P 197b880fa39d432a9b472845876bfaf0 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260129 02:59:33.760916 20436 raft_consensus.cc:2468] T 17d6431cc77a491e8464d68db4081a51 P 197b880fa39d432a9b472845876bfaf0 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1c50de89a5a845668696dcb28c3badd6 in term 1.
I20260129 02:59:33.678503 20587 raft_consensus.cc:2468] T fd8311b779af4e1fa1402b2a02a602f2 P 1c50de89a5a845668696dcb28c3badd6 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7a9daf5d42a84615a6383cfec9a10608 in term 2.
I20260129 02:59:33.716645 20405 leader_election.cc:304] T 42d27a28d89b409e8807ec0efd97af69 P 197b880fa39d432a9b472845876bfaf0 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 197b880fa39d432a9b472845876bfaf0, 7a9daf5d42a84615a6383cfec9a10608; no voters: 1c50de89a5a845668696dcb28c3badd6
I20260129 02:59:33.704530 22213 raft_consensus.cc:493] T a8479d1fe0bd41e5946186e3a6ca8c7a P 1c50de89a5a845668696dcb28c3badd6 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 974b802141bd4f13beffef2aa03f88f9)
I20260129 02:59:33.756167 20437 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0f20af7fbe5d4db89a6f17948d98912c" candidate_uuid: "1c50de89a5a845668696dcb28c3badd6" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "197b880fa39d432a9b472845876bfaf0"
I20260129 02:59:33.762087 22213 raft_consensus.cc:515] T a8479d1fe0bd41e5946186e3a6ca8c7a P 1c50de89a5a845668696dcb28c3badd6 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:59:33.674646 22120 raft_consensus.cc:3060] T 8055232cf0004d40a85f3bf983fe00a9 P 197b880fa39d432a9b472845876bfaf0 [term 1 FOLLOWER]: Advancing to term 2
W20260129 02:59:33.679160 21099 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.211.1:42027, user_credentials={real_user=slave}} blocked reactor thread for 64854.3us
I20260129 02:59:33.721560 20440 rpcz_store.cc:276] 0129 02:59:29.707032 (+ 0us) service_pool.cc:168] Inserting onto call queue
0129 02:59:33.544724 (+3837692us) service_pool.cc:225] Handling call
0129 02:59:33.678262 (+133538us) raft_consensus.cc:1471] Updating replica for 0 ops
0129 02:59:33.715258 (+ 36996us) raft_consensus.cc:1534] Early marking committed up to index 0
0129 02:59:33.715285 (+ 27us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0129 02:59:33.715335 (+ 50us) raft_consensus.cc:1645] Marking committed up to 6
0129 02:59:33.715380 (+ 45us) raft_consensus.cc:1697] Filling consensus response to leader.
0129 02:59:33.715451 (+ 71us) raft_consensus.cc:1691] UpdateReplicas() finished
0129 02:59:33.715555 (+ 104us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260129 02:59:33.746428 20441 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "724a236524cb4e7ba3428880681d6cc2" candidate_uuid: "1c50de89a5a845668696dcb28c3badd6" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "197b880fa39d432a9b472845876bfaf0"
I20260129 02:59:33.729005 20515 raft_consensus.cc:2393] T a644c02b16af47679824ccd060ac10fd P 974b802141bd4f13beffef2aa03f88f9 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 7a9daf5d42a84615a6383cfec9a10608 in current term 2: Already voted for candidate 974b802141bd4f13beffef2aa03f88f9 in this term.
I20260129 02:59:33.610039 22222 leader_election.cc:290] T 96609ac68bdf4993a12e1c23355ca013 P 7a9daf5d42a84615a6383cfec9a10608 [CANDIDATE]: Term 4 election: Requested vote from peers 197b880fa39d432a9b472845876bfaf0 (127.18.211.1:42027), 974b802141bd4f13beffef2aa03f88f9 (127.18.211.2:34517), 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3:37735)
I20260129 02:59:33.752028 20588 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "75b0b6e82ddc4f92b1fe1d94f9a0b73b" candidate_uuid: "974b802141bd4f13beffef2aa03f88f9" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "1c50de89a5a845668696dcb28c3badd6" is_pre_election: true
I20260129 02:59:33.763820 20438 raft_consensus.cc:2468] T 96609ac68bdf4993a12e1c23355ca013 P 197b880fa39d432a9b472845876bfaf0 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 974b802141bd4f13beffef2aa03f88f9 in term 3.
I20260129 02:59:33.769603 20433 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a4c1226b987b468fa84dfc6ba0e2dbe1" candidate_uuid: "1c50de89a5a845668696dcb28c3badd6" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "197b880fa39d432a9b472845876bfaf0"
I20260129 02:59:33.611404 21128 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00e79d8f25a34676b1d866eb2068a2b2" candidate_uuid: "974b802141bd4f13beffef2aa03f88f9" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "7a9daf5d42a84615a6383cfec9a10608"
I20260129 02:59:33.617348 20434 raft_consensus.cc:2468] T 1ce4ddf5b10e4ae8b193b13b2e5936ee P 197b880fa39d432a9b472845876bfaf0 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1c50de89a5a845668696dcb28c3badd6 in term 4.
I20260129 02:59:33.609141 22201 raft_consensus.cc:493] T 3e2d15f8640b407195210c476e09df91 P 974b802141bd4f13beffef2aa03f88f9 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 1c50de89a5a845668696dcb28c3badd6)
W20260129 02:59:33.777513 20434 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55676 (request call id 1096) took 1993 ms (client timeout 1862 ms). Trace:
W20260129 02:59:33.710048 20369 debug-util.cc:398] Leaking SignalData structure 0x7b08002c3d00 after lost signal to thread 19279
I20260129 02:59:33.824230 20440 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "18f30ebaa4174fd28c958e72d0cc7e24" candidate_uuid: "1c50de89a5a845668696dcb28c3badd6" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "197b880fa39d432a9b472845876bfaf0" is_pre_election: true
I20260129 02:59:33.958747 20435 raft_consensus.cc:2468] T a5a05fbcbbcf4916bbc01fd1a5b5e90d P 197b880fa39d432a9b472845876bfaf0 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 974b802141bd4f13beffef2aa03f88f9 in term 5.
I20260129 02:59:33.925148 22201 raft_consensus.cc:515] T 3e2d15f8640b407195210c476e09df91 P 974b802141bd4f13beffef2aa03f88f9 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } }
I20260129 02:59:33.720336 21100 leader_election.cc:304] T cb0b3f259ebc4310927338a8740e87b8 P 7a9daf5d42a84615a6383cfec9a10608 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7a9daf5d42a84615a6383cfec9a10608; no voters: 1c50de89a5a845668696dcb28c3badd6, 974b802141bd4f13beffef2aa03f88f9
I20260129 02:59:33.912007 20441 raft_consensus.cc:3060] T 724a236524cb4e7ba3428880681d6cc2 P 197b880fa39d432a9b472845876bfaf0 [term 1 FOLLOWER]: Advancing to term 2
I20260129 02:59:33.720912 22212 raft_consensus.cc:493] T febffe4a8b394368b69b1d164308fb9a P 7a9daf5d42a84615a6383cfec9a10608 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 197b880fa39d432a9b472845876bfaf0)
I20260129 02:59:33.805456 20438 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c2633a6fc34c44579de348d208b0c01e" candidate_uuid: "1c50de89a5a845668696dcb28c3badd6" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "197b880fa39d432a9b472845876bfaf0"
I20260129 02:59:33.821002 20436 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1299723a4476420ab02fd8bfd7aa0645" candidate_uuid: "1c50de89a5a845668696dcb28c3badd6" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "197b880fa39d432a9b472845876bfaf0"
I20260129 02:59:33.882499 20439 raft_consensus.cc:3060] T 4f8f3a38f6e24ba4857875181f447442 P 197b880fa39d432a9b472845876bfaf0 [term 3 FOLLOWER]: Advancing to term 4
W20260129 02:59:33.819156 22287 log.cc:927] Time spent T b189f08e16a74a11980cc55d99359a0a P 974b802141bd4f13beffef2aa03f88f9: Append to log took a long time: real 0.074s user 0.000s sys 0.000s
I20260129 02:59:33.549479 22082 raft_consensus.cc:515] T 0f20af7fbe5d4db89a6f17948d98912c P 197b880fa39d432a9b472845876bfaf0 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:59:33.618718 22112 consensus_queue.cc:260] T 505580c93c694bbb8ea96fdf9b17ab46 P 974b802141bd4f13beffef2aa03f88f9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } }
I20260129 02:59:33.771690 20590 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "75f45538dc704d9a8ec4ff8ad9f3ea5b" candidate_uuid: "7a9daf5d42a84615a6383cfec9a10608" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "1c50de89a5a845668696dcb28c3badd6"
I20260129 02:59:33.825014 20433 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7a9a69cd82724b1faed1d10aa091fc4a" candidate_uuid: "1c50de89a5a845668696dcb28c3badd6" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "197b880fa39d432a9b472845876bfaf0" is_pre_election: true
W20260129 02:59:33.818543 21101 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260129 02:59:33.959616 20369 debug-util.cc:398] Leaking SignalData structure 0x7b08003219a0 after lost signal to thread 20372
W20260129 02:59:33.960721 20480 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.211.3:37735, user_credentials={real_user=slave}} blocked reactor thread for 254597us
W20260129 02:59:33.977792 20369 debug-util.cc:398] Leaking SignalData structure 0x7b080031ad60 after lost signal to thread 20387
W20260129 02:59:33.965739 21102 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.211.2:34517, user_credentials={real_user=slave}} blocked reactor thread for 195474us
W20260129 02:59:33.778069 20434 rpcz_store.cc:269] 0129 02:59:31.783742 (+ 0us) service_pool.cc:168] Inserting onto call queue
0129 02:59:33.280095 (+1496353us) service_pool.cc:225] Handling call
0129 02:59:33.777416 (+497321us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260129 02:59:33.759681 22069 raft_consensus.cc:515] T cb4727386a3e48aca5d71cc6c611c197 P 197b880fa39d432a9b472845876bfaf0 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
W20260129 02:59:33.966648 20479 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.211.1:42027, user_credentials={real_user=slave}} blocked reactor thread for 345979us
I20260129 02:59:33.967914 20514 raft_consensus.cc:2468] T 75b0b6e82ddc4f92b1fe1d94f9a0b73b P 974b802141bd4f13beffef2aa03f88f9 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7a9daf5d42a84615a6383cfec9a10608 in term 2.
W20260129 02:59:33.968086 20556 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.211.2:34517, user_credentials={real_user=slave}} blocked reactor thread for 292094us
I20260129 02:59:33.956041 20585 raft_consensus.cc:2468] T a0fcc58d6fb6473cafbd9d34d5195269 P 1c50de89a5a845668696dcb28c3badd6 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 974b802141bd4f13beffef2aa03f88f9 in term 2.
W20260129 02:59:33.972223 20405 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.211.4:40439, user_credentials={real_user=slave}} blocked reactor thread for 266412us
I20260129 02:59:33.973785 20407 leader_election.cc:304] T 4e420f90884f4796997b5ea41e49e411 P 197b880fa39d432a9b472845876bfaf0 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 197b880fa39d432a9b472845876bfaf0, 974b802141bd4f13beffef2aa03f88f9; no voters: 7a9daf5d42a84615a6383cfec9a10608
I20260129 02:59:33.960785 20588 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a644c02b16af47679824ccd060ac10fd" candidate_uuid: "7a9daf5d42a84615a6383cfec9a10608" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "1c50de89a5a845668696dcb28c3badd6" is_pre_election: true
W20260129 02:59:33.978909 20369 debug-util.cc:398] Leaking SignalData structure 0x7b0800034ea0 after lost signal to thread 20465
I20260129 02:59:33.984180 20405 leader_election.cc:304] T 2ac659d732314e1288849f08fe5e074e P 197b880fa39d432a9b472845876bfaf0 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 197b880fa39d432a9b472845876bfaf0, 7a9daf5d42a84615a6383cfec9a10608; no voters:
W20260129 02:59:33.984891 20553 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.211.1:42027, user_credentials={real_user=slave}} blocked reactor thread for 237950us
I20260129 02:59:33.984833 20479 leader_election.cc:304] T a609ac02fae84725ab743f95f51d33ce P 974b802141bd4f13beffef2aa03f88f9 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 197b880fa39d432a9b472845876bfaf0, 974b802141bd4f13beffef2aa03f88f9; no voters: 7a9daf5d42a84615a6383cfec9a10608
I20260129 02:59:33.985059 22079 raft_consensus.cc:2749] T b189f08e16a74a11980cc55d99359a0a P 1c50de89a5a845668696dcb28c3badd6 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260129 02:59:33.985976 20480 leader_election.cc:304] T 45cc5f3d2688486d9b6dbe7e666dedde P 974b802141bd4f13beffef2aa03f88f9 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 974b802141bd4f13beffef2aa03f88f9; no voters: 1c50de89a5a845668696dcb28c3badd6, 7a9daf5d42a84615a6383cfec9a10608
I20260129 02:59:33.986092 22069 leader_election.cc:290] T cb4727386a3e48aca5d71cc6c611c197 P 197b880fa39d432a9b472845876bfaf0 [CANDIDATE]: Term 2 election: Requested vote from peers 974b802141bd4f13beffef2aa03f88f9 (127.18.211.2:34517), 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3:37735)
I20260129 02:59:33.971199 22160 raft_consensus.cc:2804] T e7d83d4ccf684e76acb29a0dc844e1b8 P 974b802141bd4f13beffef2aa03f88f9 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260129 02:59:33.987035 20437 raft_consensus.cc:3060] T 0f20af7fbe5d4db89a6f17948d98912c P 197b880fa39d432a9b472845876bfaf0 [term 3 FOLLOWER]: Advancing to term 4
W20260129 02:59:33.987303 20369 debug-util.cc:398] Leaking SignalData structure 0x7b0800034680 after lost signal to thread 20540
W20260129 02:59:34.002786 20369 debug-util.cc:398] Leaking SignalData structure 0x7b0800034660 after lost signal to thread 20541
I20260129 02:59:33.954553 22092 raft_consensus.cc:2804] T 42d27a28d89b409e8807ec0efd97af69 P 197b880fa39d432a9b472845876bfaf0 [term 4 FOLLOWER]: Leader election won for term 4
I20260129 02:59:33.971199 22164 consensus_queue.cc:1059] T 76f713c3331540d2a5b76164103e9eb1 P 1c50de89a5a845668696dcb28c3badd6 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 }, Status: INVALID_TERM, Last received: 2.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.025s
I20260129 02:59:33.996624 20588 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1299723a4476420ab02fd8bfd7aa0645" candidate_uuid: "7a9daf5d42a84615a6383cfec9a10608" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "1c50de89a5a845668696dcb28c3badd6" is_pre_election: true
I20260129 02:59:34.008262 22160 raft_consensus.cc:493] T e7d83d4ccf684e76acb29a0dc844e1b8 P 974b802141bd4f13beffef2aa03f88f9 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 1c50de89a5a845668696dcb28c3badd6)
I20260129 02:59:34.008726 20588 raft_consensus.cc:2393] T 1299723a4476420ab02fd8bfd7aa0645 P 1c50de89a5a845668696dcb28c3badd6 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 7a9daf5d42a84615a6383cfec9a10608 in current term 5: Already voted for candidate 1c50de89a5a845668696dcb28c3badd6 in this term.
I20260129 02:59:33.996837 22061 raft_consensus.cc:3055] T 75b0b6e82ddc4f92b1fe1d94f9a0b73b P 1c50de89a5a845668696dcb28c3badd6 [term 1 LEADER]: Stepping down as leader of term 1
I20260129 02:59:33.997462 20553 leader_election.cc:304] T 96609ac68bdf4993a12e1c23355ca013 P 1c50de89a5a845668696dcb28c3badd6 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1c50de89a5a845668696dcb28c3badd6; no voters: 197b880fa39d432a9b472845876bfaf0, 974b802141bd4f13beffef2aa03f88f9
I20260129 02:59:33.997296 22197 consensus_queue.cc:1059] T 505580c93c694bbb8ea96fdf9b17ab46 P 974b802141bd4f13beffef2aa03f88f9 [NON_LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260129 02:59:34.013242 22061 raft_consensus.cc:740] T 75b0b6e82ddc4f92b1fe1d94f9a0b73b P 1c50de89a5a845668696dcb28c3badd6 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 1c50de89a5a845668696dcb28c3badd6, State: Running, Role: LEADER
I20260129 02:59:34.013228 22247 consensus_queue.cc:1059] T 76f713c3331540d2a5b76164103e9eb1 P 1c50de89a5a845668696dcb28c3badd6 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 }, Status: INVALID_TERM, Last received: 2.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.008s
I20260129 02:59:34.017491 20407 leader_election.cc:400] T 75be4bd6bd45474795fd472eff2953ac P 197b880fa39d432a9b472845876bfaf0 [CANDIDATE]: Term 6 election: Vote denied by peer 974b802141bd4f13beffef2aa03f88f9 (127.18.211.2:34517) with higher term. Message: Invalid argument: T 75be4bd6bd45474795fd472eff2953ac P 974b802141bd4f13beffef2aa03f88f9 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 197b880fa39d432a9b472845876bfaf0 for earlier term 6. Current term is 7.
I20260129 02:59:34.017829 20407 leader_election.cc:403] T 75be4bd6bd45474795fd472eff2953ac P 197b880fa39d432a9b472845876bfaf0 [CANDIDATE]: Term 6 election: Cancelling election due to peer responding with higher term
I20260129 02:59:34.017868 22247 consensus_queue.cc:1243] T 76f713c3331540d2a5b76164103e9eb1 P 1c50de89a5a845668696dcb28c3badd6 [LEADER]: Peer 197b880fa39d432a9b472845876bfaf0 log is divergent from this leader: its last log entry 3.3 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 2
I20260129 02:59:34.018579 22237 raft_consensus.cc:2804] T 2ac659d732314e1288849f08fe5e074e P 197b880fa39d432a9b472845876bfaf0 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260129 02:59:34.019006 22237 raft_consensus.cc:493] T 2ac659d732314e1288849f08fe5e074e P 197b880fa39d432a9b472845876bfaf0 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 974b802141bd4f13beffef2aa03f88f9)
I20260129 02:59:34.019207 22091 raft_consensus.cc:2804] T 4e420f90884f4796997b5ea41e49e411 P 197b880fa39d432a9b472845876bfaf0 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260129 02:59:34.019420 22237 raft_consensus.cc:3060] T 2ac659d732314e1288849f08fe5e074e P 197b880fa39d432a9b472845876bfaf0 [term 1 FOLLOWER]: Advancing to term 2
I20260129 02:59:34.019636 22091 raft_consensus.cc:493] T 4e420f90884f4796997b5ea41e49e411 P 197b880fa39d432a9b472845876bfaf0 [term 3 FOLLOWER]: Starting leader election (detected failure of leader 7a9daf5d42a84615a6383cfec9a10608)
I20260129 02:59:34.020201 22091 raft_consensus.cc:3060] T 4e420f90884f4796997b5ea41e49e411 P 197b880fa39d432a9b472845876bfaf0 [term 3 FOLLOWER]: Advancing to term 4
I20260129 02:59:33.999904 22157 raft_consensus.cc:2749] T 45cc5f3d2688486d9b6dbe7e666dedde P 974b802141bd4f13beffef2aa03f88f9 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260129 02:59:33.999926 22036 raft_consensus.cc:2804] T a609ac02fae84725ab743f95f51d33ce P 974b802141bd4f13beffef2aa03f88f9 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260129 02:59:34.022647 22036 raft_consensus.cc:493] T a609ac02fae84725ab743f95f51d33ce P 974b802141bd4f13beffef2aa03f88f9 [term 4 FOLLOWER]: Starting leader election (detected failure of leader 7a9daf5d42a84615a6383cfec9a10608)
I20260129 02:59:34.023116 22036 raft_consensus.cc:3060] T a609ac02fae84725ab743f95f51d33ce P 974b802141bd4f13beffef2aa03f88f9 [term 4 FOLLOWER]: Advancing to term 5
I20260129 02:59:34.025211 22233 raft_consensus.cc:2749] T 75be4bd6bd45474795fd472eff2953ac P 197b880fa39d432a9b472845876bfaf0 [term 7 FOLLOWER]: Leader election lost for term 6. Reason: Vote denied by peer 974b802141bd4f13beffef2aa03f88f9 (127.18.211.2:34517) with higher term. Message: Invalid argument: T 75be4bd6bd45474795fd472eff2953ac P 974b802141bd4f13beffef2aa03f88f9 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 197b880fa39d432a9b472845876bfaf0 for earlier term 6. Current term is 7.
I20260129 02:59:34.008826 22160 raft_consensus.cc:3060] T e7d83d4ccf684e76acb29a0dc844e1b8 P 974b802141bd4f13beffef2aa03f88f9 [term 2 FOLLOWER]: Advancing to term 3
I20260129 02:59:34.027035 20515 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1ce4ddf5b10e4ae8b193b13b2e5936ee" candidate_uuid: "1c50de89a5a845668696dcb28c3badd6" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "974b802141bd4f13beffef2aa03f88f9"
I20260129 02:59:33.999094 20480 leader_election.cc:304] T cedce331e1584ac8b636c9b8a72b2ba0 P 974b802141bd4f13beffef2aa03f88f9 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 974b802141bd4f13beffef2aa03f88f9; no voters: 197b880fa39d432a9b472845876bfaf0, 1c50de89a5a845668696dcb28c3badd6
I20260129 02:59:34.027769 20515 raft_consensus.cc:3060] T 1ce4ddf5b10e4ae8b193b13b2e5936ee P 974b802141bd4f13beffef2aa03f88f9 [term 3 FOLLOWER]: Advancing to term 4
I20260129 02:59:34.028955 20480 leader_election.cc:304] T c2633a6fc34c44579de348d208b0c01e P 974b802141bd4f13beffef2aa03f88f9 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 974b802141bd4f13beffef2aa03f88f9; no voters: 197b880fa39d432a9b472845876bfaf0, 1c50de89a5a845668696dcb28c3badd6
I20260129 02:59:34.029152 20509 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "36ff33be666a4c82954731691dbcf327" candidate_uuid: "1c50de89a5a845668696dcb28c3badd6" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "974b802141bd4f13beffef2aa03f88f9" is_pre_election: true
I20260129 02:59:34.030285 20480 leader_election.cc:304] T a1a39dcd328e4268ae41b988c66b4a75 P 974b802141bd4f13beffef2aa03f88f9 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 974b802141bd4f13beffef2aa03f88f9; no voters: 197b880fa39d432a9b472845876bfaf0, 1c50de89a5a845668696dcb28c3badd6
I20260129 02:59:34.031076 20510 raft_consensus.cc:1275] T fcddab6bca684adf8ea032289a00f90e P 974b802141bd4f13beffef2aa03f88f9 [term 6 FOLLOWER]: Refusing update from remote peer 1c50de89a5a845668696dcb28c3badd6: Log matching property violated. Preceding OpId in replica: term: 4 index: 4. Preceding OpId from leader: term: 6 index: 5. (index mismatch)
I20260129 02:59:34.008635 22164 raft_consensus.cc:3055] T 76f713c3331540d2a5b76164103e9eb1 P 1c50de89a5a845668696dcb28c3badd6 [term 2 LEADER]: Stepping down as leader of term 2
I20260129 02:59:34.032104 22164 raft_consensus.cc:740] T 76f713c3331540d2a5b76164103e9eb1 P 1c50de89a5a845668696dcb28c3badd6 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 1c50de89a5a845668696dcb28c3badd6, State: Running, Role: LEADER
I20260129 02:59:34.032213 22184 raft_consensus.cc:2749] T cedce331e1584ac8b636c9b8a72b2ba0 P 974b802141bd4f13beffef2aa03f88f9 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260129 02:59:34.032315 20513 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "913fc7d751ed4b7388d889aac30a5f9b" candidate_uuid: "1c50de89a5a845668696dcb28c3badd6" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "974b802141bd4f13beffef2aa03f88f9" is_pre_election: true
I20260129 02:59:34.033041 22184 raft_consensus.cc:2749] T c2633a6fc34c44579de348d208b0c01e P 974b802141bd4f13beffef2aa03f88f9 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260129 02:59:34.033352 22157 raft_consensus.cc:2749] T a1a39dcd328e4268ae41b988c66b4a75 P 974b802141bd4f13beffef2aa03f88f9 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260129 02:59:34.033635 20511 raft_consensus.cc:1240] T a644c02b16af47679824ccd060ac10fd P 974b802141bd4f13beffef2aa03f88f9 [term 2 FOLLOWER]: Rejecting Update request from peer 1c50de89a5a845668696dcb28c3badd6 for earlier term 1. Current term is 2. Ops: []
I20260129 02:59:34.000453 20479 leader_election.cc:304] T 1299723a4476420ab02fd8bfd7aa0645 P 974b802141bd4f13beffef2aa03f88f9 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 974b802141bd4f13beffef2aa03f88f9; no voters: 197b880fa39d432a9b472845876bfaf0, 7a9daf5d42a84615a6383cfec9a10608
I20260129 02:59:34.033070 20513 raft_consensus.cc:2468] T 913fc7d751ed4b7388d889aac30a5f9b P 974b802141bd4f13beffef2aa03f88f9 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1c50de89a5a845668696dcb28c3badd6 in term 1.
I20260129 02:59:34.034417 22164 consensus_queue.cc:260] T 76f713c3331540d2a5b76164103e9eb1 P 1c50de89a5a845668696dcb28c3badd6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:59:34.035343 22079 raft_consensus.cc:2749] T 96609ac68bdf4993a12e1c23355ca013 P 1c50de89a5a845668696dcb28c3badd6 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260129 02:59:34.035740 20479 leader_election.cc:304] T e33a417cf9c54cf5af73a4ee3aac4d3a P 974b802141bd4f13beffef2aa03f88f9 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 197b880fa39d432a9b472845876bfaf0, 974b802141bd4f13beffef2aa03f88f9; no voters:
I20260129 02:59:34.036046 22197 raft_consensus.cc:2749] T 1299723a4476420ab02fd8bfd7aa0645 P 974b802141bd4f13beffef2aa03f88f9 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260129 02:59:34.036787 22197 raft_consensus.cc:2804] T e33a417cf9c54cf5af73a4ee3aac4d3a P 974b802141bd4f13beffef2aa03f88f9 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260129 02:59:34.037137 22197 raft_consensus.cc:493] T e33a417cf9c54cf5af73a4ee3aac4d3a P 974b802141bd4f13beffef2aa03f88f9 [term 3 FOLLOWER]: Starting leader election (detected failure of leader 1c50de89a5a845668696dcb28c3badd6)
I20260129 02:59:34.037529 22197 raft_consensus.cc:3060] T e33a417cf9c54cf5af73a4ee3aac4d3a P 974b802141bd4f13beffef2aa03f88f9 [term 3 FOLLOWER]: Advancing to term 4
I20260129 02:59:34.023011 20407 leader_election.cc:304] T 368953c9102643b2a21f99af36b51905 P 197b880fa39d432a9b472845876bfaf0 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 197b880fa39d432a9b472845876bfaf0; no voters: 1c50de89a5a845668696dcb28c3badd6, 974b802141bd4f13beffef2aa03f88f9
I20260129 02:59:34.040666 20336 rpcz_store.cc:275] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:55364 (request call id 259) took 6936 ms. Trace:
I20260129 02:59:34.042163 22233 raft_consensus.cc:2749] T 368953c9102643b2a21f99af36b51905 P 197b880fa39d432a9b472845876bfaf0 [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260129 02:59:34.035816 22164 raft_consensus.cc:3060] T 76f713c3331540d2a5b76164103e9eb1 P 1c50de89a5a845668696dcb28c3badd6 [term 2 FOLLOWER]: Advancing to term 3
I20260129 02:59:34.033597 20553 leader_election.cc:304] T 4e3805ffa70242e8ade66da260dddf66 P 1c50de89a5a845668696dcb28c3badd6 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 197b880fa39d432a9b472845876bfaf0, 1c50de89a5a845668696dcb28c3badd6; no voters: 7a9daf5d42a84615a6383cfec9a10608
I20260129 02:59:34.047639 20515 raft_consensus.cc:2468] T 1ce4ddf5b10e4ae8b193b13b2e5936ee P 974b802141bd4f13beffef2aa03f88f9 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1c50de89a5a845668696dcb28c3badd6 in term 4.
I20260129 02:59:34.018090 22061 consensus_queue.cc:260] T 75b0b6e82ddc4f92b1fe1d94f9a0b73b P 1c50de89a5a845668696dcb28c3badd6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } }
W20260129 02:59:34.010152 20369 debug-util.cc:398] Leaking SignalData structure 0x7b08003c4180 after lost signal to thread 20614
W20260129 02:59:34.054129 20369 debug-util.cc:398] Leaking SignalData structure 0x7b08003c41c0 after lost signal to thread 20615
I20260129 02:59:34.054631 20437 raft_consensus.cc:2468] T 0f20af7fbe5d4db89a6f17948d98912c P 197b880fa39d432a9b472845876bfaf0 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1c50de89a5a845668696dcb28c3badd6 in term 4.
W20260129 02:59:34.055312 20369 debug-util.cc:398] Leaking SignalData structure 0x7b080003a960 after lost signal to thread 21160
W20260129 02:59:34.055796 20369 debug-util.cc:398] Leaking SignalData structure 0x7b0800353f80 after lost signal to thread 22049
I20260129 02:59:34.051009 20514 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a0fcc58d6fb6473cafbd9d34d5195269" candidate_uuid: "1c50de89a5a845668696dcb28c3badd6" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "974b802141bd4f13beffef2aa03f88f9" is_pre_election: true
I20260129 02:59:34.052347 20440 raft_consensus.cc:2468] T 18f30ebaa4174fd28c958e72d0cc7e24 P 197b880fa39d432a9b472845876bfaf0 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1c50de89a5a845668696dcb28c3badd6 in term 2.
I20260129 02:59:33.973418 22212 raft_consensus.cc:3060] T febffe4a8b394368b69b1d164308fb9a P 7a9daf5d42a84615a6383cfec9a10608 [term 1 FOLLOWER]: Advancing to term 2
I20260129 02:59:34.061288 22222 raft_consensus.cc:3060] T cb0b3f259ebc4310927338a8740e87b8 P 7a9daf5d42a84615a6383cfec9a10608 [term 3 FOLLOWER]: Advancing to term 4
I20260129 02:59:34.062052 20588 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "75b0b6e82ddc4f92b1fe1d94f9a0b73b" candidate_uuid: "7a9daf5d42a84615a6383cfec9a10608" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "1c50de89a5a845668696dcb28c3badd6"
I20260129 02:59:34.041052 20336 rpcz_store.cc:276] 0129 02:59:27.103914 (+ 0us) service_pool.cc:168] Inserting onto call queue
0129 02:59:27.104088 (+ 174us) service_pool.cc:225] Handling call
0129 02:59:27.485033 (+380945us) spinlock_profiling.cc:243] Waited 99.6 ms on lock 0x7b54003ac854. stack: 00007f8fe044e373 00007f8fe044e270 00007f8fdfda0574 00007f8fe9a5c6b4 00007f8fe9a5c649 00007f8fe34980c0 00007f8fe3536bba 00007f8fe92b7eac 00007f8fe93b210e 00007f8fe78ad4a4 00007f8fe78ad432 00007f8fe78ad361 00007f8fe78ad2dc 00007f8fe78ac592 00007f8fe186d34c 00007f8fe186c786
0129 02:59:34.040458 (+6555425us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0129 02:59:29.380513 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0129 02:59:29.528425 (+147912us) write_op.cc:432] Acquiring schema lock in shared mode
0129 02:59:29.553942 (+ 25517us) write_op.cc:435] Acquired schema lock
0129 02:59:29.565103 (+ 11161us) tablet.cc:660] Decoding operations
0129 02:59:29.675640 (+110537us) write_op.cc:620] Acquiring the partition lock for write op
0129 02:59:29.682289 (+ 6649us) write_op.cc:641] Partition lock acquired for write op
0129 02:59:29.690502 (+ 8213us) tablet.cc:683] Acquiring locks for 1 operations
0129 02:59:29.726251 (+ 35749us) tablet.cc:699] Row locks acquired
0129 02:59:29.728488 (+ 2237us) write_op.cc:260] PREPARE: finished
0129 02:59:29.803237 (+ 74749us) write_op.cc:270] Start()
0129 02:59:29.813487 (+ 10250us) write_op.cc:276] Timestamp: P: 1769655569796496 usec, L: 0
0129 02:59:29.826718 (+ 13231us) op_driver.cc:348] REPLICATION: starting
0129 02:59:29.911696 (+ 84978us) log.cc:844] Serialized 533 byte log entry
0129 02:59:33.436210 (+3524514us) op_driver.cc:464] REPLICATION: finished
0129 02:59:34.001868 (+565658us) write_op.cc:301] APPLY: starting
0129 02:59:34.001976 (+ 108us) tablet.cc:1365] starting BulkCheckPresence
0129 02:59:34.002124 (+ 148us) tablet.cc:1368] finished BulkCheckPresence
0129 02:59:34.008429 (+ 6305us) tablet.cc:1370] starting ApplyRowOperation cycle
0129 02:59:34.008759 (+ 330us) tablet.cc:1381] finished ApplyRowOperation cycle
0129 02:59:34.008983 (+ 224us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0129 02:59:34.009045 (+ 62us) write_op.cc:312] APPLY: finished
0129 02:59:34.036981 (+ 27936us) log.cc:844] Serialized 23 byte log entry
0129 02:59:34.037195 (+ 214us) write_op.cc:489] Releasing partition, row and schema locks
0129 02:59:34.037321 (+ 126us) write_op.cc:454] Released schema lock
0129 02:59:34.037375 (+ 54us) write_op.cc:341] FINISH: Updating metrics
0129 02:59:34.037882 (+ 507us) write_op.cc:489] Releasing partition, row and schema locks
0129 02:59:34.037917 (+ 35us) write_op.cc:454] Released schema lock
Metrics: {"reported_tablets":1,"spinlock_wait_cycles":229018880,"child_traces":[["op",{"apply.queue_time_us":565316,"apply.run_cpu_time_us":2880,"apply.run_wall_time_us":35933,"num_ops":1,"prepare.queue_time_us":952486,"prepare.run_cpu_time_us":12857,"prepare.run_wall_time_us":572337,"replication_time_us":3609222,"thread_start_us":346178,"threads_started":2}]]}
I20260129 02:59:34.063455 20511 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0f20af7fbe5d4db89a6f17948d98912c" candidate_uuid: "1c50de89a5a845668696dcb28c3badd6" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "974b802141bd4f13beffef2aa03f88f9"
I20260129 02:59:34.064121 20516 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "cb0b3f259ebc4310927338a8740e87b8" candidate_uuid: "1c50de89a5a845668696dcb28c3badd6" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "974b802141bd4f13beffef2aa03f88f9"
I20260129 02:59:34.064359 22224 raft_consensus.cc:2804] T 4e3805ffa70242e8ade66da260dddf66 P 1c50de89a5a845668696dcb28c3badd6 [term 3 FOLLOWER]: Leader election won for term 3
I20260129 02:59:34.064541 20553 leader_election.cc:304] T 368953c9102643b2a21f99af36b51905 P 1c50de89a5a845668696dcb28c3badd6 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 197b880fa39d432a9b472845876bfaf0, 1c50de89a5a845668696dcb28c3badd6; no voters: 974b802141bd4f13beffef2aa03f88f9
I20260129 02:59:34.065743 20335 catalog_manager.cc:5654] T a5a05fbcbbcf4916bbc01fd1a5b5e90d P 1c50de89a5a845668696dcb28c3badd6 reported cstate change: term changed from 2 to 4. New cstate: current_term: 4 leader_uuid: "1c50de89a5a845668696dcb28c3badd6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } health_report { overall_health: HEALTHY } } }
W20260129 02:59:34.066792 22243 consensus_meta.cc:287] T 0340229d521b432580ae926f6062e3f7 P 1c50de89a5a845668696dcb28c3badd6: Time spent flushing consensus metadata: real 0.518s user 0.009s sys 0.020s
I20260129 02:59:34.067054 20507 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4f8f3a38f6e24ba4857875181f447442" candidate_uuid: "1c50de89a5a845668696dcb28c3badd6" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "974b802141bd4f13beffef2aa03f88f9"
I20260129 02:59:34.067953 20479 leader_election.cc:304] T 4b1876f9a70b49edb3aa74105e66d3b3 P 974b802141bd4f13beffef2aa03f88f9 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 197b880fa39d432a9b472845876bfaf0, 974b802141bd4f13beffef2aa03f88f9; no voters:
I20260129 02:59:34.069113 20589 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "75be4bd6bd45474795fd472eff2953ac" candidate_uuid: "197b880fa39d432a9b472845876bfaf0" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "1c50de89a5a845668696dcb28c3badd6"
I20260129 02:59:34.070156 22103 raft_consensus.cc:2749] T 75f45538dc704d9a8ec4ff8ad9f3ea5b P 1c50de89a5a845668696dcb28c3badd6 [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260129 02:59:34.070827 20587 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "dc2f4cb752cf4d59b5f6bc689f5d8104" candidate_uuid: "197b880fa39d432a9b472845876bfaf0" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "1c50de89a5a845668696dcb28c3badd6" is_pre_election: true
I20260129 02:59:34.071555 22082 leader_election.cc:290] T 0f20af7fbe5d4db89a6f17948d98912c P 197b880fa39d432a9b472845876bfaf0 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 974b802141bd4f13beffef2aa03f88f9 (127.18.211.2:34517), 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3:37735)
I20260129 02:59:34.071959 20581 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fcddab6bca684adf8ea032289a00f90e" candidate_uuid: "197b880fa39d432a9b472845876bfaf0" candidate_term: 7 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "1c50de89a5a845668696dcb28c3badd6" is_pre_election: true
I20260129 02:59:34.088056 20581 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "926a250fa5834e88b96ccecbefbf5086" candidate_uuid: "197b880fa39d432a9b472845876bfaf0" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "1c50de89a5a845668696dcb28c3badd6" is_pre_election: true
I20260129 02:59:34.089402 20581 raft_consensus.cc:1240] T c2633a6fc34c44579de348d208b0c01e P 1c50de89a5a845668696dcb28c3badd6 [term 2 FOLLOWER]: Rejecting Update request from peer 197b880fa39d432a9b472845876bfaf0 for earlier term 1. Current term is 2. Ops: []
I20260129 02:59:34.072203 22213 leader_election.cc:290] T a8479d1fe0bd41e5946186e3a6ca8c7a P 1c50de89a5a845668696dcb28c3badd6 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 197b880fa39d432a9b472845876bfaf0 (127.18.211.1:42027), 974b802141bd4f13beffef2aa03f88f9 (127.18.211.2:34517)
I20260129 02:59:34.089591 22160 raft_consensus.cc:515] T e7d83d4ccf684e76acb29a0dc844e1b8 P 974b802141bd4f13beffef2aa03f88f9 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:59:34.072887 20585 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c891d2bff4ce4568861bbc992e91fbb0" candidate_uuid: "197b880fa39d432a9b472845876bfaf0" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "1c50de89a5a845668696dcb28c3badd6" is_pre_election: true
I20260129 02:59:34.073029 20584 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e33a417cf9c54cf5af73a4ee3aac4d3a" candidate_uuid: "197b880fa39d432a9b472845876bfaf0" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "1c50de89a5a845668696dcb28c3badd6" is_pre_election: true
I20260129 02:59:34.074239 20508 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c2633a6fc34c44579de348d208b0c01e" candidate_uuid: "1c50de89a5a845668696dcb28c3badd6" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "974b802141bd4f13beffef2aa03f88f9"
I20260129 02:59:34.090590 20582 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "96609ac68bdf4993a12e1c23355ca013" candidate_uuid: "197b880fa39d432a9b472845876bfaf0" candidate_term: 4 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "1c50de89a5a845668696dcb28c3badd6" is_pre_election: true
I20260129 02:59:34.091081 20508 raft_consensus.cc:3060] T c2633a6fc34c44579de348d208b0c01e P 974b802141bd4f13beffef2aa03f88f9 [term 1 FOLLOWER]: Advancing to term 2
I20260129 02:59:34.091187 20581 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e4d18c35b0a7493e85d0a4b92956a48f" candidate_uuid: "197b880fa39d432a9b472845876bfaf0" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "1c50de89a5a845668696dcb28c3badd6"
I20260129 02:59:34.091329 20582 raft_consensus.cc:2468] T 96609ac68bdf4993a12e1c23355ca013 P 1c50de89a5a845668696dcb28c3badd6 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 197b880fa39d432a9b472845876bfaf0 in term 3.
I20260129 02:59:34.092586 20509 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4f5a77e94fef4b16a2ba7b067c8b144a" candidate_uuid: "1c50de89a5a845668696dcb28c3badd6" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "974b802141bd4f13beffef2aa03f88f9"
I20260129 02:59:34.093655 20584 raft_consensus.cc:1240] T a4c1226b987b468fa84dfc6ba0e2dbe1 P 1c50de89a5a845668696dcb28c3badd6 [term 3 LEADER]: Rejecting Update request from peer 197b880fa39d432a9b472845876bfaf0 for earlier term 2. Current term is 3. Ops: [2.2-2.2]
I20260129 02:59:34.094255 20509 raft_consensus.cc:1275] T 75be4bd6bd45474795fd472eff2953ac P 974b802141bd4f13beffef2aa03f88f9 [term 7 FOLLOWER]: Refusing update from remote peer 1c50de89a5a845668696dcb28c3badd6: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 7 index: 5. (index mismatch)
W20260129 02:59:34.096431 21100 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.211.3:37735, user_credentials={real_user=slave}} blocked reactor thread for 376176us
I20260129 02:59:34.098526 20508 raft_consensus.cc:2468] T c2633a6fc34c44579de348d208b0c01e P 974b802141bd4f13beffef2aa03f88f9 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1c50de89a5a845668696dcb28c3badd6 in term 2.
I20260129 02:59:34.098881 20556 leader_election.cc:304] T 36ff33be666a4c82954731691dbcf327 P 1c50de89a5a845668696dcb28c3badd6 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1c50de89a5a845668696dcb28c3badd6; no voters: 197b880fa39d432a9b472845876bfaf0, 974b802141bd4f13beffef2aa03f88f9
I20260129 02:59:34.099037 21100 leader_election.cc:304] T fd8311b779af4e1fa1402b2a02a602f2 P 7a9daf5d42a84615a6383cfec9a10608 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1c50de89a5a845668696dcb28c3badd6, 7a9daf5d42a84615a6383cfec9a10608; no voters:
I20260129 02:59:34.099486 20585 raft_consensus.cc:2468] T c891d2bff4ce4568861bbc992e91fbb0 P 1c50de89a5a845668696dcb28c3badd6 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 197b880fa39d432a9b472845876bfaf0 in term 6.
I20260129 02:59:34.099828 20584 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2e58e9a5a8aa4cecafb464addb4f35ed" candidate_uuid: "197b880fa39d432a9b472845876bfaf0" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "1c50de89a5a845668696dcb28c3badd6"
I20260129 02:59:34.100432 20585 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e7d83d4ccf684e76acb29a0dc844e1b8" candidate_uuid: "197b880fa39d432a9b472845876bfaf0" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "1c50de89a5a845668696dcb28c3badd6" is_pre_election: true
I20260129 02:59:34.100850 22201 leader_election.cc:290] T 3e2d15f8640b407195210c476e09df91 P 974b802141bd4f13beffef2aa03f88f9 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3:37735), 7a9daf5d42a84615a6383cfec9a10608 (127.18.211.4:40439)
I20260129 02:59:34.100925 20584 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "36ff33be666a4c82954731691dbcf327" candidate_uuid: "197b880fa39d432a9b472845876bfaf0" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1c50de89a5a845668696dcb28c3badd6"
I20260129 02:59:34.101480 20585 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4f8f3a38f6e24ba4857875181f447442" candidate_uuid: "197b880fa39d432a9b472845876bfaf0" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "1c50de89a5a845668696dcb28c3badd6" is_pre_election: true
I20260129 02:59:34.101769 20584 raft_consensus.cc:2393] T 36ff33be666a4c82954731691dbcf327 P 1c50de89a5a845668696dcb28c3badd6 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 197b880fa39d432a9b472845876bfaf0 in current term 2: Already voted for candidate 1c50de89a5a845668696dcb28c3badd6 in this term.
I20260129 02:59:34.102154 20585 raft_consensus.cc:2393] T 4f8f3a38f6e24ba4857875181f447442 P 1c50de89a5a845668696dcb28c3badd6 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 197b880fa39d432a9b472845876bfaf0 in current term 4: Already voted for candidate 1c50de89a5a845668696dcb28c3badd6 in this term.
I20260129 02:59:34.103432 20514 raft_consensus.cc:1240] T add8d4923d1a489e86882c3a25692b95 P 974b802141bd4f13beffef2aa03f88f9 [term 2 FOLLOWER]: Rejecting Update request from peer 1c50de89a5a845668696dcb28c3badd6 for earlier term 1. Current term is 2. Ops: []
I20260129 02:59:34.103554 20581 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4b1876f9a70b49edb3aa74105e66d3b3" candidate_uuid: "197b880fa39d432a9b472845876bfaf0" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "1c50de89a5a845668696dcb28c3badd6" is_pre_election: true
I20260129 02:59:34.104449 20515 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7a9a69cd82724b1faed1d10aa091fc4a" candidate_uuid: "1c50de89a5a845668696dcb28c3badd6" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "974b802141bd4f13beffef2aa03f88f9"
W20260129 02:59:34.104789 22257 consensus_meta.cc:287] T 00e79d8f25a34676b1d866eb2068a2b2 P 7a9daf5d42a84615a6383cfec9a10608: Time spent flushing consensus metadata: real 0.743s user 0.014s sys 0.000s
I20260129 02:59:34.105010 20515 raft_consensus.cc:3060] T 7a9a69cd82724b1faed1d10aa091fc4a P 974b802141bd4f13beffef2aa03f88f9 [term 3 FOLLOWER]: Advancing to term 4
I20260129 02:59:34.105247 22257 raft_consensus.cc:515] T 00e79d8f25a34676b1d866eb2068a2b2 P 7a9daf5d42a84615a6383cfec9a10608 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } }
I20260129 02:59:34.105618 22243 raft_consensus.cc:515] T 0340229d521b432580ae926f6062e3f7 P 1c50de89a5a845668696dcb28c3badd6 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:59:34.107779 22257 leader_election.cc:290] T 00e79d8f25a34676b1d866eb2068a2b2 P 7a9daf5d42a84615a6383cfec9a10608 [CANDIDATE]: Term 4 election: Requested vote from peers 197b880fa39d432a9b472845876bfaf0 (127.18.211.1:42027), 974b802141bd4f13beffef2aa03f88f9 (127.18.211.2:34517)
I20260129 02:59:34.108131 20510 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "724a236524cb4e7ba3428880681d6cc2" candidate_uuid: "1c50de89a5a845668696dcb28c3badd6" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "974b802141bd4f13beffef2aa03f88f9"
I20260129 02:59:34.109004 22257 raft_consensus.cc:2804] T fd8311b779af4e1fa1402b2a02a602f2 P 7a9daf5d42a84615a6383cfec9a10608 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260129 02:59:34.109380 22257 raft_consensus.cc:493] T fd8311b779af4e1fa1402b2a02a602f2 P 7a9daf5d42a84615a6383cfec9a10608 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 197b880fa39d432a9b472845876bfaf0)
I20260129 02:59:34.109723 22257 raft_consensus.cc:3060] T fd8311b779af4e1fa1402b2a02a602f2 P 7a9daf5d42a84615a6383cfec9a10608 [term 2 FOLLOWER]: Advancing to term 3
I20260129 02:59:34.109855 20479 leader_election.cc:304] T 96609ac68bdf4993a12e1c23355ca013 P 974b802141bd4f13beffef2aa03f88f9 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 197b880fa39d432a9b472845876bfaf0, 7a9daf5d42a84615a6383cfec9a10608, 974b802141bd4f13beffef2aa03f88f9; no voters:
I20260129 02:59:34.110858 20479 leader_election.cc:304] T a5a05fbcbbcf4916bbc01fd1a5b5e90d P 974b802141bd4f13beffef2aa03f88f9 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 197b880fa39d432a9b472845876bfaf0, 974b802141bd4f13beffef2aa03f88f9; no voters:
I20260129 02:59:34.111274 20590 raft_consensus.cc:2468] T 75f45538dc704d9a8ec4ff8ad9f3ea5b P 1c50de89a5a845668696dcb28c3badd6 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7a9daf5d42a84615a6383cfec9a10608 in term 2.
I20260129 02:59:34.112200 20512 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "36ff33be666a4c82954731691dbcf327" candidate_uuid: "1c50de89a5a845668696dcb28c3badd6" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "974b802141bd4f13beffef2aa03f88f9"
I20260129 02:59:34.112788 20512 raft_consensus.cc:3055] T 36ff33be666a4c82954731691dbcf327 P 974b802141bd4f13beffef2aa03f88f9 [term 1 LEADER]: Stepping down as leader of term 1
I20260129 02:59:34.113056 20512 raft_consensus.cc:740] T 36ff33be666a4c82954731691dbcf327 P 974b802141bd4f13beffef2aa03f88f9 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 974b802141bd4f13beffef2aa03f88f9, State: Running, Role: LEADER
I20260129 02:59:34.113808 20512 consensus_queue.cc:260] T 36ff33be666a4c82954731691dbcf327 P 974b802141bd4f13beffef2aa03f88f9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:59:34.113955 20335 catalog_manager.cc:5654] T 7d9494f5584d4d5db8b9adf1c8fc888b P 1c50de89a5a845668696dcb28c3badd6 reported cstate change: term changed from 1 to 2, leader changed from 974b802141bd4f13beffef2aa03f88f9 (127.18.211.2) to 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3). New cstate: current_term: 2 leader_uuid: "1c50de89a5a845668696dcb28c3badd6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } health_report { overall_health: HEALTHY } } }
I20260129 02:59:34.114764 20512 raft_consensus.cc:3060] T 36ff33be666a4c82954731691dbcf327 P 974b802141bd4f13beffef2aa03f88f9 [term 1 FOLLOWER]: Advancing to term 2
I20260129 02:59:34.081732 20435 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55672 (request call id 1384) took 4065 ms. Trace:
I20260129 02:59:34.086578 20583 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a5a05fbcbbcf4916bbc01fd1a5b5e90d" candidate_uuid: "197b880fa39d432a9b472845876bfaf0" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "1c50de89a5a845668696dcb28c3badd6" is_pre_election: true
I20260129 02:59:34.116206 20435 rpcz_store.cc:276] 0129 02:59:29.990488 (+ 0us) service_pool.cc:168] Inserting onto call queue
0129 02:59:33.970705 (+3980217us) service_pool.cc:225] Handling call
0129 02:59:33.990519 (+ 19814us) raft_consensus.cc:1471] Updating replica for 0 ops
0129 02:59:34.055160 (+ 64641us) raft_consensus.cc:1534] Early marking committed up to index 0
0129 02:59:34.055193 (+ 33us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0129 02:59:34.055236 (+ 43us) raft_consensus.cc:1645] Marking committed up to 1
0129 02:59:34.055264 (+ 28us) raft_consensus.cc:1697] Filling consensus response to leader.
0129 02:59:34.055424 (+ 160us) raft_consensus.cc:1691] UpdateReplicas() finished
0129 02:59:34.055539 (+ 115us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20260129 02:59:34.082137 20405 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.211.4:40439, user_credentials={real_user=slave}} blocked reactor thread for 65521.5us
I20260129 02:59:34.084323 20438 raft_consensus.cc:1240] T 0f20af7fbe5d4db89a6f17948d98912c P 197b880fa39d432a9b472845876bfaf0 [term 4 FOLLOWER]: Rejecting Update request from peer 974b802141bd4f13beffef2aa03f88f9 for earlier term 3. Current term is 4. Ops: []
I20260129 02:59:34.085090 20589 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4f5a77e94fef4b16a2ba7b067c8b144a" candidate_uuid: "7a9daf5d42a84615a6383cfec9a10608" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "1c50de89a5a845668696dcb28c3badd6"
I20260129 02:59:34.119536 20589 raft_consensus.cc:2393] T 4f5a77e94fef4b16a2ba7b067c8b144a P 1c50de89a5a845668696dcb28c3badd6 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 7a9daf5d42a84615a6383cfec9a10608 in current term 2: Already voted for candidate 1c50de89a5a845668696dcb28c3badd6 in this term.
W20260129 02:59:34.086159 22190 consensus_meta.cc:287] T a8479d1fe0bd41e5946186e3a6ca8c7a P 197b880fa39d432a9b472845876bfaf0: Time spent flushing consensus metadata: real 0.625s user 0.012s sys 0.000s
W20260129 02:59:34.055029 22137 consensus_meta.cc:287] T febffe4a8b394368b69b1d164308fb9a P 974b802141bd4f13beffef2aa03f88f9: Time spent flushing consensus metadata: real 0.611s user 0.008s sys 0.004s
I20260129 02:59:34.115635 20335 catalog_manager.cc:5654] T 926a250fa5834e88b96ccecbefbf5086 P 1c50de89a5a845668696dcb28c3badd6 reported cstate change: term changed from 2 to 4. New cstate: current_term: 4 leader_uuid: "1c50de89a5a845668696dcb28c3badd6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } health_report { overall_health: HEALTHY } } }
I20260129 02:59:34.086457 20587 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "724a236524cb4e7ba3428880681d6cc2" candidate_uuid: "197b880fa39d432a9b472845876bfaf0" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1c50de89a5a845668696dcb28c3badd6" is_pre_election: true
I20260129 02:59:34.072647 22061 raft_consensus.cc:3060] T 75b0b6e82ddc4f92b1fe1d94f9a0b73b P 1c50de89a5a845668696dcb28c3badd6 [term 1 FOLLOWER]: Advancing to term 2
I20260129 02:59:34.070875 22112 raft_consensus.cc:3060] T 505580c93c694bbb8ea96fdf9b17ab46 P 974b802141bd4f13beffef2aa03f88f9 [term 2 FOLLOWER]: Advancing to term 3
I20260129 02:59:34.131729 20438 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55672 (request call id 1388) took 3553 ms. Trace:
I20260129 02:59:34.125236 20439 raft_consensus.cc:2468] T 4f8f3a38f6e24ba4857875181f447442 P 197b880fa39d432a9b472845876bfaf0 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1c50de89a5a845668696dcb28c3badd6 in term 4.
I20260129 02:59:34.107672 22243 leader_election.cc:290] T 0340229d521b432580ae926f6062e3f7 P 1c50de89a5a845668696dcb28c3badd6 [CANDIDATE]: Term 4 election: Requested vote from peers 197b880fa39d432a9b472845876bfaf0 (127.18.211.1:42027), 974b802141bd4f13beffef2aa03f88f9 (127.18.211.2:34517)
I20260129 02:59:34.132157 20438 rpcz_store.cc:276] 0129 02:59:30.578354 (+ 0us) service_pool.cc:168] Inserting onto call queue
0129 02:59:33.991644 (+3413290us) service_pool.cc:225] Handling call
0129 02:59:34.084071 (+ 92427us) raft_consensus.cc:1471] Updating replica for 0 ops
0129 02:59:34.118836 (+ 34765us) raft_consensus.cc:1697] Filling consensus response to leader.
0129 02:59:34.131516 (+ 12680us) spinlock_profiling.cc:243] Waited 85.9 ms on lock 0x7b54002e0c50. stack: 00007f8fe044e373 00007f8fe044e270 00007f8fdfda0574 00007f8fe9a5c6b4 00007f8fe9a5c649 00007f8fe34975a1 00007f8fe352d52d 00007f8fe84e1f49 00007f8fe1a6ed34 00007f8fe1a6ecc2 00007f8fe1a6ebf1 00007f8fe1a6eb6c 00007f8fe1a6de22 00007f8fe186d34c 00007f8fe186c786 00007f8fe186c12f
0129 02:59:34.131671 (+ 155us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":197616384}
I20260129 02:59:34.132738 20432 raft_consensus.cc:1240] T 4f8f3a38f6e24ba4857875181f447442 P 197b880fa39d432a9b472845876bfaf0 [term 4 FOLLOWER]: Rejecting Update request from peer 974b802141bd4f13beffef2aa03f88f9 for earlier term 3. Current term is 4. Ops: []
I20260129 02:59:34.133589 20587 raft_consensus.cc:2393] T 724a236524cb4e7ba3428880681d6cc2 P 1c50de89a5a845668696dcb28c3badd6 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 197b880fa39d432a9b472845876bfaf0 in current term 2: Already voted for candidate 1c50de89a5a845668696dcb28c3badd6 in this term.
I20260129 02:59:34.117070 20583 raft_consensus.cc:2468] T a5a05fbcbbcf4916bbc01fd1a5b5e90d P 1c50de89a5a845668696dcb28c3badd6 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 197b880fa39d432a9b472845876bfaf0 in term 5.
I20260129 02:59:34.137702 20335 catalog_manager.cc:5654] T cedce331e1584ac8b636c9b8a72b2ba0 P 1c50de89a5a845668696dcb28c3badd6 reported cstate change: term changed from 2 to 4. New cstate: current_term: 4 leader_uuid: "1c50de89a5a845668696dcb28c3badd6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } health_report { overall_health: HEALTHY } } }
I20260129 02:59:34.140470 20515 raft_consensus.cc:2468] T 7a9a69cd82724b1faed1d10aa091fc4a P 974b802141bd4f13beffef2aa03f88f9 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1c50de89a5a845668696dcb28c3badd6 in term 4.
W20260129 02:59:34.141744 21099 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260129 02:59:34.143234 20553 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.211.1:42027, user_credentials={real_user=slave}} blocked reactor thread for 78782.7us
I20260129 02:59:34.146188 20507 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a8479d1fe0bd41e5946186e3a6ca8c7a" candidate_uuid: "1c50de89a5a845668696dcb28c3badd6" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "974b802141bd4f13beffef2aa03f88f9" is_pre_election: true
W20260129 02:59:34.133265 22231 consensus_meta.cc:287] T 1299723a4476420ab02fd8bfd7aa0645 P 197b880fa39d432a9b472845876bfaf0: Time spent flushing consensus metadata: real 0.601s user 0.010s sys 0.000s
I20260129 02:59:34.147650 20441 raft_consensus.cc:2468] T 724a236524cb4e7ba3428880681d6cc2 P 197b880fa39d432a9b472845876bfaf0 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1c50de89a5a845668696dcb28c3badd6 in term 2.
W20260129 02:59:34.157567 20556 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.211.2:34517, user_credentials={real_user=slave}} blocked reactor thread for 58792.4us
I20260129 02:59:34.148990 20582 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4408006eb5174de1896b037cf3f18457" candidate_uuid: "197b880fa39d432a9b472845876bfaf0" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1c50de89a5a845668696dcb28c3badd6" is_pre_election: true
I20260129 02:59:34.117939 22092 raft_consensus.cc:697] T 42d27a28d89b409e8807ec0efd97af69 P 197b880fa39d432a9b472845876bfaf0 [term 4 LEADER]: Becoming Leader. State: Replica: 197b880fa39d432a9b472845876bfaf0, State: Running, Role: LEADER
I20260129 02:59:34.161406 22092 consensus_queue.cc:237] T 42d27a28d89b409e8807ec0efd97af69 P 197b880fa39d432a9b472845876bfaf0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } }
I20260129 02:59:34.150434 22103 raft_consensus.cc:2804] T 368953c9102643b2a21f99af36b51905 P 1c50de89a5a845668696dcb28c3badd6 [term 4 FOLLOWER]: Leader election won for term 4
I20260129 02:59:34.164139 20587 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "96609ac68bdf4993a12e1c23355ca013" candidate_uuid: "7a9daf5d42a84615a6383cfec9a10608" candidate_term: 4 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "1c50de89a5a845668696dcb28c3badd6"
I20260129 02:59:34.144094 22201 raft_consensus.cc:2804] T 96609ac68bdf4993a12e1c23355ca013 P 974b802141bd4f13beffef2aa03f88f9 [term 3 FOLLOWER]: Leader election won for term 3
I20260129 02:59:34.164804 20587 raft_consensus.cc:3060] T 96609ac68bdf4993a12e1c23355ca013 P 1c50de89a5a845668696dcb28c3badd6 [term 3 FOLLOWER]: Advancing to term 4
I20260129 02:59:34.165036 22201 raft_consensus.cc:697] T 96609ac68bdf4993a12e1c23355ca013 P 974b802141bd4f13beffef2aa03f88f9 [term 3 LEADER]: Becoming Leader. State: Replica: 974b802141bd4f13beffef2aa03f88f9, State: Running, Role: LEADER
I20260129 02:59:34.165159 21100 leader_election.cc:304] T 75f45538dc704d9a8ec4ff8ad9f3ea5b P 7a9daf5d42a84615a6383cfec9a10608 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1c50de89a5a845668696dcb28c3badd6, 7a9daf5d42a84615a6383cfec9a10608; no voters:
I20260129 02:59:34.165436 20441 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55676 (request call id 1081) took 3271 ms. Trace:
I20260129 02:59:34.165823 20441 rpcz_store.cc:276] 0129 02:59:30.893762 (+ 0us) service_pool.cc:168] Inserting onto call queue
0129 02:59:34.157439 (+3263677us) service_pool.cc:225] Handling call
0129 02:59:34.164946 (+ 7507us) raft_consensus.cc:1471] Updating replica for 0 ops
0129 02:59:34.165113 (+ 167us) raft_consensus.cc:1534] Early marking committed up to index 0
0129 02:59:34.165137 (+ 24us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0129 02:59:34.165177 (+ 40us) raft_consensus.cc:1645] Marking committed up to 2
0129 02:59:34.165219 (+ 42us) raft_consensus.cc:1697] Filling consensus response to leader.
0129 02:59:34.165269 (+ 50us) raft_consensus.cc:1691] UpdateReplicas() finished
0129 02:59:34.165365 (+ 96us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260129 02:59:34.166153 22201 consensus_queue.cc:237] T 96609ac68bdf4993a12e1c23355ca013 P 974b802141bd4f13beffef2aa03f88f9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 2.4, Last appended by leader: 4, Current term: 3, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } attrs { replace: true } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } }
I20260129 02:59:34.133090 22157 raft_consensus.cc:2804] T 4b1876f9a70b49edb3aa74105e66d3b3 P 974b802141bd4f13beffef2aa03f88f9 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260129 02:59:34.167762 22157 raft_consensus.cc:493] T 4b1876f9a70b49edb3aa74105e66d3b3 P 974b802141bd4f13beffef2aa03f88f9 [term 3 FOLLOWER]: Starting leader election (detected failure of leader 1c50de89a5a845668696dcb28c3badd6)
I20260129 02:59:34.168243 22157 raft_consensus.cc:3060] T 4b1876f9a70b49edb3aa74105e66d3b3 P 974b802141bd4f13beffef2aa03f88f9 [term 3 FOLLOWER]: Advancing to term 4
I20260129 02:59:34.137410 22120 raft_consensus.cc:515] T 8055232cf0004d40a85f3bf983fe00a9 P 197b880fa39d432a9b472845876bfaf0 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:59:34.170557 21130 raft_consensus.cc:1240] T 00e79d8f25a34676b1d866eb2068a2b2 P 7a9daf5d42a84615a6383cfec9a10608 [term 4 FOLLOWER]: Rejecting Update request from peer 197b880fa39d432a9b472845876bfaf0 for earlier term 3. Current term is 4. Ops: []
I20260129 02:59:34.170920 22120 leader_election.cc:290] T 8055232cf0004d40a85f3bf983fe00a9 P 197b880fa39d432a9b472845876bfaf0 [CANDIDATE]: Term 2 election: Requested vote from peers 974b802141bd4f13beffef2aa03f88f9 (127.18.211.2:34517), 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3:37735)
I20260129 02:59:34.167744 20441 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55676 (request call id 1082) took 3273 ms. Trace:
I20260129 02:59:34.172169 20441 rpcz_store.cc:276] 0129 02:59:30.894592 (+ 0us) service_pool.cc:168] Inserting onto call queue
0129 02:59:34.166895 (+3272303us) service_pool.cc:225] Handling call
0129 02:59:34.167231 (+ 336us) raft_consensus.cc:1471] Updating replica for 0 ops
0129 02:59:34.167437 (+ 206us) raft_consensus.cc:1534] Early marking committed up to index 0
0129 02:59:34.167460 (+ 23us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0129 02:59:34.167499 (+ 39us) raft_consensus.cc:1645] Marking committed up to 2
0129 02:59:34.167530 (+ 31us) raft_consensus.cc:1697] Filling consensus response to leader.
0129 02:59:34.167583 (+ 53us) raft_consensus.cc:1691] UpdateReplicas() finished
0129 02:59:34.167678 (+ 95us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260129 02:59:34.111325 22197 raft_consensus.cc:515] T e33a417cf9c54cf5af73a4ee3aac4d3a P 974b802141bd4f13beffef2aa03f88f9 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:59:34.150585 20335 catalog_manager.cc:5654] T 4b1876f9a70b49edb3aa74105e66d3b3 P 1c50de89a5a845668696dcb28c3badd6 reported cstate change: term changed from 2 to 3, leader changed from 197b880fa39d432a9b472845876bfaf0 (127.18.211.1) to 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3). New cstate: current_term: 3 leader_uuid: "1c50de89a5a845668696dcb28c3badd6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } health_report { overall_health: HEALTHY } } }
I20260129 02:59:34.155666 20553 leader_election.cc:304] T fd8311b779af4e1fa1402b2a02a602f2 P 1c50de89a5a845668696dcb28c3badd6 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1c50de89a5a845668696dcb28c3badd6; no voters: 197b880fa39d432a9b472845876bfaf0, 7a9daf5d42a84615a6383cfec9a10608
W20260129 02:59:34.160000 20482 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260129 02:59:34.131055 22137 raft_consensus.cc:515] T febffe4a8b394368b69b1d164308fb9a P 974b802141bd4f13beffef2aa03f88f9 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } }
I20260129 02:59:34.159466 22164 raft_consensus.cc:2749] T 36ff33be666a4c82954731691dbcf327 P 1c50de89a5a845668696dcb28c3badd6 [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260129 02:59:34.200623 20432 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55672 (request call id 1383) took 4226 ms. Trace:
I20260129 02:59:34.131454 22190 raft_consensus.cc:515] T a8479d1fe0bd41e5946186e3a6ca8c7a P 197b880fa39d432a9b472845876bfaf0 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:59:34.137090 22237 raft_consensus.cc:515] T 2ac659d732314e1288849f08fe5e074e P 197b880fa39d432a9b472845876bfaf0 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } }
I20260129 02:59:34.185902 20439 raft_consensus.cc:1240] T c934d31efd3f4e5195605ddc354d14e1 P 197b880fa39d432a9b472845876bfaf0 [term 3 LEADER]: Rejecting Update request from peer 7a9daf5d42a84615a6383cfec9a10608 for earlier term 2. Current term is 3. Ops: []
I20260129 02:59:34.228389 20512 raft_consensus.cc:2468] T 36ff33be666a4c82954731691dbcf327 P 974b802141bd4f13beffef2aa03f88f9 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1c50de89a5a845668696dcb28c3badd6 in term 2.
I20260129 02:59:34.220153 20335 catalog_manager.cc:5654] T 76f713c3331540d2a5b76164103e9eb1 P 1c50de89a5a845668696dcb28c3badd6 reported cstate change: term changed from 1 to 2, leader changed from 974b802141bd4f13beffef2aa03f88f9 (127.18.211.2) to 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3). New cstate: current_term: 2 leader_uuid: "1c50de89a5a845668696dcb28c3badd6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } health_report { overall_health: HEALTHY } } }
I20260129 02:59:34.183353 20434 raft_consensus.cc:1240] T 8055232cf0004d40a85f3bf983fe00a9 P 197b880fa39d432a9b472845876bfaf0 [term 2 FOLLOWER]: Rejecting Update request from peer 974b802141bd4f13beffef2aa03f88f9 for earlier term 1. Current term is 2. Ops: []
I20260129 02:59:34.146673 22082 consensus_queue.cc:1059] T 1299723a4476420ab02fd8bfd7aa0645 P 197b880fa39d432a9b472845876bfaf0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false }, Status: INVALID_TERM, Last received: 4.6, Next index: 7, Last known committed idx: 4, Time since last communication: 0.043s
I20260129 02:59:34.202608 20513 raft_consensus.cc:1240] T e33a417cf9c54cf5af73a4ee3aac4d3a P 974b802141bd4f13beffef2aa03f88f9 [term 4 FOLLOWER]: Rejecting Update request from peer 1c50de89a5a845668696dcb28c3badd6 for earlier term 3. Current term is 4. Ops: []
I20260129 02:59:34.179196 22255 raft_consensus.cc:2804] T 75f45538dc704d9a8ec4ff8ad9f3ea5b P 7a9daf5d42a84615a6383cfec9a10608 [term 2 FOLLOWER]: Leader election won for term 2
I20260129 02:59:34.136878 22091 raft_consensus.cc:515] T 4e420f90884f4796997b5ea41e49e411 P 197b880fa39d432a9b472845876bfaf0 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } }
I20260129 02:59:34.228849 20439 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:48356 (request call id 214) took 3552 ms. Trace:
I20260129 02:59:34.227088 20432 rpcz_store.cc:276] 0129 02:59:29.956851 (+ 0us) service_pool.cc:168] Inserting onto call queue
0129 02:59:33.571559 (+3614708us) service_pool.cc:225] Handling call
0129 02:59:34.132510 (+560951us) raft_consensus.cc:1471] Updating replica for 0 ops
0129 02:59:34.138233 (+ 5723us) raft_consensus.cc:1697] Filling consensus response to leader.
0129 02:59:34.183361 (+ 45128us) spinlock_profiling.cc:243] Waited 451 ms on lock 0x7b5400075dd0. stack: 00007f8fe044e373 00007f8fe044e270 00007f8fdfda0574 00007f8fe9a5c6b4 00007f8fe9a5c649 00007f8fe34975a1 00007f8fe352d52d 00007f8fe84e1f49 00007f8fe1a6ed34 00007f8fe1a6ecc2 00007f8fe1a6ebf1 00007f8fe1a6eb6c 00007f8fe1a6de22 00007f8fe186d34c 00007f8fe186c786 00007f8fe186c12f
0129 02:59:34.183522 (+ 161us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1036740992}
I20260129 02:59:34.253296 20439 rpcz_store.cc:276] 0129 02:59:30.676823 (+ 0us) service_pool.cc:168] Inserting onto call queue
0129 02:59:34.133773 (+3456950us) service_pool.cc:225] Handling call
0129 02:59:34.147269 (+ 13496us) raft_consensus.cc:1471] Updating replica for 0 ops
0129 02:59:34.228528 (+ 81259us) raft_consensus.cc:1697] Filling consensus response to leader.
0129 02:59:34.228781 (+ 253us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20260129 02:59:34.212459 20553 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.211.1:42027, user_credentials={real_user=slave}} blocked reactor thread for 56865.6us
I20260129 02:59:34.223836 20441 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55676 (request call id 1083) took 3328 ms. Trace:
I20260129 02:59:34.186635 22243 consensus_queue.cc:1048] T fcddab6bca684adf8ea032289a00f90e P 1c50de89a5a845668696dcb28c3badd6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.050s
I20260129 02:59:34.223505 20436 raft_consensus.cc:1240] T cb4727386a3e48aca5d71cc6c611c197 P 197b880fa39d432a9b472845876bfaf0 [term 2 FOLLOWER]: Rejecting Update request from peer 974b802141bd4f13beffef2aa03f88f9 for earlier term 1. Current term is 2. Ops: []
I20260129 02:59:34.238176 22137 leader_election.cc:290] T febffe4a8b394368b69b1d164308fb9a P 974b802141bd4f13beffef2aa03f88f9 [CANDIDATE]: Term 2 election: Requested vote from peers 197b880fa39d432a9b472845876bfaf0 (127.18.211.1:42027), 7a9daf5d42a84615a6383cfec9a10608 (127.18.211.4:40439)
I20260129 02:59:34.212713 22212 raft_consensus.cc:515] T febffe4a8b394368b69b1d164308fb9a P 7a9daf5d42a84615a6383cfec9a10608 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } }
I20260129 02:59:34.212757 22213 raft_consensus.cc:2749] T fd8311b779af4e1fa1402b2a02a602f2 P 1c50de89a5a845668696dcb28c3badd6 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260129 02:59:34.241868 22222 raft_consensus.cc:2749] T cb0b3f259ebc4310927338a8740e87b8 P 7a9daf5d42a84615a6383cfec9a10608 [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260129 02:59:34.242851 20335 catalog_manager.cc:5654] T e33a417cf9c54cf5af73a4ee3aac4d3a P 1c50de89a5a845668696dcb28c3badd6 reported cstate change: term changed from 1 to 3. New cstate: current_term: 3 leader_uuid: "1c50de89a5a845668696dcb28c3badd6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } health_report { overall_health: HEALTHY } } }
I20260129 02:59:34.244364 20437 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55672 (request call id 1394) took 3596 ms. Trace:
I20260129 02:59:34.245343 22255 raft_consensus.cc:697] T 75f45538dc704d9a8ec4ff8ad9f3ea5b P 7a9daf5d42a84615a6383cfec9a10608 [term 2 LEADER]: Becoming Leader. State: Replica: 7a9daf5d42a84615a6383cfec9a10608, State: Running, Role: LEADER
I20260129 02:59:34.255925 20587 raft_consensus.cc:2468] T 96609ac68bdf4993a12e1c23355ca013 P 1c50de89a5a845668696dcb28c3badd6 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7a9daf5d42a84615a6383cfec9a10608 in term 4.
I20260129 02:59:34.256112 20441 rpcz_store.cc:276] 0129 02:59:30.895845 (+ 0us) service_pool.cc:168] Inserting onto call queue
0129 02:59:34.173294 (+3277449us) service_pool.cc:225] Handling call
0129 02:59:34.222423 (+ 49129us) raft_consensus.cc:1471] Updating replica for 0 ops
0129 02:59:34.223389 (+ 966us) raft_consensus.cc:1534] Early marking committed up to index 0
0129 02:59:34.223434 (+ 45us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0129 02:59:34.223495 (+ 61us) raft_consensus.cc:1645] Marking committed up to 4
0129 02:59:34.223537 (+ 42us) raft_consensus.cc:1697] Filling consensus response to leader.
0129 02:59:34.223622 (+ 85us) raft_consensus.cc:1691] UpdateReplicas() finished
0129 02:59:34.223777 (+ 155us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":343,"raft.run_cpu_time_us":118,"raft.run_wall_time_us":120}
I20260129 02:59:34.272647 20432 raft_consensus.cc:1240] T 36ff33be666a4c82954731691dbcf327 P 197b880fa39d432a9b472845876bfaf0 [term 2 FOLLOWER]: Rejecting Update request from peer 974b802141bd4f13beffef2aa03f88f9 for earlier term 1. Current term is 2. Ops: []
I20260129 02:59:34.255163 20553 leader_election.cc:304] T 17d6431cc77a491e8464d68db4081a51 P 1c50de89a5a845668696dcb28c3badd6 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 197b880fa39d432a9b472845876bfaf0, 1c50de89a5a845668696dcb28c3badd6; no voters: 974b802141bd4f13beffef2aa03f88f9
I20260129 02:59:34.273991 20433 raft_consensus.cc:1240] T 2ac659d732314e1288849f08fe5e074e P 197b880fa39d432a9b472845876bfaf0 [term 2 FOLLOWER]: Rejecting Update request from peer 974b802141bd4f13beffef2aa03f88f9 for earlier term 1. Current term is 2. Ops: []
I20260129 02:59:34.285447 20437 rpcz_store.cc:276] 0129 02:59:30.648257 (+ 0us) service_pool.cc:168] Inserting onto call queue
0129 02:59:34.084287 (+3436030us) service_pool.cc:225] Handling call
0129 02:59:34.243397 (+159110us) spinlock_profiling.cc:243] Waited 5.17 ms on lock 0x7b5800000f98. stack: 00007f8fe044e373 00007f8fe044e270 00007f8fdfda0574 00007f8fe9a5c6b4 00007f8fe9a5c649 00007f8fe02c08b1 00007f8fe0442db1 00007f8fe0442c62 00007f8fe92eddd9 00007f8fe846d260 00007f8fe8535dbb 00007f8fe8535e41 00007f8fe84f7af0 00007f8fe84e2459 00007f8fe84e1f02 00007f8fe1a6ed34
0129 02:59:34.243647 (+ 250us) raft_consensus.cc:1471] Updating replica for 0 ops
0129 02:59:34.243899 (+ 252us) raft_consensus.cc:1534] Early marking committed up to index 0
0129 02:59:34.243934 (+ 35us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0129 02:59:34.244023 (+ 89us) raft_consensus.cc:1645] Marking committed up to 1
0129 02:59:34.244066 (+ 43us) raft_consensus.cc:1697] Filling consensus response to leader.
0129 02:59:34.244145 (+ 79us) raft_consensus.cc:1691] UpdateReplicas() finished
0129 02:59:34.244288 (+ 143us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":11887360}
I20260129 02:59:34.292565 20432 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55672 (request call id 1403) took 2726 ms. Trace:
I20260129 02:59:34.293012 20432 rpcz_store.cc:276] 0129 02:59:31.562590 (+ 0us) service_pool.cc:168] Inserting onto call queue
0129 02:59:34.255730 (+2693140us) service_pool.cc:225] Handling call
0129 02:59:34.272412 (+ 16682us) raft_consensus.cc:1471] Updating replica for 0 ops
0129 02:59:34.288125 (+ 15713us) raft_consensus.cc:1697] Filling consensus response to leader.
0129 02:59:34.288377 (+ 252us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260129 02:59:34.292186 20335 catalog_manager.cc:5654] T 78aa39096868415d9afc2ffcb248e198 P 1c50de89a5a845668696dcb28c3badd6 reported cstate change: term changed from 1 to 3. New cstate: current_term: 3 leader_uuid: "1c50de89a5a845668696dcb28c3badd6" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20260129 02:59:34.301640 20432 raft_consensus.cc:1240] T 368953c9102643b2a21f99af36b51905 P 197b880fa39d432a9b472845876bfaf0 [term 4 FOLLOWER]: Rejecting Update request from peer 974b802141bd4f13beffef2aa03f88f9 for earlier term 3. Current term is 4. Ops: []
I20260129 02:59:34.302112 21127 raft_consensus.cc:1240] T febffe4a8b394368b69b1d164308fb9a P 7a9daf5d42a84615a6383cfec9a10608 [term 2 FOLLOWER]: Rejecting Update request from peer 197b880fa39d432a9b472845876bfaf0 for earlier term 1. Current term is 2. Ops: []
I20260129 02:59:34.302373 20432 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55672 (request call id 1404) took 2278 ms. Trace:
I20260129 02:59:34.302738 20432 rpcz_store.cc:276] 0129 02:59:32.023852 (+ 0us) service_pool.cc:168] Inserting onto call queue
0129 02:59:34.293692 (+2269840us) service_pool.cc:225] Handling call
0129 02:59:34.301445 (+ 7753us) raft_consensus.cc:1471] Updating replica for 0 ops
0129 02:59:34.302108 (+ 663us) raft_consensus.cc:1697] Filling consensus response to leader.
0129 02:59:34.302301 (+ 193us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260129 02:59:34.304332 22255 consensus_queue.cc:237] T 75f45538dc704d9a8ec4ff8ad9f3ea5b P 7a9daf5d42a84615a6383cfec9a10608 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } }
I20260129 02:59:34.306628 22257 raft_consensus.cc:515] T fd8311b779af4e1fa1402b2a02a602f2 P 7a9daf5d42a84615a6383cfec9a10608 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } }
I20260129 02:59:34.296290 20436 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55672 (request call id 1395) took 3641 ms. Trace:
I20260129 02:59:34.290961 22091 leader_election.cc:290] T 4e420f90884f4796997b5ea41e49e411 P 197b880fa39d432a9b472845876bfaf0 [CANDIDATE]: Term 4 election: Requested vote from peers 974b802141bd4f13beffef2aa03f88f9 (127.18.211.2:34517), 7a9daf5d42a84615a6383cfec9a10608 (127.18.211.4:40439)
I20260129 02:59:34.317835 20588 raft_consensus.cc:2468] T 75b0b6e82ddc4f92b1fe1d94f9a0b73b P 1c50de89a5a845668696dcb28c3badd6 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7a9daf5d42a84615a6383cfec9a10608 in term 2.
I20260129 02:59:34.320999 22257 leader_election.cc:290] T fd8311b779af4e1fa1402b2a02a602f2 P 7a9daf5d42a84615a6383cfec9a10608 [CANDIDATE]: Term 3 election: Requested vote from peers 197b880fa39d432a9b472845876bfaf0 (127.18.211.1:42027), 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3:37735)
I20260129 02:59:34.323248 21100 leader_election.cc:304] T 75b0b6e82ddc4f92b1fe1d94f9a0b73b P 7a9daf5d42a84615a6383cfec9a10608 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1c50de89a5a845668696dcb28c3badd6, 7a9daf5d42a84615a6383cfec9a10608; no voters:
I20260129 02:59:34.323937 20587 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fd8311b779af4e1fa1402b2a02a602f2" candidate_uuid: "7a9daf5d42a84615a6383cfec9a10608" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "1c50de89a5a845668696dcb28c3badd6"
I20260129 02:59:34.324381 22257 raft_consensus.cc:2804] T 75b0b6e82ddc4f92b1fe1d94f9a0b73b P 7a9daf5d42a84615a6383cfec9a10608 [term 2 FOLLOWER]: Leader election won for term 2
I20260129 02:59:34.324879 22257 raft_consensus.cc:697] T 75b0b6e82ddc4f92b1fe1d94f9a0b73b P 7a9daf5d42a84615a6383cfec9a10608 [term 2 LEADER]: Becoming Leader. State: Replica: 7a9daf5d42a84615a6383cfec9a10608, State: Running, Role: LEADER
I20260129 02:59:34.325874 22257 consensus_queue.cc:237] T 75b0b6e82ddc4f92b1fe1d94f9a0b73b P 7a9daf5d42a84615a6383cfec9a10608 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } }
I20260129 02:59:34.299099 22137 raft_consensus.cc:2804] T a5a05fbcbbcf4916bbc01fd1a5b5e90d P 974b802141bd4f13beffef2aa03f88f9 [term 5 FOLLOWER]: Leader election won for term 5
I20260129 02:59:34.310957 22157 raft_consensus.cc:515] T 4b1876f9a70b49edb3aa74105e66d3b3 P 974b802141bd4f13beffef2aa03f88f9 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:59:34.312026 20434 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55672 (request call id 1393) took 3671 ms. Trace:
I20260129 02:59:34.312954 19276 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260129 02:59:34.313021 22212 leader_election.cc:290] T febffe4a8b394368b69b1d164308fb9a P 7a9daf5d42a84615a6383cfec9a10608 [CANDIDATE]: Term 2 election: Requested vote from peers 197b880fa39d432a9b472845876bfaf0 (127.18.211.1:42027), 974b802141bd4f13beffef2aa03f88f9 (127.18.211.2:34517)
I20260129 02:59:34.314090 20441 raft_consensus.cc:1240] T a5a05fbcbbcf4916bbc01fd1a5b5e90d P 197b880fa39d432a9b472845876bfaf0 [term 5 FOLLOWER]: Rejecting Update request from peer 1c50de89a5a845668696dcb28c3badd6 for earlier term 4. Current term is 5. Ops: []
I20260129 02:59:34.362479 20441 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55676 (request call id 1094) took 2678 ms. Trace:
I20260129 02:59:34.362922 20441 rpcz_store.cc:276] 0129 02:59:31.683924 (+ 0us) service_pool.cc:168] Inserting onto call queue
0129 02:59:34.287734 (+2603810us) service_pool.cc:225] Handling call
0129 02:59:34.313894 (+ 26160us) raft_consensus.cc:1471] Updating replica for 0 ops
0129 02:59:34.362187 (+ 48293us) raft_consensus.cc:1697] Filling consensus response to leader.
0129 02:59:34.362417 (+ 230us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20260129 02:59:34.315826 20553 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.211.1:42027, user_credentials={real_user=slave}} blocked reactor thread for 60727.6us
I20260129 02:59:34.316745 22213 raft_consensus.cc:2804] T 17d6431cc77a491e8464d68db4081a51 P 1c50de89a5a845668696dcb28c3badd6 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260129 02:59:34.301795 22237 leader_election.cc:290] T 2ac659d732314e1288849f08fe5e074e P 197b880fa39d432a9b472845876bfaf0 [CANDIDATE]: Term 2 election: Requested vote from peers 974b802141bd4f13beffef2aa03f88f9 (127.18.211.2:34517), 7a9daf5d42a84615a6383cfec9a10608 (127.18.211.4:40439)
I20260129 02:59:34.317494 20436 rpcz_store.cc:276] 0129 02:59:30.651231 (+ 0us) service_pool.cc:168] Inserting onto call queue
0129 02:59:34.131820 (+3480589us) service_pool.cc:225] Handling call
0129 02:59:34.183926 (+ 52106us) raft_consensus.cc:1471] Updating replica for 0 ops
0129 02:59:34.280221 (+ 96295us) raft_consensus.cc:1697] Filling consensus response to leader.
0129 02:59:34.292529 (+ 12308us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260129 02:59:34.330279 20433 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55672 (request call id 1391) took 3700 ms. Trace:
I20260129 02:59:34.309046 22190 leader_election.cc:290] T a8479d1fe0bd41e5946186e3a6ca8c7a P 197b880fa39d432a9b472845876bfaf0 [CANDIDATE]: Term 3 election: Requested vote from peers 974b802141bd4f13beffef2aa03f88f9 (127.18.211.2:34517), 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3:37735)
I20260129 02:59:34.336283 20432 raft_consensus.cc:1240] T 724a236524cb4e7ba3428880681d6cc2 P 197b880fa39d432a9b472845876bfaf0 [term 2 FOLLOWER]: Rejecting Update request from peer 974b802141bd4f13beffef2aa03f88f9 for earlier term 1. Current term is 2. Ops: []
I20260129 02:59:34.370747 20432 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55672 (request call id 1405) took 2332 ms. Trace:
I20260129 02:59:34.285683 22036 raft_consensus.cc:515] T a609ac02fae84725ab743f95f51d33ce P 974b802141bd4f13beffef2aa03f88f9 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } }
W20260129 02:59:34.340004 21099 leader_election.cc:336] T 96609ac68bdf4993a12e1c23355ca013 P 7a9daf5d42a84615a6383cfec9a10608 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 197b880fa39d432a9b472845876bfaf0 (127.18.211.1:42027): Timed out: RequestConsensusVote RPC to 127.18.211.1:42027 timed out after 2.178s (SENT)
I20260129 02:59:34.340672 22160 leader_election.cc:290] T e7d83d4ccf684e76acb29a0dc844e1b8 P 974b802141bd4f13beffef2aa03f88f9 [CANDIDATE]: Term 3 election: Requested vote from peers 197b880fa39d432a9b472845876bfaf0 (127.18.211.1:42027), 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3:37735)
I20260129 02:59:34.342141 22197 leader_election.cc:290] T e33a417cf9c54cf5af73a4ee3aac4d3a P 974b802141bd4f13beffef2aa03f88f9 [CANDIDATE]: Term 4 election: Requested vote from peers 197b880fa39d432a9b472845876bfaf0 (127.18.211.1:42027), 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3:37735)
I20260129 02:59:34.368060 20433 rpcz_store.cc:276] 0129 02:59:30.629955 (+ 0us) service_pool.cc:168] Inserting onto call queue
0129 02:59:33.991919 (+3361964us) service_pool.cc:225] Handling call
0129 02:59:34.067371 (+ 75452us) raft_consensus.cc:1471] Updating replica for 0 ops
0129 02:59:34.289122 (+221751us) raft_consensus.cc:1697] Filling consensus response to leader.
0129 02:59:34.329983 (+ 40861us) spinlock_profiling.cc:243] Waited 206 ms on lock 0x7b54002ae754. stack: 00007f8fe044e373 00007f8fe044e270 00007f8fdfda0574 00007f8fe9a5c6b4 00007f8fe9a5c649 00007f8fe34975a1 00007f8fe352ea81 00007f8fe352d4be 00007f8fe84e1f49 00007f8fe1a6ed34 00007f8fe1a6ecc2 00007f8fe1a6ebf1 00007f8fe1a6eb6c 00007f8fe1a6de22 00007f8fe186d34c 00007f8fe186c786
0129 02:59:34.330204 (+ 221us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":474733952}
I20260129 02:59:34.361999 22137 raft_consensus.cc:697] T a5a05fbcbbcf4916bbc01fd1a5b5e90d P 974b802141bd4f13beffef2aa03f88f9 [term 5 LEADER]: Becoming Leader. State: Replica: 974b802141bd4f13beffef2aa03f88f9, State: Running, Role: LEADER
I20260129 02:59:34.371346 20516 raft_consensus.cc:1240] T 4b1876f9a70b49edb3aa74105e66d3b3 P 974b802141bd4f13beffef2aa03f88f9 [term 4 FOLLOWER]: Rejecting Update request from peer 1c50de89a5a845668696dcb28c3badd6 for earlier term 3. Current term is 4. Ops: []
I20260129 02:59:34.366554 22213 raft_consensus.cc:493] T 17d6431cc77a491e8464d68db4081a51 P 1c50de89a5a845668696dcb28c3badd6 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 974b802141bd4f13beffef2aa03f88f9)
I20260129 02:59:34.371114 20432 rpcz_store.cc:276] 0129 02:59:32.038414 (+ 0us) service_pool.cc:168] Inserting onto call queue
0129 02:59:34.303467 (+2265053us) service_pool.cc:225] Handling call
0129 02:59:34.303803 (+ 336us) raft_consensus.cc:1471] Updating replica for 0 ops
0129 02:59:34.370483 (+ 66680us) raft_consensus.cc:1697] Filling consensus response to leader.
0129 02:59:34.370695 (+ 212us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260129 02:59:34.373394 20441 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:48356 (request call id 215) took 2131 ms. Trace:
I20260129 02:59:34.378602 20441 rpcz_store.cc:276] 0129 02:59:32.241749 (+ 0us) service_pool.cc:168] Inserting onto call queue
0129 02:59:34.363749 (+2122000us) service_pool.cc:225] Handling call
0129 02:59:34.367195 (+ 3446us) raft_consensus.cc:1471] Updating replica for 0 ops
0129 02:59:34.370239 (+ 3044us) raft_consensus.cc:1534] Early marking committed up to index 0
0129 02:59:34.370282 (+ 43us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0129 02:59:34.373013 (+ 2731us) raft_consensus.cc:1645] Marking committed up to 5
0129 02:59:34.373070 (+ 57us) raft_consensus.cc:1697] Filling consensus response to leader.
0129 02:59:34.373159 (+ 89us) raft_consensus.cc:1691] UpdateReplicas() finished
0129 02:59:34.373323 (+ 164us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20260129 02:59:34.367714 20480 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.211.3:37735, user_credentials={real_user=slave}} blocked reactor thread for 336283us
W20260129 02:59:34.381150 20407 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.211.2:34517, user_credentials={real_user=slave}} blocked reactor thread for 338895us
I20260129 02:59:34.381250 20553 leader_election.cc:304] T 18f30ebaa4174fd28c958e72d0cc7e24 P 1c50de89a5a845668696dcb28c3badd6 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 197b880fa39d432a9b472845876bfaf0, 1c50de89a5a845668696dcb28c3badd6; no voters: 7a9daf5d42a84615a6383cfec9a10608
W20260129 02:59:34.387013 21099 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.211.1:42027, user_credentials={real_user=slave}} blocked reactor thread for 47045.6us
I20260129 02:59:34.393944 20432 raft_consensus.cc:1240] T e4d18c35b0a7493e85d0a4b92956a48f P 197b880fa39d432a9b472845876bfaf0 [term 3 FOLLOWER]: Rejecting Update request from peer 974b802141bd4f13beffef2aa03f88f9 for earlier term 2. Current term is 3. Ops: []
I20260129 02:59:34.380177 22213 raft_consensus.cc:3060] T 17d6431cc77a491e8464d68db4081a51 P 1c50de89a5a845668696dcb28c3badd6 [term 1 FOLLOWER]: Advancing to term 2
I20260129 02:59:34.343689 20434 rpcz_store.cc:276] 0129 02:59:30.641194 (+ 0us) service_pool.cc:168] Inserting onto call queue
0129 02:59:34.012842 (+3371648us) service_pool.cc:225] Handling call
0129 02:59:34.146127 (+133285us) raft_consensus.cc:1471] Updating replica for 0 ops
0129 02:59:34.229296 (+ 83169us) raft_consensus.cc:1697] Filling consensus response to leader.
0129 02:59:34.311628 (+ 82332us) spinlock_profiling.cc:243] Waited 27.1 ms on lock 0x7b54002af654. stack: 00007f8fe044e373 00007f8fe044e270 00007f8fdfda0574 00007f8fe9a5c6b4 00007f8fe9a5c649 00007f8fe34975a1 00007f8fe352ea81 00007f8fe352d4be 00007f8fe84e1f49 00007f8fe1a6ed34 00007f8fe1a6ecc2 00007f8fe1a6ebf1 00007f8fe1a6eb6c 00007f8fe1a6de22 00007f8fe186d34c 00007f8fe186c786
0129 02:59:34.311904 (+ 276us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":62273408}
I20260129 02:59:34.432271 20480 leader_election.cc:304] T 7d9494f5584d4d5db8b9adf1c8fc888b P 974b802141bd4f13beffef2aa03f88f9 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 974b802141bd4f13beffef2aa03f88f9; no voters: 197b880fa39d432a9b472845876bfaf0, 1c50de89a5a845668696dcb28c3badd6
I20260129 02:59:34.431221 22243 raft_consensus.cc:2804] T 18f30ebaa4174fd28c958e72d0cc7e24 P 1c50de89a5a845668696dcb28c3badd6 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260129 02:59:34.444649 20407 leader_election.cc:304] T 87d5fe48a44a408aaf03e894a67ecb2f P 197b880fa39d432a9b472845876bfaf0 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 197b880fa39d432a9b472845876bfaf0; no voters: 7a9daf5d42a84615a6383cfec9a10608, 974b802141bd4f13beffef2aa03f88f9
I20260129 02:59:34.448480 22243 raft_consensus.cc:493] T 18f30ebaa4174fd28c958e72d0cc7e24 P 1c50de89a5a845668696dcb28c3badd6 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 7a9daf5d42a84615a6383cfec9a10608)
I20260129 02:59:34.436657 20436 raft_consensus.cc:1240] T a8479d1fe0bd41e5946186e3a6ca8c7a P 197b880fa39d432a9b472845876bfaf0 [term 3 FOLLOWER]: Rejecting Update request from peer 974b802141bd4f13beffef2aa03f88f9 for earlier term 2. Current term is 3. Ops: []
W20260129 02:59:34.453210 20479 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.211.1:42027, user_credentials={real_user=slave}} blocked reactor thread for 342416us
I20260129 02:59:34.442847 22197 raft_consensus.cc:2749] T 7d9494f5584d4d5db8b9adf1c8fc888b P 974b802141bd4f13beffef2aa03f88f9 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20260129 02:59:34.473996 20480 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.211.3:37735, user_credentials={real_user=slave}} blocked reactor thread for 85029.5us
W20260129 02:59:34.475724 20407 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.211.2:34517, user_credentials={real_user=slave}} blocked reactor thread for 44406.2us
W20260129 02:59:34.477614 20553 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.211.1:42027, user_credentials={real_user=slave}} blocked reactor thread for 96456us
I20260129 02:59:34.478489 20553 leader_election.cc:304] T 1299723a4476420ab02fd8bfd7aa0645 P 1c50de89a5a845668696dcb28c3badd6 [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1c50de89a5a845668696dcb28c3badd6; no voters: 197b880fa39d432a9b472845876bfaf0, 7a9daf5d42a84615a6383cfec9a10608
I20260129 02:59:34.476876 22237 raft_consensus.cc:2749] T 87d5fe48a44a408aaf03e894a67ecb2f P 197b880fa39d432a9b472845876bfaf0 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260129 02:59:34.483158 20553 leader_election.cc:304] T 0f20af7fbe5d4db89a6f17948d98912c P 1c50de89a5a845668696dcb28c3badd6 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 197b880fa39d432a9b472845876bfaf0, 1c50de89a5a845668696dcb28c3badd6; no voters:
I20260129 02:59:34.484133 20553 leader_election.cc:304] T 4f8f3a38f6e24ba4857875181f447442 P 1c50de89a5a845668696dcb28c3badd6 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 197b880fa39d432a9b472845876bfaf0, 1c50de89a5a845668696dcb28c3badd6; no voters:
I20260129 02:59:34.484958 20553 leader_election.cc:304] T 724a236524cb4e7ba3428880681d6cc2 P 1c50de89a5a845668696dcb28c3badd6 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 197b880fa39d432a9b472845876bfaf0, 1c50de89a5a845668696dcb28c3badd6; no voters:
I20260129 02:59:34.452378 22243 raft_consensus.cc:3060] T 18f30ebaa4174fd28c958e72d0cc7e24 P 1c50de89a5a845668696dcb28c3badd6 [term 2 FOLLOWER]: Advancing to term 3
I20260129 02:59:34.488139 20436 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55672 (request call id 1416) took 1047 ms. Trace:
I20260129 02:59:34.477236 22184 raft_consensus.cc:493] T 75be4bd6bd45474795fd472eff2953ac P 974b802141bd4f13beffef2aa03f88f9 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260129 02:59:34.477301 22036 leader_election.cc:290] T a609ac02fae84725ab743f95f51d33ce P 974b802141bd4f13beffef2aa03f88f9 [CANDIDATE]: Term 5 election: Requested vote from peers 197b880fa39d432a9b472845876bfaf0 (127.18.211.1:42027), 7a9daf5d42a84615a6383cfec9a10608 (127.18.211.4:40439)
I20260129 02:59:34.502915 20480 leader_election.cc:304] T a4c1226b987b468fa84dfc6ba0e2dbe1 P 974b802141bd4f13beffef2aa03f88f9 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 974b802141bd4f13beffef2aa03f88f9; no voters: 197b880fa39d432a9b472845876bfaf0, 1c50de89a5a845668696dcb28c3badd6
I20260129 02:59:34.488399 22079 raft_consensus.cc:2749] T 1299723a4476420ab02fd8bfd7aa0645 P 1c50de89a5a845668696dcb28c3badd6 [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20260129 02:59:34.492251 22061 raft_consensus.cc:2804] T 0f20af7fbe5d4db89a6f17948d98912c P 1c50de89a5a845668696dcb28c3badd6 [term 4 FOLLOWER]: Leader election won for term 4
I20260129 02:59:34.488780 22233 consensus_queue.cc:1059] T 00e79d8f25a34676b1d866eb2068a2b2 P 197b880fa39d432a9b472845876bfaf0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 }, Status: INVALID_TERM, Last received: 3.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.018s
I20260129 02:59:34.501545 22157 leader_election.cc:290] T 4b1876f9a70b49edb3aa74105e66d3b3 P 974b802141bd4f13beffef2aa03f88f9 [CANDIDATE]: Term 4 election: Requested vote from peers 197b880fa39d432a9b472845876bfaf0 (127.18.211.1:42027), 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3:37735)
I20260129 02:59:34.511256 20407 leader_election.cc:304] T fcddab6bca684adf8ea032289a00f90e P 197b880fa39d432a9b472845876bfaf0 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 197b880fa39d432a9b472845876bfaf0, 974b802141bd4f13beffef2aa03f88f9; no voters:
W20260129 02:59:34.512253 20556 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.211.2:34517, user_credentials={real_user=slave}} blocked reactor thread for 337439us
I20260129 02:59:34.498692 20436 rpcz_store.cc:276] 0129 02:59:33.438843 (+ 0us) service_pool.cc:168] Inserting onto call queue
0129 02:59:34.380444 (+941601us) service_pool.cc:225] Handling call
0129 02:59:34.429551 (+ 49107us) raft_consensus.cc:1471] Updating replica for 0 ops
0129 02:59:34.459187 (+ 29636us) raft_consensus.cc:1697] Filling consensus response to leader.
0129 02:59:34.486278 (+ 27091us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260129 02:59:34.508783 22164 raft_consensus.cc:2804] T 4f8f3a38f6e24ba4857875181f447442 P 1c50de89a5a845668696dcb28c3badd6 [term 4 FOLLOWER]: Leader election won for term 4
W20260129 02:59:34.526227 20405 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.211.3:37735, user_credentials={real_user=slave}} blocked reactor thread for 356882us
I20260129 02:59:34.515028 22079 raft_consensus.cc:2804] T 724a236524cb4e7ba3428880681d6cc2 P 1c50de89a5a845668696dcb28c3badd6 [term 2 FOLLOWER]: Leader election won for term 2
I20260129 02:59:34.529850 20407 leader_election.cc:304] T e33a417cf9c54cf5af73a4ee3aac4d3a P 197b880fa39d432a9b472845876bfaf0 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 197b880fa39d432a9b472845876bfaf0, 974b802141bd4f13beffef2aa03f88f9; no voters:
I20260129 02:59:34.488356 22137 consensus_queue.cc:237] T a5a05fbcbbcf4916bbc01fd1a5b5e90d P 974b802141bd4f13beffef2aa03f88f9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 4.3, Last appended by leader: 3, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:59:34.511986 22061 raft_consensus.cc:697] T 0f20af7fbe5d4db89a6f17948d98912c P 1c50de89a5a845668696dcb28c3badd6 [term 4 LEADER]: Becoming Leader. State: Replica: 1c50de89a5a845668696dcb28c3badd6, State: Running, Role: LEADER
I20260129 02:59:34.523128 22103 raft_consensus.cc:697] T 368953c9102643b2a21f99af36b51905 P 1c50de89a5a845668696dcb28c3badd6 [term 4 LEADER]: Becoming Leader. State: Replica: 1c50de89a5a845668696dcb28c3badd6, State: Running, Role: LEADER
I20260129 02:59:34.527076 22036 raft_consensus.cc:2749] T a4c1226b987b468fa84dfc6ba0e2dbe1 P 974b802141bd4f13beffef2aa03f88f9 [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20260129 02:59:34.566021 21101 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260129 02:59:34.551710 22112 consensus_queue.cc:1059] T 4f8f3a38f6e24ba4857875181f447442 P 974b802141bd4f13beffef2aa03f88f9 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 }, Status: INVALID_TERM, Last received: 3.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260129 02:59:34.553021 22091 raft_consensus.cc:3055] T 00e79d8f25a34676b1d866eb2068a2b2 P 197b880fa39d432a9b472845876bfaf0 [term 3 LEADER]: Stepping down as leader of term 3
I20260129 02:59:34.534140 22164 raft_consensus.cc:697] T 4f8f3a38f6e24ba4857875181f447442 P 1c50de89a5a845668696dcb28c3badd6 [term 4 LEADER]: Becoming Leader. State: Replica: 1c50de89a5a845668696dcb28c3badd6, State: Running, Role: LEADER
I20260129 02:59:34.526717 22197 consensus_queue.cc:1059] T 0f20af7fbe5d4db89a6f17948d98912c P 974b802141bd4f13beffef2aa03f88f9 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 }, Status: INVALID_TERM, Last received: 3.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.029s
W20260129 02:59:34.556349 20480 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.211.3:37735, user_credentials={real_user=slave}} blocked reactor thread for 67019.8us
I20260129 02:59:34.509756 22184 raft_consensus.cc:515] T 75be4bd6bd45474795fd472eff2953ac P 974b802141bd4f13beffef2aa03f88f9 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:59:34.557861 22293 consensus_queue.cc:1059] T cb4727386a3e48aca5d71cc6c611c197 P 974b802141bd4f13beffef2aa03f88f9 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.006s
I20260129 02:59:34.530699 22190 raft_consensus.cc:2804] T fcddab6bca684adf8ea032289a00f90e P 197b880fa39d432a9b472845876bfaf0 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260129 02:59:34.557673 22160 consensus_queue.cc:1059] T 368953c9102643b2a21f99af36b51905 P 974b802141bd4f13beffef2aa03f88f9 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 }, Status: INVALID_TERM, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.024s
I20260129 02:59:34.561260 20407 leader_election.cc:304] T 0678b8b468bd43cdbc3e485900844821 P 197b880fa39d432a9b472845876bfaf0 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 197b880fa39d432a9b472845876bfaf0, 974b802141bd4f13beffef2aa03f88f9; no voters: 7a9daf5d42a84615a6383cfec9a10608
I20260129 02:59:34.514487 22237 consensus_queue.cc:1059] T a4c1226b987b468fa84dfc6ba0e2dbe1 P 197b880fa39d432a9b472845876bfaf0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.053s
I20260129 02:59:34.593663 22091 raft_consensus.cc:740] T 00e79d8f25a34676b1d866eb2068a2b2 P 197b880fa39d432a9b472845876bfaf0 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 197b880fa39d432a9b472845876bfaf0, State: Running, Role: LEADER
I20260129 02:59:34.558591 22292 consensus_queue.cc:1059] T a644c02b16af47679824ccd060ac10fd P 1c50de89a5a845668696dcb28c3badd6 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.058s
W20260129 02:59:34.620308 20482 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260129 02:59:34.625485 22103 consensus_queue.cc:237] T 368953c9102643b2a21f99af36b51905 P 1c50de89a5a845668696dcb28c3badd6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:59:34.556558 22061 consensus_queue.cc:237] T 0f20af7fbe5d4db89a6f17948d98912c P 1c50de89a5a845668696dcb28c3badd6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:59:34.620600 20436 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3ccac82a01a2445fa5b0ce6c0fcf2295" candidate_uuid: "974b802141bd4f13beffef2aa03f88f9" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "197b880fa39d432a9b472845876bfaf0"
I20260129 02:59:34.632625 20436 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a609ac02fae84725ab743f95f51d33ce" candidate_uuid: "974b802141bd4f13beffef2aa03f88f9" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "197b880fa39d432a9b472845876bfaf0"
I20260129 02:59:34.633877 20436 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e7d83d4ccf684e76acb29a0dc844e1b8" candidate_uuid: "974b802141bd4f13beffef2aa03f88f9" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "197b880fa39d432a9b472845876bfaf0"
I20260129 02:59:34.634960 20436 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4b1876f9a70b49edb3aa74105e66d3b3" candidate_uuid: "974b802141bd4f13beffef2aa03f88f9" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "197b880fa39d432a9b472845876bfaf0"
I20260129 02:59:34.635529 20436 raft_consensus.cc:3060] T 4b1876f9a70b49edb3aa74105e66d3b3 P 197b880fa39d432a9b472845876bfaf0 [term 3 FOLLOWER]: Advancing to term 4
I20260129 02:59:34.611997 22190 raft_consensus.cc:493] T fcddab6bca684adf8ea032289a00f90e P 197b880fa39d432a9b472845876bfaf0 [term 6 FOLLOWER]: Starting leader election (detected failure of leader 1c50de89a5a845668696dcb28c3badd6)
I20260129 02:59:34.572762 22157 consensus_queue.cc:1059] T 2ac659d732314e1288849f08fe5e074e P 974b802141bd4f13beffef2aa03f88f9 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.045s
W20260129 02:59:34.635565 21099 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260129 02:59:34.560042 22233 consensus_queue.cc:1059] T 3906b080a5b34a74b8b64128635e8997 P 197b880fa39d432a9b472845876bfaf0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.059s
W20260129 02:59:34.653115 21099 leader_election.cc:336] T 00e79d8f25a34676b1d866eb2068a2b2 P 7a9daf5d42a84615a6383cfec9a10608 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 197b880fa39d432a9b472845876bfaf0 (127.18.211.1:42027): Timed out: RequestConsensusVote RPC to 127.18.211.1:42027 timed out after 1.551s (SENT)
I20260129 02:59:34.640654 22190 raft_consensus.cc:3060] T fcddab6bca684adf8ea032289a00f90e P 197b880fa39d432a9b472845876bfaf0 [term 6 FOLLOWER]: Advancing to term 7
W20260129 02:59:34.693270 20479 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.211.1:42027, user_credentials={real_user=slave}} blocked reactor thread for 137861us
W20260129 02:59:34.706954 20556 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.211.2:34517, user_credentials={real_user=slave}} blocked reactor thread for 172706us
I20260129 02:59:34.712534 22120 raft_consensus.cc:2804] T 0678b8b468bd43cdbc3e485900844821 P 197b880fa39d432a9b472845876bfaf0 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260129 02:59:34.714545 22198 raft_consensus.cc:2804] T e33a417cf9c54cf5af73a4ee3aac4d3a P 197b880fa39d432a9b472845876bfaf0 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260129 02:59:34.642377 20434 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "febffe4a8b394368b69b1d164308fb9a" candidate_uuid: "974b802141bd4f13beffef2aa03f88f9" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "197b880fa39d432a9b472845876bfaf0"
I20260129 02:59:34.717108 20441 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55676 (request call id 1105) took 1058 ms. Trace:
W20260129 02:59:34.650748 22224 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.504s user 0.012s sys 0.026s
I20260129 02:59:34.643950 22247 raft_consensus.cc:3055] T a644c02b16af47679824ccd060ac10fd P 1c50de89a5a845668696dcb28c3badd6 [term 1 LEADER]: Stepping down as leader of term 1
I20260129 02:59:34.719156 20480 leader_election.cc:304] T 913fc7d751ed4b7388d889aac30a5f9b P 974b802141bd4f13beffef2aa03f88f9 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 974b802141bd4f13beffef2aa03f88f9; no voters: 197b880fa39d432a9b472845876bfaf0, 1c50de89a5a845668696dcb28c3badd6
I20260129 02:59:34.719102 20441 rpcz_store.cc:276] 0129 02:59:33.656659 (+ 0us) service_pool.cc:168] Inserting onto call queue
0129 02:59:34.388470 (+731811us) service_pool.cc:225] Handling call
0129 02:59:34.393506 (+ 5036us) raft_consensus.cc:1471] Updating replica for 0 ops
0129 02:59:34.430047 (+ 36541us) raft_consensus.cc:1534] Early marking committed up to index 2
0129 02:59:34.700928 (+270881us) log.cc:844] Serialized 14 byte log entry
0129 02:59:34.713952 (+ 13024us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0129 02:59:34.714202 (+ 250us) raft_consensus.cc:1645] Marking committed up to 2
0129 02:59:34.714292 (+ 90us) raft_consensus.cc:1697] Filling consensus response to leader.
0129 02:59:34.715190 (+ 898us) raft_consensus.cc:1691] UpdateReplicas() finished
0129 02:59:34.715411 (+ 221us) inbound_call.cc:173] Queueing success response
Metrics: {"thread_start_us":269993,"threads_started":2}
I20260129 02:59:34.714617 20435 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:48356 (request call id 212) took 4064 ms. Trace:
I20260129 02:59:34.715565 20556 leader_election.cc:304] T 913fc7d751ed4b7388d889aac30a5f9b P 1c50de89a5a845668696dcb28c3badd6 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1c50de89a5a845668696dcb28c3badd6, 974b802141bd4f13beffef2aa03f88f9; no voters: 197b880fa39d432a9b472845876bfaf0
I20260129 02:59:34.730420 20435 rpcz_store.cc:276] 0129 02:59:30.650197 (+ 0us) service_pool.cc:168] Inserting onto call queue
0129 02:59:34.117222 (+3467025us) service_pool.cc:225] Handling call
0129 02:59:34.212556 (+ 95334us) spinlock_profiling.cc:243] Waited 979 us on lock 0x7b5800000f98. stack: 00007f8fe044e373 00007f8fe044e270 00007f8fdfda0574 00007f8fe9a5c6b4 00007f8fe9a5c649 00007f8fe02c08b1 00007f8fe0442db1 00007f8fe0442c62 00007f8fe92eddd9 00007f8fe846d260 00007f8fe8535dbb 00007f8fe8535e41 00007f8fe84f7af0 00007f8fe84e2459 00007f8fe84e1f02 00007f8fe1a6ed34
0129 02:59:34.212782 (+ 226us) raft_consensus.cc:1471] Updating replica for 0 ops
0129 02:59:34.213180 (+ 398us) raft_consensus.cc:1534] Early marking committed up to index 6
0129 02:59:34.531440 (+318260us) log.cc:844] Serialized 14 byte log entry
0129 02:59:34.710296 (+178856us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0129 02:59:34.710885 (+ 589us) raft_consensus.cc:1645] Marking committed up to 6
0129 02:59:34.711682 (+ 797us) raft_consensus.cc:1697] Filling consensus response to leader.
0129 02:59:34.713019 (+ 1337us) raft_consensus.cc:1691] UpdateReplicas() finished
0129 02:59:34.714124 (+ 1105us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2252544,"thread_start_us":475324,"threads_started":2}
I20260129 02:59:34.642509 22036 consensus_queue.cc:1059] T 724a236524cb4e7ba3428880681d6cc2 P 974b802141bd4f13beffef2aa03f88f9 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.086s
I20260129 02:59:34.689333 22231 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.ChangeConfig from 127.0.0.1:55666 (request call id 982) took 11729 ms. Trace:
I20260129 02:59:34.743120 20556 leader_election.cc:304] T 1ce4ddf5b10e4ae8b193b13b2e5936ee P 1c50de89a5a845668696dcb28c3badd6 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1c50de89a5a845668696dcb28c3badd6, 974b802141bd4f13beffef2aa03f88f9; no voters:
I20260129 02:59:34.728560 22247 raft_consensus.cc:740] T a644c02b16af47679824ccd060ac10fd P 1c50de89a5a845668696dcb28c3badd6 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 1c50de89a5a845668696dcb28c3badd6, State: Running, Role: LEADER
W20260129 02:59:34.735623 22224 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.589s user 0.018s sys 0.027s
I20260129 02:59:34.745874 20441 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e33a417cf9c54cf5af73a4ee3aac4d3a" candidate_uuid: "974b802141bd4f13beffef2aa03f88f9" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "197b880fa39d432a9b472845876bfaf0"
I20260129 02:59:34.735714 22112 raft_consensus.cc:3055] T 0f20af7fbe5d4db89a6f17948d98912c P 974b802141bd4f13beffef2aa03f88f9 [term 3 LEADER]: Stepping down as leader of term 3
I20260129 02:59:34.749720 20556 leader_election.cc:304] T cb0b3f259ebc4310927338a8740e87b8 P 1c50de89a5a845668696dcb28c3badd6 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1c50de89a5a845668696dcb28c3badd6; no voters: 7a9daf5d42a84615a6383cfec9a10608, 974b802141bd4f13beffef2aa03f88f9
W20260129 02:59:34.764801 20480 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.211.3:37735, user_credentials={real_user=slave}} blocked reactor thread for 45744.7us
I20260129 02:59:34.748098 22224 raft_consensus.cc:697] T 4e3805ffa70242e8ade66da260dddf66 P 1c50de89a5a845668696dcb28c3badd6 [term 3 LEADER]: Becoming Leader. State: Replica: 1c50de89a5a845668696dcb28c3badd6, State: Running, Role: LEADER
I20260129 02:59:34.772023 20556 leader_election.cc:304] T 4f5a77e94fef4b16a2ba7b067c8b144a P 1c50de89a5a845668696dcb28c3badd6 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1c50de89a5a845668696dcb28c3badd6; no voters: 7a9daf5d42a84615a6383cfec9a10608, 974b802141bd4f13beffef2aa03f88f9
W20260129 02:59:34.771260 20553 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.211.1:42027, user_credentials={real_user=slave}} blocked reactor thread for 285125us
I20260129 02:59:34.731367 22079 raft_consensus.cc:697] T 724a236524cb4e7ba3428880681d6cc2 P 1c50de89a5a845668696dcb28c3badd6 [term 2 LEADER]: Becoming Leader. State: Replica: 1c50de89a5a845668696dcb28c3badd6, State: Running, Role: LEADER
I20260129 02:59:34.786983 20437 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55676 (request call id 1097) took 2945 ms. Trace:
I20260129 02:59:34.765193 22112 raft_consensus.cc:740] T 0f20af7fbe5d4db89a6f17948d98912c P 974b802141bd4f13beffef2aa03f88f9 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 974b802141bd4f13beffef2aa03f88f9, State: Running, Role: LEADER
W20260129 02:59:34.788152 20405 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.211.3:37735, user_credentials={real_user=slave}} blocked reactor thread for 150131us
I20260129 02:59:34.746531 22294 consensus_queue.cc:1059] T 1299723a4476420ab02fd8bfd7aa0645 P 197b880fa39d432a9b472845876bfaf0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 }, Status: INVALID_TERM, Last received: 0.0, Next index: 6, Last known committed idx: 4, Time since last communication: 0.031s
W20260129 02:59:34.802033 20407 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.211.2:34517, user_credentials={real_user=slave}} blocked reactor thread for 240751us
I20260129 02:59:34.729333 22164 consensus_queue.cc:237] T 4f8f3a38f6e24ba4857875181f447442 P 1c50de89a5a845668696dcb28c3badd6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:59:34.800594 20438 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55676 (request call id 1080) took 4082 ms. Trace:
I20260129 02:59:34.803887 20407 leader_election.cc:304] T 926a250fa5834e88b96ccecbefbf5086 P 197b880fa39d432a9b472845876bfaf0 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 197b880fa39d432a9b472845876bfaf0, 974b802141bd4f13beffef2aa03f88f9; no voters:
I20260129 02:59:34.809875 20438 rpcz_store.cc:276] 0129 02:59:30.717777 (+ 0us) service_pool.cc:168] Inserting onto call queue
0129 02:59:34.138201 (+3420424us) service_pool.cc:225] Handling call
0129 02:59:34.170336 (+ 32135us) raft_consensus.cc:1471] Updating replica for 0 ops
0129 02:59:34.170599 (+ 263us) raft_consensus.cc:1534] Early marking committed up to index 3
0129 02:59:34.526857 (+356258us) log.cc:844] Serialized 14 byte log entry
0129 02:59:34.800052 (+273195us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0129 02:59:34.800131 (+ 79us) raft_consensus.cc:1645] Marking committed up to 3
0129 02:59:34.800194 (+ 63us) raft_consensus.cc:1697] Filling consensus response to leader.
0129 02:59:34.800310 (+ 116us) raft_consensus.cc:1691] UpdateReplicas() finished
0129 02:59:34.800503 (+ 193us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":14815,"prepare.queue_time_us":568127,"prepare.run_cpu_time_us":42,"prepare.run_wall_time_us":45,"thread_start_us":557427,"threads_started":2}
I20260129 02:59:34.820575 20480 leader_election.cc:304] T 0340229d521b432580ae926f6062e3f7 P 974b802141bd4f13beffef2aa03f88f9 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 974b802141bd4f13beffef2aa03f88f9; no voters: 197b880fa39d432a9b472845876bfaf0, 1c50de89a5a845668696dcb28c3badd6
I20260129 02:59:34.753976 22091 consensus_queue.cc:260] T 00e79d8f25a34676b1d866eb2068a2b2 P 197b880fa39d432a9b472845876bfaf0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 3.6, Last appended by leader: 6, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } }
I20260129 02:59:34.838409 22323 raft_consensus.cc:2749] T 913fc7d751ed4b7388d889aac30a5f9b P 974b802141bd4f13beffef2aa03f88f9 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260129 02:59:34.838922 22091 raft_consensus.cc:3060] T 00e79d8f25a34676b1d866eb2068a2b2 P 197b880fa39d432a9b472845876bfaf0 [term 3 FOLLOWER]: Advancing to term 4
I20260129 02:59:34.839249 22323 consensus_queue.cc:1059] T e4d18c35b0a7493e85d0a4b92956a48f P 974b802141bd4f13beffef2aa03f88f9 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 }, Status: INVALID_TERM, Last received: 2.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260129 02:59:34.841435 22157 consensus_queue.cc:1059] T 8055232cf0004d40a85f3bf983fe00a9 P 974b802141bd4f13beffef2aa03f88f9 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260129 02:59:34.841881 22318 raft_consensus.cc:3055] T 2ac659d732314e1288849f08fe5e074e P 974b802141bd4f13beffef2aa03f88f9 [term 1 LEADER]: Stepping down as leader of term 1
I20260129 02:59:34.842293 22318 raft_consensus.cc:740] T 2ac659d732314e1288849f08fe5e074e P 974b802141bd4f13beffef2aa03f88f9 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 974b802141bd4f13beffef2aa03f88f9, State: Running, Role: LEADER
I20260129 02:59:34.843503 22224 consensus_queue.cc:237] T 4e3805ffa70242e8ade66da260dddf66 P 1c50de89a5a845668696dcb28c3badd6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } }
W20260129 02:59:34.849771 20553 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260129 02:59:34.850673 20479 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.211.1:42027, user_credentials={real_user=slave}} blocked reactor thread for 84159.5us
I20260129 02:59:34.816130 22308 raft_consensus.cc:2804] T 913fc7d751ed4b7388d889aac30a5f9b P 1c50de89a5a845668696dcb28c3badd6 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260129 02:59:34.843230 22318 consensus_queue.cc:260] T 2ac659d732314e1288849f08fe5e074e P 974b802141bd4f13beffef2aa03f88f9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } }
I20260129 02:59:34.833027 22184 leader_election.cc:290] T 75be4bd6bd45474795fd472eff2953ac P 974b802141bd4f13beffef2aa03f88f9 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 197b880fa39d432a9b472845876bfaf0 (127.18.211.1:42027), 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3:37735)
W20260129 02:59:34.903429 20405 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.211.3:37735, user_credentials={real_user=slave}} blocked reactor thread for 105489us
W20260129 02:59:34.904702 20480 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.211.3:37735, user_credentials={real_user=slave}} blocked reactor thread for 97970us
W20260129 02:59:34.906704 20556 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.211.2:34517, user_credentials={real_user=slave}} blocked reactor thread for 134949us
I20260129 02:59:34.856603 22061 raft_consensus.cc:2804] T 1ce4ddf5b10e4ae8b193b13b2e5936ee P 1c50de89a5a845668696dcb28c3badd6 [term 4 FOLLOWER]: Leader election won for term 4
I20260129 02:59:34.915261 20405 leader_election.cc:304] T dc2f4cb752cf4d59b5f6bc689f5d8104 P 197b880fa39d432a9b472845876bfaf0 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 197b880fa39d432a9b472845876bfaf0; no voters: 1c50de89a5a845668696dcb28c3badd6, 7a9daf5d42a84615a6383cfec9a10608
W20260129 02:59:34.918351 21099 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.211.1:42027, user_credentials={real_user=slave}} blocked reactor thread for 120249us
W20260129 02:59:34.919380 21099 leader_election.cc:336] T 3ccac82a01a2445fa5b0ce6c0fcf2295 P 7a9daf5d42a84615a6383cfec9a10608 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 197b880fa39d432a9b472845876bfaf0 (127.18.211.1:42027): Timed out: RequestConsensusVote RPC to 127.18.211.1:42027 timed out after 1.737s (SENT)
I20260129 02:59:34.919831 21099 leader_election.cc:304] T 3ccac82a01a2445fa5b0ce6c0fcf2295 P 7a9daf5d42a84615a6383cfec9a10608 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7a9daf5d42a84615a6383cfec9a10608; no voters: 197b880fa39d432a9b472845876bfaf0, 974b802141bd4f13beffef2aa03f88f9
I20260129 02:59:34.873637 22036 raft_consensus.cc:2749] T 0340229d521b432580ae926f6062e3f7 P 974b802141bd4f13beffef2aa03f88f9 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260129 02:59:34.874248 22308 raft_consensus.cc:493] T 913fc7d751ed4b7388d889aac30a5f9b P 1c50de89a5a845668696dcb28c3badd6 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 197b880fa39d432a9b472845876bfaf0)
I20260129 02:59:34.932614 20432 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55676 (request call id 1119) took 1181 ms. Trace:
I20260129 02:59:34.933969 20583 raft_consensus.cc:1240] T 4e3805ffa70242e8ade66da260dddf66 P 1c50de89a5a845668696dcb28c3badd6 [term 3 LEADER]: Rejecting Update request from peer 7a9daf5d42a84615a6383cfec9a10608 for earlier term 2. Current term is 3. Ops: []
I20260129 02:59:34.802862 22247 consensus_queue.cc:260] T a644c02b16af47679824ccd060ac10fd P 1c50de89a5a845668696dcb28c3badd6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } }
W20260129 02:59:34.934330 20407 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.211.2:34517, user_credentials={real_user=slave}} blocked reactor thread for 130706us
I20260129 02:59:34.934455 20556 leader_election.cc:304] T c2633a6fc34c44579de348d208b0c01e P 1c50de89a5a845668696dcb28c3badd6 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1c50de89a5a845668696dcb28c3badd6, 974b802141bd4f13beffef2aa03f88f9; no voters: 197b880fa39d432a9b472845876bfaf0
I20260129 02:59:34.935999 20432 rpcz_store.cc:276] 0129 02:59:33.750665 (+ 0us) service_pool.cc:168] Inserting onto call queue
0129 02:59:34.407636 (+656971us) service_pool.cc:225] Handling call
0129 02:59:34.434320 (+ 26684us) raft_consensus.cc:1471] Updating replica for 0 ops
0129 02:59:34.449724 (+ 15404us) raft_consensus.cc:1534] Early marking committed up to index 3
0129 02:59:34.727182 (+277458us) log.cc:844] Serialized 14 byte log entry
0129 02:59:34.930928 (+203746us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0129 02:59:34.931245 (+ 317us) raft_consensus.cc:1645] Marking committed up to 3
0129 02:59:34.931531 (+ 286us) raft_consensus.cc:1697] Filling consensus response to leader.
0129 02:59:34.931764 (+ 233us) raft_consensus.cc:1691] UpdateReplicas() finished
0129 02:59:34.932390 (+ 626us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":354403,"prepare.run_cpu_time_us":49,"prepare.run_wall_time_us":50,"thread_start_us":435719,"threads_started":2}
I20260129 02:59:34.944262 20407 leader_election.cc:304] T 4b1876f9a70b49edb3aa74105e66d3b3 P 197b880fa39d432a9b472845876bfaf0 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 197b880fa39d432a9b472845876bfaf0, 974b802141bd4f13beffef2aa03f88f9; no voters:
I20260129 02:59:34.934444 22184 raft_consensus.cc:3055] T e4d18c35b0a7493e85d0a4b92956a48f P 974b802141bd4f13beffef2aa03f88f9 [term 2 LEADER]: Stepping down as leader of term 2
W20260129 02:59:34.956414 21099 leader_election.cc:336] T 78aa39096868415d9afc2ffcb248e198 P 7a9daf5d42a84615a6383cfec9a10608 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 197b880fa39d432a9b472845876bfaf0 (127.18.211.1:42027): Timed out: RequestConsensusVote RPC to 127.18.211.1:42027 timed out after 1.738s (SENT)
I20260129 02:59:34.956890 20439 raft_consensus.cc:1240] T 42d27a28d89b409e8807ec0efd97af69 P 197b880fa39d432a9b472845876bfaf0 [term 4 LEADER]: Rejecting Update request from peer 1c50de89a5a845668696dcb28c3badd6 for earlier term 3. Current term is 4. Ops: [3.6-3.6]
I20260129 02:59:34.957883 21099 leader_election.cc:304] T 78aa39096868415d9afc2ffcb248e198 P 7a9daf5d42a84615a6383cfec9a10608 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7a9daf5d42a84615a6383cfec9a10608; no voters: 197b880fa39d432a9b472845876bfaf0, 1c50de89a5a845668696dcb28c3badd6
I20260129 02:59:34.937669 22201 raft_consensus.cc:3055] T 8055232cf0004d40a85f3bf983fe00a9 P 974b802141bd4f13beffef2aa03f88f9 [term 1 LEADER]: Stepping down as leader of term 1
I20260129 02:59:34.959146 20405 leader_election.cc:304] T 96609ac68bdf4993a12e1c23355ca013 P 197b880fa39d432a9b472845876bfaf0 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 4 responses out of 4 voters: 3 yes votes; 1 no votes. yes voters: 197b880fa39d432a9b472845876bfaf0, 1c50de89a5a845668696dcb28c3badd6, 974b802141bd4f13beffef2aa03f88f9; no voters: 7a9daf5d42a84615a6383cfec9a10608
I20260129 02:59:34.938630 22308 raft_consensus.cc:3060] T 913fc7d751ed4b7388d889aac30a5f9b P 1c50de89a5a845668696dcb28c3badd6 [term 1 FOLLOWER]: Advancing to term 2
I20260129 02:59:34.944670 22326 raft_consensus.cc:2804] T 926a250fa5834e88b96ccecbefbf5086 P 197b880fa39d432a9b472845876bfaf0 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260129 02:59:34.943308 22257 raft_consensus.cc:2749] T 3ccac82a01a2445fa5b0ce6c0fcf2295 P 7a9daf5d42a84615a6383cfec9a10608 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260129 02:59:34.967324 20438 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3906b080a5b34a74b8b64128635e8997" candidate_uuid: "1c50de89a5a845668696dcb28c3badd6" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "197b880fa39d432a9b472845876bfaf0"
I20260129 02:59:34.956597 22318 raft_consensus.cc:3060] T 2ac659d732314e1288849f08fe5e074e P 974b802141bd4f13beffef2aa03f88f9 [term 1 FOLLOWER]: Advancing to term 2
I20260129 02:59:34.974133 20480 leader_election.cc:304] T 76f713c3331540d2a5b76164103e9eb1 P 974b802141bd4f13beffef2aa03f88f9 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 974b802141bd4f13beffef2aa03f88f9; no voters: 197b880fa39d432a9b472845876bfaf0, 1c50de89a5a845668696dcb28c3badd6
I20260129 02:59:34.951712 22212 raft_consensus.cc:493] T 3e2d15f8640b407195210c476e09df91 P 7a9daf5d42a84615a6383cfec9a10608 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 1c50de89a5a845668696dcb28c3badd6)
I20260129 02:59:34.923800 22112 consensus_queue.cc:260] T 0f20af7fbe5d4db89a6f17948d98912c P 974b802141bd4f13beffef2aa03f88f9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:59:34.962958 22201 raft_consensus.cc:740] T 8055232cf0004d40a85f3bf983fe00a9 P 974b802141bd4f13beffef2aa03f88f9 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 974b802141bd4f13beffef2aa03f88f9, State: Running, Role: LEADER
I20260129 02:59:34.986577 20438 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1ce4ddf5b10e4ae8b193b13b2e5936ee" candidate_uuid: "1c50de89a5a845668696dcb28c3badd6" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "197b880fa39d432a9b472845876bfaf0"
I20260129 02:59:34.957369 22184 raft_consensus.cc:740] T e4d18c35b0a7493e85d0a4b92956a48f P 974b802141bd4f13beffef2aa03f88f9 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 974b802141bd4f13beffef2aa03f88f9, State: Running, Role: LEADER
I20260129 02:59:34.989912 20435 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a8479d1fe0bd41e5946186e3a6ca8c7a" candidate_uuid: "1c50de89a5a845668696dcb28c3badd6" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "197b880fa39d432a9b472845876bfaf0" is_pre_election: true
I20260129 02:59:34.997572 20480 leader_election.cc:304] T 7a9a69cd82724b1faed1d10aa091fc4a P 974b802141bd4f13beffef2aa03f88f9 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 974b802141bd4f13beffef2aa03f88f9; no voters: 197b880fa39d432a9b472845876bfaf0, 1c50de89a5a845668696dcb28c3badd6
I20260129 02:59:34.991422 22299 raft_consensus.cc:2749] T cb0b3f259ebc4310927338a8740e87b8 P 1c50de89a5a845668696dcb28c3badd6 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
W20260129 02:59:34.991672 22295 log.cc:927] Time spent T dc2f4cb752cf4d59b5f6bc689f5d8104 P 197b880fa39d432a9b472845876bfaf0: Append to log took a long time: real 0.294s user 0.001s sys 0.000s
I20260129 02:59:34.742491 22231 rpcz_store.cc:276] 0129 02:59:22.921032 (+ 0us) service_pool.cc:168] Inserting onto call queue
0129 02:59:22.921632 (+ 600us) service_pool.cc:225] Handling call
0129 02:59:23.225773 (+304141us) log.cc:844] Serialized 494 byte log entry
0129 02:59:23.477845 (+252072us) spinlock_profiling.cc:243] Waited 2.17 ms on lock 0x7b180032b420. stack: 00007f8fe044e373 00007f8fe044e270 00007f8fdfda0574 00007f8fe9a5c6b4 00007f8fe9a5c649 00007f8fe92d4c91 00007f8fe92e1671 00007f8fe04802d3 00007f8fe04826a1 00007f8fe0482659 00007f8fe04825e9 00007f8fe04825b1 00007f8fe04818ad 0000000000379254 0000000000379099 00007f8fe046369a
0129 02:59:34.643995 (+11166150us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":4020,"raft.queue_time_us":17497,"raft.run_cpu_time_us":1292,"raft.run_wall_time_us":27630,"spinlock_wait_cycles":4990208,"thread_start_us":179863,"threads_started":1,"wal-append.queue_time_us":204547,"wal-append.run_cpu_time_us":1934,"wal-append.run_wall_time_us":1005066}
I20260129 02:59:35.053216 22115 raft_consensus.cc:2804] T c2633a6fc34c44579de348d208b0c01e P 1c50de89a5a845668696dcb28c3badd6 [term 2 FOLLOWER]: Leader election won for term 2
I20260129 02:59:35.053817 22115 raft_consensus.cc:697] T c2633a6fc34c44579de348d208b0c01e P 1c50de89a5a845668696dcb28c3badd6 [term 2 LEADER]: Becoming Leader. State: Replica: 1c50de89a5a845668696dcb28c3badd6, State: Running, Role: LEADER
I20260129 02:59:35.054472 22292 consensus_queue.cc:1059] T add8d4923d1a489e86882c3a25692b95 P 1c50de89a5a845668696dcb28c3badd6 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20260129 02:59:34.993852 20441 raft_consensus.cc:1275] T 75be4bd6bd45474795fd472eff2953ac P 197b880fa39d432a9b472845876bfaf0 [term 7 FOLLOWER]: Refusing update from remote peer 1c50de89a5a845668696dcb28c3badd6: Log matching property violated. Preceding OpId in replica: term: 5 index: 4. Preceding OpId from leader: term: 7 index: 5. (index mismatch)
I20260129 02:59:35.057956 20441 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0340229d521b432580ae926f6062e3f7" candidate_uuid: "1c50de89a5a845668696dcb28c3badd6" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "197b880fa39d432a9b472845876bfaf0"
I20260129 02:59:34.951930 22036 consensus_queue.cc:1059] T a8479d1fe0bd41e5946186e3a6ca8c7a P 974b802141bd4f13beffef2aa03f88f9 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 }, Status: INVALID_TERM, Last received: 2.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.037s
I20260129 02:59:34.966259 22255 raft_consensus.cc:493] T 45cc5f3d2688486d9b6dbe7e666dedde P 7a9daf5d42a84615a6383cfec9a10608 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 1c50de89a5a845668696dcb28c3badd6)
I20260129 02:59:35.004956 20435 raft_consensus.cc:2393] T a8479d1fe0bd41e5946186e3a6ca8c7a P 197b880fa39d432a9b472845876bfaf0 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1c50de89a5a845668696dcb28c3badd6 in current term 3: Already voted for candidate 197b880fa39d432a9b472845876bfaf0 in this term.
I20260129 02:59:35.059734 22255 raft_consensus.cc:515] T 45cc5f3d2688486d9b6dbe7e666dedde P 7a9daf5d42a84615a6383cfec9a10608 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } }
W20260129 02:59:35.064350 22307 log.cc:927] Time spent T 75f45538dc704d9a8ec4ff8ad9f3ea5b P 7a9daf5d42a84615a6383cfec9a10608: Append to log took a long time: real 0.109s user 0.000s sys 0.001s
I20260129 02:59:35.064762 22338 raft_consensus.cc:493] T 8055232cf0004d40a85f3bf983fe00a9 P 1c50de89a5a845668696dcb28c3badd6 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 974b802141bd4f13beffef2aa03f88f9)
I20260129 02:59:35.065198 22115 consensus_queue.cc:237] T c2633a6fc34c44579de348d208b0c01e P 1c50de89a5a845668696dcb28c3badd6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:59:35.065519 22338 raft_consensus.cc:515] T 8055232cf0004d40a85f3bf983fe00a9 P 1c50de89a5a845668696dcb28c3badd6 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:59:35.070237 22112 raft_consensus.cc:3060] T 0f20af7fbe5d4db89a6f17948d98912c P 974b802141bd4f13beffef2aa03f88f9 [term 3 FOLLOWER]: Advancing to term 4
I20260129 02:59:35.063566 22255 leader_election.cc:290] T 45cc5f3d2688486d9b6dbe7e666dedde P 7a9daf5d42a84615a6383cfec9a10608 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 974b802141bd4f13beffef2aa03f88f9 (127.18.211.2:34517), 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3:37735)
W20260129 02:59:35.076731 22321 log.cc:927] Time spent T 926a250fa5834e88b96ccecbefbf5086 P 197b880fa39d432a9b472845876bfaf0: Append to log took a long time: real 0.053s user 0.000s sys 0.000s
I20260129 02:59:35.077276 22231 raft_consensus.cc:3055] T 1299723a4476420ab02fd8bfd7aa0645 P 197b880fa39d432a9b472845876bfaf0 [term 4 LEADER]: Stepping down as leader of term 4
I20260129 02:59:34.972194 20432 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "75be4bd6bd45474795fd472eff2953ac" candidate_uuid: "974b802141bd4f13beffef2aa03f88f9" candidate_term: 8 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "197b880fa39d432a9b472845876bfaf0" is_pre_election: true
I20260129 02:59:34.976559 22326 raft_consensus.cc:493] T 926a250fa5834e88b96ccecbefbf5086 P 197b880fa39d432a9b472845876bfaf0 [term 4 FOLLOWER]: Starting leader election (detected failure of leader 1c50de89a5a845668696dcb28c3badd6)
I20260129 02:59:35.083612 22326 raft_consensus.cc:3060] T 926a250fa5834e88b96ccecbefbf5086 P 197b880fa39d432a9b472845876bfaf0 [term 4 FOLLOWER]: Advancing to term 5
W20260129 02:59:35.007790 22317 log.cc:927] Time spent T a5a05fbcbbcf4916bbc01fd1a5b5e90d P 974b802141bd4f13beffef2aa03f88f9: Append to log took a long time: real 0.098s user 0.000s sys 0.000s
I20260129 02:59:35.008388 22077 raft_consensus.cc:3055] T a4c1226b987b468fa84dfc6ba0e2dbe1 P 197b880fa39d432a9b472845876bfaf0 [term 2 LEADER]: Stepping down as leader of term 2
I20260129 02:59:35.088158 22184 consensus_queue.cc:260] T e4d18c35b0a7493e85d0a4b92956a48f P 974b802141bd4f13beffef2aa03f88f9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:59:35.089000 22077 raft_consensus.cc:740] T a4c1226b987b468fa84dfc6ba0e2dbe1 P 197b880fa39d432a9b472845876bfaf0 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 197b880fa39d432a9b472845876bfaf0, State: Running, Role: LEADER
I20260129 02:59:35.089386 22184 raft_consensus.cc:3060] T e4d18c35b0a7493e85d0a4b92956a48f P 974b802141bd4f13beffef2aa03f88f9 [term 2 FOLLOWER]: Advancing to term 3
W20260129 02:59:35.090488 22243 consensus_meta.cc:287] T 18f30ebaa4174fd28c958e72d0cc7e24 P 1c50de89a5a845668696dcb28c3badd6: Time spent flushing consensus metadata: real 0.589s user 0.009s sys 0.000s
W20260129 02:59:35.011763 21101 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260129 02:59:35.013113 20588 raft_consensus.cc:1275] T 75b0b6e82ddc4f92b1fe1d94f9a0b73b P 1c50de89a5a845668696dcb28c3badd6 [term 2 FOLLOWER]: Refusing update from remote peer 7a9daf5d42a84615a6383cfec9a10608: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260129 02:59:34.987757 22315 raft_consensus.cc:2749] T 4f5a77e94fef4b16a2ba7b067c8b144a P 1c50de89a5a845668696dcb28c3badd6 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260129 02:59:35.009171 22197 raft_consensus.cc:3055] T 368953c9102643b2a21f99af36b51905 P 974b802141bd4f13beffef2aa03f88f9 [term 3 LEADER]: Stepping down as leader of term 3
I20260129 02:59:35.092020 22197 raft_consensus.cc:740] T 368953c9102643b2a21f99af36b51905 P 974b802141bd4f13beffef2aa03f88f9 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 974b802141bd4f13beffef2aa03f88f9, State: Running, Role: LEADER
W20260129 02:59:35.093086 22213 consensus_meta.cc:287] T 17d6431cc77a491e8464d68db4081a51 P 1c50de89a5a845668696dcb28c3badd6: Time spent flushing consensus metadata: real 0.659s user 0.012s sys 0.000s
I20260129 02:59:35.037675 22322 consensus_queue.cc:1059] T c2633a6fc34c44579de348d208b0c01e P 197b880fa39d432a9b472845876bfaf0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.040s
I20260129 02:59:35.090955 22243 raft_consensus.cc:515] T 18f30ebaa4174fd28c958e72d0cc7e24 P 1c50de89a5a845668696dcb28c3badd6 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } }
W20260129 02:59:35.038601 22304 log.cc:927] Time spent T 42d27a28d89b409e8807ec0efd97af69 P 197b880fa39d432a9b472845876bfaf0: Append to log took a long time: real 0.230s user 0.001s sys 0.000s
W20260129 02:59:35.100947 20406 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260129 02:59:35.104218 20436 raft_consensus.cc:2468] T 4b1876f9a70b49edb3aa74105e66d3b3 P 197b880fa39d432a9b472845876bfaf0 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 974b802141bd4f13beffef2aa03f88f9 in term 4.
I20260129 02:59:35.104826 22294 raft_consensus.cc:2764] T 4b1876f9a70b49edb3aa74105e66d3b3 P 197b880fa39d432a9b472845876bfaf0 [term 4 FOLLOWER]: Leader pre-election decision vote started in defunct term 3: won
I20260129 02:59:35.105787 20441 raft_consensus.cc:1240] T 4b1876f9a70b49edb3aa74105e66d3b3 P 197b880fa39d432a9b472845876bfaf0 [term 4 FOLLOWER]: Rejecting Update request from peer 1c50de89a5a845668696dcb28c3badd6 for earlier term 3. Current term is 4. Ops: []
I20260129 02:59:35.108817 22247 raft_consensus.cc:3060] T a644c02b16af47679824ccd060ac10fd P 1c50de89a5a845668696dcb28c3badd6 [term 1 FOLLOWER]: Advancing to term 2
I20260129 02:59:35.110236 22198 raft_consensus.cc:493] T e33a417cf9c54cf5af73a4ee3aac4d3a P 197b880fa39d432a9b472845876bfaf0 [term 3 FOLLOWER]: Starting leader election (detected failure of leader 1c50de89a5a845668696dcb28c3badd6)
I20260129 02:59:35.110754 22198 raft_consensus.cc:3060] T e33a417cf9c54cf5af73a4ee3aac4d3a P 197b880fa39d432a9b472845876bfaf0 [term 3 FOLLOWER]: Advancing to term 4
I20260129 02:59:35.092939 22197 consensus_queue.cc:260] T 368953c9102643b2a21f99af36b51905 P 974b802141bd4f13beffef2aa03f88f9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.3, Last appended by leader: 3, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:59:35.116871 22197 raft_consensus.cc:3060] T 368953c9102643b2a21f99af36b51905 P 974b802141bd4f13beffef2aa03f88f9 [term 3 FOLLOWER]: Advancing to term 4
I20260129 02:59:35.039746 22201 consensus_queue.cc:260] T 8055232cf0004d40a85f3bf983fe00a9 P 974b802141bd4f13beffef2aa03f88f9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
W20260129 02:59:35.027516 22309 log.cc:927] Time spent T 96609ac68bdf4993a12e1c23355ca013 P 974b802141bd4f13beffef2aa03f88f9: Append to log took a long time: real 0.230s user 0.000s sys 0.001s
I20260129 02:59:35.021749 20434 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a0fcc58d6fb6473cafbd9d34d5195269" candidate_uuid: "1c50de89a5a845668696dcb28c3badd6" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "197b880fa39d432a9b472845876bfaf0" is_pre_election: true
W20260129 02:59:35.042274 22291 log.cc:927] Time spent T dc2f4cb752cf4d59b5f6bc689f5d8104 P 1c50de89a5a845668696dcb28c3badd6: Append to log took a long time: real 0.325s user 0.001s sys 0.002s
I20260129 02:59:35.122507 20586 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:48526 (request call id 143) took 1549 ms. Trace:
I20260129 02:59:35.122882 22201 raft_consensus.cc:3060] T 8055232cf0004d40a85f3bf983fe00a9 P 974b802141bd4f13beffef2aa03f88f9 [term 1 FOLLOWER]: Advancing to term 2
I20260129 02:59:35.122960 20586 rpcz_store.cc:276] 0129 02:59:33.623930 (+ 0us) service_pool.cc:168] Inserting onto call queue
0129 02:59:33.660748 (+ 36818us) service_pool.cc:225] Handling call
0129 02:59:33.710286 (+ 49538us) raft_consensus.cc:1471] Updating replica for 1 ops
0129 02:59:33.985444 (+275158us) raft_consensus.cc:1534] Early marking committed up to index 0
0129 02:59:33.985475 (+ 31us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0129 02:59:33.985990 (+ 515us) log.cc:844] Serialized 26 byte log entry
0129 02:59:34.457936 (+471946us) raft_consensus.cc:1645] Marking committed up to 4
0129 02:59:34.457989 (+ 53us) raft_consensus.cc:1697] Filling consensus response to leader.
0129 02:59:34.458077 (+ 88us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0129 02:59:35.041621 (+583544us) spinlock_profiling.cc:243] Waited 2.85 ms on lock 0x7b440002304c. stack: 00007f8fe044e373 00007f8fe044e270 00007f8fdfda0574 00007f8fe9a5c6b4 00007f8fe9a5c649 00007f8fe02c08b1 00007f8fe038f005 00007f8fe038fa22 00007f8fe30d957b 00007f8fe30d4e4f 00007f8fe30c8c0e 00007f8fe30c7f56 00007f8fe30c7b78 00007f8fe30d15c1 00007f8fe30d1579 00007f8fe30d1509
0129 02:59:35.122230 (+ 80609us) raft_consensus.cc:1685] finished
0129 02:59:35.122259 (+ 29us) raft_consensus.cc:1691] UpdateReplicas() finished
0129 02:59:35.122450 (+ 191us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":218878,"raft.queue_time_us":219027,"raft.run_cpu_time_us":113,"raft.run_wall_time_us":113,"spinlock_wait_cycles":6557440,"thread_start_us":471545,"threads_started":1,"wal-append.queue_time_us":629953}
I20260129 02:59:35.124325 22323 raft_consensus.cc:3055] T 724a236524cb4e7ba3428880681d6cc2 P 974b802141bd4f13beffef2aa03f88f9 [term 1 LEADER]: Stepping down as leader of term 1
I20260129 02:59:35.125478 22120 raft_consensus.cc:493] T 0678b8b468bd43cdbc3e485900844821 P 197b880fa39d432a9b472845876bfaf0 [term 4 FOLLOWER]: Starting leader election (detected failure of leader 7a9daf5d42a84615a6383cfec9a10608)
I20260129 02:59:35.126026 22120 raft_consensus.cc:3060] T 0678b8b468bd43cdbc3e485900844821 P 197b880fa39d432a9b472845876bfaf0 [term 4 FOLLOWER]: Advancing to term 5
I20260129 02:59:35.043761 22091 raft_consensus.cc:2804] T 96609ac68bdf4993a12e1c23355ca013 P 197b880fa39d432a9b472845876bfaf0 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260129 02:59:35.127804 22091 raft_consensus.cc:493] T 96609ac68bdf4993a12e1c23355ca013 P 197b880fa39d432a9b472845876bfaf0 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260129 02:59:35.128216 22091 raft_consensus.cc:3060] T 96609ac68bdf4993a12e1c23355ca013 P 197b880fa39d432a9b472845876bfaf0 [term 3 FOLLOWER]: Advancing to term 4
I20260129 02:59:35.130208 20439 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55676 (request call id 1085) took 4226 ms. Trace:
I20260129 02:59:35.130704 20439 rpcz_store.cc:276] 0129 02:59:30.903457 (+ 0us) service_pool.cc:168] Inserting onto call queue
0129 02:59:34.254514 (+3351057us) service_pool.cc:225] Handling call
0129 02:59:34.285410 (+ 30896us) raft_consensus.cc:1471] Updating replica for 1 ops
0129 02:59:34.957957 (+672547us) raft_consensus.cc:1697] Filling consensus response to leader.
0129 02:59:35.129853 (+171896us) spinlock_profiling.cc:243] Waited 671 ms on lock 0x7b5400230854. stack: 00007f8fe044e373 00007f8fe044e270 00007f8fdfda0574 00007f8fe9a5c6b4 00007f8fe9a5c649 00007f8fe34975a1 00007f8fe352ea81 00007f8fe352d4be 00007f8fe84e1f49 00007f8fe1a6ed34 00007f8fe1a6ecc2 00007f8fe1a6ebf1 00007f8fe1a6eb6c 00007f8fe1a6de22 00007f8fe186d34c 00007f8fe186c786
0129 02:59:35.130133 (+ 280us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1543557888}
I20260129 02:59:35.044971 20433 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55676 (request call id 1118) took 1333 ms. Trace:
I20260129 02:59:35.097993 22338 leader_election.cc:290] T 8055232cf0004d40a85f3bf983fe00a9 P 1c50de89a5a845668696dcb28c3badd6 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 197b880fa39d432a9b472845876bfaf0 (127.18.211.1:42027), 974b802141bd4f13beffef2aa03f88f9 (127.18.211.2:34517)
I20260129 02:59:35.132341 20433 rpcz_store.cc:276] 0129 02:59:33.711487 (+ 0us) service_pool.cc:168] Inserting onto call queue
0129 02:59:34.406937 (+695450us) service_pool.cc:225] Handling call
0129 02:59:34.432064 (+ 25127us) raft_consensus.cc:1471] Updating replica for 0 ops
0129 02:59:34.434760 (+ 2696us) raft_consensus.cc:1534] Early marking committed up to index 5
0129 02:59:34.692569 (+257809us) log.cc:844] Serialized 14 byte log entry
0129 02:59:35.044351 (+351782us) spinlock_profiling.cc:243] Waited 12 ms on lock 0x7b18001d02c0. stack: 00007f8fe044e373 00007f8fe044e270 00007f8fdfda0574 00007f8fe9a5c6b4 00007f8fe9a5c649 00007f8fe92d4c91 00007f8fe92e1671 00007f8fe30d585c 00007f8fe04632eb 00007f8fe924b709 00007f8fe047f57b 00007f8fe047d81f 00007f8fe047fae2 00007f8fe30c75ab 00007f8fe30cd0c3 00007f8fe30cd5da
0129 02:59:35.044552 (+ 201us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0129 02:59:35.044616 (+ 64us) raft_consensus.cc:1645] Marking committed up to 5
0129 02:59:35.044669 (+ 53us) raft_consensus.cc:1697] Filling consensus response to leader.
0129 02:59:35.044756 (+ 87us) raft_consensus.cc:1691] UpdateReplicas() finished
0129 02:59:35.044902 (+ 146us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":8910,"prepare.queue_time_us":349638,"prepare.run_cpu_time_us":113,"prepare.run_wall_time_us":5995,"raft.queue_time_us":10488,"raft.run_cpu_time_us":98,"raft.run_wall_time_us":97,"spinlock_wait_cycles":27580160,"thread_start_us":373225,"threads_started":2,"wal-append.queue_time_us":259532}
I20260129 02:59:35.100327 22213 raft_consensus.cc:515] T 17d6431cc77a491e8464d68db4081a51 P 1c50de89a5a845668696dcb28c3badd6 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:59:35.025238 22296 raft_consensus.cc:3055] T 4f8f3a38f6e24ba4857875181f447442 P 974b802141bd4f13beffef2aa03f88f9 [term 3 LEADER]: Stepping down as leader of term 3
I20260129 02:59:35.134547 22296 raft_consensus.cc:740] T 4f8f3a38f6e24ba4857875181f447442 P 974b802141bd4f13beffef2aa03f88f9 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 974b802141bd4f13beffef2aa03f88f9, State: Running, Role: LEADER
I20260129 02:59:34.993105 22212 raft_consensus.cc:515] T 3e2d15f8640b407195210c476e09df91 P 7a9daf5d42a84615a6383cfec9a10608 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } }
I20260129 02:59:35.124879 22323 raft_consensus.cc:740] T 724a236524cb4e7ba3428880681d6cc2 P 974b802141bd4f13beffef2aa03f88f9 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 974b802141bd4f13beffef2aa03f88f9, State: Running, Role: LEADER
I20260129 02:59:35.135483 22213 leader_election.cc:290] T 17d6431cc77a491e8464d68db4081a51 P 1c50de89a5a845668696dcb28c3badd6 [CANDIDATE]: Term 2 election: Requested vote from peers 197b880fa39d432a9b472845876bfaf0 (127.18.211.1:42027), 974b802141bd4f13beffef2aa03f88f9 (127.18.211.2:34517)
I20260129 02:59:35.120184 22333 raft_consensus.cc:493] T 505580c93c694bbb8ea96fdf9b17ab46 P 974b802141bd4f13beffef2aa03f88f9 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260129 02:59:35.136993 22212 leader_election.cc:290] T 3e2d15f8640b407195210c476e09df91 P 7a9daf5d42a84615a6383cfec9a10608 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 974b802141bd4f13beffef2aa03f88f9 (127.18.211.2:34517), 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3:37735)
I20260129 02:59:35.136706 22333 raft_consensus.cc:515] T 505580c93c694bbb8ea96fdf9b17ab46 P 974b802141bd4f13beffef2aa03f88f9 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } }
I20260129 02:59:34.789677 20437 rpcz_store.cc:276] 0129 02:59:31.841075 (+ 0us) service_pool.cc:168] Inserting onto call queue
0129 02:59:34.289899 (+2448824us) service_pool.cc:225] Handling call
0129 02:59:34.290319 (+ 420us) raft_consensus.cc:1471] Updating replica for 0 ops
0129 02:59:34.290588 (+ 269us) raft_consensus.cc:1534] Early marking committed up to index 2
0129 02:59:34.510011 (+219423us) log.cc:844] Serialized 14 byte log entry
0129 02:59:34.786060 (+276049us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0129 02:59:34.786160 (+ 100us) raft_consensus.cc:1645] Marking committed up to 2
0129 02:59:34.786282 (+ 122us) raft_consensus.cc:1697] Filling consensus response to leader.
0129 02:59:34.786471 (+ 189us) raft_consensus.cc:1691] UpdateReplicas() finished
0129 02:59:34.786761 (+ 290us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":477550,"prepare.run_cpu_time_us":64,"prepare.run_wall_time_us":1587,"thread_start_us":470581,"threads_started":2}
I20260129 02:59:34.997942 20405 leader_election.cc:304] T e4d18c35b0a7493e85d0a4b92956a48f P 197b880fa39d432a9b472845876bfaf0 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 197b880fa39d432a9b472845876bfaf0; no voters: 1c50de89a5a845668696dcb28c3badd6, 974b802141bd4f13beffef2aa03f88f9
W20260129 02:59:35.138199 20405 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.211.3:37735, user_credentials={real_user=slave}} blocked reactor thread for 140356us
I20260129 02:59:35.135475 22296 consensus_queue.cc:260] T 4f8f3a38f6e24ba4857875181f447442 P 974b802141bd4f13beffef2aa03f88f9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:59:35.139029 22333 leader_election.cc:290] T 505580c93c694bbb8ea96fdf9b17ab46 P 974b802141bd4f13beffef2aa03f88f9 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3:37735), 7a9daf5d42a84615a6383cfec9a10608 (127.18.211.4:40439)
I20260129 02:59:35.141278 20433 raft_consensus.cc:1240] T 0f20af7fbe5d4db89a6f17948d98912c P 197b880fa39d432a9b472845876bfaf0 [term 4 FOLLOWER]: Rejecting Update request from peer 974b802141bd4f13beffef2aa03f88f9 for earlier term 3. Current term is 4. Ops: []
I20260129 02:59:35.045687 22190 raft_consensus.cc:515] T fcddab6bca684adf8ea032289a00f90e P 197b880fa39d432a9b472845876bfaf0 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:59:35.056480 22341 consensus_queue.cc:1048] T 75be4bd6bd45474795fd472eff2953ac P 1c50de89a5a845668696dcb28c3badd6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 3, Time since last communication: 0.005s
I20260129 02:59:34.995150 20438 raft_consensus.cc:2376] T 1ce4ddf5b10e4ae8b193b13b2e5936ee P 197b880fa39d432a9b472845876bfaf0 [term 4 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 1c50de89a5a845668696dcb28c3badd6 in term 4. Re-sending same reply.
I20260129 02:59:35.089967 22077 consensus_queue.cc:260] T a4c1226b987b468fa84dfc6ba0e2dbe1 P 197b880fa39d432a9b472845876bfaf0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:59:34.993021 22330 raft_consensus.cc:2749] T dc2f4cb752cf4d59b5f6bc689f5d8104 P 197b880fa39d432a9b472845876bfaf0 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260129 02:59:35.093924 20432 raft_consensus.cc:2410] T 75be4bd6bd45474795fd472eff2953ac P 197b880fa39d432a9b472845876bfaf0 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 974b802141bd4f13beffef2aa03f88f9 for term 8 because replica has last-logged OpId of term: 5 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 4 index: 3.
I20260129 02:59:35.157325 22077 raft_consensus.cc:3060] T a4c1226b987b468fa84dfc6ba0e2dbe1 P 197b880fa39d432a9b472845876bfaf0 [term 2 FOLLOWER]: Advancing to term 3
I20260129 02:59:35.158918 22190 leader_election.cc:290] T fcddab6bca684adf8ea032289a00f90e P 197b880fa39d432a9b472845876bfaf0 [CANDIDATE]: Term 7 election: Requested vote from peers 974b802141bd4f13beffef2aa03f88f9 (127.18.211.2:34517), 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3:37735)
I20260129 02:59:34.952565 22079 consensus_queue.cc:237] T 724a236524cb4e7ba3428880681d6cc2 P 1c50de89a5a845668696dcb28c3badd6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:59:35.038157 22257 raft_consensus.cc:2749] T 78aa39096868415d9afc2ffcb248e198 P 7a9daf5d42a84615a6383cfec9a10608 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260129 02:59:35.154464 22212 raft_consensus.cc:493] T 505580c93c694bbb8ea96fdf9b17ab46 P 7a9daf5d42a84615a6383cfec9a10608 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260129 02:59:35.162426 22212 raft_consensus.cc:515] T 505580c93c694bbb8ea96fdf9b17ab46 P 7a9daf5d42a84615a6383cfec9a10608 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } }
I20260129 02:59:35.099046 22337 raft_consensus.cc:3055] T add8d4923d1a489e86882c3a25692b95 P 1c50de89a5a845668696dcb28c3badd6 [term 1 LEADER]: Stepping down as leader of term 1
I20260129 02:59:35.099166 22036 raft_consensus.cc:2749] T 76f713c3331540d2a5b76164103e9eb1 P 974b802141bd4f13beffef2aa03f88f9 [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20260129 02:59:35.118021 20554 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.211.4:40439, user_credentials={real_user=slave}} blocked reactor thread for 105522us
I20260129 02:59:35.096118 22231 raft_consensus.cc:740] T 1299723a4476420ab02fd8bfd7aa0645 P 197b880fa39d432a9b472845876bfaf0 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 197b880fa39d432a9b472845876bfaf0, State: Running, Role: LEADER
I20260129 02:59:35.120146 20479 leader_election.cc:304] T 4b1876f9a70b49edb3aa74105e66d3b3 P 974b802141bd4f13beffef2aa03f88f9 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 197b880fa39d432a9b472845876bfaf0, 974b802141bd4f13beffef2aa03f88f9; no voters:
I20260129 02:59:35.120335 22302 raft_consensus.cc:3055] T a8479d1fe0bd41e5946186e3a6ca8c7a P 974b802141bd4f13beffef2aa03f88f9 [term 2 LEADER]: Stepping down as leader of term 2
I20260129 02:59:35.117374 22322 raft_consensus.cc:3055] T c2633a6fc34c44579de348d208b0c01e P 197b880fa39d432a9b472845876bfaf0 [term 1 LEADER]: Stepping down as leader of term 1
I20260129 02:59:35.054970 20440 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:48356 (request call id 211) took 4408 ms. Trace:
I20260129 02:59:35.144372 21127 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4e420f90884f4796997b5ea41e49e411" candidate_uuid: "197b880fa39d432a9b472845876bfaf0" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "7a9daf5d42a84615a6383cfec9a10608"
I20260129 02:59:35.145854 21135 raft_consensus.cc:1275] T c934d31efd3f4e5195605ddc354d14e1 P 7a9daf5d42a84615a6383cfec9a10608 [term 3 FOLLOWER]: Refusing update from remote peer 197b880fa39d432a9b472845876bfaf0: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 3 index: 6. (index mismatch)
I20260129 02:59:35.146654 21136 raft_consensus.cc:1240] T 1299723a4476420ab02fd8bfd7aa0645 P 7a9daf5d42a84615a6383cfec9a10608 [term 5 FOLLOWER]: Rejecting Update request from peer 197b880fa39d432a9b472845876bfaf0 for earlier term 4. Current term is 5. Ops: []
I20260129 02:59:35.199029 20440 rpcz_store.cc:276] 0129 02:59:30.647161 (+ 0us) service_pool.cc:168] Inserting onto call queue
0129 02:59:34.060695 (+3413534us) service_pool.cc:225] Handling call
0129 02:59:34.104251 (+ 43556us) raft_consensus.cc:1471] Updating replica for 1 ops
0129 02:59:34.104503 (+ 252us) raft_consensus.cc:1534] Early marking committed up to index 0
0129 02:59:34.104527 (+ 24us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0129 02:59:34.105013 (+ 486us) log.cc:844] Serialized 26 byte log entry
0129 02:59:34.436559 (+331546us) raft_consensus.cc:1645] Marking committed up to 4
0129 02:59:34.436599 (+ 40us) raft_consensus.cc:1697] Filling consensus response to leader.
0129 02:59:34.436665 (+ 66us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0129 02:59:35.054768 (+618103us) raft_consensus.cc:1685] finished
0129 02:59:35.054793 (+ 25us) raft_consensus.cc:1691] UpdateReplicas() finished
0129 02:59:35.054913 (+ 120us) inbound_call.cc:173] Queueing success response
Metrics: {"thread_start_us":331163,"threads_started":1,"wal-append.queue_time_us":500022}
I20260129 02:59:35.147506 21128 raft_consensus.cc:1240] T fd8311b779af4e1fa1402b2a02a602f2 P 7a9daf5d42a84615a6383cfec9a10608 [term 3 FOLLOWER]: Rejecting Update request from peer 197b880fa39d432a9b472845876bfaf0 for earlier term 2. Current term is 3. Ops: []
I20260129 02:59:35.144369 21130 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2ac659d732314e1288849f08fe5e074e" candidate_uuid: "197b880fa39d432a9b472845876bfaf0" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "7a9daf5d42a84615a6383cfec9a10608"
I20260129 02:59:35.142997 22335 consensus_queue.cc:1059] T febffe4a8b394368b69b1d164308fb9a P 197b880fa39d432a9b472845876bfaf0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.012s
I20260129 02:59:35.142575 22336 raft_consensus.cc:2749] T e4d18c35b0a7493e85d0a4b92956a48f P 197b880fa39d432a9b472845876bfaf0 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260129 02:59:35.139487 22296 raft_consensus.cc:3060] T 4f8f3a38f6e24ba4857875181f447442 P 974b802141bd4f13beffef2aa03f88f9 [term 3 FOLLOWER]: Advancing to term 4
I20260129 02:59:35.178612 20587 raft_consensus.cc:1240] T 18f30ebaa4174fd28c958e72d0cc7e24 P 1c50de89a5a845668696dcb28c3badd6 [term 3 FOLLOWER]: Rejecting Update request from peer 7a9daf5d42a84615a6383cfec9a10608 for earlier term 2. Current term is 3. Ops: []
W20260129 02:59:35.181555 20479 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.211.1:42027, user_credentials={real_user=slave}} blocked reactor thread for 62783us
I20260129 02:59:35.117492 22160 raft_consensus.cc:2749] T 7a9a69cd82724b1faed1d10aa091fc4a P 974b802141bd4f13beffef2aa03f88f9 [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260129 02:59:35.171231 22337 raft_consensus.cc:740] T add8d4923d1a489e86882c3a25692b95 P 1c50de89a5a845668696dcb28c3badd6 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 1c50de89a5a845668696dcb28c3badd6, State: Running, Role: LEADER
I20260129 02:59:35.139953 22323 consensus_queue.cc:260] T 724a236524cb4e7ba3428880681d6cc2 P 974b802141bd4f13beffef2aa03f88f9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
W20260129 02:59:35.209415 20556 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.211.2:34517, user_credentials={real_user=slave}} blocked reactor thread for 250370us
I20260129 02:59:35.204272 22112 raft_consensus.cc:2804] T 4b1876f9a70b49edb3aa74105e66d3b3 P 974b802141bd4f13beffef2aa03f88f9 [term 4 FOLLOWER]: Leader election won for term 4
I20260129 02:59:35.187506 22302 raft_consensus.cc:740] T a8479d1fe0bd41e5946186e3a6ca8c7a P 974b802141bd4f13beffef2aa03f88f9 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 974b802141bd4f13beffef2aa03f88f9, State: Running, Role: LEADER
I20260129 02:59:35.222801 20439 raft_consensus.cc:1240] T e4d18c35b0a7493e85d0a4b92956a48f P 197b880fa39d432a9b472845876bfaf0 [term 3 FOLLOWER]: Rejecting Update request from peer 974b802141bd4f13beffef2aa03f88f9 for earlier term 2. Current term is 3. Ops: []
I20260129 02:59:35.198968 22322 raft_consensus.cc:740] T c2633a6fc34c44579de348d208b0c01e P 197b880fa39d432a9b472845876bfaf0 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 197b880fa39d432a9b472845876bfaf0, State: Running, Role: LEADER
W20260129 02:59:35.232757 21099 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.211.1:42027, user_credentials={real_user=slave}} blocked reactor thread for 276347us
I20260129 02:59:35.209554 21130 raft_consensus.cc:3060] T 2ac659d732314e1288849f08fe5e074e P 7a9daf5d42a84615a6383cfec9a10608 [term 1 FOLLOWER]: Advancing to term 2
I20260129 02:59:35.228657 20556 leader_election.cc:304] T 7a9a69cd82724b1faed1d10aa091fc4a P 1c50de89a5a845668696dcb28c3badd6 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1c50de89a5a845668696dcb28c3badd6, 974b802141bd4f13beffef2aa03f88f9; no voters: 197b880fa39d432a9b472845876bfaf0
W20260129 02:59:35.211309 22293 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.574s user 0.006s sys 0.020s
I20260129 02:59:35.205529 22061 raft_consensus.cc:697] T 1ce4ddf5b10e4ae8b193b13b2e5936ee P 1c50de89a5a845668696dcb28c3badd6 [term 4 LEADER]: Becoming Leader. State: Replica: 1c50de89a5a845668696dcb28c3badd6, State: Running, Role: LEADER
I20260129 02:59:35.239050 20440 raft_consensus.cc:1275] T a5a05fbcbbcf4916bbc01fd1a5b5e90d P 197b880fa39d432a9b472845876bfaf0 [term 5 FOLLOWER]: Refusing update from remote peer 974b802141bd4f13beffef2aa03f88f9: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 5 index: 4. (index mismatch)
W20260129 02:59:35.242321 20482 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260129 02:59:35.242761 21099 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260129 02:59:35.245501 20482 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260129 02:59:35.245606 20480 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.211.3:37735, user_credentials={real_user=slave}} blocked reactor thread for 248122us
W20260129 02:59:35.243896 20407 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.211.2:34517, user_credentials={real_user=slave}} blocked reactor thread for 299759us
W20260129 02:59:35.240175 22293 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.619s user 0.006s sys 0.021s
I20260129 02:59:35.253028 20480 leader_election.cc:304] T 3906b080a5b34a74b8b64128635e8997 P 974b802141bd4f13beffef2aa03f88f9 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 974b802141bd4f13beffef2aa03f88f9; no voters: 197b880fa39d432a9b472845876bfaf0, 1c50de89a5a845668696dcb28c3badd6
I20260129 02:59:35.237919 22112 raft_consensus.cc:697] T 4b1876f9a70b49edb3aa74105e66d3b3 P 974b802141bd4f13beffef2aa03f88f9 [term 4 LEADER]: Becoming Leader. State: Replica: 974b802141bd4f13beffef2aa03f88f9, State: Running, Role: LEADER
I20260129 02:59:35.215143 22082 consensus_queue.cc:1059] T 3906b080a5b34a74b8b64128635e8997 P 197b880fa39d432a9b472845876bfaf0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.033s
I20260129 02:59:35.240847 22344 raft_consensus.cc:493] T 42d27a28d89b409e8807ec0efd97af69 P 1c50de89a5a845668696dcb28c3badd6 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260129 02:59:35.265496 21102 leader_election.cc:336] T 2ac659d732314e1288849f08fe5e074e P 7a9daf5d42a84615a6383cfec9a10608 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 974b802141bd4f13beffef2aa03f88f9 (127.18.211.2:34517): Timed out: RequestConsensusVote RPC to 127.18.211.2:34517 timed out after 1.962s (SENT)
I20260129 02:59:35.265973 21102 leader_election.cc:304] T 2ac659d732314e1288849f08fe5e074e P 7a9daf5d42a84615a6383cfec9a10608 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7a9daf5d42a84615a6383cfec9a10608; no voters: 197b880fa39d432a9b472845876bfaf0, 974b802141bd4f13beffef2aa03f88f9
I20260129 02:59:35.264052 20553 leader_election.cc:304] T a8479d1fe0bd41e5946186e3a6ca8c7a P 1c50de89a5a845668696dcb28c3badd6 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1c50de89a5a845668696dcb28c3badd6; no voters: 197b880fa39d432a9b472845876bfaf0, 974b802141bd4f13beffef2aa03f88f9
I20260129 02:59:35.269375 20553 leader_election.cc:304] T a0fcc58d6fb6473cafbd9d34d5195269 P 1c50de89a5a845668696dcb28c3badd6 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1c50de89a5a845668696dcb28c3badd6; no voters: 197b880fa39d432a9b472845876bfaf0, 974b802141bd4f13beffef2aa03f88f9
I20260129 02:59:35.226854 22231 consensus_queue.cc:260] T 1299723a4476420ab02fd8bfd7aa0645 P 197b880fa39d432a9b472845876bfaf0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } }
I20260129 02:59:35.267410 22337 consensus_queue.cc:260] T add8d4923d1a489e86882c3a25692b95 P 1c50de89a5a845668696dcb28c3badd6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } }
I20260129 02:59:35.258617 22323 raft_consensus.cc:3060] T 724a236524cb4e7ba3428880681d6cc2 P 974b802141bd4f13beffef2aa03f88f9 [term 1 FOLLOWER]: Advancing to term 2
I20260129 02:59:35.267617 22160 raft_consensus.cc:2749] T 3906b080a5b34a74b8b64128635e8997 P 974b802141bd4f13beffef2aa03f88f9 [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260129 02:59:35.258421 22257 consensus_queue.cc:1059] T 4e3805ffa70242e8ade66da260dddf66 P 7a9daf5d42a84615a6383cfec9a10608 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.030s
I20260129 02:59:35.247016 22336 consensus_queue.cc:1059] T fd8311b779af4e1fa1402b2a02a602f2 P 197b880fa39d432a9b472845876bfaf0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.041s
I20260129 02:59:35.268179 22344 raft_consensus.cc:515] T 42d27a28d89b409e8807ec0efd97af69 P 1c50de89a5a845668696dcb28c3badd6 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } }
I20260129 02:59:35.269435 22212 leader_election.cc:290] T 505580c93c694bbb8ea96fdf9b17ab46 P 7a9daf5d42a84615a6383cfec9a10608 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 974b802141bd4f13beffef2aa03f88f9 (127.18.211.2:34517), 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3:37735)
I20260129 02:59:35.269717 20512 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a609ac02fae84725ab743f95f51d33ce" candidate_uuid: "197b880fa39d432a9b472845876bfaf0" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "974b802141bd4f13beffef2aa03f88f9" is_pre_election: true
I20260129 02:59:35.271811 22319 raft_consensus.cc:493] T 4408006eb5174de1896b037cf3f18457 P 1c50de89a5a845668696dcb28c3badd6 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 974b802141bd4f13beffef2aa03f88f9)
I20260129 02:59:35.275143 20513 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e7d83d4ccf684e76acb29a0dc844e1b8" candidate_uuid: "197b880fa39d432a9b472845876bfaf0" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "974b802141bd4f13beffef2aa03f88f9" is_pre_election: true
W20260129 02:59:35.383724 20479 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.211.1:42027, user_credentials={real_user=slave}} blocked reactor thread for 80636.1us
I20260129 02:59:35.272136 22358 consensus_queue.cc:1048] T c934d31efd3f4e5195605ddc354d14e1 P 197b880fa39d432a9b472845876bfaf0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.175s
W20260129 02:59:35.384631 20479 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260129 02:59:35.385191 20556 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.211.2:34517, user_credentials={real_user=slave}} blocked reactor thread for 141289us
I20260129 02:59:35.385591 22308 raft_consensus.cc:515] T 913fc7d751ed4b7388d889aac30a5f9b P 1c50de89a5a845668696dcb28c3badd6 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:59:35.385946 20556 leader_election.cc:304] T 36ff33be666a4c82954731691dbcf327 P 1c50de89a5a845668696dcb28c3badd6 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1c50de89a5a845668696dcb28c3badd6, 974b802141bd4f13beffef2aa03f88f9; no voters: 197b880fa39d432a9b472845876bfaf0
I20260129 02:59:35.386058 20511 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0f20af7fbe5d4db89a6f17948d98912c" candidate_uuid: "197b880fa39d432a9b472845876bfaf0" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "974b802141bd4f13beffef2aa03f88f9" is_pre_election: true
I20260129 02:59:35.386758 20511 raft_consensus.cc:2468] T 0f20af7fbe5d4db89a6f17948d98912c P 974b802141bd4f13beffef2aa03f88f9 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 197b880fa39d432a9b472845876bfaf0 in term 4.
I20260129 02:59:35.288527 22257 consensus_queue.cc:1243] T 4e3805ffa70242e8ade66da260dddf66 P 7a9daf5d42a84615a6383cfec9a10608 [LEADER]: Peer 1c50de89a5a845668696dcb28c3badd6 log is divergent from this leader: its last log entry 3.6 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 5
W20260129 02:59:35.385351 20479 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260129 02:59:35.280292 22343 consensus_queue.cc:1048] T 75b0b6e82ddc4f92b1fe1d94f9a0b73b P 7a9daf5d42a84615a6383cfec9a10608 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.141s
I20260129 02:59:35.297798 22319 raft_consensus.cc:515] T 4408006eb5174de1896b037cf3f18457 P 1c50de89a5a845668696dcb28c3badd6 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:59:35.280110 22302 consensus_queue.cc:260] T a8479d1fe0bd41e5946186e3a6ca8c7a P 974b802141bd4f13beffef2aa03f88f9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:59:35.300382 20513 raft_consensus.cc:2393] T e7d83d4ccf684e76acb29a0dc844e1b8 P 974b802141bd4f13beffef2aa03f88f9 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 197b880fa39d432a9b472845876bfaf0 in current term 3: Already voted for candidate 974b802141bd4f13beffef2aa03f88f9 in this term.
I20260129 02:59:35.267345 22349 raft_consensus.cc:2804] T 7a9a69cd82724b1faed1d10aa091fc4a P 1c50de89a5a845668696dcb28c3badd6 [term 4 FOLLOWER]: Leader election won for term 4
W20260129 02:59:35.366801 20404 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260129 02:59:35.290830 20512 raft_consensus.cc:2393] T a609ac02fae84725ab743f95f51d33ce P 974b802141bd4f13beffef2aa03f88f9 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 197b880fa39d432a9b472845876bfaf0 in current term 5: Already voted for candidate 974b802141bd4f13beffef2aa03f88f9 in this term.
I20260129 02:59:35.280700 20507 raft_consensus.cc:1275] T 76f713c3331540d2a5b76164103e9eb1 P 974b802141bd4f13beffef2aa03f88f9 [term 3 FOLLOWER]: Refusing update from remote peer 197b880fa39d432a9b472845876bfaf0: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
W20260129 02:59:35.388634 22360 log.cc:927] Time spent T 00000000000000000000000000000000 P a91e6696d0a44ee19bcf2489fd709c38: Append to log took a long time: real 0.099s user 0.001s sys 0.000s
W20260129 02:59:35.392114 21099 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.211.1:42027, user_credentials={real_user=slave}} blocked reactor thread for 88593.3us
I20260129 02:59:35.300216 20515 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "cb4727386a3e48aca5d71cc6c611c197" candidate_uuid: "197b880fa39d432a9b472845876bfaf0" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "974b802141bd4f13beffef2aa03f88f9"
I20260129 02:59:35.283022 22337 raft_consensus.cc:3060] T add8d4923d1a489e86882c3a25692b95 P 1c50de89a5a845668696dcb28c3badd6 [term 1 FOLLOWER]: Advancing to term 2
I20260129 02:59:35.403887 20508 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4e420f90884f4796997b5ea41e49e411" candidate_uuid: "197b880fa39d432a9b472845876bfaf0" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "974b802141bd4f13beffef2aa03f88f9"
W20260129 02:59:35.295956 20405 leader_election.cc:336] T c891d2bff4ce4568861bbc992e91fbb0 P 197b880fa39d432a9b472845876bfaf0 [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3:37735): Timed out: RequestConsensusVote RPC to 127.18.211.3:37735 timed out after 1.560s (SENT)
I20260129 02:59:35.271552 20516 tablet_service.cc:2066] Received LeaderStepDown RPC: tablet_id: "76f713c3331540d2a5b76164103e9eb1"
dest_uuid: "974b802141bd4f13beffef2aa03f88f9"
mode: GRACEFUL
new_leader_uuid: "1c50de89a5a845668696dcb28c3badd6"
from {username='slave'} at 127.0.0.1:52008
I20260129 02:59:35.419579 20508 raft_consensus.cc:3060] T 4e420f90884f4796997b5ea41e49e411 P 974b802141bd4f13beffef2aa03f88f9 [term 3 FOLLOWER]: Advancing to term 4
I20260129 02:59:35.412648 20511 raft_consensus.cc:1275] T c934d31efd3f4e5195605ddc354d14e1 P 974b802141bd4f13beffef2aa03f88f9 [term 3 FOLLOWER]: Refusing update from remote peer 197b880fa39d432a9b472845876bfaf0: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 3 index: 6. (index mismatch)
I20260129 02:59:35.389433 22115 raft_consensus.cc:2749] T a0fcc58d6fb6473cafbd9d34d5195269 P 1c50de89a5a845668696dcb28c3badd6 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260129 02:59:35.276302 22231 raft_consensus.cc:3060] T 1299723a4476420ab02fd8bfd7aa0645 P 197b880fa39d432a9b472845876bfaf0 [term 4 FOLLOWER]: Advancing to term 5
I20260129 02:59:35.412804 20514 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "8055232cf0004d40a85f3bf983fe00a9" candidate_uuid: "197b880fa39d432a9b472845876bfaf0" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "974b802141bd4f13beffef2aa03f88f9"
I20260129 02:59:35.383742 20322 catalog_manager.cc:5167] ChangeConfig:REMOVE_PEER RPC for tablet 1299723a4476420ab02fd8bfd7aa0645 with cas_config_opid_index 4: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260129 02:59:35.288897 20510 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "36ff33be666a4c82954731691dbcf327" candidate_uuid: "197b880fa39d432a9b472845876bfaf0" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "974b802141bd4f13beffef2aa03f88f9"
I20260129 02:59:35.289064 22336 raft_consensus.cc:3055] T fd8311b779af4e1fa1402b2a02a602f2 P 197b880fa39d432a9b472845876bfaf0 [term 2 LEADER]: Stepping down as leader of term 2
W20260129 02:59:35.416092 20554 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.211.4:40439, user_credentials={real_user=slave}} blocked reactor thread for 134608us
I20260129 02:59:35.288208 22243 leader_election.cc:290] T 18f30ebaa4174fd28c958e72d0cc7e24 P 1c50de89a5a845668696dcb28c3badd6 [CANDIDATE]: Term 3 election: Requested vote from peers 197b880fa39d432a9b472845876bfaf0 (127.18.211.1:42027), 7a9daf5d42a84615a6383cfec9a10608 (127.18.211.4:40439)
I20260129 02:59:35.424458 20405 leader_election.cc:304] T c891d2bff4ce4568861bbc992e91fbb0 P 197b880fa39d432a9b472845876bfaf0 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 197b880fa39d432a9b472845876bfaf0; no voters: 1c50de89a5a845668696dcb28c3badd6, 974b802141bd4f13beffef2aa03f88f9
I20260129 02:59:35.425015 20510 raft_consensus.cc:2393] T 36ff33be666a4c82954731691dbcf327 P 974b802141bd4f13beffef2aa03f88f9 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 197b880fa39d432a9b472845876bfaf0 in current term 2: Already voted for candidate 1c50de89a5a845668696dcb28c3badd6 in this term.
I20260129 02:59:35.402448 22257 raft_consensus.cc:493] T 1299723a4476420ab02fd8bfd7aa0645 P 7a9daf5d42a84615a6383cfec9a10608 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260129 02:59:35.418980 20512 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a8479d1fe0bd41e5946186e3a6ca8c7a" candidate_uuid: "197b880fa39d432a9b472845876bfaf0" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "974b802141bd4f13beffef2aa03f88f9"
I20260129 02:59:35.387724 22308 leader_election.cc:290] T 913fc7d751ed4b7388d889aac30a5f9b P 1c50de89a5a845668696dcb28c3badd6 [CANDIDATE]: Term 2 election: Requested vote from peers 197b880fa39d432a9b472845876bfaf0 (127.18.211.1:42027), 974b802141bd4f13beffef2aa03f88f9 (127.18.211.2:34517)
W20260129 02:59:35.384649 20553 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.211.1:42027, user_credentials={real_user=slave}} blocked reactor thread for 84990.1us
I20260129 02:59:35.420791 20516 raft_consensus.cc:606] T 76f713c3331540d2a5b76164103e9eb1 P 974b802141bd4f13beffef2aa03f88f9 [term 3 FOLLOWER]: Received request to transfer leadership to TS 1c50de89a5a845668696dcb28c3badd6
I20260129 02:59:35.436954 20516 raft_consensus.cc:614] T 76f713c3331540d2a5b76164103e9eb1 P 974b802141bd4f13beffef2aa03f88f9 [term 3 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260129 02:59:35.297174 22292 raft_consensus.cc:3060] T a8479d1fe0bd41e5946186e3a6ca8c7a P 1c50de89a5a845668696dcb28c3badd6 [term 2 FOLLOWER]: Advancing to term 3
I20260129 02:59:35.426828 20510 raft_consensus.cc:1240] T febffe4a8b394368b69b1d164308fb9a P 974b802141bd4f13beffef2aa03f88f9 [term 2 FOLLOWER]: Rejecting Update request from peer 197b880fa39d432a9b472845876bfaf0 for earlier term 1. Current term is 2. Ops: []
I20260129 02:59:35.429061 20440 raft_consensus.cc:1240] T 4e420f90884f4796997b5ea41e49e411 P 197b880fa39d432a9b472845876bfaf0 [term 4 FOLLOWER]: Rejecting Update request from peer 7a9daf5d42a84615a6383cfec9a10608 for earlier term 3. Current term is 4. Ops: []
I20260129 02:59:35.429178 20586 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3e2d15f8640b407195210c476e09df91" candidate_uuid: "7a9daf5d42a84615a6383cfec9a10608" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "1c50de89a5a845668696dcb28c3badd6" is_pre_election: true
W20260129 02:59:35.430657 20322 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.ChangeConfig -> {remote=127.18.211.1:42027, user_credentials={real_user=slave}} blocked reactor thread for 46981us
I20260129 02:59:35.428987 21127 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "18f30ebaa4174fd28c958e72d0cc7e24" candidate_uuid: "1c50de89a5a845668696dcb28c3badd6" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "7a9daf5d42a84615a6383cfec9a10608"
W20260129 02:59:35.431138 20481 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260129 02:59:35.431798 20515 raft_consensus.cc:1240] T c2633a6fc34c44579de348d208b0c01e P 974b802141bd4f13beffef2aa03f88f9 [term 2 FOLLOWER]: Rejecting Update request from peer 197b880fa39d432a9b472845876bfaf0 for earlier term 1. Current term is 2. Ops: []
I20260129 02:59:35.274799 22091 raft_consensus.cc:515] T 96609ac68bdf4993a12e1c23355ca013 P 197b880fa39d432a9b472845876bfaf0 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } attrs { replace: true } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } }
I20260129 02:59:35.271282 22351 consensus_queue.cc:1048] T a5a05fbcbbcf4916bbc01fd1a5b5e90d P 974b802141bd4f13beffef2aa03f88f9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 2, Time since last communication: 0.173s
I20260129 02:59:35.300323 22341 consensus_queue.cc:1048] T 75be4bd6bd45474795fd472eff2953ac P 1c50de89a5a845668696dcb28c3badd6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 3, Time since last communication: 0.123s
W20260129 02:59:35.432140 20556 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.211.2:34517, user_credentials={real_user=slave}} blocked reactor thread for 44967.1us
I20260129 02:59:35.432196 20509 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2ac659d732314e1288849f08fe5e074e" candidate_uuid: "197b880fa39d432a9b472845876bfaf0" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "974b802141bd4f13beffef2aa03f88f9"
I20260129 02:59:35.410888 22344 leader_election.cc:290] T 42d27a28d89b409e8807ec0efd97af69 P 1c50de89a5a845668696dcb28c3badd6 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 197b880fa39d432a9b472845876bfaf0 (127.18.211.1:42027), 7a9daf5d42a84615a6383cfec9a10608 (127.18.211.4:40439)
I20260129 02:59:35.414098 22338 raft_consensus.cc:2804] T 36ff33be666a4c82954731691dbcf327 P 1c50de89a5a845668696dcb28c3badd6 [term 2 FOLLOWER]: Leader election won for term 2
I20260129 02:59:35.443996 20432 raft_consensus.cc:1240] T 2e58e9a5a8aa4cecafb464addb4f35ed P 197b880fa39d432a9b472845876bfaf0 [term 3 FOLLOWER]: Rejecting Update request from peer 7a9daf5d42a84615a6383cfec9a10608 for earlier term 2. Current term is 3. Ops: []
W20260129 02:59:35.445992 20481 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260129 02:59:35.546733 20583 raft_consensus.cc:1275] T 75f45538dc704d9a8ec4ff8ad9f3ea5b P 1c50de89a5a845668696dcb28c3badd6 [term 2 FOLLOWER]: Refusing update from remote peer 7a9daf5d42a84615a6383cfec9a10608: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260129 02:59:35.547289 20432 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1299723a4476420ab02fd8bfd7aa0645" candidate_uuid: "7a9daf5d42a84615a6383cfec9a10608" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "197b880fa39d432a9b472845876bfaf0" is_pre_election: true
I20260129 02:59:35.448105 20508 raft_consensus.cc:2468] T 4e420f90884f4796997b5ea41e49e411 P 974b802141bd4f13beffef2aa03f88f9 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 197b880fa39d432a9b472845876bfaf0 in term 4.
I20260129 02:59:35.548153 22091 leader_election.cc:290] T 96609ac68bdf4993a12e1c23355ca013 P 197b880fa39d432a9b472845876bfaf0 [CANDIDATE]: Term 4 election: Requested vote from peers 974b802141bd4f13beffef2aa03f88f9 (127.18.211.2:34517), 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3:37735), 7a9daf5d42a84615a6383cfec9a10608 (127.18.211.4:40439)
I20260129 02:59:35.412986 22061 consensus_queue.cc:237] T 1ce4ddf5b10e4ae8b193b13b2e5936ee P 1c50de89a5a845668696dcb28c3badd6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 2.2, Last appended by leader: 2, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:59:35.560140 22120 raft_consensus.cc:515] T 0678b8b468bd43cdbc3e485900844821 P 197b880fa39d432a9b472845876bfaf0 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } }
I20260129 02:59:35.562371 22120 leader_election.cc:290] T 0678b8b468bd43cdbc3e485900844821 P 197b880fa39d432a9b472845876bfaf0 [CANDIDATE]: Term 5 election: Requested vote from peers 974b802141bd4f13beffef2aa03f88f9 (127.18.211.2:34517), 7a9daf5d42a84615a6383cfec9a10608 (127.18.211.4:40439)
I20260129 02:59:35.450485 22354 raft_consensus.cc:2749] T c891d2bff4ce4568861bbc992e91fbb0 P 197b880fa39d432a9b472845876bfaf0 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260129 02:59:35.450802 20587 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "45cc5f3d2688486d9b6dbe7e666dedde" candidate_uuid: "7a9daf5d42a84615a6383cfec9a10608" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "1c50de89a5a845668696dcb28c3badd6" is_pre_election: true
I20260129 02:59:35.565119 20434 raft_consensus.cc:1240] T 96609ac68bdf4993a12e1c23355ca013 P 197b880fa39d432a9b472845876bfaf0 [term 4 FOLLOWER]: Rejecting Update request from peer 974b802141bd4f13beffef2aa03f88f9 for earlier term 3. Current term is 4. Ops: []
I20260129 02:59:35.451455 20586 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "505580c93c694bbb8ea96fdf9b17ab46" candidate_uuid: "7a9daf5d42a84615a6383cfec9a10608" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "1c50de89a5a845668696dcb28c3badd6" is_pre_election: true
I20260129 02:59:35.476830 22349 raft_consensus.cc:697] T 7a9a69cd82724b1faed1d10aa091fc4a P 1c50de89a5a845668696dcb28c3badd6 [term 4 LEADER]: Becoming Leader. State: Replica: 1c50de89a5a845668696dcb28c3badd6, State: Running, Role: LEADER
I20260129 02:59:35.443569 22257 raft_consensus.cc:515] T 1299723a4476420ab02fd8bfd7aa0645 P 7a9daf5d42a84615a6383cfec9a10608 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } }
I20260129 02:59:35.566983 22349 consensus_queue.cc:237] T 7a9a69cd82724b1faed1d10aa091fc4a P 1c50de89a5a845668696dcb28c3badd6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:59:35.454725 20435 raft_consensus.cc:3055] T 3906b080a5b34a74b8b64128635e8997 P 197b880fa39d432a9b472845876bfaf0 [term 1 LEADER]: Stepping down as leader of term 1
I20260129 02:59:35.568130 20435 raft_consensus.cc:740] T 3906b080a5b34a74b8b64128635e8997 P 197b880fa39d432a9b472845876bfaf0 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 197b880fa39d432a9b472845876bfaf0, State: Running, Role: LEADER
W20260129 02:59:35.159281 22324 consensus_queue.cc:1175] T a4c1226b987b468fa84dfc6ba0e2dbe1 P 197b880fa39d432a9b472845876bfaf0 [NON_LEADER]: Queue is closed or peer was untracked, disregarding peer response. Response: responder_uuid: "1c50de89a5a845668696dcb28c3badd6" responder_term: 3 status { last_received { term: 3 index: 2 } last_committed_idx: 1 error { code: INVALID_TERM status { code: ILLEGAL_STATE message: "Rejecting Update request from peer 197b880fa39d432a9b472845876bfaf0 for earlier term 2. Current term is 3. Ops: [2.2-2.2]" } } last_received_current_leader { term: 0 index: 0 } }
I20260129 02:59:35.460234 21128 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "42d27a28d89b409e8807ec0efd97af69" candidate_uuid: "1c50de89a5a845668696dcb28c3badd6" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "7a9daf5d42a84615a6383cfec9a10608" is_pre_election: true
W20260129 02:59:35.461319 20480 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.211.3:37735, user_credentials={real_user=slave}} blocked reactor thread for 52718.7us
I20260129 02:59:35.461391 20438 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "96609ac68bdf4993a12e1c23355ca013" candidate_uuid: "7a9daf5d42a84615a6383cfec9a10608" candidate_term: 4 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "197b880fa39d432a9b472845876bfaf0"
I20260129 02:59:35.272670 22326 raft_consensus.cc:515] T 926a250fa5834e88b96ccecbefbf5086 P 197b880fa39d432a9b472845876bfaf0 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:59:35.405966 22343 consensus_queue.cc:1059] T 4e3805ffa70242e8ade66da260dddf66 P 7a9daf5d42a84615a6383cfec9a10608 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.027s
I20260129 02:59:35.422513 22302 raft_consensus.cc:3060] T a8479d1fe0bd41e5946186e3a6ca8c7a P 974b802141bd4f13beffef2aa03f88f9 [term 2 FOLLOWER]: Advancing to term 3
I20260129 02:59:35.275938 22322 consensus_queue.cc:260] T c2633a6fc34c44579de348d208b0c01e P 197b880fa39d432a9b472845876bfaf0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:59:35.287338 22198 raft_consensus.cc:515] T e33a417cf9c54cf5af73a4ee3aac4d3a P 197b880fa39d432a9b472845876bfaf0 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
W20260129 02:59:35.292325 22233 consensus_queue.cc:1175] T 1299723a4476420ab02fd8bfd7aa0645 P 197b880fa39d432a9b472845876bfaf0 [NON_LEADER]: Queue is closed or peer was untracked, disregarding peer response. Response: responder_uuid: "7a9daf5d42a84615a6383cfec9a10608" responder_term: 5 status { last_received { term: 4 index: 6 } last_committed_idx: 4 error { code: INVALID_TERM status { code: ILLEGAL_STATE message: "Rejecting Update request from peer 197b880fa39d432a9b472845876bfaf0 for earlier term 4. Current term is 5. Ops: []" } } last_received_current_leader { term: 0 index: 0 } }
I20260129 02:59:35.588325 20511 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0678b8b468bd43cdbc3e485900844821" candidate_uuid: "197b880fa39d432a9b472845876bfaf0" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "974b802141bd4f13beffef2aa03f88f9"
I20260129 02:59:35.434088 22336 raft_consensus.cc:740] T fd8311b779af4e1fa1402b2a02a602f2 P 197b880fa39d432a9b472845876bfaf0 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 197b880fa39d432a9b472845876bfaf0, State: Running, Role: LEADER
I20260129 02:59:35.414242 22319 leader_election.cc:290] T 4408006eb5174de1896b037cf3f18457 P 1c50de89a5a845668696dcb28c3badd6 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 197b880fa39d432a9b472845876bfaf0 (127.18.211.1:42027), 974b802141bd4f13beffef2aa03f88f9 (127.18.211.2:34517)
I20260129 02:59:35.588958 20511 raft_consensus.cc:3060] T 0678b8b468bd43cdbc3e485900844821 P 974b802141bd4f13beffef2aa03f88f9 [term 4 FOLLOWER]: Advancing to term 5
I20260129 02:59:35.586953 20407 leader_election.cc:304] T 0f20af7fbe5d4db89a6f17948d98912c P 197b880fa39d432a9b472845876bfaf0 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 197b880fa39d432a9b472845876bfaf0, 974b802141bd4f13beffef2aa03f88f9; no voters:
I20260129 02:59:35.280908 22112 consensus_queue.cc:237] T 4b1876f9a70b49edb3aa74105e66d3b3 P 974b802141bd4f13beffef2aa03f88f9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
W20260129 02:59:35.529346 20405 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.211.3:37735, user_credentials={real_user=slave}} blocked reactor thread for 233396us
I20260129 02:59:35.531581 20437 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "febffe4a8b394368b69b1d164308fb9a" candidate_uuid: "7a9daf5d42a84615a6383cfec9a10608" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "197b880fa39d432a9b472845876bfaf0"
W20260129 02:59:35.532192 20553 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.211.1:42027, user_credentials={real_user=slave}} blocked reactor thread for 95414us
I20260129 02:59:35.528805 20433 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00e79d8f25a34676b1d866eb2068a2b2" candidate_uuid: "7a9daf5d42a84615a6383cfec9a10608" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "197b880fa39d432a9b472845876bfaf0"
W20260129 02:59:35.546072 21101 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260129 02:59:35.453218 22212 raft_consensus.cc:3055] T 4e3805ffa70242e8ade66da260dddf66 P 7a9daf5d42a84615a6383cfec9a10608 [term 2 LEADER]: Stepping down as leader of term 2
I20260129 02:59:35.569187 20508 raft_consensus.cc:1240] T 3ccac82a01a2445fa5b0ce6c0fcf2295 P 974b802141bd4f13beffef2aa03f88f9 [term 2 FOLLOWER]: Rejecting Update request from peer 197b880fa39d432a9b472845876bfaf0 for earlier term 1. Current term is 2. Ops: []
W20260129 02:59:35.575732 20556 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.211.2:34517, user_credentials={real_user=slave}} blocked reactor thread for 114370us
I20260129 02:59:35.578323 22300 consensus_queue.cc:1059] T e33a417cf9c54cf5af73a4ee3aac4d3a P 1c50de89a5a845668696dcb28c3badd6 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 }, Status: INVALID_TERM, Last received: 3.2, Next index: 3, Last known committed idx: 1, Time since last communication: 0.134s
I20260129 02:59:35.578840 20515 raft_consensus.cc:1240] T 7a9a69cd82724b1faed1d10aa091fc4a P 974b802141bd4f13beffef2aa03f88f9 [term 4 FOLLOWER]: Rejecting Update request from peer 197b880fa39d432a9b472845876bfaf0 for earlier term 3. Current term is 4. Ops: []
I20260129 02:59:35.595331 20516 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fcddab6bca684adf8ea032289a00f90e" candidate_uuid: "197b880fa39d432a9b472845876bfaf0" candidate_term: 7 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "974b802141bd4f13beffef2aa03f88f9"
I20260129 02:59:35.595911 20516 raft_consensus.cc:3060] T fcddab6bca684adf8ea032289a00f90e P 974b802141bd4f13beffef2aa03f88f9 [term 6 FOLLOWER]: Advancing to term 7
I20260129 02:59:35.580770 22338 raft_consensus.cc:697] T 36ff33be666a4c82954731691dbcf327 P 1c50de89a5a845668696dcb28c3badd6 [term 2 LEADER]: Becoming Leader. State: Replica: 1c50de89a5a845668696dcb28c3badd6, State: Running, Role: LEADER
I20260129 02:59:35.580492 20435 consensus_queue.cc:260] T 3906b080a5b34a74b8b64128635e8997 P 197b880fa39d432a9b472845876bfaf0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:59:35.581337 22257 leader_election.cc:290] T 1299723a4476420ab02fd8bfd7aa0645 P 7a9daf5d42a84615a6383cfec9a10608 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 197b880fa39d432a9b472845876bfaf0 (127.18.211.1:42027), 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3:37735)
W20260129 02:59:35.578141 21100 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.211.3:37735, user_credentials={real_user=slave}} blocked reactor thread for 44570us
I20260129 02:59:35.470044 20514 raft_consensus.cc:2468] T 8055232cf0004d40a85f3bf983fe00a9 P 974b802141bd4f13beffef2aa03f88f9 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 197b880fa39d432a9b472845876bfaf0 in term 2.
I20260129 02:59:35.584318 20510 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "96609ac68bdf4993a12e1c23355ca013" candidate_uuid: "197b880fa39d432a9b472845876bfaf0" candidate_term: 4 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "974b802141bd4f13beffef2aa03f88f9"
W20260129 02:59:35.585569 21099 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.211.1:42027, user_credentials={real_user=slave}} blocked reactor thread for 124990us
I20260129 02:59:35.528470 20441 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fd8311b779af4e1fa1402b2a02a602f2" candidate_uuid: "7a9daf5d42a84615a6383cfec9a10608" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "197b880fa39d432a9b472845876bfaf0"
I20260129 02:59:35.580320 22213 raft_consensus.cc:493] T b189f08e16a74a11980cc55d99359a0a P 1c50de89a5a845668696dcb28c3badd6 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 197b880fa39d432a9b472845876bfaf0)
I20260129 02:59:35.640821 20510 tablet_service.cc:2066] Received LeaderStepDown RPC: tablet_id: "8055232cf0004d40a85f3bf983fe00a9"
dest_uuid: "974b802141bd4f13beffef2aa03f88f9"
mode: GRACEFUL
new_leader_uuid: "197b880fa39d432a9b472845876bfaf0"
from {username='slave'} at 127.0.0.1:52008
I20260129 02:59:35.641235 22190 raft_consensus.cc:3055] T febffe4a8b394368b69b1d164308fb9a P 197b880fa39d432a9b472845876bfaf0 [term 1 LEADER]: Stepping down as leader of term 1
I20260129 02:59:35.641464 20510 raft_consensus.cc:606] T 8055232cf0004d40a85f3bf983fe00a9 P 974b802141bd4f13beffef2aa03f88f9 [term 2 FOLLOWER]: Received request to transfer leadership to TS 197b880fa39d432a9b472845876bfaf0
I20260129 02:59:35.641711 22190 raft_consensus.cc:740] T febffe4a8b394368b69b1d164308fb9a P 197b880fa39d432a9b472845876bfaf0 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 197b880fa39d432a9b472845876bfaf0, State: Running, Role: LEADER
I20260129 02:59:35.641938 20510 raft_consensus.cc:614] T 8055232cf0004d40a85f3bf983fe00a9 P 974b802141bd4f13beffef2aa03f88f9 [term 2 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260129 02:59:35.642443 22213 raft_consensus.cc:515] T b189f08e16a74a11980cc55d99359a0a P 1c50de89a5a845668696dcb28c3badd6 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:59:35.642642 22190 consensus_queue.cc:260] T febffe4a8b394368b69b1d164308fb9a P 197b880fa39d432a9b472845876bfaf0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } }
I20260129 02:59:35.645017 22213 leader_election.cc:290] T b189f08e16a74a11980cc55d99359a0a P 1c50de89a5a845668696dcb28c3badd6 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 197b880fa39d432a9b472845876bfaf0 (127.18.211.1:42027), 974b802141bd4f13beffef2aa03f88f9 (127.18.211.2:34517)
I20260129 02:59:35.652769 20515 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0340229d521b432580ae926f6062e3f7" candidate_uuid: "1c50de89a5a845668696dcb28c3badd6" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "974b802141bd4f13beffef2aa03f88f9"
I20260129 02:59:35.657033 20515 tablet_service.cc:2066] Received LeaderStepDown RPC: tablet_id: "cb4727386a3e48aca5d71cc6c611c197"
dest_uuid: "974b802141bd4f13beffef2aa03f88f9"
mode: GRACEFUL
new_leader_uuid: "197b880fa39d432a9b472845876bfaf0"
from {username='slave'} at 127.0.0.1:52008
I20260129 02:59:35.657723 20515 raft_consensus.cc:606] T cb4727386a3e48aca5d71cc6c611c197 P 974b802141bd4f13beffef2aa03f88f9 [term 1 LEADER]: Received request to transfer leadership to TS 197b880fa39d432a9b472845876bfaf0
I20260129 02:59:35.658995 22323 raft_consensus.cc:493] T 45cc5f3d2688486d9b6dbe7e666dedde P 974b802141bd4f13beffef2aa03f88f9 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 1c50de89a5a845668696dcb28c3badd6)
I20260129 02:59:35.660301 20586 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1299723a4476420ab02fd8bfd7aa0645" candidate_uuid: "7a9daf5d42a84615a6383cfec9a10608" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "1c50de89a5a845668696dcb28c3badd6" is_pre_election: true
I20260129 02:59:35.660578 22160 raft_consensus.cc:493] T cedce331e1584ac8b636c9b8a72b2ba0 P 974b802141bd4f13beffef2aa03f88f9 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 1c50de89a5a845668696dcb28c3badd6)
I20260129 02:59:35.661088 20586 raft_consensus.cc:2468] T 1299723a4476420ab02fd8bfd7aa0645 P 1c50de89a5a845668696dcb28c3badd6 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7a9daf5d42a84615a6383cfec9a10608 in term 5.
I20260129 02:59:35.661257 22160 raft_consensus.cc:515] T cedce331e1584ac8b636c9b8a72b2ba0 P 974b802141bd4f13beffef2aa03f88f9 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:59:35.662045 22353 consensus_queue.cc:1059] T 4b1876f9a70b49edb3aa74105e66d3b3 P 1c50de89a5a845668696dcb28c3badd6 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 }, Status: INVALID_TERM, Last received: 3.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.077s
I20260129 02:59:35.663450 22353 raft_consensus.cc:3055] T 4b1876f9a70b49edb3aa74105e66d3b3 P 1c50de89a5a845668696dcb28c3badd6 [term 3 LEADER]: Stepping down as leader of term 3
I20260129 02:59:35.663903 22353 raft_consensus.cc:740] T 4b1876f9a70b49edb3aa74105e66d3b3 P 1c50de89a5a845668696dcb28c3badd6 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 1c50de89a5a845668696dcb28c3badd6, State: Running, Role: LEADER
I20260129 02:59:35.663946 22160 leader_election.cc:290] T cedce331e1584ac8b636c9b8a72b2ba0 P 974b802141bd4f13beffef2aa03f88f9 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 197b880fa39d432a9b472845876bfaf0 (127.18.211.1:42027), 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3:37735)
I20260129 02:59:35.665076 22353 consensus_queue.cc:260] T 4b1876f9a70b49edb3aa74105e66d3b3 P 1c50de89a5a845668696dcb28c3badd6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:59:35.666582 22353 raft_consensus.cc:3060] T 4b1876f9a70b49edb3aa74105e66d3b3 P 1c50de89a5a845668696dcb28c3badd6 [term 3 FOLLOWER]: Advancing to term 4
I20260129 02:59:35.667064 20441 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "cedce331e1584ac8b636c9b8a72b2ba0" candidate_uuid: "974b802141bd4f13beffef2aa03f88f9" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "197b880fa39d432a9b472845876bfaf0" is_pre_election: true
I20260129 02:59:35.659673 22323 raft_consensus.cc:515] T 45cc5f3d2688486d9b6dbe7e666dedde P 974b802141bd4f13beffef2aa03f88f9 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } }
I20260129 02:59:35.670548 22323 leader_election.cc:290] T 45cc5f3d2688486d9b6dbe7e666dedde P 974b802141bd4f13beffef2aa03f88f9 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3:37735), 7a9daf5d42a84615a6383cfec9a10608 (127.18.211.4:40439)
I20260129 02:59:35.595009 22335 raft_consensus.cc:2764] T 0f20af7fbe5d4db89a6f17948d98912c P 197b880fa39d432a9b472845876bfaf0 [term 4 FOLLOWER]: Leader pre-election decision vote started in defunct term 3: won
W20260129 02:59:35.635605 20553 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.211.1:42027, user_credentials={real_user=slave}} blocked reactor thread for 43496.4us
I20260129 02:59:35.637305 20407 leader_election.cc:304] T a609ac02fae84725ab743f95f51d33ce P 197b880fa39d432a9b472845876bfaf0 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 197b880fa39d432a9b472845876bfaf0; no voters: 7a9daf5d42a84615a6383cfec9a10608, 974b802141bd4f13beffef2aa03f88f9
W20260129 02:59:35.673346 20407 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.211.2:34517, user_credentials={real_user=slave}} blocked reactor thread for 82915.8us
I20260129 02:59:35.673487 22326 leader_election.cc:290] T 926a250fa5834e88b96ccecbefbf5086 P 197b880fa39d432a9b472845876bfaf0 [CANDIDATE]: Term 5 election: Requested vote from peers 974b802141bd4f13beffef2aa03f88f9 (127.18.211.2:34517), 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3:37735)
I20260129 02:59:35.673961 22190 raft_consensus.cc:3060] T febffe4a8b394368b69b1d164308fb9a P 197b880fa39d432a9b472845876bfaf0 [term 1 FOLLOWER]: Advancing to term 2
I20260129 02:59:35.674082 22326 raft_consensus.cc:3060] T a609ac02fae84725ab743f95f51d33ce P 197b880fa39d432a9b472845876bfaf0 [term 4 FOLLOWER]: Advancing to term 5
W20260129 02:59:35.637998 20405 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.211.3:37735, user_credentials={real_user=slave}} blocked reactor thread for 44909.9us
I20260129 02:59:35.639007 21135 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0678b8b468bd43cdbc3e485900844821" candidate_uuid: "197b880fa39d432a9b472845876bfaf0" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "7a9daf5d42a84615a6383cfec9a10608"
I20260129 02:59:35.677992 20437 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "17d6431cc77a491e8464d68db4081a51" candidate_uuid: "1c50de89a5a845668696dcb28c3badd6" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "197b880fa39d432a9b472845876bfaf0"
W20260129 02:59:35.680617 20405 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260129 02:59:35.681061 20514 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "8055232cf0004d40a85f3bf983fe00a9" candidate_uuid: "1c50de89a5a845668696dcb28c3badd6" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "974b802141bd4f13beffef2aa03f88f9" is_pre_election: true
I20260129 02:59:35.681788 20514 raft_consensus.cc:2393] T 8055232cf0004d40a85f3bf983fe00a9 P 974b802141bd4f13beffef2aa03f88f9 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1c50de89a5a845668696dcb28c3badd6 in current term 2: Already voted for candidate 197b880fa39d432a9b472845876bfaf0 in this term.
I20260129 02:59:35.687422 20433 raft_consensus.cc:2468] T 00e79d8f25a34676b1d866eb2068a2b2 P 197b880fa39d432a9b472845876bfaf0 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7a9daf5d42a84615a6383cfec9a10608 in term 4.
W20260129 02:59:35.688994 20480 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.211.3:37735, user_credentials={real_user=slave}} blocked reactor thread for 49687.6us
I20260129 02:59:35.679088 20407 leader_election.cc:304] T 4e420f90884f4796997b5ea41e49e411 P 197b880fa39d432a9b472845876bfaf0 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 197b880fa39d432a9b472845876bfaf0, 974b802141bd4f13beffef2aa03f88f9; no voters: 7a9daf5d42a84615a6383cfec9a10608
I20260129 02:59:35.690639 20510 raft_consensus.cc:1240] T e7d83d4ccf684e76acb29a0dc844e1b8 P 974b802141bd4f13beffef2aa03f88f9 [term 3 FOLLOWER]: Rejecting Update request from peer 1c50de89a5a845668696dcb28c3badd6 for earlier term 2. Current term is 3. Ops: []
W20260129 02:59:35.697292 21099 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.211.1:42027, user_credentials={real_user=slave}} blocked reactor thread for 55612.4us
I20260129 02:59:35.699800 20438 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "18f30ebaa4174fd28c958e72d0cc7e24" candidate_uuid: "1c50de89a5a845668696dcb28c3badd6" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "197b880fa39d432a9b472845876bfaf0"
I20260129 02:59:35.593467 22212 raft_consensus.cc:740] T 4e3805ffa70242e8ade66da260dddf66 P 7a9daf5d42a84615a6383cfec9a10608 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 7a9daf5d42a84615a6383cfec9a10608, State: Running, Role: LEADER
I20260129 02:59:35.708904 21100 leader_election.cc:304] T 1299723a4476420ab02fd8bfd7aa0645 P 7a9daf5d42a84615a6383cfec9a10608 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1c50de89a5a845668696dcb28c3badd6, 7a9daf5d42a84615a6383cfec9a10608; no voters:
I20260129 02:59:35.710844 20514 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "17d6431cc77a491e8464d68db4081a51" candidate_uuid: "1c50de89a5a845668696dcb28c3badd6" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "974b802141bd4f13beffef2aa03f88f9"
I20260129 02:59:35.712061 20433 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "913fc7d751ed4b7388d889aac30a5f9b" candidate_uuid: "1c50de89a5a845668696dcb28c3badd6" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "197b880fa39d432a9b472845876bfaf0"
W20260129 02:59:35.713541 20482 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260129 02:59:35.677245 22354 consensus_queue.cc:1048] T 76f713c3331540d2a5b76164103e9eb1 P 197b880fa39d432a9b472845876bfaf0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
W20260129 02:59:35.707623 20556 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260129 02:59:35.715046 20437 raft_consensus.cc:1240] T fcddab6bca684adf8ea032289a00f90e P 197b880fa39d432a9b472845876bfaf0 [term 7 FOLLOWER]: Rejecting Update request from peer 1c50de89a5a845668696dcb28c3badd6 for earlier term 6. Current term is 7. Ops: [6.5-6.5]
I20260129 02:59:35.720530 20510 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "913fc7d751ed4b7388d889aac30a5f9b" candidate_uuid: "1c50de89a5a845668696dcb28c3badd6" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "974b802141bd4f13beffef2aa03f88f9"
I20260129 02:59:35.709318 20440 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "42d27a28d89b409e8807ec0efd97af69" candidate_uuid: "1c50de89a5a845668696dcb28c3badd6" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "197b880fa39d432a9b472845876bfaf0" is_pre_election: true
I20260129 02:59:35.772084 20510 raft_consensus.cc:3060] T 913fc7d751ed4b7388d889aac30a5f9b P 974b802141bd4f13beffef2aa03f88f9 [term 1 FOLLOWER]: Advancing to term 2
I20260129 02:59:35.775897 20507 raft_consensus.cc:1275] T 3906b080a5b34a74b8b64128635e8997 P 974b802141bd4f13beffef2aa03f88f9 [term 2 FOLLOWER]: Refusing update from remote peer 1c50de89a5a845668696dcb28c3badd6: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260129 02:59:35.781919 20441 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b189f08e16a74a11980cc55d99359a0a" candidate_uuid: "1c50de89a5a845668696dcb28c3badd6" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "197b880fa39d432a9b472845876bfaf0" is_pre_election: true
I20260129 02:59:35.780472 20553 leader_election.cc:304] T 18f30ebaa4174fd28c958e72d0cc7e24 P 1c50de89a5a845668696dcb28c3badd6 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1c50de89a5a845668696dcb28c3badd6; no voters: 197b880fa39d432a9b472845876bfaf0, 7a9daf5d42a84615a6383cfec9a10608
I20260129 02:59:35.603351 22296 consensus_queue.cc:1059] T 4f5a77e94fef4b16a2ba7b067c8b144a P 974b802141bd4f13beffef2aa03f88f9 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.110s
I20260129 02:59:35.781986 20513 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4408006eb5174de1896b037cf3f18457" candidate_uuid: "1c50de89a5a845668696dcb28c3badd6" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "974b802141bd4f13beffef2aa03f88f9" is_pre_election: true
I20260129 02:59:35.709988 22358 raft_consensus.cc:2804] T 4e420f90884f4796997b5ea41e49e411 P 197b880fa39d432a9b472845876bfaf0 [term 4 FOLLOWER]: Leader election won for term 4
I20260129 02:59:35.786731 20433 raft_consensus.cc:1275] T 4e3805ffa70242e8ade66da260dddf66 P 197b880fa39d432a9b472845876bfaf0 [term 3 FOLLOWER]: Refusing update from remote peer 1c50de89a5a845668696dcb28c3badd6: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 3 index: 6. (index mismatch)
I20260129 02:59:35.787575 20438 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4408006eb5174de1896b037cf3f18457" candidate_uuid: "1c50de89a5a845668696dcb28c3badd6" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "197b880fa39d432a9b472845876bfaf0" is_pre_election: true
W20260129 02:59:35.678318 22319 consensus_queue.cc:1175] T 4b1876f9a70b49edb3aa74105e66d3b3 P 1c50de89a5a845668696dcb28c3badd6 [NON_LEADER]: Queue is closed or peer was untracked, disregarding peer response. Response: responder_uuid: "197b880fa39d432a9b472845876bfaf0" responder_term: 4 status { last_received { term: 3 index: 3 } last_committed_idx: 3 error { code: INVALID_TERM status { code: ILLEGAL_STATE message: "Rejecting Update request from peer 1c50de89a5a845668696dcb28c3badd6 for earlier term 3. Current term is 4. Ops: []" } } last_received_current_leader { term: 0 index: 0 } }
I20260129 02:59:35.791309 22300 raft_consensus.cc:493] T e4d18c35b0a7493e85d0a4b92956a48f P 1c50de89a5a845668696dcb28c3badd6 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 974b802141bd4f13beffef2aa03f88f9)
I20260129 02:59:35.791550 22320 raft_consensus.cc:493] T cb4727386a3e48aca5d71cc6c611c197 P 1c50de89a5a845668696dcb28c3badd6 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 974b802141bd4f13beffef2aa03f88f9)
I20260129 02:59:35.793409 20553 leader_election.cc:304] T 42d27a28d89b409e8807ec0efd97af69 P 1c50de89a5a845668696dcb28c3badd6 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1c50de89a5a845668696dcb28c3badd6; no voters: 197b880fa39d432a9b472845876bfaf0, 7a9daf5d42a84615a6383cfec9a10608
I20260129 02:59:35.794063 22061 raft_consensus.cc:2749] T 18f30ebaa4174fd28c958e72d0cc7e24 P 1c50de89a5a845668696dcb28c3badd6 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260129 02:59:35.794909 22296 raft_consensus.cc:3055] T 4f5a77e94fef4b16a2ba7b067c8b144a P 974b802141bd4f13beffef2aa03f88f9 [term 1 LEADER]: Stepping down as leader of term 1
I20260129 02:59:35.699234 22120 consensus_queue.cc:1059] T c2633a6fc34c44579de348d208b0c01e P 197b880fa39d432a9b472845876bfaf0 [NON_LEADER]: Peer responded invalid term: Peer: permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.090s
W20260129 02:59:35.804183 20479 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260129 02:59:35.691932 22343 consensus_queue.cc:1059] T 18f30ebaa4174fd28c958e72d0cc7e24 P 7a9daf5d42a84615a6383cfec9a10608 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.111s
I20260129 02:59:35.812160 22300 raft_consensus.cc:515] T e4d18c35b0a7493e85d0a4b92956a48f P 1c50de89a5a845668696dcb28c3badd6 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:59:35.812295 22320 raft_consensus.cc:515] T cb4727386a3e48aca5d71cc6c611c197 P 1c50de89a5a845668696dcb28c3badd6 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:59:35.814950 22212 consensus_queue.cc:260] T 4e3805ffa70242e8ade66da260dddf66 P 7a9daf5d42a84615a6383cfec9a10608 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } }
I20260129 02:59:35.702706 22323 consensus_queue.cc:1059] T cb0b3f259ebc4310927338a8740e87b8 P 974b802141bd4f13beffef2aa03f88f9 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 }, Status: INVALID_TERM, Last received: 3.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.100s
I20260129 02:59:35.816052 22296 raft_consensus.cc:740] T 4f5a77e94fef4b16a2ba7b067c8b144a P 974b802141bd4f13beffef2aa03f88f9 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 974b802141bd4f13beffef2aa03f88f9, State: Running, Role: LEADER
I20260129 02:59:35.816560 22368 raft_consensus.cc:2804] T 1299723a4476420ab02fd8bfd7aa0645 P 7a9daf5d42a84615a6383cfec9a10608 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260129 02:59:35.817785 22137 raft_consensus.cc:3055] T cb4727386a3e48aca5d71cc6c611c197 P 974b802141bd4f13beffef2aa03f88f9 [term 1 LEADER]: Stepping down as leader of term 1
I20260129 02:59:35.821154 22213 raft_consensus.cc:3055] T e33a417cf9c54cf5af73a4ee3aac4d3a P 1c50de89a5a845668696dcb28c3badd6 [term 3 LEADER]: Stepping down as leader of term 3
I20260129 02:59:35.851132 20581 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a8479d1fe0bd41e5946186e3a6ca8c7a" candidate_uuid: "197b880fa39d432a9b472845876bfaf0" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "1c50de89a5a845668696dcb28c3badd6"
I20260129 02:59:35.851449 20585 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0f20af7fbe5d4db89a6f17948d98912c" candidate_uuid: "197b880fa39d432a9b472845876bfaf0" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "1c50de89a5a845668696dcb28c3badd6" is_pre_election: true
I20260129 02:59:35.804713 20434 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "8055232cf0004d40a85f3bf983fe00a9" candidate_uuid: "1c50de89a5a845668696dcb28c3badd6" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "197b880fa39d432a9b472845876bfaf0" is_pre_election: true
I20260129 02:59:35.822541 20436 raft_consensus.cc:1275] T a4c1226b987b468fa84dfc6ba0e2dbe1 P 197b880fa39d432a9b472845876bfaf0 [term 3 FOLLOWER]: Refusing update from remote peer 1c50de89a5a845668696dcb28c3badd6: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 2. (term mismatch)
I20260129 02:59:35.698649 22351 consensus_queue.cc:1059] T 96609ac68bdf4993a12e1c23355ca013 P 974b802141bd4f13beffef2aa03f88f9 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 }, Status: INVALID_TERM, Last received: 0.0, Next index: 5, Last known committed idx: 3, Time since last communication: 0.115s
I20260129 02:59:35.850936 22320 leader_election.cc:290] T cb4727386a3e48aca5d71cc6c611c197 P 1c50de89a5a845668696dcb28c3badd6 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 197b880fa39d432a9b472845876bfaf0 (127.18.211.1:42027), 974b802141bd4f13beffef2aa03f88f9 (127.18.211.2:34517)
I20260129 02:59:35.698791 22082 consensus_queue.cc:1048] T c934d31efd3f4e5195605ddc354d14e1 P 197b880fa39d432a9b472845876bfaf0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.116s
W20260129 02:59:35.853295 21130 consensus_meta.cc:287] T 2ac659d732314e1288849f08fe5e074e P 7a9daf5d42a84615a6383cfec9a10608: Time spent flushing consensus metadata: real 0.606s user 0.006s sys 0.004s
I20260129 02:59:35.833391 20480 leader_election.cc:304] T b189f08e16a74a11980cc55d99359a0a P 974b802141bd4f13beffef2aa03f88f9 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1c50de89a5a845668696dcb28c3badd6, 974b802141bd4f13beffef2aa03f88f9; no voters: 197b880fa39d432a9b472845876bfaf0
I20260129 02:59:35.834466 20405 leader_election.cc:304] T 2e58e9a5a8aa4cecafb464addb4f35ed P 197b880fa39d432a9b472845876bfaf0 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 197b880fa39d432a9b472845876bfaf0; no voters: 1c50de89a5a845668696dcb28c3badd6, 7a9daf5d42a84615a6383cfec9a10608
I20260129 02:59:35.834887 20514 raft_consensus.cc:1275] T c2633a6fc34c44579de348d208b0c01e P 974b802141bd4f13beffef2aa03f88f9 [term 2 FOLLOWER]: Refusing update from remote peer 1c50de89a5a845668696dcb28c3badd6: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260129 02:59:35.836752 20507 tablet_service.cc:2066] Received LeaderStepDown RPC: tablet_id: "e4d18c35b0a7493e85d0a4b92956a48f"
dest_uuid: "974b802141bd4f13beffef2aa03f88f9"
mode: GRACEFUL
new_leader_uuid: "1c50de89a5a845668696dcb28c3badd6"
from {username='slave'} at 127.0.0.1:52008
I20260129 02:59:35.868124 20507 raft_consensus.cc:606] T e4d18c35b0a7493e85d0a4b92956a48f P 974b802141bd4f13beffef2aa03f88f9 [term 3 FOLLOWER]: Received request to transfer leadership to TS 1c50de89a5a845668696dcb28c3badd6
I20260129 02:59:35.868577 20507 raft_consensus.cc:614] T e4d18c35b0a7493e85d0a4b92956a48f P 974b802141bd4f13beffef2aa03f88f9 [term 3 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260129 02:59:35.837556 20513 raft_consensus.cc:1275] T 0f20af7fbe5d4db89a6f17948d98912c P 974b802141bd4f13beffef2aa03f88f9 [term 4 FOLLOWER]: Refusing update from remote peer 1c50de89a5a845668696dcb28c3badd6: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20260129 02:59:35.842597 22077 raft_consensus.cc:493] T a1a39dcd328e4268ae41b988c66b4a75 P 197b880fa39d432a9b472845876bfaf0 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 1c50de89a5a845668696dcb28c3badd6)
I20260129 02:59:35.845793 22368 raft_consensus.cc:493] T 1299723a4476420ab02fd8bfd7aa0645 P 7a9daf5d42a84615a6383cfec9a10608 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260129 02:59:35.815259 22336 consensus_queue.cc:260] T fd8311b779af4e1fa1402b2a02a602f2 P 197b880fa39d432a9b472845876bfaf0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } }
I20260129 02:59:35.840204 20590 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "8055232cf0004d40a85f3bf983fe00a9" candidate_uuid: "197b880fa39d432a9b472845876bfaf0" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1c50de89a5a845668696dcb28c3badd6"
I20260129 02:59:35.878798 20590 raft_consensus.cc:3060] T 8055232cf0004d40a85f3bf983fe00a9 P 1c50de89a5a845668696dcb28c3badd6 [term 1 FOLLOWER]: Advancing to term 2
I20260129 02:59:35.878674 21136 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "96609ac68bdf4993a12e1c23355ca013" candidate_uuid: "197b880fa39d432a9b472845876bfaf0" candidate_term: 4 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "7a9daf5d42a84615a6383cfec9a10608"
I20260129 02:59:35.879505 21136 raft_consensus.cc:2393] T 96609ac68bdf4993a12e1c23355ca013 P 7a9daf5d42a84615a6383cfec9a10608 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 197b880fa39d432a9b472845876bfaf0 in current term 4: Already voted for candidate 7a9daf5d42a84615a6383cfec9a10608 in this term.
I20260129 02:59:35.848340 20582 raft_consensus.cc:1275] T 76f713c3331540d2a5b76164103e9eb1 P 1c50de89a5a845668696dcb28c3badd6 [term 3 FOLLOWER]: Refusing update from remote peer 197b880fa39d432a9b472845876bfaf0: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260129 02:59:35.838809 22061 consensus_queue.cc:1048] T 4e3805ffa70242e8ade66da260dddf66 P 1c50de89a5a845668696dcb28c3badd6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.011s
I20260129 02:59:35.847549 22137 raft_consensus.cc:740] T cb4727386a3e48aca5d71cc6c611c197 P 974b802141bd4f13beffef2aa03f88f9 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 974b802141bd4f13beffef2aa03f88f9, State: Running, Role: LEADER
I20260129 02:59:35.850538 20584 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "17d6431cc77a491e8464d68db4081a51" candidate_uuid: "197b880fa39d432a9b472845876bfaf0" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1c50de89a5a845668696dcb28c3badd6" is_pre_election: true
I20260129 02:59:35.886765 20584 raft_consensus.cc:2393] T 17d6431cc77a491e8464d68db4081a51 P 1c50de89a5a845668696dcb28c3badd6 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 197b880fa39d432a9b472845876bfaf0 in current term 2: Already voted for candidate 1c50de89a5a845668696dcb28c3badd6 in this term.
I20260129 02:59:35.853868 20583 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "cb4727386a3e48aca5d71cc6c611c197" candidate_uuid: "197b880fa39d432a9b472845876bfaf0" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1c50de89a5a845668696dcb28c3badd6"
I20260129 02:59:35.887341 20583 raft_consensus.cc:3060] T cb4727386a3e48aca5d71cc6c611c197 P 1c50de89a5a845668696dcb28c3badd6 [term 1 FOLLOWER]: Advancing to term 2
I20260129 02:59:35.837369 22344 raft_consensus.cc:2749] T 42d27a28d89b409e8807ec0efd97af69 P 1c50de89a5a845668696dcb28c3badd6 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260129 02:59:35.859313 21135 raft_consensus.cc:1275] T 505580c93c694bbb8ea96fdf9b17ab46 P 7a9daf5d42a84615a6383cfec9a10608 [term 3 FOLLOWER]: Refusing update from remote peer 1c50de89a5a845668696dcb28c3badd6: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 3 index: 6. (index mismatch)
I20260129 02:59:35.859247 20588 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e7d83d4ccf684e76acb29a0dc844e1b8" candidate_uuid: "197b880fa39d432a9b472845876bfaf0" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "1c50de89a5a845668696dcb28c3badd6" is_pre_election: true
I20260129 02:59:35.854566 20436 pending_rounds.cc:85] T a4c1226b987b468fa84dfc6ba0e2dbe1 P 197b880fa39d432a9b472845876bfaf0: Aborting all ops after (but not including) 1
I20260129 02:59:35.897840 20436 pending_rounds.cc:107] T a4c1226b987b468fa84dfc6ba0e2dbe1 P 197b880fa39d432a9b472845876bfaf0: Aborting uncommitted NO_OP operation due to leader change: 2.2
I20260129 02:59:35.898170 20436 raft_consensus.cc:2889] T a4c1226b987b468fa84dfc6ba0e2dbe1 P 197b880fa39d432a9b472845876bfaf0 [term 3 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260129 02:59:35.899164 22296 consensus_queue.cc:260] T 4f5a77e94fef4b16a2ba7b067c8b144a P 974b802141bd4f13beffef2aa03f88f9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } }
I20260129 02:59:35.886648 22077 raft_consensus.cc:515] T a1a39dcd328e4268ae41b988c66b4a75 P 197b880fa39d432a9b472845876bfaf0 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } } peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } }
I20260129 02:59:35.851382 22338 consensus_queue.cc:237] T 36ff33be666a4c82954731691dbcf327 P 1c50de89a5a845668696dcb28c3badd6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:59:35.900611 22296 raft_consensus.cc:3060] T 4f5a77e94fef4b16a2ba7b067c8b144a P 974b802141bd4f13beffef2aa03f88f9 [term 1 FOLLOWER]: Advancing to term 2
I20260129 02:59:35.901656 22077 leader_election.cc:290] T a1a39dcd328e4268ae41b988c66b4a75 P 197b880fa39d432a9b472845876bfaf0 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 974b802141bd4f13beffef2aa03f88f9 (127.18.211.2:34517), 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3:37735)
I20260129 02:59:35.902472 22300 leader_election.cc:290] T e4d18c35b0a7493e85d0a4b92956a48f P 1c50de89a5a845668696dcb28c3badd6 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 197b880fa39d432a9b472845876bfaf0 (127.18.211.1:42027), 974b802141bd4f13beffef2aa03f88f9 (127.18.211.2:34517)
I20260129 02:59:35.903535 22184 raft_consensus.cc:3055] T 96609ac68bdf4993a12e1c23355ca013 P 974b802141bd4f13beffef2aa03f88f9 [term 3 LEADER]: Stepping down as leader of term 3
I20260129 02:59:35.955386 20582 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "96609ac68bdf4993a12e1c23355ca013" candidate_uuid: "197b880fa39d432a9b472845876bfaf0" candidate_term: 4 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "1c50de89a5a845668696dcb28c3badd6"
I20260129 02:59:35.956540 20582 raft_consensus.cc:2393] T 96609ac68bdf4993a12e1c23355ca013 P 1c50de89a5a845668696dcb28c3badd6 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 197b880fa39d432a9b472845876bfaf0 in current term 4: Already voted for candidate 7a9daf5d42a84615a6383cfec9a10608 in this term.
I20260129 02:59:35.874442 20511 raft_consensus.cc:2468] T 0678b8b468bd43cdbc3e485900844821 P 974b802141bd4f13beffef2aa03f88f9 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 197b880fa39d432a9b472845876bfaf0 in term 5.
I20260129 02:59:35.813984 20440 raft_consensus.cc:1275] T 4f8f3a38f6e24ba4857875181f447442 P 197b880fa39d432a9b472845876bfaf0 [term 4 FOLLOWER]: Refusing update from remote peer 1c50de89a5a845668696dcb28c3badd6: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
W20260129 02:59:35.880445 20405 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.211.3:37735, user_credentials={real_user=slave}} blocked reactor thread for 46038us
I20260129 02:59:35.850023 20587 raft_consensus.cc:1240] T 0340229d521b432580ae926f6062e3f7 P 1c50de89a5a845668696dcb28c3badd6 [term 4 FOLLOWER]: Rejecting Update request from peer 197b880fa39d432a9b472845876bfaf0 for earlier term 3. Current term is 4. Ops: []
W20260129 02:59:35.883280 20480 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.211.3:37735, user_credentials={real_user=slave}} blocked reactor thread for 49991.8us
I20260129 02:59:35.883693 20585 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fcddab6bca684adf8ea032289a00f90e" candidate_uuid: "197b880fa39d432a9b472845876bfaf0" candidate_term: 7 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "1c50de89a5a845668696dcb28c3badd6"
I20260129 02:59:35.839006 22079 consensus_queue.cc:1059] T fcddab6bca684adf8ea032289a00f90e P 1c50de89a5a845668696dcb28c3badd6 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 }, Status: INVALID_TERM, Last received: 4.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.026s
I20260129 02:59:35.851593 22213 raft_consensus.cc:740] T e33a417cf9c54cf5af73a4ee3aac4d3a P 1c50de89a5a845668696dcb28c3badd6 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 1c50de89a5a845668696dcb28c3badd6, State: Running, Role: LEADER
I20260129 02:59:35.891597 20586 raft_consensus.cc:1240] T 7a9a69cd82724b1faed1d10aa091fc4a P 1c50de89a5a845668696dcb28c3badd6 [term 4 LEADER]: Rejecting Update request from peer 197b880fa39d432a9b472845876bfaf0 for earlier term 3. Current term is 4. Ops: []
I20260129 02:59:35.878127 22368 raft_consensus.cc:3060] T 1299723a4476420ab02fd8bfd7aa0645 P 7a9daf5d42a84615a6383cfec9a10608 [term 5 FOLLOWER]: Advancing to term 6
I20260129 02:59:35.894152 22355 raft_consensus.cc:2804] T b189f08e16a74a11980cc55d99359a0a P 974b802141bd4f13beffef2aa03f88f9 [term 4 FOLLOWER]: Leader pre-election won for term 5
W20260129 02:59:35.894742 20479 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260129 02:59:35.845302 22339 consensus_queue.cc:1059] T 3ccac82a01a2445fa5b0ce6c0fcf2295 P 197b880fa39d432a9b472845876bfaf0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.033s
I20260129 02:59:35.896063 19276 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 4
I20260129 02:59:35.904053 20432 raft_consensus.cc:2468] T 1299723a4476420ab02fd8bfd7aa0645 P 197b880fa39d432a9b472845876bfaf0 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7a9daf5d42a84615a6383cfec9a10608 in term 5.
I20260129 02:59:35.904321 20434 raft_consensus.cc:2393] T 8055232cf0004d40a85f3bf983fe00a9 P 197b880fa39d432a9b472845876bfaf0 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1c50de89a5a845668696dcb28c3badd6 in current term 2: Already voted for candidate 197b880fa39d432a9b472845876bfaf0 in this term.
I20260129 02:59:35.882200 22343 raft_consensus.cc:3055] T 18f30ebaa4174fd28c958e72d0cc7e24 P 7a9daf5d42a84615a6383cfec9a10608 [term 2 LEADER]: Stepping down as leader of term 2
I20260129 02:59:35.904466 22371 raft_consensus.cc:2749] T 2e58e9a5a8aa4cecafb464addb4f35ed P 197b880fa39d432a9b472845876bfaf0 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260129 02:59:35.871755 21130 raft_consensus.cc:2468] T 2ac659d732314e1288849f08fe5e074e P 7a9daf5d42a84615a6383cfec9a10608 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 197b880fa39d432a9b472845876bfaf0 in term 2.
I20260129 02:59:35.906064 20514 raft_consensus.cc:1275] T 505580c93c694bbb8ea96fdf9b17ab46 P 974b802141bd4f13beffef2aa03f88f9 [term 3 FOLLOWER]: Refusing update from remote peer 1c50de89a5a845668696dcb28c3badd6: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 3 index: 6. (index mismatch)
I20260129 02:59:35.871913 22323 raft_consensus.cc:3055] T cb0b3f259ebc4310927338a8740e87b8 P 974b802141bd4f13beffef2aa03f88f9 [term 3 LEADER]: Stepping down as leader of term 3
I20260129 02:59:35.904075 22198 leader_election.cc:290] T e33a417cf9c54cf5af73a4ee3aac4d3a P 197b880fa39d432a9b472845876bfaf0 [CANDIDATE]: Term 4 election: Requested vote from peers 974b802141bd4f13beffef2aa03f88f9 (127.18.211.2:34517), 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3:37735)
I20260129 02:59:35.907032 20507 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b189f08e16a74a11980cc55d99359a0a" candidate_uuid: "1c50de89a5a845668696dcb28c3badd6" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "974b802141bd4f13beffef2aa03f88f9" is_pre_election: true
I20260129 02:59:35.908214 20439 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "cb4727386a3e48aca5d71cc6c611c197" candidate_uuid: "1c50de89a5a845668696dcb28c3badd6" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "197b880fa39d432a9b472845876bfaf0" is_pre_election: true
I20260129 02:59:35.908896 22326 raft_consensus.cc:2749] T a609ac02fae84725ab743f95f51d33ce P 197b880fa39d432a9b472845876bfaf0 [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260129 02:59:35.909348 22300 consensus_queue.cc:1048] T 505580c93c694bbb8ea96fdf9b17ab46 P 1c50de89a5a845668696dcb28c3badd6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.002s
I20260129 02:59:35.908998 22322 raft_consensus.cc:3060] T c2633a6fc34c44579de348d208b0c01e P 197b880fa39d432a9b472845876bfaf0 [term 1 FOLLOWER]: Advancing to term 2
I20260129 02:59:35.909680 22137 consensus_queue.cc:260] T cb4727386a3e48aca5d71cc6c611c197 P 974b802141bd4f13beffef2aa03f88f9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:59:35.914667 20438 raft_consensus.cc:1275] T 0f20af7fbe5d4db89a6f17948d98912c P 197b880fa39d432a9b472845876bfaf0 [term 4 FOLLOWER]: Refusing update from remote peer 1c50de89a5a845668696dcb28c3badd6: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20260129 02:59:35.871755 20516 raft_consensus.cc:2468] T fcddab6bca684adf8ea032289a00f90e P 974b802141bd4f13beffef2aa03f88f9 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 197b880fa39d432a9b472845876bfaf0 in term 7.
W20260129 02:59:35.919631 21099 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.211.1:42027, user_credentials={real_user=slave}} blocked reactor thread for 210414us
I20260129 02:59:36.036816 20515 raft_consensus.cc:1240] T fcddab6bca684adf8ea032289a00f90e P 974b802141bd4f13beffef2aa03f88f9 [term 7 FOLLOWER]: Rejecting Update request from peer 1c50de89a5a845668696dcb28c3badd6 for earlier term 6. Current term is 7. Ops: [6.5-6.5]
I20260129 02:59:35.921702 20556 leader_election.cc:304] T 0340229d521b432580ae926f6062e3f7 P 1c50de89a5a845668696dcb28c3badd6 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1c50de89a5a845668696dcb28c3badd6; no voters: 197b880fa39d432a9b472845876bfaf0, 974b802141bd4f13beffef2aa03f88f9
I20260129 02:59:35.940899 20512 raft_consensus.cc:2468] T a8479d1fe0bd41e5946186e3a6ca8c7a P 974b802141bd4f13beffef2aa03f88f9 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 197b880fa39d432a9b472845876bfaf0 in term 3.
W20260129 02:59:36.038327 20556 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.211.2:34517, user_credentials={real_user=slave}} blocked reactor thread for 116679us
I20260129 02:59:35.960983 20480 leader_election.cc:304] T a0fcc58d6fb6473cafbd9d34d5195269 P 974b802141bd4f13beffef2aa03f88f9 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1c50de89a5a845668696dcb28c3badd6, 974b802141bd4f13beffef2aa03f88f9; no voters: 197b880fa39d432a9b472845876bfaf0
I20260129 02:59:35.961524 20511 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "cb4727386a3e48aca5d71cc6c611c197" candidate_uuid: "1c50de89a5a845668696dcb28c3badd6" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "974b802141bd4f13beffef2aa03f88f9" is_pre_election: true
I20260129 02:59:35.967011 20588 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4b1876f9a70b49edb3aa74105e66d3b3" candidate_uuid: "197b880fa39d432a9b472845876bfaf0" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "1c50de89a5a845668696dcb28c3badd6" is_pre_election: true
I20260129 02:59:36.040443 20588 raft_consensus.cc:2468] T 4b1876f9a70b49edb3aa74105e66d3b3 P 1c50de89a5a845668696dcb28c3badd6 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 197b880fa39d432a9b472845876bfaf0 in term 4.
W20260129 02:59:35.965770 20407 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.211.2:34517, user_credentials={real_user=slave}} blocked reactor thread for 234334us
I20260129 02:59:35.967440 20405 leader_election.cc:304] T 4f8f3a38f6e24ba4857875181f447442 P 197b880fa39d432a9b472845876bfaf0 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 197b880fa39d432a9b472845876bfaf0; no voters: 1c50de89a5a845668696dcb28c3badd6, 974b802141bd4f13beffef2aa03f88f9
I20260129 02:59:35.962637 20583 raft_consensus.cc:2468] T cb4727386a3e48aca5d71cc6c611c197 P 1c50de89a5a845668696dcb28c3badd6 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 197b880fa39d432a9b472845876bfaf0 in term 2.
I20260129 02:59:35.976562 20584 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "926a250fa5834e88b96ccecbefbf5086" candidate_uuid: "197b880fa39d432a9b472845876bfaf0" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "1c50de89a5a845668696dcb28c3badd6"
I20260129 02:59:35.982605 20585 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a5a05fbcbbcf4916bbc01fd1a5b5e90d" candidate_uuid: "197b880fa39d432a9b472845876bfaf0" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "1c50de89a5a845668696dcb28c3badd6" is_pre_election: true
I20260129 02:59:36.039368 20556 leader_election.cc:304] T 8055232cf0004d40a85f3bf983fe00a9 P 1c50de89a5a845668696dcb28c3badd6 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1c50de89a5a845668696dcb28c3badd6; no voters: 197b880fa39d432a9b472845876bfaf0, 974b802141bd4f13beffef2aa03f88f9
I20260129 02:59:35.912690 22292 raft_consensus.cc:2749] T a8479d1fe0bd41e5946186e3a6ca8c7a P 1c50de89a5a845668696dcb28c3badd6 [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260129 02:59:35.985044 20440 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e4d18c35b0a7493e85d0a4b92956a48f" candidate_uuid: "1c50de89a5a845668696dcb28c3badd6" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "197b880fa39d432a9b472845876bfaf0" is_pre_election: true
I20260129 02:59:36.043656 20440 raft_consensus.cc:2393] T e4d18c35b0a7493e85d0a4b92956a48f P 197b880fa39d432a9b472845876bfaf0 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1c50de89a5a845668696dcb28c3badd6 in current term 3: Already voted for candidate 197b880fa39d432a9b472845876bfaf0 in this term.
W20260129 02:59:36.044373 20480 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.211.3:37735, user_credentials={real_user=slave}} blocked reactor thread for 83479.6us
W20260129 02:59:35.985925 21102 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260129 02:59:35.987522 20587 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4408006eb5174de1896b037cf3f18457" candidate_uuid: "197b880fa39d432a9b472845876bfaf0" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1c50de89a5a845668696dcb28c3badd6" is_pre_election: true
I20260129 02:59:35.979517 19276 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20260129 02:59:35.954790 22184 raft_consensus.cc:740] T 96609ac68bdf4993a12e1c23355ca013 P 974b802141bd4f13beffef2aa03f88f9 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 974b802141bd4f13beffef2aa03f88f9, State: Running, Role: LEADER
I20260129 02:59:35.962442 22231 raft_consensus.cc:493] T 18f30ebaa4174fd28c958e72d0cc7e24 P 197b880fa39d432a9b472845876bfaf0 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 7a9daf5d42a84615a6383cfec9a10608)
I20260129 02:59:35.969050 22164 raft_consensus.cc:493] T 76f713c3331540d2a5b76164103e9eb1 P 1c50de89a5a845668696dcb28c3badd6 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260129 02:59:36.003996 20513 raft_consensus.cc:1275] T a4c1226b987b468fa84dfc6ba0e2dbe1 P 974b802141bd4f13beffef2aa03f88f9 [term 3 FOLLOWER]: Refusing update from remote peer 1c50de89a5a845668696dcb28c3badd6: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20260129 02:59:36.012658 20590 raft_consensus.cc:2468] T 8055232cf0004d40a85f3bf983fe00a9 P 1c50de89a5a845668696dcb28c3badd6 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 197b880fa39d432a9b472845876bfaf0 in term 2.
I20260129 02:59:35.989414 22350 raft_consensus.cc:493] T add8d4923d1a489e86882c3a25692b95 P 7a9daf5d42a84615a6383cfec9a10608 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 1c50de89a5a845668696dcb28c3badd6)
I20260129 02:59:36.015239 22323 raft_consensus.cc:740] T cb0b3f259ebc4310927338a8740e87b8 P 974b802141bd4f13beffef2aa03f88f9 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 974b802141bd4f13beffef2aa03f88f9, State: Running, Role: LEADER
I20260129 02:59:36.018934 20582 raft_consensus.cc:1240] T 1299723a4476420ab02fd8bfd7aa0645 P 1c50de89a5a845668696dcb28c3badd6 [term 5 FOLLOWER]: Rejecting Update request from peer 197b880fa39d432a9b472845876bfaf0 for earlier term 4. Current term is 5. Ops: [4.6-4.6]
I20260129 02:59:36.024271 20439 raft_consensus.cc:2393] T cb4727386a3e48aca5d71cc6c611c197 P 197b880fa39d432a9b472845876bfaf0 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1c50de89a5a845668696dcb28c3badd6 in current term 2: Already voted for candidate 197b880fa39d432a9b472845876bfaf0 in this term.
I20260129 02:59:35.973091 22355 raft_consensus.cc:493] T b189f08e16a74a11980cc55d99359a0a P 974b802141bd4f13beffef2aa03f88f9 [term 4 FOLLOWER]: Starting leader election (detected failure of leader 197b880fa39d432a9b472845876bfaf0)
I20260129 02:59:36.045449 20587 raft_consensus.cc:2468] T 4408006eb5174de1896b037cf3f18457 P 1c50de89a5a845668696dcb28c3badd6 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 197b880fa39d432a9b472845876bfaf0 in term 1.
I20260129 02:59:35.998903 22344 consensus_queue.cc:1048] T 4f8f3a38f6e24ba4857875181f447442 P 1c50de89a5a845668696dcb28c3badd6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.060s
I20260129 02:59:35.985605 20509 raft_consensus.cc:2468] T 2ac659d732314e1288849f08fe5e074e P 974b802141bd4f13beffef2aa03f88f9 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 197b880fa39d432a9b472845876bfaf0 in term 2.
I20260129 02:59:36.049563 20514 raft_consensus.cc:1240] T e33a417cf9c54cf5af73a4ee3aac4d3a P 974b802141bd4f13beffef2aa03f88f9 [term 4 FOLLOWER]: Rejecting Update request from peer 1c50de89a5a845668696dcb28c3badd6 for earlier term 3. Current term is 4. Ops: []
I20260129 02:59:36.042601 20585 raft_consensus.cc:2468] T a5a05fbcbbcf4916bbc01fd1a5b5e90d P 1c50de89a5a845668696dcb28c3badd6 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 197b880fa39d432a9b472845876bfaf0 in term 5.
I20260129 02:59:36.053577 20583 raft_consensus.cc:1240] T 4e3805ffa70242e8ade66da260dddf66 P 1c50de89a5a845668696dcb28c3badd6 [term 3 LEADER]: Rejecting Update request from peer 7a9daf5d42a84615a6383cfec9a10608 for earlier term 2. Current term is 3. Ops: []
I20260129 02:59:36.054801 20556 leader_election.cc:304] T 17d6431cc77a491e8464d68db4081a51 P 1c50de89a5a845668696dcb28c3badd6 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1c50de89a5a845668696dcb28c3badd6; no voters: 197b880fa39d432a9b472845876bfaf0, 974b802141bd4f13beffef2aa03f88f9
I20260129 02:59:35.999006 22366 consensus_queue.cc:1048] T 75f45538dc704d9a8ec4ff8ad9f3ea5b P 7a9daf5d42a84615a6383cfec9a10608 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.236s
I20260129 02:59:36.058173 21135 raft_consensus.cc:1240] T cb0b3f259ebc4310927338a8740e87b8 P 7a9daf5d42a84615a6383cfec9a10608 [term 4 FOLLOWER]: Rejecting Update request from peer 974b802141bd4f13beffef2aa03f88f9 for earlier term 3. Current term is 4. Ops: []
I20260129 02:59:36.065601 21136 raft_consensus.cc:1240] T 4f5a77e94fef4b16a2ba7b067c8b144a P 7a9daf5d42a84615a6383cfec9a10608 [term 2 FOLLOWER]: Rejecting Update request from peer 974b802141bd4f13beffef2aa03f88f9 for earlier term 1. Current term is 2. Ops: []
I20260129 02:59:36.066208 20590 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e7d83d4ccf684e76acb29a0dc844e1b8" candidate_uuid: "974b802141bd4f13beffef2aa03f88f9" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "1c50de89a5a845668696dcb28c3badd6"
I20260129 02:59:36.053992 22358 raft_consensus.cc:697] T 4e420f90884f4796997b5ea41e49e411 P 197b880fa39d432a9b472845876bfaf0 [term 4 LEADER]: Becoming Leader. State: Replica: 197b880fa39d432a9b472845876bfaf0, State: Running, Role: LEADER
I20260129 02:59:35.989198 22343 raft_consensus.cc:740] T 18f30ebaa4174fd28c958e72d0cc7e24 P 7a9daf5d42a84615a6383cfec9a10608 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 7a9daf5d42a84615a6383cfec9a10608, State: Running, Role: LEADER
I20260129 02:59:36.078836 20584 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3e2d15f8640b407195210c476e09df91" candidate_uuid: "974b802141bd4f13beffef2aa03f88f9" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "1c50de89a5a845668696dcb28c3badd6" is_pre_election: true
W20260129 02:59:36.084012 20482 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260129 02:59:36.053187 22115 raft_consensus.cc:2749] T 0340229d521b432580ae926f6062e3f7 P 1c50de89a5a845668696dcb28c3badd6 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20260129 02:59:36.084424 20556 leader_election.cc:304] T 4408006eb5174de1896b037cf3f18457 P 1c50de89a5a845668696dcb28c3badd6 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1c50de89a5a845668696dcb28c3badd6; no voters: 197b880fa39d432a9b472845876bfaf0, 974b802141bd4f13beffef2aa03f88f9
I20260129 02:59:36.048039 22371 raft_consensus.cc:493] T 78aa39096868415d9afc2ffcb248e198 P 197b880fa39d432a9b472845876bfaf0 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 1c50de89a5a845668696dcb28c3badd6)
I20260129 02:59:36.087477 20582 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4b1876f9a70b49edb3aa74105e66d3b3" candidate_uuid: "974b802141bd4f13beffef2aa03f88f9" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "1c50de89a5a845668696dcb28c3badd6"
I20260129 02:59:36.057701 22342 raft_consensus.cc:2749] T 2ac659d732314e1288849f08fe5e074e P 7a9daf5d42a84615a6383cfec9a10608 [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260129 02:59:36.091006 20407 leader_election.cc:304] T 8055232cf0004d40a85f3bf983fe00a9 P 197b880fa39d432a9b472845876bfaf0 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 197b880fa39d432a9b472845876bfaf0, 974b802141bd4f13beffef2aa03f88f9; no voters:
I20260129 02:59:36.087082 21130 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3e2d15f8640b407195210c476e09df91" candidate_uuid: "974b802141bd4f13beffef2aa03f88f9" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "7a9daf5d42a84615a6383cfec9a10608" is_pre_election: true
I20260129 02:59:36.106812 22371 raft_consensus.cc:515] T 78aa39096868415d9afc2ffcb248e198 P 197b880fa39d432a9b472845876bfaf0 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } }
I20260129 02:59:36.116382 20585 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "75be4bd6bd45474795fd472eff2953ac" candidate_uuid: "974b802141bd4f13beffef2aa03f88f9" candidate_term: 8 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "1c50de89a5a845668696dcb28c3badd6" is_pre_election: true
I20260129 02:59:36.117787 20582 raft_consensus.cc:2468] T 4b1876f9a70b49edb3aa74105e66d3b3 P 1c50de89a5a845668696dcb28c3badd6 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 974b802141bd4f13beffef2aa03f88f9 in term 4.
W20260129 02:59:36.110886 21099 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.211.1:42027, user_credentials={real_user=slave}} blocked reactor thread for 58181.4us
I20260129 02:59:36.058840 22079 raft_consensus.cc:2749] T 8055232cf0004d40a85f3bf983fe00a9 P 1c50de89a5a845668696dcb28c3badd6 [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260129 02:59:36.122016 20587 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e33a417cf9c54cf5af73a4ee3aac4d3a" candidate_uuid: "974b802141bd4f13beffef2aa03f88f9" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "1c50de89a5a845668696dcb28c3badd6"
I20260129 02:59:36.122156 20588 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "505580c93c694bbb8ea96fdf9b17ab46" candidate_uuid: "974b802141bd4f13beffef2aa03f88f9" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "1c50de89a5a845668696dcb28c3badd6" is_pre_election: true
I20260129 02:59:36.121901 20583 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "cedce331e1584ac8b636c9b8a72b2ba0" candidate_uuid: "974b802141bd4f13beffef2aa03f88f9" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "1c50de89a5a845668696dcb28c3badd6" is_pre_election: true
I20260129 02:59:36.123198 21133 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a609ac02fae84725ab743f95f51d33ce" candidate_uuid: "974b802141bd4f13beffef2aa03f88f9" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "7a9daf5d42a84615a6383cfec9a10608"
I20260129 02:59:36.078490 21127 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3ccac82a01a2445fa5b0ce6c0fcf2295" candidate_uuid: "974b802141bd4f13beffef2aa03f88f9" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "7a9daf5d42a84615a6383cfec9a10608"
I20260129 02:59:36.076047 22355 raft_consensus.cc:3060] T b189f08e16a74a11980cc55d99359a0a P 974b802141bd4f13beffef2aa03f88f9 [term 4 FOLLOWER]: Advancing to term 5
W20260129 02:59:36.119890 20406 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260129 02:59:36.120148 21135 raft_consensus.cc:1240] T 96609ac68bdf4993a12e1c23355ca013 P 7a9daf5d42a84615a6383cfec9a10608 [term 4 FOLLOWER]: Rejecting Update request from peer 974b802141bd4f13beffef2aa03f88f9 for earlier term 3. Current term is 4. Ops: []
I20260129 02:59:36.120044 20439 raft_consensus.cc:1240] T 96609ac68bdf4993a12e1c23355ca013 P 197b880fa39d432a9b472845876bfaf0 [term 4 FOLLOWER]: Rejecting Update request from peer 974b802141bd4f13beffef2aa03f88f9 for earlier term 3. Current term is 4. Ops: [3.5-3.5]
I20260129 02:59:36.125736 20588 raft_consensus.cc:1240] T 96609ac68bdf4993a12e1c23355ca013 P 1c50de89a5a845668696dcb28c3badd6 [term 4 FOLLOWER]: Rejecting Update request from peer 974b802141bd4f13beffef2aa03f88f9 for earlier term 3. Current term is 4. Ops: []
I20260129 02:59:36.118917 20582 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "45cc5f3d2688486d9b6dbe7e666dedde" candidate_uuid: "974b802141bd4f13beffef2aa03f88f9" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "1c50de89a5a845668696dcb28c3badd6" is_pre_election: true
I20260129 02:59:36.031093 22213 consensus_queue.cc:260] T e33a417cf9c54cf5af73a4ee3aac4d3a P 1c50de89a5a845668696dcb28c3badd6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:59:36.056131 22061 consensus_queue.cc:1059] T e7d83d4ccf684e76acb29a0dc844e1b8 P 1c50de89a5a845668696dcb28c3badd6 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 }, Status: INVALID_TERM, Last received: 2.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.045s
I20260129 02:59:36.085520 22338 raft_consensus.cc:493] T 75f45538dc704d9a8ec4ff8ad9f3ea5b P 1c50de89a5a845668696dcb28c3badd6 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260129 02:59:36.128991 20405 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.211.3:37735, user_credentials={real_user=slave}} blocked reactor thread for 168776us
I20260129 02:59:36.090296 22243 raft_consensus.cc:2749] T 17d6431cc77a491e8464d68db4081a51 P 1c50de89a5a845668696dcb28c3badd6 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260129 02:59:36.133704 20590 raft_consensus.cc:1240] T 8055232cf0004d40a85f3bf983fe00a9 P 1c50de89a5a845668696dcb28c3badd6 [term 2 FOLLOWER]: Rejecting Update request from peer 974b802141bd4f13beffef2aa03f88f9 for earlier term 1. Current term is 2. Ops: []
I20260129 02:59:36.055155 22299 consensus_queue.cc:1048] T 0f20af7fbe5d4db89a6f17948d98912c P 1c50de89a5a845668696dcb28c3badd6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.061s
W20260129 02:59:36.137059 20482 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260129 02:59:36.099345 22190 raft_consensus.cc:2749] T 4f8f3a38f6e24ba4857875181f447442 P 197b880fa39d432a9b472845876bfaf0 [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260129 02:59:36.066977 22164 raft_consensus.cc:515] T 76f713c3331540d2a5b76164103e9eb1 P 1c50de89a5a845668696dcb28c3badd6 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
W20260129 02:59:36.139374 20407 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.211.2:34517, user_credentials={real_user=slave}} blocked reactor thread for 48453.4us
I20260129 02:59:36.059466 22231 raft_consensus.cc:515] T 18f30ebaa4174fd28c958e72d0cc7e24 P 197b880fa39d432a9b472845876bfaf0 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } }
I20260129 02:59:36.151122 20510 raft_consensus.cc:2468] T 913fc7d751ed4b7388d889aac30a5f9b P 974b802141bd4f13beffef2aa03f88f9 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1c50de89a5a845668696dcb28c3badd6 in term 2.
I20260129 02:59:36.116709 21128 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "febffe4a8b394368b69b1d164308fb9a" candidate_uuid: "974b802141bd4f13beffef2aa03f88f9" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "7a9daf5d42a84615a6383cfec9a10608"
I20260129 02:59:36.142795 21136 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "505580c93c694bbb8ea96fdf9b17ab46" candidate_uuid: "974b802141bd4f13beffef2aa03f88f9" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "7a9daf5d42a84615a6383cfec9a10608" is_pre_election: true
I20260129 02:59:36.187947 21134 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "45cc5f3d2688486d9b6dbe7e666dedde" candidate_uuid: "974b802141bd4f13beffef2aa03f88f9" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "7a9daf5d42a84615a6383cfec9a10608" is_pre_election: true
W20260129 02:59:36.188687 20556 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.211.2:34517, user_credentials={real_user=slave}} blocked reactor thread for 98178us
I20260129 02:59:36.099180 22292 consensus_queue.cc:1048] T 3906b080a5b34a74b8b64128635e8997 P 1c50de89a5a845668696dcb28c3badd6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.060s
I20260129 02:59:36.191826 20584 raft_consensus.cc:1240] T 17d6431cc77a491e8464d68db4081a51 P 1c50de89a5a845668696dcb28c3badd6 [term 2 FOLLOWER]: Rejecting Update request from peer 974b802141bd4f13beffef2aa03f88f9 for earlier term 1. Current term is 2. Ops: []
I20260129 02:59:36.192294 21136 raft_consensus.cc:2468] T 505580c93c694bbb8ea96fdf9b17ab46 P 7a9daf5d42a84615a6383cfec9a10608 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 974b802141bd4f13beffef2aa03f88f9 in term 3.
I20260129 02:59:36.127998 22300 raft_consensus.cc:2749] T 4408006eb5174de1896b037cf3f18457 P 1c50de89a5a845668696dcb28c3badd6 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260129 02:59:36.119340 22082 consensus_queue.cc:1059] T 7a9a69cd82724b1faed1d10aa091fc4a P 197b880fa39d432a9b472845876bfaf0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 }, Status: INVALID_TERM, Last received: 3.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.049s
I20260129 02:59:36.072618 22350 raft_consensus.cc:515] T add8d4923d1a489e86882c3a25692b95 P 7a9daf5d42a84615a6383cfec9a10608 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } }
I20260129 02:59:36.152650 22339 raft_consensus.cc:493] T 368953c9102643b2a21f99af36b51905 P 197b880fa39d432a9b472845876bfaf0 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260129 02:59:36.223887 20405 leader_election.cc:304] T 96609ac68bdf4993a12e1c23355ca013 P 197b880fa39d432a9b472845876bfaf0 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 197b880fa39d432a9b472845876bfaf0; no voters: 7a9daf5d42a84615a6383cfec9a10608, 974b802141bd4f13beffef2aa03f88f9
I20260129 02:59:36.229059 19276 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260129 02:59:36.204747 22091 raft_consensus.cc:493] T cedce331e1584ac8b636c9b8a72b2ba0 P 197b880fa39d432a9b472845876bfaf0 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 1c50de89a5a845668696dcb28c3badd6)
I20260129 02:59:36.232869 20405 leader_election.cc:304] T 2ac659d732314e1288849f08fe5e074e P 197b880fa39d432a9b472845876bfaf0 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 197b880fa39d432a9b472845876bfaf0, 7a9daf5d42a84615a6383cfec9a10608; no voters:
I20260129 02:59:36.126037 22184 consensus_queue.cc:260] T 96609ac68bdf4993a12e1c23355ca013 P 974b802141bd4f13beffef2aa03f88f9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.5, Last appended by leader: 5, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } attrs { replace: true } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } }
I20260129 02:59:36.134492 22323 consensus_queue.cc:260] T cb0b3f259ebc4310927338a8740e87b8 P 974b802141bd4f13beffef2aa03f88f9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 3.6, Last appended by leader: 6, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } }
I20260129 02:59:36.190300 22338 raft_consensus.cc:515] T 75f45538dc704d9a8ec4ff8ad9f3ea5b P 1c50de89a5a845668696dcb28c3badd6 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } }
I20260129 02:59:36.221597 22112 raft_consensus.cc:493] T 1ce4ddf5b10e4ae8b193b13b2e5936ee P 974b802141bd4f13beffef2aa03f88f9 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260129 02:59:36.327677 20482 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260129 02:59:36.278308 22353 raft_consensus.cc:3055] T e7d83d4ccf684e76acb29a0dc844e1b8 P 1c50de89a5a845668696dcb28c3badd6 [term 2 LEADER]: Stepping down as leader of term 2
W20260129 02:59:36.366740 20482 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260129 02:59:36.392812 20435 raft_consensus.cc:3060] T 3906b080a5b34a74b8b64128635e8997 P 197b880fa39d432a9b472845876bfaf0 [term 1 FOLLOWER]: Advancing to term 2
I20260129 02:59:36.276814 21128 raft_consensus.cc:2393] T febffe4a8b394368b69b1d164308fb9a P 7a9daf5d42a84615a6383cfec9a10608 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 974b802141bd4f13beffef2aa03f88f9 in current term 2: Already voted for candidate 7a9daf5d42a84615a6383cfec9a10608 in this term.
I20260129 02:59:36.328336 20556 leader_election.cc:304] T b189f08e16a74a11980cc55d99359a0a P 1c50de89a5a845668696dcb28c3badd6 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1c50de89a5a845668696dcb28c3badd6; no voters: 197b880fa39d432a9b472845876bfaf0, 974b802141bd4f13beffef2aa03f88f9
I20260129 02:59:36.233155 22213 raft_consensus.cc:3060] T e33a417cf9c54cf5af73a4ee3aac4d3a P 1c50de89a5a845668696dcb28c3badd6 [term 3 FOLLOWER]: Advancing to term 4
W20260129 02:59:36.368435 20405 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.211.4:40439, user_credentials={real_user=slave}} blocked reactor thread for 135642us
I20260129 02:59:36.326382 22091 raft_consensus.cc:515] T cedce331e1584ac8b636c9b8a72b2ba0 P 197b880fa39d432a9b472845876bfaf0 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:59:36.370851 22391 raft_consensus.cc:493] T cb0b3f259ebc4310927338a8740e87b8 P 7a9daf5d42a84615a6383cfec9a10608 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260129 02:59:36.273874 22357 raft_consensus.cc:2804] T a0fcc58d6fb6473cafbd9d34d5195269 P 974b802141bd4f13beffef2aa03f88f9 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260129 02:59:36.371681 22350 leader_election.cc:290] T add8d4923d1a489e86882c3a25692b95 P 7a9daf5d42a84615a6383cfec9a10608 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 974b802141bd4f13beffef2aa03f88f9 (127.18.211.2:34517), 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3:37735)
I20260129 02:59:36.274081 22385 raft_consensus.cc:493] T 7d9494f5584d4d5db8b9adf1c8fc888b P 197b880fa39d432a9b472845876bfaf0 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 1c50de89a5a845668696dcb28c3badd6)
I20260129 02:59:36.372089 22198 raft_consensus.cc:2804] T 2ac659d732314e1288849f08fe5e074e P 197b880fa39d432a9b472845876bfaf0 [term 2 FOLLOWER]: Leader election won for term 2
I20260129 02:59:36.372757 22164 leader_election.cc:290] T 76f713c3331540d2a5b76164103e9eb1 P 1c50de89a5a845668696dcb28c3badd6 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 197b880fa39d432a9b472845876bfaf0 (127.18.211.1:42027), 974b802141bd4f13beffef2aa03f88f9 (127.18.211.2:34517)
W20260129 02:59:36.377403 22369 log.cc:927] Time spent T 75b0b6e82ddc4f92b1fe1d94f9a0b73b P 1c50de89a5a845668696dcb28c3badd6: Append to log took a long time: real 0.528s user 0.001s sys 0.000s
I20260129 02:59:36.232638 22374 consensus_queue.cc:1059] T 4e420f90884f4796997b5ea41e49e411 P 7a9daf5d42a84615a6383cfec9a10608 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 }, Status: INVALID_TERM, Last received: 3.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.132s
W20260129 02:59:36.413384 20480 connection.cc:232] RPC call timeout handler was delayed by -1.1914565150000271s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.15300000000000002s for a response
I20260129 02:59:36.418388 20586 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "add8d4923d1a489e86882c3a25692b95" candidate_uuid: "7a9daf5d42a84615a6383cfec9a10608" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "1c50de89a5a845668696dcb28c3badd6" is_pre_election: true
I20260129 02:59:36.286801 22373 raft_consensus.cc:2804] T 8055232cf0004d40a85f3bf983fe00a9 P 197b880fa39d432a9b472845876bfaf0 [term 2 FOLLOWER]: Leader election won for term 2
I20260129 02:59:36.276942 22255 consensus_queue.cc:1059] T 2e58e9a5a8aa4cecafb464addb4f35ed P 7a9daf5d42a84615a6383cfec9a10608 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.126s
I20260129 02:59:36.276750 22300 consensus_queue.cc:1048] T 0f20af7fbe5d4db89a6f17948d98912c P 1c50de89a5a845668696dcb28c3badd6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.138s
I20260129 02:59:36.423503 20586 raft_consensus.cc:2468] T add8d4923d1a489e86882c3a25692b95 P 1c50de89a5a845668696dcb28c3badd6 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7a9daf5d42a84615a6383cfec9a10608 in term 2.
W20260129 02:59:36.423266 20480 connection.cc:232] RPC call timeout handler was delayed by -1.1320565149999311s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.15960000000000002s for a response
I20260129 02:59:36.428927 21100 leader_election.cc:304] T add8d4923d1a489e86882c3a25692b95 P 7a9daf5d42a84615a6383cfec9a10608 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1c50de89a5a845668696dcb28c3badd6, 7a9daf5d42a84615a6383cfec9a10608; no voters:
I20260129 02:59:36.382256 22353 raft_consensus.cc:740] T e7d83d4ccf684e76acb29a0dc844e1b8 P 1c50de89a5a845668696dcb28c3badd6 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 1c50de89a5a845668696dcb28c3badd6, State: Running, Role: LEADER
I20260129 02:59:36.405067 22292 raft_consensus.cc:493] T c891d2bff4ce4568861bbc992e91fbb0 P 1c50de89a5a845668696dcb28c3badd6 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 974b802141bd4f13beffef2aa03f88f9)
I20260129 02:59:36.228735 22339 raft_consensus.cc:515] T 368953c9102643b2a21f99af36b51905 P 197b880fa39d432a9b472845876bfaf0 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:59:36.207162 22343 consensus_queue.cc:260] T 18f30ebaa4174fd28c958e72d0cc7e24 P 7a9daf5d42a84615a6383cfec9a10608 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } }
I20260129 02:59:36.392285 22112 raft_consensus.cc:515] T 1ce4ddf5b10e4ae8b193b13b2e5936ee P 974b802141bd4f13beffef2aa03f88f9 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:59:36.423267 20434 raft_consensus.cc:1240] T 926a250fa5834e88b96ccecbefbf5086 P 197b880fa39d432a9b472845876bfaf0 [term 5 FOLLOWER]: Rejecting Update request from peer 1c50de89a5a845668696dcb28c3badd6 for earlier term 4. Current term is 5. Ops: []
W20260129 02:59:36.442355 20404 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260129 02:59:36.444790 20435 raft_consensus.cc:1275] T 3906b080a5b34a74b8b64128635e8997 P 197b880fa39d432a9b472845876bfaf0 [term 2 FOLLOWER]: Refusing update from remote peer 1c50de89a5a845668696dcb28c3badd6: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260129 02:59:36.417644 22357 raft_consensus.cc:493] T a0fcc58d6fb6473cafbd9d34d5195269 P 974b802141bd4f13beffef2aa03f88f9 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 197b880fa39d432a9b472845876bfaf0)
W20260129 02:59:36.450513 20556 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.211.2:34517, user_credentials={real_user=slave}} blocked reactor thread for 122288us
I20260129 02:59:36.423960 22198 raft_consensus.cc:697] T 2ac659d732314e1288849f08fe5e074e P 197b880fa39d432a9b472845876bfaf0 [term 2 LEADER]: Becoming Leader. State: Replica: 197b880fa39d432a9b472845876bfaf0, State: Running, Role: LEADER
I20260129 02:59:36.452891 20435 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55676 (request call id 1127) took 1481 ms. Trace:
I20260129 02:59:36.454679 20435 rpcz_store.cc:276] 0129 02:59:34.967553 (+ 0us) service_pool.cc:168] Inserting onto call queue
0129 02:59:35.060376 (+ 92823us) service_pool.cc:225] Handling call
0129 02:59:35.060711 (+ 335us) raft_consensus.cc:1471] Updating replica for 0 ops
0129 02:59:35.454628 (+393917us) spinlock_profiling.cc:243] Waited 215 ms on lock 0x7b5400411ca0. stack: 00007f8fe044e373 00007f8fe044e270 00007f8fdfda0574 00007f8fe9a5c6b4 00007f8fe9a5c649 00007f8fe34975a1 00007f8fe34a5df0 00007f8fe352fe40 00007f8fe3530eb1 00007f8fe352da0f 00007f8fe352d4be 00007f8fe84e1f49 00007f8fe1a6ed34 00007f8fe1a6ecc2 00007f8fe1a6ebf1 00007f8fe1a6eb6c
0129 02:59:35.824636 (+370008us) spinlock_profiling.cc:243] Waited 227 ms on lock 0x7b180021ce40. stack: 00007f8fe044e373 00007f8fe044e270 00007f8fdfda0574 00007f8fe9a5c6b4 00007f8fe9a5c649 00007f8fe34975a1 00007f8fe3569496 00007f8fe34a35be 00007f8fe3524c1e 00007f8fe352645e 00007f8fe35307c9 00007f8fe3531096 00007f8fe352da0f 00007f8fe352d4be 00007f8fe84e1f49 00007f8fe1a6ed34
0129 02:59:36.377871 (+553235us) spinlock_profiling.cc:243] Waited 552 ms on lock 0x7b58003302d0. stack: 00007f8fe044e373 00007f8fe044e270 00007f8fdfda0574 00007f8fe9a5c6b4 00007f8fe9a5c649 00007f8fe34975a1 00007f8fe348a0da 00007f8fe350c177 00007f8fe3524c67 00007f8fe352645e 00007f8fe35307c9 00007f8fe3531096 00007f8fe352da0f 00007f8fe352d4be 00007f8fe84e1f49 00007f8fe1a6ed34
0129 02:59:36.449126 (+ 71255us) raft_consensus.cc:1697] Filling consensus response to leader.
0129 02:59:36.449336 (+ 210us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2284070016}
I20260129 02:59:36.367481 22224 consensus_queue.cc:1048] T 505580c93c694bbb8ea96fdf9b17ab46 P 1c50de89a5a845668696dcb28c3badd6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.057s
I20260129 02:59:36.457095 20556 leader_election.cc:304] T 913fc7d751ed4b7388d889aac30a5f9b P 1c50de89a5a845668696dcb28c3badd6 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1c50de89a5a845668696dcb28c3badd6, 974b802141bd4f13beffef2aa03f88f9; no voters: 197b880fa39d432a9b472845876bfaf0
I20260129 02:59:36.457414 20435 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "76f713c3331540d2a5b76164103e9eb1" candidate_uuid: "1c50de89a5a845668696dcb28c3badd6" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "197b880fa39d432a9b472845876bfaf0" is_pre_election: true
I20260129 02:59:36.274502 22358 consensus_queue.cc:237] T 4e420f90884f4796997b5ea41e49e411 P 197b880fa39d432a9b472845876bfaf0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.5, Last appended by leader: 5, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } }
I20260129 02:59:36.466190 20510 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e4d18c35b0a7493e85d0a4b92956a48f" candidate_uuid: "1c50de89a5a845668696dcb28c3badd6" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "974b802141bd4f13beffef2aa03f88f9" is_pre_election: true
I20260129 02:59:36.466882 20510 raft_consensus.cc:2468] T e4d18c35b0a7493e85d0a4b92956a48f P 974b802141bd4f13beffef2aa03f88f9 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1c50de89a5a845668696dcb28c3badd6 in term 3.
I20260129 02:59:36.468801 20510 raft_consensus.cc:1275] T 368953c9102643b2a21f99af36b51905 P 974b802141bd4f13beffef2aa03f88f9 [term 4 FOLLOWER]: Refusing update from remote peer 1c50de89a5a845668696dcb28c3badd6: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 3. (term mismatch)
I20260129 02:59:36.468750 20405 leader_election.cc:304] T e7d83d4ccf684e76acb29a0dc844e1b8 P 197b880fa39d432a9b472845876bfaf0 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 197b880fa39d432a9b472845876bfaf0; no voters: 1c50de89a5a845668696dcb28c3badd6, 974b802141bd4f13beffef2aa03f88f9
I20260129 02:59:36.469226 20510 pending_rounds.cc:85] T 368953c9102643b2a21f99af36b51905 P 974b802141bd4f13beffef2aa03f88f9: Aborting all ops after (but not including) 2
I20260129 02:59:36.469465 20510 pending_rounds.cc:107] T 368953c9102643b2a21f99af36b51905 P 974b802141bd4f13beffef2aa03f88f9: Aborting uncommitted NO_OP operation due to leader change: 3.3
I20260129 02:59:36.469664 20510 raft_consensus.cc:2889] T 368953c9102643b2a21f99af36b51905 P 974b802141bd4f13beffef2aa03f88f9 [term 4 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260129 02:59:36.469969 20405 leader_election.cc:304] T 17d6431cc77a491e8464d68db4081a51 P 197b880fa39d432a9b472845876bfaf0 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 197b880fa39d432a9b472845876bfaf0; no voters: 1c50de89a5a845668696dcb28c3badd6, 974b802141bd4f13beffef2aa03f88f9
I20260129 02:59:36.433540 22373 raft_consensus.cc:697] T 8055232cf0004d40a85f3bf983fe00a9 P 197b880fa39d432a9b472845876bfaf0 [term 2 LEADER]: Becoming Leader. State: Replica: 197b880fa39d432a9b472845876bfaf0, State: Running, Role: LEADER
I20260129 02:59:36.472162 20514 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "76f713c3331540d2a5b76164103e9eb1" candidate_uuid: "1c50de89a5a845668696dcb28c3badd6" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "974b802141bd4f13beffef2aa03f88f9" is_pre_election: true
I20260129 02:59:36.472762 20514 raft_consensus.cc:2468] T 76f713c3331540d2a5b76164103e9eb1 P 974b802141bd4f13beffef2aa03f88f9 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1c50de89a5a845668696dcb28c3badd6 in term 3.
I20260129 02:59:36.450150 22357 raft_consensus.cc:3060] T a0fcc58d6fb6473cafbd9d34d5195269 P 974b802141bd4f13beffef2aa03f88f9 [term 2 FOLLOWER]: Advancing to term 3
I20260129 02:59:36.449923 22299 raft_consensus.cc:493] T a644c02b16af47679824ccd060ac10fd P 1c50de89a5a845668696dcb28c3badd6 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260129 02:59:36.456736 22366 raft_consensus.cc:2804] T add8d4923d1a489e86882c3a25692b95 P 7a9daf5d42a84615a6383cfec9a10608 [term 1 FOLLOWER]: Leader pre-election won for term 2
W20260129 02:59:36.457962 22237 raft_consensus.cc:936] Couldn't advance consensus term.: Illegal state: Can't advance term to: 2 current term: 2 is higher.
W20260129 02:59:36.480144 20480 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.211.4:40439, user_credentials={real_user=slave}} blocked reactor thread for 46091us
I20260129 02:59:36.483584 20435 raft_consensus.cc:3055] T 7a9a69cd82724b1faed1d10aa091fc4a P 197b880fa39d432a9b472845876bfaf0 [term 3 LEADER]: Stepping down as leader of term 3
I20260129 02:59:36.469569 22092 raft_consensus.cc:3060] T e7d83d4ccf684e76acb29a0dc844e1b8 P 197b880fa39d432a9b472845876bfaf0 [term 2 FOLLOWER]: Advancing to term 3
I20260129 02:59:36.476713 22164 raft_consensus.cc:2804] T 913fc7d751ed4b7388d889aac30a5f9b P 1c50de89a5a845668696dcb28c3badd6 [term 2 FOLLOWER]: Leader election won for term 2
I20260129 02:59:36.502346 20513 raft_consensus.cc:1275] T 4f8f3a38f6e24ba4857875181f447442 P 974b802141bd4f13beffef2aa03f88f9 [term 4 FOLLOWER]: Refusing update from remote peer 1c50de89a5a845668696dcb28c3badd6: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20260129 02:59:36.541692 20435 raft_consensus.cc:740] T 7a9a69cd82724b1faed1d10aa091fc4a P 197b880fa39d432a9b472845876bfaf0 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 197b880fa39d432a9b472845876bfaf0, State: Running, Role: LEADER
I20260129 02:59:36.542838 20480 leader_election.cc:304] T add8d4923d1a489e86882c3a25692b95 P 974b802141bd4f13beffef2aa03f88f9 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 974b802141bd4f13beffef2aa03f88f9; no voters: 1c50de89a5a845668696dcb28c3badd6, 7a9daf5d42a84615a6383cfec9a10608
I20260129 02:59:36.475893 22231 leader_election.cc:290] T 18f30ebaa4174fd28c958e72d0cc7e24 P 197b880fa39d432a9b472845876bfaf0 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3:37735), 7a9daf5d42a84615a6383cfec9a10608 (127.18.211.4:40439)
I20260129 02:59:36.484931 22077 raft_consensus.cc:3060] T 17d6431cc77a491e8464d68db4081a51 P 197b880fa39d432a9b472845876bfaf0 [term 1 FOLLOWER]: Advancing to term 2
I20260129 02:59:36.546255 22372 raft_consensus.cc:493] T c934d31efd3f4e5195605ddc354d14e1 P 7a9daf5d42a84615a6383cfec9a10608 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 197b880fa39d432a9b472845876bfaf0)
I20260129 02:59:36.487341 22371 leader_election.cc:290] T 78aa39096868415d9afc2ffcb248e198 P 197b880fa39d432a9b472845876bfaf0 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3:37735), 7a9daf5d42a84615a6383cfec9a10608 (127.18.211.4:40439)
I20260129 02:59:36.445494 22292 raft_consensus.cc:515] T c891d2bff4ce4568861bbc992e91fbb0 P 1c50de89a5a845668696dcb28c3badd6 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
W20260129 02:59:36.554768 20441 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.533s user 0.019s sys 0.025s
I20260129 02:59:36.464274 22061 consensus_queue.cc:1048] T a4c1226b987b468fa84dfc6ba0e2dbe1 P 1c50de89a5a845668696dcb28c3badd6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.076s
I20260129 02:59:36.558017 20435 consensus_queue.cc:260] T 7a9a69cd82724b1faed1d10aa091fc4a P 197b880fa39d432a9b472845876bfaf0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:59:36.420640 22391 raft_consensus.cc:515] T cb0b3f259ebc4310927338a8740e87b8 P 7a9daf5d42a84615a6383cfec9a10608 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } }
I20260129 02:59:36.482098 22395 raft_consensus.cc:493] T fd8311b779af4e1fa1402b2a02a602f2 P 1c50de89a5a845668696dcb28c3badd6 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 197b880fa39d432a9b472845876bfaf0)
I20260129 02:59:36.502323 22366 raft_consensus.cc:493] T add8d4923d1a489e86882c3a25692b95 P 7a9daf5d42a84615a6383cfec9a10608 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 1c50de89a5a845668696dcb28c3badd6)
I20260129 02:59:36.546166 22082 consensus_queue.cc:1059] T 0340229d521b432580ae926f6062e3f7 P 197b880fa39d432a9b472845876bfaf0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 }, Status: INVALID_TERM, Last received: 3.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
W20260129 02:59:36.573586 20405 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.211.3:37735, user_credentials={real_user=slave}} blocked reactor thread for 95455us
W20260129 02:59:36.744590 21100 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260129 02:59:36.432572 22385 raft_consensus.cc:515] T 7d9494f5584d4d5db8b9adf1c8fc888b P 197b880fa39d432a9b472845876bfaf0 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
W20260129 02:59:36.585008 20441 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.563s user 0.019s sys 0.025s
I20260129 02:59:36.502661 22346 consensus_queue.cc:1048] T c2633a6fc34c44579de348d208b0c01e P 1c50de89a5a845668696dcb28c3badd6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.304s
I20260129 02:59:36.477926 22378 consensus_queue.cc:1048] T 76f713c3331540d2a5b76164103e9eb1 P 197b880fa39d432a9b472845876bfaf0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.389s
W20260129 02:59:36.679558 20581 consensus_meta.cc:287] T a8479d1fe0bd41e5946186e3a6ca8c7a P 1c50de89a5a845668696dcb28c3badd6: Time spent flushing consensus metadata: real 0.609s user 0.010s sys 0.001s
I20260129 02:59:36.583289 22164 raft_consensus.cc:697] T 913fc7d751ed4b7388d889aac30a5f9b P 1c50de89a5a845668696dcb28c3badd6 [term 2 LEADER]: Becoming Leader. State: Replica: 1c50de89a5a845668696dcb28c3badd6, State: Running, Role: LEADER
W20260129 02:59:36.658610 22383 log.cc:927] Time spent T 36ff33be666a4c82954731691dbcf327 P 1c50de89a5a845668696dcb28c3badd6: Append to log took a long time: real 0.212s user 0.001s sys 0.000s
I20260129 02:59:36.543635 22384 raft_consensus.cc:2749] T 96609ac68bdf4993a12e1c23355ca013 P 197b880fa39d432a9b472845876bfaf0 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20260129 02:59:36.661791 22061 raft_consensus.cc:493] T cb0b3f259ebc4310927338a8740e87b8 P 1c50de89a5a845668696dcb28c3badd6 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260129 02:59:36.650431 22390 log.cc:927] Time spent T 75be4bd6bd45474795fd472eff2953ac P 197b880fa39d432a9b472845876bfaf0: Append to log took a long time: real 0.110s user 0.000s sys 0.000s
I20260129 02:59:36.672039 22338 leader_election.cc:290] T 75f45538dc704d9a8ec4ff8ad9f3ea5b P 1c50de89a5a845668696dcb28c3badd6 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 974b802141bd4f13beffef2aa03f88f9 (127.18.211.2:34517), 7a9daf5d42a84615a6383cfec9a10608 (127.18.211.4:40439)
I20260129 02:59:36.503345 22299 raft_consensus.cc:515] T a644c02b16af47679824ccd060ac10fd P 1c50de89a5a845668696dcb28c3badd6 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } }
I20260129 02:59:36.540932 22353 consensus_queue.cc:260] T e7d83d4ccf684e76acb29a0dc844e1b8 P 1c50de89a5a845668696dcb28c3badd6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
W20260129 02:59:36.688589 20480 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.211.4:40439, user_credentials={real_user=slave}} blocked reactor thread for 145856us
I20260129 02:59:36.677558 22091 leader_election.cc:290] T cedce331e1584ac8b636c9b8a72b2ba0 P 197b880fa39d432a9b472845876bfaf0 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 974b802141bd4f13beffef2aa03f88f9 (127.18.211.2:34517), 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3:37735)
W20260129 02:59:36.715302 21099 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.525s user 0.001s sys 0.046s
I20260129 02:59:36.645107 22396 consensus_queue.cc:1059] T 96609ac68bdf4993a12e1c23355ca013 P 974b802141bd4f13beffef2aa03f88f9 [NON_LEADER]: Peer responded invalid term: Peer: permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 }, Status: INVALID_TERM, Last received: 2.4, Next index: 5, Last known committed idx: 3, Time since last communication: 0.283s
I20260129 02:59:36.747200 20581 raft_consensus.cc:2468] T a8479d1fe0bd41e5946186e3a6ca8c7a P 1c50de89a5a845668696dcb28c3badd6 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 197b880fa39d432a9b472845876bfaf0 in term 3.
I20260129 02:59:36.748591 20405 leader_election.cc:304] T 4408006eb5174de1896b037cf3f18457 P 197b880fa39d432a9b472845876bfaf0 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 197b880fa39d432a9b472845876bfaf0, 1c50de89a5a845668696dcb28c3badd6; no voters: 974b802141bd4f13beffef2aa03f88f9
W20260129 02:59:36.749027 21101 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.557s user 0.006s sys 0.013s
W20260129 02:59:36.751377 20479 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.561s user 0.000s sys 0.015s
W20260129 02:59:36.822528 21101 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.630s user 0.006s sys 0.013s
W20260129 02:59:36.822642 20479 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.661s user 0.000s sys 0.016s
W20260129 02:59:36.822808 21135 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.617s user 0.006s sys 0.013s
I20260129 02:59:36.675684 22157 consensus_queue.cc:1059] T 75f45538dc704d9a8ec4ff8ad9f3ea5b P 974b802141bd4f13beffef2aa03f88f9 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.038s
I20260129 02:59:36.705294 22389 raft_consensus.cc:2749] T b189f08e16a74a11980cc55d99359a0a P 1c50de89a5a845668696dcb28c3badd6 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
W20260129 02:59:36.744267 22382 log.cc:927] Time spent T dc2f4cb752cf4d59b5f6bc689f5d8104 P 7a9daf5d42a84615a6383cfec9a10608: Append to log took a long time: real 0.249s user 0.000s sys 0.001s
I20260129 02:59:36.587038 22372 raft_consensus.cc:515] T c934d31efd3f4e5195605ddc354d14e1 P 7a9daf5d42a84615a6383cfec9a10608 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } }
I20260129 02:59:36.642948 22198 consensus_queue.cc:237] T 2ac659d732314e1288849f08fe5e074e P 197b880fa39d432a9b472845876bfaf0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } }
I20260129 02:59:36.745124 22366 raft_consensus.cc:3060] T add8d4923d1a489e86882c3a25692b95 P 7a9daf5d42a84615a6383cfec9a10608 [term 1 FOLLOWER]: Advancing to term 2
I20260129 02:59:36.772329 22339 leader_election.cc:290] T 368953c9102643b2a21f99af36b51905 P 197b880fa39d432a9b472845876bfaf0 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 974b802141bd4f13beffef2aa03f88f9 (127.18.211.2:34517), 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3:37735)
I20260129 02:59:36.717496 22401 raft_consensus.cc:493] T 7d9494f5584d4d5db8b9adf1c8fc888b P 974b802141bd4f13beffef2aa03f88f9 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 1c50de89a5a845668696dcb28c3badd6)
I20260129 02:59:36.809118 20480 leader_election.cc:304] T a644c02b16af47679824ccd060ac10fd P 974b802141bd4f13beffef2aa03f88f9 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 974b802141bd4f13beffef2aa03f88f9; no voters: 1c50de89a5a845668696dcb28c3badd6, 7a9daf5d42a84615a6383cfec9a10608
W20260129 02:59:36.768069 21102 thread.cc:641] dns-resolver [worker] (thread pool) Time spent creating pthread: real 0.574s user 0.003s sys 0.019s
W20260129 02:59:36.846192 21102 thread.cc:608] dns-resolver [worker] (thread pool) Time spent starting thread: real 0.652s user 0.004s sys 0.027s
W20260129 02:59:36.846580 21102 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.211.2:34517, user_credentials={real_user=slave}} blocked reactor thread for 653204us
W20260129 02:59:36.848246 21102 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260129 02:59:36.811745 21099 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.621s user 0.001s sys 0.046s
W20260129 02:59:36.823350 21135 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.618s user 0.006s sys 0.013s
W20260129 02:59:36.823076 20479 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.211.1:42027, user_credentials={real_user=slave}} blocked reactor thread for 661466us
W20260129 02:59:36.760429 22379 log.cc:927] Time spent T 75be4bd6bd45474795fd472eff2953ac P 974b802141bd4f13beffef2aa03f88f9: Append to log took a long time: real 0.308s user 0.000s sys 0.001s
W20260129 02:59:36.849092 21099 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.211.1:42027, user_credentials={real_user=slave}} blocked reactor thread for 658582us
I20260129 02:59:36.746729 22411 raft_consensus.cc:493] T 4b1876f9a70b49edb3aa74105e66d3b3 P 197b880fa39d432a9b472845876bfaf0 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260129 02:59:36.852185 21099 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260129 02:59:36.852619 21099 connection.cc:232] RPC call timeout handler was delayed by -1.1352241540000705s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.15510000000000002s for a response
W20260129 02:59:36.853034 21099 connection.cc:232] RPC call timeout handler was delayed by -1.0442193970000062s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1888s for a response
W20260129 02:59:36.839666 20405 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.211.3:37735, user_credentials={real_user=slave}} blocked reactor thread for 91129.9us
W20260129 02:59:36.787202 20482 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260129 02:59:36.750128 22388 consensus_queue.cc:1048] T 3906b080a5b34a74b8b64128635e8997 P 1c50de89a5a845668696dcb28c3badd6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.353s
I20260129 02:59:36.753543 22112 leader_election.cc:290] T 1ce4ddf5b10e4ae8b193b13b2e5936ee P 974b802141bd4f13beffef2aa03f88f9 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 197b880fa39d432a9b472845876bfaf0 (127.18.211.1:42027), 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3:37735)
W20260129 02:59:36.850126 20479 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260129 02:59:36.850886 20480 leader_election.cc:304] T c934d31efd3f4e5195605ddc354d14e1 P 974b802141bd4f13beffef2aa03f88f9 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 7a9daf5d42a84615a6383cfec9a10608, 974b802141bd4f13beffef2aa03f88f9; no voters: 197b880fa39d432a9b472845876bfaf0
W20260129 02:59:36.855973 20482 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260129 02:59:36.855113 20405 leader_election.cc:304] T a5a05fbcbbcf4916bbc01fd1a5b5e90d P 197b880fa39d432a9b472845876bfaf0 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 197b880fa39d432a9b472845876bfaf0, 1c50de89a5a845668696dcb28c3badd6; no voters: 974b802141bd4f13beffef2aa03f88f9
I20260129 02:59:36.864320 22335 raft_consensus.cc:2764] T a5a05fbcbbcf4916bbc01fd1a5b5e90d P 197b880fa39d432a9b472845876bfaf0 [term 5 FOLLOWER]: Leader pre-election decision vote started in defunct term 4: won
I20260129 02:59:36.871691 22409 raft_consensus.cc:493] T 96609ac68bdf4993a12e1c23355ca013 P 1c50de89a5a845668696dcb28c3badd6 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260129 02:59:36.872380 22409 raft_consensus.cc:515] T 96609ac68bdf4993a12e1c23355ca013 P 1c50de89a5a845668696dcb28c3badd6 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } attrs { replace: true } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: NON_VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: true } }
I20260129 02:59:36.874832 22409 leader_election.cc:290] T 96609ac68bdf4993a12e1c23355ca013 P 1c50de89a5a845668696dcb28c3badd6 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 197b880fa39d432a9b472845876bfaf0 (127.18.211.1:42027), 974b802141bd4f13beffef2aa03f88f9 (127.18.211.2:34517)
I20260129 02:59:36.878618 20441 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55672 (request call id 1441) took 1105 ms. Trace:
I20260129 02:59:36.879037 20441 rpcz_store.cc:276] 0129 02:59:35.772964 (+ 0us) service_pool.cc:168] Inserting onto call queue
0129 02:59:35.801402 (+ 28438us) service_pool.cc:225] Handling call
0129 02:59:35.805640 (+ 4238us) raft_consensus.cc:1471] Updating replica for 1 ops
0129 02:59:35.962693 (+157053us) raft_consensus.cc:1534] Early marking committed up to index 2
0129 02:59:35.962729 (+ 36us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0129 02:59:35.971533 (+ 8804us) log.cc:844] Serialized 26 byte log entry
0129 02:59:36.745476 (+773943us) raft_consensus.cc:1645] Marking committed up to 2
0129 02:59:36.745528 (+ 52us) raft_consensus.cc:1697] Filling consensus response to leader.
0129 02:59:36.745597 (+ 69us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0129 02:59:36.878397 (+132800us) raft_consensus.cc:1685] finished
0129 02:59:36.878421 (+ 24us) raft_consensus.cc:1691] UpdateReplicas() finished
0129 02:59:36.878559 (+ 138us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":42012,"raft.run_cpu_time_us":70,"raft.run_wall_time_us":70,"thread_start_us":824052,"threads_started":2,"wal-append.queue_time_us":753300}
I20260129 02:59:36.880196 20437 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55676 (request call id 1139) took 1198 ms. Trace:
W20260129 02:59:36.880268 20482 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260129 02:59:36.880613 20437 rpcz_store.cc:276] 0129 02:59:35.681732 (+ 0us) service_pool.cc:168] Inserting onto call queue
0129 02:59:35.781197 (+ 99465us) service_pool.cc:225] Handling call
0129 02:59:35.810772 (+ 29575us) raft_consensus.cc:1471] Updating replica for 1 ops
0129 02:59:35.971960 (+161188us) raft_consensus.cc:1534] Early marking committed up to index 3
0129 02:59:35.972007 (+ 47us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0129 02:59:35.986768 (+ 14761us) log.cc:844] Serialized 26 byte log entry
0129 02:59:36.271677 (+284909us) raft_consensus.cc:1645] Marking committed up to 3
0129 02:59:36.271730 (+ 53us) raft_consensus.cc:1697] Filling consensus response to leader.
0129 02:59:36.271823 (+ 93us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0129 02:59:36.879787 (+607964us) raft_consensus.cc:1685] finished
0129 02:59:36.879819 (+ 32us) raft_consensus.cc:1691] UpdateReplicas() finished
0129 02:59:36.880141 (+ 322us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":26866,"raft.queue_time_us":39155,"raft.run_cpu_time_us":57,"raft.run_wall_time_us":57,"thread_start_us":338598,"threads_started":2,"wal-append.queue_time_us":405673}
W20260129 02:59:36.881052 20482 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260129 02:59:36.841624 22091 raft_consensus.cc:2804] T 4408006eb5174de1896b037cf3f18457 P 197b880fa39d432a9b472845876bfaf0 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260129 02:59:36.893285 22416 raft_consensus.cc:2749] T a644c02b16af47679824ccd060ac10fd P 974b802141bd4f13beffef2aa03f88f9 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260129 02:59:36.706363 22373 consensus_queue.cc:237] T 8055232cf0004d40a85f3bf983fe00a9 P 197b880fa39d432a9b472845876bfaf0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:59:36.772169 22061 raft_consensus.cc:515] T cb0b3f259ebc4310927338a8740e87b8 P 1c50de89a5a845668696dcb28c3badd6 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } }
I20260129 02:59:36.887955 22164 consensus_queue.cc:237] T 913fc7d751ed4b7388d889aac30a5f9b P 1c50de89a5a845668696dcb28c3badd6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
W20260129 02:59:36.904428 20480 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.211.4:40439, user_credentials={real_user=slave}} blocked reactor thread for 53597us
I20260129 02:59:36.759963 22395 raft_consensus.cc:515] T fd8311b779af4e1fa1402b2a02a602f2 P 1c50de89a5a845668696dcb28c3badd6 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } }
I20260129 02:59:36.879235 22396 raft_consensus.cc:493] T 368953c9102643b2a21f99af36b51905 P 974b802141bd4f13beffef2aa03f88f9 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260129 02:59:36.925858 20480 leader_election.cc:304] T 75b0b6e82ddc4f92b1fe1d94f9a0b73b P 974b802141bd4f13beffef2aa03f88f9 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 974b802141bd4f13beffef2aa03f88f9; no voters: 1c50de89a5a845668696dcb28c3badd6, 7a9daf5d42a84615a6383cfec9a10608
W20260129 02:59:36.905006 22296 consensus_meta.cc:287] T 4f5a77e94fef4b16a2ba7b067c8b144a P 974b802141bd4f13beffef2aa03f88f9: Time spent flushing consensus metadata: real 1.004s user 0.013s sys 0.000s
I20260129 02:59:36.927755 20480 leader_election.cc:304] T 00e79d8f25a34676b1d866eb2068a2b2 P 974b802141bd4f13beffef2aa03f88f9 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 974b802141bd4f13beffef2aa03f88f9; no voters: 197b880fa39d432a9b472845876bfaf0, 7a9daf5d42a84615a6383cfec9a10608
I20260129 02:59:36.891084 22091 raft_consensus.cc:493] T 4408006eb5174de1896b037cf3f18457 P 197b880fa39d432a9b472845876bfaf0 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 974b802141bd4f13beffef2aa03f88f9)
I20260129 02:59:36.900864 22292 leader_election.cc:290] T c891d2bff4ce4568861bbc992e91fbb0 P 1c50de89a5a845668696dcb28c3badd6 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 197b880fa39d432a9b472845876bfaf0 (127.18.211.1:42027), 974b802141bd4f13beffef2aa03f88f9 (127.18.211.2:34517)
W20260129 02:59:36.932524 20404 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260129 02:59:36.861310 22404 raft_consensus.cc:493] T 87d5fe48a44a408aaf03e894a67ecb2f P 974b802141bd4f13beffef2aa03f88f9 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 7a9daf5d42a84615a6383cfec9a10608)
I20260129 02:59:36.940626 20588 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "724a236524cb4e7ba3428880681d6cc2" candidate_uuid: "197b880fa39d432a9b472845876bfaf0" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1c50de89a5a845668696dcb28c3badd6" is_pre_election: true
I20260129 02:59:36.924333 21127 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "18f30ebaa4174fd28c958e72d0cc7e24" candidate_uuid: "197b880fa39d432a9b472845876bfaf0" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "7a9daf5d42a84615a6383cfec9a10608" is_pre_election: true
I20260129 02:59:36.944085 20441 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1ce4ddf5b10e4ae8b193b13b2e5936ee" candidate_uuid: "974b802141bd4f13beffef2aa03f88f9" candidate_term: 5 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "197b880fa39d432a9b472845876bfaf0" is_pre_election: true
I20260129 02:59:36.945577 20588 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e33a417cf9c54cf5af73a4ee3aac4d3a" candidate_uuid: "197b880fa39d432a9b472845876bfaf0" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "1c50de89a5a845668696dcb28c3badd6"
I20260129 02:59:36.947110 20588 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "cb4727386a3e48aca5d71cc6c611c197" candidate_uuid: "197b880fa39d432a9b472845876bfaf0" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1c50de89a5a845668696dcb28c3badd6"
W20260129 02:59:36.948179 20479 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260129 02:59:36.949008 20583 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a1a39dcd328e4268ae41b988c66b4a75" candidate_uuid: "197b880fa39d432a9b472845876bfaf0" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "1c50de89a5a845668696dcb28c3badd6" is_pre_election: true
I20260129 02:59:36.943397 22404 raft_consensus.cc:515] T 87d5fe48a44a408aaf03e894a67ecb2f P 974b802141bd4f13beffef2aa03f88f9 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } }
I20260129 02:59:36.950171 20583 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "cedce331e1584ac8b636c9b8a72b2ba0" candidate_uuid: "197b880fa39d432a9b472845876bfaf0" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "1c50de89a5a845668696dcb28c3badd6" is_pre_election: true
I20260129 02:59:36.950889 20510 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "75f45538dc704d9a8ec4ff8ad9f3ea5b" candidate_uuid: "1c50de89a5a845668696dcb28c3badd6" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "974b802141bd4f13beffef2aa03f88f9" is_pre_election: true
I20260129 02:59:36.951058 22404 leader_election.cc:290] T 87d5fe48a44a408aaf03e894a67ecb2f P 974b802141bd4f13beffef2aa03f88f9 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 197b880fa39d432a9b472845876bfaf0 (127.18.211.1:42027), 7a9daf5d42a84615a6383cfec9a10608 (127.18.211.4:40439)
W20260129 02:59:36.951659 22407 log.cc:927] Time spent T 75f45538dc704d9a8ec4ff8ad9f3ea5b P 1c50de89a5a845668696dcb28c3badd6: Append to log took a long time: real 0.085s user 0.001s sys 0.000s
I20260129 02:59:36.951915 22385 leader_election.cc:290] T 7d9494f5584d4d5db8b9adf1c8fc888b P 197b880fa39d432a9b472845876bfaf0 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 974b802141bd4f13beffef2aa03f88f9 (127.18.211.2:34517), 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3:37735)
I20260129 02:59:36.933209 22091 raft_consensus.cc:3060] T 4408006eb5174de1896b037cf3f18457 P 197b880fa39d432a9b472845876bfaf0 [term 1 FOLLOWER]: Advancing to term 2
I20260129 02:59:36.954386 20441 raft_consensus.cc:2410] T 1ce4ddf5b10e4ae8b193b13b2e5936ee P 197b880fa39d432a9b472845876bfaf0 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 974b802141bd4f13beffef2aa03f88f9 for term 5 because replica has last-logged OpId of term: 2 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 1.
I20260129 02:59:36.951213 20583 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "18f30ebaa4174fd28c958e72d0cc7e24" candidate_uuid: "197b880fa39d432a9b472845876bfaf0" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "1c50de89a5a845668696dcb28c3badd6" is_pre_election: true
I20260129 02:59:36.955252 20583 raft_consensus.cc:2393] T 18f30ebaa4174fd28c958e72d0cc7e24 P 1c50de89a5a845668696dcb28c3badd6 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 197b880fa39d432a9b472845876bfaf0 in current term 3: Already voted for candidate 1c50de89a5a845668696dcb28c3badd6 in this term.
I20260129 02:59:36.947764 20588 raft_consensus.cc:2376] T cb4727386a3e48aca5d71cc6c611c197 P 1c50de89a5a845668696dcb28c3badd6 [term 2 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 197b880fa39d432a9b472845876bfaf0 in term 2. Re-sending same reply.
I20260129 02:59:36.959293 20441 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "96609ac68bdf4993a12e1c23355ca013" candidate_uuid: "1c50de89a5a845668696dcb28c3badd6" candidate_term: 5 candidate_status { last_received { term: 2 index: 3 } } ignore_live_leader: false dest_uuid: "197b880fa39d432a9b472845876bfaf0" is_pre_election: true
I20260129 02:59:36.959794 20583 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "78aa39096868415d9afc2ffcb248e198" candidate_uuid: "197b880fa39d432a9b472845876bfaf0" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "1c50de89a5a845668696dcb28c3badd6" is_pre_election: true
I20260129 02:59:36.956955 20586 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "36ff33be666a4c82954731691dbcf327" candidate_uuid: "197b880fa39d432a9b472845876bfaf0" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1c50de89a5a845668696dcb28c3badd6"
I20260129 02:59:36.961333 20583 raft_consensus.cc:1275] T 42d27a28d89b409e8807ec0efd97af69 P 1c50de89a5a845668696dcb28c3badd6 [term 4 FOLLOWER]: Refusing update from remote peer 197b880fa39d432a9b472845876bfaf0: Log matching property violated. Preceding OpId in replica: term: 3 index: 6. Preceding OpId from leader: term: 4 index: 6. (term mismatch)
I20260129 02:59:36.961750 20583 pending_rounds.cc:85] T 42d27a28d89b409e8807ec0efd97af69 P 1c50de89a5a845668696dcb28c3badd6: Aborting all ops after (but not including) 5
I20260129 02:59:36.962010 20583 pending_rounds.cc:107] T 42d27a28d89b409e8807ec0efd97af69 P 1c50de89a5a845668696dcb28c3badd6: Aborting uncommitted NO_OP operation due to leader change: 3.6
I20260129 02:59:36.962248 20583 raft_consensus.cc:2889] T 42d27a28d89b409e8807ec0efd97af69 P 1c50de89a5a845668696dcb28c3badd6 [term 4 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260129 02:59:36.938989 22389 consensus_queue.cc:1059] T 926a250fa5834e88b96ccecbefbf5086 P 1c50de89a5a845668696dcb28c3badd6 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 }, Status: INVALID_TERM, Last received: 4.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.025s
I20260129 02:59:36.960404 20588 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "368953c9102643b2a21f99af36b51905" candidate_uuid: "197b880fa39d432a9b472845876bfaf0" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "1c50de89a5a845668696dcb28c3badd6" is_pre_election: true
W20260129 02:59:36.963346 22405 log.cc:927] Time spent T 4f8f3a38f6e24ba4857875181f447442 P 197b880fa39d432a9b472845876bfaf0: Append to log took a long time: real 0.277s user 0.001s sys 0.000s
I20260129 02:59:36.964973 20516 raft_consensus.cc:1275] T 724a236524cb4e7ba3428880681d6cc2 P 974b802141bd4f13beffef2aa03f88f9 [term 2 FOLLOWER]: Refusing update from remote peer 1c50de89a5a845668696dcb28c3badd6: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260129 02:59:36.965190 20441 raft_consensus.cc:2410] T 96609ac68bdf4993a12e1c23355ca013 P 197b880fa39d432a9b472845876bfaf0 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1c50de89a5a845668696dcb28c3badd6 for term 5 because replica has last-logged OpId of term: 2 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 3.
I20260129 02:59:36.861791 22411 raft_consensus.cc:515] T 4b1876f9a70b49edb3aa74105e66d3b3 P 197b880fa39d432a9b472845876bfaf0 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:59:36.970942 20556 leader_election.cc:304] T e4d18c35b0a7493e85d0a4b92956a48f P 1c50de89a5a845668696dcb28c3badd6 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1c50de89a5a845668696dcb28c3badd6, 974b802141bd4f13beffef2aa03f88f9; no voters: 197b880fa39d432a9b472845876bfaf0
I20260129 02:59:36.955209 22296 raft_consensus.cc:2804] T c934d31efd3f4e5195605ddc354d14e1 P 974b802141bd4f13beffef2aa03f88f9 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260129 02:59:36.942689 22292 raft_consensus.cc:493] T 2e58e9a5a8aa4cecafb464addb4f35ed P 1c50de89a5a845668696dcb28c3badd6 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 7a9daf5d42a84615a6383cfec9a10608)
I20260129 02:59:36.979151 20405 leader_election.cc:304] T 724a236524cb4e7ba3428880681d6cc2 P 197b880fa39d432a9b472845876bfaf0 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 197b880fa39d432a9b472845876bfaf0; no voters: 1c50de89a5a845668696dcb28c3badd6, 974b802141bd4f13beffef2aa03f88f9
I20260129 02:59:36.982081 20437 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c891d2bff4ce4568861bbc992e91fbb0" candidate_uuid: "1c50de89a5a845668696dcb28c3badd6" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "197b880fa39d432a9b472845876bfaf0" is_pre_election: true
I20260129 02:59:36.986287 20516 raft_consensus.cc:1275] T 36ff33be666a4c82954731691dbcf327 P 974b802141bd4f13beffef2aa03f88f9 [term 2 FOLLOWER]: Refusing update from remote peer 1c50de89a5a845668696dcb28c3badd6: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260129 02:59:36.987825 20556 leader_election.cc:304] T 76f713c3331540d2a5b76164103e9eb1 P 1c50de89a5a845668696dcb28c3badd6 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1c50de89a5a845668696dcb28c3badd6, 974b802141bd4f13beffef2aa03f88f9; no voters: 197b880fa39d432a9b472845876bfaf0
I20260129 02:59:36.992206 20441 raft_consensus.cc:1275] T 724a236524cb4e7ba3428880681d6cc2 P 197b880fa39d432a9b472845876bfaf0 [term 2 FOLLOWER]: Refusing update from remote peer 1c50de89a5a845668696dcb28c3badd6: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260129 02:59:36.997943 22432 raft_consensus.cc:493] T dc2f4cb752cf4d59b5f6bc689f5d8104 P 197b880fa39d432a9b472845876bfaf0 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 7a9daf5d42a84615a6383cfec9a10608)
I20260129 02:59:36.998713 22432 raft_consensus.cc:515] T dc2f4cb752cf4d59b5f6bc689f5d8104 P 197b880fa39d432a9b472845876bfaf0 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } }
I20260129 02:59:36.931707 22396 raft_consensus.cc:515] T 368953c9102643b2a21f99af36b51905 P 974b802141bd4f13beffef2aa03f88f9 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:59:37.002116 20516 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c891d2bff4ce4568861bbc992e91fbb0" candidate_uuid: "1c50de89a5a845668696dcb28c3badd6" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "974b802141bd4f13beffef2aa03f88f9" is_pre_election: true
I20260129 02:59:36.899660 22401 raft_consensus.cc:515] T 7d9494f5584d4d5db8b9adf1c8fc888b P 974b802141bd4f13beffef2aa03f88f9 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
W20260129 02:59:37.005831 21099 leader_election.cc:336] T 1299723a4476420ab02fd8bfd7aa0645 P 7a9daf5d42a84615a6383cfec9a10608 [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer 197b880fa39d432a9b472845876bfaf0 (127.18.211.1:42027): Timed out: RequestConsensusVote RPC to 127.18.211.1:42027 timed out after 1.543s (SENT)
I20260129 02:59:37.006325 21099 leader_election.cc:304] T 1299723a4476420ab02fd8bfd7aa0645 P 7a9daf5d42a84615a6383cfec9a10608 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7a9daf5d42a84615a6383cfec9a10608; no voters: 197b880fa39d432a9b472845876bfaf0, 1c50de89a5a845668696dcb28c3badd6
I20260129 02:59:37.006762 20508 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57584 (request call id 1199) took 1317 ms. Trace:
W20260129 02:59:37.007849 21102 leader_election.cc:336] T 75b0b6e82ddc4f92b1fe1d94f9a0b73b P 7a9daf5d42a84615a6383cfec9a10608 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 974b802141bd4f13beffef2aa03f88f9 (127.18.211.2:34517): Timed out: RequestConsensusVote RPC to 127.18.211.2:34517 timed out after 1.594s (SENT)
I20260129 02:59:36.986450 22296 raft_consensus.cc:493] T c934d31efd3f4e5195605ddc354d14e1 P 974b802141bd4f13beffef2aa03f88f9 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260129 02:59:36.985996 22391 leader_election.cc:290] T cb0b3f259ebc4310927338a8740e87b8 P 7a9daf5d42a84615a6383cfec9a10608 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 974b802141bd4f13beffef2aa03f88f9 (127.18.211.2:34517), 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3:37735)
I20260129 02:59:37.007089 20508 rpcz_store.cc:276] 0129 02:59:35.683027 (+ 0us) service_pool.cc:168] Inserting onto call queue
0129 02:59:35.683155 (+ 128us) service_pool.cc:225] Handling call
0129 02:59:35.704477 (+ 21322us) raft_consensus.cc:1471] Updating replica for 2 ops
0129 02:59:35.729353 (+ 24876us) raft_consensus.cc:1534] Early marking committed up to index 0
0129 02:59:35.729381 (+ 28us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0129 02:59:35.777522 (+ 48141us) log.cc:844] Serialized 52 byte log entry
0129 02:59:35.830734 (+ 53212us) spinlock_profiling.cc:243] Waited 8.88 ms on lock 0x7b1800256fc0. stack: 00007f8fe044e373 00007f8fe044e270 00007f8fdfda0574 00007f8fe9a5c6b4 00007f8fe9a5c649 00007f8fe92d4c91 00007f8fe92e1671 00007f8fe0462c67 00007f8fe924b709 00007f8fe047f57b 00007f8fe047d81f 00007f8fe047fae2 00007f8fe30c75ab 00007f8fe30cd0c3 00007f8fe30cd471 00007f8fe34e973f
0129 02:59:36.086165 (+255431us) raft_consensus.cc:1645] Marking committed up to 3
0129 02:59:36.086201 (+ 36us) raft_consensus.cc:1697] Filling consensus response to leader.
0129 02:59:36.086268 (+ 67us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0129 02:59:36.989340 (+903072us) raft_consensus.cc:1685] finished
0129 02:59:36.989374 (+ 34us) raft_consensus.cc:1691] UpdateReplicas() finished
0129 02:59:37.000809 (+ 11435us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":17093,"raft.run_cpu_time_us":175,"raft.run_wall_time_us":8837,"spinlock_wait_cycles":20434176,"thread_start_us":255234,"threads_started":1,"wal-append.queue_time_us":414574}
I20260129 02:59:37.016237 20516 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "96609ac68bdf4993a12e1c23355ca013" candidate_uuid: "1c50de89a5a845668696dcb28c3badd6" candidate_term: 5 candidate_status { last_received { term: 2 index: 3 } } ignore_live_leader: false dest_uuid: "974b802141bd4f13beffef2aa03f88f9" is_pre_election: true
I20260129 02:59:37.021106 20437 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "87d5fe48a44a408aaf03e894a67ecb2f" candidate_uuid: "974b802141bd4f13beffef2aa03f88f9" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "197b880fa39d432a9b472845876bfaf0" is_pre_election: true
W20260129 02:59:37.025588 20480 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.211.4:40439, user_credentials={real_user=slave}} blocked reactor thread for 97927.5us
I20260129 02:59:37.028055 20441 raft_consensus.cc:1240] T fcddab6bca684adf8ea032289a00f90e P 197b880fa39d432a9b472845876bfaf0 [term 7 FOLLOWER]: Rejecting Update request from peer 1c50de89a5a845668696dcb28c3badd6 for earlier term 6. Current term is 7. Ops: [6.5-6.5]
W20260129 02:59:37.030129 21102 leader_election.cc:336] T 75f45538dc704d9a8ec4ff8ad9f3ea5b P 7a9daf5d42a84615a6383cfec9a10608 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 974b802141bd4f13beffef2aa03f88f9 (127.18.211.2:34517): Timed out: RequestConsensusVote RPC to 127.18.211.2:34517 timed out after 1.818s (SENT)
I20260129 02:59:37.041939 20441 raft_consensus.cc:1275] T 36ff33be666a4c82954731691dbcf327 P 197b880fa39d432a9b472845876bfaf0 [term 2 FOLLOWER]: Refusing update from remote peer 1c50de89a5a845668696dcb28c3badd6: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260129 02:59:37.003417 22299 leader_election.cc:290] T a644c02b16af47679824ccd060ac10fd P 1c50de89a5a845668696dcb28c3badd6 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 974b802141bd4f13beffef2aa03f88f9 (127.18.211.2:34517), 7a9daf5d42a84615a6383cfec9a10608 (127.18.211.4:40439)
I20260129 02:59:37.011976 22406 raft_consensus.cc:2749] T add8d4923d1a489e86882c3a25692b95 P 974b802141bd4f13beffef2aa03f88f9 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260129 02:59:37.048561 20588 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "cb0b3f259ebc4310927338a8740e87b8" candidate_uuid: "7a9daf5d42a84615a6383cfec9a10608" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "1c50de89a5a845668696dcb28c3badd6" is_pre_election: true
I20260129 02:59:37.049389 20588 raft_consensus.cc:2468] T cb0b3f259ebc4310927338a8740e87b8 P 1c50de89a5a845668696dcb28c3badd6 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7a9daf5d42a84615a6383cfec9a10608 in term 4.
I20260129 02:59:36.994714 22292 raft_consensus.cc:515] T 2e58e9a5a8aa4cecafb464addb4f35ed P 1c50de89a5a845668696dcb28c3badd6 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } }
I20260129 02:59:37.014631 22296 raft_consensus.cc:3060] T c934d31efd3f4e5195605ddc354d14e1 P 974b802141bd4f13beffef2aa03f88f9 [term 3 FOLLOWER]: Advancing to term 4
I20260129 02:59:37.014639 22335 raft_consensus.cc:2749] T 724a236524cb4e7ba3428880681d6cc2 P 197b880fa39d432a9b472845876bfaf0 [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260129 02:59:37.057505 21100 leader_election.cc:304] T cb0b3f259ebc4310927338a8740e87b8 P 7a9daf5d42a84615a6383cfec9a10608 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1c50de89a5a845668696dcb28c3badd6, 7a9daf5d42a84615a6383cfec9a10608; no voters:
I20260129 02:59:37.021237 22430 raft_consensus.cc:493] T 4f8f3a38f6e24ba4857875181f447442 P 974b802141bd4f13beffef2aa03f88f9 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260129 02:59:37.032135 22077 raft_consensus.cc:2749] T 17d6431cc77a491e8464d68db4081a51 P 197b880fa39d432a9b472845876bfaf0 [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260129 02:59:37.040994 22388 raft_consensus.cc:493] T 4f5a77e94fef4b16a2ba7b067c8b144a P 1c50de89a5a845668696dcb28c3badd6 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260129 02:59:37.033982 21128 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a644c02b16af47679824ccd060ac10fd" candidate_uuid: "1c50de89a5a845668696dcb28c3badd6" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "7a9daf5d42a84615a6383cfec9a10608" is_pre_election: true
W20260129 02:59:37.059480 22399 log.cc:927] Time spent T 4e3805ffa70242e8ade66da260dddf66 P 197b880fa39d432a9b472845876bfaf0: Append to log took a long time: real 0.352s user 0.001s sys 0.000s
I20260129 02:59:37.067689 20480 leader_election.cc:304] T a609ac02fae84725ab743f95f51d33ce P 974b802141bd4f13beffef2aa03f88f9 [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 974b802141bd4f13beffef2aa03f88f9; no voters: 197b880fa39d432a9b472845876bfaf0, 7a9daf5d42a84615a6383cfec9a10608
I20260129 02:59:37.096910 20440 raft_consensus.cc:1240] T 17d6431cc77a491e8464d68db4081a51 P 197b880fa39d432a9b472845876bfaf0 [term 2 FOLLOWER]: Rejecting Update request from peer 974b802141bd4f13beffef2aa03f88f9 for earlier term 1. Current term is 2. Ops: []
I20260129 02:59:37.099764 21135 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "cb0b3f259ebc4310927338a8740e87b8" candidate_uuid: "1c50de89a5a845668696dcb28c3badd6" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "7a9daf5d42a84615a6383cfec9a10608" is_pre_election: true
I20260129 02:59:37.102753 22061 leader_election.cc:290] T cb0b3f259ebc4310927338a8740e87b8 P 1c50de89a5a845668696dcb28c3badd6 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 974b802141bd4f13beffef2aa03f88f9 (127.18.211.2:34517), 7a9daf5d42a84615a6383cfec9a10608 (127.18.211.4:40439)
I20260129 02:59:37.103485 22061 consensus_queue.cc:1048] T 724a236524cb4e7ba3428880681d6cc2 P 1c50de89a5a845668696dcb28c3badd6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260129 02:59:37.103717 22437 raft_consensus.cc:493] T 36ff33be666a4c82954731691dbcf327 P 974b802141bd4f13beffef2aa03f88f9 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260129 02:59:37.065447 22255 raft_consensus.cc:2804] T cb0b3f259ebc4310927338a8740e87b8 P 7a9daf5d42a84615a6383cfec9a10608 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260129 02:59:37.101233 21128 raft_consensus.cc:2468] T a644c02b16af47679824ccd060ac10fd P 7a9daf5d42a84615a6383cfec9a10608 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1c50de89a5a845668696dcb28c3badd6 in term 1.
I20260129 02:59:37.105841 22437 raft_consensus.cc:515] T 36ff33be666a4c82954731691dbcf327 P 974b802141bd4f13beffef2aa03f88f9 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:59:37.126986 22411 leader_election.cc:290] T 4b1876f9a70b49edb3aa74105e66d3b3 P 197b880fa39d432a9b472845876bfaf0 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 974b802141bd4f13beffef2aa03f88f9 (127.18.211.2:34517), 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3:37735)
W20260129 02:59:37.136250 20481 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260129 02:59:37.137892 20554 leader_election.cc:304] T a644c02b16af47679824ccd060ac10fd P 1c50de89a5a845668696dcb28c3badd6 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1c50de89a5a845668696dcb28c3badd6, 7a9daf5d42a84615a6383cfec9a10608; no voters:
I20260129 02:59:37.065642 22430 raft_consensus.cc:515] T 4f8f3a38f6e24ba4857875181f447442 P 974b802141bd4f13beffef2aa03f88f9 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:59:37.108592 22255 raft_consensus.cc:493] T cb0b3f259ebc4310927338a8740e87b8 P 7a9daf5d42a84615a6383cfec9a10608 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260129 02:59:37.144053 22437 leader_election.cc:290] T 36ff33be666a4c82954731691dbcf327 P 974b802141bd4f13beffef2aa03f88f9 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 197b880fa39d432a9b472845876bfaf0 (127.18.211.1:42027), 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3:37735)
I20260129 02:59:37.145071 22429 raft_consensus.cc:2749] T 00e79d8f25a34676b1d866eb2068a2b2 P 974b802141bd4f13beffef2aa03f88f9 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20260129 02:59:37.121589 22425 raft_consensus.cc:2804] T e4d18c35b0a7493e85d0a4b92956a48f P 1c50de89a5a845668696dcb28c3badd6 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260129 02:59:37.152047 22425 raft_consensus.cc:493] T e4d18c35b0a7493e85d0a4b92956a48f P 1c50de89a5a845668696dcb28c3badd6 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 974b802141bd4f13beffef2aa03f88f9)
I20260129 02:59:37.152588 22425 raft_consensus.cc:3060] T e4d18c35b0a7493e85d0a4b92956a48f P 1c50de89a5a845668696dcb28c3badd6 [term 2 FOLLOWER]: Advancing to term 3
I20260129 02:59:37.064257 22431 raft_consensus.cc:2749] T 75b0b6e82ddc4f92b1fe1d94f9a0b73b P 974b802141bd4f13beffef2aa03f88f9 [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260129 02:59:37.096377 22388 raft_consensus.cc:515] T 4f5a77e94fef4b16a2ba7b067c8b144a P 1c50de89a5a845668696dcb28c3badd6 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } }
I20260129 02:59:37.165766 22437 raft_consensus.cc:2749] T a609ac02fae84725ab743f95f51d33ce P 974b802141bd4f13beffef2aa03f88f9 [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20260129 02:59:37.166783 20407 leader_election.cc:304] T 0678b8b468bd43cdbc3e485900844821 P 197b880fa39d432a9b472845876bfaf0 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 197b880fa39d432a9b472845876bfaf0, 974b802141bd4f13beffef2aa03f88f9; no voters: 7a9daf5d42a84615a6383cfec9a10608
I20260129 02:59:37.149741 22255 raft_consensus.cc:3060] T cb0b3f259ebc4310927338a8740e87b8 P 7a9daf5d42a84615a6383cfec9a10608 [term 4 FOLLOWER]: Advancing to term 5
W20260129 02:59:37.170164 20407 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.211.2:34517, user_credentials={real_user=slave}} blocked reactor thread for 963916us
I20260129 02:59:37.171118 20407 leader_election.cc:304] T fcddab6bca684adf8ea032289a00f90e P 197b880fa39d432a9b472845876bfaf0 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 197b880fa39d432a9b472845876bfaf0, 974b802141bd4f13beffef2aa03f88f9; no voters: 1c50de89a5a845668696dcb28c3badd6
W20260129 02:59:37.168960 21102 leader_election.cc:336] T a644c02b16af47679824ccd060ac10fd P 7a9daf5d42a84615a6383cfec9a10608 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 974b802141bd4f13beffef2aa03f88f9 (127.18.211.2:34517): Timed out: RequestConsensusVote RPC to 127.18.211.2:34517 timed out after 1.935s (SENT)
I20260129 02:59:37.140544 22421 raft_consensus.cc:493] T 724a236524cb4e7ba3428880681d6cc2 P 974b802141bd4f13beffef2aa03f88f9 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260129 02:59:37.172482 21102 leader_election.cc:304] T a644c02b16af47679824ccd060ac10fd P 7a9daf5d42a84615a6383cfec9a10608 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7a9daf5d42a84615a6383cfec9a10608; no voters: 1c50de89a5a845668696dcb28c3badd6, 974b802141bd4f13beffef2aa03f88f9
I20260129 02:59:37.156903 22395 leader_election.cc:290] T fd8311b779af4e1fa1402b2a02a602f2 P 1c50de89a5a845668696dcb28c3badd6 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 197b880fa39d432a9b472845876bfaf0 (127.18.211.1:42027), 7a9daf5d42a84615a6383cfec9a10608 (127.18.211.4:40439)
I20260129 02:59:37.176875 22411 raft_consensus.cc:2804] T fcddab6bca684adf8ea032289a00f90e P 197b880fa39d432a9b472845876bfaf0 [term 7 FOLLOWER]: Leader election won for term 7
I20260129 02:59:37.178896 22411 raft_consensus.cc:697] T fcddab6bca684adf8ea032289a00f90e P 197b880fa39d432a9b472845876bfaf0 [term 7 LEADER]: Becoming Leader. State: Replica: 197b880fa39d432a9b472845876bfaf0, State: Running, Role: LEADER
I20260129 02:59:37.179056 22430 leader_election.cc:290] T 4f8f3a38f6e24ba4857875181f447442 P 974b802141bd4f13beffef2aa03f88f9 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 197b880fa39d432a9b472845876bfaf0 (127.18.211.1:42027), 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3:37735)
I20260129 02:59:37.179747 22411 consensus_queue.cc:237] T fcddab6bca684adf8ea032289a00f90e P 197b880fa39d432a9b472845876bfaf0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 4.4, Last appended by leader: 4, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:59:37.165823 22401 leader_election.cc:290] T 7d9494f5584d4d5db8b9adf1c8fc888b P 974b802141bd4f13beffef2aa03f88f9 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 197b880fa39d432a9b472845876bfaf0 (127.18.211.1:42027), 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3:37735)
I20260129 02:59:37.184546 20433 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55676 (request call id 1145) took 1159 ms. Trace:
I20260129 02:59:37.186220 20433 rpcz_store.cc:276] 0129 02:59:36.025395 (+ 0us) service_pool.cc:168] Inserting onto call queue
0129 02:59:36.025882 (+ 487us) service_pool.cc:225] Handling call
0129 02:59:36.053595 (+ 27713us) raft_consensus.cc:1471] Updating replica for 1 ops
0129 02:59:36.139299 (+ 85704us) raft_consensus.cc:1534] Early marking committed up to index 0
0129 02:59:36.139333 (+ 34us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0129 02:59:36.190893 (+ 51560us) log.cc:844] Serialized 26 byte log entry
0129 02:59:36.394478 (+203585us) raft_consensus.cc:1645] Marking committed up to 5
0129 02:59:36.394543 (+ 65us) raft_consensus.cc:1697] Filling consensus response to leader.
0129 02:59:36.394645 (+ 102us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0129 02:59:37.184325 (+789680us) raft_consensus.cc:1685] finished
0129 02:59:37.184347 (+ 22us) raft_consensus.cc:1691] UpdateReplicas() finished
0129 02:59:37.184487 (+ 140us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":72256,"raft.queue_time_us":85695,"raft.run_cpu_time_us":222,"raft.run_wall_time_us":47188,"thread_start_us":122731,"threads_started":1,"wal-append.queue_time_us":258177}
W20260129 02:59:37.186939 20407 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260129 02:59:37.187688 20405 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.211.4:40439, user_credentials={real_user=slave}} blocked reactor thread for 184315us
I20260129 02:59:37.174405 21128 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fd8311b779af4e1fa1402b2a02a602f2" candidate_uuid: "1c50de89a5a845668696dcb28c3badd6" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "7a9daf5d42a84615a6383cfec9a10608" is_pre_election: true
I20260129 02:59:37.196739 22418 raft_consensus.cc:2804] T 76f713c3331540d2a5b76164103e9eb1 P 1c50de89a5a845668696dcb28c3badd6 [term 3 FOLLOWER]: Leader pre-election won for term 4
W20260129 02:59:37.202836 20432 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:48356 (request call id 238) took 1724 ms (client timeout 1543 ms). Trace:
I20260129 02:59:37.202991 22378 raft_consensus.cc:493] T 1ce4ddf5b10e4ae8b193b13b2e5936ee P 197b880fa39d432a9b472845876bfaf0 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260129 02:59:37.203379 20432 rpcz_store.cc:269] 0129 02:59:35.478673 (+ 0us) service_pool.cc:168] Inserting onto call queue
0129 02:59:35.547025 (+ 68352us) service_pool.cc:225] Handling call
0129 02:59:37.202653 (+1655628us) spinlock_profiling.cc:243] Waited 356 ms on lock 0x7b54003511d4. stack: 00007f8fe044e373 00007f8fe044e270 00007f8fdfda0574 00007f8fe9a5c6b4 00007f8fe9a5c649 00007f8fe34975a1 00007f8fe3531f7a 00007f8fe84e3647 00007f8fe1a73834 00007f8fe1a737c2 00007f8fe1a736f1 00007f8fe1a7366c 00007f8fe1a72922 00007f8fe186d34c 00007f8fe186c786 00007f8fe186c12f
0129 02:59:37.202767 (+ 114us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":818928384}
I20260129 02:59:37.203572 22378 raft_consensus.cc:515] T 1ce4ddf5b10e4ae8b193b13b2e5936ee P 197b880fa39d432a9b472845876bfaf0 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:59:37.182389 22391 raft_consensus.cc:2749] T a644c02b16af47679824ccd060ac10fd P 7a9daf5d42a84615a6383cfec9a10608 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260129 02:59:37.212270 21136 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "dc2f4cb752cf4d59b5f6bc689f5d8104" candidate_uuid: "197b880fa39d432a9b472845876bfaf0" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "7a9daf5d42a84615a6383cfec9a10608" is_pre_election: true
I20260129 02:59:37.194010 22421 raft_consensus.cc:515] T 724a236524cb4e7ba3428880681d6cc2 P 974b802141bd4f13beffef2aa03f88f9 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:59:37.184314 22395 raft_consensus.cc:2804] T a644c02b16af47679824ccd060ac10fd P 1c50de89a5a845668696dcb28c3badd6 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260129 02:59:37.217479 20433 raft_consensus.cc:1275] T 4b1876f9a70b49edb3aa74105e66d3b3 P 197b880fa39d432a9b472845876bfaf0 [term 4 FOLLOWER]: Refusing update from remote peer 974b802141bd4f13beffef2aa03f88f9: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
W20260129 02:59:37.193638 22368 consensus_meta.cc:287] T 1299723a4476420ab02fd8bfd7aa0645 P 7a9daf5d42a84615a6383cfec9a10608: Time spent flushing consensus metadata: real 1.213s user 0.012s sys 0.002s
I20260129 02:59:37.231551 21136 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2e58e9a5a8aa4cecafb464addb4f35ed" candidate_uuid: "1c50de89a5a845668696dcb28c3badd6" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "7a9daf5d42a84615a6383cfec9a10608" is_pre_election: true
I20260129 02:59:37.220592 22395 raft_consensus.cc:493] T a644c02b16af47679824ccd060ac10fd P 1c50de89a5a845668696dcb28c3badd6 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260129 02:59:37.228310 22421 leader_election.cc:290] T 724a236524cb4e7ba3428880681d6cc2 P 974b802141bd4f13beffef2aa03f88f9 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 197b880fa39d432a9b472845876bfaf0 (127.18.211.1:42027), 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3:37735)
I20260129 02:59:37.237663 21128 raft_consensus.cc:2393] T fd8311b779af4e1fa1402b2a02a602f2 P 7a9daf5d42a84615a6383cfec9a10608 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1c50de89a5a845668696dcb28c3badd6 in current term 3: Already voted for candidate 7a9daf5d42a84615a6383cfec9a10608 in this term.
W20260129 02:59:37.268327 20482 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260129 02:59:37.232532 22292 leader_election.cc:290] T 2e58e9a5a8aa4cecafb464addb4f35ed P 1c50de89a5a845668696dcb28c3badd6 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 197b880fa39d432a9b472845876bfaf0 (127.18.211.1:42027), 7a9daf5d42a84615a6383cfec9a10608 (127.18.211.4:40439)
I20260129 02:59:37.266467 22350 raft_consensus.cc:3055] T 4e420f90884f4796997b5ea41e49e411 P 7a9daf5d42a84615a6383cfec9a10608 [term 3 LEADER]: Stepping down as leader of term 3
W20260129 02:59:37.244168 22423 log.cc:927] Time spent T 00000000000000000000000000000000 P a91e6696d0a44ee19bcf2489fd709c38: Append to log took a long time: real 0.193s user 0.001s sys 0.000s
I20260129 02:59:37.256165 22427 raft_consensus.cc:2804] T 0678b8b468bd43cdbc3e485900844821 P 197b880fa39d432a9b472845876bfaf0 [term 5 FOLLOWER]: Leader election won for term 5
I20260129 02:59:37.229096 22368 raft_consensus.cc:515] T 1299723a4476420ab02fd8bfd7aa0645 P 7a9daf5d42a84615a6383cfec9a10608 [term 6 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } }
I20260129 02:59:37.257309 22077 raft_consensus.cc:493] T 87d5fe48a44a408aaf03e894a67ecb2f P 197b880fa39d432a9b472845876bfaf0 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 7a9daf5d42a84615a6383cfec9a10608)
I20260129 02:59:37.260087 22395 raft_consensus.cc:3060] T a644c02b16af47679824ccd060ac10fd P 1c50de89a5a845668696dcb28c3badd6 [term 2 FOLLOWER]: Advancing to term 3
I20260129 02:59:37.246222 22413 raft_consensus.cc:493] T 8055232cf0004d40a85f3bf983fe00a9 P 974b802141bd4f13beffef2aa03f88f9 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260129 02:59:37.296627 20482 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260129 02:59:37.297953 22378 leader_election.cc:290] T 1ce4ddf5b10e4ae8b193b13b2e5936ee P 197b880fa39d432a9b472845876bfaf0 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 974b802141bd4f13beffef2aa03f88f9 (127.18.211.2:34517), 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3:37735)
I20260129 02:59:37.299247 22296 raft_consensus.cc:515] T c934d31efd3f4e5195605ddc354d14e1 P 974b802141bd4f13beffef2aa03f88f9 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } }
I20260129 02:59:37.271644 22350 raft_consensus.cc:740] T 4e420f90884f4796997b5ea41e49e411 P 7a9daf5d42a84615a6383cfec9a10608 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 7a9daf5d42a84615a6383cfec9a10608, State: Running, Role: LEADER
I20260129 02:59:37.272274 22091 raft_consensus.cc:515] T 4408006eb5174de1896b037cf3f18457 P 197b880fa39d432a9b472845876bfaf0 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:59:37.283004 22396 leader_election.cc:290] T 368953c9102643b2a21f99af36b51905 P 974b802141bd4f13beffef2aa03f88f9 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 197b880fa39d432a9b472845876bfaf0 (127.18.211.1:42027), 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3:37735)
W20260129 02:59:37.309396 21100 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260129 02:59:37.307947 22434 log.cc:927] Time spent T 0f20af7fbe5d4db89a6f17948d98912c P 197b880fa39d432a9b472845876bfaf0: Append to log took a long time: real 0.167s user 0.000s sys 0.001s
I20260129 02:59:37.313498 22418 raft_consensus.cc:493] T 76f713c3331540d2a5b76164103e9eb1 P 1c50de89a5a845668696dcb28c3badd6 [term 3 FOLLOWER]: Starting leader election (detected failure of leader 197b880fa39d432a9b472845876bfaf0)
I20260129 02:59:37.313755 21128 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4f5a77e94fef4b16a2ba7b067c8b144a" candidate_uuid: "1c50de89a5a845668696dcb28c3badd6" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "7a9daf5d42a84615a6383cfec9a10608" is_pre_election: true
I20260129 02:59:37.314580 21128 raft_consensus.cc:2468] T 4f5a77e94fef4b16a2ba7b067c8b144a P 7a9daf5d42a84615a6383cfec9a10608 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1c50de89a5a845668696dcb28c3badd6 in term 2.
I20260129 02:59:37.314888 22409 raft_consensus.cc:3055] T 926a250fa5834e88b96ccecbefbf5086 P 1c50de89a5a845668696dcb28c3badd6 [term 4 LEADER]: Stepping down as leader of term 4
I20260129 02:59:37.315090 22419 raft_consensus.cc:2749] T 1299723a4476420ab02fd8bfd7aa0645 P 7a9daf5d42a84615a6383cfec9a10608 [term 6 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260129 02:59:37.316458 20554 leader_election.cc:304] T 4f5a77e94fef4b16a2ba7b067c8b144a P 1c50de89a5a845668696dcb28c3badd6 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1c50de89a5a845668696dcb28c3badd6, 7a9daf5d42a84615a6383cfec9a10608; no voters:
I20260129 02:59:37.314049 22418 raft_consensus.cc:3060] T 76f713c3331540d2a5b76164103e9eb1 P 1c50de89a5a845668696dcb28c3badd6 [term 3 FOLLOWER]: Advancing to term 4
I20260129 02:59:37.302203 22077 raft_consensus.cc:515] T 87d5fe48a44a408aaf03e894a67ecb2f P 197b880fa39d432a9b472845876bfaf0 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } }
I20260129 02:59:37.300012 22397 raft_consensus.cc:3055] T 2e58e9a5a8aa4cecafb464addb4f35ed P 7a9daf5d42a84615a6383cfec9a10608 [term 2 LEADER]: Stepping down as leader of term 2
I20260129 02:59:37.324093 22243 raft_consensus.cc:2804] T 4f5a77e94fef4b16a2ba7b067c8b144a P 1c50de89a5a845668696dcb28c3badd6 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260129 02:59:37.329262 22243 raft_consensus.cc:493] T 4f5a77e94fef4b16a2ba7b067c8b144a P 1c50de89a5a845668696dcb28c3badd6 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260129 02:59:37.329624 22243 raft_consensus.cc:3060] T 4f5a77e94fef4b16a2ba7b067c8b144a P 1c50de89a5a845668696dcb28c3badd6 [term 2 FOLLOWER]: Advancing to term 3
I20260129 02:59:37.329730 22077 leader_election.cc:290] T 87d5fe48a44a408aaf03e894a67ecb2f P 197b880fa39d432a9b472845876bfaf0 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 974b802141bd4f13beffef2aa03f88f9 (127.18.211.2:34517), 7a9daf5d42a84615a6383cfec9a10608 (127.18.211.4:40439)
I20260129 02:59:37.335037 20588 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1299723a4476420ab02fd8bfd7aa0645" candidate_uuid: "7a9daf5d42a84615a6383cfec9a10608" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "1c50de89a5a845668696dcb28c3badd6"
I20260129 02:59:37.335711 20588 raft_consensus.cc:3060] T 1299723a4476420ab02fd8bfd7aa0645 P 1c50de89a5a845668696dcb28c3badd6 [term 5 FOLLOWER]: Advancing to term 6
I20260129 02:59:37.239075 22335 raft_consensus.cc:493] T a4c1226b987b468fa84dfc6ba0e2dbe1 P 197b880fa39d432a9b472845876bfaf0 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260129 02:59:37.303026 22427 raft_consensus.cc:697] T 0678b8b468bd43cdbc3e485900844821 P 197b880fa39d432a9b472845876bfaf0 [term 5 LEADER]: Becoming Leader. State: Replica: 197b880fa39d432a9b472845876bfaf0, State: Running, Role: LEADER
I20260129 02:59:37.315379 22409 raft_consensus.cc:740] T 926a250fa5834e88b96ccecbefbf5086 P 1c50de89a5a845668696dcb28c3badd6 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 1c50de89a5a845668696dcb28c3badd6, State: Running, Role: LEADER
I20260129 02:59:37.316210 22388 leader_election.cc:290] T 4f5a77e94fef4b16a2ba7b067c8b144a P 1c50de89a5a845668696dcb28c3badd6 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 974b802141bd4f13beffef2aa03f88f9 (127.18.211.2:34517), 7a9daf5d42a84615a6383cfec9a10608 (127.18.211.4:40439)
W20260129 02:59:37.333823 22322 consensus_meta.cc:287] T c2633a6fc34c44579de348d208b0c01e P 197b880fa39d432a9b472845876bfaf0: Time spent flushing consensus metadata: real 1.299s user 0.007s sys 0.008s
I20260129 02:59:37.349922 22409 consensus_queue.cc:260] T 926a250fa5834e88b96ccecbefbf5086 P 1c50de89a5a845668696dcb28c3badd6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:59:37.350678 22091 leader_election.cc:290] T 4408006eb5174de1896b037cf3f18457 P 197b880fa39d432a9b472845876bfaf0 [CANDIDATE]: Term 2 election: Requested vote from peers 974b802141bd4f13beffef2aa03f88f9 (127.18.211.2:34517), 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3:37735)
I20260129 02:59:37.308089 22413 raft_consensus.cc:515] T 8055232cf0004d40a85f3bf983fe00a9 P 974b802141bd4f13beffef2aa03f88f9 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:59:37.351424 22409 raft_consensus.cc:3060] T 926a250fa5834e88b96ccecbefbf5086 P 1c50de89a5a845668696dcb28c3badd6 [term 4 FOLLOWER]: Advancing to term 5
I20260129 02:59:37.355545 20439 raft_consensus.cc:1275] T c2633a6fc34c44579de348d208b0c01e P 197b880fa39d432a9b472845876bfaf0 [term 2 FOLLOWER]: Refusing update from remote peer 1c50de89a5a845668696dcb28c3badd6: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260129 02:59:37.335505 22397 raft_consensus.cc:740] T 2e58e9a5a8aa4cecafb464addb4f35ed P 7a9daf5d42a84615a6383cfec9a10608 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 7a9daf5d42a84615a6383cfec9a10608, State: Running, Role: LEADER
I20260129 02:59:37.356410 22326 raft_consensus.cc:493] T c2633a6fc34c44579de348d208b0c01e P 197b880fa39d432a9b472845876bfaf0 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260129 02:59:37.336686 22368 leader_election.cc:290] T 1299723a4476420ab02fd8bfd7aa0645 P 7a9daf5d42a84615a6383cfec9a10608 [CANDIDATE]: Term 6 election: Requested vote from peers 197b880fa39d432a9b472845876bfaf0 (127.18.211.1:42027), 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3:37735)
I20260129 02:59:37.357949 20433 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1299723a4476420ab02fd8bfd7aa0645" candidate_uuid: "7a9daf5d42a84615a6383cfec9a10608" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "197b880fa39d432a9b472845876bfaf0" is_pre_election: true
I20260129 02:59:37.360519 20433 raft_consensus.cc:2468] T 1299723a4476420ab02fd8bfd7aa0645 P 197b880fa39d432a9b472845876bfaf0 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7a9daf5d42a84615a6383cfec9a10608 in term 5.
W20260129 02:59:37.357192 22355 consensus_meta.cc:287] T b189f08e16a74a11980cc55d99359a0a P 974b802141bd4f13beffef2aa03f88f9: Time spent flushing consensus metadata: real 1.165s user 0.011s sys 0.003s
I20260129 02:59:37.343277 22296 leader_election.cc:290] T c934d31efd3f4e5195605ddc354d14e1 P 974b802141bd4f13beffef2aa03f88f9 [CANDIDATE]: Term 4 election: Requested vote from peers 197b880fa39d432a9b472845876bfaf0 (127.18.211.1:42027), 7a9daf5d42a84615a6383cfec9a10608 (127.18.211.4:40439)
I20260129 02:59:37.361843 22355 raft_consensus.cc:515] T b189f08e16a74a11980cc55d99359a0a P 974b802141bd4f13beffef2aa03f88f9 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
W20260129 02:59:37.373013 21102 leader_election.cc:336] T 4f5a77e94fef4b16a2ba7b067c8b144a P 7a9daf5d42a84615a6383cfec9a10608 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 974b802141bd4f13beffef2aa03f88f9 (127.18.211.2:34517): Timed out: RequestConsensusVote RPC to 127.18.211.2:34517 timed out after 1.943s (SENT)
I20260129 02:59:37.373631 21102 leader_election.cc:304] T 4f5a77e94fef4b16a2ba7b067c8b144a P 7a9daf5d42a84615a6383cfec9a10608 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7a9daf5d42a84615a6383cfec9a10608; no voters: 1c50de89a5a845668696dcb28c3badd6, 974b802141bd4f13beffef2aa03f88f9
I20260129 02:59:37.384806 22296 raft_consensus.cc:493] T 7a9a69cd82724b1faed1d10aa091fc4a P 974b802141bd4f13beffef2aa03f88f9 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260129 02:59:37.385702 22296 raft_consensus.cc:515] T 7a9a69cd82724b1faed1d10aa091fc4a P 974b802141bd4f13beffef2aa03f88f9 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:59:37.388603 20588 raft_consensus.cc:2468] T 1299723a4476420ab02fd8bfd7aa0645 P 1c50de89a5a845668696dcb28c3badd6 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7a9daf5d42a84615a6383cfec9a10608 in term 6.
W20260129 02:59:37.368942 22213 consensus_meta.cc:287] T e33a417cf9c54cf5af73a4ee3aac4d3a P 1c50de89a5a845668696dcb28c3badd6: Time spent flushing consensus metadata: real 0.956s user 0.012s sys 0.001s
I20260129 02:59:37.356841 22397 consensus_queue.cc:260] T 2e58e9a5a8aa4cecafb464addb4f35ed P 7a9daf5d42a84615a6383cfec9a10608 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } }
I20260129 02:59:37.391825 22432 leader_election.cc:290] T dc2f4cb752cf4d59b5f6bc689f5d8104 P 197b880fa39d432a9b472845876bfaf0 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3:37735), 7a9daf5d42a84615a6383cfec9a10608 (127.18.211.4:40439)
I20260129 02:59:37.392746 22112 raft_consensus.cc:3055] T 75f45538dc704d9a8ec4ff8ad9f3ea5b P 974b802141bd4f13beffef2aa03f88f9 [term 1 LEADER]: Stepping down as leader of term 1
I20260129 02:59:37.393646 20434 raft_consensus.cc:1240] T a609ac02fae84725ab743f95f51d33ce P 197b880fa39d432a9b472845876bfaf0 [term 5 FOLLOWER]: Rejecting Update request from peer 7a9daf5d42a84615a6383cfec9a10608 for earlier term 4. Current term is 5. Ops: []
I20260129 02:59:37.400067 21100 leader_election.cc:304] T 1299723a4476420ab02fd8bfd7aa0645 P 7a9daf5d42a84615a6383cfec9a10608 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1c50de89a5a845668696dcb28c3badd6, 7a9daf5d42a84615a6383cfec9a10608; no voters:
W20260129 02:59:37.400655 20404 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260129 02:59:37.336074 22350 consensus_queue.cc:260] T 4e420f90884f4796997b5ea41e49e411 P 7a9daf5d42a84615a6383cfec9a10608 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.5, Last appended by leader: 5, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } }
I20260129 02:59:37.356896 22326 raft_consensus.cc:515] T c2633a6fc34c44579de348d208b0c01e P 197b880fa39d432a9b472845876bfaf0 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:59:37.388396 22368 raft_consensus.cc:2749] T 4f5a77e94fef4b16a2ba7b067c8b144a P 7a9daf5d42a84615a6383cfec9a10608 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260129 02:59:37.407320 22391 raft_consensus.cc:2804] T 1299723a4476420ab02fd8bfd7aa0645 P 7a9daf5d42a84615a6383cfec9a10608 [term 6 FOLLOWER]: Leader election won for term 6
I20260129 02:59:37.403155 22112 raft_consensus.cc:740] T 75f45538dc704d9a8ec4ff8ad9f3ea5b P 974b802141bd4f13beffef2aa03f88f9 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 974b802141bd4f13beffef2aa03f88f9, State: Running, Role: LEADER
I20260129 02:59:37.371805 22255 raft_consensus.cc:515] T cb0b3f259ebc4310927338a8740e87b8 P 7a9daf5d42a84615a6383cfec9a10608 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } }
I20260129 02:59:37.406782 22296 leader_election.cc:290] T 7a9a69cd82724b1faed1d10aa091fc4a P 974b802141bd4f13beffef2aa03f88f9 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 197b880fa39d432a9b472845876bfaf0 (127.18.211.1:42027), 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3:37735)
I20260129 02:59:37.423961 22391 raft_consensus.cc:697] T 1299723a4476420ab02fd8bfd7aa0645 P 7a9daf5d42a84615a6383cfec9a10608 [term 6 LEADER]: Becoming Leader. State: Replica: 7a9daf5d42a84615a6383cfec9a10608, State: Running, Role: LEADER
I20260129 02:59:37.434326 22112 consensus_queue.cc:260] T 75f45538dc704d9a8ec4ff8ad9f3ea5b P 974b802141bd4f13beffef2aa03f88f9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } }
I20260129 02:59:37.414044 22397 raft_consensus.cc:3060] T 2e58e9a5a8aa4cecafb464addb4f35ed P 7a9daf5d42a84615a6383cfec9a10608 [term 2 FOLLOWER]: Advancing to term 3
W20260129 02:59:37.426518 22357 consensus_meta.cc:287] T a0fcc58d6fb6473cafbd9d34d5195269 P 974b802141bd4f13beffef2aa03f88f9: Time spent flushing consensus metadata: real 0.888s user 0.002s sys 0.011s
I20260129 02:59:37.348750 22335 raft_consensus.cc:515] T a4c1226b987b468fa84dfc6ba0e2dbe1 P 197b880fa39d432a9b472845876bfaf0 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:59:37.442885 22335 leader_election.cc:290] T a4c1226b987b468fa84dfc6ba0e2dbe1 P 197b880fa39d432a9b472845876bfaf0 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 974b802141bd4f13beffef2aa03f88f9 (127.18.211.2:34517), 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3:37735)
I20260129 02:59:37.444717 22395 raft_consensus.cc:515] T a644c02b16af47679824ccd060ac10fd P 1c50de89a5a845668696dcb28c3badd6 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } }
I20260129 02:59:37.446795 22395 leader_election.cc:290] T a644c02b16af47679824ccd060ac10fd P 1c50de89a5a845668696dcb28c3badd6 [CANDIDATE]: Term 3 election: Requested vote from peers 974b802141bd4f13beffef2aa03f88f9 (127.18.211.2:34517), 7a9daf5d42a84615a6383cfec9a10608 (127.18.211.4:40439)
I20260129 02:59:37.448226 21128 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a644c02b16af47679824ccd060ac10fd" candidate_uuid: "1c50de89a5a845668696dcb28c3badd6" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "7a9daf5d42a84615a6383cfec9a10608"
I20260129 02:59:37.448987 21128 raft_consensus.cc:3060] T a644c02b16af47679824ccd060ac10fd P 7a9daf5d42a84615a6383cfec9a10608 [term 1 FOLLOWER]: Advancing to term 3
I20260129 02:59:37.449148 22418 raft_consensus.cc:515] T 76f713c3331540d2a5b76164103e9eb1 P 1c50de89a5a845668696dcb28c3badd6 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
W20260129 02:59:37.453311 20553 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.211.1:42027, user_credentials={real_user=slave}} blocked reactor thread for 443381us
I20260129 02:59:37.371632 22427 consensus_queue.cc:237] T 0678b8b468bd43cdbc3e485900844821 P 197b880fa39d432a9b472845876bfaf0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } }
I20260129 02:59:37.422072 22243 raft_consensus.cc:515] T 4f5a77e94fef4b16a2ba7b067c8b144a P 1c50de89a5a845668696dcb28c3badd6 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } }
I20260129 02:59:37.440640 22350 raft_consensus.cc:3060] T 4e420f90884f4796997b5ea41e49e411 P 7a9daf5d42a84615a6383cfec9a10608 [term 3 FOLLOWER]: Advancing to term 4
I20260129 02:59:37.451824 22418 leader_election.cc:290] T 76f713c3331540d2a5b76164103e9eb1 P 1c50de89a5a845668696dcb28c3badd6 [CANDIDATE]: Term 4 election: Requested vote from peers 197b880fa39d432a9b472845876bfaf0 (127.18.211.1:42027), 974b802141bd4f13beffef2aa03f88f9 (127.18.211.2:34517)
W20260129 02:59:37.460894 21102 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260129 02:59:37.462888 21135 raft_consensus.cc:2393] T cb0b3f259ebc4310927338a8740e87b8 P 7a9daf5d42a84615a6383cfec9a10608 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1c50de89a5a845668696dcb28c3badd6 in current term 5: Already voted for candidate 7a9daf5d42a84615a6383cfec9a10608 in this term.
I20260129 02:59:37.490113 21136 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4f5a77e94fef4b16a2ba7b067c8b144a" candidate_uuid: "1c50de89a5a845668696dcb28c3badd6" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "7a9daf5d42a84615a6383cfec9a10608"
W20260129 02:59:37.459750 20553 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260129 02:59:37.498543 21136 raft_consensus.cc:3060] T 4f5a77e94fef4b16a2ba7b067c8b144a P 7a9daf5d42a84615a6383cfec9a10608 [term 2 FOLLOWER]: Advancing to term 3
W20260129 02:59:37.464689 21099 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260129 02:59:37.486485 22092 consensus_meta.cc:287] T e7d83d4ccf684e76acb29a0dc844e1b8 P 197b880fa39d432a9b472845876bfaf0: Time spent flushing consensus metadata: real 0.928s user 0.012s sys 0.002s
I20260129 02:59:37.501463 22092 raft_consensus.cc:2749] T e7d83d4ccf684e76acb29a0dc844e1b8 P 197b880fa39d432a9b472845876bfaf0 [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260129 02:59:37.485662 22391 consensus_queue.cc:237] T 1299723a4476420ab02fd8bfd7aa0645 P 7a9daf5d42a84615a6383cfec9a10608 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 4.6, Last appended by leader: 6, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } }
W20260129 02:59:37.502115 20482 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260129 02:59:37.492251 22243 leader_election.cc:290] T 4f5a77e94fef4b16a2ba7b067c8b144a P 1c50de89a5a845668696dcb28c3badd6 [CANDIDATE]: Term 3 election: Requested vote from peers 974b802141bd4f13beffef2aa03f88f9 (127.18.211.2:34517), 7a9daf5d42a84615a6383cfec9a10608 (127.18.211.4:40439)
I20260129 02:59:37.495133 20438 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55676 (request call id 1148) took 1255 ms. Trace:
I20260129 02:59:37.502944 20438 rpcz_store.cc:276] 0129 02:59:36.240095 (+ 0us) service_pool.cc:168] Inserting onto call queue
0129 02:59:36.240234 (+ 139us) service_pool.cc:225] Handling call
0129 02:59:36.271495 (+ 31261us) raft_consensus.cc:1471] Updating replica for 0 ops
0129 02:59:36.425073 (+153578us) raft_consensus.cc:1534] Early marking committed up to index 0
0129 02:59:36.425127 (+ 54us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0129 02:59:36.425189 (+ 62us) raft_consensus.cc:1645] Marking committed up to 3
0129 02:59:36.425233 (+ 44us) raft_consensus.cc:1697] Filling consensus response to leader.
0129 02:59:37.494883 (+1069650us) spinlock_profiling.cc:243] Waited 153 ms on lock 0x7b54000a78d4. stack: 00007f8fe044e373 00007f8fe044e270 00007f8fdfda0574 00007f8fe9a5c6b4 00007f8fe9a5c649 00007f8fe34975a1 00007f8fe352ea81 00007f8fe352d4be 00007f8fe84e1f49 00007f8fe1a6ed34 00007f8fe1a6ecc2 00007f8fe1a6ebf1 00007f8fe1a6eb6c 00007f8fe1a6de22 00007f8fe186d34c 00007f8fe186c786
0129 02:59:37.494937 (+ 54us) raft_consensus.cc:1691] UpdateReplicas() finished
0129 02:59:37.495072 (+ 135us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":352576256}
I20260129 02:59:37.438479 22413 leader_election.cc:290] T 8055232cf0004d40a85f3bf983fe00a9 P 974b802141bd4f13beffef2aa03f88f9 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 197b880fa39d432a9b472845876bfaf0 (127.18.211.1:42027), 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3:37735)
I20260129 02:59:37.505772 20434 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fd8311b779af4e1fa1402b2a02a602f2" candidate_uuid: "7a9daf5d42a84615a6383cfec9a10608" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "197b880fa39d432a9b472845876bfaf0"
W20260129 02:59:37.515367 20406 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260129 02:59:37.521066 20438 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "febffe4a8b394368b69b1d164308fb9a" candidate_uuid: "7a9daf5d42a84615a6383cfec9a10608" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "197b880fa39d432a9b472845876bfaf0"
W20260129 02:59:37.504889 22366 consensus_meta.cc:287] T add8d4923d1a489e86882c3a25692b95 P 7a9daf5d42a84615a6383cfec9a10608: Time spent flushing consensus metadata: real 0.644s user 0.011s sys 0.000s
W20260129 02:59:37.522420 20553 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260129 02:59:37.487946 22355 leader_election.cc:290] T b189f08e16a74a11980cc55d99359a0a P 974b802141bd4f13beffef2aa03f88f9 [CANDIDATE]: Term 5 election: Requested vote from peers 197b880fa39d432a9b472845876bfaf0 (127.18.211.1:42027), 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3:37735)
I20260129 02:59:37.524605 20587 raft_consensus.cc:2468] T e33a417cf9c54cf5af73a4ee3aac4d3a P 1c50de89a5a845668696dcb28c3badd6 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 974b802141bd4f13beffef2aa03f88f9 in term 4.
I20260129 02:59:37.536221 20582 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:48526 (request call id 166) took 1303 ms. Trace:
I20260129 02:59:37.457171 22357 raft_consensus.cc:515] T a0fcc58d6fb6473cafbd9d34d5195269 P 974b802141bd4f13beffef2aa03f88f9 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:59:37.536625 20582 rpcz_store.cc:276] 0129 02:59:36.232865 (+ 0us) service_pool.cc:168] Inserting onto call queue
0129 02:59:36.233087 (+ 222us) service_pool.cc:225] Handling call
0129 02:59:36.285139 (+ 52052us) raft_consensus.cc:1471] Updating replica for 1 ops
0129 02:59:36.444530 (+159391us) raft_consensus.cc:1534] Early marking committed up to index 0
0129 02:59:36.444565 (+ 35us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0129 02:59:36.445013 (+ 448us) log.cc:844] Serialized 26 byte log entry
0129 02:59:36.508952 (+ 63939us) raft_consensus.cc:1645] Marking committed up to 4
0129 02:59:36.509013 (+ 61us) raft_consensus.cc:1697] Filling consensus response to leader.
0129 02:59:37.525889 (+1016876us) spinlock_profiling.cc:243] Waited 111 ms on lock 0x7b5400087fd4. stack: 00007f8fe044e373 00007f8fe044e270 00007f8fdfda0574 00007f8fe9a5c6b4 00007f8fe9a5c649 00007f8fe34975a1 00007f8fe352ea81 00007f8fe352d4be 00007f8fe84e1f49 00007f8fe1a6ed34 00007f8fe1a6ecc2 00007f8fe1a6ebf1 00007f8fe1a6eb6c 00007f8fe1a6de22 00007f8fe186d34c 00007f8fe186c786
0129 02:59:37.525960 (+ 71us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0129 02:59:37.526026 (+ 66us) raft_consensus.cc:1685] finished
0129 02:59:37.526046 (+ 20us) raft_consensus.cc:1691] UpdateReplicas() finished
0129 02:59:37.536133 (+ 10087us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":38244,"raft.queue_time_us":35693,"raft.run_cpu_time_us":168,"raft.run_wall_time_us":15773,"spinlock_wait_cycles":256315776,"thread_start_us":28202,"threads_started":1,"wal-append.queue_time_us":215107}
I20260129 02:59:37.524958 22380 raft_consensus.cc:3055] T 3ccac82a01a2445fa5b0ce6c0fcf2295 P 197b880fa39d432a9b472845876bfaf0 [term 1 LEADER]: Stepping down as leader of term 1
I20260129 02:59:37.535028 22366 raft_consensus.cc:515] T add8d4923d1a489e86882c3a25692b95 P 7a9daf5d42a84615a6383cfec9a10608 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } }
I20260129 02:59:37.581642 22366 leader_election.cc:290] T add8d4923d1a489e86882c3a25692b95 P 7a9daf5d42a84615a6383cfec9a10608 [CANDIDATE]: Term 2 election: Requested vote from peers 974b802141bd4f13beffef2aa03f88f9 (127.18.211.2:34517), 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3:37735)
I20260129 02:59:37.582931 20582 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "add8d4923d1a489e86882c3a25692b95" candidate_uuid: "7a9daf5d42a84615a6383cfec9a10608" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "1c50de89a5a845668696dcb28c3badd6"
W20260129 02:59:37.567996 20556 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.579s user 0.003s sys 0.034s
W20260129 02:59:37.588034 20556 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.599s user 0.004s sys 0.034s
W20260129 02:59:37.588637 20556 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.211.2:34517, user_credentials={real_user=slave}} blocked reactor thread for 601175us
W20260129 02:59:37.589107 21100 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260129 02:59:37.465296 22425 raft_consensus.cc:515] T e4d18c35b0a7493e85d0a4b92956a48f P 1c50de89a5a845668696dcb28c3badd6 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
W20260129 02:59:37.560212 22373 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.507s user 0.038s sys 0.023s
I20260129 02:59:37.544344 22349 consensus_queue.cc:1059] T fcddab6bca684adf8ea032289a00f90e P 1c50de89a5a845668696dcb28c3badd6 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 }, Status: INVALID_TERM, Last received: 4.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.025s
I20260129 02:59:37.545053 22409 consensus_queue.cc:1048] T 36ff33be666a4c82954731691dbcf327 P 1c50de89a5a845668696dcb28c3badd6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.022s
W20260129 02:59:37.597656 20513 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.606s user 0.013s sys 0.020s
W20260129 02:59:37.600143 20513 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.608s user 0.014s sys 0.020s
I20260129 02:59:37.563061 22380 raft_consensus.cc:740] T 3ccac82a01a2445fa5b0ce6c0fcf2295 P 197b880fa39d432a9b472845876bfaf0 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 197b880fa39d432a9b472845876bfaf0, State: Running, Role: LEADER
I20260129 02:59:37.605459 22330 raft_consensus.cc:493] T 00e79d8f25a34676b1d866eb2068a2b2 P 197b880fa39d432a9b472845876bfaf0 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260129 02:59:37.615783 21100 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260129 02:59:37.621451 20589 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.797s user 0.018s sys 0.024s
W20260129 02:59:37.597785 22373 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.545s user 0.038s sys 0.023s
W20260129 02:59:37.621775 20589 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.798s user 0.018s sys 0.024s
W20260129 02:59:37.622457 20480 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.526s user 0.024s sys 0.056s
I20260129 02:59:37.554699 22342 consensus_queue.cc:1059] T a609ac02fae84725ab743f95f51d33ce P 7a9daf5d42a84615a6383cfec9a10608 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 }, Status: INVALID_TERM, Last received: 4.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.067s
W20260129 02:59:37.626883 20480 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.530s user 0.024s sys 0.056s
I20260129 02:59:37.627413 22342 raft_consensus.cc:3055] T a609ac02fae84725ab743f95f51d33ce P 7a9daf5d42a84615a6383cfec9a10608 [term 4 LEADER]: Stepping down as leader of term 4
I20260129 02:59:37.627823 22342 raft_consensus.cc:740] T a609ac02fae84725ab743f95f51d33ce P 7a9daf5d42a84615a6383cfec9a10608 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 7a9daf5d42a84615a6383cfec9a10608, State: Running, Role: LEADER
I20260129 02:59:37.627923 20586 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "cb0b3f259ebc4310927338a8740e87b8" candidate_uuid: "7a9daf5d42a84615a6383cfec9a10608" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "1c50de89a5a845668696dcb28c3badd6"
I20260129 02:59:37.628492 22425 leader_election.cc:290] T e4d18c35b0a7493e85d0a4b92956a48f P 1c50de89a5a845668696dcb28c3badd6 [CANDIDATE]: Term 3 election: Requested vote from peers 197b880fa39d432a9b472845876bfaf0 (127.18.211.1:42027), 974b802141bd4f13beffef2aa03f88f9 (127.18.211.2:34517)
I20260129 02:59:37.628571 20586 raft_consensus.cc:3060] T cb0b3f259ebc4310927338a8740e87b8 P 1c50de89a5a845668696dcb28c3badd6 [term 4 FOLLOWER]: Advancing to term 5
I20260129 02:59:37.633539 20582 raft_consensus.cc:2468] T add8d4923d1a489e86882c3a25692b95 P 1c50de89a5a845668696dcb28c3badd6 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7a9daf5d42a84615a6383cfec9a10608 in term 2.
W20260129 02:59:37.639513 20441 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.574s user 0.016s sys 0.012s
W20260129 02:59:37.641505 20480 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.211.4:40439, user_credentials={real_user=slave}} blocked reactor thread for 573877us
W20260129 02:59:37.641784 20441 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.577s user 0.016s sys 0.012s
I20260129 02:59:37.615322 22255 leader_election.cc:290] T cb0b3f259ebc4310927338a8740e87b8 P 7a9daf5d42a84615a6383cfec9a10608 [CANDIDATE]: Term 5 election: Requested vote from peers 974b802141bd4f13beffef2aa03f88f9 (127.18.211.2:34517), 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3:37735)
I20260129 02:59:37.647977 21100 leader_election.cc:304] T add8d4923d1a489e86882c3a25692b95 P 7a9daf5d42a84615a6383cfec9a10608 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1c50de89a5a845668696dcb28c3badd6, 7a9daf5d42a84615a6383cfec9a10608; no voters:
W20260129 02:59:37.800352 21100 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.211.3:37735, user_credentials={real_user=slave}} blocked reactor thread for 154458us
W20260129 02:59:37.801539 21100 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260129 02:59:37.680855 20482 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260129 02:59:37.622977 22426 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.558s user 0.002s sys 0.018s
I20260129 02:59:37.611383 22330 raft_consensus.cc:515] T 00e79d8f25a34676b1d866eb2068a2b2 P 197b880fa39d432a9b472845876bfaf0 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } }
I20260129 02:59:37.628786 22342 consensus_queue.cc:260] T a609ac02fae84725ab743f95f51d33ce P 7a9daf5d42a84615a6383cfec9a10608 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } }
W20260129 02:59:37.780474 20404 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260129 02:59:37.825800 20482 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260129 02:59:37.826814 20514 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "926a250fa5834e88b96ccecbefbf5086" candidate_uuid: "197b880fa39d432a9b472845876bfaf0" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "974b802141bd4f13beffef2aa03f88f9"
I20260129 02:59:37.827574 20514 raft_consensus.cc:3060] T 926a250fa5834e88b96ccecbefbf5086 P 974b802141bd4f13beffef2aa03f88f9 [term 4 FOLLOWER]: Advancing to term 5
I20260129 02:59:37.646540 20480 leader_election.cc:304] T 505580c93c694bbb8ea96fdf9b17ab46 P 974b802141bd4f13beffef2aa03f88f9 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7a9daf5d42a84615a6383cfec9a10608, 974b802141bd4f13beffef2aa03f88f9; no voters:
I20260129 02:59:37.649099 22409 consensus_queue.cc:1048] T 368953c9102643b2a21f99af36b51905 P 1c50de89a5a845668696dcb28c3badd6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.108s
I20260129 02:59:37.648000 22373 raft_consensus.cc:493] T e4d18c35b0a7493e85d0a4b92956a48f P 197b880fa39d432a9b472845876bfaf0 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260129 02:59:37.852113 20437 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1299723a4476420ab02fd8bfd7aa0645" candidate_uuid: "7a9daf5d42a84615a6383cfec9a10608" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "197b880fa39d432a9b472845876bfaf0"
I20260129 02:59:37.852747 20437 raft_consensus.cc:3060] T 1299723a4476420ab02fd8bfd7aa0645 P 197b880fa39d432a9b472845876bfaf0 [term 5 FOLLOWER]: Advancing to term 6
I20260129 02:59:37.853279 22342 raft_consensus.cc:3060] T a609ac02fae84725ab743f95f51d33ce P 7a9daf5d42a84615a6383cfec9a10608 [term 4 FOLLOWER]: Advancing to term 5
I20260129 02:59:37.857884 20440 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00e79d8f25a34676b1d866eb2068a2b2" candidate_uuid: "7a9daf5d42a84615a6383cfec9a10608" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "197b880fa39d432a9b472845876bfaf0"
W20260129 02:59:37.828876 20480 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.211.4:40439, user_credentials={real_user=slave}} blocked reactor thread for 182409us
I20260129 02:59:37.860100 20480 leader_election.cc:304] T 3ccac82a01a2445fa5b0ce6c0fcf2295 P 974b802141bd4f13beffef2aa03f88f9 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 974b802141bd4f13beffef2aa03f88f9; no voters: 197b880fa39d432a9b472845876bfaf0, 7a9daf5d42a84615a6383cfec9a10608
I20260129 02:59:37.620517 22432 raft_consensus.cc:493] T 2e58e9a5a8aa4cecafb464addb4f35ed P 197b880fa39d432a9b472845876bfaf0 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260129 02:59:37.803169 22390 log.cc:927] Time spent T 75be4bd6bd45474795fd472eff2953ac P 197b880fa39d432a9b472845876bfaf0: Append to log took a long time: real 0.108s user 0.000s sys 0.004s
I20260129 02:59:37.836695 22212 raft_consensus.cc:3060] T 4e3805ffa70242e8ade66da260dddf66 P 7a9daf5d42a84615a6383cfec9a10608 [term 2 FOLLOWER]: Advancing to term 3
I20260129 02:59:37.838577 22330 leader_election.cc:290] T 00e79d8f25a34676b1d866eb2068a2b2 P 197b880fa39d432a9b472845876bfaf0 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 974b802141bd4f13beffef2aa03f88f9 (127.18.211.2:34517), 7a9daf5d42a84615a6383cfec9a10608 (127.18.211.4:40439)
I20260129 02:59:37.824208 22257 raft_consensus.cc:2804] T add8d4923d1a489e86882c3a25692b95 P 7a9daf5d42a84615a6383cfec9a10608 [term 2 FOLLOWER]: Leader election won for term 2
W20260129 02:59:37.847492 21099 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260129 02:59:37.848374 22357 leader_election.cc:290] T a0fcc58d6fb6473cafbd9d34d5195269 P 974b802141bd4f13beffef2aa03f88f9 [CANDIDATE]: Term 3 election: Requested vote from peers 197b880fa39d432a9b472845876bfaf0 (127.18.211.1:42027), 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3:37735)
I20260129 02:59:37.849105 20441 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2e58e9a5a8aa4cecafb464addb4f35ed" candidate_uuid: "1c50de89a5a845668696dcb28c3badd6" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "197b880fa39d432a9b472845876bfaf0" is_pre_election: true
I20260129 02:59:37.851686 22373 raft_consensus.cc:515] T e4d18c35b0a7493e85d0a4b92956a48f P 197b880fa39d432a9b472845876bfaf0 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:59:37.646365 22349 raft_consensus.cc:493] T a0fcc58d6fb6473cafbd9d34d5195269 P 1c50de89a5a845668696dcb28c3badd6 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 197b880fa39d432a9b472845876bfaf0)
I20260129 02:59:37.810082 20438 raft_consensus.cc:2468] T febffe4a8b394368b69b1d164308fb9a P 197b880fa39d432a9b472845876bfaf0 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7a9daf5d42a84615a6383cfec9a10608 in term 2.
I20260129 02:59:37.885054 20438 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "76f713c3331540d2a5b76164103e9eb1" candidate_uuid: "1c50de89a5a845668696dcb28c3badd6" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "197b880fa39d432a9b472845876bfaf0"
I20260129 02:59:37.885486 21099 leader_election.cc:304] T febffe4a8b394368b69b1d164308fb9a P 7a9daf5d42a84615a6383cfec9a10608 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 197b880fa39d432a9b472845876bfaf0, 7a9daf5d42a84615a6383cfec9a10608; no voters:
I20260129 02:59:37.886847 20438 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fd8311b779af4e1fa1402b2a02a602f2" candidate_uuid: "1c50de89a5a845668696dcb28c3badd6" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "197b880fa39d432a9b472845876bfaf0" is_pre_election: true
I20260129 02:59:37.855229 20586 raft_consensus.cc:2468] T cb0b3f259ebc4310927338a8740e87b8 P 1c50de89a5a845668696dcb28c3badd6 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7a9daf5d42a84615a6383cfec9a10608 in term 5.
W20260129 02:59:37.813279 22426 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.757s user 0.003s sys 0.019s
W20260129 02:59:37.824542 22438 log.cc:927] Time spent T 4e420f90884f4796997b5ea41e49e411 P 197b880fa39d432a9b472845876bfaf0: Append to log took a long time: real 0.187s user 0.000s sys 0.005s
W20260129 02:59:37.889309 20507 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.651s user 0.009s sys 0.021s
I20260129 02:59:37.739905 22380 consensus_queue.cc:260] T 3ccac82a01a2445fa5b0ce6c0fcf2295 P 197b880fa39d432a9b472845876bfaf0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } }
W20260129 02:59:37.889890 20507 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.652s user 0.009s sys 0.021s
I20260129 02:59:37.890332 22380 raft_consensus.cc:3060] T 3ccac82a01a2445fa5b0ce6c0fcf2295 P 197b880fa39d432a9b472845876bfaf0 [term 1 FOLLOWER]: Advancing to term 2
I20260129 02:59:37.858664 20440 raft_consensus.cc:2376] T 00e79d8f25a34676b1d866eb2068a2b2 P 197b880fa39d432a9b472845876bfaf0 [term 4 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 7a9daf5d42a84615a6383cfec9a10608 in term 4. Re-sending same reply.
I20260129 02:59:37.866380 20480 leader_election.cc:304] T febffe4a8b394368b69b1d164308fb9a P 974b802141bd4f13beffef2aa03f88f9 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 974b802141bd4f13beffef2aa03f88f9; no voters: 197b880fa39d432a9b472845876bfaf0, 7a9daf5d42a84615a6383cfec9a10608
I20260129 02:59:37.811370 22366 raft_consensus.cc:493] T 78aa39096868415d9afc2ffcb248e198 P 7a9daf5d42a84615a6383cfec9a10608 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 1c50de89a5a845668696dcb28c3badd6)
I20260129 02:59:37.899451 22366 raft_consensus.cc:515] T 78aa39096868415d9afc2ffcb248e198 P 7a9daf5d42a84615a6383cfec9a10608 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } }
I20260129 02:59:37.902067 22366 leader_election.cc:290] T 78aa39096868415d9afc2ffcb248e198 P 7a9daf5d42a84615a6383cfec9a10608 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 197b880fa39d432a9b472845876bfaf0 (127.18.211.1:42027), 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3:37735)
W20260129 02:59:37.810989 22435 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.508s user 0.010s sys 0.022s
W20260129 02:59:37.902760 22435 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.665s user 0.010s sys 0.023s
I20260129 02:59:37.903986 22413 raft_consensus.cc:2749] T febffe4a8b394368b69b1d164308fb9a P 974b802141bd4f13beffef2aa03f88f9 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260129 02:59:37.879428 22257 raft_consensus.cc:697] T add8d4923d1a489e86882c3a25692b95 P 7a9daf5d42a84615a6383cfec9a10608 [term 2 LEADER]: Becoming Leader. State: Replica: 7a9daf5d42a84615a6383cfec9a10608, State: Running, Role: LEADER
I20260129 02:59:37.906966 22257 consensus_queue.cc:237] T add8d4923d1a489e86882c3a25692b95 P 7a9daf5d42a84615a6383cfec9a10608 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } }
W20260129 02:59:37.922948 20407 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260129 02:59:37.870431 22411 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.633s user 0.008s sys 0.036s
W20260129 02:59:37.930022 22411 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.693s user 0.009s sys 0.036s
I20260129 02:59:37.872366 20436 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55676 (request call id 1128) took 2900 ms. Trace:
I20260129 02:59:37.930938 20436 rpcz_store.cc:276] 0129 02:59:34.967812 (+ 0us) service_pool.cc:168] Inserting onto call queue
0129 02:59:35.107926 (+140114us) service_pool.cc:225] Handling call
0129 02:59:35.108315 (+ 389us) raft_consensus.cc:1471] Updating replica for 0 ops
0129 02:59:35.822156 (+713841us) spinlock_profiling.cc:243] Waited 6.3 ms on lock 0x7b54004117a0. stack: 00007f8fe044e373 00007f8fe044e270 00007f8fdfda0574 00007f8fe9a5c6b4 00007f8fe9a5c649 00007f8fe34975a1 00007f8fe34a5df0 00007f8fe352fe40 00007f8fe3530eb1 00007f8fe352da0f 00007f8fe352d4be 00007f8fe84e1f49 00007f8fe1a6ed34 00007f8fe1a6ecc2 00007f8fe1a6ebf1 00007f8fe1a6eb6c
0129 02:59:35.898659 (+ 76503us) raft_consensus.cc:1697] Filling consensus response to leader.
0129 02:59:37.853484 (+1954825us) spinlock_profiling.cc:243] Waited 461 ms on lock 0x7b5400323cd4. stack: 00007f8fe044e373 00007f8fe044e270 00007f8fdfda0574 00007f8fe9a5c6b4 00007f8fe9a5c649 00007f8fe34975a1 00007f8fe352ea81 00007f8fe352d4be 00007f8fe84e1f49 00007f8fe1a6ed34 00007f8fe1a6ecc2 00007f8fe1a6ebf1 00007f8fe1a6eb6c 00007f8fe1a6de22 00007f8fe186d34c 00007f8fe186c786
0129 02:59:37.864359 (+ 10875us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1073862400}
I20260129 02:59:37.932956 20436 raft_consensus.cc:1275] T 1ce4ddf5b10e4ae8b193b13b2e5936ee P 197b880fa39d432a9b472845876bfaf0 [term 4 FOLLOWER]: Refusing update from remote peer 1c50de89a5a845668696dcb28c3badd6: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
W20260129 02:59:37.934243 21099 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.211.1:42027, user_credentials={real_user=slave}} blocked reactor thread for 48868.8us
W20260129 02:59:37.936651 20404 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260129 02:59:37.856827 21136 raft_consensus.cc:2468] T 4f5a77e94fef4b16a2ba7b067c8b144a P 7a9daf5d42a84615a6383cfec9a10608 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1c50de89a5a845668696dcb28c3badd6 in term 3.
I20260129 02:59:37.874969 22448 raft_consensus.cc:2749] T 3ccac82a01a2445fa5b0ce6c0fcf2295 P 974b802141bd4f13beffef2aa03f88f9 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
W20260129 02:59:37.876143 20479 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.887s user 0.017s sys 0.042s
W20260129 02:59:37.941514 20479 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.955s user 0.018s sys 0.043s
W20260129 02:59:37.941768 20482 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260129 02:59:37.941819 20479 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.211.1:42027, user_credentials={real_user=slave}} blocked reactor thread for 963654us
W20260129 02:59:37.873250 22354 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.527s user 0.008s sys 0.036s
I20260129 02:59:37.948408 20554 leader_election.cc:304] T 4f5a77e94fef4b16a2ba7b067c8b144a P 1c50de89a5a845668696dcb28c3badd6 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1c50de89a5a845668696dcb28c3badd6, 7a9daf5d42a84615a6383cfec9a10608; no voters:
W20260129 02:59:37.950227 20405 thread.cc:641] dns-resolver [worker] (thread pool) Time spent creating pthread: real 0.728s user 0.000s sys 0.025s
W20260129 02:59:38.010205 20405 thread.cc:608] dns-resolver [worker] (thread pool) Time spent starting thread: real 0.788s user 0.001s sys 0.026s
W20260129 02:59:38.010602 20405 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.211.3:37735, user_credentials={real_user=slave}} blocked reactor thread for 788730us
I20260129 02:59:37.950731 20514 raft_consensus.cc:2468] T 926a250fa5834e88b96ccecbefbf5086 P 974b802141bd4f13beffef2aa03f88f9 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 197b880fa39d432a9b472845876bfaf0 in term 5.
W20260129 02:59:37.954818 21101 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260129 02:59:38.015561 21101 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260129 02:59:37.941313 20508 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.882s user 0.015s sys 0.039s
I20260129 02:59:37.941102 22355 raft_consensus.cc:2804] T 505580c93c694bbb8ea96fdf9b17ab46 P 974b802141bd4f13beffef2aa03f88f9 [term 3 FOLLOWER]: Leader pre-election won for term 4
W20260129 02:59:38.021435 20554 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.211.4:40439, user_credentials={real_user=slave}} blocked reactor thread for 80810.5us
I20260129 02:59:38.024797 20407 leader_election.cc:304] T 926a250fa5834e88b96ccecbefbf5086 P 197b880fa39d432a9b472845876bfaf0 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 197b880fa39d432a9b472845876bfaf0, 974b802141bd4f13beffef2aa03f88f9; no voters: 1c50de89a5a845668696dcb28c3badd6
W20260129 02:59:38.028121 20406 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260129 02:59:37.954962 22198 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.806s user 0.014s sys 0.048s
I20260129 02:59:37.872917 22432 raft_consensus.cc:515] T 2e58e9a5a8aa4cecafb464addb4f35ed P 197b880fa39d432a9b472845876bfaf0 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } }
I20260129 02:59:37.948087 22425 raft_consensus.cc:493] T 4b1876f9a70b49edb3aa74105e66d3b3 P 1c50de89a5a845668696dcb28c3badd6 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260129 02:59:38.034170 20508 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.975s user 0.018s sys 0.047s
I20260129 02:59:38.035969 20514 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a1a39dcd328e4268ae41b988c66b4a75" candidate_uuid: "197b880fa39d432a9b472845876bfaf0" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "974b802141bd4f13beffef2aa03f88f9" is_pre_election: true
W20260129 02:59:38.097143 20407 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.211.2:34517, user_credentials={real_user=slave}} blocked reactor thread for 72480us
W20260129 02:59:38.034802 20556 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.211.2:34517, user_credentials={real_user=slave}} blocked reactor thread for 444388us
I20260129 02:59:37.957028 22366 raft_consensus.cc:2804] T febffe4a8b394368b69b1d164308fb9a P 7a9daf5d42a84615a6383cfec9a10608 [term 2 FOLLOWER]: Leader election won for term 2
W20260129 02:59:38.049289 20404 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260129 02:59:38.007496 22354 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.677s user 0.009s sys 0.040s
I20260129 02:59:38.112932 20514 raft_consensus.cc:2468] T a1a39dcd328e4268ae41b988c66b4a75 P 974b802141bd4f13beffef2aa03f88f9 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 197b880fa39d432a9b472845876bfaf0 in term 2.
I20260129 02:59:38.006074 22450 raft_consensus.cc:493] T 0340229d521b432580ae926f6062e3f7 P 1c50de89a5a845668696dcb28c3badd6 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260129 02:59:38.091986 22198 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.959s user 0.022s sys 0.048s
I20260129 02:59:38.122126 20407 leader_election.cc:304] T a1a39dcd328e4268ae41b988c66b4a75 P 197b880fa39d432a9b472845876bfaf0 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 197b880fa39d432a9b472845876bfaf0, 974b802141bd4f13beffef2aa03f88f9; no voters:
I20260129 02:59:37.940505 22349 raft_consensus.cc:515] T a0fcc58d6fb6473cafbd9d34d5195269 P 1c50de89a5a845668696dcb28c3badd6 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:59:38.023718 20437 raft_consensus.cc:2468] T 1299723a4476420ab02fd8bfd7aa0645 P 197b880fa39d432a9b472845876bfaf0 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7a9daf5d42a84615a6383cfec9a10608 in term 6.
I20260129 02:59:38.052915 22435 raft_consensus.cc:2804] T 4f5a77e94fef4b16a2ba7b067c8b144a P 1c50de89a5a845668696dcb28c3badd6 [term 3 FOLLOWER]: Leader election won for term 3
I20260129 02:59:38.124157 20441 raft_consensus.cc:2393] T 2e58e9a5a8aa4cecafb464addb4f35ed P 197b880fa39d432a9b472845876bfaf0 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1c50de89a5a845668696dcb28c3badd6 in current term 3: Already voted for candidate 197b880fa39d432a9b472845876bfaf0 in this term.
I20260129 02:59:38.034816 22115 raft_consensus.cc:493] T 42d27a28d89b409e8807ec0efd97af69 P 1c50de89a5a845668696dcb28c3badd6 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260129 02:59:38.036253 22390 log.cc:927] Time spent T 75be4bd6bd45474795fd472eff2953ac P 197b880fa39d432a9b472845876bfaf0: Append to log took a long time: real 0.086s user 0.000s sys 0.001s
I20260129 02:59:38.111796 22330 raft_consensus.cc:2804] T 926a250fa5834e88b96ccecbefbf5086 P 197b880fa39d432a9b472845876bfaf0 [term 5 FOLLOWER]: Leader election won for term 5
I20260129 02:59:38.131405 20437 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "78aa39096868415d9afc2ffcb248e198" candidate_uuid: "7a9daf5d42a84615a6383cfec9a10608" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "197b880fa39d432a9b472845876bfaf0" is_pre_election: true
I20260129 02:59:38.113997 22448 raft_consensus.cc:493] T e4d18c35b0a7493e85d0a4b92956a48f P 974b802141bd4f13beffef2aa03f88f9 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260129 02:59:38.151477 20437 raft_consensus.cc:2468] T 78aa39096868415d9afc2ffcb248e198 P 197b880fa39d432a9b472845876bfaf0 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7a9daf5d42a84615a6383cfec9a10608 in term 3.
I20260129 02:59:38.146342 20514 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "cedce331e1584ac8b636c9b8a72b2ba0" candidate_uuid: "197b880fa39d432a9b472845876bfaf0" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "974b802141bd4f13beffef2aa03f88f9" is_pre_election: true
I20260129 02:59:38.158903 20514 raft_consensus.cc:2468] T cedce331e1584ac8b636c9b8a72b2ba0 P 974b802141bd4f13beffef2aa03f88f9 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 197b880fa39d432a9b472845876bfaf0 in term 4.
I20260129 02:59:38.128600 22366 raft_consensus.cc:697] T febffe4a8b394368b69b1d164308fb9a P 7a9daf5d42a84615a6383cfec9a10608 [term 2 LEADER]: Becoming Leader. State: Replica: 7a9daf5d42a84615a6383cfec9a10608, State: Running, Role: LEADER
I20260129 02:59:38.169853 20514 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4b1876f9a70b49edb3aa74105e66d3b3" candidate_uuid: "197b880fa39d432a9b472845876bfaf0" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "974b802141bd4f13beffef2aa03f88f9" is_pre_election: true
W20260129 02:59:38.170842 20405 leader_election.cc:336] T cb4727386a3e48aca5d71cc6c611c197 P 197b880fa39d432a9b472845876bfaf0 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3:37735): Timed out: RequestConsensusVote RPC to 127.18.211.3:37735 timed out after 1.581s (SENT)
I20260129 02:59:38.114717 22425 raft_consensus.cc:515] T 4b1876f9a70b49edb3aa74105e66d3b3 P 1c50de89a5a845668696dcb28c3badd6 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:59:38.134415 22348 raft_consensus.cc:3055] T fcddab6bca684adf8ea032289a00f90e P 1c50de89a5a845668696dcb28c3badd6 [term 6 LEADER]: Stepping down as leader of term 6
I20260129 02:59:38.172044 20405 leader_election.cc:304] T cb4727386a3e48aca5d71cc6c611c197 P 197b880fa39d432a9b472845876bfaf0 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 197b880fa39d432a9b472845876bfaf0; no voters: 1c50de89a5a845668696dcb28c3badd6, 974b802141bd4f13beffef2aa03f88f9
W20260129 02:59:38.136327 22391 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.630s user 0.007s sys 0.035s
I20260129 02:59:38.184751 22348 raft_consensus.cc:740] T fcddab6bca684adf8ea032289a00f90e P 1c50de89a5a845668696dcb28c3badd6 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 1c50de89a5a845668696dcb28c3badd6, State: Running, Role: LEADER
W20260129 02:59:38.185441 22403 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 1.085s user 0.018s sys 0.045s
W20260129 02:59:38.187088 20405 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.18.211.3:37735 timed out after 1.671s (SENT) [suppressed 41 similar messages]
I20260129 02:59:38.146768 22432 leader_election.cc:290] T 2e58e9a5a8aa4cecafb464addb4f35ed P 197b880fa39d432a9b472845876bfaf0 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3:37735), 7a9daf5d42a84615a6383cfec9a10608 (127.18.211.4:40439)
I20260129 02:59:38.147467 22120 raft_consensus.cc:2804] T a1a39dcd328e4268ae41b988c66b4a75 P 197b880fa39d432a9b472845876bfaf0 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260129 02:59:38.159394 22330 raft_consensus.cc:697] T 926a250fa5834e88b96ccecbefbf5086 P 197b880fa39d432a9b472845876bfaf0 [term 5 LEADER]: Becoming Leader. State: Replica: 197b880fa39d432a9b472845876bfaf0, State: Running, Role: LEADER
I20260129 02:59:38.125108 22450 raft_consensus.cc:515] T 0340229d521b432580ae926f6062e3f7 P 1c50de89a5a845668696dcb28c3badd6 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:59:38.202462 20514 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e33a417cf9c54cf5af73a4ee3aac4d3a" candidate_uuid: "197b880fa39d432a9b472845876bfaf0" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "974b802141bd4f13beffef2aa03f88f9"
W20260129 02:59:38.203660 20405 leader_election.cc:336] T 36ff33be666a4c82954731691dbcf327 P 197b880fa39d432a9b472845876bfaf0 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3:37735): Timed out: RequestConsensusVote RPC to 127.18.211.3:37735 timed out after 1.912s (SENT)
I20260129 02:59:38.206647 20405 leader_election.cc:304] T 36ff33be666a4c82954731691dbcf327 P 197b880fa39d432a9b472845876bfaf0 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 197b880fa39d432a9b472845876bfaf0; no voters: 1c50de89a5a845668696dcb28c3badd6, 974b802141bd4f13beffef2aa03f88f9
I20260129 02:59:38.146122 22357 consensus_queue.cc:1059] T 17d6431cc77a491e8464d68db4081a51 P 974b802141bd4f13beffef2aa03f88f9 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.043s
W20260129 02:59:38.207958 22444 log.cc:927] Time spent T 8055232cf0004d40a85f3bf983fe00a9 P 197b880fa39d432a9b472845876bfaf0: Append to log took a long time: real 0.234s user 0.001s sys 0.000s
W20260129 02:59:38.162741 21128 consensus_meta.cc:287] T a644c02b16af47679824ccd060ac10fd P 7a9daf5d42a84615a6383cfec9a10608: Time spent flushing consensus metadata: real 0.713s user 0.013s sys 0.002s
I20260129 02:59:38.217484 21128 raft_consensus.cc:2468] T a644c02b16af47679824ccd060ac10fd P 7a9daf5d42a84615a6383cfec9a10608 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1c50de89a5a845668696dcb28c3badd6 in term 3.
I20260129 02:59:38.196331 22465 raft_consensus.cc:493] T 17d6431cc77a491e8464d68db4081a51 P 1c50de89a5a845668696dcb28c3badd6 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260129 02:59:38.209352 20515 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57584 (request call id 1195) took 2533 ms. Trace:
W20260129 02:59:38.210793 20404 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260129 02:59:38.232129 20515 rpcz_store.cc:276] 0129 02:59:35.676544 (+ 0us) service_pool.cc:168] Inserting onto call queue
0129 02:59:35.680266 (+ 3722us) service_pool.cc:225] Handling call
0129 02:59:36.036478 (+356212us) raft_consensus.cc:1471] Updating replica for 1 ops
0129 02:59:36.037343 (+ 865us) raft_consensus.cc:1697] Filling consensus response to leader.
0129 02:59:38.209062 (+2171719us) spinlock_profiling.cc:243] Waited 329 ms on lock 0x7b5400128c50. stack: 00007f8fe044e373 00007f8fe044e270 00007f8fdfda0574 00007f8fe9a5c6b4 00007f8fe9a5c649 00007f8fe34975a1 00007f8fe352d52d 00007f8fe84e1f49 00007f8fe1a6ed34 00007f8fe1a6ecc2 00007f8fe1a6ebf1 00007f8fe1a6eb6c 00007f8fe1a6de22 00007f8fe186d34c 00007f8fe186c786 00007f8fe186c12f
0129 02:59:38.209274 (+ 212us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":757131648}
I20260129 02:59:38.188755 22380 raft_consensus.cc:2749] T cb4727386a3e48aca5d71cc6c611c197 P 197b880fa39d432a9b472845876bfaf0 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260129 02:59:38.201833 22120 raft_consensus.cc:493] T a1a39dcd328e4268ae41b988c66b4a75 P 197b880fa39d432a9b472845876bfaf0 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 1c50de89a5a845668696dcb28c3badd6)
I20260129 02:59:38.254927 21134 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "78aa39096868415d9afc2ffcb248e198" candidate_uuid: "197b880fa39d432a9b472845876bfaf0" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "7a9daf5d42a84615a6383cfec9a10608" is_pre_election: true
I20260129 02:59:38.260953 21134 raft_consensus.cc:2468] T 78aa39096868415d9afc2ffcb248e198 P 7a9daf5d42a84615a6383cfec9a10608 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 197b880fa39d432a9b472845876bfaf0 in term 3.
I20260129 02:59:38.261662 21134 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34060 (request call id 458) took 1374 ms. Trace:
I20260129 02:59:38.200542 22448 raft_consensus.cc:515] T e4d18c35b0a7493e85d0a4b92956a48f P 974b802141bd4f13beffef2aa03f88f9 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
W20260129 02:59:38.263686 22464 log.cc:927] Time spent T 505580c93c694bbb8ea96fdf9b17ab46 P 974b802141bd4f13beffef2aa03f88f9: Append to log took a long time: real 0.073s user 0.000s sys 0.001s
I20260129 02:59:38.173112 22373 leader_election.cc:290] T e4d18c35b0a7493e85d0a4b92956a48f P 197b880fa39d432a9b472845876bfaf0 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 974b802141bd4f13beffef2aa03f88f9 (127.18.211.2:34517), 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3:37735)
I20260129 02:59:38.207211 20514 raft_consensus.cc:2393] T e33a417cf9c54cf5af73a4ee3aac4d3a P 974b802141bd4f13beffef2aa03f88f9 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 197b880fa39d432a9b472845876bfaf0 in current term 4: Already voted for candidate 974b802141bd4f13beffef2aa03f88f9 in this term.
W20260129 02:59:38.204094 22403 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 1.360s user 0.018s sys 0.046s
I20260129 02:59:38.277814 20514 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:57600 (request call id 1678) took 1095 ms. Trace:
I20260129 02:59:38.278323 20514 rpcz_store.cc:276] 0129 02:59:37.182041 (+ 0us) service_pool.cc:168] Inserting onto call queue
0129 02:59:38.190714 (+1008673us) service_pool.cc:225] Handling call
0129 02:59:38.277733 (+ 87019us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20260129 02:59:38.250911 22397 consensus_meta.cc:287] T 2e58e9a5a8aa4cecafb464addb4f35ed P 7a9daf5d42a84615a6383cfec9a10608: Time spent flushing consensus metadata: real 0.797s user 0.000s sys 0.008s
I20260129 02:59:38.122313 22445 consensus_queue.cc:1048] T 4b1876f9a70b49edb3aa74105e66d3b3 P 974b802141bd4f13beffef2aa03f88f9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.108s
W20260129 02:59:38.261679 20482 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260129 02:59:38.253975 22425 leader_election.cc:290] T 4b1876f9a70b49edb3aa74105e66d3b3 P 1c50de89a5a845668696dcb28c3badd6 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 197b880fa39d432a9b472845876bfaf0 (127.18.211.1:42027), 974b802141bd4f13beffef2aa03f88f9 (127.18.211.2:34517)
W20260129 02:59:38.189249 22391 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.680s user 0.007s sys 0.040s
I20260129 02:59:38.262399 21134 rpcz_store.cc:276] 0129 02:59:36.887139 (+ 0us) service_pool.cc:168] Inserting onto call queue
0129 02:59:36.887328 (+ 189us) service_pool.cc:225] Handling call
0129 02:59:38.254741 (+1367413us) spinlock_profiling.cc:243] Waited 4 ms on lock 0x7b5800111f98. stack: 00007f8fe044e373 00007f8fe044e270 00007f8fdfda0574 00007f8fe9a5c6b4 00007f8fe9a5c649 00007f8fe02c08b1 00007f8fe0442db1 00007f8fe0442c62 00007f8fe92eddd9 00007f8fe846d260 00007f8fe8535dbb 00007f8fe8535e41 00007f8fe84e3bd0 00007f8fe84e34b4 00007f8fe1a73834 00007f8fe1a737c2
0129 02:59:38.261602 (+ 6861us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":9211264}
I20260129 02:59:38.186793 20436 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e4d18c35b0a7493e85d0a4b92956a48f" candidate_uuid: "1c50de89a5a845668696dcb28c3badd6" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "197b880fa39d432a9b472845876bfaf0"
I20260129 02:59:38.185746 22447 raft_consensus.cc:493] T 18f30ebaa4174fd28c958e72d0cc7e24 P 1c50de89a5a845668696dcb28c3badd6 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260129 02:59:38.267313 22453 log.cc:927] Time spent T 75be4bd6bd45474795fd472eff2953ac P 1c50de89a5a845668696dcb28c3badd6: Append to log took a long time: real 0.223s user 0.000s sys 0.001s
I20260129 02:59:38.295770 22451 raft_consensus.cc:3055] T 17d6431cc77a491e8464d68db4081a51 P 974b802141bd4f13beffef2aa03f88f9 [term 1 LEADER]: Stepping down as leader of term 1
I20260129 02:59:38.296340 22451 raft_consensus.cc:740] T 17d6431cc77a491e8464d68db4081a51 P 974b802141bd4f13beffef2aa03f88f9 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 974b802141bd4f13beffef2aa03f88f9, State: Running, Role: LEADER
I20260129 02:59:38.245911 22120 raft_consensus.cc:3060] T a1a39dcd328e4268ae41b988c66b4a75 P 197b880fa39d432a9b472845876bfaf0 [term 2 FOLLOWER]: Advancing to term 3
W20260129 02:59:38.246563 22461 log.cc:927] Time spent T 4f8f3a38f6e24ba4857875181f447442 P 1c50de89a5a845668696dcb28c3badd6: Append to log took a long time: real 0.087s user 0.000s sys 0.000s
I20260129 02:59:38.253042 20509 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57584 (request call id 1216) took 1784 ms. Trace:
I20260129 02:59:38.270362 22212 raft_consensus.cc:493] T 2ac659d732314e1288849f08fe5e074e P 7a9daf5d42a84615a6383cfec9a10608 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260129 02:59:38.270570 21133 raft_consensus.cc:1275] T 4e3805ffa70242e8ade66da260dddf66 P 7a9daf5d42a84615a6383cfec9a10608 [term 3 FOLLOWER]: Refusing update from remote peer 1c50de89a5a845668696dcb28c3badd6: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 3 index: 6. (index mismatch)
I20260129 02:59:38.221769 22348 consensus_queue.cc:260] T fcddab6bca684adf8ea032289a00f90e P 1c50de89a5a845668696dcb28c3badd6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 6.5, Last appended by leader: 5, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:59:38.142951 22115 raft_consensus.cc:515] T 42d27a28d89b409e8807ec0efd97af69 P 1c50de89a5a845668696dcb28c3badd6 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } }
W20260129 02:59:38.263342 22350 consensus_meta.cc:287] T 4e420f90884f4796997b5ea41e49e411 P 7a9daf5d42a84615a6383cfec9a10608: Time spent flushing consensus metadata: real 0.776s user 0.008s sys 0.003s
I20260129 02:59:38.258862 20515 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "87d5fe48a44a408aaf03e894a67ecb2f" candidate_uuid: "197b880fa39d432a9b472845876bfaf0" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "974b802141bd4f13beffef2aa03f88f9" is_pre_election: true
I20260129 02:59:38.285248 20514 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7d9494f5584d4d5db8b9adf1c8fc888b" candidate_uuid: "197b880fa39d432a9b472845876bfaf0" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "974b802141bd4f13beffef2aa03f88f9" is_pre_election: true
I20260129 02:59:38.294847 22447 raft_consensus.cc:515] T 18f30ebaa4174fd28c958e72d0cc7e24 P 1c50de89a5a845668696dcb28c3badd6 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } }
I20260129 02:59:38.305747 20515 raft_consensus.cc:2468] T 87d5fe48a44a408aaf03e894a67ecb2f P 974b802141bd4f13beffef2aa03f88f9 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 197b880fa39d432a9b472845876bfaf0 in term 4.
I20260129 02:59:38.307137 20507 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57584 (request call id 1222) took 1357 ms. Trace:
I20260129 02:59:38.314358 20514 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:57600 (request call id 1686) took 1124 ms. Trace:
I20260129 02:59:38.296494 22320 consensus_queue.cc:1048] T 4f8f3a38f6e24ba4857875181f447442 P 1c50de89a5a845668696dcb28c3badd6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260129 02:59:38.314888 20514 rpcz_store.cc:276] 0129 02:59:37.189856 (+ 0us) service_pool.cc:168] Inserting onto call queue
0129 02:59:38.283372 (+1093516us) service_pool.cc:225] Handling call
0129 02:59:38.314288 (+ 30916us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260129 02:59:38.317329 20514 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "368953c9102643b2a21f99af36b51905" candidate_uuid: "197b880fa39d432a9b472845876bfaf0" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "974b802141bd4f13beffef2aa03f88f9" is_pre_election: true
I20260129 02:59:38.258015 22330 consensus_queue.cc:237] T 926a250fa5834e88b96ccecbefbf5086 P 197b880fa39d432a9b472845876bfaf0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:59:38.321595 20515 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00e79d8f25a34676b1d866eb2068a2b2" candidate_uuid: "197b880fa39d432a9b472845876bfaf0" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "974b802141bd4f13beffef2aa03f88f9" is_pre_election: true
I20260129 02:59:38.320317 20514 raft_consensus.cc:2468] T 368953c9102643b2a21f99af36b51905 P 974b802141bd4f13beffef2aa03f88f9 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 197b880fa39d432a9b472845876bfaf0 in term 4.
I20260129 02:59:38.303982 22414 raft_consensus.cc:493] T 4e3805ffa70242e8ade66da260dddf66 P 7a9daf5d42a84615a6383cfec9a10608 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260129 02:59:38.324971 20514 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:57600 (request call id 1685) took 1135 ms. Trace:
I20260129 02:59:38.310851 20507 rpcz_store.cc:276] 0129 02:59:36.949292 (+ 0us) service_pool.cc:168] Inserting onto call queue
0129 02:59:36.953792 (+ 4500us) service_pool.cc:225] Handling call
0129 02:59:36.973290 (+ 19498us) raft_consensus.cc:1471] Updating replica for 1 ops
0129 02:59:37.181245 (+207955us) raft_consensus.cc:1534] Early marking committed up to index 0
0129 02:59:37.181294 (+ 49us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0129 02:59:37.206966 (+ 25672us) log.cc:844] Serialized 26 byte log entry
0129 02:59:37.890395 (+683429us) raft_consensus.cc:1645] Marking committed up to 5
0129 02:59:37.890453 (+ 58us) raft_consensus.cc:1697] Filling consensus response to leader.
0129 02:59:37.890547 (+ 94us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0129 02:59:38.306882 (+416335us) raft_consensus.cc:1685] finished
0129 02:59:38.306911 (+ 29us) raft_consensus.cc:1691] UpdateReplicas() finished
0129 02:59:38.307064 (+ 153us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":20768,"raft.queue_time_us":189674,"raft.run_cpu_time_us":189,"raft.run_wall_time_us":1850,"thread_start_us":841560,"threads_started":2,"wal-append.queue_time_us":962793}
I20260129 02:59:38.322945 20515 raft_consensus.cc:2468] T 00e79d8f25a34676b1d866eb2068a2b2 P 974b802141bd4f13beffef2aa03f88f9 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 197b880fa39d432a9b472845876bfaf0 in term 4.
I20260129 02:59:38.326694 20514 rpcz_store.cc:276] 0129 02:59:37.189567 (+ 0us) service_pool.cc:168] Inserting onto call queue
0129 02:59:38.316979 (+1127412us) service_pool.cc:225] Handling call
0129 02:59:38.324895 (+ 7916us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260129 02:59:38.244549 22465 raft_consensus.cc:515] T 17d6431cc77a491e8464d68db4081a51 P 1c50de89a5a845668696dcb28c3badd6 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:59:38.312222 22349 leader_election.cc:290] T a0fcc58d6fb6473cafbd9d34d5195269 P 1c50de89a5a845668696dcb28c3badd6 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 197b880fa39d432a9b472845876bfaf0 (127.18.211.1:42027), 974b802141bd4f13beffef2aa03f88f9 (127.18.211.2:34517)
I20260129 02:59:38.318907 20436 raft_consensus.cc:2393] T e4d18c35b0a7493e85d0a4b92956a48f P 197b880fa39d432a9b472845876bfaf0 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 1c50de89a5a845668696dcb28c3badd6 in current term 3: Already voted for candidate 197b880fa39d432a9b472845876bfaf0 in this term.
I20260129 02:59:38.282680 22366 consensus_queue.cc:237] T febffe4a8b394368b69b1d164308fb9a P 7a9daf5d42a84615a6383cfec9a10608 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } }
I20260129 02:59:38.316469 22450 leader_election.cc:290] T 0340229d521b432580ae926f6062e3f7 P 1c50de89a5a845668696dcb28c3badd6 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 197b880fa39d432a9b472845876bfaf0 (127.18.211.1:42027), 974b802141bd4f13beffef2aa03f88f9 (127.18.211.2:34517)
W20260129 02:59:38.322261 22402 log.cc:927] Time spent T c934d31efd3f4e5195605ddc354d14e1 P 7a9daf5d42a84615a6383cfec9a10608: Append to log took a long time: real 1.669s user 0.002s sys 0.004s
W20260129 02:59:38.353533 20482 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260129 02:59:38.357556 20582 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.525s user 0.000s sys 0.046s
I20260129 02:59:38.358491 20507 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4408006eb5174de1896b037cf3f18457" candidate_uuid: "197b880fa39d432a9b472845876bfaf0" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "974b802141bd4f13beffef2aa03f88f9"
W20260129 02:59:38.358109 20513 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.527s user 0.014s sys 0.032s
W20260129 02:59:38.360394 20513 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.530s user 0.014s sys 0.033s
I20260129 02:59:38.360618 20515 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "75b0b6e82ddc4f92b1fe1d94f9a0b73b" candidate_uuid: "7a9daf5d42a84615a6383cfec9a10608" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "974b802141bd4f13beffef2aa03f88f9"
I20260129 02:59:38.362757 20515 raft_consensus.cc:2376] T 75b0b6e82ddc4f92b1fe1d94f9a0b73b P 974b802141bd4f13beffef2aa03f88f9 [term 2 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 7a9daf5d42a84615a6383cfec9a10608 in term 2. Re-sending same reply.
W20260129 02:59:38.359467 20582 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.527s user 0.000s sys 0.047s
I20260129 02:59:38.364244 20514 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "febffe4a8b394368b69b1d164308fb9a" candidate_uuid: "7a9daf5d42a84615a6383cfec9a10608" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "974b802141bd4f13beffef2aa03f88f9"
I20260129 02:59:38.365274 20514 raft_consensus.cc:2393] T febffe4a8b394368b69b1d164308fb9a P 974b802141bd4f13beffef2aa03f88f9 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 7a9daf5d42a84615a6383cfec9a10608 in current term 2: Already voted for candidate 974b802141bd4f13beffef2aa03f88f9 in this term.
W20260129 02:59:38.370651 20432 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.512s user 0.008s sys 0.012s
I20260129 02:59:38.328594 22466 raft_consensus.cc:493] T 75b0b6e82ddc4f92b1fe1d94f9a0b73b P 1c50de89a5a845668696dcb28c3badd6 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 7a9daf5d42a84615a6383cfec9a10608)
I20260129 02:59:38.403180 20515 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00e79d8f25a34676b1d866eb2068a2b2" candidate_uuid: "7a9daf5d42a84615a6383cfec9a10608" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "974b802141bd4f13beffef2aa03f88f9"
I20260129 02:59:38.405409 20515 raft_consensus.cc:2393] T 00e79d8f25a34676b1d866eb2068a2b2 P 974b802141bd4f13beffef2aa03f88f9 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 7a9daf5d42a84615a6383cfec9a10608 in current term 4: Already voted for candidate 974b802141bd4f13beffef2aa03f88f9 in this term.
I20260129 02:59:38.314424 22212 raft_consensus.cc:515] T 2ac659d732314e1288849f08fe5e074e P 7a9daf5d42a84615a6383cfec9a10608 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } }
I20260129 02:59:38.379771 20584 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35752 (request call id 1476) took 2299 ms. Trace:
I20260129 02:59:38.407970 20584 rpcz_store.cc:276] 0129 02:59:36.056535 (+ 0us) service_pool.cc:168] Inserting onto call queue
0129 02:59:36.086595 (+ 30060us) service_pool.cc:225] Handling call
0129 02:59:36.122864 (+ 36269us) raft_consensus.cc:1471] Updating replica for 0 ops
0129 02:59:36.197229 (+ 74365us) raft_consensus.cc:1697] Filling consensus response to leader.
0129 02:59:38.335022 (+2137793us) spinlock_profiling.cc:243] Waited 68.8 ms on lock 0x7b5400108954. stack: 00007f8fe044e373 00007f8fe044e270 00007f8fdfda0574 00007f8fe9a5c6b4 00007f8fe9a5c649 00007f8fe34975a1 00007f8fe352ea81 00007f8fe352d4be 00007f8fe84e1f49 00007f8fe1a6ed34 00007f8fe1a6ecc2 00007f8fe1a6ebf1 00007f8fe1a6eb6c 00007f8fe1a6de22 00007f8fe186d34c 00007f8fe186c786
0129 02:59:38.356271 (+ 21249us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":158184704}
W20260129 02:59:38.297401 22368 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.672s user 0.016s sys 0.034s
W20260129 02:59:38.402225 20432 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.543s user 0.009s sys 0.013s
I20260129 02:59:38.410954 20514 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "45cc5f3d2688486d9b6dbe7e666dedde" candidate_uuid: "7a9daf5d42a84615a6383cfec9a10608" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "974b802141bd4f13beffef2aa03f88f9" is_pre_election: true
I20260129 02:59:38.413836 20514 raft_consensus.cc:2468] T 45cc5f3d2688486d9b6dbe7e666dedde P 974b802141bd4f13beffef2aa03f88f9 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7a9daf5d42a84615a6383cfec9a10608 in term 3.
I20260129 02:59:38.412206 20507 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a4c1226b987b468fa84dfc6ba0e2dbe1" candidate_uuid: "197b880fa39d432a9b472845876bfaf0" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "974b802141bd4f13beffef2aa03f88f9" is_pre_election: true
I20260129 02:59:38.333439 22414 raft_consensus.cc:515] T 4e3805ffa70242e8ade66da260dddf66 P 7a9daf5d42a84615a6383cfec9a10608 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } }
I20260129 02:59:38.417740 20515 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "add8d4923d1a489e86882c3a25692b95" candidate_uuid: "7a9daf5d42a84615a6383cfec9a10608" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "974b802141bd4f13beffef2aa03f88f9" is_pre_election: true
I20260129 02:59:38.418798 20515 raft_consensus.cc:2393] T add8d4923d1a489e86882c3a25692b95 P 974b802141bd4f13beffef2aa03f88f9 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 7a9daf5d42a84615a6383cfec9a10608 in current term 2: Already voted for candidate 974b802141bd4f13beffef2aa03f88f9 in this term.
I20260129 02:59:38.419633 20514 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3e2d15f8640b407195210c476e09df91" candidate_uuid: "7a9daf5d42a84615a6383cfec9a10608" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "974b802141bd4f13beffef2aa03f88f9" is_pre_election: true
I20260129 02:59:38.420511 20514 raft_consensus.cc:2468] T 3e2d15f8640b407195210c476e09df91 P 974b802141bd4f13beffef2aa03f88f9 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7a9daf5d42a84615a6383cfec9a10608 in term 1.
I20260129 02:59:38.421586 20514 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "45cc5f3d2688486d9b6dbe7e666dedde" candidate_uuid: "7a9daf5d42a84615a6383cfec9a10608" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "974b802141bd4f13beffef2aa03f88f9" is_pre_election: true
I20260129 02:59:38.422313 20514 raft_consensus.cc:2468] T 45cc5f3d2688486d9b6dbe7e666dedde P 974b802141bd4f13beffef2aa03f88f9 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7a9daf5d42a84615a6383cfec9a10608 in term 3.
I20260129 02:59:38.425208 20514 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3e2d15f8640b407195210c476e09df91" candidate_uuid: "7a9daf5d42a84615a6383cfec9a10608" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "974b802141bd4f13beffef2aa03f88f9" is_pre_election: true
I20260129 02:59:38.425933 20514 raft_consensus.cc:2468] T 3e2d15f8640b407195210c476e09df91 P 974b802141bd4f13beffef2aa03f88f9 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7a9daf5d42a84615a6383cfec9a10608 in term 1.
W20260129 02:59:38.392091 22437 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.536s user 0.001s sys 0.023s
I20260129 02:59:38.410579 22466 raft_consensus.cc:515] T 75b0b6e82ddc4f92b1fe1d94f9a0b73b P 1c50de89a5a845668696dcb28c3badd6 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } }
I20260129 02:59:38.431398 20514 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "75f45538dc704d9a8ec4ff8ad9f3ea5b" candidate_uuid: "7a9daf5d42a84615a6383cfec9a10608" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "974b802141bd4f13beffef2aa03f88f9"
I20260129 02:59:38.403189 22440 raft_consensus.cc:493] T 4e3805ffa70242e8ade66da260dddf66 P 197b880fa39d432a9b472845876bfaf0 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 1c50de89a5a845668696dcb28c3badd6)
I20260129 02:59:38.429616 20432 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:48356 (request call id 242) took 1068 ms. Trace:
W20260129 02:59:38.443452 22437 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.602s user 0.001s sys 0.023s
I20260129 02:59:38.303045 20509 rpcz_store.cc:276] 0129 02:59:36.468388 (+ 0us) service_pool.cc:168] Inserting onto call queue
0129 02:59:36.471630 (+ 3242us) service_pool.cc:225] Handling call
0129 02:59:36.542172 (+ 70542us) raft_consensus.cc:1471] Updating replica for 1 ops
0129 02:59:37.034444 (+492272us) raft_consensus.cc:1534] Early marking committed up to index 0
0129 02:59:37.042023 (+ 7579us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0129 02:59:37.124574 (+ 82551us) log.cc:844] Serialized 26 byte log entry
0129 02:59:37.812870 (+688296us) raft_consensus.cc:1645] Marking committed up to 1
0129 02:59:37.829994 (+ 17124us) raft_consensus.cc:1697] Filling consensus response to leader.
0129 02:59:37.843365 (+ 13371us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0129 02:59:38.252753 (+409388us) raft_consensus.cc:1685] finished
0129 02:59:38.252778 (+ 25us) raft_consensus.cc:1691] UpdateReplicas() finished
0129 02:59:38.252967 (+ 189us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":4161,"raft.queue_time_us":261078,"raft.run_cpu_time_us":185,"raft.run_wall_time_us":12260,"thread_start_us":863069,"threads_started":2,"wal-append.queue_time_us":1017900}
I20260129 02:59:38.449285 22447 leader_election.cc:290] T 18f30ebaa4174fd28c958e72d0cc7e24 P 1c50de89a5a845668696dcb28c3badd6 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 197b880fa39d432a9b472845876bfaf0 (127.18.211.1:42027), 7a9daf5d42a84615a6383cfec9a10608 (127.18.211.4:40439)
I20260129 02:59:38.453007 22115 leader_election.cc:290] T 42d27a28d89b409e8807ec0efd97af69 P 1c50de89a5a845668696dcb28c3badd6 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 197b880fa39d432a9b472845876bfaf0 (127.18.211.1:42027), 7a9daf5d42a84615a6383cfec9a10608 (127.18.211.4:40439)
W20260129 02:59:38.416577 22342 consensus_meta.cc:287] T a609ac02fae84725ab743f95f51d33ce P 7a9daf5d42a84615a6383cfec9a10608: Time spent flushing consensus metadata: real 0.548s user 0.006s sys 0.006s
I20260129 02:59:38.455018 21131 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34060 (request call id 452) took 2862 ms. Trace:
I20260129 02:59:38.460103 20515 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "cb0b3f259ebc4310927338a8740e87b8" candidate_uuid: "7a9daf5d42a84615a6383cfec9a10608" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "974b802141bd4f13beffef2aa03f88f9" is_pre_election: true
I20260129 02:59:38.462996 21131 rpcz_store.cc:276] 0129 02:59:35.592945 (+ 0us) service_pool.cc:168] Inserting onto call queue
0129 02:59:35.593081 (+ 136us) service_pool.cc:225] Handling call
0129 02:59:35.717642 (+124561us) spinlock_profiling.cc:243] Waited 41.4 ms on lock 0x7b5800111f98. stack: 00007f8fe044e373 00007f8fe044e270 00007f8fdfda0574 00007f8fe9a5c6b4 00007f8fe9a5c649 00007f8fe02c08b1 00007f8fe0442db1 00007f8fe0442c62 00007f8fe92eddd9 00007f8fe846d260 00007f8fe8535dbb 00007f8fe8535e41 00007f8fe84f7af0 00007f8fe84e2459 00007f8fe84e1f02 00007f8fe1a6ed34
0129 02:59:35.717888 (+ 246us) raft_consensus.cc:1471] Updating replica for 1 ops
0129 02:59:35.877619 (+159731us) raft_consensus.cc:1534] Early marking committed up to index 0
0129 02:59:35.877649 (+ 30us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0129 02:59:35.919413 (+ 41764us) log.cc:844] Serialized 26 byte log entry
0129 02:59:36.434377 (+514964us) raft_consensus.cc:1645] Marking committed up to 5
0129 02:59:36.434412 (+ 35us) raft_consensus.cc:1697] Filling consensus response to leader.
0129 02:59:36.434513 (+ 101us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0129 02:59:38.295608 (+1861095us) spinlock_profiling.cc:243] Waited 9 ms on lock 0x7b440002304c. stack: 00007f8fe044e373 00007f8fe044e270 00007f8fdfda0574 00007f8fe9a5c6b4 00007f8fe9a5c649 00007f8fe02c08b1 00007f8fe038f005 00007f8fe038fa22 00007f8fe30d957b 00007f8fe30d4e4f 00007f8fe30c8c0e 00007f8fe30c7f56 00007f8fe30c7b78 00007f8fe30d15c1 00007f8fe30d1579 00007f8fe30d1509
0129 02:59:38.454760 (+159152us) raft_consensus.cc:1685] finished
0129 02:59:38.454789 (+ 29us) raft_consensus.cc:1691] UpdateReplicas() finished
0129 02:59:38.454957 (+ 168us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":10352,"raft.queue_time_us":112970,"raft.run_cpu_time_us":116,"raft.run_wall_time_us":117,"spinlock_wait_cycles":115986560,"thread_start_us":594688,"threads_started":2,"wal-append.queue_time_us":569189}
W20260129 02:59:38.467280 20553 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260129 02:59:38.465839 20507 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "febffe4a8b394368b69b1d164308fb9a" candidate_uuid: "7a9daf5d42a84615a6383cfec9a10608" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "974b802141bd4f13beffef2aa03f88f9"
I20260129 02:59:38.468346 20507 raft_consensus.cc:2393] T febffe4a8b394368b69b1d164308fb9a P 974b802141bd4f13beffef2aa03f88f9 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 7a9daf5d42a84615a6383cfec9a10608 in current term 2: Already voted for candidate 974b802141bd4f13beffef2aa03f88f9 in this term.
W20260129 02:59:38.428766 22368 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.807s user 0.016s sys 0.036s
I20260129 02:59:38.457104 22411 raft_consensus.cc:493] T c891d2bff4ce4568861bbc992e91fbb0 P 197b880fa39d432a9b472845876bfaf0 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 974b802141bd4f13beffef2aa03f88f9)
I20260129 02:59:38.455830 20432 rpcz_store.cc:276] 0129 02:59:37.367271 (+ 0us) service_pool.cc:168] Inserting onto call queue
0129 02:59:37.370064 (+ 2793us) service_pool.cc:225] Handling call
0129 02:59:37.372313 (+ 2249us) raft_consensus.cc:1471] Updating replica for 0 ops
0129 02:59:37.372587 (+ 274us) raft_consensus.cc:1534] Early marking committed up to index 5
0129 02:59:37.858624 (+486037us) log.cc:844] Serialized 14 byte log entry
0129 02:59:38.429217 (+570593us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0129 02:59:38.429289 (+ 72us) raft_consensus.cc:1645] Marking committed up to 5
0129 02:59:38.429348 (+ 59us) raft_consensus.cc:1697] Filling consensus response to leader.
0129 02:59:38.429427 (+ 79us) raft_consensus.cc:1691] UpdateReplicas() finished
0129 02:59:38.429553 (+ 126us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":753986,"prepare.run_cpu_time_us":85,"prepare.run_wall_time_us":336,"raft.queue_time_us":132,"raft.run_cpu_time_us":169,"raft.run_wall_time_us":16337,"thread_start_us":1027130,"threads_started":2}
W20260129 02:59:38.461432 20479 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.211.1:42027, user_credentials={real_user=slave}} blocked reactor thread for 411794us
I20260129 02:59:38.430461 22435 raft_consensus.cc:697] T 4f5a77e94fef4b16a2ba7b067c8b144a P 1c50de89a5a845668696dcb28c3badd6 [term 3 LEADER]: Becoming Leader. State: Replica: 1c50de89a5a845668696dcb28c3badd6, State: Running, Role: LEADER
I20260129 02:59:38.348536 22451 consensus_queue.cc:260] T 17d6431cc77a491e8464d68db4081a51 P 974b802141bd4f13beffef2aa03f88f9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
W20260129 02:59:38.439276 21099 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.504s user 0.009s sys 0.009s
W20260129 02:59:38.475791 21099 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.540s user 0.009s sys 0.009s
I20260129 02:59:38.461841 22450 raft_consensus.cc:493] T a5a05fbcbbcf4916bbc01fd1a5b5e90d P 1c50de89a5a845668696dcb28c3badd6 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260129 02:59:38.478127 20553 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260129 02:59:38.464839 22448 leader_election.cc:290] T e4d18c35b0a7493e85d0a4b92956a48f P 974b802141bd4f13beffef2aa03f88f9 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 197b880fa39d432a9b472845876bfaf0 (127.18.211.1:42027), 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3:37735)
I20260129 02:59:38.477005 20432 raft_consensus.cc:1275] T 368953c9102643b2a21f99af36b51905 P 197b880fa39d432a9b472845876bfaf0 [term 4 FOLLOWER]: Refusing update from remote peer 1c50de89a5a845668696dcb28c3badd6: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
W20260129 02:59:38.483458 20407 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.211.2:34517, user_credentials={real_user=slave}} blocked reactor thread for 361383us
I20260129 02:59:38.510622 20432 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7d9494f5584d4d5db8b9adf1c8fc888b" candidate_uuid: "974b802141bd4f13beffef2aa03f88f9" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "197b880fa39d432a9b472845876bfaf0" is_pre_election: true
I20260129 02:59:38.485646 21099 leader_election.cc:304] T 00e79d8f25a34676b1d866eb2068a2b2 P 7a9daf5d42a84615a6383cfec9a10608 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 197b880fa39d432a9b472845876bfaf0, 7a9daf5d42a84615a6383cfec9a10608; no voters:
W20260129 02:59:38.476655 21102 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.529s user 0.009s sys 0.005s
W20260129 02:59:38.528282 21102 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.581s user 0.009s sys 0.005s
I20260129 02:59:38.459424 22440 raft_consensus.cc:515] T 4e3805ffa70242e8ade66da260dddf66 P 197b880fa39d432a9b472845876bfaf0 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } }
I20260129 02:59:38.497429 20507 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a644c02b16af47679824ccd060ac10fd" candidate_uuid: "7a9daf5d42a84615a6383cfec9a10608" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "974b802141bd4f13beffef2aa03f88f9" is_pre_election: true
W20260129 02:59:38.496918 22426 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.526s user 0.000s sys 0.024s
I20260129 02:59:38.494341 22450 raft_consensus.cc:515] T a5a05fbcbbcf4916bbc01fd1a5b5e90d P 1c50de89a5a845668696dcb28c3badd6 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:59:38.498852 22451 raft_consensus.cc:3060] T 17d6431cc77a491e8464d68db4081a51 P 974b802141bd4f13beffef2aa03f88f9 [term 1 FOLLOWER]: Advancing to term 2
I20260129 02:59:38.517712 22435 consensus_queue.cc:237] T 4f5a77e94fef4b16a2ba7b067c8b144a P 1c50de89a5a845668696dcb28c3badd6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } }
I20260129 02:59:38.496939 20509 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "add8d4923d1a489e86882c3a25692b95" candidate_uuid: "7a9daf5d42a84615a6383cfec9a10608" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "974b802141bd4f13beffef2aa03f88f9"
I20260129 02:59:38.522187 20513 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57584 (request call id 1219) took 1573 ms. Trace:
I20260129 02:59:38.523289 20432 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "724a236524cb4e7ba3428880681d6cc2" candidate_uuid: "974b802141bd4f13beffef2aa03f88f9" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "197b880fa39d432a9b472845876bfaf0" is_pre_election: true
I20260129 02:59:38.544317 21131 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "87d5fe48a44a408aaf03e894a67ecb2f" candidate_uuid: "197b880fa39d432a9b472845876bfaf0" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "7a9daf5d42a84615a6383cfec9a10608" is_pre_election: true
I20260129 02:59:38.474462 22411 raft_consensus.cc:515] T c891d2bff4ce4568861bbc992e91fbb0 P 197b880fa39d432a9b472845876bfaf0 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
W20260129 02:59:38.528674 20406 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260129 02:59:38.530701 20507 raft_consensus.cc:2393] T a644c02b16af47679824ccd060ac10fd P 974b802141bd4f13beffef2aa03f88f9 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 7a9daf5d42a84615a6383cfec9a10608 in current term 2: Already voted for candidate 974b802141bd4f13beffef2aa03f88f9 in this term.
W20260129 02:59:38.550941 20404 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260129 02:59:38.528475 20407 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260129 02:59:38.513471 22465 leader_election.cc:290] T 17d6431cc77a491e8464d68db4081a51 P 1c50de89a5a845668696dcb28c3badd6 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 197b880fa39d432a9b472845876bfaf0 (127.18.211.1:42027), 974b802141bd4f13beffef2aa03f88f9 (127.18.211.2:34517)
W20260129 02:59:38.637576 22453 log.cc:927] Time spent T 75be4bd6bd45474795fd472eff2953ac P 1c50de89a5a845668696dcb28c3badd6: Append to log took a long time: real 0.091s user 0.000s sys 0.000s
W20260129 02:59:38.639770 20404 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260129 02:59:38.537154 22342 raft_consensus.cc:2804] T 00e79d8f25a34676b1d866eb2068a2b2 P 7a9daf5d42a84615a6383cfec9a10608 [term 4 FOLLOWER]: Leader election won for term 4
I20260129 02:59:38.543190 20432 raft_consensus.cc:2468] T 724a236524cb4e7ba3428880681d6cc2 P 197b880fa39d432a9b472845876bfaf0 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 974b802141bd4f13beffef2aa03f88f9 in term 2.
I20260129 02:59:38.653631 20432 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "8055232cf0004d40a85f3bf983fe00a9" candidate_uuid: "974b802141bd4f13beffef2aa03f88f9" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "197b880fa39d432a9b472845876bfaf0" is_pre_election: true
W20260129 02:59:38.654599 21101 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260129 02:59:38.536674 22457 log.cc:927] Time spent T 2ac659d732314e1288849f08fe5e074e P 197b880fa39d432a9b472845876bfaf0: Append to log took a long time: real 0.179s user 0.000s sys 0.001s
W20260129 02:59:38.531595 21099 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.211.1:42027, user_credentials={real_user=slave}} blocked reactor thread for 46057.3us
I20260129 02:59:38.550465 20507 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00e79d8f25a34676b1d866eb2068a2b2" candidate_uuid: "7a9daf5d42a84615a6383cfec9a10608" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "974b802141bd4f13beffef2aa03f88f9"
I20260129 02:59:38.656905 20507 raft_consensus.cc:2393] T 00e79d8f25a34676b1d866eb2068a2b2 P 974b802141bd4f13beffef2aa03f88f9 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 7a9daf5d42a84615a6383cfec9a10608 in current term 4: Already voted for candidate 974b802141bd4f13beffef2aa03f88f9 in this term.
I20260129 02:59:38.480824 20553 leader_election.cc:304] T fd8311b779af4e1fa1402b2a02a602f2 P 1c50de89a5a845668696dcb28c3badd6 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1c50de89a5a845668696dcb28c3badd6; no voters: 197b880fa39d432a9b472845876bfaf0, 7a9daf5d42a84615a6383cfec9a10608
I20260129 02:59:38.541828 20513 rpcz_store.cc:276] 0129 02:59:36.948378 (+ 0us) service_pool.cc:168] Inserting onto call queue
0129 02:59:36.949986 (+ 1608us) service_pool.cc:225] Handling call
0129 02:59:36.950355 (+ 369us) raft_consensus.cc:1471] Updating replica for 1 ops
0129 02:59:37.600608 (+650253us) raft_consensus.cc:1534] Early marking committed up to index 0
0129 02:59:37.600653 (+ 45us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0129 02:59:37.624521 (+ 23868us) log.cc:844] Serialized 26 byte log entry
0129 02:59:38.364583 (+740062us) raft_consensus.cc:1645] Marking committed up to 2
0129 02:59:38.364626 (+ 43us) raft_consensus.cc:1697] Filling consensus response to leader.
0129 02:59:38.364693 (+ 67us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0129 02:59:38.521910 (+157217us) raft_consensus.cc:1685] finished
0129 02:59:38.521939 (+ 29us) raft_consensus.cc:1691] UpdateReplicas() finished
0129 02:59:38.522104 (+ 165us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":41085,"raft.queue_time_us":214668,"raft.run_cpu_time_us":92,"raft.run_wall_time_us":93,"thread_start_us":1143279,"threads_started":2,"wal-append.queue_time_us":870727}
I20260129 02:59:38.661475 20432 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4f8f3a38f6e24ba4857875181f447442" candidate_uuid: "974b802141bd4f13beffef2aa03f88f9" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "197b880fa39d432a9b472845876bfaf0" is_pre_election: true
I20260129 02:59:38.672719 21131 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00e79d8f25a34676b1d866eb2068a2b2" candidate_uuid: "197b880fa39d432a9b472845876bfaf0" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "7a9daf5d42a84615a6383cfec9a10608" is_pre_election: true
I20260129 02:59:38.673697 21130 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2e58e9a5a8aa4cecafb464addb4f35ed" candidate_uuid: "197b880fa39d432a9b472845876bfaf0" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "7a9daf5d42a84615a6383cfec9a10608" is_pre_election: true
I20260129 02:59:38.674117 22440 leader_election.cc:290] T 4e3805ffa70242e8ade66da260dddf66 P 197b880fa39d432a9b472845876bfaf0 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3:37735), 7a9daf5d42a84615a6383cfec9a10608 (127.18.211.4:40439)
I20260129 02:59:38.674422 21130 raft_consensus.cc:2468] T 2e58e9a5a8aa4cecafb464addb4f35ed P 7a9daf5d42a84615a6383cfec9a10608 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 197b880fa39d432a9b472845876bfaf0 in term 3.
I20260129 02:59:38.674669 22440 raft_consensus.cc:493] T a609ac02fae84725ab743f95f51d33ce P 197b880fa39d432a9b472845876bfaf0 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260129 02:59:38.675122 22440 raft_consensus.cc:515] T a609ac02fae84725ab743f95f51d33ce P 197b880fa39d432a9b472845876bfaf0 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } }
I20260129 02:59:38.677551 20507 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4f5a77e94fef4b16a2ba7b067c8b144a" candidate_uuid: "7a9daf5d42a84615a6383cfec9a10608" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "974b802141bd4f13beffef2aa03f88f9"
I20260129 02:59:38.677747 22479 raft_consensus.cc:493] T 3906b080a5b34a74b8b64128635e8997 P 974b802141bd4f13beffef2aa03f88f9 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 1c50de89a5a845668696dcb28c3badd6)
I20260129 02:59:38.678257 22479 raft_consensus.cc:515] T 3906b080a5b34a74b8b64128635e8997 P 974b802141bd4f13beffef2aa03f88f9 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:59:38.675381 21130 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "87d5fe48a44a408aaf03e894a67ecb2f" candidate_uuid: "974b802141bd4f13beffef2aa03f88f9" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "7a9daf5d42a84615a6383cfec9a10608" is_pre_election: true
I20260129 02:59:38.689703 20432 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a0fcc58d6fb6473cafbd9d34d5195269" candidate_uuid: "974b802141bd4f13beffef2aa03f88f9" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "197b880fa39d432a9b472845876bfaf0"
I20260129 02:59:38.691615 22479 leader_election.cc:290] T 3906b080a5b34a74b8b64128635e8997 P 974b802141bd4f13beffef2aa03f88f9 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 197b880fa39d432a9b472845876bfaf0 (127.18.211.1:42027), 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3:37735)
W20260129 02:59:38.694391 21099 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260129 02:59:38.694903 21099 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260129 02:59:38.696183 21099 leader_election.cc:304] T 78aa39096868415d9afc2ffcb248e198 P 7a9daf5d42a84615a6383cfec9a10608 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 197b880fa39d432a9b472845876bfaf0, 7a9daf5d42a84615a6383cfec9a10608; no voters:
I20260129 02:59:38.655439 21128 raft_consensus.cc:1240] T fd8311b779af4e1fa1402b2a02a602f2 P 7a9daf5d42a84615a6383cfec9a10608 [term 3 FOLLOWER]: Rejecting Update request from peer 197b880fa39d432a9b472845876bfaf0 for earlier term 2. Current term is 3. Ops: []
I20260129 02:59:38.698514 20513 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "96609ac68bdf4993a12e1c23355ca013" candidate_uuid: "7a9daf5d42a84615a6383cfec9a10608" candidate_term: 4 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "974b802141bd4f13beffef2aa03f88f9"
I20260129 02:59:38.659994 22212 leader_election.cc:290] T 2ac659d732314e1288849f08fe5e074e P 7a9daf5d42a84615a6383cfec9a10608 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 197b880fa39d432a9b472845876bfaf0 (127.18.211.1:42027), 974b802141bd4f13beffef2aa03f88f9 (127.18.211.2:34517)
I20260129 02:59:38.700060 20432 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "36ff33be666a4c82954731691dbcf327" candidate_uuid: "974b802141bd4f13beffef2aa03f88f9" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "197b880fa39d432a9b472845876bfaf0" is_pre_election: true
W20260129 02:59:38.700644 20512 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:57600 (request call id 1666) took 3287 ms (client timeout 1895 ms). Trace:
I20260129 02:59:38.660658 22448 raft_consensus.cc:493] T 2ac659d732314e1288849f08fe5e074e P 974b802141bd4f13beffef2aa03f88f9 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260129 02:59:38.709174 20432 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "368953c9102643b2a21f99af36b51905" candidate_uuid: "974b802141bd4f13beffef2aa03f88f9" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "197b880fa39d432a9b472845876bfaf0" is_pre_election: true
I20260129 02:59:38.694283 20509 raft_consensus.cc:2393] T add8d4923d1a489e86882c3a25692b95 P 974b802141bd4f13beffef2aa03f88f9 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 7a9daf5d42a84615a6383cfec9a10608 in current term 2: Already voted for candidate 974b802141bd4f13beffef2aa03f88f9 in this term.
W20260129 02:59:38.719712 20553 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.211.1:42027, user_credentials={real_user=slave}} blocked reactor thread for 238978us
I20260129 02:59:38.698726 20590 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35752 (request call id 1473) took 2645 ms. Trace:
I20260129 02:59:38.709961 20432 raft_consensus.cc:2468] T 368953c9102643b2a21f99af36b51905 P 197b880fa39d432a9b472845876bfaf0 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 974b802141bd4f13beffef2aa03f88f9 in term 4.
W20260129 02:59:38.708531 20512 rpcz_store.cc:269] 0129 02:59:35.413391 (+ 0us) service_pool.cc:168] Inserting onto call queue
0129 02:59:35.413523 (+ 132us) service_pool.cc:225] Handling call
0129 02:59:38.700429 (+3286906us) spinlock_profiling.cc:243] Waited 463 ms on lock 0x7b5400300cd4. stack: 00007f8fe044e373 00007f8fe044e270 00007f8fdfda0574 00007f8fe9a5c6b4 00007f8fe9a5c649 00007f8fe34975a1 00007f8fe3531f7a 00007f8fe84e3647 00007f8fe1a73834 00007f8fe1a737c2 00007f8fe1a736f1 00007f8fe1a7366c 00007f8fe1a72922 00007f8fe186d34c 00007f8fe186c786 00007f8fe186c12f
0129 02:59:38.700562 (+ 133us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1065446272}
I20260129 02:59:38.721206 20432 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e4d18c35b0a7493e85d0a4b92956a48f" candidate_uuid: "974b802141bd4f13beffef2aa03f88f9" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "197b880fa39d432a9b472845876bfaf0" is_pre_election: true
I20260129 02:59:38.722039 20432 raft_consensus.cc:2468] T e4d18c35b0a7493e85d0a4b92956a48f P 197b880fa39d432a9b472845876bfaf0 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 974b802141bd4f13beffef2aa03f88f9 in term 3.
I20260129 02:59:38.722316 20512 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "96609ac68bdf4993a12e1c23355ca013" candidate_uuid: "7a9daf5d42a84615a6383cfec9a10608" candidate_term: 4 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "974b802141bd4f13beffef2aa03f88f9"
I20260129 02:59:38.722906 20432 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b189f08e16a74a11980cc55d99359a0a" candidate_uuid: "974b802141bd4f13beffef2aa03f88f9" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "197b880fa39d432a9b472845876bfaf0"
W20260129 02:59:38.710762 20404 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260129 02:59:38.701557 21100 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.754s user 0.019s sys 0.051s
W20260129 02:59:38.543650 20556 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.211.2:34517, user_credentials={real_user=slave}} blocked reactor thread for 431229us
I20260129 02:59:38.689428 22373 raft_consensus.cc:493] T 0f20af7fbe5d4db89a6f17948d98912c P 197b880fa39d432a9b472845876bfaf0 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 1c50de89a5a845668696dcb28c3badd6)
W20260129 02:59:38.648274 20554 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.211.4:40439, user_credentials={real_user=slave}} blocked reactor thread for 407887us
W20260129 02:59:38.724686 21100 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.778s user 0.019s sys 0.052s
I20260129 02:59:38.732105 21100 leader_election.cc:304] T cb0b3f259ebc4310927338a8740e87b8 P 7a9daf5d42a84615a6383cfec9a10608 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1c50de89a5a845668696dcb28c3badd6, 7a9daf5d42a84615a6383cfec9a10608; no voters:
I20260129 02:59:38.733577 20513 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1ce4ddf5b10e4ae8b193b13b2e5936ee" candidate_uuid: "197b880fa39d432a9b472845876bfaf0" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "974b802141bd4f13beffef2aa03f88f9" is_pre_election: true
I20260129 02:59:38.724582 20440 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55676 (request call id 1162) took 1268 ms. Trace:
W20260129 02:59:38.733633 21100 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260129 02:59:38.720634 20590 rpcz_store.cc:276] 0129 02:59:36.053263 (+ 0us) service_pool.cc:168] Inserting onto call queue
0129 02:59:36.084454 (+ 31191us) service_pool.cc:225] Handling call
0129 02:59:36.117767 (+ 33313us) raft_consensus.cc:1471] Updating replica for 0 ops
0129 02:59:36.161695 (+ 43928us) raft_consensus.cc:1697] Filling consensus response to leader.
0129 02:59:38.698499 (+2536804us) spinlock_profiling.cc:243] Waited 15.7 ms on lock 0x7b54000c9c54. stack: 00007f8fe044e373 00007f8fe044e270 00007f8fdfda0574 00007f8fe9a5c6b4 00007f8fe9a5c649 00007f8fe34975a1 00007f8fe352ea81 00007f8fe352d4be 00007f8fe84e1f49 00007f8fe1a6ed34 00007f8fe1a6ecc2 00007f8fe1a6ebf1 00007f8fe1a6eb6c 00007f8fe1a6de22 00007f8fe186d34c 00007f8fe186c786
0129 02:59:38.698665 (+ 166us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":35998976}
I20260129 02:59:38.734272 20513 raft_consensus.cc:2468] T 1ce4ddf5b10e4ae8b193b13b2e5936ee P 974b802141bd4f13beffef2aa03f88f9 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 197b880fa39d432a9b472845876bfaf0 in term 4.
I20260129 02:59:38.734926 20513 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:57600 (request call id 1691) took 1173 ms. Trace:
I20260129 02:59:38.735355 20513 rpcz_store.cc:276] 0129 02:59:37.561422 (+ 0us) service_pool.cc:168] Inserting onto call queue
0129 02:59:38.703524 (+1142102us) service_pool.cc:225] Handling call
0129 02:59:38.734859 (+ 31335us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260129 02:59:38.724716 22475 raft_consensus.cc:493] T 42d27a28d89b409e8807ec0efd97af69 P 7a9daf5d42a84615a6383cfec9a10608 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 197b880fa39d432a9b472845876bfaf0)
I20260129 02:59:38.737540 20432 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c934d31efd3f4e5195605ddc354d14e1" candidate_uuid: "974b802141bd4f13beffef2aa03f88f9" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "197b880fa39d432a9b472845876bfaf0"
I20260129 02:59:38.737783 22475 raft_consensus.cc:515] T 42d27a28d89b409e8807ec0efd97af69 P 7a9daf5d42a84615a6383cfec9a10608 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } }
I20260129 02:59:38.738754 20432 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7a9a69cd82724b1faed1d10aa091fc4a" candidate_uuid: "974b802141bd4f13beffef2aa03f88f9" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "197b880fa39d432a9b472845876bfaf0" is_pre_election: true
W20260129 02:59:38.739617 20482 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260129 02:59:38.740216 20482 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260129 02:59:38.735251 20440 rpcz_store.cc:276] 0129 02:59:37.455896 (+ 0us) service_pool.cc:168] Inserting onto call queue
0129 02:59:37.895063 (+439167us) service_pool.cc:225] Handling call
0129 02:59:37.895465 (+ 402us) raft_consensus.cc:1471] Updating replica for 0 ops
0129 02:59:37.895686 (+ 221us) raft_consensus.cc:1534] Early marking committed up to index 3
0129 02:59:38.392897 (+497211us) log.cc:844] Serialized 14 byte log entry
0129 02:59:38.724144 (+331247us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0129 02:59:38.724208 (+ 64us) raft_consensus.cc:1645] Marking committed up to 3
0129 02:59:38.724261 (+ 53us) raft_consensus.cc:1697] Filling consensus response to leader.
0129 02:59:38.724352 (+ 91us) raft_consensus.cc:1691] UpdateReplicas() finished
0129 02:59:38.724521 (+ 169us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":6462,"prepare.queue_time_us":236977,"prepare.run_cpu_time_us":47,"prepare.run_wall_time_us":48,"thread_start_us":810001,"threads_started":2}
I20260129 02:59:38.727183 20480 leader_election.cc:304] T e7d83d4ccf684e76acb29a0dc844e1b8 P 974b802141bd4f13beffef2aa03f88f9 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 974b802141bd4f13beffef2aa03f88f9; no voters: 197b880fa39d432a9b472845876bfaf0, 1c50de89a5a845668696dcb28c3badd6
I20260129 02:59:38.700577 22432 raft_consensus.cc:2749] T 36ff33be666a4c82954731691dbcf327 P 197b880fa39d432a9b472845876bfaf0 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260129 02:59:38.744295 20440 raft_consensus.cc:1240] T e7d83d4ccf684e76acb29a0dc844e1b8 P 197b880fa39d432a9b472845876bfaf0 [term 3 FOLLOWER]: Rejecting Update request from peer 1c50de89a5a845668696dcb28c3badd6 for earlier term 2. Current term is 3. Ops: []
I20260129 02:59:38.700209 22342 raft_consensus.cc:697] T 00e79d8f25a34676b1d866eb2068a2b2 P 7a9daf5d42a84615a6383cfec9a10608 [term 4 LEADER]: Becoming Leader. State: Replica: 7a9daf5d42a84615a6383cfec9a10608, State: Running, Role: LEADER
I20260129 02:59:38.708500 22403 raft_consensus.cc:2804] T 78aa39096868415d9afc2ffcb248e198 P 7a9daf5d42a84615a6383cfec9a10608 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260129 02:59:38.746344 20440 raft_consensus.cc:1240] T 4408006eb5174de1896b037cf3f18457 P 197b880fa39d432a9b472845876bfaf0 [term 2 FOLLOWER]: Rejecting Update request from peer 974b802141bd4f13beffef2aa03f88f9 for earlier term 1. Current term is 2. Ops: []
I20260129 02:59:38.707005 22082 raft_consensus.cc:493] T 17d6431cc77a491e8464d68db4081a51 P 197b880fa39d432a9b472845876bfaf0 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260129 02:59:38.734298 20507 raft_consensus.cc:2468] T 4f5a77e94fef4b16a2ba7b067c8b144a P 974b802141bd4f13beffef2aa03f88f9 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7a9daf5d42a84615a6383cfec9a10608 in term 2.
I20260129 02:59:38.746379 20512 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "505580c93c694bbb8ea96fdf9b17ab46" candidate_uuid: "7a9daf5d42a84615a6383cfec9a10608" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "974b802141bd4f13beffef2aa03f88f9" is_pre_election: true
I20260129 02:59:38.725442 22426 raft_consensus.cc:493] T 926a250fa5834e88b96ccecbefbf5086 P 1c50de89a5a845668696dcb28c3badd6 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260129 02:59:38.720608 22466 leader_election.cc:290] T 75b0b6e82ddc4f92b1fe1d94f9a0b73b P 1c50de89a5a845668696dcb28c3badd6 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 974b802141bd4f13beffef2aa03f88f9 (127.18.211.2:34517), 7a9daf5d42a84615a6383cfec9a10608 (127.18.211.4:40439)
I20260129 02:59:38.720273 22411 leader_election.cc:290] T c891d2bff4ce4568861bbc992e91fbb0 P 197b880fa39d432a9b472845876bfaf0 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 974b802141bd4f13beffef2aa03f88f9 (127.18.211.2:34517), 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3:37735)
I20260129 02:59:38.757659 20512 raft_consensus.cc:2410] T 505580c93c694bbb8ea96fdf9b17ab46 P 974b802141bd4f13beffef2aa03f88f9 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 7a9daf5d42a84615a6383cfec9a10608 for term 4 because replica has last-logged OpId of term: 3 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 5.
I20260129 02:59:38.758311 20513 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "cb0b3f259ebc4310927338a8740e87b8" candidate_uuid: "7a9daf5d42a84615a6383cfec9a10608" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "974b802141bd4f13beffef2aa03f88f9"
I20260129 02:59:38.730954 22448 raft_consensus.cc:515] T 2ac659d732314e1288849f08fe5e074e P 974b802141bd4f13beffef2aa03f88f9 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } }
I20260129 02:59:38.761530 21128 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "18f30ebaa4174fd28c958e72d0cc7e24" candidate_uuid: "1c50de89a5a845668696dcb28c3badd6" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "7a9daf5d42a84615a6383cfec9a10608" is_pre_election: true
I20260129 02:59:38.762069 22440 leader_election.cc:290] T a609ac02fae84725ab743f95f51d33ce P 197b880fa39d432a9b472845876bfaf0 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 974b802141bd4f13beffef2aa03f88f9 (127.18.211.2:34517), 7a9daf5d42a84615a6383cfec9a10608 (127.18.211.4:40439)
I20260129 02:59:38.763005 20509 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57584 (request call id 1228) took 1781 ms. Trace:
I20260129 02:59:38.753628 22481 raft_consensus.cc:493] T 3906b080a5b34a74b8b64128635e8997 P 197b880fa39d432a9b472845876bfaf0 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 1c50de89a5a845668696dcb28c3badd6)
I20260129 02:59:38.766881 20480 leader_election.cc:304] T 3e2d15f8640b407195210c476e09df91 P 974b802141bd4f13beffef2aa03f88f9 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 974b802141bd4f13beffef2aa03f88f9; no voters: 1c50de89a5a845668696dcb28c3badd6, 7a9daf5d42a84615a6383cfec9a10608
W20260129 02:59:38.828876 20480 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.211.3:37735, user_credentials={real_user=slave}} blocked reactor thread for 62101.3us
I20260129 02:59:38.768347 20405 leader_election.cc:304] T 78aa39096868415d9afc2ffcb248e198 P 197b880fa39d432a9b472845876bfaf0 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 197b880fa39d432a9b472845876bfaf0, 7a9daf5d42a84615a6383cfec9a10608; no voters:
W20260129 02:59:38.839262 20405 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.211.4:40439, user_credentials={real_user=slave}} blocked reactor thread for 71016.2us
W20260129 02:59:38.830441 22120 consensus_meta.cc:287] T a1a39dcd328e4268ae41b988c66b4a75 P 197b880fa39d432a9b472845876bfaf0: Time spent flushing consensus metadata: real 0.526s user 0.011s sys 0.000s
I20260129 02:59:38.784598 22491 raft_consensus.cc:493] T a8479d1fe0bd41e5946186e3a6ca8c7a P 974b802141bd4f13beffef2aa03f88f9 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260129 02:59:38.836987 20513 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e4d18c35b0a7493e85d0a4b92956a48f" candidate_uuid: "197b880fa39d432a9b472845876bfaf0" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "974b802141bd4f13beffef2aa03f88f9" is_pre_election: true
I20260129 02:59:38.825479 22308 consensus_queue.cc:1048] T 1ce4ddf5b10e4ae8b193b13b2e5936ee P 1c50de89a5a845668696dcb28c3badd6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 1, Time since last communication: 0.015s
I20260129 02:59:38.842239 20513 raft_consensus.cc:2468] T e4d18c35b0a7493e85d0a4b92956a48f P 974b802141bd4f13beffef2aa03f88f9 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 197b880fa39d432a9b472845876bfaf0 in term 3.
I20260129 02:59:38.843323 22308 consensus_queue.cc:1048] T 724a236524cb4e7ba3428880681d6cc2 P 1c50de89a5a845668696dcb28c3badd6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260129 02:59:38.839165 21130 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "75b0b6e82ddc4f92b1fe1d94f9a0b73b" candidate_uuid: "1c50de89a5a845668696dcb28c3badd6" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "7a9daf5d42a84615a6383cfec9a10608" is_pre_election: true
I20260129 02:59:38.845858 21130 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4e3805ffa70242e8ade66da260dddf66" candidate_uuid: "197b880fa39d432a9b472845876bfaf0" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "7a9daf5d42a84615a6383cfec9a10608" is_pre_election: true
I20260129 02:59:38.847112 21130 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c934d31efd3f4e5195605ddc354d14e1" candidate_uuid: "974b802141bd4f13beffef2aa03f88f9" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "7a9daf5d42a84615a6383cfec9a10608"
I20260129 02:59:38.843685 20513 raft_consensus.cc:1240] T a0fcc58d6fb6473cafbd9d34d5195269 P 974b802141bd4f13beffef2aa03f88f9 [term 3 FOLLOWER]: Rejecting Update request from peer 197b880fa39d432a9b472845876bfaf0 for earlier term 2. Current term is 3. Ops: []
I20260129 02:59:38.850025 20405 leader_election.cc:304] T 2e58e9a5a8aa4cecafb464addb4f35ed P 197b880fa39d432a9b472845876bfaf0 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 197b880fa39d432a9b472845876bfaf0, 7a9daf5d42a84615a6383cfec9a10608; no voters:
I20260129 02:59:38.855913 20507 raft_consensus.cc:1240] T 926a250fa5834e88b96ccecbefbf5086 P 974b802141bd4f13beffef2aa03f88f9 [term 5 FOLLOWER]: Rejecting Update request from peer 1c50de89a5a845668696dcb28c3badd6 for earlier term 4. Current term is 5. Ops: []
I20260129 02:59:38.853881 20513 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57600 (request call id 1677) took 1676 ms. Trace:
I20260129 02:59:38.830339 20440 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.GetConsensusState from 127.0.0.1:55666 (request call id 1049) took 1274 ms. Trace:
I20260129 02:59:38.834431 22330 raft_consensus.cc:493] T a5a05fbcbbcf4916bbc01fd1a5b5e90d P 197b880fa39d432a9b472845876bfaf0 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 974b802141bd4f13beffef2aa03f88f9)
I20260129 02:59:38.857311 20440 rpcz_store.cc:276] 0129 02:59:37.553331 (+ 0us) service_pool.cc:168] Inserting onto call queue
0129 02:59:38.788735 (+1235404us) service_pool.cc:225] Handling call
0129 02:59:38.825815 (+ 37080us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260129 02:59:38.859783 22493 raft_consensus.cc:2749] T e7d83d4ccf684e76acb29a0dc844e1b8 P 974b802141bd4f13beffef2aa03f88f9 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260129 02:59:38.855289 21136 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "42d27a28d89b409e8807ec0efd97af69" candidate_uuid: "1c50de89a5a845668696dcb28c3badd6" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "7a9daf5d42a84615a6383cfec9a10608" is_pre_election: true
I20260129 02:59:38.862362 21136 raft_consensus.cc:2410] T 42d27a28d89b409e8807ec0efd97af69 P 7a9daf5d42a84615a6383cfec9a10608 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1c50de89a5a845668696dcb28c3badd6 for term 5 because replica has last-logged OpId of term: 4 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 5.
I20260129 02:59:38.848775 22440 raft_consensus.cc:2804] T 78aa39096868415d9afc2ffcb248e198 P 197b880fa39d432a9b472845876bfaf0 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260129 02:59:38.858435 20513 rpcz_store.cc:276] 0129 02:59:37.177963 (+ 0us) service_pool.cc:168] Inserting onto call queue
0129 02:59:38.843118 (+1665155us) service_pool.cc:225] Handling call
0129 02:59:38.843483 (+ 365us) raft_consensus.cc:1471] Updating replica for 0 ops
0129 02:59:38.853305 (+ 9822us) raft_consensus.cc:1697] Filling consensus response to leader.
0129 02:59:38.853810 (+ 505us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20260129 02:59:38.812240 21099 leader_election.cc:336] T fd8311b779af4e1fa1402b2a02a602f2 P 7a9daf5d42a84615a6383cfec9a10608 [CANDIDATE]: Term 3 election: RPC error from VoteRequest() call to peer 197b880fa39d432a9b472845876bfaf0 (127.18.211.1:42027): Timed out: RequestConsensusVote RPC to 127.18.211.1:42027 timed out after 1.551s (SENT)
I20260129 02:59:38.834147 20480 leader_election.cc:304] T 75be4bd6bd45474795fd472eff2953ac P 974b802141bd4f13beffef2aa03f88f9 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 974b802141bd4f13beffef2aa03f88f9; no voters: 197b880fa39d432a9b472845876bfaf0, 1c50de89a5a845668696dcb28c3badd6
I20260129 02:59:38.870390 20480 leader_election.cc:304] T cedce331e1584ac8b636c9b8a72b2ba0 P 974b802141bd4f13beffef2aa03f88f9 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 974b802141bd4f13beffef2aa03f88f9; no voters: 197b880fa39d432a9b472845876bfaf0, 1c50de89a5a845668696dcb28c3badd6
I20260129 02:59:38.856802 20507 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57584 (request call id 1230) took 1864 ms. Trace:
I20260129 02:59:38.872309 20480 leader_election.cc:304] T 45cc5f3d2688486d9b6dbe7e666dedde P 974b802141bd4f13beffef2aa03f88f9 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 974b802141bd4f13beffef2aa03f88f9; no voters: 1c50de89a5a845668696dcb28c3badd6, 7a9daf5d42a84615a6383cfec9a10608
I20260129 02:59:38.872779 20507 rpcz_store.cc:276] 0129 02:59:36.992107 (+ 0us) service_pool.cc:168] Inserting onto call queue
0129 02:59:38.760494 (+1768387us) service_pool.cc:225] Handling call
0129 02:59:38.760885 (+ 391us) raft_consensus.cc:1471] Updating replica for 0 ops
0129 02:59:38.856559 (+ 95674us) raft_consensus.cc:1697] Filling consensus response to leader.
0129 02:59:38.856748 (+ 189us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260129 02:59:38.833424 22486 raft_consensus.cc:493] T 3ccac82a01a2445fa5b0ce6c0fcf2295 P 7a9daf5d42a84615a6383cfec9a10608 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 197b880fa39d432a9b472845876bfaf0)
I20260129 02:59:38.750538 22483 raft_consensus.cc:2804] T cb0b3f259ebc4310927338a8740e87b8 P 7a9daf5d42a84615a6383cfec9a10608 [term 5 FOLLOWER]: Leader election won for term 5
I20260129 02:59:38.751278 22478 raft_consensus.cc:3060] T fd8311b779af4e1fa1402b2a02a602f2 P 1c50de89a5a845668696dcb28c3badd6 [term 2 FOLLOWER]: Advancing to term 3
W20260129 02:59:38.831543 20482 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260129 02:59:38.852718 22120 raft_consensus.cc:515] T a1a39dcd328e4268ae41b988c66b4a75 P 197b880fa39d432a9b472845876bfaf0 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } } peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } }
I20260129 02:59:38.743487 22373 raft_consensus.cc:515] T 0f20af7fbe5d4db89a6f17948d98912c P 197b880fa39d432a9b472845876bfaf0 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:59:38.873929 22112 raft_consensus.cc:3060] T 75f45538dc704d9a8ec4ff8ad9f3ea5b P 974b802141bd4f13beffef2aa03f88f9 [term 1 FOLLOWER]: Advancing to term 2
I20260129 02:59:38.832518 22426 raft_consensus.cc:515] T 926a250fa5834e88b96ccecbefbf5086 P 1c50de89a5a845668696dcb28c3badd6 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:59:38.754088 22403 raft_consensus.cc:493] T 78aa39096868415d9afc2ffcb248e198 P 7a9daf5d42a84615a6383cfec9a10608 [term 3 FOLLOWER]: Starting leader election (detected failure of leader 1c50de89a5a845668696dcb28c3badd6)
I20260129 02:59:38.756721 22479 raft_consensus.cc:493] T 4f5a77e94fef4b16a2ba7b067c8b144a P 974b802141bd4f13beffef2aa03f88f9 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260129 02:59:38.765686 22082 raft_consensus.cc:515] T 17d6431cc77a491e8464d68db4081a51 P 197b880fa39d432a9b472845876bfaf0 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:59:38.861402 22450 leader_election.cc:290] T a5a05fbcbbcf4916bbc01fd1a5b5e90d P 1c50de89a5a845668696dcb28c3badd6 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 197b880fa39d432a9b472845876bfaf0 (127.18.211.1:42027), 974b802141bd4f13beffef2aa03f88f9 (127.18.211.2:34517)
I20260129 02:59:38.864311 22302 raft_consensus.cc:2749] T 3e2d15f8640b407195210c476e09df91 P 974b802141bd4f13beffef2aa03f88f9 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260129 02:59:38.825924 20512 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a8479d1fe0bd41e5946186e3a6ca8c7a" candidate_uuid: "197b880fa39d432a9b472845876bfaf0" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "974b802141bd4f13beffef2aa03f88f9"
I20260129 02:59:38.781812 22164 consensus_queue.cc:1048] T a4c1226b987b468fa84dfc6ba0e2dbe1 P 1c50de89a5a845668696dcb28c3badd6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.059s
I20260129 02:59:38.866372 22440 raft_consensus.cc:493] T 78aa39096868415d9afc2ffcb248e198 P 197b880fa39d432a9b472845876bfaf0 [term 3 FOLLOWER]: Starting leader election (detected failure of leader 1c50de89a5a845668696dcb28c3badd6)
I20260129 02:59:38.894529 20513 raft_consensus.cc:1240] T c934d31efd3f4e5195605ddc354d14e1 P 974b802141bd4f13beffef2aa03f88f9 [term 4 FOLLOWER]: Rejecting Update request from peer 197b880fa39d432a9b472845876bfaf0 for earlier term 3. Current term is 4. Ops: [3.6-3.6]
I20260129 02:59:38.906446 20583 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:48526 (request call id 170) took 1861 ms. Trace:
I20260129 02:59:38.906940 20513 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57600 (request call id 1681) took 1718 ms. Trace:
I20260129 02:59:38.876987 21099 leader_election.cc:304] T fd8311b779af4e1fa1402b2a02a602f2 P 7a9daf5d42a84615a6383cfec9a10608 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7a9daf5d42a84615a6383cfec9a10608; no voters: 197b880fa39d432a9b472845876bfaf0, 1c50de89a5a845668696dcb28c3badd6
I20260129 02:59:38.828058 20509 rpcz_store.cc:276] 0129 02:59:36.981449 (+ 0us) service_pool.cc:168] Inserting onto call queue
0129 02:59:38.761998 (+1780549us) service_pool.cc:225] Handling call
0129 02:59:38.762359 (+ 361us) raft_consensus.cc:1471] Updating replica for 0 ops
0129 02:59:38.762613 (+ 254us) raft_consensus.cc:1534] Early marking committed up to index 0
0129 02:59:38.762653 (+ 40us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0129 02:59:38.762710 (+ 57us) raft_consensus.cc:1645] Marking committed up to 4
0129 02:59:38.762746 (+ 36us) raft_consensus.cc:1697] Filling consensus response to leader.
0129 02:59:38.762820 (+ 74us) raft_consensus.cc:1691] UpdateReplicas() finished
0129 02:59:38.762925 (+ 105us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260129 02:59:38.836544 22481 raft_consensus.cc:515] T 3906b080a5b34a74b8b64128635e8997 P 197b880fa39d432a9b472845876bfaf0 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
W20260129 02:59:38.910305 21099 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.211.1:42027, user_credentials={real_user=slave}} blocked reactor thread for 98127.1us
I20260129 02:59:38.889771 22486 raft_consensus.cc:515] T 3ccac82a01a2445fa5b0ce6c0fcf2295 P 7a9daf5d42a84615a6383cfec9a10608 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } }
I20260129 02:59:38.907411 20513 rpcz_store.cc:276] 0129 02:59:37.188144 (+ 0us) service_pool.cc:168] Inserting onto call queue
0129 02:59:38.877759 (+1689615us) service_pool.cc:225] Handling call
0129 02:59:38.894228 (+ 16469us) raft_consensus.cc:1471] Updating replica for 1 ops
0129 02:59:38.906628 (+ 12400us) raft_consensus.cc:1697] Filling consensus response to leader.
0129 02:59:38.906878 (+ 250us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260129 02:59:38.906971 20583 rpcz_store.cc:276] 0129 02:59:37.045669 (+ 0us) service_pool.cc:168] Inserting onto call queue
0129 02:59:37.046187 (+ 518us) service_pool.cc:225] Handling call
0129 02:59:37.046488 (+ 301us) raft_consensus.cc:1471] Updating replica for 0 ops
0129 02:59:37.067514 (+ 21026us) raft_consensus.cc:1534] Early marking committed up to index 0
0129 02:59:37.067549 (+ 35us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0129 02:59:37.067589 (+ 40us) raft_consensus.cc:1645] Marking committed up to 5
0129 02:59:37.067617 (+ 28us) raft_consensus.cc:1697] Filling consensus response to leader.
0129 02:59:38.906178 (+1838561us) spinlock_profiling.cc:243] Waited 20.8 ms on lock 0x7b540009b5d4. stack: 00007f8fe044e373 00007f8fe044e270 00007f8fdfda0574 00007f8fe9a5c6b4 00007f8fe9a5c649 00007f8fe34975a1 00007f8fe352ea81 00007f8fe352d4be 00007f8fe84e1f49 00007f8fe1a6ed34 00007f8fe1a6ecc2 00007f8fe1a6ebf1 00007f8fe1a6eb6c 00007f8fe1a6de22 00007f8fe186d34c 00007f8fe186c786
0129 02:59:38.906231 (+ 53us) raft_consensus.cc:1691] UpdateReplicas() finished
0129 02:59:38.906372 (+ 141us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":47872768}
I20260129 02:59:38.866030 22330 raft_consensus.cc:515] T a5a05fbcbbcf4916bbc01fd1a5b5e90d P 197b880fa39d432a9b472845876bfaf0 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:59:38.861100 22342 consensus_queue.cc:237] T 00e79d8f25a34676b1d866eb2068a2b2 P 7a9daf5d42a84615a6383cfec9a10608 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 3.6, Last appended by leader: 6, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } }
I20260129 02:59:38.917986 22330 leader_election.cc:290] T a5a05fbcbbcf4916bbc01fd1a5b5e90d P 197b880fa39d432a9b472845876bfaf0 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 974b802141bd4f13beffef2aa03f88f9 (127.18.211.2:34517), 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3:37735)
I20260129 02:59:38.917894 22481 leader_election.cc:290] T 3906b080a5b34a74b8b64128635e8997 P 197b880fa39d432a9b472845876bfaf0 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 974b802141bd4f13beffef2aa03f88f9 (127.18.211.2:34517), 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3:37735)
I20260129 02:59:38.854982 22491 raft_consensus.cc:515] T a8479d1fe0bd41e5946186e3a6ca8c7a P 974b802141bd4f13beffef2aa03f88f9 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
W20260129 02:59:38.923604 20482 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260129 02:59:38.903806 22403 raft_consensus.cc:3060] T 78aa39096868415d9afc2ffcb248e198 P 7a9daf5d42a84615a6383cfec9a10608 [term 3 FOLLOWER]: Advancing to term 4
I20260129 02:59:38.922312 20407 leader_election.cc:304] T cedce331e1584ac8b636c9b8a72b2ba0 P 197b880fa39d432a9b472845876bfaf0 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 197b880fa39d432a9b472845876bfaf0, 974b802141bd4f13beffef2aa03f88f9; no voters:
I20260129 02:59:38.919728 22302 raft_consensus.cc:2749] T 75be4bd6bd45474795fd472eff2953ac P 974b802141bd4f13beffef2aa03f88f9 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260129 02:59:38.914716 22212 raft_consensus.cc:2749] T fd8311b779af4e1fa1402b2a02a602f2 P 7a9daf5d42a84615a6383cfec9a10608 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260129 02:59:38.928309 20512 raft_consensus.cc:2376] T a8479d1fe0bd41e5946186e3a6ca8c7a P 974b802141bd4f13beffef2aa03f88f9 [term 3 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 197b880fa39d432a9b472845876bfaf0 in term 3. Re-sending same reply.
I20260129 02:59:38.928642 20439 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55676 (request call id 1146) took 2692 ms. Trace:
I20260129 02:59:38.933140 20439 rpcz_store.cc:276] 0129 02:59:36.236358 (+ 0us) service_pool.cc:168] Inserting onto call queue
0129 02:59:36.236543 (+ 185us) service_pool.cc:225] Handling call
0129 02:59:36.236940 (+ 397us) raft_consensus.cc:1471] Updating replica for 0 ops
0129 02:59:37.356021 (+1119081us) raft_consensus.cc:1697] Filling consensus response to leader.
0129 02:59:38.928378 (+1572357us) spinlock_profiling.cc:243] Waited 1.12 s on lock 0x7b54003232d4. stack: 00007f8fe044e373 00007f8fe044e270 00007f8fdfda0574 00007f8fe9a5c6b4 00007f8fe9a5c649 00007f8fe34975a1 00007f8fe352ea81 00007f8fe352d4be 00007f8fe84e1f49 00007f8fe1a6ed34 00007f8fe1a6ecc2 00007f8fe1a6ebf1 00007f8fe1a6eb6c 00007f8fe1a6de22 00007f8fe186d34c 00007f8fe186c786
0129 02:59:38.928582 (+ 204us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2571633536}
I20260129 02:59:38.909461 22483 raft_consensus.cc:697] T cb0b3f259ebc4310927338a8740e87b8 P 7a9daf5d42a84615a6383cfec9a10608 [term 5 LEADER]: Becoming Leader. State: Replica: 7a9daf5d42a84615a6383cfec9a10608, State: Running, Role: LEADER
I20260129 02:59:38.931823 20513 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57600 (request call id 1683) took 1743 ms. Trace:
I20260129 02:59:38.909276 22440 raft_consensus.cc:3060] T 78aa39096868415d9afc2ffcb248e198 P 197b880fa39d432a9b472845876bfaf0 [term 3 FOLLOWER]: Advancing to term 4
I20260129 02:59:38.934985 20513 rpcz_store.cc:276] 0129 02:59:37.188851 (+ 0us) service_pool.cc:168] Inserting onto call queue
0129 02:59:38.916008 (+1727157us) service_pool.cc:225] Handling call
0129 02:59:38.929504 (+ 13496us) raft_consensus.cc:1471] Updating replica for 0 ops
0129 02:59:38.931383 (+ 1879us) raft_consensus.cc:1534] Early marking committed up to index 0
0129 02:59:38.931429 (+ 46us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0129 02:59:38.931485 (+ 56us) raft_consensus.cc:1645] Marking committed up to 3
0129 02:59:38.931532 (+ 47us) raft_consensus.cc:1697] Filling consensus response to leader.
0129 02:59:38.931612 (+ 80us) raft_consensus.cc:1691] UpdateReplicas() finished
0129 02:59:38.931754 (+ 142us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260129 02:59:38.946914 22478 raft_consensus.cc:2749] T fd8311b779af4e1fa1402b2a02a602f2 P 1c50de89a5a845668696dcb28c3badd6 [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260129 02:59:38.906087 22479 raft_consensus.cc:515] T 4f5a77e94fef4b16a2ba7b067c8b144a P 974b802141bd4f13beffef2aa03f88f9 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } }
I20260129 02:59:38.950999 20583 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "78aa39096868415d9afc2ffcb248e198" candidate_uuid: "7a9daf5d42a84615a6383cfec9a10608" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "1c50de89a5a845668696dcb28c3badd6" is_pre_election: true
I20260129 02:59:38.953015 20407 leader_election.cc:304] T 87d5fe48a44a408aaf03e894a67ecb2f P 197b880fa39d432a9b472845876bfaf0 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 197b880fa39d432a9b472845876bfaf0, 974b802141bd4f13beffef2aa03f88f9; no voters: 7a9daf5d42a84615a6383cfec9a10608
I20260129 02:59:38.956748 22082 leader_election.cc:290] T 17d6431cc77a491e8464d68db4081a51 P 197b880fa39d432a9b472845876bfaf0 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 974b802141bd4f13beffef2aa03f88f9 (127.18.211.2:34517), 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3:37735)
I20260129 02:59:38.957531 22082 raft_consensus.cc:2804] T cedce331e1584ac8b636c9b8a72b2ba0 P 197b880fa39d432a9b472845876bfaf0 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260129 02:59:38.957897 22082 raft_consensus.cc:493] T cedce331e1584ac8b636c9b8a72b2ba0 P 197b880fa39d432a9b472845876bfaf0 [term 4 FOLLOWER]: Starting leader election (detected failure of leader 1c50de89a5a845668696dcb28c3badd6)
I20260129 02:59:38.958233 22082 raft_consensus.cc:3060] T cedce331e1584ac8b636c9b8a72b2ba0 P 197b880fa39d432a9b472845876bfaf0 [term 4 FOLLOWER]: Advancing to term 5
I20260129 02:59:38.960100 22302 raft_consensus.cc:2749] T 45cc5f3d2688486d9b6dbe7e666dedde P 974b802141bd4f13beffef2aa03f88f9 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260129 02:59:38.919803 22426 leader_election.cc:290] T 926a250fa5834e88b96ccecbefbf5086 P 1c50de89a5a845668696dcb28c3badd6 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 197b880fa39d432a9b472845876bfaf0 (127.18.211.1:42027), 974b802141bd4f13beffef2aa03f88f9 (127.18.211.2:34517)
I20260129 02:59:38.928825 22337 raft_consensus.cc:493] T dc2f4cb752cf4d59b5f6bc689f5d8104 P 1c50de89a5a845668696dcb28c3badd6 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 7a9daf5d42a84615a6383cfec9a10608)
I20260129 02:59:38.929710 22330 raft_consensus.cc:2804] T 2e58e9a5a8aa4cecafb464addb4f35ed P 197b880fa39d432a9b472845876bfaf0 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260129 02:59:38.930692 22496 raft_consensus.cc:2749] T cedce331e1584ac8b636c9b8a72b2ba0 P 974b802141bd4f13beffef2aa03f88f9 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260129 02:59:38.962138 20407 leader_election.cc:304] T 00e79d8f25a34676b1d866eb2068a2b2 P 197b880fa39d432a9b472845876bfaf0 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 197b880fa39d432a9b472845876bfaf0, 974b802141bd4f13beffef2aa03f88f9; no voters:
I20260129 02:59:38.961194 20513 raft_consensus.cc:1240] T 3ccac82a01a2445fa5b0ce6c0fcf2295 P 974b802141bd4f13beffef2aa03f88f9 [term 2 FOLLOWER]: Rejecting Update request from peer 197b880fa39d432a9b472845876bfaf0 for earlier term 1. Current term is 2. Ops: []
I20260129 02:59:38.978691 22501 raft_consensus.cc:2804] T 87d5fe48a44a408aaf03e894a67ecb2f P 197b880fa39d432a9b472845876bfaf0 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260129 02:59:38.981016 22437 raft_consensus.cc:493] T 0340229d521b432580ae926f6062e3f7 P 974b802141bd4f13beffef2aa03f88f9 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 197b880fa39d432a9b472845876bfaf0)
I20260129 02:59:38.985402 20407 leader_election.cc:304] T 368953c9102643b2a21f99af36b51905 P 197b880fa39d432a9b472845876bfaf0 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 197b880fa39d432a9b472845876bfaf0, 974b802141bd4f13beffef2aa03f88f9; no voters:
I20260129 02:59:39.025063 20513 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57600 (request call id 1687) took 1816 ms. Trace:
I20260129 02:59:39.025625 20513 rpcz_store.cc:276] 0129 02:59:37.208562 (+ 0us) service_pool.cc:168] Inserting onto call queue
0129 02:59:38.936880 (+1728318us) service_pool.cc:225] Handling call
0129 02:59:38.961026 (+ 24146us) raft_consensus.cc:1471] Updating replica for 0 ops
0129 02:59:39.024738 (+ 63712us) raft_consensus.cc:1697] Filling consensus response to leader.
0129 02:59:39.024989 (+ 251us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20260129 02:59:39.029172 20407 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.211.2:34517, user_credentials={real_user=slave}} blocked reactor thread for 43882.5us
W20260129 02:59:39.030118 21101 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260129 02:59:39.024086 22302 raft_consensus.cc:493] T 913fc7d751ed4b7388d889aac30a5f9b P 974b802141bd4f13beffef2aa03f88f9 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260129 02:59:39.031535 22302 raft_consensus.cc:515] T 913fc7d751ed4b7388d889aac30a5f9b P 974b802141bd4f13beffef2aa03f88f9 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:59:39.033771 22302 leader_election.cc:290] T 913fc7d751ed4b7388d889aac30a5f9b P 974b802141bd4f13beffef2aa03f88f9 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 197b880fa39d432a9b472845876bfaf0 (127.18.211.1:42027), 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3:37735)
I20260129 02:59:39.034580 22355 raft_consensus.cc:493] T 505580c93c694bbb8ea96fdf9b17ab46 P 974b802141bd4f13beffef2aa03f88f9 [term 3 FOLLOWER]: Starting leader election (detected failure of leader 1c50de89a5a845668696dcb28c3badd6)
I20260129 02:59:39.035102 22355 raft_consensus.cc:3060] T 505580c93c694bbb8ea96fdf9b17ab46 P 974b802141bd4f13beffef2aa03f88f9 [term 3 FOLLOWER]: Advancing to term 4
W20260129 02:59:39.039755 22469 log.cc:927] Time spent T a5a05fbcbbcf4916bbc01fd1a5b5e90d P 974b802141bd4f13beffef2aa03f88f9: Append to log took a long time: real 0.058s user 0.001s sys 0.000s
W20260129 02:59:39.032019 22369 log.cc:927] Time spent T 75b0b6e82ddc4f92b1fe1d94f9a0b73b P 1c50de89a5a845668696dcb28c3badd6: Append to log took a long time: real 2.211s user 0.005s sys 0.010s
I20260129 02:59:39.040735 20407 leader_election.cc:304] T 1ce4ddf5b10e4ae8b193b13b2e5936ee P 197b880fa39d432a9b472845876bfaf0 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 197b880fa39d432a9b472845876bfaf0, 974b802141bd4f13beffef2aa03f88f9; no voters:
I20260129 02:59:39.042152 20590 raft_consensus.cc:1275] T 1299723a4476420ab02fd8bfd7aa0645 P 1c50de89a5a845668696dcb28c3badd6 [term 6 FOLLOWER]: Refusing update from remote peer 7a9daf5d42a84615a6383cfec9a10608: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20260129 02:59:39.044023 20584 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "42d27a28d89b409e8807ec0efd97af69" candidate_uuid: "7a9daf5d42a84615a6383cfec9a10608" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "1c50de89a5a845668696dcb28c3badd6" is_pre_election: true
I20260129 02:59:39.044886 20584 raft_consensus.cc:2468] T 42d27a28d89b409e8807ec0efd97af69 P 1c50de89a5a845668696dcb28c3badd6 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7a9daf5d42a84615a6383cfec9a10608 in term 4.
I20260129 02:59:39.033767 22237 raft_consensus.cc:2804] T 00e79d8f25a34676b1d866eb2068a2b2 P 197b880fa39d432a9b472845876bfaf0 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260129 02:59:39.048909 22237 raft_consensus.cc:493] T 00e79d8f25a34676b1d866eb2068a2b2 P 197b880fa39d432a9b472845876bfaf0 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260129 02:59:39.049288 22237 raft_consensus.cc:3060] T 00e79d8f25a34676b1d866eb2068a2b2 P 197b880fa39d432a9b472845876bfaf0 [term 4 FOLLOWER]: Advancing to term 5
I20260129 02:59:39.052340 20407 leader_election.cc:304] T e4d18c35b0a7493e85d0a4b92956a48f P 197b880fa39d432a9b472845876bfaf0 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 197b880fa39d432a9b472845876bfaf0, 974b802141bd4f13beffef2aa03f88f9; no voters:
I20260129 02:59:39.090937 22082 raft_consensus.cc:515] T cedce331e1584ac8b636c9b8a72b2ba0 P 197b880fa39d432a9b472845876bfaf0 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:59:39.093804 22326 leader_election.cc:290] T c2633a6fc34c44579de348d208b0c01e P 197b880fa39d432a9b472845876bfaf0 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 974b802141bd4f13beffef2aa03f88f9 (127.18.211.2:34517), 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3:37735)
I20260129 02:59:39.094121 22323 raft_consensus.cc:3060] T cb0b3f259ebc4310927338a8740e87b8 P 974b802141bd4f13beffef2aa03f88f9 [term 3 FOLLOWER]: Advancing to term 4
I20260129 02:59:39.094521 22326 raft_consensus.cc:2804] T 368953c9102643b2a21f99af36b51905 P 197b880fa39d432a9b472845876bfaf0 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260129 02:59:39.094940 22326 raft_consensus.cc:493] T 368953c9102643b2a21f99af36b51905 P 197b880fa39d432a9b472845876bfaf0 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260129 02:59:39.095299 22326 raft_consensus.cc:3060] T 368953c9102643b2a21f99af36b51905 P 197b880fa39d432a9b472845876bfaf0 [term 4 FOLLOWER]: Advancing to term 5
I20260129 02:59:39.028168 20509 raft_consensus.cc:1240] T 3906b080a5b34a74b8b64128635e8997 P 974b802141bd4f13beffef2aa03f88f9 [term 2 FOLLOWER]: Rejecting Update request from peer 197b880fa39d432a9b472845876bfaf0 for earlier term 1. Current term is 2. Ops: []
I20260129 02:59:39.085179 20513 raft_consensus.cc:1240] T b189f08e16a74a11980cc55d99359a0a P 974b802141bd4f13beffef2aa03f88f9 [term 5 FOLLOWER]: Rejecting Update request from peer 197b880fa39d432a9b472845876bfaf0 for earlier term 4. Current term is 5. Ops: []
I20260129 02:59:39.108886 20513 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57600 (request call id 1688) took 1884 ms. Trace:
I20260129 02:59:39.110893 20513 rpcz_store.cc:276] 0129 02:59:37.224373 (+ 0us) service_pool.cc:168] Inserting onto call queue
0129 02:59:39.026604 (+1802231us) service_pool.cc:225] Handling call
0129 02:59:39.084978 (+ 58374us) raft_consensus.cc:1471] Updating replica for 0 ops
0129 02:59:39.108520 (+ 23542us) raft_consensus.cc:1697] Filling consensus response to leader.
0129 02:59:39.108812 (+ 292us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260129 02:59:39.115496 20514 raft_consensus.cc:2468] T 75f45538dc704d9a8ec4ff8ad9f3ea5b P 974b802141bd4f13beffef2aa03f88f9 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7a9daf5d42a84615a6383cfec9a10608 in term 2.
W20260129 02:59:39.107147 22495 log.cc:927] Time spent T 926a250fa5834e88b96ccecbefbf5086 P 197b880fa39d432a9b472845876bfaf0: Append to log took a long time: real 0.068s user 0.001s sys 0.000s
I20260129 02:59:38.982909 22330 raft_consensus.cc:493] T 2e58e9a5a8aa4cecafb464addb4f35ed P 197b880fa39d432a9b472845876bfaf0 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260129 02:59:39.118321 22082 leader_election.cc:290] T cedce331e1584ac8b636c9b8a72b2ba0 P 197b880fa39d432a9b472845876bfaf0 [CANDIDATE]: Term 5 election: Requested vote from peers 974b802141bd4f13beffef2aa03f88f9 (127.18.211.2:34517), 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3:37735)
I20260129 02:59:39.153031 22491 leader_election.cc:290] T a8479d1fe0bd41e5946186e3a6ca8c7a P 974b802141bd4f13beffef2aa03f88f9 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 197b880fa39d432a9b472845876bfaf0 (127.18.211.1:42027), 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3:37735)
I20260129 02:59:39.153172 22336 raft_consensus.cc:3060] T fd8311b779af4e1fa1402b2a02a602f2 P 197b880fa39d432a9b472845876bfaf0 [term 2 FOLLOWER]: Advancing to term 3
I20260129 02:59:39.119762 22479 leader_election.cc:290] T 4f5a77e94fef4b16a2ba7b067c8b144a P 974b802141bd4f13beffef2aa03f88f9 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3:37735), 7a9daf5d42a84615a6383cfec9a10608 (127.18.211.4:40439)
I20260129 02:59:38.975519 22475 leader_election.cc:290] T 42d27a28d89b409e8807ec0efd97af69 P 7a9daf5d42a84615a6383cfec9a10608 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 197b880fa39d432a9b472845876bfaf0 (127.18.211.1:42027), 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3:37735)
I20260129 02:59:39.160099 22373 leader_election.cc:290] T 0f20af7fbe5d4db89a6f17948d98912c P 197b880fa39d432a9b472845876bfaf0 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 974b802141bd4f13beffef2aa03f88f9 (127.18.211.2:34517), 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3:37735)
W20260129 02:59:39.120849 22415 log.cc:927] Time spent T 505580c93c694bbb8ea96fdf9b17ab46 P 7a9daf5d42a84615a6383cfec9a10608: Append to log took a long time: real 2.179s user 0.000s sys 0.004s
I20260129 02:59:39.122066 22353 raft_consensus.cc:3060] T e7d83d4ccf684e76acb29a0dc844e1b8 P 1c50de89a5a845668696dcb28c3badd6 [term 2 FOLLOWER]: Advancing to term 3
I20260129 02:59:39.119503 22355 raft_consensus.cc:515] T 505580c93c694bbb8ea96fdf9b17ab46 P 974b802141bd4f13beffef2aa03f88f9 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } }
I20260129 02:59:39.021730 22501 raft_consensus.cc:493] T 87d5fe48a44a408aaf03e894a67ecb2f P 197b880fa39d432a9b472845876bfaf0 [term 4 FOLLOWER]: Starting leader election (detected failure of leader 7a9daf5d42a84615a6383cfec9a10608)
I20260129 02:59:39.137888 22231 raft_consensus.cc:2804] T 1ce4ddf5b10e4ae8b193b13b2e5936ee P 197b880fa39d432a9b472845876bfaf0 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260129 02:59:39.132964 22302 raft_consensus.cc:493] T 75f45538dc704d9a8ec4ff8ad9f3ea5b P 974b802141bd4f13beffef2aa03f88f9 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260129 02:59:39.141212 22384 raft_consensus.cc:3055] T 0340229d521b432580ae926f6062e3f7 P 197b880fa39d432a9b472845876bfaf0 [term 3 LEADER]: Stepping down as leader of term 3
I20260129 02:59:39.141533 20515 raft_consensus.cc:2468] T cb0b3f259ebc4310927338a8740e87b8 P 974b802141bd4f13beffef2aa03f88f9 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7a9daf5d42a84615a6383cfec9a10608 in term 4.
I20260129 02:59:39.142014 22371 raft_consensus.cc:2804] T e4d18c35b0a7493e85d0a4b92956a48f P 197b880fa39d432a9b472845876bfaf0 [term 3 FOLLOWER]: Leader pre-election won for term 4
W20260129 02:59:39.144702 20404 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260129 02:59:39.151286 20581 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:35754 (request call id 1569) took 3311 ms (client timeout 1895 ms). Trace:
W20260129 02:59:39.052619 21135 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34046 (request call id 414) took 1953 ms (client timeout 1674 ms). Trace:
W20260129 02:59:39.170686 20581 rpcz_store.cc:269] 0129 02:59:35.839987 (+ 0us) service_pool.cc:168] Inserting onto call queue
0129 02:59:35.845480 (+ 5493us) service_pool.cc:225] Handling call
0129 02:59:39.151083 (+3305603us) spinlock_profiling.cc:243] Waited 215 ms on lock 0x7b54003a0054. stack: 00007f8fe044e373 00007f8fe044e270 00007f8fdfda0574 00007f8fe9a5c6b4 00007f8fe9a5c649 00007f8fe34975a1 00007f8fe3531f7a 00007f8fe84e3647 00007f8fe1a73834 00007f8fe1a737c2 00007f8fe1a736f1 00007f8fe1a7366c 00007f8fe1a72922 00007f8fe186d34c 00007f8fe186c786 00007f8fe186c12f
0129 02:59:39.151203 (+ 120us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":493510016}
I20260129 02:59:39.151980 20585 raft_consensus.cc:1240] T a8479d1fe0bd41e5946186e3a6ca8c7a P 1c50de89a5a845668696dcb28c3badd6 [term 3 FOLLOWER]: Rejecting Update request from peer 974b802141bd4f13beffef2aa03f88f9 for earlier term 2. Current term is 3. Ops: []
I20260129 02:59:39.155469 20513 raft_consensus.cc:1240] T 913fc7d751ed4b7388d889aac30a5f9b P 974b802141bd4f13beffef2aa03f88f9 [term 2 FOLLOWER]: Rejecting Update request from peer 197b880fa39d432a9b472845876bfaf0 for earlier term 1. Current term is 2. Ops: []
I20260129 02:59:39.157975 21129 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "75f45538dc704d9a8ec4ff8ad9f3ea5b" candidate_uuid: "1c50de89a5a845668696dcb28c3badd6" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "7a9daf5d42a84615a6383cfec9a10608" is_pre_election: true
I20260129 02:59:39.174693 20513 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57600 (request call id 1690) took 1621 ms. Trace:
I20260129 02:59:39.167198 22384 raft_consensus.cc:740] T 0340229d521b432580ae926f6062e3f7 P 197b880fa39d432a9b472845876bfaf0 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 197b880fa39d432a9b472845876bfaf0, State: Running, Role: LEADER
I20260129 02:59:39.029801 22337 raft_consensus.cc:515] T dc2f4cb752cf4d59b5f6bc689f5d8104 P 1c50de89a5a845668696dcb28c3badd6 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } }
I20260129 02:59:39.181833 22486 leader_election.cc:290] T 3ccac82a01a2445fa5b0ce6c0fcf2295 P 7a9daf5d42a84615a6383cfec9a10608 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 197b880fa39d432a9b472845876bfaf0 (127.18.211.1:42027), 974b802141bd4f13beffef2aa03f88f9 (127.18.211.2:34517)
I20260129 02:59:39.155961 22330 raft_consensus.cc:3060] T 2e58e9a5a8aa4cecafb464addb4f35ed P 197b880fa39d432a9b472845876bfaf0 [term 3 FOLLOWER]: Advancing to term 4
I20260129 02:59:39.170017 22302 raft_consensus.cc:515] T 75f45538dc704d9a8ec4ff8ad9f3ea5b P 974b802141bd4f13beffef2aa03f88f9 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } }
I20260129 02:59:39.165241 22483 consensus_queue.cc:237] T cb0b3f259ebc4310927338a8740e87b8 P 7a9daf5d42a84615a6383cfec9a10608 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 3.6, Last appended by leader: 6, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } }
I20260129 02:59:39.167852 22231 raft_consensus.cc:493] T 1ce4ddf5b10e4ae8b193b13b2e5936ee P 197b880fa39d432a9b472845876bfaf0 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260129 02:59:39.176522 21132 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34046 (request call id 409) took 3040 ms. Trace:
W20260129 02:59:39.177448 21129 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34046 (request call id 412) took 2343 ms (client timeout 1563 ms). Trace:
I20260129 02:59:39.153916 22448 leader_election.cc:290] T 2ac659d732314e1288849f08fe5e074e P 974b802141bd4f13beffef2aa03f88f9 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 197b880fa39d432a9b472845876bfaf0 (127.18.211.1:42027), 7a9daf5d42a84615a6383cfec9a10608 (127.18.211.4:40439)
I20260129 02:59:39.245368 20515 raft_consensus.cc:1240] T 0678b8b468bd43cdbc3e485900844821 P 974b802141bd4f13beffef2aa03f88f9 [term 5 FOLLOWER]: Rejecting Update request from peer 7a9daf5d42a84615a6383cfec9a10608 for earlier term 4. Current term is 5. Ops: []
I20260129 02:59:39.246114 20515 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:48544 (request call id 138) took 1336 ms. Trace:
I20260129 02:59:39.246425 20515 rpcz_store.cc:276] 0129 02:59:37.910369 (+ 0us) service_pool.cc:168] Inserting onto call queue
0129 02:59:39.244802 (+1334433us) service_pool.cc:225] Handling call
0129 02:59:39.245164 (+ 362us) raft_consensus.cc:1471] Updating replica for 0 ops
0129 02:59:39.245857 (+ 693us) raft_consensus.cc:1697] Filling consensus response to leader.
0129 02:59:39.246060 (+ 203us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260129 02:59:39.247526 22302 leader_election.cc:290] T 75f45538dc704d9a8ec4ff8ad9f3ea5b P 974b802141bd4f13beffef2aa03f88f9 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3:37735), 7a9daf5d42a84615a6383cfec9a10608 (127.18.211.4:40439)
I20260129 02:59:39.154096 22120 leader_election.cc:290] T a1a39dcd328e4268ae41b988c66b4a75 P 197b880fa39d432a9b472845876bfaf0 [CANDIDATE]: Term 3 election: Requested vote from peers 974b802141bd4f13beffef2aa03f88f9 (127.18.211.2:34517), 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3:37735)
I20260129 02:59:39.162585 20509 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57600 (request call id 1682) took 1965 ms. Trace:
I20260129 02:59:39.028978 22437 raft_consensus.cc:515] T 0340229d521b432580ae926f6062e3f7 P 974b802141bd4f13beffef2aa03f88f9 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:59:39.170353 22371 raft_consensus.cc:493] T e4d18c35b0a7493e85d0a4b92956a48f P 197b880fa39d432a9b472845876bfaf0 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260129 02:59:39.168687 22501 raft_consensus.cc:3060] T 87d5fe48a44a408aaf03e894a67ecb2f P 197b880fa39d432a9b472845876bfaf0 [term 4 FOLLOWER]: Advancing to term 5
I20260129 02:59:39.143623 22237 raft_consensus.cc:515] T 00e79d8f25a34676b1d866eb2068a2b2 P 197b880fa39d432a9b472845876bfaf0 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } }
I20260129 02:59:39.189935 22372 leader_election.cc:290] T c934d31efd3f4e5195605ddc354d14e1 P 7a9daf5d42a84615a6383cfec9a10608 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 197b880fa39d432a9b472845876bfaf0 (127.18.211.1:42027), 974b802141bd4f13beffef2aa03f88f9 (127.18.211.2:34517)
I20260129 02:59:39.201301 22355 leader_election.cc:290] T 505580c93c694bbb8ea96fdf9b17ab46 P 974b802141bd4f13beffef2aa03f88f9 [CANDIDATE]: Term 4 election: Requested vote from peers 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3:37735), 7a9daf5d42a84615a6383cfec9a10608 (127.18.211.4:40439)
I20260129 02:59:39.204713 22184 raft_consensus.cc:3060] T 96609ac68bdf4993a12e1c23355ca013 P 974b802141bd4f13beffef2aa03f88f9 [term 3 FOLLOWER]: Advancing to term 4
I20260129 02:59:39.181704 22384 consensus_queue.cc:260] T 0340229d521b432580ae926f6062e3f7 P 197b880fa39d432a9b472845876bfaf0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:59:37.614454 19276 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260129 02:59:39.211540 20335 rpcz_store.cc:275] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:55376 (request call id 242) took 10316 ms. Trace:
W20260129 02:59:39.212745 20587 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:35752 (request call id 1475) took 3158 ms (client timeout 1911 ms). Trace:
I20260129 02:59:39.218021 20512 raft_consensus.cc:1275] T 4e420f90884f4796997b5ea41e49e411 P 974b802141bd4f13beffef2aa03f88f9 [term 4 FOLLOWER]: Refusing update from remote peer 197b880fa39d432a9b472845876bfaf0: Log matching property violated. Preceding OpId in replica: term: 3 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20260129 02:59:39.175040 20513 rpcz_store.cc:276] 0129 02:59:37.552966 (+ 0us) service_pool.cc:168] Inserting onto call queue
0129 02:59:39.131972 (+1579006us) service_pool.cc:225] Handling call
0129 02:59:39.155255 (+ 23283us) raft_consensus.cc:1471] Updating replica for 0 ops
0129 02:59:39.172539 (+ 17284us) raft_consensus.cc:1697] Filling consensus response to leader.
0129 02:59:39.174638 (+ 2099us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20260129 02:59:39.227067 20404 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260129 02:59:39.238154 22231 raft_consensus.cc:3060] T 1ce4ddf5b10e4ae8b193b13b2e5936ee P 197b880fa39d432a9b472845876bfaf0 [term 4 FOLLOWER]: Advancing to term 5
W20260129 02:59:39.238828 21129 rpcz_store.cc:269] 0129 02:59:36.833582 (+ 0us) service_pool.cc:168] Inserting onto call queue
0129 02:59:36.836272 (+ 2690us) service_pool.cc:225] Handling call
0129 02:59:39.157791 (+2321519us) spinlock_profiling.cc:243] Waited 10.1 ms on lock 0x7b5800111f98. stack: 00007f8fe044e373 00007f8fe044e270 00007f8fdfda0574 00007f8fe9a5c6b4 00007f8fe9a5c649 00007f8fe02c08b1 00007f8fe0442db1 00007f8fe0442c62 00007f8fe92eddd9 00007f8fe846d260 00007f8fe8535dbb 00007f8fe8535e41 00007f8fe84e3bd0 00007f8fe84e34b4 00007f8fe1a73834 00007f8fe1a737c2
0129 02:59:39.177359 (+ 19568us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":23131520}
I20260129 02:59:39.238745 21132 rpcz_store.cc:276] 0129 02:59:36.136300 (+ 0us) service_pool.cc:168] Inserting onto call queue
0129 02:59:36.153017 (+ 16717us) service_pool.cc:225] Handling call
0129 02:59:36.270201 (+117184us) raft_consensus.cc:1471] Updating replica for 1 ops
0129 02:59:36.455970 (+185769us) raft_consensus.cc:1534] Early marking committed up to index 0
0129 02:59:36.464161 (+ 8191us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0129 02:59:36.481373 (+ 17212us) log.cc:844] Serialized 26 byte log entry
0129 02:59:36.761530 (+280157us) raft_consensus.cc:1645] Marking committed up to 5
0129 02:59:36.768730 (+ 7200us) raft_consensus.cc:1697] Filling consensus response to leader.
0129 02:59:36.810610 (+ 41880us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0129 02:59:39.120188 (+2309578us) spinlock_profiling.cc:243] Waited 2.21 ms on lock 0x7b440002304c. stack: 00007f8fe044e373 00007f8fe044e270 00007f8fdfda0574 00007f8fe9a5c6b4 00007f8fe9a5c649 00007f8fe02c08b1 00007f8fe038f005 00007f8fe038fa22 00007f8fe30d957b 00007f8fe30d4e4f 00007f8fe30c8c0e 00007f8fe30c7f56 00007f8fe30c7b78 00007f8fe30d15c1 00007f8fe30d1579 00007f8fe30d1509
0129 02:59:39.162386 (+ 42198us) raft_consensus.cc:1685] finished
0129 02:59:39.165326 (+ 2940us) raft_consensus.cc:1691] UpdateReplicas() finished
0129 02:59:39.176443 (+ 11117us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":60055,"raft.run_cpu_time_us":95,"raft.run_wall_time_us":95,"spinlock_wait_cycles":5089664,"thread_start_us":239620,"threads_started":2,"wal-append.queue_time_us":369871}
W20260129 02:59:39.171416 21135 rpcz_store.cc:269] 0129 02:59:37.099243 (+ 0us) service_pool.cc:168] Inserting onto call queue
0129 02:59:37.099446 (+ 203us) service_pool.cc:225] Handling call
0129 02:59:39.052333 (+1952887us) spinlock_profiling.cc:243] Waited 353 ms on lock 0x7b54004ca254. stack: 00007f8fe044e373 00007f8fe044e270 00007f8fdfda0574 00007f8fe9a5c6b4 00007f8fe9a5c649 00007f8fe34975a1 00007f8fe3531f7a 00007f8fe84e3647 00007f8fe1a73834 00007f8fe1a737c2 00007f8fe1a736f1 00007f8fe1a7366c 00007f8fe1a72922 00007f8fe186d34c 00007f8fe186c786 00007f8fe186c12f
0129 02:59:39.052527 (+ 194us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":811238656}
W20260129 02:59:39.240404 22451 consensus_meta.cc:287] T 17d6431cc77a491e8464d68db4081a51 P 974b802141bd4f13beffef2aa03f88f9: Time spent flushing consensus metadata: real 0.695s user 0.008s sys 0.004s
I20260129 02:59:39.249279 20338 catalog_manager.cc:5654] T a0fcc58d6fb6473cafbd9d34d5195269 P 197b880fa39d432a9b472845876bfaf0 reported cstate change: term changed from 1 to 2, leader changed from 974b802141bd4f13beffef2aa03f88f9 (127.18.211.2) to 197b880fa39d432a9b472845876bfaf0 (127.18.211.1). New cstate: current_term: 2 leader_uuid: "197b880fa39d432a9b472845876bfaf0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } health_report { overall_health: HEALTHY } } }
W20260129 02:59:39.271268 21101 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260129 02:59:39.275657 21100 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.211.3:37735, user_credentials={real_user=slave}} blocked reactor thread for 175303us
W20260129 02:59:39.278187 20406 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.598s user 0.014s sys 0.067s
W20260129 02:59:39.280552 20482 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260129 02:59:39.289602 22337 leader_election.cc:290] T dc2f4cb752cf4d59b5f6bc689f5d8104 P 1c50de89a5a845668696dcb28c3badd6 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 197b880fa39d432a9b472845876bfaf0 (127.18.211.1:42027), 7a9daf5d42a84615a6383cfec9a10608 (127.18.211.4:40439)
W20260129 02:59:39.291328 21099 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260129 02:59:39.297094 20556 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.544s user 0.031s sys 0.065s
I20260129 02:59:39.397878 22437 leader_election.cc:290] T 0340229d521b432580ae926f6062e3f7 P 974b802141bd4f13beffef2aa03f88f9 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 197b880fa39d432a9b472845876bfaf0 (127.18.211.1:42027), 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3:37735)
W20260129 02:59:39.400354 20554 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.613s user 0.016s sys 0.035s
W20260129 02:59:39.400732 20404 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260129 02:59:39.401162 20432 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.573s user 0.017s sys 0.035s
W20260129 02:59:39.401422 20404 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260129 02:59:39.399960 20556 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.647s user 0.032s sys 0.070s
I20260129 02:59:39.403314 21100 leader_election.cc:304] T 42d27a28d89b409e8807ec0efd97af69 P 7a9daf5d42a84615a6383cfec9a10608 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1c50de89a5a845668696dcb28c3badd6, 7a9daf5d42a84615a6383cfec9a10608; no voters:
W20260129 02:59:39.404112 20556 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.211.2:34517, user_credentials={real_user=slave}} blocked reactor thread for 673842us
I20260129 02:59:39.406075 20513 raft_consensus.cc:1240] T a609ac02fae84725ab743f95f51d33ce P 974b802141bd4f13beffef2aa03f88f9 [term 5 FOLLOWER]: Rejecting Update request from peer 7a9daf5d42a84615a6383cfec9a10608 for earlier term 4. Current term is 5. Ops: []
I20260129 02:59:39.407012 20513 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:48544 (request call id 143) took 1494 ms. Trace:
I20260129 02:59:39.407514 20513 rpcz_store.cc:276] 0129 02:59:37.913093 (+ 0us) service_pool.cc:168] Inserting onto call queue
0129 02:59:39.400415 (+1487322us) service_pool.cc:225] Handling call
0129 02:59:39.405879 (+ 5464us) raft_consensus.cc:1471] Updating replica for 0 ops
0129 02:59:39.406697 (+ 818us) raft_consensus.cc:1697] Filling consensus response to leader.
0129 02:59:39.406944 (+ 247us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20260129 02:59:39.400794 20554 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.614s user 0.016s sys 0.035s
W20260129 02:59:39.412979 20587 rpcz_store.cc:269] 0129 02:59:36.053887 (+ 0us) service_pool.cc:168] Inserting onto call queue
0129 02:59:36.078900 (+ 25013us) service_pool.cc:225] Handling call
0129 02:59:39.212498 (+3133598us) spinlock_profiling.cc:243] Waited 1.28 s on lock 0x7b54000c9254. stack: 00007f8fe044e373 00007f8fe044e270 00007f8fdfda0574 00007f8fe9a5c6b4 00007f8fe9a5c649 00007f8fe34975a1 00007f8fe3531f7a 00007f8fe84e3647 00007f8fe1a73834 00007f8fe1a737c2 00007f8fe1a736f1 00007f8fe1a7366c 00007f8fe1a72922 00007f8fe186d34c 00007f8fe186c786 00007f8fe186c12f
0129 02:59:39.212652 (+ 154us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2941966080}
I20260129 02:59:39.412586 20512 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57600 (request call id 1695) took 1786 ms. Trace:
I20260129 02:59:39.416003 20512 rpcz_store.cc:276] 0129 02:59:37.625821 (+ 0us) service_pool.cc:168] Inserting onto call queue
0129 02:59:39.217165 (+1591344us) service_pool.cc:225] Handling call
0129 02:59:39.217587 (+ 422us) raft_consensus.cc:1471] Updating replica for 0 ops
0129 02:59:39.412210 (+194623us) raft_consensus.cc:1697] Filling consensus response to leader.
0129 02:59:39.412513 (+ 303us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260129 02:59:39.410796 20338 catalog_manager.cc:5654] T b189f08e16a74a11980cc55d99359a0a P 197b880fa39d432a9b472845876bfaf0 reported cstate change: term changed from 2 to 4. New cstate: current_term: 4 leader_uuid: "197b880fa39d432a9b472845876bfaf0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } health_report { overall_health: HEALTHY } } }
I20260129 02:59:39.440248 20556 leader_election.cc:304] T c891d2bff4ce4568861bbc992e91fbb0 P 1c50de89a5a845668696dcb28c3badd6 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1c50de89a5a845668696dcb28c3badd6; no voters: 197b880fa39d432a9b472845876bfaf0, 974b802141bd4f13beffef2aa03f88f9
W20260129 02:59:39.451339 20406 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.796s user 0.014s sys 0.069s
W20260129 02:59:39.453733 21101 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260129 02:59:39.455376 20553 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.668s user 0.013s sys 0.047s
W20260129 02:59:39.468156 20553 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.681s user 0.013s sys 0.047s
W20260129 02:59:39.468484 20553 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.211.1:42027, user_credentials={real_user=slave}} blocked reactor thread for 723619us
I20260129 02:59:39.273756 22371 raft_consensus.cc:3060] T e4d18c35b0a7493e85d0a4b92956a48f P 197b880fa39d432a9b472845876bfaf0 [term 3 FOLLOWER]: Advancing to term 4
W20260129 02:59:39.456461 20554 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260129 02:59:39.469563 22387 raft_consensus.cc:2804] T 42d27a28d89b409e8807ec0efd97af69 P 7a9daf5d42a84615a6383cfec9a10608 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260129 02:59:39.459242 20338 catalog_manager.cc:5654] T 1ce4ddf5b10e4ae8b193b13b2e5936ee P 197b880fa39d432a9b472845876bfaf0 reported cstate change: term changed from 1 to 2, leader changed from 197b880fa39d432a9b472845876bfaf0 (127.18.211.1) to 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3). New cstate: current_term: 2 leader_uuid: "1c50de89a5a845668696dcb28c3badd6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } } }
I20260129 02:59:39.397070 22384 raft_consensus.cc:3060] T 0340229d521b432580ae926f6062e3f7 P 197b880fa39d432a9b472845876bfaf0 [term 3 FOLLOWER]: Advancing to term 4
W20260129 02:59:39.463984 21102 thread.cc:641] dns-resolver [worker] (thread pool) Time spent creating pthread: real 0.773s user 0.003s sys 0.012s
W20260129 02:59:39.470920 21102 thread.cc:608] dns-resolver [worker] (thread pool) Time spent starting thread: real 0.780s user 0.003s sys 0.012s
W20260129 02:59:39.471232 21102 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.211.2:34517, user_credentials={real_user=slave}} blocked reactor thread for 780914us
W20260129 02:59:39.465077 20482 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260129 02:59:39.469991 22387 raft_consensus.cc:493] T 42d27a28d89b409e8807ec0efd97af69 P 7a9daf5d42a84615a6383cfec9a10608 [term 4 FOLLOWER]: Starting leader election (detected failure of leader 197b880fa39d432a9b472845876bfaf0)
I20260129 02:59:39.472169 22387 raft_consensus.cc:3060] T 42d27a28d89b409e8807ec0efd97af69 P 7a9daf5d42a84615a6383cfec9a10608 [term 4 FOLLOWER]: Advancing to term 5
W20260129 02:59:39.401482 20432 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.573s user 0.017s sys 0.035s
W20260129 02:59:39.473208 21102 leader_election.cc:336] T 45cc5f3d2688486d9b6dbe7e666dedde P 7a9daf5d42a84615a6383cfec9a10608 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 974b802141bd4f13beffef2aa03f88f9 (127.18.211.2:34517): Timed out: RequestConsensusVote RPC to 127.18.211.2:34517 timed out after 1.648s (SENT)
W20260129 02:59:39.470513 21100 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.211.3:37735, user_credentials={real_user=slave}} blocked reactor thread for 67310.6us
I20260129 02:59:39.473652 21102 leader_election.cc:304] T 45cc5f3d2688486d9b6dbe7e666dedde P 7a9daf5d42a84615a6383cfec9a10608 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7a9daf5d42a84615a6383cfec9a10608; no voters: 1c50de89a5a845668696dcb28c3badd6, 974b802141bd4f13beffef2aa03f88f9
I20260129 02:59:39.475235 20587 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4e3805ffa70242e8ade66da260dddf66" candidate_uuid: "7a9daf5d42a84615a6383cfec9a10608" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "1c50de89a5a845668696dcb28c3badd6" is_pre_election: true
I20260129 02:59:39.474925 20338 catalog_manager.cc:5654] T a4c1226b987b468fa84dfc6ba0e2dbe1 P 197b880fa39d432a9b472845876bfaf0 reported cstate change: term changed from 1 to 2, leader changed from 974b802141bd4f13beffef2aa03f88f9 (127.18.211.2) to 197b880fa39d432a9b472845876bfaf0 (127.18.211.1). New cstate: current_term: 2 leader_uuid: "197b880fa39d432a9b472845876bfaf0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } health_report { overall_health: UNKNOWN } } }
I20260129 02:59:39.477066 20338 catalog_manager.cc:5654] T 1299723a4476420ab02fd8bfd7aa0645 P 197b880fa39d432a9b472845876bfaf0 reported cstate change: term changed from 2 to 4, now has a pending config: VOTER 197b880fa39d432a9b472845876bfaf0 (127.18.211.1), VOTER 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3), VOTER 7a9daf5d42a84615a6383cfec9a10608 (127.18.211.4). New cstate: current_term: 4 leader_uuid: "197b880fa39d432a9b472845876bfaf0" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } attrs { replace: true } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } health_report { overall_health: UNKNOWN } } } pending_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } } }
I20260129 02:59:39.383281 22224 raft_consensus.cc:493] T 8055232cf0004d40a85f3bf983fe00a9 P 1c50de89a5a845668696dcb28c3badd6 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260129 02:59:39.485603 20554 leader_election.cc:304] T a644c02b16af47679824ccd060ac10fd P 1c50de89a5a845668696dcb28c3badd6 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1c50de89a5a845668696dcb28c3badd6, 7a9daf5d42a84615a6383cfec9a10608; no voters:
W20260129 02:59:39.486070 20556 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.211.2:34517, user_credentials={real_user=slave}} blocked reactor thread for 46054.5us
W20260129 02:59:39.455540 22435 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.586s user 0.002s sys 0.000s
I20260129 02:59:39.286993 22416 raft_consensus.cc:493] T 75b0b6e82ddc4f92b1fe1d94f9a0b73b P 974b802141bd4f13beffef2aa03f88f9 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260129 02:59:39.343631 22414 leader_election.cc:290] T 4e3805ffa70242e8ade66da260dddf66 P 7a9daf5d42a84615a6383cfec9a10608 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 197b880fa39d432a9b472845876bfaf0 (127.18.211.1:42027), 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3:37735)
W20260129 02:59:39.494839 21100 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260129 02:59:39.494676 21102 leader_election.cc:336] T 3e2d15f8640b407195210c476e09df91 P 7a9daf5d42a84615a6383cfec9a10608 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 974b802141bd4f13beffef2aa03f88f9 (127.18.211.2:34517): Timed out: RequestConsensusVote RPC to 127.18.211.2:34517 timed out after 1.676s (SENT)
W20260129 02:59:39.585373 20404 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260129 02:59:39.611101 21102 leader_election.cc:304] T 3e2d15f8640b407195210c476e09df91 P 7a9daf5d42a84615a6383cfec9a10608 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7a9daf5d42a84615a6383cfec9a10608; no voters: 1c50de89a5a845668696dcb28c3badd6, 974b802141bd4f13beffef2aa03f88f9
I20260129 02:59:39.640751 22237 leader_election.cc:290] T 00e79d8f25a34676b1d866eb2068a2b2 P 197b880fa39d432a9b472845876bfaf0 [CANDIDATE]: Term 5 election: Requested vote from peers 974b802141bd4f13beffef2aa03f88f9 (127.18.211.2:34517), 7a9daf5d42a84615a6383cfec9a10608 (127.18.211.4:40439)
I20260129 02:59:39.646296 22486 raft_consensus.cc:493] T a609ac02fae84725ab743f95f51d33ce P 7a9daf5d42a84615a6383cfec9a10608 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260129 02:59:39.647066 22486 raft_consensus.cc:515] T a609ac02fae84725ab743f95f51d33ce P 7a9daf5d42a84615a6383cfec9a10608 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } }
W20260129 02:59:39.655473 20553 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.211.1:42027, user_credentials={real_user=slave}} blocked reactor thread for 152416us
W20260129 02:59:39.688781 20554 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.211.4:40439, user_credentials={real_user=slave}} blocked reactor thread for 51409.4us
W20260129 02:59:39.503961 22435 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.634s user 0.005s sys 0.002s
I20260129 02:59:39.438287 22374 consensus_queue.cc:1048] T 1299723a4476420ab02fd8bfd7aa0645 P 7a9daf5d42a84615a6383cfec9a10608 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 4, Time since last communication: 0.081s
I20260129 02:59:39.676733 22437 raft_consensus.cc:493] T 17d6431cc77a491e8464d68db4081a51 P 974b802141bd4f13beffef2aa03f88f9 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260129 02:59:39.487336 22320 raft_consensus.cc:493] T b189f08e16a74a11980cc55d99359a0a P 1c50de89a5a845668696dcb28c3badd6 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 197b880fa39d432a9b472845876bfaf0)
I20260129 02:59:39.501585 22372 raft_consensus.cc:2749] T 45cc5f3d2688486d9b6dbe7e666dedde P 7a9daf5d42a84615a6383cfec9a10608 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260129 02:59:39.676608 22376 raft_consensus.cc:2749] T 3e2d15f8640b407195210c476e09df91 P 7a9daf5d42a84615a6383cfec9a10608 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260129 02:59:39.649422 22418 raft_consensus.cc:2804] T a644c02b16af47679824ccd060ac10fd P 1c50de89a5a845668696dcb28c3badd6 [term 3 FOLLOWER]: Leader election won for term 3
I20260129 02:59:39.519245 22338 raft_consensus.cc:2749] T c891d2bff4ce4568861bbc992e91fbb0 P 1c50de89a5a845668696dcb28c3badd6 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260129 02:59:39.518085 22500 raft_consensus.cc:493] T 4f8f3a38f6e24ba4857875181f447442 P 197b880fa39d432a9b472845876bfaf0 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 1c50de89a5a845668696dcb28c3badd6)
I20260129 02:59:39.456358 22326 raft_consensus.cc:515] T 368953c9102643b2a21f99af36b51905 P 197b880fa39d432a9b472845876bfaf0 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:59:39.715327 22437 raft_consensus.cc:515] T 17d6431cc77a491e8464d68db4081a51 P 974b802141bd4f13beffef2aa03f88f9 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:59:39.726964 20440 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a0fcc58d6fb6473cafbd9d34d5195269" candidate_uuid: "1c50de89a5a845668696dcb28c3badd6" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "197b880fa39d432a9b472845876bfaf0" is_pre_election: true
I20260129 02:59:39.689826 22343 raft_consensus.cc:3060] T 18f30ebaa4174fd28c958e72d0cc7e24 P 7a9daf5d42a84615a6383cfec9a10608 [term 2 FOLLOWER]: Advancing to term 3
I20260129 02:59:39.739432 20437 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0340229d521b432580ae926f6062e3f7" candidate_uuid: "1c50de89a5a845668696dcb28c3badd6" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "197b880fa39d432a9b472845876bfaf0" is_pre_election: true
I20260129 02:59:39.455988 22388 consensus_queue.cc:1048] T 36ff33be666a4c82954731691dbcf327 P 1c50de89a5a845668696dcb28c3badd6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.226s
I20260129 02:59:39.733947 20439 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4b1876f9a70b49edb3aa74105e66d3b3" candidate_uuid: "1c50de89a5a845668696dcb28c3badd6" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "197b880fa39d432a9b472845876bfaf0" is_pre_election: true
I20260129 02:59:39.502490 22349 consensus_queue.cc:1048] T 368953c9102643b2a21f99af36b51905 P 1c50de89a5a845668696dcb28c3badd6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.218s
I20260129 02:59:39.679092 22416 raft_consensus.cc:515] T 75b0b6e82ddc4f92b1fe1d94f9a0b73b P 974b802141bd4f13beffef2aa03f88f9 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } }
I20260129 02:59:39.503577 22224 raft_consensus.cc:515] T 8055232cf0004d40a85f3bf983fe00a9 P 1c50de89a5a845668696dcb28c3badd6 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:59:39.715760 22341 consensus_queue.cc:1048] T c2633a6fc34c44579de348d208b0c01e P 1c50de89a5a845668696dcb28c3badd6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.037s
I20260129 02:59:39.759904 20439 raft_consensus.cc:3055] T 913fc7d751ed4b7388d889aac30a5f9b P 197b880fa39d432a9b472845876bfaf0 [term 1 LEADER]: Stepping down as leader of term 1
W20260129 02:59:39.762799 20554 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.211.4:40439, user_credentials={real_user=slave}} blocked reactor thread for 51846.8us
I20260129 02:59:39.763087 20439 raft_consensus.cc:740] T 913fc7d751ed4b7388d889aac30a5f9b P 197b880fa39d432a9b472845876bfaf0 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 197b880fa39d432a9b472845876bfaf0, State: Running, Role: LEADER
I20260129 02:59:39.306702 20335 rpcz_store.cc:276] 0129 02:59:28.894918 (+ 0us) service_pool.cc:168] Inserting onto call queue
0129 02:59:28.895080 (+ 162us) service_pool.cc:225] Handling call
0129 02:59:39.211443 (+10316363us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0129 02:59:35.072914 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0129 02:59:35.073683 (+ 769us) write_op.cc:432] Acquiring schema lock in shared mode
0129 02:59:35.073748 (+ 65us) write_op.cc:435] Acquired schema lock
0129 02:59:35.073770 (+ 22us) tablet.cc:660] Decoding operations
0129 02:59:35.074971 (+ 1201us) write_op.cc:620] Acquiring the partition lock for write op
0129 02:59:35.075090 (+ 119us) write_op.cc:641] Partition lock acquired for write op
0129 02:59:35.075128 (+ 38us) tablet.cc:683] Acquiring locks for 8 operations
0129 02:59:35.092023 (+ 16895us) tablet.cc:699] Row locks acquired
0129 02:59:35.092045 (+ 22us) write_op.cc:260] PREPARE: finished
0129 02:59:35.092208 (+ 163us) write_op.cc:270] Start()
0129 02:59:35.092380 (+ 172us) write_op.cc:276] Timestamp: P: 1769655575092144 usec, L: 0
0129 02:59:35.092399 (+ 19us) op_driver.cc:348] REPLICATION: starting
0129 02:59:35.147946 (+ 55547us) log.cc:844] Serialized 3229 byte log entry
0129 02:59:35.882428 (+734482us) op_driver.cc:464] REPLICATION: finished
0129 02:59:36.134735 (+252307us) write_op.cc:301] APPLY: starting
0129 02:59:36.150745 (+ 16010us) tablet.cc:1365] starting BulkCheckPresence
0129 02:59:36.216686 (+ 65941us) tablet.cc:1368] finished BulkCheckPresence
0129 02:59:36.274560 (+ 57874us) tablet.cc:1370] starting ApplyRowOperation cycle
0129 02:59:36.651004 (+376444us) tablet.cc:1381] finished ApplyRowOperation cycle
0129 02:59:36.651244 (+ 240us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=8
0129 02:59:36.651297 (+ 53us) write_op.cc:312] APPLY: finished
0129 02:59:36.651623 (+ 326us) log.cc:844] Serialized 65 byte log entry
0129 02:59:36.842056 (+190433us) write_op.cc:489] Releasing partition, row and schema locks
0129 02:59:36.857998 (+ 15942us) write_op.cc:454] Released schema lock
0129 02:59:36.858620 (+ 622us) write_op.cc:341] FINISH: Updating metrics
0129 02:59:36.859001 (+ 381us) write_op.cc:489] Releasing partition, row and schema locks
0129 02:59:36.859029 (+ 28us) write_op.cc:454] Released schema lock
Metrics: {"reported_tablets":10,"child_traces":[["op",{"apply.queue_time_us":205531,"apply.run_cpu_time_us":48829,"apply.run_wall_time_us":734153,"num_ops":8,"prepare.queue_time_us":765724,"prepare.run_cpu_time_us":41791,"prepare.run_wall_time_us":195430,"raft.queue_time_us":382031,"raft.run_cpu_time_us":18345,"raft.run_wall_time_us":177933,"replication_time_us":761213,"thread_start_us":870378,"threads_started":5,"wal-append.queue_time_us":381213,"wal-append.run_cpu_time_us":53577,"wal-append.run_wall_time_us":2756884}]]}
I20260129 02:59:39.760097 20440 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "18f30ebaa4174fd28c958e72d0cc7e24" candidate_uuid: "1c50de89a5a845668696dcb28c3badd6" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "197b880fa39d432a9b472845876bfaf0" is_pre_election: true
I20260129 02:59:39.781788 21133 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34046 (request call id 410) took 3629 ms. Trace:
W20260129 02:59:39.784339 21102 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.211.2:34517, user_credentials={real_user=slave}} blocked reactor thread for 289659us
I20260129 02:59:39.765486 20440 raft_consensus.cc:2468] T 18f30ebaa4174fd28c958e72d0cc7e24 P 197b880fa39d432a9b472845876bfaf0 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1c50de89a5a845668696dcb28c3badd6 in term 2.
I20260129 02:59:39.786679 20440 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "42d27a28d89b409e8807ec0efd97af69" candidate_uuid: "1c50de89a5a845668696dcb28c3badd6" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "197b880fa39d432a9b472845876bfaf0" is_pre_election: true
I20260129 02:59:39.788035 20584 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1ce4ddf5b10e4ae8b193b13b2e5936ee" candidate_uuid: "974b802141bd4f13beffef2aa03f88f9" candidate_term: 5 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1c50de89a5a845668696dcb28c3badd6" is_pre_election: true
W20260129 02:59:39.785514 21102 leader_election.cc:336] T febffe4a8b394368b69b1d164308fb9a P 7a9daf5d42a84615a6383cfec9a10608 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 974b802141bd4f13beffef2aa03f88f9 (127.18.211.2:34517): Timed out: RequestConsensusVote RPC to 127.18.211.2:34517 timed out after 1.821s (SENT)
I20260129 02:59:39.788666 20440 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "17d6431cc77a491e8464d68db4081a51" candidate_uuid: "1c50de89a5a845668696dcb28c3badd6" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "197b880fa39d432a9b472845876bfaf0" is_pre_election: true
I20260129 02:59:39.782526 21133 rpcz_store.cc:276] 0129 02:59:36.152619 (+ 0us) service_pool.cc:168] Inserting onto call queue
0129 02:59:36.187582 (+ 34963us) service_pool.cc:225] Handling call
0129 02:59:36.208356 (+ 20774us) raft_consensus.cc:1471] Updating replica for 0 ops
0129 02:59:38.303639 (+2095283us) raft_consensus.cc:1697] Filling consensus response to leader.
0129 02:59:39.781420 (+1477781us) spinlock_profiling.cc:243] Waited 2.06 s on lock 0x7b54005cc954. stack: 00007f8fe044e373 00007f8fe044e270 00007f8fdfda0574 00007f8fe9a5c6b4 00007f8fe9a5c649 00007f8fe34975a1 00007f8fe352ea81 00007f8fe352d4be 00007f8fe84e1f49 00007f8fe1a6ed34 00007f8fe1a6ecc2 00007f8fe1a6ebf1 00007f8fe1a6eb6c 00007f8fe1a6de22 00007f8fe186d34c 00007f8fe186c786
0129 02:59:39.781702 (+ 282us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":4741612288}
I20260129 02:59:39.789328 20440 raft_consensus.cc:2468] T 17d6431cc77a491e8464d68db4081a51 P 197b880fa39d432a9b472845876bfaf0 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1c50de89a5a845668696dcb28c3badd6 in term 2.
W20260129 02:59:39.790498 22482 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.640s user 0.004s sys 0.048s
I20260129 02:59:39.790510 22352 raft_consensus.cc:493] T 4408006eb5174de1896b037cf3f18457 P 1c50de89a5a845668696dcb28c3badd6 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 974b802141bd4f13beffef2aa03f88f9)
W20260129 02:59:39.791149 22482 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.958s user 0.004s sys 0.049s
I20260129 02:59:39.791518 22352 raft_consensus.cc:515] T 4408006eb5174de1896b037cf3f18457 P 1c50de89a5a845668696dcb28c3badd6 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:59:39.794381 22352 leader_election.cc:290] T 4408006eb5174de1896b037cf3f18457 P 1c50de89a5a845668696dcb28c3badd6 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 197b880fa39d432a9b472845876bfaf0 (127.18.211.1:42027), 974b802141bd4f13beffef2aa03f88f9 (127.18.211.2:34517)
W20260129 02:59:39.754588 22353 consensus_meta.cc:287] T e7d83d4ccf684e76acb29a0dc844e1b8 P 1c50de89a5a845668696dcb28c3badd6: Time spent flushing consensus metadata: real 0.593s user 0.010s sys 0.001s
I20260129 02:59:39.799794 20512 raft_consensus.cc:1275] T 75b0b6e82ddc4f92b1fe1d94f9a0b73b P 974b802141bd4f13beffef2aa03f88f9 [term 2 FOLLOWER]: Refusing update from remote peer 7a9daf5d42a84615a6383cfec9a10608: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260129 02:59:39.250998 20509 rpcz_store.cc:276] 0129 02:59:37.188422 (+ 0us) service_pool.cc:168] Inserting onto call queue
0129 02:59:38.920528 (+1732106us) service_pool.cc:225] Handling call
0129 02:59:38.961205 (+ 40677us) raft_consensus.cc:1471] Updating replica for 0 ops
0129 02:59:39.123822 (+162617us) raft_consensus.cc:1697] Filling consensus response to leader.
0129 02:59:39.154164 (+ 30342us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20260129 02:59:39.486912 22442 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.724s user 0.016s sys 0.035s
I20260129 02:59:39.803884 20584 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "36ff33be666a4c82954731691dbcf327" candidate_uuid: "974b802141bd4f13beffef2aa03f88f9" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1c50de89a5a845668696dcb28c3badd6" is_pre_election: true
I20260129 02:59:39.808794 20586 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4f8f3a38f6e24ba4857875181f447442" candidate_uuid: "974b802141bd4f13beffef2aa03f88f9" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "1c50de89a5a845668696dcb28c3badd6" is_pre_election: true
I20260129 02:59:39.799822 22344 raft_consensus.cc:493] T e33a417cf9c54cf5af73a4ee3aac4d3a P 1c50de89a5a845668696dcb28c3badd6 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260129 02:59:39.830364 21132 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "dc2f4cb752cf4d59b5f6bc689f5d8104" candidate_uuid: "1c50de89a5a845668696dcb28c3badd6" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "7a9daf5d42a84615a6383cfec9a10608" is_pre_election: true
I20260129 02:59:39.830786 20586 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a0fcc58d6fb6473cafbd9d34d5195269" candidate_uuid: "974b802141bd4f13beffef2aa03f88f9" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "1c50de89a5a845668696dcb28c3badd6"
I20260129 02:59:39.831450 20586 raft_consensus.cc:3060] T a0fcc58d6fb6473cafbd9d34d5195269 P 1c50de89a5a845668696dcb28c3badd6 [term 2 FOLLOWER]: Advancing to term 3
I20260129 02:59:39.832191 22353 consensus_queue.cc:1048] T 4e3805ffa70242e8ade66da260dddf66 P 1c50de89a5a845668696dcb28c3badd6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
I20260129 02:59:39.830420 22344 raft_consensus.cc:515] T e33a417cf9c54cf5af73a4ee3aac4d3a P 1c50de89a5a845668696dcb28c3badd6 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:59:39.809834 22437 leader_election.cc:290] T 17d6431cc77a491e8464d68db4081a51 P 974b802141bd4f13beffef2aa03f88f9 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 197b880fa39d432a9b472845876bfaf0 (127.18.211.1:42027), 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3:37735)
I20260129 02:59:39.764003 20439 consensus_queue.cc:260] T 913fc7d751ed4b7388d889aac30a5f9b P 197b880fa39d432a9b472845876bfaf0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:59:39.811187 20587 raft_consensus.cc:1275] T 4b1876f9a70b49edb3aa74105e66d3b3 P 1c50de89a5a845668696dcb28c3badd6 [term 4 FOLLOWER]: Refusing update from remote peer 974b802141bd4f13beffef2aa03f88f9: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
W20260129 02:59:39.811461 21102 leader_election.cc:336] T 00e79d8f25a34676b1d866eb2068a2b2 P 7a9daf5d42a84615a6383cfec9a10608 [CANDIDATE]: Term 4 election: RPC error from VoteRequest() call to peer 974b802141bd4f13beffef2aa03f88f9 (127.18.211.2:34517): Timed out: RequestConsensusVote RPC to 127.18.211.2:34517 timed out after 1.888s (SENT)
W20260129 02:59:39.812594 22440 consensus_meta.cc:287] T 78aa39096868415d9afc2ffcb248e198 P 197b880fa39d432a9b472845876bfaf0: Time spent flushing consensus metadata: real 0.858s user 0.010s sys 0.000s
I20260129 02:59:39.815212 20589 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "724a236524cb4e7ba3428880681d6cc2" candidate_uuid: "974b802141bd4f13beffef2aa03f88f9" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1c50de89a5a845668696dcb28c3badd6" is_pre_election: true
I20260129 02:59:39.848779 20587 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b189f08e16a74a11980cc55d99359a0a" candidate_uuid: "974b802141bd4f13beffef2aa03f88f9" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "1c50de89a5a845668696dcb28c3badd6"
W20260129 02:59:39.839087 20556 connection.cc:232] RPC call timeout handler was delayed by -2.2640846719999672s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.15080000000000002s for a response
W20260129 02:59:39.816840 22184 consensus_meta.cc:287] T 96609ac68bdf4993a12e1c23355ca013 P 974b802141bd4f13beffef2aa03f88f9: Time spent flushing consensus metadata: real 0.511s user 0.010s sys 0.000s
I20260129 02:59:39.804744 22292 raft_consensus.cc:493] T e7d83d4ccf684e76acb29a0dc844e1b8 P 1c50de89a5a845668696dcb28c3badd6 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260129 02:59:39.821587 20588 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "368953c9102643b2a21f99af36b51905" candidate_uuid: "974b802141bd4f13beffef2aa03f88f9" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "1c50de89a5a845668696dcb28c3badd6" is_pre_election: true
I20260129 02:59:39.822722 20590 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7d9494f5584d4d5db8b9adf1c8fc888b" candidate_uuid: "974b802141bd4f13beffef2aa03f88f9" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "1c50de89a5a845668696dcb28c3badd6" is_pre_election: true
I20260129 02:59:39.827147 20584 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "8055232cf0004d40a85f3bf983fe00a9" candidate_uuid: "974b802141bd4f13beffef2aa03f88f9" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1c50de89a5a845668696dcb28c3badd6" is_pre_election: true
I20260129 02:59:39.838049 20440 raft_consensus.cc:1240] T cedce331e1584ac8b636c9b8a72b2ba0 P 197b880fa39d432a9b472845876bfaf0 [term 5 FOLLOWER]: Rejecting Update request from peer 1c50de89a5a845668696dcb28c3badd6 for earlier term 4. Current term is 5. Ops: []
I20260129 02:59:39.799940 22499 raft_consensus.cc:493] T 3ccac82a01a2445fa5b0ce6c0fcf2295 P 974b802141bd4f13beffef2aa03f88f9 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260129 02:59:39.844471 20586 raft_consensus.cc:2468] T a0fcc58d6fb6473cafbd9d34d5195269 P 1c50de89a5a845668696dcb28c3badd6 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 974b802141bd4f13beffef2aa03f88f9 in term 3.
I20260129 02:59:39.852348 20439 raft_consensus.cc:3060] T 913fc7d751ed4b7388d889aac30a5f9b P 197b880fa39d432a9b472845876bfaf0 [term 1 FOLLOWER]: Advancing to term 2
I20260129 02:59:39.795725 22418 raft_consensus.cc:697] T a644c02b16af47679824ccd060ac10fd P 1c50de89a5a845668696dcb28c3badd6 [term 3 LEADER]: Becoming Leader. State: Replica: 1c50de89a5a845668696dcb28c3badd6, State: Running, Role: LEADER
W20260129 02:59:39.853045 21101 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260129 02:59:39.827785 22442 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 1.072s user 0.017s sys 0.036s
I20260129 02:59:39.737702 22320 raft_consensus.cc:515] T b189f08e16a74a11980cc55d99359a0a P 1c50de89a5a845668696dcb28c3badd6 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
W20260129 02:59:39.855646 20582 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 1.090s user 0.032s sys 0.100s
I20260129 02:59:39.859916 20516 raft_consensus.cc:2410] T 96609ac68bdf4993a12e1c23355ca013 P 974b802141bd4f13beffef2aa03f88f9 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1c50de89a5a845668696dcb28c3badd6 for term 5 because replica has last-logged OpId of term: 3 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 3.
I20260129 02:59:39.856033 20584 raft_consensus.cc:2468] T 8055232cf0004d40a85f3bf983fe00a9 P 1c50de89a5a845668696dcb28c3badd6 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 974b802141bd4f13beffef2aa03f88f9 in term 2.
I20260129 02:59:39.861613 20588 raft_consensus.cc:1240] T cb0b3f259ebc4310927338a8740e87b8 P 1c50de89a5a845668696dcb28c3badd6 [term 5 FOLLOWER]: Rejecting Update request from peer 974b802141bd4f13beffef2aa03f88f9 for earlier term 3. Current term is 5. Ops: []
W20260129 02:59:39.868614 20582 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 1.110s user 0.032s sys 0.100s
I20260129 02:59:39.869515 20582 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:48526 (request call id 176) took 2067 ms. Trace:
W20260129 02:59:39.862950 20556 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260129 02:59:39.861662 20589 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7a9a69cd82724b1faed1d10aa091fc4a" candidate_uuid: "974b802141bd4f13beffef2aa03f88f9" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "1c50de89a5a845668696dcb28c3badd6" is_pre_election: true
I20260129 02:59:39.848707 22440 raft_consensus.cc:515] T 78aa39096868415d9afc2ffcb248e198 P 197b880fa39d432a9b472845876bfaf0 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } }
W20260129 02:59:39.852054 22366 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 1.088s user 0.028s sys 0.102s
I20260129 02:59:39.869906 20582 rpcz_store.cc:276] 0129 02:59:37.802488 (+ 0us) service_pool.cc:168] Inserting onto call queue
0129 02:59:37.803305 (+ 817us) service_pool.cc:225] Handling call
0129 02:59:37.803693 (+ 388us) raft_consensus.cc:1471] Updating replica for 0 ops
0129 02:59:37.803952 (+ 259us) raft_consensus.cc:1534] Early marking committed up to index 5
0129 02:59:38.364206 (+560254us) log.cc:844] Serialized 14 byte log entry
0129 02:59:39.869132 (+1504926us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0129 02:59:39.869189 (+ 57us) raft_consensus.cc:1645] Marking committed up to 5
0129 02:59:39.869241 (+ 52us) raft_consensus.cc:1697] Filling consensus response to leader.
0129 02:59:39.869318 (+ 77us) raft_consensus.cc:1691] UpdateReplicas() finished
0129 02:59:39.869452 (+ 134us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":26699,"prepare.queue_time_us":669265,"prepare.run_cpu_time_us":60,"prepare.run_wall_time_us":5858,"raft.queue_time_us":116676,"raft.run_cpu_time_us":96,"raft.run_wall_time_us":95,"thread_start_us":2006711,"threads_started":3,"wal-append.queue_time_us":498392}
W20260129 02:59:39.879673 20407 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.789s user 0.026s sys 0.011s
W20260129 02:59:39.886300 20407 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.796s user 0.026s sys 0.011s
W20260129 02:59:39.887341 22366 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 1.121s user 0.029s sys 0.102s
I20260129 02:59:39.894044 22320 leader_election.cc:290] T b189f08e16a74a11980cc55d99359a0a P 1c50de89a5a845668696dcb28c3badd6 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 197b880fa39d432a9b472845876bfaf0 (127.18.211.1:42027), 974b802141bd4f13beffef2aa03f88f9 (127.18.211.2:34517)
I20260129 02:59:39.894666 22320 raft_consensus.cc:493] T a8479d1fe0bd41e5946186e3a6ca8c7a P 1c50de89a5a845668696dcb28c3badd6 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260129 02:59:39.860769 22292 raft_consensus.cc:515] T e7d83d4ccf684e76acb29a0dc844e1b8 P 1c50de89a5a845668696dcb28c3badd6 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:59:39.895110 22320 raft_consensus.cc:515] T a8479d1fe0bd41e5946186e3a6ca8c7a P 1c50de89a5a845668696dcb28c3badd6 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:59:39.896237 20587 raft_consensus.cc:3060] T b189f08e16a74a11980cc55d99359a0a P 1c50de89a5a845668696dcb28c3badd6 [term 4 FOLLOWER]: Advancing to term 5
I20260129 02:59:39.896324 20584 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "dc2f4cb752cf4d59b5f6bc689f5d8104" candidate_uuid: "197b880fa39d432a9b472845876bfaf0" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "1c50de89a5a845668696dcb28c3badd6" is_pre_election: true
W20260129 02:59:39.887668 20407 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.211.2:34517, user_credentials={real_user=slave}} blocked reactor thread for 835349us
I20260129 02:59:39.897596 20584 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a1a39dcd328e4268ae41b988c66b4a75" candidate_uuid: "197b880fa39d432a9b472845876bfaf0" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "1c50de89a5a845668696dcb28c3badd6" is_pre_election: true
W20260129 02:59:39.870757 22336 consensus_meta.cc:287] T fd8311b779af4e1fa1402b2a02a602f2 P 197b880fa39d432a9b472845876bfaf0: Time spent flushing consensus metadata: real 0.717s user 0.009s sys 0.002s
I20260129 02:59:39.906723 20589 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4b1876f9a70b49edb3aa74105e66d3b3" candidate_uuid: "197b880fa39d432a9b472845876bfaf0" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "1c50de89a5a845668696dcb28c3badd6" is_pre_election: true
I20260129 02:59:39.907618 20589 raft_consensus.cc:2468] T 4b1876f9a70b49edb3aa74105e66d3b3 P 1c50de89a5a845668696dcb28c3badd6 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 197b880fa39d432a9b472845876bfaf0 in term 4.
W20260129 02:59:39.909392 20514 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:48544 (request call id 148) took 1713 ms (client timeout 1818 ms). Trace:
W20260129 02:59:39.909849 20514 rpcz_store.cc:269] 0129 02:59:38.195935 (+ 0us) service_pool.cc:168] Inserting onto call queue
0129 02:59:38.431049 (+235114us) service_pool.cc:225] Handling call
0129 02:59:39.909170 (+1478121us) spinlock_profiling.cc:243] Waited 606 ms on lock 0x7b5400167454. stack: 00007f8fe044e373 00007f8fe044e270 00007f8fdfda0574 00007f8fe9a5c6b4 00007f8fe9a5c649 00007f8fe34975a1 00007f8fe3531f7a 00007f8fe84e3647 00007f8fe1a73834 00007f8fe1a737c2 00007f8fe1a736f1 00007f8fe1a7366c 00007f8fe1a72922 00007f8fe186d34c 00007f8fe186c786 00007f8fe186c12f
0129 02:59:39.909303 (+ 133us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1393898752}
I20260129 02:59:39.911721 20513 raft_consensus.cc:1275] T 75f45538dc704d9a8ec4ff8ad9f3ea5b P 974b802141bd4f13beffef2aa03f88f9 [term 2 FOLLOWER]: Refusing update from remote peer 7a9daf5d42a84615a6383cfec9a10608: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260129 02:59:39.913148 20514 raft_consensus.cc:1275] T 8055232cf0004d40a85f3bf983fe00a9 P 974b802141bd4f13beffef2aa03f88f9 [term 2 FOLLOWER]: Refusing update from remote peer 197b880fa39d432a9b472845876bfaf0: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260129 02:59:39.909307 22486 leader_election.cc:290] T a609ac02fae84725ab743f95f51d33ce P 7a9daf5d42a84615a6383cfec9a10608 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 197b880fa39d432a9b472845876bfaf0 (127.18.211.1:42027), 974b802141bd4f13beffef2aa03f88f9 (127.18.211.2:34517)
I20260129 02:59:39.754724 22500 raft_consensus.cc:515] T 4f8f3a38f6e24ba4857875181f447442 P 197b880fa39d432a9b472845876bfaf0 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:59:39.897071 22499 raft_consensus.cc:515] T 3ccac82a01a2445fa5b0ce6c0fcf2295 P 974b802141bd4f13beffef2aa03f88f9 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } }
I20260129 02:59:39.909413 22237 consensus_queue.cc:1059] T c934d31efd3f4e5195605ddc354d14e1 P 197b880fa39d432a9b472845876bfaf0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.302s
I20260129 02:59:40.050248 21133 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "75f45538dc704d9a8ec4ff8ad9f3ea5b" candidate_uuid: "1c50de89a5a845668696dcb28c3badd6" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "7a9daf5d42a84615a6383cfec9a10608" is_pre_election: true
W20260129 02:59:40.156989 20507 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 1.263s user 0.023s sys 0.024s
I20260129 02:59:40.032403 20590 raft_consensus.cc:1240] T 913fc7d751ed4b7388d889aac30a5f9b P 1c50de89a5a845668696dcb28c3badd6 [term 2 LEADER]: Rejecting Update request from peer 197b880fa39d432a9b472845876bfaf0 for earlier term 1. Current term is 2. Ops: []
W20260129 02:59:40.212957 20507 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 1.319s user 0.026s sys 0.024s
I20260129 02:59:39.915396 20586 raft_consensus.cc:1240] T 0340229d521b432580ae926f6062e3f7 P 1c50de89a5a845668696dcb28c3badd6 [term 4 FOLLOWER]: Rejecting Update request from peer 197b880fa39d432a9b472845876bfaf0 for earlier term 3. Current term is 4. Ops: []
W20260129 02:59:40.083998 21102 leader_election.cc:336] T 96609ac68bdf4993a12e1c23355ca013 P 7a9daf5d42a84615a6383cfec9a10608 [CANDIDATE]: Term 4 election: RPC error from VoteRequest() call to peer 974b802141bd4f13beffef2aa03f88f9 (127.18.211.2:34517): Timed out: RequestConsensusVote RPC to 127.18.211.2:34517 timed out after 2.727s (SENT)
I20260129 02:59:40.214313 21102 leader_election.cc:304] T 96609ac68bdf4993a12e1c23355ca013 P 7a9daf5d42a84615a6383cfec9a10608 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 1c50de89a5a845668696dcb28c3badd6, 7a9daf5d42a84615a6383cfec9a10608; no voters: 197b880fa39d432a9b472845876bfaf0, 974b802141bd4f13beffef2aa03f88f9
I20260129 02:59:40.080401 20582 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7d9494f5584d4d5db8b9adf1c8fc888b" candidate_uuid: "197b880fa39d432a9b472845876bfaf0" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "1c50de89a5a845668696dcb28c3badd6" is_pre_election: true
I20260129 02:59:39.999981 20589 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e33a417cf9c54cf5af73a4ee3aac4d3a" candidate_uuid: "197b880fa39d432a9b472845876bfaf0" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "1c50de89a5a845668696dcb28c3badd6"
W20260129 02:59:39.999549 20482 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260129 02:59:40.217226 20482 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260129 02:59:40.217612 20583 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "18f30ebaa4174fd28c958e72d0cc7e24" candidate_uuid: "197b880fa39d432a9b472845876bfaf0" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "1c50de89a5a845668696dcb28c3badd6" is_pre_election: true
I20260129 02:59:39.916170 20584 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a8479d1fe0bd41e5946186e3a6ca8c7a" candidate_uuid: "197b880fa39d432a9b472845876bfaf0" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "1c50de89a5a845668696dcb28c3badd6"
I20260129 02:59:39.924999 20514 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57600 (request call id 1697) took 1173 ms. Trace:
I20260129 02:59:40.220068 20514 rpcz_store.cc:276] 0129 02:59:38.741099 (+ 0us) service_pool.cc:168] Inserting onto call queue
0129 02:59:39.910768 (+1169669us) service_pool.cc:225] Handling call
0129 02:59:39.912727 (+ 1959us) raft_consensus.cc:1471] Updating replica for 0 ops
0129 02:59:39.913936 (+ 1209us) raft_consensus.cc:1697] Filling consensus response to leader.
0129 02:59:39.914183 (+ 247us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20260129 02:59:40.058669 20405 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 1.184s user 0.026s sys 0.040s
W20260129 02:59:40.222370 20405 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 1.348s user 0.030s sys 0.045s
W20260129 02:59:40.222852 20405 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.211.4:40439, user_credentials={real_user=slave}} blocked reactor thread for 1.34805e+06us
W20260129 02:59:39.938422 21101 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260129 02:59:40.233165 20589 raft_consensus.cc:2393] T e33a417cf9c54cf5af73a4ee3aac4d3a P 1c50de89a5a845668696dcb28c3badd6 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 197b880fa39d432a9b472845876bfaf0 in current term 4: Already voted for candidate 974b802141bd4f13beffef2aa03f88f9 in this term.
I20260129 02:59:40.241870 20583 raft_consensus.cc:2393] T 18f30ebaa4174fd28c958e72d0cc7e24 P 1c50de89a5a845668696dcb28c3badd6 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 197b880fa39d432a9b472845876bfaf0 in current term 3: Already voted for candidate 1c50de89a5a845668696dcb28c3badd6 in this term.
I20260129 02:59:40.242376 21132 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a609ac02fae84725ab743f95f51d33ce" candidate_uuid: "197b880fa39d432a9b472845876bfaf0" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "7a9daf5d42a84615a6383cfec9a10608" is_pre_election: true
I20260129 02:59:40.243084 22376 raft_consensus.cc:2749] T 96609ac68bdf4993a12e1c23355ca013 P 7a9daf5d42a84615a6383cfec9a10608 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20260129 02:59:40.244338 20590 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4e3805ffa70242e8ade66da260dddf66" candidate_uuid: "197b880fa39d432a9b472845876bfaf0" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "1c50de89a5a845668696dcb28c3badd6" is_pre_election: true
I20260129 02:59:40.243026 20583 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a4c1226b987b468fa84dfc6ba0e2dbe1" candidate_uuid: "197b880fa39d432a9b472845876bfaf0" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1c50de89a5a845668696dcb28c3badd6" is_pre_election: true
I20260129 02:59:40.249399 20582 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4408006eb5174de1896b037cf3f18457" candidate_uuid: "197b880fa39d432a9b472845876bfaf0" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1c50de89a5a845668696dcb28c3badd6"
I20260129 02:59:40.250789 20582 raft_consensus.cc:3060] T 4408006eb5174de1896b037cf3f18457 P 1c50de89a5a845668696dcb28c3badd6 [term 1 FOLLOWER]: Advancing to term 2
I20260129 02:59:40.219080 20584 raft_consensus.cc:2376] T a8479d1fe0bd41e5946186e3a6ca8c7a P 1c50de89a5a845668696dcb28c3badd6 [term 3 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 197b880fa39d432a9b472845876bfaf0 in term 3. Re-sending same reply.
I20260129 02:59:40.279939 20589 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c891d2bff4ce4568861bbc992e91fbb0" candidate_uuid: "197b880fa39d432a9b472845876bfaf0" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "1c50de89a5a845668696dcb28c3badd6" is_pre_election: true
I20260129 02:59:40.285507 20514 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57600 (request call id 1698) took 1541 ms. Trace:
I20260129 02:59:40.286082 20514 rpcz_store.cc:276] 0129 02:59:38.769038 (+ 0us) service_pool.cc:168] Inserting onto call queue
0129 02:59:40.221131 (+1452093us) service_pool.cc:225] Handling call
0129 02:59:40.221525 (+ 394us) raft_consensus.cc:1471] Updating replica for 0 ops
0129 02:59:40.285085 (+ 63560us) raft_consensus.cc:1534] Early marking committed up to index 0
0129 02:59:40.285116 (+ 31us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0129 02:59:40.285167 (+ 51us) raft_consensus.cc:1645] Marking committed up to 4
0129 02:59:40.285204 (+ 37us) raft_consensus.cc:1697] Filling consensus response to leader.
0129 02:59:40.285272 (+ 68us) raft_consensus.cc:1691] UpdateReplicas() finished
0129 02:59:40.285444 (+ 172us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":23407}
I20260129 02:59:40.302529 20590 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e4d18c35b0a7493e85d0a4b92956a48f" candidate_uuid: "197b880fa39d432a9b472845876bfaf0" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "1c50de89a5a845668696dcb28c3badd6" is_pre_election: true
I20260129 02:59:40.303287 20590 raft_consensus.cc:2468] T e4d18c35b0a7493e85d0a4b92956a48f P 1c50de89a5a845668696dcb28c3badd6 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 197b880fa39d432a9b472845876bfaf0 in term 3.
W20260129 02:59:40.193346 22432 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 1.040s user 0.021s sys 0.044s
W20260129 02:59:39.915815 22425 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.996s user 0.019s sys 0.019s
W20260129 02:59:40.227692 22427 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 1.267s user 0.022s sys 0.045s
W20260129 02:59:40.329468 21102 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.211.2:34517, user_credentials={real_user=slave}} blocked reactor thread for 245511us
I20260129 02:59:40.329864 20586 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "78aa39096868415d9afc2ffcb248e198" candidate_uuid: "197b880fa39d432a9b472845876bfaf0" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "1c50de89a5a845668696dcb28c3badd6" is_pre_election: true
I20260129 02:59:40.037273 20509 raft_consensus.cc:1240] T 4e420f90884f4796997b5ea41e49e411 P 974b802141bd4f13beffef2aa03f88f9 [term 4 FOLLOWER]: Rejecting Update request from peer 7a9daf5d42a84615a6383cfec9a10608 for earlier term 3. Current term is 4. Ops: []
I20260129 02:59:40.160254 22326 leader_election.cc:290] T 368953c9102643b2a21f99af36b51905 P 197b880fa39d432a9b472845876bfaf0 [CANDIDATE]: Term 5 election: Requested vote from peers 974b802141bd4f13beffef2aa03f88f9 (127.18.211.2:34517), 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3:37735)
I20260129 02:59:40.059775 22354 raft_consensus.cc:493] T fd8311b779af4e1fa1402b2a02a602f2 P 197b880fa39d432a9b472845876bfaf0 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260129 02:59:40.305115 20584 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2e58e9a5a8aa4cecafb464addb4f35ed" candidate_uuid: "197b880fa39d432a9b472845876bfaf0" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "1c50de89a5a845668696dcb28c3badd6" is_pre_election: true
I20260129 02:59:40.058610 22366 raft_consensus.cc:493] T 4e420f90884f4796997b5ea41e49e411 P 7a9daf5d42a84615a6383cfec9a10608 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 197b880fa39d432a9b472845876bfaf0)
W20260129 02:59:40.319813 21101 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260129 02:59:40.357627 20405 connection.cc:232] RPC call timeout handler was delayed by -1.373136350999971s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.21070000000000003s for a response
I20260129 02:59:40.358470 20589 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a5a05fbcbbcf4916bbc01fd1a5b5e90d" candidate_uuid: "197b880fa39d432a9b472845876bfaf0" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "1c50de89a5a845668696dcb28c3badd6" is_pre_election: true
I20260129 02:59:40.359380 20589 raft_consensus.cc:2468] T a5a05fbcbbcf4916bbc01fd1a5b5e90d P 1c50de89a5a845668696dcb28c3badd6 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 197b880fa39d432a9b472845876bfaf0 in term 5.
W20260129 02:59:40.345657 20480 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 1.462s user 0.023s sys 0.083s
W20260129 02:59:40.347258 20407 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.211.2:34517, user_credentials={real_user=slave}} blocked reactor thread for 344035us
I20260129 02:59:40.348423 20584 raft_consensus.cc:2468] T 2e58e9a5a8aa4cecafb464addb4f35ed P 1c50de89a5a845668696dcb28c3badd6 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 197b880fa39d432a9b472845876bfaf0 in term 2.
I20260129 02:59:40.342144 20554 leader_election.cc:304] T 75f45538dc704d9a8ec4ff8ad9f3ea5b P 1c50de89a5a845668696dcb28c3badd6 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1c50de89a5a845668696dcb28c3badd6; no voters: 7a9daf5d42a84615a6383cfec9a10608, 974b802141bd4f13beffef2aa03f88f9
W20260129 02:59:40.304558 22443 log.cc:927] Time spent T a4c1226b987b468fa84dfc6ba0e2dbe1 P 974b802141bd4f13beffef2aa03f88f9: Append to log took a long time: real 2.175s user 0.004s sys 0.019s
I20260129 02:59:40.351753 20583 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1ce4ddf5b10e4ae8b193b13b2e5936ee" candidate_uuid: "197b880fa39d432a9b472845876bfaf0" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "1c50de89a5a845668696dcb28c3badd6" is_pre_election: true
W20260129 02:59:40.306928 22432 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 1.155s user 0.022s sys 0.045s
W20260129 02:59:40.362635 20439 consensus_meta.cc:287] T 913fc7d751ed4b7388d889aac30a5f9b P 197b880fa39d432a9b472845876bfaf0: Time spent flushing consensus metadata: real 0.500s user 0.014s sys 0.026s
W20260129 02:59:40.365761 20480 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 1.482s user 0.023s sys 0.083s
I20260129 02:59:40.363375 20439 raft_consensus.cc:1275] T 913fc7d751ed4b7388d889aac30a5f9b P 197b880fa39d432a9b472845876bfaf0 [term 2 FOLLOWER]: Refusing update from remote peer 1c50de89a5a845668696dcb28c3badd6: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
W20260129 02:59:40.366475 20480 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.211.3:37735, user_credentials={real_user=slave}} blocked reactor thread for 1.4941e+06us
I20260129 02:59:40.368993 20439 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a5a05fbcbbcf4916bbc01fd1a5b5e90d" candidate_uuid: "1c50de89a5a845668696dcb28c3badd6" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "197b880fa39d432a9b472845876bfaf0" is_pre_election: true
W20260129 02:59:40.374294 20554 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260129 02:59:40.374349 20584 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "cedce331e1584ac8b636c9b8a72b2ba0" candidate_uuid: "197b880fa39d432a9b472845876bfaf0" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "1c50de89a5a845668696dcb28c3badd6" is_pre_election: true
W20260129 02:59:40.329082 22425 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 1.409s user 0.024s sys 0.036s
W20260129 02:59:40.329430 22427 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 1.370s user 0.024s sys 0.046s
I20260129 02:59:40.378155 20589 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "368953c9102643b2a21f99af36b51905" candidate_uuid: "197b880fa39d432a9b472845876bfaf0" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "1c50de89a5a845668696dcb28c3badd6"
I20260129 02:59:40.379128 20590 raft_consensus.cc:1275] T 8055232cf0004d40a85f3bf983fe00a9 P 1c50de89a5a845668696dcb28c3badd6 [term 2 FOLLOWER]: Refusing update from remote peer 197b880fa39d432a9b472845876bfaf0: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260129 02:59:40.380084 20583 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3906b080a5b34a74b8b64128635e8997" candidate_uuid: "197b880fa39d432a9b472845876bfaf0" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "1c50de89a5a845668696dcb28c3badd6" is_pre_election: true
I20260129 02:59:40.384624 20584 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "17d6431cc77a491e8464d68db4081a51" candidate_uuid: "197b880fa39d432a9b472845876bfaf0" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1c50de89a5a845668696dcb28c3badd6" is_pre_election: true
I20260129 02:59:40.385380 20584 raft_consensus.cc:2468] T 17d6431cc77a491e8464d68db4081a51 P 1c50de89a5a845668696dcb28c3badd6 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 197b880fa39d432a9b472845876bfaf0 in term 2.
I20260129 02:59:40.386396 20584 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0f20af7fbe5d4db89a6f17948d98912c" candidate_uuid: "197b880fa39d432a9b472845876bfaf0" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "1c50de89a5a845668696dcb28c3badd6" is_pre_election: true
I20260129 02:59:40.376572 20407 leader_election.cc:304] T a8479d1fe0bd41e5946186e3a6ca8c7a P 197b880fa39d432a9b472845876bfaf0 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 197b880fa39d432a9b472845876bfaf0, 974b802141bd4f13beffef2aa03f88f9; no voters:
I20260129 02:59:40.373629 20439 raft_consensus.cc:2410] T a5a05fbcbbcf4916bbc01fd1a5b5e90d P 197b880fa39d432a9b472845876bfaf0 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1c50de89a5a845668696dcb28c3badd6 for term 6 because replica has last-logged OpId of term: 5 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 4 index: 3.
I20260129 02:59:40.243134 21132 raft_consensus.cc:2468] T a609ac02fae84725ab743f95f51d33ce P 7a9daf5d42a84615a6383cfec9a10608 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 197b880fa39d432a9b472845876bfaf0 in term 5.
I20260129 02:59:40.049501 22502 consensus_queue.cc:1059] T a0fcc58d6fb6473cafbd9d34d5195269 P 197b880fa39d432a9b472845876bfaf0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 }, Status: INVALID_TERM, Last received: 2.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.446s
I20260129 02:59:40.399298 21136 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00e79d8f25a34676b1d866eb2068a2b2" candidate_uuid: "197b880fa39d432a9b472845876bfaf0" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "7a9daf5d42a84615a6383cfec9a10608"
I20260129 02:59:40.389974 20584 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c2633a6fc34c44579de348d208b0c01e" candidate_uuid: "197b880fa39d432a9b472845876bfaf0" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1c50de89a5a845668696dcb28c3badd6" is_pre_election: true
I20260129 02:59:40.390997 20586 raft_consensus.cc:1240] T a0fcc58d6fb6473cafbd9d34d5195269 P 1c50de89a5a845668696dcb28c3badd6 [term 3 FOLLOWER]: Rejecting Update request from peer 197b880fa39d432a9b472845876bfaf0 for earlier term 2. Current term is 3. Ops: []
I20260129 02:59:40.402314 20589 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a1a39dcd328e4268ae41b988c66b4a75" candidate_uuid: "197b880fa39d432a9b472845876bfaf0" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "1c50de89a5a845668696dcb28c3badd6"
I20260129 02:59:40.412349 22499 leader_election.cc:290] T 3ccac82a01a2445fa5b0ce6c0fcf2295 P 974b802141bd4f13beffef2aa03f88f9 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 197b880fa39d432a9b472845876bfaf0 (127.18.211.1:42027), 7a9daf5d42a84615a6383cfec9a10608 (127.18.211.4:40439)
I20260129 02:59:40.417527 20514 tablet_service.cc:2066] Received LeaderStepDown RPC: tablet_id: "4408006eb5174de1896b037cf3f18457"
dest_uuid: "974b802141bd4f13beffef2aa03f88f9"
mode: GRACEFUL
new_leader_uuid: "1c50de89a5a845668696dcb28c3badd6"
from {username='slave'} at 127.0.0.1:52008
I20260129 02:59:40.418190 20514 raft_consensus.cc:606] T 4408006eb5174de1896b037cf3f18457 P 974b802141bd4f13beffef2aa03f88f9 [term 1 LEADER]: Received request to transfer leadership to TS 1c50de89a5a845668696dcb28c3badd6
I20260129 02:59:40.421566 20514 tablet_service.cc:2066] Received LeaderStepDown RPC: tablet_id: "4f5a77e94fef4b16a2ba7b067c8b144a"
dest_uuid: "974b802141bd4f13beffef2aa03f88f9"
mode: GRACEFUL
new_leader_uuid: "1c50de89a5a845668696dcb28c3badd6"
from {username='slave'} at 127.0.0.1:52008
I20260129 02:59:40.422102 20514 raft_consensus.cc:606] T 4f5a77e94fef4b16a2ba7b067c8b144a P 974b802141bd4f13beffef2aa03f88f9 [term 2 FOLLOWER]: Received request to transfer leadership to TS 1c50de89a5a845668696dcb28c3badd6
I20260129 02:59:40.422432 20514 raft_consensus.cc:614] T 4f5a77e94fef4b16a2ba7b067c8b144a P 974b802141bd4f13beffef2aa03f88f9 [term 2 FOLLOWER]: Rejecting request to transer leadership while not leader
W20260129 02:59:40.444435 20480 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.211.3:37735, user_credentials={real_user=slave}} blocked reactor thread for 47033.1us
I20260129 02:59:40.457702 20590 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "cedce331e1584ac8b636c9b8a72b2ba0" candidate_uuid: "197b880fa39d432a9b472845876bfaf0" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "1c50de89a5a845668696dcb28c3badd6"
W20260129 02:59:40.458732 20407 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.211.2:34517, user_credentials={real_user=slave}} blocked reactor thread for 58449.5us
I20260129 02:59:40.356853 22224 leader_election.cc:290] T 8055232cf0004d40a85f3bf983fe00a9 P 1c50de89a5a845668696dcb28c3badd6 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 197b880fa39d432a9b472845876bfaf0 (127.18.211.1:42027), 974b802141bd4f13beffef2aa03f88f9 (127.18.211.2:34517)
I20260129 02:59:40.357877 22344 leader_election.cc:290] T e33a417cf9c54cf5af73a4ee3aac4d3a P 1c50de89a5a845668696dcb28c3badd6 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 197b880fa39d432a9b472845876bfaf0 (127.18.211.1:42027), 974b802141bd4f13beffef2aa03f88f9 (127.18.211.2:34517)
I20260129 02:59:40.465683 20433 raft_consensus.cc:1240] T 0678b8b468bd43cdbc3e485900844821 P 197b880fa39d432a9b472845876bfaf0 [term 5 LEADER]: Rejecting Update request from peer 7a9daf5d42a84615a6383cfec9a10608 for earlier term 4. Current term is 5. Ops: []
I20260129 02:59:40.469910 20480 leader_election.cc:304] T e33a417cf9c54cf5af73a4ee3aac4d3a P 974b802141bd4f13beffef2aa03f88f9 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1c50de89a5a845668696dcb28c3badd6, 974b802141bd4f13beffef2aa03f88f9; no voters: 197b880fa39d432a9b472845876bfaf0
I20260129 02:59:40.471658 20480 leader_election.cc:304] T 1ce4ddf5b10e4ae8b193b13b2e5936ee P 974b802141bd4f13beffef2aa03f88f9 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 974b802141bd4f13beffef2aa03f88f9; no voters: 197b880fa39d432a9b472845876bfaf0, 1c50de89a5a845668696dcb28c3badd6
W20260129 02:59:40.473110 20587 consensus_meta.cc:287] T b189f08e16a74a11980cc55d99359a0a P 1c50de89a5a845668696dcb28c3badd6: Time spent flushing consensus metadata: real 0.576s user 0.027s sys 0.010s
I20260129 02:59:40.373847 22507 raft_consensus.cc:2749] T 75f45538dc704d9a8ec4ff8ad9f3ea5b P 1c50de89a5a845668696dcb28c3badd6 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260129 02:59:40.477362 20587 raft_consensus.cc:2468] T b189f08e16a74a11980cc55d99359a0a P 1c50de89a5a845668696dcb28c3badd6 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 974b802141bd4f13beffef2aa03f88f9 in term 5.
I20260129 02:59:40.160467 22418 consensus_queue.cc:237] T a644c02b16af47679824ccd060ac10fd P 1c50de89a5a845668696dcb28c3badd6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } }
I20260129 02:59:40.493353 20582 raft_consensus.cc:2468] T 4408006eb5174de1896b037cf3f18457 P 1c50de89a5a845668696dcb28c3badd6 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 197b880fa39d432a9b472845876bfaf0 in term 2.
I20260129 02:59:40.502760 22502 raft_consensus.cc:3055] T a0fcc58d6fb6473cafbd9d34d5195269 P 197b880fa39d432a9b472845876bfaf0 [term 2 LEADER]: Stepping down as leader of term 2
I20260129 02:59:40.503264 22502 raft_consensus.cc:740] T a0fcc58d6fb6473cafbd9d34d5195269 P 197b880fa39d432a9b472845876bfaf0 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 197b880fa39d432a9b472845876bfaf0, State: Running, Role: LEADER
W20260129 02:59:40.504097 22512 log.cc:927] Time spent T 4b1876f9a70b49edb3aa74105e66d3b3 P 197b880fa39d432a9b472845876bfaf0: Append to log took a long time: real 0.363s user 0.001s sys 0.000s
I20260129 02:59:40.504906 20435 raft_consensus.cc:3060] T 7a9a69cd82724b1faed1d10aa091fc4a P 197b880fa39d432a9b472845876bfaf0 [term 3 FOLLOWER]: Advancing to term 4
I20260129 02:59:40.505501 20432 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55672 (request call id 1460) took 1972 ms. Trace:
I20260129 02:59:40.424229 20509 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:48544 (request call id 155) took 2096 ms. Trace:
I20260129 02:59:40.462582 22198 consensus_queue.cc:1059] T b189f08e16a74a11980cc55d99359a0a P 197b880fa39d432a9b472845876bfaf0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 }, Status: INVALID_TERM, Last received: 4.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.041s
I20260129 02:59:40.505869 20432 rpcz_store.cc:276] 0129 02:59:38.532770 (+ 0us) service_pool.cc:168] Inserting onto call queue
0129 02:59:38.753862 (+221092us) service_pool.cc:225] Handling call
0129 02:59:38.754218 (+ 356us) raft_consensus.cc:1471] Updating replica for 1 ops
0129 02:59:38.754578 (+ 360us) raft_consensus.cc:1534] Early marking committed up to index 0
0129 02:59:38.754603 (+ 25us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0129 02:59:38.755099 (+ 496us) log.cc:844] Serialized 26 byte log entry
0129 02:59:39.473464 (+718365us) raft_consensus.cc:1645] Marking committed up to 3
0129 02:59:39.473505 (+ 41us) raft_consensus.cc:1697] Filling consensus response to leader.
0129 02:59:39.474147 (+ 642us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0129 02:59:40.505240 (+1031093us) raft_consensus.cc:1685] finished
0129 02:59:40.505273 (+ 33us) raft_consensus.cc:1691] UpdateReplicas() finished
0129 02:59:40.505442 (+ 169us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":73360,"raft.run_cpu_time_us":151,"raft.run_wall_time_us":5012,"thread_start_us":645290,"threads_started":1,"wal-append.queue_time_us":1103577}
I20260129 02:59:40.515130 20432 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "926a250fa5834e88b96ccecbefbf5086" candidate_uuid: "1c50de89a5a845668696dcb28c3badd6" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "197b880fa39d432a9b472845876bfaf0" is_pre_election: true
I20260129 02:59:40.518006 20510 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57584 (request call id 1224) took 3568 ms. Trace:
I20260129 02:59:40.518394 20510 rpcz_store.cc:276] 0129 02:59:36.949828 (+ 0us) service_pool.cc:168] Inserting onto call queue
0129 02:59:36.951954 (+ 2126us) service_pool.cc:225] Handling call
0129 02:59:36.952266 (+ 312us) raft_consensus.cc:1471] Updating replica for 1 ops
0129 02:59:36.991058 (+ 38792us) raft_consensus.cc:1534] Early marking committed up to index 0
0129 02:59:36.991087 (+ 29us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0129 02:59:37.000457 (+ 9370us) log.cc:844] Serialized 26 byte log entry
0129 02:59:37.324158 (+323701us) raft_consensus.cc:1645] Marking committed up to 1
0129 02:59:37.324202 (+ 44us) raft_consensus.cc:1697] Filling consensus response to leader.
0129 02:59:37.324333 (+ 131us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0129 02:59:39.483392 (+2159059us) spinlock_profiling.cc:243] Waited 1.63 ms on lock 0x7b440002304c. stack: 00007f8fe044e373 00007f8fe044e270 00007f8fdfda0574 00007f8fe9a5c6b4 00007f8fe9a5c649 00007f8fe02c08b1 00007f8fe038f005 00007f8fe038fa22 00007f8fe30d957b 00007f8fe30d4e4f 00007f8fe30c8c0e 00007f8fe30c7f56 00007f8fe30c7b78 00007f8fe30d15c1 00007f8fe30d1579 00007f8fe30d1509
0129 02:59:40.517717 (+1034325us) raft_consensus.cc:1685] finished
0129 02:59:40.517742 (+ 25us) raft_consensus.cc:1691] UpdateReplicas() finished
0129 02:59:40.517944 (+ 202us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":38429,"raft.queue_time_us":197910,"raft.run_cpu_time_us":164,"raft.run_wall_time_us":165,"spinlock_wait_cycles":3755392,"thread_start_us":282669,"threads_started":1,"wal-append.queue_time_us":577318}
I20260129 02:59:40.508066 22036 raft_consensus.cc:2749] T 1ce4ddf5b10e4ae8b193b13b2e5936ee P 974b802141bd4f13beffef2aa03f88f9 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260129 02:59:40.522872 22292 leader_election.cc:290] T e7d83d4ccf684e76acb29a0dc844e1b8 P 1c50de89a5a845668696dcb28c3badd6 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 197b880fa39d432a9b472845876bfaf0 (127.18.211.1:42027), 974b802141bd4f13beffef2aa03f88f9 (127.18.211.2:34517)
W20260129 02:59:40.523631 20404 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260129 02:59:40.524199 20441 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55676 (request call id 1163) took 3067 ms (client timeout 1886 ms). Trace:
W20260129 02:59:40.564661 20441 rpcz_store.cc:269] 0129 02:59:37.456583 (+ 0us) service_pool.cc:168] Inserting onto call queue
0129 02:59:37.813463 (+356880us) service_pool.cc:225] Handling call
0129 02:59:40.523933 (+2710470us) spinlock_profiling.cc:243] Waited 248 ms on lock 0x7b54002fc454. stack: 00007f8fe044e373 00007f8fe044e270 00007f8fdfda0574 00007f8fe9a5c6b4 00007f8fe9a5c649 00007f8fe34975a1 00007f8fe3531f7a 00007f8fe84e3647 00007f8fe1a73834 00007f8fe1a737c2 00007f8fe1a736f1 00007f8fe1a7366c 00007f8fe1a72922 00007f8fe186d34c 00007f8fe186c786 00007f8fe186c12f
0129 02:59:40.524111 (+ 178us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":569971584}
I20260129 02:59:40.525125 20585 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35752 (request call id 1478) took 4460 ms. Trace:
I20260129 02:59:40.566022 20585 rpcz_store.cc:276] 0129 02:59:36.064650 (+ 0us) service_pool.cc:168] Inserting onto call queue
0129 02:59:36.125833 (+ 61183us) service_pool.cc:225] Handling call
0129 02:59:39.151770 (+3025937us) raft_consensus.cc:1471] Updating replica for 0 ops
0129 02:59:39.171923 (+ 20153us) raft_consensus.cc:1697] Filling consensus response to leader.
0129 02:59:40.524918 (+1352995us) spinlock_profiling.cc:243] Waited 3 s on lock 0x7b54003a0050. stack: 00007f8fe044e373 00007f8fe044e270 00007f8fdfda0574 00007f8fe9a5c6b4 00007f8fe9a5c649 00007f8fe34975a1 00007f8fe352d52d 00007f8fe84e1f49 00007f8fe1a6ed34 00007f8fe1a6ecc2 00007f8fe1a6ebf1 00007f8fe1a6eb6c 00007f8fe1a6de22 00007f8fe186d34c 00007f8fe186c786 00007f8fe186c12f
0129 02:59:40.525048 (+ 130us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":6891082112}
W20260129 02:59:40.526422 22455 log.cc:927] Time spent T c2633a6fc34c44579de348d208b0c01e P 974b802141bd4f13beffef2aa03f88f9: Append to log took a long time: real 2.133s user 0.014s sys 0.000s
W20260129 02:59:40.568046 20404 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260129 02:59:40.531221 20516 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:57584 (request call id 1227) took 3558 ms. Trace:
W20260129 02:59:40.532442 22403 consensus_meta.cc:287] T 78aa39096868415d9afc2ffcb248e198 P 7a9daf5d42a84615a6383cfec9a10608: Time spent flushing consensus metadata: real 1.597s user 0.011s sys 0.001s
I20260129 02:59:40.534030 20513 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:48544 (request call id 144) took 2620 ms. Trace:
I20260129 02:59:40.535116 20435 raft_consensus.cc:1275] T 7a9a69cd82724b1faed1d10aa091fc4a P 197b880fa39d432a9b472845876bfaf0 [term 4 FOLLOWER]: Refusing update from remote peer 1c50de89a5a845668696dcb28c3badd6: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20260129 02:59:40.568785 20513 rpcz_store.cc:276] 0129 02:59:37.948370 (+ 0us) service_pool.cc:168] Inserting onto call queue
0129 02:59:39.408584 (+1460214us) service_pool.cc:225] Handling call
0129 02:59:39.911270 (+502686us) raft_consensus.cc:1471] Updating replica for 0 ops
0129 02:59:39.912323 (+ 1053us) raft_consensus.cc:1697] Filling consensus response to leader.
0129 02:59:40.533834 (+621511us) spinlock_profiling.cc:243] Waited 408 ms on lock 0x7b5400167450. stack: 00007f8fe044e373 00007f8fe044e270 00007f8fdfda0574 00007f8fe9a5c6b4 00007f8fe9a5c649 00007f8fe34975a1 00007f8fe352d52d 00007f8fe84e1f49 00007f8fe1a6ed34 00007f8fe1a6ecc2 00007f8fe1a6ebf1 00007f8fe1a6eb6c 00007f8fe1a6de22 00007f8fe186d34c 00007f8fe186c786 00007f8fe186c12f
0129 02:59:40.533973 (+ 139us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":938272000}
I20260129 02:59:40.569203 20435 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55676 (request call id 1150) took 4280 ms. Trace:
I20260129 02:59:40.569518 20435 rpcz_store.cc:276] 0129 02:59:36.289359 (+ 0us) service_pool.cc:168] Inserting onto call queue
0129 02:59:36.458586 (+169227us) service_pool.cc:225] Handling call
0129 02:59:36.474339 (+ 15753us) raft_consensus.cc:1471] Updating replica for 0 ops
0129 02:59:37.395000 (+920661us) spinlock_profiling.cc:243] Waited 12.6 ms on lock 0x7b54002f88a0. stack: 00007f8fe044e373 00007f8fe044e270 00007f8fdfda0574 00007f8fe9a5c6b4 00007f8fe9a5c649 00007f8fe34975a1 00007f8fe34a3a4d 00007f8fe348a1d4 00007f8fe350c177 00007f8fe3524c67 00007f8fe352645e 00007f8fe35307c9 00007f8fe3531096 00007f8fe352da0f 00007f8fe352d4be 00007f8fe84e1f49
0129 02:59:40.504511 (+3109511us) spinlock_profiling.cc:243] Waited 1.71 s on lock 0x7b58001aabd0. stack: 00007f8fe044e373 00007f8fe044e270 00007f8fdfda0574 00007f8fe9a5c6b4 00007f8fe9a5c649 00007f8fe34975a1 00007f8fe348a0da 00007f8fe350c177 00007f8fe3524c67 00007f8fe352645e 00007f8fe35307c9 00007f8fe3531096 00007f8fe352da0f 00007f8fe352d4be 00007f8fe84e1f49 00007f8fe1a6ed34
0129 02:59:40.568922 (+ 64411us) raft_consensus.cc:1697] Filling consensus response to leader.
0129 02:59:40.569153 (+ 231us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":3963018368}
I20260129 02:59:40.570896 20435 raft_consensus.cc:1240] T 78aa39096868415d9afc2ffcb248e198 P 197b880fa39d432a9b472845876bfaf0 [term 4 FOLLOWER]: Rejecting Update request from peer 1c50de89a5a845668696dcb28c3badd6 for earlier term 3. Current term is 4. Ops: []
W20260129 02:59:40.572026 22324 raft_consensus.cc:936] Couldn't advance consensus term.: Illegal state: Can't advance term to: 4 current term: 4 is higher.
I20260129 02:59:40.573283 22385 raft_consensus.cc:493] T 7a9a69cd82724b1faed1d10aa091fc4a P 197b880fa39d432a9b472845876bfaf0 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260129 02:59:40.573854 22385 raft_consensus.cc:515] T 7a9a69cd82724b1faed1d10aa091fc4a P 197b880fa39d432a9b472845876bfaf0 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:59:40.568549 20516 rpcz_store.cc:276] 0129 02:59:36.972828 (+ 0us) service_pool.cc:168] Inserting onto call queue
0129 02:59:37.003113 (+ 30285us) service_pool.cc:225] Handling call
0129 02:59:40.531016 (+3527903us) spinlock_profiling.cc:243] Waited 2.84 s on lock 0x7b54000fcb54. stack: 00007f8fe044e373 00007f8fe044e270 00007f8fdfda0574 00007f8fe9a5c6b4 00007f8fe9a5c649 00007f8fe34975a1 00007f8fe3531f7a 00007f8fe84e3647 00007f8fe1a73834 00007f8fe1a737c2 00007f8fe1a736f1 00007f8fe1a7366c 00007f8fe1a72922 00007f8fe186d34c 00007f8fe186c786 00007f8fe186c12f
0129 02:59:40.531152 (+ 136us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":6536438656}
W20260129 02:59:40.538532 21100 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260129 02:59:40.374740 22354 raft_consensus.cc:515] T fd8311b779af4e1fa1402b2a02a602f2 P 197b880fa39d432a9b472845876bfaf0 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } }
I20260129 02:59:40.466775 22432 raft_consensus.cc:2804] T a8479d1fe0bd41e5946186e3a6ca8c7a P 197b880fa39d432a9b472845876bfaf0 [term 3 FOLLOWER]: Leader election won for term 3
I20260129 02:59:40.375924 22366 raft_consensus.cc:515] T 4e420f90884f4796997b5ea41e49e411 P 7a9daf5d42a84615a6383cfec9a10608 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } }
I20260129 02:59:40.482277 22157 raft_consensus.cc:2804] T e33a417cf9c54cf5af73a4ee3aac4d3a P 974b802141bd4f13beffef2aa03f88f9 [term 4 FOLLOWER]: Leader election won for term 4
I20260129 02:59:40.529680 22500 leader_election.cc:290] T 4f8f3a38f6e24ba4857875181f447442 P 197b880fa39d432a9b472845876bfaf0 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 974b802141bd4f13beffef2aa03f88f9 (127.18.211.2:34517), 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3:37735)
I20260129 02:59:40.573345 22391 raft_consensus.cc:493] T 2e58e9a5a8aa4cecafb464addb4f35ed P 7a9daf5d42a84615a6383cfec9a10608 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260129 02:59:40.475647 22440 leader_election.cc:290] T 78aa39096868415d9afc2ffcb248e198 P 197b880fa39d432a9b472845876bfaf0 [CANDIDATE]: Term 4 election: Requested vote from peers 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3:37735), 7a9daf5d42a84615a6383cfec9a10608 (127.18.211.4:40439)
I20260129 02:59:40.560345 20432 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "dc2f4cb752cf4d59b5f6bc689f5d8104" candidate_uuid: "1c50de89a5a845668696dcb28c3badd6" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "197b880fa39d432a9b472845876bfaf0" is_pre_election: true
I20260129 02:59:40.578857 20441 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4408006eb5174de1896b037cf3f18457" candidate_uuid: "1c50de89a5a845668696dcb28c3badd6" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "197b880fa39d432a9b472845876bfaf0" is_pre_election: true
I20260129 02:59:40.590000 20512 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:48544 (request call id 145) took 2482 ms. Trace:
I20260129 02:59:40.597172 20432 raft_consensus.cc:2468] T dc2f4cb752cf4d59b5f6bc689f5d8104 P 197b880fa39d432a9b472845876bfaf0 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1c50de89a5a845668696dcb28c3badd6 in term 3.
I20260129 02:59:40.597426 20512 rpcz_store.cc:276] 0129 02:59:38.107742 (+ 0us) service_pool.cc:168] Inserting onto call queue
0129 02:59:39.441819 (+1334077us) service_pool.cc:225] Handling call
0129 02:59:39.442156 (+ 337us) raft_consensus.cc:1471] Updating replica for 0 ops
0129 02:59:39.800335 (+358179us) raft_consensus.cc:1697] Filling consensus response to leader.
0129 02:59:40.589683 (+789348us) spinlock_profiling.cc:243] Waited 346 ms on lock 0x7b54002ee854. stack: 00007f8fe044e373 00007f8fe044e270 00007f8fdfda0574 00007f8fe9a5c6b4 00007f8fe9a5c649 00007f8fe34975a1 00007f8fe352ea81 00007f8fe352d4be 00007f8fe84e1f49 00007f8fe1a6ed34 00007f8fe1a6ecc2 00007f8fe1a6ebf1 00007f8fe1a6eb6c 00007f8fe1a6de22 00007f8fe186d34c 00007f8fe186c786
0129 02:59:40.589922 (+ 239us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":795859712}
I20260129 02:59:40.598191 20441 raft_consensus.cc:2393] T 4408006eb5174de1896b037cf3f18457 P 197b880fa39d432a9b472845876bfaf0 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1c50de89a5a845668696dcb28c3badd6 in current term 2: Already voted for candidate 197b880fa39d432a9b472845876bfaf0 in this term.
I20260129 02:59:40.582290 22416 leader_election.cc:290] T 75b0b6e82ddc4f92b1fe1d94f9a0b73b P 974b802141bd4f13beffef2aa03f88f9 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3:37735), 7a9daf5d42a84615a6383cfec9a10608 (127.18.211.4:40439)
W20260129 02:59:40.581329 22342 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 1.389s user 0.028s sys 0.057s
I20260129 02:59:40.599326 22212 raft_consensus.cc:493] T fd8311b779af4e1fa1402b2a02a602f2 P 7a9daf5d42a84615a6383cfec9a10608 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260129 02:59:40.646657 22342 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 1.455s user 0.030s sys 0.059s
I20260129 02:59:40.599519 22432 raft_consensus.cc:697] T a8479d1fe0bd41e5946186e3a6ca8c7a P 197b880fa39d432a9b472845876bfaf0 [term 3 LEADER]: Becoming Leader. State: Replica: 197b880fa39d432a9b472845876bfaf0, State: Running, Role: LEADER
W20260129 02:59:40.630534 20436 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 1.557s user 0.024s sys 0.067s
I20260129 02:59:40.584232 22326 consensus_queue.cc:1048] T 4e420f90884f4796997b5ea41e49e411 P 197b880fa39d432a9b472845876bfaf0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.060s
I20260129 02:59:40.504128 22502 consensus_queue.cc:260] T a0fcc58d6fb6473cafbd9d34d5195269 P 197b880fa39d432a9b472845876bfaf0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:59:40.607769 22500 consensus_queue.cc:1048] T 8055232cf0004d40a85f3bf983fe00a9 P 197b880fa39d432a9b472845876bfaf0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.054s
I20260129 02:59:40.513468 20509 rpcz_store.cc:276] 0129 02:59:38.301830 (+ 0us) service_pool.cc:168] Inserting onto call queue
0129 02:59:39.851541 (+1549711us) service_pool.cc:225] Handling call
0129 02:59:39.903655 (+ 52114us) raft_consensus.cc:1471] Updating replica for 0 ops
0129 02:59:40.356050 (+452395us) raft_consensus.cc:1697] Filling consensus response to leader.
0129 02:59:40.398410 (+ 42360us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260129 02:59:40.621155 22380 raft_consensus.cc:3055] T b189f08e16a74a11980cc55d99359a0a P 197b880fa39d432a9b472845876bfaf0 [term 4 LEADER]: Stepping down as leader of term 4
I20260129 02:59:40.466486 22486 raft_consensus.cc:493] T 4f5a77e94fef4b16a2ba7b067c8b144a P 7a9daf5d42a84615a6383cfec9a10608 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260129 02:59:40.672258 20556 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260129 02:59:40.673405 20480 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.211.3:37735, user_credentials={real_user=slave}} blocked reactor thread for 171116us
I20260129 02:59:40.584563 22403 raft_consensus.cc:515] T 78aa39096868415d9afc2ffcb248e198 P 7a9daf5d42a84615a6383cfec9a10608 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } }
W20260129 02:59:40.662426 20338 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 1.143s user 0.020s sys 0.079s
W20260129 02:59:40.664747 20436 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 1.617s user 0.024s sys 0.068s
W20260129 02:59:40.709280 20338 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 1.190s user 0.021s sys 0.080s
I20260129 02:59:40.607193 22391 raft_consensus.cc:515] T 2e58e9a5a8aa4cecafb464addb4f35ed P 7a9daf5d42a84615a6383cfec9a10608 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } }
W20260129 02:59:40.720283 20481 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260129 02:59:40.724218 22416 raft_consensus.cc:493] T 4e420f90884f4796997b5ea41e49e411 P 974b802141bd4f13beffef2aa03f88f9 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260129 02:59:40.724809 22416 raft_consensus.cc:515] T 4e420f90884f4796997b5ea41e49e411 P 974b802141bd4f13beffef2aa03f88f9 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } }
I20260129 02:59:40.725520 20556 leader_election.cc:304] T 96609ac68bdf4993a12e1c23355ca013 P 1c50de89a5a845668696dcb28c3badd6 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1c50de89a5a845668696dcb28c3badd6; no voters: 197b880fa39d432a9b472845876bfaf0, 974b802141bd4f13beffef2aa03f88f9
I20260129 02:59:40.727260 22416 leader_election.cc:290] T 4e420f90884f4796997b5ea41e49e411 P 974b802141bd4f13beffef2aa03f88f9 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 197b880fa39d432a9b472845876bfaf0 (127.18.211.1:42027), 7a9daf5d42a84615a6383cfec9a10608 (127.18.211.4:40439)
I20260129 02:59:40.728058 20508 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57584 (request call id 1226) took 3758 ms. Trace:
I20260129 02:59:40.728956 22337 raft_consensus.cc:2749] T 96609ac68bdf4993a12e1c23355ca013 P 1c50de89a5a845668696dcb28c3badd6 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260129 02:59:40.731884 20480 leader_election.cc:304] T a0fcc58d6fb6473cafbd9d34d5195269 P 974b802141bd4f13beffef2aa03f88f9 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1c50de89a5a845668696dcb28c3badd6, 974b802141bd4f13beffef2aa03f88f9; no voters:
I20260129 02:59:40.733517 20480 leader_election.cc:304] T 8055232cf0004d40a85f3bf983fe00a9 P 974b802141bd4f13beffef2aa03f88f9 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1c50de89a5a845668696dcb28c3badd6, 974b802141bd4f13beffef2aa03f88f9; no voters:
I20260129 02:59:40.733623 22400 raft_consensus.cc:493] T 45cc5f3d2688486d9b6dbe7e666dedde P 974b802141bd4f13beffef2aa03f88f9 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 1c50de89a5a845668696dcb28c3badd6)
I20260129 02:59:40.734433 22160 consensus_queue.cc:1048] T 4b1876f9a70b49edb3aa74105e66d3b3 P 974b802141bd4f13beffef2aa03f88f9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260129 02:59:40.734341 22400 raft_consensus.cc:515] T 45cc5f3d2688486d9b6dbe7e666dedde P 974b802141bd4f13beffef2aa03f88f9 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } }
W20260129 02:59:40.735028 21101 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260129 02:59:40.735742 22160 raft_consensus.cc:2804] T a0fcc58d6fb6473cafbd9d34d5195269 P 974b802141bd4f13beffef2aa03f88f9 [term 3 FOLLOWER]: Leader election won for term 3
I20260129 02:59:40.736545 22493 raft_consensus.cc:2804] T 8055232cf0004d40a85f3bf983fe00a9 P 974b802141bd4f13beffef2aa03f88f9 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260129 02:59:40.737023 22493 raft_consensus.cc:493] T 8055232cf0004d40a85f3bf983fe00a9 P 974b802141bd4f13beffef2aa03f88f9 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260129 02:59:40.737409 22493 raft_consensus.cc:3060] T 8055232cf0004d40a85f3bf983fe00a9 P 974b802141bd4f13beffef2aa03f88f9 [term 2 FOLLOWER]: Advancing to term 3
W20260129 02:59:40.745652 22343 consensus_meta.cc:287] T 18f30ebaa4174fd28c958e72d0cc7e24 P 7a9daf5d42a84615a6383cfec9a10608: Time spent flushing consensus metadata: real 0.999s user 0.008s sys 0.022s
I20260129 02:59:40.746555 21127 raft_consensus.cc:2468] T 18f30ebaa4174fd28c958e72d0cc7e24 P 7a9daf5d42a84615a6383cfec9a10608 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 197b880fa39d432a9b472845876bfaf0 in term 3.
I20260129 02:59:40.747252 22255 raft_consensus.cc:493] T 18f30ebaa4174fd28c958e72d0cc7e24 P 7a9daf5d42a84615a6383cfec9a10608 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260129 02:59:40.709681 22516 raft_consensus.cc:493] T add8d4923d1a489e86882c3a25692b95 P 1c50de89a5a845668696dcb28c3badd6 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260129 02:59:40.752638 20482 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260129 02:59:40.728525 20508 rpcz_store.cc:276] 0129 02:59:36.969968 (+ 0us) service_pool.cc:168] Inserting onto call queue
0129 02:59:37.015906 (+ 45938us) service_pool.cc:225] Handling call
0129 02:59:37.041158 (+ 25252us) raft_consensus.cc:1471] Updating replica for 1 ops
0129 02:59:37.045781 (+ 4623us) raft_consensus.cc:1534] Early marking committed up to index 0
0129 02:59:37.045813 (+ 32us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0129 02:59:37.059004 (+ 13191us) log.cc:844] Serialized 26 byte log entry
0129 02:59:38.110870 (+1051866us) raft_consensus.cc:1645] Marking committed up to 1
0129 02:59:38.110911 (+ 41us) raft_consensus.cc:1697] Filling consensus response to leader.
0129 02:59:38.110972 (+ 61us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0129 02:59:40.525819 (+2414847us) spinlock_profiling.cc:243] Waited 8.57 ms on lock 0x7b440002304c. stack: 00007f8fe044e373 00007f8fe044e270 00007f8fdfda0574 00007f8fe9a5c6b4 00007f8fe9a5c649 00007f8fe02c08b1 00007f8fe038f005 00007f8fe038fa22 00007f8fe30d957b 00007f8fe30d4e4f 00007f8fe30c8c0e 00007f8fe30c7f56 00007f8fe30c7b78 00007f8fe30d15c1 00007f8fe30d1579 00007f8fe30d1509
0129 02:59:40.722996 (+197177us) raft_consensus.cc:1685] finished
0129 02:59:40.723031 (+ 35us) raft_consensus.cc:1691] UpdateReplicas() finished
0129 02:59:40.727974 (+ 4943us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":4266,"raft.queue_time_us":124111,"raft.run_cpu_time_us":67,"raft.run_wall_time_us":67,"spinlock_wait_cycles":19706752,"thread_start_us":1051470,"threads_started":1,"wal-append.queue_time_us":1254485}
I20260129 02:59:40.765687 22437 raft_consensus.cc:493] T a609ac02fae84725ab743f95f51d33ce P 974b802141bd4f13beffef2aa03f88f9 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260129 02:59:40.766371 22437 raft_consensus.cc:515] T a609ac02fae84725ab743f95f51d33ce P 974b802141bd4f13beffef2aa03f88f9 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } }
I20260129 02:59:40.768357 20589 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e4d18c35b0a7493e85d0a4b92956a48f" candidate_uuid: "974b802141bd4f13beffef2aa03f88f9" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "1c50de89a5a845668696dcb28c3badd6" is_pre_election: true
I20260129 02:59:40.689388 22380 raft_consensus.cc:740] T b189f08e16a74a11980cc55d99359a0a P 197b880fa39d432a9b472845876bfaf0 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 197b880fa39d432a9b472845876bfaf0, State: Running, Role: LEADER
I20260129 02:59:40.769227 20589 raft_consensus.cc:2468] T e4d18c35b0a7493e85d0a4b92956a48f P 1c50de89a5a845668696dcb28c3badd6 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 974b802141bd4f13beffef2aa03f88f9 in term 3.
I20260129 02:59:40.773659 20585 raft_consensus.cc:1275] T a5a05fbcbbcf4916bbc01fd1a5b5e90d P 1c50de89a5a845668696dcb28c3badd6 [term 5 FOLLOWER]: Refusing update from remote peer 974b802141bd4f13beffef2aa03f88f9: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 5 index: 4. (index mismatch)
I20260129 02:59:40.776880 20589 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3906b080a5b34a74b8b64128635e8997" candidate_uuid: "974b802141bd4f13beffef2aa03f88f9" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "1c50de89a5a845668696dcb28c3badd6" is_pre_election: true
I20260129 02:59:40.718219 22137 raft_consensus.cc:3060] T cb4727386a3e48aca5d71cc6c611c197 P 974b802141bd4f13beffef2aa03f88f9 [term 1 FOLLOWER]: Advancing to term 2
I20260129 02:59:40.663107 22212 raft_consensus.cc:515] T fd8311b779af4e1fa1402b2a02a602f2 P 7a9daf5d42a84615a6383cfec9a10608 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } }
W20260129 02:59:40.764232 22384 consensus_meta.cc:287] T 0340229d521b432580ae926f6062e3f7 P 197b880fa39d432a9b472845876bfaf0: Time spent flushing consensus metadata: real 1.294s user 0.019s sys 0.010s
I20260129 02:59:40.715736 22198 raft_consensus.cc:493] T 36ff33be666a4c82954731691dbcf327 P 197b880fa39d432a9b472845876bfaf0 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 1c50de89a5a845668696dcb28c3badd6)
I20260129 02:59:40.785202 20585 raft_consensus.cc:1240] T 75f45538dc704d9a8ec4ff8ad9f3ea5b P 1c50de89a5a845668696dcb28c3badd6 [term 2 FOLLOWER]: Rejecting Update request from peer 974b802141bd4f13beffef2aa03f88f9 for earlier term 1. Current term is 2. Ops: []
I20260129 02:59:40.749792 21132 raft_consensus.cc:1240] T 78aa39096868415d9afc2ffcb248e198 P 7a9daf5d42a84615a6383cfec9a10608 [term 4 FOLLOWER]: Rejecting Update request from peer 1c50de89a5a845668696dcb28c3badd6 for earlier term 3. Current term is 4. Ops: []
I20260129 02:59:40.791543 21136 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4f5a77e94fef4b16a2ba7b067c8b144a" candidate_uuid: "974b802141bd4f13beffef2aa03f88f9" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "7a9daf5d42a84615a6383cfec9a10608" is_pre_election: true
W20260129 02:59:40.788959 20554 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260129 02:59:40.719910 22157 raft_consensus.cc:697] T e33a417cf9c54cf5af73a4ee3aac4d3a P 974b802141bd4f13beffef2aa03f88f9 [term 4 LEADER]: Becoming Leader. State: Replica: 974b802141bd4f13beffef2aa03f88f9, State: Running, Role: LEADER
I20260129 02:59:40.789628 20582 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "913fc7d751ed4b7388d889aac30a5f9b" candidate_uuid: "974b802141bd4f13beffef2aa03f88f9" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1c50de89a5a845668696dcb28c3badd6" is_pre_election: true
I20260129 02:59:40.761859 22036 raft_consensus.cc:493] T add8d4923d1a489e86882c3a25692b95 P 974b802141bd4f13beffef2aa03f88f9 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260129 02:59:40.797137 20585 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4f5a77e94fef4b16a2ba7b067c8b144a" candidate_uuid: "974b802141bd4f13beffef2aa03f88f9" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "1c50de89a5a845668696dcb28c3badd6" is_pre_election: true
I20260129 02:59:40.798256 20437 raft_consensus.cc:2468] T 0340229d521b432580ae926f6062e3f7 P 197b880fa39d432a9b472845876bfaf0 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1c50de89a5a845668696dcb28c3badd6 in term 4.
I20260129 02:59:40.804212 20584 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "75f45538dc704d9a8ec4ff8ad9f3ea5b" candidate_uuid: "974b802141bd4f13beffef2aa03f88f9" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "1c50de89a5a845668696dcb28c3badd6" is_pre_election: true
I20260129 02:59:40.764679 22403 leader_election.cc:290] T 78aa39096868415d9afc2ffcb248e198 P 7a9daf5d42a84615a6383cfec9a10608 [CANDIDATE]: Term 4 election: Requested vote from peers 197b880fa39d432a9b472845876bfaf0 (127.18.211.1:42027), 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3:37735)
I20260129 02:59:40.774227 22502 raft_consensus.cc:3060] T a0fcc58d6fb6473cafbd9d34d5195269 P 197b880fa39d432a9b472845876bfaf0 [term 2 FOLLOWER]: Advancing to term 3
I20260129 02:59:40.806196 20589 raft_consensus.cc:1240] T 4408006eb5174de1896b037cf3f18457 P 1c50de89a5a845668696dcb28c3badd6 [term 2 FOLLOWER]: Rejecting Update request from peer 974b802141bd4f13beffef2aa03f88f9 for earlier term 1. Current term is 2. Ops: []
I20260129 02:59:40.770129 22320 leader_election.cc:290] T a8479d1fe0bd41e5946186e3a6ca8c7a P 1c50de89a5a845668696dcb28c3badd6 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 197b880fa39d432a9b472845876bfaf0 (127.18.211.1:42027), 974b802141bd4f13beffef2aa03f88f9 (127.18.211.2:34517)
I20260129 02:59:40.772691 22160 raft_consensus.cc:697] T a0fcc58d6fb6473cafbd9d34d5195269 P 974b802141bd4f13beffef2aa03f88f9 [term 3 LEADER]: Becoming Leader. State: Replica: 974b802141bd4f13beffef2aa03f88f9, State: Running, Role: LEADER
I20260129 02:59:40.812057 20584 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0340229d521b432580ae926f6062e3f7" candidate_uuid: "974b802141bd4f13beffef2aa03f88f9" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "1c50de89a5a845668696dcb28c3badd6" is_pre_election: true
I20260129 02:59:40.716572 22486 raft_consensus.cc:515] T 4f5a77e94fef4b16a2ba7b067c8b144a P 7a9daf5d42a84615a6383cfec9a10608 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } }
I20260129 02:59:40.814193 20584 raft_consensus.cc:2393] T 0340229d521b432580ae926f6062e3f7 P 1c50de89a5a845668696dcb28c3badd6 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 974b802141bd4f13beffef2aa03f88f9 in current term 4: Already voted for candidate 1c50de89a5a845668696dcb28c3badd6 in this term.
I20260129 02:59:40.816439 21130 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2ac659d732314e1288849f08fe5e074e" candidate_uuid: "974b802141bd4f13beffef2aa03f88f9" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "7a9daf5d42a84615a6383cfec9a10608" is_pre_election: true
I20260129 02:59:40.816620 20590 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "505580c93c694bbb8ea96fdf9b17ab46" candidate_uuid: "974b802141bd4f13beffef2aa03f88f9" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "1c50de89a5a845668696dcb28c3badd6"
I20260129 02:59:40.817314 21130 raft_consensus.cc:2468] T 2ac659d732314e1288849f08fe5e074e P 7a9daf5d42a84615a6383cfec9a10608 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 974b802141bd4f13beffef2aa03f88f9 in term 2.
I20260129 02:59:40.750264 22516 raft_consensus.cc:515] T add8d4923d1a489e86882c3a25692b95 P 1c50de89a5a845668696dcb28c3badd6 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } }
W20260129 02:59:40.830533 20480 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260129 02:59:40.818168 22374 consensus_queue.cc:1059] T 0678b8b468bd43cdbc3e485900844821 P 7a9daf5d42a84615a6383cfec9a10608 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 }, Status: INVALID_TERM, Last received: 4.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.174s
W20260129 02:59:40.992208 21102 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.211.2:34517, user_credentials={real_user=slave}} blocked reactor thread for 163760us
I20260129 02:59:40.994853 20480 leader_election.cc:304] T 87d5fe48a44a408aaf03e894a67ecb2f P 974b802141bd4f13beffef2aa03f88f9 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 974b802141bd4f13beffef2aa03f88f9; no voters: 197b880fa39d432a9b472845876bfaf0, 7a9daf5d42a84615a6383cfec9a10608
I20260129 02:59:40.995255 20589 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "75b0b6e82ddc4f92b1fe1d94f9a0b73b" candidate_uuid: "974b802141bd4f13beffef2aa03f88f9" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "1c50de89a5a845668696dcb28c3badd6" is_pre_election: true
I20260129 02:59:40.997325 21130 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "505580c93c694bbb8ea96fdf9b17ab46" candidate_uuid: "974b802141bd4f13beffef2aa03f88f9" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "7a9daf5d42a84615a6383cfec9a10608"
I20260129 02:59:40.998735 22357 raft_consensus.cc:493] T 96609ac68bdf4993a12e1c23355ca013 P 974b802141bd4f13beffef2aa03f88f9 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260129 02:59:40.999354 22320 raft_consensus.cc:493] T fd8311b779af4e1fa1402b2a02a602f2 P 1c50de89a5a845668696dcb28c3badd6 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260129 02:59:40.996713 20480 leader_election.cc:304] T 2ac659d732314e1288849f08fe5e074e P 974b802141bd4f13beffef2aa03f88f9 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7a9daf5d42a84615a6383cfec9a10608, 974b802141bd4f13beffef2aa03f88f9; no voters:
I20260129 02:59:40.999815 22157 consensus_queue.cc:237] T e33a417cf9c54cf5af73a4ee3aac4d3a P 974b802141bd4f13beffef2aa03f88f9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 3.2, Last appended by leader: 2, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:59:40.991086 22403 consensus_queue.cc:1048] T 75f45538dc704d9a8ec4ff8ad9f3ea5b P 7a9daf5d42a84615a6383cfec9a10608 [LEADER]: Connected to new peer: Peer: permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20260129 02:59:41.001739 22364 raft_consensus.cc:3055] T 0678b8b468bd43cdbc3e485900844821 P 7a9daf5d42a84615a6383cfec9a10608 [term 4 LEADER]: Stepping down as leader of term 4
I20260129 02:59:41.022886 20590 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "17d6431cc77a491e8464d68db4081a51" candidate_uuid: "974b802141bd4f13beffef2aa03f88f9" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1c50de89a5a845668696dcb28c3badd6" is_pre_election: true
I20260129 02:59:41.041045 21136 raft_consensus.cc:2393] T 4f5a77e94fef4b16a2ba7b067c8b144a P 7a9daf5d42a84615a6383cfec9a10608 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 974b802141bd4f13beffef2aa03f88f9 in current term 3: Already voted for candidate 1c50de89a5a845668696dcb28c3badd6 in this term.
I20260129 02:59:40.810341 22036 raft_consensus.cc:515] T add8d4923d1a489e86882c3a25692b95 P 974b802141bd4f13beffef2aa03f88f9 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } }
I20260129 02:59:41.067037 20590 raft_consensus.cc:2468] T 17d6431cc77a491e8464d68db4081a51 P 1c50de89a5a845668696dcb28c3badd6 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 974b802141bd4f13beffef2aa03f88f9 in term 2.
I20260129 02:59:41.071197 20512 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "cb0b3f259ebc4310927338a8740e87b8" candidate_uuid: "1c50de89a5a845668696dcb28c3badd6" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "974b802141bd4f13beffef2aa03f88f9" is_pre_election: true
I20260129 02:59:41.037678 20585 raft_consensus.cc:1240] T cb4727386a3e48aca5d71cc6c611c197 P 1c50de89a5a845668696dcb28c3badd6 [term 2 FOLLOWER]: Rejecting Update request from peer 974b802141bd4f13beffef2aa03f88f9 for earlier term 1. Current term is 2. Ops: []
I20260129 02:59:40.747845 22255 raft_consensus.cc:515] T 18f30ebaa4174fd28c958e72d0cc7e24 P 7a9daf5d42a84615a6383cfec9a10608 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } }
I20260129 02:59:40.821537 20582 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a8479d1fe0bd41e5946186e3a6ca8c7a" candidate_uuid: "974b802141bd4f13beffef2aa03f88f9" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "1c50de89a5a845668696dcb28c3badd6" is_pre_election: true
I20260129 02:59:41.013540 22499 raft_consensus.cc:493] T 00e79d8f25a34676b1d866eb2068a2b2 P 974b802141bd4f13beffef2aa03f88f9 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260129 02:59:40.798229 22198 raft_consensus.cc:515] T 36ff33be666a4c82954731691dbcf327 P 197b880fa39d432a9b472845876bfaf0 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:59:41.032295 22400 leader_election.cc:290] T 45cc5f3d2688486d9b6dbe7e666dedde P 974b802141bd4f13beffef2aa03f88f9 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3:37735), 7a9daf5d42a84615a6383cfec9a10608 (127.18.211.4:40439)
I20260129 02:59:41.076277 20582 raft_consensus.cc:2468] T a8479d1fe0bd41e5946186e3a6ca8c7a P 1c50de89a5a845668696dcb28c3badd6 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 974b802141bd4f13beffef2aa03f88f9 in term 3.
I20260129 02:59:41.077579 20514 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "76f713c3331540d2a5b76164103e9eb1" candidate_uuid: "1c50de89a5a845668696dcb28c3badd6" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "974b802141bd4f13beffef2aa03f88f9"
W20260129 02:59:41.089455 21100 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.211.3:37735, user_credentials={real_user=slave}} blocked reactor thread for 393034us
I20260129 02:59:41.072129 20512 raft_consensus.cc:2468] T cb0b3f259ebc4310927338a8740e87b8 P 974b802141bd4f13beffef2aa03f88f9 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1c50de89a5a845668696dcb28c3badd6 in term 4.
I20260129 02:59:40.992797 22374 consensus_queue.cc:1048] T 75b0b6e82ddc4f92b1fe1d94f9a0b73b P 7a9daf5d42a84615a6383cfec9a10608 [LEADER]: Connected to new peer: Peer: permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.066s
W20260129 02:59:41.099588 20480 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.211.4:40439, user_credentials={real_user=slave}} blocked reactor thread for 102946us
I20260129 02:59:41.111487 20516 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4f5a77e94fef4b16a2ba7b067c8b144a" candidate_uuid: "1c50de89a5a845668696dcb28c3badd6" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "974b802141bd4f13beffef2aa03f88f9" is_pre_election: true
I20260129 02:59:41.114003 20510 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a644c02b16af47679824ccd060ac10fd" candidate_uuid: "1c50de89a5a845668696dcb28c3badd6" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "974b802141bd4f13beffef2aa03f88f9"
I20260129 02:59:41.175804 20510 raft_consensus.cc:3060] T a644c02b16af47679824ccd060ac10fd P 974b802141bd4f13beffef2aa03f88f9 [term 2 FOLLOWER]: Advancing to term 3
I20260129 02:59:41.051213 22366 leader_election.cc:290] T 4e420f90884f4796997b5ea41e49e411 P 7a9daf5d42a84615a6383cfec9a10608 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 197b880fa39d432a9b472845876bfaf0 (127.18.211.1:42027), 974b802141bd4f13beffef2aa03f88f9 (127.18.211.2:34517)
I20260129 02:59:41.039846 22320 raft_consensus.cc:515] T fd8311b779af4e1fa1402b2a02a602f2 P 1c50de89a5a845668696dcb28c3badd6 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } }
I20260129 02:59:41.144598 20588 raft_consensus.cc:1240] T b189f08e16a74a11980cc55d99359a0a P 1c50de89a5a845668696dcb28c3badd6 [term 5 FOLLOWER]: Rejecting Update request from peer 197b880fa39d432a9b472845876bfaf0 for earlier term 4. Current term is 5. Ops: []
I20260129 02:59:41.146247 20514 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4f5a77e94fef4b16a2ba7b067c8b144a" candidate_uuid: "1c50de89a5a845668696dcb28c3badd6" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "974b802141bd4f13beffef2aa03f88f9"
I20260129 02:59:41.190541 20514 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4e420f90884f4796997b5ea41e49e411" candidate_uuid: "7a9daf5d42a84615a6383cfec9a10608" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "974b802141bd4f13beffef2aa03f88f9" is_pre_election: true
I20260129 02:59:41.191311 20514 raft_consensus.cc:2468] T 4e420f90884f4796997b5ea41e49e411 P 974b802141bd4f13beffef2aa03f88f9 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7a9daf5d42a84615a6383cfec9a10608 in term 4.
I20260129 02:59:41.196054 20514 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "add8d4923d1a489e86882c3a25692b95" candidate_uuid: "7a9daf5d42a84615a6383cfec9a10608" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "974b802141bd4f13beffef2aa03f88f9"
I20260129 02:59:41.144368 20587 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:35752 (request call id 1495) took 1325 ms. Trace:
I20260129 02:59:41.197378 20514 raft_consensus.cc:2393] T add8d4923d1a489e86882c3a25692b95 P 974b802141bd4f13beffef2aa03f88f9 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 7a9daf5d42a84615a6383cfec9a10608 in current term 2: Already voted for candidate 974b802141bd4f13beffef2aa03f88f9 in this term.
I20260129 02:59:41.197187 20587 rpcz_store.cc:276] 0129 02:59:39.819153 (+ 0us) service_pool.cc:168] Inserting onto call queue
0129 02:59:39.841506 (+ 22353us) service_pool.cc:225] Handling call
0129 02:59:41.122766 (+1281260us) spinlock_profiling.cc:243] Waited 46.8 ms on lock 0x7b54001536d4. stack: 00007f8fe044e373 00007f8fe044e270 00007f8fdfda0574 00007f8fe9a5c6b4 00007f8fe9a5c649 00007f8fe34975a1 00007f8fe3531f7a 00007f8fe84e3647 00007f8fe1a73834 00007f8fe1a737c2 00007f8fe1a736f1 00007f8fe1a7366c 00007f8fe1a72922 00007f8fe186d34c 00007f8fe186c786 00007f8fe186c12f
0129 02:59:41.144287 (+ 21521us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":107558656}
W20260129 02:59:41.196494 20553 thread.cc:641] dns-resolver [worker] (thread pool) Time spent creating pthread: real 1.312s user 0.050s sys 0.069s
I20260129 02:59:41.150355 22400 raft_consensus.cc:2804] T 2ac659d732314e1288849f08fe5e074e P 974b802141bd4f13beffef2aa03f88f9 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260129 02:59:40.815229 22432 consensus_queue.cc:237] T a8479d1fe0bd41e5946186e3a6ca8c7a P 197b880fa39d432a9b472845876bfaf0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:59:41.069828 22364 raft_consensus.cc:740] T 0678b8b468bd43cdbc3e485900844821 P 7a9daf5d42a84615a6383cfec9a10608 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 7a9daf5d42a84615a6383cfec9a10608, State: Running, Role: LEADER
W20260129 02:59:41.209878 22231 consensus_meta.cc:287] T 1ce4ddf5b10e4ae8b193b13b2e5936ee P 197b880fa39d432a9b472845876bfaf0: Time spent flushing consensus metadata: real 1.723s user 0.008s sys 0.003s
I20260129 02:59:41.209744 22364 consensus_queue.cc:260] T 0678b8b468bd43cdbc3e485900844821 P 7a9daf5d42a84615a6383cfec9a10608 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } }
I20260129 02:59:41.210295 22231 raft_consensus.cc:515] T 1ce4ddf5b10e4ae8b193b13b2e5936ee P 197b880fa39d432a9b472845876bfaf0 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:59:41.210829 22364 raft_consensus.cc:3060] T 0678b8b468bd43cdbc3e485900844821 P 7a9daf5d42a84615a6383cfec9a10608 [term 4 FOLLOWER]: Advancing to term 5
I20260129 02:59:41.211349 20439 raft_consensus.cc:1240] T 1ce4ddf5b10e4ae8b193b13b2e5936ee P 197b880fa39d432a9b472845876bfaf0 [term 5 FOLLOWER]: Rejecting Update request from peer 1c50de89a5a845668696dcb28c3badd6 for earlier term 4. Current term is 5. Ops: [4.3-4.3]
W20260129 02:59:41.171392 20406 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260129 02:59:41.147064 20513 raft_consensus.cc:1275] T 7a9a69cd82724b1faed1d10aa091fc4a P 974b802141bd4f13beffef2aa03f88f9 [term 4 FOLLOWER]: Refusing update from remote peer 1c50de89a5a845668696dcb28c3badd6: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20260129 02:59:41.212565 20439 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55676 (request call id 1182) took 1072 ms. Trace:
I20260129 02:59:41.212922 20439 rpcz_store.cc:276] 0129 02:59:40.330926 (+ 0us) service_pool.cc:168] Inserting onto call queue
0129 02:59:40.425505 (+ 94579us) service_pool.cc:225] Handling call
0129 02:59:40.461039 (+ 35534us) raft_consensus.cc:1471] Updating replica for 1 ops
0129 02:59:41.211728 (+750689us) raft_consensus.cc:1697] Filling consensus response to leader.
0129 02:59:41.212302 (+ 574us) spinlock_profiling.cc:243] Waited 750 ms on lock 0x7b540025f154. stack: 00007f8fe044e373 00007f8fe044e270 00007f8fdfda0574 00007f8fe9a5c6b4 00007f8fe9a5c649 00007f8fe34975a1 00007f8fe352ea81 00007f8fe352d4be 00007f8fe84e1f49 00007f8fe1a6ed34 00007f8fe1a6ecc2 00007f8fe1a6ebf1 00007f8fe1a6eb6c 00007f8fe1a6de22 00007f8fe186d34c 00007f8fe186c786
0129 02:59:41.212498 (+ 196us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1724562688}
I20260129 02:59:41.213825 20513 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e4d18c35b0a7493e85d0a4b92956a48f" candidate_uuid: "1c50de89a5a845668696dcb28c3badd6" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "974b802141bd4f13beffef2aa03f88f9"
I20260129 02:59:41.215154 20439 raft_consensus.cc:1240] T 368953c9102643b2a21f99af36b51905 P 197b880fa39d432a9b472845876bfaf0 [term 5 FOLLOWER]: Rejecting Update request from peer 1c50de89a5a845668696dcb28c3badd6 for earlier term 4. Current term is 5. Ops: [4.3-4.3]
W20260129 02:59:41.216634 20482 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260129 02:59:41.216797 22486 leader_election.cc:290] T 4f5a77e94fef4b16a2ba7b067c8b144a P 7a9daf5d42a84615a6383cfec9a10608 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 974b802141bd4f13beffef2aa03f88f9 (127.18.211.2:34517), 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3:37735)
I20260129 02:59:41.222131 22320 leader_election.cc:290] T fd8311b779af4e1fa1402b2a02a602f2 P 1c50de89a5a845668696dcb28c3badd6 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 197b880fa39d432a9b472845876bfaf0 (127.18.211.1:42027), 7a9daf5d42a84615a6383cfec9a10608 (127.18.211.4:40439)
I20260129 02:59:41.223798 22198 leader_election.cc:290] T 36ff33be666a4c82954731691dbcf327 P 197b880fa39d432a9b472845876bfaf0 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 974b802141bd4f13beffef2aa03f88f9 (127.18.211.2:34517), 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3:37735)
I20260129 02:59:41.224102 22514 raft_consensus.cc:493] T fcddab6bca684adf8ea032289a00f90e P 974b802141bd4f13beffef2aa03f88f9 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260129 02:59:41.224851 22514 raft_consensus.cc:515] T fcddab6bca684adf8ea032289a00f90e P 974b802141bd4f13beffef2aa03f88f9 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:59:41.204810 22348 raft_consensus.cc:3060] T fcddab6bca684adf8ea032289a00f90e P 1c50de89a5a845668696dcb28c3badd6 [term 6 FOLLOWER]: Advancing to term 7
I20260129 02:59:41.120321 22499 raft_consensus.cc:515] T 00e79d8f25a34676b1d866eb2068a2b2 P 974b802141bd4f13beffef2aa03f88f9 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } }
I20260129 02:59:41.206327 20514 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3ccac82a01a2445fa5b0ce6c0fcf2295" candidate_uuid: "7a9daf5d42a84615a6383cfec9a10608" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "974b802141bd4f13beffef2aa03f88f9" is_pre_election: true
I20260129 02:59:41.228165 20514 raft_consensus.cc:2393] T 3ccac82a01a2445fa5b0ce6c0fcf2295 P 974b802141bd4f13beffef2aa03f88f9 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 7a9daf5d42a84615a6383cfec9a10608 in current term 2: Already voted for candidate 974b802141bd4f13beffef2aa03f88f9 in this term.
I20260129 02:59:41.228701 22499 leader_election.cc:290] T 00e79d8f25a34676b1d866eb2068a2b2 P 974b802141bd4f13beffef2aa03f88f9 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 197b880fa39d432a9b472845876bfaf0 (127.18.211.1:42027), 7a9daf5d42a84615a6383cfec9a10608 (127.18.211.4:40439)
W20260129 02:59:41.128765 22408 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 1.289s user 0.054s sys 0.088s
I20260129 02:59:41.229266 20514 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a609ac02fae84725ab743f95f51d33ce" candidate_uuid: "7a9daf5d42a84615a6383cfec9a10608" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "974b802141bd4f13beffef2aa03f88f9" is_pre_election: true
I20260129 02:59:41.040903 22357 raft_consensus.cc:515] T 96609ac68bdf4993a12e1c23355ca013 P 974b802141bd4f13beffef2aa03f88f9 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } attrs { replace: true } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } }
W20260129 02:59:41.230430 22519 log.cc:927] Time spent T 0678b8b468bd43cdbc3e485900844821 P 197b880fa39d432a9b472845876bfaf0: Append to log took a long time: real 0.080s user 0.000s sys 0.000s
I20260129 02:59:41.232491 22357 leader_election.cc:290] T 96609ac68bdf4993a12e1c23355ca013 P 974b802141bd4f13beffef2aa03f88f9 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 197b880fa39d432a9b472845876bfaf0 (127.18.211.1:42027), 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3:37735), 7a9daf5d42a84615a6383cfec9a10608 (127.18.211.4:40439)
I20260129 02:59:41.150913 22160 consensus_queue.cc:237] T a0fcc58d6fb6473cafbd9d34d5195269 P 974b802141bd4f13beffef2aa03f88f9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:59:40.880545 22380 consensus_queue.cc:260] T b189f08e16a74a11980cc55d99359a0a P 197b880fa39d432a9b472845876bfaf0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
W20260129 02:59:41.185833 20515 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 1.916s user 0.029s sys 0.059s
I20260129 02:59:41.181679 20584 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "45cc5f3d2688486d9b6dbe7e666dedde" candidate_uuid: "974b802141bd4f13beffef2aa03f88f9" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "1c50de89a5a845668696dcb28c3badd6" is_pre_election: true
I20260129 02:59:41.240798 22380 raft_consensus.cc:3060] T b189f08e16a74a11980cc55d99359a0a P 197b880fa39d432a9b472845876bfaf0 [term 4 FOLLOWER]: Advancing to term 5
W20260129 02:59:41.241551 20515 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 1.972s user 0.029s sys 0.060s
I20260129 02:59:41.242581 20515 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:48544 (request call id 140) took 3331 ms. Trace:
W20260129 02:59:41.143268 22483 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 1.411s user 0.018s sys 0.067s
W20260129 02:59:41.244076 22483 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 1.515s user 0.018s sys 0.067s
I20260129 02:59:41.195927 20512 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "cb0b3f259ebc4310927338a8740e87b8" candidate_uuid: "7a9daf5d42a84615a6383cfec9a10608" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "974b802141bd4f13beffef2aa03f88f9" is_pre_election: true
I20260129 02:59:41.151059 22342 raft_consensus.cc:493] T a644c02b16af47679824ccd060ac10fd P 7a9daf5d42a84615a6383cfec9a10608 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260129 02:59:41.244727 20512 raft_consensus.cc:2468] T cb0b3f259ebc4310927338a8740e87b8 P 974b802141bd4f13beffef2aa03f88f9 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7a9daf5d42a84615a6383cfec9a10608 in term 4.
I20260129 02:59:41.244863 22342 raft_consensus.cc:515] T a644c02b16af47679824ccd060ac10fd P 7a9daf5d42a84615a6383cfec9a10608 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } }
I20260129 02:59:41.245995 20512 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c934d31efd3f4e5195605ddc354d14e1" candidate_uuid: "7a9daf5d42a84615a6383cfec9a10608" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "974b802141bd4f13beffef2aa03f88f9" is_pre_election: true
I20260129 02:59:41.246799 20512 raft_consensus.cc:2393] T c934d31efd3f4e5195605ddc354d14e1 P 974b802141bd4f13beffef2aa03f88f9 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 7a9daf5d42a84615a6383cfec9a10608 in current term 4: Already voted for candidate 974b802141bd4f13beffef2aa03f88f9 in this term.
I20260129 02:59:41.247051 22342 leader_election.cc:290] T a644c02b16af47679824ccd060ac10fd P 7a9daf5d42a84615a6383cfec9a10608 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 974b802141bd4f13beffef2aa03f88f9 (127.18.211.2:34517), 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3:37735)
I20260129 02:59:41.247622 22416 raft_consensus.cc:2749] T 87d5fe48a44a408aaf03e894a67ecb2f P 974b802141bd4f13beffef2aa03f88f9 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260129 02:59:41.148239 20585 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "78aa39096868415d9afc2ffcb248e198" candidate_uuid: "7a9daf5d42a84615a6383cfec9a10608" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "1c50de89a5a845668696dcb28c3badd6"
W20260129 02:59:41.199558 20553 thread.cc:608] dns-resolver [worker] (thread pool) Time spent starting thread: real 1.315s user 0.050s sys 0.069s
W20260129 02:59:41.200682 20479 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 2.271s user 0.045s sys 0.103s
I20260129 02:59:41.149564 22437 leader_election.cc:290] T a609ac02fae84725ab743f95f51d33ce P 974b802141bd4f13beffef2aa03f88f9 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 197b880fa39d432a9b472845876bfaf0 (127.18.211.1:42027), 7a9daf5d42a84615a6383cfec9a10608 (127.18.211.4:40439)
I20260129 02:59:41.243055 20515 rpcz_store.cc:276] 0129 02:59:37.911484 (+ 0us) service_pool.cc:168] Inserting onto call queue
0129 02:59:39.247219 (+1335735us) service_pool.cc:225] Handling call
0129 02:59:39.269116 (+ 21897us) raft_consensus.cc:1471] Updating replica for 0 ops
0129 02:59:39.269344 (+ 228us) raft_consensus.cc:1534] Early marking committed up to index 6
0129 02:59:39.269671 (+ 327us) log.cc:844] Serialized 14 byte log entry
0129 02:59:41.242168 (+1972497us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0129 02:59:41.242233 (+ 65us) raft_consensus.cc:1645] Marking committed up to 6
0129 02:59:41.242284 (+ 51us) raft_consensus.cc:1697] Filling consensus response to leader.
0129 02:59:41.242384 (+ 100us) raft_consensus.cc:1691] UpdateReplicas() finished
0129 02:59:41.242524 (+ 140us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":1293,"prepare.run_cpu_time_us":42,"prepare.run_wall_time_us":44,"thread_start_us":1972101,"threads_started":1}
W20260129 02:59:41.248378 20553 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.211.1:42027, user_credentials={real_user=slave}} blocked reactor thread for 1.36438e+06us
I20260129 02:59:41.247829 20512 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2ac659d732314e1288849f08fe5e074e" candidate_uuid: "7a9daf5d42a84615a6383cfec9a10608" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "974b802141bd4f13beffef2aa03f88f9" is_pre_election: true
W20260129 02:59:41.229637 22408 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 1.393s user 0.054s sys 0.088s
I20260129 02:59:41.175237 20516 raft_consensus.cc:2468] T 4f5a77e94fef4b16a2ba7b067c8b144a P 974b802141bd4f13beffef2aa03f88f9 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1c50de89a5a845668696dcb28c3badd6 in term 2.
W20260129 02:59:41.139699 21100 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260129 02:59:41.260298 21100 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260129 02:59:41.248521 20479 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 2.322s user 0.045s sys 0.103s
I20260129 02:59:41.255307 20510 raft_consensus.cc:2468] T a644c02b16af47679824ccd060ac10fd P 974b802141bd4f13beffef2aa03f88f9 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1c50de89a5a845668696dcb28c3badd6 in term 3.
I20260129 02:59:41.206054 22212 leader_election.cc:290] T fd8311b779af4e1fa1402b2a02a602f2 P 7a9daf5d42a84615a6383cfec9a10608 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 197b880fa39d432a9b472845876bfaf0 (127.18.211.1:42027), 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3:37735)
W20260129 02:59:41.274749 20479 connection.cc:232] RPC call timeout handler was delayed by -1.4005055989999846s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1543s for a response
I20260129 02:59:41.263451 20584 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4f5a77e94fef4b16a2ba7b067c8b144a" candidate_uuid: "7a9daf5d42a84615a6383cfec9a10608" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "1c50de89a5a845668696dcb28c3badd6" is_pre_election: true
I20260129 02:59:41.264241 20585 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fd8311b779af4e1fa1402b2a02a602f2" candidate_uuid: "7a9daf5d42a84615a6383cfec9a10608" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "1c50de89a5a845668696dcb28c3badd6" is_pre_election: true
I20260129 02:59:41.290850 20585 raft_consensus.cc:2468] T fd8311b779af4e1fa1402b2a02a602f2 P 1c50de89a5a845668696dcb28c3badd6 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7a9daf5d42a84615a6383cfec9a10608 in term 3.
I20260129 02:59:41.198791 20508 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "add8d4923d1a489e86882c3a25692b95" candidate_uuid: "7a9daf5d42a84615a6383cfec9a10608" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "974b802141bd4f13beffef2aa03f88f9" is_pre_election: true
I20260129 02:59:41.264850 20510 raft_consensus.cc:3060] T cb0b3f259ebc4310927338a8740e87b8 P 974b802141bd4f13beffef2aa03f88f9 [term 4 FOLLOWER]: Advancing to term 5
I20260129 02:59:41.316468 20508 raft_consensus.cc:2393] T add8d4923d1a489e86882c3a25692b95 P 974b802141bd4f13beffef2aa03f88f9 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 7a9daf5d42a84615a6383cfec9a10608 in current term 2: Already voted for candidate 974b802141bd4f13beffef2aa03f88f9 in this term.
W20260129 02:59:41.147269 20581 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 1.337s user 0.012s sys 0.050s
W20260129 02:59:41.317991 20581 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 1.508s user 0.013s sys 0.050s
I20260129 02:59:41.265712 20514 raft_consensus.cc:2468] T a609ac02fae84725ab743f95f51d33ce P 974b802141bd4f13beffef2aa03f88f9 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7a9daf5d42a84615a6383cfec9a10608 in term 5.
I20260129 02:59:41.269335 20513 raft_consensus.cc:2468] T e4d18c35b0a7493e85d0a4b92956a48f P 974b802141bd4f13beffef2aa03f88f9 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1c50de89a5a845668696dcb28c3badd6 in term 3.
I20260129 02:59:41.348929 20513 raft_consensus.cc:1240] T 0678b8b468bd43cdbc3e485900844821 P 974b802141bd4f13beffef2aa03f88f9 [term 5 FOLLOWER]: Rejecting Update request from peer 7a9daf5d42a84615a6383cfec9a10608 for earlier term 4. Current term is 5. Ops: []
W20260129 02:59:41.369551 20406 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260129 02:59:41.387629 20516 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4b1876f9a70b49edb3aa74105e66d3b3" candidate_uuid: "1c50de89a5a845668696dcb28c3badd6" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "974b802141bd4f13beffef2aa03f88f9" is_pre_election: true
I20260129 02:59:41.389268 20516 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a0fcc58d6fb6473cafbd9d34d5195269" candidate_uuid: "1c50de89a5a845668696dcb28c3badd6" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "974b802141bd4f13beffef2aa03f88f9" is_pre_election: true
W20260129 02:59:41.275343 20479 connection.cc:232] RPC call timeout handler was delayed by -1.3753055989999439s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.15710000000000002s for a response
W20260129 02:59:41.391692 20479 connection.cc:232] RPC call timeout handler was delayed by -1.3483055989999002s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.16010000000000002s for a response
W20260129 02:59:41.392102 20479 connection.cc:232] RPC call timeout handler was delayed by -1.2781055989999004s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.16790000000000002s for a response
W20260129 02:59:41.392449 20479 connection.cc:232] RPC call timeout handler was delayed by -1.1980055989999983s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1768s for a response
I20260129 02:59:41.396000 20553 leader_election.cc:304] T 18f30ebaa4174fd28c958e72d0cc7e24 P 1c50de89a5a845668696dcb28c3badd6 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 197b880fa39d432a9b472845876bfaf0, 1c50de89a5a845668696dcb28c3badd6; no voters: 7a9daf5d42a84615a6383cfec9a10608
I20260129 02:59:41.404346 20585 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a644c02b16af47679824ccd060ac10fd" candidate_uuid: "7a9daf5d42a84615a6383cfec9a10608" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "1c50de89a5a845668696dcb28c3badd6" is_pre_election: true
I20260129 02:59:41.397480 20553 leader_election.cc:304] T 42d27a28d89b409e8807ec0efd97af69 P 1c50de89a5a845668696dcb28c3badd6 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1c50de89a5a845668696dcb28c3badd6; no voters: 197b880fa39d432a9b472845876bfaf0, 7a9daf5d42a84615a6383cfec9a10608
I20260129 02:59:41.408504 20515 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "505580c93c694bbb8ea96fdf9b17ab46" candidate_uuid: "7a9daf5d42a84615a6383cfec9a10608" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "974b802141bd4f13beffef2aa03f88f9" is_pre_election: true
I20260129 02:59:41.409561 20515 raft_consensus.cc:2393] T 505580c93c694bbb8ea96fdf9b17ab46 P 974b802141bd4f13beffef2aa03f88f9 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 7a9daf5d42a84615a6383cfec9a10608 in current term 4: Already voted for candidate 974b802141bd4f13beffef2aa03f88f9 in this term.
I20260129 02:59:41.204331 21130 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "75f45538dc704d9a8ec4ff8ad9f3ea5b" candidate_uuid: "974b802141bd4f13beffef2aa03f88f9" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "7a9daf5d42a84615a6383cfec9a10608" is_pre_election: true
W20260129 02:59:41.271898 22330 consensus_meta.cc:287] T 2e58e9a5a8aa4cecafb464addb4f35ed P 197b880fa39d432a9b472845876bfaf0: Time spent flushing consensus metadata: real 2.021s user 0.009s sys 0.022s
I20260129 02:59:41.416879 21100 leader_election.cc:304] T fd8311b779af4e1fa1402b2a02a602f2 P 7a9daf5d42a84615a6383cfec9a10608 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1c50de89a5a845668696dcb28c3badd6, 7a9daf5d42a84615a6383cfec9a10608; no voters:
I20260129 02:59:41.417469 20479 leader_election.cc:304] T 7d9494f5584d4d5db8b9adf1c8fc888b P 974b802141bd4f13beffef2aa03f88f9 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 974b802141bd4f13beffef2aa03f88f9; no voters: 197b880fa39d432a9b472845876bfaf0, 1c50de89a5a845668696dcb28c3badd6
I20260129 02:59:41.248993 20509 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a644c02b16af47679824ccd060ac10fd" candidate_uuid: "1c50de89a5a845668696dcb28c3badd6" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "974b802141bd4f13beffef2aa03f88f9" is_pre_election: true
I20260129 02:59:41.204373 22400 raft_consensus.cc:493] T 2ac659d732314e1288849f08fe5e074e P 974b802141bd4f13beffef2aa03f88f9 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 197b880fa39d432a9b472845876bfaf0)
I20260129 02:59:41.391168 22416 raft_consensus.cc:493] T 1299723a4476420ab02fd8bfd7aa0645 P 974b802141bd4f13beffef2aa03f88f9 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260129 02:59:41.400684 22501 consensus_meta.cc:287] T 87d5fe48a44a408aaf03e894a67ecb2f P 197b880fa39d432a9b472845876bfaf0: Time spent flushing consensus metadata: real 2.113s user 0.019s sys 0.014s
I20260129 02:59:41.434854 20514 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "17d6431cc77a491e8464d68db4081a51" candidate_uuid: "1c50de89a5a845668696dcb28c3badd6" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "974b802141bd4f13beffef2aa03f88f9" is_pre_election: true
I20260129 02:59:41.403307 22437 raft_consensus.cc:493] T febffe4a8b394368b69b1d164308fb9a P 974b802141bd4f13beffef2aa03f88f9 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 7a9daf5d42a84615a6383cfec9a10608)
I20260129 02:59:41.436405 20479 leader_election.cc:304] T 724a236524cb4e7ba3428880681d6cc2 P 974b802141bd4f13beffef2aa03f88f9 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 197b880fa39d432a9b472845876bfaf0, 974b802141bd4f13beffef2aa03f88f9; no voters: 1c50de89a5a845668696dcb28c3badd6
W20260129 02:59:41.354784 22387 consensus_meta.cc:287] T 42d27a28d89b409e8807ec0efd97af69 P 7a9daf5d42a84615a6383cfec9a10608: Time spent flushing consensus metadata: real 1.882s user 0.014s sys 0.020s
I20260129 02:59:41.270762 22391 leader_election.cc:290] T 2e58e9a5a8aa4cecafb464addb4f35ed P 7a9daf5d42a84615a6383cfec9a10608 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 197b880fa39d432a9b472845876bfaf0 (127.18.211.1:42027), 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3:37735)
I20260129 02:59:41.444671 20584 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2e58e9a5a8aa4cecafb464addb4f35ed" candidate_uuid: "7a9daf5d42a84615a6383cfec9a10608" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "1c50de89a5a845668696dcb28c3badd6" is_pre_election: true
I20260129 02:59:41.448920 20514 raft_consensus.cc:2468] T 17d6431cc77a491e8464d68db4081a51 P 974b802141bd4f13beffef2aa03f88f9 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1c50de89a5a845668696dcb28c3badd6 in term 2.
I20260129 02:59:41.416831 22435 raft_consensus.cc:2804] T 18f30ebaa4174fd28c958e72d0cc7e24 P 1c50de89a5a845668696dcb28c3badd6 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260129 02:59:41.450568 20584 raft_consensus.cc:2468] T 2e58e9a5a8aa4cecafb464addb4f35ed P 1c50de89a5a845668696dcb28c3badd6 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7a9daf5d42a84615a6383cfec9a10608 in term 2.
I20260129 02:59:41.446842 20510 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "75b0b6e82ddc4f92b1fe1d94f9a0b73b" candidate_uuid: "1c50de89a5a845668696dcb28c3badd6" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "974b802141bd4f13beffef2aa03f88f9" is_pre_election: true
I20260129 02:59:41.407596 22212 raft_consensus.cc:493] T 45cc5f3d2688486d9b6dbe7e666dedde P 7a9daf5d42a84615a6383cfec9a10608 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 1c50de89a5a845668696dcb28c3badd6)
I20260129 02:59:41.450770 20515 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0340229d521b432580ae926f6062e3f7" candidate_uuid: "1c50de89a5a845668696dcb28c3badd6" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "974b802141bd4f13beffef2aa03f88f9" is_pre_election: true
I20260129 02:59:41.454418 20515 raft_consensus.cc:2468] T 0340229d521b432580ae926f6062e3f7 P 974b802141bd4f13beffef2aa03f88f9 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1c50de89a5a845668696dcb28c3badd6 in term 3.
I20260129 02:59:41.453490 20510 raft_consensus.cc:2468] T 75b0b6e82ddc4f92b1fe1d94f9a0b73b P 974b802141bd4f13beffef2aa03f88f9 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1c50de89a5a845668696dcb28c3badd6 in term 2.
I20260129 02:59:41.442852 22343 raft_consensus.cc:2804] T fd8311b779af4e1fa1402b2a02a602f2 P 7a9daf5d42a84615a6383cfec9a10608 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260129 02:59:41.448063 22400 raft_consensus.cc:3060] T 2ac659d732314e1288849f08fe5e074e P 974b802141bd4f13beffef2aa03f88f9 [term 2 FOLLOWER]: Advancing to term 3
I20260129 02:59:41.443039 22499 raft_consensus.cc:2749] T 7d9494f5584d4d5db8b9adf1c8fc888b P 974b802141bd4f13beffef2aa03f88f9 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260129 02:59:41.452760 22435 raft_consensus.cc:493] T 18f30ebaa4174fd28c958e72d0cc7e24 P 1c50de89a5a845668696dcb28c3badd6 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260129 02:59:41.443794 22330 raft_consensus.cc:515] T 2e58e9a5a8aa4cecafb464addb4f35ed P 197b880fa39d432a9b472845876bfaf0 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } }
I20260129 02:59:41.460634 22343 raft_consensus.cc:493] T fd8311b779af4e1fa1402b2a02a602f2 P 7a9daf5d42a84615a6383cfec9a10608 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260129 02:59:41.510262 20513 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a5a05fbcbbcf4916bbc01fd1a5b5e90d" candidate_uuid: "1c50de89a5a845668696dcb28c3badd6" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "974b802141bd4f13beffef2aa03f88f9" is_pre_election: true
I20260129 02:59:41.450328 20509 raft_consensus.cc:2376] T a644c02b16af47679824ccd060ac10fd P 974b802141bd4f13beffef2aa03f88f9 [term 3 FOLLOWER]: Leader pre-election vote request: Already granted yes vote for candidate 1c50de89a5a845668696dcb28c3badd6 in term 3. Re-sending same reply.
W20260129 02:59:41.537809 21099 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260129 02:59:41.463799 22416 raft_consensus.cc:515] T 1299723a4476420ab02fd8bfd7aa0645 P 974b802141bd4f13beffef2aa03f88f9 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } attrs { replace: true } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } }
I20260129 02:59:41.543241 22416 leader_election.cc:290] T 1299723a4476420ab02fd8bfd7aa0645 P 974b802141bd4f13beffef2aa03f88f9 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 197b880fa39d432a9b472845876bfaf0 (127.18.211.1:42027), 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3:37735), 7a9daf5d42a84615a6383cfec9a10608 (127.18.211.4:40439)
I20260129 02:59:41.453102 22501 raft_consensus.cc:515] T 87d5fe48a44a408aaf03e894a67ecb2f P 197b880fa39d432a9b472845876bfaf0 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } }
I20260129 02:59:41.454316 22437 raft_consensus.cc:515] T febffe4a8b394368b69b1d164308fb9a P 974b802141bd4f13beffef2aa03f88f9 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } }
I20260129 02:59:41.453034 22387 raft_consensus.cc:515] T 42d27a28d89b409e8807ec0efd97af69 P 7a9daf5d42a84615a6383cfec9a10608 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } }
I20260129 02:59:41.580785 21130 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fd8311b779af4e1fa1402b2a02a602f2" candidate_uuid: "1c50de89a5a845668696dcb28c3badd6" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "7a9daf5d42a84615a6383cfec9a10608" is_pre_election: true
I20260129 02:59:41.458243 22212 raft_consensus.cc:515] T 45cc5f3d2688486d9b6dbe7e666dedde P 7a9daf5d42a84615a6383cfec9a10608 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } }
I20260129 02:59:41.582195 20438 raft_consensus.cc:1240] T 2e58e9a5a8aa4cecafb464addb4f35ed P 197b880fa39d432a9b472845876bfaf0 [term 4 FOLLOWER]: Rejecting Update request from peer 7a9daf5d42a84615a6383cfec9a10608 for earlier term 2. Current term is 4. Ops: []
I20260129 02:59:41.487821 22341 raft_consensus.cc:2749] T 42d27a28d89b409e8807ec0efd97af69 P 1c50de89a5a845668696dcb28c3badd6 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
W20260129 02:59:41.597520 20482 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260129 02:59:41.539450 22343 raft_consensus.cc:3060] T fd8311b779af4e1fa1402b2a02a602f2 P 7a9daf5d42a84615a6383cfec9a10608 [term 3 FOLLOWER]: Advancing to term 4
W20260129 02:59:41.616470 21135 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 1.372s user 0.009s sys 0.040s
W20260129 02:59:41.640705 21135 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 1.397s user 0.009s sys 0.040s
I20260129 02:59:41.549760 22036 leader_election.cc:290] T add8d4923d1a489e86882c3a25692b95 P 974b802141bd4f13beffef2aa03f88f9 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3:37735), 7a9daf5d42a84615a6383cfec9a10608 (127.18.211.4:40439)
I20260129 02:59:41.550721 22516 leader_election.cc:290] T add8d4923d1a489e86882c3a25692b95 P 1c50de89a5a845668696dcb28c3badd6 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 974b802141bd4f13beffef2aa03f88f9 (127.18.211.2:34517), 7a9daf5d42a84615a6383cfec9a10608 (127.18.211.4:40439)
I20260129 02:59:41.554680 22514 leader_election.cc:290] T fcddab6bca684adf8ea032289a00f90e P 974b802141bd4f13beffef2aa03f88f9 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 197b880fa39d432a9b472845876bfaf0 (127.18.211.1:42027), 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3:37735)
I20260129 02:59:41.486810 22435 raft_consensus.cc:3060] T 18f30ebaa4174fd28c958e72d0cc7e24 P 1c50de89a5a845668696dcb28c3badd6 [term 3 FOLLOWER]: Advancing to term 4
W20260129 02:59:41.631652 20440 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 1.741s user 0.026s sys 0.057s
I20260129 02:59:41.635056 20513 raft_consensus.cc:1275] T 1ce4ddf5b10e4ae8b193b13b2e5936ee P 974b802141bd4f13beffef2aa03f88f9 [term 4 FOLLOWER]: Refusing update from remote peer 1c50de89a5a845668696dcb28c3badd6: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20260129 02:59:41.618422 22320 raft_consensus.cc:493] T 1299723a4476420ab02fd8bfd7aa0645 P 1c50de89a5a845668696dcb28c3badd6 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 7a9daf5d42a84615a6383cfec9a10608)
W20260129 02:59:41.657478 20440 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 1.788s user 0.026s sys 0.057s
W20260129 02:59:41.660192 20405 thread.cc:641] dns-resolver [worker] (thread pool) Time spent creating pthread: real 1.295s user 0.015s sys 0.055s
W20260129 02:59:41.660622 20405 thread.cc:608] dns-resolver [worker] (thread pool) Time spent starting thread: real 1.296s user 0.015s sys 0.055s
W20260129 02:59:41.660940 20405 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.211.3:37735, user_credentials={real_user=slave}} blocked reactor thread for 1.29734e+06us
W20260129 02:59:41.662325 20405 connection.cc:232] RPC call timeout handler was delayed by -1.1855204250000497s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2391s for a response
W20260129 02:59:41.662940 21133 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 1.251s user 0.020s sys 0.020s
W20260129 02:59:41.663391 21133 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 1.251s user 0.020s sys 0.020s
W20260129 02:59:41.654346 22371 consensus_meta.cc:287] T e4d18c35b0a7493e85d0a4b92956a48f P 197b880fa39d432a9b472845876bfaf0: Time spent flushing consensus metadata: real 2.184s user 0.008s sys 0.005s
W20260129 02:59:41.656666 22522 log.cc:927] Time spent T 3906b080a5b34a74b8b64128635e8997 P 1c50de89a5a845668696dcb28c3badd6: Append to log took a long time: real 0.386s user 0.000s sys 0.001s
I20260129 02:59:41.651652 20509 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "926a250fa5834e88b96ccecbefbf5086" candidate_uuid: "1c50de89a5a845668696dcb28c3badd6" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "974b802141bd4f13beffef2aa03f88f9" is_pre_election: true
I20260129 02:59:41.650271 22231 leader_election.cc:290] T 1ce4ddf5b10e4ae8b193b13b2e5936ee P 197b880fa39d432a9b472845876bfaf0 [CANDIDATE]: Term 5 election: Requested vote from peers 974b802141bd4f13beffef2aa03f88f9 (127.18.211.2:34517), 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3:37735)
W20260129 02:59:41.699467 20507 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 1.317s user 0.017s sys 0.052s
W20260129 02:59:41.704140 20507 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 1.330s user 0.018s sys 0.052s
W20260129 02:59:41.706326 22524 log.cc:927] Time spent T 00e79d8f25a34676b1d866eb2068a2b2 P 7a9daf5d42a84615a6383cfec9a10608: Append to log took a long time: real 0.095s user 0.000s sys 0.000s
I20260129 02:59:41.665253 21128 raft_consensus.cc:1240] T 42d27a28d89b409e8807ec0efd97af69 P 7a9daf5d42a84615a6383cfec9a10608 [term 5 FOLLOWER]: Rejecting Update request from peer 197b880fa39d432a9b472845876bfaf0 for earlier term 4. Current term is 5. Ops: []
I20260129 02:59:41.722276 20588 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35754 (request call id 1595) took 1829 ms. Trace:
I20260129 02:59:41.722853 20588 rpcz_store.cc:276] 0129 02:59:39.892415 (+ 0us) service_pool.cc:168] Inserting onto call queue
0129 02:59:39.898162 (+ 5747us) service_pool.cc:225] Handling call
0129 02:59:41.144294 (+1246132us) raft_consensus.cc:1471] Updating replica for 0 ops
0129 02:59:41.189121 (+ 44827us) raft_consensus.cc:1697] Filling consensus response to leader.
0129 02:59:41.721960 (+532839us) spinlock_profiling.cc:243] Waited 1.25 s on lock 0x7b54001536d0. stack: 00007f8fe044e373 00007f8fe044e270 00007f8fdfda0574 00007f8fe9a5c6b4 00007f8fe9a5c649 00007f8fe34975a1 00007f8fe352d52d 00007f8fe84e1f49 00007f8fe1a6ed34 00007f8fe1a6ecc2 00007f8fe1a6ebf1 00007f8fe1a6eb6c 00007f8fe1a6de22 00007f8fe186d34c 00007f8fe186c786 00007f8fe186c12f
0129 02:59:41.722172 (+ 212us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2864826752}
I20260129 02:59:41.724992 20513 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4408006eb5174de1896b037cf3f18457" candidate_uuid: "1c50de89a5a845668696dcb28c3badd6" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "974b802141bd4f13beffef2aa03f88f9" is_pre_election: true
I20260129 02:59:41.690536 22416 raft_consensus.cc:2804] T 724a236524cb4e7ba3428880681d6cc2 P 974b802141bd4f13beffef2aa03f88f9 [term 2 FOLLOWER]: Leader pre-election won for term 3
W20260129 02:59:41.750043 20407 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 1.281s user 0.014s sys 0.030s
W20260129 02:59:41.750599 20407 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 1.282s user 0.014s sys 0.030s
W20260129 02:59:41.750923 20407 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.211.2:34517, user_credentials={real_user=slave}} blocked reactor thread for 1.28216e+06us
I20260129 02:59:41.664160 22320 raft_consensus.cc:515] T 1299723a4476420ab02fd8bfd7aa0645 P 1c50de89a5a845668696dcb28c3badd6 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } }
I20260129 02:59:41.697247 22371 raft_consensus.cc:515] T e4d18c35b0a7493e85d0a4b92956a48f P 197b880fa39d432a9b472845876bfaf0 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:59:41.731726 22231 raft_consensus.cc:493] T 3ccac82a01a2445fa5b0ce6c0fcf2295 P 197b880fa39d432a9b472845876bfaf0 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260129 02:59:41.734067 22416 raft_consensus.cc:493] T 724a236524cb4e7ba3428880681d6cc2 P 974b802141bd4f13beffef2aa03f88f9 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260129 02:59:41.726300 22518 raft_consensus.cc:493] T e7d83d4ccf684e76acb29a0dc844e1b8 P 197b880fa39d432a9b472845876bfaf0 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260129 02:59:41.739044 22237 raft_consensus.cc:493] T cb4727386a3e48aca5d71cc6c611c197 P 197b880fa39d432a9b472845876bfaf0 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260129 02:59:41.807371 20508 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a609ac02fae84725ab743f95f51d33ce" candidate_uuid: "197b880fa39d432a9b472845876bfaf0" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "974b802141bd4f13beffef2aa03f88f9" is_pre_election: true
I20260129 02:59:41.733784 20509 raft_consensus.cc:2468] T 926a250fa5834e88b96ccecbefbf5086 P 974b802141bd4f13beffef2aa03f88f9 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1c50de89a5a845668696dcb28c3badd6 in term 5.
I20260129 02:59:41.811089 20433 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:48356 (request call id 241) took 4449 ms. Trace:
I20260129 02:59:41.811522 20433 rpcz_store.cc:276] 0129 02:59:37.361323 (+ 0us) service_pool.cc:168] Inserting onto call queue
0129 02:59:37.366522 (+ 5199us) service_pool.cc:225] Handling call
0129 02:59:37.366927 (+ 405us) raft_consensus.cc:1471] Updating replica for 0 ops
0129 02:59:40.466279 (+3099352us) raft_consensus.cc:1697] Filling consensus response to leader.
0129 02:59:41.810831 (+1344552us) spinlock_profiling.cc:243] Waited 3.06 s on lock 0x7b5400266bd4. stack: 00007f8fe044e373 00007f8fe044e270 00007f8fdfda0574 00007f8fe9a5c6b4 00007f8fe9a5c649 00007f8fe34975a1 00007f8fe352ea81 00007f8fe352d4be 00007f8fe84e1f49 00007f8fe1a6ed34 00007f8fe1a6ecc2 00007f8fe1a6ebf1 00007f8fe1a6eb6c 00007f8fe1a6de22 00007f8fe186d34c 00007f8fe186c786
0129 02:59:41.811013 (+ 182us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":7034318592}
I20260129 02:59:41.780186 20512 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c891d2bff4ce4568861bbc992e91fbb0" candidate_uuid: "197b880fa39d432a9b472845876bfaf0" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "974b802141bd4f13beffef2aa03f88f9" is_pre_election: true
W20260129 02:59:41.780727 20406 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260129 02:59:41.813426 20406 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260129 02:59:41.785030 20516 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3906b080a5b34a74b8b64128635e8997" candidate_uuid: "197b880fa39d432a9b472845876bfaf0" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "974b802141bd4f13beffef2aa03f88f9" is_pre_election: true
W20260129 02:59:41.789685 21100 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260129 02:59:41.758280 22257 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 1.185s user 0.017s sys 0.023s
I20260129 02:59:41.806901 22330 leader_election.cc:290] T 2e58e9a5a8aa4cecafb464addb4f35ed P 197b880fa39d432a9b472845876bfaf0 [CANDIDATE]: Term 4 election: Requested vote from peers 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3:37735), 7a9daf5d42a84615a6383cfec9a10608 (127.18.211.4:40439)
I20260129 02:59:41.808104 20508 raft_consensus.cc:2468] T a609ac02fae84725ab743f95f51d33ce P 974b802141bd4f13beffef2aa03f88f9 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 197b880fa39d432a9b472845876bfaf0 in term 5.
I20260129 02:59:41.808557 20513 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b189f08e16a74a11980cc55d99359a0a" candidate_uuid: "1c50de89a5a845668696dcb28c3badd6" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "974b802141bd4f13beffef2aa03f88f9" is_pre_election: true
I20260129 02:59:41.814324 20516 raft_consensus.cc:2468] T 3906b080a5b34a74b8b64128635e8997 P 974b802141bd4f13beffef2aa03f88f9 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 197b880fa39d432a9b472845876bfaf0 in term 2.
W20260129 02:59:41.814881 20435 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 1.169s user 0.014s sys 0.013s
W20260129 02:59:41.817890 20435 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 1.172s user 0.014s sys 0.013s
I20260129 02:59:41.818187 20516 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "368953c9102643b2a21f99af36b51905" candidate_uuid: "197b880fa39d432a9b472845876bfaf0" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "974b802141bd4f13beffef2aa03f88f9"
I20260129 02:59:41.819576 20513 raft_consensus.cc:2393] T b189f08e16a74a11980cc55d99359a0a P 974b802141bd4f13beffef2aa03f88f9 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1c50de89a5a845668696dcb28c3badd6 in current term 5: Already voted for candidate 974b802141bd4f13beffef2aa03f88f9 in this term.
W20260129 02:59:41.821375 20441 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 1.165s user 0.013s sys 0.015s
W20260129 02:59:41.821983 20441 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 1.166s user 0.013s sys 0.015s
I20260129 02:59:41.823179 20512 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "17d6431cc77a491e8464d68db4081a51" candidate_uuid: "197b880fa39d432a9b472845876bfaf0" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "974b802141bd4f13beffef2aa03f88f9" is_pre_election: true
I20260129 02:59:41.823853 20512 raft_consensus.cc:2468] T 17d6431cc77a491e8464d68db4081a51 P 974b802141bd4f13beffef2aa03f88f9 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 197b880fa39d432a9b472845876bfaf0 in term 2.
I20260129 02:59:41.824707 20512 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0f20af7fbe5d4db89a6f17948d98912c" candidate_uuid: "197b880fa39d432a9b472845876bfaf0" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "974b802141bd4f13beffef2aa03f88f9" is_pre_election: true
I20260129 02:59:41.825265 20512 raft_consensus.cc:2468] T 0f20af7fbe5d4db89a6f17948d98912c P 974b802141bd4f13beffef2aa03f88f9 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 197b880fa39d432a9b472845876bfaf0 in term 4.
W20260129 02:59:41.825279 20404 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 1.212s user 0.009s sys 0.019s
W20260129 02:59:41.826946 20404 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 1.214s user 0.009s sys 0.019s
I20260129 02:59:41.788172 22416 raft_consensus.cc:3060] T 724a236524cb4e7ba3428880681d6cc2 P 974b802141bd4f13beffef2aa03f88f9 [term 2 FOLLOWER]: Advancing to term 3
I20260129 02:59:41.930384 20435 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55676 (request call id 1185) took 1410 ms. Trace:
I20260129 02:59:42.145309 20435 rpcz_store.cc:276] 0129 02:59:40.520063 (+ 0us) service_pool.cc:168] Inserting onto call queue
0129 02:59:40.571547 (+ 51484us) service_pool.cc:225] Handling call
0129 02:59:40.571876 (+ 329us) raft_consensus.cc:1471] Updating replica for 1 ops
0129 02:59:40.613271 (+ 41395us) raft_consensus.cc:1534] Early marking committed up to index 0
0129 02:59:40.613302 (+ 31us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0129 02:59:40.613746 (+ 444us) log.cc:844] Serialized 26 byte log entry
0129 02:59:41.818312 (+1204566us) raft_consensus.cc:1645] Marking committed up to 1
0129 02:59:41.818354 (+ 42us) raft_consensus.cc:1697] Filling consensus response to leader.
0129 02:59:41.818422 (+ 68us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0129 02:59:41.929993 (+111571us) raft_consensus.cc:1685] finished
0129 02:59:41.930031 (+ 38us) raft_consensus.cc:1691] UpdateReplicas() finished
0129 02:59:41.930281 (+ 250us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":35987,"raft.queue_time_us":80064,"raft.run_cpu_time_us":156,"raft.run_wall_time_us":2316,"thread_start_us":1172046,"threads_started":1,"wal-append.queue_time_us":1282011}
W20260129 02:59:41.858181 21100 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260129 02:59:42.085237 20432 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 1.392s user 0.014s sys 0.060s
W20260129 02:59:42.149480 20432 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 1.456s user 0.014s sys 0.065s
W20260129 02:59:41.836699 20405 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.211.3:37735, user_credentials={real_user=slave}} blocked reactor thread for 100980us
W20260129 02:59:41.815387 22257 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 1.242s user 0.018s sys 0.024s
W20260129 02:59:42.099853 22364 consensus_meta.cc:287] T 0678b8b468bd43cdbc3e485900844821 P 7a9daf5d42a84615a6383cfec9a10608: Time spent flushing consensus metadata: real 0.889s user 0.010s sys 0.003s
I20260129 02:59:41.835150 20516 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00e79d8f25a34676b1d866eb2068a2b2" candidate_uuid: "197b880fa39d432a9b472845876bfaf0" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "974b802141bd4f13beffef2aa03f88f9"
W20260129 02:59:42.140226 21129 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 1.471s user 0.011s sys 0.038s
I20260129 02:59:41.984383 20513 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "cedce331e1584ac8b636c9b8a72b2ba0" candidate_uuid: "197b880fa39d432a9b472845876bfaf0" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "974b802141bd4f13beffef2aa03f88f9"
W20260129 02:59:42.108798 21099 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260129 02:59:41.857749 20433 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4e3805ffa70242e8ade66da260dddf66" candidate_uuid: "7a9daf5d42a84615a6383cfec9a10608" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "197b880fa39d432a9b472845876bfaf0" is_pre_election: true
I20260129 02:59:42.114264 20512 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4f8f3a38f6e24ba4857875181f447442" candidate_uuid: "197b880fa39d432a9b472845876bfaf0" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "974b802141bd4f13beffef2aa03f88f9" is_pre_election: true
I20260129 02:59:42.085237 20508 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a5a05fbcbbcf4916bbc01fd1a5b5e90d" candidate_uuid: "197b880fa39d432a9b472845876bfaf0" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "974b802141bd4f13beffef2aa03f88f9" is_pre_election: true
I20260129 02:59:41.981590 21136 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:47844 (request call id 288) took 1190 ms. Trace:
W20260129 02:59:42.069274 22528 log.cc:927] Time spent T cb0b3f259ebc4310927338a8740e87b8 P 7a9daf5d42a84615a6383cfec9a10608: Append to log took a long time: real 0.298s user 0.000s sys 0.001s
I20260129 02:59:41.807814 22237 raft_consensus.cc:515] T cb4727386a3e48aca5d71cc6c611c197 P 197b880fa39d432a9b472845876bfaf0 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
W20260129 02:59:42.075385 22527 log.cc:927] Time spent T 87d5fe48a44a408aaf03e894a67ecb2f P 974b802141bd4f13beffef2aa03f88f9: Append to log took a long time: real 0.338s user 0.001s sys 0.000s
W20260129 02:59:42.076603 22337 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 1.346s user 0.020s sys 0.053s
W20260129 02:59:42.076157 22529 log.cc:927] Time spent T 1299723a4476420ab02fd8bfd7aa0645 P 1c50de89a5a845668696dcb28c3badd6: Append to log took a long time: real 0.339s user 0.001s sys 0.000s
W20260129 02:59:42.082433 22400 consensus_meta.cc:287] T 2ac659d732314e1288849f08fe5e074e P 974b802141bd4f13beffef2aa03f88f9: Time spent flushing consensus metadata: real 0.601s user 0.009s sys 0.000s
W20260129 02:59:41.937517 21102 leader_election.cc:336] T add8d4923d1a489e86882c3a25692b95 P 7a9daf5d42a84615a6383cfec9a10608 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 974b802141bd4f13beffef2aa03f88f9 (127.18.211.2:34517): Timed out: RequestConsensusVote RPC to 127.18.211.2:34517 timed out after 1.605s (SENT)
W20260129 02:59:42.247192 21102 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.211.2:34517, user_credentials={real_user=slave}} blocked reactor thread for 309715us
I20260129 02:59:41.822675 22501 leader_election.cc:290] T 87d5fe48a44a408aaf03e894a67ecb2f P 197b880fa39d432a9b472845876bfaf0 [CANDIDATE]: Term 5 election: Requested vote from peers 974b802141bd4f13beffef2aa03f88f9 (127.18.211.2:34517), 7a9daf5d42a84615a6383cfec9a10608 (127.18.211.4:40439)
I20260129 02:59:42.250067 20508 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c2633a6fc34c44579de348d208b0c01e" candidate_uuid: "197b880fa39d432a9b472845876bfaf0" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "974b802141bd4f13beffef2aa03f88f9" is_pre_election: true
I20260129 02:59:42.156098 20509 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a1a39dcd328e4268ae41b988c66b4a75" candidate_uuid: "197b880fa39d432a9b472845876bfaf0" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "974b802141bd4f13beffef2aa03f88f9"
W20260129 02:59:42.203353 21134 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 1.535s user 0.047s sys 0.030s
W20260129 02:59:42.257907 21134 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 1.590s user 0.047s sys 0.030s
W20260129 02:59:42.210793 21100 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260129 02:59:42.111986 22212 leader_election.cc:290] T 45cc5f3d2688486d9b6dbe7e666dedde P 7a9daf5d42a84615a6383cfec9a10608 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 974b802141bd4f13beffef2aa03f88f9 (127.18.211.2:34517), 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3:37735)
I20260129 02:59:41.964629 22520 raft_consensus.cc:493] T 75be4bd6bd45474795fd472eff2953ac P 197b880fa39d432a9b472845876bfaf0 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 1c50de89a5a845668696dcb28c3badd6)
W20260129 02:59:42.267931 20406 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260129 02:59:42.245734 22337 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 1.509s user 0.022s sys 0.056s
I20260129 02:59:41.929363 22374 raft_consensus.cc:493] T 3e2d15f8640b407195210c476e09df91 P 7a9daf5d42a84615a6383cfec9a10608 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 1c50de89a5a845668696dcb28c3badd6)
I20260129 02:59:41.776775 22231 raft_consensus.cc:515] T 3ccac82a01a2445fa5b0ce6c0fcf2295 P 197b880fa39d432a9b472845876bfaf0 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } }
I20260129 02:59:42.236263 20516 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e7d83d4ccf684e76acb29a0dc844e1b8" candidate_uuid: "1c50de89a5a845668696dcb28c3badd6" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "974b802141bd4f13beffef2aa03f88f9" is_pre_election: true
I20260129 02:59:42.283761 20516 raft_consensus.cc:2468] T e7d83d4ccf684e76acb29a0dc844e1b8 P 974b802141bd4f13beffef2aa03f88f9 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1c50de89a5a845668696dcb28c3badd6 in term 3.
I20260129 02:59:42.236999 20435 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "42d27a28d89b409e8807ec0efd97af69" candidate_uuid: "7a9daf5d42a84615a6383cfec9a10608" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "197b880fa39d432a9b472845876bfaf0" is_pre_election: true
I20260129 02:59:42.284471 20513 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e33a417cf9c54cf5af73a4ee3aac4d3a" candidate_uuid: "1c50de89a5a845668696dcb28c3badd6" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "974b802141bd4f13beffef2aa03f88f9" is_pre_election: true
I20260129 02:59:42.239245 20512 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "8055232cf0004d40a85f3bf983fe00a9" candidate_uuid: "1c50de89a5a845668696dcb28c3badd6" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "974b802141bd4f13beffef2aa03f88f9" is_pre_election: true
I20260129 02:59:41.810871 22518 raft_consensus.cc:515] T e7d83d4ccf684e76acb29a0dc844e1b8 P 197b880fa39d432a9b472845876bfaf0 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:59:42.296929 20516 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a8479d1fe0bd41e5946186e3a6ca8c7a" candidate_uuid: "1c50de89a5a845668696dcb28c3badd6" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "974b802141bd4f13beffef2aa03f88f9" is_pre_election: true
I20260129 02:59:41.985373 22387 leader_election.cc:290] T 42d27a28d89b409e8807ec0efd97af69 P 7a9daf5d42a84615a6383cfec9a10608 [CANDIDATE]: Term 5 election: Requested vote from peers 197b880fa39d432a9b472845876bfaf0 (127.18.211.1:42027), 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3:37735)
W20260129 02:59:42.240013 20440 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.549s user 0.023s sys 0.032s
W20260129 02:59:42.502209 20440 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.812s user 0.023s sys 0.032s
W20260129 02:59:42.243814 21131 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34060 (request call id 467) took 3604 ms (client timeout 1706 ms). Trace:
W20260129 02:59:42.502923 21131 rpcz_store.cc:269] 0129 02:59:38.639035 (+ 0us) service_pool.cc:168] Inserting onto call queue
0129 02:59:38.659891 (+ 20856us) service_pool.cc:225] Handling call
0129 02:59:42.243531 (+3583640us) spinlock_profiling.cc:243] Waited 1.98 s on lock 0x7b5400670cd4. stack: 00007f8fe044e373 00007f8fe044e270 00007f8fdfda0574 00007f8fe9a5c6b4 00007f8fe9a5c649 00007f8fe34975a1 00007f8fe3531f7a 00007f8fe84e3647 00007f8fe1a73834 00007f8fe1a737c2 00007f8fe1a736f1 00007f8fe1a7366c 00007f8fe1a72922 00007f8fe186d34c 00007f8fe186c786 00007f8fe186c12f
0129 02:59:42.243718 (+ 187us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":4564409728}
W20260129 02:59:42.249017 21102 leader_election.cc:336] T cb0b3f259ebc4310927338a8740e87b8 P 7a9daf5d42a84615a6383cfec9a10608 [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer 974b802141bd4f13beffef2aa03f88f9 (127.18.211.2:34517): Timed out: RequestConsensusVote RPC to 127.18.211.2:34517 timed out after 1.774s (SENT)
W20260129 02:59:42.504868 21102 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.211.2:34517, user_credentials={real_user=slave}} blocked reactor thread for 255850us
W20260129 02:59:42.506301 21102 leader_election.cc:336] T add8d4923d1a489e86882c3a25692b95 P 7a9daf5d42a84615a6383cfec9a10608 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 974b802141bd4f13beffef2aa03f88f9 (127.18.211.2:34517): Timed out: RequestConsensusVote RPC to 127.18.211.2:34517 timed out after 1.824s (SENT)
I20260129 02:59:42.251228 20508 raft_consensus.cc:2410] T c2633a6fc34c44579de348d208b0c01e P 974b802141bd4f13beffef2aa03f88f9 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 197b880fa39d432a9b472845876bfaf0 for term 3 because replica has last-logged OpId of term: 2 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 1.
W20260129 02:59:41.840144 22521 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 1.174s user 0.007s sys 0.022s
W20260129 02:59:42.224386 21129 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 1.562s user 0.012s sys 0.039s
I20260129 02:59:42.252972 20433 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3ccac82a01a2445fa5b0ce6c0fcf2295" candidate_uuid: "7a9daf5d42a84615a6383cfec9a10608" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "197b880fa39d432a9b472845876bfaf0" is_pre_election: true
I20260129 02:59:42.525381 20433 raft_consensus.cc:2468] T 3ccac82a01a2445fa5b0ce6c0fcf2295 P 197b880fa39d432a9b472845876bfaf0 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7a9daf5d42a84615a6383cfec9a10608 in term 2.
I20260129 02:59:42.152547 22440 raft_consensus.cc:493] T 724a236524cb4e7ba3428880681d6cc2 P 197b880fa39d432a9b472845876bfaf0 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 1c50de89a5a845668696dcb28c3badd6)
I20260129 02:59:41.931669 22330 consensus_queue.cc:1048] T 42d27a28d89b409e8807ec0efd97af69 P 197b880fa39d432a9b472845876bfaf0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.304s
W20260129 02:59:42.472352 22137 consensus_meta.cc:287] T cb4727386a3e48aca5d71cc6c611c197 P 974b802141bd4f13beffef2aa03f88f9: Time spent flushing consensus metadata: real 1.681s user 0.010s sys 0.003s
I20260129 02:59:42.220350 22320 leader_election.cc:290] T 1299723a4476420ab02fd8bfd7aa0645 P 1c50de89a5a845668696dcb28c3badd6 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 197b880fa39d432a9b472845876bfaf0 (127.18.211.1:42027), 7a9daf5d42a84615a6383cfec9a10608 (127.18.211.4:40439)
I20260129 02:59:42.230453 21136 rpcz_store.cc:276] 0129 02:59:40.791050 (+ 0us) service_pool.cc:168] Inserting onto call queue
0129 02:59:40.791239 (+ 189us) service_pool.cc:225] Handling call
0129 02:59:41.981276 (+1190037us) spinlock_profiling.cc:243] Waited 248 ms on lock 0x7b540062f554. stack: 00007f8fe044e373 00007f8fe044e270 00007f8fdfda0574 00007f8fe9a5c6b4 00007f8fe9a5c649 00007f8fe34975a1 00007f8fe3531f7a 00007f8fe84e3647 00007f8fe1a73834 00007f8fe1a737c2 00007f8fe1a736f1 00007f8fe1a7366c 00007f8fe1a72922 00007f8fe186d34c 00007f8fe186c786 00007f8fe186c12f
0129 02:59:41.981490 (+ 214us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":571299712}
I20260129 02:59:42.281769 20405 leader_election.cc:304] T 7d9494f5584d4d5db8b9adf1c8fc888b P 197b880fa39d432a9b472845876bfaf0 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 197b880fa39d432a9b472845876bfaf0; no voters: 1c50de89a5a845668696dcb28c3badd6, 974b802141bd4f13beffef2aa03f88f9
W20260129 02:59:42.421651 20589 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 1.172s user 0.060s sys 0.063s
I20260129 02:59:42.452450 20435 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "78aa39096868415d9afc2ffcb248e198" candidate_uuid: "7a9daf5d42a84615a6383cfec9a10608" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "197b880fa39d432a9b472845876bfaf0"
W20260129 02:59:42.704286 20589 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 1.455s user 0.063s sys 0.071s
I20260129 02:59:42.704671 20435 raft_consensus.cc:2393] T 78aa39096868415d9afc2ffcb248e198 P 197b880fa39d432a9b472845876bfaf0 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 7a9daf5d42a84615a6383cfec9a10608 in current term 4: Already voted for candidate 197b880fa39d432a9b472845876bfaf0 in this term.
W20260129 02:59:42.452801 21099 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.211.1:42027, user_credentials={real_user=slave}} blocked reactor thread for 59210.8us
I20260129 02:59:42.453253 20516 raft_consensus.cc:2468] T a8479d1fe0bd41e5946186e3a6ca8c7a P 974b802141bd4f13beffef2aa03f88f9 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1c50de89a5a845668696dcb28c3badd6 in term 3.
I20260129 02:59:42.707636 20440 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55676 (request call id 1180) took 2851 ms. Trace:
I20260129 02:59:42.708168 20440 rpcz_store.cc:276] 0129 02:59:39.858760 (+ 0us) service_pool.cc:168] Inserting onto call queue
0129 02:59:39.858908 (+ 148us) service_pool.cc:225] Handling call
0129 02:59:39.859312 (+ 404us) raft_consensus.cc:1471] Updating replica for 0 ops
0129 02:59:39.859562 (+ 250us) raft_consensus.cc:1534] Early marking committed up to index 2
0129 02:59:41.664499 (+1804937us) log.cc:844] Serialized 14 byte log entry
0129 02:59:42.707236 (+1042737us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0129 02:59:42.707300 (+ 64us) raft_consensus.cc:1645] Marking committed up to 2
0129 02:59:42.707363 (+ 63us) raft_consensus.cc:1697] Filling consensus response to leader.
0129 02:59:42.707423 (+ 60us) raft_consensus.cc:1691] UpdateReplicas() finished
0129 02:59:42.707556 (+ 133us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":282582,"prepare.queue_time_us":2379688,"prepare.run_cpu_time_us":65,"prepare.run_wall_time_us":66,"thread_start_us":2626333,"threads_started":2}
I20260129 02:59:42.256780 22400 raft_consensus.cc:515] T 2ac659d732314e1288849f08fe5e074e P 974b802141bd4f13beffef2aa03f88f9 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } }
W20260129 02:59:42.453944 20406 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260129 02:59:42.724442 20406 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260129 02:59:42.724687 20405 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.211.3:37735, user_credentials={real_user=slave}} blocked reactor thread for 442992us
I20260129 02:59:42.725492 20405 leader_election.cc:304] T e33a417cf9c54cf5af73a4ee3aac4d3a P 197b880fa39d432a9b472845876bfaf0 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 197b880fa39d432a9b472845876bfaf0; no voters: 1c50de89a5a845668696dcb28c3badd6, 974b802141bd4f13beffef2aa03f88f9
W20260129 02:59:42.726392 21099 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260129 02:59:42.396481 20556 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.994s user 0.020s sys 0.036s
W20260129 02:59:42.458834 20480 thread.cc:641] dns-resolver [worker] (thread pool) Time spent creating pthread: real 1.238s user 0.017s sys 0.055s
W20260129 02:59:42.454392 22432 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 1.245s user 0.045s sys 0.026s
I20260129 02:59:42.297726 22520 raft_consensus.cc:515] T 75be4bd6bd45474795fd472eff2953ac P 197b880fa39d432a9b472845876bfaf0 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
W20260129 02:59:42.648929 22521 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 1.985s user 0.008s sys 0.025s
I20260129 02:59:42.478147 20581 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:48526 (request call id 184) took 2690 ms. Trace:
I20260129 02:59:42.769367 20581 rpcz_store.cc:276] 0129 02:59:39.787530 (+ 0us) service_pool.cc:168] Inserting onto call queue
0129 02:59:39.787671 (+ 141us) service_pool.cc:225] Handling call
0129 02:59:39.788032 (+ 361us) raft_consensus.cc:1471] Updating replica for 2 ops
0129 02:59:39.800718 (+ 12686us) raft_consensus.cc:1534] Early marking committed up to index 4
0129 02:59:39.800747 (+ 29us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0129 02:59:39.809545 (+ 8798us) log.cc:844] Serialized 52 byte log entry
0129 02:59:41.318483 (+1508938us) raft_consensus.cc:1645] Marking committed up to 4
0129 02:59:41.318559 (+ 76us) raft_consensus.cc:1697] Filling consensus response to leader.
0129 02:59:41.318660 (+ 101us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0129 02:59:42.477870 (+1159210us) raft_consensus.cc:1685] finished
0129 02:59:42.477896 (+ 26us) raft_consensus.cc:1691] UpdateReplicas() finished
0129 02:59:42.478071 (+ 175us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":12230,"raft.queue_time_us":12359,"raft.run_cpu_time_us":223,"raft.run_wall_time_us":37538,"thread_start_us":1508476,"threads_started":1,"wal-append.queue_time_us":1854562}
W20260129 02:59:42.496729 20553 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 1.082s user 0.000s sys 0.060s
W20260129 02:59:42.771183 20553 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 1.357s user 0.000s sys 0.060s
W20260129 02:59:42.771797 20553 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.211.1:42027, user_credentials={real_user=slave}} blocked reactor thread for 1.37426e+06us
I20260129 02:59:42.772550 20553 leader_election.cc:304] T 17d6431cc77a491e8464d68db4081a51 P 1c50de89a5a845668696dcb28c3badd6 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 197b880fa39d432a9b472845876bfaf0, 1c50de89a5a845668696dcb28c3badd6; no voters:
I20260129 02:59:42.775074 20553 leader_election.cc:304] T dc2f4cb752cf4d59b5f6bc689f5d8104 P 1c50de89a5a845668696dcb28c3badd6 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 197b880fa39d432a9b472845876bfaf0, 1c50de89a5a845668696dcb28c3badd6; no voters: 7a9daf5d42a84615a6383cfec9a10608
W20260129 02:59:42.350175 20479 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.900s user 0.000s sys 0.043s
W20260129 02:59:42.776083 20479 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 1.326s user 0.004s sys 0.047s
W20260129 02:59:42.776791 20479 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.211.1:42027, user_credentials={real_user=slave}} blocked reactor thread for 1.34034e+06us
I20260129 02:59:42.778294 20479 leader_election.cc:304] T 4f8f3a38f6e24ba4857875181f447442 P 974b802141bd4f13beffef2aa03f88f9 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 974b802141bd4f13beffef2aa03f88f9; no voters: 197b880fa39d432a9b472845876bfaf0, 1c50de89a5a845668696dcb28c3badd6
W20260129 02:59:42.649175 20481 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260129 02:59:42.151923 22198 raft_consensus.cc:493] T 913fc7d751ed4b7388d889aac30a5f9b P 197b880fa39d432a9b472845876bfaf0 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260129 02:59:42.452795 22184 raft_consensus.cc:493] T 3e2d15f8640b407195210c476e09df91 P 974b802141bd4f13beffef2aa03f88f9 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 1c50de89a5a845668696dcb28c3badd6)
I20260129 02:59:42.711328 20433 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4e420f90884f4796997b5ea41e49e411" candidate_uuid: "7a9daf5d42a84615a6383cfec9a10608" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "197b880fa39d432a9b472845876bfaf0" is_pre_election: true
I20260129 02:59:42.326033 22374 raft_consensus.cc:515] T 3e2d15f8640b407195210c476e09df91 P 7a9daf5d42a84615a6383cfec9a10608 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } }
I20260129 02:59:42.748456 20511 raft_consensus.cc:2468] T cb4727386a3e48aca5d71cc6c611c197 P 974b802141bd4f13beffef2aa03f88f9 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1c50de89a5a845668696dcb28c3badd6 in term 2.
I20260129 02:59:42.750015 20440 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e33a417cf9c54cf5af73a4ee3aac4d3a" candidate_uuid: "1c50de89a5a845668696dcb28c3badd6" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "197b880fa39d432a9b472845876bfaf0" is_pre_election: true
W20260129 02:59:42.727211 20556 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 1.325s user 0.020s sys 0.040s
W20260129 02:59:42.751149 20480 thread.cc:608] dns-resolver [worker] (thread pool) Time spent starting thread: real 1.531s user 0.018s sys 0.057s
I20260129 02:59:42.751760 21136 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3ccac82a01a2445fa5b0ce6c0fcf2295" candidate_uuid: "974b802141bd4f13beffef2aa03f88f9" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "7a9daf5d42a84615a6383cfec9a10608" is_pre_election: true
I20260129 02:59:42.753412 20435 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4e3805ffa70242e8ade66da260dddf66" candidate_uuid: "7a9daf5d42a84615a6383cfec9a10608" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "197b880fa39d432a9b472845876bfaf0" is_pre_election: true
W20260129 02:59:42.650128 22160 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 1.261s user 0.019s sys 0.042s
W20260129 02:59:42.753988 20555 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.977s user 0.046s sys 0.060s
W20260129 02:59:42.800205 20555 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 1.383s user 0.046s sys 0.061s
W20260129 02:59:42.800454 20480 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.211.3:37735, user_credentials={real_user=slave}} blocked reactor thread for 1.58012e+06us
I20260129 02:59:42.801549 20480 leader_election.cc:304] T e4d18c35b0a7493e85d0a4b92956a48f P 974b802141bd4f13beffef2aa03f88f9 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1c50de89a5a845668696dcb28c3badd6, 974b802141bd4f13beffef2aa03f88f9; no voters:
I20260129 02:59:42.753927 21131 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4e420f90884f4796997b5ea41e49e411" candidate_uuid: "974b802141bd4f13beffef2aa03f88f9" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "7a9daf5d42a84615a6383cfec9a10608" is_pre_election: true
I20260129 02:59:42.754372 22400 leader_election.cc:290] T 2ac659d732314e1288849f08fe5e074e P 974b802141bd4f13beffef2aa03f88f9 [CANDIDATE]: Term 3 election: Requested vote from peers 197b880fa39d432a9b472845876bfaf0 (127.18.211.1:42027), 7a9daf5d42a84615a6383cfec9a10608 (127.18.211.4:40439)
I20260129 02:59:42.771528 22336 raft_consensus.cc:2749] T 7d9494f5584d4d5db8b9adf1c8fc888b P 197b880fa39d432a9b472845876bfaf0 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20260129 02:59:42.761397 20590 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 1.317s user 0.037s sys 0.065s
W20260129 02:59:42.425509 22408 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.991s user 0.024s sys 0.053s
W20260129 02:59:42.817977 20590 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 1.387s user 0.037s sys 0.065s
W20260129 02:59:42.767263 20585 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 1.349s user 0.022s sys 0.070s
I20260129 02:59:42.790233 20479 leader_election.cc:304] T 36ff33be666a4c82954731691dbcf327 P 974b802141bd4f13beffef2aa03f88f9 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 974b802141bd4f13beffef2aa03f88f9; no voters: 197b880fa39d432a9b472845876bfaf0, 1c50de89a5a845668696dcb28c3badd6
I20260129 02:59:42.687340 22371 leader_election.cc:290] T e4d18c35b0a7493e85d0a4b92956a48f P 197b880fa39d432a9b472845876bfaf0 [CANDIDATE]: Term 4 election: Requested vote from peers 974b802141bd4f13beffef2aa03f88f9 (127.18.211.2:34517), 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3:37735)
W20260129 02:59:42.764110 22432 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 1.555s user 0.046s sys 0.027s
I20260129 02:59:42.799393 21136 raft_consensus.cc:2468] T 3ccac82a01a2445fa5b0ce6c0fcf2295 P 7a9daf5d42a84615a6383cfec9a10608 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 974b802141bd4f13beffef2aa03f88f9 in term 1.
I20260129 02:59:42.840725 20582 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "42d27a28d89b409e8807ec0efd97af69" candidate_uuid: "7a9daf5d42a84615a6383cfec9a10608" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "1c50de89a5a845668696dcb28c3badd6"
W20260129 02:59:42.806434 20556 connection.cc:232] RPC call timeout handler was delayed by -1.502025351000043s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.19930000000000003s for a response
W20260129 02:59:42.841594 20556 connection.cc:232] RPC call timeout handler was delayed by -1.4948253510000313s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2001s for a response
I20260129 02:59:42.841629 20582 raft_consensus.cc:3060] T 42d27a28d89b409e8807ec0efd97af69 P 1c50de89a5a845668696dcb28c3badd6 [term 4 FOLLOWER]: Advancing to term 5
W20260129 02:59:42.758533 21100 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260129 02:59:42.800472 22320 raft_consensus.cc:2804] T 17d6431cc77a491e8464d68db4081a51 P 1c50de89a5a845668696dcb28c3badd6 [term 2 FOLLOWER]: Leader pre-election won for term 3
W20260129 02:59:42.809212 21131 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:47844 (request call id 294) took 1761 ms (client timeout 2218 ms). Trace:
W20260129 02:59:42.844151 21131 rpcz_store.cc:269] 0129 02:59:41.048348 (+ 0us) service_pool.cc:168] Inserting onto call queue
0129 02:59:42.700963 (+1652615us) service_pool.cc:225] Handling call
0129 02:59:42.809152 (+108189us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20260129 02:59:42.817003 21133 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 1.152s user 0.073s sys 0.189s
I20260129 02:59:42.766872 22364 raft_consensus.cc:493] T 0678b8b468bd43cdbc3e485900844821 P 7a9daf5d42a84615a6383cfec9a10608 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260129 02:59:42.799968 22160 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 1.563s user 0.019s sys 0.042s
W20260129 02:59:42.821256 20515 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 1.210s user 0.039s sys 0.083s
W20260129 02:59:42.855917 20515 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 1.245s user 0.039s sys 0.083s
I20260129 02:59:42.701511 22440 raft_consensus.cc:515] T 724a236524cb4e7ba3428880681d6cc2 P 197b880fa39d432a9b472845876bfaf0 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
W20260129 02:59:42.797289 22502 consensus_meta.cc:287] T a0fcc58d6fb6473cafbd9d34d5195269 P 197b880fa39d432a9b472845876bfaf0: Time spent flushing consensus metadata: real 1.983s user 0.010s sys 0.004s
I20260129 02:59:42.824154 20587 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "45cc5f3d2688486d9b6dbe7e666dedde" candidate_uuid: "7a9daf5d42a84615a6383cfec9a10608" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "1c50de89a5a845668696dcb28c3badd6" is_pre_election: true
I20260129 02:59:42.794072 22521 raft_consensus.cc:2749] T e33a417cf9c54cf5af73a4ee3aac4d3a P 197b880fa39d432a9b472845876bfaf0 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
W20260129 02:59:42.827545 20585 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 1.410s user 0.022s sys 0.070s
I20260129 02:59:42.822129 22445 raft_consensus.cc:493] T cedce331e1584ac8b636c9b8a72b2ba0 P 974b802141bd4f13beffef2aa03f88f9 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 1c50de89a5a845668696dcb28c3badd6)
I20260129 02:59:42.865903 20585 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:48526 (request call id 197) took 1450 ms. Trace:
I20260129 02:59:42.809542 22425 consensus_queue.cc:1059] T cedce331e1584ac8b636c9b8a72b2ba0 P 1c50de89a5a845668696dcb28c3badd6 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 }, Status: INVALID_TERM, Last received: 4.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.053s
I20260129 02:59:42.799664 22197 raft_consensus.cc:493] T cb4727386a3e48aca5d71cc6c611c197 P 974b802141bd4f13beffef2aa03f88f9 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260129 02:59:42.800789 22184 raft_consensus.cc:515] T 3e2d15f8640b407195210c476e09df91 P 974b802141bd4f13beffef2aa03f88f9 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } }
W20260129 02:59:42.845149 21133 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 1.181s user 0.073s sys 0.189s
I20260129 02:59:42.870216 20585 rpcz_store.cc:276] 0129 02:59:41.415994 (+ 0us) service_pool.cc:168] Inserting onto call queue
0129 02:59:41.417126 (+ 1132us) service_pool.cc:225] Handling call
0129 02:59:41.417426 (+ 300us) raft_consensus.cc:1471] Updating replica for 0 ops
0129 02:59:42.865513 (+1448087us) raft_consensus.cc:1534] Early marking committed up to index 0
0129 02:59:42.865546 (+ 33us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0129 02:59:42.865590 (+ 44us) raft_consensus.cc:1645] Marking committed up to 4
0129 02:59:42.865629 (+ 39us) raft_consensus.cc:1697] Filling consensus response to leader.
0129 02:59:42.865711 (+ 82us) raft_consensus.cc:1691] UpdateReplicas() finished
0129 02:59:42.865844 (+ 133us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":234214,"raft.run_cpu_time_us":150,"raft.run_wall_time_us":6032,"thread_start_us":1447716,"threads_started":1}
W20260129 02:59:42.818395 21135 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 1.176s user 0.029s sys 0.087s
W20260129 02:59:42.877600 21135 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 1.236s user 0.029s sys 0.088s
I20260129 02:59:42.843470 21100 leader_election.cc:304] T 2e58e9a5a8aa4cecafb464addb4f35ed P 7a9daf5d42a84615a6383cfec9a10608 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1c50de89a5a845668696dcb28c3badd6, 7a9daf5d42a84615a6383cfec9a10608; no voters:
W20260129 02:59:42.821053 22408 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 1.401s user 0.024s sys 0.054s
I20260129 02:59:42.820554 22504 raft_consensus.cc:2804] T dc2f4cb752cf4d59b5f6bc689f5d8104 P 1c50de89a5a845668696dcb28c3badd6 [term 3 FOLLOWER]: Leader pre-election won for term 4
W20260129 02:59:42.873291 21099 leader_election.cc:336] T 4e3805ffa70242e8ade66da260dddf66 P 7a9daf5d42a84615a6383cfec9a10608 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 197b880fa39d432a9b472845876bfaf0 (127.18.211.1:42027): Timed out: RequestConsensusVote RPC to 127.18.211.1:42027 timed out after 1.674s (SENT)
I20260129 02:59:42.884522 21099 leader_election.cc:304] T 4e3805ffa70242e8ade66da260dddf66 P 7a9daf5d42a84615a6383cfec9a10608 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7a9daf5d42a84615a6383cfec9a10608; no voters: 197b880fa39d432a9b472845876bfaf0, 1c50de89a5a845668696dcb28c3badd6
I20260129 02:59:42.844656 22400 raft_consensus.cc:2804] T e4d18c35b0a7493e85d0a4b92956a48f P 974b802141bd4f13beffef2aa03f88f9 [term 3 FOLLOWER]: Leader pre-election won for term 4
W20260129 02:59:42.840945 21136 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:47844 (request call id 292) took 1826 ms (client timeout 2280 ms). Trace:
I20260129 02:59:42.838214 22237 leader_election.cc:290] T cb4727386a3e48aca5d71cc6c611c197 P 197b880fa39d432a9b472845876bfaf0 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 974b802141bd4f13beffef2aa03f88f9 (127.18.211.2:34517), 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3:37735)
I20260129 02:59:42.797428 22198 raft_consensus.cc:515] T 913fc7d751ed4b7388d889aac30a5f9b P 197b880fa39d432a9b472845876bfaf0 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:59:42.811162 22344 consensus_queue.cc:1048] T 913fc7d751ed4b7388d889aac30a5f9b P 1c50de89a5a845668696dcb28c3badd6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.050s
I20260129 02:59:42.879949 21131 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "add8d4923d1a489e86882c3a25692b95" candidate_uuid: "1c50de89a5a845668696dcb28c3badd6" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "7a9daf5d42a84615a6383cfec9a10608" is_pre_election: true
I20260129 02:59:42.897572 21131 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34046 (request call id 434) took 1345 ms. Trace:
I20260129 02:59:42.897910 21131 rpcz_store.cc:276] 0129 02:59:41.552440 (+ 0us) service_pool.cc:168] Inserting onto call queue
0129 02:59:42.856133 (+1303693us) service_pool.cc:225] Handling call
0129 02:59:42.897506 (+ 41373us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260129 02:59:42.844962 21132 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34046 (request call id 430) took 2292 ms. Trace:
I20260129 02:59:42.819079 22103 consensus_queue.cc:1048] T 7a9a69cd82724b1faed1d10aa091fc4a P 1c50de89a5a845668696dcb28c3badd6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.046s
I20260129 02:59:42.852104 22320 raft_consensus.cc:493] T 17d6431cc77a491e8464d68db4081a51 P 1c50de89a5a845668696dcb28c3badd6 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260129 02:59:42.891693 21136 rpcz_store.cc:269] 0129 02:59:41.014635 (+ 0us) service_pool.cc:168] Inserting onto call queue
0129 02:59:42.751408 (+1736773us) service_pool.cc:225] Handling call
0129 02:59:42.840860 (+ 89452us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260129 02:59:42.869609 22540 raft_consensus.cc:3055] T c934d31efd3f4e5195605ddc354d14e1 P 197b880fa39d432a9b472845876bfaf0 [term 3 LEADER]: Stepping down as leader of term 3
I20260129 02:59:42.820056 22506 consensus_queue.cc:1059] T 78aa39096868415d9afc2ffcb248e198 P 1c50de89a5a845668696dcb28c3badd6 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 }, Status: INVALID_TERM, Last received: 3.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.059s
I20260129 02:59:42.839489 22404 consensus_queue.cc:1048] T a5a05fbcbbcf4916bbc01fd1a5b5e90d P 974b802141bd4f13beffef2aa03f88f9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.040s
I20260129 02:59:42.855919 22213 consensus_queue.cc:1059] T 1ce4ddf5b10e4ae8b193b13b2e5936ee P 1c50de89a5a845668696dcb28c3badd6 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 }, Status: INVALID_TERM, Last received: 2.2, Next index: 3, Last known committed idx: 1, Time since last communication: 0.034s
I20260129 02:59:42.861110 22364 raft_consensus.cc:515] T 0678b8b468bd43cdbc3e485900844821 P 7a9daf5d42a84615a6383cfec9a10608 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } }
I20260129 02:59:42.891647 22400 raft_consensus.cc:493] T e4d18c35b0a7493e85d0a4b92956a48f P 974b802141bd4f13beffef2aa03f88f9 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260129 02:59:42.879347 22197 raft_consensus.cc:515] T cb4727386a3e48aca5d71cc6c611c197 P 974b802141bd4f13beffef2aa03f88f9 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:59:42.792627 20433 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fd8311b779af4e1fa1402b2a02a602f2" candidate_uuid: "1c50de89a5a845668696dcb28c3badd6" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "197b880fa39d432a9b472845876bfaf0" is_pre_election: true
I20260129 02:59:43.074062 20405 leader_election.cc:304] T 4e3805ffa70242e8ade66da260dddf66 P 197b880fa39d432a9b472845876bfaf0 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 197b880fa39d432a9b472845876bfaf0; no voters: 1c50de89a5a845668696dcb28c3badd6, 7a9daf5d42a84615a6383cfec9a10608
I20260129 02:59:43.074311 20435 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "42d27a28d89b409e8807ec0efd97af69" candidate_uuid: "7a9daf5d42a84615a6383cfec9a10608" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "197b880fa39d432a9b472845876bfaf0" is_pre_election: true
W20260129 02:59:43.076001 20439 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 1.642s user 0.040s sys 0.078s
I20260129 02:59:43.077353 20440 raft_consensus.cc:2468] T e33a417cf9c54cf5af73a4ee3aac4d3a P 197b880fa39d432a9b472845876bfaf0 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1c50de89a5a845668696dcb28c3badd6 in term 4.
W20260129 02:59:43.078536 20441 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 1.248s user 0.004s sys 0.037s
W20260129 02:59:43.080734 20437 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55676 (request call id 1174) took 3352 ms (client timeout 2623 ms). Trace:
W20260129 02:59:43.082832 20434 consensus_meta.cc:287] T fd8311b779af4e1fa1402b2a02a602f2 P 197b880fa39d432a9b472845876bfaf0: Time spent flushing consensus metadata: real 2.436s user 0.011s sys 0.012s
I20260129 02:59:42.833247 20479 leader_election.cc:304] T 368953c9102643b2a21f99af36b51905 P 974b802141bd4f13beffef2aa03f88f9 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 197b880fa39d432a9b472845876bfaf0, 974b802141bd4f13beffef2aa03f88f9; no voters: 1c50de89a5a845668696dcb28c3badd6
W20260129 02:59:43.089450 20510 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 1.632s user 0.005s sys 0.015s
I20260129 02:59:43.089993 20438 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:48356 (request call id 243) took 5707 ms. Trace:
I20260129 02:59:42.815995 20509 raft_consensus.cc:3060] T 4f5a77e94fef4b16a2ba7b067c8b144a P 974b802141bd4f13beffef2aa03f88f9 [term 2 FOLLOWER]: Advancing to term 3
I20260129 02:59:42.882433 20585 tablet_service.cc:2066] Received LeaderStepDown RPC: tablet_id: "926a250fa5834e88b96ccecbefbf5086"
dest_uuid: "1c50de89a5a845668696dcb28c3badd6"
mode: GRACEFUL
new_leader_uuid: "197b880fa39d432a9b472845876bfaf0"
from {username='slave'} at 127.0.0.1:54860
I20260129 02:59:42.921692 21136 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "75b0b6e82ddc4f92b1fe1d94f9a0b73b" candidate_uuid: "974b802141bd4f13beffef2aa03f88f9" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "7a9daf5d42a84615a6383cfec9a10608" is_pre_election: true
W20260129 02:59:43.090058 20514 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 1.317s user 0.009s sys 0.040s
W20260129 02:59:43.240793 20514 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 1.468s user 0.010s sys 0.041s
W20260129 02:59:43.103044 21127 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34060 (request call id 457) took 6216 ms (client timeout 1989 ms). Trace:
I20260129 02:59:43.104707 21131 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "45cc5f3d2688486d9b6dbe7e666dedde" candidate_uuid: "974b802141bd4f13beffef2aa03f88f9" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "7a9daf5d42a84615a6383cfec9a10608" is_pre_election: true
I20260129 02:59:43.242233 22255 leader_election.cc:290] T 18f30ebaa4174fd28c958e72d0cc7e24 P 7a9daf5d42a84615a6383cfec9a10608 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 197b880fa39d432a9b472845876bfaf0 (127.18.211.1:42027), 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3:37735)
I20260129 02:59:43.242574 22364 leader_election.cc:290] T 0678b8b468bd43cdbc3e485900844821 P 7a9daf5d42a84615a6383cfec9a10608 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 197b880fa39d432a9b472845876bfaf0 (127.18.211.1:42027), 974b802141bd4f13beffef2aa03f88f9 (127.18.211.2:34517)
I20260129 02:59:43.242883 21131 raft_consensus.cc:2468] T 45cc5f3d2688486d9b6dbe7e666dedde P 7a9daf5d42a84615a6383cfec9a10608 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 974b802141bd4f13beffef2aa03f88f9 in term 3.
W20260129 02:59:43.105701 20511 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:57584 (request call id 1212) took 7184 ms (client timeout 1508 ms). Trace:
I20260129 02:59:42.902155 21132 rpcz_store.cc:276] 0129 02:59:40.552894 (+ 0us) service_pool.cc:168] Inserting onto call queue
0129 02:59:40.561193 (+ 8299us) service_pool.cc:225] Handling call
0129 02:59:40.570513 (+ 9320us) raft_consensus.cc:1471] Updating replica for 0 ops
0129 02:59:40.795933 (+225420us) raft_consensus.cc:1697] Filling consensus response to leader.
0129 02:59:42.816702 (+2020769us) spinlock_profiling.cc:243] Waited 179 ms on lock 0x7b540068e2d4. stack: 00007f8fe044e373 00007f8fe044e270 00007f8fdfda0574 00007f8fe9a5c6b4 00007f8fe9a5c649 00007f8fe34975a1 00007f8fe352ea81 00007f8fe352d4be 00007f8fe84e1f49 00007f8fe1a6ed34 00007f8fe1a6ecc2 00007f8fe1a6ebf1 00007f8fe1a6eb6c 00007f8fe1a6de22 00007f8fe186d34c 00007f8fe186c786
0129 02:59:42.836411 (+ 19709us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":411582848}
I20260129 02:59:43.107653 21128 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34060 (request call id 473) took 2715 ms. Trace:
I20260129 02:59:42.923313 22231 leader_election.cc:290] T 3ccac82a01a2445fa5b0ce6c0fcf2295 P 197b880fa39d432a9b472845876bfaf0 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 974b802141bd4f13beffef2aa03f88f9 (127.18.211.2:34517), 7a9daf5d42a84615a6383cfec9a10608 (127.18.211.4:40439)
W20260129 02:59:43.249045 21099 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.211.1:42027, user_credentials={real_user=slave}} blocked reactor thread for 375623us
W20260129 02:59:43.242277 21127 rpcz_store.cc:269] 0129 02:59:36.886769 (+ 0us) service_pool.cc:168] Inserting onto call queue
0129 02:59:36.891220 (+ 4451us) service_pool.cc:225] Handling call
0129 02:59:42.893915 (+6002695us) spinlock_profiling.cc:243] Waited 3.79 s on lock 0x7b5400670554. stack: 00007f8fe044e373 00007f8fe044e270 00007f8fdfda0574 00007f8fe9a5c6b4 00007f8fe9a5c649 00007f8fe34975a1 00007f8fe3531f7a 00007f8fe84e3647 00007f8fe1a73834 00007f8fe1a737c2 00007f8fe1a736f1 00007f8fe1a7366c 00007f8fe1a72922 00007f8fe186d34c 00007f8fe186c786 00007f8fe186c12f
0129 02:59:43.102658 (+208743us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":8721527680}
I20260129 02:59:43.249759 21132 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a609ac02fae84725ab743f95f51d33ce" candidate_uuid: "974b802141bd4f13beffef2aa03f88f9" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "7a9daf5d42a84615a6383cfec9a10608" is_pre_election: true
I20260129 02:59:43.250044 21133 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34060 (request call id 471) took 2885 ms. Trace:
I20260129 02:59:42.918802 22320 raft_consensus.cc:3060] T 17d6431cc77a491e8464d68db4081a51 P 1c50de89a5a845668696dcb28c3badd6 [term 2 FOLLOWER]: Advancing to term 3
I20260129 02:59:43.251106 21127 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1299723a4476420ab02fd8bfd7aa0645" candidate_uuid: "1c50de89a5a845668696dcb28c3badd6" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "7a9daf5d42a84615a6383cfec9a10608" is_pre_election: true
I20260129 02:59:42.901155 22354 leader_election.cc:290] T fd8311b779af4e1fa1402b2a02a602f2 P 197b880fa39d432a9b472845876bfaf0 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3:37735), 7a9daf5d42a84615a6383cfec9a10608 (127.18.211.4:40439)
I20260129 02:59:42.971940 22374 leader_election.cc:290] T 3e2d15f8640b407195210c476e09df91 P 7a9daf5d42a84615a6383cfec9a10608 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 974b802141bd4f13beffef2aa03f88f9 (127.18.211.2:34517), 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3:37735)
I20260129 02:59:42.973450 22385 leader_election.cc:290] T 7a9a69cd82724b1faed1d10aa091fc4a P 197b880fa39d432a9b472845876bfaf0 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 974b802141bd4f13beffef2aa03f88f9 (127.18.211.2:34517), 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3:37735)
I20260129 02:59:43.264781 22566 raft_consensus.cc:493] T cb0b3f259ebc4310927338a8740e87b8 P 974b802141bd4f13beffef2aa03f88f9 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 7a9daf5d42a84615a6383cfec9a10608)
I20260129 02:59:42.973891 22437 leader_election.cc:290] T febffe4a8b394368b69b1d164308fb9a P 974b802141bd4f13beffef2aa03f88f9 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 197b880fa39d432a9b472845876bfaf0 (127.18.211.1:42027), 7a9daf5d42a84615a6383cfec9a10608 (127.18.211.4:40439)
W20260129 02:59:42.930019 22493 consensus_meta.cc:287] T 8055232cf0004d40a85f3bf983fe00a9 P 974b802141bd4f13beffef2aa03f88f9: Time spent flushing consensus metadata: real 2.192s user 0.015s sys 0.006s
W20260129 02:59:43.270785 20404 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260129 02:59:42.935290 22518 leader_election.cc:290] T e7d83d4ccf684e76acb29a0dc844e1b8 P 197b880fa39d432a9b472845876bfaf0 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 974b802141bd4f13beffef2aa03f88f9 (127.18.211.2:34517), 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3:37735)
I20260129 02:59:42.992619 22520 leader_election.cc:290] T 75be4bd6bd45474795fd472eff2953ac P 197b880fa39d432a9b472845876bfaf0 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 974b802141bd4f13beffef2aa03f88f9 (127.18.211.2:34517), 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3:37735)
W20260129 02:59:42.990862 22535 log.cc:927] Time spent T 76f713c3331540d2a5b76164103e9eb1 P 974b802141bd4f13beffef2aa03f88f9: Append to log took a long time: real 0.275s user 0.001s sys 0.000s
I20260129 02:59:43.278026 22573 raft_consensus.cc:2749] T 4e3805ffa70242e8ade66da260dddf66 P 197b880fa39d432a9b472845876bfaf0 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
W20260129 02:59:43.278496 20405 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.211.3:37735, user_credentials={real_user=slave}} blocked reactor thread for 205066us
W20260129 02:59:42.969856 22538 log.cc:927] Time spent T add8d4923d1a489e86882c3a25692b95 P 7a9daf5d42a84615a6383cfec9a10608: Append to log took a long time: real 0.169s user 0.000s sys 0.000s
I20260129 02:59:42.917439 22540 raft_consensus.cc:740] T c934d31efd3f4e5195605ddc354d14e1 P 197b880fa39d432a9b472845876bfaf0 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 197b880fa39d432a9b472845876bfaf0, State: Running, Role: LEADER
W20260129 02:59:42.970919 22544 log.cc:927] Time spent T 724a236524cb4e7ba3428880681d6cc2 P 197b880fa39d432a9b472845876bfaf0: Append to log took a long time: real 0.095s user 0.000s sys 0.000s
W20260129 02:59:43.032269 22547 log.cc:927] Time spent T a8479d1fe0bd41e5946186e3a6ca8c7a P 197b880fa39d432a9b472845876bfaf0: Append to log took a long time: real 0.057s user 0.000s sys 0.002s
I20260129 02:59:42.965798 22551 consensus_queue.cc:1059] T 368953c9102643b2a21f99af36b51905 P 1c50de89a5a845668696dcb28c3badd6 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 }, Status: INVALID_TERM, Last received: 2.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.014s
W20260129 02:59:43.034087 22550 log.cc:927] Time spent T 4b1876f9a70b49edb3aa74105e66d3b3 P 1c50de89a5a845668696dcb28c3badd6: Append to log took a long time: real 0.065s user 0.003s sys 0.001s
W20260129 02:59:43.033728 22561 log.cc:927] Time spent T 3906b080a5b34a74b8b64128635e8997 P 197b880fa39d432a9b472845876bfaf0: Append to log took a long time: real 0.070s user 0.000s sys 0.001s
I20260129 02:59:43.295565 20589 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35752 (request call id 1511) took 2115 ms. Trace:
I20260129 02:59:42.933495 22400 raft_consensus.cc:3060] T e4d18c35b0a7493e85d0a4b92956a48f P 974b802141bd4f13beffef2aa03f88f9 [term 3 FOLLOWER]: Advancing to term 4
I20260129 02:59:43.296651 22364 raft_consensus.cc:2804] T 2e58e9a5a8aa4cecafb464addb4f35ed P 7a9daf5d42a84615a6383cfec9a10608 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260129 02:59:43.297394 22364 raft_consensus.cc:493] T 2e58e9a5a8aa4cecafb464addb4f35ed P 7a9daf5d42a84615a6383cfec9a10608 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260129 02:59:43.014602 22501 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.618s user 0.023s sys 0.005s
I20260129 02:59:43.298038 22364 raft_consensus.cc:3060] T 2e58e9a5a8aa4cecafb464addb4f35ed P 7a9daf5d42a84615a6383cfec9a10608 [term 3 FOLLOWER]: Advancing to term 4
W20260129 02:59:43.298341 22501 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.905s user 0.024s sys 0.006s
W20260129 02:59:43.204373 22157 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 1.123s user 0.023s sys 0.034s
W20260129 02:59:43.217201 20437 rpcz_store.cc:269] 0129 02:59:39.727914 (+ 0us) service_pool.cc:168] Inserting onto call queue
0129 02:59:39.728068 (+ 154us) service_pool.cc:225] Handling call
0129 02:59:43.079839 (+3351771us) spinlock_profiling.cc:243] Waited 1.06 s on lock 0x7b54002666d4. stack: 00007f8fe044e373 00007f8fe044e270 00007f8fdfda0574 00007f8fe9a5c6b4 00007f8fe9a5c649 00007f8fe34975a1 00007f8fe3531f7a 00007f8fe84e3647 00007f8fe1a73834 00007f8fe1a737c2 00007f8fe1a736f1 00007f8fe1a7366c 00007f8fe1a72922 00007f8fe186d34c 00007f8fe186c786 00007f8fe186c12f
0129 02:59:43.080276 (+ 437us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2433055488}
I20260129 02:59:42.890851 22504 raft_consensus.cc:493] T dc2f4cb752cf4d59b5f6bc689f5d8104 P 1c50de89a5a845668696dcb28c3badd6 [term 3 FOLLOWER]: Starting leader election (detected failure of leader 7a9daf5d42a84615a6383cfec9a10608)
W20260129 02:59:43.299690 22157 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 1.493s user 0.023s sys 0.035s
I20260129 02:59:43.300253 22504 raft_consensus.cc:3060] T dc2f4cb752cf4d59b5f6bc689f5d8104 P 1c50de89a5a845668696dcb28c3badd6 [term 3 FOLLOWER]: Advancing to term 4
I20260129 02:59:43.300778 20432 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55676 (request call id 1186) took 2780 ms. Trace:
W20260129 02:59:43.226982 20439 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 1.793s user 0.040s sys 0.079s
I20260129 02:59:43.301030 20437 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c934d31efd3f4e5195605ddc354d14e1" candidate_uuid: "7a9daf5d42a84615a6383cfec9a10608" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "197b880fa39d432a9b472845876bfaf0" is_pre_election: true
W20260129 02:59:43.224901 20441 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 1.394s user 0.005s sys 0.037s
I20260129 02:59:43.302429 20439 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55676 (request call id 1192) took 2780 ms. Trace:
I20260129 02:59:43.231771 22308 raft_consensus.cc:493] T 96609ac68bdf4993a12e1c23355ca013 P 1c50de89a5a845668696dcb28c3badd6 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260129 02:59:43.301294 20432 rpcz_store.cc:276] 0129 02:59:40.520426 (+ 0us) service_pool.cc:168] Inserting onto call queue
0129 02:59:40.608321 (+ 87895us) service_pool.cc:225] Handling call
0129 02:59:40.608782 (+ 461us) raft_consensus.cc:1471] Updating replica for 1 ops
0129 02:59:40.609178 (+ 396us) raft_consensus.cc:1534] Early marking committed up to index 0
0129 02:59:40.609214 (+ 36us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0129 02:59:40.645300 (+ 36086us) log.cc:844] Serialized 26 byte log entry
0129 02:59:42.149947 (+1504647us) raft_consensus.cc:1645] Marking committed up to 1
0129 02:59:42.149987 (+ 40us) raft_consensus.cc:1697] Filling consensus response to leader.
0129 02:59:42.150061 (+ 74us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0129 02:59:42.830041 (+679980us) spinlock_profiling.cc:243] Waited 6.21 ms on lock 0x7b48001c0158. stack: 00007f8fe044e373 00007f8fe044e270 00007f8fdfda0574 00007f8fe9a5c6b4 00007f8fe9a5c649 00007f8fe8447e21 00007f8fe843eeb7 00007f8fe843ed4a 00007f8fe8535f7f 00007f8fe8535c47 00007f8fe854a26c 00007f8fe854a218 00007f8fe854a198 00007f8fe854a153 00007f8fe85493fc 00007f8fe74845b6
0129 02:59:43.300080 (+470039us) raft_consensus.cc:1685] finished
0129 02:59:43.300131 (+ 51us) raft_consensus.cc:1691] UpdateReplicas() finished
0129 02:59:43.300527 (+ 396us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":27355,"raft.queue_time_us":37469,"raft.run_cpu_time_us":14002,"raft.run_wall_time_us":2196617,"spinlock_wait_cycles":14289152,"thread_start_us":1456239,"threads_started":1,"wal-append.queue_time_us":2157025}
I20260129 02:59:43.302816 20441 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55676 (request call id 1188) took 2781 ms. Trace:
W20260129 02:59:43.237686 20510 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 1.780s user 0.006s sys 0.016s
I20260129 02:59:43.304037 22520 raft_consensus.cc:493] T b189f08e16a74a11980cc55d99359a0a P 197b880fa39d432a9b472845876bfaf0 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260129 02:59:43.304471 20554 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.18.211.4:40439 timed out after 1.796s (SENT) [suppressed 36 similar messages]
W20260129 02:59:43.239681 21100 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.211.3:37735, user_credentials={real_user=slave}} blocked reactor thread for 396122us
W20260129 02:59:43.243824 21131 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:47844 (request call id 295) took 2058 ms (client timeout 2635 ms). Trace:
I20260129 02:59:43.236193 20435 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:48356 (request call id 263) took 1084 ms. Trace:
I20260129 02:59:43.229151 20438 rpcz_store.cc:276] 0129 02:59:37.382326 (+ 0us) service_pool.cc:168] Inserting onto call queue
0129 02:59:37.887981 (+505655us) service_pool.cc:225] Handling call
0129 02:59:40.525497 (+2637516us) raft_consensus.cc:1471] Updating replica for 0 ops
0129 02:59:41.624864 (+1099367us) raft_consensus.cc:1697] Filling consensus response to leader.
0129 02:59:42.484896 (+860032us) spinlock_profiling.cc:243] Waited 1.06 s on lock 0x7b54002fc454. stack: 00007f8fe044e373 00007f8fe044e270 00007f8fdfda0574 00007f8fe9a5c6b4 00007f8fe9a5c649 00007f8fe34975a1 00007f8fe352ea81 00007f8fe352d4be 00007f8fe84e1f49 00007f8fe1a6ed34 00007f8fe1a6ecc2 00007f8fe1a6ebf1 00007f8fe1a6eb6c 00007f8fe1a6de22 00007f8fe186d34c 00007f8fe186c786
0129 02:59:43.089277 (+604381us) spinlock_profiling.cc:243] Waited 2.64 s on lock 0x7b54002fc450. stack: 00007f8fe044e373 00007f8fe044e270 00007f8fdfda0574 00007f8fe9a5c6b4 00007f8fe9a5c649 00007f8fe34975a1 00007f8fe352d52d 00007f8fe84e1f49 00007f8fe1a6ed34 00007f8fe1a6ecc2 00007f8fe1a6ebf1 00007f8fe1a6eb6c 00007f8fe1a6de22 00007f8fe186d34c 00007f8fe186c786 00007f8fe186c12f
0129 02:59:43.089669 (+ 392us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":8494267520}
W20260129 02:59:43.244240 20511 rpcz_store.cc:269] 0129 02:59:35.921398 (+ 0us) service_pool.cc:168] Inserting onto call queue
0129 02:59:35.961178 (+ 39780us) service_pool.cc:225] Handling call
0129 02:59:43.105051 (+7143873us) spinlock_profiling.cc:243] Waited 6.71 s on lock 0x7b5400229a54. stack: 00007f8fe044e373 00007f8fe044e270 00007f8fdfda0574 00007f8fe9a5c6b4 00007f8fe9a5c649 00007f8fe34975a1 00007f8fe3531f7a 00007f8fe84e3647 00007f8fe1a73834 00007f8fe1a737c2 00007f8fe1a736f1 00007f8fe1a7366c 00007f8fe1a72922 00007f8fe186d34c 00007f8fe186c786 00007f8fe186c12f
0129 02:59:43.105317 (+ 266us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":15426562432}
I20260129 02:59:43.239738 21129 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34046 (request call id 428) took 3013 ms. Trace:
I20260129 02:59:43.248314 21128 rpcz_store.cc:276] 0129 02:59:40.391698 (+ 0us) service_pool.cc:168] Inserting onto call queue
0129 02:59:40.400580 (+ 8882us) service_pool.cc:225] Handling call
0129 02:59:40.495306 (+ 94726us) raft_consensus.cc:1471] Updating replica for 0 ops
0129 02:59:41.733562 (+1238256us) raft_consensus.cc:1697] Filling consensus response to leader.
0129 02:59:43.106861 (+1373299us) spinlock_profiling.cc:243] Waited 1.15 s on lock 0x7b54003b3dd4. stack: 00007f8fe044e373 00007f8fe044e270 00007f8fdfda0574 00007f8fe9a5c6b4 00007f8fe9a5c649 00007f8fe34975a1 00007f8fe352ea81 00007f8fe352d4be 00007f8fe84e1f49 00007f8fe1a6ed34 00007f8fe1a6ecc2 00007f8fe1a6ebf1 00007f8fe1a6eb6c 00007f8fe1a6de22 00007f8fe186d34c 00007f8fe186c786
0129 02:59:43.107377 (+ 516us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2645278592}
I20260129 02:59:43.250684 21133 rpcz_store.cc:276] 0129 02:59:40.364899 (+ 0us) service_pool.cc:168] Inserting onto call queue
0129 02:59:40.365045 (+ 146us) service_pool.cc:225] Handling call
0129 02:59:40.411719 (+ 46674us) raft_consensus.cc:1471] Updating replica for 0 ops
0129 02:59:40.411981 (+ 262us) raft_consensus.cc:1534] Early marking committed up to index 6
0129 02:59:41.663925 (+1251944us) log.cc:844] Serialized 14 byte log entry
0129 02:59:43.249295 (+1585370us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0129 02:59:43.249420 (+ 125us) raft_consensus.cc:1645] Marking committed up to 6
0129 02:59:43.249566 (+ 146us) raft_consensus.cc:1697] Filling consensus response to leader.
0129 02:59:43.249652 (+ 86us) raft_consensus.cc:1691] UpdateReplicas() finished
0129 02:59:43.249847 (+ 195us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":1820743,"prepare.run_cpu_time_us":33,"prepare.run_wall_time_us":33,"thread_start_us":2814036,"threads_started":3,"wal-append.queue_time_us":1334931}
I20260129 02:59:43.257870 20590 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:48526 (request call id 191) took 2088 ms. Trace:
I20260129 02:59:43.258818 21127 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34046 (request call id 435) took 1039 ms. Trace:
I20260129 02:59:43.305912 20435 rpcz_store.cc:276] 0129 02:59:42.145529 (+ 0us) service_pool.cc:168] Inserting onto call queue
0129 02:59:42.803522 (+657993us) service_pool.cc:225] Handling call
0129 02:59:43.227839 (+424317us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20260129 02:59:43.305800 21131 rpcz_store.cc:269] 0129 02:59:41.185067 (+ 0us) service_pool.cc:168] Inserting onto call queue
0129 02:59:42.917418 (+1732351us) service_pool.cc:225] Handling call
0129 02:59:43.243715 (+326297us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260129 02:59:43.281817 20507 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57600 (request call id 1679) took 6098 ms. Trace:
I20260129 02:59:43.225689 20434 raft_consensus.cc:2468] T fd8311b779af4e1fa1402b2a02a602f2 P 197b880fa39d432a9b472845876bfaf0 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7a9daf5d42a84615a6383cfec9a10608 in term 3.
W20260129 02:59:43.237411 22337 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.697s user 0.009s sys 0.007s
I20260129 02:59:43.268327 20436 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55676 (request call id 1169) took 5130 ms. Trace:
W20260129 02:59:43.268499 21099 leader_election.cc:336] T 42d27a28d89b409e8807ec0efd97af69 P 7a9daf5d42a84615a6383cfec9a10608 [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer 197b880fa39d432a9b472845876bfaf0 (127.18.211.1:42027): Timed out: RequestConsensusVote RPC to 127.18.211.1:42027 timed out after 1.873s (SENT)
I20260129 02:59:43.297313 22374 raft_consensus.cc:2749] T 4e3805ffa70242e8ade66da260dddf66 P 7a9daf5d42a84615a6383cfec9a10608 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260129 02:59:43.232780 21134 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34060 (request call id 472) took 2854 ms. Trace:
I20260129 02:59:43.250840 21132 raft_consensus.cc:2468] T a609ac02fae84725ab743f95f51d33ce P 7a9daf5d42a84615a6383cfec9a10608 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 974b802141bd4f13beffef2aa03f88f9 in term 5.
W20260129 02:59:43.303516 20556 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.211.2:34517, user_credentials={real_user=slave}} blocked reactor thread for 461080us
I20260129 02:59:43.279851 20405 leader_election.cc:304] T a4c1226b987b468fa84dfc6ba0e2dbe1 P 197b880fa39d432a9b472845876bfaf0 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 197b880fa39d432a9b472845876bfaf0; no voters: 1c50de89a5a845668696dcb28c3badd6, 974b802141bd4f13beffef2aa03f88f9
I20260129 02:59:43.304804 22520 raft_consensus.cc:515] T b189f08e16a74a11980cc55d99359a0a P 197b880fa39d432a9b472845876bfaf0 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:59:43.303440 22308 raft_consensus.cc:515] T 96609ac68bdf4993a12e1c23355ca013 P 1c50de89a5a845668696dcb28c3badd6 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } attrs { replace: true } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: NON_VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: true } }
W20260129 02:59:43.503158 22416 consensus_meta.cc:287] T 724a236524cb4e7ba3428880681d6cc2 P 974b802141bd4f13beffef2aa03f88f9: Time spent flushing consensus metadata: real 1.306s user 0.008s sys 0.008s
I20260129 02:59:43.296150 20589 rpcz_store.cc:276] 0129 02:59:41.180297 (+ 0us) service_pool.cc:168] Inserting onto call queue
0129 02:59:41.180530 (+ 233us) service_pool.cc:225] Handling call
0129 02:59:41.182396 (+ 1866us) raft_consensus.cc:1471] Updating replica for 1 ops
0129 02:59:41.182834 (+ 438us) raft_consensus.cc:1534] Early marking committed up to index 0
0129 02:59:41.182876 (+ 42us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0129 02:59:41.193615 (+ 10739us) log.cc:844] Serialized 26 byte log entry
0129 02:59:42.704782 (+1511167us) raft_consensus.cc:1645] Marking committed up to 3
0129 02:59:42.704827 (+ 45us) raft_consensus.cc:1697] Filling consensus response to leader.
0129 02:59:42.704893 (+ 66us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0129 02:59:43.007410 (+302517us) spinlock_profiling.cc:243] Waited 10.8 ms on lock 0x7b440002304c. stack: 00007f8fe044e373 00007f8fe044e270 00007f8fdfda0574 00007f8fe9a5c6b4 00007f8fe9a5c649 00007f8fe02c08b1 00007f8fe038f005 00007f8fe038fa22 00007f8fe30d957b 00007f8fe30d4e4f 00007f8fe30c8c0e 00007f8fe30c7f56 00007f8fe30c7b78 00007f8fe30d15c1 00007f8fe30d1579 00007f8fe30d1509
0129 02:59:43.294889 (+287479us) raft_consensus.cc:1685] finished
0129 02:59:43.294932 (+ 43us) raft_consensus.cc:1691] UpdateReplicas() finished
0129 02:59:43.295368 (+ 436us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":680,"raft.run_cpu_time_us":156,"raft.run_wall_time_us":9201,"spinlock_wait_cycles":24734464,"thread_start_us":1455585,"threads_started":1,"wal-append.queue_time_us":1719831}
I20260129 02:59:43.303344 20441 rpcz_store.cc:276] 0129 02:59:40.520988 (+ 0us) service_pool.cc:168] Inserting onto call queue
0129 02:59:40.609928 (+ 88940us) service_pool.cc:225] Handling call
0129 02:59:40.611856 (+ 1928us) raft_consensus.cc:1471] Updating replica for 0 ops
0129 02:59:40.612127 (+ 271us) raft_consensus.cc:1534] Early marking committed up to index 3
0129 02:59:41.830330 (+1218203us) log.cc:844] Serialized 14 byte log entry
0129 02:59:43.302264 (+1471934us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0129 02:59:43.302361 (+ 97us) raft_consensus.cc:1645] Marking committed up to 3
0129 02:59:43.302500 (+ 139us) raft_consensus.cc:1697] Filling consensus response to leader.
0129 02:59:43.302594 (+ 94us) raft_consensus.cc:1691] UpdateReplicas() finished
0129 02:59:43.302763 (+ 169us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":1844030,"prepare.run_cpu_time_us":64,"prepare.run_wall_time_us":66,"thread_start_us":2637482,"threads_started":2,"wal-append.queue_time_us":1191511}
I20260129 02:59:43.499928 20435 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3ccac82a01a2445fa5b0ce6c0fcf2295" candidate_uuid: "7a9daf5d42a84615a6383cfec9a10608" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "197b880fa39d432a9b472845876bfaf0" is_pre_election: true
I20260129 02:59:43.489029 20516 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57584 (request call id 1262) took 1329 ms. Trace:
W20260129 02:59:43.512885 21099 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.211.1:42027, user_credentials={real_user=slave}} blocked reactor thread for 251967us
I20260129 02:59:43.376089 20433 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55676 (request call id 1201) took 1121 ms. Trace:
W20260129 02:59:43.379269 21100 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260129 02:59:43.331518 20507 rpcz_store.cc:276] 0129 02:59:37.182770 (+ 0us) service_pool.cc:168] Inserting onto call queue
0129 02:59:38.874495 (+1691725us) service_pool.cc:225] Handling call
0129 02:59:38.893751 (+ 19256us) raft_consensus.cc:1471] Updating replica for 1 ops
0129 02:59:40.213476 (+1319725us) raft_consensus.cc:1534] Early marking committed up to index 0
0129 02:59:40.213503 (+ 27us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0129 02:59:40.284517 (+ 71014us) log.cc:844] Serialized 26 byte log entry
0129 02:59:41.704582 (+1420065us) raft_consensus.cc:1645] Marking committed up to 2
0129 02:59:41.704633 (+ 51us) raft_consensus.cc:1697] Filling consensus response to leader.
0129 02:59:41.704714 (+ 81us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0129 02:59:43.281137 (+1576423us) raft_consensus.cc:1685] finished
0129 02:59:43.281191 (+ 54us) raft_consensus.cc:1691] UpdateReplicas() finished
0129 02:59:43.281490 (+ 299us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":86637,"raft.run_cpu_time_us":126,"raft.run_wall_time_us":127,"thread_start_us":2649526,"threads_started":2,"wal-append.queue_time_us":1956628}
I20260129 02:59:43.512403 20515 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57584 (request call id 1248) took 2046 ms. Trace:
I20260129 02:59:43.302851 20439 rpcz_store.cc:276] 0129 02:59:40.522080 (+ 0us) service_pool.cc:168] Inserting onto call queue
0129 02:59:41.263156 (+741076us) service_pool.cc:225] Handling call
0129 02:59:41.263550 (+ 394us) raft_consensus.cc:1471] Updating replica for 1 ops
0129 02:59:41.265570 (+ 2020us) raft_consensus.cc:1534] Early marking committed up to index 0
0129 02:59:41.265615 (+ 45us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0129 02:59:41.266204 (+ 589us) log.cc:844] Serialized 26 byte log entry
0129 02:59:43.301866 (+2035662us) raft_consensus.cc:1645] Marking committed up to 1
0129 02:59:43.301964 (+ 98us) raft_consensus.cc:1697] Filling consensus response to leader.
0129 02:59:43.302069 (+ 105us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0129 02:59:43.302135 (+ 66us) raft_consensus.cc:1685] finished
0129 02:59:43.302151 (+ 16us) raft_consensus.cc:1691] UpdateReplicas() finished
0129 02:59:43.302345 (+ 194us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":104,"raft.run_cpu_time_us":188,"raft.run_wall_time_us":147482,"thread_start_us":1867561,"threads_started":1,"wal-append.queue_time_us":1591580}
I20260129 02:59:43.494552 20514 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57584 (request call id 1254) took 1779 ms. Trace:
I20260129 02:59:43.490521 20438 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "913fc7d751ed4b7388d889aac30a5f9b" candidate_uuid: "974b802141bd4f13beffef2aa03f88f9" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "197b880fa39d432a9b472845876bfaf0" is_pre_election: true
I20260129 02:59:43.285844 22231 raft_consensus.cc:493] T 96609ac68bdf4993a12e1c23355ca013 P 197b880fa39d432a9b472845876bfaf0 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260129 02:59:43.344978 20406 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260129 02:59:43.385784 20432 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2e58e9a5a8aa4cecafb464addb4f35ed" candidate_uuid: "7a9daf5d42a84615a6383cfec9a10608" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "197b880fa39d432a9b472845876bfaf0" is_pre_election: true
I20260129 02:59:43.385552 21131 raft_consensus.cc:1275] T 4f5a77e94fef4b16a2ba7b067c8b144a P 7a9daf5d42a84615a6383cfec9a10608 [term 3 FOLLOWER]: Refusing update from remote peer 1c50de89a5a845668696dcb28c3badd6: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 3 index: 5. (index mismatch)
I20260129 02:59:43.372009 21135 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34046 (request call id 429) took 3145 ms. Trace:
W20260129 02:59:43.380230 20481 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260129 02:59:43.515609 20433 rpcz_store.cc:276] 0129 02:59:42.166397 (+ 0us) service_pool.cc:168] Inserting onto call queue
0129 02:59:42.792343 (+625946us) service_pool.cc:225] Handling call
0129 02:59:43.273796 (+481453us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260129 02:59:43.345310 20440 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55676 (request call id 1195) took 1177 ms. Trace:
I20260129 02:59:43.372747 20511 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "cb4727386a3e48aca5d71cc6c611c197" candidate_uuid: "1c50de89a5a845668696dcb28c3badd6" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "974b802141bd4f13beffef2aa03f88f9" is_pre_election: true
W20260129 02:59:43.501542 20582 consensus_meta.cc:287] T 42d27a28d89b409e8807ec0efd97af69 P 1c50de89a5a845668696dcb28c3badd6: Time spent flushing consensus metadata: real 0.659s user 0.005s sys 0.013s
I20260129 02:59:43.376142 21136 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:47844 (request call id 293) took 2211 ms. Trace:
I20260129 02:59:43.400465 20581 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "18f30ebaa4174fd28c958e72d0cc7e24" candidate_uuid: "7a9daf5d42a84615a6383cfec9a10608" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "1c50de89a5a845668696dcb28c3badd6" is_pre_election: true
I20260129 02:59:43.531984 20440 rpcz_store.cc:276] 0129 02:59:42.164528 (+ 0us) service_pool.cc:168] Inserting onto call queue
0129 02:59:42.749646 (+585118us) service_pool.cc:225] Handling call
0129 02:59:43.305546 (+555900us) spinlock_profiling.cc:243] Waited 273 ms on lock 0x7b540026e654. stack: 00007f8fe044e373 00007f8fe044e270 00007f8fdfda0574 00007f8fe9a5c6b4 00007f8fe9a5c649 00007f8fe34975a1 00007f8fe3531f7a 00007f8fe84e3647 00007f8fe1a73834 00007f8fe1a737c2 00007f8fe1a736f1 00007f8fe1a7366c 00007f8fe1a72922 00007f8fe186d34c 00007f8fe186c786 00007f8fe186c12f
0129 02:59:43.341609 (+ 36063us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":627659648}
I20260129 02:59:43.399608 20587 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3e2d15f8640b407195210c476e09df91" candidate_uuid: "7a9daf5d42a84615a6383cfec9a10608" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "1c50de89a5a845668696dcb28c3badd6" is_pre_election: true
I20260129 02:59:43.400935 22237 raft_consensus.cc:493] T febffe4a8b394368b69b1d164308fb9a P 197b880fa39d432a9b472845876bfaf0 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260129 02:59:43.375746 22577 raft_consensus.cc:493] T 96609ac68bdf4993a12e1c23355ca013 P 7a9daf5d42a84615a6383cfec9a10608 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260129 02:59:43.534965 22308 leader_election.cc:290] T 96609ac68bdf4993a12e1c23355ca013 P 1c50de89a5a845668696dcb28c3badd6 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 197b880fa39d432a9b472845876bfaf0 (127.18.211.1:42027), 974b802141bd4f13beffef2aa03f88f9 (127.18.211.2:34517)
W20260129 02:59:43.535355 21099 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260129 02:59:43.535686 22577 raft_consensus.cc:515] T 96609ac68bdf4993a12e1c23355ca013 P 7a9daf5d42a84615a6383cfec9a10608 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } attrs { replace: true } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } }
I20260129 02:59:43.538812 22552 raft_consensus.cc:3055] T 1ce4ddf5b10e4ae8b193b13b2e5936ee P 1c50de89a5a845668696dcb28c3badd6 [term 4 LEADER]: Stepping down as leader of term 4
I20260129 02:59:43.539398 22552 raft_consensus.cc:740] T 1ce4ddf5b10e4ae8b193b13b2e5936ee P 1c50de89a5a845668696dcb28c3badd6 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 1c50de89a5a845668696dcb28c3badd6, State: Running, Role: LEADER
W20260129 02:59:43.540236 22556 log.cc:927] Time spent T 4f8f3a38f6e24ba4857875181f447442 P 974b802141bd4f13beffef2aa03f88f9: Append to log took a long time: real 0.139s user 0.000s sys 0.001s
W20260129 02:59:43.385955 22337 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.920s user 0.010s sys 0.008s
W20260129 02:59:43.379994 20480 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.211.3:37735, user_credentials={real_user=slave}} blocked reactor thread for 462009us
W20260129 02:59:43.511359 22343 consensus_meta.cc:287] T fd8311b779af4e1fa1402b2a02a602f2 P 7a9daf5d42a84615a6383cfec9a10608: Time spent flushing consensus metadata: real 1.857s user 0.012s sys 0.004s
I20260129 02:59:43.500433 22137 raft_consensus.cc:2749] T 4f8f3a38f6e24ba4857875181f447442 P 974b802141bd4f13beffef2aa03f88f9 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260129 02:59:43.546610 20582 raft_consensus.cc:2468] T 42d27a28d89b409e8807ec0efd97af69 P 1c50de89a5a845668696dcb28c3badd6 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7a9daf5d42a84615a6383cfec9a10608 in term 5.
W20260129 02:59:43.382232 22348 consensus_meta.cc:287] T fcddab6bca684adf8ea032289a00f90e P 1c50de89a5a845668696dcb28c3badd6: Time spent flushing consensus metadata: real 2.156s user 0.015s sys 0.002s
I20260129 02:59:43.539853 20441 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3906b080a5b34a74b8b64128635e8997" candidate_uuid: "974b802141bd4f13beffef2aa03f88f9" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "197b880fa39d432a9b472845876bfaf0" is_pre_election: true
I20260129 02:59:43.532222 21136 rpcz_store.cc:276] 0129 02:59:41.041696 (+ 0us) service_pool.cc:168] Inserting onto call queue
0129 02:59:42.921402 (+1879706us) service_pool.cc:225] Handling call
0129 02:59:43.244826 (+323424us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260129 02:59:43.309332 20590 rpcz_store.cc:276] 0129 02:59:41.169492 (+ 0us) service_pool.cc:168] Inserting onto call queue
0129 02:59:41.172119 (+ 2627us) service_pool.cc:225] Handling call
0129 02:59:41.194056 (+ 21937us) raft_consensus.cc:1471] Updating replica for 1 ops
0129 02:59:41.194341 (+ 285us) raft_consensus.cc:1534] Early marking committed up to index 0
0129 02:59:41.194378 (+ 37us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0129 02:59:41.200427 (+ 6049us) log.cc:844] Serialized 26 byte log entry
0129 02:59:42.818486 (+1618059us) raft_consensus.cc:1645] Marking committed up to 6
0129 02:59:42.818529 (+ 43us) raft_consensus.cc:1697] Filling consensus response to leader.
0129 02:59:42.818621 (+ 92us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0129 02:59:43.008786 (+190165us) spinlock_profiling.cc:243] Waited 1.28 ms on lock 0x7b440002304c. stack: 00007f8fe044e373 00007f8fe044e270 00007f8fdfda0574 00007f8fe9a5c6b4 00007f8fe9a5c649 00007f8fe02c08b1 00007f8fe038f005 00007f8fe038fa22 00007f8fe30d957b 00007f8fe30d4e4f 00007f8fe30c8c0e 00007f8fe30c7f56 00007f8fe30c7b78 00007f8fe30d15c1 00007f8fe30d1579 00007f8fe30d1509
0129 02:59:43.257014 (+248228us) raft_consensus.cc:1685] finished
0129 02:59:43.257070 (+ 56us) raft_consensus.cc:1691] UpdateReplicas() finished
0129 02:59:43.257528 (+ 458us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2953984,"thread_start_us":1387472,"threads_started":1,"wal-append.queue_time_us":1576115}
I20260129 02:59:43.545710 21131 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34046 (request call id 432) took 2074 ms. Trace:
W20260129 02:59:43.392731 21102 leader_election.cc:336] T 3ccac82a01a2445fa5b0ce6c0fcf2295 P 7a9daf5d42a84615a6383cfec9a10608 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 974b802141bd4f13beffef2aa03f88f9 (127.18.211.2:34517): Timed out: RequestConsensusVote RPC to 127.18.211.2:34517 timed out after 1.914s (SENT)
I20260129 02:59:43.280246 22445 raft_consensus.cc:515] T cedce331e1584ac8b636c9b8a72b2ba0 P 974b802141bd4f13beffef2aa03f88f9 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
W20260129 02:59:43.396104 20434 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:48356 (request call id 244) took 5998 ms (client timeout 1551 ms). Trace:
I20260129 02:59:43.376391 20585 raft_consensus.cc:606] T 926a250fa5834e88b96ccecbefbf5086 P 1c50de89a5a845668696dcb28c3badd6 [term 5 FOLLOWER]: Received request to transfer leadership to TS 197b880fa39d432a9b472845876bfaf0
I20260129 02:59:43.332144 21127 rpcz_store.cc:276] 0129 02:59:42.219799 (+ 0us) service_pool.cc:168] Inserting onto call queue
0129 02:59:43.250614 (+1030815us) service_pool.cc:225] Handling call
0129 02:59:43.258720 (+ 8106us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260129 02:59:43.516949 20515 rpcz_store.cc:276] 0129 02:59:41.465671 (+ 0us) service_pool.cc:168] Inserting onto call queue
0129 02:59:41.465998 (+ 327us) service_pool.cc:225] Handling call
0129 02:59:41.485600 (+ 19602us) raft_consensus.cc:1471] Updating replica for 1 ops
0129 02:59:41.486956 (+ 1356us) raft_consensus.cc:1534] Early marking committed up to index 0
0129 02:59:41.487001 (+ 45us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0129 02:59:41.490701 (+ 3700us) log.cc:844] Serialized 26 byte log entry
0129 02:59:42.856363 (+1365662us) raft_consensus.cc:1645] Marking committed up to 3
0129 02:59:43.276433 (+420070us) log.cc:844] Serialized 14 byte log entry
0129 02:59:43.494039 (+217606us) raft_consensus.cc:1697] Filling consensus response to leader.
0129 02:59:43.494178 (+ 139us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0129 02:59:43.494261 (+ 83us) raft_consensus.cc:1685] finished
0129 02:59:43.494275 (+ 14us) raft_consensus.cc:1691] UpdateReplicas() finished
0129 02:59:43.512217 (+ 17942us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":854,"prepare.queue_time_us":162704,"prepare.run_cpu_time_us":41,"prepare.run_wall_time_us":41,"raft.queue_time_us":157998,"raft.run_cpu_time_us":329,"raft.run_wall_time_us":77171,"thread_start_us":1613995,"threads_started":2,"wal-append.queue_time_us":1454584}
I20260129 02:59:43.519589 20514 rpcz_store.cc:276] 0129 02:59:41.715675 (+ 0us) service_pool.cc:168] Inserting onto call queue
0129 02:59:41.724424 (+ 8749us) service_pool.cc:225] Handling call
0129 02:59:41.742452 (+ 18028us) raft_consensus.cc:1471] Updating replica for 0 ops
0129 02:59:41.742770 (+ 318us) raft_consensus.cc:1534] Early marking committed up to index 5
0129 02:59:43.241477 (+1498707us) log.cc:844] Serialized 14 byte log entry
0129 02:59:43.493798 (+252321us) log.cc:844] Serialized 14 byte log entry
0129 02:59:43.494039 (+ 241us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0129 02:59:43.494115 (+ 76us) raft_consensus.cc:1645] Marking committed up to 5
0129 02:59:43.494202 (+ 87us) raft_consensus.cc:1697] Filling consensus response to leader.
0129 02:59:43.494328 (+ 126us) raft_consensus.cc:1691] UpdateReplicas() finished
0129 02:59:43.494460 (+ 132us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":1274957,"prepare.run_cpu_time_us":96,"prepare.run_wall_time_us":99,"raft.queue_time_us":2309,"raft.run_cpu_time_us":259,"raft.run_wall_time_us":259,"thread_start_us":1645039,"threads_started":2}
I20260129 02:59:43.494766 22437 raft_consensus.cc:493] T 75be4bd6bd45474795fd472eff2953ac P 974b802141bd4f13beffef2aa03f88f9 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 1c50de89a5a845668696dcb28c3badd6)
I20260129 02:59:43.345161 21134 rpcz_store.cc:276] 0129 02:59:40.378725 (+ 0us) service_pool.cc:168] Inserting onto call queue
0129 02:59:40.378881 (+ 156us) service_pool.cc:225] Handling call
0129 02:59:40.462455 (+ 83574us) raft_consensus.cc:1471] Updating replica for 1 ops
0129 02:59:40.616090 (+153635us) raft_consensus.cc:1534] Early marking committed up to index 0
0129 02:59:40.616135 (+ 45us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0129 02:59:40.616658 (+ 523us) log.cc:844] Serialized 26 byte log entry
0129 02:59:42.258376 (+1641718us) raft_consensus.cc:1645] Marking committed up to 5
0129 02:59:42.258435 (+ 59us) raft_consensus.cc:1697] Filling consensus response to leader.
0129 02:59:43.094074 (+835639us) spinlock_profiling.cc:243] Waited 153 ms on lock 0x7b54005dedd4. stack: 00007f8fe044e373 00007f8fe044e270 00007f8fdfda0574 00007f8fe9a5c6b4 00007f8fe9a5c649 00007f8fe34975a1 00007f8fe352ea81 00007f8fe352d4be 00007f8fe84e1f49 00007f8fe1a6ed34 00007f8fe1a6ecc2 00007f8fe1a6ebf1 00007f8fe1a6eb6c 00007f8fe1a6de22 00007f8fe186d34c 00007f8fe186c786
0129 02:59:43.094167 (+ 93us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0129 02:59:43.232091 (+137924us) raft_consensus.cc:1685] finished
0129 02:59:43.232117 (+ 26us) raft_consensus.cc:1691] UpdateReplicas() finished
0129 02:59:43.232578 (+ 461us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":352579968,"thread_start_us":1590078,"threads_started":1,"wal-append.queue_time_us":2231784}
W20260129 02:59:43.579069 21099 leader_election.cc:336] T 3ccac82a01a2445fa5b0ce6c0fcf2295 P 7a9daf5d42a84615a6383cfec9a10608 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 197b880fa39d432a9b472845876bfaf0 (127.18.211.1:42027): Timed out: RequestConsensusVote RPC to 127.18.211.1:42027 timed out after 1.914s (SENT)
I20260129 02:59:43.380635 22198 leader_election.cc:290] T 913fc7d751ed4b7388d889aac30a5f9b P 197b880fa39d432a9b472845876bfaf0 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 974b802141bd4f13beffef2aa03f88f9 (127.18.211.2:34517), 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3:37735)
I20260129 02:59:43.507313 22440 leader_election.cc:290] T 724a236524cb4e7ba3428880681d6cc2 P 197b880fa39d432a9b472845876bfaf0 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 974b802141bd4f13beffef2aa03f88f9 (127.18.211.2:34517), 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3:37735)
I20260129 02:59:43.556890 20441 raft_consensus.cc:2468] T 3906b080a5b34a74b8b64128635e8997 P 197b880fa39d432a9b472845876bfaf0 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 974b802141bd4f13beffef2aa03f88f9 in term 2.
I20260129 02:59:43.396737 22184 leader_election.cc:290] T 3e2d15f8640b407195210c476e09df91 P 974b802141bd4f13beffef2aa03f88f9 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3:37735), 7a9daf5d42a84615a6383cfec9a10608 (127.18.211.4:40439)
I20260129 02:59:43.495662 20509 raft_consensus.cc:1275] T 4f5a77e94fef4b16a2ba7b067c8b144a P 974b802141bd4f13beffef2aa03f88f9 [term 3 FOLLOWER]: Refusing update from remote peer 1c50de89a5a845668696dcb28c3badd6: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 3 index: 5. (index mismatch)
I20260129 02:59:43.397046 22425 raft_consensus.cc:3055] T 368953c9102643b2a21f99af36b51905 P 1c50de89a5a845668696dcb28c3badd6 [term 4 LEADER]: Stepping down as leader of term 4
I20260129 02:59:43.387204 21128 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34046 (request call id 433) took 1904 ms. Trace:
I20260129 02:59:43.571919 21131 rpcz_store.cc:276] 0129 02:59:41.456785 (+ 0us) service_pool.cc:168] Inserting onto call queue
0129 02:59:43.346300 (+1889515us) service_pool.cc:225] Handling call
0129 02:59:43.384882 (+ 38582us) raft_consensus.cc:1471] Updating replica for 0 ops
0129 02:59:43.529877 (+144995us) raft_consensus.cc:1697] Filling consensus response to leader.
0129 02:59:43.531046 (+ 1169us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260129 02:59:43.577152 20585 raft_consensus.cc:614] T 926a250fa5834e88b96ccecbefbf5086 P 1c50de89a5a845668696dcb28c3badd6 [term 5 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260129 02:59:43.525054 21132 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:47844 (request call id 296) took 2337 ms. Trace:
I20260129 02:59:43.540788 22552 consensus_queue.cc:260] T 1ce4ddf5b10e4ae8b193b13b2e5936ee P 1c50de89a5a845668696dcb28c3badd6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 4.3, Last appended by leader: 3, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:59:43.560338 22504 raft_consensus.cc:515] T dc2f4cb752cf4d59b5f6bc689f5d8104 P 1c50de89a5a845668696dcb28c3badd6 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } }
W20260129 02:59:43.579774 20479 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.211.1:42027, user_credentials={real_user=slave}} blocked reactor thread for 683960us
I20260129 02:59:43.559484 22571 raft_consensus.cc:2749] T a4c1226b987b468fa84dfc6ba0e2dbe1 P 197b880fa39d432a9b472845876bfaf0 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260129 02:59:43.394261 22566 raft_consensus.cc:515] T cb0b3f259ebc4310927338a8740e87b8 P 974b802141bd4f13beffef2aa03f88f9 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } }
I20260129 02:59:43.608184 22504 leader_election.cc:290] T dc2f4cb752cf4d59b5f6bc689f5d8104 P 1c50de89a5a845668696dcb28c3badd6 [CANDIDATE]: Term 4 election: Requested vote from peers 197b880fa39d432a9b472845876bfaf0 (127.18.211.1:42027), 7a9daf5d42a84615a6383cfec9a10608 (127.18.211.4:40439)
I20260129 02:59:43.609346 20584 raft_consensus.cc:1240] T dc2f4cb752cf4d59b5f6bc689f5d8104 P 1c50de89a5a845668696dcb28c3badd6 [term 4 FOLLOWER]: Rejecting Update request from peer 7a9daf5d42a84615a6383cfec9a10608 for earlier term 3. Current term is 4. Ops: []
I20260129 02:59:43.560475 20438 raft_consensus.cc:2468] T 913fc7d751ed4b7388d889aac30a5f9b P 197b880fa39d432a9b472845876bfaf0 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 974b802141bd4f13beffef2aa03f88f9 in term 2.
I20260129 02:59:43.610879 20438 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55672 (request call id 1465) took 1391 ms. Trace:
I20260129 02:59:43.273087 22493 raft_consensus.cc:515] T 8055232cf0004d40a85f3bf983fe00a9 P 974b802141bd4f13beffef2aa03f88f9 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
W20260129 02:59:43.563236 20406 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260129 02:59:43.580432 20586 raft_consensus.cc:1275] T fcddab6bca684adf8ea032289a00f90e P 1c50de89a5a845668696dcb28c3badd6 [term 7 FOLLOWER]: Refusing update from remote peer 197b880fa39d432a9b472845876bfaf0: Log matching property violated. Preceding OpId in replica: term: 6 index: 5. Preceding OpId from leader: term: 7 index: 5. (term mismatch)
W20260129 02:59:43.578979 21102 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.211.2:34517, user_credentials={real_user=slave}} blocked reactor thread for 186501us
I20260129 02:59:43.590344 20441 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55672 (request call id 1463) took 1422 ms. Trace:
I20260129 02:59:43.531466 21135 rpcz_store.cc:276] 0129 02:59:40.226444 (+ 0us) service_pool.cc:168] Inserting onto call queue
0129 02:59:40.226591 (+ 147us) service_pool.cc:225] Handling call
0129 02:59:40.226980 (+ 389us) raft_consensus.cc:1471] Updating replica for 0 ops
0129 02:59:40.227247 (+ 267us) raft_consensus.cc:1534] Early marking committed up to index 6
0129 02:59:41.641239 (+1413992us) log.cc:844] Serialized 14 byte log entry
0129 02:59:43.252167 (+1610928us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0129 02:59:43.291565 (+ 39398us) raft_consensus.cc:1645] Marking committed up to 6
0129 02:59:43.345236 (+ 53671us) raft_consensus.cc:1697] Filling consensus response to leader.
0129 02:59:43.368980 (+ 23744us) raft_consensus.cc:1691] UpdateReplicas() finished
0129 02:59:43.371741 (+ 2761us) inbound_call.cc:173] Queueing success response
0129 02:59:43.508567 (+136826us) spinlock_profiling.cc:243] Waited 42.4 ms on lock 0x7b4800061458. stack: 00007f8fe044e373 00007f8fe044e270 00007f8fdfda0574 00007f8fe9a5c6b4 00007f8fe9a5c649 00007f8fe8447e21 00007f8fe843eeb7 00007f8fe843ed4a 00007f8fe8535f7f 00007f8fe8535c47 00007f8fe854a26c 00007f8fe854a218 00007f8fe854a198 00007f8fe854a153 00007f8fe85493fc 00007f8fe74845b6
Metrics: {"prepare.queue_time_us":2005210,"prepare.run_cpu_time_us":38,"prepare.run_wall_time_us":39,"raft.queue_time_us":398021,"raft.run_cpu_time_us":1102,"raft.run_wall_time_us":167196,"spinlock_wait_cycles":97539968,"thread_start_us":2978452,"threads_started":3,"wal-append.queue_time_us":1356697}
I20260129 02:59:43.514446 20516 rpcz_store.cc:276] 0129 02:59:42.159617 (+ 0us) service_pool.cc:168] Inserting onto call queue
0129 02:59:42.706250 (+546633us) service_pool.cc:225] Handling call
0129 02:59:42.706678 (+ 428us) raft_consensus.cc:1471] Updating replica for 0 ops
0129 02:59:42.706940 (+ 262us) raft_consensus.cc:1534] Early marking committed up to index 2
0129 02:59:43.224819 (+517879us) log.cc:844] Serialized 14 byte log entry
0129 02:59:43.488298 (+263479us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0129 02:59:43.488405 (+ 107us) raft_consensus.cc:1645] Marking committed up to 2
0129 02:59:43.488555 (+ 150us) raft_consensus.cc:1697] Filling consensus response to leader.
0129 02:59:43.488676 (+ 121us) raft_consensus.cc:1691] UpdateReplicas() finished
0129 02:59:43.488839 (+ 163us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":88727,"prepare.queue_time_us":311860,"prepare.run_cpu_time_us":35,"prepare.run_wall_time_us":36,"raft.queue_time_us":158955,"raft.run_cpu_time_us":207,"raft.run_wall_time_us":209,"thread_start_us":495399,"threads_started":2,"wal-append.queue_time_us":132344}
I20260129 02:59:43.580271 20435 raft_consensus.cc:2468] T 3ccac82a01a2445fa5b0ce6c0fcf2295 P 197b880fa39d432a9b472845876bfaf0 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7a9daf5d42a84615a6383cfec9a10608 in term 2.
I20260129 02:59:43.594070 20511 raft_consensus.cc:2468] T cb4727386a3e48aca5d71cc6c611c197 P 974b802141bd4f13beffef2aa03f88f9 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1c50de89a5a845668696dcb28c3badd6 in term 2.
I20260129 02:59:43.561110 22553 raft_consensus.cc:3055] T cedce331e1584ac8b636c9b8a72b2ba0 P 1c50de89a5a845668696dcb28c3badd6 [term 4 LEADER]: Stepping down as leader of term 4
I20260129 02:59:43.534935 22237 raft_consensus.cc:515] T febffe4a8b394368b69b1d164308fb9a P 197b880fa39d432a9b472845876bfaf0 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } }
I20260129 02:59:43.283613 22540 consensus_queue.cc:260] T c934d31efd3f4e5195605ddc354d14e1 P 197b880fa39d432a9b472845876bfaf0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 3.6, Last appended by leader: 6, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } }
I20260129 02:59:43.611411 20438 rpcz_store.cc:276] 0129 02:59:42.220953 (+ 0us) service_pool.cc:168] Inserting onto call queue
0129 02:59:43.372963 (+1152010us) service_pool.cc:225] Handling call
0129 02:59:43.610778 (+237815us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260129 02:59:43.530560 22416 raft_consensus.cc:515] T 724a236524cb4e7ba3428880681d6cc2 P 974b802141bd4f13beffef2aa03f88f9 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:59:43.574048 22343 raft_consensus.cc:515] T fd8311b779af4e1fa1402b2a02a602f2 P 7a9daf5d42a84615a6383cfec9a10608 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } }
I20260129 02:59:43.632761 20432 raft_consensus.cc:2393] T 2e58e9a5a8aa4cecafb464addb4f35ed P 197b880fa39d432a9b472845876bfaf0 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 7a9daf5d42a84615a6383cfec9a10608 in current term 4: Already voted for candidate 197b880fa39d432a9b472845876bfaf0 in this term.
I20260129 02:59:43.663655 20515 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a644c02b16af47679824ccd060ac10fd" candidate_uuid: "1c50de89a5a845668696dcb28c3badd6" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "974b802141bd4f13beffef2aa03f88f9" is_pre_election: true
I20260129 02:59:43.595405 20509 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57584 (request call id 1259) took 1477 ms. Trace:
I20260129 02:59:43.677958 20441 rpcz_store.cc:276] 0129 02:59:42.167028 (+ 0us) service_pool.cc:168] Inserting onto call queue
0129 02:59:43.539350 (+1372322us) service_pool.cc:225] Handling call
0129 02:59:43.589493 (+ 50143us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260129 02:59:43.677806 20514 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "75b0b6e82ddc4f92b1fe1d94f9a0b73b" candidate_uuid: "1c50de89a5a845668696dcb28c3badd6" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "974b802141bd4f13beffef2aa03f88f9" is_pre_election: true
I20260129 02:59:43.674963 20433 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2ac659d732314e1288849f08fe5e074e" candidate_uuid: "974b802141bd4f13beffef2aa03f88f9" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "197b880fa39d432a9b472845876bfaf0" is_pre_election: true
I20260129 02:59:43.677369 20586 pending_rounds.cc:85] T fcddab6bca684adf8ea032289a00f90e P 1c50de89a5a845668696dcb28c3badd6: Aborting all ops after (but not including) 4
I20260129 02:59:43.592196 22425 raft_consensus.cc:740] T 368953c9102643b2a21f99af36b51905 P 1c50de89a5a845668696dcb28c3badd6 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 1c50de89a5a845668696dcb28c3badd6, State: Running, Role: LEADER
W20260129 02:59:43.679019 20435 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:48356 (request call id 267) took 1449 ms (client timeout 1914 ms). Trace:
W20260129 02:59:43.684597 20435 rpcz_store.cc:269] 0129 02:59:42.229350 (+ 0us) service_pool.cc:168] Inserting onto call queue
0129 02:59:43.375071 (+1145721us) service_pool.cc:225] Handling call
0129 02:59:43.678962 (+303891us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20260129 02:59:43.679672 20432 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:48356 (request call id 265) took 1506 ms (client timeout 1958 ms). Trace:
W20260129 02:59:43.685442 20432 rpcz_store.cc:269] 0129 02:59:42.173828 (+ 0us) service_pool.cc:168] Inserting onto call queue
0129 02:59:43.303795 (+1129967us) service_pool.cc:225] Handling call
0129 02:59:43.679613 (+375818us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260129 02:59:43.669338 20507 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "cb0b3f259ebc4310927338a8740e87b8" candidate_uuid: "1c50de89a5a845668696dcb28c3badd6" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "974b802141bd4f13beffef2aa03f88f9" is_pre_election: true
I20260129 02:59:43.682862 20586 pending_rounds.cc:107] T fcddab6bca684adf8ea032289a00f90e P 1c50de89a5a845668696dcb28c3badd6: Aborting uncommitted NO_OP operation due to leader change: 6.5
I20260129 02:59:43.687065 20586 raft_consensus.cc:2889] T fcddab6bca684adf8ea032289a00f90e P 1c50de89a5a845668696dcb28c3badd6 [term 7 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260129 02:59:43.684427 20439 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a609ac02fae84725ab743f95f51d33ce" candidate_uuid: "7a9daf5d42a84615a6383cfec9a10608" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "197b880fa39d432a9b472845876bfaf0" is_pre_election: true
I20260129 02:59:43.686337 20507 raft_consensus.cc:2468] T cb0b3f259ebc4310927338a8740e87b8 P 974b802141bd4f13beffef2aa03f88f9 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1c50de89a5a845668696dcb28c3badd6 in term 5.
W20260129 02:59:43.691229 20405 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.211.3:37735, user_credentials={real_user=slave}} blocked reactor thread for 411618us
I20260129 02:59:43.673796 22552 raft_consensus.cc:3060] T 1ce4ddf5b10e4ae8b193b13b2e5936ee P 1c50de89a5a845668696dcb28c3badd6 [term 4 FOLLOWER]: Advancing to term 5
I20260129 02:59:43.594612 21132 rpcz_store.cc:276] 0129 02:59:41.187305 (+ 0us) service_pool.cc:168] Inserting onto call queue
0129 02:59:43.249148 (+2061843us) service_pool.cc:225] Handling call
0129 02:59:43.516442 (+267294us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260129 02:59:43.649047 22445 leader_election.cc:290] T cedce331e1584ac8b636c9b8a72b2ba0 P 974b802141bd4f13beffef2aa03f88f9 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 197b880fa39d432a9b472845876bfaf0 (127.18.211.1:42027), 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3:37735)
I20260129 02:59:43.678469 22553 raft_consensus.cc:740] T cedce331e1584ac8b636c9b8a72b2ba0 P 1c50de89a5a845668696dcb28c3badd6 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 1c50de89a5a845668696dcb28c3badd6, State: Running, Role: LEADER
W20260129 02:59:43.573467 20434 rpcz_store.cc:269] 0129 02:59:37.397516 (+ 0us) service_pool.cc:168] Inserting onto call queue
0129 02:59:37.456007 (+ 58491us) service_pool.cc:225] Handling call
0129 02:59:43.395748 (+5939741us) spinlock_profiling.cc:243] Waited 3.09 s on lock 0x7b54002e1654. stack: 00007f8fe044e373 00007f8fe044e270 00007f8fdfda0574 00007f8fe9a5c6b4 00007f8fe9a5c649 00007f8fe34975a1 00007f8fe3531f7a 00007f8fe84e3647 00007f8fe1a73834 00007f8fe1a737c2 00007f8fe1a736f1 00007f8fe1a7366c 00007f8fe1a72922 00007f8fe186d34c 00007f8fe186c786 00007f8fe186c12f
0129 02:59:43.395995 (+ 247us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":7113058176}
I20260129 02:59:43.688189 21100 leader_election.cc:304] T 42d27a28d89b409e8807ec0efd97af69 P 7a9daf5d42a84615a6383cfec9a10608 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1c50de89a5a845668696dcb28c3badd6, 7a9daf5d42a84615a6383cfec9a10608; no voters:
I20260129 02:59:43.594388 21128 rpcz_store.cc:276] 0129 02:59:41.482833 (+ 0us) service_pool.cc:168] Inserting onto call queue
0129 02:59:43.346245 (+1863412us) service_pool.cc:225] Handling call
0129 02:59:43.386376 (+ 40131us) raft_consensus.cc:1471] Updating replica for 0 ops
0129 02:59:43.386756 (+ 380us) raft_consensus.cc:1534] Early marking committed up to index 0
0129 02:59:43.386787 (+ 31us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0129 02:59:43.386858 (+ 71us) raft_consensus.cc:1645] Marking committed up to 4
0129 02:59:43.386918 (+ 60us) raft_consensus.cc:1697] Filling consensus response to leader.
0129 02:59:43.386988 (+ 70us) raft_consensus.cc:1691] UpdateReplicas() finished
0129 02:59:43.387120 (+ 132us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260129 02:59:43.595729 22231 raft_consensus.cc:515] T 96609ac68bdf4993a12e1c23355ca013 P 197b880fa39d432a9b472845876bfaf0 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } attrs { replace: true } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } }
I20260129 02:59:43.682885 22197 leader_election.cc:290] T cb4727386a3e48aca5d71cc6c611c197 P 974b802141bd4f13beffef2aa03f88f9 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 197b880fa39d432a9b472845876bfaf0 (127.18.211.1:42027), 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3:37735)
I20260129 02:59:43.699458 20433 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55672 (request call id 1466) took 1459 ms. Trace:
I20260129 02:59:43.700536 21130 raft_consensus.cc:2393] T fd8311b779af4e1fa1402b2a02a602f2 P 7a9daf5d42a84615a6383cfec9a10608 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1c50de89a5a845668696dcb28c3badd6 in current term 4: Already voted for candidate 7a9daf5d42a84615a6383cfec9a10608 in this term.
W20260129 02:59:43.695209 21102 leader_election.cc:336] T 505580c93c694bbb8ea96fdf9b17ab46 P 7a9daf5d42a84615a6383cfec9a10608 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 974b802141bd4f13beffef2aa03f88f9 (127.18.211.2:34517): Timed out: RequestConsensusVote RPC to 127.18.211.2:34517 timed out after 3.068s (SENT)
I20260129 02:59:43.700851 20439 raft_consensus.cc:2468] T a609ac02fae84725ab743f95f51d33ce P 197b880fa39d432a9b472845876bfaf0 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7a9daf5d42a84615a6383cfec9a10608 in term 5.
I20260129 02:59:43.704164 20515 raft_consensus.cc:2376] T a644c02b16af47679824ccd060ac10fd P 974b802141bd4f13beffef2aa03f88f9 [term 3 FOLLOWER]: Leader pre-election vote request: Already granted yes vote for candidate 1c50de89a5a845668696dcb28c3badd6 in term 3. Re-sending same reply.
I20260129 02:59:43.648175 22160 raft_consensus.cc:2749] T 36ff33be666a4c82954731691dbcf327 P 974b802141bd4f13beffef2aa03f88f9 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260129 02:59:43.706180 20507 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a5a05fbcbbcf4916bbc01fd1a5b5e90d" candidate_uuid: "1c50de89a5a845668696dcb28c3badd6" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "974b802141bd4f13beffef2aa03f88f9" is_pre_election: true
I20260129 02:59:43.690588 22566 leader_election.cc:290] T cb0b3f259ebc4310927338a8740e87b8 P 974b802141bd4f13beffef2aa03f88f9 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3:37735), 7a9daf5d42a84615a6383cfec9a10608 (127.18.211.4:40439)
I20260129 02:59:43.644112 22184 raft_consensus.cc:493] T 7d9494f5584d4d5db8b9adf1c8fc888b P 974b802141bd4f13beffef2aa03f88f9 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 1c50de89a5a845668696dcb28c3badd6)
I20260129 02:59:43.709895 22520 leader_election.cc:290] T b189f08e16a74a11980cc55d99359a0a P 197b880fa39d432a9b472845876bfaf0 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 974b802141bd4f13beffef2aa03f88f9 (127.18.211.2:34517), 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3:37735)
I20260129 02:59:43.709012 20433 rpcz_store.cc:276] 0129 02:59:42.239460 (+ 0us) service_pool.cc:168] Inserting onto call queue
0129 02:59:43.600540 (+1361080us) service_pool.cc:225] Handling call
0129 02:59:43.696912 (+ 96372us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20260129 02:59:43.710776 20439 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:48356 (request call id 259) took 1633 ms (client timeout 2131 ms). Trace:
I20260129 02:59:43.664891 22437 raft_consensus.cc:515] T 75be4bd6bd45474795fd472eff2953ac P 974b802141bd4f13beffef2aa03f88f9 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:59:43.681694 20509 rpcz_store.cc:276] 0129 02:59:42.118452 (+ 0us) service_pool.cc:168] Inserting onto call queue
0129 02:59:42.474069 (+355617us) service_pool.cc:225] Handling call
0129 02:59:42.792346 (+318277us) raft_consensus.cc:1471] Updating replica for 0 ops
0129 02:59:43.591825 (+799479us) raft_consensus.cc:1697] Filling consensus response to leader.
0129 02:59:43.595329 (+ 3504us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260129 02:59:43.710757 20514 raft_consensus.cc:2468] T 75b0b6e82ddc4f92b1fe1d94f9a0b73b P 974b802141bd4f13beffef2aa03f88f9 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1c50de89a5a845668696dcb28c3badd6 in term 2.
I20260129 02:59:43.717219 20438 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0340229d521b432580ae926f6062e3f7" candidate_uuid: "974b802141bd4f13beffef2aa03f88f9" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "197b880fa39d432a9b472845876bfaf0" is_pre_election: true
I20260129 02:59:43.705678 20511 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3ccac82a01a2445fa5b0ce6c0fcf2295" candidate_uuid: "197b880fa39d432a9b472845876bfaf0" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "974b802141bd4f13beffef2aa03f88f9" is_pre_election: true
W20260129 02:59:43.715832 20439 rpcz_store.cc:269] 0129 02:59:42.077488 (+ 0us) service_pool.cc:168] Inserting onto call queue
0129 02:59:43.664418 (+1586930us) service_pool.cc:225] Handling call
0129 02:59:43.710715 (+ 46297us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260129 02:59:43.713415 21102 leader_election.cc:304] T 505580c93c694bbb8ea96fdf9b17ab46 P 7a9daf5d42a84615a6383cfec9a10608 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7a9daf5d42a84615a6383cfec9a10608; no voters: 1c50de89a5a845668696dcb28c3badd6, 974b802141bd4f13beffef2aa03f88f9
W20260129 02:59:43.716868 21100 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.211.3:37735, user_credentials={real_user=slave}} blocked reactor thread for 85028us
I20260129 02:59:43.720321 20516 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a0fcc58d6fb6473cafbd9d34d5195269" candidate_uuid: "1c50de89a5a845668696dcb28c3badd6" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "974b802141bd4f13beffef2aa03f88f9" is_pre_election: true
I20260129 02:59:43.721200 20512 raft_consensus.cc:2393] T 8055232cf0004d40a85f3bf983fe00a9 P 974b802141bd4f13beffef2aa03f88f9 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1c50de89a5a845668696dcb28c3badd6 in current term 3: Already voted for candidate 974b802141bd4f13beffef2aa03f88f9 in this term.
I20260129 02:59:43.721115 20513 raft_consensus.cc:1240] T 724a236524cb4e7ba3428880681d6cc2 P 974b802141bd4f13beffef2aa03f88f9 [term 3 FOLLOWER]: Rejecting Update request from peer 1c50de89a5a845668696dcb28c3badd6 for earlier term 2. Current term is 3. Ops: [2.2-2.2]
I20260129 02:59:43.723556 20508 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57584 (request call id 1261) took 1514 ms. Trace:
I20260129 02:59:43.718010 22554 raft_consensus.cc:3055] T 78aa39096868415d9afc2ffcb248e198 P 1c50de89a5a845668696dcb28c3badd6 [term 3 LEADER]: Stepping down as leader of term 3
I20260129 02:59:43.712059 22586 raft_consensus.cc:2804] T 368953c9102643b2a21f99af36b51905 P 974b802141bd4f13beffef2aa03f88f9 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260129 02:59:43.725378 22511 raft_consensus.cc:493] T fcddab6bca684adf8ea032289a00f90e P 1c50de89a5a845668696dcb28c3badd6 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260129 02:59:43.727216 20515 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "add8d4923d1a489e86882c3a25692b95" candidate_uuid: "1c50de89a5a845668696dcb28c3badd6" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "974b802141bd4f13beffef2aa03f88f9" is_pre_election: true
I20260129 02:59:43.727288 20514 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "724a236524cb4e7ba3428880681d6cc2" candidate_uuid: "197b880fa39d432a9b472845876bfaf0" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "974b802141bd4f13beffef2aa03f88f9" is_pre_election: true
I20260129 02:59:43.726702 22590 raft_consensus.cc:2804] T 42d27a28d89b409e8807ec0efd97af69 P 7a9daf5d42a84615a6383cfec9a10608 [term 5 FOLLOWER]: Leader election won for term 5
I20260129 02:59:43.732219 22590 raft_consensus.cc:697] T 42d27a28d89b409e8807ec0efd97af69 P 7a9daf5d42a84615a6383cfec9a10608 [term 5 LEADER]: Becoming Leader. State: Replica: 7a9daf5d42a84615a6383cfec9a10608, State: Running, Role: LEADER
I20260129 02:59:43.698676 20590 raft_consensus.cc:2955] T 1299723a4476420ab02fd8bfd7aa0645 P 1c50de89a5a845668696dcb28c3badd6 [term 6 FOLLOWER]: Committing config change with OpId 2.5: config changed from index 4 to 5, VOTER 974b802141bd4f13beffef2aa03f88f9 (127.18.211.2) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } } }
I20260129 02:59:43.719035 22577 leader_election.cc:290] T 96609ac68bdf4993a12e1c23355ca013 P 7a9daf5d42a84615a6383cfec9a10608 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 197b880fa39d432a9b472845876bfaf0 (127.18.211.1:42027), 974b802141bd4f13beffef2aa03f88f9 (127.18.211.2:34517), 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3:37735)
I20260129 02:59:43.734285 20507 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e7d83d4ccf684e76acb29a0dc844e1b8" candidate_uuid: "197b880fa39d432a9b472845876bfaf0" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "974b802141bd4f13beffef2aa03f88f9" is_pre_election: true
I20260129 02:59:43.734869 22342 raft_consensus.cc:2749] T 505580c93c694bbb8ea96fdf9b17ab46 P 7a9daf5d42a84615a6383cfec9a10608 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260129 02:59:43.725556 22554 raft_consensus.cc:740] T 78aa39096868415d9afc2ffcb248e198 P 1c50de89a5a845668696dcb28c3badd6 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 1c50de89a5a845668696dcb28c3badd6, State: Running, Role: LEADER
I20260129 02:59:43.731858 22511 raft_consensus.cc:515] T fcddab6bca684adf8ea032289a00f90e P 1c50de89a5a845668696dcb28c3badd6 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:59:43.728804 22540 raft_consensus.cc:3060] T c934d31efd3f4e5195605ddc354d14e1 P 197b880fa39d432a9b472845876bfaf0 [term 3 FOLLOWER]: Advancing to term 4
I20260129 02:59:43.739945 20434 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "96609ac68bdf4993a12e1c23355ca013" candidate_uuid: "974b802141bd4f13beffef2aa03f88f9" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "197b880fa39d432a9b472845876bfaf0" is_pre_election: true
I20260129 02:59:43.744992 20434 raft_consensus.cc:2468] T 96609ac68bdf4993a12e1c23355ca013 P 197b880fa39d432a9b472845876bfaf0 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 974b802141bd4f13beffef2aa03f88f9 in term 4.
I20260129 02:59:43.744939 22364 raft_consensus.cc:515] T 2e58e9a5a8aa4cecafb464addb4f35ed P 7a9daf5d42a84615a6383cfec9a10608 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } }
I20260129 02:59:43.744333 20510 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57584 (request call id 1247) took 2288 ms. Trace:
I20260129 02:59:43.745476 20512 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:57584 (request call id 1257) took 1920 ms. Trace:
I20260129 02:59:43.743647 20439 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "42d27a28d89b409e8807ec0efd97af69" candidate_uuid: "7a9daf5d42a84615a6383cfec9a10608" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "197b880fa39d432a9b472845876bfaf0"
I20260129 02:59:43.550521 22403 raft_consensus.cc:2955] T 1299723a4476420ab02fd8bfd7aa0645 P 7a9daf5d42a84615a6383cfec9a10608 [term 6 LEADER]: Committing config change with OpId 2.5: config changed from index 4 to 5, VOTER 974b802141bd4f13beffef2aa03f88f9 (127.18.211.2) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } } }
I20260129 02:59:43.724035 20508 rpcz_store.cc:276] 0129 02:59:42.150117 (+ 0us) service_pool.cc:168] Inserting onto call queue
0129 02:59:42.724717 (+574600us) service_pool.cc:225] Handling call
0129 02:59:42.725062 (+ 345us) raft_consensus.cc:1471] Updating replica for 1 ops
0129 02:59:42.725409 (+ 347us) raft_consensus.cc:1534] Early marking committed up to index 0
0129 02:59:42.725436 (+ 27us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0129 02:59:42.753457 (+ 28021us) log.cc:844] Serialized 26 byte log entry
0129 02:59:43.089947 (+336490us) raft_consensus.cc:1645] Marking committed up to 2
0129 02:59:43.090825 (+ 878us) log.cc:844] Serialized 14 byte log entry
0129 02:59:43.169936 (+ 79111us) raft_consensus.cc:1697] Filling consensus response to leader.
0129 02:59:43.636273 (+466337us) spinlock_profiling.cc:243] Waited 38.5 ms on lock 0x7b5400039420. stack: 00007f8fe044e373 00007f8fe044e270 00007f8fdfda0574 00007f8fe9a5c6b4 00007f8fe9a5c649 00007f8fe34975a1 00007f8fe34a5df0 00007f8fe3531682 00007f8fe352ea5e 00007f8fe352d4be 00007f8fe84e1f49 00007f8fe1a6ed34 00007f8fe1a6ecc2 00007f8fe1a6ebf1 00007f8fe1a6eb6c 00007f8fe1a6de22
0129 02:59:43.636389 (+ 116us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0129 02:59:43.636439 (+ 50us) raft_consensus.cc:1685] finished
0129 02:59:43.659616 (+ 23177us) raft_consensus.cc:1691] UpdateReplicas() finished
0129 02:59:43.664688 (+ 5072us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":27136,"prepare.queue_time_us":72107,"prepare.run_cpu_time_us":118,"prepare.run_wall_time_us":6181,"raft.queue_time_us":117566,"raft.run_cpu_time_us":569,"raft.run_wall_time_us":14878,"spinlock_wait_cycles":88660224,"thread_start_us":335687,"threads_started":1,"wal-append.queue_time_us":264976}
W20260129 02:59:43.754968 20554 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260129 02:59:43.748909 20438 raft_consensus.cc:2468] T 0340229d521b432580ae926f6062e3f7 P 197b880fa39d432a9b472845876bfaf0 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 974b802141bd4f13beffef2aa03f88f9 in term 4.
I20260129 02:59:43.756060 20438 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55672 (request call id 1468) took 1106 ms. Trace:
I20260129 02:59:43.756358 20438 rpcz_store.cc:276] 0129 02:59:42.649926 (+ 0us) service_pool.cc:168] Inserting onto call queue
0129 02:59:43.690956 (+1041030us) service_pool.cc:225] Handling call
0129 02:59:43.756013 (+ 65057us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260129 02:59:43.757097 20514 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "75be4bd6bd45474795fd472eff2953ac" candidate_uuid: "197b880fa39d432a9b472845876bfaf0" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "974b802141bd4f13beffef2aa03f88f9" is_pre_election: true
I20260129 02:59:43.758098 20514 raft_consensus.cc:2468] T 75be4bd6bd45474795fd472eff2953ac P 974b802141bd4f13beffef2aa03f88f9 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 197b880fa39d432a9b472845876bfaf0 in term 7.
I20260129 02:59:43.752312 20511 raft_consensus.cc:2468] T 3ccac82a01a2445fa5b0ce6c0fcf2295 P 974b802141bd4f13beffef2aa03f88f9 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 197b880fa39d432a9b472845876bfaf0 in term 2.
I20260129 02:59:43.761894 20516 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7a9a69cd82724b1faed1d10aa091fc4a" candidate_uuid: "197b880fa39d432a9b472845876bfaf0" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "974b802141bd4f13beffef2aa03f88f9" is_pre_election: true
I20260129 02:59:43.757112 20515 raft_consensus.cc:2468] T add8d4923d1a489e86882c3a25692b95 P 974b802141bd4f13beffef2aa03f88f9 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1c50de89a5a845668696dcb28c3badd6 in term 2.
I20260129 02:59:43.751343 20512 rpcz_store.cc:276] 0129 02:59:41.825105 (+ 0us) service_pool.cc:168] Inserting onto call queue
0129 02:59:42.238494 (+413389us) service_pool.cc:225] Handling call
0129 02:59:43.745234 (+1506740us) spinlock_profiling.cc:243] Waited 1.41 s on lock 0x7b54001b03d4. stack: 00007f8fe044e373 00007f8fe044e270 00007f8fdfda0574 00007f8fe9a5c6b4 00007f8fe9a5c649 00007f8fe34975a1 00007f8fe3531f7a 00007f8fe84e3647 00007f8fe1a73834 00007f8fe1a737c2 00007f8fe1a736f1 00007f8fe1a7366c 00007f8fe1a72922 00007f8fe186d34c 00007f8fe186c786 00007f8fe186c12f
0129 02:59:43.745397 (+ 163us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":3251787648}
I20260129 02:59:43.765895 20514 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "36ff33be666a4c82954731691dbcf327" candidate_uuid: "197b880fa39d432a9b472845876bfaf0" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "974b802141bd4f13beffef2aa03f88f9" is_pre_election: true
I20260129 02:59:43.750332 22416 leader_election.cc:290] T 724a236524cb4e7ba3428880681d6cc2 P 974b802141bd4f13beffef2aa03f88f9 [CANDIDATE]: Term 3 election: Requested vote from peers 197b880fa39d432a9b472845876bfaf0 (127.18.211.1:42027), 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3:37735)
I20260129 02:59:43.766670 20507 raft_consensus.cc:2468] T e7d83d4ccf684e76acb29a0dc844e1b8 P 974b802141bd4f13beffef2aa03f88f9 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 197b880fa39d432a9b472845876bfaf0 in term 3.
I20260129 02:59:43.767355 20508 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "913fc7d751ed4b7388d889aac30a5f9b" candidate_uuid: "197b880fa39d432a9b472845876bfaf0" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "974b802141bd4f13beffef2aa03f88f9" is_pre_election: true
I20260129 02:59:43.743685 20509 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "87d5fe48a44a408aaf03e894a67ecb2f" candidate_uuid: "197b880fa39d432a9b472845876bfaf0" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "974b802141bd4f13beffef2aa03f88f9"
I20260129 02:59:43.757282 20585 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "96609ac68bdf4993a12e1c23355ca013" candidate_uuid: "7a9daf5d42a84615a6383cfec9a10608" candidate_term: 5 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "1c50de89a5a845668696dcb28c3badd6" is_pre_election: true
I20260129 02:59:43.752104 22343 leader_election.cc:290] T fd8311b779af4e1fa1402b2a02a602f2 P 7a9daf5d42a84615a6383cfec9a10608 [CANDIDATE]: Term 4 election: Requested vote from peers 197b880fa39d432a9b472845876bfaf0 (127.18.211.1:42027), 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3:37735)
W20260129 02:59:43.760820 20556 connection.cc:232] RPC call timeout handler was delayed by -1.1915253509999957s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2338s for a response
W20260129 02:59:43.772634 20482 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260129 02:59:43.769047 20479 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.211.1:42027, user_credentials={real_user=slave}} blocked reactor thread for 56159us
I20260129 02:59:43.772104 20582 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fd8311b779af4e1fa1402b2a02a602f2" candidate_uuid: "7a9daf5d42a84615a6383cfec9a10608" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "1c50de89a5a845668696dcb28c3badd6"
I20260129 02:59:43.751248 20510 rpcz_store.cc:276] 0129 02:59:41.456189 (+ 0us) service_pool.cc:168] Inserting onto call queue
0129 02:59:41.456528 (+ 339us) service_pool.cc:225] Handling call
0129 02:59:41.456870 (+ 342us) raft_consensus.cc:1471] Updating replica for 1 ops
0129 02:59:43.304277 (+1847407us) raft_consensus.cc:1534] Early marking committed up to index 0
0129 02:59:43.304329 (+ 52us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0129 02:59:43.305085 (+ 756us) log.cc:844] Serialized 26 byte log entry
0129 02:59:43.694730 (+389645us) raft_consensus.cc:1645] Marking committed up to 2
0129 02:59:43.694787 (+ 57us) raft_consensus.cc:1697] Filling consensus response to leader.
0129 02:59:43.701221 (+ 6434us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0129 02:59:43.737687 (+ 36466us) raft_consensus.cc:1685] finished
0129 02:59:43.737708 (+ 21us) raft_consensus.cc:1691] UpdateReplicas() finished
0129 02:59:43.744254 (+ 6546us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":89191,"raft.run_cpu_time_us":149,"raft.run_wall_time_us":30687,"thread_start_us":2235887,"threads_started":2,"wal-append.queue_time_us":411724}
W20260129 02:59:43.774668 20405 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.211.3:37735, user_credentials={real_user=slave}} blocked reactor thread for 70940.6us
I20260129 02:59:43.770823 20585 raft_consensus.cc:2468] T 96609ac68bdf4993a12e1c23355ca013 P 1c50de89a5a845668696dcb28c3badd6 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7a9daf5d42a84615a6383cfec9a10608 in term 4.
I20260129 02:59:43.776319 20509 raft_consensus.cc:3060] T 87d5fe48a44a408aaf03e894a67ecb2f P 974b802141bd4f13beffef2aa03f88f9 [term 4 FOLLOWER]: Advancing to term 5
I20260129 02:59:43.773892 20508 raft_consensus.cc:2468] T 913fc7d751ed4b7388d889aac30a5f9b P 974b802141bd4f13beffef2aa03f88f9 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 197b880fa39d432a9b472845876bfaf0 in term 2.
I20260129 02:59:43.774603 20511 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "cb4727386a3e48aca5d71cc6c611c197" candidate_uuid: "197b880fa39d432a9b472845876bfaf0" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "974b802141bd4f13beffef2aa03f88f9" is_pre_election: true
I20260129 02:59:43.764943 22237 leader_election.cc:290] T febffe4a8b394368b69b1d164308fb9a P 197b880fa39d432a9b472845876bfaf0 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 974b802141bd4f13beffef2aa03f88f9 (127.18.211.2:34517), 7a9daf5d42a84615a6383cfec9a10608 (127.18.211.4:40439)
I20260129 02:59:43.778035 20511 raft_consensus.cc:2468] T cb4727386a3e48aca5d71cc6c611c197 P 974b802141bd4f13beffef2aa03f88f9 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 197b880fa39d432a9b472845876bfaf0 in term 2.
I20260129 02:59:43.780786 20439 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2ac659d732314e1288849f08fe5e074e" candidate_uuid: "7a9daf5d42a84615a6383cfec9a10608" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "197b880fa39d432a9b472845876bfaf0" is_pre_election: true
I20260129 02:59:43.781041 20514 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e4d18c35b0a7493e85d0a4b92956a48f" candidate_uuid: "197b880fa39d432a9b472845876bfaf0" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "974b802141bd4f13beffef2aa03f88f9"
I20260129 02:59:43.784096 20515 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1ce4ddf5b10e4ae8b193b13b2e5936ee" candidate_uuid: "197b880fa39d432a9b472845876bfaf0" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "974b802141bd4f13beffef2aa03f88f9"
I20260129 02:59:43.789052 20439 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:48356 (request call id 251) took 1954 ms. Trace:
I20260129 02:59:43.789507 20439 rpcz_store.cc:276] 0129 02:59:41.827402 (+ 0us) service_pool.cc:168] Inserting onto call queue
0129 02:59:43.760989 (+1933587us) service_pool.cc:225] Handling call
0129 02:59:43.782018 (+ 21029us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":808832}
I20260129 02:59:43.791337 20438 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b189f08e16a74a11980cc55d99359a0a" candidate_uuid: "1c50de89a5a845668696dcb28c3badd6" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "197b880fa39d432a9b472845876bfaf0" is_pre_election: true
W20260129 02:59:43.776711 21130 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34046 (request call id 431) took 2553 ms (client timeout 1796 ms). Trace:
W20260129 02:59:43.792531 21102 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.211.2:34517, user_credentials={real_user=slave}} blocked reactor thread for 104383us
I20260129 02:59:43.792590 20438 raft_consensus.cc:2468] T b189f08e16a74a11980cc55d99359a0a P 197b880fa39d432a9b472845876bfaf0 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1c50de89a5a845668696dcb28c3badd6 in term 5.
W20260129 02:59:43.794487 20556 connection.cc:232] RPC call timeout handler was delayed by -1.0781253510000397s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2464s for a response
I20260129 02:59:43.784951 20516 raft_consensus.cc:2468] T 7a9a69cd82724b1faed1d10aa091fc4a P 974b802141bd4f13beffef2aa03f88f9 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 197b880fa39d432a9b472845876bfaf0 in term 4.
I20260129 02:59:43.756151 22184 raft_consensus.cc:515] T 7d9494f5584d4d5db8b9adf1c8fc888b P 974b802141bd4f13beffef2aa03f88f9 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:59:43.735627 22590 consensus_queue.cc:237] T 42d27a28d89b409e8807ec0efd97af69 P 7a9daf5d42a84615a6383cfec9a10608 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } }
I20260129 02:59:43.765028 22581 consensus_queue.cc:1059] T 78aa39096868415d9afc2ffcb248e198 P 1c50de89a5a845668696dcb28c3badd6 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false }, Status: INVALID_TERM, Last received: 3.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.029s
I20260129 02:59:43.797106 20440 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e7d83d4ccf684e76acb29a0dc844e1b8" candidate_uuid: "1c50de89a5a845668696dcb28c3badd6" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "197b880fa39d432a9b472845876bfaf0" is_pre_election: true
W20260129 02:59:43.794003 21130 rpcz_store.cc:269] 0129 02:59:41.222838 (+ 0us) service_pool.cc:168] Inserting onto call queue
0129 02:59:41.468467 (+245629us) service_pool.cc:225] Handling call
0129 02:59:43.776426 (+2307959us) spinlock_profiling.cc:243] Waited 2.06 s on lock 0x7b54003da754. stack: 00007f8fe044e373 00007f8fe044e270 00007f8fdfda0574 00007f8fe9a5c6b4 00007f8fe9a5c649 00007f8fe34975a1 00007f8fe3531f7a 00007f8fe84e3647 00007f8fe1a73834 00007f8fe1a737c2 00007f8fe1a736f1 00007f8fe1a7366c 00007f8fe1a72922 00007f8fe186d34c 00007f8fe186c786 00007f8fe186c12f
0129 02:59:43.776615 (+ 189us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":4731541376}
I20260129 02:59:43.798169 21099 leader_election.cc:304] T 3ccac82a01a2445fa5b0ce6c0fcf2295 P 7a9daf5d42a84615a6383cfec9a10608 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7a9daf5d42a84615a6383cfec9a10608; no voters: 197b880fa39d432a9b472845876bfaf0, 974b802141bd4f13beffef2aa03f88f9
I20260129 02:59:43.805574 22342 raft_consensus.cc:2749] T 3ccac82a01a2445fa5b0ce6c0fcf2295 P 7a9daf5d42a84615a6383cfec9a10608 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20260129 02:59:43.805819 21099 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.211.1:42027, user_credentials={real_user=slave}} blocked reactor thread for 269070us
I20260129 02:59:43.798791 20508 raft_consensus.cc:1240] T 505580c93c694bbb8ea96fdf9b17ab46 P 974b802141bd4f13beffef2aa03f88f9 [term 4 FOLLOWER]: Rejecting Update request from peer 1c50de89a5a845668696dcb28c3badd6 for earlier term 3. Current term is 4. Ops: []
I20260129 02:59:43.808768 20508 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57584 (request call id 1266) took 1556 ms. Trace:
I20260129 02:59:43.808858 20479 leader_election.cc:304] T c934d31efd3f4e5195605ddc354d14e1 P 974b802141bd4f13beffef2aa03f88f9 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 974b802141bd4f13beffef2aa03f88f9; no voters: 197b880fa39d432a9b472845876bfaf0, 7a9daf5d42a84615a6383cfec9a10608
I20260129 02:59:43.809260 20508 rpcz_store.cc:276] 0129 02:59:42.252125 (+ 0us) service_pool.cc:168] Inserting onto call queue
0129 02:59:43.783830 (+1531705us) service_pool.cc:225] Handling call
0129 02:59:43.792719 (+ 8889us) raft_consensus.cc:1471] Updating replica for 0 ops
0129 02:59:43.808383 (+ 15664us) raft_consensus.cc:1697] Filling consensus response to leader.
0129 02:59:43.808643 (+ 260us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20260129 02:59:43.810808 22400 consensus_meta.cc:287] T e4d18c35b0a7493e85d0a4b92956a48f P 974b802141bd4f13beffef2aa03f88f9: Time spent flushing consensus metadata: real 0.513s user 0.009s sys 0.004s
I20260129 02:59:43.811192 20508 raft_consensus.cc:1240] T c934d31efd3f4e5195605ddc354d14e1 P 974b802141bd4f13beffef2aa03f88f9 [term 4 FOLLOWER]: Rejecting Update request from peer 197b880fa39d432a9b472845876bfaf0 for earlier term 3. Current term is 4. Ops: [3.6-3.6]
I20260129 02:59:43.814098 20440 raft_consensus.cc:2468] T e7d83d4ccf684e76acb29a0dc844e1b8 P 197b880fa39d432a9b472845876bfaf0 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1c50de89a5a845668696dcb28c3badd6 in term 3.
I20260129 02:59:43.827649 22437 leader_election.cc:290] T 75be4bd6bd45474795fd472eff2953ac P 974b802141bd4f13beffef2aa03f88f9 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 197b880fa39d432a9b472845876bfaf0 (127.18.211.1:42027), 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3:37735)
I20260129 02:59:43.816895 20439 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4e420f90884f4796997b5ea41e49e411" candidate_uuid: "974b802141bd4f13beffef2aa03f88f9" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "197b880fa39d432a9b472845876bfaf0" is_pre_election: true
I20260129 02:59:43.817612 22511 leader_election.cc:290] T fcddab6bca684adf8ea032289a00f90e P 1c50de89a5a845668696dcb28c3badd6 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 197b880fa39d432a9b472845876bfaf0 (127.18.211.1:42027), 974b802141bd4f13beffef2aa03f88f9 (127.18.211.2:34517)
W20260129 02:59:43.821108 22320 consensus_meta.cc:287] T 17d6431cc77a491e8464d68db4081a51 P 1c50de89a5a845668696dcb28c3badd6: Time spent flushing consensus metadata: real 0.569s user 0.011s sys 0.000s
I20260129 02:59:43.824048 20516 raft_consensus.cc:1275] T fcddab6bca684adf8ea032289a00f90e P 974b802141bd4f13beffef2aa03f88f9 [term 7 FOLLOWER]: Refusing update from remote peer 197b880fa39d432a9b472845876bfaf0: Log matching property violated. Preceding OpId in replica: term: 4 index: 4. Preceding OpId from leader: term: 7 index: 5. (index mismatch)
I20260129 02:59:43.825042 20405 leader_election.cc:304] T a5a05fbcbbcf4916bbc01fd1a5b5e90d P 197b880fa39d432a9b472845876bfaf0 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 197b880fa39d432a9b472845876bfaf0, 1c50de89a5a845668696dcb28c3badd6; no voters:
I20260129 02:59:43.794565 20438 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55676 (request call id 1193) took 1634 ms. Trace:
I20260129 02:59:43.853688 20438 rpcz_store.cc:276] 0129 02:59:42.159842 (+ 0us) service_pool.cc:168] Inserting onto call queue
0129 02:59:43.764626 (+1604784us) service_pool.cc:225] Handling call
0129 02:59:43.794503 (+ 29877us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260129 02:59:43.798454 20582 raft_consensus.cc:3060] T fd8311b779af4e1fa1402b2a02a602f2 P 1c50de89a5a845668696dcb28c3badd6 [term 3 FOLLOWER]: Advancing to term 4
I20260129 02:59:43.827171 22586 raft_consensus.cc:493] T 368953c9102643b2a21f99af36b51905 P 974b802141bd4f13beffef2aa03f88f9 [term 4 FOLLOWER]: Starting leader election (detected failure of leader 1c50de89a5a845668696dcb28c3badd6)
W20260129 02:59:43.827963 20440 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55676 (request call id 1197) took 1662 ms (client timeout 2113 ms). Trace:
W20260129 02:59:43.862131 20440 rpcz_store.cc:269] 0129 02:59:42.165164 (+ 0us) service_pool.cc:168] Inserting onto call queue
0129 02:59:43.791539 (+1626375us) service_pool.cc:225] Handling call
0129 02:59:43.827881 (+ 36342us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20260129 02:59:43.829914 20404 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260129 02:59:43.832005 20435 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a8479d1fe0bd41e5946186e3a6ca8c7a" candidate_uuid: "974b802141bd4f13beffef2aa03f88f9" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "197b880fa39d432a9b472845876bfaf0" is_pre_election: true
I20260129 02:59:43.863534 20515 raft_consensus.cc:3060] T 1ce4ddf5b10e4ae8b193b13b2e5936ee P 974b802141bd4f13beffef2aa03f88f9 [term 4 FOLLOWER]: Advancing to term 5
I20260129 02:59:43.834190 20433 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "17d6431cc77a491e8464d68db4081a51" candidate_uuid: "974b802141bd4f13beffef2aa03f88f9" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "197b880fa39d432a9b472845876bfaf0" is_pre_election: true
I20260129 02:59:43.342365 21129 rpcz_store.cc:276] 0129 02:59:40.226051 (+ 0us) service_pool.cc:168] Inserting onto call queue
0129 02:59:40.243187 (+ 17136us) service_pool.cc:225] Handling call
0129 02:59:40.243558 (+ 371us) raft_consensus.cc:1471] Updating replica for 1 ops
0129 02:59:40.358129 (+114571us) raft_consensus.cc:1534] Early marking committed up to index 0
0129 02:59:40.365122 (+ 6993us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0129 02:59:40.564063 (+198941us) log.cc:844] Serialized 26 byte log entry
0129 02:59:42.752513 (+2188450us) raft_consensus.cc:1645] Marking committed up to 6
0129 02:59:42.819545 (+ 67032us) log.cc:844] Serialized 14 byte log entry
0129 02:59:42.852321 (+ 32776us) raft_consensus.cc:1697] Filling consensus response to leader.
0129 02:59:42.862787 (+ 10466us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0129 02:59:43.029196 (+166409us) spinlock_profiling.cc:243] Waited 2.18 ms on lock 0x7b1800387be0. stack: 00007f8fe044e373 00007f8fe044e270 00007f8fdfda0574 00007f8fe9a5c6b4 00007f8fe9a5c649 00007f8fe92d4c91 00007f8fe92e1671 00007f8fe04802d3 00007f8fe04826a1 00007f8fe0482659 00007f8fe04825e9 00007f8fe04825b1 00007f8fe04818ad 0000000000379254 0000000000379099 00007f8fe046369a
0129 02:59:43.237122 (+207926us) raft_consensus.cc:1685] finished
0129 02:59:43.237314 (+ 192us) raft_consensus.cc:1691] UpdateReplicas() finished
0129 02:59:43.239489 (+ 2175us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":1295,"prepare.queue_time_us":38756,"prepare.run_cpu_time_us":80,"prepare.run_wall_time_us":6336,"raft.queue_time_us":32000,"raft.run_cpu_time_us":359,"raft.run_wall_time_us":32105,"spinlock_wait_cycles":15671040,"thread_start_us":1989587,"threads_started":1,"wal-append.queue_time_us":2250355}
W20260129 02:59:43.814750 22435 consensus_meta.cc:287] T 18f30ebaa4174fd28c958e72d0cc7e24 P 1c50de89a5a845668696dcb28c3badd6: Time spent flushing consensus metadata: real 2.150s user 0.000s sys 0.016s
W20260129 02:59:43.844789 22589 log.cc:927] Time spent T 75be4bd6bd45474795fd472eff2953ac P 974b802141bd4f13beffef2aa03f88f9: Append to log took a long time: real 0.113s user 0.005s sys 0.000s
W20260129 02:59:43.870568 20404 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260129 02:59:43.880450 20479 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.211.1:42027, user_credentials={real_user=slave}} blocked reactor thread for 71763.1us
I20260129 02:59:43.881949 20479 leader_election.cc:304] T 7a9a69cd82724b1faed1d10aa091fc4a P 974b802141bd4f13beffef2aa03f88f9 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 974b802141bd4f13beffef2aa03f88f9; no voters: 197b880fa39d432a9b472845876bfaf0, 1c50de89a5a845668696dcb28c3badd6
I20260129 02:59:43.883976 20586 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35754 (request call id 1605) took 3989 ms. Trace:
I20260129 02:59:43.884903 20434 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a609ac02fae84725ab743f95f51d33ce" candidate_uuid: "7a9daf5d42a84615a6383cfec9a10608" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "197b880fa39d432a9b472845876bfaf0" is_pre_election: true
I20260129 02:59:43.885671 20434 raft_consensus.cc:2468] T a609ac02fae84725ab743f95f51d33ce P 197b880fa39d432a9b472845876bfaf0 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7a9daf5d42a84615a6383cfec9a10608 in term 5.
I20260129 02:59:43.886160 20434 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:48356 (request call id 271) took 1131 ms. Trace:
I20260129 02:59:43.886538 20434 rpcz_store.cc:276] 0129 02:59:42.754946 (+ 0us) service_pool.cc:168] Inserting onto call queue
0129 02:59:43.777246 (+1022300us) service_pool.cc:225] Handling call
0129 02:59:43.886103 (+108857us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260129 02:59:43.887454 20432 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00e79d8f25a34676b1d866eb2068a2b2" candidate_uuid: "974b802141bd4f13beffef2aa03f88f9" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "197b880fa39d432a9b472845876bfaf0" is_pre_election: true
I20260129 02:59:43.884483 20586 rpcz_store.cc:276] 0129 02:59:39.894955 (+ 0us) service_pool.cc:168] Inserting onto call queue
0129 02:59:40.459032 (+564077us) service_pool.cc:225] Handling call
0129 02:59:40.460646 (+ 1614us) raft_consensus.cc:1471] Updating replica for 0 ops
0129 02:59:43.716764 (+3256118us) raft_consensus.cc:1697] Filling consensus response to leader.
0129 02:59:43.883566 (+166802us) spinlock_profiling.cc:243] Waited 3.12 s on lock 0x7b540003edd4. stack: 00007f8fe044e373 00007f8fe044e270 00007f8fdfda0574 00007f8fe9a5c6b4 00007f8fe9a5c649 00007f8fe34975a1 00007f8fe352ea81 00007f8fe352d4be 00007f8fe84e1f49 00007f8fe1a6ed34 00007f8fe1a6ecc2 00007f8fe1a6ebf1 00007f8fe1a6eb6c 00007f8fe1a6de22 00007f8fe186d34c 00007f8fe186c786
0129 02:59:43.883856 (+ 290us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":7166784384}
I20260129 02:59:43.888679 20432 raft_consensus.cc:2393] T 00e79d8f25a34676b1d866eb2068a2b2 P 197b880fa39d432a9b472845876bfaf0 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 974b802141bd4f13beffef2aa03f88f9 in current term 5: Already voted for candidate 197b880fa39d432a9b472845876bfaf0 in this term.
I20260129 02:59:43.889186 20432 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55672 (request call id 1474) took 1093 ms. Trace:
I20260129 02:59:43.889309 20511 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57584 (request call id 1267) took 1615 ms. Trace:
I20260129 02:59:43.890506 20586 tablet_service.cc:2066] Received LeaderStepDown RPC: tablet_id: "a1a39dcd328e4268ae41b988c66b4a75"
dest_uuid: "1c50de89a5a845668696dcb28c3badd6"
mode: GRACEFUL
new_leader_uuid: "197b880fa39d432a9b472845876bfaf0"
from {username='slave'} at 127.0.0.1:54860
I20260129 02:59:43.890719 20441 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c934d31efd3f4e5195605ddc354d14e1" candidate_uuid: "7a9daf5d42a84615a6383cfec9a10608" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "197b880fa39d432a9b472845876bfaf0" is_pre_election: true
I20260129 02:59:43.891302 20586 raft_consensus.cc:606] T a1a39dcd328e4268ae41b988c66b4a75 P 1c50de89a5a845668696dcb28c3badd6 [term 2 LEADER]: Received request to transfer leadership to TS 197b880fa39d432a9b472845876bfaf0
I20260129 02:59:43.891665 20441 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:48356 (request call id 268) took 1631 ms. Trace:
I20260129 02:59:43.770865 22425 consensus_queue.cc:260] T 368953c9102643b2a21f99af36b51905 P 1c50de89a5a845668696dcb28c3badd6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 4.3, Last appended by leader: 3, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:59:43.861526 22160 raft_consensus.cc:2749] T c934d31efd3f4e5195605ddc354d14e1 P 974b802141bd4f13beffef2aa03f88f9 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20260129 02:59:43.865160 22197 consensus_queue.cc:1059] T 4408006eb5174de1896b037cf3f18457 P 974b802141bd4f13beffef2aa03f88f9 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260129 02:59:43.893278 22425 raft_consensus.cc:3060] T 368953c9102643b2a21f99af36b51905 P 1c50de89a5a845668696dcb28c3badd6 [term 4 FOLLOWER]: Advancing to term 5
I20260129 02:59:43.893613 20435 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55672 (request call id 1467) took 1471 ms. Trace:
I20260129 02:59:43.893726 20440 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3ccac82a01a2445fa5b0ce6c0fcf2295" candidate_uuid: "974b802141bd4f13beffef2aa03f88f9" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "197b880fa39d432a9b472845876bfaf0" is_pre_election: true
I20260129 02:59:43.894173 20435 rpcz_store.cc:276] 0129 02:59:42.497594 (+ 0us) service_pool.cc:168] Inserting onto call queue
0129 02:59:43.831576 (+1333982us) service_pool.cc:225] Handling call
0129 02:59:43.893555 (+ 61979us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260129 02:59:43.894526 20440 raft_consensus.cc:2468] T 3ccac82a01a2445fa5b0ce6c0fcf2295 P 197b880fa39d432a9b472845876bfaf0 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 974b802141bd4f13beffef2aa03f88f9 in term 2.
I20260129 02:59:43.895241 20435 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4e420f90884f4796997b5ea41e49e411" candidate_uuid: "7a9daf5d42a84615a6383cfec9a10608" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "197b880fa39d432a9b472845876bfaf0" is_pre_election: true
I20260129 02:59:43.895408 20440 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55672 (request call id 1470) took 1146 ms. Trace:
I20260129 02:59:43.793442 22493 leader_election.cc:290] T 8055232cf0004d40a85f3bf983fe00a9 P 974b802141bd4f13beffef2aa03f88f9 [CANDIDATE]: Term 3 election: Requested vote from peers 197b880fa39d432a9b472845876bfaf0 (127.18.211.1:42027), 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3:37735)
W20260129 02:59:43.868414 21102 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.211.2:34517, user_credentials={real_user=slave}} blocked reactor thread for 70673.6us
I20260129 02:59:43.852679 20439 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55672 (request call id 1472) took 1069 ms. Trace:
I20260129 02:59:43.917626 20439 rpcz_store.cc:276] 0129 02:59:42.783493 (+ 0us) service_pool.cc:168] Inserting onto call queue
0129 02:59:43.816411 (+1032918us) service_pool.cc:225] Handling call
0129 02:59:43.852588 (+ 36177us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260129 02:59:43.868218 20513 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57584 (request call id 1256) took 2052 ms. Trace:
I20260129 02:59:43.804030 20480 leader_election.cc:304] T 17d6431cc77a491e8464d68db4081a51 P 974b802141bd4f13beffef2aa03f88f9 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1c50de89a5a845668696dcb28c3badd6, 974b802141bd4f13beffef2aa03f88f9; no voters:
I20260129 02:59:43.343953 20436 rpcz_store.cc:276] 0129 02:59:38.129765 (+ 0us) service_pool.cc:168] Inserting onto call queue
0129 02:59:38.410066 (+280301us) service_pool.cc:225] Handling call
0129 02:59:38.454579 (+ 44513us) raft_consensus.cc:1471] Updating replica for 0 ops
0129 02:59:38.637099 (+182520us) raft_consensus.cc:1534] Early marking committed up to index 6
0129 02:59:38.698958 (+ 61859us) log.cc:844] Serialized 14 byte log entry
0129 02:59:40.722586 (+2023628us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0129 02:59:40.753136 (+ 30550us) raft_consensus.cc:1645] Marking committed up to 6
0129 02:59:40.768911 (+ 15775us) raft_consensus.cc:1697] Filling consensus response to leader.
0129 02:59:43.260086 (+2491175us) spinlock_profiling.cc:243] Waited 90.9 ms on lock 0x7b54002670d4. stack: 00007f8fe044e373 00007f8fe044e270 00007f8fdfda0574 00007f8fe9a5c6b4 00007f8fe9a5c649 00007f8fe34975a1 00007f8fe352ea81 00007f8fe352d4be 00007f8fe84e1f49 00007f8fe1a6ed34 00007f8fe1a6ecc2 00007f8fe1a6ebf1 00007f8fe1a6eb6c 00007f8fe1a6de22 00007f8fe186d34c 00007f8fe186c786
0129 02:59:43.260190 (+ 104us) raft_consensus.cc:1691] UpdateReplicas() finished
0129 02:59:43.260479 (+ 289us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":55740,"prepare.queue_time_us":56011,"prepare.run_cpu_time_us":90,"prepare.run_wall_time_us":11294,"raft.queue_time_us":74535,"raft.run_cpu_time_us":308,"raft.run_wall_time_us":2728,"spinlock_wait_cycles":209156864,"thread_start_us":1965736,"threads_started":2,"wal-append.queue_time_us":275142,"wal-append.run_cpu_time_us":1210,"wal-append.run_wall_time_us":1019955}
I20260129 02:59:43.892096 20441 rpcz_store.cc:276] 0129 02:59:42.260555 (+ 0us) service_pool.cc:168] Inserting onto call queue
0129 02:59:43.890369 (+1629814us) service_pool.cc:225] Handling call
0129 02:59:43.891583 (+ 1214us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260129 02:59:43.919449 20589 tablet_service.cc:2066] Received LeaderStepDown RPC: tablet_id: "a644c02b16af47679824ccd060ac10fd"
dest_uuid: "1c50de89a5a845668696dcb28c3badd6"
mode: GRACEFUL
new_leader_uuid: "974b802141bd4f13beffef2aa03f88f9"
from {username='slave'} at 127.0.0.1:54860
I20260129 02:59:43.865298 20438 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "78aa39096868415d9afc2ffcb248e198" candidate_uuid: "7a9daf5d42a84615a6383cfec9a10608" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "197b880fa39d432a9b472845876bfaf0"
I20260129 02:59:43.896864 22231 leader_election.cc:290] T 96609ac68bdf4993a12e1c23355ca013 P 197b880fa39d432a9b472845876bfaf0 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 974b802141bd4f13beffef2aa03f88f9 (127.18.211.2:34517), 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3:37735), 7a9daf5d42a84615a6383cfec9a10608 (127.18.211.4:40439)
I20260129 02:59:43.897379 22435 raft_consensus.cc:515] T 18f30ebaa4174fd28c958e72d0cc7e24 P 1c50de89a5a845668696dcb28c3badd6 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } }
I20260129 02:59:43.898334 20433 raft_consensus.cc:2468] T 17d6431cc77a491e8464d68db4081a51 P 197b880fa39d432a9b472845876bfaf0 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 974b802141bd4f13beffef2aa03f88f9 in term 2.
I20260129 02:59:43.811311 22400 raft_consensus.cc:515] T e4d18c35b0a7493e85d0a4b92956a48f P 974b802141bd4f13beffef2aa03f88f9 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
W20260129 02:59:43.898864 20405 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.211.3:37735, user_credentials={real_user=slave}} blocked reactor thread for 73990.1us
I20260129 02:59:43.899499 22237 raft_consensus.cc:2804] T a5a05fbcbbcf4916bbc01fd1a5b5e90d P 197b880fa39d432a9b472845876bfaf0 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260129 02:59:43.868278 22586 raft_consensus.cc:3060] T 368953c9102643b2a21f99af36b51905 P 974b802141bd4f13beffef2aa03f88f9 [term 4 FOLLOWER]: Advancing to term 5
I20260129 02:59:43.893304 20584 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:48526 (request call id 200) took 1091 ms. Trace:
I20260129 02:59:43.863291 22320 raft_consensus.cc:515] T 17d6431cc77a491e8464d68db4081a51 P 1c50de89a5a845668696dcb28c3badd6 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:59:43.889814 20511 rpcz_store.cc:276] 0129 02:59:42.273716 (+ 0us) service_pool.cc:168] Inserting onto call queue
0129 02:59:43.788123 (+1514407us) service_pool.cc:225] Handling call
0129 02:59:43.821133 (+ 33010us) raft_consensus.cc:1471] Updating replica for 0 ops
0129 02:59:43.888398 (+ 67265us) raft_consensus.cc:1534] Early marking committed up to index 0
0129 02:59:43.888437 (+ 39us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0129 02:59:43.888479 (+ 42us) raft_consensus.cc:1645] Marking committed up to 2
0129 02:59:43.888519 (+ 40us) raft_consensus.cc:1697] Filling consensus response to leader.
0129 02:59:43.889051 (+ 532us) spinlock_profiling.cc:243] Waited 67 ms on lock 0x7b5400317254. stack: 00007f8fe044e373 00007f8fe044e270 00007f8fdfda0574 00007f8fe9a5c6b4 00007f8fe9a5c649 00007f8fe34975a1 00007f8fe352ea81 00007f8fe352d4be 00007f8fe84e1f49 00007f8fe1a6ed34 00007f8fe1a6ecc2 00007f8fe1a6ebf1 00007f8fe1a6eb6c 00007f8fe1a6de22 00007f8fe186d34c 00007f8fe186c786
0129 02:59:43.889103 (+ 52us) raft_consensus.cc:1691] UpdateReplicas() finished
0129 02:59:43.889234 (+ 131us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":154042624}
I20260129 02:59:43.904811 20516 raft_consensus.cc:1275] T 926a250fa5834e88b96ccecbefbf5086 P 974b802141bd4f13beffef2aa03f88f9 [term 5 FOLLOWER]: Refusing update from remote peer 197b880fa39d432a9b472845876bfaf0: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 5 index: 4. (index mismatch)
I20260129 02:59:43.904706 20435 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2ac659d732314e1288849f08fe5e074e" candidate_uuid: "7a9daf5d42a84615a6383cfec9a10608" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "197b880fa39d432a9b472845876bfaf0" is_pre_election: true
I20260129 02:59:43.889578 20432 rpcz_store.cc:276] 0129 02:59:42.795366 (+ 0us) service_pool.cc:168] Inserting onto call queue
0129 02:59:43.887062 (+1091696us) service_pool.cc:225] Handling call
0129 02:59:43.889092 (+ 2030us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260129 02:59:43.777261 22553 consensus_queue.cc:260] T cedce331e1584ac8b636c9b8a72b2ba0 P 1c50de89a5a845668696dcb28c3badd6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:59:43.915549 20440 rpcz_store.cc:276] 0129 02:59:42.748924 (+ 0us) service_pool.cc:168] Inserting onto call queue
0129 02:59:43.871422 (+1122498us) service_pool.cc:225] Handling call
0129 02:59:43.895202 (+ 23780us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260129 02:59:43.887555 20434 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2e58e9a5a8aa4cecafb464addb4f35ed" candidate_uuid: "1c50de89a5a845668696dcb28c3badd6" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "197b880fa39d432a9b472845876bfaf0" is_pre_election: true
I20260129 02:59:43.892138 22184 leader_election.cc:290] T 7d9494f5584d4d5db8b9adf1c8fc888b P 974b802141bd4f13beffef2aa03f88f9 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 197b880fa39d432a9b472845876bfaf0 (127.18.211.1:42027), 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3:37735)
I20260129 02:59:43.933198 20435 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:48356 (request call id 269) took 1477 ms. Trace:
I20260129 02:59:43.920107 20589 raft_consensus.cc:606] T a644c02b16af47679824ccd060ac10fd P 1c50de89a5a845668696dcb28c3badd6 [term 3 LEADER]: Received request to transfer leadership to TS 974b802141bd4f13beffef2aa03f88f9
I20260129 02:59:43.943331 20435 rpcz_store.cc:276] 0129 02:59:42.455741 (+ 0us) service_pool.cc:168] Inserting onto call queue
0129 02:59:43.904356 (+1448615us) service_pool.cc:225] Handling call
0129 02:59:43.933137 (+ 28781us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260129 02:59:43.926513 20433 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55672 (request call id 1469) took 1199 ms. Trace:
W20260129 02:59:43.931406 20407 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.211.2:34517, user_credentials={real_user=slave}} blocked reactor thread for 192949us
I20260129 02:59:43.944258 20433 rpcz_store.cc:276] 0129 02:59:42.724180 (+ 0us) service_pool.cc:168] Inserting onto call queue
0129 02:59:43.719067 (+994887us) service_pool.cc:225] Handling call
0129 02:59:43.834047 (+114980us) spinlock_profiling.cc:243] Waited 1.17 ms on lock 0x7b5800000f98. stack: 00007f8fe044e373 00007f8fe044e270 00007f8fdfda0574 00007f8fe9a5c6b4 00007f8fe9a5c649 00007f8fe02c08b1 00007f8fe0442db1 00007f8fe044312e 000000000035a3c9 00007f8fe846d560 00007f8fe8535dd4 00007f8fe8535e41 00007f8fe84e3bd0 00007f8fe84e34b4 00007f8fe1a73834 00007f8fe1a737c2
0129 02:59:43.923050 (+ 89003us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2687744}
I20260129 02:59:43.945067 20514 raft_consensus.cc:2393] T e4d18c35b0a7493e85d0a4b92956a48f P 974b802141bd4f13beffef2aa03f88f9 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 197b880fa39d432a9b472845876bfaf0 in current term 4: Already voted for candidate 974b802141bd4f13beffef2aa03f88f9 in this term.
I20260129 02:59:43.930802 20479 leader_election.cc:304] T 913fc7d751ed4b7388d889aac30a5f9b P 974b802141bd4f13beffef2aa03f88f9 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 197b880fa39d432a9b472845876bfaf0, 974b802141bd4f13beffef2aa03f88f9; no voters: 1c50de89a5a845668696dcb28c3badd6
I20260129 02:59:43.929315 20553 leader_election.cc:304] T b189f08e16a74a11980cc55d99359a0a P 1c50de89a5a845668696dcb28c3badd6 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 197b880fa39d432a9b472845876bfaf0, 1c50de89a5a845668696dcb28c3badd6; no voters:
I20260129 02:59:43.935356 20434 raft_consensus.cc:2368] T 2e58e9a5a8aa4cecafb464addb4f35ed P 197b880fa39d432a9b472845876bfaf0 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1c50de89a5a845668696dcb28c3badd6 for earlier term 3. Current term is 4.
I20260129 02:59:43.935488 20439 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fd8311b779af4e1fa1402b2a02a602f2" candidate_uuid: "7a9daf5d42a84615a6383cfec9a10608" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "197b880fa39d432a9b472845876bfaf0" is_pre_election: true
I20260129 02:59:43.946238 20439 raft_consensus.cc:2468] T fd8311b779af4e1fa1402b2a02a602f2 P 197b880fa39d432a9b472845876bfaf0 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7a9daf5d42a84615a6383cfec9a10608 in term 3.
I20260129 02:59:43.946594 20434 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fcddab6bca684adf8ea032289a00f90e" candidate_uuid: "1c50de89a5a845668696dcb28c3badd6" candidate_term: 8 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "197b880fa39d432a9b472845876bfaf0" is_pre_election: true
I20260129 02:59:43.947443 22435 leader_election.cc:290] T 18f30ebaa4174fd28c958e72d0cc7e24 P 1c50de89a5a845668696dcb28c3badd6 [CANDIDATE]: Term 4 election: Requested vote from peers 197b880fa39d432a9b472845876bfaf0 (127.18.211.1:42027), 7a9daf5d42a84615a6383cfec9a10608 (127.18.211.4:40439)
I20260129 02:59:43.948165 20434 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "dc2f4cb752cf4d59b5f6bc689f5d8104" candidate_uuid: "1c50de89a5a845668696dcb28c3badd6" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "197b880fa39d432a9b472845876bfaf0"
I20260129 02:59:43.948861 20434 raft_consensus.cc:3060] T dc2f4cb752cf4d59b5f6bc689f5d8104 P 197b880fa39d432a9b472845876bfaf0 [term 3 FOLLOWER]: Advancing to term 4
I20260129 02:59:43.941268 22137 consensus_queue.cc:1059] T 4408006eb5174de1896b037cf3f18457 P 974b802141bd4f13beffef2aa03f88f9 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.008s
I20260129 02:59:43.951028 20432 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1299723a4476420ab02fd8bfd7aa0645" candidate_uuid: "1c50de89a5a845668696dcb28c3badd6" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "197b880fa39d432a9b472845876bfaf0" is_pre_election: true
I20260129 02:59:43.935920 20440 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e33a417cf9c54cf5af73a4ee3aac4d3a" candidate_uuid: "1c50de89a5a845668696dcb28c3badd6" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "197b880fa39d432a9b472845876bfaf0" is_pre_election: true
I20260129 02:59:43.952245 20435 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0678b8b468bd43cdbc3e485900844821" candidate_uuid: "7a9daf5d42a84615a6383cfec9a10608" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "197b880fa39d432a9b472845876bfaf0" is_pre_election: true
I20260129 02:59:43.952382 20440 raft_consensus.cc:2468] T e33a417cf9c54cf5af73a4ee3aac4d3a P 197b880fa39d432a9b472845876bfaf0 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1c50de89a5a845668696dcb28c3badd6 in term 4.
I20260129 02:59:43.953401 20440 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a609ac02fae84725ab743f95f51d33ce" candidate_uuid: "974b802141bd4f13beffef2aa03f88f9" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "197b880fa39d432a9b472845876bfaf0" is_pre_election: true
I20260129 02:59:43.953469 20435 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e7d83d4ccf684e76acb29a0dc844e1b8" candidate_uuid: "1c50de89a5a845668696dcb28c3badd6" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "197b880fa39d432a9b472845876bfaf0" is_pre_election: true
I20260129 02:59:43.954113 20440 raft_consensus.cc:2468] T a609ac02fae84725ab743f95f51d33ce P 197b880fa39d432a9b472845876bfaf0 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 974b802141bd4f13beffef2aa03f88f9 in term 5.
I20260129 02:59:43.954267 20435 raft_consensus.cc:2468] T e7d83d4ccf684e76acb29a0dc844e1b8 P 197b880fa39d432a9b472845876bfaf0 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1c50de89a5a845668696dcb28c3badd6 in term 3.
I20260129 02:59:43.954679 20440 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55672 (request call id 1473) took 1159 ms. Trace:
I20260129 02:59:43.955317 20435 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0340229d521b432580ae926f6062e3f7" candidate_uuid: "1c50de89a5a845668696dcb28c3badd6" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "197b880fa39d432a9b472845876bfaf0" is_pre_election: true
I20260129 02:59:43.940186 22553 raft_consensus.cc:3060] T cedce331e1584ac8b636c9b8a72b2ba0 P 1c50de89a5a845668696dcb28c3badd6 [term 4 FOLLOWER]: Advancing to term 5
I20260129 02:59:43.942293 22364 leader_election.cc:290] T 2e58e9a5a8aa4cecafb464addb4f35ed P 7a9daf5d42a84615a6383cfec9a10608 [CANDIDATE]: Term 4 election: Requested vote from peers 197b880fa39d432a9b472845876bfaf0 (127.18.211.1:42027), 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3:37735)
I20260129 02:59:43.918455 20513 rpcz_store.cc:276] 0129 02:59:41.815743 (+ 0us) service_pool.cc:168] Inserting onto call queue
0129 02:59:42.453453 (+637710us) service_pool.cc:225] Handling call
0129 02:59:42.476675 (+ 23222us) raft_consensus.cc:1471] Updating replica for 1 ops
0129 02:59:43.725874 (+1249199us) raft_consensus.cc:1697] Filling consensus response to leader.
0129 02:59:43.867918 (+142044us) spinlock_profiling.cc:243] Waited 1.22 s on lock 0x7b54000fbc54. stack: 00007f8fe044e373 00007f8fe044e270 00007f8fdfda0574 00007f8fe9a5c6b4 00007f8fe9a5c649 00007f8fe34975a1 00007f8fe352ea81 00007f8fe352d4be 00007f8fe84e1f49 00007f8fe1a6ed34 00007f8fe1a6ecc2 00007f8fe1a6ebf1 00007f8fe1a6eb6c 00007f8fe1a6de22 00007f8fe186d34c 00007f8fe186c786
0129 02:59:43.868152 (+ 234us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2801495296}
I20260129 02:59:43.926199 20584 rpcz_store.cc:276] 0129 02:59:42.802300 (+ 0us) service_pool.cc:168] Inserting onto call queue
0129 02:59:42.802899 (+ 599us) service_pool.cc:225] Handling call
0129 02:59:42.827107 (+ 24208us) raft_consensus.cc:1471] Updating replica for 0 ops
0129 02:59:43.609922 (+782815us) raft_consensus.cc:1697] Filling consensus response to leader.
0129 02:59:43.892993 (+283071us) spinlock_profiling.cc:243] Waited 782 ms on lock 0x7b5400108e54. stack: 00007f8fe044e373 00007f8fe044e270 00007f8fdfda0574 00007f8fe9a5c6b4 00007f8fe9a5c649 00007f8fe34975a1 00007f8fe352ea81 00007f8fe352d4be 00007f8fe84e1f49 00007f8fe1a6ed34 00007f8fe1a6ecc2 00007f8fe1a6ebf1 00007f8fe1a6eb6c 00007f8fe1a6de22 00007f8fe186d34c 00007f8fe186c786
0129 02:59:43.893237 (+ 244us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1797831936}
I20260129 02:59:43.946760 20439 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:48356 (request call id 266) took 1726 ms. Trace:
I20260129 02:59:43.965271 20439 rpcz_store.cc:276] 0129 02:59:42.220917 (+ 0us) service_pool.cc:168] Inserting onto call queue
0129 02:59:43.918310 (+1697393us) service_pool.cc:225] Handling call
0129 02:59:43.946707 (+ 28397us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260129 02:59:43.953640 20441 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "18f30ebaa4174fd28c958e72d0cc7e24" candidate_uuid: "7a9daf5d42a84615a6383cfec9a10608" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "197b880fa39d432a9b472845876bfaf0" is_pre_election: true
I20260129 02:59:43.937256 20436 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a8479d1fe0bd41e5946186e3a6ca8c7a" candidate_uuid: "1c50de89a5a845668696dcb28c3badd6" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "197b880fa39d432a9b472845876bfaf0" is_pre_election: true
I20260129 02:59:43.954304 20438 raft_consensus.cc:2393] T 78aa39096868415d9afc2ffcb248e198 P 197b880fa39d432a9b472845876bfaf0 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 7a9daf5d42a84615a6383cfec9a10608 in current term 4: Already voted for candidate 197b880fa39d432a9b472845876bfaf0 in this term.
I20260129 02:59:43.951730 20432 raft_consensus.cc:2468] T 1299723a4476420ab02fd8bfd7aa0645 P 197b880fa39d432a9b472845876bfaf0 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1c50de89a5a845668696dcb28c3badd6 in term 6.
I20260129 02:59:43.939330 22320 leader_election.cc:290] T 17d6431cc77a491e8464d68db4081a51 P 1c50de89a5a845668696dcb28c3badd6 [CANDIDATE]: Term 3 election: Requested vote from peers 197b880fa39d432a9b472845876bfaf0 (127.18.211.1:42027), 974b802141bd4f13beffef2aa03f88f9 (127.18.211.2:34517)
I20260129 02:59:43.924540 22566 raft_consensus.cc:2749] T 7a9a69cd82724b1faed1d10aa091fc4a P 974b802141bd4f13beffef2aa03f88f9 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260129 02:59:43.956225 20581 raft_consensus.cc:2393] T 18f30ebaa4174fd28c958e72d0cc7e24 P 1c50de89a5a845668696dcb28c3badd6 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 7a9daf5d42a84615a6383cfec9a10608 in current term 4: Already voted for candidate 1c50de89a5a845668696dcb28c3badd6 in this term.
W20260129 02:59:43.953405 20480 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.211.3:37735, user_credentials={real_user=slave}} blocked reactor thread for 149541us
I20260129 02:59:43.936199 22552 consensus_queue.cc:1048] T 4f5a77e94fef4b16a2ba7b067c8b144a P 1c50de89a5a845668696dcb28c3badd6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20260129 02:59:43.955087 20440 rpcz_store.cc:276] 0129 02:59:42.794986 (+ 0us) service_pool.cc:168] Inserting onto call queue
0129 02:59:43.953090 (+1158104us) service_pool.cc:225] Handling call
0129 02:59:43.954611 (+ 1521us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260129 02:59:43.977198 20510 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57584 (request call id 1269) took 1157 ms. Trace:
I20260129 02:59:43.975697 20434 raft_consensus.cc:2468] T dc2f4cb752cf4d59b5f6bc689f5d8104 P 197b880fa39d432a9b472845876bfaf0 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1c50de89a5a845668696dcb28c3badd6 in term 4.
I20260129 02:59:43.957247 20511 raft_consensus.cc:1240] T 2ac659d732314e1288849f08fe5e074e P 974b802141bd4f13beffef2aa03f88f9 [term 3 FOLLOWER]: Rejecting Update request from peer 197b880fa39d432a9b472845876bfaf0 for earlier term 2. Current term is 3. Ops: [2.5-2.5]
I20260129 02:59:43.977702 20510 rpcz_store.cc:276] 0129 02:59:42.807578 (+ 0us) service_pool.cc:168] Inserting onto call queue
0129 02:59:43.783736 (+976158us) service_pool.cc:225] Handling call
0129 02:59:43.870466 (+ 86730us) raft_consensus.cc:1471] Updating replica for 0 ops
0129 02:59:43.932132 (+ 61666us) raft_consensus.cc:1534] Early marking committed up to index 0
0129 02:59:43.932157 (+ 25us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0129 02:59:43.944749 (+ 12592us) raft_consensus.cc:1645] Marking committed up to 4
0129 02:59:43.964322 (+ 19573us) raft_consensus.cc:1697] Filling consensus response to leader.
0129 02:59:43.964404 (+ 82us) raft_consensus.cc:1691] UpdateReplicas() finished
0129 02:59:43.964526 (+ 122us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260129 02:59:43.961828 20582 raft_consensus.cc:2468] T fd8311b779af4e1fa1402b2a02a602f2 P 1c50de89a5a845668696dcb28c3badd6 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7a9daf5d42a84615a6383cfec9a10608 in term 4.
I20260129 02:59:43.962564 20435 raft_consensus.cc:2468] T 0340229d521b432580ae926f6062e3f7 P 197b880fa39d432a9b472845876bfaf0 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1c50de89a5a845668696dcb28c3badd6 in term 4.
I20260129 02:59:43.968788 22184 raft_consensus.cc:2804] T 17d6431cc77a491e8464d68db4081a51 P 974b802141bd4f13beffef2aa03f88f9 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260129 02:59:43.970198 20513 raft_consensus.cc:1275] T 0678b8b468bd43cdbc3e485900844821 P 974b802141bd4f13beffef2aa03f88f9 [term 5 FOLLOWER]: Refusing update from remote peer 197b880fa39d432a9b472845876bfaf0: Log matching property violated. Preceding OpId in replica: term: 4 index: 6. Preceding OpId from leader: term: 5 index: 7. (index mismatch)
I20260129 02:59:43.926709 22237 raft_consensus.cc:493] T a5a05fbcbbcf4916bbc01fd1a5b5e90d P 197b880fa39d432a9b472845876bfaf0 [term 5 FOLLOWER]: Starting leader election (detected failure of leader 974b802141bd4f13beffef2aa03f88f9)
I20260129 02:59:43.958586 22400 leader_election.cc:290] T e4d18c35b0a7493e85d0a4b92956a48f P 974b802141bd4f13beffef2aa03f88f9 [CANDIDATE]: Term 4 election: Requested vote from peers 197b880fa39d432a9b472845876bfaf0 (127.18.211.1:42027), 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3:37735)
I20260129 02:59:43.982080 20512 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57584 (request call id 1264) took 1752 ms. Trace:
I20260129 02:59:43.954975 20556 leader_election.cc:304] T 76f713c3331540d2a5b76164103e9eb1 P 1c50de89a5a845668696dcb28c3badd6 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1c50de89a5a845668696dcb28c3badd6; no voters: 197b880fa39d432a9b472845876bfaf0, 974b802141bd4f13beffef2aa03f88f9
I20260129 02:59:43.972334 20436 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55676 (request call id 1200) took 1806 ms. Trace:
I20260129 02:59:43.987272 20512 rpcz_store.cc:276] 0129 02:59:42.230039 (+ 0us) service_pool.cc:168] Inserting onto call queue
0129 02:59:43.775546 (+1545507us) service_pool.cc:225] Handling call
0129 02:59:43.818436 (+ 42890us) raft_consensus.cc:1471] Updating replica for 0 ops
0129 02:59:43.818772 (+ 336us) raft_consensus.cc:1534] Early marking committed up to index 3
0129 02:59:43.862528 (+ 43756us) log.cc:844] Serialized 14 byte log entry
0129 02:59:43.981716 (+119188us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0129 02:59:43.981774 (+ 58us) raft_consensus.cc:1645] Marking committed up to 3
0129 02:59:43.981819 (+ 45us) raft_consensus.cc:1697] Filling consensus response to leader.
0129 02:59:43.981885 (+ 66us) raft_consensus.cc:1691] UpdateReplicas() finished
0129 02:59:43.982011 (+ 126us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":44433,"prepare.run_cpu_time_us":50,"prepare.run_wall_time_us":52,"raft.queue_time_us":115,"raft.run_cpu_time_us":110,"raft.run_wall_time_us":110,"thread_start_us":92231,"threads_started":1}
I20260129 02:59:43.986977 20433 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fd8311b779af4e1fa1402b2a02a602f2" candidate_uuid: "1c50de89a5a845668696dcb28c3badd6" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "197b880fa39d432a9b472845876bfaf0" is_pre_election: true
I20260129 02:59:43.985436 22237 raft_consensus.cc:3060] T a5a05fbcbbcf4916bbc01fd1a5b5e90d P 197b880fa39d432a9b472845876bfaf0 [term 5 FOLLOWER]: Advancing to term 6
I20260129 02:59:43.999027 21100 leader_election.cc:304] T fd8311b779af4e1fa1402b2a02a602f2 P 7a9daf5d42a84615a6383cfec9a10608 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1c50de89a5a845668696dcb28c3badd6, 7a9daf5d42a84615a6383cfec9a10608; no voters:
I20260129 02:59:43.995884 20439 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "8055232cf0004d40a85f3bf983fe00a9" candidate_uuid: "1c50de89a5a845668696dcb28c3badd6" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "197b880fa39d432a9b472845876bfaf0" is_pre_election: true
I20260129 02:59:43.997144 20438 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:48356 (request call id 253) took 2165 ms. Trace:
I20260129 02:59:44.002624 20438 rpcz_store.cc:276] 0129 02:59:41.834282 (+ 0us) service_pool.cc:168] Inserting onto call queue
0129 02:59:43.985182 (+2150900us) service_pool.cc:225] Handling call
0129 02:59:43.996378 (+ 11196us) raft_consensus.cc:1471] Updating replica for 0 ops
0129 02:59:43.996752 (+ 374us) raft_consensus.cc:1534] Early marking committed up to index 0
0129 02:59:43.996784 (+ 32us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0129 02:59:43.996847 (+ 63us) raft_consensus.cc:1645] Marking committed up to 4
0129 02:59:43.996885 (+ 38us) raft_consensus.cc:1697] Filling consensus response to leader.
0129 02:59:43.996947 (+ 62us) raft_consensus.cc:1691] UpdateReplicas() finished
0129 02:59:43.997066 (+ 119us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260129 02:59:43.995400 20433 raft_consensus.cc:2468] T fd8311b779af4e1fa1402b2a02a602f2 P 197b880fa39d432a9b472845876bfaf0 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1c50de89a5a845668696dcb28c3badd6 in term 3.
I20260129 02:59:44.009647 20439 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55676 (request call id 1196) took 1844 ms. Trace:
I20260129 02:59:44.010210 20439 rpcz_store.cc:276] 0129 02:59:42.164834 (+ 0us) service_pool.cc:168] Inserting onto call queue
0129 02:59:43.965966 (+1801132us) service_pool.cc:225] Handling call
0129 02:59:44.009567 (+ 43601us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260129 02:59:44.012544 22255 raft_consensus.cc:2804] T fd8311b779af4e1fa1402b2a02a602f2 P 7a9daf5d42a84615a6383cfec9a10608 [term 4 FOLLOWER]: Leader election won for term 4
I20260129 02:59:44.013161 22255 raft_consensus.cc:697] T fd8311b779af4e1fa1402b2a02a602f2 P 7a9daf5d42a84615a6383cfec9a10608 [term 4 LEADER]: Becoming Leader. State: Replica: 7a9daf5d42a84615a6383cfec9a10608, State: Running, Role: LEADER
I20260129 02:59:44.014279 22255 consensus_queue.cc:237] T fd8311b779af4e1fa1402b2a02a602f2 P 7a9daf5d42a84615a6383cfec9a10608 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } }
I20260129 02:59:44.007848 20586 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2e58e9a5a8aa4cecafb464addb4f35ed" candidate_uuid: "7a9daf5d42a84615a6383cfec9a10608" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "1c50de89a5a845668696dcb28c3badd6"
I20260129 02:59:44.016297 20586 raft_consensus.cc:3060] T 2e58e9a5a8aa4cecafb464addb4f35ed P 1c50de89a5a845668696dcb28c3badd6 [term 2 FOLLOWER]: Advancing to term 4
W20260129 02:59:44.016929 20479 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.211.1:42027, user_credentials={real_user=slave}} blocked reactor thread for 92556.5us
I20260129 02:59:44.017719 20435 raft_consensus.cc:1240] T dc2f4cb752cf4d59b5f6bc689f5d8104 P 197b880fa39d432a9b472845876bfaf0 [term 4 FOLLOWER]: Rejecting Update request from peer 7a9daf5d42a84615a6383cfec9a10608 for earlier term 3. Current term is 4. Ops: []
I20260129 02:59:44.017840 20479 leader_election.cc:304] T 3906b080a5b34a74b8b64128635e8997 P 974b802141bd4f13beffef2aa03f88f9 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 197b880fa39d432a9b472845876bfaf0, 974b802141bd4f13beffef2aa03f88f9; no voters: 1c50de89a5a845668696dcb28c3badd6
I20260129 02:59:44.018473 20435 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:48356 (request call id 258) took 1941 ms. Trace:
I20260129 02:59:44.018829 20435 rpcz_store.cc:276] 0129 02:59:42.077080 (+ 0us) service_pool.cc:168] Inserting onto call queue
0129 02:59:43.986665 (+1909585us) service_pool.cc:225] Handling call
0129 02:59:44.017523 (+ 30858us) raft_consensus.cc:1471] Updating replica for 0 ops
0129 02:59:44.018212 (+ 689us) raft_consensus.cc:1697] Filling consensus response to leader.
0129 02:59:44.018417 (+ 205us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260129 02:59:44.016494 20510 raft_consensus.cc:1240] T 8055232cf0004d40a85f3bf983fe00a9 P 974b802141bd4f13beffef2aa03f88f9 [term 3 FOLLOWER]: Rejecting Update request from peer 197b880fa39d432a9b472845876bfaf0 for earlier term 2. Current term is 3. Ops: [2.2-2.2]
I20260129 02:59:44.016444 20432 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "96609ac68bdf4993a12e1c23355ca013" candidate_uuid: "1c50de89a5a845668696dcb28c3badd6" candidate_term: 5 candidate_status { last_received { term: 2 index: 3 } } ignore_live_leader: false dest_uuid: "197b880fa39d432a9b472845876bfaf0" is_pre_election: true
W20260129 02:59:44.026693 20406 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260129 02:59:43.980449 22184 raft_consensus.cc:493] T 17d6431cc77a491e8464d68db4081a51 P 974b802141bd4f13beffef2aa03f88f9 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260129 02:59:43.994386 22437 raft_consensus.cc:2804] T 913fc7d751ed4b7388d889aac30a5f9b P 974b802141bd4f13beffef2aa03f88f9 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260129 02:59:43.992769 20436 rpcz_store.cc:276] 0129 02:59:42.166088 (+ 0us) service_pool.cc:168] Inserting onto call queue
0129 02:59:43.932252 (+1766164us) service_pool.cc:225] Handling call
0129 02:59:43.972247 (+ 39995us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260129 02:59:44.036785 20440 raft_consensus.cc:1240] T 00e79d8f25a34676b1d866eb2068a2b2 P 197b880fa39d432a9b472845876bfaf0 [term 5 FOLLOWER]: Rejecting Update request from peer 7a9daf5d42a84615a6383cfec9a10608 for earlier term 4. Current term is 5. Ops: []
I20260129 02:59:44.040735 20434 raft_consensus.cc:1240] T 87d5fe48a44a408aaf03e894a67ecb2f P 197b880fa39d432a9b472845876bfaf0 [term 5 FOLLOWER]: Rejecting Update request from peer 7a9daf5d42a84615a6383cfec9a10608 for earlier term 4. Current term is 5. Ops: []
I20260129 02:59:44.041213 20432 raft_consensus.cc:2410] T 96609ac68bdf4993a12e1c23355ca013 P 197b880fa39d432a9b472845876bfaf0 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1c50de89a5a845668696dcb28c3badd6 for term 5 because replica has last-logged OpId of term: 2 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 3.
W20260129 02:59:44.043430 21099 leader_election.cc:336] T c934d31efd3f4e5195605ddc354d14e1 P 7a9daf5d42a84615a6383cfec9a10608 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 197b880fa39d432a9b472845876bfaf0 (127.18.211.1:42027): Timed out: RequestConsensusVote RPC to 127.18.211.1:42027 timed out after 2.185s (SENT)
W20260129 02:59:44.024080 20553 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.211.1:42027, user_credentials={real_user=slave}} blocked reactor thread for 95114.6us
I20260129 02:59:44.044483 20479 leader_election.cc:304] T 0340229d521b432580ae926f6062e3f7 P 974b802141bd4f13beffef2aa03f88f9 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 197b880fa39d432a9b472845876bfaf0, 974b802141bd4f13beffef2aa03f88f9; no voters: 1c50de89a5a845668696dcb28c3badd6
I20260129 02:59:44.046415 20434 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:48356 (request call id 257) took 2194 ms. Trace:
W20260129 02:59:44.046723 20481 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260129 02:59:44.046993 20434 rpcz_store.cc:276] 0129 02:59:41.847009 (+ 0us) service_pool.cc:168] Inserting onto call queue
0129 02:59:44.001500 (+2154491us) service_pool.cc:225] Handling call
0129 02:59:44.031843 (+ 30343us) raft_consensus.cc:1471] Updating replica for 0 ops
0129 02:59:44.041430 (+ 9587us) raft_consensus.cc:1697] Filling consensus response to leader.
0129 02:59:44.041718 (+ 288us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260129 02:59:44.048862 20440 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:48356 (request call id 260) took 1940 ms. Trace:
I20260129 02:59:44.049297 20440 rpcz_store.cc:276] 0129 02:59:42.102013 (+ 0us) service_pool.cc:168] Inserting onto call queue
0129 02:59:43.981134 (+1879121us) service_pool.cc:225] Handling call
0129 02:59:44.036521 (+ 55387us) raft_consensus.cc:1471] Updating replica for 0 ops
0129 02:59:44.042221 (+ 5700us) raft_consensus.cc:1697] Filling consensus response to leader.
0129 02:59:44.042441 (+ 220us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260129 02:59:44.035187 22184 raft_consensus.cc:3060] T 17d6431cc77a491e8464d68db4081a51 P 974b802141bd4f13beffef2aa03f88f9 [term 2 FOLLOWER]: Advancing to term 3
I20260129 02:59:44.047575 20441 raft_consensus.cc:2468] T 18f30ebaa4174fd28c958e72d0cc7e24 P 197b880fa39d432a9b472845876bfaf0 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7a9daf5d42a84615a6383cfec9a10608 in term 2.
I20260129 02:59:44.039116 22437 raft_consensus.cc:493] T 913fc7d751ed4b7388d889aac30a5f9b P 974b802141bd4f13beffef2aa03f88f9 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 1c50de89a5a845668696dcb28c3badd6)
I20260129 02:59:44.030146 22577 consensus_queue.cc:1059] T dc2f4cb752cf4d59b5f6bc689f5d8104 P 7a9daf5d42a84615a6383cfec9a10608 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 }, Status: INVALID_TERM, Last received: 3.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.021s
I20260129 02:59:44.041844 22581 raft_consensus.cc:2764] T b189f08e16a74a11980cc55d99359a0a P 1c50de89a5a845668696dcb28c3badd6 [term 5 FOLLOWER]: Leader pre-election decision vote started in defunct term 4: won
I20260129 02:59:44.067049 21132 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "dc2f4cb752cf4d59b5f6bc689f5d8104" candidate_uuid: "1c50de89a5a845668696dcb28c3badd6" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "7a9daf5d42a84615a6383cfec9a10608"
W20260129 02:59:44.074723 20479 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.211.1:42027, user_credentials={real_user=slave}} blocked reactor thread for 49691.4us
W20260129 02:59:44.075809 20556 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.211.2:34517, user_credentials={real_user=slave}} blocked reactor thread for 120964us
I20260129 02:59:44.075114 20586 raft_consensus.cc:2468] T 2e58e9a5a8aa4cecafb464addb4f35ed P 1c50de89a5a845668696dcb28c3badd6 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7a9daf5d42a84615a6383cfec9a10608 in term 4.
I20260129 02:59:44.050534 22435 raft_consensus.cc:2749] T 76f713c3331540d2a5b76164103e9eb1 P 1c50de89a5a845668696dcb28c3badd6 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20260129 02:59:44.080281 20435 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55672 (request call id 1462) took 1913 ms. Trace:
I20260129 02:59:44.080821 20435 rpcz_store.cc:276] 0129 02:59:42.166719 (+ 0us) service_pool.cc:168] Inserting onto call queue
0129 02:59:44.019666 (+1852947us) service_pool.cc:225] Handling call
0129 02:59:44.054665 (+ 34999us) raft_consensus.cc:1471] Updating replica for 0 ops
0129 02:59:44.079954 (+ 25289us) raft_consensus.cc:1534] Early marking committed up to index 0
0129 02:59:44.079999 (+ 45us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0129 02:59:44.080039 (+ 40us) raft_consensus.cc:1645] Marking committed up to 5
0129 02:59:44.080071 (+ 32us) raft_consensus.cc:1697] Filling consensus response to leader.
0129 02:59:44.080132 (+ 61us) raft_consensus.cc:1691] UpdateReplicas() finished
0129 02:59:44.080231 (+ 99us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260129 02:59:44.085868 21100 leader_election.cc:304] T 2e58e9a5a8aa4cecafb464addb4f35ed P 7a9daf5d42a84615a6383cfec9a10608 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1c50de89a5a845668696dcb28c3badd6, 7a9daf5d42a84615a6383cfec9a10608; no voters:
I20260129 02:59:44.087011 22343 raft_consensus.cc:2804] T 2e58e9a5a8aa4cecafb464addb4f35ed P 7a9daf5d42a84615a6383cfec9a10608 [term 4 FOLLOWER]: Leader election won for term 4
I20260129 02:59:44.087383 20479 leader_election.cc:304] T 3ccac82a01a2445fa5b0ce6c0fcf2295 P 974b802141bd4f13beffef2aa03f88f9 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 197b880fa39d432a9b472845876bfaf0, 974b802141bd4f13beffef2aa03f88f9; no voters:
I20260129 02:59:44.087584 22343 raft_consensus.cc:697] T 2e58e9a5a8aa4cecafb464addb4f35ed P 7a9daf5d42a84615a6383cfec9a10608 [term 4 LEADER]: Becoming Leader. State: Replica: 7a9daf5d42a84615a6383cfec9a10608, State: Running, Role: LEADER
I20260129 02:59:44.088290 22566 raft_consensus.cc:2804] T 0340229d521b432580ae926f6062e3f7 P 974b802141bd4f13beffef2aa03f88f9 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260129 02:59:44.088595 20509 raft_consensus.cc:2468] T 87d5fe48a44a408aaf03e894a67ecb2f P 974b802141bd4f13beffef2aa03f88f9 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 197b880fa39d432a9b472845876bfaf0 in term 5.
I20260129 02:59:44.088783 22566 raft_consensus.cc:493] T 0340229d521b432580ae926f6062e3f7 P 974b802141bd4f13beffef2aa03f88f9 [term 3 FOLLOWER]: Starting leader election (detected failure of leader 197b880fa39d432a9b472845876bfaf0)
I20260129 02:59:44.089363 22566 raft_consensus.cc:3060] T 0340229d521b432580ae926f6062e3f7 P 974b802141bd4f13beffef2aa03f88f9 [term 3 FOLLOWER]: Advancing to term 4
I20260129 02:59:44.090931 20433 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55676 (request call id 1198) took 1925 ms. Trace:
I20260129 02:59:44.091338 20433 rpcz_store.cc:276] 0129 02:59:42.165474 (+ 0us) service_pool.cc:168] Inserting onto call queue
0129 02:59:44.006039 (+1840565us) service_pool.cc:225] Handling call
0129 02:59:44.079598 (+ 73559us) raft_consensus.cc:1471] Updating replica for 0 ops
0129 02:59:44.090452 (+ 10854us) raft_consensus.cc:1534] Early marking committed up to index 0
0129 02:59:44.090477 (+ 25us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0129 02:59:44.090582 (+ 105us) raft_consensus.cc:1645] Marking committed up to 5
0129 02:59:44.090618 (+ 36us) raft_consensus.cc:1697] Filling consensus response to leader.
0129 02:59:44.090714 (+ 96us) raft_consensus.cc:1691] UpdateReplicas() finished
0129 02:59:44.090851 (+ 137us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260129 02:59:44.088675 22343 consensus_queue.cc:237] T 2e58e9a5a8aa4cecafb464addb4f35ed P 7a9daf5d42a84615a6383cfec9a10608 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } }
I20260129 02:59:44.070134 22437 raft_consensus.cc:3060] T 913fc7d751ed4b7388d889aac30a5f9b P 974b802141bd4f13beffef2aa03f88f9 [term 2 FOLLOWER]: Advancing to term 3
I20260129 02:59:44.096200 20480 leader_election.cc:304] T a8479d1fe0bd41e5946186e3a6ca8c7a P 974b802141bd4f13beffef2aa03f88f9 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1c50de89a5a845668696dcb28c3badd6, 974b802141bd4f13beffef2aa03f88f9; no voters: 197b880fa39d432a9b472845876bfaf0
I20260129 02:59:44.098021 20434 raft_consensus.cc:1240] T cb4727386a3e48aca5d71cc6c611c197 P 197b880fa39d432a9b472845876bfaf0 [term 2 FOLLOWER]: Rejecting Update request from peer 974b802141bd4f13beffef2aa03f88f9 for earlier term 1. Current term is 2. Ops: []
I20260129 02:59:44.098733 20434 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55672 (request call id 1471) took 1326 ms. Trace:
I20260129 02:59:44.099038 20434 rpcz_store.cc:276] 0129 02:59:42.772633 (+ 0us) service_pool.cc:168] Inserting onto call queue
0129 02:59:44.062577 (+1289944us) service_pool.cc:225] Handling call
0129 02:59:44.097797 (+ 35220us) raft_consensus.cc:1471] Updating replica for 0 ops
0129 02:59:44.098458 (+ 661us) raft_consensus.cc:1697] Filling consensus response to leader.
0129 02:59:44.098668 (+ 210us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260129 02:59:44.104106 20438 raft_consensus.cc:1240] T a1a39dcd328e4268ae41b988c66b4a75 P 197b880fa39d432a9b472845876bfaf0 [term 3 FOLLOWER]: Rejecting Update request from peer 1c50de89a5a845668696dcb28c3badd6 for earlier term 2. Current term is 3. Ops: []
I20260129 02:59:44.104987 20438 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55676 (request call id 1194) took 1940 ms. Trace:
I20260129 02:59:44.105333 20438 rpcz_store.cc:276] 0129 02:59:42.164193 (+ 0us) service_pool.cc:168] Inserting onto call queue
0129 02:59:44.030359 (+1866166us) service_pool.cc:225] Handling call
0129 02:59:44.076220 (+ 45861us) raft_consensus.cc:1471] Updating replica for 0 ops
0129 02:59:44.104705 (+ 28485us) raft_consensus.cc:1697] Filling consensus response to leader.
0129 02:59:44.104927 (+ 222us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260129 02:59:44.106947 20438 raft_consensus.cc:1240] T 1ce4ddf5b10e4ae8b193b13b2e5936ee P 197b880fa39d432a9b472845876bfaf0 [term 5 FOLLOWER]: Rejecting Update request from peer 1c50de89a5a845668696dcb28c3badd6 for earlier term 4. Current term is 5. Ops: [4.3-4.3]
W20260129 02:59:44.108031 20405 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.211.3:37735, user_credentials={real_user=slave}} blocked reactor thread for 114996us
I20260129 02:59:44.109743 21135 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "18f30ebaa4174fd28c958e72d0cc7e24" candidate_uuid: "1c50de89a5a845668696dcb28c3badd6" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "7a9daf5d42a84615a6383cfec9a10608"
I20260129 02:59:44.110366 21135 raft_consensus.cc:3060] T 18f30ebaa4174fd28c958e72d0cc7e24 P 7a9daf5d42a84615a6383cfec9a10608 [term 3 FOLLOWER]: Advancing to term 4
I20260129 02:59:44.112319 20439 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55676 (request call id 1199) took 1946 ms. Trace:
I20260129 02:59:44.112807 20439 rpcz_store.cc:276] 0129 02:59:42.165774 (+ 0us) service_pool.cc:168] Inserting onto call queue
0129 02:59:44.032373 (+1866599us) service_pool.cc:225] Handling call
0129 02:59:44.069179 (+ 36806us) raft_consensus.cc:1471] Updating replica for 0 ops
0129 02:59:44.111984 (+ 42805us) raft_consensus.cc:1534] Early marking committed up to index 0
0129 02:59:44.112008 (+ 24us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0129 02:59:44.112051 (+ 43us) raft_consensus.cc:1645] Marking committed up to 3
0129 02:59:44.112086 (+ 35us) raft_consensus.cc:1697] Filling consensus response to leader.
0129 02:59:44.112155 (+ 69us) raft_consensus.cc:1691] UpdateReplicas() finished
0129 02:59:44.112266 (+ 111us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260129 02:59:44.112345 21130 raft_consensus.cc:1240] T 78aa39096868415d9afc2ffcb248e198 P 7a9daf5d42a84615a6383cfec9a10608 [term 4 FOLLOWER]: Rejecting Update request from peer 1c50de89a5a845668696dcb28c3badd6 for earlier term 3. Current term is 4. Ops: []
I20260129 02:59:44.114431 20439 raft_consensus.cc:1240] T 78aa39096868415d9afc2ffcb248e198 P 197b880fa39d432a9b472845876bfaf0 [term 4 FOLLOWER]: Rejecting Update request from peer 1c50de89a5a845668696dcb28c3badd6 for earlier term 3. Current term is 4. Ops: []
I20260129 02:59:44.109433 22237 raft_consensus.cc:515] T a5a05fbcbbcf4916bbc01fd1a5b5e90d P 197b880fa39d432a9b472845876bfaf0 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:59:44.116989 22237 leader_election.cc:290] T a5a05fbcbbcf4916bbc01fd1a5b5e90d P 197b880fa39d432a9b472845876bfaf0 [CANDIDATE]: Term 6 election: Requested vote from peers 974b802141bd4f13beffef2aa03f88f9 (127.18.211.2:34517), 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3:37735)
I20260129 02:59:44.118865 21128 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fd8311b779af4e1fa1402b2a02a602f2" candidate_uuid: "1c50de89a5a845668696dcb28c3badd6" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "7a9daf5d42a84615a6383cfec9a10608" is_pre_election: true
I20260129 02:59:44.120623 22337 consensus_queue.cc:1059] T 78aa39096868415d9afc2ffcb248e198 P 1c50de89a5a845668696dcb28c3badd6 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false }, Status: INVALID_TERM, Last received: 3.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260129 02:59:44.125166 20439 raft_consensus.cc:1240] T 368953c9102643b2a21f99af36b51905 P 197b880fa39d432a9b472845876bfaf0 [term 5 FOLLOWER]: Rejecting Update request from peer 1c50de89a5a845668696dcb28c3badd6 for earlier term 4. Current term is 5. Ops: [4.3-4.3]
W20260129 02:59:44.135136 21099 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260129 02:59:44.138810 20508 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "febffe4a8b394368b69b1d164308fb9a" candidate_uuid: "197b880fa39d432a9b472845876bfaf0" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "974b802141bd4f13beffef2aa03f88f9" is_pre_election: true
W20260129 02:59:44.139487 21102 leader_election.cc:336] T c934d31efd3f4e5195605ddc354d14e1 P 7a9daf5d42a84615a6383cfec9a10608 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 974b802141bd4f13beffef2aa03f88f9 (127.18.211.2:34517): Timed out: RequestConsensusVote RPC to 127.18.211.2:34517 timed out after 2.185s (SENT)
I20260129 02:59:44.139705 20508 raft_consensus.cc:2468] T febffe4a8b394368b69b1d164308fb9a P 974b802141bd4f13beffef2aa03f88f9 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 197b880fa39d432a9b472845876bfaf0 in term 2.
I20260129 02:59:44.140034 21102 leader_election.cc:304] T c934d31efd3f4e5195605ddc354d14e1 P 7a9daf5d42a84615a6383cfec9a10608 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7a9daf5d42a84615a6383cfec9a10608; no voters: 197b880fa39d432a9b472845876bfaf0, 974b802141bd4f13beffef2aa03f88f9
I20260129 02:59:44.140949 22577 raft_consensus.cc:2749] T c934d31efd3f4e5195605ddc354d14e1 P 7a9daf5d42a84615a6383cfec9a10608 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260129 02:59:44.143373 22197 raft_consensus.cc:2804] T 3ccac82a01a2445fa5b0ce6c0fcf2295 P 974b802141bd4f13beffef2aa03f88f9 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260129 02:59:44.143899 22197 raft_consensus.cc:493] T 3ccac82a01a2445fa5b0ce6c0fcf2295 P 974b802141bd4f13beffef2aa03f88f9 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260129 02:59:44.144280 22197 raft_consensus.cc:3060] T 3ccac82a01a2445fa5b0ce6c0fcf2295 P 974b802141bd4f13beffef2aa03f88f9 [term 2 FOLLOWER]: Advancing to term 3
I20260129 02:59:44.145824 22586 raft_consensus.cc:515] T 368953c9102643b2a21f99af36b51905 P 974b802141bd4f13beffef2aa03f88f9 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:59:44.147293 22445 raft_consensus.cc:3055] T 4408006eb5174de1896b037cf3f18457 P 974b802141bd4f13beffef2aa03f88f9 [term 1 LEADER]: Stepping down as leader of term 1
I20260129 02:59:44.147713 22445 raft_consensus.cc:740] T 4408006eb5174de1896b037cf3f18457 P 974b802141bd4f13beffef2aa03f88f9 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 974b802141bd4f13beffef2aa03f88f9, State: Running, Role: LEADER
I20260129 02:59:44.148738 22445 consensus_queue.cc:260] T 4408006eb5174de1896b037cf3f18457 P 974b802141bd4f13beffef2aa03f88f9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:59:44.150157 20509 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b189f08e16a74a11980cc55d99359a0a" candidate_uuid: "197b880fa39d432a9b472845876bfaf0" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "974b802141bd4f13beffef2aa03f88f9" is_pre_election: true
I20260129 02:59:44.150550 22445 raft_consensus.cc:3060] T 4408006eb5174de1896b037cf3f18457 P 974b802141bd4f13beffef2aa03f88f9 [term 1 FOLLOWER]: Advancing to term 2
I20260129 02:59:44.150995 20509 raft_consensus.cc:2468] T b189f08e16a74a11980cc55d99359a0a P 974b802141bd4f13beffef2aa03f88f9 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 197b880fa39d432a9b472845876bfaf0 in term 5.
I20260129 02:59:44.152547 22372 raft_consensus.cc:3055] T dc2f4cb752cf4d59b5f6bc689f5d8104 P 7a9daf5d42a84615a6383cfec9a10608 [term 3 LEADER]: Stepping down as leader of term 3
I20260129 02:59:44.153041 22372 raft_consensus.cc:740] T dc2f4cb752cf4d59b5f6bc689f5d8104 P 7a9daf5d42a84615a6383cfec9a10608 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 7a9daf5d42a84615a6383cfec9a10608, State: Running, Role: LEADER
I20260129 02:59:44.154115 22372 consensus_queue.cc:260] T dc2f4cb752cf4d59b5f6bc689f5d8104 P 7a9daf5d42a84615a6383cfec9a10608 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.5, Last appended by leader: 5, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } }
W20260129 02:59:44.155155 20555 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260129 02:59:44.155588 22372 raft_consensus.cc:3060] T dc2f4cb752cf4d59b5f6bc689f5d8104 P 7a9daf5d42a84615a6383cfec9a10608 [term 3 FOLLOWER]: Advancing to term 4
I20260129 02:59:44.152474 22554 consensus_queue.cc:260] T 78aa39096868415d9afc2ffcb248e198 P 1c50de89a5a845668696dcb28c3badd6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.5, Last appended by leader: 5, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } }
I20260129 02:59:44.158913 20436 raft_consensus.cc:1240] T a5a05fbcbbcf4916bbc01fd1a5b5e90d P 197b880fa39d432a9b472845876bfaf0 [term 6 FOLLOWER]: Rejecting Update request from peer 974b802141bd4f13beffef2aa03f88f9 for earlier term 5. Current term is 6. Ops: []
I20260129 02:59:44.156857 20553 leader_election.cc:400] T 2e58e9a5a8aa4cecafb464addb4f35ed P 1c50de89a5a845668696dcb28c3badd6 [CANDIDATE]: Term 3 pre-election: Vote denied by peer 197b880fa39d432a9b472845876bfaf0 (127.18.211.1:42027) with higher term. Message: Invalid argument: T 2e58e9a5a8aa4cecafb464addb4f35ed P 197b880fa39d432a9b472845876bfaf0 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1c50de89a5a845668696dcb28c3badd6 for earlier term 3. Current term is 4.
I20260129 02:59:44.155965 22160 raft_consensus.cc:2804] T a8479d1fe0bd41e5946186e3a6ca8c7a P 974b802141bd4f13beffef2aa03f88f9 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260129 02:59:44.158551 20556 leader_election.cc:304] T e4d18c35b0a7493e85d0a4b92956a48f P 1c50de89a5a845668696dcb28c3badd6 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1c50de89a5a845668696dcb28c3badd6, 974b802141bd4f13beffef2aa03f88f9; no voters: 197b880fa39d432a9b472845876bfaf0
I20260129 02:59:44.196964 20553 leader_election.cc:403] T 2e58e9a5a8aa4cecafb464addb4f35ed P 1c50de89a5a845668696dcb28c3badd6 [CANDIDATE]: Term 3 pre-election: Cancelling election due to peer responding with higher term
W20260129 02:59:44.304080 20404 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260129 02:59:44.300732 20406 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260129 02:59:44.262503 21099 leader_election.cc:304] T a609ac02fae84725ab743f95f51d33ce P 7a9daf5d42a84615a6383cfec9a10608 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 197b880fa39d432a9b472845876bfaf0, 7a9daf5d42a84615a6383cfec9a10608; no voters:
I20260129 02:59:44.198596 22586 leader_election.cc:290] T 368953c9102643b2a21f99af36b51905 P 974b802141bd4f13beffef2aa03f88f9 [CANDIDATE]: Term 5 election: Requested vote from peers 197b880fa39d432a9b472845876bfaf0 (127.18.211.1:42027), 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3:37735)
I20260129 02:59:44.258168 22554 raft_consensus.cc:3060] T 78aa39096868415d9afc2ffcb248e198 P 1c50de89a5a845668696dcb28c3badd6 [term 3 FOLLOWER]: Advancing to term 4
W20260129 02:59:44.240757 20482 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260129 02:59:44.260630 22344 consensus_queue.cc:1048] T 7a9a69cd82724b1faed1d10aa091fc4a P 1c50de89a5a845668696dcb28c3badd6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.069s
I20260129 02:59:44.280167 20479 leader_election.cc:304] T a609ac02fae84725ab743f95f51d33ce P 974b802141bd4f13beffef2aa03f88f9 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 197b880fa39d432a9b472845876bfaf0, 974b802141bd4f13beffef2aa03f88f9; no voters:
I20260129 02:59:44.262940 20438 raft_consensus.cc:1240] T cedce331e1584ac8b636c9b8a72b2ba0 P 197b880fa39d432a9b472845876bfaf0 [term 5 FOLLOWER]: Rejecting Update request from peer 1c50de89a5a845668696dcb28c3badd6 for earlier term 4. Current term is 5. Ops: []
W20260129 02:59:44.251049 20480 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.211.3:37735, user_credentials={real_user=slave}} blocked reactor thread for 154951us
W20260129 02:59:44.261775 22540 consensus_meta.cc:287] T c934d31efd3f4e5195605ddc354d14e1 P 197b880fa39d432a9b472845876bfaf0: Time spent flushing consensus metadata: real 0.506s user 0.000s sys 0.009s
W20260129 02:59:44.283821 20405 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.211.3:37735, user_credentials={real_user=slave}} blocked reactor thread for 175006us
I20260129 02:59:44.282742 22197 raft_consensus.cc:515] T 3ccac82a01a2445fa5b0ce6c0fcf2295 P 974b802141bd4f13beffef2aa03f88f9 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } }
I20260129 02:59:44.289110 20336 catalog_manager.cc:5654] T 87d5fe48a44a408aaf03e894a67ecb2f P 7a9daf5d42a84615a6383cfec9a10608 reported cstate change: term changed from 2 to 4. New cstate: current_term: 4 leader_uuid: "7a9daf5d42a84615a6383cfec9a10608" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260129 02:59:44.305080 21135 raft_consensus.cc:2468] T 18f30ebaa4174fd28c958e72d0cc7e24 P 7a9daf5d42a84615a6383cfec9a10608 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1c50de89a5a845668696dcb28c3badd6 in term 4.
I20260129 02:59:44.198488 22160 raft_consensus.cc:493] T a8479d1fe0bd41e5946186e3a6ca8c7a P 974b802141bd4f13beffef2aa03f88f9 [term 3 FOLLOWER]: Starting leader election (detected failure of leader 197b880fa39d432a9b472845876bfaf0)
W20260129 02:59:44.308799 21099 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.211.1:42027, user_credentials={real_user=slave}} blocked reactor thread for 112826us
W20260129 02:59:44.310238 21102 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260129 02:59:44.315579 20437 raft_consensus.cc:2468] T c934d31efd3f4e5195605ddc354d14e1 P 197b880fa39d432a9b472845876bfaf0 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7a9daf5d42a84615a6383cfec9a10608 in term 4.
W20260129 02:59:44.320199 20479 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.211.1:42027, user_credentials={real_user=slave}} blocked reactor thread for 57460.9us
W20260129 02:59:44.324730 20556 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.211.2:34517, user_credentials={real_user=slave}} blocked reactor thread for 166268us
I20260129 02:59:44.316017 22160 raft_consensus.cc:3060] T a8479d1fe0bd41e5946186e3a6ca8c7a P 974b802141bd4f13beffef2aa03f88f9 [term 3 FOLLOWER]: Advancing to term 4
I20260129 02:59:44.315699 22590 raft_consensus.cc:2804] T a609ac02fae84725ab743f95f51d33ce P 7a9daf5d42a84615a6383cfec9a10608 [term 5 FOLLOWER]: Leader pre-election won for term 6
W20260129 02:59:44.329664 20553 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.211.1:42027, user_credentials={real_user=slave}} blocked reactor thread for 173030us
I20260129 02:59:44.328089 20554 leader_election.cc:304] T 18f30ebaa4174fd28c958e72d0cc7e24 P 1c50de89a5a845668696dcb28c3badd6 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1c50de89a5a845668696dcb28c3badd6, 7a9daf5d42a84615a6383cfec9a10608; no voters:
I20260129 02:59:44.329636 20507 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57584 (request call id 1263) took 2107 ms. Trace:
I20260129 02:59:44.321200 22425 raft_consensus.cc:2804] T e4d18c35b0a7493e85d0a4b92956a48f P 1c50de89a5a845668696dcb28c3badd6 [term 3 FOLLOWER]: Leader election won for term 3
I20260129 02:59:44.338289 20553 leader_election.cc:304] T e33a417cf9c54cf5af73a4ee3aac4d3a P 1c50de89a5a845668696dcb28c3badd6 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 197b880fa39d432a9b472845876bfaf0, 1c50de89a5a845668696dcb28c3badd6; no voters:
I20260129 02:59:44.334626 20507 rpcz_store.cc:276] 0129 02:59:42.221991 (+ 0us) service_pool.cc:168] Inserting onto call queue
0129 02:59:43.774968 (+1552977us) service_pool.cc:225] Handling call
0129 02:59:43.918298 (+143330us) raft_consensus.cc:1471] Updating replica for 0 ops
0129 02:59:43.966655 (+ 48357us) raft_consensus.cc:1534] Early marking committed up to index 2
0129 02:59:44.033833 (+ 67178us) log.cc:844] Serialized 14 byte log entry
0129 02:59:44.329269 (+295436us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0129 02:59:44.329321 (+ 52us) raft_consensus.cc:1645] Marking committed up to 2
0129 02:59:44.329382 (+ 61us) raft_consensus.cc:1697] Filling consensus response to leader.
0129 02:59:44.329466 (+ 84us) raft_consensus.cc:1691] UpdateReplicas() finished
0129 02:59:44.329584 (+ 118us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":55886,"prepare.run_cpu_time_us":49,"prepare.run_wall_time_us":50,"raft.queue_time_us":4268,"thread_start_us":259525,"threads_started":1}
I20260129 02:59:44.340008 20514 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a5a05fbcbbcf4916bbc01fd1a5b5e90d" candidate_uuid: "197b880fa39d432a9b472845876bfaf0" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "974b802141bd4f13beffef2aa03f88f9" is_pre_election: true
I20260129 02:59:44.324769 22348 raft_consensus.cc:2749] T 2e58e9a5a8aa4cecafb464addb4f35ed P 1c50de89a5a845668696dcb28c3badd6 [term 4 FOLLOWER]: Leader pre-election lost for term 3. Reason: Vote denied by peer 197b880fa39d432a9b472845876bfaf0 (127.18.211.1:42027) with higher term. Message: Invalid argument: T 2e58e9a5a8aa4cecafb464addb4f35ed P 197b880fa39d432a9b472845876bfaf0 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1c50de89a5a845668696dcb28c3badd6 for earlier term 3. Current term is 4.
I20260129 02:59:44.346788 22425 raft_consensus.cc:697] T e4d18c35b0a7493e85d0a4b92956a48f P 1c50de89a5a845668696dcb28c3badd6 [term 3 LEADER]: Becoming Leader. State: Replica: 1c50de89a5a845668696dcb28c3badd6, State: Running, Role: LEADER
I20260129 02:59:44.348137 22425 consensus_queue.cc:237] T e4d18c35b0a7493e85d0a4b92956a48f P 1c50de89a5a845668696dcb28c3badd6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:59:44.349263 20436 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55672 (request call id 1464) took 2182 ms. Trace:
I20260129 02:59:44.349732 20436 rpcz_store.cc:276] 0129 02:59:42.167340 (+ 0us) service_pool.cc:168] Inserting onto call queue
0129 02:59:44.048446 (+1881106us) service_pool.cc:225] Handling call
0129 02:59:44.109200 (+ 60754us) raft_consensus.cc:1471] Updating replica for 0 ops
0129 02:59:44.201586 (+ 92386us) raft_consensus.cc:1697] Filling consensus response to leader.
0129 02:59:44.348960 (+147374us) spinlock_profiling.cc:243] Waited 13.2 ms on lock 0x7b54002cf6d4. stack: 00007f8fe044e373 00007f8fe044e270 00007f8fdfda0574 00007f8fe9a5c6b4 00007f8fe9a5c649 00007f8fe34975a1 00007f8fe352ea81 00007f8fe352d4be 00007f8fe84e1f49 00007f8fe1a6ed34 00007f8fe1a6ecc2 00007f8fe1a6ebf1 00007f8fe1a6eb6c 00007f8fe1a6de22 00007f8fe186d34c 00007f8fe186c786
0129 02:59:44.349174 (+ 214us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":30334848}
I20260129 02:59:44.353446 20553 leader_election.cc:304] T e7d83d4ccf684e76acb29a0dc844e1b8 P 1c50de89a5a845668696dcb28c3badd6 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 197b880fa39d432a9b472845876bfaf0, 1c50de89a5a845668696dcb28c3badd6; no voters:
I20260129 02:59:44.354463 20553 leader_election.cc:304] T 1299723a4476420ab02fd8bfd7aa0645 P 1c50de89a5a845668696dcb28c3badd6 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 197b880fa39d432a9b472845876bfaf0, 1c50de89a5a845668696dcb28c3badd6; no voters: 7a9daf5d42a84615a6383cfec9a10608
I20260129 02:59:44.355335 22337 raft_consensus.cc:2804] T e7d83d4ccf684e76acb29a0dc844e1b8 P 1c50de89a5a845668696dcb28c3badd6 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260129 02:59:44.355788 22337 raft_consensus.cc:493] T e7d83d4ccf684e76acb29a0dc844e1b8 P 1c50de89a5a845668696dcb28c3badd6 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260129 02:59:44.356197 22337 raft_consensus.cc:3060] T e7d83d4ccf684e76acb29a0dc844e1b8 P 1c50de89a5a845668696dcb28c3badd6 [term 3 FOLLOWER]: Advancing to term 4
I20260129 02:59:44.353499 22581 raft_consensus.cc:2804] T e33a417cf9c54cf5af73a4ee3aac4d3a P 1c50de89a5a845668696dcb28c3badd6 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260129 02:59:44.370474 20515 raft_consensus.cc:2468] T 1ce4ddf5b10e4ae8b193b13b2e5936ee P 974b802141bd4f13beffef2aa03f88f9 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 197b880fa39d432a9b472845876bfaf0 in term 5.
I20260129 02:59:44.339773 20509 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "96609ac68bdf4993a12e1c23355ca013" candidate_uuid: "197b880fa39d432a9b472845876bfaf0" candidate_term: 5 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "974b802141bd4f13beffef2aa03f88f9" is_pre_election: true
I20260129 02:59:44.333482 22590 raft_consensus.cc:493] T a609ac02fae84725ab743f95f51d33ce P 7a9daf5d42a84615a6383cfec9a10608 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260129 02:59:44.376839 20508 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a609ac02fae84725ab743f95f51d33ce" candidate_uuid: "197b880fa39d432a9b472845876bfaf0" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "974b802141bd4f13beffef2aa03f88f9" is_pre_election: true
I20260129 02:59:44.377739 20515 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:57600 (request call id 1717) took 1113 ms. Trace:
W20260129 02:59:44.378172 20554 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260129 02:59:44.378130 20515 rpcz_store.cc:276] 0129 02:59:43.259961 (+ 0us) service_pool.cc:168] Inserting onto call queue
0129 02:59:43.765839 (+505878us) service_pool.cc:225] Handling call
0129 02:59:44.373515 (+607676us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260129 02:59:44.333282 22400 raft_consensus.cc:2804] T 3906b080a5b34a74b8b64128635e8997 P 974b802141bd4f13beffef2aa03f88f9 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260129 02:59:44.355623 20553 leader_election.cc:304] T dc2f4cb752cf4d59b5f6bc689f5d8104 P 1c50de89a5a845668696dcb28c3badd6 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 197b880fa39d432a9b472845876bfaf0, 1c50de89a5a845668696dcb28c3badd6; no voters: 7a9daf5d42a84615a6383cfec9a10608
I20260129 02:59:44.333442 22586 raft_consensus.cc:2804] T a609ac02fae84725ab743f95f51d33ce P 974b802141bd4f13beffef2aa03f88f9 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260129 02:59:44.381075 20507 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "368953c9102643b2a21f99af36b51905" candidate_uuid: "197b880fa39d432a9b472845876bfaf0" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "974b802141bd4f13beffef2aa03f88f9"
W20260129 02:59:44.395620 20437 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:48356 (request call id 255) took 2531 ms (client timeout 2185 ms). Trace:
W20260129 02:59:44.399217 20437 rpcz_store.cc:269] 0129 02:59:41.846383 (+ 0us) service_pool.cc:168] Inserting onto call queue
0129 02:59:43.300488 (+1454105us) service_pool.cc:225] Handling call
0129 02:59:44.377914 (+1077426us) spinlock_profiling.cc:243] Waited 1.01 s on lock 0x7b54002e2054. stack: 00007f8fe044e373 00007f8fe044e270 00007f8fdfda0574 00007f8fe9a5c6b4 00007f8fe9a5c649 00007f8fe34975a1 00007f8fe3531f7a 00007f8fe84e3647 00007f8fe1a73834 00007f8fe1a737c2 00007f8fe1a736f1 00007f8fe1a7366c 00007f8fe1a72922 00007f8fe186d34c 00007f8fe186c786 00007f8fe186c12f
0129 02:59:44.378026 (+ 112us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2331188480}
I20260129 02:59:44.372542 22504 raft_consensus.cc:2804] T 18f30ebaa4174fd28c958e72d0cc7e24 P 1c50de89a5a845668696dcb28c3badd6 [term 4 FOLLOWER]: Leader election won for term 4
I20260129 02:59:44.379472 22590 raft_consensus.cc:3060] T a609ac02fae84725ab743f95f51d33ce P 7a9daf5d42a84615a6383cfec9a10608 [term 5 FOLLOWER]: Advancing to term 6
W20260129 02:59:44.396214 20480 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.211.3:37735, user_credentials={real_user=slave}} blocked reactor thread for 56063.3us
I20260129 02:59:44.388592 20509 raft_consensus.cc:2410] T 96609ac68bdf4993a12e1c23355ca013 P 974b802141bd4f13beffef2aa03f88f9 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 197b880fa39d432a9b472845876bfaf0 for term 5 because replica has last-logged OpId of term: 3 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 4.
I20260129 02:59:44.372992 22581 raft_consensus.cc:493] T e33a417cf9c54cf5af73a4ee3aac4d3a P 1c50de89a5a845668696dcb28c3badd6 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260129 02:59:44.408262 22504 raft_consensus.cc:697] T 18f30ebaa4174fd28c958e72d0cc7e24 P 1c50de89a5a845668696dcb28c3badd6 [term 4 LEADER]: Becoming Leader. State: Replica: 1c50de89a5a845668696dcb28c3badd6, State: Running, Role: LEADER
I20260129 02:59:44.409622 22584 raft_consensus.cc:2804] T dc2f4cb752cf4d59b5f6bc689f5d8104 P 1c50de89a5a845668696dcb28c3badd6 [term 4 FOLLOWER]: Leader election won for term 4
I20260129 02:59:44.412040 22320 raft_consensus.cc:2804] T 1299723a4476420ab02fd8bfd7aa0645 P 1c50de89a5a845668696dcb28c3badd6 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260129 02:59:44.394964 20336 catalog_manager.cc:5654] T dc2f4cb752cf4d59b5f6bc689f5d8104 P 7a9daf5d42a84615a6383cfec9a10608 reported cstate change: term changed from 1 to 3, leader changed from 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3) to 7a9daf5d42a84615a6383cfec9a10608 (127.18.211.4). New cstate: current_term: 3 leader_uuid: "7a9daf5d42a84615a6383cfec9a10608" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
W20260129 02:59:44.405648 20553 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.211.1:42027, user_credentials={real_user=slave}} blocked reactor thread for 50083.2us
W20260129 02:59:44.422402 20404 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260129 02:59:44.417434 22584 raft_consensus.cc:697] T dc2f4cb752cf4d59b5f6bc689f5d8104 P 1c50de89a5a845668696dcb28c3badd6 [term 4 LEADER]: Becoming Leader. State: Replica: 1c50de89a5a845668696dcb28c3badd6, State: Running, Role: LEADER
I20260129 02:59:44.398892 22586 raft_consensus.cc:493] T a609ac02fae84725ab743f95f51d33ce P 974b802141bd4f13beffef2aa03f88f9 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260129 02:59:44.416545 22581 raft_consensus.cc:3060] T e33a417cf9c54cf5af73a4ee3aac4d3a P 1c50de89a5a845668696dcb28c3badd6 [term 4 FOLLOWER]: Advancing to term 5
W20260129 02:59:44.436039 20405 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.211.3:37735, user_credentials={real_user=slave}} blocked reactor thread for 118198us
I20260129 02:59:44.395020 22437 raft_consensus.cc:515] T 913fc7d751ed4b7388d889aac30a5f9b P 974b802141bd4f13beffef2aa03f88f9 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
W20260129 02:59:44.446362 21099 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.211.1:42027, user_credentials={real_user=slave}} blocked reactor thread for 74435.6us
W20260129 02:59:44.446919 20556 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.211.2:34517, user_credentials={real_user=slave}} blocked reactor thread for 75357.2us
I20260129 02:59:44.449753 20336 catalog_manager.cc:5654] T 75f45538dc704d9a8ec4ff8ad9f3ea5b P 7a9daf5d42a84615a6383cfec9a10608 reported cstate change: term changed from 1 to 2, leader changed from 974b802141bd4f13beffef2aa03f88f9 (127.18.211.2) to 7a9daf5d42a84615a6383cfec9a10608 (127.18.211.4). New cstate: current_term: 2 leader_uuid: "7a9daf5d42a84615a6383cfec9a10608" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260129 02:59:44.477876 20553 leader_election.cc:304] T 0340229d521b432580ae926f6062e3f7 P 1c50de89a5a845668696dcb28c3badd6 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 197b880fa39d432a9b472845876bfaf0, 1c50de89a5a845668696dcb28c3badd6; no voters:
I20260129 02:59:44.477100 20480 leader_election.cc:304] T 505580c93c694bbb8ea96fdf9b17ab46 P 974b802141bd4f13beffef2aa03f88f9 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 974b802141bd4f13beffef2aa03f88f9; no voters: 1c50de89a5a845668696dcb28c3badd6, 7a9daf5d42a84615a6383cfec9a10608
I20260129 02:59:44.436369 22586 raft_consensus.cc:3060] T a609ac02fae84725ab743f95f51d33ce P 974b802141bd4f13beffef2aa03f88f9 [term 5 FOLLOWER]: Advancing to term 6
I20260129 02:59:44.483786 21099 leader_election.cc:304] T 78aa39096868415d9afc2ffcb248e198 P 7a9daf5d42a84615a6383cfec9a10608 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7a9daf5d42a84615a6383cfec9a10608; no voters: 197b880fa39d432a9b472845876bfaf0, 1c50de89a5a845668696dcb28c3badd6
W20260129 02:59:44.489073 20406 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260129 02:59:44.491770 22435 raft_consensus.cc:2804] T 0340229d521b432580ae926f6062e3f7 P 1c50de89a5a845668696dcb28c3badd6 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260129 02:59:44.492269 22435 raft_consensus.cc:493] T 0340229d521b432580ae926f6062e3f7 P 1c50de89a5a845668696dcb28c3badd6 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260129 02:59:44.492630 22435 raft_consensus.cc:3060] T 0340229d521b432580ae926f6062e3f7 P 1c50de89a5a845668696dcb28c3badd6 [term 4 FOLLOWER]: Advancing to term 5
I20260129 02:59:44.498027 22584 consensus_queue.cc:237] T dc2f4cb752cf4d59b5f6bc689f5d8104 P 1c50de89a5a845668696dcb28c3badd6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.5, Last appended by leader: 5, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } }
I20260129 02:59:44.501868 22435 raft_consensus.cc:515] T 0340229d521b432580ae926f6062e3f7 P 1c50de89a5a845668696dcb28c3badd6 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:59:44.502313 22437 leader_election.cc:290] T 913fc7d751ed4b7388d889aac30a5f9b P 974b802141bd4f13beffef2aa03f88f9 [CANDIDATE]: Term 3 election: Requested vote from peers 197b880fa39d432a9b472845876bfaf0 (127.18.211.1:42027), 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3:37735)
I20260129 02:59:44.503108 22160 raft_consensus.cc:515] T a8479d1fe0bd41e5946186e3a6ca8c7a P 974b802141bd4f13beffef2aa03f88f9 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:59:44.504563 22435 leader_election.cc:290] T 0340229d521b432580ae926f6062e3f7 P 1c50de89a5a845668696dcb28c3badd6 [CANDIDATE]: Term 5 election: Requested vote from peers 197b880fa39d432a9b472845876bfaf0 (127.18.211.1:42027), 974b802141bd4f13beffef2aa03f88f9 (127.18.211.2:34517)
W20260129 02:59:44.505880 22553 consensus_meta.cc:287] T cedce331e1584ac8b636c9b8a72b2ba0 P 1c50de89a5a845668696dcb28c3badd6: Time spent flushing consensus metadata: real 0.529s user 0.008s sys 0.002s
W20260129 02:59:44.508529 22614 log.cc:927] Time spent T 7a9a69cd82724b1faed1d10aa091fc4a P 197b880fa39d432a9b472845876bfaf0: Append to log took a long time: real 0.096s user 0.001s sys 0.000s
I20260129 02:59:44.509326 20338 rpcz_store.cc:275] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:55348 (request call id 257) took 11171 ms. Trace:
W20260129 02:59:44.516317 22603 log.cc:927] Time spent T 0f20af7fbe5d4db89a6f17948d98912c P 974b802141bd4f13beffef2aa03f88f9: Append to log took a long time: real 0.183s user 0.001s sys 0.000s
W20260129 02:59:44.529309 20553 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.211.1:42027, user_credentials={real_user=slave}} blocked reactor thread for 100929us
I20260129 02:59:44.531826 20437 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "96609ac68bdf4993a12e1c23355ca013" candidate_uuid: "7a9daf5d42a84615a6383cfec9a10608" candidate_term: 5 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "197b880fa39d432a9b472845876bfaf0" is_pre_election: true
I20260129 02:59:44.532298 20436 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fd8311b779af4e1fa1402b2a02a602f2" candidate_uuid: "7a9daf5d42a84615a6383cfec9a10608" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "197b880fa39d432a9b472845876bfaf0"
I20260129 02:59:44.532567 20437 raft_consensus.cc:2468] T 96609ac68bdf4993a12e1c23355ca013 P 197b880fa39d432a9b472845876bfaf0 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7a9daf5d42a84615a6383cfec9a10608 in term 4.
I20260129 02:59:44.532917 20436 raft_consensus.cc:3060] T fd8311b779af4e1fa1402b2a02a602f2 P 197b880fa39d432a9b472845876bfaf0 [term 3 FOLLOWER]: Advancing to term 4
I20260129 02:59:44.533355 20407 leader_election.cc:304] T 3906b080a5b34a74b8b64128635e8997 P 197b880fa39d432a9b472845876bfaf0 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 197b880fa39d432a9b472845876bfaf0, 974b802141bd4f13beffef2aa03f88f9; no voters:
I20260129 02:59:44.533830 20437 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "42d27a28d89b409e8807ec0efd97af69" candidate_uuid: "7a9daf5d42a84615a6383cfec9a10608" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "197b880fa39d432a9b472845876bfaf0"
I20260129 02:59:44.534266 22237 raft_consensus.cc:2804] T 3906b080a5b34a74b8b64128635e8997 P 197b880fa39d432a9b472845876bfaf0 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260129 02:59:44.534464 20407 leader_election.cc:304] T c891d2bff4ce4568861bbc992e91fbb0 P 197b880fa39d432a9b472845876bfaf0 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 197b880fa39d432a9b472845876bfaf0; no voters: 1c50de89a5a845668696dcb28c3badd6, 974b802141bd4f13beffef2aa03f88f9
I20260129 02:59:44.534715 22237 raft_consensus.cc:493] T 3906b080a5b34a74b8b64128635e8997 P 197b880fa39d432a9b472845876bfaf0 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 1c50de89a5a845668696dcb28c3badd6)
I20260129 02:59:44.535080 20438 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2e58e9a5a8aa4cecafb464addb4f35ed" candidate_uuid: "7a9daf5d42a84615a6383cfec9a10608" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "197b880fa39d432a9b472845876bfaf0"
I20260129 02:59:44.535259 22231 raft_consensus.cc:2749] T c891d2bff4ce4568861bbc992e91fbb0 P 197b880fa39d432a9b472845876bfaf0 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260129 02:59:44.535935 20438 raft_consensus.cc:2393] T 2e58e9a5a8aa4cecafb464addb4f35ed P 197b880fa39d432a9b472845876bfaf0 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 7a9daf5d42a84615a6383cfec9a10608 in current term 4: Already voted for candidate 197b880fa39d432a9b472845876bfaf0 in this term.
I20260129 02:59:44.535113 22237 raft_consensus.cc:3060] T 3906b080a5b34a74b8b64128635e8997 P 197b880fa39d432a9b472845876bfaf0 [term 2 FOLLOWER]: Advancing to term 3
I20260129 02:59:44.536864 22437 raft_consensus.cc:2749] T 505580c93c694bbb8ea96fdf9b17ab46 P 974b802141bd4f13beffef2aa03f88f9 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20260129 02:59:44.537175 20439 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2e58e9a5a8aa4cecafb464addb4f35ed" candidate_uuid: "7a9daf5d42a84615a6383cfec9a10608" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "197b880fa39d432a9b472845876bfaf0" is_pre_election: true
I20260129 02:59:44.537551 22437 consensus_queue.cc:1059] T a5a05fbcbbcf4916bbc01fd1a5b5e90d P 974b802141bd4f13beffef2aa03f88f9 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 }, Status: INVALID_TERM, Last received: 5.4, Next index: 5, Last known committed idx: 2, Time since last communication: 0.000s
W20260129 02:59:44.538218 20405 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.211.3:37735, user_credentials={real_user=slave}} blocked reactor thread for 47385.9us
I20260129 02:59:44.538478 22437 raft_consensus.cc:3055] T a5a05fbcbbcf4916bbc01fd1a5b5e90d P 974b802141bd4f13beffef2aa03f88f9 [term 5 LEADER]: Stepping down as leader of term 5
I20260129 02:59:44.538900 22437 raft_consensus.cc:740] T a5a05fbcbbcf4916bbc01fd1a5b5e90d P 974b802141bd4f13beffef2aa03f88f9 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 974b802141bd4f13beffef2aa03f88f9, State: Running, Role: LEADER
I20260129 02:59:44.539089 20590 raft_consensus.cc:1275] T fd8311b779af4e1fa1402b2a02a602f2 P 1c50de89a5a845668696dcb28c3badd6 [term 4 FOLLOWER]: Refusing update from remote peer 7a9daf5d42a84615a6383cfec9a10608: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20260129 02:59:44.537930 20439 raft_consensus.cc:2393] T 2e58e9a5a8aa4cecafb464addb4f35ed P 197b880fa39d432a9b472845876bfaf0 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 7a9daf5d42a84615a6383cfec9a10608 in current term 4: Already voted for candidate 197b880fa39d432a9b472845876bfaf0 in this term.
W20260129 02:59:44.540298 22612 log.cc:927] Time spent T 1299723a4476420ab02fd8bfd7aa0645 P 1c50de89a5a845668696dcb28c3badd6: Append to log took a long time: real 0.091s user 0.001s sys 0.000s
I20260129 02:59:44.540638 22437 consensus_queue.cc:260] T a5a05fbcbbcf4916bbc01fd1a5b5e90d P 974b802141bd4f13beffef2aa03f88f9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 5.4, Last appended by leader: 4, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:59:44.541342 22364 consensus_queue.cc:1048] T fd8311b779af4e1fa1402b2a02a602f2 P 7a9daf5d42a84615a6383cfec9a10608 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260129 02:59:44.541976 22437 raft_consensus.cc:3060] T a5a05fbcbbcf4916bbc01fd1a5b5e90d P 974b802141bd4f13beffef2aa03f88f9 [term 5 FOLLOWER]: Advancing to term 6
W20260129 02:59:44.542152 22613 log.cc:927] Time spent T 1299723a4476420ab02fd8bfd7aa0645 P 7a9daf5d42a84615a6383cfec9a10608: Append to log took a long time: real 0.108s user 0.000s sys 0.000s
I20260129 02:59:44.547957 22403 raft_consensus.cc:2749] T 78aa39096868415d9afc2ffcb248e198 P 7a9daf5d42a84615a6383cfec9a10608 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20260129 02:59:44.547708 22337 raft_consensus.cc:515] T e7d83d4ccf684e76acb29a0dc844e1b8 P 1c50de89a5a845668696dcb28c3badd6 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:59:44.550129 22337 leader_election.cc:290] T e7d83d4ccf684e76acb29a0dc844e1b8 P 1c50de89a5a845668696dcb28c3badd6 [CANDIDATE]: Term 4 election: Requested vote from peers 197b880fa39d432a9b472845876bfaf0 (127.18.211.1:42027), 974b802141bd4f13beffef2aa03f88f9 (127.18.211.2:34517)
I20260129 02:59:44.551998 22160 leader_election.cc:290] T a8479d1fe0bd41e5946186e3a6ca8c7a P 974b802141bd4f13beffef2aa03f88f9 [CANDIDATE]: Term 4 election: Requested vote from peers 197b880fa39d432a9b472845876bfaf0 (127.18.211.1:42027), 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3:37735)
I20260129 02:59:44.554808 20436 raft_consensus.cc:2468] T fd8311b779af4e1fa1402b2a02a602f2 P 197b880fa39d432a9b472845876bfaf0 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7a9daf5d42a84615a6383cfec9a10608 in term 4.
I20260129 02:59:44.561153 22237 raft_consensus.cc:515] T 3906b080a5b34a74b8b64128635e8997 P 197b880fa39d432a9b472845876bfaf0 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:59:44.573309 22320 raft_consensus.cc:493] T 1299723a4476420ab02fd8bfd7aa0645 P 1c50de89a5a845668696dcb28c3badd6 [term 6 FOLLOWER]: Starting leader election (detected failure of leader 7a9daf5d42a84615a6383cfec9a10608)
I20260129 02:59:44.573958 22320 raft_consensus.cc:3060] T 1299723a4476420ab02fd8bfd7aa0645 P 1c50de89a5a845668696dcb28c3badd6 [term 6 FOLLOWER]: Advancing to term 7
I20260129 02:59:44.565776 22237 leader_election.cc:290] T 3906b080a5b34a74b8b64128635e8997 P 197b880fa39d432a9b472845876bfaf0 [CANDIDATE]: Term 3 election: Requested vote from peers 974b802141bd4f13beffef2aa03f88f9 (127.18.211.2:34517), 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3:37735)
I20260129 02:59:44.578190 22400 raft_consensus.cc:493] T 3906b080a5b34a74b8b64128635e8997 P 974b802141bd4f13beffef2aa03f88f9 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 1c50de89a5a845668696dcb28c3badd6)
I20260129 02:59:44.578812 22400 raft_consensus.cc:3060] T 3906b080a5b34a74b8b64128635e8997 P 974b802141bd4f13beffef2aa03f88f9 [term 2 FOLLOWER]: Advancing to term 3
I20260129 02:59:44.490370 22540 consensus_queue.cc:1048] T 8055232cf0004d40a85f3bf983fe00a9 P 197b880fa39d432a9b472845876bfaf0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.100s
I20260129 02:59:44.517164 22197 leader_election.cc:290] T 3ccac82a01a2445fa5b0ce6c0fcf2295 P 974b802141bd4f13beffef2aa03f88f9 [CANDIDATE]: Term 3 election: Requested vote from peers 197b880fa39d432a9b472845876bfaf0 (127.18.211.1:42027), 7a9daf5d42a84615a6383cfec9a10608 (127.18.211.4:40439)
W20260129 02:59:44.592728 21099 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.211.1:42027, user_credentials={real_user=slave}} blocked reactor thread for 116017us
W20260129 02:59:44.597572 21102 leader_election.cc:336] T 4e420f90884f4796997b5ea41e49e411 P 7a9daf5d42a84615a6383cfec9a10608 [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer 974b802141bd4f13beffef2aa03f88f9 (127.18.211.2:34517): Timed out: RequestConsensusVote RPC to 127.18.211.2:34517 timed out after 1.651s (SENT)
I20260129 02:59:44.598212 21102 leader_election.cc:304] T 4e420f90884f4796997b5ea41e49e411 P 7a9daf5d42a84615a6383cfec9a10608 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7a9daf5d42a84615a6383cfec9a10608; no voters: 197b880fa39d432a9b472845876bfaf0, 974b802141bd4f13beffef2aa03f88f9
I20260129 02:59:44.599414 22372 raft_consensus.cc:2749] T 4e420f90884f4796997b5ea41e49e411 P 7a9daf5d42a84615a6383cfec9a10608 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260129 02:59:44.600833 20441 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55676 (request call id 1206) took 1208 ms. Trace:
I20260129 02:59:44.601240 20441 rpcz_store.cc:276] 0129 02:59:43.392715 (+ 0us) service_pool.cc:168] Inserting onto call queue
0129 02:59:44.066078 (+673363us) service_pool.cc:225] Handling call
0129 02:59:44.066457 (+ 379us) raft_consensus.cc:1471] Updating replica for 1 ops
0129 02:59:44.066971 (+ 514us) raft_consensus.cc:1534] Early marking committed up to index 0
0129 02:59:44.067012 (+ 41us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0129 02:59:44.067588 (+ 576us) log.cc:844] Serialized 26 byte log entry
0129 02:59:44.308920 (+241332us) raft_consensus.cc:1645] Marking committed up to 2
0129 02:59:44.308961 (+ 41us) raft_consensus.cc:1697] Filling consensus response to leader.
0129 02:59:44.309030 (+ 69us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0129 02:59:44.600582 (+291552us) raft_consensus.cc:1685] finished
0129 02:59:44.600604 (+ 22us) raft_consensus.cc:1691] UpdateReplicas() finished
0129 02:59:44.600749 (+ 145us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":134,"raft.run_cpu_time_us":122,"raft.run_wall_time_us":123,"thread_start_us":240844,"threads_started":1,"wal-append.queue_time_us":344337}
W20260129 02:59:44.536015 20479 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.211.1:42027, user_credentials={real_user=slave}} blocked reactor thread for 48601.9us
W20260129 02:59:44.599753 21102 leader_election.cc:336] T 2ac659d732314e1288849f08fe5e074e P 7a9daf5d42a84615a6383cfec9a10608 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 974b802141bd4f13beffef2aa03f88f9 (127.18.211.2:34517): Timed out: RequestConsensusVote RPC to 127.18.211.2:34517 timed out after 2.865s (SENT)
I20260129 02:59:44.621461 21102 leader_election.cc:304] T 2ac659d732314e1288849f08fe5e074e P 7a9daf5d42a84615a6383cfec9a10608 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7a9daf5d42a84615a6383cfec9a10608; no voters: 197b880fa39d432a9b472845876bfaf0, 974b802141bd4f13beffef2aa03f88f9
I20260129 02:59:44.538954 20405 leader_election.cc:304] T 17d6431cc77a491e8464d68db4081a51 P 197b880fa39d432a9b472845876bfaf0 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 197b880fa39d432a9b472845876bfaf0, 1c50de89a5a845668696dcb28c3badd6; no voters:
W20260129 02:59:44.603616 22611 log.cc:927] Time spent T fd8311b779af4e1fa1402b2a02a602f2 P 7a9daf5d42a84615a6383cfec9a10608: Append to log took a long time: real 0.213s user 0.000s sys 0.001s
W20260129 02:59:44.630596 20405 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.211.3:37735, user_credentials={real_user=slave}} blocked reactor thread for 91756.3us
I20260129 02:59:44.633919 20407 leader_election.cc:304] T 0f20af7fbe5d4db89a6f17948d98912c P 197b880fa39d432a9b472845876bfaf0 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 197b880fa39d432a9b472845876bfaf0, 974b802141bd4f13beffef2aa03f88f9; no voters:
I20260129 02:59:44.628221 20553 leader_election.cc:304] T fd8311b779af4e1fa1402b2a02a602f2 P 1c50de89a5a845668696dcb28c3badd6 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 197b880fa39d432a9b472845876bfaf0, 1c50de89a5a845668696dcb28c3badd6; no voters: 7a9daf5d42a84615a6383cfec9a10608
W20260129 02:59:44.635818 20407 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.211.2:34517, user_credentials={real_user=slave}} blocked reactor thread for 100036us
I20260129 02:59:44.487110 22504 consensus_queue.cc:237] T 18f30ebaa4174fd28c958e72d0cc7e24 P 1c50de89a5a845668696dcb28c3badd6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } }
I20260129 02:59:44.652665 20433 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55676 (request call id 1207) took 1102 ms. Trace:
W20260129 02:59:44.677850 21102 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.211.2:34517, user_credentials={real_user=slave}} blocked reactor thread for 78118.8us
I20260129 02:59:44.645800 22540 raft_consensus.cc:2804] T 17d6431cc77a491e8464d68db4081a51 P 197b880fa39d432a9b472845876bfaf0 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260129 02:59:44.676759 20433 rpcz_store.cc:276] 0129 02:59:43.549923 (+ 0us) service_pool.cc:168] Inserting onto call queue
0129 02:59:44.092439 (+542516us) service_pool.cc:225] Handling call
0129 02:59:44.092786 (+ 347us) raft_consensus.cc:1471] Updating replica for 0 ops
0129 02:59:44.093042 (+ 256us) raft_consensus.cc:1534] Early marking committed up to index 2
0129 02:59:44.320105 (+227063us) log.cc:844] Serialized 14 byte log entry
0129 02:59:44.652127 (+332022us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0129 02:59:44.652209 (+ 82us) raft_consensus.cc:1645] Marking committed up to 2
0129 02:59:44.652273 (+ 64us) raft_consensus.cc:1697] Filling consensus response to leader.
0129 02:59:44.652373 (+ 100us) raft_consensus.cc:1691] UpdateReplicas() finished
0129 02:59:44.652550 (+ 177us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":21606,"prepare.queue_time_us":373632,"prepare.run_cpu_time_us":87,"prepare.run_wall_time_us":134,"raft.queue_time_us":21774,"thread_start_us":511298,"threads_started":2}
W20260129 02:59:44.690557 21099 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260129 02:59:44.700186 20556 leader_election.cc:304] T 4408006eb5174de1896b037cf3f18457 P 1c50de89a5a845668696dcb28c3badd6 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1c50de89a5a845668696dcb28c3badd6; no voters: 197b880fa39d432a9b472845876bfaf0, 974b802141bd4f13beffef2aa03f88f9
I20260129 02:59:44.700397 20441 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1299723a4476420ab02fd8bfd7aa0645" candidate_uuid: "974b802141bd4f13beffef2aa03f88f9" candidate_term: 5 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "197b880fa39d432a9b472845876bfaf0" is_pre_election: true
I20260129 02:59:44.702201 20441 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "cedce331e1584ac8b636c9b8a72b2ba0" candidate_uuid: "974b802141bd4f13beffef2aa03f88f9" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "197b880fa39d432a9b472845876bfaf0" is_pre_election: true
I20260129 02:59:44.703197 20441 raft_consensus.cc:2393] T cedce331e1584ac8b636c9b8a72b2ba0 P 197b880fa39d432a9b472845876bfaf0 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 974b802141bd4f13beffef2aa03f88f9 in current term 5: Already voted for candidate 197b880fa39d432a9b472845876bfaf0 in this term.
I20260129 02:59:44.704046 22425 raft_consensus.cc:2749] T 4408006eb5174de1896b037cf3f18457 P 1c50de89a5a845668696dcb28c3badd6 [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260129 02:59:44.704304 20439 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2ac659d732314e1288849f08fe5e074e" candidate_uuid: "974b802141bd4f13beffef2aa03f88f9" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "197b880fa39d432a9b472845876bfaf0"
W20260129 02:59:44.664103 22184 consensus_meta.cc:287] T 17d6431cc77a491e8464d68db4081a51 P 974b802141bd4f13beffef2aa03f88f9: Time spent flushing consensus metadata: real 0.588s user 0.007s sys 0.003s
W20260129 02:59:44.706622 21102 leader_election.cc:336] T a609ac02fae84725ab743f95f51d33ce P 7a9daf5d42a84615a6383cfec9a10608 [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer 974b802141bd4f13beffef2aa03f88f9 (127.18.211.2:34517): Timed out: RequestConsensusVote RPC to 127.18.211.2:34517 timed out after 2.131s (SENT)
I20260129 02:59:44.672837 22237 raft_consensus.cc:2804] T 0f20af7fbe5d4db89a6f17948d98912c P 197b880fa39d432a9b472845876bfaf0 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260129 02:59:44.703577 20556 leader_election.cc:304] T a8479d1fe0bd41e5946186e3a6ca8c7a P 1c50de89a5a845668696dcb28c3badd6 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1c50de89a5a845668696dcb28c3badd6, 974b802141bd4f13beffef2aa03f88f9; no voters:
I20260129 02:59:44.709039 20439 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "724a236524cb4e7ba3428880681d6cc2" candidate_uuid: "974b802141bd4f13beffef2aa03f88f9" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "197b880fa39d432a9b472845876bfaf0"
I20260129 02:59:44.685500 22364 raft_consensus.cc:2749] T 2ac659d732314e1288849f08fe5e074e P 7a9daf5d42a84615a6383cfec9a10608 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260129 02:59:44.725195 20437 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "cb4727386a3e48aca5d71cc6c611c197" candidate_uuid: "974b802141bd4f13beffef2aa03f88f9" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "197b880fa39d432a9b472845876bfaf0" is_pre_election: true
I20260129 02:59:44.728001 20589 raft_consensus.cc:1275] T 2e58e9a5a8aa4cecafb464addb4f35ed P 1c50de89a5a845668696dcb28c3badd6 [term 4 FOLLOWER]: Refusing update from remote peer 7a9daf5d42a84615a6383cfec9a10608: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20260129 02:59:44.773022 22197 raft_consensus.cc:493] T a644c02b16af47679824ccd060ac10fd P 974b802141bd4f13beffef2aa03f88f9 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260129 02:59:44.709244 20438 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "febffe4a8b394368b69b1d164308fb9a" candidate_uuid: "974b802141bd4f13beffef2aa03f88f9" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "197b880fa39d432a9b472845876bfaf0" is_pre_election: true
I20260129 02:59:44.725558 20441 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "8055232cf0004d40a85f3bf983fe00a9" candidate_uuid: "974b802141bd4f13beffef2aa03f88f9" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "197b880fa39d432a9b472845876bfaf0"
I20260129 02:59:44.729187 20586 raft_consensus.cc:1275] T 42d27a28d89b409e8807ec0efd97af69 P 1c50de89a5a845668696dcb28c3badd6 [term 5 FOLLOWER]: Refusing update from remote peer 7a9daf5d42a84615a6383cfec9a10608: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 5 index: 7. (index mismatch)
I20260129 02:59:44.752703 20433 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fcddab6bca684adf8ea032289a00f90e" candidate_uuid: "974b802141bd4f13beffef2aa03f88f9" candidate_term: 8 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "197b880fa39d432a9b472845876bfaf0" is_pre_election: true
W20260129 02:59:44.753654 20555 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260129 02:59:44.798480 20553 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.211.1:42027, user_credentials={real_user=slave}} blocked reactor thread for 91346.5us
W20260129 02:59:44.754977 20405 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.211.3:37735, user_credentials={real_user=slave}} blocked reactor thread for 109404us
I20260129 02:59:44.706255 22540 raft_consensus.cc:493] T 17d6431cc77a491e8464d68db4081a51 P 197b880fa39d432a9b472845876bfaf0 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260129 02:59:44.708720 20407 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.211.2:34517, user_credentials={real_user=slave}} blocked reactor thread for 59786.4us
W20260129 02:59:44.728989 22566 consensus_meta.cc:287] T 0340229d521b432580ae926f6062e3f7 P 974b802141bd4f13beffef2aa03f88f9: Time spent flushing consensus metadata: real 0.639s user 0.006s sys 0.005s
W20260129 02:59:44.763263 20556 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.211.2:34517, user_credentials={real_user=slave}} blocked reactor thread for 59755.6us
W20260129 02:59:44.763259 21102 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.211.2:34517, user_credentials={real_user=slave}} blocked reactor thread for 56665.5us
I20260129 02:59:44.764082 20439 raft_consensus.cc:3060] T 724a236524cb4e7ba3428880681d6cc2 P 197b880fa39d432a9b472845876bfaf0 [term 2 FOLLOWER]: Advancing to term 3
I20260129 02:59:44.530416 20338 rpcz_store.cc:276] 0129 02:59:33.337771 (+ 0us) service_pool.cc:168] Inserting onto call queue
0129 02:59:33.337977 (+ 206us) service_pool.cc:225] Handling call
0129 02:59:44.509097 (+11171120us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0129 02:59:41.638017 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0129 02:59:41.777081 (+139064us) write_op.cc:432] Acquiring schema lock in shared mode
0129 02:59:41.809636 (+ 32555us) write_op.cc:435] Acquired schema lock
0129 02:59:41.810693 (+ 1057us) tablet.cc:660] Decoding operations
0129 02:59:42.250398 (+439705us) write_op.cc:620] Acquiring the partition lock for write op
0129 02:59:42.272226 (+ 21828us) write_op.cc:641] Partition lock acquired for write op
0129 02:59:42.282532 (+ 10306us) tablet.cc:683] Acquiring locks for 5 operations
0129 02:59:42.473209 (+190677us) tablet.cc:699] Row locks acquired
0129 02:59:42.476526 (+ 3317us) write_op.cc:260] PREPARE: finished
0129 02:59:42.708620 (+232094us) write_op.cc:270] Start()
0129 02:59:42.775097 (+ 66477us) write_op.cc:276] Timestamp: P: 1769655582664942 usec, L: 0
0129 02:59:42.790863 (+ 15766us) op_driver.cc:348] REPLICATION: starting
0129 02:59:42.923140 (+132277us) log.cc:844] Serialized 2321 byte log entry
0129 02:59:43.692719 (+769579us) op_driver.cc:464] REPLICATION: finished
0129 02:59:43.935229 (+242510us) write_op.cc:301] APPLY: starting
0129 02:59:43.935378 (+ 149us) tablet.cc:1365] starting BulkCheckPresence
0129 02:59:43.935611 (+ 233us) tablet.cc:1368] finished BulkCheckPresence
0129 02:59:43.935776 (+ 165us) tablet.cc:1370] starting ApplyRowOperation cycle
0129 02:59:43.936730 (+ 954us) tablet.cc:1381] finished ApplyRowOperation cycle
0129 02:59:43.937048 (+ 318us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=5
0129 02:59:43.941852 (+ 4804us) write_op.cc:312] APPLY: finished
0129 02:59:43.942305 (+ 453us) log.cc:844] Serialized 47 byte log entry
0129 02:59:43.942566 (+ 261us) write_op.cc:489] Releasing partition, row and schema locks
0129 02:59:43.951960 (+ 9394us) write_op.cc:454] Released schema lock
0129 02:59:43.952036 (+ 76us) write_op.cc:341] FINISH: Updating metrics
0129 02:59:43.952581 (+ 545us) write_op.cc:489] Releasing partition, row and schema locks
0129 02:59:43.952611 (+ 30us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":67310,"reported_tablets":8,"child_traces":[["op",{"apply.queue_time_us":233790,"apply.run_cpu_time_us":3472,"apply.run_wall_time_us":17344,"num_ops":5,"prepare.queue_time_us":2092539,"prepare.run_cpu_time_us":21171,"prepare.run_wall_time_us":1642611,"raft.queue_time_us":329390,"raft.run_cpu_time_us":14740,"raft.run_wall_time_us":293873,"replication_time_us":856027,"thread_start_us":1737721,"threads_started":4,"wal-append.queue_time_us":113642}]]}
I20260129 02:59:44.762071 22445 raft_consensus.cc:493] T 76f713c3331540d2a5b76164103e9eb1 P 974b802141bd4f13beffef2aa03f88f9 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 197b880fa39d432a9b472845876bfaf0)
I20260129 02:59:44.768004 22425 raft_consensus.cc:2804] T a8479d1fe0bd41e5946186e3a6ca8c7a P 1c50de89a5a845668696dcb28c3badd6 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260129 02:59:44.758183 22437 raft_consensus.cc:493] T 0678b8b468bd43cdbc3e485900844821 P 974b802141bd4f13beffef2aa03f88f9 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260129 02:59:44.740252 22237 raft_consensus.cc:493] T 0f20af7fbe5d4db89a6f17948d98912c P 197b880fa39d432a9b472845876bfaf0 [term 4 FOLLOWER]: Starting leader election (detected failure of leader 1c50de89a5a845668696dcb28c3badd6)
I20260129 02:59:44.726943 22590 raft_consensus.cc:515] T a609ac02fae84725ab743f95f51d33ce P 7a9daf5d42a84615a6383cfec9a10608 [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } }
I20260129 02:59:44.709154 22184 raft_consensus.cc:515] T 17d6431cc77a491e8464d68db4081a51 P 974b802141bd4f13beffef2aa03f88f9 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:59:44.794401 20437 raft_consensus.cc:2468] T cb4727386a3e48aca5d71cc6c611c197 P 197b880fa39d432a9b472845876bfaf0 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 974b802141bd4f13beffef2aa03f88f9 in term 2.
W20260129 02:59:44.724421 20406 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260129 02:59:44.772976 21128 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.629s user 0.013s sys 0.019s
I20260129 02:59:44.803493 20438 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "368953c9102643b2a21f99af36b51905" candidate_uuid: "974b802141bd4f13beffef2aa03f88f9" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "197b880fa39d432a9b472845876bfaf0"
I20260129 02:59:44.805824 20441 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "75be4bd6bd45474795fd472eff2953ac" candidate_uuid: "974b802141bd4f13beffef2aa03f88f9" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "197b880fa39d432a9b472845876bfaf0" is_pre_election: true
I20260129 02:59:44.802611 22540 raft_consensus.cc:3060] T 17d6431cc77a491e8464d68db4081a51 P 197b880fa39d432a9b472845876bfaf0 [term 2 FOLLOWER]: Advancing to term 3
I20260129 02:59:44.820901 20438 raft_consensus.cc:2393] T 368953c9102643b2a21f99af36b51905 P 197b880fa39d432a9b472845876bfaf0 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 974b802141bd4f13beffef2aa03f88f9 in current term 5: Already voted for candidate 197b880fa39d432a9b472845876bfaf0 in this term.
I20260129 02:59:44.828547 20438 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e4d18c35b0a7493e85d0a4b92956a48f" candidate_uuid: "974b802141bd4f13beffef2aa03f88f9" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "197b880fa39d432a9b472845876bfaf0"
I20260129 02:59:44.829635 20437 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3ccac82a01a2445fa5b0ce6c0fcf2295" candidate_uuid: "974b802141bd4f13beffef2aa03f88f9" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "197b880fa39d432a9b472845876bfaf0"
I20260129 02:59:44.830385 20437 raft_consensus.cc:3060] T 3ccac82a01a2445fa5b0ce6c0fcf2295 P 197b880fa39d432a9b472845876bfaf0 [term 2 FOLLOWER]: Advancing to term 3
I20260129 02:59:44.813776 22237 raft_consensus.cc:3060] T 0f20af7fbe5d4db89a6f17948d98912c P 197b880fa39d432a9b472845876bfaf0 [term 4 FOLLOWER]: Advancing to term 5
I20260129 02:59:44.821079 20433 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "913fc7d751ed4b7388d889aac30a5f9b" candidate_uuid: "974b802141bd4f13beffef2aa03f88f9" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "197b880fa39d432a9b472845876bfaf0"
I20260129 02:59:44.806603 22425 raft_consensus.cc:493] T a8479d1fe0bd41e5946186e3a6ca8c7a P 1c50de89a5a845668696dcb28c3badd6 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260129 02:59:44.838477 20441 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a8479d1fe0bd41e5946186e3a6ca8c7a" candidate_uuid: "974b802141bd4f13beffef2aa03f88f9" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "197b880fa39d432a9b472845876bfaf0"
W20260129 02:59:44.824092 20404 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260129 02:59:44.834606 20438 raft_consensus.cc:2393] T e4d18c35b0a7493e85d0a4b92956a48f P 197b880fa39d432a9b472845876bfaf0 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 974b802141bd4f13beffef2aa03f88f9 in current term 4: Already voted for candidate 197b880fa39d432a9b472845876bfaf0 in this term.
I20260129 02:59:44.808401 22197 raft_consensus.cc:515] T a644c02b16af47679824ccd060ac10fd P 974b802141bd4f13beffef2aa03f88f9 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } }
I20260129 02:59:44.808943 22445 raft_consensus.cc:515] T 76f713c3331540d2a5b76164103e9eb1 P 974b802141bd4f13beffef2aa03f88f9 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:59:44.836387 22184 leader_election.cc:290] T 17d6431cc77a491e8464d68db4081a51 P 974b802141bd4f13beffef2aa03f88f9 [CANDIDATE]: Term 3 election: Requested vote from peers 197b880fa39d432a9b472845876bfaf0 (127.18.211.1:42027), 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3:37735)
W20260129 02:59:44.837824 22620 log.cc:927] Time spent T 1299723a4476420ab02fd8bfd7aa0645 P 197b880fa39d432a9b472845876bfaf0: Append to log took a long time: real 0.076s user 0.000s sys 0.001s
W20260129 02:59:44.850535 20480 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.211.4:40439, user_credentials={real_user=slave}} blocked reactor thread for 373542us
W20260129 02:59:44.820962 21128 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.677s user 0.013s sys 0.020s
I20260129 02:59:44.852447 20480 leader_election.cc:304] T 75f45538dc704d9a8ec4ff8ad9f3ea5b P 974b802141bd4f13beffef2aa03f88f9 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 974b802141bd4f13beffef2aa03f88f9; no voters: 1c50de89a5a845668696dcb28c3badd6, 7a9daf5d42a84615a6383cfec9a10608
W20260129 02:59:44.855209 20404 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260129 02:59:44.861876 20480 leader_election.cc:304] T 4f5a77e94fef4b16a2ba7b067c8b144a P 974b802141bd4f13beffef2aa03f88f9 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 974b802141bd4f13beffef2aa03f88f9; no voters: 1c50de89a5a845668696dcb28c3badd6, 7a9daf5d42a84615a6383cfec9a10608
I20260129 02:59:44.863276 20480 leader_election.cc:304] T 4e420f90884f4796997b5ea41e49e411 P 974b802141bd4f13beffef2aa03f88f9 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 974b802141bd4f13beffef2aa03f88f9; no voters: 197b880fa39d432a9b472845876bfaf0, 7a9daf5d42a84615a6383cfec9a10608
I20260129 02:59:44.809267 22566 raft_consensus.cc:515] T 0340229d521b432580ae926f6062e3f7 P 974b802141bd4f13beffef2aa03f88f9 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:59:44.859962 20407 leader_election.cc:304] T 75be4bd6bd45474795fd472eff2953ac P 197b880fa39d432a9b472845876bfaf0 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 197b880fa39d432a9b472845876bfaf0, 974b802141bd4f13beffef2aa03f88f9; no voters:
I20260129 02:59:44.863317 22184 raft_consensus.cc:2749] T 4f5a77e94fef4b16a2ba7b067c8b144a P 974b802141bd4f13beffef2aa03f88f9 [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260129 02:59:44.813483 22372 consensus_queue.cc:1048] T 2e58e9a5a8aa4cecafb464addb4f35ed P 7a9daf5d42a84615a6383cfec9a10608 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.037s
I20260129 02:59:44.865999 20441 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7d9494f5584d4d5db8b9adf1c8fc888b" candidate_uuid: "974b802141bd4f13beffef2aa03f88f9" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "197b880fa39d432a9b472845876bfaf0" is_pre_election: true
I20260129 02:59:44.820684 22403 consensus_queue.cc:1048] T 42d27a28d89b409e8807ec0efd97af69 P 7a9daf5d42a84615a6383cfec9a10608 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 5, Time since last communication: 0.032s
I20260129 02:59:44.848452 22425 raft_consensus.cc:3060] T a8479d1fe0bd41e5946186e3a6ca8c7a P 1c50de89a5a845668696dcb28c3badd6 [term 3 FOLLOWER]: Advancing to term 4
I20260129 02:59:44.815984 22437 raft_consensus.cc:515] T 0678b8b468bd43cdbc3e485900844821 P 974b802141bd4f13beffef2aa03f88f9 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } }
I20260129 02:59:44.873976 20432 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55676 (request call id 1204) took 1571 ms. Trace:
I20260129 02:59:44.874382 20432 rpcz_store.cc:276] 0129 02:59:43.302382 (+ 0us) service_pool.cc:168] Inserting onto call queue
0129 02:59:44.056760 (+754378us) service_pool.cc:225] Handling call
0129 02:59:44.121283 (+ 64523us) raft_consensus.cc:1471] Updating replica for 0 ops
0129 02:59:44.121664 (+ 381us) raft_consensus.cc:1534] Early marking committed up to index 2
0129 02:59:44.376815 (+255151us) log.cc:844] Serialized 14 byte log entry
0129 02:59:44.873615 (+496800us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0129 02:59:44.873676 (+ 61us) raft_consensus.cc:1645] Marking committed up to 2
0129 02:59:44.873729 (+ 53us) raft_consensus.cc:1697] Filling consensus response to leader.
0129 02:59:44.873803 (+ 74us) raft_consensus.cc:1691] UpdateReplicas() finished
0129 02:59:44.873921 (+ 118us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":434255,"prepare.run_cpu_time_us":42,"prepare.run_wall_time_us":43,"thread_start_us":685406,"threads_started":2}
I20260129 02:59:44.862028 22493 raft_consensus.cc:2749] T 75f45538dc704d9a8ec4ff8ad9f3ea5b P 974b802141bd4f13beffef2aa03f88f9 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260129 02:59:44.892722 20480 leader_election.cc:304] T 45cc5f3d2688486d9b6dbe7e666dedde P 974b802141bd4f13beffef2aa03f88f9 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 7a9daf5d42a84615a6383cfec9a10608, 974b802141bd4f13beffef2aa03f88f9; no voters: 1c50de89a5a845668696dcb28c3badd6
I20260129 02:59:44.866602 22553 raft_consensus.cc:2764] T fd8311b779af4e1fa1402b2a02a602f2 P 1c50de89a5a845668696dcb28c3badd6 [term 4 FOLLOWER]: Leader pre-election decision vote started in defunct term 3: won
I20260129 02:59:44.868341 22160 raft_consensus.cc:2749] T 4e420f90884f4796997b5ea41e49e411 P 974b802141bd4f13beffef2aa03f88f9 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260129 02:59:44.905475 20556 leader_election.cc:304] T cb0b3f259ebc4310927338a8740e87b8 P 1c50de89a5a845668696dcb28c3badd6 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1c50de89a5a845668696dcb28c3badd6, 974b802141bd4f13beffef2aa03f88f9; no voters: 7a9daf5d42a84615a6383cfec9a10608
I20260129 02:59:44.906471 20479 leader_election.cc:304] T cb4727386a3e48aca5d71cc6c611c197 P 974b802141bd4f13beffef2aa03f88f9 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 197b880fa39d432a9b472845876bfaf0, 974b802141bd4f13beffef2aa03f88f9; no voters:
W20260129 02:59:44.901525 22623 log.cc:927] Time spent T a4c1226b987b468fa84dfc6ba0e2dbe1 P 197b880fa39d432a9b472845876bfaf0: Append to log took a long time: real 0.069s user 0.000s sys 0.001s
I20260129 02:59:44.911510 22197 leader_election.cc:290] T a644c02b16af47679824ccd060ac10fd P 974b802141bd4f13beffef2aa03f88f9 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3:37735), 7a9daf5d42a84615a6383cfec9a10608 (127.18.211.4:40439)
W20260129 02:59:44.905268 22621 log.cc:927] Time spent T 913fc7d751ed4b7388d889aac30a5f9b P 197b880fa39d432a9b472845876bfaf0: Append to log took a long time: real 0.129s user 0.000s sys 0.001s
I20260129 02:59:44.912828 20435 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:48356 (request call id 276) took 1374 ms. Trace:
I20260129 02:59:44.919005 20480 leader_election.cc:304] T 75b0b6e82ddc4f92b1fe1d94f9a0b73b P 974b802141bd4f13beffef2aa03f88f9 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 974b802141bd4f13beffef2aa03f88f9; no voters: 1c50de89a5a845668696dcb28c3badd6, 7a9daf5d42a84615a6383cfec9a10608
I20260129 02:59:44.875888 22435 consensus_queue.cc:1059] T a1a39dcd328e4268ae41b988c66b4a75 P 1c50de89a5a845668696dcb28c3badd6 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 }, Status: INVALID_TERM, Last received: 2.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.032s
I20260129 02:59:44.915772 22348 raft_consensus.cc:2764] T cb0b3f259ebc4310927338a8740e87b8 P 1c50de89a5a845668696dcb28c3badd6 [term 5 FOLLOWER]: Leader pre-election decision vote started in defunct term 4: won
I20260129 02:59:44.907785 22184 raft_consensus.cc:2804] T 45cc5f3d2688486d9b6dbe7e666dedde P 974b802141bd4f13beffef2aa03f88f9 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260129 02:59:44.892863 22586 raft_consensus.cc:515] T a609ac02fae84725ab743f95f51d33ce P 974b802141bd4f13beffef2aa03f88f9 [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } }
I20260129 02:59:44.925681 20441 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "17d6431cc77a491e8464d68db4081a51" candidate_uuid: "974b802141bd4f13beffef2aa03f88f9" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "197b880fa39d432a9b472845876bfaf0"
I20260129 02:59:44.919190 20435 rpcz_store.cc:276] 0129 02:59:43.538846 (+ 0us) service_pool.cc:168] Inserting onto call queue
0129 02:59:44.082091 (+543245us) service_pool.cc:225] Handling call
0129 02:59:44.117839 (+ 35748us) raft_consensus.cc:1471] Updating replica for 1 ops
0129 02:59:44.134954 (+ 17115us) raft_consensus.cc:1534] Early marking committed up to index 0
0129 02:59:44.134995 (+ 41us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0129 02:59:44.148490 (+ 13495us) log.cc:844] Serialized 26 byte log entry
0129 02:59:44.412811 (+264321us) raft_consensus.cc:1645] Marking committed up to 4
0129 02:59:44.412850 (+ 39us) raft_consensus.cc:1697] Filling consensus response to leader.
0129 02:59:44.412916 (+ 66us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0129 02:59:44.912589 (+499673us) raft_consensus.cc:1685] finished
0129 02:59:44.912612 (+ 23us) raft_consensus.cc:1691] UpdateReplicas() finished
0129 02:59:44.912765 (+ 153us) inbound_call.cc:173] Queueing success response
Metrics: {"thread_start_us":263932,"threads_started":1,"wal-append.queue_time_us":513839}
I20260129 02:59:44.953327 22320 raft_consensus.cc:515] T 1299723a4476420ab02fd8bfd7aa0645 P 1c50de89a5a845668696dcb28c3badd6 [term 7 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } }
I20260129 02:59:44.956702 22590 leader_election.cc:290] T a609ac02fae84725ab743f95f51d33ce P 7a9daf5d42a84615a6383cfec9a10608 [CANDIDATE]: Term 6 election: Requested vote from peers 197b880fa39d432a9b472845876bfaf0 (127.18.211.1:42027), 974b802141bd4f13beffef2aa03f88f9 (127.18.211.2:34517)
I20260129 02:59:44.957581 20435 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "76f713c3331540d2a5b76164103e9eb1" candidate_uuid: "974b802141bd4f13beffef2aa03f88f9" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "197b880fa39d432a9b472845876bfaf0" is_pre_election: true
I20260129 02:59:44.934057 20405 leader_election.cc:304] T c2633a6fc34c44579de348d208b0c01e P 197b880fa39d432a9b472845876bfaf0 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 197b880fa39d432a9b472845876bfaf0; no voters: 1c50de89a5a845668696dcb28c3badd6, 974b802141bd4f13beffef2aa03f88f9
I20260129 02:59:44.959257 20440 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55676 (request call id 1205) took 1656 ms. Trace:
I20260129 02:59:44.960754 20586 raft_consensus.cc:1240] T 1299723a4476420ab02fd8bfd7aa0645 P 1c50de89a5a845668696dcb28c3badd6 [term 7 FOLLOWER]: Rejecting Update request from peer 7a9daf5d42a84615a6383cfec9a10608 for earlier term 6. Current term is 7. Ops: []
I20260129 02:59:44.962357 20508 raft_consensus.cc:2393] T a609ac02fae84725ab743f95f51d33ce P 974b802141bd4f13beffef2aa03f88f9 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 197b880fa39d432a9b472845876bfaf0 in current term 6: Already voted for candidate 974b802141bd4f13beffef2aa03f88f9 in this term.
I20260129 02:59:44.923091 22184 raft_consensus.cc:493] T 45cc5f3d2688486d9b6dbe7e666dedde P 974b802141bd4f13beffef2aa03f88f9 [term 3 FOLLOWER]: Starting leader election (detected failure of leader 1c50de89a5a845668696dcb28c3badd6)
I20260129 02:59:44.959681 20440 rpcz_store.cc:276] 0129 02:59:43.302802 (+ 0us) service_pool.cc:168] Inserting onto call queue
0129 02:59:44.056928 (+754126us) service_pool.cc:225] Handling call
0129 02:59:44.068916 (+ 11988us) raft_consensus.cc:1471] Updating replica for 1 ops
0129 02:59:44.082790 (+ 13874us) raft_consensus.cc:1534] Early marking committed up to index 0
0129 02:59:44.082958 (+ 168us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0129 02:59:44.160384 (+ 77426us) log.cc:844] Serialized 26 byte log entry
0129 02:59:44.605791 (+445407us) raft_consensus.cc:1645] Marking committed up to 1
0129 02:59:44.605843 (+ 52us) raft_consensus.cc:1697] Filling consensus response to leader.
0129 02:59:44.605911 (+ 68us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0129 02:59:44.959015 (+353104us) raft_consensus.cc:1685] finished
0129 02:59:44.959043 (+ 28us) raft_consensus.cc:1691] UpdateReplicas() finished
0129 02:59:44.959195 (+ 152us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":13193,"raft.run_cpu_time_us":113,"raft.run_wall_time_us":114,"thread_start_us":445026,"threads_started":1,"wal-append.queue_time_us":604827}
W20260129 02:59:44.970202 20480 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.211.4:40439, user_credentials={real_user=slave}} blocked reactor thread for 51271us
I20260129 02:59:44.974768 22237 raft_consensus.cc:515] T 0f20af7fbe5d4db89a6f17948d98912c P 197b880fa39d432a9b472845876bfaf0 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:59:44.964432 22320 leader_election.cc:290] T 1299723a4476420ab02fd8bfd7aa0645 P 1c50de89a5a845668696dcb28c3badd6 [CANDIDATE]: Term 7 election: Requested vote from peers 197b880fa39d432a9b472845876bfaf0 (127.18.211.1:42027), 7a9daf5d42a84615a6383cfec9a10608 (127.18.211.4:40439)
I20260129 02:59:44.977870 22586 leader_election.cc:290] T a609ac02fae84725ab743f95f51d33ce P 974b802141bd4f13beffef2aa03f88f9 [CANDIDATE]: Term 6 election: Requested vote from peers 197b880fa39d432a9b472845876bfaf0 (127.18.211.1:42027), 7a9daf5d42a84615a6383cfec9a10608 (127.18.211.4:40439)
W20260129 02:59:44.984124 20554 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.211.4:40439, user_credentials={real_user=slave}} blocked reactor thread for 78092.3us
I20260129 02:59:44.919809 22493 raft_consensus.cc:2804] T cb4727386a3e48aca5d71cc6c611c197 P 974b802141bd4f13beffef2aa03f88f9 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260129 02:59:44.959175 22628 raft_consensus.cc:2749] T 75b0b6e82ddc4f92b1fe1d94f9a0b73b P 974b802141bd4f13beffef2aa03f88f9 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20260129 02:59:44.992583 20407 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.211.2:34517, user_credentials={real_user=slave}} blocked reactor thread for 132760us
I20260129 02:59:44.922976 22581 raft_consensus.cc:515] T e33a417cf9c54cf5af73a4ee3aac4d3a P 1c50de89a5a845668696dcb28c3badd6 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
W20260129 02:59:44.994776 20556 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.211.2:34517, user_credentials={real_user=slave}} blocked reactor thread for 94793.2us
I20260129 02:59:44.982170 22184 raft_consensus.cc:3060] T 45cc5f3d2688486d9b6dbe7e666dedde P 974b802141bd4f13beffef2aa03f88f9 [term 3 FOLLOWER]: Advancing to term 4
I20260129 02:59:44.998342 20407 leader_election.cc:304] T 3ccac82a01a2445fa5b0ce6c0fcf2295 P 197b880fa39d432a9b472845876bfaf0 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 197b880fa39d432a9b472845876bfaf0, 974b802141bd4f13beffef2aa03f88f9; no voters:
I20260129 02:59:44.990825 22445 leader_election.cc:290] T 76f713c3331540d2a5b76164103e9eb1 P 974b802141bd4f13beffef2aa03f88f9 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 197b880fa39d432a9b472845876bfaf0 (127.18.211.1:42027), 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3:37735)
I20260129 02:59:44.994043 22493 raft_consensus.cc:493] T cb4727386a3e48aca5d71cc6c611c197 P 974b802141bd4f13beffef2aa03f88f9 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260129 02:59:45.009114 20556 leader_election.cc:304] T cb4727386a3e48aca5d71cc6c611c197 P 1c50de89a5a845668696dcb28c3badd6 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1c50de89a5a845668696dcb28c3badd6, 974b802141bd4f13beffef2aa03f88f9; no voters: 197b880fa39d432a9b472845876bfaf0
W20260129 02:59:45.011132 20553 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.211.1:42027, user_credentials={real_user=slave}} blocked reactor thread for 50337.7us
W20260129 02:59:45.013760 20405 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.211.3:37735, user_credentials={real_user=slave}} blocked reactor thread for 79821.1us
I20260129 02:59:45.014571 20405 leader_election.cc:304] T cedce331e1584ac8b636c9b8a72b2ba0 P 197b880fa39d432a9b472845876bfaf0 [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 197b880fa39d432a9b472845876bfaf0; no voters: 1c50de89a5a845668696dcb28c3badd6, 974b802141bd4f13beffef2aa03f88f9
I20260129 02:59:45.017701 22641 raft_consensus.cc:2749] T c2633a6fc34c44579de348d208b0c01e P 197b880fa39d432a9b472845876bfaf0 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260129 02:59:45.000047 22566 leader_election.cc:290] T 0340229d521b432580ae926f6062e3f7 P 974b802141bd4f13beffef2aa03f88f9 [CANDIDATE]: Term 4 election: Requested vote from peers 197b880fa39d432a9b472845876bfaf0 (127.18.211.1:42027), 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3:37735)
I20260129 02:59:45.018154 22554 raft_consensus.cc:493] T 1ce4ddf5b10e4ae8b193b13b2e5936ee P 1c50de89a5a845668696dcb28c3badd6 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260129 02:59:45.012053 22437 leader_election.cc:290] T 0678b8b468bd43cdbc3e485900844821 P 974b802141bd4f13beffef2aa03f88f9 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 197b880fa39d432a9b472845876bfaf0 (127.18.211.1:42027), 7a9daf5d42a84615a6383cfec9a10608 (127.18.211.4:40439)
I20260129 02:59:45.018738 22554 raft_consensus.cc:515] T 1ce4ddf5b10e4ae8b193b13b2e5936ee P 1c50de89a5a845668696dcb28c3badd6 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:59:45.020942 22554 leader_election.cc:290] T 1ce4ddf5b10e4ae8b193b13b2e5936ee P 1c50de89a5a845668696dcb28c3badd6 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 197b880fa39d432a9b472845876bfaf0 (127.18.211.1:42027), 974b802141bd4f13beffef2aa03f88f9 (127.18.211.2:34517)
I20260129 02:59:45.023124 19276 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 5
I20260129 02:59:45.023715 21135 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00e79d8f25a34676b1d866eb2068a2b2" candidate_uuid: "974b802141bd4f13beffef2aa03f88f9" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "7a9daf5d42a84615a6383cfec9a10608" is_pre_election: true
I20260129 02:59:45.026044 21135 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "96609ac68bdf4993a12e1c23355ca013" candidate_uuid: "974b802141bd4f13beffef2aa03f88f9" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "7a9daf5d42a84615a6383cfec9a10608" is_pre_election: true
I20260129 02:59:45.026314 19276 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20260129 02:59:45.026762 21135 raft_consensus.cc:2468] T 96609ac68bdf4993a12e1c23355ca013 P 7a9daf5d42a84615a6383cfec9a10608 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 974b802141bd4f13beffef2aa03f88f9 in term 4.
W20260129 02:59:45.022351 20553 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260129 02:59:45.027773 20582 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "96609ac68bdf4993a12e1c23355ca013" candidate_uuid: "974b802141bd4f13beffef2aa03f88f9" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "1c50de89a5a845668696dcb28c3badd6" is_pre_election: true
I20260129 02:59:45.029270 20582 raft_consensus.cc:2468] T 96609ac68bdf4993a12e1c23355ca013 P 1c50de89a5a845668696dcb28c3badd6 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 974b802141bd4f13beffef2aa03f88f9 in term 4.
I20260129 02:59:45.004069 22633 raft_consensus.cc:2804] T 75be4bd6bd45474795fd472eff2953ac P 197b880fa39d432a9b472845876bfaf0 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260129 02:59:45.030831 20438 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "18f30ebaa4174fd28c958e72d0cc7e24" candidate_uuid: "1c50de89a5a845668696dcb28c3badd6" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "197b880fa39d432a9b472845876bfaf0"
I20260129 02:59:45.040671 20438 raft_consensus.cc:3060] T 18f30ebaa4174fd28c958e72d0cc7e24 P 197b880fa39d432a9b472845876bfaf0 [term 2 FOLLOWER]: Advancing to term 4
I20260129 02:59:45.045334 20440 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a609ac02fae84725ab743f95f51d33ce" candidate_uuid: "974b802141bd4f13beffef2aa03f88f9" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "197b880fa39d432a9b472845876bfaf0"
I20260129 02:59:45.046099 20440 raft_consensus.cc:3060] T a609ac02fae84725ab743f95f51d33ce P 197b880fa39d432a9b472845876bfaf0 [term 5 FOLLOWER]: Advancing to term 6
I20260129 02:59:45.047147 20435 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "17d6431cc77a491e8464d68db4081a51" candidate_uuid: "1c50de89a5a845668696dcb28c3badd6" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "197b880fa39d432a9b472845876bfaf0"
I20260129 02:59:45.048543 20439 raft_consensus.cc:2410] T 724a236524cb4e7ba3428880681d6cc2 P 197b880fa39d432a9b472845876bfaf0 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 974b802141bd4f13beffef2aa03f88f9 for term 3 because replica has last-logged OpId of term: 2 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 1.
I20260129 02:59:45.029322 22493 raft_consensus.cc:3060] T cb4727386a3e48aca5d71cc6c611c197 P 974b802141bd4f13beffef2aa03f88f9 [term 2 FOLLOWER]: Advancing to term 3
I20260129 02:59:45.052940 20434 raft_consensus.cc:1240] T 724a236524cb4e7ba3428880681d6cc2 P 197b880fa39d432a9b472845876bfaf0 [term 3 FOLLOWER]: Rejecting Update request from peer 1c50de89a5a845668696dcb28c3badd6 for earlier term 2. Current term is 3. Ops: []
I20260129 02:59:45.054847 21135 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1299723a4476420ab02fd8bfd7aa0645" candidate_uuid: "974b802141bd4f13beffef2aa03f88f9" candidate_term: 5 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "7a9daf5d42a84615a6383cfec9a10608" is_pre_election: true
I20260129 02:59:45.037176 21130 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1299723a4476420ab02fd8bfd7aa0645" candidate_uuid: "1c50de89a5a845668696dcb28c3badd6" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "7a9daf5d42a84615a6383cfec9a10608"
I20260129 02:59:45.058947 20439 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0678b8b468bd43cdbc3e485900844821" candidate_uuid: "974b802141bd4f13beffef2aa03f88f9" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "197b880fa39d432a9b472845876bfaf0" is_pre_election: true
I20260129 02:59:45.038260 22633 raft_consensus.cc:493] T 75be4bd6bd45474795fd472eff2953ac P 197b880fa39d432a9b472845876bfaf0 [term 7 FOLLOWER]: Starting leader election (detected failure of leader 1c50de89a5a845668696dcb28c3badd6)
I20260129 02:59:45.063225 20435 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1299723a4476420ab02fd8bfd7aa0645" candidate_uuid: "1c50de89a5a845668696dcb28c3badd6" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "197b880fa39d432a9b472845876bfaf0"
I20260129 02:59:45.066875 20585 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "add8d4923d1a489e86882c3a25692b95" candidate_uuid: "974b802141bd4f13beffef2aa03f88f9" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "1c50de89a5a845668696dcb28c3badd6" is_pre_election: true
W20260129 02:59:45.072253 21099 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260129 02:59:45.073940 20439 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0340229d521b432580ae926f6062e3f7" candidate_uuid: "1c50de89a5a845668696dcb28c3badd6" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "197b880fa39d432a9b472845876bfaf0"
I20260129 02:59:45.061686 22636 raft_consensus.cc:2749] T cedce331e1584ac8b636c9b8a72b2ba0 P 197b880fa39d432a9b472845876bfaf0 [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20260129 02:59:45.074618 20439 raft_consensus.cc:3060] T 0340229d521b432580ae926f6062e3f7 P 197b880fa39d432a9b472845876bfaf0 [term 4 FOLLOWER]: Advancing to term 5
W20260129 02:59:45.074836 22629 log.cc:927] Time spent T fd8311b779af4e1fa1402b2a02a602f2 P 1c50de89a5a845668696dcb28c3badd6: Append to log took a long time: real 0.163s user 0.001s sys 0.001s
I20260129 02:59:45.068006 22633 raft_consensus.cc:3060] T 75be4bd6bd45474795fd472eff2953ac P 197b880fa39d432a9b472845876bfaf0 [term 7 FOLLOWER]: Advancing to term 8
I20260129 02:59:45.075719 22320 raft_consensus.cc:493] T c891d2bff4ce4568861bbc992e91fbb0 P 1c50de89a5a845668696dcb28c3badd6 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 974b802141bd4f13beffef2aa03f88f9)
I20260129 02:59:45.076293 22320 raft_consensus.cc:515] T c891d2bff4ce4568861bbc992e91fbb0 P 1c50de89a5a845668696dcb28c3badd6 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:59:45.077577 20434 raft_consensus.cc:1240] T 3906b080a5b34a74b8b64128635e8997 P 197b880fa39d432a9b472845876bfaf0 [term 3 FOLLOWER]: Rejecting Update request from peer 1c50de89a5a845668696dcb28c3badd6 for earlier term 2. Current term is 3. Ops: []
I20260129 02:59:45.078220 22320 leader_election.cc:290] T c891d2bff4ce4568861bbc992e91fbb0 P 1c50de89a5a845668696dcb28c3badd6 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 197b880fa39d432a9b472845876bfaf0 (127.18.211.1:42027), 974b802141bd4f13beffef2aa03f88f9 (127.18.211.2:34517)
I20260129 02:59:45.078984 20434 raft_consensus.cc:1275] T dc2f4cb752cf4d59b5f6bc689f5d8104 P 197b880fa39d432a9b472845876bfaf0 [term 4 FOLLOWER]: Refusing update from remote peer 1c50de89a5a845668696dcb28c3badd6: Log matching property violated. Preceding OpId in replica: term: 3 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20260129 02:59:45.080456 20433 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e7d83d4ccf684e76acb29a0dc844e1b8" candidate_uuid: "1c50de89a5a845668696dcb28c3badd6" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "197b880fa39d432a9b472845876bfaf0"
W20260129 02:59:45.065562 20407 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.211.2:34517, user_credentials={real_user=slave}} blocked reactor thread for 67300.4us
I20260129 02:59:45.081039 20433 raft_consensus.cc:3060] T e7d83d4ccf684e76acb29a0dc844e1b8 P 197b880fa39d432a9b472845876bfaf0 [term 3 FOLLOWER]: Advancing to term 4
I20260129 02:59:45.099277 21135 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2ac659d732314e1288849f08fe5e074e" candidate_uuid: "974b802141bd4f13beffef2aa03f88f9" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "7a9daf5d42a84615a6383cfec9a10608"
I20260129 02:59:45.100234 21135 raft_consensus.cc:3060] T 2ac659d732314e1288849f08fe5e074e P 7a9daf5d42a84615a6383cfec9a10608 [term 2 FOLLOWER]: Advancing to term 3
I20260129 02:59:45.104032 20590 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1299723a4476420ab02fd8bfd7aa0645" candidate_uuid: "974b802141bd4f13beffef2aa03f88f9" candidate_term: 5 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "1c50de89a5a845668696dcb28c3badd6" is_pre_election: true
I20260129 02:59:45.111702 20439 raft_consensus.cc:2468] T 0340229d521b432580ae926f6062e3f7 P 197b880fa39d432a9b472845876bfaf0 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1c50de89a5a845668696dcb28c3badd6 in term 5.
I20260129 02:59:45.120595 20437 raft_consensus.cc:2468] T 3ccac82a01a2445fa5b0ce6c0fcf2295 P 197b880fa39d432a9b472845876bfaf0 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 974b802141bd4f13beffef2aa03f88f9 in term 3.
I20260129 02:59:45.089643 21129 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "add8d4923d1a489e86882c3a25692b95" candidate_uuid: "974b802141bd4f13beffef2aa03f88f9" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "7a9daf5d42a84615a6383cfec9a10608" is_pre_election: true
I20260129 02:59:45.132447 20479 leader_election.cc:304] T 3ccac82a01a2445fa5b0ce6c0fcf2295 P 974b802141bd4f13beffef2aa03f88f9 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 197b880fa39d432a9b472845876bfaf0, 974b802141bd4f13beffef2aa03f88f9; no voters:
W20260129 02:59:45.136195 20405 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.211.3:37735, user_credentials={real_user=slave}} blocked reactor thread for 121741us
I20260129 02:59:45.130677 20407 leader_election.cc:304] T e7d83d4ccf684e76acb29a0dc844e1b8 P 197b880fa39d432a9b472845876bfaf0 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 197b880fa39d432a9b472845876bfaf0, 974b802141bd4f13beffef2aa03f88f9; no voters:
I20260129 02:59:45.140842 20407 leader_election.cc:304] T 913fc7d751ed4b7388d889aac30a5f9b P 197b880fa39d432a9b472845876bfaf0 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 197b880fa39d432a9b472845876bfaf0, 974b802141bd4f13beffef2aa03f88f9; no voters:
I20260129 02:59:45.141804 20407 leader_election.cc:304] T cb4727386a3e48aca5d71cc6c611c197 P 197b880fa39d432a9b472845876bfaf0 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 197b880fa39d432a9b472845876bfaf0, 974b802141bd4f13beffef2aa03f88f9; no voters:
I20260129 02:59:45.137050 20405 leader_election.cc:304] T a1a39dcd328e4268ae41b988c66b4a75 P 197b880fa39d432a9b472845876bfaf0 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 197b880fa39d432a9b472845876bfaf0; no voters: 1c50de89a5a845668696dcb28c3badd6, 974b802141bd4f13beffef2aa03f88f9
I20260129 02:59:45.149040 20590 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fcddab6bca684adf8ea032289a00f90e" candidate_uuid: "974b802141bd4f13beffef2aa03f88f9" candidate_term: 8 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "1c50de89a5a845668696dcb28c3badd6" is_pre_election: true
I20260129 02:59:45.149981 20590 raft_consensus.cc:2468] T fcddab6bca684adf8ea032289a00f90e P 1c50de89a5a845668696dcb28c3badd6 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 974b802141bd4f13beffef2aa03f88f9 in term 7.
I20260129 02:59:45.090595 22403 consensus_queue.cc:1059] T 1299723a4476420ab02fd8bfd7aa0645 P 7a9daf5d42a84615a6383cfec9a10608 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 }, Status: INVALID_TERM, Last received: 6.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.045s
I20260129 02:59:45.086539 22540 raft_consensus.cc:515] T 17d6431cc77a491e8464d68db4081a51 P 197b880fa39d432a9b472845876bfaf0 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:59:45.110788 22640 raft_consensus.cc:2764] T cb4727386a3e48aca5d71cc6c611c197 P 1c50de89a5a845668696dcb28c3badd6 [term 2 FOLLOWER]: Leader pre-election decision vote started in defunct term 1: won
I20260129 02:59:45.121378 22641 raft_consensus.cc:2764] T 3ccac82a01a2445fa5b0ce6c0fcf2295 P 197b880fa39d432a9b472845876bfaf0 [term 3 FOLLOWER]: Leader pre-election decision vote started in defunct term 2: won
I20260129 02:59:45.161324 20585 raft_consensus.cc:1240] T e33a417cf9c54cf5af73a4ee3aac4d3a P 1c50de89a5a845668696dcb28c3badd6 [term 5 FOLLOWER]: Rejecting Update request from peer 974b802141bd4f13beffef2aa03f88f9 for earlier term 4. Current term is 5. Ops: []
I20260129 02:59:45.142247 22566 raft_consensus.cc:2804] T 3ccac82a01a2445fa5b0ce6c0fcf2295 P 974b802141bd4f13beffef2aa03f88f9 [term 3 FOLLOWER]: Leader election won for term 3
I20260129 02:59:45.165405 20586 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b189f08e16a74a11980cc55d99359a0a" candidate_uuid: "974b802141bd4f13beffef2aa03f88f9" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "1c50de89a5a845668696dcb28c3badd6"
I20260129 02:59:45.173221 20582 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3e2d15f8640b407195210c476e09df91" candidate_uuid: "974b802141bd4f13beffef2aa03f88f9" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "1c50de89a5a845668696dcb28c3badd6" is_pre_election: true
I20260129 02:59:45.180299 20441 raft_consensus.cc:2393] T 17d6431cc77a491e8464d68db4081a51 P 197b880fa39d432a9b472845876bfaf0 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 974b802141bd4f13beffef2aa03f88f9 in current term 3: Already voted for candidate 197b880fa39d432a9b472845876bfaf0 in this term.
I20260129 02:59:45.165159 21136 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "cb0b3f259ebc4310927338a8740e87b8" candidate_uuid: "974b802141bd4f13beffef2aa03f88f9" candidate_term: 6 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "7a9daf5d42a84615a6383cfec9a10608" is_pre_election: true
I20260129 02:59:45.186218 21131 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3ccac82a01a2445fa5b0ce6c0fcf2295" candidate_uuid: "974b802141bd4f13beffef2aa03f88f9" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "7a9daf5d42a84615a6383cfec9a10608"
I20260129 02:59:45.159582 22635 raft_consensus.cc:2804] T 913fc7d751ed4b7388d889aac30a5f9b P 197b880fa39d432a9b472845876bfaf0 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260129 02:59:45.163717 22590 raft_consensus.cc:3055] T 1299723a4476420ab02fd8bfd7aa0645 P 7a9daf5d42a84615a6383cfec9a10608 [term 6 LEADER]: Stepping down as leader of term 6
I20260129 02:59:45.160524 21130 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "febffe4a8b394368b69b1d164308fb9a" candidate_uuid: "974b802141bd4f13beffef2aa03f88f9" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "7a9daf5d42a84615a6383cfec9a10608" is_pre_election: true
W20260129 02:59:45.186965 20407 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.211.2:34517, user_credentials={real_user=slave}} blocked reactor thread for 45268.2us
I20260129 02:59:45.166189 22581 leader_election.cc:290] T e33a417cf9c54cf5af73a4ee3aac4d3a P 1c50de89a5a845668696dcb28c3badd6 [CANDIDATE]: Term 5 election: Requested vote from peers 197b880fa39d432a9b472845876bfaf0 (127.18.211.1:42027), 974b802141bd4f13beffef2aa03f88f9 (127.18.211.2:34517)
I20260129 02:59:45.194993 21131 raft_consensus.cc:3060] T 3ccac82a01a2445fa5b0ce6c0fcf2295 P 7a9daf5d42a84615a6383cfec9a10608 [term 1 FOLLOWER]: Advancing to term 3
I20260129 02:59:45.165246 22636 raft_consensus.cc:2749] T a1a39dcd328e4268ae41b988c66b4a75 P 197b880fa39d432a9b472845876bfaf0 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260129 02:59:45.163893 22237 leader_election.cc:290] T 0f20af7fbe5d4db89a6f17948d98912c P 197b880fa39d432a9b472845876bfaf0 [CANDIDATE]: Term 5 election: Requested vote from peers 974b802141bd4f13beffef2aa03f88f9 (127.18.211.2:34517), 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3:37735)
I20260129 02:59:45.216490 20407 leader_election.cc:304] T 7a9a69cd82724b1faed1d10aa091fc4a P 197b880fa39d432a9b472845876bfaf0 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 197b880fa39d432a9b472845876bfaf0, 974b802141bd4f13beffef2aa03f88f9; no voters:
I20260129 02:59:45.218727 21136 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a609ac02fae84725ab743f95f51d33ce" candidate_uuid: "974b802141bd4f13beffef2aa03f88f9" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "7a9daf5d42a84615a6383cfec9a10608"
I20260129 02:59:45.204079 21134 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a644c02b16af47679824ccd060ac10fd" candidate_uuid: "974b802141bd4f13beffef2aa03f88f9" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "7a9daf5d42a84615a6383cfec9a10608" is_pre_election: true
I20260129 02:59:45.219635 21136 raft_consensus.cc:2393] T a609ac02fae84725ab743f95f51d33ce P 7a9daf5d42a84615a6383cfec9a10608 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 974b802141bd4f13beffef2aa03f88f9 in current term 6: Already voted for candidate 7a9daf5d42a84615a6383cfec9a10608 in this term.
I20260129 02:59:45.217463 20585 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "cb4727386a3e48aca5d71cc6c611c197" candidate_uuid: "974b802141bd4f13beffef2aa03f88f9" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1c50de89a5a845668696dcb28c3badd6" is_pre_election: true
I20260129 02:59:45.218528 20584 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "cb0b3f259ebc4310927338a8740e87b8" candidate_uuid: "974b802141bd4f13beffef2aa03f88f9" candidate_term: 6 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "1c50de89a5a845668696dcb28c3badd6" is_pre_election: true
I20260129 02:59:45.225262 21131 raft_consensus.cc:2468] T 3ccac82a01a2445fa5b0ce6c0fcf2295 P 7a9daf5d42a84615a6383cfec9a10608 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 974b802141bd4f13beffef2aa03f88f9 in term 3.
I20260129 02:59:45.227607 21135 raft_consensus.cc:2468] T 2ac659d732314e1288849f08fe5e074e P 7a9daf5d42a84615a6383cfec9a10608 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 974b802141bd4f13beffef2aa03f88f9 in term 3.
W20260129 02:59:45.204910 21099 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.211.1:42027, user_credentials={real_user=slave}} blocked reactor thread for 106722us
I20260129 02:59:45.189916 22566 raft_consensus.cc:697] T 3ccac82a01a2445fa5b0ce6c0fcf2295 P 974b802141bd4f13beffef2aa03f88f9 [term 3 LEADER]: Becoming Leader. State: Replica: 974b802141bd4f13beffef2aa03f88f9, State: Running, Role: LEADER
I20260129 02:59:45.229668 20405 leader_election.cc:304] T 4408006eb5174de1896b037cf3f18457 P 197b880fa39d432a9b472845876bfaf0 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 197b880fa39d432a9b472845876bfaf0, 1c50de89a5a845668696dcb28c3badd6; no voters: 974b802141bd4f13beffef2aa03f88f9
W20260129 02:59:45.235934 20406 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260129 02:59:45.235664 20586 raft_consensus.cc:2376] T b189f08e16a74a11980cc55d99359a0a P 1c50de89a5a845668696dcb28c3badd6 [term 5 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 974b802141bd4f13beffef2aa03f88f9 in term 5. Re-sending same reply.
I20260129 02:59:45.193262 22635 raft_consensus.cc:493] T 913fc7d751ed4b7388d889aac30a5f9b P 197b880fa39d432a9b472845876bfaf0 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 1c50de89a5a845668696dcb28c3badd6)
I20260129 02:59:45.197393 22590 raft_consensus.cc:740] T 1299723a4476420ab02fd8bfd7aa0645 P 7a9daf5d42a84615a6383cfec9a10608 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 7a9daf5d42a84615a6383cfec9a10608, State: Running, Role: LEADER
I20260129 02:59:45.194769 22646 raft_consensus.cc:2804] T cb4727386a3e48aca5d71cc6c611c197 P 197b880fa39d432a9b472845876bfaf0 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260129 02:59:45.205272 21132 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3e2d15f8640b407195210c476e09df91" candidate_uuid: "974b802141bd4f13beffef2aa03f88f9" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "7a9daf5d42a84615a6383cfec9a10608" is_pre_election: true
W20260129 02:59:45.237349 21102 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.211.2:34517, user_credentials={real_user=slave}} blocked reactor thread for 70891.3us
I20260129 02:59:45.240967 20585 raft_consensus.cc:2468] T cb4727386a3e48aca5d71cc6c611c197 P 1c50de89a5a845668696dcb28c3badd6 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 974b802141bd4f13beffef2aa03f88f9 in term 2.
I20260129 02:59:45.235584 22197 raft_consensus.cc:493] T c2633a6fc34c44579de348d208b0c01e P 974b802141bd4f13beffef2aa03f88f9 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 1c50de89a5a845668696dcb28c3badd6)
I20260129 02:59:45.245427 22197 raft_consensus.cc:515] T c2633a6fc34c44579de348d208b0c01e P 974b802141bd4f13beffef2aa03f88f9 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
W20260129 02:59:45.246343 22639 log.cc:927] Time spent T 4b1876f9a70b49edb3aa74105e66d3b3 P 197b880fa39d432a9b472845876bfaf0: Append to log took a long time: real 0.112s user 0.001s sys 0.000s
I20260129 02:59:45.239776 22647 raft_consensus.cc:2804] T 7a9a69cd82724b1faed1d10aa091fc4a P 197b880fa39d432a9b472845876bfaf0 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260129 02:59:45.247483 22647 raft_consensus.cc:493] T 7a9a69cd82724b1faed1d10aa091fc4a P 197b880fa39d432a9b472845876bfaf0 [term 4 FOLLOWER]: Starting leader election (detected failure of leader 1c50de89a5a845668696dcb28c3badd6)
I20260129 02:59:45.247915 22647 raft_consensus.cc:3060] T 7a9a69cd82724b1faed1d10aa091fc4a P 197b880fa39d432a9b472845876bfaf0 [term 4 FOLLOWER]: Advancing to term 5
I20260129 02:59:45.248116 22637 raft_consensus.cc:493] T 368953c9102643b2a21f99af36b51905 P 1c50de89a5a845668696dcb28c3badd6 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260129 02:59:45.248200 20508 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4f5a77e94fef4b16a2ba7b067c8b144a" candidate_uuid: "7a9daf5d42a84615a6383cfec9a10608" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "974b802141bd4f13beffef2aa03f88f9" is_pre_election: true
I20260129 02:59:45.248415 20437 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c2633a6fc34c44579de348d208b0c01e" candidate_uuid: "974b802141bd4f13beffef2aa03f88f9" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "197b880fa39d432a9b472845876bfaf0" is_pre_election: true
I20260129 02:59:45.248898 20508 raft_consensus.cc:2468] T 4f5a77e94fef4b16a2ba7b067c8b144a P 974b802141bd4f13beffef2aa03f88f9 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7a9daf5d42a84615a6383cfec9a10608 in term 3.
I20260129 02:59:45.248785 22637 raft_consensus.cc:515] T 368953c9102643b2a21f99af36b51905 P 1c50de89a5a845668696dcb28c3badd6 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:59:45.229321 22445 raft_consensus.cc:493] T a4c1226b987b468fa84dfc6ba0e2dbe1 P 974b802141bd4f13beffef2aa03f88f9 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 1c50de89a5a845668696dcb28c3badd6)
I20260129 02:59:45.250018 22445 raft_consensus.cc:515] T a4c1226b987b468fa84dfc6ba0e2dbe1 P 974b802141bd4f13beffef2aa03f88f9 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:59:45.250960 22637 leader_election.cc:290] T 368953c9102643b2a21f99af36b51905 P 1c50de89a5a845668696dcb28c3badd6 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 197b880fa39d432a9b472845876bfaf0 (127.18.211.1:42027), 974b802141bd4f13beffef2aa03f88f9 (127.18.211.2:34517)
W20260129 02:59:45.252008 22625 log.cc:927] Time spent T 4f5a77e94fef4b16a2ba7b067c8b144a P 7a9daf5d42a84615a6383cfec9a10608: Append to log took a long time: real 0.139s user 0.000s sys 0.001s
W20260129 02:59:45.254536 22648 log.cc:927] Time spent T 00000000000000000000000000000000 P a91e6696d0a44ee19bcf2489fd709c38: Append to log took a long time: real 0.091s user 0.000s sys 0.000s
I20260129 02:59:45.256219 21128 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34046 (request call id 442) took 1132 ms. Trace:
I20260129 02:59:45.256624 21128 rpcz_store.cc:276] 0129 02:59:44.124339 (+ 0us) service_pool.cc:168] Inserting onto call queue
0129 02:59:44.141758 (+ 17419us) service_pool.cc:225] Handling call
0129 02:59:44.142179 (+ 421us) raft_consensus.cc:1471] Updating replica for 1 ops
0129 02:59:44.142628 (+ 449us) raft_consensus.cc:1534] Early marking committed up to index 0
0129 02:59:44.142667 (+ 39us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0129 02:59:44.143277 (+ 610us) log.cc:844] Serialized 26 byte log entry
0129 02:59:44.416190 (+272913us) spinlock_profiling.cc:243] Waited 5.78 ms on lock 0x7b440002304c. stack: 00007f8fe044e373 00007f8fe044e270 00007f8fdfda0574 00007f8fe9a5c6b4 00007f8fe9a5c649 00007f8fe02c08b1 00007f8fe038f005 00007f8fe038fa22 00007f8fe30d957b 00007f8fe30d4e4f 00007f8fe0462f0c 00007f8fe924b709 00007f8fe047f57b 00007f8fe047d81f 00007f8fe047fae2 00007f8fe30c75ab
0129 02:59:44.871006 (+454816us) raft_consensus.cc:1645] Marking committed up to 4
0129 02:59:44.872654 (+ 1648us) raft_consensus.cc:1697] Filling consensus response to leader.
0129 02:59:44.884773 (+ 12119us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0129 02:59:44.998283 (+113510us) spinlock_profiling.cc:243] Waited 2.05 ms on lock 0x7b1800387ac0. stack: 00007f8fe044e373 00007f8fe044e270 00007f8fdfda0574 00007f8fe9a5c6b4 00007f8fe9a5c649 00007f8fe92d4c91 00007f8fe92e1671 00007f8fe04802d3 00007f8fe04826a1 00007f8fe0482659 00007f8fe04825e9 00007f8fe04825b1 00007f8fe04818ad 0000000000379254 0000000000379099 00007f8fe046369a
0129 02:59:45.255943 (+257660us) raft_consensus.cc:1685] finished
0129 02:59:45.255984 (+ 41us) raft_consensus.cc:1691] UpdateReplicas() finished
0129 02:59:45.256128 (+ 144us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":1461,"raft.run_cpu_time_us":148,"raft.run_wall_time_us":148,"spinlock_wait_cycles":18010752,"thread_start_us":717007,"threads_started":1,"wal-append.queue_time_us":718950}
I20260129 02:59:45.240689 22635 raft_consensus.cc:3060] T 913fc7d751ed4b7388d889aac30a5f9b P 197b880fa39d432a9b472845876bfaf0 [term 2 FOLLOWER]: Advancing to term 3
W20260129 02:59:45.260735 22400 consensus_meta.cc:287] T 3906b080a5b34a74b8b64128635e8997 P 974b802141bd4f13beffef2aa03f88f9: Time spent flushing consensus metadata: real 0.681s user 0.012s sys 0.000s
I20260129 02:59:45.261147 22400 raft_consensus.cc:515] T 3906b080a5b34a74b8b64128635e8997 P 974b802141bd4f13beffef2aa03f88f9 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:59:45.263991 20437 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3906b080a5b34a74b8b64128635e8997" candidate_uuid: "974b802141bd4f13beffef2aa03f88f9" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "197b880fa39d432a9b472845876bfaf0"
I20260129 02:59:45.267266 20586 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7d9494f5584d4d5db8b9adf1c8fc888b" candidate_uuid: "974b802141bd4f13beffef2aa03f88f9" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "1c50de89a5a845668696dcb28c3badd6" is_pre_election: true
I20260129 02:59:45.269279 20585 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "8055232cf0004d40a85f3bf983fe00a9" candidate_uuid: "974b802141bd4f13beffef2aa03f88f9" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1c50de89a5a845668696dcb28c3badd6"
I20260129 02:59:45.269912 20585 raft_consensus.cc:3060] T 8055232cf0004d40a85f3bf983fe00a9 P 1c50de89a5a845668696dcb28c3badd6 [term 2 FOLLOWER]: Advancing to term 3
I20260129 02:59:45.230369 22586 raft_consensus.cc:493] T 926a250fa5834e88b96ccecbefbf5086 P 974b802141bd4f13beffef2aa03f88f9 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260129 02:59:45.242667 22646 raft_consensus.cc:493] T cb4727386a3e48aca5d71cc6c611c197 P 197b880fa39d432a9b472845876bfaf0 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260129 02:59:45.272863 22646 raft_consensus.cc:3060] T cb4727386a3e48aca5d71cc6c611c197 P 197b880fa39d432a9b472845876bfaf0 [term 2 FOLLOWER]: Advancing to term 3
I20260129 02:59:45.259563 20433 raft_consensus.cc:2468] T e7d83d4ccf684e76acb29a0dc844e1b8 P 197b880fa39d432a9b472845876bfaf0 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1c50de89a5a845668696dcb28c3badd6 in term 4.
I20260129 02:59:45.272295 20587 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "368953c9102643b2a21f99af36b51905" candidate_uuid: "974b802141bd4f13beffef2aa03f88f9" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "1c50de89a5a845668696dcb28c3badd6"
W20260129 02:59:45.258149 22650 log.cc:927] Time spent T a0fcc58d6fb6473cafbd9d34d5195269 P 197b880fa39d432a9b472845876bfaf0: Append to log took a long time: real 0.067s user 0.000s sys 0.000s
I20260129 02:59:45.285859 20438 raft_consensus.cc:2468] T 18f30ebaa4174fd28c958e72d0cc7e24 P 197b880fa39d432a9b472845876bfaf0 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1c50de89a5a845668696dcb28c3badd6 in term 4.
I20260129 02:59:45.271946 22630 raft_consensus.cc:493] T a1a39dcd328e4268ae41b988c66b4a75 P 974b802141bd4f13beffef2aa03f88f9 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 1c50de89a5a845668696dcb28c3badd6)
I20260129 02:59:45.288098 22636 consensus_queue.cc:1048] T fcddab6bca684adf8ea032289a00f90e P 197b880fa39d432a9b472845876bfaf0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
W20260129 02:59:45.289834 22627 log.cc:927] Time spent T c2633a6fc34c44579de348d208b0c01e P 197b880fa39d432a9b472845876bfaf0: Append to log took a long time: real 0.361s user 0.003s sys 0.002s
I20260129 02:59:45.282248 21132 raft_consensus.cc:2468] T 3e2d15f8640b407195210c476e09df91 P 7a9daf5d42a84615a6383cfec9a10608 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 974b802141bd4f13beffef2aa03f88f9 in term 1.
I20260129 02:59:45.292897 20581 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "724a236524cb4e7ba3428880681d6cc2" candidate_uuid: "974b802141bd4f13beffef2aa03f88f9" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1c50de89a5a845668696dcb28c3badd6"
I20260129 02:59:45.287716 22237 raft_consensus.cc:2804] T 4408006eb5174de1896b037cf3f18457 P 197b880fa39d432a9b472845876bfaf0 [term 2 FOLLOWER]: Leader election won for term 2
I20260129 02:59:45.287945 22630 raft_consensus.cc:515] T a1a39dcd328e4268ae41b988c66b4a75 P 974b802141bd4f13beffef2aa03f88f9 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } } peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } }
I20260129 02:59:45.303347 20584 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "75be4bd6bd45474795fd472eff2953ac" candidate_uuid: "974b802141bd4f13beffef2aa03f88f9" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "1c50de89a5a845668696dcb28c3badd6" is_pre_election: true
I20260129 02:59:45.305147 20586 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e4d18c35b0a7493e85d0a4b92956a48f" candidate_uuid: "974b802141bd4f13beffef2aa03f88f9" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "1c50de89a5a845668696dcb28c3badd6"
I20260129 02:59:45.247610 22197 leader_election.cc:290] T c2633a6fc34c44579de348d208b0c01e P 974b802141bd4f13beffef2aa03f88f9 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 197b880fa39d432a9b472845876bfaf0 (127.18.211.1:42027), 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3:37735)
I20260129 02:59:45.298336 20582 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "cedce331e1584ac8b636c9b8a72b2ba0" candidate_uuid: "974b802141bd4f13beffef2aa03f88f9" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "1c50de89a5a845668696dcb28c3badd6" is_pre_election: true
I20260129 02:59:45.338269 20582 raft_consensus.cc:2468] T cedce331e1584ac8b636c9b8a72b2ba0 P 1c50de89a5a845668696dcb28c3badd6 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 974b802141bd4f13beffef2aa03f88f9 in term 5.
I20260129 02:59:45.297904 20440 raft_consensus.cc:2468] T a609ac02fae84725ab743f95f51d33ce P 197b880fa39d432a9b472845876bfaf0 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 974b802141bd4f13beffef2aa03f88f9 in term 6.
W20260129 02:59:45.312358 20404 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260129 02:59:45.340965 20479 leader_election.cc:304] T a609ac02fae84725ab743f95f51d33ce P 974b802141bd4f13beffef2aa03f88f9 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 197b880fa39d432a9b472845876bfaf0, 974b802141bd4f13beffef2aa03f88f9; no voters:
I20260129 02:59:45.296484 22540 leader_election.cc:290] T 17d6431cc77a491e8464d68db4081a51 P 197b880fa39d432a9b472845876bfaf0 [CANDIDATE]: Term 3 election: Requested vote from peers 974b802141bd4f13beffef2aa03f88f9 (127.18.211.2:34517), 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3:37735)
I20260129 02:59:45.346658 20515 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2ac659d732314e1288849f08fe5e074e" candidate_uuid: "7a9daf5d42a84615a6383cfec9a10608" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "974b802141bd4f13beffef2aa03f88f9" is_pre_election: true
I20260129 02:59:45.347649 20507 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3ccac82a01a2445fa5b0ce6c0fcf2295" candidate_uuid: "7a9daf5d42a84615a6383cfec9a10608" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "974b802141bd4f13beffef2aa03f88f9" is_pre_election: true
I20260129 02:59:45.349814 20440 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a4c1226b987b468fa84dfc6ba0e2dbe1" candidate_uuid: "974b802141bd4f13beffef2aa03f88f9" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "197b880fa39d432a9b472845876bfaf0" is_pre_election: true
I20260129 02:59:45.350910 20515 raft_consensus.cc:2393] T 2ac659d732314e1288849f08fe5e074e P 974b802141bd4f13beffef2aa03f88f9 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 7a9daf5d42a84615a6383cfec9a10608 in current term 3: Already voted for candidate 974b802141bd4f13beffef2aa03f88f9 in this term.
I20260129 02:59:45.272289 22586 raft_consensus.cc:515] T 926a250fa5834e88b96ccecbefbf5086 P 974b802141bd4f13beffef2aa03f88f9 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
W20260129 02:59:45.351666 20405 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.211.3:37735, user_credentials={real_user=slave}} blocked reactor thread for 122139us
I20260129 02:59:45.355360 20432 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1ce4ddf5b10e4ae8b193b13b2e5936ee" candidate_uuid: "1c50de89a5a845668696dcb28c3badd6" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "197b880fa39d432a9b472845876bfaf0" is_pre_election: true
I20260129 02:59:45.356076 20432 raft_consensus.cc:2468] T 1ce4ddf5b10e4ae8b193b13b2e5936ee P 197b880fa39d432a9b472845876bfaf0 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1c50de89a5a845668696dcb28c3badd6 in term 5.
I20260129 02:59:45.356853 22348 raft_consensus.cc:3055] T a1a39dcd328e4268ae41b988c66b4a75 P 1c50de89a5a845668696dcb28c3badd6 [term 2 LEADER]: Stepping down as leader of term 2
I20260129 02:59:45.357259 22348 raft_consensus.cc:740] T a1a39dcd328e4268ae41b988c66b4a75 P 1c50de89a5a845668696dcb28c3badd6 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 1c50de89a5a845668696dcb28c3badd6, State: Running, Role: LEADER
I20260129 02:59:45.358052 22348 consensus_queue.cc:260] T a1a39dcd328e4268ae41b988c66b4a75 P 1c50de89a5a845668696dcb28c3badd6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } } peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } }
I20260129 02:59:45.358434 22644 consensus_queue.cc:1048] T 0678b8b468bd43cdbc3e485900844821 P 197b880fa39d432a9b472845876bfaf0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.012s
I20260129 02:59:45.359355 22348 raft_consensus.cc:3060] T a1a39dcd328e4268ae41b988c66b4a75 P 1c50de89a5a845668696dcb28c3badd6 [term 2 FOLLOWER]: Advancing to term 3
I20260129 02:59:45.359932 22644 consensus_queue.cc:1059] T 8055232cf0004d40a85f3bf983fe00a9 P 197b880fa39d432a9b472845876bfaf0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260129 02:59:45.360456 22645 raft_consensus.cc:2764] T e7d83d4ccf684e76acb29a0dc844e1b8 P 197b880fa39d432a9b472845876bfaf0 [term 4 FOLLOWER]: Leader pre-election decision vote started in defunct term 3: won
I20260129 02:59:45.361121 22630 leader_election.cc:290] T a1a39dcd328e4268ae41b988c66b4a75 P 974b802141bd4f13beffef2aa03f88f9 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3:37735), 197b880fa39d432a9b472845876bfaf0 (127.18.211.1:42027)
I20260129 02:59:45.361726 20436 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0340229d521b432580ae926f6062e3f7" candidate_uuid: "974b802141bd4f13beffef2aa03f88f9" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "197b880fa39d432a9b472845876bfaf0"
I20260129 02:59:45.362491 20436 raft_consensus.cc:2368] T 0340229d521b432580ae926f6062e3f7 P 197b880fa39d432a9b472845876bfaf0 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 974b802141bd4f13beffef2aa03f88f9 for earlier term 4. Current term is 5.
I20260129 02:59:45.345032 22633 raft_consensus.cc:515] T 75be4bd6bd45474795fd472eff2953ac P 197b880fa39d432a9b472845876bfaf0 [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:59:45.366031 19276 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260129 02:59:45.367364 22633 leader_election.cc:290] T 75be4bd6bd45474795fd472eff2953ac P 197b880fa39d432a9b472845876bfaf0 [CANDIDATE]: Term 8 election: Requested vote from peers 974b802141bd4f13beffef2aa03f88f9 (127.18.211.2:34517), 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3:37735)
I20260129 02:59:45.337113 22636 consensus_queue.cc:1048] T 926a250fa5834e88b96ccecbefbf5086 P 197b880fa39d432a9b472845876bfaf0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.031s
W20260129 02:59:45.368813 21102 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.211.2:34517, user_credentials={real_user=slave}} blocked reactor thread for 50447.6us
I20260129 02:59:45.369323 22636 raft_consensus.cc:3055] T 8055232cf0004d40a85f3bf983fe00a9 P 197b880fa39d432a9b472845876bfaf0 [term 2 LEADER]: Stepping down as leader of term 2
I20260129 02:59:45.369724 22636 raft_consensus.cc:740] T 8055232cf0004d40a85f3bf983fe00a9 P 197b880fa39d432a9b472845876bfaf0 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 197b880fa39d432a9b472845876bfaf0, State: Running, Role: LEADER
I20260129 02:59:45.370419 20586 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a8479d1fe0bd41e5946186e3a6ca8c7a" candidate_uuid: "974b802141bd4f13beffef2aa03f88f9" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "1c50de89a5a845668696dcb28c3badd6"
I20260129 02:59:45.370627 22636 consensus_queue.cc:260] T 8055232cf0004d40a85f3bf983fe00a9 P 197b880fa39d432a9b472845876bfaf0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:59:45.372021 20589 raft_consensus.cc:1275] T a0fcc58d6fb6473cafbd9d34d5195269 P 1c50de89a5a845668696dcb28c3badd6 [term 3 FOLLOWER]: Refusing update from remote peer 974b802141bd4f13beffef2aa03f88f9: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260129 02:59:45.377647 20433 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e33a417cf9c54cf5af73a4ee3aac4d3a" candidate_uuid: "1c50de89a5a845668696dcb28c3badd6" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "197b880fa39d432a9b472845876bfaf0"
I20260129 02:59:45.378427 20433 raft_consensus.cc:3060] T e33a417cf9c54cf5af73a4ee3aac4d3a P 197b880fa39d432a9b472845876bfaf0 [term 4 FOLLOWER]: Advancing to term 5
I20260129 02:59:45.379109 20585 raft_consensus.cc:2468] T 8055232cf0004d40a85f3bf983fe00a9 P 1c50de89a5a845668696dcb28c3badd6 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 974b802141bd4f13beffef2aa03f88f9 in term 3.
I20260129 02:59:45.382694 20435 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "368953c9102643b2a21f99af36b51905" candidate_uuid: "1c50de89a5a845668696dcb28c3badd6" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "197b880fa39d432a9b472845876bfaf0" is_pre_election: true
I20260129 02:59:45.383294 20435 raft_consensus.cc:2468] T 368953c9102643b2a21f99af36b51905 P 197b880fa39d432a9b472845876bfaf0 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1c50de89a5a845668696dcb28c3badd6 in term 5.
I20260129 02:59:45.388476 20479 leader_election.cc:400] T 0340229d521b432580ae926f6062e3f7 P 974b802141bd4f13beffef2aa03f88f9 [CANDIDATE]: Term 4 election: Vote denied by peer 197b880fa39d432a9b472845876bfaf0 (127.18.211.1:42027) with higher term. Message: Invalid argument: T 0340229d521b432580ae926f6062e3f7 P 197b880fa39d432a9b472845876bfaf0 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 974b802141bd4f13beffef2aa03f88f9 for earlier term 4. Current term is 5.
I20260129 02:59:45.388953 20479 leader_election.cc:403] T 0340229d521b432580ae926f6062e3f7 P 974b802141bd4f13beffef2aa03f88f9 [CANDIDATE]: Term 4 election: Cancelling election due to peer responding with higher term
I20260129 02:59:45.389382 20514 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c934d31efd3f4e5195605ddc354d14e1" candidate_uuid: "7a9daf5d42a84615a6383cfec9a10608" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "974b802141bd4f13beffef2aa03f88f9" is_pre_election: true
I20260129 02:59:45.346349 22197 raft_consensus.cc:2804] T a609ac02fae84725ab743f95f51d33ce P 974b802141bd4f13beffef2aa03f88f9 [term 6 FOLLOWER]: Leader election won for term 6
I20260129 02:59:45.400164 20440 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c891d2bff4ce4568861bbc992e91fbb0" candidate_uuid: "1c50de89a5a845668696dcb28c3badd6" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "197b880fa39d432a9b472845876bfaf0" is_pre_election: true
I20260129 02:59:45.409977 20514 raft_consensus.cc:2393] T c934d31efd3f4e5195605ddc354d14e1 P 974b802141bd4f13beffef2aa03f88f9 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 7a9daf5d42a84615a6383cfec9a10608 in current term 4: Already voted for candidate 974b802141bd4f13beffef2aa03f88f9 in this term.
I20260129 02:59:45.410440 20515 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "45cc5f3d2688486d9b6dbe7e666dedde" candidate_uuid: "7a9daf5d42a84615a6383cfec9a10608" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "974b802141bd4f13beffef2aa03f88f9" is_pre_election: true
I20260129 02:59:45.353798 22400 leader_election.cc:290] T 3906b080a5b34a74b8b64128635e8997 P 974b802141bd4f13beffef2aa03f88f9 [CANDIDATE]: Term 3 election: Requested vote from peers 197b880fa39d432a9b472845876bfaf0 (127.18.211.1:42027), 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3:37735)
I20260129 02:59:45.415378 20437 raft_consensus.cc:1240] T 75be4bd6bd45474795fd472eff2953ac P 197b880fa39d432a9b472845876bfaf0 [term 8 FOLLOWER]: Rejecting Update request from peer 1c50de89a5a845668696dcb28c3badd6 for earlier term 7. Current term is 8. Ops: []
I20260129 02:59:45.416383 20432 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a1a39dcd328e4268ae41b988c66b4a75" candidate_uuid: "974b802141bd4f13beffef2aa03f88f9" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "197b880fa39d432a9b472845876bfaf0" is_pre_election: true
I20260129 02:59:45.418076 20581 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "17d6431cc77a491e8464d68db4081a51" candidate_uuid: "974b802141bd4f13beffef2aa03f88f9" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1c50de89a5a845668696dcb28c3badd6"
I20260129 02:59:45.420562 20507 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a609ac02fae84725ab743f95f51d33ce" candidate_uuid: "7a9daf5d42a84615a6383cfec9a10608" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "974b802141bd4f13beffef2aa03f88f9" is_pre_election: true
I20260129 02:59:45.425710 20581 raft_consensus.cc:2393] T 17d6431cc77a491e8464d68db4081a51 P 1c50de89a5a845668696dcb28c3badd6 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 974b802141bd4f13beffef2aa03f88f9 in current term 3: Already voted for candidate 1c50de89a5a845668696dcb28c3badd6 in this term.
W20260129 02:59:45.425119 21100 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.211.3:37735, user_credentials={real_user=slave}} blocked reactor thread for 150370us
I20260129 02:59:45.425774 20582 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "913fc7d751ed4b7388d889aac30a5f9b" candidate_uuid: "974b802141bd4f13beffef2aa03f88f9" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1c50de89a5a845668696dcb28c3badd6"
I20260129 02:59:45.426648 20516 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4e420f90884f4796997b5ea41e49e411" candidate_uuid: "7a9daf5d42a84615a6383cfec9a10608" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "974b802141bd4f13beffef2aa03f88f9" is_pre_election: true
I20260129 02:59:45.344692 22231 consensus_queue.cc:1059] T 2ac659d732314e1288849f08fe5e074e P 197b880fa39d432a9b472845876bfaf0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.053s
I20260129 02:59:45.358069 22586 leader_election.cc:290] T 926a250fa5834e88b96ccecbefbf5086 P 974b802141bd4f13beffef2aa03f88f9 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 197b880fa39d432a9b472845876bfaf0 (127.18.211.1:42027), 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3:37735)
W20260129 02:59:45.404325 22611 log.cc:927] Time spent T fd8311b779af4e1fa1402b2a02a602f2 P 7a9daf5d42a84615a6383cfec9a10608: Append to log took a long time: real 0.055s user 0.000s sys 0.000s
I20260129 02:59:45.397328 20509 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a644c02b16af47679824ccd060ac10fd" candidate_uuid: "7a9daf5d42a84615a6383cfec9a10608" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "974b802141bd4f13beffef2aa03f88f9" is_pre_election: true
I20260129 02:59:45.409142 22630 raft_consensus.cc:3060] T 0340229d521b432580ae926f6062e3f7 P 974b802141bd4f13beffef2aa03f88f9 [term 4 FOLLOWER]: Advancing to term 5
I20260129 02:59:45.449602 20514 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3e2d15f8640b407195210c476e09df91" candidate_uuid: "7a9daf5d42a84615a6383cfec9a10608" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "974b802141bd4f13beffef2aa03f88f9" is_pre_election: true
I20260129 02:59:45.453001 20514 raft_consensus.cc:2468] T 3e2d15f8640b407195210c476e09df91 P 974b802141bd4f13beffef2aa03f88f9 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7a9daf5d42a84615a6383cfec9a10608 in term 1.
W20260129 02:59:45.455174 20407 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.211.2:34517, user_credentials={real_user=slave}} blocked reactor thread for 158705us
I20260129 02:59:45.409634 22445 leader_election.cc:290] T a4c1226b987b468fa84dfc6ba0e2dbe1 P 974b802141bd4f13beffef2aa03f88f9 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 197b880fa39d432a9b472845876bfaf0 (127.18.211.1:42027), 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3:37735)
W20260129 02:59:45.439203 22425 consensus_meta.cc:287] T a8479d1fe0bd41e5946186e3a6ca8c7a P 1c50de89a5a845668696dcb28c3badd6: Time spent flushing consensus metadata: real 0.542s user 0.007s sys 0.004s
I20260129 02:59:45.428256 20432 raft_consensus.cc:2393] T a1a39dcd328e4268ae41b988c66b4a75 P 197b880fa39d432a9b472845876bfaf0 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 974b802141bd4f13beffef2aa03f88f9 in current term 3: Already voted for candidate 197b880fa39d432a9b472845876bfaf0 in this term.
I20260129 02:59:45.469875 20433 raft_consensus.cc:2468] T e33a417cf9c54cf5af73a4ee3aac4d3a P 197b880fa39d432a9b472845876bfaf0 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1c50de89a5a845668696dcb28c3badd6 in term 5.
I20260129 02:59:45.349891 22590 consensus_queue.cc:260] T 1299723a4476420ab02fd8bfd7aa0645 P 7a9daf5d42a84615a6383cfec9a10608 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 6.7, Last appended by leader: 7, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } }
W20260129 02:59:45.477895 20556 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.211.2:34517, user_credentials={real_user=slave}} blocked reactor thread for 468841us
I20260129 02:59:45.482029 22647 raft_consensus.cc:515] T 7a9a69cd82724b1faed1d10aa091fc4a P 197b880fa39d432a9b472845876bfaf0 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:59:45.486469 20588 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a644c02b16af47679824ccd060ac10fd" candidate_uuid: "974b802141bd4f13beffef2aa03f88f9" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "1c50de89a5a845668696dcb28c3badd6" is_pre_election: true
I20260129 02:59:45.487669 22445 raft_consensus.cc:493] T 36ff33be666a4c82954731691dbcf327 P 974b802141bd4f13beffef2aa03f88f9 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 1c50de89a5a845668696dcb28c3badd6)
I20260129 02:59:45.488274 22445 raft_consensus.cc:515] T 36ff33be666a4c82954731691dbcf327 P 974b802141bd4f13beffef2aa03f88f9 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:59:45.490178 22445 leader_election.cc:290] T 36ff33be666a4c82954731691dbcf327 P 974b802141bd4f13beffef2aa03f88f9 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 197b880fa39d432a9b472845876bfaf0 (127.18.211.1:42027), 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3:37735)
I20260129 02:59:45.460263 22372 raft_consensus.cc:493] T 505580c93c694bbb8ea96fdf9b17ab46 P 7a9daf5d42a84615a6383cfec9a10608 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 1c50de89a5a845668696dcb28c3badd6)
I20260129 02:59:45.505257 20407 leader_election.cc:304] T 87d5fe48a44a408aaf03e894a67ecb2f P 197b880fa39d432a9b472845876bfaf0 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 197b880fa39d432a9b472845876bfaf0, 974b802141bd4f13beffef2aa03f88f9; no voters:
I20260129 02:59:45.505110 22372 raft_consensus.cc:515] T 505580c93c694bbb8ea96fdf9b17ab46 P 7a9daf5d42a84615a6383cfec9a10608 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } }
I20260129 02:59:45.506783 20407 leader_election.cc:304] T febffe4a8b394368b69b1d164308fb9a P 197b880fa39d432a9b472845876bfaf0 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 197b880fa39d432a9b472845876bfaf0, 974b802141bd4f13beffef2aa03f88f9; no voters:
I20260129 02:59:45.491745 20433 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "36ff33be666a4c82954731691dbcf327" candidate_uuid: "974b802141bd4f13beffef2aa03f88f9" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "197b880fa39d432a9b472845876bfaf0" is_pre_election: true
I20260129 02:59:45.451970 20436 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "926a250fa5834e88b96ccecbefbf5086" candidate_uuid: "974b802141bd4f13beffef2aa03f88f9" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "197b880fa39d432a9b472845876bfaf0" is_pre_election: true
I20260129 02:59:45.492991 20439 raft_consensus.cc:1240] T 7a9a69cd82724b1faed1d10aa091fc4a P 197b880fa39d432a9b472845876bfaf0 [term 5 FOLLOWER]: Rejecting Update request from peer 1c50de89a5a845668696dcb28c3badd6 for earlier term 4. Current term is 5. Ops: []
I20260129 02:59:45.456063 22586 raft_consensus.cc:493] T c934d31efd3f4e5195605ddc354d14e1 P 974b802141bd4f13beffef2aa03f88f9 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260129 02:59:45.507787 20407 leader_election.cc:304] T b189f08e16a74a11980cc55d99359a0a P 197b880fa39d432a9b472845876bfaf0 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 197b880fa39d432a9b472845876bfaf0, 974b802141bd4f13beffef2aa03f88f9; no voters:
I20260129 02:59:45.510103 22633 raft_consensus.cc:2804] T 87d5fe48a44a408aaf03e894a67ecb2f P 197b880fa39d432a9b472845876bfaf0 [term 5 FOLLOWER]: Leader election won for term 5
I20260129 02:59:45.476838 22197 raft_consensus.cc:697] T a609ac02fae84725ab743f95f51d33ce P 974b802141bd4f13beffef2aa03f88f9 [term 6 LEADER]: Becoming Leader. State: Replica: 974b802141bd4f13beffef2aa03f88f9, State: Running, Role: LEADER
I20260129 02:59:45.507597 22372 leader_election.cc:290] T 505580c93c694bbb8ea96fdf9b17ab46 P 7a9daf5d42a84615a6383cfec9a10608 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 974b802141bd4f13beffef2aa03f88f9 (127.18.211.2:34517), 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3:37735)
I20260129 02:59:45.405251 22566 consensus_queue.cc:237] T 3ccac82a01a2445fa5b0ce6c0fcf2295 P 974b802141bd4f13beffef2aa03f88f9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } }
I20260129 02:59:45.460328 20509 raft_consensus.cc:2468] T a644c02b16af47679824ccd060ac10fd P 974b802141bd4f13beffef2aa03f88f9 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7a9daf5d42a84615a6383cfec9a10608 in term 3.
I20260129 02:59:45.526163 20587 raft_consensus.cc:2468] T 368953c9102643b2a21f99af36b51905 P 1c50de89a5a845668696dcb28c3badd6 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 974b802141bd4f13beffef2aa03f88f9 in term 5.
I20260129 02:59:45.519959 20405 leader_election.cc:304] T 00e79d8f25a34676b1d866eb2068a2b2 P 197b880fa39d432a9b472845876bfaf0 [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 197b880fa39d432a9b472845876bfaf0; no voters: 7a9daf5d42a84615a6383cfec9a10608, 974b802141bd4f13beffef2aa03f88f9
I20260129 02:59:45.560883 20405 leader_election.cc:304] T a609ac02fae84725ab743f95f51d33ce P 197b880fa39d432a9b472845876bfaf0 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 197b880fa39d432a9b472845876bfaf0, 7a9daf5d42a84615a6383cfec9a10608; no voters:
I20260129 02:59:45.512948 22590 raft_consensus.cc:3060] T 1299723a4476420ab02fd8bfd7aa0645 P 7a9daf5d42a84615a6383cfec9a10608 [term 6 FOLLOWER]: Advancing to term 7
I20260129 02:59:45.478757 22343 raft_consensus.cc:493] T 4e3805ffa70242e8ade66da260dddf66 P 7a9daf5d42a84615a6383cfec9a10608 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 1c50de89a5a845668696dcb28c3badd6)
W20260129 02:59:45.591825 21101 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260129 02:59:45.544345 22231 raft_consensus.cc:2804] T febffe4a8b394368b69b1d164308fb9a P 197b880fa39d432a9b472845876bfaf0 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260129 02:59:45.555743 22237 raft_consensus.cc:697] T 4408006eb5174de1896b037cf3f18457 P 197b880fa39d432a9b472845876bfaf0 [term 2 LEADER]: Becoming Leader. State: Replica: 197b880fa39d432a9b472845876bfaf0, State: Running, Role: LEADER
I20260129 02:59:45.473227 22425 raft_consensus.cc:515] T a8479d1fe0bd41e5946186e3a6ca8c7a P 1c50de89a5a845668696dcb28c3badd6 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
W20260129 02:59:45.600013 20556 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.211.2:34517, user_credentials={real_user=slave}} blocked reactor thread for 59025.3us
I20260129 02:59:45.600207 20514 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0678b8b468bd43cdbc3e485900844821" candidate_uuid: "7a9daf5d42a84615a6383cfec9a10608" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "974b802141bd4f13beffef2aa03f88f9" is_pre_election: true
I20260129 02:59:45.622141 20514 raft_consensus.cc:2468] T 0678b8b468bd43cdbc3e485900844821 P 974b802141bd4f13beffef2aa03f88f9 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7a9daf5d42a84615a6383cfec9a10608 in term 5.
I20260129 02:59:45.604421 20516 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "96609ac68bdf4993a12e1c23355ca013" candidate_uuid: "7a9daf5d42a84615a6383cfec9a10608" candidate_term: 5 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "974b802141bd4f13beffef2aa03f88f9" is_pre_election: true
I20260129 02:59:45.605017 20512 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "45cc5f3d2688486d9b6dbe7e666dedde" candidate_uuid: "7a9daf5d42a84615a6383cfec9a10608" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "974b802141bd4f13beffef2aa03f88f9" is_pre_election: true
I20260129 02:59:45.605223 20511 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4f5a77e94fef4b16a2ba7b067c8b144a" candidate_uuid: "7a9daf5d42a84615a6383cfec9a10608" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "974b802141bd4f13beffef2aa03f88f9" is_pre_election: true
I20260129 02:59:45.623451 20511 raft_consensus.cc:2468] T 4f5a77e94fef4b16a2ba7b067c8b144a P 974b802141bd4f13beffef2aa03f88f9 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7a9daf5d42a84615a6383cfec9a10608 in term 3.
I20260129 02:59:45.613071 21102 leader_election.cc:304] T 3e2d15f8640b407195210c476e09df91 P 7a9daf5d42a84615a6383cfec9a10608 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 7a9daf5d42a84615a6383cfec9a10608, 974b802141bd4f13beffef2aa03f88f9; no voters: 1c50de89a5a845668696dcb28c3badd6
W20260129 02:59:45.614172 20480 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260129 02:59:45.605479 20513 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a644c02b16af47679824ccd060ac10fd" candidate_uuid: "7a9daf5d42a84615a6383cfec9a10608" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "974b802141bd4f13beffef2aa03f88f9" is_pre_election: true
I20260129 02:59:45.626834 20513 raft_consensus.cc:2468] T a644c02b16af47679824ccd060ac10fd P 974b802141bd4f13beffef2aa03f88f9 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7a9daf5d42a84615a6383cfec9a10608 in term 3.
I20260129 02:59:45.631283 20516 raft_consensus.cc:2410] T 96609ac68bdf4993a12e1c23355ca013 P 974b802141bd4f13beffef2aa03f88f9 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 7a9daf5d42a84615a6383cfec9a10608 for term 5 because replica has last-logged OpId of term: 3 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 4.
I20260129 02:59:45.520812 22184 raft_consensus.cc:515] T 45cc5f3d2688486d9b6dbe7e666dedde P 974b802141bd4f13beffef2aa03f88f9 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } }
I20260129 02:59:45.634361 20480 leader_election.cc:304] T 96609ac68bdf4993a12e1c23355ca013 P 974b802141bd4f13beffef2aa03f88f9 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 197b880fa39d432a9b472845876bfaf0, 1c50de89a5a845668696dcb28c3badd6, 974b802141bd4f13beffef2aa03f88f9; no voters:
W20260129 02:59:45.653841 21102 leader_election.cc:336] T 45cc5f3d2688486d9b6dbe7e666dedde P 7a9daf5d42a84615a6383cfec9a10608 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 974b802141bd4f13beffef2aa03f88f9 (127.18.211.2:34517): Timed out: RequestConsensusVote RPC to 127.18.211.2:34517 timed out after 1.717s (SENT)
I20260129 02:59:45.654454 21102 leader_election.cc:304] T 45cc5f3d2688486d9b6dbe7e666dedde P 7a9daf5d42a84615a6383cfec9a10608 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7a9daf5d42a84615a6383cfec9a10608; no voters: 1c50de89a5a845668696dcb28c3badd6, 974b802141bd4f13beffef2aa03f88f9
I20260129 02:59:45.598484 22647 leader_election.cc:290] T 7a9a69cd82724b1faed1d10aa091fc4a P 197b880fa39d432a9b472845876bfaf0 [CANDIDATE]: Term 5 election: Requested vote from peers 974b802141bd4f13beffef2aa03f88f9 (127.18.211.2:34517), 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3:37735)
I20260129 02:59:45.597664 22231 raft_consensus.cc:493] T febffe4a8b394368b69b1d164308fb9a P 197b880fa39d432a9b472845876bfaf0 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 7a9daf5d42a84615a6383cfec9a10608)
I20260129 02:59:45.659458 20586 raft_consensus.cc:2393] T a8479d1fe0bd41e5946186e3a6ca8c7a P 1c50de89a5a845668696dcb28c3badd6 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 974b802141bd4f13beffef2aa03f88f9 in current term 4: Already voted for candidate 1c50de89a5a845668696dcb28c3badd6 in this term.
I20260129 02:59:45.552697 22586 raft_consensus.cc:515] T c934d31efd3f4e5195605ddc354d14e1 P 974b802141bd4f13beffef2aa03f88f9 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } }
I20260129 02:59:45.661208 20511 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0678b8b468bd43cdbc3e485900844821" candidate_uuid: "7a9daf5d42a84615a6383cfec9a10608" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "974b802141bd4f13beffef2aa03f88f9" is_pre_election: true
I20260129 02:59:45.666555 20556 leader_election.cc:304] T 75b0b6e82ddc4f92b1fe1d94f9a0b73b P 1c50de89a5a845668696dcb28c3badd6 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1c50de89a5a845668696dcb28c3badd6, 974b802141bd4f13beffef2aa03f88f9; no voters: 7a9daf5d42a84615a6383cfec9a10608
W20260129 02:59:45.668632 20407 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.211.2:34517, user_credentials={real_user=slave}} blocked reactor thread for 160916us
I20260129 02:59:45.668802 20513 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "505580c93c694bbb8ea96fdf9b17ab46" candidate_uuid: "7a9daf5d42a84615a6383cfec9a10608" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "974b802141bd4f13beffef2aa03f88f9" is_pre_election: true
I20260129 02:59:45.667343 20511 raft_consensus.cc:2468] T 0678b8b468bd43cdbc3e485900844821 P 974b802141bd4f13beffef2aa03f88f9 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7a9daf5d42a84615a6383cfec9a10608 in term 5.
I20260129 02:59:45.672621 20515 raft_consensus.cc:2393] T 45cc5f3d2688486d9b6dbe7e666dedde P 974b802141bd4f13beffef2aa03f88f9 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 7a9daf5d42a84615a6383cfec9a10608 in current term 4: Already voted for candidate 974b802141bd4f13beffef2aa03f88f9 in this term.
W20260129 02:59:45.656606 22493 consensus_meta.cc:287] T cb4727386a3e48aca5d71cc6c611c197 P 974b802141bd4f13beffef2aa03f88f9: Time spent flushing consensus metadata: real 0.591s user 0.005s sys 0.005s
I20260129 02:59:45.699954 20480 leader_election.cc:304] T 1299723a4476420ab02fd8bfd7aa0645 P 974b802141bd4f13beffef2aa03f88f9 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 974b802141bd4f13beffef2aa03f88f9; no voters: 197b880fa39d432a9b472845876bfaf0, 1c50de89a5a845668696dcb28c3badd6
I20260129 02:59:45.700275 20513 raft_consensus.cc:2393] T 505580c93c694bbb8ea96fdf9b17ab46 P 974b802141bd4f13beffef2aa03f88f9 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 7a9daf5d42a84615a6383cfec9a10608 in current term 4: Already voted for candidate 974b802141bd4f13beffef2aa03f88f9 in this term.
I20260129 02:59:45.667915 22231 raft_consensus.cc:3060] T febffe4a8b394368b69b1d164308fb9a P 197b880fa39d432a9b472845876bfaf0 [term 2 FOLLOWER]: Advancing to term 3
I20260129 02:59:45.628278 22343 raft_consensus.cc:515] T 4e3805ffa70242e8ade66da260dddf66 P 7a9daf5d42a84615a6383cfec9a10608 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } }
I20260129 02:59:45.663236 22633 raft_consensus.cc:697] T 87d5fe48a44a408aaf03e894a67ecb2f P 197b880fa39d432a9b472845876bfaf0 [term 5 LEADER]: Becoming Leader. State: Replica: 197b880fa39d432a9b472845876bfaf0, State: Running, Role: LEADER
I20260129 02:59:45.629259 22646 raft_consensus.cc:515] T cb4727386a3e48aca5d71cc6c611c197 P 197b880fa39d432a9b472845876bfaf0 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:59:45.600036 22197 consensus_queue.cc:237] T a609ac02fae84725ab743f95f51d33ce P 974b802141bd4f13beffef2aa03f88f9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } }
I20260129 02:59:45.739832 20511 raft_consensus.cc:1240] T 87d5fe48a44a408aaf03e894a67ecb2f P 974b802141bd4f13beffef2aa03f88f9 [term 5 FOLLOWER]: Rejecting Update request from peer 7a9daf5d42a84615a6383cfec9a10608 for earlier term 4. Current term is 5. Ops: []
W20260129 02:59:45.706233 20405 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.211.4:40439, user_credentials={real_user=slave}} blocked reactor thread for 145448us
W20260129 02:59:45.769376 20407 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.211.2:34517, user_credentials={real_user=slave}} blocked reactor thread for 82859us
I20260129 02:59:45.716079 20480 leader_election.cc:304] T fcddab6bca684adf8ea032289a00f90e P 974b802141bd4f13beffef2aa03f88f9 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1c50de89a5a845668696dcb28c3badd6, 974b802141bd4f13beffef2aa03f88f9; no voters: 197b880fa39d432a9b472845876bfaf0
I20260129 02:59:45.745558 22630 raft_consensus.cc:2749] T 0340229d521b432580ae926f6062e3f7 P 974b802141bd4f13beffef2aa03f88f9 [term 5 FOLLOWER]: Leader election lost for term 4. Reason: Vote denied by peer 197b880fa39d432a9b472845876bfaf0 (127.18.211.1:42027) with higher term. Message: Invalid argument: T 0340229d521b432580ae926f6062e3f7 P 197b880fa39d432a9b472845876bfaf0 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 974b802141bd4f13beffef2aa03f88f9 for earlier term 4. Current term is 5.
I20260129 02:59:45.737838 22672 raft_consensus.cc:2804] T 3e2d15f8640b407195210c476e09df91 P 7a9daf5d42a84615a6383cfec9a10608 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260129 02:59:45.744382 22237 consensus_queue.cc:237] T 4408006eb5174de1896b037cf3f18457 P 197b880fa39d432a9b472845876bfaf0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:59:45.744549 22579 raft_consensus.cc:2749] T 45cc5f3d2688486d9b6dbe7e666dedde P 7a9daf5d42a84615a6383cfec9a10608 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260129 02:59:45.702654 22680 raft_consensus.cc:2804] T 96609ac68bdf4993a12e1c23355ca013 P 974b802141bd4f13beffef2aa03f88f9 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260129 02:59:45.740016 22400 raft_consensus.cc:2749] T 1299723a4476420ab02fd8bfd7aa0645 P 974b802141bd4f13beffef2aa03f88f9 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260129 02:59:45.793838 22645 raft_consensus.cc:2764] T a609ac02fae84725ab743f95f51d33ce P 197b880fa39d432a9b472845876bfaf0 [term 6 FOLLOWER]: Leader pre-election decision vote started in defunct term 5: won
I20260129 02:59:45.707516 20509 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a609ac02fae84725ab743f95f51d33ce" candidate_uuid: "7a9daf5d42a84615a6383cfec9a10608" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "974b802141bd4f13beffef2aa03f88f9"
I20260129 02:59:45.702342 21130 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0678b8b468bd43cdbc3e485900844821" candidate_uuid: "974b802141bd4f13beffef2aa03f88f9" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "7a9daf5d42a84615a6383cfec9a10608" is_pre_election: true
I20260129 02:59:45.736617 22647 raft_consensus.cc:2749] T 00e79d8f25a34676b1d866eb2068a2b2 P 197b880fa39d432a9b472845876bfaf0 [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20260129 02:59:45.701766 22677 raft_consensus.cc:2804] T b189f08e16a74a11980cc55d99359a0a P 197b880fa39d432a9b472845876bfaf0 [term 5 FOLLOWER]: Leader pre-election won for term 6
W20260129 02:59:45.823622 21099 leader_election.cc:336] T 42d27a28d89b409e8807ec0efd97af69 P 7a9daf5d42a84615a6383cfec9a10608 [CANDIDATE]: Term 5 election: RPC error from VoteRequest() call to peer 197b880fa39d432a9b472845876bfaf0 (127.18.211.1:42027): Timed out: RequestConsensusVote RPC to 127.18.211.1:42027 timed out after 1.909s (SENT)
I20260129 02:59:45.820353 22680 raft_consensus.cc:493] T 96609ac68bdf4993a12e1c23355ca013 P 974b802141bd4f13beffef2aa03f88f9 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260129 02:59:45.742970 22635 raft_consensus.cc:515] T 913fc7d751ed4b7388d889aac30a5f9b P 197b880fa39d432a9b472845876bfaf0 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:59:45.793881 22688 raft_consensus.cc:493] T a4c1226b987b468fa84dfc6ba0e2dbe1 P 197b880fa39d432a9b472845876bfaf0 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 1c50de89a5a845668696dcb28c3badd6)
I20260129 02:59:45.744058 22184 leader_election.cc:290] T 45cc5f3d2688486d9b6dbe7e666dedde P 974b802141bd4f13beffef2aa03f88f9 [CANDIDATE]: Term 4 election: Requested vote from peers 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3:37735), 7a9daf5d42a84615a6383cfec9a10608 (127.18.211.4:40439)
W20260129 02:59:45.837195 20480 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.211.3:37735, user_credentials={real_user=slave}} blocked reactor thread for 121246us
I20260129 02:59:45.696941 22493 raft_consensus.cc:515] T cb4727386a3e48aca5d71cc6c611c197 P 974b802141bd4f13beffef2aa03f88f9 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:59:45.830040 22630 raft_consensus.cc:493] T 4408006eb5174de1896b037cf3f18457 P 974b802141bd4f13beffef2aa03f88f9 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260129 02:59:45.814574 22445 raft_consensus.cc:2804] T fcddab6bca684adf8ea032289a00f90e P 974b802141bd4f13beffef2aa03f88f9 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260129 02:59:45.848291 20407 leader_election.cc:304] T 1ce4ddf5b10e4ae8b193b13b2e5936ee P 197b880fa39d432a9b472845876bfaf0 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 197b880fa39d432a9b472845876bfaf0, 974b802141bd4f13beffef2aa03f88f9; no voters:
I20260129 02:59:45.861320 20480 leader_election.cc:304] T b189f08e16a74a11980cc55d99359a0a P 974b802141bd4f13beffef2aa03f88f9 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1c50de89a5a845668696dcb28c3badd6, 974b802141bd4f13beffef2aa03f88f9; no voters: 197b880fa39d432a9b472845876bfaf0
I20260129 02:59:45.849656 22680 raft_consensus.cc:3060] T 96609ac68bdf4993a12e1c23355ca013 P 974b802141bd4f13beffef2aa03f88f9 [term 4 FOLLOWER]: Advancing to term 5
I20260129 02:59:45.870031 20438 raft_consensus.cc:1240] T 913fc7d751ed4b7388d889aac30a5f9b P 197b880fa39d432a9b472845876bfaf0 [term 3 FOLLOWER]: Rejecting Update request from peer 1c50de89a5a845668696dcb28c3badd6 for earlier term 2. Current term is 3. Ops: []
W20260129 02:59:45.863063 22682 log.cc:927] Time spent T a5a05fbcbbcf4916bbc01fd1a5b5e90d P 1c50de89a5a845668696dcb28c3badd6: Append to log took a long time: real 0.090s user 0.000s sys 0.001s
I20260129 02:59:45.848222 22677 raft_consensus.cc:493] T b189f08e16a74a11980cc55d99359a0a P 197b880fa39d432a9b472845876bfaf0 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260129 02:59:45.872813 20407 leader_election.cc:304] T 368953c9102643b2a21f99af36b51905 P 197b880fa39d432a9b472845876bfaf0 [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 197b880fa39d432a9b472845876bfaf0; no voters: 1c50de89a5a845668696dcb28c3badd6, 974b802141bd4f13beffef2aa03f88f9
W20260129 02:59:45.873374 21099 leader_election.cc:336] T 2e58e9a5a8aa4cecafb464addb4f35ed P 7a9daf5d42a84615a6383cfec9a10608 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 197b880fa39d432a9b472845876bfaf0 (127.18.211.1:42027): Timed out: RequestConsensusVote RPC to 127.18.211.1:42027 timed out after 1.958s (SENT)
I20260129 02:59:45.848716 22586 leader_election.cc:290] T c934d31efd3f4e5195605ddc354d14e1 P 974b802141bd4f13beffef2aa03f88f9 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 197b880fa39d432a9b472845876bfaf0 (127.18.211.1:42027), 7a9daf5d42a84615a6383cfec9a10608 (127.18.211.4:40439)
W20260129 02:59:45.865492 22676 log.cc:927] Time spent T 75b0b6e82ddc4f92b1fe1d94f9a0b73b P 974b802141bd4f13beffef2aa03f88f9: Append to log took a long time: real 0.314s user 0.001s sys 0.000s
I20260129 02:59:45.876789 20433 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c934d31efd3f4e5195605ddc354d14e1" candidate_uuid: "974b802141bd4f13beffef2aa03f88f9" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "197b880fa39d432a9b472845876bfaf0" is_pre_election: true
I20260129 02:59:45.862843 22445 raft_consensus.cc:493] T fcddab6bca684adf8ea032289a00f90e P 974b802141bd4f13beffef2aa03f88f9 [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260129 02:59:45.879360 20433 raft_consensus.cc:2410] T c934d31efd3f4e5195605ddc354d14e1 P 197b880fa39d432a9b472845876bfaf0 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 974b802141bd4f13beffef2aa03f88f9 for term 5 because replica has last-logged OpId of term: 3 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 5.
I20260129 02:59:45.880074 20441 raft_consensus.cc:1240] T e33a417cf9c54cf5af73a4ee3aac4d3a P 197b880fa39d432a9b472845876bfaf0 [term 5 FOLLOWER]: Rejecting Update request from peer 974b802141bd4f13beffef2aa03f88f9 for earlier term 4. Current term is 5. Ops: []
W20260129 02:59:45.882795 20556 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.211.2:34517, user_credentials={real_user=slave}} blocked reactor thread for 234207us
W20260129 02:59:45.883121 20553 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.211.1:42027, user_credentials={real_user=slave}} blocked reactor thread for 487213us
I20260129 02:59:45.883203 22633 consensus_queue.cc:237] T 87d5fe48a44a408aaf03e894a67ecb2f P 197b880fa39d432a9b472845876bfaf0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } }
I20260129 02:59:45.883975 20556 leader_election.cc:304] T a5a05fbcbbcf4916bbc01fd1a5b5e90d P 1c50de89a5a845668696dcb28c3badd6 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1c50de89a5a845668696dcb28c3badd6; no voters: 197b880fa39d432a9b472845876bfaf0, 974b802141bd4f13beffef2aa03f88f9
I20260129 02:59:45.884756 22652 raft_consensus.cc:2804] T 75b0b6e82ddc4f92b1fe1d94f9a0b73b P 1c50de89a5a845668696dcb28c3badd6 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260129 02:59:45.884910 20553 leader_election.cc:304] T 1299723a4476420ab02fd8bfd7aa0645 P 1c50de89a5a845668696dcb28c3badd6 [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1c50de89a5a845668696dcb28c3badd6; no voters: 197b880fa39d432a9b472845876bfaf0, 7a9daf5d42a84615a6383cfec9a10608
I20260129 02:59:45.885222 22652 raft_consensus.cc:493] T 75b0b6e82ddc4f92b1fe1d94f9a0b73b P 1c50de89a5a845668696dcb28c3badd6 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 7a9daf5d42a84615a6383cfec9a10608)
I20260129 02:59:45.885677 22652 raft_consensus.cc:3060] T 75b0b6e82ddc4f92b1fe1d94f9a0b73b P 1c50de89a5a845668696dcb28c3badd6 [term 2 FOLLOWER]: Advancing to term 3
I20260129 02:59:45.886092 20553 leader_election.cc:304] T 0340229d521b432580ae926f6062e3f7 P 1c50de89a5a845668696dcb28c3badd6 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 197b880fa39d432a9b472845876bfaf0, 1c50de89a5a845668696dcb28c3badd6; no voters:
I20260129 02:59:45.867472 22425 leader_election.cc:290] T a8479d1fe0bd41e5946186e3a6ca8c7a P 1c50de89a5a845668696dcb28c3badd6 [CANDIDATE]: Term 4 election: Requested vote from peers 197b880fa39d432a9b472845876bfaf0 (127.18.211.1:42027), 974b802141bd4f13beffef2aa03f88f9 (127.18.211.2:34517)
I20260129 02:59:45.907964 20587 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0340229d521b432580ae926f6062e3f7" candidate_uuid: "974b802141bd4f13beffef2aa03f88f9" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "1c50de89a5a845668696dcb28c3badd6"
I20260129 02:59:45.908842 20587 raft_consensus.cc:2368] T 0340229d521b432580ae926f6062e3f7 P 1c50de89a5a845668696dcb28c3badd6 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 974b802141bd4f13beffef2aa03f88f9 for earlier term 4. Current term is 5.
I20260129 02:59:45.909639 20585 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "926a250fa5834e88b96ccecbefbf5086" candidate_uuid: "974b802141bd4f13beffef2aa03f88f9" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "1c50de89a5a845668696dcb28c3badd6" is_pre_election: true
I20260129 02:59:45.877766 22647 raft_consensus.cc:2804] T 1ce4ddf5b10e4ae8b193b13b2e5936ee P 197b880fa39d432a9b472845876bfaf0 [term 5 FOLLOWER]: Leader election won for term 5
I20260129 02:59:45.867072 21130 raft_consensus.cc:2468] T 0678b8b468bd43cdbc3e485900844821 P 7a9daf5d42a84615a6383cfec9a10608 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 974b802141bd4f13beffef2aa03f88f9 in term 5.
W20260129 02:59:45.917719 20407 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.211.2:34517, user_credentials={real_user=slave}} blocked reactor thread for 44958.1us
I20260129 02:59:45.912227 22645 raft_consensus.cc:2749] T 368953c9102643b2a21f99af36b51905 P 197b880fa39d432a9b472845876bfaf0 [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20260129 02:59:45.918797 20586 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "76f713c3331540d2a5b76164103e9eb1" candidate_uuid: "974b802141bd4f13beffef2aa03f88f9" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "1c50de89a5a845668696dcb28c3badd6" is_pre_election: true
I20260129 02:59:45.887547 22445 raft_consensus.cc:3060] T fcddab6bca684adf8ea032289a00f90e P 974b802141bd4f13beffef2aa03f88f9 [term 7 FOLLOWER]: Advancing to term 8
I20260129 02:59:45.924422 20582 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3906b080a5b34a74b8b64128635e8997" candidate_uuid: "974b802141bd4f13beffef2aa03f88f9" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "1c50de89a5a845668696dcb28c3badd6"
I20260129 02:59:45.925483 20585 raft_consensus.cc:2468] T 926a250fa5834e88b96ccecbefbf5086 P 1c50de89a5a845668696dcb28c3badd6 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 974b802141bd4f13beffef2aa03f88f9 in term 5.
I20260129 02:59:45.865826 22630 raft_consensus.cc:515] T 4408006eb5174de1896b037cf3f18457 P 974b802141bd4f13beffef2aa03f88f9 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:59:45.879258 22677 raft_consensus.cc:3060] T b189f08e16a74a11980cc55d99359a0a P 197b880fa39d432a9b472845876bfaf0 [term 5 FOLLOWER]: Advancing to term 6
I20260129 02:59:45.936848 22435 raft_consensus.cc:2749] T 1299723a4476420ab02fd8bfd7aa0645 P 1c50de89a5a845668696dcb28c3badd6 [term 7 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
I20260129 02:59:45.867249 22688 raft_consensus.cc:515] T a4c1226b987b468fa84dfc6ba0e2dbe1 P 197b880fa39d432a9b472845876bfaf0 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:59:45.953951 22343 leader_election.cc:290] T 4e3805ffa70242e8ade66da260dddf66 P 7a9daf5d42a84615a6383cfec9a10608 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 197b880fa39d432a9b472845876bfaf0 (127.18.211.1:42027), 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3:37735)
I20260129 02:59:45.954645 22681 raft_consensus.cc:493] T dc2f4cb752cf4d59b5f6bc689f5d8104 P 7a9daf5d42a84615a6383cfec9a10608 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260129 02:59:45.927438 22344 consensus_queue.cc:1059] T 3906b080a5b34a74b8b64128635e8997 P 1c50de89a5a845668696dcb28c3badd6 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 }, Status: INVALID_TERM, Last received: 2.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.030s
I20260129 02:59:45.955825 22493 leader_election.cc:290] T cb4727386a3e48aca5d71cc6c611c197 P 974b802141bd4f13beffef2aa03f88f9 [CANDIDATE]: Term 3 election: Requested vote from peers 197b880fa39d432a9b472845876bfaf0 (127.18.211.1:42027), 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3:37735)
I20260129 02:59:45.957203 22630 leader_election.cc:290] T 4408006eb5174de1896b037cf3f18457 P 974b802141bd4f13beffef2aa03f88f9 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 197b880fa39d432a9b472845876bfaf0 (127.18.211.1:42027), 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3:37735)
I20260129 02:59:45.875562 22184 consensus_queue.cc:1059] T e33a417cf9c54cf5af73a4ee3aac4d3a P 974b802141bd4f13beffef2aa03f88f9 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 }, Status: INVALID_TERM, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.067s
I20260129 02:59:45.912002 22554 consensus_queue.cc:1048] T 4f5a77e94fef4b16a2ba7b067c8b144a P 1c50de89a5a845668696dcb28c3badd6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.046s
I20260129 02:59:45.865257 22684 consensus_queue.cc:1059] T 42d27a28d89b409e8807ec0efd97af69 P 197b880fa39d432a9b472845876bfaf0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false }, Status: INVALID_TERM, Last received: 4.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.100s
I20260129 02:59:45.924995 22640 consensus_queue.cc:1059] T 724a236524cb4e7ba3428880681d6cc2 P 1c50de89a5a845668696dcb28c3badd6 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 }, Status: INVALID_TERM, Last received: 2.2, Next index: 3, Last known committed idx: 1, Time since last communication: 0.146s
W20260129 02:59:46.163529 20407 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.211.2:34517, user_credentials={real_user=slave}} blocked reactor thread for 234830us
I20260129 02:59:46.166038 20582 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "505580c93c694bbb8ea96fdf9b17ab46" candidate_uuid: "7a9daf5d42a84615a6383cfec9a10608" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "1c50de89a5a845668696dcb28c3badd6" is_pre_election: true
W20260129 02:59:45.962793 21099 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.211.1:42027, user_credentials={real_user=slave}} blocked reactor thread for 89468.4us
I20260129 02:59:46.028795 20586 raft_consensus.cc:2393] T 76f713c3331540d2a5b76164103e9eb1 P 1c50de89a5a845668696dcb28c3badd6 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 974b802141bd4f13beffef2aa03f88f9 in current term 4: Already voted for candidate 1c50de89a5a845668696dcb28c3badd6 in this term.
W20260129 02:59:46.140262 20553 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.211.1:42027, user_credentials={real_user=slave}} blocked reactor thread for 254200us
I20260129 02:59:46.169328 20553 leader_election.cc:304] T e7d83d4ccf684e76acb29a0dc844e1b8 P 1c50de89a5a845668696dcb28c3badd6 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 197b880fa39d432a9b472845876bfaf0, 1c50de89a5a845668696dcb28c3badd6; no voters:
W20260129 02:59:46.097985 20480 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.211.3:37735, user_credentials={real_user=slave}} blocked reactor thread for 236761us
I20260129 02:59:46.171195 20553 leader_election.cc:304] T 1ce4ddf5b10e4ae8b193b13b2e5936ee P 1c50de89a5a845668696dcb28c3badd6 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 197b880fa39d432a9b472845876bfaf0, 1c50de89a5a845668696dcb28c3badd6; no voters:
W20260129 02:59:46.117606 21101 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260129 02:59:46.172240 20480 leader_election.cc:304] T 7d9494f5584d4d5db8b9adf1c8fc888b P 974b802141bd4f13beffef2aa03f88f9 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 974b802141bd4f13beffef2aa03f88f9; no voters: 197b880fa39d432a9b472845876bfaf0, 1c50de89a5a845668696dcb28c3badd6
I20260129 02:59:46.031126 20441 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "cb4727386a3e48aca5d71cc6c611c197" candidate_uuid: "974b802141bd4f13beffef2aa03f88f9" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "197b880fa39d432a9b472845876bfaf0"
W20260129 02:59:45.957876 20556 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.211.2:34517, user_credentials={real_user=slave}} blocked reactor thread for 74000.2us
I20260129 02:59:46.174351 20553 leader_election.cc:304] T 368953c9102643b2a21f99af36b51905 P 1c50de89a5a845668696dcb28c3badd6 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 197b880fa39d432a9b472845876bfaf0, 1c50de89a5a845668696dcb28c3badd6; no voters:
I20260129 02:59:46.174445 20588 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c2633a6fc34c44579de348d208b0c01e" candidate_uuid: "974b802141bd4f13beffef2aa03f88f9" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "1c50de89a5a845668696dcb28c3badd6" is_pre_election: true
I20260129 02:59:46.240890 20441 raft_consensus.cc:2393] T cb4727386a3e48aca5d71cc6c611c197 P 197b880fa39d432a9b472845876bfaf0 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 974b802141bd4f13beffef2aa03f88f9 in current term 3: Already voted for candidate 197b880fa39d432a9b472845876bfaf0 in this term.
W20260129 02:59:46.227416 20480 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.211.3:37735, user_credentials={real_user=slave}} blocked reactor thread for 55238.3us
I20260129 02:59:46.225729 21134 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2e58e9a5a8aa4cecafb464addb4f35ed" candidate_uuid: "197b880fa39d432a9b472845876bfaf0" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "7a9daf5d42a84615a6383cfec9a10608"
I20260129 02:59:46.242676 20480 leader_election.cc:304] T 724a236524cb4e7ba3428880681d6cc2 P 974b802141bd4f13beffef2aa03f88f9 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 974b802141bd4f13beffef2aa03f88f9; no voters: 197b880fa39d432a9b472845876bfaf0, 1c50de89a5a845668696dcb28c3badd6
W20260129 02:59:46.246174 20553 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.211.1:42027, user_credentials={real_user=slave}} blocked reactor thread for 71912.6us
I20260129 02:59:45.918195 22586 raft_consensus.cc:2804] T b189f08e16a74a11980cc55d99359a0a P 974b802141bd4f13beffef2aa03f88f9 [term 5 FOLLOWER]: Leader election won for term 5
I20260129 02:59:46.167753 22646 leader_election.cc:290] T cb4727386a3e48aca5d71cc6c611c197 P 197b880fa39d432a9b472845876bfaf0 [CANDIDATE]: Term 3 election: Requested vote from peers 974b802141bd4f13beffef2aa03f88f9 (127.18.211.2:34517), 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3:37735)
I20260129 02:59:46.250056 22664 raft_consensus.cc:2804] T e7d83d4ccf684e76acb29a0dc844e1b8 P 1c50de89a5a845668696dcb28c3badd6 [term 4 FOLLOWER]: Leader election won for term 4
I20260129 02:59:46.015141 22584 raft_consensus.cc:493] T 78aa39096868415d9afc2ffcb248e198 P 1c50de89a5a845668696dcb28c3badd6 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260129 02:59:46.250923 22553 raft_consensus.cc:2804] T 368953c9102643b2a21f99af36b51905 P 1c50de89a5a845668696dcb28c3badd6 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260129 02:59:45.961086 22647 raft_consensus.cc:697] T 1ce4ddf5b10e4ae8b193b13b2e5936ee P 197b880fa39d432a9b472845876bfaf0 [term 5 LEADER]: Becoming Leader. State: Replica: 197b880fa39d432a9b472845876bfaf0, State: Running, Role: LEADER
I20260129 02:59:46.162876 22637 raft_consensus.cc:2804] T 0340229d521b432580ae926f6062e3f7 P 1c50de89a5a845668696dcb28c3badd6 [term 5 FOLLOWER]: Leader election won for term 5
I20260129 02:59:46.029407 21128 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "78aa39096868415d9afc2ffcb248e198" candidate_uuid: "197b880fa39d432a9b472845876bfaf0" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "7a9daf5d42a84615a6383cfec9a10608"
I20260129 02:59:46.261368 20556 leader_election.cc:304] T a0fcc58d6fb6473cafbd9d34d5195269 P 1c50de89a5a845668696dcb28c3badd6 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1c50de89a5a845668696dcb28c3badd6; no voters: 197b880fa39d432a9b472845876bfaf0, 974b802141bd4f13beffef2aa03f88f9
I20260129 02:59:46.267998 22590 raft_consensus.cc:493] T 78aa39096868415d9afc2ffcb248e198 P 7a9daf5d42a84615a6383cfec9a10608 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260129 02:59:46.268160 22672 raft_consensus.cc:493] T 3e2d15f8640b407195210c476e09df91 P 7a9daf5d42a84615a6383cfec9a10608 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 1c50de89a5a845668696dcb28c3badd6)
W20260129 02:59:46.268715 22686 log.cc:927] Time spent T 75f45538dc704d9a8ec4ff8ad9f3ea5b P 974b802141bd4f13beffef2aa03f88f9: Append to log took a long time: real 0.382s user 0.000s sys 0.001s
I20260129 02:59:46.268626 22184 raft_consensus.cc:2749] T 7d9494f5584d4d5db8b9adf1c8fc888b P 974b802141bd4f13beffef2aa03f88f9 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20260129 02:59:46.247006 22348 consensus_meta.cc:287] T a1a39dcd328e4268ae41b988c66b4a75 P 1c50de89a5a845668696dcb28c3badd6: Time spent flushing consensus metadata: real 0.887s user 0.014s sys 0.001s
I20260129 02:59:46.250466 22638 consensus_queue.cc:1048] T dc2f4cb752cf4d59b5f6bc689f5d8104 P 1c50de89a5a845668696dcb28c3badd6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.318s
I20260129 02:59:46.226486 22694 raft_consensus.cc:493] T 4f8f3a38f6e24ba4857875181f447442 P 974b802141bd4f13beffef2aa03f88f9 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 1c50de89a5a845668696dcb28c3badd6)
I20260129 02:59:46.272178 22684 raft_consensus.cc:493] T 7d9494f5584d4d5db8b9adf1c8fc888b P 197b880fa39d432a9b472845876bfaf0 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 1c50de89a5a845668696dcb28c3badd6)
W20260129 02:59:46.312500 22698 log.cc:927] Time spent T add8d4923d1a489e86882c3a25692b95 P 974b802141bd4f13beffef2aa03f88f9: Append to log took a long time: real 0.066s user 0.000s sys 0.000s
I20260129 02:59:46.285861 22672 raft_consensus.cc:3060] T 3e2d15f8640b407195210c476e09df91 P 7a9daf5d42a84615a6383cfec9a10608 [term 1 FOLLOWER]: Advancing to term 2
I20260129 02:59:46.314375 22647 consensus_queue.cc:237] T 1ce4ddf5b10e4ae8b193b13b2e5936ee P 197b880fa39d432a9b472845876bfaf0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 2.2, Last appended by leader: 2, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:59:46.274250 22553 raft_consensus.cc:493] T 368953c9102643b2a21f99af36b51905 P 1c50de89a5a845668696dcb28c3badd6 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260129 02:59:46.316027 22553 raft_consensus.cc:3060] T 368953c9102643b2a21f99af36b51905 P 1c50de89a5a845668696dcb28c3badd6 [term 5 FOLLOWER]: Advancing to term 6
I20260129 02:59:46.319286 22554 consensus_queue.cc:1059] T 75be4bd6bd45474795fd472eff2953ac P 1c50de89a5a845668696dcb28c3badd6 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 }, Status: INVALID_TERM, Last received: 7.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260129 02:59:46.320443 22554 raft_consensus.cc:2749] T a0fcc58d6fb6473cafbd9d34d5195269 P 1c50de89a5a845668696dcb28c3badd6 [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260129 02:59:46.320689 20433 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4408006eb5174de1896b037cf3f18457" candidate_uuid: "974b802141bd4f13beffef2aa03f88f9" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "197b880fa39d432a9b472845876bfaf0" is_pre_election: true
I20260129 02:59:46.321086 22554 raft_consensus.cc:3055] T 75be4bd6bd45474795fd472eff2953ac P 1c50de89a5a845668696dcb28c3badd6 [term 7 LEADER]: Stepping down as leader of term 7
I20260129 02:59:46.321424 22554 raft_consensus.cc:740] T 75be4bd6bd45474795fd472eff2953ac P 1c50de89a5a845668696dcb28c3badd6 [term 7 LEADER]: Becoming Follower/Learner. State: Replica: 1c50de89a5a845668696dcb28c3badd6, State: Running, Role: LEADER
I20260129 02:59:46.321907 20433 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "96609ac68bdf4993a12e1c23355ca013" candidate_uuid: "7a9daf5d42a84615a6383cfec9a10608" candidate_term: 5 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "197b880fa39d432a9b472845876bfaf0" is_pre_election: true
I20260129 02:59:46.322506 20433 raft_consensus.cc:2468] T 96609ac68bdf4993a12e1c23355ca013 P 197b880fa39d432a9b472845876bfaf0 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7a9daf5d42a84615a6383cfec9a10608 in term 4.
I20260129 02:59:46.322214 22554 consensus_queue.cc:260] T 75be4bd6bd45474795fd472eff2953ac P 1c50de89a5a845668696dcb28c3badd6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 7.5, Last appended by leader: 5, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:59:46.281801 21130 raft_consensus.cc:1240] T 2ac659d732314e1288849f08fe5e074e P 7a9daf5d42a84615a6383cfec9a10608 [term 3 FOLLOWER]: Rejecting Update request from peer 197b880fa39d432a9b472845876bfaf0 for earlier term 2. Current term is 3. Ops: [2.5-2.5]
I20260129 02:59:46.323480 20433 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4e3805ffa70242e8ade66da260dddf66" candidate_uuid: "7a9daf5d42a84615a6383cfec9a10608" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "197b880fa39d432a9b472845876bfaf0" is_pre_election: true
I20260129 02:59:46.287619 21131 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3ccac82a01a2445fa5b0ce6c0fcf2295" candidate_uuid: "197b880fa39d432a9b472845876bfaf0" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "7a9daf5d42a84615a6383cfec9a10608" is_pre_election: true
I20260129 02:59:46.331604 21131 raft_consensus.cc:2393] T 3ccac82a01a2445fa5b0ce6c0fcf2295 P 7a9daf5d42a84615a6383cfec9a10608 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 197b880fa39d432a9b472845876bfaf0 in current term 3: Already voted for candidate 974b802141bd4f13beffef2aa03f88f9 in this term.
I20260129 02:59:46.291209 20553 leader_election.cc:304] T e33a417cf9c54cf5af73a4ee3aac4d3a P 1c50de89a5a845668696dcb28c3badd6 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 197b880fa39d432a9b472845876bfaf0, 1c50de89a5a845668696dcb28c3badd6; no voters:
W20260129 02:59:46.331732 20554 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260129 02:59:46.272784 20480 leader_election.cc:304] T e4d18c35b0a7493e85d0a4b92956a48f P 974b802141bd4f13beffef2aa03f88f9 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 974b802141bd4f13beffef2aa03f88f9; no voters: 197b880fa39d432a9b472845876bfaf0, 1c50de89a5a845668696dcb28c3badd6
I20260129 02:59:46.332664 21099 leader_election.cc:304] T 96609ac68bdf4993a12e1c23355ca013 P 7a9daf5d42a84615a6383cfec9a10608 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 197b880fa39d432a9b472845876bfaf0, 1c50de89a5a845668696dcb28c3badd6, 7a9daf5d42a84615a6383cfec9a10608; no voters:
I20260129 02:59:46.262380 22343 consensus_queue.cc:1059] T 87d5fe48a44a408aaf03e894a67ecb2f P 7a9daf5d42a84615a6383cfec9a10608 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 }, Status: INVALID_TERM, Last received: 4.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.036s
I20260129 02:59:46.297887 22668 raft_consensus.cc:2804] T 1ce4ddf5b10e4ae8b193b13b2e5936ee P 1c50de89a5a845668696dcb28c3badd6 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260129 02:59:46.298223 21134 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "18f30ebaa4174fd28c958e72d0cc7e24" candidate_uuid: "197b880fa39d432a9b472845876bfaf0" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "7a9daf5d42a84615a6383cfec9a10608" is_pre_election: true
I20260129 02:59:46.298694 22694 raft_consensus.cc:515] T 4f8f3a38f6e24ba4857875181f447442 P 974b802141bd4f13beffef2aa03f88f9 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
W20260129 02:59:46.339540 20480 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.211.3:37735, user_credentials={real_user=slave}} blocked reactor thread for 66888us
I20260129 02:59:46.302471 21129 raft_consensus.cc:1275] T 0678b8b468bd43cdbc3e485900844821 P 7a9daf5d42a84615a6383cfec9a10608 [term 5 FOLLOWER]: Refusing update from remote peer 197b880fa39d432a9b472845876bfaf0: Log matching property violated. Preceding OpId in replica: term: 4 index: 6. Preceding OpId from leader: term: 5 index: 7. (index mismatch)
I20260129 02:59:46.304169 20441 raft_consensus.cc:3055] T 42d27a28d89b409e8807ec0efd97af69 P 197b880fa39d432a9b472845876bfaf0 [term 4 LEADER]: Stepping down as leader of term 4
I20260129 02:59:46.340149 20441 raft_consensus.cc:740] T 42d27a28d89b409e8807ec0efd97af69 P 197b880fa39d432a9b472845876bfaf0 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 197b880fa39d432a9b472845876bfaf0, State: Running, Role: LEADER
I20260129 02:59:46.340301 20480 leader_election.cc:304] T cedce331e1584ac8b636c9b8a72b2ba0 P 974b802141bd4f13beffef2aa03f88f9 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1c50de89a5a845668696dcb28c3badd6, 974b802141bd4f13beffef2aa03f88f9; no voters: 197b880fa39d432a9b472845876bfaf0
I20260129 02:59:46.305271 20435 raft_consensus.cc:1275] T fd8311b779af4e1fa1402b2a02a602f2 P 197b880fa39d432a9b472845876bfaf0 [term 4 FOLLOWER]: Refusing update from remote peer 7a9daf5d42a84615a6383cfec9a10608: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20260129 02:59:46.305869 20440 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a609ac02fae84725ab743f95f51d33ce" candidate_uuid: "7a9daf5d42a84615a6383cfec9a10608" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "197b880fa39d432a9b472845876bfaf0"
I20260129 02:59:46.341845 20440 raft_consensus.cc:2393] T a609ac02fae84725ab743f95f51d33ce P 197b880fa39d432a9b472845876bfaf0 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 7a9daf5d42a84615a6383cfec9a10608 in current term 6: Already voted for candidate 974b802141bd4f13beffef2aa03f88f9 in this term.
I20260129 02:59:46.306970 20432 raft_consensus.cc:1275] T 2e58e9a5a8aa4cecafb464addb4f35ed P 197b880fa39d432a9b472845876bfaf0 [term 4 FOLLOWER]: Refusing update from remote peer 7a9daf5d42a84615a6383cfec9a10608: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20260129 02:59:46.306766 22671 consensus_queue.cc:1059] T 00e79d8f25a34676b1d866eb2068a2b2 P 7a9daf5d42a84615a6383cfec9a10608 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 }, Status: INVALID_TERM, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.045s
I20260129 02:59:46.307665 20434 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "18f30ebaa4174fd28c958e72d0cc7e24" candidate_uuid: "7a9daf5d42a84615a6383cfec9a10608" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "197b880fa39d432a9b472845876bfaf0" is_pre_election: true
I20260129 02:59:46.269232 22566 raft_consensus.cc:2749] T 724a236524cb4e7ba3428880681d6cc2 P 974b802141bd4f13beffef2aa03f88f9 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260129 02:59:46.327000 22540 raft_consensus.cc:3055] T 2ac659d732314e1288849f08fe5e074e P 197b880fa39d432a9b472845876bfaf0 [term 2 LEADER]: Stepping down as leader of term 2
I20260129 02:59:46.352000 22540 raft_consensus.cc:740] T 2ac659d732314e1288849f08fe5e074e P 197b880fa39d432a9b472845876bfaf0 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 197b880fa39d432a9b472845876bfaf0, State: Running, Role: LEADER
W20260129 02:59:46.353466 22692 log.cc:927] Time spent T a0fcc58d6fb6473cafbd9d34d5195269 P 974b802141bd4f13beffef2aa03f88f9: Append to log took a long time: real 0.211s user 0.003s sys 0.000s
I20260129 02:59:46.354846 21135 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "87d5fe48a44a408aaf03e894a67ecb2f" candidate_uuid: "197b880fa39d432a9b472845876bfaf0" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "7a9daf5d42a84615a6383cfec9a10608"
I20260129 02:59:46.357016 20440 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4f8f3a38f6e24ba4857875181f447442" candidate_uuid: "974b802141bd4f13beffef2aa03f88f9" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "197b880fa39d432a9b472845876bfaf0" is_pre_election: true
I20260129 02:59:46.357273 22668 raft_consensus.cc:493] T 1ce4ddf5b10e4ae8b193b13b2e5936ee P 1c50de89a5a845668696dcb28c3badd6 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260129 02:59:46.285550 22590 raft_consensus.cc:515] T 78aa39096868415d9afc2ffcb248e198 P 7a9daf5d42a84615a6383cfec9a10608 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } }
I20260129 02:59:46.357664 20440 raft_consensus.cc:2468] T 4f8f3a38f6e24ba4857875181f447442 P 197b880fa39d432a9b472845876bfaf0 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 974b802141bd4f13beffef2aa03f88f9 in term 4.
I20260129 02:59:46.329221 22688 leader_election.cc:290] T a4c1226b987b468fa84dfc6ba0e2dbe1 P 197b880fa39d432a9b472845876bfaf0 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 974b802141bd4f13beffef2aa03f88f9 (127.18.211.2:34517), 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3:37735)
I20260129 02:59:46.277104 22586 raft_consensus.cc:697] T b189f08e16a74a11980cc55d99359a0a P 974b802141bd4f13beffef2aa03f88f9 [term 5 LEADER]: Becoming Leader. State: Replica: 974b802141bd4f13beffef2aa03f88f9, State: Running, Role: LEADER
I20260129 02:59:46.359170 22590 leader_election.cc:290] T 78aa39096868415d9afc2ffcb248e198 P 7a9daf5d42a84615a6383cfec9a10608 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 197b880fa39d432a9b472845876bfaf0 (127.18.211.1:42027), 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3:37735)
I20260129 02:59:46.359107 22586 consensus_queue.cc:237] T b189f08e16a74a11980cc55d99359a0a P 974b802141bd4f13beffef2aa03f88f9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:59:46.359928 22590 consensus_queue.cc:1048] T 2e58e9a5a8aa4cecafb464addb4f35ed P 7a9daf5d42a84615a6383cfec9a10608 [LEADER]: Connected to new peer: Peer: permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260129 02:59:46.360397 20479 leader_election.cc:304] T 4f8f3a38f6e24ba4857875181f447442 P 974b802141bd4f13beffef2aa03f88f9 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 197b880fa39d432a9b472845876bfaf0, 974b802141bd4f13beffef2aa03f88f9; no voters:
I20260129 02:59:46.360966 22636 raft_consensus.cc:3060] T 8055232cf0004d40a85f3bf983fe00a9 P 197b880fa39d432a9b472845876bfaf0 [term 2 FOLLOWER]: Advancing to term 3
I20260129 02:59:46.361251 22566 raft_consensus.cc:2804] T 4f8f3a38f6e24ba4857875181f447442 P 974b802141bd4f13beffef2aa03f88f9 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260129 02:59:46.361696 22566 raft_consensus.cc:493] T 4f8f3a38f6e24ba4857875181f447442 P 974b802141bd4f13beffef2aa03f88f9 [term 4 FOLLOWER]: Starting leader election (detected failure of leader 1c50de89a5a845668696dcb28c3badd6)
I20260129 02:59:46.248775 22197 consensus_queue.cc:1059] T e33a417cf9c54cf5af73a4ee3aac4d3a P 974b802141bd4f13beffef2aa03f88f9 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 }, Status: INVALID_TERM, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.041s
I20260129 02:59:46.362076 22566 raft_consensus.cc:3060] T 4f8f3a38f6e24ba4857875181f447442 P 974b802141bd4f13beffef2aa03f88f9 [term 4 FOLLOWER]: Advancing to term 5
I20260129 02:59:46.362756 20441 consensus_queue.cc:260] T 42d27a28d89b409e8807ec0efd97af69 P 197b880fa39d432a9b472845876bfaf0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } }
I20260129 02:59:46.357683 22668 raft_consensus.cc:3060] T 1ce4ddf5b10e4ae8b193b13b2e5936ee P 1c50de89a5a845668696dcb28c3badd6 [term 5 FOLLOWER]: Advancing to term 6
I20260129 02:59:46.295817 21136 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fd8311b779af4e1fa1402b2a02a602f2" candidate_uuid: "197b880fa39d432a9b472845876bfaf0" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "7a9daf5d42a84615a6383cfec9a10608" is_pre_election: true
I20260129 02:59:46.368340 22581 raft_consensus.cc:2749] T a5a05fbcbbcf4916bbc01fd1a5b5e90d P 1c50de89a5a845668696dcb28c3badd6 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260129 02:59:46.289824 22635 leader_election.cc:290] T 913fc7d751ed4b7388d889aac30a5f9b P 197b880fa39d432a9b472845876bfaf0 [CANDIDATE]: Term 3 election: Requested vote from peers 974b802141bd4f13beffef2aa03f88f9 (127.18.211.2:34517), 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3:37735)
I20260129 02:59:46.347383 20590 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35752 (request call id 1521) took 1183 ms. Trace:
I20260129 02:59:46.369318 20590 rpcz_store.cc:276] 0129 02:59:45.163571 (+ 0us) service_pool.cc:168] Inserting onto call queue
0129 02:59:45.163778 (+ 207us) service_pool.cc:225] Handling call
0129 02:59:45.172461 (+ 8683us) raft_consensus.cc:1471] Updating replica for 1 ops
0129 02:59:45.353213 (+180752us) raft_consensus.cc:1534] Early marking committed up to index 0
0129 02:59:45.414473 (+ 61260us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0129 02:59:45.425616 (+ 11143us) log.cc:844] Serialized 26 byte log entry
0129 02:59:45.559963 (+134347us) raft_consensus.cc:1645] Marking committed up to 4
0129 02:59:46.307650 (+747687us) log.cc:844] Serialized 14 byte log entry
0129 02:59:46.347037 (+ 39387us) raft_consensus.cc:1697] Filling consensus response to leader.
0129 02:59:46.347137 (+ 100us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0129 02:59:46.347183 (+ 46us) raft_consensus.cc:1685] finished
0129 02:59:46.347196 (+ 13us) raft_consensus.cc:1691] UpdateReplicas() finished
0129 02:59:46.347324 (+ 128us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":771460,"prepare.queue_time_us":748402,"prepare.run_cpu_time_us":30,"prepare.run_wall_time_us":30,"raft.queue_time_us":563025,"raft.run_cpu_time_us":226,"raft.run_wall_time_us":226,"thread_start_us":208105,"threads_started":2,"wal-append.queue_time_us":209246}
I20260129 02:59:46.312527 22684 raft_consensus.cc:515] T 7d9494f5584d4d5db8b9adf1c8fc888b P 197b880fa39d432a9b472845876bfaf0 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:59:46.301239 20583 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "36ff33be666a4c82954731691dbcf327" candidate_uuid: "974b802141bd4f13beffef2aa03f88f9" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "1c50de89a5a845668696dcb28c3badd6" is_pre_election: true
I20260129 02:59:46.343611 22400 raft_consensus.cc:2749] T e4d18c35b0a7493e85d0a4b92956a48f P 974b802141bd4f13beffef2aa03f88f9 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20260129 02:59:46.344112 22160 raft_consensus.cc:2804] T cedce331e1584ac8b636c9b8a72b2ba0 P 974b802141bd4f13beffef2aa03f88f9 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260129 02:59:46.344188 20480 leader_election.cc:304] T 75be4bd6bd45474795fd472eff2953ac P 974b802141bd4f13beffef2aa03f88f9 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 974b802141bd4f13beffef2aa03f88f9; no voters: 197b880fa39d432a9b472845876bfaf0, 1c50de89a5a845668696dcb28c3badd6
I20260129 02:59:46.379555 20441 raft_consensus.cc:3060] T 42d27a28d89b409e8807ec0efd97af69 P 197b880fa39d432a9b472845876bfaf0 [term 4 FOLLOWER]: Advancing to term 5
I20260129 02:59:46.344960 21131 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "febffe4a8b394368b69b1d164308fb9a" candidate_uuid: "197b880fa39d432a9b472845876bfaf0" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "7a9daf5d42a84615a6383cfec9a10608" is_pre_election: true
I20260129 02:59:46.346578 22403 raft_consensus.cc:493] T c934d31efd3f4e5195605ddc354d14e1 P 7a9daf5d42a84615a6383cfec9a10608 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 197b880fa39d432a9b472845876bfaf0)
I20260129 02:59:46.380882 20480 leader_election.cc:304] T 8055232cf0004d40a85f3bf983fe00a9 P 974b802141bd4f13beffef2aa03f88f9 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1c50de89a5a845668696dcb28c3badd6, 974b802141bd4f13beffef2aa03f88f9; no voters: 197b880fa39d432a9b472845876bfaf0
I20260129 02:59:46.348625 21134 raft_consensus.cc:2368] T 18f30ebaa4174fd28c958e72d0cc7e24 P 7a9daf5d42a84615a6383cfec9a10608 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 197b880fa39d432a9b472845876bfaf0 for earlier term 3. Current term is 4.
I20260129 02:59:46.349239 22671 raft_consensus.cc:3055] T 00e79d8f25a34676b1d866eb2068a2b2 P 7a9daf5d42a84615a6383cfec9a10608 [term 4 LEADER]: Stepping down as leader of term 4
I20260129 02:59:46.350035 22343 raft_consensus.cc:3055] T 87d5fe48a44a408aaf03e894a67ecb2f P 7a9daf5d42a84615a6383cfec9a10608 [term 4 LEADER]: Stepping down as leader of term 4
I20260129 02:59:46.350509 20434 raft_consensus.cc:2393] T 18f30ebaa4174fd28c958e72d0cc7e24 P 197b880fa39d432a9b472845876bfaf0 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 7a9daf5d42a84615a6383cfec9a10608 in current term 4: Already voted for candidate 1c50de89a5a845668696dcb28c3badd6 in this term.
I20260129 02:59:46.351325 20433 raft_consensus.cc:1275] T a609ac02fae84725ab743f95f51d33ce P 197b880fa39d432a9b472845876bfaf0 [term 6 FOLLOWER]: Refusing update from remote peer 974b802141bd4f13beffef2aa03f88f9: Log matching property violated. Preceding OpId in replica: term: 4 index: 6. Preceding OpId from leader: term: 6 index: 7. (index mismatch)
I20260129 02:59:46.352937 22540 consensus_queue.cc:260] T 2ac659d732314e1288849f08fe5e074e P 197b880fa39d432a9b472845876bfaf0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 2.5, Last appended by leader: 5, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } }
I20260129 02:59:46.367779 21130 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "96609ac68bdf4993a12e1c23355ca013" candidate_uuid: "197b880fa39d432a9b472845876bfaf0" candidate_term: 5 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "7a9daf5d42a84615a6383cfec9a10608" is_pre_election: true
W20260129 02:59:46.373608 20556 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.211.2:34517, user_credentials={real_user=slave}} blocked reactor thread for 125773us
I20260129 02:59:46.335235 20588 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4e3805ffa70242e8ade66da260dddf66" candidate_uuid: "7a9daf5d42a84615a6383cfec9a10608" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "1c50de89a5a845668696dcb28c3badd6" is_pre_election: true
I20260129 02:59:46.293236 22664 raft_consensus.cc:697] T e7d83d4ccf684e76acb29a0dc844e1b8 P 1c50de89a5a845668696dcb28c3badd6 [term 4 LEADER]: Becoming Leader. State: Replica: 1c50de89a5a845668696dcb28c3badd6, State: Running, Role: LEADER
I20260129 02:59:46.378942 22704 raft_consensus.cc:2804] T e33a417cf9c54cf5af73a4ee3aac4d3a P 1c50de89a5a845668696dcb28c3badd6 [term 5 FOLLOWER]: Leader election won for term 5
W20260129 02:59:46.376117 20553 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.211.1:42027, user_credentials={real_user=slave}} blocked reactor thread for 85027.6us
W20260129 02:59:46.380755 20405 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260129 02:59:46.346900 22372 raft_consensus.cc:2804] T 96609ac68bdf4993a12e1c23355ca013 P 7a9daf5d42a84615a6383cfec9a10608 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260129 02:59:46.245738 22681 raft_consensus.cc:515] T dc2f4cb752cf4d59b5f6bc689f5d8104 P 7a9daf5d42a84615a6383cfec9a10608 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } }
I20260129 02:59:46.355616 22694 leader_election.cc:290] T 4f8f3a38f6e24ba4857875181f447442 P 974b802141bd4f13beffef2aa03f88f9 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 197b880fa39d432a9b472845876bfaf0 (127.18.211.1:42027), 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3:37735)
I20260129 02:59:46.381873 20440 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "78aa39096868415d9afc2ffcb248e198" candidate_uuid: "7a9daf5d42a84615a6383cfec9a10608" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "197b880fa39d432a9b472845876bfaf0" is_pre_election: true
I20260129 02:59:46.485702 20440 raft_consensus.cc:2468] T 78aa39096868415d9afc2ffcb248e198 P 197b880fa39d432a9b472845876bfaf0 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7a9daf5d42a84615a6383cfec9a10608 in term 4.
I20260129 02:59:46.386880 22343 raft_consensus.cc:740] T 87d5fe48a44a408aaf03e894a67ecb2f P 7a9daf5d42a84615a6383cfec9a10608 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 7a9daf5d42a84615a6383cfec9a10608, State: Running, Role: LEADER
I20260129 02:59:46.384915 22628 raft_consensus.cc:2749] T 75be4bd6bd45474795fd472eff2953ac P 974b802141bd4f13beffef2aa03f88f9 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260129 02:59:46.338662 22637 raft_consensus.cc:697] T 0340229d521b432580ae926f6062e3f7 P 1c50de89a5a845668696dcb28c3badd6 [term 5 LEADER]: Becoming Leader. State: Replica: 1c50de89a5a845668696dcb28c3badd6, State: Running, Role: LEADER
W20260129 02:59:46.478308 22633 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.592s user 0.008s sys 0.015s
W20260129 02:59:46.491937 22633 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.606s user 0.008s sys 0.015s
I20260129 02:59:46.492307 22637 consensus_queue.cc:237] T 0340229d521b432580ae926f6062e3f7 P 1c50de89a5a845668696dcb28c3badd6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:59:46.281122 22584 raft_consensus.cc:515] T 78aa39096868415d9afc2ffcb248e198 P 1c50de89a5a845668696dcb28c3badd6 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } }
I20260129 02:59:46.389189 21099 leader_election.cc:304] T 18f30ebaa4174fd28c958e72d0cc7e24 P 7a9daf5d42a84615a6383cfec9a10608 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7a9daf5d42a84615a6383cfec9a10608; no voters: 197b880fa39d432a9b472845876bfaf0, 1c50de89a5a845668696dcb28c3badd6
I20260129 02:59:46.408849 20480 leader_election.cc:304] T 913fc7d751ed4b7388d889aac30a5f9b P 974b802141bd4f13beffef2aa03f88f9 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 974b802141bd4f13beffef2aa03f88f9; no voters: 197b880fa39d432a9b472845876bfaf0, 1c50de89a5a845668696dcb28c3badd6
W20260129 02:59:46.498548 20480 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.211.3:37735, user_credentials={real_user=slave}} blocked reactor thread for 89812.6us
I20260129 02:59:46.499569 20480 leader_election.cc:304] T 17d6431cc77a491e8464d68db4081a51 P 974b802141bd4f13beffef2aa03f88f9 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 974b802141bd4f13beffef2aa03f88f9; no voters: 197b880fa39d432a9b472845876bfaf0, 1c50de89a5a845668696dcb28c3badd6
I20260129 02:59:46.501293 20480 leader_election.cc:304] T 368953c9102643b2a21f99af36b51905 P 974b802141bd4f13beffef2aa03f88f9 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1c50de89a5a845668696dcb28c3badd6, 974b802141bd4f13beffef2aa03f88f9; no voters: 197b880fa39d432a9b472845876bfaf0
I20260129 02:59:46.414561 20516 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3906b080a5b34a74b8b64128635e8997" candidate_uuid: "197b880fa39d432a9b472845876bfaf0" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "974b802141bd4f13beffef2aa03f88f9"
I20260129 02:59:46.502844 20516 raft_consensus.cc:2393] T 3906b080a5b34a74b8b64128635e8997 P 974b802141bd4f13beffef2aa03f88f9 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 197b880fa39d432a9b472845876bfaf0 in current term 3: Already voted for candidate 974b802141bd4f13beffef2aa03f88f9 in this term.
I20260129 02:59:46.504128 20516 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "75be4bd6bd45474795fd472eff2953ac" candidate_uuid: "197b880fa39d432a9b472845876bfaf0" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "974b802141bd4f13beffef2aa03f88f9"
I20260129 02:59:46.379840 22160 raft_consensus.cc:493] T cedce331e1584ac8b636c9b8a72b2ba0 P 974b802141bd4f13beffef2aa03f88f9 [term 4 FOLLOWER]: Starting leader election (detected failure of leader 1c50de89a5a845668696dcb28c3badd6)
I20260129 02:59:46.505113 22160 raft_consensus.cc:3060] T cedce331e1584ac8b636c9b8a72b2ba0 P 974b802141bd4f13beffef2aa03f88f9 [term 4 FOLLOWER]: Advancing to term 5
W20260129 02:59:46.479564 20553 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260129 02:59:46.412055 22231 consensus_meta.cc:287] T febffe4a8b394368b69b1d164308fb9a P 197b880fa39d432a9b472845876bfaf0: Time spent flushing consensus metadata: real 0.599s user 0.010s sys 0.001s
I20260129 02:59:46.508581 22231 raft_consensus.cc:515] T febffe4a8b394368b69b1d164308fb9a P 197b880fa39d432a9b472845876bfaf0 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } }
I20260129 02:59:46.510080 20437 raft_consensus.cc:1240] T febffe4a8b394368b69b1d164308fb9a P 197b880fa39d432a9b472845876bfaf0 [term 3 FOLLOWER]: Rejecting Update request from peer 7a9daf5d42a84615a6383cfec9a10608 for earlier term 2. Current term is 3. Ops: [2.5-2.5]
I20260129 02:59:46.511067 22231 leader_election.cc:290] T febffe4a8b394368b69b1d164308fb9a P 197b880fa39d432a9b472845876bfaf0 [CANDIDATE]: Term 3 election: Requested vote from peers 974b802141bd4f13beffef2aa03f88f9 (127.18.211.2:34517), 7a9daf5d42a84615a6383cfec9a10608 (127.18.211.4:40439)
I20260129 02:59:46.512390 22704 raft_consensus.cc:697] T e33a417cf9c54cf5af73a4ee3aac4d3a P 1c50de89a5a845668696dcb28c3badd6 [term 5 LEADER]: Becoming Leader. State: Replica: 1c50de89a5a845668696dcb28c3badd6, State: Running, Role: LEADER
I20260129 02:59:46.513295 22691 raft_consensus.cc:493] T c934d31efd3f4e5195605ddc354d14e1 P 197b880fa39d432a9b472845876bfaf0 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260129 02:59:46.513459 22704 consensus_queue.cc:237] T e33a417cf9c54cf5af73a4ee3aac4d3a P 1c50de89a5a845668696dcb28c3badd6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:59:46.513842 22691 raft_consensus.cc:515] T c934d31efd3f4e5195605ddc354d14e1 P 197b880fa39d432a9b472845876bfaf0 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } }
I20260129 02:59:46.516427 20433 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a8479d1fe0bd41e5946186e3a6ca8c7a" candidate_uuid: "1c50de89a5a845668696dcb28c3badd6" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "197b880fa39d432a9b472845876bfaf0"
W20260129 02:59:46.523459 20553 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260129 02:59:46.483530 20508 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "17d6431cc77a491e8464d68db4081a51" candidate_uuid: "197b880fa39d432a9b472845876bfaf0" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "974b802141bd4f13beffef2aa03f88f9" is_pre_election: true
I20260129 02:59:46.382695 22554 raft_consensus.cc:3060] T 75be4bd6bd45474795fd472eff2953ac P 1c50de89a5a845668696dcb28c3badd6 [term 7 FOLLOWER]: Advancing to term 8
I20260129 02:59:46.484153 20515 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0f20af7fbe5d4db89a6f17948d98912c" candidate_uuid: "197b880fa39d432a9b472845876bfaf0" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "974b802141bd4f13beffef2aa03f88f9"
I20260129 02:59:46.492380 22343 consensus_queue.cc:260] T 87d5fe48a44a408aaf03e894a67ecb2f P 7a9daf5d42a84615a6383cfec9a10608 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } }
I20260129 02:59:46.385255 22671 raft_consensus.cc:740] T 00e79d8f25a34676b1d866eb2068a2b2 P 7a9daf5d42a84615a6383cfec9a10608 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 7a9daf5d42a84615a6383cfec9a10608, State: Running, Role: LEADER
I20260129 02:59:46.381903 22590 consensus_queue.cc:1048] T fd8311b779af4e1fa1402b2a02a602f2 P 7a9daf5d42a84615a6383cfec9a10608 [LEADER]: Connected to new peer: Peer: permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.096s
I20260129 02:59:46.380614 22445 raft_consensus.cc:515] T fcddab6bca684adf8ea032289a00f90e P 974b802141bd4f13beffef2aa03f88f9 [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:59:46.534301 20508 raft_consensus.cc:2393] T 17d6431cc77a491e8464d68db4081a51 P 974b802141bd4f13beffef2aa03f88f9 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 197b880fa39d432a9b472845876bfaf0 in current term 3: Already voted for candidate 974b802141bd4f13beffef2aa03f88f9 in this term.
W20260129 02:59:46.534909 20556 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.211.2:34517, user_credentials={real_user=slave}} blocked reactor thread for 46793.5us
I20260129 02:59:46.384985 22400 consensus_queue.cc:1048] T a0fcc58d6fb6473cafbd9d34d5195269 P 974b802141bd4f13beffef2aa03f88f9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.097s
I20260129 02:59:46.485370 21128 raft_consensus.cc:2393] T 78aa39096868415d9afc2ffcb248e198 P 7a9daf5d42a84615a6383cfec9a10608 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 197b880fa39d432a9b472845876bfaf0 in current term 4: Already voted for candidate 7a9daf5d42a84615a6383cfec9a10608 in this term.
I20260129 02:59:46.484037 20510 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a5a05fbcbbcf4916bbc01fd1a5b5e90d" candidate_uuid: "197b880fa39d432a9b472845876bfaf0" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "974b802141bd4f13beffef2aa03f88f9"
I20260129 02:59:46.487428 20513 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "17d6431cc77a491e8464d68db4081a51" candidate_uuid: "197b880fa39d432a9b472845876bfaf0" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "974b802141bd4f13beffef2aa03f88f9"
I20260129 02:59:46.484967 22184 raft_consensus.cc:493] T 4e420f90884f4796997b5ea41e49e411 P 974b802141bd4f13beffef2aa03f88f9 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 197b880fa39d432a9b472845876bfaf0)
I20260129 02:59:46.478814 22372 raft_consensus.cc:493] T 96609ac68bdf4993a12e1c23355ca013 P 7a9daf5d42a84615a6383cfec9a10608 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260129 02:59:46.567480 20556 leader_election.cc:304] T add8d4923d1a489e86882c3a25692b95 P 1c50de89a5a845668696dcb28c3badd6 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1c50de89a5a845668696dcb28c3badd6, 974b802141bd4f13beffef2aa03f88f9; no voters: 7a9daf5d42a84615a6383cfec9a10608
I20260129 02:59:46.551003 20516 raft_consensus.cc:3060] T 75be4bd6bd45474795fd472eff2953ac P 974b802141bd4f13beffef2aa03f88f9 [term 7 FOLLOWER]: Advancing to term 8
W20260129 02:59:46.551460 20407 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260129 02:59:46.570017 20407 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260129 02:59:46.386626 22403 raft_consensus.cc:515] T c934d31efd3f4e5195605ddc354d14e1 P 7a9daf5d42a84615a6383cfec9a10608 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } }
I20260129 02:59:46.555788 20434 raft_consensus.cc:1240] T 0f20af7fbe5d4db89a6f17948d98912c P 197b880fa39d432a9b472845876bfaf0 [term 5 FOLLOWER]: Rejecting Update request from peer 1c50de89a5a845668696dcb28c3badd6 for earlier term 4. Current term is 5. Ops: []
I20260129 02:59:46.418797 22699 raft_consensus.cc:2804] T 8055232cf0004d40a85f3bf983fe00a9 P 974b802141bd4f13beffef2aa03f88f9 [term 3 FOLLOWER]: Leader election won for term 3
I20260129 02:59:46.542021 22348 raft_consensus.cc:493] T 76f713c3331540d2a5b76164103e9eb1 P 1c50de89a5a845668696dcb28c3badd6 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260129 02:59:46.576489 20508 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "febffe4a8b394368b69b1d164308fb9a" candidate_uuid: "197b880fa39d432a9b472845876bfaf0" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "974b802141bd4f13beffef2aa03f88f9"
I20260129 02:59:46.581399 20508 raft_consensus.cc:3060] T febffe4a8b394368b69b1d164308fb9a P 974b802141bd4f13beffef2aa03f88f9 [term 2 FOLLOWER]: Advancing to term 3
W20260129 02:59:46.563242 20405 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260129 02:59:46.559500 22628 raft_consensus.cc:2749] T 17d6431cc77a491e8464d68db4081a51 P 974b802141bd4f13beffef2aa03f88f9 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260129 02:59:46.574304 22590 raft_consensus.cc:493] T 1299723a4476420ab02fd8bfd7aa0645 P 7a9daf5d42a84615a6383cfec9a10608 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260129 02:59:46.574128 22671 consensus_queue.cc:260] T 00e79d8f25a34676b1d866eb2068a2b2 P 7a9daf5d42a84615a6383cfec9a10608 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.7, Last appended by leader: 7, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } }
I20260129 02:59:46.575812 22445 leader_election.cc:290] T fcddab6bca684adf8ea032289a00f90e P 974b802141bd4f13beffef2aa03f88f9 [CANDIDATE]: Term 8 election: Requested vote from peers 197b880fa39d432a9b472845876bfaf0 (127.18.211.1:42027), 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3:37735)
I20260129 02:59:46.577008 20511 raft_consensus.cc:1240] T fcddab6bca684adf8ea032289a00f90e P 974b802141bd4f13beffef2aa03f88f9 [term 8 FOLLOWER]: Rejecting Update request from peer 197b880fa39d432a9b472845876bfaf0 for earlier term 7. Current term is 8. Ops: [7.5-7.5]
I20260129 02:59:46.542725 22694 raft_consensus.cc:2749] T 913fc7d751ed4b7388d889aac30a5f9b P 974b802141bd4f13beffef2aa03f88f9 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260129 02:59:46.537181 20515 raft_consensus.cc:3060] T 0f20af7fbe5d4db89a6f17948d98912c P 974b802141bd4f13beffef2aa03f88f9 [term 4 FOLLOWER]: Advancing to term 5
I20260129 02:59:46.476227 22700 consensus_queue.cc:1048] T a609ac02fae84725ab743f95f51d33ce P 974b802141bd4f13beffef2aa03f88f9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.085s
W20260129 02:59:46.580199 21100 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.211.3:37735, user_credentials={real_user=slave}} blocked reactor thread for 295594us
I20260129 02:59:46.571797 22372 raft_consensus.cc:3060] T 96609ac68bdf4993a12e1c23355ca013 P 7a9daf5d42a84615a6383cfec9a10608 [term 4 FOLLOWER]: Advancing to term 5
W20260129 02:59:46.600122 21100 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260129 02:59:46.596911 20434 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fcddab6bca684adf8ea032289a00f90e" candidate_uuid: "974b802141bd4f13beffef2aa03f88f9" candidate_term: 8 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "197b880fa39d432a9b472845876bfaf0"
I20260129 02:59:46.571978 22633 raft_consensus.cc:493] T 4e3805ffa70242e8ade66da260dddf66 P 197b880fa39d432a9b472845876bfaf0 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 1c50de89a5a845668696dcb28c3badd6)
I20260129 02:59:46.602766 20513 raft_consensus.cc:2393] T 17d6431cc77a491e8464d68db4081a51 P 974b802141bd4f13beffef2aa03f88f9 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 197b880fa39d432a9b472845876bfaf0 in current term 3: Already voted for candidate 974b802141bd4f13beffef2aa03f88f9 in this term.
I20260129 02:59:46.532617 22677 raft_consensus.cc:515] T b189f08e16a74a11980cc55d99359a0a P 197b880fa39d432a9b472845876bfaf0 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:59:46.646246 20588 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "78aa39096868415d9afc2ffcb248e198" candidate_uuid: "7a9daf5d42a84615a6383cfec9a10608" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "1c50de89a5a845668696dcb28c3badd6" is_pre_election: true
I20260129 02:59:46.599375 22343 raft_consensus.cc:3060] T 87d5fe48a44a408aaf03e894a67ecb2f P 7a9daf5d42a84615a6383cfec9a10608 [term 4 FOLLOWER]: Advancing to term 5
W20260129 02:59:46.657567 20405 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.211.4:40439, user_credentials={real_user=slave}} blocked reactor thread for 59919.9us
I20260129 02:59:46.587713 22699 raft_consensus.cc:697] T 8055232cf0004d40a85f3bf983fe00a9 P 974b802141bd4f13beffef2aa03f88f9 [term 3 LEADER]: Becoming Leader. State: Replica: 974b802141bd4f13beffef2aa03f88f9, State: Running, Role: LEADER
I20260129 02:59:46.647150 20588 raft_consensus.cc:2468] T 78aa39096868415d9afc2ffcb248e198 P 1c50de89a5a845668696dcb28c3badd6 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7a9daf5d42a84615a6383cfec9a10608 in term 4.
I20260129 02:59:46.725906 22654 raft_consensus.cc:493] T b189f08e16a74a11980cc55d99359a0a P 1c50de89a5a845668696dcb28c3badd6 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260129 02:59:46.588723 22184 raft_consensus.cc:515] T 4e420f90884f4796997b5ea41e49e411 P 974b802141bd4f13beffef2aa03f88f9 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } }
I20260129 02:59:46.709226 20405 leader_election.cc:400] T 18f30ebaa4174fd28c958e72d0cc7e24 P 197b880fa39d432a9b472845876bfaf0 [CANDIDATE]: Term 3 pre-election: Vote denied by peer 7a9daf5d42a84615a6383cfec9a10608 (127.18.211.4:40439) with higher term. Message: Invalid argument: T 18f30ebaa4174fd28c958e72d0cc7e24 P 7a9daf5d42a84615a6383cfec9a10608 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 197b880fa39d432a9b472845876bfaf0 for earlier term 3. Current term is 4.
I20260129 02:59:46.762756 20405 leader_election.cc:403] T 18f30ebaa4174fd28c958e72d0cc7e24 P 197b880fa39d432a9b472845876bfaf0 [CANDIDATE]: Term 3 pre-election: Cancelling election due to peer responding with higher term
W20260129 02:59:46.688504 20336 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:33836 (request call id 152) took 11697 ms (client timeout 15000 ms). Trace:
I20260129 02:59:46.730881 22715 raft_consensus.cc:2749] T 18f30ebaa4174fd28c958e72d0cc7e24 P 7a9daf5d42a84615a6383cfec9a10608 [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
W20260129 02:59:46.730743 20553 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.211.1:42027, user_credentials={real_user=slave}} blocked reactor thread for 164194us
I20260129 02:59:46.737792 20584 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35752 (request call id 1531) took 1392 ms. Trace:
I20260129 02:59:46.656284 22633 raft_consensus.cc:515] T 4e3805ffa70242e8ade66da260dddf66 P 197b880fa39d432a9b472845876bfaf0 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } }
I20260129 02:59:46.591143 22590 raft_consensus.cc:515] T 1299723a4476420ab02fd8bfd7aa0645 P 7a9daf5d42a84615a6383cfec9a10608 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } }
I20260129 02:59:46.588790 22348 raft_consensus.cc:515] T 76f713c3331540d2a5b76164103e9eb1 P 1c50de89a5a845668696dcb28c3badd6 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:59:46.649044 22671 raft_consensus.cc:3060] T 00e79d8f25a34676b1d866eb2068a2b2 P 7a9daf5d42a84615a6383cfec9a10608 [term 4 FOLLOWER]: Advancing to term 5
I20260129 02:59:46.767647 20515 raft_consensus.cc:2468] T 0f20af7fbe5d4db89a6f17948d98912c P 974b802141bd4f13beffef2aa03f88f9 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 197b880fa39d432a9b472845876bfaf0 in term 5.
I20260129 02:59:46.820430 20441 raft_consensus.cc:1275] T 42d27a28d89b409e8807ec0efd97af69 P 197b880fa39d432a9b472845876bfaf0 [term 5 FOLLOWER]: Refusing update from remote peer 7a9daf5d42a84615a6383cfec9a10608: Log matching property violated. Preceding OpId in replica: term: 4 index: 6. Preceding OpId from leader: term: 5 index: 7. (index mismatch)
I20260129 02:59:46.821264 20515 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "913fc7d751ed4b7388d889aac30a5f9b" candidate_uuid: "197b880fa39d432a9b472845876bfaf0" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "974b802141bd4f13beffef2aa03f88f9"
I20260129 02:59:46.821506 20407 leader_election.cc:304] T 0f20af7fbe5d4db89a6f17948d98912c P 197b880fa39d432a9b472845876bfaf0 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 197b880fa39d432a9b472845876bfaf0, 974b802141bd4f13beffef2aa03f88f9; no voters:
I20260129 02:59:46.822013 20515 raft_consensus.cc:2393] T 913fc7d751ed4b7388d889aac30a5f9b P 974b802141bd4f13beffef2aa03f88f9 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 197b880fa39d432a9b472845876bfaf0 in current term 3: Already voted for candidate 974b802141bd4f13beffef2aa03f88f9 in this term.
I20260129 02:59:46.821831 20441 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "78aa39096868415d9afc2ffcb248e198" candidate_uuid: "1c50de89a5a845668696dcb28c3badd6" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "197b880fa39d432a9b472845876bfaf0" is_pre_election: true
W20260129 02:59:46.786774 21099 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.211.1:42027, user_credentials={real_user=slave}} blocked reactor thread for 397685us
I20260129 02:59:46.798444 22320 raft_consensus.cc:493] T 4408006eb5174de1896b037cf3f18457 P 1c50de89a5a845668696dcb28c3badd6 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260129 02:59:46.802037 20556 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.211.2:34517, user_credentials={real_user=slave}} blocked reactor thread for 242610us
I20260129 02:59:46.736122 22708 raft_consensus.cc:2804] T add8d4923d1a489e86882c3a25692b95 P 1c50de89a5a845668696dcb28c3badd6 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260129 02:59:46.822788 20441 raft_consensus.cc:2468] T 78aa39096868415d9afc2ffcb248e198 P 197b880fa39d432a9b472845876bfaf0 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1c50de89a5a845668696dcb28c3badd6 in term 4.
I20260129 02:59:46.730999 22584 leader_election.cc:290] T 78aa39096868415d9afc2ffcb248e198 P 1c50de89a5a845668696dcb28c3badd6 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 197b880fa39d432a9b472845876bfaf0 (127.18.211.1:42027), 7a9daf5d42a84615a6383cfec9a10608 (127.18.211.4:40439)
I20260129 02:59:46.732991 22684 leader_election.cc:290] T 7d9494f5584d4d5db8b9adf1c8fc888b P 197b880fa39d432a9b472845876bfaf0 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 974b802141bd4f13beffef2aa03f88f9 (127.18.211.2:34517), 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3:37735)
I20260129 02:59:46.768644 20584 rpcz_store.cc:276] 0129 02:59:45.345031 (+ 0us) service_pool.cc:168] Inserting onto call queue
0129 02:59:45.349175 (+ 4144us) service_pool.cc:225] Handling call
0129 02:59:45.356198 (+ 7023us) raft_consensus.cc:1471] Updating replica for 0 ops
0129 02:59:45.356464 (+ 266us) raft_consensus.cc:1534] Early marking committed up to index 4
0129 02:59:46.292284 (+935820us) log.cc:844] Serialized 14 byte log entry
0129 02:59:46.737420 (+445136us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0129 02:59:46.737479 (+ 59us) raft_consensus.cc:1645] Marking committed up to 4
0129 02:59:46.737522 (+ 43us) raft_consensus.cc:1697] Filling consensus response to leader.
0129 02:59:46.737588 (+ 66us) raft_consensus.cc:1691] UpdateReplicas() finished
0129 02:59:46.737724 (+ 136us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":930162,"prepare.queue_time_us":951596,"prepare.run_cpu_time_us":51,"prepare.run_wall_time_us":52,"raft.queue_time_us":24067,"raft.run_cpu_time_us":128,"raft.run_wall_time_us":3185,"thread_start_us":404369,"threads_started":2,"wal-append.queue_time_us":408313}
I20260129 02:59:46.730248 22707 raft_consensus.cc:2804] T 368953c9102643b2a21f99af36b51905 P 974b802141bd4f13beffef2aa03f88f9 [term 5 FOLLOWER]: Leader election won for term 5
I20260129 02:59:46.720670 22641 consensus_queue.cc:1048] T 0678b8b468bd43cdbc3e485900844821 P 197b880fa39d432a9b472845876bfaf0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.080s
I20260129 02:59:46.928435 20556 leader_election.cc:304] T 8055232cf0004d40a85f3bf983fe00a9 P 1c50de89a5a845668696dcb28c3badd6 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1c50de89a5a845668696dcb28c3badd6; no voters: 197b880fa39d432a9b472845876bfaf0, 974b802141bd4f13beffef2aa03f88f9
I20260129 02:59:46.835876 20553 leader_election.cc:304] T 78aa39096868415d9afc2ffcb248e198 P 1c50de89a5a845668696dcb28c3badd6 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 197b880fa39d432a9b472845876bfaf0, 1c50de89a5a845668696dcb28c3badd6; no voters:
I20260129 02:59:46.824364 21099 leader_election.cc:304] T 78aa39096868415d9afc2ffcb248e198 P 7a9daf5d42a84615a6383cfec9a10608 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 197b880fa39d432a9b472845876bfaf0, 7a9daf5d42a84615a6383cfec9a10608; no voters:
W20260129 02:59:46.842818 20480 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.211.3:37735, user_credentials={real_user=slave}} blocked reactor thread for 341593us
W20260129 02:59:46.874552 20554 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260129 02:59:46.928588 21102 leader_election.cc:336] T a644c02b16af47679824ccd060ac10fd P 7a9daf5d42a84615a6383cfec9a10608 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 974b802141bd4f13beffef2aa03f88f9 (127.18.211.2:34517): Timed out: RequestConsensusVote RPC to 127.18.211.2:34517 timed out after 2.682s (SENT)
I20260129 02:59:46.929980 20480 leader_election.cc:304] T a8479d1fe0bd41e5946186e3a6ca8c7a P 974b802141bd4f13beffef2aa03f88f9 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 974b802141bd4f13beffef2aa03f88f9; no voters: 197b880fa39d432a9b472845876bfaf0, 1c50de89a5a845668696dcb28c3badd6
I20260129 02:59:46.930157 21102 leader_election.cc:304] T a644c02b16af47679824ccd060ac10fd P 7a9daf5d42a84615a6383cfec9a10608 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7a9daf5d42a84615a6383cfec9a10608; no voters: 1c50de89a5a845668696dcb28c3badd6, 974b802141bd4f13beffef2aa03f88f9
I20260129 02:59:46.931314 20480 leader_election.cc:400] T 0340229d521b432580ae926f6062e3f7 P 974b802141bd4f13beffef2aa03f88f9 [CANDIDATE]: Term 4 election: Vote denied by peer 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3:37735) with higher term. Message: Invalid argument: T 0340229d521b432580ae926f6062e3f7 P 1c50de89a5a845668696dcb28c3badd6 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 974b802141bd4f13beffef2aa03f88f9 for earlier term 4. Current term is 5.
I20260129 02:59:46.835422 20434 raft_consensus.cc:1240] T e4d18c35b0a7493e85d0a4b92956a48f P 197b880fa39d432a9b472845876bfaf0 [term 4 FOLLOWER]: Rejecting Update request from peer 1c50de89a5a845668696dcb28c3badd6 for earlier term 3. Current term is 4. Ops: []
I20260129 02:59:46.932403 20480 leader_election.cc:304] T 926a250fa5834e88b96ccecbefbf5086 P 974b802141bd4f13beffef2aa03f88f9 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1c50de89a5a845668696dcb28c3badd6, 974b802141bd4f13beffef2aa03f88f9; no voters: 197b880fa39d432a9b472845876bfaf0
I20260129 02:59:46.936172 20441 raft_consensus.cc:1240] T 3906b080a5b34a74b8b64128635e8997 P 197b880fa39d432a9b472845876bfaf0 [term 3 FOLLOWER]: Rejecting Update request from peer 1c50de89a5a845668696dcb28c3badd6 for earlier term 2. Current term is 3. Ops: []
I20260129 02:59:46.656872 22664 consensus_queue.cc:237] T e7d83d4ccf684e76acb29a0dc844e1b8 P 1c50de89a5a845668696dcb28c3badd6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
W20260129 02:59:46.942003 21099 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.211.1:42027, user_credentials={real_user=slave}} blocked reactor thread for 117749us
I20260129 02:59:46.942641 20515 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a4c1226b987b468fa84dfc6ba0e2dbe1" candidate_uuid: "197b880fa39d432a9b472845876bfaf0" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "974b802141bd4f13beffef2aa03f88f9" is_pre_election: true
I20260129 02:59:46.767369 22691 leader_election.cc:290] T c934d31efd3f4e5195605ddc354d14e1 P 197b880fa39d432a9b472845876bfaf0 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 974b802141bd4f13beffef2aa03f88f9 (127.18.211.2:34517), 7a9daf5d42a84615a6383cfec9a10608 (127.18.211.4:40439)
I20260129 02:59:46.947011 20515 raft_consensus.cc:2468] T a4c1226b987b468fa84dfc6ba0e2dbe1 P 974b802141bd4f13beffef2aa03f88f9 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 197b880fa39d432a9b472845876bfaf0 in term 3.
W20260129 02:59:46.951122 20553 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.211.1:42027, user_credentials={real_user=slave}} blocked reactor thread for 115339us
W20260129 02:59:46.953447 20553 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260129 02:59:46.834830 22646 raft_consensus.cc:493] T a1a39dcd328e4268ae41b988c66b4a75 P 197b880fa39d432a9b472845876bfaf0 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260129 02:59:46.834518 22694 raft_consensus.cc:493] T 75b0b6e82ddc4f92b1fe1d94f9a0b73b P 974b802141bd4f13beffef2aa03f88f9 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 7a9daf5d42a84615a6383cfec9a10608)
I20260129 02:59:46.950258 21100 leader_election.cc:304] T 4e3805ffa70242e8ade66da260dddf66 P 7a9daf5d42a84615a6383cfec9a10608 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7a9daf5d42a84615a6383cfec9a10608; no voters: 197b880fa39d432a9b472845876bfaf0, 1c50de89a5a845668696dcb28c3badd6
W20260129 02:59:46.960129 21101 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260129 02:59:46.741573 22656 consensus_queue.cc:1059] T 7a9a69cd82724b1faed1d10aa091fc4a P 1c50de89a5a845668696dcb28c3badd6 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 }, Status: INVALID_TERM, Last received: 4.3, Next index: 4, Last known committed idx: 2, Time since last communication: 0.286s
I20260129 02:59:46.839743 22708 raft_consensus.cc:493] T add8d4923d1a489e86882c3a25692b95 P 1c50de89a5a845668696dcb28c3badd6 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 7a9daf5d42a84615a6383cfec9a10608)
I20260129 02:59:46.839748 22231 raft_consensus.cc:2749] T 18f30ebaa4174fd28c958e72d0cc7e24 P 197b880fa39d432a9b472845876bfaf0 [term 4 FOLLOWER]: Leader pre-election lost for term 3. Reason: Vote denied by peer 7a9daf5d42a84615a6383cfec9a10608 (127.18.211.4:40439) with higher term. Message: Invalid argument: T 18f30ebaa4174fd28c958e72d0cc7e24 P 7a9daf5d42a84615a6383cfec9a10608 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 197b880fa39d432a9b472845876bfaf0 for earlier term 3. Current term is 4.
I20260129 02:59:46.975539 20589 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a4c1226b987b468fa84dfc6ba0e2dbe1" candidate_uuid: "974b802141bd4f13beffef2aa03f88f9" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "1c50de89a5a845668696dcb28c3badd6" is_pre_election: true
W20260129 02:59:46.979264 20405 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.211.4:40439, user_credentials={real_user=slave}} blocked reactor thread for 270168us
W20260129 02:59:46.954236 22672 consensus_meta.cc:287] T 3e2d15f8640b407195210c476e09df91 P 7a9daf5d42a84615a6383cfec9a10608: Time spent flushing consensus metadata: real 0.641s user 0.008s sys 0.003s
W20260129 02:59:47.035585 20407 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.211.2:34517, user_credentials={real_user=slave}} blocked reactor thread for 214153us
W20260129 02:59:47.011236 20480 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.211.3:37735, user_credentials={real_user=slave}} blocked reactor thread for 76965.6us
I20260129 02:59:46.828601 22504 consensus_queue.cc:1059] T 0f20af7fbe5d4db89a6f17948d98912c P 1c50de89a5a845668696dcb28c3badd6 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 }, Status: INVALID_TERM, Last received: 4.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.134s
I20260129 02:59:47.044831 21134 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "78aa39096868415d9afc2ffcb248e198" candidate_uuid: "1c50de89a5a845668696dcb28c3badd6" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "7a9daf5d42a84615a6383cfec9a10608" is_pre_election: true
I20260129 02:59:47.037026 20480 leader_election.cc:304] T 3906b080a5b34a74b8b64128635e8997 P 974b802141bd4f13beffef2aa03f88f9 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 974b802141bd4f13beffef2aa03f88f9; no voters: 197b880fa39d432a9b472845876bfaf0, 1c50de89a5a845668696dcb28c3badd6
W20260129 02:59:46.764683 20336 rpcz_store.cc:269] 0129 02:59:35.018157 (+ 0us) service_pool.cc:168] Inserting onto call queue
0129 02:59:35.030568 (+ 12411us) service_pool.cc:225] Handling call
0129 02:59:46.663006 (+11632438us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0129 02:59:44.862867 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0129 02:59:44.863546 (+ 679us) write_op.cc:432] Acquiring schema lock in shared mode
0129 02:59:44.863594 (+ 48us) write_op.cc:435] Acquired schema lock
0129 02:59:44.863612 (+ 18us) tablet.cc:660] Decoding operations
0129 02:59:44.905710 (+ 42098us) write_op.cc:620] Acquiring the partition lock for write op
0129 02:59:44.922162 (+ 16452us) write_op.cc:641] Partition lock acquired for write op
0129 02:59:44.923133 (+ 971us) tablet.cc:683] Acquiring locks for 3 operations
0129 02:59:44.929086 (+ 5953us) tablet.cc:699] Row locks acquired
0129 02:59:44.929116 (+ 30us) write_op.cc:260] PREPARE: finished
0129 02:59:44.929354 (+ 238us) write_op.cc:270] Start()
0129 02:59:44.929562 (+ 208us) write_op.cc:276] Timestamp: P: 1769655584929261 usec, L: 0
0129 02:59:44.929586 (+ 24us) op_driver.cc:348] REPLICATION: starting
0129 02:59:45.017887 (+ 88301us) log.cc:844] Serialized 1293 byte log entry
0129 02:59:45.881346 (+863459us) op_driver.cc:464] REPLICATION: finished
0129 02:59:46.260187 (+378841us) write_op.cc:301] APPLY: starting
0129 02:59:46.260279 (+ 92us) tablet.cc:1365] starting BulkCheckPresence
0129 02:59:46.260408 (+ 129us) tablet.cc:1368] finished BulkCheckPresence
0129 02:59:46.263876 (+ 3468us) tablet.cc:1370] starting ApplyRowOperation cycle
0129 02:59:46.264483 (+ 607us) tablet.cc:1381] finished ApplyRowOperation cycle
0129 02:59:46.264725 (+ 242us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=3
0129 02:59:46.264794 (+ 69us) write_op.cc:312] APPLY: finished
0129 02:59:46.265057 (+ 263us) log.cc:844] Serialized 35 byte log entry
0129 02:59:46.265245 (+ 188us) write_op.cc:489] Releasing partition, row and schema locks
0129 02:59:46.265361 (+ 116us) write_op.cc:454] Released schema lock
0129 02:59:46.265402 (+ 41us) write_op.cc:341] FINISH: Updating metrics
0129 02:59:46.265801 (+ 399us) write_op.cc:489] Releasing partition, row and schema locks
0129 02:59:46.265830 (+ 29us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":153215,"reported_tablets":7,"child_traces":[["op",{"apply.queue_time_us":373254,"apply.run_cpu_time_us":2558,"apply.run_wall_time_us":5583,"num_ops":3,"prepare.queue_time_us":311028,"prepare.run_cpu_time_us":12972,"prepare.run_wall_time_us":203012,"raft.queue_time_us":594379,"raft.run_cpu_time_us":9069,"raft.run_wall_time_us":200016,"replication_time_us":940677,"thread_start_us":668799,"threads_started":4,"wal-append.queue_time_us":98336}]]}
I20260129 02:59:46.784293 22654 raft_consensus.cc:515] T b189f08e16a74a11980cc55d99359a0a P 1c50de89a5a845668696dcb28c3badd6 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
W20260129 02:59:47.044374 21102 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.211.2:34517, user_credentials={real_user=slave}} blocked reactor thread for 115806us
I20260129 02:59:46.944657 22684 raft_consensus.cc:2804] T 0f20af7fbe5d4db89a6f17948d98912c P 197b880fa39d432a9b472845876bfaf0 [term 5 FOLLOWER]: Leader election won for term 5
I20260129 02:59:46.839995 22320 raft_consensus.cc:515] T 4408006eb5174de1896b037cf3f18457 P 1c50de89a5a845668696dcb28c3badd6 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:59:47.054183 20515 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c934d31efd3f4e5195605ddc354d14e1" candidate_uuid: "197b880fa39d432a9b472845876bfaf0" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "974b802141bd4f13beffef2aa03f88f9" is_pre_election: true
I20260129 02:59:47.061441 20515 raft_consensus.cc:2468] T c934d31efd3f4e5195605ddc354d14e1 P 974b802141bd4f13beffef2aa03f88f9 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 197b880fa39d432a9b472845876bfaf0 in term 4.
I20260129 02:59:46.968226 22344 raft_consensus.cc:2804] T 78aa39096868415d9afc2ffcb248e198 P 1c50de89a5a845668696dcb28c3badd6 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260129 02:59:47.052995 20407 leader_election.cc:304] T a4c1226b987b468fa84dfc6ba0e2dbe1 P 197b880fa39d432a9b472845876bfaf0 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 197b880fa39d432a9b472845876bfaf0, 974b802141bd4f13beffef2aa03f88f9; no voters:
I20260129 02:59:47.059221 20480 leader_election.cc:304] T 76f713c3331540d2a5b76164103e9eb1 P 974b802141bd4f13beffef2aa03f88f9 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 974b802141bd4f13beffef2aa03f88f9; no voters: 197b880fa39d432a9b472845876bfaf0, 1c50de89a5a845668696dcb28c3badd6
W20260129 02:59:47.063212 21100 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.211.3:37735, user_credentials={real_user=slave}} blocked reactor thread for 86295us
I20260129 02:59:47.063706 21129 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c934d31efd3f4e5195605ddc354d14e1" candidate_uuid: "197b880fa39d432a9b472845876bfaf0" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "7a9daf5d42a84615a6383cfec9a10608" is_pre_election: true
I20260129 02:59:47.065939 20480 leader_election.cc:304] T c2633a6fc34c44579de348d208b0c01e P 974b802141bd4f13beffef2aa03f88f9 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 974b802141bd4f13beffef2aa03f88f9; no voters: 197b880fa39d432a9b472845876bfaf0, 1c50de89a5a845668696dcb28c3badd6
I20260129 02:59:47.071944 21135 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "febffe4a8b394368b69b1d164308fb9a" candidate_uuid: "197b880fa39d432a9b472845876bfaf0" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "7a9daf5d42a84615a6383cfec9a10608"
I20260129 02:59:46.985381 22708 raft_consensus.cc:3060] T add8d4923d1a489e86882c3a25692b95 P 1c50de89a5a845668696dcb28c3badd6 [term 2 FOLLOWER]: Advancing to term 3
I20260129 02:59:46.978909 22638 raft_consensus.cc:493] T cb4727386a3e48aca5d71cc6c611c197 P 1c50de89a5a845668696dcb28c3badd6 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260129 02:59:47.073272 20516 consensus_meta.cc:287] T 75be4bd6bd45474795fd472eff2953ac P 974b802141bd4f13beffef2aa03f88f9: Time spent flushing consensus metadata: real 0.504s user 0.008s sys 0.000s
I20260129 02:59:46.986246 22197 raft_consensus.cc:2804] T 926a250fa5834e88b96ccecbefbf5086 P 974b802141bd4f13beffef2aa03f88f9 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260129 02:59:47.080837 20432 raft_consensus.cc:1275] T 18f30ebaa4174fd28c958e72d0cc7e24 P 197b880fa39d432a9b472845876bfaf0 [term 4 FOLLOWER]: Refusing update from remote peer 1c50de89a5a845668696dcb28c3badd6: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20260129 02:59:47.047331 22644 raft_consensus.cc:493] T 3ccac82a01a2445fa5b0ce6c0fcf2295 P 197b880fa39d432a9b472845876bfaf0 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260129 02:59:47.079763 20516 raft_consensus.cc:2468] T 75be4bd6bd45474795fd472eff2953ac P 974b802141bd4f13beffef2aa03f88f9 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 197b880fa39d432a9b472845876bfaf0 in term 8.
I20260129 02:59:46.981168 22403 leader_election.cc:290] T c934d31efd3f4e5195605ddc354d14e1 P 7a9daf5d42a84615a6383cfec9a10608 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 197b880fa39d432a9b472845876bfaf0 (127.18.211.1:42027), 974b802141bd4f13beffef2aa03f88f9 (127.18.211.2:34517)
I20260129 02:59:47.079463 20589 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:35752 (request call id 1544) took 1161 ms. Trace:
I20260129 02:59:46.988529 22681 leader_election.cc:290] T dc2f4cb752cf4d59b5f6bc689f5d8104 P 7a9daf5d42a84615a6383cfec9a10608 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 197b880fa39d432a9b472845876bfaf0 (127.18.211.1:42027), 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3:37735)
I20260129 02:59:46.980114 22646 raft_consensus.cc:515] T a1a39dcd328e4268ae41b988c66b4a75 P 197b880fa39d432a9b472845876bfaf0 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } } peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } }
I20260129 02:59:46.979676 22694 raft_consensus.cc:515] T 75b0b6e82ddc4f92b1fe1d94f9a0b73b P 974b802141bd4f13beffef2aa03f88f9 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } }
I20260129 02:59:46.840260 22699 consensus_queue.cc:237] T 8055232cf0004d40a85f3bf983fe00a9 P 974b802141bd4f13beffef2aa03f88f9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:59:47.054562 22672 raft_consensus.cc:515] T 3e2d15f8640b407195210c476e09df91 P 7a9daf5d42a84615a6383cfec9a10608 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } }
I20260129 02:59:47.374830 20587 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a1a39dcd328e4268ae41b988c66b4a75" candidate_uuid: "974b802141bd4f13beffef2aa03f88f9" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "1c50de89a5a845668696dcb28c3badd6" is_pre_election: true
I20260129 02:59:47.072055 22344 raft_consensus.cc:493] T 78aa39096868415d9afc2ffcb248e198 P 1c50de89a5a845668696dcb28c3badd6 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260129 02:59:47.064239 22400 raft_consensus.cc:2749] T 3906b080a5b34a74b8b64128635e8997 P 974b802141bd4f13beffef2aa03f88f9 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
W20260129 02:59:47.459355 20553 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.211.1:42027, user_credentials={real_user=slave}} blocked reactor thread for 395341us
W20260129 02:59:47.598585 20434 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.519s user 0.001s sys 0.017s
I20260129 02:59:47.086304 20515 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7d9494f5584d4d5db8b9adf1c8fc888b" candidate_uuid: "197b880fa39d432a9b472845876bfaf0" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "974b802141bd4f13beffef2aa03f88f9" is_pre_election: true
W20260129 02:59:47.618232 20508 consensus_meta.cc:287] T febffe4a8b394368b69b1d164308fb9a P 974b802141bd4f13beffef2aa03f88f9: Time spent flushing consensus metadata: real 1.029s user 0.009s sys 0.001s
I20260129 02:59:47.594808 20441 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "dc2f4cb752cf4d59b5f6bc689f5d8104" candidate_uuid: "7a9daf5d42a84615a6383cfec9a10608" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "197b880fa39d432a9b472845876bfaf0" is_pre_election: true
I20260129 02:59:47.621726 20508 raft_consensus.cc:2468] T febffe4a8b394368b69b1d164308fb9a P 974b802141bd4f13beffef2aa03f88f9 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 197b880fa39d432a9b472845876bfaf0 in term 3.
W20260129 02:59:47.428037 20405 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260129 02:59:47.065508 22700 raft_consensus.cc:493] T e7d83d4ccf684e76acb29a0dc844e1b8 P 974b802141bd4f13beffef2aa03f88f9 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260129 02:59:47.595461 20556 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.211.2:34517, user_credentials={real_user=slave}} blocked reactor thread for 755029us
I20260129 02:59:47.594856 20587 raft_consensus.cc:2468] T a1a39dcd328e4268ae41b988c66b4a75 P 1c50de89a5a845668696dcb28c3badd6 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 974b802141bd4f13beffef2aa03f88f9 in term 3.
W20260129 02:59:47.294701 20479 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.211.1:42027, user_credentials={real_user=slave}} blocked reactor thread for 218744us
I20260129 02:59:47.622378 20508 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:57600 (request call id 1754) took 1085 ms. Trace:
W20260129 02:59:47.249286 20554 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.211.4:40439, user_credentials={real_user=slave}} blocked reactor thread for 213114us
W20260129 02:59:47.294523 21100 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.211.3:37735, user_credentials={real_user=slave}} blocked reactor thread for 227545us
I20260129 02:59:47.622275 20589 rpcz_store.cc:276] 0129 02:59:45.911312 (+ 0us) service_pool.cc:168] Inserting onto call queue
0129 02:59:45.911450 (+ 138us) service_pool.cc:225] Handling call
0129 02:59:46.975420 (+1063970us) spinlock_profiling.cc:243] Waited 6.57 ms on lock 0x7b58001cb398. stack: 00007f8fe044e373 00007f8fe044e270 00007f8fdfda0574 00007f8fe9a5c6b4 00007f8fe9a5c649 00007f8fe02c08b1 00007f8fe0442db1 00007f8fe044312e 000000000035a3c9 00007f8fe846d560 00007f8fe8535dd4 00007f8fe8535e41 00007f8fe84e3bd0 00007f8fe84e34b4 00007f8fe1a73834 00007f8fe1a737c2
0129 02:59:47.072323 (+ 96903us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":15119616}
W20260129 02:59:47.614912 21101 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260129 02:59:47.584638 22553 consensus_meta.cc:287] T 368953c9102643b2a21f99af36b51905 P 1c50de89a5a845668696dcb28c3badd6: Time spent flushing consensus metadata: real 1.268s user 0.008s sys 0.003s
W20260129 02:59:47.588238 22713 log.cc:927] Time spent T 4f5a77e94fef4b16a2ba7b067c8b144a P 7a9daf5d42a84615a6383cfec9a10608: Append to log took a long time: real 0.729s user 0.000s sys 0.001s
I20260129 02:59:47.707443 20440 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c934d31efd3f4e5195605ddc354d14e1" candidate_uuid: "7a9daf5d42a84615a6383cfec9a10608" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "197b880fa39d432a9b472845876bfaf0" is_pre_election: true
I20260129 02:59:47.727595 20440 raft_consensus.cc:2468] T c934d31efd3f4e5195605ddc354d14e1 P 197b880fa39d432a9b472845876bfaf0 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7a9daf5d42a84615a6383cfec9a10608 in term 4.
W20260129 02:59:47.709175 20587 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:35752 (request call id 1543) took 1798 ms (client timeout 2060 ms). Trace:
I20260129 02:59:47.710455 20508 rpcz_store.cc:276] 0129 02:59:46.537407 (+ 0us) service_pool.cc:168] Inserting onto call queue
0129 02:59:46.559126 (+ 21719us) service_pool.cc:225] Handling call
0129 02:59:47.622324 (+1063198us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20260129 02:59:47.519196 20510 consensus_meta.cc:287] T a5a05fbcbbcf4916bbc01fd1a5b5e90d P 974b802141bd4f13beffef2aa03f88f9: Time spent flushing consensus metadata: real 0.959s user 0.011s sys 0.000s
I20260129 02:59:47.073015 22677 leader_election.cc:290] T b189f08e16a74a11980cc55d99359a0a P 197b880fa39d432a9b472845876bfaf0 [CANDIDATE]: Term 6 election: Requested vote from peers 974b802141bd4f13beffef2aa03f88f9 (127.18.211.2:34517), 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3:37735)
I20260129 02:59:47.730732 20515 raft_consensus.cc:2468] T 7d9494f5584d4d5db8b9adf1c8fc888b P 974b802141bd4f13beffef2aa03f88f9 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 197b880fa39d432a9b472845876bfaf0 in term 2.
I20260129 02:59:47.733458 20510 raft_consensus.cc:2468] T a5a05fbcbbcf4916bbc01fd1a5b5e90d P 974b802141bd4f13beffef2aa03f88f9 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 197b880fa39d432a9b472845876bfaf0 in term 6.
I20260129 02:59:47.733645 20515 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:57600 (request call id 1756) took 1077 ms. Trace:
W20260129 02:59:47.731308 20434 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.651s user 0.003s sys 0.024s
I20260129 02:59:47.734117 20510 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:57600 (request call id 1738) took 1320 ms. Trace:
W20260129 02:59:47.181977 22711 log.cc:927] Time spent T a0fcc58d6fb6473cafbd9d34d5195269 P 197b880fa39d432a9b472845876bfaf0: Append to log took a long time: real 0.220s user 0.000s sys 0.001s
W20260129 02:59:47.286021 22710 log.cc:927] Time spent T fd8311b779af4e1fa1402b2a02a602f2 P 1c50de89a5a845668696dcb28c3badd6: Append to log took a long time: real 0.427s user 0.001s sys 0.000s
W20260129 02:59:47.289444 22717 log.cc:927] Time spent T 2e58e9a5a8aa4cecafb464addb4f35ed P 1c50de89a5a845668696dcb28c3badd6: Append to log took a long time: real 0.313s user 0.000s sys 0.000s
W20260129 02:59:47.372270 22733 log.cc:927] Time spent T e33a417cf9c54cf5af73a4ee3aac4d3a P 1c50de89a5a845668696dcb28c3badd6: Append to log took a long time: real 0.093s user 0.000s sys 0.001s
I20260129 02:59:47.734174 20515 rpcz_store.cc:276] 0129 02:59:46.655781 (+ 0us) service_pool.cc:168] Inserting onto call queue
0129 02:59:47.062197 (+406416us) service_pool.cc:225] Handling call
0129 02:59:47.733580 (+671383us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260129 02:59:47.734544 20510 rpcz_store.cc:276] 0129 02:59:46.413699 (+ 0us) service_pool.cc:168] Inserting onto call queue
0129 02:59:46.413878 (+ 179us) service_pool.cc:225] Handling call
0129 02:59:47.734059 (+1320181us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260129 02:59:47.802242 20588 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1299723a4476420ab02fd8bfd7aa0645" candidate_uuid: "7a9daf5d42a84615a6383cfec9a10608" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "1c50de89a5a845668696dcb28c3badd6" is_pre_election: true
W20260129 02:59:47.287935 22709 log.cc:927] Time spent T 4b1876f9a70b49edb3aa74105e66d3b3 P 1c50de89a5a845668696dcb28c3badd6: Append to log took a long time: real 0.468s user 0.000s sys 0.000s
I20260129 02:59:47.251588 22684 raft_consensus.cc:697] T 0f20af7fbe5d4db89a6f17948d98912c P 197b880fa39d432a9b472845876bfaf0 [term 5 LEADER]: Becoming Leader. State: Replica: 197b880fa39d432a9b472845876bfaf0, State: Running, Role: LEADER
W20260129 02:59:47.729475 20587 rpcz_store.cc:269] 0129 02:59:45.910954 (+ 0us) service_pool.cc:168] Inserting onto call queue
0129 02:59:45.926195 (+ 15241us) service_pool.cc:225] Handling call
0129 02:59:47.374651 (+1448456us) spinlock_profiling.cc:243] Waited 8.84 ms on lock 0x7b58001cb398. stack: 00007f8fe044e373 00007f8fe044e270 00007f8fdfda0574 00007f8fe9a5c6b4 00007f8fe9a5c649 00007f8fe02c08b1 00007f8fe0442db1 00007f8fe0442c62 00007f8fe92eddd9 00007f8fe846d260 00007f8fe8535dbb 00007f8fe8535e41 00007f8fe84e3bd0 00007f8fe84e34b4 00007f8fe1a73834 00007f8fe1a737c2
0129 02:59:47.709117 (+334466us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":20341120}
I20260129 02:59:47.784958 20508 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "cb4727386a3e48aca5d71cc6c611c197" candidate_uuid: "197b880fa39d432a9b472845876bfaf0" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "974b802141bd4f13beffef2aa03f88f9"
I20260129 02:59:47.286365 22658 raft_consensus.cc:493] T 4e420f90884f4796997b5ea41e49e411 P 7a9daf5d42a84615a6383cfec9a10608 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 197b880fa39d432a9b472845876bfaf0)
I20260129 02:59:47.562333 22641 raft_consensus.cc:2804] T a4c1226b987b468fa84dfc6ba0e2dbe1 P 197b880fa39d432a9b472845876bfaf0 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260129 02:59:47.828063 22641 raft_consensus.cc:493] T a4c1226b987b468fa84dfc6ba0e2dbe1 P 197b880fa39d432a9b472845876bfaf0 [term 3 FOLLOWER]: Starting leader election (detected failure of leader 1c50de89a5a845668696dcb28c3badd6)
I20260129 02:59:47.828641 22641 raft_consensus.cc:3060] T a4c1226b987b468fa84dfc6ba0e2dbe1 P 197b880fa39d432a9b472845876bfaf0 [term 3 FOLLOWER]: Advancing to term 4
I20260129 02:59:47.787932 22700 raft_consensus.cc:515] T e7d83d4ccf684e76acb29a0dc844e1b8 P 974b802141bd4f13beffef2aa03f88f9 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:59:47.796829 22320 leader_election.cc:290] T 4408006eb5174de1896b037cf3f18457 P 1c50de89a5a845668696dcb28c3badd6 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 197b880fa39d432a9b472845876bfaf0 (127.18.211.1:42027), 974b802141bd4f13beffef2aa03f88f9 (127.18.211.2:34517)
W20260129 02:59:47.831694 22652 consensus_meta.cc:287] T 75b0b6e82ddc4f92b1fe1d94f9a0b73b P 1c50de89a5a845668696dcb28c3badd6: Time spent flushing consensus metadata: real 1.946s user 0.010s sys 0.006s
I20260129 02:59:47.832248 22652 raft_consensus.cc:515] T 75b0b6e82ddc4f92b1fe1d94f9a0b73b P 1c50de89a5a845668696dcb28c3badd6 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } }
I20260129 02:59:47.833505 20582 raft_consensus.cc:1240] T 75b0b6e82ddc4f92b1fe1d94f9a0b73b P 1c50de89a5a845668696dcb28c3badd6 [term 3 FOLLOWER]: Rejecting Update request from peer 7a9daf5d42a84615a6383cfec9a10608 for earlier term 2. Current term is 3. Ops: []
W20260129 02:59:47.798041 22637 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 1.218s user 0.043s sys 0.039s
I20260129 02:59:47.285744 22724 raft_consensus.cc:2749] T 8055232cf0004d40a85f3bf983fe00a9 P 1c50de89a5a845668696dcb28c3badd6 [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20260129 02:59:47.742220 20554 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260129 02:59:47.562137 22738 raft_consensus.cc:493] T 4f5a77e94fef4b16a2ba7b067c8b144a P 974b802141bd4f13beffef2aa03f88f9 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260129 02:59:47.287254 22716 log.cc:927] Time spent T 42d27a28d89b409e8807ec0efd97af69 P 1c50de89a5a845668696dcb28c3badd6: Append to log took a long time: real 0.447s user 0.000s sys 0.001s
W20260129 02:59:47.148061 22680 consensus_meta.cc:287] T 96609ac68bdf4993a12e1c23355ca013 P 974b802141bd4f13beffef2aa03f88f9: Time spent flushing consensus metadata: real 1.272s user 0.011s sys 0.000s
W20260129 02:59:47.251710 22706 log.cc:927] Time spent T 87d5fe48a44a408aaf03e894a67ecb2f P 197b880fa39d432a9b472845876bfaf0: Append to log took a long time: real 0.086s user 0.000s sys 0.001s
I20260129 02:59:47.879354 22746 raft_consensus.cc:493] T e7d83d4ccf684e76acb29a0dc844e1b8 P 197b880fa39d432a9b472845876bfaf0 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260129 02:59:47.359891 22445 raft_consensus.cc:3060] T 76f713c3331540d2a5b76164103e9eb1 P 974b802141bd4f13beffef2aa03f88f9 [term 3 FOLLOWER]: Advancing to term 4
I20260129 02:59:47.880154 22746 raft_consensus.cc:515] T e7d83d4ccf684e76acb29a0dc844e1b8 P 197b880fa39d432a9b472845876bfaf0 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:59:47.373782 22628 raft_consensus.cc:2749] T c2633a6fc34c44579de348d208b0c01e P 974b802141bd4f13beffef2aa03f88f9 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260129 02:59:47.371359 22742 raft_consensus.cc:2749] T 4e3805ffa70242e8ade66da260dddf66 P 7a9daf5d42a84615a6383cfec9a10608 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
W20260129 02:59:47.288707 22712 log.cc:927] Time spent T 4f5a77e94fef4b16a2ba7b067c8b144a P 1c50de89a5a845668696dcb28c3badd6: Append to log took a long time: real 0.233s user 0.000s sys 0.001s
I20260129 02:59:47.478536 22184 leader_election.cc:290] T 4e420f90884f4796997b5ea41e49e411 P 974b802141bd4f13beffef2aa03f88f9 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 197b880fa39d432a9b472845876bfaf0 (127.18.211.1:42027), 7a9daf5d42a84615a6383cfec9a10608 (127.18.211.4:40439)
I20260129 02:59:47.616183 22344 raft_consensus.cc:3060] T 78aa39096868415d9afc2ffcb248e198 P 1c50de89a5a845668696dcb28c3badd6 [term 4 FOLLOWER]: Advancing to term 5
W20260129 02:59:47.905722 21128 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34060 (request call id 475) took 1998 ms (client timeout 2007 ms). Trace:
W20260129 02:59:47.906229 21128 rpcz_store.cc:269] 0129 02:59:45.907678 (+ 0us) service_pool.cc:168] Inserting onto call queue
0129 02:59:45.908506 (+ 828us) service_pool.cc:225] Handling call
0129 02:59:47.905435 (+1996929us) spinlock_profiling.cc:243] Waited 91 ms on lock 0x7b540068e2d4. stack: 00007f8fe044e373 00007f8fe044e270 00007f8fdfda0574 00007f8fe9a5c6b4 00007f8fe9a5c649 00007f8fe34975a1 00007f8fe3531f7a 00007f8fe84e3647 00007f8fe1a73834 00007f8fe1a737c2 00007f8fe1a736f1 00007f8fe1a7366c 00007f8fe1a72922 00007f8fe186d34c 00007f8fe186c786 00007f8fe186c12f
0129 02:59:47.905619 (+ 184us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":209402752}
I20260129 02:59:47.460037 22348 leader_election.cc:290] T 76f713c3331540d2a5b76164103e9eb1 P 1c50de89a5a845668696dcb28c3badd6 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 197b880fa39d432a9b472845876bfaf0 (127.18.211.1:42027), 974b802141bd4f13beffef2aa03f88f9 (127.18.211.2:34517)
I20260129 02:59:47.501659 22633 leader_election.cc:290] T 4e3805ffa70242e8ade66da260dddf66 P 197b880fa39d432a9b472845876bfaf0 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3:37735), 7a9daf5d42a84615a6383cfec9a10608 (127.18.211.4:40439)
I20260129 02:59:47.495602 22590 leader_election.cc:290] T 1299723a4476420ab02fd8bfd7aa0645 P 7a9daf5d42a84615a6383cfec9a10608 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 197b880fa39d432a9b472845876bfaf0 (127.18.211.1:42027), 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3:37735)
W20260129 02:59:47.707667 22707 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.587s user 0.004s sys 0.006s
W20260129 02:59:47.908617 22707 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.923s user 0.004s sys 0.006s
I20260129 02:59:47.619159 20509 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7a9a69cd82724b1faed1d10aa091fc4a" candidate_uuid: "197b880fa39d432a9b472845876bfaf0" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "974b802141bd4f13beffef2aa03f88f9"
I20260129 02:59:47.908900 22707 raft_consensus.cc:697] T 368953c9102643b2a21f99af36b51905 P 974b802141bd4f13beffef2aa03f88f9 [term 5 LEADER]: Becoming Leader. State: Replica: 974b802141bd4f13beffef2aa03f88f9, State: Running, Role: LEADER
I20260129 02:59:47.909271 20509 raft_consensus.cc:3060] T 7a9a69cd82724b1faed1d10aa091fc4a P 974b802141bd4f13beffef2aa03f88f9 [term 4 FOLLOWER]: Advancing to term 5
I20260129 02:59:47.909838 22707 consensus_queue.cc:237] T 368953c9102643b2a21f99af36b51905 P 974b802141bd4f13beffef2aa03f88f9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 4.3, Last appended by leader: 3, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:59:47.274585 22715 raft_consensus.cc:2749] T a644c02b16af47679824ccd060ac10fd P 7a9daf5d42a84615a6383cfec9a10608 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260129 02:59:47.085575 22197 raft_consensus.cc:493] T 926a250fa5834e88b96ccecbefbf5086 P 974b802141bd4f13beffef2aa03f88f9 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260129 02:59:47.912066 22197 raft_consensus.cc:3060] T 926a250fa5834e88b96ccecbefbf5086 P 974b802141bd4f13beffef2aa03f88f9 [term 5 FOLLOWER]: Advancing to term 6
I20260129 02:59:47.912391 20438 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55676 (request call id 1232) took 2563 ms. Trace:
I20260129 02:59:47.912323 22684 consensus_queue.cc:237] T 0f20af7fbe5d4db89a6f17948d98912c P 197b880fa39d432a9b472845876bfaf0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:59:47.912909 20438 rpcz_store.cc:276] 0129 02:59:45.348908 (+ 0us) service_pool.cc:168] Inserting onto call queue
0129 02:59:45.378232 (+ 29324us) service_pool.cc:225] Handling call
0129 02:59:45.378602 (+ 370us) raft_consensus.cc:1471] Updating replica for 0 ops
0129 02:59:45.870552 (+491950us) raft_consensus.cc:1697] Filling consensus response to leader.
0129 02:59:47.912147 (+2041595us) spinlock_profiling.cc:243] Waited 491 ms on lock 0x7b54002fce54. stack: 00007f8fe044e373 00007f8fe044e270 00007f8fdfda0574 00007f8fe9a5c6b4 00007f8fe9a5c649 00007f8fe34975a1 00007f8fe352ea81 00007f8fe352d4be 00007f8fe84e1f49 00007f8fe1a6ed34 00007f8fe1a6ecc2 00007f8fe1a6ebf1 00007f8fe1a6eb6c 00007f8fe1a6de22 00007f8fe186d34c 00007f8fe186c786
0129 02:59:47.912329 (+ 182us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1129720576}
I20260129 02:59:47.914297 20438 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e7d83d4ccf684e76acb29a0dc844e1b8" candidate_uuid: "974b802141bd4f13beffef2aa03f88f9" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "197b880fa39d432a9b472845876bfaf0" is_pre_election: true
I20260129 02:59:47.915050 20438 raft_consensus.cc:2393] T e7d83d4ccf684e76acb29a0dc844e1b8 P 197b880fa39d432a9b472845876bfaf0 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 974b802141bd4f13beffef2aa03f88f9 in current term 4: Already voted for candidate 1c50de89a5a845668696dcb28c3badd6 in this term.
I20260129 02:59:47.802959 20588 raft_consensus.cc:2468] T 1299723a4476420ab02fd8bfd7aa0645 P 1c50de89a5a845668696dcb28c3badd6 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7a9daf5d42a84615a6383cfec9a10608 in term 7.
I20260129 02:59:47.882730 22739 raft_consensus.cc:493] T 368953c9102643b2a21f99af36b51905 P 197b880fa39d432a9b472845876bfaf0 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260129 02:59:47.804265 21127 raft_consensus.cc:1240] T 3e2d15f8640b407195210c476e09df91 P 7a9daf5d42a84615a6383cfec9a10608 [term 2 FOLLOWER]: Rejecting Update request from peer 1c50de89a5a845668696dcb28c3badd6 for earlier term 1. Current term is 2. Ops: []
I20260129 02:59:47.916589 20513 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:57600 (request call id 1744) took 1500 ms. Trace:
I20260129 02:59:47.917042 20513 rpcz_store.cc:276] 0129 02:59:46.415779 (+ 0us) service_pool.cc:168] Inserting onto call queue
0129 02:59:46.416348 (+ 569us) service_pool.cc:225] Handling call
0129 02:59:47.916375 (+1500027us) spinlock_profiling.cc:243] Waited 37.1 ms on lock 0x7b54001ae854. stack: 00007f8fe044e373 00007f8fe044e270 00007f8fdfda0574 00007f8fe9a5c6b4 00007f8fe9a5c649 00007f8fe34975a1 00007f8fe3531f7a 00007f8fe84e3647 00007f8fe1a73834 00007f8fe1a737c2 00007f8fe1a736f1 00007f8fe1a7366c 00007f8fe1a72922 00007f8fe186d34c 00007f8fe186c786 00007f8fe186c12f
0129 02:59:47.916524 (+ 149us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":85223680}
I20260129 02:59:47.804709 20436 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "76f713c3331540d2a5b76164103e9eb1" candidate_uuid: "1c50de89a5a845668696dcb28c3badd6" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "197b880fa39d432a9b472845876bfaf0" is_pre_election: true
I20260129 02:59:47.723944 22728 consensus_queue.cc:1059] T 3906b080a5b34a74b8b64128635e8997 P 1c50de89a5a845668696dcb28c3badd6 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 }, Status: INVALID_TERM, Last received: 2.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.095s
I20260129 02:59:47.729233 22644 raft_consensus.cc:515] T 3ccac82a01a2445fa5b0ce6c0fcf2295 P 197b880fa39d432a9b472845876bfaf0 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } }
I20260129 02:59:47.804347 20584 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "dc2f4cb752cf4d59b5f6bc689f5d8104" candidate_uuid: "7a9daf5d42a84615a6383cfec9a10608" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "1c50de89a5a845668696dcb28c3badd6" is_pre_election: true
W20260129 02:59:47.808594 22723 log.cc:927] Time spent T 2e58e9a5a8aa4cecafb464addb4f35ed P 197b880fa39d432a9b472845876bfaf0: Append to log took a long time: real 0.318s user 0.000s sys 0.000s
I20260129 02:59:47.920290 22744 raft_consensus.cc:493] T 42d27a28d89b409e8807ec0efd97af69 P 1c50de89a5a845668696dcb28c3badd6 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 7a9daf5d42a84615a6383cfec9a10608)
W20260129 02:59:47.808728 22554 consensus_meta.cc:287] T 75be4bd6bd45474795fd472eff2953ac P 1c50de89a5a845668696dcb28c3badd6: Time spent flushing consensus metadata: real 1.251s user 0.006s sys 0.004s
I20260129 02:59:47.921104 22672 leader_election.cc:290] T 3e2d15f8640b407195210c476e09df91 P 7a9daf5d42a84615a6383cfec9a10608 [CANDIDATE]: Term 2 election: Requested vote from peers 974b802141bd4f13beffef2aa03f88f9 (127.18.211.2:34517), 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3:37735)
I20260129 02:59:47.920883 22744 raft_consensus.cc:515] T 42d27a28d89b409e8807ec0efd97af69 P 1c50de89a5a845668696dcb28c3badd6 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } }
I20260129 02:59:47.826054 22400 raft_consensus.cc:493] T 75f45538dc704d9a8ec4ff8ad9f3ea5b P 974b802141bd4f13beffef2aa03f88f9 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 7a9daf5d42a84615a6383cfec9a10608)
I20260129 02:59:47.921737 22554 consensus_queue.cc:1059] T 505580c93c694bbb8ea96fdf9b17ab46 P 1c50de89a5a845668696dcb28c3badd6 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 }, Status: INVALID_TERM, Last received: 3.6, Next index: 7, Last known committed idx: 5, Time since last communication: 0.000s
I20260129 02:59:47.922349 22400 raft_consensus.cc:515] T 75f45538dc704d9a8ec4ff8ad9f3ea5b P 974b802141bd4f13beffef2aa03f88f9 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } }
I20260129 02:59:47.922868 22554 raft_consensus.cc:3055] T 505580c93c694bbb8ea96fdf9b17ab46 P 1c50de89a5a845668696dcb28c3badd6 [term 3 LEADER]: Stepping down as leader of term 3
I20260129 02:59:47.923061 22744 leader_election.cc:290] T 42d27a28d89b409e8807ec0efd97af69 P 1c50de89a5a845668696dcb28c3badd6 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 197b880fa39d432a9b472845876bfaf0 (127.18.211.1:42027), 7a9daf5d42a84615a6383cfec9a10608 (127.18.211.4:40439)
I20260129 02:59:47.923312 22554 raft_consensus.cc:740] T 505580c93c694bbb8ea96fdf9b17ab46 P 1c50de89a5a845668696dcb28c3badd6 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 1c50de89a5a845668696dcb28c3badd6, State: Running, Role: LEADER
I20260129 02:59:47.879458 22680 raft_consensus.cc:515] T 96609ac68bdf4993a12e1c23355ca013 P 974b802141bd4f13beffef2aa03f88f9 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } attrs { replace: true } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } }
I20260129 02:59:47.925050 20435 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:48356 (request call id 291) took 1549 ms. Trace:
I20260129 02:59:47.926126 21132 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34046 (request call id 445) took 2379 ms. Trace:
I20260129 02:59:47.925410 20435 rpcz_store.cc:276] 0129 02:59:46.375904 (+ 0us) service_pool.cc:168] Inserting onto call queue
0129 02:59:46.376028 (+ 124us) service_pool.cc:225] Handling call
0129 02:59:46.376344 (+ 316us) raft_consensus.cc:1471] Updating replica for 1 ops
0129 02:59:46.411559 (+ 35215us) raft_consensus.cc:1534] Early marking committed up to index 0
0129 02:59:46.411592 (+ 33us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0129 02:59:46.465578 (+ 53986us) log.cc:844] Serialized 26 byte log entry
0129 02:59:46.820420 (+354842us) raft_consensus.cc:1645] Marking committed up to 6
0129 02:59:46.935170 (+114750us) log.cc:844] Serialized 14 byte log entry
0129 02:59:47.046484 (+111314us) raft_consensus.cc:1697] Filling consensus response to leader.
0129 02:59:47.046561 (+ 77us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0129 02:59:47.924842 (+878281us) raft_consensus.cc:1685] finished
0129 02:59:47.924865 (+ 23us) raft_consensus.cc:1691] UpdateReplicas() finished
0129 02:59:47.924994 (+ 129us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":4082,"prepare.queue_time_us":114546,"prepare.run_cpu_time_us":48,"prepare.run_wall_time_us":51,"raft.queue_time_us":235386,"raft.run_cpu_time_us":216,"raft.run_wall_time_us":49246,"spinlock_wait_cycles":31173888,"thread_start_us":330554,"threads_started":2,"wal-append.queue_time_us":643189}
W20260129 02:59:47.812510 22732 log.cc:927] Time spent T dc2f4cb752cf4d59b5f6bc689f5d8104 P 197b880fa39d432a9b472845876bfaf0: Append to log took a long time: real 0.455s user 0.000s sys 0.001s
I20260129 02:59:47.927488 22663 raft_consensus.cc:3055] T 3906b080a5b34a74b8b64128635e8997 P 1c50de89a5a845668696dcb28c3badd6 [term 2 LEADER]: Stepping down as leader of term 2
I20260129 02:59:47.813133 22727 consensus_queue.cc:1059] T e4d18c35b0a7493e85d0a4b92956a48f P 1c50de89a5a845668696dcb28c3badd6 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 }, Status: INVALID_TERM, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.421s
I20260129 02:59:47.927932 22663 raft_consensus.cc:740] T 3906b080a5b34a74b8b64128635e8997 P 1c50de89a5a845668696dcb28c3badd6 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 1c50de89a5a845668696dcb28c3badd6, State: Running, Role: LEADER
W20260129 02:59:47.928097 22160 consensus_meta.cc:287] T cedce331e1584ac8b636c9b8a72b2ba0 P 974b802141bd4f13beffef2aa03f88f9: Time spent flushing consensus metadata: real 1.422s user 0.009s sys 0.003s
I20260129 02:59:47.928383 22727 raft_consensus.cc:3055] T e4d18c35b0a7493e85d0a4b92956a48f P 1c50de89a5a845668696dcb28c3badd6 [term 3 LEADER]: Stepping down as leader of term 3
I20260129 02:59:47.928887 22727 raft_consensus.cc:740] T e4d18c35b0a7493e85d0a4b92956a48f P 1c50de89a5a845668696dcb28c3badd6 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 1c50de89a5a845668696dcb28c3badd6, State: Running, Role: LEADER
I20260129 02:59:47.928539 22160 raft_consensus.cc:515] T cedce331e1584ac8b636c9b8a72b2ba0 P 974b802141bd4f13beffef2aa03f88f9 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
W20260129 02:59:47.929514 20507 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:48544 (request call id 184) took 2550 ms (client timeout 2131 ms). Trace:
W20260129 02:59:47.929926 20507 rpcz_store.cc:269] 0129 02:59:45.379246 (+ 0us) service_pool.cc:168] Inserting onto call queue
0129 02:59:45.397857 (+ 18611us) service_pool.cc:225] Handling call
0129 02:59:47.929269 (+2531412us) spinlock_profiling.cc:243] Waited 676 ms on lock 0x7b54002ee354. stack: 00007f8fe044e373 00007f8fe044e270 00007f8fdfda0574 00007f8fe9a5c6b4 00007f8fe9a5c649 00007f8fe34975a1 00007f8fe3531f7a 00007f8fe84e3647 00007f8fe1a73834 00007f8fe1a737c2 00007f8fe1a736f1 00007f8fe1a7366c 00007f8fe1a72922 00007f8fe186d34c 00007f8fe186c786 00007f8fe186c12f
0129 02:59:47.929414 (+ 145us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1555369216}
I20260129 02:59:47.930781 20433 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55676 (request call id 1244) took 1441 ms. Trace:
W20260129 02:59:47.815474 22671 consensus_meta.cc:287] T 00e79d8f25a34676b1d866eb2068a2b2 P 7a9daf5d42a84615a6383cfec9a10608: Time spent flushing consensus metadata: real 0.981s user 0.006s sys 0.003s
I20260129 02:59:47.931396 22654 leader_election.cc:290] T b189f08e16a74a11980cc55d99359a0a P 1c50de89a5a845668696dcb28c3badd6 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 197b880fa39d432a9b472845876bfaf0 (127.18.211.1:42027), 974b802141bd4f13beffef2aa03f88f9 (127.18.211.2:34517)
I20260129 02:59:47.291986 22719 raft_consensus.cc:2749] T a8479d1fe0bd41e5946186e3a6ca8c7a P 974b802141bd4f13beffef2aa03f88f9 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20260129 02:59:47.821836 20508 raft_consensus.cc:2393] T cb4727386a3e48aca5d71cc6c611c197 P 974b802141bd4f13beffef2aa03f88f9 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 197b880fa39d432a9b472845876bfaf0 in current term 3: Already voted for candidate 974b802141bd4f13beffef2aa03f88f9 in this term.
I20260129 02:59:47.787312 20440 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4e420f90884f4796997b5ea41e49e411" candidate_uuid: "974b802141bd4f13beffef2aa03f88f9" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "197b880fa39d432a9b472845876bfaf0" is_pre_election: true
I20260129 02:59:47.085758 21129 raft_consensus.cc:2468] T c934d31efd3f4e5195605ddc354d14e1 P 7a9daf5d42a84615a6383cfec9a10608 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 197b880fa39d432a9b472845876bfaf0 in term 3.
I20260129 02:59:47.932344 20508 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:57600 (request call id 1751) took 1455 ms. Trace:
I20260129 02:59:47.556011 22729 raft_consensus.cc:2804] T 78aa39096868415d9afc2ffcb248e198 P 7a9daf5d42a84615a6383cfec9a10608 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260129 02:59:47.932682 20508 rpcz_store.cc:276] 0129 02:59:46.476681 (+ 0us) service_pool.cc:168] Inserting onto call queue
0129 02:59:47.784637 (+1307956us) service_pool.cc:225] Handling call
0129 02:59:47.932283 (+147646us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260129 02:59:47.933182 22729 raft_consensus.cc:493] T 78aa39096868415d9afc2ffcb248e198 P 7a9daf5d42a84615a6383cfec9a10608 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260129 02:59:47.835937 22637 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 1.258s user 0.044s sys 0.039s
I20260129 02:59:47.933646 22729 raft_consensus.cc:3060] T 78aa39096868415d9afc2ffcb248e198 P 7a9daf5d42a84615a6383cfec9a10608 [term 4 FOLLOWER]: Advancing to term 5
I20260129 02:59:47.935400 22637 raft_consensus.cc:493] T 75be4bd6bd45474795fd472eff2953ac P 1c50de89a5a845668696dcb28c3badd6 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260129 02:59:47.852905 22745 raft_consensus.cc:493] T 87d5fe48a44a408aaf03e894a67ecb2f P 974b802141bd4f13beffef2aa03f88f9 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260129 02:59:47.935923 22637 raft_consensus.cc:515] T 75be4bd6bd45474795fd472eff2953ac P 1c50de89a5a845668696dcb28c3badd6 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:59:47.936132 22745 raft_consensus.cc:515] T 87d5fe48a44a408aaf03e894a67ecb2f P 974b802141bd4f13beffef2aa03f88f9 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } }
I20260129 02:59:47.937021 22652 leader_election.cc:290] T 75b0b6e82ddc4f92b1fe1d94f9a0b73b P 1c50de89a5a845668696dcb28c3badd6 [CANDIDATE]: Term 3 election: Requested vote from peers 974b802141bd4f13beffef2aa03f88f9 (127.18.211.2:34517), 7a9daf5d42a84615a6383cfec9a10608 (127.18.211.4:40439)
I20260129 02:59:47.937925 22637 leader_election.cc:290] T 75be4bd6bd45474795fd472eff2953ac P 1c50de89a5a845668696dcb28c3badd6 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 197b880fa39d432a9b472845876bfaf0 (127.18.211.1:42027), 974b802141bd4f13beffef2aa03f88f9 (127.18.211.2:34517)
I20260129 02:59:47.857329 20515 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b189f08e16a74a11980cc55d99359a0a" candidate_uuid: "197b880fa39d432a9b472845876bfaf0" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "974b802141bd4f13beffef2aa03f88f9"
W20260129 02:59:47.938602 22668 consensus_meta.cc:287] T 1ce4ddf5b10e4ae8b193b13b2e5936ee P 1c50de89a5a845668696dcb28c3badd6: Time spent flushing consensus metadata: real 1.574s user 0.001s sys 0.013s
W20260129 02:59:47.860584 22566 consensus_meta.cc:287] T 4f8f3a38f6e24ba4857875181f447442 P 974b802141bd4f13beffef2aa03f88f9: Time spent flushing consensus metadata: real 1.498s user 0.008s sys 0.004s
I20260129 02:59:47.853214 22738 raft_consensus.cc:515] T 4f5a77e94fef4b16a2ba7b067c8b144a P 974b802141bd4f13beffef2aa03f88f9 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } }
W20260129 02:59:47.882990 20480 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.806s user 0.007s sys 0.029s
I20260129 02:59:47.939572 22566 raft_consensus.cc:515] T 4f8f3a38f6e24ba4857875181f447442 P 974b802141bd4f13beffef2aa03f88f9 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
W20260129 02:59:47.940658 22697 log.cc:927] Time spent T 00e79d8f25a34676b1d866eb2068a2b2 P 974b802141bd4f13beffef2aa03f88f9: Append to log took a long time: real 2.057s user 0.004s sys 0.000s
I20260129 02:59:47.931171 20433 rpcz_store.cc:276] 0129 02:59:46.489497 (+ 0us) service_pool.cc:168] Inserting onto call queue
0129 02:59:46.517796 (+ 28299us) service_pool.cc:225] Handling call
0129 02:59:46.518245 (+ 449us) raft_consensus.cc:1471] Updating replica for 1 ops
0129 02:59:46.518710 (+ 465us) raft_consensus.cc:1534] Early marking committed up to index 0
0129 02:59:46.518752 (+ 42us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0129 02:59:46.519323 (+ 571us) log.cc:844] Serialized 26 byte log entry
0129 02:59:46.839495 (+320172us) raft_consensus.cc:1645] Marking committed up to 5
0129 02:59:46.841741 (+ 2246us) raft_consensus.cc:1697] Filling consensus response to leader.
0129 02:59:46.842203 (+ 462us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0129 02:59:47.930576 (+1088373us) raft_consensus.cc:1685] finished
0129 02:59:47.930597 (+ 21us) raft_consensus.cc:1691] UpdateReplicas() finished
0129 02:59:47.930725 (+ 128us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":15685,"raft.queue_time_us":18013,"raft.run_cpu_time_us":146,"raft.run_wall_time_us":146,"thread_start_us":319726,"threads_started":1,"wal-append.queue_time_us":554599}
I20260129 02:59:47.732632 22553 raft_consensus.cc:515] T 368953c9102643b2a21f99af36b51905 P 1c50de89a5a845668696dcb28c3badd6 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
W20260129 02:59:47.958671 20554 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.211.4:40439, user_credentials={real_user=slave}} blocked reactor thread for 47018.6us
I20260129 02:59:47.836136 22677 raft_consensus.cc:493] T 00e79d8f25a34676b1d866eb2068a2b2 P 197b880fa39d432a9b472845876bfaf0 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260129 02:59:47.926708 22337 raft_consensus.cc:3055] T 724a236524cb4e7ba3428880681d6cc2 P 1c50de89a5a845668696dcb28c3badd6 [term 2 LEADER]: Stepping down as leader of term 2
I20260129 02:59:47.929733 22727 consensus_queue.cc:260] T e4d18c35b0a7493e85d0a4b92956a48f P 1c50de89a5a845668696dcb28c3badd6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.3, Last appended by leader: 3, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:59:47.973279 22727 raft_consensus.cc:3060] T e4d18c35b0a7493e85d0a4b92956a48f P 1c50de89a5a845668696dcb28c3badd6 [term 3 FOLLOWER]: Advancing to term 4
I20260129 02:59:47.973338 22553 leader_election.cc:290] T 368953c9102643b2a21f99af36b51905 P 1c50de89a5a845668696dcb28c3badd6 [CANDIDATE]: Term 6 election: Requested vote from peers 197b880fa39d432a9b472845876bfaf0 (127.18.211.1:42027), 974b802141bd4f13beffef2aa03f88f9 (127.18.211.2:34517)
I20260129 02:59:47.974738 20439 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55676 (request call id 1246) took 1425 ms. Trace:
I20260129 02:59:47.974674 22677 raft_consensus.cc:515] T 00e79d8f25a34676b1d866eb2068a2b2 P 197b880fa39d432a9b472845876bfaf0 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } }
I20260129 02:59:47.975244 20439 rpcz_store.cc:276] 0129 02:59:46.549148 (+ 0us) service_pool.cc:168] Inserting onto call queue
0129 02:59:46.566104 (+ 16956us) service_pool.cc:225] Handling call
0129 02:59:46.582552 (+ 16448us) raft_consensus.cc:1471] Updating replica for 0 ops
0129 02:59:46.844036 (+261484us) raft_consensus.cc:1534] Early marking committed up to index 0
0129 02:59:46.844068 (+ 32us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0129 02:59:46.844112 (+ 44us) raft_consensus.cc:1645] Marking committed up to 6
0129 02:59:46.844143 (+ 31us) raft_consensus.cc:1697] Filling consensus response to leader.
0129 02:59:47.974488 (+1130345us) spinlock_profiling.cc:243] Waited 261 ms on lock 0x7b54002670d4. stack: 00007f8fe044e373 00007f8fe044e270 00007f8fdfda0574 00007f8fe9a5c6b4 00007f8fe9a5c649 00007f8fe34975a1 00007f8fe352ea81 00007f8fe352d4be 00007f8fe84e1f49 00007f8fe1a6ed34 00007f8fe1a6ecc2 00007f8fe1a6ebf1 00007f8fe1a6eb6c 00007f8fe1a6de22 00007f8fe186d34c 00007f8fe186c786
0129 02:59:47.974564 (+ 76us) raft_consensus.cc:1691] UpdateReplicas() finished
0129 02:59:47.974684 (+ 120us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":600798976}
W20260129 02:59:47.976112 22372 consensus_meta.cc:287] T 96609ac68bdf4993a12e1c23355ca013 P 7a9daf5d42a84615a6383cfec9a10608: Time spent flushing consensus metadata: real 1.313s user 0.011s sys 0.000s
I20260129 02:59:47.809499 21133 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4e3805ffa70242e8ade66da260dddf66" candidate_uuid: "197b880fa39d432a9b472845876bfaf0" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "7a9daf5d42a84615a6383cfec9a10608" is_pre_election: true
I20260129 02:59:47.979095 21133 raft_consensus.cc:2468] T 4e3805ffa70242e8ade66da260dddf66 P 7a9daf5d42a84615a6383cfec9a10608 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 197b880fa39d432a9b472845876bfaf0 in term 3.
I20260129 02:59:47.249523 22656 raft_consensus.cc:3055] T 7a9a69cd82724b1faed1d10aa091fc4a P 1c50de89a5a845668696dcb28c3badd6 [term 4 LEADER]: Stepping down as leader of term 4
I20260129 02:59:47.372253 22735 consensus_queue.cc:1048] T 42d27a28d89b409e8807ec0efd97af69 P 7a9daf5d42a84615a6383cfec9a10608 [LEADER]: Connected to new peer: Peer: permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.454s
W20260129 02:59:47.936264 20516 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:57600 (request call id 1746) took 1518 ms (client timeout 1563 ms). Trace:
W20260129 02:59:47.938652 22685 log.cc:927] Time spent T 42d27a28d89b409e8807ec0efd97af69 P 7a9daf5d42a84615a6383cfec9a10608: Append to log took a long time: real 2.006s user 0.001s sys 0.001s
W20260129 02:59:47.940053 20480 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.863s user 0.007s sys 0.029s
I20260129 02:59:47.147544 22638 raft_consensus.cc:515] T cb4727386a3e48aca5d71cc6c611c197 P 1c50de89a5a845668696dcb28c3badd6 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
W20260129 02:59:47.860076 22636 consensus_meta.cc:287] T 8055232cf0004d40a85f3bf983fe00a9 P 197b880fa39d432a9b472845876bfaf0: Time spent flushing consensus metadata: real 1.499s user 0.000s sys 0.012s
I20260129 02:59:47.916303 22739 raft_consensus.cc:515] T 368953c9102643b2a21f99af36b51905 P 197b880fa39d432a9b472845876bfaf0 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:59:47.957733 20438 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b189f08e16a74a11980cc55d99359a0a" candidate_uuid: "1c50de89a5a845668696dcb28c3badd6" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "197b880fa39d432a9b472845876bfaf0" is_pre_election: true
I20260129 02:59:47.808091 22750 raft_consensus.cc:493] T cedce331e1584ac8b636c9b8a72b2ba0 P 197b880fa39d432a9b472845876bfaf0 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260129 02:59:47.895237 22343 consensus_meta.cc:287] T 87d5fe48a44a408aaf03e894a67ecb2f P 7a9daf5d42a84615a6383cfec9a10608: Time spent flushing consensus metadata: real 1.204s user 0.006s sys 0.006s
I20260129 02:59:47.966861 22337 raft_consensus.cc:740] T 724a236524cb4e7ba3428880681d6cc2 P 1c50de89a5a845668696dcb28c3badd6 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 1c50de89a5a845668696dcb28c3badd6, State: Running, Role: LEADER
I20260129 02:59:47.926488 21132 rpcz_store.cc:276] 0129 02:59:45.547225 (+ 0us) service_pool.cc:168] Inserting onto call queue
0129 02:59:45.556223 (+ 8998us) service_pool.cc:225] Handling call
0129 02:59:45.564453 (+ 8230us) raft_consensus.cc:1471] Updating replica for 0 ops
0129 02:59:45.834102 (+269649us) raft_consensus.cc:1534] Early marking committed up to index 0
0129 02:59:45.834137 (+ 35us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0129 02:59:45.834191 (+ 54us) raft_consensus.cc:1645] Marking committed up to 6
0129 02:59:45.834240 (+ 49us) raft_consensus.cc:1697] Filling consensus response to leader.
0129 02:59:47.925893 (+2091653us) spinlock_profiling.cc:243] Waited 255 ms on lock 0x7b54005cc954. stack: 00007f8fe044e373 00007f8fe044e270 00007f8fdfda0574 00007f8fe9a5c6b4 00007f8fe9a5c649 00007f8fe34975a1 00007f8fe352ea81 00007f8fe352d4be 00007f8fe84e1f49 00007f8fe1a6ed34 00007f8fe1a6ecc2 00007f8fe1a6ebf1 00007f8fe1a6eb6c 00007f8fe1a6de22 00007f8fe186d34c 00007f8fe186c786
0129 02:59:47.925939 (+ 46us) raft_consensus.cc:1691] UpdateReplicas() finished
0129 02:59:47.926067 (+ 128us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":585471360}
W20260129 02:59:47.854974 22504 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.609s user 0.016s sys 0.035s
W20260129 02:59:47.996090 20556 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.211.2:34517, user_credentials={real_user=slave}} blocked reactor thread for 178193us
I20260129 02:59:47.996644 21100 leader_election.cc:304] T 1299723a4476420ab02fd8bfd7aa0645 P 7a9daf5d42a84615a6383cfec9a10608 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1c50de89a5a845668696dcb28c3badd6, 7a9daf5d42a84615a6383cfec9a10608; no voters:
W20260129 02:59:47.992141 20407 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.927s user 0.007s sys 0.004s
I20260129 02:59:48.076835 20441 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4408006eb5174de1896b037cf3f18457" candidate_uuid: "1c50de89a5a845668696dcb28c3badd6" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "197b880fa39d432a9b472845876bfaf0" is_pre_election: true
I20260129 02:59:48.100683 20438 raft_consensus.cc:2393] T b189f08e16a74a11980cc55d99359a0a P 197b880fa39d432a9b472845876bfaf0 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1c50de89a5a845668696dcb28c3badd6 in current term 6: Already voted for candidate 197b880fa39d432a9b472845876bfaf0 in this term.
I20260129 02:59:48.005456 20585 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:48526 (request call id 229) took 2063 ms. Trace:
W20260129 02:59:48.100696 20480 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.211.3:37735, user_credentials={real_user=slave}} blocked reactor thread for 1.03474e+06us
W20260129 02:59:48.100644 20516 rpcz_store.cc:269] 0129 02:59:46.417502 (+ 0us) service_pool.cc:168] Inserting onto call queue
0129 02:59:46.503756 (+ 86254us) service_pool.cc:225] Handling call
0129 02:59:47.936102 (+1432346us) spinlock_profiling.cc:243] Waited 46.1 ms on lock 0x7b540026eb54. stack: 00007f8fe044e373 00007f8fe044e270 00007f8fdfda0574 00007f8fe9a5c6b4 00007f8fe9a5c649 00007f8fe34975a1 00007f8fe3531f7a 00007f8fe84e3647 00007f8fe1a73834 00007f8fe1a737c2 00007f8fe1a736f1 00007f8fe1a7366c 00007f8fe1a72922 00007f8fe186d34c 00007f8fe186c786 00007f8fe186c12f
0129 02:59:47.936207 (+ 105us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":106028928}
W20260129 02:59:48.015827 20553 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.211.1:42027, user_credentials={real_user=slave}} blocked reactor thread for 56044.9us
W20260129 02:59:48.104015 20405 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.211.3:37735, user_credentials={real_user=slave}} blocked reactor thread for 393874us
I20260129 02:59:48.106627 20514 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:48544 (request call id 194) took 2505 ms. Trace:
I20260129 02:59:48.141765 20438 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "87d5fe48a44a408aaf03e894a67ecb2f" candidate_uuid: "974b802141bd4f13beffef2aa03f88f9" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "197b880fa39d432a9b472845876bfaf0" is_pre_election: true
I20260129 02:59:48.128237 20439 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "75be4bd6bd45474795fd472eff2953ac" candidate_uuid: "1c50de89a5a845668696dcb28c3badd6" candidate_term: 9 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "197b880fa39d432a9b472845876bfaf0" is_pre_election: true
I20260129 02:59:48.141394 20514 rpcz_store.cc:276] 0129 02:59:45.601059 (+ 0us) service_pool.cc:168] Inserting onto call queue
0129 02:59:45.638317 (+ 37258us) service_pool.cc:225] Handling call
0129 02:59:45.669701 (+ 31384us) raft_consensus.cc:1471] Updating replica for 1 ops
0129 02:59:45.737664 (+ 67963us) raft_consensus.cc:1534] Early marking committed up to index 0
0129 02:59:45.737695 (+ 31us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0129 02:59:45.765893 (+ 28198us) log.cc:844] Serialized 26 byte log entry
0129 02:59:45.866664 (+100771us) raft_consensus.cc:1645] Marking committed up to 6
0129 02:59:45.866720 (+ 56us) raft_consensus.cc:1697] Filling consensus response to leader.
0129 02:59:45.866814 (+ 94us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0129 02:59:47.939987 (+2073173us) spinlock_profiling.cc:243] Waited 4.93 ms on lock 0x7b440002304c. stack: 00007f8fe044e373 00007f8fe044e270 00007f8fdfda0574 00007f8fe9a5c6b4 00007f8fe9a5c649 00007f8fe02c08b1 00007f8fe038f005 00007f8fe038fa22 00007f8fe30d957b 00007f8fe30d4e4f 00007f8fe30c8c0e 00007f8fe30c7f56 00007f8fe30c7b78 00007f8fe30d15c1 00007f8fe30d1579 00007f8fe30d1509
0129 02:59:48.106162 (+166175us) raft_consensus.cc:1685] finished
0129 02:59:48.106196 (+ 34us) raft_consensus.cc:1691] UpdateReplicas() finished
0129 02:59:48.106518 (+ 322us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":11331968,"thread_start_us":32036,"threads_started":1,"wal-append.queue_time_us":74527}
I20260129 02:59:48.143419 20439 raft_consensus.cc:2468] T 75be4bd6bd45474795fd472eff2953ac P 197b880fa39d432a9b472845876bfaf0 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1c50de89a5a845668696dcb28c3badd6 in term 8.
W20260129 02:59:48.142412 20407 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 1.077s user 0.007s sys 0.005s
I20260129 02:59:48.143896 20438 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4f8f3a38f6e24ba4857875181f447442" candidate_uuid: "974b802141bd4f13beffef2aa03f88f9" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "197b880fa39d432a9b472845876bfaf0"
W20260129 02:59:47.992041 22237 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.608s user 0.006s sys 0.006s
I20260129 02:59:48.147804 21134 raft_consensus.cc:1275] T 87d5fe48a44a408aaf03e894a67ecb2f P 7a9daf5d42a84615a6383cfec9a10608 [term 5 FOLLOWER]: Refusing update from remote peer 197b880fa39d432a9b472845876bfaf0: Log matching property violated. Preceding OpId in replica: term: 4 index: 6. Preceding OpId from leader: term: 5 index: 7. (index mismatch)
I20260129 02:59:48.140595 20585 rpcz_store.cc:276] 0129 02:59:45.941701 (+ 0us) service_pool.cc:168] Inserting onto call queue
0129 02:59:45.958005 (+ 16304us) service_pool.cc:225] Handling call
0129 02:59:48.004454 (+2046449us) spinlock_profiling.cc:243] Waited 1.39 ms on lock 0x7b58001cb398. stack: 00007f8fe044e373 00007f8fe044e270 00007f8fdfda0574 00007f8fe9a5c6b4 00007f8fe9a5c649 00007f8fe02c08b1 00007f8fe0442db1 00007f8fe0442c62 00007f8fe92eddd9 00007f8fe846d260 00007f8fe8535dbb 00007f8fe8535e41 00007f8fe84f7af0 00007f8fe84e2459 00007f8fe84e1f02 00007f8fe1a6ed34
0129 02:59:48.004760 (+ 306us) raft_consensus.cc:1471] Updating replica for 0 ops
0129 02:59:48.005012 (+ 252us) raft_consensus.cc:1534] Early marking committed up to index 0
0129 02:59:48.005054 (+ 42us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0129 02:59:48.005112 (+ 58us) raft_consensus.cc:1645] Marking committed up to 5
0129 02:59:48.005158 (+ 46us) raft_consensus.cc:1697] Filling consensus response to leader.
0129 02:59:48.005233 (+ 75us) raft_consensus.cc:1691] UpdateReplicas() finished
0129 02:59:48.005378 (+ 145us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":3199360}
W20260129 02:59:48.148487 21100 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.211.3:37735, user_credentials={real_user=slave}} blocked reactor thread for 151958us
W20260129 02:59:48.149029 21100 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260129 02:59:48.144896 20407 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.211.2:34517, user_credentials={real_user=slave}} blocked reactor thread for 1.09188e+06us
I20260129 02:59:48.152061 20440 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "42d27a28d89b409e8807ec0efd97af69" candidate_uuid: "1c50de89a5a845668696dcb28c3badd6" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "197b880fa39d432a9b472845876bfaf0" is_pre_election: true
I20260129 02:59:48.150969 21099 leader_election.cc:304] T dc2f4cb752cf4d59b5f6bc689f5d8104 P 7a9daf5d42a84615a6383cfec9a10608 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7a9daf5d42a84615a6383cfec9a10608; no voters: 197b880fa39d432a9b472845876bfaf0, 1c50de89a5a845668696dcb28c3badd6
I20260129 02:59:47.998399 22656 raft_consensus.cc:740] T 7a9a69cd82724b1faed1d10aa091fc4a P 1c50de89a5a845668696dcb28c3badd6 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 1c50de89a5a845668696dcb28c3badd6, State: Running, Role: LEADER
I20260129 02:59:48.165766 20584 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3e2d15f8640b407195210c476e09df91" candidate_uuid: "7a9daf5d42a84615a6383cfec9a10608" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "1c50de89a5a845668696dcb28c3badd6"
W20260129 02:59:48.113015 22708 consensus_meta.cc:287] T add8d4923d1a489e86882c3a25692b95 P 1c50de89a5a845668696dcb28c3badd6: Time spent flushing consensus metadata: real 0.983s user 0.009s sys 0.002s
I20260129 02:59:47.997989 22372 raft_consensus.cc:515] T 96609ac68bdf4993a12e1c23355ca013 P 7a9daf5d42a84615a6383cfec9a10608 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } attrs { replace: true } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } }
I20260129 02:59:48.110584 22493 raft_consensus.cc:3055] T e33a417cf9c54cf5af73a4ee3aac4d3a P 974b802141bd4f13beffef2aa03f88f9 [term 4 LEADER]: Stepping down as leader of term 4
I20260129 02:59:48.161808 20441 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "96609ac68bdf4993a12e1c23355ca013" candidate_uuid: "974b802141bd4f13beffef2aa03f88f9" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "197b880fa39d432a9b472845876bfaf0"
W20260129 02:59:48.188030 21102 thread.cc:641] dns-resolver [worker] (thread pool) Time spent creating pthread: real 1.005s user 0.029s sys 0.044s
I20260129 02:59:47.870059 22658 raft_consensus.cc:515] T 4e420f90884f4796997b5ea41e49e411 P 7a9daf5d42a84615a6383cfec9a10608 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } }
I20260129 02:59:48.156549 21099 leader_election.cc:304] T c934d31efd3f4e5195605ddc354d14e1 P 7a9daf5d42a84615a6383cfec9a10608 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 197b880fa39d432a9b472845876bfaf0, 7a9daf5d42a84615a6383cfec9a10608; no voters:
I20260129 02:59:48.157292 20480 leader_election.cc:304] T 36ff33be666a4c82954731691dbcf327 P 974b802141bd4f13beffef2aa03f88f9 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 974b802141bd4f13beffef2aa03f88f9; no voters: 197b880fa39d432a9b472845876bfaf0, 1c50de89a5a845668696dcb28c3badd6
I20260129 02:59:48.197821 20553 leader_election.cc:304] T 75be4bd6bd45474795fd472eff2953ac P 1c50de89a5a845668696dcb28c3badd6 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 197b880fa39d432a9b472845876bfaf0, 1c50de89a5a845668696dcb28c3badd6; no voters:
W20260129 02:59:48.142207 22504 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.959s user 0.016s sys 0.036s
I20260129 02:59:48.188987 20433 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "cedce331e1584ac8b636c9b8a72b2ba0" candidate_uuid: "974b802141bd4f13beffef2aa03f88f9" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "197b880fa39d432a9b472845876bfaf0"
I20260129 02:59:48.074178 22579 raft_consensus.cc:493] T 3ccac82a01a2445fa5b0ce6c0fcf2295 P 7a9daf5d42a84615a6383cfec9a10608 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260129 02:59:48.107043 22688 raft_consensus.cc:493] T 8055232cf0004d40a85f3bf983fe00a9 P 197b880fa39d432a9b472845876bfaf0 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260129 02:59:48.152895 20440 raft_consensus.cc:2468] T 42d27a28d89b409e8807ec0efd97af69 P 197b880fa39d432a9b472845876bfaf0 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1c50de89a5a845668696dcb28c3badd6 in term 5.
W20260129 02:59:48.160851 20405 leader_election.cc:336] T 4b1876f9a70b49edb3aa74105e66d3b3 P 197b880fa39d432a9b472845876bfaf0 [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3:37735): Timed out: RequestConsensusVote RPC to 127.18.211.3:37735 timed out after 1.686s (SENT)
W20260129 02:59:48.153687 22237 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.770s user 0.009s sys 0.013s
W20260129 02:59:48.188962 20479 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.211.1:42027, user_credentials={real_user=slave}} blocked reactor thread for 91490.8us
I20260129 02:59:48.168064 22667 raft_consensus.cc:2804] T 1299723a4476420ab02fd8bfd7aa0645 P 7a9daf5d42a84615a6383cfec9a10608 [term 7 FOLLOWER]: Leader pre-election won for term 8
W20260129 02:59:48.199666 20553 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260129 02:59:48.190151 22742 raft_consensus.cc:2749] T dc2f4cb752cf4d59b5f6bc689f5d8104 P 7a9daf5d42a84615a6383cfec9a10608 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260129 02:59:48.195339 20441 raft_consensus.cc:3060] T 96609ac68bdf4993a12e1c23355ca013 P 197b880fa39d432a9b472845876bfaf0 [term 4 FOLLOWER]: Advancing to term 5
I20260129 02:59:48.201197 20438 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "368953c9102643b2a21f99af36b51905" candidate_uuid: "1c50de89a5a845668696dcb28c3badd6" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "197b880fa39d432a9b472845876bfaf0"
I20260129 02:59:48.223528 20438 raft_consensus.cc:3060] T 368953c9102643b2a21f99af36b51905 P 197b880fa39d432a9b472845876bfaf0 [term 5 FOLLOWER]: Advancing to term 6
I20260129 02:59:48.223408 20405 leader_election.cc:304] T 4b1876f9a70b49edb3aa74105e66d3b3 P 197b880fa39d432a9b472845876bfaf0 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 197b880fa39d432a9b472845876bfaf0; no voters: 1c50de89a5a845668696dcb28c3badd6, 974b802141bd4f13beffef2aa03f88f9
I20260129 02:59:48.168701 22715 raft_consensus.cc:493] T 87d5fe48a44a408aaf03e894a67ecb2f P 7a9daf5d42a84615a6383cfec9a10608 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260129 02:59:48.189733 20335 catalog_manager.cc:5654] T 4b1876f9a70b49edb3aa74105e66d3b3 P 974b802141bd4f13beffef2aa03f88f9 reported cstate change: term changed from 3 to 4, leader changed from 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3) to 974b802141bd4f13beffef2aa03f88f9 (127.18.211.2). New cstate: current_term: 4 leader_uuid: "974b802141bd4f13beffef2aa03f88f9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } health_report { overall_health: UNKNOWN } } }
I20260129 02:59:48.184409 21127 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "75b0b6e82ddc4f92b1fe1d94f9a0b73b" candidate_uuid: "1c50de89a5a845668696dcb28c3badd6" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "7a9daf5d42a84615a6383cfec9a10608"
I20260129 02:59:48.147724 22581 consensus_queue.cc:1059] T 3e2d15f8640b407195210c476e09df91 P 1c50de89a5a845668696dcb28c3badd6 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.078s
W20260129 02:59:48.200567 21102 thread.cc:608] dns-resolver [worker] (thread pool) Time spent starting thread: real 1.016s user 0.029s sys 0.044s
I20260129 02:59:48.074156 22724 consensus_queue.cc:1059] T 913fc7d751ed4b7388d889aac30a5f9b P 1c50de89a5a845668696dcb28c3badd6 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 }, Status: INVALID_TERM, Last received: 2.2, Next index: 3, Last known committed idx: 1, Time since last communication: 0.129s
I20260129 02:59:47.976137 22554 consensus_queue.cc:260] T 505580c93c694bbb8ea96fdf9b17ab46 P 1c50de89a5a845668696dcb28c3badd6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 3.6, Last appended by leader: 6, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } }
I20260129 02:59:47.965083 22668 raft_consensus.cc:515] T 1ce4ddf5b10e4ae8b193b13b2e5936ee P 1c50de89a5a845668696dcb28c3badd6 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
W20260129 02:59:48.209724 22647 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 1.680s user 0.035s sys 0.018s
I20260129 02:59:48.218827 22337 consensus_queue.cc:260] T 724a236524cb4e7ba3428880681d6cc2 P 1c50de89a5a845668696dcb28c3badd6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:59:48.230341 22743 raft_consensus.cc:2804] T 75be4bd6bd45474795fd472eff2953ac P 1c50de89a5a845668696dcb28c3badd6 [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260129 02:59:48.234622 21130 raft_consensus.cc:2393] T 96609ac68bdf4993a12e1c23355ca013 P 7a9daf5d42a84615a6383cfec9a10608 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 197b880fa39d432a9b472845876bfaf0 in current term 5: Already voted for candidate 7a9daf5d42a84615a6383cfec9a10608 in this term.
I20260129 02:59:48.249276 22554 raft_consensus.cc:3060] T 505580c93c694bbb8ea96fdf9b17ab46 P 1c50de89a5a845668696dcb28c3badd6 [term 3 FOLLOWER]: Advancing to term 4
I20260129 02:59:48.250022 22581 raft_consensus.cc:3055] T 3e2d15f8640b407195210c476e09df91 P 1c50de89a5a845668696dcb28c3badd6 [term 1 LEADER]: Stepping down as leader of term 1
I20260129 02:59:48.250443 22581 raft_consensus.cc:740] T 3e2d15f8640b407195210c476e09df91 P 1c50de89a5a845668696dcb28c3badd6 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 1c50de89a5a845668696dcb28c3badd6, State: Running, Role: LEADER
I20260129 02:59:48.250475 20335 catalog_manager.cc:5654] T a5a05fbcbbcf4916bbc01fd1a5b5e90d P 974b802141bd4f13beffef2aa03f88f9 reported cstate change: term changed from 4 to 5, leader changed from 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3) to 974b802141bd4f13beffef2aa03f88f9 (127.18.211.2). New cstate: current_term: 5 leader_uuid: "974b802141bd4f13beffef2aa03f88f9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } health_report { overall_health: UNKNOWN } } }
I20260129 02:59:48.234216 22197 raft_consensus.cc:515] T 926a250fa5834e88b96ccecbefbf5086 P 974b802141bd4f13beffef2aa03f88f9 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:59:48.202714 22681 raft_consensus.cc:2804] T c934d31efd3f4e5195605ddc354d14e1 P 7a9daf5d42a84615a6383cfec9a10608 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260129 02:59:48.235587 22677 leader_election.cc:290] T 00e79d8f25a34676b1d866eb2068a2b2 P 197b880fa39d432a9b472845876bfaf0 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 974b802141bd4f13beffef2aa03f88f9 (127.18.211.2:34517), 7a9daf5d42a84615a6383cfec9a10608 (127.18.211.4:40439)
I20260129 02:59:48.252739 22681 raft_consensus.cc:493] T c934d31efd3f4e5195605ddc354d14e1 P 7a9daf5d42a84615a6383cfec9a10608 [term 3 FOLLOWER]: Starting leader election (detected failure of leader 197b880fa39d432a9b472845876bfaf0)
I20260129 02:59:48.253218 22681 raft_consensus.cc:3060] T c934d31efd3f4e5195605ddc354d14e1 P 7a9daf5d42a84615a6383cfec9a10608 [term 3 FOLLOWER]: Advancing to term 4
I20260129 02:59:48.253270 22758 raft_consensus.cc:2749] T 4b1876f9a70b49edb3aa74105e66d3b3 P 197b880fa39d432a9b472845876bfaf0 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
W20260129 02:59:48.236320 21102 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.211.2:34517, user_credentials={real_user=slave}} blocked reactor thread for 1.05402e+06us
I20260129 02:59:48.226702 22184 raft_consensus.cc:493] T 913fc7d751ed4b7388d889aac30a5f9b P 974b802141bd4f13beffef2aa03f88f9 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260129 02:59:48.269098 22184 raft_consensus.cc:515] T 913fc7d751ed4b7388d889aac30a5f9b P 974b802141bd4f13beffef2aa03f88f9 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:59:48.269614 21135 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34060 (request call id 489) took 1184 ms. Trace:
I20260129 02:59:48.269975 21135 rpcz_store.cc:276] 0129 02:59:47.085574 (+ 0us) service_pool.cc:168] Inserting onto call queue
0129 02:59:47.085810 (+ 236us) service_pool.cc:225] Handling call
0129 02:59:47.600578 (+514768us) raft_consensus.cc:1471] Updating replica for 1 ops
0129 02:59:47.707923 (+107345us) raft_consensus.cc:1534] Early marking committed up to index 0
0129 02:59:47.707956 (+ 33us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0129 02:59:47.708440 (+ 484us) log.cc:844] Serialized 26 byte log entry
0129 02:59:48.147422 (+438982us) raft_consensus.cc:1645] Marking committed up to 6
0129 02:59:48.147462 (+ 40us) raft_consensus.cc:1697] Filling consensus response to leader.
0129 02:59:48.147524 (+ 62us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0129 02:59:48.269379 (+121855us) raft_consensus.cc:1685] finished
0129 02:59:48.269400 (+ 21us) raft_consensus.cc:1691] UpdateReplicas() finished
0129 02:59:48.269559 (+ 159us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":111703,"raft.run_cpu_time_us":265,"raft.run_wall_time_us":73415,"spinlock_wait_cycles":16640,"thread_start_us":320363,"threads_started":1,"wal-append.queue_time_us":433323}
I20260129 02:59:48.271375 22184 leader_election.cc:290] T 913fc7d751ed4b7388d889aac30a5f9b P 974b802141bd4f13beffef2aa03f88f9 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 197b880fa39d432a9b472845876bfaf0 (127.18.211.1:42027), 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3:37735)
I20260129 02:59:48.223115 22667 raft_consensus.cc:493] T 1299723a4476420ab02fd8bfd7aa0645 P 7a9daf5d42a84615a6383cfec9a10608 [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260129 02:59:48.272301 22667 raft_consensus.cc:3060] T 1299723a4476420ab02fd8bfd7aa0645 P 7a9daf5d42a84615a6383cfec9a10608 [term 7 FOLLOWER]: Advancing to term 8
I20260129 02:59:48.263702 22337 raft_consensus.cc:3060] T 724a236524cb4e7ba3428880681d6cc2 P 1c50de89a5a845668696dcb28c3badd6 [term 2 FOLLOWER]: Advancing to term 3
W20260129 02:59:48.237841 20405 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.211.3:37735, user_credentials={real_user=slave}} blocked reactor thread for 77020us
I20260129 02:59:48.110736 22750 raft_consensus.cc:515] T cedce331e1584ac8b636c9b8a72b2ba0 P 197b880fa39d432a9b472845876bfaf0 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:59:48.242633 20440 raft_consensus.cc:1240] T 7a9a69cd82724b1faed1d10aa091fc4a P 197b880fa39d432a9b472845876bfaf0 [term 5 FOLLOWER]: Rejecting Update request from peer 1c50de89a5a845668696dcb28c3badd6 for earlier term 4. Current term is 5. Ops: []
I20260129 02:59:48.221308 22646 leader_election.cc:290] T a1a39dcd328e4268ae41b988c66b4a75 P 197b880fa39d432a9b472845876bfaf0 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 974b802141bd4f13beffef2aa03f88f9 (127.18.211.2:34517), 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3:37735)
I20260129 02:59:48.103063 22700 leader_election.cc:290] T e7d83d4ccf684e76acb29a0dc844e1b8 P 974b802141bd4f13beffef2aa03f88f9 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 197b880fa39d432a9b472845876bfaf0 (127.18.211.1:42027), 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3:37735)
I20260129 02:59:48.246487 20438 raft_consensus.cc:2468] T 368953c9102643b2a21f99af36b51905 P 197b880fa39d432a9b472845876bfaf0 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1c50de89a5a845668696dcb28c3badd6 in term 6.
W20260129 02:59:48.256721 21102 leader_election.cc:336] T 4f5a77e94fef4b16a2ba7b067c8b144a P 7a9daf5d42a84615a6383cfec9a10608 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 974b802141bd4f13beffef2aa03f88f9 (127.18.211.2:34517): Timed out: RequestConsensusVote RPC to 127.18.211.2:34517 timed out after 2.786s (SENT)
I20260129 02:59:48.202932 22746 leader_election.cc:290] T e7d83d4ccf684e76acb29a0dc844e1b8 P 197b880fa39d432a9b472845876bfaf0 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 974b802141bd4f13beffef2aa03f88f9 (127.18.211.2:34517), 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3:37735)
W20260129 02:59:48.260169 21101 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260129 02:59:48.243098 22755 raft_consensus.cc:2749] T 36ff33be666a4c82954731691dbcf327 P 974b802141bd4f13beffef2aa03f88f9 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260129 02:59:48.262903 22344 raft_consensus.cc:515] T 78aa39096868415d9afc2ffcb248e198 P 1c50de89a5a845668696dcb28c3badd6 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } }
I20260129 02:59:48.235520 22715 raft_consensus.cc:515] T 87d5fe48a44a408aaf03e894a67ecb2f P 7a9daf5d42a84615a6383cfec9a10608 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } }
W20260129 02:59:48.264956 22751 consensus_queue.cc:1175] T 724a236524cb4e7ba3428880681d6cc2 P 1c50de89a5a845668696dcb28c3badd6 [NON_LEADER]: Queue is closed or peer was untracked, disregarding peer response. Response: responder_uuid: "974b802141bd4f13beffef2aa03f88f9" responder_term: 3 status { last_received { term: 1 index: 1 } last_committed_idx: 1 error { code: INVALID_TERM status { code: ILLEGAL_STATE message: "Rejecting Update request from peer 1c50de89a5a845668696dcb28c3badd6 for earlier term 2. Current term is 3. Ops: [2.2-2.2]" } } last_received_current_leader { term: 0 index: 0 } }
I20260129 02:59:48.267094 20513 raft_consensus.cc:1275] T 87d5fe48a44a408aaf03e894a67ecb2f P 974b802141bd4f13beffef2aa03f88f9 [term 5 FOLLOWER]: Refusing update from remote peer 197b880fa39d432a9b472845876bfaf0: Log matching property violated. Preceding OpId in replica: term: 4 index: 6. Preceding OpId from leader: term: 5 index: 7. (index mismatch)
I20260129 02:59:48.280488 20438 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "913fc7d751ed4b7388d889aac30a5f9b" candidate_uuid: "974b802141bd4f13beffef2aa03f88f9" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "197b880fa39d432a9b472845876bfaf0" is_pre_election: true
I20260129 02:59:48.268325 20509 raft_consensus.cc:2468] T 7a9a69cd82724b1faed1d10aa091fc4a P 974b802141bd4f13beffef2aa03f88f9 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 197b880fa39d432a9b472845876bfaf0 in term 5.
I20260129 02:59:48.268452 20437 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:48356 (request call id 284) took 1964 ms. Trace:
I20260129 02:59:48.281803 20513 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57600 (request call id 1755) took 1744 ms. Trace:
I20260129 02:59:48.281939 20438 raft_consensus.cc:2410] T 913fc7d751ed4b7388d889aac30a5f9b P 197b880fa39d432a9b472845876bfaf0 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 974b802141bd4f13beffef2aa03f88f9 for term 4 because replica has last-logged OpId of term: 2 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 1.
I20260129 02:59:48.282348 20513 rpcz_store.cc:276] 0129 02:59:46.537728 (+ 0us) service_pool.cc:168] Inserting onto call queue
0129 02:59:48.266421 (+1728693us) service_pool.cc:225] Handling call
0129 02:59:48.266760 (+ 339us) raft_consensus.cc:1471] Updating replica for 0 ops
0129 02:59:48.281563 (+ 14803us) raft_consensus.cc:1697] Filling consensus response to leader.
0129 02:59:48.281753 (+ 190us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260129 02:59:48.194996 22493 raft_consensus.cc:740] T e33a417cf9c54cf5af73a4ee3aac4d3a P 974b802141bd4f13beffef2aa03f88f9 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 974b802141bd4f13beffef2aa03f88f9, State: Running, Role: LEADER
I20260129 02:59:48.197134 21129 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "42d27a28d89b409e8807ec0efd97af69" candidate_uuid: "1c50de89a5a845668696dcb28c3badd6" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "7a9daf5d42a84615a6383cfec9a10608" is_pre_election: true
I20260129 02:59:48.273175 22658 leader_election.cc:290] T 4e420f90884f4796997b5ea41e49e411 P 7a9daf5d42a84615a6383cfec9a10608 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 197b880fa39d432a9b472845876bfaf0 (127.18.211.1:42027), 974b802141bd4f13beffef2aa03f88f9 (127.18.211.2:34517)
W20260129 02:59:48.244007 22647 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 1.755s user 0.037s sys 0.021s
I20260129 02:59:48.276451 20335 catalog_manager.cc:5654] T 75be4bd6bd45474795fd472eff2953ac P 974b802141bd4f13beffef2aa03f88f9 reported cstate change: term changed from 5 to 7. New cstate: current_term: 7 leader_uuid: "1c50de89a5a845668696dcb28c3badd6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } } }
I20260129 02:59:48.195104 22708 raft_consensus.cc:515] T add8d4923d1a489e86882c3a25692b95 P 1c50de89a5a845668696dcb28c3badd6 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } }
I20260129 02:59:48.281864 20437 rpcz_store.cc:276] 0129 02:59:46.304248 (+ 0us) service_pool.cc:168] Inserting onto call queue
0129 02:59:46.305838 (+ 1590us) service_pool.cc:225] Handling call
0129 02:59:46.306319 (+ 481us) raft_consensus.cc:1471] Updating replica for 1 ops
0129 02:59:46.510455 (+204136us) raft_consensus.cc:1697] Filling consensus response to leader.
0129 02:59:48.268121 (+1757666us) spinlock_profiling.cc:243] Waited 203 ms on lock 0x7b54000a6ed4. stack: 00007f8fe044e373 00007f8fe044e270 00007f8fdfda0574 00007f8fe9a5c6b4 00007f8fe9a5c649 00007f8fe34975a1 00007f8fe352ea81 00007f8fe352d4be 00007f8fe84e1f49 00007f8fe1a6ed34 00007f8fe1a6ecc2 00007f8fe1a6ebf1 00007f8fe1a6eb6c 00007f8fe1a6de22 00007f8fe186d34c 00007f8fe186c786
0129 02:59:48.268386 (+ 265us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":467992960}
I20260129 02:59:48.342500 20437 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4e420f90884f4796997b5ea41e49e411" candidate_uuid: "7a9daf5d42a84615a6383cfec9a10608" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "197b880fa39d432a9b472845876bfaf0" is_pre_election: true
I20260129 02:59:48.343119 22708 leader_election.cc:290] T add8d4923d1a489e86882c3a25692b95 P 1c50de89a5a845668696dcb28c3badd6 [CANDIDATE]: Term 3 election: Requested vote from peers 974b802141bd4f13beffef2aa03f88f9 (127.18.211.2:34517), 7a9daf5d42a84615a6383cfec9a10608 (127.18.211.4:40439)
I20260129 02:59:48.343833 20589 raft_consensus.cc:1240] T add8d4923d1a489e86882c3a25692b95 P 1c50de89a5a845668696dcb28c3badd6 [term 3 FOLLOWER]: Rejecting Update request from peer 7a9daf5d42a84615a6383cfec9a10608 for earlier term 2. Current term is 3. Ops: []
W20260129 02:59:48.273818 20405 leader_election.cc:336] T dc2f4cb752cf4d59b5f6bc689f5d8104 P 197b880fa39d432a9b472845876bfaf0 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3:37735): Timed out: RequestConsensusVote RPC to 127.18.211.3:37735 timed out after 1.696s (SENT)
I20260129 02:59:48.319348 21102 leader_election.cc:304] T 4f5a77e94fef4b16a2ba7b067c8b144a P 7a9daf5d42a84615a6383cfec9a10608 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7a9daf5d42a84615a6383cfec9a10608; no voters: 1c50de89a5a845668696dcb28c3badd6, 974b802141bd4f13beffef2aa03f88f9
I20260129 02:59:48.345468 20553 leader_election.cc:304] T 42d27a28d89b409e8807ec0efd97af69 P 1c50de89a5a845668696dcb28c3badd6 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 197b880fa39d432a9b472845876bfaf0, 1c50de89a5a845668696dcb28c3badd6; no voters:
I20260129 02:59:48.282997 22504 raft_consensus.cc:3055] T 913fc7d751ed4b7388d889aac30a5f9b P 1c50de89a5a845668696dcb28c3badd6 [term 2 LEADER]: Stepping down as leader of term 2
I20260129 02:59:48.355258 20405 leader_election.cc:304] T dc2f4cb752cf4d59b5f6bc689f5d8104 P 197b880fa39d432a9b472845876bfaf0 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 197b880fa39d432a9b472845876bfaf0; no voters: 1c50de89a5a845668696dcb28c3badd6, 7a9daf5d42a84615a6383cfec9a10608
I20260129 02:59:48.359992 20553 leader_election.cc:304] T 368953c9102643b2a21f99af36b51905 P 1c50de89a5a845668696dcb28c3badd6 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 197b880fa39d432a9b472845876bfaf0, 1c50de89a5a845668696dcb28c3badd6; no voters:
I20260129 02:59:48.251371 22581 consensus_queue.cc:260] T 3e2d15f8640b407195210c476e09df91 P 1c50de89a5a845668696dcb28c3badd6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } }
I20260129 02:59:48.246694 22743 raft_consensus.cc:493] T 75be4bd6bd45474795fd472eff2953ac P 1c50de89a5a845668696dcb28c3badd6 [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260129 02:59:48.329248 20556 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.211.2:34517, user_credentials={real_user=slave}} blocked reactor thread for 48055.7us
I20260129 02:59:48.243292 22656 consensus_queue.cc:260] T 7a9a69cd82724b1faed1d10aa091fc4a P 1c50de89a5a845668696dcb28c3badd6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:59:48.332844 20507 raft_consensus.cc:1240] T 926a250fa5834e88b96ccecbefbf5086 P 974b802141bd4f13beffef2aa03f88f9 [term 6 FOLLOWER]: Rejecting Update request from peer 197b880fa39d432a9b472845876bfaf0 for earlier term 5. Current term is 6. Ops: [5.4-5.4]
I20260129 02:59:48.334502 22668 leader_election.cc:290] T 1ce4ddf5b10e4ae8b193b13b2e5936ee P 1c50de89a5a845668696dcb28c3badd6 [CANDIDATE]: Term 6 election: Requested vote from peers 197b880fa39d432a9b472845876bfaf0 (127.18.211.1:42027), 974b802141bd4f13beffef2aa03f88f9 (127.18.211.2:34517)
W20260129 02:59:48.335292 20554 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260129 02:59:48.380398 20554 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260129 02:59:48.337113 20433 raft_consensus.cc:2393] T cedce331e1584ac8b636c9b8a72b2ba0 P 197b880fa39d432a9b472845876bfaf0 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 974b802141bd4f13beffef2aa03f88f9 in current term 5: Already voted for candidate 197b880fa39d432a9b472845876bfaf0 in this term.
I20260129 02:59:48.337069 20438 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1299723a4476420ab02fd8bfd7aa0645" candidate_uuid: "7a9daf5d42a84615a6383cfec9a10608" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "197b880fa39d432a9b472845876bfaf0" is_pre_election: true
I20260129 02:59:48.337957 20440 raft_consensus.cc:1240] T b189f08e16a74a11980cc55d99359a0a P 197b880fa39d432a9b472845876bfaf0 [term 6 FOLLOWER]: Rejecting Update request from peer 974b802141bd4f13beffef2aa03f88f9 for earlier term 5. Current term is 6. Ops: []
I20260129 02:59:48.327966 20407 leader_election.cc:304] T c934d31efd3f4e5195605ddc354d14e1 P 197b880fa39d432a9b472845876bfaf0 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 197b880fa39d432a9b472845876bfaf0, 974b802141bd4f13beffef2aa03f88f9; no voters:
I20260129 02:59:48.224656 22688 raft_consensus.cc:515] T 8055232cf0004d40a85f3bf983fe00a9 P 197b880fa39d432a9b472845876bfaf0 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
W20260129 02:59:48.328037 20479 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.211.1:42027, user_credentials={real_user=slave}} blocked reactor thread for 43479.6us
I20260129 02:59:48.223290 22579 raft_consensus.cc:515] T 3ccac82a01a2445fa5b0ce6c0fcf2295 P 7a9daf5d42a84615a6383cfec9a10608 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } }
I20260129 02:59:48.385249 21135 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "add8d4923d1a489e86882c3a25692b95" candidate_uuid: "1c50de89a5a845668696dcb28c3badd6" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "7a9daf5d42a84615a6383cfec9a10608"
I20260129 02:59:48.328310 22646 raft_consensus.cc:493] T 1299723a4476420ab02fd8bfd7aa0645 P 197b880fa39d432a9b472845876bfaf0 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 7a9daf5d42a84615a6383cfec9a10608)
I20260129 02:59:48.388245 22581 raft_consensus.cc:3060] T 3e2d15f8640b407195210c476e09df91 P 1c50de89a5a845668696dcb28c3badd6 [term 1 FOLLOWER]: Advancing to term 2
I20260129 02:59:48.347291 22403 raft_consensus.cc:2749] T 4f5a77e94fef4b16a2ba7b067c8b144a P 7a9daf5d42a84615a6383cfec9a10608 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
W20260129 02:59:48.338311 20509 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:57600 (request call id 1749) took 1920 ms (client timeout 2158 ms). Trace:
W20260129 02:59:48.390136 20407 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.211.2:34517, user_credentials={real_user=slave}} blocked reactor thread for 62307.9us
I20260129 02:59:48.418892 20407 leader_election.cc:304] T febffe4a8b394368b69b1d164308fb9a P 197b880fa39d432a9b472845876bfaf0 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 197b880fa39d432a9b472845876bfaf0, 974b802141bd4f13beffef2aa03f88f9; no voters: 7a9daf5d42a84615a6383cfec9a10608
W20260129 02:59:48.415207 20405 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.211.3:37735, user_credentials={real_user=slave}} blocked reactor thread for 141406us
W20260129 02:59:48.422931 20405 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.18.211.3:37735 timed out after 1.725s (SENT) [suppressed 21 similar messages]
I20260129 02:59:48.349236 22504 raft_consensus.cc:740] T 913fc7d751ed4b7388d889aac30a5f9b P 1c50de89a5a845668696dcb28c3badd6 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 1c50de89a5a845668696dcb28c3badd6, State: Running, Role: LEADER
I20260129 02:59:48.378000 22743 raft_consensus.cc:3060] T 75be4bd6bd45474795fd472eff2953ac P 1c50de89a5a845668696dcb28c3badd6 [term 8 FOLLOWER]: Advancing to term 9
W20260129 02:59:48.427877 20407 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260129 02:59:48.428403 20407 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260129 02:59:48.376600 22637 consensus_queue.cc:1175] T 3e2d15f8640b407195210c476e09df91 P 1c50de89a5a845668696dcb28c3badd6 [NON_LEADER]: Queue is closed or peer was untracked, disregarding peer response. Response: responder_uuid: "974b802141bd4f13beffef2aa03f88f9" responder_term: 1 status { last_received { term: 1 index: 4 } last_committed_idx: 4 last_received_current_leader { term: 1 index: 4 } }
I20260129 02:59:48.389343 22554 raft_consensus.cc:2804] T 368953c9102643b2a21f99af36b51905 P 1c50de89a5a845668696dcb28c3badd6 [term 6 FOLLOWER]: Leader election won for term 6
I20260129 02:59:48.406881 22647 raft_consensus.cc:2804] T c934d31efd3f4e5195605ddc354d14e1 P 197b880fa39d432a9b472845876bfaf0 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260129 02:59:48.441097 20440 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1ce4ddf5b10e4ae8b193b13b2e5936ee" candidate_uuid: "1c50de89a5a845668696dcb28c3badd6" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "197b880fa39d432a9b472845876bfaf0"
W20260129 02:59:48.456315 20556 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260129 02:59:48.443848 21102 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.211.2:34517, user_credentials={real_user=slave}} blocked reactor thread for 187154us
I20260129 02:59:48.462884 20437 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "96609ac68bdf4993a12e1c23355ca013" candidate_uuid: "7a9daf5d42a84615a6383cfec9a10608" candidate_term: 5 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "197b880fa39d432a9b472845876bfaf0"
I20260129 02:59:48.400892 22654 consensus_queue.cc:1059] T 7a9a69cd82724b1faed1d10aa091fc4a P 1c50de89a5a845668696dcb28c3badd6 [NON_LEADER]: Peer responded invalid term: Peer: permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 }, Status: INVALID_TERM, Last received: 4.3, Next index: 4, Last known committed idx: 2, Time since last communication: 0.052s
I20260129 02:59:48.431710 22758 raft_consensus.cc:2804] T febffe4a8b394368b69b1d164308fb9a P 197b880fa39d432a9b472845876bfaf0 [term 3 FOLLOWER]: Leader election won for term 3
I20260129 02:59:48.441020 22638 leader_election.cc:290] T cb4727386a3e48aca5d71cc6c611c197 P 1c50de89a5a845668696dcb28c3badd6 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 197b880fa39d432a9b472845876bfaf0 (127.18.211.1:42027), 974b802141bd4f13beffef2aa03f88f9 (127.18.211.2:34517)
I20260129 02:59:48.411808 22646 raft_consensus.cc:515] T 1299723a4476420ab02fd8bfd7aa0645 P 197b880fa39d432a9b472845876bfaf0 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } }
I20260129 02:59:48.469894 20435 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "87d5fe48a44a408aaf03e894a67ecb2f" candidate_uuid: "7a9daf5d42a84615a6383cfec9a10608" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "197b880fa39d432a9b472845876bfaf0" is_pre_election: true
I20260129 02:59:48.600190 20587 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "78aa39096868415d9afc2ffcb248e198" candidate_uuid: "197b880fa39d432a9b472845876bfaf0" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "1c50de89a5a845668696dcb28c3badd6"
I20260129 02:59:48.609851 21135 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00e79d8f25a34676b1d866eb2068a2b2" candidate_uuid: "197b880fa39d432a9b472845876bfaf0" candidate_term: 6 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "7a9daf5d42a84615a6383cfec9a10608" is_pre_election: true
I20260129 02:59:48.610319 20587 raft_consensus.cc:2368] T 78aa39096868415d9afc2ffcb248e198 P 1c50de89a5a845668696dcb28c3badd6 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 197b880fa39d432a9b472845876bfaf0 for earlier term 4. Current term is 5.
I20260129 02:59:48.588986 20585 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "96609ac68bdf4993a12e1c23355ca013" candidate_uuid: "7a9daf5d42a84615a6383cfec9a10608" candidate_term: 5 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "1c50de89a5a845668696dcb28c3badd6"
I20260129 02:59:48.611451 20585 raft_consensus.cc:3060] T 96609ac68bdf4993a12e1c23355ca013 P 1c50de89a5a845668696dcb28c3badd6 [term 4 FOLLOWER]: Advancing to term 5
I20260129 02:59:48.475399 20440 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "cb4727386a3e48aca5d71cc6c611c197" candidate_uuid: "1c50de89a5a845668696dcb28c3badd6" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "197b880fa39d432a9b472845876bfaf0" is_pre_election: true
I20260129 02:59:48.612777 20440 raft_consensus.cc:2393] T cb4727386a3e48aca5d71cc6c611c197 P 197b880fa39d432a9b472845876bfaf0 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1c50de89a5a845668696dcb28c3badd6 in current term 3: Already voted for candidate 197b880fa39d432a9b472845876bfaf0 in this term.
I20260129 02:59:48.609592 20437 raft_consensus.cc:1240] T 3906b080a5b34a74b8b64128635e8997 P 197b880fa39d432a9b472845876bfaf0 [term 3 FOLLOWER]: Rejecting Update request from peer 1c50de89a5a845668696dcb28c3badd6 for earlier term 2. Current term is 3. Ops: []
I20260129 02:59:48.442482 22658 consensus_queue.cc:1059] T febffe4a8b394368b69b1d164308fb9a P 7a9daf5d42a84615a6383cfec9a10608 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.087s
I20260129 02:59:48.441792 22739 leader_election.cc:290] T 368953c9102643b2a21f99af36b51905 P 197b880fa39d432a9b472845876bfaf0 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 974b802141bd4f13beffef2aa03f88f9 (127.18.211.2:34517), 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3:37735)
I20260129 02:59:48.413702 22755 consensus_queue.cc:1059] T b189f08e16a74a11980cc55d99359a0a P 974b802141bd4f13beffef2aa03f88f9 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 }, Status: INVALID_TERM, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.113s
I20260129 02:59:48.473342 20441 raft_consensus.cc:2468] T 96609ac68bdf4993a12e1c23355ca013 P 197b880fa39d432a9b472845876bfaf0 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 974b802141bd4f13beffef2aa03f88f9 in term 5.
I20260129 02:59:48.459758 22647 raft_consensus.cc:493] T c934d31efd3f4e5195605ddc354d14e1 P 197b880fa39d432a9b472845876bfaf0 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260129 02:59:48.537106 20405 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.211.4:40439, user_credentials={real_user=slave}} blocked reactor thread for 50239.7us
W20260129 02:59:48.640543 20405 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260129 02:59:48.641556 20585 raft_consensus.cc:2468] T 96609ac68bdf4993a12e1c23355ca013 P 1c50de89a5a845668696dcb28c3badd6 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7a9daf5d42a84615a6383cfec9a10608 in term 5.
I20260129 02:59:48.464385 22231 raft_consensus.cc:2749] T dc2f4cb752cf4d59b5f6bc689f5d8104 P 197b880fa39d432a9b472845876bfaf0 [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
W20260129 02:59:48.470161 21102 leader_election.cc:336] T 96609ac68bdf4993a12e1c23355ca013 P 7a9daf5d42a84615a6383cfec9a10608 [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer 974b802141bd4f13beffef2aa03f88f9 (127.18.211.2:34517): Timed out: RequestConsensusVote RPC to 127.18.211.2:34517 timed out after 1.807s (SENT)
W20260129 02:59:48.587319 22762 log.cc:927] Time spent T 1ce4ddf5b10e4ae8b193b13b2e5936ee P 197b880fa39d432a9b472845876bfaf0: Append to log took a long time: real 0.054s user 0.001s sys 0.001s
I20260129 02:59:48.632408 20436 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55676 (request call id 1251) took 1009 ms. Trace:
I20260129 02:59:48.632823 20587 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4f8f3a38f6e24ba4857875181f447442" candidate_uuid: "197b880fa39d432a9b472845876bfaf0" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "1c50de89a5a845668696dcb28c3badd6" is_pre_election: true
I20260129 02:59:48.605180 22566 leader_election.cc:290] T 4f8f3a38f6e24ba4857875181f447442 P 974b802141bd4f13beffef2aa03f88f9 [CANDIDATE]: Term 5 election: Requested vote from peers 197b880fa39d432a9b472845876bfaf0 (127.18.211.1:42027), 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3:37735)
W20260129 02:59:48.633972 22645 raft_consensus.cc:936] Couldn't advance consensus term.: Illegal state: Can't advance term to: 5 current term: 5 is higher.
I20260129 02:59:48.634367 22715 leader_election.cc:290] T 87d5fe48a44a408aaf03e894a67ecb2f P 7a9daf5d42a84615a6383cfec9a10608 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 197b880fa39d432a9b472845876bfaf0 (127.18.211.1:42027), 974b802141bd4f13beffef2aa03f88f9 (127.18.211.2:34517)
I20260129 02:59:48.431027 22493 consensus_queue.cc:260] T e33a417cf9c54cf5af73a4ee3aac4d3a P 974b802141bd4f13beffef2aa03f88f9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 4.3, Last appended by leader: 3, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
W20260129 02:59:48.636101 22734 log.cc:927] Time spent T a609ac02fae84725ab743f95f51d33ce P 197b880fa39d432a9b472845876bfaf0: Append to log took a long time: real 0.077s user 0.000s sys 0.000s
I20260129 02:59:48.604280 22504 consensus_queue.cc:260] T 913fc7d751ed4b7388d889aac30a5f9b P 1c50de89a5a845668696dcb28c3badd6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
W20260129 02:59:48.637279 22445 consensus_meta.cc:287] T 76f713c3331540d2a5b76164103e9eb1 P 974b802141bd4f13beffef2aa03f88f9: Time spent flushing consensus metadata: real 0.757s user 0.005s sys 0.006s
I20260129 02:59:48.603744 22681 raft_consensus.cc:515] T c934d31efd3f4e5195605ddc354d14e1 P 7a9daf5d42a84615a6383cfec9a10608 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } }
I20260129 02:59:48.589213 22740 raft_consensus.cc:2804] T 42d27a28d89b409e8807ec0efd97af69 P 1c50de89a5a845668696dcb28c3badd6 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260129 02:59:48.637917 20438 raft_consensus.cc:2468] T 1299723a4476420ab02fd8bfd7aa0645 P 197b880fa39d432a9b472845876bfaf0 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7a9daf5d42a84615a6383cfec9a10608 in term 6.
I20260129 02:59:48.749788 20513 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a644c02b16af47679824ccd060ac10fd" candidate_uuid: "1c50de89a5a845668696dcb28c3badd6" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "974b802141bd4f13beffef2aa03f88f9"
W20260129 02:59:48.606637 22693 log.cc:927] Time spent T cb0b3f259ebc4310927338a8740e87b8 P 974b802141bd4f13beffef2aa03f88f9: Append to log took a long time: real 2.591s user 0.000s sys 0.004s
I20260129 02:59:48.463969 22554 raft_consensus.cc:697] T 368953c9102643b2a21f99af36b51905 P 1c50de89a5a845668696dcb28c3badd6 [term 6 LEADER]: Becoming Leader. State: Replica: 1c50de89a5a845668696dcb28c3badd6, State: Running, Role: LEADER
W20260129 02:59:48.624601 20480 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.211.3:37735, user_credentials={real_user=slave}} blocked reactor thread for 467382us
I20260129 02:59:48.550701 22584 raft_consensus.cc:493] T a0fcc58d6fb6473cafbd9d34d5195269 P 1c50de89a5a845668696dcb28c3badd6 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260129 02:59:48.470252 22403 raft_consensus.cc:493] T 2ac659d732314e1288849f08fe5e074e P 7a9daf5d42a84615a6383cfec9a10608 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260129 02:59:48.664940 20441 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3ccac82a01a2445fa5b0ce6c0fcf2295" candidate_uuid: "7a9daf5d42a84615a6383cfec9a10608" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "197b880fa39d432a9b472845876bfaf0" is_pre_election: true
I20260129 02:59:48.763988 20441 raft_consensus.cc:2468] T 3ccac82a01a2445fa5b0ce6c0fcf2295 P 197b880fa39d432a9b472845876bfaf0 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7a9daf5d42a84615a6383cfec9a10608 in term 3.
W20260129 02:59:48.718811 20407 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.211.2:34517, user_credentials={real_user=slave}} blocked reactor thread for 55961.4us
W20260129 02:59:48.645027 21102 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.211.2:34517, user_credentials={real_user=slave}} blocked reactor thread for 174880us
I20260129 02:59:48.705834 20440 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "78aa39096868415d9afc2ffcb248e198" candidate_uuid: "1c50de89a5a845668696dcb28c3badd6" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "197b880fa39d432a9b472845876bfaf0"
W20260129 02:59:48.418195 20509 rpcz_store.cc:269] 0129 02:59:46.418352 (+ 0us) service_pool.cc:168] Inserting onto call queue
0129 02:59:47.279615 (+861263us) service_pool.cc:225] Handling call
0129 02:59:48.328904 (+1049289us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260129 02:59:48.769109 20509 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "926a250fa5834e88b96ccecbefbf5086" candidate_uuid: "1c50de89a5a845668696dcb28c3badd6" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "974b802141bd4f13beffef2aa03f88f9" is_pre_election: true
W20260129 02:59:48.775038 22729 consensus_meta.cc:287] T 78aa39096868415d9afc2ffcb248e198 P 7a9daf5d42a84615a6383cfec9a10608: Time spent flushing consensus metadata: real 0.841s user 0.011s sys 0.001s
W20260129 02:59:48.658046 21100 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260129 02:59:48.689085 20553 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.211.1:42027, user_credentials={real_user=slave}} blocked reactor thread for 53677.5us
W20260129 02:59:48.721613 20554 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.211.4:40439, user_credentials={real_user=slave}} blocked reactor thread for 84996.4us
W20260129 02:59:48.630352 22732 log.cc:927] Time spent T dc2f4cb752cf4d59b5f6bc689f5d8104 P 197b880fa39d432a9b472845876bfaf0: Append to log took a long time: real 0.117s user 0.000s sys 0.000s
I20260129 02:59:48.550154 22372 leader_election.cc:290] T 96609ac68bdf4993a12e1c23355ca013 P 7a9daf5d42a84615a6383cfec9a10608 [CANDIDATE]: Term 5 election: Requested vote from peers 197b880fa39d432a9b472845876bfaf0 (127.18.211.1:42027), 974b802141bd4f13beffef2aa03f88f9 (127.18.211.2:34517), 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3:37735)
I20260129 02:59:48.724169 20433 raft_consensus.cc:1240] T 0f20af7fbe5d4db89a6f17948d98912c P 197b880fa39d432a9b472845876bfaf0 [term 5 LEADER]: Rejecting Update request from peer 1c50de89a5a845668696dcb28c3badd6 for earlier term 4. Current term is 5. Ops: []
I20260129 02:59:48.775812 22688 leader_election.cc:290] T 8055232cf0004d40a85f3bf983fe00a9 P 197b880fa39d432a9b472845876bfaf0 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 974b802141bd4f13beffef2aa03f88f9 (127.18.211.2:34517), 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3:37735)
I20260129 02:59:48.666170 20405 leader_election.cc:304] T 4e3805ffa70242e8ade66da260dddf66 P 197b880fa39d432a9b472845876bfaf0 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 197b880fa39d432a9b472845876bfaf0, 7a9daf5d42a84615a6383cfec9a10608; no voters:
I20260129 02:59:48.781144 20584 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1ce4ddf5b10e4ae8b193b13b2e5936ee" candidate_uuid: "197b880fa39d432a9b472845876bfaf0" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "1c50de89a5a845668696dcb28c3badd6"
I20260129 02:59:48.777045 22160 leader_election.cc:290] T cedce331e1584ac8b636c9b8a72b2ba0 P 974b802141bd4f13beffef2aa03f88f9 [CANDIDATE]: Term 5 election: Requested vote from peers 197b880fa39d432a9b472845876bfaf0 (127.18.211.1:42027), 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3:37735)
W20260129 02:59:48.781813 20405 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.211.4:40439, user_credentials={real_user=slave}} blocked reactor thread for 115730us
I20260129 02:59:48.782543 20583 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "cb4727386a3e48aca5d71cc6c611c197" candidate_uuid: "197b880fa39d432a9b472845876bfaf0" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1c50de89a5a845668696dcb28c3badd6" is_pre_election: true
I20260129 02:59:48.783149 20583 raft_consensus.cc:2468] T cb4727386a3e48aca5d71cc6c611c197 P 1c50de89a5a845668696dcb28c3badd6 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 197b880fa39d432a9b472845876bfaf0 in term 2.
W20260129 02:59:48.728485 20555 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260129 02:59:48.784157 20555 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260129 02:59:48.783964 22403 raft_consensus.cc:515] T 2ac659d732314e1288849f08fe5e074e P 7a9daf5d42a84615a6383cfec9a10608 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } }
I20260129 02:59:48.785259 20586 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "dc2f4cb752cf4d59b5f6bc689f5d8104" candidate_uuid: "197b880fa39d432a9b472845876bfaf0" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "1c50de89a5a845668696dcb28c3badd6" is_pre_election: true
I20260129 02:59:48.730166 20437 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "926a250fa5834e88b96ccecbefbf5086" candidate_uuid: "974b802141bd4f13beffef2aa03f88f9" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "197b880fa39d432a9b472845876bfaf0"
I20260129 02:59:48.786034 22403 leader_election.cc:290] T 2ac659d732314e1288849f08fe5e074e P 7a9daf5d42a84615a6383cfec9a10608 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 197b880fa39d432a9b472845876bfaf0 (127.18.211.1:42027), 974b802141bd4f13beffef2aa03f88f9 (127.18.211.2:34517)
I20260129 02:59:48.786377 20586 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e7d83d4ccf684e76acb29a0dc844e1b8" candidate_uuid: "197b880fa39d432a9b472845876bfaf0" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "1c50de89a5a845668696dcb28c3badd6" is_pre_election: true
I20260129 02:59:48.787487 20586 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fd8311b779af4e1fa1402b2a02a602f2" candidate_uuid: "197b880fa39d432a9b472845876bfaf0" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "1c50de89a5a845668696dcb28c3badd6" is_pre_election: true
I20260129 02:59:48.591231 22667 raft_consensus.cc:515] T 1299723a4476420ab02fd8bfd7aa0645 P 7a9daf5d42a84615a6383cfec9a10608 [term 8 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } }
I20260129 02:59:48.788946 20512 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:48544 (request call id 195) took 3187 ms. Trace:
I20260129 02:59:48.740803 20514 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57600 (request call id 1750) took 2322 ms. Trace:
I20260129 02:59:48.630038 22758 raft_consensus.cc:697] T febffe4a8b394368b69b1d164308fb9a P 197b880fa39d432a9b472845876bfaf0 [term 3 LEADER]: Becoming Leader. State: Replica: 197b880fa39d432a9b472845876bfaf0, State: Running, Role: LEADER
I20260129 02:59:48.661293 22647 raft_consensus.cc:3060] T c934d31efd3f4e5195605ddc354d14e1 P 197b880fa39d432a9b472845876bfaf0 [term 4 FOLLOWER]: Advancing to term 5
I20260129 02:59:48.643697 22671 raft_consensus.cc:3055] T febffe4a8b394368b69b1d164308fb9a P 7a9daf5d42a84615a6383cfec9a10608 [term 2 LEADER]: Stepping down as leader of term 2
W20260129 02:59:48.588129 22761 log.cc:927] Time spent T 0f20af7fbe5d4db89a6f17948d98912c P 197b880fa39d432a9b472845876bfaf0: Append to log took a long time: real 0.181s user 0.001s sys 0.000s
I20260129 02:59:48.723291 22197 leader_election.cc:290] T 926a250fa5834e88b96ccecbefbf5086 P 974b802141bd4f13beffef2aa03f88f9 [CANDIDATE]: Term 6 election: Requested vote from peers 197b880fa39d432a9b472845876bfaf0 (127.18.211.1:42027), 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3:37735)
I20260129 02:59:48.722998 22739 raft_consensus.cc:493] T e33a417cf9c54cf5af73a4ee3aac4d3a P 197b880fa39d432a9b472845876bfaf0 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260129 02:59:48.816147 22739 raft_consensus.cc:515] T e33a417cf9c54cf5af73a4ee3aac4d3a P 197b880fa39d432a9b472845876bfaf0 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:59:48.818554 22739 leader_election.cc:290] T e33a417cf9c54cf5af73a4ee3aac4d3a P 197b880fa39d432a9b472845876bfaf0 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 974b802141bd4f13beffef2aa03f88f9 (127.18.211.2:34517), 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3:37735)
I20260129 02:59:48.750488 20513 raft_consensus.cc:2376] T a644c02b16af47679824ccd060ac10fd P 974b802141bd4f13beffef2aa03f88f9 [term 3 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 1c50de89a5a845668696dcb28c3badd6 in term 3. Re-sending same reply.
I20260129 02:59:48.819824 20513 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fcddab6bca684adf8ea032289a00f90e" candidate_uuid: "1c50de89a5a845668696dcb28c3badd6" candidate_term: 8 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "974b802141bd4f13beffef2aa03f88f9" is_pre_election: true
I20260129 02:59:48.820609 20513 raft_consensus.cc:2393] T fcddab6bca684adf8ea032289a00f90e P 974b802141bd4f13beffef2aa03f88f9 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1c50de89a5a845668696dcb28c3badd6 in current term 8: Already voted for candidate 974b802141bd4f13beffef2aa03f88f9 in this term.
I20260129 02:59:48.820868 22646 leader_election.cc:290] T 1299723a4476420ab02fd8bfd7aa0645 P 197b880fa39d432a9b472845876bfaf0 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3:37735), 7a9daf5d42a84615a6383cfec9a10608 (127.18.211.4:40439)
I20260129 02:59:48.821933 22554 consensus_queue.cc:237] T 368953c9102643b2a21f99af36b51905 P 1c50de89a5a845668696dcb28c3badd6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 4.3, Last appended by leader: 3, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:59:48.823443 21130 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34060 (request call id 484) took 2482 ms. Trace:
I20260129 02:59:48.756909 20585 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "36ff33be666a4c82954731691dbcf327" candidate_uuid: "197b880fa39d432a9b472845876bfaf0" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "1c50de89a5a845668696dcb28c3badd6" is_pre_election: true
I20260129 02:59:48.738031 22344 leader_election.cc:290] T 78aa39096868415d9afc2ffcb248e198 P 1c50de89a5a845668696dcb28c3badd6 [CANDIDATE]: Term 5 election: Requested vote from peers 197b880fa39d432a9b472845876bfaf0 (127.18.211.1:42027), 7a9daf5d42a84615a6383cfec9a10608 (127.18.211.4:40439)
I20260129 02:59:48.759267 20515 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "96609ac68bdf4993a12e1c23355ca013" candidate_uuid: "1c50de89a5a845668696dcb28c3badd6" candidate_term: 5 candidate_status { last_received { term: 2 index: 3 } } ignore_live_leader: false dest_uuid: "974b802141bd4f13beffef2aa03f88f9" is_pre_election: true
I20260129 02:59:48.829186 20515 raft_consensus.cc:2393] T 96609ac68bdf4993a12e1c23355ca013 P 974b802141bd4f13beffef2aa03f88f9 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1c50de89a5a845668696dcb28c3badd6 in current term 5: Already voted for candidate 974b802141bd4f13beffef2aa03f88f9 in this term.
I20260129 02:59:48.761257 20587 raft_consensus.cc:1275] T 926a250fa5834e88b96ccecbefbf5086 P 1c50de89a5a845668696dcb28c3badd6 [term 5 FOLLOWER]: Refusing update from remote peer 197b880fa39d432a9b472845876bfaf0: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 5 index: 4. (index mismatch)
W20260129 02:59:48.765704 21102 leader_election.cc:336] T 0678b8b468bd43cdbc3e485900844821 P 7a9daf5d42a84615a6383cfec9a10608 [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer 974b802141bd4f13beffef2aa03f88f9 (127.18.211.2:34517): Timed out: RequestConsensusVote RPC to 127.18.211.2:34517 timed out after 2.118s (SENT)
I20260129 02:59:48.830132 21102 leader_election.cc:304] T 0678b8b468bd43cdbc3e485900844821 P 7a9daf5d42a84615a6383cfec9a10608 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7a9daf5d42a84615a6383cfec9a10608; no voters: 197b880fa39d432a9b472845876bfaf0, 974b802141bd4f13beffef2aa03f88f9
I20260129 02:59:48.830469 19276 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
W20260129 02:59:48.831070 21102 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.211.2:34517, user_credentials={real_user=slave}} blocked reactor thread for 65402.3us
I20260129 02:59:48.765731 20440 raft_consensus.cc:3060] T 78aa39096868415d9afc2ffcb248e198 P 197b880fa39d432a9b472845876bfaf0 [term 4 FOLLOWER]: Advancing to term 5
I20260129 02:59:48.726394 22684 consensus_queue.cc:1048] T fcddab6bca684adf8ea032289a00f90e P 197b880fa39d432a9b472845876bfaf0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.033s
I20260129 02:59:48.745774 22687 raft_consensus.cc:493] T e4d18c35b0a7493e85d0a4b92956a48f P 974b802141bd4f13beffef2aa03f88f9 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260129 02:59:48.745035 22700 raft_consensus.cc:3055] T b189f08e16a74a11980cc55d99359a0a P 974b802141bd4f13beffef2aa03f88f9 [term 5 LEADER]: Stepping down as leader of term 5
I20260129 02:59:48.766700 21099 leader_election.cc:304] T 3ccac82a01a2445fa5b0ce6c0fcf2295 P 7a9daf5d42a84615a6383cfec9a10608 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 197b880fa39d432a9b472845876bfaf0, 7a9daf5d42a84615a6383cfec9a10608; no voters:
W20260129 02:59:48.838464 21099 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.211.1:42027, user_credentials={real_user=slave}} blocked reactor thread for 71848.7us
I20260129 02:59:48.787395 22680 leader_election.cc:290] T 96609ac68bdf4993a12e1c23355ca013 P 974b802141bd4f13beffef2aa03f88f9 [CANDIDATE]: Term 5 election: Requested vote from peers 197b880fa39d432a9b472845876bfaf0 (127.18.211.1:42027), 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3:37735), 7a9daf5d42a84615a6383cfec9a10608 (127.18.211.4:40439)
I20260129 02:59:48.767731 20511 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57600 (request call id 1743) took 2352 ms. Trace:
I20260129 02:59:48.765357 22504 raft_consensus.cc:3060] T 913fc7d751ed4b7388d889aac30a5f9b P 1c50de89a5a845668696dcb28c3badd6 [term 2 FOLLOWER]: Advancing to term 3
W20260129 02:59:48.776819 20553 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260129 02:59:48.781975 20584 raft_consensus.cc:2368] T 1ce4ddf5b10e4ae8b193b13b2e5936ee P 1c50de89a5a845668696dcb28c3badd6 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 197b880fa39d432a9b472845876bfaf0 for earlier term 5. Current term is 6.
I20260129 02:59:48.783068 22688 raft_consensus.cc:2804] T 4e3805ffa70242e8ade66da260dddf66 P 197b880fa39d432a9b472845876bfaf0 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260129 02:59:48.605597 22750 leader_election.cc:290] T cedce331e1584ac8b636c9b8a72b2ba0 P 197b880fa39d432a9b472845876bfaf0 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 974b802141bd4f13beffef2aa03f88f9 (127.18.211.2:34517), 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3:37735)
I20260129 02:59:48.607111 22579 leader_election.cc:290] T 3ccac82a01a2445fa5b0ce6c0fcf2295 P 7a9daf5d42a84615a6383cfec9a10608 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 197b880fa39d432a9b472845876bfaf0 (127.18.211.1:42027), 974b802141bd4f13beffef2aa03f88f9 (127.18.211.2:34517)
I20260129 02:59:48.506935 22672 raft_consensus.cc:493] T 00e79d8f25a34676b1d866eb2068a2b2 P 7a9daf5d42a84615a6383cfec9a10608 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260129 02:59:48.784186 20583 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7a9a69cd82724b1faed1d10aa091fc4a" candidate_uuid: "197b880fa39d432a9b472845876bfaf0" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "1c50de89a5a845668696dcb28c3badd6" is_pre_election: true
I20260129 02:59:48.731422 22337 consensus_queue.cc:1059] T 3906b080a5b34a74b8b64128635e8997 P 1c50de89a5a845668696dcb28c3badd6 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 }, Status: INVALID_TERM, Last received: 2.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.030s
I20260129 02:59:48.751144 22445 raft_consensus.cc:2749] T 76f713c3331540d2a5b76164103e9eb1 P 974b802141bd4f13beffef2aa03f88f9 [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260129 02:59:48.789726 20514 rpcz_store.cc:276] 0129 02:59:46.418621 (+ 0us) service_pool.cc:168] Inserting onto call queue
0129 02:59:48.155915 (+1737294us) service_pool.cc:225] Handling call
0129 02:59:48.199652 (+ 43737us) raft_consensus.cc:1471] Updating replica for 0 ops
0129 02:59:48.199949 (+ 297us) raft_consensus.cc:1534] Early marking committed up to index 6
0129 02:59:48.425281 (+225332us) log.cc:844] Serialized 14 byte log entry
0129 02:59:48.740428 (+315147us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0129 02:59:48.740499 (+ 71us) raft_consensus.cc:1645] Marking committed up to 6
0129 02:59:48.740548 (+ 49us) raft_consensus.cc:1697] Filling consensus response to leader.
0129 02:59:48.740612 (+ 64us) raft_consensus.cc:1691] UpdateReplicas() finished
0129 02:59:48.740742 (+ 130us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":95759,"prepare.queue_time_us":388213,"prepare.run_cpu_time_us":38,"prepare.run_wall_time_us":41,"raft.queue_time_us":101118,"raft.run_cpu_time_us":162,"raft.run_wall_time_us":11816,"thread_start_us":419983,"threads_started":2,"wal-append.queue_time_us":346553}
I20260129 02:59:48.796800 20509 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4b1876f9a70b49edb3aa74105e66d3b3" candidate_uuid: "1c50de89a5a845668696dcb28c3badd6" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "974b802141bd4f13beffef2aa03f88f9" is_pre_election: true
I20260129 02:59:48.796902 20590 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4b1876f9a70b49edb3aa74105e66d3b3" candidate_uuid: "197b880fa39d432a9b472845876bfaf0" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "1c50de89a5a845668696dcb28c3badd6" is_pre_election: true
I20260129 02:59:48.797559 22667 leader_election.cc:290] T 1299723a4476420ab02fd8bfd7aa0645 P 7a9daf5d42a84615a6383cfec9a10608 [CANDIDATE]: Term 8 election: Requested vote from peers 197b880fa39d432a9b472845876bfaf0 (127.18.211.1:42027), 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3:37735)
I20260129 02:59:48.797653 20588 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2e58e9a5a8aa4cecafb464addb4f35ed" candidate_uuid: "197b880fa39d432a9b472845876bfaf0" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "1c50de89a5a845668696dcb28c3badd6"
I20260129 02:59:48.798285 20581 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e4d18c35b0a7493e85d0a4b92956a48f" candidate_uuid: "197b880fa39d432a9b472845876bfaf0" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "1c50de89a5a845668696dcb28c3badd6"
I20260129 02:59:48.799835 22681 leader_election.cc:290] T c934d31efd3f4e5195605ddc354d14e1 P 7a9daf5d42a84615a6383cfec9a10608 [CANDIDATE]: Term 4 election: Requested vote from peers 197b880fa39d432a9b472845876bfaf0 (127.18.211.1:42027), 974b802141bd4f13beffef2aa03f88f9 (127.18.211.2:34517)
I20260129 02:59:48.800808 22237 raft_consensus.cc:493] T 18f30ebaa4174fd28c958e72d0cc7e24 P 197b880fa39d432a9b472845876bfaf0 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260129 02:59:48.800648 22758 consensus_queue.cc:237] T febffe4a8b394368b69b1d164308fb9a P 197b880fa39d432a9b472845876bfaf0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } }
I20260129 02:59:48.801322 22738 leader_election.cc:290] T 4f5a77e94fef4b16a2ba7b067c8b144a P 974b802141bd4f13beffef2aa03f88f9 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3:37735), 7a9daf5d42a84615a6383cfec9a10608 (127.18.211.4:40439)
I20260129 02:59:48.801829 20582 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:48526 (request call id 233) took 2575 ms. Trace:
I20260129 02:59:48.802055 20480 leader_election.cc:304] T a4c1226b987b468fa84dfc6ba0e2dbe1 P 974b802141bd4f13beffef2aa03f88f9 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 974b802141bd4f13beffef2aa03f88f9; no voters: 197b880fa39d432a9b472845876bfaf0, 1c50de89a5a845668696dcb28c3badd6
I20260129 02:59:48.793042 22493 raft_consensus.cc:3060] T e33a417cf9c54cf5af73a4ee3aac4d3a P 974b802141bd4f13beffef2aa03f88f9 [term 4 FOLLOWER]: Advancing to term 5
W20260129 02:59:48.617004 22641 consensus_meta.cc:287] T a4c1226b987b468fa84dfc6ba0e2dbe1 P 197b880fa39d432a9b472845876bfaf0: Time spent flushing consensus metadata: real 0.788s user 0.005s sys 0.005s
I20260129 02:59:48.789325 20512 rpcz_store.cc:276] 0129 02:59:45.601364 (+ 0us) service_pool.cc:168] Inserting onto call queue
0129 02:59:45.649408 (+ 48044us) service_pool.cc:225] Handling call
0129 02:59:45.666416 (+ 17008us) raft_consensus.cc:1471] Updating replica for 1 ops
0129 02:59:45.673315 (+ 6899us) raft_consensus.cc:1534] Early marking committed up to index 0
0129 02:59:45.673343 (+ 28us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0129 02:59:45.706722 (+ 33379us) log.cc:844] Serialized 26 byte log entry
0129 02:59:45.805978 (+ 99256us) raft_consensus.cc:1645] Marking committed up to 7
0129 02:59:45.845564 (+ 39586us) log.cc:844] Serialized 14 byte log entry
0129 02:59:45.845809 (+ 245us) raft_consensus.cc:1697] Filling consensus response to leader.
0129 02:59:45.845923 (+ 114us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0129 02:59:46.325214 (+479291us) spinlock_profiling.cc:243] Waited 5.95 ms on lock 0x7b18004f0180. stack: 00007f8fe044e373 00007f8fe044e270 00007f8fdfda0574 00007f8fe9a5c6b4 00007f8fe9a5c649 00007f8fe92d4c91 00007f8fe92e1671 00007f8fe0480402 00007f8fe04826a1 00007f8fe0482659 00007f8fe04825e9 00007f8fe04825b1 00007f8fe04818ad 0000000000379254 0000000000379099 00007f8fe046369a
0129 02:59:48.605692 (+2280478us) spinlock_profiling.cc:243] Waited 39.8 ms on lock 0x7b440002304c. stack: 00007f8fe044e373 00007f8fe044e270 00007f8fdfda0574 00007f8fe9a5c6b4 00007f8fe9a5c649 00007f8fe02c08b1 00007f8fe038f005 00007f8fe038fa22 00007f8fe30d957b 00007f8fe30d4e4f 00007f8fe30c8c0e 00007f8fe30c7f56 00007f8fe30c7b78 00007f8fe30d15c1 00007f8fe30d1579 00007f8fe30d1509
0129 02:59:48.788743 (+183051us) raft_consensus.cc:1685] finished
0129 02:59:48.788767 (+ 24us) raft_consensus.cc:1691] UpdateReplicas() finished
0129 02:59:48.788887 (+ 120us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":10416,"prepare.queue_time_us":39936,"prepare.run_cpu_time_us":71,"prepare.run_wall_time_us":3422,"raft.queue_time_us":13393,"raft.run_cpu_time_us":166,"raft.run_wall_time_us":9948,"spinlock_wait_cycles":105286400,"thread_start_us":41588,"threads_started":1,"wal-append.queue_time_us":117963}
I20260129 02:59:48.756690 22740 raft_consensus.cc:493] T 42d27a28d89b409e8807ec0efd97af69 P 1c50de89a5a845668696dcb28c3badd6 [term 5 FOLLOWER]: Starting leader election (detected failure of leader 7a9daf5d42a84615a6383cfec9a10608)
W20260129 02:59:48.809154 20405 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260129 02:59:48.811455 20554 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260129 02:59:48.835620 21101 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260129 02:59:48.841145 20511 rpcz_store.cc:276] 0129 02:59:46.415440 (+ 0us) service_pool.cc:168] Inserting onto call queue
0129 02:59:46.416263 (+ 823us) service_pool.cc:225] Handling call
0129 02:59:46.483560 (+ 67297us) raft_consensus.cc:1471] Updating replica for 1 ops
0129 02:59:46.589767 (+106207us) raft_consensus.cc:1697] Filling consensus response to leader.
0129 02:59:48.764990 (+2175223us) spinlock_profiling.cc:243] Waited 93.2 ms on lock 0x7b5400128c54. stack: 00007f8fe044e373 00007f8fe044e270 00007f8fdfda0574 00007f8fe9a5c6b4 00007f8fe9a5c649 00007f8fe34975a1 00007f8fe352ea81 00007f8fe352d4be 00007f8fe84e1f49 00007f8fe1a6ed34 00007f8fe1a6ecc2 00007f8fe1a6ebf1 00007f8fe1a6eb6c 00007f8fe1a6de22 00007f8fe186d34c 00007f8fe186c786
0129 02:59:48.767658 (+ 2668us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":214337920}
I20260129 02:59:48.840408 20515 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "17d6431cc77a491e8464d68db4081a51" candidate_uuid: "1c50de89a5a845668696dcb28c3badd6" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "974b802141bd4f13beffef2aa03f88f9"
I20260129 02:59:48.809430 22671 raft_consensus.cc:740] T febffe4a8b394368b69b1d164308fb9a P 7a9daf5d42a84615a6383cfec9a10608 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 7a9daf5d42a84615a6383cfec9a10608, State: Running, Role: LEADER
I20260129 02:59:48.775583 22729 raft_consensus.cc:515] T 78aa39096868415d9afc2ffcb248e198 P 7a9daf5d42a84615a6383cfec9a10608 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } }
I20260129 02:59:48.766366 22584 raft_consensus.cc:515] T a0fcc58d6fb6473cafbd9d34d5195269 P 1c50de89a5a845668696dcb28c3badd6 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:59:49.246609 20440 raft_consensus.cc:2468] T 78aa39096868415d9afc2ffcb248e198 P 197b880fa39d432a9b472845876bfaf0 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1c50de89a5a845668696dcb28c3badd6 in term 5.
I20260129 02:59:49.405495 20590 raft_consensus.cc:2410] T 4b1876f9a70b49edb3aa74105e66d3b3 P 1c50de89a5a845668696dcb28c3badd6 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 197b880fa39d432a9b472845876bfaf0 for term 5 because replica has last-logged OpId of term: 4 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 3.
I20260129 02:59:49.406008 20515 raft_consensus.cc:2393] T 17d6431cc77a491e8464d68db4081a51 P 974b802141bd4f13beffef2aa03f88f9 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 1c50de89a5a845668696dcb28c3badd6 in current term 3: Already voted for candidate 974b802141bd4f13beffef2aa03f88f9 in this term.
I20260129 02:59:48.952699 21133 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "78aa39096868415d9afc2ffcb248e198" candidate_uuid: "1c50de89a5a845668696dcb28c3badd6" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "7a9daf5d42a84615a6383cfec9a10608"
W20260129 02:59:49.406937 20554 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260129 02:59:49.407135 20515 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0340229d521b432580ae926f6062e3f7" candidate_uuid: "1c50de89a5a845668696dcb28c3badd6" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "974b802141bd4f13beffef2aa03f88f9"
I20260129 02:59:48.846915 20585 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1299723a4476420ab02fd8bfd7aa0645" candidate_uuid: "7a9daf5d42a84615a6383cfec9a10608" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "1c50de89a5a845668696dcb28c3badd6"
I20260129 02:59:49.020071 20587 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "75be4bd6bd45474795fd472eff2953ac" candidate_uuid: "197b880fa39d432a9b472845876bfaf0" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "1c50de89a5a845668696dcb28c3badd6" is_pre_election: true
I20260129 02:59:49.408038 20585 raft_consensus.cc:3060] T 1299723a4476420ab02fd8bfd7aa0645 P 1c50de89a5a845668696dcb28c3badd6 [term 7 FOLLOWER]: Advancing to term 8
W20260129 02:59:49.154114 21100 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260129 02:59:48.993283 20407 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260129 02:59:48.720688 20436 rpcz_store.cc:276] 0129 02:59:47.623274 (+ 0us) service_pool.cc:168] Inserting onto call queue
0129 02:59:47.918554 (+295280us) service_pool.cc:225] Handling call
0129 02:59:48.608269 (+689715us) spinlock_profiling.cc:243] Waited 3.17 ms on lock 0x7b5800000f98. stack: 00007f8fe044e373 00007f8fe044e270 00007f8fdfda0574 00007f8fe9a5c6b4 00007f8fe9a5c649 00007f8fe02c08b1 00007f8fe0442db1 00007f8fe0442c62 00007f8fe92eddd9 00007f8fe846d260 00007f8fe8535dbb 00007f8fe8535e41 00007f8fe84f7af0 00007f8fe84e2459 00007f8fe84e1f02 00007f8fe1a6ed34
0129 02:59:48.625366 (+ 17097us) raft_consensus.cc:1471] Updating replica for 0 ops
0129 02:59:48.632060 (+ 6694us) raft_consensus.cc:1534] Early marking committed up to index 0
0129 02:59:48.632088 (+ 28us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0129 02:59:48.632141 (+ 53us) raft_consensus.cc:1645] Marking committed up to 2
0129 02:59:48.632176 (+ 35us) raft_consensus.cc:1697] Filling consensus response to leader.
0129 02:59:48.632241 (+ 65us) raft_consensus.cc:1691] UpdateReplicas() finished
0129 02:59:48.632354 (+ 113us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":7298944}
I20260129 02:59:49.404917 20582 rpcz_store.cc:276] 0129 02:59:46.226151 (+ 0us) service_pool.cc:168] Inserting onto call queue
0129 02:59:46.226600 (+ 449us) service_pool.cc:225] Handling call
0129 02:59:46.250533 (+ 23933us) raft_consensus.cc:1471] Updating replica for 0 ops
0129 02:59:47.833914 (+1583381us) raft_consensus.cc:1697] Filling consensus response to leader.
0129 02:59:48.801555 (+967641us) spinlock_profiling.cc:243] Waited 1.58 s on lock 0x7b540003f2d4. stack: 00007f8fe044e373 00007f8fe044e270 00007f8fdfda0574 00007f8fe9a5c6b4 00007f8fe9a5c649 00007f8fe34975a1 00007f8fe352ea81 00007f8fe352d4be 00007f8fe84e1f49 00007f8fe1a6ed34 00007f8fe1a6ecc2 00007f8fe1a6ebf1 00007f8fe1a6eb6c 00007f8fe1a6de22 00007f8fe186d34c 00007f8fe186c786
0129 02:59:48.801768 (+ 213us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":3639907200}
I20260129 02:59:49.034822 20433 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1299723a4476420ab02fd8bfd7aa0645" candidate_uuid: "7a9daf5d42a84615a6383cfec9a10608" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "197b880fa39d432a9b472845876bfaf0"
W20260129 02:59:49.193439 20555 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260129 02:59:49.409861 20407 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260129 02:59:49.407558 20581 raft_consensus.cc:2410] T e4d18c35b0a7493e85d0a4b92956a48f P 1c50de89a5a845668696dcb28c3badd6 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 197b880fa39d432a9b472845876bfaf0 for term 4 because replica has last-logged OpId of term: 3 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 2.
I20260129 02:59:49.034309 20441 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2ac659d732314e1288849f08fe5e074e" candidate_uuid: "7a9daf5d42a84615a6383cfec9a10608" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "197b880fa39d432a9b472845876bfaf0" is_pre_election: true
W20260129 02:59:49.437188 20480 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.211.3:37735, user_credentials={real_user=slave}} blocked reactor thread for 635168us
I20260129 02:59:49.451985 20590 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "724a236524cb4e7ba3428880681d6cc2" candidate_uuid: "197b880fa39d432a9b472845876bfaf0" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "1c50de89a5a845668696dcb28c3badd6" is_pre_election: true
W20260129 02:59:49.443348 20554 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260129 02:59:49.435842 20512 raft_consensus.cc:1240] T 7a9a69cd82724b1faed1d10aa091fc4a P 974b802141bd4f13beffef2aa03f88f9 [term 5 FOLLOWER]: Rejecting Update request from peer 1c50de89a5a845668696dcb28c3badd6 for earlier term 4. Current term is 5. Ops: [4.3-4.3]
I20260129 02:59:49.461553 20511 raft_consensus.cc:1240] T 913fc7d751ed4b7388d889aac30a5f9b P 974b802141bd4f13beffef2aa03f88f9 [term 3 FOLLOWER]: Rejecting Update request from peer 1c50de89a5a845668696dcb28c3badd6 for earlier term 2. Current term is 3. Ops: [2.2-2.2]
W20260129 02:59:49.463408 20555 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260129 02:59:49.457974 20581 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a5a05fbcbbcf4916bbc01fd1a5b5e90d" candidate_uuid: "197b880fa39d432a9b472845876bfaf0" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "1c50de89a5a845668696dcb28c3badd6"
I20260129 02:59:49.465731 20581 raft_consensus.cc:3060] T a5a05fbcbbcf4916bbc01fd1a5b5e90d P 1c50de89a5a845668696dcb28c3badd6 [term 5 FOLLOWER]: Advancing to term 6
I20260129 02:59:48.837771 22590 raft_consensus.cc:2749] T 0678b8b468bd43cdbc3e485900844821 P 7a9daf5d42a84615a6383cfec9a10608 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
W20260129 02:59:49.329119 22645 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.543s user 0.038s sys 0.043s
I20260129 02:59:49.451460 20480 leader_election.cc:304] T a1a39dcd328e4268ae41b988c66b4a75 P 974b802141bd4f13beffef2aa03f88f9 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1c50de89a5a845668696dcb28c3badd6, 974b802141bd4f13beffef2aa03f88f9; no voters: 197b880fa39d432a9b472845876bfaf0
I20260129 02:59:49.196377 21129 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "78aa39096868415d9afc2ffcb248e198" candidate_uuid: "197b880fa39d432a9b472845876bfaf0" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "7a9daf5d42a84615a6383cfec9a10608"
W20260129 02:59:49.468952 20405 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260129 02:59:49.472801 20582 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "913fc7d751ed4b7388d889aac30a5f9b" candidate_uuid: "197b880fa39d432a9b472845876bfaf0" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1c50de89a5a845668696dcb28c3badd6" is_pre_election: true
W20260129 02:59:49.247124 22768 log.cc:927] Time spent T 8055232cf0004d40a85f3bf983fe00a9 P 974b802141bd4f13beffef2aa03f88f9: Append to log took a long time: real 0.293s user 0.000s sys 0.004s
I20260129 02:59:49.474985 20588 raft_consensus.cc:1240] T 76f713c3331540d2a5b76164103e9eb1 P 1c50de89a5a845668696dcb28c3badd6 [term 4 FOLLOWER]: Rejecting Update request from peer 197b880fa39d432a9b472845876bfaf0 for earlier term 3. Current term is 4. Ops: []
I20260129 02:59:48.842896 22700 raft_consensus.cc:740] T b189f08e16a74a11980cc55d99359a0a P 974b802141bd4f13beffef2aa03f88f9 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 974b802141bd4f13beffef2aa03f88f9, State: Running, Role: LEADER
I20260129 02:59:49.477012 20515 raft_consensus.cc:2468] T 0340229d521b432580ae926f6062e3f7 P 974b802141bd4f13beffef2aa03f88f9 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1c50de89a5a845668696dcb28c3badd6 in term 5.
I20260129 02:59:49.443992 22184 raft_consensus.cc:2749] T a4c1226b987b468fa84dfc6ba0e2dbe1 P 974b802141bd4f13beffef2aa03f88f9 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260129 02:59:49.510475 20581 raft_consensus.cc:2468] T a5a05fbcbbcf4916bbc01fd1a5b5e90d P 1c50de89a5a845668696dcb28c3badd6 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 197b880fa39d432a9b472845876bfaf0 in term 6.
I20260129 02:59:49.478448 21133 raft_consensus.cc:2393] T 78aa39096868415d9afc2ffcb248e198 P 7a9daf5d42a84615a6383cfec9a10608 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 1c50de89a5a845668696dcb28c3badd6 in current term 5: Already voted for candidate 7a9daf5d42a84615a6383cfec9a10608 in this term.
I20260129 02:59:49.467343 20584 raft_consensus.cc:1240] T a8479d1fe0bd41e5946186e3a6ca8c7a P 1c50de89a5a845668696dcb28c3badd6 [term 4 FOLLOWER]: Rejecting Update request from peer 197b880fa39d432a9b472845876bfaf0 for earlier term 3. Current term is 4. Ops: []
I20260129 02:59:48.823940 21130 rpcz_store.cc:276] 0129 02:59:46.340953 (+ 0us) service_pool.cc:168] Inserting onto call queue
0129 02:59:46.367481 (+ 26528us) service_pool.cc:225] Handling call
0129 02:59:48.823223 (+2455742us) spinlock_profiling.cc:243] Waited 1.8 s on lock 0x7b54000b27d4. stack: 00007f8fe044e373 00007f8fe044e270 00007f8fdfda0574 00007f8fe9a5c6b4 00007f8fe9a5c649 00007f8fe34975a1 00007f8fe3531f7a 00007f8fe84e3647 00007f8fe1a73834 00007f8fe1a737c2 00007f8fe1a736f1 00007f8fe1a7366c 00007f8fe1a72922 00007f8fe186d34c 00007f8fe186c786 00007f8fe186c12f
0129 02:59:48.823376 (+ 153us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":4148317056}
I20260129 02:59:49.245558 22372 raft_consensus.cc:493] T 18f30ebaa4174fd28c958e72d0cc7e24 P 7a9daf5d42a84615a6383cfec9a10608 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260129 02:59:49.445274 22740 raft_consensus.cc:3060] T 42d27a28d89b409e8807ec0efd97af69 P 1c50de89a5a845668696dcb28c3badd6 [term 5 FOLLOWER]: Advancing to term 6
I20260129 02:59:48.853578 22776 raft_consensus.cc:2804] T 3ccac82a01a2445fa5b0ce6c0fcf2295 P 7a9daf5d42a84615a6383cfec9a10608 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260129 02:59:49.403942 22724 raft_consensus.cc:493] T 724a236524cb4e7ba3428880681d6cc2 P 1c50de89a5a845668696dcb28c3badd6 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260129 02:59:48.843461 22737 consensus_queue.cc:1048] T 87d5fe48a44a408aaf03e894a67ecb2f P 197b880fa39d432a9b472845876bfaf0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.639s
I20260129 02:59:49.443688 22237 raft_consensus.cc:515] T 18f30ebaa4174fd28c958e72d0cc7e24 P 197b880fa39d432a9b472845876bfaf0 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } }
I20260129 02:59:49.443269 22672 raft_consensus.cc:515] T 00e79d8f25a34676b1d866eb2068a2b2 P 7a9daf5d42a84615a6383cfec9a10608 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } }
I20260129 02:59:49.020534 22344 consensus_queue.cc:1059] T 0f20af7fbe5d4db89a6f17948d98912c P 1c50de89a5a845668696dcb28c3badd6 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 }, Status: INVALID_TERM, Last received: 4.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.462s
I20260129 02:59:49.486032 20512 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e7d83d4ccf684e76acb29a0dc844e1b8" candidate_uuid: "1c50de89a5a845668696dcb28c3badd6" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "974b802141bd4f13beffef2aa03f88f9"
I20260129 02:59:49.584395 20512 raft_consensus.cc:3060] T e7d83d4ccf684e76acb29a0dc844e1b8 P 974b802141bd4f13beffef2aa03f88f9 [term 3 FOLLOWER]: Advancing to term 4
I20260129 02:59:49.515292 20435 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55676 (request call id 1265) took 1170 ms. Trace:
I20260129 02:59:49.477521 20585 raft_consensus.cc:2468] T 1299723a4476420ab02fd8bfd7aa0645 P 1c50de89a5a845668696dcb28c3badd6 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7a9daf5d42a84615a6383cfec9a10608 in term 8.
I20260129 02:59:49.585280 20435 rpcz_store.cc:276] 0129 02:59:48.345129 (+ 0us) service_pool.cc:168] Inserting onto call queue
0129 02:59:48.626412 (+281283us) service_pool.cc:225] Handling call
0129 02:59:48.626745 (+ 333us) raft_consensus.cc:1471] Updating replica for 1 ops
0129 02:59:48.626974 (+ 229us) raft_consensus.cc:1534] Early marking committed up to index 0
0129 02:59:48.626997 (+ 23us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0129 02:59:48.644541 (+ 17544us) log.cc:844] Serialized 26 byte log entry
0129 02:59:48.828068 (+183527us) raft_consensus.cc:1645] Marking committed up to 3
0129 02:59:48.828107 (+ 39us) raft_consensus.cc:1697] Filling consensus response to leader.
0129 02:59:48.828174 (+ 67us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0129 02:59:49.515031 (+686857us) raft_consensus.cc:1685] finished
0129 02:59:49.515058 (+ 27us) raft_consensus.cc:1691] UpdateReplicas() finished
0129 02:59:49.515225 (+ 167us) inbound_call.cc:173] Queueing success response
Metrics: {"thread_start_us":79857,"threads_started":1,"wal-append.queue_time_us":783452}
I20260129 02:59:49.588676 20435 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1299723a4476420ab02fd8bfd7aa0645" candidate_uuid: "7a9daf5d42a84615a6383cfec9a10608" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "197b880fa39d432a9b472845876bfaf0" is_pre_election: true
I20260129 02:59:49.487540 20437 raft_consensus.cc:1240] T 368953c9102643b2a21f99af36b51905 P 197b880fa39d432a9b472845876bfaf0 [term 6 FOLLOWER]: Rejecting Update request from peer 974b802141bd4f13beffef2aa03f88f9 for earlier term 5. Current term is 6. Ops: []
I20260129 02:59:49.486312 20480 leader_election.cc:304] T 00e79d8f25a34676b1d866eb2068a2b2 P 974b802141bd4f13beffef2aa03f88f9 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 974b802141bd4f13beffef2aa03f88f9; no voters: 197b880fa39d432a9b472845876bfaf0, 7a9daf5d42a84615a6383cfec9a10608
I20260129 02:59:49.458171 22750 raft_consensus.cc:493] T a0fcc58d6fb6473cafbd9d34d5195269 P 197b880fa39d432a9b472845876bfaf0 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 974b802141bd4f13beffef2aa03f88f9)
I20260129 02:59:49.593442 20585 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "913fc7d751ed4b7388d889aac30a5f9b" candidate_uuid: "197b880fa39d432a9b472845876bfaf0" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "1c50de89a5a845668696dcb28c3badd6"
I20260129 02:59:49.486913 20432 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55676 (request call id 1248) took 2686 ms. Trace:
I20260129 02:59:49.561681 20515 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "add8d4923d1a489e86882c3a25692b95" candidate_uuid: "1c50de89a5a845668696dcb28c3badd6" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "974b802141bd4f13beffef2aa03f88f9"
I20260129 02:59:48.845547 22688 raft_consensus.cc:493] T 4e3805ffa70242e8ade66da260dddf66 P 197b880fa39d432a9b472845876bfaf0 [term 3 FOLLOWER]: Starting leader election (detected failure of leader 1c50de89a5a845668696dcb28c3badd6)
W20260129 02:59:49.563946 20553 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260129 02:59:49.564934 20581 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "75be4bd6bd45474795fd472eff2953ac" candidate_uuid: "197b880fa39d432a9b472845876bfaf0" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "1c50de89a5a845668696dcb28c3badd6"
I20260129 02:59:49.567643 22776 raft_consensus.cc:493] T 3ccac82a01a2445fa5b0ce6c0fcf2295 P 7a9daf5d42a84615a6383cfec9a10608 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260129 02:59:49.506282 22647 consensus_meta.cc:287] T c934d31efd3f4e5195605ddc354d14e1 P 197b880fa39d432a9b472845876bfaf0: Time spent flushing consensus metadata: real 0.693s user 0.009s sys 0.000s
I20260129 02:59:49.525309 22337 raft_consensus.cc:493] T e4d18c35b0a7493e85d0a4b92956a48f P 1c50de89a5a845668696dcb28c3badd6 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260129 02:59:49.572741 22372 raft_consensus.cc:515] T 18f30ebaa4174fd28c958e72d0cc7e24 P 7a9daf5d42a84615a6383cfec9a10608 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } }
I20260129 02:59:49.543771 22667 consensus_queue.cc:1059] T 75b0b6e82ddc4f92b1fe1d94f9a0b73b P 7a9daf5d42a84615a6383cfec9a10608 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.031s
I20260129 02:59:49.574152 20584 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2e58e9a5a8aa4cecafb464addb4f35ed" candidate_uuid: "197b880fa39d432a9b472845876bfaf0" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "1c50de89a5a845668696dcb28c3badd6"
I20260129 02:59:49.500825 20436 raft_consensus.cc:1240] T b189f08e16a74a11980cc55d99359a0a P 197b880fa39d432a9b472845876bfaf0 [term 6 FOLLOWER]: Rejecting Update request from peer 974b802141bd4f13beffef2aa03f88f9 for earlier term 5. Current term is 6. Ops: [5.4-5.4]
I20260129 02:59:49.606133 20589 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:48526 (request call id 240) took 1810 ms. Trace:
I20260129 02:59:48.845316 22687 raft_consensus.cc:515] T e4d18c35b0a7493e85d0a4b92956a48f P 974b802141bd4f13beffef2aa03f88f9 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:59:49.606508 22372 leader_election.cc:290] T 18f30ebaa4174fd28c958e72d0cc7e24 P 7a9daf5d42a84615a6383cfec9a10608 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 197b880fa39d432a9b472845876bfaf0 (127.18.211.1:42027), 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3:37735)
I20260129 02:59:49.608965 22687 leader_election.cc:290] T e4d18c35b0a7493e85d0a4b92956a48f P 974b802141bd4f13beffef2aa03f88f9 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 197b880fa39d432a9b472845876bfaf0 (127.18.211.1:42027), 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3:37735)
I20260129 02:59:49.445639 22641 raft_consensus.cc:515] T a4c1226b987b468fa84dfc6ba0e2dbe1 P 197b880fa39d432a9b472845876bfaf0 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:59:49.609741 20511 raft_consensus.cc:1240] T e4d18c35b0a7493e85d0a4b92956a48f P 974b802141bd4f13beffef2aa03f88f9 [term 4 FOLLOWER]: Rejecting Update request from peer 1c50de89a5a845668696dcb28c3badd6 for earlier term 3. Current term is 4. Ops: []
I20260129 02:59:49.612051 20439 raft_consensus.cc:1240] T a4c1226b987b468fa84dfc6ba0e2dbe1 P 197b880fa39d432a9b472845876bfaf0 [term 4 FOLLOWER]: Rejecting Update request from peer 1c50de89a5a845668696dcb28c3badd6 for earlier term 3. Current term is 4. Ops: []
I20260129 02:59:49.612244 22641 leader_election.cc:290] T a4c1226b987b468fa84dfc6ba0e2dbe1 P 197b880fa39d432a9b472845876bfaf0 [CANDIDATE]: Term 4 election: Requested vote from peers 974b802141bd4f13beffef2aa03f88f9 (127.18.211.2:34517), 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3:37735)
I20260129 02:59:49.481637 22681 raft_consensus.cc:493] T a644c02b16af47679824ccd060ac10fd P 7a9daf5d42a84615a6383cfec9a10608 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 1c50de89a5a845668696dcb28c3badd6)
I20260129 02:59:49.613248 22681 raft_consensus.cc:515] T a644c02b16af47679824ccd060ac10fd P 7a9daf5d42a84615a6383cfec9a10608 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } }
I20260129 02:59:49.613832 20508 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "75b0b6e82ddc4f92b1fe1d94f9a0b73b" candidate_uuid: "1c50de89a5a845668696dcb28c3badd6" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "974b802141bd4f13beffef2aa03f88f9"
I20260129 02:59:49.614431 20508 raft_consensus.cc:3060] T 75b0b6e82ddc4f92b1fe1d94f9a0b73b P 974b802141bd4f13beffef2aa03f88f9 [term 2 FOLLOWER]: Advancing to term 3
I20260129 02:59:49.482754 22348 raft_consensus.cc:493] T 505580c93c694bbb8ea96fdf9b17ab46 P 1c50de89a5a845668696dcb28c3badd6 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260129 02:59:49.615711 22237 leader_election.cc:290] T 18f30ebaa4174fd28c958e72d0cc7e24 P 197b880fa39d432a9b472845876bfaf0 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3:37735), 7a9daf5d42a84615a6383cfec9a10608 (127.18.211.4:40439)
I20260129 02:59:49.616214 20585 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "724a236524cb4e7ba3428880681d6cc2" candidate_uuid: "197b880fa39d432a9b472845876bfaf0" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "1c50de89a5a845668696dcb28c3badd6" is_pre_election: true
I20260129 02:59:49.581063 22677 raft_consensus.cc:493] T c891d2bff4ce4568861bbc992e91fbb0 P 197b880fa39d432a9b472845876bfaf0 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 974b802141bd4f13beffef2aa03f88f9)
I20260129 02:59:49.599740 22688 raft_consensus.cc:3060] T 4e3805ffa70242e8ade66da260dddf66 P 197b880fa39d432a9b472845876bfaf0 [term 3 FOLLOWER]: Advancing to term 4
I20260129 02:59:49.617375 22677 raft_consensus.cc:515] T c891d2bff4ce4568861bbc992e91fbb0 P 197b880fa39d432a9b472845876bfaf0 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:59:49.617950 20585 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3906b080a5b34a74b8b64128635e8997" candidate_uuid: "197b880fa39d432a9b472845876bfaf0" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "1c50de89a5a845668696dcb28c3badd6"
I20260129 02:59:49.619387 22677 leader_election.cc:290] T c891d2bff4ce4568861bbc992e91fbb0 P 197b880fa39d432a9b472845876bfaf0 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 974b802141bd4f13beffef2aa03f88f9 (127.18.211.2:34517), 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3:37735)
I20260129 02:59:49.616209 22348 raft_consensus.cc:515] T 505580c93c694bbb8ea96fdf9b17ab46 P 1c50de89a5a845668696dcb28c3badd6 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } }
I20260129 02:59:49.620297 20581 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a4c1226b987b468fa84dfc6ba0e2dbe1" candidate_uuid: "197b880fa39d432a9b472845876bfaf0" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "1c50de89a5a845668696dcb28c3badd6" is_pre_election: true
I20260129 02:59:49.621447 20581 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0f20af7fbe5d4db89a6f17948d98912c" candidate_uuid: "197b880fa39d432a9b472845876bfaf0" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "1c50de89a5a845668696dcb28c3badd6"
I20260129 02:59:49.621619 20437 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "78aa39096868415d9afc2ffcb248e198" candidate_uuid: "7a9daf5d42a84615a6383cfec9a10608" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "197b880fa39d432a9b472845876bfaf0"
I20260129 02:59:49.542148 20509 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "75be4bd6bd45474795fd472eff2953ac" candidate_uuid: "1c50de89a5a845668696dcb28c3badd6" candidate_term: 9 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "974b802141bd4f13beffef2aa03f88f9" is_pre_election: true
I20260129 02:59:49.622314 20437 raft_consensus.cc:2393] T 78aa39096868415d9afc2ffcb248e198 P 197b880fa39d432a9b472845876bfaf0 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 7a9daf5d42a84615a6383cfec9a10608 in current term 5: Already voted for candidate 1c50de89a5a845668696dcb28c3badd6 in this term.
I20260129 02:59:49.622462 20581 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3906b080a5b34a74b8b64128635e8997" candidate_uuid: "197b880fa39d432a9b472845876bfaf0" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "1c50de89a5a845668696dcb28c3badd6"
I20260129 02:59:49.622534 20509 raft_consensus.cc:2468] T 75be4bd6bd45474795fd472eff2953ac P 974b802141bd4f13beffef2aa03f88f9 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1c50de89a5a845668696dcb28c3badd6 in term 8.
I20260129 02:59:49.623147 20437 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00e79d8f25a34676b1d866eb2068a2b2" candidate_uuid: "7a9daf5d42a84615a6383cfec9a10608" candidate_term: 6 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "197b880fa39d432a9b472845876bfaf0" is_pre_election: true
I20260129 02:59:49.623476 20581 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7d9494f5584d4d5db8b9adf1c8fc888b" candidate_uuid: "197b880fa39d432a9b472845876bfaf0" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "1c50de89a5a845668696dcb28c3badd6" is_pre_election: true
I20260129 02:59:49.623757 20437 raft_consensus.cc:2468] T 00e79d8f25a34676b1d866eb2068a2b2 P 197b880fa39d432a9b472845876bfaf0 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7a9daf5d42a84615a6383cfec9a10608 in term 5.
I20260129 02:59:49.493580 22729 leader_election.cc:290] T 78aa39096868415d9afc2ffcb248e198 P 7a9daf5d42a84615a6383cfec9a10608 [CANDIDATE]: Term 5 election: Requested vote from peers 197b880fa39d432a9b472845876bfaf0 (127.18.211.1:42027), 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3:37735)
I20260129 02:59:49.585894 20588 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "75be4bd6bd45474795fd472eff2953ac" candidate_uuid: "197b880fa39d432a9b472845876bfaf0" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "1c50de89a5a845668696dcb28c3badd6"
I20260129 02:59:49.595471 20515 raft_consensus.cc:3060] T add8d4923d1a489e86882c3a25692b95 P 974b802141bd4f13beffef2aa03f88f9 [term 2 FOLLOWER]: Advancing to term 3
I20260129 02:59:49.595162 20432 rpcz_store.cc:276] 0129 02:59:46.800165 (+ 0us) service_pool.cc:168] Inserting onto call queue
0129 02:59:46.801757 (+ 1592us) service_pool.cc:225] Handling call
0129 02:59:46.929495 (+127738us) raft_consensus.cc:1471] Updating replica for 0 ops
0129 02:59:47.598393 (+668898us) raft_consensus.cc:1697] Filling consensus response to leader.
0129 02:59:49.486690 (+1888297us) spinlock_profiling.cc:243] Waited 151 ms on lock 0x7b54002af154. stack: 00007f8fe044e373 00007f8fe044e270 00007f8fdfda0574 00007f8fe9a5c6b4 00007f8fe9a5c649 00007f8fe34975a1 00007f8fe352ea81 00007f8fe352d4be 00007f8fe84e1f49 00007f8fe1a6ed34 00007f8fe1a6ecc2 00007f8fe1a6ebf1 00007f8fe1a6eb6c 00007f8fe1a6de22 00007f8fe186d34c 00007f8fe186c786
0129 02:59:49.486858 (+ 168us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":346886912}
W20260129 02:59:49.597878 20482 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.755s user 0.014s sys 0.027s
W20260129 02:59:49.639891 20482 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.797s user 0.015s sys 0.027s
W20260129 02:59:49.483654 22645 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.694s user 0.041s sys 0.046s
I20260129 02:59:49.600773 21135 raft_consensus.cc:2410] T 00e79d8f25a34676b1d866eb2068a2b2 P 7a9daf5d42a84615a6383cfec9a10608 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 197b880fa39d432a9b472845876bfaf0 for term 6 because replica has last-logged OpId of term: 4 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 6.
I20260129 02:59:49.605782 20584 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0f20af7fbe5d4db89a6f17948d98912c" candidate_uuid: "197b880fa39d432a9b472845876bfaf0" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "1c50de89a5a845668696dcb28c3badd6"
W20260129 02:59:49.609338 22774 log.cc:927] Time spent T 4e420f90884f4796997b5ea41e49e411 P 7a9daf5d42a84615a6383cfec9a10608: Append to log took a long time: real 0.344s user 0.000s sys 0.001s
I20260129 02:59:49.611958 22337 raft_consensus.cc:515] T e4d18c35b0a7493e85d0a4b92956a48f P 1c50de89a5a845668696dcb28c3badd6 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:59:49.599660 22750 raft_consensus.cc:515] T a0fcc58d6fb6473cafbd9d34d5195269 P 197b880fa39d432a9b472845876bfaf0 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:59:49.623986 20509 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4408006eb5174de1896b037cf3f18457" candidate_uuid: "1c50de89a5a845668696dcb28c3badd6" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "974b802141bd4f13beffef2aa03f88f9" is_pre_election: true
I20260129 02:59:49.559002 22445 raft_consensus.cc:2804] T a1a39dcd328e4268ae41b988c66b4a75 P 974b802141bd4f13beffef2aa03f88f9 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260129 02:59:49.624533 22763 raft_consensus.cc:3060] T 00e79d8f25a34676b1d866eb2068a2b2 P 974b802141bd4f13beffef2aa03f88f9 [term 4 FOLLOWER]: Advancing to term 5
I20260129 02:59:49.624609 22348 leader_election.cc:290] T 505580c93c694bbb8ea96fdf9b17ab46 P 1c50de89a5a845668696dcb28c3badd6 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 974b802141bd4f13beffef2aa03f88f9 (127.18.211.2:34517), 7a9daf5d42a84615a6383cfec9a10608 (127.18.211.4:40439)
I20260129 02:59:49.606546 20589 rpcz_store.cc:276] 0129 02:59:47.796068 (+ 0us) service_pool.cc:168] Inserting onto call queue
0129 02:59:47.796330 (+ 262us) service_pool.cc:225] Handling call
0129 02:59:47.807733 (+ 11403us) raft_consensus.cc:1471] Updating replica for 0 ops
0129 02:59:48.344260 (+536527us) raft_consensus.cc:1697] Filling consensus response to leader.
0129 02:59:49.605920 (+1261660us) spinlock_profiling.cc:243] Waited 536 ms on lock 0x7b540009bad4. stack: 00007f8fe044e373 00007f8fe044e270 00007f8fdfda0574 00007f8fe9a5c6b4 00007f8fe9a5c649 00007f8fe34975a1 00007f8fe352ea81 00007f8fe352d4be 00007f8fe84e1f49 00007f8fe1a6ed34 00007f8fe1a6ecc2 00007f8fe1a6ebf1 00007f8fe1a6eb6c 00007f8fe1a6de22 00007f8fe186d34c 00007f8fe186c786
0129 02:59:49.606073 (+ 153us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1232450816}
I20260129 02:59:49.712833 20589 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1299723a4476420ab02fd8bfd7aa0645" candidate_uuid: "197b880fa39d432a9b472845876bfaf0" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "1c50de89a5a845668696dcb28c3badd6" is_pre_election: true
I20260129 02:59:49.625972 20436 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "18f30ebaa4174fd28c958e72d0cc7e24" candidate_uuid: "7a9daf5d42a84615a6383cfec9a10608" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "197b880fa39d432a9b472845876bfaf0" is_pre_election: true
I20260129 02:59:49.626152 22344 consensus_queue.cc:1243] T 0f20af7fbe5d4db89a6f17948d98912c P 1c50de89a5a845668696dcb28c3badd6 [LEADER]: Peer 197b880fa39d432a9b472845876bfaf0 log is divergent from this leader: its last log entry 5.4 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 3
I20260129 02:59:49.626619 22637 raft_consensus.cc:493] T a1a39dcd328e4268ae41b988c66b4a75 P 1c50de89a5a845668696dcb28c3badd6 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260129 02:59:49.721247 22504 consensus_meta.cc:287] T 913fc7d751ed4b7388d889aac30a5f9b P 1c50de89a5a845668696dcb28c3badd6: Time spent flushing consensus metadata: real 0.867s user 0.007s sys 0.003s
I20260129 02:59:49.615772 22681 leader_election.cc:290] T a644c02b16af47679824ccd060ac10fd P 7a9daf5d42a84615a6383cfec9a10608 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 974b802141bd4f13beffef2aa03f88f9 (127.18.211.2:34517), 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3:37735)
I20260129 02:59:49.722468 20582 raft_consensus.cc:2410] T 913fc7d751ed4b7388d889aac30a5f9b P 1c50de89a5a845668696dcb28c3badd6 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 197b880fa39d432a9b472845876bfaf0 for term 3 because replica has last-logged OpId of term: 2 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 1.
I20260129 02:59:49.724567 20582 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "913fc7d751ed4b7388d889aac30a5f9b" candidate_uuid: "197b880fa39d432a9b472845876bfaf0" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "1c50de89a5a845668696dcb28c3badd6"
I20260129 02:59:49.626173 20553 leader_election.cc:304] T 78aa39096868415d9afc2ffcb248e198 P 1c50de89a5a845668696dcb28c3badd6 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 197b880fa39d432a9b472845876bfaf0, 1c50de89a5a845668696dcb28c3badd6; no voters:
I20260129 02:59:49.725664 22337 leader_election.cc:290] T e4d18c35b0a7493e85d0a4b92956a48f P 1c50de89a5a845668696dcb28c3badd6 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 197b880fa39d432a9b472845876bfaf0 (127.18.211.1:42027), 974b802141bd4f13beffef2aa03f88f9 (127.18.211.2:34517)
W20260129 02:59:49.725740 20553 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.211.1:42027, user_credentials={real_user=slave}} blocked reactor thread for 99640.2us
I20260129 02:59:49.726460 22337 raft_consensus.cc:2804] T 78aa39096868415d9afc2ffcb248e198 P 1c50de89a5a845668696dcb28c3badd6 [term 5 FOLLOWER]: Leader election won for term 5
I20260129 02:59:49.726995 22337 raft_consensus.cc:697] T 78aa39096868415d9afc2ffcb248e198 P 1c50de89a5a845668696dcb28c3badd6 [term 5 LEADER]: Becoming Leader. State: Replica: 1c50de89a5a845668696dcb28c3badd6, State: Running, Role: LEADER
I20260129 02:59:49.628304 20511 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "368953c9102643b2a21f99af36b51905" candidate_uuid: "1c50de89a5a845668696dcb28c3badd6" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "974b802141bd4f13beffef2aa03f88f9"
I20260129 02:59:49.582007 20514 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1ce4ddf5b10e4ae8b193b13b2e5936ee" candidate_uuid: "1c50de89a5a845668696dcb28c3badd6" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "974b802141bd4f13beffef2aa03f88f9" is_pre_election: true
I20260129 02:59:49.728317 20514 raft_consensus.cc:2468] T 1ce4ddf5b10e4ae8b193b13b2e5936ee P 974b802141bd4f13beffef2aa03f88f9 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1c50de89a5a845668696dcb28c3badd6 in term 5.
I20260129 02:59:49.728188 22737 consensus_queue.cc:1059] T fcddab6bca684adf8ea032289a00f90e P 197b880fa39d432a9b472845876bfaf0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 }, Status: INVALID_TERM, Last received: 4.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20260129 02:59:49.728826 20511 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "75be4bd6bd45474795fd472eff2953ac" candidate_uuid: "197b880fa39d432a9b472845876bfaf0" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "974b802141bd4f13beffef2aa03f88f9"
I20260129 02:59:49.729292 20514 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3ccac82a01a2445fa5b0ce6c0fcf2295" candidate_uuid: "197b880fa39d432a9b472845876bfaf0" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "974b802141bd4f13beffef2aa03f88f9" is_pre_election: true
I20260129 02:59:49.729406 20511 raft_consensus.cc:2376] T 75be4bd6bd45474795fd472eff2953ac P 974b802141bd4f13beffef2aa03f88f9 [term 8 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 197b880fa39d432a9b472845876bfaf0 in term 8. Re-sending same reply.
I20260129 02:59:49.729825 22737 raft_consensus.cc:3055] T fcddab6bca684adf8ea032289a00f90e P 197b880fa39d432a9b472845876bfaf0 [term 7 LEADER]: Stepping down as leader of term 7
I20260129 02:59:49.630198 21099 leader_election.cc:304] T 00e79d8f25a34676b1d866eb2068a2b2 P 7a9daf5d42a84615a6383cfec9a10608 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 197b880fa39d432a9b472845876bfaf0, 7a9daf5d42a84615a6383cfec9a10608; no voters:
I20260129 02:59:49.730311 22737 raft_consensus.cc:740] T fcddab6bca684adf8ea032289a00f90e P 197b880fa39d432a9b472845876bfaf0 [term 7 LEADER]: Becoming Follower/Learner. State: Replica: 197b880fa39d432a9b472845876bfaf0, State: Running, Role: LEADER
I20260129 02:59:49.730376 20511 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a5a05fbcbbcf4916bbc01fd1a5b5e90d" candidate_uuid: "197b880fa39d432a9b472845876bfaf0" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "974b802141bd4f13beffef2aa03f88f9"
W20260129 02:59:49.730893 21099 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.211.1:42027, user_credentials={real_user=slave}} blocked reactor thread for 100773us
I20260129 02:59:49.731024 20511 raft_consensus.cc:2376] T a5a05fbcbbcf4916bbc01fd1a5b5e90d P 974b802141bd4f13beffef2aa03f88f9 [term 6 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 197b880fa39d432a9b472845876bfaf0 in term 6. Re-sending same reply.
I20260129 02:59:49.732039 22667 raft_consensus.cc:2804] T 00e79d8f25a34676b1d866eb2068a2b2 P 7a9daf5d42a84615a6383cfec9a10608 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260129 02:59:49.732463 22667 raft_consensus.cc:493] T 00e79d8f25a34676b1d866eb2068a2b2 P 7a9daf5d42a84615a6383cfec9a10608 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260129 02:59:49.732806 22667 raft_consensus.cc:3060] T 00e79d8f25a34676b1d866eb2068a2b2 P 7a9daf5d42a84615a6383cfec9a10608 [term 5 FOLLOWER]: Advancing to term 6
I20260129 02:59:49.732877 20436 raft_consensus.cc:2468] T 18f30ebaa4174fd28c958e72d0cc7e24 P 197b880fa39d432a9b472845876bfaf0 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7a9daf5d42a84615a6383cfec9a10608 in term 4.
W20260129 02:59:49.593889 22743 consensus_meta.cc:287] T 75be4bd6bd45474795fd472eff2953ac P 1c50de89a5a845668696dcb28c3badd6: Time spent flushing consensus metadata: real 1.135s user 0.009s sys 0.002s
I20260129 02:59:49.631801 20512 raft_consensus.cc:2468] T e7d83d4ccf684e76acb29a0dc844e1b8 P 974b802141bd4f13beffef2aa03f88f9 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1c50de89a5a845668696dcb28c3badd6 in term 4.
I20260129 02:59:49.646823 20515 raft_consensus.cc:2468] T add8d4923d1a489e86882c3a25692b95 P 974b802141bd4f13beffef2aa03f88f9 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1c50de89a5a845668696dcb28c3badd6 in term 3.
I20260129 02:59:49.685015 20588 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7a9a69cd82724b1faed1d10aa091fc4a" candidate_uuid: "197b880fa39d432a9b472845876bfaf0" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "1c50de89a5a845668696dcb28c3badd6" is_pre_election: true
I20260129 02:59:49.701742 20437 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e4d18c35b0a7493e85d0a4b92956a48f" candidate_uuid: "974b802141bd4f13beffef2aa03f88f9" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "197b880fa39d432a9b472845876bfaf0" is_pre_election: true
I20260129 02:59:49.702323 20508 raft_consensus.cc:2468] T 75b0b6e82ddc4f92b1fe1d94f9a0b73b P 974b802141bd4f13beffef2aa03f88f9 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1c50de89a5a845668696dcb28c3badd6 in term 3.
I20260129 02:59:49.743593 20508 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "368953c9102643b2a21f99af36b51905" candidate_uuid: "1c50de89a5a845668696dcb28c3badd6" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "974b802141bd4f13beffef2aa03f88f9" is_pre_election: true
W20260129 02:59:49.703173 20480 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.211.4:40439, user_credentials={real_user=slave}} blocked reactor thread for 216927us
I20260129 02:59:49.581578 22724 raft_consensus.cc:515] T 724a236524cb4e7ba3428880681d6cc2 P 1c50de89a5a845668696dcb28c3badd6 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:59:49.604667 22776 raft_consensus.cc:3060] T 3ccac82a01a2445fa5b0ce6c0fcf2295 P 7a9daf5d42a84615a6383cfec9a10608 [term 3 FOLLOWER]: Advancing to term 4
I20260129 02:59:49.705657 20581 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a5a05fbcbbcf4916bbc01fd1a5b5e90d" candidate_uuid: "197b880fa39d432a9b472845876bfaf0" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "1c50de89a5a845668696dcb28c3badd6"
I20260129 02:59:49.614217 22672 leader_election.cc:290] T 00e79d8f25a34676b1d866eb2068a2b2 P 7a9daf5d42a84615a6383cfec9a10608 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 197b880fa39d432a9b472845876bfaf0 (127.18.211.1:42027), 974b802141bd4f13beffef2aa03f88f9 (127.18.211.2:34517)
I20260129 02:59:49.604476 22700 consensus_queue.cc:260] T b189f08e16a74a11980cc55d99359a0a P 974b802141bd4f13beffef2aa03f88f9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.4, Last appended by leader: 4, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:59:49.632305 22647 raft_consensus.cc:515] T c934d31efd3f4e5195605ddc354d14e1 P 197b880fa39d432a9b472845876bfaf0 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } }
I20260129 02:59:49.707131 20584 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "368953c9102643b2a21f99af36b51905" candidate_uuid: "197b880fa39d432a9b472845876bfaf0" candidate_term: 6 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "1c50de89a5a845668696dcb28c3badd6" is_pre_election: true
I20260129 02:59:49.708801 20509 raft_consensus.cc:2468] T 4408006eb5174de1896b037cf3f18457 P 974b802141bd4f13beffef2aa03f88f9 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1c50de89a5a845668696dcb28c3badd6 in term 2.
I20260129 02:59:49.747565 20581 raft_consensus.cc:2376] T a5a05fbcbbcf4916bbc01fd1a5b5e90d P 1c50de89a5a845668696dcb28c3badd6 [term 6 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 197b880fa39d432a9b472845876bfaf0 in term 6. Re-sending same reply.
I20260129 02:59:49.717332 20589 raft_consensus.cc:2368] T 1299723a4476420ab02fd8bfd7aa0645 P 1c50de89a5a845668696dcb28c3badd6 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 197b880fa39d432a9b472845876bfaf0 for earlier term 7. Current term is 8.
I20260129 02:59:49.720417 22584 leader_election.cc:290] T a0fcc58d6fb6473cafbd9d34d5195269 P 1c50de89a5a845668696dcb28c3badd6 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 197b880fa39d432a9b472845876bfaf0 (127.18.211.1:42027), 974b802141bd4f13beffef2aa03f88f9 (127.18.211.2:34517)
I20260129 02:59:49.720709 22656 raft_consensus.cc:3060] T 7a9a69cd82724b1faed1d10aa091fc4a P 1c50de89a5a845668696dcb28c3badd6 [term 4 FOLLOWER]: Advancing to term 5
I20260129 02:59:49.730384 20514 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7d9494f5584d4d5db8b9adf1c8fc888b" candidate_uuid: "197b880fa39d432a9b472845876bfaf0" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "974b802141bd4f13beffef2aa03f88f9" is_pre_election: true
I20260129 02:59:49.736212 20407 leader_election.cc:304] T 75be4bd6bd45474795fd472eff2953ac P 197b880fa39d432a9b472845876bfaf0 [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 197b880fa39d432a9b472845876bfaf0, 974b802141bd4f13beffef2aa03f88f9; no voters:
I20260129 02:59:49.750806 20515 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1ce4ddf5b10e4ae8b193b13b2e5936ee" candidate_uuid: "1c50de89a5a845668696dcb28c3badd6" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "974b802141bd4f13beffef2aa03f88f9"
I20260129 02:59:49.751245 20514 raft_consensus.cc:2468] T 7d9494f5584d4d5db8b9adf1c8fc888b P 974b802141bd4f13beffef2aa03f88f9 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 197b880fa39d432a9b472845876bfaf0 in term 2.
I20260129 02:59:49.751425 20515 raft_consensus.cc:3060] T 1ce4ddf5b10e4ae8b193b13b2e5936ee P 974b802141bd4f13beffef2aa03f88f9 [term 5 FOLLOWER]: Advancing to term 6
I20260129 02:59:49.747999 20511 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "368953c9102643b2a21f99af36b51905" candidate_uuid: "197b880fa39d432a9b472845876bfaf0" candidate_term: 6 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "974b802141bd4f13beffef2aa03f88f9" is_pre_election: true
I20260129 02:59:49.737217 21099 leader_election.cc:304] T 18f30ebaa4174fd28c958e72d0cc7e24 P 7a9daf5d42a84615a6383cfec9a10608 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 197b880fa39d432a9b472845876bfaf0, 7a9daf5d42a84615a6383cfec9a10608; no voters:
I20260129 02:59:49.704319 22372 raft_consensus.cc:3055] T 75b0b6e82ddc4f92b1fe1d94f9a0b73b P 7a9daf5d42a84615a6383cfec9a10608 [term 2 LEADER]: Stepping down as leader of term 2
I20260129 02:59:49.752377 20514 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a4c1226b987b468fa84dfc6ba0e2dbe1" candidate_uuid: "197b880fa39d432a9b472845876bfaf0" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "974b802141bd4f13beffef2aa03f88f9"
I20260129 02:59:49.755307 20512 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c891d2bff4ce4568861bbc992e91fbb0" candidate_uuid: "1c50de89a5a845668696dcb28c3badd6" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "974b802141bd4f13beffef2aa03f88f9" is_pre_election: true
I20260129 02:59:49.759061 20514 raft_consensus.cc:3060] T a4c1226b987b468fa84dfc6ba0e2dbe1 P 974b802141bd4f13beffef2aa03f88f9 [term 3 FOLLOWER]: Advancing to term 4
I20260129 02:59:49.750190 20437 raft_consensus.cc:2468] T e4d18c35b0a7493e85d0a4b92956a48f P 197b880fa39d432a9b472845876bfaf0 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 974b802141bd4f13beffef2aa03f88f9 in term 4.
I20260129 02:59:49.757872 20407 leader_election.cc:304] T a5a05fbcbbcf4916bbc01fd1a5b5e90d P 197b880fa39d432a9b472845876bfaf0 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 197b880fa39d432a9b472845876bfaf0, 974b802141bd4f13beffef2aa03f88f9; no voters:
I20260129 02:59:49.704842 22445 raft_consensus.cc:493] T a1a39dcd328e4268ae41b988c66b4a75 P 974b802141bd4f13beffef2aa03f88f9 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 1c50de89a5a845668696dcb28c3badd6)
I20260129 02:59:49.762143 20581 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a4c1226b987b468fa84dfc6ba0e2dbe1" candidate_uuid: "197b880fa39d432a9b472845876bfaf0" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "1c50de89a5a845668696dcb28c3badd6" is_pre_election: true
I20260129 02:59:49.775847 20508 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a1a39dcd328e4268ae41b988c66b4a75" candidate_uuid: "197b880fa39d432a9b472845876bfaf0" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "974b802141bd4f13beffef2aa03f88f9" is_pre_election: true
I20260129 02:59:49.781785 20590 raft_consensus.cc:2468] T 724a236524cb4e7ba3428880681d6cc2 P 1c50de89a5a845668696dcb28c3badd6 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 197b880fa39d432a9b472845876bfaf0 in term 3.
I20260129 02:59:49.782910 20479 leader_election.cc:304] T e4d18c35b0a7493e85d0a4b92956a48f P 974b802141bd4f13beffef2aa03f88f9 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 197b880fa39d432a9b472845876bfaf0, 974b802141bd4f13beffef2aa03f88f9; no voters:
I20260129 02:59:49.780180 20407 leader_election.cc:304] T 7d9494f5584d4d5db8b9adf1c8fc888b P 197b880fa39d432a9b472845876bfaf0 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 197b880fa39d432a9b472845876bfaf0, 974b802141bd4f13beffef2aa03f88f9; no voters:
I20260129 02:59:49.784709 20508 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e7d83d4ccf684e76acb29a0dc844e1b8" candidate_uuid: "197b880fa39d432a9b472845876bfaf0" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "974b802141bd4f13beffef2aa03f88f9" is_pre_election: true
I20260129 02:59:49.785316 20508 raft_consensus.cc:2468] T e7d83d4ccf684e76acb29a0dc844e1b8 P 974b802141bd4f13beffef2aa03f88f9 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 197b880fa39d432a9b472845876bfaf0 in term 4.
I20260129 02:59:49.786238 20588 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b189f08e16a74a11980cc55d99359a0a" candidate_uuid: "197b880fa39d432a9b472845876bfaf0" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "1c50de89a5a845668696dcb28c3badd6"
I20260129 02:59:49.786355 20480 leader_election.cc:304] T add8d4923d1a489e86882c3a25692b95 P 974b802141bd4f13beffef2aa03f88f9 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 974b802141bd4f13beffef2aa03f88f9; no voters: 1c50de89a5a845668696dcb28c3badd6, 7a9daf5d42a84615a6383cfec9a10608
I20260129 02:59:49.786844 20588 raft_consensus.cc:3060] T b189f08e16a74a11980cc55d99359a0a P 1c50de89a5a845668696dcb28c3badd6 [term 5 FOLLOWER]: Advancing to term 6
I20260129 02:59:49.787926 20437 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a0fcc58d6fb6473cafbd9d34d5195269" candidate_uuid: "1c50de89a5a845668696dcb28c3badd6" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "197b880fa39d432a9b472845876bfaf0" is_pre_election: true
I20260129 02:59:49.788167 20584 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b189f08e16a74a11980cc55d99359a0a" candidate_uuid: "197b880fa39d432a9b472845876bfaf0" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "1c50de89a5a845668696dcb28c3badd6" is_pre_election: true
I20260129 02:59:49.789815 20511 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e33a417cf9c54cf5af73a4ee3aac4d3a" candidate_uuid: "197b880fa39d432a9b472845876bfaf0" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "974b802141bd4f13beffef2aa03f88f9" is_pre_election: true
I20260129 02:59:49.786238 20508 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7a9a69cd82724b1faed1d10aa091fc4a" candidate_uuid: "197b880fa39d432a9b472845876bfaf0" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "974b802141bd4f13beffef2aa03f88f9"
I20260129 02:59:49.790771 20511 raft_consensus.cc:2410] T e33a417cf9c54cf5af73a4ee3aac4d3a P 974b802141bd4f13beffef2aa03f88f9 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 197b880fa39d432a9b472845876bfaf0 for term 6 because replica has last-logged OpId of term: 4 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 2.
I20260129 02:59:49.790916 20508 raft_consensus.cc:2376] T 7a9a69cd82724b1faed1d10aa091fc4a P 974b802141bd4f13beffef2aa03f88f9 [term 5 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 197b880fa39d432a9b472845876bfaf0 in term 5. Re-sending same reply.
I20260129 02:59:49.782894 20589 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a1a39dcd328e4268ae41b988c66b4a75" candidate_uuid: "197b880fa39d432a9b472845876bfaf0" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "1c50de89a5a845668696dcb28c3badd6" is_pre_election: true
I20260129 02:59:49.793123 20407 leader_election.cc:304] T e7d83d4ccf684e76acb29a0dc844e1b8 P 197b880fa39d432a9b472845876bfaf0 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 197b880fa39d432a9b472845876bfaf0, 974b802141bd4f13beffef2aa03f88f9; no voters:
I20260129 02:59:49.795743 20512 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e33a417cf9c54cf5af73a4ee3aac4d3a" candidate_uuid: "1c50de89a5a845668696dcb28c3badd6" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "974b802141bd4f13beffef2aa03f88f9"
I20260129 02:59:49.795780 20480 leader_election.cc:304] T cb0b3f259ebc4310927338a8740e87b8 P 974b802141bd4f13beffef2aa03f88f9 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 974b802141bd4f13beffef2aa03f88f9; no voters: 1c50de89a5a845668696dcb28c3badd6, 7a9daf5d42a84615a6383cfec9a10608
I20260129 02:59:49.758376 22372 raft_consensus.cc:740] T 75b0b6e82ddc4f92b1fe1d94f9a0b73b P 7a9daf5d42a84615a6383cfec9a10608 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 7a9daf5d42a84615a6383cfec9a10608, State: Running, Role: LEADER
I20260129 02:59:49.775664 22672 raft_consensus.cc:2804] T 18f30ebaa4174fd28c958e72d0cc7e24 P 7a9daf5d42a84615a6383cfec9a10608 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260129 02:59:49.762013 22677 raft_consensus.cc:2804] T 75be4bd6bd45474795fd472eff2953ac P 197b880fa39d432a9b472845876bfaf0 [term 8 FOLLOWER]: Leader election won for term 8
I20260129 02:59:49.800110 20508 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "8055232cf0004d40a85f3bf983fe00a9" candidate_uuid: "197b880fa39d432a9b472845876bfaf0" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "974b802141bd4f13beffef2aa03f88f9" is_pre_election: true
I20260129 02:59:49.795636 20407 leader_election.cc:304] T 7a9a69cd82724b1faed1d10aa091fc4a P 197b880fa39d432a9b472845876bfaf0 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 197b880fa39d432a9b472845876bfaf0, 974b802141bd4f13beffef2aa03f88f9; no voters:
I20260129 02:59:49.731436 22737 consensus_queue.cc:260] T fcddab6bca684adf8ea032289a00f90e P 197b880fa39d432a9b472845876bfaf0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 7.5, Last appended by leader: 5, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:59:49.779515 22700 raft_consensus.cc:3060] T b189f08e16a74a11980cc55d99359a0a P 974b802141bd4f13beffef2aa03f88f9 [term 5 FOLLOWER]: Advancing to term 6
I20260129 02:59:49.804853 20581 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e7d83d4ccf684e76acb29a0dc844e1b8" candidate_uuid: "197b880fa39d432a9b472845876bfaf0" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "1c50de89a5a845668696dcb28c3badd6" is_pre_election: true
I20260129 02:59:49.785786 22645 raft_consensus.cc:2804] T a5a05fbcbbcf4916bbc01fd1a5b5e90d P 197b880fa39d432a9b472845876bfaf0 [term 6 FOLLOWER]: Leader election won for term 6
I20260129 02:59:49.721403 22637 raft_consensus.cc:515] T a1a39dcd328e4268ae41b988c66b4a75 P 1c50de89a5a845668696dcb28c3badd6 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } } peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } }
I20260129 02:59:49.779410 22445 raft_consensus.cc:3060] T a1a39dcd328e4268ae41b988c66b4a75 P 974b802141bd4f13beffef2aa03f88f9 [term 2 FOLLOWER]: Advancing to term 3
I20260129 02:59:49.789129 22493 raft_consensus.cc:2804] T e4d18c35b0a7493e85d0a4b92956a48f P 974b802141bd4f13beffef2aa03f88f9 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260129 02:59:49.733930 22654 consensus_queue.cc:1048] T 18f30ebaa4174fd28c958e72d0cc7e24 P 1c50de89a5a845668696dcb28c3badd6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.053s
I20260129 02:59:49.857013 20437 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e4d18c35b0a7493e85d0a4b92956a48f" candidate_uuid: "1c50de89a5a845668696dcb28c3badd6" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "197b880fa39d432a9b472845876bfaf0" is_pre_election: true
I20260129 02:59:49.826069 20515 raft_consensus.cc:2468] T 1ce4ddf5b10e4ae8b193b13b2e5936ee P 974b802141bd4f13beffef2aa03f88f9 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1c50de89a5a845668696dcb28c3badd6 in term 6.
I20260129 02:59:49.874321 20584 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4f8f3a38f6e24ba4857875181f447442" candidate_uuid: "197b880fa39d432a9b472845876bfaf0" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "1c50de89a5a845668696dcb28c3badd6" is_pre_election: true
I20260129 02:59:49.873667 20437 raft_consensus.cc:2468] T e4d18c35b0a7493e85d0a4b92956a48f P 197b880fa39d432a9b472845876bfaf0 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1c50de89a5a845668696dcb28c3badd6 in term 4.
I20260129 02:59:49.804764 22184 raft_consensus.cc:2749] T add8d4923d1a489e86882c3a25692b95 P 974b802141bd4f13beffef2aa03f88f9 [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260129 02:59:49.603622 22765 consensus_queue.cc:1059] T 368953c9102643b2a21f99af36b51905 P 974b802141bd4f13beffef2aa03f88f9 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 }, Status: INVALID_TERM, Last received: 0.0, Next index: 4, Last known committed idx: 2, Time since last communication: 0.153s
W20260129 02:59:49.824230 21100 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.211.3:37735, user_credentials={real_user=slave}} blocked reactor thread for 357863us
W20260129 02:59:49.776413 22687 consensus_queue.cc:1175] T b189f08e16a74a11980cc55d99359a0a P 974b802141bd4f13beffef2aa03f88f9 [NON_LEADER]: Queue is closed or peer was untracked, disregarding peer response. Response: responder_uuid: "197b880fa39d432a9b472845876bfaf0" responder_term: 6 status { last_received { term: 4 index: 3 } last_committed_idx: 3 error { code: INVALID_TERM status { code: ILLEGAL_STATE message: "Rejecting Update request from peer 974b802141bd4f13beffef2aa03f88f9 for earlier term 5. Current term is 6. Ops: [5.4-5.4]" } } last_received_current_leader { term: 0 index: 0 } }
I20260129 02:59:49.749941 22743 raft_consensus.cc:515] T 75be4bd6bd45474795fd472eff2953ac P 1c50de89a5a845668696dcb28c3badd6 [term 9 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
W20260129 02:59:49.869141 20480 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.211.4:40439, user_credentials={real_user=slave}} blocked reactor thread for 73426.7us
I20260129 02:59:49.900709 20584 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7a9a69cd82724b1faed1d10aa091fc4a" candidate_uuid: "197b880fa39d432a9b472845876bfaf0" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "1c50de89a5a845668696dcb28c3badd6"
I20260129 02:59:49.903009 20480 leader_election.cc:304] T 2ac659d732314e1288849f08fe5e074e P 974b802141bd4f13beffef2aa03f88f9 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 7a9daf5d42a84615a6383cfec9a10608, 974b802141bd4f13beffef2aa03f88f9; no voters: 197b880fa39d432a9b472845876bfaf0
I20260129 02:59:49.871395 20581 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7d9494f5584d4d5db8b9adf1c8fc888b" candidate_uuid: "197b880fa39d432a9b472845876bfaf0" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "1c50de89a5a845668696dcb28c3badd6" is_pre_election: true
I20260129 02:59:49.874419 20515 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00e79d8f25a34676b1d866eb2068a2b2" candidate_uuid: "197b880fa39d432a9b472845876bfaf0" candidate_term: 6 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "974b802141bd4f13beffef2aa03f88f9" is_pre_election: true
W20260129 02:59:49.841884 20554 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260129 02:59:49.800452 22237 raft_consensus.cc:2804] T 7d9494f5584d4d5db8b9adf1c8fc888b P 197b880fa39d432a9b472845876bfaf0 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260129 02:59:49.811030 20582 raft_consensus.cc:2468] T 913fc7d751ed4b7388d889aac30a5f9b P 1c50de89a5a845668696dcb28c3badd6 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 197b880fa39d432a9b472845876bfaf0 in term 3.
I20260129 02:59:49.877897 20512 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "cb4727386a3e48aca5d71cc6c611c197" candidate_uuid: "197b880fa39d432a9b472845876bfaf0" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "974b802141bd4f13beffef2aa03f88f9"
I20260129 02:59:49.880093 20511 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "cedce331e1584ac8b636c9b8a72b2ba0" candidate_uuid: "197b880fa39d432a9b472845876bfaf0" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "974b802141bd4f13beffef2aa03f88f9" is_pre_election: true
I20260129 02:59:49.907011 20511 raft_consensus.cc:2468] T cedce331e1584ac8b636c9b8a72b2ba0 P 974b802141bd4f13beffef2aa03f88f9 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 197b880fa39d432a9b472845876bfaf0 in term 5.
I20260129 02:59:49.856822 22584 raft_consensus.cc:493] T 2e58e9a5a8aa4cecafb464addb4f35ed P 1c50de89a5a845668696dcb28c3badd6 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 7a9daf5d42a84615a6383cfec9a10608)
I20260129 02:59:49.879269 20553 leader_election.cc:304] T e4d18c35b0a7493e85d0a4b92956a48f P 1c50de89a5a845668696dcb28c3badd6 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 197b880fa39d432a9b472845876bfaf0, 1c50de89a5a845668696dcb28c3badd6; no voters:
I20260129 02:59:49.757481 22337 consensus_queue.cc:237] T 78aa39096868415d9afc2ffcb248e198 P 1c50de89a5a845668696dcb28c3badd6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.5, Last appended by leader: 5, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } }
I20260129 02:59:49.808444 22672 raft_consensus.cc:493] T 18f30ebaa4174fd28c958e72d0cc7e24 P 7a9daf5d42a84615a6383cfec9a10608 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260129 02:59:49.886226 20481 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260129 02:59:49.883208 21100 leader_election.cc:304] T 1299723a4476420ab02fd8bfd7aa0645 P 7a9daf5d42a84615a6383cfec9a10608 [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1c50de89a5a845668696dcb28c3badd6, 7a9daf5d42a84615a6383cfec9a10608; no voters: 197b880fa39d432a9b472845876bfaf0
W20260129 02:59:49.917701 21101 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.509s user 0.043s sys 0.051s
I20260129 02:59:49.805269 22641 raft_consensus.cc:2804] T e7d83d4ccf684e76acb29a0dc844e1b8 P 197b880fa39d432a9b472845876bfaf0 [term 4 FOLLOWER]: Leader pre-election won for term 5
W20260129 02:59:49.932222 21100 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.211.3:37735, user_credentials={real_user=slave}} blocked reactor thread for 50908.4us
I20260129 02:59:49.885178 22737 raft_consensus.cc:3060] T fcddab6bca684adf8ea032289a00f90e P 197b880fa39d432a9b472845876bfaf0 [term 7 FOLLOWER]: Advancing to term 8
W20260129 02:59:49.913897 20479 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260129 02:59:49.915469 22758 raft_consensus.cc:2804] T 7a9a69cd82724b1faed1d10aa091fc4a P 197b880fa39d432a9b472845876bfaf0 [term 5 FOLLOWER]: Leader election won for term 5
W20260129 02:59:49.918617 21102 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.673s user 0.011s sys 0.014s
I20260129 02:59:49.874177 22493 raft_consensus.cc:493] T e4d18c35b0a7493e85d0a4b92956a48f P 974b802141bd4f13beffef2aa03f88f9 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260129 02:59:49.924676 20555 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260129 02:59:49.876247 22699 raft_consensus.cc:2749] T cb0b3f259ebc4310927338a8740e87b8 P 974b802141bd4f13beffef2aa03f88f9 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
W20260129 02:59:49.969700 20480 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.211.4:40439, user_credentials={real_user=slave}} blocked reactor thread for 66787.5us
I20260129 02:59:50.256484 22493 raft_consensus.cc:3060] T e4d18c35b0a7493e85d0a4b92956a48f P 974b802141bd4f13beffef2aa03f88f9 [term 4 FOLLOWER]: Advancing to term 5
I20260129 02:59:50.256899 20480 leader_election.cc:304] T a644c02b16af47679824ccd060ac10fd P 974b802141bd4f13beffef2aa03f88f9 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 974b802141bd4f13beffef2aa03f88f9; no voters: 1c50de89a5a845668696dcb28c3badd6, 7a9daf5d42a84615a6383cfec9a10608
I20260129 02:59:50.258116 20480 leader_election.cc:304] T febffe4a8b394368b69b1d164308fb9a P 974b802141bd4f13beffef2aa03f88f9 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 974b802141bd4f13beffef2aa03f88f9; no voters: 197b880fa39d432a9b472845876bfaf0, 7a9daf5d42a84615a6383cfec9a10608
I20260129 02:59:50.258280 22699 raft_consensus.cc:493] T 36ff33be666a4c82954731691dbcf327 P 974b802141bd4f13beffef2aa03f88f9 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 1c50de89a5a845668696dcb28c3badd6)
I20260129 02:59:50.259233 20480 leader_election.cc:304] T 3e2d15f8640b407195210c476e09df91 P 974b802141bd4f13beffef2aa03f88f9 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 7a9daf5d42a84615a6383cfec9a10608, 974b802141bd4f13beffef2aa03f88f9; no voters: 1c50de89a5a845668696dcb28c3badd6
I20260129 02:59:49.873154 20509 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c891d2bff4ce4568861bbc992e91fbb0" candidate_uuid: "197b880fa39d432a9b472845876bfaf0" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "974b802141bd4f13beffef2aa03f88f9" is_pre_election: true
I20260129 02:59:50.001340 22765 raft_consensus.cc:493] T e33a417cf9c54cf5af73a4ee3aac4d3a P 974b802141bd4f13beffef2aa03f88f9 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260129 02:59:49.883701 20435 raft_consensus.cc:1240] T febffe4a8b394368b69b1d164308fb9a P 197b880fa39d432a9b472845876bfaf0 [term 3 LEADER]: Rejecting Update request from peer 7a9daf5d42a84615a6383cfec9a10608 for earlier term 2. Current term is 3. Ops: [2.5-2.5]
I20260129 02:59:49.927016 20584 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4e3805ffa70242e8ade66da260dddf66" candidate_uuid: "197b880fa39d432a9b472845876bfaf0" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "1c50de89a5a845668696dcb28c3badd6" is_pre_election: true
W20260129 02:59:49.927452 21101 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.519s user 0.043s sys 0.051s
I20260129 02:59:49.928925 20587 raft_consensus.cc:2368] T 75be4bd6bd45474795fd472eff2953ac P 1c50de89a5a845668696dcb28c3badd6 [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 197b880fa39d432a9b472845876bfaf0 for earlier term 8. Current term is 9.
I20260129 02:59:49.898952 20589 raft_consensus.cc:2468] T a1a39dcd328e4268ae41b988c66b4a75 P 1c50de89a5a845668696dcb28c3badd6 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 197b880fa39d432a9b472845876bfaf0 in term 3.
I20260129 02:59:49.876910 22784 raft_consensus.cc:493] T 3906b080a5b34a74b8b64128635e8997 P 974b802141bd4f13beffef2aa03f88f9 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260129 02:59:49.933094 20511 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b189f08e16a74a11980cc55d99359a0a" candidate_uuid: "1c50de89a5a845668696dcb28c3badd6" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "974b802141bd4f13beffef2aa03f88f9" is_pre_election: true
I20260129 02:59:49.909873 20512 raft_consensus.cc:2393] T cb4727386a3e48aca5d71cc6c611c197 P 974b802141bd4f13beffef2aa03f88f9 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 197b880fa39d432a9b472845876bfaf0 in current term 3: Already voted for candidate 974b802141bd4f13beffef2aa03f88f9 in this term.
I20260129 02:59:49.907218 22237 raft_consensus.cc:493] T 7d9494f5584d4d5db8b9adf1c8fc888b P 197b880fa39d432a9b472845876bfaf0 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 1c50de89a5a845668696dcb28c3badd6)
I20260129 02:59:49.911181 20582 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "78aa39096868415d9afc2ffcb248e198" candidate_uuid: "197b880fa39d432a9b472845876bfaf0" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "1c50de89a5a845668696dcb28c3badd6"
I20260129 02:59:49.925542 22584 raft_consensus.cc:515] T 2e58e9a5a8aa4cecafb464addb4f35ed P 1c50de89a5a845668696dcb28c3badd6 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } }
W20260129 02:59:49.947260 21102 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.701s user 0.011s sys 0.015s
I20260129 02:59:49.938882 22372 consensus_queue.cc:260] T 75b0b6e82ddc4f92b1fe1d94f9a0b73b P 7a9daf5d42a84615a6383cfec9a10608 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } }
I20260129 02:59:49.931670 22672 raft_consensus.cc:3060] T 18f30ebaa4174fd28c958e72d0cc7e24 P 7a9daf5d42a84615a6383cfec9a10608 [term 4 FOLLOWER]: Advancing to term 5
I20260129 02:59:49.931917 22641 raft_consensus.cc:493] T e7d83d4ccf684e76acb29a0dc844e1b8 P 197b880fa39d432a9b472845876bfaf0 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260129 02:59:50.188508 20405 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260129 02:59:50.293254 20405 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260129 02:59:50.293769 20405 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260129 02:59:50.207720 20553 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.211.1:42027, user_credentials={real_user=slave}} blocked reactor thread for 269438us
W20260129 02:59:50.295996 20553 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260129 02:59:49.946964 22779 raft_consensus.cc:2804] T 1299723a4476420ab02fd8bfd7aa0645 P 7a9daf5d42a84615a6383cfec9a10608 [term 8 FOLLOWER]: Leader election won for term 8
I20260129 02:59:50.133818 22724 leader_election.cc:290] T 724a236524cb4e7ba3428880681d6cc2 P 1c50de89a5a845668696dcb28c3badd6 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 197b880fa39d432a9b472845876bfaf0 (127.18.211.1:42027), 974b802141bd4f13beffef2aa03f88f9 (127.18.211.2:34517)
W20260129 02:59:50.254253 20554 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260129 02:59:50.261305 20509 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "76f713c3331540d2a5b76164103e9eb1" candidate_uuid: "1c50de89a5a845668696dcb28c3badd6" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "974b802141bd4f13beffef2aa03f88f9" is_pre_election: true
I20260129 02:59:49.926062 20581 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e7d83d4ccf684e76acb29a0dc844e1b8" candidate_uuid: "197b880fa39d432a9b472845876bfaf0" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "1c50de89a5a845668696dcb28c3badd6" is_pre_election: true
I20260129 02:59:50.259068 22699 raft_consensus.cc:515] T 36ff33be666a4c82954731691dbcf327 P 974b802141bd4f13beffef2aa03f88f9 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
W20260129 02:59:50.032341 21100 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260129 02:59:50.263564 20555 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260129 02:59:49.938813 22184 raft_consensus.cc:2804] T 2ac659d732314e1288849f08fe5e074e P 974b802141bd4f13beffef2aa03f88f9 [term 3 FOLLOWER]: Leader election won for term 3
I20260129 02:59:50.274503 22493 raft_consensus.cc:515] T e4d18c35b0a7493e85d0a4b92956a48f P 974b802141bd4f13beffef2aa03f88f9 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:59:50.276880 21132 raft_consensus.cc:1275] T dc2f4cb752cf4d59b5f6bc689f5d8104 P 7a9daf5d42a84615a6383cfec9a10608 [term 4 FOLLOWER]: Refusing update from remote peer 1c50de89a5a845668696dcb28c3badd6: Log matching property violated. Preceding OpId in replica: term: 3 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
W20260129 02:59:50.281339 21101 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260129 02:59:50.204303 20407 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.211.2:34517, user_credentials={real_user=slave}} blocked reactor thread for 408682us
W20260129 02:59:50.337183 20407 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260129 02:59:50.029666 22780 log.cc:927] Time spent T dc2f4cb752cf4d59b5f6bc689f5d8104 P 1c50de89a5a845668696dcb28c3badd6: Append to log took a long time: real 0.166s user 0.000s sys 0.001s
W20260129 02:59:50.287449 21102 connection.cc:232] RPC call timeout handler was delayed by -1.0260485779999726s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2457s for a response
W20260129 02:59:50.371757 21102 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260129 02:59:50.092939 22645 raft_consensus.cc:697] T a5a05fbcbbcf4916bbc01fd1a5b5e90d P 197b880fa39d432a9b472845876bfaf0 [term 6 LEADER]: Becoming Leader. State: Replica: 197b880fa39d432a9b472845876bfaf0, State: Running, Role: LEADER
I20260129 02:59:49.959563 22758 raft_consensus.cc:697] T 7a9a69cd82724b1faed1d10aa091fc4a P 197b880fa39d432a9b472845876bfaf0 [term 5 LEADER]: Becoming Leader. State: Replica: 197b880fa39d432a9b472845876bfaf0, State: Running, Role: LEADER
I20260129 02:59:50.034617 22750 leader_election.cc:290] T a0fcc58d6fb6473cafbd9d34d5195269 P 197b880fa39d432a9b472845876bfaf0 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 974b802141bd4f13beffef2aa03f88f9 (127.18.211.2:34517), 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3:37735)
I20260129 02:59:50.071107 22647 leader_election.cc:290] T c934d31efd3f4e5195605ddc354d14e1 P 197b880fa39d432a9b472845876bfaf0 [CANDIDATE]: Term 5 election: Requested vote from peers 974b802141bd4f13beffef2aa03f88f9 (127.18.211.2:34517), 7a9daf5d42a84615a6383cfec9a10608 (127.18.211.4:40439)
I20260129 02:59:50.363610 22400 leader_election.cc:290] T 75f45538dc704d9a8ec4ff8ad9f3ea5b P 974b802141bd4f13beffef2aa03f88f9 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3:37735), 7a9daf5d42a84615a6383cfec9a10608 (127.18.211.4:40439)
I20260129 02:59:49.947827 22729 consensus_queue.cc:1059] T add8d4923d1a489e86882c3a25692b95 P 7a9daf5d42a84615a6383cfec9a10608 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.334s
I20260129 02:59:50.220196 22677 raft_consensus.cc:697] T 75be4bd6bd45474795fd472eff2953ac P 197b880fa39d432a9b472845876bfaf0 [term 8 LEADER]: Becoming Leader. State: Replica: 197b880fa39d432a9b472845876bfaf0, State: Running, Role: LEADER
I20260129 02:59:50.414289 22729 raft_consensus.cc:3055] T add8d4923d1a489e86882c3a25692b95 P 7a9daf5d42a84615a6383cfec9a10608 [term 2 LEADER]: Stepping down as leader of term 2
I20260129 02:59:50.414798 22729 raft_consensus.cc:740] T add8d4923d1a489e86882c3a25692b95 P 7a9daf5d42a84615a6383cfec9a10608 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 7a9daf5d42a84615a6383cfec9a10608, State: Running, Role: LEADER
I20260129 02:59:50.414616 22677 consensus_queue.cc:237] T 75be4bd6bd45474795fd472eff2953ac P 197b880fa39d432a9b472845876bfaf0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 7.5, Last appended by leader: 5, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:59:50.415897 22729 consensus_queue.cc:260] T add8d4923d1a489e86882c3a25692b95 P 7a9daf5d42a84615a6383cfec9a10608 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } }
I20260129 02:59:50.417431 22729 raft_consensus.cc:3060] T add8d4923d1a489e86882c3a25692b95 P 7a9daf5d42a84615a6383cfec9a10608 [term 2 FOLLOWER]: Advancing to term 3
I20260129 02:59:50.314105 20582 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e33a417cf9c54cf5af73a4ee3aac4d3a" candidate_uuid: "197b880fa39d432a9b472845876bfaf0" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "1c50de89a5a845668696dcb28c3badd6" is_pre_election: true
I20260129 02:59:50.314285 22743 leader_election.cc:290] T 75be4bd6bd45474795fd472eff2953ac P 1c50de89a5a845668696dcb28c3badd6 [CANDIDATE]: Term 9 election: Requested vote from peers 197b880fa39d432a9b472845876bfaf0 (127.18.211.1:42027), 974b802141bd4f13beffef2aa03f88f9 (127.18.211.2:34517)
I20260129 02:59:50.418032 21135 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34060 (request call id 491) took 1951 ms. Trace:
I20260129 02:59:50.418442 21135 rpcz_store.cc:276] 0129 02:59:48.466870 (+ 0us) service_pool.cc:168] Inserting onto call queue
0129 02:59:48.467011 (+ 141us) service_pool.cc:225] Handling call
0129 02:59:50.417810 (+1950799us) spinlock_profiling.cc:243] Waited 990 ms on lock 0x7b5400670cd4. stack: 00007f8fe044e373 00007f8fe044e270 00007f8fdfda0574 00007f8fe9a5c6b4 00007f8fe9a5c649 00007f8fe34975a1 00007f8fe3531f7a 00007f8fe84e3647 00007f8fe1a73834 00007f8fe1a737c2 00007f8fe1a736f1 00007f8fe1a7366c 00007f8fe1a72922 00007f8fe186d34c 00007f8fe186c786 00007f8fe186c12f
0129 02:59:50.417976 (+ 166us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2276577792}
I20260129 02:59:50.289775 22237 raft_consensus.cc:3060] T 7d9494f5584d4d5db8b9adf1c8fc888b P 197b880fa39d432a9b472845876bfaf0 [term 2 FOLLOWER]: Advancing to term 3
I20260129 02:59:50.328557 20584 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b189f08e16a74a11980cc55d99359a0a" candidate_uuid: "197b880fa39d432a9b472845876bfaf0" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "1c50de89a5a845668696dcb28c3badd6" is_pre_election: true
W20260129 02:59:50.329928 20588 consensus_meta.cc:287] T b189f08e16a74a11980cc55d99359a0a P 1c50de89a5a845668696dcb28c3badd6: Time spent flushing consensus metadata: real 0.540s user 0.011s sys 0.000s
I20260129 02:59:50.420928 20588 raft_consensus.cc:2468] T b189f08e16a74a11980cc55d99359a0a P 1c50de89a5a845668696dcb28c3badd6 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 197b880fa39d432a9b472845876bfaf0 in term 6.
I20260129 02:59:50.422032 20588 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "75be4bd6bd45474795fd472eff2953ac" candidate_uuid: "197b880fa39d432a9b472845876bfaf0" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "1c50de89a5a845668696dcb28c3badd6" is_pre_election: true
I20260129 02:59:50.340548 22803 raft_consensus.cc:493] T 42d27a28d89b409e8807ec0efd97af69 P 197b880fa39d432a9b472845876bfaf0 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260129 02:59:50.342098 22787 log.cc:927] Time spent T 368953c9102643b2a21f99af36b51905 P 1c50de89a5a845668696dcb28c3badd6: Append to log took a long time: real 0.310s user 0.000s sys 0.001s
I20260129 02:59:50.343684 22802 raft_consensus.cc:493] T 3e2d15f8640b407195210c476e09df91 P 1c50de89a5a845668696dcb28c3badd6 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260129 02:59:50.346480 21136 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34046 (request call id 454) took 2550 ms. Trace:
W20260129 02:59:50.348985 22763 consensus_meta.cc:287] T 00e79d8f25a34676b1d866eb2068a2b2 P 974b802141bd4f13beffef2aa03f88f9: Time spent flushing consensus metadata: real 0.632s user 0.009s sys 0.000s
I20260129 02:59:50.336078 22779 raft_consensus.cc:697] T 1299723a4476420ab02fd8bfd7aa0645 P 7a9daf5d42a84615a6383cfec9a10608 [term 8 LEADER]: Becoming Leader. State: Replica: 7a9daf5d42a84615a6383cfec9a10608, State: Running, Role: LEADER
I20260129 02:59:50.291996 22784 raft_consensus.cc:515] T 3906b080a5b34a74b8b64128635e8997 P 974b802141bd4f13beffef2aa03f88f9 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:59:50.335071 20509 raft_consensus.cc:2468] T 76f713c3331540d2a5b76164103e9eb1 P 974b802141bd4f13beffef2aa03f88f9 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1c50de89a5a845668696dcb28c3badd6 in term 4.
I20260129 02:59:50.357064 22584 leader_election.cc:290] T 2e58e9a5a8aa4cecafb464addb4f35ed P 1c50de89a5a845668696dcb28c3badd6 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 197b880fa39d432a9b472845876bfaf0 (127.18.211.1:42027), 7a9daf5d42a84615a6383cfec9a10608 (127.18.211.4:40439)
W20260129 02:59:50.358315 20514 consensus_meta.cc:287] T a4c1226b987b468fa84dfc6ba0e2dbe1 P 974b802141bd4f13beffef2aa03f88f9: Time spent flushing consensus metadata: real 0.582s user 0.007s sys 0.001s
I20260129 02:59:50.358521 20433 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55672 (request call id 1532) took 1066 ms. Trace:
I20260129 02:59:50.360570 20507 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57600 (request call id 1747) took 3942 ms. Trace:
I20260129 02:59:50.364979 22493 leader_election.cc:290] T e4d18c35b0a7493e85d0a4b92956a48f P 974b802141bd4f13beffef2aa03f88f9 [CANDIDATE]: Term 5 election: Requested vote from peers 197b880fa39d432a9b472845876bfaf0 (127.18.211.1:42027), 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3:37735)
I20260129 02:59:50.367271 22699 leader_election.cc:290] T 36ff33be666a4c82954731691dbcf327 P 974b802141bd4f13beffef2aa03f88f9 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 197b880fa39d432a9b472845876bfaf0 (127.18.211.1:42027), 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3:37735)
W20260129 02:59:50.370402 20556 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260129 02:59:50.467437 20433 rpcz_store.cc:276] 0129 02:59:49.292629 (+ 0us) service_pool.cc:168] Inserting onto call queue
0129 02:59:49.458375 (+165746us) service_pool.cc:225] Handling call
0129 02:59:49.458670 (+ 295us) raft_consensus.cc:1471] Updating replica for 0 ops
0129 02:59:49.742314 (+283644us) raft_consensus.cc:1534] Early marking committed up to index 0
0129 02:59:49.742346 (+ 32us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0129 02:59:49.742391 (+ 45us) raft_consensus.cc:1645] Marking committed up to 3
0129 02:59:49.742426 (+ 35us) raft_consensus.cc:1697] Filling consensus response to leader.
0129 02:59:50.358213 (+615787us) spinlock_profiling.cc:243] Waited 283 ms on lock 0x7b54003237d4. stack: 00007f8fe044e373 00007f8fe044e270 00007f8fdfda0574 00007f8fe9a5c6b4 00007f8fe9a5c649 00007f8fe34975a1 00007f8fe352ea81 00007f8fe352d4be 00007f8fe84e1f49 00007f8fe1a6ed34 00007f8fe1a6ecc2 00007f8fe1a6ebf1 00007f8fe1a6eb6c 00007f8fe1a6de22 00007f8fe186d34c 00007f8fe186c786
0129 02:59:50.358275 (+ 62us) raft_consensus.cc:1691] UpdateReplicas() finished
0129 02:59:50.358444 (+ 169us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":651836160}
I20260129 02:59:50.370244 20581 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "913fc7d751ed4b7388d889aac30a5f9b" candidate_uuid: "197b880fa39d432a9b472845876bfaf0" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1c50de89a5a845668696dcb28c3badd6" is_pre_election: true
I20260129 02:59:50.470952 20581 raft_consensus.cc:2376] T 913fc7d751ed4b7388d889aac30a5f9b P 1c50de89a5a845668696dcb28c3badd6 [term 3 FOLLOWER]: Leader pre-election vote request: Already granted yes vote for candidate 197b880fa39d432a9b472845876bfaf0 in term 3. Re-sending same reply.
I20260129 02:59:50.370513 20440 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c934d31efd3f4e5195605ddc354d14e1" candidate_uuid: "7a9daf5d42a84615a6383cfec9a10608" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "197b880fa39d432a9b472845876bfaf0"
I20260129 02:59:50.472280 20440 raft_consensus.cc:2368] T c934d31efd3f4e5195605ddc354d14e1 P 197b880fa39d432a9b472845876bfaf0 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 7a9daf5d42a84615a6383cfec9a10608 for earlier term 4. Current term is 5.
I20260129 02:59:50.472754 20440 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:48356 (request call id 302) took 1625 ms. Trace:
I20260129 02:59:50.473116 20440 rpcz_store.cc:276] 0129 02:59:48.846878 (+ 0us) service_pool.cc:168] Inserting onto call queue
0129 02:59:49.406340 (+559462us) service_pool.cc:225] Handling call
0129 02:59:50.370343 (+964003us) spinlock_profiling.cc:243] Waited 6.02 ms on lock 0x7b5800000f98. stack: 00007f8fe044e373 00007f8fe044e270 00007f8fdfda0574 00007f8fe9a5c6b4 00007f8fe9a5c649 00007f8fe02c08b1 00007f8fe0442db1 00007f8fe044312e 000000000035a3c9 00007f8fe846d560 00007f8fe8535dd4 00007f8fe8535e41 00007f8fe84e3bd0 00007f8fe84e34b4 00007f8fe1a73834 00007f8fe1a737c2
0129 02:59:50.472690 (+102347us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":13949056}
W20260129 02:59:50.370980 20482 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260129 02:59:50.371191 20407 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260129 02:59:50.298096 22637 leader_election.cc:290] T a1a39dcd328e4268ae41b988c66b4a75 P 1c50de89a5a845668696dcb28c3badd6 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 974b802141bd4f13beffef2aa03f88f9 (127.18.211.2:34517), 197b880fa39d432a9b472845876bfaf0 (127.18.211.1:42027)
W20260129 02:59:50.121913 22783 log.cc:927] Time spent T c2633a6fc34c44579de348d208b0c01e P 974b802141bd4f13beffef2aa03f88f9: Append to log took a long time: real 0.418s user 0.001s sys 0.000s
I20260129 02:59:50.297981 22641 raft_consensus.cc:3060] T e7d83d4ccf684e76acb29a0dc844e1b8 P 197b880fa39d432a9b472845876bfaf0 [term 4 FOLLOWER]: Advancing to term 5
I20260129 02:59:50.284883 22765 raft_consensus.cc:515] T e33a417cf9c54cf5af73a4ee3aac4d3a P 974b802141bd4f13beffef2aa03f88f9 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
W20260129 02:59:50.298628 22781 log.cc:927] Time spent T 0340229d521b432580ae926f6062e3f7 P 197b880fa39d432a9b472845876bfaf0: Append to log took a long time: real 0.166s user 0.003s sys 0.006s
W20260129 02:59:50.383644 20405 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.211.4:40439, user_credentials={real_user=slave}} blocked reactor thread for 89354us
I20260129 02:59:50.488605 20581 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a4c1226b987b468fa84dfc6ba0e2dbe1" candidate_uuid: "197b880fa39d432a9b472845876bfaf0" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "1c50de89a5a845668696dcb28c3badd6"
I20260129 02:59:50.489039 20440 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2e58e9a5a8aa4cecafb464addb4f35ed" candidate_uuid: "1c50de89a5a845668696dcb28c3badd6" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "197b880fa39d432a9b472845876bfaf0" is_pre_election: true
I20260129 02:59:50.489543 20407 leader_election.cc:304] T cedce331e1584ac8b636c9b8a72b2ba0 P 197b880fa39d432a9b472845876bfaf0 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 197b880fa39d432a9b472845876bfaf0, 974b802141bd4f13beffef2aa03f88f9; no voters:
I20260129 02:59:50.260423 20480 leader_election.cc:304] T 0678b8b468bd43cdbc3e485900844821 P 974b802141bd4f13beffef2aa03f88f9 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 7a9daf5d42a84615a6383cfec9a10608, 974b802141bd4f13beffef2aa03f88f9; no voters: 197b880fa39d432a9b472845876bfaf0
W20260129 02:59:50.491643 20480 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.211.4:40439, user_credentials={real_user=slave}} blocked reactor thread for 231277us
I20260129 02:59:50.334872 22372 raft_consensus.cc:3060] T 75b0b6e82ddc4f92b1fe1d94f9a0b73b P 7a9daf5d42a84615a6383cfec9a10608 [term 2 FOLLOWER]: Advancing to term 3
W20260129 02:59:50.401491 22445 consensus_meta.cc:287] T a1a39dcd328e4268ae41b988c66b4a75 P 974b802141bd4f13beffef2aa03f88f9: Time spent flushing consensus metadata: real 0.523s user 0.004s sys 0.004s
I20260129 02:59:50.402171 21129 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1299723a4476420ab02fd8bfd7aa0645" candidate_uuid: "197b880fa39d432a9b472845876bfaf0" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "7a9daf5d42a84615a6383cfec9a10608" is_pre_election: true
I20260129 02:59:50.404426 22758 consensus_queue.cc:237] T 7a9a69cd82724b1faed1d10aa091fc4a P 197b880fa39d432a9b472845876bfaf0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 4.3, Last appended by leader: 3, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
W20260129 02:59:50.410156 22688 consensus_meta.cc:287] T 4e3805ffa70242e8ade66da260dddf66 P 197b880fa39d432a9b472845876bfaf0: Time spent flushing consensus metadata: real 0.792s user 0.008s sys 0.000s
W20260129 02:59:50.410874 22700 consensus_meta.cc:287] T b189f08e16a74a11980cc55d99359a0a P 974b802141bd4f13beffef2aa03f88f9: Time spent flushing consensus metadata: real 0.543s user 0.001s sys 0.004s
I20260129 02:59:50.412480 22540 raft_consensus.cc:3060] T 2ac659d732314e1288849f08fe5e074e P 197b880fa39d432a9b472845876bfaf0 [term 2 FOLLOWER]: Advancing to term 3
W20260129 02:59:50.413108 22740 consensus_meta.cc:287] T 42d27a28d89b409e8807ec0efd97af69 P 1c50de89a5a845668696dcb28c3badd6: Time spent flushing consensus metadata: real 0.834s user 0.010s sys 0.000s
I20260129 02:59:49.957512 22786 raft_consensus.cc:2804] T e4d18c35b0a7493e85d0a4b92956a48f P 1c50de89a5a845668696dcb28c3badd6 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260129 02:59:50.425585 20588 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fd8311b779af4e1fa1402b2a02a602f2" candidate_uuid: "197b880fa39d432a9b472845876bfaf0" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "1c50de89a5a845668696dcb28c3badd6" is_pre_election: true
I20260129 02:59:50.375247 22722 raft_consensus.cc:2749] T a644c02b16af47679824ccd060ac10fd P 974b802141bd4f13beffef2aa03f88f9 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260129 02:59:50.375003 21128 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "505580c93c694bbb8ea96fdf9b17ab46" candidate_uuid: "1c50de89a5a845668696dcb28c3badd6" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "7a9daf5d42a84615a6383cfec9a10608" is_pre_election: true
I20260129 02:59:50.428184 21136 rpcz_store.cc:276] 0129 02:59:47.795829 (+ 0us) service_pool.cc:168] Inserting onto call queue
0129 02:59:47.796089 (+ 260us) service_pool.cc:225] Handling call
0129 02:59:47.826169 (+ 30080us) raft_consensus.cc:1471] Updating replica for 0 ops
0129 02:59:47.924090 (+ 97921us) raft_consensus.cc:1534] Early marking committed up to index 0
0129 02:59:47.924126 (+ 36us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0129 02:59:47.924168 (+ 42us) raft_consensus.cc:1645] Marking committed up to 5
0129 02:59:47.924198 (+ 30us) raft_consensus.cc:1697] Filling consensus response to leader.
0129 02:59:50.346175 (+2421977us) spinlock_profiling.cc:243] Waited 97.7 ms on lock 0x7b54003d9ad4. stack: 00007f8fe044e373 00007f8fe044e270 00007f8fdfda0574 00007f8fe9a5c6b4 00007f8fe9a5c649 00007f8fe34975a1 00007f8fe352ea81 00007f8fe352d4be 00007f8fe84e1f49 00007f8fe1a6ed34 00007f8fe1a6ecc2 00007f8fe1a6ebf1 00007f8fe1a6eb6c 00007f8fe1a6de22 00007f8fe186d34c 00007f8fe186c786
0129 02:59:50.346236 (+ 61us) raft_consensus.cc:1691] UpdateReplicas() finished
0129 02:59:50.346400 (+ 164us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":224659328}
W20260129 02:59:50.587960 20554 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.211.4:40439, user_credentials={real_user=slave}} blocked reactor thread for 252189us
I20260129 02:59:50.375636 22687 raft_consensus.cc:2749] T febffe4a8b394368b69b1d164308fb9a P 974b802141bd4f13beffef2aa03f88f9 [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260129 02:59:50.461009 20582 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "18f30ebaa4174fd28c958e72d0cc7e24" candidate_uuid: "7a9daf5d42a84615a6383cfec9a10608" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "1c50de89a5a845668696dcb28c3badd6" is_pre_election: true
I20260129 02:59:50.467371 20514 raft_consensus.cc:2468] T a4c1226b987b468fa84dfc6ba0e2dbe1 P 974b802141bd4f13beffef2aa03f88f9 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 197b880fa39d432a9b472845876bfaf0 in term 4.
I20260129 02:59:50.468170 20507 rpcz_store.cc:276] 0129 02:59:46.417808 (+ 0us) service_pool.cc:168] Inserting onto call queue
0129 02:59:48.109159 (+1691351us) service_pool.cc:225] Handling call
0129 02:59:48.154353 (+ 45194us) raft_consensus.cc:1471] Updating replica for 1 ops
0129 02:59:48.377462 (+223109us) raft_consensus.cc:1697] Filling consensus response to leader.
0129 02:59:50.360255 (+1982793us) spinlock_profiling.cc:243] Waited 176 ms on lock 0x7b54002ef254. stack: 00007f8fe044e373 00007f8fe044e270 00007f8fdfda0574 00007f8fe9a5c6b4 00007f8fe9a5c649 00007f8fe34975a1 00007f8fe352ea81 00007f8fe352d4be 00007f8fe84e1f49 00007f8fe1a6ed34 00007f8fe1a6ecc2 00007f8fe1a6ebf1 00007f8fe1a6eb6c 00007f8fe1a6de22 00007f8fe186d34c 00007f8fe186c786
0129 02:59:50.360492 (+ 237us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":404590592}
I20260129 02:59:50.475513 20584 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "cb4727386a3e48aca5d71cc6c611c197" candidate_uuid: "197b880fa39d432a9b472845876bfaf0" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1c50de89a5a845668696dcb28c3badd6"
I20260129 02:59:50.595323 20514 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a0fcc58d6fb6473cafbd9d34d5195269" candidate_uuid: "197b880fa39d432a9b472845876bfaf0" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "974b802141bd4f13beffef2aa03f88f9" is_pre_election: true
I20260129 02:59:50.595547 20584 raft_consensus.cc:3060] T cb4727386a3e48aca5d71cc6c611c197 P 1c50de89a5a845668696dcb28c3badd6 [term 2 FOLLOWER]: Advancing to term 3
I20260129 02:59:50.596925 20512 raft_consensus.cc:1240] T a4c1226b987b468fa84dfc6ba0e2dbe1 P 974b802141bd4f13beffef2aa03f88f9 [term 4 FOLLOWER]: Rejecting Update request from peer 1c50de89a5a845668696dcb28c3badd6 for earlier term 3. Current term is 4. Ops: []
I20260129 02:59:50.426956 22400 raft_consensus.cc:2804] T 3e2d15f8640b407195210c476e09df91 P 974b802141bd4f13beffef2aa03f88f9 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260129 02:59:50.426823 21132 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c934d31efd3f4e5195605ddc354d14e1" candidate_uuid: "197b880fa39d432a9b472845876bfaf0" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "7a9daf5d42a84615a6383cfec9a10608"
I20260129 02:59:50.432812 20436 raft_consensus.cc:1275] T 368953c9102643b2a21f99af36b51905 P 197b880fa39d432a9b472845876bfaf0 [term 6 FOLLOWER]: Refusing update from remote peer 1c50de89a5a845668696dcb28c3badd6: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20260129 02:59:50.485983 20433 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a1a39dcd328e4268ae41b988c66b4a75" candidate_uuid: "1c50de89a5a845668696dcb28c3badd6" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "197b880fa39d432a9b472845876bfaf0" is_pre_election: true
I20260129 02:59:50.429637 22803 raft_consensus.cc:515] T 42d27a28d89b409e8807ec0efd97af69 P 197b880fa39d432a9b472845876bfaf0 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } }
I20260129 02:59:50.433169 21130 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "18f30ebaa4174fd28c958e72d0cc7e24" candidate_uuid: "197b880fa39d432a9b472845876bfaf0" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "7a9daf5d42a84615a6383cfec9a10608" is_pre_election: true
I20260129 02:59:50.466367 22763 raft_consensus.cc:2749] T 00e79d8f25a34676b1d866eb2068a2b2 P 974b802141bd4f13beffef2aa03f88f9 [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260129 02:59:50.519443 21099 leader_election.cc:400] T c934d31efd3f4e5195605ddc354d14e1 P 7a9daf5d42a84615a6383cfec9a10608 [CANDIDATE]: Term 4 election: Vote denied by peer 197b880fa39d432a9b472845876bfaf0 (127.18.211.1:42027) with higher term. Message: Invalid argument: T c934d31efd3f4e5195605ddc354d14e1 P 197b880fa39d432a9b472845876bfaf0 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 7a9daf5d42a84615a6383cfec9a10608 for earlier term 4. Current term is 5.
I20260129 02:59:50.614272 21099 leader_election.cc:403] T c934d31efd3f4e5195605ddc354d14e1 P 7a9daf5d42a84615a6383cfec9a10608 [CANDIDATE]: Term 4 election: Cancelling election due to peer responding with higher term
W20260129 02:59:50.523967 20481 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260129 02:59:50.547132 20405 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.211.3:37735, user_credentials={real_user=slave}} blocked reactor thread for 56025.8us
I20260129 02:59:50.614840 21136 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "45cc5f3d2688486d9b6dbe7e666dedde" candidate_uuid: "974b802141bd4f13beffef2aa03f88f9" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "7a9daf5d42a84615a6383cfec9a10608"
I20260129 02:59:50.479558 22584 raft_consensus.cc:493] T cedce331e1584ac8b636c9b8a72b2ba0 P 1c50de89a5a845668696dcb28c3badd6 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260129 02:59:50.454245 22403 raft_consensus.cc:493] T dc2f4cb752cf4d59b5f6bc689f5d8104 P 7a9daf5d42a84615a6383cfec9a10608 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260129 02:59:50.577293 20556 leader_election.cc:304] T 926a250fa5834e88b96ccecbefbf5086 P 1c50de89a5a845668696dcb28c3badd6 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1c50de89a5a845668696dcb28c3badd6; no voters: 197b880fa39d432a9b472845876bfaf0, 974b802141bd4f13beffef2aa03f88f9
W20260129 02:59:50.517554 22344 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.694s user 0.011s sys 0.007s
I20260129 02:59:50.583074 20511 raft_consensus.cc:2410] T b189f08e16a74a11980cc55d99359a0a P 974b802141bd4f13beffef2aa03f88f9 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1c50de89a5a845668696dcb28c3badd6 for term 6 because replica has last-logged OpId of term: 5 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 4 index: 3.
W20260129 02:59:50.583408 20553 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.211.1:42027, user_credentials={real_user=slave}} blocked reactor thread for 85887.1us
I20260129 02:59:50.585551 20437 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "75be4bd6bd45474795fd472eff2953ac" candidate_uuid: "1c50de89a5a845668696dcb28c3badd6" candidate_term: 9 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "197b880fa39d432a9b472845876bfaf0"
I20260129 02:59:50.592664 20581 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7a9a69cd82724b1faed1d10aa091fc4a" candidate_uuid: "197b880fa39d432a9b472845876bfaf0" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "1c50de89a5a845668696dcb28c3badd6"
I20260129 02:59:50.521358 22638 raft_consensus.cc:493] T 75f45538dc704d9a8ec4ff8ad9f3ea5b P 1c50de89a5a845668696dcb28c3badd6 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 7a9daf5d42a84615a6383cfec9a10608)
I20260129 02:59:50.598055 20588 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "8055232cf0004d40a85f3bf983fe00a9" candidate_uuid: "197b880fa39d432a9b472845876bfaf0" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "1c50de89a5a845668696dcb28c3badd6" is_pre_election: true
I20260129 02:59:50.598333 20440 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e4d18c35b0a7493e85d0a4b92956a48f" candidate_uuid: "974b802141bd4f13beffef2aa03f88f9" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "197b880fa39d432a9b472845876bfaf0"
I20260129 02:59:50.623589 20588 raft_consensus.cc:2468] T 8055232cf0004d40a85f3bf983fe00a9 P 1c50de89a5a845668696dcb28c3badd6 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 197b880fa39d432a9b472845876bfaf0 in term 3.
W20260129 02:59:50.623777 21099 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.211.1:42027, user_credentials={real_user=slave}} blocked reactor thread for 104558us
I20260129 02:59:50.624059 20440 raft_consensus.cc:3060] T e4d18c35b0a7493e85d0a4b92956a48f P 197b880fa39d432a9b472845876bfaf0 [term 4 FOLLOWER]: Advancing to term 5
I20260129 02:59:50.598884 20407 leader_election.cc:304] T a4c1226b987b468fa84dfc6ba0e2dbe1 P 197b880fa39d432a9b472845876bfaf0 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 197b880fa39d432a9b472845876bfaf0, 974b802141bd4f13beffef2aa03f88f9; no voters:
I20260129 02:59:50.573388 22684 raft_consensus.cc:2804] T cedce331e1584ac8b636c9b8a72b2ba0 P 197b880fa39d432a9b472845876bfaf0 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260129 02:59:50.462299 22802 raft_consensus.cc:515] T 3e2d15f8640b407195210c476e09df91 P 1c50de89a5a845668696dcb28c3badd6 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } }
I20260129 02:59:50.602216 20514 raft_consensus.cc:1240] T 75be4bd6bd45474795fd472eff2953ac P 974b802141bd4f13beffef2aa03f88f9 [term 8 FOLLOWER]: Rejecting Update request from peer 1c50de89a5a845668696dcb28c3badd6 for earlier term 7. Current term is 8. Ops: []
I20260129 02:59:50.605545 20508 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a8479d1fe0bd41e5946186e3a6ca8c7a" candidate_uuid: "1c50de89a5a845668696dcb28c3badd6" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "974b802141bd4f13beffef2aa03f88f9"
I20260129 02:59:50.580471 22184 raft_consensus.cc:697] T 2ac659d732314e1288849f08fe5e074e P 974b802141bd4f13beffef2aa03f88f9 [term 3 LEADER]: Becoming Leader. State: Replica: 974b802141bd4f13beffef2aa03f88f9, State: Running, Role: LEADER
I20260129 02:59:50.644753 22184 consensus_queue.cc:237] T 2ac659d732314e1288849f08fe5e074e P 974b802141bd4f13beffef2aa03f88f9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } }
I20260129 02:59:50.645406 21131 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34060 (request call id 485) took 3673 ms. Trace:
I20260129 02:59:50.646708 20440 raft_consensus.cc:2468] T e4d18c35b0a7493e85d0a4b92956a48f P 197b880fa39d432a9b472845876bfaf0 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 974b802141bd4f13beffef2aa03f88f9 in term 5.
I20260129 02:59:50.645792 21131 rpcz_store.cc:276] 0129 02:59:46.972176 (+ 0us) service_pool.cc:168] Inserting onto call queue
0129 02:59:46.972323 (+ 147us) service_pool.cc:225] Handling call
0129 02:59:47.054862 (+ 82539us) raft_consensus.cc:1471] Updating replica for 0 ops
0129 02:59:48.235381 (+1180519us) raft_consensus.cc:1534] Early marking committed up to index 6
0129 02:59:48.458469 (+223088us) log.cc:844] Serialized 14 byte log entry
0129 02:59:48.747269 (+288800us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0129 02:59:48.747348 (+ 79us) raft_consensus.cc:1645] Marking committed up to 6
0129 02:59:48.747416 (+ 68us) raft_consensus.cc:1697] Filling consensus response to leader.
0129 02:59:49.608767 (+861351us) spinlock_profiling.cc:243] Waited 37 ms on lock 0x7b440002304c. stack: 00007f8fe044e373 00007f8fe044e270 00007f8fdfda0574 00007f8fe9a5c6b4 00007f8fe9a5c649 00007f8fe02c08b1 00007f8fe038f005 00007f8fe038fa22 00007f8fe30d957b 00007f8fe30d4e4f 00007f8fe30c8c0e 00007f8fe30c7f56 00007f8fe30c7b78 00007f8fe30d15c1 00007f8fe30d1579 00007f8fe30d1509
0129 02:59:50.645163 (+1036396us) spinlock_profiling.cc:243] Waited 1.17 s on lock 0x7b54005dedd4. stack: 00007f8fe044e373 00007f8fe044e270 00007f8fdfda0574 00007f8fe9a5c6b4 00007f8fe9a5c649 00007f8fe34975a1 00007f8fe352ea81 00007f8fe352d4be 00007f8fe84e1f49 00007f8fe1a6ed34 00007f8fe1a6ecc2 00007f8fe1a6ebf1 00007f8fe1a6eb6c 00007f8fe1a6de22 00007f8fe186d34c 00007f8fe186c786
0129 02:59:50.645213 (+ 50us) raft_consensus.cc:1691] UpdateReplicas() finished
0129 02:59:50.645343 (+ 130us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":351510,"prepare.run_cpu_time_us":51,"prepare.run_wall_time_us":2895,"raft.queue_time_us":19906,"raft.run_cpu_time_us":118,"raft.run_wall_time_us":11663,"spinlock_wait_cycles":2777782016,"thread_start_us":438397,"threads_started":2,"wal-append.queue_time_us":379257}
I20260129 02:59:50.648355 21131 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "96609ac68bdf4993a12e1c23355ca013" candidate_uuid: "974b802141bd4f13beffef2aa03f88f9" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "7a9daf5d42a84615a6383cfec9a10608"
I20260129 02:59:50.607571 20433 raft_consensus.cc:2468] T a1a39dcd328e4268ae41b988c66b4a75 P 197b880fa39d432a9b472845876bfaf0 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1c50de89a5a845668696dcb28c3badd6 in term 3.
I20260129 02:59:50.649238 21131 raft_consensus.cc:2393] T 96609ac68bdf4993a12e1c23355ca013 P 7a9daf5d42a84615a6383cfec9a10608 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 974b802141bd4f13beffef2aa03f88f9 in current term 5: Already voted for candidate 7a9daf5d42a84615a6383cfec9a10608 in this term.
I20260129 02:59:50.610850 20507 raft_consensus.cc:1240] T 4f8f3a38f6e24ba4857875181f447442 P 974b802141bd4f13beffef2aa03f88f9 [term 5 FOLLOWER]: Rejecting Update request from peer 1c50de89a5a845668696dcb28c3badd6 for earlier term 4. Current term is 5. Ops: []
I20260129 02:59:50.609165 22493 raft_consensus.cc:493] T a5a05fbcbbcf4916bbc01fd1a5b5e90d P 974b802141bd4f13beffef2aa03f88f9 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260129 02:59:50.587002 22786 raft_consensus.cc:493] T e4d18c35b0a7493e85d0a4b92956a48f P 1c50de89a5a845668696dcb28c3badd6 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260129 02:59:50.589779 22707 raft_consensus.cc:493] T 0f20af7fbe5d4db89a6f17948d98912c P 974b802141bd4f13beffef2aa03f88f9 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260129 02:59:50.626540 20514 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57584 (request call id 1297) took 1124 ms. Trace:
I20260129 02:59:50.603241 22400 raft_consensus.cc:493] T 3e2d15f8640b407195210c476e09df91 P 974b802141bd4f13beffef2aa03f88f9 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 1c50de89a5a845668696dcb28c3badd6)
I20260129 02:59:50.660358 20508 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:57584 (request call id 1296) took 1158 ms. Trace:
I20260129 02:59:50.656308 20479 leader_election.cc:304] T e4d18c35b0a7493e85d0a4b92956a48f P 974b802141bd4f13beffef2aa03f88f9 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 197b880fa39d432a9b472845876bfaf0, 974b802141bd4f13beffef2aa03f88f9; no voters:
W20260129 02:59:50.621474 22344 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.824s user 0.011s sys 0.007s
I20260129 02:59:50.628374 22403 raft_consensus.cc:515] T dc2f4cb752cf4d59b5f6bc689f5d8104 P 7a9daf5d42a84615a6383cfec9a10608 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } }
I20260129 02:59:50.656136 22786 raft_consensus.cc:3060] T e4d18c35b0a7493e85d0a4b92956a48f P 1c50de89a5a845668696dcb28c3badd6 [term 4 FOLLOWER]: Advancing to term 5
I20260129 02:59:50.591035 22798 consensus_queue.cc:1059] T 76f713c3331540d2a5b76164103e9eb1 P 197b880fa39d432a9b472845876bfaf0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 }, Status: INVALID_TERM, Last received: 3.3, Next index: 4, Last known committed idx: 2, Time since last communication: 0.073s
I20260129 02:59:50.593576 22740 raft_consensus.cc:515] T 42d27a28d89b409e8807ec0efd97af69 P 1c50de89a5a845668696dcb28c3badd6 [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } }
I20260129 02:59:50.611482 21132 raft_consensus.cc:3060] T c934d31efd3f4e5195605ddc354d14e1 P 7a9daf5d42a84615a6383cfec9a10608 [term 4 FOLLOWER]: Advancing to term 5
W20260129 02:59:50.632308 20556 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.211.2:34517, user_credentials={real_user=slave}} blocked reactor thread for 144008us
I20260129 02:59:50.624958 22584 raft_consensus.cc:515] T cedce331e1584ac8b636c9b8a72b2ba0 P 1c50de89a5a845668696dcb28c3badd6 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:59:50.619952 22687 raft_consensus.cc:493] T c2633a6fc34c44579de348d208b0c01e P 974b802141bd4f13beffef2aa03f88f9 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 1c50de89a5a845668696dcb28c3badd6)
I20260129 02:59:50.633459 22724 raft_consensus.cc:2749] T 926a250fa5834e88b96ccecbefbf5086 P 1c50de89a5a845668696dcb28c3badd6 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260129 02:59:50.636165 20582 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "cedce331e1584ac8b636c9b8a72b2ba0" candidate_uuid: "197b880fa39d432a9b472845876bfaf0" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "1c50de89a5a845668696dcb28c3badd6" is_pre_election: true
I20260129 02:59:50.636065 22638 raft_consensus.cc:515] T 75f45538dc704d9a8ec4ff8ad9f3ea5b P 1c50de89a5a845668696dcb28c3badd6 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } }
I20260129 02:59:50.687716 22784 leader_election.cc:290] T 3906b080a5b34a74b8b64128635e8997 P 974b802141bd4f13beffef2aa03f88f9 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 197b880fa39d432a9b472845876bfaf0 (127.18.211.1:42027), 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3:37735)
I20260129 02:59:50.665419 22707 raft_consensus.cc:515] T 0f20af7fbe5d4db89a6f17948d98912c P 974b802141bd4f13beffef2aa03f88f9 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:59:50.688764 22638 leader_election.cc:290] T 75f45538dc704d9a8ec4ff8ad9f3ea5b P 1c50de89a5a845668696dcb28c3badd6 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 974b802141bd4f13beffef2aa03f88f9 (127.18.211.2:34517), 7a9daf5d42a84615a6383cfec9a10608 (127.18.211.4:40439)
I20260129 02:59:50.640136 22803 leader_election.cc:290] T 42d27a28d89b409e8807ec0efd97af69 P 197b880fa39d432a9b472845876bfaf0 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3:37735), 7a9daf5d42a84615a6383cfec9a10608 (127.18.211.4:40439)
I20260129 02:59:50.689051 22644 leader_election.cc:290] T 3ccac82a01a2445fa5b0ce6c0fcf2295 P 197b880fa39d432a9b472845876bfaf0 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 974b802141bd4f13beffef2aa03f88f9 (127.18.211.2:34517), 7a9daf5d42a84615a6383cfec9a10608 (127.18.211.4:40439)
I20260129 02:59:50.602563 22645 consensus_queue.cc:237] T a5a05fbcbbcf4916bbc01fd1a5b5e90d P 197b880fa39d432a9b472845876bfaf0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 5.4, Last appended by leader: 4, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:59:50.648298 21129 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4f5a77e94fef4b16a2ba7b067c8b144a" candidate_uuid: "974b802141bd4f13beffef2aa03f88f9" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "7a9daf5d42a84615a6383cfec9a10608" is_pre_election: true
W20260129 02:59:50.610677 21102 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.211.2:34517, user_credentials={real_user=slave}} blocked reactor thread for 140502us
I20260129 02:59:50.581178 22750 consensus_queue.cc:1048] T 926a250fa5834e88b96ccecbefbf5086 P 197b880fa39d432a9b472845876bfaf0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.035s
I20260129 02:59:50.651809 20584 raft_consensus.cc:2468] T cb4727386a3e48aca5d71cc6c611c197 P 1c50de89a5a845668696dcb28c3badd6 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 197b880fa39d432a9b472845876bfaf0 in term 3.
I20260129 02:59:50.649860 20433 raft_consensus.cc:1240] T 0f20af7fbe5d4db89a6f17948d98912c P 197b880fa39d432a9b472845876bfaf0 [term 5 LEADER]: Rejecting Update request from peer 1c50de89a5a845668696dcb28c3badd6 for earlier term 4. Current term is 5. Ops: []
I20260129 02:59:50.655543 20553 leader_election.cc:304] T a1a39dcd328e4268ae41b988c66b4a75 P 1c50de89a5a845668696dcb28c3badd6 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 197b880fa39d432a9b472845876bfaf0, 1c50de89a5a845668696dcb28c3badd6; no voters:
I20260129 02:59:50.706581 20582 raft_consensus.cc:2468] T cedce331e1584ac8b636c9b8a72b2ba0 P 1c50de89a5a845668696dcb28c3badd6 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 197b880fa39d432a9b472845876bfaf0 in term 5.
I20260129 02:59:50.656359 20514 rpcz_store.cc:276] 0129 02:59:49.502042 (+ 0us) service_pool.cc:168] Inserting onto call queue
0129 02:59:50.596525 (+1094483us) service_pool.cc:225] Handling call
0129 02:59:50.601979 (+ 5454us) raft_consensus.cc:1471] Updating replica for 0 ops
0129 02:59:50.626290 (+ 24311us) raft_consensus.cc:1697] Filling consensus response to leader.
0129 02:59:50.626483 (+ 193us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260129 02:59:50.657132 20515 raft_consensus.cc:2410] T 00e79d8f25a34676b1d866eb2068a2b2 P 974b802141bd4f13beffef2aa03f88f9 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 197b880fa39d432a9b472845876bfaf0 for term 6 because replica has last-logged OpId of term: 4 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 6.
I20260129 02:59:50.576326 22445 raft_consensus.cc:515] T a1a39dcd328e4268ae41b988c66b4a75 P 974b802141bd4f13beffef2aa03f88f9 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } } peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } }
I20260129 02:59:50.660807 21131 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "87d5fe48a44a408aaf03e894a67ecb2f" candidate_uuid: "974b802141bd4f13beffef2aa03f88f9" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "7a9daf5d42a84615a6383cfec9a10608" is_pre_election: true
I20260129 02:59:50.660733 20507 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57584 (request call id 1295) took 1163 ms. Trace:
W20260129 02:59:50.706158 20553 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.211.1:42027, user_credentials={real_user=slave}} blocked reactor thread for 50681.1us
I20260129 02:59:50.711898 20507 rpcz_store.cc:276] 0129 02:59:49.496861 (+ 0us) service_pool.cc:168] Inserting onto call queue
0129 02:59:50.595206 (+1098345us) service_pool.cc:225] Handling call
0129 02:59:50.610664 (+ 15458us) raft_consensus.cc:1471] Updating replica for 0 ops
0129 02:59:50.660469 (+ 49805us) raft_consensus.cc:1697] Filling consensus response to leader.
0129 02:59:50.660676 (+ 207us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20260129 02:59:50.623605 22737 consensus_meta.cc:287] T fcddab6bca684adf8ea032289a00f90e P 197b880fa39d432a9b472845876bfaf0: Time spent flushing consensus metadata: real 0.673s user 0.008s sys 0.002s
I20260129 02:59:50.660892 20508 rpcz_store.cc:276] 0129 02:59:49.501721 (+ 0us) service_pool.cc:168] Inserting onto call queue
0129 02:59:49.869246 (+367525us) service_pool.cc:225] Handling call
0129 02:59:50.605343 (+736097us) spinlock_profiling.cc:243] Waited 9.15 ms on lock 0x7b5800254d98. stack: 00007f8fe044e373 00007f8fe044e270 00007f8fdfda0574 00007f8fe9a5c6b4 00007f8fe9a5c649 00007f8fe02c08b1 00007f8fe0442db1 00007f8fe0442c62 00007f8fe92eddd9 00007f8fe846d260 00007f8fe8535dbb 00007f8fe8535e41 00007f8fe84e3bd0 00007f8fe84e34b4 00007f8fe1a73834 00007f8fe1a737c2
0129 02:59:50.660283 (+ 54940us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":21054720}
W20260129 02:59:50.714783 21099 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260129 02:59:50.582449 22688 raft_consensus.cc:515] T 4e3805ffa70242e8ade66da260dddf66 P 197b880fa39d432a9b472845876bfaf0 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } }
I20260129 02:59:50.665473 21136 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4e420f90884f4796997b5ea41e49e411" candidate_uuid: "974b802141bd4f13beffef2aa03f88f9" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "7a9daf5d42a84615a6383cfec9a10608" is_pre_election: true
I20260129 02:59:50.665822 20437 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "724a236524cb4e7ba3428880681d6cc2" candidate_uuid: "1c50de89a5a845668696dcb28c3badd6" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "197b880fa39d432a9b472845876bfaf0" is_pre_election: true
I20260129 02:59:50.615792 20509 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c934d31efd3f4e5195605ddc354d14e1" candidate_uuid: "197b880fa39d432a9b472845876bfaf0" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "974b802141bd4f13beffef2aa03f88f9"
I20260129 02:59:50.716486 21136 raft_consensus.cc:2468] T 4e420f90884f4796997b5ea41e49e411 P 7a9daf5d42a84615a6383cfec9a10608 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 974b802141bd4f13beffef2aa03f88f9 in term 4.
I20260129 02:59:50.668092 20588 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "cb4727386a3e48aca5d71cc6c611c197" candidate_uuid: "197b880fa39d432a9b472845876bfaf0" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1c50de89a5a845668696dcb28c3badd6" is_pre_election: true
I20260129 02:59:50.718871 20588 raft_consensus.cc:2376] T cb4727386a3e48aca5d71cc6c611c197 P 1c50de89a5a845668696dcb28c3badd6 [term 3 FOLLOWER]: Leader pre-election vote request: Already granted yes vote for candidate 197b880fa39d432a9b472845876bfaf0 in term 3. Re-sending same reply.
I20260129 02:59:50.716550 20437 raft_consensus.cc:2468] T 724a236524cb4e7ba3428880681d6cc2 P 197b880fa39d432a9b472845876bfaf0 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1c50de89a5a845668696dcb28c3badd6 in term 3.
I20260129 02:59:50.628989 20436 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "36ff33be666a4c82954731691dbcf327" candidate_uuid: "974b802141bd4f13beffef2aa03f88f9" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "197b880fa39d432a9b472845876bfaf0" is_pre_election: true
I20260129 02:59:50.625891 22684 raft_consensus.cc:493] T cedce331e1584ac8b636c9b8a72b2ba0 P 197b880fa39d432a9b472845876bfaf0 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260129 02:59:50.630146 22802 leader_election.cc:290] T 3e2d15f8640b407195210c476e09df91 P 1c50de89a5a845668696dcb28c3badd6 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 974b802141bd4f13beffef2aa03f88f9 (127.18.211.2:34517), 7a9daf5d42a84615a6383cfec9a10608 (127.18.211.4:40439)
I20260129 02:59:50.665143 22400 raft_consensus.cc:3060] T 3e2d15f8640b407195210c476e09df91 P 974b802141bd4f13beffef2aa03f88f9 [term 1 FOLLOWER]: Advancing to term 2
I20260129 02:59:50.668435 22763 raft_consensus.cc:2804] T e4d18c35b0a7493e85d0a4b92956a48f P 974b802141bd4f13beffef2aa03f88f9 [term 5 FOLLOWER]: Leader election won for term 5
I20260129 02:59:50.667616 22700 raft_consensus.cc:493] T 505580c93c694bbb8ea96fdf9b17ab46 P 974b802141bd4f13beffef2aa03f88f9 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260129 02:59:50.660166 22435 consensus_queue.cc:1048] T 368953c9102643b2a21f99af36b51905 P 1c50de89a5a845668696dcb28c3badd6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 2, Time since last communication: 0.023s
I20260129 02:59:50.660997 22493 raft_consensus.cc:515] T a5a05fbcbbcf4916bbc01fd1a5b5e90d P 974b802141bd4f13beffef2aa03f88f9 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:59:50.686079 22403 leader_election.cc:290] T dc2f4cb752cf4d59b5f6bc689f5d8104 P 7a9daf5d42a84615a6383cfec9a10608 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 197b880fa39d432a9b472845876bfaf0 (127.18.211.1:42027), 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3:37735)
I20260129 02:59:50.604225 21128 raft_consensus.cc:2468] T 505580c93c694bbb8ea96fdf9b17ab46 P 7a9daf5d42a84615a6383cfec9a10608 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1c50de89a5a845668696dcb28c3badd6 in term 3.
I20260129 02:59:50.695195 20586 raft_consensus.cc:1217] T 42d27a28d89b409e8807ec0efd97af69 P 1c50de89a5a845668696dcb28c3badd6 [term 6 FOLLOWER]: Deduplicated request from leader. Original: 2.5->[4.6-4.6] Dedup: 4.6->[]
W20260129 02:59:50.696365 21102 leader_election.cc:336] T a609ac02fae84725ab743f95f51d33ce P 7a9daf5d42a84615a6383cfec9a10608 [CANDIDATE]: Term 6 election: RPC error from VoteRequest() call to peer 974b802141bd4f13beffef2aa03f88f9 (127.18.211.2:34517): Timed out: RequestConsensusVote RPC to 127.18.211.2:34517 timed out after 1.923s (SENT)
I20260129 02:59:50.690044 22707 leader_election.cc:290] T 0f20af7fbe5d4db89a6f17948d98912c P 974b802141bd4f13beffef2aa03f88f9 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 197b880fa39d432a9b472845876bfaf0 (127.18.211.1:42027), 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3:37735)
I20260129 02:59:50.708485 20584 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:35754 (request call id 1650) took 1248 ms. Trace:
I20260129 02:59:50.705533 20433 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3906b080a5b34a74b8b64128635e8997" candidate_uuid: "974b802141bd4f13beffef2aa03f88f9" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "197b880fa39d432a9b472845876bfaf0" is_pre_election: true
I20260129 02:59:50.703734 21129 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "75f45538dc704d9a8ec4ff8ad9f3ea5b" candidate_uuid: "974b802141bd4f13beffef2aa03f88f9" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "7a9daf5d42a84615a6383cfec9a10608" is_pre_election: true
I20260129 02:59:50.712772 20556 leader_election.cc:304] T fcddab6bca684adf8ea032289a00f90e P 1c50de89a5a845668696dcb28c3badd6 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1c50de89a5a845668696dcb28c3badd6; no voters: 197b880fa39d432a9b472845876bfaf0, 974b802141bd4f13beffef2aa03f88f9
I20260129 02:59:50.737289 20516 raft_consensus.cc:1240] T a8479d1fe0bd41e5946186e3a6ca8c7a P 974b802141bd4f13beffef2aa03f88f9 [term 4 FOLLOWER]: Rejecting Update request from peer 197b880fa39d432a9b472845876bfaf0 for earlier term 3. Current term is 4. Ops: [3.3-3.3]
I20260129 02:59:50.725069 20513 raft_consensus.cc:1240] T a1a39dcd328e4268ae41b988c66b4a75 P 974b802141bd4f13beffef2aa03f88f9 [term 3 FOLLOWER]: Rejecting Update request from peer 1c50de89a5a845668696dcb28c3badd6 for earlier term 2. Current term is 3. Ops: []
I20260129 02:59:50.715248 20508 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "87d5fe48a44a408aaf03e894a67ecb2f" candidate_uuid: "7a9daf5d42a84615a6383cfec9a10608" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "974b802141bd4f13beffef2aa03f88f9" is_pre_election: true
I20260129 02:59:50.717545 20514 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00e79d8f25a34676b1d866eb2068a2b2" candidate_uuid: "7a9daf5d42a84615a6383cfec9a10608" candidate_term: 6 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "974b802141bd4f13beffef2aa03f88f9" is_pre_election: true
I20260129 02:59:50.739166 20508 raft_consensus.cc:2468] T 87d5fe48a44a408aaf03e894a67ecb2f P 974b802141bd4f13beffef2aa03f88f9 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7a9daf5d42a84615a6383cfec9a10608 in term 5.
I20260129 02:59:50.601732 22779 consensus_queue.cc:237] T 1299723a4476420ab02fd8bfd7aa0645 P 7a9daf5d42a84615a6383cfec9a10608 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 6.7, Last appended by leader: 7, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } }
I20260129 02:59:50.666922 20581 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a644c02b16af47679824ccd060ac10fd" candidate_uuid: "7a9daf5d42a84615a6383cfec9a10608" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "1c50de89a5a845668696dcb28c3badd6" is_pre_election: true
I20260129 02:59:50.720417 20553 leader_election.cc:304] T 724a236524cb4e7ba3428880681d6cc2 P 1c50de89a5a845668696dcb28c3badd6 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 197b880fa39d432a9b472845876bfaf0, 1c50de89a5a845668696dcb28c3badd6; no voters:
I20260129 02:59:50.721091 20507 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a1a39dcd328e4268ae41b988c66b4a75" candidate_uuid: "1c50de89a5a845668696dcb28c3badd6" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "974b802141bd4f13beffef2aa03f88f9" is_pre_election: true
I20260129 02:59:50.740094 20508 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4e420f90884f4796997b5ea41e49e411" candidate_uuid: "7a9daf5d42a84615a6383cfec9a10608" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "974b802141bd4f13beffef2aa03f88f9" is_pre_election: true
I20260129 02:59:50.716002 22785 consensus_queue.cc:1059] T 0f20af7fbe5d4db89a6f17948d98912c P 1c50de89a5a845668696dcb28c3badd6 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 }, Status: INVALID_TERM, Last received: 4.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.007s
I20260129 02:59:50.722896 22237 raft_consensus.cc:515] T 7d9494f5584d4d5db8b9adf1c8fc888b P 197b880fa39d432a9b472845876bfaf0 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:59:50.745469 22445 leader_election.cc:290] T a1a39dcd328e4268ae41b988c66b4a75 P 974b802141bd4f13beffef2aa03f88f9 [CANDIDATE]: Term 3 election: Requested vote from peers 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3:37735), 197b880fa39d432a9b472845876bfaf0 (127.18.211.1:42027)
I20260129 02:59:50.745626 20581 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a0fcc58d6fb6473cafbd9d34d5195269" candidate_uuid: "197b880fa39d432a9b472845876bfaf0" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "1c50de89a5a845668696dcb28c3badd6" is_pre_election: true
I20260129 02:59:50.746275 20581 raft_consensus.cc:2468] T a0fcc58d6fb6473cafbd9d34d5195269 P 1c50de89a5a845668696dcb28c3badd6 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 197b880fa39d432a9b472845876bfaf0 in term 3.
I20260129 02:59:50.733966 20586 raft_consensus.cc:1240] T 42d27a28d89b409e8807ec0efd97af69 P 1c50de89a5a845668696dcb28c3badd6 [term 6 FOLLOWER]: Rejecting Update request from peer 197b880fa39d432a9b472845876bfaf0 for earlier term 4. Current term is 6. Ops: [4.6-4.6]
I20260129 02:59:50.746937 22237 leader_election.cc:290] T 7d9494f5584d4d5db8b9adf1c8fc888b P 197b880fa39d432a9b472845876bfaf0 [CANDIDATE]: Term 3 election: Requested vote from peers 974b802141bd4f13beffef2aa03f88f9 (127.18.211.2:34517), 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3:37735)
I20260129 02:59:50.747120 20581 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "36ff33be666a4c82954731691dbcf327" candidate_uuid: "197b880fa39d432a9b472845876bfaf0" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "1c50de89a5a845668696dcb28c3badd6" is_pre_election: true
I20260129 02:59:50.732707 22493 leader_election.cc:290] T a5a05fbcbbcf4916bbc01fd1a5b5e90d P 974b802141bd4f13beffef2aa03f88f9 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 197b880fa39d432a9b472845876bfaf0 (127.18.211.1:42027), 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3:37735)
I20260129 02:59:50.747607 20586 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35754 (request call id 1633) took 1965 ms. Trace:
I20260129 02:59:50.734531 21102 leader_election.cc:304] T a609ac02fae84725ab743f95f51d33ce P 7a9daf5d42a84615a6383cfec9a10608 [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7a9daf5d42a84615a6383cfec9a10608; no voters: 197b880fa39d432a9b472845876bfaf0, 974b802141bd4f13beffef2aa03f88f9
I20260129 02:59:50.692871 22687 raft_consensus.cc:515] T c2633a6fc34c44579de348d208b0c01e P 974b802141bd4f13beffef2aa03f88f9 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
W20260129 02:59:50.714509 22776 consensus_meta.cc:287] T 3ccac82a01a2445fa5b0ce6c0fcf2295 P 7a9daf5d42a84615a6383cfec9a10608: Time spent flushing consensus metadata: real 0.950s user 0.009s sys 0.000s
I20260129 02:59:50.736323 22435 raft_consensus.cc:3060] T fcddab6bca684adf8ea032289a00f90e P 1c50de89a5a845668696dcb28c3badd6 [term 7 FOLLOWER]: Advancing to term 8
I20260129 02:59:50.751406 22776 raft_consensus.cc:515] T 3ccac82a01a2445fa5b0ce6c0fcf2295 P 7a9daf5d42a84615a6383cfec9a10608 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } }
I20260129 02:59:50.753530 22776 leader_election.cc:290] T 3ccac82a01a2445fa5b0ce6c0fcf2295 P 7a9daf5d42a84615a6383cfec9a10608 [CANDIDATE]: Term 4 election: Requested vote from peers 197b880fa39d432a9b472845876bfaf0 (127.18.211.1:42027), 974b802141bd4f13beffef2aa03f88f9 (127.18.211.2:34517)
I20260129 02:59:50.724917 21136 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2e58e9a5a8aa4cecafb464addb4f35ed" candidate_uuid: "1c50de89a5a845668696dcb28c3badd6" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "7a9daf5d42a84615a6383cfec9a10608" is_pre_election: true
I20260129 02:59:50.754035 22671 consensus_queue.cc:260] T febffe4a8b394368b69b1d164308fb9a P 7a9daf5d42a84615a6383cfec9a10608 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 2.5, Last appended by leader: 5, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } }
W20260129 02:59:50.719075 22656 consensus_meta.cc:287] T 7a9a69cd82724b1faed1d10aa091fc4a P 1c50de89a5a845668696dcb28c3badd6: Time spent flushing consensus metadata: real 0.957s user 0.010s sys 0.001s
I20260129 02:59:50.741398 20508 raft_consensus.cc:2468] T 4e420f90884f4796997b5ea41e49e411 P 974b802141bd4f13beffef2aa03f88f9 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7a9daf5d42a84615a6383cfec9a10608 in term 4.
W20260129 02:59:50.727674 20554 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260129 02:59:50.735184 20433 raft_consensus.cc:2468] T 3906b080a5b34a74b8b64128635e8997 P 197b880fa39d432a9b472845876bfaf0 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 974b802141bd4f13beffef2aa03f88f9 in term 3.
I20260129 02:59:50.736166 20556 leader_election.cc:304] T 96609ac68bdf4993a12e1c23355ca013 P 1c50de89a5a845668696dcb28c3badd6 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1c50de89a5a845668696dcb28c3badd6; no voters: 197b880fa39d432a9b472845876bfaf0, 974b802141bd4f13beffef2aa03f88f9
I20260129 02:59:50.736879 22684 raft_consensus.cc:3060] T cedce331e1584ac8b636c9b8a72b2ba0 P 197b880fa39d432a9b472845876bfaf0 [term 5 FOLLOWER]: Advancing to term 6
I20260129 02:59:50.730701 21132 raft_consensus.cc:2468] T c934d31efd3f4e5195605ddc354d14e1 P 7a9daf5d42a84615a6383cfec9a10608 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 197b880fa39d432a9b472845876bfaf0 in term 5.
I20260129 02:59:50.743356 20436 raft_consensus.cc:2468] T 36ff33be666a4c82954731691dbcf327 P 197b880fa39d432a9b472845876bfaf0 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 974b802141bd4f13beffef2aa03f88f9 in term 2.
I20260129 02:59:50.731191 22700 raft_consensus.cc:515] T 505580c93c694bbb8ea96fdf9b17ab46 P 974b802141bd4f13beffef2aa03f88f9 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } }
I20260129 02:59:50.737807 22160 raft_consensus.cc:493] T a8479d1fe0bd41e5946186e3a6ca8c7a P 974b802141bd4f13beffef2aa03f88f9 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260129 02:59:50.729597 22672 raft_consensus.cc:515] T 18f30ebaa4174fd28c958e72d0cc7e24 P 7a9daf5d42a84615a6383cfec9a10608 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } }
I20260129 02:59:50.744959 20507 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "505580c93c694bbb8ea96fdf9b17ab46" candidate_uuid: "1c50de89a5a845668696dcb28c3badd6" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "974b802141bd4f13beffef2aa03f88f9" is_pre_election: true
I20260129 02:59:50.736239 20584 rpcz_store.cc:276] 0129 02:59:49.459698 (+ 0us) service_pool.cc:168] Inserting onto call queue
0129 02:59:50.430609 (+970911us) service_pool.cc:225] Handling call
0129 02:59:50.708441 (+277832us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260129 02:59:50.714448 22638 raft_consensus.cc:2804] T a1a39dcd328e4268ae41b988c66b4a75 P 1c50de89a5a845668696dcb28c3badd6 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260129 02:59:50.748020 20586 rpcz_store.cc:276] 0129 02:59:48.782496 (+ 0us) service_pool.cc:168] Inserting onto call queue
0129 02:59:48.789950 (+ 7454us) service_pool.cc:225] Handling call
0129 02:59:48.790225 (+ 275us) raft_consensus.cc:1471] Updating replica for 1 ops
0129 02:59:50.746874 (+1956649us) raft_consensus.cc:1697] Filling consensus response to leader.
0129 02:59:50.747399 (+ 525us) spinlock_profiling.cc:243] Waited 1.9 s on lock 0x7b5400134f54. stack: 00007f8fe044e373 00007f8fe044e270 00007f8fdfda0574 00007f8fe9a5c6b4 00007f8fe9a5c649 00007f8fe34975a1 00007f8fe352ea81 00007f8fe352d4be 00007f8fe84e1f49 00007f8fe1a6ed34 00007f8fe1a6ecc2 00007f8fe1a6ebf1 00007f8fe1a6eb6c 00007f8fe1a6de22 00007f8fe186d34c 00007f8fe186c786
0129 02:59:50.747555 (+ 156us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":4380698880}
I20260129 02:59:50.758612 20437 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e33a417cf9c54cf5af73a4ee3aac4d3a" candidate_uuid: "974b802141bd4f13beffef2aa03f88f9" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "197b880fa39d432a9b472845876bfaf0" is_pre_election: true
W20260129 02:59:50.846930 20553 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260129 02:59:50.759567 20588 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "75f45538dc704d9a8ec4ff8ad9f3ea5b" candidate_uuid: "974b802141bd4f13beffef2aa03f88f9" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "1c50de89a5a845668696dcb28c3badd6" is_pre_election: true
I20260129 02:59:50.826789 20581 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c891d2bff4ce4568861bbc992e91fbb0" candidate_uuid: "197b880fa39d432a9b472845876bfaf0" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "1c50de89a5a845668696dcb28c3badd6" is_pre_election: true
I20260129 02:59:50.736771 20480 leader_election.cc:304] T 4e420f90884f4796997b5ea41e49e411 P 974b802141bd4f13beffef2aa03f88f9 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 7a9daf5d42a84615a6383cfec9a10608, 974b802141bd4f13beffef2aa03f88f9; no voters: 197b880fa39d432a9b472845876bfaf0
W20260129 02:59:50.757395 21133 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34046 (request call id 463) took 1922 ms (client timeout 1973 ms). Trace:
W20260129 02:59:50.883337 20590 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:35754 (request call id 1640) took 1784 ms (client timeout 1605 ms). Trace:
I20260129 02:59:50.885119 20588 raft_consensus.cc:2468] T 75f45538dc704d9a8ec4ff8ad9f3ea5b P 1c50de89a5a845668696dcb28c3badd6 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 974b802141bd4f13beffef2aa03f88f9 in term 2.
W20260129 02:59:50.884867 20590 rpcz_store.cc:269] 0129 02:59:49.060624 (+ 0us) service_pool.cc:168] Inserting onto call queue
0129 02:59:49.451671 (+391047us) service_pool.cc:225] Handling call
0129 02:59:50.844396 (+1392725us) spinlock_profiling.cc:243] Waited 329 ms on lock 0x7b540033f254. stack: 00007f8fe044e373 00007f8fe044e270 00007f8fdfda0574 00007f8fe9a5c6b4 00007f8fe9a5c649 00007f8fe34975a1 00007f8fe3531f7a 00007f8fe84e3647 00007f8fe1a73834 00007f8fe1a737c2 00007f8fe1a736f1 00007f8fe1a7366c 00007f8fe1a72922 00007f8fe186d34c 00007f8fe186c786 00007f8fe186c12f
0129 02:59:50.844559 (+ 163us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":756492032}
I20260129 02:59:50.764675 20514 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "96609ac68bdf4993a12e1c23355ca013" candidate_uuid: "7a9daf5d42a84615a6383cfec9a10608" candidate_term: 5 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "974b802141bd4f13beffef2aa03f88f9" is_pre_election: true
W20260129 02:59:50.887184 21100 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260129 02:59:50.886543 20590 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b189f08e16a74a11980cc55d99359a0a" candidate_uuid: "197b880fa39d432a9b472845876bfaf0" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "1c50de89a5a845668696dcb28c3badd6"
W20260129 02:59:50.888088 21102 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.211.2:34517, user_credentials={real_user=slave}} blocked reactor thread for 191723us
I20260129 02:59:50.888263 20590 raft_consensus.cc:2376] T b189f08e16a74a11980cc55d99359a0a P 1c50de89a5a845668696dcb28c3badd6 [term 6 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 197b880fa39d432a9b472845876bfaf0 in term 6. Re-sending same reply.
I20260129 02:59:50.844702 20439 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55676 (request call id 1261) took 2643 ms. Trace:
W20260129 02:59:50.758822 20555 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260129 02:59:50.887704 20514 raft_consensus.cc:2393] T 96609ac68bdf4993a12e1c23355ca013 P 974b802141bd4f13beffef2aa03f88f9 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 7a9daf5d42a84615a6383cfec9a10608 in current term 5: Already voted for candidate 974b802141bd4f13beffef2aa03f88f9 in this term.
I20260129 02:59:50.889490 20590 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3906b080a5b34a74b8b64128635e8997" candidate_uuid: "974b802141bd4f13beffef2aa03f88f9" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "1c50de89a5a845668696dcb28c3badd6" is_pre_election: true
W20260129 02:59:50.885991 21133 rpcz_store.cc:269] 0129 02:59:48.835400 (+ 0us) service_pool.cc:168] Inserting onto call queue
0129 02:59:48.835598 (+ 198us) service_pool.cc:225] Handling call
0129 02:59:50.757154 (+1921556us) spinlock_profiling.cc:243] Waited 71.1 ms on lock 0x7b540068e2d4. stack: 00007f8fe044e373 00007f8fe044e270 00007f8fdfda0574 00007f8fe9a5c6b4 00007f8fe9a5c649 00007f8fe34975a1 00007f8fe3531f7a 00007f8fe84e3647 00007f8fe1a73834 00007f8fe1a737c2 00007f8fe1a736f1 00007f8fe1a7366c 00007f8fe1a72922 00007f8fe186d34c 00007f8fe186c786 00007f8fe186c12f
0129 02:59:50.757322 (+ 168us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":163457408}
W20260129 02:59:50.889820 20555 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260129 02:59:50.737071 21131 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c934d31efd3f4e5195605ddc354d14e1" candidate_uuid: "974b802141bd4f13beffef2aa03f88f9" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "7a9daf5d42a84615a6383cfec9a10608" is_pre_election: true
W20260129 02:59:50.726725 22667 consensus_meta.cc:287] T 00e79d8f25a34676b1d866eb2068a2b2 P 7a9daf5d42a84615a6383cfec9a10608: Time spent flushing consensus metadata: real 0.993s user 0.010s sys 0.002s
I20260129 02:59:50.884334 20581 raft_consensus.cc:2468] T c891d2bff4ce4568861bbc992e91fbb0 P 1c50de89a5a845668696dcb28c3badd6 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 197b880fa39d432a9b472845876bfaf0 in term 6.
I20260129 02:59:50.910108 20590 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e4d18c35b0a7493e85d0a4b92956a48f" candidate_uuid: "974b802141bd4f13beffef2aa03f88f9" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "1c50de89a5a845668696dcb28c3badd6" is_pre_election: true
I20260129 02:59:50.900821 20515 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:57600 (request call id 1763) took 1258 ms. Trace:
I20260129 02:59:50.881402 20437 raft_consensus.cc:2468] T e33a417cf9c54cf5af73a4ee3aac4d3a P 197b880fa39d432a9b472845876bfaf0 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 974b802141bd4f13beffef2aa03f88f9 in term 5.
I20260129 02:59:50.898181 20584 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "cb4727386a3e48aca5d71cc6c611c197" candidate_uuid: "197b880fa39d432a9b472845876bfaf0" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1c50de89a5a845668696dcb28c3badd6"
I20260129 02:59:50.976032 20584 raft_consensus.cc:2376] T cb4727386a3e48aca5d71cc6c611c197 P 1c50de89a5a845668696dcb28c3badd6 [term 3 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 197b880fa39d432a9b472845876bfaf0 in term 3. Re-sending same reply.
I20260129 02:59:50.976999 20584 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:35754 (request call id 1687) took 1118 ms. Trace:
I20260129 02:59:50.977586 20584 rpcz_store.cc:276] 0129 02:59:49.858332 (+ 0us) service_pool.cc:168] Inserting onto call queue
0129 02:59:50.897914 (+1039582us) service_pool.cc:225] Handling call
0129 02:59:50.976924 (+ 79010us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20260129 02:59:50.932086 20482 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260129 02:59:50.976965 20515 rpcz_store.cc:276] 0129 02:59:49.642117 (+ 0us) service_pool.cc:168] Inserting onto call queue
0129 02:59:49.874210 (+232093us) service_pool.cc:225] Handling call
0129 02:59:50.900495 (+1026285us) spinlock_profiling.cc:243] Waited 753 ms on lock 0x7b54002efc54. stack: 00007f8fe044e373 00007f8fe044e270 00007f8fdfda0574 00007f8fe9a5c6b4 00007f8fe9a5c649 00007f8fe34975a1 00007f8fe3531f7a 00007f8fe84e3647 00007f8fe1a73834 00007f8fe1a737c2 00007f8fe1a736f1 00007f8fe1a7366c 00007f8fe1a72922 00007f8fe186d34c 00007f8fe186c786 00007f8fe186c12f
0129 02:59:50.900745 (+ 250us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1731002624}
I20260129 02:59:50.987141 20515 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a0fcc58d6fb6473cafbd9d34d5195269" candidate_uuid: "1c50de89a5a845668696dcb28c3badd6" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "974b802141bd4f13beffef2aa03f88f9" is_pre_election: true
I20260129 02:59:50.991951 20515 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a609ac02fae84725ab743f95f51d33ce" candidate_uuid: "7a9daf5d42a84615a6383cfec9a10608" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "974b802141bd4f13beffef2aa03f88f9"
I20260129 02:59:50.996886 20515 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "cb4727386a3e48aca5d71cc6c611c197" candidate_uuid: "1c50de89a5a845668696dcb28c3badd6" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "974b802141bd4f13beffef2aa03f88f9" is_pre_election: true
I20260129 02:59:50.754779 21136 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "75f45538dc704d9a8ec4ff8ad9f3ea5b" candidate_uuid: "1c50de89a5a845668696dcb28c3badd6" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "7a9daf5d42a84615a6383cfec9a10608" is_pre_election: true
W20260129 02:59:50.891002 21101 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260129 02:59:50.890504 20556 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.211.2:34517, user_credentials={real_user=slave}} blocked reactor thread for 154390us
W20260129 02:59:50.882004 20554 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260129 02:59:50.724961 20509 raft_consensus.cc:3060] T c934d31efd3f4e5195605ddc354d14e1 P 974b802141bd4f13beffef2aa03f88f9 [term 4 FOLLOWER]: Advancing to term 5
I20260129 02:59:50.881006 20508 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3e2d15f8640b407195210c476e09df91" candidate_uuid: "7a9daf5d42a84615a6383cfec9a10608" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "974b802141bd4f13beffef2aa03f88f9"
I20260129 02:59:50.920539 20433 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c2633a6fc34c44579de348d208b0c01e" candidate_uuid: "974b802141bd4f13beffef2aa03f88f9" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "197b880fa39d432a9b472845876bfaf0" is_pre_election: true
W20260129 02:59:50.822005 22813 log.cc:927] Time spent T 368953c9102643b2a21f99af36b51905 P 974b802141bd4f13beffef2aa03f88f9: Append to log took a long time: real 0.071s user 0.000s sys 0.000s
W20260129 02:59:51.060240 20480 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.211.4:40439, user_credentials={real_user=slave}} blocked reactor thread for 323518us
I20260129 02:59:51.063792 20586 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4f8f3a38f6e24ba4857875181f447442" candidate_uuid: "974b802141bd4f13beffef2aa03f88f9" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "1c50de89a5a845668696dcb28c3badd6" is_pre_election: true
I20260129 02:59:50.889374 20439 rpcz_store.cc:276] 0129 02:59:48.201002 (+ 0us) service_pool.cc:168] Inserting onto call queue
0129 02:59:48.201159 (+ 157us) service_pool.cc:225] Handling call
0129 02:59:48.218712 (+ 17553us) raft_consensus.cc:1471] Updating replica for 0 ops
0129 02:59:49.612488 (+1393776us) raft_consensus.cc:1697] Filling consensus response to leader.
0129 02:59:50.844357 (+1231869us) spinlock_profiling.cc:243] Waited 1.39 s on lock 0x7b5400323cd4. stack: 00007f8fe044e373 00007f8fe044e270 00007f8fdfda0574 00007f8fe9a5c6b4 00007f8fe9a5c649 00007f8fe34975a1 00007f8fe352ea81 00007f8fe352d4be 00007f8fe84e1f49 00007f8fe1a6ed34 00007f8fe1a6ecc2 00007f8fe1a6ebf1 00007f8fe1a6eb6c 00007f8fe1a6de22 00007f8fe186d34c 00007f8fe186c786
0129 02:59:50.844623 (+ 266us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":3203879168}
W20260129 02:59:51.063971 21102 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.211.2:34517, user_credentials={real_user=slave}} blocked reactor thread for 174433us
I20260129 02:59:51.069409 20507 raft_consensus.cc:2468] T 505580c93c694bbb8ea96fdf9b17ab46 P 974b802141bd4f13beffef2aa03f88f9 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1c50de89a5a845668696dcb28c3badd6 in term 4.
I20260129 02:59:51.067034 20515 raft_consensus.cc:2393] T cb4727386a3e48aca5d71cc6c611c197 P 974b802141bd4f13beffef2aa03f88f9 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1c50de89a5a845668696dcb28c3badd6 in current term 3: Already voted for candidate 974b802141bd4f13beffef2aa03f88f9 in this term.
I20260129 02:59:51.065515 20514 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4e420f90884f4796997b5ea41e49e411" candidate_uuid: "7a9daf5d42a84615a6383cfec9a10608" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "974b802141bd4f13beffef2aa03f88f9" is_pre_election: true
I20260129 02:59:51.070394 20556 leader_election.cc:304] T 17d6431cc77a491e8464d68db4081a51 P 1c50de89a5a845668696dcb28c3badd6 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1c50de89a5a845668696dcb28c3badd6; no voters: 197b880fa39d432a9b472845876bfaf0, 974b802141bd4f13beffef2aa03f88f9
I20260129 02:59:50.881765 22638 raft_consensus.cc:493] T a1a39dcd328e4268ae41b988c66b4a75 P 1c50de89a5a845668696dcb28c3badd6 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260129 02:59:50.756541 22403 raft_consensus.cc:2749] T a609ac02fae84725ab743f95f51d33ce P 7a9daf5d42a84615a6383cfec9a10608 [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20260129 02:59:50.859629 22425 raft_consensus.cc:3055] T 0f20af7fbe5d4db89a6f17948d98912c P 1c50de89a5a845668696dcb28c3badd6 [term 4 LEADER]: Stepping down as leader of term 4
I20260129 02:59:50.745493 22785 consensus_queue.cc:1243] T 0f20af7fbe5d4db89a6f17948d98912c P 1c50de89a5a845668696dcb28c3badd6 [LEADER]: Peer 197b880fa39d432a9b472845876bfaf0 log is divergent from this leader: its last log entry 5.4 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 3
I20260129 02:59:50.755072 22671 raft_consensus.cc:3060] T febffe4a8b394368b69b1d164308fb9a P 7a9daf5d42a84615a6383cfec9a10608 [term 2 FOLLOWER]: Advancing to term 3
I20260129 02:59:50.757093 22691 raft_consensus.cc:493] T 2ac659d732314e1288849f08fe5e074e P 197b880fa39d432a9b472845876bfaf0 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260129 02:59:51.061352 21134 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34060 (request call id 487) took 4005 ms. Trace:
I20260129 02:59:51.068455 20480 leader_election.cc:304] T 87d5fe48a44a408aaf03e894a67ecb2f P 974b802141bd4f13beffef2aa03f88f9 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 974b802141bd4f13beffef2aa03f88f9; no voters: 197b880fa39d432a9b472845876bfaf0, 7a9daf5d42a84615a6383cfec9a10608
I20260129 02:59:50.727017 22802 consensus_queue.cc:1048] T dc2f4cb752cf4d59b5f6bc689f5d8104 P 1c50de89a5a845668696dcb28c3badd6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.345s
I20260129 02:59:51.071466 21133 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3e2d15f8640b407195210c476e09df91" candidate_uuid: "1c50de89a5a845668696dcb28c3badd6" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "7a9daf5d42a84615a6383cfec9a10608" is_pre_election: true
I20260129 02:59:51.091406 21133 raft_consensus.cc:2468] T 3e2d15f8640b407195210c476e09df91 P 7a9daf5d42a84615a6383cfec9a10608 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1c50de89a5a845668696dcb28c3badd6 in term 2.
I20260129 02:59:51.091655 20586 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4e3805ffa70242e8ade66da260dddf66" candidate_uuid: "197b880fa39d432a9b472845876bfaf0" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "1c50de89a5a845668696dcb28c3badd6" is_pre_election: true
I20260129 02:59:51.072867 20515 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e4d18c35b0a7493e85d0a4b92956a48f" candidate_uuid: "1c50de89a5a845668696dcb28c3badd6" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "974b802141bd4f13beffef2aa03f88f9" is_pre_election: true
I20260129 02:59:51.073829 20588 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "cb4727386a3e48aca5d71cc6c611c197" candidate_uuid: "974b802141bd4f13beffef2aa03f88f9" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1c50de89a5a845668696dcb28c3badd6"
I20260129 02:59:51.074752 20437 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a5a05fbcbbcf4916bbc01fd1a5b5e90d" candidate_uuid: "974b802141bd4f13beffef2aa03f88f9" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "197b880fa39d432a9b472845876bfaf0" is_pre_election: true
I20260129 02:59:51.066776 20583 raft_consensus.cc:2410] T 7a9a69cd82724b1faed1d10aa091fc4a P 1c50de89a5a845668696dcb28c3badd6 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 197b880fa39d432a9b472845876bfaf0 for term 5 because replica has last-logged OpId of term: 4 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 2.
W20260129 02:59:51.093508 20589 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:35754 (request call id 1658) took 1590 ms (client timeout 2023 ms). Trace:
I20260129 02:59:50.754877 22584 leader_election.cc:290] T cedce331e1584ac8b636c9b8a72b2ba0 P 1c50de89a5a845668696dcb28c3badd6 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 197b880fa39d432a9b472845876bfaf0 (127.18.211.1:42027), 974b802141bd4f13beffef2aa03f88f9 (127.18.211.2:34517)
W20260129 02:59:51.093923 20589 rpcz_store.cc:269] 0129 02:59:49.502675 (+ 0us) service_pool.cc:168] Inserting onto call queue
0129 02:59:49.763248 (+260573us) service_pool.cc:225] Handling call
0129 02:59:51.093351 (+1330103us) spinlock_profiling.cc:243] Waited 107 ms on lock 0x7b5400087ad4. stack: 00007f8fe044e373 00007f8fe044e270 00007f8fdfda0574 00007f8fe9a5c6b4 00007f8fe9a5c649 00007f8fe34975a1 00007f8fe3531f7a 00007f8fe84e3647 00007f8fe1a73834 00007f8fe1a737c2 00007f8fe1a736f1 00007f8fe1a7366c 00007f8fe1a72922 00007f8fe186d34c 00007f8fe186c786 00007f8fe186c12f
0129 02:59:51.093449 (+ 98us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":245333376}
I20260129 02:59:51.068092 22707 raft_consensus.cc:2804] T 4e420f90884f4796997b5ea41e49e411 P 974b802141bd4f13beffef2aa03f88f9 [term 4 FOLLOWER]: Leader pre-election won for term 5
W20260129 02:59:51.067251 21101 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260129 02:59:50.859654 22763 raft_consensus.cc:697] T e4d18c35b0a7493e85d0a4b92956a48f P 974b802141bd4f13beffef2aa03f88f9 [term 5 LEADER]: Becoming Leader. State: Replica: 974b802141bd4f13beffef2aa03f88f9, State: Running, Role: LEADER
I20260129 02:59:50.883168 22742 raft_consensus.cc:2749] T c934d31efd3f4e5195605ddc354d14e1 P 7a9daf5d42a84615a6383cfec9a10608 [term 5 FOLLOWER]: Leader election lost for term 4. Reason: Vote denied by peer 197b880fa39d432a9b472845876bfaf0 (127.18.211.1:42027) with higher term. Message: Invalid argument: T c934d31efd3f4e5195605ddc354d14e1 P 197b880fa39d432a9b472845876bfaf0 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 7a9daf5d42a84615a6383cfec9a10608 for earlier term 4. Current term is 5.
I20260129 02:59:50.728649 22740 leader_election.cc:290] T 42d27a28d89b409e8807ec0efd97af69 P 1c50de89a5a845668696dcb28c3badd6 [CANDIDATE]: Term 6 election: Requested vote from peers 197b880fa39d432a9b472845876bfaf0 (127.18.211.1:42027), 7a9daf5d42a84615a6383cfec9a10608 (127.18.211.4:40439)
I20260129 02:59:50.821836 22765 leader_election.cc:290] T e33a417cf9c54cf5af73a4ee3aac4d3a P 974b802141bd4f13beffef2aa03f88f9 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 197b880fa39d432a9b472845876bfaf0 (127.18.211.1:42027), 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3:37735)
I20260129 02:59:50.925309 22581 raft_consensus.cc:2749] T 96609ac68bdf4993a12e1c23355ca013 P 1c50de89a5a845668696dcb28c3badd6 [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260129 02:59:50.756968 22687 leader_election.cc:290] T c2633a6fc34c44579de348d208b0c01e P 974b802141bd4f13beffef2aa03f88f9 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 197b880fa39d432a9b472845876bfaf0 (127.18.211.1:42027), 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3:37735)
I20260129 02:59:51.113411 22435 raft_consensus.cc:2749] T fcddab6bca684adf8ea032289a00f90e P 1c50de89a5a845668696dcb28c3badd6 [term 8 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260129 02:59:50.831954 22814 raft_consensus.cc:2804] T 724a236524cb4e7ba3428880681d6cc2 P 1c50de89a5a845668696dcb28c3badd6 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260129 02:59:51.084789 21130 raft_consensus.cc:2393] T 18f30ebaa4174fd28c958e72d0cc7e24 P 7a9daf5d42a84615a6383cfec9a10608 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 197b880fa39d432a9b472845876bfaf0 in current term 5: Already voted for candidate 7a9daf5d42a84615a6383cfec9a10608 in this term.
W20260129 02:59:51.122699 20438 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:48356 (request call id 295) took 2839 ms (client timeout 2212 ms). Trace:
W20260129 02:59:51.123164 20438 rpcz_store.cc:269] 0129 02:59:48.283604 (+ 0us) service_pool.cc:168] Inserting onto call queue
0129 02:59:48.283703 (+ 99us) service_pool.cc:225] Handling call
0129 02:59:51.122498 (+2838795us) spinlock_profiling.cc:243] Waited 254 ms on lock 0x7b54003511d4. stack: 00007f8fe044e373 00007f8fe044e270 00007f8fdfda0574 00007f8fe9a5c6b4 00007f8fe9a5c649 00007f8fe34975a1 00007f8fe3531f7a 00007f8fe84e3647 00007f8fe1a73834 00007f8fe1a737c2 00007f8fe1a736f1 00007f8fe1a7366c 00007f8fe1a72922 00007f8fe186d34c 00007f8fe186c786 00007f8fe186c12f
0129 02:59:51.122624 (+ 126us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":585022208}
I20260129 02:59:51.124357 22763 consensus_queue.cc:237] T e4d18c35b0a7493e85d0a4b92956a48f P 974b802141bd4f13beffef2aa03f88f9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:59:51.124868 20510 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57600 (request call id 1745) took 4707 ms. Trace:
I20260129 02:59:51.125617 20438 raft_consensus.cc:1240] T 368953c9102643b2a21f99af36b51905 P 197b880fa39d432a9b472845876bfaf0 [term 6 FOLLOWER]: Rejecting Update request from peer 974b802141bd4f13beffef2aa03f88f9 for earlier term 5. Current term is 6. Ops: [4.3-4.3]
I20260129 02:59:51.125449 20510 rpcz_store.cc:276] 0129 02:59:46.417163 (+ 0us) service_pool.cc:168] Inserting onto call queue
0129 02:59:47.819537 (+1402374us) service_pool.cc:225] Handling call
0129 02:59:48.099083 (+279546us) raft_consensus.cc:1471] Updating replica for 1 ops
0129 02:59:48.189114 (+ 90031us) raft_consensus.cc:1534] Early marking committed up to index 0
0129 02:59:48.189148 (+ 34us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0129 02:59:50.751410 (+2562262us) spinlock_profiling.cc:243] Waited 2.02 ms on lock 0x7b2c00010828. stack: 00007f8fe044e373 00007f8fe044e270 00007f8fdfda0574 00007f8fe9a5c6b4 00007f8fe9a5c649 00007f8fe1e2c9b1 00007f8fe1e3bfe2 00007f8fe3569851 00007f8fe352e0ba 00007f8fe352d4be 00007f8fe84e1f49 00007f8fe1a6ed34 00007f8fe1a6ecc2 00007f8fe1a6ebf1 00007f8fe1a6eb6c 00007f8fe1a6de22
0129 02:59:50.751850 (+ 440us) log.cc:844] Serialized 26 byte log entry
0129 02:59:50.924827 (+172977us) raft_consensus.cc:1645] Marking committed up to 6
0129 02:59:50.924865 (+ 38us) raft_consensus.cc:1697] Filling consensus response to leader.
0129 02:59:50.924919 (+ 54us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0129 02:59:51.124470 (+199551us) raft_consensus.cc:1685] finished
0129 02:59:51.124502 (+ 32us) raft_consensus.cc:1691] UpdateReplicas() finished
0129 02:59:51.124774 (+ 272us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":89557,"raft.queue_time_us":94456,"raft.run_cpu_time_us":120,"raft.run_wall_time_us":6191,"spinlock_wait_cycles":4646656,"thread_start_us":144771,"threads_started":1,"wal-append.queue_time_us":321942}
I20260129 02:59:51.129150 22672 leader_election.cc:290] T 18f30ebaa4174fd28c958e72d0cc7e24 P 7a9daf5d42a84615a6383cfec9a10608 [CANDIDATE]: Term 5 election: Requested vote from peers 197b880fa39d432a9b472845876bfaf0 (127.18.211.1:42027), 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3:37735)
I20260129 02:59:51.130184 21133 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "42d27a28d89b409e8807ec0efd97af69" candidate_uuid: "1c50de89a5a845668696dcb28c3badd6" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "7a9daf5d42a84615a6383cfec9a10608"
I20260129 02:59:51.131759 22688 leader_election.cc:290] T 4e3805ffa70242e8ade66da260dddf66 P 197b880fa39d432a9b472845876bfaf0 [CANDIDATE]: Term 4 election: Requested vote from peers 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3:37735), 7a9daf5d42a84615a6383cfec9a10608 (127.18.211.4:40439)
I20260129 02:59:51.096225 22348 raft_consensus.cc:493] T 7a9a69cd82724b1faed1d10aa091fc4a P 1c50de89a5a845668696dcb28c3badd6 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260129 02:59:51.134382 22348 raft_consensus.cc:515] T 7a9a69cd82724b1faed1d10aa091fc4a P 1c50de89a5a845668696dcb28c3badd6 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:59:51.103062 20436 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "42d27a28d89b409e8807ec0efd97af69" candidate_uuid: "1c50de89a5a845668696dcb28c3badd6" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "197b880fa39d432a9b472845876bfaf0"
I20260129 02:59:51.136487 20436 raft_consensus.cc:3060] T 42d27a28d89b409e8807ec0efd97af69 P 197b880fa39d432a9b472845876bfaf0 [term 5 FOLLOWER]: Advancing to term 6
I20260129 02:59:51.136883 20438 raft_consensus.cc:1240] T 7d9494f5584d4d5db8b9adf1c8fc888b P 197b880fa39d432a9b472845876bfaf0 [term 3 FOLLOWER]: Rejecting Update request from peer 1c50de89a5a845668696dcb28c3badd6 for earlier term 2. Current term is 3. Ops: []
I20260129 02:59:51.138047 22348 leader_election.cc:290] T 7a9a69cd82724b1faed1d10aa091fc4a P 1c50de89a5a845668696dcb28c3badd6 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 197b880fa39d432a9b472845876bfaf0 (127.18.211.1:42027), 974b802141bd4f13beffef2aa03f88f9 (127.18.211.2:34517)
I20260129 02:59:51.140404 22700 leader_election.cc:290] T 505580c93c694bbb8ea96fdf9b17ab46 P 974b802141bd4f13beffef2aa03f88f9 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3:37735), 7a9daf5d42a84615a6383cfec9a10608 (127.18.211.4:40439)
I20260129 02:59:51.140666 20516 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57600 (request call id 1748) took 4722 ms. Trace:
I20260129 02:59:51.116436 22814 raft_consensus.cc:493] T 724a236524cb4e7ba3428880681d6cc2 P 1c50de89a5a845668696dcb28c3badd6 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260129 02:59:51.141327 22814 raft_consensus.cc:3060] T 724a236524cb4e7ba3428880681d6cc2 P 1c50de89a5a845668696dcb28c3badd6 [term 3 FOLLOWER]: Advancing to term 4
I20260129 02:59:51.141032 20516 rpcz_store.cc:276] 0129 02:59:46.418076 (+ 0us) service_pool.cc:168] Inserting onto call queue
0129 02:59:48.153920 (+1735844us) service_pool.cc:225] Handling call
0129 02:59:48.154353 (+ 433us) raft_consensus.cc:1471] Updating replica for 1 ops
0129 02:59:50.737664 (+2583311us) raft_consensus.cc:1697] Filling consensus response to leader.
0129 02:59:51.140417 (+402753us) spinlock_profiling.cc:243] Waited 2.58 s on lock 0x7b5400300cd4. stack: 00007f8fe044e373 00007f8fe044e270 00007f8fdfda0574 00007f8fe9a5c6b4 00007f8fe9a5c649 00007f8fe34975a1 00007f8fe352ea81 00007f8fe352d4be 00007f8fe84e1f49 00007f8fe1a6ed34 00007f8fe1a6ecc2 00007f8fe1a6ebf1 00007f8fe1a6eb6c 00007f8fe1a6de22 00007f8fe186d34c 00007f8fe186c786
0129 02:59:51.140608 (+ 191us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":5939663872}
I20260129 02:59:51.101171 22820 raft_consensus.cc:2749] T 87d5fe48a44a408aaf03e894a67ecb2f P 974b802141bd4f13beffef2aa03f88f9 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260129 02:59:51.142330 20516 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "724a236524cb4e7ba3428880681d6cc2" candidate_uuid: "1c50de89a5a845668696dcb28c3badd6" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "974b802141bd4f13beffef2aa03f88f9" is_pre_election: true
I20260129 02:59:50.822500 22641 raft_consensus.cc:515] T e7d83d4ccf684e76acb29a0dc844e1b8 P 197b880fa39d432a9b472845876bfaf0 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:59:51.086006 20581 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "17d6431cc77a491e8464d68db4081a51" candidate_uuid: "197b880fa39d432a9b472845876bfaf0" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1c50de89a5a845668696dcb28c3badd6"
I20260129 02:59:51.101858 22707 raft_consensus.cc:493] T 4e420f90884f4796997b5ea41e49e411 P 974b802141bd4f13beffef2aa03f88f9 [term 4 FOLLOWER]: Starting leader election (detected failure of leader 197b880fa39d432a9b472845876bfaf0)
I20260129 02:59:51.088199 21134 rpcz_store.cc:276] 0129 02:59:47.056401 (+ 0us) service_pool.cc:168] Inserting onto call queue
0129 02:59:47.066135 (+ 9734us) service_pool.cc:225] Handling call
0129 02:59:47.130324 (+ 64189us) raft_consensus.cc:1471] Updating replica for 0 ops
0129 02:59:48.148378 (+1018054us) raft_consensus.cc:1697] Filling consensus response to leader.
0129 02:59:51.061095 (+2912717us) spinlock_profiling.cc:243] Waited 980 ms on lock 0x7b54005631d4. stack: 00007f8fe044e373 00007f8fe044e270 00007f8fdfda0574 00007f8fe9a5c6b4 00007f8fe9a5c649 00007f8fe34975a1 00007f8fe352ea81 00007f8fe352d4be 00007f8fe84e1f49 00007f8fe1a6ed34 00007f8fe1a6ecc2 00007f8fe1a6ebf1 00007f8fe1a6eb6c 00007f8fe1a6de22 00007f8fe186d34c 00007f8fe186c786
0129 02:59:51.061291 (+ 196us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2255127936}
I20260129 02:59:51.058313 22807 raft_consensus.cc:2804] T 0678b8b468bd43cdbc3e485900844821 P 974b802141bd4f13beffef2aa03f88f9 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260129 02:59:50.735134 22400 raft_consensus.cc:515] T 3e2d15f8640b407195210c476e09df91 P 974b802141bd4f13beffef2aa03f88f9 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } }
I20260129 02:59:51.151512 20510 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "75be4bd6bd45474795fd472eff2953ac" candidate_uuid: "1c50de89a5a845668696dcb28c3badd6" candidate_term: 9 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "974b802141bd4f13beffef2aa03f88f9"
I20260129 02:59:51.152390 20510 raft_consensus.cc:3060] T 75be4bd6bd45474795fd472eff2953ac P 974b802141bd4f13beffef2aa03f88f9 [term 8 FOLLOWER]: Advancing to term 9
I20260129 02:59:51.115463 22694 leader_election.cc:290] T 75b0b6e82ddc4f92b1fe1d94f9a0b73b P 974b802141bd4f13beffef2aa03f88f9 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3:37735), 7a9daf5d42a84615a6383cfec9a10608 (127.18.211.4:40439)
I20260129 02:59:51.068652 20433 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a1a39dcd328e4268ae41b988c66b4a75" candidate_uuid: "974b802141bd4f13beffef2aa03f88f9" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "197b880fa39d432a9b472845876bfaf0"
I20260129 02:59:50.860023 22745 leader_election.cc:290] T 87d5fe48a44a408aaf03e894a67ecb2f P 974b802141bd4f13beffef2aa03f88f9 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 197b880fa39d432a9b472845876bfaf0 (127.18.211.1:42027), 7a9daf5d42a84615a6383cfec9a10608 (127.18.211.4:40439)
I20260129 02:59:51.065105 22815 raft_consensus.cc:493] T b189f08e16a74a11980cc55d99359a0a P 974b802141bd4f13beffef2aa03f88f9 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260129 02:59:51.098035 20588 raft_consensus.cc:2393] T cb4727386a3e48aca5d71cc6c611c197 P 1c50de89a5a845668696dcb28c3badd6 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 974b802141bd4f13beffef2aa03f88f9 in current term 3: Already voted for candidate 197b880fa39d432a9b472845876bfaf0 in this term.
I20260129 02:59:50.756320 22663 consensus_queue.cc:260] T 3906b080a5b34a74b8b64128635e8997 P 1c50de89a5a845668696dcb28c3badd6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:59:51.087764 22656 raft_consensus.cc:2749] T 17d6431cc77a491e8464d68db4081a51 P 1c50de89a5a845668696dcb28c3badd6 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260129 02:59:51.100296 20480 leader_election.cc:304] T 75f45538dc704d9a8ec4ff8ad9f3ea5b P 974b802141bd4f13beffef2aa03f88f9 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1c50de89a5a845668696dcb28c3badd6, 974b802141bd4f13beffef2aa03f88f9; no voters: 7a9daf5d42a84615a6383cfec9a10608
W20260129 02:59:51.163763 20480 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.211.3:37735, user_credentials={real_user=slave}} blocked reactor thread for 63537.2us
I20260129 02:59:51.082619 22638 raft_consensus.cc:3060] T a1a39dcd328e4268ae41b988c66b4a75 P 1c50de89a5a845668696dcb28c3badd6 [term 3 FOLLOWER]: Advancing to term 4
I20260129 02:59:50.897533 22160 raft_consensus.cc:515] T a8479d1fe0bd41e5946186e3a6ca8c7a P 974b802141bd4f13beffef2aa03f88f9 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:59:51.101611 20589 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4408006eb5174de1896b037cf3f18457" candidate_uuid: "974b802141bd4f13beffef2aa03f88f9" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1c50de89a5a845668696dcb28c3badd6" is_pre_election: true
I20260129 02:59:51.076460 22729 raft_consensus.cc:493] T 0678b8b468bd43cdbc3e485900844821 P 7a9daf5d42a84615a6383cfec9a10608 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 197b880fa39d432a9b472845876bfaf0)
I20260129 02:59:51.090643 20439 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0f20af7fbe5d4db89a6f17948d98912c" candidate_uuid: "974b802141bd4f13beffef2aa03f88f9" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "197b880fa39d432a9b472845876bfaf0" is_pre_election: true
W20260129 02:59:51.104388 20587 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:35754 (request call id 1638) took 2313 ms (client timeout 2087 ms). Trace:
W20260129 02:59:51.176061 20587 rpcz_store.cc:269] 0129 02:59:48.824473 (+ 0us) service_pool.cc:168] Inserting onto call queue
0129 02:59:48.835068 (+ 10595us) service_pool.cc:225] Handling call
0129 02:59:51.094985 (+2259917us) spinlock_profiling.cc:243] Waited 520 ms on lock 0x7b540018bfd4. stack: 00007f8fe044e373 00007f8fe044e270 00007f8fdfda0574 00007f8fe9a5c6b4 00007f8fe9a5c649 00007f8fe34975a1 00007f8fe3531f7a 00007f8fe84e3647 00007f8fe1a73834 00007f8fe1a737c2 00007f8fe1a736f1 00007f8fe1a7366c 00007f8fe1a72922 00007f8fe186d34c 00007f8fe186c786 00007f8fe186c12f
0129 02:59:51.096944 (+ 1959us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1197134720}
I20260129 02:59:51.085640 22425 raft_consensus.cc:740] T 0f20af7fbe5d4db89a6f17948d98912c P 1c50de89a5a845668696dcb28c3badd6 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 1c50de89a5a845668696dcb28c3badd6, State: Running, Role: LEADER
I20260129 02:59:51.064040 22667 raft_consensus.cc:515] T 00e79d8f25a34676b1d866eb2068a2b2 P 7a9daf5d42a84615a6383cfec9a10608 [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } }
I20260129 02:59:51.108031 21131 raft_consensus.cc:2393] T c934d31efd3f4e5195605ddc354d14e1 P 7a9daf5d42a84615a6383cfec9a10608 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 974b802141bd4f13beffef2aa03f88f9 in current term 5: Already voted for candidate 197b880fa39d432a9b472845876bfaf0 in this term.
I20260129 02:59:51.108609 20437 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "cedce331e1584ac8b636c9b8a72b2ba0" candidate_uuid: "1c50de89a5a845668696dcb28c3badd6" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "197b880fa39d432a9b472845876bfaf0" is_pre_election: true
W20260129 02:59:51.114630 20481 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260129 02:59:51.118700 20586 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "cedce331e1584ac8b636c9b8a72b2ba0" candidate_uuid: "974b802141bd4f13beffef2aa03f88f9" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "1c50de89a5a845668696dcb28c3badd6"
I20260129 02:59:51.073491 20584 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4f5a77e94fef4b16a2ba7b067c8b144a" candidate_uuid: "974b802141bd4f13beffef2aa03f88f9" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "1c50de89a5a845668696dcb28c3badd6" is_pre_election: true
I20260129 02:59:51.143019 20516 raft_consensus.cc:2468] T 724a236524cb4e7ba3428880681d6cc2 P 974b802141bd4f13beffef2aa03f88f9 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1c50de89a5a845668696dcb28c3badd6 in term 3.
I20260129 02:59:51.094887 20556 leader_election.cc:304] T 4b1876f9a70b49edb3aa74105e66d3b3 P 1c50de89a5a845668696dcb28c3badd6 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1c50de89a5a845668696dcb28c3badd6; no voters: 197b880fa39d432a9b472845876bfaf0, 974b802141bd4f13beffef2aa03f88f9
I20260129 02:59:51.148674 22707 raft_consensus.cc:3060] T 4e420f90884f4796997b5ea41e49e411 P 974b802141bd4f13beffef2aa03f88f9 [term 4 FOLLOWER]: Advancing to term 5
I20260129 02:59:51.205375 21133 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "75b0b6e82ddc4f92b1fe1d94f9a0b73b" candidate_uuid: "974b802141bd4f13beffef2aa03f88f9" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "7a9daf5d42a84615a6383cfec9a10608" is_pre_election: true
I20260129 02:59:51.206372 21133 raft_consensus.cc:2468] T 75b0b6e82ddc4f92b1fe1d94f9a0b73b P 7a9daf5d42a84615a6383cfec9a10608 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 974b802141bd4f13beffef2aa03f88f9 in term 3.
I20260129 02:59:51.163029 22663 raft_consensus.cc:3060] T 3906b080a5b34a74b8b64128635e8997 P 1c50de89a5a845668696dcb28c3badd6 [term 2 FOLLOWER]: Advancing to term 3
I20260129 02:59:51.210991 20480 leader_election.cc:304] T 75b0b6e82ddc4f92b1fe1d94f9a0b73b P 974b802141bd4f13beffef2aa03f88f9 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7a9daf5d42a84615a6383cfec9a10608, 974b802141bd4f13beffef2aa03f88f9; no voters:
I20260129 02:59:51.161626 20433 raft_consensus.cc:2393] T a1a39dcd328e4268ae41b988c66b4a75 P 197b880fa39d432a9b472845876bfaf0 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 974b802141bd4f13beffef2aa03f88f9 in current term 3: Already voted for candidate 197b880fa39d432a9b472845876bfaf0 in this term.
I20260129 02:59:51.212864 22745 raft_consensus.cc:2764] T 75b0b6e82ddc4f92b1fe1d94f9a0b73b P 974b802141bd4f13beffef2aa03f88f9 [term 3 FOLLOWER]: Leader pre-election decision vote started in defunct term 2: won
I20260129 02:59:51.213382 22667 leader_election.cc:290] T 00e79d8f25a34676b1d866eb2068a2b2 P 7a9daf5d42a84615a6383cfec9a10608 [CANDIDATE]: Term 6 election: Requested vote from peers 197b880fa39d432a9b472845876bfaf0 (127.18.211.1:42027), 974b802141bd4f13beffef2aa03f88f9 (127.18.211.2:34517)
I20260129 02:59:51.173493 22820 raft_consensus.cc:2804] T 75f45538dc704d9a8ec4ff8ad9f3ea5b P 974b802141bd4f13beffef2aa03f88f9 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260129 02:59:51.214375 22820 raft_consensus.cc:493] T 75f45538dc704d9a8ec4ff8ad9f3ea5b P 974b802141bd4f13beffef2aa03f88f9 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 7a9daf5d42a84615a6383cfec9a10608)
I20260129 02:59:51.214745 22820 raft_consensus.cc:3060] T 75f45538dc704d9a8ec4ff8ad9f3ea5b P 974b802141bd4f13beffef2aa03f88f9 [term 2 FOLLOWER]: Advancing to term 3
I20260129 02:59:51.216687 20584 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "17d6431cc77a491e8464d68db4081a51" candidate_uuid: "197b880fa39d432a9b472845876bfaf0" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1c50de89a5a845668696dcb28c3badd6"
W20260129 02:59:51.217110 22816 log.cc:927] Time spent T 78aa39096868415d9afc2ffcb248e198 P 1c50de89a5a845668696dcb28c3badd6: Append to log took a long time: real 0.521s user 0.015s sys 0.017s
I20260129 02:59:51.217523 20584 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:35754 (request call id 1688) took 1339 ms. Trace:
I20260129 02:59:51.217908 20584 rpcz_store.cc:276] 0129 02:59:49.881153 (+ 0us) service_pool.cc:168] Inserting onto call queue
0129 02:59:51.216349 (+1335196us) service_pool.cc:225] Handling call
0129 02:59:51.217463 (+ 1114us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260129 02:59:51.218855 20584 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "45cc5f3d2688486d9b6dbe7e666dedde" candidate_uuid: "974b802141bd4f13beffef2aa03f88f9" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "1c50de89a5a845668696dcb28c3badd6"
I20260129 02:59:51.181281 22729 raft_consensus.cc:515] T 0678b8b468bd43cdbc3e485900844821 P 7a9daf5d42a84615a6383cfec9a10608 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } }
I20260129 02:59:51.226442 22729 leader_election.cc:290] T 0678b8b468bd43cdbc3e485900844821 P 7a9daf5d42a84615a6383cfec9a10608 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 197b880fa39d432a9b472845876bfaf0 (127.18.211.1:42027), 974b802141bd4f13beffef2aa03f88f9 (127.18.211.2:34517)
I20260129 02:59:51.227044 22656 raft_consensus.cc:2749] T 4b1876f9a70b49edb3aa74105e66d3b3 P 1c50de89a5a845668696dcb28c3badd6 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260129 02:59:51.228761 20511 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:57584 (request call id 1304) took 1719 ms. Trace:
I20260129 02:59:51.229174 20511 rpcz_store.cc:276] 0129 02:59:49.508953 (+ 0us) service_pool.cc:168] Inserting onto call queue
0129 02:59:49.910273 (+401320us) service_pool.cc:225] Handling call
0129 02:59:51.228536 (+1318263us) spinlock_profiling.cc:243] Waited 300 ms on lock 0x7b5400128754. stack: 00007f8fe044e373 00007f8fe044e270 00007f8fdfda0574 00007f8fe9a5c6b4 00007f8fe9a5c649 00007f8fe34975a1 00007f8fe3531f7a 00007f8fe84e3647 00007f8fe1a73834 00007f8fe1a737c2 00007f8fe1a736f1 00007f8fe1a7366c 00007f8fe1a72922 00007f8fe186d34c 00007f8fe186c786 00007f8fe186c12f
0129 02:59:51.228699 (+ 163us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":689091328}
I20260129 02:59:51.232951 20584 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "75b0b6e82ddc4f92b1fe1d94f9a0b73b" candidate_uuid: "974b802141bd4f13beffef2aa03f88f9" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "1c50de89a5a845668696dcb28c3badd6" is_pre_election: true
I20260129 02:59:51.166826 21136 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "505580c93c694bbb8ea96fdf9b17ab46" candidate_uuid: "974b802141bd4f13beffef2aa03f88f9" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "7a9daf5d42a84615a6383cfec9a10608" is_pre_election: true
W20260129 02:59:51.169633 20554 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.211.4:40439, user_credentials={real_user=slave}} blocked reactor thread for 82917.2us
I20260129 02:59:51.131742 22786 raft_consensus.cc:515] T e4d18c35b0a7493e85d0a4b92956a48f P 1c50de89a5a845668696dcb28c3badd6 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:59:51.173347 20589 raft_consensus.cc:2468] T 4408006eb5174de1896b037cf3f18457 P 1c50de89a5a845668696dcb28c3badd6 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 974b802141bd4f13beffef2aa03f88f9 in term 2.
W20260129 02:59:51.234102 21099 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260129 02:59:51.173275 20438 raft_consensus.cc:1240] T e7d83d4ccf684e76acb29a0dc844e1b8 P 197b880fa39d432a9b472845876bfaf0 [term 5 FOLLOWER]: Rejecting Update request from peer 1c50de89a5a845668696dcb28c3badd6 for earlier term 4. Current term is 5. Ops: []
I20260129 02:59:51.102562 22691 raft_consensus.cc:515] T 2ac659d732314e1288849f08fe5e074e P 197b880fa39d432a9b472845876bfaf0 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } }
I20260129 02:59:51.177721 20581 raft_consensus.cc:2393] T 17d6431cc77a491e8464d68db4081a51 P 1c50de89a5a845668696dcb28c3badd6 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 197b880fa39d432a9b472845876bfaf0 in current term 3: Already voted for candidate 1c50de89a5a845668696dcb28c3badd6 in this term.
I20260129 02:59:51.178045 20588 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "18f30ebaa4174fd28c958e72d0cc7e24" candidate_uuid: "197b880fa39d432a9b472845876bfaf0" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "1c50de89a5a845668696dcb28c3badd6" is_pre_election: true
I20260129 02:59:51.236675 20584 raft_consensus.cc:2393] T 75b0b6e82ddc4f92b1fe1d94f9a0b73b P 1c50de89a5a845668696dcb28c3badd6 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 974b802141bd4f13beffef2aa03f88f9 in current term 3: Already voted for candidate 1c50de89a5a845668696dcb28c3badd6 in this term.
I20260129 02:59:51.236748 20588 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:35754 (request call id 1692) took 1007 ms. Trace:
I20260129 02:59:51.234705 21136 raft_consensus.cc:2468] T 505580c93c694bbb8ea96fdf9b17ab46 P 7a9daf5d42a84615a6383cfec9a10608 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 974b802141bd4f13beffef2aa03f88f9 in term 3.
I20260129 02:59:51.180262 21134 raft_consensus.cc:1275] T 2ac659d732314e1288849f08fe5e074e P 7a9daf5d42a84615a6383cfec9a10608 [term 3 FOLLOWER]: Refusing update from remote peer 974b802141bd4f13beffef2aa03f88f9: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 3 index: 5. (index mismatch)
I20260129 02:59:51.190346 20516 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3ccac82a01a2445fa5b0ce6c0fcf2295" candidate_uuid: "7a9daf5d42a84615a6383cfec9a10608" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "974b802141bd4f13beffef2aa03f88f9"
I20260129 02:59:51.238267 20480 leader_election.cc:304] T 505580c93c694bbb8ea96fdf9b17ab46 P 974b802141bd4f13beffef2aa03f88f9 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7a9daf5d42a84615a6383cfec9a10608, 974b802141bd4f13beffef2aa03f88f9; no voters:
I20260129 02:59:51.238632 20554 leader_election.cc:304] T 505580c93c694bbb8ea96fdf9b17ab46 P 1c50de89a5a845668696dcb28c3badd6 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1c50de89a5a845668696dcb28c3badd6, 7a9daf5d42a84615a6383cfec9a10608; no voters:
I20260129 02:59:51.238925 20516 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a644c02b16af47679824ccd060ac10fd" candidate_uuid: "7a9daf5d42a84615a6383cfec9a10608" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "974b802141bd4f13beffef2aa03f88f9" is_pre_election: true
I20260129 02:59:51.239593 20516 raft_consensus.cc:2468] T a644c02b16af47679824ccd060ac10fd P 974b802141bd4f13beffef2aa03f88f9 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7a9daf5d42a84615a6383cfec9a10608 in term 3.
I20260129 02:59:51.241237 20516 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3e2d15f8640b407195210c476e09df91" candidate_uuid: "7a9daf5d42a84615a6383cfec9a10608" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "974b802141bd4f13beffef2aa03f88f9"
I20260129 02:59:51.241431 21102 leader_election.cc:304] T a644c02b16af47679824ccd060ac10fd P 7a9daf5d42a84615a6383cfec9a10608 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 7a9daf5d42a84615a6383cfec9a10608, 974b802141bd4f13beffef2aa03f88f9; no voters: 1c50de89a5a845668696dcb28c3badd6
I20260129 02:59:51.237355 20588 rpcz_store.cc:276] 0129 02:59:50.229513 (+ 0us) service_pool.cc:168] Inserting onto call queue
0129 02:59:51.161516 (+932003us) service_pool.cc:225] Handling call
0129 02:59:51.236669 (+ 75153us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260129 02:59:51.242282 20516 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0678b8b468bd43cdbc3e485900844821" candidate_uuid: "7a9daf5d42a84615a6383cfec9a10608" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "974b802141bd4f13beffef2aa03f88f9" is_pre_election: true
I20260129 02:59:51.242928 20516 raft_consensus.cc:2468] T 0678b8b468bd43cdbc3e485900844821 P 974b802141bd4f13beffef2aa03f88f9 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7a9daf5d42a84615a6383cfec9a10608 in term 5.
I20260129 02:59:51.243467 21102 leader_election.cc:304] T 3e2d15f8640b407195210c476e09df91 P 7a9daf5d42a84615a6383cfec9a10608 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7a9daf5d42a84615a6383cfec9a10608; no voters: 1c50de89a5a845668696dcb28c3badd6, 974b802141bd4f13beffef2aa03f88f9
I20260129 02:59:51.159366 20508 raft_consensus.cc:2393] T 3e2d15f8640b407195210c476e09df91 P 974b802141bd4f13beffef2aa03f88f9 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 7a9daf5d42a84615a6383cfec9a10608 in current term 2: Already voted for candidate 974b802141bd4f13beffef2aa03f88f9 in this term.
I20260129 02:59:51.235703 20587 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "96609ac68bdf4993a12e1c23355ca013" candidate_uuid: "974b802141bd4f13beffef2aa03f88f9" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "1c50de89a5a845668696dcb28c3badd6"
I20260129 02:59:51.243729 20516 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c934d31efd3f4e5195605ddc354d14e1" candidate_uuid: "7a9daf5d42a84615a6383cfec9a10608" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "974b802141bd4f13beffef2aa03f88f9" is_pre_election: true
I20260129 02:59:51.244853 21102 leader_election.cc:304] T 0678b8b468bd43cdbc3e485900844821 P 7a9daf5d42a84615a6383cfec9a10608 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7a9daf5d42a84615a6383cfec9a10608, 974b802141bd4f13beffef2aa03f88f9; no voters:
I20260129 02:59:51.163190 22815 raft_consensus.cc:515] T b189f08e16a74a11980cc55d99359a0a P 974b802141bd4f13beffef2aa03f88f9 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
W20260129 02:59:51.245656 20556 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.211.2:34517, user_credentials={real_user=slave}} blocked reactor thread for 150857us
I20260129 02:59:51.248637 20589 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e4d18c35b0a7493e85d0a4b92956a48f" candidate_uuid: "974b802141bd4f13beffef2aa03f88f9" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "1c50de89a5a845668696dcb28c3badd6"
I20260129 02:59:51.249506 20587 raft_consensus.cc:2393] T 96609ac68bdf4993a12e1c23355ca013 P 1c50de89a5a845668696dcb28c3badd6 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 974b802141bd4f13beffef2aa03f88f9 in current term 5: Already voted for candidate 7a9daf5d42a84615a6383cfec9a10608 in this term.
I20260129 02:59:51.249553 20510 raft_consensus.cc:2468] T 75be4bd6bd45474795fd472eff2953ac P 974b802141bd4f13beffef2aa03f88f9 [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1c50de89a5a845668696dcb28c3badd6 in term 9.
I20260129 02:59:51.251139 20554 leader_election.cc:304] T 2e58e9a5a8aa4cecafb464addb4f35ed P 1c50de89a5a845668696dcb28c3badd6 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1c50de89a5a845668696dcb28c3badd6; no voters: 197b880fa39d432a9b472845876bfaf0, 7a9daf5d42a84615a6383cfec9a10608
W20260129 02:59:51.244468 20583 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:35754 (request call id 1637) took 2460 ms (client timeout 1888 ms). Trace:
I20260129 02:59:51.273180 20516 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00e79d8f25a34676b1d866eb2068a2b2" candidate_uuid: "7a9daf5d42a84615a6383cfec9a10608" candidate_term: 6 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "974b802141bd4f13beffef2aa03f88f9"
I20260129 02:59:51.281103 20516 raft_consensus.cc:3060] T 00e79d8f25a34676b1d866eb2068a2b2 P 974b802141bd4f13beffef2aa03f88f9 [term 5 FOLLOWER]: Advancing to term 6
I20260129 02:59:51.281607 20554 leader_election.cc:304] T 3e2d15f8640b407195210c476e09df91 P 1c50de89a5a845668696dcb28c3badd6 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1c50de89a5a845668696dcb28c3badd6, 7a9daf5d42a84615a6383cfec9a10608; no voters:
I20260129 02:59:51.246181 22820 raft_consensus.cc:515] T 75f45538dc704d9a8ec4ff8ad9f3ea5b P 974b802141bd4f13beffef2aa03f88f9 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } }
W20260129 02:59:51.280838 20583 rpcz_store.cc:269] 0129 02:59:48.783669 (+ 0us) service_pool.cc:168] Inserting onto call queue
0129 02:59:48.783958 (+ 289us) service_pool.cc:225] Handling call
0129 02:59:51.229549 (+2445591us) spinlock_profiling.cc:243] Waited 1.66 s on lock 0x7b54000889d4. stack: 00007f8fe044e373 00007f8fe044e270 00007f8fdfda0574 00007f8fe9a5c6b4 00007f8fe9a5c649 00007f8fe34975a1 00007f8fe3531f7a 00007f8fe84e3647 00007f8fe1a73834 00007f8fe1a737c2 00007f8fe1a736f1 00007f8fe1a7366c 00007f8fe1a72922 00007f8fe186d34c 00007f8fe186c786 00007f8fe186c12f
0129 02:59:51.244369 (+ 14820us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":3814010752}
I20260129 02:59:51.284669 20441 raft_consensus.cc:2410] T 2ac659d732314e1288849f08fe5e074e P 197b880fa39d432a9b472845876bfaf0 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 7a9daf5d42a84615a6383cfec9a10608 for term 4 because replica has last-logged OpId of term: 2 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 4.
I20260129 02:59:51.288385 20584 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0f20af7fbe5d4db89a6f17948d98912c" candidate_uuid: "974b802141bd4f13beffef2aa03f88f9" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "1c50de89a5a845668696dcb28c3badd6" is_pre_election: true
I20260129 02:59:51.285180 20590 raft_consensus.cc:2393] T e4d18c35b0a7493e85d0a4b92956a48f P 1c50de89a5a845668696dcb28c3badd6 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 974b802141bd4f13beffef2aa03f88f9 in current term 5: Already voted for candidate 1c50de89a5a845668696dcb28c3badd6 in this term.
I20260129 02:59:51.280670 20511 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "87d5fe48a44a408aaf03e894a67ecb2f" candidate_uuid: "7a9daf5d42a84615a6383cfec9a10608" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "974b802141bd4f13beffef2aa03f88f9" is_pre_election: true
I20260129 02:59:51.289650 20511 raft_consensus.cc:2468] T 87d5fe48a44a408aaf03e894a67ecb2f P 974b802141bd4f13beffef2aa03f88f9 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7a9daf5d42a84615a6383cfec9a10608 in term 5.
I20260129 02:59:51.289853 20589 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e4d18c35b0a7493e85d0a4b92956a48f" candidate_uuid: "197b880fa39d432a9b472845876bfaf0" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "1c50de89a5a845668696dcb28c3badd6"
I20260129 02:59:51.290835 21102 leader_election.cc:304] T 87d5fe48a44a408aaf03e894a67ecb2f P 7a9daf5d42a84615a6383cfec9a10608 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 7a9daf5d42a84615a6383cfec9a10608, 974b802141bd4f13beffef2aa03f88f9; no voters: 197b880fa39d432a9b472845876bfaf0
I20260129 02:59:51.282472 20556 leader_election.cc:304] T add8d4923d1a489e86882c3a25692b95 P 1c50de89a5a845668696dcb28c3badd6 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1c50de89a5a845668696dcb28c3badd6, 974b802141bd4f13beffef2aa03f88f9; no voters: 7a9daf5d42a84615a6383cfec9a10608
W20260129 02:59:51.264008 21099 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260129 02:59:51.259761 20553 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260129 02:59:51.294111 20556 leader_election.cc:304] T 75b0b6e82ddc4f92b1fe1d94f9a0b73b P 1c50de89a5a845668696dcb28c3badd6 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1c50de89a5a845668696dcb28c3badd6, 974b802141bd4f13beffef2aa03f88f9; no voters: 7a9daf5d42a84615a6383cfec9a10608
I20260129 02:59:51.286866 20510 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "505580c93c694bbb8ea96fdf9b17ab46" candidate_uuid: "7a9daf5d42a84615a6383cfec9a10608" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "974b802141bd4f13beffef2aa03f88f9" is_pre_election: true
I20260129 02:59:51.295606 20588 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1ce4ddf5b10e4ae8b193b13b2e5936ee" candidate_uuid: "197b880fa39d432a9b472845876bfaf0" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "1c50de89a5a845668696dcb28c3badd6"
I20260129 02:59:51.296379 20588 raft_consensus.cc:2368] T 1ce4ddf5b10e4ae8b193b13b2e5936ee P 1c50de89a5a845668696dcb28c3badd6 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 197b880fa39d432a9b472845876bfaf0 for earlier term 5. Current term is 6.
I20260129 02:59:51.296762 20588 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:35754 (request call id 1689) took 1398 ms. Trace:
I20260129 02:59:51.297032 20588 rpcz_store.cc:276] 0129 02:59:49.898796 (+ 0us) service_pool.cc:168] Inserting onto call queue
0129 02:59:51.258515 (+1359719us) service_pool.cc:225] Handling call
0129 02:59:51.296708 (+ 38193us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260129 02:59:51.297771 20588 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e33a417cf9c54cf5af73a4ee3aac4d3a" candidate_uuid: "974b802141bd4f13beffef2aa03f88f9" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "1c50de89a5a845668696dcb28c3badd6" is_pre_election: true
W20260129 02:59:51.246331 22791 log.cc:927] Time spent T 00000000000000000000000000000000 P a91e6696d0a44ee19bcf2489fd709c38: Append to log took a long time: real 0.071s user 0.000s sys 0.001s
I20260129 02:59:51.299391 20589 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:35754 (request call id 1690) took 1383 ms. Trace:
I20260129 02:59:51.299896 21102 leader_election.cc:304] T 505580c93c694bbb8ea96fdf9b17ab46 P 7a9daf5d42a84615a6383cfec9a10608 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7a9daf5d42a84615a6383cfec9a10608; no voters: 1c50de89a5a845668696dcb28c3badd6, 974b802141bd4f13beffef2aa03f88f9
I20260129 02:59:51.252801 22672 raft_consensus.cc:2749] T 3e2d15f8640b407195210c476e09df91 P 7a9daf5d42a84615a6383cfec9a10608 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260129 02:59:51.246155 22745 raft_consensus.cc:2804] T 505580c93c694bbb8ea96fdf9b17ab46 P 974b802141bd4f13beffef2aa03f88f9 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260129 02:59:51.302314 20556 leader_election.cc:304] T 4408006eb5174de1896b037cf3f18457 P 1c50de89a5a845668696dcb28c3badd6 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1c50de89a5a845668696dcb28c3badd6, 974b802141bd4f13beffef2aa03f88f9; no voters: 197b880fa39d432a9b472845876bfaf0
I20260129 02:59:51.306561 20511 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c934d31efd3f4e5195605ddc354d14e1" candidate_uuid: "7a9daf5d42a84615a6383cfec9a10608" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "974b802141bd4f13beffef2aa03f88f9"
I20260129 02:59:51.307132 20556 leader_election.cc:304] T c891d2bff4ce4568861bbc992e91fbb0 P 1c50de89a5a845668696dcb28c3badd6 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1c50de89a5a845668696dcb28c3badd6; no voters: 197b880fa39d432a9b472845876bfaf0, 974b802141bd4f13beffef2aa03f88f9
I20260129 02:59:51.302651 20588 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a1a39dcd328e4268ae41b988c66b4a75" candidate_uuid: "974b802141bd4f13beffef2aa03f88f9" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "1c50de89a5a845668696dcb28c3badd6"
I20260129 02:59:51.303602 22681 raft_consensus.cc:2749] T 505580c93c694bbb8ea96fdf9b17ab46 P 7a9daf5d42a84615a6383cfec9a10608 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260129 02:59:51.298895 22343 raft_consensus.cc:2804] T 87d5fe48a44a408aaf03e894a67ecb2f P 7a9daf5d42a84615a6383cfec9a10608 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260129 02:59:51.300583 22344 raft_consensus.cc:2804] T add8d4923d1a489e86882c3a25692b95 P 1c50de89a5a845668696dcb28c3badd6 [term 3 FOLLOWER]: Leader election won for term 3
I20260129 02:59:51.304064 22751 raft_consensus.cc:2804] T 75b0b6e82ddc4f92b1fe1d94f9a0b73b P 1c50de89a5a845668696dcb28c3badd6 [term 3 FOLLOWER]: Leader election won for term 3
I20260129 02:59:51.279336 22403 raft_consensus.cc:2804] T 0678b8b468bd43cdbc3e485900844821 P 7a9daf5d42a84615a6383cfec9a10608 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260129 02:59:51.252497 22694 consensus_queue.cc:1048] T 2ac659d732314e1288849f08fe5e074e P 974b802141bd4f13beffef2aa03f88f9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.053s
I20260129 02:59:51.324270 20516 raft_consensus.cc:2468] T 00e79d8f25a34676b1d866eb2068a2b2 P 974b802141bd4f13beffef2aa03f88f9 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7a9daf5d42a84615a6383cfec9a10608 in term 6.
I20260129 02:59:51.283197 22708 raft_consensus.cc:2749] T 2e58e9a5a8aa4cecafb464addb4f35ed P 1c50de89a5a845668696dcb28c3badd6 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260129 02:59:51.308126 22554 raft_consensus.cc:2749] T c891d2bff4ce4568861bbc992e91fbb0 P 1c50de89a5a845668696dcb28c3badd6 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260129 02:59:51.309319 20583 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a5a05fbcbbcf4916bbc01fd1a5b5e90d" candidate_uuid: "974b802141bd4f13beffef2aa03f88f9" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "1c50de89a5a845668696dcb28c3badd6" is_pre_election: true
I20260129 02:59:51.310971 22348 raft_consensus.cc:2804] T 4408006eb5174de1896b037cf3f18457 P 1c50de89a5a845668696dcb28c3badd6 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260129 02:59:51.332746 20583 raft_consensus.cc:2468] T a5a05fbcbbcf4916bbc01fd1a5b5e90d P 1c50de89a5a845668696dcb28c3badd6 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 974b802141bd4f13beffef2aa03f88f9 in term 6.
I20260129 02:59:51.292196 22805 raft_consensus.cc:2804] T 3e2d15f8640b407195210c476e09df91 P 1c50de89a5a845668696dcb28c3badd6 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260129 02:59:51.253684 22667 raft_consensus.cc:2804] T a644c02b16af47679824ccd060ac10fd P 7a9daf5d42a84615a6383cfec9a10608 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260129 02:59:51.312215 20587 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c2633a6fc34c44579de348d208b0c01e" candidate_uuid: "974b802141bd4f13beffef2aa03f88f9" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "1c50de89a5a845668696dcb28c3badd6" is_pre_election: true
I20260129 02:59:51.312142 20556 leader_election.cc:304] T 1ce4ddf5b10e4ae8b193b13b2e5936ee P 1c50de89a5a845668696dcb28c3badd6 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1c50de89a5a845668696dcb28c3badd6, 974b802141bd4f13beffef2aa03f88f9; no voters: 197b880fa39d432a9b472845876bfaf0
I20260129 02:59:51.280292 22656 raft_consensus.cc:2804] T 505580c93c694bbb8ea96fdf9b17ab46 P 1c50de89a5a845668696dcb28c3badd6 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260129 02:59:51.316139 20510 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2ac659d732314e1288849f08fe5e074e" candidate_uuid: "7a9daf5d42a84615a6383cfec9a10608" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "974b802141bd4f13beffef2aa03f88f9" is_pre_election: true
I20260129 02:59:51.317624 20480 leader_election.cc:304] T cb4727386a3e48aca5d71cc6c611c197 P 974b802141bd4f13beffef2aa03f88f9 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 974b802141bd4f13beffef2aa03f88f9; no voters: 197b880fa39d432a9b472845876bfaf0, 1c50de89a5a845668696dcb28c3badd6
I20260129 02:59:51.337906 20507 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3ccac82a01a2445fa5b0ce6c0fcf2295" candidate_uuid: "7a9daf5d42a84615a6383cfec9a10608" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "974b802141bd4f13beffef2aa03f88f9" is_pre_election: true
W20260129 02:59:51.317873 20555 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260129 02:59:51.338846 21102 leader_election.cc:304] T 00e79d8f25a34676b1d866eb2068a2b2 P 7a9daf5d42a84615a6383cfec9a10608 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7a9daf5d42a84615a6383cfec9a10608, 974b802141bd4f13beffef2aa03f88f9; no voters:
I20260129 02:59:51.300109 22814 raft_consensus.cc:515] T 724a236524cb4e7ba3428880681d6cc2 P 1c50de89a5a845668696dcb28c3badd6 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:59:51.338848 20480 leader_election.cc:304] T cedce331e1584ac8b636c9b8a72b2ba0 P 974b802141bd4f13beffef2aa03f88f9 [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 974b802141bd4f13beffef2aa03f88f9; no voters: 197b880fa39d432a9b472845876bfaf0, 1c50de89a5a845668696dcb28c3badd6
I20260129 02:59:51.341377 20480 leader_election.cc:304] T 4f5a77e94fef4b16a2ba7b067c8b144a P 974b802141bd4f13beffef2aa03f88f9 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 974b802141bd4f13beffef2aa03f88f9; no voters: 1c50de89a5a845668696dcb28c3badd6, 7a9daf5d42a84615a6383cfec9a10608
I20260129 02:59:51.305805 20439 raft_consensus.cc:1275] T e33a417cf9c54cf5af73a4ee3aac4d3a P 197b880fa39d432a9b472845876bfaf0 [term 5 FOLLOWER]: Refusing update from remote peer 1c50de89a5a845668696dcb28c3badd6: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 5 index: 3. (index mismatch)
I20260129 02:59:51.327358 20511 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c934d31efd3f4e5195605ddc354d14e1" candidate_uuid: "7a9daf5d42a84615a6383cfec9a10608" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "974b802141bd4f13beffef2aa03f88f9" is_pre_election: true
I20260129 02:59:51.309962 22745 raft_consensus.cc:493] T 505580c93c694bbb8ea96fdf9b17ab46 P 974b802141bd4f13beffef2aa03f88f9 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260129 02:59:51.329850 20516 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "96609ac68bdf4993a12e1c23355ca013" candidate_uuid: "7a9daf5d42a84615a6383cfec9a10608" candidate_term: 5 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "974b802141bd4f13beffef2aa03f88f9"
I20260129 02:59:51.331660 20432 raft_consensus.cc:1240] T 3906b080a5b34a74b8b64128635e8997 P 197b880fa39d432a9b472845876bfaf0 [term 3 FOLLOWER]: Rejecting Update request from peer 1c50de89a5a845668696dcb28c3badd6 for earlier term 2. Current term is 3. Ops: []
I20260129 02:59:51.313102 22343 raft_consensus.cc:493] T 87d5fe48a44a408aaf03e894a67ecb2f P 7a9daf5d42a84615a6383cfec9a10608 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260129 02:59:51.344630 20432 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55676 (request call id 1273) took 1603 ms. Trace:
I20260129 02:59:51.319118 22403 raft_consensus.cc:493] T 0678b8b468bd43cdbc3e485900844821 P 7a9daf5d42a84615a6383cfec9a10608 [term 5 FOLLOWER]: Starting leader election (detected failure of leader 197b880fa39d432a9b472845876bfaf0)
I20260129 02:59:51.345031 20432 rpcz_store.cc:276] 0129 02:59:49.741012 (+ 0us) service_pool.cc:168] Inserting onto call queue
0129 02:59:49.741754 (+ 742us) service_pool.cc:225] Handling call
0129 02:59:51.331263 (+1589509us) spinlock_profiling.cc:243] Waited 3.85 ms on lock 0x7b5800000f98. stack: 00007f8fe044e373 00007f8fe044e270 00007f8fdfda0574 00007f8fe9a5c6b4 00007f8fe9a5c649 00007f8fe02c08b1 00007f8fe0442db1 00007f8fe0442c62 00007f8fe92eddd9 00007f8fe846d260 00007f8fe8535dbb 00007f8fe8535e41 00007f8fe84f7af0 00007f8fe84e2459 00007f8fe84e1f02 00007f8fe1a6ed34
0129 02:59:51.331460 (+ 197us) raft_consensus.cc:1471] Updating replica for 0 ops
0129 02:59:51.344366 (+ 12906us) raft_consensus.cc:1697] Filling consensus response to leader.
0129 02:59:51.344568 (+ 202us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":8851200}
I20260129 02:59:51.345956 20583 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e7d83d4ccf684e76acb29a0dc844e1b8" candidate_uuid: "974b802141bd4f13beffef2aa03f88f9" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "1c50de89a5a845668696dcb28c3badd6" is_pre_election: true
I20260129 02:59:51.310777 20436 raft_consensus.cc:2468] T 42d27a28d89b409e8807ec0efd97af69 P 197b880fa39d432a9b472845876bfaf0 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1c50de89a5a845668696dcb28c3badd6 in term 6.
I20260129 02:59:51.347105 20583 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fcddab6bca684adf8ea032289a00f90e" candidate_uuid: "974b802141bd4f13beffef2aa03f88f9" candidate_term: 8 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "1c50de89a5a845668696dcb28c3badd6"
I20260129 02:59:51.347254 20556 leader_election.cc:304] T 76f713c3331540d2a5b76164103e9eb1 P 1c50de89a5a845668696dcb28c3badd6 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1c50de89a5a845668696dcb28c3badd6, 974b802141bd4f13beffef2aa03f88f9; no voters: 197b880fa39d432a9b472845876bfaf0
I20260129 02:59:51.327666 22344 raft_consensus.cc:697] T add8d4923d1a489e86882c3a25692b95 P 1c50de89a5a845668696dcb28c3badd6 [term 3 LEADER]: Becoming Leader. State: Replica: 1c50de89a5a845668696dcb28c3badd6, State: Running, Role: LEADER
I20260129 02:59:51.362097 20587 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "78aa39096868415d9afc2ffcb248e198" candidate_uuid: "7a9daf5d42a84615a6383cfec9a10608" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "1c50de89a5a845668696dcb28c3badd6"
I20260129 02:59:51.363440 20587 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:48526 (request call id 254) took 1482 ms. Trace:
I20260129 02:59:51.362480 22640 raft_consensus.cc:2804] T 1ce4ddf5b10e4ae8b193b13b2e5936ee P 1c50de89a5a845668696dcb28c3badd6 [term 6 FOLLOWER]: Leader election won for term 6
I20260129 02:59:51.346215 22184 raft_consensus.cc:2749] T cedce331e1584ac8b636c9b8a72b2ba0 P 974b802141bd4f13beffef2aa03f88f9 [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20260129 02:59:51.342407 22667 raft_consensus.cc:493] T a644c02b16af47679824ccd060ac10fd P 7a9daf5d42a84615a6383cfec9a10608 [term 3 FOLLOWER]: Starting leader election (detected failure of leader 1c50de89a5a845668696dcb28c3badd6)
I20260129 02:59:51.362476 22815 leader_election.cc:290] T b189f08e16a74a11980cc55d99359a0a P 974b802141bd4f13beffef2aa03f88f9 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 197b880fa39d432a9b472845876bfaf0 (127.18.211.1:42027), 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3:37735)
I20260129 02:59:51.368628 22667 raft_consensus.cc:3060] T a644c02b16af47679824ccd060ac10fd P 7a9daf5d42a84615a6383cfec9a10608 [term 3 FOLLOWER]: Advancing to term 4
I20260129 02:59:51.358724 20433 raft_consensus.cc:1240] T 42d27a28d89b409e8807ec0efd97af69 P 197b880fa39d432a9b472845876bfaf0 [term 6 FOLLOWER]: Rejecting Update request from peer 7a9daf5d42a84615a6383cfec9a10608 for earlier term 5. Current term is 6. Ops: [5.7-5.7]
I20260129 02:59:51.359294 20480 leader_election.cc:304] T 45cc5f3d2688486d9b6dbe7e666dedde P 974b802141bd4f13beffef2aa03f88f9 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 974b802141bd4f13beffef2aa03f88f9; no voters: 1c50de89a5a845668696dcb28c3badd6, 7a9daf5d42a84615a6383cfec9a10608
I20260129 02:59:51.295674 22425 consensus_queue.cc:260] T 0f20af7fbe5d4db89a6f17948d98912c P 1c50de89a5a845668696dcb28c3badd6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:59:51.370817 22425 raft_consensus.cc:3060] T 0f20af7fbe5d4db89a6f17948d98912c P 1c50de89a5a845668696dcb28c3badd6 [term 4 FOLLOWER]: Advancing to term 5
I20260129 02:59:51.336787 22751 raft_consensus.cc:697] T 75b0b6e82ddc4f92b1fe1d94f9a0b73b P 1c50de89a5a845668696dcb28c3badd6 [term 3 LEADER]: Becoming Leader. State: Replica: 1c50de89a5a845668696dcb28c3badd6, State: Running, Role: LEADER
I20260129 02:59:51.372047 22751 consensus_queue.cc:237] T 75b0b6e82ddc4f92b1fe1d94f9a0b73b P 1c50de89a5a845668696dcb28c3badd6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } }
I20260129 02:59:51.372545 22638 raft_consensus.cc:515] T a1a39dcd328e4268ae41b988c66b4a75 P 1c50de89a5a845668696dcb28c3badd6 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } } peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } }
I20260129 02:59:51.373692 20588 raft_consensus.cc:2368] T a1a39dcd328e4268ae41b988c66b4a75 P 1c50de89a5a845668696dcb28c3badd6 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 974b802141bd4f13beffef2aa03f88f9 for earlier term 3. Current term is 4.
I20260129 02:59:51.375145 20588 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "505580c93c694bbb8ea96fdf9b17ab46" candidate_uuid: "974b802141bd4f13beffef2aa03f88f9" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "1c50de89a5a845668696dcb28c3badd6" is_pre_election: true
I20260129 02:59:51.376363 22344 consensus_queue.cc:237] T add8d4923d1a489e86882c3a25692b95 P 1c50de89a5a845668696dcb28c3badd6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } }
I20260129 02:59:51.300105 20589 rpcz_store.cc:276] 0129 02:59:49.917984 (+ 0us) service_pool.cc:168] Inserting onto call queue
0129 02:59:51.280561 (+1362577us) service_pool.cc:225] Handling call
0129 02:59:51.299305 (+ 18744us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260129 02:59:51.345911 22729 raft_consensus.cc:2804] T 00e79d8f25a34676b1d866eb2068a2b2 P 7a9daf5d42a84615a6383cfec9a10608 [term 6 FOLLOWER]: Leader election won for term 6
I20260129 02:59:51.371016 20480 leader_election.cc:304] T 4408006eb5174de1896b037cf3f18457 P 974b802141bd4f13beffef2aa03f88f9 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1c50de89a5a845668696dcb28c3badd6, 974b802141bd4f13beffef2aa03f88f9; no voters: 197b880fa39d432a9b472845876bfaf0
I20260129 02:59:51.347703 20516 raft_consensus.cc:2393] T 96609ac68bdf4993a12e1c23355ca013 P 974b802141bd4f13beffef2aa03f88f9 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 7a9daf5d42a84615a6383cfec9a10608 in current term 5: Already voted for candidate 974b802141bd4f13beffef2aa03f88f9 in this term.
I20260129 02:59:51.336797 22348 raft_consensus.cc:493] T 4408006eb5174de1896b037cf3f18457 P 1c50de89a5a845668696dcb28c3badd6 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260129 02:59:51.340315 22656 raft_consensus.cc:493] T 505580c93c694bbb8ea96fdf9b17ab46 P 1c50de89a5a845668696dcb28c3badd6 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260129 02:59:51.346098 20507 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57584 (request call id 1299) took 1839 ms. Trace:
I20260129 02:59:51.385207 20507 rpcz_store.cc:276] 0129 02:59:49.502608 (+ 0us) service_pool.cc:168] Inserting onto call queue
0129 02:59:51.341145 (+1838537us) service_pool.cc:225] Handling call
0129 02:59:51.341364 (+ 219us) raft_consensus.cc:1471] Updating replica for 0 ops
0129 02:59:51.341492 (+ 128us) raft_consensus.cc:1534] Early marking committed up to index 0
0129 02:59:51.341507 (+ 15us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0129 02:59:51.341533 (+ 26us) raft_consensus.cc:1645] Marking committed up to 2
0129 02:59:51.341553 (+ 20us) raft_consensus.cc:1697] Filling consensus response to leader.
0129 02:59:51.341586 (+ 33us) raft_consensus.cc:1691] UpdateReplicas() finished
0129 02:59:51.341690 (+ 104us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260129 02:59:51.385912 20516 raft_consensus.cc:1275] T e33a417cf9c54cf5af73a4ee3aac4d3a P 974b802141bd4f13beffef2aa03f88f9 [term 5 FOLLOWER]: Refusing update from remote peer 1c50de89a5a845668696dcb28c3badd6: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 5 index: 3. (term mismatch)
I20260129 02:59:51.386294 20516 pending_rounds.cc:85] T e33a417cf9c54cf5af73a4ee3aac4d3a P 974b802141bd4f13beffef2aa03f88f9: Aborting all ops after (but not including) 2
I20260129 02:59:51.380668 20587 rpcz_store.cc:276] 0129 02:59:49.880885 (+ 0us) service_pool.cc:168] Inserting onto call queue
0129 02:59:51.337820 (+1456935us) service_pool.cc:225] Handling call
0129 02:59:51.363376 (+ 25556us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260129 02:59:51.339347 22493 raft_consensus.cc:2749] T cb4727386a3e48aca5d71cc6c611c197 P 974b802141bd4f13beffef2aa03f88f9 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260129 02:59:51.386401 20480 leader_election.cc:304] T 96609ac68bdf4993a12e1c23355ca013 P 974b802141bd4f13beffef2aa03f88f9 [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 197b880fa39d432a9b472845876bfaf0, 974b802141bd4f13beffef2aa03f88f9; no voters: 1c50de89a5a845668696dcb28c3badd6, 7a9daf5d42a84615a6383cfec9a10608
I20260129 02:59:51.364116 22641 leader_election.cc:290] T e7d83d4ccf684e76acb29a0dc844e1b8 P 197b880fa39d432a9b472845876bfaf0 [CANDIDATE]: Term 5 election: Requested vote from peers 974b802141bd4f13beffef2aa03f88f9 (127.18.211.2:34517), 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3:37735)
I20260129 02:59:51.388944 20432 raft_consensus.cc:1275] T 3ccac82a01a2445fa5b0ce6c0fcf2295 P 197b880fa39d432a9b472845876bfaf0 [term 3 FOLLOWER]: Refusing update from remote peer 974b802141bd4f13beffef2aa03f88f9: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 3 index: 5. (index mismatch)
I20260129 02:59:51.389436 20480 leader_election.cc:304] T a5a05fbcbbcf4916bbc01fd1a5b5e90d P 974b802141bd4f13beffef2aa03f88f9 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1c50de89a5a845668696dcb28c3badd6, 974b802141bd4f13beffef2aa03f88f9; no voters:
I20260129 02:59:51.386549 20516 pending_rounds.cc:107] T e33a417cf9c54cf5af73a4ee3aac4d3a P 974b802141bd4f13beffef2aa03f88f9: Aborting uncommitted NO_OP operation due to leader change: 4.3
I20260129 02:59:51.337124 22805 raft_consensus.cc:493] T 3e2d15f8640b407195210c476e09df91 P 1c50de89a5a845668696dcb28c3badd6 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260129 02:59:51.391517 22805 raft_consensus.cc:3060] T 3e2d15f8640b407195210c476e09df91 P 1c50de89a5a845668696dcb28c3badd6 [term 2 FOLLOWER]: Advancing to term 3
I20260129 02:59:51.387383 21100 leader_election.cc:304] T 78aa39096868415d9afc2ffcb248e198 P 7a9daf5d42a84615a6383cfec9a10608 [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7a9daf5d42a84615a6383cfec9a10608; no voters: 197b880fa39d432a9b472845876bfaf0, 1c50de89a5a845668696dcb28c3badd6
I20260129 02:59:51.390862 20516 raft_consensus.cc:2889] T e33a417cf9c54cf5af73a4ee3aac4d3a P 974b802141bd4f13beffef2aa03f88f9 [term 5 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260129 02:59:51.345605 22745 raft_consensus.cc:3060] T 505580c93c694bbb8ea96fdf9b17ab46 P 974b802141bd4f13beffef2aa03f88f9 [term 4 FOLLOWER]: Advancing to term 5
I20260129 02:59:51.343510 22400 leader_election.cc:290] T 3e2d15f8640b407195210c476e09df91 P 974b802141bd4f13beffef2aa03f88f9 [CANDIDATE]: Term 2 election: Requested vote from peers 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3:37735), 7a9daf5d42a84615a6383cfec9a10608 (127.18.211.4:40439)
I20260129 02:59:51.343019 22581 raft_consensus.cc:493] T cb0b3f259ebc4310927338a8740e87b8 P 1c50de89a5a845668696dcb28c3badd6 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 7a9daf5d42a84615a6383cfec9a10608)
I20260129 02:59:51.392198 20589 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3e2d15f8640b407195210c476e09df91" candidate_uuid: "974b802141bd4f13beffef2aa03f88f9" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "1c50de89a5a845668696dcb28c3badd6"
I20260129 02:59:51.345496 22343 raft_consensus.cc:3060] T 87d5fe48a44a408aaf03e894a67ecb2f P 7a9daf5d42a84615a6383cfec9a10608 [term 5 FOLLOWER]: Advancing to term 6
I20260129 02:59:51.393478 20480 leader_election.cc:304] T e7d83d4ccf684e76acb29a0dc844e1b8 P 974b802141bd4f13beffef2aa03f88f9 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 974b802141bd4f13beffef2aa03f88f9; no voters: 197b880fa39d432a9b472845876bfaf0, 1c50de89a5a845668696dcb28c3badd6
I20260129 02:59:51.394136 21135 raft_consensus.cc:1275] T febffe4a8b394368b69b1d164308fb9a P 7a9daf5d42a84615a6383cfec9a10608 [term 3 FOLLOWER]: Refusing update from remote peer 197b880fa39d432a9b472845876bfaf0: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 3 index: 5. (term mismatch)
I20260129 02:59:51.394556 20507 raft_consensus.cc:1240] T 1ce4ddf5b10e4ae8b193b13b2e5936ee P 974b802141bd4f13beffef2aa03f88f9 [term 6 FOLLOWER]: Rejecting Update request from peer 197b880fa39d432a9b472845876bfaf0 for earlier term 5. Current term is 6. Ops: []
I20260129 02:59:51.403101 21135 pending_rounds.cc:85] T febffe4a8b394368b69b1d164308fb9a P 7a9daf5d42a84615a6383cfec9a10608: Aborting all ops after (but not including) 4
I20260129 02:59:51.403461 21135 pending_rounds.cc:107] T febffe4a8b394368b69b1d164308fb9a P 7a9daf5d42a84615a6383cfec9a10608: Aborting uncommitted NO_OP operation due to leader change: 2.5
I20260129 02:59:51.404249 20507 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57600 (request call id 1759) took 1763 ms. Trace:
I20260129 02:59:51.404852 20556 leader_election.cc:304] T a8479d1fe0bd41e5946186e3a6ca8c7a P 1c50de89a5a845668696dcb28c3badd6 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1c50de89a5a845668696dcb28c3badd6; no voters: 197b880fa39d432a9b472845876bfaf0, 974b802141bd4f13beffef2aa03f88f9
I20260129 02:59:51.361997 22763 raft_consensus.cc:2749] T 4f5a77e94fef4b16a2ba7b067c8b144a P 974b802141bd4f13beffef2aa03f88f9 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260129 02:59:51.387888 22638 leader_election.cc:290] T a1a39dcd328e4268ae41b988c66b4a75 P 1c50de89a5a845668696dcb28c3badd6 [CANDIDATE]: Term 4 election: Requested vote from peers 974b802141bd4f13beffef2aa03f88f9 (127.18.211.2:34517), 197b880fa39d432a9b472845876bfaf0 (127.18.211.1:42027)
I20260129 02:59:51.397224 22184 raft_consensus.cc:2804] T 4408006eb5174de1896b037cf3f18457 P 974b802141bd4f13beffef2aa03f88f9 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260129 02:59:51.387741 22348 raft_consensus.cc:3060] T 4408006eb5174de1896b037cf3f18457 P 1c50de89a5a845668696dcb28c3badd6 [term 2 FOLLOWER]: Advancing to term 3
I20260129 02:59:51.402634 20587 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "913fc7d751ed4b7388d889aac30a5f9b" candidate_uuid: "974b802141bd4f13beffef2aa03f88f9" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1c50de89a5a845668696dcb28c3badd6" is_pre_election: true
I20260129 02:59:51.404609 20507 rpcz_store.cc:276] 0129 02:59:49.640819 (+ 0us) service_pool.cc:168] Inserting onto call queue
0129 02:59:51.386167 (+1745348us) service_pool.cc:225] Handling call
0129 02:59:51.386441 (+ 274us) raft_consensus.cc:1471] Updating replica for 0 ops
0129 02:59:51.404001 (+ 17560us) raft_consensus.cc:1697] Filling consensus response to leader.
0129 02:59:51.404191 (+ 190us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260129 02:59:51.388163 22815 raft_consensus.cc:2749] T 45cc5f3d2688486d9b6dbe7e666dedde P 974b802141bd4f13beffef2aa03f88f9 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20260129 02:59:51.347810 22403 raft_consensus.cc:3060] T 0678b8b468bd43cdbc3e485900844821 P 7a9daf5d42a84615a6383cfec9a10608 [term 5 FOLLOWER]: Advancing to term 6
I20260129 02:59:51.389963 22656 raft_consensus.cc:3060] T 505580c93c694bbb8ea96fdf9b17ab46 P 1c50de89a5a845668696dcb28c3badd6 [term 4 FOLLOWER]: Advancing to term 5
I20260129 02:59:51.403715 21135 raft_consensus.cc:2889] T febffe4a8b394368b69b1d164308fb9a P 7a9daf5d42a84615a6383cfec9a10608 [term 3 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260129 02:59:51.407585 20480 leader_election.cc:400] T a1a39dcd328e4268ae41b988c66b4a75 P 974b802141bd4f13beffef2aa03f88f9 [CANDIDATE]: Term 3 election: Vote denied by peer 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3:37735) with higher term. Message: Invalid argument: T a1a39dcd328e4268ae41b988c66b4a75 P 1c50de89a5a845668696dcb28c3badd6 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 974b802141bd4f13beffef2aa03f88f9 for earlier term 3. Current term is 4.
I20260129 02:59:51.420274 20480 leader_election.cc:403] T a1a39dcd328e4268ae41b988c66b4a75 P 974b802141bd4f13beffef2aa03f88f9 [CANDIDATE]: Term 3 election: Cancelling election due to peer responding with higher term
I20260129 02:59:51.391683 22640 raft_consensus.cc:697] T 1ce4ddf5b10e4ae8b193b13b2e5936ee P 1c50de89a5a845668696dcb28c3badd6 [term 6 LEADER]: Becoming Leader. State: Replica: 1c50de89a5a845668696dcb28c3badd6, State: Running, Role: LEADER
I20260129 02:59:51.394557 20511 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57584 (request call id 1301) took 1891 ms. Trace:
I20260129 02:59:51.401279 20516 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57584 (request call id 1303) took 1897 ms. Trace:
I20260129 02:59:51.398521 22719 raft_consensus.cc:2804] T a5a05fbcbbcf4916bbc01fd1a5b5e90d P 974b802141bd4f13beffef2aa03f88f9 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260129 02:59:51.423794 22719 raft_consensus.cc:493] T a5a05fbcbbcf4916bbc01fd1a5b5e90d P 974b802141bd4f13beffef2aa03f88f9 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260129 02:59:51.424297 22719 raft_consensus.cc:3060] T a5a05fbcbbcf4916bbc01fd1a5b5e90d P 974b802141bd4f13beffef2aa03f88f9 [term 6 FOLLOWER]: Advancing to term 7
I20260129 02:59:51.424643 21129 raft_consensus.cc:1275] T a609ac02fae84725ab743f95f51d33ce P 7a9daf5d42a84615a6383cfec9a10608 [term 6 FOLLOWER]: Refusing update from remote peer 974b802141bd4f13beffef2aa03f88f9: Log matching property violated. Preceding OpId in replica: term: 4 index: 6. Preceding OpId from leader: term: 6 index: 7. (index mismatch)
I20260129 02:59:51.426674 20508 raft_consensus.cc:1240] T 76f713c3331540d2a5b76164103e9eb1 P 974b802141bd4f13beffef2aa03f88f9 [term 4 FOLLOWER]: Rejecting Update request from peer 197b880fa39d432a9b472845876bfaf0 for earlier term 3. Current term is 4. Ops: []
I20260129 02:59:51.427284 20508 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57600 (request call id 1774) took 1705 ms. Trace:
I20260129 02:59:51.396962 22694 raft_consensus.cc:2749] T 96609ac68bdf4993a12e1c23355ca013 P 974b802141bd4f13beffef2aa03f88f9 [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20260129 02:59:51.427577 20508 rpcz_store.cc:276] 0129 02:59:49.722356 (+ 0us) service_pool.cc:168] Inserting onto call queue
0129 02:59:51.426180 (+1703824us) service_pool.cc:225] Handling call
0129 02:59:51.426494 (+ 314us) raft_consensus.cc:1471] Updating replica for 0 ops
0129 02:59:51.427057 (+ 563us) raft_consensus.cc:1697] Filling consensus response to leader.
0129 02:59:51.427231 (+ 174us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260129 02:59:51.428633 22820 leader_election.cc:290] T 75f45538dc704d9a8ec4ff8ad9f3ea5b P 974b802141bd4f13beffef2aa03f88f9 [CANDIDATE]: Term 3 election: Requested vote from peers 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3:37735), 7a9daf5d42a84615a6383cfec9a10608 (127.18.211.4:40439)
I20260129 02:59:51.414865 22664 raft_consensus.cc:2749] T a8479d1fe0bd41e5946186e3a6ca8c7a P 1c50de89a5a845668696dcb28c3badd6 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20260129 02:59:51.423885 22400 raft_consensus.cc:3060] T a1a39dcd328e4268ae41b988c66b4a75 P 974b802141bd4f13beffef2aa03f88f9 [term 3 FOLLOWER]: Advancing to term 4
I20260129 02:59:51.432997 22807 raft_consensus.cc:493] T 0678b8b468bd43cdbc3e485900844821 P 974b802141bd4f13beffef2aa03f88f9 [term 5 FOLLOWER]: Starting leader election (detected failure of leader 197b880fa39d432a9b472845876bfaf0)
I20260129 02:59:51.433496 22807 raft_consensus.cc:3060] T 0678b8b468bd43cdbc3e485900844821 P 974b802141bd4f13beffef2aa03f88f9 [term 5 FOLLOWER]: Advancing to term 6
I20260129 02:59:51.433224 22684 raft_consensus.cc:515] T cedce331e1584ac8b636c9b8a72b2ba0 P 197b880fa39d432a9b472845876bfaf0 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:59:51.434581 20437 raft_consensus.cc:2393] T cedce331e1584ac8b636c9b8a72b2ba0 P 197b880fa39d432a9b472845876bfaf0 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1c50de89a5a845668696dcb28c3badd6 in current term 6: Already voted for candidate 197b880fa39d432a9b472845876bfaf0 in this term.
I20260129 02:59:51.435146 20581 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:35754 (request call id 1647) took 1976 ms. Trace:
I20260129 02:59:51.435570 20581 rpcz_store.cc:276] 0129 02:59:49.458856 (+ 0us) service_pool.cc:168] Inserting onto call queue
0129 02:59:51.064106 (+1605250us) service_pool.cc:225] Handling call
0129 02:59:51.434952 (+370846us) spinlock_profiling.cc:243] Waited 28.8 ms on lock 0x7b5400108954. stack: 00007f8fe044e373 00007f8fe044e270 00007f8fdfda0574 00007f8fe9a5c6b4 00007f8fe9a5c649 00007f8fe34975a1 00007f8fe3531f7a 00007f8fe84e3647 00007f8fe1a73834 00007f8fe1a737c2 00007f8fe1a736f1 00007f8fe1a7366c 00007f8fe1a72922 00007f8fe186d34c 00007f8fe186c786 00007f8fe186c12f
0129 02:59:51.435083 (+ 131us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":66245504}
I20260129 02:59:51.436722 20581 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4f8f3a38f6e24ba4857875181f447442" candidate_uuid: "974b802141bd4f13beffef2aa03f88f9" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "1c50de89a5a845668696dcb28c3badd6"
I20260129 02:59:51.437903 20581 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "96609ac68bdf4993a12e1c23355ca013" candidate_uuid: "197b880fa39d432a9b472845876bfaf0" candidate_term: 5 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "1c50de89a5a845668696dcb28c3badd6" is_pre_election: true
I20260129 02:59:51.438645 20581 raft_consensus.cc:2393] T 96609ac68bdf4993a12e1c23355ca013 P 1c50de89a5a845668696dcb28c3badd6 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 197b880fa39d432a9b472845876bfaf0 in current term 5: Already voted for candidate 7a9daf5d42a84615a6383cfec9a10608 in this term.
I20260129 02:59:51.438988 20581 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:35754 (request call id 1642) took 1981 ms. Trace:
I20260129 02:59:51.401959 22729 raft_consensus.cc:697] T 00e79d8f25a34676b1d866eb2068a2b2 P 7a9daf5d42a84615a6383cfec9a10608 [term 6 LEADER]: Becoming Leader. State: Replica: 7a9daf5d42a84615a6383cfec9a10608, State: Running, Role: LEADER
I20260129 02:59:51.439270 20581 rpcz_store.cc:276] 0129 02:59:49.457294 (+ 0us) service_pool.cc:168] Inserting onto call queue
0129 02:59:51.437639 (+1980345us) service_pool.cc:225] Handling call
0129 02:59:51.438943 (+ 1304us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260129 02:59:51.428501 22640 consensus_queue.cc:237] T 1ce4ddf5b10e4ae8b193b13b2e5936ee P 1c50de89a5a845668696dcb28c3badd6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 4.3, Last appended by leader: 3, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:59:51.416989 20587 raft_consensus.cc:2410] T 913fc7d751ed4b7388d889aac30a5f9b P 1c50de89a5a845668696dcb28c3badd6 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 974b802141bd4f13beffef2aa03f88f9 for term 4 because replica has last-logged OpId of term: 2 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 1.
I20260129 02:59:51.418413 20584 raft_consensus.cc:2468] T 0f20af7fbe5d4db89a6f17948d98912c P 1c50de89a5a845668696dcb28c3badd6 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 974b802141bd4f13beffef2aa03f88f9 in term 5.
I20260129 02:59:51.398996 22728 raft_consensus.cc:2804] T 76f713c3331540d2a5b76164103e9eb1 P 1c50de89a5a845668696dcb28c3badd6 [term 4 FOLLOWER]: Leader pre-election won for term 5
W20260129 02:59:51.436857 22822 log.cc:927] Time spent T 2ac659d732314e1288849f08fe5e074e P 974b802141bd4f13beffef2aa03f88f9: Append to log took a long time: real 0.711s user 0.002s sys 0.000s
W20260129 02:59:51.488618 20582 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:35754 (request call id 1661) took 1985 ms (client timeout 2545 ms). Trace:
W20260129 02:59:51.498135 20582 rpcz_store.cc:269] 0129 02:59:49.503607 (+ 0us) service_pool.cc:168] Inserting onto call queue
0129 02:59:50.618700 (+1115093us) service_pool.cc:225] Handling call
0129 02:59:51.488398 (+869698us) spinlock_profiling.cc:243] Waited 19.3 ms on lock 0x7b54003b0e54. stack: 00007f8fe044e373 00007f8fe044e270 00007f8fdfda0574 00007f8fe9a5c6b4 00007f8fe9a5c649 00007f8fe34975a1 00007f8fe3531f7a 00007f8fe84e3647 00007f8fe1a73834 00007f8fe1a737c2 00007f8fe1a736f1 00007f8fe1a7366c 00007f8fe1a72922 00007f8fe186d34c 00007f8fe186c786 00007f8fe186c12f
0129 02:59:51.488537 (+ 139us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":44408192}
I20260129 02:59:51.499771 20582 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b189f08e16a74a11980cc55d99359a0a" candidate_uuid: "974b802141bd4f13beffef2aa03f88f9" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "1c50de89a5a845668696dcb28c3badd6" is_pre_election: true
I20260129 02:59:51.501681 20582 raft_consensus.cc:2468] T b189f08e16a74a11980cc55d99359a0a P 1c50de89a5a845668696dcb28c3badd6 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 974b802141bd4f13beffef2aa03f88f9 in term 6.
I20260129 02:59:51.416548 20507 raft_consensus.cc:1275] T 0f20af7fbe5d4db89a6f17948d98912c P 974b802141bd4f13beffef2aa03f88f9 [term 5 FOLLOWER]: Refusing update from remote peer 197b880fa39d432a9b472845876bfaf0: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 5 index: 4. (index mismatch)
I20260129 02:59:51.423643 20516 rpcz_store.cc:276] 0129 02:59:49.503708 (+ 0us) service_pool.cc:168] Inserting onto call queue
0129 02:59:51.385311 (+1881603us) service_pool.cc:225] Handling call
0129 02:59:51.385592 (+ 281us) raft_consensus.cc:1471] Updating replica for 0 ops
0129 02:59:51.401018 (+ 15426us) raft_consensus.cc:1697] Filling consensus response to leader.
0129 02:59:51.401226 (+ 208us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260129 02:59:51.422569 20511 rpcz_store.cc:276] 0129 02:59:49.503185 (+ 0us) service_pool.cc:168] Inserting onto call queue
0129 02:59:51.346727 (+1843542us) service_pool.cc:225] Handling call
0129 02:59:51.390439 (+ 43712us) raft_consensus.cc:1471] Updating replica for 0 ops
0129 02:59:51.394100 (+ 3661us) raft_consensus.cc:1534] Early marking committed up to index 0
0129 02:59:51.394128 (+ 28us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0129 02:59:51.394170 (+ 42us) raft_consensus.cc:1645] Marking committed up to 3
0129 02:59:51.394205 (+ 35us) raft_consensus.cc:1697] Filling consensus response to leader.
0129 02:59:51.394392 (+ 187us) raft_consensus.cc:1691] UpdateReplicas() finished
0129 02:59:51.394504 (+ 112us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":3280,"raft.queue_time_us":5195,"raft.run_cpu_time_us":131,"raft.run_wall_time_us":131}
I20260129 02:59:51.451736 20480 leader_election.cc:304] T 4f8f3a38f6e24ba4857875181f447442 P 974b802141bd4f13beffef2aa03f88f9 [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 974b802141bd4f13beffef2aa03f88f9; no voters: 197b880fa39d432a9b472845876bfaf0, 1c50de89a5a845668696dcb28c3badd6
I20260129 02:59:51.503203 20507 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57600 (request call id 1771) took 1781 ms. Trace:
W20260129 02:59:51.503964 20480 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.211.3:37735, user_credentials={real_user=slave}} blocked reactor thread for 52323.1us
I20260129 02:59:51.503744 20507 rpcz_store.cc:276] 0129 02:59:49.721418 (+ 0us) service_pool.cc:168] Inserting onto call queue
0129 02:59:51.415729 (+1694311us) service_pool.cc:225] Handling call
0129 02:59:51.416146 (+ 417us) raft_consensus.cc:1471] Updating replica for 0 ops
0129 02:59:51.502873 (+ 86727us) raft_consensus.cc:1697] Filling consensus response to leader.
0129 02:59:51.503135 (+ 262us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260129 02:59:51.412454 22687 raft_consensus.cc:2749] T e7d83d4ccf684e76acb29a0dc844e1b8 P 974b802141bd4f13beffef2aa03f88f9 [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260129 02:59:51.445369 21102 leader_election.cc:304] T 96609ac68bdf4993a12e1c23355ca013 P 7a9daf5d42a84615a6383cfec9a10608 [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 1c50de89a5a845668696dcb28c3badd6, 7a9daf5d42a84615a6383cfec9a10608; no voters: 197b880fa39d432a9b472845876bfaf0, 974b802141bd4f13beffef2aa03f88f9
I20260129 02:59:51.507525 20480 leader_election.cc:304] T 913fc7d751ed4b7388d889aac30a5f9b P 974b802141bd4f13beffef2aa03f88f9 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 974b802141bd4f13beffef2aa03f88f9; no voters: 197b880fa39d432a9b472845876bfaf0, 1c50de89a5a845668696dcb28c3badd6
W20260129 02:59:51.466275 20556 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.211.2:34517, user_credentials={real_user=slave}} blocked reactor thread for 61500.2us
I20260129 02:59:51.413944 22184 raft_consensus.cc:493] T 4408006eb5174de1896b037cf3f18457 P 974b802141bd4f13beffef2aa03f88f9 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260129 02:59:51.512387 20587 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "75f45538dc704d9a8ec4ff8ad9f3ea5b" candidate_uuid: "974b802141bd4f13beffef2aa03f88f9" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "1c50de89a5a845668696dcb28c3badd6"
I20260129 02:59:51.450909 20585 raft_consensus.cc:2468] T 3906b080a5b34a74b8b64128635e8997 P 1c50de89a5a845668696dcb28c3badd6 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 197b880fa39d432a9b472845876bfaf0 in term 3.
I20260129 02:59:51.509323 20581 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "dc2f4cb752cf4d59b5f6bc689f5d8104" candidate_uuid: "7a9daf5d42a84615a6383cfec9a10608" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "1c50de89a5a845668696dcb28c3badd6" is_pre_election: true
W20260129 02:59:51.509639 20555 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260129 02:59:51.510032 20480 leader_election.cc:304] T b189f08e16a74a11980cc55d99359a0a P 974b802141bd4f13beffef2aa03f88f9 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1c50de89a5a845668696dcb28c3badd6, 974b802141bd4f13beffef2aa03f88f9; no voters:
I20260129 02:59:51.420758 22672 raft_consensus.cc:2749] T 78aa39096868415d9afc2ffcb248e198 P 7a9daf5d42a84615a6383cfec9a10608 [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20260129 02:59:51.514709 22672 raft_consensus.cc:2749] T 96609ac68bdf4993a12e1c23355ca013 P 7a9daf5d42a84615a6383cfec9a10608 [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20260129 02:59:51.448122 22742 raft_consensus.cc:493] T febffe4a8b394368b69b1d164308fb9a P 7a9daf5d42a84615a6383cfec9a10608 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260129 02:59:51.443856 22786 leader_election.cc:290] T e4d18c35b0a7493e85d0a4b92956a48f P 1c50de89a5a845668696dcb28c3badd6 [CANDIDATE]: Term 5 election: Requested vote from peers 197b880fa39d432a9b472845876bfaf0 (127.18.211.1:42027), 974b802141bd4f13beffef2aa03f88f9 (127.18.211.2:34517)
I20260129 02:59:51.516331 20433 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:48356 (request call id 310) took 1027 ms. Trace:
I20260129 02:59:51.522140 20433 rpcz_store.cc:276] 0129 02:59:50.488835 (+ 0us) service_pool.cc:168] Inserting onto call queue
0129 02:59:51.212309 (+723474us) service_pool.cc:225] Handling call
0129 02:59:51.358553 (+146244us) raft_consensus.cc:1471] Updating replica for 1 ops
0129 02:59:51.369553 (+ 11000us) raft_consensus.cc:1697] Filling consensus response to leader.
0129 02:59:51.516156 (+146603us) spinlock_profiling.cc:243] Waited 146 ms on lock 0x7b5400230850. stack: 00007f8fe044e373 00007f8fe044e270 00007f8fdfda0574 00007f8fe9a5c6b4 00007f8fe9a5c649 00007f8fe34975a1 00007f8fe352d52d 00007f8fe84e1f49 00007f8fe1a6ed34 00007f8fe1a6ecc2 00007f8fe1a6ebf1 00007f8fe1a6eb6c 00007f8fe1a6de22 00007f8fe186d34c 00007f8fe186c786 00007f8fe186c12f
0129 02:59:51.516277 (+ 121us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":335480704}
I20260129 02:59:51.513207 20587 raft_consensus.cc:3060] T 75f45538dc704d9a8ec4ff8ad9f3ea5b P 1c50de89a5a845668696dcb28c3badd6 [term 2 FOLLOWER]: Advancing to term 3
I20260129 02:59:51.489046 22728 raft_consensus.cc:493] T 76f713c3331540d2a5b76164103e9eb1 P 1c50de89a5a845668696dcb28c3badd6 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260129 02:59:51.529321 22728 raft_consensus.cc:3060] T 76f713c3331540d2a5b76164103e9eb1 P 1c50de89a5a845668696dcb28c3badd6 [term 4 FOLLOWER]: Advancing to term 5
I20260129 02:59:51.497323 22738 raft_consensus.cc:3055] T 368953c9102643b2a21f99af36b51905 P 974b802141bd4f13beffef2aa03f88f9 [term 5 LEADER]: Stepping down as leader of term 5
I20260129 02:59:51.405987 22581 raft_consensus.cc:515] T cb0b3f259ebc4310927338a8740e87b8 P 1c50de89a5a845668696dcb28c3badd6 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } }
W20260129 02:59:51.512794 21102 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.211.2:34517, user_credentials={real_user=slave}} blocked reactor thread for 67519.4us
I20260129 02:59:51.518190 20435 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:48356 (request call id 305) took 1955 ms. Trace:
I20260129 02:59:51.512362 22763 raft_consensus.cc:2749] T 913fc7d751ed4b7388d889aac30a5f9b P 974b802141bd4f13beffef2aa03f88f9 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260129 02:59:51.519521 20582 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "18f30ebaa4174fd28c958e72d0cc7e24" candidate_uuid: "7a9daf5d42a84615a6383cfec9a10608" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "1c50de89a5a845668696dcb28c3badd6"
I20260129 02:59:51.517710 20556 leader_election.cc:304] T a0fcc58d6fb6473cafbd9d34d5195269 P 1c50de89a5a845668696dcb28c3badd6 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1c50de89a5a845668696dcb28c3badd6; no voters: 197b880fa39d432a9b472845876bfaf0, 974b802141bd4f13beffef2aa03f88f9
I20260129 02:59:51.504290 22700 raft_consensus.cc:2749] T 4f8f3a38f6e24ba4857875181f447442 P 974b802141bd4f13beffef2aa03f88f9 [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20260129 02:59:51.408800 22320 consensus_queue.cc:1059] T 4f8f3a38f6e24ba4857875181f447442 P 1c50de89a5a845668696dcb28c3badd6 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 }, Status: INVALID_TERM, Last received: 4.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.100s
I20260129 02:59:51.521883 22820 raft_consensus.cc:2804] T b189f08e16a74a11980cc55d99359a0a P 974b802141bd4f13beffef2aa03f88f9 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260129 02:59:51.524353 22742 raft_consensus.cc:515] T febffe4a8b394368b69b1d164308fb9a P 7a9daf5d42a84615a6383cfec9a10608 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } }
I20260129 02:59:51.515820 22184 raft_consensus.cc:3060] T 4408006eb5174de1896b037cf3f18457 P 974b802141bd4f13beffef2aa03f88f9 [term 2 FOLLOWER]: Advancing to term 3
I20260129 02:59:51.526974 20507 raft_consensus.cc:1275] T e7d83d4ccf684e76acb29a0dc844e1b8 P 974b802141bd4f13beffef2aa03f88f9 [term 4 FOLLOWER]: Refusing update from remote peer 1c50de89a5a845668696dcb28c3badd6: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20260129 02:59:51.518759 20480 leader_election.cc:304] T c934d31efd3f4e5195605ddc354d14e1 P 974b802141bd4f13beffef2aa03f88f9 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 974b802141bd4f13beffef2aa03f88f9; no voters: 197b880fa39d432a9b472845876bfaf0, 7a9daf5d42a84615a6383cfec9a10608
I20260129 02:59:51.528839 22684 leader_election.cc:290] T cedce331e1584ac8b636c9b8a72b2ba0 P 197b880fa39d432a9b472845876bfaf0 [CANDIDATE]: Term 6 election: Requested vote from peers 974b802141bd4f13beffef2aa03f88f9 (127.18.211.2:34517), 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3:37735)
I20260129 02:59:51.531787 20510 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57584 (request call id 1298) took 2029 ms. Trace:
I20260129 02:59:51.521265 21131 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3e2d15f8640b407195210c476e09df91" candidate_uuid: "974b802141bd4f13beffef2aa03f88f9" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "7a9daf5d42a84615a6383cfec9a10608"
I20260129 02:59:51.546068 21131 raft_consensus.cc:2393] T 3e2d15f8640b407195210c476e09df91 P 7a9daf5d42a84615a6383cfec9a10608 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 974b802141bd4f13beffef2aa03f88f9 in current term 2: Already voted for candidate 7a9daf5d42a84615a6383cfec9a10608 in this term.
I20260129 02:59:51.545727 20510 rpcz_store.cc:276] 0129 02:59:49.502326 (+ 0us) service_pool.cc:168] Inserting onto call queue
0129 02:59:51.338083 (+1835757us) service_pool.cc:225] Handling call
0129 02:59:51.361146 (+ 23063us) raft_consensus.cc:1471] Updating replica for 1 ops
0129 02:59:51.421250 (+ 60104us) raft_consensus.cc:1534] Early marking committed up to index 0
0129 02:59:51.421300 (+ 50us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0129 02:59:51.440797 (+ 19497us) log.cc:844] Serialized 26 byte log entry
0129 02:59:51.498268 (+ 57471us) raft_consensus.cc:1645] Marking committed up to 5
0129 02:59:51.514466 (+ 16198us) log.cc:844] Serialized 14 byte log entry
0129 02:59:51.514710 (+ 244us) raft_consensus.cc:1697] Filling consensus response to leader.
0129 02:59:51.516424 (+ 1714us) spinlock_profiling.cc:243] Waited 51.8 ms on lock 0x7b54003007d4. stack: 00007f8fe044e373 00007f8fe044e270 00007f8fdfda0574 00007f8fe9a5c6b4 00007f8fe9a5c649 00007f8fe34975a1 00007f8fe352ea81 00007f8fe352d4be 00007f8fe84e1f49 00007f8fe1a6ed34 00007f8fe1a6ecc2 00007f8fe1a6ebf1 00007f8fe1a6eb6c 00007f8fe1a6de22 00007f8fe186d34c 00007f8fe186c786
0129 02:59:51.516490 (+ 66us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0129 02:59:51.516556 (+ 66us) raft_consensus.cc:1685] finished
0129 02:59:51.524213 (+ 7657us) raft_consensus.cc:1691] UpdateReplicas() finished
0129 02:59:51.531727 (+ 7514us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":20943,"prepare.queue_time_us":38151,"raft.queue_time_us":8261,"raft.run_cpu_time_us":240,"raft.run_wall_time_us":9877,"spinlock_wait_cycles":119158528,"thread_start_us":72756,"threads_started":2,"wal-append.queue_time_us":57315}
I20260129 02:59:51.547062 21131 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "75f45538dc704d9a8ec4ff8ad9f3ea5b" candidate_uuid: "974b802141bd4f13beffef2aa03f88f9" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "7a9daf5d42a84615a6383cfec9a10608"
I20260129 02:59:51.534946 20513 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57584 (request call id 1290) took 2039 ms. Trace:
W20260129 02:59:51.535686 21100 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.211.3:37735, user_credentials={real_user=slave}} blocked reactor thread for 148381us
I20260129 02:59:51.535199 20435 rpcz_store.cc:276] 0129 02:59:49.562390 (+ 0us) service_pool.cc:168] Inserting onto call queue
0129 02:59:49.595699 (+ 33309us) service_pool.cc:225] Handling call
0129 02:59:49.596135 (+ 436us) raft_consensus.cc:1471] Updating replica for 1 ops
0129 02:59:50.278370 (+682235us) raft_consensus.cc:1697] Filling consensus response to leader.
0129 02:59:51.517929 (+1239559us) spinlock_profiling.cc:243] Waited 287 ms on lock 0x7b54000a6ed4. stack: 00007f8fe044e373 00007f8fe044e270 00007f8fdfda0574 00007f8fe9a5c6b4 00007f8fe9a5c649 00007f8fe34975a1 00007f8fe352ea81 00007f8fe352d4be 00007f8fe84e1f49 00007f8fe1a6ed34 00007f8fe1a6ecc2 00007f8fe1a6ebf1 00007f8fe1a6eb6c 00007f8fe1a6de22 00007f8fe186d34c 00007f8fe186c786
0129 02:59:51.518114 (+ 185us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":660678016}
I20260129 02:59:51.519587 20511 raft_consensus.cc:1240] T 3906b080a5b34a74b8b64128635e8997 P 974b802141bd4f13beffef2aa03f88f9 [term 3 FOLLOWER]: Rejecting Update request from peer 1c50de89a5a845668696dcb28c3badd6 for earlier term 2. Current term is 3. Ops: []
I20260129 02:59:51.538791 20581 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "36ff33be666a4c82954731691dbcf327" candidate_uuid: "974b802141bd4f13beffef2aa03f88f9" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "1c50de89a5a845668696dcb28c3badd6" is_pre_election: true
I20260129 02:59:51.547870 20513 rpcz_store.cc:276] 0129 02:59:49.495334 (+ 0us) service_pool.cc:168] Inserting onto call queue
0129 02:59:49.495458 (+ 124us) service_pool.cc:225] Handling call
0129 02:59:49.495751 (+ 293us) raft_consensus.cc:1471] Updating replica for 0 ops
0129 02:59:50.738255 (+1242504us) raft_consensus.cc:1697] Filling consensus response to leader.
0129 02:59:51.534714 (+796459us) spinlock_profiling.cc:243] Waited 1.23 s on lock 0x7b5400166f54. stack: 00007f8fe044e373 00007f8fe044e270 00007f8fdfda0574 00007f8fe9a5c6b4 00007f8fe9a5c649 00007f8fe34975a1 00007f8fe352ea81 00007f8fe352d4be 00007f8fe84e1f49 00007f8fe1a6ed34 00007f8fe1a6ecc2 00007f8fe1a6ebf1 00007f8fe1a6eb6c 00007f8fe1a6de22 00007f8fe186d34c 00007f8fe186c786
0129 02:59:51.534890 (+ 176us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2826711808}
I20260129 02:59:51.517215 22654 raft_consensus.cc:493] T 3906b080a5b34a74b8b64128635e8997 P 1c50de89a5a845668696dcb28c3badd6 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260129 02:59:51.542800 20556 leader_election.cc:304] T cb4727386a3e48aca5d71cc6c611c197 P 1c50de89a5a845668696dcb28c3badd6 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1c50de89a5a845668696dcb28c3badd6; no voters: 197b880fa39d432a9b472845876bfaf0, 974b802141bd4f13beffef2aa03f88f9
I20260129 02:59:51.549408 20513 raft_consensus.cc:1240] T febffe4a8b394368b69b1d164308fb9a P 974b802141bd4f13beffef2aa03f88f9 [term 3 FOLLOWER]: Rejecting Update request from peer 7a9daf5d42a84615a6383cfec9a10608 for earlier term 2. Current term is 3. Ops: [2.5-2.5]
I20260129 02:59:51.540215 20434 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:48356 (request call id 292) took 4489 ms. Trace:
W20260129 02:59:51.543202 20405 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.211.3:37735, user_credentials={real_user=slave}} blocked reactor thread for 927690us
W20260129 02:59:51.543342 20407 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.211.2:34517, user_credentials={real_user=slave}} blocked reactor thread for 944522us
I20260129 02:59:51.550745 20405 leader_election.cc:304] T 2e58e9a5a8aa4cecafb464addb4f35ed P 197b880fa39d432a9b472845876bfaf0 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 197b880fa39d432a9b472845876bfaf0; no voters: 1c50de89a5a845668696dcb28c3badd6, 7a9daf5d42a84615a6383cfec9a10608
I20260129 02:59:51.550264 20434 rpcz_store.cc:276] 0129 02:59:47.051554 (+ 0us) service_pool.cc:168] Inserting onto call queue
0129 02:59:47.051703 (+ 149us) service_pool.cc:225] Handling call
0129 02:59:47.074420 (+ 22717us) raft_consensus.cc:1471] Updating replica for 1 ops
0129 02:59:50.293611 (+3219191us) spinlock_profiling.cc:243] Waited 53.7 ms on lock 0x7b18003e8c00. stack: 00007f8fe044e373 00007f8fe044e270 00007f8fdfda0574 00007f8fe9a5c6b4 00007f8fe9a5c649 00007f8fe92d4c91 00007f8fe92e1671 00007f8fe30d585c 00007f8fe04632eb 00007f8fe924b709 00007f8fe047f57b 00007f8fe047d81f 00007f8fe047d335 00007f8fe3525ce0 00007f8fe352687b 00007f8fe35312cb
0129 02:59:50.312701 (+ 19090us) raft_consensus.cc:1534] Early marking committed up to index 0
0129 02:59:50.312721 (+ 20us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0129 02:59:50.321486 (+ 8765us) log.cc:844] Serialized 26 byte log entry
0129 02:59:50.583147 (+261661us) raft_consensus.cc:1645] Marking committed up to 6
0129 02:59:50.609952 (+ 26805us) log.cc:844] Serialized 14 byte log entry
0129 02:59:51.539682 (+929730us) spinlock_profiling.cc:243] Waited 2.55 ms on lock 0x7b18003e8c00. stack: 00007f8fe044e373 00007f8fe044e270 00007f8fdfda0574 00007f8fe9a5c6b4 00007f8fe9a5c649 00007f8fe92d4c91 00007f8fe92e1671 00007f8fe03f546a 00007f8fe9a7b5c9 00007f8fe02b5160 00007f8fe047d3c4 00007f8fe047d335 00007f8fe3525ce0 00007f8fe3538396 00007f8fe3548f07 00007f8fe3548e88
0129 02:59:51.539899 (+ 217us) raft_consensus.cc:1697] Filling consensus response to leader.
0129 02:59:51.539976 (+ 77us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0129 02:59:51.540017 (+ 41us) raft_consensus.cc:1685] finished
0129 02:59:51.540031 (+ 14us) raft_consensus.cc:1691] UpdateReplicas() finished
0129 02:59:51.540156 (+ 125us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":25164,"prepare.queue_time_us":27222,"prepare.run_cpu_time_us":37,"prepare.run_wall_time_us":40,"raft.queue_time_us":634417,"raft.run_cpu_time_us":173,"raft.run_wall_time_us":20191,"spinlock_wait_cycles":129434112,"thread_start_us":884678,"threads_started":2,"wal-append.queue_time_us":329668}
I20260129 02:59:51.556706 22796 raft_consensus.cc:2804] T a4c1226b987b468fa84dfc6ba0e2dbe1 P 197b880fa39d432a9b472845876bfaf0 [term 4 FOLLOWER]: Leader election won for term 4
I20260129 02:59:51.557307 22796 raft_consensus.cc:697] T a4c1226b987b468fa84dfc6ba0e2dbe1 P 197b880fa39d432a9b472845876bfaf0 [term 4 LEADER]: Becoming Leader. State: Replica: 197b880fa39d432a9b472845876bfaf0, State: Running, Role: LEADER
I20260129 02:59:51.558228 22796 consensus_queue.cc:237] T a4c1226b987b468fa84dfc6ba0e2dbe1 P 197b880fa39d432a9b472845876bfaf0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:59:51.533172 22738 raft_consensus.cc:740] T 368953c9102643b2a21f99af36b51905 P 974b802141bd4f13beffef2aa03f88f9 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 974b802141bd4f13beffef2aa03f88f9, State: Running, Role: LEADER
I20260129 02:59:51.532086 22694 consensus_queue.cc:1048] T a609ac02fae84725ab743f95f51d33ce P 974b802141bd4f13beffef2aa03f88f9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.042s
I20260129 02:59:51.505314 22729 consensus_queue.cc:237] T 00e79d8f25a34676b1d866eb2068a2b2 P 7a9daf5d42a84615a6383cfec9a10608 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.7, Last appended by leader: 7, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } }
I20260129 02:59:51.532624 22667 raft_consensus.cc:515] T a644c02b16af47679824ccd060ac10fd P 7a9daf5d42a84615a6383cfec9a10608 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } }
I20260129 02:59:51.565791 20582 tablet_service.cc:2066] Received LeaderStepDown RPC: tablet_id: "75b0b6e82ddc4f92b1fe1d94f9a0b73b"
dest_uuid: "1c50de89a5a845668696dcb28c3badd6"
mode: GRACEFUL
new_leader_uuid: "974b802141bd4f13beffef2aa03f88f9"
from {username='slave'} at 127.0.0.1:54860
W20260129 02:59:51.605463 20482 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260129 02:59:51.655292 20582 raft_consensus.cc:606] T 75b0b6e82ddc4f92b1fe1d94f9a0b73b P 1c50de89a5a845668696dcb28c3badd6 [term 3 LEADER]: Received request to transfer leadership to TS 974b802141bd4f13beffef2aa03f88f9
I20260129 02:59:51.650396 21131 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "cb0b3f259ebc4310927338a8740e87b8" candidate_uuid: "1c50de89a5a845668696dcb28c3badd6" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "7a9daf5d42a84615a6383cfec9a10608" is_pre_election: true
W20260129 02:59:51.585143 20481 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260129 02:59:51.651919 20405 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.211.3:37735, user_credentials={real_user=slave}} blocked reactor thread for 101243us
I20260129 02:59:51.660392 20405 leader_election.cc:304] T 724a236524cb4e7ba3428880681d6cc2 P 197b880fa39d432a9b472845876bfaf0 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 197b880fa39d432a9b472845876bfaf0; no voters: 1c50de89a5a845668696dcb28c3badd6, 974b802141bd4f13beffef2aa03f88f9
I20260129 02:59:51.615031 20479 leader_election.cc:304] T 3906b080a5b34a74b8b64128635e8997 P 974b802141bd4f13beffef2aa03f88f9 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 197b880fa39d432a9b472845876bfaf0, 974b802141bd4f13beffef2aa03f88f9; no voters: 1c50de89a5a845668696dcb28c3badd6
I20260129 02:59:51.620656 20433 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b189f08e16a74a11980cc55d99359a0a" candidate_uuid: "974b802141bd4f13beffef2aa03f88f9" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "197b880fa39d432a9b472845876bfaf0" is_pre_election: true
I20260129 02:59:51.610172 20581 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:48526 (request call id 252) took 2802 ms. Trace:
I20260129 02:59:51.561167 20556 leader_election.cc:304] T b189f08e16a74a11980cc55d99359a0a P 1c50de89a5a845668696dcb28c3badd6 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1c50de89a5a845668696dcb28c3badd6; no voters: 197b880fa39d432a9b472845876bfaf0, 974b802141bd4f13beffef2aa03f88f9
I20260129 02:59:51.710685 20587 raft_consensus.cc:2468] T 75f45538dc704d9a8ec4ff8ad9f3ea5b P 1c50de89a5a845668696dcb28c3badd6 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 974b802141bd4f13beffef2aa03f88f9 in term 3.
I20260129 02:59:51.674063 21135 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34060 (request call id 497) took 1182 ms. Trace:
W20260129 02:59:51.613072 20407 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.211.2:34517, user_credentials={real_user=slave}} blocked reactor thread for 62339.6us
W20260129 02:59:51.722090 20405 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.211.3:37735, user_credentials={real_user=slave}} blocked reactor thread for 61779.8us
I20260129 02:59:51.709753 20432 raft_consensus.cc:1275] T 78aa39096868415d9afc2ffcb248e198 P 197b880fa39d432a9b472845876bfaf0 [term 5 FOLLOWER]: Refusing update from remote peer 1c50de89a5a845668696dcb28c3badd6: Log matching property violated. Preceding OpId in replica: term: 3 index: 5. Preceding OpId from leader: term: 5 index: 6. (index mismatch)
I20260129 02:59:51.721580 20480 leader_election.cc:304] T 75f45538dc704d9a8ec4ff8ad9f3ea5b P 974b802141bd4f13beffef2aa03f88f9 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1c50de89a5a845668696dcb28c3badd6, 974b802141bd4f13beffef2aa03f88f9; no voters: 7a9daf5d42a84615a6383cfec9a10608
I20260129 02:59:51.724442 20587 raft_consensus.cc:1240] T 75f45538dc704d9a8ec4ff8ad9f3ea5b P 1c50de89a5a845668696dcb28c3badd6 [term 3 FOLLOWER]: Rejecting Update request from peer 7a9daf5d42a84615a6383cfec9a10608 for earlier term 2. Current term is 3. Ops: []
I20260129 02:59:51.725241 20587 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:48526 (request call id 253) took 2260 ms. Trace:
I20260129 02:59:51.723564 20405 leader_election.cc:304] T 3906b080a5b34a74b8b64128635e8997 P 197b880fa39d432a9b472845876bfaf0 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 197b880fa39d432a9b472845876bfaf0; no voters: 1c50de89a5a845668696dcb28c3badd6, 974b802141bd4f13beffef2aa03f88f9
I20260129 02:59:51.725641 20587 rpcz_store.cc:276] 0129 02:59:49.464837 (+ 0us) service_pool.cc:168] Inserting onto call queue
0129 02:59:51.721076 (+2256239us) service_pool.cc:225] Handling call
0129 02:59:51.724185 (+ 3109us) raft_consensus.cc:1471] Updating replica for 0 ops
0129 02:59:51.724926 (+ 741us) raft_consensus.cc:1697] Filling consensus response to leader.
0129 02:59:51.725176 (+ 250us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260129 02:59:51.727236 20587 raft_consensus.cc:1240] T 1ce4ddf5b10e4ae8b193b13b2e5936ee P 1c50de89a5a845668696dcb28c3badd6 [term 6 LEADER]: Rejecting Update request from peer 197b880fa39d432a9b472845876bfaf0 for earlier term 5. Current term is 6. Ops: []
I20260129 02:59:51.728645 20405 leader_election.cc:304] T 368953c9102643b2a21f99af36b51905 P 197b880fa39d432a9b472845876bfaf0 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 197b880fa39d432a9b472845876bfaf0; no voters: 1c50de89a5a845668696dcb28c3badd6, 974b802141bd4f13beffef2aa03f88f9
I20260129 02:59:51.729697 20434 raft_consensus.cc:1275] T 8055232cf0004d40a85f3bf983fe00a9 P 197b880fa39d432a9b472845876bfaf0 [term 3 FOLLOWER]: Refusing update from remote peer 974b802141bd4f13beffef2aa03f88f9: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 2. (term mismatch)
I20260129 02:59:51.730350 20434 pending_rounds.cc:85] T 8055232cf0004d40a85f3bf983fe00a9 P 197b880fa39d432a9b472845876bfaf0: Aborting all ops after (but not including) 1
I20260129 02:59:51.730638 20434 pending_rounds.cc:107] T 8055232cf0004d40a85f3bf983fe00a9 P 197b880fa39d432a9b472845876bfaf0: Aborting uncommitted NO_OP operation due to leader change: 2.2
I20260129 02:59:51.730928 20434 raft_consensus.cc:2889] T 8055232cf0004d40a85f3bf983fe00a9 P 197b880fa39d432a9b472845876bfaf0 [term 3 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260129 02:59:51.542112 22820 raft_consensus.cc:493] T b189f08e16a74a11980cc55d99359a0a P 974b802141bd4f13beffef2aa03f88f9 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260129 02:59:51.543164 22748 raft_consensus.cc:2749] T a0fcc58d6fb6473cafbd9d34d5195269 P 1c50de89a5a845668696dcb28c3badd6 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260129 02:59:51.612039 21128 raft_consensus.cc:1240] T 3ccac82a01a2445fa5b0ce6c0fcf2295 P 7a9daf5d42a84615a6383cfec9a10608 [term 4 FOLLOWER]: Rejecting Update request from peer 974b802141bd4f13beffef2aa03f88f9 for earlier term 3. Current term is 4. Ops: []
I20260129 02:59:51.628636 22400 raft_consensus.cc:2749] T a1a39dcd328e4268ae41b988c66b4a75 P 974b802141bd4f13beffef2aa03f88f9 [term 4 FOLLOWER]: Leader election lost for term 3. Reason: Vote denied by peer 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3:37735) with higher term. Message: Invalid argument: T a1a39dcd328e4268ae41b988c66b4a75 P 1c50de89a5a845668696dcb28c3badd6 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 974b802141bd4f13beffef2aa03f88f9 for earlier term 3. Current term is 4.
I20260129 02:59:51.628746 22814 leader_election.cc:290] T 724a236524cb4e7ba3428880681d6cc2 P 1c50de89a5a845668696dcb28c3badd6 [CANDIDATE]: Term 4 election: Requested vote from peers 197b880fa39d432a9b472845876bfaf0 (127.18.211.1:42027), 974b802141bd4f13beffef2aa03f88f9 (127.18.211.2:34517)
I20260129 02:59:51.609197 22742 leader_election.cc:290] T febffe4a8b394368b69b1d164308fb9a P 7a9daf5d42a84615a6383cfec9a10608 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 197b880fa39d432a9b472845876bfaf0 (127.18.211.1:42027), 974b802141bd4f13beffef2aa03f88f9 (127.18.211.2:34517)
I20260129 02:59:51.745373 21134 raft_consensus.cc:1240] T a644c02b16af47679824ccd060ac10fd P 7a9daf5d42a84615a6383cfec9a10608 [term 4 FOLLOWER]: Rejecting Update request from peer 1c50de89a5a845668696dcb28c3badd6 for earlier term 3. Current term is 4. Ops: []
I20260129 02:59:51.752182 20587 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35754 (request call id 1655) took 2243 ms. Trace:
W20260129 02:59:51.743428 20554 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260129 02:59:51.746501 20511 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "febffe4a8b394368b69b1d164308fb9a" candidate_uuid: "7a9daf5d42a84615a6383cfec9a10608" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "974b802141bd4f13beffef2aa03f88f9" is_pre_election: true
I20260129 02:59:51.723944 22832 raft_consensus.cc:493] T add8d4923d1a489e86882c3a25692b95 P 7a9daf5d42a84615a6383cfec9a10608 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260129 02:59:51.768702 20511 raft_consensus.cc:2468] T febffe4a8b394368b69b1d164308fb9a P 974b802141bd4f13beffef2aa03f88f9 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7a9daf5d42a84615a6383cfec9a10608 in term 3.
I20260129 02:59:51.768335 20587 rpcz_store.cc:276] 0129 02:59:49.501722 (+ 0us) service_pool.cc:168] Inserting onto call queue
0129 02:59:51.726638 (+2224916us) service_pool.cc:225] Handling call
0129 02:59:51.727011 (+ 373us) raft_consensus.cc:1471] Updating replica for 0 ops
0129 02:59:51.727661 (+ 650us) raft_consensus.cc:1697] Filling consensus response to leader.
0129 02:59:51.745142 (+ 17481us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260129 02:59:51.753324 20581 rpcz_store.cc:276] 0129 02:59:48.807344 (+ 0us) service_pool.cc:168] Inserting onto call queue
0129 02:59:51.578420 (+2771076us) service_pool.cc:225] Handling call
0129 02:59:51.609691 (+ 31271us) raft_consensus.cc:1471] Updating replica for 0 ops
0129 02:59:51.609881 (+ 190us) raft_consensus.cc:1534] Early marking committed up to index 0
0129 02:59:51.609905 (+ 24us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0129 02:59:51.609947 (+ 42us) raft_consensus.cc:1645] Marking committed up to 6
0129 02:59:51.609981 (+ 34us) raft_consensus.cc:1697] Filling consensus response to leader.
0129 02:59:51.610030 (+ 49us) raft_consensus.cc:1691] UpdateReplicas() finished
0129 02:59:51.610120 (+ 90us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20260129 02:59:51.687296 21101 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260129 02:59:51.720481 21135 rpcz_store.cc:276] 0129 02:59:50.491333 (+ 0us) service_pool.cc:168] Inserting onto call queue
0129 02:59:50.491491 (+ 158us) service_pool.cc:225] Handling call
0129 02:59:50.491768 (+ 277us) raft_consensus.cc:1471] Updating replica for 0 ops
0129 02:59:51.420549 (+928781us) raft_consensus.cc:1697] Filling consensus response to leader.
0129 02:59:51.673699 (+253150us) spinlock_profiling.cc:243] Waited 902 ms on lock 0x7b5400441454. stack: 00007f8fe044e373 00007f8fe044e270 00007f8fdfda0574 00007f8fe9a5c6b4 00007f8fe9a5c649 00007f8fe34975a1 00007f8fe352ea81 00007f8fe352d4be 00007f8fe84e1f49 00007f8fe1a6ed34 00007f8fe1a6ecc2 00007f8fe1a6ebf1 00007f8fe1a6eb6c 00007f8fe1a6de22 00007f8fe186d34c 00007f8fe186c786
0129 02:59:51.673972 (+ 273us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2074123136}
I20260129 02:59:51.609210 22581 leader_election.cc:290] T cb0b3f259ebc4310927338a8740e87b8 P 1c50de89a5a845668696dcb28c3badd6 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 974b802141bd4f13beffef2aa03f88f9 (127.18.211.2:34517), 7a9daf5d42a84615a6383cfec9a10608 (127.18.211.4:40439)
W20260129 02:59:51.696241 22836 log.cc:927] Time spent T 75b0b6e82ddc4f92b1fe1d94f9a0b73b P 1c50de89a5a845668696dcb28c3badd6: Append to log took a long time: real 0.135s user 0.001s sys 0.000s
I20260129 02:59:51.757457 20582 raft_consensus.cc:1240] T 8055232cf0004d40a85f3bf983fe00a9 P 1c50de89a5a845668696dcb28c3badd6 [term 3 FOLLOWER]: Rejecting Update request from peer 197b880fa39d432a9b472845876bfaf0 for earlier term 2. Current term is 3. Ops: [2.2-2.2]
I20260129 02:59:51.559746 22348 raft_consensus.cc:515] T 4408006eb5174de1896b037cf3f18457 P 1c50de89a5a845668696dcb28c3badd6 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:59:51.778261 20582 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35754 (request call id 1645) took 2320 ms. Trace:
I20260129 02:59:51.757820 20405 leader_election.cc:400] T 1299723a4476420ab02fd8bfd7aa0645 P 197b880fa39d432a9b472845876bfaf0 [CANDIDATE]: Term 7 pre-election: Vote denied by peer 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3:37735) with higher term. Message: Invalid argument: T 1299723a4476420ab02fd8bfd7aa0645 P 1c50de89a5a845668696dcb28c3badd6 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 197b880fa39d432a9b472845876bfaf0 for earlier term 7. Current term is 8.
I20260129 02:59:51.628676 22540 raft_consensus.cc:493] T 4b1876f9a70b49edb3aa74105e66d3b3 P 197b880fa39d432a9b472845876bfaf0 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 974b802141bd4f13beffef2aa03f88f9)
I20260129 02:59:51.561328 21129 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "78aa39096868415d9afc2ffcb248e198" candidate_uuid: "1c50de89a5a845668696dcb28c3badd6" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "7a9daf5d42a84615a6383cfec9a10608"
I20260129 02:59:51.652100 22647 raft_consensus.cc:2749] T 2e58e9a5a8aa4cecafb464addb4f35ed P 197b880fa39d432a9b472845876bfaf0 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
W20260129 02:59:51.760761 20479 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.211.1:42027, user_credentials={real_user=slave}} blocked reactor thread for 145795us
I20260129 02:59:51.761408 20433 raft_consensus.cc:2468] T b189f08e16a74a11980cc55d99359a0a P 197b880fa39d432a9b472845876bfaf0 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 974b802141bd4f13beffef2aa03f88f9 in term 6.
W20260129 02:59:51.593847 22834 log.cc:927] Time spent T 4f5a77e94fef4b16a2ba7b067c8b144a P 974b802141bd4f13beffef2aa03f88f9: Append to log took a long time: real 0.057s user 0.001s sys 0.000s
W20260129 02:59:51.602617 22830 log.cc:927] Time spent T dc2f4cb752cf4d59b5f6bc689f5d8104 P 7a9daf5d42a84615a6383cfec9a10608: Append to log took a long time: real 0.064s user 0.000s sys 0.000s
I20260129 02:59:51.578730 20513 raft_consensus.cc:1240] T 75f45538dc704d9a8ec4ff8ad9f3ea5b P 974b802141bd4f13beffef2aa03f88f9 [term 3 FOLLOWER]: Rejecting Update request from peer 7a9daf5d42a84615a6383cfec9a10608 for earlier term 2. Current term is 3. Ops: []
I20260129 02:59:51.762547 20438 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a1a39dcd328e4268ae41b988c66b4a75" candidate_uuid: "1c50de89a5a845668696dcb28c3badd6" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "197b880fa39d432a9b472845876bfaf0"
I20260129 02:59:51.799183 20438 raft_consensus.cc:3060] T a1a39dcd328e4268ae41b988c66b4a75 P 197b880fa39d432a9b472845876bfaf0 [term 3 FOLLOWER]: Advancing to term 4
I20260129 02:59:51.745757 20432 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7a9a69cd82724b1faed1d10aa091fc4a" candidate_uuid: "1c50de89a5a845668696dcb28c3badd6" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "197b880fa39d432a9b472845876bfaf0" is_pre_election: true
I20260129 02:59:51.577462 22751 raft_consensus.cc:2749] T cb4727386a3e48aca5d71cc6c611c197 P 1c50de89a5a845668696dcb28c3badd6 [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260129 02:59:51.657477 22815 raft_consensus.cc:493] T 17d6431cc77a491e8464d68db4081a51 P 974b802141bd4f13beffef2aa03f88f9 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260129 02:59:51.559993 22719 raft_consensus.cc:515] T a5a05fbcbbcf4916bbc01fd1a5b5e90d P 974b802141bd4f13beffef2aa03f88f9 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
W20260129 02:59:51.764288 20556 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.211.2:34517, user_credentials={real_user=slave}} blocked reactor thread for 203183us
I20260129 02:59:51.745599 21130 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34060 (request call id 495) took 1406 ms. Trace:
I20260129 02:59:51.808775 21130 rpcz_store.cc:276] 0129 02:59:50.338898 (+ 0us) service_pool.cc:168] Inserting onto call queue
0129 02:59:50.371661 (+ 32763us) service_pool.cc:225] Handling call
0129 02:59:51.659586 (+1287925us) spinlock_profiling.cc:243] Waited 444 ms on lock 0x7b5400670554. stack: 00007f8fe044e373 00007f8fe044e270 00007f8fdfda0574 00007f8fe9a5c6b4 00007f8fe9a5c649 00007f8fe34975a1 00007f8fe3531f7a 00007f8fe84e3647 00007f8fe1a73834 00007f8fe1a737c2 00007f8fe1a736f1 00007f8fe1a7366c 00007f8fe1a72922 00007f8fe186d34c 00007f8fe186c786 00007f8fe186c12f
0129 02:59:51.740592 (+ 81006us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1021871488}
I20260129 02:59:51.766865 20434 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e4d18c35b0a7493e85d0a4b92956a48f" candidate_uuid: "1c50de89a5a845668696dcb28c3badd6" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "197b880fa39d432a9b472845876bfaf0"
I20260129 02:59:51.810217 20434 raft_consensus.cc:2393] T e4d18c35b0a7493e85d0a4b92956a48f P 197b880fa39d432a9b472845876bfaf0 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 1c50de89a5a845668696dcb28c3badd6 in current term 5: Already voted for candidate 974b802141bd4f13beffef2aa03f88f9 in this term.
I20260129 02:59:51.801254 22445 raft_consensus.cc:2804] T 75f45538dc704d9a8ec4ff8ad9f3ea5b P 974b802141bd4f13beffef2aa03f88f9 [term 3 FOLLOWER]: Leader election won for term 3
I20260129 02:59:51.779825 22750 consensus_queue.cc:1048] T 0f20af7fbe5d4db89a6f17948d98912c P 197b880fa39d432a9b472845876bfaf0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.033s
I20260129 02:59:51.814299 22403 raft_consensus.cc:515] T 0678b8b468bd43cdbc3e485900844821 P 7a9daf5d42a84615a6383cfec9a10608 [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } }
I20260129 02:59:51.747990 22641 raft_consensus.cc:2749] T 3906b080a5b34a74b8b64128635e8997 P 197b880fa39d432a9b472845876bfaf0 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260129 02:59:51.797997 22694 raft_consensus.cc:493] T add8d4923d1a489e86882c3a25692b95 P 974b802141bd4f13beffef2aa03f88f9 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260129 02:59:51.746672 22820 raft_consensus.cc:3060] T b189f08e16a74a11980cc55d99359a0a P 974b802141bd4f13beffef2aa03f88f9 [term 6 FOLLOWER]: Advancing to term 7
I20260129 02:59:51.817821 22694 raft_consensus.cc:515] T add8d4923d1a489e86882c3a25692b95 P 974b802141bd4f13beffef2aa03f88f9 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } }
I20260129 02:59:51.740437 22798 raft_consensus.cc:2749] T 724a236524cb4e7ba3428880681d6cc2 P 197b880fa39d432a9b472845876bfaf0 [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260129 02:59:51.770582 21102 leader_election.cc:304] T febffe4a8b394368b69b1d164308fb9a P 7a9daf5d42a84615a6383cfec9a10608 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7a9daf5d42a84615a6383cfec9a10608, 974b802141bd4f13beffef2aa03f88f9; no voters:
I20260129 02:59:51.779121 20405 leader_election.cc:403] T 1299723a4476420ab02fd8bfd7aa0645 P 197b880fa39d432a9b472845876bfaf0 [CANDIDATE]: Term 7 pre-election: Cancelling election due to peer responding with higher term
I20260129 02:59:51.778652 20582 rpcz_store.cc:276] 0129 02:59:49.458264 (+ 0us) service_pool.cc:168] Inserting onto call queue
0129 02:59:51.661135 (+2202871us) service_pool.cc:225] Handling call
0129 02:59:51.757107 (+ 95972us) raft_consensus.cc:1471] Updating replica for 1 ops
0129 02:59:51.778007 (+ 20900us) raft_consensus.cc:1697] Filling consensus response to leader.
0129 02:59:51.778202 (+ 195us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260129 02:59:51.822242 20507 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57584 (request call id 1312) took 1204 ms. Trace:
I20260129 02:59:51.778270 20437 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "724a236524cb4e7ba3428880681d6cc2" candidate_uuid: "1c50de89a5a845668696dcb28c3badd6" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "197b880fa39d432a9b472845876bfaf0"
W20260129 02:59:51.823725 20585 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:35754 (request call id 1644) took 2365 ms (client timeout 1733 ms). Trace:
I20260129 02:59:51.823603 20438 raft_consensus.cc:2468] T a1a39dcd328e4268ae41b988c66b4a75 P 197b880fa39d432a9b472845876bfaf0 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1c50de89a5a845668696dcb28c3badd6 in term 4.
I20260129 02:59:51.783612 20479 leader_election.cc:304] T e33a417cf9c54cf5af73a4ee3aac4d3a P 974b802141bd4f13beffef2aa03f88f9 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 197b880fa39d432a9b472845876bfaf0, 974b802141bd4f13beffef2aa03f88f9; no voters: 1c50de89a5a845668696dcb28c3badd6
I20260129 02:59:51.784157 20433 raft_consensus.cc:1240] T 4e3805ffa70242e8ade66da260dddf66 P 197b880fa39d432a9b472845876bfaf0 [term 4 FOLLOWER]: Rejecting Update request from peer 1c50de89a5a845668696dcb28c3badd6 for earlier term 3. Current term is 4. Ops: []
I20260129 02:59:51.577679 22654 raft_consensus.cc:515] T 3906b080a5b34a74b8b64128635e8997 P 1c50de89a5a845668696dcb28c3badd6 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:59:51.773185 22231 consensus_queue.cc:1059] T 76f713c3331540d2a5b76164103e9eb1 P 197b880fa39d432a9b472845876bfaf0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 }, Status: INVALID_TERM, Last received: 3.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.012s
I20260129 02:59:51.785393 22540 raft_consensus.cc:515] T 4b1876f9a70b49edb3aa74105e66d3b3 P 197b880fa39d432a9b472845876bfaf0 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:59:51.766212 21132 raft_consensus.cc:1240] T 18f30ebaa4174fd28c958e72d0cc7e24 P 7a9daf5d42a84615a6383cfec9a10608 [term 5 FOLLOWER]: Rejecting Update request from peer 1c50de89a5a845668696dcb28c3badd6 for earlier term 4. Current term is 5. Ops: []
I20260129 02:59:51.769284 22237 consensus_queue.cc:1059] T 1ce4ddf5b10e4ae8b193b13b2e5936ee P 197b880fa39d432a9b472845876bfaf0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 }, Status: INVALID_TERM, Last received: 0.0, Next index: 3, Last known committed idx: 1, Time since last communication: 0.017s
W20260129 02:59:51.786561 22829 log.cc:927] Time spent T 1299723a4476420ab02fd8bfd7aa0645 P 7a9daf5d42a84615a6383cfec9a10608: Append to log took a long time: real 0.172s user 0.000s sys 0.000s
I20260129 02:59:51.786906 22640 consensus_queue.cc:1048] T e33a417cf9c54cf5af73a4ee3aac4d3a P 1c50de89a5a845668696dcb28c3badd6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260129 02:59:51.790027 22691 leader_election.cc:290] T 2ac659d732314e1288849f08fe5e074e P 197b880fa39d432a9b472845876bfaf0 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 974b802141bd4f13beffef2aa03f88f9 (127.18.211.2:34517), 7a9daf5d42a84615a6383cfec9a10608 (127.18.211.4:40439)
I20260129 02:59:51.780309 22737 raft_consensus.cc:493] T fcddab6bca684adf8ea032289a00f90e P 197b880fa39d432a9b472845876bfaf0 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260129 02:59:51.710153 22425 consensus_queue.cc:1059] T a4c1226b987b468fa84dfc6ba0e2dbe1 P 1c50de89a5a845668696dcb28c3badd6 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 }, Status: INVALID_TERM, Last received: 3.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.067s
I20260129 02:59:51.605263 22707 raft_consensus.cc:515] T 4e420f90884f4796997b5ea41e49e411 P 974b802141bd4f13beffef2aa03f88f9 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } }
I20260129 02:59:51.768262 22493 raft_consensus.cc:2804] T 3906b080a5b34a74b8b64128635e8997 P 974b802141bd4f13beffef2aa03f88f9 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260129 02:59:51.880502 22493 raft_consensus.cc:493] T 3906b080a5b34a74b8b64128635e8997 P 974b802141bd4f13beffef2aa03f88f9 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260129 02:59:51.881138 22493 raft_consensus.cc:3060] T 3906b080a5b34a74b8b64128635e8997 P 974b802141bd4f13beffef2aa03f88f9 [term 3 FOLLOWER]: Advancing to term 4
I20260129 02:59:51.765820 22320 consensus_queue.cc:1059] T 7d9494f5584d4d5db8b9adf1c8fc888b P 1c50de89a5a845668696dcb28c3badd6 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 }, Status: INVALID_TERM, Last received: 2.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.020s
I20260129 02:59:51.796775 20511 raft_consensus.cc:1275] T febffe4a8b394368b69b1d164308fb9a P 974b802141bd4f13beffef2aa03f88f9 [term 3 FOLLOWER]: Refusing update from remote peer 197b880fa39d432a9b472845876bfaf0: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 3 index: 5. (index mismatch)
I20260129 02:59:51.884950 20511 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "cedce331e1584ac8b636c9b8a72b2ba0" candidate_uuid: "197b880fa39d432a9b472845876bfaf0" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "974b802141bd4f13beffef2aa03f88f9"
I20260129 02:59:51.885087 22343 raft_consensus.cc:515] T 87d5fe48a44a408aaf03e894a67ecb2f P 7a9daf5d42a84615a6383cfec9a10608 [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } }
I20260129 02:59:51.885824 20511 raft_consensus.cc:3060] T cedce331e1584ac8b636c9b8a72b2ba0 P 974b802141bd4f13beffef2aa03f88f9 [term 5 FOLLOWER]: Advancing to term 6
I20260129 02:59:51.770342 22344 raft_consensus.cc:2749] T b189f08e16a74a11980cc55d99359a0a P 1c50de89a5a845668696dcb28c3badd6 [term 6 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260129 02:59:51.801728 20556 leader_election.cc:304] T 75be4bd6bd45474795fd472eff2953ac P 1c50de89a5a845668696dcb28c3badd6 [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1c50de89a5a845668696dcb28c3badd6, 974b802141bd4f13beffef2aa03f88f9; no voters: 197b880fa39d432a9b472845876bfaf0
I20260129 02:59:51.802506 20587 raft_consensus.cc:1240] T 4408006eb5174de1896b037cf3f18457 P 1c50de89a5a845668696dcb28c3badd6 [term 3 FOLLOWER]: Rejecting Update request from peer 197b880fa39d432a9b472845876bfaf0 for earlier term 2. Current term is 3. Ops: []
I20260129 02:59:51.794423 20553 leader_election.cc:304] T 42d27a28d89b409e8807ec0efd97af69 P 1c50de89a5a845668696dcb28c3badd6 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 197b880fa39d432a9b472845876bfaf0, 1c50de89a5a845668696dcb28c3badd6; no voters: 7a9daf5d42a84615a6383cfec9a10608
I20260129 02:59:51.799752 20513 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7d9494f5584d4d5db8b9adf1c8fc888b" candidate_uuid: "197b880fa39d432a9b472845876bfaf0" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "974b802141bd4f13beffef2aa03f88f9"
W20260129 02:59:51.804513 22835 log.cc:927] Time spent T 87d5fe48a44a408aaf03e894a67ecb2f P 974b802141bd4f13beffef2aa03f88f9: Append to log took a long time: real 0.206s user 0.001s sys 0.000s
I20260129 02:59:51.805752 22719 leader_election.cc:290] T a5a05fbcbbcf4916bbc01fd1a5b5e90d P 974b802141bd4f13beffef2aa03f88f9 [CANDIDATE]: Term 7 election: Requested vote from peers 197b880fa39d432a9b472845876bfaf0 (127.18.211.1:42027), 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3:37735)
I20260129 02:59:51.888904 20513 raft_consensus.cc:3060] T 7d9494f5584d4d5db8b9adf1c8fc888b P 974b802141bd4f13beffef2aa03f88f9 [term 2 FOLLOWER]: Advancing to term 3
I20260129 02:59:51.889523 22719 raft_consensus.cc:2804] T e33a417cf9c54cf5af73a4ee3aac4d3a P 974b802141bd4f13beffef2aa03f88f9 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260129 02:59:51.889768 20508 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57600 (request call id 1777) took 2166 ms. Trace:
I20260129 02:59:51.889935 22719 raft_consensus.cc:493] T e33a417cf9c54cf5af73a4ee3aac4d3a P 974b802141bd4f13beffef2aa03f88f9 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260129 02:59:51.806394 22815 raft_consensus.cc:515] T 17d6431cc77a491e8464d68db4081a51 P 974b802141bd4f13beffef2aa03f88f9 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:59:51.890213 20508 rpcz_store.cc:276] 0129 02:59:49.723210 (+ 0us) service_pool.cc:168] Inserting onto call queue
0129 02:59:51.428367 (+1705157us) service_pool.cc:225] Handling call
0129 02:59:51.428697 (+ 330us) raft_consensus.cc:1471] Updating replica for 1 ops
0129 02:59:51.441667 (+ 12970us) raft_consensus.cc:1534] Early marking committed up to index 0
0129 02:59:51.441699 (+ 32us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0129 02:59:51.442145 (+ 446us) log.cc:844] Serialized 26 byte log entry
0129 02:59:51.501080 (+ 58935us) raft_consensus.cc:1645] Marking committed up to 6
0129 02:59:51.501185 (+ 105us) raft_consensus.cc:1697] Filling consensus response to leader.
0129 02:59:51.501268 (+ 83us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0129 02:59:51.889541 (+388273us) raft_consensus.cc:1685] finished
0129 02:59:51.889562 (+ 21us) raft_consensus.cc:1691] UpdateReplicas() finished
0129 02:59:51.889707 (+ 145us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":12388,"raft.run_cpu_time_us":88,"raft.run_wall_time_us":88,"thread_start_us":58495,"threads_started":1,"wal-append.queue_time_us":81214}
I20260129 02:59:51.892122 22815 leader_election.cc:290] T 17d6431cc77a491e8464d68db4081a51 P 974b802141bd4f13beffef2aa03f88f9 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 197b880fa39d432a9b472845876bfaf0 (127.18.211.1:42027), 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3:37735)
I20260129 02:59:51.892275 20508 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e7d83d4ccf684e76acb29a0dc844e1b8" candidate_uuid: "197b880fa39d432a9b472845876bfaf0" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "974b802141bd4f13beffef2aa03f88f9"
I20260129 02:59:51.892930 20508 raft_consensus.cc:3060] T e7d83d4ccf684e76acb29a0dc844e1b8 P 974b802141bd4f13beffef2aa03f88f9 [term 4 FOLLOWER]: Advancing to term 5
I20260129 02:59:51.769423 22832 raft_consensus.cc:515] T add8d4923d1a489e86882c3a25692b95 P 7a9daf5d42a84615a6383cfec9a10608 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } }
I20260129 02:59:51.887784 22343 leader_election.cc:290] T 87d5fe48a44a408aaf03e894a67ecb2f P 7a9daf5d42a84615a6383cfec9a10608 [CANDIDATE]: Term 6 election: Requested vote from peers 197b880fa39d432a9b472845876bfaf0 (127.18.211.1:42027), 974b802141bd4f13beffef2aa03f88f9 (127.18.211.2:34517)
I20260129 02:59:51.895092 22832 leader_election.cc:290] T add8d4923d1a489e86882c3a25692b95 P 7a9daf5d42a84615a6383cfec9a10608 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 974b802141bd4f13beffef2aa03f88f9 (127.18.211.2:34517), 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3:37735)
I20260129 02:59:51.785436 21129 raft_consensus.cc:2393] T 78aa39096868415d9afc2ffcb248e198 P 7a9daf5d42a84615a6383cfec9a10608 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 1c50de89a5a845668696dcb28c3badd6 in current term 5: Already voted for candidate 7a9daf5d42a84615a6383cfec9a10608 in this term.
I20260129 02:59:51.821172 20583 raft_consensus.cc:2468] T fcddab6bca684adf8ea032289a00f90e P 1c50de89a5a845668696dcb28c3badd6 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 974b802141bd4f13beffef2aa03f88f9 in term 8.
I20260129 02:59:51.896934 20583 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:35752 (request call id 1550) took 1291 ms. Trace:
I20260129 02:59:51.897028 20513 raft_consensus.cc:2468] T 7d9494f5584d4d5db8b9adf1c8fc888b P 974b802141bd4f13beffef2aa03f88f9 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 197b880fa39d432a9b472845876bfaf0 in term 3.
W20260129 02:59:51.822270 21102 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.211.2:34517, user_credentials={real_user=slave}} blocked reactor thread for 51809.5us
I20260129 02:59:51.613544 22688 consensus_queue.cc:1059] T 926a250fa5834e88b96ccecbefbf5086 P 197b880fa39d432a9b472845876bfaf0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 }, Status: INVALID_TERM, Last received: 4.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.152s
I20260129 02:59:51.827138 20590 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:35752 (request call id 1565) took 1123 ms. Trace:
W20260129 02:59:51.898942 20555 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260129 02:59:51.826687 20584 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:35752 (request call id 1570) took 1100 ms. Trace:
I20260129 02:59:51.828084 20581 raft_consensus.cc:1275] T 0f20af7fbe5d4db89a6f17948d98912c P 1c50de89a5a845668696dcb28c3badd6 [term 5 FOLLOWER]: Refusing update from remote peer 197b880fa39d432a9b472845876bfaf0: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 5 index: 4. (index mismatch)
I20260129 02:59:51.898869 20590 rpcz_store.cc:276] 0129 02:59:50.703568 (+ 0us) service_pool.cc:168] Inserting onto call queue
0129 02:59:50.909766 (+206198us) service_pool.cc:225] Handling call
0129 02:59:51.826960 (+917194us) spinlock_profiling.cc:243] Waited 308 ms on lock 0x7b540033f754. stack: 00007f8fe044e373 00007f8fe044e270 00007f8fdfda0574 00007f8fe9a5c6b4 00007f8fe9a5c649 00007f8fe34975a1 00007f8fe3531f7a 00007f8fe84e3647 00007f8fe1a73834 00007f8fe1a737c2 00007f8fe1a736f1 00007f8fe1a7366c 00007f8fe1a72922 00007f8fe186d34c 00007f8fe186c786 00007f8fe186c12f
0129 02:59:51.827074 (+ 114us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":707867008}
I20260129 02:59:51.899468 20584 rpcz_store.cc:276] 0129 02:59:50.726104 (+ 0us) service_pool.cc:168] Inserting onto call queue
0129 02:59:51.237478 (+511374us) service_pool.cc:225] Handling call
0129 02:59:51.826502 (+589024us) spinlock_profiling.cc:243] Waited 129 ms on lock 0x7b54000c8354. stack: 00007f8fe044e373 00007f8fe044e270 00007f8fdfda0574 00007f8fe9a5c6b4 00007f8fe9a5c649 00007f8fe34975a1 00007f8fe3531f7a 00007f8fe84e3647 00007f8fe1a73834 00007f8fe1a737c2 00007f8fe1a736f1 00007f8fe1a7366c 00007f8fe1a72922 00007f8fe186d34c 00007f8fe186c786 00007f8fe186c12f
0129 02:59:51.826633 (+ 131us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":296721280}
I20260129 02:59:51.900691 20590 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "add8d4923d1a489e86882c3a25692b95" candidate_uuid: "7a9daf5d42a84615a6383cfec9a10608" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "1c50de89a5a845668696dcb28c3badd6" is_pre_election: true
I20260129 02:59:51.829398 20582 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a5a05fbcbbcf4916bbc01fd1a5b5e90d" candidate_uuid: "974b802141bd4f13beffef2aa03f88f9" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "1c50de89a5a845668696dcb28c3badd6"
I20260129 02:59:51.901980 20582 raft_consensus.cc:3060] T a5a05fbcbbcf4916bbc01fd1a5b5e90d P 1c50de89a5a845668696dcb28c3badd6 [term 6 FOLLOWER]: Advancing to term 7
W20260129 02:59:51.778955 20407 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260129 02:59:51.824298 20585 rpcz_store.cc:269] 0129 02:59:49.457923 (+ 0us) service_pool.cc:168] Inserting onto call queue
0129 02:59:49.617629 (+159706us) service_pool.cc:225] Handling call
0129 02:59:51.823544 (+2205915us) spinlock_profiling.cc:243] Waited 1.82 s on lock 0x7b540010a754. stack: 00007f8fe044e373 00007f8fe044e270 00007f8fdfda0574 00007f8fe9a5c6b4 00007f8fe9a5c649 00007f8fe34975a1 00007f8fe3531f7a 00007f8fe84e3647 00007f8fe1a73834 00007f8fe1a737c2 00007f8fe1a736f1 00007f8fe1a7366c 00007f8fe1a72922 00007f8fe186d34c 00007f8fe186c786 00007f8fe186c12f
0129 02:59:51.823657 (+ 113us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":4184948992}
I20260129 02:59:51.906324 20585 raft_consensus.cc:1240] T fcddab6bca684adf8ea032289a00f90e P 1c50de89a5a845668696dcb28c3badd6 [term 8 FOLLOWER]: Rejecting Update request from peer 197b880fa39d432a9b472845876bfaf0 for earlier term 7. Current term is 8. Ops: [7.5-7.5]
I20260129 02:59:51.907122 20585 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35754 (request call id 1680) took 2195 ms. Trace:
I20260129 02:59:51.908247 20513 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0678b8b468bd43cdbc3e485900844821" candidate_uuid: "7a9daf5d42a84615a6383cfec9a10608" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "974b802141bd4f13beffef2aa03f88f9"
I20260129 02:59:51.909919 20432 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "dc2f4cb752cf4d59b5f6bc689f5d8104" candidate_uuid: "7a9daf5d42a84615a6383cfec9a10608" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "197b880fa39d432a9b472845876bfaf0" is_pre_election: true
I20260129 02:59:51.783145 21128 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:47844 (request call id 311) took 1208 ms. Trace:
W20260129 02:59:51.857887 20441 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:48356 (request call id 300) took 3014 ms (client timeout 2854 ms). Trace:
I20260129 02:59:51.860409 20437 raft_consensus.cc:3060] T 724a236524cb4e7ba3428880681d6cc2 P 197b880fa39d432a9b472845876bfaf0 [term 3 FOLLOWER]: Advancing to term 4
I20260129 02:59:51.797523 22400 raft_consensus.cc:493] T 7a9a69cd82724b1faed1d10aa091fc4a P 974b802141bd4f13beffef2aa03f88f9 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260129 02:59:51.861207 20435 raft_consensus.cc:1275] T 2ac659d732314e1288849f08fe5e074e P 197b880fa39d432a9b472845876bfaf0 [term 3 FOLLOWER]: Refusing update from remote peer 974b802141bd4f13beffef2aa03f88f9: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 3 index: 5. (term mismatch)
I20260129 02:59:51.764312 22831 consensus_queue.cc:1059] T 75f45538dc704d9a8ec4ff8ad9f3ea5b P 7a9daf5d42a84615a6383cfec9a10608 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.055s
I20260129 02:59:51.832541 22742 raft_consensus.cc:2804] T febffe4a8b394368b69b1d164308fb9a P 7a9daf5d42a84615a6383cfec9a10608 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260129 02:59:51.907518 20585 rpcz_store.cc:276] 0129 02:59:49.711460 (+ 0us) service_pool.cc:168] Inserting onto call queue
0129 02:59:51.905537 (+2194077us) service_pool.cc:225] Handling call
0129 02:59:51.906002 (+ 465us) raft_consensus.cc:1471] Updating replica for 1 ops
0129 02:59:51.906821 (+ 819us) raft_consensus.cc:1697] Filling consensus response to leader.
0129 02:59:51.907056 (+ 235us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260129 02:59:51.580185 22736 consensus_queue.cc:1059] T a8479d1fe0bd41e5946186e3a6ca8c7a P 197b880fa39d432a9b472845876bfaf0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 }, Status: INVALID_TERM, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.196s
I20260129 02:59:51.834265 22641 raft_consensus.cc:3060] T 1299723a4476420ab02fd8bfd7aa0645 P 197b880fa39d432a9b472845876bfaf0 [term 6 FOLLOWER]: Advancing to term 8
I20260129 02:59:51.835462 22728 raft_consensus.cc:515] T 76f713c3331540d2a5b76164103e9eb1 P 1c50de89a5a845668696dcb28c3badd6 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:59:51.876308 21127 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34060 (request call id 493) took 3042 ms. Trace:
I20260129 02:59:51.801427 22814 consensus_queue.cc:1059] T e7d83d4ccf684e76acb29a0dc844e1b8 P 1c50de89a5a845668696dcb28c3badd6 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 }, Status: INVALID_TERM, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.014s
I20260129 02:59:51.897473 22707 leader_election.cc:290] T 4e420f90884f4796997b5ea41e49e411 P 974b802141bd4f13beffef2aa03f88f9 [CANDIDATE]: Term 5 election: Requested vote from peers 197b880fa39d432a9b472845876bfaf0 (127.18.211.1:42027), 7a9daf5d42a84615a6383cfec9a10608 (127.18.211.4:40439)
I20260129 02:59:51.898070 20514 raft_consensus.cc:2393] T 4e420f90884f4796997b5ea41e49e411 P 974b802141bd4f13beffef2aa03f88f9 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 7a9daf5d42a84615a6383cfec9a10608 in current term 5: Already voted for candidate 974b802141bd4f13beffef2aa03f88f9 in this term.
I20260129 02:59:51.825904 20512 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57584 (request call id 1291) took 2330 ms. Trace:
W20260129 02:59:51.826578 20405 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.211.3:37735, user_credentials={real_user=slave}} blocked reactor thread for 68966.4us
I20260129 02:59:51.822896 20507 rpcz_store.cc:276] 0129 02:59:50.618087 (+ 0us) service_pool.cc:168] Inserting onto call queue
0129 02:59:51.507487 (+889400us) service_pool.cc:225] Handling call
0129 02:59:51.508462 (+ 975us) raft_consensus.cc:1471] Updating replica for 0 ops
0129 02:59:51.544039 (+ 35577us) raft_consensus.cc:1697] Filling consensus response to leader.
0129 02:59:51.821940 (+277901us) spinlock_profiling.cc:243] Waited 10.5 ms on lock 0x7b540028f354. stack: 00007f8fe044e373 00007f8fe044e270 00007f8fdfda0574 00007f8fe9a5c6b4 00007f8fe9a5c649 00007f8fe34975a1 00007f8fe352ea81 00007f8fe352d4be 00007f8fe84e1f49 00007f8fe1a6ed34 00007f8fe1a6ecc2 00007f8fe1a6ebf1 00007f8fe1a6eb6c 00007f8fe1a6de22 00007f8fe186d34c 00007f8fe186c786
0129 02:59:51.822168 (+ 228us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":24107264}
I20260129 02:59:51.922284 20435 pending_rounds.cc:85] T 2ac659d732314e1288849f08fe5e074e P 197b880fa39d432a9b472845876bfaf0: Aborting all ops after (but not including) 4
I20260129 02:59:51.962896 20435 pending_rounds.cc:107] T 2ac659d732314e1288849f08fe5e074e P 197b880fa39d432a9b472845876bfaf0: Aborting uncommitted NO_OP operation due to leader change: 2.5
I20260129 02:59:51.897310 20583 rpcz_store.cc:276] 0129 02:59:50.605632 (+ 0us) service_pool.cc:168] Inserting onto call queue
0129 02:59:51.346884 (+741252us) service_pool.cc:225] Handling call
0129 02:59:51.896871 (+549987us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20260129 02:59:51.922956 20441 rpcz_store.cc:269] 0129 02:59:48.843311 (+ 0us) service_pool.cc:168] Inserting onto call queue
0129 02:59:48.845895 (+ 2584us) service_pool.cc:225] Handling call
0129 02:59:51.835989 (+2990094us) spinlock_profiling.cc:243] Waited 1.83 s on lock 0x7b54002ae754. stack: 00007f8fe044e373 00007f8fe044e270 00007f8fdfda0574 00007f8fe9a5c6b4 00007f8fe9a5c649 00007f8fe34975a1 00007f8fe3531f7a 00007f8fe84e3647 00007f8fe1a73834 00007f8fe1a737c2 00007f8fe1a736f1 00007f8fe1a7366c 00007f8fe1a72922 00007f8fe186d34c 00007f8fe186c786 00007f8fe186c12f
0129 02:59:51.857807 (+ 21818us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":4217384832}
I20260129 02:59:51.962592 20512 rpcz_store.cc:276] 0129 02:59:49.495659 (+ 0us) service_pool.cc:168] Inserting onto call queue
0129 02:59:50.285920 (+790261us) service_pool.cc:225] Handling call
0129 02:59:50.596666 (+310746us) raft_consensus.cc:1471] Updating replica for 0 ops
0129 02:59:50.597424 (+ 758us) raft_consensus.cc:1697] Filling consensus response to leader.
0129 02:59:51.825709 (+1228285us) spinlock_profiling.cc:243] Waited 281 ms on lock 0x7b540012a050. stack: 00007f8fe044e373 00007f8fe044e270 00007f8fdfda0574 00007f8fe9a5c6b4 00007f8fe9a5c649 00007f8fe34975a1 00007f8fe352d52d 00007f8fe84e1f49 00007f8fe1a6ed34 00007f8fe1a6ecc2 00007f8fe1a6ebf1 00007f8fe1a6eb6c 00007f8fe1a6de22 00007f8fe186d34c 00007f8fe186c786 00007f8fe186c12f
0129 02:59:51.825846 (+ 137us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":646046080}
I20260129 02:59:51.964365 20583 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:48526 (request call id 259) took 1588 ms. Trace:
I20260129 02:59:51.963214 20435 raft_consensus.cc:2889] T 2ac659d732314e1288849f08fe5e074e P 197b880fa39d432a9b472845876bfaf0 [term 3 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260129 02:59:51.916498 20480 leader_election.cc:304] T 0f20af7fbe5d4db89a6f17948d98912c P 974b802141bd4f13beffef2aa03f88f9 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1c50de89a5a845668696dcb28c3badd6, 974b802141bd4f13beffef2aa03f88f9; no voters:
I20260129 02:59:51.890398 22719 raft_consensus.cc:3060] T e33a417cf9c54cf5af73a4ee3aac4d3a P 974b802141bd4f13beffef2aa03f88f9 [term 5 FOLLOWER]: Advancing to term 6
I20260129 02:59:51.979182 20432 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "febffe4a8b394368b69b1d164308fb9a" candidate_uuid: "7a9daf5d42a84615a6383cfec9a10608" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "197b880fa39d432a9b472845876bfaf0" is_pre_election: true
I20260129 02:59:51.931522 20438 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3ccac82a01a2445fa5b0ce6c0fcf2295" candidate_uuid: "7a9daf5d42a84615a6383cfec9a10608" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "197b880fa39d432a9b472845876bfaf0"
I20260129 02:59:51.981312 20438 raft_consensus.cc:3060] T 3ccac82a01a2445fa5b0ce6c0fcf2295 P 197b880fa39d432a9b472845876bfaf0 [term 3 FOLLOWER]: Advancing to term 4
I20260129 02:59:51.934407 20584 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "17d6431cc77a491e8464d68db4081a51" candidate_uuid: "974b802141bd4f13beffef2aa03f88f9" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1c50de89a5a845668696dcb28c3badd6" is_pre_election: true
I20260129 02:59:51.981992 20405 leader_election.cc:304] T 4f8f3a38f6e24ba4857875181f447442 P 197b880fa39d432a9b472845876bfaf0 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 197b880fa39d432a9b472845876bfaf0; no voters: 1c50de89a5a845668696dcb28c3badd6, 974b802141bd4f13beffef2aa03f88f9
I20260129 02:59:51.982620 20584 raft_consensus.cc:2468] T 17d6431cc77a491e8464d68db4081a51 P 1c50de89a5a845668696dcb28c3badd6 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 974b802141bd4f13beffef2aa03f88f9 in term 3.
I20260129 02:59:51.819626 22403 leader_election.cc:290] T 0678b8b468bd43cdbc3e485900844821 P 7a9daf5d42a84615a6383cfec9a10608 [CANDIDATE]: Term 6 election: Requested vote from peers 197b880fa39d432a9b472845876bfaf0 (127.18.211.1:42027), 974b802141bd4f13beffef2aa03f88f9 (127.18.211.2:34517)
W20260129 02:59:51.936162 20556 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.211.2:34517, user_credentials={real_user=slave}} blocked reactor thread for 134519us
I20260129 02:59:51.776196 22738 consensus_queue.cc:260] T 368953c9102643b2a21f99af36b51905 P 974b802141bd4f13beffef2aa03f88f9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 5.4, Last appended by leader: 4, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
W20260129 02:59:51.946591 20553 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.211.1:42027, user_credentials={real_user=slave}} blocked reactor thread for 152256us
I20260129 02:59:51.948346 20433 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0678b8b468bd43cdbc3e485900844821" candidate_uuid: "7a9daf5d42a84615a6383cfec9a10608" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "197b880fa39d432a9b472845876bfaf0" is_pre_election: true
I20260129 02:59:51.860010 22751 consensus_queue.cc:1059] T a644c02b16af47679824ccd060ac10fd P 1c50de89a5a845668696dcb28c3badd6 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false }, Status: INVALID_TERM, Last received: 3.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.082s
I20260129 02:59:51.898669 22688 raft_consensus.cc:3055] T 926a250fa5834e88b96ccecbefbf5086 P 197b880fa39d432a9b472845876bfaf0 [term 5 LEADER]: Stepping down as leader of term 5
I20260129 02:59:51.955206 20434 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "18f30ebaa4174fd28c958e72d0cc7e24" candidate_uuid: "7a9daf5d42a84615a6383cfec9a10608" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "197b880fa39d432a9b472845876bfaf0"
I20260129 02:59:51.964926 20507 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "87d5fe48a44a408aaf03e894a67ecb2f" candidate_uuid: "7a9daf5d42a84615a6383cfec9a10608" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "974b802141bd4f13beffef2aa03f88f9"
W20260129 02:59:51.965873 20480 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.211.3:37735, user_credentials={real_user=slave}} blocked reactor thread for 49474.2us
I20260129 02:59:51.964717 20583 rpcz_store.cc:276] 0129 02:59:50.376324 (+ 0us) service_pool.cc:168] Inserting onto call queue
0129 02:59:51.963524 (+1587200us) service_pool.cc:225] Handling call
0129 02:59:51.963887 (+ 363us) raft_consensus.cc:1471] Updating replica for 0 ops
0129 02:59:51.964071 (+ 184us) raft_consensus.cc:1534] Early marking committed up to index 0
0129 02:59:51.964094 (+ 23us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0129 02:59:51.964133 (+ 39us) raft_consensus.cc:1645] Marking committed up to 7
0129 02:59:51.964162 (+ 29us) raft_consensus.cc:1697] Filling consensus response to leader.
0129 02:59:51.964213 (+ 51us) raft_consensus.cc:1691] UpdateReplicas() finished
0129 02:59:51.964312 (+ 99us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260129 02:59:51.946297 22742 raft_consensus.cc:493] T febffe4a8b394368b69b1d164308fb9a P 7a9daf5d42a84615a6383cfec9a10608 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260129 02:59:51.946043 22348 leader_election.cc:290] T 4408006eb5174de1896b037cf3f18457 P 1c50de89a5a845668696dcb28c3badd6 [CANDIDATE]: Term 3 election: Requested vote from peers 197b880fa39d432a9b472845876bfaf0 (127.18.211.1:42027), 974b802141bd4f13beffef2aa03f88f9 (127.18.211.2:34517)
W20260129 02:59:51.950205 22805 consensus_meta.cc:287] T 3e2d15f8640b407195210c476e09df91 P 1c50de89a5a845668696dcb28c3badd6: Time spent flushing consensus metadata: real 0.545s user 0.008s sys 0.001s
I20260129 02:59:51.946652 22237 consensus_queue.cc:1048] T febffe4a8b394368b69b1d164308fb9a P 197b880fa39d432a9b472845876bfaf0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.057s
I20260129 02:59:51.873677 22737 raft_consensus.cc:515] T fcddab6bca684adf8ea032289a00f90e P 197b880fa39d432a9b472845876bfaf0 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:59:51.960376 20407 leader_election.cc:304] T 7d9494f5584d4d5db8b9adf1c8fc888b P 197b880fa39d432a9b472845876bfaf0 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 197b880fa39d432a9b472845876bfaf0, 974b802141bd4f13beffef2aa03f88f9; no voters:
I20260129 02:59:51.929426 22400 raft_consensus.cc:515] T 7a9a69cd82724b1faed1d10aa091fc4a P 974b802141bd4f13beffef2aa03f88f9 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:59:51.977041 22645 consensus_queue.cc:1059] T a8479d1fe0bd41e5946186e3a6ca8c7a P 197b880fa39d432a9b472845876bfaf0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 }, Status: INVALID_TERM, Last received: 2.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.131s
I20260129 02:59:51.951519 22832 consensus_queue.cc:1059] T 75f45538dc704d9a8ec4ff8ad9f3ea5b P 7a9daf5d42a84615a6383cfec9a10608 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.187s
I20260129 02:59:51.985210 20585 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:48526 (request call id 258) took 1655 ms. Trace:
W20260129 02:59:51.963213 22807 consensus_meta.cc:287] T 0678b8b468bd43cdbc3e485900844821 P 974b802141bd4f13beffef2aa03f88f9: Time spent flushing consensus metadata: real 0.521s user 0.009s sys 0.003s
I20260129 02:59:51.998752 20441 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00e79d8f25a34676b1d866eb2068a2b2" candidate_uuid: "7a9daf5d42a84615a6383cfec9a10608" candidate_term: 6 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "197b880fa39d432a9b472845876bfaf0"
I20260129 02:59:51.999349 20507 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2ac659d732314e1288849f08fe5e074e" candidate_uuid: "197b880fa39d432a9b472845876bfaf0" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "974b802141bd4f13beffef2aa03f88f9" is_pre_election: true
I20260129 02:59:51.961901 22708 raft_consensus.cc:2804] T 75be4bd6bd45474795fd472eff2953ac P 1c50de89a5a845668696dcb28c3badd6 [term 9 FOLLOWER]: Leader election won for term 9
W20260129 02:59:52.224383 21099 thread.cc:641] dns-resolver [worker] (thread pool) Time spent creating pthread: real 0.520s user 0.005s sys 0.005s
I20260129 02:59:52.082605 20440 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55676 (request call id 1278) took 1779 ms. Trace:
I20260129 02:59:51.963056 22554 raft_consensus.cc:2804] T 42d27a28d89b409e8807ec0efd97af69 P 1c50de89a5a845668696dcb28c3badd6 [term 6 FOLLOWER]: Leader election won for term 6
W20260129 02:59:52.092876 20556 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.211.2:34517, user_credentials={real_user=slave}} blocked reactor thread for 108256us
I20260129 02:59:52.108281 20586 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "926a250fa5834e88b96ccecbefbf5086" candidate_uuid: "974b802141bd4f13beffef2aa03f88f9" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "1c50de89a5a845668696dcb28c3badd6"
I20260129 02:59:52.108352 21134 raft_consensus.cc:1275] T 75b0b6e82ddc4f92b1fe1d94f9a0b73b P 7a9daf5d42a84615a6383cfec9a10608 [term 3 FOLLOWER]: Refusing update from remote peer 1c50de89a5a845668696dcb28c3badd6: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 3 index: 6. (index mismatch)
I20260129 02:59:52.241466 20586 raft_consensus.cc:3060] T 926a250fa5834e88b96ccecbefbf5086 P 1c50de89a5a845668696dcb28c3badd6 [term 5 FOLLOWER]: Advancing to term 6
I20260129 02:59:52.134545 20582 raft_consensus.cc:2468] T a5a05fbcbbcf4916bbc01fd1a5b5e90d P 1c50de89a5a845668696dcb28c3badd6 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 974b802141bd4f13beffef2aa03f88f9 in term 7.
I20260129 02:59:52.000420 20437 raft_consensus.cc:2468] T 724a236524cb4e7ba3428880681d6cc2 P 197b880fa39d432a9b472845876bfaf0 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1c50de89a5a845668696dcb28c3badd6 in term 4.
W20260129 02:59:52.176645 20482 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260129 02:59:52.096781 20480 leader_election.cc:304] T fcddab6bca684adf8ea032289a00f90e P 974b802141bd4f13beffef2aa03f88f9 [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1c50de89a5a845668696dcb28c3badd6, 974b802141bd4f13beffef2aa03f88f9; no voters: 197b880fa39d432a9b472845876bfaf0
I20260129 02:59:52.023286 20512 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "add8d4923d1a489e86882c3a25692b95" candidate_uuid: "7a9daf5d42a84615a6383cfec9a10608" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "974b802141bd4f13beffef2aa03f88f9" is_pre_election: true
W20260129 02:59:52.093075 20553 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.211.1:42027, user_credentials={real_user=slave}} blocked reactor thread for 107974us
I20260129 02:59:52.244328 20512 raft_consensus.cc:2468] T add8d4923d1a489e86882c3a25692b95 P 974b802141bd4f13beffef2aa03f88f9 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7a9daf5d42a84615a6383cfec9a10608 in term 3.
I20260129 02:59:52.220643 20511 raft_consensus.cc:2468] T cedce331e1584ac8b636c9b8a72b2ba0 P 974b802141bd4f13beffef2aa03f88f9 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 197b880fa39d432a9b472845876bfaf0 in term 6.
I20260129 02:59:52.012650 21133 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "add8d4923d1a489e86882c3a25692b95" candidate_uuid: "974b802141bd4f13beffef2aa03f88f9" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "7a9daf5d42a84615a6383cfec9a10608" is_pre_election: true
I20260129 02:59:52.006789 20590 raft_consensus.cc:1240] T 75b0b6e82ddc4f92b1fe1d94f9a0b73b P 1c50de89a5a845668696dcb28c3badd6 [term 3 LEADER]: Rejecting Update request from peer 7a9daf5d42a84615a6383cfec9a10608 for earlier term 2. Current term is 3. Ops: []
W20260129 02:59:52.212316 20479 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.211.1:42027, user_credentials={real_user=slave}} blocked reactor thread for 430395us
I20260129 02:59:52.108912 20515 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:57584 (request call id 1315) took 1490 ms. Trace:
W20260129 02:59:52.027968 20405 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.211.3:37735, user_credentials={real_user=slave}} blocked reactor thread for 46088.6us
I20260129 02:59:52.246665 20479 leader_election.cc:304] T 36ff33be666a4c82954731691dbcf327 P 974b802141bd4f13beffef2aa03f88f9 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 197b880fa39d432a9b472845876bfaf0, 974b802141bd4f13beffef2aa03f88f9; no voters: 1c50de89a5a845668696dcb28c3badd6
I20260129 02:59:52.233078 20507 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a644c02b16af47679824ccd060ac10fd" candidate_uuid: "7a9daf5d42a84615a6383cfec9a10608" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "974b802141bd4f13beffef2aa03f88f9"
I20260129 02:59:52.004201 20583 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a644c02b16af47679824ccd060ac10fd" candidate_uuid: "7a9daf5d42a84615a6383cfec9a10608" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "1c50de89a5a845668696dcb28c3badd6"
W20260129 02:59:52.149240 20481 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260129 02:59:52.246475 20515 rpcz_store.cc:276] 0129 02:59:50.618898 (+ 0us) service_pool.cc:168] Inserting onto call queue
0129 02:59:51.072620 (+453722us) service_pool.cc:225] Handling call
0129 02:59:52.108655 (+1036035us) spinlock_profiling.cc:243] Waited 250 ms on lock 0x7b5400229054. stack: 00007f8fe044e373 00007f8fe044e270 00007f8fdfda0574 00007f8fe9a5c6b4 00007f8fe9a5c649 00007f8fe34975a1 00007f8fe3531f7a 00007f8fe84e3647 00007f8fe1a73834 00007f8fe1a737c2 00007f8fe1a736f1 00007f8fe1a7366c 00007f8fe1a72922 00007f8fe186d34c 00007f8fe186c786 00007f8fe186c12f
0129 02:59:52.108829 (+ 174us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":573958400}
I20260129 02:59:52.248301 20507 raft_consensus.cc:3060] T a644c02b16af47679824ccd060ac10fd P 974b802141bd4f13beffef2aa03f88f9 [term 3 FOLLOWER]: Advancing to term 4
I20260129 02:59:52.245983 21133 raft_consensus.cc:2468] T add8d4923d1a489e86882c3a25692b95 P 7a9daf5d42a84615a6383cfec9a10608 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 974b802141bd4f13beffef2aa03f88f9 in term 3.
W20260129 02:59:52.246583 21099 thread.cc:608] dns-resolver [worker] (thread pool) Time spent starting thread: real 0.542s user 0.008s sys 0.005s
W20260129 02:59:52.086989 20514 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:48544 (request call id 214) took 1381 ms (client timeout 1831 ms). Trace:
I20260129 02:59:52.247745 20405 leader_election.cc:304] T 913fc7d751ed4b7388d889aac30a5f9b P 197b880fa39d432a9b472845876bfaf0 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 197b880fa39d432a9b472845876bfaf0, 1c50de89a5a845668696dcb28c3badd6; no voters: 974b802141bd4f13beffef2aa03f88f9
I20260129 02:59:52.233299 20585 rpcz_store.cc:276] 0129 02:59:50.329665 (+ 0us) service_pool.cc:168] Inserting onto call queue
0129 02:59:51.951664 (+1621999us) service_pool.cc:225] Handling call
0129 02:59:51.984736 (+ 33072us) raft_consensus.cc:1471] Updating replica for 0 ops
0129 02:59:51.984921 (+ 185us) raft_consensus.cc:1534] Early marking committed up to index 0
0129 02:59:51.984943 (+ 22us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0129 02:59:51.984984 (+ 41us) raft_consensus.cc:1645] Marking committed up to 6
0129 02:59:51.985018 (+ 34us) raft_consensus.cc:1697] Filling consensus response to leader.
0129 02:59:51.985069 (+ 51us) raft_consensus.cc:1691] UpdateReplicas() finished
0129 02:59:51.985160 (+ 91us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260129 02:59:52.114930 20587 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35754 (request call id 1663) took 2569 ms. Trace:
I20260129 02:59:52.031034 20438 raft_consensus.cc:2468] T 3ccac82a01a2445fa5b0ce6c0fcf2295 P 197b880fa39d432a9b472845876bfaf0 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7a9daf5d42a84615a6383cfec9a10608 in term 4.
I20260129 02:59:52.227900 20441 raft_consensus.cc:3060] T 00e79d8f25a34676b1d866eb2068a2b2 P 197b880fa39d432a9b472845876bfaf0 [term 5 FOLLOWER]: Advancing to term 6
I20260129 02:59:51.984706 21129 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4e420f90884f4796997b5ea41e49e411" candidate_uuid: "974b802141bd4f13beffef2aa03f88f9" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "7a9daf5d42a84615a6383cfec9a10608"
I20260129 02:59:52.279109 22647 raft_consensus.cc:2804] T 7d9494f5584d4d5db8b9adf1c8fc888b P 197b880fa39d432a9b472845876bfaf0 [term 3 FOLLOWER]: Leader election won for term 3
I20260129 02:59:52.279338 20508 raft_consensus.cc:2468] T e7d83d4ccf684e76acb29a0dc844e1b8 P 974b802141bd4f13beffef2aa03f88f9 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 197b880fa39d432a9b472845876bfaf0 in term 5.
I20260129 02:59:52.279726 22647 raft_consensus.cc:697] T 7d9494f5584d4d5db8b9adf1c8fc888b P 197b880fa39d432a9b472845876bfaf0 [term 3 LEADER]: Becoming Leader. State: Replica: 197b880fa39d432a9b472845876bfaf0, State: Running, Role: LEADER
W20260129 02:59:52.001341 22839 log.cc:927] Time spent T 2ac659d732314e1288849f08fe5e074e P 7a9daf5d42a84615a6383cfec9a10608: Append to log took a long time: real 0.100s user 0.000s sys 0.000s
I20260129 02:59:52.112002 20581 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35754 (request call id 1678) took 2401 ms. Trace:
I20260129 02:59:52.243191 20582 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "add8d4923d1a489e86882c3a25692b95" candidate_uuid: "974b802141bd4f13beffef2aa03f88f9" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "1c50de89a5a845668696dcb28c3badd6" is_pre_election: true
I20260129 02:59:52.246498 20511 raft_consensus.cc:1240] T 75be4bd6bd45474795fd472eff2953ac P 974b802141bd4f13beffef2aa03f88f9 [term 9 FOLLOWER]: Rejecting Update request from peer 197b880fa39d432a9b472845876bfaf0 for earlier term 8. Current term is 9. Ops: []
W20260129 02:59:52.276510 21099 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.211.1:42027, user_credentials={real_user=slave}} blocked reactor thread for 572337us
I20260129 02:59:52.251653 20590 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:48526 (request call id 257) took 2327 ms. Trace:
W20260129 02:59:52.251470 20439 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.637s user 0.023s sys 0.034s
I20260129 02:59:52.300843 20583 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a8479d1fe0bd41e5946186e3a6ca8c7a" candidate_uuid: "974b802141bd4f13beffef2aa03f88f9" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "1c50de89a5a845668696dcb28c3badd6" is_pre_election: true
W20260129 02:59:52.301154 20439 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.687s user 0.025s sys 0.036s
I20260129 02:59:52.301600 20583 raft_consensus.cc:2468] T a8479d1fe0bd41e5946186e3a6ca8c7a P 1c50de89a5a845668696dcb28c3badd6 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 974b802141bd4f13beffef2aa03f88f9 in term 4.
I20260129 02:59:51.983809 22815 raft_consensus.cc:2804] T 0f20af7fbe5d4db89a6f17948d98912c P 974b802141bd4f13beffef2aa03f88f9 [term 5 FOLLOWER]: Leader pre-election won for term 6
W20260129 02:59:52.101985 22745 consensus_meta.cc:287] T 505580c93c694bbb8ea96fdf9b17ab46 P 974b802141bd4f13beffef2aa03f88f9: Time spent flushing consensus metadata: real 0.694s user 0.007s sys 0.002s
W20260129 02:59:52.004920 20509 consensus_meta.cc:287] T c934d31efd3f4e5195605ddc354d14e1 P 974b802141bd4f13beffef2aa03f88f9: Time spent flushing consensus metadata: real 0.921s user 0.009s sys 0.001s
I20260129 02:59:52.231812 20440 rpcz_store.cc:276] 0129 02:59:50.303045 (+ 0us) service_pool.cc:168] Inserting onto call queue
0129 02:59:50.649317 (+346272us) service_pool.cc:225] Handling call
0129 02:59:50.649625 (+ 308us) raft_consensus.cc:1471] Updating replica for 1 ops
0129 02:59:51.560256 (+910631us) raft_consensus.cc:1534] Early marking committed up to index 0
0129 02:59:51.560285 (+ 29us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0129 02:59:51.560782 (+ 497us) log.cc:844] Serialized 26 byte log entry
0129 02:59:51.949831 (+389049us) raft_consensus.cc:1645] Marking committed up to 5
0129 02:59:51.949880 (+ 49us) raft_consensus.cc:1697] Filling consensus response to leader.
0129 02:59:51.949957 (+ 77us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0129 02:59:52.082349 (+132392us) raft_consensus.cc:1685] finished
0129 02:59:52.082373 (+ 24us) raft_consensus.cc:1691] UpdateReplicas() finished
0129 02:59:52.082533 (+ 160us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":910259,"raft.queue_time_us":910530,"raft.run_cpu_time_us":76,"raft.run_wall_time_us":76,"thread_start_us":388666,"threads_started":1,"wal-append.queue_time_us":519942}
W20260129 02:59:52.277082 20514 rpcz_store.cc:269] 0129 02:59:50.705687 (+ 0us) service_pool.cc:168] Inserting onto call queue
0129 02:59:50.921141 (+215454us) service_pool.cc:225] Handling call
0129 02:59:52.086773 (+1165632us) spinlock_profiling.cc:243] Waited 827 ms on lock 0x7b54003cdcd4. stack: 00007f8fe044e373 00007f8fe044e270 00007f8fdfda0574 00007f8fe9a5c6b4 00007f8fe9a5c649 00007f8fe34975a1 00007f8fe3531f7a 00007f8fe84e3647 00007f8fe1a73834 00007f8fe1a737c2 00007f8fe1a736f1 00007f8fe1a7366c 00007f8fe1a72922 00007f8fe186d34c 00007f8fe186c786 00007f8fe186c12f
0129 02:59:52.086927 (+ 154us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1902152576}
I20260129 02:59:52.302562 20590 rpcz_store.cc:276] 0129 02:59:49.924484 (+ 0us) service_pool.cc:168] Inserting onto call queue
0129 02:59:51.905644 (+1981160us) service_pool.cc:225] Handling call
0129 02:59:51.906087 (+ 443us) raft_consensus.cc:1471] Updating replica for 0 ops
0129 02:59:52.006602 (+100515us) spinlock_profiling.cc:243] Waited 34.4 ms on lock 0x7b54001a84a0. stack: 00007f8fe044e373 00007f8fe044e270 00007f8fdfda0574 00007f8fe9a5c6b4 00007f8fe9a5c649 00007f8fe34975a1 00007f8fe34a5df0 00007f8fe352fe40 00007f8fe3530eb1 00007f8fe352da0f 00007f8fe352d4be 00007f8fe84e1f49 00007f8fe1a6ed34 00007f8fe1a6ecc2 00007f8fe1a6ebf1 00007f8fe1a6eb6c
0129 02:59:52.251403 (+244801us) raft_consensus.cc:1697] Filling consensus response to leader.
0129 02:59:52.251598 (+ 195us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":79017472}
W20260129 02:59:51.996376 22830 log.cc:927] Time spent T dc2f4cb752cf4d59b5f6bc689f5d8104 P 7a9daf5d42a84615a6383cfec9a10608: Append to log took a long time: real 0.051s user 0.000s sys 0.000s
I20260129 02:59:52.115396 22758 raft_consensus.cc:3055] T 76f713c3331540d2a5b76164103e9eb1 P 197b880fa39d432a9b472845876bfaf0 [term 3 LEADER]: Stepping down as leader of term 3
W20260129 02:59:52.290599 20481 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260129 02:59:51.922170 21128 rpcz_store.cc:276] 0129 02:59:50.575081 (+ 0us) service_pool.cc:168] Inserting onto call queue
0129 02:59:50.733530 (+158449us) service_pool.cc:225] Handling call
0129 02:59:51.611531 (+878001us) spinlock_profiling.cc:243] Waited 2.16 ms on lock 0x7b5800111f98. stack: 00007f8fe044e373 00007f8fe044e270 00007f8fdfda0574 00007f8fe9a5c6b4 00007f8fe9a5c649 00007f8fe02c08b1 00007f8fe0442db1 00007f8fe0442c62 00007f8fe92eddd9 00007f8fe846d260 00007f8fe8535dbb 00007f8fe8535e41 00007f8fe84f7af0 00007f8fe84e2459 00007f8fe84e1f02 00007f8fe1a6ed34
0129 02:59:51.611767 (+ 236us) raft_consensus.cc:1471] Updating replica for 0 ops
0129 02:59:51.763210 (+151443us) raft_consensus.cc:1697] Filling consensus response to leader.
0129 02:59:51.781205 (+ 17995us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":4972928}
W20260129 02:59:52.262735 20480 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.211.3:37735, user_credentials={real_user=slave}} blocked reactor thread for 166032us
I20260129 02:59:51.989148 22688 raft_consensus.cc:740] T 926a250fa5834e88b96ccecbefbf5086 P 197b880fa39d432a9b472845876bfaf0 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 197b880fa39d432a9b472845876bfaf0, State: Running, Role: LEADER
I20260129 02:59:52.297753 20581 rpcz_store.cc:276] 0129 02:59:49.710989 (+ 0us) service_pool.cc:168] Inserting onto call queue
0129 02:59:51.771411 (+2060422us) service_pool.cc:225] Handling call
0129 02:59:51.827730 (+ 56319us) raft_consensus.cc:1471] Updating replica for 0 ops
0129 02:59:51.899749 (+ 72019us) raft_consensus.cc:1697] Filling consensus response to leader.
0129 02:59:52.111726 (+211977us) spinlock_profiling.cc:243] Waited 34.1 ms on lock 0x7b54000c8350. stack: 00007f8fe044e373 00007f8fe044e270 00007f8fdfda0574 00007f8fe9a5c6b4 00007f8fe9a5c649 00007f8fe34975a1 00007f8fe352d52d 00007f8fe84e1f49 00007f8fe1a6ed34 00007f8fe1a6ecc2 00007f8fe1a6ebf1 00007f8fe1a6eb6c 00007f8fe1a6de22 00007f8fe186d34c 00007f8fe186c786 00007f8fe186c12f
0129 02:59:52.111923 (+ 197us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":78461184}
I20260129 02:59:52.008508 22320 raft_consensus.cc:3055] T a644c02b16af47679824ccd060ac10fd P 1c50de89a5a845668696dcb28c3badd6 [term 3 LEADER]: Stepping down as leader of term 3
I20260129 02:59:51.999315 22742 raft_consensus.cc:3060] T febffe4a8b394368b69b1d164308fb9a P 7a9daf5d42a84615a6383cfec9a10608 [term 3 FOLLOWER]: Advancing to term 4
I20260129 02:59:52.306255 22684 raft_consensus.cc:2749] T 368953c9102643b2a21f99af36b51905 P 197b880fa39d432a9b472845876bfaf0 [term 6 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260129 02:59:52.240520 22807 raft_consensus.cc:515] T 0678b8b468bd43cdbc3e485900844821 P 974b802141bd4f13beffef2aa03f88f9 [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } }
I20260129 02:59:52.001663 22694 leader_election.cc:290] T add8d4923d1a489e86882c3a25692b95 P 974b802141bd4f13beffef2aa03f88f9 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3:37735), 7a9daf5d42a84615a6383cfec9a10608 (127.18.211.4:40439)
I20260129 02:59:52.112800 22540 leader_election.cc:290] T 4b1876f9a70b49edb3aa74105e66d3b3 P 197b880fa39d432a9b472845876bfaf0 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 974b802141bd4f13beffef2aa03f88f9 (127.18.211.2:34517), 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3:37735)
I20260129 02:59:52.304663 20512 raft_consensus.cc:1240] T 4e420f90884f4796997b5ea41e49e411 P 974b802141bd4f13beffef2aa03f88f9 [term 5 FOLLOWER]: Rejecting Update request from peer 197b880fa39d432a9b472845876bfaf0 for earlier term 4. Current term is 5. Ops: []
I20260129 02:59:52.091102 22160 leader_election.cc:290] T a8479d1fe0bd41e5946186e3a6ca8c7a P 974b802141bd4f13beffef2aa03f88f9 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 197b880fa39d432a9b472845876bfaf0 (127.18.211.1:42027), 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3:37735)
I20260129 02:59:51.988902 22667 leader_election.cc:290] T a644c02b16af47679824ccd060ac10fd P 7a9daf5d42a84615a6383cfec9a10608 [CANDIDATE]: Term 4 election: Requested vote from peers 974b802141bd4f13beffef2aa03f88f9 (127.18.211.2:34517), 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3:37735)
I20260129 02:59:51.998492 22736 raft_consensus.cc:2749] T 4f8f3a38f6e24ba4857875181f447442 P 197b880fa39d432a9b472845876bfaf0 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260129 02:59:52.084717 22231 raft_consensus.cc:3055] T a8479d1fe0bd41e5946186e3a6ca8c7a P 197b880fa39d432a9b472845876bfaf0 [term 3 LEADER]: Stepping down as leader of term 3
W20260129 02:59:52.305922 21099 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260129 02:59:52.092752 22640 raft_consensus.cc:3055] T e7d83d4ccf684e76acb29a0dc844e1b8 P 1c50de89a5a845668696dcb28c3badd6 [term 4 LEADER]: Stepping down as leader of term 4
I20260129 02:59:51.978468 22656 raft_consensus.cc:515] T 505580c93c694bbb8ea96fdf9b17ab46 P 1c50de89a5a845668696dcb28c3badd6 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } }
I20260129 02:59:52.118455 22581 raft_consensus.cc:3055] T a4c1226b987b468fa84dfc6ba0e2dbe1 P 1c50de89a5a845668696dcb28c3badd6 [term 3 LEADER]: Stepping down as leader of term 3
I20260129 02:59:52.310563 22745 raft_consensus.cc:515] T 505580c93c694bbb8ea96fdf9b17ab46 P 974b802141bd4f13beffef2aa03f88f9 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } }
I20260129 02:59:52.386514 20513 raft_consensus.cc:2393] T 0678b8b468bd43cdbc3e485900844821 P 974b802141bd4f13beffef2aa03f88f9 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 7a9daf5d42a84615a6383cfec9a10608 in current term 6: Already voted for candidate 974b802141bd4f13beffef2aa03f88f9 in this term.
I20260129 02:59:52.097776 22728 leader_election.cc:290] T 76f713c3331540d2a5b76164103e9eb1 P 1c50de89a5a845668696dcb28c3badd6 [CANDIDATE]: Term 5 election: Requested vote from peers 197b880fa39d432a9b472845876bfaf0 (127.18.211.1:42027), 974b802141bd4f13beffef2aa03f88f9 (127.18.211.2:34517)
I20260129 02:59:52.117942 22425 raft_consensus.cc:3055] T 7d9494f5584d4d5db8b9adf1c8fc888b P 1c50de89a5a845668696dcb28c3badd6 [term 2 LEADER]: Stepping down as leader of term 2
I20260129 02:59:52.089306 21132 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34046 (request call id 470) took 1430 ms. Trace:
I20260129 02:59:52.231351 22445 raft_consensus.cc:697] T 75f45538dc704d9a8ec4ff8ad9f3ea5b P 974b802141bd4f13beffef2aa03f88f9 [term 3 LEADER]: Becoming Leader. State: Replica: 974b802141bd4f13beffef2aa03f88f9, State: Running, Role: LEADER
I20260129 02:59:52.111065 22654 leader_election.cc:290] T 3906b080a5b34a74b8b64128635e8997 P 1c50de89a5a845668696dcb28c3badd6 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 197b880fa39d432a9b472845876bfaf0 (127.18.211.1:42027), 974b802141bd4f13beffef2aa03f88f9 (127.18.211.2:34517)
I20260129 02:59:52.083885 22786 raft_consensus.cc:3055] T 4f8f3a38f6e24ba4857875181f447442 P 1c50de89a5a845668696dcb28c3badd6 [term 4 LEADER]: Stepping down as leader of term 4
I20260129 02:59:52.309453 20590 raft_consensus.cc:1275] T 8055232cf0004d40a85f3bf983fe00a9 P 1c50de89a5a845668696dcb28c3badd6 [term 3 FOLLOWER]: Refusing update from remote peer 974b802141bd4f13beffef2aa03f88f9: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20260129 02:59:52.412539 20590 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35752 (request call id 1563) took 1730 ms. Trace:
I20260129 02:59:52.412885 20590 rpcz_store.cc:276] 0129 02:59:50.682657 (+ 0us) service_pool.cc:168] Inserting onto call queue
0129 02:59:52.308717 (+1626060us) service_pool.cc:225] Handling call
0129 02:59:52.309019 (+ 302us) raft_consensus.cc:1471] Updating replica for 0 ops
0129 02:59:52.412234 (+103215us) raft_consensus.cc:1697] Filling consensus response to leader.
0129 02:59:52.412471 (+ 237us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260129 02:59:52.283542 21129 raft_consensus.cc:3060] T 4e420f90884f4796997b5ea41e49e411 P 7a9daf5d42a84615a6383cfec9a10608 [term 4 FOLLOWER]: Advancing to term 5
I20260129 02:59:52.280696 22647 consensus_queue.cc:237] T 7d9494f5584d4d5db8b9adf1c8fc888b P 197b880fa39d432a9b472845876bfaf0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:59:52.279407 22644 raft_consensus.cc:2804] T 913fc7d751ed4b7388d889aac30a5f9b P 197b880fa39d432a9b472845876bfaf0 [term 3 FOLLOWER]: Leader election won for term 3
I20260129 02:59:52.393385 22231 raft_consensus.cc:740] T a8479d1fe0bd41e5946186e3a6ca8c7a P 197b880fa39d432a9b472845876bfaf0 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 197b880fa39d432a9b472845876bfaf0, State: Running, Role: LEADER
I20260129 02:59:52.420723 22644 raft_consensus.cc:697] T 913fc7d751ed4b7388d889aac30a5f9b P 197b880fa39d432a9b472845876bfaf0 [term 3 LEADER]: Becoming Leader. State: Replica: 197b880fa39d432a9b472845876bfaf0, State: Running, Role: LEADER
I20260129 02:59:51.954742 21127 rpcz_store.cc:276] 0129 02:59:48.830833 (+ 0us) service_pool.cc:168] Inserting onto call queue
0129 02:59:48.835086 (+ 4253us) service_pool.cc:225] Handling call
0129 02:59:49.480806 (+645720us) raft_consensus.cc:1471] Updating replica for 0 ops
0129 02:59:49.569728 (+ 88922us) raft_consensus.cc:1534] Early marking committed up to index 7
0129 02:59:49.939517 (+369789us) log.cc:844] Serialized 14 byte log entry
0129 02:59:50.321749 (+382232us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0129 02:59:50.334232 (+ 12483us) raft_consensus.cc:1645] Marking committed up to 7
0129 02:59:50.339229 (+ 4997us) raft_consensus.cc:1697] Filling consensus response to leader.
0129 02:59:51.859563 (+1520334us) spinlock_profiling.cc:243] Waited 78.3 ms on lock 0x7b540068ef54. stack: 00007f8fe044e373 00007f8fe044e270 00007f8fdfda0574 00007f8fe9a5c6b4 00007f8fe9a5c649 00007f8fe34975a1 00007f8fe352ea81 00007f8fe352d4be 00007f8fe84e1f49 00007f8fe1a6ed34 00007f8fe1a6ecc2 00007f8fe1a6ebf1 00007f8fe1a6eb6c 00007f8fe1a6de22 00007f8fe186d34c 00007f8fe186c786
0129 02:59:51.864580 (+ 5017us) raft_consensus.cc:1691] UpdateReplicas() finished
0129 02:59:51.872767 (+ 8187us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":728553,"prepare.run_cpu_time_us":58,"prepare.run_wall_time_us":7116,"raft.queue_time_us":417,"raft.run_cpu_time_us":185,"raft.run_wall_time_us":21093,"spinlock_wait_cycles":180018816,"thread_start_us":719638,"threads_started":2,"wal-append.queue_time_us":401040,"wal-append.run_cpu_time_us":998,"wal-append.run_wall_time_us":1002108}
I20260129 02:59:52.421130 22231 consensus_queue.cc:260] T a8479d1fe0bd41e5946186e3a6ca8c7a P 197b880fa39d432a9b472845876bfaf0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.3, Last appended by leader: 3, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:59:52.421661 22644 consensus_queue.cc:237] T 913fc7d751ed4b7388d889aac30a5f9b P 197b880fa39d432a9b472845876bfaf0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:59:52.422238 22231 raft_consensus.cc:3060] T a8479d1fe0bd41e5946186e3a6ca8c7a P 197b880fa39d432a9b472845876bfaf0 [term 3 FOLLOWER]: Advancing to term 4
I20260129 02:59:52.422964 22737 leader_election.cc:290] T fcddab6bca684adf8ea032289a00f90e P 197b880fa39d432a9b472845876bfaf0 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 974b802141bd4f13beffef2aa03f88f9 (127.18.211.2:34517), 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3:37735)
I20260129 02:59:52.420007 22403 consensus_queue.cc:1059] T 42d27a28d89b409e8807ec0efd97af69 P 7a9daf5d42a84615a6383cfec9a10608 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 }, Status: INVALID_TERM, Last received: 4.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.004s
I20260129 02:59:52.384133 22688 consensus_queue.cc:260] T 926a250fa5834e88b96ccecbefbf5086 P 197b880fa39d432a9b472845876bfaf0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.4, Last appended by leader: 4, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:59:52.424568 22403 raft_consensus.cc:3055] T 42d27a28d89b409e8807ec0efd97af69 P 7a9daf5d42a84615a6383cfec9a10608 [term 5 LEADER]: Stepping down as leader of term 5
I20260129 02:59:52.424863 22688 raft_consensus.cc:3060] T 926a250fa5834e88b96ccecbefbf5086 P 197b880fa39d432a9b472845876bfaf0 [term 5 FOLLOWER]: Advancing to term 6
I20260129 02:59:52.424983 22403 raft_consensus.cc:740] T 42d27a28d89b409e8807ec0efd97af69 P 7a9daf5d42a84615a6383cfec9a10608 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 7a9daf5d42a84615a6383cfec9a10608, State: Running, Role: LEADER
I20260129 02:59:52.426018 22403 consensus_queue.cc:260] T 42d27a28d89b409e8807ec0efd97af69 P 7a9daf5d42a84615a6383cfec9a10608 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 5.7, Last appended by leader: 7, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } }
I20260129 02:59:52.426466 22445 consensus_queue.cc:237] T 75f45538dc704d9a8ec4ff8ad9f3ea5b P 974b802141bd4f13beffef2aa03f88f9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } }
I20260129 02:59:52.427681 22403 raft_consensus.cc:3060] T 42d27a28d89b409e8807ec0efd97af69 P 7a9daf5d42a84615a6383cfec9a10608 [term 5 FOLLOWER]: Advancing to term 6
I20260129 02:59:52.356916 22719 raft_consensus.cc:515] T e33a417cf9c54cf5af73a4ee3aac4d3a P 974b802141bd4f13beffef2aa03f88f9 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:59:52.431212 22719 leader_election.cc:290] T e33a417cf9c54cf5af73a4ee3aac4d3a P 974b802141bd4f13beffef2aa03f88f9 [CANDIDATE]: Term 6 election: Requested vote from peers 197b880fa39d432a9b472845876bfaf0 (127.18.211.1:42027), 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3:37735)
I20260129 02:59:52.385128 22400 leader_election.cc:290] T 7a9a69cd82724b1faed1d10aa091fc4a P 974b802141bd4f13beffef2aa03f88f9 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 197b880fa39d432a9b472845876bfaf0 (127.18.211.1:42027), 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3:37735)
I20260129 02:59:52.312513 20335 rpcz_store.cc:275] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:55364 (request call id 260) took 11074 ms. Trace:
W20260129 02:59:52.338904 21101 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260129 02:59:52.314417 20583 raft_consensus.cc:1240] T b189f08e16a74a11980cc55d99359a0a P 1c50de89a5a845668696dcb28c3badd6 [term 6 FOLLOWER]: Rejecting Update request from peer 974b802141bd4f13beffef2aa03f88f9 for earlier term 5. Current term is 6. Ops: []
I20260129 02:59:52.269850 20479 leader_election.cc:304] T c2633a6fc34c44579de348d208b0c01e P 974b802141bd4f13beffef2aa03f88f9 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 974b802141bd4f13beffef2aa03f88f9; no voters: 197b880fa39d432a9b472845876bfaf0, 1c50de89a5a845668696dcb28c3badd6
I20260129 02:59:52.296916 22687 raft_consensus.cc:2804] T fcddab6bca684adf8ea032289a00f90e P 974b802141bd4f13beffef2aa03f88f9 [term 8 FOLLOWER]: Leader election won for term 8
I20260129 02:59:52.349743 20480 leader_election.cc:304] T 17d6431cc77a491e8464d68db4081a51 P 974b802141bd4f13beffef2aa03f88f9 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1c50de89a5a845668696dcb28c3badd6, 974b802141bd4f13beffef2aa03f88f9; no voters:
I20260129 02:59:52.440296 20441 raft_consensus.cc:2468] T 00e79d8f25a34676b1d866eb2068a2b2 P 197b880fa39d432a9b472845876bfaf0 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7a9daf5d42a84615a6383cfec9a10608 in term 6.
I20260129 02:59:52.003377 22184 raft_consensus.cc:515] T 4408006eb5174de1896b037cf3f18457 P 974b802141bd4f13beffef2aa03f88f9 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:59:52.457113 22184 leader_election.cc:290] T 4408006eb5174de1896b037cf3f18457 P 974b802141bd4f13beffef2aa03f88f9 [CANDIDATE]: Term 3 election: Requested vote from peers 197b880fa39d432a9b472845876bfaf0 (127.18.211.1:42027), 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3:37735)
I20260129 02:59:52.439831 20335 rpcz_store.cc:276] 0129 02:59:41.232267 (+ 0us) service_pool.cc:168] Inserting onto call queue
0129 02:59:41.232493 (+ 226us) service_pool.cc:225] Handling call
0129 02:59:52.306835 (+11074342us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0129 02:59:48.633537 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0129 02:59:48.786902 (+153365us) write_op.cc:432] Acquiring schema lock in shared mode
0129 02:59:48.786944 (+ 42us) write_op.cc:435] Acquired schema lock
0129 02:59:48.786968 (+ 24us) tablet.cc:660] Decoding operations
0129 02:59:49.025301 (+238333us) write_op.cc:620] Acquiring the partition lock for write op
0129 02:59:49.118990 (+ 93689us) write_op.cc:641] Partition lock acquired for write op
0129 02:59:49.223334 (+104344us) tablet.cc:683] Acquiring locks for 3 operations
0129 02:59:49.482038 (+258704us) tablet.cc:699] Row locks acquired
0129 02:59:49.482059 (+ 21us) write_op.cc:260] PREPARE: finished
0129 02:59:49.528508 (+ 46449us) write_op.cc:270] Start()
0129 02:59:49.528687 (+ 179us) write_op.cc:276] Timestamp: P: 1769655589528427 usec, L: 0
0129 02:59:49.528709 (+ 22us) op_driver.cc:348] REPLICATION: starting
0129 02:59:49.529591 (+ 882us) log.cc:844] Serialized 1308 byte log entry
0129 02:59:50.659707 (+1130116us) op_driver.cc:464] REPLICATION: finished
0129 02:59:51.132822 (+473115us) write_op.cc:301] APPLY: starting
0129 02:59:51.132936 (+ 114us) tablet.cc:1365] starting BulkCheckPresence
0129 02:59:51.133087 (+ 151us) tablet.cc:1368] finished BulkCheckPresence
0129 02:59:51.133209 (+ 122us) tablet.cc:1370] starting ApplyRowOperation cycle
0129 02:59:51.133712 (+ 503us) tablet.cc:1381] finished ApplyRowOperation cycle
0129 02:59:51.133919 (+ 207us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=3
0129 02:59:51.133974 (+ 55us) write_op.cc:312] APPLY: finished
0129 02:59:51.157360 (+ 23386us) log.cc:844] Serialized 35 byte log entry
0129 02:59:51.162702 (+ 5342us) write_op.cc:489] Releasing partition, row and schema locks
0129 02:59:51.162869 (+ 167us) write_op.cc:454] Released schema lock
0129 02:59:51.162946 (+ 77us) write_op.cc:341] FINISH: Updating metrics
0129 02:59:51.213743 (+ 50797us) write_op.cc:489] Releasing partition, row and schema locks
0129 02:59:51.213777 (+ 34us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":23287,"reported_tablets":15,"child_traces":[["op",{"apply.queue_time_us":440158,"apply.run_cpu_time_us":2585,"apply.run_wall_time_us":69539,"num_ops":3,"prepare.queue_time_us":288883,"prepare.run_cpu_time_us":29572,"prepare.run_wall_time_us":1272430,"raft.queue_time_us":583393,"raft.run_cpu_time_us":60096,"raft.run_wall_time_us":468954,"replication_time_us":1130744,"thread_start_us":1239952,"threads_started":4,"wal-append.queue_time_us":375471,"wal-append.run_cpu_time_us":25459,"wal-append.run_wall_time_us":2409321}]]}
I20260129 02:59:52.457835 20516 raft_consensus.cc:1240] T 4408006eb5174de1896b037cf3f18457 P 974b802141bd4f13beffef2aa03f88f9 [term 3 FOLLOWER]: Rejecting Update request from peer 197b880fa39d432a9b472845876bfaf0 for earlier term 2. Current term is 3. Ops: []
I20260129 02:59:52.004005 22738 raft_consensus.cc:3060] T 368953c9102643b2a21f99af36b51905 P 974b802141bd4f13beffef2aa03f88f9 [term 5 FOLLOWER]: Advancing to term 6
I20260129 02:59:52.285156 22554 raft_consensus.cc:697] T 42d27a28d89b409e8807ec0efd97af69 P 1c50de89a5a845668696dcb28c3badd6 [term 6 LEADER]: Becoming Leader. State: Replica: 1c50de89a5a845668696dcb28c3badd6, State: Running, Role: LEADER
I20260129 02:59:52.231660 22805 raft_consensus.cc:515] T 3e2d15f8640b407195210c476e09df91 P 1c50de89a5a845668696dcb28c3badd6 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } }
I20260129 02:59:52.281143 22748 consensus_queue.cc:1048] T e33a417cf9c54cf5af73a4ee3aac4d3a P 1c50de89a5a845668696dcb28c3badd6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 1, Time since last communication: 0.089s
I20260129 02:59:52.121138 22798 raft_consensus.cc:3055] T 1ce4ddf5b10e4ae8b193b13b2e5936ee P 197b880fa39d432a9b472845876bfaf0 [term 5 LEADER]: Stepping down as leader of term 5
I20260129 02:59:52.469275 22798 raft_consensus.cc:740] T 1ce4ddf5b10e4ae8b193b13b2e5936ee P 197b880fa39d432a9b472845876bfaf0 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 197b880fa39d432a9b472845876bfaf0, State: Running, Role: LEADER
I20260129 02:59:52.470347 22798 consensus_queue.cc:260] T 1ce4ddf5b10e4ae8b193b13b2e5936ee P 197b880fa39d432a9b472845876bfaf0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 5.3, Last appended by leader: 3, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:59:52.473491 22805 leader_election.cc:290] T 3e2d15f8640b407195210c476e09df91 P 1c50de89a5a845668696dcb28c3badd6 [CANDIDATE]: Term 3 election: Requested vote from peers 974b802141bd4f13beffef2aa03f88f9 (127.18.211.2:34517), 7a9daf5d42a84615a6383cfec9a10608 (127.18.211.4:40439)
I20260129 02:59:52.140511 22493 raft_consensus.cc:515] T 3906b080a5b34a74b8b64128635e8997 P 974b802141bd4f13beffef2aa03f88f9 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:59:52.481802 22493 leader_election.cc:290] T 3906b080a5b34a74b8b64128635e8997 P 974b802141bd4f13beffef2aa03f88f9 [CANDIDATE]: Term 4 election: Requested vote from peers 197b880fa39d432a9b472845876bfaf0 (127.18.211.1:42027), 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3:37735)
I20260129 02:59:52.482815 20589 raft_consensus.cc:2368] T 3e2d15f8640b407195210c476e09df91 P 1c50de89a5a845668696dcb28c3badd6 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 974b802141bd4f13beffef2aa03f88f9 for earlier term 2. Current term is 3.
I20260129 02:59:52.484278 20589 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:35752 (request call id 1577) took 1167 ms. Trace:
I20260129 02:59:52.305737 22700 raft_consensus.cc:2804] T 36ff33be666a4c82954731691dbcf327 P 974b802141bd4f13beffef2aa03f88f9 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260129 02:59:52.484978 22554 consensus_queue.cc:237] T 42d27a28d89b409e8807ec0efd97af69 P 1c50de89a5a845668696dcb28c3badd6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 5.7, Last appended by leader: 7, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } }
I20260129 02:59:52.306660 22815 raft_consensus.cc:493] T 0f20af7fbe5d4db89a6f17948d98912c P 974b802141bd4f13beffef2aa03f88f9 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260129 02:59:52.394696 20507 raft_consensus.cc:2468] T a644c02b16af47679824ccd060ac10fd P 974b802141bd4f13beffef2aa03f88f9 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7a9daf5d42a84615a6383cfec9a10608 in term 4.
I20260129 02:59:52.486089 22815 raft_consensus.cc:3060] T 0f20af7fbe5d4db89a6f17948d98912c P 974b802141bd4f13beffef2aa03f88f9 [term 5 FOLLOWER]: Advancing to term 6
I20260129 02:59:52.307590 20509 raft_consensus.cc:2468] T c934d31efd3f4e5195605ddc354d14e1 P 974b802141bd4f13beffef2aa03f88f9 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 197b880fa39d432a9b472845876bfaf0 in term 5.
I20260129 02:59:52.486512 22833 raft_consensus.cc:3055] T 75f45538dc704d9a8ec4ff8ad9f3ea5b P 7a9daf5d42a84615a6383cfec9a10608 [term 2 LEADER]: Stepping down as leader of term 2
W20260129 02:59:52.486804 20509 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:57600 (request call id 1779) took 2005 ms (client timeout 1795 ms). Trace:
I20260129 02:59:52.486970 22833 raft_consensus.cc:740] T 75f45538dc704d9a8ec4ff8ad9f3ea5b P 7a9daf5d42a84615a6383cfec9a10608 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 7a9daf5d42a84615a6383cfec9a10608, State: Running, Role: LEADER
I20260129 02:59:52.363495 20581 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35752 (request call id 1564) took 1660 ms. Trace:
I20260129 02:59:52.353086 22320 raft_consensus.cc:740] T a644c02b16af47679824ccd060ac10fd P 1c50de89a5a845668696dcb28c3badd6 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 1c50de89a5a845668696dcb28c3badd6, State: Running, Role: LEADER
I20260129 02:59:52.461709 22687 raft_consensus.cc:697] T fcddab6bca684adf8ea032289a00f90e P 974b802141bd4f13beffef2aa03f88f9 [term 8 LEADER]: Becoming Leader. State: Replica: 974b802141bd4f13beffef2aa03f88f9, State: Running, Role: LEADER
I20260129 02:59:52.487941 22833 consensus_queue.cc:260] T 75f45538dc704d9a8ec4ff8ad9f3ea5b P 7a9daf5d42a84615a6383cfec9a10608 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } }
I20260129 02:59:52.488500 22320 consensus_queue.cc:260] T a644c02b16af47679824ccd060ac10fd P 1c50de89a5a845668696dcb28c3badd6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.5, Last appended by leader: 5, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } }
I20260129 02:59:52.489033 22833 raft_consensus.cc:3060] T 75f45538dc704d9a8ec4ff8ad9f3ea5b P 7a9daf5d42a84615a6383cfec9a10608 [term 2 FOLLOWER]: Advancing to term 3
I20260129 02:59:52.488747 22687 consensus_queue.cc:237] T fcddab6bca684adf8ea032289a00f90e P 974b802141bd4f13beffef2aa03f88f9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 4.4, Last appended by leader: 4, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:59:52.489560 22320 raft_consensus.cc:3060] T a644c02b16af47679824ccd060ac10fd P 1c50de89a5a845668696dcb28c3badd6 [term 3 FOLLOWER]: Advancing to term 4
I20260129 02:59:52.340358 22758 raft_consensus.cc:740] T 76f713c3331540d2a5b76164103e9eb1 P 197b880fa39d432a9b472845876bfaf0 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 197b880fa39d432a9b472845876bfaf0, State: Running, Role: LEADER
I20260129 02:59:52.415344 20588 raft_consensus.cc:2393] T 505580c93c694bbb8ea96fdf9b17ab46 P 1c50de89a5a845668696dcb28c3badd6 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 974b802141bd4f13beffef2aa03f88f9 in current term 5: Already voted for candidate 1c50de89a5a845668696dcb28c3badd6 in this term.
I20260129 02:59:52.392611 22640 raft_consensus.cc:740] T e7d83d4ccf684e76acb29a0dc844e1b8 P 1c50de89a5a845668696dcb28c3badd6 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 1c50de89a5a845668696dcb28c3badd6, State: Running, Role: LEADER
I20260129 02:59:52.484740 20589 rpcz_store.cc:276] 0129 02:59:51.316712 (+ 0us) service_pool.cc:168] Inserting onto call queue
0129 02:59:51.380045 (+ 63333us) service_pool.cc:225] Handling call
0129 02:59:52.484025 (+1103980us) spinlock_profiling.cc:243] Waited 1.08 s on lock 0x7b540003e3d4. stack: 00007f8fe044e373 00007f8fe044e270 00007f8fdfda0574 00007f8fe9a5c6b4 00007f8fe9a5c649 00007f8fe34975a1 00007f8fe3531f7a 00007f8fe84e3647 00007f8fe1a73834 00007f8fe1a737c2 00007f8fe1a736f1 00007f8fe1a7366c 00007f8fe1a72922 00007f8fe186d34c 00007f8fe186c786 00007f8fe186c12f
0129 02:59:52.484205 (+ 180us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2484507904}
I20260129 02:59:52.492179 20589 raft_consensus.cc:1240] T 368953c9102643b2a21f99af36b51905 P 1c50de89a5a845668696dcb28c3badd6 [term 6 LEADER]: Rejecting Update request from peer 974b802141bd4f13beffef2aa03f88f9 for earlier term 5. Current term is 6. Ops: []
I20260129 02:59:52.492865 20589 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35752 (request call id 1575) took 1186 ms. Trace:
I20260129 02:59:52.493165 20589 rpcz_store.cc:276] 0129 02:59:51.306592 (+ 0us) service_pool.cc:168] Inserting onto call queue
0129 02:59:52.491674 (+1185082us) service_pool.cc:225] Handling call
0129 02:59:52.492011 (+ 337us) raft_consensus.cc:1471] Updating replica for 0 ops
0129 02:59:52.492629 (+ 618us) raft_consensus.cc:1697] Filling consensus response to leader.
0129 02:59:52.492817 (+ 188us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260129 02:59:52.387236 22807 leader_election.cc:290] T 0678b8b468bd43cdbc3e485900844821 P 974b802141bd4f13beffef2aa03f88f9 [CANDIDATE]: Term 6 election: Requested vote from peers 197b880fa39d432a9b472845876bfaf0 (127.18.211.1:42027), 7a9daf5d42a84615a6383cfec9a10608 (127.18.211.4:40439)
I20260129 02:59:52.276959 20587 rpcz_store.cc:276] 0129 02:59:49.545898 (+ 0us) service_pool.cc:168] Inserting onto call queue
0129 02:59:51.770568 (+2224670us) service_pool.cc:225] Handling call
0129 02:59:51.775380 (+ 4812us) raft_consensus.cc:1471] Updating replica for 0 ops
0129 02:59:51.888307 (+112927us) raft_consensus.cc:1697] Filling consensus response to leader.
0129 02:59:52.114671 (+226364us) spinlock_profiling.cc:243] Waited 26.9 ms on lock 0x7b5400109354. stack: 00007f8fe044e373 00007f8fe044e270 00007f8fdfda0574 00007f8fe9a5c6b4 00007f8fe9a5c649 00007f8fe34975a1 00007f8fe352ea81 00007f8fe352d4be 00007f8fe84e1f49 00007f8fe1a6ed34 00007f8fe1a6ecc2 00007f8fe1a6ebf1 00007f8fe1a6eb6c 00007f8fe1a6de22 00007f8fe186d34c 00007f8fe186c786
0129 02:59:52.114856 (+ 185us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":61893376}
I20260129 02:59:52.426507 20590 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35752 (request call id 1568) took 1702 ms. Trace:
I20260129 02:59:52.486110 20516 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57600 (request call id 1780) took 2004 ms. Trace:
I20260129 02:59:52.494788 20516 rpcz_store.cc:276] 0129 02:59:50.481215 (+ 0us) service_pool.cc:168] Inserting onto call queue
0129 02:59:51.503176 (+1021961us) service_pool.cc:225] Handling call
0129 02:59:51.503467 (+ 291us) raft_consensus.cc:1471] Updating replica for 0 ops
0129 02:59:52.458257 (+954790us) raft_consensus.cc:1697] Filling consensus response to leader.
0129 02:59:52.485886 (+ 27629us) spinlock_profiling.cc:243] Waited 954 ms on lock 0x7b5400315e54. stack: 00007f8fe044e373 00007f8fe044e270 00007f8fdfda0574 00007f8fe9a5c6b4 00007f8fe9a5c649 00007f8fe34975a1 00007f8fe352ea81 00007f8fe352d4be 00007f8fe84e1f49 00007f8fe1a6ed34 00007f8fe1a6ecc2 00007f8fe1a6ebf1 00007f8fe1a6eb6c 00007f8fe1a6de22 00007f8fe186d34c 00007f8fe186c786
0129 02:59:52.486052 (+ 166us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2194343168}
I20260129 02:59:52.440649 20583 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35752 (request call id 1553) took 1834 ms. Trace:
I20260129 02:59:52.496448 20583 rpcz_store.cc:276] 0129 02:59:50.606357 (+ 0us) service_pool.cc:168] Inserting onto call queue
0129 02:59:52.302242 (+1695885us) service_pool.cc:225] Handling call
0129 02:59:52.314244 (+ 12002us) raft_consensus.cc:1471] Updating replica for 0 ops
0129 02:59:52.440383 (+126139us) raft_consensus.cc:1697] Filling consensus response to leader.
0129 02:59:52.440593 (+ 210us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260129 02:59:52.393594 22581 raft_consensus.cc:740] T a4c1226b987b468fa84dfc6ba0e2dbe1 P 1c50de89a5a845668696dcb28c3badd6 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 1c50de89a5a845668696dcb28c3badd6, State: Running, Role: LEADER
W20260129 02:59:52.440953 20479 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.211.1:42027, user_credentials={real_user=slave}} blocked reactor thread for 171245us
I20260129 02:59:52.507663 22763 raft_consensus.cc:2749] T c934d31efd3f4e5195605ddc354d14e1 P 974b802141bd4f13beffef2aa03f88f9 [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
W20260129 02:59:52.441110 20480 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.211.3:37735, user_credentials={real_user=slave}} blocked reactor thread for 91506.2us
I20260129 02:59:52.134603 22820 raft_consensus.cc:515] T b189f08e16a74a11980cc55d99359a0a P 974b802141bd4f13beffef2aa03f88f9 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:59:52.400211 22425 raft_consensus.cc:740] T 7d9494f5584d4d5db8b9adf1c8fc888b P 1c50de89a5a845668696dcb28c3badd6 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 1c50de89a5a845668696dcb28c3badd6, State: Running, Role: LEADER
I20260129 02:59:52.412020 22786 raft_consensus.cc:740] T 4f8f3a38f6e24ba4857875181f447442 P 1c50de89a5a845668696dcb28c3badd6 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 1c50de89a5a845668696dcb28c3badd6, State: Running, Role: LEADER
I20260129 02:59:52.397500 22814 consensus_queue.cc:1059] T a4c1226b987b468fa84dfc6ba0e2dbe1 P 1c50de89a5a845668696dcb28c3badd6 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 }, Status: INVALID_TERM, Last received: 3.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.062s
I20260129 02:59:52.388000 22344 consensus_queue.cc:1048] T 75b0b6e82ddc4f92b1fe1d94f9a0b73b P 1c50de89a5a845668696dcb28c3badd6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.071s
W20260129 02:59:52.528154 21102 leader_election.cc:336] T 4e420f90884f4796997b5ea41e49e411 P 7a9daf5d42a84615a6383cfec9a10608 [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer 974b802141bd4f13beffef2aa03f88f9 (127.18.211.2:34517): Timed out: RequestConsensusVote RPC to 127.18.211.2:34517 timed out after 1.831s (SENT)
I20260129 02:59:52.528805 21102 leader_election.cc:304] T 4e420f90884f4796997b5ea41e49e411 P 7a9daf5d42a84615a6383cfec9a10608 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7a9daf5d42a84615a6383cfec9a10608; no voters: 197b880fa39d432a9b472845876bfaf0, 974b802141bd4f13beffef2aa03f88f9
W20260129 02:59:52.528995 20407 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.211.2:34517, user_credentials={real_user=slave}} blocked reactor thread for 568703us
I20260129 02:59:52.460932 22719 raft_consensus.cc:2749] T c2633a6fc34c44579de348d208b0c01e P 974b802141bd4f13beffef2aa03f88f9 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260129 02:59:52.411427 21132 rpcz_store.cc:276] 0129 02:59:50.658507 (+ 0us) service_pool.cc:168] Inserting onto call queue
0129 02:59:51.073623 (+415116us) service_pool.cc:225] Handling call
0129 02:59:51.721448 (+647825us) raft_consensus.cc:1471] Updating replica for 0 ops
0129 02:59:51.836328 (+114880us) raft_consensus.cc:1697] Filling consensus response to leader.
0129 02:59:52.089050 (+252722us) spinlock_profiling.cc:243] Waited 613 ms on lock 0x7b5400670550. stack: 00007f8fe044e373 00007f8fe044e270 00007f8fdfda0574 00007f8fe9a5c6b4 00007f8fe9a5c649 00007f8fe34975a1 00007f8fe352d52d 00007f8fe84e1f49 00007f8fe1a6ed34 00007f8fe1a6ecc2 00007f8fe1a6ebf1 00007f8fe1a6eb6c 00007f8fe1a6de22 00007f8fe186d34c 00007f8fe186c786 00007f8fe186c12f
0129 02:59:52.089222 (+ 172us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1410253056}
W20260129 02:59:52.468838 22641 consensus_meta.cc:287] T 1299723a4476420ab02fd8bfd7aa0645 P 197b880fa39d432a9b472845876bfaf0: Time spent flushing consensus metadata: real 0.506s user 0.003s sys 0.006s
I20260129 02:59:52.485414 22700 raft_consensus.cc:493] T 36ff33be666a4c82954731691dbcf327 P 974b802141bd4f13beffef2aa03f88f9 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 1c50de89a5a845668696dcb28c3badd6)
I20260129 02:59:52.565181 20588 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:35752 (request call id 1576) took 1254 ms. Trace:
I20260129 02:59:52.568033 20588 rpcz_store.cc:276] 0129 02:59:51.310989 (+ 0us) service_pool.cc:168] Inserting onto call queue
0129 02:59:51.374879 (+ 63890us) service_pool.cc:225] Handling call
0129 02:59:52.564900 (+1190021us) spinlock_profiling.cc:243] Waited 1.04 s on lock 0x7b540033fc54. stack: 00007f8fe044e373 00007f8fe044e270 00007f8fdfda0574 00007f8fe9a5c6b4 00007f8fe9a5c649 00007f8fe34975a1 00007f8fe3531f7a 00007f8fe84e3647 00007f8fe1a73834 00007f8fe1a737c2 00007f8fe1a736f1 00007f8fe1a7366c 00007f8fe1a72922 00007f8fe186d34c 00007f8fe186c786 00007f8fe186c12f
0129 02:59:52.565102 (+ 202us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2390338432}
W20260129 02:59:52.487265 20509 rpcz_store.cc:269] 0129 02:59:50.480921 (+ 0us) service_pool.cc:168] Inserting onto call queue
0129 02:59:50.521067 (+ 40146us) service_pool.cc:225] Handling call
0129 02:59:52.486727 (+1965660us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260129 02:59:52.501891 22640 consensus_queue.cc:260] T e7d83d4ccf684e76acb29a0dc844e1b8 P 1c50de89a5a845668696dcb28c3badd6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 4.3, Last appended by leader: 3, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
W20260129 02:59:52.502662 20482 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260129 02:59:52.497789 22798 raft_consensus.cc:3060] T 1ce4ddf5b10e4ae8b193b13b2e5936ee P 197b880fa39d432a9b472845876bfaf0 [term 5 FOLLOWER]: Advancing to term 6
I20260129 02:59:52.503572 22758 consensus_queue.cc:260] T 76f713c3331540d2a5b76164103e9eb1 P 197b880fa39d432a9b472845876bfaf0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:59:52.511602 20480 leader_election.cc:304] T a5a05fbcbbcf4916bbc01fd1a5b5e90d P 974b802141bd4f13beffef2aa03f88f9 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1c50de89a5a845668696dcb28c3badd6, 974b802141bd4f13beffef2aa03f88f9; no voters:
I20260129 02:59:52.518275 20589 raft_consensus.cc:1275] T 1299723a4476420ab02fd8bfd7aa0645 P 1c50de89a5a845668696dcb28c3badd6 [term 8 FOLLOWER]: Refusing update from remote peer 7a9daf5d42a84615a6383cfec9a10608: Log matching property violated. Preceding OpId in replica: term: 6 index: 7. Preceding OpId from leader: term: 8 index: 8. (index mismatch)
I20260129 02:59:52.310395 22751 consensus_queue.cc:1048] T 78aa39096868415d9afc2ffcb248e198 P 1c50de89a5a845668696dcb28c3badd6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.149s
I20260129 02:59:52.529731 20515 raft_consensus.cc:1240] T 0678b8b468bd43cdbc3e485900844821 P 974b802141bd4f13beffef2aa03f88f9 [term 6 FOLLOWER]: Rejecting Update request from peer 197b880fa39d432a9b472845876bfaf0 for earlier term 5. Current term is 6. Ops: []
I20260129 02:59:52.530441 21099 leader_election.cc:304] T dc2f4cb752cf4d59b5f6bc689f5d8104 P 7a9daf5d42a84615a6383cfec9a10608 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7a9daf5d42a84615a6383cfec9a10608; no voters: 197b880fa39d432a9b472845876bfaf0, 1c50de89a5a845668696dcb28c3badd6
W20260129 02:59:52.600525 21099 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.211.1:42027, user_credentials={real_user=slave}} blocked reactor thread for 70218.2us
I20260129 02:59:52.602294 21099 leader_election.cc:304] T 18f30ebaa4174fd28c958e72d0cc7e24 P 7a9daf5d42a84615a6383cfec9a10608 [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7a9daf5d42a84615a6383cfec9a10608; no voters: 197b880fa39d432a9b472845876bfaf0, 1c50de89a5a845668696dcb28c3badd6
I20260129 02:59:52.487496 20581 rpcz_store.cc:276] 0129 02:59:50.703210 (+ 0us) service_pool.cc:168] Inserting onto call queue
0129 02:59:52.348918 (+1645708us) service_pool.cc:225] Handling call
0129 02:59:52.362859 (+ 13941us) raft_consensus.cc:1471] Updating replica for 0 ops
0129 02:59:52.363104 (+ 245us) raft_consensus.cc:1534] Early marking committed up to index 0
0129 02:59:52.363142 (+ 38us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0129 02:59:52.363201 (+ 59us) raft_consensus.cc:1645] Marking committed up to 4
0129 02:59:52.363247 (+ 46us) raft_consensus.cc:1697] Filling consensus response to leader.
0129 02:59:52.363319 (+ 72us) raft_consensus.cc:1691] UpdateReplicas() finished
0129 02:59:52.363441 (+ 122us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260129 02:59:52.494215 20590 rpcz_store.cc:276] 0129 02:59:50.724240 (+ 0us) service_pool.cc:168] Inserting onto call queue
0129 02:59:52.414007 (+1689767us) service_pool.cc:225] Handling call
0129 02:59:52.426002 (+ 11995us) raft_consensus.cc:1471] Updating replica for 0 ops
0129 02:59:52.426177 (+ 175us) raft_consensus.cc:1534] Early marking committed up to index 0
0129 02:59:52.426203 (+ 26us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0129 02:59:52.426245 (+ 42us) raft_consensus.cc:1645] Marking committed up to 5
0129 02:59:52.426274 (+ 29us) raft_consensus.cc:1697] Filling consensus response to leader.
0129 02:59:52.426328 (+ 54us) raft_consensus.cc:1691] UpdateReplicas() finished
0129 02:59:52.426453 (+ 125us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20260129 02:59:52.547070 20405 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.211.3:37735, user_credentials={real_user=slave}} blocked reactor thread for 299360us
I20260129 02:59:52.604200 22831 raft_consensus.cc:2749] T 18f30ebaa4174fd28c958e72d0cc7e24 P 7a9daf5d42a84615a6383cfec9a10608 [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20260129 02:59:52.603765 22343 raft_consensus.cc:2749] T dc2f4cb752cf4d59b5f6bc689f5d8104 P 7a9daf5d42a84615a6383cfec9a10608 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260129 02:59:52.605000 20405 leader_election.cc:304] T 78aa39096868415d9afc2ffcb248e198 P 197b880fa39d432a9b472845876bfaf0 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 197b880fa39d432a9b472845876bfaf0; no voters: 1c50de89a5a845668696dcb28c3badd6, 7a9daf5d42a84615a6383cfec9a10608
I20260129 02:59:52.606184 22737 raft_consensus.cc:2749] T 78aa39096868415d9afc2ffcb248e198 P 197b880fa39d432a9b472845876bfaf0 [term 5 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20260129 02:59:52.606664 20405 leader_election.cc:304] T 00e79d8f25a34676b1d866eb2068a2b2 P 197b880fa39d432a9b472845876bfaf0 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 197b880fa39d432a9b472845876bfaf0; no voters: 7a9daf5d42a84615a6383cfec9a10608, 974b802141bd4f13beffef2aa03f88f9
I20260129 02:59:52.608809 22737 raft_consensus.cc:2749] T 00e79d8f25a34676b1d866eb2068a2b2 P 197b880fa39d432a9b472845876bfaf0 [term 6 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260129 02:59:52.609153 20405 leader_election.cc:304] T c934d31efd3f4e5195605ddc354d14e1 P 197b880fa39d432a9b472845876bfaf0 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 197b880fa39d432a9b472845876bfaf0, 7a9daf5d42a84615a6383cfec9a10608; no voters:
W20260129 02:59:52.559689 21135 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.755s user 0.051s sys 0.070s
I20260129 02:59:52.460426 22745 leader_election.cc:290] T 505580c93c694bbb8ea96fdf9b17ab46 P 974b802141bd4f13beffef2aa03f88f9 [CANDIDATE]: Term 5 election: Requested vote from peers 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3:37735), 7a9daf5d42a84615a6383cfec9a10608 (127.18.211.4:40439)
I20260129 02:59:52.359288 20586 raft_consensus.cc:2468] T 926a250fa5834e88b96ccecbefbf5086 P 1c50de89a5a845668696dcb28c3badd6 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 974b802141bd4f13beffef2aa03f88f9 in term 6.
I20260129 02:59:52.578816 20516 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4b1876f9a70b49edb3aa74105e66d3b3" candidate_uuid: "197b880fa39d432a9b472845876bfaf0" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "974b802141bd4f13beffef2aa03f88f9" is_pre_election: true
W20260129 02:59:52.610610 20586 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:35752 (request call id 1562) took 1948 ms (client timeout 2076 ms). Trace:
W20260129 02:59:52.610966 20586 rpcz_store.cc:269] 0129 02:59:50.662012 (+ 0us) service_pool.cc:168] Inserting onto call queue
0129 02:59:51.223952 (+561940us) service_pool.cc:225] Handling call
0129 02:59:52.108130 (+884178us) spinlock_profiling.cc:243] Waited 8.6 ms on lock 0x7b58001cb398. stack: 00007f8fe044e373 00007f8fe044e270 00007f8fdfda0574 00007f8fe9a5c6b4 00007f8fe9a5c649 00007f8fe02c08b1 00007f8fe0442db1 00007f8fe0442c62 00007f8fe92eddd9 00007f8fe846d260 00007f8fe8535dbb 00007f8fe8535e41 00007f8fe84e3bd0 00007f8fe84e34b4 00007f8fe1a73834 00007f8fe1a737c2
0129 02:59:52.610536 (+502406us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":19783936}
I20260129 02:59:52.508946 22694 raft_consensus.cc:2804] T 17d6431cc77a491e8464d68db4081a51 P 974b802141bd4f13beffef2aa03f88f9 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260129 02:59:52.560508 22656 leader_election.cc:290] T 505580c93c694bbb8ea96fdf9b17ab46 P 1c50de89a5a845668696dcb28c3badd6 [CANDIDATE]: Term 5 election: Requested vote from peers 974b802141bd4f13beffef2aa03f88f9 (127.18.211.2:34517), 7a9daf5d42a84615a6383cfec9a10608 (127.18.211.4:40439)
W20260129 02:59:52.571467 20407 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260129 02:59:52.589555 20555 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260129 02:59:52.612040 21135 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.808s user 0.052s sys 0.070s
I20260129 02:59:52.567705 20582 raft_consensus.cc:1240] T e33a417cf9c54cf5af73a4ee3aac4d3a P 1c50de89a5a845668696dcb28c3badd6 [term 5 LEADER]: Rejecting Update request from peer 974b802141bd4f13beffef2aa03f88f9 for earlier term 4. Current term is 5. Ops: [4.3-4.3]
I20260129 02:59:52.563820 22641 raft_consensus.cc:2749] T 1299723a4476420ab02fd8bfd7aa0645 P 197b880fa39d432a9b472845876bfaf0 [term 8 FOLLOWER]: Leader pre-election lost for term 7. Reason: Vote denied by peer 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3:37735) with higher term. Message: Invalid argument: T 1299723a4476420ab02fd8bfd7aa0645 P 1c50de89a5a845668696dcb28c3badd6 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 197b880fa39d432a9b472845876bfaf0 for earlier term 7. Current term is 8.
I20260129 02:59:52.613389 22694 raft_consensus.cc:493] T 17d6431cc77a491e8464d68db4081a51 P 974b802141bd4f13beffef2aa03f88f9 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260129 02:59:52.572047 22700 raft_consensus.cc:3060] T 36ff33be666a4c82954731691dbcf327 P 974b802141bd4f13beffef2aa03f88f9 [term 2 FOLLOWER]: Advancing to term 3
I20260129 02:59:52.617956 20511 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fcddab6bca684adf8ea032289a00f90e" candidate_uuid: "197b880fa39d432a9b472845876bfaf0" candidate_term: 9 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "974b802141bd4f13beffef2aa03f88f9" is_pre_election: true
I20260129 02:59:52.626863 20407 leader_election.cc:304] T cedce331e1584ac8b636c9b8a72b2ba0 P 197b880fa39d432a9b472845876bfaf0 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 197b880fa39d432a9b472845876bfaf0, 974b802141bd4f13beffef2aa03f88f9; no voters:
I20260129 02:59:52.618839 20582 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35752 (request call id 1552) took 2012 ms. Trace:
I20260129 02:59:52.629556 20582 rpcz_store.cc:276] 0129 02:59:50.606134 (+ 0us) service_pool.cc:168] Inserting onto call queue
0129 02:59:52.300312 (+1694178us) service_pool.cc:225] Handling call
0129 02:59:52.356134 (+ 55822us) raft_consensus.cc:1471] Updating replica for 1 ops
0129 02:59:52.567353 (+211219us) spinlock_profiling.cc:243] Waited 128 ms on lock 0x7b54002da3a0. stack: 00007f8fe044e373 00007f8fe044e270 00007f8fdfda0574 00007f8fe9a5c6b4 00007f8fe9a5c649 00007f8fe34975a1 00007f8fe34a5df0 00007f8fe352fe40 00007f8fe3530eb1 00007f8fe352da0f 00007f8fe352d4be 00007f8fe84e1f49 00007f8fe1a6ed34 00007f8fe1a6ecc2 00007f8fe1a6ebf1 00007f8fe1a6eb6c
0129 02:59:52.618545 (+ 51192us) raft_consensus.cc:1697] Filling consensus response to leader.
0129 02:59:52.618781 (+ 236us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":294699904}
W20260129 02:59:52.615844 20480 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.211.3:37735, user_credentials={real_user=slave}} blocked reactor thread for 104371us
W20260129 02:59:52.632318 20481 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260129 02:59:52.635228 20480 leader_election.cc:304] T add8d4923d1a489e86882c3a25692b95 P 974b802141bd4f13beffef2aa03f88f9 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7a9daf5d42a84615a6383cfec9a10608, 974b802141bd4f13beffef2aa03f88f9; no voters:
I20260129 02:59:52.636214 21102 leader_election.cc:304] T add8d4923d1a489e86882c3a25692b95 P 7a9daf5d42a84615a6383cfec9a10608 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 7a9daf5d42a84615a6383cfec9a10608, 974b802141bd4f13beffef2aa03f88f9; no voters: 1c50de89a5a845668696dcb28c3badd6
I20260129 02:59:52.619566 20438 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a5a05fbcbbcf4916bbc01fd1a5b5e90d" candidate_uuid: "974b802141bd4f13beffef2aa03f88f9" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "197b880fa39d432a9b472845876bfaf0"
I20260129 02:59:52.620020 20437 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "87d5fe48a44a408aaf03e894a67ecb2f" candidate_uuid: "7a9daf5d42a84615a6383cfec9a10608" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "197b880fa39d432a9b472845876bfaf0"
W20260129 02:59:52.617058 20554 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.211.4:40439, user_credentials={real_user=slave}} blocked reactor thread for 364492us
I20260129 02:59:52.621985 20512 raft_consensus.cc:1240] T 76f713c3331540d2a5b76164103e9eb1 P 974b802141bd4f13beffef2aa03f88f9 [term 4 FOLLOWER]: Rejecting Update request from peer 197b880fa39d432a9b472845876bfaf0 for earlier term 3. Current term is 4. Ops: []
I20260129 02:59:52.622537 20433 raft_consensus.cc:1275] T 00e79d8f25a34676b1d866eb2068a2b2 P 197b880fa39d432a9b472845876bfaf0 [term 6 FOLLOWER]: Refusing update from remote peer 7a9daf5d42a84615a6383cfec9a10608: Log matching property violated. Preceding OpId in replica: term: 3 index: 6. Preceding OpId from leader: term: 6 index: 8. (index mismatch)
I20260129 02:59:52.620849 20435 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4e420f90884f4796997b5ea41e49e411" candidate_uuid: "974b802141bd4f13beffef2aa03f88f9" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "197b880fa39d432a9b472845876bfaf0"
I20260129 02:59:52.640370 20507 raft_consensus.cc:1275] T 00e79d8f25a34676b1d866eb2068a2b2 P 974b802141bd4f13beffef2aa03f88f9 [term 6 FOLLOWER]: Refusing update from remote peer 7a9daf5d42a84615a6383cfec9a10608: Log matching property violated. Preceding OpId in replica: term: 4 index: 7. Preceding OpId from leader: term: 6 index: 8. (index mismatch)
I20260129 02:59:52.642972 20435 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4408006eb5174de1896b037cf3f18457" candidate_uuid: "974b802141bd4f13beffef2aa03f88f9" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "197b880fa39d432a9b472845876bfaf0"
I20260129 02:59:52.644135 20433 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0678b8b468bd43cdbc3e485900844821" candidate_uuid: "974b802141bd4f13beffef2aa03f88f9" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "197b880fa39d432a9b472845876bfaf0"
I20260129 02:59:52.645222 21136 raft_consensus.cc:1240] T a644c02b16af47679824ccd060ac10fd P 7a9daf5d42a84615a6383cfec9a10608 [term 4 FOLLOWER]: Rejecting Update request from peer 1c50de89a5a845668696dcb28c3badd6 for earlier term 3. Current term is 4. Ops: []
I20260129 02:59:52.646188 20440 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0678b8b468bd43cdbc3e485900844821" candidate_uuid: "7a9daf5d42a84615a6383cfec9a10608" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "197b880fa39d432a9b472845876bfaf0"
I20260129 02:59:52.646955 20441 raft_consensus.cc:1275] T 1299723a4476420ab02fd8bfd7aa0645 P 197b880fa39d432a9b472845876bfaf0 [term 8 FOLLOWER]: Refusing update from remote peer 7a9daf5d42a84615a6383cfec9a10608: Log matching property violated. Preceding OpId in replica: term: 6 index: 7. Preceding OpId from leader: term: 8 index: 8. (index mismatch)
I20260129 02:59:52.643760 21102 leader_election.cc:304] T a644c02b16af47679824ccd060ac10fd P 7a9daf5d42a84615a6383cfec9a10608 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 7a9daf5d42a84615a6383cfec9a10608, 974b802141bd4f13beffef2aa03f88f9; no voters: 1c50de89a5a845668696dcb28c3badd6
I20260129 02:59:52.571560 22400 consensus_queue.cc:1048] T 3ccac82a01a2445fa5b0ce6c0fcf2295 P 974b802141bd4f13beffef2aa03f88f9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.054s
I20260129 02:59:52.650365 20434 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2ac659d732314e1288849f08fe5e074e" candidate_uuid: "7a9daf5d42a84615a6383cfec9a10608" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "197b880fa39d432a9b472845876bfaf0" is_pre_election: true
W20260129 02:59:52.651702 20554 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260129 02:59:52.651583 20435 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a8479d1fe0bd41e5946186e3a6ca8c7a" candidate_uuid: "974b802141bd4f13beffef2aa03f88f9" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "197b880fa39d432a9b472845876bfaf0" is_pre_election: true
I20260129 02:59:52.617082 22237 raft_consensus.cc:2804] T c934d31efd3f4e5195605ddc354d14e1 P 197b880fa39d432a9b472845876bfaf0 [term 5 FOLLOWER]: Leader election won for term 5
I20260129 02:59:52.652511 20433 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3906b080a5b34a74b8b64128635e8997" candidate_uuid: "974b802141bd4f13beffef2aa03f88f9" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "197b880fa39d432a9b472845876bfaf0"
I20260129 02:59:52.652834 20438 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e33a417cf9c54cf5af73a4ee3aac4d3a" candidate_uuid: "974b802141bd4f13beffef2aa03f88f9" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "197b880fa39d432a9b472845876bfaf0"
I20260129 02:59:52.656095 20438 raft_consensus.cc:3060] T e33a417cf9c54cf5af73a4ee3aac4d3a P 197b880fa39d432a9b472845876bfaf0 [term 5 FOLLOWER]: Advancing to term 6
I20260129 02:59:52.653151 20432 raft_consensus.cc:1275] T e4d18c35b0a7493e85d0a4b92956a48f P 197b880fa39d432a9b472845876bfaf0 [term 5 FOLLOWER]: Refusing update from remote peer 974b802141bd4f13beffef2aa03f88f9: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 5 index: 3. (index mismatch)
I20260129 02:59:52.653105 21136 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "505580c93c694bbb8ea96fdf9b17ab46" candidate_uuid: "1c50de89a5a845668696dcb28c3badd6" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "7a9daf5d42a84615a6383cfec9a10608"
I20260129 02:59:52.658478 21136 raft_consensus.cc:3060] T 505580c93c694bbb8ea96fdf9b17ab46 P 7a9daf5d42a84615a6383cfec9a10608 [term 3 FOLLOWER]: Advancing to term 5
I20260129 02:59:52.657384 20586 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7a9a69cd82724b1faed1d10aa091fc4a" candidate_uuid: "974b802141bd4f13beffef2aa03f88f9" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "1c50de89a5a845668696dcb28c3badd6" is_pre_election: true
I20260129 02:59:52.659406 20586 raft_consensus.cc:2410] T 7a9a69cd82724b1faed1d10aa091fc4a P 1c50de89a5a845668696dcb28c3badd6 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 974b802141bd4f13beffef2aa03f88f9 for term 6 because replica has last-logged OpId of term: 4 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 2.
I20260129 02:59:52.597237 22184 consensus_queue.cc:1048] T 8055232cf0004d40a85f3bf983fe00a9 P 974b802141bd4f13beffef2aa03f88f9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.050s
I20260129 02:59:52.658356 20433 raft_consensus.cc:3060] T 3906b080a5b34a74b8b64128635e8997 P 197b880fa39d432a9b472845876bfaf0 [term 3 FOLLOWER]: Advancing to term 4
I20260129 02:59:52.659754 20437 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3ccac82a01a2445fa5b0ce6c0fcf2295" candidate_uuid: "7a9daf5d42a84615a6383cfec9a10608" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "197b880fa39d432a9b472845876bfaf0"
I20260129 02:59:52.668105 20407 leader_election.cc:304] T e7d83d4ccf684e76acb29a0dc844e1b8 P 197b880fa39d432a9b472845876bfaf0 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 197b880fa39d432a9b472845876bfaf0, 974b802141bd4f13beffef2aa03f88f9; no voters:
I20260129 02:59:52.670354 20437 raft_consensus.cc:2376] T 3ccac82a01a2445fa5b0ce6c0fcf2295 P 197b880fa39d432a9b472845876bfaf0 [term 4 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 7a9daf5d42a84615a6383cfec9a10608 in term 4. Re-sending same reply.
I20260129 02:59:52.654099 21132 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0678b8b468bd43cdbc3e485900844821" candidate_uuid: "974b802141bd4f13beffef2aa03f88f9" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "7a9daf5d42a84615a6383cfec9a10608"
I20260129 02:59:52.657918 20590 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "505580c93c694bbb8ea96fdf9b17ab46" candidate_uuid: "974b802141bd4f13beffef2aa03f88f9" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "1c50de89a5a845668696dcb28c3badd6"
I20260129 02:59:52.671231 21132 raft_consensus.cc:2393] T 0678b8b468bd43cdbc3e485900844821 P 7a9daf5d42a84615a6383cfec9a10608 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 974b802141bd4f13beffef2aa03f88f9 in current term 6: Already voted for candidate 7a9daf5d42a84615a6383cfec9a10608 in this term.
I20260129 02:59:52.626148 22694 raft_consensus.cc:3060] T 17d6431cc77a491e8464d68db4081a51 P 974b802141bd4f13beffef2aa03f88f9 [term 3 FOLLOWER]: Advancing to term 4
I20260129 02:59:52.627460 22758 raft_consensus.cc:3060] T 76f713c3331540d2a5b76164103e9eb1 P 197b880fa39d432a9b472845876bfaf0 [term 3 FOLLOWER]: Advancing to term 4
W20260129 02:59:52.668732 20406 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260129 02:59:52.673417 20590 raft_consensus.cc:2393] T 505580c93c694bbb8ea96fdf9b17ab46 P 1c50de89a5a845668696dcb28c3badd6 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 974b802141bd4f13beffef2aa03f88f9 in current term 5: Already voted for candidate 1c50de89a5a845668696dcb28c3badd6 in this term.
I20260129 02:59:52.660017 20589 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4408006eb5174de1896b037cf3f18457" candidate_uuid: "974b802141bd4f13beffef2aa03f88f9" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1c50de89a5a845668696dcb28c3badd6"
I20260129 02:59:52.674252 20589 raft_consensus.cc:2393] T 4408006eb5174de1896b037cf3f18457 P 1c50de89a5a845668696dcb28c3badd6 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 974b802141bd4f13beffef2aa03f88f9 in current term 3: Already voted for candidate 1c50de89a5a845668696dcb28c3badd6 in this term.
I20260129 02:59:52.661698 20588 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3906b080a5b34a74b8b64128635e8997" candidate_uuid: "974b802141bd4f13beffef2aa03f88f9" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "1c50de89a5a845668696dcb28c3badd6"
I20260129 02:59:52.624934 22640 raft_consensus.cc:3060] T e7d83d4ccf684e76acb29a0dc844e1b8 P 1c50de89a5a845668696dcb28c3badd6 [term 4 FOLLOWER]: Advancing to term 5
I20260129 02:59:52.655624 20434 raft_consensus.cc:2468] T 2ac659d732314e1288849f08fe5e074e P 197b880fa39d432a9b472845876bfaf0 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7a9daf5d42a84615a6383cfec9a10608 in term 3.
I20260129 02:59:52.674711 20588 raft_consensus.cc:3060] T 3906b080a5b34a74b8b64128635e8997 P 1c50de89a5a845668696dcb28c3badd6 [term 3 FOLLOWER]: Advancing to term 4
I20260129 02:59:52.676069 20582 raft_consensus.cc:1275] T e4d18c35b0a7493e85d0a4b92956a48f P 1c50de89a5a845668696dcb28c3badd6 [term 5 FOLLOWER]: Refusing update from remote peer 974b802141bd4f13beffef2aa03f88f9: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 5 index: 3. (term mismatch)
I20260129 02:59:52.642187 22160 raft_consensus.cc:493] T 76f713c3331540d2a5b76164103e9eb1 P 974b802141bd4f13beffef2aa03f88f9 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260129 02:59:52.614241 22719 consensus_queue.cc:1048] T 2ac659d732314e1288849f08fe5e074e P 974b802141bd4f13beffef2aa03f88f9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.042s
I20260129 02:59:52.624573 22708 raft_consensus.cc:697] T 75be4bd6bd45474795fd472eff2953ac P 1c50de89a5a845668696dcb28c3badd6 [term 9 LEADER]: Becoming Leader. State: Replica: 1c50de89a5a845668696dcb28c3badd6, State: Running, Role: LEADER
I20260129 02:59:52.676453 20582 pending_rounds.cc:85] T e4d18c35b0a7493e85d0a4b92956a48f P 1c50de89a5a845668696dcb28c3badd6: Aborting all ops after (but not including) 2
W20260129 02:59:52.679232 22848 log.cc:927] Time spent T 368953c9102643b2a21f99af36b51905 P 197b880fa39d432a9b472845876bfaf0: Append to log took a long time: real 0.066s user 0.001s sys 0.000s
I20260129 02:59:52.652613 22343 raft_consensus.cc:2804] T add8d4923d1a489e86882c3a25692b95 P 7a9daf5d42a84615a6383cfec9a10608 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260129 02:59:52.673446 22817 raft_consensus.cc:493] T dc2f4cb752cf4d59b5f6bc689f5d8104 P 197b880fa39d432a9b472845876bfaf0 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 1c50de89a5a845668696dcb28c3badd6)
I20260129 02:59:52.682308 22581 consensus_queue.cc:260] T a4c1226b987b468fa84dfc6ba0e2dbe1 P 1c50de89a5a845668696dcb28c3badd6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:59:52.684185 22736 raft_consensus.cc:2804] T e7d83d4ccf684e76acb29a0dc844e1b8 P 197b880fa39d432a9b472845876bfaf0 [term 5 FOLLOWER]: Leader election won for term 5
I20260129 02:59:52.672757 21127 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "505580c93c694bbb8ea96fdf9b17ab46" candidate_uuid: "974b802141bd4f13beffef2aa03f88f9" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "7a9daf5d42a84615a6383cfec9a10608"
I20260129 02:59:52.672816 21128 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3e2d15f8640b407195210c476e09df91" candidate_uuid: "1c50de89a5a845668696dcb28c3badd6" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "7a9daf5d42a84615a6383cfec9a10608"
I20260129 02:59:52.689420 20581 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e33a417cf9c54cf5af73a4ee3aac4d3a" candidate_uuid: "974b802141bd4f13beffef2aa03f88f9" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "1c50de89a5a845668696dcb28c3badd6"
I20260129 02:59:52.642422 22847 raft_consensus.cc:2804] T a5a05fbcbbcf4916bbc01fd1a5b5e90d P 974b802141bd4f13beffef2aa03f88f9 [term 7 FOLLOWER]: Leader election won for term 7
I20260129 02:59:52.692660 20582 pending_rounds.cc:107] T e4d18c35b0a7493e85d0a4b92956a48f P 1c50de89a5a845668696dcb28c3badd6: Aborting uncommitted NO_OP operation due to leader change: 3.3
I20260129 02:59:52.652432 22831 raft_consensus.cc:2804] T a644c02b16af47679824ccd060ac10fd P 7a9daf5d42a84615a6383cfec9a10608 [term 4 FOLLOWER]: Leader election won for term 4
I20260129 02:59:52.679033 22745 consensus_queue.cc:1048] T e4d18c35b0a7493e85d0a4b92956a48f P 974b802141bd4f13beffef2aa03f88f9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260129 02:59:52.725576 22343 raft_consensus.cc:493] T add8d4923d1a489e86882c3a25692b95 P 7a9daf5d42a84615a6383cfec9a10608 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260129 02:59:52.660251 22844 raft_consensus.cc:2804] T add8d4923d1a489e86882c3a25692b95 P 974b802141bd4f13beffef2aa03f88f9 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260129 02:59:52.650983 22493 raft_consensus.cc:493] T 1299723a4476420ab02fd8bfd7aa0645 P 974b802141bd4f13beffef2aa03f88f9 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260129 02:59:52.729610 20556 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.211.2:34517, user_credentials={real_user=slave}} blocked reactor thread for 450393us
W20260129 02:59:52.735028 20407 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.211.2:34517, user_credentials={real_user=slave}} blocked reactor thread for 57197us
W20260129 02:59:52.735663 20405 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.211.4:40439, user_credentials={real_user=slave}} blocked reactor thread for 122954us
W20260129 02:59:52.792292 21099 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260129 02:59:52.793215 21099 leader_election.cc:304] T 87d5fe48a44a408aaf03e894a67ecb2f P 7a9daf5d42a84615a6383cfec9a10608 [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7a9daf5d42a84615a6383cfec9a10608; no voters: 197b880fa39d432a9b472845876bfaf0, 974b802141bd4f13beffef2aa03f88f9
I20260129 02:59:52.668550 22852 raft_consensus.cc:2804] T cedce331e1584ac8b636c9b8a72b2ba0 P 197b880fa39d432a9b472845876bfaf0 [term 6 FOLLOWER]: Leader election won for term 6
I20260129 02:59:52.768771 20582 raft_consensus.cc:2889] T e4d18c35b0a7493e85d0a4b92956a48f P 1c50de89a5a845668696dcb28c3badd6 [term 5 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
W20260129 02:59:52.769349 20553 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.211.1:42027, user_credentials={real_user=slave}} blocked reactor thread for 524389us
I20260129 02:59:52.794523 20515 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3e2d15f8640b407195210c476e09df91" candidate_uuid: "1c50de89a5a845668696dcb28c3badd6" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "974b802141bd4f13beffef2aa03f88f9" is_pre_election: true
I20260129 02:59:52.795284 20515 raft_consensus.cc:2468] T 3e2d15f8640b407195210c476e09df91 P 974b802141bd4f13beffef2aa03f88f9 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1c50de89a5a845668696dcb28c3badd6 in term 2.
I20260129 02:59:52.734704 22237 raft_consensus.cc:697] T c934d31efd3f4e5195605ddc354d14e1 P 197b880fa39d432a9b472845876bfaf0 [term 5 LEADER]: Becoming Leader. State: Replica: 197b880fa39d432a9b472845876bfaf0, State: Running, Role: LEADER
I20260129 02:59:52.704537 20436 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "17d6431cc77a491e8464d68db4081a51" candidate_uuid: "974b802141bd4f13beffef2aa03f88f9" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "197b880fa39d432a9b472845876bfaf0" is_pre_election: true
I20260129 02:59:52.705077 22641 consensus_queue.cc:1059] T 75be4bd6bd45474795fd472eff2953ac P 197b880fa39d432a9b472845876bfaf0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 }, Status: INVALID_TERM, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.075s
I20260129 02:59:52.704931 22796 consensus_queue.cc:1059] T 4e420f90884f4796997b5ea41e49e411 P 197b880fa39d432a9b472845876bfaf0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 }, Status: INVALID_TERM, Last received: 4.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.075s
W20260129 02:59:52.803047 20406 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260129 02:59:52.799413 20553 leader_election.cc:304] T a1a39dcd328e4268ae41b988c66b4a75 P 1c50de89a5a845668696dcb28c3badd6 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 197b880fa39d432a9b472845876bfaf0, 1c50de89a5a845668696dcb28c3badd6; no voters:
I20260129 02:59:52.803920 20507 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "cedce331e1584ac8b636c9b8a72b2ba0" candidate_uuid: "1c50de89a5a845668696dcb28c3badd6" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "974b802141bd4f13beffef2aa03f88f9" is_pre_election: true
I20260129 02:59:52.777709 20433 raft_consensus.cc:2468] T 3906b080a5b34a74b8b64128635e8997 P 197b880fa39d432a9b472845876bfaf0 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 974b802141bd4f13beffef2aa03f88f9 in term 4.
W20260129 02:59:52.716645 20404 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260129 02:59:52.627314 22425 consensus_queue.cc:260] T 7d9494f5584d4d5db8b9adf1c8fc888b P 1c50de89a5a845668696dcb28c3badd6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:59:52.780428 20584 raft_consensus.cc:1240] T 42d27a28d89b409e8807ec0efd97af69 P 1c50de89a5a845668696dcb28c3badd6 [term 6 LEADER]: Rejecting Update request from peer 7a9daf5d42a84615a6383cfec9a10608 for earlier term 5. Current term is 6. Ops: []
I20260129 02:59:52.780994 21128 raft_consensus.cc:3060] T 3e2d15f8640b407195210c476e09df91 P 7a9daf5d42a84615a6383cfec9a10608 [term 2 FOLLOWER]: Advancing to term 3
I20260129 02:59:52.782624 22742 raft_consensus.cc:515] T febffe4a8b394368b69b1d164308fb9a P 7a9daf5d42a84615a6383cfec9a10608 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } }
I20260129 02:59:52.643334 22786 consensus_queue.cc:260] T 4f8f3a38f6e24ba4857875181f447442 P 1c50de89a5a845668696dcb28c3badd6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
W20260129 02:59:52.784260 20482 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260129 02:59:52.814365 20588 raft_consensus.cc:2468] T 3906b080a5b34a74b8b64128635e8997 P 1c50de89a5a845668696dcb28c3badd6 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 974b802141bd4f13beffef2aa03f88f9 in term 4.
I20260129 02:59:52.676456 22656 consensus_queue.cc:1059] T 18f30ebaa4174fd28c958e72d0cc7e24 P 1c50de89a5a845668696dcb28c3badd6 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false }, Status: INVALID_TERM, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.093s
I20260129 02:59:52.797006 20435 raft_consensus.cc:2410] T a8479d1fe0bd41e5946186e3a6ca8c7a P 197b880fa39d432a9b472845876bfaf0 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 974b802141bd4f13beffef2aa03f88f9 for term 5 because replica has last-logged OpId of term: 3 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 2.
I20260129 02:59:52.800153 20515 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "75f45538dc704d9a8ec4ff8ad9f3ea5b" candidate_uuid: "1c50de89a5a845668696dcb28c3badd6" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "974b802141bd4f13beffef2aa03f88f9" is_pre_election: true
I20260129 02:59:52.777268 20438 raft_consensus.cc:2468] T e33a417cf9c54cf5af73a4ee3aac4d3a P 197b880fa39d432a9b472845876bfaf0 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 974b802141bd4f13beffef2aa03f88f9 in term 6.
I20260129 02:59:52.804984 20507 raft_consensus.cc:2393] T cedce331e1584ac8b636c9b8a72b2ba0 P 974b802141bd4f13beffef2aa03f88f9 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1c50de89a5a845668696dcb28c3badd6 in current term 6: Already voted for candidate 197b880fa39d432a9b472845876bfaf0 in this term.
W20260129 02:59:52.805860 20404 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260129 02:59:52.819033 20479 leader_election.cc:304] T 3906b080a5b34a74b8b64128635e8997 P 974b802141bd4f13beffef2aa03f88f9 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 197b880fa39d432a9b472845876bfaf0, 974b802141bd4f13beffef2aa03f88f9; no voters:
I20260129 02:59:52.776716 22581 raft_consensus.cc:3060] T a4c1226b987b468fa84dfc6ba0e2dbe1 P 1c50de89a5a845668696dcb28c3badd6 [term 3 FOLLOWER]: Advancing to term 4
I20260129 02:59:52.770944 22736 raft_consensus.cc:697] T e7d83d4ccf684e76acb29a0dc844e1b8 P 197b880fa39d432a9b472845876bfaf0 [term 5 LEADER]: Becoming Leader. State: Replica: 197b880fa39d432a9b472845876bfaf0, State: Running, Role: LEADER
I20260129 02:59:52.839319 20584 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:48526 (request call id 260) took 2459 ms. Trace:
I20260129 02:59:52.839913 20584 rpcz_store.cc:276] 0129 02:59:50.379692 (+ 0us) service_pool.cc:168] Inserting onto call queue
0129 02:59:51.983586 (+1603894us) service_pool.cc:225] Handling call
0129 02:59:52.004249 (+ 20663us) raft_consensus.cc:1471] Updating replica for 0 ops
0129 02:59:52.806324 (+802075us) raft_consensus.cc:1697] Filling consensus response to leader.
0129 02:59:52.838993 (+ 32669us) spinlock_profiling.cc:243] Waited 776 ms on lock 0x7b5400134f54. stack: 00007f8fe044e373 00007f8fe044e270 00007f8fdfda0574 00007f8fe9a5c6b4 00007f8fe9a5c649 00007f8fe34975a1 00007f8fe352ea81 00007f8fe352d4be 00007f8fe84e1f49 00007f8fe1a6ed34 00007f8fe1a6ecc2 00007f8fe1a6ebf1 00007f8fe1a6eb6c 00007f8fe1a6de22 00007f8fe186d34c 00007f8fe186c786
0129 02:59:52.839238 (+ 245us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1784526592}
I20260129 02:59:52.809832 20513 raft_consensus.cc:1240] T 7d9494f5584d4d5db8b9adf1c8fc888b P 974b802141bd4f13beffef2aa03f88f9 [term 3 FOLLOWER]: Rejecting Update request from peer 1c50de89a5a845668696dcb28c3badd6 for earlier term 2. Current term is 3. Ops: []
I20260129 02:59:52.810133 20553 leader_election.cc:304] T 724a236524cb4e7ba3428880681d6cc2 P 1c50de89a5a845668696dcb28c3badd6 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 197b880fa39d432a9b472845876bfaf0, 1c50de89a5a845668696dcb28c3badd6; no voters:
I20260129 02:59:52.738046 22817 raft_consensus.cc:515] T dc2f4cb752cf4d59b5f6bc689f5d8104 P 197b880fa39d432a9b472845876bfaf0 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } }
I20260129 02:59:52.812825 20516 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7a9a69cd82724b1faed1d10aa091fc4a" candidate_uuid: "1c50de89a5a845668696dcb28c3badd6" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "974b802141bd4f13beffef2aa03f88f9" is_pre_election: true
I20260129 02:59:52.779718 22667 consensus_queue.cc:1048] T 1299723a4476420ab02fd8bfd7aa0645 P 7a9daf5d42a84615a6383cfec9a10608 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.050s
I20260129 02:59:52.786211 20589 raft_consensus.cc:1240] T 75f45538dc704d9a8ec4ff8ad9f3ea5b P 1c50de89a5a845668696dcb28c3badd6 [term 3 FOLLOWER]: Rejecting Update request from peer 7a9daf5d42a84615a6383cfec9a10608 for earlier term 2. Current term is 3. Ops: []
I20260129 02:59:52.820673 20438 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b189f08e16a74a11980cc55d99359a0a" candidate_uuid: "974b802141bd4f13beffef2aa03f88f9" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "197b880fa39d432a9b472845876bfaf0"
I20260129 02:59:52.857473 20438 raft_consensus.cc:3060] T b189f08e16a74a11980cc55d99359a0a P 197b880fa39d432a9b472845876bfaf0 [term 6 FOLLOWER]: Advancing to term 7
I20260129 02:59:52.789274 22343 raft_consensus.cc:3060] T add8d4923d1a489e86882c3a25692b95 P 7a9daf5d42a84615a6383cfec9a10608 [term 3 FOLLOWER]: Advancing to term 4
I20260129 02:59:52.821426 20479 leader_election.cc:304] T e33a417cf9c54cf5af73a4ee3aac4d3a P 974b802141bd4f13beffef2aa03f88f9 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 197b880fa39d432a9b472845876bfaf0, 974b802141bd4f13beffef2aa03f88f9; no voters:
I20260129 02:59:52.705508 22160 raft_consensus.cc:515] T 76f713c3331540d2a5b76164103e9eb1 P 974b802141bd4f13beffef2aa03f88f9 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:59:52.825287 20440 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7a9a69cd82724b1faed1d10aa091fc4a" candidate_uuid: "974b802141bd4f13beffef2aa03f88f9" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "197b880fa39d432a9b472845876bfaf0" is_pre_election: true
I20260129 02:59:52.825855 20515 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a1a39dcd328e4268ae41b988c66b4a75" candidate_uuid: "1c50de89a5a845668696dcb28c3badd6" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "974b802141bd4f13beffef2aa03f88f9"
I20260129 02:59:52.797415 22844 raft_consensus.cc:493] T add8d4923d1a489e86882c3a25692b95 P 974b802141bd4f13beffef2aa03f88f9 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260129 02:59:52.824823 20480 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.211.4:40439, user_credentials={real_user=slave}} blocked reactor thread for 189542us
I20260129 02:59:52.869432 20480 leader_election.cc:304] T 0678b8b468bd43cdbc3e485900844821 P 974b802141bd4f13beffef2aa03f88f9 [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 974b802141bd4f13beffef2aa03f88f9; no voters: 197b880fa39d432a9b472845876bfaf0, 7a9daf5d42a84615a6383cfec9a10608
I20260129 02:59:52.785313 22847 raft_consensus.cc:697] T a5a05fbcbbcf4916bbc01fd1a5b5e90d P 974b802141bd4f13beffef2aa03f88f9 [term 7 LEADER]: Becoming Leader. State: Replica: 974b802141bd4f13beffef2aa03f88f9, State: Running, Role: LEADER
W20260129 02:59:52.826699 20481 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260129 02:59:52.874821 20407 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.211.2:34517, user_credentials={real_user=slave}} blocked reactor thread for 59842us
I20260129 02:59:52.740639 22691 consensus_queue.cc:1048] T 87d5fe48a44a408aaf03e894a67ecb2f P 197b880fa39d432a9b472845876bfaf0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.064s
W20260129 02:59:52.876917 21100 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.211.3:37735, user_credentials={real_user=slave}} blocked reactor thread for 86239.6us
I20260129 02:59:52.848258 20512 raft_consensus.cc:1240] T 45cc5f3d2688486d9b6dbe7e666dedde P 974b802141bd4f13beffef2aa03f88f9 [term 4 FOLLOWER]: Rejecting Update request from peer 1c50de89a5a845668696dcb28c3badd6 for earlier term 3. Current term is 4. Ops: []
I20260129 02:59:52.850934 20516 raft_consensus.cc:2468] T 7a9a69cd82724b1faed1d10aa091fc4a P 974b802141bd4f13beffef2aa03f88f9 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1c50de89a5a845668696dcb28c3badd6 in term 5.
I20260129 02:59:52.738154 22809 consensus_queue.cc:1059] T 4408006eb5174de1896b037cf3f18457 P 197b880fa39d432a9b472845876bfaf0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 }, Status: INVALID_TERM, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.064s
I20260129 02:59:52.855393 20434 raft_consensus.cc:1240] T 3ccac82a01a2445fa5b0ce6c0fcf2295 P 197b880fa39d432a9b472845876bfaf0 [term 4 FOLLOWER]: Rejecting Update request from peer 974b802141bd4f13beffef2aa03f88f9 for earlier term 3. Current term is 4. Ops: [3.5-3.5]
W20260129 02:59:52.865751 22835 log.cc:927] Time spent T 87d5fe48a44a408aaf03e894a67ecb2f P 974b802141bd4f13beffef2aa03f88f9: Append to log took a long time: real 0.055s user 0.001s sys 0.000s
I20260129 02:59:52.865007 22664 raft_consensus.cc:2804] T 724a236524cb4e7ba3428880681d6cc2 P 1c50de89a5a845668696dcb28c3badd6 [term 4 FOLLOWER]: Leader election won for term 4
I20260129 02:59:52.823275 22554 consensus_queue.cc:1059] T 4e3805ffa70242e8ade66da260dddf66 P 1c50de89a5a845668696dcb28c3badd6 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 }, Status: INVALID_TERM, Last received: 3.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.055s
I20260129 02:59:52.809149 22854 raft_consensus.cc:2804] T a1a39dcd328e4268ae41b988c66b4a75 P 1c50de89a5a845668696dcb28c3badd6 [term 4 FOLLOWER]: Leader election won for term 4
W20260129 02:59:52.854471 22849 log.cc:927] Time spent T a4c1226b987b468fa84dfc6ba0e2dbe1 P 197b880fa39d432a9b472845876bfaf0: Append to log took a long time: real 0.309s user 0.001s sys 0.001s
I20260129 02:59:52.814749 22820 leader_election.cc:290] T b189f08e16a74a11980cc55d99359a0a P 974b802141bd4f13beffef2aa03f88f9 [CANDIDATE]: Term 7 election: Requested vote from peers 197b880fa39d432a9b472845876bfaf0 (127.18.211.1:42027), 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3:37735)
I20260129 02:59:52.840660 22807 raft_consensus.cc:2804] T 3906b080a5b34a74b8b64128635e8997 P 974b802141bd4f13beffef2aa03f88f9 [term 4 FOLLOWER]: Leader election won for term 4
I20260129 02:59:52.812220 20436 raft_consensus.cc:2468] T 17d6431cc77a491e8464d68db4081a51 P 197b880fa39d432a9b472845876bfaf0 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 974b802141bd4f13beffef2aa03f88f9 in term 3.
I20260129 02:59:52.881755 20516 raft_consensus.cc:1240] T a644c02b16af47679824ccd060ac10fd P 974b802141bd4f13beffef2aa03f88f9 [term 4 FOLLOWER]: Rejecting Update request from peer 1c50de89a5a845668696dcb28c3badd6 for earlier term 3. Current term is 4. Ops: [3.5-3.5]
I20260129 02:59:52.883973 20433 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "76f713c3331540d2a5b76164103e9eb1" candidate_uuid: "1c50de89a5a845668696dcb28c3badd6" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "197b880fa39d432a9b472845876bfaf0"
I20260129 02:59:52.884756 20513 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e4d18c35b0a7493e85d0a4b92956a48f" candidate_uuid: "1c50de89a5a845668696dcb28c3badd6" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "974b802141bd4f13beffef2aa03f88f9"
I20260129 02:59:52.937129 20513 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3e2d15f8640b407195210c476e09df91" candidate_uuid: "1c50de89a5a845668696dcb28c3badd6" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "974b802141bd4f13beffef2aa03f88f9"
I20260129 02:59:52.937011 20516 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "cb0b3f259ebc4310927338a8740e87b8" candidate_uuid: "1c50de89a5a845668696dcb28c3badd6" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "974b802141bd4f13beffef2aa03f88f9" is_pre_election: true
I20260129 02:59:52.937779 20513 raft_consensus.cc:3060] T 3e2d15f8640b407195210c476e09df91 P 974b802141bd4f13beffef2aa03f88f9 [term 2 FOLLOWER]: Advancing to term 3
I20260129 02:59:52.861426 20439 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55676 (request call id 1292) took 2033 ms. Trace:
I20260129 02:59:52.856899 22425 raft_consensus.cc:3060] T 7d9494f5584d4d5db8b9adf1c8fc888b P 1c50de89a5a845668696dcb28c3badd6 [term 2 FOLLOWER]: Advancing to term 3
I20260129 02:59:52.938444 20439 rpcz_store.cc:276] 0129 02:59:50.827685 (+ 0us) service_pool.cc:168] Inserting onto call queue
0129 02:59:51.362521 (+534836us) service_pool.cc:225] Handling call
0129 02:59:51.394039 (+ 31518us) raft_consensus.cc:1471] Updating replica for 2 ops
0129 02:59:51.613023 (+218984us) raft_consensus.cc:1534] Early marking committed up to index 0
0129 02:59:51.613060 (+ 37us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0129 02:59:51.613678 (+ 618us) log.cc:844] Serialized 52 byte log entry
0129 02:59:52.301667 (+687989us) raft_consensus.cc:1645] Marking committed up to 2
0129 02:59:52.301725 (+ 58us) raft_consensus.cc:1697] Filling consensus response to leader.
0129 02:59:52.301850 (+ 125us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0129 02:59:52.861129 (+559279us) raft_consensus.cc:1685] finished
0129 02:59:52.861156 (+ 27us) raft_consensus.cc:1691] UpdateReplicas() finished
0129 02:59:52.861351 (+ 195us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":218585,"raft.queue_time_us":218926,"raft.run_cpu_time_us":108,"raft.run_wall_time_us":109,"thread_start_us":687566,"threads_started":1,"wal-append.queue_time_us":946591}
I20260129 02:59:52.845158 22644 raft_consensus.cc:493] T a609ac02fae84725ab743f95f51d33ce P 197b880fa39d432a9b472845876bfaf0 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 974b802141bd4f13beffef2aa03f88f9)
I20260129 02:59:52.826788 22708 consensus_queue.cc:237] T 75be4bd6bd45474795fd472eff2953ac P 1c50de89a5a845668696dcb28c3badd6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 7.5, Last appended by leader: 5, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:59:52.814457 22647 consensus_queue.cc:1059] T 0678b8b468bd43cdbc3e485900844821 P 197b880fa39d432a9b472845876bfaf0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 }, Status: INVALID_TERM, Last received: 5.7, Next index: 8, Last known committed idx: 6, Time since last communication: 0.058s
W20260129 02:59:52.866139 20479 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.211.1:42027, user_credentials={real_user=slave}} blocked reactor thread for 44836.4us
I20260129 02:59:52.851454 22831 raft_consensus.cc:697] T a644c02b16af47679824ccd060ac10fd P 7a9daf5d42a84615a6383cfec9a10608 [term 4 LEADER]: Becoming Leader. State: Replica: 7a9daf5d42a84615a6383cfec9a10608, State: Running, Role: LEADER
I20260129 02:59:52.909044 20441 raft_consensus.cc:1240] T e33a417cf9c54cf5af73a4ee3aac4d3a P 197b880fa39d432a9b472845876bfaf0 [term 6 FOLLOWER]: Rejecting Update request from peer 1c50de89a5a845668696dcb28c3badd6 for earlier term 5. Current term is 6. Ops: [5.3-5.3]
I20260129 02:59:52.909371 20438 raft_consensus.cc:2468] T b189f08e16a74a11980cc55d99359a0a P 197b880fa39d432a9b472845876bfaf0 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 974b802141bd4f13beffef2aa03f88f9 in term 7.
W20260129 02:59:52.910017 20405 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.211.4:40439, user_credentials={real_user=slave}} blocked reactor thread for 117879us
I20260129 02:59:52.911837 20440 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4408006eb5174de1896b037cf3f18457" candidate_uuid: "1c50de89a5a845668696dcb28c3badd6" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "197b880fa39d432a9b472845876bfaf0"
I20260129 02:59:52.874022 22184 raft_consensus.cc:2804] T e33a417cf9c54cf5af73a4ee3aac4d3a P 974b802141bd4f13beffef2aa03f88f9 [term 6 FOLLOWER]: Leader election won for term 6
I20260129 02:59:52.872018 22844 raft_consensus.cc:3060] T add8d4923d1a489e86882c3a25692b95 P 974b802141bd4f13beffef2aa03f88f9 [term 3 FOLLOWER]: Advancing to term 4
I20260129 02:59:52.928853 20434 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3906b080a5b34a74b8b64128635e8997" candidate_uuid: "1c50de89a5a845668696dcb28c3badd6" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "197b880fa39d432a9b472845876bfaf0" is_pre_election: true
I20260129 02:59:52.854769 22540 consensus_queue.cc:1048] T febffe4a8b394368b69b1d164308fb9a P 197b880fa39d432a9b472845876bfaf0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.048s
W20260129 02:59:52.933727 20480 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.211.4:40439, user_credentials={real_user=slave}} blocked reactor thread for 64403.6us
I20260129 02:59:53.028807 20589 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b189f08e16a74a11980cc55d99359a0a" candidate_uuid: "974b802141bd4f13beffef2aa03f88f9" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "1c50de89a5a845668696dcb28c3badd6"
I20260129 02:59:53.029609 20589 raft_consensus.cc:3060] T b189f08e16a74a11980cc55d99359a0a P 1c50de89a5a845668696dcb28c3badd6 [term 6 FOLLOWER]: Advancing to term 7
I20260129 02:59:52.938707 20516 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4408006eb5174de1896b037cf3f18457" candidate_uuid: "1c50de89a5a845668696dcb28c3badd6" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "974b802141bd4f13beffef2aa03f88f9"
I20260129 02:59:52.823432 22493 raft_consensus.cc:515] T 1299723a4476420ab02fd8bfd7aa0645 P 974b802141bd4f13beffef2aa03f88f9 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } attrs { replace: true } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } }
I20260129 02:59:53.040876 20516 raft_consensus.cc:2393] T 4408006eb5174de1896b037cf3f18457 P 974b802141bd4f13beffef2aa03f88f9 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 1c50de89a5a845668696dcb28c3badd6 in current term 3: Already voted for candidate 974b802141bd4f13beffef2aa03f88f9 in this term.
I20260129 02:59:52.953222 20515 raft_consensus.cc:2468] T a1a39dcd328e4268ae41b988c66b4a75 P 974b802141bd4f13beffef2aa03f88f9 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1c50de89a5a845668696dcb28c3badd6 in term 4.
I20260129 02:59:53.042584 20515 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "724a236524cb4e7ba3428880681d6cc2" candidate_uuid: "1c50de89a5a845668696dcb28c3badd6" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "974b802141bd4f13beffef2aa03f88f9"
I20260129 02:59:53.043371 20515 raft_consensus.cc:3060] T 724a236524cb4e7ba3428880681d6cc2 P 974b802141bd4f13beffef2aa03f88f9 [term 3 FOLLOWER]: Advancing to term 4
W20260129 02:59:53.045583 20405 connection.cc:232] RPC call timeout handler was delayed by -2.4153859520000651s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.202s for a response
W20260129 02:59:53.046326 20405 connection.cc:232] RPC call timeout handler was delayed by -2.3946859520000316s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.20430000000000004s for a response
I20260129 02:59:52.901376 22786 raft_consensus.cc:3060] T 4f8f3a38f6e24ba4857875181f447442 P 1c50de89a5a845668696dcb28c3badd6 [term 4 FOLLOWER]: Advancing to term 5
I20260129 02:59:53.049801 21133 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "42d27a28d89b409e8807ec0efd97af69" candidate_uuid: "197b880fa39d432a9b472845876bfaf0" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "7a9daf5d42a84615a6383cfec9a10608" is_pre_election: true
I20260129 02:59:52.990622 22688 raft_consensus.cc:3055] T 4408006eb5174de1896b037cf3f18457 P 197b880fa39d432a9b472845876bfaf0 [term 2 LEADER]: Stepping down as leader of term 2
I20260129 02:59:53.008440 22554 raft_consensus.cc:493] T 913fc7d751ed4b7388d889aac30a5f9b P 1c50de89a5a845668696dcb28c3badd6 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260129 02:59:53.006016 20556 leader_election.cc:304] T 75f45538dc704d9a8ec4ff8ad9f3ea5b P 1c50de89a5a845668696dcb28c3badd6 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1c50de89a5a845668696dcb28c3badd6; no voters: 7a9daf5d42a84615a6383cfec9a10608, 974b802141bd4f13beffef2aa03f88f9
W20260129 02:59:53.062561 20556 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.211.2:34517, user_credentials={real_user=slave}} blocked reactor thread for 56621.3us
I20260129 02:59:53.063432 20556 leader_election.cc:304] T cedce331e1584ac8b636c9b8a72b2ba0 P 1c50de89a5a845668696dcb28c3badd6 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1c50de89a5a845668696dcb28c3badd6; no voters: 197b880fa39d432a9b472845876bfaf0, 974b802141bd4f13beffef2aa03f88f9
W20260129 02:59:53.009661 20553 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260129 02:59:52.921083 22687 raft_consensus.cc:2749] T 0678b8b468bd43cdbc3e485900844821 P 974b802141bd4f13beffef2aa03f88f9 [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
W20260129 02:59:53.011765 21100 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.211.3:37735, user_credentials={real_user=slave}} blocked reactor thread for 133892us
I20260129 02:59:52.887985 22719 consensus_queue.cc:1059] T 3ccac82a01a2445fa5b0ce6c0fcf2295 P 974b802141bd4f13beffef2aa03f88f9 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false }, Status: INVALID_TERM, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.077s
I20260129 02:59:53.071120 21134 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "96609ac68bdf4993a12e1c23355ca013" candidate_uuid: "197b880fa39d432a9b472845876bfaf0" candidate_term: 5 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "7a9daf5d42a84615a6383cfec9a10608" is_pre_election: true
I20260129 02:59:53.072202 21134 raft_consensus.cc:2393] T 96609ac68bdf4993a12e1c23355ca013 P 7a9daf5d42a84615a6383cfec9a10608 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 197b880fa39d432a9b472845876bfaf0 in current term 5: Already voted for candidate 7a9daf5d42a84615a6383cfec9a10608 in this term.
I20260129 02:59:52.906960 21129 raft_consensus.cc:2468] T 4e420f90884f4796997b5ea41e49e411 P 7a9daf5d42a84615a6383cfec9a10608 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 974b802141bd4f13beffef2aa03f88f9 in term 5.
I20260129 02:59:53.073004 20516 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "505580c93c694bbb8ea96fdf9b17ab46" candidate_uuid: "1c50de89a5a845668696dcb28c3badd6" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "974b802141bd4f13beffef2aa03f88f9"
I20260129 02:59:53.017140 22820 raft_consensus.cc:493] T 4f5a77e94fef4b16a2ba7b067c8b144a P 974b802141bd4f13beffef2aa03f88f9 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 1c50de89a5a845668696dcb28c3badd6)
I20260129 02:59:53.018158 20437 raft_consensus.cc:1240] T 7d9494f5584d4d5db8b9adf1c8fc888b P 197b880fa39d432a9b472845876bfaf0 [term 3 LEADER]: Rejecting Update request from peer 1c50de89a5a845668696dcb28c3badd6 for earlier term 2. Current term is 3. Ops: []
I20260129 02:59:53.020813 20434 raft_consensus.cc:2393] T 3906b080a5b34a74b8b64128635e8997 P 197b880fa39d432a9b472845876bfaf0 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1c50de89a5a845668696dcb28c3badd6 in current term 4: Already voted for candidate 974b802141bd4f13beffef2aa03f88f9 in this term.
I20260129 02:59:52.929104 20512 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3906b080a5b34a74b8b64128635e8997" candidate_uuid: "1c50de89a5a845668696dcb28c3badd6" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "974b802141bd4f13beffef2aa03f88f9" is_pre_election: true
W20260129 02:59:52.892457 21099 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.211.1:42027, user_credentials={real_user=slave}} blocked reactor thread for 99338.8us
I20260129 02:59:53.005962 20513 raft_consensus.cc:2468] T 3e2d15f8640b407195210c476e09df91 P 974b802141bd4f13beffef2aa03f88f9 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1c50de89a5a845668696dcb28c3badd6 in term 3.
I20260129 02:59:53.078612 20513 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "76f713c3331540d2a5b76164103e9eb1" candidate_uuid: "1c50de89a5a845668696dcb28c3badd6" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "974b802141bd4f13beffef2aa03f88f9"
I20260129 02:59:53.079463 20513 raft_consensus.cc:3060] T 76f713c3331540d2a5b76164103e9eb1 P 974b802141bd4f13beffef2aa03f88f9 [term 4 FOLLOWER]: Advancing to term 5
I20260129 02:59:52.964887 22873 raft_consensus.cc:2749] T 87d5fe48a44a408aaf03e894a67ecb2f P 7a9daf5d42a84615a6383cfec9a10608 [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
W20260129 02:59:53.053876 20554 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260129 02:59:52.867683 22736 consensus_queue.cc:237] T e7d83d4ccf684e76acb29a0dc844e1b8 P 197b880fa39d432a9b472845876bfaf0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:59:53.060808 20515 raft_consensus.cc:2468] T 724a236524cb4e7ba3428880681d6cc2 P 974b802141bd4f13beffef2aa03f88f9 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1c50de89a5a845668696dcb28c3badd6 in term 4.
I20260129 02:59:52.995790 22807 raft_consensus.cc:697] T 3906b080a5b34a74b8b64128635e8997 P 974b802141bd4f13beffef2aa03f88f9 [term 4 LEADER]: Becoming Leader. State: Replica: 974b802141bd4f13beffef2aa03f88f9, State: Running, Role: LEADER
I20260129 02:59:53.073843 20516 raft_consensus.cc:2393] T 505580c93c694bbb8ea96fdf9b17ab46 P 974b802141bd4f13beffef2aa03f88f9 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 1c50de89a5a845668696dcb28c3badd6 in current term 5: Already voted for candidate 974b802141bd4f13beffef2aa03f88f9 in this term.
I20260129 02:59:52.965706 22809 raft_consensus.cc:493] T c2633a6fc34c44579de348d208b0c01e P 197b880fa39d432a9b472845876bfaf0 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 1c50de89a5a845668696dcb28c3badd6)
W20260129 02:59:53.068285 22738 consensus_meta.cc:287] T 368953c9102643b2a21f99af36b51905 P 974b802141bd4f13beffef2aa03f88f9: Time spent flushing consensus metadata: real 0.606s user 0.014s sys 0.009s
I20260129 02:59:53.013738 22667 raft_consensus.cc:493] T 45cc5f3d2688486d9b6dbe7e666dedde P 7a9daf5d42a84615a6383cfec9a10608 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 1c50de89a5a845668696dcb28c3badd6)
W20260129 02:59:53.084993 20555 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260129 02:59:53.086687 20556 leader_election.cc:304] T 7a9a69cd82724b1faed1d10aa091fc4a P 1c50de89a5a845668696dcb28c3badd6 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1c50de89a5a845668696dcb28c3badd6, 974b802141bd4f13beffef2aa03f88f9; no voters: 197b880fa39d432a9b472845876bfaf0
I20260129 02:59:53.089681 21131 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4e3805ffa70242e8ade66da260dddf66" candidate_uuid: "197b880fa39d432a9b472845876bfaf0" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "7a9daf5d42a84615a6383cfec9a10608"
I20260129 02:59:53.102741 21131 raft_consensus.cc:3060] T 4e3805ffa70242e8ade66da260dddf66 P 7a9daf5d42a84615a6383cfec9a10608 [term 3 FOLLOWER]: Advancing to term 4
I20260129 02:59:52.902429 22237 consensus_queue.cc:237] T c934d31efd3f4e5195605ddc354d14e1 P 197b880fa39d432a9b472845876bfaf0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 3.6, Last appended by leader: 6, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } }
I20260129 02:59:53.099947 20515 raft_consensus.cc:1275] T 1ce4ddf5b10e4ae8b193b13b2e5936ee P 974b802141bd4f13beffef2aa03f88f9 [term 6 FOLLOWER]: Refusing update from remote peer 1c50de89a5a845668696dcb28c3badd6: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 6 index: 4. (index mismatch)
I20260129 02:59:53.100071 20584 tablet_service.cc:2066] Received LeaderStepDown RPC: tablet_id: "7d9494f5584d4d5db8b9adf1c8fc888b"
dest_uuid: "1c50de89a5a845668696dcb28c3badd6"
mode: GRACEFUL
new_leader_uuid: "197b880fa39d432a9b472845876bfaf0"
from {username='slave'} at 127.0.0.1:54860
I20260129 02:59:53.080888 22664 raft_consensus.cc:697] T 724a236524cb4e7ba3428880681d6cc2 P 1c50de89a5a845668696dcb28c3badd6 [term 4 LEADER]: Becoming Leader. State: Replica: 1c50de89a5a845668696dcb28c3badd6, State: Running, Role: LEADER
I20260129 02:59:53.107362 21136 raft_consensus.cc:2468] T 505580c93c694bbb8ea96fdf9b17ab46 P 7a9daf5d42a84615a6383cfec9a10608 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1c50de89a5a845668696dcb28c3badd6 in term 5.
W20260129 02:59:53.098500 22815 consensus_meta.cc:287] T 0f20af7fbe5d4db89a6f17948d98912c P 974b802141bd4f13beffef2aa03f88f9: Time spent flushing consensus metadata: real 0.612s user 0.005s sys 0.005s
I20260129 02:59:53.169284 20513 raft_consensus.cc:2468] T 76f713c3331540d2a5b76164103e9eb1 P 974b802141bd4f13beffef2aa03f88f9 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1c50de89a5a845668696dcb28c3badd6 in term 5.
I20260129 02:59:53.001657 22644 raft_consensus.cc:515] T a609ac02fae84725ab743f95f51d33ce P 197b880fa39d432a9b472845876bfaf0 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } }
W20260129 02:59:53.168412 20554 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260129 02:59:53.176551 20510 raft_consensus.cc:1275] T 368953c9102643b2a21f99af36b51905 P 974b802141bd4f13beffef2aa03f88f9 [term 6 FOLLOWER]: Refusing update from remote peer 1c50de89a5a845668696dcb28c3badd6: Log matching property violated. Preceding OpId in replica: term: 5 index: 4. Preceding OpId from leader: term: 6 index: 4. (term mismatch)
W20260129 02:59:53.177532 20407 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260129 02:59:53.067901 22852 raft_consensus.cc:697] T cedce331e1584ac8b636c9b8a72b2ba0 P 197b880fa39d432a9b472845876bfaf0 [term 6 LEADER]: Becoming Leader. State: Replica: 197b880fa39d432a9b472845876bfaf0, State: Running, Role: LEADER
W20260129 02:59:53.179448 20556 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.211.2:34517, user_credentials={real_user=slave}} blocked reactor thread for 92897.2us
I20260129 02:59:53.178386 21131 raft_consensus.cc:2468] T 4e3805ffa70242e8ade66da260dddf66 P 7a9daf5d42a84615a6383cfec9a10608 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 197b880fa39d432a9b472845876bfaf0 in term 4.
I20260129 02:59:53.180209 21136 raft_consensus.cc:1240] T 87d5fe48a44a408aaf03e894a67ecb2f P 7a9daf5d42a84615a6383cfec9a10608 [term 6 FOLLOWER]: Rejecting Update request from peer 197b880fa39d432a9b472845876bfaf0 for earlier term 5. Current term is 6. Ops: [5.7-5.7]
I20260129 02:59:53.057518 22688 raft_consensus.cc:740] T 4408006eb5174de1896b037cf3f18457 P 197b880fa39d432a9b472845876bfaf0 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 197b880fa39d432a9b472845876bfaf0, State: Running, Role: LEADER
I20260129 02:59:53.181912 21134 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2ac659d732314e1288849f08fe5e074e" candidate_uuid: "197b880fa39d432a9b472845876bfaf0" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "7a9daf5d42a84615a6383cfec9a10608" is_pre_election: true
I20260129 02:59:53.087466 22729 raft_consensus.cc:2749] T 4e420f90884f4796997b5ea41e49e411 P 7a9daf5d42a84615a6383cfec9a10608 [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260129 02:59:53.189314 20510 pending_rounds.cc:85] T 368953c9102643b2a21f99af36b51905 P 974b802141bd4f13beffef2aa03f88f9: Aborting all ops after (but not including) 3
I20260129 02:59:53.230911 20510 pending_rounds.cc:107] T 368953c9102643b2a21f99af36b51905 P 974b802141bd4f13beffef2aa03f88f9: Aborting uncommitted NO_OP operation due to leader change: 5.4
I20260129 02:59:53.231323 20510 raft_consensus.cc:2889] T 368953c9102643b2a21f99af36b51905 P 974b802141bd4f13beffef2aa03f88f9 [term 6 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
W20260129 02:59:53.231544 20555 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260129 02:59:53.195399 20556 leader_election.cc:304] T e4d18c35b0a7493e85d0a4b92956a48f P 1c50de89a5a845668696dcb28c3badd6 [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1c50de89a5a845668696dcb28c3badd6; no voters: 197b880fa39d432a9b472845876bfaf0, 974b802141bd4f13beffef2aa03f88f9
I20260129 02:59:53.181360 21130 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3ccac82a01a2445fa5b0ce6c0fcf2295" candidate_uuid: "197b880fa39d432a9b472845876bfaf0" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "7a9daf5d42a84615a6383cfec9a10608" is_pre_election: true
W20260129 02:59:53.176410 22843 log.cc:927] Time spent T a609ac02fae84725ab743f95f51d33ce P 7a9daf5d42a84615a6383cfec9a10608: Append to log took a long time: real 0.391s user 0.000s sys 0.001s
I20260129 02:59:53.073082 22554 raft_consensus.cc:515] T 913fc7d751ed4b7388d889aac30a5f9b P 1c50de89a5a845668696dcb28c3badd6 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:59:53.099187 21129 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:47844 (request call id 325) took 1201 ms. Trace:
I20260129 02:59:53.188118 22881 raft_consensus.cc:3055] T 4e3805ffa70242e8ade66da260dddf66 P 1c50de89a5a845668696dcb28c3badd6 [term 3 LEADER]: Stepping down as leader of term 3
I20260129 02:59:53.248469 22881 raft_consensus.cc:740] T 4e3805ffa70242e8ade66da260dddf66 P 1c50de89a5a845668696dcb28c3badd6 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 1c50de89a5a845668696dcb28c3badd6, State: Running, Role: LEADER
I20260129 02:59:53.248680 22872 consensus_queue.cc:1048] T 00e79d8f25a34676b1d866eb2068a2b2 P 7a9daf5d42a84615a6383cfec9a10608 [LEADER]: Connected to new peer: Peer: permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 6, Time since last communication: 0.001s
I20260129 02:59:53.249856 22881 consensus_queue.cc:260] T 4e3805ffa70242e8ade66da260dddf66 P 1c50de89a5a845668696dcb28c3badd6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 3.6, Last appended by leader: 6, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } }
I20260129 02:59:53.252610 22880 consensus_queue.cc:1048] T 00e79d8f25a34676b1d866eb2068a2b2 P 7a9daf5d42a84615a6383cfec9a10608 [LEADER]: Connected to new peer: Peer: permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 6, Time since last communication: 0.001s
I20260129 02:59:53.183630 22873 raft_consensus.cc:493] T 4f5a77e94fef4b16a2ba7b067c8b144a P 7a9daf5d42a84615a6383cfec9a10608 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 1c50de89a5a845668696dcb28c3badd6)
I20260129 02:59:53.190539 22184 raft_consensus.cc:697] T e33a417cf9c54cf5af73a4ee3aac4d3a P 974b802141bd4f13beffef2aa03f88f9 [term 6 LEADER]: Becoming Leader. State: Replica: 974b802141bd4f13beffef2aa03f88f9, State: Running, Role: LEADER
I20260129 02:59:53.263273 22817 leader_election.cc:290] T dc2f4cb752cf4d59b5f6bc689f5d8104 P 197b880fa39d432a9b472845876bfaf0 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3:37735), 7a9daf5d42a84615a6383cfec9a10608 (127.18.211.4:40439)
I20260129 02:59:53.096143 22820 raft_consensus.cc:515] T 4f5a77e94fef4b16a2ba7b067c8b144a P 974b802141bd4f13beffef2aa03f88f9 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } }
I20260129 02:59:53.222270 22763 raft_consensus.cc:493] T 75be4bd6bd45474795fd472eff2953ac P 974b802141bd4f13beffef2aa03f88f9 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260129 02:59:53.280941 20553 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.211.1:42027, user_credentials={real_user=slave}} blocked reactor thread for 179325us
W20260129 02:59:53.189787 20554 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260129 02:59:53.310072 21130 raft_consensus.cc:2393] T 3ccac82a01a2445fa5b0ce6c0fcf2295 P 7a9daf5d42a84615a6383cfec9a10608 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 197b880fa39d432a9b472845876bfaf0 in current term 4: Already voted for candidate 7a9daf5d42a84615a6383cfec9a10608 in this term.
W20260129 02:59:53.311048 22833 consensus_meta.cc:287] T 75f45538dc704d9a8ec4ff8ad9f3ea5b P 7a9daf5d42a84615a6383cfec9a10608: Time spent flushing consensus metadata: real 0.822s user 0.013s sys 0.000s
I20260129 02:59:53.311221 22729 raft_consensus.cc:493] T a609ac02fae84725ab743f95f51d33ce P 7a9daf5d42a84615a6383cfec9a10608 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 974b802141bd4f13beffef2aa03f88f9)
I20260129 02:59:53.062675 22847 consensus_queue.cc:237] T a5a05fbcbbcf4916bbc01fd1a5b5e90d P 974b802141bd4f13beffef2aa03f88f9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 5.4, Last appended by leader: 4, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:59:53.182384 22667 raft_consensus.cc:515] T 45cc5f3d2688486d9b6dbe7e666dedde P 7a9daf5d42a84615a6383cfec9a10608 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } }
I20260129 02:59:53.194358 22815 raft_consensus.cc:515] T 0f20af7fbe5d4db89a6f17948d98912c P 974b802141bd4f13beffef2aa03f88f9 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:59:53.263746 21127 raft_consensus.cc:1275] T 78aa39096868415d9afc2ffcb248e198 P 7a9daf5d42a84615a6383cfec9a10608 [term 5 FOLLOWER]: Refusing update from remote peer 1c50de89a5a845668696dcb28c3badd6: Log matching property violated. Preceding OpId in replica: term: 3 index: 5. Preceding OpId from leader: term: 5 index: 6. (index mismatch)
I20260129 02:59:53.256189 22742 leader_election.cc:290] T febffe4a8b394368b69b1d164308fb9a P 7a9daf5d42a84615a6383cfec9a10608 [CANDIDATE]: Term 4 election: Requested vote from peers 197b880fa39d432a9b472845876bfaf0 (127.18.211.1:42027), 974b802141bd4f13beffef2aa03f88f9 (127.18.211.2:34517)
W20260129 02:59:53.302119 22863 log.cc:927] Time spent T 87d5fe48a44a408aaf03e894a67ecb2f P 197b880fa39d432a9b472845876bfaf0: Append to log took a long time: real 0.439s user 0.000s sys 0.001s
I20260129 02:59:53.277241 22160 leader_election.cc:290] T 76f713c3331540d2a5b76164103e9eb1 P 974b802141bd4f13beffef2aa03f88f9 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 197b880fa39d432a9b472845876bfaf0 (127.18.211.1:42027), 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3:37735)
I20260129 02:59:53.275493 22865 raft_consensus.cc:493] T 45cc5f3d2688486d9b6dbe7e666dedde P 974b802141bd4f13beffef2aa03f88f9 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260129 02:59:53.321101 20556 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.211.2:34517, user_credentials={real_user=slave}} blocked reactor thread for 125827us
W20260129 02:59:53.329928 22861 log.cc:927] Time spent T 42d27a28d89b409e8807ec0efd97af69 P 1c50de89a5a845668696dcb28c3badd6: Append to log took a long time: real 0.433s user 0.001s sys 0.000s
I20260129 02:59:53.241436 21129 rpcz_store.cc:276] 0129 02:59:51.898078 (+ 0us) service_pool.cc:168] Inserting onto call queue
0129 02:59:51.934270 (+ 36192us) service_pool.cc:225] Handling call
0129 02:59:53.090944 (+1156674us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260129 02:59:53.187927 22809 raft_consensus.cc:515] T c2633a6fc34c44579de348d208b0c01e P 197b880fa39d432a9b472845876bfaf0 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:59:53.277789 22887 raft_consensus.cc:2749] T 75f45538dc704d9a8ec4ff8ad9f3ea5b P 1c50de89a5a845668696dcb28c3badd6 [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260129 02:59:53.355830 20508 raft_consensus.cc:1240] T 0f20af7fbe5d4db89a6f17948d98912c P 974b802141bd4f13beffef2aa03f88f9 [term 6 FOLLOWER]: Rejecting Update request from peer 197b880fa39d432a9b472845876bfaf0 for earlier term 5. Current term is 6. Ops: [5.4-5.4]
I20260129 02:59:53.357239 20556 leader_election.cc:304] T cb0b3f259ebc4310927338a8740e87b8 P 1c50de89a5a845668696dcb28c3badd6 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1c50de89a5a845668696dcb28c3badd6; no voters: 7a9daf5d42a84615a6383cfec9a10608, 974b802141bd4f13beffef2aa03f88f9
I20260129 02:59:53.360153 21135 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:47844 (request call id 324) took 1567 ms. Trace:
I20260129 02:59:53.317946 22882 raft_consensus.cc:3060] T cedce331e1584ac8b636c9b8a72b2ba0 P 1c50de89a5a845668696dcb28c3badd6 [term 5 FOLLOWER]: Advancing to term 6
I20260129 02:59:53.385571 21135 rpcz_store.cc:276] 0129 02:59:51.792770 (+ 0us) service_pool.cc:168] Inserting onto call queue
0129 02:59:51.792926 (+ 156us) service_pool.cc:225] Handling call
0129 02:59:51.793356 (+ 430us) raft_consensus.cc:1471] Updating replica for 1 ops
0129 02:59:51.793752 (+ 396us) raft_consensus.cc:1534] Early marking committed up to index 0
0129 02:59:51.793784 (+ 32us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0129 02:59:51.801207 (+ 7423us) log.cc:844] Serialized 26 byte log entry
0129 02:59:52.621738 (+820531us) raft_consensus.cc:1645] Marking committed up to 7
0129 02:59:52.865283 (+243545us) log.cc:844] Serialized 14 byte log entry
0129 02:59:52.865703 (+ 420us) raft_consensus.cc:1697] Filling consensus response to leader.
0129 02:59:52.865922 (+ 219us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0129 02:59:53.359930 (+494008us) raft_consensus.cc:1685] finished
0129 02:59:53.359952 (+ 22us) raft_consensus.cc:1691] UpdateReplicas() finished
0129 02:59:53.360090 (+ 138us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":629790,"prepare.run_cpu_time_us":52,"prepare.run_wall_time_us":53,"raft.queue_time_us":207813,"raft.run_cpu_time_us":332,"raft.run_wall_time_us":38866,"thread_start_us":1031841,"threads_started":2,"wal-append.queue_time_us":900324}
W20260129 02:59:53.388062 21102 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.211.2:34517, user_credentials={real_user=slave}} blocked reactor thread for 360319us
I20260129 02:59:53.242831 22831 consensus_queue.cc:237] T a644c02b16af47679824ccd060ac10fd P 7a9daf5d42a84615a6383cfec9a10608 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.5, Last appended by leader: 5, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } }
I20260129 02:59:53.396595 22820 leader_election.cc:290] T 4f5a77e94fef4b16a2ba7b067c8b144a P 974b802141bd4f13beffef2aa03f88f9 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3:37735), 7a9daf5d42a84615a6383cfec9a10608 (127.18.211.4:40439)
I20260129 02:59:53.315469 22729 raft_consensus.cc:515] T a609ac02fae84725ab743f95f51d33ce P 7a9daf5d42a84615a6383cfec9a10608 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } }
I20260129 02:59:53.316701 22763 raft_consensus.cc:515] T 75be4bd6bd45474795fd472eff2953ac P 974b802141bd4f13beffef2aa03f88f9 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:59:53.392920 20556 leader_election.cc:304] T 3e2d15f8640b407195210c476e09df91 P 1c50de89a5a845668696dcb28c3badd6 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1c50de89a5a845668696dcb28c3badd6, 974b802141bd4f13beffef2aa03f88f9; no voters:
I20260129 02:59:53.376585 22742 raft_consensus.cc:493] T c934d31efd3f4e5195605ddc354d14e1 P 7a9daf5d42a84615a6383cfec9a10608 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260129 02:59:53.384047 22862 log.cc:927] Time spent T 913fc7d751ed4b7388d889aac30a5f9b P 197b880fa39d432a9b472845876bfaf0: Append to log took a long time: real 0.456s user 0.000s sys 0.001s
W20260129 02:59:53.355183 22859 log.cc:927] Time spent T 7d9494f5584d4d5db8b9adf1c8fc888b P 197b880fa39d432a9b472845876bfaf0: Append to log took a long time: real 0.458s user 0.000s sys 0.001s
I20260129 02:59:53.418970 20589 raft_consensus.cc:2468] T b189f08e16a74a11980cc55d99359a0a P 1c50de89a5a845668696dcb28c3badd6 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 974b802141bd4f13beffef2aa03f88f9 in term 7.
I20260129 02:59:53.281132 22873 raft_consensus.cc:515] T 4f5a77e94fef4b16a2ba7b067c8b144a P 7a9daf5d42a84615a6383cfec9a10608 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } }
W20260129 02:59:53.412012 22858 log.cc:927] Time spent T a0fcc58d6fb6473cafbd9d34d5195269 P 1c50de89a5a845668696dcb28c3badd6: Append to log took a long time: real 0.487s user 0.001s sys 0.000s
I20260129 02:59:53.386466 22891 raft_consensus.cc:2804] T 7a9a69cd82724b1faed1d10aa091fc4a P 1c50de89a5a845668696dcb28c3badd6 [term 5 FOLLOWER]: Leader pre-election won for term 6
W20260129 02:59:53.421341 22860 log.cc:927] Time spent T 75f45538dc704d9a8ec4ff8ad9f3ea5b P 974b802141bd4f13beffef2aa03f88f9: Append to log took a long time: real 0.349s user 0.001s sys 0.000s
I20260129 02:59:53.331738 22807 consensus_queue.cc:237] T 3906b080a5b34a74b8b64128635e8997 P 974b802141bd4f13beffef2aa03f88f9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:59:53.420584 22855 raft_consensus.cc:2749] T e4d18c35b0a7493e85d0a4b92956a48f P 1c50de89a5a845668696dcb28c3badd6 [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20260129 02:59:53.334404 22664 consensus_queue.cc:237] T 724a236524cb4e7ba3428880681d6cc2 P 1c50de89a5a845668696dcb28c3badd6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
W20260129 02:59:53.425663 22868 log.cc:927] Time spent T fcddab6bca684adf8ea032289a00f90e P 974b802141bd4f13beffef2aa03f88f9: Append to log took a long time: real 0.239s user 0.000s sys 0.001s
I20260129 02:59:53.323031 22876 raft_consensus.cc:493] T 75b0b6e82ddc4f92b1fe1d94f9a0b73b P 7a9daf5d42a84615a6383cfec9a10608 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260129 02:59:53.359251 22866 raft_consensus.cc:493] T 4f8f3a38f6e24ba4857875181f447442 P 974b802141bd4f13beffef2aa03f88f9 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260129 02:59:53.430471 22891 raft_consensus.cc:493] T 7a9a69cd82724b1faed1d10aa091fc4a P 1c50de89a5a845668696dcb28c3badd6 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260129 02:59:53.408414 22654 consensus_queue.cc:1059] T 45cc5f3d2688486d9b6dbe7e666dedde P 1c50de89a5a845668696dcb28c3badd6 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 }, Status: INVALID_TERM, Last received: 3.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.027s
I20260129 02:59:53.348122 22852 consensus_queue.cc:237] T cedce331e1584ac8b636c9b8a72b2ba0 P 197b880fa39d432a9b472845876bfaf0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:59:53.437109 22886 raft_consensus.cc:2749] T cb0b3f259ebc4310927338a8740e87b8 P 1c50de89a5a845668696dcb28c3badd6 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260129 02:59:53.421460 22742 raft_consensus.cc:515] T c934d31efd3f4e5195605ddc354d14e1 P 7a9daf5d42a84615a6383cfec9a10608 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } }
I20260129 02:59:53.438500 22854 raft_consensus.cc:697] T a1a39dcd328e4268ae41b988c66b4a75 P 1c50de89a5a845668696dcb28c3badd6 [term 4 LEADER]: Becoming Leader. State: Replica: 1c50de89a5a845668696dcb28c3badd6, State: Running, Role: LEADER
I20260129 02:59:53.454183 22745 consensus_queue.cc:1059] T 3ccac82a01a2445fa5b0ce6c0fcf2295 P 974b802141bd4f13beffef2aa03f88f9 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
W20260129 02:59:53.444209 22832 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.538s user 0.000s sys 0.006s
I20260129 02:59:53.450970 22866 raft_consensus.cc:515] T 4f8f3a38f6e24ba4857875181f447442 P 974b802141bd4f13beffef2aa03f88f9 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:59:53.466327 22809 leader_election.cc:290] T c2633a6fc34c44579de348d208b0c01e P 197b880fa39d432a9b472845876bfaf0 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 974b802141bd4f13beffef2aa03f88f9 (127.18.211.2:34517), 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3:37735)
I20260129 02:59:53.445938 22897 raft_consensus.cc:493] T 78aa39096868415d9afc2ffcb248e198 P 7a9daf5d42a84615a6383cfec9a10608 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260129 02:59:53.469918 22815 leader_election.cc:290] T 0f20af7fbe5d4db89a6f17948d98912c P 974b802141bd4f13beffef2aa03f88f9 [CANDIDATE]: Term 6 election: Requested vote from peers 197b880fa39d432a9b472845876bfaf0 (127.18.211.1:42027), 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3:37735)
I20260129 02:59:53.469728 22897 raft_consensus.cc:515] T 78aa39096868415d9afc2ffcb248e198 P 7a9daf5d42a84615a6383cfec9a10608 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } }
I20260129 02:59:53.469955 22854 consensus_queue.cc:237] T a1a39dcd328e4268ae41b988c66b4a75 P 1c50de89a5a845668696dcb28c3badd6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } } peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } }
I20260129 02:59:53.471974 22897 leader_election.cc:290] T 78aa39096868415d9afc2ffcb248e198 P 7a9daf5d42a84615a6383cfec9a10608 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 197b880fa39d432a9b472845876bfaf0 (127.18.211.1:42027), 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3:37735)
W20260129 02:59:53.463064 22758 consensus_meta.cc:287] T 76f713c3331540d2a5b76164103e9eb1 P 197b880fa39d432a9b472845876bfaf0: Time spent flushing consensus metadata: real 0.771s user 0.012s sys 0.000s
I20260129 02:59:53.456314 22729 leader_election.cc:290] T a609ac02fae84725ab743f95f51d33ce P 7a9daf5d42a84615a6383cfec9a10608 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 197b880fa39d432a9b472845876bfaf0 (127.18.211.1:42027), 974b802141bd4f13beffef2aa03f88f9 (127.18.211.2:34517)
I20260129 02:59:53.455804 22748 raft_consensus.cc:3055] T 18f30ebaa4174fd28c958e72d0cc7e24 P 1c50de89a5a845668696dcb28c3badd6 [term 4 LEADER]: Stepping down as leader of term 4
I20260129 02:59:53.477219 22748 raft_consensus.cc:740] T 18f30ebaa4174fd28c958e72d0cc7e24 P 1c50de89a5a845668696dcb28c3badd6 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 1c50de89a5a845668696dcb28c3badd6, State: Running, Role: LEADER
I20260129 02:59:53.477653 22742 leader_election.cc:290] T c934d31efd3f4e5195605ddc354d14e1 P 7a9daf5d42a84615a6383cfec9a10608 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 197b880fa39d432a9b472845876bfaf0 (127.18.211.1:42027), 974b802141bd4f13beffef2aa03f88f9 (127.18.211.2:34517)
I20260129 02:59:53.480518 22763 leader_election.cc:290] T 75be4bd6bd45474795fd472eff2953ac P 974b802141bd4f13beffef2aa03f88f9 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 197b880fa39d432a9b472845876bfaf0 (127.18.211.1:42027), 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3:37735)
I20260129 02:59:53.451129 22891 raft_consensus.cc:3060] T 7a9a69cd82724b1faed1d10aa091fc4a P 1c50de89a5a845668696dcb28c3badd6 [term 5 FOLLOWER]: Advancing to term 6
W20260129 02:59:53.464176 22832 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.562s user 0.000s sys 0.007s
I20260129 02:59:53.476375 22873 leader_election.cc:290] T 4f5a77e94fef4b16a2ba7b067c8b144a P 7a9daf5d42a84615a6383cfec9a10608 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 974b802141bd4f13beffef2aa03f88f9 (127.18.211.2:34517), 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3:37735)
I20260129 02:59:53.450220 22876 raft_consensus.cc:515] T 75b0b6e82ddc4f92b1fe1d94f9a0b73b P 7a9daf5d42a84615a6383cfec9a10608 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } }
I20260129 02:59:53.442483 22865 raft_consensus.cc:515] T 45cc5f3d2688486d9b6dbe7e666dedde P 974b802141bd4f13beffef2aa03f88f9 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } }
I20260129 02:59:53.484784 22844 raft_consensus.cc:515] T add8d4923d1a489e86882c3a25692b95 P 974b802141bd4f13beffef2aa03f88f9 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } }
I20260129 02:59:53.490372 22865 leader_election.cc:290] T 45cc5f3d2688486d9b6dbe7e666dedde P 974b802141bd4f13beffef2aa03f88f9 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3:37735), 7a9daf5d42a84615a6383cfec9a10608 (127.18.211.4:40439)
I20260129 02:59:53.490705 22844 leader_election.cc:290] T add8d4923d1a489e86882c3a25692b95 P 974b802141bd4f13beffef2aa03f88f9 [CANDIDATE]: Term 4 election: Requested vote from peers 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3:37735), 7a9daf5d42a84615a6383cfec9a10608 (127.18.211.4:40439)
I20260129 02:59:53.483780 22866 leader_election.cc:290] T 4f8f3a38f6e24ba4857875181f447442 P 974b802141bd4f13beffef2aa03f88f9 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 197b880fa39d432a9b472845876bfaf0 (127.18.211.1:42027), 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3:37735)
I20260129 02:59:53.490882 22876 leader_election.cc:290] T 75b0b6e82ddc4f92b1fe1d94f9a0b73b P 7a9daf5d42a84615a6383cfec9a10608 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 974b802141bd4f13beffef2aa03f88f9 (127.18.211.2:34517), 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3:37735)
I20260129 02:59:53.501005 22882 raft_consensus.cc:2749] T cedce331e1584ac8b636c9b8a72b2ba0 P 1c50de89a5a845668696dcb28c3badd6 [term 6 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
W20260129 02:59:53.528188 20406 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260129 02:59:53.528231 20405 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.211.3:37735, user_credentials={real_user=slave}} blocked reactor thread for 449189us
I20260129 02:59:53.531831 20433 raft_consensus.cc:3060] T 76f713c3331540d2a5b76164103e9eb1 P 197b880fa39d432a9b472845876bfaf0 [term 4 FOLLOWER]: Advancing to term 5
W20260129 02:59:53.532436 20480 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.211.3:37735, user_credentials={real_user=slave}} blocked reactor thread for 425034us
I20260129 02:59:53.545179 20585 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35752 (request call id 1551) took 2939 ms. Trace:
I20260129 02:59:53.405925 22184 consensus_queue.cc:237] T e33a417cf9c54cf5af73a4ee3aac4d3a P 974b802141bd4f13beffef2aa03f88f9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 3.2, Last appended by leader: 3, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:59:53.553000 22554 leader_election.cc:290] T 913fc7d751ed4b7388d889aac30a5f9b P 1c50de89a5a845668696dcb28c3badd6 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 197b880fa39d432a9b472845876bfaf0 (127.18.211.1:42027), 974b802141bd4f13beffef2aa03f88f9 (127.18.211.2:34517)
I20260129 02:59:53.555473 21129 raft_consensus.cc:1240] T 4e3805ffa70242e8ade66da260dddf66 P 7a9daf5d42a84615a6383cfec9a10608 [term 4 FOLLOWER]: Rejecting Update request from peer 1c50de89a5a845668696dcb28c3badd6 for earlier term 3. Current term is 4. Ops: []
I20260129 02:59:53.560992 22493 leader_election.cc:290] T 1299723a4476420ab02fd8bfd7aa0645 P 974b802141bd4f13beffef2aa03f88f9 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 197b880fa39d432a9b472845876bfaf0 (127.18.211.1:42027), 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3:37735), 7a9daf5d42a84615a6383cfec9a10608 (127.18.211.4:40439)
I20260129 02:59:53.569072 22667 leader_election.cc:290] T 45cc5f3d2688486d9b6dbe7e666dedde P 7a9daf5d42a84615a6383cfec9a10608 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 974b802141bd4f13beffef2aa03f88f9 (127.18.211.2:34517), 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3:37735)
I20260129 02:59:53.449097 22644 leader_election.cc:290] T a609ac02fae84725ab743f95f51d33ce P 197b880fa39d432a9b472845876bfaf0 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 974b802141bd4f13beffef2aa03f88f9 (127.18.211.2:34517), 7a9daf5d42a84615a6383cfec9a10608 (127.18.211.4:40439)
W20260129 02:59:53.457572 22700 consensus_meta.cc:287] T 36ff33be666a4c82954731691dbcf327 P 974b802141bd4f13beffef2aa03f88f9: Time spent flushing consensus metadata: real 0.807s user 0.016s sys 0.000s
W20260129 02:59:53.600683 20588 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.524s user 0.003s sys 0.007s
W20260129 02:59:53.602293 21099 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.501s user 0.002s sys 0.015s
I20260129 02:59:53.636400 22878 raft_consensus.cc:493] T 0340229d521b432580ae926f6062e3f7 P 974b802141bd4f13beffef2aa03f88f9 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 1c50de89a5a845668696dcb28c3badd6)
W20260129 02:59:53.750226 21099 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.649s user 0.002s sys 0.015s
W20260129 02:59:53.720350 22403 consensus_meta.cc:287] T 42d27a28d89b409e8807ec0efd97af69 P 7a9daf5d42a84615a6383cfec9a10608: Time spent flushing consensus metadata: real 1.292s user 0.023s sys 0.008s
W20260129 02:59:53.639587 20556 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.211.2:34517, user_credentials={real_user=slave}} blocked reactor thread for 246599us
I20260129 02:59:53.685587 22656 raft_consensus.cc:2804] T 3e2d15f8640b407195210c476e09df91 P 1c50de89a5a845668696dcb28c3badd6 [term 3 FOLLOWER]: Leader election won for term 3
W20260129 02:59:53.687477 22798 consensus_meta.cc:287] T 1ce4ddf5b10e4ae8b193b13b2e5936ee P 197b880fa39d432a9b472845876bfaf0: Time spent flushing consensus metadata: real 1.074s user 0.020s sys 0.000s
I20260129 02:59:53.689069 22728 raft_consensus.cc:493] T 96609ac68bdf4993a12e1c23355ca013 P 1c50de89a5a845668696dcb28c3badd6 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260129 02:59:53.692091 22855 raft_consensus.cc:493] T 8055232cf0004d40a85f3bf983fe00a9 P 1c50de89a5a845668696dcb28c3badd6 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260129 02:59:53.683198 20482 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260129 02:59:53.701627 20405 connection.cc:232] RPC call timeout handler was delayed by -2.2416859520000116s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.22130000000000002s for a response
I20260129 02:59:53.703251 20432 raft_consensus.cc:1240] T a4c1226b987b468fa84dfc6ba0e2dbe1 P 197b880fa39d432a9b472845876bfaf0 [term 4 LEADER]: Rejecting Update request from peer 1c50de89a5a845668696dcb28c3badd6 for earlier term 3. Current term is 4. Ops: []
I20260129 02:59:53.710072 22891 raft_consensus.cc:515] T 7a9a69cd82724b1faed1d10aa091fc4a P 1c50de89a5a845668696dcb28c3badd6 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
W20260129 02:59:53.753314 20588 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.677s user 0.004s sys 0.008s
I20260129 02:59:53.752885 22700 raft_consensus.cc:515] T 36ff33be666a4c82954731691dbcf327 P 974b802141bd4f13beffef2aa03f88f9 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:59:53.766721 22700 leader_election.cc:290] T 36ff33be666a4c82954731691dbcf327 P 974b802141bd4f13beffef2aa03f88f9 [CANDIDATE]: Term 3 election: Requested vote from peers 197b880fa39d432a9b472845876bfaf0 (127.18.211.1:42027), 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3:37735)
W20260129 02:59:53.767488 22320 consensus_meta.cc:287] T a644c02b16af47679824ccd060ac10fd P 1c50de89a5a845668696dcb28c3badd6: Time spent flushing consensus metadata: real 1.277s user 0.015s sys 0.021s
I20260129 02:59:53.769102 20516 raft_consensus.cc:1240] T 36ff33be666a4c82954731691dbcf327 P 974b802141bd4f13beffef2aa03f88f9 [term 3 FOLLOWER]: Rejecting Update request from peer 1c50de89a5a845668696dcb28c3badd6 for earlier term 2. Current term is 3. Ops: []
I20260129 02:59:53.773087 21133 raft_consensus.cc:2410] T 42d27a28d89b409e8807ec0efd97af69 P 7a9daf5d42a84615a6383cfec9a10608 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 197b880fa39d432a9b472845876bfaf0 for term 6 because replica has last-logged OpId of term: 5 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 4 index: 6.
W20260129 02:59:53.778473 22581 consensus_meta.cc:287] T a4c1226b987b468fa84dfc6ba0e2dbe1 P 1c50de89a5a845668696dcb28c3badd6: Time spent flushing consensus metadata: real 0.940s user 0.018s sys 0.003s
I20260129 02:59:53.666276 20585 rpcz_store.cc:276] 0129 02:59:50.605888 (+ 0us) service_pool.cc:168] Inserting onto call queue
0129 02:59:52.282971 (+1677083us) service_pool.cc:225] Handling call
0129 02:59:52.307629 (+ 24658us) raft_consensus.cc:1471] Updating replica for 1 ops
0129 02:59:52.308017 (+ 388us) raft_consensus.cc:1534] Early marking committed up to index 0
0129 02:59:52.308044 (+ 27us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0129 02:59:52.339102 (+ 31058us) log.cc:844] Serialized 26 byte log entry
0129 02:59:52.660250 (+321148us) raft_consensus.cc:1645] Marking committed up to 3
0129 02:59:52.873543 (+213293us) log.cc:844] Serialized 14 byte log entry
0129 02:59:52.896462 (+ 22919us) raft_consensus.cc:1697] Filling consensus response to leader.
0129 02:59:52.896551 (+ 89us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0129 02:59:53.544397 (+647846us) raft_consensus.cc:1685] finished
0129 02:59:53.544445 (+ 48us) raft_consensus.cc:1691] UpdateReplicas() finished
0129 02:59:53.544834 (+ 389us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":22604,"prepare.queue_time_us":261156,"prepare.run_cpu_time_us":35,"prepare.run_wall_time_us":38,"raft.queue_time_us":224467,"raft.run_cpu_time_us":255,"raft.run_wall_time_us":5814,"spinlock_wait_cycles":856320,"thread_start_us":386473,"threads_started":2,"wal-append.queue_time_us":470291}
I20260129 02:59:53.787925 22878 raft_consensus.cc:515] T 0340229d521b432580ae926f6062e3f7 P 974b802141bd4f13beffef2aa03f88f9 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:59:53.789376 22231 raft_consensus.cc:3055] T 75be4bd6bd45474795fd472eff2953ac P 197b880fa39d432a9b472845876bfaf0 [term 8 LEADER]: Stepping down as leader of term 8
I20260129 02:59:53.840276 22231 raft_consensus.cc:740] T 75be4bd6bd45474795fd472eff2953ac P 197b880fa39d432a9b472845876bfaf0 [term 8 LEADER]: Becoming Follower/Learner. State: Replica: 197b880fa39d432a9b472845876bfaf0, State: Running, Role: LEADER
I20260129 02:59:53.840633 22748 consensus_queue.cc:260] T 18f30ebaa4174fd28c958e72d0cc7e24 P 1c50de89a5a845668696dcb28c3badd6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } }
W20260129 02:59:53.771626 21128 consensus_meta.cc:287] T 3e2d15f8640b407195210c476e09df91 P 7a9daf5d42a84615a6383cfec9a10608: Time spent flushing consensus metadata: real 0.917s user 0.006s sys 0.011s
W20260129 02:59:53.762787 21099 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.211.1:42027, user_credentials={real_user=slave}} blocked reactor thread for 681986us
I20260129 02:59:53.846736 20433 raft_consensus.cc:2468] T 76f713c3331540d2a5b76164103e9eb1 P 197b880fa39d432a9b472845876bfaf0 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1c50de89a5a845668696dcb28c3badd6 in term 5.
I20260129 02:59:53.824838 22855 raft_consensus.cc:515] T 8055232cf0004d40a85f3bf983fe00a9 P 1c50de89a5a845668696dcb28c3badd6 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:59:53.826007 22728 raft_consensus.cc:515] T 96609ac68bdf4993a12e1c23355ca013 P 1c50de89a5a845668696dcb28c3badd6 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } attrs { replace: true } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: NON_VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: true } }
W20260129 02:59:53.833118 20553 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260129 02:59:53.836956 20508 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57600 (request call id 1789) took 1904 ms. Trace:
I20260129 02:59:53.838133 20511 raft_consensus.cc:1240] T 0f20af7fbe5d4db89a6f17948d98912c P 974b802141bd4f13beffef2aa03f88f9 [term 6 FOLLOWER]: Rejecting Update request from peer 1c50de89a5a845668696dcb28c3badd6 for earlier term 4. Current term is 6. Ops: []
I20260129 02:59:53.839367 22872 raft_consensus.cc:493] T 96609ac68bdf4993a12e1c23355ca013 P 7a9daf5d42a84615a6383cfec9a10608 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260129 02:59:53.841917 22231 consensus_queue.cc:260] T 75be4bd6bd45474795fd472eff2953ac P 197b880fa39d432a9b472845876bfaf0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 8.6, Last appended by leader: 6, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:59:53.792760 22688 consensus_queue.cc:260] T 4408006eb5174de1896b037cf3f18457 P 197b880fa39d432a9b472845876bfaf0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
W20260129 02:59:53.824244 20405 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260129 02:59:53.938788 22872 raft_consensus.cc:515] T 96609ac68bdf4993a12e1c23355ca013 P 7a9daf5d42a84615a6383cfec9a10608 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } attrs { replace: true } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } }
I20260129 02:59:53.940212 21128 raft_consensus.cc:2468] T 3e2d15f8640b407195210c476e09df91 P 7a9daf5d42a84615a6383cfec9a10608 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1c50de89a5a845668696dcb28c3badd6 in term 3.
W20260129 02:59:53.901849 22425 consensus_meta.cc:287] T 7d9494f5584d4d5db8b9adf1c8fc888b P 1c50de89a5a845668696dcb28c3badd6: Time spent flushing consensus metadata: real 0.803s user 0.010s sys 0.010s
I20260129 02:59:53.948279 21099 leader_election.cc:304] T 0678b8b468bd43cdbc3e485900844821 P 7a9daf5d42a84615a6383cfec9a10608 [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7a9daf5d42a84615a6383cfec9a10608; no voters: 197b880fa39d432a9b472845876bfaf0, 974b802141bd4f13beffef2aa03f88f9
I20260129 02:59:54.071818 20516 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57584 (request call id 1337) took 1188 ms. Trace:
I20260129 02:59:54.105746 20433 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55676 (request call id 1303) took 1223 ms. Trace:
W20260129 02:59:53.825440 22640 consensus_meta.cc:287] T e7d83d4ccf684e76acb29a0dc844e1b8 P 1c50de89a5a845668696dcb28c3badd6: Time spent flushing consensus metadata: real 1.140s user 0.016s sys 0.010s
I20260129 02:59:54.125622 20516 rpcz_store.cc:276] 0129 02:59:52.883038 (+ 0us) service_pool.cc:168] Inserting onto call queue
0129 02:59:53.096567 (+213529us) service_pool.cc:225] Handling call
0129 02:59:53.096909 (+ 342us) raft_consensus.cc:1471] Updating replica for 0 ops
0129 02:59:53.770987 (+674078us) raft_consensus.cc:1697] Filling consensus response to leader.
0129 02:59:54.071419 (+300432us) spinlock_profiling.cc:243] Waited 672 ms on lock 0x7b5400168354. stack: 00007f8fe044e373 00007f8fe044e270 00007f8fdfda0574 00007f8fe9a5c6b4 00007f8fe9a5c649 00007f8fe34975a1 00007f8fe352ea81 00007f8fe352d4be 00007f8fe84e1f49 00007f8fe1a6ed34 00007f8fe1a6ecc2 00007f8fe1a6ebf1 00007f8fe1a6eb6c 00007f8fe1a6de22 00007f8fe186d34c 00007f8fe186c786
0129 02:59:54.071735 (+ 316us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1544767744}
I20260129 02:59:53.952143 20438 raft_consensus.cc:1275] T fcddab6bca684adf8ea032289a00f90e P 197b880fa39d432a9b472845876bfaf0 [term 8 FOLLOWER]: Refusing update from remote peer 974b802141bd4f13beffef2aa03f88f9: Log matching property violated. Preceding OpId in replica: term: 7 index: 5. Preceding OpId from leader: term: 8 index: 5. (term mismatch)
I20260129 02:59:54.066638 20513 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57584 (request call id 1342) took 1131 ms. Trace:
W20260129 02:59:54.543547 20482 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260129 02:59:54.162340 20556 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.211.2:34517, user_credentials={real_user=slave}} blocked reactor thread for 387323us
W20260129 02:59:54.175629 20481 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.718s user 0.007s sys 0.009s
I20260129 02:59:54.055997 21133 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34060 (request call id 498) took 1006 ms. Trace:
W20260129 02:59:54.237777 20406 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260129 02:59:54.622435 20480 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.211.3:37735, user_credentials={real_user=slave}} blocked reactor thread for 461497us
I20260129 02:59:53.953457 20510 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57584 (request call id 1318) took 3333 ms. Trace:
I20260129 02:59:54.270673 20436 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0f20af7fbe5d4db89a6f17948d98912c" candidate_uuid: "974b802141bd4f13beffef2aa03f88f9" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "197b880fa39d432a9b472845876bfaf0"
I20260129 02:59:53.950074 20437 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "76f713c3331540d2a5b76164103e9eb1" candidate_uuid: "974b802141bd4f13beffef2aa03f88f9" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "197b880fa39d432a9b472845876bfaf0" is_pre_election: true
I20260129 02:59:53.955842 20439 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1299723a4476420ab02fd8bfd7aa0645" candidate_uuid: "974b802141bd4f13beffef2aa03f88f9" candidate_term: 5 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "197b880fa39d432a9b472845876bfaf0" is_pre_election: true
W20260129 02:59:54.073175 22899 log.cc:927] Time spent T 2ac659d732314e1288849f08fe5e074e P 974b802141bd4f13beffef2aa03f88f9: Append to log took a long time: real 0.597s user 0.002s sys 0.000s
W20260129 02:59:54.396855 20515 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.550s user 0.006s sys 0.005s
I20260129 02:59:54.126329 20433 rpcz_store.cc:276] 0129 02:59:52.882002 (+ 0us) service_pool.cc:168] Inserting onto call queue
0129 02:59:52.883692 (+ 1690us) service_pool.cc:225] Handling call
0129 02:59:54.105329 (+1221637us) spinlock_profiling.cc:243] Waited 595 ms on lock 0x7b5400330754. stack: 00007f8fe044e373 00007f8fe044e270 00007f8fdfda0574 00007f8fe9a5c6b4 00007f8fe9a5c649 00007f8fe34975a1 00007f8fe3531f7a 00007f8fe84e3647 00007f8fe1a73834 00007f8fe1a737c2 00007f8fe1a736f1 00007f8fe1a7366c 00007f8fe1a72922 00007f8fe186d34c 00007f8fe186c786 00007f8fe186c12f
0129 02:59:54.105619 (+ 290us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1368519040}
I20260129 02:59:53.932430 20508 rpcz_store.cc:276] 0129 02:59:51.932826 (+ 0us) service_pool.cc:168] Inserting onto call queue
0129 02:59:52.280497 (+347671us) service_pool.cc:225] Handling call
0129 02:59:52.280846 (+ 349us) raft_consensus.cc:1471] Updating replica for 1 ops
0129 02:59:53.385088 (+1104242us) raft_consensus.cc:1697] Filling consensus response to leader.
0129 02:59:53.836389 (+451301us) spinlock_profiling.cc:243] Waited 1.07 s on lock 0x7b54001ae354. stack: 00007f8fe044e373 00007f8fe044e270 00007f8fdfda0574 00007f8fe9a5c6b4 00007f8fe9a5c649 00007f8fe34975a1 00007f8fe352ea81 00007f8fe352d4be 00007f8fe84e1f49 00007f8fe1a6ed34 00007f8fe1a6ecc2 00007f8fe1a6ebf1 00007f8fe1a6eb6c 00007f8fe1a6de22 00007f8fe186d34c 00007f8fe186c786
0129 02:59:53.836761 (+ 372us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2471404800}
W20260129 02:59:54.397215 20553 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260129 02:59:54.251890 20405 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260129 02:59:54.060031 20511 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57584 (request call id 1323) took 1258 ms. Trace:
I20260129 02:59:54.662003 20516 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c2633a6fc34c44579de348d208b0c01e" candidate_uuid: "197b880fa39d432a9b472845876bfaf0" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "974b802141bd4f13beffef2aa03f88f9" is_pre_election: true
I20260129 02:59:54.058569 21135 raft_consensus.cc:1275] T 42d27a28d89b409e8807ec0efd97af69 P 7a9daf5d42a84615a6383cfec9a10608 [term 6 FOLLOWER]: Refusing update from remote peer 1c50de89a5a845668696dcb28c3badd6: Log matching property violated. Preceding OpId in replica: term: 5 index: 7. Preceding OpId from leader: term: 6 index: 8. (index mismatch)
W20260129 02:59:54.160907 20555 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260129 02:59:54.159721 20554 leader_election.cc:304] T 505580c93c694bbb8ea96fdf9b17ab46 P 1c50de89a5a845668696dcb28c3badd6 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1c50de89a5a845668696dcb28c3badd6, 7a9daf5d42a84615a6383cfec9a10608; no voters: 974b802141bd4f13beffef2aa03f88f9
W20260129 02:59:54.622473 21099 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.211.1:42027, user_credentials={real_user=slave}} blocked reactor thread for 674309us
W20260129 02:59:54.677249 20515 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 1.102s user 0.012s sys 0.015s
W20260129 02:59:54.667448 20481 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 1.482s user 0.017s sys 0.022s
I20260129 02:59:54.694769 20437 raft_consensus.cc:2393] T 76f713c3331540d2a5b76164103e9eb1 P 197b880fa39d432a9b472845876bfaf0 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 974b802141bd4f13beffef2aa03f88f9 in current term 5: Already voted for candidate 1c50de89a5a845668696dcb28c3badd6 in this term.
W20260129 02:59:54.388235 22915 log.cc:927] Time spent T 724a236524cb4e7ba3428880681d6cc2 P 1c50de89a5a845668696dcb28c3badd6: Append to log took a long time: real 0.226s user 0.003s sys 0.000s
W20260129 02:59:54.610785 22807 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.520s user 0.014s sys 0.007s
I20260129 02:59:54.420647 20441 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4f8f3a38f6e24ba4857875181f447442" candidate_uuid: "974b802141bd4f13beffef2aa03f88f9" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "197b880fa39d432a9b472845876bfaf0" is_pre_election: true
I20260129 02:59:54.659722 21133 rpcz_store.cc:276] 0129 02:59:53.049112 (+ 0us) service_pool.cc:168] Inserting onto call queue
0129 02:59:53.049503 (+ 391us) service_pool.cc:225] Handling call
0129 02:59:54.055616 (+1006113us) spinlock_profiling.cc:243] Waited 712 ms on lock 0x7b54003b3dd4. stack: 00007f8fe044e373 00007f8fe044e270 00007f8fdfda0574 00007f8fe9a5c6b4 00007f8fe9a5c649 00007f8fe34975a1 00007f8fe3531f7a 00007f8fe84e3647 00007f8fe1a73834 00007f8fe1a737c2 00007f8fe1a736f1 00007f8fe1a7366c 00007f8fe1a72922 00007f8fe186d34c 00007f8fe186c786 00007f8fe186c12f
0129 02:59:54.055892 (+ 276us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1638241152}
I20260129 02:59:54.658116 20438 pending_rounds.cc:85] T fcddab6bca684adf8ea032289a00f90e P 197b880fa39d432a9b472845876bfaf0: Aborting all ops after (but not including) 4
W20260129 02:59:54.070412 22883 log.cc:927] Time spent T febffe4a8b394368b69b1d164308fb9a P 974b802141bd4f13beffef2aa03f88f9: Append to log took a long time: real 0.730s user 0.002s sys 0.000s
I20260129 02:59:54.658278 20513 rpcz_store.cc:276] 0129 02:59:52.934991 (+ 0us) service_pool.cc:168] Inserting onto call queue
0129 02:59:53.189833 (+254842us) service_pool.cc:225] Handling call
0129 02:59:53.190172 (+ 339us) raft_consensus.cc:1471] Updating replica for 0 ops
0129 02:59:53.339234 (+149062us) raft_consensus.cc:1534] Early marking committed up to index 0
0129 02:59:53.339276 (+ 42us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0129 02:59:53.339330 (+ 54us) raft_consensus.cc:1645] Marking committed up to 5
0129 02:59:53.339374 (+ 44us) raft_consensus.cc:1697] Filling consensus response to leader.
0129 02:59:54.066075 (+726701us) spinlock_profiling.cc:243] Waited 149 ms on lock 0x7b54003007d4. stack: 00007f8fe044e373 00007f8fe044e270 00007f8fdfda0574 00007f8fe9a5c6b4 00007f8fe9a5c649 00007f8fe34975a1 00007f8fe352ea81 00007f8fe352d4be 00007f8fe84e1f49 00007f8fe1a6ed34 00007f8fe1a6ecc2 00007f8fe1a6ebf1 00007f8fe1a6eb6c 00007f8fe1a6de22 00007f8fe186d34c 00007f8fe186c786
0129 02:59:54.066160 (+ 85us) raft_consensus.cc:1691] UpdateReplicas() finished
0129 02:59:54.066423 (+ 263us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":342289792}
W20260129 02:59:53.966605 22343 consensus_meta.cc:287] T add8d4923d1a489e86882c3a25692b95 P 7a9daf5d42a84615a6383cfec9a10608: Time spent flushing consensus metadata: real 0.963s user 0.015s sys 0.004s
I20260129 02:59:53.955971 22888 raft_consensus.cc:3055] T 3ccac82a01a2445fa5b0ce6c0fcf2295 P 974b802141bd4f13beffef2aa03f88f9 [term 3 LEADER]: Stepping down as leader of term 3
W20260129 02:59:54.702584 20407 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260129 02:59:54.677659 20510 rpcz_store.cc:276] 0129 02:59:50.619667 (+ 0us) service_pool.cc:168] Inserting onto call queue
0129 02:59:51.547591 (+927924us) service_pool.cc:225] Handling call
0129 02:59:51.601375 (+ 53784us) raft_consensus.cc:1471] Updating replica for 0 ops
0129 02:59:53.231957 (+1630582us) raft_consensus.cc:1697] Filling consensus response to leader.
0129 02:59:53.953006 (+721049us) spinlock_profiling.cc:243] Waited 1.57 s on lock 0x7b54000fc654. stack: 00007f8fe044e373 00007f8fe044e270 00007f8fdfda0574 00007f8fe9a5c6b4 00007f8fe9a5c649 00007f8fe34975a1 00007f8fe352ea81 00007f8fe352d4be 00007f8fe84e1f49 00007f8fe1a6ed34 00007f8fe1a6ecc2 00007f8fe1a6ebf1 00007f8fe1a6eb6c 00007f8fe1a6de22 00007f8fe186d34c 00007f8fe186c786
0129 02:59:53.953268 (+ 262us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":3612359040}
I20260129 02:59:54.158023 20584 raft_consensus.cc:606] T 7d9494f5584d4d5db8b9adf1c8fc888b P 1c50de89a5a845668696dcb28c3badd6 [term 3 FOLLOWER]: Received request to transfer leadership to TS 197b880fa39d432a9b472845876bfaf0
I20260129 02:59:54.160935 22656 raft_consensus.cc:697] T 3e2d15f8640b407195210c476e09df91 P 1c50de89a5a845668696dcb28c3badd6 [term 3 LEADER]: Becoming Leader. State: Replica: 1c50de89a5a845668696dcb28c3badd6, State: Running, Role: LEADER
I20260129 02:59:54.249816 22728 leader_election.cc:290] T 96609ac68bdf4993a12e1c23355ca013 P 1c50de89a5a845668696dcb28c3badd6 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 197b880fa39d432a9b472845876bfaf0 (127.18.211.1:42027), 974b802141bd4f13beffef2aa03f88f9 (127.18.211.2:34517)
I20260129 02:59:54.168232 22700 raft_consensus.cc:493] T 96609ac68bdf4993a12e1c23355ca013 P 974b802141bd4f13beffef2aa03f88f9 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260129 02:59:54.710435 20508 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a609ac02fae84725ab743f95f51d33ce" candidate_uuid: "197b880fa39d432a9b472845876bfaf0" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "974b802141bd4f13beffef2aa03f88f9" is_pre_election: true
I20260129 02:59:54.056578 22691 raft_consensus.cc:3055] T 4e420f90884f4796997b5ea41e49e411 P 197b880fa39d432a9b472845876bfaf0 [term 4 LEADER]: Stepping down as leader of term 4
I20260129 02:59:54.678881 20511 rpcz_store.cc:276] 0129 02:59:52.801361 (+ 0us) service_pool.cc:168] Inserting onto call queue
0129 02:59:52.801570 (+ 209us) service_pool.cc:225] Handling call
0129 02:59:53.837617 (+1036047us) raft_consensus.cc:1471] Updating replica for 0 ops
0129 02:59:53.952170 (+114553us) raft_consensus.cc:1697] Filling consensus response to leader.
0129 02:59:54.059668 (+107498us) spinlock_profiling.cc:243] Waited 1.01 s on lock 0x7b54001ae350. stack: 00007f8fe044e373 00007f8fe044e270 00007f8fdfda0574 00007f8fe9a5c6b4 00007f8fe9a5c649 00007f8fe34975a1 00007f8fe352d52d 00007f8fe84e1f49 00007f8fe1a6ed34 00007f8fe1a6ecc2 00007f8fe1a6ebf1 00007f8fe1a6eb6c 00007f8fe1a6de22 00007f8fe186d34c 00007f8fe186c786 00007f8fe186c12f
0129 02:59:54.059948 (+ 280us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2326775168}
I20260129 02:59:54.715435 22640 raft_consensus.cc:2804] T 505580c93c694bbb8ea96fdf9b17ab46 P 1c50de89a5a845668696dcb28c3badd6 [term 5 FOLLOWER]: Leader election won for term 5
I20260129 02:59:54.706749 22888 raft_consensus.cc:740] T 3ccac82a01a2445fa5b0ce6c0fcf2295 P 974b802141bd4f13beffef2aa03f88f9 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 974b802141bd4f13beffef2aa03f88f9, State: Running, Role: LEADER
I20260129 02:59:54.716205 22640 raft_consensus.cc:697] T 505580c93c694bbb8ea96fdf9b17ab46 P 1c50de89a5a845668696dcb28c3badd6 [term 5 LEADER]: Becoming Leader. State: Replica: 1c50de89a5a845668696dcb28c3badd6, State: Running, Role: LEADER
I20260129 02:59:54.715636 22700 raft_consensus.cc:515] T 96609ac68bdf4993a12e1c23355ca013 P 974b802141bd4f13beffef2aa03f88f9 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } attrs { replace: true } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } }
I20260129 02:59:54.717139 22888 consensus_queue.cc:260] T 3ccac82a01a2445fa5b0ce6c0fcf2295 P 974b802141bd4f13beffef2aa03f88f9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 3.5, Last appended by leader: 5, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } }
I20260129 02:59:54.717485 22640 consensus_queue.cc:237] T 505580c93c694bbb8ea96fdf9b17ab46 P 1c50de89a5a845668696dcb28c3badd6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 3.6, Last appended by leader: 6, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } }
I20260129 02:59:54.719034 22888 raft_consensus.cc:3060] T 3ccac82a01a2445fa5b0ce6c0fcf2295 P 974b802141bd4f13beffef2aa03f88f9 [term 3 FOLLOWER]: Advancing to term 4
I20260129 02:59:54.719370 22700 leader_election.cc:290] T 96609ac68bdf4993a12e1c23355ca013 P 974b802141bd4f13beffef2aa03f88f9 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 197b880fa39d432a9b472845876bfaf0 (127.18.211.1:42027), 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3:37735), 7a9daf5d42a84615a6383cfec9a10608 (127.18.211.4:40439)
I20260129 02:59:54.715770 20508 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:57600 (request call id 1802) took 1175 ms. Trace:
W20260129 02:59:54.158427 21128 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34046 (request call id 485) took 1513 ms (client timeout 1671 ms). Trace:
I20260129 02:59:54.164903 22857 raft_consensus.cc:2749] T 0678b8b468bd43cdbc3e485900844821 P 7a9daf5d42a84615a6383cfec9a10608 [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20260129 02:59:54.672840 22852 raft_consensus.cc:493] T 3906b080a5b34a74b8b64128635e8997 P 197b880fa39d432a9b472845876bfaf0 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260129 02:59:54.288517 22719 raft_consensus.cc:493] T 00e79d8f25a34676b1d866eb2068a2b2 P 974b802141bd4f13beffef2aa03f88f9 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260129 02:59:54.161270 22900 raft_consensus.cc:493] T c891d2bff4ce4568861bbc992e91fbb0 P 1c50de89a5a845668696dcb28c3badd6 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 974b802141bd4f13beffef2aa03f88f9)
I20260129 02:59:54.165951 22667 raft_consensus.cc:493] T 75f45538dc704d9a8ec4ff8ad9f3ea5b P 7a9daf5d42a84615a6383cfec9a10608 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260129 02:59:54.067603 22851 raft_consensus.cc:3055] T 0678b8b468bd43cdbc3e485900844821 P 197b880fa39d432a9b472845876bfaf0 [term 5 LEADER]: Stepping down as leader of term 5
I20260129 02:59:54.707715 20584 raft_consensus.cc:614] T 7d9494f5584d4d5db8b9adf1c8fc888b P 1c50de89a5a845668696dcb28c3badd6 [term 3 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260129 02:59:54.658025 22855 leader_election.cc:290] T 8055232cf0004d40a85f3bf983fe00a9 P 1c50de89a5a845668696dcb28c3badd6 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 197b880fa39d432a9b472845876bfaf0 (127.18.211.1:42027), 974b802141bd4f13beffef2aa03f88f9 (127.18.211.2:34517)
I20260129 02:59:54.704051 20438 pending_rounds.cc:107] T fcddab6bca684adf8ea032289a00f90e P 197b880fa39d432a9b472845876bfaf0: Aborting uncommitted NO_OP operation due to leader change: 7.5
W20260129 02:59:54.075145 22694 consensus_meta.cc:287] T 17d6431cc77a491e8464d68db4081a51 P 974b802141bd4f13beffef2aa03f88f9: Time spent flushing consensus metadata: real 1.396s user 0.017s sys 0.009s
I20260129 02:59:54.391676 22688 raft_consensus.cc:3060] T 4408006eb5174de1896b037cf3f18457 P 197b880fa39d432a9b472845876bfaf0 [term 2 FOLLOWER]: Advancing to term 3
I20260129 02:59:54.126978 22748 raft_consensus.cc:3060] T 18f30ebaa4174fd28c958e72d0cc7e24 P 1c50de89a5a845668696dcb28c3badd6 [term 4 FOLLOWER]: Advancing to term 5
I20260129 02:59:54.167899 22581 raft_consensus.cc:493] T 0f20af7fbe5d4db89a6f17948d98912c P 1c50de89a5a845668696dcb28c3badd6 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260129 02:59:54.681322 20479 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.18.211.1:42027 timed out after 2.062s (SENT) [suppressed 41 similar messages]
W20260129 02:59:54.702925 22807 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 1.227s user 0.019s sys 0.010s
I20260129 02:59:54.664242 22645 raft_consensus.cc:493] T 368953c9102643b2a21f99af36b51905 P 197b880fa39d432a9b472845876bfaf0 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 1c50de89a5a845668696dcb28c3badd6)
I20260129 02:59:54.681540 22833 raft_consensus.cc:493] T 42d27a28d89b409e8807ec0efd97af69 P 7a9daf5d42a84615a6383cfec9a10608 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260129 02:59:54.167312 22493 raft_consensus.cc:493] T cb0b3f259ebc4310927338a8740e87b8 P 974b802141bd4f13beffef2aa03f88f9 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 7a9daf5d42a84615a6383cfec9a10608)
I20260129 02:59:54.162091 22763 raft_consensus.cc:493] T 368953c9102643b2a21f99af36b51905 P 974b802141bd4f13beffef2aa03f88f9 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260129 02:59:54.286847 22875 raft_consensus.cc:493] T 1ce4ddf5b10e4ae8b193b13b2e5936ee P 197b880fa39d432a9b472845876bfaf0 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260129 02:59:54.175392 22891 leader_election.cc:290] T 7a9a69cd82724b1faed1d10aa091fc4a P 1c50de89a5a845668696dcb28c3badd6 [CANDIDATE]: Term 6 election: Requested vote from peers 197b880fa39d432a9b472845876bfaf0 (127.18.211.1:42027), 974b802141bd4f13beffef2aa03f88f9 (127.18.211.2:34517)
I20260129 02:59:54.701382 22878 leader_election.cc:290] T 0340229d521b432580ae926f6062e3f7 P 974b802141bd4f13beffef2aa03f88f9 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 197b880fa39d432a9b472845876bfaf0 (127.18.211.1:42027), 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3:37735)
W20260129 02:59:54.714627 20407 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260129 02:59:54.720211 20508 rpcz_store.cc:276] 0129 02:59:53.642038 (+ 0us) service_pool.cc:168] Inserting onto call queue
0129 02:59:54.683616 (+1041578us) service_pool.cc:225] Handling call
0129 02:59:54.715650 (+ 32034us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260129 02:59:54.168316 22903 raft_consensus.cc:493] T 4b1876f9a70b49edb3aa74105e66d3b3 P 1c50de89a5a845668696dcb28c3badd6 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 974b802141bd4f13beffef2aa03f88f9)
I20260129 02:59:54.707106 22231 raft_consensus.cc:3060] T 75be4bd6bd45474795fd472eff2953ac P 197b880fa39d432a9b472845876bfaf0 [term 8 FOLLOWER]: Advancing to term 9
W20260129 02:59:54.710431 22847 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.669s user 0.002s sys 0.010s
W20260129 02:59:54.723242 20554 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.211.4:40439, user_credentials={real_user=slave}} blocked reactor thread for 563772us
I20260129 02:59:54.723897 20516 raft_consensus.cc:2468] T c2633a6fc34c44579de348d208b0c01e P 974b802141bd4f13beffef2aa03f88f9 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 197b880fa39d432a9b472845876bfaf0 in term 2.
I20260129 02:59:54.711921 22877 raft_consensus.cc:493] T 2e58e9a5a8aa4cecafb464addb4f35ed P 197b880fa39d432a9b472845876bfaf0 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 7a9daf5d42a84615a6383cfec9a10608)
I20260129 02:59:54.726016 20433 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0340229d521b432580ae926f6062e3f7" candidate_uuid: "974b802141bd4f13beffef2aa03f88f9" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "197b880fa39d432a9b472845876bfaf0" is_pre_election: true
I20260129 02:59:54.726433 20436 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "36ff33be666a4c82954731691dbcf327" candidate_uuid: "974b802141bd4f13beffef2aa03f88f9" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "197b880fa39d432a9b472845876bfaf0"
I20260129 02:59:54.726454 21099 leader_election.cc:304] T 3ccac82a01a2445fa5b0ce6c0fcf2295 P 7a9daf5d42a84615a6383cfec9a10608 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 197b880fa39d432a9b472845876bfaf0, 7a9daf5d42a84615a6383cfec9a10608; no voters: 974b802141bd4f13beffef2aa03f88f9
I20260129 02:59:54.727561 20439 raft_consensus.cc:1813] T 1299723a4476420ab02fd8bfd7aa0645 P 197b880fa39d432a9b472845876bfaf0 [term 8 FOLLOWER]: Handling vote request from an unknown peer 974b802141bd4f13beffef2aa03f88f9
W20260129 02:59:54.731405 20405 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260129 02:59:54.732465 20513 raft_consensus.cc:1275] T 913fc7d751ed4b7388d889aac30a5f9b P 974b802141bd4f13beffef2aa03f88f9 [term 3 FOLLOWER]: Refusing update from remote peer 197b880fa39d432a9b472845876bfaf0: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260129 02:59:54.733695 20434 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "75be4bd6bd45474795fd472eff2953ac" candidate_uuid: "974b802141bd4f13beffef2aa03f88f9" candidate_term: 10 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "197b880fa39d432a9b472845876bfaf0" is_pre_election: true
I20260129 02:59:54.734692 22900 raft_consensus.cc:515] T c891d2bff4ce4568861bbc992e91fbb0 P 1c50de89a5a845668696dcb28c3badd6 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:59:54.714927 22691 raft_consensus.cc:740] T 4e420f90884f4796997b5ea41e49e411 P 197b880fa39d432a9b472845876bfaf0 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 197b880fa39d432a9b472845876bfaf0, State: Running, Role: LEADER
I20260129 02:59:54.736562 20438 raft_consensus.cc:2889] T fcddab6bca684adf8ea032289a00f90e P 197b880fa39d432a9b472845876bfaf0 [term 8 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260129 02:59:54.725565 22872 leader_election.cc:290] T 96609ac68bdf4993a12e1c23355ca013 P 7a9daf5d42a84615a6383cfec9a10608 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 197b880fa39d432a9b472845876bfaf0 (127.18.211.1:42027), 974b802141bd4f13beffef2aa03f88f9 (127.18.211.2:34517), 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3:37735)
I20260129 02:59:54.738482 20441 raft_consensus.cc:2468] T 4f8f3a38f6e24ba4857875181f447442 P 197b880fa39d432a9b472845876bfaf0 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 974b802141bd4f13beffef2aa03f88f9 in term 4.
I20260129 02:59:54.764858 20439 raft_consensus.cc:2368] T 1299723a4476420ab02fd8bfd7aa0645 P 197b880fa39d432a9b472845876bfaf0 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 974b802141bd4f13beffef2aa03f88f9 for earlier term 5. Current term is 8.
I20260129 02:59:54.710793 22343 raft_consensus.cc:515] T add8d4923d1a489e86882c3a25692b95 P 7a9daf5d42a84615a6383cfec9a10608 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } }
I20260129 02:59:54.745687 22645 raft_consensus.cc:515] T 368953c9102643b2a21f99af36b51905 P 197b880fa39d432a9b472845876bfaf0 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:59:54.766956 22900 leader_election.cc:290] T c891d2bff4ce4568861bbc992e91fbb0 P 1c50de89a5a845668696dcb28c3badd6 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 197b880fa39d432a9b472845876bfaf0 (127.18.211.1:42027), 974b802141bd4f13beffef2aa03f88f9 (127.18.211.2:34517)
I20260129 02:59:54.767338 22425 consensus_queue.cc:1048] T 78aa39096868415d9afc2ffcb248e198 P 1c50de89a5a845668696dcb28c3badd6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260129 02:59:54.768110 22645 leader_election.cc:290] T 368953c9102643b2a21f99af36b51905 P 197b880fa39d432a9b472845876bfaf0 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 974b802141bd4f13beffef2aa03f88f9 (127.18.211.2:34517), 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3:37735)
I20260129 02:59:54.768590 20441 raft_consensus.cc:3055] T a5a05fbcbbcf4916bbc01fd1a5b5e90d P 197b880fa39d432a9b472845876bfaf0 [term 6 LEADER]: Stepping down as leader of term 6
I20260129 02:59:54.769003 20441 raft_consensus.cc:740] T a5a05fbcbbcf4916bbc01fd1a5b5e90d P 197b880fa39d432a9b472845876bfaf0 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 197b880fa39d432a9b472845876bfaf0, State: Running, Role: LEADER
I20260129 02:59:54.746498 22833 raft_consensus.cc:515] T 42d27a28d89b409e8807ec0efd97af69 P 7a9daf5d42a84615a6383cfec9a10608 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } }
W20260129 02:59:54.758880 22847 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 1.279s user 0.002s sys 0.010s
I20260129 02:59:54.770025 20441 consensus_queue.cc:260] T a5a05fbcbbcf4916bbc01fd1a5b5e90d P 197b880fa39d432a9b472845876bfaf0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 6.5, Last appended by leader: 5, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:59:54.749054 20514 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57600 (request call id 1794) took 2171 ms. Trace:
W20260129 02:59:54.749198 20512 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:57584 (request call id 1336) took 1866 ms (client timeout 1508 ms). Trace:
I20260129 02:59:54.772728 20441 raft_consensus.cc:3060] T a5a05fbcbbcf4916bbc01fd1a5b5e90d P 197b880fa39d432a9b472845876bfaf0 [term 6 FOLLOWER]: Advancing to term 7
W20260129 02:59:54.772953 20512 rpcz_store.cc:269] 0129 02:59:52.882820 (+ 0us) service_pool.cc:168] Inserting onto call queue
0129 02:59:52.888779 (+ 5959us) service_pool.cc:225] Handling call
0129 02:59:54.748782 (+1860003us) spinlock_profiling.cc:243] Waited 1.67 s on lock 0x7b540026f054. stack: 00007f8fe044e373 00007f8fe044e270 00007f8fdfda0574 00007f8fe9a5c6b4 00007f8fe9a5c649 00007f8fe34975a1 00007f8fe3531f7a 00007f8fe84e3647 00007f8fe1a73834 00007f8fe1a737c2 00007f8fe1a736f1 00007f8fe1a7366c 00007f8fe1a72922 00007f8fe186d34c 00007f8fe186c786 00007f8fe186c12f
0129 02:59:54.748985 (+ 203us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":3840583424}
I20260129 02:59:54.747740 20510 raft_consensus.cc:1240] T 4408006eb5174de1896b037cf3f18457 P 974b802141bd4f13beffef2aa03f88f9 [term 3 FOLLOWER]: Rejecting Update request from peer 197b880fa39d432a9b472845876bfaf0 for earlier term 2. Current term is 3. Ops: [2.2-2.2]
I20260129 02:59:54.774845 20512 raft_consensus.cc:1240] T 0678b8b468bd43cdbc3e485900844821 P 974b802141bd4f13beffef2aa03f88f9 [term 6 FOLLOWER]: Rejecting Update request from peer 197b880fa39d432a9b472845876bfaf0 for earlier term 5. Current term is 6. Ops: []
I20260129 02:59:54.775110 22903 raft_consensus.cc:515] T 4b1876f9a70b49edb3aa74105e66d3b3 P 1c50de89a5a845668696dcb28c3badd6 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:59:54.741827 20480 leader_election.cc:304] T a8479d1fe0bd41e5946186e3a6ca8c7a P 974b802141bd4f13beffef2aa03f88f9 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1c50de89a5a845668696dcb28c3badd6, 974b802141bd4f13beffef2aa03f88f9; no voters:
I20260129 02:59:54.775792 20510 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57600 (request call id 1799) took 1237 ms. Trace:
I20260129 02:59:54.760461 20513 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57600 (request call id 1798) took 1220 ms. Trace:
I20260129 02:59:54.761024 20516 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:57600 (request call id 1804) took 1075 ms. Trace:
I20260129 02:59:54.778235 20516 rpcz_store.cc:276] 0129 02:59:53.682155 (+ 0us) service_pool.cc:168] Inserting onto call queue
0129 02:59:54.661659 (+979504us) service_pool.cc:225] Handling call
0129 02:59:54.757718 (+ 96059us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260129 02:59:54.763768 20585 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "78aa39096868415d9afc2ffcb248e198" candidate_uuid: "7a9daf5d42a84615a6383cfec9a10608" candidate_term: 6 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "1c50de89a5a845668696dcb28c3badd6" is_pre_election: true
I20260129 02:59:54.732556 22881 raft_consensus.cc:3060] T 4e3805ffa70242e8ade66da260dddf66 P 1c50de89a5a845668696dcb28c3badd6 [term 3 FOLLOWER]: Advancing to term 4
W20260129 02:59:54.778980 20554 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260129 02:59:54.724593 22654 raft_consensus.cc:493] T a4c1226b987b468fa84dfc6ba0e2dbe1 P 1c50de89a5a845668696dcb28c3badd6 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260129 02:59:54.779950 20554 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260129 02:59:54.780618 22878 raft_consensus.cc:2804] T a8479d1fe0bd41e5946186e3a6ca8c7a P 974b802141bd4f13beffef2aa03f88f9 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260129 02:59:54.781165 22878 raft_consensus.cc:493] T a8479d1fe0bd41e5946186e3a6ca8c7a P 974b802141bd4f13beffef2aa03f88f9 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260129 02:59:54.781539 22878 raft_consensus.cc:3060] T a8479d1fe0bd41e5946186e3a6ca8c7a P 974b802141bd4f13beffef2aa03f88f9 [term 4 FOLLOWER]: Advancing to term 5
I20260129 02:59:54.770606 20508 raft_consensus.cc:1240] T 4e420f90884f4796997b5ea41e49e411 P 974b802141bd4f13beffef2aa03f88f9 [term 5 FOLLOWER]: Rejecting Update request from peer 197b880fa39d432a9b472845876bfaf0 for earlier term 4. Current term is 5. Ops: []
I20260129 02:59:54.782531 20508 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57600 (request call id 1801) took 1243 ms. Trace:
I20260129 02:59:54.777912 20513 rpcz_store.cc:276] 0129 02:59:53.538039 (+ 0us) service_pool.cc:168] Inserting onto call queue
0129 02:59:54.708058 (+1170019us) service_pool.cc:225] Handling call
0129 02:59:54.728527 (+ 20469us) raft_consensus.cc:1471] Updating replica for 0 ops
0129 02:59:54.758462 (+ 29935us) raft_consensus.cc:1697] Filling consensus response to leader.
0129 02:59:54.758643 (+ 181us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260129 02:59:54.782869 20508 rpcz_store.cc:276] 0129 02:59:53.539501 (+ 0us) service_pool.cc:168] Inserting onto call queue
0129 02:59:54.769921 (+1230420us) service_pool.cc:225] Handling call
0129 02:59:54.770286 (+ 365us) raft_consensus.cc:1471] Updating replica for 0 ops
0129 02:59:54.782243 (+ 11957us) raft_consensus.cc:1697] Filling consensus response to leader.
0129 02:59:54.782448 (+ 205us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260129 02:59:54.784194 22425 raft_consensus.cc:493] T cb4727386a3e48aca5d71cc6c611c197 P 1c50de89a5a845668696dcb28c3badd6 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260129 02:59:54.736545 22851 raft_consensus.cc:740] T 0678b8b468bd43cdbc3e485900844821 P 197b880fa39d432a9b472845876bfaf0 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 197b880fa39d432a9b472845876bfaf0, State: Running, Role: LEADER
I20260129 02:59:54.784996 22425 raft_consensus.cc:515] T cb4727386a3e48aca5d71cc6c611c197 P 1c50de89a5a845668696dcb28c3badd6 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:59:54.786137 22851 consensus_queue.cc:260] T 0678b8b468bd43cdbc3e485900844821 P 197b880fa39d432a9b472845876bfaf0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 5.7, Last appended by leader: 7, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } }
I20260129 02:59:54.764155 22877 raft_consensus.cc:515] T 2e58e9a5a8aa4cecafb464addb4f35ed P 197b880fa39d432a9b472845876bfaf0 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } }
I20260129 02:59:54.772388 20514 rpcz_store.cc:276] 0129 02:59:52.577910 (+ 0us) service_pool.cc:168] Inserting onto call queue
0129 02:59:52.580068 (+ 2158us) service_pool.cc:225] Handling call
0129 02:59:52.580424 (+ 356us) raft_consensus.cc:1471] Updating replica for 1 ops
0129 02:59:52.846853 (+266429us) raft_consensus.cc:1534] Early marking committed up to index 0
0129 02:59:52.846900 (+ 47us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0129 02:59:52.907316 (+ 60416us) log.cc:844] Serialized 26 byte log entry
0129 02:59:52.936188 (+ 28872us) raft_consensus.cc:1645] Marking committed up to 4
0129 02:59:52.936224 (+ 36us) raft_consensus.cc:1697] Filling consensus response to leader.
0129 02:59:52.936294 (+ 70us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0129 02:59:54.069328 (+1133034us) spinlock_profiling.cc:243] Waited 5.99 ms on lock 0x7b440002304c. stack: 00007f8fe044e373 00007f8fe044e270 00007f8fdfda0574 00007f8fe9a5c6b4 00007f8fe9a5c649 00007f8fe02c08b1 00007f8fe038f005 00007f8fe038fa22 00007f8fe30d957b 00007f8fe30d4e4f 00007f8fe30c8c0e 00007f8fe30c7f56 00007f8fe30c7b78 00007f8fe30d15c1 00007f8fe30d1579 00007f8fe30d1509
0129 02:59:54.748372 (+679044us) raft_consensus.cc:1685] finished
0129 02:59:54.748408 (+ 36us) raft_consensus.cc:1691] UpdateReplicas() finished
0129 02:59:54.748820 (+ 412us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":35638,"raft.queue_time_us":43739,"raft.run_cpu_time_us":174,"raft.run_wall_time_us":21899,"spinlock_wait_cycles":13782400,"thread_start_us":209780,"threads_started":2,"wal-append.queue_time_us":305261}
I20260129 02:59:54.769189 22691 consensus_queue.cc:260] T 4e420f90884f4796997b5ea41e49e411 P 197b880fa39d432a9b472845876bfaf0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } }
I20260129 02:59:54.749673 22856 raft_consensus.cc:493] T e7d83d4ccf684e76acb29a0dc844e1b8 P 1c50de89a5a845668696dcb28c3badd6 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260129 02:59:54.788319 22691 raft_consensus.cc:3060] T 4e420f90884f4796997b5ea41e49e411 P 197b880fa39d432a9b472845876bfaf0 [term 4 FOLLOWER]: Advancing to term 5
I20260129 02:59:54.788154 22856 raft_consensus.cc:515] T e7d83d4ccf684e76acb29a0dc844e1b8 P 1c50de89a5a845668696dcb28c3badd6 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:59:54.788961 22877 leader_election.cc:290] T 2e58e9a5a8aa4cecafb464addb4f35ed P 197b880fa39d432a9b472845876bfaf0 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3:37735), 7a9daf5d42a84615a6383cfec9a10608 (127.18.211.4:40439)
W20260129 02:59:54.777429 20480 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.211.3:37735, user_credentials={real_user=slave}} blocked reactor thread for 56551.1us
I20260129 02:59:54.751138 22667 raft_consensus.cc:515] T 75f45538dc704d9a8ec4ff8ad9f3ea5b P 7a9daf5d42a84615a6383cfec9a10608 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } }
I20260129 02:59:54.786108 22900 raft_consensus.cc:493] T 7d9494f5584d4d5db8b9adf1c8fc888b P 1c50de89a5a845668696dcb28c3badd6 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260129 02:59:54.791322 22900 raft_consensus.cc:515] T 7d9494f5584d4d5db8b9adf1c8fc888b P 1c50de89a5a845668696dcb28c3badd6 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:59:54.777868 20510 rpcz_store.cc:276] 0129 02:59:53.538512 (+ 0us) service_pool.cc:168] Inserting onto call queue
0129 02:59:54.711254 (+1172742us) service_pool.cc:225] Handling call
0129 02:59:54.741109 (+ 29855us) raft_consensus.cc:1471] Updating replica for 1 ops
0129 02:59:54.775208 (+ 34099us) raft_consensus.cc:1697] Filling consensus response to leader.
0129 02:59:54.775546 (+ 338us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260129 02:59:54.793111 20507 raft_consensus.cc:1240] T a5a05fbcbbcf4916bbc01fd1a5b5e90d P 974b802141bd4f13beffef2aa03f88f9 [term 7 LEADER]: Rejecting Update request from peer 197b880fa39d432a9b472845876bfaf0 for earlier term 6. Current term is 7. Ops: []
I20260129 02:59:54.793501 20513 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "368953c9102643b2a21f99af36b51905" candidate_uuid: "197b880fa39d432a9b472845876bfaf0" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "974b802141bd4f13beffef2aa03f88f9" is_pre_election: true
I20260129 02:59:54.787204 22654 raft_consensus.cc:515] T a4c1226b987b468fa84dfc6ba0e2dbe1 P 1c50de89a5a845668696dcb28c3badd6 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:59:54.795923 22847 consensus_queue.cc:1048] T 8055232cf0004d40a85f3bf983fe00a9 P 974b802141bd4f13beffef2aa03f88f9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260129 02:59:54.796543 20480 leader_election.cc:400] T 3e2d15f8640b407195210c476e09df91 P 974b802141bd4f13beffef2aa03f88f9 [CANDIDATE]: Term 2 election: Vote denied by peer 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3:37735) with higher term. Message: Invalid argument: T 3e2d15f8640b407195210c476e09df91 P 1c50de89a5a845668696dcb28c3badd6 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 974b802141bd4f13beffef2aa03f88f9 for earlier term 2. Current term is 3.
I20260129 02:59:54.797025 22654 leader_election.cc:290] T a4c1226b987b468fa84dfc6ba0e2dbe1 P 1c50de89a5a845668696dcb28c3badd6 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 197b880fa39d432a9b472845876bfaf0 (127.18.211.1:42027), 974b802141bd4f13beffef2aa03f88f9 (127.18.211.2:34517)
I20260129 02:59:54.775614 20512 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57600 (request call id 1803) took 1094 ms. Trace:
W20260129 02:59:54.722327 21128 rpcz_store.cc:269] 0129 02:59:52.644603 (+ 0us) service_pool.cc:168] Inserting onto call queue
0129 02:59:52.659294 (+ 14691us) service_pool.cc:225] Handling call
0129 02:59:54.158308 (+1499014us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":30080}
I20260129 02:59:54.796976 20480 leader_election.cc:403] T 3e2d15f8640b407195210c476e09df91 P 974b802141bd4f13beffef2aa03f88f9 [CANDIDATE]: Term 2 election: Cancelling election due to peer responding with higher term
I20260129 02:59:54.799607 20512 rpcz_store.cc:276] 0129 02:59:53.681611 (+ 0us) service_pool.cc:168] Inserting onto call queue
0129 02:59:54.774176 (+1092565us) service_pool.cc:225] Handling call
0129 02:59:54.774537 (+ 361us) raft_consensus.cc:1471] Updating replica for 0 ops
0129 02:59:54.775316 (+ 779us) raft_consensus.cc:1697] Filling consensus response to leader.
0129 02:59:54.775527 (+ 211us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20260129 02:59:54.842407 20405 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260129 02:59:54.845273 20433 raft_consensus.cc:2468] T 0340229d521b432580ae926f6062e3f7 P 197b880fa39d432a9b472845876bfaf0 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 974b802141bd4f13beffef2aa03f88f9 in term 5.
I20260129 02:59:54.847429 22737 raft_consensus.cc:493] T 1299723a4476420ab02fd8bfd7aa0645 P 197b880fa39d432a9b472845876bfaf0 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260129 02:59:54.848186 22737 raft_consensus.cc:515] T 1299723a4476420ab02fd8bfd7aa0645 P 197b880fa39d432a9b472845876bfaf0 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } }
I20260129 02:59:54.849460 22900 leader_election.cc:290] T 7d9494f5584d4d5db8b9adf1c8fc888b P 1c50de89a5a845668696dcb28c3badd6 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 197b880fa39d432a9b472845876bfaf0 (127.18.211.1:42027), 974b802141bd4f13beffef2aa03f88f9 (127.18.211.2:34517)
I20260129 02:59:54.799360 22719 raft_consensus.cc:515] T 00e79d8f25a34676b1d866eb2068a2b2 P 974b802141bd4f13beffef2aa03f88f9 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } }
I20260129 02:59:54.852027 20511 raft_consensus.cc:1275] T 7a9a69cd82724b1faed1d10aa091fc4a P 974b802141bd4f13beffef2aa03f88f9 [term 5 FOLLOWER]: Refusing update from remote peer 197b880fa39d432a9b472845876bfaf0: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 5 index: 4. (index mismatch)
I20260129 02:59:54.852182 22719 leader_election.cc:290] T 00e79d8f25a34676b1d866eb2068a2b2 P 974b802141bd4f13beffef2aa03f88f9 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 197b880fa39d432a9b472845876bfaf0 (127.18.211.1:42027), 7a9daf5d42a84615a6383cfec9a10608 (127.18.211.4:40439)
I20260129 02:59:54.761953 22493 raft_consensus.cc:515] T cb0b3f259ebc4310927338a8740e87b8 P 974b802141bd4f13beffef2aa03f88f9 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } }
I20260129 02:59:54.852749 20511 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57600 (request call id 1800) took 1313 ms. Trace:
I20260129 02:59:54.853080 20511 rpcz_store.cc:276] 0129 02:59:53.539028 (+ 0us) service_pool.cc:168] Inserting onto call queue
0129 02:59:54.724850 (+1185822us) service_pool.cc:225] Handling call
0129 02:59:54.757260 (+ 32410us) raft_consensus.cc:1471] Updating replica for 0 ops
0129 02:59:54.852454 (+ 95194us) raft_consensus.cc:1697] Filling consensus response to leader.
0129 02:59:54.852674 (+ 220us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260129 02:59:54.823127 22645 raft_consensus.cc:493] T 00e79d8f25a34676b1d866eb2068a2b2 P 197b880fa39d432a9b472845876bfaf0 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260129 02:59:54.854346 22493 leader_election.cc:290] T cb0b3f259ebc4310927338a8740e87b8 P 974b802141bd4f13beffef2aa03f88f9 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3:37735), 7a9daf5d42a84615a6383cfec9a10608 (127.18.211.4:40439)
W20260129 02:59:54.856554 21099 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.211.1:42027, user_credentials={real_user=slave}} blocked reactor thread for 145196us
I20260129 02:59:54.799599 22581 raft_consensus.cc:515] T 0f20af7fbe5d4db89a6f17948d98912c P 1c50de89a5a845668696dcb28c3badd6 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:59:54.857604 21099 leader_election.cc:304] T 2ac659d732314e1288849f08fe5e074e P 7a9daf5d42a84615a6383cfec9a10608 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 197b880fa39d432a9b472845876bfaf0, 7a9daf5d42a84615a6383cfec9a10608; no voters: 974b802141bd4f13beffef2aa03f88f9
I20260129 02:59:54.858760 22581 leader_election.cc:290] T 0f20af7fbe5d4db89a6f17948d98912c P 1c50de89a5a845668696dcb28c3badd6 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 197b880fa39d432a9b472845876bfaf0 (127.18.211.1:42027), 974b802141bd4f13beffef2aa03f88f9 (127.18.211.2:34517)
I20260129 02:59:54.860126 22656 consensus_queue.cc:237] T 3e2d15f8640b407195210c476e09df91 P 1c50de89a5a845668696dcb28c3badd6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } }
I20260129 02:59:54.755467 22852 raft_consensus.cc:515] T 3906b080a5b34a74b8b64128635e8997 P 197b880fa39d432a9b472845876bfaf0 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:59:54.865284 22852 leader_election.cc:290] T 3906b080a5b34a74b8b64128635e8997 P 197b880fa39d432a9b472845876bfaf0 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 974b802141bd4f13beffef2aa03f88f9 (127.18.211.2:34517), 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3:37735)
I20260129 02:59:54.750741 22694 raft_consensus.cc:515] T 17d6431cc77a491e8464d68db4081a51 P 974b802141bd4f13beffef2aa03f88f9 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:59:54.867442 22872 raft_consensus.cc:2804] T 3ccac82a01a2445fa5b0ce6c0fcf2295 P 7a9daf5d42a84615a6383cfec9a10608 [term 4 FOLLOWER]: Leader election won for term 4
I20260129 02:59:54.867897 22763 raft_consensus.cc:515] T 368953c9102643b2a21f99af36b51905 P 974b802141bd4f13beffef2aa03f88f9 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:59:54.868396 20585 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4f5a77e94fef4b16a2ba7b067c8b144a" candidate_uuid: "7a9daf5d42a84615a6383cfec9a10608" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "1c50de89a5a845668696dcb28c3badd6" is_pre_election: true
I20260129 02:59:54.868710 22857 raft_consensus.cc:493] T 4e3805ffa70242e8ade66da260dddf66 P 7a9daf5d42a84615a6383cfec9a10608 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260129 02:59:54.868156 22872 raft_consensus.cc:697] T 3ccac82a01a2445fa5b0ce6c0fcf2295 P 7a9daf5d42a84615a6383cfec9a10608 [term 4 LEADER]: Becoming Leader. State: Replica: 7a9daf5d42a84615a6383cfec9a10608, State: Running, Role: LEADER
I20260129 02:59:54.869922 20513 raft_consensus.cc:2468] T 368953c9102643b2a21f99af36b51905 P 974b802141bd4f13beffef2aa03f88f9 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 197b880fa39d432a9b472845876bfaf0 in term 6.
I20260129 02:59:54.870685 21131 raft_consensus.cc:1240] T add8d4923d1a489e86882c3a25692b95 P 7a9daf5d42a84615a6383cfec9a10608 [term 4 FOLLOWER]: Rejecting Update request from peer 1c50de89a5a845668696dcb28c3badd6 for earlier term 3. Current term is 4. Ops: []
I20260129 02:59:54.871553 20436 raft_consensus.cc:3060] T 36ff33be666a4c82954731691dbcf327 P 197b880fa39d432a9b472845876bfaf0 [term 2 FOLLOWER]: Advancing to term 3
I20260129 02:59:54.871778 20588 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "45cc5f3d2688486d9b6dbe7e666dedde" candidate_uuid: "7a9daf5d42a84615a6383cfec9a10608" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "1c50de89a5a845668696dcb28c3badd6" is_pre_election: true
I20260129 02:59:54.870560 22872 consensus_queue.cc:237] T 3ccac82a01a2445fa5b0ce6c0fcf2295 P 7a9daf5d42a84615a6383cfec9a10608 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } }
I20260129 02:59:54.875502 20582 raft_consensus.cc:1275] T a644c02b16af47679824ccd060ac10fd P 1c50de89a5a845668696dcb28c3badd6 [term 4 FOLLOWER]: Refusing update from remote peer 7a9daf5d42a84615a6383cfec9a10608: Log matching property violated. Preceding OpId in replica: term: 3 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20260129 02:59:54.869403 22857 raft_consensus.cc:515] T 4e3805ffa70242e8ade66da260dddf66 P 7a9daf5d42a84615a6383cfec9a10608 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } }
I20260129 02:59:54.877418 22903 leader_election.cc:290] T 4b1876f9a70b49edb3aa74105e66d3b3 P 1c50de89a5a845668696dcb28c3badd6 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 197b880fa39d432a9b472845876bfaf0 (127.18.211.1:42027), 974b802141bd4f13beffef2aa03f88f9 (127.18.211.2:34517)
I20260129 02:59:54.877950 20586 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "96609ac68bdf4993a12e1c23355ca013" candidate_uuid: "7a9daf5d42a84615a6383cfec9a10608" candidate_term: 6 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "1c50de89a5a845668696dcb28c3badd6" is_pre_election: true
I20260129 02:59:54.878602 20586 raft_consensus.cc:2468] T 96609ac68bdf4993a12e1c23355ca013 P 1c50de89a5a845668696dcb28c3badd6 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7a9daf5d42a84615a6383cfec9a10608 in term 5.
I20260129 02:59:54.878908 22857 leader_election.cc:290] T 4e3805ffa70242e8ade66da260dddf66 P 7a9daf5d42a84615a6383cfec9a10608 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 197b880fa39d432a9b472845876bfaf0 (127.18.211.1:42027), 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3:37735)
I20260129 02:59:54.879745 20581 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "75f45538dc704d9a8ec4ff8ad9f3ea5b" candidate_uuid: "7a9daf5d42a84615a6383cfec9a10608" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "1c50de89a5a845668696dcb28c3badd6" is_pre_election: true
I20260129 02:59:54.880594 20581 raft_consensus.cc:2468] T 75f45538dc704d9a8ec4ff8ad9f3ea5b P 1c50de89a5a845668696dcb28c3badd6 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7a9daf5d42a84615a6383cfec9a10608 in term 3.
I20260129 02:59:54.758589 22875 raft_consensus.cc:515] T 1ce4ddf5b10e4ae8b193b13b2e5936ee P 197b880fa39d432a9b472845876bfaf0 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:59:54.885089 20434 raft_consensus.cc:2410] T 75be4bd6bd45474795fd472eff2953ac P 197b880fa39d432a9b472845876bfaf0 [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 974b802141bd4f13beffef2aa03f88f9 for term 10 because replica has last-logged OpId of term: 8 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 7 index: 5.
I20260129 02:59:54.870054 22763 leader_election.cc:290] T 368953c9102643b2a21f99af36b51905 P 974b802141bd4f13beffef2aa03f88f9 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 197b880fa39d432a9b472845876bfaf0 (127.18.211.1:42027), 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3:37735)
I20260129 02:59:54.790373 22856 leader_election.cc:290] T e7d83d4ccf684e76acb29a0dc844e1b8 P 1c50de89a5a845668696dcb28c3badd6 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 197b880fa39d432a9b472845876bfaf0 (127.18.211.1:42027), 974b802141bd4f13beffef2aa03f88f9 (127.18.211.2:34517)
W20260129 02:59:54.856686 20481 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260129 02:59:54.887305 20405 connection.cc:232] RPC call timeout handler was delayed by -1.9428859520000969s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2545s for a response
I20260129 02:59:54.787196 22425 leader_election.cc:290] T cb4727386a3e48aca5d71cc6c611c197 P 1c50de89a5a845668696dcb28c3badd6 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 197b880fa39d432a9b472845876bfaf0 (127.18.211.1:42027), 974b802141bd4f13beffef2aa03f88f9 (127.18.211.2:34517)
I20260129 02:59:54.888327 21135 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34046 (request call id 492) took 1287 ms. Trace:
I20260129 02:59:54.801050 22667 leader_election.cc:290] T 75f45538dc704d9a8ec4ff8ad9f3ea5b P 7a9daf5d42a84615a6383cfec9a10608 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 974b802141bd4f13beffef2aa03f88f9 (127.18.211.2:34517), 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3:37735)
I20260129 02:59:54.897961 22847 raft_consensus.cc:493] T 0678b8b468bd43cdbc3e485900844821 P 974b802141bd4f13beffef2aa03f88f9 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260129 02:59:54.902352 20481 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260129 02:59:54.899959 22719 raft_consensus.cc:2749] T 3e2d15f8640b407195210c476e09df91 P 974b802141bd4f13beffef2aa03f88f9 [term 3 FOLLOWER]: Leader election lost for term 2. Reason: Vote denied by peer 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3:37735) with higher term. Message: Invalid argument: T 3e2d15f8640b407195210c476e09df91 P 1c50de89a5a845668696dcb28c3badd6 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 974b802141bd4f13beffef2aa03f88f9 for earlier term 2. Current term is 3.
I20260129 02:59:54.902647 22847 raft_consensus.cc:515] T 0678b8b468bd43cdbc3e485900844821 P 974b802141bd4f13beffef2aa03f88f9 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } }
I20260129 02:59:54.905515 22763 raft_consensus.cc:493] T 913fc7d751ed4b7388d889aac30a5f9b P 974b802141bd4f13beffef2aa03f88f9 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260129 02:59:54.906092 22847 leader_election.cc:290] T 0678b8b468bd43cdbc3e485900844821 P 974b802141bd4f13beffef2aa03f88f9 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 197b880fa39d432a9b472845876bfaf0 (127.18.211.1:42027), 7a9daf5d42a84615a6383cfec9a10608 (127.18.211.4:40439)
I20260129 02:59:54.906004 22763 raft_consensus.cc:515] T 913fc7d751ed4b7388d889aac30a5f9b P 974b802141bd4f13beffef2aa03f88f9 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:59:54.907573 20512 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7a9a69cd82724b1faed1d10aa091fc4a" candidate_uuid: "1c50de89a5a845668696dcb28c3badd6" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "974b802141bd4f13beffef2aa03f88f9"
I20260129 02:59:54.908346 20512 raft_consensus.cc:3060] T 7a9a69cd82724b1faed1d10aa091fc4a P 974b802141bd4f13beffef2aa03f88f9 [term 5 FOLLOWER]: Advancing to term 6
I20260129 02:59:54.903786 22831 raft_consensus.cc:2804] T 2ac659d732314e1288849f08fe5e074e P 7a9daf5d42a84615a6383cfec9a10608 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260129 02:59:54.909083 22875 leader_election.cc:290] T 1ce4ddf5b10e4ae8b193b13b2e5936ee P 197b880fa39d432a9b472845876bfaf0 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 974b802141bd4f13beffef2aa03f88f9 (127.18.211.2:34517), 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3:37735)
I20260129 02:59:54.909278 22831 raft_consensus.cc:493] T 2ac659d732314e1288849f08fe5e074e P 7a9daf5d42a84615a6383cfec9a10608 [term 3 FOLLOWER]: Starting leader election (detected failure of leader 974b802141bd4f13beffef2aa03f88f9)
I20260129 02:59:54.909677 22831 raft_consensus.cc:3060] T 2ac659d732314e1288849f08fe5e074e P 7a9daf5d42a84615a6383cfec9a10608 [term 3 FOLLOWER]: Advancing to term 4
I20260129 02:59:54.902653 20585 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "75b0b6e82ddc4f92b1fe1d94f9a0b73b" candidate_uuid: "7a9daf5d42a84615a6383cfec9a10608" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "1c50de89a5a845668696dcb28c3badd6" is_pre_election: true
I20260129 02:59:54.903187 22719 raft_consensus.cc:493] T cb4727386a3e48aca5d71cc6c611c197 P 974b802141bd4f13beffef2aa03f88f9 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260129 02:59:54.908363 20516 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c891d2bff4ce4568861bbc992e91fbb0" candidate_uuid: "1c50de89a5a845668696dcb28c3badd6" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "974b802141bd4f13beffef2aa03f88f9" is_pre_election: true
I20260129 02:59:54.900447 21135 rpcz_store.cc:276] 0129 02:59:53.556820 (+ 0us) service_pool.cc:168] Inserting onto call queue
0129 02:59:53.561563 (+ 4743us) service_pool.cc:225] Handling call
0129 02:59:54.057915 (+496352us) raft_consensus.cc:1471] Updating replica for 0 ops
0129 02:59:54.679548 (+621633us) raft_consensus.cc:1697] Filling consensus response to leader.
0129 02:59:54.844262 (+164714us) spinlock_profiling.cc:243] Waited 496 ms on lock 0x7b54003b3dd0. stack: 00007f8fe044e373 00007f8fe044e270 00007f8fdfda0574 00007f8fe9a5c6b4 00007f8fe9a5c649 00007f8fe34975a1 00007f8fe352d52d 00007f8fe84e1f49 00007f8fe1a6ed34 00007f8fe1a6ecc2 00007f8fe1a6ebf1 00007f8fe1a6eb6c 00007f8fe1a6de22 00007f8fe186d34c 00007f8fe186c786 00007f8fe186c12f
0129 02:59:54.844468 (+ 206us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1140065024}
I20260129 02:59:54.905527 20584 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.LeaderStepDown from 127.0.0.1:54860 (request call id 12) took 1838 ms. Trace:
W20260129 02:59:54.915853 20405 connection.cc:232] RPC call timeout handler was delayed by -1.7870819760000813s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.19970000000000002s for a response
I20260129 02:59:54.927131 20511 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "913fc7d751ed4b7388d889aac30a5f9b" candidate_uuid: "1c50de89a5a845668696dcb28c3badd6" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "974b802141bd4f13beffef2aa03f88f9" is_pre_election: true
I20260129 02:59:54.929415 20511 raft_consensus.cc:2468] T 913fc7d751ed4b7388d889aac30a5f9b P 974b802141bd4f13beffef2aa03f88f9 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1c50de89a5a845668696dcb28c3badd6 in term 3.
W20260129 02:59:54.930860 20405 connection.cc:232] RPC call timeout handler was delayed by -1.6530267780000258s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.20870000000000002s for a response
I20260129 02:59:54.933924 20516 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "cb4727386a3e48aca5d71cc6c611c197" candidate_uuid: "1c50de89a5a845668696dcb28c3badd6" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "974b802141bd4f13beffef2aa03f88f9" is_pre_election: true
I20260129 02:59:54.915585 20584 rpcz_store.cc:276] 0129 02:59:53.067243 (+ 0us) service_pool.cc:168] Inserting onto call queue
0129 02:59:53.067401 (+ 158us) service_pool.cc:225] Handling call
0129 02:59:54.905293 (+1837892us) spinlock_profiling.cc:243] Waited 979 ms on lock 0x7b540018c4d4. stack: 00007f8fe044e373 00007f8fe044e270 00007f8fdfda0574 00007f8fe9a5c6b4 00007f8fe9a5c649 00007f8fe34975a1 00007f8fe35284d3 00007f8fe84e613a 00007f8fe1a81934 00007f8fe1a818c2 00007f8fe1a817f1 00007f8fe1a8176c 00007f8fe1a80a22 00007f8fe186d34c 00007f8fe186c786 00007f8fe186c12f
0129 02:59:54.905420 (+ 127us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2252406144}
I20260129 02:59:54.935968 22719 raft_consensus.cc:515] T cb4727386a3e48aca5d71cc6c611c197 P 974b802141bd4f13beffef2aa03f88f9 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:59:54.915654 22763 leader_election.cc:290] T 913fc7d751ed4b7388d889aac30a5f9b P 974b802141bd4f13beffef2aa03f88f9 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 197b880fa39d432a9b472845876bfaf0 (127.18.211.1:42027), 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3:37735)
I20260129 02:59:54.932195 22694 leader_election.cc:290] T 17d6431cc77a491e8464d68db4081a51 P 974b802141bd4f13beffef2aa03f88f9 [CANDIDATE]: Term 4 election: Requested vote from peers 197b880fa39d432a9b472845876bfaf0 (127.18.211.1:42027), 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3:37735)
I20260129 02:59:54.931733 22857 consensus_queue.cc:1048] T a644c02b16af47679824ccd060ac10fd P 7a9daf5d42a84615a6383cfec9a10608 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.006s
I20260129 02:59:54.938043 22719 leader_election.cc:290] T cb4727386a3e48aca5d71cc6c611c197 P 974b802141bd4f13beffef2aa03f88f9 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 197b880fa39d432a9b472845876bfaf0 (127.18.211.1:42027), 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3:37735)
I20260129 02:59:54.938727 20514 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "8055232cf0004d40a85f3bf983fe00a9" candidate_uuid: "1c50de89a5a845668696dcb28c3badd6" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "974b802141bd4f13beffef2aa03f88f9" is_pre_election: true
I20260129 02:59:54.939952 20514 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e7d83d4ccf684e76acb29a0dc844e1b8" candidate_uuid: "1c50de89a5a845668696dcb28c3badd6" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "974b802141bd4f13beffef2aa03f88f9" is_pre_election: true
I20260129 02:59:54.940725 20514 raft_consensus.cc:2468] T e7d83d4ccf684e76acb29a0dc844e1b8 P 974b802141bd4f13beffef2aa03f88f9 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1c50de89a5a845668696dcb28c3badd6 in term 5.
I20260129 02:59:54.941576 22654 raft_consensus.cc:493] T 4e3805ffa70242e8ade66da260dddf66 P 1c50de89a5a845668696dcb28c3badd6 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260129 02:59:54.942271 22654 raft_consensus.cc:515] T 4e3805ffa70242e8ade66da260dddf66 P 1c50de89a5a845668696dcb28c3badd6 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } }
W20260129 02:59:54.944062 21099 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.211.1:42027, user_credentials={real_user=slave}} blocked reactor thread for 86557.3us
I20260129 02:59:54.910697 22851 raft_consensus.cc:3060] T 0678b8b468bd43cdbc3e485900844821 P 197b880fa39d432a9b472845876bfaf0 [term 5 FOLLOWER]: Advancing to term 6
W20260129 02:59:54.956375 20405 connection.cc:232] RPC call timeout handler was delayed by -1.6365656839999474s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.19640000000000002s for a response
I20260129 02:59:54.957254 20516 raft_consensus.cc:2468] T cb4727386a3e48aca5d71cc6c611c197 P 974b802141bd4f13beffef2aa03f88f9 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1c50de89a5a845668696dcb28c3badd6 in term 3.
W20260129 02:59:54.960896 20482 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260129 02:59:54.854360 22645 raft_consensus.cc:515] T 00e79d8f25a34676b1d866eb2068a2b2 P 197b880fa39d432a9b472845876bfaf0 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } }
I20260129 02:59:54.961795 20510 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "96609ac68bdf4993a12e1c23355ca013" candidate_uuid: "1c50de89a5a845668696dcb28c3badd6" candidate_term: 6 candidate_status { last_received { term: 2 index: 3 } } ignore_live_leader: false dest_uuid: "974b802141bd4f13beffef2aa03f88f9" is_pre_election: true
I20260129 02:59:54.964807 20585 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4e3805ffa70242e8ade66da260dddf66" candidate_uuid: "7a9daf5d42a84615a6383cfec9a10608" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "1c50de89a5a845668696dcb28c3badd6" is_pre_election: true
I20260129 02:59:54.965392 20510 raft_consensus.cc:2410] T 96609ac68bdf4993a12e1c23355ca013 P 974b802141bd4f13beffef2aa03f88f9 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1c50de89a5a845668696dcb28c3badd6 for term 6 because replica has last-logged OpId of term: 3 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 3.
I20260129 02:59:54.962357 20508 raft_consensus.cc:1275] T c934d31efd3f4e5195605ddc354d14e1 P 974b802141bd4f13beffef2aa03f88f9 [term 5 FOLLOWER]: Refusing update from remote peer 197b880fa39d432a9b472845876bfaf0: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 5 index: 7. (index mismatch)
I20260129 02:59:54.965372 20585 raft_consensus.cc:2468] T 4e3805ffa70242e8ade66da260dddf66 P 1c50de89a5a845668696dcb28c3badd6 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7a9daf5d42a84615a6383cfec9a10608 in term 4.
W20260129 02:59:54.968456 20405 connection.cc:232] RPC call timeout handler was delayed by -1.6112047179999536s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.19830000000000003s for a response
W20260129 02:59:54.968976 20405 connection.cc:232] RPC call timeout handler was delayed by -1.5995457410000427s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.21110000000000004s for a response
W20260129 02:59:54.969368 20405 connection.cc:232] RPC call timeout handler was delayed by -1.5572457410000879s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2158s for a response
W20260129 02:59:54.972889 20405 connection.cc:232] RPC call timeout handler was delayed by -1.5106518570000844s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.19970000000000002s for a response
I20260129 02:59:54.931340 22737 leader_election.cc:290] T 1299723a4476420ab02fd8bfd7aa0645 P 197b880fa39d432a9b472845876bfaf0 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3:37735), 7a9daf5d42a84615a6383cfec9a10608 (127.18.211.4:40439)
I20260129 02:59:54.976639 22645 leader_election.cc:290] T 00e79d8f25a34676b1d866eb2068a2b2 P 197b880fa39d432a9b472845876bfaf0 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 974b802141bd4f13beffef2aa03f88f9 (127.18.211.2:34517), 7a9daf5d42a84615a6383cfec9a10608 (127.18.211.4:40439)
W20260129 02:59:54.977409 20480 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.211.3:37735, user_credentials={real_user=slave}} blocked reactor thread for 181091us
I20260129 02:59:54.980819 20436 raft_consensus.cc:2468] T 36ff33be666a4c82954731691dbcf327 P 197b880fa39d432a9b472845876bfaf0 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 974b802141bd4f13beffef2aa03f88f9 in term 3.
W20260129 02:59:54.987017 20405 connection.cc:232] RPC call timeout handler was delayed by -1.391851857000006s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2129s for a response
W20260129 02:59:54.987700 20405 connection.cc:232] RPC call timeout handler was delayed by -1.3574457409999923s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.238s for a response
W20260129 02:59:54.988195 20405 connection.cc:232] RPC call timeout handler was delayed by -1.3223457410000492s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2419s for a response
W20260129 02:59:54.988629 20405 connection.cc:232] RPC call timeout handler was delayed by -1.2935457410000026s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2451s for a response
W20260129 02:59:54.999001 20405 connection.cc:232] RPC call timeout handler was delayed by -1.2476047180000478s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.23870000000000002s for a response
W20260129 02:59:55.000000 20405 connection.cc:232] RPC call timeout handler was delayed by -1.1884448779999275s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.18220000000000003s for a response
W20260129 02:59:55.009791 20404 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260129 02:59:55.036252 22403 consensus_queue.cc:1048] T 1299723a4476420ab02fd8bfd7aa0645 P 7a9daf5d42a84615a6383cfec9a10608 [LEADER]: Connected to new peer: Peer: permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 4, Time since last communication: 0.001s
I20260129 02:59:55.038070 22833 leader_election.cc:290] T 42d27a28d89b409e8807ec0efd97af69 P 7a9daf5d42a84615a6383cfec9a10608 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 197b880fa39d432a9b472845876bfaf0 (127.18.211.1:42027), 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3:37735)
W20260129 02:59:55.039264 20404 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260129 02:59:55.056310 20512 raft_consensus.cc:2468] T 7a9a69cd82724b1faed1d10aa091fc4a P 974b802141bd4f13beffef2aa03f88f9 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1c50de89a5a845668696dcb28c3badd6 in term 6.
W20260129 02:59:55.056481 20480 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.211.3:37735, user_credentials={real_user=slave}} blocked reactor thread for 57690.2us
I20260129 02:59:55.047504 22833 raft_consensus.cc:493] T dc2f4cb752cf4d59b5f6bc689f5d8104 P 7a9daf5d42a84615a6383cfec9a10608 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 1c50de89a5a845668696dcb28c3badd6)
I20260129 02:59:55.064729 20439 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "913fc7d751ed4b7388d889aac30a5f9b" candidate_uuid: "1c50de89a5a845668696dcb28c3badd6" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "197b880fa39d432a9b472845876bfaf0" is_pre_election: true
I20260129 02:59:55.060647 20507 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57600 (request call id 1796) took 2239 ms. Trace:
I20260129 02:59:55.069022 20434 raft_consensus.cc:1240] T 36ff33be666a4c82954731691dbcf327 P 197b880fa39d432a9b472845876bfaf0 [term 3 FOLLOWER]: Rejecting Update request from peer 1c50de89a5a845668696dcb28c3badd6 for earlier term 2. Current term is 3. Ops: []
I20260129 02:59:55.072258 21131 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34046 (request call id 489) took 1883 ms. Trace:
I20260129 02:59:55.019958 22878 raft_consensus.cc:515] T a8479d1fe0bd41e5946186e3a6ca8c7a P 974b802141bd4f13beffef2aa03f88f9 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:59:55.068166 20507 rpcz_store.cc:276] 0129 02:59:52.820806 (+ 0us) service_pool.cc:168] Inserting onto call queue
0129 02:59:52.822346 (+ 1540us) service_pool.cc:225] Handling call
0129 02:59:52.822766 (+ 420us) raft_consensus.cc:1471] Updating replica for 0 ops
0129 02:59:54.793639 (+1970873us) raft_consensus.cc:1697] Filling consensus response to leader.
0129 02:59:55.039179 (+245540us) spinlock_profiling.cc:243] Waited 1.97 s on lock 0x7b5400228b54. stack: 00007f8fe044e373 00007f8fe044e270 00007f8fdfda0574 00007f8fe9a5c6b4 00007f8fe9a5c649 00007f8fe34975a1 00007f8fe352ea81 00007f8fe352d4be 00007f8fe84e1f49 00007f8fe1a6ed34 00007f8fe1a6ecc2 00007f8fe1a6ebf1 00007f8fe1a6eb6c 00007f8fe1a6de22 00007f8fe186d34c 00007f8fe186c786
0129 02:59:55.060570 (+ 21391us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":4530158464}
I20260129 02:59:55.054701 22737 raft_consensus.cc:493] T a8479d1fe0bd41e5946186e3a6ca8c7a P 197b880fa39d432a9b472845876bfaf0 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260129 02:59:55.059614 22343 leader_election.cc:290] T add8d4923d1a489e86882c3a25692b95 P 7a9daf5d42a84615a6383cfec9a10608 [CANDIDATE]: Term 4 election: Requested vote from peers 974b802141bd4f13beffef2aa03f88f9 (127.18.211.2:34517), 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3:37735)
I20260129 02:59:55.072791 21131 rpcz_store.cc:276] 0129 02:59:53.188907 (+ 0us) service_pool.cc:168] Inserting onto call queue
0129 02:59:53.229437 (+ 40530us) service_pool.cc:225] Handling call
0129 02:59:53.229820 (+ 383us) raft_consensus.cc:1471] Updating replica for 0 ops
0129 02:59:54.871236 (+1641416us) raft_consensus.cc:1697] Filling consensus response to leader.
0129 02:59:55.056243 (+185007us) spinlock_profiling.cc:243] Waited 1.62 s on lock 0x7b54005712d4. stack: 00007f8fe044e373 00007f8fe044e270 00007f8fdfda0574 00007f8fe9a5c6b4 00007f8fe9a5c649 00007f8fe34975a1 00007f8fe352ea81 00007f8fe352d4be 00007f8fe84e1f49 00007f8fe1a6ed34 00007f8fe1a6ecc2 00007f8fe1a6ebf1 00007f8fe1a6eb6c 00007f8fe1a6de22 00007f8fe186d34c 00007f8fe186c786
0129 02:59:55.072180 (+ 15937us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":3717594368}
W20260129 02:59:55.081950 22917 log.cc:927] Time spent T a4c1226b987b468fa84dfc6ba0e2dbe1 P 974b802141bd4f13beffef2aa03f88f9: Append to log took a long time: real 0.349s user 0.004s sys 0.004s
I20260129 02:59:55.083711 20515 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57600 (request call id 1797) took 1546 ms. Trace:
I20260129 02:59:55.068049 22877 raft_consensus.cc:493] T 78aa39096868415d9afc2ffcb248e198 P 197b880fa39d432a9b472845876bfaf0 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260129 02:59:55.079339 20405 leader_election.cc:304] T e33a417cf9c54cf5af73a4ee3aac4d3a P 197b880fa39d432a9b472845876bfaf0 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 197b880fa39d432a9b472845876bfaf0; no voters: 1c50de89a5a845668696dcb28c3badd6, 974b802141bd4f13beffef2aa03f88f9
I20260129 02:59:55.084183 20515 rpcz_store.cc:276] 0129 02:59:53.537622 (+ 0us) service_pool.cc:168] Inserting onto call queue
0129 02:59:53.537902 (+ 280us) service_pool.cc:225] Handling call
0129 02:59:53.538667 (+ 765us) raft_consensus.cc:1471] Updating replica for 1 ops
0129 02:59:53.539289 (+ 622us) raft_consensus.cc:1534] Early marking committed up to index 0
0129 02:59:53.539340 (+ 51us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0129 02:59:53.574686 (+ 35346us) log.cc:844] Serialized 26 byte log entry
0129 02:59:54.696518 (+1121832us) raft_consensus.cc:1645] Marking committed up to 2
0129 02:59:54.696579 (+ 61us) raft_consensus.cc:1697] Filling consensus response to leader.
0129 02:59:54.696660 (+ 81us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0129 02:59:55.073432 (+376772us) spinlock_profiling.cc:243] Waited 2.03 ms on lock 0x7b440002304c. stack: 00007f8fe044e373 00007f8fe044e270 00007f8fdfda0574 00007f8fe9a5c6b4 00007f8fe9a5c649 00007f8fe02c08b1 00007f8fe038f005 00007f8fe038fa22 00007f8fe30d957b 00007f8fe30d4e4f 00007f8fe30c8c0e 00007f8fe30c7f56 00007f8fe30c7b78 00007f8fe30d15c1 00007f8fe30d1579 00007f8fe30d1509
0129 02:59:55.083348 (+ 9916us) raft_consensus.cc:1685] finished
0129 02:59:55.083378 (+ 30us) raft_consensus.cc:1691] UpdateReplicas() finished
0129 02:59:55.083635 (+ 257us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":4663552,"thread_start_us":1120471,"threads_started":1,"wal-append.queue_time_us":1157085}
I20260129 02:59:55.106894 20439 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7a9a69cd82724b1faed1d10aa091fc4a" candidate_uuid: "1c50de89a5a845668696dcb28c3badd6" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "197b880fa39d432a9b472845876bfaf0"
I20260129 02:59:55.123651 20434 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "96609ac68bdf4993a12e1c23355ca013" candidate_uuid: "1c50de89a5a845668696dcb28c3badd6" candidate_term: 6 candidate_status { last_received { term: 2 index: 3 } } ignore_live_leader: false dest_uuid: "197b880fa39d432a9b472845876bfaf0" is_pre_election: true
I20260129 02:59:55.127030 20437 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "8055232cf0004d40a85f3bf983fe00a9" candidate_uuid: "1c50de89a5a845668696dcb28c3badd6" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "197b880fa39d432a9b472845876bfaf0" is_pre_election: true
I20260129 02:59:55.093052 22833 raft_consensus.cc:515] T dc2f4cb752cf4d59b5f6bc689f5d8104 P 7a9daf5d42a84615a6383cfec9a10608 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } }
I20260129 02:59:55.133257 20434 raft_consensus.cc:2410] T 96609ac68bdf4993a12e1c23355ca013 P 197b880fa39d432a9b472845876bfaf0 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1c50de89a5a845668696dcb28c3badd6 for term 6 because replica has last-logged OpId of term: 2 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 3.
I20260129 02:59:55.120718 22691 raft_consensus.cc:2749] T e33a417cf9c54cf5af73a4ee3aac4d3a P 197b880fa39d432a9b472845876bfaf0 [term 6 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260129 02:59:55.128638 22343 raft_consensus.cc:493] T 4e420f90884f4796997b5ea41e49e411 P 7a9daf5d42a84615a6383cfec9a10608 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260129 02:59:55.141031 20406 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260129 02:59:55.105626 22737 raft_consensus.cc:515] T a8479d1fe0bd41e5946186e3a6ca8c7a P 197b880fa39d432a9b472845876bfaf0 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
W20260129 02:59:55.181790 20405 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.211.3:37735, user_credentials={real_user=slave}} blocked reactor thread for 126404us
I20260129 02:59:55.181907 22581 raft_consensus.cc:493] T fcddab6bca684adf8ea032289a00f90e P 1c50de89a5a845668696dcb28c3badd6 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260129 02:59:55.187160 20407 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.211.2:34517, user_credentials={real_user=slave}} blocked reactor thread for 45233.1us
I20260129 02:59:55.131856 22877 raft_consensus.cc:515] T 78aa39096868415d9afc2ffcb248e198 P 197b880fa39d432a9b472845876bfaf0 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } }
W20260129 02:59:55.193483 20479 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.211.1:42027, user_credentials={real_user=slave}} blocked reactor thread for 512300us
W20260129 02:59:55.194562 20480 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.211.3:37735, user_credentials={real_user=slave}} blocked reactor thread for 74399.7us
I20260129 02:59:55.185745 22884 raft_consensus.cc:3055] T 45cc5f3d2688486d9b6dbe7e666dedde P 1c50de89a5a845668696dcb28c3badd6 [term 3 LEADER]: Stepping down as leader of term 3
I20260129 02:59:55.181951 22895 raft_consensus.cc:493] T cb0b3f259ebc4310927338a8740e87b8 P 1c50de89a5a845668696dcb28c3badd6 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 7a9daf5d42a84615a6383cfec9a10608)
I20260129 02:59:55.144253 22688 consensus_queue.cc:1059] T 0f20af7fbe5d4db89a6f17948d98912c P 197b880fa39d432a9b472845876bfaf0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 }, Status: INVALID_TERM, Last received: 4.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.049s
I20260129 02:59:55.193670 20407 leader_election.cc:304] T c2633a6fc34c44579de348d208b0c01e P 197b880fa39d432a9b472845876bfaf0 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 197b880fa39d432a9b472845876bfaf0, 974b802141bd4f13beffef2aa03f88f9; no voters:
I20260129 02:59:55.209187 20434 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "96609ac68bdf4993a12e1c23355ca013" candidate_uuid: "974b802141bd4f13beffef2aa03f88f9" candidate_term: 6 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "197b880fa39d432a9b472845876bfaf0" is_pre_election: true
I20260129 02:59:55.214905 20433 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "368953c9102643b2a21f99af36b51905" candidate_uuid: "974b802141bd4f13beffef2aa03f88f9" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "197b880fa39d432a9b472845876bfaf0" is_pre_election: true
I20260129 02:59:55.210069 20434 raft_consensus.cc:2468] T 96609ac68bdf4993a12e1c23355ca013 P 197b880fa39d432a9b472845876bfaf0 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 974b802141bd4f13beffef2aa03f88f9 in term 5.
I20260129 02:59:55.197351 22654 leader_election.cc:290] T 4e3805ffa70242e8ade66da260dddf66 P 1c50de89a5a845668696dcb28c3badd6 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 197b880fa39d432a9b472845876bfaf0 (127.18.211.1:42027), 7a9daf5d42a84615a6383cfec9a10608 (127.18.211.4:40439)
I20260129 02:59:55.221063 20437 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00e79d8f25a34676b1d866eb2068a2b2" candidate_uuid: "974b802141bd4f13beffef2aa03f88f9" candidate_term: 7 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "197b880fa39d432a9b472845876bfaf0" is_pre_election: true
W20260129 02:59:55.223515 20553 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.211.1:42027, user_credentials={real_user=slave}} blocked reactor thread for 523446us
I20260129 02:59:55.231000 20439 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "17d6431cc77a491e8464d68db4081a51" candidate_uuid: "974b802141bd4f13beffef2aa03f88f9" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "197b880fa39d432a9b472845876bfaf0"
I20260129 02:59:55.225322 20437 raft_consensus.cc:2468] T 00e79d8f25a34676b1d866eb2068a2b2 P 197b880fa39d432a9b472845876bfaf0 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 974b802141bd4f13beffef2aa03f88f9 in term 6.
I20260129 02:59:55.195909 22903 raft_consensus.cc:493] T 4f8f3a38f6e24ba4857875181f447442 P 1c50de89a5a845668696dcb28c3badd6 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260129 02:59:55.200639 22884 raft_consensus.cc:740] T 45cc5f3d2688486d9b6dbe7e666dedde P 1c50de89a5a845668696dcb28c3badd6 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 1c50de89a5a845668696dcb28c3badd6, State: Running, Role: LEADER
I20260129 02:59:55.237107 20434 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0678b8b468bd43cdbc3e485900844821" candidate_uuid: "974b802141bd4f13beffef2aa03f88f9" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "197b880fa39d432a9b472845876bfaf0" is_pre_election: true
I20260129 02:59:55.213385 22878 leader_election.cc:290] T a8479d1fe0bd41e5946186e3a6ca8c7a P 974b802141bd4f13beffef2aa03f88f9 [CANDIDATE]: Term 5 election: Requested vote from peers 197b880fa39d432a9b472845876bfaf0 (127.18.211.1:42027), 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3:37735)
I20260129 02:59:55.206636 22719 raft_consensus.cc:493] T cedce331e1584ac8b636c9b8a72b2ba0 P 974b802141bd4f13beffef2aa03f88f9 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260129 02:59:55.236641 20407 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.211.2:34517, user_credentials={real_user=slave}} blocked reactor thread for 48486.3us
I20260129 02:59:55.196084 22343 raft_consensus.cc:515] T 4e420f90884f4796997b5ea41e49e411 P 7a9daf5d42a84615a6383cfec9a10608 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } }
I20260129 02:59:55.222365 22748 raft_consensus.cc:493] T a8479d1fe0bd41e5946186e3a6ca8c7a P 1c50de89a5a845668696dcb28c3badd6 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260129 02:59:55.207415 22581 raft_consensus.cc:515] T fcddab6bca684adf8ea032289a00f90e P 1c50de89a5a845668696dcb28c3badd6 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:59:55.252630 20437 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "cb4727386a3e48aca5d71cc6c611c197" candidate_uuid: "974b802141bd4f13beffef2aa03f88f9" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "197b880fa39d432a9b472845876bfaf0" is_pre_election: true
I20260129 02:59:55.256537 22719 raft_consensus.cc:515] T cedce331e1584ac8b636c9b8a72b2ba0 P 974b802141bd4f13beffef2aa03f88f9 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:59:55.257922 22737 leader_election.cc:290] T a8479d1fe0bd41e5946186e3a6ca8c7a P 197b880fa39d432a9b472845876bfaf0 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 974b802141bd4f13beffef2aa03f88f9 (127.18.211.2:34517), 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3:37735)
I20260129 02:59:55.259004 22719 leader_election.cc:290] T cedce331e1584ac8b636c9b8a72b2ba0 P 974b802141bd4f13beffef2aa03f88f9 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 197b880fa39d432a9b472845876bfaf0 (127.18.211.1:42027), 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3:37735)
I20260129 02:59:55.259220 22708 raft_consensus.cc:493] T a644c02b16af47679824ccd060ac10fd P 1c50de89a5a845668696dcb28c3badd6 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260129 02:59:55.259802 22708 raft_consensus.cc:515] T a644c02b16af47679824ccd060ac10fd P 1c50de89a5a845668696dcb28c3badd6 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } }
I20260129 02:59:55.261272 22343 leader_election.cc:290] T 4e420f90884f4796997b5ea41e49e411 P 7a9daf5d42a84615a6383cfec9a10608 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 197b880fa39d432a9b472845876bfaf0 (127.18.211.1:42027), 974b802141bd4f13beffef2aa03f88f9 (127.18.211.2:34517)
I20260129 02:59:55.261175 22831 raft_consensus.cc:515] T 2ac659d732314e1288849f08fe5e074e P 7a9daf5d42a84615a6383cfec9a10608 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } }
I20260129 02:59:55.240346 22231 raft_consensus.cc:2804] T c2633a6fc34c44579de348d208b0c01e P 197b880fa39d432a9b472845876bfaf0 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260129 02:59:55.263214 20439 raft_consensus.cc:3060] T 17d6431cc77a491e8464d68db4081a51 P 197b880fa39d432a9b472845876bfaf0 [term 3 FOLLOWER]: Advancing to term 4
I20260129 02:59:55.265573 20433 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c891d2bff4ce4568861bbc992e91fbb0" candidate_uuid: "1c50de89a5a845668696dcb28c3badd6" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "197b880fa39d432a9b472845876bfaf0" is_pre_election: true
I20260129 02:59:55.241575 22640 raft_consensus.cc:493] T a5a05fbcbbcf4916bbc01fd1a5b5e90d P 1c50de89a5a845668696dcb28c3badd6 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260129 02:59:55.265501 21135 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4e3805ffa70242e8ade66da260dddf66" candidate_uuid: "1c50de89a5a845668696dcb28c3badd6" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "7a9daf5d42a84615a6383cfec9a10608" is_pre_election: true
W20260129 02:59:55.274859 20406 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260129 02:59:55.272192 20437 raft_consensus.cc:2468] T cb4727386a3e48aca5d71cc6c611c197 P 197b880fa39d432a9b472845876bfaf0 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 974b802141bd4f13beffef2aa03f88f9 in term 3.
I20260129 02:59:55.275063 21135 raft_consensus.cc:2468] T 4e3805ffa70242e8ade66da260dddf66 P 7a9daf5d42a84615a6383cfec9a10608 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1c50de89a5a845668696dcb28c3badd6 in term 4.
I20260129 02:59:55.252337 22900 raft_consensus.cc:493] T 1299723a4476420ab02fd8bfd7aa0645 P 1c50de89a5a845668696dcb28c3badd6 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 7a9daf5d42a84615a6383cfec9a10608)
I20260129 02:59:55.266944 22833 leader_election.cc:290] T dc2f4cb752cf4d59b5f6bc689f5d8104 P 7a9daf5d42a84615a6383cfec9a10608 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 197b880fa39d432a9b472845876bfaf0 (127.18.211.1:42027), 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3:37735)
W20260129 02:59:55.284528 20555 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260129 02:59:55.277281 22888 consensus_meta.cc:287] T 3ccac82a01a2445fa5b0ce6c0fcf2295 P 974b802141bd4f13beffef2aa03f88f9: Time spent flushing consensus metadata: real 0.558s user 0.001s sys 0.014s
I20260129 02:59:55.245340 22895 raft_consensus.cc:515] T cb0b3f259ebc4310927338a8740e87b8 P 1c50de89a5a845668696dcb28c3badd6 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } }
I20260129 02:59:55.273452 22231 raft_consensus.cc:493] T c2633a6fc34c44579de348d208b0c01e P 197b880fa39d432a9b472845876bfaf0 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 1c50de89a5a845668696dcb28c3badd6)
I20260129 02:59:55.296505 20437 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a4c1226b987b468fa84dfc6ba0e2dbe1" candidate_uuid: "1c50de89a5a845668696dcb28c3badd6" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "197b880fa39d432a9b472845876bfaf0" is_pre_election: true
I20260129 02:59:55.291978 20433 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "cb4727386a3e48aca5d71cc6c611c197" candidate_uuid: "1c50de89a5a845668696dcb28c3badd6" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "197b880fa39d432a9b472845876bfaf0" is_pre_election: true
I20260129 02:59:55.298316 20433 raft_consensus.cc:2468] T cb4727386a3e48aca5d71cc6c611c197 P 197b880fa39d432a9b472845876bfaf0 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1c50de89a5a845668696dcb28c3badd6 in term 3.
I20260129 02:59:55.263770 22748 raft_consensus.cc:515] T a8479d1fe0bd41e5946186e3a6ca8c7a P 1c50de89a5a845668696dcb28c3badd6 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
W20260129 02:59:55.290933 22919 log.cc:927] Time spent T 8055232cf0004d40a85f3bf983fe00a9 P 197b880fa39d432a9b472845876bfaf0: Append to log took a long time: real 0.213s user 0.000s sys 0.002s
I20260129 02:59:55.284030 22708 leader_election.cc:290] T a644c02b16af47679824ccd060ac10fd P 1c50de89a5a845668696dcb28c3badd6 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 974b802141bd4f13beffef2aa03f88f9 (127.18.211.2:34517), 7a9daf5d42a84615a6383cfec9a10608 (127.18.211.4:40439)
I20260129 02:59:55.270159 22903 raft_consensus.cc:515] T 4f8f3a38f6e24ba4857875181f447442 P 1c50de89a5a845668696dcb28c3badd6 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:59:55.276969 22645 consensus_queue.cc:1048] T 913fc7d751ed4b7388d889aac30a5f9b P 197b880fa39d432a9b472845876bfaf0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 1, Time since last communication: 0.026s
W20260129 02:59:55.324128 21100 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.211.3:37735, user_credentials={real_user=slave}} blocked reactor thread for 320634us
I20260129 02:59:55.327551 20585 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "42d27a28d89b409e8807ec0efd97af69" candidate_uuid: "197b880fa39d432a9b472845876bfaf0" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "1c50de89a5a845668696dcb28c3badd6" is_pre_election: true
I20260129 02:59:55.329274 20437 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7d9494f5584d4d5db8b9adf1c8fc888b" candidate_uuid: "1c50de89a5a845668696dcb28c3badd6" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "197b880fa39d432a9b472845876bfaf0" is_pre_election: true
I20260129 02:59:55.329679 20433 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4b1876f9a70b49edb3aa74105e66d3b3" candidate_uuid: "1c50de89a5a845668696dcb28c3badd6" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "197b880fa39d432a9b472845876bfaf0" is_pre_election: true
I20260129 02:59:55.299752 22231 raft_consensus.cc:3060] T c2633a6fc34c44579de348d208b0c01e P 197b880fa39d432a9b472845876bfaf0 [term 2 FOLLOWER]: Advancing to term 3
I20260129 02:59:55.347532 20589 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "cedce331e1584ac8b636c9b8a72b2ba0" candidate_uuid: "197b880fa39d432a9b472845876bfaf0" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "1c50de89a5a845668696dcb28c3badd6"
I20260129 02:59:55.349647 20407 leader_election.cc:304] T 368953c9102643b2a21f99af36b51905 P 197b880fa39d432a9b472845876bfaf0 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 197b880fa39d432a9b472845876bfaf0, 974b802141bd4f13beffef2aa03f88f9; no voters:
I20260129 02:59:55.350033 20588 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4b1876f9a70b49edb3aa74105e66d3b3" candidate_uuid: "197b880fa39d432a9b472845876bfaf0" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "1c50de89a5a845668696dcb28c3badd6" is_pre_election: true
I20260129 02:59:55.350731 20588 raft_consensus.cc:2468] T 4b1876f9a70b49edb3aa74105e66d3b3 P 1c50de89a5a845668696dcb28c3badd6 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 197b880fa39d432a9b472845876bfaf0 in term 4.
I20260129 02:59:55.351471 22877 leader_election.cc:290] T 78aa39096868415d9afc2ffcb248e198 P 197b880fa39d432a9b472845876bfaf0 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3:37735), 7a9daf5d42a84615a6383cfec9a10608 (127.18.211.4:40439)
I20260129 02:59:55.334360 20509 raft_consensus.cc:1240] T e33a417cf9c54cf5af73a4ee3aac4d3a P 974b802141bd4f13beffef2aa03f88f9 [term 6 LEADER]: Rejecting Update request from peer 1c50de89a5a845668696dcb28c3badd6 for earlier term 5. Current term is 6. Ops: [5.3-5.3]
I20260129 02:59:55.336156 20584 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "96609ac68bdf4993a12e1c23355ca013" candidate_uuid: "197b880fa39d432a9b472845876bfaf0" candidate_term: 5 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "1c50de89a5a845668696dcb28c3badd6" is_pre_election: true
I20260129 02:59:55.352476 20584 raft_consensus.cc:2393] T 96609ac68bdf4993a12e1c23355ca013 P 1c50de89a5a845668696dcb28c3badd6 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 197b880fa39d432a9b472845876bfaf0 in current term 5: Already voted for candidate 7a9daf5d42a84615a6383cfec9a10608 in this term.
I20260129 02:59:55.352849 21100 leader_election.cc:304] T 75f45538dc704d9a8ec4ff8ad9f3ea5b P 7a9daf5d42a84615a6383cfec9a10608 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1c50de89a5a845668696dcb28c3badd6, 7a9daf5d42a84615a6383cfec9a10608; no voters:
I20260129 02:59:55.336767 22581 leader_election.cc:290] T fcddab6bca684adf8ea032289a00f90e P 1c50de89a5a845668696dcb28c3badd6 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 197b880fa39d432a9b472845876bfaf0 (127.18.211.1:42027), 974b802141bd4f13beffef2aa03f88f9 (127.18.211.2:34517)
I20260129 02:59:55.353565 20515 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a4c1226b987b468fa84dfc6ba0e2dbe1" candidate_uuid: "1c50de89a5a845668696dcb28c3badd6" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "974b802141bd4f13beffef2aa03f88f9" is_pre_election: true
I20260129 02:59:55.340943 22184 raft_consensus.cc:493] T a4c1226b987b468fa84dfc6ba0e2dbe1 P 974b802141bd4f13beffef2aa03f88f9 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 197b880fa39d432a9b472845876bfaf0)
I20260129 02:59:55.293491 22900 raft_consensus.cc:515] T 1299723a4476420ab02fd8bfd7aa0645 P 1c50de89a5a845668696dcb28c3badd6 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } }
I20260129 02:59:55.354319 22184 raft_consensus.cc:515] T a4c1226b987b468fa84dfc6ba0e2dbe1 P 974b802141bd4f13beffef2aa03f88f9 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:59:55.355006 22880 raft_consensus.cc:2804] T 75f45538dc704d9a8ec4ff8ad9f3ea5b P 7a9daf5d42a84615a6383cfec9a10608 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260129 02:59:55.355388 22880 raft_consensus.cc:493] T 75f45538dc704d9a8ec4ff8ad9f3ea5b P 7a9daf5d42a84615a6383cfec9a10608 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260129 02:59:55.355703 22880 raft_consensus.cc:3060] T 75f45538dc704d9a8ec4ff8ad9f3ea5b P 7a9daf5d42a84615a6383cfec9a10608 [term 3 FOLLOWER]: Advancing to term 4
I20260129 02:59:55.355916 22900 leader_election.cc:290] T 1299723a4476420ab02fd8bfd7aa0645 P 1c50de89a5a845668696dcb28c3badd6 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 197b880fa39d432a9b472845876bfaf0 (127.18.211.1:42027), 7a9daf5d42a84615a6383cfec9a10608 (127.18.211.4:40439)
I20260129 02:59:55.356305 22184 leader_election.cc:290] T a4c1226b987b468fa84dfc6ba0e2dbe1 P 974b802141bd4f13beffef2aa03f88f9 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 197b880fa39d432a9b472845876bfaf0 (127.18.211.1:42027), 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3:37735)
I20260129 02:59:55.357115 20515 raft_consensus.cc:2410] T a4c1226b987b468fa84dfc6ba0e2dbe1 P 974b802141bd4f13beffef2aa03f88f9 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1c50de89a5a845668696dcb28c3badd6 for term 5 because replica has last-logged OpId of term: 4 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 2.
I20260129 02:59:55.342773 20581 raft_consensus.cc:1240] T 926a250fa5834e88b96ccecbefbf5086 P 1c50de89a5a845668696dcb28c3badd6 [term 6 FOLLOWER]: Rejecting Update request from peer 197b880fa39d432a9b472845876bfaf0 for earlier term 5. Current term is 6. Ops: [5.4-5.4]
W20260129 02:59:55.335093 20405 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.211.3:37735, user_credentials={real_user=slave}} blocked reactor thread for 83382.1us
I20260129 02:59:55.344456 20432 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55672 (request call id 1568) took 1547 ms. Trace:
I20260129 02:59:55.343487 20586 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7d9494f5584d4d5db8b9adf1c8fc888b" candidate_uuid: "197b880fa39d432a9b472845876bfaf0" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "1c50de89a5a845668696dcb28c3badd6"
I20260129 02:59:55.346212 20583 raft_consensus.cc:1240] T 76f713c3331540d2a5b76164103e9eb1 P 1c50de89a5a845668696dcb28c3badd6 [term 5 FOLLOWER]: Rejecting Update request from peer 197b880fa39d432a9b472845876bfaf0 for earlier term 3. Current term is 5. Ops: []
I20260129 02:59:55.344947 20582 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e7d83d4ccf684e76acb29a0dc844e1b8" candidate_uuid: "197b880fa39d432a9b472845876bfaf0" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "1c50de89a5a845668696dcb28c3badd6"
I20260129 02:59:55.359608 20582 raft_consensus.cc:2410] T e7d83d4ccf684e76acb29a0dc844e1b8 P 1c50de89a5a845668696dcb28c3badd6 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 197b880fa39d432a9b472845876bfaf0 for term 5 because replica has last-logged OpId of term: 4 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 2.
I20260129 02:59:55.342458 20554 leader_election.cc:304] T 4e3805ffa70242e8ade66da260dddf66 P 1c50de89a5a845668696dcb28c3badd6 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1c50de89a5a845668696dcb28c3badd6, 7a9daf5d42a84615a6383cfec9a10608; no voters:
W20260129 02:59:55.359995 20482 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260129 02:59:55.334762 22926 log.cc:927] Time spent T 3e2d15f8640b407195210c476e09df91 P 1c50de89a5a845668696dcb28c3badd6: Append to log took a long time: real 0.050s user 0.000s sys 0.000s
I20260129 02:59:55.349574 22925 consensus_queue.cc:1048] T 7a9a69cd82724b1faed1d10aa091fc4a P 197b880fa39d432a9b472845876bfaf0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 2, Time since last communication: 0.001s
I20260129 02:59:55.350574 22875 consensus_queue.cc:1048] T c934d31efd3f4e5195605ddc354d14e1 P 197b880fa39d432a9b472845876bfaf0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 5, Time since last communication: 0.000s
I20260129 02:59:55.354063 21100 leader_election.cc:304] T 4e3805ffa70242e8ade66da260dddf66 P 7a9daf5d42a84615a6383cfec9a10608 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1c50de89a5a845668696dcb28c3badd6, 7a9daf5d42a84615a6383cfec9a10608; no voters:
I20260129 02:59:55.357052 22748 leader_election.cc:290] T a8479d1fe0bd41e5946186e3a6ca8c7a P 1c50de89a5a845668696dcb28c3badd6 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 197b880fa39d432a9b472845876bfaf0 (127.18.211.1:42027), 974b802141bd4f13beffef2aa03f88f9 (127.18.211.2:34517)
I20260129 02:59:55.344372 20590 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4e3805ffa70242e8ade66da260dddf66" candidate_uuid: "197b880fa39d432a9b472845876bfaf0" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "1c50de89a5a845668696dcb28c3badd6"
I20260129 02:59:55.346357 22656 consensus_queue.cc:1059] T add8d4923d1a489e86882c3a25692b95 P 1c50de89a5a845668696dcb28c3badd6 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false }, Status: INVALID_TERM, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260129 02:59:55.346961 20440 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55672 (request call id 1567) took 2343 ms. Trace:
I20260129 02:59:55.353606 20584 raft_consensus.cc:1240] T a8479d1fe0bd41e5946186e3a6ca8c7a P 1c50de89a5a845668696dcb28c3badd6 [term 4 FOLLOWER]: Rejecting Update request from peer 197b880fa39d432a9b472845876bfaf0 for earlier term 3. Current term is 4. Ops: [3.3-3.3]
I20260129 02:59:55.393798 21135 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a644c02b16af47679824ccd060ac10fd" candidate_uuid: "1c50de89a5a845668696dcb28c3badd6" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "7a9daf5d42a84615a6383cfec9a10608" is_pre_election: true
W20260129 02:59:55.370487 20407 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260129 02:59:55.384457 20512 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00e79d8f25a34676b1d866eb2068a2b2" candidate_uuid: "197b880fa39d432a9b472845876bfaf0" candidate_term: 7 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "974b802141bd4f13beffef2aa03f88f9" is_pre_election: true
I20260129 02:59:55.293174 22640 raft_consensus.cc:515] T a5a05fbcbbcf4916bbc01fd1a5b5e90d P 1c50de89a5a845668696dcb28c3badd6 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:59:55.351768 20588 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fcddab6bca684adf8ea032289a00f90e" candidate_uuid: "197b880fa39d432a9b472845876bfaf0" candidate_term: 9 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "1c50de89a5a845668696dcb28c3badd6" is_pre_election: true
I20260129 02:59:55.397993 20588 raft_consensus.cc:2468] T fcddab6bca684adf8ea032289a00f90e P 1c50de89a5a845668696dcb28c3badd6 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 197b880fa39d432a9b472845876bfaf0 in term 8.
I20260129 02:59:55.398586 20511 raft_consensus.cc:1275] T cedce331e1584ac8b636c9b8a72b2ba0 P 974b802141bd4f13beffef2aa03f88f9 [term 6 FOLLOWER]: Refusing update from remote peer 197b880fa39d432a9b472845876bfaf0: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 6 index: 4. (index mismatch)
W20260129 02:59:55.394363 20553 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260129 02:59:55.396034 20512 raft_consensus.cc:2410] T 00e79d8f25a34676b1d866eb2068a2b2 P 974b802141bd4f13beffef2aa03f88f9 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 197b880fa39d432a9b472845876bfaf0 for term 7 because replica has last-logged OpId of term: 4 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 6.
W20260129 02:59:55.395905 20407 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260129 02:59:55.360159 22691 raft_consensus.cc:3055] T 0f20af7fbe5d4db89a6f17948d98912c P 197b880fa39d432a9b472845876bfaf0 [term 5 LEADER]: Stepping down as leader of term 5
I20260129 02:59:55.402946 20510 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a8479d1fe0bd41e5946186e3a6ca8c7a" candidate_uuid: "197b880fa39d432a9b472845876bfaf0" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "974b802141bd4f13beffef2aa03f88f9" is_pre_election: true
I20260129 02:59:55.321812 22719 raft_consensus.cc:493] T 7d9494f5584d4d5db8b9adf1c8fc888b P 974b802141bd4f13beffef2aa03f88f9 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260129 02:59:55.405504 20510 raft_consensus.cc:2393] T a8479d1fe0bd41e5946186e3a6ca8c7a P 974b802141bd4f13beffef2aa03f88f9 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 197b880fa39d432a9b472845876bfaf0 in current term 5: Already voted for candidate 974b802141bd4f13beffef2aa03f88f9 in this term.
I20260129 02:59:55.407189 20514 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7d9494f5584d4d5db8b9adf1c8fc888b" candidate_uuid: "1c50de89a5a845668696dcb28c3badd6" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "974b802141bd4f13beffef2aa03f88f9" is_pre_election: true
I20260129 02:59:55.405678 20437 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a8479d1fe0bd41e5946186e3a6ca8c7a" candidate_uuid: "974b802141bd4f13beffef2aa03f88f9" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "197b880fa39d432a9b472845876bfaf0"
I20260129 02:59:55.408094 20437 raft_consensus.cc:3060] T a8479d1fe0bd41e5946186e3a6ca8c7a P 197b880fa39d432a9b472845876bfaf0 [term 4 FOLLOWER]: Advancing to term 5
I20260129 02:59:55.358042 20432 rpcz_store.cc:276] 0129 02:59:53.797217 (+ 0us) service_pool.cc:168] Inserting onto call queue
0129 02:59:53.816130 (+ 18913us) service_pool.cc:225] Handling call
0129 02:59:53.816715 (+ 585us) raft_consensus.cc:1471] Updating replica for 1 ops
0129 02:59:54.652323 (+835608us) raft_consensus.cc:1534] Early marking committed up to index 0
0129 02:59:54.652373 (+ 50us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0129 02:59:54.679021 (+ 26648us) log.cc:844] Serialized 26 byte log entry
0129 02:59:55.196345 (+517324us) raft_consensus.cc:1645] Marking committed up to 4
0129 02:59:55.196409 (+ 64us) raft_consensus.cc:1697] Filling consensus response to leader.
0129 02:59:55.196519 (+ 110us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0129 02:59:55.344214 (+147695us) raft_consensus.cc:1685] finished
0129 02:59:55.344236 (+ 22us) raft_consensus.cc:1691] UpdateReplicas() finished
0129 02:59:55.344372 (+ 136us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":462475,"raft.queue_time_us":472797,"raft.run_cpu_time_us":187,"raft.run_wall_time_us":9429,"thread_start_us":476051,"threads_started":1,"wal-append.queue_time_us":567726}
W20260129 02:59:55.400242 20407 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260129 02:59:55.402395 20507 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3906b080a5b34a74b8b64128635e8997" candidate_uuid: "197b880fa39d432a9b472845876bfaf0" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "974b802141bd4f13beffef2aa03f88f9" is_pre_election: true
I20260129 02:59:55.367373 22900 raft_consensus.cc:2804] T 4e3805ffa70242e8ade66da260dddf66 P 1c50de89a5a845668696dcb28c3badd6 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260129 02:59:55.405303 20441 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4e3805ffa70242e8ade66da260dddf66" candidate_uuid: "1c50de89a5a845668696dcb28c3badd6" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "197b880fa39d432a9b472845876bfaf0" is_pre_election: true
W20260129 02:59:55.413331 20405 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.211.3:37735, user_credentials={real_user=slave}} blocked reactor thread for 55075.9us
I20260129 02:59:55.413878 20433 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0f20af7fbe5d4db89a6f17948d98912c" candidate_uuid: "1c50de89a5a845668696dcb28c3badd6" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "197b880fa39d432a9b472845876bfaf0" is_pre_election: true
I20260129 02:59:55.411048 20513 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1ce4ddf5b10e4ae8b193b13b2e5936ee" candidate_uuid: "197b880fa39d432a9b472845876bfaf0" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "974b802141bd4f13beffef2aa03f88f9" is_pre_election: true
W20260129 02:59:55.420938 20480 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.211.3:37735, user_credentials={real_user=slave}} blocked reactor thread for 56870.7us
W20260129 02:59:55.414067 20479 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.211.1:42027, user_credentials={real_user=slave}} blocked reactor thread for 52669us
I20260129 02:59:55.424932 21135 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1299723a4476420ab02fd8bfd7aa0645" candidate_uuid: "1c50de89a5a845668696dcb28c3badd6" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "7a9daf5d42a84615a6383cfec9a10608" is_pre_election: true
I20260129 02:59:55.433636 20441 raft_consensus.cc:2468] T 4e3805ffa70242e8ade66da260dddf66 P 197b880fa39d432a9b472845876bfaf0 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1c50de89a5a845668696dcb28c3badd6 in term 4.
I20260129 02:59:55.392933 20440 rpcz_store.cc:276] 0129 02:59:53.003911 (+ 0us) service_pool.cc:168] Inserting onto call queue
0129 02:59:53.798136 (+794225us) service_pool.cc:225] Handling call
0129 02:59:53.798802 (+ 666us) raft_consensus.cc:1471] Updating replica for 1 ops
0129 02:59:54.284444 (+485642us) raft_consensus.cc:1534] Early marking committed up to index 0
0129 02:59:54.284491 (+ 47us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0129 02:59:54.287159 (+ 2668us) log.cc:844] Serialized 26 byte log entry
0129 02:59:54.754048 (+466889us) raft_consensus.cc:1645] Marking committed up to 1
0129 02:59:54.754076 (+ 28us) raft_consensus.cc:1697] Filling consensus response to leader.
0129 02:59:54.754124 (+ 48us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0129 02:59:55.180986 (+426862us) spinlock_profiling.cc:243] Waited 4.96 ms on lock 0x7b440002304c. stack: 00007f8fe044e373 00007f8fe044e270 00007f8fdfda0574 00007f8fe9a5c6b4 00007f8fe9a5c649 00007f8fe02c08b1 00007f8fe038f005 00007f8fe038fa22 00007f8fe30d957b 00007f8fe30d4e4f 00007f8fe30c8c0e 00007f8fe30c7f56 00007f8fe30c7b78 00007f8fe30d15c1 00007f8fe30d1579 00007f8fe30d1509
0129 02:59:55.346541 (+165555us) raft_consensus.cc:1685] finished
0129 02:59:55.346565 (+ 24us) raft_consensus.cc:1691] UpdateReplicas() finished
0129 02:59:55.346791 (+ 226us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":461122,"raft.queue_time_us":479060,"raft.run_cpu_time_us":131,"raft.run_wall_time_us":134,"spinlock_wait_cycles":11407744,"thread_start_us":466419,"threads_started":1,"wal-append.queue_time_us":728285}
I20260129 02:59:55.429267 20508 raft_consensus.cc:1275] T a1a39dcd328e4268ae41b988c66b4a75 P 974b802141bd4f13beffef2aa03f88f9 [term 4 FOLLOWER]: Refusing update from remote peer 1c50de89a5a845668696dcb28c3badd6: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
W20260129 02:59:55.428800 21100 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.211.3:37735, user_credentials={real_user=slave}} blocked reactor thread for 74788.1us
I20260129 02:59:55.397341 22891 raft_consensus.cc:493] T 18f30ebaa4174fd28c958e72d0cc7e24 P 1c50de89a5a845668696dcb28c3badd6 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260129 02:59:55.437196 20509 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57584 (request call id 1339) took 2553 ms. Trace:
W20260129 02:59:55.437899 21102 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.211.2:34517, user_credentials={real_user=slave}} blocked reactor thread for 121490us
I20260129 02:59:55.402287 22691 raft_consensus.cc:740] T 0f20af7fbe5d4db89a6f17948d98912c P 197b880fa39d432a9b472845876bfaf0 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 197b880fa39d432a9b472845876bfaf0, State: Running, Role: LEADER
I20260129 02:59:55.438776 20511 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0f20af7fbe5d4db89a6f17948d98912c" candidate_uuid: "1c50de89a5a845668696dcb28c3badd6" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "974b802141bd4f13beffef2aa03f88f9" is_pre_election: true
I20260129 02:59:55.301621 22884 consensus_queue.cc:260] T 45cc5f3d2688486d9b6dbe7e666dedde P 1c50de89a5a845668696dcb28c3badd6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.5, Last appended by leader: 5, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } }
I20260129 02:59:55.409505 22856 raft_consensus.cc:493] T a0fcc58d6fb6473cafbd9d34d5195269 P 1c50de89a5a845668696dcb28c3badd6 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 974b802141bd4f13beffef2aa03f88f9)
I20260129 02:59:55.442821 20509 rpcz_store.cc:276] 0129 02:59:52.883449 (+ 0us) service_pool.cc:168] Inserting onto call queue
0129 02:59:53.179556 (+296107us) service_pool.cc:225] Handling call
0129 02:59:53.324030 (+144474us) raft_consensus.cc:1471] Updating replica for 1 ops
0129 02:59:55.352110 (+2028080us) raft_consensus.cc:1697] Filling consensus response to leader.
0129 02:59:55.410613 (+ 58503us) spinlock_profiling.cc:243] Waited 1.87 s on lock 0x7b54003ff054. stack: 00007f8fe044e373 00007f8fe044e270 00007f8fdfda0574 00007f8fe9a5c6b4 00007f8fe9a5c649 00007f8fe34975a1 00007f8fe352ea81 00007f8fe352d4be 00007f8fe84e1f49 00007f8fe1a6ed34 00007f8fe1a6ecc2 00007f8fe1a6ebf1 00007f8fe1a6eb6c 00007f8fe1a6de22 00007f8fe186d34c 00007f8fe186c786
0129 02:59:55.437117 (+ 26504us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":4294141696}
I20260129 02:59:55.451354 20513 raft_consensus.cc:2468] T 1ce4ddf5b10e4ae8b193b13b2e5936ee P 974b802141bd4f13beffef2aa03f88f9 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 197b880fa39d432a9b472845876bfaf0 in term 6.
I20260129 02:59:55.438947 20479 leader_election.cc:304] T 4f8f3a38f6e24ba4857875181f447442 P 974b802141bd4f13beffef2aa03f88f9 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 197b880fa39d432a9b472845876bfaf0, 974b802141bd4f13beffef2aa03f88f9; no voters:
I20260129 02:59:55.444512 20432 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e7d83d4ccf684e76acb29a0dc844e1b8" candidate_uuid: "1c50de89a5a845668696dcb28c3badd6" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "197b880fa39d432a9b472845876bfaf0" is_pre_election: true
I20260129 02:59:55.453294 20588 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "add8d4923d1a489e86882c3a25692b95" candidate_uuid: "7a9daf5d42a84615a6383cfec9a10608" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "1c50de89a5a845668696dcb28c3badd6"
I20260129 02:59:55.454514 20433 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "febffe4a8b394368b69b1d164308fb9a" candidate_uuid: "7a9daf5d42a84615a6383cfec9a10608" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "197b880fa39d432a9b472845876bfaf0"
I20260129 02:59:55.421733 22748 raft_consensus.cc:3055] T add8d4923d1a489e86882c3a25692b95 P 1c50de89a5a845668696dcb28c3badd6 [term 3 LEADER]: Stepping down as leader of term 3
I20260129 02:59:55.456065 20511 raft_consensus.cc:2393] T 0f20af7fbe5d4db89a6f17948d98912c P 974b802141bd4f13beffef2aa03f88f9 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1c50de89a5a845668696dcb28c3badd6 in current term 6: Already voted for candidate 974b802141bd4f13beffef2aa03f88f9 in this term.
I20260129 02:59:55.458207 20440 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "913fc7d751ed4b7388d889aac30a5f9b" candidate_uuid: "974b802141bd4f13beffef2aa03f88f9" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "197b880fa39d432a9b472845876bfaf0" is_pre_election: true
I20260129 02:59:55.457274 20584 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "42d27a28d89b409e8807ec0efd97af69" candidate_uuid: "7a9daf5d42a84615a6383cfec9a10608" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "1c50de89a5a845668696dcb28c3badd6" is_pre_election: true
I20260129 02:59:55.457598 20512 raft_consensus.cc:1275] T e7d83d4ccf684e76acb29a0dc844e1b8 P 974b802141bd4f13beffef2aa03f88f9 [term 5 FOLLOWER]: Refusing update from remote peer 197b880fa39d432a9b472845876bfaf0: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 5 index: 3. (index mismatch)
I20260129 02:59:55.461826 20435 raft_consensus.cc:1275] T a1a39dcd328e4268ae41b988c66b4a75 P 197b880fa39d432a9b472845876bfaf0 [term 4 FOLLOWER]: Refusing update from remote peer 1c50de89a5a845668696dcb28c3badd6: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20260129 02:59:55.462168 22922 consensus_queue.cc:1048] T 42d27a28d89b409e8807ec0efd97af69 P 1c50de89a5a845668696dcb28c3badd6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.129s
I20260129 02:59:55.462908 20435 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "78aa39096868415d9afc2ffcb248e198" candidate_uuid: "7a9daf5d42a84615a6383cfec9a10608" candidate_term: 6 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "197b880fa39d432a9b472845876bfaf0" is_pre_election: true
I20260129 02:59:55.463136 22932 raft_consensus.cc:493] T 87d5fe48a44a408aaf03e894a67ecb2f P 7a9daf5d42a84615a6383cfec9a10608 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260129 02:59:55.464206 20553 leader_election.cc:304] T 4408006eb5174de1896b037cf3f18457 P 1c50de89a5a845668696dcb28c3badd6 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1c50de89a5a845668696dcb28c3badd6; no voters: 197b880fa39d432a9b472845876bfaf0, 974b802141bd4f13beffef2aa03f88f9
I20260129 02:59:55.463902 22932 raft_consensus.cc:515] T 87d5fe48a44a408aaf03e894a67ecb2f P 7a9daf5d42a84615a6383cfec9a10608 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } }
W20260129 02:59:55.465147 22928 log.cc:927] Time spent T 75be4bd6bd45474795fd472eff2953ac P 1c50de89a5a845668696dcb28c3badd6: Append to log took a long time: real 0.148s user 0.000s sys 0.000s
I20260129 02:59:55.466400 22719 raft_consensus.cc:515] T 7d9494f5584d4d5db8b9adf1c8fc888b P 974b802141bd4f13beffef2aa03f88f9 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:59:55.467984 20480 leader_election.cc:304] T 7a9a69cd82724b1faed1d10aa091fc4a P 974b802141bd4f13beffef2aa03f88f9 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 974b802141bd4f13beffef2aa03f88f9; no voters: 197b880fa39d432a9b472845876bfaf0, 1c50de89a5a845668696dcb28c3badd6
I20260129 02:59:55.461691 20583 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "dc2f4cb752cf4d59b5f6bc689f5d8104" candidate_uuid: "7a9daf5d42a84615a6383cfec9a10608" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "1c50de89a5a845668696dcb28c3badd6" is_pre_election: true
I20260129 02:59:55.440268 22857 raft_consensus.cc:2804] T 4e3805ffa70242e8ade66da260dddf66 P 7a9daf5d42a84615a6383cfec9a10608 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260129 02:59:55.470082 20514 raft_consensus.cc:2468] T 7d9494f5584d4d5db8b9adf1c8fc888b P 974b802141bd4f13beffef2aa03f88f9 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1c50de89a5a845668696dcb28c3badd6 in term 3.
I20260129 02:59:55.471323 20441 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a609ac02fae84725ab743f95f51d33ce" candidate_uuid: "7a9daf5d42a84615a6383cfec9a10608" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "197b880fa39d432a9b472845876bfaf0" is_pre_election: true
I20260129 02:59:55.474813 20441 raft_consensus.cc:2468] T a609ac02fae84725ab743f95f51d33ce P 197b880fa39d432a9b472845876bfaf0 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7a9daf5d42a84615a6383cfec9a10608 in term 6.
W20260129 02:59:55.474399 20479 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.211.1:42027, user_credentials={real_user=slave}} blocked reactor thread for 49930.8us
I20260129 02:59:55.479812 20405 leader_election.cc:304] T fd8311b779af4e1fa1402b2a02a602f2 P 197b880fa39d432a9b472845876bfaf0 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 197b880fa39d432a9b472845876bfaf0; no voters: 1c50de89a5a845668696dcb28c3badd6, 7a9daf5d42a84615a6383cfec9a10608
I20260129 02:59:55.482586 21127 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3e2d15f8640b407195210c476e09df91" candidate_uuid: "1c50de89a5a845668696dcb28c3badd6" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "7a9daf5d42a84615a6383cfec9a10608"
I20260129 02:59:55.483309 21127 raft_consensus.cc:2376] T 3e2d15f8640b407195210c476e09df91 P 7a9daf5d42a84615a6383cfec9a10608 [term 3 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 1c50de89a5a845668696dcb28c3badd6 in term 3. Re-sending same reply.
I20260129 02:59:55.485626 20433 raft_consensus.cc:1275] T 42d27a28d89b409e8807ec0efd97af69 P 197b880fa39d432a9b472845876bfaf0 [term 6 FOLLOWER]: Refusing update from remote peer 1c50de89a5a845668696dcb28c3badd6: Log matching property violated. Preceding OpId in replica: term: 4 index: 6. Preceding OpId from leader: term: 6 index: 8. (index mismatch)
I20260129 02:59:55.487399 21132 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "cb0b3f259ebc4310927338a8740e87b8" candidate_uuid: "1c50de89a5a845668696dcb28c3badd6" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "7a9daf5d42a84615a6383cfec9a10608" is_pre_election: true
I20260129 02:59:55.488940 22903 leader_election.cc:290] T 4f8f3a38f6e24ba4857875181f447442 P 1c50de89a5a845668696dcb28c3badd6 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 197b880fa39d432a9b472845876bfaf0 (127.18.211.1:42027), 974b802141bd4f13beffef2aa03f88f9 (127.18.211.2:34517)
I20260129 02:59:55.489540 22903 consensus_queue.cc:1048] T 1ce4ddf5b10e4ae8b193b13b2e5936ee P 1c50de89a5a845668696dcb28c3badd6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 1, Time since last communication: 0.000s
I20260129 02:59:55.476647 22857 raft_consensus.cc:493] T 4e3805ffa70242e8ade66da260dddf66 P 7a9daf5d42a84615a6383cfec9a10608 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260129 02:59:55.492633 22857 raft_consensus.cc:3060] T 4e3805ffa70242e8ade66da260dddf66 P 7a9daf5d42a84615a6383cfec9a10608 [term 4 FOLLOWER]: Advancing to term 5
I20260129 02:59:55.497689 20586 raft_consensus.cc:2468] T 7d9494f5584d4d5db8b9adf1c8fc888b P 1c50de89a5a845668696dcb28c3badd6 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 197b880fa39d432a9b472845876bfaf0 in term 3.
I20260129 02:59:55.498754 20590 raft_consensus.cc:2468] T 4e3805ffa70242e8ade66da260dddf66 P 1c50de89a5a845668696dcb28c3badd6 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 197b880fa39d432a9b472845876bfaf0 in term 4.
I20260129 02:59:55.458011 22831 leader_election.cc:290] T 2ac659d732314e1288849f08fe5e074e P 7a9daf5d42a84615a6383cfec9a10608 [CANDIDATE]: Term 4 election: Requested vote from peers 197b880fa39d432a9b472845876bfaf0 (127.18.211.1:42027), 974b802141bd4f13beffef2aa03f88f9 (127.18.211.2:34517)
I20260129 02:59:55.457916 22748 raft_consensus.cc:740] T add8d4923d1a489e86882c3a25692b95 P 1c50de89a5a845668696dcb28c3badd6 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 1c50de89a5a845668696dcb28c3badd6, State: Running, Role: LEADER
I20260129 02:59:55.511572 22691 consensus_queue.cc:260] T 0f20af7fbe5d4db89a6f17948d98912c P 197b880fa39d432a9b472845876bfaf0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.4, Last appended by leader: 4, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:59:55.502183 22922 raft_consensus.cc:2749] T 4408006eb5174de1896b037cf3f18457 P 1c50de89a5a845668696dcb28c3badd6 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260129 02:59:55.512665 22878 raft_consensus.cc:2804] T 4f8f3a38f6e24ba4857875181f447442 P 974b802141bd4f13beffef2aa03f88f9 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260129 02:59:55.512687 20516 raft_consensus.cc:1240] T 45cc5f3d2688486d9b6dbe7e666dedde P 974b802141bd4f13beffef2aa03f88f9 [term 4 FOLLOWER]: Rejecting Update request from peer 1c50de89a5a845668696dcb28c3badd6 for earlier term 3. Current term is 4. Ops: []
I20260129 02:59:55.519008 20441 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c934d31efd3f4e5195605ddc354d14e1" candidate_uuid: "7a9daf5d42a84615a6383cfec9a10608" candidate_term: 6 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "197b880fa39d432a9b472845876bfaf0" is_pre_election: true
I20260129 02:59:55.447538 22891 raft_consensus.cc:515] T 18f30ebaa4174fd28c958e72d0cc7e24 P 1c50de89a5a845668696dcb28c3badd6 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } }
W20260129 02:59:55.529837 20404 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260129 02:59:55.525815 20556 leader_election.cc:304] T 76f713c3331540d2a5b76164103e9eb1 P 1c50de89a5a845668696dcb28c3badd6 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1c50de89a5a845668696dcb28c3badd6, 974b802141bd4f13beffef2aa03f88f9; no voters:
I20260129 02:59:55.508003 22694 raft_consensus.cc:2749] T 7a9a69cd82724b1faed1d10aa091fc4a P 974b802141bd4f13beffef2aa03f88f9 [term 6 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260129 02:59:55.500455 22900 raft_consensus.cc:493] T 4e3805ffa70242e8ade66da260dddf66 P 1c50de89a5a845668696dcb28c3badd6 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260129 02:59:55.535970 22900 raft_consensus.cc:3060] T 4e3805ffa70242e8ade66da260dddf66 P 1c50de89a5a845668696dcb28c3badd6 [term 4 FOLLOWER]: Advancing to term 5
I20260129 02:59:55.537003 22640 leader_election.cc:290] T a5a05fbcbbcf4916bbc01fd1a5b5e90d P 1c50de89a5a845668696dcb28c3badd6 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 197b880fa39d432a9b472845876bfaf0 (127.18.211.1:42027), 974b802141bd4f13beffef2aa03f88f9 (127.18.211.2:34517)
W20260129 02:59:55.538597 22851 consensus_meta.cc:287] T 0678b8b468bd43cdbc3e485900844821 P 197b880fa39d432a9b472845876bfaf0: Time spent flushing consensus metadata: real 0.590s user 0.008s sys 0.004s
I20260129 02:59:55.539335 20434 raft_consensus.cc:2468] T 0678b8b468bd43cdbc3e485900844821 P 197b880fa39d432a9b472845876bfaf0 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 974b802141bd4f13beffef2aa03f88f9 in term 6.
I20260129 02:59:55.469592 22856 raft_consensus.cc:515] T a0fcc58d6fb6473cafbd9d34d5195269 P 1c50de89a5a845668696dcb28c3badd6 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:59:55.541679 22856 leader_election.cc:290] T a0fcc58d6fb6473cafbd9d34d5195269 P 1c50de89a5a845668696dcb28c3badd6 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 197b880fa39d432a9b472845876bfaf0 (127.18.211.1:42027), 974b802141bd4f13beffef2aa03f88f9 (127.18.211.2:34517)
I20260129 02:59:55.468945 22728 consensus_queue.cc:1059] T e33a417cf9c54cf5af73a4ee3aac4d3a P 1c50de89a5a845668696dcb28c3badd6 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 }, Status: INVALID_TERM, Last received: 3.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.053s
I20260129 02:59:55.548532 22691 raft_consensus.cc:3060] T 0f20af7fbe5d4db89a6f17948d98912c P 197b880fa39d432a9b472845876bfaf0 [term 5 FOLLOWER]: Advancing to term 6
I20260129 02:59:55.549278 20479 leader_election.cc:400] T 1299723a4476420ab02fd8bfd7aa0645 P 974b802141bd4f13beffef2aa03f88f9 [CANDIDATE]: Term 5 pre-election: Vote denied by peer 197b880fa39d432a9b472845876bfaf0 (127.18.211.1:42027) with higher term. Message: Invalid argument: T 1299723a4476420ab02fd8bfd7aa0645 P 197b880fa39d432a9b472845876bfaf0 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 974b802141bd4f13beffef2aa03f88f9 for earlier term 5. Current term is 8.
I20260129 02:59:55.549613 20479 leader_election.cc:403] T 1299723a4476420ab02fd8bfd7aa0645 P 974b802141bd4f13beffef2aa03f88f9 [CANDIDATE]: Term 5 pre-election: Cancelling election due to peer responding with higher term
I20260129 02:59:55.494408 22895 leader_election.cc:290] T cb0b3f259ebc4310927338a8740e87b8 P 1c50de89a5a845668696dcb28c3badd6 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 974b802141bd4f13beffef2aa03f88f9 (127.18.211.2:34517), 7a9daf5d42a84615a6383cfec9a10608 (127.18.211.4:40439)
W20260129 02:59:55.551635 20556 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.211.2:34517, user_credentials={real_user=slave}} blocked reactor thread for 48020.1us
W20260129 02:59:55.553453 20480 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.211.3:37735, user_credentials={real_user=slave}} blocked reactor thread for 114041us
W20260129 02:59:55.571939 20404 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260129 02:59:55.577190 20440 raft_consensus.cc:1275] T 1ce4ddf5b10e4ae8b193b13b2e5936ee P 197b880fa39d432a9b472845876bfaf0 [term 6 FOLLOWER]: Refusing update from remote peer 1c50de89a5a845668696dcb28c3badd6: Log matching property violated. Preceding OpId in replica: term: 5 index: 3. Preceding OpId from leader: term: 6 index: 4. (index mismatch)
I20260129 02:59:55.526397 22878 raft_consensus.cc:493] T 4f8f3a38f6e24ba4857875181f447442 P 974b802141bd4f13beffef2aa03f88f9 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260129 02:59:55.534191 22645 raft_consensus.cc:2749] T fd8311b779af4e1fa1402b2a02a602f2 P 197b880fa39d432a9b472845876bfaf0 [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
W20260129 02:59:55.590373 20554 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260129 02:59:55.590845 20405 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.211.3:37735, user_credentials={real_user=slave}} blocked reactor thread for 145830us
I20260129 02:59:55.550916 22719 leader_election.cc:290] T 7d9494f5584d4d5db8b9adf1c8fc888b P 974b802141bd4f13beffef2aa03f88f9 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 197b880fa39d432a9b472845876bfaf0 (127.18.211.1:42027), 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3:37735)
I20260129 02:59:55.576964 22640 raft_consensus.cc:2804] T 76f713c3331540d2a5b76164103e9eb1 P 1c50de89a5a845668696dcb28c3badd6 [term 5 FOLLOWER]: Leader election won for term 5
I20260129 02:59:55.559566 22932 leader_election.cc:290] T 87d5fe48a44a408aaf03e894a67ecb2f P 7a9daf5d42a84615a6383cfec9a10608 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 197b880fa39d432a9b472845876bfaf0 (127.18.211.1:42027), 974b802141bd4f13beffef2aa03f88f9 (127.18.211.2:34517)
I20260129 02:59:55.589545 22880 raft_consensus.cc:515] T 75f45538dc704d9a8ec4ff8ad9f3ea5b P 7a9daf5d42a84615a6383cfec9a10608 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } }
I20260129 02:59:55.613729 20441 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "96609ac68bdf4993a12e1c23355ca013" candidate_uuid: "7a9daf5d42a84615a6383cfec9a10608" candidate_term: 6 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "197b880fa39d432a9b472845876bfaf0" is_pre_election: true
I20260129 02:59:55.619431 20441 raft_consensus.cc:2468] T 96609ac68bdf4993a12e1c23355ca013 P 197b880fa39d432a9b472845876bfaf0 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7a9daf5d42a84615a6383cfec9a10608 in term 5.
I20260129 02:59:55.619747 20480 leader_election.cc:304] T 505580c93c694bbb8ea96fdf9b17ab46 P 974b802141bd4f13beffef2aa03f88f9 [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 974b802141bd4f13beffef2aa03f88f9; no voters: 1c50de89a5a845668696dcb28c3badd6, 7a9daf5d42a84615a6383cfec9a10608
W20260129 02:59:55.616875 20556 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.211.2:34517, user_credentials={real_user=slave}} blocked reactor thread for 44463.6us
I20260129 02:59:55.591183 22888 raft_consensus.cc:3060] T 1299723a4476420ab02fd8bfd7aa0645 P 974b802141bd4f13beffef2aa03f88f9 [term 4 FOLLOWER]: Advancing to term 8
I20260129 02:59:55.583927 22878 raft_consensus.cc:3060] T 4f8f3a38f6e24ba4857875181f447442 P 974b802141bd4f13beffef2aa03f88f9 [term 5 FOLLOWER]: Advancing to term 6
I20260129 02:59:55.611428 20589 raft_consensus.cc:2468] T cedce331e1584ac8b636c9b8a72b2ba0 P 1c50de89a5a845668696dcb28c3badd6 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 197b880fa39d432a9b472845876bfaf0 in term 6.
W20260129 02:59:55.629149 20554 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260129 02:59:55.613138 22942 log.cc:927] Time spent T a5a05fbcbbcf4916bbc01fd1a5b5e90d P 197b880fa39d432a9b472845876bfaf0: Append to log took a long time: real 0.140s user 0.001s sys 0.000s
I20260129 02:59:55.629155 22640 raft_consensus.cc:697] T 76f713c3331540d2a5b76164103e9eb1 P 1c50de89a5a845668696dcb28c3badd6 [term 5 LEADER]: Becoming Leader. State: Replica: 1c50de89a5a845668696dcb28c3badd6, State: Running, Role: LEADER
I20260129 02:59:55.614602 22748 consensus_queue.cc:260] T add8d4923d1a489e86882c3a25692b95 P 1c50de89a5a845668696dcb28c3badd6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.6, Last appended by leader: 6, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } }
I20260129 02:59:55.644147 22922 raft_consensus.cc:493] T b189f08e16a74a11980cc55d99359a0a P 1c50de89a5a845668696dcb28c3badd6 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260129 02:59:55.644464 22748 raft_consensus.cc:3060] T add8d4923d1a489e86882c3a25692b95 P 1c50de89a5a845668696dcb28c3badd6 [term 3 FOLLOWER]: Advancing to term 4
I20260129 02:59:55.650132 20437 raft_consensus.cc:2410] T a8479d1fe0bd41e5946186e3a6ca8c7a P 197b880fa39d432a9b472845876bfaf0 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 974b802141bd4f13beffef2aa03f88f9 for term 5 because replica has last-logged OpId of term: 3 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 2.
I20260129 02:59:55.612643 22895 raft_consensus.cc:493] T e4d18c35b0a7493e85d0a4b92956a48f P 1c50de89a5a845668696dcb28c3badd6 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260129 02:59:55.648658 22640 consensus_queue.cc:237] T 76f713c3331540d2a5b76164103e9eb1 P 1c50de89a5a845668696dcb28c3badd6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.3, Last appended by leader: 3, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
W20260129 02:59:55.681818 20405 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.211.3:37735, user_credentials={real_user=slave}} blocked reactor thread for 69836.3us
I20260129 02:59:55.682706 20405 leader_election.cc:304] T 8055232cf0004d40a85f3bf983fe00a9 P 197b880fa39d432a9b472845876bfaf0 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 197b880fa39d432a9b472845876bfaf0, 1c50de89a5a845668696dcb28c3badd6; no voters: 974b802141bd4f13beffef2aa03f88f9
I20260129 02:59:55.690876 20439 raft_consensus.cc:2468] T 17d6431cc77a491e8464d68db4081a51 P 197b880fa39d432a9b472845876bfaf0 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 974b802141bd4f13beffef2aa03f88f9 in term 4.
W20260129 02:59:55.692168 20556 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.211.2:34517, user_credentials={real_user=slave}} blocked reactor thread for 53889.8us
I20260129 02:59:55.698040 20437 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4e3805ffa70242e8ade66da260dddf66" candidate_uuid: "7a9daf5d42a84615a6383cfec9a10608" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "197b880fa39d432a9b472845876bfaf0" is_pre_election: true
I20260129 02:59:55.700614 22645 raft_consensus.cc:2804] T 8055232cf0004d40a85f3bf983fe00a9 P 197b880fa39d432a9b472845876bfaf0 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260129 02:59:55.701205 22645 raft_consensus.cc:493] T 8055232cf0004d40a85f3bf983fe00a9 P 197b880fa39d432a9b472845876bfaf0 [term 3 FOLLOWER]: Starting leader election (detected failure of leader 974b802141bd4f13beffef2aa03f88f9)
I20260129 02:59:55.644733 22922 raft_consensus.cc:515] T b189f08e16a74a11980cc55d99359a0a P 1c50de89a5a845668696dcb28c3badd6 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:59:55.615430 22719 consensus_queue.cc:1048] T fcddab6bca684adf8ea032289a00f90e P 974b802141bd4f13beffef2aa03f88f9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.065s
I20260129 02:59:55.705122 20437 raft_consensus.cc:2468] T 4e3805ffa70242e8ade66da260dddf66 P 197b880fa39d432a9b472845876bfaf0 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7a9daf5d42a84615a6383cfec9a10608 in term 4.
I20260129 02:59:55.701653 22645 raft_consensus.cc:3060] T 8055232cf0004d40a85f3bf983fe00a9 P 197b880fa39d432a9b472845876bfaf0 [term 3 FOLLOWER]: Advancing to term 4
I20260129 02:59:55.707080 20440 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "42d27a28d89b409e8807ec0efd97af69" candidate_uuid: "7a9daf5d42a84615a6383cfec9a10608" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "197b880fa39d432a9b472845876bfaf0" is_pre_election: true
I20260129 02:59:55.717106 20436 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2ac659d732314e1288849f08fe5e074e" candidate_uuid: "7a9daf5d42a84615a6383cfec9a10608" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "197b880fa39d432a9b472845876bfaf0"
I20260129 02:59:55.717311 20433 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "87d5fe48a44a408aaf03e894a67ecb2f" candidate_uuid: "7a9daf5d42a84615a6383cfec9a10608" candidate_term: 7 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "197b880fa39d432a9b472845876bfaf0" is_pre_election: true
W20260129 02:59:55.717796 20553 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.211.1:42027, user_credentials={real_user=slave}} blocked reactor thread for 253711us
I20260129 02:59:55.719938 22953 consensus_queue.cc:1059] T 36ff33be666a4c82954731691dbcf327 P 1c50de89a5a845668696dcb28c3badd6 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 }, Status: INVALID_TERM, Last received: 2.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260129 02:59:55.721251 22922 leader_election.cc:290] T b189f08e16a74a11980cc55d99359a0a P 1c50de89a5a845668696dcb28c3badd6 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 197b880fa39d432a9b472845876bfaf0 (127.18.211.1:42027), 974b802141bd4f13beffef2aa03f88f9 (127.18.211.2:34517)
I20260129 02:59:55.721483 22953 raft_consensus.cc:3055] T 36ff33be666a4c82954731691dbcf327 P 1c50de89a5a845668696dcb28c3badd6 [term 2 LEADER]: Stepping down as leader of term 2
I20260129 02:59:55.721874 22953 raft_consensus.cc:740] T 36ff33be666a4c82954731691dbcf327 P 1c50de89a5a845668696dcb28c3badd6 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 1c50de89a5a845668696dcb28c3badd6, State: Running, Role: LEADER
W20260129 02:59:55.723191 20406 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260129 02:59:55.725510 22891 leader_election.cc:290] T 18f30ebaa4174fd28c958e72d0cc7e24 P 1c50de89a5a845668696dcb28c3badd6 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 197b880fa39d432a9b472845876bfaf0 (127.18.211.1:42027), 7a9daf5d42a84615a6383cfec9a10608 (127.18.211.4:40439)
I20260129 02:59:55.726606 21099 leader_election.cc:304] T 78aa39096868415d9afc2ffcb248e198 P 7a9daf5d42a84615a6383cfec9a10608 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7a9daf5d42a84615a6383cfec9a10608; no voters: 197b880fa39d432a9b472845876bfaf0, 1c50de89a5a845668696dcb28c3badd6
I20260129 02:59:55.727510 22933 raft_consensus.cc:2749] T 78aa39096868415d9afc2ffcb248e198 P 7a9daf5d42a84615a6383cfec9a10608 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260129 02:59:55.728253 21099 leader_election.cc:304] T a609ac02fae84725ab743f95f51d33ce P 7a9daf5d42a84615a6383cfec9a10608 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 197b880fa39d432a9b472845876bfaf0, 7a9daf5d42a84615a6383cfec9a10608; no voters:
I20260129 02:59:55.729496 22833 raft_consensus.cc:2804] T a609ac02fae84725ab743f95f51d33ce P 7a9daf5d42a84615a6383cfec9a10608 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260129 02:59:55.689708 22763 raft_consensus.cc:2749] T 505580c93c694bbb8ea96fdf9b17ab46 P 974b802141bd4f13beffef2aa03f88f9 [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20260129 02:59:55.720124 22581 consensus_queue.cc:1048] T 368953c9102643b2a21f99af36b51905 P 1c50de89a5a845668696dcb28c3badd6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 2, Time since last communication: 0.000s
I20260129 02:59:55.717460 20556 leader_election.cc:304] T e7d83d4ccf684e76acb29a0dc844e1b8 P 1c50de89a5a845668696dcb28c3badd6 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1c50de89a5a845668696dcb28c3badd6, 974b802141bd4f13beffef2aa03f88f9; no voters:
I20260129 02:59:55.722738 22953 consensus_queue.cc:260] T 36ff33be666a4c82954731691dbcf327 P 1c50de89a5a845668696dcb28c3badd6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:59:55.729501 21099 leader_election.cc:304] T 96609ac68bdf4993a12e1c23355ca013 P 7a9daf5d42a84615a6383cfec9a10608 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 197b880fa39d432a9b472845876bfaf0, 1c50de89a5a845668696dcb28c3badd6, 7a9daf5d42a84615a6383cfec9a10608; no voters:
I20260129 02:59:55.737260 20440 raft_consensus.cc:2468] T 42d27a28d89b409e8807ec0efd97af69 P 197b880fa39d432a9b472845876bfaf0 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7a9daf5d42a84615a6383cfec9a10608 in term 6.
I20260129 02:59:55.719956 20435 raft_consensus.cc:1275] T 3ccac82a01a2445fa5b0ce6c0fcf2295 P 197b880fa39d432a9b472845876bfaf0 [term 4 FOLLOWER]: Refusing update from remote peer 7a9daf5d42a84615a6383cfec9a10608: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 4 index: 5. (index mismatch)
I20260129 02:59:55.739614 20439 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4e420f90884f4796997b5ea41e49e411" candidate_uuid: "7a9daf5d42a84615a6383cfec9a10608" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "197b880fa39d432a9b472845876bfaf0" is_pre_election: true
I20260129 02:59:55.740363 20439 raft_consensus.cc:2468] T 4e420f90884f4796997b5ea41e49e411 P 197b880fa39d432a9b472845876bfaf0 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7a9daf5d42a84615a6383cfec9a10608 in term 5.
I20260129 02:59:55.729890 22833 raft_consensus.cc:493] T a609ac02fae84725ab743f95f51d33ce P 7a9daf5d42a84615a6383cfec9a10608 [term 6 FOLLOWER]: Starting leader election (detected failure of leader 974b802141bd4f13beffef2aa03f88f9)
I20260129 02:59:55.743904 21132 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "18f30ebaa4174fd28c958e72d0cc7e24" candidate_uuid: "1c50de89a5a845668696dcb28c3badd6" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "7a9daf5d42a84615a6383cfec9a10608" is_pre_election: true
W20260129 02:59:55.746047 20405 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.211.3:37735, user_credentials={real_user=slave}} blocked reactor thread for 63443.2us
I20260129 02:59:55.690616 22895 raft_consensus.cc:515] T e4d18c35b0a7493e85d0a4b92956a48f P 1c50de89a5a845668696dcb28c3badd6 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:59:55.689435 22231 raft_consensus.cc:515] T c2633a6fc34c44579de348d208b0c01e P 197b880fa39d432a9b472845876bfaf0 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:59:55.749792 20434 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "dc2f4cb752cf4d59b5f6bc689f5d8104" candidate_uuid: "7a9daf5d42a84615a6383cfec9a10608" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "197b880fa39d432a9b472845876bfaf0" is_pre_election: true
I20260129 02:59:55.750586 20556 leader_election.cc:304] T 913fc7d751ed4b7388d889aac30a5f9b P 1c50de89a5a845668696dcb28c3badd6 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1c50de89a5a845668696dcb28c3badd6, 974b802141bd4f13beffef2aa03f88f9; no voters:
I20260129 02:59:55.751127 21132 raft_consensus.cc:2468] T 18f30ebaa4174fd28c958e72d0cc7e24 P 7a9daf5d42a84615a6383cfec9a10608 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1c50de89a5a845668696dcb28c3badd6 in term 5.
W20260129 02:59:55.751909 20479 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.211.1:42027, user_credentials={real_user=slave}} blocked reactor thread for 201263us
I20260129 02:59:55.753908 20554 leader_election.cc:304] T 18f30ebaa4174fd28c958e72d0cc7e24 P 1c50de89a5a845668696dcb28c3badd6 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1c50de89a5a845668696dcb28c3badd6, 7a9daf5d42a84615a6383cfec9a10608; no voters:
I20260129 02:59:55.755084 20479 leader_election.cc:304] T 0340229d521b432580ae926f6062e3f7 P 974b802141bd4f13beffef2aa03f88f9 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 197b880fa39d432a9b472845876bfaf0, 974b802141bd4f13beffef2aa03f88f9; no voters:
W20260129 02:59:55.757115 20406 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260129 02:59:55.759724 20407 leader_election.cc:304] T 1ce4ddf5b10e4ae8b193b13b2e5936ee P 197b880fa39d432a9b472845876bfaf0 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 197b880fa39d432a9b472845876bfaf0, 974b802141bd4f13beffef2aa03f88f9; no voters:
I20260129 02:59:55.747586 22933 raft_consensus.cc:2804] T 96609ac68bdf4993a12e1c23355ca013 P 7a9daf5d42a84615a6383cfec9a10608 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260129 02:59:55.744127 22654 raft_consensus.cc:2804] T e7d83d4ccf684e76acb29a0dc844e1b8 P 1c50de89a5a845668696dcb28c3badd6 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260129 02:59:55.747845 22833 raft_consensus.cc:3060] T a609ac02fae84725ab743f95f51d33ce P 7a9daf5d42a84615a6383cfec9a10608 [term 6 FOLLOWER]: Advancing to term 7
I20260129 02:59:55.772738 20556 leader_election.cc:304] T cb4727386a3e48aca5d71cc6c611c197 P 1c50de89a5a845668696dcb28c3badd6 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1c50de89a5a845668696dcb28c3badd6, 974b802141bd4f13beffef2aa03f88f9; no voters:
I20260129 02:59:55.774902 20516 raft_consensus.cc:1240] T 7a9a69cd82724b1faed1d10aa091fc4a P 974b802141bd4f13beffef2aa03f88f9 [term 6 FOLLOWER]: Rejecting Update request from peer 197b880fa39d432a9b472845876bfaf0 for earlier term 5. Current term is 6. Ops: [4.3-5.4]
W20260129 02:59:55.777175 20480 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.211.3:37735, user_credentials={real_user=slave}} blocked reactor thread for 192838us
I20260129 02:59:55.755970 22640 raft_consensus.cc:2804] T 18f30ebaa4174fd28c958e72d0cc7e24 P 1c50de89a5a845668696dcb28c3badd6 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260129 02:59:55.756139 22688 raft_consensus.cc:493] T 724a236524cb4e7ba3428880681d6cc2 P 197b880fa39d432a9b472845876bfaf0 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260129 02:59:55.821437 20480 leader_election.cc:304] T 4408006eb5174de1896b037cf3f18457 P 974b802141bd4f13beffef2aa03f88f9 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 974b802141bd4f13beffef2aa03f88f9; no voters: 197b880fa39d432a9b472845876bfaf0, 1c50de89a5a845668696dcb28c3badd6
I20260129 02:59:55.773376 22933 raft_consensus.cc:493] T 96609ac68bdf4993a12e1c23355ca013 P 7a9daf5d42a84615a6383cfec9a10608 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260129 02:59:55.777563 22851 raft_consensus.cc:2804] T 1ce4ddf5b10e4ae8b193b13b2e5936ee P 197b880fa39d432a9b472845876bfaf0 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260129 02:59:55.771678 22654 raft_consensus.cc:493] T e7d83d4ccf684e76acb29a0dc844e1b8 P 1c50de89a5a845668696dcb28c3badd6 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260129 02:59:55.777034 22856 raft_consensus.cc:2804] T 913fc7d751ed4b7388d889aac30a5f9b P 1c50de89a5a845668696dcb28c3badd6 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260129 02:59:55.788210 22953 raft_consensus.cc:3060] T 36ff33be666a4c82954731691dbcf327 P 1c50de89a5a845668696dcb28c3badd6 [term 2 FOLLOWER]: Advancing to term 3
W20260129 02:59:55.839308 20553 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.211.1:42027, user_credentials={real_user=slave}} blocked reactor thread for 48359.2us
I20260129 02:59:55.764380 22691 consensus_queue.cc:1048] T cedce331e1584ac8b636c9b8a72b2ba0 P 197b880fa39d432a9b472845876bfaf0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.064s
I20260129 02:59:55.791832 22763 raft_consensus.cc:2804] T 0340229d521b432580ae926f6062e3f7 P 974b802141bd4f13beffef2aa03f88f9 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260129 02:59:55.820259 22640 raft_consensus.cc:493] T 18f30ebaa4174fd28c958e72d0cc7e24 P 1c50de89a5a845668696dcb28c3badd6 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260129 02:59:55.778548 22857 raft_consensus.cc:515] T 4e3805ffa70242e8ade66da260dddf66 P 7a9daf5d42a84615a6383cfec9a10608 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } }
I20260129 02:59:55.776744 22875 consensus_queue.cc:1048] T e7d83d4ccf684e76acb29a0dc844e1b8 P 197b880fa39d432a9b472845876bfaf0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.061s
I20260129 02:59:55.833102 22933 raft_consensus.cc:3060] T 96609ac68bdf4993a12e1c23355ca013 P 7a9daf5d42a84615a6383cfec9a10608 [term 5 FOLLOWER]: Advancing to term 6
W20260129 02:59:55.873114 20405 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.211.3:37735, user_credentials={real_user=slave}} blocked reactor thread for 117859us
I20260129 02:59:55.832706 22851 raft_consensus.cc:493] T 1ce4ddf5b10e4ae8b193b13b2e5936ee P 197b880fa39d432a9b472845876bfaf0 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260129 02:59:55.836968 22949 log.cc:927] Time spent T 78aa39096868415d9afc2ffcb248e198 P 7a9daf5d42a84615a6383cfec9a10608: Append to log took a long time: real 0.056s user 0.000s sys 0.000s
I20260129 02:59:55.837625 22654 raft_consensus.cc:3060] T e7d83d4ccf684e76acb29a0dc844e1b8 P 1c50de89a5a845668696dcb28c3badd6 [term 5 FOLLOWER]: Advancing to term 6
I20260129 02:59:55.859445 20553 leader_election.cc:304] T 8055232cf0004d40a85f3bf983fe00a9 P 1c50de89a5a845668696dcb28c3badd6 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1c50de89a5a845668696dcb28c3badd6; no voters: 197b880fa39d432a9b472845876bfaf0, 974b802141bd4f13beffef2aa03f88f9
W20260129 02:59:55.885170 20482 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260129 02:59:55.841725 22922 raft_consensus.cc:2804] T cb4727386a3e48aca5d71cc6c611c197 P 1c50de89a5a845668696dcb28c3badd6 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260129 02:59:55.837668 22856 raft_consensus.cc:493] T 913fc7d751ed4b7388d889aac30a5f9b P 1c50de89a5a845668696dcb28c3badd6 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260129 02:59:55.906559 20553 leader_election.cc:304] T 96609ac68bdf4993a12e1c23355ca013 P 1c50de89a5a845668696dcb28c3badd6 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1c50de89a5a845668696dcb28c3badd6; no voters: 197b880fa39d432a9b472845876bfaf0, 974b802141bd4f13beffef2aa03f88f9
I20260129 02:59:55.872817 22737 raft_consensus.cc:493] T 4408006eb5174de1896b037cf3f18457 P 197b880fa39d432a9b472845876bfaf0 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260129 02:59:55.882401 22763 raft_consensus.cc:493] T 0340229d521b432580ae926f6062e3f7 P 974b802141bd4f13beffef2aa03f88f9 [term 5 FOLLOWER]: Starting leader election (detected failure of leader 1c50de89a5a845668696dcb28c3badd6)
I20260129 02:59:55.886946 22640 raft_consensus.cc:3060] T 18f30ebaa4174fd28c958e72d0cc7e24 P 1c50de89a5a845668696dcb28c3badd6 [term 5 FOLLOWER]: Advancing to term 6
I20260129 02:59:55.911388 20553 leader_election.cc:304] T c891d2bff4ce4568861bbc992e91fbb0 P 1c50de89a5a845668696dcb28c3badd6 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1c50de89a5a845668696dcb28c3badd6; no voters: 197b880fa39d432a9b472845876bfaf0, 974b802141bd4f13beffef2aa03f88f9
I20260129 02:59:55.893443 22851 raft_consensus.cc:3060] T 1ce4ddf5b10e4ae8b193b13b2e5936ee P 197b880fa39d432a9b472845876bfaf0 [term 6 FOLLOWER]: Advancing to term 7
I20260129 02:59:55.909818 22748 raft_consensus.cc:2749] T 8055232cf0004d40a85f3bf983fe00a9 P 1c50de89a5a845668696dcb28c3badd6 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260129 02:59:55.929591 22878 raft_consensus.cc:515] T 4f8f3a38f6e24ba4857875181f447442 P 974b802141bd4f13beffef2aa03f88f9 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:59:55.866204 22688 raft_consensus.cc:515] T 724a236524cb4e7ba3428880681d6cc2 P 197b880fa39d432a9b472845876bfaf0 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:59:55.914484 22763 raft_consensus.cc:3060] T 0340229d521b432580ae926f6062e3f7 P 974b802141bd4f13beffef2aa03f88f9 [term 5 FOLLOWER]: Advancing to term 6
I20260129 02:59:55.914363 22924 raft_consensus.cc:2749] T 96609ac68bdf4993a12e1c23355ca013 P 1c50de89a5a845668696dcb28c3badd6 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260129 02:59:55.895854 22645 raft_consensus.cc:515] T 8055232cf0004d40a85f3bf983fe00a9 P 197b880fa39d432a9b472845876bfaf0 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:59:55.918128 21131 raft_consensus.cc:1275] T 3e2d15f8640b407195210c476e09df91 P 7a9daf5d42a84615a6383cfec9a10608 [term 3 FOLLOWER]: Refusing update from remote peer 1c50de89a5a845668696dcb28c3badd6: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 3 index: 5. (index mismatch)
I20260129 02:59:55.918432 20581 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "75f45538dc704d9a8ec4ff8ad9f3ea5b" candidate_uuid: "7a9daf5d42a84615a6383cfec9a10608" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "1c50de89a5a845668696dcb28c3badd6"
I20260129 02:59:55.930394 21135 raft_consensus.cc:1275] T 505580c93c694bbb8ea96fdf9b17ab46 P 7a9daf5d42a84615a6383cfec9a10608 [term 5 FOLLOWER]: Refusing update from remote peer 1c50de89a5a845668696dcb28c3badd6: Log matching property violated. Preceding OpId in replica: term: 3 index: 6. Preceding OpId from leader: term: 5 index: 7. (index mismatch)
I20260129 02:59:55.931115 22231 leader_election.cc:290] T c2633a6fc34c44579de348d208b0c01e P 197b880fa39d432a9b472845876bfaf0 [CANDIDATE]: Term 3 election: Requested vote from peers 974b802141bd4f13beffef2aa03f88f9 (127.18.211.2:34517), 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3:37735)
I20260129 02:59:55.906072 22880 leader_election.cc:290] T 75f45538dc704d9a8ec4ff8ad9f3ea5b P 7a9daf5d42a84615a6383cfec9a10608 [CANDIDATE]: Term 4 election: Requested vote from peers 974b802141bd4f13beffef2aa03f88f9 (127.18.211.2:34517), 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3:37735)
W20260129 02:59:55.905674 22957 log.cc:927] Time spent T 75b0b6e82ddc4f92b1fe1d94f9a0b73b P 7a9daf5d42a84615a6383cfec9a10608: Append to log took a long time: real 0.166s user 0.001s sys 0.000s
W20260129 02:59:55.938966 22946 log.cc:927] Time spent T 18f30ebaa4174fd28c958e72d0cc7e24 P 197b880fa39d432a9b472845876bfaf0: Append to log took a long time: real 0.267s user 0.001s sys 0.000s
W20260129 02:59:55.941246 22958 log.cc:927] Time spent T a644c02b16af47679824ccd060ac10fd P 1c50de89a5a845668696dcb28c3badd6: Append to log took a long time: real 0.118s user 0.001s sys 0.000s
I20260129 02:59:55.912168 22856 raft_consensus.cc:3060] T 913fc7d751ed4b7388d889aac30a5f9b P 1c50de89a5a845668696dcb28c3badd6 [term 3 FOLLOWER]: Advancing to term 4
I20260129 02:59:55.908119 22922 raft_consensus.cc:493] T cb4727386a3e48aca5d71cc6c611c197 P 1c50de89a5a845668696dcb28c3badd6 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260129 02:59:55.944237 20405 leader_election.cc:304] T a0fcc58d6fb6473cafbd9d34d5195269 P 197b880fa39d432a9b472845876bfaf0 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 197b880fa39d432a9b472845876bfaf0, 1c50de89a5a845668696dcb28c3badd6; no voters: 974b802141bd4f13beffef2aa03f88f9
I20260129 02:59:55.952333 20581 raft_consensus.cc:3060] T 75f45538dc704d9a8ec4ff8ad9f3ea5b P 1c50de89a5a845668696dcb28c3badd6 [term 3 FOLLOWER]: Advancing to term 4
W20260129 02:59:55.946969 20480 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.211.3:37735, user_credentials={real_user=slave}} blocked reactor thread for 158863us
I20260129 02:59:55.914599 22737 raft_consensus.cc:515] T 4408006eb5174de1896b037cf3f18457 P 197b880fa39d432a9b472845876bfaf0 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
W20260129 02:59:55.963408 20479 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.211.1:42027, user_credentials={real_user=slave}} blocked reactor thread for 198997us
I20260129 02:59:55.947110 22963 raft_consensus.cc:2749] T 4408006eb5174de1896b037cf3f18457 P 974b802141bd4f13beffef2aa03f88f9 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260129 02:59:55.952908 22581 raft_consensus.cc:2749] T c891d2bff4ce4568861bbc992e91fbb0 P 1c50de89a5a845668696dcb28c3badd6 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
W20260129 02:59:55.981681 20404 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260129 02:59:55.984094 22855 consensus_queue.cc:1048] T 505580c93c694bbb8ea96fdf9b17ab46 P 1c50de89a5a845668696dcb28c3badd6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20260129 02:59:55.963910 22895 leader_election.cc:290] T e4d18c35b0a7493e85d0a4b92956a48f P 1c50de89a5a845668696dcb28c3badd6 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 197b880fa39d432a9b472845876bfaf0 (127.18.211.1:42027), 974b802141bd4f13beffef2aa03f88f9 (127.18.211.2:34517)
I20260129 02:59:55.984825 22924 consensus_queue.cc:1048] T a1a39dcd328e4268ae41b988c66b4a75 P 1c50de89a5a845668696dcb28c3badd6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260129 02:59:55.916298 22900 raft_consensus.cc:515] T 4e3805ffa70242e8ade66da260dddf66 P 1c50de89a5a845668696dcb28c3badd6 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } }
I20260129 02:59:55.989053 20479 leader_election.cc:304] T 36ff33be666a4c82954731691dbcf327 P 974b802141bd4f13beffef2aa03f88f9 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 197b880fa39d432a9b472845876bfaf0, 974b802141bd4f13beffef2aa03f88f9; no voters:
I20260129 02:59:56.006179 20479 leader_election.cc:304] T 00e79d8f25a34676b1d866eb2068a2b2 P 974b802141bd4f13beffef2aa03f88f9 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 197b880fa39d432a9b472845876bfaf0, 974b802141bd4f13beffef2aa03f88f9; no voters:
I20260129 02:59:56.007383 20479 leader_election.cc:304] T cb4727386a3e48aca5d71cc6c611c197 P 974b802141bd4f13beffef2aa03f88f9 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 197b880fa39d432a9b472845876bfaf0, 974b802141bd4f13beffef2aa03f88f9; no voters:
I20260129 02:59:55.981967 22884 raft_consensus.cc:3060] T 45cc5f3d2688486d9b6dbe7e666dedde P 1c50de89a5a845668696dcb28c3badd6 [term 3 FOLLOWER]: Advancing to term 4
I20260129 02:59:56.008157 22967 raft_consensus.cc:2804] T 36ff33be666a4c82954731691dbcf327 P 974b802141bd4f13beffef2aa03f88f9 [term 3 FOLLOWER]: Leader election won for term 3
I20260129 02:59:56.008714 22967 raft_consensus.cc:697] T 36ff33be666a4c82954731691dbcf327 P 974b802141bd4f13beffef2aa03f88f9 [term 3 LEADER]: Becoming Leader. State: Replica: 974b802141bd4f13beffef2aa03f88f9, State: Running, Role: LEADER
I20260129 02:59:56.009630 22967 consensus_queue.cc:237] T 36ff33be666a4c82954731691dbcf327 P 974b802141bd4f13beffef2aa03f88f9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:59:55.965350 22645 leader_election.cc:290] T 8055232cf0004d40a85f3bf983fe00a9 P 197b880fa39d432a9b472845876bfaf0 [CANDIDATE]: Term 4 election: Requested vote from peers 974b802141bd4f13beffef2aa03f88f9 (127.18.211.2:34517), 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3:37735)
I20260129 02:59:55.987974 22922 raft_consensus.cc:3060] T cb4727386a3e48aca5d71cc6c611c197 P 1c50de89a5a845668696dcb28c3badd6 [term 3 FOLLOWER]: Advancing to term 4
I20260129 02:59:56.028563 20581 raft_consensus.cc:2468] T 75f45538dc704d9a8ec4ff8ad9f3ea5b P 1c50de89a5a845668696dcb28c3badd6 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7a9daf5d42a84615a6383cfec9a10608 in term 4.
I20260129 02:59:56.030448 21100 leader_election.cc:304] T 75f45538dc704d9a8ec4ff8ad9f3ea5b P 7a9daf5d42a84615a6383cfec9a10608 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1c50de89a5a845668696dcb28c3badd6, 7a9daf5d42a84615a6383cfec9a10608; no voters:
I20260129 02:59:55.971202 22748 consensus_queue.cc:1048] T 3e2d15f8640b407195210c476e09df91 P 1c50de89a5a845668696dcb28c3badd6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.047s
I20260129 02:59:56.013705 22877 raft_consensus.cc:2804] T 368953c9102643b2a21f99af36b51905 P 197b880fa39d432a9b472845876bfaf0 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260129 02:59:56.039909 20434 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1299723a4476420ab02fd8bfd7aa0645" candidate_uuid: "1c50de89a5a845668696dcb28c3badd6" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "197b880fa39d432a9b472845876bfaf0" is_pre_election: true
I20260129 02:59:56.042346 20434 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "18f30ebaa4174fd28c958e72d0cc7e24" candidate_uuid: "1c50de89a5a845668696dcb28c3badd6" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "197b880fa39d432a9b472845876bfaf0" is_pre_election: true
I20260129 02:59:56.043401 20434 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e4d18c35b0a7493e85d0a4b92956a48f" candidate_uuid: "1c50de89a5a845668696dcb28c3badd6" candidate_term: 6 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "197b880fa39d432a9b472845876bfaf0" is_pre_election: true
I20260129 02:59:56.044198 20434 raft_consensus.cc:2410] T e4d18c35b0a7493e85d0a4b92956a48f P 197b880fa39d432a9b472845876bfaf0 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1c50de89a5a845668696dcb28c3badd6 for term 6 because replica has last-logged OpId of term: 5 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 2.
I20260129 02:59:56.044827 20434 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b189f08e16a74a11980cc55d99359a0a" candidate_uuid: "1c50de89a5a845668696dcb28c3badd6" candidate_term: 8 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "197b880fa39d432a9b472845876bfaf0" is_pre_election: true
I20260129 02:59:56.045395 20434 raft_consensus.cc:2468] T b189f08e16a74a11980cc55d99359a0a P 197b880fa39d432a9b472845876bfaf0 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1c50de89a5a845668696dcb28c3badd6 in term 7.
I20260129 02:59:56.046401 20434 raft_consensus.cc:1240] T e33a417cf9c54cf5af73a4ee3aac4d3a P 197b880fa39d432a9b472845876bfaf0 [term 6 FOLLOWER]: Rejecting Update request from peer 1c50de89a5a845668696dcb28c3badd6 for earlier term 5. Current term is 6. Ops: [5.3-5.3]
I20260129 02:59:56.042387 20480 leader_election.cc:304] T b189f08e16a74a11980cc55d99359a0a P 974b802141bd4f13beffef2aa03f88f9 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1c50de89a5a845668696dcb28c3badd6, 974b802141bd4f13beffef2aa03f88f9; no voters:
I20260129 02:59:56.051307 20440 raft_consensus.cc:1275] T 724a236524cb4e7ba3428880681d6cc2 P 197b880fa39d432a9b472845876bfaf0 [term 4 FOLLOWER]: Refusing update from remote peer 1c50de89a5a845668696dcb28c3badd6: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20260129 02:59:56.053301 20439 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4f8f3a38f6e24ba4857875181f447442" candidate_uuid: "1c50de89a5a845668696dcb28c3badd6" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "197b880fa39d432a9b472845876bfaf0" is_pre_election: true
I20260129 02:59:56.054359 20441 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3906b080a5b34a74b8b64128635e8997" candidate_uuid: "1c50de89a5a845668696dcb28c3badd6" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "197b880fa39d432a9b472845876bfaf0" is_pre_election: true
I20260129 02:59:56.057206 20435 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a8479d1fe0bd41e5946186e3a6ca8c7a" candidate_uuid: "1c50de89a5a845668696dcb28c3badd6" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "197b880fa39d432a9b472845876bfaf0" is_pre_election: true
I20260129 02:59:56.057540 20581 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4e3805ffa70242e8ade66da260dddf66" candidate_uuid: "7a9daf5d42a84615a6383cfec9a10608" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "1c50de89a5a845668696dcb28c3badd6"
I20260129 02:59:56.058229 20581 raft_consensus.cc:2393] T 4e3805ffa70242e8ade66da260dddf66 P 1c50de89a5a845668696dcb28c3badd6 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 7a9daf5d42a84615a6383cfec9a10608 in current term 5: Already voted for candidate 1c50de89a5a845668696dcb28c3badd6 in this term.
I20260129 02:59:56.058163 20433 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a0fcc58d6fb6473cafbd9d34d5195269" candidate_uuid: "1c50de89a5a845668696dcb28c3badd6" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "197b880fa39d432a9b472845876bfaf0" is_pre_election: true
I20260129 02:59:56.061264 20439 raft_consensus.cc:2468] T 4f8f3a38f6e24ba4857875181f447442 P 197b880fa39d432a9b472845876bfaf0 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1c50de89a5a845668696dcb28c3badd6 in term 4.
I20260129 02:59:56.000010 22903 consensus_queue.cc:1048] T 42d27a28d89b409e8807ec0efd97af69 P 1c50de89a5a845668696dcb28c3badd6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 6, Time since last communication: 0.037s
I20260129 02:59:56.065107 20435 raft_consensus.cc:2410] T a8479d1fe0bd41e5946186e3a6ca8c7a P 197b880fa39d432a9b472845876bfaf0 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1c50de89a5a845668696dcb28c3badd6 for term 5 because replica has last-logged OpId of term: 3 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 2.
I20260129 02:59:56.062310 20438 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fcddab6bca684adf8ea032289a00f90e" candidate_uuid: "1c50de89a5a845668696dcb28c3badd6" candidate_term: 9 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "197b880fa39d432a9b472845876bfaf0" is_pre_election: true
I20260129 02:59:56.062888 20436 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a5a05fbcbbcf4916bbc01fd1a5b5e90d" candidate_uuid: "1c50de89a5a845668696dcb28c3badd6" candidate_term: 8 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "197b880fa39d432a9b472845876bfaf0" is_pre_election: true
W20260129 02:59:56.063196 20556 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.211.2:34517, user_credentials={real_user=slave}} blocked reactor thread for 290518us
I20260129 02:59:56.022073 22963 raft_consensus.cc:2804] T cb4727386a3e48aca5d71cc6c611c197 P 974b802141bd4f13beffef2aa03f88f9 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260129 02:59:56.038058 22932 raft_consensus.cc:2804] T 75f45538dc704d9a8ec4ff8ad9f3ea5b P 7a9daf5d42a84615a6383cfec9a10608 [term 4 FOLLOWER]: Leader election won for term 4
I20260129 02:59:56.067598 20438 raft_consensus.cc:2468] T fcddab6bca684adf8ea032289a00f90e P 197b880fa39d432a9b472845876bfaf0 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1c50de89a5a845668696dcb28c3badd6 in term 8.
I20260129 02:59:56.068555 20556 leader_election.cc:304] T 7a9a69cd82724b1faed1d10aa091fc4a P 1c50de89a5a845668696dcb28c3badd6 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1c50de89a5a845668696dcb28c3badd6, 974b802141bd4f13beffef2aa03f88f9; no voters: 197b880fa39d432a9b472845876bfaf0
I20260129 02:59:56.021868 22231 raft_consensus.cc:2804] T a0fcc58d6fb6473cafbd9d34d5195269 P 197b880fa39d432a9b472845876bfaf0 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260129 02:59:56.039830 22857 leader_election.cc:290] T 4e3805ffa70242e8ade66da260dddf66 P 7a9daf5d42a84615a6383cfec9a10608 [CANDIDATE]: Term 5 election: Requested vote from peers 197b880fa39d432a9b472845876bfaf0 (127.18.211.1:42027), 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3:37735)
I20260129 02:59:56.147756 22833 raft_consensus.cc:515] T a609ac02fae84725ab743f95f51d33ce P 7a9daf5d42a84615a6383cfec9a10608 [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } }
W20260129 02:59:56.084093 20405 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.211.3:37735, user_credentials={real_user=slave}} blocked reactor thread for 173325us
I20260129 02:59:56.149714 20405 leader_election.cc:304] T 36ff33be666a4c82954731691dbcf327 P 197b880fa39d432a9b472845876bfaf0 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 197b880fa39d432a9b472845876bfaf0; no voters: 1c50de89a5a845668696dcb28c3badd6, 974b802141bd4f13beffef2aa03f88f9
I20260129 02:59:56.084908 22737 leader_election.cc:290] T 4408006eb5174de1896b037cf3f18457 P 197b880fa39d432a9b472845876bfaf0 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 974b802141bd4f13beffef2aa03f88f9 (127.18.211.2:34517), 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3:37735)
I20260129 02:59:56.070175 22963 raft_consensus.cc:493] T cb4727386a3e48aca5d71cc6c611c197 P 974b802141bd4f13beffef2aa03f88f9 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260129 02:59:56.125739 22959 log.cc:927] Time spent T 42d27a28d89b409e8807ec0efd97af69 P 7a9daf5d42a84615a6383cfec9a10608: Append to log took a long time: real 0.056s user 0.001s sys 0.000s
I20260129 02:59:56.109030 20513 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4b1876f9a70b49edb3aa74105e66d3b3" candidate_uuid: "1c50de89a5a845668696dcb28c3badd6" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "974b802141bd4f13beffef2aa03f88f9" is_pre_election: true
I20260129 02:59:56.105381 22969 raft_consensus.cc:2804] T 00e79d8f25a34676b1d866eb2068a2b2 P 974b802141bd4f13beffef2aa03f88f9 [term 6 FOLLOWER]: Leader pre-election won for term 7
W20260129 02:59:56.108428 22964 log.cc:927] Time spent T 76f713c3331540d2a5b76164103e9eb1 P 1c50de89a5a845668696dcb28c3badd6: Append to log took a long time: real 0.242s user 0.001s sys 0.000s
I20260129 02:59:56.061652 22719 consensus_queue.cc:1048] T e4d18c35b0a7493e85d0a4b92956a48f P 974b802141bd4f13beffef2aa03f88f9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.076s
I20260129 02:59:56.087074 22763 raft_consensus.cc:515] T 0340229d521b432580ae926f6062e3f7 P 974b802141bd4f13beffef2aa03f88f9 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:59:56.019228 22895 consensus_queue.cc:1048] T 1ce4ddf5b10e4ae8b193b13b2e5936ee P 1c50de89a5a845668696dcb28c3badd6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 1, Time since last communication: 0.045s
I20260129 02:59:56.148881 20507 raft_consensus.cc:1275] T 724a236524cb4e7ba3428880681d6cc2 P 974b802141bd4f13beffef2aa03f88f9 [term 4 FOLLOWER]: Refusing update from remote peer 1c50de89a5a845668696dcb28c3badd6: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20260129 02:59:56.081506 22881 raft_consensus.cc:3055] T e33a417cf9c54cf5af73a4ee3aac4d3a P 1c50de89a5a845668696dcb28c3badd6 [term 5 LEADER]: Stepping down as leader of term 5
I20260129 02:59:56.140017 20516 raft_consensus.cc:1240] T add8d4923d1a489e86882c3a25692b95 P 974b802141bd4f13beffef2aa03f88f9 [term 4 FOLLOWER]: Rejecting Update request from peer 1c50de89a5a845668696dcb28c3badd6 for earlier term 3. Current term is 4. Ops: []
I20260129 02:59:56.141909 20508 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fcddab6bca684adf8ea032289a00f90e" candidate_uuid: "1c50de89a5a845668696dcb28c3badd6" candidate_term: 9 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "974b802141bd4f13beffef2aa03f88f9" is_pre_election: true
I20260129 02:59:56.139364 20509 raft_consensus.cc:1275] T 75b0b6e82ddc4f92b1fe1d94f9a0b73b P 974b802141bd4f13beffef2aa03f88f9 [term 3 FOLLOWER]: Refusing update from remote peer 1c50de89a5a845668696dcb28c3badd6: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 3 index: 6. (index mismatch)
I20260129 02:59:56.140365 20512 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a644c02b16af47679824ccd060ac10fd" candidate_uuid: "1c50de89a5a845668696dcb28c3badd6" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "974b802141bd4f13beffef2aa03f88f9" is_pre_election: true
W20260129 02:59:56.148025 22888 consensus_meta.cc:287] T 1299723a4476420ab02fd8bfd7aa0645 P 974b802141bd4f13beffef2aa03f88f9: Time spent flushing consensus metadata: real 0.510s user 0.005s sys 0.006s
W20260129 02:59:56.176028 20404 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260129 02:59:56.190371 20406 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260129 02:59:56.173715 22969 raft_consensus.cc:493] T 00e79d8f25a34676b1d866eb2068a2b2 P 974b802141bd4f13beffef2aa03f88f9 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260129 02:59:56.199770 20480 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.211.3:37735, user_credentials={real_user=slave}} blocked reactor thread for 157464us
I20260129 02:59:56.253437 20512 raft_consensus.cc:2468] T a644c02b16af47679824ccd060ac10fd P 974b802141bd4f13beffef2aa03f88f9 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1c50de89a5a845668696dcb28c3badd6 in term 4.
I20260129 02:59:56.224628 22979 raft_consensus.cc:2804] T b189f08e16a74a11980cc55d99359a0a P 974b802141bd4f13beffef2aa03f88f9 [term 7 FOLLOWER]: Leader election won for term 7
I20260129 02:59:56.256234 22979 raft_consensus.cc:697] T b189f08e16a74a11980cc55d99359a0a P 974b802141bd4f13beffef2aa03f88f9 [term 7 LEADER]: Becoming Leader. State: Replica: 974b802141bd4f13beffef2aa03f88f9, State: Running, Role: LEADER
I20260129 02:59:56.257169 22979 consensus_queue.cc:237] T b189f08e16a74a11980cc55d99359a0a P 974b802141bd4f13beffef2aa03f88f9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.4, Last appended by leader: 4, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
W20260129 02:59:56.221451 20479 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.211.1:42027, user_credentials={real_user=slave}} blocked reactor thread for 178804us
I20260129 02:59:56.141295 22878 leader_election.cc:290] T 4f8f3a38f6e24ba4857875181f447442 P 974b802141bd4f13beffef2aa03f88f9 [CANDIDATE]: Term 6 election: Requested vote from peers 197b880fa39d432a9b472845876bfaf0 (127.18.211.1:42027), 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3:37735)
I20260129 02:59:56.142216 22688 leader_election.cc:290] T 724a236524cb4e7ba3428880681d6cc2 P 197b880fa39d432a9b472845876bfaf0 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 974b802141bd4f13beffef2aa03f88f9 (127.18.211.2:34517), 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3:37735)
I20260129 02:59:56.172644 22691 raft_consensus.cc:2749] T 36ff33be666a4c82954731691dbcf327 P 197b880fa39d432a9b472845876bfaf0 [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260129 02:59:56.172706 22667 raft_consensus.cc:493] T 505580c93c694bbb8ea96fdf9b17ab46 P 7a9daf5d42a84615a6383cfec9a10608 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260129 02:59:56.138387 22955 log.cc:927] Time spent T febffe4a8b394368b69b1d164308fb9a P 197b880fa39d432a9b472845876bfaf0: Append to log took a long time: real 0.225s user 0.004s sys 0.001s
I20260129 02:59:56.172837 22231 raft_consensus.cc:493] T a0fcc58d6fb6473cafbd9d34d5195269 P 197b880fa39d432a9b472845876bfaf0 [term 3 FOLLOWER]: Starting leader election (detected failure of leader 974b802141bd4f13beffef2aa03f88f9)
I20260129 02:59:56.150727 22932 raft_consensus.cc:697] T 75f45538dc704d9a8ec4ff8ad9f3ea5b P 7a9daf5d42a84615a6383cfec9a10608 [term 4 LEADER]: Becoming Leader. State: Replica: 7a9daf5d42a84615a6383cfec9a10608, State: Running, Role: LEADER
I20260129 02:59:56.173053 22963 raft_consensus.cc:3060] T cb4727386a3e48aca5d71cc6c611c197 P 974b802141bd4f13beffef2aa03f88f9 [term 3 FOLLOWER]: Advancing to term 4
W20260129 02:59:56.260553 20556 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.211.2:34517, user_credentials={real_user=slave}} blocked reactor thread for 192066us
I20260129 02:59:56.137095 22933 raft_consensus.cc:515] T 96609ac68bdf4993a12e1c23355ca013 P 7a9daf5d42a84615a6383cfec9a10608 [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } attrs { replace: true } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } }
I20260129 02:59:56.269217 20553 leader_election.cc:304] T 1299723a4476420ab02fd8bfd7aa0645 P 1c50de89a5a845668696dcb28c3badd6 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1c50de89a5a845668696dcb28c3badd6; no voters: 197b880fa39d432a9b472845876bfaf0, 7a9daf5d42a84615a6383cfec9a10608
I20260129 02:59:56.245230 22881 raft_consensus.cc:740] T e33a417cf9c54cf5af73a4ee3aac4d3a P 1c50de89a5a845668696dcb28c3badd6 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 1c50de89a5a845668696dcb28c3badd6, State: Running, Role: LEADER
I20260129 02:59:56.243636 22895 consensus_queue.cc:1243] T 1ce4ddf5b10e4ae8b193b13b2e5936ee P 1c50de89a5a845668696dcb28c3badd6 [LEADER]: Peer 197b880fa39d432a9b472845876bfaf0 log is divergent from this leader: its last log entry 5.3 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 1
I20260129 02:59:56.252234 22888 raft_consensus.cc:2749] T 1299723a4476420ab02fd8bfd7aa0645 P 974b802141bd4f13beffef2aa03f88f9 [term 8 FOLLOWER]: Leader pre-election lost for term 5. Reason: Vote denied by peer 197b880fa39d432a9b472845876bfaf0 (127.18.211.1:42027) with higher term. Message: Invalid argument: T 1299723a4476420ab02fd8bfd7aa0645 P 197b880fa39d432a9b472845876bfaf0 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 974b802141bd4f13beffef2aa03f88f9 for earlier term 5. Current term is 8.
I20260129 02:59:56.294250 20438 raft_consensus.cc:1240] T c2633a6fc34c44579de348d208b0c01e P 197b880fa39d432a9b472845876bfaf0 [term 3 FOLLOWER]: Rejecting Update request from peer 1c50de89a5a845668696dcb28c3badd6 for earlier term 2. Current term is 3. Ops: []
I20260129 02:59:56.261092 22969 raft_consensus.cc:3060] T 00e79d8f25a34676b1d866eb2068a2b2 P 974b802141bd4f13beffef2aa03f88f9 [term 6 FOLLOWER]: Advancing to term 7
I20260129 02:59:56.296307 21135 raft_consensus.cc:1240] T 2ac659d732314e1288849f08fe5e074e P 7a9daf5d42a84615a6383cfec9a10608 [term 4 FOLLOWER]: Rejecting Update request from peer 974b802141bd4f13beffef2aa03f88f9 for earlier term 3. Current term is 4. Ops: []
I20260129 02:59:56.371815 20583 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "75be4bd6bd45474795fd472eff2953ac" candidate_uuid: "974b802141bd4f13beffef2aa03f88f9" candidate_term: 10 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "1c50de89a5a845668696dcb28c3badd6" is_pre_election: true
I20260129 02:59:56.424194 20588 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "add8d4923d1a489e86882c3a25692b95" candidate_uuid: "974b802141bd4f13beffef2aa03f88f9" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "1c50de89a5a845668696dcb28c3badd6"
I20260129 02:59:56.311297 20581 raft_consensus.cc:1275] T fcddab6bca684adf8ea032289a00f90e P 1c50de89a5a845668696dcb28c3badd6 [term 8 FOLLOWER]: Refusing update from remote peer 974b802141bd4f13beffef2aa03f88f9: Log matching property violated. Preceding OpId in replica: term: 4 index: 4. Preceding OpId from leader: term: 8 index: 5. (index mismatch)
W20260129 02:59:56.323446 20556 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260129 02:59:56.296752 21131 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4f5a77e94fef4b16a2ba7b067c8b144a" candidate_uuid: "974b802141bd4f13beffef2aa03f88f9" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "7a9daf5d42a84615a6383cfec9a10608" is_pre_election: true
I20260129 02:59:56.347071 20479 leader_election.cc:304] T 17d6431cc77a491e8464d68db4081a51 P 974b802141bd4f13beffef2aa03f88f9 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 197b880fa39d432a9b472845876bfaf0, 974b802141bd4f13beffef2aa03f88f9; no voters:
I20260129 02:59:56.451550 21131 raft_consensus.cc:2468] T 4f5a77e94fef4b16a2ba7b067c8b144a P 7a9daf5d42a84615a6383cfec9a10608 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 974b802141bd4f13beffef2aa03f88f9 in term 3.
I20260129 02:59:56.450099 20588 raft_consensus.cc:2410] T add8d4923d1a489e86882c3a25692b95 P 1c50de89a5a845668696dcb28c3badd6 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 974b802141bd4f13beffef2aa03f88f9 for term 4 because replica has last-logged OpId of term: 3 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 5.
I20260129 02:59:56.427713 20582 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "45cc5f3d2688486d9b6dbe7e666dedde" candidate_uuid: "974b802141bd4f13beffef2aa03f88f9" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "1c50de89a5a845668696dcb28c3badd6" is_pre_election: true
I20260129 02:59:56.421645 20589 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4f5a77e94fef4b16a2ba7b067c8b144a" candidate_uuid: "974b802141bd4f13beffef2aa03f88f9" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "1c50de89a5a845668696dcb28c3badd6" is_pre_election: true
W20260129 02:59:56.311120 20405 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.211.3:37735, user_credentials={real_user=slave}} blocked reactor thread for 50839.1us
I20260129 02:59:56.372129 20584 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4f8f3a38f6e24ba4857875181f447442" candidate_uuid: "974b802141bd4f13beffef2aa03f88f9" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "1c50de89a5a845668696dcb28c3badd6" is_pre_election: true
W20260129 02:59:56.314846 20406 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260129 02:59:56.455840 20584 raft_consensus.cc:2468] T 4f8f3a38f6e24ba4857875181f447442 P 1c50de89a5a845668696dcb28c3badd6 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 974b802141bd4f13beffef2aa03f88f9 in term 5.
I20260129 02:59:56.414911 20585 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1299723a4476420ab02fd8bfd7aa0645" candidate_uuid: "974b802141bd4f13beffef2aa03f88f9" candidate_term: 5 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "1c50de89a5a845668696dcb28c3badd6" is_pre_election: true
I20260129 02:59:56.290081 22231 raft_consensus.cc:3060] T a0fcc58d6fb6473cafbd9d34d5195269 P 197b880fa39d432a9b472845876bfaf0 [term 3 FOLLOWER]: Advancing to term 4
I20260129 02:59:56.453644 20582 raft_consensus.cc:2468] T 45cc5f3d2688486d9b6dbe7e666dedde P 1c50de89a5a845668696dcb28c3badd6 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 974b802141bd4f13beffef2aa03f88f9 in term 4.
W20260129 02:59:56.462986 20553 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.211.1:42027, user_credentials={real_user=slave}} blocked reactor thread for 158458us
I20260129 02:59:56.465282 20553 leader_election.cc:304] T b189f08e16a74a11980cc55d99359a0a P 1c50de89a5a845668696dcb28c3badd6 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 197b880fa39d432a9b472845876bfaf0, 1c50de89a5a845668696dcb28c3badd6; no voters:
I20260129 02:59:56.371490 20586 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0f20af7fbe5d4db89a6f17948d98912c" candidate_uuid: "974b802141bd4f13beffef2aa03f88f9" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "1c50de89a5a845668696dcb28c3badd6"
I20260129 02:59:56.467459 20586 raft_consensus.cc:3060] T 0f20af7fbe5d4db89a6f17948d98912c P 1c50de89a5a845668696dcb28c3badd6 [term 5 FOLLOWER]: Advancing to term 6
I20260129 02:59:56.366856 21133 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "45cc5f3d2688486d9b6dbe7e666dedde" candidate_uuid: "974b802141bd4f13beffef2aa03f88f9" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "7a9daf5d42a84615a6383cfec9a10608" is_pre_election: true
I20260129 02:59:56.343842 20590 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "76f713c3331540d2a5b76164103e9eb1" candidate_uuid: "974b802141bd4f13beffef2aa03f88f9" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "1c50de89a5a845668696dcb28c3badd6" is_pre_election: true
W20260129 02:59:56.472295 20556 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260129 02:59:56.448350 21127 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1299723a4476420ab02fd8bfd7aa0645" candidate_uuid: "974b802141bd4f13beffef2aa03f88f9" candidate_term: 5 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "7a9daf5d42a84615a6383cfec9a10608" is_pre_election: true
I20260129 02:59:56.476806 20405 leader_election.cc:304] T b189f08e16a74a11980cc55d99359a0a P 197b880fa39d432a9b472845876bfaf0 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 197b880fa39d432a9b472845876bfaf0, 1c50de89a5a845668696dcb28c3badd6; no voters: 974b802141bd4f13beffef2aa03f88f9
I20260129 02:59:56.364549 22976 raft_consensus.cc:493] T 17d6431cc77a491e8464d68db4081a51 P 1c50de89a5a845668696dcb28c3badd6 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260129 02:59:56.477525 20585 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a8479d1fe0bd41e5946186e3a6ca8c7a" candidate_uuid: "974b802141bd4f13beffef2aa03f88f9" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "1c50de89a5a845668696dcb28c3badd6"
W20260129 02:59:56.366037 22974 log.cc:927] Time spent T e4d18c35b0a7493e85d0a4b92956a48f P 974b802141bd4f13beffef2aa03f88f9: Append to log took a long time: real 0.069s user 0.000s sys 0.000s
I20260129 02:59:56.484052 20582 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a4c1226b987b468fa84dfc6ba0e2dbe1" candidate_uuid: "974b802141bd4f13beffef2aa03f88f9" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "1c50de89a5a845668696dcb28c3badd6" is_pre_election: true
I20260129 02:59:56.443580 21128 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "add8d4923d1a489e86882c3a25692b95" candidate_uuid: "974b802141bd4f13beffef2aa03f88f9" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "7a9daf5d42a84615a6383cfec9a10608"
I20260129 02:59:56.489303 20512 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "cb0b3f259ebc4310927338a8740e87b8" candidate_uuid: "1c50de89a5a845668696dcb28c3badd6" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "974b802141bd4f13beffef2aa03f88f9" is_pre_election: true
I20260129 02:59:56.490165 20512 raft_consensus.cc:2468] T cb0b3f259ebc4310927338a8740e87b8 P 974b802141bd4f13beffef2aa03f88f9 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1c50de89a5a845668696dcb28c3badd6 in term 5.
I20260129 02:59:56.480822 20581 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "cb4727386a3e48aca5d71cc6c611c197" candidate_uuid: "974b802141bd4f13beffef2aa03f88f9" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1c50de89a5a845668696dcb28c3badd6" is_pre_election: true
I20260129 02:59:56.481105 20585 raft_consensus.cc:3060] T a8479d1fe0bd41e5946186e3a6ca8c7a P 1c50de89a5a845668696dcb28c3badd6 [term 4 FOLLOWER]: Advancing to term 5
I20260129 02:59:56.481443 20589 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "368953c9102643b2a21f99af36b51905" candidate_uuid: "974b802141bd4f13beffef2aa03f88f9" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "1c50de89a5a845668696dcb28c3badd6" is_pre_election: true
I20260129 02:59:56.482280 20583 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "cb0b3f259ebc4310927338a8740e87b8" candidate_uuid: "974b802141bd4f13beffef2aa03f88f9" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "1c50de89a5a845668696dcb28c3badd6" is_pre_election: true
I20260129 02:59:56.331364 22884 raft_consensus.cc:2749] T 1299723a4476420ab02fd8bfd7aa0645 P 1c50de89a5a845668696dcb28c3badd6 [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260129 02:59:56.424476 22833 leader_election.cc:290] T a609ac02fae84725ab743f95f51d33ce P 7a9daf5d42a84615a6383cfec9a10608 [CANDIDATE]: Term 7 election: Requested vote from peers 197b880fa39d432a9b472845876bfaf0 (127.18.211.1:42027), 974b802141bd4f13beffef2aa03f88f9 (127.18.211.2:34517)
I20260129 02:59:56.293516 22667 raft_consensus.cc:515] T 505580c93c694bbb8ea96fdf9b17ab46 P 7a9daf5d42a84615a6383cfec9a10608 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } }
I20260129 02:59:56.296705 22977 raft_consensus.cc:2804] T 7a9a69cd82724b1faed1d10aa091fc4a P 1c50de89a5a845668696dcb28c3badd6 [term 6 FOLLOWER]: Leader election won for term 6
I20260129 02:59:56.486197 21135 raft_consensus.cc:1240] T 3ccac82a01a2445fa5b0ce6c0fcf2295 P 7a9daf5d42a84615a6383cfec9a10608 [term 4 LEADER]: Rejecting Update request from peer 974b802141bd4f13beffef2aa03f88f9 for earlier term 3. Current term is 4. Ops: [3.5-3.5]
W20260129 02:59:56.506564 20482 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260129 02:59:56.364785 22962 raft_consensus.cc:493] T c934d31efd3f4e5195605ddc354d14e1 P 974b802141bd4f13beffef2aa03f88f9 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 197b880fa39d432a9b472845876bfaf0)
I20260129 02:59:56.493039 20588 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "926a250fa5834e88b96ccecbefbf5086" candidate_uuid: "974b802141bd4f13beffef2aa03f88f9" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "1c50de89a5a845668696dcb28c3badd6"
W20260129 02:59:56.492693 20479 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.211.1:42027, user_credentials={real_user=slave}} blocked reactor thread for 145739us
I20260129 02:59:56.495368 20405 leader_election.cc:304] T c891d2bff4ce4568861bbc992e91fbb0 P 197b880fa39d432a9b472845876bfaf0 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 197b880fa39d432a9b472845876bfaf0, 1c50de89a5a845668696dcb28c3badd6; no voters: 974b802141bd4f13beffef2aa03f88f9
I20260129 02:59:56.495450 22878 raft_consensus.cc:2804] T 17d6431cc77a491e8464d68db4081a51 P 974b802141bd4f13beffef2aa03f88f9 [term 4 FOLLOWER]: Leader election won for term 4
I20260129 02:59:56.495586 21130 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "96609ac68bdf4993a12e1c23355ca013" candidate_uuid: "974b802141bd4f13beffef2aa03f88f9" candidate_term: 6 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "7a9daf5d42a84615a6383cfec9a10608" is_pre_election: true
W20260129 02:59:56.496515 22940 log.cc:927] Time spent T 78aa39096868415d9afc2ffcb248e198 P 197b880fa39d432a9b472845876bfaf0: Append to log took a long time: real 0.079s user 0.000s sys 0.000s
I20260129 02:59:56.497104 20514 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a8479d1fe0bd41e5946186e3a6ca8c7a" candidate_uuid: "1c50de89a5a845668696dcb28c3badd6" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "974b802141bd4f13beffef2aa03f88f9" is_pre_election: true
I20260129 02:59:56.510710 20479 leader_election.cc:304] T 0678b8b468bd43cdbc3e485900844821 P 974b802141bd4f13beffef2aa03f88f9 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 197b880fa39d432a9b472845876bfaf0, 974b802141bd4f13beffef2aa03f88f9; no voters:
I20260129 02:59:56.511358 20514 raft_consensus.cc:2393] T a8479d1fe0bd41e5946186e3a6ca8c7a P 974b802141bd4f13beffef2aa03f88f9 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1c50de89a5a845668696dcb28c3badd6 in current term 5: Already voted for candidate 974b802141bd4f13beffef2aa03f88f9 in this term.
I20260129 02:59:56.476708 22971 raft_consensus.cc:2804] T b189f08e16a74a11980cc55d99359a0a P 1c50de89a5a845668696dcb28c3badd6 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260129 02:59:56.482450 20556 leader_election.cc:304] T a4c1226b987b468fa84dfc6ba0e2dbe1 P 1c50de89a5a845668696dcb28c3badd6 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1c50de89a5a845668696dcb28c3badd6; no voters: 197b880fa39d432a9b472845876bfaf0, 974b802141bd4f13beffef2aa03f88f9
I20260129 02:59:56.481217 20584 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "36ff33be666a4c82954731691dbcf327" candidate_uuid: "974b802141bd4f13beffef2aa03f88f9" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "1c50de89a5a845668696dcb28c3badd6"
I20260129 02:59:56.514020 20556 leader_election.cc:304] T 0f20af7fbe5d4db89a6f17948d98912c P 1c50de89a5a845668696dcb28c3badd6 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1c50de89a5a845668696dcb28c3badd6; no voters: 197b880fa39d432a9b472845876bfaf0, 974b802141bd4f13beffef2aa03f88f9
I20260129 02:59:56.487229 20590 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0340229d521b432580ae926f6062e3f7" candidate_uuid: "974b802141bd4f13beffef2aa03f88f9" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "1c50de89a5a845668696dcb28c3badd6" is_pre_election: true
I20260129 02:59:56.516053 22891 raft_consensus.cc:2749] T a4c1226b987b468fa84dfc6ba0e2dbe1 P 1c50de89a5a845668696dcb28c3badd6 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
W20260129 02:59:56.516537 20404 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260129 02:59:56.516808 22891 consensus_queue.cc:1048] T a1a39dcd328e4268ae41b988c66b4a75 P 1c50de89a5a845668696dcb28c3badd6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
W20260129 02:59:56.509580 20480 connection.cc:232] RPC call timeout handler was delayed by -2.4509415280000439s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2117s for a response
I20260129 02:59:56.520829 20512 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "febffe4a8b394368b69b1d164308fb9a" candidate_uuid: "7a9daf5d42a84615a6383cfec9a10608" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "974b802141bd4f13beffef2aa03f88f9"
I20260129 02:59:56.484228 22691 raft_consensus.cc:2764] T b189f08e16a74a11980cc55d99359a0a P 197b880fa39d432a9b472845876bfaf0 [term 7 FOLLOWER]: Leader election decision vote started in defunct term 6: won
I20260129 02:59:56.523494 20512 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "45cc5f3d2688486d9b6dbe7e666dedde" candidate_uuid: "7a9daf5d42a84615a6383cfec9a10608" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "974b802141bd4f13beffef2aa03f88f9" is_pre_election: true
I20260129 02:59:56.484279 22425 raft_consensus.cc:493] T 926a250fa5834e88b96ccecbefbf5086 P 1c50de89a5a845668696dcb28c3badd6 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260129 02:59:56.533123 20508 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4f5a77e94fef4b16a2ba7b067c8b144a" candidate_uuid: "7a9daf5d42a84615a6383cfec9a10608" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "974b802141bd4f13beffef2aa03f88f9" is_pre_election: true
I20260129 02:59:56.534157 20589 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "cedce331e1584ac8b636c9b8a72b2ba0" candidate_uuid: "974b802141bd4f13beffef2aa03f88f9" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "1c50de89a5a845668696dcb28c3badd6" is_pre_election: true
I20260129 02:59:56.545444 20589 raft_consensus.cc:2468] T cedce331e1584ac8b636c9b8a72b2ba0 P 1c50de89a5a845668696dcb28c3badd6 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 974b802141bd4f13beffef2aa03f88f9 in term 6.
I20260129 02:59:56.534559 20512 raft_consensus.cc:2393] T 45cc5f3d2688486d9b6dbe7e666dedde P 974b802141bd4f13beffef2aa03f88f9 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 7a9daf5d42a84615a6383cfec9a10608 in current term 4: Already voted for candidate 974b802141bd4f13beffef2aa03f88f9 in this term.
I20260129 02:59:56.547271 20583 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "96609ac68bdf4993a12e1c23355ca013" candidate_uuid: "974b802141bd4f13beffef2aa03f88f9" candidate_term: 6 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "1c50de89a5a845668696dcb28c3badd6" is_pre_election: true
I20260129 02:59:56.550554 20435 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "17d6431cc77a491e8464d68db4081a51" candidate_uuid: "974b802141bd4f13beffef2aa03f88f9" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "197b880fa39d432a9b472845876bfaf0" is_pre_election: true
I20260129 02:59:56.551507 20586 raft_consensus.cc:2468] T 0f20af7fbe5d4db89a6f17948d98912c P 1c50de89a5a845668696dcb28c3badd6 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 974b802141bd4f13beffef2aa03f88f9 in term 6.
I20260129 02:59:56.556667 20590 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "17d6431cc77a491e8464d68db4081a51" candidate_uuid: "974b802141bd4f13beffef2aa03f88f9" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1c50de89a5a845668696dcb28c3badd6"
I20260129 02:59:56.558040 20582 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7d9494f5584d4d5db8b9adf1c8fc888b" candidate_uuid: "974b802141bd4f13beffef2aa03f88f9" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "1c50de89a5a845668696dcb28c3badd6" is_pre_election: true
I20260129 02:59:56.555434 20434 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7d9494f5584d4d5db8b9adf1c8fc888b" candidate_uuid: "974b802141bd4f13beffef2aa03f88f9" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "197b880fa39d432a9b472845876bfaf0" is_pre_election: true
I20260129 02:59:56.549155 20516 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4f8f3a38f6e24ba4857875181f447442" candidate_uuid: "1c50de89a5a845668696dcb28c3badd6" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "974b802141bd4f13beffef2aa03f88f9" is_pre_election: true
I20260129 02:59:56.507228 22763 leader_election.cc:290] T 0340229d521b432580ae926f6062e3f7 P 974b802141bd4f13beffef2aa03f88f9 [CANDIDATE]: Term 6 election: Requested vote from peers 197b880fa39d432a9b472845876bfaf0 (127.18.211.1:42027), 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3:37735)
I20260129 02:59:56.560266 20516 raft_consensus.cc:2393] T 4f8f3a38f6e24ba4857875181f447442 P 974b802141bd4f13beffef2aa03f88f9 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1c50de89a5a845668696dcb28c3badd6 in current term 6: Already voted for candidate 974b802141bd4f13beffef2aa03f88f9 in this term.
I20260129 02:59:56.549335 20583 raft_consensus.cc:2468] T 96609ac68bdf4993a12e1c23355ca013 P 1c50de89a5a845668696dcb28c3badd6 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 974b802141bd4f13beffef2aa03f88f9 in term 5.
I20260129 02:59:56.371759 20587 tablet_service.cc:2066] Received LeaderStepDown RPC: tablet_id: "fcddab6bca684adf8ea032289a00f90e"
dest_uuid: "1c50de89a5a845668696dcb28c3badd6"
mode: GRACEFUL
new_leader_uuid: "974b802141bd4f13beffef2aa03f88f9"
from {username='slave'} at 127.0.0.1:54860
I20260129 02:59:56.564623 20587 raft_consensus.cc:606] T fcddab6bca684adf8ea032289a00f90e P 1c50de89a5a845668696dcb28c3badd6 [term 8 FOLLOWER]: Received request to transfer leadership to TS 974b802141bd4f13beffef2aa03f88f9
I20260129 02:59:56.565129 20587 raft_consensus.cc:614] T fcddab6bca684adf8ea032289a00f90e P 1c50de89a5a845668696dcb28c3badd6 [term 8 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260129 02:59:56.558722 20582 raft_consensus.cc:2468] T 7d9494f5584d4d5db8b9adf1c8fc888b P 1c50de89a5a845668696dcb28c3badd6 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 974b802141bd4f13beffef2aa03f88f9 in term 3.
I20260129 02:59:56.562295 21131 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "cb0b3f259ebc4310927338a8740e87b8" candidate_uuid: "974b802141bd4f13beffef2aa03f88f9" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "7a9daf5d42a84615a6383cfec9a10608" is_pre_election: true
I20260129 02:59:56.562969 20586 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "913fc7d751ed4b7388d889aac30a5f9b" candidate_uuid: "974b802141bd4f13beffef2aa03f88f9" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1c50de89a5a845668696dcb28c3badd6" is_pre_election: true
I20260129 02:59:56.551155 20435 raft_consensus.cc:2376] T 17d6431cc77a491e8464d68db4081a51 P 197b880fa39d432a9b472845876bfaf0 [term 4 FOLLOWER]: Leader pre-election vote request: Already granted yes vote for candidate 974b802141bd4f13beffef2aa03f88f9 in term 4. Re-sending same reply.
I20260129 02:59:56.567907 20512 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b189f08e16a74a11980cc55d99359a0a" candidate_uuid: "1c50de89a5a845668696dcb28c3badd6" candidate_term: 8 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "974b802141bd4f13beffef2aa03f88f9" is_pre_election: true
I20260129 02:59:56.491153 22924 consensus_queue.cc:1048] T 724a236524cb4e7ba3428880681d6cc2 P 1c50de89a5a845668696dcb28c3badd6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 1, Time since last communication: 0.057s
I20260129 02:59:56.606511 20589 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4f8f3a38f6e24ba4857875181f447442" candidate_uuid: "974b802141bd4f13beffef2aa03f88f9" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "1c50de89a5a845668696dcb28c3badd6"
I20260129 02:59:56.607213 20589 raft_consensus.cc:3060] T 4f8f3a38f6e24ba4857875181f447442 P 1c50de89a5a845668696dcb28c3badd6 [term 5 FOLLOWER]: Advancing to term 6
I20260129 02:59:56.607656 20512 raft_consensus.cc:1275] T 505580c93c694bbb8ea96fdf9b17ab46 P 974b802141bd4f13beffef2aa03f88f9 [term 5 FOLLOWER]: Refusing update from remote peer 1c50de89a5a845668696dcb28c3badd6: Log matching property violated. Preceding OpId in replica: term: 3 index: 6. Preceding OpId from leader: term: 5 index: 7. (index mismatch)
I20260129 02:59:56.562124 20508 raft_consensus.cc:2468] T 4f5a77e94fef4b16a2ba7b067c8b144a P 974b802141bd4f13beffef2aa03f88f9 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7a9daf5d42a84615a6383cfec9a10608 in term 3.
I20260129 02:59:56.545194 22967 raft_consensus.cc:2804] T 0678b8b468bd43cdbc3e485900844821 P 974b802141bd4f13beffef2aa03f88f9 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260129 02:59:56.576843 20507 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3906b080a5b34a74b8b64128635e8997" candidate_uuid: "1c50de89a5a845668696dcb28c3badd6" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "974b802141bd4f13beffef2aa03f88f9" is_pre_election: true
I20260129 02:59:56.575644 20511 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a5a05fbcbbcf4916bbc01fd1a5b5e90d" candidate_uuid: "1c50de89a5a845668696dcb28c3badd6" candidate_term: 8 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "974b802141bd4f13beffef2aa03f88f9" is_pre_election: true
I20260129 02:59:56.575682 20436 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a4c1226b987b468fa84dfc6ba0e2dbe1" candidate_uuid: "974b802141bd4f13beffef2aa03f88f9" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "197b880fa39d432a9b472845876bfaf0" is_pre_election: true
I20260129 02:59:56.577796 20515 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a0fcc58d6fb6473cafbd9d34d5195269" candidate_uuid: "1c50de89a5a845668696dcb28c3badd6" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "974b802141bd4f13beffef2aa03f88f9" is_pre_election: true
I20260129 02:59:56.568456 21135 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00e79d8f25a34676b1d866eb2068a2b2" candidate_uuid: "974b802141bd4f13beffef2aa03f88f9" candidate_term: 7 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "7a9daf5d42a84615a6383cfec9a10608" is_pre_election: true
I20260129 02:59:56.523826 22184 raft_consensus.cc:493] T 75b0b6e82ddc4f92b1fe1d94f9a0b73b P 974b802141bd4f13beffef2aa03f88f9 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260129 02:59:56.563099 20438 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "cedce331e1584ac8b636c9b8a72b2ba0" candidate_uuid: "974b802141bd4f13beffef2aa03f88f9" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "197b880fa39d432a9b472845876bfaf0" is_pre_election: true
I20260129 02:59:56.576725 20433 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b189f08e16a74a11980cc55d99359a0a" candidate_uuid: "974b802141bd4f13beffef2aa03f88f9" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "197b880fa39d432a9b472845876bfaf0"
I20260129 02:59:56.545240 22977 raft_consensus.cc:697] T 7a9a69cd82724b1faed1d10aa091fc4a P 1c50de89a5a845668696dcb28c3badd6 [term 6 LEADER]: Becoming Leader. State: Replica: 1c50de89a5a845668696dcb28c3badd6, State: Running, Role: LEADER
I20260129 02:59:56.553049 21128 raft_consensus.cc:2393] T add8d4923d1a489e86882c3a25692b95 P 7a9daf5d42a84615a6383cfec9a10608 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 974b802141bd4f13beffef2aa03f88f9 in current term 4: Already voted for candidate 7a9daf5d42a84615a6383cfec9a10608 in this term.
W20260129 02:59:56.576215 20405 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.211.3:37735, user_credentials={real_user=slave}} blocked reactor thread for 80980.9us
I20260129 02:59:56.548103 22645 raft_consensus.cc:2804] T c891d2bff4ce4568861bbc992e91fbb0 P 197b880fa39d432a9b472845876bfaf0 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260129 02:59:56.545984 22878 raft_consensus.cc:697] T 17d6431cc77a491e8464d68db4081a51 P 974b802141bd4f13beffef2aa03f88f9 [term 4 LEADER]: Becoming Leader. State: Replica: 974b802141bd4f13beffef2aa03f88f9, State: Running, Role: LEADER
W20260129 02:59:56.607244 22654 consensus_meta.cc:287] T e7d83d4ccf684e76acb29a0dc844e1b8 P 1c50de89a5a845668696dcb28c3badd6: Time spent flushing consensus metadata: real 0.711s user 0.007s sys 0.003s
I20260129 02:59:56.578181 20509 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a609ac02fae84725ab743f95f51d33ce" candidate_uuid: "7a9daf5d42a84615a6383cfec9a10608" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "974b802141bd4f13beffef2aa03f88f9" is_pre_election: true
I20260129 02:59:56.629257 20583 raft_consensus.cc:1240] T 75f45538dc704d9a8ec4ff8ad9f3ea5b P 1c50de89a5a845668696dcb28c3badd6 [term 4 FOLLOWER]: Rejecting Update request from peer 974b802141bd4f13beffef2aa03f88f9 for earlier term 3. Current term is 4. Ops: []
I20260129 02:59:56.620906 20433 raft_consensus.cc:2376] T b189f08e16a74a11980cc55d99359a0a P 197b880fa39d432a9b472845876bfaf0 [term 7 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 974b802141bd4f13beffef2aa03f88f9 in term 7. Re-sending same reply.
W20260129 02:59:56.605978 22851 consensus_meta.cc:287] T 1ce4ddf5b10e4ae8b193b13b2e5936ee P 197b880fa39d432a9b472845876bfaf0: Time spent flushing consensus metadata: real 0.687s user 0.012s sys 0.000s
I20260129 02:59:56.494302 22976 raft_consensus.cc:515] T 17d6431cc77a491e8464d68db4081a51 P 1c50de89a5a845668696dcb28c3badd6 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:59:56.558220 22895 raft_consensus.cc:2749] T 0f20af7fbe5d4db89a6f17948d98912c P 1c50de89a5a845668696dcb28c3badd6 [term 6 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260129 02:59:56.626533 20508 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c934d31efd3f4e5195605ddc354d14e1" candidate_uuid: "7a9daf5d42a84615a6383cfec9a10608" candidate_term: 6 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "974b802141bd4f13beffef2aa03f88f9" is_pre_election: true
I20260129 02:59:56.524945 22971 raft_consensus.cc:493] T b189f08e16a74a11980cc55d99359a0a P 1c50de89a5a845668696dcb28c3badd6 [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260129 02:59:56.488565 22881 consensus_queue.cc:260] T e33a417cf9c54cf5af73a4ee3aac4d3a P 1c50de89a5a845668696dcb28c3badd6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 5.3, Last appended by leader: 3, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:59:56.620038 21130 raft_consensus.cc:2393] T 96609ac68bdf4993a12e1c23355ca013 P 7a9daf5d42a84615a6383cfec9a10608 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 974b802141bd4f13beffef2aa03f88f9 in current term 6: Already voted for candidate 7a9daf5d42a84615a6383cfec9a10608 in this term.
I20260129 02:59:56.631975 20440 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4f8f3a38f6e24ba4857875181f447442" candidate_uuid: "974b802141bd4f13beffef2aa03f88f9" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "197b880fa39d432a9b472845876bfaf0"
I20260129 02:59:56.478070 22932 consensus_queue.cc:237] T 75f45538dc704d9a8ec4ff8ad9f3ea5b P 7a9daf5d42a84615a6383cfec9a10608 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } }
I20260129 02:59:56.618140 22967 raft_consensus.cc:493] T 0678b8b468bd43cdbc3e485900844821 P 974b802141bd4f13beffef2aa03f88f9 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260129 02:59:56.641589 20516 raft_consensus.cc:1275] T 3e2d15f8640b407195210c476e09df91 P 974b802141bd4f13beffef2aa03f88f9 [term 3 FOLLOWER]: Refusing update from remote peer 1c50de89a5a845668696dcb28c3badd6: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 3 index: 5. (index mismatch)
I20260129 02:59:56.562508 22425 raft_consensus.cc:515] T 926a250fa5834e88b96ccecbefbf5086 P 1c50de89a5a845668696dcb28c3badd6 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:59:56.646698 20405 leader_election.cc:304] T cb4727386a3e48aca5d71cc6c611c197 P 197b880fa39d432a9b472845876bfaf0 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 197b880fa39d432a9b472845876bfaf0, 1c50de89a5a845668696dcb28c3badd6; no voters: 974b802141bd4f13beffef2aa03f88f9
W20260129 02:59:56.627075 22640 consensus_meta.cc:287] T 18f30ebaa4174fd28c958e72d0cc7e24 P 1c50de89a5a845668696dcb28c3badd6: Time spent flushing consensus metadata: real 0.682s user 0.010s sys 0.000s
I20260129 02:59:56.547240 22962 raft_consensus.cc:515] T c934d31efd3f4e5195605ddc354d14e1 P 974b802141bd4f13beffef2aa03f88f9 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } }
I20260129 02:59:56.656919 20512 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4e420f90884f4796997b5ea41e49e411" candidate_uuid: "7a9daf5d42a84615a6383cfec9a10608" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "974b802141bd4f13beffef2aa03f88f9" is_pre_election: true
I20260129 02:59:56.658061 20582 raft_consensus.cc:1275] T 3906b080a5b34a74b8b64128635e8997 P 1c50de89a5a845668696dcb28c3badd6 [term 4 FOLLOWER]: Refusing update from remote peer 974b802141bd4f13beffef2aa03f88f9: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20260129 02:59:56.677680 20507 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "96609ac68bdf4993a12e1c23355ca013" candidate_uuid: "7a9daf5d42a84615a6383cfec9a10608" candidate_term: 6 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "974b802141bd4f13beffef2aa03f88f9" is_pre_election: true
I20260129 02:59:56.636029 22851 raft_consensus.cc:515] T 1ce4ddf5b10e4ae8b193b13b2e5936ee P 197b880fa39d432a9b472845876bfaf0 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:59:56.656984 20508 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "75b0b6e82ddc4f92b1fe1d94f9a0b73b" candidate_uuid: "7a9daf5d42a84615a6383cfec9a10608" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "974b802141bd4f13beffef2aa03f88f9" is_pre_election: true
I20260129 02:59:56.625649 22645 raft_consensus.cc:493] T c891d2bff4ce4568861bbc992e91fbb0 P 197b880fa39d432a9b472845876bfaf0 [term 6 FOLLOWER]: Starting leader election (detected failure of leader 974b802141bd4f13beffef2aa03f88f9)
W20260129 02:59:56.659912 20556 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.211.2:34517, user_credentials={real_user=slave}} blocked reactor thread for 145960us
I20260129 02:59:56.661726 20590 raft_consensus.cc:3060] T 17d6431cc77a491e8464d68db4081a51 P 1c50de89a5a845668696dcb28c3badd6 [term 3 FOLLOWER]: Advancing to term 4
W20260129 02:59:56.662637 20553 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.211.1:42027, user_credentials={real_user=slave}} blocked reactor thread for 189143us
I20260129 02:59:56.686976 20507 raft_consensus.cc:2410] T 96609ac68bdf4993a12e1c23355ca013 P 974b802141bd4f13beffef2aa03f88f9 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 7a9daf5d42a84615a6383cfec9a10608 for term 6 because replica has last-logged OpId of term: 3 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 4.
I20260129 02:59:56.687038 20553 leader_election.cc:304] T 4f8f3a38f6e24ba4857875181f447442 P 1c50de89a5a845668696dcb28c3badd6 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 197b880fa39d432a9b472845876bfaf0, 1c50de89a5a845668696dcb28c3badd6; no voters:
I20260129 02:59:56.665954 20440 raft_consensus.cc:3060] T 4f8f3a38f6e24ba4857875181f447442 P 197b880fa39d432a9b472845876bfaf0 [term 4 FOLLOWER]: Advancing to term 6
I20260129 02:59:56.688438 20589 raft_consensus.cc:2468] T 4f8f3a38f6e24ba4857875181f447442 P 1c50de89a5a845668696dcb28c3badd6 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 974b802141bd4f13beffef2aa03f88f9 in term 6.
W20260129 02:59:56.659492 22856 consensus_meta.cc:287] T 913fc7d751ed4b7388d889aac30a5f9b P 1c50de89a5a845668696dcb28c3badd6: Time spent flushing consensus metadata: real 0.639s user 0.001s sys 0.008s
I20260129 02:59:56.689719 20512 raft_consensus.cc:2468] T 4e420f90884f4796997b5ea41e49e411 P 974b802141bd4f13beffef2aa03f88f9 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7a9daf5d42a84615a6383cfec9a10608 in term 5.
I20260129 02:59:56.651616 21127 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0678b8b468bd43cdbc3e485900844821" candidate_uuid: "974b802141bd4f13beffef2aa03f88f9" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "7a9daf5d42a84615a6383cfec9a10608" is_pre_election: true
W20260129 02:59:56.664302 22922 consensus_meta.cc:287] T cb4727386a3e48aca5d71cc6c611c197 P 1c50de89a5a845668696dcb28c3badd6: Time spent flushing consensus metadata: real 0.627s user 0.003s sys 0.007s
I20260129 02:59:56.633554 22654 raft_consensus.cc:515] T e7d83d4ccf684e76acb29a0dc844e1b8 P 1c50de89a5a845668696dcb28c3badd6 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:59:56.685150 20509 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e4d18c35b0a7493e85d0a4b92956a48f" candidate_uuid: "1c50de89a5a845668696dcb28c3badd6" candidate_term: 6 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "974b802141bd4f13beffef2aa03f88f9" is_pre_election: true
I20260129 02:59:56.639657 22971 raft_consensus.cc:3060] T b189f08e16a74a11980cc55d99359a0a P 1c50de89a5a845668696dcb28c3badd6 [term 7 FOLLOWER]: Advancing to term 8
I20260129 02:59:56.626332 22184 raft_consensus.cc:515] T 75b0b6e82ddc4f92b1fe1d94f9a0b73b P 974b802141bd4f13beffef2aa03f88f9 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } }
I20260129 02:59:56.679008 20511 raft_consensus.cc:1275] T a644c02b16af47679824ccd060ac10fd P 974b802141bd4f13beffef2aa03f88f9 [term 4 FOLLOWER]: Refusing update from remote peer 7a9daf5d42a84615a6383cfec9a10608: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 3 index: 5. (index mismatch)
I20260129 02:59:56.696853 20511 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "87d5fe48a44a408aaf03e894a67ecb2f" candidate_uuid: "7a9daf5d42a84615a6383cfec9a10608" candidate_term: 7 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "974b802141bd4f13beffef2aa03f88f9" is_pre_election: true
I20260129 02:59:56.697296 22995 raft_consensus.cc:2764] T 4f8f3a38f6e24ba4857875181f447442 P 1c50de89a5a845668696dcb28c3badd6 [term 6 FOLLOWER]: Leader pre-election decision vote started in defunct term 5: won
I20260129 02:59:56.698359 20511 raft_consensus.cc:1275] T 3ccac82a01a2445fa5b0ce6c0fcf2295 P 974b802141bd4f13beffef2aa03f88f9 [term 4 FOLLOWER]: Refusing update from remote peer 7a9daf5d42a84615a6383cfec9a10608: Log matching property violated. Preceding OpId in replica: term: 3 index: 5. Preceding OpId from leader: term: 4 index: 5. (term mismatch)
I20260129 02:59:56.698772 20511 pending_rounds.cc:85] T 3ccac82a01a2445fa5b0ce6c0fcf2295 P 974b802141bd4f13beffef2aa03f88f9: Aborting all ops after (but not including) 4
I20260129 02:59:56.698997 20511 pending_rounds.cc:107] T 3ccac82a01a2445fa5b0ce6c0fcf2295 P 974b802141bd4f13beffef2aa03f88f9: Aborting uncommitted NO_OP operation due to leader change: 3.5
I20260129 02:59:56.699226 20511 raft_consensus.cc:2889] T 3ccac82a01a2445fa5b0ce6c0fcf2295 P 974b802141bd4f13beffef2aa03f88f9 [term 4 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
W20260129 02:59:56.700148 20406 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260129 02:59:56.701241 20512 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "add8d4923d1a489e86882c3a25692b95" candidate_uuid: "7a9daf5d42a84615a6383cfec9a10608" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "974b802141bd4f13beffef2aa03f88f9"
I20260129 02:59:56.702176 20512 raft_consensus.cc:2393] T add8d4923d1a489e86882c3a25692b95 P 974b802141bd4f13beffef2aa03f88f9 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 7a9daf5d42a84615a6383cfec9a10608 in current term 4: Already voted for candidate 974b802141bd4f13beffef2aa03f88f9 in this term.
I20260129 02:59:56.673187 22967 raft_consensus.cc:3060] T 0678b8b468bd43cdbc3e485900844821 P 974b802141bd4f13beffef2aa03f88f9 [term 6 FOLLOWER]: Advancing to term 7
I20260129 02:59:56.701535 20516 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2ac659d732314e1288849f08fe5e074e" candidate_uuid: "7a9daf5d42a84615a6383cfec9a10608" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "974b802141bd4f13beffef2aa03f88f9"
I20260129 02:59:56.705719 20509 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "75f45538dc704d9a8ec4ff8ad9f3ea5b" candidate_uuid: "7a9daf5d42a84615a6383cfec9a10608" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "974b802141bd4f13beffef2aa03f88f9" is_pre_election: true
I20260129 02:59:56.707762 20588 raft_consensus.cc:2376] T 926a250fa5834e88b96ccecbefbf5086 P 1c50de89a5a845668696dcb28c3badd6 [term 6 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 974b802141bd4f13beffef2aa03f88f9 in term 6. Re-sending same reply.
I20260129 02:59:56.690363 22645 raft_consensus.cc:3060] T c891d2bff4ce4568861bbc992e91fbb0 P 197b880fa39d432a9b472845876bfaf0 [term 6 FOLLOWER]: Advancing to term 7
I20260129 02:59:56.717342 20508 raft_consensus.cc:2468] T 75b0b6e82ddc4f92b1fe1d94f9a0b73b P 974b802141bd4f13beffef2aa03f88f9 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7a9daf5d42a84615a6383cfec9a10608 in term 3.
I20260129 02:59:56.724499 20512 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a609ac02fae84725ab743f95f51d33ce" candidate_uuid: "7a9daf5d42a84615a6383cfec9a10608" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "974b802141bd4f13beffef2aa03f88f9" is_pre_election: true
W20260129 02:59:56.725405 20405 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.211.3:37735, user_credentials={real_user=slave}} blocked reactor thread for 86546.8us
W20260129 02:59:56.726238 20404 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260129 02:59:56.672271 22891 consensus_queue.cc:1059] T e33a417cf9c54cf5af73a4ee3aac4d3a P 1c50de89a5a845668696dcb28c3badd6 [NON_LEADER]: Peer responded invalid term: Peer: permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 }, Status: INVALID_TERM, Last received: 3.2, Next index: 3, Last known committed idx: 1, Time since last communication: 0.040s
I20260129 02:59:56.704582 22691 raft_consensus.cc:2804] T cb4727386a3e48aca5d71cc6c611c197 P 197b880fa39d432a9b472845876bfaf0 [term 3 FOLLOWER]: Leader election won for term 3
I20260129 02:59:56.704814 22933 leader_election.cc:290] T 96609ac68bdf4993a12e1c23355ca013 P 7a9daf5d42a84615a6383cfec9a10608 [CANDIDATE]: Term 6 election: Requested vote from peers 197b880fa39d432a9b472845876bfaf0 (127.18.211.1:42027), 974b802141bd4f13beffef2aa03f88f9 (127.18.211.2:34517), 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3:37735)
I20260129 02:59:56.755832 20509 raft_consensus.cc:1240] T 0340229d521b432580ae926f6062e3f7 P 974b802141bd4f13beffef2aa03f88f9 [term 6 FOLLOWER]: Rejecting Update request from peer 1c50de89a5a845668696dcb28c3badd6 for earlier term 5. Current term is 6. Ops: []
W20260129 02:59:56.755925 20555 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260129 02:59:56.758113 21132 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4e3805ffa70242e8ade66da260dddf66" candidate_uuid: "1c50de89a5a845668696dcb28c3badd6" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "7a9daf5d42a84615a6383cfec9a10608"
I20260129 02:59:56.758936 21132 raft_consensus.cc:2393] T 4e3805ffa70242e8ade66da260dddf66 P 7a9daf5d42a84615a6383cfec9a10608 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 1c50de89a5a845668696dcb28c3badd6 in current term 5: Already voted for candidate 7a9daf5d42a84615a6383cfec9a10608 in this term.
I20260129 02:59:56.707816 21127 raft_consensus.cc:2468] T 0678b8b468bd43cdbc3e485900844821 P 7a9daf5d42a84615a6383cfec9a10608 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 974b802141bd4f13beffef2aa03f88f9 in term 6.
W20260129 02:59:56.762090 22953 consensus_meta.cc:287] T 36ff33be666a4c82954731691dbcf327 P 1c50de89a5a845668696dcb28c3badd6: Time spent flushing consensus metadata: real 0.916s user 0.006s sys 0.010s
I20260129 02:59:56.724570 22977 consensus_queue.cc:237] T 7a9a69cd82724b1faed1d10aa091fc4a P 1c50de89a5a845668696dcb28c3badd6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:59:56.763332 20582 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "96609ac68bdf4993a12e1c23355ca013" candidate_uuid: "7a9daf5d42a84615a6383cfec9a10608" candidate_term: 6 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "1c50de89a5a845668696dcb28c3badd6"
I20260129 02:59:56.786705 20582 raft_consensus.cc:3060] T 96609ac68bdf4993a12e1c23355ca013 P 1c50de89a5a845668696dcb28c3badd6 [term 5 FOLLOWER]: Advancing to term 6
I20260129 02:59:56.765020 22425 leader_election.cc:290] T 926a250fa5834e88b96ccecbefbf5086 P 1c50de89a5a845668696dcb28c3badd6 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 197b880fa39d432a9b472845876bfaf0 (127.18.211.1:42027), 974b802141bd4f13beffef2aa03f88f9 (127.18.211.2:34517)
I20260129 02:59:56.724800 22900 leader_election.cc:290] T 4e3805ffa70242e8ade66da260dddf66 P 1c50de89a5a845668696dcb28c3badd6 [CANDIDATE]: Term 5 election: Requested vote from peers 197b880fa39d432a9b472845876bfaf0 (127.18.211.1:42027), 7a9daf5d42a84615a6383cfec9a10608 (127.18.211.4:40439)
I20260129 02:59:56.685137 22640 raft_consensus.cc:515] T 18f30ebaa4174fd28c958e72d0cc7e24 P 1c50de89a5a845668696dcb28c3badd6 [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } }
W20260129 02:59:56.804042 20556 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.211.2:34517, user_credentials={real_user=slave}} blocked reactor thread for 110703us
I20260129 02:59:56.693506 22856 raft_consensus.cc:515] T 913fc7d751ed4b7388d889aac30a5f9b P 1c50de89a5a845668696dcb28c3badd6 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:59:56.788853 20440 raft_consensus.cc:2468] T 4f8f3a38f6e24ba4857875181f447442 P 197b880fa39d432a9b472845876bfaf0 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 974b802141bd4f13beffef2aa03f88f9 in term 6.
I20260129 02:59:56.727471 22888 raft_consensus.cc:493] T 3e2d15f8640b407195210c476e09df91 P 974b802141bd4f13beffef2aa03f88f9 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260129 02:59:56.768699 22691 raft_consensus.cc:697] T cb4727386a3e48aca5d71cc6c611c197 P 197b880fa39d432a9b472845876bfaf0 [term 3 LEADER]: Becoming Leader. State: Replica: 197b880fa39d432a9b472845876bfaf0, State: Running, Role: LEADER
I20260129 02:59:56.704972 22922 raft_consensus.cc:515] T cb4727386a3e48aca5d71cc6c611c197 P 1c50de89a5a845668696dcb28c3badd6 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:59:56.790179 20507 raft_consensus.cc:1275] T 75be4bd6bd45474795fd472eff2953ac P 974b802141bd4f13beffef2aa03f88f9 [term 9 FOLLOWER]: Refusing update from remote peer 1c50de89a5a845668696dcb28c3badd6: Log matching property violated. Preceding OpId in replica: term: 7 index: 5. Preceding OpId from leader: term: 9 index: 6. (index mismatch)
I20260129 02:59:56.798921 20584 raft_consensus.cc:2468] T 36ff33be666a4c82954731691dbcf327 P 1c50de89a5a845668696dcb28c3badd6 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 974b802141bd4f13beffef2aa03f88f9 in term 3.
I20260129 02:59:56.808638 20479 leader_election.cc:304] T 4f8f3a38f6e24ba4857875181f447442 P 974b802141bd4f13beffef2aa03f88f9 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 197b880fa39d432a9b472845876bfaf0, 974b802141bd4f13beffef2aa03f88f9; no voters:
I20260129 02:59:56.807654 20509 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4f5a77e94fef4b16a2ba7b067c8b144a" candidate_uuid: "7a9daf5d42a84615a6383cfec9a10608" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "974b802141bd4f13beffef2aa03f88f9" is_pre_election: true
I20260129 02:59:56.811798 20590 raft_consensus.cc:2468] T 17d6431cc77a491e8464d68db4081a51 P 1c50de89a5a845668696dcb28c3badd6 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 974b802141bd4f13beffef2aa03f88f9 in term 4.
W20260129 02:59:56.815753 21102 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.211.2:34517, user_credentials={real_user=slave}} blocked reactor thread for 111216us
I20260129 02:59:56.816457 20440 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0340229d521b432580ae926f6062e3f7" candidate_uuid: "974b802141bd4f13beffef2aa03f88f9" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "197b880fa39d432a9b472845876bfaf0"
I20260129 02:59:56.818150 20512 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "75b0b6e82ddc4f92b1fe1d94f9a0b73b" candidate_uuid: "7a9daf5d42a84615a6383cfec9a10608" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "974b802141bd4f13beffef2aa03f88f9" is_pre_election: true
I20260129 02:59:56.788120 22884 raft_consensus.cc:493] T add8d4923d1a489e86882c3a25692b95 P 1c50de89a5a845668696dcb28c3badd6 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260129 02:59:56.710672 22878 consensus_queue.cc:237] T 17d6431cc77a491e8464d68db4081a51 P 974b802141bd4f13beffef2aa03f88f9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:59:56.790041 22667 leader_election.cc:290] T 505580c93c694bbb8ea96fdf9b17ab46 P 7a9daf5d42a84615a6383cfec9a10608 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 974b802141bd4f13beffef2aa03f88f9 (127.18.211.2:34517), 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3:37735)
I20260129 02:59:56.828507 20586 raft_consensus.cc:2393] T 913fc7d751ed4b7388d889aac30a5f9b P 1c50de89a5a845668696dcb28c3badd6 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 974b802141bd4f13beffef2aa03f88f9 in current term 4: Already voted for candidate 1c50de89a5a845668696dcb28c3badd6 in this term.
I20260129 02:59:56.828917 20556 leader_election.cc:304] T 4b1876f9a70b49edb3aa74105e66d3b3 P 1c50de89a5a845668696dcb28c3badd6 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1c50de89a5a845668696dcb28c3badd6; no voters: 197b880fa39d432a9b472845876bfaf0, 974b802141bd4f13beffef2aa03f88f9
I20260129 02:59:56.830650 20556 leader_election.cc:304] T a644c02b16af47679824ccd060ac10fd P 1c50de89a5a845668696dcb28c3badd6 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1c50de89a5a845668696dcb28c3badd6, 974b802141bd4f13beffef2aa03f88f9; no voters: 7a9daf5d42a84615a6383cfec9a10608
I20260129 02:59:56.832865 20581 raft_consensus.cc:2393] T cb4727386a3e48aca5d71cc6c611c197 P 1c50de89a5a845668696dcb28c3badd6 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 974b802141bd4f13beffef2aa03f88f9 in current term 4: Already voted for candidate 1c50de89a5a845668696dcb28c3badd6 in this term.
I20260129 02:59:56.835485 20556 leader_election.cc:304] T 7d9494f5584d4d5db8b9adf1c8fc888b P 1c50de89a5a845668696dcb28c3badd6 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1c50de89a5a845668696dcb28c3badd6, 974b802141bd4f13beffef2aa03f88f9; no voters: 197b880fa39d432a9b472845876bfaf0
I20260129 02:59:56.835937 20432 pending_rounds.cc:85] T a5a05fbcbbcf4916bbc01fd1a5b5e90d P 197b880fa39d432a9b472845876bfaf0: Aborting all ops after (but not including) 4
I20260129 02:59:56.831238 22953 consensus_queue.cc:1048] T 75b0b6e82ddc4f92b1fe1d94f9a0b73b P 1c50de89a5a845668696dcb28c3badd6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.026s
I20260129 02:59:56.808748 20587 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "505580c93c694bbb8ea96fdf9b17ab46" candidate_uuid: "7a9daf5d42a84615a6383cfec9a10608" candidate_term: 6 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "1c50de89a5a845668696dcb28c3badd6" is_pre_election: true
I20260129 02:59:56.857476 22851 leader_election.cc:290] T 1ce4ddf5b10e4ae8b193b13b2e5936ee P 197b880fa39d432a9b472845876bfaf0 [CANDIDATE]: Term 7 election: Requested vote from peers 974b802141bd4f13beffef2aa03f88f9 (127.18.211.2:34517), 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3:37735)
I20260129 02:59:56.858284 22953 raft_consensus.cc:2749] T 4b1876f9a70b49edb3aa74105e66d3b3 P 1c50de89a5a845668696dcb28c3badd6 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260129 02:59:56.829895 22884 raft_consensus.cc:515] T add8d4923d1a489e86882c3a25692b95 P 1c50de89a5a845668696dcb28c3badd6 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } }
I20260129 02:59:56.859035 22953 raft_consensus.cc:2804] T a644c02b16af47679824ccd060ac10fd P 1c50de89a5a845668696dcb28c3badd6 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260129 02:59:56.859411 22953 raft_consensus.cc:493] T a644c02b16af47679824ccd060ac10fd P 1c50de89a5a845668696dcb28c3badd6 [term 4 FOLLOWER]: Starting leader election (detected failure of leader 7a9daf5d42a84615a6383cfec9a10608)
I20260129 02:59:56.860272 22884 leader_election.cc:290] T add8d4923d1a489e86882c3a25692b95 P 1c50de89a5a845668696dcb28c3badd6 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 974b802141bd4f13beffef2aa03f88f9 (127.18.211.2:34517), 7a9daf5d42a84615a6383cfec9a10608 (127.18.211.4:40439)
I20260129 02:59:56.860556 20439 raft_consensus.cc:1240] T 8055232cf0004d40a85f3bf983fe00a9 P 197b880fa39d432a9b472845876bfaf0 [term 4 FOLLOWER]: Rejecting Update request from peer 974b802141bd4f13beffef2aa03f88f9 for earlier term 3. Current term is 4. Ops: []
I20260129 02:59:56.860854 22884 raft_consensus.cc:2804] T 7d9494f5584d4d5db8b9adf1c8fc888b P 1c50de89a5a845668696dcb28c3badd6 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260129 02:59:56.784934 22891 consensus_queue.cc:1243] T e33a417cf9c54cf5af73a4ee3aac4d3a P 1c50de89a5a845668696dcb28c3badd6 [NON_LEADER]: Peer 974b802141bd4f13beffef2aa03f88f9 log is divergent from this leader: its last log entry 6.3 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 1
I20260129 02:59:56.830710 22986 raft_consensus.cc:2804] T 4f8f3a38f6e24ba4857875181f447442 P 974b802141bd4f13beffef2aa03f88f9 [term 6 FOLLOWER]: Leader election won for term 6
W20260129 02:59:56.836434 20553 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.211.1:42027, user_credentials={real_user=slave}} blocked reactor thread for 144979us
W20260129 02:59:56.813277 22990 log.cc:927] Time spent T a644c02b16af47679824ccd060ac10fd P 7a9daf5d42a84615a6383cfec9a10608: Append to log took a long time: real 0.154s user 0.000s sys 0.001s
I20260129 02:59:56.860880 22856 leader_election.cc:290] T 913fc7d751ed4b7388d889aac30a5f9b P 1c50de89a5a845668696dcb28c3badd6 [CANDIDATE]: Term 4 election: Requested vote from peers 197b880fa39d432a9b472845876bfaf0 (127.18.211.1:42027), 974b802141bd4f13beffef2aa03f88f9 (127.18.211.2:34517)
I20260129 02:59:56.868973 20439 raft_consensus.cc:1275] T 36ff33be666a4c82954731691dbcf327 P 197b880fa39d432a9b472845876bfaf0 [term 3 FOLLOWER]: Refusing update from remote peer 974b802141bd4f13beffef2aa03f88f9: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260129 02:59:56.869603 20509 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "45cc5f3d2688486d9b6dbe7e666dedde" candidate_uuid: "7a9daf5d42a84615a6383cfec9a10608" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "974b802141bd4f13beffef2aa03f88f9" is_pre_election: true
I20260129 02:59:56.904064 20509 raft_consensus.cc:2393] T 45cc5f3d2688486d9b6dbe7e666dedde P 974b802141bd4f13beffef2aa03f88f9 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 7a9daf5d42a84615a6383cfec9a10608 in current term 4: Already voted for candidate 974b802141bd4f13beffef2aa03f88f9 in this term.
I20260129 02:59:56.859760 22953 raft_consensus.cc:3060] T a644c02b16af47679824ccd060ac10fd P 1c50de89a5a845668696dcb28c3badd6 [term 4 FOLLOWER]: Advancing to term 5
W20260129 02:59:56.861337 20405 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.211.3:37735, user_credentials={real_user=slave}} blocked reactor thread for 99344.6us
I20260129 02:59:56.857093 20432 pending_rounds.cc:107] T a5a05fbcbbcf4916bbc01fd1a5b5e90d P 197b880fa39d432a9b472845876bfaf0: Aborting uncommitted NO_OP operation due to leader change: 6.5
W20260129 02:59:56.886023 20406 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260129 02:59:56.896881 20482 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260129 02:59:56.871982 22884 raft_consensus.cc:493] T 7d9494f5584d4d5db8b9adf1c8fc888b P 1c50de89a5a845668696dcb28c3badd6 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260129 02:59:56.906595 20507 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "febffe4a8b394368b69b1d164308fb9a" candidate_uuid: "7a9daf5d42a84615a6383cfec9a10608" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "974b802141bd4f13beffef2aa03f88f9"
I20260129 02:59:56.904856 20553 leader_election.cc:304] T fcddab6bca684adf8ea032289a00f90e P 1c50de89a5a845668696dcb28c3badd6 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 197b880fa39d432a9b472845876bfaf0, 1c50de89a5a845668696dcb28c3badd6; no voters: 974b802141bd4f13beffef2aa03f88f9
I20260129 02:59:56.831336 22425 consensus_queue.cc:1048] T 724a236524cb4e7ba3428880681d6cc2 P 1c50de89a5a845668696dcb28c3badd6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 1, Time since last communication: 0.070s
I20260129 02:59:56.908314 22184 leader_election.cc:290] T 75b0b6e82ddc4f92b1fe1d94f9a0b73b P 974b802141bd4f13beffef2aa03f88f9 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3:37735), 7a9daf5d42a84615a6383cfec9a10608 (127.18.211.4:40439)
I20260129 02:59:56.908887 22976 leader_election.cc:290] T 17d6431cc77a491e8464d68db4081a51 P 1c50de89a5a845668696dcb28c3badd6 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 197b880fa39d432a9b472845876bfaf0 (127.18.211.1:42027), 974b802141bd4f13beffef2aa03f88f9 (127.18.211.2:34517)
I20260129 02:59:56.910106 22640 leader_election.cc:290] T 18f30ebaa4174fd28c958e72d0cc7e24 P 1c50de89a5a845668696dcb28c3badd6 [CANDIDATE]: Term 6 election: Requested vote from peers 197b880fa39d432a9b472845876bfaf0 (127.18.211.1:42027), 7a9daf5d42a84615a6383cfec9a10608 (127.18.211.4:40439)
I20260129 02:59:56.910631 22654 leader_election.cc:290] T e7d83d4ccf684e76acb29a0dc844e1b8 P 1c50de89a5a845668696dcb28c3badd6 [CANDIDATE]: Term 6 election: Requested vote from peers 197b880fa39d432a9b472845876bfaf0 (127.18.211.1:42027), 974b802141bd4f13beffef2aa03f88f9 (127.18.211.2:34517)
I20260129 02:59:56.900094 22986 raft_consensus.cc:697] T 4f8f3a38f6e24ba4857875181f447442 P 974b802141bd4f13beffef2aa03f88f9 [term 6 LEADER]: Becoming Leader. State: Replica: 974b802141bd4f13beffef2aa03f88f9, State: Running, Role: LEADER
I20260129 02:59:56.827898 22888 raft_consensus.cc:515] T 3e2d15f8640b407195210c476e09df91 P 974b802141bd4f13beffef2aa03f88f9 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } }
I20260129 02:59:56.916502 20432 raft_consensus.cc:2889] T a5a05fbcbbcf4916bbc01fd1a5b5e90d P 197b880fa39d432a9b472845876bfaf0 [term 7 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
W20260129 02:59:56.916429 20556 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.211.2:34517, user_credentials={real_user=slave}} blocked reactor thread for 81019.7us
I20260129 02:59:56.927346 20507 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "75f45538dc704d9a8ec4ff8ad9f3ea5b" candidate_uuid: "7a9daf5d42a84615a6383cfec9a10608" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "974b802141bd4f13beffef2aa03f88f9"
I20260129 02:59:56.910442 22691 consensus_queue.cc:237] T cb4727386a3e48aca5d71cc6c611c197 P 197b880fa39d432a9b472845876bfaf0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:59:56.877487 22581 consensus_queue.cc:1059] T e33a417cf9c54cf5af73a4ee3aac4d3a P 1c50de89a5a845668696dcb28c3badd6 [NON_LEADER]: Peer responded invalid term: Peer: permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 }, Status: INVALID_TERM, Last received: 3.2, Next index: 3, Last known committed idx: 1, Time since last communication: 0.044s
I20260129 02:59:56.905812 21127 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "add8d4923d1a489e86882c3a25692b95" candidate_uuid: "1c50de89a5a845668696dcb28c3badd6" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "7a9daf5d42a84615a6383cfec9a10608" is_pre_election: true
I20260129 02:59:56.932482 20556 leader_election.cc:304] T a8479d1fe0bd41e5946186e3a6ca8c7a P 1c50de89a5a845668696dcb28c3badd6 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1c50de89a5a845668696dcb28c3badd6; no voters: 197b880fa39d432a9b472845876bfaf0, 974b802141bd4f13beffef2aa03f88f9
I20260129 02:59:56.921239 22884 raft_consensus.cc:3060] T 7d9494f5584d4d5db8b9adf1c8fc888b P 1c50de89a5a845668696dcb28c3badd6 [term 3 FOLLOWER]: Advancing to term 4
I20260129 02:59:56.938326 20405 leader_election.cc:304] T 17d6431cc77a491e8464d68db4081a51 P 197b880fa39d432a9b472845876bfaf0 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 197b880fa39d432a9b472845876bfaf0; no voters: 1c50de89a5a845668696dcb28c3badd6, 974b802141bd4f13beffef2aa03f88f9
I20260129 02:59:56.931816 21133 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "18f30ebaa4174fd28c958e72d0cc7e24" candidate_uuid: "1c50de89a5a845668696dcb28c3badd6" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "7a9daf5d42a84615a6383cfec9a10608"
I20260129 02:59:56.957458 21133 raft_consensus.cc:3060] T 18f30ebaa4174fd28c958e72d0cc7e24 P 7a9daf5d42a84615a6383cfec9a10608 [term 5 FOLLOWER]: Advancing to term 6
I20260129 02:59:56.899670 22878 consensus_queue.cc:1059] T 8055232cf0004d40a85f3bf983fe00a9 P 974b802141bd4f13beffef2aa03f88f9 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 }, Status: INVALID_TERM, Last received: 3.2, Next index: 3, Last known committed idx: 1, Time since last communication: 0.030s
I20260129 02:59:56.947929 21129 raft_consensus.cc:1240] T 75f45538dc704d9a8ec4ff8ad9f3ea5b P 7a9daf5d42a84615a6383cfec9a10608 [term 4 LEADER]: Rejecting Update request from peer 974b802141bd4f13beffef2aa03f88f9 for earlier term 3. Current term is 4. Ops: []
I20260129 02:59:56.952531 22986 consensus_queue.cc:237] T 4f8f3a38f6e24ba4857875181f447442 P 974b802141bd4f13beffef2aa03f88f9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:59:56.953083 22877 raft_consensus.cc:493] T 368953c9102643b2a21f99af36b51905 P 197b880fa39d432a9b472845876bfaf0 [term 6 FOLLOWER]: Starting leader election (detected failure of leader 1c50de89a5a845668696dcb28c3badd6)
I20260129 02:59:56.967641 20585 raft_consensus.cc:2468] T a8479d1fe0bd41e5946186e3a6ca8c7a P 1c50de89a5a845668696dcb28c3badd6 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 974b802141bd4f13beffef2aa03f88f9 in term 5.
I20260129 02:59:56.955252 20509 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a609ac02fae84725ab743f95f51d33ce" candidate_uuid: "7a9daf5d42a84615a6383cfec9a10608" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "974b802141bd4f13beffef2aa03f88f9"
I20260129 02:59:56.939131 23005 raft_consensus.cc:2804] T fcddab6bca684adf8ea032289a00f90e P 1c50de89a5a845668696dcb28c3badd6 [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260129 02:59:56.975446 22877 raft_consensus.cc:3060] T 368953c9102643b2a21f99af36b51905 P 197b880fa39d432a9b472845876bfaf0 [term 6 FOLLOWER]: Advancing to term 7
I20260129 02:59:56.958885 22654 consensus_queue.cc:1059] T c2633a6fc34c44579de348d208b0c01e P 1c50de89a5a845668696dcb28c3badd6 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 }, Status: INVALID_TERM, Last received: 2.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.021s
I20260129 02:59:56.941632 22979 consensus_queue.cc:1048] T 36ff33be666a4c82954731691dbcf327 P 974b802141bd4f13beffef2aa03f88f9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.039s
W20260129 02:59:56.982301 20405 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.211.3:37735, user_credentials={real_user=slave}} blocked reactor thread for 61491.5us
I20260129 02:59:56.983155 20405 leader_election.cc:304] T 18f30ebaa4174fd28c958e72d0cc7e24 P 197b880fa39d432a9b472845876bfaf0 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 197b880fa39d432a9b472845876bfaf0; no voters: 1c50de89a5a845668696dcb28c3badd6, 7a9daf5d42a84615a6383cfec9a10608
I20260129 02:59:56.984485 20405 leader_election.cc:400] T 1ce4ddf5b10e4ae8b193b13b2e5936ee P 197b880fa39d432a9b472845876bfaf0 [CANDIDATE]: Term 5 election: Vote denied by peer 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3:37735) with higher term. Message: Invalid argument: T 1ce4ddf5b10e4ae8b193b13b2e5936ee P 1c50de89a5a845668696dcb28c3badd6 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 197b880fa39d432a9b472845876bfaf0 for earlier term 5. Current term is 6.
I20260129 02:59:56.985374 20405 leader_election.cc:304] T e4d18c35b0a7493e85d0a4b92956a48f P 197b880fa39d432a9b472845876bfaf0 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 197b880fa39d432a9b472845876bfaf0; no voters: 1c50de89a5a845668696dcb28c3badd6, 974b802141bd4f13beffef2aa03f88f9
I20260129 02:59:56.987500 22688 raft_consensus.cc:3060] T 18f30ebaa4174fd28c958e72d0cc7e24 P 197b880fa39d432a9b472845876bfaf0 [term 4 FOLLOWER]: Advancing to term 5
I20260129 02:59:56.989018 20507 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "505580c93c694bbb8ea96fdf9b17ab46" candidate_uuid: "7a9daf5d42a84615a6383cfec9a10608" candidate_term: 6 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "974b802141bd4f13beffef2aa03f88f9" is_pre_election: true
I20260129 02:59:56.989701 20507 raft_consensus.cc:2468] T 505580c93c694bbb8ea96fdf9b17ab46 P 974b802141bd4f13beffef2aa03f88f9 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7a9daf5d42a84615a6383cfec9a10608 in term 5.
I20260129 02:59:56.992287 20512 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "75f45538dc704d9a8ec4ff8ad9f3ea5b" candidate_uuid: "7a9daf5d42a84615a6383cfec9a10608" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "974b802141bd4f13beffef2aa03f88f9" is_pre_election: true
W20260129 02:59:56.993670 22973 log.cc:927] Time spent T c934d31efd3f4e5195605ddc354d14e1 P 974b802141bd4f13beffef2aa03f88f9: Append to log took a long time: real 0.851s user 0.003s sys 0.000s
I20260129 02:59:56.993606 20516 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "add8d4923d1a489e86882c3a25692b95" candidate_uuid: "7a9daf5d42a84615a6383cfec9a10608" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "974b802141bd4f13beffef2aa03f88f9"
I20260129 02:59:56.994489 20516 raft_consensus.cc:2393] T add8d4923d1a489e86882c3a25692b95 P 974b802141bd4f13beffef2aa03f88f9 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 7a9daf5d42a84615a6383cfec9a10608 in current term 4: Already voted for candidate 974b802141bd4f13beffef2aa03f88f9 in this term.
I20260129 02:59:56.995538 22925 raft_consensus.cc:2749] T e4d18c35b0a7493e85d0a4b92956a48f P 197b880fa39d432a9b472845876bfaf0 [term 5 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
W20260129 02:59:56.996968 20556 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.211.2:34517, user_credentials={real_user=slave}} blocked reactor thread for 64572.4us
I20260129 02:59:56.997239 22996 raft_consensus.cc:2749] T a8479d1fe0bd41e5946186e3a6ca8c7a P 1c50de89a5a845668696dcb28c3badd6 [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260129 02:59:56.997735 20556 leader_election.cc:304] T cb0b3f259ebc4310927338a8740e87b8 P 1c50de89a5a845668696dcb28c3badd6 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1c50de89a5a845668696dcb28c3badd6, 974b802141bd4f13beffef2aa03f88f9; no voters: 7a9daf5d42a84615a6383cfec9a10608
I20260129 02:59:57.000448 20511 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4e420f90884f4796997b5ea41e49e411" candidate_uuid: "7a9daf5d42a84615a6383cfec9a10608" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "974b802141bd4f13beffef2aa03f88f9" is_pre_election: true
I20260129 02:59:57.001098 20511 raft_consensus.cc:2468] T 4e420f90884f4796997b5ea41e49e411 P 974b802141bd4f13beffef2aa03f88f9 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7a9daf5d42a84615a6383cfec9a10608 in term 5.
I20260129 02:59:57.002110 22881 raft_consensus.cc:3060] T e33a417cf9c54cf5af73a4ee3aac4d3a P 1c50de89a5a845668696dcb28c3badd6 [term 5 FOLLOWER]: Advancing to term 6
I20260129 02:59:57.002697 22640 consensus_queue.cc:1048] T 505580c93c694bbb8ea96fdf9b17ab46 P 1c50de89a5a845668696dcb28c3badd6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 5, Time since last communication: 0.000s
I20260129 02:59:57.002926 19276 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 3
I20260129 02:59:57.002234 22976 raft_consensus.cc:2804] T cb0b3f259ebc4310927338a8740e87b8 P 1c50de89a5a845668696dcb28c3badd6 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260129 02:59:57.006213 20556 leader_election.cc:304] T 3906b080a5b34a74b8b64128635e8997 P 1c50de89a5a845668696dcb28c3badd6 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1c50de89a5a845668696dcb28c3badd6; no voters: 197b880fa39d432a9b472845876bfaf0, 974b802141bd4f13beffef2aa03f88f9
I20260129 02:59:57.009115 20582 raft_consensus.cc:2468] T 96609ac68bdf4993a12e1c23355ca013 P 1c50de89a5a845668696dcb28c3badd6 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7a9daf5d42a84615a6383cfec9a10608 in term 6.
I20260129 02:59:57.002126 20510 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57600 (request call id 1817) took 1244 ms. Trace:
I20260129 02:59:57.020115 20509 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "96609ac68bdf4993a12e1c23355ca013" candidate_uuid: "7a9daf5d42a84615a6383cfec9a10608" candidate_term: 6 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "974b802141bd4f13beffef2aa03f88f9"
I20260129 02:59:57.021000 20509 raft_consensus.cc:3060] T 96609ac68bdf4993a12e1c23355ca013 P 974b802141bd4f13beffef2aa03f88f9 [term 5 FOLLOWER]: Advancing to term 6
I20260129 02:59:56.972208 22851 raft_consensus.cc:2749] T 17d6431cc77a491e8464d68db4081a51 P 197b880fa39d432a9b472845876bfaf0 [term 4 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260129 02:59:57.003690 22688 raft_consensus.cc:2749] T 18f30ebaa4174fd28c958e72d0cc7e24 P 197b880fa39d432a9b472845876bfaf0 [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260129 02:59:56.968379 21127 raft_consensus.cc:2468] T add8d4923d1a489e86882c3a25692b95 P 7a9daf5d42a84615a6383cfec9a10608 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1c50de89a5a845668696dcb28c3badd6 in term 4.
I20260129 02:59:57.045718 20433 raft_consensus.cc:1275] T 75be4bd6bd45474795fd472eff2953ac P 197b880fa39d432a9b472845876bfaf0 [term 9 FOLLOWER]: Refusing update from remote peer 1c50de89a5a845668696dcb28c3badd6: Log matching property violated. Preceding OpId in replica: term: 8 index: 6. Preceding OpId from leader: term: 9 index: 6. (term mismatch)
W20260129 02:59:57.019908 22969 consensus_meta.cc:287] T 00e79d8f25a34676b1d866eb2068a2b2 P 974b802141bd4f13beffef2aa03f88f9: Time spent flushing consensus metadata: real 0.542s user 0.005s sys 0.004s
W20260129 02:59:57.046453 20405 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.211.3:37735, user_credentials={real_user=slave}} blocked reactor thread for 47051.3us
I20260129 02:59:57.011557 20510 rpcz_store.cc:276] 0129 02:59:55.757612 (+ 0us) service_pool.cc:168] Inserting onto call queue
0129 02:59:55.757789 (+ 177us) service_pool.cc:225] Handling call
0129 02:59:55.785700 (+ 27911us) raft_consensus.cc:1471] Updating replica for 2 ops
0129 02:59:55.825477 (+ 39777us) raft_consensus.cc:1534] Early marking committed up to index 0
0129 02:59:55.825515 (+ 38us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0129 02:59:55.893191 (+ 67676us) log.cc:844] Serialized 52 byte log entry
0129 02:59:56.017975 (+124784us) raft_consensus.cc:1645] Marking committed up to 6
0129 02:59:56.052817 (+ 34842us) log.cc:844] Serialized 14 byte log entry
0129 02:59:56.285539 (+232722us) raft_consensus.cc:1697] Filling consensus response to leader.
0129 02:59:56.285703 (+ 164us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0129 02:59:56.993092 (+707389us) spinlock_profiling.cc:243] Waited 6.89 ms on lock 0x7b440002304c. stack: 00007f8fe044e373 00007f8fe044e270 00007f8fdfda0574 00007f8fe9a5c6b4 00007f8fe9a5c649 00007f8fe02c08b1 00007f8fe038f005 00007f8fe038fa22 00007f8fe30d957b 00007f8fe30d4e4f 00007f8fe30c8c0e 00007f8fe30c7f56 00007f8fe30c7b78 00007f8fe30d15c1 00007f8fe30d1579 00007f8fe30d1509
0129 02:59:57.001824 (+ 8732us) raft_consensus.cc:1685] finished
0129 02:59:57.001933 (+ 109us) raft_consensus.cc:1691] UpdateReplicas() finished
0129 02:59:57.002069 (+ 136us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":7301,"prepare.queue_time_us":34922,"prepare.run_cpu_time_us":65,"prepare.run_wall_time_us":66,"raft.queue_time_us":287939,"raft.run_cpu_time_us":328,"raft.run_wall_time_us":35354,"spinlock_wait_cycles":15843072,"thread_start_us":295521,"threads_started":2,"wal-append.queue_time_us":172294}
I20260129 02:59:57.005043 23005 raft_consensus.cc:493] T fcddab6bca684adf8ea032289a00f90e P 1c50de89a5a845668696dcb28c3badd6 [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260129 02:59:57.051617 23005 raft_consensus.cc:3060] T fcddab6bca684adf8ea032289a00f90e P 1c50de89a5a845668696dcb28c3badd6 [term 8 FOLLOWER]: Advancing to term 9
I20260129 02:59:57.061808 23005 raft_consensus.cc:515] T fcddab6bca684adf8ea032289a00f90e P 1c50de89a5a845668696dcb28c3badd6 [term 9 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:59:57.064462 23005 leader_election.cc:290] T fcddab6bca684adf8ea032289a00f90e P 1c50de89a5a845668696dcb28c3badd6 [CANDIDATE]: Term 9 election: Requested vote from peers 197b880fa39d432a9b472845876bfaf0 (127.18.211.1:42027), 974b802141bd4f13beffef2aa03f88f9 (127.18.211.2:34517)
I20260129 02:59:57.069851 20440 raft_consensus.cc:1217] T 1ce4ddf5b10e4ae8b193b13b2e5936ee P 197b880fa39d432a9b472845876bfaf0 [term 7 FOLLOWER]: Deduplicated request from leader. Original: 1.1->[2.2-6.4] Dedup: 2.2->[4.3-6.4]
I20260129 02:59:57.047585 20480 leader_election.cc:304] T 4e420f90884f4796997b5ea41e49e411 P 974b802141bd4f13beffef2aa03f88f9 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 7a9daf5d42a84615a6383cfec9a10608, 974b802141bd4f13beffef2aa03f88f9; no voters: 197b880fa39d432a9b472845876bfaf0
I20260129 02:59:57.070502 20440 raft_consensus.cc:1240] T 1ce4ddf5b10e4ae8b193b13b2e5936ee P 197b880fa39d432a9b472845876bfaf0 [term 7 FOLLOWER]: Rejecting Update request from peer 1c50de89a5a845668696dcb28c3badd6 for earlier term 6. Current term is 7. Ops: [2.2-6.4]
I20260129 02:59:57.048564 20433 pending_rounds.cc:85] T 75be4bd6bd45474795fd472eff2953ac P 197b880fa39d432a9b472845876bfaf0: Aborting all ops after (but not including) 5
I20260129 02:59:57.072378 20433 pending_rounds.cc:107] T 75be4bd6bd45474795fd472eff2953ac P 197b880fa39d432a9b472845876bfaf0: Aborting uncommitted NO_OP operation due to leader change: 8.6
I20260129 02:59:57.072726 20433 raft_consensus.cc:2889] T 75be4bd6bd45474795fd472eff2953ac P 197b880fa39d432a9b472845876bfaf0 [term 9 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260129 02:59:57.019701 22976 raft_consensus.cc:493] T cb0b3f259ebc4310927338a8740e87b8 P 1c50de89a5a845668696dcb28c3badd6 [term 5 FOLLOWER]: Starting leader election (detected failure of leader 7a9daf5d42a84615a6383cfec9a10608)
I20260129 02:59:57.077601 22979 raft_consensus.cc:2804] T 4e420f90884f4796997b5ea41e49e411 P 974b802141bd4f13beffef2aa03f88f9 [term 5 FOLLOWER]: Leader election won for term 5
I20260129 02:59:57.078256 20439 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4e3805ffa70242e8ade66da260dddf66" candidate_uuid: "1c50de89a5a845668696dcb28c3badd6" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "197b880fa39d432a9b472845876bfaf0"
I20260129 02:59:57.078603 20480 leader_election.cc:304] T 4f5a77e94fef4b16a2ba7b067c8b144a P 974b802141bd4f13beffef2aa03f88f9 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7a9daf5d42a84615a6383cfec9a10608, 974b802141bd4f13beffef2aa03f88f9; no voters:
I20260129 02:59:57.100889 20440 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e7d83d4ccf684e76acb29a0dc844e1b8" candidate_uuid: "1c50de89a5a845668696dcb28c3badd6" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "197b880fa39d432a9b472845876bfaf0"
I20260129 02:59:57.105438 20439 raft_consensus.cc:3060] T 4e3805ffa70242e8ade66da260dddf66 P 197b880fa39d432a9b472845876bfaf0 [term 4 FOLLOWER]: Advancing to term 5
W20260129 02:59:57.108301 20404 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260129 02:59:57.050334 22922 leader_election.cc:290] T cb4727386a3e48aca5d71cc6c611c197 P 1c50de89a5a845668696dcb28c3badd6 [CANDIDATE]: Term 4 election: Requested vote from peers 197b880fa39d432a9b472845876bfaf0 (127.18.211.1:42027), 974b802141bd4f13beffef2aa03f88f9 (127.18.211.2:34517)
I20260129 02:59:57.150310 20433 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "18f30ebaa4174fd28c958e72d0cc7e24" candidate_uuid: "1c50de89a5a845668696dcb28c3badd6" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "197b880fa39d432a9b472845876bfaf0"
I20260129 02:59:57.086323 22976 raft_consensus.cc:3060] T cb0b3f259ebc4310927338a8740e87b8 P 1c50de89a5a845668696dcb28c3badd6 [term 5 FOLLOWER]: Advancing to term 6
I20260129 02:59:57.154538 20440 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "913fc7d751ed4b7388d889aac30a5f9b" candidate_uuid: "1c50de89a5a845668696dcb28c3badd6" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "197b880fa39d432a9b472845876bfaf0"
W20260129 02:59:57.154102 20554 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.211.4:40439, user_credentials={real_user=slave}} blocked reactor thread for 78390.4us
I20260129 02:59:57.044711 22971 raft_consensus.cc:515] T b189f08e16a74a11980cc55d99359a0a P 1c50de89a5a845668696dcb28c3badd6 [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
W20260129 02:59:57.178304 20556 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.211.2:34517, user_credentials={real_user=slave}} blocked reactor thread for 172277us
W20260129 02:59:57.179598 20555 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260129 02:59:57.181376 20436 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "926a250fa5834e88b96ccecbefbf5086" candidate_uuid: "1c50de89a5a845668696dcb28c3badd6" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "197b880fa39d432a9b472845876bfaf0" is_pre_election: true
W20260129 02:59:57.182502 20553 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.211.1:42027, user_credentials={real_user=slave}} blocked reactor thread for 68246.3us
I20260129 02:59:57.101516 22656 raft_consensus.cc:2749] T 3906b080a5b34a74b8b64128635e8997 P 1c50de89a5a845668696dcb28c3badd6 [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260129 02:59:57.156654 20554 leader_election.cc:304] T add8d4923d1a489e86882c3a25692b95 P 1c50de89a5a845668696dcb28c3badd6 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1c50de89a5a845668696dcb28c3badd6, 7a9daf5d42a84615a6383cfec9a10608; no voters:
I20260129 02:59:57.113101 22979 raft_consensus.cc:697] T 4e420f90884f4796997b5ea41e49e411 P 974b802141bd4f13beffef2aa03f88f9 [term 5 LEADER]: Becoming Leader. State: Replica: 974b802141bd4f13beffef2aa03f88f9, State: Running, Role: LEADER
I20260129 02:59:57.186455 20440 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fcddab6bca684adf8ea032289a00f90e" candidate_uuid: "1c50de89a5a845668696dcb28c3badd6" candidate_term: 9 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "197b880fa39d432a9b472845876bfaf0"
I20260129 02:59:57.152668 22184 raft_consensus.cc:2804] T 4f5a77e94fef4b16a2ba7b067c8b144a P 974b802141bd4f13beffef2aa03f88f9 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260129 02:59:57.188555 20436 raft_consensus.cc:2410] T 926a250fa5834e88b96ccecbefbf5086 P 197b880fa39d432a9b472845876bfaf0 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1c50de89a5a845668696dcb28c3badd6 for term 7 because replica has last-logged OpId of term: 5 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 4 index: 3.
I20260129 02:59:57.198915 20556 leader_election.cc:304] T a5a05fbcbbcf4916bbc01fd1a5b5e90d P 1c50de89a5a845668696dcb28c3badd6 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1c50de89a5a845668696dcb28c3badd6; no voters: 197b880fa39d432a9b472845876bfaf0, 974b802141bd4f13beffef2aa03f88f9
I20260129 02:59:57.202311 20436 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "17d6431cc77a491e8464d68db4081a51" candidate_uuid: "1c50de89a5a845668696dcb28c3badd6" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "197b880fa39d432a9b472845876bfaf0" is_pre_election: true
I20260129 02:59:57.216751 20436 raft_consensus.cc:2393] T 17d6431cc77a491e8464d68db4081a51 P 197b880fa39d432a9b472845876bfaf0 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1c50de89a5a845668696dcb28c3badd6 in current term 4: Already voted for candidate 974b802141bd4f13beffef2aa03f88f9 in this term.
I20260129 02:59:57.203024 19276 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20260129 02:59:57.205791 22922 consensus_queue.cc:1059] T 1ce4ddf5b10e4ae8b193b13b2e5936ee P 1c50de89a5a845668696dcb28c3badd6 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 }, Status: INVALID_TERM, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260129 02:59:57.219612 22922 consensus_queue.cc:1243] T 1ce4ddf5b10e4ae8b193b13b2e5936ee P 1c50de89a5a845668696dcb28c3badd6 [LEADER]: Peer 197b880fa39d432a9b472845876bfaf0 log is divergent from this leader: its last log entry 5.3 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 1
I20260129 02:59:57.220254 22922 consensus_queue.cc:1048] T 75be4bd6bd45474795fd472eff2953ac P 1c50de89a5a845668696dcb28c3badd6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260129 02:59:57.221060 22961 raft_consensus.cc:3055] T 8055232cf0004d40a85f3bf983fe00a9 P 974b802141bd4f13beffef2aa03f88f9 [term 3 LEADER]: Stepping down as leader of term 3
I20260129 02:59:57.221459 22961 raft_consensus.cc:740] T 8055232cf0004d40a85f3bf983fe00a9 P 974b802141bd4f13beffef2aa03f88f9 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 974b802141bd4f13beffef2aa03f88f9, State: Running, Role: LEADER
I20260129 02:59:57.222070 22922 raft_consensus.cc:3055] T 1ce4ddf5b10e4ae8b193b13b2e5936ee P 1c50de89a5a845668696dcb28c3badd6 [term 6 LEADER]: Stepping down as leader of term 6
I20260129 02:59:57.209141 22967 raft_consensus.cc:515] T 0678b8b468bd43cdbc3e485900844821 P 974b802141bd4f13beffef2aa03f88f9 [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } }
I20260129 02:59:57.222627 22922 raft_consensus.cc:740] T 1ce4ddf5b10e4ae8b193b13b2e5936ee P 1c50de89a5a845668696dcb28c3badd6 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 1c50de89a5a845668696dcb28c3badd6, State: Running, Role: LEADER
I20260129 02:59:57.222378 22961 consensus_queue.cc:260] T 8055232cf0004d40a85f3bf983fe00a9 P 974b802141bd4f13beffef2aa03f88f9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:59:57.223495 22922 consensus_queue.cc:260] T 1ce4ddf5b10e4ae8b193b13b2e5936ee P 1c50de89a5a845668696dcb28c3badd6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 6.4, Last appended by leader: 4, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:59:57.224189 22967 leader_election.cc:290] T 0678b8b468bd43cdbc3e485900844821 P 974b802141bd4f13beffef2aa03f88f9 [CANDIDATE]: Term 7 election: Requested vote from peers 197b880fa39d432a9b472845876bfaf0 (127.18.211.1:42027), 7a9daf5d42a84615a6383cfec9a10608 (127.18.211.4:40439)
I20260129 02:59:57.211459 21132 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "75b0b6e82ddc4f92b1fe1d94f9a0b73b" candidate_uuid: "974b802141bd4f13beffef2aa03f88f9" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "7a9daf5d42a84615a6383cfec9a10608" is_pre_election: true
I20260129 02:59:57.220417 19276 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
W20260129 02:59:57.208385 23014 log.cc:927] Time spent T 2ac659d732314e1288849f08fe5e074e P 197b880fa39d432a9b472845876bfaf0: Append to log took a long time: real 0.054s user 0.000s sys 0.000s
W20260129 02:59:57.207516 23016 log.cc:927] Time spent T 368953c9102643b2a21f99af36b51905 P 974b802141bd4f13beffef2aa03f88f9: Append to log took a long time: real 0.053s user 0.000s sys 0.000s
I20260129 02:59:57.075199 22969 raft_consensus.cc:515] T 00e79d8f25a34676b1d866eb2068a2b2 P 974b802141bd4f13beffef2aa03f88f9 [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } }
I20260129 02:59:57.105811 22987 consensus_queue.cc:1059] T 2ac659d732314e1288849f08fe5e074e P 974b802141bd4f13beffef2aa03f88f9 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false }, Status: INVALID_TERM, Last received: 3.5, Next index: 6, Last known committed idx: 4, Time since last communication: 0.093s
W20260129 02:59:57.236680 20407 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 1.466s user 0.007s sys 0.015s
W20260129 02:59:57.260500 20407 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 1.495s user 0.011s sys 0.020s
I20260129 02:59:57.229516 20440 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "cb4727386a3e48aca5d71cc6c611c197" candidate_uuid: "1c50de89a5a845668696dcb28c3badd6" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "197b880fa39d432a9b472845876bfaf0"
W20260129 02:59:57.242777 20404 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260129 02:59:57.224817 22922 raft_consensus.cc:3060] T 1ce4ddf5b10e4ae8b193b13b2e5936ee P 1c50de89a5a845668696dcb28c3badd6 [term 6 FOLLOWER]: Advancing to term 7
W20260129 02:59:57.262071 20554 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.211.4:40439, user_credentials={real_user=slave}} blocked reactor thread for 105767us
I20260129 02:59:57.266877 20509 raft_consensus.cc:2410] T 96609ac68bdf4993a12e1c23355ca013 P 974b802141bd4f13beffef2aa03f88f9 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 7a9daf5d42a84615a6383cfec9a10608 for term 6 because replica has last-logged OpId of term: 3 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 4.
I20260129 02:59:57.199764 20480 leader_election.cc:304] T 75be4bd6bd45474795fd472eff2953ac P 974b802141bd4f13beffef2aa03f88f9 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 974b802141bd4f13beffef2aa03f88f9; no voters: 197b880fa39d432a9b472845876bfaf0, 1c50de89a5a845668696dcb28c3badd6
I20260129 02:59:57.264288 21102 leader_election.cc:304] T 87d5fe48a44a408aaf03e894a67ecb2f P 7a9daf5d42a84615a6383cfec9a10608 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7a9daf5d42a84615a6383cfec9a10608; no voters: 197b880fa39d432a9b472845876bfaf0, 974b802141bd4f13beffef2aa03f88f9
I20260129 02:59:57.265784 22891 raft_consensus.cc:2749] T a5a05fbcbbcf4916bbc01fd1a5b5e90d P 1c50de89a5a845668696dcb28c3badd6 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
W20260129 02:59:57.263252 20407 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.211.2:34517, user_credentials={real_user=slave}} blocked reactor thread for 1.4975e+06us
I20260129 02:59:57.247579 21133 raft_consensus.cc:2468] T 18f30ebaa4174fd28c958e72d0cc7e24 P 7a9daf5d42a84615a6383cfec9a10608 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1c50de89a5a845668696dcb28c3badd6 in term 6.
I20260129 02:59:57.223645 22961 raft_consensus.cc:3060] T 8055232cf0004d40a85f3bf983fe00a9 P 974b802141bd4f13beffef2aa03f88f9 [term 3 FOLLOWER]: Advancing to term 4
W20260129 02:59:57.235847 22963 consensus_meta.cc:287] T cb4727386a3e48aca5d71cc6c611c197 P 974b802141bd4f13beffef2aa03f88f9: Time spent flushing consensus metadata: real 0.907s user 0.011s sys 0.000s
W20260129 02:59:57.225521 23013 log.cc:927] Time spent T 17d6431cc77a491e8464d68db4081a51 P 974b802141bd4f13beffef2aa03f88f9: Append to log took a long time: real 0.111s user 0.000s sys 0.001s
W20260129 02:59:57.262897 20556 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.211.2:34517, user_credentials={real_user=slave}} blocked reactor thread for 64129.5us
I20260129 02:59:57.269872 20585 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "75b0b6e82ddc4f92b1fe1d94f9a0b73b" candidate_uuid: "974b802141bd4f13beffef2aa03f88f9" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "1c50de89a5a845668696dcb28c3badd6" is_pre_election: true
I20260129 02:59:57.230624 22691 raft_consensus.cc:493] T 0f20af7fbe5d4db89a6f17948d98912c P 197b880fa39d432a9b472845876bfaf0 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260129 02:59:57.304718 23024 log.cc:927] Time spent T a1a39dcd328e4268ae41b988c66b4a75 P 1c50de89a5a845668696dcb28c3badd6: Append to log took a long time: real 0.104s user 0.000s sys 0.000s
I20260129 02:59:57.304244 22691 raft_consensus.cc:515] T 0f20af7fbe5d4db89a6f17948d98912c P 197b880fa39d432a9b472845876bfaf0 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:59:57.306993 22691 leader_election.cc:290] T 0f20af7fbe5d4db89a6f17948d98912c P 197b880fa39d432a9b472845876bfaf0 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 974b802141bd4f13beffef2aa03f88f9 (127.18.211.2:34517), 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3:37735)
W20260129 02:59:57.233017 22231 consensus_meta.cc:287] T a0fcc58d6fb6473cafbd9d34d5195269 P 197b880fa39d432a9b472845876bfaf0: Time spent flushing consensus metadata: real 0.756s user 0.012s sys 0.004s
I20260129 02:59:57.307958 22231 raft_consensus.cc:515] T a0fcc58d6fb6473cafbd9d34d5195269 P 197b880fa39d432a9b472845876bfaf0 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:59:57.310539 22231 leader_election.cc:290] T a0fcc58d6fb6473cafbd9d34d5195269 P 197b880fa39d432a9b472845876bfaf0 [CANDIDATE]: Term 4 election: Requested vote from peers 974b802141bd4f13beffef2aa03f88f9 (127.18.211.2:34517), 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3:37735)
I20260129 02:59:57.274019 21099 leader_election.cc:304] T 42d27a28d89b409e8807ec0efd97af69 P 7a9daf5d42a84615a6383cfec9a10608 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 197b880fa39d432a9b472845876bfaf0, 7a9daf5d42a84615a6383cfec9a10608; no voters: 1c50de89a5a845668696dcb28c3badd6
I20260129 02:59:57.312865 21099 leader_election.cc:304] T dc2f4cb752cf4d59b5f6bc689f5d8104 P 7a9daf5d42a84615a6383cfec9a10608 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7a9daf5d42a84615a6383cfec9a10608; no voters: 197b880fa39d432a9b472845876bfaf0, 1c50de89a5a845668696dcb28c3badd6
I20260129 02:59:57.274732 20440 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0678b8b468bd43cdbc3e485900844821" candidate_uuid: "974b802141bd4f13beffef2aa03f88f9" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "197b880fa39d432a9b472845876bfaf0"
I20260129 02:59:57.314947 20440 raft_consensus.cc:3060] T 0678b8b468bd43cdbc3e485900844821 P 197b880fa39d432a9b472845876bfaf0 [term 6 FOLLOWER]: Advancing to term 7
W20260129 02:59:57.275476 20406 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260129 02:59:57.271126 22983 raft_consensus.cc:493] T e7d83d4ccf684e76acb29a0dc844e1b8 P 974b802141bd4f13beffef2aa03f88f9 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260129 02:59:57.270844 22992 consensus_queue.cc:1048] T a644c02b16af47679824ccd060ac10fd P 7a9daf5d42a84615a6383cfec9a10608 [LEADER]: Connected to new peer: Peer: permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 4, Time since last communication: 0.011s
I20260129 02:59:57.277585 22933 raft_consensus.cc:2749] T 87d5fe48a44a408aaf03e894a67ecb2f P 7a9daf5d42a84615a6383cfec9a10608 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260129 02:59:57.279948 20587 raft_consensus.cc:1275] T 36ff33be666a4c82954731691dbcf327 P 1c50de89a5a845668696dcb28c3badd6 [term 3 FOLLOWER]: Refusing update from remote peer 974b802141bd4f13beffef2aa03f88f9: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260129 02:59:57.280251 20438 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b189f08e16a74a11980cc55d99359a0a" candidate_uuid: "1c50de89a5a845668696dcb28c3badd6" candidate_term: 8 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "197b880fa39d432a9b472845876bfaf0"
I20260129 02:59:57.328855 20438 raft_consensus.cc:3060] T b189f08e16a74a11980cc55d99359a0a P 197b880fa39d432a9b472845876bfaf0 [term 7 FOLLOWER]: Advancing to term 8
I20260129 02:59:57.280535 22962 leader_election.cc:290] T c934d31efd3f4e5195605ddc354d14e1 P 974b802141bd4f13beffef2aa03f88f9 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 197b880fa39d432a9b472845876bfaf0 (127.18.211.1:42027), 7a9daf5d42a84615a6383cfec9a10608 (127.18.211.4:40439)
I20260129 02:59:57.262112 22986 consensus_queue.cc:1048] T fcddab6bca684adf8ea032289a00f90e P 974b802141bd4f13beffef2aa03f88f9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.070s
I20260129 02:59:57.332499 20554 leader_election.cc:304] T 18f30ebaa4174fd28c958e72d0cc7e24 P 1c50de89a5a845668696dcb28c3badd6 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1c50de89a5a845668696dcb28c3badd6, 7a9daf5d42a84615a6383cfec9a10608; no voters:
I20260129 02:59:57.281342 20514 raft_consensus.cc:1240] T 00e79d8f25a34676b1d866eb2068a2b2 P 974b802141bd4f13beffef2aa03f88f9 [term 7 FOLLOWER]: Rejecting Update request from peer 7a9daf5d42a84615a6383cfec9a10608 for earlier term 6. Current term is 7. Ops: [6.8-6.8]
I20260129 02:59:57.199806 22184 raft_consensus.cc:493] T 4f5a77e94fef4b16a2ba7b067c8b144a P 974b802141bd4f13beffef2aa03f88f9 [term 3 FOLLOWER]: Starting leader election (detected failure of leader 1c50de89a5a845668696dcb28c3badd6)
I20260129 02:59:57.333971 22891 raft_consensus.cc:2804] T 18f30ebaa4174fd28c958e72d0cc7e24 P 1c50de89a5a845668696dcb28c3badd6 [term 6 FOLLOWER]: Leader election won for term 6
I20260129 02:59:57.334297 22184 raft_consensus.cc:3060] T 4f5a77e94fef4b16a2ba7b067c8b144a P 974b802141bd4f13beffef2aa03f88f9 [term 3 FOLLOWER]: Advancing to term 4
I20260129 02:59:57.334506 22891 raft_consensus.cc:697] T 18f30ebaa4174fd28c958e72d0cc7e24 P 1c50de89a5a845668696dcb28c3badd6 [term 6 LEADER]: Becoming Leader. State: Replica: 1c50de89a5a845668696dcb28c3badd6, State: Running, Role: LEADER
I20260129 02:59:57.338403 22976 raft_consensus.cc:515] T cb0b3f259ebc4310927338a8740e87b8 P 1c50de89a5a845668696dcb28c3badd6 [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } }
I20260129 02:59:57.339570 20590 raft_consensus.cc:1240] T cb0b3f259ebc4310927338a8740e87b8 P 1c50de89a5a845668696dcb28c3badd6 [term 6 FOLLOWER]: Rejecting Update request from peer 7a9daf5d42a84615a6383cfec9a10608 for earlier term 5. Current term is 6. Ops: []
I20260129 02:59:57.281459 20589 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3e2d15f8640b407195210c476e09df91" candidate_uuid: "974b802141bd4f13beffef2aa03f88f9" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "1c50de89a5a845668696dcb28c3badd6" is_pre_election: true
I20260129 02:59:57.285117 20509 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c2633a6fc34c44579de348d208b0c01e" candidate_uuid: "197b880fa39d432a9b472845876bfaf0" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "974b802141bd4f13beffef2aa03f88f9"
I20260129 02:59:57.342108 20509 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "724a236524cb4e7ba3428880681d6cc2" candidate_uuid: "197b880fa39d432a9b472845876bfaf0" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "974b802141bd4f13beffef2aa03f88f9" is_pre_election: true
I20260129 02:59:57.342723 20509 raft_consensus.cc:2468] T 724a236524cb4e7ba3428880681d6cc2 P 974b802141bd4f13beffef2aa03f88f9 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 197b880fa39d432a9b472845876bfaf0 in term 4.
I20260129 02:59:57.343492 20509 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1ce4ddf5b10e4ae8b193b13b2e5936ee" candidate_uuid: "197b880fa39d432a9b472845876bfaf0" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "974b802141bd4f13beffef2aa03f88f9"
I20260129 02:59:57.345187 22884 raft_consensus.cc:515] T 7d9494f5584d4d5db8b9adf1c8fc888b P 1c50de89a5a845668696dcb28c3badd6 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:59:57.290805 20556 leader_election.cc:304] T a0fcc58d6fb6473cafbd9d34d5195269 P 1c50de89a5a845668696dcb28c3badd6 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1c50de89a5a845668696dcb28c3badd6; no voters: 197b880fa39d432a9b472845876bfaf0, 974b802141bd4f13beffef2aa03f88f9
I20260129 02:59:57.291329 20433 raft_consensus.cc:3060] T 18f30ebaa4174fd28c958e72d0cc7e24 P 197b880fa39d432a9b472845876bfaf0 [term 5 FOLLOWER]: Advancing to term 6
I20260129 02:59:57.346235 20509 raft_consensus.cc:3060] T 1ce4ddf5b10e4ae8b193b13b2e5936ee P 974b802141bd4f13beffef2aa03f88f9 [term 6 FOLLOWER]: Advancing to term 7
W20260129 02:59:57.346654 20556 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.211.2:34517, user_credentials={real_user=slave}} blocked reactor thread for 55932us
I20260129 02:59:57.347249 22884 leader_election.cc:290] T 7d9494f5584d4d5db8b9adf1c8fc888b P 1c50de89a5a845668696dcb28c3badd6 [CANDIDATE]: Term 4 election: Requested vote from peers 197b880fa39d432a9b472845876bfaf0 (127.18.211.1:42027), 974b802141bd4f13beffef2aa03f88f9 (127.18.211.2:34517)
I20260129 02:59:57.323060 22831 raft_consensus.cc:2804] T 42d27a28d89b409e8807ec0efd97af69 P 7a9daf5d42a84615a6383cfec9a10608 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260129 02:59:57.348309 22831 raft_consensus.cc:493] T 42d27a28d89b409e8807ec0efd97af69 P 7a9daf5d42a84615a6383cfec9a10608 [term 6 FOLLOWER]: Starting leader election (detected failure of leader 1c50de89a5a845668696dcb28c3badd6)
I20260129 02:59:57.348670 22831 raft_consensus.cc:3060] T 42d27a28d89b409e8807ec0efd97af69 P 7a9daf5d42a84615a6383cfec9a10608 [term 6 FOLLOWER]: Advancing to term 7
I20260129 02:59:57.225082 22971 leader_election.cc:290] T b189f08e16a74a11980cc55d99359a0a P 1c50de89a5a845668696dcb28c3badd6 [CANDIDATE]: Term 8 election: Requested vote from peers 197b880fa39d432a9b472845876bfaf0 (127.18.211.1:42027), 974b802141bd4f13beffef2aa03f88f9 (127.18.211.2:34517)
I20260129 02:59:57.350039 22971 consensus_queue.cc:1048] T 3e2d15f8640b407195210c476e09df91 P 1c50de89a5a845668696dcb28c3badd6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260129 02:59:57.291738 20405 leader_election.cc:304] T 4b1876f9a70b49edb3aa74105e66d3b3 P 197b880fa39d432a9b472845876bfaf0 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 197b880fa39d432a9b472845876bfaf0, 1c50de89a5a845668696dcb28c3badd6; no voters: 974b802141bd4f13beffef2aa03f88f9
W20260129 02:59:57.291885 20407 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260129 02:59:57.293102 20584 raft_consensus.cc:1275] T 75f45538dc704d9a8ec4ff8ad9f3ea5b P 1c50de89a5a845668696dcb28c3badd6 [term 4 FOLLOWER]: Refusing update from remote peer 7a9daf5d42a84615a6383cfec9a10608: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
W20260129 02:59:57.294234 22645 consensus_meta.cc:287] T c891d2bff4ce4568861bbc992e91fbb0 P 197b880fa39d432a9b472845876bfaf0: Time spent flushing consensus metadata: real 0.536s user 0.009s sys 0.000s
I20260129 02:59:57.371599 22645 raft_consensus.cc:515] T c891d2bff4ce4568861bbc992e91fbb0 P 197b880fa39d432a9b472845876bfaf0 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:59:57.372887 20441 raft_consensus.cc:1240] T c891d2bff4ce4568861bbc992e91fbb0 P 197b880fa39d432a9b472845876bfaf0 [term 7 FOLLOWER]: Rejecting Update request from peer 974b802141bd4f13beffef2aa03f88f9 for earlier term 6. Current term is 7. Ops: []
I20260129 02:59:57.373876 22645 leader_election.cc:290] T c891d2bff4ce4568861bbc992e91fbb0 P 197b880fa39d432a9b472845876bfaf0 [CANDIDATE]: Term 7 election: Requested vote from peers 974b802141bd4f13beffef2aa03f88f9 (127.18.211.2:34517), 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3:37735)
I20260129 02:59:57.294785 20511 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4408006eb5174de1896b037cf3f18457" candidate_uuid: "197b880fa39d432a9b472845876bfaf0" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "974b802141bd4f13beffef2aa03f88f9" is_pre_election: true
I20260129 02:59:57.228963 22979 consensus_queue.cc:237] T 4e420f90884f4796997b5ea41e49e411 P 974b802141bd4f13beffef2aa03f88f9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } }
I20260129 02:59:57.294804 20510 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "8055232cf0004d40a85f3bf983fe00a9" candidate_uuid: "197b880fa39d432a9b472845876bfaf0" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "974b802141bd4f13beffef2aa03f88f9"
W20260129 02:59:57.318693 20553 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.211.1:42027, user_credentials={real_user=slave}} blocked reactor thread for 47448.8us
I20260129 02:59:57.319581 20434 raft_consensus.cc:1240] T a0fcc58d6fb6473cafbd9d34d5195269 P 197b880fa39d432a9b472845876bfaf0 [term 4 FOLLOWER]: Rejecting Update request from peer 974b802141bd4f13beffef2aa03f88f9 for earlier term 3. Current term is 4. Ops: []
I20260129 02:59:57.239171 22877 raft_consensus.cc:515] T 368953c9102643b2a21f99af36b51905 P 197b880fa39d432a9b472845876bfaf0 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:59:57.323387 22932 consensus_queue.cc:1048] T 3ccac82a01a2445fa5b0ce6c0fcf2295 P 7a9daf5d42a84615a6383cfec9a10608 [LEADER]: Connected to new peer: Peer: permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.021s
I20260129 02:59:57.270152 22977 raft_consensus.cc:2804] T add8d4923d1a489e86882c3a25692b95 P 1c50de89a5a845668696dcb28c3badd6 [term 4 FOLLOWER]: Leader pre-election won for term 5
W20260129 02:59:57.319720 23016 log.cc:927] Time spent T 368953c9102643b2a21f99af36b51905 P 974b802141bd4f13beffef2aa03f88f9: Append to log took a long time: real 0.053s user 0.000s sys 0.000s
I20260129 02:59:57.320582 20439 raft_consensus.cc:2468] T 4e3805ffa70242e8ade66da260dddf66 P 197b880fa39d432a9b472845876bfaf0 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1c50de89a5a845668696dcb28c3badd6 in term 5.
I20260129 02:59:57.281023 20586 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0340229d521b432580ae926f6062e3f7" candidate_uuid: "974b802141bd4f13beffef2aa03f88f9" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "1c50de89a5a845668696dcb28c3badd6"
I20260129 02:59:57.347726 20556 leader_election.cc:304] T e4d18c35b0a7493e85d0a4b92956a48f P 1c50de89a5a845668696dcb28c3badd6 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1c50de89a5a845668696dcb28c3badd6; no voters: 197b880fa39d432a9b472845876bfaf0, 974b802141bd4f13beffef2aa03f88f9
I20260129 02:59:57.385325 20407 leader_election.cc:304] T 724a236524cb4e7ba3428880681d6cc2 P 197b880fa39d432a9b472845876bfaf0 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 197b880fa39d432a9b472845876bfaf0, 974b802141bd4f13beffef2aa03f88f9; no voters:
I20260129 02:59:57.386150 20439 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c934d31efd3f4e5195605ddc354d14e1" candidate_uuid: "974b802141bd4f13beffef2aa03f88f9" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "197b880fa39d432a9b472845876bfaf0" is_pre_election: true
I20260129 02:59:57.360992 20433 raft_consensus.cc:2468] T 18f30ebaa4174fd28c958e72d0cc7e24 P 197b880fa39d432a9b472845876bfaf0 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1c50de89a5a845668696dcb28c3badd6 in term 6.
I20260129 02:59:57.364166 20509 raft_consensus.cc:2468] T 1ce4ddf5b10e4ae8b193b13b2e5936ee P 974b802141bd4f13beffef2aa03f88f9 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 197b880fa39d432a9b472845876bfaf0 in term 7.
I20260129 02:59:57.374851 20511 raft_consensus.cc:2468] T 4408006eb5174de1896b037cf3f18457 P 974b802141bd4f13beffef2aa03f88f9 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 197b880fa39d432a9b472845876bfaf0 in term 3.
I20260129 02:59:57.391436 20511 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "add8d4923d1a489e86882c3a25692b95" candidate_uuid: "1c50de89a5a845668696dcb28c3badd6" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "974b802141bd4f13beffef2aa03f88f9" is_pre_election: true
I20260129 02:59:57.393275 20509 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e7d83d4ccf684e76acb29a0dc844e1b8" candidate_uuid: "1c50de89a5a845668696dcb28c3badd6" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "974b802141bd4f13beffef2aa03f88f9"
I20260129 02:59:57.381510 20438 raft_consensus.cc:2468] T b189f08e16a74a11980cc55d99359a0a P 197b880fa39d432a9b472845876bfaf0 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1c50de89a5a845668696dcb28c3badd6 in term 8.
I20260129 02:59:57.325613 22983 raft_consensus.cc:515] T e7d83d4ccf684e76acb29a0dc844e1b8 P 974b802141bd4f13beffef2aa03f88f9 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:59:57.327467 22992 raft_consensus.cc:2749] T dc2f4cb752cf4d59b5f6bc689f5d8104 P 7a9daf5d42a84615a6383cfec9a10608 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260129 02:59:57.392161 20511 raft_consensus.cc:2468] T add8d4923d1a489e86882c3a25692b95 P 974b802141bd4f13beffef2aa03f88f9 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1c50de89a5a845668696dcb28c3badd6 in term 4.
I20260129 02:59:57.397157 20438 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4e3805ffa70242e8ade66da260dddf66" candidate_uuid: "7a9daf5d42a84615a6383cfec9a10608" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "197b880fa39d432a9b472845876bfaf0"
I20260129 02:59:57.399307 20438 raft_consensus.cc:2393] T 4e3805ffa70242e8ade66da260dddf66 P 197b880fa39d432a9b472845876bfaf0 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 7a9daf5d42a84615a6383cfec9a10608 in current term 5: Already voted for candidate 1c50de89a5a845668696dcb28c3badd6 in this term.
I20260129 02:59:57.362596 22884 raft_consensus.cc:2749] T a0fcc58d6fb6473cafbd9d34d5195269 P 1c50de89a5a845668696dcb28c3badd6 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
W20260129 02:59:57.404475 20479 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.211.1:42027, user_credentials={real_user=slave}} blocked reactor thread for 113597us
I20260129 02:59:57.408560 20407 leader_election.cc:304] T 4408006eb5174de1896b037cf3f18457 P 197b880fa39d432a9b472845876bfaf0 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 197b880fa39d432a9b472845876bfaf0, 974b802141bd4f13beffef2aa03f88f9; no voters:
I20260129 02:59:57.414451 20440 raft_consensus.cc:2468] T 0678b8b468bd43cdbc3e485900844821 P 197b880fa39d432a9b472845876bfaf0 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 974b802141bd4f13beffef2aa03f88f9 in term 7.
I20260129 02:59:57.289875 22953 raft_consensus.cc:515] T a644c02b16af47679824ccd060ac10fd P 1c50de89a5a845668696dcb28c3badd6 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } }
I20260129 02:59:57.335456 22891 consensus_queue.cc:237] T 18f30ebaa4174fd28c958e72d0cc7e24 P 1c50de89a5a845668696dcb28c3badd6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 4.6, Last appended by leader: 6, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } }
I20260129 02:59:57.391892 22645 raft_consensus.cc:2804] T 4b1876f9a70b49edb3aa74105e66d3b3 P 197b880fa39d432a9b472845876bfaf0 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260129 02:59:57.441884 20511 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a0fcc58d6fb6473cafbd9d34d5195269" candidate_uuid: "197b880fa39d432a9b472845876bfaf0" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "974b802141bd4f13beffef2aa03f88f9"
I20260129 02:59:57.450161 20440 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "96609ac68bdf4993a12e1c23355ca013" candidate_uuid: "7a9daf5d42a84615a6383cfec9a10608" candidate_term: 6 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "197b880fa39d432a9b472845876bfaf0"
I20260129 02:59:57.379024 22763 raft_consensus.cc:2749] T 75be4bd6bd45474795fd472eff2953ac P 974b802141bd4f13beffef2aa03f88f9 [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20260129 02:59:57.442543 20435 raft_consensus.cc:1240] T 368953c9102643b2a21f99af36b51905 P 197b880fa39d432a9b472845876bfaf0 [term 7 FOLLOWER]: Rejecting Update request from peer 1c50de89a5a845668696dcb28c3badd6 for earlier term 6. Current term is 7. Ops: []
W20260129 02:59:57.443557 20405 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.211.3:37735, user_credentials={real_user=slave}} blocked reactor thread for 151867us
W20260129 02:59:57.432816 21099 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.211.1:42027, user_credentials={real_user=slave}} blocked reactor thread for 118635us
I20260129 02:59:57.394140 22977 raft_consensus.cc:493] T add8d4923d1a489e86882c3a25692b95 P 1c50de89a5a845668696dcb28c3badd6 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260129 02:59:57.447398 20480 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.211.3:37735, user_credentials={real_user=slave}} blocked reactor thread for 247784us
I20260129 02:59:57.416797 20553 leader_election.cc:304] T 4e3805ffa70242e8ade66da260dddf66 P 1c50de89a5a845668696dcb28c3badd6 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 197b880fa39d432a9b472845876bfaf0, 1c50de89a5a845668696dcb28c3badd6; no voters: 7a9daf5d42a84615a6383cfec9a10608
W20260129 02:59:57.448450 20482 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260129 02:59:57.404903 22903 raft_consensus.cc:2749] T e4d18c35b0a7493e85d0a4b92956a48f P 1c50de89a5a845668696dcb28c3badd6 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260129 02:59:57.451069 20439 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a609ac02fae84725ab743f95f51d33ce" candidate_uuid: "7a9daf5d42a84615a6383cfec9a10608" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "197b880fa39d432a9b472845876bfaf0"
I20260129 02:59:57.453528 20556 leader_election.cc:304] T e7d83d4ccf684e76acb29a0dc844e1b8 P 1c50de89a5a845668696dcb28c3badd6 [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1c50de89a5a845668696dcb28c3badd6; no voters: 197b880fa39d432a9b472845876bfaf0, 974b802141bd4f13beffef2aa03f88f9
I20260129 02:59:57.396322 22851 raft_consensus.cc:2804] T 724a236524cb4e7ba3428880681d6cc2 P 197b880fa39d432a9b472845876bfaf0 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260129 02:59:57.454169 20553 leader_election.cc:304] T b189f08e16a74a11980cc55d99359a0a P 1c50de89a5a845668696dcb28c3badd6 [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 197b880fa39d432a9b472845876bfaf0, 1c50de89a5a845668696dcb28c3badd6; no voters:
I20260129 02:59:57.450803 20440 raft_consensus.cc:3060] T 96609ac68bdf4993a12e1c23355ca013 P 197b880fa39d432a9b472845876bfaf0 [term 5 FOLLOWER]: Advancing to term 6
I20260129 02:59:57.454198 20509 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fcddab6bca684adf8ea032289a00f90e" candidate_uuid: "1c50de89a5a845668696dcb28c3badd6" candidate_term: 9 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "974b802141bd4f13beffef2aa03f88f9"
I20260129 02:59:57.457908 20438 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4e420f90884f4796997b5ea41e49e411" candidate_uuid: "7a9daf5d42a84615a6383cfec9a10608" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "197b880fa39d432a9b472845876bfaf0" is_pre_election: true
I20260129 02:59:57.461426 20438 raft_consensus.cc:2468] T 4e420f90884f4796997b5ea41e49e411 P 197b880fa39d432a9b472845876bfaf0 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7a9daf5d42a84615a6383cfec9a10608 in term 5.
I20260129 02:59:57.461894 20582 raft_consensus.cc:1240] T a644c02b16af47679824ccd060ac10fd P 1c50de89a5a845668696dcb28c3badd6 [term 5 FOLLOWER]: Rejecting Update request from peer 7a9daf5d42a84615a6383cfec9a10608 for earlier term 4. Current term is 5. Ops: []
I20260129 02:59:57.451246 20479 leader_election.cc:304] T 0678b8b468bd43cdbc3e485900844821 P 974b802141bd4f13beffef2aa03f88f9 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 197b880fa39d432a9b472845876bfaf0, 974b802141bd4f13beffef2aa03f88f9; no voters:
I20260129 02:59:57.317906 22963 raft_consensus.cc:515] T cb4727386a3e48aca5d71cc6c611c197 P 974b802141bd4f13beffef2aa03f88f9 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:59:57.393316 22691 consensus_queue.cc:1059] T 7a9a69cd82724b1faed1d10aa091fc4a P 197b880fa39d432a9b472845876bfaf0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 }, Status: INVALID_TERM, Last received: 3.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.058s
W20260129 02:59:57.463544 21102 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.211.2:34517, user_credentials={real_user=slave}} blocked reactor thread for 199478us
I20260129 02:59:57.465828 20556 leader_election.cc:304] T fcddab6bca684adf8ea032289a00f90e P 1c50de89a5a845668696dcb28c3badd6 [CANDIDATE]: Term 9 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1c50de89a5a845668696dcb28c3badd6; no voters: 197b880fa39d432a9b472845876bfaf0, 974b802141bd4f13beffef2aa03f88f9
I20260129 02:59:57.463902 22884 raft_consensus.cc:2804] T 4e3805ffa70242e8ade66da260dddf66 P 1c50de89a5a845668696dcb28c3badd6 [term 5 FOLLOWER]: Leader election won for term 5
I20260129 02:59:57.477767 20405 leader_election.cc:304] T 96609ac68bdf4993a12e1c23355ca013 P 197b880fa39d432a9b472845876bfaf0 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 197b880fa39d432a9b472845876bfaf0; no voters: 1c50de89a5a845668696dcb28c3badd6, 974b802141bd4f13beffef2aa03f88f9
I20260129 02:59:57.474099 20511 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7d9494f5584d4d5db8b9adf1c8fc888b" candidate_uuid: "1c50de89a5a845668696dcb28c3badd6" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "974b802141bd4f13beffef2aa03f88f9"
I20260129 02:59:57.476312 20509 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0f20af7fbe5d4db89a6f17948d98912c" candidate_uuid: "197b880fa39d432a9b472845876bfaf0" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "974b802141bd4f13beffef2aa03f88f9" is_pre_election: true
I20260129 02:59:57.478984 20511 raft_consensus.cc:3060] T 7d9494f5584d4d5db8b9adf1c8fc888b P 974b802141bd4f13beffef2aa03f88f9 [term 3 FOLLOWER]: Advancing to term 4
I20260129 02:59:57.479125 20509 raft_consensus.cc:2468] T 0f20af7fbe5d4db89a6f17948d98912c P 974b802141bd4f13beffef2aa03f88f9 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 197b880fa39d432a9b472845876bfaf0 in term 6.
I20260129 02:59:57.484560 20480 leader_election.cc:304] T 45cc5f3d2688486d9b6dbe7e666dedde P 974b802141bd4f13beffef2aa03f88f9 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1c50de89a5a845668696dcb28c3badd6, 974b802141bd4f13beffef2aa03f88f9; no voters: 7a9daf5d42a84615a6383cfec9a10608
W20260129 02:59:57.500564 20407 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.211.2:34517, user_credentials={real_user=slave}} blocked reactor thread for 92107.2us
I20260129 02:59:57.449821 22645 raft_consensus.cc:493] T 4b1876f9a70b49edb3aa74105e66d3b3 P 197b880fa39d432a9b472845876bfaf0 [term 4 FOLLOWER]: Starting leader election (detected failure of leader 974b802141bd4f13beffef2aa03f88f9)
I20260129 02:59:57.454559 22977 raft_consensus.cc:3060] T add8d4923d1a489e86882c3a25692b95 P 1c50de89a5a845668696dcb28c3badd6 [term 4 FOLLOWER]: Advancing to term 5
I20260129 02:59:57.502573 20407 leader_election.cc:304] T 1ce4ddf5b10e4ae8b193b13b2e5936ee P 197b880fa39d432a9b472845876bfaf0 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 197b880fa39d432a9b472845876bfaf0, 974b802141bd4f13beffef2aa03f88f9; no voters:
I20260129 02:59:57.504315 20509 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "cb0b3f259ebc4310927338a8740e87b8" candidate_uuid: "1c50de89a5a845668696dcb28c3badd6" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "974b802141bd4f13beffef2aa03f88f9"
I20260129 02:59:57.461936 22851 raft_consensus.cc:493] T 724a236524cb4e7ba3428880681d6cc2 P 197b880fa39d432a9b472845876bfaf0 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260129 02:59:57.469512 22856 raft_consensus.cc:2749] T e7d83d4ccf684e76acb29a0dc844e1b8 P 1c50de89a5a845668696dcb28c3badd6 [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20260129 02:59:57.477242 23005 raft_consensus.cc:2804] T b189f08e16a74a11980cc55d99359a0a P 1c50de89a5a845668696dcb28c3badd6 [term 8 FOLLOWER]: Leader election won for term 8
W20260129 02:59:57.528141 23016 log.cc:927] Time spent T 368953c9102643b2a21f99af36b51905 P 974b802141bd4f13beffef2aa03f88f9: Append to log took a long time: real 0.051s user 0.000s sys 0.000s
I20260129 02:59:57.483965 22763 raft_consensus.cc:2804] T 0678b8b468bd43cdbc3e485900844821 P 974b802141bd4f13beffef2aa03f88f9 [term 7 FOLLOWER]: Leader election won for term 7
I20260129 02:59:57.522233 22903 raft_consensus.cc:2749] T fcddab6bca684adf8ea032289a00f90e P 1c50de89a5a845668696dcb28c3badd6 [term 9 FOLLOWER]: Leader election lost for term 9. Reason: could not achieve majority
I20260129 02:59:57.545686 20509 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "913fc7d751ed4b7388d889aac30a5f9b" candidate_uuid: "1c50de89a5a845668696dcb28c3badd6" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "974b802141bd4f13beffef2aa03f88f9"
I20260129 02:59:57.573483 20509 raft_consensus.cc:3060] T 913fc7d751ed4b7388d889aac30a5f9b P 974b802141bd4f13beffef2aa03f88f9 [term 3 FOLLOWER]: Advancing to term 4
I20260129 02:59:57.528721 22851 raft_consensus.cc:3060] T 724a236524cb4e7ba3428880681d6cc2 P 197b880fa39d432a9b472845876bfaf0 [term 4 FOLLOWER]: Advancing to term 5
W20260129 02:59:57.581025 20480 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.211.3:37735, user_credentials={real_user=slave}} blocked reactor thread for 105727us
I20260129 02:59:57.582876 20480 leader_election.cc:304] T 76f713c3331540d2a5b76164103e9eb1 P 974b802141bd4f13beffef2aa03f88f9 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 974b802141bd4f13beffef2aa03f88f9; no voters: 197b880fa39d432a9b472845876bfaf0, 1c50de89a5a845668696dcb28c3badd6
I20260129 02:59:57.585121 21133 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "cb0b3f259ebc4310927338a8740e87b8" candidate_uuid: "1c50de89a5a845668696dcb28c3badd6" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "7a9daf5d42a84615a6383cfec9a10608"
I20260129 02:59:57.565953 20440 raft_consensus.cc:2468] T 96609ac68bdf4993a12e1c23355ca013 P 197b880fa39d432a9b472845876bfaf0 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7a9daf5d42a84615a6383cfec9a10608 in term 6.
I20260129 02:59:57.522112 22645 raft_consensus.cc:3060] T 4b1876f9a70b49edb3aa74105e66d3b3 P 197b880fa39d432a9b472845876bfaf0 [term 4 FOLLOWER]: Advancing to term 5
I20260129 02:59:57.540786 22976 leader_election.cc:290] T cb0b3f259ebc4310927338a8740e87b8 P 1c50de89a5a845668696dcb28c3badd6 [CANDIDATE]: Term 6 election: Requested vote from peers 974b802141bd4f13beffef2aa03f88f9 (127.18.211.2:34517), 7a9daf5d42a84615a6383cfec9a10608 (127.18.211.4:40439)
I20260129 02:59:57.399102 23010 consensus_queue.cc:1048] T 75be4bd6bd45474795fd472eff2953ac P 1c50de89a5a845668696dcb28c3badd6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.079s
I20260129 02:59:57.537328 22884 raft_consensus.cc:697] T 4e3805ffa70242e8ade66da260dddf66 P 1c50de89a5a845668696dcb28c3badd6 [term 5 LEADER]: Becoming Leader. State: Replica: 1c50de89a5a845668696dcb28c3badd6, State: Running, Role: LEADER
W20260129 02:59:57.591113 21102 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.211.2:34517, user_credentials={real_user=slave}} blocked reactor thread for 116566us
I20260129 02:59:57.569739 22691 raft_consensus.cc:2804] T 4408006eb5174de1896b037cf3f18457 P 197b880fa39d432a9b472845876bfaf0 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260129 02:59:57.523618 22932 consensus_queue.cc:1048] T 75f45538dc704d9a8ec4ff8ad9f3ea5b P 7a9daf5d42a84615a6383cfec9a10608 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.056s
W20260129 02:59:57.571954 22881 consensus_meta.cc:287] T e33a417cf9c54cf5af73a4ee3aac4d3a P 1c50de89a5a845668696dcb28c3badd6: Time spent flushing consensus metadata: real 0.567s user 0.006s sys 0.007s
W20260129 02:59:57.578699 20407 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.211.2:34517, user_credentials={real_user=slave}} blocked reactor thread for 76325.7us
I20260129 02:59:57.557850 20507 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "926a250fa5834e88b96ccecbefbf5086" candidate_uuid: "1c50de89a5a845668696dcb28c3badd6" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "974b802141bd4f13beffef2aa03f88f9" is_pre_election: true
I20260129 02:59:57.397586 23025 consensus_queue.cc:1059] T 0340229d521b432580ae926f6062e3f7 P 1c50de89a5a845668696dcb28c3badd6 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 }, Status: INVALID_TERM, Last received: 5.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.076s
I20260129 02:59:57.593228 21102 leader_election.cc:304] T 75b0b6e82ddc4f92b1fe1d94f9a0b73b P 7a9daf5d42a84615a6383cfec9a10608 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7a9daf5d42a84615a6383cfec9a10608; no voters: 1c50de89a5a845668696dcb28c3badd6, 974b802141bd4f13beffef2aa03f88f9
I20260129 02:59:57.597137 20440 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7d9494f5584d4d5db8b9adf1c8fc888b" candidate_uuid: "1c50de89a5a845668696dcb28c3badd6" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "197b880fa39d432a9b472845876bfaf0"
I20260129 02:59:57.594450 22691 raft_consensus.cc:493] T 4408006eb5174de1896b037cf3f18457 P 197b880fa39d432a9b472845876bfaf0 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260129 02:59:57.602409 20440 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00e79d8f25a34676b1d866eb2068a2b2" candidate_uuid: "974b802141bd4f13beffef2aa03f88f9" candidate_term: 7 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "197b880fa39d432a9b472845876bfaf0"
I20260129 02:59:57.608687 20440 raft_consensus.cc:3060] T 00e79d8f25a34676b1d866eb2068a2b2 P 197b880fa39d432a9b472845876bfaf0 [term 6 FOLLOWER]: Advancing to term 7
I20260129 02:59:57.581369 22856 raft_consensus.cc:493] T fd8311b779af4e1fa1402b2a02a602f2 P 1c50de89a5a845668696dcb28c3badd6 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 7a9daf5d42a84615a6383cfec9a10608)
I20260129 02:59:57.606074 20509 raft_consensus.cc:2410] T 913fc7d751ed4b7388d889aac30a5f9b P 974b802141bd4f13beffef2aa03f88f9 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 1c50de89a5a845668696dcb28c3badd6 for term 4 because replica has last-logged OpId of term: 3 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 2.
I20260129 02:59:57.577512 22969 leader_election.cc:290] T 00e79d8f25a34676b1d866eb2068a2b2 P 974b802141bd4f13beffef2aa03f88f9 [CANDIDATE]: Term 7 election: Requested vote from peers 197b880fa39d432a9b472845876bfaf0 (127.18.211.1:42027), 7a9daf5d42a84615a6383cfec9a10608 (127.18.211.4:40439)
I20260129 02:59:57.607213 20407 leader_election.cc:304] T 0f20af7fbe5d4db89a6f17948d98912c P 197b880fa39d432a9b472845876bfaf0 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 197b880fa39d432a9b472845876bfaf0, 974b802141bd4f13beffef2aa03f88f9; no voters:
I20260129 02:59:57.613533 20509 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c891d2bff4ce4568861bbc992e91fbb0" candidate_uuid: "197b880fa39d432a9b472845876bfaf0" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "974b802141bd4f13beffef2aa03f88f9"
I20260129 02:59:57.614717 20583 raft_consensus.cc:1275] T e33a417cf9c54cf5af73a4ee3aac4d3a P 1c50de89a5a845668696dcb28c3badd6 [term 6 FOLLOWER]: Refusing update from remote peer 974b802141bd4f13beffef2aa03f88f9: Log matching property violated. Preceding OpId in replica: term: 5 index: 3. Preceding OpId from leader: term: 6 index: 3. (term mismatch)
I20260129 02:59:57.587857 22763 raft_consensus.cc:697] T 0678b8b468bd43cdbc3e485900844821 P 974b802141bd4f13beffef2aa03f88f9 [term 7 LEADER]: Becoming Leader. State: Replica: 974b802141bd4f13beffef2aa03f88f9, State: Running, Role: LEADER
I20260129 02:59:57.615056 20507 raft_consensus.cc:2468] T 926a250fa5834e88b96ccecbefbf5086 P 974b802141bd4f13beffef2aa03f88f9 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1c50de89a5a845668696dcb28c3badd6 in term 6.
I20260129 02:59:57.620779 20507 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b189f08e16a74a11980cc55d99359a0a" candidate_uuid: "1c50de89a5a845668696dcb28c3badd6" candidate_term: 8 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "974b802141bd4f13beffef2aa03f88f9"
I20260129 02:59:57.623129 20440 raft_consensus.cc:2410] T 00e79d8f25a34676b1d866eb2068a2b2 P 197b880fa39d432a9b472845876bfaf0 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 974b802141bd4f13beffef2aa03f88f9 for term 7 because replica has last-logged OpId of term: 6 index: 8, which is greater than that of the candidate, which has last-logged OpId of term: 4 index: 7.
I20260129 02:59:57.617304 20583 pending_rounds.cc:85] T e33a417cf9c54cf5af73a4ee3aac4d3a P 1c50de89a5a845668696dcb28c3badd6: Aborting all ops after (but not including) 2
I20260129 02:59:57.628633 20583 pending_rounds.cc:107] T e33a417cf9c54cf5af73a4ee3aac4d3a P 1c50de89a5a845668696dcb28c3badd6: Aborting uncommitted NO_OP operation due to leader change: 5.3
I20260129 02:59:57.628952 20583 raft_consensus.cc:2889] T e33a417cf9c54cf5af73a4ee3aac4d3a P 1c50de89a5a845668696dcb28c3badd6 [term 6 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260129 02:59:57.633133 20509 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "17d6431cc77a491e8464d68db4081a51" candidate_uuid: "1c50de89a5a845668696dcb28c3badd6" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "974b802141bd4f13beffef2aa03f88f9" is_pre_election: true
I20260129 02:59:57.606848 22982 raft_consensus.cc:2749] T 76f713c3331540d2a5b76164103e9eb1 P 974b802141bd4f13beffef2aa03f88f9 [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260129 02:59:57.595794 23032 raft_consensus.cc:2804] T 45cc5f3d2688486d9b6dbe7e666dedde P 974b802141bd4f13beffef2aa03f88f9 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260129 02:59:57.609819 22691 raft_consensus.cc:3060] T 4408006eb5174de1896b037cf3f18457 P 197b880fa39d432a9b472845876bfaf0 [term 3 FOLLOWER]: Advancing to term 4
I20260129 02:59:57.647065 20515 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "cb4727386a3e48aca5d71cc6c611c197" candidate_uuid: "1c50de89a5a845668696dcb28c3badd6" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "974b802141bd4f13beffef2aa03f88f9"
I20260129 02:59:57.653724 20515 raft_consensus.cc:2393] T cb4727386a3e48aca5d71cc6c611c197 P 974b802141bd4f13beffef2aa03f88f9 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 1c50de89a5a845668696dcb28c3badd6 in current term 4: Already voted for candidate 974b802141bd4f13beffef2aa03f88f9 in this term.
I20260129 02:59:57.658229 22763 consensus_queue.cc:237] T 0678b8b468bd43cdbc3e485900844821 P 974b802141bd4f13beffef2aa03f88f9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 5.7, Last appended by leader: 7, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } }
I20260129 02:59:57.665042 20510 raft_consensus.cc:2468] T 8055232cf0004d40a85f3bf983fe00a9 P 974b802141bd4f13beffef2aa03f88f9 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 197b880fa39d432a9b472845876bfaf0 in term 4.
I20260129 02:59:57.650300 23032 raft_consensus.cc:493] T 45cc5f3d2688486d9b6dbe7e666dedde P 974b802141bd4f13beffef2aa03f88f9 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260129 02:59:57.666535 23032 raft_consensus.cc:3060] T 45cc5f3d2688486d9b6dbe7e666dedde P 974b802141bd4f13beffef2aa03f88f9 [term 4 FOLLOWER]: Advancing to term 5
I20260129 02:59:57.668042 22953 leader_election.cc:290] T a644c02b16af47679824ccd060ac10fd P 1c50de89a5a845668696dcb28c3badd6 [CANDIDATE]: Term 5 election: Requested vote from peers 974b802141bd4f13beffef2aa03f88f9 (127.18.211.2:34517), 7a9daf5d42a84615a6383cfec9a10608 (127.18.211.4:40439)
I20260129 02:59:57.620591 23037 consensus_queue.cc:1048] T 3ccac82a01a2445fa5b0ce6c0fcf2295 P 7a9daf5d42a84615a6383cfec9a10608 [LEADER]: Connected to new peer: Peer: permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.059s
W20260129 02:59:57.669101 23036 log.cc:927] Time spent T 75be4bd6bd45474795fd472eff2953ac P 197b880fa39d432a9b472845876bfaf0: Append to log took a long time: real 0.075s user 0.000s sys 0.000s
I20260129 02:59:57.644683 22184 raft_consensus.cc:515] T 4f5a77e94fef4b16a2ba7b067c8b144a P 974b802141bd4f13beffef2aa03f88f9 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } }
I20260129 02:59:57.670643 20436 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e7d83d4ccf684e76acb29a0dc844e1b8" candidate_uuid: "974b802141bd4f13beffef2aa03f88f9" candidate_term: 6 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "197b880fa39d432a9b472845876bfaf0" is_pre_election: true
I20260129 02:59:57.671842 20436 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "cb4727386a3e48aca5d71cc6c611c197" candidate_uuid: "974b802141bd4f13beffef2aa03f88f9" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "197b880fa39d432a9b472845876bfaf0"
I20260129 02:59:57.672258 22184 leader_election.cc:290] T 4f5a77e94fef4b16a2ba7b067c8b144a P 974b802141bd4f13beffef2aa03f88f9 [CANDIDATE]: Term 4 election: Requested vote from peers 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3:37735), 7a9daf5d42a84615a6383cfec9a10608 (127.18.211.4:40439)
I20260129 02:59:57.671918 22884 consensus_queue.cc:237] T 4e3805ffa70242e8ade66da260dddf66 P 1c50de89a5a845668696dcb28c3badd6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 3.6, Last appended by leader: 6, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } }
I20260129 02:59:57.627812 22856 raft_consensus.cc:515] T fd8311b779af4e1fa1402b2a02a602f2 P 1c50de89a5a845668696dcb28c3badd6 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } }
I20260129 02:59:57.674106 20436 raft_consensus.cc:1275] T 17d6431cc77a491e8464d68db4081a51 P 197b880fa39d432a9b472845876bfaf0 [term 4 FOLLOWER]: Refusing update from remote peer 974b802141bd4f13beffef2aa03f88f9: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 4 index: 2. (index mismatch)
W20260129 02:59:57.668241 23030 log.cc:927] Time spent T 36ff33be666a4c82954731691dbcf327 P 197b880fa39d432a9b472845876bfaf0: Append to log took a long time: real 0.091s user 0.000s sys 0.001s
I20260129 02:59:57.641295 22877 leader_election.cc:290] T 368953c9102643b2a21f99af36b51905 P 197b880fa39d432a9b472845876bfaf0 [CANDIDATE]: Term 7 election: Requested vote from peers 974b802141bd4f13beffef2aa03f88f9 (127.18.211.2:34517), 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3:37735)
I20260129 02:59:57.678671 20588 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:35752 (request call id 1600) took 1237 ms. Trace:
W20260129 02:59:57.694636 20404 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260129 02:59:57.713604 20404 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260129 02:59:57.726929 20436 raft_consensus.cc:3055] T 7a9a69cd82724b1faed1d10aa091fc4a P 197b880fa39d432a9b472845876bfaf0 [term 5 LEADER]: Stepping down as leader of term 5
I20260129 02:59:57.727530 20436 raft_consensus.cc:740] T 7a9a69cd82724b1faed1d10aa091fc4a P 197b880fa39d432a9b472845876bfaf0 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 197b880fa39d432a9b472845876bfaf0, State: Running, Role: LEADER
I20260129 02:59:57.687573 20588 rpcz_store.cc:276] 0129 02:59:56.424734 (+ 0us) service_pool.cc:168] Inserting onto call queue
0129 02:59:56.481957 (+ 57223us) service_pool.cc:225] Handling call
0129 02:59:57.662291 (+1180334us) spinlock_profiling.cc:243] Waited 194 ms on lock 0x7b540018bad4. stack: 00007f8fe044e373 00007f8fe044e270 00007f8fdfda0574 00007f8fe9a5c6b4 00007f8fe9a5c649 00007f8fe34975a1 00007f8fe3531f7a 00007f8fe84e3647 00007f8fe1a73834 00007f8fe1a737c2 00007f8fe1a736f1 00007f8fe1a7366c 00007f8fe1a72922 00007f8fe186d34c 00007f8fe186c786 00007f8fe186c12f
0129 02:59:57.662392 (+ 101us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":446023040}
W20260129 02:59:57.733960 20553 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.211.1:42027, user_credentials={real_user=slave}} blocked reactor thread for 55861.4us
I20260129 02:59:57.665261 22932 raft_consensus.cc:2749] T 75b0b6e82ddc4f92b1fe1d94f9a0b73b P 7a9daf5d42a84615a6383cfec9a10608 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
W20260129 02:59:57.734990 20480 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.211.3:37735, user_credentials={real_user=slave}} blocked reactor thread for 152218us
I20260129 02:59:57.741305 20480 leader_election.cc:304] T add8d4923d1a489e86882c3a25692b95 P 974b802141bd4f13beffef2aa03f88f9 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 974b802141bd4f13beffef2aa03f88f9; no voters: 1c50de89a5a845668696dcb28c3badd6, 7a9daf5d42a84615a6383cfec9a10608
I20260129 02:59:57.731959 22983 leader_election.cc:290] T e7d83d4ccf684e76acb29a0dc844e1b8 P 974b802141bd4f13beffef2aa03f88f9 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 197b880fa39d432a9b472845876bfaf0 (127.18.211.1:42027), 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3:37735)
W20260129 02:59:57.749552 20554 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260129 02:59:57.750085 21133 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a644c02b16af47679824ccd060ac10fd" candidate_uuid: "1c50de89a5a845668696dcb28c3badd6" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "7a9daf5d42a84615a6383cfec9a10608"
I20260129 02:59:57.752413 21127 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fd8311b779af4e1fa1402b2a02a602f2" candidate_uuid: "1c50de89a5a845668696dcb28c3badd6" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "7a9daf5d42a84615a6383cfec9a10608" is_pre_election: true
I20260129 02:59:57.756290 20554 leader_election.cc:304] T cb0b3f259ebc4310927338a8740e87b8 P 1c50de89a5a845668696dcb28c3badd6 [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1c50de89a5a845668696dcb28c3badd6; no voters: 7a9daf5d42a84615a6383cfec9a10608, 974b802141bd4f13beffef2aa03f88f9
I20260129 02:59:57.743139 20436 consensus_queue.cc:260] T 7a9a69cd82724b1faed1d10aa091fc4a P 197b880fa39d432a9b472845876bfaf0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 5.4, Last appended by leader: 4, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:59:57.767534 20480 leader_election.cc:304] T 368953c9102643b2a21f99af36b51905 P 974b802141bd4f13beffef2aa03f88f9 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 974b802141bd4f13beffef2aa03f88f9; no voters: 197b880fa39d432a9b472845876bfaf0, 1c50de89a5a845668696dcb28c3badd6
I20260129 02:59:57.744202 22963 leader_election.cc:290] T cb4727386a3e48aca5d71cc6c611c197 P 974b802141bd4f13beffef2aa03f88f9 [CANDIDATE]: Term 4 election: Requested vote from peers 197b880fa39d432a9b472845876bfaf0 (127.18.211.1:42027), 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3:37735)
W20260129 02:59:57.772936 20405 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.211.3:37735, user_credentials={real_user=slave}} blocked reactor thread for 295229us
I20260129 02:59:57.777241 20436 raft_consensus.cc:3060] T 7a9a69cd82724b1faed1d10aa091fc4a P 197b880fa39d432a9b472845876bfaf0 [term 5 FOLLOWER]: Advancing to term 6
I20260129 02:59:57.748260 22856 leader_election.cc:290] T fd8311b779af4e1fa1402b2a02a602f2 P 1c50de89a5a845668696dcb28c3badd6 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 197b880fa39d432a9b472845876bfaf0 (127.18.211.1:42027), 7a9daf5d42a84615a6383cfec9a10608 (127.18.211.4:40439)
I20260129 02:59:57.760180 23010 raft_consensus.cc:2749] T cb0b3f259ebc4310927338a8740e87b8 P 1c50de89a5a845668696dcb28c3badd6 [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20260129 02:59:57.786984 20480 leader_election.cc:304] T cb0b3f259ebc4310927338a8740e87b8 P 974b802141bd4f13beffef2aa03f88f9 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 974b802141bd4f13beffef2aa03f88f9; no voters: 1c50de89a5a845668696dcb28c3badd6, 7a9daf5d42a84615a6383cfec9a10608
I20260129 02:59:57.773123 22982 raft_consensus.cc:2749] T add8d4923d1a489e86882c3a25692b95 P 974b802141bd4f13beffef2aa03f88f9 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20260129 02:59:57.812994 22691 raft_consensus.cc:515] T 4408006eb5174de1896b037cf3f18457 P 197b880fa39d432a9b472845876bfaf0 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:59:57.795279 20436 raft_consensus.cc:1275] T 7a9a69cd82724b1faed1d10aa091fc4a P 197b880fa39d432a9b472845876bfaf0 [term 6 FOLLOWER]: Refusing update from remote peer 1c50de89a5a845668696dcb28c3badd6: Log matching property violated. Preceding OpId in replica: term: 5 index: 4. Preceding OpId from leader: term: 6 index: 4. (term mismatch)
I20260129 02:59:57.827478 20436 pending_rounds.cc:85] T 7a9a69cd82724b1faed1d10aa091fc4a P 197b880fa39d432a9b472845876bfaf0: Aborting all ops after (but not including) 3
I20260129 02:59:57.827740 20436 pending_rounds.cc:107] T 7a9a69cd82724b1faed1d10aa091fc4a P 197b880fa39d432a9b472845876bfaf0: Aborting uncommitted NO_OP operation due to leader change: 5.4
I20260129 02:59:57.827997 20436 raft_consensus.cc:2889] T 7a9a69cd82724b1faed1d10aa091fc4a P 197b880fa39d432a9b472845876bfaf0 [term 6 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260129 02:59:57.828646 20480 leader_election.cc:304] T a4c1226b987b468fa84dfc6ba0e2dbe1 P 974b802141bd4f13beffef2aa03f88f9 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 974b802141bd4f13beffef2aa03f88f9; no voters: 197b880fa39d432a9b472845876bfaf0, 1c50de89a5a845668696dcb28c3badd6
I20260129 02:59:57.795099 22645 raft_consensus.cc:515] T 4b1876f9a70b49edb3aa74105e66d3b3 P 197b880fa39d432a9b472845876bfaf0 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:59:57.801239 22888 leader_election.cc:290] T 3e2d15f8640b407195210c476e09df91 P 974b802141bd4f13beffef2aa03f88f9 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3:37735), 7a9daf5d42a84615a6383cfec9a10608 (127.18.211.4:40439)
I20260129 02:59:57.802205 22831 raft_consensus.cc:515] T 42d27a28d89b409e8807ec0efd97af69 P 7a9daf5d42a84615a6383cfec9a10608 [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } }
I20260129 02:59:57.855125 22581 consensus_queue.cc:1048] T 7a9a69cd82724b1faed1d10aa091fc4a P 1c50de89a5a845668696dcb28c3badd6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 2, Time since last communication: 0.000s
I20260129 02:59:57.806653 22851 raft_consensus.cc:515] T 724a236524cb4e7ba3428880681d6cc2 P 197b880fa39d432a9b472845876bfaf0 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:59:57.808043 22989 raft_consensus.cc:2749] T 368953c9102643b2a21f99af36b51905 P 974b802141bd4f13beffef2aa03f88f9 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
W20260129 02:59:57.813731 23040 log.cc:927] Time spent T a5a05fbcbbcf4916bbc01fd1a5b5e90d P 974b802141bd4f13beffef2aa03f88f9: Append to log took a long time: real 0.077s user 0.000s sys 0.000s
W20260129 02:59:57.871850 23029 log.cc:927] Time spent T 8055232cf0004d40a85f3bf983fe00a9 P 974b802141bd4f13beffef2aa03f88f9: Append to log took a long time: real 0.492s user 0.000s sys 0.002s
I20260129 02:59:57.898284 22833 consensus_queue.cc:1059] T cb0b3f259ebc4310927338a8740e87b8 P 7a9daf5d42a84615a6383cfec9a10608 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 }, Status: INVALID_TERM, Last received: 5.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20260129 02:59:57.899550 23033 raft_consensus.cc:3055] T cb0b3f259ebc4310927338a8740e87b8 P 7a9daf5d42a84615a6383cfec9a10608 [term 5 LEADER]: Stepping down as leader of term 5
I20260129 02:59:57.900038 23033 raft_consensus.cc:740] T cb0b3f259ebc4310927338a8740e87b8 P 7a9daf5d42a84615a6383cfec9a10608 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 7a9daf5d42a84615a6383cfec9a10608, State: Running, Role: LEADER
I20260129 02:59:57.901062 23033 consensus_queue.cc:260] T cb0b3f259ebc4310927338a8740e87b8 P 7a9daf5d42a84615a6383cfec9a10608 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 5.7, Last appended by leader: 7, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } }
I20260129 02:59:57.902676 23033 raft_consensus.cc:3060] T cb0b3f259ebc4310927338a8740e87b8 P 7a9daf5d42a84615a6383cfec9a10608 [term 5 FOLLOWER]: Advancing to term 6
I20260129 02:59:57.788938 20432 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fd8311b779af4e1fa1402b2a02a602f2" candidate_uuid: "1c50de89a5a845668696dcb28c3badd6" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "197b880fa39d432a9b472845876bfaf0" is_pre_election: true
I20260129 02:59:57.904843 20513 raft_consensus.cc:1240] T 1ce4ddf5b10e4ae8b193b13b2e5936ee P 974b802141bd4f13beffef2aa03f88f9 [term 7 FOLLOWER]: Rejecting Update request from peer 1c50de89a5a845668696dcb28c3badd6 for earlier term 6. Current term is 7. Ops: [2.2-6.4]
I20260129 02:59:57.904713 20432 raft_consensus.cc:2468] T fd8311b779af4e1fa1402b2a02a602f2 P 197b880fa39d432a9b472845876bfaf0 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1c50de89a5a845668696dcb28c3badd6 in term 4.
I20260129 02:59:57.887804 22640 raft_consensus.cc:3055] T c2633a6fc34c44579de348d208b0c01e P 1c50de89a5a845668696dcb28c3badd6 [term 2 LEADER]: Stepping down as leader of term 2
I20260129 02:59:57.905647 20513 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57584 (request call id 1365) took 1397 ms. Trace:
I20260129 02:59:57.905778 22640 raft_consensus.cc:740] T c2633a6fc34c44579de348d208b0c01e P 1c50de89a5a845668696dcb28c3badd6 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 1c50de89a5a845668696dcb28c3badd6, State: Running, Role: LEADER
I20260129 02:59:57.906000 20513 rpcz_store.cc:276] 0129 02:59:56.508557 (+ 0us) service_pool.cc:168] Inserting onto call queue
0129 02:59:56.508698 (+ 141us) service_pool.cc:225] Handling call
0129 02:59:57.904349 (+1395651us) spinlock_profiling.cc:243] Waited 1.51 ms on lock 0x7b5800254d98. stack: 00007f8fe044e373 00007f8fe044e270 00007f8fdfda0574 00007f8fe9a5c6b4 00007f8fe9a5c649 00007f8fe02c08b1 00007f8fe0442db1 00007f8fe0442c62 00007f8fe92eddd9 00007f8fe846d260 00007f8fe8535dbb 00007f8fe8535e41 00007f8fe84f7af0 00007f8fe84e2459 00007f8fe84e1f02 00007f8fe1a6ed34
0129 02:59:57.904537 (+ 188us) raft_consensus.cc:1471] Updating replica for 3 ops
0129 02:59:57.905355 (+ 818us) raft_consensus.cc:1697] Filling consensus response to leader.
0129 02:59:57.905570 (+ 215us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":3463936}
I20260129 02:59:57.906647 22640 consensus_queue.cc:260] T c2633a6fc34c44579de348d208b0c01e P 1c50de89a5a845668696dcb28c3badd6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:59:57.832115 23046 raft_consensus.cc:2749] T cb0b3f259ebc4310927338a8740e87b8 P 974b802141bd4f13beffef2aa03f88f9 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260129 02:59:57.833751 20436 raft_consensus.cc:1240] T c2633a6fc34c44579de348d208b0c01e P 197b880fa39d432a9b472845876bfaf0 [term 3 FOLLOWER]: Rejecting Update request from peer 1c50de89a5a845668696dcb28c3badd6 for earlier term 2. Current term is 3. Ops: []
I20260129 02:59:57.833134 22691 leader_election.cc:290] T 4408006eb5174de1896b037cf3f18457 P 197b880fa39d432a9b472845876bfaf0 [CANDIDATE]: Term 4 election: Requested vote from peers 974b802141bd4f13beffef2aa03f88f9 (127.18.211.2:34517), 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3:37735)
I20260129 02:59:57.836197 22983 raft_consensus.cc:2749] T a4c1226b987b468fa84dfc6ba0e2dbe1 P 974b802141bd4f13beffef2aa03f88f9 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
W20260129 02:59:57.838733 22988 log.cc:927] Time spent T 1299723a4476420ab02fd8bfd7aa0645 P 197b880fa39d432a9b472845876bfaf0: Append to log took a long time: real 1.394s user 0.006s sys 0.000s
I20260129 02:59:57.909453 23042 raft_consensus.cc:2804] T 1ce4ddf5b10e4ae8b193b13b2e5936ee P 197b880fa39d432a9b472845876bfaf0 [term 7 FOLLOWER]: Leader election won for term 7
I20260129 02:59:57.910010 23042 raft_consensus.cc:697] T 1ce4ddf5b10e4ae8b193b13b2e5936ee P 197b880fa39d432a9b472845876bfaf0 [term 7 LEADER]: Becoming Leader. State: Replica: 197b880fa39d432a9b472845876bfaf0, State: Running, Role: LEADER
W20260129 02:59:57.848726 23045 log.cc:927] Time spent T 42d27a28d89b409e8807ec0efd97af69 P 197b880fa39d432a9b472845876bfaf0: Append to log took a long time: real 0.079s user 0.000s sys 0.000s
I20260129 02:59:57.910898 23042 consensus_queue.cc:237] T 1ce4ddf5b10e4ae8b193b13b2e5936ee P 197b880fa39d432a9b472845876bfaf0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 5.3, Last appended by leader: 3, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:59:57.911603 20437 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:48356 (request call id 333) took 2196 ms. Trace:
I20260129 02:59:57.912060 22231 raft_consensus.cc:2749] T 96609ac68bdf4993a12e1c23355ca013 P 197b880fa39d432a9b472845876bfaf0 [term 6 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260129 02:59:57.912015 20437 rpcz_store.cc:276] 0129 02:59:55.715481 (+ 0us) service_pool.cc:168] Inserting onto call queue
0129 02:59:55.730913 (+ 15432us) service_pool.cc:225] Handling call
0129 02:59:55.744401 (+ 13488us) raft_consensus.cc:1471] Updating replica for 1 ops
0129 02:59:55.755045 (+ 10644us) raft_consensus.cc:1534] Early marking committed up to index 7
0129 02:59:55.789772 (+ 34727us) log.cc:844] Serialized 14 byte log entry
0129 02:59:56.177473 (+387701us) spinlock_profiling.cc:243] Waited 15.4 ms on lock 0x7b1800374a40. stack: 00007f8fe044e373 00007f8fe044e270 00007f8fdfda0574 00007f8fe9a5c6b4 00007f8fe9a5c649 00007f8fe92d4c91 00007f8fe92e1671 00007f8fe0462c67 00007f8fe924b709 00007f8fe047f57b 00007f8fe047d81f 00007f8fe047fae2 00007f8fe30c75ab 00007f8fe30cd0c3 00007f8fe30cd5da 00007f8fe35386aa
0129 02:59:56.363697 (+186224us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0129 02:59:56.450120 (+ 86423us) log.cc:844] Serialized 26 byte log entry
0129 02:59:56.450285 (+ 165us) raft_consensus.cc:1645] Marking committed up to 8
0129 02:59:56.608837 (+158552us) log.cc:844] Serialized 14 byte log entry
0129 02:59:56.609111 (+ 274us) raft_consensus.cc:1697] Filling consensus response to leader.
0129 02:59:56.609178 (+ 67us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0129 02:59:57.472229 (+863051us) spinlock_profiling.cc:243] Waited 1.77 ms on lock 0x7b48001c0158. stack: 00007f8fe044e373 00007f8fe044e270 00007f8fdfda0574 00007f8fe9a5c6b4 00007f8fe9a5c649 00007f8fe8447e21 00007f8fe843eeb7 00007f8fe843ed4a 00007f8fe8535f7f 00007f8fe8535c47 00007f8fe854a26c 00007f8fe854a218 00007f8fe854a198 00007f8fe854a153 00007f8fe85493fc 00007f8fe74845b6
0129 02:59:57.837870 (+365641us) spinlock_profiling.cc:243] Waited 6.61 ms on lock 0x7b440002304c. stack: 00007f8fe044e373 00007f8fe044e270 00007f8fdfda0574 00007f8fe9a5c6b4 00007f8fe9a5c649 00007f8fe02c08b1 00007f8fe038f005 00007f8fe038fa22 00007f8fe30d957b 00007f8fe30d4e4f 00007f8fe30c8c0e 00007f8fe30c7f56 00007f8fe30c7b78 00007f8fe30d15c1 00007f8fe30d1579 00007f8fe30d1509
0129 02:59:57.911356 (+ 73486us) raft_consensus.cc:1685] finished
0129 02:59:57.911393 (+ 37us) raft_consensus.cc:1691] UpdateReplicas() finished
0129 02:59:57.911517 (+ 124us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":26616,"prepare.queue_time_us":218183,"prepare.run_cpu_time_us":94,"prepare.run_wall_time_us":95,"raft.queue_time_us":29436,"raft.run_cpu_time_us":3492,"raft.run_wall_time_us":1137579,"spinlock_wait_cycles":54576128,"thread_start_us":274328,"threads_started":2,"wal-append.queue_time_us":611546}
W20260129 02:59:57.871592 22972 log.cc:927] Time spent T 42d27a28d89b409e8807ec0efd97af69 P 1c50de89a5a845668696dcb28c3badd6: Append to log took a long time: real 1.730s user 0.008s sys 0.000s
I20260129 02:59:57.807706 23034 consensus_queue.cc:1048] T 17d6431cc77a491e8464d68db4081a51 P 974b802141bd4f13beffef2aa03f88f9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.120s
I20260129 02:59:57.916441 20508 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:48544 (request call id 236) took 1277 ms. Trace:
I20260129 02:59:57.916903 20508 rpcz_store.cc:276] 0129 02:59:56.638784 (+ 0us) service_pool.cc:168] Inserting onto call queue
0129 02:59:56.645569 (+ 6785us) service_pool.cc:225] Handling call
0129 02:59:57.916265 (+1270696us) spinlock_profiling.cc:243] Waited 38.7 ms on lock 0x7b54002ee854. stack: 00007f8fe044e373 00007f8fe044e270 00007f8fdfda0574 00007f8fe9a5c6b4 00007f8fe9a5c649 00007f8fe34975a1 00007f8fe3531f7a 00007f8fe84e3647 00007f8fe1a73834 00007f8fe1a737c2 00007f8fe1a736f1 00007f8fe1a7366c 00007f8fe1a72922 00007f8fe186d34c 00007f8fe186c786 00007f8fe186c12f
0129 02:59:57.916381 (+ 116us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":89051264}
I20260129 02:59:57.873945 20516 raft_consensus.cc:1275] T 76f713c3331540d2a5b76164103e9eb1 P 974b802141bd4f13beffef2aa03f88f9 [term 5 FOLLOWER]: Refusing update from remote peer 1c50de89a5a845668696dcb28c3badd6: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 5 index: 4. (index mismatch)
I20260129 02:59:57.875581 20441 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55672 (request call id 1599) took 1321 ms. Trace:
I20260129 02:59:57.918529 20508 raft_consensus.cc:1275] T 7a9a69cd82724b1faed1d10aa091fc4a P 974b802141bd4f13beffef2aa03f88f9 [term 6 FOLLOWER]: Refusing update from remote peer 1c50de89a5a845668696dcb28c3badd6: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 6 index: 4. (index mismatch)
I20260129 02:59:57.894865 20514 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:48544 (request call id 231) took 1374 ms. Trace:
I20260129 02:59:57.838101 20480 leader_election.cc:304] T cedce331e1584ac8b636c9b8a72b2ba0 P 974b802141bd4f13beffef2aa03f88f9 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1c50de89a5a845668696dcb28c3badd6, 974b802141bd4f13beffef2aa03f88f9; no voters: 197b880fa39d432a9b472845876bfaf0
I20260129 02:59:57.897200 22995 consensus_queue.cc:1059] T 368953c9102643b2a21f99af36b51905 P 1c50de89a5a845668696dcb28c3badd6 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 }, Status: INVALID_TERM, Last received: 6.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20260129 02:59:57.873797 22900 raft_consensus.cc:3055] T 0340229d521b432580ae926f6062e3f7 P 1c50de89a5a845668696dcb28c3badd6 [term 5 LEADER]: Stepping down as leader of term 5
I20260129 02:59:57.817283 20433 raft_consensus.cc:1275] T 4f8f3a38f6e24ba4857875181f447442 P 197b880fa39d432a9b472845876bfaf0 [term 6 FOLLOWER]: Refusing update from remote peer 974b802141bd4f13beffef2aa03f88f9: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 6 index: 4. (index mismatch)
I20260129 02:59:57.918761 20441 rpcz_store.cc:276] 0129 02:59:56.554437 (+ 0us) service_pool.cc:168] Inserting onto call queue
0129 02:59:56.555746 (+ 1309us) service_pool.cc:225] Handling call
0129 02:59:56.616757 (+ 61011us) raft_consensus.cc:1471] Updating replica for 0 ops
0129 02:59:57.373281 (+756524us) raft_consensus.cc:1697] Filling consensus response to leader.
0129 02:59:57.875263 (+501982us) spinlock_profiling.cc:243] Waited 756 ms on lock 0x7b54003d1654. stack: 00007f8fe044e373 00007f8fe044e270 00007f8fdfda0574 00007f8fe9a5c6b4 00007f8fe9a5c649 00007f8fe34975a1 00007f8fe352ea81 00007f8fe352d4be 00007f8fe84e1f49 00007f8fe1a6ed34 00007f8fe1a6ecc2 00007f8fe1a6ebf1 00007f8fe1a6eb6c 00007f8fe1a6de22 00007f8fe186d34c 00007f8fe186c786
0129 02:59:57.875489 (+ 226us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1738424192}
W20260129 02:59:57.923882 20480 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.211.3:37735, user_credentials={real_user=slave}} blocked reactor thread for 85908.6us
I20260129 02:59:57.924796 20480 leader_election.cc:304] T 0f20af7fbe5d4db89a6f17948d98912c P 974b802141bd4f13beffef2aa03f88f9 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1c50de89a5a845668696dcb28c3badd6, 974b802141bd4f13beffef2aa03f88f9; no voters: 197b880fa39d432a9b472845876bfaf0
I20260129 02:59:57.919718 20514 rpcz_store.cc:276] 0129 02:59:56.520844 (+ 0us) service_pool.cc:168] Inserting onto call queue
0129 02:59:56.548238 (+ 27394us) service_pool.cc:225] Handling call
0129 02:59:56.577778 (+ 29540us) raft_consensus.cc:1471] Updating replica for 1 ops
0129 02:59:57.333891 (+756113us) raft_consensus.cc:1697] Filling consensus response to leader.
0129 02:59:57.894596 (+560705us) spinlock_profiling.cc:243] Waited 697 ms on lock 0x7b54002efc54. stack: 00007f8fe044e373 00007f8fe044e270 00007f8fdfda0574 00007f8fe9a5c6b4 00007f8fe9a5c649 00007f8fe34975a1 00007f8fe352ea81 00007f8fe352d4be 00007f8fe84e1f49 00007f8fe1a6ed34 00007f8fe1a6ecc2 00007f8fe1a6ebf1 00007f8fe1a6eb6c 00007f8fe1a6de22 00007f8fe186d34c 00007f8fe186c786
0129 02:59:57.894805 (+ 209us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1603883264}
I20260129 02:59:57.873206 22967 raft_consensus.cc:3055] T 2ac659d732314e1288849f08fe5e074e P 974b802141bd4f13beffef2aa03f88f9 [term 3 LEADER]: Stepping down as leader of term 3
I20260129 02:59:57.881207 20434 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55672 (request call id 1602) took 1320 ms. Trace:
I20260129 02:59:57.921118 20439 raft_consensus.cc:1275] T 76f713c3331540d2a5b76164103e9eb1 P 197b880fa39d432a9b472845876bfaf0 [term 5 FOLLOWER]: Refusing update from remote peer 1c50de89a5a845668696dcb28c3badd6: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 5 index: 4. (index mismatch)
W20260129 02:59:57.927042 20407 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.211.2:34517, user_credentials={real_user=slave}} blocked reactor thread for 319914us
I20260129 02:59:57.927318 23043 raft_consensus.cc:2804] T 0f20af7fbe5d4db89a6f17948d98912c P 197b880fa39d432a9b472845876bfaf0 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260129 02:59:57.927743 23043 raft_consensus.cc:493] T 0f20af7fbe5d4db89a6f17948d98912c P 197b880fa39d432a9b472845876bfaf0 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260129 02:59:57.925881 20434 rpcz_store.cc:276] 0129 02:59:56.560668 (+ 0us) service_pool.cc:168] Inserting onto call queue
0129 02:59:56.576377 (+ 15709us) service_pool.cc:225] Handling call
0129 02:59:56.576673 (+ 296us) raft_consensus.cc:1471] Updating replica for 0 ops
0129 02:59:57.376474 (+799801us) raft_consensus.cc:1697] Filling consensus response to leader.
0129 02:59:57.880897 (+504423us) spinlock_profiling.cc:243] Waited 738 ms on lock 0x7b54003237d4. stack: 00007f8fe044e373 00007f8fe044e270 00007f8fdfda0574 00007f8fe9a5c6b4 00007f8fe9a5c649 00007f8fe34975a1 00007f8fe352ea81 00007f8fe352d4be 00007f8fe84e1f49 00007f8fe1a6ed34 00007f8fe1a6ecc2 00007f8fe1a6ebf1 00007f8fe1a6eb6c 00007f8fe1a6de22 00007f8fe186d34c 00007f8fe186c786
0129 02:59:57.881130 (+ 233us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1696603520}
W20260129 02:59:57.928068 20405 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.211.3:37735, user_credentials={real_user=slave}} blocked reactor thread for 143225us
I20260129 02:59:57.877246 22977 raft_consensus.cc:515] T add8d4923d1a489e86882c3a25692b95 P 1c50de89a5a845668696dcb28c3badd6 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } }
W20260129 02:59:57.929703 20407 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260129 02:59:57.930091 20407 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260129 02:59:57.929957 20516 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4408006eb5174de1896b037cf3f18457" candidate_uuid: "197b880fa39d432a9b472845876bfaf0" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "974b802141bd4f13beffef2aa03f88f9"
I20260129 02:59:57.930500 20405 leader_election.cc:304] T fcddab6bca684adf8ea032289a00f90e P 197b880fa39d432a9b472845876bfaf0 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 197b880fa39d432a9b472845876bfaf0, 1c50de89a5a845668696dcb28c3badd6; no voters: 974b802141bd4f13beffef2aa03f88f9
I20260129 02:59:57.932204 21099 leader_election.cc:304] T 4e420f90884f4796997b5ea41e49e411 P 7a9daf5d42a84615a6383cfec9a10608 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 197b880fa39d432a9b472845876bfaf0, 7a9daf5d42a84615a6383cfec9a10608; no voters:
I20260129 02:59:57.930532 20516 raft_consensus.cc:3060] T 4408006eb5174de1896b037cf3f18457 P 974b802141bd4f13beffef2aa03f88f9 [term 3 FOLLOWER]: Advancing to term 4
I20260129 02:59:57.933473 21099 leader_election.cc:304] T 96609ac68bdf4993a12e1c23355ca013 P 7a9daf5d42a84615a6383cfec9a10608 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 197b880fa39d432a9b472845876bfaf0, 1c50de89a5a845668696dcb28c3badd6, 7a9daf5d42a84615a6383cfec9a10608; no voters:
I20260129 02:59:57.922442 20553 leader_election.cc:304] T fd8311b779af4e1fa1402b2a02a602f2 P 1c50de89a5a845668696dcb28c3badd6 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 197b880fa39d432a9b472845876bfaf0, 1c50de89a5a845668696dcb28c3badd6; no voters: 7a9daf5d42a84615a6383cfec9a10608
I20260129 02:59:57.907701 22640 raft_consensus.cc:3060] T c2633a6fc34c44579de348d208b0c01e P 1c50de89a5a845668696dcb28c3badd6 [term 2 FOLLOWER]: Advancing to term 3
I20260129 02:59:57.935987 20435 raft_consensus.cc:1240] T 4b1876f9a70b49edb3aa74105e66d3b3 P 197b880fa39d432a9b472845876bfaf0 [term 5 FOLLOWER]: Rejecting Update request from peer 974b802141bd4f13beffef2aa03f88f9 for earlier term 4. Current term is 5. Ops: []
W20260129 02:59:57.937584 20406 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260129 02:59:57.871824 22922 consensus_meta.cc:287] T 1ce4ddf5b10e4ae8b193b13b2e5936ee P 1c50de89a5a845668696dcb28c3badd6: Time spent flushing consensus metadata: real 0.601s user 0.006s sys 0.002s
I20260129 02:59:57.939376 20440 raft_consensus.cc:1240] T 724a236524cb4e7ba3428880681d6cc2 P 197b880fa39d432a9b472845876bfaf0 [term 5 FOLLOWER]: Rejecting Update request from peer 1c50de89a5a845668696dcb28c3badd6 for earlier term 4. Current term is 5. Ops: [4.3-4.3]
I20260129 02:59:57.928095 23043 raft_consensus.cc:3060] T 0f20af7fbe5d4db89a6f17948d98912c P 197b880fa39d432a9b472845876bfaf0 [term 6 FOLLOWER]: Advancing to term 7
I20260129 02:59:57.940809 22691 raft_consensus.cc:2804] T fcddab6bca684adf8ea032289a00f90e P 197b880fa39d432a9b472845876bfaf0 [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260129 02:59:57.941337 22691 raft_consensus.cc:493] T fcddab6bca684adf8ea032289a00f90e P 197b880fa39d432a9b472845876bfaf0 [term 8 FOLLOWER]: Starting leader election (detected failure of leader 974b802141bd4f13beffef2aa03f88f9)
I20260129 02:59:57.941802 22691 raft_consensus.cc:3060] T fcddab6bca684adf8ea032289a00f90e P 197b880fa39d432a9b472845876bfaf0 [term 8 FOLLOWER]: Advancing to term 9
I20260129 02:59:57.943719 22708 raft_consensus.cc:2804] T fd8311b779af4e1fa1402b2a02a602f2 P 1c50de89a5a845668696dcb28c3badd6 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260129 02:59:57.944164 22708 raft_consensus.cc:493] T fd8311b779af4e1fa1402b2a02a602f2 P 1c50de89a5a845668696dcb28c3badd6 [term 4 FOLLOWER]: Starting leader election (detected failure of leader 7a9daf5d42a84615a6383cfec9a10608)
I20260129 02:59:57.944507 22708 raft_consensus.cc:3060] T fd8311b779af4e1fa1402b2a02a602f2 P 1c50de89a5a845668696dcb28c3badd6 [term 4 FOLLOWER]: Advancing to term 5
I20260129 02:59:57.945303 22933 raft_consensus.cc:2804] T 4e420f90884f4796997b5ea41e49e411 P 7a9daf5d42a84615a6383cfec9a10608 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260129 02:59:57.945809 22933 raft_consensus.cc:493] T 4e420f90884f4796997b5ea41e49e411 P 7a9daf5d42a84615a6383cfec9a10608 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260129 02:59:57.946225 22933 raft_consensus.cc:3060] T 4e420f90884f4796997b5ea41e49e411 P 7a9daf5d42a84615a6383cfec9a10608 [term 5 FOLLOWER]: Advancing to term 6
I20260129 02:59:57.948211 21129 raft_consensus.cc:1240] T 42d27a28d89b409e8807ec0efd97af69 P 7a9daf5d42a84615a6383cfec9a10608 [term 7 FOLLOWER]: Rejecting Update request from peer 1c50de89a5a845668696dcb28c3badd6 for earlier term 6. Current term is 7. Ops: []
I20260129 02:59:57.948151 22995 consensus_queue.cc:1048] T 76f713c3331540d2a5b76164103e9eb1 P 1c50de89a5a845668696dcb28c3badd6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260129 02:59:57.948957 20439 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4e3805ffa70242e8ade66da260dddf66" candidate_uuid: "7a9daf5d42a84615a6383cfec9a10608" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "197b880fa39d432a9b472845876bfaf0"
I20260129 02:59:57.949819 20439 raft_consensus.cc:2393] T 4e3805ffa70242e8ade66da260dddf66 P 197b880fa39d432a9b472845876bfaf0 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 7a9daf5d42a84615a6383cfec9a10608 in current term 5: Already voted for candidate 1c50de89a5a845668696dcb28c3badd6 in this term.
I20260129 02:59:57.950639 20433 raft_consensus.cc:1240] T 00e79d8f25a34676b1d866eb2068a2b2 P 197b880fa39d432a9b472845876bfaf0 [term 7 FOLLOWER]: Rejecting Update request from peer 7a9daf5d42a84615a6383cfec9a10608 for earlier term 6. Current term is 7. Ops: []
I20260129 02:59:57.947584 20516 raft_consensus.cc:2468] T 4408006eb5174de1896b037cf3f18457 P 974b802141bd4f13beffef2aa03f88f9 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 197b880fa39d432a9b472845876bfaf0 in term 4.
I20260129 02:59:57.951726 22667 raft_consensus.cc:2804] T 96609ac68bdf4993a12e1c23355ca013 P 7a9daf5d42a84615a6383cfec9a10608 [term 6 FOLLOWER]: Leader election won for term 6
I20260129 02:59:57.952323 22667 raft_consensus.cc:697] T 96609ac68bdf4993a12e1c23355ca013 P 7a9daf5d42a84615a6383cfec9a10608 [term 6 LEADER]: Becoming Leader. State: Replica: 7a9daf5d42a84615a6383cfec9a10608, State: Running, Role: LEADER
I20260129 02:59:57.932761 22967 raft_consensus.cc:740] T 2ac659d732314e1288849f08fe5e074e P 974b802141bd4f13beffef2aa03f88f9 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 974b802141bd4f13beffef2aa03f88f9, State: Running, Role: LEADER
I20260129 02:59:57.922071 22900 raft_consensus.cc:740] T 0340229d521b432580ae926f6062e3f7 P 1c50de89a5a845668696dcb28c3badd6 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 1c50de89a5a845668696dcb28c3badd6, State: Running, Role: LEADER
I20260129 02:59:57.954607 22989 raft_consensus.cc:2804] T cedce331e1584ac8b636c9b8a72b2ba0 P 974b802141bd4f13beffef2aa03f88f9 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260129 02:59:57.964263 20556 leader_election.cc:304] T 913fc7d751ed4b7388d889aac30a5f9b P 1c50de89a5a845668696dcb28c3badd6 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1c50de89a5a845668696dcb28c3badd6; no voters: 197b880fa39d432a9b472845876bfaf0, 974b802141bd4f13beffef2aa03f88f9
I20260129 02:59:57.958573 20511 raft_consensus.cc:2468] T 7d9494f5584d4d5db8b9adf1c8fc888b P 974b802141bd4f13beffef2aa03f88f9 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1c50de89a5a845668696dcb28c3badd6 in term 4.
I20260129 02:59:57.961540 20508 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "368953c9102643b2a21f99af36b51905" candidate_uuid: "197b880fa39d432a9b472845876bfaf0" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "974b802141bd4f13beffef2aa03f88f9"
W20260129 02:59:57.969566 21102 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.211.2:34517, user_credentials={real_user=slave}} blocked reactor thread for 376546us
I20260129 02:59:57.970714 21102 leader_election.cc:304] T 4f5a77e94fef4b16a2ba7b067c8b144a P 7a9daf5d42a84615a6383cfec9a10608 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7a9daf5d42a84615a6383cfec9a10608; no voters: 1c50de89a5a845668696dcb28c3badd6, 974b802141bd4f13beffef2aa03f88f9
I20260129 02:59:57.972191 23037 raft_consensus.cc:2749] T 4f5a77e94fef4b16a2ba7b067c8b144a P 7a9daf5d42a84615a6383cfec9a10608 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260129 02:59:57.973177 21102 leader_election.cc:304] T 45cc5f3d2688486d9b6dbe7e666dedde P 7a9daf5d42a84615a6383cfec9a10608 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7a9daf5d42a84615a6383cfec9a10608; no voters: 1c50de89a5a845668696dcb28c3badd6, 974b802141bd4f13beffef2aa03f88f9
I20260129 02:59:57.974709 20511 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a644c02b16af47679824ccd060ac10fd" candidate_uuid: "1c50de89a5a845668696dcb28c3badd6" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "974b802141bd4f13beffef2aa03f88f9"
I20260129 02:59:57.976239 20514 raft_consensus.cc:1240] T 0340229d521b432580ae926f6062e3f7 P 974b802141bd4f13beffef2aa03f88f9 [term 6 FOLLOWER]: Rejecting Update request from peer 1c50de89a5a845668696dcb28c3badd6 for earlier term 5. Current term is 6. Ops: []
I20260129 02:59:57.978106 20511 raft_consensus.cc:3060] T a644c02b16af47679824ccd060ac10fd P 974b802141bd4f13beffef2aa03f88f9 [term 4 FOLLOWER]: Advancing to term 5
I20260129 02:59:57.980099 20514 raft_consensus.cc:1240] T 4f5a77e94fef4b16a2ba7b067c8b144a P 974b802141bd4f13beffef2aa03f88f9 [term 4 FOLLOWER]: Rejecting Update request from peer 1c50de89a5a845668696dcb28c3badd6 for earlier term 3. Current term is 4. Ops: []
I20260129 02:59:57.981647 23037 raft_consensus.cc:3060] T 45cc5f3d2688486d9b6dbe7e666dedde P 7a9daf5d42a84615a6383cfec9a10608 [term 3 FOLLOWER]: Advancing to term 4
I20260129 02:59:57.982712 20514 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "add8d4923d1a489e86882c3a25692b95" candidate_uuid: "1c50de89a5a845668696dcb28c3badd6" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "974b802141bd4f13beffef2aa03f88f9"
I20260129 02:59:57.983392 20514 raft_consensus.cc:3060] T add8d4923d1a489e86882c3a25692b95 P 974b802141bd4f13beffef2aa03f88f9 [term 4 FOLLOWER]: Advancing to term 5
W20260129 02:59:57.984792 20482 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260129 02:59:57.988611 20511 raft_consensus.cc:2468] T a644c02b16af47679824ccd060ac10fd P 974b802141bd4f13beffef2aa03f88f9 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1c50de89a5a845668696dcb28c3badd6 in term 5.
I20260129 02:59:57.993682 21102 leader_election.cc:304] T febffe4a8b394368b69b1d164308fb9a P 7a9daf5d42a84615a6383cfec9a10608 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7a9daf5d42a84615a6383cfec9a10608; no voters: 197b880fa39d432a9b472845876bfaf0, 974b802141bd4f13beffef2aa03f88f9
I20260129 02:59:57.997361 22833 raft_consensus.cc:2749] T febffe4a8b394368b69b1d164308fb9a P 7a9daf5d42a84615a6383cfec9a10608 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20260129 02:59:58.000109 21102 leader_election.cc:304] T a609ac02fae84725ab743f95f51d33ce P 7a9daf5d42a84615a6383cfec9a10608 [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7a9daf5d42a84615a6383cfec9a10608; no voters: 197b880fa39d432a9b472845876bfaf0, 974b802141bd4f13beffef2aa03f88f9
I20260129 02:59:58.000707 20514 raft_consensus.cc:2468] T add8d4923d1a489e86882c3a25692b95 P 974b802141bd4f13beffef2aa03f88f9 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1c50de89a5a845668696dcb28c3badd6 in term 5.
I20260129 02:59:58.003791 22833 raft_consensus.cc:2749] T a609ac02fae84725ab743f95f51d33ce P 7a9daf5d42a84615a6383cfec9a10608 [term 7 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
I20260129 02:59:58.007570 21102 leader_election.cc:304] T add8d4923d1a489e86882c3a25692b95 P 7a9daf5d42a84615a6383cfec9a10608 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7a9daf5d42a84615a6383cfec9a10608; no voters: 1c50de89a5a845668696dcb28c3badd6, 974b802141bd4f13beffef2aa03f88f9
W20260129 02:59:58.009954 20556 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.211.2:34517, user_credentials={real_user=slave}} blocked reactor thread for 44275.5us
W20260129 02:59:58.011483 20479 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.211.1:42027, user_credentials={real_user=slave}} blocked reactor thread for 63465.8us
I20260129 02:59:58.010864 20556 leader_election.cc:304] T 926a250fa5834e88b96ccecbefbf5086 P 1c50de89a5a845668696dcb28c3badd6 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1c50de89a5a845668696dcb28c3badd6, 974b802141bd4f13beffef2aa03f88f9; no voters: 197b880fa39d432a9b472845876bfaf0
W20260129 02:59:58.012828 20407 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.211.2:34517, user_credentials={real_user=slave}} blocked reactor thread for 76426.8us
I20260129 02:59:58.014743 20407 leader_election.cc:304] T 8055232cf0004d40a85f3bf983fe00a9 P 197b880fa39d432a9b472845876bfaf0 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 197b880fa39d432a9b472845876bfaf0, 974b802141bd4f13beffef2aa03f88f9; no voters:
I20260129 02:59:58.014590 21133 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "add8d4923d1a489e86882c3a25692b95" candidate_uuid: "1c50de89a5a845668696dcb28c3badd6" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "7a9daf5d42a84615a6383cfec9a10608"
I20260129 02:59:58.016322 21133 raft_consensus.cc:3060] T add8d4923d1a489e86882c3a25692b95 P 7a9daf5d42a84615a6383cfec9a10608 [term 4 FOLLOWER]: Advancing to term 5
I20260129 02:59:58.016719 20556 leader_election.cc:304] T 17d6431cc77a491e8464d68db4081a51 P 1c50de89a5a845668696dcb28c3badd6 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1c50de89a5a845668696dcb28c3badd6; no voters: 197b880fa39d432a9b472845876bfaf0, 974b802141bd4f13beffef2aa03f88f9
W20260129 02:59:58.017778 20480 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.211.3:37735, user_credentials={real_user=slave}} blocked reactor thread for 93086.2us
I20260129 02:59:58.018659 20480 leader_election.cc:304] T 96609ac68bdf4993a12e1c23355ca013 P 974b802141bd4f13beffef2aa03f88f9 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 4 responses out of 4 voters: 3 yes votes; 1 no votes. yes voters: 197b880fa39d432a9b472845876bfaf0, 1c50de89a5a845668696dcb28c3badd6, 974b802141bd4f13beffef2aa03f88f9; no voters: 7a9daf5d42a84615a6383cfec9a10608
I20260129 02:59:57.960119 22977 leader_election.cc:290] T add8d4923d1a489e86882c3a25692b95 P 1c50de89a5a845668696dcb28c3badd6 [CANDIDATE]: Term 5 election: Requested vote from peers 974b802141bd4f13beffef2aa03f88f9 (127.18.211.2:34517), 7a9daf5d42a84615a6383cfec9a10608 (127.18.211.4:40439)
W20260129 02:59:58.021684 20405 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.211.3:37735, user_credentials={real_user=slave}} blocked reactor thread for 87756.6us
I20260129 02:59:57.963377 22989 raft_consensus.cc:493] T cedce331e1584ac8b636c9b8a72b2ba0 P 974b802141bd4f13beffef2aa03f88f9 [term 6 FOLLOWER]: Starting leader election (detected failure of leader 197b880fa39d432a9b472845876bfaf0)
I20260129 02:59:58.022303 20480 leader_election.cc:304] T 7d9494f5584d4d5db8b9adf1c8fc888b P 974b802141bd4f13beffef2aa03f88f9 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1c50de89a5a845668696dcb28c3badd6, 974b802141bd4f13beffef2aa03f88f9; no voters: 197b880fa39d432a9b472845876bfaf0
I20260129 02:59:58.008055 23005 raft_consensus.cc:697] T b189f08e16a74a11980cc55d99359a0a P 1c50de89a5a845668696dcb28c3badd6 [term 8 LEADER]: Becoming Leader. State: Replica: 1c50de89a5a845668696dcb28c3badd6, State: Running, Role: LEADER
I20260129 02:59:58.021716 21102 leader_election.cc:304] T 505580c93c694bbb8ea96fdf9b17ab46 P 7a9daf5d42a84615a6383cfec9a10608 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 7a9daf5d42a84615a6383cfec9a10608, 974b802141bd4f13beffef2aa03f88f9; no voters: 1c50de89a5a845668696dcb28c3badd6
I20260129 02:59:58.011971 22881 raft_consensus.cc:2749] T 913fc7d751ed4b7388d889aac30a5f9b P 1c50de89a5a845668696dcb28c3badd6 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20260129 02:59:58.008790 23034 raft_consensus.cc:2804] T 0f20af7fbe5d4db89a6f17948d98912c P 974b802141bd4f13beffef2aa03f88f9 [term 6 FOLLOWER]: Leader election won for term 6
I20260129 02:59:57.956089 22667 consensus_queue.cc:237] T 96609ac68bdf4993a12e1c23355ca013 P 7a9daf5d42a84615a6383cfec9a10608 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 2.4, Last appended by leader: 4, Current term: 6, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } attrs { replace: true } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } }
I20260129 02:59:58.035610 22975 consensus_queue.cc:1048] T 4f8f3a38f6e24ba4857875181f447442 P 974b802141bd4f13beffef2aa03f88f9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.079s
I20260129 02:59:58.035964 22980 raft_consensus.cc:2804] T 8055232cf0004d40a85f3bf983fe00a9 P 197b880fa39d432a9b472845876bfaf0 [term 4 FOLLOWER]: Leader election won for term 4
I20260129 02:59:58.040796 22645 leader_election.cc:290] T 4b1876f9a70b49edb3aa74105e66d3b3 P 197b880fa39d432a9b472845876bfaf0 [CANDIDATE]: Term 5 election: Requested vote from peers 974b802141bd4f13beffef2aa03f88f9 (127.18.211.2:34517), 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3:37735)
I20260129 02:59:58.041963 22851 leader_election.cc:290] T 724a236524cb4e7ba3428880681d6cc2 P 197b880fa39d432a9b472845876bfaf0 [CANDIDATE]: Term 5 election: Requested vote from peers 974b802141bd4f13beffef2aa03f88f9 (127.18.211.2:34517), 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3:37735)
I20260129 02:59:58.022229 22995 raft_consensus.cc:2804] T 926a250fa5834e88b96ccecbefbf5086 P 1c50de89a5a845668696dcb28c3badd6 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260129 02:59:58.046422 20556 leader_election.cc:304] T cb4727386a3e48aca5d71cc6c611c197 P 1c50de89a5a845668696dcb28c3badd6 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1c50de89a5a845668696dcb28c3badd6; no voters: 197b880fa39d432a9b472845876bfaf0, 974b802141bd4f13beffef2aa03f88f9
I20260129 02:59:58.035640 22900 consensus_queue.cc:260] T 0340229d521b432580ae926f6062e3f7 P 1c50de89a5a845668696dcb28c3badd6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 5.4, Last appended by leader: 4, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
W20260129 02:59:58.080020 20407 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.211.2:34517, user_credentials={real_user=slave}} blocked reactor thread for 51954.3us
W20260129 02:59:58.081436 20581 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:35752 (request call id 1611) took 1607 ms (client timeout 1501 ms). Trace:
W20260129 02:59:58.088501 20480 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.211.3:37735, user_credentials={real_user=slave}} blocked reactor thread for 66280.6us
I20260129 02:59:58.090497 20407 leader_election.cc:304] T 4408006eb5174de1896b037cf3f18457 P 197b880fa39d432a9b472845876bfaf0 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 197b880fa39d432a9b472845876bfaf0, 974b802141bd4f13beffef2aa03f88f9; no voters:
W20260129 02:59:58.088935 20581 rpcz_store.cc:269] 0129 02:59:56.443547 (+ 0us) service_pool.cc:168] Inserting onto call queue
0129 02:59:56.470734 (+ 27187us) service_pool.cc:225] Handling call
0129 02:59:58.050547 (+1579813us) spinlock_profiling.cc:243] Waited 341 ms on lock 0x7b5400134554. stack: 00007f8fe044e373 00007f8fe044e270 00007f8fdfda0574 00007f8fe9a5c6b4 00007f8fe9a5c649 00007f8fe34975a1 00007f8fe3531f7a 00007f8fe84e3647 00007f8fe1a73834 00007f8fe1a737c2 00007f8fe1a736f1 00007f8fe1a7366c 00007f8fe1a72922 00007f8fe186d34c 00007f8fe186c786 00007f8fe186c12f
0129 02:59:58.050701 (+ 154us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":785037696}
I20260129 02:59:58.030097 22986 raft_consensus.cc:2764] T 96609ac68bdf4993a12e1c23355ca013 P 974b802141bd4f13beffef2aa03f88f9 [term 6 FOLLOWER]: Leader pre-election decision vote started in defunct term 5: won
I20260129 02:59:58.047708 22989 raft_consensus.cc:3060] T cedce331e1584ac8b636c9b8a72b2ba0 P 974b802141bd4f13beffef2aa03f88f9 [term 6 FOLLOWER]: Advancing to term 7
I20260129 02:59:58.017824 22888 consensus_queue.cc:1059] T c891d2bff4ce4568861bbc992e91fbb0 P 974b802141bd4f13beffef2aa03f88f9 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 }, Status: INVALID_TERM, Last received: 6.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.063s
I20260129 02:59:58.053033 22857 raft_consensus.cc:2804] T 505580c93c694bbb8ea96fdf9b17ab46 P 7a9daf5d42a84615a6383cfec9a10608 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260129 02:59:58.023059 22961 consensus_queue.cc:1059] T a0fcc58d6fb6473cafbd9d34d5195269 P 974b802141bd4f13beffef2aa03f88f9 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 }, Status: INVALID_TERM, Last received: 3.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.064s
I20260129 02:59:58.096305 20556 leader_election.cc:304] T 7d9494f5584d4d5db8b9adf1c8fc888b P 1c50de89a5a845668696dcb28c3badd6 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1c50de89a5a845668696dcb28c3badd6, 974b802141bd4f13beffef2aa03f88f9; no voters: 197b880fa39d432a9b472845876bfaf0
I20260129 02:59:58.000718 23037 raft_consensus.cc:2749] T 45cc5f3d2688486d9b6dbe7e666dedde P 7a9daf5d42a84615a6383cfec9a10608 [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260129 02:59:58.054121 22903 raft_consensus.cc:2749] T 17d6431cc77a491e8464d68db4081a51 P 1c50de89a5a845668696dcb28c3badd6 [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260129 02:59:58.105026 20514 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4b1876f9a70b49edb3aa74105e66d3b3" candidate_uuid: "197b880fa39d432a9b472845876bfaf0" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "974b802141bd4f13beffef2aa03f88f9"
I20260129 02:59:58.111014 21133 raft_consensus.cc:2468] T add8d4923d1a489e86882c3a25692b95 P 7a9daf5d42a84615a6383cfec9a10608 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1c50de89a5a845668696dcb28c3badd6 in term 5.
I20260129 02:59:58.101840 20511 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "724a236524cb4e7ba3428880681d6cc2" candidate_uuid: "197b880fa39d432a9b472845876bfaf0" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "974b802141bd4f13beffef2aa03f88f9"
I20260129 02:59:58.122977 20556 leader_election.cc:304] T a644c02b16af47679824ccd060ac10fd P 1c50de89a5a845668696dcb28c3badd6 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1c50de89a5a845668696dcb28c3badd6, 974b802141bd4f13beffef2aa03f88f9; no voters: 7a9daf5d42a84615a6383cfec9a10608
I20260129 02:59:58.122876 22654 consensus_queue.cc:1059] T 4f5a77e94fef4b16a2ba7b067c8b144a P 1c50de89a5a845668696dcb28c3badd6 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 }, Status: INVALID_TERM, Last received: 3.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
I20260129 02:59:58.123710 22728 raft_consensus.cc:493] T e33a417cf9c54cf5af73a4ee3aac4d3a P 1c50de89a5a845668696dcb28c3badd6 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260129 02:59:58.086477 22980 raft_consensus.cc:697] T 8055232cf0004d40a85f3bf983fe00a9 P 197b880fa39d432a9b472845876bfaf0 [term 4 LEADER]: Becoming Leader. State: Replica: 197b880fa39d432a9b472845876bfaf0, State: Running, Role: LEADER
I20260129 02:59:58.083976 23032 raft_consensus.cc:515] T 45cc5f3d2688486d9b6dbe7e666dedde P 974b802141bd4f13beffef2aa03f88f9 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } }
W20260129 02:59:58.141647 20405 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.211.3:37735, user_credentials={real_user=slave}} blocked reactor thread for 95895.5us
I20260129 02:59:58.084636 22995 raft_consensus.cc:493] T 926a250fa5834e88b96ccecbefbf5086 P 1c50de89a5a845668696dcb28c3badd6 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260129 02:59:58.093573 22922 raft_consensus.cc:2749] T cb4727386a3e48aca5d71cc6c611c197 P 1c50de89a5a845668696dcb28c3badd6 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20260129 02:59:58.136632 20554 leader_election.cc:304] T add8d4923d1a489e86882c3a25692b95 P 1c50de89a5a845668696dcb28c3badd6 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1c50de89a5a845668696dcb28c3badd6, 7a9daf5d42a84615a6383cfec9a10608; no voters:
W20260129 02:59:58.138574 20479 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.211.1:42027, user_credentials={real_user=slave}} blocked reactor thread for 46061.6us
I20260129 02:59:58.166874 20581 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "42d27a28d89b409e8807ec0efd97af69" candidate_uuid: "7a9daf5d42a84615a6383cfec9a10608" candidate_term: 7 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "1c50de89a5a845668696dcb28c3badd6"
I20260129 02:59:58.127015 20440 raft_consensus.cc:1275] T 18f30ebaa4174fd28c958e72d0cc7e24 P 197b880fa39d432a9b472845876bfaf0 [term 6 FOLLOWER]: Refusing update from remote peer 1c50de89a5a845668696dcb28c3badd6: Log matching property violated. Preceding OpId in replica: term: 4 index: 6. Preceding OpId from leader: term: 6 index: 7. (index mismatch)
I20260129 02:59:58.128688 21099 leader_election.cc:304] T 4e3805ffa70242e8ade66da260dddf66 P 7a9daf5d42a84615a6383cfec9a10608 [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7a9daf5d42a84615a6383cfec9a10608; no voters: 197b880fa39d432a9b472845876bfaf0, 1c50de89a5a845668696dcb28c3badd6
I20260129 02:59:58.133198 22924 raft_consensus.cc:2804] T 7d9494f5584d4d5db8b9adf1c8fc888b P 1c50de89a5a845668696dcb28c3badd6 [term 4 FOLLOWER]: Leader election won for term 4
I20260129 02:59:58.137068 22891 raft_consensus.cc:2804] T a644c02b16af47679824ccd060ac10fd P 1c50de89a5a845668696dcb28c3badd6 [term 5 FOLLOWER]: Leader election won for term 5
I20260129 02:59:58.145923 20508 raft_consensus.cc:1240] T 913fc7d751ed4b7388d889aac30a5f9b P 974b802141bd4f13beffef2aa03f88f9 [term 4 FOLLOWER]: Rejecting Update request from peer 197b880fa39d432a9b472845876bfaf0 for earlier term 3. Current term is 4. Ops: []
I20260129 02:59:58.092016 23034 raft_consensus.cc:697] T 0f20af7fbe5d4db89a6f17948d98912c P 974b802141bd4f13beffef2aa03f88f9 [term 6 LEADER]: Becoming Leader. State: Replica: 974b802141bd4f13beffef2aa03f88f9, State: Running, Role: LEADER
I20260129 02:59:58.165690 20405 leader_election.cc:304] T 4e3805ffa70242e8ade66da260dddf66 P 197b880fa39d432a9b472845876bfaf0 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 197b880fa39d432a9b472845876bfaf0, 1c50de89a5a845668696dcb28c3badd6; no voters:
I20260129 02:59:58.136776 22728 raft_consensus.cc:515] T e33a417cf9c54cf5af73a4ee3aac4d3a P 1c50de89a5a845668696dcb28c3badd6 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:59:58.150861 22995 raft_consensus.cc:3060] T 926a250fa5834e88b96ccecbefbf5086 P 1c50de89a5a845668696dcb28c3badd6 [term 6 FOLLOWER]: Advancing to term 7
I20260129 02:59:58.118240 22851 raft_consensus.cc:2804] T 4408006eb5174de1896b037cf3f18457 P 197b880fa39d432a9b472845876bfaf0 [term 4 FOLLOWER]: Leader election won for term 4
I20260129 02:59:58.143249 20439 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "42d27a28d89b409e8807ec0efd97af69" candidate_uuid: "7a9daf5d42a84615a6383cfec9a10608" candidate_term: 7 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "197b880fa39d432a9b472845876bfaf0"
I20260129 02:59:58.133917 22884 consensus_queue.cc:1059] T 724a236524cb4e7ba3428880681d6cc2 P 1c50de89a5a845668696dcb28c3badd6 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 }, Status: INVALID_TERM, Last received: 2.2, Next index: 3, Last known committed idx: 1, Time since last communication: 0.029s
I20260129 02:59:58.170989 21132 raft_consensus.cc:1275] T 4e3805ffa70242e8ade66da260dddf66 P 7a9daf5d42a84615a6383cfec9a10608 [term 5 FOLLOWER]: Refusing update from remote peer 1c50de89a5a845668696dcb28c3badd6: Log matching property violated. Preceding OpId in replica: term: 3 index: 6. Preceding OpId from leader: term: 5 index: 7. (index mismatch)
I20260129 02:59:58.135120 23055 raft_consensus.cc:2764] T 7d9494f5584d4d5db8b9adf1c8fc888b P 974b802141bd4f13beffef2aa03f88f9 [term 4 FOLLOWER]: Leader pre-election decision vote started in defunct term 3: won
I20260129 02:59:58.129024 22857 raft_consensus.cc:493] T 505580c93c694bbb8ea96fdf9b17ab46 P 7a9daf5d42a84615a6383cfec9a10608 [term 5 FOLLOWER]: Starting leader election (detected failure of leader 1c50de89a5a845668696dcb28c3badd6)
I20260129 02:59:58.080336 22967 consensus_queue.cc:260] T 2ac659d732314e1288849f08fe5e074e P 974b802141bd4f13beffef2aa03f88f9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.5, Last appended by leader: 5, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } }
I20260129 02:59:58.093393 22992 consensus_queue.cc:1059] T a644c02b16af47679824ccd060ac10fd P 7a9daf5d42a84615a6383cfec9a10608 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 }, Status: INVALID_TERM, Last received: 4.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.056s
I20260129 02:59:58.118773 22708 raft_consensus.cc:515] T fd8311b779af4e1fa1402b2a02a602f2 P 1c50de89a5a845668696dcb28c3badd6 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } }
W20260129 02:59:58.222378 21099 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.211.1:42027, user_credentials={real_user=slave}} blocked reactor thread for 53793.2us
I20260129 02:59:58.121222 23010 consensus_queue.cc:1048] T 7a9a69cd82724b1faed1d10aa091fc4a P 1c50de89a5a845668696dcb28c3badd6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 2, Time since last communication: 0.047s
I20260129 02:59:58.147389 22831 leader_election.cc:290] T 42d27a28d89b409e8807ec0efd97af69 P 7a9daf5d42a84615a6383cfec9a10608 [CANDIDATE]: Term 7 election: Requested vote from peers 197b880fa39d432a9b472845876bfaf0 (127.18.211.1:42027), 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3:37735)
I20260129 02:59:58.137720 22833 raft_consensus.cc:2749] T add8d4923d1a489e86882c3a25692b95 P 7a9daf5d42a84615a6383cfec9a10608 [term 5 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
W20260129 02:59:58.245896 20553 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.211.1:42027, user_credentials={real_user=slave}} blocked reactor thread for 75875.7us
I20260129 02:59:58.247284 22891 raft_consensus.cc:697] T a644c02b16af47679824ccd060ac10fd P 1c50de89a5a845668696dcb28c3badd6 [term 5 LEADER]: Becoming Leader. State: Replica: 1c50de89a5a845668696dcb28c3badd6, State: Running, Role: LEADER
I20260129 02:59:58.150283 22982 consensus_queue.cc:1059] T 4b1876f9a70b49edb3aa74105e66d3b3 P 974b802141bd4f13beffef2aa03f88f9 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 }, Status: INVALID_TERM, Last received: 4.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.097s
I20260129 02:59:58.218689 22903 raft_consensus.cc:2804] T add8d4923d1a489e86882c3a25692b95 P 1c50de89a5a845668696dcb28c3badd6 [term 5 FOLLOWER]: Leader election won for term 5
I20260129 02:59:58.219831 22900 raft_consensus.cc:3060] T 0340229d521b432580ae926f6062e3f7 P 1c50de89a5a845668696dcb28c3badd6 [term 5 FOLLOWER]: Advancing to term 6
W20260129 02:59:58.261328 20480 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.211.3:37735, user_credentials={real_user=slave}} blocked reactor thread for 129591us
I20260129 02:59:58.149201 22763 consensus_queue.cc:1059] T 75f45538dc704d9a8ec4ff8ad9f3ea5b P 974b802141bd4f13beffef2aa03f88f9 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 }, Status: INVALID_TERM, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.106s
I20260129 02:59:58.241561 23037 raft_consensus.cc:2749] T 4e3805ffa70242e8ade66da260dddf66 P 7a9daf5d42a84615a6383cfec9a10608 [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20260129 02:59:58.301359 20590 raft_consensus.cc:1240] T fd8311b779af4e1fa1402b2a02a602f2 P 1c50de89a5a845668696dcb28c3badd6 [term 5 FOLLOWER]: Rejecting Update request from peer 7a9daf5d42a84615a6383cfec9a10608 for earlier term 4. Current term is 5. Ops: []
I20260129 02:59:58.242712 22986 raft_consensus.cc:3055] T a0fcc58d6fb6473cafbd9d34d5195269 P 974b802141bd4f13beffef2aa03f88f9 [term 3 LEADER]: Stepping down as leader of term 3
W20260129 02:59:58.250566 20405 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.211.3:37735, user_credentials={real_user=slave}} blocked reactor thread for 84937.8us
I20260129 02:59:58.127712 23051 consensus_queue.cc:1048] T 76f713c3331540d2a5b76164103e9eb1 P 1c50de89a5a845668696dcb28c3badd6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.143s
W20260129 02:59:58.334193 20479 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.211.1:42027, user_credentials={real_user=slave}} blocked reactor thread for 78975.7us
I20260129 02:59:58.251731 22857 raft_consensus.cc:3060] T 505580c93c694bbb8ea96fdf9b17ab46 P 7a9daf5d42a84615a6383cfec9a10608 [term 5 FOLLOWER]: Advancing to term 6
I20260129 02:59:58.253690 22851 raft_consensus.cc:697] T 4408006eb5174de1896b037cf3f18457 P 197b880fa39d432a9b472845876bfaf0 [term 4 LEADER]: Becoming Leader. State: Replica: 197b880fa39d432a9b472845876bfaf0, State: Running, Role: LEADER
I20260129 02:59:58.256651 22581 raft_consensus.cc:493] T 0f20af7fbe5d4db89a6f17948d98912c P 1c50de89a5a845668696dcb28c3badd6 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260129 02:59:58.251977 22922 consensus_queue.cc:1048] T 18f30ebaa4174fd28c958e72d0cc7e24 P 1c50de89a5a845668696dcb28c3badd6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.057s
I20260129 02:59:58.275506 22884 raft_consensus.cc:3055] T 724a236524cb4e7ba3428880681d6cc2 P 1c50de89a5a845668696dcb28c3badd6 [term 4 LEADER]: Stepping down as leader of term 4
W20260129 02:59:58.282423 23050 log.cc:927] Time spent T a1a39dcd328e4268ae41b988c66b4a75 P 974b802141bd4f13beffef2aa03f88f9: Append to log took a long time: real 0.188s user 0.001s sys 0.000s
W20260129 02:59:58.363903 20554 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260129 02:59:58.338044 22986 raft_consensus.cc:740] T a0fcc58d6fb6473cafbd9d34d5195269 P 974b802141bd4f13beffef2aa03f88f9 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 974b802141bd4f13beffef2aa03f88f9, State: Running, Role: LEADER
I20260129 02:59:58.231034 23005 consensus_queue.cc:237] T b189f08e16a74a11980cc55d99359a0a P 1c50de89a5a845668696dcb28c3badd6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:59:58.419919 22884 raft_consensus.cc:740] T 724a236524cb4e7ba3428880681d6cc2 P 1c50de89a5a845668696dcb28c3badd6 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 1c50de89a5a845668696dcb28c3badd6, State: Running, Role: LEADER
I20260129 02:59:58.309109 22924 raft_consensus.cc:697] T 7d9494f5584d4d5db8b9adf1c8fc888b P 1c50de89a5a845668696dcb28c3badd6 [term 4 LEADER]: Becoming Leader. State: Replica: 1c50de89a5a845668696dcb28c3badd6, State: Running, Role: LEADER
W20260129 02:59:58.422996 21101 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260129 02:59:58.433677 21132 raft_consensus.cc:1240] T febffe4a8b394368b69b1d164308fb9a P 7a9daf5d42a84615a6383cfec9a10608 [term 4 FOLLOWER]: Rejecting Update request from peer 197b880fa39d432a9b472845876bfaf0 for earlier term 3. Current term is 4. Ops: [3.5-3.5]
I20260129 02:59:58.267052 23057 raft_consensus.cc:2764] T 4e3805ffa70242e8ade66da260dddf66 P 197b880fa39d432a9b472845876bfaf0 [term 5 FOLLOWER]: Leader election decision vote started in defunct term 4: won
W20260129 02:59:58.445901 21099 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.211.1:42027, user_credentials={real_user=slave}} blocked reactor thread for 179121us
I20260129 02:59:58.409791 23059 raft_consensus.cc:3055] T a644c02b16af47679824ccd060ac10fd P 7a9daf5d42a84615a6383cfec9a10608 [term 4 LEADER]: Stepping down as leader of term 4
W20260129 02:59:58.426823 23068 log.cc:927] Time spent T 75b0b6e82ddc4f92b1fe1d94f9a0b73b P 974b802141bd4f13beffef2aa03f88f9: Append to log took a long time: real 0.181s user 0.001s sys 0.000s
W20260129 02:59:58.432658 23069 log.cc:927] Time spent T 3ccac82a01a2445fa5b0ce6c0fcf2295 P 197b880fa39d432a9b472845876bfaf0: Append to log took a long time: real 0.194s user 0.000s sys 0.000s
W20260129 02:59:58.520954 20407 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260129 02:59:58.421159 22855 raft_consensus.cc:493] T 45cc5f3d2688486d9b6dbe7e666dedde P 1c50de89a5a845668696dcb28c3badd6 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260129 02:59:58.364147 22831 consensus_queue.cc:1059] T 00e79d8f25a34676b1d866eb2068a2b2 P 7a9daf5d42a84615a6383cfec9a10608 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 }, Status: INVALID_TERM, Last received: 6.8, Next index: 9, Last known committed idx: 6, Time since last communication: 0.081s
I20260129 02:59:58.431164 22708 leader_election.cc:290] T fd8311b779af4e1fa1402b2a02a602f2 P 1c50de89a5a845668696dcb28c3badd6 [CANDIDATE]: Term 5 election: Requested vote from peers 197b880fa39d432a9b472845876bfaf0 (127.18.211.1:42027), 7a9daf5d42a84615a6383cfec9a10608 (127.18.211.4:40439)
I20260129 02:59:58.531559 21132 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a609ac02fae84725ab743f95f51d33ce" candidate_uuid: "197b880fa39d432a9b472845876bfaf0" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "7a9daf5d42a84615a6383cfec9a10608" is_pre_election: true
I20260129 02:59:58.588039 21132 raft_consensus.cc:2393] T a609ac02fae84725ab743f95f51d33ce P 7a9daf5d42a84615a6383cfec9a10608 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 197b880fa39d432a9b472845876bfaf0 in current term 7: Already voted for candidate 7a9daf5d42a84615a6383cfec9a10608 in this term.
I20260129 02:59:58.339393 22231 consensus_queue.cc:1059] T 913fc7d751ed4b7388d889aac30a5f9b P 197b880fa39d432a9b472845876bfaf0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 }, Status: INVALID_TERM, Last received: 3.3, Next index: 4, Last known committed idx: 1, Time since last communication: 0.097s
W20260129 02:59:58.536871 20407 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260129 02:59:58.531383 20480 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.211.3:37735, user_credentials={real_user=slave}} blocked reactor thread for 113184us
W20260129 02:59:58.588454 21099 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.211.1:42027, user_credentials={real_user=slave}} blocked reactor thread for 107618us
W20260129 02:59:58.550639 20405 connection.cc:232] RPC call timeout handler was delayed by -3.9214400390000037s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1655s for a response
W20260129 02:59:58.567274 20554 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.211.4:40439, user_credentials={real_user=slave}} blocked reactor thread for 133173us
W20260129 02:59:58.589931 20405 connection.cc:232] RPC call timeout handler was delayed by -3.7711400389999881s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.18220000000000003s for a response
W20260129 02:59:58.590294 20405 connection.cc:232] RPC call timeout handler was delayed by -3.7495400390000668s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.18460000000000001s for a response
I20260129 02:59:58.589336 21099 leader_election.cc:304] T 42d27a28d89b409e8807ec0efd97af69 P 7a9daf5d42a84615a6383cfec9a10608 [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7a9daf5d42a84615a6383cfec9a10608; no voters: 197b880fa39d432a9b472845876bfaf0, 1c50de89a5a845668696dcb28c3badd6
W20260129 02:59:58.590639 20405 connection.cc:232] RPC call timeout handler was delayed by -3.5056400390000135s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2117s for a response
W20260129 02:59:58.591056 20405 connection.cc:232] RPC call timeout handler was delayed by -1.5913400389999879s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.4244s for a response
I20260129 02:59:58.443234 22903 raft_consensus.cc:697] T add8d4923d1a489e86882c3a25692b95 P 1c50de89a5a845668696dcb28c3badd6 [term 5 LEADER]: Becoming Leader. State: Replica: 1c50de89a5a845668696dcb28c3badd6, State: Running, Role: LEADER
W20260129 02:59:58.612727 20479 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.211.1:42027, user_credentials={real_user=slave}} blocked reactor thread for 171532us
I20260129 02:59:58.459702 23059 raft_consensus.cc:740] T a644c02b16af47679824ccd060ac10fd P 7a9daf5d42a84615a6383cfec9a10608 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 7a9daf5d42a84615a6383cfec9a10608, State: Running, Role: LEADER
I20260129 02:59:58.425722 22581 raft_consensus.cc:515] T 0f20af7fbe5d4db89a6f17948d98912c P 1c50de89a5a845668696dcb28c3badd6 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:59:58.527276 22922 raft_consensus.cc:493] T 4f8f3a38f6e24ba4857875181f447442 P 1c50de89a5a845668696dcb28c3badd6 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260129 02:59:58.300103 22980 consensus_queue.cc:237] T 8055232cf0004d40a85f3bf983fe00a9 P 197b880fa39d432a9b472845876bfaf0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 3.2, Last appended by leader: 2, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:59:58.618381 20507 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57584 (request call id 1377) took 1231 ms. Trace:
W20260129 02:59:58.626819 23060 log.cc:927] Time spent T 505580c93c694bbb8ea96fdf9b17ab46 P 974b802141bd4f13beffef2aa03f88f9: Append to log took a long time: real 0.368s user 0.000s sys 0.009s
I20260129 02:59:58.537863 22977 raft_consensus.cc:493] T 4b1876f9a70b49edb3aa74105e66d3b3 P 1c50de89a5a845668696dcb28c3badd6 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 974b802141bd4f13beffef2aa03f88f9)
I20260129 02:59:58.445734 22962 consensus_queue.cc:1048] T 3906b080a5b34a74b8b64128635e8997 P 974b802141bd4f13beffef2aa03f88f9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.146s
I20260129 02:59:58.596717 21129 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "dc2f4cb752cf4d59b5f6bc689f5d8104" candidate_uuid: "197b880fa39d432a9b472845876bfaf0" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "7a9daf5d42a84615a6383cfec9a10608" is_pre_election: true
I20260129 02:59:58.597018 22925 raft_consensus.cc:493] T 926a250fa5834e88b96ccecbefbf5086 P 197b880fa39d432a9b472845876bfaf0 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260129 02:59:58.617125 23010 raft_consensus.cc:493] T a4c1226b987b468fa84dfc6ba0e2dbe1 P 1c50de89a5a845668696dcb28c3badd6 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 197b880fa39d432a9b472845876bfaf0)
I20260129 02:59:58.735050 22748 raft_consensus.cc:493] T 1299723a4476420ab02fd8bfd7aa0645 P 1c50de89a5a845668696dcb28c3badd6 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 7a9daf5d42a84615a6383cfec9a10608)
W20260129 02:59:58.735816 23073 log.cc:927] Time spent T 76f713c3331540d2a5b76164103e9eb1 P 197b880fa39d432a9b472845876bfaf0: Append to log took a long time: real 0.282s user 0.000s sys 0.000s
I20260129 02:59:58.428166 22884 consensus_queue.cc:260] T 724a236524cb4e7ba3428880681d6cc2 P 1c50de89a5a845668696dcb28c3badd6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 4.3, Last appended by leader: 3, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:59:58.736770 21133 raft_consensus.cc:1240] T 0678b8b468bd43cdbc3e485900844821 P 7a9daf5d42a84615a6383cfec9a10608 [term 6 FOLLOWER]: Rejecting Update request from peer 197b880fa39d432a9b472845876bfaf0 for earlier term 5. Current term is 6. Ops: []
I20260129 02:59:58.738125 20440 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fd8311b779af4e1fa1402b2a02a602f2" candidate_uuid: "1c50de89a5a845668696dcb28c3badd6" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "197b880fa39d432a9b472845876bfaf0"
I20260129 02:59:58.851843 20440 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e33a417cf9c54cf5af73a4ee3aac4d3a" candidate_uuid: "1c50de89a5a845668696dcb28c3badd6" candidate_term: 7 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "197b880fa39d432a9b472845876bfaf0" is_pre_election: true
I20260129 02:59:58.697525 20507 rpcz_store.cc:276] 0129 02:59:57.387215 (+ 0us) service_pool.cc:168] Inserting onto call queue
0129 02:59:57.625993 (+238778us) service_pool.cc:225] Handling call
0129 02:59:57.626269 (+ 276us) raft_consensus.cc:1471] Updating replica for 1 ops
0129 02:59:57.626624 (+ 355us) raft_consensus.cc:1534] Early marking committed up to index 0
0129 02:59:57.626651 (+ 27us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0129 02:59:57.627124 (+ 473us) log.cc:844] Serialized 26 byte log entry
0129 02:59:57.755207 (+128083us) raft_consensus.cc:1645] Marking committed up to 2
0129 02:59:57.755248 (+ 41us) raft_consensus.cc:1697] Filling consensus response to leader.
0129 02:59:57.755322 (+ 74us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0129 02:59:58.618077 (+862755us) raft_consensus.cc:1685] finished
0129 02:59:58.618107 (+ 30us) raft_consensus.cc:1691] UpdateReplicas() finished
0129 02:59:58.618301 (+ 194us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":17421,"raft.run_cpu_time_us":170,"raft.run_wall_time_us":15258,"thread_start_us":127713,"threads_started":1,"wal-append.queue_time_us":387138}
I20260129 02:59:58.628451 22667 raft_consensus.cc:2749] T 42d27a28d89b409e8807ec0efd97af69 P 7a9daf5d42a84615a6383cfec9a10608 [term 7 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
I20260129 02:59:58.594542 22953 consensus_queue.cc:1048] T 4e3805ffa70242e8ade66da260dddf66 P 1c50de89a5a845668696dcb28c3badd6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.135s
I20260129 02:59:58.415724 23034 consensus_queue.cc:237] T 0f20af7fbe5d4db89a6f17948d98912c P 974b802141bd4f13beffef2aa03f88f9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:59:58.597285 22855 raft_consensus.cc:515] T 45cc5f3d2688486d9b6dbe7e666dedde P 1c50de89a5a845668696dcb28c3badd6 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } }
W20260129 02:59:58.813684 20480 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.211.3:37735, user_credentials={real_user=slave}} blocked reactor thread for 224158us
I20260129 02:59:58.661231 22708 raft_consensus.cc:493] T 36ff33be666a4c82954731691dbcf327 P 1c50de89a5a845668696dcb28c3badd6 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260129 02:59:58.621798 22851 consensus_queue.cc:237] T 4408006eb5174de1896b037cf3f18457 P 197b880fa39d432a9b472845876bfaf0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 2.2, Last appended by leader: 2, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:59:58.739477 22977 raft_consensus.cc:515] T 4b1876f9a70b49edb3aa74105e66d3b3 P 1c50de89a5a845668696dcb28c3badd6 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
W20260129 02:59:58.817176 23061 log.cc:927] Time spent T 75f45538dc704d9a8ec4ff8ad9f3ea5b P 1c50de89a5a845668696dcb28c3badd6: Append to log took a long time: real 0.190s user 0.000s sys 0.000s
I20260129 02:59:58.817610 22903 consensus_queue.cc:237] T add8d4923d1a489e86882c3a25692b95 P 1c50de89a5a845668696dcb28c3badd6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.6, Last appended by leader: 6, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } }
I20260129 02:59:58.482244 22891 consensus_queue.cc:237] T a644c02b16af47679824ccd060ac10fd P 1c50de89a5a845668696dcb28c3badd6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } }
I20260129 02:59:58.430150 22924 consensus_queue.cc:237] T 7d9494f5584d4d5db8b9adf1c8fc888b P 1c50de89a5a845668696dcb28c3badd6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:59:58.662243 23032 leader_election.cc:290] T 45cc5f3d2688486d9b6dbe7e666dedde P 974b802141bd4f13beffef2aa03f88f9 [CANDIDATE]: Term 5 election: Requested vote from peers 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3:37735), 7a9daf5d42a84615a6383cfec9a10608 (127.18.211.4:40439)
W20260129 02:59:58.731737 23062 log.cc:927] Time spent T 724a236524cb4e7ba3428880681d6cc2 P 974b802141bd4f13beffef2aa03f88f9: Append to log took a long time: real 0.201s user 0.000s sys 0.001s
I20260129 02:59:58.628924 23051 consensus_queue.cc:1059] T 42d27a28d89b409e8807ec0efd97af69 P 1c50de89a5a845668696dcb28c3badd6 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false }, Status: INVALID_TERM, Last received: 6.8, Next index: 9, Last known committed idx: 8, Time since last communication: 0.218s
I20260129 02:59:59.104254 20434 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:48356 (request call id 344) took 1173 ms. Trace:
I20260129 02:59:58.912241 19276 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
W20260129 02:59:58.876336 20554 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260129 02:59:59.115150 20554 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260129 02:59:59.103622 20440 raft_consensus.cc:1240] T 368953c9102643b2a21f99af36b51905 P 197b880fa39d432a9b472845876bfaf0 [term 7 FOLLOWER]: Rejecting Update request from peer 1c50de89a5a845668696dcb28c3badd6 for earlier term 6. Current term is 7. Ops: []
W20260129 02:59:59.110352 20405 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.211.4:40439, user_credentials={real_user=slave}} blocked reactor thread for 513952us
I20260129 02:59:58.909013 21131 raft_consensus.cc:1275] T c934d31efd3f4e5195605ddc354d14e1 P 7a9daf5d42a84615a6383cfec9a10608 [term 5 FOLLOWER]: Refusing update from remote peer 197b880fa39d432a9b472845876bfaf0: Log matching property violated. Preceding OpId in replica: term: 3 index: 6. Preceding OpId from leader: term: 5 index: 7. (index mismatch)
W20260129 02:59:59.169941 20406 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260129 02:59:58.960848 20512 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57600 (request call id 1820) took 1685 ms. Trace:
I20260129 02:59:59.183907 20512 rpcz_store.cc:276] 0129 02:59:57.275625 (+ 0us) service_pool.cc:168] Inserting onto call queue
0129 02:59:57.275765 (+ 140us) service_pool.cc:225] Handling call
0129 02:59:57.276084 (+ 319us) raft_consensus.cc:1471] Updating replica for 1 ops
0129 02:59:57.462022 (+185938us) raft_consensus.cc:1534] Early marking committed up to index 0
0129 02:59:57.462070 (+ 48us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0129 02:59:57.528183 (+ 66113us) log.cc:844] Serialized 26 byte log entry
0129 02:59:57.963395 (+435212us) raft_consensus.cc:1645] Marking committed up to 2
0129 02:59:57.963453 (+ 58us) raft_consensus.cc:1697] Filling consensus response to leader.
0129 02:59:58.045983 (+ 82530us) spinlock_profiling.cc:243] Waited 171 ms on lock 0x7b540028f354. stack: 00007f8fe044e373 00007f8fe044e270 00007f8fdfda0574 00007f8fe9a5c6b4 00007f8fe9a5c649 00007f8fe34975a1 00007f8fe352ea81 00007f8fe352d4be 00007f8fe84e1f49 00007f8fe1a6ed34 00007f8fe1a6ecc2 00007f8fe1a6ebf1 00007f8fe1a6eb6c 00007f8fe1a6de22 00007f8fe186d34c 00007f8fe186c786
0129 02:59:58.046033 (+ 50us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0129 02:59:58.909804 (+863771us) raft_consensus.cc:1685] finished
0129 02:59:58.909835 (+ 31us) raft_consensus.cc:1691] UpdateReplicas() finished
0129 02:59:58.960749 (+ 50914us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":26735,"raft.queue_time_us":93,"raft.run_cpu_time_us":216,"raft.run_wall_time_us":26429,"spinlock_wait_cycles":393242368,"thread_start_us":382009,"threads_started":1,"wal-append.queue_time_us":716950}
I20260129 02:59:59.111361 20434 rpcz_store.cc:276] 0129 02:59:57.930657 (+ 0us) service_pool.cc:168] Inserting onto call queue
0129 02:59:57.931063 (+ 406us) service_pool.cc:225] Handling call
0129 02:59:57.931364 (+ 301us) raft_consensus.cc:1471] Updating replica for 1 ops
0129 02:59:57.931747 (+ 383us) raft_consensus.cc:1534] Early marking committed up to index 0
0129 02:59:57.931772 (+ 25us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0129 02:59:57.932260 (+ 488us) log.cc:844] Serialized 26 byte log entry
0129 02:59:57.971911 (+ 39651us) raft_consensus.cc:1645] Marking committed up to 4
0129 02:59:57.971951 (+ 40us) raft_consensus.cc:1697] Filling consensus response to leader.
0129 02:59:57.972022 (+ 71us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0129 02:59:59.104014 (+1131992us) raft_consensus.cc:1685] finished
0129 02:59:59.104039 (+ 25us) raft_consensus.cc:1691] UpdateReplicas() finished
0129 02:59:59.104192 (+ 153us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":16497,"raft.queue_time_us":5387,"raft.run_cpu_time_us":155,"raft.run_wall_time_us":156,"thread_start_us":39253,"threads_started":1,"wal-append.queue_time_us":208985}
I20260129 02:59:59.192098 20433 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55676 (request call id 1375) took 1233 ms. Trace:
I20260129 02:59:59.192627 20433 rpcz_store.cc:276] 0129 02:59:57.958995 (+ 0us) service_pool.cc:168] Inserting onto call queue
0129 02:59:57.959209 (+ 214us) service_pool.cc:225] Handling call
0129 02:59:57.959598 (+ 389us) raft_consensus.cc:1471] Updating replica for 1 ops
0129 02:59:58.027510 (+ 67912us) raft_consensus.cc:1534] Early marking committed up to index 0
0129 02:59:58.027551 (+ 41us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0129 02:59:58.038303 (+ 10752us) log.cc:844] Serialized 26 byte log entry
0129 02:59:58.227057 (+188754us) raft_consensus.cc:1645] Marking committed up to 2
0129 02:59:58.227117 (+ 60us) raft_consensus.cc:1697] Filling consensus response to leader.
0129 02:59:58.227212 (+ 95us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0129 02:59:59.191755 (+964543us) raft_consensus.cc:1685] finished
0129 02:59:59.191791 (+ 36us) raft_consensus.cc:1691] UpdateReplicas() finished
0129 02:59:59.192019 (+ 228us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":72611,"raft.queue_time_us":59120,"raft.run_cpu_time_us":134,"raft.run_wall_time_us":8472,"thread_start_us":188359,"threads_started":1,"wal-append.queue_time_us":243833}
I20260129 02:59:58.867620 22748 raft_consensus.cc:515] T 1299723a4476420ab02fd8bfd7aa0645 P 1c50de89a5a845668696dcb28c3badd6 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } }
I20260129 02:59:59.205262 21130 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1299723a4476420ab02fd8bfd7aa0645" candidate_uuid: "197b880fa39d432a9b472845876bfaf0" candidate_term: 9 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "7a9daf5d42a84615a6383cfec9a10608" is_pre_election: true
W20260129 02:59:59.206221 22640 consensus_meta.cc:287] T c2633a6fc34c44579de348d208b0c01e P 1c50de89a5a845668696dcb28c3badd6: Time spent flushing consensus metadata: real 1.245s user 0.012s sys 0.000s
I20260129 02:59:59.207115 22950 raft_consensus.cc:3055] T 913fc7d751ed4b7388d889aac30a5f9b P 197b880fa39d432a9b472845876bfaf0 [term 3 LEADER]: Stepping down as leader of term 3
I20260129 02:59:59.086759 22694 raft_consensus.cc:3055] T c891d2bff4ce4568861bbc992e91fbb0 P 974b802141bd4f13beffef2aa03f88f9 [term 6 LEADER]: Stepping down as leader of term 6
I20260129 02:59:59.207449 20586 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:48526 (request call id 308) took 1554 ms. Trace:
I20260129 02:59:59.213434 22694 raft_consensus.cc:740] T c891d2bff4ce4568861bbc992e91fbb0 P 974b802141bd4f13beffef2aa03f88f9 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 974b802141bd4f13beffef2aa03f88f9, State: Running, Role: LEADER
I20260129 02:59:59.213708 20586 rpcz_store.cc:276] 0129 02:59:57.653378 (+ 0us) service_pool.cc:168] Inserting onto call queue
0129 02:59:57.653526 (+ 148us) service_pool.cc:225] Handling call
0129 02:59:57.726493 (+ 72967us) raft_consensus.cc:1471] Updating replica for 1 ops
0129 02:59:57.745477 (+ 18984us) raft_consensus.cc:1534] Early marking committed up to index 0
0129 02:59:57.745512 (+ 35us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0129 02:59:57.755787 (+ 10275us) log.cc:844] Serialized 26 byte log entry
0129 02:59:58.148461 (+392674us) raft_consensus.cc:1645] Marking committed up to 5
0129 02:59:58.148558 (+ 97us) raft_consensus.cc:1697] Filling consensus response to leader.
0129 02:59:58.148710 (+ 152us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0129 02:59:59.207178 (+1058468us) raft_consensus.cc:1685] finished
0129 02:59:59.207212 (+ 34us) raft_consensus.cc:1691] UpdateReplicas() finished
0129 02:59:59.207373 (+ 161us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":9471,"raft.queue_time_us":20943,"raft.run_cpu_time_us":132,"raft.run_wall_time_us":2980,"thread_start_us":351646,"threads_started":1,"wal-append.queue_time_us":652817}
W20260129 02:59:59.215032 22933 consensus_meta.cc:287] T 4e420f90884f4796997b5ea41e49e411 P 7a9daf5d42a84615a6383cfec9a10608: Time spent flushing consensus metadata: real 1.268s user 0.010s sys 0.000s
I20260129 02:59:59.211642 22748 leader_election.cc:290] T 1299723a4476420ab02fd8bfd7aa0645 P 1c50de89a5a845668696dcb28c3badd6 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 197b880fa39d432a9b472845876bfaf0 (127.18.211.1:42027), 7a9daf5d42a84615a6383cfec9a10608 (127.18.211.4:40439)
I20260129 02:59:59.214303 22694 consensus_queue.cc:260] T c891d2bff4ce4568861bbc992e91fbb0 P 974b802141bd4f13beffef2aa03f88f9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 6.5, Last appended by leader: 5, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:59:59.086074 23033 raft_consensus.cc:493] T 0678b8b468bd43cdbc3e485900844821 P 7a9daf5d42a84615a6383cfec9a10608 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260129 02:59:58.855504 22728 leader_election.cc:290] T e33a417cf9c54cf5af73a4ee3aac4d3a P 1c50de89a5a845668696dcb28c3badd6 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 197b880fa39d432a9b472845876bfaf0 (127.18.211.1:42027), 974b802141bd4f13beffef2aa03f88f9 (127.18.211.2:34517)
W20260129 02:59:58.910248 23060 log.cc:927] Time spent T 505580c93c694bbb8ea96fdf9b17ab46 P 974b802141bd4f13beffef2aa03f88f9: Append to log took a long time: real 0.168s user 0.000s sys 0.001s
I20260129 02:59:59.275420 22694 raft_consensus.cc:3060] T c891d2bff4ce4568861bbc992e91fbb0 P 974b802141bd4f13beffef2aa03f88f9 [term 6 FOLLOWER]: Advancing to term 7
I20260129 02:59:59.275569 21130 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00e79d8f25a34676b1d866eb2068a2b2" candidate_uuid: "197b880fa39d432a9b472845876bfaf0" candidate_term: 7 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "7a9daf5d42a84615a6383cfec9a10608" is_pre_election: true
W20260129 02:59:59.215576 22857 consensus_meta.cc:287] T 505580c93c694bbb8ea96fdf9b17ab46 P 7a9daf5d42a84615a6383cfec9a10608: Time spent flushing consensus metadata: real 0.791s user 0.005s sys 0.001s
W20260129 02:59:59.278087 20405 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.211.4:40439, user_credentials={real_user=slave}} blocked reactor thread for 119185us
W20260129 02:59:59.190250 23075 log.cc:927] Time spent T 75be4bd6bd45474795fd472eff2953ac P 974b802141bd4f13beffef2aa03f88f9: Append to log took a long time: real 0.821s user 0.000s sys 0.001s
I20260129 02:59:58.851398 23010 raft_consensus.cc:515] T a4c1226b987b468fa84dfc6ba0e2dbe1 P 1c50de89a5a845668696dcb28c3badd6 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
W20260129 02:59:59.218261 23072 log.cc:927] Time spent T 3e2d15f8640b407195210c476e09df91 P 974b802141bd4f13beffef2aa03f88f9: Append to log took a long time: real 0.482s user 0.000s sys 0.000s
W20260129 02:59:59.216359 23074 log.cc:927] Time spent T 17d6431cc77a491e8464d68db4081a51 P 197b880fa39d432a9b472845876bfaf0: Append to log took a long time: real 0.793s user 0.001s sys 0.000s
W20260129 02:59:59.210561 23043 consensus_meta.cc:287] T 0f20af7fbe5d4db89a6f17948d98912c P 197b880fa39d432a9b472845876bfaf0: Time spent flushing consensus metadata: real 1.270s user 0.002s sys 0.006s
I20260129 02:59:59.207531 22950 raft_consensus.cc:740] T 913fc7d751ed4b7388d889aac30a5f9b P 197b880fa39d432a9b472845876bfaf0 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 197b880fa39d432a9b472845876bfaf0, State: Running, Role: LEADER
W20260129 02:59:59.217139 23063 log.cc:927] Time spent T 4e3805ffa70242e8ade66da260dddf66 P 1c50de89a5a845668696dcb28c3badd6: Append to log took a long time: real 0.475s user 0.000s sys 0.000s
I20260129 02:59:59.215688 22933 raft_consensus.cc:515] T 4e420f90884f4796997b5ea41e49e411 P 7a9daf5d42a84615a6383cfec9a10608 [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } }
I20260129 02:59:58.733194 22922 raft_consensus.cc:515] T 4f8f3a38f6e24ba4857875181f447442 P 1c50de89a5a845668696dcb28c3badd6 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:59:59.362874 22667 raft_consensus.cc:493] T 78aa39096868415d9afc2ffcb248e198 P 7a9daf5d42a84615a6383cfec9a10608 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 1c50de89a5a845668696dcb28c3badd6)
I20260129 02:59:59.377759 21134 raft_consensus.cc:1275] T 18f30ebaa4174fd28c958e72d0cc7e24 P 7a9daf5d42a84615a6383cfec9a10608 [term 6 FOLLOWER]: Refusing update from remote peer 1c50de89a5a845668696dcb28c3badd6: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 6 index: 7. (index mismatch)
W20260129 02:59:59.426546 21101 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260129 02:59:59.347435 20480 leader_election.cc:304] T a8479d1fe0bd41e5946186e3a6ca8c7a P 974b802141bd4f13beffef2aa03f88f9 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1c50de89a5a845668696dcb28c3badd6, 974b802141bd4f13beffef2aa03f88f9; no voters: 197b880fa39d432a9b472845876bfaf0
I20260129 02:59:59.346621 21133 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2e58e9a5a8aa4cecafb464addb4f35ed" candidate_uuid: "197b880fa39d432a9b472845876bfaf0" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "7a9daf5d42a84615a6383cfec9a10608" is_pre_election: true
I20260129 02:59:58.845434 22925 raft_consensus.cc:515] T 926a250fa5834e88b96ccecbefbf5086 P 197b880fa39d432a9b472845876bfaf0 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
W20260129 02:59:59.436466 20405 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.211.4:40439, user_credentials={real_user=slave}} blocked reactor thread for 58325.7us
I20260129 02:59:59.354938 22884 raft_consensus.cc:3060] T 724a236524cb4e7ba3428880681d6cc2 P 1c50de89a5a845668696dcb28c3badd6 [term 4 FOLLOWER]: Advancing to term 5
I20260129 02:59:59.452752 21132 raft_consensus.cc:1240] T 4e420f90884f4796997b5ea41e49e411 P 7a9daf5d42a84615a6383cfec9a10608 [term 6 FOLLOWER]: Rejecting Update request from peer 197b880fa39d432a9b472845876bfaf0 for earlier term 4. Current term is 6. Ops: []
I20260129 02:59:59.293649 23033 raft_consensus.cc:515] T 0678b8b468bd43cdbc3e485900844821 P 7a9daf5d42a84615a6383cfec9a10608 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } }
I20260129 02:59:59.458215 20405 leader_election.cc:304] T 2ac659d732314e1288849f08fe5e074e P 197b880fa39d432a9b472845876bfaf0 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 197b880fa39d432a9b472845876bfaf0; no voters: 7a9daf5d42a84615a6383cfec9a10608, 974b802141bd4f13beffef2aa03f88f9
I20260129 02:59:59.259229 22708 raft_consensus.cc:515] T 36ff33be666a4c82954731691dbcf327 P 1c50de89a5a845668696dcb28c3badd6 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:59:59.364126 22857 raft_consensus.cc:515] T 505580c93c694bbb8ea96fdf9b17ab46 P 7a9daf5d42a84615a6383cfec9a10608 [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } }
I20260129 02:59:59.440500 21129 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "78aa39096868415d9afc2ffcb248e198" candidate_uuid: "197b880fa39d432a9b472845876bfaf0" candidate_term: 6 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "7a9daf5d42a84615a6383cfec9a10608" is_pre_election: true
I20260129 02:59:59.470379 20433 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1299723a4476420ab02fd8bfd7aa0645" candidate_uuid: "1c50de89a5a845668696dcb28c3badd6" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "197b880fa39d432a9b472845876bfaf0" is_pre_election: true
I20260129 02:59:59.471051 20433 raft_consensus.cc:2468] T 1299723a4476420ab02fd8bfd7aa0645 P 197b880fa39d432a9b472845876bfaf0 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1c50de89a5a845668696dcb28c3badd6 in term 8.
I20260129 02:59:59.436484 23087 raft_consensus.cc:493] T 36ff33be666a4c82954731691dbcf327 P 197b880fa39d432a9b472845876bfaf0 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 974b802141bd4f13beffef2aa03f88f9)
I20260129 02:59:59.472723 20516 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57584 (request call id 1395) took 1503 ms. Trace:
I20260129 02:59:59.473479 20516 rpcz_store.cc:276] 0129 02:59:57.968938 (+ 0us) service_pool.cc:168] Inserting onto call queue
0129 02:59:57.973892 (+ 4954us) service_pool.cc:225] Handling call
0129 02:59:57.974307 (+ 415us) raft_consensus.cc:1471] Updating replica for 1 ops
0129 02:59:58.017959 (+ 43652us) raft_consensus.cc:1534] Early marking committed up to index 0
0129 02:59:58.017996 (+ 37us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0129 02:59:58.023725 (+ 5729us) log.cc:844] Serialized 26 byte log entry
0129 02:59:58.124588 (+100863us) raft_consensus.cc:1645] Marking committed up to 5
0129 02:59:58.124642 (+ 54us) raft_consensus.cc:1697] Filling consensus response to leader.
0129 02:59:58.124741 (+ 99us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0129 02:59:59.472463 (+1347722us) raft_consensus.cc:1685] finished
0129 02:59:59.472491 (+ 28us) raft_consensus.cc:1691] UpdateReplicas() finished
0129 02:59:59.472649 (+ 158us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":40161,"raft.queue_time_us":39122,"raft.run_cpu_time_us":102,"raft.run_wall_time_us":5971,"thread_start_us":30557,"threads_started":1,"wal-append.queue_time_us":202170}
W20260129 02:59:59.476295 20407 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.863s user 0.015s sys 0.009s
I20260129 02:59:59.478101 20433 raft_consensus.cc:1275] T b189f08e16a74a11980cc55d99359a0a P 197b880fa39d432a9b472845876bfaf0 [term 8 FOLLOWER]: Refusing update from remote peer 1c50de89a5a845668696dcb28c3badd6: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 8 index: 4. (index mismatch)
I20260129 02:59:59.482211 20433 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0f20af7fbe5d4db89a6f17948d98912c" candidate_uuid: "1c50de89a5a845668696dcb28c3badd6" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "197b880fa39d432a9b472845876bfaf0" is_pre_election: true
I20260129 02:59:59.460202 22581 leader_election.cc:290] T 0f20af7fbe5d4db89a6f17948d98912c P 1c50de89a5a845668696dcb28c3badd6 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 197b880fa39d432a9b472845876bfaf0 (127.18.211.1:42027), 974b802141bd4f13beffef2aa03f88f9 (127.18.211.2:34517)
W20260129 02:59:59.478502 20407 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.866s user 0.016s sys 0.009s
W20260129 02:59:59.472623 20555 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 1.033s user 0.010s sys 0.014s
I20260129 02:59:59.483844 20441 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55672 (request call id 1633) took 1565 ms. Trace:
I20260129 02:59:59.482067 20553 leader_election.cc:304] T 1299723a4476420ab02fd8bfd7aa0645 P 1c50de89a5a845668696dcb28c3badd6 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 197b880fa39d432a9b472845876bfaf0, 1c50de89a5a845668696dcb28c3badd6; no voters:
I20260129 02:59:59.427404 23043 raft_consensus.cc:515] T 0f20af7fbe5d4db89a6f17948d98912c P 197b880fa39d432a9b472845876bfaf0 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
W20260129 02:59:59.486402 20555 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 1.046s user 0.010s sys 0.014s
I20260129 02:59:59.457772 22953 consensus_queue.cc:1048] T 18f30ebaa4174fd28c958e72d0cc7e24 P 1c50de89a5a845668696dcb28c3badd6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 5, Time since last communication: 0.033s
I20260129 02:59:59.486428 20441 rpcz_store.cc:276] 0129 02:59:57.918609 (+ 0us) service_pool.cc:168] Inserting onto call queue
0129 02:59:57.937898 (+ 19289us) service_pool.cc:225] Handling call
0129 02:59:58.010630 (+ 72732us) raft_consensus.cc:1471] Updating replica for 1 ops
0129 02:59:58.044205 (+ 33575us) raft_consensus.cc:1534] Early marking committed up to index 0
0129 02:59:58.044240 (+ 35us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0129 02:59:58.044759 (+ 519us) log.cc:844] Serialized 26 byte log entry
0129 02:59:58.164437 (+119678us) raft_consensus.cc:1645] Marking committed up to 1
0129 02:59:58.164482 (+ 45us) raft_consensus.cc:1697] Filling consensus response to leader.
0129 02:59:58.164552 (+ 70us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0129 02:59:59.483634 (+1319082us) raft_consensus.cc:1685] finished
0129 02:59:59.483654 (+ 20us) raft_consensus.cc:1691] UpdateReplicas() finished
0129 02:59:59.483782 (+ 128us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":33155,"raft.queue_time_us":42937,"raft.run_cpu_time_us":441,"raft.run_wall_time_us":63436,"thread_start_us":119294,"threads_started":1,"wal-append.queue_time_us":260905}
I20260129 02:59:59.496150 20405 leader_election.cc:304] T 3ccac82a01a2445fa5b0ce6c0fcf2295 P 197b880fa39d432a9b472845876bfaf0 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 197b880fa39d432a9b472845876bfaf0; no voters: 7a9daf5d42a84615a6383cfec9a10608, 974b802141bd4f13beffef2aa03f88f9
I20260129 02:59:59.497021 20515 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57584 (request call id 1385) took 2107 ms. Trace:
I20260129 02:59:59.450732 22667 raft_consensus.cc:515] T 78aa39096868415d9afc2ffcb248e198 P 7a9daf5d42a84615a6383cfec9a10608 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } }
I20260129 02:59:59.497459 20515 rpcz_store.cc:276] 0129 02:59:57.389784 (+ 0us) service_pool.cc:168] Inserting onto call queue
0129 02:59:57.732975 (+343191us) service_pool.cc:225] Handling call
0129 02:59:57.741761 (+ 8786us) raft_consensus.cc:1471] Updating replica for 2 ops
0129 02:59:57.769429 (+ 27668us) raft_consensus.cc:1534] Early marking committed up to index 0
0129 02:59:57.769482 (+ 53us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0129 02:59:57.778894 (+ 9412us) log.cc:844] Serialized 52 byte log entry
0129 02:59:57.975554 (+196660us) raft_consensus.cc:1645] Marking committed up to 2
0129 02:59:57.975926 (+ 372us) log.cc:844] Serialized 14 byte log entry
0129 02:59:58.013653 (+ 37727us) raft_consensus.cc:1697] Filling consensus response to leader.
0129 02:59:58.013749 (+ 96us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0129 02:59:58.037899 (+ 24150us) spinlock_profiling.cc:243] Waited 3.24 ms on lock 0x7b48000615d8. stack: 00007f8fe044e373 00007f8fe044e270 00007f8fdfda0574 00007f8fe9a5c6b4 00007f8fe9a5c649 00007f8fe8447e21 00007f8fe843eeb7 00007f8fe843ed4a 00007f8fe8535f7f 00007f8fe8535c47 00007f8fe854a26c 00007f8fe854a218 00007f8fe854a198 00007f8fe854a153 00007f8fe85493fc 00007f8fe74845b6
0129 02:59:59.487113 (+1449214us) raft_consensus.cc:1685] finished
0129 02:59:59.496816 (+ 9703us) raft_consensus.cc:1691] UpdateReplicas() finished
0129 02:59:59.496963 (+ 147us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":145996,"prepare.queue_time_us":222,"prepare.run_cpu_time_us":36,"prepare.run_wall_time_us":37,"raft.queue_time_us":64459,"raft.run_cpu_time_us":2095,"raft.run_wall_time_us":28088,"spinlock_wait_cycles":7441664,"thread_start_us":163263,"threads_started":1,"wal-append.queue_time_us":350676}
I20260129 02:59:59.510455 20433 raft_consensus.cc:2393] T 0f20af7fbe5d4db89a6f17948d98912c P 197b880fa39d432a9b472845876bfaf0 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1c50de89a5a845668696dcb28c3badd6 in current term 7: Already voted for candidate 197b880fa39d432a9b472845876bfaf0 in this term.
I20260129 02:59:59.511355 20513 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57584 (request call id 1391) took 2059 ms. Trace:
I20260129 02:59:59.492146 22656 raft_consensus.cc:2804] T 1299723a4476420ab02fd8bfd7aa0645 P 1c50de89a5a845668696dcb28c3badd6 [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260129 02:59:59.511022 21136 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1299723a4476420ab02fd8bfd7aa0645" candidate_uuid: "1c50de89a5a845668696dcb28c3badd6" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "7a9daf5d42a84615a6383cfec9a10608" is_pre_election: true
I20260129 02:59:59.518289 20510 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57584 (request call id 1384) took 2128 ms. Trace:
W20260129 02:59:59.480024 23042 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.612s user 0.007s sys 0.019s
I20260129 02:59:59.489506 22640 raft_consensus.cc:493] T a0fcc58d6fb6473cafbd9d34d5195269 P 1c50de89a5a845668696dcb28c3badd6 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 974b802141bd4f13beffef2aa03f88f9)
I20260129 02:59:59.482481 23005 consensus_queue.cc:1059] T 368953c9102643b2a21f99af36b51905 P 1c50de89a5a845668696dcb28c3badd6 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 }, Status: INVALID_TERM, Last received: 6.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.025s
I20260129 02:59:59.520462 20510 rpcz_store.cc:276] 0129 02:59:57.389480 (+ 0us) service_pool.cc:168] Inserting onto call queue
0129 02:59:57.684153 (+294673us) service_pool.cc:225] Handling call
0129 02:59:57.684607 (+ 454us) raft_consensus.cc:1471] Updating replica for 1 ops
0129 02:59:57.687908 (+ 3301us) raft_consensus.cc:1534] Early marking committed up to index 6
0129 02:59:57.736041 (+ 48133us) log.cc:844] Serialized 14 byte log entry
0129 02:59:57.897571 (+161530us) spinlock_profiling.cc:243] Waited 2.75 ms on lock 0x7b18003b62c0. stack: 00007f8fe044e373 00007f8fe044e270 00007f8fdfda0574 00007f8fe9a5c6b4 00007f8fe9a5c649 00007f8fe92d4c91 00007f8fe92e1671 00007f8fe04802d3 00007f8fe04826a1 00007f8fe0482659 00007f8fe04825e9 00007f8fe04825b1 00007f8fe04818ad 0000000000379254 0000000000379099 00007f8fe046369a
0129 02:59:58.012321 (+114750us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0129 02:59:58.031599 (+ 19278us) log.cc:844] Serialized 26 byte log entry
0129 02:59:58.031802 (+ 203us) raft_consensus.cc:1645] Marking committed up to 7
0129 02:59:58.080737 (+ 48935us) log.cc:844] Serialized 14 byte log entry
0129 02:59:58.093786 (+ 13049us) raft_consensus.cc:1697] Filling consensus response to leader.
0129 02:59:58.093891 (+ 105us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0129 02:59:59.518014 (+1424123us) raft_consensus.cc:1685] finished
0129 02:59:59.518048 (+ 34us) raft_consensus.cc:1691] UpdateReplicas() finished
0129 02:59:59.518216 (+ 168us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":376278,"prepare.queue_time_us":100157,"prepare.run_cpu_time_us":73,"prepare.run_wall_time_us":76,"raft.queue_time_us":149367,"raft.run_cpu_time_us":432,"raft.run_wall_time_us":17567,"spinlock_wait_cycles":6320512,"thread_start_us":206632,"threads_started":1,"wal-append.queue_time_us":347443}
I20260129 02:59:59.547390 20434 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a4c1226b987b468fa84dfc6ba0e2dbe1" candidate_uuid: "1c50de89a5a845668696dcb28c3badd6" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "197b880fa39d432a9b472845876bfaf0" is_pre_election: true
I20260129 02:59:59.511951 20513 rpcz_store.cc:276] 0129 02:59:57.451663 (+ 0us) service_pool.cc:168] Inserting onto call queue
0129 02:59:57.906941 (+455278us) service_pool.cc:225] Handling call
0129 02:59:57.907275 (+ 334us) raft_consensus.cc:1471] Updating replica for 1 ops
0129 02:59:57.907630 (+ 355us) raft_consensus.cc:1534] Early marking committed up to index 0
0129 02:59:57.907659 (+ 29us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0129 02:59:57.961182 (+ 53523us) log.cc:844] Serialized 26 byte log entry
0129 02:59:58.112744 (+151562us) raft_consensus.cc:1645] Marking committed up to 5
0129 02:59:58.146323 (+ 33579us) log.cc:844] Serialized 14 byte log entry
0129 02:59:58.273989 (+127666us) raft_consensus.cc:1697] Filling consensus response to leader.
0129 02:59:58.274064 (+ 75us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0129 02:59:59.511123 (+1237059us) raft_consensus.cc:1685] finished
0129 02:59:59.511148 (+ 25us) raft_consensus.cc:1691] UpdateReplicas() finished
0129 02:59:59.511293 (+ 145us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":195063,"prepare.queue_time_us":33320,"prepare.run_cpu_time_us":8632,"prepare.run_wall_time_us":48650,"raft.queue_time_us":130718,"raft.run_cpu_time_us":450,"raft.run_wall_time_us":16278,"thread_start_us":98299,"threads_started":1,"wal-append.queue_time_us":519776}
W20260129 02:59:59.515724 22995 consensus_meta.cc:287] T 926a250fa5834e88b96ccecbefbf5086 P 1c50de89a5a845668696dcb28c3badd6: Time spent flushing consensus metadata: real 1.250s user 0.006s sys 0.005s
I20260129 02:59:59.560592 20441 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4b1876f9a70b49edb3aa74105e66d3b3" candidate_uuid: "1c50de89a5a845668696dcb28c3badd6" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "197b880fa39d432a9b472845876bfaf0" is_pre_election: true
W20260129 02:59:59.525990 23042 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.675s user 0.007s sys 0.020s
I20260129 02:59:59.513455 23086 raft_consensus.cc:493] T e4d18c35b0a7493e85d0a4b92956a48f P 197b880fa39d432a9b472845876bfaf0 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 974b802141bd4f13beffef2aa03f88f9)
I20260129 02:59:59.501529 22581 consensus_queue.cc:1048] T b189f08e16a74a11980cc55d99359a0a P 1c50de89a5a845668696dcb28c3badd6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.049s
I20260129 02:59:59.522325 22656 raft_consensus.cc:493] T 1299723a4476420ab02fd8bfd7aa0645 P 1c50de89a5a845668696dcb28c3badd6 [term 8 FOLLOWER]: Starting leader election (detected failure of leader 7a9daf5d42a84615a6383cfec9a10608)
I20260129 02:59:59.564195 20441 raft_consensus.cc:2393] T 4b1876f9a70b49edb3aa74105e66d3b3 P 197b880fa39d432a9b472845876bfaf0 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1c50de89a5a845668696dcb28c3badd6 in current term 5: Already voted for candidate 197b880fa39d432a9b472845876bfaf0 in this term.
W20260129 02:59:59.566040 21099 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.211.1:42027, user_credentials={real_user=slave}} blocked reactor thread for 78317.7us
W20260129 02:59:59.597210 20405 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.211.4:40439, user_credentials={real_user=slave}} blocked reactor thread for 118408us
W20260129 02:59:59.597399 21100 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.211.3:37735, user_credentials={real_user=slave}} blocked reactor thread for 102317us
I20260129 02:59:59.598358 20405 leader_election.cc:304] T 42d27a28d89b409e8807ec0efd97af69 P 197b880fa39d432a9b472845876bfaf0 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 197b880fa39d432a9b472845876bfaf0; no voters: 1c50de89a5a845668696dcb28c3badd6, 7a9daf5d42a84615a6383cfec9a10608
I20260129 02:59:59.561468 22855 leader_election.cc:290] T 45cc5f3d2688486d9b6dbe7e666dedde P 1c50de89a5a845668696dcb28c3badd6 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 974b802141bd4f13beffef2aa03f88f9 (127.18.211.2:34517), 7a9daf5d42a84615a6383cfec9a10608 (127.18.211.4:40439)
I20260129 02:59:59.567154 22977 leader_election.cc:290] T 4b1876f9a70b49edb3aa74105e66d3b3 P 1c50de89a5a845668696dcb28c3badd6 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 197b880fa39d432a9b472845876bfaf0 (127.18.211.1:42027), 974b802141bd4f13beffef2aa03f88f9 (127.18.211.2:34517)
I20260129 02:59:59.592845 23010 leader_election.cc:290] T a4c1226b987b468fa84dfc6ba0e2dbe1 P 1c50de89a5a845668696dcb28c3badd6 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 197b880fa39d432a9b472845876bfaf0 (127.18.211.1:42027), 974b802141bd4f13beffef2aa03f88f9 (127.18.211.2:34517)
W20260129 02:59:59.615513 23081 log.cc:927] Time spent T 96609ac68bdf4993a12e1c23355ca013 P 7a9daf5d42a84615a6383cfec9a10608: Append to log took a long time: real 0.096s user 0.000s sys 0.000s
I20260129 02:59:59.616559 23091 consensus_queue.cc:1059] T 87d5fe48a44a408aaf03e894a67ecb2f P 197b880fa39d432a9b472845876bfaf0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false }, Status: INVALID_TERM, Last received: 4.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.103s
I20260129 02:59:59.561795 23092 raft_consensus.cc:2749] T 2ac659d732314e1288849f08fe5e074e P 197b880fa39d432a9b472845876bfaf0 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260129 02:59:59.594419 22656 raft_consensus.cc:3060] T 1299723a4476420ab02fd8bfd7aa0645 P 1c50de89a5a845668696dcb28c3badd6 [term 8 FOLLOWER]: Advancing to term 9
I20260129 02:59:59.557822 22640 raft_consensus.cc:515] T a0fcc58d6fb6473cafbd9d34d5195269 P 1c50de89a5a845668696dcb28c3badd6 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
W20260129 02:59:59.605257 23076 log.cc:927] Time spent T 36ff33be666a4c82954731691dbcf327 P 974b802141bd4f13beffef2aa03f88f9: Append to log took a long time: real 0.861s user 0.000s sys 0.001s
I20260129 02:59:59.594915 23042 raft_consensus.cc:493] T a5a05fbcbbcf4916bbc01fd1a5b5e90d P 197b880fa39d432a9b472845876bfaf0 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 974b802141bd4f13beffef2aa03f88f9)
I20260129 02:59:59.622877 21129 raft_consensus.cc:2410] T 78aa39096868415d9afc2ffcb248e198 P 7a9daf5d42a84615a6383cfec9a10608 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 197b880fa39d432a9b472845876bfaf0 for term 6 because replica has last-logged OpId of term: 5 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 5.
W20260129 02:59:59.651072 20407 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.211.2:34517, user_credentials={real_user=slave}} blocked reactor thread for 162325us
W20260129 02:59:59.629408 23070 log.cc:927] Time spent T 3906b080a5b34a74b8b64128635e8997 P 974b802141bd4f13beffef2aa03f88f9: Append to log took a long time: real 0.079s user 0.000s sys 0.001s
I20260129 02:59:59.478181 23087 raft_consensus.cc:515] T 36ff33be666a4c82954731691dbcf327 P 197b880fa39d432a9b472845876bfaf0 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:59:59.569875 22995 raft_consensus.cc:515] T 926a250fa5834e88b96ccecbefbf5086 P 1c50de89a5a845668696dcb28c3badd6 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
W20260129 02:59:59.660709 20480 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.211.3:37735, user_credentials={real_user=slave}} blocked reactor thread for 313394us
W20260129 02:59:59.665915 20482 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260129 02:59:59.623322 23100 raft_consensus.cc:493] T dc2f4cb752cf4d59b5f6bc689f5d8104 P 7a9daf5d42a84615a6383cfec9a10608 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 1c50de89a5a845668696dcb28c3badd6)
I20260129 02:59:59.633584 22855 raft_consensus.cc:493] T e7d83d4ccf684e76acb29a0dc844e1b8 P 1c50de89a5a845668696dcb28c3badd6 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260129 02:59:59.626448 22933 leader_election.cc:290] T 4e420f90884f4796997b5ea41e49e411 P 7a9daf5d42a84615a6383cfec9a10608 [CANDIDATE]: Term 6 election: Requested vote from peers 197b880fa39d432a9b472845876bfaf0 (127.18.211.1:42027), 974b802141bd4f13beffef2aa03f88f9 (127.18.211.2:34517)
I20260129 02:59:59.584563 23086 raft_consensus.cc:515] T e4d18c35b0a7493e85d0a4b92956a48f P 197b880fa39d432a9b472845876bfaf0 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:59:59.190702 23059 consensus_queue.cc:260] T a644c02b16af47679824ccd060ac10fd P 7a9daf5d42a84615a6383cfec9a10608 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } }
I20260129 02:59:59.669960 20434 raft_consensus.cc:1275] T 96609ac68bdf4993a12e1c23355ca013 P 197b880fa39d432a9b472845876bfaf0 [term 6 FOLLOWER]: Refusing update from remote peer 7a9daf5d42a84615a6383cfec9a10608: Log matching property violated. Preceding OpId in replica: term: 2 index: 4. Preceding OpId from leader: term: 6 index: 5. (index mismatch)
I20260129 02:59:59.671001 20480 leader_election.cc:304] T 913fc7d751ed4b7388d889aac30a5f9b P 974b802141bd4f13beffef2aa03f88f9 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 974b802141bd4f13beffef2aa03f88f9; no voters: 197b880fa39d432a9b472845876bfaf0, 1c50de89a5a845668696dcb28c3badd6
I20260129 02:59:59.469476 22950 consensus_queue.cc:260] T 913fc7d751ed4b7388d889aac30a5f9b P 197b880fa39d432a9b472845876bfaf0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:59:59.668221 23033 leader_election.cc:290] T 0678b8b468bd43cdbc3e485900844821 P 7a9daf5d42a84615a6383cfec9a10608 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 197b880fa39d432a9b472845876bfaf0 (127.18.211.1:42027), 974b802141bd4f13beffef2aa03f88f9 (127.18.211.2:34517)
I20260129 02:59:59.675244 20434 raft_consensus.cc:1240] T 00e79d8f25a34676b1d866eb2068a2b2 P 197b880fa39d432a9b472845876bfaf0 [term 7 FOLLOWER]: Rejecting Update request from peer 7a9daf5d42a84615a6383cfec9a10608 for earlier term 6. Current term is 7. Ops: []
W20260129 02:59:59.678589 20554 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260129 02:59:59.657407 22922 leader_election.cc:290] T 4f8f3a38f6e24ba4857875181f447442 P 1c50de89a5a845668696dcb28c3badd6 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 197b880fa39d432a9b472845876bfaf0 (127.18.211.1:42027), 974b802141bd4f13beffef2aa03f88f9 (127.18.211.2:34517)
I20260129 02:59:59.657122 23078 raft_consensus.cc:2804] T a8479d1fe0bd41e5946186e3a6ca8c7a P 974b802141bd4f13beffef2aa03f88f9 [term 5 FOLLOWER]: Leader election won for term 5
I20260129 02:59:59.659904 22925 leader_election.cc:290] T 926a250fa5834e88b96ccecbefbf5086 P 197b880fa39d432a9b472845876bfaf0 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 974b802141bd4f13beffef2aa03f88f9 (127.18.211.2:34517), 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3:37735)
W20260129 02:59:59.682139 20405 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.211.4:40439, user_credentials={real_user=slave}} blocked reactor thread for 67381.9us
I20260129 02:59:59.685330 20405 leader_election.cc:304] T a609ac02fae84725ab743f95f51d33ce P 197b880fa39d432a9b472845876bfaf0 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 197b880fa39d432a9b472845876bfaf0; no voters: 7a9daf5d42a84615a6383cfec9a10608, 974b802141bd4f13beffef2aa03f88f9
I20260129 02:59:59.688817 20434 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4e420f90884f4796997b5ea41e49e411" candidate_uuid: "7a9daf5d42a84615a6383cfec9a10608" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "197b880fa39d432a9b472845876bfaf0"
I20260129 02:59:59.689693 20434 raft_consensus.cc:3060] T 4e420f90884f4796997b5ea41e49e411 P 197b880fa39d432a9b472845876bfaf0 [term 5 FOLLOWER]: Advancing to term 6
I20260129 02:59:59.693403 20440 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0678b8b468bd43cdbc3e485900844821" candidate_uuid: "7a9daf5d42a84615a6383cfec9a10608" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "197b880fa39d432a9b472845876bfaf0" is_pre_election: true
I20260129 02:59:59.694530 20440 raft_consensus.cc:2393] T 0678b8b468bd43cdbc3e485900844821 P 197b880fa39d432a9b472845876bfaf0 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 7a9daf5d42a84615a6383cfec9a10608 in current term 7: Already voted for candidate 974b802141bd4f13beffef2aa03f88f9 in this term.
I20260129 02:59:59.672556 23090 raft_consensus.cc:493] T 4f5a77e94fef4b16a2ba7b067c8b144a P 7a9daf5d42a84615a6383cfec9a10608 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 1c50de89a5a845668696dcb28c3badd6)
I20260129 02:59:59.673444 22857 leader_election.cc:290] T 505580c93c694bbb8ea96fdf9b17ab46 P 7a9daf5d42a84615a6383cfec9a10608 [CANDIDATE]: Term 6 election: Requested vote from peers 974b802141bd4f13beffef2aa03f88f9 (127.18.211.2:34517), 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3:37735)
I20260129 02:59:59.674383 23034 raft_consensus.cc:2749] T 913fc7d751ed4b7388d889aac30a5f9b P 974b802141bd4f13beffef2aa03f88f9 [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260129 02:59:59.680166 22708 leader_election.cc:290] T 36ff33be666a4c82954731691dbcf327 P 1c50de89a5a845668696dcb28c3badd6 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 197b880fa39d432a9b472845876bfaf0 (127.18.211.1:42027), 974b802141bd4f13beffef2aa03f88f9 (127.18.211.2:34517)
I20260129 02:59:59.666854 23042 raft_consensus.cc:515] T a5a05fbcbbcf4916bbc01fd1a5b5e90d P 197b880fa39d432a9b472845876bfaf0 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:59:59.728380 20510 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "926a250fa5834e88b96ccecbefbf5086" candidate_uuid: "197b880fa39d432a9b472845876bfaf0" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "974b802141bd4f13beffef2aa03f88f9" is_pre_election: true
I20260129 02:59:59.729319 20510 raft_consensus.cc:2468] T 926a250fa5834e88b96ccecbefbf5086 P 974b802141bd4f13beffef2aa03f88f9 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 197b880fa39d432a9b472845876bfaf0 in term 6.
I20260129 02:59:59.685063 23092 raft_consensus.cc:2749] T 42d27a28d89b409e8807ec0efd97af69 P 197b880fa39d432a9b472845876bfaf0 [term 6 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260129 02:59:59.735831 20586 raft_consensus.cc:1275] T 96609ac68bdf4993a12e1c23355ca013 P 1c50de89a5a845668696dcb28c3badd6 [term 6 FOLLOWER]: Refusing update from remote peer 7a9daf5d42a84615a6383cfec9a10608: Log matching property violated. Preceding OpId in replica: term: 2 index: 3. Preceding OpId from leader: term: 6 index: 5. (index mismatch)
I20260129 02:59:59.732613 20583 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35752 (request call id 1615) took 3275 ms. Trace:
I20260129 02:59:59.685793 23096 raft_consensus.cc:2749] T 3ccac82a01a2445fa5b0ce6c0fcf2295 P 197b880fa39d432a9b472845876bfaf0 [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260129 02:59:59.737237 20583 rpcz_store.cc:276] 0129 02:59:56.445073 (+ 0us) service_pool.cc:168] Inserting onto call queue
0129 02:59:56.658101 (+213028us) service_pool.cc:225] Handling call
0129 02:59:56.658401 (+ 300us) raft_consensus.cc:1471] Updating replica for 0 ops
0129 02:59:57.648947 (+990546us) raft_consensus.cc:1697] Filling consensus response to leader.
0129 02:59:59.720102 (+2071155us) spinlock_profiling.cc:243] Waited 936 ms on lock 0x7b54000c9254. stack: 00007f8fe044e373 00007f8fe044e270 00007f8fdfda0574 00007f8fe9a5c6b4 00007f8fe9a5c649 00007f8fe34975a1 00007f8fe352ea81 00007f8fe352d4be 00007f8fe84e1f49 00007f8fe1a6ed34 00007f8fe1a6ecc2 00007f8fe1a6ebf1 00007f8fe1a6eb6c 00007f8fe1a6de22 00007f8fe186d34c 00007f8fe186c786
0129 02:59:59.720291 (+ 189us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2153716096}
I20260129 02:59:59.739683 20510 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0f20af7fbe5d4db89a6f17948d98912c" candidate_uuid: "197b880fa39d432a9b472845876bfaf0" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "974b802141bd4f13beffef2aa03f88f9"
I20260129 02:59:59.673709 23100 raft_consensus.cc:515] T dc2f4cb752cf4d59b5f6bc689f5d8104 P 7a9daf5d42a84615a6383cfec9a10608 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } }
I20260129 02:59:59.715692 23010 raft_consensus.cc:493] T 3906b080a5b34a74b8b64128635e8997 P 1c50de89a5a845668696dcb28c3badd6 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260129 02:59:59.696599 22922 raft_consensus.cc:493] T c891d2bff4ce4568861bbc992e91fbb0 P 1c50de89a5a845668696dcb28c3badd6 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 974b802141bd4f13beffef2aa03f88f9)
I20260129 02:59:59.742311 20441 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "36ff33be666a4c82954731691dbcf327" candidate_uuid: "1c50de89a5a845668696dcb28c3badd6" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "197b880fa39d432a9b472845876bfaf0" is_pre_election: true
I20260129 02:59:59.740844 20582 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "505580c93c694bbb8ea96fdf9b17ab46" candidate_uuid: "7a9daf5d42a84615a6383cfec9a10608" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "1c50de89a5a845668696dcb28c3badd6"
I20260129 02:59:59.754346 20441 raft_consensus.cc:2410] T 36ff33be666a4c82954731691dbcf327 P 197b880fa39d432a9b472845876bfaf0 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1c50de89a5a845668696dcb28c3badd6 for term 4 because replica has last-logged OpId of term: 3 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 2.
I20260129 02:59:59.714655 23043 leader_election.cc:290] T 0f20af7fbe5d4db89a6f17948d98912c P 197b880fa39d432a9b472845876bfaf0 [CANDIDATE]: Term 7 election: Requested vote from peers 974b802141bd4f13beffef2aa03f88f9 (127.18.211.2:34517), 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3:37735)
I20260129 02:59:59.675143 22855 raft_consensus.cc:515] T e7d83d4ccf684e76acb29a0dc844e1b8 P 1c50de89a5a845668696dcb28c3badd6 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
W20260129 02:59:59.735767 22691 consensus_meta.cc:287] T fcddab6bca684adf8ea032289a00f90e P 197b880fa39d432a9b472845876bfaf0: Time spent flushing consensus metadata: real 1.793s user 0.013s sys 0.003s
W20260129 02:59:59.752331 21102 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.18.211.2:34517 timed out after 3.022s (SENT) [suppressed 16 similar messages]
I20260129 02:59:59.738795 23033 raft_consensus.cc:493] T 3e2d15f8640b407195210c476e09df91 P 7a9daf5d42a84615a6383cfec9a10608 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 1c50de89a5a845668696dcb28c3badd6)
I20260129 02:59:59.741546 23080 raft_consensus.cc:493] T cb0b3f259ebc4310927338a8740e87b8 P 7a9daf5d42a84615a6383cfec9a10608 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260129 02:59:59.717885 22925 consensus_queue.cc:1059] T febffe4a8b394368b69b1d164308fb9a P 197b880fa39d432a9b472845876bfaf0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.035s
I20260129 02:59:59.750443 22667 leader_election.cc:290] T 78aa39096868415d9afc2ffcb248e198 P 7a9daf5d42a84615a6383cfec9a10608 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 197b880fa39d432a9b472845876bfaf0 (127.18.211.1:42027), 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3:37735)
W20260129 02:59:59.783013 20480 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.211.3:37735, user_credentials={real_user=slave}} blocked reactor thread for 101634us
I20260129 02:59:59.785034 20480 leader_election.cc:304] T 0340229d521b432580ae926f6062e3f7 P 974b802141bd4f13beffef2aa03f88f9 [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 974b802141bd4f13beffef2aa03f88f9; no voters: 197b880fa39d432a9b472845876bfaf0, 1c50de89a5a845668696dcb28c3badd6
I20260129 02:59:59.767277 20441 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "78aa39096868415d9afc2ffcb248e198" candidate_uuid: "7a9daf5d42a84615a6383cfec9a10608" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "197b880fa39d432a9b472845876bfaf0" is_pre_election: true
W20260129 02:59:59.786641 23071 log.cc:927] Time spent T e33a417cf9c54cf5af73a4ee3aac4d3a P 974b802141bd4f13beffef2aa03f88f9: Append to log took a long time: real 0.919s user 0.001s sys 0.002s
I20260129 02:59:59.787386 20441 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a0fcc58d6fb6473cafbd9d34d5195269" candidate_uuid: "1c50de89a5a845668696dcb28c3badd6" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "197b880fa39d432a9b472845876bfaf0" is_pre_election: true
I20260129 02:59:59.789415 20441 raft_consensus.cc:2393] T a0fcc58d6fb6473cafbd9d34d5195269 P 197b880fa39d432a9b472845876bfaf0 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1c50de89a5a845668696dcb28c3badd6 in current term 4: Already voted for candidate 197b880fa39d432a9b472845876bfaf0 in this term.
I20260129 02:59:59.745162 22933 consensus_queue.cc:1059] T 00e79d8f25a34676b1d866eb2068a2b2 P 7a9daf5d42a84615a6383cfec9a10608 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 }, Status: INVALID_TERM, Last received: 6.8, Next index: 9, Last known committed idx: 6, Time since last communication: 0.036s
I20260129 02:59:59.819327 21136 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "45cc5f3d2688486d9b6dbe7e666dedde" candidate_uuid: "1c50de89a5a845668696dcb28c3badd6" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "7a9daf5d42a84615a6383cfec9a10608" is_pre_election: true
W20260129 02:59:59.819685 23083 log.cc:927] Time spent T 7a9a69cd82724b1faed1d10aa091fc4a P 197b880fa39d432a9b472845876bfaf0: Append to log took a long time: real 0.126s user 0.000s sys 0.000s
I20260129 02:59:59.820711 21136 raft_consensus.cc:2468] T 45cc5f3d2688486d9b6dbe7e666dedde P 7a9daf5d42a84615a6383cfec9a10608 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1c50de89a5a845668696dcb28c3badd6 in term 4.
I20260129 02:59:59.781159 23033 raft_consensus.cc:515] T 3e2d15f8640b407195210c476e09df91 P 7a9daf5d42a84615a6383cfec9a10608 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } }
W20260129 02:59:59.821067 20406 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260129 02:59:59.829697 20406 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260129 02:59:59.831441 20407 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.211.2:34517, user_credentials={real_user=slave}} blocked reactor thread for 99570.8us
W20260129 02:59:59.831784 20405 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.211.4:40439, user_credentials={real_user=slave}} blocked reactor thread for 90617.5us
I20260129 02:59:59.779706 23096 raft_consensus.cc:3060] T a609ac02fae84725ab743f95f51d33ce P 197b880fa39d432a9b472845876bfaf0 [term 6 FOLLOWER]: Advancing to term 7
I20260129 02:59:59.753707 23034 consensus_queue.cc:1048] T 36ff33be666a4c82954731691dbcf327 P 974b802141bd4f13beffef2aa03f88f9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.035s
I20260129 02:59:59.833245 20405 leader_election.cc:304] T 00e79d8f25a34676b1d866eb2068a2b2 P 197b880fa39d432a9b472845876bfaf0 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 197b880fa39d432a9b472845876bfaf0; no voters: 7a9daf5d42a84615a6383cfec9a10608, 974b802141bd4f13beffef2aa03f88f9
I20260129 02:59:59.835186 20480 leader_election.cc:304] T 926a250fa5834e88b96ccecbefbf5086 P 974b802141bd4f13beffef2aa03f88f9 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1c50de89a5a845668696dcb28c3badd6, 974b802141bd4f13beffef2aa03f88f9; no voters: 197b880fa39d432a9b472845876bfaf0
W20260129 02:59:59.833956 21100 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.211.3:37735, user_credentials={real_user=slave}} blocked reactor thread for 78424.9us
I20260129 02:59:59.741035 23098 consensus_queue.cc:1048] T 96609ac68bdf4993a12e1c23355ca013 P 7a9daf5d42a84615a6383cfec9a10608 [LEADER]: Connected to new peer: Peer: permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 3, Time since last communication: 0.055s
I20260129 02:59:59.764809 23010 raft_consensus.cc:515] T 3906b080a5b34a74b8b64128635e8997 P 1c50de89a5a845668696dcb28c3badd6 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 02:59:59.837658 20441 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4f8f3a38f6e24ba4857875181f447442" candidate_uuid: "1c50de89a5a845668696dcb28c3badd6" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "197b880fa39d432a9b472845876bfaf0" is_pre_election: true
I20260129 02:59:59.838220 20582 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "78aa39096868415d9afc2ffcb248e198" candidate_uuid: "7a9daf5d42a84615a6383cfec9a10608" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "1c50de89a5a845668696dcb28c3badd6" is_pre_election: true
I20260129 02:59:59.723532 23090 raft_consensus.cc:515] T 4f5a77e94fef4b16a2ba7b067c8b144a P 7a9daf5d42a84615a6383cfec9a10608 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } }
I20260129 02:59:59.768726 22922 raft_consensus.cc:515] T c891d2bff4ce4568861bbc992e91fbb0 P 1c50de89a5a845668696dcb28c3badd6 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
W20260129 02:59:59.838778 21102 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.211.2:34517, user_credentials={real_user=slave}} blocked reactor thread for 86489.4us
I20260129 02:59:59.774056 22691 raft_consensus.cc:515] T fcddab6bca684adf8ea032289a00f90e P 197b880fa39d432a9b472845876bfaf0 [term 9 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
W20260129 02:59:59.865535 21101 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260129 02:59:59.866799 21102 leader_election.cc:336] T 2ac659d732314e1288849f08fe5e074e P 7a9daf5d42a84615a6383cfec9a10608 [CANDIDATE]: Term 4 election: RPC error from VoteRequest() call to peer 974b802141bd4f13beffef2aa03f88f9 (127.18.211.2:34517): Timed out: RequestConsensusVote RPC to 127.18.211.2:34517 timed out after 1.710s (SENT)
I20260129 02:59:59.867468 21102 leader_election.cc:304] T 2ac659d732314e1288849f08fe5e074e P 7a9daf5d42a84615a6383cfec9a10608 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7a9daf5d42a84615a6383cfec9a10608; no voters: 197b880fa39d432a9b472845876bfaf0, 974b802141bd4f13beffef2aa03f88f9
I20260129 02:59:59.873554 20441 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "926a250fa5834e88b96ccecbefbf5086" candidate_uuid: "1c50de89a5a845668696dcb28c3badd6" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "197b880fa39d432a9b472845876bfaf0"
I20260129 02:59:59.876744 20441 raft_consensus.cc:3060] T 926a250fa5834e88b96ccecbefbf5086 P 197b880fa39d432a9b472845876bfaf0 [term 6 FOLLOWER]: Advancing to term 7
I20260129 02:59:59.839112 22640 leader_election.cc:290] T a0fcc58d6fb6473cafbd9d34d5195269 P 1c50de89a5a845668696dcb28c3badd6 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 197b880fa39d432a9b472845876bfaf0 (127.18.211.1:42027), 974b802141bd4f13beffef2aa03f88f9 (127.18.211.2:34517)
I20260129 02:59:59.888561 20435 raft_consensus.cc:1240] T fcddab6bca684adf8ea032289a00f90e P 197b880fa39d432a9b472845876bfaf0 [term 9 FOLLOWER]: Rejecting Update request from peer 974b802141bd4f13beffef2aa03f88f9 for earlier term 8. Current term is 9. Ops: []
I20260129 02:59:59.833582 22851 consensus_queue.cc:1048] T c934d31efd3f4e5195605ddc354d14e1 P 197b880fa39d432a9b472845876bfaf0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.042s
I20260129 02:59:59.847887 22667 consensus_queue.cc:1048] T 96609ac68bdf4993a12e1c23355ca013 P 7a9daf5d42a84615a6383cfec9a10608 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 3, Time since last communication: 0.049s
I20260129 02:59:59.899216 23097 raft_consensus.cc:2749] T 0340229d521b432580ae926f6062e3f7 P 974b802141bd4f13beffef2aa03f88f9 [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20260129 02:59:59.781934 23080 raft_consensus.cc:515] T cb0b3f259ebc4310927338a8740e87b8 P 7a9daf5d42a84615a6383cfec9a10608 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } }
I20260129 02:59:59.907878 23035 raft_consensus.cc:493] T 45cc5f3d2688486d9b6dbe7e666dedde P 7a9daf5d42a84615a6383cfec9a10608 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260129 02:59:59.908844 23099 log.cc:927] Time spent T 18f30ebaa4174fd28c958e72d0cc7e24 P 197b880fa39d432a9b472845876bfaf0: Append to log took a long time: real 0.156s user 0.001s sys 0.000s
I20260129 02:59:59.908556 23035 raft_consensus.cc:515] T 45cc5f3d2688486d9b6dbe7e666dedde P 7a9daf5d42a84615a6383cfec9a10608 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } }
I20260129 02:59:59.910255 20438 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55676 (request call id 1379) took 1172 ms. Trace:
I20260129 02:59:59.925974 20434 raft_consensus.cc:2468] T 4e420f90884f4796997b5ea41e49e411 P 197b880fa39d432a9b472845876bfaf0 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7a9daf5d42a84615a6383cfec9a10608 in term 6.
I20260129 02:59:59.910655 20438 rpcz_store.cc:276] 0129 02:59:58.737401 (+ 0us) service_pool.cc:168] Inserting onto call queue
0129 02:59:58.737628 (+ 227us) service_pool.cc:225] Handling call
0129 02:59:58.814246 (+ 76618us) raft_consensus.cc:1471] Updating replica for 1 ops
0129 02:59:59.476811 (+662565us) raft_consensus.cc:1534] Early marking committed up to index 0
0129 02:59:59.477223 (+ 412us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0129 02:59:59.478552 (+ 1329us) log.cc:844] Serialized 26 byte log entry
0129 02:59:59.648369 (+169817us) raft_consensus.cc:1645] Marking committed up to 5
0129 02:59:59.648410 (+ 41us) raft_consensus.cc:1697] Filling consensus response to leader.
0129 02:59:59.648486 (+ 76us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0129 02:59:59.910007 (+261521us) raft_consensus.cc:1685] finished
0129 02:59:59.910032 (+ 25us) raft_consensus.cc:1691] UpdateReplicas() finished
0129 02:59:59.910195 (+ 163us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":298487,"raft.queue_time_us":621808,"raft.run_cpu_time_us":169,"raft.run_wall_time_us":16686,"thread_start_us":532902,"threads_started":2,"wal-append.queue_time_us":239473}
I20260129 02:59:59.888775 22995 leader_election.cc:290] T 926a250fa5834e88b96ccecbefbf5086 P 1c50de89a5a845668696dcb28c3badd6 [CANDIDATE]: Term 7 election: Requested vote from peers 197b880fa39d432a9b472845876bfaf0 (127.18.211.1:42027), 974b802141bd4f13beffef2aa03f88f9 (127.18.211.2:34517)
I20260129 02:59:59.942420 20434 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c891d2bff4ce4568861bbc992e91fbb0" candidate_uuid: "1c50de89a5a845668696dcb28c3badd6" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "197b880fa39d432a9b472845876bfaf0" is_pre_election: true
I20260129 02:59:59.944602 20434 raft_consensus.cc:2393] T c891d2bff4ce4568861bbc992e91fbb0 P 197b880fa39d432a9b472845876bfaf0 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1c50de89a5a845668696dcb28c3badd6 in current term 7: Already voted for candidate 197b880fa39d432a9b472845876bfaf0 in this term.
I20260129 02:59:59.941198 23086 leader_election.cc:290] T e4d18c35b0a7493e85d0a4b92956a48f P 197b880fa39d432a9b472845876bfaf0 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 974b802141bd4f13beffef2aa03f88f9 (127.18.211.2:34517), 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3:37735)
I20260129 02:59:59.902858 23087 leader_election.cc:290] T 36ff33be666a4c82954731691dbcf327 P 197b880fa39d432a9b472845876bfaf0 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 974b802141bd4f13beffef2aa03f88f9 (127.18.211.2:34517), 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3:37735)
I20260129 02:59:59.993824 20434 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e7d83d4ccf684e76acb29a0dc844e1b8" candidate_uuid: "1c50de89a5a845668696dcb28c3badd6" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "197b880fa39d432a9b472845876bfaf0" is_pre_election: true
I20260129 02:59:59.964920 23118 raft_consensus.cc:2749] T 00e79d8f25a34676b1d866eb2068a2b2 P 197b880fa39d432a9b472845876bfaf0 [term 7 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260129 02:59:59.972697 22855 leader_election.cc:290] T e7d83d4ccf684e76acb29a0dc844e1b8 P 1c50de89a5a845668696dcb28c3badd6 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 197b880fa39d432a9b472845876bfaf0 (127.18.211.1:42027), 974b802141bd4f13beffef2aa03f88f9 (127.18.211.2:34517)
I20260129 02:59:59.973022 22691 leader_election.cc:290] T fcddab6bca684adf8ea032289a00f90e P 197b880fa39d432a9b472845876bfaf0 [CANDIDATE]: Term 9 election: Requested vote from peers 974b802141bd4f13beffef2aa03f88f9 (127.18.211.2:34517), 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3:37735)
I20260129 02:59:59.973929 23035 leader_election.cc:290] T 45cc5f3d2688486d9b6dbe7e666dedde P 7a9daf5d42a84615a6383cfec9a10608 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 974b802141bd4f13beffef2aa03f88f9 (127.18.211.2:34517), 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3:37735)
I20260129 02:59:59.975387 23100 leader_election.cc:290] T dc2f4cb752cf4d59b5f6bc689f5d8104 P 7a9daf5d42a84615a6383cfec9a10608 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 197b880fa39d432a9b472845876bfaf0 (127.18.211.1:42027), 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3:37735)
I20260129 02:59:59.912178 22933 raft_consensus.cc:493] T a609ac02fae84725ab743f95f51d33ce P 7a9daf5d42a84615a6383cfec9a10608 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260129 02:59:59.965220 23107 raft_consensus.cc:2749] T 2ac659d732314e1288849f08fe5e074e P 7a9daf5d42a84615a6383cfec9a10608 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20260129 02:59:59.977973 20436 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55672 (request call id 1632) took 2120 ms. Trace:
I20260129 03:00:00.023896 20436 rpcz_store.cc:276] 0129 02:59:57.857969 (+ 0us) service_pool.cc:168] Inserting onto call queue
0129 02:59:57.908711 (+ 50742us) service_pool.cc:225] Handling call
0129 02:59:59.977291 (+2068580us) spinlock_profiling.cc:243] Waited 12.1 ms on lock 0x7b5800000f98. stack: 00007f8fe044e373 00007f8fe044e270 00007f8fdfda0574 00007f8fe9a5c6b4 00007f8fe9a5c649 00007f8fe02c08b1 00007f8fe0442db1 00007f8fe0442c62 00007f8fe92eddd9 00007f8fe846d260 00007f8fe8535dbb 00007f8fe8535e41 00007f8fe84f7af0 00007f8fe84e2459 00007f8fe84e1f02 00007f8fe1a6ed34
0129 02:59:59.977486 (+ 195us) raft_consensus.cc:1471] Updating replica for 0 ops
0129 02:59:59.977678 (+ 192us) raft_consensus.cc:1534] Early marking committed up to index 0
0129 02:59:59.977704 (+ 26us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0129 02:59:59.977748 (+ 44us) raft_consensus.cc:1645] Marking committed up to 5
0129 02:59:59.977776 (+ 28us) raft_consensus.cc:1697] Filling consensus response to leader.
0129 02:59:59.977832 (+ 56us) raft_consensus.cc:1691] UpdateReplicas() finished
0129 02:59:59.977926 (+ 94us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":27801472}
I20260129 03:00:00.025883 20436 raft_consensus.cc:1240] T 368953c9102643b2a21f99af36b51905 P 197b880fa39d432a9b472845876bfaf0 [term 7 FOLLOWER]: Rejecting Update request from peer 1c50de89a5a845668696dcb28c3badd6 for earlier term 6. Current term is 7. Ops: []
I20260129 02:59:59.968138 23047 raft_consensus.cc:493] T add8d4923d1a489e86882c3a25692b95 P 974b802141bd4f13beffef2aa03f88f9 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260129 02:59:59.968564 23026 raft_consensus.cc:3055] T 368953c9102643b2a21f99af36b51905 P 1c50de89a5a845668696dcb28c3badd6 [term 6 LEADER]: Stepping down as leader of term 6
I20260129 02:59:59.900194 22922 leader_election.cc:290] T c891d2bff4ce4568861bbc992e91fbb0 P 1c50de89a5a845668696dcb28c3badd6 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 197b880fa39d432a9b472845876bfaf0 (127.18.211.1:42027), 974b802141bd4f13beffef2aa03f88f9 (127.18.211.2:34517)
W20260129 02:59:59.981520 20407 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.211.2:34517, user_credentials={real_user=slave}} blocked reactor thread for 114926us
I20260129 03:00:00.033686 20407 leader_election.cc:304] T 926a250fa5834e88b96ccecbefbf5086 P 197b880fa39d432a9b472845876bfaf0 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 197b880fa39d432a9b472845876bfaf0, 974b802141bd4f13beffef2aa03f88f9; no voters:
I20260129 02:59:59.989539 23010 leader_election.cc:290] T 3906b080a5b34a74b8b64128635e8997 P 1c50de89a5a845668696dcb28c3badd6 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 197b880fa39d432a9b472845876bfaf0 (127.18.211.1:42027), 974b802141bd4f13beffef2aa03f88f9 (127.18.211.2:34517)
I20260129 02:59:59.941869 23034 raft_consensus.cc:2804] T 926a250fa5834e88b96ccecbefbf5086 P 974b802141bd4f13beffef2aa03f88f9 [term 6 FOLLOWER]: Leader election won for term 6
I20260129 03:00:00.053136 20436 raft_consensus.cc:3055] T 7d9494f5584d4d5db8b9adf1c8fc888b P 197b880fa39d432a9b472845876bfaf0 [term 3 LEADER]: Stepping down as leader of term 3
I20260129 03:00:00.053848 20436 raft_consensus.cc:740] T 7d9494f5584d4d5db8b9adf1c8fc888b P 197b880fa39d432a9b472845876bfaf0 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 197b880fa39d432a9b472845876bfaf0, State: Running, Role: LEADER
W20260129 02:59:59.990664 22694 consensus_meta.cc:287] T c891d2bff4ce4568861bbc992e91fbb0 P 974b802141bd4f13beffef2aa03f88f9: Time spent flushing consensus metadata: real 0.656s user 0.006s sys 0.003s
I20260129 02:59:59.991175 23042 leader_election.cc:290] T a5a05fbcbbcf4916bbc01fd1a5b5e90d P 197b880fa39d432a9b472845876bfaf0 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 974b802141bd4f13beffef2aa03f88f9 (127.18.211.2:34517), 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3:37735)
I20260129 02:59:59.992324 23090 leader_election.cc:290] T 4f5a77e94fef4b16a2ba7b067c8b144a P 7a9daf5d42a84615a6383cfec9a10608 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 974b802141bd4f13beffef2aa03f88f9 (127.18.211.2:34517), 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3:37735)
I20260129 02:59:59.995749 20434 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3906b080a5b34a74b8b64128635e8997" candidate_uuid: "1c50de89a5a845668696dcb28c3badd6" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "197b880fa39d432a9b472845876bfaf0" is_pre_election: true
I20260129 03:00:00.057503 20434 raft_consensus.cc:2410] T 3906b080a5b34a74b8b64128635e8997 P 197b880fa39d432a9b472845876bfaf0 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1c50de89a5a845668696dcb28c3badd6 for term 5 because replica has last-logged OpId of term: 4 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 2.
W20260129 02:59:59.990567 22900 consensus_meta.cc:287] T 0340229d521b432580ae926f6062e3f7 P 1c50de89a5a845668696dcb28c3badd6: Time spent flushing consensus metadata: real 1.622s user 0.010s sys 0.003s
I20260129 02:59:59.999370 20441 raft_consensus.cc:2410] T 926a250fa5834e88b96ccecbefbf5086 P 197b880fa39d432a9b472845876bfaf0 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 1c50de89a5a845668696dcb28c3badd6 for term 7 because replica has last-logged OpId of term: 5 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 4 index: 3.
I20260129 02:59:59.933100 23092 raft_consensus.cc:493] T 4e420f90884f4796997b5ea41e49e411 P 197b880fa39d432a9b472845876bfaf0 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260129 03:00:00.063704 23092 raft_consensus.cc:515] T 4e420f90884f4796997b5ea41e49e411 P 197b880fa39d432a9b472845876bfaf0 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } }
I20260129 03:00:00.066005 23080 leader_election.cc:290] T cb0b3f259ebc4310927338a8740e87b8 P 7a9daf5d42a84615a6383cfec9a10608 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 974b802141bd4f13beffef2aa03f88f9 (127.18.211.2:34517), 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3:37735)
W20260129 03:00:00.066649 23117 log.cc:927] Time spent T 75be4bd6bd45474795fd472eff2953ac P 1c50de89a5a845668696dcb28c3badd6: Append to log took a long time: real 0.064s user 0.000s sys 0.001s
I20260129 03:00:00.068239 22922 consensus_queue.cc:1059] T 368953c9102643b2a21f99af36b51905 P 1c50de89a5a845668696dcb28c3badd6 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 }, Status: INVALID_TERM, Last received: 6.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260129 03:00:00.014757 20437 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55676 (request call id 1373) took 2156 ms. Trace:
I20260129 03:00:00.071197 20436 consensus_queue.cc:260] T 7d9494f5584d4d5db8b9adf1c8fc888b P 197b880fa39d432a9b472845876bfaf0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.3, Last appended by leader: 3, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 03:00:00.016204 20588 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "45cc5f3d2688486d9b6dbe7e666dedde" candidate_uuid: "7a9daf5d42a84615a6383cfec9a10608" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "1c50de89a5a845668696dcb28c3badd6" is_pre_election: true
I20260129 02:59:59.973935 23033 leader_election.cc:290] T 3e2d15f8640b407195210c476e09df91 P 7a9daf5d42a84615a6383cfec9a10608 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 974b802141bd4f13beffef2aa03f88f9 (127.18.211.2:34517), 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3:37735)
I20260129 03:00:00.022667 20583 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3e2d15f8640b407195210c476e09df91" candidate_uuid: "7a9daf5d42a84615a6383cfec9a10608" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "1c50de89a5a845668696dcb28c3badd6" is_pre_election: true
W20260129 03:00:00.014557 23103 log.cc:927] Time spent T 0f20af7fbe5d4db89a6f17948d98912c P 974b802141bd4f13beffef2aa03f88f9: Append to log took a long time: real 0.275s user 0.001s sys 0.001s
I20260129 03:00:00.073378 20588 raft_consensus.cc:2468] T 45cc5f3d2688486d9b6dbe7e666dedde P 1c50de89a5a845668696dcb28c3badd6 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7a9daf5d42a84615a6383cfec9a10608 in term 4.
I20260129 03:00:00.041579 20586 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "dc2f4cb752cf4d59b5f6bc689f5d8104" candidate_uuid: "7a9daf5d42a84615a6383cfec9a10608" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "1c50de89a5a845668696dcb28c3badd6" is_pre_election: true
I20260129 03:00:00.041656 20581 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4f5a77e94fef4b16a2ba7b067c8b144a" candidate_uuid: "7a9daf5d42a84615a6383cfec9a10608" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "1c50de89a5a845668696dcb28c3badd6" is_pre_election: true
W20260129 03:00:00.053084 20556 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.211.2:34517, user_credentials={real_user=slave}} blocked reactor thread for 1.91588e+06us
W20260129 03:00:00.074126 20482 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260129 03:00:00.042284 23026 raft_consensus.cc:740] T 368953c9102643b2a21f99af36b51905 P 1c50de89a5a845668696dcb28c3badd6 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 1c50de89a5a845668696dcb28c3badd6, State: Running, Role: LEADER
I20260129 03:00:00.113857 21136 raft_consensus.cc:1240] T febffe4a8b394368b69b1d164308fb9a P 7a9daf5d42a84615a6383cfec9a10608 [term 4 FOLLOWER]: Rejecting Update request from peer 197b880fa39d432a9b472845876bfaf0 for earlier term 3. Current term is 4. Ops: [3.5-3.5]
W20260129 03:00:00.114612 22989 consensus_meta.cc:287] T cedce331e1584ac8b636c9b8a72b2ba0 P 974b802141bd4f13beffef2aa03f88f9: Time spent flushing consensus metadata: real 1.987s user 0.014s sys 0.000s
I20260129 03:00:00.117883 22656 raft_consensus.cc:515] T 1299723a4476420ab02fd8bfd7aa0645 P 1c50de89a5a845668696dcb28c3badd6 [term 9 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } }
I20260129 03:00:00.115024 20584 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "cb0b3f259ebc4310927338a8740e87b8" candidate_uuid: "7a9daf5d42a84615a6383cfec9a10608" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "1c50de89a5a845668696dcb28c3badd6" is_pre_election: true
I20260129 03:00:00.077602 21100 leader_election.cc:304] T 45cc5f3d2688486d9b6dbe7e666dedde P 7a9daf5d42a84615a6383cfec9a10608 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1c50de89a5a845668696dcb28c3badd6, 7a9daf5d42a84615a6383cfec9a10608; no voters:
I20260129 03:00:00.069970 20437 rpcz_store.cc:276] 0129 02:59:57.858513 (+ 0us) service_pool.cc:168] Inserting onto call queue
0129 02:59:57.914268 (+ 55755us) service_pool.cc:225] Handling call
0129 02:59:58.267508 (+353240us) spinlock_profiling.cc:243] Waited 6.43 ms on lock 0x7b5800000f98. stack: 00007f8fe044e373 00007f8fe044e270 00007f8fdfda0574 00007f8fe9a5c6b4 00007f8fe9a5c649 00007f8fe02c08b1 00007f8fe0442db1 00007f8fe0442c62 00007f8fe92eddd9 00007f8fe846d260 00007f8fe8535dbb 00007f8fe8535e41 00007f8fe84f7af0 00007f8fe84e2459 00007f8fe84e1f02 00007f8fe1a6ed34
0129 02:59:58.267745 (+ 237us) raft_consensus.cc:1471] Updating replica for 1 ops
0129 02:59:58.268180 (+ 435us) raft_consensus.cc:1534] Early marking committed up to index 3
0129 02:59:58.287953 (+ 19773us) log.cc:844] Serialized 14 byte log entry
0129 02:59:59.349046 (+1061093us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0129 02:59:59.359747 (+ 10701us) log.cc:844] Serialized 26 byte log entry
0129 02:59:59.384583 (+ 24836us) raft_consensus.cc:1645] Marking committed up to 3
0129 02:59:59.386596 (+ 2013us) raft_consensus.cc:1697] Filling consensus response to leader.
0129 02:59:59.386891 (+ 295us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0129 02:59:59.994960 (+608069us) raft_consensus.cc:1685] finished
0129 03:00:00.014533 (+ 19573us) raft_consensus.cc:1691] UpdateReplicas() finished
0129 03:00:00.014695 (+ 162us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":19575,"prepare.run_cpu_time_us":42,"prepare.run_wall_time_us":43,"raft.queue_time_us":689593,"raft.run_cpu_time_us":388,"raft.run_wall_time_us":67355,"spinlock_wait_cycles":14796544,"thread_start_us":496687,"threads_started":2,"wal-append.queue_time_us":1317227}
I20260129 03:00:00.143392 20584 raft_consensus.cc:2468] T cb0b3f259ebc4310927338a8740e87b8 P 1c50de89a5a845668696dcb28c3badd6 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7a9daf5d42a84615a6383cfec9a10608 in term 6.
I20260129 03:00:00.135409 20512 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0f20af7fbe5d4db89a6f17948d98912c" candidate_uuid: "1c50de89a5a845668696dcb28c3badd6" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "974b802141bd4f13beffef2aa03f88f9" is_pre_election: true
I20260129 03:00:00.150422 20515 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e33a417cf9c54cf5af73a4ee3aac4d3a" candidate_uuid: "1c50de89a5a845668696dcb28c3badd6" candidate_term: 7 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "974b802141bd4f13beffef2aa03f88f9" is_pre_election: true
I20260129 03:00:00.069605 22953 raft_consensus.cc:493] T 0340229d521b432580ae926f6062e3f7 P 1c50de89a5a845668696dcb28c3badd6 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260129 03:00:00.075735 23042 raft_consensus.cc:493] T 0678b8b468bd43cdbc3e485900844821 P 197b880fa39d432a9b472845876bfaf0 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260129 03:00:00.192025 20516 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4b1876f9a70b49edb3aa74105e66d3b3" candidate_uuid: "1c50de89a5a845668696dcb28c3badd6" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "974b802141bd4f13beffef2aa03f88f9" is_pre_election: true
I20260129 03:00:00.194767 20507 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a4c1226b987b468fa84dfc6ba0e2dbe1" candidate_uuid: "1c50de89a5a845668696dcb28c3badd6" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "974b802141bd4f13beffef2aa03f88f9" is_pre_election: true
W20260129 03:00:00.238886 20406 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260129 03:00:00.075301 22851 raft_consensus.cc:2764] T 926a250fa5834e88b96ccecbefbf5086 P 197b880fa39d432a9b472845876bfaf0 [term 7 FOLLOWER]: Leader pre-election decision vote started in defunct term 6: won
I20260129 03:00:00.179533 20512 raft_consensus.cc:3055] T b189f08e16a74a11980cc55d99359a0a P 974b802141bd4f13beffef2aa03f88f9 [term 7 LEADER]: Stepping down as leader of term 7
I20260129 03:00:00.258162 20512 raft_consensus.cc:740] T b189f08e16a74a11980cc55d99359a0a P 974b802141bd4f13beffef2aa03f88f9 [term 7 LEADER]: Becoming Follower/Learner. State: Replica: 974b802141bd4f13beffef2aa03f88f9, State: Running, Role: LEADER
W20260129 03:00:00.250324 21100 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.211.3:37735, user_credentials={real_user=slave}} blocked reactor thread for 172814us
I20260129 03:00:00.255158 20511 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "36ff33be666a4c82954731691dbcf327" candidate_uuid: "1c50de89a5a845668696dcb28c3badd6" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "974b802141bd4f13beffef2aa03f88f9" is_pre_election: true
I20260129 03:00:00.105890 23078 raft_consensus.cc:697] T a8479d1fe0bd41e5946186e3a6ca8c7a P 974b802141bd4f13beffef2aa03f88f9 [term 5 LEADER]: Becoming Leader. State: Replica: 974b802141bd4f13beffef2aa03f88f9, State: Running, Role: LEADER
I20260129 03:00:00.120208 23118 raft_consensus.cc:493] T 75be4bd6bd45474795fd472eff2953ac P 197b880fa39d432a9b472845876bfaf0 [term 9 FOLLOWER]: Starting pre-election (detected failure of leader 1c50de89a5a845668696dcb28c3badd6)
I20260129 03:00:00.120258 22694 raft_consensus.cc:493] T 368953c9102643b2a21f99af36b51905 P 974b802141bd4f13beffef2aa03f88f9 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 1c50de89a5a845668696dcb28c3badd6)
I20260129 03:00:00.120474 23111 raft_consensus.cc:493] T 0340229d521b432580ae926f6062e3f7 P 197b880fa39d432a9b472845876bfaf0 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 1c50de89a5a845668696dcb28c3badd6)
I20260129 03:00:00.146580 22900 raft_consensus.cc:493] T fcddab6bca684adf8ea032289a00f90e P 1c50de89a5a845668696dcb28c3badd6 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260129 03:00:00.258956 20512 consensus_queue.cc:260] T b189f08e16a74a11980cc55d99359a0a P 974b802141bd4f13beffef2aa03f88f9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 7.5, Last appended by leader: 5, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
W20260129 03:00:00.286011 20407 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.211.2:34517, user_credentials={real_user=slave}} blocked reactor thread for 231865us
I20260129 03:00:00.308455 20512 raft_consensus.cc:3060] T b189f08e16a74a11980cc55d99359a0a P 974b802141bd4f13beffef2aa03f88f9 [term 7 FOLLOWER]: Advancing to term 8
I20260129 03:00:00.286536 20515 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "45cc5f3d2688486d9b6dbe7e666dedde" candidate_uuid: "1c50de89a5a845668696dcb28c3badd6" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "974b802141bd4f13beffef2aa03f88f9" is_pre_election: true
W20260129 03:00:00.311995 20405 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.211.3:37735, user_credentials={real_user=slave}} blocked reactor thread for 243792us
I20260129 03:00:00.034974 22933 raft_consensus.cc:515] T a609ac02fae84725ab743f95f51d33ce P 7a9daf5d42a84615a6383cfec9a10608 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } }
I20260129 03:00:00.142366 22691 raft_consensus.cc:493] T 96609ac68bdf4993a12e1c23355ca013 P 197b880fa39d432a9b472845876bfaf0 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260129 03:00:00.288216 20508 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4f8f3a38f6e24ba4857875181f447442" candidate_uuid: "1c50de89a5a845668696dcb28c3badd6" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "974b802141bd4f13beffef2aa03f88f9" is_pre_election: true
I20260129 03:00:00.314559 21100 leader_election.cc:304] T cb0b3f259ebc4310927338a8740e87b8 P 7a9daf5d42a84615a6383cfec9a10608 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1c50de89a5a845668696dcb28c3badd6, 7a9daf5d42a84615a6383cfec9a10608; no voters:
I20260129 03:00:00.315943 21099 leader_election.cc:304] T 78aa39096868415d9afc2ffcb248e198 P 7a9daf5d42a84615a6383cfec9a10608 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7a9daf5d42a84615a6383cfec9a10608; no voters: 197b880fa39d432a9b472845876bfaf0, 1c50de89a5a845668696dcb28c3badd6
I20260129 03:00:00.323494 21099 leader_election.cc:304] T 4e420f90884f4796997b5ea41e49e411 P 7a9daf5d42a84615a6383cfec9a10608 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 197b880fa39d432a9b472845876bfaf0, 7a9daf5d42a84615a6383cfec9a10608; no voters:
W20260129 03:00:00.317981 20407 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260129 03:00:00.323539 20515 raft_consensus.cc:2393] T 45cc5f3d2688486d9b6dbe7e666dedde P 974b802141bd4f13beffef2aa03f88f9 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1c50de89a5a845668696dcb28c3badd6 in current term 5: Already voted for candidate 974b802141bd4f13beffef2aa03f88f9 in this term.
W20260129 03:00:00.340188 21101 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260129 03:00:00.348039 20432 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55672 (request call id 1631) took 2513 ms. Trace:
I20260129 03:00:00.286566 22922 raft_consensus.cc:493] T 913fc7d751ed4b7388d889aac30a5f9b P 1c50de89a5a845668696dcb28c3badd6 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260129 03:00:00.350176 20432 rpcz_store.cc:276] 0129 02:59:57.835079 (+ 0us) service_pool.cc:168] Inserting onto call queue
0129 02:59:57.905741 (+ 70662us) service_pool.cc:225] Handling call
0129 02:59:59.791060 (+1885319us) spinlock_profiling.cc:243] Waited 15.1 ms on lock 0x7b5800000f98. stack: 00007f8fe044e373 00007f8fe044e270 00007f8fdfda0574 00007f8fe9a5c6b4 00007f8fe9a5c649 00007f8fe02c08b1 00007f8fe0442db1 00007f8fe0442c62 00007f8fe92eddd9 00007f8fe846d260 00007f8fe8535dbb 00007f8fe8535e41 00007f8fe84f7af0 00007f8fe84e2459 00007f8fe84e1f02 00007f8fe1a6ed34
0129 02:59:59.791262 (+ 202us) raft_consensus.cc:1471] Updating replica for 0 ops
0129 02:59:59.791455 (+ 193us) raft_consensus.cc:1534] Early marking committed up to index 3
0129 03:00:00.041245 (+249790us) log.cc:844] Serialized 14 byte log entry
0129 03:00:00.347658 (+306413us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0129 03:00:00.347717 (+ 59us) raft_consensus.cc:1645] Marking committed up to 3
0129 03:00:00.347768 (+ 51us) raft_consensus.cc:1697] Filling consensus response to leader.
0129 03:00:00.347849 (+ 81us) raft_consensus.cc:1691] UpdateReplicas() finished
0129 03:00:00.347986 (+ 137us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":151627,"prepare.queue_time_us":111124,"prepare.run_cpu_time_us":78,"prepare.run_wall_time_us":8144,"spinlock_wait_cycles":34692992,"thread_start_us":358213,"threads_started":2}
I20260129 03:00:00.054382 23047 raft_consensus.cc:515] T add8d4923d1a489e86882c3a25692b95 P 974b802141bd4f13beffef2aa03f88f9 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } }
I20260129 03:00:00.306452 23080 raft_consensus.cc:2804] T 45cc5f3d2688486d9b6dbe7e666dedde P 7a9daf5d42a84615a6383cfec9a10608 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260129 03:00:00.352792 22691 raft_consensus.cc:515] T 96609ac68bdf4993a12e1c23355ca013 P 197b880fa39d432a9b472845876bfaf0 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } attrs { replace: true } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } }
I20260129 03:00:00.381567 20590 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:48526 (request call id 311) took 2368 ms. Trace:
I20260129 03:00:00.381982 20590 rpcz_store.cc:276] 0129 02:59:58.012757 (+ 0us) service_pool.cc:168] Inserting onto call queue
0129 02:59:58.012966 (+ 209us) service_pool.cc:225] Handling call
0129 02:59:58.022855 (+ 9889us) raft_consensus.cc:1471] Updating replica for 0 ops
0129 02:59:58.335827 (+312972us) raft_consensus.cc:1697] Filling consensus response to leader.
0129 03:00:00.381291 (+2045464us) spinlock_profiling.cc:243] Waited 278 ms on lock 0x7b540009bfd4. stack: 00007f8fe044e373 00007f8fe044e270 00007f8fdfda0574 00007f8fe9a5c6b4 00007f8fe9a5c649 00007f8fe34975a1 00007f8fe352ea81 00007f8fe352d4be 00007f8fe84e1f49 00007f8fe1a6ed34 00007f8fe1a6ecc2 00007f8fe1a6ebf1 00007f8fe1a6eb6c 00007f8fe1a6de22 00007f8fe186d34c 00007f8fe186c786
0129 03:00:00.381501 (+ 210us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":640016640}
I20260129 03:00:00.382829 22922 raft_consensus.cc:515] T 913fc7d751ed4b7388d889aac30a5f9b P 1c50de89a5a845668696dcb28c3badd6 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 03:00:00.386754 22922 leader_election.cc:290] T 913fc7d751ed4b7388d889aac30a5f9b P 1c50de89a5a845668696dcb28c3badd6 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 197b880fa39d432a9b472845876bfaf0 (127.18.211.1:42027), 974b802141bd4f13beffef2aa03f88f9 (127.18.211.2:34517)
I20260129 03:00:00.387748 22691 leader_election.cc:290] T 96609ac68bdf4993a12e1c23355ca013 P 197b880fa39d432a9b472845876bfaf0 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 974b802141bd4f13beffef2aa03f88f9 (127.18.211.2:34517), 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3:37735), 7a9daf5d42a84615a6383cfec9a10608 (127.18.211.4:40439)
I20260129 03:00:00.351485 23026 consensus_queue.cc:260] T 368953c9102643b2a21f99af36b51905 P 1c50de89a5a845668696dcb28c3badd6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 6.4, Last appended by leader: 4, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 03:00:00.388767 23101 raft_consensus.cc:3055] T 87d5fe48a44a408aaf03e894a67ecb2f P 197b880fa39d432a9b472845876bfaf0 [term 5 LEADER]: Stepping down as leader of term 5
I20260129 03:00:00.389185 23101 raft_consensus.cc:740] T 87d5fe48a44a408aaf03e894a67ecb2f P 197b880fa39d432a9b472845876bfaf0 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 197b880fa39d432a9b472845876bfaf0, State: Running, Role: LEADER
I20260129 03:00:00.363219 23035 raft_consensus.cc:2804] T 4e420f90884f4796997b5ea41e49e411 P 7a9daf5d42a84615a6383cfec9a10608 [term 6 FOLLOWER]: Leader election won for term 6
I20260129 03:00:00.357713 23100 raft_consensus.cc:493] T febffe4a8b394368b69b1d164308fb9a P 7a9daf5d42a84615a6383cfec9a10608 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260129 03:00:00.390756 23100 raft_consensus.cc:515] T febffe4a8b394368b69b1d164308fb9a P 7a9daf5d42a84615a6383cfec9a10608 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } }
I20260129 03:00:00.391422 23121 raft_consensus.cc:2749] T 78aa39096868415d9afc2ffcb248e198 P 7a9daf5d42a84615a6383cfec9a10608 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260129 03:00:00.393042 23100 leader_election.cc:290] T febffe4a8b394368b69b1d164308fb9a P 7a9daf5d42a84615a6383cfec9a10608 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 197b880fa39d432a9b472845876bfaf0 (127.18.211.1:42027), 974b802141bd4f13beffef2aa03f88f9 (127.18.211.2:34517)
I20260129 03:00:00.393481 23119 raft_consensus.cc:2804] T cb0b3f259ebc4310927338a8740e87b8 P 7a9daf5d42a84615a6383cfec9a10608 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260129 03:00:00.393937 23119 raft_consensus.cc:493] T cb0b3f259ebc4310927338a8740e87b8 P 7a9daf5d42a84615a6383cfec9a10608 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260129 03:00:00.394291 23119 raft_consensus.cc:3060] T cb0b3f259ebc4310927338a8740e87b8 P 7a9daf5d42a84615a6383cfec9a10608 [term 6 FOLLOWER]: Advancing to term 7
W20260129 03:00:00.397550 23095 log.cc:927] Time spent T 1ce4ddf5b10e4ae8b193b13b2e5936ee P 197b880fa39d432a9b472845876bfaf0: Append to log took a long time: real 0.762s user 0.001s sys 0.000s
I20260129 03:00:00.146797 22989 raft_consensus.cc:515] T cedce331e1584ac8b636c9b8a72b2ba0 P 974b802141bd4f13beffef2aa03f88f9 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 03:00:00.260001 22953 raft_consensus.cc:515] T 0340229d521b432580ae926f6062e3f7 P 1c50de89a5a845668696dcb28c3badd6 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 03:00:00.307365 22694 raft_consensus.cc:515] T 368953c9102643b2a21f99af36b51905 P 974b802141bd4f13beffef2aa03f88f9 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 03:00:00.288610 23042 raft_consensus.cc:515] T 0678b8b468bd43cdbc3e485900844821 P 197b880fa39d432a9b472845876bfaf0 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } }
I20260129 03:00:00.325167 23111 raft_consensus.cc:515] T 0340229d521b432580ae926f6062e3f7 P 197b880fa39d432a9b472845876bfaf0 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 03:00:00.459095 21136 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "96609ac68bdf4993a12e1c23355ca013" candidate_uuid: "197b880fa39d432a9b472845876bfaf0" candidate_term: 7 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "7a9daf5d42a84615a6383cfec9a10608" is_pre_election: true
I20260129 03:00:00.473938 20432 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "dc2f4cb752cf4d59b5f6bc689f5d8104" candidate_uuid: "7a9daf5d42a84615a6383cfec9a10608" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "197b880fa39d432a9b472845876bfaf0" is_pre_election: true
I20260129 03:00:00.518119 20512 raft_consensus.cc:1275] T b189f08e16a74a11980cc55d99359a0a P 974b802141bd4f13beffef2aa03f88f9 [term 8 FOLLOWER]: Refusing update from remote peer 1c50de89a5a845668696dcb28c3badd6: Log matching property violated. Preceding OpId in replica: term: 7 index: 5. Preceding OpId from leader: term: 8 index: 4. (term mismatch)
W20260129 03:00:00.425170 20406 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260129 03:00:00.591697 20512 pending_rounds.cc:85] T b189f08e16a74a11980cc55d99359a0a P 974b802141bd4f13beffef2aa03f88f9: Aborting all ops after (but not including) 3
I20260129 03:00:00.592682 20512 pending_rounds.cc:107] T b189f08e16a74a11980cc55d99359a0a P 974b802141bd4f13beffef2aa03f88f9: Aborting uncommitted NO_OP operation due to leader change: 5.4
I20260129 03:00:00.568241 20437 raft_consensus.cc:1240] T 00e79d8f25a34676b1d866eb2068a2b2 P 197b880fa39d432a9b472845876bfaf0 [term 7 FOLLOWER]: Rejecting Update request from peer 7a9daf5d42a84615a6383cfec9a10608 for earlier term 6. Current term is 7. Ops: []
I20260129 03:00:00.357540 23107 raft_consensus.cc:493] T 87d5fe48a44a408aaf03e894a67ecb2f P 7a9daf5d42a84615a6383cfec9a10608 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260129 03:00:00.328472 23096 raft_consensus.cc:2749] T a609ac02fae84725ab743f95f51d33ce P 197b880fa39d432a9b472845876bfaf0 [term 7 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260129 03:00:00.594681 21099 leader_election.cc:304] T dc2f4cb752cf4d59b5f6bc689f5d8104 P 7a9daf5d42a84615a6383cfec9a10608 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7a9daf5d42a84615a6383cfec9a10608; no voters: 197b880fa39d432a9b472845876bfaf0, 1c50de89a5a845668696dcb28c3badd6
I20260129 03:00:00.594074 20434 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "febffe4a8b394368b69b1d164308fb9a" candidate_uuid: "7a9daf5d42a84615a6383cfec9a10608" candidate_term: 5 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "197b880fa39d432a9b472845876bfaf0" is_pre_election: true
I20260129 03:00:00.351394 22640 raft_consensus.cc:493] T 8055232cf0004d40a85f3bf983fe00a9 P 1c50de89a5a845668696dcb28c3badd6 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 974b802141bd4f13beffef2aa03f88f9)
I20260129 03:00:00.402793 23092 leader_election.cc:290] T 4e420f90884f4796997b5ea41e49e411 P 197b880fa39d432a9b472845876bfaf0 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 974b802141bd4f13beffef2aa03f88f9 (127.18.211.2:34517), 7a9daf5d42a84615a6383cfec9a10608 (127.18.211.4:40439)
W20260129 03:00:00.602734 20405 leader_election.cc:336] T a1a39dcd328e4268ae41b988c66b4a75 P 197b880fa39d432a9b472845876bfaf0 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3:37735): Timed out: RequestConsensusVote RPC to 127.18.211.3:37735 timed out after 2.023s (SENT)
I20260129 03:00:00.612460 20405 leader_election.cc:304] T a1a39dcd328e4268ae41b988c66b4a75 P 197b880fa39d432a9b472845876bfaf0 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 197b880fa39d432a9b472845876bfaf0; no voters: 1c50de89a5a845668696dcb28c3badd6, 974b802141bd4f13beffef2aa03f88f9
I20260129 03:00:00.613379 20511 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "926a250fa5834e88b96ccecbefbf5086" candidate_uuid: "1c50de89a5a845668696dcb28c3badd6" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "974b802141bd4f13beffef2aa03f88f9"
I20260129 03:00:00.614255 22688 raft_consensus.cc:2749] T a1a39dcd328e4268ae41b988c66b4a75 P 197b880fa39d432a9b472845876bfaf0 [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260129 03:00:00.617708 22986 consensus_queue.cc:260] T a0fcc58d6fb6473cafbd9d34d5195269 P 974b802141bd4f13beffef2aa03f88f9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 03:00:00.619083 23042 leader_election.cc:290] T 0678b8b468bd43cdbc3e485900844821 P 197b880fa39d432a9b472845876bfaf0 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 974b802141bd4f13beffef2aa03f88f9 (127.18.211.2:34517), 7a9daf5d42a84615a6383cfec9a10608 (127.18.211.4:40439)
I20260129 03:00:00.619335 22986 raft_consensus.cc:3060] T a0fcc58d6fb6473cafbd9d34d5195269 P 974b802141bd4f13beffef2aa03f88f9 [term 3 FOLLOWER]: Advancing to term 4
I20260129 03:00:00.621089 23111 leader_election.cc:290] T 0340229d521b432580ae926f6062e3f7 P 197b880fa39d432a9b472845876bfaf0 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 974b802141bd4f13beffef2aa03f88f9 (127.18.211.2:34517), 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3:37735)
W20260129 03:00:00.623299 23143 log.cc:927] Time spent T 36ff33be666a4c82954731691dbcf327 P 197b880fa39d432a9b472845876bfaf0: Append to log took a long time: real 0.119s user 0.000s sys 0.001s
W20260129 03:00:00.624294 23127 log.cc:927] Time spent T 3ccac82a01a2445fa5b0ce6c0fcf2295 P 7a9daf5d42a84615a6383cfec9a10608: Append to log took a long time: real 0.467s user 0.001s sys 0.000s
I20260129 03:00:00.624922 23116 raft_consensus.cc:493] T e33a417cf9c54cf5af73a4ee3aac4d3a P 197b880fa39d432a9b472845876bfaf0 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 974b802141bd4f13beffef2aa03f88f9)
I20260129 03:00:00.625473 21127 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34046 (request call id 532) took 2710 ms. Trace:
I20260129 03:00:00.625559 23116 raft_consensus.cc:515] T e33a417cf9c54cf5af73a4ee3aac4d3a P 197b880fa39d432a9b472845876bfaf0 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 03:00:00.626071 21127 rpcz_store.cc:276] 0129 02:59:57.915489 (+ 0us) service_pool.cc:168] Inserting onto call queue
0129 02:59:57.938033 (+ 22544us) service_pool.cc:225] Handling call
0129 03:00:00.624575 (+2686542us) spinlock_profiling.cc:243] Waited 2.94 ms on lock 0x7b5800111f98. stack: 00007f8fe044e373 00007f8fe044e270 00007f8fdfda0574 00007f8fe9a5c6b4 00007f8fe9a5c649 00007f8fe02c08b1 00007f8fe0442db1 00007f8fe0442c62 00007f8fe92eddd9 00007f8fe846d260 00007f8fe8535dbb 00007f8fe8535e41 00007f8fe84f7af0 00007f8fe84e2459 00007f8fe84e1f02 00007f8fe1a6ed34
0129 03:00:00.624851 (+ 276us) raft_consensus.cc:1471] Updating replica for 0 ops
0129 03:00:00.625076 (+ 225us) raft_consensus.cc:1534] Early marking committed up to index 0
0129 03:00:00.625106 (+ 30us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0129 03:00:00.625154 (+ 48us) raft_consensus.cc:1645] Marking committed up to 5
0129 03:00:00.625195 (+ 41us) raft_consensus.cc:1697] Filling consensus response to leader.
0129 03:00:00.625283 (+ 88us) raft_consensus.cc:1691] UpdateReplicas() finished
0129 03:00:00.625390 (+ 107us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":6758656}
W20260129 03:00:00.628189 23138 log.cc:927] Time spent T 18f30ebaa4174fd28c958e72d0cc7e24 P 7a9daf5d42a84615a6383cfec9a10608: Append to log took a long time: real 0.077s user 0.000s sys 0.001s
I20260129 03:00:00.593729 20512 raft_consensus.cc:2889] T b189f08e16a74a11980cc55d99359a0a P 974b802141bd4f13beffef2aa03f88f9 [term 8 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260129 03:00:00.631040 20512 pending_rounds.cc:107] T b189f08e16a74a11980cc55d99359a0a P 974b802141bd4f13beffef2aa03f88f9: Aborting uncommitted NO_OP operation due to leader change: 7.5
W20260129 03:00:00.631374 23130 log.cc:927] Time spent T e7d83d4ccf684e76acb29a0dc844e1b8 P 197b880fa39d432a9b472845876bfaf0: Append to log took a long time: real 0.052s user 0.000s sys 0.000s
I20260129 03:00:00.404603 23080 raft_consensus.cc:493] T 45cc5f3d2688486d9b6dbe7e666dedde P 7a9daf5d42a84615a6383cfec9a10608 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260129 03:00:00.609812 20508 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a0fcc58d6fb6473cafbd9d34d5195269" candidate_uuid: "1c50de89a5a845668696dcb28c3badd6" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "974b802141bd4f13beffef2aa03f88f9" is_pre_election: true
W20260129 03:00:00.607156 20438 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.614s user 0.000s sys 0.019s
W20260129 03:00:00.636960 20438 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.644s user 0.000s sys 0.020s
I20260129 03:00:00.631404 20512 raft_consensus.cc:2889] T b189f08e16a74a11980cc55d99359a0a P 974b802141bd4f13beffef2aa03f88f9 [term 8 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260129 03:00:00.633417 20515 raft_consensus.cc:1275] T add8d4923d1a489e86882c3a25692b95 P 974b802141bd4f13beffef2aa03f88f9 [term 5 FOLLOWER]: Refusing update from remote peer 1c50de89a5a845668696dcb28c3badd6: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 3 index: 6. (index mismatch)
I20260129 03:00:00.353830 23118 raft_consensus.cc:515] T 75be4bd6bd45474795fd472eff2953ac P 197b880fa39d432a9b472845876bfaf0 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 03:00:00.635667 20514 raft_consensus.cc:1240] T cedce331e1584ac8b636c9b8a72b2ba0 P 974b802141bd4f13beffef2aa03f88f9 [term 7 FOLLOWER]: Rejecting Update request from peer 197b880fa39d432a9b472845876bfaf0 for earlier term 6. Current term is 7. Ops: []
W20260129 03:00:00.642022 20482 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260129 03:00:00.322901 22900 raft_consensus.cc:515] T fcddab6bca684adf8ea032289a00f90e P 1c50de89a5a845668696dcb28c3badd6 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 03:00:00.474481 23109 raft_consensus.cc:493] T add8d4923d1a489e86882c3a25692b95 P 7a9daf5d42a84615a6383cfec9a10608 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260129 03:00:00.641306 20480 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.801s user 0.012s sys 0.004s
I20260129 03:00:00.488097 23034 raft_consensus.cc:697] T 926a250fa5834e88b96ccecbefbf5086 P 974b802141bd4f13beffef2aa03f88f9 [term 6 LEADER]: Becoming Leader. State: Replica: 974b802141bd4f13beffef2aa03f88f9, State: Running, Role: LEADER
W20260129 03:00:00.674063 20482 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260129 03:00:00.604288 23044 raft_consensus.cc:493] T cb4727386a3e48aca5d71cc6c611c197 P 1c50de89a5a845668696dcb28c3badd6 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260129 03:00:00.402904 23043 consensus_queue.cc:1059] T febffe4a8b394368b69b1d164308fb9a P 197b880fa39d432a9b472845876bfaf0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.238s
I20260129 03:00:00.678315 20512 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c891d2bff4ce4568861bbc992e91fbb0" candidate_uuid: "1c50de89a5a845668696dcb28c3badd6" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "974b802141bd4f13beffef2aa03f88f9" is_pre_election: true
W20260129 03:00:00.681543 20480 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.842s user 0.014s sys 0.004s
I20260129 03:00:00.667703 21134 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4e420f90884f4796997b5ea41e49e411" candidate_uuid: "197b880fa39d432a9b472845876bfaf0" candidate_term: 7 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "7a9daf5d42a84615a6383cfec9a10608" is_pre_election: true
I20260129 03:00:00.640727 22656 leader_election.cc:290] T 1299723a4476420ab02fd8bfd7aa0645 P 1c50de89a5a845668696dcb28c3badd6 [CANDIDATE]: Term 9 election: Requested vote from peers 197b880fa39d432a9b472845876bfaf0 (127.18.211.1:42027), 7a9daf5d42a84615a6383cfec9a10608 (127.18.211.4:40439)
I20260129 03:00:00.634135 22640 raft_consensus.cc:515] T 8055232cf0004d40a85f3bf983fe00a9 P 1c50de89a5a845668696dcb28c3badd6 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 03:00:00.455610 23078 consensus_queue.cc:237] T a8479d1fe0bd41e5946186e3a6ca8c7a P 974b802141bd4f13beffef2aa03f88f9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 03:00:00.605394 23107 raft_consensus.cc:515] T 87d5fe48a44a408aaf03e894a67ecb2f P 7a9daf5d42a84615a6383cfec9a10608 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } }
W20260129 03:00:00.879058 20407 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260129 03:00:00.794531 20406 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260129 03:00:00.885761 20556 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.534s user 0.005s sys 0.013s
W20260129 03:00:00.898583 20556 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.546s user 0.005s sys 0.014s
I20260129 03:00:00.642477 23080 raft_consensus.cc:3060] T 45cc5f3d2688486d9b6dbe7e666dedde P 7a9daf5d42a84615a6383cfec9a10608 [term 4 FOLLOWER]: Advancing to term 5
I20260129 03:00:00.886250 20512 raft_consensus.cc:2468] T c891d2bff4ce4568861bbc992e91fbb0 P 974b802141bd4f13beffef2aa03f88f9 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1c50de89a5a845668696dcb28c3badd6 in term 7.
I20260129 03:00:00.737228 22978 raft_consensus.cc:493] T 505580c93c694bbb8ea96fdf9b17ab46 P 974b802141bd4f13beffef2aa03f88f9 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 1c50de89a5a845668696dcb28c3badd6)
I20260129 03:00:00.879045 20507 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3906b080a5b34a74b8b64128635e8997" candidate_uuid: "1c50de89a5a845668696dcb28c3badd6" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "974b802141bd4f13beffef2aa03f88f9" is_pre_election: true
W20260129 03:00:00.887140 20480 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.211.3:37735, user_credentials={real_user=slave}} blocked reactor thread for 1.09403e+06us
I20260129 03:00:00.736927 23150 raft_consensus.cc:2749] T dc2f4cb752cf4d59b5f6bc689f5d8104 P 7a9daf5d42a84615a6383cfec9a10608 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260129 03:00:00.723115 23109 raft_consensus.cc:515] T add8d4923d1a489e86882c3a25692b95 P 7a9daf5d42a84615a6383cfec9a10608 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } }
I20260129 03:00:00.745604 23055 raft_consensus.cc:3055] T 75f45538dc704d9a8ec4ff8ad9f3ea5b P 974b802141bd4f13beffef2aa03f88f9 [term 3 LEADER]: Stepping down as leader of term 3
I20260129 03:00:00.841459 23096 raft_consensus.cc:493] T 76f713c3331540d2a5b76164103e9eb1 P 197b880fa39d432a9b472845876bfaf0 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 1c50de89a5a845668696dcb28c3badd6)
W20260129 03:00:00.892496 21135 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.636s user 0.017s sys 0.025s
I20260129 03:00:00.728514 20516 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "36ff33be666a4c82954731691dbcf327" candidate_uuid: "197b880fa39d432a9b472845876bfaf0" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "974b802141bd4f13beffef2aa03f88f9" is_pre_election: true
W20260129 03:00:00.938092 21135 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.682s user 0.018s sys 0.026s
I20260129 03:00:00.687184 20515 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e7d83d4ccf684e76acb29a0dc844e1b8" candidate_uuid: "1c50de89a5a845668696dcb28c3badd6" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "974b802141bd4f13beffef2aa03f88f9" is_pre_election: true
W20260129 03:00:00.878701 21099 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.211.1:42027, user_credentials={real_user=slave}} blocked reactor thread for 284097us
W20260129 03:00:00.902122 22884 consensus_meta.cc:287] T 724a236524cb4e7ba3428880681d6cc2 P 1c50de89a5a845668696dcb28c3badd6: Time spent flushing consensus metadata: real 1.447s user 0.007s sys 0.008s
I20260129 03:00:00.744053 22694 leader_election.cc:290] T 368953c9102643b2a21f99af36b51905 P 974b802141bd4f13beffef2aa03f88f9 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 197b880fa39d432a9b472845876bfaf0 (127.18.211.1:42027), 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3:37735)
I20260129 03:00:00.641052 23035 raft_consensus.cc:697] T 4e420f90884f4796997b5ea41e49e411 P 7a9daf5d42a84615a6383cfec9a10608 [term 6 LEADER]: Becoming Leader. State: Replica: 7a9daf5d42a84615a6383cfec9a10608, State: Running, Role: LEADER
I20260129 03:00:00.929322 20512 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fcddab6bca684adf8ea032289a00f90e" candidate_uuid: "197b880fa39d432a9b472845876bfaf0" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "974b802141bd4f13beffef2aa03f88f9"
I20260129 03:00:00.946017 20516 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "96609ac68bdf4993a12e1c23355ca013" candidate_uuid: "197b880fa39d432a9b472845876bfaf0" candidate_term: 7 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "974b802141bd4f13beffef2aa03f88f9" is_pre_election: true
I20260129 03:00:00.946779 20512 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e4d18c35b0a7493e85d0a4b92956a48f" candidate_uuid: "197b880fa39d432a9b472845876bfaf0" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "974b802141bd4f13beffef2aa03f88f9" is_pre_election: true
I20260129 03:00:00.744297 23116 leader_election.cc:290] T e33a417cf9c54cf5af73a4ee3aac4d3a P 197b880fa39d432a9b472845876bfaf0 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 974b802141bd4f13beffef2aa03f88f9 (127.18.211.2:34517), 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3:37735)
I20260129 03:00:00.901194 22881 raft_consensus.cc:493] T a5a05fbcbbcf4916bbc01fd1a5b5e90d P 1c50de89a5a845668696dcb28c3badd6 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 974b802141bd4f13beffef2aa03f88f9)
I20260129 03:00:00.939797 21136 raft_consensus.cc:1240] T 87d5fe48a44a408aaf03e894a67ecb2f P 7a9daf5d42a84615a6383cfec9a10608 [term 6 FOLLOWER]: Rejecting Update request from peer 197b880fa39d432a9b472845876bfaf0 for earlier term 5. Current term is 6. Ops: [5.7-5.7]
I20260129 03:00:00.955243 20585 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3906b080a5b34a74b8b64128635e8997" candidate_uuid: "197b880fa39d432a9b472845876bfaf0" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "1c50de89a5a845668696dcb28c3badd6" is_pre_election: true
I20260129 03:00:00.922427 23047 leader_election.cc:290] T add8d4923d1a489e86882c3a25692b95 P 974b802141bd4f13beffef2aa03f88f9 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3:37735), 7a9daf5d42a84615a6383cfec9a10608 (127.18.211.4:40439)
I20260129 03:00:00.737584 22975 raft_consensus.cc:3055] T 4b1876f9a70b49edb3aa74105e66d3b3 P 974b802141bd4f13beffef2aa03f88f9 [term 4 LEADER]: Stepping down as leader of term 4
I20260129 03:00:00.737793 23133 raft_consensus.cc:493] T 17d6431cc77a491e8464d68db4081a51 P 197b880fa39d432a9b472845876bfaf0 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 974b802141bd4f13beffef2aa03f88f9)
W20260129 03:00:00.941625 20555 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260129 03:00:00.942389 20440 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a609ac02fae84725ab743f95f51d33ce" candidate_uuid: "7a9daf5d42a84615a6383cfec9a10608" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "197b880fa39d432a9b472845876bfaf0" is_pre_election: true
I20260129 03:00:00.900151 22933 leader_election.cc:290] T a609ac02fae84725ab743f95f51d33ce P 7a9daf5d42a84615a6383cfec9a10608 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 197b880fa39d432a9b472845876bfaf0 (127.18.211.1:42027), 974b802141bd4f13beffef2aa03f88f9 (127.18.211.2:34517)
I20260129 03:00:00.972741 20588 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "368953c9102643b2a21f99af36b51905" candidate_uuid: "197b880fa39d432a9b472845876bfaf0" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "1c50de89a5a845668696dcb28c3badd6" is_pre_election: true
I20260129 03:00:00.947127 20516 raft_consensus.cc:2410] T 96609ac68bdf4993a12e1c23355ca013 P 974b802141bd4f13beffef2aa03f88f9 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 197b880fa39d432a9b472845876bfaf0 for term 7 because replica has last-logged OpId of term: 3 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 4.
I20260129 03:00:00.948508 20512 raft_consensus.cc:1240] T cb4727386a3e48aca5d71cc6c611c197 P 974b802141bd4f13beffef2aa03f88f9 [term 4 FOLLOWER]: Rejecting Update request from peer 197b880fa39d432a9b472845876bfaf0 for earlier term 3. Current term is 4. Ops: []
I20260129 03:00:00.934350 20507 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a5a05fbcbbcf4916bbc01fd1a5b5e90d" candidate_uuid: "197b880fa39d432a9b472845876bfaf0" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "974b802141bd4f13beffef2aa03f88f9" is_pre_election: true
I20260129 03:00:00.743167 21130 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34046 (request call id 540) took 1304 ms. Trace:
I20260129 03:00:00.878501 23037 consensus_queue.cc:1059] T fd8311b779af4e1fa1402b2a02a602f2 P 7a9daf5d42a84615a6383cfec9a10608 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 }, Status: INVALID_TERM, Last received: 4.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.058s
I20260129 03:00:00.952206 20581 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a1a39dcd328e4268ae41b988c66b4a75" candidate_uuid: "197b880fa39d432a9b472845876bfaf0" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "1c50de89a5a845668696dcb28c3badd6" is_pre_election: true
I20260129 03:00:00.952510 20590 raft_consensus.cc:1240] T 913fc7d751ed4b7388d889aac30a5f9b P 1c50de89a5a845668696dcb28c3badd6 [term 4 FOLLOWER]: Rejecting Update request from peer 197b880fa39d432a9b472845876bfaf0 for earlier term 3. Current term is 4. Ops: []
I20260129 03:00:00.934167 22656 raft_consensus.cc:493] T a8479d1fe0bd41e5946186e3a6ca8c7a P 1c50de89a5a845668696dcb28c3badd6 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260129 03:00:00.934005 23055 raft_consensus.cc:740] T 75f45538dc704d9a8ec4ff8ad9f3ea5b P 974b802141bd4f13beffef2aa03f88f9 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 974b802141bd4f13beffef2aa03f88f9, State: Running, Role: LEADER
W20260129 03:00:00.953042 20480 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260129 03:00:00.940693 20515 raft_consensus.cc:2410] T e7d83d4ccf684e76acb29a0dc844e1b8 P 974b802141bd4f13beffef2aa03f88f9 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1c50de89a5a845668696dcb28c3badd6 for term 7 because replica has last-logged OpId of term: 5 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 4 index: 3.
I20260129 03:00:00.956429 20585 raft_consensus.cc:2468] T 3906b080a5b34a74b8b64128635e8997 P 1c50de89a5a845668696dcb28c3badd6 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 197b880fa39d432a9b472845876bfaf0 in term 4.
I20260129 03:00:00.958226 20584 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "dc2f4cb752cf4d59b5f6bc689f5d8104" candidate_uuid: "197b880fa39d432a9b472845876bfaf0" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "1c50de89a5a845668696dcb28c3badd6" is_pre_election: true
I20260129 03:00:00.960011 20589 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2e58e9a5a8aa4cecafb464addb4f35ed" candidate_uuid: "197b880fa39d432a9b472845876bfaf0" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "1c50de89a5a845668696dcb28c3badd6" is_pre_election: true
I20260129 03:00:00.989789 20589 raft_consensus.cc:1240] T e7d83d4ccf684e76acb29a0dc844e1b8 P 1c50de89a5a845668696dcb28c3badd6 [term 6 FOLLOWER]: Rejecting Update request from peer 197b880fa39d432a9b472845876bfaf0 for earlier term 5. Current term is 6. Ops: []
I20260129 03:00:00.989555 20507 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0678b8b468bd43cdbc3e485900844821" candidate_uuid: "197b880fa39d432a9b472845876bfaf0" candidate_term: 8 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "974b802141bd4f13beffef2aa03f88f9" is_pre_election: true
I20260129 03:00:00.989634 20584 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "78aa39096868415d9afc2ffcb248e198" candidate_uuid: "197b880fa39d432a9b472845876bfaf0" candidate_term: 6 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "1c50de89a5a845668696dcb28c3badd6" is_pre_election: true
W20260129 03:00:00.991999 20406 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260129 03:00:00.971483 20440 raft_consensus.cc:2468] T a609ac02fae84725ab743f95f51d33ce P 197b880fa39d432a9b472845876bfaf0 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7a9daf5d42a84615a6383cfec9a10608 in term 7.
I20260129 03:00:00.968215 20586 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c2633a6fc34c44579de348d208b0c01e" candidate_uuid: "197b880fa39d432a9b472845876bfaf0" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "1c50de89a5a845668696dcb28c3badd6" is_pre_election: true
I20260129 03:00:00.741037 21127 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0678b8b468bd43cdbc3e485900844821" candidate_uuid: "197b880fa39d432a9b472845876bfaf0" candidate_term: 8 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "7a9daf5d42a84615a6383cfec9a10608" is_pre_election: true
I20260129 03:00:00.937881 23128 raft_consensus.cc:493] T cedce331e1584ac8b636c9b8a72b2ba0 P 1c50de89a5a845668696dcb28c3badd6 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260129 03:00:00.947744 22924 raft_consensus.cc:493] T e4d18c35b0a7493e85d0a4b92956a48f P 1c50de89a5a845668696dcb28c3badd6 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260129 03:00:00.967280 22975 raft_consensus.cc:740] T 4b1876f9a70b49edb3aa74105e66d3b3 P 974b802141bd4f13beffef2aa03f88f9 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 974b802141bd4f13beffef2aa03f88f9, State: Running, Role: LEADER
I20260129 03:00:00.947978 22728 raft_consensus.cc:493] T 1ce4ddf5b10e4ae8b193b13b2e5936ee P 1c50de89a5a845668696dcb28c3badd6 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260129 03:00:00.974119 23154 raft_consensus.cc:493] T 724a236524cb4e7ba3428880681d6cc2 P 974b802141bd4f13beffef2aa03f88f9 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 1c50de89a5a845668696dcb28c3badd6)
I20260129 03:00:00.948554 23096 raft_consensus.cc:515] T 76f713c3331540d2a5b76164103e9eb1 P 197b880fa39d432a9b472845876bfaf0 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 03:00:00.952142 23145 raft_consensus.cc:493] T 3ccac82a01a2445fa5b0ce6c0fcf2295 P 974b802141bd4f13beffef2aa03f88f9 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260129 03:00:00.984526 20512 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4e420f90884f4796997b5ea41e49e411" candidate_uuid: "197b880fa39d432a9b472845876bfaf0" candidate_term: 7 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "974b802141bd4f13beffef2aa03f88f9" is_pre_election: true
I20260129 03:00:00.992861 20585 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1ce4ddf5b10e4ae8b193b13b2e5936ee" candidate_uuid: "197b880fa39d432a9b472845876bfaf0" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "1c50de89a5a845668696dcb28c3badd6" is_pre_election: true
W20260129 03:00:00.957652 22903 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 1.440s user 0.019s sys 0.044s
I20260129 03:00:00.994597 20584 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c2633a6fc34c44579de348d208b0c01e" candidate_uuid: "197b880fa39d432a9b472845876bfaf0" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "1c50de89a5a845668696dcb28c3badd6"
I20260129 03:00:00.956040 23147 raft_consensus.cc:493] T c2633a6fc34c44579de348d208b0c01e P 1c50de89a5a845668696dcb28c3badd6 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260129 03:00:00.996330 20507 raft_consensus.cc:1275] T 4408006eb5174de1896b037cf3f18457 P 974b802141bd4f13beffef2aa03f88f9 [term 4 FOLLOWER]: Refusing update from remote peer 197b880fa39d432a9b472845876bfaf0: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
W20260129 03:00:00.998919 21100 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.211.3:37735, user_credentials={real_user=slave}} blocked reactor thread for 388305us
W20260129 03:00:01.000824 20480 connection.cc:232] RPC call timeout handler was delayed by -1.7499531219999653s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.221s for a response
I20260129 03:00:00.967408 23116 raft_consensus.cc:493] T 4e3805ffa70242e8ade66da260dddf66 P 197b880fa39d432a9b472845876bfaf0 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260129 03:00:00.957758 23152 raft_consensus.cc:493] T a1a39dcd328e4268ae41b988c66b4a75 P 974b802141bd4f13beffef2aa03f88f9 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 1c50de89a5a845668696dcb28c3badd6)
I20260129 03:00:01.003614 20581 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1299723a4476420ab02fd8bfd7aa0645" candidate_uuid: "197b880fa39d432a9b472845876bfaf0" candidate_term: 9 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "1c50de89a5a845668696dcb28c3badd6" is_pre_election: true
I20260129 03:00:01.000852 20590 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a8479d1fe0bd41e5946186e3a6ca8c7a" candidate_uuid: "197b880fa39d432a9b472845876bfaf0" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "1c50de89a5a845668696dcb28c3badd6" is_pre_election: true
W20260129 03:00:00.976114 21101 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260129 03:00:00.979362 22989 leader_election.cc:290] T cedce331e1584ac8b636c9b8a72b2ba0 P 974b802141bd4f13beffef2aa03f88f9 [CANDIDATE]: Term 7 election: Requested vote from peers 197b880fa39d432a9b472845876bfaf0 (127.18.211.1:42027), 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3:37735)
I20260129 03:00:00.922405 23044 raft_consensus.cc:515] T cb4727386a3e48aca5d71cc6c611c197 P 1c50de89a5a845668696dcb28c3badd6 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 03:00:00.980639 20583 raft_consensus.cc:1240] T 7a9a69cd82724b1faed1d10aa091fc4a P 1c50de89a5a845668696dcb28c3badd6 [term 6 LEADER]: Rejecting Update request from peer 197b880fa39d432a9b472845876bfaf0 for earlier term 5. Current term is 6. Ops: []
I20260129 03:00:00.944542 23146 consensus_queue.cc:1048] T e33a417cf9c54cf5af73a4ee3aac4d3a P 974b802141bd4f13beffef2aa03f88f9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.066s
I20260129 03:00:01.004607 21127 raft_consensus.cc:2468] T 0678b8b468bd43cdbc3e485900844821 P 7a9daf5d42a84615a6383cfec9a10608 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 197b880fa39d432a9b472845876bfaf0 in term 6.
I20260129 03:00:00.991178 22656 raft_consensus.cc:515] T a8479d1fe0bd41e5946186e3a6ca8c7a P 1c50de89a5a845668696dcb28c3badd6 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 03:00:00.967305 22881 raft_consensus.cc:515] T a5a05fbcbbcf4916bbc01fd1a5b5e90d P 1c50de89a5a845668696dcb28c3badd6 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 03:00:01.011327 22900 leader_election.cc:290] T fcddab6bca684adf8ea032289a00f90e P 1c50de89a5a845668696dcb28c3badd6 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 197b880fa39d432a9b472845876bfaf0 (127.18.211.1:42027), 974b802141bd4f13beffef2aa03f88f9 (127.18.211.2:34517)
I20260129 03:00:00.980192 23133 raft_consensus.cc:515] T 17d6431cc77a491e8464d68db4081a51 P 197b880fa39d432a9b472845876bfaf0 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 03:00:01.011790 20515 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0340229d521b432580ae926f6062e3f7" candidate_uuid: "197b880fa39d432a9b472845876bfaf0" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "974b802141bd4f13beffef2aa03f88f9" is_pre_election: true
I20260129 03:00:01.012081 23141 raft_consensus.cc:493] T 42d27a28d89b409e8807ec0efd97af69 P 7a9daf5d42a84615a6383cfec9a10608 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260129 03:00:01.065891 20515 raft_consensus.cc:2393] T 0340229d521b432580ae926f6062e3f7 P 974b802141bd4f13beffef2aa03f88f9 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 197b880fa39d432a9b472845876bfaf0 in current term 6: Already voted for candidate 974b802141bd4f13beffef2aa03f88f9 in this term.
W20260129 03:00:01.012283 23140 log.cc:927] Time spent T 96609ac68bdf4993a12e1c23355ca013 P 1c50de89a5a845668696dcb28c3badd6: Append to log took a long time: real 0.271s user 0.000s sys 0.001s
I20260129 03:00:01.012562 23109 leader_election.cc:290] T add8d4923d1a489e86882c3a25692b95 P 7a9daf5d42a84615a6383cfec9a10608 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 974b802141bd4f13beffef2aa03f88f9 (127.18.211.2:34517), 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3:37735)
I20260129 03:00:01.013023 23154 raft_consensus.cc:515] T 724a236524cb4e7ba3428880681d6cc2 P 974b802141bd4f13beffef2aa03f88f9 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 03:00:01.014515 23107 leader_election.cc:290] T 87d5fe48a44a408aaf03e894a67ecb2f P 7a9daf5d42a84615a6383cfec9a10608 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 197b880fa39d432a9b472845876bfaf0 (127.18.211.1:42027), 974b802141bd4f13beffef2aa03f88f9 (127.18.211.2:34517)
W20260129 03:00:01.014977 23058 log.cc:927] Time spent T 913fc7d751ed4b7388d889aac30a5f9b P 197b880fa39d432a9b472845876bfaf0: Append to log took a long time: real 2.795s user 0.004s sys 0.011s
I20260129 03:00:01.015295 23118 leader_election.cc:290] T 75be4bd6bd45474795fd472eff2953ac P 197b880fa39d432a9b472845876bfaf0 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 974b802141bd4f13beffef2aa03f88f9 (127.18.211.2:34517), 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3:37735)
I20260129 03:00:01.009461 22924 raft_consensus.cc:515] T e4d18c35b0a7493e85d0a4b92956a48f P 1c50de89a5a845668696dcb28c3badd6 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
W20260129 03:00:01.017704 23148 log.cc:927] Time spent T b189f08e16a74a11980cc55d99359a0a P 197b880fa39d432a9b472845876bfaf0: Append to log took a long time: real 0.141s user 0.001s sys 0.000s
I20260129 03:00:01.019228 23153 raft_consensus.cc:493] T c2633a6fc34c44579de348d208b0c01e P 974b802141bd4f13beffef2aa03f88f9 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 1c50de89a5a845668696dcb28c3badd6)
I20260129 03:00:00.935120 22978 raft_consensus.cc:515] T 505580c93c694bbb8ea96fdf9b17ab46 P 974b802141bd4f13beffef2aa03f88f9 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } }
I20260129 03:00:01.028968 21132 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34060 (request call id 507) took 2406 ms. Trace:
I20260129 03:00:01.049129 20480 leader_election.cc:304] T 75b0b6e82ddc4f92b1fe1d94f9a0b73b P 974b802141bd4f13beffef2aa03f88f9 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 974b802141bd4f13beffef2aa03f88f9; no voters: 1c50de89a5a845668696dcb28c3badd6, 7a9daf5d42a84615a6383cfec9a10608
I20260129 03:00:01.050767 20589 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "8055232cf0004d40a85f3bf983fe00a9" candidate_uuid: "197b880fa39d432a9b472845876bfaf0" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "1c50de89a5a845668696dcb28c3badd6"
I20260129 03:00:01.144546 20589 raft_consensus.cc:3060] T 8055232cf0004d40a85f3bf983fe00a9 P 1c50de89a5a845668696dcb28c3badd6 [term 3 FOLLOWER]: Advancing to term 4
I20260129 03:00:01.053133 20581 raft_consensus.cc:2393] T 1299723a4476420ab02fd8bfd7aa0645 P 1c50de89a5a845668696dcb28c3badd6 [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 197b880fa39d432a9b472845876bfaf0 in current term 9: Already voted for candidate 1c50de89a5a845668696dcb28c3badd6 in this term.
W20260129 03:00:01.145078 20480 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.211.4:40439, user_credentials={real_user=slave}} blocked reactor thread for 96062.4us
I20260129 03:00:01.053203 21134 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0678b8b468bd43cdbc3e485900844821" candidate_uuid: "974b802141bd4f13beffef2aa03f88f9" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "7a9daf5d42a84615a6383cfec9a10608"
I20260129 03:00:01.146123 21134 raft_consensus.cc:3060] T 0678b8b468bd43cdbc3e485900844821 P 7a9daf5d42a84615a6383cfec9a10608 [term 6 FOLLOWER]: Advancing to term 7
I20260129 03:00:01.059365 20512 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3e2d15f8640b407195210c476e09df91" candidate_uuid: "7a9daf5d42a84615a6383cfec9a10608" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "974b802141bd4f13beffef2aa03f88f9" is_pre_election: true
I20260129 03:00:01.147028 20512 raft_consensus.cc:2468] T 3e2d15f8640b407195210c476e09df91 P 974b802141bd4f13beffef2aa03f88f9 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7a9daf5d42a84615a6383cfec9a10608 in term 3.
I20260129 03:00:01.009056 22640 leader_election.cc:290] T 8055232cf0004d40a85f3bf983fe00a9 P 1c50de89a5a845668696dcb28c3badd6 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 197b880fa39d432a9b472845876bfaf0 (127.18.211.1:42027), 974b802141bd4f13beffef2aa03f88f9 (127.18.211.2:34517)
W20260129 03:00:01.035985 22903 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 1.520s user 0.020s sys 0.045s
I20260129 03:00:01.153937 20512 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4f5a77e94fef4b16a2ba7b067c8b144a" candidate_uuid: "7a9daf5d42a84615a6383cfec9a10608" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "974b802141bd4f13beffef2aa03f88f9" is_pre_election: true
I20260129 03:00:01.154641 20512 raft_consensus.cc:2393] T 4f5a77e94fef4b16a2ba7b067c8b144a P 974b802141bd4f13beffef2aa03f88f9 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 7a9daf5d42a84615a6383cfec9a10608 in current term 4: Already voted for candidate 974b802141bd4f13beffef2aa03f88f9 in this term.
I20260129 03:00:01.058532 20507 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2ac659d732314e1288849f08fe5e074e" candidate_uuid: "7a9daf5d42a84615a6383cfec9a10608" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "974b802141bd4f13beffef2aa03f88f9"
I20260129 03:00:01.129858 21128 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c934d31efd3f4e5195605ddc354d14e1" candidate_uuid: "974b802141bd4f13beffef2aa03f88f9" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "7a9daf5d42a84615a6383cfec9a10608" is_pre_election: true
I20260129 03:00:01.107391 23042 consensus_queue.cc:1059] T cb4727386a3e48aca5d71cc6c611c197 P 197b880fa39d432a9b472845876bfaf0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 }, Status: INVALID_TERM, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.040s
W20260129 03:00:01.089438 20406 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260129 03:00:01.098034 20515 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0678b8b468bd43cdbc3e485900844821" candidate_uuid: "7a9daf5d42a84615a6383cfec9a10608" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "974b802141bd4f13beffef2aa03f88f9" is_pre_election: true
W20260129 03:00:01.016672 23142 log.cc:927] Time spent T 75f45538dc704d9a8ec4ff8ad9f3ea5b P 7a9daf5d42a84615a6383cfec9a10608: Append to log took a long time: real 0.081s user 0.000s sys 0.000s
I20260129 03:00:01.024732 22728 raft_consensus.cc:515] T 1ce4ddf5b10e4ae8b193b13b2e5936ee P 1c50de89a5a845668696dcb28c3badd6 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 03:00:01.098052 20590 raft_consensus.cc:2393] T a8479d1fe0bd41e5946186e3a6ca8c7a P 1c50de89a5a845668696dcb28c3badd6 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 197b880fa39d432a9b472845876bfaf0 in current term 5: Already voted for candidate 974b802141bd4f13beffef2aa03f88f9 in this term.
I20260129 03:00:01.104492 20407 leader_election.cc:304] T 96609ac68bdf4993a12e1c23355ca013 P 197b880fa39d432a9b472845876bfaf0 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 197b880fa39d432a9b472845876bfaf0; no voters: 7a9daf5d42a84615a6383cfec9a10608, 974b802141bd4f13beffef2aa03f88f9
I20260129 03:00:01.053733 23147 raft_consensus.cc:515] T c2633a6fc34c44579de348d208b0c01e P 1c50de89a5a845668696dcb28c3badd6 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 03:00:01.107884 20438 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55676 (request call id 1391) took 1384 ms. Trace:
I20260129 03:00:01.129424 20405 leader_election.cc:304] T 0678b8b468bd43cdbc3e485900844821 P 197b880fa39d432a9b472845876bfaf0 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 197b880fa39d432a9b472845876bfaf0, 7a9daf5d42a84615a6383cfec9a10608; no voters:
I20260129 03:00:01.131479 21099 leader_election.cc:304] T a609ac02fae84725ab743f95f51d33ce P 7a9daf5d42a84615a6383cfec9a10608 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 197b880fa39d432a9b472845876bfaf0, 7a9daf5d42a84615a6383cfec9a10608; no voters:
I20260129 03:00:01.107234 21127 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4f5a77e94fef4b16a2ba7b067c8b144a" candidate_uuid: "974b802141bd4f13beffef2aa03f88f9" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "7a9daf5d42a84615a6383cfec9a10608"
I20260129 03:00:01.187219 20438 rpcz_store.cc:276] 0129 02:59:59.723959 (+ 0us) service_pool.cc:168] Inserting onto call queue
0129 02:59:59.947279 (+223320us) service_pool.cc:225] Handling call
0129 02:59:59.976978 (+ 29699us) raft_consensus.cc:1471] Updating replica for 1 ops
0129 02:59:59.977295 (+ 317us) raft_consensus.cc:1534] Early marking committed up to index 0
0129 02:59:59.977323 (+ 28us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0129 02:59:59.992899 (+ 15576us) log.cc:844] Serialized 26 byte log entry
0129 03:00:00.637440 (+644541us) raft_consensus.cc:1645] Marking committed up to 3
0129 03:00:00.637494 (+ 54us) raft_consensus.cc:1697] Filling consensus response to leader.
0129 03:00:00.637584 (+ 90us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0129 03:00:01.107643 (+470059us) raft_consensus.cc:1685] finished
0129 03:00:01.107664 (+ 21us) raft_consensus.cc:1691] UpdateReplicas() finished
0129 03:00:01.107808 (+ 144us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":876,"raft.queue_time_us":1138,"raft.run_cpu_time_us":124,"raft.run_wall_time_us":18262,"thread_start_us":644140,"threads_started":1,"wal-append.queue_time_us":861895}
I20260129 03:00:01.005779 23128 raft_consensus.cc:515] T cedce331e1584ac8b636c9b8a72b2ba0 P 1c50de89a5a845668696dcb28c3badd6 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 03:00:01.057395 23116 raft_consensus.cc:515] T 4e3805ffa70242e8ade66da260dddf66 P 197b880fa39d432a9b472845876bfaf0 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } }
I20260129 03:00:01.111804 21132 rpcz_store.cc:276] 0129 02:59:58.623147 (+ 0us) service_pool.cc:168] Inserting onto call queue
0129 02:59:58.623314 (+ 167us) service_pool.cc:225] Handling call
0129 02:59:58.623682 (+ 368us) raft_consensus.cc:1471] Updating replica for 0 ops
0129 02:59:59.453245 (+829563us) raft_consensus.cc:1697] Filling consensus response to leader.
0129 03:00:01.028713 (+1575468us) spinlock_profiling.cc:243] Waited 829 ms on lock 0x7b54005dedd4. stack: 00007f8fe044e373 00007f8fe044e270 00007f8fdfda0574 00007f8fe9a5c6b4 00007f8fe9a5c649 00007f8fe34975a1 00007f8fe352ea81 00007f8fe352d4be 00007f8fe84e1f49 00007f8fe1a6ed34 00007f8fe1a6ecc2 00007f8fe1a6ebf1 00007f8fe1a6eb6c 00007f8fe1a6de22 00007f8fe186d34c 00007f8fe186c786
0129 03:00:01.028902 (+ 189us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1906139520}
I20260129 03:00:01.104979 23163 raft_consensus.cc:3055] T fd8311b779af4e1fa1402b2a02a602f2 P 7a9daf5d42a84615a6383cfec9a10608 [term 4 LEADER]: Stepping down as leader of term 4
I20260129 03:00:01.071846 20584 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "926a250fa5834e88b96ccecbefbf5086" candidate_uuid: "197b880fa39d432a9b472845876bfaf0" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "1c50de89a5a845668696dcb28c3badd6" is_pre_election: true
W20260129 03:00:01.155692 20553 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.211.1:42027, user_credentials={real_user=slave}} blocked reactor thread for 166685us
I20260129 03:00:01.090850 23141 raft_consensus.cc:515] T 42d27a28d89b409e8807ec0efd97af69 P 7a9daf5d42a84615a6383cfec9a10608 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } }
I20260129 03:00:00.987449 23034 consensus_queue.cc:237] T 926a250fa5834e88b96ccecbefbf5086 P 974b802141bd4f13beffef2aa03f88f9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
W20260129 03:00:01.217244 21099 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.211.1:42027, user_credentials={real_user=slave}} blocked reactor thread for 85860.4us
I20260129 03:00:01.161283 20581 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "724a236524cb4e7ba3428880681d6cc2" candidate_uuid: "197b880fa39d432a9b472845876bfaf0" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "1c50de89a5a845668696dcb28c3badd6"
W20260129 03:00:01.218748 23162 log.cc:927] Time spent T 7a9a69cd82724b1faed1d10aa091fc4a P 974b802141bd4f13beffef2aa03f88f9: Append to log took a long time: real 0.061s user 0.000s sys 0.000s
I20260129 03:00:01.219059 20581 raft_consensus.cc:2410] T 724a236524cb4e7ba3428880681d6cc2 P 1c50de89a5a845668696dcb28c3badd6 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 197b880fa39d432a9b472845876bfaf0 for term 5 because replica has last-logged OpId of term: 4 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 2.
I20260129 03:00:01.220000 20581 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "368953c9102643b2a21f99af36b51905" candidate_uuid: "197b880fa39d432a9b472845876bfaf0" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "1c50de89a5a845668696dcb28c3badd6"
I20260129 03:00:01.220317 20513 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57584 (request call id 1401) took 1087 ms. Trace:
I20260129 03:00:01.063237 23035 consensus_queue.cc:237] T 4e420f90884f4796997b5ea41e49e411 P 7a9daf5d42a84615a6383cfec9a10608 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } }
I20260129 03:00:01.255805 20585 raft_consensus.cc:2410] T 1ce4ddf5b10e4ae8b193b13b2e5936ee P 1c50de89a5a845668696dcb28c3badd6 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 197b880fa39d432a9b472845876bfaf0 for term 7 because replica has last-logged OpId of term: 6 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 5 index: 3.
I20260129 03:00:01.037042 23145 raft_consensus.cc:515] T 3ccac82a01a2445fa5b0ce6c0fcf2295 P 974b802141bd4f13beffef2aa03f88f9 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } }
I20260129 03:00:01.072045 23152 raft_consensus.cc:515] T a1a39dcd328e4268ae41b988c66b4a75 P 974b802141bd4f13beffef2aa03f88f9 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } } peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } }
I20260129 03:00:01.184424 20582 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:48526 (request call id 319) took 1205 ms. Trace:
I20260129 03:00:00.977576 21130 rpcz_store.cc:276] 0129 02:59:59.439008 (+ 0us) service_pool.cc:168] Inserting onto call queue
0129 02:59:59.441066 (+ 2058us) service_pool.cc:225] Handling call
0129 02:59:59.484787 (+ 43721us) raft_consensus.cc:1471] Updating replica for 1 ops
0129 02:59:59.733285 (+248498us) raft_consensus.cc:1534] Early marking committed up to index 0
0129 02:59:59.739707 (+ 6422us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0129 02:59:59.790439 (+ 50732us) log.cc:844] Serialized 26 byte log entry
0129 03:00:00.148108 (+357669us) raft_consensus.cc:1645] Marking committed up to 6
0129 03:00:00.150043 (+ 1935us) raft_consensus.cc:1697] Filling consensus response to leader.
0129 03:00:00.258831 (+108788us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0129 03:00:00.742911 (+484080us) raft_consensus.cc:1685] finished
0129 03:00:00.742940 (+ 29us) raft_consensus.cc:1691] UpdateReplicas() finished
0129 03:00:00.743095 (+ 155us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":1414,"raft.queue_time_us":126327,"raft.run_cpu_time_us":168,"raft.run_wall_time_us":20914,"thread_start_us":397341,"threads_started":2,"wal-append.queue_time_us":816843}
I20260129 03:00:01.186168 20515 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e33a417cf9c54cf5af73a4ee3aac4d3a" candidate_uuid: "197b880fa39d432a9b472845876bfaf0" candidate_term: 7 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "974b802141bd4f13beffef2aa03f88f9" is_pre_election: true
W20260129 03:00:01.185374 20407 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.211.2:34517, user_credentials={real_user=slave}} blocked reactor thread for 80983.8us
I20260129 03:00:01.196127 23146 raft_consensus.cc:493] T febffe4a8b394368b69b1d164308fb9a P 974b802141bd4f13beffef2aa03f88f9 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 197b880fa39d432a9b472845876bfaf0)
I20260129 03:00:01.201500 22881 leader_election.cc:290] T a5a05fbcbbcf4916bbc01fd1a5b5e90d P 1c50de89a5a845668696dcb28c3badd6 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 197b880fa39d432a9b472845876bfaf0 (127.18.211.1:42027), 974b802141bd4f13beffef2aa03f88f9 (127.18.211.2:34517)
I20260129 03:00:01.152273 23153 raft_consensus.cc:515] T c2633a6fc34c44579de348d208b0c01e P 974b802141bd4f13beffef2aa03f88f9 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 03:00:01.086720 22975 consensus_queue.cc:260] T 4b1876f9a70b49edb3aa74105e66d3b3 P 974b802141bd4f13beffef2aa03f88f9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 4.4, Last appended by leader: 4, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 03:00:01.157021 20512 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "75be4bd6bd45474795fd472eff2953ac" candidate_uuid: "197b880fa39d432a9b472845876bfaf0" candidate_term: 10 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "974b802141bd4f13beffef2aa03f88f9" is_pre_election: true
I20260129 03:00:01.333009 20512 raft_consensus.cc:2468] T 75be4bd6bd45474795fd472eff2953ac P 974b802141bd4f13beffef2aa03f88f9 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 197b880fa39d432a9b472845876bfaf0 in term 9.
I20260129 03:00:01.334326 20512 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "45cc5f3d2688486d9b6dbe7e666dedde" candidate_uuid: "7a9daf5d42a84615a6383cfec9a10608" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "974b802141bd4f13beffef2aa03f88f9" is_pre_election: true
I20260129 03:00:01.335310 20512 raft_consensus.cc:2393] T 45cc5f3d2688486d9b6dbe7e666dedde P 974b802141bd4f13beffef2aa03f88f9 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 7a9daf5d42a84615a6383cfec9a10608 in current term 5: Already voted for candidate 974b802141bd4f13beffef2aa03f88f9 in this term.
I20260129 03:00:01.160043 23078 raft_consensus.cc:2749] T 75b0b6e82ddc4f92b1fe1d94f9a0b73b P 974b802141bd4f13beffef2aa03f88f9 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
W20260129 03:00:01.208179 23156 log.cc:927] Time spent T c934d31efd3f4e5195605ddc354d14e1 P 7a9daf5d42a84615a6383cfec9a10608: Append to log took a long time: real 0.076s user 0.000s sys 0.001s
I20260129 03:00:01.211691 20584 raft_consensus.cc:2393] T 926a250fa5834e88b96ccecbefbf5086 P 1c50de89a5a845668696dcb28c3badd6 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 197b880fa39d432a9b472845876bfaf0 in current term 7: Already voted for candidate 1c50de89a5a845668696dcb28c3badd6 in this term.
I20260129 03:00:01.129385 23165 raft_consensus.cc:493] T 75b0b6e82ddc4f92b1fe1d94f9a0b73b P 7a9daf5d42a84615a6383cfec9a10608 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 1c50de89a5a845668696dcb28c3badd6)
I20260129 03:00:01.220745 20513 rpcz_store.cc:276] 0129 03:00:00.133437 (+ 0us) service_pool.cc:168] Inserting onto call queue
0129 03:00:00.133609 (+ 172us) service_pool.cc:225] Handling call
0129 03:00:00.155218 (+ 21609us) raft_consensus.cc:1471] Updating replica for 2 ops
0129 03:00:00.557332 (+402114us) raft_consensus.cc:1534] Early marking committed up to index 0
0129 03:00:00.557383 (+ 51us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0129 03:00:00.574155 (+ 16772us) log.cc:844] Serialized 52 byte log entry
0129 03:00:00.983496 (+409341us) raft_consensus.cc:1645] Marking committed up to 3
0129 03:00:01.146920 (+163424us) log.cc:844] Serialized 14 byte log entry
0129 03:00:01.147164 (+ 244us) raft_consensus.cc:1697] Filling consensus response to leader.
0129 03:00:01.147244 (+ 80us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0129 03:00:01.220095 (+ 72851us) raft_consensus.cc:1685] finished
0129 03:00:01.220118 (+ 23us) raft_consensus.cc:1691] UpdateReplicas() finished
0129 03:00:01.220260 (+ 142us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":21656,"prepare.queue_time_us":34875,"prepare.run_cpu_time_us":32,"prepare.run_wall_time_us":33,"raft.queue_time_us":461417,"raft.run_cpu_time_us":211,"raft.run_wall_time_us":5817,"thread_start_us":768139,"threads_started":3,"wal-append.queue_time_us":484415}
I20260129 03:00:01.229676 20589 raft_consensus.cc:2468] T 8055232cf0004d40a85f3bf983fe00a9 P 1c50de89a5a845668696dcb28c3badd6 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 197b880fa39d432a9b472845876bfaf0 in term 4.
I20260129 03:00:01.252265 23096 leader_election.cc:290] T 76f713c3331540d2a5b76164103e9eb1 P 197b880fa39d432a9b472845876bfaf0 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 974b802141bd4f13beffef2aa03f88f9 (127.18.211.2:34517), 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3:37735)
I20260129 03:00:01.252513 23133 leader_election.cc:290] T 17d6431cc77a491e8464d68db4081a51 P 197b880fa39d432a9b472845876bfaf0 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 974b802141bd4f13beffef2aa03f88f9 (127.18.211.2:34517), 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3:37735)
I20260129 03:00:01.253690 21128 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "45cc5f3d2688486d9b6dbe7e666dedde" candidate_uuid: "974b802141bd4f13beffef2aa03f88f9" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "7a9daf5d42a84615a6383cfec9a10608"
I20260129 03:00:01.253959 23128 leader_election.cc:290] T cedce331e1584ac8b636c9b8a72b2ba0 P 1c50de89a5a845668696dcb28c3badd6 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 197b880fa39d432a9b472845876bfaf0 (127.18.211.1:42027), 974b802141bd4f13beffef2aa03f88f9 (127.18.211.2:34517)
I20260129 03:00:01.196434 22980 raft_consensus.cc:3055] T cb4727386a3e48aca5d71cc6c611c197 P 197b880fa39d432a9b472845876bfaf0 [term 3 LEADER]: Stepping down as leader of term 3
I20260129 03:00:01.254537 23041 raft_consensus.cc:2804] T a609ac02fae84725ab743f95f51d33ce P 7a9daf5d42a84615a6383cfec9a10608 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260129 03:00:01.210438 21132 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "78aa39096868415d9afc2ffcb248e198" candidate_uuid: "197b880fa39d432a9b472845876bfaf0" candidate_term: 6 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "7a9daf5d42a84615a6383cfec9a10608" is_pre_election: true
I20260129 03:00:01.419096 20589 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "724a236524cb4e7ba3428880681d6cc2" candidate_uuid: "197b880fa39d432a9b472845876bfaf0" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "1c50de89a5a845668696dcb28c3badd6" is_pre_election: true
I20260129 03:00:01.419927 20589 raft_consensus.cc:2410] T 724a236524cb4e7ba3428880681d6cc2 P 1c50de89a5a845668696dcb28c3badd6 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 197b880fa39d432a9b472845876bfaf0 for term 5 because replica has last-logged OpId of term: 4 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 2.
I20260129 03:00:01.257179 20586 raft_consensus.cc:2468] T c2633a6fc34c44579de348d208b0c01e P 1c50de89a5a845668696dcb28c3badd6 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 197b880fa39d432a9b472845876bfaf0 in term 3.
I20260129 03:00:01.258347 20480 leader_election.cc:304] T c934d31efd3f4e5195605ddc354d14e1 P 974b802141bd4f13beffef2aa03f88f9 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 974b802141bd4f13beffef2aa03f88f9; no voters: 197b880fa39d432a9b472845876bfaf0, 7a9daf5d42a84615a6383cfec9a10608
I20260129 03:00:01.304958 23168 raft_consensus.cc:493] T 4e3805ffa70242e8ade66da260dddf66 P 7a9daf5d42a84615a6383cfec9a10608 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 1c50de89a5a845668696dcb28c3badd6)
W20260129 03:00:01.452813 20480 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.211.4:40439, user_credentials={real_user=slave}} blocked reactor thread for 194565us
W20260129 03:00:01.363288 23162 log.cc:927] Time spent T 7a9a69cd82724b1faed1d10aa091fc4a P 974b802141bd4f13beffef2aa03f88f9: Append to log took a long time: real 0.143s user 0.000s sys 0.001s
I20260129 03:00:01.275025 20587 raft_consensus.cc:1275] T cedce331e1584ac8b636c9b8a72b2ba0 P 1c50de89a5a845668696dcb28c3badd6 [term 6 FOLLOWER]: Refusing update from remote peer 197b880fa39d432a9b472845876bfaf0: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 6 index: 4. (index mismatch)
I20260129 03:00:01.331012 20515 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "add8d4923d1a489e86882c3a25692b95" candidate_uuid: "7a9daf5d42a84615a6383cfec9a10608" candidate_term: 6 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "974b802141bd4f13beffef2aa03f88f9" is_pre_election: true
I20260129 03:00:01.331068 21127 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3e2d15f8640b407195210c476e09df91" candidate_uuid: "974b802141bd4f13beffef2aa03f88f9" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "7a9daf5d42a84615a6383cfec9a10608" is_pre_election: true
I20260129 03:00:01.339149 23153 leader_election.cc:290] T c2633a6fc34c44579de348d208b0c01e P 974b802141bd4f13beffef2aa03f88f9 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 197b880fa39d432a9b472845876bfaf0 (127.18.211.1:42027), 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3:37735)
I20260129 03:00:01.287834 20582 rpcz_store.cc:276] 0129 02:59:59.979197 (+ 0us) service_pool.cc:168] Inserting onto call queue
0129 02:59:59.979339 (+ 142us) service_pool.cc:225] Handling call
0129 02:59:59.979915 (+ 576us) raft_consensus.cc:1471] Updating replica for 2 ops
0129 02:59:59.988584 (+ 8669us) raft_consensus.cc:1534] Early marking committed up to index 0
0129 02:59:59.988619 (+ 35us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0129 02:59:59.989727 (+ 1108us) log.cc:844] Serialized 582 byte log entry
0129 03:00:00.047032 (+ 57305us) raft_consensus.cc:1645] Marking committed up to 3
0129 03:00:00.047132 (+ 100us) raft_consensus.cc:1697] Filling consensus response to leader.
0129 03:00:00.047341 (+ 209us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0129 03:00:01.184190 (+1136849us) raft_consensus.cc:1685] finished
0129 03:00:01.184213 (+ 23us) raft_consensus.cc:1691] UpdateReplicas() finished
0129 03:00:01.184364 (+ 151us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":188,"raft.queue_time_us":9867,"raft.run_cpu_time_us":103,"raft.run_wall_time_us":105,"thread_start_us":56493,"threads_started":1,"wal-append.queue_time_us":330302}
I20260129 03:00:01.206465 23163 raft_consensus.cc:740] T fd8311b779af4e1fa1402b2a02a602f2 P 7a9daf5d42a84615a6383cfec9a10608 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 7a9daf5d42a84615a6383cfec9a10608, State: Running, Role: LEADER
I20260129 03:00:01.162662 23167 consensus_queue.cc:1059] T 00e79d8f25a34676b1d866eb2068a2b2 P 7a9daf5d42a84615a6383cfec9a10608 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 }, Status: INVALID_TERM, Last received: 6.8, Next index: 9, Last known committed idx: 6, Time since last communication: 0.131s
I20260129 03:00:01.131008 23055 consensus_queue.cc:260] T 75f45538dc704d9a8ec4ff8ad9f3ea5b P 974b802141bd4f13beffef2aa03f88f9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.6, Last appended by leader: 6, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } }
I20260129 03:00:01.386871 21134 raft_consensus.cc:2468] T 0678b8b468bd43cdbc3e485900844821 P 7a9daf5d42a84615a6383cfec9a10608 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 974b802141bd4f13beffef2aa03f88f9 in term 7.
I20260129 03:00:01.413482 20581 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4408006eb5174de1896b037cf3f18457" candidate_uuid: "197b880fa39d432a9b472845876bfaf0" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "1c50de89a5a845668696dcb28c3badd6" is_pre_election: true
I20260129 03:00:01.416850 20512 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "cb0b3f259ebc4310927338a8740e87b8" candidate_uuid: "7a9daf5d42a84615a6383cfec9a10608" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "974b802141bd4f13beffef2aa03f88f9" is_pre_election: true
I20260129 03:00:01.418455 20584 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4408006eb5174de1896b037cf3f18457" candidate_uuid: "197b880fa39d432a9b472845876bfaf0" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "1c50de89a5a845668696dcb28c3badd6"
W20260129 03:00:01.424804 20555 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260129 03:00:01.451658 21132 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "add8d4923d1a489e86882c3a25692b95" candidate_uuid: "974b802141bd4f13beffef2aa03f88f9" candidate_term: 6 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "7a9daf5d42a84615a6383cfec9a10608" is_pre_election: true
W20260129 03:00:01.456813 20406 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260129 03:00:01.455332 20515 raft_consensus.cc:2468] T add8d4923d1a489e86882c3a25692b95 P 974b802141bd4f13beffef2aa03f88f9 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7a9daf5d42a84615a6383cfec9a10608 in term 5.
I20260129 03:00:01.460433 20582 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fcddab6bca684adf8ea032289a00f90e" candidate_uuid: "197b880fa39d432a9b472845876bfaf0" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "1c50de89a5a845668696dcb28c3badd6"
I20260129 03:00:01.461155 21102 leader_election.cc:304] T 3e2d15f8640b407195210c476e09df91 P 7a9daf5d42a84615a6383cfec9a10608 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 7a9daf5d42a84615a6383cfec9a10608, 974b802141bd4f13beffef2aa03f88f9; no voters: 1c50de89a5a845668696dcb28c3badd6
W20260129 03:00:01.416065 23065 log.cc:927] Time spent T 4e420f90884f4796997b5ea41e49e411 P 974b802141bd4f13beffef2aa03f88f9: Append to log took a long time: real 3.160s user 0.018s sys 0.033s
W20260129 03:00:01.537451 20480 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260129 03:00:01.537603 20589 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "add8d4923d1a489e86882c3a25692b95" candidate_uuid: "7a9daf5d42a84615a6383cfec9a10608" candidate_term: 6 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "1c50de89a5a845668696dcb28c3badd6" is_pre_election: true
I20260129 03:00:01.445981 22980 raft_consensus.cc:740] T cb4727386a3e48aca5d71cc6c611c197 P 197b880fa39d432a9b472845876bfaf0 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 197b880fa39d432a9b472845876bfaf0, State: Running, Role: LEADER
I20260129 03:00:01.449611 23118 raft_consensus.cc:2749] T 96609ac68bdf4993a12e1c23355ca013 P 197b880fa39d432a9b472845876bfaf0 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
W20260129 03:00:01.447315 23158 log.cc:927] Time spent T add8d4923d1a489e86882c3a25692b95 P 1c50de89a5a845668696dcb28c3badd6: Append to log took a long time: real 0.238s user 0.001s sys 0.000s
W20260129 03:00:01.506671 21101 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260129 03:00:01.446244 23041 raft_consensus.cc:493] T a609ac02fae84725ab743f95f51d33ce P 7a9daf5d42a84615a6383cfec9a10608 [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260129 03:00:01.540779 20582 raft_consensus.cc:2393] T fcddab6bca684adf8ea032289a00f90e P 1c50de89a5a845668696dcb28c3badd6 [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate 197b880fa39d432a9b472845876bfaf0 in current term 9: Already voted for candidate 1c50de89a5a845668696dcb28c3badd6 in this term.
I20260129 03:00:01.542485 20512 raft_consensus.cc:2468] T cb0b3f259ebc4310927338a8740e87b8 P 974b802141bd4f13beffef2aa03f88f9 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7a9daf5d42a84615a6383cfec9a10608 in term 5.
W20260129 03:00:01.473065 23119 consensus_meta.cc:287] T cb0b3f259ebc4310927338a8740e87b8 P 7a9daf5d42a84615a6383cfec9a10608: Time spent flushing consensus metadata: real 1.078s user 0.019s sys 0.007s
I20260129 03:00:01.466926 20513 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "505580c93c694bbb8ea96fdf9b17ab46" candidate_uuid: "7a9daf5d42a84615a6383cfec9a10608" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "974b802141bd4f13beffef2aa03f88f9"
W20260129 03:00:01.467175 20482 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260129 03:00:01.547596 20589 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4b1876f9a70b49edb3aa74105e66d3b3" candidate_uuid: "197b880fa39d432a9b472845876bfaf0" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "1c50de89a5a845668696dcb28c3badd6"
I20260129 03:00:01.554682 20513 raft_consensus.cc:3060] T 505580c93c694bbb8ea96fdf9b17ab46 P 974b802141bd4f13beffef2aa03f88f9 [term 5 FOLLOWER]: Advancing to term 6
I20260129 03:00:01.554945 20589 raft_consensus.cc:3060] T 4b1876f9a70b49edb3aa74105e66d3b3 P 1c50de89a5a845668696dcb28c3badd6 [term 4 FOLLOWER]: Advancing to term 5
W20260129 03:00:01.556221 20482 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260129 03:00:01.559399 21135 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34060 (request call id 515) took 1486 ms. Trace:
I20260129 03:00:01.538758 20581 raft_consensus.cc:2468] T 4408006eb5174de1896b037cf3f18457 P 1c50de89a5a845668696dcb28c3badd6 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 197b880fa39d432a9b472845876bfaf0 in term 3.
I20260129 03:00:01.539043 21132 raft_consensus.cc:2468] T add8d4923d1a489e86882c3a25692b95 P 7a9daf5d42a84615a6383cfec9a10608 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 974b802141bd4f13beffef2aa03f88f9 in term 5.
I20260129 03:00:01.548669 21134 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "505580c93c694bbb8ea96fdf9b17ab46" candidate_uuid: "974b802141bd4f13beffef2aa03f88f9" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "7a9daf5d42a84615a6383cfec9a10608" is_pre_election: true
I20260129 03:00:01.556267 20512 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a609ac02fae84725ab743f95f51d33ce" candidate_uuid: "7a9daf5d42a84615a6383cfec9a10608" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "974b802141bd4f13beffef2aa03f88f9" is_pre_election: true
I20260129 03:00:01.346019 23146 raft_consensus.cc:515] T febffe4a8b394368b69b1d164308fb9a P 974b802141bd4f13beffef2aa03f88f9 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } }
I20260129 03:00:01.460187 22978 leader_election.cc:290] T 505580c93c694bbb8ea96fdf9b17ab46 P 974b802141bd4f13beffef2aa03f88f9 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3:37735), 7a9daf5d42a84615a6383cfec9a10608 (127.18.211.4:40439)
I20260129 03:00:01.562152 21134 raft_consensus.cc:2393] T 505580c93c694bbb8ea96fdf9b17ab46 P 7a9daf5d42a84615a6383cfec9a10608 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 974b802141bd4f13beffef2aa03f88f9 in current term 6: Already voted for candidate 7a9daf5d42a84615a6383cfec9a10608 in this term.
I20260129 03:00:01.563436 20515 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4e420f90884f4796997b5ea41e49e411" candidate_uuid: "7a9daf5d42a84615a6383cfec9a10608" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "974b802141bd4f13beffef2aa03f88f9"
I20260129 03:00:01.564213 20582 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "368953c9102643b2a21f99af36b51905" candidate_uuid: "197b880fa39d432a9b472845876bfaf0" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "1c50de89a5a845668696dcb28c3badd6" is_pre_election: true
I20260129 03:00:01.562129 22975 raft_consensus.cc:3060] T 4b1876f9a70b49edb3aa74105e66d3b3 P 974b802141bd4f13beffef2aa03f88f9 [term 4 FOLLOWER]: Advancing to term 5
I20260129 03:00:01.469233 22888 raft_consensus.cc:2749] T c934d31efd3f4e5195605ddc354d14e1 P 974b802141bd4f13beffef2aa03f88f9 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260129 03:00:01.462420 23042 raft_consensus.cc:493] T 2ac659d732314e1288849f08fe5e074e P 197b880fa39d432a9b472845876bfaf0 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 974b802141bd4f13beffef2aa03f88f9)
I20260129 03:00:01.567268 20584 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a0fcc58d6fb6473cafbd9d34d5195269" candidate_uuid: "197b880fa39d432a9b472845876bfaf0" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "1c50de89a5a845668696dcb28c3badd6"
I20260129 03:00:01.545270 21130 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00e79d8f25a34676b1d866eb2068a2b2" candidate_uuid: "974b802141bd4f13beffef2aa03f88f9" candidate_term: 7 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "7a9daf5d42a84615a6383cfec9a10608"
I20260129 03:00:01.569258 20584 raft_consensus.cc:3060] T a0fcc58d6fb6473cafbd9d34d5195269 P 1c50de89a5a845668696dcb28c3badd6 [term 3 FOLLOWER]: Advancing to term 4
W20260129 03:00:01.557130 23166 log.cc:927] Time spent T a8479d1fe0bd41e5946186e3a6ca8c7a P 974b802141bd4f13beffef2aa03f88f9: Append to log took a long time: real 0.143s user 0.000s sys 0.001s
I20260129 03:00:01.424448 23165 raft_consensus.cc:515] T 75b0b6e82ddc4f92b1fe1d94f9a0b73b P 7a9daf5d42a84615a6383cfec9a10608 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } }
I20260129 03:00:01.570794 20582 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0f20af7fbe5d4db89a6f17948d98912c" candidate_uuid: "197b880fa39d432a9b472845876bfaf0" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "1c50de89a5a845668696dcb28c3badd6"
I20260129 03:00:01.577049 20582 raft_consensus.cc:3060] T 0f20af7fbe5d4db89a6f17948d98912c P 1c50de89a5a845668696dcb28c3badd6 [term 6 FOLLOWER]: Advancing to term 7
I20260129 03:00:01.553634 23041 raft_consensus.cc:3060] T a609ac02fae84725ab743f95f51d33ce P 7a9daf5d42a84615a6383cfec9a10608 [term 7 FOLLOWER]: Advancing to term 8
I20260129 03:00:01.571825 20515 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "febffe4a8b394368b69b1d164308fb9a" candidate_uuid: "7a9daf5d42a84615a6383cfec9a10608" candidate_term: 5 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "974b802141bd4f13beffef2aa03f88f9" is_pre_election: true
I20260129 03:00:01.572572 20581 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0f20af7fbe5d4db89a6f17948d98912c" candidate_uuid: "197b880fa39d432a9b472845876bfaf0" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "1c50de89a5a845668696dcb28c3badd6" is_pre_election: true
I20260129 03:00:01.550820 23044 leader_election.cc:290] T cb4727386a3e48aca5d71cc6c611c197 P 1c50de89a5a845668696dcb28c3badd6 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 197b880fa39d432a9b472845876bfaf0 (127.18.211.1:42027), 974b802141bd4f13beffef2aa03f88f9 (127.18.211.2:34517)
I20260129 03:00:01.561170 21135 rpcz_store.cc:276] 0129 03:00:00.072705 (+ 0us) service_pool.cc:168] Inserting onto call queue
0129 03:00:00.073727 (+ 1022us) service_pool.cc:225] Handling call
0129 03:00:00.119690 (+ 45963us) raft_consensus.cc:1471] Updating replica for 1 ops
0129 03:00:00.150692 (+ 31002us) raft_consensus.cc:1534] Early marking committed up to index 0
0129 03:00:00.150720 (+ 28us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0129 03:00:00.151237 (+ 517us) log.cc:844] Serialized 26 byte log entry
0129 03:00:00.949250 (+798013us) raft_consensus.cc:1645] Marking committed up to 7
0129 03:00:01.147908 (+198658us) log.cc:844] Serialized 14 byte log entry
0129 03:00:01.154675 (+ 6767us) raft_consensus.cc:1697] Filling consensus response to leader.
0129 03:00:01.154748 (+ 73us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0129 03:00:01.559180 (+404432us) raft_consensus.cc:1685] finished
0129 03:00:01.559219 (+ 39us) raft_consensus.cc:1691] UpdateReplicas() finished
0129 03:00:01.559338 (+ 119us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":21720,"prepare.queue_time_us":588527,"raft.queue_time_us":38810,"raft.run_cpu_time_us":332,"raft.run_wall_time_us":123112,"thread_start_us":860604,"threads_started":2,"wal-append.queue_time_us":807604}
I20260129 03:00:01.571749 20512 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "96609ac68bdf4993a12e1c23355ca013" candidate_uuid: "7a9daf5d42a84615a6383cfec9a10608" candidate_term: 6 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "974b802141bd4f13beffef2aa03f88f9"
W20260129 03:00:01.583920 20406 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260129 03:00:01.583740 20512 raft_consensus.cc:2410] T 96609ac68bdf4993a12e1c23355ca013 P 974b802141bd4f13beffef2aa03f88f9 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 7a9daf5d42a84615a6383cfec9a10608 for term 6 because replica has last-logged OpId of term: 3 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 4.
I20260129 03:00:01.579278 20515 raft_consensus.cc:2410] T febffe4a8b394368b69b1d164308fb9a P 974b802141bd4f13beffef2aa03f88f9 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 7a9daf5d42a84615a6383cfec9a10608 for term 5 because replica has last-logged OpId of term: 3 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 4.
I20260129 03:00:01.562350 22656 leader_election.cc:290] T a8479d1fe0bd41e5946186e3a6ca8c7a P 1c50de89a5a845668696dcb28c3badd6 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 197b880fa39d432a9b472845876bfaf0 (127.18.211.1:42027), 974b802141bd4f13beffef2aa03f88f9 (127.18.211.2:34517)
I20260129 03:00:01.592973 20589 raft_consensus.cc:2468] T 4b1876f9a70b49edb3aa74105e66d3b3 P 1c50de89a5a845668696dcb28c3badd6 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 197b880fa39d432a9b472845876bfaf0 in term 5.
I20260129 03:00:01.596675 20512 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "87d5fe48a44a408aaf03e894a67ecb2f" candidate_uuid: "7a9daf5d42a84615a6383cfec9a10608" candidate_term: 7 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "974b802141bd4f13beffef2aa03f88f9" is_pre_election: true
I20260129 03:00:01.565688 23154 leader_election.cc:290] T 724a236524cb4e7ba3428880681d6cc2 P 974b802141bd4f13beffef2aa03f88f9 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 197b880fa39d432a9b472845876bfaf0 (127.18.211.1:42027), 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3:37735)
W20260129 03:00:01.612354 20406 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260129 03:00:01.547017 22989 consensus_queue.cc:1059] T 75f45538dc704d9a8ec4ff8ad9f3ea5b P 974b802141bd4f13beffef2aa03f88f9 [NON_LEADER]: Peer responded invalid term: Peer: permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false }, Status: INVALID_TERM, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.029s
I20260129 03:00:01.568274 23118 raft_consensus.cc:2804] T 0678b8b468bd43cdbc3e485900844821 P 197b880fa39d432a9b472845876bfaf0 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260129 03:00:01.557013 23107 consensus_queue.cc:1059] T 00e79d8f25a34676b1d866eb2068a2b2 P 7a9daf5d42a84615a6383cfec9a10608 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 }, Status: INVALID_TERM, Last received: 4.7, Next index: 8, Last known committed idx: 6, Time since last communication: 0.025s
I20260129 03:00:01.625779 20512 raft_consensus.cc:1240] T a644c02b16af47679824ccd060ac10fd P 974b802141bd4f13beffef2aa03f88f9 [term 5 FOLLOWER]: Rejecting Update request from peer 7a9daf5d42a84615a6383cfec9a10608 for earlier term 4. Current term is 5. Ops: [3.5-4.6]
I20260129 03:00:01.622524 20582 raft_consensus.cc:2468] T 0f20af7fbe5d4db89a6f17948d98912c P 1c50de89a5a845668696dcb28c3badd6 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 197b880fa39d432a9b472845876bfaf0 in term 7.
I20260129 03:00:01.470192 23168 raft_consensus.cc:515] T 4e3805ffa70242e8ade66da260dddf66 P 7a9daf5d42a84615a6383cfec9a10608 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } }
W20260129 03:00:01.630118 20482 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260129 03:00:01.576087 23175 raft_consensus.cc:2804] T 3e2d15f8640b407195210c476e09df91 P 7a9daf5d42a84615a6383cfec9a10608 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260129 03:00:01.637842 20515 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c934d31efd3f4e5195605ddc354d14e1" candidate_uuid: "7a9daf5d42a84615a6383cfec9a10608" candidate_term: 6 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "974b802141bd4f13beffef2aa03f88f9" is_pre_election: true
I20260129 03:00:01.563892 23119 raft_consensus.cc:515] T cb0b3f259ebc4310927338a8740e87b8 P 7a9daf5d42a84615a6383cfec9a10608 [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } }
I20260129 03:00:01.647286 20515 raft_consensus.cc:2410] T c934d31efd3f4e5195605ddc354d14e1 P 974b802141bd4f13beffef2aa03f88f9 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 7a9daf5d42a84615a6383cfec9a10608 for term 6 because replica has last-logged OpId of term: 5 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 6.
I20260129 03:00:01.611799 22924 leader_election.cc:290] T e4d18c35b0a7493e85d0a4b92956a48f P 1c50de89a5a845668696dcb28c3badd6 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 197b880fa39d432a9b472845876bfaf0 (127.18.211.1:42027), 974b802141bd4f13beffef2aa03f88f9 (127.18.211.2:34517)
I20260129 03:00:01.636631 20581 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "724a236524cb4e7ba3428880681d6cc2" candidate_uuid: "197b880fa39d432a9b472845876bfaf0" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "1c50de89a5a845668696dcb28c3badd6"
I20260129 03:00:01.655082 20581 raft_consensus.cc:2410] T 724a236524cb4e7ba3428880681d6cc2 P 1c50de89a5a845668696dcb28c3badd6 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 197b880fa39d432a9b472845876bfaf0 for term 5 because replica has last-logged OpId of term: 4 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 2.
I20260129 03:00:01.576804 23042 raft_consensus.cc:515] T 2ac659d732314e1288849f08fe5e074e P 197b880fa39d432a9b472845876bfaf0 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } }
I20260129 03:00:01.623485 20589 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "dc2f4cb752cf4d59b5f6bc689f5d8104" candidate_uuid: "197b880fa39d432a9b472845876bfaf0" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "1c50de89a5a845668696dcb28c3badd6" is_pre_election: true
W20260129 03:00:01.639920 23156 log.cc:927] Time spent T c934d31efd3f4e5195605ddc354d14e1 P 7a9daf5d42a84615a6383cfec9a10608: Append to log took a long time: real 0.053s user 0.000s sys 0.000s
I20260129 03:00:01.626621 23118 raft_consensus.cc:493] T 0678b8b468bd43cdbc3e485900844821 P 197b880fa39d432a9b472845876bfaf0 [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260129 03:00:01.630653 22989 consensus_queue.cc:1243] T 75f45538dc704d9a8ec4ff8ad9f3ea5b P 974b802141bd4f13beffef2aa03f88f9 [NON_LEADER]: Peer 7a9daf5d42a84615a6383cfec9a10608 log is divergent from this leader: its last log entry 4.6 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 5
W20260129 03:00:01.633718 23174 log.cc:927] Time spent T 75b0b6e82ddc4f92b1fe1d94f9a0b73b P 974b802141bd4f13beffef2aa03f88f9: Append to log took a long time: real 0.171s user 0.000s sys 0.001s
W20260129 03:00:01.661055 20407 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.211.2:34517, user_credentials={real_user=slave}} blocked reactor thread for 371358us
I20260129 03:00:01.630072 22728 leader_election.cc:290] T 1ce4ddf5b10e4ae8b193b13b2e5936ee P 1c50de89a5a845668696dcb28c3badd6 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 197b880fa39d432a9b472845876bfaf0 (127.18.211.1:42027), 974b802141bd4f13beffef2aa03f88f9 (127.18.211.2:34517)
I20260129 03:00:01.637305 23175 raft_consensus.cc:493] T 3e2d15f8640b407195210c476e09df91 P 7a9daf5d42a84615a6383cfec9a10608 [term 3 FOLLOWER]: Starting leader election (detected failure of leader 1c50de89a5a845668696dcb28c3badd6)
W20260129 03:00:01.657297 21100 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.211.3:37735, user_credentials={real_user=slave}} blocked reactor thread for 110176us
I20260129 03:00:01.650054 20584 raft_consensus.cc:2468] T a0fcc58d6fb6473cafbd9d34d5195269 P 1c50de89a5a845668696dcb28c3badd6 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 197b880fa39d432a9b472845876bfaf0 in term 4.
I20260129 03:00:01.578807 22980 consensus_queue.cc:260] T cb4727386a3e48aca5d71cc6c611c197 P 197b880fa39d432a9b472845876bfaf0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 3.2, Last appended by leader: 2, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 03:00:01.816819 21134 raft_consensus.cc:1275] T add8d4923d1a489e86882c3a25692b95 P 7a9daf5d42a84615a6383cfec9a10608 [term 5 FOLLOWER]: Refusing update from remote peer 1c50de89a5a845668696dcb28c3badd6: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 3 index: 6. (index mismatch)
W20260129 03:00:01.687829 21102 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.211.2:34517, user_credentials={real_user=slave}} blocked reactor thread for 226761us
I20260129 03:00:01.696089 23151 raft_consensus.cc:493] T 0340229d521b432580ae926f6062e3f7 P 974b802141bd4f13beffef2aa03f88f9 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260129 03:00:01.838201 20405 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.211.4:40439, user_credentials={real_user=slave}} blocked reactor thread for 708794us
I20260129 03:00:01.671363 20480 leader_election.cc:304] T add8d4923d1a489e86882c3a25692b95 P 974b802141bd4f13beffef2aa03f88f9 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7a9daf5d42a84615a6383cfec9a10608, 974b802141bd4f13beffef2aa03f88f9; no voters:
I20260129 03:00:01.820762 20407 leader_election.cc:304] T 4e420f90884f4796997b5ea41e49e411 P 197b880fa39d432a9b472845876bfaf0 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 197b880fa39d432a9b472845876bfaf0; no voters: 7a9daf5d42a84615a6383cfec9a10608, 974b802141bd4f13beffef2aa03f88f9
I20260129 03:00:01.821012 23165 leader_election.cc:290] T 75b0b6e82ddc4f92b1fe1d94f9a0b73b P 7a9daf5d42a84615a6383cfec9a10608 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 974b802141bd4f13beffef2aa03f88f9 (127.18.211.2:34517), 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3:37735)
W20260129 03:00:01.841048 20480 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.211.4:40439, user_credentials={real_user=slave}} blocked reactor thread for 169794us
I20260129 03:00:01.841306 22978 raft_consensus.cc:2804] T add8d4923d1a489e86882c3a25692b95 P 974b802141bd4f13beffef2aa03f88f9 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260129 03:00:01.841454 21134 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1299723a4476420ab02fd8bfd7aa0645" candidate_uuid: "1c50de89a5a845668696dcb28c3badd6" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "7a9daf5d42a84615a6383cfec9a10608"
I20260129 03:00:01.841910 22978 raft_consensus.cc:493] T add8d4923d1a489e86882c3a25692b95 P 974b802141bd4f13beffef2aa03f88f9 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260129 03:00:01.842411 22978 raft_consensus.cc:3060] T add8d4923d1a489e86882c3a25692b95 P 974b802141bd4f13beffef2aa03f88f9 [term 5 FOLLOWER]: Advancing to term 6
I20260129 03:00:01.842950 21134 raft_consensus.cc:1240] T 505580c93c694bbb8ea96fdf9b17ab46 P 7a9daf5d42a84615a6383cfec9a10608 [term 6 FOLLOWER]: Rejecting Update request from peer 1c50de89a5a845668696dcb28c3badd6 for earlier term 5. Current term is 6. Ops: []
I20260129 03:00:01.843303 20480 leader_election.cc:304] T 3e2d15f8640b407195210c476e09df91 P 974b802141bd4f13beffef2aa03f88f9 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 974b802141bd4f13beffef2aa03f88f9; no voters: 1c50de89a5a845668696dcb28c3badd6, 7a9daf5d42a84615a6383cfec9a10608
I20260129 03:00:01.844067 22888 raft_consensus.cc:2749] T 3e2d15f8640b407195210c476e09df91 P 974b802141bd4f13beffef2aa03f88f9 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260129 03:00:01.844470 20480 leader_election.cc:304] T 00e79d8f25a34676b1d866eb2068a2b2 P 974b802141bd4f13beffef2aa03f88f9 [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 974b802141bd4f13beffef2aa03f88f9; no voters: 197b880fa39d432a9b472845876bfaf0, 7a9daf5d42a84615a6383cfec9a10608
I20260129 03:00:01.845243 22888 raft_consensus.cc:2749] T 00e79d8f25a34676b1d866eb2068a2b2 P 974b802141bd4f13beffef2aa03f88f9 [term 7 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
I20260129 03:00:01.846223 21134 raft_consensus.cc:1240] T 42d27a28d89b409e8807ec0efd97af69 P 7a9daf5d42a84615a6383cfec9a10608 [term 7 FOLLOWER]: Rejecting Update request from peer 1c50de89a5a845668696dcb28c3badd6 for earlier term 6. Current term is 7. Ops: []
I20260129 03:00:01.821309 23118 raft_consensus.cc:3060] T 0678b8b468bd43cdbc3e485900844821 P 197b880fa39d432a9b472845876bfaf0 [term 7 FOLLOWER]: Advancing to term 8
I20260129 03:00:01.847088 21134 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3ccac82a01a2445fa5b0ce6c0fcf2295" candidate_uuid: "974b802141bd4f13beffef2aa03f88f9" candidate_term: 5 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "7a9daf5d42a84615a6383cfec9a10608" is_pre_election: true
I20260129 03:00:01.821673 23175 raft_consensus.cc:3060] T 3e2d15f8640b407195210c476e09df91 P 7a9daf5d42a84615a6383cfec9a10608 [term 3 FOLLOWER]: Advancing to term 4
I20260129 03:00:01.822563 20584 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "36ff33be666a4c82954731691dbcf327" candidate_uuid: "197b880fa39d432a9b472845876bfaf0" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "1c50de89a5a845668696dcb28c3badd6" is_pre_election: true
I20260129 03:00:01.823063 21130 raft_consensus.cc:1275] T 0678b8b468bd43cdbc3e485900844821 P 7a9daf5d42a84615a6383cfec9a10608 [term 7 FOLLOWER]: Refusing update from remote peer 974b802141bd4f13beffef2aa03f88f9: Log matching property violated. Preceding OpId in replica: term: 5 index: 7. Preceding OpId from leader: term: 7 index: 8. (index mismatch)
I20260129 03:00:01.663395 20582 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a5a05fbcbbcf4916bbc01fd1a5b5e90d" candidate_uuid: "197b880fa39d432a9b472845876bfaf0" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "1c50de89a5a845668696dcb28c3badd6" is_pre_election: true
I20260129 03:00:01.852754 20584 raft_consensus.cc:2468] T 36ff33be666a4c82954731691dbcf327 P 1c50de89a5a845668696dcb28c3badd6 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 197b880fa39d432a9b472845876bfaf0 in term 3.
I20260129 03:00:01.739467 20581 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c891d2bff4ce4568861bbc992e91fbb0" candidate_uuid: "197b880fa39d432a9b472845876bfaf0" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "1c50de89a5a845668696dcb28c3badd6"
I20260129 03:00:01.853858 20581 raft_consensus.cc:3060] T c891d2bff4ce4568861bbc992e91fbb0 P 1c50de89a5a845668696dcb28c3badd6 [term 6 FOLLOWER]: Advancing to term 7
I20260129 03:00:01.825810 22950 raft_consensus.cc:3060] T 913fc7d751ed4b7388d889aac30a5f9b P 197b880fa39d432a9b472845876bfaf0 [term 3 FOLLOWER]: Advancing to term 4
I20260129 03:00:01.826320 23112 raft_consensus.cc:3055] T febffe4a8b394368b69b1d164308fb9a P 197b880fa39d432a9b472845876bfaf0 [term 3 LEADER]: Stepping down as leader of term 3
W20260129 03:00:01.827499 20433 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55676 (request call id 1385) took 2346 ms (client timeout 1631 ms). Trace:
W20260129 03:00:01.857717 20433 rpcz_store.cc:269] 0129 02:59:59.481532 (+ 0us) service_pool.cc:168] Inserting onto call queue
0129 02:59:59.481674 (+ 142us) service_pool.cc:225] Handling call
0129 03:00:01.827071 (+2345397us) spinlock_profiling.cc:243] Waited 26 ms on lock 0x7b54002e0c54. stack: 00007f8fe044e373 00007f8fe044e270 00007f8fdfda0574 00007f8fe9a5c6b4 00007f8fe9a5c649 00007f8fe34975a1 00007f8fe3531f7a 00007f8fe84e3647 00007f8fe1a73834 00007f8fe1a737c2 00007f8fe1a736f1 00007f8fe1a7366c 00007f8fe1a72922 00007f8fe186d34c 00007f8fe186c786 00007f8fe186c12f
0129 03:00:01.827232 (+ 161us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":59695488}
W20260129 03:00:01.827941 20554 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260129 03:00:01.859107 20554 connection.cc:232] RPC call timeout handler was delayed by -1.7149147770001036s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.18480000000000002s for a response
I20260129 03:00:01.855558 22851 raft_consensus.cc:493] T b189f08e16a74a11980cc55d99359a0a P 197b880fa39d432a9b472845876bfaf0 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 1c50de89a5a845668696dcb28c3badd6)
I20260129 03:00:01.860049 23146 leader_election.cc:290] T febffe4a8b394368b69b1d164308fb9a P 974b802141bd4f13beffef2aa03f88f9 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 197b880fa39d432a9b472845876bfaf0 (127.18.211.1:42027), 7a9daf5d42a84615a6383cfec9a10608 (127.18.211.4:40439)
I20260129 03:00:01.860700 23152 leader_election.cc:290] T a1a39dcd328e4268ae41b988c66b4a75 P 974b802141bd4f13beffef2aa03f88f9 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3:37735), 197b880fa39d432a9b472845876bfaf0 (127.18.211.1:42027)
I20260129 03:00:01.819124 20585 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1ce4ddf5b10e4ae8b193b13b2e5936ee" candidate_uuid: "197b880fa39d432a9b472845876bfaf0" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "1c50de89a5a845668696dcb28c3badd6"
I20260129 03:00:01.664230 23147 leader_election.cc:290] T c2633a6fc34c44579de348d208b0c01e P 1c50de89a5a845668696dcb28c3badd6 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 197b880fa39d432a9b472845876bfaf0 (127.18.211.1:42027), 974b802141bd4f13beffef2aa03f88f9 (127.18.211.2:34517)
I20260129 03:00:01.864792 20584 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "368953c9102643b2a21f99af36b51905" candidate_uuid: "197b880fa39d432a9b472845876bfaf0" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "1c50de89a5a845668696dcb28c3badd6"
I20260129 03:00:01.597093 23163 consensus_queue.cc:260] T fd8311b779af4e1fa1402b2a02a602f2 P 7a9daf5d42a84615a6383cfec9a10608 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } }
I20260129 03:00:01.868476 23163 raft_consensus.cc:3060] T fd8311b779af4e1fa1402b2a02a602f2 P 7a9daf5d42a84615a6383cfec9a10608 [term 4 FOLLOWER]: Advancing to term 5
W20260129 03:00:01.871621 23172 log.cc:927] Time spent T 8055232cf0004d40a85f3bf983fe00a9 P 197b880fa39d432a9b472845876bfaf0: Append to log took a long time: real 0.430s user 0.001s sys 0.001s
I20260129 03:00:01.876776 20584 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "96609ac68bdf4993a12e1c23355ca013" candidate_uuid: "197b880fa39d432a9b472845876bfaf0" candidate_term: 7 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "1c50de89a5a845668696dcb28c3badd6" is_pre_election: true
I20260129 03:00:01.856873 20407 leader_election.cc:304] T 75be4bd6bd45474795fd472eff2953ac P 197b880fa39d432a9b472845876bfaf0 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 197b880fa39d432a9b472845876bfaf0, 974b802141bd4f13beffef2aa03f88f9; no voters:
I20260129 03:00:01.878432 23165 raft_consensus.cc:493] T 2ac659d732314e1288849f08fe5e074e P 7a9daf5d42a84615a6383cfec9a10608 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260129 03:00:01.879094 23165 raft_consensus.cc:515] T 2ac659d732314e1288849f08fe5e074e P 7a9daf5d42a84615a6383cfec9a10608 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } }
I20260129 03:00:01.879643 23077 raft_consensus.cc:3055] T 00e79d8f25a34676b1d866eb2068a2b2 P 7a9daf5d42a84615a6383cfec9a10608 [term 6 LEADER]: Stepping down as leader of term 6
I20260129 03:00:01.880137 23077 raft_consensus.cc:740] T 00e79d8f25a34676b1d866eb2068a2b2 P 7a9daf5d42a84615a6383cfec9a10608 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 7a9daf5d42a84615a6383cfec9a10608, State: Running, Role: LEADER
I20260129 03:00:01.880026 20584 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a8479d1fe0bd41e5946186e3a6ca8c7a" candidate_uuid: "197b880fa39d432a9b472845876bfaf0" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "1c50de89a5a845668696dcb28c3badd6" is_pre_election: true
I20260129 03:00:01.881306 23165 leader_election.cc:290] T 2ac659d732314e1288849f08fe5e074e P 7a9daf5d42a84615a6383cfec9a10608 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 197b880fa39d432a9b472845876bfaf0 (127.18.211.1:42027), 974b802141bd4f13beffef2aa03f88f9 (127.18.211.2:34517)
I20260129 03:00:01.881057 23077 consensus_queue.cc:260] T 00e79d8f25a34676b1d866eb2068a2b2 P 7a9daf5d42a84615a6383cfec9a10608 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 6.8, Last appended by leader: 8, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } }
I20260129 03:00:01.881505 20584 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "cedce331e1584ac8b636c9b8a72b2ba0" candidate_uuid: "974b802141bd4f13beffef2aa03f88f9" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "1c50de89a5a845668696dcb28c3badd6"
I20260129 03:00:01.882261 23077 raft_consensus.cc:3060] T 00e79d8f25a34676b1d866eb2068a2b2 P 7a9daf5d42a84615a6383cfec9a10608 [term 6 FOLLOWER]: Advancing to term 7
I20260129 03:00:01.882562 20584 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e33a417cf9c54cf5af73a4ee3aac4d3a" candidate_uuid: "197b880fa39d432a9b472845876bfaf0" candidate_term: 7 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "1c50de89a5a845668696dcb28c3badd6" is_pre_election: true
I20260129 03:00:01.884981 23168 leader_election.cc:290] T 4e3805ffa70242e8ade66da260dddf66 P 7a9daf5d42a84615a6383cfec9a10608 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 197b880fa39d432a9b472845876bfaf0 (127.18.211.1:42027), 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3:37735)
I20260129 03:00:01.883145 20584 raft_consensus.cc:2468] T e33a417cf9c54cf5af73a4ee3aac4d3a P 1c50de89a5a845668696dcb28c3badd6 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 197b880fa39d432a9b472845876bfaf0 in term 6.
I20260129 03:00:01.886656 20584 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0340229d521b432580ae926f6062e3f7" candidate_uuid: "197b880fa39d432a9b472845876bfaf0" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "1c50de89a5a845668696dcb28c3badd6" is_pre_election: true
I20260129 03:00:01.887358 20584 raft_consensus.cc:2468] T 0340229d521b432580ae926f6062e3f7 P 1c50de89a5a845668696dcb28c3badd6 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 197b880fa39d432a9b472845876bfaf0 in term 6.
I20260129 03:00:01.857599 23112 raft_consensus.cc:740] T febffe4a8b394368b69b1d164308fb9a P 197b880fa39d432a9b472845876bfaf0 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 197b880fa39d432a9b472845876bfaf0, State: Running, Role: LEADER
I20260129 03:00:01.888126 20584 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4408006eb5174de1896b037cf3f18457" candidate_uuid: "197b880fa39d432a9b472845876bfaf0" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "1c50de89a5a845668696dcb28c3badd6" is_pre_election: true
I20260129 03:00:01.888724 20584 raft_consensus.cc:2468] T 4408006eb5174de1896b037cf3f18457 P 1c50de89a5a845668696dcb28c3badd6 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 197b880fa39d432a9b472845876bfaf0 in term 3.
I20260129 03:00:01.888890 23112 consensus_queue.cc:260] T febffe4a8b394368b69b1d164308fb9a P 197b880fa39d432a9b472845876bfaf0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.5, Last appended by leader: 5, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } }
I20260129 03:00:01.889532 20584 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4408006eb5174de1896b037cf3f18457" candidate_uuid: "197b880fa39d432a9b472845876bfaf0" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "1c50de89a5a845668696dcb28c3badd6"
I20260129 03:00:01.890268 20584 raft_consensus.cc:3060] T 4408006eb5174de1896b037cf3f18457 P 1c50de89a5a845668696dcb28c3badd6 [term 3 FOLLOWER]: Advancing to term 4
I20260129 03:00:01.890590 23112 raft_consensus.cc:3060] T febffe4a8b394368b69b1d164308fb9a P 197b880fa39d432a9b472845876bfaf0 [term 3 FOLLOWER]: Advancing to term 4
I20260129 03:00:01.854549 23091 raft_consensus.cc:2749] T 4e420f90884f4796997b5ea41e49e411 P 197b880fa39d432a9b472845876bfaf0 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260129 03:00:01.892342 23119 leader_election.cc:290] T cb0b3f259ebc4310927338a8740e87b8 P 7a9daf5d42a84615a6383cfec9a10608 [CANDIDATE]: Term 7 election: Requested vote from peers 974b802141bd4f13beffef2aa03f88f9 (127.18.211.2:34517), 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3:37735)
I20260129 03:00:01.892407 23091 raft_consensus.cc:493] T 00e79d8f25a34676b1d866eb2068a2b2 P 197b880fa39d432a9b472845876bfaf0 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260129 03:00:01.856541 22877 consensus_queue.cc:1048] T 4408006eb5174de1896b037cf3f18457 P 197b880fa39d432a9b472845876bfaf0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 1, Time since last communication: 0.037s
I20260129 03:00:01.893172 23091 raft_consensus.cc:515] T 00e79d8f25a34676b1d866eb2068a2b2 P 197b880fa39d432a9b472845876bfaf0 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } }
I20260129 03:00:01.894176 22877 raft_consensus.cc:493] T 7a9a69cd82724b1faed1d10aa091fc4a P 197b880fa39d432a9b472845876bfaf0 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 1c50de89a5a845668696dcb28c3badd6)
I20260129 03:00:01.894690 22877 raft_consensus.cc:515] T 7a9a69cd82724b1faed1d10aa091fc4a P 197b880fa39d432a9b472845876bfaf0 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 03:00:01.896543 23042 leader_election.cc:290] T 2ac659d732314e1288849f08fe5e074e P 197b880fa39d432a9b472845876bfaf0 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 974b802141bd4f13beffef2aa03f88f9 (127.18.211.2:34517), 7a9daf5d42a84615a6383cfec9a10608 (127.18.211.4:40439)
I20260129 03:00:01.896787 21130 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4e3805ffa70242e8ade66da260dddf66" candidate_uuid: "197b880fa39d432a9b472845876bfaf0" candidate_term: 6 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "7a9daf5d42a84615a6383cfec9a10608" is_pre_election: true
I20260129 03:00:01.897109 23042 raft_consensus.cc:493] T 18f30ebaa4174fd28c958e72d0cc7e24 P 197b880fa39d432a9b472845876bfaf0 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 1c50de89a5a845668696dcb28c3badd6)
I20260129 03:00:01.897953 21130 raft_consensus.cc:2410] T 4e3805ffa70242e8ade66da260dddf66 P 7a9daf5d42a84615a6383cfec9a10608 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 197b880fa39d432a9b472845876bfaf0 for term 6 because replica has last-logged OpId of term: 5 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 6.
I20260129 03:00:01.898522 20584 raft_consensus.cc:2468] T 4408006eb5174de1896b037cf3f18457 P 1c50de89a5a845668696dcb28c3badd6 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 197b880fa39d432a9b472845876bfaf0 in term 4.
I20260129 03:00:01.863131 22980 raft_consensus.cc:3060] T cb4727386a3e48aca5d71cc6c611c197 P 197b880fa39d432a9b472845876bfaf0 [term 3 FOLLOWER]: Advancing to term 4
I20260129 03:00:01.899122 21130 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "febffe4a8b394368b69b1d164308fb9a" candidate_uuid: "974b802141bd4f13beffef2aa03f88f9" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "7a9daf5d42a84615a6383cfec9a10608" is_pre_election: true
I20260129 03:00:01.899410 20584 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c2633a6fc34c44579de348d208b0c01e" candidate_uuid: "197b880fa39d432a9b472845876bfaf0" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "1c50de89a5a845668696dcb28c3badd6"
I20260129 03:00:01.900553 20584 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "75be4bd6bd45474795fd472eff2953ac" candidate_uuid: "197b880fa39d432a9b472845876bfaf0" candidate_term: 10 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "1c50de89a5a845668696dcb28c3badd6" is_pre_election: true
I20260129 03:00:01.854523 20582 raft_consensus.cc:2468] T a5a05fbcbbcf4916bbc01fd1a5b5e90d P 1c50de89a5a845668696dcb28c3badd6 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 197b880fa39d432a9b472845876bfaf0 in term 7.
I20260129 03:00:01.652244 23116 leader_election.cc:290] T 4e3805ffa70242e8ade66da260dddf66 P 197b880fa39d432a9b472845876bfaf0 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3:37735), 7a9daf5d42a84615a6383cfec9a10608 (127.18.211.4:40439)
I20260129 03:00:01.773629 20589 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3906b080a5b34a74b8b64128635e8997" candidate_uuid: "197b880fa39d432a9b472845876bfaf0" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "1c50de89a5a845668696dcb28c3badd6" is_pre_election: true
I20260129 03:00:01.863049 20585 raft_consensus.cc:2410] T 1ce4ddf5b10e4ae8b193b13b2e5936ee P 1c50de89a5a845668696dcb28c3badd6 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 197b880fa39d432a9b472845876bfaf0 for term 7 because replica has last-logged OpId of term: 6 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 5 index: 3.
I20260129 03:00:01.831566 21102 leader_election.cc:304] T 4f5a77e94fef4b16a2ba7b067c8b144a P 7a9daf5d42a84615a6383cfec9a10608 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7a9daf5d42a84615a6383cfec9a10608; no voters: 1c50de89a5a845668696dcb28c3badd6, 974b802141bd4f13beffef2aa03f88f9
I20260129 03:00:01.895685 23091 leader_election.cc:290] T 00e79d8f25a34676b1d866eb2068a2b2 P 197b880fa39d432a9b472845876bfaf0 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 974b802141bd4f13beffef2aa03f88f9 (127.18.211.2:34517), 7a9daf5d42a84615a6383cfec9a10608 (127.18.211.4:40439)
I20260129 03:00:01.739449 23145 leader_election.cc:290] T 3ccac82a01a2445fa5b0ce6c0fcf2295 P 974b802141bd4f13beffef2aa03f88f9 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 197b880fa39d432a9b472845876bfaf0 (127.18.211.1:42027), 7a9daf5d42a84615a6383cfec9a10608 (127.18.211.4:40439)
I20260129 03:00:01.743634 23141 leader_election.cc:290] T 42d27a28d89b409e8807ec0efd97af69 P 7a9daf5d42a84615a6383cfec9a10608 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 197b880fa39d432a9b472845876bfaf0 (127.18.211.1:42027), 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3:37735)
I20260129 03:00:01.867367 20581 raft_consensus.cc:2468] T c891d2bff4ce4568861bbc992e91fbb0 P 1c50de89a5a845668696dcb28c3badd6 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 197b880fa39d432a9b472845876bfaf0 in term 7.
I20260129 03:00:01.867733 21134 raft_consensus.cc:1240] T febffe4a8b394368b69b1d164308fb9a P 7a9daf5d42a84615a6383cfec9a10608 [term 4 FOLLOWER]: Rejecting Update request from peer 197b880fa39d432a9b472845876bfaf0 for earlier term 3. Current term is 4. Ops: [3.5-3.5]
I20260129 03:00:01.897635 23042 raft_consensus.cc:515] T 18f30ebaa4174fd28c958e72d0cc7e24 P 197b880fa39d432a9b472845876bfaf0 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } }
I20260129 03:00:01.902192 20582 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e4d18c35b0a7493e85d0a4b92956a48f" candidate_uuid: "197b880fa39d432a9b472845876bfaf0" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "1c50de89a5a845668696dcb28c3badd6" is_pre_election: true
I20260129 03:00:01.810206 23026 raft_consensus.cc:3060] T 368953c9102643b2a21f99af36b51905 P 1c50de89a5a845668696dcb28c3badd6 [term 6 FOLLOWER]: Advancing to term 7
I20260129 03:00:01.942498 20584 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "8055232cf0004d40a85f3bf983fe00a9" candidate_uuid: "197b880fa39d432a9b472845876bfaf0" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "1c50de89a5a845668696dcb28c3badd6"
I20260129 03:00:01.914297 20585 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "724a236524cb4e7ba3428880681d6cc2" candidate_uuid: "197b880fa39d432a9b472845876bfaf0" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "1c50de89a5a845668696dcb28c3badd6" is_pre_election: true
I20260129 03:00:01.948824 20585 raft_consensus.cc:2410] T 724a236524cb4e7ba3428880681d6cc2 P 1c50de89a5a845668696dcb28c3badd6 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 197b880fa39d432a9b472845876bfaf0 for term 5 because replica has last-logged OpId of term: 4 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 2.
W20260129 03:00:01.891149 22986 consensus_meta.cc:287] T a0fcc58d6fb6473cafbd9d34d5195269 P 974b802141bd4f13beffef2aa03f88f9: Time spent flushing consensus metadata: real 1.262s user 0.010s sys 0.001s
I20260129 03:00:01.949491 20584 raft_consensus.cc:2376] T 8055232cf0004d40a85f3bf983fe00a9 P 1c50de89a5a845668696dcb28c3badd6 [term 4 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 197b880fa39d432a9b472845876bfaf0 in term 4. Re-sending same reply.
W20260129 03:00:01.865748 23177 log.cc:927] Time spent T 926a250fa5834e88b96ccecbefbf5086 P 974b802141bd4f13beffef2aa03f88f9: Append to log took a long time: real 0.204s user 0.000s sys 0.000s
I20260129 03:00:01.950531 20581 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c2633a6fc34c44579de348d208b0c01e" candidate_uuid: "197b880fa39d432a9b472845876bfaf0" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "1c50de89a5a845668696dcb28c3badd6" is_pre_election: true
I20260129 03:00:01.947906 20582 raft_consensus.cc:2468] T e4d18c35b0a7493e85d0a4b92956a48f P 1c50de89a5a845668696dcb28c3badd6 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 197b880fa39d432a9b472845876bfaf0 in term 5.
I20260129 03:00:01.952805 20582 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "cb0b3f259ebc4310927338a8740e87b8" candidate_uuid: "7a9daf5d42a84615a6383cfec9a10608" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "1c50de89a5a845668696dcb28c3badd6"
I20260129 03:00:01.953408 20582 raft_consensus.cc:3060] T cb0b3f259ebc4310927338a8740e87b8 P 1c50de89a5a845668696dcb28c3badd6 [term 6 FOLLOWER]: Advancing to term 7
W20260129 03:00:01.954069 21102 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.211.2:34517, user_credentials={real_user=slave}} blocked reactor thread for 122618us
I20260129 03:00:01.950528 20584 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4b1876f9a70b49edb3aa74105e66d3b3" candidate_uuid: "197b880fa39d432a9b472845876bfaf0" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "1c50de89a5a845668696dcb28c3badd6"
I20260129 03:00:01.955189 21102 leader_election.cc:304] T 0678b8b468bd43cdbc3e485900844821 P 7a9daf5d42a84615a6383cfec9a10608 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7a9daf5d42a84615a6383cfec9a10608; no voters: 197b880fa39d432a9b472845876bfaf0, 974b802141bd4f13beffef2aa03f88f9
I20260129 03:00:01.955626 20584 raft_consensus.cc:2376] T 4b1876f9a70b49edb3aa74105e66d3b3 P 1c50de89a5a845668696dcb28c3badd6 [term 5 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 197b880fa39d432a9b472845876bfaf0 in term 5. Re-sending same reply.
I20260129 03:00:01.956647 20584 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e7d83d4ccf684e76acb29a0dc844e1b8" candidate_uuid: "974b802141bd4f13beffef2aa03f88f9" candidate_term: 6 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "1c50de89a5a845668696dcb28c3badd6" is_pre_election: true
I20260129 03:00:01.957376 20584 raft_consensus.cc:2393] T e7d83d4ccf684e76acb29a0dc844e1b8 P 1c50de89a5a845668696dcb28c3badd6 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 974b802141bd4f13beffef2aa03f88f9 in current term 6: Already voted for candidate 1c50de89a5a845668696dcb28c3badd6 in this term.
I20260129 03:00:01.962162 20508 raft_consensus.cc:2468] T a0fcc58d6fb6473cafbd9d34d5195269 P 974b802141bd4f13beffef2aa03f88f9 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1c50de89a5a845668696dcb28c3badd6 in term 4.
I20260129 03:00:01.964983 20589 raft_consensus.cc:2468] T 3906b080a5b34a74b8b64128635e8997 P 1c50de89a5a845668696dcb28c3badd6 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 197b880fa39d432a9b472845876bfaf0 in term 4.
I20260129 03:00:01.966298 20585 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a0fcc58d6fb6473cafbd9d34d5195269" candidate_uuid: "197b880fa39d432a9b472845876bfaf0" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "1c50de89a5a845668696dcb28c3badd6"
I20260129 03:00:01.966922 20585 raft_consensus.cc:2376] T a0fcc58d6fb6473cafbd9d34d5195269 P 1c50de89a5a845668696dcb28c3badd6 [term 4 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 197b880fa39d432a9b472845876bfaf0 in term 4. Re-sending same reply.
I20260129 03:00:01.967752 21102 leader_election.cc:304] T add8d4923d1a489e86882c3a25692b95 P 7a9daf5d42a84615a6383cfec9a10608 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 7a9daf5d42a84615a6383cfec9a10608, 974b802141bd4f13beffef2aa03f88f9; no voters: 1c50de89a5a845668696dcb28c3badd6
W20260129 03:00:01.969293 20406 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260129 03:00:01.974265 20584 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4e3805ffa70242e8ade66da260dddf66" candidate_uuid: "7a9daf5d42a84615a6383cfec9a10608" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "1c50de89a5a845668696dcb28c3badd6" is_pre_election: true
I20260129 03:00:01.860078 22851 raft_consensus.cc:515] T b189f08e16a74a11980cc55d99359a0a P 197b880fa39d432a9b472845876bfaf0 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
W20260129 03:00:01.961578 23080 consensus_meta.cc:287] T 45cc5f3d2688486d9b6dbe7e666dedde P 7a9daf5d42a84615a6383cfec9a10608: Time spent flushing consensus metadata: real 1.027s user 0.011s sys 0.000s
I20260129 03:00:01.983392 20589 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "42d27a28d89b409e8807ec0efd97af69" candidate_uuid: "7a9daf5d42a84615a6383cfec9a10608" candidate_term: 8 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "1c50de89a5a845668696dcb28c3badd6" is_pre_election: true
I20260129 03:00:01.978955 21102 leader_election.cc:304] T febffe4a8b394368b69b1d164308fb9a P 7a9daf5d42a84615a6383cfec9a10608 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7a9daf5d42a84615a6383cfec9a10608; no voters: 197b880fa39d432a9b472845876bfaf0, 974b802141bd4f13beffef2aa03f88f9
I20260129 03:00:01.984860 20582 raft_consensus.cc:2468] T cb0b3f259ebc4310927338a8740e87b8 P 1c50de89a5a845668696dcb28c3badd6 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7a9daf5d42a84615a6383cfec9a10608 in term 7.
W20260129 03:00:01.920153 23039 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.583s user 0.006s sys 0.040s
I20260129 03:00:01.985297 20581 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "75b0b6e82ddc4f92b1fe1d94f9a0b73b" candidate_uuid: "7a9daf5d42a84615a6383cfec9a10608" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "1c50de89a5a845668696dcb28c3badd6" is_pre_election: true
I20260129 03:00:01.915796 22231 raft_consensus.cc:2804] T 75be4bd6bd45474795fd472eff2953ac P 197b880fa39d432a9b472845876bfaf0 [term 9 FOLLOWER]: Leader pre-election won for term 10
I20260129 03:00:01.950268 23119 raft_consensus.cc:3060] T 4f5a77e94fef4b16a2ba7b067c8b144a P 7a9daf5d42a84615a6383cfec9a10608 [term 3 FOLLOWER]: Advancing to term 4
I20260129 03:00:01.978102 20585 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0f20af7fbe5d4db89a6f17948d98912c" candidate_uuid: "197b880fa39d432a9b472845876bfaf0" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "1c50de89a5a845668696dcb28c3badd6" is_pre_election: true
I20260129 03:00:01.947271 22967 raft_consensus.cc:3060] T 2ac659d732314e1288849f08fe5e074e P 974b802141bd4f13beffef2aa03f88f9 [term 3 FOLLOWER]: Advancing to term 4
I20260129 03:00:01.979804 23141 raft_consensus.cc:2749] T 0678b8b468bd43cdbc3e485900844821 P 7a9daf5d42a84615a6383cfec9a10608 [term 7 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260129 03:00:01.851082 23151 raft_consensus.cc:515] T 0340229d521b432580ae926f6062e3f7 P 974b802141bd4f13beffef2aa03f88f9 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 03:00:01.990345 21100 leader_election.cc:304] T cb0b3f259ebc4310927338a8740e87b8 P 7a9daf5d42a84615a6383cfec9a10608 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1c50de89a5a845668696dcb28c3badd6, 7a9daf5d42a84615a6383cfec9a10608; no voters:
I20260129 03:00:02.005343 20589 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4f5a77e94fef4b16a2ba7b067c8b144a" candidate_uuid: "974b802141bd4f13beffef2aa03f88f9" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "1c50de89a5a845668696dcb28c3badd6"
I20260129 03:00:01.993203 20581 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "724a236524cb4e7ba3428880681d6cc2" candidate_uuid: "974b802141bd4f13beffef2aa03f88f9" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "1c50de89a5a845668696dcb28c3badd6" is_pre_election: true
I20260129 03:00:02.002962 23098 raft_consensus.cc:2749] T febffe4a8b394368b69b1d164308fb9a P 7a9daf5d42a84615a6383cfec9a10608 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260129 03:00:02.028966 20581 raft_consensus.cc:2468] T 724a236524cb4e7ba3428880681d6cc2 P 1c50de89a5a845668696dcb28c3badd6 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 974b802141bd4f13beffef2aa03f88f9 in term 5.
I20260129 03:00:01.992259 20584 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c2633a6fc34c44579de348d208b0c01e" candidate_uuid: "974b802141bd4f13beffef2aa03f88f9" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "1c50de89a5a845668696dcb28c3badd6" is_pre_election: true
I20260129 03:00:01.966512 22878 raft_consensus.cc:493] T a0fcc58d6fb6473cafbd9d34d5195269 P 974b802141bd4f13beffef2aa03f88f9 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260129 03:00:01.997820 20585 raft_consensus.cc:2376] T 0f20af7fbe5d4db89a6f17948d98912c P 1c50de89a5a845668696dcb28c3badd6 [term 7 FOLLOWER]: Leader pre-election vote request: Already granted yes vote for candidate 197b880fa39d432a9b472845876bfaf0 in term 7. Re-sending same reply.
I20260129 03:00:01.988948 21102 leader_election.cc:304] T c934d31efd3f4e5195605ddc354d14e1 P 7a9daf5d42a84615a6383cfec9a10608 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7a9daf5d42a84615a6383cfec9a10608; no voters: 197b880fa39d432a9b472845876bfaf0, 974b802141bd4f13beffef2aa03f88f9
I20260129 03:00:01.995209 20582 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2e58e9a5a8aa4cecafb464addb4f35ed" candidate_uuid: "197b880fa39d432a9b472845876bfaf0" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "1c50de89a5a845668696dcb28c3badd6" is_pre_election: true
I20260129 03:00:01.914747 23152 consensus_queue.cc:1048] T 0678b8b468bd43cdbc3e485900844821 P 974b802141bd4f13beffef2aa03f88f9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.073s
I20260129 03:00:02.032724 20585 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a1a39dcd328e4268ae41b988c66b4a75" candidate_uuid: "974b802141bd4f13beffef2aa03f88f9" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "1c50de89a5a845668696dcb28c3badd6" is_pre_election: true
I20260129 03:00:02.010948 22978 raft_consensus.cc:515] T add8d4923d1a489e86882c3a25692b95 P 974b802141bd4f13beffef2aa03f88f9 [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } }
W20260129 03:00:02.015305 20406 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260129 03:00:02.003311 23184 raft_consensus.cc:2804] T add8d4923d1a489e86882c3a25692b95 P 7a9daf5d42a84615a6383cfec9a10608 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260129 03:00:01.999049 22231 raft_consensus.cc:493] T 75be4bd6bd45474795fd472eff2953ac P 197b880fa39d432a9b472845876bfaf0 [term 9 FOLLOWER]: Starting leader election (detected failure of leader 1c50de89a5a845668696dcb28c3badd6)
I20260129 03:00:02.041863 20586 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:35754 (request call id 1712) took 1099 ms. Trace:
I20260129 03:00:02.042424 20581 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "505580c93c694bbb8ea96fdf9b17ab46" candidate_uuid: "974b802141bd4f13beffef2aa03f88f9" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "1c50de89a5a845668696dcb28c3badd6" is_pre_election: true
W20260129 03:00:01.986703 23039 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.682s user 0.007s sys 0.044s
I20260129 03:00:02.027918 20513 raft_consensus.cc:2468] T 505580c93c694bbb8ea96fdf9b17ab46 P 974b802141bd4f13beffef2aa03f88f9 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7a9daf5d42a84615a6383cfec9a10608 in term 6.
I20260129 03:00:02.044790 20513 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:48544 (request call id 266) took 1002 ms. Trace:
I20260129 03:00:02.025144 23182 raft_consensus.cc:2804] T cb0b3f259ebc4310927338a8740e87b8 P 7a9daf5d42a84615a6383cfec9a10608 [term 7 FOLLOWER]: Leader election won for term 7
I20260129 03:00:02.038674 20589 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1ce4ddf5b10e4ae8b193b13b2e5936ee" candidate_uuid: "197b880fa39d432a9b472845876bfaf0" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "1c50de89a5a845668696dcb28c3badd6" is_pre_election: true
I20260129 03:00:02.029325 22877 leader_election.cc:290] T 7a9a69cd82724b1faed1d10aa091fc4a P 197b880fa39d432a9b472845876bfaf0 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 974b802141bd4f13beffef2aa03f88f9 (127.18.211.2:34517), 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3:37735)
I20260129 03:00:02.019073 20554 leader_election.cc:304] T 45cc5f3d2688486d9b6dbe7e666dedde P 1c50de89a5a845668696dcb28c3badd6 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1c50de89a5a845668696dcb28c3badd6, 7a9daf5d42a84615a6383cfec9a10608; no voters:
I20260129 03:00:01.989765 23080 raft_consensus.cc:515] T 45cc5f3d2688486d9b6dbe7e666dedde P 7a9daf5d42a84615a6383cfec9a10608 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } }
I20260129 03:00:02.034976 23077 raft_consensus.cc:2749] T c934d31efd3f4e5195605ddc354d14e1 P 7a9daf5d42a84615a6383cfec9a10608 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260129 03:00:02.042251 20586 rpcz_store.cc:276] 0129 03:00:00.942146 (+ 0us) service_pool.cc:168] Inserting onto call queue
0129 03:00:00.943698 (+ 1552us) service_pool.cc:225] Handling call
0129 03:00:02.041700 (+1098002us) spinlock_profiling.cc:243] Waited 264 ms on lock 0x7b54001540d4. stack: 00007f8fe044e373 00007f8fe044e270 00007f8fdfda0574 00007f8fe9a5c6b4 00007f8fe9a5c649 00007f8fe34975a1 00007f8fe3531f7a 00007f8fe84e3647 00007f8fe1a73834 00007f8fe1a737c2 00007f8fe1a736f1 00007f8fe1a7366c 00007f8fe1a72922 00007f8fe186d34c 00007f8fe186c786 00007f8fe186c12f
0129 03:00:02.041807 (+ 107us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":606662528}
I20260129 03:00:02.020033 23183 consensus_queue.cc:1059] T a644c02b16af47679824ccd060ac10fd P 7a9daf5d42a84615a6383cfec9a10608 [NON_LEADER]: Peer responded invalid term: Peer: permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.042s
I20260129 03:00:02.045251 20513 rpcz_store.cc:276] 0129 03:00:01.042675 (+ 0us) service_pool.cc:168] Inserting onto call queue
0129 03:00:01.420492 (+377817us) service_pool.cc:225] Handling call
0129 03:00:02.044733 (+624241us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260129 03:00:02.038745 20585 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "78aa39096868415d9afc2ffcb248e198" candidate_uuid: "197b880fa39d432a9b472845876bfaf0" candidate_term: 6 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "1c50de89a5a845668696dcb28c3badd6" is_pre_election: true
I20260129 03:00:02.031551 20582 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1299723a4476420ab02fd8bfd7aa0645" candidate_uuid: "197b880fa39d432a9b472845876bfaf0" candidate_term: 9 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "1c50de89a5a845668696dcb28c3badd6" is_pre_election: true
I20260129 03:00:02.067935 20513 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2ac659d732314e1288849f08fe5e074e" candidate_uuid: "7a9daf5d42a84615a6383cfec9a10608" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "974b802141bd4f13beffef2aa03f88f9" is_pre_election: true
I20260129 03:00:02.068380 20582 raft_consensus.cc:2393] T 1299723a4476420ab02fd8bfd7aa0645 P 1c50de89a5a845668696dcb28c3badd6 [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 197b880fa39d432a9b472845876bfaf0 in current term 9: Already voted for candidate 1c50de89a5a845668696dcb28c3badd6 in this term.
I20260129 03:00:02.053490 20589 raft_consensus.cc:2410] T 1ce4ddf5b10e4ae8b193b13b2e5936ee P 1c50de89a5a845668696dcb28c3badd6 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 197b880fa39d432a9b472845876bfaf0 for term 7 because replica has last-logged OpId of term: 6 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 5 index: 3.
I20260129 03:00:02.069224 20513 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "cb0b3f259ebc4310927338a8740e87b8" candidate_uuid: "7a9daf5d42a84615a6383cfec9a10608" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "974b802141bd4f13beffef2aa03f88f9"
I20260129 03:00:02.057484 23098 raft_consensus.cc:493] T 18f30ebaa4174fd28c958e72d0cc7e24 P 7a9daf5d42a84615a6383cfec9a10608 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 1c50de89a5a845668696dcb28c3badd6)
I20260129 03:00:02.069998 23098 raft_consensus.cc:515] T 18f30ebaa4174fd28c958e72d0cc7e24 P 7a9daf5d42a84615a6383cfec9a10608 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } }
I20260129 03:00:02.070391 20513 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "75b0b6e82ddc4f92b1fe1d94f9a0b73b" candidate_uuid: "7a9daf5d42a84615a6383cfec9a10608" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "974b802141bd4f13beffef2aa03f88f9" is_pre_election: true
I20260129 03:00:02.071095 20513 raft_consensus.cc:2468] T 75b0b6e82ddc4f92b1fe1d94f9a0b73b P 974b802141bd4f13beffef2aa03f88f9 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7a9daf5d42a84615a6383cfec9a10608 in term 3.
I20260129 03:00:02.072163 23098 leader_election.cc:290] T 18f30ebaa4174fd28c958e72d0cc7e24 P 7a9daf5d42a84615a6383cfec9a10608 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 197b880fa39d432a9b472845876bfaf0 (127.18.211.1:42027), 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3:37735)
W20260129 03:00:02.040874 21102 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260129 03:00:02.072989 21102 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260129 03:00:02.073608 22977 consensus_queue.cc:1048] T add8d4923d1a489e86882c3a25692b95 P 1c50de89a5a845668696dcb28c3badd6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 5, Time since last communication: 0.000s
I20260129 03:00:02.077165 20582 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "add8d4923d1a489e86882c3a25692b95" candidate_uuid: "974b802141bd4f13beffef2aa03f88f9" candidate_term: 6 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "1c50de89a5a845668696dcb28c3badd6" is_pre_election: true
I20260129 03:00:02.069447 20589 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "cb4727386a3e48aca5d71cc6c611c197" candidate_uuid: "974b802141bd4f13beffef2aa03f88f9" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1c50de89a5a845668696dcb28c3badd6"
I20260129 03:00:02.078243 20589 raft_consensus.cc:2393] T cb4727386a3e48aca5d71cc6c611c197 P 1c50de89a5a845668696dcb28c3badd6 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 974b802141bd4f13beffef2aa03f88f9 in current term 4: Already voted for candidate 1c50de89a5a845668696dcb28c3badd6 in this term.
W20260129 03:00:02.060467 20407 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.211.2:34517, user_credentials={real_user=slave}} blocked reactor thread for 203649us
I20260129 03:00:02.040246 23184 raft_consensus.cc:493] T add8d4923d1a489e86882c3a25692b95 P 7a9daf5d42a84615a6383cfec9a10608 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260129 03:00:02.064764 20555 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260129 03:00:02.083631 21102 leader_election.cc:304] T 505580c93c694bbb8ea96fdf9b17ab46 P 7a9daf5d42a84615a6383cfec9a10608 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 7a9daf5d42a84615a6383cfec9a10608, 974b802141bd4f13beffef2aa03f88f9; no voters: 1c50de89a5a845668696dcb28c3badd6
I20260129 03:00:02.085175 20582 raft_consensus.cc:1240] T cb4727386a3e48aca5d71cc6c611c197 P 1c50de89a5a845668696dcb28c3badd6 [term 4 FOLLOWER]: Rejecting Update request from peer 197b880fa39d432a9b472845876bfaf0 for earlier term 3. Current term is 4. Ops: []
I20260129 03:00:02.034219 22878 raft_consensus.cc:515] T a0fcc58d6fb6473cafbd9d34d5195269 P 974b802141bd4f13beffef2aa03f88f9 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 03:00:02.091593 20588 raft_consensus.cc:2468] T 368953c9102643b2a21f99af36b51905 P 1c50de89a5a845668696dcb28c3badd6 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 197b880fa39d432a9b472845876bfaf0 in term 7.
I20260129 03:00:02.065121 20586 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c891d2bff4ce4568861bbc992e91fbb0" candidate_uuid: "197b880fa39d432a9b472845876bfaf0" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "1c50de89a5a845668696dcb28c3badd6"
I20260129 03:00:02.066403 20581 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "368953c9102643b2a21f99af36b51905" candidate_uuid: "974b802141bd4f13beffef2aa03f88f9" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "1c50de89a5a845668696dcb28c3badd6" is_pre_election: true
I20260129 03:00:02.044412 22231 raft_consensus.cc:3060] T 75be4bd6bd45474795fd472eff2953ac P 197b880fa39d432a9b472845876bfaf0 [term 9 FOLLOWER]: Advancing to term 10
I20260129 03:00:02.064095 23104 raft_consensus.cc:2804] T 45cc5f3d2688486d9b6dbe7e666dedde P 1c50de89a5a845668696dcb28c3badd6 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260129 03:00:02.086050 20582 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35754 (request call id 1730) took 1064 ms. Trace:
I20260129 03:00:02.046360 20584 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "45cc5f3d2688486d9b6dbe7e666dedde" candidate_uuid: "974b802141bd4f13beffef2aa03f88f9" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "1c50de89a5a845668696dcb28c3badd6"
W20260129 03:00:02.088227 20407 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260129 03:00:02.088980 20480 leader_election.cc:304] T e7d83d4ccf684e76acb29a0dc844e1b8 P 974b802141bd4f13beffef2aa03f88f9 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 974b802141bd4f13beffef2aa03f88f9; no voters: 197b880fa39d432a9b472845876bfaf0, 1c50de89a5a845668696dcb28c3badd6
I20260129 03:00:02.095506 20582 rpcz_store.cc:276] 0129 03:00:01.021489 (+ 0us) service_pool.cc:168] Inserting onto call queue
0129 03:00:02.084572 (+1063083us) service_pool.cc:225] Handling call
0129 03:00:02.084930 (+ 358us) raft_consensus.cc:1471] Updating replica for 0 ops
0129 03:00:02.085743 (+ 813us) raft_consensus.cc:1697] Filling consensus response to leader.
0129 03:00:02.085961 (+ 218us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260129 03:00:02.096655 20480 leader_election.cc:304] T 724a236524cb4e7ba3428880681d6cc2 P 974b802141bd4f13beffef2aa03f88f9 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1c50de89a5a845668696dcb28c3badd6, 974b802141bd4f13beffef2aa03f88f9; no voters:
I20260129 03:00:02.089449 21102 leader_election.cc:304] T 75b0b6e82ddc4f92b1fe1d94f9a0b73b P 7a9daf5d42a84615a6383cfec9a10608 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 7a9daf5d42a84615a6383cfec9a10608, 974b802141bd4f13beffef2aa03f88f9; no voters: 1c50de89a5a845668696dcb28c3badd6
I20260129 03:00:02.092396 20586 raft_consensus.cc:2376] T c891d2bff4ce4568861bbc992e91fbb0 P 1c50de89a5a845668696dcb28c3badd6 [term 7 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 197b880fa39d432a9b472845876bfaf0 in term 7. Re-sending same reply.
I20260129 03:00:02.067495 22986 raft_consensus.cc:493] T 913fc7d751ed4b7388d889aac30a5f9b P 974b802141bd4f13beffef2aa03f88f9 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260129 03:00:02.100291 20589 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1ce4ddf5b10e4ae8b193b13b2e5936ee" candidate_uuid: "197b880fa39d432a9b472845876bfaf0" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "1c50de89a5a845668696dcb28c3badd6"
I20260129 03:00:02.066772 23182 raft_consensus.cc:697] T cb0b3f259ebc4310927338a8740e87b8 P 7a9daf5d42a84615a6383cfec9a10608 [term 7 LEADER]: Becoming Leader. State: Replica: 7a9daf5d42a84615a6383cfec9a10608, State: Running, Role: LEADER
I20260129 03:00:02.069535 22851 leader_election.cc:290] T b189f08e16a74a11980cc55d99359a0a P 197b880fa39d432a9b472845876bfaf0 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 974b802141bd4f13beffef2aa03f88f9 (127.18.211.2:34517), 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3:37735)
I20260129 03:00:02.062201 23163 raft_consensus.cc:493] T 78aa39096868415d9afc2ffcb248e198 P 7a9daf5d42a84615a6383cfec9a10608 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 1c50de89a5a845668696dcb28c3badd6)
I20260129 03:00:02.082742 22694 raft_consensus.cc:493] T a644c02b16af47679824ccd060ac10fd P 974b802141bd4f13beffef2aa03f88f9 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260129 03:00:02.103515 23168 raft_consensus.cc:2804] T 75b0b6e82ddc4f92b1fe1d94f9a0b73b P 7a9daf5d42a84615a6383cfec9a10608 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260129 03:00:02.101454 20480 leader_election.cc:304] T 4f5a77e94fef4b16a2ba7b067c8b144a P 974b802141bd4f13beffef2aa03f88f9 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 974b802141bd4f13beffef2aa03f88f9; no voters: 1c50de89a5a845668696dcb28c3badd6, 7a9daf5d42a84615a6383cfec9a10608
I20260129 03:00:02.302681 23168 raft_consensus.cc:493] T 75b0b6e82ddc4f92b1fe1d94f9a0b73b P 7a9daf5d42a84615a6383cfec9a10608 [term 3 FOLLOWER]: Starting leader election (detected failure of leader 1c50de89a5a845668696dcb28c3badd6)
I20260129 03:00:02.303184 23168 raft_consensus.cc:3060] T 75b0b6e82ddc4f92b1fe1d94f9a0b73b P 7a9daf5d42a84615a6383cfec9a10608 [term 3 FOLLOWER]: Advancing to term 4
I20260129 03:00:02.103017 21135 raft_consensus.cc:1240] T 4e420f90884f4796997b5ea41e49e411 P 7a9daf5d42a84615a6383cfec9a10608 [term 6 LEADER]: Rejecting Update request from peer 974b802141bd4f13beffef2aa03f88f9 for earlier term 5. Current term is 6. Ops: []
I20260129 03:00:02.304919 20587 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35754 (request call id 1716) took 1361 ms. Trace:
I20260129 03:00:02.305372 20587 rpcz_store.cc:276] 0129 03:00:00.943478 (+ 0us) service_pool.cc:168] Inserting onto call queue
0129 03:00:00.953020 (+ 9542us) service_pool.cc:225] Handling call
0129 03:00:00.973802 (+ 20782us) raft_consensus.cc:1471] Updating replica for 0 ops
0129 03:00:01.454706 (+480904us) raft_consensus.cc:1697] Filling consensus response to leader.
0129 03:00:02.304620 (+849914us) spinlock_profiling.cc:243] Waited 282 ms on lock 0x7b54003b0e54. stack: 00007f8fe044e373 00007f8fe044e270 00007f8fdfda0574 00007f8fe9a5c6b4 00007f8fe9a5c649 00007f8fe34975a1 00007f8fe352ea81 00007f8fe352d4be 00007f8fe84e1f49 00007f8fe1a6ed34 00007f8fe1a6ecc2 00007f8fe1a6ebf1 00007f8fe1a6eb6c 00007f8fe1a6de22 00007f8fe186d34c 00007f8fe186c786
0129 03:00:02.304851 (+ 231us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":647830784}
I20260129 03:00:02.306504 21135 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fd8311b779af4e1fa1402b2a02a602f2" candidate_uuid: "1c50de89a5a845668696dcb28c3badd6" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "7a9daf5d42a84615a6383cfec9a10608"
I20260129 03:00:02.306612 20587 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4e3805ffa70242e8ade66da260dddf66" candidate_uuid: "197b880fa39d432a9b472845876bfaf0" candidate_term: 6 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "1c50de89a5a845668696dcb28c3badd6" is_pre_election: true
I20260129 03:00:02.307209 23053 raft_consensus.cc:3055] T 42d27a28d89b409e8807ec0efd97af69 P 1c50de89a5a845668696dcb28c3badd6 [term 6 LEADER]: Stepping down as leader of term 6
I20260129 03:00:02.082626 23185 raft_consensus.cc:493] T 96609ac68bdf4993a12e1c23355ca013 P 974b802141bd4f13beffef2aa03f88f9 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260129 03:00:02.101193 20589 raft_consensus.cc:2410] T 1ce4ddf5b10e4ae8b193b13b2e5936ee P 1c50de89a5a845668696dcb28c3badd6 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 197b880fa39d432a9b472845876bfaf0 for term 7 because replica has last-logged OpId of term: 6 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 5 index: 3.
W20260129 03:00:02.094105 20406 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260129 03:00:02.100750 23116 raft_consensus.cc:493] T 3ccac82a01a2445fa5b0ce6c0fcf2295 P 197b880fa39d432a9b472845876bfaf0 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 7a9daf5d42a84615a6383cfec9a10608)
W20260129 03:00:02.315121 20480 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.211.3:37735, user_credentials={real_user=slave}} blocked reactor thread for 213739us
I20260129 03:00:02.315268 23116 raft_consensus.cc:515] T 3ccac82a01a2445fa5b0ce6c0fcf2295 P 197b880fa39d432a9b472845876bfaf0 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } }
I20260129 03:00:02.308177 23185 raft_consensus.cc:515] T 96609ac68bdf4993a12e1c23355ca013 P 974b802141bd4f13beffef2aa03f88f9 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } attrs { replace: true } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } }
I20260129 03:00:02.316401 20480 leader_election.cc:304] T 505580c93c694bbb8ea96fdf9b17ab46 P 974b802141bd4f13beffef2aa03f88f9 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 974b802141bd4f13beffef2aa03f88f9; no voters: 1c50de89a5a845668696dcb28c3badd6, 7a9daf5d42a84615a6383cfec9a10608
I20260129 03:00:02.317410 23116 leader_election.cc:290] T 3ccac82a01a2445fa5b0ce6c0fcf2295 P 197b880fa39d432a9b472845876bfaf0 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 974b802141bd4f13beffef2aa03f88f9 (127.18.211.2:34517), 7a9daf5d42a84615a6383cfec9a10608 (127.18.211.4:40439)
I20260129 03:00:02.318058 20480 leader_election.cc:304] T cb4727386a3e48aca5d71cc6c611c197 P 974b802141bd4f13beffef2aa03f88f9 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 974b802141bd4f13beffef2aa03f88f9; no voters: 197b880fa39d432a9b472845876bfaf0, 1c50de89a5a845668696dcb28c3badd6
I20260129 03:00:02.318480 23185 leader_election.cc:290] T 96609ac68bdf4993a12e1c23355ca013 P 974b802141bd4f13beffef2aa03f88f9 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 197b880fa39d432a9b472845876bfaf0 (127.18.211.1:42027), 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3:37735), 7a9daf5d42a84615a6383cfec9a10608 (127.18.211.4:40439)
I20260129 03:00:02.319022 20589 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "17d6431cc77a491e8464d68db4081a51" candidate_uuid: "197b880fa39d432a9b472845876bfaf0" candidate_term: 5 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "1c50de89a5a845668696dcb28c3badd6" is_pre_election: true
I20260129 03:00:02.319270 22983 raft_consensus.cc:2749] T 4f5a77e94fef4b16a2ba7b067c8b144a P 974b802141bd4f13beffef2aa03f88f9 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20260129 03:00:02.104228 23152 raft_consensus.cc:3060] T e7d83d4ccf684e76acb29a0dc844e1b8 P 974b802141bd4f13beffef2aa03f88f9 [term 5 FOLLOWER]: Advancing to term 6
I20260129 03:00:02.319726 20589 raft_consensus.cc:2468] T 17d6431cc77a491e8464d68db4081a51 P 1c50de89a5a845668696dcb28c3badd6 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 197b880fa39d432a9b472845876bfaf0 in term 4.
I20260129 03:00:02.320258 20513 raft_consensus.cc:1275] T 96609ac68bdf4993a12e1c23355ca013 P 974b802141bd4f13beffef2aa03f88f9 [term 6 FOLLOWER]: Refusing update from remote peer 7a9daf5d42a84615a6383cfec9a10608: Log matching property violated. Preceding OpId in replica: term: 3 index: 5. Preceding OpId from leader: term: 6 index: 5. (term mismatch)
I20260129 03:00:02.320638 20513 pending_rounds.cc:85] T 96609ac68bdf4993a12e1c23355ca013 P 974b802141bd4f13beffef2aa03f88f9: Aborting all ops after (but not including) 4
I20260129 03:00:02.320854 20513 pending_rounds.cc:107] T 96609ac68bdf4993a12e1c23355ca013 P 974b802141bd4f13beffef2aa03f88f9: Aborting uncommitted NO_OP operation due to leader change: 3.5
I20260129 03:00:02.321079 20513 raft_consensus.cc:2889] T 96609ac68bdf4993a12e1c23355ca013 P 974b802141bd4f13beffef2aa03f88f9 [term 6 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
W20260129 03:00:02.154791 23084 log.cc:927] Time spent T 4f8f3a38f6e24ba4857875181f447442 P 197b880fa39d432a9b472845876bfaf0: Append to log took a long time: real 2.937s user 0.007s sys 0.010s
I20260129 03:00:02.322778 20439 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55672 (request call id 1635) took 4074 ms. Trace:
I20260129 03:00:02.323127 20439 rpcz_store.cc:276] 0129 02:59:58.248813 (+ 0us) service_pool.cc:168] Inserting onto call queue
0129 02:59:58.248981 (+ 168us) service_pool.cc:225] Handling call
0129 02:59:58.275591 (+ 26610us) raft_consensus.cc:1471] Updating replica for 1 ops
0129 02:59:58.282583 (+ 6992us) raft_consensus.cc:1534] Early marking committed up to index 0
0129 02:59:58.282624 (+ 41us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0129 02:59:58.363482 (+ 80858us) log.cc:844] Serialized 26 byte log entry
0129 02:59:58.592682 (+229200us) raft_consensus.cc:1645] Marking committed up to 3
0129 02:59:58.592729 (+ 47us) raft_consensus.cc:1697] Filling consensus response to leader.
0129 02:59:58.592795 (+ 66us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0129 03:00:02.064546 (+3471751us) spinlock_profiling.cc:243] Waited 134 ms on lock 0x7b440002304c. stack: 00007f8fe044e373 00007f8fe044e270 00007f8fdfda0574 00007f8fe9a5c6b4 00007f8fe9a5c649 00007f8fe02c08b1 00007f8fe038f005 00007f8fe038fa22 00007f8fe30d957b 00007f8fe30d4e4f 00007f8fe30c8c0e 00007f8fe30c7f56 00007f8fe30c7b78 00007f8fe30d15c1 00007f8fe30d1579 00007f8fe30d1509
0129 03:00:02.322495 (+257949us) raft_consensus.cc:1685] finished
0129 03:00:02.322522 (+ 27us) raft_consensus.cc:1691] UpdateReplicas() finished
0129 03:00:02.322715 (+ 193us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":1971,"raft.queue_time_us":24092,"raft.run_cpu_time_us":142,"raft.run_wall_time_us":33417,"spinlock_wait_cycles":308045568,"thread_start_us":147827,"threads_started":1,"wal-append.queue_time_us":463678}
I20260129 03:00:02.324249 23032 raft_consensus.cc:2749] T cb4727386a3e48aca5d71cc6c611c197 P 974b802141bd4f13beffef2aa03f88f9 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20260129 03:00:02.324767 23131 raft_consensus.cc:493] T 913fc7d751ed4b7388d889aac30a5f9b P 197b880fa39d432a9b472845876bfaf0 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260129 03:00:02.325271 23131 raft_consensus.cc:515] T 913fc7d751ed4b7388d889aac30a5f9b P 197b880fa39d432a9b472845876bfaf0 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 03:00:02.326304 22983 raft_consensus.cc:2749] T 505580c93c694bbb8ea96fdf9b17ab46 P 974b802141bd4f13beffef2aa03f88f9 [term 6 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260129 03:00:02.167567 20581 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "18f30ebaa4174fd28c958e72d0cc7e24" candidate_uuid: "7a9daf5d42a84615a6383cfec9a10608" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "1c50de89a5a845668696dcb28c3badd6" is_pre_election: true
W20260129 03:00:02.162132 22975 consensus_meta.cc:287] T 4b1876f9a70b49edb3aa74105e66d3b3 P 974b802141bd4f13beffef2aa03f88f9: Time spent flushing consensus metadata: real 0.593s user 0.009s sys 0.000s
I20260129 03:00:02.155092 23163 raft_consensus.cc:515] T 78aa39096868415d9afc2ffcb248e198 P 7a9daf5d42a84615a6383cfec9a10608 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } }
I20260129 03:00:02.087116 23184 raft_consensus.cc:3060] T add8d4923d1a489e86882c3a25692b95 P 7a9daf5d42a84615a6383cfec9a10608 [term 5 FOLLOWER]: Advancing to term 6
I20260129 03:00:02.087260 23043 raft_consensus.cc:493] T 42d27a28d89b409e8807ec0efd97af69 P 197b880fa39d432a9b472845876bfaf0 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 1c50de89a5a845668696dcb28c3badd6)
I20260129 03:00:02.328799 23043 raft_consensus.cc:515] T 42d27a28d89b409e8807ec0efd97af69 P 197b880fa39d432a9b472845876bfaf0 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } }
I20260129 03:00:02.329969 23163 leader_election.cc:290] T 78aa39096868415d9afc2ffcb248e198 P 7a9daf5d42a84615a6383cfec9a10608 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 197b880fa39d432a9b472845876bfaf0 (127.18.211.1:42027), 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3:37735)
I20260129 03:00:02.331053 23043 leader_election.cc:290] T 42d27a28d89b409e8807ec0efd97af69 P 197b880fa39d432a9b472845876bfaf0 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3:37735), 7a9daf5d42a84615a6383cfec9a10608 (127.18.211.4:40439)
I20260129 03:00:02.331713 23152 raft_consensus.cc:2749] T e7d83d4ccf684e76acb29a0dc844e1b8 P 974b802141bd4f13beffef2aa03f88f9 [term 6 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
W20260129 03:00:02.309767 20406 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260129 03:00:02.308830 20511 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:57584 (request call id 1412) took 1696 ms. Trace:
I20260129 03:00:02.333668 20511 rpcz_store.cc:276] 0129 03:00:00.612965 (+ 0us) service_pool.cc:168] Inserting onto call queue
0129 03:00:00.613100 (+ 135us) service_pool.cc:225] Handling call
0129 03:00:02.308618 (+1695518us) spinlock_profiling.cc:243] Waited 926 ms on lock 0x7b54002ef254. stack: 00007f8fe044e373 00007f8fe044e270 00007f8fdfda0574 00007f8fe9a5c6b4 00007f8fe9a5c649 00007f8fe34975a1 00007f8fe3531f7a 00007f8fe84e3647 00007f8fe1a73834 00007f8fe1a737c2 00007f8fe1a736f1 00007f8fe1a7366c 00007f8fe1a72922 00007f8fe186d34c 00007f8fe186c786 00007f8fe186c12f
0129 03:00:02.308774 (+ 156us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2128899968}
I20260129 03:00:02.158270 23175 raft_consensus.cc:515] T 3e2d15f8640b407195210c476e09df91 P 7a9daf5d42a84615a6383cfec9a10608 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } }
I20260129 03:00:02.094084 23183 raft_consensus.cc:2804] T 505580c93c694bbb8ea96fdf9b17ab46 P 7a9daf5d42a84615a6383cfec9a10608 [term 6 FOLLOWER]: Leader election won for term 6
I20260129 03:00:02.179654 20586 raft_consensus.cc:1240] T fd8311b779af4e1fa1402b2a02a602f2 P 1c50de89a5a845668696dcb28c3badd6 [term 5 FOLLOWER]: Rejecting Update request from peer 7a9daf5d42a84615a6383cfec9a10608 for earlier term 4. Current term is 5. Ops: []
I20260129 03:00:02.334944 23183 raft_consensus.cc:697] T 505580c93c694bbb8ea96fdf9b17ab46 P 7a9daf5d42a84615a6383cfec9a10608 [term 6 LEADER]: Becoming Leader. State: Replica: 7a9daf5d42a84615a6383cfec9a10608, State: Running, Role: LEADER
I20260129 03:00:02.335193 20586 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:48526 (request call id 329) took 1132 ms. Trace:
I20260129 03:00:02.335353 20511 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00e79d8f25a34676b1d866eb2068a2b2" candidate_uuid: "197b880fa39d432a9b472845876bfaf0" candidate_term: 8 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "974b802141bd4f13beffef2aa03f88f9" is_pre_election: true
I20260129 03:00:02.336244 20511 raft_consensus.cc:2468] T 00e79d8f25a34676b1d866eb2068a2b2 P 974b802141bd4f13beffef2aa03f88f9 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 197b880fa39d432a9b472845876bfaf0 in term 7.
I20260129 03:00:02.101574 23104 raft_consensus.cc:493] T 45cc5f3d2688486d9b6dbe7e666dedde P 1c50de89a5a845668696dcb28c3badd6 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260129 03:00:02.336958 23104 raft_consensus.cc:3060] T 45cc5f3d2688486d9b6dbe7e666dedde P 1c50de89a5a845668696dcb28c3badd6 [term 4 FOLLOWER]: Advancing to term 5
I20260129 03:00:02.121196 22967 raft_consensus.cc:2804] T 724a236524cb4e7ba3428880681d6cc2 P 974b802141bd4f13beffef2aa03f88f9 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260129 03:00:02.340252 22967 raft_consensus.cc:493] T 724a236524cb4e7ba3428880681d6cc2 P 974b802141bd4f13beffef2aa03f88f9 [term 4 FOLLOWER]: Starting leader election (detected failure of leader 1c50de89a5a845668696dcb28c3badd6)
I20260129 03:00:02.340651 22967 raft_consensus.cc:3060] T 724a236524cb4e7ba3428880681d6cc2 P 974b802141bd4f13beffef2aa03f88f9 [term 4 FOLLOWER]: Advancing to term 5
I20260129 03:00:02.342491 22231 raft_consensus.cc:515] T 75be4bd6bd45474795fd472eff2953ac P 197b880fa39d432a9b472845876bfaf0 [term 10 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 03:00:02.093237 22581 consensus_queue.cc:1059] T 42d27a28d89b409e8807ec0efd97af69 P 1c50de89a5a845668696dcb28c3badd6 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false }, Status: INVALID_TERM, Last received: 6.8, Next index: 9, Last known committed idx: 8, Time since last communication: 0.089s
I20260129 03:00:02.201179 20435 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55672 (request call id 1637) took 3762 ms. Trace:
I20260129 03:00:02.212788 23151 leader_election.cc:290] T 0340229d521b432580ae926f6062e3f7 P 974b802141bd4f13beffef2aa03f88f9 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 197b880fa39d432a9b472845876bfaf0 (127.18.211.1:42027), 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3:37735)
I20260129 03:00:02.220475 23182 consensus_queue.cc:237] T cb0b3f259ebc4310927338a8740e87b8 P 7a9daf5d42a84615a6383cfec9a10608 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 5.7, Last appended by leader: 7, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } }
I20260129 03:00:02.224987 22978 leader_election.cc:290] T add8d4923d1a489e86882c3a25692b95 P 974b802141bd4f13beffef2aa03f88f9 [CANDIDATE]: Term 6 election: Requested vote from peers 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3:37735), 7a9daf5d42a84615a6383cfec9a10608 (127.18.211.4:40439)
I20260129 03:00:02.185251 23042 leader_election.cc:290] T 18f30ebaa4174fd28c958e72d0cc7e24 P 197b880fa39d432a9b472845876bfaf0 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3:37735), 7a9daf5d42a84615a6383cfec9a10608 (127.18.211.4:40439)
I20260129 03:00:02.108081 22986 raft_consensus.cc:515] T 913fc7d751ed4b7388d889aac30a5f9b P 974b802141bd4f13beffef2aa03f88f9 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
W20260129 03:00:02.262313 21101 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260129 03:00:02.347395 21135 raft_consensus.cc:2468] T fd8311b779af4e1fa1402b2a02a602f2 P 7a9daf5d42a84615a6383cfec9a10608 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1c50de89a5a845668696dcb28c3badd6 in term 5.
W20260129 03:00:02.123371 20482 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260129 03:00:02.291357 20433 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "87d5fe48a44a408aaf03e894a67ecb2f" candidate_uuid: "7a9daf5d42a84615a6383cfec9a10608" candidate_term: 7 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "197b880fa39d432a9b472845876bfaf0" is_pre_election: true
I20260129 03:00:02.299665 23133 raft_consensus.cc:493] T 926a250fa5834e88b96ccecbefbf5086 P 197b880fa39d432a9b472845876bfaf0 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260129 03:00:02.300956 21131 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34046 (request call id 539) took 2915 ms (client timeout 1848 ms). Trace:
W20260129 03:00:02.352490 21131 rpcz_store.cc:269] 0129 02:59:59.385640 (+ 0us) service_pool.cc:168] Inserting onto call queue
0129 03:00:02.300694 (+2915054us) service_pool.cc:212] Skipping call since client already timed out
0129 03:00:02.300880 (+ 186us) inbound_call.cc:173] Queueing failure response
Metrics: {}
I20260129 03:00:02.355083 20512 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:48544 (request call id 261) took 1313 ms. Trace:
I20260129 03:00:02.104203 20582 raft_consensus.cc:1240] T 1299723a4476420ab02fd8bfd7aa0645 P 1c50de89a5a845668696dcb28c3badd6 [term 9 FOLLOWER]: Rejecting Update request from peer 7a9daf5d42a84615a6383cfec9a10608 for earlier term 8. Current term is 9. Ops: []
I20260129 03:00:02.179921 22694 raft_consensus.cc:515] T a644c02b16af47679824ccd060ac10fd P 974b802141bd4f13beffef2aa03f88f9 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } }
W20260129 03:00:02.356526 23160 log.cc:927] Time spent T 76f713c3331540d2a5b76164103e9eb1 P 974b802141bd4f13beffef2aa03f88f9: Append to log took a long time: real 1.221s user 0.008s sys 0.007s
I20260129 03:00:02.307605 23053 raft_consensus.cc:740] T 42d27a28d89b409e8807ec0efd97af69 P 1c50de89a5a845668696dcb28c3badd6 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 1c50de89a5a845668696dcb28c3badd6, State: Running, Role: LEADER
W20260129 03:00:02.312743 20479 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260129 03:00:02.313761 20507 raft_consensus.cc:2468] T 2ac659d732314e1288849f08fe5e074e P 974b802141bd4f13beffef2aa03f88f9 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7a9daf5d42a84615a6383cfec9a10608 in term 4.
I20260129 03:00:02.321197 20589 raft_consensus.cc:1275] T 4408006eb5174de1896b037cf3f18457 P 1c50de89a5a845668696dcb28c3badd6 [term 4 FOLLOWER]: Refusing update from remote peer 197b880fa39d432a9b472845876bfaf0: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20260129 03:00:02.335978 23183 consensus_queue.cc:237] T 505580c93c694bbb8ea96fdf9b17ab46 P 7a9daf5d42a84615a6383cfec9a10608 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 5.7, Last appended by leader: 7, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } }
I20260129 03:00:02.178638 21128 raft_consensus.cc:2393] T 45cc5f3d2688486d9b6dbe7e666dedde P 7a9daf5d42a84615a6383cfec9a10608 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 974b802141bd4f13beffef2aa03f88f9 in current term 5: Already voted for candidate 7a9daf5d42a84615a6383cfec9a10608 in this term.
I20260129 03:00:02.368999 20514 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57600 (request call id 1839) took 4030 ms. Trace:
I20260129 03:00:02.369505 20514 rpcz_store.cc:276] 0129 02:59:58.338227 (+ 0us) service_pool.cc:168] Inserting onto call queue
0129 02:59:58.338463 (+ 236us) service_pool.cc:225] Handling call
0129 02:59:58.411474 (+ 73011us) raft_consensus.cc:1471] Updating replica for 0 ops
0129 03:00:00.638875 (+2227401us) raft_consensus.cc:1697] Filling consensus response to leader.
0129 03:00:02.368709 (+1729834us) spinlock_profiling.cc:243] Waited 2.22 s on lock 0x7b5400168854. stack: 00007f8fe044e373 00007f8fe044e270 00007f8fdfda0574 00007f8fe9a5c6b4 00007f8fe9a5c649 00007f8fe34975a1 00007f8fe352ea81 00007f8fe352d4be 00007f8fe84e1f49 00007f8fe1a6ed34 00007f8fe1a6ecc2 00007f8fe1a6ebf1 00007f8fe1a6eb6c 00007f8fe1a6de22 00007f8fe186d34c 00007f8fe186c786
0129 03:00:02.368923 (+ 214us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":5114500480}
I20260129 03:00:02.342530 23175 leader_election.cc:290] T 3e2d15f8640b407195210c476e09df91 P 7a9daf5d42a84615a6383cfec9a10608 [CANDIDATE]: Term 4 election: Requested vote from peers 974b802141bd4f13beffef2aa03f88f9 (127.18.211.2:34517), 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3:37735)
I20260129 03:00:02.342657 20581 raft_consensus.cc:1240] T fcddab6bca684adf8ea032289a00f90e P 1c50de89a5a845668696dcb28c3badd6 [term 9 FOLLOWER]: Rejecting Update request from peer 974b802141bd4f13beffef2aa03f88f9 for earlier term 8. Current term is 9. Ops: [8.5-8.5]
I20260129 03:00:02.343461 20435 rpcz_store.cc:276] 0129 02:59:58.438306 (+ 0us) service_pool.cc:168] Inserting onto call queue
0129 02:59:58.438430 (+ 124us) service_pool.cc:225] Handling call
0129 02:59:58.459158 (+ 20728us) raft_consensus.cc:1471] Updating replica for 0 ops
0129 02:59:59.892551 (+1433393us) raft_consensus.cc:1697] Filling consensus response to leader.
0129 03:00:02.200872 (+2308321us) spinlock_profiling.cc:243] Waited 1.35 s on lock 0x7b5400265cd4. stack: 00007f8fe044e373 00007f8fe044e270 00007f8fdfda0574 00007f8fe9a5c6b4 00007f8fe9a5c649 00007f8fe34975a1 00007f8fe352ea81 00007f8fe352d4be 00007f8fe84e1f49 00007f8fe1a6ed34 00007f8fe1a6ecc2 00007f8fe1a6ebf1 00007f8fe1a6eb6c 00007f8fe1a6de22 00007f8fe186d34c 00007f8fe186c786
0129 03:00:02.201109 (+ 237us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":3108980096}
I20260129 03:00:02.086952 22728 consensus_queue.cc:1059] T 505580c93c694bbb8ea96fdf9b17ab46 P 1c50de89a5a845668696dcb28c3badd6 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false }, Status: INVALID_TERM, Last received: 5.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.088s
I20260129 03:00:02.376533 22728 raft_consensus.cc:3055] T 505580c93c694bbb8ea96fdf9b17ab46 P 1c50de89a5a845668696dcb28c3badd6 [term 5 LEADER]: Stepping down as leader of term 5
I20260129 03:00:02.335606 20586 rpcz_store.cc:276] 0129 03:00:01.202446 (+ 0us) service_pool.cc:168] Inserting onto call queue
0129 03:00:02.104438 (+901992us) service_pool.cc:225] Handling call
0129 03:00:02.179382 (+ 74944us) raft_consensus.cc:1471] Updating replica for 0 ops
0129 03:00:02.334916 (+155534us) raft_consensus.cc:1697] Filling consensus response to leader.
0129 03:00:02.335124 (+ 208us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260129 03:00:02.376966 22728 raft_consensus.cc:740] T 505580c93c694bbb8ea96fdf9b17ab46 P 1c50de89a5a845668696dcb28c3badd6 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 1c50de89a5a845668696dcb28c3badd6, State: Running, Role: LEADER
I20260129 03:00:02.377578 20586 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "78aa39096868415d9afc2ffcb248e198" candidate_uuid: "7a9daf5d42a84615a6383cfec9a10608" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "1c50de89a5a845668696dcb28c3badd6" is_pre_election: true
I20260129 03:00:02.377859 22728 consensus_queue.cc:260] T 505580c93c694bbb8ea96fdf9b17ab46 P 1c50de89a5a845668696dcb28c3badd6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 5.7, Last appended by leader: 7, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } }
I20260129 03:00:02.378809 20586 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0340229d521b432580ae926f6062e3f7" candidate_uuid: "974b802141bd4f13beffef2aa03f88f9" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "1c50de89a5a845668696dcb28c3badd6" is_pre_election: true
I20260129 03:00:02.379242 22728 raft_consensus.cc:3060] T 505580c93c694bbb8ea96fdf9b17ab46 P 1c50de89a5a845668696dcb28c3badd6 [term 5 FOLLOWER]: Advancing to term 6
I20260129 03:00:02.379417 20586 raft_consensus.cc:2468] T 0340229d521b432580ae926f6062e3f7 P 1c50de89a5a845668696dcb28c3badd6 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 974b802141bd4f13beffef2aa03f88f9 in term 6.
I20260129 03:00:02.355505 20512 rpcz_store.cc:276] 0129 03:00:01.041261 (+ 0us) service_pool.cc:168] Inserting onto call queue
0129 03:00:01.627055 (+585794us) service_pool.cc:225] Handling call
0129 03:00:01.651388 (+ 24333us) raft_consensus.cc:1471] Updating replica for 1 ops
0129 03:00:01.652785 (+ 1397us) raft_consensus.cc:1534] Early marking committed up to index 0
0129 03:00:01.652828 (+ 43us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0129 03:00:01.653437 (+ 609us) log.cc:844] Serialized 26 byte log entry
0129 03:00:02.061033 (+407596us) raft_consensus.cc:1645] Marking committed up to 4
0129 03:00:02.061075 (+ 42us) raft_consensus.cc:1697] Filling consensus response to leader.
0129 03:00:02.061147 (+ 72us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0129 03:00:02.354876 (+293729us) raft_consensus.cc:1685] finished
0129 03:00:02.354898 (+ 22us) raft_consensus.cc:1691] UpdateReplicas() finished
0129 03:00:02.355026 (+ 128us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":866,"raft.queue_time_us":1840,"raft.run_cpu_time_us":169,"raft.run_wall_time_us":6179,"thread_start_us":239189,"threads_started":1,"wal-append.queue_time_us":482637}
W20260129 03:00:02.362316 23041 consensus_meta.cc:287] T a609ac02fae84725ab743f95f51d33ce P 7a9daf5d42a84615a6383cfec9a10608: Time spent flushing consensus metadata: real 0.753s user 0.009s sys 0.003s
I20260129 03:00:02.357712 20511 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "17d6431cc77a491e8464d68db4081a51" candidate_uuid: "197b880fa39d432a9b472845876bfaf0" candidate_term: 5 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "974b802141bd4f13beffef2aa03f88f9" is_pre_election: true
I20260129 03:00:02.387305 20480 leader_election.cc:304] T 0340229d521b432580ae926f6062e3f7 P 974b802141bd4f13beffef2aa03f88f9 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1c50de89a5a845668696dcb28c3badd6, 974b802141bd4f13beffef2aa03f88f9; no voters:
I20260129 03:00:02.388571 20511 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b189f08e16a74a11980cc55d99359a0a" candidate_uuid: "197b880fa39d432a9b472845876bfaf0" candidate_term: 9 candidate_status { last_received { term: 8 index: 4 } } ignore_live_leader: false dest_uuid: "974b802141bd4f13beffef2aa03f88f9" is_pre_election: true
I20260129 03:00:02.389199 20511 raft_consensus.cc:2468] T b189f08e16a74a11980cc55d99359a0a P 974b802141bd4f13beffef2aa03f88f9 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 197b880fa39d432a9b472845876bfaf0 in term 8.
I20260129 03:00:02.389981 20511 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7a9a69cd82724b1faed1d10aa091fc4a" candidate_uuid: "197b880fa39d432a9b472845876bfaf0" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "974b802141bd4f13beffef2aa03f88f9" is_pre_election: true
I20260129 03:00:02.390480 20511 raft_consensus.cc:2468] T 7a9a69cd82724b1faed1d10aa091fc4a P 974b802141bd4f13beffef2aa03f88f9 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 197b880fa39d432a9b472845876bfaf0 in term 6.
I20260129 03:00:02.391625 20514 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "76f713c3331540d2a5b76164103e9eb1" candidate_uuid: "197b880fa39d432a9b472845876bfaf0" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "974b802141bd4f13beffef2aa03f88f9" is_pre_election: true
I20260129 03:00:02.362021 23055 raft_consensus.cc:3060] T 75f45538dc704d9a8ec4ff8ad9f3ea5b P 974b802141bd4f13beffef2aa03f88f9 [term 3 FOLLOWER]: Advancing to term 4
I20260129 03:00:02.374769 22694 leader_election.cc:290] T a644c02b16af47679824ccd060ac10fd P 974b802141bd4f13beffef2aa03f88f9 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3:37735), 7a9daf5d42a84615a6383cfec9a10608 (127.18.211.4:40439)
I20260129 03:00:02.394968 20434 raft_consensus.cc:1240] T 0f20af7fbe5d4db89a6f17948d98912c P 197b880fa39d432a9b472845876bfaf0 [term 7 FOLLOWER]: Rejecting Update request from peer 974b802141bd4f13beffef2aa03f88f9 for earlier term 6. Current term is 7. Ops: []
I20260129 03:00:02.375036 20510 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57584 (request call id 1400) took 2242 ms. Trace:
I20260129 03:00:02.376037 23053 consensus_queue.cc:260] T 42d27a28d89b409e8807ec0efd97af69 P 1c50de89a5a845668696dcb28c3badd6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 6.8, Last appended by leader: 8, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } }
W20260129 03:00:02.351658 20482 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260129 03:00:02.361186 23133 raft_consensus.cc:515] T 926a250fa5834e88b96ccecbefbf5086 P 197b880fa39d432a9b472845876bfaf0 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 03:00:02.381059 20516 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57600 (request call id 1852) took 1479 ms. Trace:
I20260129 03:00:02.376037 20439 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "42d27a28d89b409e8807ec0efd97af69" candidate_uuid: "7a9daf5d42a84615a6383cfec9a10608" candidate_term: 8 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "197b880fa39d432a9b472845876bfaf0" is_pre_election: true
I20260129 03:00:02.397657 20586 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b189f08e16a74a11980cc55d99359a0a" candidate_uuid: "197b880fa39d432a9b472845876bfaf0" candidate_term: 9 candidate_status { last_received { term: 8 index: 4 } } ignore_live_leader: false dest_uuid: "1c50de89a5a845668696dcb28c3badd6" is_pre_election: true
W20260129 03:00:02.399153 20482 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260129 03:00:02.400352 20512 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3e2d15f8640b407195210c476e09df91" candidate_uuid: "7a9daf5d42a84615a6383cfec9a10608" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "974b802141bd4f13beffef2aa03f88f9"
W20260129 03:00:02.388046 23118 consensus_meta.cc:287] T 0678b8b468bd43cdbc3e485900844821 P 197b880fa39d432a9b472845876bfaf0: Time spent flushing consensus metadata: real 0.540s user 0.011s sys 0.000s
I20260129 03:00:02.384227 20556 leader_election.cc:304] T e33a417cf9c54cf5af73a4ee3aac4d3a P 1c50de89a5a845668696dcb28c3badd6 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1c50de89a5a845668696dcb28c3badd6; no voters: 197b880fa39d432a9b472845876bfaf0, 974b802141bd4f13beffef2aa03f88f9
I20260129 03:00:02.104961 21134 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00e79d8f25a34676b1d866eb2068a2b2" candidate_uuid: "197b880fa39d432a9b472845876bfaf0" candidate_term: 8 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "7a9daf5d42a84615a6383cfec9a10608" is_pre_election: true
I20260129 03:00:02.307819 20587 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "76f713c3331540d2a5b76164103e9eb1" candidate_uuid: "197b880fa39d432a9b472845876bfaf0" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "1c50de89a5a845668696dcb28c3badd6" is_pre_election: true
I20260129 03:00:02.384959 20589 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3e2d15f8640b407195210c476e09df91" candidate_uuid: "7a9daf5d42a84615a6383cfec9a10608" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "1c50de89a5a845668696dcb28c3badd6"
I20260129 03:00:02.407009 23026 raft_consensus.cc:2749] T e33a417cf9c54cf5af73a4ee3aac4d3a P 1c50de89a5a845668696dcb28c3badd6 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260129 03:00:02.357890 20582 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:48526 (request call id 328) took 1249 ms. Trace:
I20260129 03:00:02.407269 20587 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "18f30ebaa4174fd28c958e72d0cc7e24" candidate_uuid: "197b880fa39d432a9b472845876bfaf0" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "1c50de89a5a845668696dcb28c3badd6" is_pre_election: true
I20260129 03:00:02.391913 20511 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "45cc5f3d2688486d9b6dbe7e666dedde" candidate_uuid: "7a9daf5d42a84615a6383cfec9a10608" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "974b802141bd4f13beffef2aa03f88f9"
I20260129 03:00:02.407827 20582 rpcz_store.cc:276] 0129 03:00:01.108793 (+ 0us) service_pool.cc:168] Inserting onto call queue
0129 03:00:02.103679 (+994886us) service_pool.cc:225] Handling call
0129 03:00:02.104041 (+ 362us) raft_consensus.cc:1471] Updating replica for 0 ops
0129 03:00:02.357596 (+253555us) raft_consensus.cc:1697] Filling consensus response to leader.
0129 03:00:02.357835 (+ 239us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260129 03:00:02.392995 20554 leader_election.cc:304] T fd8311b779af4e1fa1402b2a02a602f2 P 1c50de89a5a845668696dcb28c3badd6 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1c50de89a5a845668696dcb28c3badd6, 7a9daf5d42a84615a6383cfec9a10608; no voters: 197b880fa39d432a9b472845876bfaf0
I20260129 03:00:02.394132 20440 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2ac659d732314e1288849f08fe5e074e" candidate_uuid: "7a9daf5d42a84615a6383cfec9a10608" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "197b880fa39d432a9b472845876bfaf0" is_pre_election: true
I20260129 03:00:02.408283 20589 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "96609ac68bdf4993a12e1c23355ca013" candidate_uuid: "974b802141bd4f13beffef2aa03f88f9" candidate_term: 7 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "1c50de89a5a845668696dcb28c3badd6" is_pre_election: true
I20260129 03:00:02.408892 23026 raft_consensus.cc:2804] T fd8311b779af4e1fa1402b2a02a602f2 P 1c50de89a5a845668696dcb28c3badd6 [term 5 FOLLOWER]: Leader election won for term 5
I20260129 03:00:02.409461 20582 raft_consensus.cc:1275] T 17d6431cc77a491e8464d68db4081a51 P 1c50de89a5a845668696dcb28c3badd6 [term 4 FOLLOWER]: Refusing update from remote peer 974b802141bd4f13beffef2aa03f88f9: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 4 index: 2. (index mismatch)
I20260129 03:00:02.410440 20441 raft_consensus.cc:1240] T c891d2bff4ce4568861bbc992e91fbb0 P 197b880fa39d432a9b472845876bfaf0 [term 7 FOLLOWER]: Rejecting Update request from peer 974b802141bd4f13beffef2aa03f88f9 for earlier term 6. Current term is 7. Ops: []
I20260129 03:00:02.408382 20511 raft_consensus.cc:2393] T 45cc5f3d2688486d9b6dbe7e666dedde P 974b802141bd4f13beffef2aa03f88f9 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 7a9daf5d42a84615a6383cfec9a10608 in current term 5: Already voted for candidate 974b802141bd4f13beffef2aa03f88f9 in this term.
I20260129 03:00:02.359750 21135 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2ac659d732314e1288849f08fe5e074e" candidate_uuid: "197b880fa39d432a9b472845876bfaf0" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "7a9daf5d42a84615a6383cfec9a10608" is_pre_election: true
I20260129 03:00:02.408540 20587 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "add8d4923d1a489e86882c3a25692b95" candidate_uuid: "974b802141bd4f13beffef2aa03f88f9" candidate_term: 6 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "1c50de89a5a845668696dcb28c3badd6"
I20260129 03:00:02.395569 20510 rpcz_store.cc:276] 0129 03:00:00.132605 (+ 0us) service_pool.cc:168] Inserting onto call queue
0129 03:00:00.132777 (+ 172us) service_pool.cc:225] Handling call
0129 03:00:00.142435 (+ 9658us) raft_consensus.cc:1471] Updating replica for 1 ops
0129 03:00:00.601198 (+458763us) raft_consensus.cc:1534] Early marking committed up to index 0
0129 03:00:00.601247 (+ 49us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0129 03:00:00.631732 (+ 30485us) log.cc:844] Serialized 26 byte log entry
0129 03:00:00.842352 (+210620us) raft_consensus.cc:1645] Marking committed up to 2
0129 03:00:00.842397 (+ 45us) raft_consensus.cc:1697] Filling consensus response to leader.
0129 03:00:00.842490 (+ 93us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0129 03:00:02.355988 (+1513498us) spinlock_profiling.cc:243] Waited 2.22 ms on lock 0x7b440002304c. stack: 00007f8fe044e373 00007f8fe044e270 00007f8fdfda0574 00007f8fe9a5c6b4 00007f8fe9a5c649 00007f8fe02c08b1 00007f8fe038f005 00007f8fe038fa22 00007f8fe30d957b 00007f8fe30d4e4f 00007f8fe30c8c0e 00007f8fe30c7f56 00007f8fe30c7b78 00007f8fe30d15c1 00007f8fe30d1579 00007f8fe30d1509
0129 03:00:02.374727 (+ 18739us) raft_consensus.cc:1685] finished
0129 03:00:02.374752 (+ 25us) raft_consensus.cc:1691] UpdateReplicas() finished
0129 03:00:02.374971 (+ 219us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":465732,"raft.run_cpu_time_us":100,"raft.run_wall_time_us":100,"spinlock_wait_cycles":5111680,"thread_start_us":668449,"threads_started":2,"wal-append.queue_time_us":430256}
I20260129 03:00:02.400925 20512 raft_consensus.cc:3060] T 3e2d15f8640b407195210c476e09df91 P 974b802141bd4f13beffef2aa03f88f9 [term 3 FOLLOWER]: Advancing to term 4
I20260129 03:00:02.401386 20435 raft_consensus.cc:1240] T a0fcc58d6fb6473cafbd9d34d5195269 P 197b880fa39d432a9b472845876bfaf0 [term 4 FOLLOWER]: Rejecting Update request from peer 974b802141bd4f13beffef2aa03f88f9 for earlier term 3. Current term is 4. Ops: []
I20260129 03:00:02.398682 20439 raft_consensus.cc:2468] T 42d27a28d89b409e8807ec0efd97af69 P 197b880fa39d432a9b472845876bfaf0 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7a9daf5d42a84615a6383cfec9a10608 in term 6.
W20260129 03:00:02.394654 20405 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.211.3:37735, user_credentials={real_user=slave}} blocked reactor thread for 292774us
I20260129 03:00:02.401782 20438 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4e3805ffa70242e8ade66da260dddf66" candidate_uuid: "7a9daf5d42a84615a6383cfec9a10608" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "197b880fa39d432a9b472845876bfaf0" is_pre_election: true
I20260129 03:00:02.397769 20581 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7a9a69cd82724b1faed1d10aa091fc4a" candidate_uuid: "197b880fa39d432a9b472845876bfaf0" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "1c50de89a5a845668696dcb28c3badd6" is_pre_election: true
I20260129 03:00:02.403044 20434 raft_consensus.cc:1240] T 4b1876f9a70b49edb3aa74105e66d3b3 P 197b880fa39d432a9b472845876bfaf0 [term 5 FOLLOWER]: Rejecting Update request from peer 974b802141bd4f13beffef2aa03f88f9 for earlier term 4. Current term is 5. Ops: []
I20260129 03:00:02.409099 20440 raft_consensus.cc:2468] T 2ac659d732314e1288849f08fe5e074e P 197b880fa39d432a9b472845876bfaf0 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7a9daf5d42a84615a6383cfec9a10608 in term 3.
I20260129 03:00:02.403124 20516 rpcz_store.cc:276] 0129 03:00:00.901726 (+ 0us) service_pool.cc:168] Inserting onto call queue
0129 03:00:00.986210 (+ 84484us) service_pool.cc:225] Handling call
0129 03:00:00.992157 (+ 5947us) raft_consensus.cc:1471] Updating replica for 0 ops
0129 03:00:00.992369 (+ 212us) raft_consensus.cc:1534] Early marking committed up to index 3
0129 03:00:01.143204 (+150835us) log.cc:844] Serialized 14 byte log entry
0129 03:00:01.686791 (+543587us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0129 03:00:01.686873 (+ 82us) raft_consensus.cc:1645] Marking committed up to 3
0129 03:00:01.686949 (+ 76us) raft_consensus.cc:1697] Filling consensus response to leader.
0129 03:00:01.687050 (+ 101us) raft_consensus.cc:1691] UpdateReplicas() finished
0129 03:00:02.380891 (+693841us) spinlock_profiling.cc:243] Waited 5.65 ms on lock 0x7b540028f350. stack: 00007f8fe044e373 00007f8fe044e270 00007f8fdfda0574 00007f8fe9a5c6b4 00007f8fe9a5c649 00007f8fe34975a1 00007f8fe352d52d 00007f8fe84e1f49 00007f8fe1a6ed34 00007f8fe1a6ecc2 00007f8fe1a6ebf1 00007f8fe1a6eb6c 00007f8fe1a6de22 00007f8fe186d34c 00007f8fe186c786 00007f8fe186c12f
0129 03:00:02.380999 (+ 108us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":25822,"prepare.run_cpu_time_us":37,"prepare.run_wall_time_us":37,"raft.queue_time_us":11257,"raft.run_cpu_time_us":135,"raft.run_wall_time_us":7338,"spinlock_wait_cycles":12996480,"thread_start_us":680958,"threads_started":3,"wal-append.queue_time_us":598166}
I20260129 03:00:02.376530 22986 leader_election.cc:290] T 913fc7d751ed4b7388d889aac30a5f9b P 974b802141bd4f13beffef2aa03f88f9 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 197b880fa39d432a9b472845876bfaf0 (127.18.211.1:42027), 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3:37735)
I20260129 03:00:02.406548 21134 raft_consensus.cc:2468] T 00e79d8f25a34676b1d866eb2068a2b2 P 7a9daf5d42a84615a6383cfec9a10608 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 197b880fa39d432a9b472845876bfaf0 in term 7.
I20260129 03:00:02.407095 20556 leader_election.cc:304] T 4b1876f9a70b49edb3aa74105e66d3b3 P 1c50de89a5a845668696dcb28c3badd6 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1c50de89a5a845668696dcb28c3badd6; no voters: 197b880fa39d432a9b472845876bfaf0, 974b802141bd4f13beffef2aa03f88f9
I20260129 03:00:02.413139 20438 raft_consensus.cc:2468] T 4e3805ffa70242e8ade66da260dddf66 P 197b880fa39d432a9b472845876bfaf0 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7a9daf5d42a84615a6383cfec9a10608 in term 5.
I20260129 03:00:02.386965 21131 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "add8d4923d1a489e86882c3a25692b95" candidate_uuid: "974b802141bd4f13beffef2aa03f88f9" candidate_term: 6 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "7a9daf5d42a84615a6383cfec9a10608"
I20260129 03:00:02.412314 21135 raft_consensus.cc:2393] T 2ac659d732314e1288849f08fe5e074e P 7a9daf5d42a84615a6383cfec9a10608 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 197b880fa39d432a9b472845876bfaf0 in current term 4: Already voted for candidate 7a9daf5d42a84615a6383cfec9a10608 in this term.
I20260129 03:00:02.384023 22953 leader_election.cc:290] T 0340229d521b432580ae926f6062e3f7 P 1c50de89a5a845668696dcb28c3badd6 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 197b880fa39d432a9b472845876bfaf0 (127.18.211.1:42027), 974b802141bd4f13beffef2aa03f88f9 (127.18.211.2:34517)
I20260129 03:00:02.404835 20514 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2ac659d732314e1288849f08fe5e074e" candidate_uuid: "197b880fa39d432a9b472845876bfaf0" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "974b802141bd4f13beffef2aa03f88f9" is_pre_election: true
I20260129 03:00:02.452216 20510 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57600 (request call id 1860) took 1402 ms. Trace:
I20260129 03:00:02.452713 20510 rpcz_store.cc:276] 0129 03:00:01.050078 (+ 0us) service_pool.cc:168] Inserting onto call queue
0129 03:00:02.418695 (+1368617us) service_pool.cc:225] Handling call
0129 03:00:02.451670 (+ 32975us) raft_consensus.cc:1471] Updating replica for 0 ops
0129 03:00:02.451855 (+ 185us) raft_consensus.cc:1534] Early marking committed up to index 0
0129 03:00:02.451911 (+ 56us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0129 03:00:02.451965 (+ 54us) raft_consensus.cc:1645] Marking committed up to 7
0129 03:00:02.451998 (+ 33us) raft_consensus.cc:1697] Filling consensus response to leader.
0129 03:00:02.452055 (+ 57us) raft_consensus.cc:1691] UpdateReplicas() finished
0129 03:00:02.452156 (+ 101us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20260129 03:00:02.454530 21102 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260129 03:00:02.394861 23080 leader_election.cc:290] T 45cc5f3d2688486d9b6dbe7e666dedde P 7a9daf5d42a84615a6383cfec9a10608 [CANDIDATE]: Term 5 election: Requested vote from peers 974b802141bd4f13beffef2aa03f88f9 (127.18.211.2:34517), 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3:37735)
I20260129 03:00:02.546048 20581 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "913fc7d751ed4b7388d889aac30a5f9b" candidate_uuid: "974b802141bd4f13beffef2aa03f88f9" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "1c50de89a5a845668696dcb28c3badd6" is_pre_election: true
I20260129 03:00:02.489252 20516 raft_consensus.cc:1275] T 1ce4ddf5b10e4ae8b193b13b2e5936ee P 974b802141bd4f13beffef2aa03f88f9 [term 7 FOLLOWER]: Refusing update from remote peer 197b880fa39d432a9b472845876bfaf0: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 5 index: 3. (index mismatch)
I20260129 03:00:02.489806 20586 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "45cc5f3d2688486d9b6dbe7e666dedde" candidate_uuid: "7a9daf5d42a84615a6383cfec9a10608" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "1c50de89a5a845668696dcb28c3badd6"
I20260129 03:00:02.547019 20581 raft_consensus.cc:2468] T 913fc7d751ed4b7388d889aac30a5f9b P 1c50de89a5a845668696dcb28c3badd6 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 974b802141bd4f13beffef2aa03f88f9 in term 4.
I20260129 03:00:02.409466 23026 raft_consensus.cc:697] T fd8311b779af4e1fa1402b2a02a602f2 P 1c50de89a5a845668696dcb28c3badd6 [term 5 LEADER]: Becoming Leader. State: Replica: 1c50de89a5a845668696dcb28c3badd6, State: Running, Role: LEADER
I20260129 03:00:02.393622 23041 raft_consensus.cc:515] T a609ac02fae84725ab743f95f51d33ce P 7a9daf5d42a84615a6383cfec9a10608 [term 8 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } }
I20260129 03:00:02.495776 20407 leader_election.cc:304] T 00e79d8f25a34676b1d866eb2068a2b2 P 197b880fa39d432a9b472845876bfaf0 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 197b880fa39d432a9b472845876bfaf0, 974b802141bd4f13beffef2aa03f88f9; no voters:
I20260129 03:00:02.510274 20440 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "368953c9102643b2a21f99af36b51905" candidate_uuid: "974b802141bd4f13beffef2aa03f88f9" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "197b880fa39d432a9b472845876bfaf0" is_pre_election: true
I20260129 03:00:02.575501 20440 raft_consensus.cc:2393] T 368953c9102643b2a21f99af36b51905 P 197b880fa39d432a9b472845876bfaf0 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 974b802141bd4f13beffef2aa03f88f9 in current term 7: Already voted for candidate 197b880fa39d432a9b472845876bfaf0 in this term.
I20260129 03:00:02.515089 21135 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "96609ac68bdf4993a12e1c23355ca013" candidate_uuid: "974b802141bd4f13beffef2aa03f88f9" candidate_term: 7 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "7a9daf5d42a84615a6383cfec9a10608" is_pre_election: true
I20260129 03:00:02.508208 20441 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "18f30ebaa4174fd28c958e72d0cc7e24" candidate_uuid: "7a9daf5d42a84615a6383cfec9a10608" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "197b880fa39d432a9b472845876bfaf0" is_pre_election: true
I20260129 03:00:02.576726 20441 raft_consensus.cc:2468] T 18f30ebaa4174fd28c958e72d0cc7e24 P 197b880fa39d432a9b472845876bfaf0 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7a9daf5d42a84615a6383cfec9a10608 in term 6.
W20260129 03:00:02.515959 23112 consensus_meta.cc:287] T febffe4a8b394368b69b1d164308fb9a P 197b880fa39d432a9b472845876bfaf0: Time spent flushing consensus metadata: real 0.625s user 0.011s sys 0.002s
I20260129 03:00:02.514302 21134 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a644c02b16af47679824ccd060ac10fd" candidate_uuid: "974b802141bd4f13beffef2aa03f88f9" candidate_term: 6 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "7a9daf5d42a84615a6383cfec9a10608" is_pre_election: true
W20260129 03:00:02.506049 20556 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.211.2:34517, user_credentials={real_user=slave}} blocked reactor thread for 99019.8us
W20260129 03:00:02.508657 20405 leader_election.cc:336] T 368953c9102643b2a21f99af36b51905 P 197b880fa39d432a9b472845876bfaf0 [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3:37735): Timed out: RequestConsensusVote RPC to 127.18.211.3:37735 timed out after 1.512s (SENT)
I20260129 03:00:02.542871 23133 leader_election.cc:290] T 926a250fa5834e88b96ccecbefbf5086 P 197b880fa39d432a9b472845876bfaf0 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 974b802141bd4f13beffef2aa03f88f9 (127.18.211.2:34517), 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3:37735)
W20260129 03:00:02.590955 20407 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.211.2:34517, user_credentials={real_user=slave}} blocked reactor thread for 95316.4us
I20260129 03:00:02.510183 20435 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c2633a6fc34c44579de348d208b0c01e" candidate_uuid: "974b802141bd4f13beffef2aa03f88f9" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "197b880fa39d432a9b472845876bfaf0" is_pre_election: true
I20260129 03:00:02.397321 22983 raft_consensus.cc:2804] T 0340229d521b432580ae926f6062e3f7 P 974b802141bd4f13beffef2aa03f88f9 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260129 03:00:02.393635 23032 consensus_queue.cc:1059] T fcddab6bca684adf8ea032289a00f90e P 974b802141bd4f13beffef2aa03f88f9 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 }, Status: INVALID_TERM, Last received: 8.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.096s
I20260129 03:00:02.548596 20438 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "cedce331e1584ac8b636c9b8a72b2ba0" candidate_uuid: "974b802141bd4f13beffef2aa03f88f9" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "197b880fa39d432a9b472845876bfaf0"
I20260129 03:00:02.556893 20434 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "724a236524cb4e7ba3428880681d6cc2" candidate_uuid: "974b802141bd4f13beffef2aa03f88f9" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "197b880fa39d432a9b472845876bfaf0" is_pre_election: true
I20260129 03:00:02.558221 21100 leader_election.cc:304] T 45cc5f3d2688486d9b6dbe7e666dedde P 7a9daf5d42a84615a6383cfec9a10608 [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7a9daf5d42a84615a6383cfec9a10608; no voters: 1c50de89a5a845668696dcb28c3badd6, 974b802141bd4f13beffef2aa03f88f9
I20260129 03:00:02.368237 22978 consensus_queue.cc:1059] T 4e420f90884f4796997b5ea41e49e411 P 974b802141bd4f13beffef2aa03f88f9 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false }, Status: INVALID_TERM, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.039s
I20260129 03:00:02.563071 20437 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "78aa39096868415d9afc2ffcb248e198" candidate_uuid: "7a9daf5d42a84615a6383cfec9a10608" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "197b880fa39d432a9b472845876bfaf0" is_pre_election: true
I20260129 03:00:02.553217 22231 leader_election.cc:290] T 75be4bd6bd45474795fd472eff2953ac P 197b880fa39d432a9b472845876bfaf0 [CANDIDATE]: Term 10 election: Requested vote from peers 974b802141bd4f13beffef2aa03f88f9 (127.18.211.2:34517), 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3:37735)
I20260129 03:00:02.508848 20587 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a644c02b16af47679824ccd060ac10fd" candidate_uuid: "974b802141bd4f13beffef2aa03f88f9" candidate_term: 6 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "1c50de89a5a845668696dcb28c3badd6" is_pre_election: true
W20260129 03:00:02.577715 20482 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260129 03:00:02.583554 21135 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "18f30ebaa4174fd28c958e72d0cc7e24" candidate_uuid: "197b880fa39d432a9b472845876bfaf0" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "7a9daf5d42a84615a6383cfec9a10608" is_pre_election: true
I20260129 03:00:02.585930 20556 leader_election.cc:304] T a4c1226b987b468fa84dfc6ba0e2dbe1 P 1c50de89a5a845668696dcb28c3badd6 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1c50de89a5a845668696dcb28c3badd6; no voters: 197b880fa39d432a9b472845876bfaf0, 974b802141bd4f13beffef2aa03f88f9
W20260129 03:00:02.586097 20405 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.211.3:37735, user_credentials={real_user=slave}} blocked reactor thread for 77474.5us
I20260129 03:00:02.605940 21135 raft_consensus.cc:2468] T 18f30ebaa4174fd28c958e72d0cc7e24 P 7a9daf5d42a84615a6383cfec9a10608 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 197b880fa39d432a9b472845876bfaf0 in term 6.
I20260129 03:00:02.586817 20581 tablet_service.cc:2066] Received LeaderStepDown RPC: tablet_id: "76f713c3331540d2a5b76164103e9eb1"
dest_uuid: "1c50de89a5a845668696dcb28c3badd6"
mode: GRACEFUL
new_leader_uuid: "197b880fa39d432a9b472845876bfaf0"
from {username='slave'} at 127.0.0.1:54860
I20260129 03:00:02.606279 21132 raft_consensus.cc:1240] T a609ac02fae84725ab743f95f51d33ce P 7a9daf5d42a84615a6383cfec9a10608 [term 8 FOLLOWER]: Rejecting Update request from peer 974b802141bd4f13beffef2aa03f88f9 for earlier term 6. Current term is 8. Ops: []
I20260129 03:00:02.606575 20581 raft_consensus.cc:606] T 76f713c3331540d2a5b76164103e9eb1 P 1c50de89a5a845668696dcb28c3badd6 [term 5 LEADER]: Received request to transfer leadership to TS 197b880fa39d432a9b472845876bfaf0
W20260129 03:00:02.606317 20405 leader_election.cc:336] T dc2f4cb752cf4d59b5f6bc689f5d8104 P 197b880fa39d432a9b472845876bfaf0 [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3:37735): Timed out: RequestConsensusVote RPC to 127.18.211.3:37735 timed out after 1.523s (SENT)
I20260129 03:00:02.607668 20405 leader_election.cc:304] T dc2f4cb752cf4d59b5f6bc689f5d8104 P 197b880fa39d432a9b472845876bfaf0 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 197b880fa39d432a9b472845876bfaf0; no voters: 1c50de89a5a845668696dcb28c3badd6, 7a9daf5d42a84615a6383cfec9a10608
I20260129 03:00:02.588101 20480 leader_election.cc:304] T 913fc7d751ed4b7388d889aac30a5f9b P 974b802141bd4f13beffef2aa03f88f9 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1c50de89a5a845668696dcb28c3badd6, 974b802141bd4f13beffef2aa03f88f9; no voters:
I20260129 03:00:02.590690 20512 raft_consensus.cc:2468] T 3e2d15f8640b407195210c476e09df91 P 974b802141bd4f13beffef2aa03f88f9 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7a9daf5d42a84615a6383cfec9a10608 in term 4.
I20260129 03:00:02.512287 23146 consensus_queue.cc:1048] T 17d6431cc77a491e8464d68db4081a51 P 974b802141bd4f13beffef2aa03f88f9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.031s
I20260129 03:00:02.591941 20435 raft_consensus.cc:2393] T c2633a6fc34c44579de348d208b0c01e P 197b880fa39d432a9b472845876bfaf0 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 974b802141bd4f13beffef2aa03f88f9 in current term 3: Already voted for candidate 197b880fa39d432a9b472845876bfaf0 in this term.
I20260129 03:00:02.510914 23053 raft_consensus.cc:3060] T 42d27a28d89b409e8807ec0efd97af69 P 1c50de89a5a845668696dcb28c3badd6 [term 6 FOLLOWER]: Advancing to term 7
I20260129 03:00:02.507725 22924 raft_consensus.cc:2749] T 4b1876f9a70b49edb3aa74105e66d3b3 P 1c50de89a5a845668696dcb28c3badd6 [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260129 03:00:02.427649 23182 consensus_queue.cc:1059] T 1299723a4476420ab02fd8bfd7aa0645 P 7a9daf5d42a84615a6383cfec9a10608 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 }, Status: INVALID_TERM, Last received: 8.8, Next index: 9, Last known committed idx: 8, Time since last communication: 0.142s
I20260129 03:00:02.593786 20586 raft_consensus.cc:1217] T 75f45538dc704d9a8ec4ff8ad9f3ea5b P 1c50de89a5a845668696dcb28c3badd6 [term 4 FOLLOWER]: Deduplicated request from leader. Original: 2.5->[3.6-3.6] Dedup: 3.6->[]
I20260129 03:00:02.613976 20516 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3ccac82a01a2445fa5b0ce6c0fcf2295" candidate_uuid: "197b880fa39d432a9b472845876bfaf0" candidate_term: 5 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "974b802141bd4f13beffef2aa03f88f9" is_pre_election: true
I20260129 03:00:02.415768 23118 raft_consensus.cc:515] T 0678b8b468bd43cdbc3e485900844821 P 197b880fa39d432a9b472845876bfaf0 [term 8 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } }
I20260129 03:00:02.599596 20437 raft_consensus.cc:2468] T 78aa39096868415d9afc2ffcb248e198 P 197b880fa39d432a9b472845876bfaf0 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7a9daf5d42a84615a6383cfec9a10608 in term 5.
I20260129 03:00:02.571688 21099 leader_election.cc:304] T 42d27a28d89b409e8807ec0efd97af69 P 7a9daf5d42a84615a6383cfec9a10608 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 197b880fa39d432a9b472845876bfaf0, 7a9daf5d42a84615a6383cfec9a10608; no voters: 1c50de89a5a845668696dcb28c3badd6
I20260129 03:00:02.600391 20407 leader_election.cc:304] T b189f08e16a74a11980cc55d99359a0a P 197b880fa39d432a9b472845876bfaf0 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 197b880fa39d432a9b472845876bfaf0, 974b802141bd4f13beffef2aa03f88f9; no voters:
I20260129 03:00:02.601809 20440 raft_consensus.cc:1240] T 926a250fa5834e88b96ccecbefbf5086 P 197b880fa39d432a9b472845876bfaf0 [term 7 FOLLOWER]: Rejecting Update request from peer 974b802141bd4f13beffef2aa03f88f9 for earlier term 6. Current term is 7. Ops: []
I20260129 03:00:02.626595 20440 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0340229d521b432580ae926f6062e3f7" candidate_uuid: "974b802141bd4f13beffef2aa03f88f9" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "197b880fa39d432a9b472845876bfaf0" is_pre_election: true
I20260129 03:00:02.627439 20440 raft_consensus.cc:2468] T 0340229d521b432580ae926f6062e3f7 P 197b880fa39d432a9b472845876bfaf0 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 974b802141bd4f13beffef2aa03f88f9 in term 5.
W20260129 03:00:02.608896 20555 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260129 03:00:02.597656 23039 raft_consensus.cc:2804] T 00e79d8f25a34676b1d866eb2068a2b2 P 197b880fa39d432a9b472845876bfaf0 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260129 03:00:02.614142 20586 raft_consensus.cc:1240] T 75f45538dc704d9a8ec4ff8ad9f3ea5b P 1c50de89a5a845668696dcb28c3badd6 [term 4 FOLLOWER]: Rejecting Update request from peer 974b802141bd4f13beffef2aa03f88f9 for earlier term 3. Current term is 4. Ops: [3.6-3.6]
I20260129 03:00:02.595777 20434 raft_consensus.cc:2393] T 724a236524cb4e7ba3428880681d6cc2 P 197b880fa39d432a9b472845876bfaf0 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 974b802141bd4f13beffef2aa03f88f9 in current term 5: Already voted for candidate 197b880fa39d432a9b472845876bfaf0 in this term.
I20260129 03:00:02.614487 20435 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a1a39dcd328e4268ae41b988c66b4a75" candidate_uuid: "974b802141bd4f13beffef2aa03f88f9" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "197b880fa39d432a9b472845876bfaf0" is_pre_election: true
I20260129 03:00:02.647250 20435 raft_consensus.cc:2468] T a1a39dcd328e4268ae41b988c66b4a75 P 197b880fa39d432a9b472845876bfaf0 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 974b802141bd4f13beffef2aa03f88f9 in term 4.
I20260129 03:00:02.647626 20434 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "913fc7d751ed4b7388d889aac30a5f9b" candidate_uuid: "974b802141bd4f13beffef2aa03f88f9" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "197b880fa39d432a9b472845876bfaf0" is_pre_election: true
I20260129 03:00:02.648658 20434 raft_consensus.cc:2468] T 913fc7d751ed4b7388d889aac30a5f9b P 197b880fa39d432a9b472845876bfaf0 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 974b802141bd4f13beffef2aa03f88f9 in term 4.
I20260129 03:00:02.614768 20556 leader_election.cc:304] T 36ff33be666a4c82954731691dbcf327 P 1c50de89a5a845668696dcb28c3badd6 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1c50de89a5a845668696dcb28c3badd6; no voters: 197b880fa39d432a9b472845876bfaf0, 974b802141bd4f13beffef2aa03f88f9
I20260129 03:00:02.616590 20438 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3ccac82a01a2445fa5b0ce6c0fcf2295" candidate_uuid: "974b802141bd4f13beffef2aa03f88f9" candidate_term: 5 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "197b880fa39d432a9b472845876bfaf0" is_pre_election: true
I20260129 03:00:02.619573 20480 leader_election.cc:304] T add8d4923d1a489e86882c3a25692b95 P 974b802141bd4f13beffef2aa03f88f9 [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 974b802141bd4f13beffef2aa03f88f9; no voters: 1c50de89a5a845668696dcb28c3badd6, 7a9daf5d42a84615a6383cfec9a10608
I20260129 03:00:02.602535 20441 raft_consensus.cc:1240] T 4e420f90884f4796997b5ea41e49e411 P 197b880fa39d432a9b472845876bfaf0 [term 6 FOLLOWER]: Rejecting Update request from peer 974b802141bd4f13beffef2aa03f88f9 for earlier term 5. Current term is 6. Ops: []
I20260129 03:00:02.593226 22983 raft_consensus.cc:493] T 0340229d521b432580ae926f6062e3f7 P 974b802141bd4f13beffef2aa03f88f9 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260129 03:00:02.601377 23163 raft_consensus.cc:2749] T 45cc5f3d2688486d9b6dbe7e666dedde P 7a9daf5d42a84615a6383cfec9a10608 [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20260129 03:00:02.609239 20512 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "75be4bd6bd45474795fd472eff2953ac" candidate_uuid: "197b880fa39d432a9b472845876bfaf0" candidate_term: 10 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "974b802141bd4f13beffef2aa03f88f9"
W20260129 03:00:02.635356 20406 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260129 03:00:02.656260 20405 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.211.3:37735, user_credentials={real_user=slave}} blocked reactor thread for 49973.6us
W20260129 03:00:02.656745 20407 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.211.2:34517, user_credentials={real_user=slave}} blocked reactor thread for 56429.1us
I20260129 03:00:02.657742 20407 leader_election.cc:304] T 7a9a69cd82724b1faed1d10aa091fc4a P 197b880fa39d432a9b472845876bfaf0 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 197b880fa39d432a9b472845876bfaf0, 974b802141bd4f13beffef2aa03f88f9; no voters:
I20260129 03:00:02.663690 20581 raft_consensus.cc:1275] T 4f8f3a38f6e24ba4857875181f447442 P 1c50de89a5a845668696dcb28c3badd6 [term 6 FOLLOWER]: Refusing update from remote peer 974b802141bd4f13beffef2aa03f88f9: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 6 index: 4. (index mismatch)
I20260129 03:00:02.664651 21135 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3ccac82a01a2445fa5b0ce6c0fcf2295" candidate_uuid: "197b880fa39d432a9b472845876bfaf0" candidate_term: 5 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "7a9daf5d42a84615a6383cfec9a10608" is_pre_election: true
I20260129 03:00:02.656021 20512 raft_consensus.cc:3060] T 75be4bd6bd45474795fd472eff2953ac P 974b802141bd4f13beffef2aa03f88f9 [term 9 FOLLOWER]: Advancing to term 10
I20260129 03:00:02.669106 20437 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "febffe4a8b394368b69b1d164308fb9a" candidate_uuid: "974b802141bd4f13beffef2aa03f88f9" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "197b880fa39d432a9b472845876bfaf0" is_pre_election: true
I20260129 03:00:02.669891 20437 raft_consensus.cc:2468] T febffe4a8b394368b69b1d164308fb9a P 197b880fa39d432a9b472845876bfaf0 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 974b802141bd4f13beffef2aa03f88f9 in term 4.
I20260129 03:00:02.673760 20439 raft_consensus.cc:1240] T 0678b8b468bd43cdbc3e485900844821 P 197b880fa39d432a9b472845876bfaf0 [term 8 FOLLOWER]: Rejecting Update request from peer 974b802141bd4f13beffef2aa03f88f9 for earlier term 7. Current term is 8. Ops: []
I20260129 03:00:02.622792 23151 raft_consensus.cc:493] T a4c1226b987b468fa84dfc6ba0e2dbe1 P 974b802141bd4f13beffef2aa03f88f9 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 197b880fa39d432a9b472845876bfaf0)
I20260129 03:00:02.628260 22581 raft_consensus.cc:2749] T a4c1226b987b468fa84dfc6ba0e2dbe1 P 1c50de89a5a845668696dcb28c3badd6 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260129 03:00:02.629114 23180 raft_consensus.cc:2749] T dc2f4cb752cf4d59b5f6bc689f5d8104 P 197b880fa39d432a9b472845876bfaf0 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260129 03:00:02.684885 21135 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "42d27a28d89b409e8807ec0efd97af69" candidate_uuid: "197b880fa39d432a9b472845876bfaf0" candidate_term: 7 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "7a9daf5d42a84615a6383cfec9a10608" is_pre_election: true
I20260129 03:00:02.607659 22967 raft_consensus.cc:515] T 724a236524cb4e7ba3428880681d6cc2 P 974b802141bd4f13beffef2aa03f88f9 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 03:00:02.663125 22983 raft_consensus.cc:3060] T 0340229d521b432580ae926f6062e3f7 P 974b802141bd4f13beffef2aa03f88f9 [term 6 FOLLOWER]: Advancing to term 7
I20260129 03:00:02.612660 23112 consensus_queue.cc:1059] T cedce331e1584ac8b636c9b8a72b2ba0 P 197b880fa39d432a9b472845876bfaf0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 }, Status: INVALID_TERM, Last received: 6.4, Next index: 5, Last known committed idx: 3, Time since last communication: 0.062s
I20260129 03:00:02.655972 23039 raft_consensus.cc:493] T 00e79d8f25a34676b1d866eb2068a2b2 P 197b880fa39d432a9b472845876bfaf0 [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260129 03:00:02.713824 20440 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "96609ac68bdf4993a12e1c23355ca013" candidate_uuid: "974b802141bd4f13beffef2aa03f88f9" candidate_term: 7 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "197b880fa39d432a9b472845876bfaf0" is_pre_election: true
W20260129 03:00:02.693481 20406 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260129 03:00:02.694082 20556 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.211.2:34517, user_credentials={real_user=slave}} blocked reactor thread for 79425.8us
I20260129 03:00:02.716527 20556 leader_election.cc:304] T 4f8f3a38f6e24ba4857875181f447442 P 1c50de89a5a845668696dcb28c3badd6 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1c50de89a5a845668696dcb28c3badd6; no voters: 197b880fa39d432a9b472845876bfaf0, 974b802141bd4f13beffef2aa03f88f9
I20260129 03:00:02.679716 22667 raft_consensus.cc:2804] T 42d27a28d89b409e8807ec0efd97af69 P 7a9daf5d42a84615a6383cfec9a10608 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260129 03:00:02.668439 23188 raft_consensus.cc:2804] T b189f08e16a74a11980cc55d99359a0a P 197b880fa39d432a9b472845876bfaf0 [term 8 FOLLOWER]: Leader pre-election won for term 9
W20260129 03:00:02.679037 20405 leader_election.cc:336] T 724a236524cb4e7ba3428880681d6cc2 P 197b880fa39d432a9b472845876bfaf0 [CANDIDATE]: Term 5 election: RPC error from VoteRequest() call to peer 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3:37735): Timed out: RequestConsensusVote RPC to 127.18.211.3:37735 timed out after 1.613s (SENT)
I20260129 03:00:02.678622 23043 raft_consensus.cc:2804] T 7a9a69cd82724b1faed1d10aa091fc4a P 197b880fa39d432a9b472845876bfaf0 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260129 03:00:02.669776 23184 raft_consensus.cc:515] T add8d4923d1a489e86882c3a25692b95 P 7a9daf5d42a84615a6383cfec9a10608 [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } }
W20260129 03:00:02.678135 23119 consensus_meta.cc:287] T 4f5a77e94fef4b16a2ba7b067c8b144a P 7a9daf5d42a84615a6383cfec9a10608: Time spent flushing consensus metadata: real 0.671s user 0.011s sys 0.001s
I20260129 03:00:02.719779 20556 leader_election.cc:304] T c891d2bff4ce4568861bbc992e91fbb0 P 1c50de89a5a845668696dcb28c3badd6 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1c50de89a5a845668696dcb28c3badd6, 974b802141bd4f13beffef2aa03f88f9; no voters: 197b880fa39d432a9b472845876bfaf0
I20260129 03:00:02.688164 23185 raft_consensus.cc:493] T 87d5fe48a44a408aaf03e894a67ecb2f P 974b802141bd4f13beffef2aa03f88f9 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 197b880fa39d432a9b472845876bfaf0)
I20260129 03:00:02.729998 20405 leader_election.cc:304] T 724a236524cb4e7ba3428880681d6cc2 P 197b880fa39d432a9b472845876bfaf0 [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 197b880fa39d432a9b472845876bfaf0; no voters: 1c50de89a5a845668696dcb28c3badd6, 974b802141bd4f13beffef2aa03f88f9
W20260129 03:00:02.761400 20405 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.211.3:37735, user_credentials={real_user=slave}} blocked reactor thread for 82417.5us
I20260129 03:00:02.761744 20556 leader_election.cc:304] T 3906b080a5b34a74b8b64128635e8997 P 1c50de89a5a845668696dcb28c3badd6 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1c50de89a5a845668696dcb28c3badd6; no voters: 197b880fa39d432a9b472845876bfaf0, 974b802141bd4f13beffef2aa03f88f9
W20260129 03:00:02.762144 20405 leader_election.cc:336] T 3906b080a5b34a74b8b64128635e8997 P 197b880fa39d432a9b472845876bfaf0 [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3:37735): Timed out: RequestConsensusVote RPC to 127.18.211.3:37735 timed out after 1.625s (SENT)
I20260129 03:00:02.762543 20405 leader_election.cc:304] T 3906b080a5b34a74b8b64128635e8997 P 197b880fa39d432a9b472845876bfaf0 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 197b880fa39d432a9b472845876bfaf0; no voters: 1c50de89a5a845668696dcb28c3badd6, 974b802141bd4f13beffef2aa03f88f9
I20260129 03:00:02.722386 22953 raft_consensus.cc:2749] T 4f8f3a38f6e24ba4857875181f447442 P 1c50de89a5a845668696dcb28c3badd6 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260129 03:00:02.724169 22975 consensus_queue.cc:1059] T fcddab6bca684adf8ea032289a00f90e P 974b802141bd4f13beffef2aa03f88f9 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 }, Status: INVALID_TERM, Last received: 4.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.019s
I20260129 03:00:02.766376 23185 raft_consensus.cc:515] T 87d5fe48a44a408aaf03e894a67ecb2f P 974b802141bd4f13beffef2aa03f88f9 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } }
I20260129 03:00:02.722054 22667 raft_consensus.cc:493] T 42d27a28d89b409e8807ec0efd97af69 P 7a9daf5d42a84615a6383cfec9a10608 [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260129 03:00:02.768574 22667 raft_consensus.cc:3060] T 42d27a28d89b409e8807ec0efd97af69 P 7a9daf5d42a84615a6383cfec9a10608 [term 7 FOLLOWER]: Advancing to term 8
I20260129 03:00:02.769402 23116 raft_consensus.cc:2749] T 724a236524cb4e7ba3428880681d6cc2 P 197b880fa39d432a9b472845876bfaf0 [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20260129 03:00:02.769857 23185 leader_election.cc:290] T 87d5fe48a44a408aaf03e894a67ecb2f P 974b802141bd4f13beffef2aa03f88f9 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 197b880fa39d432a9b472845876bfaf0 (127.18.211.1:42027), 7a9daf5d42a84615a6383cfec9a10608 (127.18.211.4:40439)
I20260129 03:00:02.689040 23151 raft_consensus.cc:515] T a4c1226b987b468fa84dfc6ba0e2dbe1 P 974b802141bd4f13beffef2aa03f88f9 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
W20260129 03:00:02.744318 23191 log.cc:927] Time spent T cedce331e1584ac8b636c9b8a72b2ba0 P 197b880fa39d432a9b472845876bfaf0: Append to log took a long time: real 0.170s user 0.001s sys 0.000s
I20260129 03:00:02.771286 23133 raft_consensus.cc:2749] T 3906b080a5b34a74b8b64128635e8997 P 197b880fa39d432a9b472845876bfaf0 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260129 03:00:02.744616 23118 leader_election.cc:290] T 0678b8b468bd43cdbc3e485900844821 P 197b880fa39d432a9b472845876bfaf0 [CANDIDATE]: Term 8 election: Requested vote from peers 974b802141bd4f13beffef2aa03f88f9 (127.18.211.2:34517), 7a9daf5d42a84615a6383cfec9a10608 (127.18.211.4:40439)
I20260129 03:00:02.715565 23039 raft_consensus.cc:3060] T 00e79d8f25a34676b1d866eb2068a2b2 P 197b880fa39d432a9b472845876bfaf0 [term 7 FOLLOWER]: Advancing to term 8
I20260129 03:00:02.773124 21133 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34046 (request call id 544) took 2992 ms. Trace:
I20260129 03:00:02.678995 23104 raft_consensus.cc:515] T 45cc5f3d2688486d9b6dbe7e666dedde P 1c50de89a5a845668696dcb28c3badd6 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } }
I20260129 03:00:02.773509 21133 rpcz_store.cc:276] 0129 02:59:59.780671 (+ 0us) service_pool.cc:168] Inserting onto call queue
0129 02:59:59.780779 (+ 108us) service_pool.cc:225] Handling call
0129 02:59:59.782279 (+ 1500us) raft_consensus.cc:1471] Updating replica for 0 ops
0129 02:59:59.865410 (+ 83131us) raft_consensus.cc:1534] Early marking committed up to index 0
0129 02:59:59.865442 (+ 32us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0129 02:59:59.865482 (+ 40us) raft_consensus.cc:1645] Marking committed up to 5
0129 02:59:59.865509 (+ 27us) raft_consensus.cc:1697] Filling consensus response to leader.
0129 03:00:02.772866 (+2907357us) spinlock_profiling.cc:243] Waited 81.3 ms on lock 0x7b540062e8d4. stack: 00007f8fe044e373 00007f8fe044e270 00007f8fdfda0574 00007f8fe9a5c6b4 00007f8fe9a5c649 00007f8fe34975a1 00007f8fe352ea81 00007f8fe352d4be 00007f8fe84e1f49 00007f8fe1a6ed34 00007f8fe1a6ecc2 00007f8fe1a6ebf1 00007f8fe1a6eb6c 00007f8fe1a6de22 00007f8fe186d34c 00007f8fe186c786
0129 03:00:02.772922 (+ 56us) raft_consensus.cc:1691] UpdateReplicas() finished
0129 03:00:02.773062 (+ 140us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":186942336}
I20260129 03:00:02.775316 23104 leader_election.cc:290] T 45cc5f3d2688486d9b6dbe7e666dedde P 1c50de89a5a845668696dcb28c3badd6 [CANDIDATE]: Term 5 election: Requested vote from peers 974b802141bd4f13beffef2aa03f88f9 (127.18.211.2:34517), 7a9daf5d42a84615a6383cfec9a10608 (127.18.211.4:40439)
I20260129 03:00:02.776242 20584 raft_consensus.cc:2393] T 45cc5f3d2688486d9b6dbe7e666dedde P 1c50de89a5a845668696dcb28c3badd6 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 974b802141bd4f13beffef2aa03f88f9 in current term 5: Already voted for candidate 1c50de89a5a845668696dcb28c3badd6 in this term.
I20260129 03:00:02.729321 23188 raft_consensus.cc:493] T b189f08e16a74a11980cc55d99359a0a P 197b880fa39d432a9b472845876bfaf0 [term 8 FOLLOWER]: Starting leader election (detected failure of leader 1c50de89a5a845668696dcb28c3badd6)
I20260129 03:00:02.741797 23112 consensus_queue.cc:1048] T 1ce4ddf5b10e4ae8b193b13b2e5936ee P 197b880fa39d432a9b472845876bfaf0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 1, Time since last communication: 0.007s
I20260129 03:00:02.742097 23043 raft_consensus.cc:493] T 7a9a69cd82724b1faed1d10aa091fc4a P 197b880fa39d432a9b472845876bfaf0 [term 6 FOLLOWER]: Starting leader election (detected failure of leader 1c50de89a5a845668696dcb28c3badd6)
W20260129 03:00:02.801374 23196 log.cc:927] Time spent T 4e3805ffa70242e8ade66da260dddf66 P 1c50de89a5a845668696dcb28c3badd6: Append to log took a long time: real 0.114s user 0.001s sys 0.000s
I20260129 03:00:02.801574 23043 raft_consensus.cc:3060] T 7a9a69cd82724b1faed1d10aa091fc4a P 197b880fa39d432a9b472845876bfaf0 [term 6 FOLLOWER]: Advancing to term 7
I20260129 03:00:02.802254 23205 raft_consensus.cc:2749] T add8d4923d1a489e86882c3a25692b95 P 974b802141bd4f13beffef2aa03f88f9 [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20260129 03:00:02.724148 22977 consensus_queue.cc:1048] T add8d4923d1a489e86882c3a25692b95 P 1c50de89a5a845668696dcb28c3badd6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 5, Time since last communication: 0.061s
I20260129 03:00:02.790488 23188 raft_consensus.cc:3060] T b189f08e16a74a11980cc55d99359a0a P 197b880fa39d432a9b472845876bfaf0 [term 8 FOLLOWER]: Advancing to term 9
I20260129 03:00:02.748838 21128 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:47844 (request call id 347) took 1644 ms. Trace:
W20260129 03:00:02.741801 23082 log.cc:927] Time spent T b189f08e16a74a11980cc55d99359a0a P 1c50de89a5a845668696dcb28c3badd6: Append to log took a long time: real 3.181s user 0.009s sys 0.024s
I20260129 03:00:02.749541 23184 leader_election.cc:290] T add8d4923d1a489e86882c3a25692b95 P 7a9daf5d42a84615a6383cfec9a10608 [CANDIDATE]: Term 6 election: Requested vote from peers 974b802141bd4f13beffef2aa03f88f9 (127.18.211.2:34517), 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3:37735)
I20260129 03:00:02.753217 20516 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "926a250fa5834e88b96ccecbefbf5086" candidate_uuid: "197b880fa39d432a9b472845876bfaf0" candidate_term: 8 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "974b802141bd4f13beffef2aa03f88f9" is_pre_election: true
I20260129 03:00:02.765635 20405 leader_election.cc:304] T 2ac659d732314e1288849f08fe5e074e P 197b880fa39d432a9b472845876bfaf0 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 197b880fa39d432a9b472845876bfaf0; no voters: 7a9daf5d42a84615a6383cfec9a10608, 974b802141bd4f13beffef2aa03f88f9
I20260129 03:00:02.767992 22953 raft_consensus.cc:2749] T 3906b080a5b34a74b8b64128635e8997 P 1c50de89a5a845668696dcb28c3badd6 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260129 03:00:02.738914 20512 raft_consensus.cc:2468] T 75be4bd6bd45474795fd472eff2953ac P 974b802141bd4f13beffef2aa03f88f9 [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 197b880fa39d432a9b472845876bfaf0 in term 10.
W20260129 03:00:02.810130 20405 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.211.4:40439, user_credentials={real_user=slave}} blocked reactor thread for 44602.5us
I20260129 03:00:02.763604 23005 raft_consensus.cc:2764] T c891d2bff4ce4568861bbc992e91fbb0 P 1c50de89a5a845668696dcb28c3badd6 [term 7 FOLLOWER]: Leader pre-election decision vote started in defunct term 6: won
I20260129 03:00:02.811062 20405 leader_election.cc:304] T 18f30ebaa4174fd28c958e72d0cc7e24 P 197b880fa39d432a9b472845876bfaf0 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 197b880fa39d432a9b472845876bfaf0, 7a9daf5d42a84615a6383cfec9a10608; no voters:
I20260129 03:00:02.780277 23119 raft_consensus.cc:2749] T 4f5a77e94fef4b16a2ba7b067c8b144a P 7a9daf5d42a84615a6383cfec9a10608 [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260129 03:00:02.730450 22581 consensus_queue.cc:1048] T b189f08e16a74a11980cc55d99359a0a P 1c50de89a5a845668696dcb28c3badd6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.070s
I20260129 03:00:02.707433 23026 consensus_queue.cc:237] T fd8311b779af4e1fa1402b2a02a602f2 P 1c50de89a5a845668696dcb28c3badd6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } }
I20260129 03:00:02.812950 20407 leader_election.cc:304] T 75be4bd6bd45474795fd472eff2953ac P 197b880fa39d432a9b472845876bfaf0 [CANDIDATE]: Term 10 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 197b880fa39d432a9b472845876bfaf0, 974b802141bd4f13beffef2aa03f88f9; no voters:
W20260129 03:00:02.811451 20480 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.211.4:40439, user_credentials={real_user=slave}} blocked reactor thread for 191988us
I20260129 03:00:02.806622 22978 consensus_queue.cc:1243] T 4e420f90884f4796997b5ea41e49e411 P 974b802141bd4f13beffef2aa03f88f9 [LEADER]: Peer 7a9daf5d42a84615a6383cfec9a10608 log is divergent from this leader: its last log entry 6.7 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 6
W20260129 03:00:02.816134 20482 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260129 03:00:02.820232 20512 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0678b8b468bd43cdbc3e485900844821" candidate_uuid: "197b880fa39d432a9b472845876bfaf0" candidate_term: 8 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "974b802141bd4f13beffef2aa03f88f9"
W20260129 03:00:02.834374 20556 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.211.2:34517, user_credentials={real_user=slave}} blocked reactor thread for 72727.5us
I20260129 03:00:02.791666 23146 raft_consensus.cc:2804] T 913fc7d751ed4b7388d889aac30a5f9b P 974b802141bd4f13beffef2aa03f88f9 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260129 03:00:02.840674 20515 raft_consensus.cc:1275] T 75f45538dc704d9a8ec4ff8ad9f3ea5b P 974b802141bd4f13beffef2aa03f88f9 [term 4 FOLLOWER]: Refusing update from remote peer 7a9daf5d42a84615a6383cfec9a10608: Log matching property violated. Preceding OpId in replica: term: 3 index: 6. Preceding OpId from leader: term: 4 index: 6. (term mismatch)
I20260129 03:00:02.841100 20515 pending_rounds.cc:85] T 75f45538dc704d9a8ec4ff8ad9f3ea5b P 974b802141bd4f13beffef2aa03f88f9: Aborting all ops after (but not including) 5
I20260129 03:00:02.841320 20515 pending_rounds.cc:107] T 75f45538dc704d9a8ec4ff8ad9f3ea5b P 974b802141bd4f13beffef2aa03f88f9: Aborting uncommitted NO_OP operation due to leader change: 3.6
I20260129 03:00:02.841554 20515 raft_consensus.cc:2889] T 75f45538dc704d9a8ec4ff8ad9f3ea5b P 974b802141bd4f13beffef2aa03f88f9 [term 4 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260129 03:00:02.807216 21130 raft_consensus.cc:1240] T add8d4923d1a489e86882c3a25692b95 P 7a9daf5d42a84615a6383cfec9a10608 [term 6 FOLLOWER]: Rejecting Update request from peer 1c50de89a5a845668696dcb28c3badd6 for earlier term 5. Current term is 6. Ops: [3.6-5.7]
I20260129 03:00:02.830853 20480 leader_election.cc:304] T 96609ac68bdf4993a12e1c23355ca013 P 974b802141bd4f13beffef2aa03f88f9 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 974b802141bd4f13beffef2aa03f88f9; no voters: 1c50de89a5a845668696dcb28c3badd6, 7a9daf5d42a84615a6383cfec9a10608
I20260129 03:00:02.819497 23112 raft_consensus.cc:3060] T 2ac659d732314e1288849f08fe5e074e P 197b880fa39d432a9b472845876bfaf0 [term 3 FOLLOWER]: Advancing to term 4
I20260129 03:00:02.852883 20586 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "add8d4923d1a489e86882c3a25692b95" candidate_uuid: "7a9daf5d42a84615a6383cfec9a10608" candidate_term: 6 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "1c50de89a5a845668696dcb28c3badd6"
I20260129 03:00:02.835830 20556 leader_election.cc:304] T e7d83d4ccf684e76acb29a0dc844e1b8 P 1c50de89a5a845668696dcb28c3badd6 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1c50de89a5a845668696dcb28c3badd6; no voters: 197b880fa39d432a9b472845876bfaf0, 974b802141bd4f13beffef2aa03f88f9
I20260129 03:00:02.846565 21102 leader_election.cc:304] T 3e2d15f8640b407195210c476e09df91 P 7a9daf5d42a84615a6383cfec9a10608 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 7a9daf5d42a84615a6383cfec9a10608, 974b802141bd4f13beffef2aa03f88f9; no voters: 1c50de89a5a845668696dcb28c3badd6
I20260129 03:00:02.856647 20432 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "87d5fe48a44a408aaf03e894a67ecb2f" candidate_uuid: "974b802141bd4f13beffef2aa03f88f9" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "197b880fa39d432a9b472845876bfaf0" is_pre_election: true
I20260129 03:00:02.814591 20405 leader_election.cc:304] T 3ccac82a01a2445fa5b0ce6c0fcf2295 P 197b880fa39d432a9b472845876bfaf0 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 197b880fa39d432a9b472845876bfaf0; no voters: 7a9daf5d42a84615a6383cfec9a10608, 974b802141bd4f13beffef2aa03f88f9
W20260129 03:00:02.848096 20482 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260129 03:00:02.849502 20406 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260129 03:00:02.824625 22925 raft_consensus.cc:2804] T 18f30ebaa4174fd28c958e72d0cc7e24 P 197b880fa39d432a9b472845876bfaf0 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260129 03:00:02.822839 23041 leader_election.cc:290] T a609ac02fae84725ab743f95f51d33ce P 7a9daf5d42a84615a6383cfec9a10608 [CANDIDATE]: Term 8 election: Requested vote from peers 197b880fa39d432a9b472845876bfaf0 (127.18.211.1:42027), 974b802141bd4f13beffef2aa03f88f9 (127.18.211.2:34517)
I20260129 03:00:02.867763 20556 leader_election.cc:304] T 926a250fa5834e88b96ccecbefbf5086 P 1c50de89a5a845668696dcb28c3badd6 [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1c50de89a5a845668696dcb28c3badd6; no voters: 197b880fa39d432a9b472845876bfaf0, 974b802141bd4f13beffef2aa03f88f9
W20260129 03:00:02.869035 20482 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260129 03:00:02.828210 23151 leader_election.cc:290] T a4c1226b987b468fa84dfc6ba0e2dbe1 P 974b802141bd4f13beffef2aa03f88f9 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 197b880fa39d432a9b472845876bfaf0 (127.18.211.1:42027), 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3:37735)
W20260129 03:00:02.870102 20407 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260129 03:00:02.833461 23118 raft_consensus.cc:2804] T 75be4bd6bd45474795fd472eff2953ac P 197b880fa39d432a9b472845876bfaf0 [term 10 FOLLOWER]: Leader election won for term 10
I20260129 03:00:02.877063 20438 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a4c1226b987b468fa84dfc6ba0e2dbe1" candidate_uuid: "974b802141bd4f13beffef2aa03f88f9" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "197b880fa39d432a9b472845876bfaf0" is_pre_election: true
I20260129 03:00:02.879580 20437 raft_consensus.cc:1240] T a609ac02fae84725ab743f95f51d33ce P 197b880fa39d432a9b472845876bfaf0 [term 7 FOLLOWER]: Rejecting Update request from peer 974b802141bd4f13beffef2aa03f88f9 for earlier term 6. Current term is 7. Ops: []
I20260129 03:00:02.847481 23146 raft_consensus.cc:493] T 913fc7d751ed4b7388d889aac30a5f9b P 974b802141bd4f13beffef2aa03f88f9 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260129 03:00:02.886536 21099 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.211.1:42027, user_credentials={real_user=slave}} blocked reactor thread for 314967us
I20260129 03:00:02.887302 21099 leader_election.cc:304] T 2ac659d732314e1288849f08fe5e074e P 7a9daf5d42a84615a6383cfec9a10608 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 197b880fa39d432a9b472845876bfaf0, 7a9daf5d42a84615a6383cfec9a10608; no voters: 974b802141bd4f13beffef2aa03f88f9
W20260129 03:00:02.889037 20405 leader_election.cc:336] T 368953c9102643b2a21f99af36b51905 P 197b880fa39d432a9b472845876bfaf0 [CANDIDATE]: Term 7 election: RPC error from VoteRequest() call to peer 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3:37735): Timed out: RequestConsensusVote RPC to 127.18.211.3:37735 timed out after 1.645s (SENT)
I20260129 03:00:02.889658 20405 leader_election.cc:304] T 368953c9102643b2a21f99af36b51905 P 197b880fa39d432a9b472845876bfaf0 [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 197b880fa39d432a9b472845876bfaf0; no voters: 1c50de89a5a845668696dcb28c3badd6, 974b802141bd4f13beffef2aa03f88f9
W20260129 03:00:02.891089 20405 leader_election.cc:336] T a8479d1fe0bd41e5946186e3a6ca8c7a P 197b880fa39d432a9b472845876bfaf0 [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3:37735): Timed out: RequestConsensusVote RPC to 127.18.211.3:37735 timed out after 1.662s (SENT)
I20260129 03:00:02.884423 20586 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a4c1226b987b468fa84dfc6ba0e2dbe1" candidate_uuid: "974b802141bd4f13beffef2aa03f88f9" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "1c50de89a5a845668696dcb28c3badd6" is_pre_election: true
I20260129 03:00:02.893189 21131 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "45cc5f3d2688486d9b6dbe7e666dedde" candidate_uuid: "1c50de89a5a845668696dcb28c3badd6" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "7a9daf5d42a84615a6383cfec9a10608"
I20260129 03:00:02.852106 22978 raft_consensus.cc:2749] T 96609ac68bdf4993a12e1c23355ca013 P 974b802141bd4f13beffef2aa03f88f9 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260129 03:00:02.806449 21128 rpcz_store.cc:276] 0129 03:00:01.104857 (+ 0us) service_pool.cc:168] Inserting onto call queue
0129 03:00:01.253367 (+148510us) service_pool.cc:225] Handling call
0129 03:00:02.748603 (+1495236us) spinlock_profiling.cc:243] Waited 673 ms on lock 0x7b54006d79d4. stack: 00007f8fe044e373 00007f8fe044e270 00007f8fdfda0574 00007f8fe9a5c6b4 00007f8fe9a5c649 00007f8fe34975a1 00007f8fe3531f7a 00007f8fe84e3647 00007f8fe1a73834 00007f8fe1a737c2 00007f8fe1a736f1 00007f8fe1a7366c 00007f8fe1a72922 00007f8fe186d34c 00007f8fe186c786 00007f8fe186c12f
0129 03:00:02.748773 (+ 170us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1548596096}
I20260129 03:00:02.895798 21131 raft_consensus.cc:2393] T 45cc5f3d2688486d9b6dbe7e666dedde P 7a9daf5d42a84615a6383cfec9a10608 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 1c50de89a5a845668696dcb28c3badd6 in current term 5: Already voted for candidate 7a9daf5d42a84615a6383cfec9a10608 in this term.
I20260129 03:00:02.859767 23055 raft_consensus.cc:493] T c891d2bff4ce4568861bbc992e91fbb0 P 974b802141bd4f13beffef2aa03f88f9 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260129 03:00:02.891620 20405 leader_election.cc:304] T a8479d1fe0bd41e5946186e3a6ca8c7a P 197b880fa39d432a9b472845876bfaf0 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 197b880fa39d432a9b472845876bfaf0; no voters: 1c50de89a5a845668696dcb28c3badd6, 974b802141bd4f13beffef2aa03f88f9
I20260129 03:00:02.859752 23133 raft_consensus.cc:2749] T 3ccac82a01a2445fa5b0ce6c0fcf2295 P 197b880fa39d432a9b472845876bfaf0 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260129 03:00:02.866084 23051 raft_consensus.cc:2749] T e7d83d4ccf684e76acb29a0dc844e1b8 P 1c50de89a5a845668696dcb28c3badd6 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260129 03:00:02.866415 23202 raft_consensus.cc:493] T 75f45538dc704d9a8ec4ff8ad9f3ea5b P 974b802141bd4f13beffef2aa03f88f9 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260129 03:00:02.866930 23175 raft_consensus.cc:2804] T 3e2d15f8640b407195210c476e09df91 P 7a9daf5d42a84615a6383cfec9a10608 [term 4 FOLLOWER]: Leader election won for term 4
I20260129 03:00:02.895321 22950 raft_consensus.cc:2749] T 368953c9102643b2a21f99af36b51905 P 197b880fa39d432a9b472845876bfaf0 [term 7 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
I20260129 03:00:02.866815 22925 raft_consensus.cc:493] T 18f30ebaa4174fd28c958e72d0cc7e24 P 197b880fa39d432a9b472845876bfaf0 [term 6 FOLLOWER]: Starting leader election (detected failure of leader 1c50de89a5a845668696dcb28c3badd6)
I20260129 03:00:02.872246 23104 raft_consensus.cc:2749] T 926a250fa5834e88b96ccecbefbf5086 P 1c50de89a5a845668696dcb28c3badd6 [term 7 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
I20260129 03:00:02.894584 23146 raft_consensus.cc:3060] T 913fc7d751ed4b7388d889aac30a5f9b P 974b802141bd4f13beffef2aa03f88f9 [term 4 FOLLOWER]: Advancing to term 5
I20260129 03:00:02.903142 23151 raft_consensus.cc:493] T 7d9494f5584d4d5db8b9adf1c8fc888b P 974b802141bd4f13beffef2aa03f88f9 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260129 03:00:02.963392 21099 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.211.1:42027, user_credentials={real_user=slave}} blocked reactor thread for 76136.1us
I20260129 03:00:02.969095 21131 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0678b8b468bd43cdbc3e485900844821" candidate_uuid: "197b880fa39d432a9b472845876bfaf0" candidate_term: 8 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "7a9daf5d42a84615a6383cfec9a10608"
I20260129 03:00:02.913468 22925 raft_consensus.cc:3060] T 18f30ebaa4174fd28c958e72d0cc7e24 P 197b880fa39d432a9b472845876bfaf0 [term 6 FOLLOWER]: Advancing to term 7
W20260129 03:00:02.976497 20556 connection.cc:232] RPC call timeout handler was delayed by -1.3161568430000443s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1587s for a response
I20260129 03:00:02.968084 20586 raft_consensus.cc:1240] T 926a250fa5834e88b96ccecbefbf5086 P 1c50de89a5a845668696dcb28c3badd6 [term 7 FOLLOWER]: Rejecting Update request from peer 974b802141bd4f13beffef2aa03f88f9 for earlier term 6. Current term is 7. Ops: []
I20260129 03:00:02.967465 22950 raft_consensus.cc:2749] T a8479d1fe0bd41e5946186e3a6ca8c7a P 197b880fa39d432a9b472845876bfaf0 [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260129 03:00:02.962306 23032 raft_consensus.cc:493] T 4408006eb5174de1896b037cf3f18457 P 974b802141bd4f13beffef2aa03f88f9 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 197b880fa39d432a9b472845876bfaf0)
I20260129 03:00:02.962733 22967 leader_election.cc:290] T 724a236524cb4e7ba3428880681d6cc2 P 974b802141bd4f13beffef2aa03f88f9 [CANDIDATE]: Term 5 election: Requested vote from peers 197b880fa39d432a9b472845876bfaf0 (127.18.211.1:42027), 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3:37735)
W20260129 03:00:03.018570 21099 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.211.1:42027, user_credentials={real_user=slave}} blocked reactor thread for 44464.3us
I20260129 03:00:03.019496 21099 leader_election.cc:304] T 4e3805ffa70242e8ade66da260dddf66 P 7a9daf5d42a84615a6383cfec9a10608 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 197b880fa39d432a9b472845876bfaf0, 7a9daf5d42a84615a6383cfec9a10608; no voters: 1c50de89a5a845668696dcb28c3badd6
W20260129 03:00:03.000254 20405 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.211.3:37735, user_credentials={real_user=slave}} blocked reactor thread for 109190us
I20260129 03:00:02.913295 23055 raft_consensus.cc:515] T c891d2bff4ce4568861bbc992e91fbb0 P 974b802141bd4f13beffef2aa03f88f9 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 03:00:02.966060 23216 raft_consensus.cc:493] T 8055232cf0004d40a85f3bf983fe00a9 P 974b802141bd4f13beffef2aa03f88f9 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260129 03:00:03.021386 20405 leader_election.cc:336] T 4408006eb5174de1896b037cf3f18457 P 197b880fa39d432a9b472845876bfaf0 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3:37735): Timed out: RequestConsensusVote RPC to 127.18.211.3:37735 timed out after 1.704s (SENT)
I20260129 03:00:03.022866 21099 leader_election.cc:304] T 18f30ebaa4174fd28c958e72d0cc7e24 P 7a9daf5d42a84615a6383cfec9a10608 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 197b880fa39d432a9b472845876bfaf0, 7a9daf5d42a84615a6383cfec9a10608; no voters: 1c50de89a5a845668696dcb28c3badd6
W20260129 03:00:02.977844 20480 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.211.3:37735, user_credentials={real_user=slave}} blocked reactor thread for 81000.3us
I20260129 03:00:02.975930 22891 raft_consensus.cc:2749] T 36ff33be666a4c82954731691dbcf327 P 1c50de89a5a845668696dcb28c3badd6 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260129 03:00:03.021031 21128 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "87d5fe48a44a408aaf03e894a67ecb2f" candidate_uuid: "974b802141bd4f13beffef2aa03f88f9" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "7a9daf5d42a84615a6383cfec9a10608" is_pre_election: true
I20260129 03:00:02.962230 23202 raft_consensus.cc:515] T 75f45538dc704d9a8ec4ff8ad9f3ea5b P 974b802141bd4f13beffef2aa03f88f9 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } }
I20260129 03:00:03.024348 20586 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "926a250fa5834e88b96ccecbefbf5086" candidate_uuid: "197b880fa39d432a9b472845876bfaf0" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "1c50de89a5a845668696dcb28c3badd6" is_pre_election: true
W20260129 03:00:03.026508 20405 leader_election.cc:336] T 4408006eb5174de1896b037cf3f18457 P 197b880fa39d432a9b472845876bfaf0 [CANDIDATE]: Term 4 election: RPC error from VoteRequest() call to peer 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3:37735): Timed out: RequestConsensusVote RPC to 127.18.211.3:37735 timed out after 1.724s (SENT)
I20260129 03:00:03.026657 21099 leader_election.cc:304] T 78aa39096868415d9afc2ffcb248e198 P 7a9daf5d42a84615a6383cfec9a10608 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 197b880fa39d432a9b472845876bfaf0, 7a9daf5d42a84615a6383cfec9a10608; no voters: 1c50de89a5a845668696dcb28c3badd6
W20260129 03:00:03.035327 20555 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260129 03:00:03.020576 23208 consensus_queue.cc:1048] T 4f8f3a38f6e24ba4857875181f447442 P 974b802141bd4f13beffef2aa03f88f9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.048s
I20260129 03:00:02.978355 23175 raft_consensus.cc:697] T 3e2d15f8640b407195210c476e09df91 P 7a9daf5d42a84615a6383cfec9a10608 [term 4 LEADER]: Becoming Leader. State: Replica: 7a9daf5d42a84615a6383cfec9a10608, State: Running, Role: LEADER
W20260129 03:00:03.025915 23186 log.cc:927] Time spent T 3ccac82a01a2445fa5b0ce6c0fcf2295 P 974b802141bd4f13beffef2aa03f88f9: Append to log took a long time: real 0.052s user 0.000s sys 0.000s
I20260129 03:00:03.048218 20586 raft_consensus.cc:2393] T 926a250fa5834e88b96ccecbefbf5086 P 1c50de89a5a845668696dcb28c3badd6 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 197b880fa39d432a9b472845876bfaf0 in current term 7: Already voted for candidate 1c50de89a5a845668696dcb28c3badd6 in this term.
I20260129 03:00:02.973176 22983 raft_consensus.cc:515] T 0340229d521b432580ae926f6062e3f7 P 974b802141bd4f13beffef2aa03f88f9 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
W20260129 03:00:03.040870 20405 leader_election.cc:336] T c2633a6fc34c44579de348d208b0c01e P 197b880fa39d432a9b472845876bfaf0 [CANDIDATE]: Term 3 election: RPC error from VoteRequest() call to peer 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3:37735): Timed out: RequestConsensusVote RPC to 127.18.211.3:37735 timed out after 1.734s (SENT)
I20260129 03:00:03.059902 20405 leader_election.cc:304] T c2633a6fc34c44579de348d208b0c01e P 197b880fa39d432a9b472845876bfaf0 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 197b880fa39d432a9b472845876bfaf0; no voters: 1c50de89a5a845668696dcb28c3badd6, 974b802141bd4f13beffef2aa03f88f9
I20260129 03:00:03.024835 23079 raft_consensus.cc:2804] T 4e3805ffa70242e8ade66da260dddf66 P 7a9daf5d42a84615a6383cfec9a10608 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260129 03:00:03.031908 23113 raft_consensus.cc:493] T 4b1876f9a70b49edb3aa74105e66d3b3 P 974b802141bd4f13beffef2aa03f88f9 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260129 03:00:02.978492 23151 raft_consensus.cc:515] T 7d9494f5584d4d5db8b9adf1c8fc888b P 974b802141bd4f13beffef2aa03f88f9 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 03:00:02.978938 23206 raft_consensus.cc:2804] T 2ac659d732314e1288849f08fe5e074e P 7a9daf5d42a84615a6383cfec9a10608 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260129 03:00:03.021975 22978 consensus_queue.cc:1059] T 0f20af7fbe5d4db89a6f17948d98912c P 974b802141bd4f13beffef2aa03f88f9 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 }, Status: INVALID_TERM, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.044s
I20260129 03:00:03.045851 23216 raft_consensus.cc:515] T 8055232cf0004d40a85f3bf983fe00a9 P 974b802141bd4f13beffef2aa03f88f9 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 03:00:03.071132 23043 raft_consensus.cc:515] T 7a9a69cd82724b1faed1d10aa091fc4a P 197b880fa39d432a9b472845876bfaf0 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
W20260129 03:00:03.075855 20405 leader_election.cc:336] T 8055232cf0004d40a85f3bf983fe00a9 P 197b880fa39d432a9b472845876bfaf0 [CANDIDATE]: Term 4 election: RPC error from VoteRequest() call to peer 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3:37735): Timed out: RequestConsensusVote RPC to 127.18.211.3:37735 timed out after 1.744s (SENT)
I20260129 03:00:03.072305 20586 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "75be4bd6bd45474795fd472eff2953ac" candidate_uuid: "197b880fa39d432a9b472845876bfaf0" candidate_term: 10 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "1c50de89a5a845668696dcb28c3badd6"
W20260129 03:00:03.146013 20405 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.211.3:37735, user_credentials={real_user=slave}} blocked reactor thread for 70210us
W20260129 03:00:03.146770 20405 leader_election.cc:336] T 724a236524cb4e7ba3428880681d6cc2 P 197b880fa39d432a9b472845876bfaf0 [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3:37735): Timed out: RequestConsensusVote RPC to 127.18.211.3:37735 timed out after 1.753s (SENT)
W20260129 03:00:03.114215 21099 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.211.1:42027, user_credentials={real_user=slave}} blocked reactor thread for 49907us
I20260129 03:00:03.072947 22950 raft_consensus.cc:2749] T c2633a6fc34c44579de348d208b0c01e P 197b880fa39d432a9b472845876bfaf0 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
W20260129 03:00:03.119621 23219 log.cc:927] Time spent T 0678b8b468bd43cdbc3e485900844821 P 7a9daf5d42a84615a6383cfec9a10608: Append to log took a long time: real 0.056s user 0.000s sys 0.000s
I20260129 03:00:03.120213 22667 raft_consensus.cc:515] T 42d27a28d89b409e8807ec0efd97af69 P 7a9daf5d42a84615a6383cfec9a10608 [term 8 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } }
I20260129 03:00:03.124610 23107 raft_consensus.cc:2804] T 78aa39096868415d9afc2ffcb248e198 P 7a9daf5d42a84615a6383cfec9a10608 [term 5 FOLLOWER]: Leader pre-election won for term 6
W20260129 03:00:03.131162 20406 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260129 03:00:03.058857 22728 consensus_meta.cc:287] T 505580c93c694bbb8ea96fdf9b17ab46 P 1c50de89a5a845668696dcb28c3badd6: Time spent flushing consensus metadata: real 0.679s user 0.009s sys 0.003s
I20260129 03:00:03.035519 23032 raft_consensus.cc:515] T 4408006eb5174de1896b037cf3f18457 P 974b802141bd4f13beffef2aa03f88f9 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 03:00:03.051239 22986 consensus_queue.cc:1059] T 926a250fa5834e88b96ccecbefbf5086 P 974b802141bd4f13beffef2aa03f88f9 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 }, Status: INVALID_TERM, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.120s
I20260129 03:00:03.188082 20512 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "913fc7d751ed4b7388d889aac30a5f9b" candidate_uuid: "1c50de89a5a845668696dcb28c3badd6" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "974b802141bd4f13beffef2aa03f88f9" is_pre_election: true
W20260129 03:00:03.219135 20405 leader_election.cc:336] T c2633a6fc34c44579de348d208b0c01e P 197b880fa39d432a9b472845876bfaf0 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3:37735): Timed out: RequestConsensusVote RPC to 127.18.211.3:37735 timed out after 1.844s (SENT)
I20260129 03:00:03.219297 20586 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fcddab6bca684adf8ea032289a00f90e" candidate_uuid: "197b880fa39d432a9b472845876bfaf0" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "1c50de89a5a845668696dcb28c3badd6"
W20260129 03:00:03.228863 20405 leader_election.cc:336] T 4b1876f9a70b49edb3aa74105e66d3b3 P 197b880fa39d432a9b472845876bfaf0 [CANDIDATE]: Term 5 election: RPC error from VoteRequest() call to peer 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3:37735): Timed out: RequestConsensusVote RPC to 127.18.211.3:37735 timed out after 1.854s (SENT)
I20260129 03:00:03.229538 20586 raft_consensus.cc:2393] T fcddab6bca684adf8ea032289a00f90e P 1c50de89a5a845668696dcb28c3badd6 [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate 197b880fa39d432a9b472845876bfaf0 in current term 9: Already voted for candidate 1c50de89a5a845668696dcb28c3badd6 in this term.
I20260129 03:00:03.226580 21135 raft_consensus.cc:2368] T 42d27a28d89b409e8807ec0efd97af69 P 7a9daf5d42a84615a6383cfec9a10608 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 197b880fa39d432a9b472845876bfaf0 for earlier term 7. Current term is 8.
I20260129 03:00:03.073028 23079 raft_consensus.cc:493] T 4e3805ffa70242e8ade66da260dddf66 P 7a9daf5d42a84615a6383cfec9a10608 [term 5 FOLLOWER]: Starting leader election (detected failure of leader 1c50de89a5a845668696dcb28c3badd6)
I20260129 03:00:03.229425 20405 leader_election.cc:304] T 4b1876f9a70b49edb3aa74105e66d3b3 P 197b880fa39d432a9b472845876bfaf0 [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 197b880fa39d432a9b472845876bfaf0; no voters: 1c50de89a5a845668696dcb28c3badd6, 974b802141bd4f13beffef2aa03f88f9
I20260129 03:00:03.144381 23206 raft_consensus.cc:493] T 2ac659d732314e1288849f08fe5e074e P 7a9daf5d42a84615a6383cfec9a10608 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260129 03:00:03.235356 22667 leader_election.cc:290] T 42d27a28d89b409e8807ec0efd97af69 P 7a9daf5d42a84615a6383cfec9a10608 [CANDIDATE]: Term 8 election: Requested vote from peers 197b880fa39d432a9b472845876bfaf0 (127.18.211.1:42027), 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3:37735)
I20260129 03:00:03.232196 20437 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a609ac02fae84725ab743f95f51d33ce" candidate_uuid: "7a9daf5d42a84615a6383cfec9a10608" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "197b880fa39d432a9b472845876bfaf0"
I20260129 03:00:03.246846 20437 raft_consensus.cc:3060] T a609ac02fae84725ab743f95f51d33ce P 197b880fa39d432a9b472845876bfaf0 [term 7 FOLLOWER]: Advancing to term 8
I20260129 03:00:03.187495 23107 raft_consensus.cc:493] T 78aa39096868415d9afc2ffcb248e198 P 7a9daf5d42a84615a6383cfec9a10608 [term 5 FOLLOWER]: Starting leader election (detected failure of leader 1c50de89a5a845668696dcb28c3badd6)
W20260129 03:00:03.255965 20482 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260129 03:00:03.259114 20405 leader_election.cc:336] T a0fcc58d6fb6473cafbd9d34d5195269 P 197b880fa39d432a9b472845876bfaf0 [CANDIDATE]: Term 4 election: RPC error from VoteRequest() call to peer 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3:37735): Timed out: RequestConsensusVote RPC to 127.18.211.3:37735 timed out after 1.892s (SENT)
I20260129 03:00:03.259588 20405 leader_election.cc:304] T a0fcc58d6fb6473cafbd9d34d5195269 P 197b880fa39d432a9b472845876bfaf0 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 197b880fa39d432a9b472845876bfaf0; no voters: 1c50de89a5a845668696dcb28c3badd6, 974b802141bd4f13beffef2aa03f88f9
I20260129 03:00:03.219060 23097 raft_consensus.cc:493] T cb0b3f259ebc4310927338a8740e87b8 P 974b802141bd4f13beffef2aa03f88f9 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 7a9daf5d42a84615a6383cfec9a10608)
I20260129 03:00:03.261152 20583 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35754 (request call id 1711) took 2321 ms. Trace:
I20260129 03:00:03.261245 20586 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0340229d521b432580ae926f6062e3f7" candidate_uuid: "197b880fa39d432a9b472845876bfaf0" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "1c50de89a5a845668696dcb28c3badd6" is_pre_election: true
I20260129 03:00:03.265586 20583 rpcz_store.cc:276] 0129 03:00:00.939835 (+ 0us) service_pool.cc:168] Inserting onto call queue
0129 03:00:00.943606 (+ 3771us) service_pool.cc:225] Handling call
0129 03:00:00.957792 (+ 14186us) raft_consensus.cc:1471] Updating replica for 0 ops
0129 03:00:01.057711 (+ 99919us) raft_consensus.cc:1697] Filling consensus response to leader.
0129 03:00:03.260875 (+2203164us) spinlock_profiling.cc:243] Waited 18.5 ms on lock 0x7b54000889d4. stack: 00007f8fe044e373 00007f8fe044e270 00007f8fdfda0574 00007f8fe9a5c6b4 00007f8fe9a5c649 00007f8fe34975a1 00007f8fe352ea81 00007f8fe352d4be 00007f8fe84e1f49 00007f8fe1a6ed34 00007f8fe1a6ecc2 00007f8fe1a6ebf1 00007f8fe1a6eb6c 00007f8fe1a6de22 00007f8fe186d34c 00007f8fe186c786
0129 03:00:03.261074 (+ 199us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":42493312}
I20260129 03:00:03.266028 20586 raft_consensus.cc:2468] T 0340229d521b432580ae926f6062e3f7 P 1c50de89a5a845668696dcb28c3badd6 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 197b880fa39d432a9b472845876bfaf0 in term 6.
I20260129 03:00:03.231508 22978 consensus_queue.cc:1243] T 0f20af7fbe5d4db89a6f17948d98912c P 974b802141bd4f13beffef2aa03f88f9 [LEADER]: Peer 197b880fa39d432a9b472845876bfaf0 log is divergent from this leader: its last log entry 5.4 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 3
W20260129 03:00:03.277709 21101 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260129 03:00:03.293358 23225 log.cc:927] Time spent T 1ce4ddf5b10e4ae8b193b13b2e5936ee P 974b802141bd4f13beffef2aa03f88f9: Append to log took a long time: real 0.060s user 0.000s sys 0.000s
I20260129 03:00:03.262094 20441 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "42d27a28d89b409e8807ec0efd97af69" candidate_uuid: "7a9daf5d42a84615a6383cfec9a10608" candidate_term: 8 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "197b880fa39d432a9b472845876bfaf0"
I20260129 03:00:03.294092 20441 raft_consensus.cc:3060] T 42d27a28d89b409e8807ec0efd97af69 P 197b880fa39d432a9b472845876bfaf0 [term 6 FOLLOWER]: Advancing to term 8
I20260129 03:00:03.293771 23188 raft_consensus.cc:515] T b189f08e16a74a11980cc55d99359a0a P 197b880fa39d432a9b472845876bfaf0 [term 9 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 03:00:03.295310 20516 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c2633a6fc34c44579de348d208b0c01e" candidate_uuid: "1c50de89a5a845668696dcb28c3badd6" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "974b802141bd4f13beffef2aa03f88f9" is_pre_election: true
I20260129 03:00:03.242413 23079 raft_consensus.cc:3060] T 4e3805ffa70242e8ade66da260dddf66 P 7a9daf5d42a84615a6383cfec9a10608 [term 5 FOLLOWER]: Advancing to term 6
I20260129 03:00:03.295957 20516 raft_consensus.cc:2468] T c2633a6fc34c44579de348d208b0c01e P 974b802141bd4f13beffef2aa03f88f9 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1c50de89a5a845668696dcb28c3badd6 in term 2.
I20260129 03:00:03.296584 20583 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "42d27a28d89b409e8807ec0efd97af69" candidate_uuid: "7a9daf5d42a84615a6383cfec9a10608" candidate_term: 8 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "1c50de89a5a845668696dcb28c3badd6"
I20260129 03:00:03.296866 20516 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "8055232cf0004d40a85f3bf983fe00a9" candidate_uuid: "1c50de89a5a845668696dcb28c3badd6" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "974b802141bd4f13beffef2aa03f88f9" is_pre_election: true
I20260129 03:00:03.297621 20516 raft_consensus.cc:2393] T 8055232cf0004d40a85f3bf983fe00a9 P 974b802141bd4f13beffef2aa03f88f9 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1c50de89a5a845668696dcb28c3badd6 in current term 4: Already voted for candidate 197b880fa39d432a9b472845876bfaf0 in this term.
I20260129 03:00:03.245159 23206 raft_consensus.cc:3060] T 2ac659d732314e1288849f08fe5e074e P 7a9daf5d42a84615a6383cfec9a10608 [term 4 FOLLOWER]: Advancing to term 5
I20260129 03:00:03.241696 23039 raft_consensus.cc:515] T 00e79d8f25a34676b1d866eb2068a2b2 P 197b880fa39d432a9b472845876bfaf0 [term 8 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } }
I20260129 03:00:03.262621 23086 raft_consensus.cc:493] T cb4727386a3e48aca5d71cc6c611c197 P 197b880fa39d432a9b472845876bfaf0 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260129 03:00:03.291831 20586 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "724a236524cb4e7ba3428880681d6cc2" candidate_uuid: "974b802141bd4f13beffef2aa03f88f9" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "1c50de89a5a845668696dcb28c3badd6"
I20260129 03:00:03.292243 23133 raft_consensus.cc:2749] T 4b1876f9a70b49edb3aa74105e66d3b3 P 197b880fa39d432a9b472845876bfaf0 [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20260129 03:00:03.308674 23039 leader_election.cc:290] T 00e79d8f25a34676b1d866eb2068a2b2 P 197b880fa39d432a9b472845876bfaf0 [CANDIDATE]: Term 8 election: Requested vote from peers 974b802141bd4f13beffef2aa03f88f9 (127.18.211.2:34517), 7a9daf5d42a84615a6383cfec9a10608 (127.18.211.4:40439)
I20260129 03:00:03.299104 20432 raft_consensus.cc:2955] T 96609ac68bdf4993a12e1c23355ca013 P 197b880fa39d432a9b472845876bfaf0 [term 6 FOLLOWER]: Committing config change with OpId 2.4: config changed from index 2 to 4, 7a9daf5d42a84615a6383cfec9a10608 (127.18.211.4) changed from NON_VOTER to VOTER. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } attrs { replace: true } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } } }
W20260129 03:00:03.310310 23223 log.cc:927] Time spent T 4f8f3a38f6e24ba4857875181f447442 P 974b802141bd4f13beffef2aa03f88f9: Append to log took a long time: real 0.053s user 0.001s sys 0.000s
I20260129 03:00:03.303722 20511 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57600 (request call id 1866) took 1218 ms. Trace:
I20260129 03:00:03.319229 20441 raft_consensus.cc:2468] T 42d27a28d89b409e8807ec0efd97af69 P 197b880fa39d432a9b472845876bfaf0 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7a9daf5d42a84615a6383cfec9a10608 in term 8.
W20260129 03:00:03.320640 20406 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260129 03:00:03.307840 20516 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a5a05fbcbbcf4916bbc01fd1a5b5e90d" candidate_uuid: "1c50de89a5a845668696dcb28c3badd6" candidate_term: 8 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "974b802141bd4f13beffef2aa03f88f9" is_pre_election: true
I20260129 03:00:03.334367 20516 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "add8d4923d1a489e86882c3a25692b95" candidate_uuid: "7a9daf5d42a84615a6383cfec9a10608" candidate_term: 6 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "974b802141bd4f13beffef2aa03f88f9"
I20260129 03:00:03.310225 23086 raft_consensus.cc:515] T cb4727386a3e48aca5d71cc6c611c197 P 197b880fa39d432a9b472845876bfaf0 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 03:00:03.314352 20511 rpcz_store.cc:276] 0129 03:00:02.085469 (+ 0us) service_pool.cc:168] Inserting onto call queue
0129 03:00:02.415834 (+330365us) service_pool.cc:225] Handling call
0129 03:00:02.416228 (+ 394us) raft_consensus.cc:1471] Updating replica for 2 ops
0129 03:00:02.822858 (+406630us) raft_consensus.cc:1534] Early marking committed up to index 0
0129 03:00:02.822893 (+ 35us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0129 03:00:02.829622 (+ 6729us) log.cc:844] Serialized 52 byte log entry
0129 03:00:02.959451 (+129829us) raft_consensus.cc:1645] Marking committed up to 1
0129 03:00:02.959515 (+ 64us) raft_consensus.cc:1697] Filling consensus response to leader.
0129 03:00:02.959627 (+ 112us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0129 03:00:03.303438 (+343811us) raft_consensus.cc:1685] finished
0129 03:00:03.303468 (+ 30us) raft_consensus.cc:1691] UpdateReplicas() finished
0129 03:00:03.303652 (+ 184us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":149521,"raft.queue_time_us":160631,"raft.run_cpu_time_us":182,"raft.run_wall_time_us":8653,"thread_start_us":350440,"threads_started":2,"wal-append.queue_time_us":278445}
I20260129 03:00:03.374528 20511 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1ce4ddf5b10e4ae8b193b13b2e5936ee" candidate_uuid: "1c50de89a5a845668696dcb28c3badd6" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "974b802141bd4f13beffef2aa03f88f9" is_pre_election: true
W20260129 03:00:03.363933 20482 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260129 03:00:03.376732 20516 raft_consensus.cc:2393] T add8d4923d1a489e86882c3a25692b95 P 974b802141bd4f13beffef2aa03f88f9 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 7a9daf5d42a84615a6383cfec9a10608 in current term 6: Already voted for candidate 974b802141bd4f13beffef2aa03f88f9 in this term.
W20260129 03:00:03.422237 20479 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.524s user 0.008s sys 0.009s
W20260129 03:00:03.422734 20479 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.525s user 0.008s sys 0.009s
W20260129 03:00:03.423103 20479 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.211.1:42027, user_credentials={real_user=slave}} blocked reactor thread for 552847us
I20260129 03:00:03.421000 22667 raft_consensus.cc:493] T dc2f4cb752cf4d59b5f6bc689f5d8104 P 7a9daf5d42a84615a6383cfec9a10608 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 1c50de89a5a845668696dcb28c3badd6)
I20260129 03:00:03.424625 22667 raft_consensus.cc:515] T dc2f4cb752cf4d59b5f6bc689f5d8104 P 7a9daf5d42a84615a6383cfec9a10608 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } }
I20260129 03:00:03.260018 23175 consensus_queue.cc:237] T 3e2d15f8640b407195210c476e09df91 P 7a9daf5d42a84615a6383cfec9a10608 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.5, Last appended by leader: 5, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } }
I20260129 03:00:03.426653 23188 leader_election.cc:290] T b189f08e16a74a11980cc55d99359a0a P 197b880fa39d432a9b472845876bfaf0 [CANDIDATE]: Term 9 election: Requested vote from peers 974b802141bd4f13beffef2aa03f88f9 (127.18.211.2:34517), 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3:37735)
I20260129 03:00:03.428205 23086 leader_election.cc:290] T cb4727386a3e48aca5d71cc6c611c197 P 197b880fa39d432a9b472845876bfaf0 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 974b802141bd4f13beffef2aa03f88f9 (127.18.211.2:34517), 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3:37735)
I20260129 03:00:03.426775 23043 leader_election.cc:290] T 7a9a69cd82724b1faed1d10aa091fc4a P 197b880fa39d432a9b472845876bfaf0 [CANDIDATE]: Term 7 election: Requested vote from peers 974b802141bd4f13beffef2aa03f88f9 (127.18.211.2:34517), 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3:37735)
I20260129 03:00:03.302014 23216 leader_election.cc:290] T 8055232cf0004d40a85f3bf983fe00a9 P 974b802141bd4f13beffef2aa03f88f9 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 197b880fa39d432a9b472845876bfaf0 (127.18.211.1:42027), 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3:37735)
I20260129 03:00:03.432102 23055 leader_election.cc:290] T c891d2bff4ce4568861bbc992e91fbb0 P 974b802141bd4f13beffef2aa03f88f9 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 197b880fa39d432a9b472845876bfaf0 (127.18.211.1:42027), 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3:37735)
I20260129 03:00:03.432118 23133 raft_consensus.cc:2749] T a0fcc58d6fb6473cafbd9d34d5195269 P 197b880fa39d432a9b472845876bfaf0 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
W20260129 03:00:03.433666 20407 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260129 03:00:03.397962 20437 raft_consensus.cc:2468] T a609ac02fae84725ab743f95f51d33ce P 197b880fa39d432a9b472845876bfaf0 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7a9daf5d42a84615a6383cfec9a10608 in term 8.
I20260129 03:00:03.128018 23113 raft_consensus.cc:515] T 4b1876f9a70b49edb3aa74105e66d3b3 P 974b802141bd4f13beffef2aa03f88f9 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 03:00:03.412351 20511 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b189f08e16a74a11980cc55d99359a0a" candidate_uuid: "197b880fa39d432a9b472845876bfaf0" candidate_term: 9 candidate_status { last_received { term: 8 index: 4 } } ignore_live_leader: false dest_uuid: "974b802141bd4f13beffef2aa03f88f9"
I20260129 03:00:03.440845 20511 raft_consensus.cc:3060] T b189f08e16a74a11980cc55d99359a0a P 974b802141bd4f13beffef2aa03f88f9 [term 8 FOLLOWER]: Advancing to term 9
I20260129 03:00:03.441052 20441 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "dc2f4cb752cf4d59b5f6bc689f5d8104" candidate_uuid: "7a9daf5d42a84615a6383cfec9a10608" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "197b880fa39d432a9b472845876bfaf0" is_pre_election: true
I20260129 03:00:03.253554 23208 raft_consensus.cc:3055] T 926a250fa5834e88b96ccecbefbf5086 P 974b802141bd4f13beffef2aa03f88f9 [term 6 LEADER]: Stepping down as leader of term 6
I20260129 03:00:03.262697 23107 raft_consensus.cc:3060] T 78aa39096868415d9afc2ffcb248e198 P 7a9daf5d42a84615a6383cfec9a10608 [term 5 FOLLOWER]: Advancing to term 6
W20260129 03:00:03.357807 23168 consensus_meta.cc:287] T 75b0b6e82ddc4f92b1fe1d94f9a0b73b P 7a9daf5d42a84615a6383cfec9a10608: Time spent flushing consensus metadata: real 1.015s user 0.000s sys 0.012s
W20260129 03:00:03.421200 20405 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.211.3:37735, user_credentials={real_user=slave}} blocked reactor thread for 162105us
W20260129 03:00:03.449496 20405 leader_election.cc:336] T 0f20af7fbe5d4db89a6f17948d98912c P 197b880fa39d432a9b472845876bfaf0 [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3:37735): Timed out: RequestConsensusVote RPC to 127.18.211.3:37735 timed out after 1.993s (SENT)
I20260129 03:00:03.450466 20405 leader_election.cc:304] T 0678b8b468bd43cdbc3e485900844821 P 197b880fa39d432a9b472845876bfaf0 [CANDIDATE]: Term 8 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 197b880fa39d432a9b472845876bfaf0; no voters: 7a9daf5d42a84615a6383cfec9a10608, 974b802141bd4f13beffef2aa03f88f9
I20260129 03:00:03.450860 20586 raft_consensus.cc:2468] T 724a236524cb4e7ba3428880681d6cc2 P 1c50de89a5a845668696dcb28c3badd6 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 974b802141bd4f13beffef2aa03f88f9 in term 5.
I20260129 03:00:03.451599 21102 leader_election.cc:304] T add8d4923d1a489e86882c3a25692b95 P 7a9daf5d42a84615a6383cfec9a10608 [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7a9daf5d42a84615a6383cfec9a10608; no voters: 1c50de89a5a845668696dcb28c3badd6, 974b802141bd4f13beffef2aa03f88f9
I20260129 03:00:03.451910 20586 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0f20af7fbe5d4db89a6f17948d98912c" candidate_uuid: "197b880fa39d432a9b472845876bfaf0" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "1c50de89a5a845668696dcb28c3badd6"
I20260129 03:00:03.452530 20586 raft_consensus.cc:2376] T 0f20af7fbe5d4db89a6f17948d98912c P 1c50de89a5a845668696dcb28c3badd6 [term 7 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 197b880fa39d432a9b472845876bfaf0 in term 7. Re-sending same reply.
I20260129 03:00:03.452827 20480 leader_election.cc:304] T 724a236524cb4e7ba3428880681d6cc2 P 974b802141bd4f13beffef2aa03f88f9 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1c50de89a5a845668696dcb28c3badd6, 974b802141bd4f13beffef2aa03f88f9; no voters:
I20260129 03:00:03.453332 20586 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a5a05fbcbbcf4916bbc01fd1a5b5e90d" candidate_uuid: "197b880fa39d432a9b472845876bfaf0" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "1c50de89a5a845668696dcb28c3badd6" is_pre_election: true
I20260129 03:00:03.465854 20586 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "8055232cf0004d40a85f3bf983fe00a9" candidate_uuid: "974b802141bd4f13beffef2aa03f88f9" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "1c50de89a5a845668696dcb28c3badd6" is_pre_election: true
I20260129 03:00:03.466583 20586 raft_consensus.cc:2468] T 8055232cf0004d40a85f3bf983fe00a9 P 1c50de89a5a845668696dcb28c3badd6 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 974b802141bd4f13beffef2aa03f88f9 in term 4.
I20260129 03:00:03.467947 20586 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "96609ac68bdf4993a12e1c23355ca013" candidate_uuid: "197b880fa39d432a9b472845876bfaf0" candidate_term: 7 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "1c50de89a5a845668696dcb28c3badd6" is_pre_election: true
I20260129 03:00:03.469137 23168 raft_consensus.cc:515] T 75b0b6e82ddc4f92b1fe1d94f9a0b73b P 7a9daf5d42a84615a6383cfec9a10608 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } }
I20260129 03:00:03.470382 21128 raft_consensus.cc:1240] T 75b0b6e82ddc4f92b1fe1d94f9a0b73b P 7a9daf5d42a84615a6383cfec9a10608 [term 4 FOLLOWER]: Rejecting Update request from peer 1c50de89a5a845668696dcb28c3badd6 for earlier term 3. Current term is 4. Ops: []
I20260129 03:00:03.471640 23043 consensus_queue.cc:1059] T e7d83d4ccf684e76acb29a0dc844e1b8 P 197b880fa39d432a9b472845876bfaf0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 }, Status: INVALID_TERM, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.002s
I20260129 03:00:03.473469 23043 consensus_queue.cc:1243] T e7d83d4ccf684e76acb29a0dc844e1b8 P 197b880fa39d432a9b472845876bfaf0 [LEADER]: Peer 1c50de89a5a845668696dcb28c3badd6 log is divergent from this leader: its last log entry 4.3 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 2
I20260129 03:00:03.473978 23043 raft_consensus.cc:3055] T e7d83d4ccf684e76acb29a0dc844e1b8 P 197b880fa39d432a9b472845876bfaf0 [term 5 LEADER]: Stepping down as leader of term 5
I20260129 03:00:03.474303 23043 raft_consensus.cc:740] T e7d83d4ccf684e76acb29a0dc844e1b8 P 197b880fa39d432a9b472845876bfaf0 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 197b880fa39d432a9b472845876bfaf0, State: Running, Role: LEADER
W20260129 03:00:03.476155 23112 consensus_meta.cc:287] T 2ac659d732314e1288849f08fe5e074e P 197b880fa39d432a9b472845876bfaf0: Time spent flushing consensus metadata: real 0.615s user 0.007s sys 0.004s
I20260129 03:00:03.476403 23168 leader_election.cc:290] T 75b0b6e82ddc4f92b1fe1d94f9a0b73b P 7a9daf5d42a84615a6383cfec9a10608 [CANDIDATE]: Term 4 election: Requested vote from peers 974b802141bd4f13beffef2aa03f88f9 (127.18.211.2:34517), 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3:37735)
I20260129 03:00:03.441852 20441 raft_consensus.cc:2468] T dc2f4cb752cf4d59b5f6bc689f5d8104 P 197b880fa39d432a9b472845876bfaf0 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7a9daf5d42a84615a6383cfec9a10608 in term 4.
I20260129 03:00:03.476747 23112 raft_consensus.cc:2749] T 2ac659d732314e1288849f08fe5e074e P 197b880fa39d432a9b472845876bfaf0 [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260129 03:00:03.479604 23113 leader_election.cc:290] T 4b1876f9a70b49edb3aa74105e66d3b3 P 974b802141bd4f13beffef2aa03f88f9 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 197b880fa39d432a9b472845876bfaf0 (127.18.211.1:42027), 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3:37735)
I20260129 03:00:03.447366 21133 raft_consensus.cc:1240] T 4e420f90884f4796997b5ea41e49e411 P 7a9daf5d42a84615a6383cfec9a10608 [term 6 LEADER]: Rejecting Update request from peer 974b802141bd4f13beffef2aa03f88f9 for earlier term 5. Current term is 6. Ops: [5.7-5.7]
I20260129 03:00:03.439885 21099 leader_election.cc:304] T 42d27a28d89b409e8807ec0efd97af69 P 7a9daf5d42a84615a6383cfec9a10608 [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 197b880fa39d432a9b472845876bfaf0, 7a9daf5d42a84615a6383cfec9a10608; no voters:
I20260129 03:00:03.240900 23183 raft_consensus.cc:2955] T 96609ac68bdf4993a12e1c23355ca013 P 7a9daf5d42a84615a6383cfec9a10608 [term 6 LEADER]: Committing config change with OpId 2.4: config changed from index 2 to 4, 7a9daf5d42a84615a6383cfec9a10608 (127.18.211.4) changed from NON_VOTER to VOTER. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } attrs { replace: true } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } } }
I20260129 03:00:03.301769 23151 leader_election.cc:290] T 7d9494f5584d4d5db8b9adf1c8fc888b P 974b802141bd4f13beffef2aa03f88f9 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 197b880fa39d432a9b472845876bfaf0 (127.18.211.1:42027), 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3:37735)
I20260129 03:00:03.481259 21099 leader_election.cc:304] T a609ac02fae84725ab743f95f51d33ce P 7a9daf5d42a84615a6383cfec9a10608 [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 197b880fa39d432a9b472845876bfaf0, 7a9daf5d42a84615a6383cfec9a10608; no voters:
I20260129 03:00:03.481732 20513 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:48544 (request call id 264) took 2439 ms. Trace:
W20260129 03:00:03.479477 21129 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34060 (request call id 513) took 4204 ms (client timeout 2206 ms). Trace:
W20260129 03:00:03.484117 20555 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260129 03:00:03.475134 23043 consensus_queue.cc:260] T e7d83d4ccf684e76acb29a0dc844e1b8 P 197b880fa39d432a9b472845876bfaf0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.3, Last appended by leader: 3, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 03:00:03.467186 20405 leader_election.cc:304] T 2e58e9a5a8aa4cecafb464addb4f35ed P 197b880fa39d432a9b472845876bfaf0 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 197b880fa39d432a9b472845876bfaf0; no voters: 1c50de89a5a845668696dcb28c3badd6, 7a9daf5d42a84615a6383cfec9a10608
I20260129 03:00:03.490859 20480 leader_election.cc:304] T 8055232cf0004d40a85f3bf983fe00a9 P 974b802141bd4f13beffef2aa03f88f9 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1c50de89a5a845668696dcb28c3badd6, 974b802141bd4f13beffef2aa03f88f9; no voters:
W20260129 03:00:03.450500 23053 consensus_meta.cc:287] T 42d27a28d89b409e8807ec0efd97af69 P 1c50de89a5a845668696dcb28c3badd6: Time spent flushing consensus metadata: real 0.826s user 0.007s sys 0.005s
I20260129 03:00:03.482164 20513 rpcz_store.cc:276] 0129 03:00:01.042128 (+ 0us) service_pool.cc:168] Inserting onto call queue
0129 03:00:02.071725 (+1029597us) service_pool.cc:225] Handling call
0129 03:00:02.076523 (+ 4798us) raft_consensus.cc:1471] Updating replica for 0 ops
0129 03:00:02.321513 (+244990us) raft_consensus.cc:1697] Filling consensus response to leader.
0129 03:00:03.481459 (+1159946us) spinlock_profiling.cc:243] Waited 243 ms on lock 0x7b54000fcb54. stack: 00007f8fe044e373 00007f8fe044e270 00007f8fdfda0574 00007f8fe9a5c6b4 00007f8fe9a5c649 00007f8fe34975a1 00007f8fe352ea81 00007f8fe352d4be 00007f8fe84e1f49 00007f8fe1a6ed34 00007f8fe1a6ecc2 00007f8fe1a6ebf1 00007f8fe1a6eb6c 00007f8fe1a6de22 00007f8fe186d34c 00007f8fe186c786
0129 03:00:03.481644 (+ 185us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":559769856}
I20260129 03:00:03.469544 22950 raft_consensus.cc:2749] T 0678b8b468bd43cdbc3e485900844821 P 197b880fa39d432a9b472845876bfaf0 [term 8 FOLLOWER]: Leader election lost for term 8. Reason: could not achieve majority
I20260129 03:00:03.483418 20586 raft_consensus.cc:2410] T 96609ac68bdf4993a12e1c23355ca013 P 1c50de89a5a845668696dcb28c3badd6 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 197b880fa39d432a9b472845876bfaf0 for term 7 because replica has last-logged OpId of term: 6 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 4.
I20260129 03:00:03.493580 20405 leader_election.cc:304] T 1299723a4476420ab02fd8bfd7aa0645 P 197b880fa39d432a9b472845876bfaf0 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 197b880fa39d432a9b472845876bfaf0; no voters: 1c50de89a5a845668696dcb28c3badd6, 7a9daf5d42a84615a6383cfec9a10608
I20260129 03:00:03.494493 23112 raft_consensus.cc:2749] T 2e58e9a5a8aa4cecafb464addb4f35ed P 197b880fa39d432a9b472845876bfaf0 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260129 03:00:03.445766 23208 raft_consensus.cc:740] T 926a250fa5834e88b96ccecbefbf5086 P 974b802141bd4f13beffef2aa03f88f9 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 974b802141bd4f13beffef2aa03f88f9, State: Running, Role: LEADER
I20260129 03:00:03.304076 23097 raft_consensus.cc:515] T cb0b3f259ebc4310927338a8740e87b8 P 974b802141bd4f13beffef2aa03f88f9 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } }
I20260129 03:00:03.446514 20516 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0340229d521b432580ae926f6062e3f7" candidate_uuid: "1c50de89a5a845668696dcb28c3badd6" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "974b802141bd4f13beffef2aa03f88f9" is_pre_election: true
I20260129 03:00:03.491894 21099 leader_election.cc:304] T dc2f4cb752cf4d59b5f6bc689f5d8104 P 7a9daf5d42a84615a6383cfec9a10608 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 197b880fa39d432a9b472845876bfaf0, 7a9daf5d42a84615a6383cfec9a10608; no voters:
I20260129 03:00:03.495374 23151 raft_consensus.cc:2804] T 724a236524cb4e7ba3428880681d6cc2 P 974b802141bd4f13beffef2aa03f88f9 [term 5 FOLLOWER]: Leader election won for term 5
W20260129 03:00:03.483901 21129 rpcz_store.cc:269] 0129 02:59:59.275478 (+ 0us) service_pool.cc:168] Inserting onto call queue
0129 02:59:59.325573 (+ 50095us) service_pool.cc:225] Handling call
0129 03:00:03.478949 (+4153376us) spinlock_profiling.cc:243] Waited 84.3 ms on lock 0x7b540068e2d4. stack: 00007f8fe044e373 00007f8fe044e270 00007f8fdfda0574 00007f8fe9a5c6b4 00007f8fe9a5c649 00007f8fe34975a1 00007f8fe3531f7a 00007f8fe84e3647 00007f8fe1a73834 00007f8fe1a737c2 00007f8fe1a736f1 00007f8fe1a7366c 00007f8fe1a72922 00007f8fe186d34c 00007f8fe186c786 00007f8fe186c12f
0129 03:00:03.479374 (+ 425us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":193922944}
I20260129 03:00:03.482050 23182 raft_consensus.cc:2804] T 42d27a28d89b409e8807ec0efd97af69 P 7a9daf5d42a84615a6383cfec9a10608 [term 8 FOLLOWER]: Leader election won for term 8
W20260129 03:00:03.496192 20482 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260129 03:00:03.426769 22667 leader_election.cc:290] T dc2f4cb752cf4d59b5f6bc689f5d8104 P 7a9daf5d42a84615a6383cfec9a10608 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 197b880fa39d432a9b472845876bfaf0 (127.18.211.1:42027), 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3:37735)
W20260129 03:00:03.505220 20479 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.211.1:42027, user_credentials={real_user=slave}} blocked reactor thread for 81338us
I20260129 03:00:03.516464 21133 raft_consensus.cc:1240] T 3e2d15f8640b407195210c476e09df91 P 7a9daf5d42a84615a6383cfec9a10608 [term 4 LEADER]: Rejecting Update request from peer 1c50de89a5a845668696dcb28c3badd6 for earlier term 3. Current term is 4. Ops: []
I20260129 03:00:03.506147 20405 leader_election.cc:304] T 78aa39096868415d9afc2ffcb248e198 P 197b880fa39d432a9b472845876bfaf0 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 197b880fa39d432a9b472845876bfaf0; no voters: 1c50de89a5a845668696dcb28c3badd6, 7a9daf5d42a84615a6383cfec9a10608
I20260129 03:00:03.507377 23041 raft_consensus.cc:2804] T dc2f4cb752cf4d59b5f6bc689f5d8104 P 7a9daf5d42a84615a6383cfec9a10608 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260129 03:00:03.507504 20513 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "45cc5f3d2688486d9b6dbe7e666dedde" candidate_uuid: "1c50de89a5a845668696dcb28c3badd6" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "974b802141bd4f13beffef2aa03f88f9"
I20260129 03:00:03.518445 20513 raft_consensus.cc:2393] T 45cc5f3d2688486d9b6dbe7e666dedde P 974b802141bd4f13beffef2aa03f88f9 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 1c50de89a5a845668696dcb28c3badd6 in current term 5: Already voted for candidate 974b802141bd4f13beffef2aa03f88f9 in this term.
I20260129 03:00:03.518385 20405 leader_election.cc:304] T c891d2bff4ce4568861bbc992e91fbb0 P 197b880fa39d432a9b472845876bfaf0 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 197b880fa39d432a9b472845876bfaf0, 1c50de89a5a845668696dcb28c3badd6; no voters: 974b802141bd4f13beffef2aa03f88f9
I20260129 03:00:03.508067 22983 leader_election.cc:290] T 0340229d521b432580ae926f6062e3f7 P 974b802141bd4f13beffef2aa03f88f9 [CANDIDATE]: Term 7 election: Requested vote from peers 197b880fa39d432a9b472845876bfaf0 (127.18.211.1:42027), 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3:37735)
I20260129 03:00:03.520411 22950 raft_consensus.cc:2749] T 78aa39096868415d9afc2ffcb248e198 P 197b880fa39d432a9b472845876bfaf0 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260129 03:00:03.521018 23041 raft_consensus.cc:493] T dc2f4cb752cf4d59b5f6bc689f5d8104 P 7a9daf5d42a84615a6383cfec9a10608 [term 4 FOLLOWER]: Starting leader election (detected failure of leader 1c50de89a5a845668696dcb28c3badd6)
I20260129 03:00:03.521508 23041 raft_consensus.cc:3060] T dc2f4cb752cf4d59b5f6bc689f5d8104 P 7a9daf5d42a84615a6383cfec9a10608 [term 4 FOLLOWER]: Advancing to term 5
I20260129 03:00:03.498589 23168 raft_consensus.cc:2804] T a609ac02fae84725ab743f95f51d33ce P 7a9daf5d42a84615a6383cfec9a10608 [term 8 FOLLOWER]: Leader election won for term 8
I20260129 03:00:03.515689 23086 raft_consensus.cc:3060] T 1299723a4476420ab02fd8bfd7aa0645 P 197b880fa39d432a9b472845876bfaf0 [term 8 FOLLOWER]: Advancing to term 9
I20260129 03:00:03.508497 20583 raft_consensus.cc:3060] T 42d27a28d89b409e8807ec0efd97af69 P 1c50de89a5a845668696dcb28c3badd6 [term 7 FOLLOWER]: Advancing to term 8
I20260129 03:00:03.514180 20586 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "36ff33be666a4c82954731691dbcf327" candidate_uuid: "197b880fa39d432a9b472845876bfaf0" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "1c50de89a5a845668696dcb28c3badd6" is_pre_election: true
I20260129 03:00:03.532315 20586 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4b1876f9a70b49edb3aa74105e66d3b3" candidate_uuid: "974b802141bd4f13beffef2aa03f88f9" candidate_term: 6 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "1c50de89a5a845668696dcb28c3badd6" is_pre_election: true
I20260129 03:00:03.453781 23150 raft_consensus.cc:2749] T add8d4923d1a489e86882c3a25692b95 P 7a9daf5d42a84615a6383cfec9a10608 [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20260129 03:00:03.507485 20516 raft_consensus.cc:2393] T 0340229d521b432580ae926f6062e3f7 P 974b802141bd4f13beffef2aa03f88f9 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1c50de89a5a845668696dcb28c3badd6 in current term 7: Already voted for candidate 974b802141bd4f13beffef2aa03f88f9 in this term.
I20260129 03:00:03.534343 23151 raft_consensus.cc:697] T 724a236524cb4e7ba3428880681d6cc2 P 974b802141bd4f13beffef2aa03f88f9 [term 5 LEADER]: Becoming Leader. State: Replica: 974b802141bd4f13beffef2aa03f88f9, State: Running, Role: LEADER
I20260129 03:00:03.534219 23101 consensus_queue.cc:260] T 87d5fe48a44a408aaf03e894a67ecb2f P 197b880fa39d432a9b472845876bfaf0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 5.7, Last appended by leader: 7, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } }
I20260129 03:00:03.543615 23101 raft_consensus.cc:3060] T 87d5fe48a44a408aaf03e894a67ecb2f P 197b880fa39d432a9b472845876bfaf0 [term 5 FOLLOWER]: Advancing to term 6
I20260129 03:00:03.535148 23112 consensus_queue.cc:1048] T cedce331e1584ac8b636c9b8a72b2ba0 P 197b880fa39d432a9b472845876bfaf0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260129 03:00:03.502279 22925 raft_consensus.cc:515] T 18f30ebaa4174fd28c958e72d0cc7e24 P 197b880fa39d432a9b472845876bfaf0 [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } }
I20260129 03:00:03.531936 23182 raft_consensus.cc:697] T 42d27a28d89b409e8807ec0efd97af69 P 7a9daf5d42a84615a6383cfec9a10608 [term 8 LEADER]: Becoming Leader. State: Replica: 7a9daf5d42a84615a6383cfec9a10608, State: Running, Role: LEADER
I20260129 03:00:03.536248 22950 raft_consensus.cc:2804] T c891d2bff4ce4568861bbc992e91fbb0 P 197b880fa39d432a9b472845876bfaf0 [term 7 FOLLOWER]: Leader election won for term 7
I20260129 03:00:03.514300 20511 raft_consensus.cc:2468] T b189f08e16a74a11980cc55d99359a0a P 974b802141bd4f13beffef2aa03f88f9 [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 197b880fa39d432a9b472845876bfaf0 in term 9.
I20260129 03:00:03.565913 20511 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fcddab6bca684adf8ea032289a00f90e" candidate_uuid: "1c50de89a5a845668696dcb28c3badd6" candidate_term: 10 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "974b802141bd4f13beffef2aa03f88f9" is_pre_election: true
I20260129 03:00:03.537364 20405 leader_election.cc:304] T 4e3805ffa70242e8ade66da260dddf66 P 197b880fa39d432a9b472845876bfaf0 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 197b880fa39d432a9b472845876bfaf0; no voters: 1c50de89a5a845668696dcb28c3badd6, 7a9daf5d42a84615a6383cfec9a10608
I20260129 03:00:03.542128 23151 consensus_queue.cc:237] T 724a236524cb4e7ba3428880681d6cc2 P 974b802141bd4f13beffef2aa03f88f9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 4.3, Last appended by leader: 3, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 03:00:03.539475 20516 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e4d18c35b0a7493e85d0a4b92956a48f" candidate_uuid: "1c50de89a5a845668696dcb28c3badd6" candidate_term: 6 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "974b802141bd4f13beffef2aa03f88f9" is_pre_election: true
I20260129 03:00:03.544700 20586 raft_consensus.cc:2468] T 4b1876f9a70b49edb3aa74105e66d3b3 P 1c50de89a5a845668696dcb28c3badd6 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 974b802141bd4f13beffef2aa03f88f9 in term 5.
I20260129 03:00:03.544799 20513 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "75b0b6e82ddc4f92b1fe1d94f9a0b73b" candidate_uuid: "7a9daf5d42a84615a6383cfec9a10608" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "974b802141bd4f13beffef2aa03f88f9"
W20260129 03:00:03.546785 20556 thread.cc:641] dns-resolver [worker] (thread pool) Time spent creating pthread: real 0.509s user 0.023s sys 0.036s
W20260129 03:00:03.549824 23189 log.cc:927] Time spent T 4e420f90884f4796997b5ea41e49e411 P 7a9daf5d42a84615a6383cfec9a10608: Append to log took a long time: real 1.212s user 0.009s sys 0.015s
I20260129 03:00:03.538321 23043 raft_consensus.cc:3060] T e7d83d4ccf684e76acb29a0dc844e1b8 P 197b880fa39d432a9b472845876bfaf0 [term 5 FOLLOWER]: Advancing to term 6
I20260129 03:00:03.551600 23208 consensus_queue.cc:260] T 926a250fa5834e88b96ccecbefbf5086 P 974b802141bd4f13beffef2aa03f88f9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 6.4, Last appended by leader: 4, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 03:00:03.500516 23113 consensus_queue.cc:1059] T 4e420f90884f4796997b5ea41e49e411 P 974b802141bd4f13beffef2aa03f88f9 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false }, Status: INVALID_TERM, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.040s
I20260129 03:00:03.562891 23079 raft_consensus.cc:515] T 4e3805ffa70242e8ade66da260dddf66 P 7a9daf5d42a84615a6383cfec9a10608 [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } }
I20260129 03:00:03.581301 20407 leader_election.cc:304] T b189f08e16a74a11980cc55d99359a0a P 197b880fa39d432a9b472845876bfaf0 [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 197b880fa39d432a9b472845876bfaf0, 974b802141bd4f13beffef2aa03f88f9; no voters:
W20260129 03:00:03.584623 21099 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.211.1:42027, user_credentials={real_user=slave}} blocked reactor thread for 92824.1us
W20260129 03:00:03.555570 23146 consensus_meta.cc:287] T 913fc7d751ed4b7388d889aac30a5f9b P 974b802141bd4f13beffef2aa03f88f9: Time spent flushing consensus metadata: real 0.580s user 0.006s sys 0.003s
I20260129 03:00:03.607740 20513 raft_consensus.cc:3060] T 75b0b6e82ddc4f92b1fe1d94f9a0b73b P 974b802141bd4f13beffef2aa03f88f9 [term 3 FOLLOWER]: Advancing to term 4
W20260129 03:00:03.607782 20556 thread.cc:608] dns-resolver [worker] (thread pool) Time spent starting thread: real 0.570s user 0.024s sys 0.036s
I20260129 03:00:03.608168 20586 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "75b0b6e82ddc4f92b1fe1d94f9a0b73b" candidate_uuid: "7a9daf5d42a84615a6383cfec9a10608" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "1c50de89a5a845668696dcb28c3badd6"
W20260129 03:00:03.643533 20556 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.211.2:34517, user_credentials={real_user=slave}} blocked reactor thread for 606033us
I20260129 03:00:03.608748 20480 leader_election.cc:304] T 4b1876f9a70b49edb3aa74105e66d3b3 P 974b802141bd4f13beffef2aa03f88f9 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1c50de89a5a845668696dcb28c3badd6, 974b802141bd4f13beffef2aa03f88f9; no voters:
W20260129 03:00:03.650316 20556 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260129 03:00:03.650817 20586 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "926a250fa5834e88b96ccecbefbf5086" candidate_uuid: "197b880fa39d432a9b472845876bfaf0" candidate_term: 8 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "1c50de89a5a845668696dcb28c3badd6" is_pre_election: true
I20260129 03:00:03.651248 20556 leader_election.cc:304] T c2633a6fc34c44579de348d208b0c01e P 1c50de89a5a845668696dcb28c3badd6 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1c50de89a5a845668696dcb28c3badd6, 974b802141bd4f13beffef2aa03f88f9; no voters:
I20260129 03:00:03.651456 20586 raft_consensus.cc:2468] T 926a250fa5834e88b96ccecbefbf5086 P 1c50de89a5a845668696dcb28c3badd6 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 197b880fa39d432a9b472845876bfaf0 in term 7.
I20260129 03:00:03.652571 20586 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c891d2bff4ce4568861bbc992e91fbb0" candidate_uuid: "974b802141bd4f13beffef2aa03f88f9" candidate_term: 8 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "1c50de89a5a845668696dcb28c3badd6" is_pre_election: true
I20260129 03:00:03.539366 23232 raft_consensus.cc:2804] T 8055232cf0004d40a85f3bf983fe00a9 P 974b802141bd4f13beffef2aa03f88f9 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260129 03:00:03.621308 21131 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34046 (request call id 555) took 1258 ms. Trace:
I20260129 03:00:03.533169 23175 consensus_queue.cc:1048] T 96609ac68bdf4993a12e1c23355ca013 P 7a9daf5d42a84615a6383cfec9a10608 [LEADER]: Connected to new peer: Peer: permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 3, Time since last communication: 0.050s
I20260129 03:00:03.635659 23163 raft_consensus.cc:2804] T 18f30ebaa4174fd28c958e72d0cc7e24 P 7a9daf5d42a84615a6383cfec9a10608 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260129 03:00:03.632427 20516 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "cb4727386a3e48aca5d71cc6c611c197" candidate_uuid: "197b880fa39d432a9b472845876bfaf0" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "974b802141bd4f13beffef2aa03f88f9" is_pre_election: true
I20260129 03:00:03.564028 23097 leader_election.cc:290] T cb0b3f259ebc4310927338a8740e87b8 P 974b802141bd4f13beffef2aa03f88f9 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3:37735), 7a9daf5d42a84615a6383cfec9a10608 (127.18.211.4:40439)
I20260129 03:00:03.610262 20479 leader_election.cc:304] T 368953c9102643b2a21f99af36b51905 P 974b802141bd4f13beffef2aa03f88f9 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 974b802141bd4f13beffef2aa03f88f9; no voters: 197b880fa39d432a9b472845876bfaf0, 1c50de89a5a845668696dcb28c3badd6
W20260129 03:00:03.724187 20479 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.211.1:42027, user_credentials={real_user=slave}} blocked reactor thread for 114063us
I20260129 03:00:03.640166 23113 consensus_queue.cc:1243] T 4e420f90884f4796997b5ea41e49e411 P 974b802141bd4f13beffef2aa03f88f9 [LEADER]: Peer 7a9daf5d42a84615a6383cfec9a10608 log is divergent from this leader: its last log entry 6.7 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 6
I20260129 03:00:03.575060 20511 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7a9a69cd82724b1faed1d10aa091fc4a" candidate_uuid: "197b880fa39d432a9b472845876bfaf0" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "974b802141bd4f13beffef2aa03f88f9"
I20260129 03:00:03.666574 20556 leader_election.cc:304] T 45cc5f3d2688486d9b6dbe7e666dedde P 1c50de89a5a845668696dcb28c3badd6 [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1c50de89a5a845668696dcb28c3badd6; no voters: 7a9daf5d42a84615a6383cfec9a10608, 974b802141bd4f13beffef2aa03f88f9
I20260129 03:00:03.641974 23208 raft_consensus.cc:3060] T 926a250fa5834e88b96ccecbefbf5086 P 974b802141bd4f13beffef2aa03f88f9 [term 6 FOLLOWER]: Advancing to term 7
W20260129 03:00:03.689360 20404 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260129 03:00:03.740653 20435 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1299723a4476420ab02fd8bfd7aa0645" candidate_uuid: "1c50de89a5a845668696dcb28c3badd6" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "197b880fa39d432a9b472845876bfaf0"
I20260129 03:00:03.684322 21131 rpcz_store.cc:276] 0129 03:00:02.362838 (+ 0us) service_pool.cc:168] Inserting onto call queue
0129 03:00:02.979012 (+616174us) service_pool.cc:225] Handling call
0129 03:00:03.024410 (+ 45398us) raft_consensus.cc:1471] Updating replica for 0 ops
0129 03:00:03.024632 (+ 222us) raft_consensus.cc:1534] Early marking committed up to index 7
0129 03:00:03.050296 (+ 25664us) log.cc:844] Serialized 14 byte log entry
0129 03:00:03.552031 (+501735us) spinlock_profiling.cc:243] Waited 3.2 ms on lock 0x7b1800487240. stack: 00007f8fe044e373 00007f8fe044e270 00007f8fdfda0574 00007f8fe9a5c6b4 00007f8fe9a5c649 00007f8fe92d4c91 00007f8fe92e1671 00007f8fe0462c67 00007f8fe924b709 00007f8fe047f57b 00007f8fe047d81f 00007f8fe047fae2 00007f8fe30c75ab 00007f8fe30cd0c3 00007f8fe30cd5da 00007f8fe35386aa
0129 03:00:03.595585 (+ 43554us) log.cc:844] Serialized 14 byte log entry
0129 03:00:03.620801 (+ 25216us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0129 03:00:03.620859 (+ 58us) raft_consensus.cc:1645] Marking committed up to 7
0129 03:00:03.620920 (+ 61us) raft_consensus.cc:1697] Filling consensus response to leader.
0129 03:00:03.621044 (+ 124us) raft_consensus.cc:1691] UpdateReplicas() finished
0129 03:00:03.621214 (+ 170us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":24865,"prepare.queue_time_us":49818,"prepare.run_cpu_time_us":97,"prepare.run_wall_time_us":10000,"raft.queue_time_us":50517,"raft.run_wall_time_us":11156,"spinlock_wait_cycles":59419648,"thread_start_us":42721,"threads_started":1}
I20260129 03:00:03.725333 20479 leader_election.cc:304] T cedce331e1584ac8b636c9b8a72b2ba0 P 974b802141bd4f13beffef2aa03f88f9 [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 974b802141bd4f13beffef2aa03f88f9; no voters: 197b880fa39d432a9b472845876bfaf0, 1c50de89a5a845668696dcb28c3badd6
I20260129 03:00:03.744328 20479 leader_election.cc:304] T c2633a6fc34c44579de348d208b0c01e P 974b802141bd4f13beffef2aa03f88f9 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 974b802141bd4f13beffef2aa03f88f9; no voters: 197b880fa39d432a9b472845876bfaf0, 1c50de89a5a845668696dcb28c3badd6
I20260129 03:00:03.747493 20479 leader_election.cc:304] T a1a39dcd328e4268ae41b988c66b4a75 P 974b802141bd4f13beffef2aa03f88f9 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 197b880fa39d432a9b472845876bfaf0, 974b802141bd4f13beffef2aa03f88f9; no voters: 1c50de89a5a845668696dcb28c3badd6
I20260129 03:00:03.726787 20516 raft_consensus.cc:2468] T cb4727386a3e48aca5d71cc6c611c197 P 974b802141bd4f13beffef2aa03f88f9 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 197b880fa39d432a9b472845876bfaf0 in term 4.
I20260129 03:00:03.732548 20511 raft_consensus.cc:3060] T 7a9a69cd82724b1faed1d10aa091fc4a P 974b802141bd4f13beffef2aa03f88f9 [term 6 FOLLOWER]: Advancing to term 7
I20260129 03:00:03.750279 20437 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "913fc7d751ed4b7388d889aac30a5f9b" candidate_uuid: "1c50de89a5a845668696dcb28c3badd6" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "197b880fa39d432a9b472845876bfaf0" is_pre_election: true
I20260129 03:00:03.751577 20437 raft_consensus.cc:2410] T 913fc7d751ed4b7388d889aac30a5f9b P 197b880fa39d432a9b472845876bfaf0 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1c50de89a5a845668696dcb28c3badd6 for term 5 because replica has last-logged OpId of term: 3 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 2.
I20260129 03:00:03.664333 20586 raft_consensus.cc:2468] T c891d2bff4ce4568861bbc992e91fbb0 P 1c50de89a5a845668696dcb28c3badd6 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 974b802141bd4f13beffef2aa03f88f9 in term 7.
I20260129 03:00:03.648847 23216 raft_consensus.cc:2804] T 4b1876f9a70b49edb3aa74105e66d3b3 P 974b802141bd4f13beffef2aa03f88f9 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260129 03:00:03.760491 20435 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "8055232cf0004d40a85f3bf983fe00a9" candidate_uuid: "1c50de89a5a845668696dcb28c3badd6" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "197b880fa39d432a9b472845876bfaf0" is_pre_election: true
I20260129 03:00:03.762535 20511 raft_consensus.cc:2468] T 7a9a69cd82724b1faed1d10aa091fc4a P 974b802141bd4f13beffef2aa03f88f9 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 197b880fa39d432a9b472845876bfaf0 in term 7.
I20260129 03:00:03.768482 21129 raft_consensus.cc:1240] T 4e3805ffa70242e8ade66da260dddf66 P 7a9daf5d42a84615a6383cfec9a10608 [term 6 FOLLOWER]: Rejecting Update request from peer 1c50de89a5a845668696dcb28c3badd6 for earlier term 5. Current term is 6. Ops: []
I20260129 03:00:03.772491 20438 raft_consensus.cc:1275] T 4e3805ffa70242e8ade66da260dddf66 P 197b880fa39d432a9b472845876bfaf0 [term 5 FOLLOWER]: Refusing update from remote peer 1c50de89a5a845668696dcb28c3badd6: Log matching property violated. Preceding OpId in replica: term: 3 index: 6. Preceding OpId from leader: term: 5 index: 7. (index mismatch)
I20260129 03:00:03.773572 20438 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fcddab6bca684adf8ea032289a00f90e" candidate_uuid: "1c50de89a5a845668696dcb28c3badd6" candidate_term: 10 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "197b880fa39d432a9b472845876bfaf0" is_pre_election: true
I20260129 03:00:03.774446 20438 raft_consensus.cc:2410] T fcddab6bca684adf8ea032289a00f90e P 197b880fa39d432a9b472845876bfaf0 [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1c50de89a5a845668696dcb28c3badd6 for term 10 because replica has last-logged OpId of term: 8 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 4 index: 4.
I20260129 03:00:03.775295 20438 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "cedce331e1584ac8b636c9b8a72b2ba0" candidate_uuid: "1c50de89a5a845668696dcb28c3badd6" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "197b880fa39d432a9b472845876bfaf0" is_pre_election: true
W20260129 03:00:03.776620 20556 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.211.2:34517, user_credentials={real_user=slave}} blocked reactor thread for 110109us
I20260129 03:00:03.781904 23104 raft_consensus.cc:2749] T 45cc5f3d2688486d9b6dbe7e666dedde P 1c50de89a5a845668696dcb28c3badd6 [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
W20260129 03:00:03.785074 21101 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260129 03:00:03.788712 20440 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a5a05fbcbbcf4916bbc01fd1a5b5e90d" candidate_uuid: "1c50de89a5a845668696dcb28c3badd6" candidate_term: 8 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "197b880fa39d432a9b472845876bfaf0" is_pre_election: true
I20260129 03:00:03.791145 21131 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "cb0b3f259ebc4310927338a8740e87b8" candidate_uuid: "974b802141bd4f13beffef2aa03f88f9" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "7a9daf5d42a84615a6383cfec9a10608" is_pre_election: true
W20260129 03:00:03.793777 20554 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.211.4:40439, user_credentials={real_user=slave}} blocked reactor thread for 44216.7us
I20260129 03:00:03.797344 20435 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "cb4727386a3e48aca5d71cc6c611c197" candidate_uuid: "1c50de89a5a845668696dcb28c3badd6" candidate_term: 5 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "197b880fa39d432a9b472845876bfaf0" is_pre_election: true
I20260129 03:00:03.722782 23232 raft_consensus.cc:493] T 8055232cf0004d40a85f3bf983fe00a9 P 974b802141bd4f13beffef2aa03f88f9 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260129 03:00:03.721606 23200 log.cc:927] Time spent T 96609ac68bdf4993a12e1c23355ca013 P 197b880fa39d432a9b472845876bfaf0: Append to log took a long time: real 0.085s user 0.001s sys 0.000s
I20260129 03:00:03.768800 20516 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a0fcc58d6fb6473cafbd9d34d5195269" candidate_uuid: "1c50de89a5a845668696dcb28c3badd6" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "974b802141bd4f13beffef2aa03f88f9" is_pre_election: true
I20260129 03:00:03.769572 20586 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7d9494f5584d4d5db8b9adf1c8fc888b" candidate_uuid: "974b802141bd4f13beffef2aa03f88f9" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "1c50de89a5a845668696dcb28c3badd6" is_pre_election: true
I20260129 03:00:03.870662 20435 raft_consensus.cc:2410] T cb4727386a3e48aca5d71cc6c611c197 P 197b880fa39d432a9b472845876bfaf0 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1c50de89a5a845668696dcb28c3badd6 for term 5 because replica has last-logged OpId of term: 3 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 1.
I20260129 03:00:03.675168 20583 raft_consensus.cc:2468] T 42d27a28d89b409e8807ec0efd97af69 P 1c50de89a5a845668696dcb28c3badd6 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7a9daf5d42a84615a6383cfec9a10608 in term 8.
W20260129 03:00:03.883095 20479 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.211.1:42027, user_credentials={real_user=slave}} blocked reactor thread for 120319us
I20260129 03:00:03.884284 20479 leader_election.cc:304] T 3ccac82a01a2445fa5b0ce6c0fcf2295 P 974b802141bd4f13beffef2aa03f88f9 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 974b802141bd4f13beffef2aa03f88f9; no voters: 197b880fa39d432a9b472845876bfaf0, 7a9daf5d42a84615a6383cfec9a10608
I20260129 03:00:03.723790 23026 raft_consensus.cc:2804] T c2633a6fc34c44579de348d208b0c01e P 1c50de89a5a845668696dcb28c3badd6 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260129 03:00:03.892643 20437 raft_consensus.cc:1240] T 7a9a69cd82724b1faed1d10aa091fc4a P 197b880fa39d432a9b472845876bfaf0 [term 7 FOLLOWER]: Rejecting Update request from peer 1c50de89a5a845668696dcb28c3badd6 for earlier term 6. Current term is 7. Ops: []
I20260129 03:00:03.894964 20516 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a609ac02fae84725ab743f95f51d33ce" candidate_uuid: "7a9daf5d42a84615a6383cfec9a10608" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "974b802141bd4f13beffef2aa03f88f9"
I20260129 03:00:03.896821 20556 leader_election.cc:304] T a0fcc58d6fb6473cafbd9d34d5195269 P 1c50de89a5a845668696dcb28c3badd6 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1c50de89a5a845668696dcb28c3badd6; no voters: 197b880fa39d432a9b472845876bfaf0, 974b802141bd4f13beffef2aa03f88f9
I20260129 03:00:03.735975 23097 raft_consensus.cc:3060] T 368953c9102643b2a21f99af36b51905 P 974b802141bd4f13beffef2aa03f88f9 [term 6 FOLLOWER]: Advancing to term 7
I20260129 03:00:03.905952 20586 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "cb0b3f259ebc4310927338a8740e87b8" candidate_uuid: "974b802141bd4f13beffef2aa03f88f9" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "1c50de89a5a845668696dcb28c3badd6" is_pre_election: true
I20260129 03:00:03.650198 23146 raft_consensus.cc:515] T 913fc7d751ed4b7388d889aac30a5f9b P 974b802141bd4f13beffef2aa03f88f9 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 03:00:03.768554 22982 raft_consensus.cc:2749] T cedce331e1584ac8b636c9b8a72b2ba0 P 974b802141bd4f13beffef2aa03f88f9 [term 7 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
I20260129 03:00:03.905952 20435 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e4d18c35b0a7493e85d0a4b92956a48f" candidate_uuid: "1c50de89a5a845668696dcb28c3badd6" candidate_term: 6 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "197b880fa39d432a9b472845876bfaf0" is_pre_election: true
I20260129 03:00:03.903592 20516 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:48544 (request call id 285) took 1049 ms. Trace:
I20260129 03:00:03.960443 20516 rpcz_store.cc:276] 0129 03:00:02.854366 (+ 0us) service_pool.cc:168] Inserting onto call queue
0129 03:00:03.894647 (+1040281us) service_pool.cc:225] Handling call
0129 03:00:03.903510 (+ 8863us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260129 03:00:03.910115 20438 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a8479d1fe0bd41e5946186e3a6ca8c7a" candidate_uuid: "1c50de89a5a845668696dcb28c3badd6" candidate_term: 6 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "197b880fa39d432a9b472845876bfaf0" is_pre_election: true
I20260129 03:00:03.911141 20586 raft_consensus.cc:2368] T cb0b3f259ebc4310927338a8740e87b8 P 1c50de89a5a845668696dcb28c3badd6 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 974b802141bd4f13beffef2aa03f88f9 for earlier term 6. Current term is 7.
I20260129 03:00:03.962139 20438 raft_consensus.cc:2410] T a8479d1fe0bd41e5946186e3a6ca8c7a P 197b880fa39d432a9b472845876bfaf0 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1c50de89a5a845668696dcb28c3badd6 for term 6 because replica has last-logged OpId of term: 3 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 2.
I20260129 03:00:03.962389 20586 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "dc2f4cb752cf4d59b5f6bc689f5d8104" candidate_uuid: "7a9daf5d42a84615a6383cfec9a10608" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "1c50de89a5a845668696dcb28c3badd6" is_pre_election: true
I20260129 03:00:03.963122 20516 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00e79d8f25a34676b1d866eb2068a2b2" candidate_uuid: "197b880fa39d432a9b472845876bfaf0" candidate_term: 8 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "974b802141bd4f13beffef2aa03f88f9"
I20260129 03:00:03.963474 20586 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "42d27a28d89b409e8807ec0efd97af69" candidate_uuid: "197b880fa39d432a9b472845876bfaf0" candidate_term: 7 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "1c50de89a5a845668696dcb28c3badd6" is_pre_election: true
I20260129 03:00:03.769512 23216 raft_consensus.cc:493] T 4b1876f9a70b49edb3aa74105e66d3b3 P 974b802141bd4f13beffef2aa03f88f9 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260129 03:00:03.920441 21136 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34060 (request call id 517) took 3318 ms. Trace:
I20260129 03:00:03.796888 23113 raft_consensus.cc:3060] T c2633a6fc34c44579de348d208b0c01e P 974b802141bd4f13beffef2aa03f88f9 [term 2 FOLLOWER]: Advancing to term 3
I20260129 03:00:03.808089 22694 raft_consensus.cc:2804] T a1a39dcd328e4268ae41b988c66b4a75 P 974b802141bd4f13beffef2aa03f88f9 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260129 03:00:03.963776 20516 raft_consensus.cc:3060] T 00e79d8f25a34676b1d866eb2068a2b2 P 974b802141bd4f13beffef2aa03f88f9 [term 7 FOLLOWER]: Advancing to term 8
W20260129 03:00:04.047645 20482 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260129 03:00:04.081590 20515 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:48544 (request call id 263) took 3039 ms. Trace:
I20260129 03:00:04.082319 20515 rpcz_store.cc:276] 0129 03:00:01.041832 (+ 0us) service_pool.cc:168] Inserting onto call queue
0129 03:00:01.782086 (+740254us) service_pool.cc:225] Handling call
0129 03:00:01.850543 (+ 68457us) raft_consensus.cc:1471] Updating replica for 0 ops
0129 03:00:02.865894 (+1015351us) raft_consensus.cc:1697] Filling consensus response to leader.
0129 03:00:04.081240 (+1215346us) spinlock_profiling.cc:243] Waited 990 ms on lock 0x7b5400167454. stack: 00007f8fe044e373 00007f8fe044e270 00007f8fdfda0574 00007f8fe9a5c6b4 00007f8fe9a5c649 00007f8fe34975a1 00007f8fe352ea81 00007f8fe352d4be 00007f8fe84e1f49 00007f8fe1a6ed34 00007f8fe1a6ecc2 00007f8fe1a6ebf1 00007f8fe1a6eb6c 00007f8fe1a6de22 00007f8fe186d34c 00007f8fe186c786
0129 03:00:04.081503 (+ 263us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2276040960}
I20260129 03:00:04.024349 21136 rpcz_store.cc:276] 0129 03:00:00.602022 (+ 0us) service_pool.cc:168] Inserting onto call queue
0129 03:00:00.602472 (+ 450us) service_pool.cc:225] Handling call
0129 03:00:00.649931 (+ 47459us) raft_consensus.cc:1471] Updating replica for 1 ops
0129 03:00:00.968356 (+318425us) raft_consensus.cc:1697] Filling consensus response to leader.
0129 03:00:03.920064 (+2951708us) spinlock_profiling.cc:243] Waited 289 ms on lock 0x7b54005631d4. stack: 00007f8fe044e373 00007f8fe044e270 00007f8fdfda0574 00007f8fe9a5c6b4 00007f8fe9a5c649 00007f8fe34975a1 00007f8fe352ea81 00007f8fe352d4be 00007f8fe84e1f49 00007f8fe1a6ed34 00007f8fe1a6ecc2 00007f8fe1a6ebf1 00007f8fe1a6eb6c 00007f8fe1a6de22 00007f8fe186d34c 00007f8fe186c786
0129 03:00:03.920356 (+ 292us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":665823104}
I20260129 03:00:03.900216 23026 raft_consensus.cc:493] T c2633a6fc34c44579de348d208b0c01e P 1c50de89a5a845668696dcb28c3badd6 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260129 03:00:04.085713 20515 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "cb4727386a3e48aca5d71cc6c611c197" candidate_uuid: "1c50de89a5a845668696dcb28c3badd6" candidate_term: 5 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "974b802141bd4f13beffef2aa03f88f9" is_pre_election: true
I20260129 03:00:04.020045 20511 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "cedce331e1584ac8b636c9b8a72b2ba0" candidate_uuid: "1c50de89a5a845668696dcb28c3badd6" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "974b802141bd4f13beffef2aa03f88f9" is_pre_election: true
I20260129 03:00:03.770133 20513 raft_consensus.cc:2468] T 75b0b6e82ddc4f92b1fe1d94f9a0b73b P 974b802141bd4f13beffef2aa03f88f9 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7a9daf5d42a84615a6383cfec9a10608 in term 4.
I20260129 03:00:04.034014 20581 raft_consensus.cc:1275] T a8479d1fe0bd41e5946186e3a6ca8c7a P 1c50de89a5a845668696dcb28c3badd6 [term 5 FOLLOWER]: Refusing update from remote peer 974b802141bd4f13beffef2aa03f88f9: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 5 index: 3. (index mismatch)
I20260129 03:00:03.760059 23168 raft_consensus.cc:697] T a609ac02fae84725ab743f95f51d33ce P 7a9daf5d42a84615a6383cfec9a10608 [term 8 LEADER]: Becoming Leader. State: Replica: 7a9daf5d42a84615a6383cfec9a10608, State: Running, Role: LEADER
I20260129 03:00:04.036784 20586 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:35754 (request call id 1776) took 1054 ms. Trace:
I20260129 03:00:04.144735 20586 rpcz_store.cc:276] 0129 03:00:02.983020 (+ 0us) service_pool.cc:168] Inserting onto call queue
0129 03:00:03.963257 (+980237us) service_pool.cc:225] Handling call
0129 03:00:04.036680 (+ 73423us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260129 03:00:04.147894 20586 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e4d18c35b0a7493e85d0a4b92956a48f" candidate_uuid: "197b880fa39d432a9b472845876bfaf0" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "1c50de89a5a845668696dcb28c3badd6" is_pre_election: true
W20260129 03:00:04.093376 20555 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260129 03:00:04.138578 20590 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:35754 (request call id 1721) took 3061 ms (client timeout 1662 ms). Trace:
W20260129 03:00:04.151494 20590 rpcz_store.cc:269] 0129 03:00:00.973017 (+ 0us) service_pool.cc:168] Inserting onto call queue
0129 03:00:00.986165 (+ 13148us) service_pool.cc:225] Handling call
0129 03:00:04.033431 (+3047266us) spinlock_profiling.cc:243] Waited 45.9 ms on lock 0x7b54003a0054. stack: 00007f8fe044e373 00007f8fe044e270 00007f8fdfda0574 00007f8fe9a5c6b4 00007f8fe9a5c649 00007f8fe34975a1 00007f8fe3531f7a 00007f8fe84e3647 00007f8fe1a73834 00007f8fe1a737c2 00007f8fe1a736f1 00007f8fe1a7366c 00007f8fe1a72922 00007f8fe186d34c 00007f8fe186c786 00007f8fe186c12f
0129 03:00:04.033610 (+ 179us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":105563520}
I20260129 03:00:04.138846 20509 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57584 (request call id 1379) took 6644 ms. Trace:
I20260129 03:00:03.909960 22977 raft_consensus.cc:2749] T a0fcc58d6fb6473cafbd9d34d5195269 P 1c50de89a5a845668696dcb28c3badd6 [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260129 03:00:03.891762 23232 raft_consensus.cc:3060] T 8055232cf0004d40a85f3bf983fe00a9 P 974b802141bd4f13beffef2aa03f88f9 [term 4 FOLLOWER]: Advancing to term 5
W20260129 03:00:03.948853 23206 consensus_meta.cc:287] T 2ac659d732314e1288849f08fe5e074e P 7a9daf5d42a84615a6383cfec9a10608: Time spent flushing consensus metadata: real 0.538s user 0.011s sys 0.000s
I20260129 03:00:04.157521 20512 raft_consensus.cc:2393] T 913fc7d751ed4b7388d889aac30a5f9b P 974b802141bd4f13beffef2aa03f88f9 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1c50de89a5a845668696dcb28c3badd6 in current term 5: Already voted for candidate 974b802141bd4f13beffef2aa03f88f9 in this term.
I20260129 03:00:04.159060 20437 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1ce4ddf5b10e4ae8b193b13b2e5936ee" candidate_uuid: "1c50de89a5a845668696dcb28c3badd6" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "197b880fa39d432a9b472845876bfaf0" is_pre_election: true
I20260129 03:00:04.166083 20515 raft_consensus.cc:2468] T cb4727386a3e48aca5d71cc6c611c197 P 974b802141bd4f13beffef2aa03f88f9 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1c50de89a5a845668696dcb28c3badd6 in term 4.
I20260129 03:00:04.166621 20513 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a8479d1fe0bd41e5946186e3a6ca8c7a" candidate_uuid: "1c50de89a5a845668696dcb28c3badd6" candidate_term: 6 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "974b802141bd4f13beffef2aa03f88f9" is_pre_election: true
I20260129 03:00:04.206626 20515 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:57584 (request call id 1422) took 1291 ms. Trace:
I20260129 03:00:04.206913 20513 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:57584 (request call id 1423) took 1285 ms. Trace:
I20260129 03:00:04.207057 20515 rpcz_store.cc:276] 0129 03:00:02.915538 (+ 0us) service_pool.cc:168] Inserting onto call queue
0129 03:00:04.084614 (+1169076us) service_pool.cc:225] Handling call
0129 03:00:04.206562 (+121948us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260129 03:00:04.168071 20511 raft_consensus.cc:2393] T cedce331e1584ac8b636c9b8a72b2ba0 P 974b802141bd4f13beffef2aa03f88f9 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1c50de89a5a845668696dcb28c3badd6 in current term 7: Already voted for candidate 974b802141bd4f13beffef2aa03f88f9 in this term.
I20260129 03:00:04.024923 23202 leader_election.cc:290] T 75f45538dc704d9a8ec4ff8ad9f3ea5b P 974b802141bd4f13beffef2aa03f88f9 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3:37735), 7a9daf5d42a84615a6383cfec9a10608 (127.18.211.4:40439)
I20260129 03:00:04.153115 20509 rpcz_store.cc:276] 0129 02:59:57.387792 (+ 0us) service_pool.cc:168] Inserting onto call queue
0129 02:59:57.645473 (+257681us) service_pool.cc:225] Handling call
0129 02:59:57.916457 (+270984us) raft_consensus.cc:1471] Updating replica for 1 ops
0129 02:59:57.916938 (+ 481us) raft_consensus.cc:1534] Early marking committed up to index 0
0129 02:59:57.916982 (+ 44us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0129 02:59:57.917566 (+ 584us) log.cc:844] Serialized 26 byte log entry
0129 03:00:00.256445 (+2338879us) spinlock_profiling.cc:243] Waited 37.9 ms on lock 0x7b18000e12e0. stack: 00007f8fe044e373 00007f8fe044e270 00007f8fdfda0574 00007f8fe9a5c6b4 00007f8fe9a5c649 00007f8fe92d4c91 00007f8fe92e1671 00007f8fe30d585c 00007f8fe04632eb 00007f8fe924b709 00007f8fe047f57b 00007f8fe047d81f 00007f8fe047fae2 00007f8fe30c75ab 00007f8fe30cd0c3 00007f8fe30cd471
0129 03:00:00.256603 (+ 158us) raft_consensus.cc:1645] Marking committed up to 6
0129 03:00:00.658077 (+401474us) log.cc:844] Serialized 14 byte log entry
0129 03:00:01.158259 (+500182us) raft_consensus.cc:1697] Filling consensus response to leader.
0129 03:00:01.158355 (+ 96us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0129 03:00:01.158539 (+ 184us) raft_consensus.cc:1685] finished
0129 03:00:01.197974 (+ 39435us) raft_consensus.cc:1691] UpdateReplicas() finished
0129 03:00:04.032445 (+2834471us) spinlock_profiling.cc:243] Waited 271 ms on lock 0x7b54002ee850. stack: 00007f8fe044e373 00007f8fe044e270 00007f8fdfda0574 00007f8fe9a5c6b4 00007f8fe9a5c649 00007f8fe34975a1 00007f8fe352d52d 00007f8fe84e1f49 00007f8fe1a6ed34 00007f8fe1a6ecc2 00007f8fe1a6ebf1 00007f8fe1a6eb6c 00007f8fe1a6de22 00007f8fe186d34c 00007f8fe186c786 00007f8fe186c12f
0129 03:00:04.032597 (+ 152us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":10286,"prepare.queue_time_us":734407,"prepare.run_cpu_time_us":52,"prepare.run_wall_time_us":4290,"raft.queue_time_us":5211,"raft.run_cpu_time_us":379,"raft.run_wall_time_us":42051,"spinlock_wait_cycles":709474816,"thread_start_us":897534,"threads_started":3,"wal-append.queue_time_us":888505,"wal-append.run_cpu_time_us":27798,"wal-append.run_wall_time_us":3225058}
I20260129 03:00:04.207274 20590 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0340229d521b432580ae926f6062e3f7" candidate_uuid: "974b802141bd4f13beffef2aa03f88f9" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "1c50de89a5a845668696dcb28c3badd6"
I20260129 03:00:03.731927 23182 consensus_queue.cc:237] T 42d27a28d89b409e8807ec0efd97af69 P 7a9daf5d42a84615a6383cfec9a10608 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 6.8, Last appended by leader: 8, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } }
I20260129 03:00:03.906538 23055 consensus_queue.cc:1059] T 4e420f90884f4796997b5ea41e49e411 P 974b802141bd4f13beffef2aa03f88f9 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 }, Status: INVALID_TERM, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.254s
I20260129 03:00:03.905167 22748 raft_consensus.cc:493] T 2e58e9a5a8aa4cecafb464addb4f35ed P 1c50de89a5a845668696dcb28c3badd6 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 7a9daf5d42a84615a6383cfec9a10608)
I20260129 03:00:04.035295 23216 raft_consensus.cc:3060] T 4b1876f9a70b49edb3aa74105e66d3b3 P 974b802141bd4f13beffef2aa03f88f9 [term 5 FOLLOWER]: Advancing to term 6
I20260129 03:00:04.207401 20513 rpcz_store.cc:276] 0129 03:00:02.921623 (+ 0us) service_pool.cc:168] Inserting onto call queue
0129 03:00:04.166361 (+1244738us) service_pool.cc:225] Handling call
0129 03:00:04.206850 (+ 40489us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260129 03:00:04.208174 20440 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c2633a6fc34c44579de348d208b0c01e" candidate_uuid: "1c50de89a5a845668696dcb28c3badd6" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "197b880fa39d432a9b472845876bfaf0" is_pre_election: true
I20260129 03:00:04.233244 20440 raft_consensus.cc:2468] T c2633a6fc34c44579de348d208b0c01e P 197b880fa39d432a9b472845876bfaf0 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1c50de89a5a845668696dcb28c3badd6 in term 3.
I20260129 03:00:04.212899 20511 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:57584 (request call id 1421) took 1297 ms. Trace:
I20260129 03:00:04.214164 20590 raft_consensus.cc:3060] T 0340229d521b432580ae926f6062e3f7 P 1c50de89a5a845668696dcb28c3badd6 [term 6 FOLLOWER]: Advancing to term 7
I20260129 03:00:04.156062 23026 raft_consensus.cc:3060] T c2633a6fc34c44579de348d208b0c01e P 1c50de89a5a845668696dcb28c3badd6 [term 3 FOLLOWER]: Advancing to term 4
I20260129 03:00:04.085884 22982 raft_consensus.cc:493] T c934d31efd3f4e5195605ddc354d14e1 P 974b802141bd4f13beffef2aa03f88f9 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 197b880fa39d432a9b472845876bfaf0)
I20260129 03:00:04.167470 22694 raft_consensus.cc:493] T a1a39dcd328e4268ae41b988c66b4a75 P 974b802141bd4f13beffef2aa03f88f9 [term 4 FOLLOWER]: Starting leader election (detected failure of leader 1c50de89a5a845668696dcb28c3badd6)
W20260129 03:00:04.234268 21102 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.552s user 0.008s sys 0.023s
I20260129 03:00:04.234216 20511 rpcz_store.cc:276] 0129 03:00:02.915214 (+ 0us) service_pool.cc:168] Inserting onto call queue
0129 03:00:04.019660 (+1104446us) service_pool.cc:225] Handling call
0129 03:00:04.212839 (+193179us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260129 03:00:04.241539 20511 raft_consensus.cc:3060] T cb0b3f259ebc4310927338a8740e87b8 P 974b802141bd4f13beffef2aa03f88f9 [term 5 FOLLOWER]: Advancing to term 7
I20260129 03:00:04.171952 23053 raft_consensus.cc:493] T 96609ac68bdf4993a12e1c23355ca013 P 1c50de89a5a845668696dcb28c3badd6 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 7a9daf5d42a84615a6383cfec9a10608)
I20260129 03:00:04.241979 20516 raft_consensus.cc:2468] T 00e79d8f25a34676b1d866eb2068a2b2 P 974b802141bd4f13beffef2aa03f88f9 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 197b880fa39d432a9b472845876bfaf0 in term 8.
I20260129 03:00:04.231941 22983 raft_consensus.cc:2749] T 3ccac82a01a2445fa5b0ce6c0fcf2295 P 974b802141bd4f13beffef2aa03f88f9 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260129 03:00:04.238893 20437 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0f20af7fbe5d4db89a6f17948d98912c" candidate_uuid: "1c50de89a5a845668696dcb28c3badd6" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "197b880fa39d432a9b472845876bfaf0" is_pre_election: true
I20260129 03:00:04.244374 20515 tablet_service.cc:2066] Received LeaderStepDown RPC: tablet_id: "c891d2bff4ce4568861bbc992e91fbb0"
dest_uuid: "974b802141bd4f13beffef2aa03f88f9"
mode: GRACEFUL
new_leader_uuid: "1c50de89a5a845668696dcb28c3badd6"
from {username='slave'} at 127.0.0.1:52008
I20260129 03:00:04.248998 20513 raft_consensus.cc:1240] T 4f5a77e94fef4b16a2ba7b067c8b144a P 974b802141bd4f13beffef2aa03f88f9 [term 4 FOLLOWER]: Rejecting Update request from peer 1c50de89a5a845668696dcb28c3badd6 for earlier term 3. Current term is 4. Ops: []
I20260129 03:00:04.265451 20515 raft_consensus.cc:606] T c891d2bff4ce4568861bbc992e91fbb0 P 974b802141bd4f13beffef2aa03f88f9 [term 7 FOLLOWER]: Received request to transfer leadership to TS 1c50de89a5a845668696dcb28c3badd6
I20260129 03:00:04.266130 20515 raft_consensus.cc:614] T c891d2bff4ce4568861bbc992e91fbb0 P 974b802141bd4f13beffef2aa03f88f9 [term 7 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260129 03:00:04.266404 20513 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57584 (request call id 1427) took 1246 ms. Trace:
I20260129 03:00:04.266963 20513 rpcz_store.cc:276] 0129 03:00:03.020308 (+ 0us) service_pool.cc:168] Inserting onto call queue
0129 03:00:04.233115 (+1212807us) service_pool.cc:225] Handling call
0129 03:00:04.248759 (+ 15644us) raft_consensus.cc:1471] Updating replica for 0 ops
0129 03:00:04.266030 (+ 17271us) raft_consensus.cc:1697] Filling consensus response to leader.
0129 03:00:04.266325 (+ 295us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20260129 03:00:04.268059 21102 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.586s user 0.008s sys 0.024s
I20260129 03:00:04.268471 20516 raft_consensus.cc:1275] T 7d9494f5584d4d5db8b9adf1c8fc888b P 974b802141bd4f13beffef2aa03f88f9 [term 4 FOLLOWER]: Refusing update from remote peer 1c50de89a5a845668696dcb28c3badd6: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20260129 03:00:04.269773 20516 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57584 (request call id 1428) took 1235 ms. Trace:
W20260129 03:00:04.238154 20480 thread.cc:641] dns-resolver [worker] (thread pool) Time spent creating pthread: real 0.579s user 0.018s sys 0.006s
W20260129 03:00:04.270809 20480 thread.cc:608] dns-resolver [worker] (thread pool) Time spent starting thread: real 0.612s user 0.018s sys 0.006s
W20260129 03:00:04.271488 20480 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.211.3:37735, user_credentials={real_user=slave}} blocked reactor thread for 612978us
I20260129 03:00:04.225867 22925 leader_election.cc:290] T 18f30ebaa4174fd28c958e72d0cc7e24 P 197b880fa39d432a9b472845876bfaf0 [CANDIDATE]: Term 7 election: Requested vote from peers 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3:37735), 7a9daf5d42a84615a6383cfec9a10608 (127.18.211.4:40439)
I20260129 03:00:04.238340 23202 raft_consensus.cc:493] T e7d83d4ccf684e76acb29a0dc844e1b8 P 974b802141bd4f13beffef2aa03f88f9 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260129 03:00:04.283427 20511 raft_consensus.cc:1275] T cb0b3f259ebc4310927338a8740e87b8 P 974b802141bd4f13beffef2aa03f88f9 [term 7 FOLLOWER]: Refusing update from remote peer 7a9daf5d42a84615a6383cfec9a10608: Log matching property violated. Preceding OpId in replica: term: 5 index: 7. Preceding OpId from leader: term: 7 index: 8. (index mismatch)
I20260129 03:00:04.270226 20516 rpcz_store.cc:276] 0129 03:00:03.034845 (+ 0us) service_pool.cc:168] Inserting onto call queue
0129 03:00:04.243853 (+1209008us) service_pool.cc:225] Handling call
0129 03:00:04.245916 (+ 2063us) raft_consensus.cc:1471] Updating replica for 0 ops
0129 03:00:04.269404 (+ 23488us) raft_consensus.cc:1697] Filling consensus response to leader.
0129 03:00:04.269700 (+ 296us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260129 03:00:04.242841 22996 raft_consensus.cc:3055] T 4f5a77e94fef4b16a2ba7b067c8b144a P 1c50de89a5a845668696dcb28c3badd6 [term 3 LEADER]: Stepping down as leader of term 3
I20260129 03:00:04.211505 23206 raft_consensus.cc:515] T 2ac659d732314e1288849f08fe5e074e P 7a9daf5d42a84615a6383cfec9a10608 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } }
I20260129 03:00:04.237538 22748 raft_consensus.cc:515] T 2e58e9a5a8aa4cecafb464addb4f35ed P 1c50de89a5a845668696dcb28c3badd6 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } }
I20260129 03:00:04.295987 20515 raft_consensus.cc:1240] T 75be4bd6bd45474795fd472eff2953ac P 974b802141bd4f13beffef2aa03f88f9 [term 10 FOLLOWER]: Rejecting Update request from peer 1c50de89a5a845668696dcb28c3badd6 for earlier term 9. Current term is 10. Ops: []
I20260129 03:00:04.248406 22982 raft_consensus.cc:515] T c934d31efd3f4e5195605ddc354d14e1 P 974b802141bd4f13beffef2aa03f88f9 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } }
I20260129 03:00:04.304212 20515 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57584 (request call id 1430) took 1253 ms. Trace:
I20260129 03:00:04.294323 21102 leader_election.cc:304] T 75b0b6e82ddc4f92b1fe1d94f9a0b73b P 7a9daf5d42a84615a6383cfec9a10608 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 7a9daf5d42a84615a6383cfec9a10608, 974b802141bd4f13beffef2aa03f88f9; no voters: 1c50de89a5a845668696dcb28c3badd6
I20260129 03:00:04.304642 20515 rpcz_store.cc:276] 0129 03:00:03.062276 (+ 0us) service_pool.cc:168] Inserting onto call queue
0129 03:00:04.281943 (+1219667us) service_pool.cc:225] Handling call
0129 03:00:04.295706 (+ 13763us) raft_consensus.cc:1471] Updating replica for 0 ops
0129 03:00:04.303839 (+ 8133us) raft_consensus.cc:1697] Filling consensus response to leader.
0129 03:00:04.304143 (+ 304us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260129 03:00:04.248503 22694 raft_consensus.cc:3060] T a1a39dcd328e4268ae41b988c66b4a75 P 974b802141bd4f13beffef2aa03f88f9 [term 4 FOLLOWER]: Advancing to term 5
I20260129 03:00:04.279942 23079 leader_election.cc:290] T 4e3805ffa70242e8ade66da260dddf66 P 7a9daf5d42a84615a6383cfec9a10608 [CANDIDATE]: Term 6 election: Requested vote from peers 197b880fa39d432a9b472845876bfaf0 (127.18.211.1:42027), 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3:37735)
I20260129 03:00:04.283752 22992 raft_consensus.cc:493] T 00e79d8f25a34676b1d866eb2068a2b2 P 7a9daf5d42a84615a6383cfec9a10608 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260129 03:00:04.305696 23125 log.cc:927] Time spent T 76f713c3331540d2a5b76164103e9eb1 P 1c50de89a5a845668696dcb28c3badd6: Append to log took a long time: real 3.704s user 0.025s sys 0.061s
I20260129 03:00:04.317943 20590 raft_consensus.cc:2468] T 0340229d521b432580ae926f6062e3f7 P 1c50de89a5a845668696dcb28c3badd6 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 974b802141bd4f13beffef2aa03f88f9 in term 7.
I20260129 03:00:04.269986 23053 raft_consensus.cc:515] T 96609ac68bdf4993a12e1c23355ca013 P 1c50de89a5a845668696dcb28c3badd6 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } attrs { replace: true } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } }
I20260129 03:00:04.240345 22977 raft_consensus.cc:493] T 368953c9102643b2a21f99af36b51905 P 1c50de89a5a845668696dcb28c3badd6 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260129 03:00:04.303381 22996 raft_consensus.cc:740] T 4f5a77e94fef4b16a2ba7b067c8b144a P 1c50de89a5a845668696dcb28c3badd6 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 1c50de89a5a845668696dcb28c3badd6, State: Running, Role: LEADER
I20260129 03:00:04.330277 22977 raft_consensus.cc:515] T 368953c9102643b2a21f99af36b51905 P 1c50de89a5a845668696dcb28c3badd6 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 03:00:04.319737 22992 raft_consensus.cc:515] T 00e79d8f25a34676b1d866eb2068a2b2 P 7a9daf5d42a84615a6383cfec9a10608 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } }
I20260129 03:00:04.279742 23168 consensus_queue.cc:237] T a609ac02fae84725ab743f95f51d33ce P 7a9daf5d42a84615a6383cfec9a10608 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 6.7, Last appended by leader: 7, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } }
I20260129 03:00:04.291174 23202 raft_consensus.cc:515] T e7d83d4ccf684e76acb29a0dc844e1b8 P 974b802141bd4f13beffef2aa03f88f9 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 03:00:04.332265 23037 raft_consensus.cc:2804] T 75b0b6e82ddc4f92b1fe1d94f9a0b73b P 7a9daf5d42a84615a6383cfec9a10608 [term 4 FOLLOWER]: Leader election won for term 4
W20260129 03:00:04.337667 23242 log.cc:927] Time spent T 18f30ebaa4174fd28c958e72d0cc7e24 P 7a9daf5d42a84615a6383cfec9a10608: Append to log took a long time: real 0.394s user 0.001s sys 0.000s
W20260129 03:00:04.339365 23183 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.571s user 0.031s sys 0.043s
I20260129 03:00:04.353849 22996 consensus_queue.cc:260] T 4f5a77e94fef4b16a2ba7b067c8b144a P 1c50de89a5a845668696dcb28c3badd6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.5, Last appended by leader: 5, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } }
I20260129 03:00:04.324497 23079 consensus_queue.cc:1048] T 75f45538dc704d9a8ec4ff8ad9f3ea5b P 7a9daf5d42a84615a6383cfec9a10608 [LEADER]: Connected to new peer: Peer: permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.026s
I20260129 03:00:04.346192 22992 leader_election.cc:290] T 00e79d8f25a34676b1d866eb2068a2b2 P 7a9daf5d42a84615a6383cfec9a10608 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 197b880fa39d432a9b472845876bfaf0 (127.18.211.1:42027), 974b802141bd4f13beffef2aa03f88f9 (127.18.211.2:34517)
I20260129 03:00:04.346239 23094 raft_consensus.cc:493] T fd8311b779af4e1fa1402b2a02a602f2 P 7a9daf5d42a84615a6383cfec9a10608 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260129 03:00:04.345844 23243 log.cc:927] Time spent T 1ce4ddf5b10e4ae8b193b13b2e5936ee P 197b880fa39d432a9b472845876bfaf0: Append to log took a long time: real 0.062s user 0.000s sys 0.000s
I20260129 03:00:04.356524 23131 leader_election.cc:290] T 913fc7d751ed4b7388d889aac30a5f9b P 197b880fa39d432a9b472845876bfaf0 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 974b802141bd4f13beffef2aa03f88f9 (127.18.211.2:34517), 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3:37735)
I20260129 03:00:04.354827 23202 leader_election.cc:290] T e7d83d4ccf684e76acb29a0dc844e1b8 P 974b802141bd4f13beffef2aa03f88f9 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 197b880fa39d432a9b472845876bfaf0 (127.18.211.1:42027), 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3:37735)
I20260129 03:00:04.345715 23035 raft_consensus.cc:3055] T 1299723a4476420ab02fd8bfd7aa0645 P 7a9daf5d42a84615a6383cfec9a10608 [term 8 LEADER]: Stepping down as leader of term 8
W20260129 03:00:04.366236 23183 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.595s user 0.031s sys 0.043s
I20260129 03:00:04.370674 23035 raft_consensus.cc:740] T 1299723a4476420ab02fd8bfd7aa0645 P 7a9daf5d42a84615a6383cfec9a10608 [term 8 LEADER]: Becoming Follower/Learner. State: Replica: 7a9daf5d42a84615a6383cfec9a10608, State: Running, Role: LEADER
I20260129 03:00:04.363219 23037 raft_consensus.cc:697] T 75b0b6e82ddc4f92b1fe1d94f9a0b73b P 7a9daf5d42a84615a6383cfec9a10608 [term 4 LEADER]: Becoming Leader. State: Replica: 7a9daf5d42a84615a6383cfec9a10608, State: Running, Role: LEADER
I20260129 03:00:04.423974 19276 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 2
I20260129 03:00:04.427963 20511 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:48544 (request call id 286) took 1394 ms. Trace:
I20260129 03:00:04.464325 20511 rpcz_store.cc:276] 0129 03:00:03.033916 (+ 0us) service_pool.cc:168] Inserting onto call queue
0129 03:00:04.241000 (+1207084us) service_pool.cc:225] Handling call
0129 03:00:04.241382 (+ 382us) raft_consensus.cc:1471] Updating replica for 0 ops
0129 03:00:04.284281 (+ 42899us) raft_consensus.cc:1697] Filling consensus response to leader.
0129 03:00:04.427674 (+143393us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260129 03:00:04.430135 20515 raft_consensus.cc:1240] T 3e2d15f8640b407195210c476e09df91 P 974b802141bd4f13beffef2aa03f88f9 [term 4 FOLLOWER]: Rejecting Update request from peer 1c50de89a5a845668696dcb28c3badd6 for earlier term 3. Current term is 4. Ops: []
I20260129 03:00:04.431550 20438 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0340229d521b432580ae926f6062e3f7" candidate_uuid: "1c50de89a5a845668696dcb28c3badd6" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "197b880fa39d432a9b472845876bfaf0" is_pre_election: true
W20260129 03:00:04.468174 20479 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.211.1:42027, user_credentials={real_user=slave}} blocked reactor thread for 583869us
I20260129 03:00:04.434458 20437 raft_consensus.cc:2393] T 0f20af7fbe5d4db89a6f17948d98912c P 197b880fa39d432a9b472845876bfaf0 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1c50de89a5a845668696dcb28c3badd6 in current term 7: Already voted for candidate 197b880fa39d432a9b472845876bfaf0 in this term.
I20260129 03:00:04.468673 20438 raft_consensus.cc:2468] T 0340229d521b432580ae926f6062e3f7 P 197b880fa39d432a9b472845876bfaf0 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1c50de89a5a845668696dcb28c3badd6 in term 5.
I20260129 03:00:04.433652 20435 raft_consensus.cc:1240] T 42d27a28d89b409e8807ec0efd97af69 P 197b880fa39d432a9b472845876bfaf0 [term 8 FOLLOWER]: Rejecting Update request from peer 1c50de89a5a845668696dcb28c3badd6 for earlier term 6. Current term is 8. Ops: []
W20260129 03:00:04.436949 20441 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.665s user 0.014s sys 0.023s
W20260129 03:00:04.437772 20556 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260129 03:00:04.471208 20441 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.699s user 0.015s sys 0.023s
I20260129 03:00:04.440124 20590 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:48526 (request call id 337) took 2355 ms. Trace:
W20260129 03:00:04.440891 21101 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.549s user 0.001s sys 0.001s
I20260129 03:00:04.472414 20556 leader_election.cc:304] T cb4727386a3e48aca5d71cc6c611c197 P 1c50de89a5a845668696dcb28c3badd6 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1c50de89a5a845668696dcb28c3badd6, 974b802141bd4f13beffef2aa03f88f9; no voters:
I20260129 03:00:04.453321 22996 raft_consensus.cc:3060] T 4f5a77e94fef4b16a2ba7b067c8b144a P 1c50de89a5a845668696dcb28c3badd6 [term 3 FOLLOWER]: Advancing to term 4
I20260129 03:00:04.454123 22748 leader_election.cc:290] T 2e58e9a5a8aa4cecafb464addb4f35ed P 1c50de89a5a845668696dcb28c3badd6 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 197b880fa39d432a9b472845876bfaf0 (127.18.211.1:42027), 7a9daf5d42a84615a6383cfec9a10608 (127.18.211.4:40439)
I20260129 03:00:04.475697 22748 raft_consensus.cc:493] T 505580c93c694bbb8ea96fdf9b17ab46 P 1c50de89a5a845668696dcb28c3badd6 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260129 03:00:04.475945 23053 leader_election.cc:290] T 96609ac68bdf4993a12e1c23355ca013 P 1c50de89a5a845668696dcb28c3badd6 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 197b880fa39d432a9b472845876bfaf0 (127.18.211.1:42027), 974b802141bd4f13beffef2aa03f88f9 (127.18.211.2:34517), 7a9daf5d42a84615a6383cfec9a10608 (127.18.211.4:40439)
W20260129 03:00:04.381464 23086 consensus_meta.cc:287] T 1299723a4476420ab02fd8bfd7aa0645 P 197b880fa39d432a9b472845876bfaf0: Time spent flushing consensus metadata: real 0.852s user 0.008s sys 0.001s
I20260129 03:00:04.480602 23104 raft_consensus.cc:2804] T cb4727386a3e48aca5d71cc6c611c197 P 1c50de89a5a845668696dcb28c3badd6 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260129 03:00:04.457046 22982 leader_election.cc:290] T c934d31efd3f4e5195605ddc354d14e1 P 974b802141bd4f13beffef2aa03f88f9 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 197b880fa39d432a9b472845876bfaf0 (127.18.211.1:42027), 7a9daf5d42a84615a6383cfec9a10608 (127.18.211.4:40439)
I20260129 03:00:04.468400 20515 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57584 (request call id 1432) took 1335 ms. Trace:
I20260129 03:00:04.476468 22748 raft_consensus.cc:515] T 505580c93c694bbb8ea96fdf9b17ab46 P 1c50de89a5a845668696dcb28c3badd6 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } }
I20260129 03:00:04.470582 20437 raft_consensus.cc:1240] T b189f08e16a74a11980cc55d99359a0a P 197b880fa39d432a9b472845876bfaf0 [term 9 FOLLOWER]: Rejecting Update request from peer 1c50de89a5a845668696dcb28c3badd6 for earlier term 8. Current term is 9. Ops: []
I20260129 03:00:04.481287 23104 raft_consensus.cc:493] T cb4727386a3e48aca5d71cc6c611c197 P 1c50de89a5a845668696dcb28c3badd6 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260129 03:00:04.471939 20479 leader_election.cc:304] T febffe4a8b394368b69b1d164308fb9a P 974b802141bd4f13beffef2aa03f88f9 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 197b880fa39d432a9b472845876bfaf0, 974b802141bd4f13beffef2aa03f88f9; no voters: 7a9daf5d42a84615a6383cfec9a10608
W20260129 03:00:04.472714 21101 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.581s user 0.001s sys 0.001s
I20260129 03:00:04.345688 22977 leader_election.cc:290] T 368953c9102643b2a21f99af36b51905 P 1c50de89a5a845668696dcb28c3badd6 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 197b880fa39d432a9b472845876bfaf0 (127.18.211.1:42027), 974b802141bd4f13beffef2aa03f88f9 (127.18.211.2:34517)
I20260129 03:00:04.372641 23097 raft_consensus.cc:2749] T 368953c9102643b2a21f99af36b51905 P 974b802141bd4f13beffef2aa03f88f9 [term 7 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260129 03:00:04.508901 20509 raft_consensus.cc:1240] T 368953c9102643b2a21f99af36b51905 P 974b802141bd4f13beffef2aa03f88f9 [term 7 FOLLOWER]: Rejecting Update request from peer 1c50de89a5a845668696dcb28c3badd6 for earlier term 6. Current term is 7. Ops: []
I20260129 03:00:04.453940 23035 consensus_queue.cc:260] T 1299723a4476420ab02fd8bfd7aa0645 P 7a9daf5d42a84615a6383cfec9a10608 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 8.8, Last appended by leader: 8, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } }
I20260129 03:00:04.510871 23035 raft_consensus.cc:3060] T 1299723a4476420ab02fd8bfd7aa0645 P 7a9daf5d42a84615a6383cfec9a10608 [term 8 FOLLOWER]: Advancing to term 9
I20260129 03:00:04.510914 22748 leader_election.cc:290] T 505580c93c694bbb8ea96fdf9b17ab46 P 1c50de89a5a845668696dcb28c3badd6 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 974b802141bd4f13beffef2aa03f88f9 (127.18.211.2:34517), 7a9daf5d42a84615a6383cfec9a10608 (127.18.211.4:40439)
I20260129 03:00:04.368602 23094 raft_consensus.cc:515] T fd8311b779af4e1fa1402b2a02a602f2 P 7a9daf5d42a84615a6383cfec9a10608 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } }
I20260129 03:00:04.364045 23146 leader_election.cc:290] T 913fc7d751ed4b7388d889aac30a5f9b P 974b802141bd4f13beffef2aa03f88f9 [CANDIDATE]: Term 5 election: Requested vote from peers 197b880fa39d432a9b472845876bfaf0 (127.18.211.1:42027), 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3:37735)
I20260129 03:00:04.378266 23206 leader_election.cc:290] T 2ac659d732314e1288849f08fe5e074e P 7a9daf5d42a84615a6383cfec9a10608 [CANDIDATE]: Term 5 election: Requested vote from peers 197b880fa39d432a9b472845876bfaf0 (127.18.211.1:42027), 974b802141bd4f13beffef2aa03f88f9 (127.18.211.2:34517)
I20260129 03:00:04.514142 22977 raft_consensus.cc:493] T 75f45538dc704d9a8ec4ff8ad9f3ea5b P 1c50de89a5a845668696dcb28c3badd6 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 7a9daf5d42a84615a6383cfec9a10608)
I20260129 03:00:04.515165 23094 leader_election.cc:290] T fd8311b779af4e1fa1402b2a02a602f2 P 7a9daf5d42a84615a6383cfec9a10608 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 197b880fa39d432a9b472845876bfaf0 (127.18.211.1:42027), 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3:37735)
I20260129 03:00:04.514904 22977 raft_consensus.cc:515] T 75f45538dc704d9a8ec4ff8ad9f3ea5b P 1c50de89a5a845668696dcb28c3badd6 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } }
I20260129 03:00:04.467553 20511 raft_consensus.cc:1275] T a644c02b16af47679824ccd060ac10fd P 974b802141bd4f13beffef2aa03f88f9 [term 5 FOLLOWER]: Refusing update from remote peer 1c50de89a5a845668696dcb28c3badd6: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 5 index: 7. (index mismatch)
I20260129 03:00:04.436265 20440 raft_consensus.cc:1240] T 18f30ebaa4174fd28c958e72d0cc7e24 P 197b880fa39d432a9b472845876bfaf0 [term 7 FOLLOWER]: Rejecting Update request from peer 1c50de89a5a845668696dcb28c3badd6 for earlier term 6. Current term is 7. Ops: []
I20260129 03:00:04.516057 23168 raft_consensus.cc:493] T febffe4a8b394368b69b1d164308fb9a P 7a9daf5d42a84615a6383cfec9a10608 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260129 03:00:04.516324 20511 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57584 (request call id 1433) took 1298 ms. Trace:
I20260129 03:00:04.516789 20511 rpcz_store.cc:276] 0129 03:00:03.218394 (+ 0us) service_pool.cc:168] Inserting onto call queue
0129 03:00:04.466036 (+1247642us) service_pool.cc:225] Handling call
0129 03:00:04.466799 (+ 763us) raft_consensus.cc:1471] Updating replica for 0 ops
0129 03:00:04.516033 (+ 49234us) raft_consensus.cc:1697] Filling consensus response to leader.
0129 03:00:04.516241 (+ 208us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260129 03:00:04.516813 23168 raft_consensus.cc:515] T febffe4a8b394368b69b1d164308fb9a P 7a9daf5d42a84615a6383cfec9a10608 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } }
I20260129 03:00:04.519307 20511 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57600 (request call id 1878) took 1288 ms. Trace:
W20260129 03:00:04.494341 23107 consensus_meta.cc:287] T 78aa39096868415d9afc2ffcb248e198 P 7a9daf5d42a84615a6383cfec9a10608: Time spent flushing consensus metadata: real 1.011s user 0.002s sys 0.012s
I20260129 03:00:04.525040 23107 raft_consensus.cc:515] T 78aa39096868415d9afc2ffcb248e198 P 7a9daf5d42a84615a6383cfec9a10608 [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } }
W20260129 03:00:04.526758 20480 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260129 03:00:04.476215 23037 consensus_queue.cc:237] T 75b0b6e82ddc4f92b1fe1d94f9a0b73b P 7a9daf5d42a84615a6383cfec9a10608 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 3.6, Last appended by leader: 6, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } }
I20260129 03:00:04.528455 20553 leader_election.cc:304] T 1299723a4476420ab02fd8bfd7aa0645 P 1c50de89a5a845668696dcb28c3badd6 [CANDIDATE]: Term 9 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1c50de89a5a845668696dcb28c3badd6; no voters: 197b880fa39d432a9b472845876bfaf0, 7a9daf5d42a84615a6383cfec9a10608
I20260129 03:00:04.487599 23086 raft_consensus.cc:2749] T 1299723a4476420ab02fd8bfd7aa0645 P 197b880fa39d432a9b472845876bfaf0 [term 9 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260129 03:00:04.472079 20590 rpcz_store.cc:276] 0129 03:00:02.084138 (+ 0us) service_pool.cc:168] Inserting onto call queue
0129 03:00:04.318887 (+2234749us) service_pool.cc:225] Handling call
0129 03:00:04.319245 (+ 358us) raft_consensus.cc:1471] Updating replica for 0 ops
0129 03:00:04.439200 (+119955us) raft_consensus.cc:1534] Early marking committed up to index 3
0129 03:00:04.439263 (+ 63us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0129 03:00:04.439324 (+ 61us) raft_consensus.cc:1645] Marking committed up to 3
0129 03:00:04.439412 (+ 88us) raft_consensus.cc:1697] Filling consensus response to leader.
0129 03:00:04.439530 (+ 118us) raft_consensus.cc:1691] UpdateReplicas() finished
0129 03:00:04.439803 (+ 273us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260129 03:00:04.476735 23053 raft_consensus.cc:493] T e33a417cf9c54cf5af73a4ee3aac4d3a P 1c50de89a5a845668696dcb28c3badd6 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 974b802141bd4f13beffef2aa03f88f9)
I20260129 03:00:04.533298 23053 raft_consensus.cc:515] T e33a417cf9c54cf5af73a4ee3aac4d3a P 1c50de89a5a845668696dcb28c3badd6 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 03:00:04.483395 20515 rpcz_store.cc:276] 0129 03:00:03.132631 (+ 0us) service_pool.cc:168] Inserting onto call queue
0129 03:00:04.306246 (+1173615us) service_pool.cc:225] Handling call
0129 03:00:04.429656 (+123410us) raft_consensus.cc:1471] Updating replica for 0 ops
0129 03:00:04.468106 (+ 38450us) raft_consensus.cc:1697] Filling consensus response to leader.
0129 03:00:04.468343 (+ 237us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260129 03:00:04.536681 20515 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "368953c9102643b2a21f99af36b51905" candidate_uuid: "1c50de89a5a845668696dcb28c3badd6" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "974b802141bd4f13beffef2aa03f88f9" is_pre_election: true
W20260129 03:00:04.537606 20404 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260129 03:00:04.538527 20515 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00e79d8f25a34676b1d866eb2068a2b2" candidate_uuid: "7a9daf5d42a84615a6383cfec9a10608" candidate_term: 8 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "974b802141bd4f13beffef2aa03f88f9" is_pre_election: true
W20260129 03:00:04.540134 20407 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.211.2:34517, user_credentials={real_user=slave}} blocked reactor thread for 958708us
W20260129 03:00:04.551277 20507 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:48544 (request call id 258) took 3510 ms (client timeout 1710 ms). Trace:
W20260129 03:00:04.553061 20407 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260129 03:00:04.539844 20515 raft_consensus.cc:2393] T 00e79d8f25a34676b1d866eb2068a2b2 P 974b802141bd4f13beffef2aa03f88f9 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 7a9daf5d42a84615a6383cfec9a10608 in current term 8: Already voted for candidate 197b880fa39d432a9b472845876bfaf0 in this term.
I20260129 03:00:04.520608 23168 leader_election.cc:290] T febffe4a8b394368b69b1d164308fb9a P 7a9daf5d42a84615a6383cfec9a10608 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 197b880fa39d432a9b472845876bfaf0 (127.18.211.1:42027), 974b802141bd4f13beffef2aa03f88f9 (127.18.211.2:34517)
W20260129 03:00:04.528702 20482 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260129 03:00:04.523643 20556 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.211.2:34517, user_credentials={real_user=slave}} blocked reactor thread for 51468.2us
I20260129 03:00:04.488479 23104 raft_consensus.cc:3060] T cb4727386a3e48aca5d71cc6c611c197 P 1c50de89a5a845668696dcb28c3badd6 [term 4 FOLLOWER]: Advancing to term 5
I20260129 03:00:04.552467 22694 raft_consensus.cc:515] T a1a39dcd328e4268ae41b988c66b4a75 P 974b802141bd4f13beffef2aa03f88f9 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } } peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } }
W20260129 03:00:04.541307 20555 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260129 03:00:04.542498 23259 log.cc:927] Time spent T a609ac02fae84725ab743f95f51d33ce P 7a9daf5d42a84615a6383cfec9a10608: Append to log took a long time: real 0.142s user 0.001s sys 0.000s
W20260129 03:00:04.569470 23043 consensus_meta.cc:287] T e7d83d4ccf684e76acb29a0dc844e1b8 P 197b880fa39d432a9b472845876bfaf0: Time spent flushing consensus metadata: real 0.934s user 0.016s sys 0.007s
I20260129 03:00:04.524233 20511 rpcz_store.cc:276] 0129 03:00:03.231355 (+ 0us) service_pool.cc:168] Inserting onto call queue
0129 03:00:04.518055 (+1286700us) service_pool.cc:225] Handling call
0129 03:00:04.518528 (+ 473us) raft_consensus.cc:1471] Updating replica for 0 ops
0129 03:00:04.518918 (+ 390us) raft_consensus.cc:1534] Early marking committed up to index 0
0129 03:00:04.518949 (+ 31us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0129 03:00:04.519008 (+ 59us) raft_consensus.cc:1645] Marking committed up to 7
0129 03:00:04.519064 (+ 56us) raft_consensus.cc:1697] Filling consensus response to leader.
0129 03:00:04.519125 (+ 61us) raft_consensus.cc:1691] UpdateReplicas() finished
0129 03:00:04.519238 (+ 113us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260129 03:00:04.532239 20590 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4e3805ffa70242e8ade66da260dddf66" candidate_uuid: "7a9daf5d42a84615a6383cfec9a10608" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "1c50de89a5a845668696dcb28c3badd6"
I20260129 03:00:04.551018 23107 leader_election.cc:290] T 78aa39096868415d9afc2ffcb248e198 P 7a9daf5d42a84615a6383cfec9a10608 [CANDIDATE]: Term 6 election: Requested vote from peers 197b880fa39d432a9b472845876bfaf0 (127.18.211.1:42027), 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3:37735)
I20260129 03:00:04.550946 23053 leader_election.cc:290] T e33a417cf9c54cf5af73a4ee3aac4d3a P 1c50de89a5a845668696dcb28c3badd6 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 197b880fa39d432a9b472845876bfaf0 (127.18.211.1:42027), 974b802141bd4f13beffef2aa03f88f9 (127.18.211.2:34517)
I20260129 03:00:04.577402 20581 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35752 (request call id 1644) took 2590 ms. Trace:
I20260129 03:00:04.545375 22950 raft_consensus.cc:697] T c891d2bff4ce4568861bbc992e91fbb0 P 197b880fa39d432a9b472845876bfaf0 [term 7 LEADER]: Becoming Leader. State: Replica: 197b880fa39d432a9b472845876bfaf0, State: Running, Role: LEADER
I20260129 03:00:04.557320 23251 raft_consensus.cc:2749] T 1299723a4476420ab02fd8bfd7aa0645 P 1c50de89a5a845668696dcb28c3badd6 [term 9 FOLLOWER]: Leader election lost for term 9. Reason: could not achieve majority
I20260129 03:00:04.545938 22925 raft_consensus.cc:2749] T 4e3805ffa70242e8ade66da260dddf66 P 197b880fa39d432a9b472845876bfaf0 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260129 03:00:04.540418 21127 raft_consensus.cc:1240] T 78aa39096868415d9afc2ffcb248e198 P 7a9daf5d42a84615a6383cfec9a10608 [term 6 FOLLOWER]: Rejecting Update request from peer 1c50de89a5a845668696dcb28c3badd6 for earlier term 5. Current term is 6. Ops: []
W20260129 03:00:04.546056 21101 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260129 03:00:04.578008 20590 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fd8311b779af4e1fa1402b2a02a602f2" candidate_uuid: "7a9daf5d42a84615a6383cfec9a10608" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "1c50de89a5a845668696dcb28c3badd6" is_pre_election: true
I20260129 03:00:04.555073 20516 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57600 (request call id 1877) took 1429 ms. Trace:
W20260129 03:00:04.569757 20588 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:35754 (request call id 1714) took 3626 ms (client timeout 1512 ms). Trace:
I20260129 03:00:04.582015 20516 rpcz_store.cc:276] 0129 03:00:03.125918 (+ 0us) service_pool.cc:168] Inserting onto call queue
0129 03:00:04.293156 (+1167238us) service_pool.cc:225] Handling call
0129 03:00:04.293593 (+ 437us) raft_consensus.cc:1471] Updating replica for 0 ops
0129 03:00:04.431448 (+137855us) raft_consensus.cc:1534] Early marking committed up to index 0
0129 03:00:04.431528 (+ 80us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0129 03:00:04.431601 (+ 73us) raft_consensus.cc:1645] Marking committed up to 7
0129 03:00:04.431681 (+ 80us) raft_consensus.cc:1697] Filling consensus response to leader.
0129 03:00:04.554574 (+122893us) spinlock_profiling.cc:243] Waited 137 ms on lock 0x7b54002ef754. stack: 00007f8fe044e373 00007f8fe044e270 00007f8fdfda0574 00007f8fe9a5c6b4 00007f8fe9a5c649 00007f8fe34975a1 00007f8fe352ea81 00007f8fe352d4be 00007f8fe84e1f49 00007f8fe1a6ed34 00007f8fe1a6ecc2 00007f8fe1a6ebf1 00007f8fe1a6eb6c 00007f8fe1a6de22 00007f8fe186d34c 00007f8fe186c786
0129 03:00:04.554628 (+ 54us) raft_consensus.cc:1691] UpdateReplicas() finished
0129 03:00:04.554833 (+ 205us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":315311360}
W20260129 03:00:04.551836 20507 rpcz_store.cc:269] 0129 03:00:01.040285 (+ 0us) service_pool.cc:168] Inserting onto call queue
0129 03:00:01.058221 (+ 17936us) service_pool.cc:225] Handling call
0129 03:00:04.550802 (+3492581us) spinlock_profiling.cc:243] Waited 931 ms on lock 0x7b5400315454. stack: 00007f8fe044e373 00007f8fe044e270 00007f8fdfda0574 00007f8fe9a5c6b4 00007f8fe9a5c649 00007f8fe34975a1 00007f8fe3531f7a 00007f8fe84e3647 00007f8fe1a73834 00007f8fe1a737c2 00007f8fe1a736f1 00007f8fe1a7366c 00007f8fe1a72922 00007f8fe186d34c 00007f8fe186c786 00007f8fe186c12f
0129 03:00:04.551053 (+ 251us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2140860800}
I20260129 03:00:04.588366 20516 raft_consensus.cc:1240] T add8d4923d1a489e86882c3a25692b95 P 974b802141bd4f13beffef2aa03f88f9 [term 6 FOLLOWER]: Rejecting Update request from peer 1c50de89a5a845668696dcb28c3badd6 for earlier term 5. Current term is 6. Ops: [3.6-5.7]
I20260129 03:00:04.587815 22950 consensus_queue.cc:237] T c891d2bff4ce4568861bbc992e91fbb0 P 197b880fa39d432a9b472845876bfaf0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 6.5, Last appended by leader: 5, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
W20260129 03:00:04.588626 23258 log.cc:927] Time spent T 42d27a28d89b409e8807ec0efd97af69 P 7a9daf5d42a84615a6383cfec9a10608: Append to log took a long time: real 0.188s user 0.001s sys 0.000s
I20260129 03:00:04.589097 20516 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57584 (request call id 1435) took 1341 ms. Trace:
I20260129 03:00:04.589504 20516 rpcz_store.cc:276] 0129 03:00:03.247277 (+ 0us) service_pool.cc:168] Inserting onto call queue
0129 03:00:04.587327 (+1340050us) service_pool.cc:225] Handling call
0129 03:00:04.587945 (+ 618us) raft_consensus.cc:1471] Updating replica for 2 ops
0129 03:00:04.588803 (+ 858us) raft_consensus.cc:1697] Filling consensus response to leader.
0129 03:00:04.589037 (+ 234us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20260129 03:00:04.591030 20583 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:48526 (request call id 345) took 1342 ms (client timeout 1585 ms). Trace:
I20260129 03:00:04.550372 22977 leader_election.cc:290] T 75f45538dc704d9a8ec4ff8ad9f3ea5b P 1c50de89a5a845668696dcb28c3badd6 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 974b802141bd4f13beffef2aa03f88f9 (127.18.211.2:34517), 7a9daf5d42a84615a6383cfec9a10608 (127.18.211.4:40439)
W20260129 03:00:04.584972 20588 rpcz_store.cc:269] 0129 03:00:00.942844 (+ 0us) service_pool.cc:168] Inserting onto call queue
0129 03:00:00.952930 (+ 10086us) service_pool.cc:225] Handling call
0129 03:00:04.569290 (+3616360us) spinlock_profiling.cc:243] Waited 1.12 s on lock 0x7b5400134054. stack: 00007f8fe044e373 00007f8fe044e270 00007f8fdfda0574 00007f8fe9a5c6b4 00007f8fe9a5c649 00007f8fe34975a1 00007f8fe3531f7a 00007f8fe84e3647 00007f8fe1a73834 00007f8fe1a737c2 00007f8fe1a736f1 00007f8fe1a7366c 00007f8fe1a72922 00007f8fe186d34c 00007f8fe186c786 00007f8fe186c12f
0129 03:00:04.569486 (+ 196us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2571679616}
I20260129 03:00:04.577976 20581 rpcz_store.cc:276] 0129 03:00:01.986979 (+ 0us) service_pool.cc:168] Inserting onto call queue
0129 03:00:02.664817 (+677838us) service_pool.cc:225] Handling call
0129 03:00:04.033588 (+1368771us) raft_consensus.cc:1471] Updating replica for 0 ops
0129 03:00:04.168402 (+134814us) raft_consensus.cc:1697] Filling consensus response to leader.
0129 03:00:04.576980 (+408578us) spinlock_profiling.cc:243] Waited 1.35 s on lock 0x7b54003a0050. stack: 00007f8fe044e373 00007f8fe044e270 00007f8fdfda0574 00007f8fe9a5c6b4 00007f8fe9a5c649 00007f8fe34975a1 00007f8fe352d52d 00007f8fe84e1f49 00007f8fe1a6ed34 00007f8fe1a6ecc2 00007f8fe1a6ebf1 00007f8fe1a6eb6c 00007f8fe1a6de22 00007f8fe186d34c 00007f8fe186c786 00007f8fe186c12f
0129 03:00:04.577176 (+ 196us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":3100732800}
I20260129 03:00:04.596325 22691 raft_consensus.cc:2804] T b189f08e16a74a11980cc55d99359a0a P 197b880fa39d432a9b472845876bfaf0 [term 9 FOLLOWER]: Leader election won for term 9
I20260129 03:00:04.597386 23057 raft_consensus.cc:493] T 3906b080a5b34a74b8b64128635e8997 P 197b880fa39d432a9b472845876bfaf0 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 974b802141bd4f13beffef2aa03f88f9)
I20260129 03:00:04.597656 23097 raft_consensus.cc:493] T 76f713c3331540d2a5b76164103e9eb1 P 974b802141bd4f13beffef2aa03f88f9 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 1c50de89a5a845668696dcb28c3badd6)
I20260129 03:00:04.596415 20516 raft_consensus.cc:1240] T b189f08e16a74a11980cc55d99359a0a P 974b802141bd4f13beffef2aa03f88f9 [term 9 FOLLOWER]: Rejecting Update request from peer 1c50de89a5a845668696dcb28c3badd6 for earlier term 8. Current term is 9. Ops: [8.4-8.4]
W20260129 03:00:04.591477 20583 rpcz_store.cc:269] 0129 03:00:03.248204 (+ 0us) service_pool.cc:168] Inserting onto call queue
0129 03:00:03.267194 (+ 18990us) service_pool.cc:225] Handling call
0129 03:00:04.590696 (+1323502us) spinlock_profiling.cc:243] Waited 211 ms on lock 0x7b5400134f54. stack: 00007f8fe044e373 00007f8fe044e270 00007f8fdfda0574 00007f8fe9a5c6b4 00007f8fe9a5c649 00007f8fe34975a1 00007f8fe3531f7a 00007f8fe84e3647 00007f8fe1a73834 00007f8fe1a737c2 00007f8fe1a736f1 00007f8fe1a7366c 00007f8fe1a72922 00007f8fe186d34c 00007f8fe186c786 00007f8fe186c12f
0129 03:00:04.590859 (+ 163us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":485887744}
I20260129 03:00:04.604686 20516 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57584 (request call id 1436) took 1311 ms. Trace:
I20260129 03:00:04.604979 20516 rpcz_store.cc:276] 0129 03:00:03.289495 (+ 0us) service_pool.cc:168] Inserting onto call queue
0129 03:00:04.590343 (+1300848us) service_pool.cc:225] Handling call
0129 03:00:04.591733 (+ 1390us) raft_consensus.cc:1471] Updating replica for 1 ops
0129 03:00:04.599200 (+ 7467us) raft_consensus.cc:1697] Filling consensus response to leader.
0129 03:00:04.600406 (+ 1206us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20260129 03:00:04.608685 20404 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260129 03:00:04.610512 20516 raft_consensus.cc:1275] T 505580c93c694bbb8ea96fdf9b17ab46 P 974b802141bd4f13beffef2aa03f88f9 [term 6 FOLLOWER]: Refusing update from remote peer 7a9daf5d42a84615a6383cfec9a10608: Log matching property violated. Preceding OpId in replica: term: 5 index: 7. Preceding OpId from leader: term: 6 index: 8. (index mismatch)
I20260129 03:00:04.611303 20516 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:48544 (request call id 287) took 1308 ms. Trace:
I20260129 03:00:04.611336 20507 raft_consensus.cc:1240] T 7a9a69cd82724b1faed1d10aa091fc4a P 974b802141bd4f13beffef2aa03f88f9 [term 7 FOLLOWER]: Rejecting Update request from peer 1c50de89a5a845668696dcb28c3badd6 for earlier term 6. Current term is 7. Ops: []
I20260129 03:00:04.610530 20590 raft_consensus.cc:1240] T 0f20af7fbe5d4db89a6f17948d98912c P 1c50de89a5a845668696dcb28c3badd6 [term 7 FOLLOWER]: Rejecting Update request from peer 974b802141bd4f13beffef2aa03f88f9 for earlier term 6. Current term is 7. Ops: []
I20260129 03:00:04.612195 20507 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57584 (request call id 1437) took 1322 ms. Trace:
I20260129 03:00:04.612722 20590 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35752 (request call id 1648) took 2267 ms. Trace:
I20260129 03:00:04.613030 20440 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "368953c9102643b2a21f99af36b51905" candidate_uuid: "1c50de89a5a845668696dcb28c3badd6" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "197b880fa39d432a9b472845876bfaf0" is_pre_election: true
I20260129 03:00:04.613235 20590 rpcz_store.cc:276] 0129 03:00:02.345167 (+ 0us) service_pool.cc:168] Inserting onto call queue
0129 03:00:04.609634 (+2264467us) service_pool.cc:225] Handling call
0129 03:00:04.610133 (+ 499us) raft_consensus.cc:1471] Updating replica for 0 ops
0129 03:00:04.612455 (+ 2322us) raft_consensus.cc:1697] Filling consensus response to leader.
0129 03:00:04.612629 (+ 174us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260129 03:00:04.613972 20440 raft_consensus.cc:2468] T 368953c9102643b2a21f99af36b51905 P 197b880fa39d432a9b472845876bfaf0 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1c50de89a5a845668696dcb28c3badd6 in term 7.
I20260129 03:00:04.612630 20507 rpcz_store.cc:276] 0129 03:00:03.289862 (+ 0us) service_pool.cc:168] Inserting onto call queue
0129 03:00:04.601090 (+1311228us) service_pool.cc:225] Handling call
0129 03:00:04.611042 (+ 9952us) raft_consensus.cc:1471] Updating replica for 0 ops
0129 03:00:04.611928 (+ 886us) raft_consensus.cc:1697] Filling consensus response to leader.
0129 03:00:04.612119 (+ 191us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260129 03:00:04.615839 20507 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "febffe4a8b394368b69b1d164308fb9a" candidate_uuid: "7a9daf5d42a84615a6383cfec9a10608" candidate_term: 5 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "974b802141bd4f13beffef2aa03f88f9" is_pre_election: true
W20260129 03:00:04.613984 20554 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260129 03:00:04.612205 20441 raft_consensus.cc:3060] T fd8311b779af4e1fa1402b2a02a602f2 P 197b880fa39d432a9b472845876bfaf0 [term 4 FOLLOWER]: Advancing to term 5
I20260129 03:00:04.611763 20516 rpcz_store.cc:276] 0129 03:00:03.302852 (+ 0us) service_pool.cc:168] Inserting onto call queue
0129 03:00:04.608787 (+1305935us) service_pool.cc:225] Handling call
0129 03:00:04.610008 (+ 1221us) raft_consensus.cc:1471] Updating replica for 0 ops
0129 03:00:04.610993 (+ 985us) raft_consensus.cc:1697] Filling consensus response to leader.
0129 03:00:04.611230 (+ 237us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260129 03:00:04.618901 20516 raft_consensus.cc:1240] T e7d83d4ccf684e76acb29a0dc844e1b8 P 974b802141bd4f13beffef2aa03f88f9 [term 6 FOLLOWER]: Rejecting Update request from peer 197b880fa39d432a9b472845876bfaf0 for earlier term 5. Current term is 6. Ops: []
I20260129 03:00:04.619483 20516 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57600 (request call id 1881) took 1291 ms. Trace:
I20260129 03:00:04.619622 20437 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "96609ac68bdf4993a12e1c23355ca013" candidate_uuid: "1c50de89a5a845668696dcb28c3badd6" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "197b880fa39d432a9b472845876bfaf0" is_pre_election: true
I20260129 03:00:04.618916 20440 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2e58e9a5a8aa4cecafb464addb4f35ed" candidate_uuid: "1c50de89a5a845668696dcb28c3badd6" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "197b880fa39d432a9b472845876bfaf0" is_pre_election: true
I20260129 03:00:04.598184 23057 raft_consensus.cc:515] T 3906b080a5b34a74b8b64128635e8997 P 197b880fa39d432a9b472845876bfaf0 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 03:00:04.619650 23104 raft_consensus.cc:515] T cb4727386a3e48aca5d71cc6c611c197 P 1c50de89a5a845668696dcb28c3badd6 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 03:00:04.608112 22691 raft_consensus.cc:697] T b189f08e16a74a11980cc55d99359a0a P 197b880fa39d432a9b472845876bfaf0 [term 9 LEADER]: Becoming Leader. State: Replica: 197b880fa39d432a9b472845876bfaf0, State: Running, Role: LEADER
I20260129 03:00:04.622553 23057 leader_election.cc:290] T 3906b080a5b34a74b8b64128635e8997 P 197b880fa39d432a9b472845876bfaf0 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 974b802141bd4f13beffef2aa03f88f9 (127.18.211.2:34517), 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3:37735)
I20260129 03:00:04.623344 22691 consensus_queue.cc:237] T b189f08e16a74a11980cc55d99359a0a P 197b880fa39d432a9b472845876bfaf0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 8.4, Last appended by leader: 4, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 03:00:04.625010 20437 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e33a417cf9c54cf5af73a4ee3aac4d3a" candidate_uuid: "1c50de89a5a845668696dcb28c3badd6" candidate_term: 7 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "197b880fa39d432a9b472845876bfaf0" is_pre_election: true
I20260129 03:00:04.625999 20437 raft_consensus.cc:2468] T e33a417cf9c54cf5af73a4ee3aac4d3a P 197b880fa39d432a9b472845876bfaf0 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1c50de89a5a845668696dcb28c3badd6 in term 6.
W20260129 03:00:04.627245 23041 consensus_meta.cc:287] T dc2f4cb752cf4d59b5f6bc689f5d8104 P 7a9daf5d42a84615a6383cfec9a10608: Time spent flushing consensus metadata: real 1.071s user 0.016s sys 0.004s
I20260129 03:00:04.624032 23104 leader_election.cc:290] T cb4727386a3e48aca5d71cc6c611c197 P 1c50de89a5a845668696dcb28c3badd6 [CANDIDATE]: Term 5 election: Requested vote from peers 197b880fa39d432a9b472845876bfaf0 (127.18.211.1:42027), 974b802141bd4f13beffef2aa03f88f9 (127.18.211.2:34517)
I20260129 03:00:04.627770 23041 raft_consensus.cc:515] T dc2f4cb752cf4d59b5f6bc689f5d8104 P 7a9daf5d42a84615a6383cfec9a10608 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } }
I20260129 03:00:04.630391 23041 leader_election.cc:290] T dc2f4cb752cf4d59b5f6bc689f5d8104 P 7a9daf5d42a84615a6383cfec9a10608 [CANDIDATE]: Term 5 election: Requested vote from peers 197b880fa39d432a9b472845876bfaf0 (127.18.211.1:42027), 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3:37735)
W20260129 03:00:04.638661 20407 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260129 03:00:04.640800 20435 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4e3805ffa70242e8ade66da260dddf66" candidate_uuid: "7a9daf5d42a84615a6383cfec9a10608" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "197b880fa39d432a9b472845876bfaf0"
I20260129 03:00:04.641610 20435 raft_consensus.cc:3060] T 4e3805ffa70242e8ade66da260dddf66 P 197b880fa39d432a9b472845876bfaf0 [term 5 FOLLOWER]: Advancing to term 6
I20260129 03:00:04.642015 20437 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fd8311b779af4e1fa1402b2a02a602f2" candidate_uuid: "7a9daf5d42a84615a6383cfec9a10608" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "197b880fa39d432a9b472845876bfaf0" is_pre_election: true
I20260129 03:00:04.642629 20407 leader_election.cc:304] T cb4727386a3e48aca5d71cc6c611c197 P 197b880fa39d432a9b472845876bfaf0 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 197b880fa39d432a9b472845876bfaf0, 974b802141bd4f13beffef2aa03f88f9; no voters:
I20260129 03:00:04.643666 20437 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2ac659d732314e1288849f08fe5e074e" candidate_uuid: "7a9daf5d42a84615a6383cfec9a10608" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "197b880fa39d432a9b472845876bfaf0"
I20260129 03:00:04.644472 20437 raft_consensus.cc:3060] T 2ac659d732314e1288849f08fe5e074e P 197b880fa39d432a9b472845876bfaf0 [term 4 FOLLOWER]: Advancing to term 5
I20260129 03:00:04.649415 21136 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "75f45538dc704d9a8ec4ff8ad9f3ea5b" candidate_uuid: "974b802141bd4f13beffef2aa03f88f9" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "7a9daf5d42a84615a6383cfec9a10608" is_pre_election: true
I20260129 03:00:04.650993 21136 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2e58e9a5a8aa4cecafb464addb4f35ed" candidate_uuid: "1c50de89a5a845668696dcb28c3badd6" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "7a9daf5d42a84615a6383cfec9a10608" is_pre_election: true
W20260129 03:00:04.653312 21100 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.211.3:37735, user_credentials={real_user=slave}} blocked reactor thread for 46255.2us
I20260129 03:00:04.620051 22977 raft_consensus.cc:493] T 4f8f3a38f6e24ba4857875181f447442 P 1c50de89a5a845668696dcb28c3badd6 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260129 03:00:04.654419 22977 raft_consensus.cc:515] T 4f8f3a38f6e24ba4857875181f447442 P 1c50de89a5a845668696dcb28c3badd6 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 03:00:04.655102 20437 raft_consensus.cc:2468] T 2ac659d732314e1288849f08fe5e074e P 197b880fa39d432a9b472845876bfaf0 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7a9daf5d42a84615a6383cfec9a10608 in term 5.
I20260129 03:00:04.656546 20437 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "78aa39096868415d9afc2ffcb248e198" candidate_uuid: "7a9daf5d42a84615a6383cfec9a10608" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "197b880fa39d432a9b472845876bfaf0"
I20260129 03:00:04.660281 23146 raft_consensus.cc:2804] T febffe4a8b394368b69b1d164308fb9a P 974b802141bd4f13beffef2aa03f88f9 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260129 03:00:04.660864 23146 raft_consensus.cc:493] T febffe4a8b394368b69b1d164308fb9a P 974b802141bd4f13beffef2aa03f88f9 [term 3 FOLLOWER]: Starting leader election (detected failure of leader 197b880fa39d432a9b472845876bfaf0)
I20260129 03:00:04.661306 23146 raft_consensus.cc:3060] T febffe4a8b394368b69b1d164308fb9a P 974b802141bd4f13beffef2aa03f88f9 [term 3 FOLLOWER]: Advancing to term 4
I20260129 03:00:04.657991 20437 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "febffe4a8b394368b69b1d164308fb9a" candidate_uuid: "7a9daf5d42a84615a6383cfec9a10608" candidate_term: 5 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "197b880fa39d432a9b472845876bfaf0" is_pre_election: true
I20260129 03:00:04.665067 20437 raft_consensus.cc:2410] T febffe4a8b394368b69b1d164308fb9a P 197b880fa39d432a9b472845876bfaf0 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 7a9daf5d42a84615a6383cfec9a10608 for term 5 because replica has last-logged OpId of term: 3 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 4.
I20260129 03:00:04.609058 20553 leader_election.cc:304] T fcddab6bca684adf8ea032289a00f90e P 1c50de89a5a845668696dcb28c3badd6 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1c50de89a5a845668696dcb28c3badd6; no voters: 197b880fa39d432a9b472845876bfaf0, 974b802141bd4f13beffef2aa03f88f9
I20260129 03:00:04.644390 20438 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00e79d8f25a34676b1d866eb2068a2b2" candidate_uuid: "7a9daf5d42a84615a6383cfec9a10608" candidate_term: 8 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "197b880fa39d432a9b472845876bfaf0" is_pre_election: true
I20260129 03:00:04.656662 22977 leader_election.cc:290] T 4f8f3a38f6e24ba4857875181f447442 P 1c50de89a5a845668696dcb28c3badd6 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 197b880fa39d432a9b472845876bfaf0 (127.18.211.1:42027), 974b802141bd4f13beffef2aa03f88f9 (127.18.211.2:34517)
I20260129 03:00:04.607071 20515 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2ac659d732314e1288849f08fe5e074e" candidate_uuid: "7a9daf5d42a84615a6383cfec9a10608" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "974b802141bd4f13beffef2aa03f88f9"
I20260129 03:00:04.541088 23232 raft_consensus.cc:515] T 8055232cf0004d40a85f3bf983fe00a9 P 974b802141bd4f13beffef2aa03f88f9 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 03:00:04.598415 23097 raft_consensus.cc:515] T 76f713c3331540d2a5b76164103e9eb1 P 974b802141bd4f13beffef2aa03f88f9 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 03:00:04.666242 21136 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "505580c93c694bbb8ea96fdf9b17ab46" candidate_uuid: "1c50de89a5a845668696dcb28c3badd6" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "7a9daf5d42a84615a6383cfec9a10608" is_pre_election: true
I20260129 03:00:04.672818 20511 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "913fc7d751ed4b7388d889aac30a5f9b" candidate_uuid: "197b880fa39d432a9b472845876bfaf0" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "974b802141bd4f13beffef2aa03f88f9" is_pre_election: true
I20260129 03:00:04.968003 21136 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "75f45538dc704d9a8ec4ff8ad9f3ea5b" candidate_uuid: "1c50de89a5a845668696dcb28c3badd6" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "7a9daf5d42a84615a6383cfec9a10608" is_pre_election: true
I20260129 03:00:04.969506 21136 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "96609ac68bdf4993a12e1c23355ca013" candidate_uuid: "1c50de89a5a845668696dcb28c3badd6" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "7a9daf5d42a84615a6383cfec9a10608" is_pre_election: true
I20260129 03:00:04.619822 20516 rpcz_store.cc:276] 0129 03:00:03.328439 (+ 0us) service_pool.cc:168] Inserting onto call queue
0129 03:00:04.618189 (+1289750us) service_pool.cc:225] Handling call
0129 03:00:04.618579 (+ 390us) raft_consensus.cc:1471] Updating replica for 0 ops
0129 03:00:04.619266 (+ 687us) raft_consensus.cc:1697] Filling consensus response to leader.
0129 03:00:04.619438 (+ 172us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260129 03:00:04.971167 21136 raft_consensus.cc:1240] T dc2f4cb752cf4d59b5f6bc689f5d8104 P 7a9daf5d42a84615a6383cfec9a10608 [term 5 FOLLOWER]: Rejecting Update request from peer 1c50de89a5a845668696dcb28c3badd6 for earlier term 4. Current term is 5. Ops: []
I20260129 03:00:04.619771 20513 raft_consensus.cc:1240] T a1a39dcd328e4268ae41b988c66b4a75 P 974b802141bd4f13beffef2aa03f88f9 [term 5 FOLLOWER]: Rejecting Update request from peer 1c50de89a5a845668696dcb28c3badd6 for earlier term 4. Current term is 5. Ops: []
I20260129 03:00:04.973356 20407 leader_election.cc:304] T 7a9a69cd82724b1faed1d10aa091fc4a P 197b880fa39d432a9b472845876bfaf0 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 197b880fa39d432a9b472845876bfaf0, 974b802141bd4f13beffef2aa03f88f9; no voters:
I20260129 03:00:04.965691 20588 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:48526 (request call id 341) took 2364 ms. Trace:
W20260129 03:00:04.976388 20556 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.211.2:34517, user_credentials={real_user=slave}} blocked reactor thread for 72713.3us
I20260129 03:00:04.928874 20583 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "75f45538dc704d9a8ec4ff8ad9f3ea5b" candidate_uuid: "974b802141bd4f13beffef2aa03f88f9" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "1c50de89a5a845668696dcb28c3badd6" is_pre_election: true
I20260129 03:00:04.920063 19276 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20260129 03:00:04.976493 20588 rpcz_store.cc:276] 0129 03:00:02.601636 (+ 0us) service_pool.cc:168] Inserting onto call queue
0129 03:00:04.899616 (+2297980us) service_pool.cc:225] Handling call
0129 03:00:04.964991 (+ 65375us) raft_consensus.cc:1471] Updating replica for 0 ops
0129 03:00:04.965219 (+ 228us) raft_consensus.cc:1534] Early marking committed up to index 0
0129 03:00:04.965257 (+ 38us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0129 03:00:04.965329 (+ 72us) raft_consensus.cc:1645] Marking committed up to 6
0129 03:00:04.965391 (+ 62us) raft_consensus.cc:1697] Filling consensus response to leader.
0129 03:00:04.965465 (+ 74us) raft_consensus.cc:1691] UpdateReplicas() finished
0129 03:00:04.965602 (+ 137us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260129 03:00:04.977254 20581 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "78aa39096868415d9afc2ffcb248e198" candidate_uuid: "7a9daf5d42a84615a6383cfec9a10608" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "1c50de89a5a845668696dcb28c3badd6"
I20260129 03:00:04.977901 20511 raft_consensus.cc:1240] T cedce331e1584ac8b636c9b8a72b2ba0 P 974b802141bd4f13beffef2aa03f88f9 [term 7 FOLLOWER]: Rejecting Update request from peer 197b880fa39d432a9b472845876bfaf0 for earlier term 6. Current term is 7. Ops: []
I20260129 03:00:04.981557 20511 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57600 (request call id 1884) took 1545 ms. Trace:
I20260129 03:00:04.981967 20511 rpcz_store.cc:276] 0129 03:00:03.436563 (+ 0us) service_pool.cc:168] Inserting onto call queue
0129 03:00:04.968820 (+1532257us) service_pool.cc:225] Handling call
0129 03:00:04.969301 (+ 481us) raft_consensus.cc:1471] Updating replica for 0 ops
0129 03:00:04.981231 (+ 11930us) raft_consensus.cc:1697] Filling consensus response to leader.
0129 03:00:04.981502 (+ 271us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260129 03:00:04.985531 20588 raft_consensus.cc:1275] T 8055232cf0004d40a85f3bf983fe00a9 P 1c50de89a5a845668696dcb28c3badd6 [term 4 FOLLOWER]: Refusing update from remote peer 197b880fa39d432a9b472845876bfaf0: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
W20260129 03:00:04.985714 20479 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.211.1:42027, user_credentials={real_user=slave}} blocked reactor thread for 514070us
I20260129 03:00:04.986380 20588 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35754 (request call id 1775) took 2007 ms. Trace:
I20260129 03:00:04.986766 20588 rpcz_store.cc:276] 0129 03:00:02.979918 (+ 0us) service_pool.cc:168] Inserting onto call queue
0129 03:00:04.980186 (+2000268us) service_pool.cc:225] Handling call
0129 03:00:04.984952 (+ 4766us) raft_consensus.cc:1471] Updating replica for 0 ops
0129 03:00:04.986077 (+ 1125us) raft_consensus.cc:1697] Filling consensus response to leader.
0129 03:00:04.986304 (+ 227us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260129 03:00:04.594696 22748 consensus_queue.cc:1048] T 4e3805ffa70242e8ade66da260dddf66 P 1c50de89a5a845668696dcb28c3badd6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.384s
W20260129 03:00:04.989218 23208 consensus_meta.cc:287] T 926a250fa5834e88b96ccecbefbf5086 P 974b802141bd4f13beffef2aa03f88f9: Time spent flushing consensus metadata: real 1.220s user 0.007s sys 0.010s
I20260129 03:00:04.990739 21136 raft_consensus.cc:1275] T fd8311b779af4e1fa1402b2a02a602f2 P 7a9daf5d42a84615a6383cfec9a10608 [term 5 FOLLOWER]: Refusing update from remote peer 1c50de89a5a845668696dcb28c3badd6: Log matching property violated. Preceding OpId in replica: term: 4 index: 6. Preceding OpId from leader: term: 5 index: 7. (index mismatch)
I20260129 03:00:04.993382 23165 raft_consensus.cc:493] T 4f5a77e94fef4b16a2ba7b067c8b144a P 7a9daf5d42a84615a6383cfec9a10608 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260129 03:00:04.994002 23165 raft_consensus.cc:515] T 4f5a77e94fef4b16a2ba7b067c8b144a P 7a9daf5d42a84615a6383cfec9a10608 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } }
I20260129 03:00:04.994556 23168 raft_consensus.cc:493] T c934d31efd3f4e5195605ddc354d14e1 P 7a9daf5d42a84615a6383cfec9a10608 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 197b880fa39d432a9b472845876bfaf0)
I20260129 03:00:04.995235 23168 raft_consensus.cc:515] T c934d31efd3f4e5195605ddc354d14e1 P 7a9daf5d42a84615a6383cfec9a10608 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } }
I20260129 03:00:04.932039 23043 raft_consensus.cc:493] T febffe4a8b394368b69b1d164308fb9a P 197b880fa39d432a9b472845876bfaf0 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260129 03:00:04.979269 23057 raft_consensus.cc:2804] T cb4727386a3e48aca5d71cc6c611c197 P 197b880fa39d432a9b472845876bfaf0 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260129 03:00:04.984195 22694 leader_election.cc:290] T a1a39dcd328e4268ae41b988c66b4a75 P 974b802141bd4f13beffef2aa03f88f9 [CANDIDATE]: Term 5 election: Requested vote from peers 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3:37735), 197b880fa39d432a9b472845876bfaf0 (127.18.211.1:42027)
W20260129 03:00:05.024121 21101 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260129 03:00:05.011003 20515 raft_consensus.cc:3060] T 2ac659d732314e1288849f08fe5e074e P 974b802141bd4f13beffef2aa03f88f9 [term 4 FOLLOWER]: Advancing to term 5
I20260129 03:00:05.015367 20435 raft_consensus.cc:2468] T 4e3805ffa70242e8ade66da260dddf66 P 197b880fa39d432a9b472845876bfaf0 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7a9daf5d42a84615a6383cfec9a10608 in term 6.
W20260129 03:00:05.009227 20407 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.211.2:34517, user_credentials={real_user=slave}} blocked reactor thread for 365362us
W20260129 03:00:05.053005 20553 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.211.1:42027, user_credentials={real_user=slave}} blocked reactor thread for 452723us
I20260129 03:00:05.017015 20479 leader_election.cc:304] T 87d5fe48a44a408aaf03e894a67ecb2f P 974b802141bd4f13beffef2aa03f88f9 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 974b802141bd4f13beffef2aa03f88f9; no voters: 197b880fa39d432a9b472845876bfaf0, 7a9daf5d42a84615a6383cfec9a10608
W20260129 03:00:05.013140 20555 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260129 03:00:05.011170 23026 consensus_meta.cc:287] T c2633a6fc34c44579de348d208b0c01e P 1c50de89a5a845668696dcb28c3badd6: Time spent flushing consensus metadata: real 0.762s user 0.017s sys 0.002s
I20260129 03:00:05.004637 23041 raft_consensus.cc:493] T 45cc5f3d2688486d9b6dbe7e666dedde P 7a9daf5d42a84615a6383cfec9a10608 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260129 03:00:05.059072 20480 leader_election.cc:304] T c891d2bff4ce4568861bbc992e91fbb0 P 974b802141bd4f13beffef2aa03f88f9 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1c50de89a5a845668696dcb28c3badd6, 974b802141bd4f13beffef2aa03f88f9; no voters:
I20260129 03:00:05.009902 23107 raft_consensus.cc:493] T 0678b8b468bd43cdbc3e485900844821 P 7a9daf5d42a84615a6383cfec9a10608 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 974b802141bd4f13beffef2aa03f88f9)
I20260129 03:00:05.057992 23057 raft_consensus.cc:493] T cb4727386a3e48aca5d71cc6c611c197 P 197b880fa39d432a9b472845876bfaf0 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260129 03:00:05.076315 20432 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:48356 (request call id 380) took 1748 ms. Trace:
I20260129 03:00:05.076728 20432 rpcz_store.cc:276] 0129 03:00:03.247537 (+ 0us) service_pool.cc:168] Inserting onto call queue
0129 03:00:03.247666 (+ 129us) service_pool.cc:225] Handling call
0129 03:00:03.248025 (+ 359us) raft_consensus.cc:1471] Updating replica for 0 ops
0129 03:00:03.248206 (+ 181us) raft_consensus.cc:1534] Early marking committed up to index 5
0129 03:00:03.556057 (+307851us) log.cc:844] Serialized 14 byte log entry
0129 03:00:03.583305 (+ 27248us) log.cc:844] Serialized 14 byte log entry
0129 03:00:04.539253 (+955948us) spinlock_profiling.cc:243] Waited 4.61 ms on lock 0x7b18004e66a0. stack: 00007f8fe044e373 00007f8fe044e270 00007f8fdfda0574 00007f8fe9a5c6b4 00007f8fe9a5c649 00007f8fe92d4c91 00007f8fe92e1671 00007f8fe03f546a 00007f8fe9a7b5c9 00007f8fe02b5160 00007f8fe047d3c4 00007f8fe047d335 00007f8fe3525ce0 00007f8fe3538396 00007f8fe3548f07 00007f8fe3548e88
0129 03:00:04.592179 (+ 52926us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0129 03:00:04.600832 (+ 8653us) raft_consensus.cc:1645] Marking committed up to 5
0129 03:00:04.920490 (+319658us) raft_consensus.cc:1697] Filling consensus response to leader.
0129 03:00:04.962188 (+ 41698us) raft_consensus.cc:1691] UpdateReplicas() finished
0129 03:00:04.996365 (+ 34177us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":25669,"prepare.queue_time_us":31756,"prepare.run_cpu_time_us":53,"prepare.run_wall_time_us":54,"raft.queue_time_us":960923,"raft.run_cpu_time_us":419,"raft.run_wall_time_us":420,"spinlock_wait_cycles":10605440}
W20260129 03:00:05.079092 21102 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260129 03:00:05.079988 21102 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260129 03:00:05.071102 23043 raft_consensus.cc:515] T febffe4a8b394368b69b1d164308fb9a P 197b880fa39d432a9b472845876bfaf0 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } }
I20260129 03:00:05.080067 23041 raft_consensus.cc:515] T 45cc5f3d2688486d9b6dbe7e666dedde P 7a9daf5d42a84615a6383cfec9a10608 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } }
I20260129 03:00:05.082167 23043 leader_election.cc:290] T febffe4a8b394368b69b1d164308fb9a P 197b880fa39d432a9b472845876bfaf0 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 974b802141bd4f13beffef2aa03f88f9 (127.18.211.2:34517), 7a9daf5d42a84615a6383cfec9a10608 (127.18.211.4:40439)
I20260129 03:00:05.082317 23041 leader_election.cc:290] T 45cc5f3d2688486d9b6dbe7e666dedde P 7a9daf5d42a84615a6383cfec9a10608 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 974b802141bd4f13beffef2aa03f88f9 (127.18.211.2:34517), 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3:37735)
I20260129 03:00:05.082635 23255 raft_consensus.cc:493] T add8d4923d1a489e86882c3a25692b95 P 7a9daf5d42a84615a6383cfec9a10608 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260129 03:00:05.080086 23163 raft_consensus.cc:493] T 18f30ebaa4174fd28c958e72d0cc7e24 P 7a9daf5d42a84615a6383cfec9a10608 [term 6 FOLLOWER]: Starting leader election (detected failure of leader 1c50de89a5a845668696dcb28c3badd6)
I20260129 03:00:05.083563 23163 raft_consensus.cc:3060] T 18f30ebaa4174fd28c958e72d0cc7e24 P 7a9daf5d42a84615a6383cfec9a10608 [term 6 FOLLOWER]: Advancing to term 7
I20260129 03:00:05.083307 23255 raft_consensus.cc:515] T add8d4923d1a489e86882c3a25692b95 P 7a9daf5d42a84615a6383cfec9a10608 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } }
I20260129 03:00:05.085747 23255 leader_election.cc:290] T add8d4923d1a489e86882c3a25692b95 P 7a9daf5d42a84615a6383cfec9a10608 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 974b802141bd4f13beffef2aa03f88f9 (127.18.211.2:34517), 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3:37735)
I20260129 03:00:05.066570 23208 raft_consensus.cc:2749] T 87d5fe48a44a408aaf03e894a67ecb2f P 974b802141bd4f13beffef2aa03f88f9 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260129 03:00:05.086970 23262 raft_consensus.cc:493] T cb0b3f259ebc4310927338a8740e87b8 P 1c50de89a5a845668696dcb28c3badd6 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260129 03:00:05.087091 20585 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35754 (request call id 1734) took 4064 ms. Trace:
I20260129 03:00:05.087654 23262 raft_consensus.cc:515] T cb0b3f259ebc4310927338a8740e87b8 P 1c50de89a5a845668696dcb28c3badd6 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } }
I20260129 03:00:05.088065 23089 raft_consensus.cc:2804] T 7a9a69cd82724b1faed1d10aa091fc4a P 197b880fa39d432a9b472845876bfaf0 [term 7 FOLLOWER]: Leader election won for term 7
I20260129 03:00:05.088939 23089 raft_consensus.cc:697] T 7a9a69cd82724b1faed1d10aa091fc4a P 197b880fa39d432a9b472845876bfaf0 [term 7 LEADER]: Becoming Leader. State: Replica: 197b880fa39d432a9b472845876bfaf0, State: Running, Role: LEADER
W20260129 03:00:05.089598 20512 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:57584 (request call id 1416) took 2195 ms (client timeout 1502 ms). Trace:
I20260129 03:00:05.090131 23262 leader_election.cc:290] T cb0b3f259ebc4310927338a8740e87b8 P 1c50de89a5a845668696dcb28c3badd6 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 974b802141bd4f13beffef2aa03f88f9 (127.18.211.2:34517), 7a9daf5d42a84615a6383cfec9a10608 (127.18.211.4:40439)
I20260129 03:00:05.090469 20581 raft_consensus.cc:1275] T cb0b3f259ebc4310927338a8740e87b8 P 1c50de89a5a845668696dcb28c3badd6 [term 7 FOLLOWER]: Refusing update from remote peer 7a9daf5d42a84615a6383cfec9a10608: Log matching property violated. Preceding OpId in replica: term: 5 index: 7. Preceding OpId from leader: term: 7 index: 8. (index mismatch)
I20260129 03:00:05.092010 20438 raft_consensus.cc:2393] T 00e79d8f25a34676b1d866eb2068a2b2 P 197b880fa39d432a9b472845876bfaf0 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 7a9daf5d42a84615a6383cfec9a10608 in current term 8: Already voted for candidate 197b880fa39d432a9b472845876bfaf0 in this term.
I20260129 03:00:05.087584 20585 rpcz_store.cc:276] 0129 03:00:01.022683 (+ 0us) service_pool.cc:168] Inserting onto call queue
0129 03:00:02.086483 (+1063800us) service_pool.cc:225] Handling call
0129 03:00:02.087758 (+ 1275us) raft_consensus.cc:1471] Updating replica for 1 ops
0129 03:00:02.088006 (+ 248us) raft_consensus.cc:1534] Early marking committed up to index 0
0129 03:00:02.088030 (+ 24us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0129 03:00:02.100134 (+ 12104us) log.cc:844] Serialized 26 byte log entry
0129 03:00:02.400939 (+300805us) raft_consensus.cc:1645] Marking committed up to 3
0129 03:00:02.583098 (+182159us) log.cc:844] Serialized 14 byte log entry
0129 03:00:02.584416 (+ 1318us) raft_consensus.cc:1697] Filling consensus response to leader.
0129 03:00:02.584498 (+ 82us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0129 03:00:05.081338 (+2496840us) spinlock_profiling.cc:243] Waited 5.25 ms on lock 0x7b18001d47c0. stack: 00007f8fe044e373 00007f8fe044e270 00007f8fdfda0574 00007f8fe9a5c6b4 00007f8fe9a5c649 00007f8fe92d4c91 00007f8fe92e1671 00007f8fe04802d3 00007f8fe04826a1 00007f8fe0482659 00007f8fe04825e9 00007f8fe04825b1 00007f8fe04818ad 0000000000379254 0000000000379099 00007f8fe046369a
0129 03:00:05.084542 (+ 3204us) spinlock_profiling.cc:243] Waited 7.97 ms on lock 0x7b18001d47c0. stack: 00007f8fe044e373 00007f8fe044e270 00007f8fdfda0574 00007f8fe9a5c6b4 00007f8fe9a5c649 00007f8fe92d4c91 00007f8fe92e1671 00007f8fe04802d3 00007f8fe04826a1 00007f8fe0482659 00007f8fe04825e9 00007f8fe04825b1 00007f8fe04818ad 0000000000379254 0000000000379099 00007f8fe046369a
0129 03:00:05.086711 (+ 2169us) raft_consensus.cc:1685] finished
0129 03:00:05.086735 (+ 24us) raft_consensus.cc:1691] UpdateReplicas() finished
0129 03:00:05.086885 (+ 150us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":217961,"prepare.run_cpu_time_us":55,"prepare.run_wall_time_us":57,"raft.queue_time_us":85,"raft.run_cpu_time_us":150,"raft.run_wall_time_us":10491,"spinlock_wait_cycles":30392576,"thread_start_us":316697,"threads_started":2,"wal-append.queue_time_us":445345}
W20260129 03:00:05.093896 20404 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260129 03:00:05.057219 23165 leader_election.cc:290] T 4f5a77e94fef4b16a2ba7b067c8b144a P 7a9daf5d42a84615a6383cfec9a10608 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 974b802141bd4f13beffef2aa03f88f9 (127.18.211.2:34517), 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3:37735)
W20260129 03:00:05.090054 20512 rpcz_store.cc:269] 0129 03:00:02.893698 (+ 0us) service_pool.cc:168] Inserting onto call queue
0129 03:00:03.125327 (+231629us) service_pool.cc:225] Handling call
0129 03:00:05.089170 (+1963843us) spinlock_profiling.cc:243] Waited 917 ms on lock 0x7b5400229554. stack: 00007f8fe044e373 00007f8fe044e270 00007f8fdfda0574 00007f8fe9a5c6b4 00007f8fe9a5c649 00007f8fe34975a1 00007f8fe3531f7a 00007f8fe84e3647 00007f8fe1a73834 00007f8fe1a737c2 00007f8fe1a736f1 00007f8fe1a7366c 00007f8fe1a72922 00007f8fe186d34c 00007f8fe186c786 00007f8fe186c12f
0129 03:00:05.089379 (+ 209us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2109280512}
W20260129 03:00:05.074842 20556 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.211.2:34517, user_credentials={real_user=slave}} blocked reactor thread for 70709.2us
I20260129 03:00:05.100692 20441 raft_consensus.cc:1275] T fd8311b779af4e1fa1402b2a02a602f2 P 197b880fa39d432a9b472845876bfaf0 [term 5 FOLLOWER]: Refusing update from remote peer 1c50de89a5a845668696dcb28c3badd6: Log matching property violated. Preceding OpId in replica: term: 4 index: 6. Preceding OpId from leader: term: 5 index: 7. (index mismatch)
I20260129 03:00:05.062558 23104 raft_consensus.cc:2749] T fcddab6bca684adf8ea032289a00f90e P 1c50de89a5a845668696dcb28c3badd6 [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20260129 03:00:05.104131 20588 raft_consensus.cc:1240] T 1299723a4476420ab02fd8bfd7aa0645 P 1c50de89a5a845668696dcb28c3badd6 [term 9 FOLLOWER]: Rejecting Update request from peer 7a9daf5d42a84615a6383cfec9a10608 for earlier term 8. Current term is 9. Ops: []
I20260129 03:00:05.100934 20407 leader_election.cc:304] T 00e79d8f25a34676b1d866eb2068a2b2 P 197b880fa39d432a9b472845876bfaf0 [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 197b880fa39d432a9b472845876bfaf0, 974b802141bd4f13beffef2aa03f88f9; no voters:
W20260129 03:00:05.104866 20480 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.211.3:37735, user_credentials={real_user=slave}} blocked reactor thread for 125204us
I20260129 03:00:05.096194 23057 raft_consensus.cc:3060] T cb4727386a3e48aca5d71cc6c611c197 P 197b880fa39d432a9b472845876bfaf0 [term 4 FOLLOWER]: Advancing to term 5
W20260129 03:00:05.111518 20554 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260129 03:00:05.113209 20585 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "dc2f4cb752cf4d59b5f6bc689f5d8104" candidate_uuid: "7a9daf5d42a84615a6383cfec9a10608" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "1c50de89a5a845668696dcb28c3badd6"
I20260129 03:00:05.112327 20588 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:48526 (request call id 343) took 2065 ms. Trace:
I20260129 03:00:05.116812 20588 rpcz_store.cc:276] 0129 03:00:03.046755 (+ 0us) service_pool.cc:168] Inserting onto call queue
0129 03:00:04.987707 (+1940952us) service_pool.cc:225] Handling call
0129 03:00:05.061327 (+ 73620us) raft_consensus.cc:1471] Updating replica for 0 ops
0129 03:00:05.108651 (+ 47324us) raft_consensus.cc:1697] Filling consensus response to leader.
0129 03:00:05.112255 (+ 3604us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260129 03:00:05.069211 23026 raft_consensus.cc:515] T c2633a6fc34c44579de348d208b0c01e P 1c50de89a5a845668696dcb28c3badd6 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 03:00:05.120239 23026 leader_election.cc:290] T c2633a6fc34c44579de348d208b0c01e P 1c50de89a5a845668696dcb28c3badd6 [CANDIDATE]: Term 4 election: Requested vote from peers 197b880fa39d432a9b472845876bfaf0 (127.18.211.1:42027), 974b802141bd4f13beffef2aa03f88f9 (127.18.211.2:34517)
I20260129 03:00:05.083736 23180 raft_consensus.cc:3055] T cedce331e1584ac8b636c9b8a72b2ba0 P 197b880fa39d432a9b472845876bfaf0 [term 6 LEADER]: Stepping down as leader of term 6
I20260129 03:00:05.134514 20480 leader_election.cc:400] T cb0b3f259ebc4310927338a8740e87b8 P 974b802141bd4f13beffef2aa03f88f9 [CANDIDATE]: Term 6 pre-election: Vote denied by peer 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3:37735) with higher term. Message: Invalid argument: T cb0b3f259ebc4310927338a8740e87b8 P 1c50de89a5a845668696dcb28c3badd6 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 974b802141bd4f13beffef2aa03f88f9 for earlier term 6. Current term is 7.
I20260129 03:00:05.113186 20553 leader_election.cc:304] T 8055232cf0004d40a85f3bf983fe00a9 P 1c50de89a5a845668696dcb28c3badd6 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1c50de89a5a845668696dcb28c3badd6; no voters: 197b880fa39d432a9b472845876bfaf0, 974b802141bd4f13beffef2aa03f88f9
I20260129 03:00:05.134902 20480 leader_election.cc:403] T cb0b3f259ebc4310927338a8740e87b8 P 974b802141bd4f13beffef2aa03f88f9 [CANDIDATE]: Term 6 pre-election: Cancelling election due to peer responding with higher term
I20260129 03:00:05.066720 23216 raft_consensus.cc:515] T 4b1876f9a70b49edb3aa74105e66d3b3 P 974b802141bd4f13beffef2aa03f88f9 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 03:00:05.141958 20585 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4f5a77e94fef4b16a2ba7b067c8b144a" candidate_uuid: "7a9daf5d42a84615a6383cfec9a10608" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "1c50de89a5a845668696dcb28c3badd6" is_pre_election: true
I20260129 03:00:05.142179 23250 consensus_queue.cc:1059] T dc2f4cb752cf4d59b5f6bc689f5d8104 P 1c50de89a5a845668696dcb28c3badd6 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false }, Status: INVALID_TERM, Last received: 4.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.038s
I20260129 03:00:05.137992 19276 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260129 03:00:05.138262 21136 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "cb0b3f259ebc4310927338a8740e87b8" candidate_uuid: "1c50de89a5a845668696dcb28c3badd6" candidate_term: 8 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "7a9daf5d42a84615a6383cfec9a10608" is_pre_election: true
I20260129 03:00:05.121601 22694 raft_consensus.cc:2804] T c891d2bff4ce4568861bbc992e91fbb0 P 974b802141bd4f13beffef2aa03f88f9 [term 7 FOLLOWER]: Leader pre-election won for term 8
W20260129 03:00:05.147392 20404 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260129 03:00:05.148495 23101 consensus_meta.cc:287] T 87d5fe48a44a408aaf03e894a67ecb2f P 197b880fa39d432a9b472845876bfaf0: Time spent flushing consensus metadata: real 1.551s user 0.008s sys 0.011s
I20260129 03:00:05.147495 20588 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "add8d4923d1a489e86882c3a25692b95" candidate_uuid: "7a9daf5d42a84615a6383cfec9a10608" candidate_term: 7 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "1c50de89a5a845668696dcb28c3badd6" is_pre_election: true
I20260129 03:00:05.134883 23251 consensus_queue.cc:1048] T fd8311b779af4e1fa1402b2a02a602f2 P 1c50de89a5a845668696dcb28c3badd6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.017s
I20260129 03:00:05.134331 23180 raft_consensus.cc:740] T cedce331e1584ac8b636c9b8a72b2ba0 P 197b880fa39d432a9b472845876bfaf0 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 197b880fa39d432a9b472845876bfaf0, State: Running, Role: LEADER
W20260129 03:00:05.149966 20407 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.211.2:34517, user_credentials={real_user=slave}} blocked reactor thread for 80061.9us
W20260129 03:00:05.157212 23243 log.cc:927] Time spent T 1ce4ddf5b10e4ae8b193b13b2e5936ee P 197b880fa39d432a9b472845876bfaf0: Append to log took a long time: real 0.085s user 0.000s sys 0.001s
W20260129 03:00:05.160807 20405 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.211.3:37735, user_credentials={real_user=slave}} blocked reactor thread for 1.6272e+06us
I20260129 03:00:05.151628 23091 raft_consensus.cc:2804] T 00e79d8f25a34676b1d866eb2068a2b2 P 197b880fa39d432a9b472845876bfaf0 [term 8 FOLLOWER]: Leader election won for term 8
W20260129 03:00:05.166097 20479 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.211.1:42027, user_credentials={real_user=slave}} blocked reactor thread for 178182us
I20260129 03:00:05.167210 20515 raft_consensus.cc:2468] T 2ac659d732314e1288849f08fe5e074e P 974b802141bd4f13beffef2aa03f88f9 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7a9daf5d42a84615a6383cfec9a10608 in term 5.
I20260129 03:00:05.090153 23089 consensus_queue.cc:237] T 7a9a69cd82724b1faed1d10aa091fc4a P 197b880fa39d432a9b472845876bfaf0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 6.4, Last appended by leader: 4, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 03:00:05.152294 23104 raft_consensus.cc:2749] T 8055232cf0004d40a85f3bf983fe00a9 P 1c50de89a5a845668696dcb28c3badd6 [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260129 03:00:05.169180 22851 raft_consensus.cc:493] T 87d5fe48a44a408aaf03e894a67ecb2f P 197b880fa39d432a9b472845876bfaf0 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260129 03:00:05.171145 23032 leader_election.cc:290] T 4408006eb5174de1896b037cf3f18457 P 974b802141bd4f13beffef2aa03f88f9 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 197b880fa39d432a9b472845876bfaf0 (127.18.211.1:42027), 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3:37735)
W20260129 03:00:05.149179 21099 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.18.211.1:42027 timed out after 3.654s (SENT) [suppressed 36 similar messages]
I20260129 03:00:05.172978 21132 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:47844 (request call id 346) took 4074 ms. Trace:
W20260129 03:00:05.173785 20508 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:57584 (request call id 1411) took 4575 ms (client timeout 1587 ms). Trace:
I20260129 03:00:05.176039 20509 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57584 (request call id 1417) took 2281 ms. Trace:
W20260129 03:00:05.176522 22996 consensus_meta.cc:287] T 4f5a77e94fef4b16a2ba7b067c8b144a P 1c50de89a5a845668696dcb28c3badd6: Time spent flushing consensus metadata: real 0.703s user 0.004s sys 0.009s
I20260129 03:00:05.152774 22694 raft_consensus.cc:493] T c891d2bff4ce4568861bbc992e91fbb0 P 974b802141bd4f13beffef2aa03f88f9 [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260129 03:00:05.177793 23216 leader_election.cc:290] T 4b1876f9a70b49edb3aa74105e66d3b3 P 974b802141bd4f13beffef2aa03f88f9 [CANDIDATE]: Term 6 election: Requested vote from peers 197b880fa39d432a9b472845876bfaf0 (127.18.211.1:42027), 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3:37735)
I20260129 03:00:05.181442 20405 leader_election.cc:304] T 17d6431cc77a491e8464d68db4081a51 P 197b880fa39d432a9b472845876bfaf0 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 197b880fa39d432a9b472845876bfaf0, 1c50de89a5a845668696dcb28c3badd6; no voters: 974b802141bd4f13beffef2aa03f88f9
I20260129 03:00:05.181900 21127 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34046 (request call id 547) took 3368 ms. Trace:
I20260129 03:00:05.184237 23091 raft_consensus.cc:697] T 00e79d8f25a34676b1d866eb2068a2b2 P 197b880fa39d432a9b472845876bfaf0 [term 8 LEADER]: Becoming Leader. State: Replica: 197b880fa39d432a9b472845876bfaf0, State: Running, Role: LEADER
I20260129 03:00:05.185446 23180 consensus_queue.cc:260] T cedce331e1584ac8b636c9b8a72b2ba0 P 197b880fa39d432a9b472845876bfaf0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 6.4, Last appended by leader: 4, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 03:00:05.186971 23168 leader_election.cc:290] T c934d31efd3f4e5195605ddc354d14e1 P 7a9daf5d42a84615a6383cfec9a10608 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 197b880fa39d432a9b472845876bfaf0 (127.18.211.1:42027), 974b802141bd4f13beffef2aa03f88f9 (127.18.211.2:34517)
W20260129 03:00:05.187837 20553 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.211.1:42027, user_credentials={real_user=slave}} blocked reactor thread for 115953us
I20260129 03:00:05.188848 21128 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34046 (request call id 556) took 2799 ms. Trace:
W20260129 03:00:05.189224 20556 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.211.2:34517, user_credentials={real_user=slave}} blocked reactor thread for 47416.6us
I20260129 03:00:05.189692 20590 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35752 (request call id 1654) took 2342 ms. Trace:
I20260129 03:00:05.190237 23026 consensus_queue.cc:1048] T 7d9494f5584d4d5db8b9adf1c8fc888b P 1c50de89a5a845668696dcb28c3badd6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
W20260129 03:00:05.194527 20480 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.211.3:37735, user_credentials={real_user=slave}} blocked reactor thread for 73793.9us
I20260129 03:00:05.204052 21132 rpcz_store.cc:276] 0129 03:00:01.098868 (+ 0us) service_pool.cc:168] Inserting onto call queue
0129 03:00:01.566104 (+467236us) service_pool.cc:225] Handling call
0129 03:00:01.566408 (+ 304us) raft_consensus.cc:1471] Updating replica for 0 ops
0129 03:00:02.606755 (+1040347us) raft_consensus.cc:1697] Filling consensus response to leader.
0129 03:00:05.172485 (+2565730us) spinlock_profiling.cc:243] Waited 1.04 s on lock 0x7b5400563e54. stack: 00007f8fe044e373 00007f8fe044e270 00007f8fdfda0574 00007f8fe9a5c6b4 00007f8fe9a5c649 00007f8fe34975a1 00007f8fe352ea81 00007f8fe352d4be 00007f8fe84e1f49 00007f8fe1a6ed34 00007f8fe1a6ecc2 00007f8fe1a6ebf1 00007f8fe1a6eb6c 00007f8fe1a6de22 00007f8fe186d34c 00007f8fe186c786
0129 03:00:05.172776 (+ 291us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2384935168}
W20260129 03:00:05.204666 20508 rpcz_store.cc:269] 0129 03:00:00.597855 (+ 0us) service_pool.cc:168] Inserting onto call queue
0129 03:00:00.598988 (+ 1133us) service_pool.cc:225] Handling call
0129 03:00:05.173425 (+4574437us) spinlock_profiling.cc:243] Waited 1.33 s on lock 0x7b54003df9d4. stack: 00007f8fe044e373 00007f8fe044e270 00007f8fdfda0574 00007f8fe9a5c6b4 00007f8fe9a5c649 00007f8fe34975a1 00007f8fe3531f7a 00007f8fe84e3647 00007f8fe1a73834 00007f8fe1a737c2 00007f8fe1a736f1 00007f8fe1a7366c 00007f8fe1a72922 00007f8fe186d34c 00007f8fe186c786 00007f8fe186c12f
0129 03:00:05.173593 (+ 168us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":3047711616}
I20260129 03:00:05.204825 20509 rpcz_store.cc:276] 0129 03:00:02.894394 (+ 0us) service_pool.cc:168] Inserting onto call queue
0129 03:00:04.213758 (+1319364us) service_pool.cc:225] Handling call
0129 03:00:04.214153 (+ 395us) raft_consensus.cc:1471] Updating replica for 0 ops
0129 03:00:04.509421 (+295268us) raft_consensus.cc:1697] Filling consensus response to leader.
0129 03:00:05.175725 (+666304us) spinlock_profiling.cc:243] Waited 294 ms on lock 0x7b54000fc654. stack: 00007f8fe044e373 00007f8fe044e270 00007f8fdfda0574 00007f8fe9a5c6b4 00007f8fe9a5c649 00007f8fe34975a1 00007f8fe352ea81 00007f8fe352d4be 00007f8fe84e1f49 00007f8fe1a6ed34 00007f8fe1a6ecc2 00007f8fe1a6ebf1 00007f8fe1a6eb6c 00007f8fe1a6de22 00007f8fe186d34c 00007f8fe186c786
0129 03:00:05.175960 (+ 235us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":676368768}
I20260129 03:00:05.146827 23107 raft_consensus.cc:515] T 0678b8b468bd43cdbc3e485900844821 P 7a9daf5d42a84615a6383cfec9a10608 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } }
W20260129 03:00:05.203136 22950 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.611s user 0.024s sys 0.020s
I20260129 03:00:05.223837 20585 raft_consensus.cc:2468] T 4f5a77e94fef4b16a2ba7b067c8b144a P 1c50de89a5a845668696dcb28c3badd6 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7a9daf5d42a84615a6383cfec9a10608 in term 4.
I20260129 03:00:05.231547 20588 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "45cc5f3d2688486d9b6dbe7e666dedde" candidate_uuid: "7a9daf5d42a84615a6383cfec9a10608" candidate_term: 6 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "1c50de89a5a845668696dcb28c3badd6" is_pre_election: true
I20260129 03:00:05.208981 22694 raft_consensus.cc:3060] T c891d2bff4ce4568861bbc992e91fbb0 P 974b802141bd4f13beffef2aa03f88f9 [term 7 FOLLOWER]: Advancing to term 8
I20260129 03:00:05.205192 23232 leader_election.cc:290] T 8055232cf0004d40a85f3bf983fe00a9 P 974b802141bd4f13beffef2aa03f88f9 [CANDIDATE]: Term 5 election: Requested vote from peers 197b880fa39d432a9b472845876bfaf0 (127.18.211.1:42027), 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3:37735)
I20260129 03:00:05.208889 22989 raft_consensus.cc:2749] T cb0b3f259ebc4310927338a8740e87b8 P 974b802141bd4f13beffef2aa03f88f9 [term 7 FOLLOWER]: Leader pre-election lost for term 6. Reason: Vote denied by peer 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3:37735) with higher term. Message: Invalid argument: T cb0b3f259ebc4310927338a8740e87b8 P 1c50de89a5a845668696dcb28c3badd6 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 974b802141bd4f13beffef2aa03f88f9 for earlier term 6. Current term is 7.
W20260129 03:00:05.240572 20407 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.211.2:34517, user_credentials={real_user=slave}} blocked reactor thread for 44409.9us
W20260129 03:00:05.238619 22950 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.647s user 0.024s sys 0.020s
I20260129 03:00:05.252712 23091 consensus_queue.cc:237] T 00e79d8f25a34676b1d866eb2068a2b2 P 197b880fa39d432a9b472845876bfaf0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 6.8, Last appended by leader: 8, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } }
I20260129 03:00:05.253702 20514 raft_consensus.cc:3055] T 4e420f90884f4796997b5ea41e49e411 P 974b802141bd4f13beffef2aa03f88f9 [term 5 LEADER]: Stepping down as leader of term 5
I20260129 03:00:05.254094 20514 raft_consensus.cc:740] T 4e420f90884f4796997b5ea41e49e411 P 974b802141bd4f13beffef2aa03f88f9 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 974b802141bd4f13beffef2aa03f88f9, State: Running, Role: LEADER
I20260129 03:00:05.255052 20514 consensus_queue.cc:260] T 4e420f90884f4796997b5ea41e49e411 P 974b802141bd4f13beffef2aa03f88f9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 5.7, Last appended by leader: 7, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } }
I20260129 03:00:05.257091 20588 raft_consensus.cc:1275] T 505580c93c694bbb8ea96fdf9b17ab46 P 1c50de89a5a845668696dcb28c3badd6 [term 6 FOLLOWER]: Refusing update from remote peer 7a9daf5d42a84615a6383cfec9a10608: Log matching property violated. Preceding OpId in replica: term: 5 index: 7. Preceding OpId from leader: term: 6 index: 8. (index mismatch)
I20260129 03:00:05.258280 23107 leader_election.cc:290] T 0678b8b468bd43cdbc3e485900844821 P 7a9daf5d42a84615a6383cfec9a10608 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 197b880fa39d432a9b472845876bfaf0 (127.18.211.1:42027), 974b802141bd4f13beffef2aa03f88f9 (127.18.211.2:34517)
I20260129 03:00:05.235316 23086 raft_consensus.cc:493] T a8479d1fe0bd41e5946186e3a6ca8c7a P 197b880fa39d432a9b472845876bfaf0 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260129 03:00:05.259912 23086 raft_consensus.cc:515] T a8479d1fe0bd41e5946186e3a6ca8c7a P 197b880fa39d432a9b472845876bfaf0 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 03:00:05.252365 23085 raft_consensus.cc:493] T a0fcc58d6fb6473cafbd9d34d5195269 P 197b880fa39d432a9b472845876bfaf0 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260129 03:00:05.222501 21127 rpcz_store.cc:276] 0129 03:00:01.813250 (+ 0us) service_pool.cc:168] Inserting onto call queue
0129 03:00:01.814107 (+ 857us) service_pool.cc:225] Handling call
0129 03:00:03.479096 (+1664989us) raft_consensus.cc:1471] Updating replica for 0 ops
0129 03:00:04.587713 (+1108617us) raft_consensus.cc:1697] Filling consensus response to leader.
0129 03:00:05.180677 (+592964us) spinlock_profiling.cc:243] Waited 1.05 s on lock 0x7b540068e2d4. stack: 00007f8fe044e373 00007f8fe044e270 00007f8fdfda0574 00007f8fe9a5c6b4 00007f8fe9a5c649 00007f8fe34975a1 00007f8fe352ea81 00007f8fe352d4be 00007f8fe84e1f49 00007f8fe1a6ed34 00007f8fe1a6ecc2 00007f8fe1a6ebf1 00007f8fe1a6eb6c 00007f8fe1a6de22 00007f8fe186d34c 00007f8fe186c786
0129 03:00:05.181567 (+ 890us) spinlock_profiling.cc:243] Waited 1.66 s on lock 0x7b540068e2d0. stack: 00007f8fe044e373 00007f8fe044e270 00007f8fdfda0574 00007f8fe9a5c6b4 00007f8fe9a5c649 00007f8fe34975a1 00007f8fe352d52d 00007f8fe84e1f49 00007f8fe1a6ed34 00007f8fe1a6ecc2 00007f8fe1a6ebf1 00007f8fe1a6eb6c 00007f8fe1a6de22 00007f8fe186d34c 00007f8fe186c786 00007f8fe186c12f
0129 03:00:05.181807 (+ 240us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":6245750400}
I20260129 03:00:05.262437 23086 leader_election.cc:290] T a8479d1fe0bd41e5946186e3a6ca8c7a P 197b880fa39d432a9b472845876bfaf0 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 974b802141bd4f13beffef2aa03f88f9 (127.18.211.2:34517), 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3:37735)
I20260129 03:00:05.263849 20436 raft_consensus.cc:3060] T 7d9494f5584d4d5db8b9adf1c8fc888b P 197b880fa39d432a9b472845876bfaf0 [term 3 FOLLOWER]: Advancing to term 4
W20260129 03:00:05.264247 23113 consensus_meta.cc:287] T c2633a6fc34c44579de348d208b0c01e P 974b802141bd4f13beffef2aa03f88f9: Time spent flushing consensus metadata: real 1.149s user 0.009s sys 0.008s
I20260129 03:00:05.264671 20582 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35752 (request call id 1631) took 3400 ms. Trace:
I20260129 03:00:05.264837 23113 raft_consensus.cc:2749] T c2633a6fc34c44579de348d208b0c01e P 974b802141bd4f13beffef2aa03f88f9 [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20260129 03:00:05.241097 23035 consensus_meta.cc:287] T 1299723a4476420ab02fd8bfd7aa0645 P 7a9daf5d42a84615a6383cfec9a10608: Time spent flushing consensus metadata: real 0.730s user 0.012s sys 0.002s
I20260129 03:00:05.265043 20582 rpcz_store.cc:276] 0129 03:00:01.863808 (+ 0us) service_pool.cc:168] Inserting onto call queue
0129 03:00:02.410187 (+546379us) service_pool.cc:225] Handling call
0129 03:00:02.410447 (+ 260us) raft_consensus.cc:1471] Updating replica for 1 ops
0129 03:00:02.411757 (+ 1310us) raft_consensus.cc:1534] Early marking committed up to index 0
0129 03:00:02.411786 (+ 29us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0129 03:00:02.412259 (+ 473us) log.cc:844] Serialized 26 byte log entry
0129 03:00:02.610219 (+197960us) raft_consensus.cc:1645] Marking committed up to 3
0129 03:00:02.663799 (+ 53580us) log.cc:844] Serialized 14 byte log entry
0129 03:00:02.680519 (+ 16720us) raft_consensus.cc:1697] Filling consensus response to leader.
0129 03:00:02.680624 (+ 105us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0129 03:00:03.306953 (+626329us) spinlock_profiling.cc:243] Waited 3.57 ms on lock 0x7b18004b8860. stack: 00007f8fe044e373 00007f8fe044e270 00007f8fdfda0574 00007f8fe9a5c6b4 00007f8fe9a5c649 00007f8fe92d4c91 00007f8fe92e1671 00007f8fe044e2d6 00007f8fe044e270 00007f8fdfda0574 00007f8fe9a5c6b4 00007f8fe9a5c649 00007f8fe92d4c91 00007f8fe92e1671 00007f8fe0480402 00007f8fe04826a1
0129 03:00:05.099015 (+1792062us) spinlock_profiling.cc:243] Waited 7.61 ms on lock 0x7b18004b8860. stack: 00007f8fe044e373 00007f8fe044e270 00007f8fdfda0574 00007f8fe9a5c6b4 00007f8fe9a5c649 00007f8fe92d4c91 00007f8fe92e1671 00007f8fe04802d3 00007f8fe04826a1 00007f8fe0482659 00007f8fe04825e9 00007f8fe04825b1 00007f8fe04818ad 0000000000379254 0000000000379099 00007f8fe046369a
0129 03:00:05.264260 (+165245us) raft_consensus.cc:1685] finished
0129 03:00:05.264283 (+ 23us) raft_consensus.cc:1691] UpdateReplicas() finished
0129 03:00:05.264455 (+ 172us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":16292,"prepare.queue_time_us":47138,"prepare.run_wall_time_us":52,"raft.queue_time_us":17565,"spinlock_wait_cycles":51251584,"thread_start_us":64869,"threads_started":1,"wal-append.queue_time_us":188784}
I20260129 03:00:05.267854 20513 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57584 (request call id 1429) took 2225 ms. Trace:
I20260129 03:00:05.268112 23085 raft_consensus.cc:515] T a0fcc58d6fb6473cafbd9d34d5195269 P 197b880fa39d432a9b472845876bfaf0 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 03:00:05.268380 20513 rpcz_store.cc:276] 0129 03:00:03.042021 (+ 0us) service_pool.cc:168] Inserting onto call queue
0129 03:00:04.267985 (+1225964us) service_pool.cc:225] Handling call
0129 03:00:04.295160 (+ 27175us) raft_consensus.cc:1471] Updating replica for 0 ops
0129 03:00:04.980110 (+684950us) raft_consensus.cc:1697] Filling consensus response to leader.
0129 03:00:05.267543 (+287433us) spinlock_profiling.cc:243] Waited 284 ms on lock 0x7b5400166f54. stack: 00007f8fe044e373 00007f8fe044e270 00007f8fdfda0574 00007f8fe9a5c6b4 00007f8fe9a5c649 00007f8fe34975a1 00007f8fe352ea81 00007f8fe352d4be 00007f8fe84e1f49 00007f8fe1a6ed34 00007f8fe1a6ecc2 00007f8fe1a6ebf1 00007f8fe1a6eb6c 00007f8fe1a6de22 00007f8fe186d34c 00007f8fe186c786
0129 03:00:05.267799 (+ 256us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":652439296}
I20260129 03:00:05.269713 23104 consensus_queue.cc:1059] T 75be4bd6bd45474795fd472eff2953ac P 1c50de89a5a845668696dcb28c3badd6 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 }, Status: INVALID_TERM, Last received: 9.6, Next index: 7, Last known committed idx: 5, Time since last communication: 0.001s
I20260129 03:00:05.270841 23180 raft_consensus.cc:3060] T cedce331e1584ac8b636c9b8a72b2ba0 P 197b880fa39d432a9b472845876bfaf0 [term 6 FOLLOWER]: Advancing to term 7
I20260129 03:00:05.271209 20439 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55672 (request call id 1644) took 2868 ms. Trace:
I20260129 03:00:05.271518 23251 consensus_queue.cc:1059] T add8d4923d1a489e86882c3a25692b95 P 1c50de89a5a845668696dcb28c3badd6 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
I20260129 03:00:05.272059 22748 consensus_queue.cc:1059] T b189f08e16a74a11980cc55d99359a0a P 1c50de89a5a845668696dcb28c3badd6 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 }, Status: INVALID_TERM, Last received: 4.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260129 03:00:05.272702 23251 consensus_queue.cc:1059] T 7a9a69cd82724b1faed1d10aa091fc4a P 1c50de89a5a845668696dcb28c3badd6 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 }, Status: INVALID_TERM, Last received: 6.4, Next index: 5, Last known committed idx: 3, Time since last communication: 0.000s
I20260129 03:00:05.273123 22748 raft_consensus.cc:3055] T 75be4bd6bd45474795fd472eff2953ac P 1c50de89a5a845668696dcb28c3badd6 [term 9 LEADER]: Stepping down as leader of term 9
I20260129 03:00:05.273550 22748 raft_consensus.cc:740] T 75be4bd6bd45474795fd472eff2953ac P 1c50de89a5a845668696dcb28c3badd6 [term 9 LEADER]: Becoming Follower/Learner. State: Replica: 1c50de89a5a845668696dcb28c3badd6, State: Running, Role: LEADER
I20260129 03:00:05.273674 23251 raft_consensus.cc:3055] T add8d4923d1a489e86882c3a25692b95 P 1c50de89a5a845668696dcb28c3badd6 [term 5 LEADER]: Stepping down as leader of term 5
I20260129 03:00:05.274204 23251 raft_consensus.cc:740] T add8d4923d1a489e86882c3a25692b95 P 1c50de89a5a845668696dcb28c3badd6 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 1c50de89a5a845668696dcb28c3badd6, State: Running, Role: LEADER
I20260129 03:00:05.274545 20582 raft_consensus.cc:1240] T 926a250fa5834e88b96ccecbefbf5086 P 1c50de89a5a845668696dcb28c3badd6 [term 7 FOLLOWER]: Rejecting Update request from peer 974b802141bd4f13beffef2aa03f88f9 for earlier term 6. Current term is 7. Ops: [6.4-6.4]
I20260129 03:00:05.274996 23262 raft_consensus.cc:3055] T b189f08e16a74a11980cc55d99359a0a P 1c50de89a5a845668696dcb28c3badd6 [term 8 LEADER]: Stepping down as leader of term 8
I20260129 03:00:05.275276 20582 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35752 (request call id 1660) took 1837 ms. Trace:
I20260129 03:00:05.275394 23262 raft_consensus.cc:740] T b189f08e16a74a11980cc55d99359a0a P 1c50de89a5a845668696dcb28c3badd6 [term 8 LEADER]: Becoming Follower/Learner. State: Replica: 1c50de89a5a845668696dcb28c3badd6, State: Running, Role: LEADER
I20260129 03:00:05.275285 22977 raft_consensus.cc:3055] T 7a9a69cd82724b1faed1d10aa091fc4a P 1c50de89a5a845668696dcb28c3badd6 [term 6 LEADER]: Stepping down as leader of term 6
I20260129 03:00:05.276659 22977 raft_consensus.cc:740] T 7a9a69cd82724b1faed1d10aa091fc4a P 1c50de89a5a845668696dcb28c3badd6 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 1c50de89a5a845668696dcb28c3badd6, State: Running, Role: LEADER
I20260129 03:00:05.276538 23262 consensus_queue.cc:260] T b189f08e16a74a11980cc55d99359a0a P 1c50de89a5a845668696dcb28c3badd6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 8.4, Last appended by leader: 4, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 03:00:05.274569 22748 consensus_queue.cc:260] T 75be4bd6bd45474795fd472eff2953ac P 1c50de89a5a845668696dcb28c3badd6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 9.6, Last appended by leader: 6, Current term: 9, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 03:00:05.278133 23262 raft_consensus.cc:3060] T b189f08e16a74a11980cc55d99359a0a P 1c50de89a5a845668696dcb28c3badd6 [term 8 FOLLOWER]: Advancing to term 9
I20260129 03:00:05.277911 22977 consensus_queue.cc:260] T 7a9a69cd82724b1faed1d10aa091fc4a P 1c50de89a5a845668696dcb28c3badd6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 6.4, Last appended by leader: 4, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
W20260129 03:00:05.278810 20405 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.211.3:37735, user_credentials={real_user=slave}} blocked reactor thread for 97555.5us
I20260129 03:00:05.279361 22977 raft_consensus.cc:3060] T 7a9a69cd82724b1faed1d10aa091fc4a P 1c50de89a5a845668696dcb28c3badd6 [term 6 FOLLOWER]: Advancing to term 7
I20260129 03:00:05.245427 23097 leader_election.cc:290] T 76f713c3331540d2a5b76164103e9eb1 P 974b802141bd4f13beffef2aa03f88f9 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 197b880fa39d432a9b472845876bfaf0 (127.18.211.1:42027), 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3:37735)
I20260129 03:00:05.281255 20405 leader_election.cc:304] T 76f713c3331540d2a5b76164103e9eb1 P 197b880fa39d432a9b472845876bfaf0 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 197b880fa39d432a9b472845876bfaf0; no voters: 1c50de89a5a845668696dcb28c3badd6, 974b802141bd4f13beffef2aa03f88f9
I20260129 03:00:05.282203 23199 raft_consensus.cc:3055] T fcddab6bca684adf8ea032289a00f90e P 974b802141bd4f13beffef2aa03f88f9 [term 8 LEADER]: Stepping down as leader of term 8
I20260129 03:00:05.282647 23199 raft_consensus.cc:740] T fcddab6bca684adf8ea032289a00f90e P 974b802141bd4f13beffef2aa03f88f9 [term 8 LEADER]: Becoming Follower/Learner. State: Replica: 974b802141bd4f13beffef2aa03f88f9, State: Running, Role: LEADER
W20260129 03:00:05.283334 20479 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.211.1:42027, user_credentials={real_user=slave}} blocked reactor thread for 46486.8us
I20260129 03:00:05.283807 23199 consensus_queue.cc:260] T fcddab6bca684adf8ea032289a00f90e P 974b802141bd4f13beffef2aa03f88f9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 8.5, Last appended by leader: 5, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 03:00:05.252015 22645 raft_consensus.cc:2804] T 17d6431cc77a491e8464d68db4081a51 P 197b880fa39d432a9b472845876bfaf0 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260129 03:00:05.285120 22645 raft_consensus.cc:493] T 17d6431cc77a491e8464d68db4081a51 P 197b880fa39d432a9b472845876bfaf0 [term 4 FOLLOWER]: Starting leader election (detected failure of leader 974b802141bd4f13beffef2aa03f88f9)
I20260129 03:00:05.285224 23199 raft_consensus.cc:3060] T fcddab6bca684adf8ea032289a00f90e P 974b802141bd4f13beffef2aa03f88f9 [term 8 FOLLOWER]: Advancing to term 9
I20260129 03:00:05.285553 22645 raft_consensus.cc:3060] T 17d6431cc77a491e8464d68db4081a51 P 197b880fa39d432a9b472845876bfaf0 [term 4 FOLLOWER]: Advancing to term 5
I20260129 03:00:05.284917 20515 raft_consensus.cc:2955] T 96609ac68bdf4993a12e1c23355ca013 P 974b802141bd4f13beffef2aa03f88f9 [term 6 FOLLOWER]: Committing config change with OpId 2.4: config changed from index 2 to 4, 7a9daf5d42a84615a6383cfec9a10608 (127.18.211.4) changed from NON_VOTER to VOTER. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } attrs { replace: true } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } } }
I20260129 03:00:05.227669 21128 rpcz_store.cc:276] 0129 03:00:02.389637 (+ 0us) service_pool.cc:168] Inserting onto call queue
0129 03:00:03.221073 (+831436us) service_pool.cc:225] Handling call
0129 03:00:03.289043 (+ 67970us) raft_consensus.cc:1471] Updating replica for 0 ops
0129 03:00:03.470790 (+181747us) raft_consensus.cc:1697] Filling consensus response to leader.
0129 03:00:05.188181 (+1717391us) spinlock_profiling.cc:243] Waited 181 ms on lock 0x7b54005cd5d4. stack: 00007f8fe044e373 00007f8fe044e270 00007f8fdfda0574 00007f8fe9a5c6b4 00007f8fe9a5c649 00007f8fe34975a1 00007f8fe352ea81 00007f8fe352d4be 00007f8fe84e1f49 00007f8fe1a6ed34 00007f8fe1a6ecc2 00007f8fe1a6ebf1 00007f8fe1a6eb6c 00007f8fe1a6de22 00007f8fe186d34c 00007f8fe186c786
0129 03:00:05.188575 (+ 394us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":416545024}
I20260129 03:00:05.287675 20583 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35752 (request call id 1657) took 2053 ms. Trace:
I20260129 03:00:05.288095 20583 rpcz_store.cc:276] 0129 03:00:03.233787 (+ 0us) service_pool.cc:168] Inserting onto call queue
0129 03:00:05.058401 (+1824614us) service_pool.cc:225] Handling call
0129 03:00:05.058861 (+ 460us) raft_consensus.cc:1471] Updating replica for 1 ops
0129 03:00:05.066387 (+ 7526us) raft_consensus.cc:1534] Early marking committed up to index 0
0129 03:00:05.066423 (+ 36us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0129 03:00:05.074058 (+ 7635us) log.cc:844] Serialized 26 byte log entry
0129 03:00:05.167945 (+ 93887us) raft_consensus.cc:1645] Marking committed up to 4
0129 03:00:05.172574 (+ 4629us) log.cc:844] Serialized 14 byte log entry
0129 03:00:05.190955 (+ 18381us) raft_consensus.cc:1697] Filling consensus response to leader.
0129 03:00:05.191038 (+ 83us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0129 03:00:05.287422 (+ 96384us) raft_consensus.cc:1685] finished
0129 03:00:05.287442 (+ 20us) raft_consensus.cc:1691] UpdateReplicas() finished
0129 03:00:05.287595 (+ 153us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":18117,"prepare.queue_time_us":4114,"prepare.run_cpu_time_us":43,"prepare.run_wall_time_us":43,"raft.queue_time_us":18885,"raft.run_cpu_time_us":296,"raft.run_wall_time_us":5659,"thread_start_us":88524,"threads_started":1,"wal-append.queue_time_us":204663}
I20260129 03:00:05.276839 20581 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:48526 (request call id 344) took 2162 ms. Trace:
I20260129 03:00:05.284025 20480 leader_election.cc:304] T 0340229d521b432580ae926f6062e3f7 P 974b802141bd4f13beffef2aa03f88f9 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1c50de89a5a845668696dcb28c3badd6, 974b802141bd4f13beffef2aa03f88f9; no voters:
I20260129 03:00:05.228077 20590 rpcz_store.cc:276] 0129 03:00:02.846914 (+ 0us) service_pool.cc:168] Inserting onto call queue
0129 03:00:04.614294 (+1767380us) service_pool.cc:225] Handling call
0129 03:00:04.614826 (+ 532us) raft_consensus.cc:1471] Updating replica for 1 ops
0129 03:00:04.615535 (+ 709us) raft_consensus.cc:1534] Early marking committed up to index 0
0129 03:00:04.615567 (+ 32us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0129 03:00:04.617042 (+ 1475us) log.cc:844] Serialized 26 byte log entry
0129 03:00:04.963590 (+346548us) raft_consensus.cc:1645] Marking committed up to 2
0129 03:00:05.167249 (+203659us) log.cc:844] Serialized 14 byte log entry
0129 03:00:05.189346 (+ 22097us) raft_consensus.cc:1697] Filling consensus response to leader.
0129 03:00:05.189425 (+ 79us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0129 03:00:05.189470 (+ 45us) raft_consensus.cc:1685] finished
0129 03:00:05.189482 (+ 12us) raft_consensus.cc:1691] UpdateReplicas() finished
0129 03:00:05.189610 (+ 128us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":21814,"prepare.queue_time_us":117941,"prepare.run_cpu_time_us":30,"prepare.run_wall_time_us":31,"raft.queue_time_us":24379,"raft.run_cpu_time_us":266,"raft.run_wall_time_us":266,"thread_start_us":422151,"threads_started":2,"wal-append.queue_time_us":451986}
I20260129 03:00:05.290822 20581 rpcz_store.cc:276] 0129 03:00:03.114079 (+ 0us) service_pool.cc:168] Inserting onto call queue
0129 03:00:04.995813 (+1881734us) service_pool.cc:225] Handling call
0129 03:00:04.996286 (+ 473us) raft_consensus.cc:1471] Updating replica for 0 ops
0129 03:00:05.090836 (+ 94550us) raft_consensus.cc:1697] Filling consensus response to leader.
0129 03:00:05.276595 (+185759us) spinlock_profiling.cc:243] Waited 93.9 ms on lock 0x7b5400109854. stack: 00007f8fe044e373 00007f8fe044e270 00007f8fdfda0574 00007f8fe9a5c6b4 00007f8fe9a5c649 00007f8fe34975a1 00007f8fe352ea81 00007f8fe352d4be 00007f8fe84e1f49 00007f8fe1a6ed34 00007f8fe1a6ecc2 00007f8fe1a6ebf1 00007f8fe1a6eb6c 00007f8fe1a6de22 00007f8fe186d34c 00007f8fe186c786
0129 03:00:05.276763 (+ 168us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":215867136}
I20260129 03:00:05.275748 20582 rpcz_store.cc:276] 0129 03:00:03.437971 (+ 0us) service_pool.cc:168] Inserting onto call queue
0129 03:00:05.267079 (+1829108us) service_pool.cc:225] Handling call
0129 03:00:05.267579 (+ 500us) raft_consensus.cc:1471] Updating replica for 1 ops
0129 03:00:05.274995 (+ 7416us) raft_consensus.cc:1697] Filling consensus response to leader.
0129 03:00:05.275218 (+ 223us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260129 03:00:05.272083 20553 leader_election.cc:304] T cedce331e1584ac8b636c9b8a72b2ba0 P 1c50de89a5a845668696dcb28c3badd6 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1c50de89a5a845668696dcb28c3badd6; no voters: 197b880fa39d432a9b472845876bfaf0, 974b802141bd4f13beffef2aa03f88f9
I20260129 03:00:05.271654 20439 rpcz_store.cc:276] 0129 03:00:02.402957 (+ 0us) service_pool.cc:168] Inserting onto call queue
0129 03:00:02.413235 (+ 10278us) service_pool.cc:225] Handling call
0129 03:00:02.511127 (+ 97892us) raft_consensus.cc:1471] Updating replica for 0 ops
0129 03:00:02.677916 (+166789us) raft_consensus.cc:1697] Filling consensus response to leader.
0129 03:00:05.270656 (+2592740us) spinlock_profiling.cc:243] Waited 158 ms on lock 0x7b5400266bd4. stack: 00007f8fe044e373 00007f8fe044e270 00007f8fdfda0574 00007f8fe9a5c6b4 00007f8fe9a5c649 00007f8fe34975a1 00007f8fe352ea81 00007f8fe352d4be 00007f8fe84e1f49 00007f8fe1a6ed34 00007f8fe1a6ecc2 00007f8fe1a6ebf1 00007f8fe1a6eb6c 00007f8fe1a6de22 00007f8fe186d34c 00007f8fe186c786
0129 03:00:05.271004 (+ 348us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":362840960}
I20260129 03:00:05.297821 20588 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:48526 (request call id 347) took 1855 ms. Trace:
I20260129 03:00:05.298229 20588 rpcz_store.cc:276] 0129 03:00:03.436819 (+ 0us) service_pool.cc:168] Inserting onto call queue
0129 03:00:05.256275 (+1819456us) service_pool.cc:225] Handling call
0129 03:00:05.256755 (+ 480us) raft_consensus.cc:1471] Updating replica for 0 ops
0129 03:00:05.257488 (+ 733us) raft_consensus.cc:1697] Filling consensus response to leader.
0129 03:00:05.292042 (+ 34554us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260129 03:00:05.299404 20514 raft_consensus.cc:3060] T 4e420f90884f4796997b5ea41e49e411 P 974b802141bd4f13beffef2aa03f88f9 [term 5 FOLLOWER]: Advancing to term 6
I20260129 03:00:05.300979 20508 raft_consensus.cc:1275] T 3e2d15f8640b407195210c476e09df91 P 974b802141bd4f13beffef2aa03f88f9 [term 4 FOLLOWER]: Refusing update from remote peer 7a9daf5d42a84615a6383cfec9a10608: Log matching property violated. Preceding OpId in replica: term: 3 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20260129 03:00:05.303287 23026 raft_consensus.cc:3060] T cedce331e1584ac8b636c9b8a72b2ba0 P 1c50de89a5a845668696dcb28c3badd6 [term 6 FOLLOWER]: Advancing to term 7
I20260129 03:00:05.208053 22851 raft_consensus.cc:515] T 87d5fe48a44a408aaf03e894a67ecb2f P 197b880fa39d432a9b472845876bfaf0 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } }
I20260129 03:00:05.278810 22748 raft_consensus.cc:3060] T 75be4bd6bd45474795fd472eff2953ac P 1c50de89a5a845668696dcb28c3badd6 [term 9 FOLLOWER]: Advancing to term 10
W20260129 03:00:05.307929 21099 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.211.1:42027, user_credentials={real_user=slave}} blocked reactor thread for 251294us
W20260129 03:00:05.313795 20480 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.211.3:37735, user_credentials={real_user=slave}} blocked reactor thread for 70861.1us
I20260129 03:00:05.302372 20556 leader_election.cc:304] T 913fc7d751ed4b7388d889aac30a5f9b P 1c50de89a5a845668696dcb28c3badd6 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1c50de89a5a845668696dcb28c3badd6; no voters: 197b880fa39d432a9b472845876bfaf0, 974b802141bd4f13beffef2aa03f88f9
I20260129 03:00:05.316208 20583 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35752 (request call id 1665) took 1771 ms. Trace:
I20260129 03:00:05.322517 20583 rpcz_store.cc:276] 0129 03:00:03.539021 (+ 0us) service_pool.cc:168] Inserting onto call queue
0129 03:00:05.296149 (+1757128us) service_pool.cc:225] Handling call
0129 03:00:05.306027 (+ 9878us) raft_consensus.cc:1471] Updating replica for 0 ops
0129 03:00:05.309832 (+ 3805us) raft_consensus.cc:1534] Early marking committed up to index 0
0129 03:00:05.309869 (+ 37us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0129 03:00:05.309914 (+ 45us) raft_consensus.cc:1645] Marking committed up to 3
0129 03:00:05.309953 (+ 39us) raft_consensus.cc:1697] Filling consensus response to leader.
0129 03:00:05.310015 (+ 62us) raft_consensus.cc:1691] UpdateReplicas() finished
0129 03:00:05.310115 (+ 100us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260129 03:00:05.271060 22996 consensus_queue.cc:1059] T 3e2d15f8640b407195210c476e09df91 P 1c50de89a5a845668696dcb28c3badd6 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 }, Status: INVALID_TERM, Last received: 3.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.044s
I20260129 03:00:05.288959 23251 consensus_queue.cc:260] T add8d4923d1a489e86882c3a25692b95 P 1c50de89a5a845668696dcb28c3badd6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 5.7, Last appended by leader: 7, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } }
I20260129 03:00:05.304131 23181 raft_consensus.cc:2749] T 76f713c3331540d2a5b76164103e9eb1 P 197b880fa39d432a9b472845876bfaf0 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260129 03:00:05.270870 23104 consensus_queue.cc:1048] T a644c02b16af47679824ccd060ac10fd P 1c50de89a5a845668696dcb28c3badd6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 4, Time since last communication: 0.055s
I20260129 03:00:05.337289 20553 leader_election.cc:304] T a5a05fbcbbcf4916bbc01fd1a5b5e90d P 1c50de89a5a845668696dcb28c3badd6 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1c50de89a5a845668696dcb28c3badd6; no voters: 197b880fa39d432a9b472845876bfaf0, 974b802141bd4f13beffef2aa03f88f9
W20260129 03:00:05.339085 20405 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.211.3:37735, user_credentials={real_user=slave}} blocked reactor thread for 58011.9us
I20260129 03:00:05.342763 23109 consensus_queue.cc:1048] T cb0b3f259ebc4310927338a8740e87b8 P 7a9daf5d42a84615a6383cfec9a10608 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.010s
I20260129 03:00:05.345579 21129 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34046 (request call id 559) took 1829 ms. Trace:
I20260129 03:00:05.345978 21129 rpcz_store.cc:276] 0129 03:00:03.516088 (+ 0us) service_pool.cc:168] Inserting onto call queue
0129 03:00:03.524154 (+ 8066us) service_pool.cc:225] Handling call
0129 03:00:03.524598 (+ 444us) raft_consensus.cc:1471] Updating replica for 0 ops
0129 03:00:03.770694 (+246096us) raft_consensus.cc:1697] Filling consensus response to leader.
0129 03:00:05.345115 (+1574421us) spinlock_profiling.cc:243] Waited 201 ms on lock 0x7b54005cc954. stack: 00007f8fe044e373 00007f8fe044e270 00007f8fdfda0574 00007f8fe9a5c6b4 00007f8fe9a5c649 00007f8fe34975a1 00007f8fe352ea81 00007f8fe352d4be 00007f8fe84e1f49 00007f8fe1a6ed34 00007f8fe1a6ecc2 00007f8fe1a6ebf1 00007f8fe1a6eb6c 00007f8fe1a6de22 00007f8fe186d34c 00007f8fe186c786
0129 03:00:05.345418 (+ 303us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":463015296}
W20260129 03:00:05.348212 23210 log.cc:927] Time spent T 36ff33be666a4c82954731691dbcf327 P 1c50de89a5a845668696dcb28c3badd6: Append to log took a long time: real 2.380s user 0.009s sys 0.013s
I20260129 03:00:05.350032 21133 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34046 (request call id 558) took 1911 ms. Trace:
I20260129 03:00:05.350644 22851 leader_election.cc:290] T 87d5fe48a44a408aaf03e894a67ecb2f P 197b880fa39d432a9b472845876bfaf0 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 974b802141bd4f13beffef2aa03f88f9 (127.18.211.2:34517), 7a9daf5d42a84615a6383cfec9a10608 (127.18.211.4:40439)
I20260129 03:00:05.351451 21130 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34046 (request call id 552) took 3261 ms. Trace:
I20260129 03:00:05.351317 23214 consensus_queue.cc:1059] T 75b0b6e82ddc4f92b1fe1d94f9a0b73b P 1c50de89a5a845668696dcb28c3badd6 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false }, Status: INVALID_TERM, Last received: 3.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.036s
I20260129 03:00:05.351912 21130 rpcz_store.cc:276] 0129 03:00:02.089649 (+ 0us) service_pool.cc:168] Inserting onto call queue
0129 03:00:02.091083 (+ 1434us) service_pool.cc:225] Handling call
0129 03:00:02.160683 (+ 69600us) raft_consensus.cc:1471] Updating replica for 2 ops
0129 03:00:02.860191 (+699508us) raft_consensus.cc:1697] Filling consensus response to leader.
0129 03:00:05.350990 (+2490799us) spinlock_profiling.cc:243] Waited 590 ms on lock 0x7b54005712d4. stack: 00007f8fe044e373 00007f8fe044e270 00007f8fdfda0574 00007f8fe9a5c6b4 00007f8fe9a5c649 00007f8fe34975a1 00007f8fe352ea81 00007f8fe352d4be 00007f8fe84e1f49 00007f8fe1a6ed34 00007f8fe1a6ecc2 00007f8fe1a6ebf1 00007f8fe1a6eb6c 00007f8fe1a6de22 00007f8fe186d34c 00007f8fe186c786
0129 03:00:05.351296 (+ 306us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1356247296}
I20260129 03:00:05.350435 21133 rpcz_store.cc:276] 0129 03:00:03.438971 (+ 0us) service_pool.cc:168] Inserting onto call queue
0129 03:00:03.480843 (+ 41872us) service_pool.cc:225] Handling call
0129 03:00:03.481148 (+ 305us) raft_consensus.cc:1471] Updating replica for 0 ops
0129 03:00:03.516913 (+ 35765us) raft_consensus.cc:1697] Filling consensus response to leader.
0129 03:00:05.349589 (+1832676us) spinlock_profiling.cc:243] Waited 35.1 ms on lock 0x7b540062e8d4. stack: 00007f8fe044e373 00007f8fe044e270 00007f8fdfda0574 00007f8fe9a5c6b4 00007f8fe9a5c649 00007f8fe34975a1 00007f8fe352ea81 00007f8fe352d4be 00007f8fe84e1f49 00007f8fe1a6ed34 00007f8fe1a6ecc2 00007f8fe1a6ebf1 00007f8fe1a6eb6c 00007f8fe1a6de22 00007f8fe186d34c 00007f8fe186c786
0129 03:00:05.349862 (+ 273us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":80743808}
I20260129 03:00:05.354058 22855 raft_consensus.cc:3055] T dc2f4cb752cf4d59b5f6bc689f5d8104 P 1c50de89a5a845668696dcb28c3badd6 [term 4 LEADER]: Stepping down as leader of term 4
I20260129 03:00:05.354247 20587 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35752 (request call id 1635) took 3488 ms. Trace:
I20260129 03:00:05.354493 22855 raft_consensus.cc:740] T dc2f4cb752cf4d59b5f6bc689f5d8104 P 1c50de89a5a845668696dcb28c3badd6 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 1c50de89a5a845668696dcb28c3badd6, State: Running, Role: LEADER
I20260129 03:00:05.354729 20587 rpcz_store.cc:276] 0129 03:00:01.866099 (+ 0us) service_pool.cc:168] Inserting onto call queue
0129 03:00:02.611085 (+744986us) service_pool.cc:225] Handling call
0129 03:00:02.644713 (+ 33628us) raft_consensus.cc:1471] Updating replica for 1 ops
0129 03:00:02.677621 (+ 32908us) raft_consensus.cc:1534] Early marking committed up to index 0
0129 03:00:02.677668 (+ 47us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0129 03:00:02.681679 (+ 4011us) log.cc:844] Serialized 26 byte log entry
0129 03:00:02.815386 (+133707us) raft_consensus.cc:1645] Marking committed up to 3
0129 03:00:02.973238 (+157852us) log.cc:844] Serialized 14 byte log entry
0129 03:00:02.974281 (+ 1043us) raft_consensus.cc:1697] Filling consensus response to leader.
0129 03:00:02.974372 (+ 91us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0129 03:00:05.347397 (+2373025us) spinlock_profiling.cc:243] Waited 54 ms on lock 0x7b440002304c. stack: 00007f8fe044e373 00007f8fe044e270 00007f8fdfda0574 00007f8fe9a5c6b4 00007f8fe9a5c649 00007f8fe02c08b1 00007f8fe038f005 00007f8fe038fa22 00007f8fe30d957b 00007f8fe30d4e4f 00007f8fe30c8c0e 00007f8fe30c7f56 00007f8fe30c7b78 00007f8fe30d15c1 00007f8fe30d1579 00007f8fe30d1509
0129 03:00:05.353820 (+ 6423us) raft_consensus.cc:1685] finished
0129 03:00:05.353924 (+ 104us) raft_consensus.cc:1691] UpdateReplicas() finished
0129 03:00:05.354084 (+ 160us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":47341,"prepare.queue_time_us":259394,"prepare.run_cpu_time_us":43,"prepare.run_wall_time_us":44,"raft.queue_time_us":33542,"raft.run_cpu_time_us":489,"raft.run_wall_time_us":9346,"spinlock_wait_cycles":124127104,"thread_start_us":211277,"threads_started":2,"wal-append.queue_time_us":157653}
W20260129 03:00:05.357071 20556 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.211.2:34517, user_credentials={real_user=slave}} blocked reactor thread for 66819.3us
I20260129 03:00:05.357440 23260 raft_consensus.cc:3060] T 913fc7d751ed4b7388d889aac30a5f9b P 1c50de89a5a845668696dcb28c3badd6 [term 4 FOLLOWER]: Advancing to term 5
I20260129 03:00:05.360893 22878 leader_election.cc:290] T a0fcc58d6fb6473cafbd9d34d5195269 P 974b802141bd4f13beffef2aa03f88f9 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 197b880fa39d432a9b472845876bfaf0 (127.18.211.1:42027), 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3:37735)
I20260129 03:00:05.361184 20508 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "96609ac68bdf4993a12e1c23355ca013" candidate_uuid: "1c50de89a5a845668696dcb28c3badd6" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "974b802141bd4f13beffef2aa03f88f9" is_pre_election: true
I20260129 03:00:05.362112 20509 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "505580c93c694bbb8ea96fdf9b17ab46" candidate_uuid: "1c50de89a5a845668696dcb28c3badd6" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "974b802141bd4f13beffef2aa03f88f9" is_pre_election: true
I20260129 03:00:05.362340 23207 raft_consensus.cc:3055] T 0f20af7fbe5d4db89a6f17948d98912c P 974b802141bd4f13beffef2aa03f88f9 [term 6 LEADER]: Stepping down as leader of term 6
I20260129 03:00:05.362618 20508 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "75f45538dc704d9a8ec4ff8ad9f3ea5b" candidate_uuid: "1c50de89a5a845668696dcb28c3badd6" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "974b802141bd4f13beffef2aa03f88f9" is_pre_election: true
I20260129 03:00:05.362813 20509 raft_consensus.cc:2468] T 505580c93c694bbb8ea96fdf9b17ab46 P 974b802141bd4f13beffef2aa03f88f9 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1c50de89a5a845668696dcb28c3badd6 in term 6.
I20260129 03:00:05.364410 23250 consensus_queue.cc:1059] T a1a39dcd328e4268ae41b988c66b4a75 P 1c50de89a5a845668696dcb28c3badd6 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 }, Status: INVALID_TERM, Last received: 4.3, Next index: 4, Last known committed idx: 2, Time since last communication: 0.001s
I20260129 03:00:05.366715 22857 consensus_queue.cc:1048] T 505580c93c694bbb8ea96fdf9b17ab46 P 7a9daf5d42a84615a6383cfec9a10608 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.001s
W20260129 03:00:05.368914 20584 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:35752 (request call id 1632) took 3504 ms (client timeout 2730 ms). Trace:
I20260129 03:00:05.307029 20582 raft_consensus.cc:2955] T 96609ac68bdf4993a12e1c23355ca013 P 1c50de89a5a845668696dcb28c3badd6 [term 6 FOLLOWER]: Committing config change with OpId 2.4: config changed from index 2 to 4, 7a9daf5d42a84615a6383cfec9a10608 (127.18.211.4) changed from NON_VOTER to VOTER. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } attrs { replace: true } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } } }
W20260129 03:00:05.369323 20584 rpcz_store.cc:269] 0129 03:00:01.864194 (+ 0us) service_pool.cc:168] Inserting onto call queue
0129 03:00:02.046080 (+181886us) service_pool.cc:225] Handling call
0129 03:00:05.368509 (+3322429us) spinlock_profiling.cc:243] Waited 680 ms on lock 0x7b5400133654. stack: 00007f8fe044e373 00007f8fe044e270 00007f8fdfda0574 00007f8fe9a5c6b4 00007f8fe9a5c649 00007f8fe34975a1 00007f8fe3531f7a 00007f8fe84e3647 00007f8fe1a73834 00007f8fe1a737c2 00007f8fe1a736f1 00007f8fe1a7366c 00007f8fe1a72922 00007f8fe186d34c 00007f8fe186c786 00007f8fe186c12f
0129 03:00:05.368717 (+ 208us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1564286848}
I20260129 03:00:05.372345 23147 consensus_queue.cc:1059] T 78aa39096868415d9afc2ffcb248e198 P 1c50de89a5a845668696dcb28c3badd6 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false }, Status: INVALID_TERM, Last received: 5.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.001s
I20260129 03:00:05.310439 23163 raft_consensus.cc:515] T 18f30ebaa4174fd28c958e72d0cc7e24 P 7a9daf5d42a84615a6383cfec9a10608 [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } }
I20260129 03:00:05.301744 22925 consensus_queue.cc:1048] T 4408006eb5174de1896b037cf3f18457 P 197b880fa39d432a9b472845876bfaf0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 1, Time since last communication: 0.076s
I20260129 03:00:05.331200 23085 leader_election.cc:290] T a0fcc58d6fb6473cafbd9d34d5195269 P 197b880fa39d432a9b472845876bfaf0 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 974b802141bd4f13beffef2aa03f88f9 (127.18.211.2:34517), 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3:37735)
I20260129 03:00:05.337992 23232 raft_consensus.cc:2804] T 0340229d521b432580ae926f6062e3f7 P 974b802141bd4f13beffef2aa03f88f9 [term 7 FOLLOWER]: Leader election won for term 7
I20260129 03:00:05.380735 21100 leader_election.cc:304] T 4f5a77e94fef4b16a2ba7b067c8b144a P 7a9daf5d42a84615a6383cfec9a10608 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1c50de89a5a845668696dcb28c3badd6, 7a9daf5d42a84615a6383cfec9a10608; no voters:
W20260129 03:00:05.384522 20480 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.211.3:37735, user_credentials={real_user=slave}} blocked reactor thread for 46470.2us
I20260129 03:00:05.399951 20439 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "dc2f4cb752cf4d59b5f6bc689f5d8104" candidate_uuid: "7a9daf5d42a84615a6383cfec9a10608" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "197b880fa39d432a9b472845876bfaf0"
I20260129 03:00:05.402433 23214 raft_consensus.cc:3055] T 78aa39096868415d9afc2ffcb248e198 P 1c50de89a5a845668696dcb28c3badd6 [term 5 LEADER]: Stepping down as leader of term 5
I20260129 03:00:05.403191 20508 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3906b080a5b34a74b8b64128635e8997" candidate_uuid: "197b880fa39d432a9b472845876bfaf0" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "974b802141bd4f13beffef2aa03f88f9" is_pre_election: true
I20260129 03:00:05.406375 20514 raft_consensus.cc:1275] T 4e420f90884f4796997b5ea41e49e411 P 974b802141bd4f13beffef2aa03f88f9 [term 6 FOLLOWER]: Refusing update from remote peer 7a9daf5d42a84615a6383cfec9a10608: Log matching property violated. Preceding OpId in replica: term: 5 index: 7. Preceding OpId from leader: term: 6 index: 7. (term mismatch)
I20260129 03:00:05.362830 23207 raft_consensus.cc:740] T 0f20af7fbe5d4db89a6f17948d98912c P 974b802141bd4f13beffef2aa03f88f9 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 974b802141bd4f13beffef2aa03f88f9, State: Running, Role: LEADER
I20260129 03:00:05.408907 21136 raft_consensus.cc:1240] T 18f30ebaa4174fd28c958e72d0cc7e24 P 7a9daf5d42a84615a6383cfec9a10608 [term 7 FOLLOWER]: Rejecting Update request from peer 1c50de89a5a845668696dcb28c3badd6 for earlier term 6. Current term is 7. Ops: []
I20260129 03:00:05.410027 20435 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0678b8b468bd43cdbc3e485900844821" candidate_uuid: "7a9daf5d42a84615a6383cfec9a10608" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "197b880fa39d432a9b472845876bfaf0" is_pre_election: true
I20260129 03:00:05.388708 23216 raft_consensus.cc:493] T 3e2d15f8640b407195210c476e09df91 P 974b802141bd4f13beffef2aa03f88f9 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260129 03:00:05.406769 20514 pending_rounds.cc:85] T 4e420f90884f4796997b5ea41e49e411 P 974b802141bd4f13beffef2aa03f88f9: Aborting all ops after (but not including) 6
I20260129 03:00:05.414562 20433 raft_consensus.cc:2410] T 87d5fe48a44a408aaf03e894a67ecb2f P 197b880fa39d432a9b472845876bfaf0 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 7a9daf5d42a84615a6383cfec9a10608 for term 7 because replica has last-logged OpId of term: 5 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 4 index: 6.
I20260129 03:00:05.412920 21099 leader_election.cc:304] T fd8311b779af4e1fa1402b2a02a602f2 P 7a9daf5d42a84615a6383cfec9a10608 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7a9daf5d42a84615a6383cfec9a10608; no voters: 197b880fa39d432a9b472845876bfaf0, 1c50de89a5a845668696dcb28c3badd6
I20260129 03:00:05.419179 20432 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c934d31efd3f4e5195605ddc354d14e1" candidate_uuid: "7a9daf5d42a84615a6383cfec9a10608" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "197b880fa39d432a9b472845876bfaf0" is_pre_election: true
I20260129 03:00:05.416031 20514 pending_rounds.cc:107] T 4e420f90884f4796997b5ea41e49e411 P 974b802141bd4f13beffef2aa03f88f9: Aborting uncommitted NO_OP operation due to leader change: 5.7
I20260129 03:00:05.420105 20514 raft_consensus.cc:2889] T 4e420f90884f4796997b5ea41e49e411 P 974b802141bd4f13beffef2aa03f88f9 [term 6 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260129 03:00:05.355566 22855 consensus_queue.cc:260] T dc2f4cb752cf4d59b5f6bc689f5d8104 P 1c50de89a5a845668696dcb28c3badd6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } }
I20260129 03:00:05.422940 20479 leader_election.cc:304] T a4c1226b987b468fa84dfc6ba0e2dbe1 P 974b802141bd4f13beffef2aa03f88f9 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 974b802141bd4f13beffef2aa03f88f9; no voters: 197b880fa39d432a9b472845876bfaf0, 1c50de89a5a845668696dcb28c3badd6
I20260129 03:00:05.424587 21099 leader_election.cc:304] T 2ac659d732314e1288849f08fe5e074e P 7a9daf5d42a84615a6383cfec9a10608 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 197b880fa39d432a9b472845876bfaf0, 7a9daf5d42a84615a6383cfec9a10608; no voters:
I20260129 03:00:05.402918 23214 raft_consensus.cc:740] T 78aa39096868415d9afc2ffcb248e198 P 1c50de89a5a845668696dcb28c3badd6 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 1c50de89a5a845668696dcb28c3badd6, State: Running, Role: LEADER
I20260129 03:00:05.429754 20512 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57584 (request call id 1438) took 1777 ms. Trace:
W20260129 03:00:05.430285 20554 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.211.4:40439, user_credentials={real_user=slave}} blocked reactor thread for 101866us
I20260129 03:00:05.432631 20556 leader_election.cc:304] T 96609ac68bdf4993a12e1c23355ca013 P 1c50de89a5a845668696dcb28c3badd6 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 1c50de89a5a845668696dcb28c3badd6; no voters: 7a9daf5d42a84615a6383cfec9a10608, 974b802141bd4f13beffef2aa03f88f9
I20260129 03:00:05.433301 20435 raft_consensus.cc:2393] T 0678b8b468bd43cdbc3e485900844821 P 197b880fa39d432a9b472845876bfaf0 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 7a9daf5d42a84615a6383cfec9a10608 in current term 8: Already voted for candidate 197b880fa39d432a9b472845876bfaf0 in this term.
I20260129 03:00:05.436460 21099 leader_election.cc:304] T 78aa39096868415d9afc2ffcb248e198 P 7a9daf5d42a84615a6383cfec9a10608 [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7a9daf5d42a84615a6383cfec9a10608; no voters: 197b880fa39d432a9b472845876bfaf0, 1c50de89a5a845668696dcb28c3badd6
I20260129 03:00:05.420779 23232 raft_consensus.cc:697] T 0340229d521b432580ae926f6062e3f7 P 974b802141bd4f13beffef2aa03f88f9 [term 7 LEADER]: Becoming Leader. State: Replica: 974b802141bd4f13beffef2aa03f88f9, State: Running, Role: LEADER
I20260129 03:00:05.439198 20514 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:48544 (request call id 282) took 3059 ms. Trace:
W20260129 03:00:05.455256 20555 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260129 03:00:05.440197 23185 raft_consensus.cc:936] Couldn't advance consensus term.: Illegal state: Can't advance term to: 6 current term: 6 is higher.
I20260129 03:00:05.402951 23032 consensus_queue.cc:1048] T a8479d1fe0bd41e5946186e3a6ca8c7a P 974b802141bd4f13beffef2aa03f88f9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.028s
I20260129 03:00:05.447958 23215 raft_consensus.cc:2749] T a4c1226b987b468fa84dfc6ba0e2dbe1 P 974b802141bd4f13beffef2aa03f88f9 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260129 03:00:05.452345 23145 consensus_queue.cc:1059] T 0678b8b468bd43cdbc3e485900844821 P 974b802141bd4f13beffef2aa03f88f9 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 }, Status: INVALID_TERM, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.001s
I20260129 03:00:05.431290 23113 consensus_queue.cc:1059] T 0f20af7fbe5d4db89a6f17948d98912c P 974b802141bd4f13beffef2aa03f88f9 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 }, Status: INVALID_TERM, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.022s
I20260129 03:00:05.453562 22932 raft_consensus.cc:2804] T 2ac659d732314e1288849f08fe5e074e P 7a9daf5d42a84615a6383cfec9a10608 [term 5 FOLLOWER]: Leader election won for term 5
W20260129 03:00:05.453816 20405 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.211.3:37735, user_credentials={real_user=slave}} blocked reactor thread for 53090.3us
I20260129 03:00:05.418644 22996 raft_consensus.cc:2749] T a5a05fbcbbcf4916bbc01fd1a5b5e90d P 1c50de89a5a845668696dcb28c3badd6 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260129 03:00:05.456442 20556 leader_election.cc:304] T 505580c93c694bbb8ea96fdf9b17ab46 P 1c50de89a5a845668696dcb28c3badd6 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1c50de89a5a845668696dcb28c3badd6, 974b802141bd4f13beffef2aa03f88f9; no voters: 7a9daf5d42a84615a6383cfec9a10608
I20260129 03:00:05.429402 22857 raft_consensus.cc:2749] T fd8311b779af4e1fa1402b2a02a602f2 P 7a9daf5d42a84615a6383cfec9a10608 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260129 03:00:05.486666 20437 raft_consensus.cc:1275] T a8479d1fe0bd41e5946186e3a6ca8c7a P 197b880fa39d432a9b472845876bfaf0 [term 5 FOLLOWER]: Refusing update from remote peer 974b802141bd4f13beffef2aa03f88f9: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 5 index: 3. (term mismatch)
I20260129 03:00:05.445689 23216 raft_consensus.cc:515] T 3e2d15f8640b407195210c476e09df91 P 974b802141bd4f13beffef2aa03f88f9 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } }
I20260129 03:00:05.455521 20514 rpcz_store.cc:276] 0129 03:00:02.373021 (+ 0us) service_pool.cc:168] Inserting onto call queue
0129 03:00:02.452790 (+ 79769us) service_pool.cc:225] Handling call
0129 03:00:02.514572 (+ 61782us) raft_consensus.cc:1471] Updating replica for 0 ops
0129 03:00:05.253438 (+2738866us) spinlock_profiling.cc:243] Waited 320 ms on lock 0x7b54002d30a0. stack: 00007f8fe044e373 00007f8fe044e270 00007f8fdfda0574 00007f8fe9a5c6b4 00007f8fe9a5c649 00007f8fe34975a1 00007f8fe34a5df0 00007f8fe352fe40 00007f8fe3530eb1 00007f8fe352da0f 00007f8fe352d4be 00007f8fe84e1f49 00007f8fe1a6ed34 00007f8fe1a6ecc2 00007f8fe1a6ebf1 00007f8fe1a6eb6c
0129 03:00:05.425262 (+171824us) raft_consensus.cc:1697] Filling consensus response to leader.
0129 03:00:05.431981 (+ 6719us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":735668480}
I20260129 03:00:05.419461 23097 consensus_queue.cc:1059] T a609ac02fae84725ab743f95f51d33ce P 974b802141bd4f13beffef2aa03f88f9 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 }, Status: INVALID_TERM, Last received: 6.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.048s
W20260129 03:00:05.465994 20553 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.211.1:42027, user_credentials={real_user=slave}} blocked reactor thread for 143857us
I20260129 03:00:05.466240 20438 raft_consensus.cc:1275] T a609ac02fae84725ab743f95f51d33ce P 197b880fa39d432a9b472845876bfaf0 [term 8 FOLLOWER]: Refusing update from remote peer 7a9daf5d42a84615a6383cfec9a10608: Log matching property violated. Preceding OpId in replica: term: 6 index: 7. Preceding OpId from leader: term: 8 index: 8. (index mismatch)
I20260129 03:00:05.473836 20440 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "8055232cf0004d40a85f3bf983fe00a9" candidate_uuid: "974b802141bd4f13beffef2aa03f88f9" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "197b880fa39d432a9b472845876bfaf0" is_pre_election: true
I20260129 03:00:05.464344 21099 leader_election.cc:304] T 4e3805ffa70242e8ade66da260dddf66 P 7a9daf5d42a84615a6383cfec9a10608 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 197b880fa39d432a9b472845876bfaf0, 7a9daf5d42a84615a6383cfec9a10608; no voters: 1c50de89a5a845668696dcb28c3badd6
I20260129 03:00:05.461313 23037 raft_consensus.cc:2749] T 78aa39096868415d9afc2ffcb248e198 P 7a9daf5d42a84615a6383cfec9a10608 [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20260129 03:00:05.511348 20514 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "87d5fe48a44a408aaf03e894a67ecb2f" candidate_uuid: "197b880fa39d432a9b472845876bfaf0" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "974b802141bd4f13beffef2aa03f88f9" is_pre_election: true
I20260129 03:00:05.512599 20432 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "724a236524cb4e7ba3428880681d6cc2" candidate_uuid: "974b802141bd4f13beffef2aa03f88f9" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "197b880fa39d432a9b472845876bfaf0"
I20260129 03:00:05.463366 20508 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e33a417cf9c54cf5af73a4ee3aac4d3a" candidate_uuid: "1c50de89a5a845668696dcb28c3badd6" candidate_term: 7 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "974b802141bd4f13beffef2aa03f88f9" is_pre_election: true
I20260129 03:00:05.487177 20437 pending_rounds.cc:85] T a8479d1fe0bd41e5946186e3a6ca8c7a P 197b880fa39d432a9b472845876bfaf0: Aborting all ops after (but not including) 2
I20260129 03:00:05.514628 20437 pending_rounds.cc:107] T a8479d1fe0bd41e5946186e3a6ca8c7a P 197b880fa39d432a9b472845876bfaf0: Aborting uncommitted NO_OP operation due to leader change: 3.3
I20260129 03:00:05.515013 20437 raft_consensus.cc:2889] T a8479d1fe0bd41e5946186e3a6ca8c7a P 197b880fa39d432a9b472845876bfaf0 [term 5 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260129 03:00:05.480572 23086 raft_consensus.cc:493] T 4b1876f9a70b49edb3aa74105e66d3b3 P 197b880fa39d432a9b472845876bfaf0 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260129 03:00:05.520522 20439 raft_consensus.cc:1240] T fcddab6bca684adf8ea032289a00f90e P 197b880fa39d432a9b472845876bfaf0 [term 9 FOLLOWER]: Rejecting Update request from peer 974b802141bd4f13beffef2aa03f88f9 for earlier term 8. Current term is 9. Ops: []
W20260129 03:00:05.519831 20556 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.211.2:34517, user_credentials={real_user=slave}} blocked reactor thread for 63528us
I20260129 03:00:05.521574 20587 raft_consensus.cc:1275] T 42d27a28d89b409e8807ec0efd97af69 P 1c50de89a5a845668696dcb28c3badd6 [term 8 FOLLOWER]: Refusing update from remote peer 7a9daf5d42a84615a6383cfec9a10608: Log matching property violated. Preceding OpId in replica: term: 6 index: 8. Preceding OpId from leader: term: 8 index: 9. (index mismatch)
I20260129 03:00:05.526021 20556 leader_election.cc:304] T 75f45538dc704d9a8ec4ff8ad9f3ea5b P 1c50de89a5a845668696dcb28c3badd6 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1c50de89a5a845668696dcb28c3badd6; no voters: 7a9daf5d42a84615a6383cfec9a10608, 974b802141bd4f13beffef2aa03f88f9
W20260129 03:00:05.516834 23146 consensus_meta.cc:287] T febffe4a8b394368b69b1d164308fb9a P 974b802141bd4f13beffef2aa03f88f9: Time spent flushing consensus metadata: real 0.855s user 0.012s sys 0.004s
I20260129 03:00:05.474790 23280 raft_consensus.cc:493] T 724a236524cb4e7ba3428880681d6cc2 P 1c50de89a5a845668696dcb28c3badd6 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260129 03:00:05.522409 23118 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 2.522s user 0.014s sys 0.041s
I20260129 03:00:05.522801 22855 raft_consensus.cc:3060] T dc2f4cb752cf4d59b5f6bc689f5d8104 P 1c50de89a5a845668696dcb28c3badd6 [term 4 FOLLOWER]: Advancing to term 5
I20260129 03:00:05.541314 20438 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c891d2bff4ce4568861bbc992e91fbb0" candidate_uuid: "974b802141bd4f13beffef2aa03f88f9" candidate_term: 8 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "197b880fa39d432a9b472845876bfaf0" is_pre_election: true
I20260129 03:00:05.542605 23134 raft_consensus.cc:493] T 78aa39096868415d9afc2ffcb248e198 P 197b880fa39d432a9b472845876bfaf0 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 1c50de89a5a845668696dcb28c3badd6)
I20260129 03:00:05.543159 23134 raft_consensus.cc:515] T 78aa39096868415d9afc2ffcb248e198 P 197b880fa39d432a9b472845876bfaf0 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } }
W20260129 03:00:05.543782 23279 log.cc:927] Time spent T fd8311b779af4e1fa1402b2a02a602f2 P 7a9daf5d42a84615a6383cfec9a10608: Append to log took a long time: real 0.076s user 0.000s sys 0.000s
I20260129 03:00:05.544519 23026 raft_consensus.cc:2749] T cedce331e1584ac8b636c9b8a72b2ba0 P 1c50de89a5a845668696dcb28c3badd6 [term 7 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260129 03:00:05.545354 23026 raft_consensus.cc:2804] T 505580c93c694bbb8ea96fdf9b17ab46 P 1c50de89a5a845668696dcb28c3badd6 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260129 03:00:05.545785 23026 raft_consensus.cc:493] T 505580c93c694bbb8ea96fdf9b17ab46 P 1c50de89a5a845668696dcb28c3badd6 [term 6 FOLLOWER]: Starting leader election (detected failure of leader 7a9daf5d42a84615a6383cfec9a10608)
I20260129 03:00:05.546160 23026 raft_consensus.cc:3060] T 505580c93c694bbb8ea96fdf9b17ab46 P 1c50de89a5a845668696dcb28c3badd6 [term 6 FOLLOWER]: Advancing to term 7
I20260129 03:00:05.548784 22975 raft_consensus.cc:3055] T 0678b8b468bd43cdbc3e485900844821 P 974b802141bd4f13beffef2aa03f88f9 [term 7 LEADER]: Stepping down as leader of term 7
I20260129 03:00:05.549221 22975 raft_consensus.cc:740] T 0678b8b468bd43cdbc3e485900844821 P 974b802141bd4f13beffef2aa03f88f9 [term 7 LEADER]: Becoming Follower/Learner. State: Replica: 974b802141bd4f13beffef2aa03f88f9, State: Running, Role: LEADER
I20260129 03:00:05.550163 22975 consensus_queue.cc:260] T 0678b8b468bd43cdbc3e485900844821 P 974b802141bd4f13beffef2aa03f88f9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 7.8, Last appended by leader: 8, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } }
I20260129 03:00:05.551505 22975 raft_consensus.cc:3060] T 0678b8b468bd43cdbc3e485900844821 P 974b802141bd4f13beffef2aa03f88f9 [term 7 FOLLOWER]: Advancing to term 8
I20260129 03:00:05.552577 20432 raft_consensus.cc:2393] T 724a236524cb4e7ba3428880681d6cc2 P 197b880fa39d432a9b472845876bfaf0 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 974b802141bd4f13beffef2aa03f88f9 in current term 5: Already voted for candidate 197b880fa39d432a9b472845876bfaf0 in this term.
I20260129 03:00:05.553051 20440 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7d9494f5584d4d5db8b9adf1c8fc888b" candidate_uuid: "974b802141bd4f13beffef2aa03f88f9" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "197b880fa39d432a9b472845876bfaf0" is_pre_election: true
I20260129 03:00:05.559367 20437 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4b1876f9a70b49edb3aa74105e66d3b3" candidate_uuid: "974b802141bd4f13beffef2aa03f88f9" candidate_term: 6 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "197b880fa39d432a9b472845876bfaf0" is_pre_election: true
I20260129 03:00:05.518088 22932 raft_consensus.cc:697] T 2ac659d732314e1288849f08fe5e074e P 7a9daf5d42a84615a6383cfec9a10608 [term 5 LEADER]: Becoming Leader. State: Replica: 7a9daf5d42a84615a6383cfec9a10608, State: Running, Role: LEADER
I20260129 03:00:05.563094 20508 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a8479d1fe0bd41e5946186e3a6ca8c7a" candidate_uuid: "197b880fa39d432a9b472845876bfaf0" candidate_term: 6 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "974b802141bd4f13beffef2aa03f88f9" is_pre_election: true
W20260129 03:00:05.545526 23278 log.cc:927] Time spent T 00e79d8f25a34676b1d866eb2068a2b2 P 197b880fa39d432a9b472845876bfaf0: Append to log took a long time: real 0.065s user 0.000s sys 0.000s
I20260129 03:00:05.565606 20439 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0340229d521b432580ae926f6062e3f7" candidate_uuid: "974b802141bd4f13beffef2aa03f88f9" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "197b880fa39d432a9b472845876bfaf0"
I20260129 03:00:05.568459 20439 raft_consensus.cc:3060] T 0340229d521b432580ae926f6062e3f7 P 197b880fa39d432a9b472845876bfaf0 [term 5 FOLLOWER]: Advancing to term 7
W20260129 03:00:05.534914 23118 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 2.632s user 0.014s sys 0.043s
I20260129 03:00:05.534147 23141 raft_consensus.cc:2804] T 4f5a77e94fef4b16a2ba7b067c8b144a P 7a9daf5d42a84615a6383cfec9a10608 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260129 03:00:05.567255 20514 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c2633a6fc34c44579de348d208b0c01e" candidate_uuid: "1c50de89a5a845668696dcb28c3badd6" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "974b802141bd4f13beffef2aa03f88f9"
W20260129 03:00:05.569001 21101 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260129 03:00:05.573513 22932 consensus_queue.cc:237] T 2ac659d732314e1288849f08fe5e074e P 7a9daf5d42a84615a6383cfec9a10608 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 3.5, Last appended by leader: 5, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } }
W20260129 03:00:05.574626 20481 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260129 03:00:05.575932 23118 raft_consensus.cc:697] T 75be4bd6bd45474795fd472eff2953ac P 197b880fa39d432a9b472845876bfaf0 [term 10 LEADER]: Becoming Leader. State: Replica: 197b880fa39d432a9b472845876bfaf0, State: Running, Role: LEADER
W20260129 03:00:05.576941 20433 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:48356 (request call id 364) took 3469 ms (client timeout 3654 ms). Trace:
I20260129 03:00:05.577137 23118 consensus_queue.cc:237] T 75be4bd6bd45474795fd472eff2953ac P 197b880fa39d432a9b472845876bfaf0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 9.6, Last appended by leader: 6, Current term: 10, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
W20260129 03:00:05.577545 20433 rpcz_store.cc:269] 0129 03:00:02.285325 (+ 0us) service_pool.cc:168] Inserting onto call queue
0129 03:00:02.291063 (+ 5738us) service_pool.cc:225] Handling call
0129 03:00:05.576668 (+3285605us) spinlock_profiling.cc:243] Waited 2.98 s on lock 0x7b54000a64d4. stack: 00007f8fe044e373 00007f8fe044e270 00007f8fdfda0574 00007f8fe9a5c6b4 00007f8fe9a5c649 00007f8fe34975a1 00007f8fe3531f7a 00007f8fe84e3647 00007f8fe1a73834 00007f8fe1a737c2 00007f8fe1a736f1 00007f8fe1a7366c 00007f8fe1a72922 00007f8fe186d34c 00007f8fe186c786 00007f8fe186c12f
0129 03:00:05.576857 (+ 189us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":6844211968}
I20260129 03:00:05.571480 23141 raft_consensus.cc:493] T 4f5a77e94fef4b16a2ba7b067c8b144a P 7a9daf5d42a84615a6383cfec9a10608 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260129 03:00:05.580044 23141 raft_consensus.cc:3060] T 4f5a77e94fef4b16a2ba7b067c8b144a P 7a9daf5d42a84615a6383cfec9a10608 [term 4 FOLLOWER]: Advancing to term 5
I20260129 03:00:05.560348 22645 raft_consensus.cc:515] T 17d6431cc77a491e8464d68db4081a51 P 197b880fa39d432a9b472845876bfaf0 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 03:00:05.593245 20514 raft_consensus.cc:3060] T c2633a6fc34c44579de348d208b0c01e P 974b802141bd4f13beffef2aa03f88f9 [term 3 FOLLOWER]: Advancing to term 4
I20260129 03:00:05.593827 23283 consensus_queue.cc:1059] T 4e3805ffa70242e8ade66da260dddf66 P 1c50de89a5a845668696dcb28c3badd6 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false }, Status: INVALID_TERM, Last received: 5.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.064s
I20260129 03:00:05.596323 23216 leader_election.cc:290] T 3e2d15f8640b407195210c476e09df91 P 974b802141bd4f13beffef2aa03f88f9 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3:37735), 7a9daf5d42a84615a6383cfec9a10608 (127.18.211.4:40439)
I20260129 03:00:05.577011 20510 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57600 (request call id 1873) took 2971 ms. Trace:
I20260129 03:00:05.562379 22996 raft_consensus.cc:2749] T 75f45538dc704d9a8ec4ff8ad9f3ea5b P 1c50de89a5a845668696dcb28c3badd6 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260129 03:00:05.614681 20433 raft_consensus.cc:1240] T 0f20af7fbe5d4db89a6f17948d98912c P 197b880fa39d432a9b472845876bfaf0 [term 7 FOLLOWER]: Rejecting Update request from peer 974b802141bd4f13beffef2aa03f88f9 for earlier term 6. Current term is 7. Ops: [6.4-6.4]
I20260129 03:00:05.616001 20433 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c934d31efd3f4e5195605ddc354d14e1" candidate_uuid: "974b802141bd4f13beffef2aa03f88f9" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "197b880fa39d432a9b472845876bfaf0" is_pre_election: true
I20260129 03:00:05.616384 22996 raft_consensus.cc:3055] T 4e3805ffa70242e8ade66da260dddf66 P 1c50de89a5a845668696dcb28c3badd6 [term 5 LEADER]: Stepping down as leader of term 5
I20260129 03:00:05.616947 22996 raft_consensus.cc:740] T 4e3805ffa70242e8ade66da260dddf66 P 1c50de89a5a845668696dcb28c3badd6 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 1c50de89a5a845668696dcb28c3badd6, State: Running, Role: LEADER
I20260129 03:00:05.618341 23163 leader_election.cc:290] T 18f30ebaa4174fd28c958e72d0cc7e24 P 7a9daf5d42a84615a6383cfec9a10608 [CANDIDATE]: Term 7 election: Requested vote from peers 197b880fa39d432a9b472845876bfaf0 (127.18.211.1:42027), 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3:37735)
I20260129 03:00:05.618098 22996 consensus_queue.cc:260] T 4e3805ffa70242e8ade66da260dddf66 P 1c50de89a5a845668696dcb28c3badd6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 5.7, Last appended by leader: 7, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } }
I20260129 03:00:05.552501 23037 consensus_queue.cc:1048] T 42d27a28d89b409e8807ec0efd97af69 P 7a9daf5d42a84615a6383cfec9a10608 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.051s
I20260129 03:00:05.598518 20510 rpcz_store.cc:276] 0129 03:00:02.605449 (+ 0us) service_pool.cc:168] Inserting onto call queue
0129 03:00:02.612563 (+ 7114us) service_pool.cc:225] Handling call
0129 03:00:02.620245 (+ 7682us) raft_consensus.cc:1471] Updating replica for 0 ops
0129 03:00:02.777113 (+156868us) raft_consensus.cc:1534] Early marking committed up to index 0
0129 03:00:02.777151 (+ 38us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0129 03:00:02.777203 (+ 52us) raft_consensus.cc:1645] Marking committed up to 3
0129 03:00:02.777248 (+ 45us) raft_consensus.cc:1697] Filling consensus response to leader.
0129 03:00:05.576559 (+2799311us) spinlock_profiling.cc:243] Waited 157 ms on lock 0x7b540012a054. stack: 00007f8fe044e373 00007f8fe044e270 00007f8fdfda0574 00007f8fe9a5c6b4 00007f8fe9a5c649 00007f8fe34975a1 00007f8fe352ea81 00007f8fe352d4be 00007f8fe84e1f49 00007f8fe1a6ed34 00007f8fe1a6ecc2 00007f8fe1a6ebf1 00007f8fe1a6eb6c 00007f8fe1a6de22 00007f8fe186d34c 00007f8fe186c786
0129 03:00:05.576628 (+ 69us) raft_consensus.cc:1691] UpdateReplicas() finished
0129 03:00:05.576799 (+ 171us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":360202112}
I20260129 03:00:05.624356 20510 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "febffe4a8b394368b69b1d164308fb9a" candidate_uuid: "197b880fa39d432a9b472845876bfaf0" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "974b802141bd4f13beffef2aa03f88f9" is_pre_election: true
I20260129 03:00:05.625126 20583 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "18f30ebaa4174fd28c958e72d0cc7e24" candidate_uuid: "7a9daf5d42a84615a6383cfec9a10608" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "1c50de89a5a845668696dcb28c3badd6"
I20260129 03:00:05.561509 23280 raft_consensus.cc:515] T 724a236524cb4e7ba3428880681d6cc2 P 1c50de89a5a845668696dcb28c3badd6 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 03:00:05.602528 23232 consensus_queue.cc:237] T 0340229d521b432580ae926f6062e3f7 P 974b802141bd4f13beffef2aa03f88f9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 5.4, Last appended by leader: 4, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 03:00:05.567075 23086 raft_consensus.cc:515] T 4b1876f9a70b49edb3aa74105e66d3b3 P 197b880fa39d432a9b472845876bfaf0 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 03:00:05.656556 20437 raft_consensus.cc:2468] T 4b1876f9a70b49edb3aa74105e66d3b3 P 197b880fa39d432a9b472845876bfaf0 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 974b802141bd4f13beffef2aa03f88f9 in term 5.
I20260129 03:00:05.540194 23146 raft_consensus.cc:515] T febffe4a8b394368b69b1d164308fb9a P 974b802141bd4f13beffef2aa03f88f9 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } }
I20260129 03:00:05.581650 23035 raft_consensus.cc:2804] T 4e3805ffa70242e8ade66da260dddf66 P 7a9daf5d42a84615a6383cfec9a10608 [term 6 FOLLOWER]: Leader election won for term 6
I20260129 03:00:05.658762 23146 leader_election.cc:290] T febffe4a8b394368b69b1d164308fb9a P 974b802141bd4f13beffef2aa03f88f9 [CANDIDATE]: Term 4 election: Requested vote from peers 197b880fa39d432a9b472845876bfaf0 (127.18.211.1:42027), 7a9daf5d42a84615a6383cfec9a10608 (127.18.211.4:40439)
I20260129 03:00:05.659742 20440 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "913fc7d751ed4b7388d889aac30a5f9b" candidate_uuid: "974b802141bd4f13beffef2aa03f88f9" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "197b880fa39d432a9b472845876bfaf0"
I20260129 03:00:05.660527 20440 raft_consensus.cc:3060] T 913fc7d751ed4b7388d889aac30a5f9b P 197b880fa39d432a9b472845876bfaf0 [term 4 FOLLOWER]: Advancing to term 5
I20260129 03:00:05.582091 23214 consensus_queue.cc:260] T 78aa39096868415d9afc2ffcb248e198 P 1c50de89a5a845668696dcb28c3badd6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 5.6, Last appended by leader: 6, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } }
I20260129 03:00:05.665019 23214 raft_consensus.cc:3060] T 78aa39096868415d9afc2ffcb248e198 P 1c50de89a5a845668696dcb28c3badd6 [term 5 FOLLOWER]: Advancing to term 6
I20260129 03:00:05.430763 20512 rpcz_store.cc:276] 0129 03:00:03.645746 (+ 0us) service_pool.cc:168] Inserting onto call queue
0129 03:00:05.134743 (+1488997us) service_pool.cc:225] Handling call
0129 03:00:05.143290 (+ 8547us) raft_consensus.cc:1471] Updating replica for 0 ops
0129 03:00:05.143617 (+ 327us) raft_consensus.cc:1534] Early marking committed up to index 4
0129 03:00:05.324700 (+181083us) log.cc:844] Serialized 14 byte log entry
0129 03:00:05.422294 (+ 97594us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0129 03:00:05.422343 (+ 49us) raft_consensus.cc:1645] Marking committed up to 4
0129 03:00:05.422505 (+ 162us) raft_consensus.cc:1697] Filling consensus response to leader.
0129 03:00:05.422594 (+ 89us) raft_consensus.cc:1691] UpdateReplicas() finished
0129 03:00:05.422855 (+ 261us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":4520,"prepare.queue_time_us":238944,"prepare.run_cpu_time_us":36,"prepare.run_wall_time_us":38,"raft.queue_time_us":8305,"raft.run_cpu_time_us":116,"raft.run_wall_time_us":3169,"thread_start_us":203464,"threads_started":2}
I20260129 03:00:05.666553 20512 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "17d6431cc77a491e8464d68db4081a51" candidate_uuid: "197b880fa39d432a9b472845876bfaf0" candidate_term: 5 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "974b802141bd4f13beffef2aa03f88f9"
I20260129 03:00:05.675213 20507 raft_consensus.cc:2410] T febffe4a8b394368b69b1d164308fb9a P 974b802141bd4f13beffef2aa03f88f9 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 7a9daf5d42a84615a6383cfec9a10608 for term 5 because replica has last-logged OpId of term: 3 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 4.
W20260129 03:00:05.675060 20555 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260129 03:00:05.722150 21100 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.211.3:37735, user_credentials={real_user=slave}} blocked reactor thread for 92876us
I20260129 03:00:05.738433 20512 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4f8f3a38f6e24ba4857875181f447442" candidate_uuid: "1c50de89a5a845668696dcb28c3badd6" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "974b802141bd4f13beffef2aa03f88f9" is_pre_election: true
I20260129 03:00:05.746227 23059 raft_consensus.cc:3060] T a644c02b16af47679824ccd060ac10fd P 7a9daf5d42a84615a6383cfec9a10608 [term 4 FOLLOWER]: Advancing to term 5
I20260129 03:00:05.747433 23104 raft_consensus.cc:3055] T a1a39dcd328e4268ae41b988c66b4a75 P 1c50de89a5a845668696dcb28c3badd6 [term 4 LEADER]: Stepping down as leader of term 4
W20260129 03:00:05.751394 21101 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260129 03:00:05.752884 20432 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a1a39dcd328e4268ae41b988c66b4a75" candidate_uuid: "974b802141bd4f13beffef2aa03f88f9" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "197b880fa39d432a9b472845876bfaf0"
I20260129 03:00:05.756340 21128 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c934d31efd3f4e5195605ddc354d14e1" candidate_uuid: "974b802141bd4f13beffef2aa03f88f9" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "7a9daf5d42a84615a6383cfec9a10608" is_pre_election: true
W20260129 03:00:05.756749 20407 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.211.2:34517, user_credentials={real_user=slave}} blocked reactor thread for 43614us
I20260129 03:00:05.757119 21128 raft_consensus.cc:2468] T c934d31efd3f4e5195605ddc354d14e1 P 7a9daf5d42a84615a6383cfec9a10608 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 974b802141bd4f13beffef2aa03f88f9 in term 5.
W20260129 03:00:05.757779 20407 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260129 03:00:05.758352 20407 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260129 03:00:05.758670 20407 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260129 03:00:05.780298 20510 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a0fcc58d6fb6473cafbd9d34d5195269" candidate_uuid: "197b880fa39d432a9b472845876bfaf0" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "974b802141bd4f13beffef2aa03f88f9" is_pre_election: true
I20260129 03:00:05.781451 20510 raft_consensus.cc:2468] T a0fcc58d6fb6473cafbd9d34d5195269 P 974b802141bd4f13beffef2aa03f88f9 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 197b880fa39d432a9b472845876bfaf0 in term 4.
W20260129 03:00:05.829190 20554 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.211.4:40439, user_credentials={real_user=slave}} blocked reactor thread for 388090us
W20260129 03:00:05.873027 20555 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260129 03:00:05.876767 23281 raft_consensus.cc:493] T 0678b8b468bd43cdbc3e485900844821 P 197b880fa39d432a9b472845876bfaf0 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260129 03:00:05.739521 23035 raft_consensus.cc:697] T 4e3805ffa70242e8ade66da260dddf66 P 7a9daf5d42a84615a6383cfec9a10608 [term 6 LEADER]: Becoming Leader. State: Replica: 7a9daf5d42a84615a6383cfec9a10608, State: Running, Role: LEADER
I20260129 03:00:05.613900 20438 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e7d83d4ccf684e76acb29a0dc844e1b8" candidate_uuid: "974b802141bd4f13beffef2aa03f88f9" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "197b880fa39d432a9b472845876bfaf0" is_pre_election: true
I20260129 03:00:05.752133 23104 raft_consensus.cc:740] T a1a39dcd328e4268ae41b988c66b4a75 P 1c50de89a5a845668696dcb28c3badd6 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 1c50de89a5a845668696dcb28c3badd6, State: Running, Role: LEADER
W20260129 03:00:05.894696 21102 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.211.2:34517, user_credentials={real_user=slave}} blocked reactor thread for 166549us
W20260129 03:00:05.911340 20436 consensus_meta.cc:287] T 7d9494f5584d4d5db8b9adf1c8fc888b P 197b880fa39d432a9b472845876bfaf0: Time spent flushing consensus metadata: real 0.647s user 0.035s sys 0.004s
I20260129 03:00:05.893985 20407 leader_election.cc:304] T a0fcc58d6fb6473cafbd9d34d5195269 P 197b880fa39d432a9b472845876bfaf0 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 197b880fa39d432a9b472845876bfaf0, 974b802141bd4f13beffef2aa03f88f9; no voters:
W20260129 03:00:05.917042 20555 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260129 03:00:05.889930 20432 raft_consensus.cc:3060] T a1a39dcd328e4268ae41b988c66b4a75 P 197b880fa39d432a9b472845876bfaf0 [term 4 FOLLOWER]: Advancing to term 5
W20260129 03:00:05.889822 20405 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.211.3:37735, user_credentials={real_user=slave}} blocked reactor thread for 372564us
W20260129 03:00:05.890681 21099 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.211.1:42027, user_credentials={real_user=slave}} blocked reactor thread for 433427us
I20260129 03:00:05.921316 20438 raft_consensus.cc:2468] T e7d83d4ccf684e76acb29a0dc844e1b8 P 197b880fa39d432a9b472845876bfaf0 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 974b802141bd4f13beffef2aa03f88f9 in term 6.
I20260129 03:00:05.933141 20512 raft_consensus.cc:1240] T 8055232cf0004d40a85f3bf983fe00a9 P 974b802141bd4f13beffef2aa03f88f9 [term 5 FOLLOWER]: Rejecting Update request from peer 197b880fa39d432a9b472845876bfaf0 for earlier term 4. Current term is 5. Ops: []
I20260129 03:00:05.911340 22857 raft_consensus.cc:493] T 1299723a4476420ab02fd8bfd7aa0645 P 7a9daf5d42a84615a6383cfec9a10608 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260129 03:00:05.948623 23080 consensus_queue.cc:1048] T cb0b3f259ebc4310927338a8740e87b8 P 7a9daf5d42a84615a6383cfec9a10608 [LEADER]: Connected to new peer: Peer: permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.012s
I20260129 03:00:05.948969 22857 raft_consensus.cc:515] T 1299723a4476420ab02fd8bfd7aa0645 P 7a9daf5d42a84615a6383cfec9a10608 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } }
W20260129 03:00:05.949928 23057 consensus_meta.cc:287] T cb4727386a3e48aca5d71cc6c611c197 P 197b880fa39d432a9b472845876bfaf0: Time spent flushing consensus metadata: real 0.804s user 0.014s sys 0.029s
I20260129 03:00:05.950358 23057 raft_consensus.cc:515] T cb4727386a3e48aca5d71cc6c611c197 P 197b880fa39d432a9b472845876bfaf0 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 03:00:05.950743 23134 leader_election.cc:290] T 78aa39096868415d9afc2ffcb248e198 P 197b880fa39d432a9b472845876bfaf0 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3:37735), 7a9daf5d42a84615a6383cfec9a10608 (127.18.211.4:40439)
I20260129 03:00:05.884218 23281 raft_consensus.cc:515] T 0678b8b468bd43cdbc3e485900844821 P 197b880fa39d432a9b472845876bfaf0 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } }
I20260129 03:00:05.951767 23280 leader_election.cc:290] T 724a236524cb4e7ba3428880681d6cc2 P 1c50de89a5a845668696dcb28c3badd6 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 197b880fa39d432a9b472845876bfaf0 (127.18.211.1:42027), 974b802141bd4f13beffef2aa03f88f9 (127.18.211.2:34517)
I20260129 03:00:05.953143 23035 consensus_queue.cc:237] T 4e3805ffa70242e8ade66da260dddf66 P 7a9daf5d42a84615a6383cfec9a10608 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 5.7, Last appended by leader: 7, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } }
I20260129 03:00:05.960477 23280 consensus_queue.cc:1059] T 3e2d15f8640b407195210c476e09df91 P 1c50de89a5a845668696dcb28c3badd6 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false }, Status: INVALID_TERM, Last received: 3.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
I20260129 03:00:05.961383 23280 consensus_queue.cc:1243] T 3e2d15f8640b407195210c476e09df91 P 1c50de89a5a845668696dcb28c3badd6 [LEADER]: Peer 7a9daf5d42a84615a6383cfec9a10608 log is divergent from this leader: its last log entry 4.6 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 5
I20260129 03:00:05.892702 23290 raft_consensus.cc:493] T 3906b080a5b34a74b8b64128635e8997 P 1c50de89a5a845668696dcb28c3badd6 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 974b802141bd4f13beffef2aa03f88f9)
I20260129 03:00:05.952033 22645 leader_election.cc:290] T 17d6431cc77a491e8464d68db4081a51 P 197b880fa39d432a9b472845876bfaf0 [CANDIDATE]: Term 5 election: Requested vote from peers 974b802141bd4f13beffef2aa03f88f9 (127.18.211.2:34517), 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3:37735)
I20260129 03:00:05.957690 23178 raft_consensus.cc:493] T 7d9494f5584d4d5db8b9adf1c8fc888b P 197b880fa39d432a9b472845876bfaf0 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 1c50de89a5a845668696dcb28c3badd6)
I20260129 03:00:05.963202 23178 raft_consensus.cc:515] T 7d9494f5584d4d5db8b9adf1c8fc888b P 197b880fa39d432a9b472845876bfaf0 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 03:00:05.969486 20511 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57600 (request call id 1886) took 2477 ms. Trace:
I20260129 03:00:05.969923 20511 rpcz_store.cc:276] 0129 03:00:03.491825 (+ 0us) service_pool.cc:168] Inserting onto call queue
0129 03:00:04.982818 (+1490993us) service_pool.cc:225] Handling call
0129 03:00:05.072175 (+ 89357us) raft_consensus.cc:1471] Updating replica for 0 ops
0129 03:00:05.135834 (+ 63659us) raft_consensus.cc:1534] Early marking committed up to index 3
0129 03:00:05.327166 (+191332us) log.cc:844] Serialized 14 byte log entry
0129 03:00:05.956362 (+629196us) log.cc:844] Serialized 14 byte log entry
0129 03:00:05.969153 (+ 12791us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0129 03:00:05.969209 (+ 56us) raft_consensus.cc:1645] Marking committed up to 3
0129 03:00:05.969250 (+ 41us) raft_consensus.cc:1697] Filling consensus response to leader.
0129 03:00:05.969316 (+ 66us) raft_consensus.cc:1691] UpdateReplicas() finished
0129 03:00:05.969432 (+ 116us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":240377,"prepare.queue_time_us":1017426,"prepare.run_cpu_time_us":63,"prepare.run_wall_time_us":58,"raft.queue_time_us":31526,"raft.run_cpu_time_us":116,"raft.run_wall_time_us":120,"thread_start_us":458924,"threads_started":2}
I20260129 03:00:05.972115 20590 raft_consensus.cc:3055] T 75b0b6e82ddc4f92b1fe1d94f9a0b73b P 1c50de89a5a845668696dcb28c3badd6 [term 3 LEADER]: Stepping down as leader of term 3
I20260129 03:00:05.972527 20590 raft_consensus.cc:740] T 75b0b6e82ddc4f92b1fe1d94f9a0b73b P 1c50de89a5a845668696dcb28c3badd6 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 1c50de89a5a845668696dcb28c3badd6, State: Running, Role: LEADER
I20260129 03:00:05.962369 23290 raft_consensus.cc:515] T 3906b080a5b34a74b8b64128635e8997 P 1c50de89a5a845668696dcb28c3badd6 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
W20260129 03:00:05.964349 20481 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260129 03:00:05.957605 20436 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55676 (request call id 1399) took 5979 ms. Trace:
I20260129 03:00:05.974119 23086 leader_election.cc:290] T 4b1876f9a70b49edb3aa74105e66d3b3 P 197b880fa39d432a9b472845876bfaf0 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 974b802141bd4f13beffef2aa03f88f9 (127.18.211.2:34517), 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3:37735)
I20260129 03:00:05.974437 20436 rpcz_store.cc:276] 0129 02:59:59.978587 (+ 0us) service_pool.cc:168] Inserting onto call queue
0129 03:00:00.026814 (+ 48227us) service_pool.cc:225] Handling call
0129 03:00:00.028631 (+ 1817us) raft_consensus.cc:1471] Updating replica for 0 ops
0129 03:00:03.542255 (+3513624us) spinlock_profiling.cc:243] Waited 1.88 s on lock 0x7b180027a980. stack: 00007f8fe044e373 00007f8fe044e270 00007f8fdfda0574 00007f8fe9a5c6b4 00007f8fe9a5c649 00007f8fe34975a1 00007f8fe3569496 00007f8fe34a35be 00007f8fe3524c1e 00007f8fe352645e 00007f8fe35307c9 00007f8fe3531096 00007f8fe352da0f 00007f8fe352d4be 00007f8fe84e1f49 00007f8fe1a6ed34
0129 03:00:05.263278 (+1721023us) spinlock_profiling.cc:243] Waited 2.07 ms on lock 0x7b58009b24d0. stack: 00007f8fe044e373 00007f8fe044e270 00007f8fdfda0574 00007f8fe9a5c6b4 00007f8fe9a5c649 00007f8fe34975a1 00007f8fe348a0da 00007f8fe350c177 00007f8fe3524c67 00007f8fe352645e 00007f8fe35307c9 00007f8fe3531096 00007f8fe352da0f 00007f8fe352d4be 00007f8fe84e1f49 00007f8fe1a6ed34
0129 03:00:05.956992 (+693714us) raft_consensus.cc:1534] Early marking committed up to index 2
0129 03:00:05.957028 (+ 36us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0129 03:00:05.957167 (+ 139us) raft_consensus.cc:1645] Marking committed up to 2
0129 03:00:05.957208 (+ 41us) raft_consensus.cc:1697] Filling consensus response to leader.
0129 03:00:05.957314 (+ 106us) raft_consensus.cc:1691] UpdateReplicas() finished
0129 03:00:05.957527 (+ 213us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":19318,"raft.queue_time_us":21134,"raft.run_cpu_time_us":90,"raft.run_wall_time_us":90,"spinlock_wait_cycles":4326642944}
I20260129 03:00:05.976140 20436 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "8055232cf0004d40a85f3bf983fe00a9" candidate_uuid: "974b802141bd4f13beffef2aa03f88f9" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "197b880fa39d432a9b472845876bfaf0"
I20260129 03:00:05.977289 20436 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4408006eb5174de1896b037cf3f18457" candidate_uuid: "974b802141bd4f13beffef2aa03f88f9" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "197b880fa39d432a9b472845876bfaf0" is_pre_election: true
I20260129 03:00:05.977912 20438 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4b1876f9a70b49edb3aa74105e66d3b3" candidate_uuid: "974b802141bd4f13beffef2aa03f88f9" candidate_term: 6 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "197b880fa39d432a9b472845876bfaf0"
I20260129 03:00:05.965492 23185 raft_consensus.cc:493] T 75be4bd6bd45474795fd472eff2953ac P 974b802141bd4f13beffef2aa03f88f9 [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260129 03:00:05.978920 23185 raft_consensus.cc:515] T 75be4bd6bd45474795fd472eff2953ac P 974b802141bd4f13beffef2aa03f88f9 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 03:00:05.966357 23145 raft_consensus.cc:493] T b189f08e16a74a11980cc55d99359a0a P 974b802141bd4f13beffef2aa03f88f9 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260129 03:00:05.966805 23216 raft_consensus.cc:493] T 4f5a77e94fef4b16a2ba7b067c8b144a P 974b802141bd4f13beffef2aa03f88f9 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260129 03:00:05.937463 23085 raft_consensus.cc:2804] T a0fcc58d6fb6473cafbd9d34d5195269 P 197b880fa39d432a9b472845876bfaf0 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260129 03:00:06.001344 23085 raft_consensus.cc:493] T a0fcc58d6fb6473cafbd9d34d5195269 P 197b880fa39d432a9b472845876bfaf0 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260129 03:00:06.002116 23085 raft_consensus.cc:3060] T a0fcc58d6fb6473cafbd9d34d5195269 P 197b880fa39d432a9b472845876bfaf0 [term 4 FOLLOWER]: Advancing to term 5
I20260129 03:00:05.956341 20581 raft_consensus.cc:3055] T 3e2d15f8640b407195210c476e09df91 P 1c50de89a5a845668696dcb28c3badd6 [term 3 LEADER]: Stepping down as leader of term 3
I20260129 03:00:05.972447 20515 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:48544 (request call id 289) took 1444 ms. Trace:
W20260129 03:00:05.994801 21099 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.211.1:42027, user_credentials={real_user=slave}} blocked reactor thread for 64369us
I20260129 03:00:05.983047 20511 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4b1876f9a70b49edb3aa74105e66d3b3" candidate_uuid: "197b880fa39d432a9b472845876bfaf0" candidate_term: 6 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "974b802141bd4f13beffef2aa03f88f9" is_pre_election: true
I20260129 03:00:06.006527 21099 leader_election.cc:304] T dc2f4cb752cf4d59b5f6bc689f5d8104 P 7a9daf5d42a84615a6383cfec9a10608 [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7a9daf5d42a84615a6383cfec9a10608; no voters: 197b880fa39d432a9b472845876bfaf0, 1c50de89a5a845668696dcb28c3badd6
I20260129 03:00:06.003958 20515 rpcz_store.cc:276] 0129 03:00:04.528336 (+ 0us) service_pool.cc:168] Inserting onto call queue
0129 03:00:05.206536 (+678200us) service_pool.cc:225] Handling call
0129 03:00:05.209411 (+ 2875us) raft_consensus.cc:1471] Updating replica for 1 ops
0129 03:00:05.284670 (+ 75259us) raft_consensus.cc:1534] Early marking committed up to index 4
0129 03:00:05.318261 (+ 33591us) log.cc:844] Serialized 14 byte log entry
0129 03:00:05.403320 (+ 85059us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0129 03:00:05.419017 (+ 15697us) log.cc:844] Serialized 26 byte log entry
0129 03:00:05.419176 (+ 159us) raft_consensus.cc:1645] Marking committed up to 5
0129 03:00:05.955686 (+536510us) log.cc:844] Serialized 14 byte log entry
0129 03:00:05.972111 (+ 16425us) raft_consensus.cc:1697] Filling consensus response to leader.
0129 03:00:05.972185 (+ 74us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0129 03:00:05.972229 (+ 44us) raft_consensus.cc:1685] finished
0129 03:00:05.972244 (+ 15us) raft_consensus.cc:1691] UpdateReplicas() finished
0129 03:00:05.972388 (+ 144us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":606217,"prepare.queue_time_us":537269,"prepare.run_cpu_time_us":35,"prepare.run_wall_time_us":35,"raft.queue_time_us":78599,"raft.run_cpu_time_us":368,"raft.run_wall_time_us":6360,"thread_start_us":8827,"threads_started":1,"wal-append.queue_time_us":91574}
I20260129 03:00:05.963577 23104 consensus_queue.cc:260] T a1a39dcd328e4268ae41b988c66b4a75 P 1c50de89a5a845668696dcb28c3badd6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } } peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } }
I20260129 03:00:06.006954 20511 raft_consensus.cc:2393] T 4b1876f9a70b49edb3aa74105e66d3b3 P 974b802141bd4f13beffef2aa03f88f9 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 197b880fa39d432a9b472845876bfaf0 in current term 6: Already voted for candidate 974b802141bd4f13beffef2aa03f88f9 in this term.
I20260129 03:00:06.003309 20581 raft_consensus.cc:740] T 3e2d15f8640b407195210c476e09df91 P 1c50de89a5a845668696dcb28c3badd6 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 1c50de89a5a845668696dcb28c3badd6, State: Running, Role: LEADER
W20260129 03:00:06.010764 20556 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.211.2:34517, user_credentials={real_user=slave}} blocked reactor thread for 284088us
W20260129 03:00:06.025978 20556 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260129 03:00:06.026722 20556 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260129 03:00:06.027148 20556 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260129 03:00:06.027577 20556 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260129 03:00:05.950574 20508 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "cb4727386a3e48aca5d71cc6c611c197" candidate_uuid: "1c50de89a5a845668696dcb28c3badd6" candidate_term: 5 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "974b802141bd4f13beffef2aa03f88f9"
I20260129 03:00:05.782634 20510 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "cb0b3f259ebc4310927338a8740e87b8" candidate_uuid: "1c50de89a5a845668696dcb28c3badd6" candidate_term: 8 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "974b802141bd4f13beffef2aa03f88f9" is_pre_election: true
I20260129 03:00:06.030848 20508 raft_consensus.cc:3060] T cb4727386a3e48aca5d71cc6c611c197 P 974b802141bd4f13beffef2aa03f88f9 [term 4 FOLLOWER]: Advancing to term 5
I20260129 03:00:06.031136 20510 raft_consensus.cc:2468] T cb0b3f259ebc4310927338a8740e87b8 P 974b802141bd4f13beffef2aa03f88f9 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1c50de89a5a845668696dcb28c3badd6 in term 7.
I20260129 03:00:06.032567 20556 leader_election.cc:304] T cb0b3f259ebc4310927338a8740e87b8 P 1c50de89a5a845668696dcb28c3badd6 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1c50de89a5a845668696dcb28c3badd6, 974b802141bd4f13beffef2aa03f88f9; no voters: 7a9daf5d42a84615a6383cfec9a10608
I20260129 03:00:06.032991 20436 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a0fcc58d6fb6473cafbd9d34d5195269" candidate_uuid: "974b802141bd4f13beffef2aa03f88f9" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "197b880fa39d432a9b472845876bfaf0" is_pre_election: true
W20260129 03:00:06.033910 20553 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.211.1:42027, user_credentials={real_user=slave}} blocked reactor thread for 377581us
I20260129 03:00:05.955132 21133 raft_consensus.cc:1217] T 4e420f90884f4796997b5ea41e49e411 P 7a9daf5d42a84615a6383cfec9a10608 [term 6 LEADER]: Deduplicated request from leader. Original: 4.6->[5.7-5.7] Dedup: 5.7->[]
W20260129 03:00:06.036600 20441 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.634s user 0.017s sys 0.024s
W20260129 03:00:06.037046 20441 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.634s user 0.017s sys 0.024s
I20260129 03:00:06.013813 20516 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57600 (request call id 1885) took 2563 ms. Trace:
I20260129 03:00:06.044255 21133 raft_consensus.cc:1240] T 4e420f90884f4796997b5ea41e49e411 P 7a9daf5d42a84615a6383cfec9a10608 [term 6 LEADER]: Rejecting Update request from peer 974b802141bd4f13beffef2aa03f88f9 for earlier term 5. Current term is 6. Ops: [5.7-5.7]
I20260129 03:00:05.954115 23091 raft_consensus.cc:493] T a609ac02fae84725ab743f95f51d33ce P 197b880fa39d432a9b472845876bfaf0 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260129 03:00:05.974113 23208 raft_consensus.cc:3055] T a609ac02fae84725ab743f95f51d33ce P 974b802141bd4f13beffef2aa03f88f9 [term 6 LEADER]: Stepping down as leader of term 6
I20260129 03:00:06.046597 20590 consensus_queue.cc:260] T 75b0b6e82ddc4f92b1fe1d94f9a0b73b P 1c50de89a5a845668696dcb28c3badd6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 3.6, Last appended by leader: 6, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } }
I20260129 03:00:06.064864 20438 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "76f713c3331540d2a5b76164103e9eb1" candidate_uuid: "974b802141bd4f13beffef2aa03f88f9" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "197b880fa39d432a9b472845876bfaf0" is_pre_election: true
I20260129 03:00:06.064103 20514 raft_consensus.cc:2468] T c2633a6fc34c44579de348d208b0c01e P 974b802141bd4f13beffef2aa03f88f9 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1c50de89a5a845668696dcb28c3badd6 in term 4.
I20260129 03:00:06.069932 20433 raft_consensus.cc:1217] T 4e420f90884f4796997b5ea41e49e411 P 197b880fa39d432a9b472845876bfaf0 [term 6 FOLLOWER]: Deduplicated request from leader. Original: 4.6->[5.7-5.7] Dedup: 5.7->[]
I20260129 03:00:06.037456 20516 rpcz_store.cc:276] 0129 03:00:03.450225 (+ 0us) service_pool.cc:168] Inserting onto call queue
0129 03:00:04.970910 (+1520685us) service_pool.cc:225] Handling call
0129 03:00:04.971412 (+ 502us) raft_consensus.cc:1471] Updating replica for 0 ops
0129 03:00:04.971986 (+ 574us) raft_consensus.cc:1534] Early marking committed up to index 4
0129 03:00:05.167087 (+195101us) log.cc:844] Serialized 14 byte log entry
0129 03:00:05.338867 (+171780us) log.cc:844] Serialized 14 byte log entry
0129 03:00:05.954413 (+615546us) spinlock_profiling.cc:243] Waited 2.09 ms on lock 0x7b18004fa800. stack: 00007f8fe044e373 00007f8fe044e270 00007f8fdfda0574 00007f8fe9a5c6b4 00007f8fe9a5c649 00007f8fe92d4c91 00007f8fe92e1671 00007f8fe03f546a 00007f8fe9a7b5c9 00007f8fe02b5160 00007f8fe047d3c4 00007f8fe047d335 00007f8fe6f28317 00007f8fe3538627 00007f8fe3547d3f 00007f8fe3547cb8
0129 03:00:05.954592 (+ 179us) log.cc:844] Serialized 14 byte log entry
0129 03:00:05.957949 (+ 3357us) spinlock_profiling.cc:243] Waited 7.05 ms on lock 0x7b18004fa800. stack: 00007f8fe044e373 00007f8fe044e270 00007f8fdfda0574 00007f8fe9a5c6b4 00007f8fe9a5c649 00007f8fe92d4c91 00007f8fe92e1671 00007f8fe04802d3 00007f8fe04826a1 00007f8fe0482659 00007f8fe04825e9 00007f8fe04825b1 00007f8fe04818ad 0000000000379254 0000000000379099 00007f8fe046369a
0129 03:00:06.013333 (+ 55384us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0129 03:00:06.013402 (+ 69us) raft_consensus.cc:1645] Marking committed up to 4
0129 03:00:06.013460 (+ 58us) raft_consensus.cc:1697] Filling consensus response to leader.
0129 03:00:06.013546 (+ 86us) raft_consensus.cc:1691] UpdateReplicas() finished
0129 03:00:06.013699 (+ 153us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":96237,"prepare.queue_time_us":349765,"prepare.run_wall_time_us":4302,"raft.queue_time_us":86107,"raft.run_cpu_time_us":244,"raft.run_wall_time_us":1982,"spinlock_wait_cycles":104765568,"thread_start_us":328173,"threads_started":2,"wal-append.queue_time_us":206332}
I20260129 03:00:06.070494 20433 raft_consensus.cc:1240] T 4e420f90884f4796997b5ea41e49e411 P 197b880fa39d432a9b472845876bfaf0 [term 6 FOLLOWER]: Rejecting Update request from peer 974b802141bd4f13beffef2aa03f88f9 for earlier term 5. Current term is 6. Ops: [5.7-5.7]
I20260129 03:00:06.071669 20556 leader_election.cc:304] T c2633a6fc34c44579de348d208b0c01e P 1c50de89a5a845668696dcb28c3badd6 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1c50de89a5a845668696dcb28c3badd6, 974b802141bd4f13beffef2aa03f88f9; no voters:
I20260129 03:00:06.071808 20510 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "724a236524cb4e7ba3428880681d6cc2" candidate_uuid: "1c50de89a5a845668696dcb28c3badd6" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "974b802141bd4f13beffef2aa03f88f9" is_pre_election: true
W20260129 03:00:06.074734 20555 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260129 03:00:06.074613 20514 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "cb4727386a3e48aca5d71cc6c611c197" candidate_uuid: "197b880fa39d432a9b472845876bfaf0" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "974b802141bd4f13beffef2aa03f88f9"
W20260129 03:00:06.059625 20480 connection.cc:232] RPC call timeout handler was delayed by -1.206519048999894s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.22970000000000002s for a response
W20260129 03:00:06.079257 22977 consensus_meta.cc:287] T 7a9a69cd82724b1faed1d10aa091fc4a P 1c50de89a5a845668696dcb28c3badd6: Time spent flushing consensus metadata: real 0.787s user 0.015s sys 0.030s
I20260129 03:00:06.079947 23113 raft_consensus.cc:493] T 1299723a4476420ab02fd8bfd7aa0645 P 974b802141bd4f13beffef2aa03f88f9 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260129 03:00:06.080312 21130 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "febffe4a8b394368b69b1d164308fb9a" candidate_uuid: "974b802141bd4f13beffef2aa03f88f9" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "7a9daf5d42a84615a6383cfec9a10608"
I20260129 03:00:06.080636 23113 raft_consensus.cc:515] T 1299723a4476420ab02fd8bfd7aa0645 P 974b802141bd4f13beffef2aa03f88f9 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } attrs { replace: true } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } }
W20260129 03:00:06.081604 21135 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34060 (request call id 527) took 3572 ms (client timeout 3251 ms). Trace:
I20260129 03:00:06.081559 23102 consensus_queue.cc:1048] T a609ac02fae84725ab743f95f51d33ce P 7a9daf5d42a84615a6383cfec9a10608 [LEADER]: Connected to new peer: Peer: permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.013s
W20260129 03:00:06.082247 21135 rpcz_store.cc:269] 0129 03:00:02.509350 (+ 0us) service_pool.cc:168] Inserting onto call queue
0129 03:00:02.666041 (+156691us) service_pool.cc:225] Handling call
0129 03:00:06.081187 (+3415146us) spinlock_profiling.cc:243] Waited 541 ms on lock 0x7b54003b3dd4. stack: 00007f8fe044e373 00007f8fe044e270 00007f8fdfda0574 00007f8fe9a5c6b4 00007f8fe9a5c649 00007f8fe34975a1 00007f8fe3531f7a 00007f8fe84e3647 00007f8fe1a73834 00007f8fe1a737c2 00007f8fe1a736f1 00007f8fe1a7366c 00007f8fe1a72922 00007f8fe186d34c 00007f8fe186c786 00007f8fe186c12f
0129 03:00:06.081392 (+ 205us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1243397376}
I20260129 03:00:06.066725 20581 consensus_queue.cc:260] T 3e2d15f8640b407195210c476e09df91 P 1c50de89a5a845668696dcb28c3badd6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.5, Last appended by leader: 5, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } }
I20260129 03:00:06.081208 21130 raft_consensus.cc:2393] T febffe4a8b394368b69b1d164308fb9a P 7a9daf5d42a84615a6383cfec9a10608 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 974b802141bd4f13beffef2aa03f88f9 in current term 4: Already voted for candidate 7a9daf5d42a84615a6383cfec9a10608 in this term.
I20260129 03:00:06.086508 20438 raft_consensus.cc:2468] T 76f713c3331540d2a5b76164103e9eb1 P 197b880fa39d432a9b472845876bfaf0 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 974b802141bd4f13beffef2aa03f88f9 in term 5.
I20260129 03:00:06.012729 23145 raft_consensus.cc:515] T b189f08e16a74a11980cc55d99359a0a P 974b802141bd4f13beffef2aa03f88f9 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 03:00:06.088176 21128 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3e2d15f8640b407195210c476e09df91" candidate_uuid: "974b802141bd4f13beffef2aa03f88f9" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "7a9daf5d42a84615a6383cfec9a10608" is_pre_election: true
I20260129 03:00:06.089448 23145 leader_election.cc:290] T b189f08e16a74a11980cc55d99359a0a P 974b802141bd4f13beffef2aa03f88f9 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 197b880fa39d432a9b472845876bfaf0 (127.18.211.1:42027), 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3:37735)
I20260129 03:00:06.060736 23109 raft_consensus.cc:2749] T dc2f4cb752cf4d59b5f6bc689f5d8104 P 7a9daf5d42a84615a6383cfec9a10608 [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20260129 03:00:06.090986 23290 leader_election.cc:290] T 3906b080a5b34a74b8b64128635e8997 P 1c50de89a5a845668696dcb28c3badd6 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 197b880fa39d432a9b472845876bfaf0 (127.18.211.1:42027), 974b802141bd4f13beffef2aa03f88f9 (127.18.211.2:34517)
I20260129 03:00:06.091274 23034 raft_consensus.cc:493] T 7a9a69cd82724b1faed1d10aa091fc4a P 974b802141bd4f13beffef2aa03f88f9 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260129 03:00:06.083516 23113 leader_election.cc:290] T 1299723a4476420ab02fd8bfd7aa0645 P 974b802141bd4f13beffef2aa03f88f9 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 197b880fa39d432a9b472845876bfaf0 (127.18.211.1:42027), 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3:37735), 7a9daf5d42a84615a6383cfec9a10608 (127.18.211.4:40439)
I20260129 03:00:06.046164 22996 raft_consensus.cc:3060] T 4e3805ffa70242e8ade66da260dddf66 P 1c50de89a5a845668696dcb28c3badd6 [term 5 FOLLOWER]: Advancing to term 6
I20260129 03:00:06.003396 23086 consensus_queue.cc:1059] T 8055232cf0004d40a85f3bf983fe00a9 P 197b880fa39d432a9b472845876bfaf0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 }, Status: INVALID_TERM, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.058s
W20260129 03:00:06.104606 20555 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260129 03:00:06.110442 20441 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "febffe4a8b394368b69b1d164308fb9a" candidate_uuid: "974b802141bd4f13beffef2aa03f88f9" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "197b880fa39d432a9b472845876bfaf0"
I20260129 03:00:05.952118 20335 catalog_manager.cc:5654] T 1299723a4476420ab02fd8bfd7aa0645 P 7a9daf5d42a84615a6383cfec9a10608 reported cstate change: config changed from index 4 to 5, term changed from 4 to 8, leader changed from 197b880fa39d432a9b472845876bfaf0 (127.18.211.1) to 7a9daf5d42a84615a6383cfec9a10608 (127.18.211.4), VOTER 974b802141bd4f13beffef2aa03f88f9 (127.18.211.2) evicted, no longer has a pending config: VOTER 197b880fa39d432a9b472845876bfaf0 (127.18.211.1), VOTER 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3), VOTER 7a9daf5d42a84615a6383cfec9a10608 (127.18.211.4). New cstate: current_term: 8 leader_uuid: "7a9daf5d42a84615a6383cfec9a10608" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260129 03:00:06.060915 23208 raft_consensus.cc:740] T a609ac02fae84725ab743f95f51d33ce P 974b802141bd4f13beffef2aa03f88f9 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 974b802141bd4f13beffef2aa03f88f9, State: Running, Role: LEADER
I20260129 03:00:06.118466 20586 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35752 (request call id 1647) took 3758 ms. Trace:
I20260129 03:00:06.114974 20590 raft_consensus.cc:3060] T 75b0b6e82ddc4f92b1fe1d94f9a0b73b P 1c50de89a5a845668696dcb28c3badd6 [term 3 FOLLOWER]: Advancing to term 4
I20260129 03:00:06.091898 23034 raft_consensus.cc:515] T 7a9a69cd82724b1faed1d10aa091fc4a P 974b802141bd4f13beffef2aa03f88f9 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 03:00:06.117874 20438 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "87d5fe48a44a408aaf03e894a67ecb2f" candidate_uuid: "7a9daf5d42a84615a6383cfec9a10608" candidate_term: 7 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "197b880fa39d432a9b472845876bfaf0" is_pre_election: true
I20260129 03:00:06.121197 20438 raft_consensus.cc:2410] T 87d5fe48a44a408aaf03e894a67ecb2f P 197b880fa39d432a9b472845876bfaf0 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 7a9daf5d42a84615a6383cfec9a10608 for term 7 because replica has last-logged OpId of term: 5 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 4 index: 6.
I20260129 03:00:06.069514 23057 leader_election.cc:290] T cb4727386a3e48aca5d71cc6c611c197 P 197b880fa39d432a9b472845876bfaf0 [CANDIDATE]: Term 5 election: Requested vote from peers 974b802141bd4f13beffef2aa03f88f9 (127.18.211.2:34517), 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3:37735)
I20260129 03:00:06.057442 23216 raft_consensus.cc:515] T 4f5a77e94fef4b16a2ba7b067c8b144a P 974b802141bd4f13beffef2aa03f88f9 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } }
W20260129 03:00:06.137099 20553 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.211.1:42027, user_credentials={real_user=slave}} blocked reactor thread for 59912.7us
I20260129 03:00:06.129513 20510 raft_consensus.cc:1275] T 00e79d8f25a34676b1d866eb2068a2b2 P 974b802141bd4f13beffef2aa03f88f9 [term 8 FOLLOWER]: Refusing update from remote peer 197b880fa39d432a9b472845876bfaf0: Log matching property violated. Preceding OpId in replica: term: 4 index: 7. Preceding OpId from leader: term: 8 index: 9. (index mismatch)
W20260129 03:00:06.113180 22855 consensus_meta.cc:287] T dc2f4cb752cf4d59b5f6bc689f5d8104 P 1c50de89a5a845668696dcb28c3badd6: Time spent flushing consensus metadata: real 0.579s user 0.014s sys 0.006s
I20260129 03:00:06.103190 23104 raft_consensus.cc:3060] T a1a39dcd328e4268ae41b988c66b4a75 P 1c50de89a5a845668696dcb28c3badd6 [term 4 FOLLOWER]: Advancing to term 5
I20260129 03:00:06.115051 23290 raft_consensus.cc:2804] T cb0b3f259ebc4310927338a8740e87b8 P 1c50de89a5a845668696dcb28c3badd6 [term 7 FOLLOWER]: Leader pre-election won for term 8
W20260129 03:00:06.140419 20582 consensus_meta.cc:287] T 96609ac68bdf4993a12e1c23355ca013 P 1c50de89a5a845668696dcb28c3badd6: Time spent flushing consensus metadata: real 0.770s user 0.021s sys 0.025s
I20260129 03:00:06.068967 23091 raft_consensus.cc:515] T a609ac02fae84725ab743f95f51d33ce P 197b880fa39d432a9b472845876bfaf0 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } }
I20260129 03:00:06.115618 23147 raft_consensus.cc:2804] T c2633a6fc34c44579de348d208b0c01e P 1c50de89a5a845668696dcb28c3badd6 [term 4 FOLLOWER]: Leader election won for term 4
I20260129 03:00:06.118999 20586 rpcz_store.cc:276] 0129 03:00:02.006568 (+ 0us) service_pool.cc:168] Inserting onto call queue
0129 03:00:04.169445 (+2162877us) service_pool.cc:225] Handling call
0129 03:00:04.169881 (+ 436us) raft_consensus.cc:1471] Updating replica for 1 ops
0129 03:00:04.232835 (+ 62954us) raft_consensus.cc:1534] Early marking committed up to index 0
0129 03:00:04.232879 (+ 44us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0129 03:00:04.233378 (+ 499us) log.cc:844] Serialized 26 byte log entry
0129 03:00:05.425138 (+1191760us) spinlock_profiling.cc:243] Waited 1.2 ms on lock 0x7b18003e5f00. stack: 00007f8fe044e373 00007f8fe044e270 00007f8fdfda0574 00007f8fe9a5c6b4 00007f8fe9a5c649 00007f8fe92d4c91 00007f8fe92e1671 00007f8fe0462c67 00007f8fe924b709 00007f8fe047f57b 00007f8fe047d81f 00007f8fe047fae2 00007f8fe30c75ab 00007f8fe30cd0c3 00007f8fe30cd471 00007f8fe34e973f
0129 03:00:05.470689 (+ 45551us) raft_consensus.cc:1645] Marking committed up to 3
0129 03:00:05.471171 (+ 482us) log.cc:844] Serialized 14 byte log entry
0129 03:00:05.609037 (+137866us) raft_consensus.cc:1697] Filling consensus response to leader.
0129 03:00:05.713609 (+104572us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0129 03:00:05.713754 (+ 145us) raft_consensus.cc:1685] finished
0129 03:00:05.740576 (+ 26822us) raft_consensus.cc:1691] UpdateReplicas() finished
0129 03:00:05.764329 (+ 23753us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":5637,"prepare.queue_time_us":213,"prepare.run_cpu_time_us":45,"prepare.run_wall_time_us":47,"raft.queue_time_us":157590,"raft.run_cpu_time_us":335,"raft.run_wall_time_us":24443,"spinlock_wait_cycles":2769792,"thread_start_us":161012,"threads_started":2,"wal-append.queue_time_us":1287689}
I20260129 03:00:06.146811 20553 leader_election.cc:304] T a8479d1fe0bd41e5946186e3a6ca8c7a P 1c50de89a5a845668696dcb28c3badd6 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1c50de89a5a845668696dcb28c3badd6; no voters: 197b880fa39d432a9b472845876bfaf0, 974b802141bd4f13beffef2aa03f88f9
W20260129 03:00:06.150985 20407 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.211.2:34517, user_credentials={real_user=slave}} blocked reactor thread for 47308.5us
W20260129 03:00:06.137952 23262 consensus_meta.cc:287] T b189f08e16a74a11980cc55d99359a0a P 1c50de89a5a845668696dcb28c3badd6: Time spent flushing consensus metadata: real 0.854s user 0.022s sys 0.026s
I20260129 03:00:06.200982 20438 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "75be4bd6bd45474795fd472eff2953ac" candidate_uuid: "974b802141bd4f13beffef2aa03f88f9" candidate_term: 11 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "197b880fa39d432a9b472845876bfaf0" is_pre_election: true
I20260129 03:00:06.132684 23185 leader_election.cc:290] T 75be4bd6bd45474795fd472eff2953ac P 974b802141bd4f13beffef2aa03f88f9 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 197b880fa39d432a9b472845876bfaf0 (127.18.211.1:42027), 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3:37735)
W20260129 03:00:06.211742 20480 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.211.4:40439, user_credentials={real_user=slave}} blocked reactor thread for 60256.4us
I20260129 03:00:06.208731 20441 raft_consensus.cc:2468] T febffe4a8b394368b69b1d164308fb9a P 197b880fa39d432a9b472845876bfaf0 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 974b802141bd4f13beffef2aa03f88f9 in term 4.
I20260129 03:00:06.145587 23290 raft_consensus.cc:493] T cb0b3f259ebc4310927338a8740e87b8 P 1c50de89a5a845668696dcb28c3badd6 [term 7 FOLLOWER]: Starting leader election (detected failure of leader 7a9daf5d42a84615a6383cfec9a10608)
I20260129 03:00:06.204721 20335 catalog_manager.cc:5654] T 42d27a28d89b409e8807ec0efd97af69 P 7a9daf5d42a84615a6383cfec9a10608 reported cstate change: term changed from 2 to 6, leader changed from 7a9daf5d42a84615a6383cfec9a10608 (127.18.211.4) to 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3). New cstate: current_term: 6 leader_uuid: "1c50de89a5a845668696dcb28c3badd6" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } } }
W20260129 03:00:06.243706 20554 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.211.4:40439, user_credentials={real_user=slave}} blocked reactor thread for 335056us
I20260129 03:00:06.243574 20441 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "18f30ebaa4174fd28c958e72d0cc7e24" candidate_uuid: "7a9daf5d42a84615a6383cfec9a10608" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "197b880fa39d432a9b472845876bfaf0"
I20260129 03:00:06.156550 23281 leader_election.cc:290] T 0678b8b468bd43cdbc3e485900844821 P 197b880fa39d432a9b472845876bfaf0 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 974b802141bd4f13beffef2aa03f88f9 (127.18.211.2:34517), 7a9daf5d42a84615a6383cfec9a10608 (127.18.211.4:40439)
I20260129 03:00:06.245455 20513 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:48544 (request call id 294) took 1663 ms. Trace:
I20260129 03:00:06.205140 23145 raft_consensus.cc:493] T cedce331e1584ac8b636c9b8a72b2ba0 P 974b802141bd4f13beffef2aa03f88f9 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260129 03:00:06.246147 20480 leader_election.cc:304] T 75f45538dc704d9a8ec4ff8ad9f3ea5b P 974b802141bd4f13beffef2aa03f88f9 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 974b802141bd4f13beffef2aa03f88f9; no voters: 1c50de89a5a845668696dcb28c3badd6, 7a9daf5d42a84615a6383cfec9a10608
W20260129 03:00:06.257344 20405 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.211.3:37735, user_credentials={real_user=slave}} blocked reactor thread for 198260us
I20260129 03:00:06.257643 20441 raft_consensus.cc:2393] T 18f30ebaa4174fd28c958e72d0cc7e24 P 197b880fa39d432a9b472845876bfaf0 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 7a9daf5d42a84615a6383cfec9a10608 in current term 7: Already voted for candidate 197b880fa39d432a9b472845876bfaf0 in this term.
W20260129 03:00:06.245374 20553 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.211.1:42027, user_credentials={real_user=slave}} blocked reactor thread for 98664.1us
I20260129 03:00:06.259964 21099 leader_election.cc:304] T 87d5fe48a44a408aaf03e894a67ecb2f P 7a9daf5d42a84615a6383cfec9a10608 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7a9daf5d42a84615a6383cfec9a10608; no voters: 197b880fa39d432a9b472845876bfaf0, 974b802141bd4f13beffef2aa03f88f9
I20260129 03:00:06.261008 20435 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b189f08e16a74a11980cc55d99359a0a" candidate_uuid: "974b802141bd4f13beffef2aa03f88f9" candidate_term: 10 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "197b880fa39d432a9b472845876bfaf0" is_pre_election: true
I20260129 03:00:06.213397 23147 raft_consensus.cc:697] T c2633a6fc34c44579de348d208b0c01e P 1c50de89a5a845668696dcb28c3badd6 [term 4 LEADER]: Becoming Leader. State: Replica: 1c50de89a5a845668696dcb28c3badd6, State: Running, Role: LEADER
I20260129 03:00:06.243003 23290 raft_consensus.cc:3060] T cb0b3f259ebc4310927338a8740e87b8 P 1c50de89a5a845668696dcb28c3badd6 [term 7 FOLLOWER]: Advancing to term 8
W20260129 03:00:06.265986 20479 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.211.1:42027, user_credentials={real_user=slave}} blocked reactor thread for 84219.8us
I20260129 03:00:06.267151 20438 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1299723a4476420ab02fd8bfd7aa0645" candidate_uuid: "7a9daf5d42a84615a6383cfec9a10608" candidate_term: 10 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "197b880fa39d432a9b472845876bfaf0" is_pre_election: true
I20260129 03:00:06.260241 20335 catalog_manager.cc:5654] T a644c02b16af47679824ccd060ac10fd P 7a9daf5d42a84615a6383cfec9a10608 reported cstate change: term changed from 1 to 4, leader changed from 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3) to 7a9daf5d42a84615a6383cfec9a10608 (127.18.211.4). New cstate: current_term: 4 leader_uuid: "7a9daf5d42a84615a6383cfec9a10608" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
W20260129 03:00:06.244457 23209 log.cc:927] Time spent T a5a05fbcbbcf4916bbc01fd1a5b5e90d P 1c50de89a5a845668696dcb28c3badd6: Append to log took a long time: real 3.273s user 0.009s sys 0.023s
I20260129 03:00:06.271436 20441 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1299723a4476420ab02fd8bfd7aa0645" candidate_uuid: "974b802141bd4f13beffef2aa03f88f9" candidate_term: 9 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "197b880fa39d432a9b472845876bfaf0" is_pre_election: true
W20260129 03:00:06.273392 22694 consensus_meta.cc:287] T c891d2bff4ce4568861bbc992e91fbb0 P 974b802141bd4f13beffef2aa03f88f9: Time spent flushing consensus metadata: real 0.991s user 0.016s sys 0.014s
I20260129 03:00:06.273578 23086 raft_consensus.cc:493] T 2e58e9a5a8aa4cecafb464addb4f35ed P 197b880fa39d432a9b472845876bfaf0 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 7a9daf5d42a84615a6383cfec9a10608)
W20260129 03:00:06.265767 23180 consensus_meta.cc:287] T cedce331e1584ac8b636c9b8a72b2ba0 P 197b880fa39d432a9b472845876bfaf0: Time spent flushing consensus metadata: real 0.994s user 0.024s sys 0.006s
I20260129 03:00:06.276994 21128 raft_consensus.cc:1240] T 75b0b6e82ddc4f92b1fe1d94f9a0b73b P 7a9daf5d42a84615a6383cfec9a10608 [term 4 LEADER]: Rejecting Update request from peer 1c50de89a5a845668696dcb28c3badd6 for earlier term 3. Current term is 4. Ops: []
I20260129 03:00:06.238054 23085 raft_consensus.cc:515] T a0fcc58d6fb6473cafbd9d34d5195269 P 197b880fa39d432a9b472845876bfaf0 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 03:00:06.255126 20513 rpcz_store.cc:276] 0129 03:00:04.582154 (+ 0us) service_pool.cc:168] Inserting onto call queue
0129 03:00:05.269480 (+687326us) service_pool.cc:225] Handling call
0129 03:00:05.269915 (+ 435us) raft_consensus.cc:1471] Updating replica for 1 ops
0129 03:00:05.313768 (+ 43853us) raft_consensus.cc:1534] Early marking committed up to index 0
0129 03:00:05.313802 (+ 34us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0129 03:00:05.333865 (+ 20063us) log.cc:844] Serialized 26 byte log entry
0129 03:00:05.569924 (+236059us) raft_consensus.cc:1645] Marking committed up to 6
0129 03:00:06.201162 (+631238us) log.cc:844] Serialized 14 byte log entry
0129 03:00:06.245038 (+ 43876us) raft_consensus.cc:1697] Filling consensus response to leader.
0129 03:00:06.245142 (+ 104us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0129 03:00:06.245195 (+ 53us) raft_consensus.cc:1685] finished
0129 03:00:06.245216 (+ 21us) raft_consensus.cc:1691] UpdateReplicas() finished
0129 03:00:06.245383 (+ 167us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":469673,"prepare.queue_time_us":386665,"prepare.run_cpu_time_us":46,"prepare.run_wall_time_us":47,"raft.queue_time_us":37439,"raft.run_cpu_time_us":126,"raft.run_wall_time_us":5479,"thread_start_us":439416,"threads_started":2,"wal-append.queue_time_us":246043}
I20260129 03:00:06.144312 23208 consensus_queue.cc:260] T a609ac02fae84725ab743f95f51d33ce P 974b802141bd4f13beffef2aa03f88f9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 6.7, Last appended by leader: 7, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } }
I20260129 03:00:06.308279 23208 raft_consensus.cc:3060] T a609ac02fae84725ab743f95f51d33ce P 974b802141bd4f13beffef2aa03f88f9 [term 6 FOLLOWER]: Advancing to term 7
I20260129 03:00:06.273885 20438 raft_consensus.cc:2468] T 1299723a4476420ab02fd8bfd7aa0645 P 197b880fa39d432a9b472845876bfaf0 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7a9daf5d42a84615a6383cfec9a10608 in term 9.
I20260129 03:00:06.286788 20516 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0678b8b468bd43cdbc3e485900844821" candidate_uuid: "197b880fa39d432a9b472845876bfaf0" candidate_term: 9 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "974b802141bd4f13beffef2aa03f88f9" is_pre_election: true
I20260129 03:00:06.285032 20335 catalog_manager.cc:5654] T dc2f4cb752cf4d59b5f6bc689f5d8104 P 7a9daf5d42a84615a6383cfec9a10608 reported cstate change: term changed from 3 to 4, leader changed from 7a9daf5d42a84615a6383cfec9a10608 (127.18.211.4) to 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3). New cstate: current_term: 4 leader_uuid: "1c50de89a5a845668696dcb28c3badd6" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } } }
I20260129 03:00:06.316596 23091 leader_election.cc:290] T a609ac02fae84725ab743f95f51d33ce P 197b880fa39d432a9b472845876bfaf0 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 974b802141bd4f13beffef2aa03f88f9 (127.18.211.2:34517), 7a9daf5d42a84615a6383cfec9a10608 (127.18.211.4:40439)
I20260129 03:00:06.316248 20335 catalog_manager.cc:5654] T 75b0b6e82ddc4f92b1fe1d94f9a0b73b P 7a9daf5d42a84615a6383cfec9a10608 reported cstate change: term changed from 1 to 3. New cstate: current_term: 3 leader_uuid: "1c50de89a5a845668696dcb28c3badd6" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } } }
I20260129 03:00:06.317780 20335 catalog_manager.cc:5654] T fd8311b779af4e1fa1402b2a02a602f2 P 7a9daf5d42a84615a6383cfec9a10608 reported cstate change: term changed from 2 to 4, leader changed from 197b880fa39d432a9b472845876bfaf0 (127.18.211.1) to 7a9daf5d42a84615a6383cfec9a10608 (127.18.211.4). New cstate: current_term: 4 leader_uuid: "7a9daf5d42a84615a6383cfec9a10608" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
W20260129 03:00:06.269160 23026 consensus_meta.cc:287] T 505580c93c694bbb8ea96fdf9b17ab46 P 1c50de89a5a845668696dcb28c3badd6: Time spent flushing consensus metadata: real 0.723s user 0.010s sys 0.017s
I20260129 03:00:06.320173 23026 raft_consensus.cc:515] T 505580c93c694bbb8ea96fdf9b17ab46 P 1c50de89a5a845668696dcb28c3badd6 [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } }
I20260129 03:00:06.322296 23026 leader_election.cc:290] T 505580c93c694bbb8ea96fdf9b17ab46 P 1c50de89a5a845668696dcb28c3badd6 [CANDIDATE]: Term 7 election: Requested vote from peers 974b802141bd4f13beffef2aa03f88f9 (127.18.211.2:34517), 7a9daf5d42a84615a6383cfec9a10608 (127.18.211.4:40439)
I20260129 03:00:06.291787 23097 raft_consensus.cc:2749] T 75f45538dc704d9a8ec4ff8ad9f3ea5b P 974b802141bd4f13beffef2aa03f88f9 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260129 03:00:06.324030 20436 raft_consensus.cc:2393] T a0fcc58d6fb6473cafbd9d34d5195269 P 197b880fa39d432a9b472845876bfaf0 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 974b802141bd4f13beffef2aa03f88f9 in current term 5: Already voted for candidate 197b880fa39d432a9b472845876bfaf0 in this term.
I20260129 03:00:06.326339 20589 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35752 (request call id 1633) took 4460 ms. Trace:
I20260129 03:00:06.326720 20589 rpcz_store.cc:276] 0129 03:00:01.865504 (+ 0us) service_pool.cc:168] Inserting onto call queue
0129 03:00:02.410399 (+544895us) service_pool.cc:225] Handling call
0129 03:00:02.419029 (+ 8630us) raft_consensus.cc:1471] Updating replica for 1 ops
0129 03:00:02.419277 (+ 248us) raft_consensus.cc:1534] Early marking committed up to index 0
0129 03:00:02.419305 (+ 28us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0129 03:00:02.419823 (+ 518us) log.cc:844] Serialized 26 byte log entry
0129 03:00:02.861136 (+441313us) raft_consensus.cc:1645] Marking committed up to 5
0129 03:00:02.981837 (+120701us) log.cc:844] Serialized 14 byte log entry
0129 03:00:02.988295 (+ 6458us) raft_consensus.cc:1697] Filling consensus response to leader.
0129 03:00:02.988389 (+ 94us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0129 03:00:04.929406 (+1941017us) spinlock_profiling.cc:243] Waited 7.77 ms on lock 0x7b18003c5380. stack: 00007f8fe044e373 00007f8fe044e270 00007f8fdfda0574 00007f8fe9a5c6b4 00007f8fe9a5c649 00007f8fe92d4c91 00007f8fe92e1671 00007f8fe044e2d6 00007f8fe044e270 00007f8fdfda0574 00007f8fe9a5c6b4 00007f8fe9a5c649 00007f8fe02c08b1 00007f8fe038f005 00007f8fe038fa22 00007f8fe30d957b
0129 03:00:05.917813 (+988407us) spinlock_profiling.cc:243] Waited 48.3 ms on lock 0x7b440002304c. stack: 00007f8fe044e373 00007f8fe044e270 00007f8fdfda0574 00007f8fe9a5c6b4 00007f8fe9a5c649 00007f8fe02c08b1 00007f8fe038f005 00007f8fe038fa22 00007f8fe30d957b 00007f8fe30d4e4f 00007f8fe30c8c0e 00007f8fe30c7f56 00007f8fe30c7b78 00007f8fe30d15c1 00007f8fe30d1579 00007f8fe30d1509
0129 03:00:06.325998 (+408185us) raft_consensus.cc:1685] finished
0129 03:00:06.326020 (+ 22us) raft_consensus.cc:1691] UpdateReplicas() finished
0129 03:00:06.326167 (+ 147us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":68743,"prepare.queue_time_us":284151,"prepare.run_cpu_time_us":98,"prepare.run_wall_time_us":1316,"raft.queue_time_us":36322,"raft.run_cpu_time_us":141,"raft.run_wall_time_us":10491,"spinlock_wait_cycles":129040000,"thread_start_us":364715,"threads_started":2,"wal-append.queue_time_us":400231}
W20260129 03:00:06.270478 23199 consensus_meta.cc:287] T fcddab6bca684adf8ea032289a00f90e P 974b802141bd4f13beffef2aa03f88f9: Time spent flushing consensus metadata: real 0.985s user 0.018s sys 0.014s
W20260129 03:00:06.282481 22748 consensus_meta.cc:287] T 75be4bd6bd45474795fd472eff2953ac P 1c50de89a5a845668696dcb28c3badd6: Time spent flushing consensus metadata: real 0.954s user 0.015s sys 0.000s
I20260129 03:00:06.294212 22645 raft_consensus.cc:493] T c2633a6fc34c44579de348d208b0c01e P 197b880fa39d432a9b472845876bfaf0 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260129 03:00:06.288765 22932 raft_consensus.cc:2749] T 87d5fe48a44a408aaf03e894a67ecb2f P 7a9daf5d42a84615a6383cfec9a10608 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
W20260129 03:00:06.295037 23294 log.cc:927] Time spent T 4f8f3a38f6e24ba4857875181f447442 P 197b880fa39d432a9b472845876bfaf0: Append to log took a long time: real 0.203s user 0.000s sys 0.001s
I20260129 03:00:06.295332 23292 raft_consensus.cc:2749] T a8479d1fe0bd41e5946186e3a6ca8c7a P 1c50de89a5a845668696dcb28c3badd6 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260129 03:00:06.268239 22855 consensus_queue.cc:1059] T 18f30ebaa4174fd28c958e72d0cc7e24 P 1c50de89a5a845668696dcb28c3badd6 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false }, Status: INVALID_TERM, Last received: 6.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.071s
W20260129 03:00:06.296015 21101 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260129 03:00:06.297354 23293 log.cc:927] Time spent T 4408006eb5174de1896b037cf3f18457 P 974b802141bd4f13beffef2aa03f88f9: Append to log took a long time: real 0.090s user 0.000s sys 0.000s
I20260129 03:00:06.269320 23185 raft_consensus.cc:493] T 368953c9102643b2a21f99af36b51905 P 974b802141bd4f13beffef2aa03f88f9 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260129 03:00:06.300590 23085 leader_election.cc:290] T a0fcc58d6fb6473cafbd9d34d5195269 P 197b880fa39d432a9b472845876bfaf0 [CANDIDATE]: Term 5 election: Requested vote from peers 974b802141bd4f13beffef2aa03f88f9 (127.18.211.2:34517), 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3:37735)
I20260129 03:00:06.301255 21099 leader_election.cc:304] T 18f30ebaa4174fd28c958e72d0cc7e24 P 7a9daf5d42a84615a6383cfec9a10608 [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7a9daf5d42a84615a6383cfec9a10608; no voters: 197b880fa39d432a9b472845876bfaf0, 1c50de89a5a845668696dcb28c3badd6
I20260129 03:00:06.288587 22694 raft_consensus.cc:515] T c891d2bff4ce4568861bbc992e91fbb0 P 974b802141bd4f13beffef2aa03f88f9 [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
W20260129 03:00:06.322398 23260 consensus_meta.cc:287] T 913fc7d751ed4b7388d889aac30a5f9b P 1c50de89a5a845668696dcb28c3badd6: Time spent flushing consensus metadata: real 0.964s user 0.010s sys 0.003s
W20260129 03:00:06.278585 22975 consensus_meta.cc:287] T 0678b8b468bd43cdbc3e485900844821 P 974b802141bd4f13beffef2aa03f88f9: Time spent flushing consensus metadata: real 0.711s user 0.012s sys 0.007s
W20260129 03:00:06.332955 20407 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260129 03:00:06.352797 20407 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260129 03:00:06.353329 20407 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260129 03:00:06.277683 20553 leader_election.cc:304] T e4d18c35b0a7493e85d0a4b92956a48f P 1c50de89a5a845668696dcb28c3badd6 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1c50de89a5a845668696dcb28c3badd6; no voters: 197b880fa39d432a9b472845876bfaf0, 974b802141bd4f13beffef2aa03f88f9
I20260129 03:00:06.292640 20435 raft_consensus.cc:1240] T a609ac02fae84725ab743f95f51d33ce P 197b880fa39d432a9b472845876bfaf0 [term 8 FOLLOWER]: Rejecting Update request from peer 974b802141bd4f13beffef2aa03f88f9 for earlier term 6. Current term is 8. Ops: []
I20260129 03:00:06.335295 20508 raft_consensus.cc:2468] T cb4727386a3e48aca5d71cc6c611c197 P 974b802141bd4f13beffef2aa03f88f9 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1c50de89a5a845668696dcb28c3badd6 in term 5.
I20260129 03:00:06.335987 20432 raft_consensus.cc:2468] T a1a39dcd328e4268ae41b988c66b4a75 P 197b880fa39d432a9b472845876bfaf0 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 974b802141bd4f13beffef2aa03f88f9 in term 5.
W20260129 03:00:06.337925 20440 consensus_meta.cc:287] T 913fc7d751ed4b7388d889aac30a5f9b P 197b880fa39d432a9b472845876bfaf0: Time spent flushing consensus metadata: real 0.677s user 0.021s sys 0.006s
I20260129 03:00:06.268826 23145 raft_consensus.cc:515] T cedce331e1584ac8b636c9b8a72b2ba0 P 974b802141bd4f13beffef2aa03f88f9 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 03:00:06.338829 22645 raft_consensus.cc:515] T c2633a6fc34c44579de348d208b0c01e P 197b880fa39d432a9b472845876bfaf0 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 03:00:06.356503 23185 raft_consensus.cc:515] T 368953c9102643b2a21f99af36b51905 P 974b802141bd4f13beffef2aa03f88f9 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 03:00:06.358062 23145 leader_election.cc:290] T cedce331e1584ac8b636c9b8a72b2ba0 P 974b802141bd4f13beffef2aa03f88f9 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 197b880fa39d432a9b472845876bfaf0 (127.18.211.1:42027), 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3:37735)
I20260129 03:00:06.358287 22645 leader_election.cc:290] T c2633a6fc34c44579de348d208b0c01e P 197b880fa39d432a9b472845876bfaf0 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 974b802141bd4f13beffef2aa03f88f9 (127.18.211.2:34517), 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3:37735)
I20260129 03:00:06.358469 23185 leader_election.cc:290] T 368953c9102643b2a21f99af36b51905 P 974b802141bd4f13beffef2aa03f88f9 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 197b880fa39d432a9b472845876bfaf0 (127.18.211.1:42027), 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3:37735)
I20260129 03:00:06.361708 20589 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "45cc5f3d2688486d9b6dbe7e666dedde" candidate_uuid: "974b802141bd4f13beffef2aa03f88f9" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "1c50de89a5a845668696dcb28c3badd6"
I20260129 03:00:06.362530 20589 raft_consensus.cc:2393] T 45cc5f3d2688486d9b6dbe7e666dedde P 1c50de89a5a845668696dcb28c3badd6 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 974b802141bd4f13beffef2aa03f88f9 in current term 5: Already voted for candidate 1c50de89a5a845668696dcb28c3badd6 in this term.
W20260129 03:00:06.340267 20480 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.211.4:40439, user_credentials={real_user=slave}} blocked reactor thread for 99844.7us
I20260129 03:00:06.363642 20480 leader_election.cc:304] T c934d31efd3f4e5195605ddc354d14e1 P 974b802141bd4f13beffef2aa03f88f9 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7a9daf5d42a84615a6383cfec9a10608, 974b802141bd4f13beffef2aa03f88f9; no voters:
I20260129 03:00:06.344267 23086 raft_consensus.cc:515] T 2e58e9a5a8aa4cecafb464addb4f35ed P 197b880fa39d432a9b472845876bfaf0 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } }
I20260129 03:00:06.364274 20508 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a609ac02fae84725ab743f95f51d33ce" candidate_uuid: "197b880fa39d432a9b472845876bfaf0" candidate_term: 9 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "974b802141bd4f13beffef2aa03f88f9" is_pre_election: true
I20260129 03:00:06.365967 22855 raft_consensus.cc:3055] T 18f30ebaa4174fd28c958e72d0cc7e24 P 1c50de89a5a845668696dcb28c3badd6 [term 6 LEADER]: Stepping down as leader of term 6
I20260129 03:00:06.366075 23086 leader_election.cc:290] T 2e58e9a5a8aa4cecafb464addb4f35ed P 197b880fa39d432a9b472845876bfaf0 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3:37735), 7a9daf5d42a84615a6383cfec9a10608 (127.18.211.4:40439)
I20260129 03:00:06.366567 22855 raft_consensus.cc:740] T 18f30ebaa4174fd28c958e72d0cc7e24 P 1c50de89a5a845668696dcb28c3badd6 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 1c50de89a5a845668696dcb28c3badd6, State: Running, Role: LEADER
I20260129 03:00:06.367169 23260 raft_consensus.cc:2749] T 913fc7d751ed4b7388d889aac30a5f9b P 1c50de89a5a845668696dcb28c3badd6 [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
W20260129 03:00:06.342628 20439 consensus_meta.cc:287] T 0340229d521b432580ae926f6062e3f7 P 197b880fa39d432a9b472845876bfaf0: Time spent flushing consensus metadata: real 0.760s user 0.008s sys 0.013s
I20260129 03:00:06.295486 23034 leader_election.cc:290] T 7a9a69cd82724b1faed1d10aa091fc4a P 974b802141bd4f13beffef2aa03f88f9 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 197b880fa39d432a9b472845876bfaf0 (127.18.211.1:42027), 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3:37735)
I20260129 03:00:06.301637 20513 raft_consensus.cc:1275] T 75be4bd6bd45474795fd472eff2953ac P 974b802141bd4f13beffef2aa03f88f9 [term 10 FOLLOWER]: Refusing update from remote peer 197b880fa39d432a9b472845876bfaf0: Log matching property violated. Preceding OpId in replica: term: 9 index: 6. Preceding OpId from leader: term: 10 index: 7. (index mismatch)
I20260129 03:00:06.368295 20439 raft_consensus.cc:2468] T 0340229d521b432580ae926f6062e3f7 P 197b880fa39d432a9b472845876bfaf0 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 974b802141bd4f13beffef2aa03f88f9 in term 7.
I20260129 03:00:06.369345 20513 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a0fcc58d6fb6473cafbd9d34d5195269" candidate_uuid: "197b880fa39d432a9b472845876bfaf0" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "974b802141bd4f13beffef2aa03f88f9"
I20260129 03:00:06.369879 20513 raft_consensus.cc:3060] T a0fcc58d6fb6473cafbd9d34d5195269 P 974b802141bd4f13beffef2aa03f88f9 [term 4 FOLLOWER]: Advancing to term 5
I20260129 03:00:06.370071 20509 raft_consensus.cc:1240] T c891d2bff4ce4568861bbc992e91fbb0 P 974b802141bd4f13beffef2aa03f88f9 [term 8 FOLLOWER]: Rejecting Update request from peer 197b880fa39d432a9b472845876bfaf0 for earlier term 7. Current term is 8. Ops: []
I20260129 03:00:06.368314 23216 leader_election.cc:290] T 4f5a77e94fef4b16a2ba7b067c8b144a P 974b802141bd4f13beffef2aa03f88f9 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3:37735), 7a9daf5d42a84615a6383cfec9a10608 (127.18.211.4:40439)
I20260129 03:00:06.369889 22694 leader_election.cc:290] T c891d2bff4ce4568861bbc992e91fbb0 P 974b802141bd4f13beffef2aa03f88f9 [CANDIDATE]: Term 8 election: Requested vote from peers 197b880fa39d432a9b472845876bfaf0 (127.18.211.1:42027), 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3:37735)
I20260129 03:00:06.347429 23026 raft_consensus.cc:493] T c891d2bff4ce4568861bbc992e91fbb0 P 1c50de89a5a845668696dcb28c3badd6 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260129 03:00:06.372700 22932 raft_consensus.cc:2749] T 18f30ebaa4174fd28c958e72d0cc7e24 P 7a9daf5d42a84615a6383cfec9a10608 [term 7 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
W20260129 03:00:06.372890 21099 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.211.1:42027, user_credentials={real_user=slave}} blocked reactor thread for 71722.9us
I20260129 03:00:06.373733 21099 leader_election.cc:304] T 1299723a4476420ab02fd8bfd7aa0645 P 7a9daf5d42a84615a6383cfec9a10608 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 197b880fa39d432a9b472845876bfaf0, 7a9daf5d42a84615a6383cfec9a10608; no voters:
I20260129 03:00:06.374635 20434 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.GetConsensusState from 127.0.0.1:55666 (request call id 1089) took 3116 ms. Trace:
I20260129 03:00:06.375013 20434 rpcz_store.cc:276] 0129 03:00:03.257914 (+ 0us) service_pool.cc:168] Inserting onto call queue
0129 03:00:03.258054 (+ 140us) service_pool.cc:225] Handling call
0129 03:00:06.373655 (+3115601us) spinlock_profiling.cc:243] Waited 1.7 s on lock 0x7b5400330254. stack: 00007f8fe044e373 00007f8fe044e270 00007f8fdfda0574 00007f8fe9a5c6b4 00007f8fe9a5c649 00007f8fe34980c0 00007f8fe3536bba 00007f8fe84e758e 00007f8fe1a86437 00007f8fe1a863c2 00007f8fe1a862f1 00007f8fe1a8626c 00007f8fe1a85522 00007f8fe186d34c 00007f8fe186c786 00007f8fe186c12f
0129 03:00:06.374430 (+ 775us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":3918584576}
W20260129 03:00:06.376049 20405 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.211.3:37735, user_credentials={real_user=slave}} blocked reactor thread for 107507us
I20260129 03:00:06.376235 20440 raft_consensus.cc:2468] T 913fc7d751ed4b7388d889aac30a5f9b P 197b880fa39d432a9b472845876bfaf0 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 974b802141bd4f13beffef2aa03f88f9 in term 5.
I20260129 03:00:06.333709 20335 catalog_manager.cc:5654] T 00e79d8f25a34676b1d866eb2068a2b2 P 7a9daf5d42a84615a6383cfec9a10608 reported cstate change: term changed from 3 to 6, leader changed from 197b880fa39d432a9b472845876bfaf0 (127.18.211.1) to 7a9daf5d42a84615a6383cfec9a10608 (127.18.211.4). New cstate: current_term: 6 leader_uuid: "7a9daf5d42a84615a6383cfec9a10608" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260129 03:00:06.368016 22855 consensus_queue.cc:260] T 18f30ebaa4174fd28c958e72d0cc7e24 P 1c50de89a5a845668696dcb28c3badd6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 6.7, Last appended by leader: 7, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } }
I20260129 03:00:06.375025 22831 raft_consensus.cc:2804] T 1299723a4476420ab02fd8bfd7aa0645 P 7a9daf5d42a84615a6383cfec9a10608 [term 9 FOLLOWER]: Leader pre-election won for term 10
I20260129 03:00:06.380589 22831 raft_consensus.cc:493] T 1299723a4476420ab02fd8bfd7aa0645 P 7a9daf5d42a84615a6383cfec9a10608 [term 9 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260129 03:00:06.381054 22831 raft_consensus.cc:3060] T 1299723a4476420ab02fd8bfd7aa0645 P 7a9daf5d42a84615a6383cfec9a10608 [term 9 FOLLOWER]: Advancing to term 10
I20260129 03:00:06.367200 20583 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e7d83d4ccf684e76acb29a0dc844e1b8" candidate_uuid: "974b802141bd4f13beffef2aa03f88f9" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "1c50de89a5a845668696dcb28c3badd6" is_pre_election: true
I20260129 03:00:06.392311 20590 raft_consensus.cc:1275] T 75b0b6e82ddc4f92b1fe1d94f9a0b73b P 1c50de89a5a845668696dcb28c3badd6 [term 4 FOLLOWER]: Refusing update from remote peer 7a9daf5d42a84615a6383cfec9a10608: Log matching property violated. Preceding OpId in replica: term: 3 index: 6. Preceding OpId from leader: term: 4 index: 7. (index mismatch)
I20260129 03:00:06.372066 23026 raft_consensus.cc:515] T c891d2bff4ce4568861bbc992e91fbb0 P 1c50de89a5a845668696dcb28c3badd6 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 03:00:06.348315 23147 consensus_queue.cc:237] T c2633a6fc34c44579de348d208b0c01e P 1c50de89a5a845668696dcb28c3badd6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 03:00:06.367744 23199 raft_consensus.cc:2804] T c934d31efd3f4e5195605ddc354d14e1 P 974b802141bd4f13beffef2aa03f88f9 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260129 03:00:06.392218 20516 raft_consensus.cc:2410] T 0678b8b468bd43cdbc3e485900844821 P 974b802141bd4f13beffef2aa03f88f9 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 197b880fa39d432a9b472845876bfaf0 for term 9 because replica has last-logged OpId of term: 7 index: 8, which is greater than that of the candidate, which has last-logged OpId of term: 5 index: 7.
I20260129 03:00:06.393355 20590 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:48526 (request call id 358) took 1255 ms. Trace:
W20260129 03:00:06.349745 20479 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.211.1:42027, user_credentials={real_user=slave}} blocked reactor thread for 66842.1us
I20260129 03:00:06.411692 20508 raft_consensus.cc:2468] T a609ac02fae84725ab743f95f51d33ce P 974b802141bd4f13beffef2aa03f88f9 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 197b880fa39d432a9b472845876bfaf0 in term 7.
I20260129 03:00:06.408448 20335 catalog_manager.cc:5654] T 4f5a77e94fef4b16a2ba7b067c8b144a P 7a9daf5d42a84615a6383cfec9a10608 reported cstate change: term changed from 1 to 3, leader changed from 974b802141bd4f13beffef2aa03f88f9 (127.18.211.2) to 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3). New cstate: current_term: 3 leader_uuid: "1c50de89a5a845668696dcb28c3badd6" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } } }
I20260129 03:00:06.412150 20513 raft_consensus.cc:2468] T a0fcc58d6fb6473cafbd9d34d5195269 P 974b802141bd4f13beffef2aa03f88f9 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 197b880fa39d432a9b472845876bfaf0 in term 5.
I20260129 03:00:06.412194 20590 rpcz_store.cc:276] 0129 03:00:05.137736 (+ 0us) service_pool.cc:168] Inserting onto call queue
0129 03:00:05.298105 (+160369us) service_pool.cc:225] Handling call
0129 03:00:05.322165 (+ 24060us) raft_consensus.cc:1471] Updating replica for 0 ops
0129 03:00:05.971995 (+649830us) spinlock_profiling.cc:243] Waited 72.5 ms on lock 0x7b54001a84a0. stack: 00007f8fe044e373 00007f8fe044e270 00007f8fdfda0574 00007f8fe9a5c6b4 00007f8fe9a5c649 00007f8fe34975a1 00007f8fe34a5df0 00007f8fe352fe40 00007f8fe3530eb1 00007f8fe352da0f 00007f8fe352d4be 00007f8fe84e1f49 00007f8fe1a6ed34 00007f8fe1a6ecc2 00007f8fe1a6ebf1 00007f8fe1a6eb6c
0129 03:00:06.392894 (+420899us) raft_consensus.cc:1697] Filling consensus response to leader.
0129 03:00:06.393278 (+ 384us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":166765824}
I20260129 03:00:06.433482 20513 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c934d31efd3f4e5195605ddc354d14e1" candidate_uuid: "7a9daf5d42a84615a6383cfec9a10608" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "974b802141bd4f13beffef2aa03f88f9" is_pre_election: true
I20260129 03:00:06.434856 20405 leader_election.cc:304] T fcddab6bca684adf8ea032289a00f90e P 197b880fa39d432a9b472845876bfaf0 [CANDIDATE]: Term 9 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 197b880fa39d432a9b472845876bfaf0; no voters: 1c50de89a5a845668696dcb28c3badd6, 974b802141bd4f13beffef2aa03f88f9
W20260129 03:00:06.395588 23059 consensus_meta.cc:287] T a644c02b16af47679824ccd060ac10fd P 7a9daf5d42a84615a6383cfec9a10608: Time spent flushing consensus metadata: real 0.649s user 0.007s sys 0.012s
I20260129 03:00:06.435097 20583 raft_consensus.cc:2468] T e7d83d4ccf684e76acb29a0dc844e1b8 P 1c50de89a5a845668696dcb28c3badd6 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 974b802141bd4f13beffef2aa03f88f9 in term 6.
W20260129 03:00:06.396517 23250 raft_consensus.cc:936] Couldn't advance consensus term.: Illegal state: Can't advance term to: 4 current term: 4 is higher.
W20260129 03:00:06.418701 23141 consensus_meta.cc:287] T 4f5a77e94fef4b16a2ba7b067c8b144a P 7a9daf5d42a84615a6383cfec9a10608: Time spent flushing consensus metadata: real 0.838s user 0.012s sys 0.004s
I20260129 03:00:06.395509 23260 raft_consensus.cc:2749] T e4d18c35b0a7493e85d0a4b92956a48f P 1c50de89a5a845668696dcb28c3badd6 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260129 03:00:06.441998 20407 leader_election.cc:304] T a0fcc58d6fb6473cafbd9d34d5195269 P 197b880fa39d432a9b472845876bfaf0 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 197b880fa39d432a9b472845876bfaf0, 974b802141bd4f13beffef2aa03f88f9; no voters:
I20260129 03:00:06.443933 20589 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "913fc7d751ed4b7388d889aac30a5f9b" candidate_uuid: "974b802141bd4f13beffef2aa03f88f9" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "1c50de89a5a845668696dcb28c3badd6"
I20260129 03:00:06.448577 21134 raft_consensus.cc:2410] T a644c02b16af47679824ccd060ac10fd P 7a9daf5d42a84615a6383cfec9a10608 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 974b802141bd4f13beffef2aa03f88f9 for term 6 because replica has last-logged OpId of term: 4 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 4.
I20260129 03:00:06.449067 21128 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1299723a4476420ab02fd8bfd7aa0645" candidate_uuid: "974b802141bd4f13beffef2aa03f88f9" candidate_term: 9 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "7a9daf5d42a84615a6383cfec9a10608" is_pre_election: true
I20260129 03:00:06.450454 23134 raft_consensus.cc:3055] T 8055232cf0004d40a85f3bf983fe00a9 P 197b880fa39d432a9b472845876bfaf0 [term 4 LEADER]: Stepping down as leader of term 4
W20260129 03:00:06.452780 23296 log.cc:927] Time spent T 75be4bd6bd45474795fd472eff2953ac P 197b880fa39d432a9b472845876bfaf0: Append to log took a long time: real 0.055s user 0.000s sys 0.000s
I20260129 03:00:06.418171 23199 raft_consensus.cc:493] T c934d31efd3f4e5195605ddc354d14e1 P 974b802141bd4f13beffef2aa03f88f9 [term 5 FOLLOWER]: Starting leader election (detected failure of leader 197b880fa39d432a9b472845876bfaf0)
I20260129 03:00:06.465555 21130 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4f5a77e94fef4b16a2ba7b067c8b144a" candidate_uuid: "974b802141bd4f13beffef2aa03f88f9" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "7a9daf5d42a84615a6383cfec9a10608" is_pre_election: true
I20260129 03:00:06.466131 23100 raft_consensus.cc:493] T a644c02b16af47679824ccd060ac10fd P 7a9daf5d42a84615a6383cfec9a10608 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260129 03:00:06.472476 22831 raft_consensus.cc:515] T 1299723a4476420ab02fd8bfd7aa0645 P 7a9daf5d42a84615a6383cfec9a10608 [term 10 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } }
I20260129 03:00:06.475329 22831 leader_election.cc:290] T 1299723a4476420ab02fd8bfd7aa0645 P 7a9daf5d42a84615a6383cfec9a10608 [CANDIDATE]: Term 10 election: Requested vote from peers 197b880fa39d432a9b472845876bfaf0 (127.18.211.1:42027), 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3:37735)
I20260129 03:00:06.486840 20513 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0678b8b468bd43cdbc3e485900844821" candidate_uuid: "7a9daf5d42a84615a6383cfec9a10608" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "974b802141bd4f13beffef2aa03f88f9" is_pre_election: true
I20260129 03:00:06.432096 23185 consensus_queue.cc:1048] T a8479d1fe0bd41e5946186e3a6ca8c7a P 974b802141bd4f13beffef2aa03f88f9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.036s
I20260129 03:00:06.462105 20335 catalog_manager.cc:5654] T 78aa39096868415d9afc2ffcb248e198 P 7a9daf5d42a84615a6383cfec9a10608 reported cstate change: term changed from 3 to 5. New cstate: current_term: 5 leader_uuid: "1c50de89a5a845668696dcb28c3badd6" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } } }
I20260129 03:00:06.462929 22691 raft_consensus.cc:2804] T a0fcc58d6fb6473cafbd9d34d5195269 P 197b880fa39d432a9b472845876bfaf0 [term 5 FOLLOWER]: Leader election won for term 5
I20260129 03:00:06.464850 23134 raft_consensus.cc:740] T 8055232cf0004d40a85f3bf983fe00a9 P 197b880fa39d432a9b472845876bfaf0 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 197b880fa39d432a9b472845876bfaf0, State: Running, Role: LEADER
I20260129 03:00:06.469794 23199 raft_consensus.cc:3060] T c934d31efd3f4e5195605ddc354d14e1 P 974b802141bd4f13beffef2aa03f88f9 [term 5 FOLLOWER]: Advancing to term 6
W20260129 03:00:06.519716 20553 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.211.1:42027, user_credentials={real_user=slave}} blocked reactor thread for 246464us
I20260129 03:00:06.522887 20435 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1299723a4476420ab02fd8bfd7aa0645" candidate_uuid: "7a9daf5d42a84615a6383cfec9a10608" candidate_term: 10 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "197b880fa39d432a9b472845876bfaf0"
I20260129 03:00:06.524591 20435 raft_consensus.cc:3060] T 1299723a4476420ab02fd8bfd7aa0645 P 197b880fa39d432a9b472845876bfaf0 [term 9 FOLLOWER]: Advancing to term 10
I20260129 03:00:06.456998 23086 consensus_queue.cc:1048] T 75be4bd6bd45474795fd472eff2953ac P 197b880fa39d432a9b472845876bfaf0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 5, Time since last communication: 0.057s
I20260129 03:00:06.489455 21128 raft_consensus.cc:1813] T 1299723a4476420ab02fd8bfd7aa0645 P 7a9daf5d42a84615a6383cfec9a10608 [term 10 FOLLOWER]: Handling vote request from an unknown peer 974b802141bd4f13beffef2aa03f88f9
I20260129 03:00:06.526407 20335 catalog_manager.cc:5654] T a609ac02fae84725ab743f95f51d33ce P 7a9daf5d42a84615a6383cfec9a10608 reported cstate change: term changed from 4 to 6, leader changed from 7a9daf5d42a84615a6383cfec9a10608 (127.18.211.4) to 974b802141bd4f13beffef2aa03f88f9 (127.18.211.2). New cstate: current_term: 6 leader_uuid: "974b802141bd4f13beffef2aa03f88f9" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } } }
I20260129 03:00:06.491972 22855 raft_consensus.cc:3060] T 18f30ebaa4174fd28c958e72d0cc7e24 P 1c50de89a5a845668696dcb28c3badd6 [term 6 FOLLOWER]: Advancing to term 7
I20260129 03:00:06.491600 23100 raft_consensus.cc:515] T a644c02b16af47679824ccd060ac10fd P 7a9daf5d42a84615a6383cfec9a10608 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } }
W20260129 03:00:06.533627 20481 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260129 03:00:06.531066 20335 catalog_manager.cc:5654] T 3ccac82a01a2445fa5b0ce6c0fcf2295 P 7a9daf5d42a84615a6383cfec9a10608 reported cstate change: term changed from 1 to 4, leader changed from 197b880fa39d432a9b472845876bfaf0 (127.18.211.1) to 7a9daf5d42a84615a6383cfec9a10608 (127.18.211.4). New cstate: current_term: 4 leader_uuid: "7a9daf5d42a84615a6383cfec9a10608" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260129 03:00:06.533860 20513 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c2633a6fc34c44579de348d208b0c01e" candidate_uuid: "197b880fa39d432a9b472845876bfaf0" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "974b802141bd4f13beffef2aa03f88f9" is_pre_election: true
I20260129 03:00:06.534937 20513 raft_consensus.cc:2393] T c2633a6fc34c44579de348d208b0c01e P 974b802141bd4f13beffef2aa03f88f9 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 197b880fa39d432a9b472845876bfaf0 in current term 4: Already voted for candidate 1c50de89a5a845668696dcb28c3badd6 in this term.
I20260129 03:00:06.462438 23141 raft_consensus.cc:515] T 4f5a77e94fef4b16a2ba7b067c8b144a P 7a9daf5d42a84615a6383cfec9a10608 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } }
W20260129 03:00:06.538339 20556 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.211.2:34517, user_credentials={real_user=slave}} blocked reactor thread for 466730us
W20260129 03:00:06.543322 20405 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.211.3:37735, user_credentials={real_user=slave}} blocked reactor thread for 151191us
I20260129 03:00:06.547394 20582 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:48526 (request call id 353) took 1974 ms. Trace:
I20260129 03:00:06.540505 20335 catalog_manager.cc:5654] T cb0b3f259ebc4310927338a8740e87b8 P 7a9daf5d42a84615a6383cfec9a10608 reported cstate change: term changed from 3 to 5, leader changed from 974b802141bd4f13beffef2aa03f88f9 (127.18.211.2) to 7a9daf5d42a84615a6383cfec9a10608 (127.18.211.4). New cstate: current_term: 5 leader_uuid: "7a9daf5d42a84615a6383cfec9a10608" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
W20260129 03:00:06.543079 20479 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.211.1:42027, user_credentials={real_user=slave}} blocked reactor thread for 51884.8us
I20260129 03:00:06.524959 22691 raft_consensus.cc:697] T a0fcc58d6fb6473cafbd9d34d5195269 P 197b880fa39d432a9b472845876bfaf0 [term 5 LEADER]: Becoming Leader. State: Replica: 197b880fa39d432a9b472845876bfaf0, State: Running, Role: LEADER
I20260129 03:00:06.548396 20479 leader_election.cc:304] T 7d9494f5584d4d5db8b9adf1c8fc888b P 974b802141bd4f13beffef2aa03f88f9 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 974b802141bd4f13beffef2aa03f88f9; no voters: 197b880fa39d432a9b472845876bfaf0, 1c50de89a5a845668696dcb28c3badd6
I20260129 03:00:06.548705 20556 leader_election.cc:304] T cb4727386a3e48aca5d71cc6c611c197 P 1c50de89a5a845668696dcb28c3badd6 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1c50de89a5a845668696dcb28c3badd6, 974b802141bd4f13beffef2aa03f88f9; no voters:
I20260129 03:00:06.550525 20513 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "add8d4923d1a489e86882c3a25692b95" candidate_uuid: "7a9daf5d42a84615a6383cfec9a10608" candidate_term: 7 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "974b802141bd4f13beffef2aa03f88f9" is_pre_election: true
I20260129 03:00:06.553310 20513 raft_consensus.cc:2468] T add8d4923d1a489e86882c3a25692b95 P 974b802141bd4f13beffef2aa03f88f9 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7a9daf5d42a84615a6383cfec9a10608 in term 6.
I20260129 03:00:06.552690 20335 catalog_manager.cc:5654] T 2e58e9a5a8aa4cecafb464addb4f35ed P 7a9daf5d42a84615a6383cfec9a10608 reported cstate change: term changed from 2 to 4. New cstate: current_term: 4 leader_uuid: "7a9daf5d42a84615a6383cfec9a10608" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260129 03:00:06.537257 22694 raft_consensus.cc:493] T fcddab6bca684adf8ea032289a00f90e P 974b802141bd4f13beffef2aa03f88f9 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260129 03:00:06.544840 23087 raft_consensus.cc:2749] T fcddab6bca684adf8ea032289a00f90e P 197b880fa39d432a9b472845876bfaf0 [term 9 FOLLOWER]: Leader election lost for term 9. Reason: could not achieve majority
I20260129 03:00:06.551303 21128 raft_consensus.cc:2368] T 1299723a4476420ab02fd8bfd7aa0645 P 7a9daf5d42a84615a6383cfec9a10608 [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 974b802141bd4f13beffef2aa03f88f9 for earlier term 9. Current term is 10.
I20260129 03:00:06.551196 21130 raft_consensus.cc:2393] T 4f5a77e94fef4b16a2ba7b067c8b144a P 7a9daf5d42a84615a6383cfec9a10608 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 974b802141bd4f13beffef2aa03f88f9 in current term 5: Already voted for candidate 7a9daf5d42a84615a6383cfec9a10608 in this term.
I20260129 03:00:06.566805 23207 consensus_queue.cc:260] T 0f20af7fbe5d4db89a6f17948d98912c P 974b802141bd4f13beffef2aa03f88f9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 6.4, Last appended by leader: 4, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 03:00:06.567821 23207 raft_consensus.cc:3060] T 0f20af7fbe5d4db89a6f17948d98912c P 974b802141bd4f13beffef2aa03f88f9 [term 6 FOLLOWER]: Advancing to term 7
W20260129 03:00:06.577410 20556 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260129 03:00:06.578392 20556 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260129 03:00:06.580170 23134 consensus_queue.cc:260] T 8055232cf0004d40a85f3bf983fe00a9 P 197b880fa39d432a9b472845876bfaf0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 4.3, Last appended by leader: 3, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 03:00:06.581327 23134 raft_consensus.cc:3060] T 8055232cf0004d40a85f3bf983fe00a9 P 197b880fa39d432a9b472845876bfaf0 [term 4 FOLLOWER]: Advancing to term 5
I20260129 03:00:06.582187 23251 raft_consensus.cc:3060] T add8d4923d1a489e86882c3a25692b95 P 1c50de89a5a845668696dcb28c3badd6 [term 5 FOLLOWER]: Advancing to term 6
I20260129 03:00:06.586794 20405 leader_election.cc:304] T 0340229d521b432580ae926f6062e3f7 P 197b880fa39d432a9b472845876bfaf0 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 197b880fa39d432a9b472845876bfaf0, 1c50de89a5a845668696dcb28c3badd6; no voters: 974b802141bd4f13beffef2aa03f88f9
I20260129 03:00:06.588671 23087 raft_consensus.cc:2764] T 0340229d521b432580ae926f6062e3f7 P 197b880fa39d432a9b472845876bfaf0 [term 7 FOLLOWER]: Leader pre-election decision vote started in defunct term 5: won
I20260129 03:00:06.589129 20509 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57600 (request call id 1889) took 1222 ms. Trace:
I20260129 03:00:06.589569 20509 rpcz_store.cc:276] 0129 03:00:05.366478 (+ 0us) service_pool.cc:168] Inserting onto call queue
0129 03:00:05.366618 (+ 140us) service_pool.cc:225] Handling call
0129 03:00:05.366961 (+ 343us) raft_consensus.cc:1471] Updating replica for 0 ops
0129 03:00:06.370466 (+1003505us) raft_consensus.cc:1697] Filling consensus response to leader.
0129 03:00:06.588789 (+218323us) spinlock_profiling.cc:243] Waited 1 s on lock 0x7b5400316854. stack: 00007f8fe044e373 00007f8fe044e270 00007f8fdfda0574 00007f8fe9a5c6b4 00007f8fe9a5c649 00007f8fe34975a1 00007f8fe352ea81 00007f8fe352d4be 00007f8fe84e1f49 00007f8fe1a6ed34 00007f8fe1a6ecc2 00007f8fe1a6ebf1 00007f8fe1a6eb6c 00007f8fe1a6de22 00007f8fe186d34c 00007f8fe186c786
0129 03:00:06.589029 (+ 240us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2306523520}
I20260129 03:00:06.590947 20509 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "505580c93c694bbb8ea96fdf9b17ab46" candidate_uuid: "1c50de89a5a845668696dcb28c3badd6" candidate_term: 7 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "974b802141bd4f13beffef2aa03f88f9"
I20260129 03:00:06.591531 20509 raft_consensus.cc:3060] T 505580c93c694bbb8ea96fdf9b17ab46 P 974b802141bd4f13beffef2aa03f88f9 [term 6 FOLLOWER]: Advancing to term 7
W20260129 03:00:06.591888 23291 log.cc:927] Time spent T 7a9a69cd82724b1faed1d10aa091fc4a P 197b880fa39d432a9b472845876bfaf0: Append to log took a long time: real 0.381s user 0.004s sys 0.000s
I20260129 03:00:06.549918 23262 raft_consensus.cc:493] T 36ff33be666a4c82954731691dbcf327 P 1c50de89a5a845668696dcb28c3badd6 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 974b802141bd4f13beffef2aa03f88f9)
I20260129 03:00:06.594746 23178 leader_election.cc:290] T 7d9494f5584d4d5db8b9adf1c8fc888b P 197b880fa39d432a9b472845876bfaf0 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 974b802141bd4f13beffef2aa03f88f9 (127.18.211.2:34517), 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3:37735)
I20260129 03:00:06.594928 23262 raft_consensus.cc:515] T 36ff33be666a4c82954731691dbcf327 P 1c50de89a5a845668696dcb28c3badd6 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 03:00:06.587083 20581 raft_consensus.cc:3060] T 3e2d15f8640b407195210c476e09df91 P 1c50de89a5a845668696dcb28c3badd6 [term 3 FOLLOWER]: Advancing to term 4
I20260129 03:00:06.597007 23262 leader_election.cc:290] T 36ff33be666a4c82954731691dbcf327 P 1c50de89a5a845668696dcb28c3badd6 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 197b880fa39d432a9b472845876bfaf0 (127.18.211.1:42027), 974b802141bd4f13beffef2aa03f88f9 (127.18.211.2:34517)
I20260129 03:00:06.588553 20553 leader_election.cc:304] T 1ce4ddf5b10e4ae8b193b13b2e5936ee P 1c50de89a5a845668696dcb28c3badd6 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1c50de89a5a845668696dcb28c3badd6; no voters: 197b880fa39d432a9b472845876bfaf0, 974b802141bd4f13beffef2aa03f88f9
I20260129 03:00:06.600520 20513 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3906b080a5b34a74b8b64128635e8997" candidate_uuid: "1c50de89a5a845668696dcb28c3badd6" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "974b802141bd4f13beffef2aa03f88f9" is_pre_election: true
I20260129 03:00:06.551157 23282 raft_consensus.cc:2749] T 96609ac68bdf4993a12e1c23355ca013 P 1c50de89a5a845668696dcb28c3badd6 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260129 03:00:06.552237 23216 raft_consensus.cc:2749] T 7d9494f5584d4d5db8b9adf1c8fc888b P 974b802141bd4f13beffef2aa03f88f9 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260129 03:00:06.604615 20440 raft_consensus.cc:1275] T 42d27a28d89b409e8807ec0efd97af69 P 197b880fa39d432a9b472845876bfaf0 [term 8 FOLLOWER]: Refusing update from remote peer 7a9daf5d42a84615a6383cfec9a10608: Log matching property violated. Preceding OpId in replica: term: 6 index: 8. Preceding OpId from leader: term: 8 index: 9. (index mismatch)
I20260129 03:00:06.608623 20435 raft_consensus.cc:2468] T 1299723a4476420ab02fd8bfd7aa0645 P 197b880fa39d432a9b472845876bfaf0 [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7a9daf5d42a84615a6383cfec9a10608 in term 10.
W20260129 03:00:06.611311 20479 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.211.1:42027, user_credentials={real_user=slave}} blocked reactor thread for 62995.5us
W20260129 03:00:06.612839 20556 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260129 03:00:06.615757 21099 leader_election.cc:304] T 1299723a4476420ab02fd8bfd7aa0645 P 7a9daf5d42a84615a6383cfec9a10608 [CANDIDATE]: Term 10 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 197b880fa39d432a9b472845876bfaf0, 7a9daf5d42a84615a6383cfec9a10608; no voters:
I20260129 03:00:06.563705 23260 raft_consensus.cc:2804] T cb4727386a3e48aca5d71cc6c611c197 P 1c50de89a5a845668696dcb28c3badd6 [term 5 FOLLOWER]: Leader election won for term 5
W20260129 03:00:06.629477 20480 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.211.4:40439, user_credentials={real_user=slave}} blocked reactor thread for 137053us
I20260129 03:00:06.647639 20513 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4f5a77e94fef4b16a2ba7b067c8b144a" candidate_uuid: "7a9daf5d42a84615a6383cfec9a10608" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "974b802141bd4f13beffef2aa03f88f9" is_pre_election: true
I20260129 03:00:06.648342 20513 raft_consensus.cc:2468] T 4f5a77e94fef4b16a2ba7b067c8b144a P 974b802141bd4f13beffef2aa03f88f9 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7a9daf5d42a84615a6383cfec9a10608 in term 4.
I20260129 03:00:06.609916 23026 leader_election.cc:290] T c891d2bff4ce4568861bbc992e91fbb0 P 1c50de89a5a845668696dcb28c3badd6 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 197b880fa39d432a9b472845876bfaf0 (127.18.211.1:42027), 974b802141bd4f13beffef2aa03f88f9 (127.18.211.2:34517)
I20260129 03:00:06.561540 22694 raft_consensus.cc:515] T fcddab6bca684adf8ea032289a00f90e P 974b802141bd4f13beffef2aa03f88f9 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 03:00:06.645329 20405 leader_election.cc:304] T 0f20af7fbe5d4db89a6f17948d98912c P 197b880fa39d432a9b472845876bfaf0 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 197b880fa39d432a9b472845876bfaf0, 1c50de89a5a845668696dcb28c3badd6; no voters: 974b802141bd4f13beffef2aa03f88f9
I20260129 03:00:06.649266 20513 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "36ff33be666a4c82954731691dbcf327" candidate_uuid: "1c50de89a5a845668696dcb28c3badd6" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "974b802141bd4f13beffef2aa03f88f9" is_pre_election: true
W20260129 03:00:06.651648 20480 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260129 03:00:06.629352 23262 raft_consensus.cc:2749] T 1ce4ddf5b10e4ae8b193b13b2e5936ee P 1c50de89a5a845668696dcb28c3badd6 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260129 03:00:06.625913 23080 raft_consensus.cc:2804] T 1299723a4476420ab02fd8bfd7aa0645 P 7a9daf5d42a84615a6383cfec9a10608 [term 10 FOLLOWER]: Leader election won for term 10
W20260129 03:00:06.646857 23214 consensus_meta.cc:287] T 78aa39096868415d9afc2ffcb248e198 P 1c50de89a5a845668696dcb28c3badd6: Time spent flushing consensus metadata: real 0.981s user 0.017s sys 0.014s
I20260129 03:00:06.607259 23086 consensus_queue.cc:1059] T c891d2bff4ce4568861bbc992e91fbb0 P 197b880fa39d432a9b472845876bfaf0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 }, Status: INVALID_TERM, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.039s
W20260129 03:00:06.659495 20553 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.211.1:42027, user_credentials={real_user=slave}} blocked reactor thread for 71036.5us
I20260129 03:00:06.664971 23080 raft_consensus.cc:697] T 1299723a4476420ab02fd8bfd7aa0645 P 7a9daf5d42a84615a6383cfec9a10608 [term 10 LEADER]: Becoming Leader. State: Replica: 7a9daf5d42a84615a6383cfec9a10608, State: Running, Role: LEADER
I20260129 03:00:06.666029 23080 consensus_queue.cc:237] T 1299723a4476420ab02fd8bfd7aa0645 P 7a9daf5d42a84615a6383cfec9a10608 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 8.8, Last appended by leader: 8, Current term: 10, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } }
I20260129 03:00:06.666589 23089 raft_consensus.cc:2804] T 0f20af7fbe5d4db89a6f17948d98912c P 197b880fa39d432a9b472845876bfaf0 [term 7 FOLLOWER]: Leader election won for term 7
I20260129 03:00:06.667726 21102 leader_election.cc:304] T 00e79d8f25a34676b1d866eb2068a2b2 P 7a9daf5d42a84615a6383cfec9a10608 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7a9daf5d42a84615a6383cfec9a10608; no voters: 197b880fa39d432a9b472845876bfaf0, 974b802141bd4f13beffef2aa03f88f9
I20260129 03:00:06.668108 22691 consensus_queue.cc:237] T a0fcc58d6fb6473cafbd9d34d5195269 P 197b880fa39d432a9b472845876bfaf0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
W20260129 03:00:06.668790 21102 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.211.2:34517, user_credentials={real_user=slave}} blocked reactor thread for 730246us
I20260129 03:00:06.671504 20437 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55672 (request call id 1672) took 1146 ms. Trace:
I20260129 03:00:06.671955 23087 raft_consensus.cc:493] T cedce331e1584ac8b636c9b8a72b2ba0 P 197b880fa39d432a9b472845876bfaf0 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260129 03:00:06.681547 20437 rpcz_store.cc:276] 0129 03:00:05.525151 (+ 0us) service_pool.cc:168] Inserting onto call queue
0129 03:00:05.525390 (+ 239us) service_pool.cc:225] Handling call
0129 03:00:06.671323 (+1145933us) spinlock_profiling.cc:243] Waited 94.4 ms on lock 0x7b54002ae254. stack: 00007f8fe044e373 00007f8fe044e270 00007f8fdfda0574 00007f8fe9a5c6b4 00007f8fe9a5c649 00007f8fe34975a1 00007f8fe3531f7a 00007f8fe84e3647 00007f8fe1a73834 00007f8fe1a737c2 00007f8fe1a736f1 00007f8fe1a7366c 00007f8fe1a72922 00007f8fe186d34c 00007f8fe186c786 00007f8fe186c12f
0129 03:00:06.671426 (+ 103us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":217184640}
I20260129 03:00:06.681775 23087 raft_consensus.cc:515] T cedce331e1584ac8b636c9b8a72b2ba0 P 197b880fa39d432a9b472845876bfaf0 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 03:00:06.664309 20479 leader_election.cc:304] T e7d83d4ccf684e76acb29a0dc844e1b8 P 974b802141bd4f13beffef2aa03f88f9 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 197b880fa39d432a9b472845876bfaf0, 974b802141bd4f13beffef2aa03f88f9; no voters:
I20260129 03:00:06.672382 23089 raft_consensus.cc:697] T 0f20af7fbe5d4db89a6f17948d98912c P 197b880fa39d432a9b472845876bfaf0 [term 7 LEADER]: Becoming Leader. State: Replica: 197b880fa39d432a9b472845876bfaf0, State: Running, Role: LEADER
I20260129 03:00:06.671531 23086 raft_consensus.cc:3055] T c891d2bff4ce4568861bbc992e91fbb0 P 197b880fa39d432a9b472845876bfaf0 [term 7 LEADER]: Stepping down as leader of term 7
I20260129 03:00:06.677995 23035 raft_consensus.cc:3060] T 00e79d8f25a34676b1d866eb2068a2b2 P 7a9daf5d42a84615a6383cfec9a10608 [term 7 FOLLOWER]: Advancing to term 8
I20260129 03:00:06.678447 23107 consensus_queue.cc:1048] T 505580c93c694bbb8ea96fdf9b17ab46 P 7a9daf5d42a84615a6383cfec9a10608 [LEADER]: Connected to new peer: Peer: permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20260129 03:00:06.679895 22932 consensus_queue.cc:1048] T 3e2d15f8640b407195210c476e09df91 P 7a9daf5d42a84615a6383cfec9a10608 [LEADER]: Connected to new peer: Peer: permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260129 03:00:06.680267 23163 consensus_queue.cc:1048] T 4e420f90884f4796997b5ea41e49e411 P 7a9daf5d42a84615a6383cfec9a10608 [LEADER]: Connected to new peer: Peer: permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.001s
I20260129 03:00:06.679458 20581 raft_consensus.cc:1275] T 3e2d15f8640b407195210c476e09df91 P 1c50de89a5a845668696dcb28c3badd6 [term 4 FOLLOWER]: Refusing update from remote peer 7a9daf5d42a84615a6383cfec9a10608: Log matching property violated. Preceding OpId in replica: term: 3 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20260129 03:00:06.649941 23260 raft_consensus.cc:697] T cb4727386a3e48aca5d71cc6c611c197 P 1c50de89a5a845668696dcb28c3badd6 [term 5 LEADER]: Becoming Leader. State: Replica: 1c50de89a5a845668696dcb28c3badd6, State: Running, Role: LEADER
W20260129 03:00:06.683694 20407 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260129 03:00:06.688686 21102 leader_election.cc:304] T c934d31efd3f4e5195605ddc354d14e1 P 7a9daf5d42a84615a6383cfec9a10608 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7a9daf5d42a84615a6383cfec9a10608; no voters: 197b880fa39d432a9b472845876bfaf0, 974b802141bd4f13beffef2aa03f88f9
I20260129 03:00:06.689726 20513 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7d9494f5584d4d5db8b9adf1c8fc888b" candidate_uuid: "197b880fa39d432a9b472845876bfaf0" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "974b802141bd4f13beffef2aa03f88f9" is_pre_election: true
I20260129 03:00:06.632064 23255 consensus_queue.cc:1048] T 42d27a28d89b409e8807ec0efd97af69 P 7a9daf5d42a84615a6383cfec9a10608 [LEADER]: Connected to new peer: Peer: permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.032s
I20260129 03:00:06.695469 20553 leader_election.cc:304] T 0f20af7fbe5d4db89a6f17948d98912c P 1c50de89a5a845668696dcb28c3badd6 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1c50de89a5a845668696dcb28c3badd6; no voters: 197b880fa39d432a9b472845876bfaf0, 974b802141bd4f13beffef2aa03f88f9
W20260129 03:00:06.709888 20479 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.211.1:42027, user_credentials={real_user=slave}} blocked reactor thread for 48274.4us
I20260129 03:00:06.691248 23086 raft_consensus.cc:740] T c891d2bff4ce4568861bbc992e91fbb0 P 197b880fa39d432a9b472845876bfaf0 [term 7 LEADER]: Becoming Follower/Learner. State: Replica: 197b880fa39d432a9b472845876bfaf0, State: Running, Role: LEADER
I20260129 03:00:06.695218 23089 consensus_queue.cc:237] T 0f20af7fbe5d4db89a6f17948d98912c P 197b880fa39d432a9b472845876bfaf0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.4, Last appended by leader: 4, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 03:00:06.696838 23087 leader_election.cc:290] T cedce331e1584ac8b636c9b8a72b2ba0 P 197b880fa39d432a9b472845876bfaf0 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 974b802141bd4f13beffef2aa03f88f9 (127.18.211.2:34517), 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3:37735)
I20260129 03:00:06.698266 23100 leader_election.cc:290] T a644c02b16af47679824ccd060ac10fd P 7a9daf5d42a84615a6383cfec9a10608 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 974b802141bd4f13beffef2aa03f88f9 (127.18.211.2:34517), 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3:37735)
I20260129 03:00:06.699980 20515 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57584 (request call id 1449) took 1235 ms. Trace:
I20260129 03:00:06.714718 20433 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55672 (request call id 1678) took 1093 ms. Trace:
I20260129 03:00:06.715049 20433 rpcz_store.cc:276] 0129 03:00:05.621434 (+ 0us) service_pool.cc:168] Inserting onto call queue
0129 03:00:05.674982 (+ 53548us) service_pool.cc:225] Handling call
0129 03:00:06.069816 (+394834us) raft_consensus.cc:1471] Updating replica for 1 ops
0129 03:00:06.070887 (+ 1071us) raft_consensus.cc:1697] Filling consensus response to leader.
0129 03:00:06.714522 (+643635us) spinlock_profiling.cc:243] Waited 180 ms on lock 0x7b54002e1150. stack: 00007f8fe044e373 00007f8fe044e270 00007f8fdfda0574 00007f8fe9a5c6b4 00007f8fe9a5c649 00007f8fe34975a1 00007f8fe352d52d 00007f8fe84e1f49 00007f8fe1a6ed34 00007f8fe1a6ecc2 00007f8fe1a6ebf1 00007f8fe1a6eb6c 00007f8fe1a6de22 00007f8fe186d34c 00007f8fe186c786 00007f8fe186c12f
0129 03:00:06.714651 (+ 129us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":412976384}
W20260129 03:00:06.716161 22891 raft_consensus.cc:936] Couldn't advance consensus term.: Illegal state: Can't advance term to: 4 current term: 4 is higher.
I20260129 03:00:06.714262 20515 rpcz_store.cc:276] 0129 03:00:05.464952 (+ 0us) service_pool.cc:168] Inserting onto call queue
0129 03:00:06.007909 (+542957us) service_pool.cc:225] Handling call
0129 03:00:06.008331 (+ 422us) raft_consensus.cc:1471] Updating replica for 1 ops
0129 03:00:06.116343 (+108012us) raft_consensus.cc:1534] Early marking committed up to index 0
0129 03:00:06.116382 (+ 39us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0129 03:00:06.123586 (+ 7204us) log.cc:844] Serialized 26 byte log entry
0129 03:00:06.525830 (+402244us) raft_consensus.cc:1645] Marking committed up to 2
0129 03:00:06.525868 (+ 38us) raft_consensus.cc:1697] Filling consensus response to leader.
0129 03:00:06.525935 (+ 67us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0129 03:00:06.699741 (+173806us) raft_consensus.cc:1685] finished
0129 03:00:06.699761 (+ 20us) raft_consensus.cc:1691] UpdateReplicas() finished
0129 03:00:06.699916 (+ 155us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":48014,"raft.queue_time_us":16741,"raft.run_cpu_time_us":452,"raft.run_wall_time_us":83575,"thread_start_us":321077,"threads_started":1,"wal-append.queue_time_us":511548}
I20260129 03:00:06.717923 20515 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "cedce331e1584ac8b636c9b8a72b2ba0" candidate_uuid: "197b880fa39d432a9b472845876bfaf0" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "974b802141bd4f13beffef2aa03f88f9" is_pre_election: true
I20260129 03:00:06.718511 20515 raft_consensus.cc:2468] T cedce331e1584ac8b636c9b8a72b2ba0 P 974b802141bd4f13beffef2aa03f88f9 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 197b880fa39d432a9b472845876bfaf0 in term 7.
I20260129 03:00:06.699923 21136 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a644c02b16af47679824ccd060ac10fd" candidate_uuid: "974b802141bd4f13beffef2aa03f88f9" candidate_term: 6 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "7a9daf5d42a84615a6383cfec9a10608" is_pre_election: true
I20260129 03:00:06.719925 20479 leader_election.cc:304] T 76f713c3331540d2a5b76164103e9eb1 P 974b802141bd4f13beffef2aa03f88f9 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 197b880fa39d432a9b472845876bfaf0, 974b802141bd4f13beffef2aa03f88f9; no voters:
I20260129 03:00:06.702518 20512 raft_consensus.cc:1240] T 7d9494f5584d4d5db8b9adf1c8fc888b P 974b802141bd4f13beffef2aa03f88f9 [term 4 FOLLOWER]: Rejecting Update request from peer 197b880fa39d432a9b472845876bfaf0 for earlier term 3. Current term is 4. Ops: []
I20260129 03:00:06.702569 20507 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:48544 (request call id 295) took 2096 ms. Trace:
I20260129 03:00:06.547781 20582 rpcz_store.cc:276] 0129 03:00:04.573276 (+ 0us) service_pool.cc:168] Inserting onto call queue
0129 03:00:05.300770 (+727494us) service_pool.cc:225] Handling call
0129 03:00:05.301675 (+ 905us) raft_consensus.cc:1471] Updating replica for 0 ops
0129 03:00:05.301897 (+ 222us) raft_consensus.cc:1534] Early marking committed up to index 5
0129 03:00:06.146040 (+844143us) log.cc:844] Serialized 14 byte log entry
0129 03:00:06.542166 (+396126us) log.cc:844] Serialized 14 byte log entry
0129 03:00:06.547079 (+ 4913us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0129 03:00:06.547127 (+ 48us) raft_consensus.cc:1645] Marking committed up to 5
0129 03:00:06.547167 (+ 40us) raft_consensus.cc:1697] Filling consensus response to leader.
0129 03:00:06.547240 (+ 73us) raft_consensus.cc:1691] UpdateReplicas() finished
0129 03:00:06.547342 (+ 102us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":4679,"prepare.queue_time_us":8145,"raft.queue_time_us":107,"raft.run_cpu_time_us":178,"raft.run_wall_time_us":5793,"spinlock_wait_cycles":6170880,"thread_start_us":291702,"threads_started":1}
I20260129 03:00:06.706291 20581 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:48526 (request call id 350) took 2265 ms. Trace:
I20260129 03:00:06.720645 20507 rpcz_store.cc:276] 0129 03:00:04.606596 (+ 0us) service_pool.cc:168] Inserting onto call queue
0129 03:00:04.615456 (+ 8860us) service_pool.cc:225] Handling call
0129 03:00:06.702339 (+2086883us) spinlock_profiling.cc:243] Waited 1.05 s on lock 0x7b54000fc154. stack: 00007f8fe044e373 00007f8fe044e270 00007f8fdfda0574 00007f8fe9a5c6b4 00007f8fe9a5c649 00007f8fe34975a1 00007f8fe3531f7a 00007f8fe84e3647 00007f8fe1a73834 00007f8fe1a737c2 00007f8fe1a736f1 00007f8fe1a7366c 00007f8fe1a72922 00007f8fe186d34c 00007f8fe186c786 00007f8fe186c12f
0129 03:00:06.702506 (+ 167us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2425604992}
I20260129 03:00:06.721010 20407 leader_election.cc:304] T cedce331e1584ac8b636c9b8a72b2ba0 P 197b880fa39d432a9b472845876bfaf0 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 197b880fa39d432a9b472845876bfaf0, 974b802141bd4f13beffef2aa03f88f9; no voters:
I20260129 03:00:06.721081 20581 rpcz_store.cc:276] 0129 03:00:04.440453 (+ 0us) service_pool.cc:168] Inserting onto call queue
0129 03:00:05.298712 (+858259us) service_pool.cc:225] Handling call
0129 03:00:05.321308 (+ 22596us) raft_consensus.cc:1471] Updating replica for 0 ops
0129 03:00:05.956247 (+634939us) spinlock_profiling.cc:243] Waited 37.1 ms on lock 0x7b54001a1ba0. stack: 00007f8fe044e373 00007f8fe044e270 00007f8fdfda0574 00007f8fe9a5c6b4 00007f8fe9a5c649 00007f8fe34975a1 00007f8fe34a5df0 00007f8fe352fe40 00007f8fe3530eb1 00007f8fe352da0f 00007f8fe352d4be 00007f8fe84e1f49 00007f8fe1a6ed34 00007f8fe1a6ecc2 00007f8fe1a6ebf1 00007f8fe1a6eb6c
0129 03:00:06.586234 (+629987us) spinlock_profiling.cc:243] Waited 159 ms on lock 0x7b5800941cd0. stack: 00007f8fe044e373 00007f8fe044e270 00007f8fdfda0574 00007f8fe9a5c6b4 00007f8fe9a5c649 00007f8fe34975a1 00007f8fe348a0da 00007f8fe350c177 00007f8fe3524c67 00007f8fe352645e 00007f8fe35307c9 00007f8fe3531096 00007f8fe352da0f 00007f8fe352d4be 00007f8fe84e1f49 00007f8fe1a6ed34
0129 03:00:06.706004 (+119770us) raft_consensus.cc:1697] Filling consensus response to leader.
0129 03:00:06.706235 (+ 231us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":452102656}
I20260129 03:00:06.711007 21102 leader_election.cc:304] T 0678b8b468bd43cdbc3e485900844821 P 7a9daf5d42a84615a6383cfec9a10608 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7a9daf5d42a84615a6383cfec9a10608; no voters: 197b880fa39d432a9b472845876bfaf0, 974b802141bd4f13beffef2aa03f88f9
W20260129 03:00:06.712203 20405 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.211.3:37735, user_credentials={real_user=slave}} blocked reactor thread for 78906.2us
I20260129 03:00:06.727559 21102 leader_election.cc:304] T add8d4923d1a489e86882c3a25692b95 P 7a9daf5d42a84615a6383cfec9a10608 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 7a9daf5d42a84615a6383cfec9a10608, 974b802141bd4f13beffef2aa03f88f9; no voters: 1c50de89a5a845668696dcb28c3badd6
I20260129 03:00:06.728287 20479 leader_election.cc:304] T febffe4a8b394368b69b1d164308fb9a P 974b802141bd4f13beffef2aa03f88f9 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 197b880fa39d432a9b472845876bfaf0, 974b802141bd4f13beffef2aa03f88f9; no voters: 7a9daf5d42a84615a6383cfec9a10608
I20260129 03:00:06.728070 23182 raft_consensus.cc:2749] T c934d31efd3f4e5195605ddc354d14e1 P 7a9daf5d42a84615a6383cfec9a10608 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260129 03:00:06.720376 23141 leader_election.cc:290] T 4f5a77e94fef4b16a2ba7b067c8b144a P 7a9daf5d42a84615a6383cfec9a10608 [CANDIDATE]: Term 5 election: Requested vote from peers 974b802141bd4f13beffef2aa03f88f9 (127.18.211.2:34517), 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3:37735)
I20260129 03:00:06.737630 20405 leader_election.cc:304] T a5a05fbcbbcf4916bbc01fd1a5b5e90d P 197b880fa39d432a9b472845876bfaf0 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 197b880fa39d432a9b472845876bfaf0; no voters: 1c50de89a5a845668696dcb28c3badd6, 974b802141bd4f13beffef2aa03f88f9
I20260129 03:00:06.738842 20507 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4f5a77e94fef4b16a2ba7b067c8b144a" candidate_uuid: "7a9daf5d42a84615a6383cfec9a10608" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "974b802141bd4f13beffef2aa03f88f9"
I20260129 03:00:06.738969 20515 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c891d2bff4ce4568861bbc992e91fbb0" candidate_uuid: "1c50de89a5a845668696dcb28c3badd6" candidate_term: 8 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "974b802141bd4f13beffef2aa03f88f9" is_pre_election: true
I20260129 03:00:06.739332 20507 raft_consensus.cc:3060] T 4f5a77e94fef4b16a2ba7b067c8b144a P 974b802141bd4f13beffef2aa03f88f9 [term 4 FOLLOWER]: Advancing to term 5
I20260129 03:00:06.736888 22986 raft_consensus.cc:2804] T febffe4a8b394368b69b1d164308fb9a P 974b802141bd4f13beffef2aa03f88f9 [term 4 FOLLOWER]: Leader election won for term 4
I20260129 03:00:06.740437 22986 raft_consensus.cc:697] T febffe4a8b394368b69b1d164308fb9a P 974b802141bd4f13beffef2aa03f88f9 [term 4 LEADER]: Becoming Leader. State: Replica: 974b802141bd4f13beffef2aa03f88f9, State: Running, Role: LEADER
I20260129 03:00:06.741426 22986 consensus_queue.cc:237] T febffe4a8b394368b69b1d164308fb9a P 974b802141bd4f13beffef2aa03f88f9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.5, Last appended by leader: 5, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } }
I20260129 03:00:06.742055 20514 raft_consensus.cc:1240] T 8055232cf0004d40a85f3bf983fe00a9 P 974b802141bd4f13beffef2aa03f88f9 [term 5 FOLLOWER]: Rejecting Update request from peer 197b880fa39d432a9b472845876bfaf0 for earlier term 4. Current term is 5. Ops: [4.3-4.3]
I20260129 03:00:06.725298 23087 raft_consensus.cc:2804] T cedce331e1584ac8b636c9b8a72b2ba0 P 197b880fa39d432a9b472845876bfaf0 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260129 03:00:06.744431 23087 raft_consensus.cc:493] T cedce331e1584ac8b636c9b8a72b2ba0 P 197b880fa39d432a9b472845876bfaf0 [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260129 03:00:06.744788 23087 raft_consensus.cc:3060] T cedce331e1584ac8b636c9b8a72b2ba0 P 197b880fa39d432a9b472845876bfaf0 [term 7 FOLLOWER]: Advancing to term 8
I20260129 03:00:06.744877 23239 raft_consensus.cc:2804] T add8d4923d1a489e86882c3a25692b95 P 7a9daf5d42a84615a6383cfec9a10608 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260129 03:00:06.745348 23239 raft_consensus.cc:493] T add8d4923d1a489e86882c3a25692b95 P 7a9daf5d42a84615a6383cfec9a10608 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260129 03:00:06.745718 23239 raft_consensus.cc:3060] T add8d4923d1a489e86882c3a25692b95 P 7a9daf5d42a84615a6383cfec9a10608 [term 6 FOLLOWER]: Advancing to term 7
I20260129 03:00:06.726847 22982 raft_consensus.cc:2804] T 76f713c3331540d2a5b76164103e9eb1 P 974b802141bd4f13beffef2aa03f88f9 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260129 03:00:06.748180 22982 raft_consensus.cc:493] T 76f713c3331540d2a5b76164103e9eb1 P 974b802141bd4f13beffef2aa03f88f9 [term 5 FOLLOWER]: Starting leader election (detected failure of leader 1c50de89a5a845668696dcb28c3badd6)
I20260129 03:00:06.748569 22982 raft_consensus.cc:3060] T 76f713c3331540d2a5b76164103e9eb1 P 974b802141bd4f13beffef2aa03f88f9 [term 5 FOLLOWER]: Advancing to term 6
I20260129 03:00:06.750481 20511 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57584 (request call id 1450) took 1266 ms. Trace:
I20260129 03:00:06.751053 20511 rpcz_store.cc:276] 0129 03:00:05.484071 (+ 0us) service_pool.cc:168] Inserting onto call queue
0129 03:00:06.050786 (+566715us) service_pool.cc:225] Handling call
0129 03:00:06.095544 (+ 44758us) raft_consensus.cc:1471] Updating replica for 3 ops
0129 03:00:06.139508 (+ 43964us) raft_consensus.cc:1534] Early marking committed up to index 0
0129 03:00:06.139529 (+ 21us) raft_consensus.cc:1539] Triggering prepare for 3 ops
0129 03:00:06.237234 (+ 97705us) log.cc:844] Serialized 78 byte log entry
0129 03:00:06.378083 (+140849us) spinlock_profiling.cc:243] Waited 58.5 ms on lock 0x7b48000615d8. stack: 00007f8fe044e373 00007f8fe044e270 00007f8fdfda0574 00007f8fe9a5c6b4 00007f8fe9a5c649 00007f8fe8447e21 00007f8fe843eeb7 00007f8fe843ed4a 00007f8fe8535f7f 00007f8fe8535c47 00007f8fe854a26c 00007f8fe854a218 00007f8fe854a198 00007f8fe854a153 00007f8fe85493fc 00007f8fe74845b6
0129 03:00:06.518750 (+140667us) raft_consensus.cc:1645] Marking committed up to 6
0129 03:00:06.541613 (+ 22863us) log.cc:844] Serialized 14 byte log entry
0129 03:00:06.544539 (+ 2926us) log.cc:844] Serialized 14 byte log entry
0129 03:00:06.544755 (+ 216us) raft_consensus.cc:1697] Filling consensus response to leader.
0129 03:00:06.545245 (+ 490us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0129 03:00:06.583665 (+ 38420us) spinlock_profiling.cc:243] Waited 3.34 ms on lock 0x7b18002d5bc0. stack: 00007f8fe044e373 00007f8fe044e270 00007f8fdfda0574 00007f8fe9a5c6b4 00007f8fe9a5c649 00007f8fe92d4c91 00007f8fe92e1671 00007f8fe04802d3 00007f8fe04826a1 00007f8fe0482659 00007f8fe04825e9 00007f8fe04825b1 00007f8fe04818ad 0000000000379254 0000000000379099 00007f8fe046369a
0129 03:00:06.701595 (+117930us) spinlock_profiling.cc:243] Waited 2.17 ms on lock 0x7b18002d5bc0. stack: 00007f8fe044e373 00007f8fe044e270 00007f8fdfda0574 00007f8fe9a5c6b4 00007f8fe9a5c649 00007f8fe92d4c91 00007f8fe92e1671 00007f8fe04802d3 00007f8fe04826a1 00007f8fe0482659 00007f8fe04825e9 00007f8fe04825b1 00007f8fe04818ad 0000000000379254 0000000000379099 00007f8fe046369a
0129 03:00:06.750185 (+ 48590us) raft_consensus.cc:1685] finished
0129 03:00:06.750219 (+ 34us) raft_consensus.cc:1691] UpdateReplicas() finished
0129 03:00:06.750409 (+ 190us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":41605,"prepare.queue_time_us":67550,"prepare.run_cpu_time_us":57,"prepare.run_wall_time_us":57,"raft.queue_time_us":28738,"raft.run_cpu_time_us":2045,"raft.run_wall_time_us":236838,"spinlock_wait_cycles":147194880,"thread_start_us":236240,"threads_started":1,"wal-append.queue_time_us":425705}
I20260129 03:00:06.753996 22645 raft_consensus.cc:2749] T a5a05fbcbbcf4916bbc01fd1a5b5e90d P 197b880fa39d432a9b472845876bfaf0 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
W20260129 03:00:06.754547 20553 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.211.1:42027, user_credentials={real_user=slave}} blocked reactor thread for 59182.9us
I20260129 03:00:06.754662 20405 leader_election.cc:304] T 36ff33be666a4c82954731691dbcf327 P 197b880fa39d432a9b472845876bfaf0 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 197b880fa39d432a9b472845876bfaf0; no voters: 1c50de89a5a845668696dcb28c3badd6, 974b802141bd4f13beffef2aa03f88f9
I20260129 03:00:06.755416 20553 leader_election.cc:304] T 0340229d521b432580ae926f6062e3f7 P 1c50de89a5a845668696dcb28c3badd6 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 197b880fa39d432a9b472845876bfaf0, 1c50de89a5a845668696dcb28c3badd6; no voters: 974b802141bd4f13beffef2aa03f88f9
I20260129 03:00:06.756048 23180 raft_consensus.cc:2749] T 36ff33be666a4c82954731691dbcf327 P 197b880fa39d432a9b472845876bfaf0 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260129 03:00:06.756733 23214 raft_consensus.cc:2764] T 0340229d521b432580ae926f6062e3f7 P 1c50de89a5a845668696dcb28c3badd6 [term 7 FOLLOWER]: Leader pre-election decision vote started in defunct term 6: won
I20260129 03:00:06.754114 20511 raft_consensus.cc:1275] T 75b0b6e82ddc4f92b1fe1d94f9a0b73b P 974b802141bd4f13beffef2aa03f88f9 [term 4 FOLLOWER]: Refusing update from remote peer 7a9daf5d42a84615a6383cfec9a10608: Log matching property violated. Preceding OpId in replica: term: 3 index: 6. Preceding OpId from leader: term: 4 index: 7. (index mismatch)
I20260129 03:00:06.757896 20515 raft_consensus.cc:2393] T c891d2bff4ce4568861bbc992e91fbb0 P 974b802141bd4f13beffef2aa03f88f9 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1c50de89a5a845668696dcb28c3badd6 in current term 8: Already voted for candidate 974b802141bd4f13beffef2aa03f88f9 in this term.
I20260129 03:00:06.724613 20480 leader_election.cc:304] T a644c02b16af47679824ccd060ac10fd P 974b802141bd4f13beffef2aa03f88f9 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 974b802141bd4f13beffef2aa03f88f9; no voters: 1c50de89a5a845668696dcb28c3badd6, 7a9daf5d42a84615a6383cfec9a10608
I20260129 03:00:06.762596 20509 raft_consensus.cc:2468] T 505580c93c694bbb8ea96fdf9b17ab46 P 974b802141bd4f13beffef2aa03f88f9 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1c50de89a5a845668696dcb28c3badd6 in term 7.
I20260129 03:00:06.767484 20556 leader_election.cc:304] T 505580c93c694bbb8ea96fdf9b17ab46 P 1c50de89a5a845668696dcb28c3badd6 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1c50de89a5a845668696dcb28c3badd6, 974b802141bd4f13beffef2aa03f88f9; no voters:
W20260129 03:00:06.768350 20407 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260129 03:00:06.770069 20513 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "45cc5f3d2688486d9b6dbe7e666dedde" candidate_uuid: "7a9daf5d42a84615a6383cfec9a10608" candidate_term: 6 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "974b802141bd4f13beffef2aa03f88f9" is_pre_election: true
I20260129 03:00:06.770602 20513 raft_consensus.cc:2468] T 45cc5f3d2688486d9b6dbe7e666dedde P 974b802141bd4f13beffef2aa03f88f9 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7a9daf5d42a84615a6383cfec9a10608 in term 5.
I20260129 03:00:06.771584 20479 leader_election.cc:304] T a1a39dcd328e4268ae41b988c66b4a75 P 974b802141bd4f13beffef2aa03f88f9 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 197b880fa39d432a9b472845876bfaf0, 974b802141bd4f13beffef2aa03f88f9; no voters:
I20260129 03:00:06.728650 23059 raft_consensus.cc:3060] T 0678b8b468bd43cdbc3e485900844821 P 7a9daf5d42a84615a6383cfec9a10608 [term 7 FOLLOWER]: Advancing to term 8
I20260129 03:00:06.777168 20479 leader_election.cc:304] T 913fc7d751ed4b7388d889aac30a5f9b P 974b802141bd4f13beffef2aa03f88f9 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 197b880fa39d432a9b472845876bfaf0, 974b802141bd4f13beffef2aa03f88f9; no voters:
I20260129 03:00:06.733017 23262 raft_consensus.cc:2749] T 0f20af7fbe5d4db89a6f17948d98912c P 1c50de89a5a845668696dcb28c3badd6 [term 7 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260129 03:00:06.789790 20553 leader_election.cc:304] T 368953c9102643b2a21f99af36b51905 P 1c50de89a5a845668696dcb28c3badd6 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 197b880fa39d432a9b472845876bfaf0, 1c50de89a5a845668696dcb28c3badd6; no voters: 974b802141bd4f13beffef2aa03f88f9
I20260129 03:00:06.782261 22924 raft_consensus.cc:2804] T 505580c93c694bbb8ea96fdf9b17ab46 P 1c50de89a5a845668696dcb28c3badd6 [term 7 FOLLOWER]: Leader election won for term 7
I20260129 03:00:06.782701 23205 raft_consensus.cc:2804] T e7d83d4ccf684e76acb29a0dc844e1b8 P 974b802141bd4f13beffef2aa03f88f9 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260129 03:00:06.772509 23208 raft_consensus.cc:2749] T a644c02b16af47679824ccd060ac10fd P 974b802141bd4f13beffef2aa03f88f9 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260129 03:00:06.774513 23086 consensus_queue.cc:260] T c891d2bff4ce4568861bbc992e91fbb0 P 197b880fa39d432a9b472845876bfaf0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 7.6, Last appended by leader: 6, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 03:00:06.786764 23087 raft_consensus.cc:515] T cedce331e1584ac8b636c9b8a72b2ba0 P 197b880fa39d432a9b472845876bfaf0 [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 03:00:06.787472 23215 raft_consensus.cc:2804] T a1a39dcd328e4268ae41b988c66b4a75 P 974b802141bd4f13beffef2aa03f88f9 [term 5 FOLLOWER]: Leader election won for term 5
I20260129 03:00:06.790767 20405 leader_election.cc:304] T 926a250fa5834e88b96ccecbefbf5086 P 197b880fa39d432a9b472845876bfaf0 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 197b880fa39d432a9b472845876bfaf0, 1c50de89a5a845668696dcb28c3badd6; no voters: 974b802141bd4f13beffef2aa03f88f9
W20260129 03:00:06.780350 20480 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.211.4:40439, user_credentials={real_user=slave}} blocked reactor thread for 55792.7us
I20260129 03:00:06.802017 20587 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:48526 (request call id 362) took 1260 ms. Trace:
I20260129 03:00:06.813684 20587 rpcz_store.cc:276] 0129 03:00:05.542153 (+ 0us) service_pool.cc:168] Inserting onto call queue
0129 03:00:05.542327 (+ 174us) service_pool.cc:225] Handling call
0129 03:00:05.565379 (+ 23052us) raft_consensus.cc:1471] Updating replica for 0 ops
0129 03:00:05.615337 (+ 49958us) raft_consensus.cc:1534] Early marking committed up to index 0
0129 03:00:05.615366 (+ 29us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0129 03:00:05.615432 (+ 66us) raft_consensus.cc:1645] Marking committed up to 6
0129 03:00:05.615467 (+ 35us) raft_consensus.cc:1697] Filling consensus response to leader.
0129 03:00:06.801716 (+1186249us) spinlock_profiling.cc:243] Waited 49.8 ms on lock 0x7b5400087fd4. stack: 00007f8fe044e373 00007f8fe044e270 00007f8fdfda0574 00007f8fe9a5c6b4 00007f8fe9a5c649 00007f8fe34975a1 00007f8fe352ea81 00007f8fe352d4be 00007f8fe84e1f49 00007f8fe1a6ed34 00007f8fe1a6ecc2 00007f8fe1a6ebf1 00007f8fe1a6eb6c 00007f8fe1a6de22 00007f8fe186d34c 00007f8fe186c786
0129 03:00:06.801784 (+ 68us) raft_consensus.cc:1691] UpdateReplicas() finished
0129 03:00:06.801946 (+ 162us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":114434304}
I20260129 03:00:06.808789 20553 leader_election.cc:304] T e33a417cf9c54cf5af73a4ee3aac4d3a P 1c50de89a5a845668696dcb28c3badd6 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 197b880fa39d432a9b472845876bfaf0, 1c50de89a5a845668696dcb28c3badd6; no voters: 974b802141bd4f13beffef2aa03f88f9
W20260129 03:00:06.789345 23310 log.cc:927] Time spent T cb0b3f259ebc4310927338a8740e87b8 P 7a9daf5d42a84615a6383cfec9a10608: Append to log took a long time: real 0.126s user 0.001s sys 0.000s
I20260129 03:00:06.817520 21102 leader_election.cc:304] T febffe4a8b394368b69b1d164308fb9a P 7a9daf5d42a84615a6383cfec9a10608 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7a9daf5d42a84615a6383cfec9a10608; no voters: 197b880fa39d432a9b472845876bfaf0, 974b802141bd4f13beffef2aa03f88f9
I20260129 03:00:06.820847 20553 leader_election.cc:304] T 2e58e9a5a8aa4cecafb464addb4f35ed P 1c50de89a5a845668696dcb28c3badd6 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1c50de89a5a845668696dcb28c3badd6; no voters: 197b880fa39d432a9b472845876bfaf0, 7a9daf5d42a84615a6383cfec9a10608
I20260129 03:00:06.805981 23205 raft_consensus.cc:493] T e7d83d4ccf684e76acb29a0dc844e1b8 P 974b802141bd4f13beffef2aa03f88f9 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260129 03:00:06.773185 23290 raft_consensus.cc:515] T cb0b3f259ebc4310927338a8740e87b8 P 1c50de89a5a845668696dcb28c3badd6 [term 8 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } }
I20260129 03:00:06.830220 20589 raft_consensus.cc:2468] T 913fc7d751ed4b7388d889aac30a5f9b P 1c50de89a5a845668696dcb28c3badd6 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 974b802141bd4f13beffef2aa03f88f9 in term 5.
I20260129 03:00:06.814908 22878 raft_consensus.cc:2804] T 913fc7d751ed4b7388d889aac30a5f9b P 974b802141bd4f13beffef2aa03f88f9 [term 5 FOLLOWER]: Leader election won for term 5
I20260129 03:00:06.787626 22982 raft_consensus.cc:515] T 76f713c3331540d2a5b76164103e9eb1 P 974b802141bd4f13beffef2aa03f88f9 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 03:00:06.812501 23309 raft_consensus.cc:493] T 3ccac82a01a2445fa5b0ce6c0fcf2295 P 974b802141bd4f13beffef2aa03f88f9 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 7a9daf5d42a84615a6383cfec9a10608)
I20260129 03:00:06.838613 21102 leader_election.cc:304] T 45cc5f3d2688486d9b6dbe7e666dedde P 7a9daf5d42a84615a6383cfec9a10608 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 7a9daf5d42a84615a6383cfec9a10608, 974b802141bd4f13beffef2aa03f88f9; no voters: 1c50de89a5a845668696dcb28c3badd6
I20260129 03:00:06.829762 23026 raft_consensus.cc:2804] T e33a417cf9c54cf5af73a4ee3aac4d3a P 1c50de89a5a845668696dcb28c3badd6 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260129 03:00:06.832407 23262 raft_consensus.cc:493] T 1299723a4476420ab02fd8bfd7aa0645 P 1c50de89a5a845668696dcb28c3badd6 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260129 03:00:06.848022 20435 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "368953c9102643b2a21f99af36b51905" candidate_uuid: "974b802141bd4f13beffef2aa03f88f9" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "197b880fa39d432a9b472845876bfaf0" is_pre_election: true
I20260129 03:00:06.832593 23091 raft_consensus.cc:2804] T 926a250fa5834e88b96ccecbefbf5086 P 197b880fa39d432a9b472845876bfaf0 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260129 03:00:06.835727 23086 raft_consensus.cc:3060] T c891d2bff4ce4568861bbc992e91fbb0 P 197b880fa39d432a9b472845876bfaf0 [term 7 FOLLOWER]: Advancing to term 8
I20260129 03:00:06.831772 23304 raft_consensus.cc:493] T 78aa39096868415d9afc2ffcb248e198 P 1c50de89a5a845668696dcb28c3badd6 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260129 03:00:06.849417 20432 raft_consensus.cc:1275] T 724a236524cb4e7ba3428880681d6cc2 P 197b880fa39d432a9b472845876bfaf0 [term 5 FOLLOWER]: Refusing update from remote peer 974b802141bd4f13beffef2aa03f88f9: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 5 index: 4. (index mismatch)
I20260129 03:00:06.839308 22694 leader_election.cc:290] T fcddab6bca684adf8ea032289a00f90e P 974b802141bd4f13beffef2aa03f88f9 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 197b880fa39d432a9b472845876bfaf0 (127.18.211.1:42027), 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3:37735)
I20260129 03:00:06.840716 23097 raft_consensus.cc:493] T 4e420f90884f4796997b5ea41e49e411 P 974b802141bd4f13beffef2aa03f88f9 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260129 03:00:06.838738 23215 raft_consensus.cc:697] T a1a39dcd328e4268ae41b988c66b4a75 P 974b802141bd4f13beffef2aa03f88f9 [term 5 LEADER]: Becoming Leader. State: Replica: 974b802141bd4f13beffef2aa03f88f9, State: Running, Role: LEADER
I20260129 03:00:06.834725 22924 raft_consensus.cc:697] T 505580c93c694bbb8ea96fdf9b17ab46 P 1c50de89a5a845668696dcb28c3badd6 [term 7 LEADER]: Becoming Leader. State: Replica: 1c50de89a5a845668696dcb28c3badd6, State: Running, Role: LEADER
W20260129 03:00:06.853470 20405 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.211.3:37735, user_credentials={real_user=slave}} blocked reactor thread for 95295.9us
I20260129 03:00:06.858590 20440 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "cedce331e1584ac8b636c9b8a72b2ba0" candidate_uuid: "974b802141bd4f13beffef2aa03f88f9" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "197b880fa39d432a9b472845876bfaf0" is_pre_election: true
I20260129 03:00:06.832258 23107 raft_consensus.cc:2749] T febffe4a8b394368b69b1d164308fb9a P 7a9daf5d42a84615a6383cfec9a10608 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260129 03:00:06.880191 20440 raft_consensus.cc:2393] T cedce331e1584ac8b636c9b8a72b2ba0 P 197b880fa39d432a9b472845876bfaf0 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 974b802141bd4f13beffef2aa03f88f9 in current term 8: Already voted for candidate 197b880fa39d432a9b472845876bfaf0 in this term.
I20260129 03:00:06.859843 20437 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7a9a69cd82724b1faed1d10aa091fc4a" candidate_uuid: "974b802141bd4f13beffef2aa03f88f9" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "197b880fa39d432a9b472845876bfaf0" is_pre_election: true
I20260129 03:00:06.806087 23260 consensus_queue.cc:237] T cb4727386a3e48aca5d71cc6c611c197 P 1c50de89a5a845668696dcb28c3badd6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 03:00:06.840862 23292 raft_consensus.cc:2749] T 2e58e9a5a8aa4cecafb464addb4f35ed P 1c50de89a5a845668696dcb28c3badd6 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260129 03:00:06.861969 20439 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c891d2bff4ce4568861bbc992e91fbb0" candidate_uuid: "974b802141bd4f13beffef2aa03f88f9" candidate_term: 8 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "197b880fa39d432a9b472845876bfaf0"
I20260129 03:00:06.835595 23205 raft_consensus.cc:3060] T e7d83d4ccf684e76acb29a0dc844e1b8 P 974b802141bd4f13beffef2aa03f88f9 [term 6 FOLLOWER]: Advancing to term 7
I20260129 03:00:06.855466 20516 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a644c02b16af47679824ccd060ac10fd" candidate_uuid: "7a9daf5d42a84615a6383cfec9a10608" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "974b802141bd4f13beffef2aa03f88f9" is_pre_election: true
I20260129 03:00:06.850720 20435 raft_consensus.cc:2468] T 368953c9102643b2a21f99af36b51905 P 197b880fa39d432a9b472845876bfaf0 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 974b802141bd4f13beffef2aa03f88f9 in term 7.
I20260129 03:00:06.896803 20435 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4f8f3a38f6e24ba4857875181f447442" candidate_uuid: "1c50de89a5a845668696dcb28c3badd6" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "197b880fa39d432a9b472845876bfaf0" is_pre_election: true
I20260129 03:00:06.867666 20432 raft_consensus.cc:1240] T 17d6431cc77a491e8464d68db4081a51 P 197b880fa39d432a9b472845876bfaf0 [term 5 FOLLOWER]: Rejecting Update request from peer 974b802141bd4f13beffef2aa03f88f9 for earlier term 4. Current term is 5. Ops: []
I20260129 03:00:06.849994 23026 raft_consensus.cc:493] T e33a417cf9c54cf5af73a4ee3aac4d3a P 1c50de89a5a845668696dcb28c3badd6 [term 6 FOLLOWER]: Starting leader election (detected failure of leader 974b802141bd4f13beffef2aa03f88f9)
I20260129 03:00:06.882645 20434 raft_consensus.cc:1275] T 0340229d521b432580ae926f6062e3f7 P 197b880fa39d432a9b472845876bfaf0 [term 7 FOLLOWER]: Refusing update from remote peer 974b802141bd4f13beffef2aa03f88f9: Log matching property violated. Preceding OpId in replica: term: 5 index: 4. Preceding OpId from leader: term: 7 index: 5. (index mismatch)
I20260129 03:00:06.857734 23091 raft_consensus.cc:493] T 926a250fa5834e88b96ccecbefbf5086 P 197b880fa39d432a9b472845876bfaf0 [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260129 03:00:06.841152 23163 consensus_queue.cc:1048] T 75b0b6e82ddc4f92b1fe1d94f9a0b73b P 7a9daf5d42a84615a6383cfec9a10608 [LEADER]: Connected to new peer: Peer: permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.024s
I20260129 03:00:06.852303 22878 raft_consensus.cc:697] T 913fc7d751ed4b7388d889aac30a5f9b P 974b802141bd4f13beffef2aa03f88f9 [term 5 LEADER]: Becoming Leader. State: Replica: 974b802141bd4f13beffef2aa03f88f9, State: Running, Role: LEADER
I20260129 03:00:06.859617 23109 raft_consensus.cc:2804] T 45cc5f3d2688486d9b6dbe7e666dedde P 7a9daf5d42a84615a6383cfec9a10608 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260129 03:00:06.865121 23280 raft_consensus.cc:493] T a4c1226b987b468fa84dfc6ba0e2dbe1 P 1c50de89a5a845668696dcb28c3badd6 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 197b880fa39d432a9b472845876bfaf0)
I20260129 03:00:06.846488 23283 raft_consensus.cc:2804] T 368953c9102643b2a21f99af36b51905 P 1c50de89a5a845668696dcb28c3badd6 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260129 03:00:06.866005 22996 raft_consensus.cc:493] T 17d6431cc77a491e8464d68db4081a51 P 1c50de89a5a845668696dcb28c3badd6 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 974b802141bd4f13beffef2aa03f88f9)
I20260129 03:00:06.849779 23309 raft_consensus.cc:515] T 3ccac82a01a2445fa5b0ce6c0fcf2295 P 974b802141bd4f13beffef2aa03f88f9 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } }
I20260129 03:00:06.963689 20435 raft_consensus.cc:1240] T a1a39dcd328e4268ae41b988c66b4a75 P 197b880fa39d432a9b472845876bfaf0 [term 5 FOLLOWER]: Rejecting Update request from peer 1c50de89a5a845668696dcb28c3badd6 for earlier term 4. Current term is 5. Ops: []
I20260129 03:00:06.934818 20432 raft_consensus.cc:1240] T 4e3805ffa70242e8ade66da260dddf66 P 197b880fa39d432a9b472845876bfaf0 [term 6 FOLLOWER]: Rejecting Update request from peer 1c50de89a5a845668696dcb28c3badd6 for earlier term 5. Current term is 6. Ops: [5.7-5.7]
I20260129 03:00:06.882076 23216 raft_consensus.cc:493] T 96609ac68bdf4993a12e1c23355ca013 P 974b802141bd4f13beffef2aa03f88f9 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 7a9daf5d42a84615a6383cfec9a10608)
I20260129 03:00:06.882371 23145 raft_consensus.cc:493] T 926a250fa5834e88b96ccecbefbf5086 P 974b802141bd4f13beffef2aa03f88f9 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260129 03:00:06.877549 23304 raft_consensus.cc:515] T 78aa39096868415d9afc2ffcb248e198 P 1c50de89a5a845668696dcb28c3badd6 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } }
I20260129 03:00:06.962320 22978 raft_consensus.cc:493] T 87d5fe48a44a408aaf03e894a67ecb2f P 974b802141bd4f13beffef2aa03f88f9 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 197b880fa39d432a9b472845876bfaf0)
I20260129 03:00:06.956017 20440 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fcddab6bca684adf8ea032289a00f90e" candidate_uuid: "974b802141bd4f13beffef2aa03f88f9" candidate_term: 10 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "197b880fa39d432a9b472845876bfaf0" is_pre_election: true
I20260129 03:00:06.900496 23034 consensus_queue.cc:1048] T 724a236524cb4e7ba3428880681d6cc2 P 974b802141bd4f13beffef2aa03f88f9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 1, Time since last communication: 0.070s
I20260129 03:00:06.853313 23299 consensus_queue.cc:1048] T fd8311b779af4e1fa1402b2a02a602f2 P 1c50de89a5a845668696dcb28c3badd6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.098s
I20260129 03:00:06.962769 20405 leader_election.cc:304] T e4d18c35b0a7493e85d0a4b92956a48f P 197b880fa39d432a9b472845876bfaf0 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 197b880fa39d432a9b472845876bfaf0; no voters: 1c50de89a5a845668696dcb28c3badd6, 974b802141bd4f13beffef2aa03f88f9
I20260129 03:00:06.888763 22645 consensus_queue.cc:1048] T 00e79d8f25a34676b1d866eb2068a2b2 P 197b880fa39d432a9b472845876bfaf0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 6, Time since last communication: 0.073s
I20260129 03:00:06.933284 22924 consensus_queue.cc:237] T 505580c93c694bbb8ea96fdf9b17ab46 P 1c50de89a5a845668696dcb28c3badd6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 6.8, Last appended by leader: 8, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } }
W20260129 03:00:07.008184 20406 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260129 03:00:06.998428 22982 leader_election.cc:290] T 76f713c3331540d2a5b76164103e9eb1 P 974b802141bd4f13beffef2aa03f88f9 [CANDIDATE]: Term 6 election: Requested vote from peers 197b880fa39d432a9b472845876bfaf0 (127.18.211.1:42027), 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3:37735)
I20260129 03:00:07.009050 23035 raft_consensus.cc:2749] T 00e79d8f25a34676b1d866eb2068a2b2 P 7a9daf5d42a84615a6383cfec9a10608 [term 8 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260129 03:00:07.009268 20479 leader_election.cc:304] T 368953c9102643b2a21f99af36b51905 P 974b802141bd4f13beffef2aa03f88f9 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 197b880fa39d432a9b472845876bfaf0, 974b802141bd4f13beffef2aa03f88f9; no voters:
I20260129 03:00:07.009996 22691 raft_consensus.cc:2749] T e4d18c35b0a7493e85d0a4b92956a48f P 197b880fa39d432a9b472845876bfaf0 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260129 03:00:07.010957 22982 raft_consensus.cc:2804] T 368953c9102643b2a21f99af36b51905 P 974b802141bd4f13beffef2aa03f88f9 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260129 03:00:07.011375 22982 raft_consensus.cc:493] T 368953c9102643b2a21f99af36b51905 P 974b802141bd4f13beffef2aa03f88f9 [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260129 03:00:07.011731 22982 raft_consensus.cc:3060] T 368953c9102643b2a21f99af36b51905 P 974b802141bd4f13beffef2aa03f88f9 [term 7 FOLLOWER]: Advancing to term 8
I20260129 03:00:07.011761 23212 consensus_queue.cc:1059] T 17d6431cc77a491e8464d68db4081a51 P 974b802141bd4f13beffef2aa03f88f9 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 }, Status: INVALID_TERM, Last received: 4.2, Next index: 3, Last known committed idx: 1, Time since last communication: 0.001s
W20260129 03:00:07.016139 23302 log.cc:927] Time spent T 2ac659d732314e1288849f08fe5e074e P 7a9daf5d42a84615a6383cfec9a10608: Append to log took a long time: real 0.066s user 0.000s sys 0.000s
W20260129 03:00:07.017802 23316 log.cc:927] Time spent T a0fcc58d6fb6473cafbd9d34d5195269 P 197b880fa39d432a9b472845876bfaf0: Append to log took a long time: real 0.053s user 0.000s sys 0.000s
I20260129 03:00:07.019349 20507 raft_consensus.cc:2468] T 4f5a77e94fef4b16a2ba7b067c8b144a P 974b802141bd4f13beffef2aa03f88f9 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7a9daf5d42a84615a6383cfec9a10608 in term 5.
I20260129 03:00:06.998973 23216 raft_consensus.cc:515] T 96609ac68bdf4993a12e1c23355ca013 P 974b802141bd4f13beffef2aa03f88f9 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } attrs { replace: true } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } }
W20260129 03:00:07.020222 20405 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.211.3:37735, user_credentials={real_user=slave}} blocked reactor thread for 57541.6us
I20260129 03:00:07.020889 22982 raft_consensus.cc:515] T 368953c9102643b2a21f99af36b51905 P 974b802141bd4f13beffef2aa03f88f9 [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 03:00:07.024704 22982 leader_election.cc:290] T 368953c9102643b2a21f99af36b51905 P 974b802141bd4f13beffef2aa03f88f9 [CANDIDATE]: Term 8 election: Requested vote from peers 197b880fa39d432a9b472845876bfaf0 (127.18.211.1:42027), 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3:37735)
W20260129 03:00:06.908495 20407 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260129 03:00:06.963539 20434 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c2633a6fc34c44579de348d208b0c01e" candidate_uuid: "1c50de89a5a845668696dcb28c3badd6" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "197b880fa39d432a9b472845876bfaf0"
I20260129 03:00:07.040175 20434 raft_consensus.cc:3060] T c2633a6fc34c44579de348d208b0c01e P 197b880fa39d432a9b472845876bfaf0 [term 3 FOLLOWER]: Advancing to term 4
I20260129 03:00:06.878038 23262 raft_consensus.cc:515] T 1299723a4476420ab02fd8bfd7aa0645 P 1c50de89a5a845668696dcb28c3badd6 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } }
I20260129 03:00:06.882520 23097 raft_consensus.cc:515] T 4e420f90884f4796997b5ea41e49e411 P 974b802141bd4f13beffef2aa03f88f9 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } }
I20260129 03:00:06.933980 23091 raft_consensus.cc:3060] T 926a250fa5834e88b96ccecbefbf5086 P 197b880fa39d432a9b472845876bfaf0 [term 7 FOLLOWER]: Advancing to term 8
I20260129 03:00:06.956904 23109 raft_consensus.cc:493] T 45cc5f3d2688486d9b6dbe7e666dedde P 7a9daf5d42a84615a6383cfec9a10608 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260129 03:00:06.975445 20432 raft_consensus.cc:1240] T 75be4bd6bd45474795fd472eff2953ac P 197b880fa39d432a9b472845876bfaf0 [term 10 LEADER]: Rejecting Update request from peer 1c50de89a5a845668696dcb28c3badd6 for earlier term 9. Current term is 10. Ops: []
I20260129 03:00:06.949968 20433 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "cb4727386a3e48aca5d71cc6c611c197" candidate_uuid: "1c50de89a5a845668696dcb28c3badd6" candidate_term: 5 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "197b880fa39d432a9b472845876bfaf0"
I20260129 03:00:07.050093 20433 raft_consensus.cc:2393] T cb4727386a3e48aca5d71cc6c611c197 P 197b880fa39d432a9b472845876bfaf0 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 1c50de89a5a845668696dcb28c3badd6 in current term 5: Already voted for candidate 197b880fa39d432a9b472845876bfaf0 in this term.
I20260129 03:00:06.980871 20440 raft_consensus.cc:2468] T fcddab6bca684adf8ea032289a00f90e P 197b880fa39d432a9b472845876bfaf0 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 974b802141bd4f13beffef2aa03f88f9 in term 9.
I20260129 03:00:06.959038 20516 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "cedce331e1584ac8b636c9b8a72b2ba0" candidate_uuid: "197b880fa39d432a9b472845876bfaf0" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "974b802141bd4f13beffef2aa03f88f9"
I20260129 03:00:07.051695 20516 raft_consensus.cc:3060] T cedce331e1584ac8b636c9b8a72b2ba0 P 974b802141bd4f13beffef2aa03f88f9 [term 7 FOLLOWER]: Advancing to term 8
I20260129 03:00:06.998767 23292 raft_consensus.cc:493] T b189f08e16a74a11980cc55d99359a0a P 1c50de89a5a845668696dcb28c3badd6 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260129 03:00:06.991552 20435 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "724a236524cb4e7ba3428880681d6cc2" candidate_uuid: "1c50de89a5a845668696dcb28c3badd6" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "197b880fa39d432a9b472845876bfaf0" is_pre_election: true
I20260129 03:00:06.960899 23283 raft_consensus.cc:493] T 368953c9102643b2a21f99af36b51905 P 1c50de89a5a845668696dcb28c3badd6 [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260129 03:00:06.969154 23087 leader_election.cc:290] T cedce331e1584ac8b636c9b8a72b2ba0 P 197b880fa39d432a9b472845876bfaf0 [CANDIDATE]: Term 8 election: Requested vote from peers 974b802141bd4f13beffef2aa03f88f9 (127.18.211.2:34517), 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3:37735)
I20260129 03:00:06.969529 23280 raft_consensus.cc:515] T a4c1226b987b468fa84dfc6ba0e2dbe1 P 1c50de89a5a845668696dcb28c3badd6 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 03:00:07.047214 20436 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "36ff33be666a4c82954731691dbcf327" candidate_uuid: "1c50de89a5a845668696dcb28c3badd6" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "197b880fa39d432a9b472845876bfaf0" is_pre_election: true
I20260129 03:00:07.093822 20436 raft_consensus.cc:2468] T 36ff33be666a4c82954731691dbcf327 P 197b880fa39d432a9b472845876bfaf0 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1c50de89a5a845668696dcb28c3badd6 in term 3.
I20260129 03:00:07.045323 23216 leader_election.cc:290] T 96609ac68bdf4993a12e1c23355ca013 P 974b802141bd4f13beffef2aa03f88f9 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 197b880fa39d432a9b472845876bfaf0 (127.18.211.1:42027), 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3:37735), 7a9daf5d42a84615a6383cfec9a10608 (127.18.211.4:40439)
I20260129 03:00:07.049007 21136 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00e79d8f25a34676b1d866eb2068a2b2" candidate_uuid: "197b880fa39d432a9b472845876bfaf0" candidate_term: 8 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "7a9daf5d42a84615a6383cfec9a10608"
I20260129 03:00:07.065044 20435 raft_consensus.cc:2468] T 724a236524cb4e7ba3428880681d6cc2 P 197b880fa39d432a9b472845876bfaf0 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1c50de89a5a845668696dcb28c3badd6 in term 5.
I20260129 03:00:07.050484 21102 leader_election.cc:304] T 4f5a77e94fef4b16a2ba7b067c8b144a P 7a9daf5d42a84615a6383cfec9a10608 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7a9daf5d42a84615a6383cfec9a10608, 974b802141bd4f13beffef2aa03f88f9; no voters:
W20260129 03:00:07.097221 21102 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.211.2:34517, user_credentials={real_user=slave}} blocked reactor thread for 46844.6us
I20260129 03:00:07.072664 20479 leader_election.cc:304] T fcddab6bca684adf8ea032289a00f90e P 974b802141bd4f13beffef2aa03f88f9 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 197b880fa39d432a9b472845876bfaf0, 974b802141bd4f13beffef2aa03f88f9; no voters:
I20260129 03:00:07.073138 20432 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "96609ac68bdf4993a12e1c23355ca013" candidate_uuid: "974b802141bd4f13beffef2aa03f88f9" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "197b880fa39d432a9b472845876bfaf0" is_pre_election: true
I20260129 03:00:07.098155 20436 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c891d2bff4ce4568861bbc992e91fbb0" candidate_uuid: "1c50de89a5a845668696dcb28c3badd6" candidate_term: 8 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "197b880fa39d432a9b472845876bfaf0" is_pre_election: true
I20260129 03:00:07.099494 20432 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "368953c9102643b2a21f99af36b51905" candidate_uuid: "974b802141bd4f13beffef2aa03f88f9" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "197b880fa39d432a9b472845876bfaf0"
I20260129 03:00:07.100343 20432 raft_consensus.cc:3060] T 368953c9102643b2a21f99af36b51905 P 197b880fa39d432a9b472845876bfaf0 [term 7 FOLLOWER]: Advancing to term 8
I20260129 03:00:06.970474 22996 raft_consensus.cc:515] T 17d6431cc77a491e8464d68db4081a51 P 1c50de89a5a845668696dcb28c3badd6 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 03:00:06.893651 22891 raft_consensus.cc:493] T 45cc5f3d2688486d9b6dbe7e666dedde P 1c50de89a5a845668696dcb28c3badd6 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260129 03:00:06.906371 23026 raft_consensus.cc:3060] T e33a417cf9c54cf5af73a4ee3aac4d3a P 1c50de89a5a845668696dcb28c3badd6 [term 6 FOLLOWER]: Advancing to term 7
I20260129 03:00:07.072100 23109 raft_consensus.cc:3060] T 45cc5f3d2688486d9b6dbe7e666dedde P 7a9daf5d42a84615a6383cfec9a10608 [term 5 FOLLOWER]: Advancing to term 6
I20260129 03:00:07.090044 23097 leader_election.cc:290] T 4e420f90884f4796997b5ea41e49e411 P 974b802141bd4f13beffef2aa03f88f9 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 197b880fa39d432a9b472845876bfaf0 (127.18.211.1:42027), 7a9daf5d42a84615a6383cfec9a10608 (127.18.211.4:40439)
W20260129 03:00:07.091075 21134 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:47844 (request call id 357) took 4695 ms (client timeout 2297 ms). Trace:
I20260129 03:00:07.091621 22878 consensus_queue.cc:237] T 913fc7d751ed4b7388d889aac30a5f9b P 974b802141bd4f13beffef2aa03f88f9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 3.3, Last appended by leader: 3, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 03:00:07.110960 20433 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3906b080a5b34a74b8b64128635e8997" candidate_uuid: "1c50de89a5a845668696dcb28c3badd6" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "197b880fa39d432a9b472845876bfaf0" is_pre_election: true
W20260129 03:00:07.115839 20482 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260129 03:00:07.115756 22996 leader_election.cc:290] T 17d6431cc77a491e8464d68db4081a51 P 1c50de89a5a845668696dcb28c3badd6 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 197b880fa39d432a9b472845876bfaf0 (127.18.211.1:42027), 974b802141bd4f13beffef2aa03f88f9 (127.18.211.2:34517)
I20260129 03:00:07.116369 22891 raft_consensus.cc:515] T 45cc5f3d2688486d9b6dbe7e666dedde P 1c50de89a5a845668696dcb28c3badd6 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } }
W20260129 03:00:07.087713 22855 consensus_meta.cc:287] T 18f30ebaa4174fd28c958e72d0cc7e24 P 1c50de89a5a845668696dcb28c3badd6: Time spent flushing consensus metadata: real 0.551s user 0.010s sys 0.000s
W20260129 03:00:07.139724 23322 log.cc:927] Time spent T 7a9a69cd82724b1faed1d10aa091fc4a P 974b802141bd4f13beffef2aa03f88f9: Append to log took a long time: real 0.164s user 0.001s sys 0.000s
I20260129 03:00:07.104621 23290 leader_election.cc:290] T cb0b3f259ebc4310927338a8740e87b8 P 1c50de89a5a845668696dcb28c3badd6 [CANDIDATE]: Term 8 election: Requested vote from peers 974b802141bd4f13beffef2aa03f88f9 (127.18.211.2:34517), 7a9daf5d42a84615a6383cfec9a10608 (127.18.211.4:40439)
I20260129 03:00:06.998742 23145 raft_consensus.cc:515] T 926a250fa5834e88b96ccecbefbf5086 P 974b802141bd4f13beffef2aa03f88f9 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 03:00:07.119380 20589 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "75be4bd6bd45474795fd472eff2953ac" candidate_uuid: "197b880fa39d432a9b472845876bfaf0" candidate_term: 10 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "1c50de89a5a845668696dcb28c3badd6" is_pre_election: true
I20260129 03:00:07.049893 23208 raft_consensus.cc:3055] T 17d6431cc77a491e8464d68db4081a51 P 974b802141bd4f13beffef2aa03f88f9 [term 4 LEADER]: Stepping down as leader of term 4
W20260129 03:00:07.123023 21134 rpcz_store.cc:269] 0129 03:00:02.390422 (+ 0us) service_pool.cc:168] Inserting onto call queue
0129 03:00:02.419055 (+ 28633us) service_pool.cc:225] Handling call
0129 03:00:07.047194 (+4628139us) spinlock_profiling.cc:243] Waited 3.86 s on lock 0x7b54003d9ad4. stack: 00007f8fe044e373 00007f8fe044e270 00007f8fdfda0574 00007f8fe9a5c6b4 00007f8fe9a5c649 00007f8fe34975a1 00007f8fe3531f7a 00007f8fe84e3647 00007f8fe1a73834 00007f8fe1a737c2 00007f8fe1a736f1 00007f8fe1a7366c 00007f8fe1a72922 00007f8fe186d34c 00007f8fe186c786 00007f8fe186c12f
0129 03:00:07.085676 (+ 38482us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":8870399872}
I20260129 03:00:07.047432 22989 consensus_queue.cc:1048] T 0340229d521b432580ae926f6062e3f7 P 974b802141bd4f13beffef2aa03f88f9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.082s
I20260129 03:00:07.086736 23283 raft_consensus.cc:3060] T 368953c9102643b2a21f99af36b51905 P 1c50de89a5a845668696dcb28c3badd6 [term 7 FOLLOWER]: Advancing to term 8
I20260129 03:00:07.131012 21135 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4e420f90884f4796997b5ea41e49e411" candidate_uuid: "974b802141bd4f13beffef2aa03f88f9" candidate_term: 7 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "7a9daf5d42a84615a6383cfec9a10608" is_pre_election: true
I20260129 03:00:07.135437 20587 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e33a417cf9c54cf5af73a4ee3aac4d3a" candidate_uuid: "197b880fa39d432a9b472845876bfaf0" candidate_term: 7 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "1c50de89a5a845668696dcb28c3badd6" is_pre_election: true
I20260129 03:00:07.110582 23035 raft_consensus.cc:2804] T 4f5a77e94fef4b16a2ba7b067c8b144a P 7a9daf5d42a84615a6383cfec9a10608 [term 5 FOLLOWER]: Leader election won for term 5
I20260129 03:00:07.119325 20440 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "76f713c3331540d2a5b76164103e9eb1" candidate_uuid: "974b802141bd4f13beffef2aa03f88f9" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "197b880fa39d432a9b472845876bfaf0"
I20260129 03:00:07.184392 20440 raft_consensus.cc:3060] T 76f713c3331540d2a5b76164103e9eb1 P 197b880fa39d432a9b472845876bfaf0 [term 5 FOLLOWER]: Advancing to term 6
I20260129 03:00:07.145778 20589 raft_consensus.cc:2468] T 75be4bd6bd45474795fd472eff2953ac P 1c50de89a5a845668696dcb28c3badd6 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 197b880fa39d432a9b472845876bfaf0 in term 10.
I20260129 03:00:07.150453 20433 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4e420f90884f4796997b5ea41e49e411" candidate_uuid: "974b802141bd4f13beffef2aa03f88f9" candidate_term: 7 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "197b880fa39d432a9b472845876bfaf0" is_pre_election: true
I20260129 03:00:07.026054 22978 raft_consensus.cc:515] T 87d5fe48a44a408aaf03e894a67ecb2f P 974b802141bd4f13beffef2aa03f88f9 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } }
I20260129 03:00:07.045332 23304 leader_election.cc:290] T 78aa39096868415d9afc2ffcb248e198 P 1c50de89a5a845668696dcb28c3badd6 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 197b880fa39d432a9b472845876bfaf0 (127.18.211.1:42027), 7a9daf5d42a84615a6383cfec9a10608 (127.18.211.4:40439)
I20260129 03:00:07.153970 20582 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7a9a69cd82724b1faed1d10aa091fc4a" candidate_uuid: "197b880fa39d432a9b472845876bfaf0" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "1c50de89a5a845668696dcb28c3badd6"
I20260129 03:00:07.126873 22694 raft_consensus.cc:2804] T fcddab6bca684adf8ea032289a00f90e P 974b802141bd4f13beffef2aa03f88f9 [term 9 FOLLOWER]: Leader pre-election won for term 10
I20260129 03:00:07.158601 20553 leader_election.cc:304] T 4f8f3a38f6e24ba4857875181f447442 P 1c50de89a5a845668696dcb28c3badd6 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1c50de89a5a845668696dcb28c3badd6; no voters: 197b880fa39d432a9b472845876bfaf0, 974b802141bd4f13beffef2aa03f88f9
I20260129 03:00:07.188301 20435 pending_rounds.cc:85] T 7d9494f5584d4d5db8b9adf1c8fc888b P 197b880fa39d432a9b472845876bfaf0: Aborting all ops after (but not including) 2
I20260129 03:00:07.141695 21130 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "96609ac68bdf4993a12e1c23355ca013" candidate_uuid: "974b802141bd4f13beffef2aa03f88f9" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "7a9daf5d42a84615a6383cfec9a10608" is_pre_election: true
I20260129 03:00:07.197647 20435 pending_rounds.cc:107] T 7d9494f5584d4d5db8b9adf1c8fc888b P 197b880fa39d432a9b472845876bfaf0: Aborting uncommitted NO_OP operation due to leader change: 3.3
I20260129 03:00:07.155557 23208 raft_consensus.cc:740] T 17d6431cc77a491e8464d68db4081a51 P 974b802141bd4f13beffef2aa03f88f9 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 974b802141bd4f13beffef2aa03f88f9, State: Running, Role: LEADER
I20260129 03:00:07.086098 23215 consensus_queue.cc:237] T a1a39dcd328e4268ae41b988c66b4a75 P 974b802141bd4f13beffef2aa03f88f9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 4.3, Last appended by leader: 3, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } } peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } }
I20260129 03:00:07.201156 20435 raft_consensus.cc:2889] T 7d9494f5584d4d5db8b9adf1c8fc888b P 197b880fa39d432a9b472845876bfaf0 [term 4 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260129 03:00:07.104491 23292 raft_consensus.cc:515] T b189f08e16a74a11980cc55d99359a0a P 1c50de89a5a845668696dcb28c3badd6 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 03:00:07.201869 20432 raft_consensus.cc:2468] T 368953c9102643b2a21f99af36b51905 P 197b880fa39d432a9b472845876bfaf0 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 974b802141bd4f13beffef2aa03f88f9 in term 8.
W20260129 03:00:07.202507 20553 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.211.1:42027, user_credentials={real_user=slave}} blocked reactor thread for 44014.4us
I20260129 03:00:07.198750 20433 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "78aa39096868415d9afc2ffcb248e198" candidate_uuid: "1c50de89a5a845668696dcb28c3badd6" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "197b880fa39d432a9b472845876bfaf0" is_pre_election: true
I20260129 03:00:07.205654 20589 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "cb4727386a3e48aca5d71cc6c611c197" candidate_uuid: "197b880fa39d432a9b472845876bfaf0" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "1c50de89a5a845668696dcb28c3badd6" is_pre_election: true
I20260129 03:00:07.181375 22855 raft_consensus.cc:493] T 4f5a77e94fef4b16a2ba7b067c8b144a P 1c50de89a5a845668696dcb28c3badd6 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260129 03:00:07.215492 20479 leader_election.cc:304] T 368953c9102643b2a21f99af36b51905 P 974b802141bd4f13beffef2aa03f88f9 [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 197b880fa39d432a9b472845876bfaf0, 974b802141bd4f13beffef2aa03f88f9; no voters:
I20260129 03:00:07.220546 20432 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "17d6431cc77a491e8464d68db4081a51" candidate_uuid: "1c50de89a5a845668696dcb28c3badd6" candidate_term: 5 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "197b880fa39d432a9b472845876bfaf0" is_pre_election: true
I20260129 03:00:07.216455 20433 raft_consensus.cc:2468] T 78aa39096868415d9afc2ffcb248e198 P 197b880fa39d432a9b472845876bfaf0 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1c50de89a5a845668696dcb28c3badd6 in term 5.
I20260129 03:00:07.224033 20440 raft_consensus.cc:2468] T 76f713c3331540d2a5b76164103e9eb1 P 197b880fa39d432a9b472845876bfaf0 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 974b802141bd4f13beffef2aa03f88f9 in term 6.
I20260129 03:00:07.225672 20432 raft_consensus.cc:2393] T 17d6431cc77a491e8464d68db4081a51 P 197b880fa39d432a9b472845876bfaf0 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1c50de89a5a845668696dcb28c3badd6 in current term 5: Already voted for candidate 197b880fa39d432a9b472845876bfaf0 in this term.
W20260129 03:00:07.203542 23313 log.cc:927] Time spent T a609ac02fae84725ab743f95f51d33ce P 197b880fa39d432a9b472845876bfaf0: Append to log took a long time: real 0.242s user 0.001s sys 0.000s
I20260129 03:00:07.195497 23097 raft_consensus.cc:493] T 75b0b6e82ddc4f92b1fe1d94f9a0b73b P 974b802141bd4f13beffef2aa03f88f9 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260129 03:00:07.131472 23239 raft_consensus.cc:515] T add8d4923d1a489e86882c3a25692b95 P 7a9daf5d42a84615a6383cfec9a10608 [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } }
I20260129 03:00:07.195710 22694 raft_consensus.cc:493] T fcddab6bca684adf8ea032289a00f90e P 974b802141bd4f13beffef2aa03f88f9 [term 9 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260129 03:00:07.198235 23035 raft_consensus.cc:697] T 4f5a77e94fef4b16a2ba7b067c8b144a P 7a9daf5d42a84615a6383cfec9a10608 [term 5 LEADER]: Becoming Leader. State: Replica: 7a9daf5d42a84615a6383cfec9a10608, State: Running, Role: LEADER
I20260129 03:00:07.309211 21134 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "78aa39096868415d9afc2ffcb248e198" candidate_uuid: "197b880fa39d432a9b472845876bfaf0" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "7a9daf5d42a84615a6383cfec9a10608" is_pre_election: true
I20260129 03:00:07.234737 21133 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "18f30ebaa4174fd28c958e72d0cc7e24" candidate_uuid: "197b880fa39d432a9b472845876bfaf0" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "7a9daf5d42a84615a6383cfec9a10608"
I20260129 03:00:07.206760 23304 raft_consensus.cc:2749] T 4f8f3a38f6e24ba4857875181f447442 P 1c50de89a5a845668696dcb28c3badd6 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260129 03:00:07.312465 21133 raft_consensus.cc:2393] T 18f30ebaa4174fd28c958e72d0cc7e24 P 7a9daf5d42a84615a6383cfec9a10608 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 197b880fa39d432a9b472845876bfaf0 in current term 7: Already voted for candidate 7a9daf5d42a84615a6383cfec9a10608 in this term.
I20260129 03:00:07.297571 21135 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "febffe4a8b394368b69b1d164308fb9a" candidate_uuid: "197b880fa39d432a9b472845876bfaf0" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "7a9daf5d42a84615a6383cfec9a10608" is_pre_election: true
W20260129 03:00:07.302021 20407 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260129 03:00:07.308557 20480 leader_election.cc:304] T 4e420f90884f4796997b5ea41e49e411 P 974b802141bd4f13beffef2aa03f88f9 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 974b802141bd4f13beffef2aa03f88f9; no voters: 197b880fa39d432a9b472845876bfaf0, 7a9daf5d42a84615a6383cfec9a10608
I20260129 03:00:07.299827 20589 raft_consensus.cc:1240] T 7d9494f5584d4d5db8b9adf1c8fc888b P 1c50de89a5a845668696dcb28c3badd6 [term 4 LEADER]: Rejecting Update request from peer 197b880fa39d432a9b472845876bfaf0 for earlier term 3. Current term is 4. Ops: []
I20260129 03:00:07.313061 21135 raft_consensus.cc:2468] T febffe4a8b394368b69b1d164308fb9a P 7a9daf5d42a84615a6383cfec9a10608 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 197b880fa39d432a9b472845876bfaf0 in term 4.
W20260129 03:00:07.315680 20405 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260129 03:00:07.316517 21134 raft_consensus.cc:2393] T 78aa39096868415d9afc2ffcb248e198 P 7a9daf5d42a84615a6383cfec9a10608 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 197b880fa39d432a9b472845876bfaf0 in current term 6: Already voted for candidate 7a9daf5d42a84615a6383cfec9a10608 in this term.
W20260129 03:00:07.318535 20405 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260129 03:00:07.318467 20553 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.211.1:42027, user_credentials={real_user=slave}} blocked reactor thread for 50956.4us
I20260129 03:00:07.320101 20553 leader_election.cc:304] T 36ff33be666a4c82954731691dbcf327 P 1c50de89a5a845668696dcb28c3badd6 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 197b880fa39d432a9b472845876bfaf0, 1c50de89a5a845668696dcb28c3badd6; no voters: 974b802141bd4f13beffef2aa03f88f9
I20260129 03:00:07.321636 20553 leader_election.cc:304] T c891d2bff4ce4568861bbc992e91fbb0 P 1c50de89a5a845668696dcb28c3badd6 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1c50de89a5a845668696dcb28c3badd6; no voters: 197b880fa39d432a9b472845876bfaf0, 974b802141bd4f13beffef2aa03f88f9
I20260129 03:00:07.318392 21132 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a609ac02fae84725ab743f95f51d33ce" candidate_uuid: "197b880fa39d432a9b472845876bfaf0" candidate_term: 9 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "7a9daf5d42a84615a6383cfec9a10608" is_pre_election: true
I20260129 03:00:07.327458 20582 raft_consensus.cc:2468] T 7a9a69cd82724b1faed1d10aa091fc4a P 1c50de89a5a845668696dcb28c3badd6 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 197b880fa39d432a9b472845876bfaf0 in term 7.
I20260129 03:00:07.338692 20553 leader_election.cc:304] T 724a236524cb4e7ba3428880681d6cc2 P 1c50de89a5a845668696dcb28c3badd6 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 197b880fa39d432a9b472845876bfaf0, 1c50de89a5a845668696dcb28c3badd6; no voters: 974b802141bd4f13beffef2aa03f88f9
I20260129 03:00:07.328337 20509 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "cb0b3f259ebc4310927338a8740e87b8" candidate_uuid: "1c50de89a5a845668696dcb28c3badd6" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "974b802141bd4f13beffef2aa03f88f9"
I20260129 03:00:07.339113 20480 leader_election.cc:304] T 96609ac68bdf4993a12e1c23355ca013 P 974b802141bd4f13beffef2aa03f88f9 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 974b802141bd4f13beffef2aa03f88f9; no voters: 197b880fa39d432a9b472845876bfaf0, 7a9daf5d42a84615a6383cfec9a10608
I20260129 03:00:07.344749 20553 leader_election.cc:304] T 3906b080a5b34a74b8b64128635e8997 P 1c50de89a5a845668696dcb28c3badd6 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1c50de89a5a845668696dcb28c3badd6; no voters: 197b880fa39d432a9b472845876bfaf0, 974b802141bd4f13beffef2aa03f88f9
I20260129 03:00:07.309175 22989 raft_consensus.cc:2804] T 368953c9102643b2a21f99af36b51905 P 974b802141bd4f13beffef2aa03f88f9 [term 8 FOLLOWER]: Leader election won for term 8
I20260129 03:00:07.309633 22694 raft_consensus.cc:3060] T fcddab6bca684adf8ea032289a00f90e P 974b802141bd4f13beffef2aa03f88f9 [term 9 FOLLOWER]: Advancing to term 10
I20260129 03:00:07.349933 20553 leader_election.cc:304] T 78aa39096868415d9afc2ffcb248e198 P 1c50de89a5a845668696dcb28c3badd6 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 197b880fa39d432a9b472845876bfaf0, 1c50de89a5a845668696dcb28c3badd6; no voters:
I20260129 03:00:07.351159 20583 raft_consensus.cc:1240] T cedce331e1584ac8b636c9b8a72b2ba0 P 1c50de89a5a845668696dcb28c3badd6 [term 7 FOLLOWER]: Rejecting Update request from peer 197b880fa39d432a9b472845876bfaf0 for earlier term 6. Current term is 7. Ops: [6.4-6.4]
I20260129 03:00:07.298379 21129 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "87d5fe48a44a408aaf03e894a67ecb2f" candidate_uuid: "197b880fa39d432a9b472845876bfaf0" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "7a9daf5d42a84615a6383cfec9a10608" is_pre_election: true
I20260129 03:00:07.357527 20589 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "18f30ebaa4174fd28c958e72d0cc7e24" candidate_uuid: "197b880fa39d432a9b472845876bfaf0" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "1c50de89a5a845668696dcb28c3badd6"
I20260129 03:00:07.230947 22855 raft_consensus.cc:515] T 4f5a77e94fef4b16a2ba7b067c8b144a P 1c50de89a5a845668696dcb28c3badd6 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } }
W20260129 03:00:07.368920 20405 connection.cc:232] RPC call timeout handler was delayed by -1.681185279000033s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.3251s for a response
I20260129 03:00:07.268424 23097 raft_consensus.cc:515] T 75b0b6e82ddc4f92b1fe1d94f9a0b73b P 974b802141bd4f13beffef2aa03f88f9 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } }
I20260129 03:00:07.350313 23290 raft_consensus.cc:2804] T 36ff33be666a4c82954731691dbcf327 P 1c50de89a5a845668696dcb28c3badd6 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260129 03:00:07.376888 20440 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1299723a4476420ab02fd8bfd7aa0645" candidate_uuid: "1c50de89a5a845668696dcb28c3badd6" candidate_term: 10 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "197b880fa39d432a9b472845876bfaf0" is_pre_election: true
I20260129 03:00:07.351946 23102 raft_consensus.cc:493] T 78aa39096868415d9afc2ffcb248e198 P 7a9daf5d42a84615a6383cfec9a10608 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260129 03:00:07.376350 21131 raft_consensus.cc:1275] T a644c02b16af47679824ccd060ac10fd P 7a9daf5d42a84615a6383cfec9a10608 [term 5 FOLLOWER]: Refusing update from remote peer 1c50de89a5a845668696dcb28c3badd6: Log matching property violated. Preceding OpId in replica: term: 4 index: 6. Preceding OpId from leader: term: 5 index: 7. (index mismatch)
I20260129 03:00:07.357172 22996 raft_consensus.cc:2749] T 3906b080a5b34a74b8b64128635e8997 P 1c50de89a5a845668696dcb28c3badd6 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260129 03:00:07.377648 20440 raft_consensus.cc:2393] T 1299723a4476420ab02fd8bfd7aa0645 P 197b880fa39d432a9b472845876bfaf0 [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1c50de89a5a845668696dcb28c3badd6 in current term 10: Already voted for candidate 7a9daf5d42a84615a6383cfec9a10608 in this term.
I20260129 03:00:07.349442 23147 raft_consensus.cc:3060] T c891d2bff4ce4568861bbc992e91fbb0 P 1c50de89a5a845668696dcb28c3badd6 [term 7 FOLLOWER]: Advancing to term 8
I20260129 03:00:07.344842 23087 raft_consensus.cc:493] T 4f8f3a38f6e24ba4857875181f447442 P 197b880fa39d432a9b472845876bfaf0 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 974b802141bd4f13beffef2aa03f88f9)
I20260129 03:00:07.350127 23309 leader_election.cc:290] T 3ccac82a01a2445fa5b0ce6c0fcf2295 P 974b802141bd4f13beffef2aa03f88f9 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 197b880fa39d432a9b472845876bfaf0 (127.18.211.1:42027), 7a9daf5d42a84615a6383cfec9a10608 (127.18.211.4:40439)
I20260129 03:00:07.386132 20582 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "913fc7d751ed4b7388d889aac30a5f9b" candidate_uuid: "197b880fa39d432a9b472845876bfaf0" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "1c50de89a5a845668696dcb28c3badd6" is_pre_election: true
I20260129 03:00:07.394589 20582 raft_consensus.cc:2393] T 913fc7d751ed4b7388d889aac30a5f9b P 1c50de89a5a845668696dcb28c3badd6 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 197b880fa39d432a9b472845876bfaf0 in current term 5: Already voted for candidate 974b802141bd4f13beffef2aa03f88f9 in this term.
I20260129 03:00:07.394775 20509 raft_consensus.cc:1240] T 505580c93c694bbb8ea96fdf9b17ab46 P 974b802141bd4f13beffef2aa03f88f9 [term 7 FOLLOWER]: Rejecting Update request from peer 7a9daf5d42a84615a6383cfec9a10608 for earlier term 6. Current term is 7. Ops: [6.8-6.8]
I20260129 03:00:07.392688 20589 raft_consensus.cc:2468] T 18f30ebaa4174fd28c958e72d0cc7e24 P 1c50de89a5a845668696dcb28c3badd6 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 197b880fa39d432a9b472845876bfaf0 in term 7.
W20260129 03:00:07.397323 20479 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.211.1:42027, user_credentials={real_user=slave}} blocked reactor thread for 181923us
I20260129 03:00:07.402103 20405 leader_election.cc:400] T 42d27a28d89b409e8807ec0efd97af69 P 197b880fa39d432a9b472845876bfaf0 [CANDIDATE]: Term 7 pre-election: Vote denied by peer 7a9daf5d42a84615a6383cfec9a10608 (127.18.211.4:40439) with higher term. Message: Invalid argument: T 42d27a28d89b409e8807ec0efd97af69 P 7a9daf5d42a84615a6383cfec9a10608 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 197b880fa39d432a9b472845876bfaf0 for earlier term 7. Current term is 8.
I20260129 03:00:07.402570 20405 leader_election.cc:403] T 42d27a28d89b409e8807ec0efd97af69 P 197b880fa39d432a9b472845876bfaf0 [CANDIDATE]: Term 7 pre-election: Cancelling election due to peer responding with higher term
I20260129 03:00:07.368060 22989 raft_consensus.cc:697] T 368953c9102643b2a21f99af36b51905 P 974b802141bd4f13beffef2aa03f88f9 [term 8 LEADER]: Becoming Leader. State: Replica: 974b802141bd4f13beffef2aa03f88f9, State: Running, Role: LEADER
I20260129 03:00:07.372613 23262 leader_election.cc:290] T 1299723a4476420ab02fd8bfd7aa0645 P 1c50de89a5a845668696dcb28c3badd6 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 197b880fa39d432a9b472845876bfaf0 (127.18.211.1:42027), 7a9daf5d42a84615a6383cfec9a10608 (127.18.211.4:40439)
I20260129 03:00:07.377976 21129 raft_consensus.cc:2468] T 87d5fe48a44a408aaf03e894a67ecb2f P 7a9daf5d42a84615a6383cfec9a10608 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 197b880fa39d432a9b472845876bfaf0 in term 6.
I20260129 03:00:07.417415 20583 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3906b080a5b34a74b8b64128635e8997" candidate_uuid: "197b880fa39d432a9b472845876bfaf0" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "1c50de89a5a845668696dcb28c3badd6" is_pre_election: true
I20260129 03:00:07.418392 20583 raft_consensus.cc:2468] T 3906b080a5b34a74b8b64128635e8997 P 1c50de89a5a845668696dcb28c3badd6 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 197b880fa39d432a9b472845876bfaf0 in term 4.
I20260129 03:00:07.374024 21127 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0678b8b468bd43cdbc3e485900844821" candidate_uuid: "197b880fa39d432a9b472845876bfaf0" candidate_term: 9 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "7a9daf5d42a84615a6383cfec9a10608" is_pre_election: true
W20260129 03:00:07.424440 20407 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.211.2:34517, user_credentials={real_user=slave}} blocked reactor thread for 111162us
I20260129 03:00:07.376997 23290 raft_consensus.cc:493] T 36ff33be666a4c82954731691dbcf327 P 1c50de89a5a845668696dcb28c3badd6 [term 3 FOLLOWER]: Starting leader election (detected failure of leader 974b802141bd4f13beffef2aa03f88f9)
I20260129 03:00:07.333727 23208 consensus_queue.cc:260] T 17d6431cc77a491e8464d68db4081a51 P 974b802141bd4f13beffef2aa03f88f9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 4.2, Last appended by leader: 2, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 03:00:07.376796 22891 leader_election.cc:290] T 45cc5f3d2688486d9b6dbe7e666dedde P 1c50de89a5a845668696dcb28c3badd6 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 974b802141bd4f13beffef2aa03f88f9 (127.18.211.2:34517), 7a9daf5d42a84615a6383cfec9a10608 (127.18.211.4:40439)
I20260129 03:00:07.383812 22986 raft_consensus.cc:2749] T 96609ac68bdf4993a12e1c23355ca013 P 974b802141bd4f13beffef2aa03f88f9 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260129 03:00:07.383551 23178 raft_consensus.cc:493] T dc2f4cb752cf4d59b5f6bc689f5d8104 P 197b880fa39d432a9b472845876bfaf0 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 1c50de89a5a845668696dcb28c3badd6)
I20260129 03:00:07.385339 23280 leader_election.cc:290] T a4c1226b987b468fa84dfc6ba0e2dbe1 P 1c50de89a5a845668696dcb28c3badd6 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 197b880fa39d432a9b472845876bfaf0 (127.18.211.1:42027), 974b802141bd4f13beffef2aa03f88f9 (127.18.211.2:34517)
W20260129 03:00:07.435143 20407 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260129 03:00:07.393253 21130 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3ccac82a01a2445fa5b0ce6c0fcf2295" candidate_uuid: "974b802141bd4f13beffef2aa03f88f9" candidate_term: 5 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "7a9daf5d42a84615a6383cfec9a10608" is_pre_election: true
I20260129 03:00:07.438674 20440 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a4c1226b987b468fa84dfc6ba0e2dbe1" candidate_uuid: "1c50de89a5a845668696dcb28c3badd6" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "197b880fa39d432a9b472845876bfaf0" is_pre_election: true
W20260129 03:00:07.423290 23059 consensus_meta.cc:287] T 0678b8b468bd43cdbc3e485900844821 P 7a9daf5d42a84615a6383cfec9a10608: Time spent flushing consensus metadata: real 0.632s user 0.009s sys 0.004s
I20260129 03:00:07.415633 22932 consensus_queue.cc:1059] T 505580c93c694bbb8ea96fdf9b17ab46 P 7a9daf5d42a84615a6383cfec9a10608 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 }, Status: INVALID_TERM, Last received: 5.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.026s
I20260129 03:00:07.441867 23325 raft_consensus.cc:2804] T 78aa39096868415d9afc2ffcb248e198 P 1c50de89a5a845668696dcb28c3badd6 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260129 03:00:07.451152 19276 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260129 03:00:07.452503 20432 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b189f08e16a74a11980cc55d99359a0a" candidate_uuid: "1c50de89a5a845668696dcb28c3badd6" candidate_term: 10 candidate_status { last_received { term: 8 index: 4 } } ignore_live_leader: false dest_uuid: "197b880fa39d432a9b472845876bfaf0" is_pre_election: true
I20260129 03:00:07.454377 20432 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3ccac82a01a2445fa5b0ce6c0fcf2295" candidate_uuid: "974b802141bd4f13beffef2aa03f88f9" candidate_term: 5 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "197b880fa39d432a9b472845876bfaf0" is_pre_election: true
W20260129 03:00:07.455396 20405 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.211.4:40439, user_credentials={real_user=slave}} blocked reactor thread for 66890.9us
W20260129 03:00:07.444697 23134 consensus_meta.cc:287] T 8055232cf0004d40a85f3bf983fe00a9 P 197b880fa39d432a9b472845876bfaf0: Time spent flushing consensus metadata: real 0.863s user 0.007s sys 0.007s
I20260129 03:00:07.440800 23290 raft_consensus.cc:3060] T 36ff33be666a4c82954731691dbcf327 P 1c50de89a5a845668696dcb28c3badd6 [term 3 FOLLOWER]: Advancing to term 4
I20260129 03:00:07.505029 20582 raft_consensus.cc:1275] T 1ce4ddf5b10e4ae8b193b13b2e5936ee P 1c50de89a5a845668696dcb28c3badd6 [term 7 FOLLOWER]: Refusing update from remote peer 197b880fa39d432a9b472845876bfaf0: Log matching property violated. Preceding OpId in replica: term: 6 index: 4. Preceding OpId from leader: term: 7 index: 4. (term mismatch)
W20260129 03:00:07.508304 20406 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260129 03:00:07.509950 20440 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "926a250fa5834e88b96ccecbefbf5086" candidate_uuid: "974b802141bd4f13beffef2aa03f88f9" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "197b880fa39d432a9b472845876bfaf0" is_pre_election: true
I20260129 03:00:07.447791 23046 raft_consensus.cc:2749] T 4e420f90884f4796997b5ea41e49e411 P 974b802141bd4f13beffef2aa03f88f9 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
W20260129 03:00:07.459154 23324 log.cc:927] Time spent T cb0b3f259ebc4310927338a8740e87b8 P 974b802141bd4f13beffef2aa03f88f9: Append to log took a long time: real 0.275s user 0.000s sys 0.001s
I20260129 03:00:07.506826 20582 pending_rounds.cc:85] T 1ce4ddf5b10e4ae8b193b13b2e5936ee P 1c50de89a5a845668696dcb28c3badd6: Aborting all ops after (but not including) 3
I20260129 03:00:07.422101 23087 raft_consensus.cc:515] T 4f8f3a38f6e24ba4857875181f447442 P 197b880fa39d432a9b472845876bfaf0 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 03:00:07.538863 23298 raft_consensus.cc:2804] T 724a236524cb4e7ba3428880681d6cc2 P 1c50de89a5a845668696dcb28c3badd6 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260129 03:00:07.538700 22989 consensus_queue.cc:237] T 368953c9102643b2a21f99af36b51905 P 974b802141bd4f13beffef2aa03f88f9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 6.4, Last appended by leader: 4, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 03:00:07.539383 23298 raft_consensus.cc:493] T 724a236524cb4e7ba3428880681d6cc2 P 1c50de89a5a845668696dcb28c3badd6 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260129 03:00:07.539762 23298 raft_consensus.cc:3060] T 724a236524cb4e7ba3428880681d6cc2 P 1c50de89a5a845668696dcb28c3badd6 [term 5 FOLLOWER]: Advancing to term 6
I20260129 03:00:07.540732 23087 leader_election.cc:290] T 4f8f3a38f6e24ba4857875181f447442 P 197b880fa39d432a9b472845876bfaf0 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 974b802141bd4f13beffef2aa03f88f9 (127.18.211.2:34517), 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3:37735)
I20260129 03:00:07.502072 23178 raft_consensus.cc:515] T dc2f4cb752cf4d59b5f6bc689f5d8104 P 197b880fa39d432a9b472845876bfaf0 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } }
I20260129 03:00:07.543241 23097 leader_election.cc:290] T 75b0b6e82ddc4f92b1fe1d94f9a0b73b P 974b802141bd4f13beffef2aa03f88f9 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3:37735), 7a9daf5d42a84615a6383cfec9a10608 (127.18.211.4:40439)
W20260129 03:00:07.543980 23331 log.cc:927] Time spent T a644c02b16af47679824ccd060ac10fd P 1c50de89a5a845668696dcb28c3badd6: Append to log took a long time: real 0.174s user 0.001s sys 0.000s
I20260129 03:00:07.544286 23178 leader_election.cc:290] T dc2f4cb752cf4d59b5f6bc689f5d8104 P 197b880fa39d432a9b472845876bfaf0 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3:37735), 7a9daf5d42a84615a6383cfec9a10608 (127.18.211.4:40439)
W20260129 03:00:07.516913 20482 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260129 03:00:07.452956 23325 raft_consensus.cc:493] T 78aa39096868415d9afc2ffcb248e198 P 1c50de89a5a845668696dcb28c3badd6 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260129 03:00:07.536686 20589 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a8479d1fe0bd41e5946186e3a6ca8c7a" candidate_uuid: "197b880fa39d432a9b472845876bfaf0" candidate_term: 6 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "1c50de89a5a845668696dcb28c3badd6" is_pre_election: true
I20260129 03:00:07.547813 20589 raft_consensus.cc:2468] T a8479d1fe0bd41e5946186e3a6ca8c7a P 1c50de89a5a845668696dcb28c3badd6 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 197b880fa39d432a9b472845876bfaf0 in term 5.
I20260129 03:00:07.503804 23208 raft_consensus.cc:3060] T 17d6431cc77a491e8464d68db4081a51 P 974b802141bd4f13beffef2aa03f88f9 [term 4 FOLLOWER]: Advancing to term 5
I20260129 03:00:07.453819 23059 raft_consensus.cc:2749] T 0678b8b468bd43cdbc3e485900844821 P 7a9daf5d42a84615a6383cfec9a10608 [term 8 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260129 03:00:07.535892 20582 pending_rounds.cc:107] T 1ce4ddf5b10e4ae8b193b13b2e5936ee P 1c50de89a5a845668696dcb28c3badd6: Aborting uncommitted NO_OP operation due to leader change: 6.4
I20260129 03:00:07.502144 23100 raft_consensus.cc:3055] T 505580c93c694bbb8ea96fdf9b17ab46 P 7a9daf5d42a84615a6383cfec9a10608 [term 6 LEADER]: Stepping down as leader of term 6
I20260129 03:00:07.534526 20434 raft_consensus.cc:2468] T c2633a6fc34c44579de348d208b0c01e P 197b880fa39d432a9b472845876bfaf0 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1c50de89a5a845668696dcb28c3badd6 in term 4.
I20260129 03:00:07.502163 23145 leader_election.cc:290] T 926a250fa5834e88b96ccecbefbf5086 P 974b802141bd4f13beffef2aa03f88f9 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 197b880fa39d432a9b472845876bfaf0 (127.18.211.1:42027), 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3:37735)
I20260129 03:00:07.503471 22982 raft_consensus.cc:493] T 2ac659d732314e1288849f08fe5e074e P 974b802141bd4f13beffef2aa03f88f9 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260129 03:00:07.387418 23102 raft_consensus.cc:515] T 78aa39096868415d9afc2ffcb248e198 P 7a9daf5d42a84615a6383cfec9a10608 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } }
I20260129 03:00:07.550043 20582 raft_consensus.cc:2889] T 1ce4ddf5b10e4ae8b193b13b2e5936ee P 1c50de89a5a845668696dcb28c3badd6 [term 7 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260129 03:00:07.508930 22932 raft_consensus.cc:493] T fd8311b779af4e1fa1402b2a02a602f2 P 7a9daf5d42a84615a6383cfec9a10608 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 1c50de89a5a845668696dcb28c3badd6)
I20260129 03:00:07.553846 20434 raft_consensus.cc:1240] T 17d6431cc77a491e8464d68db4081a51 P 197b880fa39d432a9b472845876bfaf0 [term 5 FOLLOWER]: Rejecting Update request from peer 974b802141bd4f13beffef2aa03f88f9 for earlier term 4. Current term is 5. Ops: []
W20260129 03:00:07.571996 20516 consensus_meta.cc:287] T cedce331e1584ac8b636c9b8a72b2ba0 P 974b802141bd4f13beffef2aa03f88f9: Time spent flushing consensus metadata: real 0.520s user 0.008s sys 0.003s
I20260129 03:00:07.530822 23336 raft_consensus.cc:493] T 0678b8b468bd43cdbc3e485900844821 P 974b802141bd4f13beffef2aa03f88f9 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260129 03:00:07.581820 20516 raft_consensus.cc:2468] T cedce331e1584ac8b636c9b8a72b2ba0 P 974b802141bd4f13beffef2aa03f88f9 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 197b880fa39d432a9b472845876bfaf0 in term 8.
I20260129 03:00:07.440456 23035 consensus_queue.cc:237] T 4f5a77e94fef4b16a2ba7b067c8b144a P 7a9daf5d42a84615a6383cfec9a10608 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.5, Last appended by leader: 5, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } }
W20260129 03:00:07.581022 20480 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.211.3:37735, user_credentials={real_user=slave}} blocked reactor thread for 76535.3us
I20260129 03:00:07.594089 20582 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a0fcc58d6fb6473cafbd9d34d5195269" candidate_uuid: "197b880fa39d432a9b472845876bfaf0" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "1c50de89a5a845668696dcb28c3badd6" is_pre_election: true
I20260129 03:00:07.551216 23325 raft_consensus.cc:3060] T 78aa39096868415d9afc2ffcb248e198 P 1c50de89a5a845668696dcb28c3badd6 [term 6 FOLLOWER]: Advancing to term 7
I20260129 03:00:07.597016 21130 raft_consensus.cc:1275] T febffe4a8b394368b69b1d164308fb9a P 7a9daf5d42a84615a6383cfec9a10608 [term 4 FOLLOWER]: Refusing update from remote peer 974b802141bd4f13beffef2aa03f88f9: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
W20260129 03:00:07.598018 23251 consensus_meta.cc:287] T add8d4923d1a489e86882c3a25692b95 P 1c50de89a5a845668696dcb28c3badd6: Time spent flushing consensus metadata: real 1.015s user 0.011s sys 0.008s
I20260129 03:00:07.574311 21136 raft_consensus.cc:2468] T 00e79d8f25a34676b1d866eb2068a2b2 P 7a9daf5d42a84615a6383cfec9a10608 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 197b880fa39d432a9b472845876bfaf0 in term 8.
I20260129 03:00:07.584744 20516 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4f8f3a38f6e24ba4857875181f447442" candidate_uuid: "197b880fa39d432a9b472845876bfaf0" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "974b802141bd4f13beffef2aa03f88f9" is_pre_election: true
I20260129 03:00:07.585000 20407 leader_election.cc:304] T cedce331e1584ac8b636c9b8a72b2ba0 P 197b880fa39d432a9b472845876bfaf0 [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 197b880fa39d432a9b472845876bfaf0, 974b802141bd4f13beffef2aa03f88f9; no voters:
I20260129 03:00:07.600811 23344 raft_consensus.cc:2804] T cedce331e1584ac8b636c9b8a72b2ba0 P 197b880fa39d432a9b472845876bfaf0 [term 8 FOLLOWER]: Leader election won for term 8
I20260129 03:00:07.601341 23344 raft_consensus.cc:697] T cedce331e1584ac8b636c9b8a72b2ba0 P 197b880fa39d432a9b472845876bfaf0 [term 8 LEADER]: Becoming Leader. State: Replica: 197b880fa39d432a9b472845876bfaf0, State: Running, Role: LEADER
I20260129 03:00:07.601442 20516 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "add8d4923d1a489e86882c3a25692b95" candidate_uuid: "7a9daf5d42a84615a6383cfec9a10608" candidate_term: 7 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "974b802141bd4f13beffef2aa03f88f9"
I20260129 03:00:07.595693 20582 raft_consensus.cc:2468] T a0fcc58d6fb6473cafbd9d34d5195269 P 1c50de89a5a845668696dcb28c3badd6 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 197b880fa39d432a9b472845876bfaf0 in term 4.
I20260129 03:00:07.555982 23100 raft_consensus.cc:740] T 505580c93c694bbb8ea96fdf9b17ab46 P 7a9daf5d42a84615a6383cfec9a10608 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 7a9daf5d42a84615a6383cfec9a10608, State: Running, Role: LEADER
I20260129 03:00:07.610026 23100 consensus_queue.cc:260] T 505580c93c694bbb8ea96fdf9b17ab46 P 7a9daf5d42a84615a6383cfec9a10608 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 6.8, Last appended by leader: 8, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } }
I20260129 03:00:07.609684 23109 raft_consensus.cc:515] T 45cc5f3d2688486d9b6dbe7e666dedde P 7a9daf5d42a84615a6383cfec9a10608 [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } }
I20260129 03:00:07.536885 22978 leader_election.cc:290] T 87d5fe48a44a408aaf03e894a67ecb2f P 974b802141bd4f13beffef2aa03f88f9 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 197b880fa39d432a9b472845876bfaf0 (127.18.211.1:42027), 7a9daf5d42a84615a6383cfec9a10608 (127.18.211.4:40439)
I20260129 03:00:07.559669 23292 leader_election.cc:290] T b189f08e16a74a11980cc55d99359a0a P 1c50de89a5a845668696dcb28c3badd6 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 197b880fa39d432a9b472845876bfaf0 (127.18.211.1:42027), 974b802141bd4f13beffef2aa03f88f9 (127.18.211.2:34517)
I20260129 03:00:07.549837 23239 leader_election.cc:290] T add8d4923d1a489e86882c3a25692b95 P 7a9daf5d42a84615a6383cfec9a10608 [CANDIDATE]: Term 7 election: Requested vote from peers 974b802141bd4f13beffef2aa03f88f9 (127.18.211.2:34517), 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3:37735)
I20260129 03:00:07.536192 23134 raft_consensus.cc:2749] T 42d27a28d89b409e8807ec0efd97af69 P 197b880fa39d432a9b472845876bfaf0 [term 8 FOLLOWER]: Leader pre-election lost for term 7. Reason: Vote denied by peer 7a9daf5d42a84615a6383cfec9a10608 (127.18.211.4:40439) with higher term. Message: Invalid argument: T 42d27a28d89b409e8807ec0efd97af69 P 7a9daf5d42a84615a6383cfec9a10608 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 197b880fa39d432a9b472845876bfaf0 for earlier term 7. Current term is 8.
I20260129 03:00:07.604694 21128 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:47844 (request call id 368) took 1170 ms. Trace:
W20260129 03:00:07.613517 23327 log.cc:927] Time spent T a8479d1fe0bd41e5946186e3a6ca8c7a P 197b880fa39d432a9b472845876bfaf0: Append to log took a long time: real 0.256s user 0.000s sys 0.002s
I20260129 03:00:07.624701 21128 rpcz_store.cc:276] 0129 03:00:06.434540 (+ 0us) service_pool.cc:168] Inserting onto call queue
0129 03:00:06.437260 (+ 2720us) service_pool.cc:225] Handling call
0129 03:00:07.604462 (+1167202us) spinlock_profiling.cc:243] Waited 21.3 ms on lock 0x7b5400730354. stack: 00007f8fe044e373 00007f8fe044e270 00007f8fdfda0574 00007f8fe9a5c6b4 00007f8fe9a5c649 00007f8fe34975a1 00007f8fe3531f7a 00007f8fe84e3647 00007f8fe1a73834 00007f8fe1a737c2 00007f8fe1a736f1 00007f8fe1a7366c 00007f8fe1a72922 00007f8fe186d34c 00007f8fe186c786 00007f8fe186c12f
0129 03:00:07.604622 (+ 160us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":48934656}
I20260129 03:00:07.602245 23344 consensus_queue.cc:237] T cedce331e1584ac8b636c9b8a72b2ba0 P 197b880fa39d432a9b472845876bfaf0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 6.4, Last appended by leader: 4, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 03:00:07.626499 20516 raft_consensus.cc:3060] T add8d4923d1a489e86882c3a25692b95 P 974b802141bd4f13beffef2aa03f88f9 [term 6 FOLLOWER]: Advancing to term 7
I20260129 03:00:07.606918 21136 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "87d5fe48a44a408aaf03e894a67ecb2f" candidate_uuid: "974b802141bd4f13beffef2aa03f88f9" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "7a9daf5d42a84615a6383cfec9a10608" is_pre_election: true
I20260129 03:00:07.628337 21136 raft_consensus.cc:2393] T 87d5fe48a44a408aaf03e894a67ecb2f P 7a9daf5d42a84615a6383cfec9a10608 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 974b802141bd4f13beffef2aa03f88f9 in current term 6: Already voted for candidate 7a9daf5d42a84615a6383cfec9a10608 in this term.
I20260129 03:00:07.611356 23100 raft_consensus.cc:3060] T 505580c93c694bbb8ea96fdf9b17ab46 P 7a9daf5d42a84615a6383cfec9a10608 [term 6 FOLLOWER]: Advancing to term 7
I20260129 03:00:07.615538 20581 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b189f08e16a74a11980cc55d99359a0a" candidate_uuid: "197b880fa39d432a9b472845876bfaf0" candidate_term: 9 candidate_status { last_received { term: 8 index: 4 } } ignore_live_leader: false dest_uuid: "1c50de89a5a845668696dcb28c3badd6"
I20260129 03:00:07.629807 21136 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "75b0b6e82ddc4f92b1fe1d94f9a0b73b" candidate_uuid: "974b802141bd4f13beffef2aa03f88f9" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "7a9daf5d42a84615a6383cfec9a10608" is_pre_election: true
W20260129 03:00:07.631402 23314 log.cc:927] Time spent T c2633a6fc34c44579de348d208b0c01e P 1c50de89a5a845668696dcb28c3badd6: Append to log took a long time: real 0.966s user 0.005s sys 0.000s
I20260129 03:00:07.633343 20512 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57600 (request call id 1892) took 2266 ms. Trace:
I20260129 03:00:07.633190 23325 raft_consensus.cc:515] T 78aa39096868415d9afc2ffcb248e198 P 1c50de89a5a845668696dcb28c3badd6 [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } }
I20260129 03:00:07.633692 20512 rpcz_store.cc:276] 0129 03:00:05.367374 (+ 0us) service_pool.cc:168] Inserting onto call queue
0129 03:00:05.983642 (+616268us) service_pool.cc:225] Handling call
0129 03:00:06.702310 (+718668us) raft_consensus.cc:1471] Updating replica for 0 ops
0129 03:00:06.723420 (+ 21110us) raft_consensus.cc:1697] Filling consensus response to leader.
0129 03:00:07.633151 (+909731us) spinlock_profiling.cc:243] Waited 679 ms on lock 0x7b5400317250. stack: 00007f8fe044e373 00007f8fe044e270 00007f8fdfda0574 00007f8fe9a5c6b4 00007f8fe9a5c649 00007f8fe34975a1 00007f8fe352d52d 00007f8fe84e1f49 00007f8fe1a6ed34 00007f8fe1a6ecc2 00007f8fe1a6ebf1 00007f8fe1a6eb6c 00007f8fe1a6de22 00007f8fe186d34c 00007f8fe186c786 00007f8fe186c12f
0129 03:00:07.633283 (+ 132us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1561773824}
I20260129 03:00:07.635335 23325 leader_election.cc:290] T 78aa39096868415d9afc2ffcb248e198 P 1c50de89a5a845668696dcb28c3badd6 [CANDIDATE]: Term 7 election: Requested vote from peers 197b880fa39d432a9b472845876bfaf0 (127.18.211.1:42027), 7a9daf5d42a84615a6383cfec9a10608 (127.18.211.4:40439)
I20260129 03:00:07.636569 20441 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:48356 (request call id 396) took 1480 ms. Trace:
I20260129 03:00:07.636978 20441 rpcz_store.cc:276] 0129 03:00:06.156441 (+ 0us) service_pool.cc:168] Inserting onto call queue
0129 03:00:06.286947 (+130506us) service_pool.cc:225] Handling call
0129 03:00:06.339349 (+ 52402us) raft_consensus.cc:1471] Updating replica for 0 ops
0129 03:00:06.364959 (+ 25610us) raft_consensus.cc:1534] Early marking committed up to index 0
0129 03:00:06.364987 (+ 28us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0129 03:00:06.365026 (+ 39us) raft_consensus.cc:1645] Marking committed up to 6
0129 03:00:06.365058 (+ 32us) raft_consensus.cc:1697] Filling consensus response to leader.
0129 03:00:07.636364 (+1271306us) spinlock_profiling.cc:243] Waited 25.4 ms on lock 0x7b54002fc454. stack: 00007f8fe044e373 00007f8fe044e270 00007f8fdfda0574 00007f8fe9a5c6b4 00007f8fe9a5c649 00007f8fe34975a1 00007f8fe352ea81 00007f8fe352d4be 00007f8fe84e1f49 00007f8fe1a6ed34 00007f8fe1a6ecc2 00007f8fe1a6ebf1 00007f8fe1a6eb6c 00007f8fe1a6de22 00007f8fe186d34c 00007f8fe186c786
0129 03:00:07.636409 (+ 45us) raft_consensus.cc:1691] UpdateReplicas() finished
0129 03:00:07.636517 (+ 108us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":58339072}
I20260129 03:00:07.639950 21134 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2e58e9a5a8aa4cecafb464addb4f35ed" candidate_uuid: "197b880fa39d432a9b472845876bfaf0" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "7a9daf5d42a84615a6383cfec9a10608" is_pre_election: true
I20260129 03:00:07.640457 20480 leader_election.cc:400] T 1299723a4476420ab02fd8bfd7aa0645 P 974b802141bd4f13beffef2aa03f88f9 [CANDIDATE]: Term 9 pre-election: Vote denied by peer 7a9daf5d42a84615a6383cfec9a10608 (127.18.211.4:40439) with higher term. Message: Invalid argument: T 1299723a4476420ab02fd8bfd7aa0645 P 7a9daf5d42a84615a6383cfec9a10608 [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 974b802141bd4f13beffef2aa03f88f9 for earlier term 9. Current term is 10.
I20260129 03:00:07.641001 20480 leader_election.cc:403] T 1299723a4476420ab02fd8bfd7aa0645 P 974b802141bd4f13beffef2aa03f88f9 [CANDIDATE]: Term 9 pre-election: Cancelling election due to peer responding with higher term
I20260129 03:00:07.641371 20508 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:57600 (request call id 1907) took 1292 ms. Trace:
I20260129 03:00:07.641680 22978 raft_consensus.cc:3060] T 1299723a4476420ab02fd8bfd7aa0645 P 974b802141bd4f13beffef2aa03f88f9 [term 8 FOLLOWER]: Advancing to term 10
I20260129 03:00:07.641757 20508 rpcz_store.cc:276] 0129 03:00:06.348556 (+ 0us) service_pool.cc:168] Inserting onto call queue
0129 03:00:06.364011 (+ 15455us) service_pool.cc:225] Handling call
0129 03:00:07.641201 (+1277190us) spinlock_profiling.cc:243] Waited 46.2 ms on lock 0x7b54002ee354. stack: 00007f8fe044e373 00007f8fe044e270 00007f8fdfda0574 00007f8fe9a5c6b4 00007f8fe9a5c649 00007f8fe34975a1 00007f8fe3531f7a 00007f8fe84e3647 00007f8fe1a73834 00007f8fe1a737c2 00007f8fe1a736f1 00007f8fe1a7366c 00007f8fe1a72922 00007f8fe186d34c 00007f8fe186c786 00007f8fe186c12f
0129 03:00:07.641312 (+ 111us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":106348416}
I20260129 03:00:07.641670 23152 consensus_queue.cc:1048] T febffe4a8b394368b69b1d164308fb9a P 974b802141bd4f13beffef2aa03f88f9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 4, Time since last communication: 0.002s
I20260129 03:00:07.643795 20581 raft_consensus.cc:2468] T b189f08e16a74a11980cc55d99359a0a P 1c50de89a5a845668696dcb28c3badd6 [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 197b880fa39d432a9b472845876bfaf0 in term 9.
I20260129 03:00:07.612672 23109 leader_election.cc:290] T 45cc5f3d2688486d9b6dbe7e666dedde P 7a9daf5d42a84615a6383cfec9a10608 [CANDIDATE]: Term 6 election: Requested vote from peers 974b802141bd4f13beffef2aa03f88f9 (127.18.211.2:34517), 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3:37735)
I20260129 03:00:07.648135 20514 raft_consensus.cc:3060] T a609ac02fae84725ab743f95f51d33ce P 974b802141bd4f13beffef2aa03f88f9 [term 7 FOLLOWER]: Advancing to term 8
I20260129 03:00:07.651942 22978 raft_consensus.cc:2749] T 1299723a4476420ab02fd8bfd7aa0645 P 974b802141bd4f13beffef2aa03f88f9 [term 10 FOLLOWER]: Leader pre-election lost for term 9. Reason: Vote denied by peer 7a9daf5d42a84615a6383cfec9a10608 (127.18.211.4:40439) with higher term. Message: Invalid argument: T 1299723a4476420ab02fd8bfd7aa0645 P 7a9daf5d42a84615a6383cfec9a10608 [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 974b802141bd4f13beffef2aa03f88f9 for earlier term 9. Current term is 10.
I20260129 03:00:07.655154 20586 raft_consensus.cc:1240] T 505580c93c694bbb8ea96fdf9b17ab46 P 1c50de89a5a845668696dcb28c3badd6 [term 7 LEADER]: Rejecting Update request from peer 7a9daf5d42a84615a6383cfec9a10608 for earlier term 6. Current term is 7. Ops: []
I20260129 03:00:07.655782 20586 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:48526 (request call id 368) took 1490 ms. Trace:
I20260129 03:00:07.656213 20510 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "45cc5f3d2688486d9b6dbe7e666dedde" candidate_uuid: "7a9daf5d42a84615a6383cfec9a10608" candidate_term: 6 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "974b802141bd4f13beffef2aa03f88f9"
I20260129 03:00:07.656795 20510 raft_consensus.cc:3060] T 45cc5f3d2688486d9b6dbe7e666dedde P 974b802141bd4f13beffef2aa03f88f9 [term 5 FOLLOWER]: Advancing to term 6
I20260129 03:00:07.657773 23163 consensus_queue.cc:1048] T 75b0b6e82ddc4f92b1fe1d94f9a0b73b P 7a9daf5d42a84615a6383cfec9a10608 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20260129 03:00:07.659283 23163 consensus_queue.cc:1048] T 3e2d15f8640b407195210c476e09df91 P 7a9daf5d42a84615a6383cfec9a10608 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260129 03:00:07.664726 20516 raft_consensus.cc:2468] T add8d4923d1a489e86882c3a25692b95 P 974b802141bd4f13beffef2aa03f88f9 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7a9daf5d42a84615a6383cfec9a10608 in term 7.
I20260129 03:00:07.611039 23336 raft_consensus.cc:515] T 0678b8b468bd43cdbc3e485900844821 P 974b802141bd4f13beffef2aa03f88f9 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } }
I20260129 03:00:07.665800 21102 leader_election.cc:304] T add8d4923d1a489e86882c3a25692b95 P 7a9daf5d42a84615a6383cfec9a10608 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7a9daf5d42a84615a6383cfec9a10608, 974b802141bd4f13beffef2aa03f88f9; no voters:
I20260129 03:00:07.656093 20586 rpcz_store.cc:276] 0129 03:00:06.164939 (+ 0us) service_pool.cc:168] Inserting onto call queue
0129 03:00:07.654630 (+1489691us) service_pool.cc:225] Handling call
0129 03:00:07.654972 (+ 342us) raft_consensus.cc:1471] Updating replica for 0 ops
0129 03:00:07.655535 (+ 563us) raft_consensus.cc:1697] Filling consensus response to leader.
0129 03:00:07.655710 (+ 175us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260129 03:00:07.669217 20407 leader_election.cc:304] T a609ac02fae84725ab743f95f51d33ce P 197b880fa39d432a9b472845876bfaf0 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 197b880fa39d432a9b472845876bfaf0, 974b802141bd4f13beffef2aa03f88f9; no voters:
W20260129 03:00:07.664968 23207 consensus_meta.cc:287] T 0f20af7fbe5d4db89a6f17948d98912c P 974b802141bd4f13beffef2aa03f88f9: Time spent flushing consensus metadata: real 1.097s user 0.012s sys 0.000s
W20260129 03:00:07.613232 23330 log.cc:927] Time spent T 7d9494f5584d4d5db8b9adf1c8fc888b P 1c50de89a5a845668696dcb28c3badd6: Append to log took a long time: real 0.252s user 0.001s sys 0.000s
W20260129 03:00:07.709945 20553 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.211.1:42027, user_credentials={real_user=slave}} blocked reactor thread for 105061us
I20260129 03:00:07.724200 23032 raft_consensus.cc:493] T c2633a6fc34c44579de348d208b0c01e P 974b802141bd4f13beffef2aa03f88f9 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260129 03:00:07.724649 23332 log.cc:927] Time spent T 3e2d15f8640b407195210c476e09df91 P 974b802141bd4f13beffef2aa03f88f9: Append to log took a long time: real 0.178s user 0.000s sys 0.001s
I20260129 03:00:07.724963 23032 raft_consensus.cc:515] T c2633a6fc34c44579de348d208b0c01e P 974b802141bd4f13beffef2aa03f88f9 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 03:00:07.726754 20507 raft_consensus.cc:1240] T 76f713c3331540d2a5b76164103e9eb1 P 974b802141bd4f13beffef2aa03f88f9 [term 6 FOLLOWER]: Rejecting Update request from peer 1c50de89a5a845668696dcb28c3badd6 for earlier term 5. Current term is 6. Ops: []
I20260129 03:00:07.727496 23032 leader_election.cc:290] T c2633a6fc34c44579de348d208b0c01e P 974b802141bd4f13beffef2aa03f88f9 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 197b880fa39d432a9b472845876bfaf0 (127.18.211.1:42027), 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3:37735)
I20260129 03:00:07.561003 22982 raft_consensus.cc:515] T 2ac659d732314e1288849f08fe5e074e P 974b802141bd4f13beffef2aa03f88f9 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } }
I20260129 03:00:07.728940 23102 leader_election.cc:290] T 78aa39096868415d9afc2ffcb248e198 P 7a9daf5d42a84615a6383cfec9a10608 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 197b880fa39d432a9b472845876bfaf0 (127.18.211.1:42027), 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3:37735)
I20260129 03:00:07.732535 23216 raft_consensus.cc:493] T 0f20af7fbe5d4db89a6f17948d98912c P 974b802141bd4f13beffef2aa03f88f9 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260129 03:00:07.734458 20435 raft_consensus.cc:1240] T 76f713c3331540d2a5b76164103e9eb1 P 197b880fa39d432a9b472845876bfaf0 [term 6 FOLLOWER]: Rejecting Update request from peer 1c50de89a5a845668696dcb28c3badd6 for earlier term 5. Current term is 6. Ops: []
W20260129 03:00:07.741868 20481 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260129 03:00:07.679930 20514 raft_consensus.cc:1275] T a609ac02fae84725ab743f95f51d33ce P 974b802141bd4f13beffef2aa03f88f9 [term 8 FOLLOWER]: Refusing update from remote peer 7a9daf5d42a84615a6383cfec9a10608: Log matching property violated. Preceding OpId in replica: term: 6 index: 7. Preceding OpId from leader: term: 8 index: 8. (index mismatch)
I20260129 03:00:07.618042 22855 leader_election.cc:290] T 4f5a77e94fef4b16a2ba7b067c8b144a P 1c50de89a5a845668696dcb28c3badd6 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 974b802141bd4f13beffef2aa03f88f9 (127.18.211.2:34517), 7a9daf5d42a84615a6383cfec9a10608 (127.18.211.4:40439)
I20260129 03:00:07.780867 20586 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "78aa39096868415d9afc2ffcb248e198" candidate_uuid: "197b880fa39d432a9b472845876bfaf0" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "1c50de89a5a845668696dcb28c3badd6" is_pre_election: true
I20260129 03:00:07.788650 20586 raft_consensus.cc:2368] T 78aa39096868415d9afc2ffcb248e198 P 1c50de89a5a845668696dcb28c3badd6 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 197b880fa39d432a9b472845876bfaf0 for earlier term 6. Current term is 7.
W20260129 03:00:07.768406 20479 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260129 03:00:07.774287 20510 raft_consensus.cc:2468] T 45cc5f3d2688486d9b6dbe7e666dedde P 974b802141bd4f13beffef2aa03f88f9 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7a9daf5d42a84615a6383cfec9a10608 in term 6.
W20260129 03:00:07.774565 20555 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260129 03:00:07.692652 23342 raft_consensus.cc:2804] T a609ac02fae84725ab743f95f51d33ce P 197b880fa39d432a9b472845876bfaf0 [term 8 FOLLOWER]: Leader pre-election won for term 9
W20260129 03:00:07.767583 23199 consensus_meta.cc:287] T c934d31efd3f4e5195605ddc354d14e1 P 974b802141bd4f13beffef2aa03f88f9: Time spent flushing consensus metadata: real 1.238s user 0.007s sys 0.006s
I20260129 03:00:07.580417 22932 raft_consensus.cc:515] T fd8311b779af4e1fa1402b2a02a602f2 P 7a9daf5d42a84615a6383cfec9a10608 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } }
I20260129 03:00:07.784402 20441 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "78aa39096868415d9afc2ffcb248e198" candidate_uuid: "1c50de89a5a845668696dcb28c3badd6" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "197b880fa39d432a9b472845876bfaf0"
I20260129 03:00:07.825210 20441 raft_consensus.cc:3060] T 78aa39096868415d9afc2ffcb248e198 P 197b880fa39d432a9b472845876bfaf0 [term 5 FOLLOWER]: Advancing to term 7
W20260129 03:00:07.809424 21100 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260129 03:00:07.712325 23203 raft_consensus.cc:2804] T add8d4923d1a489e86882c3a25692b95 P 7a9daf5d42a84615a6383cfec9a10608 [term 7 FOLLOWER]: Leader election won for term 7
I20260129 03:00:07.816571 20479 leader_election.cc:304] T 76f713c3331540d2a5b76164103e9eb1 P 974b802141bd4f13beffef2aa03f88f9 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 197b880fa39d432a9b472845876bfaf0, 974b802141bd4f13beffef2aa03f88f9; no voters:
I20260129 03:00:07.825681 20516 raft_consensus.cc:1275] T 2ac659d732314e1288849f08fe5e074e P 974b802141bd4f13beffef2aa03f88f9 [term 5 FOLLOWER]: Refusing update from remote peer 7a9daf5d42a84615a6383cfec9a10608: Log matching property violated. Preceding OpId in replica: term: 3 index: 5. Preceding OpId from leader: term: 5 index: 6. (index mismatch)
I20260129 03:00:07.707065 22857 leader_election.cc:290] T 1299723a4476420ab02fd8bfd7aa0645 P 7a9daf5d42a84615a6383cfec9a10608 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 197b880fa39d432a9b472845876bfaf0 (127.18.211.1:42027), 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3:37735)
I20260129 03:00:07.827709 20585 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a644c02b16af47679824ccd060ac10fd" candidate_uuid: "7a9daf5d42a84615a6383cfec9a10608" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "1c50de89a5a845668696dcb28c3badd6" is_pre_election: true
W20260129 03:00:07.784119 23324 log.cc:927] Time spent T cb0b3f259ebc4310927338a8740e87b8 P 974b802141bd4f13beffef2aa03f88f9: Append to log took a long time: real 0.096s user 0.000s sys 0.000s
I20260129 03:00:07.831343 20479 leader_election.cc:304] T 3ccac82a01a2445fa5b0ce6c0fcf2295 P 974b802141bd4f13beffef2aa03f88f9 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 974b802141bd4f13beffef2aa03f88f9; no voters: 197b880fa39d432a9b472845876bfaf0, 7a9daf5d42a84615a6383cfec9a10608
I20260129 03:00:07.832290 20582 raft_consensus.cc:1240] T cb0b3f259ebc4310927338a8740e87b8 P 1c50de89a5a845668696dcb28c3badd6 [term 8 FOLLOWER]: Rejecting Update request from peer 7a9daf5d42a84615a6383cfec9a10608 for earlier term 7. Current term is 8. Ops: []
I20260129 03:00:07.623492 23298 raft_consensus.cc:515] T 724a236524cb4e7ba3428880681d6cc2 P 1c50de89a5a845668696dcb28c3badd6 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 03:00:07.838120 20582 raft_consensus.cc:1275] T 4e3805ffa70242e8ade66da260dddf66 P 1c50de89a5a845668696dcb28c3badd6 [term 6 FOLLOWER]: Refusing update from remote peer 7a9daf5d42a84615a6383cfec9a10608: Log matching property violated. Preceding OpId in replica: term: 5 index: 7. Preceding OpId from leader: term: 6 index: 8. (index mismatch)
I20260129 03:00:07.786864 23325 raft_consensus.cc:493] T 926a250fa5834e88b96ccecbefbf5086 P 1c50de89a5a845668696dcb28c3badd6 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260129 03:00:07.839811 20405 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.211.4:40439, user_credentials={real_user=slave}} blocked reactor thread for 383196us
I20260129 03:00:07.843200 22989 raft_consensus.cc:2804] T 76f713c3331540d2a5b76164103e9eb1 P 974b802141bd4f13beffef2aa03f88f9 [term 6 FOLLOWER]: Leader election won for term 6
I20260129 03:00:07.851411 20589 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1299723a4476420ab02fd8bfd7aa0645" candidate_uuid: "7a9daf5d42a84615a6383cfec9a10608" candidate_term: 10 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "1c50de89a5a845668696dcb28c3badd6"
I20260129 03:00:07.852263 20589 raft_consensus.cc:3060] T 1299723a4476420ab02fd8bfd7aa0645 P 1c50de89a5a845668696dcb28c3badd6 [term 9 FOLLOWER]: Advancing to term 10
I20260129 03:00:07.818419 23342 raft_consensus.cc:493] T a609ac02fae84725ab743f95f51d33ce P 197b880fa39d432a9b472845876bfaf0 [term 8 FOLLOWER]: Starting leader election (detected failure of leader 7a9daf5d42a84615a6383cfec9a10608)
I20260129 03:00:07.864145 20582 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "17d6431cc77a491e8464d68db4081a51" candidate_uuid: "197b880fa39d432a9b472845876bfaf0" candidate_term: 5 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "1c50de89a5a845668696dcb28c3badd6"
I20260129 03:00:07.864781 20582 raft_consensus.cc:3060] T 17d6431cc77a491e8464d68db4081a51 P 1c50de89a5a845668696dcb28c3badd6 [term 4 FOLLOWER]: Advancing to term 5
I20260129 03:00:07.857903 20405 leader_election.cc:304] T febffe4a8b394368b69b1d164308fb9a P 197b880fa39d432a9b472845876bfaf0 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 197b880fa39d432a9b472845876bfaf0, 7a9daf5d42a84615a6383cfec9a10608; no voters: 974b802141bd4f13beffef2aa03f88f9
W20260129 03:00:07.832827 23337 log.cc:927] Time spent T 4e420f90884f4796997b5ea41e49e411 P 974b802141bd4f13beffef2aa03f88f9: Append to log took a long time: real 0.324s user 0.000s sys 0.001s
I20260129 03:00:07.889902 20586 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1299723a4476420ab02fd8bfd7aa0645" candidate_uuid: "7a9daf5d42a84615a6383cfec9a10608" candidate_term: 10 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "1c50de89a5a845668696dcb28c3badd6" is_pre_election: true
I20260129 03:00:07.890450 20588 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4f5a77e94fef4b16a2ba7b067c8b144a" candidate_uuid: "7a9daf5d42a84615a6383cfec9a10608" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "1c50de89a5a845668696dcb28c3badd6"
I20260129 03:00:07.879544 20441 raft_consensus.cc:2468] T 78aa39096868415d9afc2ffcb248e198 P 197b880fa39d432a9b472845876bfaf0 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1c50de89a5a845668696dcb28c3badd6 in term 7.
I20260129 03:00:07.891198 20588 raft_consensus.cc:3060] T 4f5a77e94fef4b16a2ba7b067c8b144a P 1c50de89a5a845668696dcb28c3badd6 [term 4 FOLLOWER]: Advancing to term 5
I20260129 03:00:07.891628 20441 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c2633a6fc34c44579de348d208b0c01e" candidate_uuid: "974b802141bd4f13beffef2aa03f88f9" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "197b880fa39d432a9b472845876bfaf0" is_pre_election: true
I20260129 03:00:07.892391 20441 raft_consensus.cc:2468] T c2633a6fc34c44579de348d208b0c01e P 197b880fa39d432a9b472845876bfaf0 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 974b802141bd4f13beffef2aa03f88f9 in term 4.
I20260129 03:00:07.827337 23304 raft_consensus.cc:493] T dc2f4cb752cf4d59b5f6bc689f5d8104 P 1c50de89a5a845668696dcb28c3badd6 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260129 03:00:07.839926 23251 raft_consensus.cc:493] T cedce331e1584ac8b636c9b8a72b2ba0 P 1c50de89a5a845668696dcb28c3badd6 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260129 03:00:07.911835 20515 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:48544 (request call id 303) took 1224 ms. Trace:
I20260129 03:00:07.920604 20515 rpcz_store.cc:276] 0129 03:00:06.687807 (+ 0us) service_pool.cc:168] Inserting onto call queue
0129 03:00:06.769753 (+ 81946us) service_pool.cc:225] Handling call
0129 03:00:06.791156 (+ 21403us) raft_consensus.cc:1471] Updating replica for 1 ops
0129 03:00:06.824410 (+ 33254us) raft_consensus.cc:1534] Early marking committed up to index 0
0129 03:00:06.824442 (+ 32us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0129 03:00:06.835605 (+ 11163us) log.cc:844] Serialized 26 byte log entry
0129 03:00:06.899232 (+ 63627us) raft_consensus.cc:1645] Marking committed up to 8
0129 03:00:06.968815 (+ 69583us) log.cc:844] Serialized 14 byte log entry
0129 03:00:06.970227 (+ 1412us) raft_consensus.cc:1697] Filling consensus response to leader.
0129 03:00:06.970306 (+ 79us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0129 03:00:07.911480 (+941174us) raft_consensus.cc:1685] finished
0129 03:00:07.911535 (+ 55us) raft_consensus.cc:1691] UpdateReplicas() finished
0129 03:00:07.911734 (+ 199us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":33913,"prepare.queue_time_us":67769,"prepare.run_cpu_time_us":38,"prepare.run_wall_time_us":39,"raft.queue_time_us":30506,"raft.run_cpu_time_us":315,"raft.run_wall_time_us":13396,"thread_start_us":36623,"threads_started":1,"wal-append.queue_time_us":234496}
I20260129 03:00:07.843568 23292 raft_consensus.cc:493] T 75f45538dc704d9a8ec4ff8ad9f3ea5b P 1c50de89a5a845668696dcb28c3badd6 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 7a9daf5d42a84615a6383cfec9a10608)
I20260129 03:00:07.853022 22891 raft_consensus.cc:493] T 4e3805ffa70242e8ade66da260dddf66 P 1c50de89a5a845668696dcb28c3badd6 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260129 03:00:07.918128 20584 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "add8d4923d1a489e86882c3a25692b95" candidate_uuid: "7a9daf5d42a84615a6383cfec9a10608" candidate_term: 7 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "1c50de89a5a845668696dcb28c3badd6"
I20260129 03:00:07.943197 20584 raft_consensus.cc:3060] T add8d4923d1a489e86882c3a25692b95 P 1c50de89a5a845668696dcb28c3badd6 [term 6 FOLLOWER]: Advancing to term 7
I20260129 03:00:07.926666 20433 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "87d5fe48a44a408aaf03e894a67ecb2f" candidate_uuid: "974b802141bd4f13beffef2aa03f88f9" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "197b880fa39d432a9b472845876bfaf0" is_pre_election: true
I20260129 03:00:07.944036 20433 raft_consensus.cc:2468] T 87d5fe48a44a408aaf03e894a67ecb2f P 197b880fa39d432a9b472845876bfaf0 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 974b802141bd4f13beffef2aa03f88f9 in term 6.
I20260129 03:00:07.850494 23262 raft_consensus.cc:493] T 3e2d15f8640b407195210c476e09df91 P 1c50de89a5a845668696dcb28c3badd6 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260129 03:00:07.850710 22978 raft_consensus.cc:2749] T 3ccac82a01a2445fa5b0ce6c0fcf2295 P 974b802141bd4f13beffef2aa03f88f9 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260129 03:00:07.821846 23216 raft_consensus.cc:515] T 0f20af7fbe5d4db89a6f17948d98912c P 974b802141bd4f13beffef2aa03f88f9 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 03:00:07.850183 23280 raft_consensus.cc:493] T 42d27a28d89b409e8807ec0efd97af69 P 1c50de89a5a845668696dcb28c3badd6 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 7a9daf5d42a84615a6383cfec9a10608)
I20260129 03:00:07.934136 20589 raft_consensus.cc:2468] T 1299723a4476420ab02fd8bfd7aa0645 P 1c50de89a5a845668696dcb28c3badd6 [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7a9daf5d42a84615a6383cfec9a10608 in term 10.
W20260129 03:00:07.894497 22878 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.671s user 0.008s sys 0.007s
I20260129 03:00:07.947171 20435 raft_consensus.cc:1275] T febffe4a8b394368b69b1d164308fb9a P 197b880fa39d432a9b472845876bfaf0 [term 4 FOLLOWER]: Refusing update from remote peer 974b802141bd4f13beffef2aa03f88f9: Log matching property violated. Preceding OpId in replica: term: 3 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20260129 03:00:07.828438 23199 raft_consensus.cc:515] T c934d31efd3f4e5195605ddc354d14e1 P 974b802141bd4f13beffef2aa03f88f9 [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } }
W20260129 03:00:07.918993 23147 consensus_meta.cc:287] T c891d2bff4ce4568861bbc992e91fbb0 P 1c50de89a5a845668696dcb28c3badd6: Time spent flushing consensus metadata: real 0.515s user 0.008s sys 0.002s
W20260129 03:00:07.956005 20407 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260129 03:00:07.960212 20582 raft_consensus.cc:2468] T 17d6431cc77a491e8464d68db4081a51 P 1c50de89a5a845668696dcb28c3badd6 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 197b880fa39d432a9b472845876bfaf0 in term 5.
I20260129 03:00:07.974588 20582 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "78aa39096868415d9afc2ffcb248e198" candidate_uuid: "7a9daf5d42a84615a6383cfec9a10608" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "1c50de89a5a845668696dcb28c3badd6" is_pre_election: true
I20260129 03:00:07.975562 20582 raft_consensus.cc:2393] T 78aa39096868415d9afc2ffcb248e198 P 1c50de89a5a845668696dcb28c3badd6 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 7a9daf5d42a84615a6383cfec9a10608 in current term 7: Already voted for candidate 1c50de89a5a845668696dcb28c3badd6 in this term.
I20260129 03:00:07.964699 20590 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a1a39dcd328e4268ae41b988c66b4a75" candidate_uuid: "974b802141bd4f13beffef2aa03f88f9" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "1c50de89a5a845668696dcb28c3badd6"
I20260129 03:00:07.977773 20586 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "45cc5f3d2688486d9b6dbe7e666dedde" candidate_uuid: "7a9daf5d42a84615a6383cfec9a10608" candidate_term: 6 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "1c50de89a5a845668696dcb28c3badd6"
I20260129 03:00:07.978408 20586 raft_consensus.cc:3060] T 45cc5f3d2688486d9b6dbe7e666dedde P 1c50de89a5a845668696dcb28c3badd6 [term 5 FOLLOWER]: Advancing to term 6
I20260129 03:00:07.863565 23250 consensus_queue.cc:1059] T 76f713c3331540d2a5b76164103e9eb1 P 1c50de89a5a845668696dcb28c3badd6 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 }, Status: INVALID_TERM, Last received: 5.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.088s
I20260129 03:00:07.895259 22989 raft_consensus.cc:697] T 76f713c3331540d2a5b76164103e9eb1 P 974b802141bd4f13beffef2aa03f88f9 [term 6 LEADER]: Becoming Leader. State: Replica: 974b802141bd4f13beffef2aa03f88f9, State: Running, Role: LEADER
I20260129 03:00:07.889010 23342 raft_consensus.cc:3060] T a609ac02fae84725ab743f95f51d33ce P 197b880fa39d432a9b472845876bfaf0 [term 8 FOLLOWER]: Advancing to term 9
W20260129 03:00:07.979521 21099 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.211.1:42027, user_credentials={real_user=slave}} blocked reactor thread for 291867us
W20260129 03:00:07.962258 22878 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.739s user 0.008s sys 0.007s
W20260129 03:00:07.968873 23343 log.cc:927] Time spent T fd8311b779af4e1fa1402b2a02a602f2 P 197b880fa39d432a9b472845876bfaf0: Append to log took a long time: real 0.059s user 0.000s sys 0.000s
I20260129 03:00:07.888828 23325 raft_consensus.cc:515] T 926a250fa5834e88b96ccecbefbf5086 P 1c50de89a5a845668696dcb28c3badd6 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 03:00:07.999157 20438 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:48356 (request call id 399) took 1755 ms. Trace:
I20260129 03:00:08.001642 20512 raft_consensus.cc:1240] T c934d31efd3f4e5195605ddc354d14e1 P 974b802141bd4f13beffef2aa03f88f9 [term 6 FOLLOWER]: Rejecting Update request from peer 197b880fa39d432a9b472845876bfaf0 for earlier term 5. Current term is 6. Ops: []
I20260129 03:00:08.017551 20582 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4b1876f9a70b49edb3aa74105e66d3b3" candidate_uuid: "197b880fa39d432a9b472845876bfaf0" candidate_term: 6 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "1c50de89a5a845668696dcb28c3badd6" is_pre_election: true
I20260129 03:00:07.984153 23147 raft_consensus.cc:2749] T c891d2bff4ce4568861bbc992e91fbb0 P 1c50de89a5a845668696dcb28c3badd6 [term 8 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260129 03:00:07.942601 23251 raft_consensus.cc:515] T cedce331e1584ac8b636c9b8a72b2ba0 P 1c50de89a5a845668696dcb28c3badd6 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 03:00:07.918828 23304 raft_consensus.cc:515] T dc2f4cb752cf4d59b5f6bc689f5d8104 P 1c50de89a5a845668696dcb28c3badd6 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } }
W20260129 03:00:08.001221 23091 consensus_meta.cc:287] T 926a250fa5834e88b96ccecbefbf5086 P 197b880fa39d432a9b472845876bfaf0: Time spent flushing consensus metadata: real 0.877s user 0.009s sys 0.002s
I20260129 03:00:07.982168 23104 raft_consensus.cc:493] T e4d18c35b0a7493e85d0a4b92956a48f P 1c50de89a5a845668696dcb28c3badd6 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 974b802141bd4f13beffef2aa03f88f9)
I20260129 03:00:07.963550 23262 raft_consensus.cc:515] T 3e2d15f8640b407195210c476e09df91 P 1c50de89a5a845668696dcb28c3badd6 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } }
I20260129 03:00:08.021368 21102 leader_election.cc:304] T 45cc5f3d2688486d9b6dbe7e666dedde P 7a9daf5d42a84615a6383cfec9a10608 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7a9daf5d42a84615a6383cfec9a10608, 974b802141bd4f13beffef2aa03f88f9; no voters:
I20260129 03:00:08.013917 20438 rpcz_store.cc:276] 0129 03:00:06.244018 (+ 0us) service_pool.cc:168] Inserting onto call queue
0129 03:00:06.314375 (+ 70357us) service_pool.cc:225] Handling call
0129 03:00:06.393692 (+ 79317us) raft_consensus.cc:1471] Updating replica for 1 ops
0129 03:00:06.421054 (+ 27362us) raft_consensus.cc:1534] Early marking committed up to index 0
0129 03:00:06.421105 (+ 51us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0129 03:00:06.436749 (+ 15644us) log.cc:844] Serialized 26 byte log entry
0129 03:00:06.644634 (+207885us) raft_consensus.cc:1645] Marking committed up to 7
0129 03:00:06.644699 (+ 65us) raft_consensus.cc:1697] Filling consensus response to leader.
0129 03:00:06.644831 (+ 132us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0129 03:00:07.420989 (+776158us) raft_consensus.cc:1685] finished
0129 03:00:07.421023 (+ 34us) raft_consensus.cc:1691] UpdateReplicas() finished
0129 03:00:07.988844 (+567821us) spinlock_profiling.cc:243] Waited 79 ms on lock 0x7b54002fc950. stack: 00007f8fe044e373 00007f8fe044e270 00007f8fdfda0574 00007f8fe9a5c6b4 00007f8fe9a5c649 00007f8fe34975a1 00007f8fe352d52d 00007f8fe84e1f49 00007f8fe1a6ed34 00007f8fe1a6ecc2 00007f8fe1a6ebf1 00007f8fe1a6eb6c 00007f8fe1a6de22 00007f8fe186d34c 00007f8fe186c786 00007f8fe186c12f
0129 03:00:07.999090 (+ 10246us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":92,"raft.run_cpu_time_us":158,"raft.run_wall_time_us":13749,"spinlock_wait_cycles":181693312,"thread_start_us":119299,"threads_started":1,"wal-append.queue_time_us":249295}
I20260129 03:00:07.949334 23292 raft_consensus.cc:515] T 75f45538dc704d9a8ec4ff8ad9f3ea5b P 1c50de89a5a845668696dcb28c3badd6 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } }
I20260129 03:00:08.000385 23344 raft_consensus.cc:2804] T febffe4a8b394368b69b1d164308fb9a P 197b880fa39d432a9b472845876bfaf0 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260129 03:00:08.024968 20582 raft_consensus.cc:2468] T 4b1876f9a70b49edb3aa74105e66d3b3 P 1c50de89a5a845668696dcb28c3badd6 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 197b880fa39d432a9b472845876bfaf0 in term 5.
I20260129 03:00:07.955361 22891 raft_consensus.cc:515] T 4e3805ffa70242e8ade66da260dddf66 P 1c50de89a5a845668696dcb28c3badd6 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } }
W20260129 03:00:08.059271 20405 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.211.4:40439, user_credentials={real_user=slave}} blocked reactor thread for 201423us
I20260129 03:00:08.083828 20405 leader_election.cc:304] T 87d5fe48a44a408aaf03e894a67ecb2f P 197b880fa39d432a9b472845876bfaf0 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 197b880fa39d432a9b472845876bfaf0, 7a9daf5d42a84615a6383cfec9a10608; no voters: 974b802141bd4f13beffef2aa03f88f9
I20260129 03:00:08.074278 20582 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4b1876f9a70b49edb3aa74105e66d3b3" candidate_uuid: "974b802141bd4f13beffef2aa03f88f9" candidate_term: 6 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "1c50de89a5a845668696dcb28c3badd6"
I20260129 03:00:08.071587 20583 raft_consensus.cc:1240] T c891d2bff4ce4568861bbc992e91fbb0 P 1c50de89a5a845668696dcb28c3badd6 [term 8 FOLLOWER]: Rejecting Update request from peer 197b880fa39d432a9b472845876bfaf0 for earlier term 7. Current term is 8. Ops: []
I20260129 03:00:08.019050 23203 raft_consensus.cc:697] T add8d4923d1a489e86882c3a25692b95 P 7a9daf5d42a84615a6383cfec9a10608 [term 7 LEADER]: Becoming Leader. State: Replica: 7a9daf5d42a84615a6383cfec9a10608, State: Running, Role: LEADER
I20260129 03:00:08.069275 20435 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "78aa39096868415d9afc2ffcb248e198" candidate_uuid: "7a9daf5d42a84615a6383cfec9a10608" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "197b880fa39d432a9b472845876bfaf0" is_pre_election: true
W20260129 03:00:08.076821 21100 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.211.3:37735, user_credentials={real_user=slave}} blocked reactor thread for 101306us
W20260129 03:00:08.049589 23086 consensus_meta.cc:287] T c891d2bff4ce4568861bbc992e91fbb0 P 197b880fa39d432a9b472845876bfaf0: Time spent flushing consensus metadata: real 1.185s user 0.009s sys 0.005s
I20260129 03:00:08.084739 20582 raft_consensus.cc:3060] T 4b1876f9a70b49edb3aa74105e66d3b3 P 1c50de89a5a845668696dcb28c3badd6 [term 5 FOLLOWER]: Advancing to term 6
I20260129 03:00:07.993607 23280 raft_consensus.cc:515] T 42d27a28d89b409e8807ec0efd97af69 P 1c50de89a5a845668696dcb28c3badd6 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } }
I20260129 03:00:08.109560 20405 leader_election.cc:304] T 0678b8b468bd43cdbc3e485900844821 P 197b880fa39d432a9b472845876bfaf0 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 197b880fa39d432a9b472845876bfaf0; no voters: 7a9daf5d42a84615a6383cfec9a10608, 974b802141bd4f13beffef2aa03f88f9
I20260129 03:00:08.099503 20479 leader_election.cc:304] T c2633a6fc34c44579de348d208b0c01e P 974b802141bd4f13beffef2aa03f88f9 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 197b880fa39d432a9b472845876bfaf0, 974b802141bd4f13beffef2aa03f88f9; no voters:
I20260129 03:00:08.110201 20435 raft_consensus.cc:2393] T 78aa39096868415d9afc2ffcb248e198 P 197b880fa39d432a9b472845876bfaf0 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 7a9daf5d42a84615a6383cfec9a10608 in current term 7: Already voted for candidate 1c50de89a5a845668696dcb28c3badd6 in this term.
W20260129 03:00:08.108657 20406 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260129 03:00:08.111353 20439 raft_consensus.cc:2410] T c891d2bff4ce4568861bbc992e91fbb0 P 197b880fa39d432a9b472845876bfaf0 [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate 974b802141bd4f13beffef2aa03f88f9 for term 8 because replica has last-logged OpId of term: 7 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 6 index: 5.
I20260129 03:00:08.112771 20590 raft_consensus.cc:2468] T a1a39dcd328e4268ae41b988c66b4a75 P 1c50de89a5a845668696dcb28c3badd6 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 974b802141bd4f13beffef2aa03f88f9 in term 5.
I20260129 03:00:08.118571 23207 raft_consensus.cc:2804] T c2633a6fc34c44579de348d208b0c01e P 974b802141bd4f13beffef2aa03f88f9 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260129 03:00:08.119001 23207 raft_consensus.cc:493] T c2633a6fc34c44579de348d208b0c01e P 974b802141bd4f13beffef2aa03f88f9 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260129 03:00:08.119035 21136 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "dc2f4cb752cf4d59b5f6bc689f5d8104" candidate_uuid: "197b880fa39d432a9b472845876bfaf0" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "7a9daf5d42a84615a6383cfec9a10608" is_pre_election: true
I20260129 03:00:08.119467 23207 raft_consensus.cc:3060] T c2633a6fc34c44579de348d208b0c01e P 974b802141bd4f13beffef2aa03f88f9 [term 4 FOLLOWER]: Advancing to term 5
I20260129 03:00:08.119810 23086 raft_consensus.cc:2749] T 0678b8b468bd43cdbc3e485900844821 P 197b880fa39d432a9b472845876bfaf0 [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260129 03:00:08.119853 21136 raft_consensus.cc:2393] T dc2f4cb752cf4d59b5f6bc689f5d8104 P 7a9daf5d42a84615a6383cfec9a10608 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 197b880fa39d432a9b472845876bfaf0 in current term 5: Already voted for candidate 7a9daf5d42a84615a6383cfec9a10608 in this term.
I20260129 03:00:08.121518 20479 leader_election.cc:304] T 87d5fe48a44a408aaf03e894a67ecb2f P 974b802141bd4f13beffef2aa03f88f9 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 197b880fa39d432a9b472845876bfaf0, 974b802141bd4f13beffef2aa03f88f9; no voters: 7a9daf5d42a84615a6383cfec9a10608
I20260129 03:00:08.122784 23333 raft_consensus.cc:2804] T 87d5fe48a44a408aaf03e894a67ecb2f P 974b802141bd4f13beffef2aa03f88f9 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260129 03:00:08.123152 23333 raft_consensus.cc:493] T 87d5fe48a44a408aaf03e894a67ecb2f P 974b802141bd4f13beffef2aa03f88f9 [term 5 FOLLOWER]: Starting leader election (detected failure of leader 197b880fa39d432a9b472845876bfaf0)
I20260129 03:00:08.123494 23333 raft_consensus.cc:3060] T 87d5fe48a44a408aaf03e894a67ecb2f P 974b802141bd4f13beffef2aa03f88f9 [term 5 FOLLOWER]: Advancing to term 6
I20260129 03:00:08.124699 20510 raft_consensus.cc:1275] T 0f20af7fbe5d4db89a6f17948d98912c P 974b802141bd4f13beffef2aa03f88f9 [term 7 FOLLOWER]: Refusing update from remote peer 197b880fa39d432a9b472845876bfaf0: Log matching property violated. Preceding OpId in replica: term: 6 index: 4. Preceding OpId from leader: term: 7 index: 5. (index mismatch)
I20260129 03:00:08.124814 20515 raft_consensus.cc:1275] T b189f08e16a74a11980cc55d99359a0a P 974b802141bd4f13beffef2aa03f88f9 [term 9 FOLLOWER]: Refusing update from remote peer 197b880fa39d432a9b472845876bfaf0: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 9 index: 5. (index mismatch)
I20260129 03:00:08.128667 23280 leader_election.cc:290] T 42d27a28d89b409e8807ec0efd97af69 P 1c50de89a5a845668696dcb28c3badd6 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 197b880fa39d432a9b472845876bfaf0 (127.18.211.1:42027), 7a9daf5d42a84615a6383cfec9a10608 (127.18.211.4:40439)
I20260129 03:00:08.126820 23086 consensus_queue.cc:1048] T 0f20af7fbe5d4db89a6f17948d98912c P 197b880fa39d432a9b472845876bfaf0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 3, Time since last communication: 0.002s
I20260129 03:00:08.132108 20433 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "724a236524cb4e7ba3428880681d6cc2" candidate_uuid: "1c50de89a5a845668696dcb28c3badd6" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "197b880fa39d432a9b472845876bfaf0"
I20260129 03:00:08.139006 20513 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:48544 (request call id 308) took 1305 ms. Trace:
I20260129 03:00:08.138681 20582 raft_consensus.cc:2468] T 4b1876f9a70b49edb3aa74105e66d3b3 P 1c50de89a5a845668696dcb28c3badd6 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 974b802141bd4f13beffef2aa03f88f9 in term 6.
W20260129 03:00:08.131114 20590 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:35752 (request call id 1670) took 1700 ms (client timeout 1923 ms). Trace:
I20260129 03:00:08.110471 23232 raft_consensus.cc:493] T a609ac02fae84725ab743f95f51d33ce P 974b802141bd4f13beffef2aa03f88f9 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260129 03:00:08.081089 23089 raft_consensus.cc:493] T a1a39dcd328e4268ae41b988c66b4a75 P 197b880fa39d432a9b472845876bfaf0 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260129 03:00:08.073285 23346 raft_consensus.cc:493] T 8055232cf0004d40a85f3bf983fe00a9 P 197b880fa39d432a9b472845876bfaf0 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260129 03:00:08.139529 20513 rpcz_store.cc:276] 0129 03:00:06.833962 (+ 0us) service_pool.cc:168] Inserting onto call queue
0129 03:00:06.883637 (+ 49675us) service_pool.cc:225] Handling call
0129 03:00:06.950940 (+ 67303us) raft_consensus.cc:1471] Updating replica for 1 ops
0129 03:00:07.093606 (+142666us) raft_consensus.cc:1534] Early marking committed up to index 0
0129 03:00:07.093636 (+ 30us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0129 03:00:07.110041 (+ 16405us) log.cc:844] Serialized 26 byte log entry
0129 03:00:07.373294 (+263253us) raft_consensus.cc:1645] Marking committed up to 7
0129 03:00:07.392559 (+ 19265us) log.cc:844] Serialized 14 byte log entry
0129 03:00:07.392785 (+ 226us) raft_consensus.cc:1697] Filling consensus response to leader.
0129 03:00:07.603704 (+210919us) spinlock_profiling.cc:243] Waited 4.19 ms on lock 0x7b18003766c0. stack: 00007f8fe044e373 00007f8fe044e270 00007f8fdfda0574 00007f8fe9a5c6b4 00007f8fe9a5c649 00007f8fe92d4c91 00007f8fe92e1671 00007f8fe04802d3 00007f8fe04826a1 00007f8fe0482659 00007f8fe04825e9 00007f8fe04825b1 00007f8fe04818ad 0000000000379254 0000000000379099 00007f8fe046369a
0129 03:00:07.709105 (+105401us) spinlock_profiling.cc:243] Waited 142 ms on lock 0x7b54003cdcd4. stack: 00007f8fe044e373 00007f8fe044e270 00007f8fdfda0574 00007f8fe9a5c6b4 00007f8fe9a5c649 00007f8fe34975a1 00007f8fe352ea81 00007f8fe352d4be 00007f8fe84e1f49 00007f8fe1a6ed34 00007f8fe1a6ecc2 00007f8fe1a6ebf1 00007f8fe1a6eb6c 00007f8fe1a6de22 00007f8fe186d34c 00007f8fe186c786
0129 03:00:07.709183 (+ 78us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0129 03:00:08.138767 (+429584us) raft_consensus.cc:1685] finished
0129 03:00:08.138797 (+ 30us) raft_consensus.cc:1691] UpdateReplicas() finished
0129 03:00:08.138939 (+ 142us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":7633,"prepare.queue_time_us":13350,"prepare.run_cpu_time_us":73,"prepare.run_wall_time_us":4055,"raft.queue_time_us":24734,"raft.run_cpu_time_us":83,"raft.run_wall_time_us":83,"spinlock_wait_cycles":357380992,"thread_start_us":255625,"threads_started":1,"wal-append.queue_time_us":289320}
I20260129 03:00:08.145502 21136 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2ac659d732314e1288849f08fe5e074e" candidate_uuid: "974b802141bd4f13beffef2aa03f88f9" candidate_term: 6 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "7a9daf5d42a84615a6383cfec9a10608" is_pre_election: true
I20260129 03:00:08.073401 23344 raft_consensus.cc:493] T febffe4a8b394368b69b1d164308fb9a P 197b880fa39d432a9b472845876bfaf0 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260129 03:00:08.051275 23091 raft_consensus.cc:515] T 926a250fa5834e88b96ccecbefbf5086 P 197b880fa39d432a9b472845876bfaf0 [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 03:00:08.087198 20553 leader_election.cc:304] T 78aa39096868415d9afc2ffcb248e198 P 1c50de89a5a845668696dcb28c3badd6 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 197b880fa39d432a9b472845876bfaf0, 1c50de89a5a845668696dcb28c3badd6; no voters:
W20260129 03:00:08.143838 20590 rpcz_store.cc:269] 0129 03:00:06.417874 (+ 0us) service_pool.cc:168] Inserting onto call queue
0129 03:00:07.964352 (+1546478us) service_pool.cc:225] Handling call
0129 03:00:08.117998 (+153646us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260129 03:00:08.151897 20441 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2ac659d732314e1288849f08fe5e074e" candidate_uuid: "974b802141bd4f13beffef2aa03f88f9" candidate_term: 6 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "197b880fa39d432a9b472845876bfaf0" is_pre_election: true
W20260129 03:00:08.159161 20482 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260129 03:00:08.175076 20441 raft_consensus.cc:2410] T 2ac659d732314e1288849f08fe5e074e P 197b880fa39d432a9b472845876bfaf0 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 974b802141bd4f13beffef2aa03f88f9 for term 6 because replica has last-logged OpId of term: 5 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 5.
I20260129 03:00:08.159261 20582 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "cb4727386a3e48aca5d71cc6c611c197" candidate_uuid: "197b880fa39d432a9b472845876bfaf0" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "1c50de89a5a845668696dcb28c3badd6"
I20260129 03:00:08.143745 23086 consensus_queue.cc:1243] T 0f20af7fbe5d4db89a6f17948d98912c P 197b880fa39d432a9b472845876bfaf0 [LEADER]: Peer 974b802141bd4f13beffef2aa03f88f9 log is divergent from this leader: its last log entry 6.4 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 3
I20260129 03:00:08.098495 23104 raft_consensus.cc:515] T e4d18c35b0a7493e85d0a4b92956a48f P 1c50de89a5a845668696dcb28c3badd6 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 03:00:08.176105 20433 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0678b8b468bd43cdbc3e485900844821" candidate_uuid: "974b802141bd4f13beffef2aa03f88f9" candidate_term: 9 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "197b880fa39d432a9b472845876bfaf0" is_pre_election: true
I20260129 03:00:08.132663 23336 leader_election.cc:290] T 0678b8b468bd43cdbc3e485900844821 P 974b802141bd4f13beffef2aa03f88f9 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 197b880fa39d432a9b472845876bfaf0 (127.18.211.1:42027), 7a9daf5d42a84615a6383cfec9a10608 (127.18.211.4:40439)
W20260129 03:00:08.161208 21102 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.211.2:34517, user_credentials={real_user=slave}} blocked reactor thread for 139915us
I20260129 03:00:08.132963 22982 leader_election.cc:290] T 2ac659d732314e1288849f08fe5e074e P 974b802141bd4f13beffef2aa03f88f9 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 197b880fa39d432a9b472845876bfaf0 (127.18.211.1:42027), 7a9daf5d42a84615a6383cfec9a10608 (127.18.211.4:40439)
I20260129 03:00:08.189585 23262 leader_election.cc:290] T 3e2d15f8640b407195210c476e09df91 P 1c50de89a5a845668696dcb28c3badd6 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 974b802141bd4f13beffef2aa03f88f9 (127.18.211.2:34517), 7a9daf5d42a84615a6383cfec9a10608 (127.18.211.4:40439)
W20260129 03:00:08.190521 23349 log.cc:927] Time spent T 368953c9102643b2a21f99af36b51905 P 974b802141bd4f13beffef2aa03f88f9: Append to log took a long time: real 0.211s user 0.001s sys 0.000s
I20260129 03:00:08.192920 22891 leader_election.cc:290] T 4e3805ffa70242e8ade66da260dddf66 P 1c50de89a5a845668696dcb28c3badd6 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 197b880fa39d432a9b472845876bfaf0 (127.18.211.1:42027), 7a9daf5d42a84615a6383cfec9a10608 (127.18.211.4:40439)
I20260129 03:00:08.162426 22932 leader_election.cc:290] T fd8311b779af4e1fa1402b2a02a602f2 P 7a9daf5d42a84615a6383cfec9a10608 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 197b880fa39d432a9b472845876bfaf0 (127.18.211.1:42027), 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3:37735)
I20260129 03:00:08.196269 22932 raft_consensus.cc:2804] T 45cc5f3d2688486d9b6dbe7e666dedde P 7a9daf5d42a84615a6383cfec9a10608 [term 6 FOLLOWER]: Leader election won for term 6
I20260129 03:00:08.196755 22932 raft_consensus.cc:697] T 45cc5f3d2688486d9b6dbe7e666dedde P 7a9daf5d42a84615a6383cfec9a10608 [term 6 LEADER]: Becoming Leader. State: Replica: 7a9daf5d42a84615a6383cfec9a10608, State: Running, Role: LEADER
I20260129 03:00:08.197701 22932 consensus_queue.cc:237] T 45cc5f3d2688486d9b6dbe7e666dedde P 7a9daf5d42a84615a6383cfec9a10608 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.5, Last appended by leader: 5, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } }
W20260129 03:00:08.198446 23205 consensus_meta.cc:287] T e7d83d4ccf684e76acb29a0dc844e1b8 P 974b802141bd4f13beffef2aa03f88f9: Time spent flushing consensus metadata: real 1.240s user 0.011s sys 0.000s
I20260129 03:00:08.141058 23352 raft_consensus.cc:2804] T 87d5fe48a44a408aaf03e894a67ecb2f P 197b880fa39d432a9b472845876bfaf0 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260129 03:00:08.204614 20584 raft_consensus.cc:2410] T add8d4923d1a489e86882c3a25692b95 P 1c50de89a5a845668696dcb28c3badd6 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 7a9daf5d42a84615a6383cfec9a10608 for term 7 because replica has last-logged OpId of term: 5 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 5.
W20260129 03:00:08.205338 20553 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.211.1:42027, user_credentials={real_user=slave}} blocked reactor thread for 118256us
I20260129 03:00:08.205928 20584 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "76f713c3331540d2a5b76164103e9eb1" candidate_uuid: "974b802141bd4f13beffef2aa03f88f9" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "1c50de89a5a845668696dcb28c3badd6" is_pre_election: true
W20260129 03:00:08.206837 21099 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.211.1:42027, user_credentials={real_user=slave}} blocked reactor thread for 132509us
I20260129 03:00:08.184612 20582 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "8055232cf0004d40a85f3bf983fe00a9" candidate_uuid: "974b802141bd4f13beffef2aa03f88f9" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "1c50de89a5a845668696dcb28c3badd6"
I20260129 03:00:08.184996 20433 raft_consensus.cc:2468] T 0678b8b468bd43cdbc3e485900844821 P 197b880fa39d432a9b472845876bfaf0 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 974b802141bd4f13beffef2aa03f88f9 in term 8.
I20260129 03:00:08.138327 23152 consensus_queue.cc:1048] T febffe4a8b394368b69b1d164308fb9a P 974b802141bd4f13beffef2aa03f88f9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.042s
I20260129 03:00:08.175536 23178 consensus_queue.cc:1048] T b189f08e16a74a11980cc55d99359a0a P 197b880fa39d432a9b472845876bfaf0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 3, Time since last communication: 0.020s
I20260129 03:00:08.163194 23298 leader_election.cc:290] T 724a236524cb4e7ba3428880681d6cc2 P 1c50de89a5a845668696dcb28c3badd6 [CANDIDATE]: Term 6 election: Requested vote from peers 197b880fa39d432a9b472845876bfaf0 (127.18.211.1:42027), 974b802141bd4f13beffef2aa03f88f9 (127.18.211.2:34517)
I20260129 03:00:08.169274 23344 raft_consensus.cc:3060] T febffe4a8b394368b69b1d164308fb9a P 197b880fa39d432a9b472845876bfaf0 [term 4 FOLLOWER]: Advancing to term 5
I20260129 03:00:08.209053 20440 raft_consensus.cc:2393] T 926a250fa5834e88b96ccecbefbf5086 P 197b880fa39d432a9b472845876bfaf0 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 974b802141bd4f13beffef2aa03f88f9 in current term 8: Already voted for candidate 197b880fa39d432a9b472845876bfaf0 in this term.
I20260129 03:00:08.222560 23251 leader_election.cc:290] T cedce331e1584ac8b636c9b8a72b2ba0 P 1c50de89a5a845668696dcb28c3badd6 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 197b880fa39d432a9b472845876bfaf0 (127.18.211.1:42027), 974b802141bd4f13beffef2aa03f88f9 (127.18.211.2:34517)
W20260129 03:00:08.202342 23026 consensus_meta.cc:287] T e33a417cf9c54cf5af73a4ee3aac4d3a P 1c50de89a5a845668696dcb28c3badd6: Time spent flushing consensus metadata: real 1.053s user 0.005s sys 0.005s
I20260129 03:00:08.181818 21128 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0678b8b468bd43cdbc3e485900844821" candidate_uuid: "974b802141bd4f13beffef2aa03f88f9" candidate_term: 9 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "7a9daf5d42a84615a6383cfec9a10608" is_pre_election: true
I20260129 03:00:08.217790 20515 pending_rounds.cc:85] T 0f20af7fbe5d4db89a6f17948d98912c P 974b802141bd4f13beffef2aa03f88f9: Aborting all ops after (but not including) 3
W20260129 03:00:08.226655 21100 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.211.3:37735, user_credentials={real_user=slave}} blocked reactor thread for 140509us
I20260129 03:00:08.227046 20515 pending_rounds.cc:107] T 0f20af7fbe5d4db89a6f17948d98912c P 974b802141bd4f13beffef2aa03f88f9: Aborting uncommitted NO_OP operation due to leader change: 6.4
I20260129 03:00:08.227896 20515 raft_consensus.cc:2889] T 0f20af7fbe5d4db89a6f17948d98912c P 974b802141bd4f13beffef2aa03f88f9 [term 7 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260129 03:00:08.155575 22989 consensus_queue.cc:237] T 76f713c3331540d2a5b76164103e9eb1 P 974b802141bd4f13beffef2aa03f88f9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 5.4, Last appended by leader: 4, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 03:00:08.203258 22855 raft_consensus.cc:2804] T 78aa39096868415d9afc2ffcb248e198 P 1c50de89a5a845668696dcb28c3badd6 [term 7 FOLLOWER]: Leader election won for term 7
I20260129 03:00:08.236791 20435 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "42d27a28d89b409e8807ec0efd97af69" candidate_uuid: "1c50de89a5a845668696dcb28c3badd6" candidate_term: 9 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "197b880fa39d432a9b472845876bfaf0" is_pre_election: true
I20260129 03:00:08.238487 20438 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4e3805ffa70242e8ade66da260dddf66" candidate_uuid: "1c50de89a5a845668696dcb28c3badd6" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "197b880fa39d432a9b472845876bfaf0" is_pre_election: true
I20260129 03:00:08.244588 20433 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fd8311b779af4e1fa1402b2a02a602f2" candidate_uuid: "7a9daf5d42a84615a6383cfec9a10608" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "197b880fa39d432a9b472845876bfaf0" is_pre_election: true
I20260129 03:00:08.170848 23089 raft_consensus.cc:515] T a1a39dcd328e4268ae41b988c66b4a75 P 197b880fa39d432a9b472845876bfaf0 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } } peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } }
I20260129 03:00:08.208655 23352 raft_consensus.cc:493] T 87d5fe48a44a408aaf03e894a67ecb2f P 197b880fa39d432a9b472845876bfaf0 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260129 03:00:08.246937 20584 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4408006eb5174de1896b037cf3f18457" candidate_uuid: "974b802141bd4f13beffef2aa03f88f9" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "1c50de89a5a845668696dcb28c3badd6" is_pre_election: true
I20260129 03:00:08.239552 20590 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a0fcc58d6fb6473cafbd9d34d5195269" candidate_uuid: "197b880fa39d432a9b472845876bfaf0" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "1c50de89a5a845668696dcb28c3badd6"
I20260129 03:00:08.248145 20590 raft_consensus.cc:3060] T a0fcc58d6fb6473cafbd9d34d5195269 P 1c50de89a5a845668696dcb28c3badd6 [term 4 FOLLOWER]: Advancing to term 5
I20260129 03:00:08.248903 20582 raft_consensus.cc:3060] T 8055232cf0004d40a85f3bf983fe00a9 P 1c50de89a5a845668696dcb28c3badd6 [term 4 FOLLOWER]: Advancing to term 5
I20260129 03:00:08.165817 23232 raft_consensus.cc:515] T a609ac02fae84725ab743f95f51d33ce P 974b802141bd4f13beffef2aa03f88f9 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } }
W20260129 03:00:08.256264 23080 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 1.578s user 0.010s sys 0.005s
I20260129 03:00:08.257863 20435 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "cedce331e1584ac8b636c9b8a72b2ba0" candidate_uuid: "1c50de89a5a845668696dcb28c3badd6" candidate_term: 8 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "197b880fa39d432a9b472845876bfaf0" is_pre_election: true
I20260129 03:00:08.176949 23346 raft_consensus.cc:515] T 8055232cf0004d40a85f3bf983fe00a9 P 197b880fa39d432a9b472845876bfaf0 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 03:00:08.231598 23178 raft_consensus.cc:493] T 4e3805ffa70242e8ade66da260dddf66 P 197b880fa39d432a9b472845876bfaf0 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260129 03:00:08.186208 23359 consensus_queue.cc:1059] T cb0b3f259ebc4310927338a8740e87b8 P 7a9daf5d42a84615a6383cfec9a10608 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 }, Status: INVALID_TERM, Last received: 7.8, Next index: 9, Last known committed idx: 8, Time since last communication: 0.085s
I20260129 03:00:08.263785 21136 raft_consensus.cc:1275] T 00e79d8f25a34676b1d866eb2068a2b2 P 7a9daf5d42a84615a6383cfec9a10608 [term 8 FOLLOWER]: Refusing update from remote peer 197b880fa39d432a9b472845876bfaf0: Log matching property violated. Preceding OpId in replica: term: 6 index: 8. Preceding OpId from leader: term: 8 index: 9. (index mismatch)
I20260129 03:00:08.214192 23205 raft_consensus.cc:515] T e7d83d4ccf684e76acb29a0dc844e1b8 P 974b802141bd4f13beffef2aa03f88f9 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 03:00:08.244704 23199 leader_election.cc:290] T c934d31efd3f4e5195605ddc354d14e1 P 974b802141bd4f13beffef2aa03f88f9 [CANDIDATE]: Term 6 election: Requested vote from peers 197b880fa39d432a9b472845876bfaf0 (127.18.211.1:42027), 7a9daf5d42a84615a6383cfec9a10608 (127.18.211.4:40439)
I20260129 03:00:08.233577 23026 raft_consensus.cc:515] T e33a417cf9c54cf5af73a4ee3aac4d3a P 1c50de89a5a845668696dcb28c3badd6 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 03:00:08.217283 20513 tablet_service.cc:2066] Received LeaderStepDown RPC: tablet_id: "4b1876f9a70b49edb3aa74105e66d3b3"
dest_uuid: "974b802141bd4f13beffef2aa03f88f9"
mode: GRACEFUL
new_leader_uuid: "1c50de89a5a845668696dcb28c3badd6"
from {username='slave'} at 127.0.0.1:52008
W20260129 03:00:08.296033 20555 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260129 03:00:08.281708 20433 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "926a250fa5834e88b96ccecbefbf5086" candidate_uuid: "1c50de89a5a845668696dcb28c3badd6" candidate_term: 8 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "197b880fa39d432a9b472845876bfaf0" is_pre_election: true
I20260129 03:00:08.296639 20584 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b189f08e16a74a11980cc55d99359a0a" candidate_uuid: "974b802141bd4f13beffef2aa03f88f9" candidate_term: 10 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "1c50de89a5a845668696dcb28c3badd6" is_pre_election: true
I20260129 03:00:08.349615 20438 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e4d18c35b0a7493e85d0a4b92956a48f" candidate_uuid: "1c50de89a5a845668696dcb28c3badd6" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "197b880fa39d432a9b472845876bfaf0" is_pre_election: true
W20260129 03:00:08.380872 21102 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.211.2:34517, user_credentials={real_user=slave}} blocked reactor thread for 191192us
I20260129 03:00:08.383234 20584 raft_consensus.cc:2410] T b189f08e16a74a11980cc55d99359a0a P 1c50de89a5a845668696dcb28c3badd6 [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 974b802141bd4f13beffef2aa03f88f9 for term 10 because replica has last-logged OpId of term: 8 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 4 index: 3.
I20260129 03:00:08.247059 21128 raft_consensus.cc:2468] T 0678b8b468bd43cdbc3e485900844821 P 7a9daf5d42a84615a6383cfec9a10608 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 974b802141bd4f13beffef2aa03f88f9 in term 8.
I20260129 03:00:08.381024 20582 raft_consensus.cc:2468] T 8055232cf0004d40a85f3bf983fe00a9 P 1c50de89a5a845668696dcb28c3badd6 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 974b802141bd4f13beffef2aa03f88f9 in term 5.
W20260129 03:00:08.261363 23080 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 1.583s user 0.010s sys 0.005s
I20260129 03:00:08.387838 20590 raft_consensus.cc:2468] T a0fcc58d6fb6473cafbd9d34d5195269 P 1c50de89a5a845668696dcb28c3badd6 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 197b880fa39d432a9b472845876bfaf0 in term 5.
W20260129 03:00:08.403510 20480 leader_election.cc:336] T 45cc5f3d2688486d9b6dbe7e666dedde P 974b802141bd4f13beffef2aa03f88f9 [CANDIDATE]: Term 5 election: RPC error from VoteRequest() call to peer 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3:37735): Timed out: RequestConsensusVote RPC to 127.18.211.3:37735 timed out after 2.730s (SENT)
I20260129 03:00:08.402700 20513 raft_consensus.cc:606] T 4b1876f9a70b49edb3aa74105e66d3b3 P 974b802141bd4f13beffef2aa03f88f9 [term 6 FOLLOWER]: Received request to transfer leadership to TS 1c50de89a5a845668696dcb28c3badd6
I20260129 03:00:08.404399 20513 raft_consensus.cc:614] T 4b1876f9a70b49edb3aa74105e66d3b3 P 974b802141bd4f13beffef2aa03f88f9 [term 6 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260129 03:00:08.246793 23104 leader_election.cc:290] T e4d18c35b0a7493e85d0a4b92956a48f P 1c50de89a5a845668696dcb28c3badd6 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 197b880fa39d432a9b472845876bfaf0 (127.18.211.1:42027), 974b802141bd4f13beffef2aa03f88f9 (127.18.211.2:34517)
I20260129 03:00:08.223698 23203 consensus_queue.cc:237] T add8d4923d1a489e86882c3a25692b95 P 7a9daf5d42a84615a6383cfec9a10608 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } }
I20260129 03:00:08.562219 23380 raft_consensus.cc:3055] T cb0b3f259ebc4310927338a8740e87b8 P 7a9daf5d42a84615a6383cfec9a10608 [term 7 LEADER]: Stepping down as leader of term 7
I20260129 03:00:08.827625 23205 leader_election.cc:290] T e7d83d4ccf684e76acb29a0dc844e1b8 P 974b802141bd4f13beffef2aa03f88f9 [CANDIDATE]: Term 7 election: Requested vote from peers 197b880fa39d432a9b472845876bfaf0 (127.18.211.1:42027), 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3:37735)
W20260129 03:00:08.389669 20479 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260129 03:00:08.619571 20482 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260129 03:00:08.297567 23345 log.cc:927] Time spent T cb4727386a3e48aca5d71cc6c611c197 P 1c50de89a5a845668696dcb28c3badd6: Append to log took a long time: real 0.284s user 0.000s sys 0.001s
I20260129 03:00:08.653831 20432 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55672 (request call id 1709) took 1254 ms. Trace:
W20260129 03:00:08.637122 20406 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260129 03:00:08.528847 20407 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260129 03:00:08.523766 20513 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "926a250fa5834e88b96ccecbefbf5086" candidate_uuid: "197b880fa39d432a9b472845876bfaf0" candidate_term: 8 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "974b802141bd4f13beffef2aa03f88f9"
I20260129 03:00:08.866349 20507 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57600 (request call id 1917) took 1677 ms. Trace:
I20260129 03:00:08.452157 20590 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "75be4bd6bd45474795fd472eff2953ac" candidate_uuid: "974b802141bd4f13beffef2aa03f88f9" candidate_term: 11 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "1c50de89a5a845668696dcb28c3badd6" is_pre_election: true
I20260129 03:00:08.915630 20590 raft_consensus.cc:2468] T 75be4bd6bd45474795fd472eff2953ac P 1c50de89a5a845668696dcb28c3badd6 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 974b802141bd4f13beffef2aa03f88f9 in term 10.
I20260129 03:00:08.251075 23352 raft_consensus.cc:3060] T 87d5fe48a44a408aaf03e894a67ecb2f P 197b880fa39d432a9b472845876bfaf0 [term 6 FOLLOWER]: Advancing to term 7
I20260129 03:00:08.438462 20582 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "cedce331e1584ac8b636c9b8a72b2ba0" candidate_uuid: "197b880fa39d432a9b472845876bfaf0" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "1c50de89a5a845668696dcb28c3badd6" is_pre_election: true
W20260129 03:00:08.592839 20586 consensus_meta.cc:287] T 45cc5f3d2688486d9b6dbe7e666dedde P 1c50de89a5a845668696dcb28c3badd6: Time spent flushing consensus metadata: real 0.601s user 0.011s sys 0.000s
W20260129 03:00:08.503217 20588 consensus_meta.cc:287] T 4f5a77e94fef4b16a2ba7b067c8b144a P 1c50de89a5a845668696dcb28c3badd6: Time spent flushing consensus metadata: real 0.597s user 0.010s sys 0.001s
I20260129 03:00:08.917991 20582 raft_consensus.cc:2468] T cedce331e1584ac8b636c9b8a72b2ba0 P 1c50de89a5a845668696dcb28c3badd6 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 197b880fa39d432a9b472845876bfaf0 in term 7.
I20260129 03:00:08.846112 20584 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7d9494f5584d4d5db8b9adf1c8fc888b" candidate_uuid: "197b880fa39d432a9b472845876bfaf0" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "1c50de89a5a845668696dcb28c3badd6" is_pre_election: true
I20260129 03:00:08.848722 20434 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "dc2f4cb752cf4d59b5f6bc689f5d8104" candidate_uuid: "1c50de89a5a845668696dcb28c3badd6" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "197b880fa39d432a9b472845876bfaf0" is_pre_election: true
I20260129 03:00:08.920035 20434 raft_consensus.cc:2468] T dc2f4cb752cf4d59b5f6bc689f5d8104 P 197b880fa39d432a9b472845876bfaf0 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1c50de89a5a845668696dcb28c3badd6 in term 4.
I20260129 03:00:08.869428 20585 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2e58e9a5a8aa4cecafb464addb4f35ed" candidate_uuid: "197b880fa39d432a9b472845876bfaf0" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "1c50de89a5a845668696dcb28c3badd6" is_pre_election: true
I20260129 03:00:08.921324 20553 leader_election.cc:304] T dc2f4cb752cf4d59b5f6bc689f5d8104 P 1c50de89a5a845668696dcb28c3badd6 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 197b880fa39d432a9b472845876bfaf0, 1c50de89a5a845668696dcb28c3badd6; no voters:
I20260129 03:00:08.786584 20512 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57600 (request call id 1915) took 1884 ms. Trace:
I20260129 03:00:08.922111 20512 rpcz_store.cc:276] 0129 03:00:06.902323 (+ 0us) service_pool.cc:168] Inserting onto call queue
0129 03:00:07.635498 (+733175us) service_pool.cc:225] Handling call
0129 03:00:07.635790 (+ 292us) raft_consensus.cc:1471] Updating replica for 0 ops
0129 03:00:08.014272 (+378482us) raft_consensus.cc:1697] Filling consensus response to leader.
0129 03:00:08.786255 (+771983us) spinlock_profiling.cc:243] Waited 366 ms on lock 0x7b54002ef754. stack: 00007f8fe044e373 00007f8fe044e270 00007f8fdfda0574 00007f8fe9a5c6b4 00007f8fe9a5c649 00007f8fe34975a1 00007f8fe352ea81 00007f8fe352d4be 00007f8fe84e1f49 00007f8fe1a6ed34 00007f8fe1a6ecc2 00007f8fe1a6ebf1 00007f8fe1a6eb6c 00007f8fe1a6de22 00007f8fe186d34c 00007f8fe186c786
0129 03:00:08.786503 (+ 248us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":840828160}
W20260129 03:00:08.889920 20555 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260129 03:00:08.906100 20405 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.211.4:40439, user_credentials={real_user=slave}} blocked reactor thread for 78602.1us
W20260129 03:00:08.925216 20405 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260129 03:00:08.564519 20438 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c934d31efd3f4e5195605ddc354d14e1" candidate_uuid: "974b802141bd4f13beffef2aa03f88f9" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "197b880fa39d432a9b472845876bfaf0"
I20260129 03:00:08.776341 20509 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57600 (request call id 1921) took 1174 ms. Trace:
I20260129 03:00:08.927814 20509 rpcz_store.cc:276] 0129 03:00:07.601556 (+ 0us) service_pool.cc:168] Inserting onto call queue
0129 03:00:07.822992 (+221436us) service_pool.cc:225] Handling call
0129 03:00:07.844919 (+ 21927us) raft_consensus.cc:1471] Updating replica for 0 ops
0129 03:00:07.845153 (+ 234us) raft_consensus.cc:1534] Early marking committed up to index 5
0129 03:00:07.907752 (+ 62599us) log.cc:844] Serialized 14 byte log entry
0129 03:00:08.775262 (+867510us) spinlock_profiling.cc:243] Waited 10.2 ms on lock 0x7b18004fad40. stack: 00007f8fe044e373 00007f8fe044e270 00007f8fdfda0574 00007f8fe9a5c6b4 00007f8fe9a5c649 00007f8fe92d4c91 00007f8fe92e1671 00007f8fe03f546a 00007f8fe9a7b5c9 00007f8fe02b5160 00007f8fe047d3c4 00007f8fe047d335 00007f8fe6f28317 00007f8fe3538627 00007f8fe3547d3f 00007f8fe3547cb8
0129 03:00:08.775551 (+ 289us) log.cc:844] Serialized 14 byte log entry
0129 03:00:08.775847 (+ 296us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0129 03:00:08.775948 (+ 101us) raft_consensus.cc:1645] Marking committed up to 5
0129 03:00:08.776012 (+ 64us) raft_consensus.cc:1697] Filling consensus response to leader.
0129 03:00:08.776106 (+ 94us) raft_consensus.cc:1691] UpdateReplicas() finished
0129 03:00:08.776266 (+ 160us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":856947,"prepare.queue_time_us":938804,"prepare.run_cpu_time_us":84,"prepare.run_wall_time_us":86,"raft.queue_time_us":13596,"raft.run_cpu_time_us":240,"raft.run_wall_time_us":11475,"spinlock_wait_cycles":23496448}
I20260129 03:00:08.411988 20480 leader_election.cc:304] T 45cc5f3d2688486d9b6dbe7e666dedde P 974b802141bd4f13beffef2aa03f88f9 [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 974b802141bd4f13beffef2aa03f88f9; no voters: 1c50de89a5a845668696dcb28c3badd6, 7a9daf5d42a84615a6383cfec9a10608
W20260129 03:00:08.820523 23283 consensus_meta.cc:287] T 368953c9102643b2a21f99af36b51905 P 1c50de89a5a845668696dcb28c3badd6: Time spent flushing consensus metadata: real 1.618s user 0.014s sys 0.000s
I20260129 03:00:08.330276 23262 raft_consensus.cc:493] T e7d83d4ccf684e76acb29a0dc844e1b8 P 1c50de89a5a845668696dcb28c3badd6 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260129 03:00:08.637521 21099 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.211.1:42027, user_credentials={real_user=slave}} blocked reactor thread for 132002us
W20260129 03:00:08.637392 21102 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.211.2:34517, user_credentials={real_user=slave}} blocked reactor thread for 235214us
I20260129 03:00:08.918037 20586 raft_consensus.cc:2468] T 45cc5f3d2688486d9b6dbe7e666dedde P 1c50de89a5a845668696dcb28c3badd6 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7a9daf5d42a84615a6383cfec9a10608 in term 6.
W20260129 03:00:08.516852 23358 log.cc:927] Time spent T 1299723a4476420ab02fd8bfd7aa0645 P 197b880fa39d432a9b472845876bfaf0: Append to log took a long time: real 0.352s user 0.000s sys 0.000s
I20260129 03:00:08.323886 23304 leader_election.cc:290] T dc2f4cb752cf4d59b5f6bc689f5d8104 P 1c50de89a5a845668696dcb28c3badd6 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 197b880fa39d432a9b472845876bfaf0 (127.18.211.1:42027), 7a9daf5d42a84615a6383cfec9a10608 (127.18.211.4:40439)
I20260129 03:00:08.914667 20507 rpcz_store.cc:276] 0129 03:00:07.188742 (+ 0us) service_pool.cc:168] Inserting onto call queue
0129 03:00:07.733583 (+544841us) service_pool.cc:225] Handling call
0129 03:00:07.733927 (+ 344us) raft_consensus.cc:1471] Updating replica for 2 ops
0129 03:00:07.734330 (+ 403us) raft_consensus.cc:1534] Early marking committed up to index 6
0129 03:00:07.734357 (+ 27us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0129 03:00:07.734957 (+ 600us) log.cc:844] Serialized 52 byte log entry
0129 03:00:08.018447 (+283490us) raft_consensus.cc:1645] Marking committed up to 6
0129 03:00:08.018491 (+ 44us) raft_consensus.cc:1697] Filling consensus response to leader.
0129 03:00:08.018553 (+ 62us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0129 03:00:08.866045 (+847492us) raft_consensus.cc:1685] finished
0129 03:00:08.866075 (+ 30us) raft_consensus.cc:1691] UpdateReplicas() finished
0129 03:00:08.866270 (+ 195us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":282027,"raft.queue_time_us":2505,"raft.run_cpu_time_us":78,"raft.run_wall_time_us":79,"thread_start_us":281938,"threads_started":1,"wal-append.queue_time_us":825790}
I20260129 03:00:08.453765 23369 raft_consensus.cc:493] T e7d83d4ccf684e76acb29a0dc844e1b8 P 197b880fa39d432a9b472845876bfaf0 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260129 03:00:08.455354 20587 raft_consensus.cc:2393] T e33a417cf9c54cf5af73a4ee3aac4d3a P 1c50de89a5a845668696dcb28c3badd6 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 197b880fa39d432a9b472845876bfaf0 in current term 7: Already voted for candidate 1c50de89a5a845668696dcb28c3badd6 in this term.
I20260129 03:00:08.950804 20438 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e33a417cf9c54cf5af73a4ee3aac4d3a" candidate_uuid: "1c50de89a5a845668696dcb28c3badd6" candidate_term: 7 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "197b880fa39d432a9b472845876bfaf0"
I20260129 03:00:08.952155 20584 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "cedce331e1584ac8b636c9b8a72b2ba0" candidate_uuid: "197b880fa39d432a9b472845876bfaf0" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "1c50de89a5a845668696dcb28c3badd6"
I20260129 03:00:08.943332 23046 raft_consensus.cc:2749] T 45cc5f3d2688486d9b6dbe7e666dedde P 974b802141bd4f13beffef2aa03f88f9 [term 6 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20260129 03:00:08.953296 20507 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "8055232cf0004d40a85f3bf983fe00a9" candidate_uuid: "197b880fa39d432a9b472845876bfaf0" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "974b802141bd4f13beffef2aa03f88f9" is_pre_election: true
I20260129 03:00:08.329448 23378 raft_consensus.cc:493] T 2ac659d732314e1288849f08fe5e074e P 197b880fa39d432a9b472845876bfaf0 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 7a9daf5d42a84615a6383cfec9a10608)
W20260129 03:00:08.773826 23377 log.cc:927] Time spent T 3e2d15f8640b407195210c476e09df91 P 7a9daf5d42a84615a6383cfec9a10608: Append to log took a long time: real 0.198s user 0.000s sys 0.000s
I20260129 03:00:08.864959 23380 raft_consensus.cc:740] T cb0b3f259ebc4310927338a8740e87b8 P 7a9daf5d42a84615a6383cfec9a10608 [term 7 LEADER]: Becoming Follower/Learner. State: Replica: 7a9daf5d42a84615a6383cfec9a10608, State: Running, Role: LEADER
I20260129 03:00:08.354616 23091 leader_election.cc:290] T 926a250fa5834e88b96ccecbefbf5086 P 197b880fa39d432a9b472845876bfaf0 [CANDIDATE]: Term 8 election: Requested vote from peers 974b802141bd4f13beffef2aa03f88f9 (127.18.211.2:34517), 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3:37735)
W20260129 03:00:08.822217 23373 log.cc:927] Time spent T 42d27a28d89b409e8807ec0efd97af69 P 7a9daf5d42a84615a6383cfec9a10608: Append to log took a long time: real 0.633s user 0.001s sys 0.000s
I20260129 03:00:08.355203 23292 leader_election.cc:290] T 75f45538dc704d9a8ec4ff8ad9f3ea5b P 1c50de89a5a845668696dcb28c3badd6 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 974b802141bd4f13beffef2aa03f88f9 (127.18.211.2:34517), 7a9daf5d42a84615a6383cfec9a10608 (127.18.211.4:40439)
W20260129 03:00:08.823493 23355 log.cc:927] Time spent T 913fc7d751ed4b7388d889aac30a5f9b P 974b802141bd4f13beffef2aa03f88f9: Append to log took a long time: real 0.685s user 0.000s sys 0.000s
I20260129 03:00:08.914592 20513 raft_consensus.cc:3060] T 926a250fa5834e88b96ccecbefbf5086 P 974b802141bd4f13beffef2aa03f88f9 [term 7 FOLLOWER]: Advancing to term 8
W20260129 03:00:08.822214 23207 consensus_meta.cc:287] T c2633a6fc34c44579de348d208b0c01e P 974b802141bd4f13beffef2aa03f88f9: Time spent flushing consensus metadata: real 0.702s user 0.010s sys 0.001s
I20260129 03:00:09.163357 23207 raft_consensus.cc:515] T c2633a6fc34c44579de348d208b0c01e P 974b802141bd4f13beffef2aa03f88f9 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 03:00:09.164583 23091 raft_consensus.cc:493] T 368953c9102643b2a21f99af36b51905 P 197b880fa39d432a9b472845876bfaf0 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260129 03:00:09.165190 23091 raft_consensus.cc:515] T 368953c9102643b2a21f99af36b51905 P 197b880fa39d432a9b472845876bfaf0 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 03:00:09.166224 23207 leader_election.cc:290] T c2633a6fc34c44579de348d208b0c01e P 974b802141bd4f13beffef2aa03f88f9 [CANDIDATE]: Term 5 election: Requested vote from peers 197b880fa39d432a9b472845876bfaf0 (127.18.211.1:42027), 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3:37735)
W20260129 03:00:08.775574 23381 log.cc:927] Time spent T 724a236524cb4e7ba3428880681d6cc2 P 197b880fa39d432a9b472845876bfaf0: Append to log took a long time: real 0.212s user 0.001s sys 0.000s
I20260129 03:00:09.167248 23091 leader_election.cc:290] T 368953c9102643b2a21f99af36b51905 P 197b880fa39d432a9b472845876bfaf0 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 974b802141bd4f13beffef2aa03f88f9 (127.18.211.2:34517), 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3:37735)
W20260129 03:00:08.755226 23333 consensus_meta.cc:287] T 87d5fe48a44a408aaf03e894a67ecb2f P 974b802141bd4f13beffef2aa03f88f9: Time spent flushing consensus metadata: real 0.631s user 0.002s sys 0.006s
I20260129 03:00:09.168962 23333 raft_consensus.cc:515] T 87d5fe48a44a408aaf03e894a67ecb2f P 974b802141bd4f13beffef2aa03f88f9 [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } }
I20260129 03:00:09.171715 23333 leader_election.cc:290] T 87d5fe48a44a408aaf03e894a67ecb2f P 974b802141bd4f13beffef2aa03f88f9 [CANDIDATE]: Term 6 election: Requested vote from peers 197b880fa39d432a9b472845876bfaf0 (127.18.211.1:42027), 7a9daf5d42a84615a6383cfec9a10608 (127.18.211.4:40439)
I20260129 03:00:09.172163 23205 raft_consensus.cc:493] T 4e420f90884f4796997b5ea41e49e411 P 974b802141bd4f13beffef2aa03f88f9 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 7a9daf5d42a84615a6383cfec9a10608)
I20260129 03:00:09.172452 23333 raft_consensus.cc:493] T 96609ac68bdf4993a12e1c23355ca013 P 974b802141bd4f13beffef2aa03f88f9 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 7a9daf5d42a84615a6383cfec9a10608)
W20260129 03:00:08.651837 23357 log.cc:927] Time spent T cedce331e1584ac8b636c9b8a72b2ba0 P 197b880fa39d432a9b472845876bfaf0: Append to log took a long time: real 0.198s user 0.001s sys 0.000s
W20260129 03:00:08.750048 22694 consensus_meta.cc:287] T fcddab6bca684adf8ea032289a00f90e P 974b802141bd4f13beffef2aa03f88f9: Time spent flushing consensus metadata: real 1.379s user 0.011s sys 0.002s
I20260129 03:00:08.353068 23232 leader_election.cc:290] T a609ac02fae84725ab743f95f51d33ce P 974b802141bd4f13beffef2aa03f88f9 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 197b880fa39d432a9b472845876bfaf0 (127.18.211.1:42027), 7a9daf5d42a84615a6383cfec9a10608 (127.18.211.4:40439)
I20260129 03:00:08.337778 21127 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c934d31efd3f4e5195605ddc354d14e1" candidate_uuid: "974b802141bd4f13beffef2aa03f88f9" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "7a9daf5d42a84615a6383cfec9a10608"
I20260129 03:00:08.262369 22855 raft_consensus.cc:697] T 78aa39096868415d9afc2ffcb248e198 P 1c50de89a5a845668696dcb28c3badd6 [term 7 LEADER]: Becoming Leader. State: Replica: 1c50de89a5a845668696dcb28c3badd6, State: Running, Role: LEADER
I20260129 03:00:08.779187 22891 raft_consensus.cc:493] T 1ce4ddf5b10e4ae8b193b13b2e5936ee P 1c50de89a5a845668696dcb28c3badd6 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260129 03:00:08.943476 20582 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7a9a69cd82724b1faed1d10aa091fc4a" candidate_uuid: "974b802141bd4f13beffef2aa03f88f9" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "1c50de89a5a845668696dcb28c3badd6" is_pre_election: true
I20260129 03:00:08.337533 23152 raft_consensus.cc:493] T a644c02b16af47679824ccd060ac10fd P 974b802141bd4f13beffef2aa03f88f9 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 1c50de89a5a845668696dcb28c3badd6)
I20260129 03:00:08.912987 20432 rpcz_store.cc:276] 0129 03:00:07.399706 (+ 0us) service_pool.cc:168] Inserting onto call queue
0129 03:00:07.502708 (+103002us) service_pool.cc:225] Handling call
0129 03:00:07.503143 (+ 435us) raft_consensus.cc:1471] Updating replica for 1 ops
0129 03:00:07.909156 (+406013us) raft_consensus.cc:1534] Early marking committed up to index 0
0129 03:00:07.909189 (+ 33us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0129 03:00:07.909661 (+ 472us) log.cc:844] Serialized 26 byte log entry
0129 03:00:08.070415 (+160754us) raft_consensus.cc:1645] Marking committed up to 4
0129 03:00:08.070457 (+ 42us) raft_consensus.cc:1697] Filling consensus response to leader.
0129 03:00:08.070523 (+ 66us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0129 03:00:08.653568 (+583045us) raft_consensus.cc:1685] finished
0129 03:00:08.653590 (+ 22us) raft_consensus.cc:1691] UpdateReplicas() finished
0129 03:00:08.653761 (+ 171us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":16332,"raft.queue_time_us":40662,"raft.run_cpu_time_us":122,"raft.run_wall_time_us":3223,"thread_start_us":476126,"threads_started":2,"wal-append.queue_time_us":598959}
I20260129 03:00:08.944506 20585 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c2633a6fc34c44579de348d208b0c01e" candidate_uuid: "197b880fa39d432a9b472845876bfaf0" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "1c50de89a5a845668696dcb28c3badd6" is_pre_election: true
I20260129 03:00:08.945101 20586 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:48526 (request call id 378) took 1061 ms. Trace:
I20260129 03:00:09.217698 20586 rpcz_store.cc:276] 0129 03:00:07.880973 (+ 0us) service_pool.cc:168] Inserting onto call queue
0129 03:00:07.940992 (+ 60019us) service_pool.cc:225] Handling call
0129 03:00:08.942165 (+1001173us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20260129 03:00:08.510931 23353 log.cc:927] Time spent T a1a39dcd328e4268ae41b988c66b4a75 P 974b802141bd4f13beffef2aa03f88f9: Append to log took a long time: real 0.187s user 0.000s sys 0.000s
I20260129 03:00:09.172729 23205 raft_consensus.cc:515] T 4e420f90884f4796997b5ea41e49e411 P 974b802141bd4f13beffef2aa03f88f9 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } }
I20260129 03:00:08.827658 23368 raft_consensus.cc:493] T c891d2bff4ce4568861bbc992e91fbb0 P 197b880fa39d432a9b472845876bfaf0 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260129 03:00:08.321292 23327 log.cc:927] Time spent T a8479d1fe0bd41e5946186e3a6ca8c7a P 197b880fa39d432a9b472845876bfaf0: Append to log took a long time: real 0.069s user 0.000s sys 0.000s
I20260129 03:00:08.664278 21128 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a609ac02fae84725ab743f95f51d33ce" candidate_uuid: "974b802141bd4f13beffef2aa03f88f9" candidate_term: 9 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "7a9daf5d42a84615a6383cfec9a10608" is_pre_election: true
I20260129 03:00:08.947589 20512 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a1a39dcd328e4268ae41b988c66b4a75" candidate_uuid: "197b880fa39d432a9b472845876bfaf0" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "974b802141bd4f13beffef2aa03f88f9" is_pre_election: true
I20260129 03:00:08.909861 23104 raft_consensus.cc:493] T 0340229d521b432580ae926f6062e3f7 P 1c50de89a5a845668696dcb28c3badd6 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260129 03:00:08.264483 22857 consensus_queue.cc:1048] T 4e3805ffa70242e8ade66da260dddf66 P 7a9daf5d42a84615a6383cfec9a10608 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.672s
I20260129 03:00:08.774662 23026 leader_election.cc:290] T e33a417cf9c54cf5af73a4ee3aac4d3a P 1c50de89a5a845668696dcb28c3badd6 [CANDIDATE]: Term 7 election: Requested vote from peers 197b880fa39d432a9b472845876bfaf0 (127.18.211.1:42027), 974b802141bd4f13beffef2aa03f88f9 (127.18.211.2:34517)
I20260129 03:00:08.952827 20584 raft_consensus.cc:3060] T cedce331e1584ac8b636c9b8a72b2ba0 P 1c50de89a5a845668696dcb28c3badd6 [term 7 FOLLOWER]: Advancing to term 8
I20260129 03:00:08.953405 23046 raft_consensus.cc:493] T 7d9494f5584d4d5db8b9adf1c8fc888b P 974b802141bd4f13beffef2aa03f88f9 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 1c50de89a5a845668696dcb28c3badd6)
I20260129 03:00:08.774847 23089 leader_election.cc:290] T a1a39dcd328e4268ae41b988c66b4a75 P 197b880fa39d432a9b472845876bfaf0 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 974b802141bd4f13beffef2aa03f88f9 (127.18.211.2:34517), 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3:37735)
I20260129 03:00:08.953958 20507 raft_consensus.cc:2468] T 8055232cf0004d40a85f3bf983fe00a9 P 974b802141bd4f13beffef2aa03f88f9 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 197b880fa39d432a9b472845876bfaf0 in term 5.
I20260129 03:00:08.262606 23325 leader_election.cc:290] T 926a250fa5834e88b96ccecbefbf5086 P 1c50de89a5a845668696dcb28c3badd6 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 197b880fa39d432a9b472845876bfaf0 (127.18.211.1:42027), 974b802141bd4f13beffef2aa03f88f9 (127.18.211.2:34517)
I20260129 03:00:08.916839 23298 raft_consensus.cc:493] T 0f20af7fbe5d4db89a6f17948d98912c P 1c50de89a5a845668696dcb28c3badd6 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260129 03:00:08.779664 23216 leader_election.cc:290] T 0f20af7fbe5d4db89a6f17948d98912c P 974b802141bd4f13beffef2aa03f88f9 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 197b880fa39d432a9b472845876bfaf0 (127.18.211.1:42027), 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3:37735)
W20260129 03:00:08.936760 23342 consensus_meta.cc:287] T a609ac02fae84725ab743f95f51d33ce P 197b880fa39d432a9b472845876bfaf0: Time spent flushing consensus metadata: real 0.911s user 0.009s sys 0.003s
I20260129 03:00:08.396394 23280 raft_consensus.cc:493] T 96609ac68bdf4993a12e1c23355ca013 P 1c50de89a5a845668696dcb28c3badd6 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 7a9daf5d42a84615a6383cfec9a10608)
W20260129 03:00:08.955442 20510 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.543s user 0.025s sys 0.007s
I20260129 03:00:09.058677 23378 raft_consensus.cc:515] T 2ac659d732314e1288849f08fe5e074e P 197b880fa39d432a9b472845876bfaf0 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } }
W20260129 03:00:09.285565 20510 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.875s user 0.025s sys 0.008s
I20260129 03:00:08.918609 20588 raft_consensus.cc:2468] T 4f5a77e94fef4b16a2ba7b067c8b144a P 1c50de89a5a845668696dcb28c3badd6 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7a9daf5d42a84615a6383cfec9a10608 in term 5.
I20260129 03:00:09.286370 20588 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:48526 (request call id 374) took 1452 ms. Trace:
W20260129 03:00:08.511819 23363 log.cc:927] Time spent T 75b0b6e82ddc4f92b1fe1d94f9a0b73b P 974b802141bd4f13beffef2aa03f88f9: Append to log took a long time: real 0.213s user 0.000s sys 0.000s
W20260129 03:00:08.946761 21100 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.714s user 0.014s sys 0.002s
W20260129 03:00:09.294239 21100 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 1.065s user 0.014s sys 0.002s
W20260129 03:00:09.294538 21100 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.211.3:37735, user_credentials={real_user=slave}} blocked reactor thread for 1.06678e+06us
I20260129 03:00:08.995574 20590 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a0fcc58d6fb6473cafbd9d34d5195269" candidate_uuid: "974b802141bd4f13beffef2aa03f88f9" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "1c50de89a5a845668696dcb28c3badd6" is_pre_election: true
I20260129 03:00:09.295367 20590 raft_consensus.cc:2393] T a0fcc58d6fb6473cafbd9d34d5195269 P 1c50de89a5a845668696dcb28c3badd6 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 974b802141bd4f13beffef2aa03f88f9 in current term 5: Already voted for candidate 197b880fa39d432a9b472845876bfaf0 in this term.
I20260129 03:00:08.542649 23376 raft_consensus.cc:493] T dc2f4cb752cf4d59b5f6bc689f5d8104 P 7a9daf5d42a84615a6383cfec9a10608 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260129 03:00:08.297996 23178 raft_consensus.cc:515] T 4e3805ffa70242e8ade66da260dddf66 P 197b880fa39d432a9b472845876bfaf0 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } }
I20260129 03:00:09.117635 20434 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a609ac02fae84725ab743f95f51d33ce" candidate_uuid: "974b802141bd4f13beffef2aa03f88f9" candidate_term: 9 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "197b880fa39d432a9b472845876bfaf0" is_pre_election: true
I20260129 03:00:08.954312 23346 leader_election.cc:290] T 8055232cf0004d40a85f3bf983fe00a9 P 197b880fa39d432a9b472845876bfaf0 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 974b802141bd4f13beffef2aa03f88f9 (127.18.211.2:34517), 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3:37735)
I20260129 03:00:08.984068 23304 raft_consensus.cc:2804] T dc2f4cb752cf4d59b5f6bc689f5d8104 P 1c50de89a5a845668696dcb28c3badd6 [term 5 FOLLOWER]: Leader pre-election won for term 6
W20260129 03:00:09.062186 23290 consensus_meta.cc:287] T 36ff33be666a4c82954731691dbcf327 P 1c50de89a5a845668696dcb28c3badd6: Time spent flushing consensus metadata: real 1.526s user 0.014s sys 0.000s
I20260129 03:00:09.208632 20436 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55672 (request call id 1704) took 2162 ms. Trace:
I20260129 03:00:09.390301 20436 rpcz_store.cc:276] 0129 03:00:07.046079 (+ 0us) service_pool.cc:168] Inserting onto call queue
0129 03:00:07.113088 (+ 67009us) service_pool.cc:225] Handling call
0129 03:00:07.130198 (+ 17110us) raft_consensus.cc:1471] Updating replica for 2 ops
0129 03:00:07.166791 (+ 36593us) raft_consensus.cc:1534] Early marking committed up to index 2
0129 03:00:08.050908 (+884117us) spinlock_profiling.cc:243] Waited 3.14 ms on lock 0x7b180037bee0. stack: 00007f8fe044e373 00007f8fe044e270 00007f8fdfda0574 00007f8fe9a5c6b4 00007f8fe9a5c649 00007f8fe92d4c91 00007f8fe92e1671 00007f8fe0462c67 00007f8fe924b709 00007f8fe047f57b 00007f8fe047d81f 00007f8fe047d335 00007f8fe6f28317 00007f8fe3538627 00007f8fe3547d3f 00007f8fe3547cb8
0129 03:00:08.115922 (+ 65014us) log.cc:844] Serialized 14 byte log entry
0129 03:00:08.253997 (+138075us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0129 03:00:08.254787 (+ 790us) log.cc:844] Serialized 52 byte log entry
0129 03:00:08.254957 (+ 170us) raft_consensus.cc:1645] Marking committed up to 2
0129 03:00:08.255003 (+ 46us) raft_consensus.cc:1697] Filling consensus response to leader.
0129 03:00:08.255095 (+ 92us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0129 03:00:09.173348 (+918253us) raft_consensus.cc:1685] finished
0129 03:00:09.204778 (+ 31430us) raft_consensus.cc:1691] UpdateReplicas() finished
0129 03:00:09.208537 (+ 3759us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":69886,"prepare.queue_time_us":383986,"prepare.run_cpu_time_us":51,"prepare.run_wall_time_us":62,"raft.queue_time_us":282267,"raft.run_cpu_time_us":305,"raft.run_wall_time_us":5612,"spinlock_wait_cycles":7218560,"thread_start_us":153602,"threads_started":2,"wal-append.queue_time_us":419412}
I20260129 03:00:09.215590 20582 raft_consensus.cc:2468] T 7a9a69cd82724b1faed1d10aa091fc4a P 1c50de89a5a845668696dcb28c3badd6 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 974b802141bd4f13beffef2aa03f88f9 in term 7.
W20260129 03:00:08.519418 23208 consensus_meta.cc:287] T 17d6431cc77a491e8464d68db4081a51 P 974b802141bd4f13beffef2aa03f88f9: Time spent flushing consensus metadata: real 0.963s user 0.006s sys 0.002s
I20260129 03:00:09.173156 23333 raft_consensus.cc:515] T 96609ac68bdf4993a12e1c23355ca013 P 974b802141bd4f13beffef2aa03f88f9 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } attrs { replace: true } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } }
W20260129 03:00:09.231384 23308 log.cc:927] Time spent T 96609ac68bdf4993a12e1c23355ca013 P 1c50de89a5a845668696dcb28c3badd6: Append to log took a long time: real 2.603s user 0.008s sys 0.014s
I20260129 03:00:09.232048 20513 raft_consensus.cc:2410] T 926a250fa5834e88b96ccecbefbf5086 P 974b802141bd4f13beffef2aa03f88f9 [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate 197b880fa39d432a9b472845876bfaf0 for term 8 because replica has last-logged OpId of term: 6 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 5 index: 4.
I20260129 03:00:08.943842 23283 raft_consensus.cc:515] T 368953c9102643b2a21f99af36b51905 P 1c50de89a5a845668696dcb28c3badd6 [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 03:00:09.232631 20437 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:48356 (request call id 414) took 1621 ms. Trace:
I20260129 03:00:09.232429 23152 raft_consensus.cc:515] T a644c02b16af47679824ccd060ac10fd P 974b802141bd4f13beffef2aa03f88f9 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } }
I20260129 03:00:09.409081 23152 leader_election.cc:290] T a644c02b16af47679824ccd060ac10fd P 974b802141bd4f13beffef2aa03f88f9 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3:37735), 7a9daf5d42a84615a6383cfec9a10608 (127.18.211.4:40439)
I20260129 03:00:09.410013 20584 raft_consensus.cc:2468] T cedce331e1584ac8b636c9b8a72b2ba0 P 1c50de89a5a845668696dcb28c3badd6 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 197b880fa39d432a9b472845876bfaf0 in term 8.
I20260129 03:00:09.233320 20585 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:35754 (request call id 1809) took 1031 ms. Trace:
I20260129 03:00:09.411623 20584 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "76f713c3331540d2a5b76164103e9eb1" candidate_uuid: "974b802141bd4f13beffef2aa03f88f9" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "1c50de89a5a845668696dcb28c3badd6"
I20260129 03:00:09.216735 22694 raft_consensus.cc:515] T fcddab6bca684adf8ea032289a00f90e P 974b802141bd4f13beffef2aa03f88f9 [term 10 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 03:00:09.241236 22855 consensus_queue.cc:237] T 78aa39096868415d9afc2ffcb248e198 P 1c50de89a5a845668696dcb28c3badd6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 5.6, Last appended by leader: 6, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } }
I20260129 03:00:09.242837 20432 raft_consensus.cc:1275] T a1a39dcd328e4268ae41b988c66b4a75 P 197b880fa39d432a9b472845876bfaf0 [term 5 FOLLOWER]: Refusing update from remote peer 974b802141bd4f13beffef2aa03f88f9: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 5 index: 4. (index mismatch)
I20260129 03:00:09.243322 22891 raft_consensus.cc:515] T 1ce4ddf5b10e4ae8b193b13b2e5936ee P 1c50de89a5a845668696dcb28c3badd6 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
W20260129 03:00:09.436332 23391 log.cc:927] Time spent T a609ac02fae84725ab743f95f51d33ce P 7a9daf5d42a84615a6383cfec9a10608: Append to log took a long time: real 0.157s user 0.001s sys 0.000s
I20260129 03:00:09.436864 22891 leader_election.cc:290] T 1ce4ddf5b10e4ae8b193b13b2e5936ee P 1c50de89a5a845668696dcb28c3badd6 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 197b880fa39d432a9b472845876bfaf0 (127.18.211.1:42027), 974b802141bd4f13beffef2aa03f88f9 (127.18.211.2:34517)
I20260129 03:00:09.246567 20586 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4f8f3a38f6e24ba4857875181f447442" candidate_uuid: "197b880fa39d432a9b472845876bfaf0" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "1c50de89a5a845668696dcb28c3badd6" is_pre_election: true
I20260129 03:00:09.437883 20586 raft_consensus.cc:2468] T 4f8f3a38f6e24ba4857875181f447442 P 1c50de89a5a845668696dcb28c3badd6 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 197b880fa39d432a9b472845876bfaf0 in term 6.
I20260129 03:00:09.439126 20586 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a1a39dcd328e4268ae41b988c66b4a75" candidate_uuid: "197b880fa39d432a9b472845876bfaf0" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "1c50de89a5a845668696dcb28c3badd6" is_pre_election: true
I20260129 03:00:09.439944 20586 raft_consensus.cc:2468] T a1a39dcd328e4268ae41b988c66b4a75 P 1c50de89a5a845668696dcb28c3badd6 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 197b880fa39d432a9b472845876bfaf0 in term 5.
I20260129 03:00:09.441051 20586 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fcddab6bca684adf8ea032289a00f90e" candidate_uuid: "974b802141bd4f13beffef2aa03f88f9" candidate_term: 10 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "1c50de89a5a845668696dcb28c3badd6" is_pre_election: true
I20260129 03:00:08.947108 23262 raft_consensus.cc:515] T e7d83d4ccf684e76acb29a0dc844e1b8 P 1c50de89a5a845668696dcb28c3badd6 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
W20260129 03:00:09.245460 20515 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.924s user 0.062s sys 0.115s
W20260129 03:00:09.442021 20515 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 1.121s user 0.062s sys 0.116s
I20260129 03:00:09.177974 23091 consensus_queue.cc:1048] T 00e79d8f25a34676b1d866eb2068a2b2 P 197b880fa39d432a9b472845876bfaf0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.098s
I20260129 03:00:09.277467 21131 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34046 (request call id 560) took 5481 ms. Trace:
I20260129 03:00:09.443217 23262 leader_election.cc:290] T e7d83d4ccf684e76acb29a0dc844e1b8 P 1c50de89a5a845668696dcb28c3badd6 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 197b880fa39d432a9b472845876bfaf0 (127.18.211.1:42027), 974b802141bd4f13beffef2aa03f88f9 (127.18.211.2:34517)
I20260129 03:00:09.443457 21131 rpcz_store.cc:276] 0129 03:00:03.795824 (+ 0us) service_pool.cc:168] Inserting onto call queue
0129 03:00:03.905381 (+109557us) service_pool.cc:225] Handling call
0129 03:00:07.127532 (+3222151us) raft_consensus.cc:1471] Updating replica for 0 ops
0129 03:00:07.381371 (+253839us) raft_consensus.cc:1697] Filling consensus response to leader.
0129 03:00:09.271289 (+1889918us) spinlock_profiling.cc:243] Waited 3.16 s on lock 0x7b54003d9ad0. stack: 00007f8fe044e373 00007f8fe044e270 00007f8fdfda0574 00007f8fe9a5c6b4 00007f8fe9a5c649 00007f8fe34975a1 00007f8fe352d52d 00007f8fe84e1f49 00007f8fe1a6ed34 00007f8fe1a6ecc2 00007f8fe1a6ebf1 00007f8fe1a6eb6c 00007f8fe1a6de22 00007f8fe186d34c 00007f8fe186c786 00007f8fe186c12f
0129 03:00:09.277390 (+ 6101us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":7279351680}
I20260129 03:00:09.444173 20586 raft_consensus.cc:2468] T fcddab6bca684adf8ea032289a00f90e P 1c50de89a5a845668696dcb28c3badd6 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 974b802141bd4f13beffef2aa03f88f9 in term 9.
I20260129 03:00:09.446373 20586 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "96609ac68bdf4993a12e1c23355ca013" candidate_uuid: "974b802141bd4f13beffef2aa03f88f9" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "1c50de89a5a845668696dcb28c3badd6" is_pre_election: true
I20260129 03:00:09.277377 23046 raft_consensus.cc:515] T 7d9494f5584d4d5db8b9adf1c8fc888b P 974b802141bd4f13beffef2aa03f88f9 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 03:00:09.300426 23280 raft_consensus.cc:515] T 96609ac68bdf4993a12e1c23355ca013 P 1c50de89a5a845668696dcb28c3badd6 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } attrs { replace: true } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } }
I20260129 03:00:09.286715 20588 rpcz_store.cc:276] 0129 03:00:07.833740 (+ 0us) service_pool.cc:168] Inserting onto call queue
0129 03:00:07.834018 (+ 278us) service_pool.cc:225] Handling call
0129 03:00:09.286310 (+1452292us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260129 03:00:09.449151 20588 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "926a250fa5834e88b96ccecbefbf5086" candidate_uuid: "197b880fa39d432a9b472845876bfaf0" candidate_term: 8 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "1c50de89a5a845668696dcb28c3badd6"
I20260129 03:00:09.449678 20588 raft_consensus.cc:3060] T 926a250fa5834e88b96ccecbefbf5086 P 1c50de89a5a845668696dcb28c3badd6 [term 7 FOLLOWER]: Advancing to term 8
I20260129 03:00:09.449558 20586 raft_consensus.cc:2468] T 96609ac68bdf4993a12e1c23355ca013 P 1c50de89a5a845668696dcb28c3badd6 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 974b802141bd4f13beffef2aa03f88f9 in term 6.
I20260129 03:00:09.290716 20589 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:48526 (request call id 379) took 1399 ms. Trace:
I20260129 03:00:09.298254 20590 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:35752 (request call id 1675) took 1229 ms. Trace:
I20260129 03:00:09.450393 20590 rpcz_store.cc:276] 0129 03:00:08.068763 (+ 0us) service_pool.cc:168] Inserting onto call queue
0129 03:00:08.938277 (+869514us) service_pool.cc:225] Handling call
0129 03:00:09.298192 (+359915us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260129 03:00:09.297465 21100 leader_election.cc:304] T a644c02b16af47679824ccd060ac10fd P 7a9daf5d42a84615a6383cfec9a10608 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7a9daf5d42a84615a6383cfec9a10608; no voters: 1c50de89a5a845668696dcb28c3badd6, 974b802141bd4f13beffef2aa03f88f9
W20260129 03:00:09.451968 21100 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.211.3:37735, user_credentials={real_user=slave}} blocked reactor thread for 154584us
I20260129 03:00:09.450265 20589 rpcz_store.cc:276] 0129 03:00:07.890889 (+ 0us) service_pool.cc:168] Inserting onto call queue
0129 03:00:07.949024 (+ 58135us) service_pool.cc:225] Handling call
0129 03:00:07.949404 (+ 380us) raft_consensus.cc:1471] Updating replica for 1 ops
0129 03:00:07.967199 (+ 17795us) raft_consensus.cc:1534] Early marking committed up to index 0
0129 03:00:07.967245 (+ 46us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0129 03:00:07.967837 (+ 592us) log.cc:844] Serialized 26 byte log entry
0129 03:00:08.473595 (+505758us) raft_consensus.cc:1645] Marking committed up to 6
0129 03:00:08.473650 (+ 55us) raft_consensus.cc:1697] Filling consensus response to leader.
0129 03:00:08.473737 (+ 87us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0129 03:00:09.290406 (+816669us) raft_consensus.cc:1685] finished
0129 03:00:09.290437 (+ 31us) raft_consensus.cc:1691] UpdateReplicas() finished
0129 03:00:09.290640 (+ 203us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":25182,"raft.queue_time_us":91,"raft.run_cpu_time_us":122,"raft.run_wall_time_us":8927,"thread_start_us":457012,"threads_started":1,"wal-append.queue_time_us":1286694}
I20260129 03:00:09.453747 21100 leader_election.cc:304] T 78aa39096868415d9afc2ffcb248e198 P 7a9daf5d42a84615a6383cfec9a10608 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7a9daf5d42a84615a6383cfec9a10608; no voters: 197b880fa39d432a9b472845876bfaf0, 1c50de89a5a845668696dcb28c3badd6
W20260129 03:00:09.286011 23344 consensus_meta.cc:287] T febffe4a8b394368b69b1d164308fb9a P 197b880fa39d432a9b472845876bfaf0: Time spent flushing consensus metadata: real 1.041s user 0.008s sys 0.005s
I20260129 03:00:09.264422 23368 raft_consensus.cc:515] T c891d2bff4ce4568861bbc992e91fbb0 P 197b880fa39d432a9b472845876bfaf0 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 03:00:09.396814 20513 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:57600 (request call id 1926) took 1016 ms. Trace:
I20260129 03:00:09.396791 20437 rpcz_store.cc:276] 0129 03:00:07.611303 (+ 0us) service_pool.cc:168] Inserting onto call queue
0129 03:00:07.629337 (+ 18034us) service_pool.cc:225] Handling call
0129 03:00:07.629629 (+ 292us) raft_consensus.cc:1471] Updating replica for 1 ops
0129 03:00:07.629827 (+ 198us) raft_consensus.cc:1534] Early marking committed up to index 0
0129 03:00:07.629853 (+ 26us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0129 03:00:07.630289 (+ 436us) log.cc:844] Serialized 26 byte log entry
0129 03:00:07.910913 (+280624us) raft_consensus.cc:1645] Marking committed up to 8
0129 03:00:07.910958 (+ 45us) raft_consensus.cc:1697] Filling consensus response to leader.
0129 03:00:07.911026 (+ 68us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0129 03:00:09.232413 (+1321387us) raft_consensus.cc:1685] finished
0129 03:00:09.232433 (+ 20us) raft_consensus.cc:1691] UpdateReplicas() finished
0129 03:00:09.232570 (+ 137us) inbound_call.cc:173] Queueing success response
Metrics: {"thread_start_us":264141,"threads_started":1,"wal-append.queue_time_us":439682}
I20260129 03:00:09.398728 22857 raft_consensus.cc:493] T 18f30ebaa4174fd28c958e72d0cc7e24 P 7a9daf5d42a84615a6383cfec9a10608 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260129 03:00:09.400924 23333 leader_election.cc:290] T 96609ac68bdf4993a12e1c23355ca013 P 974b802141bd4f13beffef2aa03f88f9 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 197b880fa39d432a9b472845876bfaf0 (127.18.211.1:42027), 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3:37735), 7a9daf5d42a84615a6383cfec9a10608 (127.18.211.4:40439)
I20260129 03:00:09.402141 20582 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3e2d15f8640b407195210c476e09df91" candidate_uuid: "974b802141bd4f13beffef2aa03f88f9" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "1c50de89a5a845668696dcb28c3badd6" is_pre_election: true
I20260129 03:00:09.286986 23298 raft_consensus.cc:515] T 0f20af7fbe5d4db89a6f17948d98912c P 1c50de89a5a845668696dcb28c3badd6 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 03:00:09.404397 23178 leader_election.cc:290] T 4e3805ffa70242e8ade66da260dddf66 P 197b880fa39d432a9b472845876bfaf0 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3:37735), 7a9daf5d42a84615a6383cfec9a10608 (127.18.211.4:40439)
I20260129 03:00:09.390822 23304 raft_consensus.cc:493] T dc2f4cb752cf4d59b5f6bc689f5d8104 P 1c50de89a5a845668696dcb28c3badd6 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260129 03:00:09.406622 20436 raft_consensus.cc:1240] T 76f713c3331540d2a5b76164103e9eb1 P 197b880fa39d432a9b472845876bfaf0 [term 6 FOLLOWER]: Rejecting Update request from peer 1c50de89a5a845668696dcb28c3badd6 for earlier term 5. Current term is 6. Ops: []
I20260129 03:00:09.278817 23104 raft_consensus.cc:515] T 0340229d521b432580ae926f6062e3f7 P 1c50de89a5a845668696dcb28c3badd6 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 03:00:09.387912 22932 raft_consensus.cc:493] T febffe4a8b394368b69b1d164308fb9a P 7a9daf5d42a84615a6383cfec9a10608 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 974b802141bd4f13beffef2aa03f88f9)
I20260129 03:00:09.421161 20441 raft_consensus.cc:1275] T 4e3805ffa70242e8ade66da260dddf66 P 197b880fa39d432a9b472845876bfaf0 [term 6 FOLLOWER]: Refusing update from remote peer 7a9daf5d42a84615a6383cfec9a10608: Log matching property violated. Preceding OpId in replica: term: 3 index: 6. Preceding OpId from leader: term: 6 index: 8. (index mismatch)
I20260129 03:00:09.412209 20585 rpcz_store.cc:276] 0129 03:00:08.201633 (+ 0us) service_pool.cc:168] Inserting onto call queue
0129 03:00:08.921723 (+720090us) service_pool.cc:225] Handling call
0129 03:00:09.233254 (+311531us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260129 03:00:09.056895 23369 raft_consensus.cc:515] T e7d83d4ccf684e76acb29a0dc844e1b8 P 197b880fa39d432a9b472845876bfaf0 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 03:00:09.288771 23389 raft_consensus.cc:493] T fcddab6bca684adf8ea032289a00f90e P 197b880fa39d432a9b472845876bfaf0 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260129 03:00:09.457825 20513 rpcz_store.cc:276] 0129 03:00:08.380656 (+ 0us) service_pool.cc:168] Inserting onto call queue
0129 03:00:08.523482 (+142826us) service_pool.cc:225] Handling call
0129 03:00:09.396749 (+873267us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260129 03:00:09.406694 23283 leader_election.cc:290] T 368953c9102643b2a21f99af36b51905 P 1c50de89a5a845668696dcb28c3badd6 [CANDIDATE]: Term 8 election: Requested vote from peers 197b880fa39d432a9b472845876bfaf0 (127.18.211.1:42027), 974b802141bd4f13beffef2aa03f88f9 (127.18.211.2:34517)
I20260129 03:00:09.464502 20582 raft_consensus.cc:2468] T 3e2d15f8640b407195210c476e09df91 P 1c50de89a5a845668696dcb28c3badd6 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 974b802141bd4f13beffef2aa03f88f9 in term 4.
I20260129 03:00:09.465945 23109 raft_consensus.cc:2749] T a644c02b16af47679824ccd060ac10fd P 7a9daf5d42a84615a6383cfec9a10608 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260129 03:00:09.396123 23290 raft_consensus.cc:515] T 36ff33be666a4c82954731691dbcf327 P 1c50de89a5a845668696dcb28c3badd6 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 03:00:09.390473 23325 raft_consensus.cc:493] T a5a05fbcbbcf4916bbc01fd1a5b5e90d P 1c50de89a5a845668696dcb28c3badd6 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 974b802141bd4f13beffef2aa03f88f9)
I20260129 03:00:09.413444 23292 raft_consensus.cc:493] T 18f30ebaa4174fd28c958e72d0cc7e24 P 1c50de89a5a845668696dcb28c3badd6 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260129 03:00:09.469179 20590 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "dc2f4cb752cf4d59b5f6bc689f5d8104" candidate_uuid: "197b880fa39d432a9b472845876bfaf0" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "1c50de89a5a845668696dcb28c3badd6" is_pre_election: true
I20260129 03:00:09.470847 20589 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c891d2bff4ce4568861bbc992e91fbb0" candidate_uuid: "974b802141bd4f13beffef2aa03f88f9" candidate_term: 8 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "1c50de89a5a845668696dcb28c3badd6"
I20260129 03:00:09.471397 20584 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "926a250fa5834e88b96ccecbefbf5086" candidate_uuid: "974b802141bd4f13beffef2aa03f88f9" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "1c50de89a5a845668696dcb28c3badd6" is_pre_election: true
I20260129 03:00:09.286696 23342 raft_consensus.cc:515] T a609ac02fae84725ab743f95f51d33ce P 197b880fa39d432a9b472845876bfaf0 [term 9 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } }
I20260129 03:00:09.473946 20508 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:48544 (request call id 312) took 2428 ms. Trace:
I20260129 03:00:09.387389 23376 raft_consensus.cc:515] T dc2f4cb752cf4d59b5f6bc689f5d8104 P 7a9daf5d42a84615a6383cfec9a10608 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } }
I20260129 03:00:09.490130 20508 rpcz_store.cc:276] 0129 03:00:07.045120 (+ 0us) service_pool.cc:168] Inserting onto call queue
0129 03:00:07.642674 (+597554us) service_pool.cc:225] Handling call
0129 03:00:07.643024 (+ 350us) raft_consensus.cc:1471] Updating replica for 1 ops
0129 03:00:07.643378 (+ 354us) raft_consensus.cc:1534] Early marking committed up to index 0
0129 03:00:07.643401 (+ 23us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0129 03:00:07.670952 (+ 27551us) log.cc:844] Serialized 26 byte log entry
0129 03:00:08.018766 (+347814us) raft_consensus.cc:1645] Marking committed up to 6
0129 03:00:08.018815 (+ 49us) raft_consensus.cc:1697] Filling consensus response to leader.
0129 03:00:08.018871 (+ 56us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0129 03:00:09.473667 (+1454796us) raft_consensus.cc:1685] finished
0129 03:00:09.473698 (+ 31us) raft_consensus.cc:1691] UpdateReplicas() finished
0129 03:00:09.473873 (+ 175us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":1658,"raft.run_cpu_time_us":96,"raft.run_wall_time_us":96,"thread_start_us":234775,"threads_started":1,"wal-append.queue_time_us":486332}
I20260129 03:00:09.464706 23141 raft_consensus.cc:3060] T 78aa39096868415d9afc2ffcb248e198 P 7a9daf5d42a84615a6383cfec9a10608 [term 6 FOLLOWER]: Advancing to term 7
I20260129 03:00:09.482146 20582 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:35752 (request call id 1677) took 1365 ms. Trace:
I20260129 03:00:09.496399 20582 rpcz_store.cc:276] 0129 03:00:08.116343 (+ 0us) service_pool.cc:168] Inserting onto call queue
0129 03:00:09.401739 (+1285396us) service_pool.cc:225] Handling call
0129 03:00:09.482087 (+ 80348us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260129 03:00:09.497370 20582 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4f5a77e94fef4b16a2ba7b067c8b144a" candidate_uuid: "974b802141bd4f13beffef2aa03f88f9" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "1c50de89a5a845668696dcb28c3badd6" is_pre_election: true
I20260129 03:00:09.498255 20582 raft_consensus.cc:2393] T 4f5a77e94fef4b16a2ba7b067c8b144a P 1c50de89a5a845668696dcb28c3badd6 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 974b802141bd4f13beffef2aa03f88f9 in current term 5: Already voted for candidate 7a9daf5d42a84615a6383cfec9a10608 in this term.
I20260129 03:00:09.394032 23026 consensus_queue.cc:1059] T 76f713c3331540d2a5b76164103e9eb1 P 1c50de89a5a845668696dcb28c3badd6 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 }, Status: INVALID_TERM, Last received: 5.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.009s
I20260129 03:00:09.492727 20585 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e7d83d4ccf684e76acb29a0dc844e1b8" candidate_uuid: "974b802141bd4f13beffef2aa03f88f9" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "1c50de89a5a845668696dcb28c3badd6" is_pre_election: true
I20260129 03:00:09.555301 20585 raft_consensus.cc:2468] T e7d83d4ccf684e76acb29a0dc844e1b8 P 1c50de89a5a845668696dcb28c3badd6 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 974b802141bd4f13beffef2aa03f88f9 in term 6.
I20260129 03:00:09.556346 20585 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a1a39dcd328e4268ae41b988c66b4a75" candidate_uuid: "974b802141bd4f13beffef2aa03f88f9" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "1c50de89a5a845668696dcb28c3badd6"
I20260129 03:00:09.556864 20585 raft_consensus.cc:2376] T a1a39dcd328e4268ae41b988c66b4a75 P 1c50de89a5a845668696dcb28c3badd6 [term 5 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 974b802141bd4f13beffef2aa03f88f9 in term 5. Re-sending same reply.
I20260129 03:00:09.557683 20585 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "cedce331e1584ac8b636c9b8a72b2ba0" candidate_uuid: "974b802141bd4f13beffef2aa03f88f9" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "1c50de89a5a845668696dcb28c3badd6" is_pre_election: true
I20260129 03:00:09.558151 20588 raft_consensus.cc:2468] T 926a250fa5834e88b96ccecbefbf5086 P 1c50de89a5a845668696dcb28c3badd6 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 197b880fa39d432a9b472845876bfaf0 in term 8.
I20260129 03:00:09.470558 23304 raft_consensus.cc:3060] T dc2f4cb752cf4d59b5f6bc689f5d8104 P 1c50de89a5a845668696dcb28c3badd6 [term 5 FOLLOWER]: Advancing to term 6
W20260129 03:00:09.547390 20555 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260129 03:00:09.493643 20553 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.557s user 0.025s sys 0.019s
W20260129 03:00:09.561138 20553 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.625s user 0.025s sys 0.019s
W20260129 03:00:09.561519 20553 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.211.1:42027, user_credentials={real_user=slave}} blocked reactor thread for 640237us
I20260129 03:00:09.470497 23107 raft_consensus.cc:493] T 505580c93c694bbb8ea96fdf9b17ab46 P 7a9daf5d42a84615a6383cfec9a10608 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260129 03:00:09.558478 20585 raft_consensus.cc:2393] T cedce331e1584ac8b636c9b8a72b2ba0 P 1c50de89a5a845668696dcb28c3badd6 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 974b802141bd4f13beffef2aa03f88f9 in current term 8: Already voted for candidate 197b880fa39d432a9b472845876bfaf0 in this term.
I20260129 03:00:09.573854 20585 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c2633a6fc34c44579de348d208b0c01e" candidate_uuid: "974b802141bd4f13beffef2aa03f88f9" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "1c50de89a5a845668696dcb28c3badd6" is_pre_election: true
W20260129 03:00:09.498693 20482 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260129 03:00:09.568184 20584 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4b1876f9a70b49edb3aa74105e66d3b3" candidate_uuid: "974b802141bd4f13beffef2aa03f88f9" candidate_term: 6 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "1c50de89a5a845668696dcb28c3badd6"
I20260129 03:00:09.575598 20584 raft_consensus.cc:2376] T 4b1876f9a70b49edb3aa74105e66d3b3 P 1c50de89a5a845668696dcb28c3badd6 [term 6 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 974b802141bd4f13beffef2aa03f88f9 in term 6. Re-sending same reply.
I20260129 03:00:09.569540 20433 raft_consensus.cc:1240] T a609ac02fae84725ab743f95f51d33ce P 197b880fa39d432a9b472845876bfaf0 [term 9 FOLLOWER]: Rejecting Update request from peer 7a9daf5d42a84615a6383cfec9a10608 for earlier term 8. Current term is 9. Ops: []
I20260129 03:00:09.570547 20582 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "75b0b6e82ddc4f92b1fe1d94f9a0b73b" candidate_uuid: "974b802141bd4f13beffef2aa03f88f9" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "1c50de89a5a845668696dcb28c3badd6" is_pre_election: true
I20260129 03:00:09.571504 20589 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fd8311b779af4e1fa1402b2a02a602f2" candidate_uuid: "7a9daf5d42a84615a6383cfec9a10608" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "1c50de89a5a845668696dcb28c3badd6" is_pre_election: true
I20260129 03:00:09.464569 23344 raft_consensus.cc:515] T febffe4a8b394368b69b1d164308fb9a P 197b880fa39d432a9b472845876bfaf0 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } }
I20260129 03:00:09.600646 21100 leader_election.cc:304] T fd8311b779af4e1fa1402b2a02a602f2 P 7a9daf5d42a84615a6383cfec9a10608 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7a9daf5d42a84615a6383cfec9a10608; no voters: 197b880fa39d432a9b472845876bfaf0, 1c50de89a5a845668696dcb28c3badd6
I20260129 03:00:09.469868 22857 raft_consensus.cc:515] T 18f30ebaa4174fd28c958e72d0cc7e24 P 7a9daf5d42a84615a6383cfec9a10608 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } }
I20260129 03:00:09.609771 20589 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1299723a4476420ab02fd8bfd7aa0645" candidate_uuid: "974b802141bd4f13beffef2aa03f88f9" candidate_term: 9 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "1c50de89a5a845668696dcb28c3badd6" is_pre_election: true
I20260129 03:00:09.610514 20589 raft_consensus.cc:1813] T 1299723a4476420ab02fd8bfd7aa0645 P 1c50de89a5a845668696dcb28c3badd6 [term 10 FOLLOWER]: Handling vote request from an unknown peer 974b802141bd4f13beffef2aa03f88f9
I20260129 03:00:09.420089 23380 consensus_queue.cc:260] T cb0b3f259ebc4310927338a8740e87b8 P 7a9daf5d42a84615a6383cfec9a10608 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 7.8, Last appended by leader: 8, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } }
I20260129 03:00:09.611050 20589 raft_consensus.cc:2368] T 1299723a4476420ab02fd8bfd7aa0645 P 1c50de89a5a845668696dcb28c3badd6 [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 974b802141bd4f13beffef2aa03f88f9 for earlier term 9. Current term is 10.
I20260129 03:00:09.613860 20589 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:35752 (request call id 1680) took 1441 ms. Trace:
I20260129 03:00:09.614228 20589 rpcz_store.cc:276] 0129 03:00:08.172303 (+ 0us) service_pool.cc:168] Inserting onto call queue
0129 03:00:09.606694 (+1434391us) service_pool.cc:225] Handling call
0129 03:00:09.613784 (+ 7090us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260129 03:00:09.611073 20585 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "368953c9102643b2a21f99af36b51905" candidate_uuid: "974b802141bd4f13beffef2aa03f88f9" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "1c50de89a5a845668696dcb28c3badd6"
I20260129 03:00:09.615521 20585 raft_consensus.cc:2393] T 368953c9102643b2a21f99af36b51905 P 1c50de89a5a845668696dcb28c3badd6 [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate 974b802141bd4f13beffef2aa03f88f9 in current term 8: Already voted for candidate 1c50de89a5a845668696dcb28c3badd6 in this term.
I20260129 03:00:09.609874 20584 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "368953c9102643b2a21f99af36b51905" candidate_uuid: "974b802141bd4f13beffef2aa03f88f9" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "1c50de89a5a845668696dcb28c3badd6" is_pre_election: true
I20260129 03:00:09.479377 22932 raft_consensus.cc:515] T febffe4a8b394368b69b1d164308fb9a P 7a9daf5d42a84615a6383cfec9a10608 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } }
I20260129 03:00:09.616318 20584 raft_consensus.cc:2393] T 368953c9102643b2a21f99af36b51905 P 1c50de89a5a845668696dcb28c3badd6 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 974b802141bd4f13beffef2aa03f88f9 in current term 8: Already voted for candidate 1c50de89a5a845668696dcb28c3badd6 in this term.
I20260129 03:00:09.489578 23389 raft_consensus.cc:515] T fcddab6bca684adf8ea032289a00f90e P 197b880fa39d432a9b472845876bfaf0 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 03:00:09.500465 23292 raft_consensus.cc:515] T 18f30ebaa4174fd28c958e72d0cc7e24 P 1c50de89a5a845668696dcb28c3badd6 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } }
I20260129 03:00:09.647506 23292 leader_election.cc:290] T 18f30ebaa4174fd28c958e72d0cc7e24 P 1c50de89a5a845668696dcb28c3badd6 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 197b880fa39d432a9b472845876bfaf0 (127.18.211.1:42027), 7a9daf5d42a84615a6383cfec9a10608 (127.18.211.4:40439)
I20260129 03:00:09.500332 23325 raft_consensus.cc:515] T a5a05fbcbbcf4916bbc01fd1a5b5e90d P 1c50de89a5a845668696dcb28c3badd6 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 03:00:09.620429 20438 raft_consensus.cc:1240] T febffe4a8b394368b69b1d164308fb9a P 197b880fa39d432a9b472845876bfaf0 [term 5 FOLLOWER]: Rejecting Update request from peer 974b802141bd4f13beffef2aa03f88f9 for earlier term 4. Current term is 5. Ops: [4.6-4.6]
I20260129 03:00:09.604403 23333 raft_consensus.cc:493] T cb4727386a3e48aca5d71cc6c611c197 P 974b802141bd4f13beffef2aa03f88f9 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260129 03:00:09.629949 23334 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.661s user 0.131s sys 0.179s
I20260129 03:00:09.632052 20582 tablet_service.cc:2066] Received LeaderStepDown RPC: tablet_id: "cedce331e1584ac8b636c9b8a72b2ba0"
dest_uuid: "1c50de89a5a845668696dcb28c3badd6"
mode: GRACEFUL
new_leader_uuid: "197b880fa39d432a9b472845876bfaf0"
from {username='slave'} at 127.0.0.1:54860
W20260129 03:00:09.653079 20407 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.728s user 0.008s sys 0.014s
I20260129 03:00:09.653354 20582 raft_consensus.cc:606] T cedce331e1584ac8b636c9b8a72b2ba0 P 1c50de89a5a845668696dcb28c3badd6 [term 8 FOLLOWER]: Received request to transfer leadership to TS 197b880fa39d432a9b472845876bfaf0
W20260129 03:00:09.653685 20407 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.729s user 0.008s sys 0.014s
I20260129 03:00:09.653937 20582 raft_consensus.cc:614] T cedce331e1584ac8b636c9b8a72b2ba0 P 1c50de89a5a845668696dcb28c3badd6 [term 8 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260129 03:00:09.633618 23290 leader_election.cc:290] T 36ff33be666a4c82954731691dbcf327 P 1c50de89a5a845668696dcb28c3badd6 [CANDIDATE]: Term 4 election: Requested vote from peers 197b880fa39d432a9b472845876bfaf0 (127.18.211.1:42027), 974b802141bd4f13beffef2aa03f88f9 (127.18.211.2:34517)
I20260129 03:00:09.634685 23368 leader_election.cc:290] T c891d2bff4ce4568861bbc992e91fbb0 P 197b880fa39d432a9b472845876bfaf0 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 974b802141bd4f13beffef2aa03f88f9 (127.18.211.2:34517), 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3:37735)
W20260129 03:00:09.638495 23364 log.cc:927] Time spent T febffe4a8b394368b69b1d164308fb9a P 7a9daf5d42a84615a6383cfec9a10608: Append to log took a long time: real 1.369s user 0.013s sys 0.015s
I20260129 03:00:09.639189 23280 leader_election.cc:290] T 96609ac68bdf4993a12e1c23355ca013 P 1c50de89a5a845668696dcb28c3badd6 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 197b880fa39d432a9b472845876bfaf0 (127.18.211.1:42027), 974b802141bd4f13beffef2aa03f88f9 (127.18.211.2:34517), 7a9daf5d42a84615a6383cfec9a10608 (127.18.211.4:40439)
I20260129 03:00:09.641726 20584 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "76f713c3331540d2a5b76164103e9eb1" candidate_uuid: "974b802141bd4f13beffef2aa03f88f9" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "1c50de89a5a845668696dcb28c3badd6" is_pre_election: true
W20260129 03:00:09.615931 23392 log.cc:927] Time spent T b189f08e16a74a11980cc55d99359a0a P 974b802141bd4f13beffef2aa03f88f9: Append to log took a long time: real 0.128s user 0.000s sys 0.000s
I20260129 03:00:09.626055 20585 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "18f30ebaa4174fd28c958e72d0cc7e24" candidate_uuid: "7a9daf5d42a84615a6383cfec9a10608" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "1c50de89a5a845668696dcb28c3badd6" is_pre_election: true
I20260129 03:00:09.687770 20585 raft_consensus.cc:2468] T 18f30ebaa4174fd28c958e72d0cc7e24 P 1c50de89a5a845668696dcb28c3badd6 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7a9daf5d42a84615a6383cfec9a10608 in term 7.
I20260129 03:00:09.689373 21100 leader_election.cc:304] T 18f30ebaa4174fd28c958e72d0cc7e24 P 7a9daf5d42a84615a6383cfec9a10608 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1c50de89a5a845668696dcb28c3badd6, 7a9daf5d42a84615a6383cfec9a10608; no voters:
W20260129 03:00:09.653175 23334 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.865s user 0.131s sys 0.179s
W20260129 03:00:09.650068 20406 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.594s user 0.009s sys 0.013s
W20260129 03:00:09.658320 20405 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.732s user 0.002s sys 0.023s
I20260129 03:00:09.615829 22857 leader_election.cc:290] T 18f30ebaa4174fd28c958e72d0cc7e24 P 7a9daf5d42a84615a6383cfec9a10608 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 197b880fa39d432a9b472845876bfaf0 (127.18.211.1:42027), 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3:37735)
W20260129 03:00:09.655668 20404 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.648s user 0.000s sys 0.023s
I20260129 03:00:09.619551 23205 leader_election.cc:290] T 4e420f90884f4796997b5ea41e49e411 P 974b802141bd4f13beffef2aa03f88f9 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 197b880fa39d432a9b472845876bfaf0 (127.18.211.1:42027), 7a9daf5d42a84615a6383cfec9a10608 (127.18.211.4:40439)
W20260129 03:00:09.699816 20405 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.774s user 0.002s sys 0.023s
W20260129 03:00:09.699811 20406 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.732s user 0.009s sys 0.013s
W20260129 03:00:09.729103 20406 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260129 03:00:09.724398 20407 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260129 03:00:09.724412 20404 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.717s user 0.000s sys 0.023s
W20260129 03:00:09.708432 20480 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.770s user 0.014s sys 0.040s
W20260129 03:00:09.730217 20480 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.792s user 0.014s sys 0.040s
I20260129 03:00:09.735262 20513 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c891d2bff4ce4568861bbc992e91fbb0" candidate_uuid: "197b880fa39d432a9b472845876bfaf0" candidate_term: 9 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "974b802141bd4f13beffef2aa03f88f9" is_pre_election: true
I20260129 03:00:09.736083 20513 raft_consensus.cc:2468] T c891d2bff4ce4568861bbc992e91fbb0 P 974b802141bd4f13beffef2aa03f88f9 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 197b880fa39d432a9b472845876bfaf0 in term 8.
I20260129 03:00:09.740752 20584 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35754 (request call id 1804) took 1797 ms. Trace:
I20260129 03:00:09.741572 20584 rpcz_store.cc:276] 0129 03:00:07.943472 (+ 0us) service_pool.cc:168] Inserting onto call queue
0129 03:00:09.675134 (+1731662us) service_pool.cc:225] Handling call
0129 03:00:09.734365 (+ 59231us) raft_consensus.cc:1471] Updating replica for 0 ops
0129 03:00:09.740287 (+ 5922us) raft_consensus.cc:1534] Early marking committed up to index 0
0129 03:00:09.740332 (+ 45us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0129 03:00:09.740391 (+ 59us) raft_consensus.cc:1645] Marking committed up to 3
0129 03:00:09.740436 (+ 45us) raft_consensus.cc:1697] Filling consensus response to leader.
0129 03:00:09.740517 (+ 81us) raft_consensus.cc:1691] UpdateReplicas() finished
0129 03:00:09.740684 (+ 167us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":5514}
I20260129 03:00:09.743628 20508 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "368953c9102643b2a21f99af36b51905" candidate_uuid: "197b880fa39d432a9b472845876bfaf0" candidate_term: 9 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "974b802141bd4f13beffef2aa03f88f9" is_pre_election: true
I20260129 03:00:09.728168 23368 raft_consensus.cc:493] T 36ff33be666a4c82954731691dbcf327 P 197b880fa39d432a9b472845876bfaf0 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 974b802141bd4f13beffef2aa03f88f9)
I20260129 03:00:09.752202 23283 consensus_queue.cc:1059] T 76f713c3331540d2a5b76164103e9eb1 P 1c50de89a5a845668696dcb28c3badd6 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 }, Status: INVALID_TERM, Last received: 5.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260129 03:00:09.752529 23368 raft_consensus.cc:515] T 36ff33be666a4c82954731691dbcf327 P 197b880fa39d432a9b472845876bfaf0 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 03:00:09.743995 23292 raft_consensus.cc:493] T c891d2bff4ce4568861bbc992e91fbb0 P 1c50de89a5a845668696dcb28c3badd6 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260129 03:00:09.753611 23292 raft_consensus.cc:515] T c891d2bff4ce4568861bbc992e91fbb0 P 1c50de89a5a845668696dcb28c3badd6 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
W20260129 03:00:09.754208 23318 log.cc:927] Time spent T 0f20af7fbe5d4db89a6f17948d98912c P 197b880fa39d432a9b472845876bfaf0: Append to log took a long time: real 2.878s user 0.018s sys 0.028s
I20260129 03:00:09.733397 23091 raft_consensus.cc:493] T a5a05fbcbbcf4916bbc01fd1a5b5e90d P 197b880fa39d432a9b472845876bfaf0 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 974b802141bd4f13beffef2aa03f88f9)
I20260129 03:00:09.755668 23091 raft_consensus.cc:515] T a5a05fbcbbcf4916bbc01fd1a5b5e90d P 197b880fa39d432a9b472845876bfaf0 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 03:00:09.756649 21134 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:47844 (request call id 377) took 2086 ms. Trace:
I20260129 03:00:09.756938 23046 leader_election.cc:290] T 7d9494f5584d4d5db8b9adf1c8fc888b P 974b802141bd4f13beffef2aa03f88f9 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 197b880fa39d432a9b472845876bfaf0 (127.18.211.1:42027), 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3:37735)
W20260129 03:00:09.758190 20480 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.211.3:37735, user_credentials={real_user=slave}} blocked reactor thread for 1.3546e+06us
I20260129 03:00:09.758415 23369 leader_election.cc:290] T e7d83d4ccf684e76acb29a0dc844e1b8 P 197b880fa39d432a9b472845876bfaf0 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 974b802141bd4f13beffef2aa03f88f9 (127.18.211.2:34517), 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3:37735)
I20260129 03:00:09.732810 22857 raft_consensus.cc:2749] T fd8311b779af4e1fa1402b2a02a602f2 P 7a9daf5d42a84615a6383cfec9a10608 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260129 03:00:09.760004 20436 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "18f30ebaa4174fd28c958e72d0cc7e24" candidate_uuid: "7a9daf5d42a84615a6383cfec9a10608" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "197b880fa39d432a9b472845876bfaf0" is_pre_election: true
I20260129 03:00:09.760620 20436 raft_consensus.cc:2468] T 18f30ebaa4174fd28c958e72d0cc7e24 P 197b880fa39d432a9b472845876bfaf0 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7a9daf5d42a84615a6383cfec9a10608 in term 7.
I20260129 03:00:09.604674 23107 raft_consensus.cc:515] T 505580c93c694bbb8ea96fdf9b17ab46 P 7a9daf5d42a84615a6383cfec9a10608 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } }
I20260129 03:00:09.775277 20434 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1ce4ddf5b10e4ae8b193b13b2e5936ee" candidate_uuid: "1c50de89a5a845668696dcb28c3badd6" candidate_term: 8 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "197b880fa39d432a9b472845876bfaf0" is_pre_election: true
I20260129 03:00:09.776865 23107 leader_election.cc:290] T 505580c93c694bbb8ea96fdf9b17ab46 P 7a9daf5d42a84615a6383cfec9a10608 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 974b802141bd4f13beffef2aa03f88f9 (127.18.211.2:34517), 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3:37735)
I20260129 03:00:09.777061 20434 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e7d83d4ccf684e76acb29a0dc844e1b8" candidate_uuid: "1c50de89a5a845668696dcb28c3badd6" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "197b880fa39d432a9b472845876bfaf0" is_pre_election: true
I20260129 03:00:09.778133 20434 raft_consensus.cc:2410] T e7d83d4ccf684e76acb29a0dc844e1b8 P 197b880fa39d432a9b472845876bfaf0 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1c50de89a5a845668696dcb28c3badd6 for term 7 because replica has last-logged OpId of term: 5 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 4 index: 3.
W20260129 03:00:09.778378 23396 log.cc:927] Time spent T 0f20af7fbe5d4db89a6f17948d98912c P 974b802141bd4f13beffef2aa03f88f9: Append to log took a long time: real 0.115s user 0.001s sys 0.000s
I20260129 03:00:09.757055 21134 rpcz_store.cc:276] 0129 03:00:07.670700 (+ 0us) service_pool.cc:168] Inserting onto call queue
0129 03:00:07.671071 (+ 371us) service_pool.cc:225] Handling call
0129 03:00:07.739026 (+ 67955us) raft_consensus.cc:1471] Updating replica for 2 ops
0129 03:00:07.739453 (+ 427us) raft_consensus.cc:1534] Early marking committed up to index 0
0129 03:00:07.739492 (+ 39us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0129 03:00:07.781345 (+ 41853us) log.cc:844] Serialized 52 byte log entry
0129 03:00:08.071048 (+289703us) raft_consensus.cc:1645] Marking committed up to 5
0129 03:00:08.240873 (+169825us) log.cc:844] Serialized 14 byte log entry
0129 03:00:08.848967 (+608094us) raft_consensus.cc:1697] Filling consensus response to leader.
0129 03:00:08.849078 (+ 111us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0129 03:00:09.637929 (+788851us) spinlock_profiling.cc:243] Waited 2.57 ms on lock 0x7b440002304c. stack: 00007f8fe044e373 00007f8fe044e270 00007f8fdfda0574 00007f8fe9a5c6b4 00007f8fe9a5c649 00007f8fe02c08b1 00007f8fe038f005 00007f8fe038fa22 00007f8fe30d957b 00007f8fe30d4e4f 00007f8fe30c8c0e 00007f8fe30c7f56 00007f8fe30c7b78 00007f8fe30d15c1 00007f8fe30d1579 00007f8fe30d1509
0129 03:00:09.756299 (+118370us) raft_consensus.cc:1685] finished
0129 03:00:09.756415 (+ 116us) raft_consensus.cc:1691] UpdateReplicas() finished
0129 03:00:09.756593 (+ 178us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":176914,"prepare.run_cpu_time_us":42,"prepare.run_wall_time_us":4330,"raft.queue_time_us":315853,"raft.run_cpu_time_us":277,"raft.run_wall_time_us":6749,"spinlock_wait_cycles":5913984,"thread_start_us":775947,"threads_started":3,"wal-append.queue_time_us":398469}
I20260129 03:00:09.779109 20432 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "96609ac68bdf4993a12e1c23355ca013" candidate_uuid: "1c50de89a5a845668696dcb28c3badd6" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "197b880fa39d432a9b472845876bfaf0" is_pre_election: true
I20260129 03:00:09.779716 20515 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57600 (request call id 1924) took 1569 ms. Trace:
I20260129 03:00:09.780517 20432 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "36ff33be666a4c82954731691dbcf327" candidate_uuid: "1c50de89a5a845668696dcb28c3badd6" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "197b880fa39d432a9b472845876bfaf0"
I20260129 03:00:09.781143 20432 raft_consensus.cc:3060] T 36ff33be666a4c82954731691dbcf327 P 197b880fa39d432a9b472845876bfaf0 [term 3 FOLLOWER]: Advancing to term 4
I20260129 03:00:09.781769 23376 leader_election.cc:290] T dc2f4cb752cf4d59b5f6bc689f5d8104 P 7a9daf5d42a84615a6383cfec9a10608 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 197b880fa39d432a9b472845876bfaf0 (127.18.211.1:42027), 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3:37735)
I20260129 03:00:09.783435 20510 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57600 (request call id 1925) took 1547 ms. Trace:
I20260129 03:00:09.780089 20515 rpcz_store.cc:276] 0129 03:00:08.210712 (+ 0us) service_pool.cc:168] Inserting onto call queue
0129 03:00:08.217245 (+ 6533us) service_pool.cc:225] Handling call
0129 03:00:08.217583 (+ 338us) raft_consensus.cc:1471] Updating replica for 2 ops
0129 03:00:08.234575 (+ 16992us) raft_consensus.cc:1534] Early marking committed up to index 0
0129 03:00:08.234617 (+ 42us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0129 03:00:08.235263 (+ 646us) log.cc:844] Serialized 52 byte log entry
0129 03:00:09.442388 (+1207125us) raft_consensus.cc:1645] Marking committed up to 3
0129 03:00:09.442428 (+ 40us) raft_consensus.cc:1697] Filling consensus response to leader.
0129 03:00:09.442493 (+ 65us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0129 03:00:09.779507 (+337014us) raft_consensus.cc:1685] finished
0129 03:00:09.779528 (+ 21us) raft_consensus.cc:1691] UpdateReplicas() finished
0129 03:00:09.779655 (+ 127us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":9844,"raft.run_cpu_time_us":155,"raft.run_wall_time_us":4151,"thread_start_us":1121110,"threads_started":1,"wal-append.queue_time_us":1335554}
I20260129 03:00:09.783783 20510 rpcz_store.cc:276] 0129 03:00:08.236042 (+ 0us) service_pool.cc:168] Inserting onto call queue
0129 03:00:08.236224 (+ 182us) service_pool.cc:225] Handling call
0129 03:00:08.259553 (+ 23329us) raft_consensus.cc:1471] Updating replica for 2 ops
0129 03:00:08.268638 (+ 9085us) raft_consensus.cc:1534] Early marking committed up to index 0
0129 03:00:08.268670 (+ 32us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0129 03:00:08.323472 (+ 54802us) log.cc:844] Serialized 52 byte log entry
0129 03:00:09.285939 (+962467us) raft_consensus.cc:1645] Marking committed up to 3
0129 03:00:09.285979 (+ 40us) raft_consensus.cc:1697] Filling consensus response to leader.
0129 03:00:09.286051 (+ 72us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0129 03:00:09.783244 (+497193us) raft_consensus.cc:1685] finished
0129 03:00:09.783265 (+ 21us) raft_consensus.cc:1691] UpdateReplicas() finished
0129 03:00:09.783383 (+ 118us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":127,"raft.run_cpu_time_us":118,"raft.run_wall_time_us":17065,"thread_start_us":875346,"threads_started":1,"wal-append.queue_time_us":1117018}
I20260129 03:00:09.726928 20582 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.LeaderStepDown from 127.0.0.1:54860 (request call id 15) took 1091 ms. Trace:
I20260129 03:00:09.786108 20582 rpcz_store.cc:276] 0129 03:00:08.563382 (+ 0us) service_pool.cc:168] Inserting onto call queue
0129 03:00:09.607951 (+1044569us) service_pool.cc:225] Handling call
0129 03:00:09.654368 (+ 46417us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260129 03:00:09.773273 20436 raft_consensus.cc:1275] T cb4727386a3e48aca5d71cc6c611c197 P 197b880fa39d432a9b472845876bfaf0 [term 5 FOLLOWER]: Refusing update from remote peer 1c50de89a5a845668696dcb28c3badd6: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 5 index: 2. (term mismatch)
W20260129 03:00:09.773501 23370 log.cc:927] Time spent T 7a9a69cd82724b1faed1d10aa091fc4a P 197b880fa39d432a9b472845876bfaf0: Append to log took a long time: real 1.632s user 0.012s sys 0.051s
I20260129 03:00:09.774288 20437 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "368953c9102643b2a21f99af36b51905" candidate_uuid: "1c50de89a5a845668696dcb28c3badd6" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "197b880fa39d432a9b472845876bfaf0"
I20260129 03:00:09.786787 20436 pending_rounds.cc:85] T cb4727386a3e48aca5d71cc6c611c197 P 197b880fa39d432a9b472845876bfaf0: Aborting all ops after (but not including) 1
I20260129 03:00:09.787078 20436 pending_rounds.cc:107] T cb4727386a3e48aca5d71cc6c611c197 P 197b880fa39d432a9b472845876bfaf0: Aborting uncommitted NO_OP operation due to leader change: 3.2
I20260129 03:00:09.787286 20436 raft_consensus.cc:2889] T cb4727386a3e48aca5d71cc6c611c197 P 197b880fa39d432a9b472845876bfaf0 [term 5 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260129 03:00:09.795691 20434 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "18f30ebaa4174fd28c958e72d0cc7e24" candidate_uuid: "1c50de89a5a845668696dcb28c3badd6" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "197b880fa39d432a9b472845876bfaf0" is_pre_election: true
I20260129 03:00:09.796561 20434 raft_consensus.cc:2468] T 18f30ebaa4174fd28c958e72d0cc7e24 P 197b880fa39d432a9b472845876bfaf0 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1c50de89a5a845668696dcb28c3badd6 in term 7.
I20260129 03:00:09.803776 19276 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 2
I20260129 03:00:09.804265 23344 leader_election.cc:290] T febffe4a8b394368b69b1d164308fb9a P 197b880fa39d432a9b472845876bfaf0 [CANDIDATE]: Term 5 election: Requested vote from peers 974b802141bd4f13beffef2aa03f88f9 (127.18.211.2:34517), 7a9daf5d42a84615a6383cfec9a10608 (127.18.211.4:40439)
I20260129 03:00:09.805657 19276 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20260129 03:00:09.806121 23389 leader_election.cc:290] T fcddab6bca684adf8ea032289a00f90e P 197b880fa39d432a9b472845876bfaf0 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 974b802141bd4f13beffef2aa03f88f9 (127.18.211.2:34517), 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3:37735)
I20260129 03:00:09.806481 19276 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
W20260129 03:00:09.807294 23394 log.cc:927] Time spent T 00000000000000000000000000000000 P a91e6696d0a44ee19bcf2489fd709c38: Append to log took a long time: real 0.233s user 0.001s sys 0.001s
I20260129 03:00:09.803283 20582 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a0fcc58d6fb6473cafbd9d34d5195269" candidate_uuid: "974b802141bd4f13beffef2aa03f88f9" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "1c50de89a5a845668696dcb28c3badd6" is_pre_election: true
I20260129 03:00:09.808666 20582 raft_consensus.cc:2393] T a0fcc58d6fb6473cafbd9d34d5195269 P 1c50de89a5a845668696dcb28c3badd6 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 974b802141bd4f13beffef2aa03f88f9 in current term 5: Already voted for candidate 197b880fa39d432a9b472845876bfaf0 in this term.
I20260129 03:00:09.657133 23333 raft_consensus.cc:515] T cb4727386a3e48aca5d71cc6c611c197 P 974b802141bd4f13beffef2aa03f88f9 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
W20260129 03:00:09.809301 20407 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.211.2:34517, user_credentials={real_user=slave}} blocked reactor thread for 48548.3us
I20260129 03:00:09.841599 20437 raft_consensus.cc:2393] T 368953c9102643b2a21f99af36b51905 P 197b880fa39d432a9b472845876bfaf0 [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate 1c50de89a5a845668696dcb28c3badd6 in current term 8: Already voted for candidate 974b802141bd4f13beffef2aa03f88f9 in this term.
I20260129 03:00:09.755586 23292 leader_election.cc:290] T c891d2bff4ce4568861bbc992e91fbb0 P 1c50de89a5a845668696dcb28c3badd6 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 197b880fa39d432a9b472845876bfaf0 (127.18.211.1:42027), 974b802141bd4f13beffef2aa03f88f9 (127.18.211.2:34517)
I20260129 03:00:09.844379 20584 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "8055232cf0004d40a85f3bf983fe00a9" candidate_uuid: "974b802141bd4f13beffef2aa03f88f9" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "1c50de89a5a845668696dcb28c3badd6"
W20260129 03:00:09.844121 20479 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.926s user 0.016s sys 0.064s
I20260129 03:00:09.845508 20584 raft_consensus.cc:2376] T 8055232cf0004d40a85f3bf983fe00a9 P 1c50de89a5a845668696dcb28c3badd6 [term 5 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 974b802141bd4f13beffef2aa03f88f9 in term 5. Re-sending same reply.
W20260129 03:00:09.845901 20479 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.928s user 0.016s sys 0.064s
I20260129 03:00:09.844427 20434 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0f20af7fbe5d4db89a6f17948d98912c" candidate_uuid: "1c50de89a5a845668696dcb28c3badd6" candidate_term: 8 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "197b880fa39d432a9b472845876bfaf0" is_pre_election: true
I20260129 03:00:09.848135 23304 raft_consensus.cc:515] T dc2f4cb752cf4d59b5f6bc689f5d8104 P 1c50de89a5a845668696dcb28c3badd6 [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } }
I20260129 03:00:09.849785 20590 raft_consensus.cc:2368] T dc2f4cb752cf4d59b5f6bc689f5d8104 P 1c50de89a5a845668696dcb28c3badd6 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 197b880fa39d432a9b472845876bfaf0 for earlier term 5. Current term is 6.
I20260129 03:00:09.733928 22694 leader_election.cc:290] T fcddab6bca684adf8ea032289a00f90e P 974b802141bd4f13beffef2aa03f88f9 [CANDIDATE]: Term 10 election: Requested vote from peers 197b880fa39d432a9b472845876bfaf0 (127.18.211.1:42027), 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3:37735)
I20260129 03:00:09.863274 20553 leader_election.cc:304] T 18f30ebaa4174fd28c958e72d0cc7e24 P 1c50de89a5a845668696dcb28c3badd6 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 197b880fa39d432a9b472845876bfaf0, 1c50de89a5a845668696dcb28c3badd6; no voters:
I20260129 03:00:09.866117 20434 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a5a05fbcbbcf4916bbc01fd1a5b5e90d" candidate_uuid: "1c50de89a5a845668696dcb28c3badd6" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "197b880fa39d432a9b472845876bfaf0" is_pre_election: true
I20260129 03:00:09.873881 20582 raft_consensus.cc:1275] T 75be4bd6bd45474795fd472eff2953ac P 1c50de89a5a845668696dcb28c3badd6 [term 10 FOLLOWER]: Refusing update from remote peer 197b880fa39d432a9b472845876bfaf0: Log matching property violated. Preceding OpId in replica: term: 9 index: 6. Preceding OpId from leader: term: 10 index: 7. (index mismatch)
I20260129 03:00:09.874606 20582 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35754 (request call id 1807) took 1794 ms. Trace:
I20260129 03:00:09.874994 20582 rpcz_store.cc:276] 0129 03:00:08.080660 (+ 0us) service_pool.cc:168] Inserting onto call queue
0129 03:00:09.809219 (+1728559us) service_pool.cc:225] Handling call
0129 03:00:09.873577 (+ 64358us) raft_consensus.cc:1471] Updating replica for 0 ops
0129 03:00:09.874379 (+ 802us) raft_consensus.cc:1697] Filling consensus response to leader.
0129 03:00:09.874556 (+ 177us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260129 03:00:09.876757 20432 raft_consensus.cc:2468] T 36ff33be666a4c82954731691dbcf327 P 197b880fa39d432a9b472845876bfaf0 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1c50de89a5a845668696dcb28c3badd6 in term 4.
I20260129 03:00:09.876886 20437 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c891d2bff4ce4568861bbc992e91fbb0" candidate_uuid: "1c50de89a5a845668696dcb28c3badd6" candidate_term: 9 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "197b880fa39d432a9b472845876bfaf0" is_pre_election: true
I20260129 03:00:09.746143 23298 leader_election.cc:290] T 0f20af7fbe5d4db89a6f17948d98912c P 1c50de89a5a845668696dcb28c3badd6 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 197b880fa39d432a9b472845876bfaf0 (127.18.211.1:42027), 974b802141bd4f13beffef2aa03f88f9 (127.18.211.2:34517)
I20260129 03:00:09.880419 20553 leader_election.cc:304] T 36ff33be666a4c82954731691dbcf327 P 1c50de89a5a845668696dcb28c3badd6 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 197b880fa39d432a9b472845876bfaf0, 1c50de89a5a845668696dcb28c3badd6; no voters:
I20260129 03:00:09.866920 20434 raft_consensus.cc:2468] T a5a05fbcbbcf4916bbc01fd1a5b5e90d P 197b880fa39d432a9b472845876bfaf0 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1c50de89a5a845668696dcb28c3badd6 in term 7.
I20260129 03:00:09.882405 20553 leader_election.cc:304] T a5a05fbcbbcf4916bbc01fd1a5b5e90d P 1c50de89a5a845668696dcb28c3badd6 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 197b880fa39d432a9b472845876bfaf0, 1c50de89a5a845668696dcb28c3badd6; no voters:
I20260129 03:00:09.883572 23325 leader_election.cc:290] T a5a05fbcbbcf4916bbc01fd1a5b5e90d P 1c50de89a5a845668696dcb28c3badd6 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 197b880fa39d432a9b472845876bfaf0 (127.18.211.1:42027), 974b802141bd4f13beffef2aa03f88f9 (127.18.211.2:34517)
I20260129 03:00:09.883587 22891 raft_consensus.cc:2804] T 36ff33be666a4c82954731691dbcf327 P 1c50de89a5a845668696dcb28c3badd6 [term 4 FOLLOWER]: Leader election won for term 4
I20260129 03:00:09.885215 20438 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55672 (request call id 1728) took 1404 ms. Trace:
W20260129 03:00:09.844264 23397 log.cc:927] Time spent T a8479d1fe0bd41e5946186e3a6ca8c7a P 974b802141bd4f13beffef2aa03f88f9: Append to log took a long time: real 0.201s user 0.000s sys 0.001s
I20260129 03:00:09.885743 20438 rpcz_store.cc:276] 0129 03:00:08.480980 (+ 0us) service_pool.cc:168] Inserting onto call queue
0129 03:00:09.245752 (+764772us) service_pool.cc:225] Handling call
0129 03:00:09.267109 (+ 21357us) raft_consensus.cc:1471] Updating replica for 1 ops
0129 03:00:09.651380 (+384271us) raft_consensus.cc:1697] Filling consensus response to leader.
0129 03:00:09.884983 (+233603us) spinlock_profiling.cc:243] Waited 331 ms on lock 0x7b54000a6ed4. stack: 00007f8fe044e373 00007f8fe044e270 00007f8fdfda0574 00007f8fe9a5c6b4 00007f8fe9a5c649 00007f8fe34975a1 00007f8fe352ea81 00007f8fe352d4be 00007f8fe84e1f49 00007f8fe1a6ed34 00007f8fe1a6ecc2 00007f8fe1a6ebf1 00007f8fe1a6eb6c 00007f8fe1a6de22 00007f8fe186d34c 00007f8fe186c786
0129 03:00:09.885160 (+ 177us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":760228736}
I20260129 03:00:09.877131 20479 leader_election.cc:304] T 2ac659d732314e1288849f08fe5e074e P 974b802141bd4f13beffef2aa03f88f9 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 974b802141bd4f13beffef2aa03f88f9; no voters: 197b880fa39d432a9b472845876bfaf0, 7a9daf5d42a84615a6383cfec9a10608
I20260129 03:00:09.886471 20436 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0340229d521b432580ae926f6062e3f7" candidate_uuid: "1c50de89a5a845668696dcb28c3badd6" candidate_term: 8 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "197b880fa39d432a9b472845876bfaf0" is_pre_election: true
I20260129 03:00:09.879739 20407 leader_election.cc:304] T 8055232cf0004d40a85f3bf983fe00a9 P 197b880fa39d432a9b472845876bfaf0 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 197b880fa39d432a9b472845876bfaf0, 974b802141bd4f13beffef2aa03f88f9; no voters:
I20260129 03:00:09.890812 20584 raft_consensus.cc:1240] T 17d6431cc77a491e8464d68db4081a51 P 1c50de89a5a845668696dcb28c3badd6 [term 5 FOLLOWER]: Rejecting Update request from peer 974b802141bd4f13beffef2aa03f88f9 for earlier term 4. Current term is 5. Ops: []
I20260129 03:00:09.891577 20584 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35752 (request call id 1678) took 1759 ms. Trace:
I20260129 03:00:09.896474 20584 rpcz_store.cc:276] 0129 03:00:08.132545 (+ 0us) service_pool.cc:168] Inserting onto call queue
0129 03:00:09.846488 (+1713943us) service_pool.cc:225] Handling call
0129 03:00:09.890512 (+ 44024us) raft_consensus.cc:1471] Updating replica for 0 ops
0129 03:00:09.891296 (+ 784us) raft_consensus.cc:1697] Filling consensus response to leader.
0129 03:00:09.891513 (+ 217us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260129 03:00:09.897848 20582 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4408006eb5174de1896b037cf3f18457" candidate_uuid: "974b802141bd4f13beffef2aa03f88f9" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "1c50de89a5a845668696dcb28c3badd6" is_pre_election: true
I20260129 03:00:09.885016 22891 raft_consensus.cc:697] T 36ff33be666a4c82954731691dbcf327 P 1c50de89a5a845668696dcb28c3badd6 [term 4 LEADER]: Becoming Leader. State: Replica: 1c50de89a5a845668696dcb28c3badd6, State: Running, Role: LEADER
I20260129 03:00:09.878613 23342 leader_election.cc:290] T a609ac02fae84725ab743f95f51d33ce P 197b880fa39d432a9b472845876bfaf0 [CANDIDATE]: Term 9 election: Requested vote from peers 974b802141bd4f13beffef2aa03f88f9 (127.18.211.2:34517), 7a9daf5d42a84615a6383cfec9a10608 (127.18.211.4:40439)
W20260129 03:00:09.921551 20556 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.532s user 0.026s sys 0.039s
I20260129 03:00:09.878253 23398 raft_consensus.cc:2804] T 18f30ebaa4174fd28c958e72d0cc7e24 P 1c50de89a5a845668696dcb28c3badd6 [term 7 FOLLOWER]: Leader pre-election won for term 8
W20260129 03:00:09.922202 20556 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.533s user 0.026s sys 0.039s
W20260129 03:00:09.924331 20556 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.211.2:34517, user_credentials={real_user=slave}} blocked reactor thread for 662227us
W20260129 03:00:09.904265 20581 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.947s user 0.010s sys 0.004s
W20260129 03:00:09.926571 20581 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.969s user 0.010s sys 0.004s
I20260129 03:00:09.906010 20588 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35752 (request call id 1682) took 1657 ms. Trace:
I20260129 03:00:09.861483 23280 raft_consensus.cc:493] T 3906b080a5b34a74b8b64128635e8997 P 1c50de89a5a845668696dcb28c3badd6 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 974b802141bd4f13beffef2aa03f88f9)
I20260129 03:00:09.909833 20479 leader_election.cc:304] T 0678b8b468bd43cdbc3e485900844821 P 974b802141bd4f13beffef2aa03f88f9 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 197b880fa39d432a9b472845876bfaf0, 974b802141bd4f13beffef2aa03f88f9; no voters:
I20260129 03:00:09.845144 23346 consensus_queue.cc:1059] T c934d31efd3f4e5195605ddc354d14e1 P 197b880fa39d432a9b472845876bfaf0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 }, Status: INVALID_TERM, Last received: 5.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.054s
I20260129 03:00:09.921417 20407 leader_election.cc:304] T c891d2bff4ce4568861bbc992e91fbb0 P 197b880fa39d432a9b472845876bfaf0 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 197b880fa39d432a9b472845876bfaf0, 974b802141bd4f13beffef2aa03f88f9; no voters:
I20260129 03:00:09.925765 20582 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b189f08e16a74a11980cc55d99359a0a" candidate_uuid: "974b802141bd4f13beffef2aa03f88f9" candidate_term: 10 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "1c50de89a5a845668696dcb28c3badd6" is_pre_election: true
I20260129 03:00:09.957414 20582 raft_consensus.cc:2410] T b189f08e16a74a11980cc55d99359a0a P 1c50de89a5a845668696dcb28c3badd6 [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 974b802141bd4f13beffef2aa03f88f9 for term 10 because replica has last-logged OpId of term: 8 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 4 index: 3.
I20260129 03:00:09.927560 20588 rpcz_store.cc:276] 0129 03:00:08.249163 (+ 0us) service_pool.cc:168] Inserting onto call queue
0129 03:00:09.905061 (+1655898us) service_pool.cc:225] Handling call
0129 03:00:09.905400 (+ 339us) raft_consensus.cc:1471] Updating replica for 0 ops
0129 03:00:09.905623 (+ 223us) raft_consensus.cc:1534] Early marking committed up to index 0
0129 03:00:09.905659 (+ 36us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0129 03:00:09.905726 (+ 67us) raft_consensus.cc:1645] Marking committed up to 3
0129 03:00:09.905769 (+ 43us) raft_consensus.cc:1697] Filling consensus response to leader.
0129 03:00:09.905839 (+ 70us) raft_consensus.cc:1691] UpdateReplicas() finished
0129 03:00:09.905950 (+ 111us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260129 03:00:09.894059 23104 leader_election.cc:290] T 0340229d521b432580ae926f6062e3f7 P 1c50de89a5a845668696dcb28c3badd6 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 197b880fa39d432a9b472845876bfaf0 (127.18.211.1:42027), 974b802141bd4f13beffef2aa03f88f9 (127.18.211.2:34517)
I20260129 03:00:09.896339 23334 raft_consensus.cc:2749] T 2ac659d732314e1288849f08fe5e074e P 974b802141bd4f13beffef2aa03f88f9 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260129 03:00:09.897354 23298 raft_consensus.cc:2804] T a5a05fbcbbcf4916bbc01fd1a5b5e90d P 1c50de89a5a845668696dcb28c3badd6 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260129 03:00:09.896215 23087 raft_consensus.cc:493] T 42d27a28d89b409e8807ec0efd97af69 P 197b880fa39d432a9b472845876bfaf0 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 7a9daf5d42a84615a6383cfec9a10608)
W20260129 03:00:09.911648 23035 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.618s user 0.008s sys 0.008s
W20260129 03:00:09.985090 20435 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.535s user 0.004s sys 0.006s
I20260129 03:00:09.889061 23292 consensus_queue.cc:1048] T cb4727386a3e48aca5d71cc6c611c197 P 1c50de89a5a845668696dcb28c3badd6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.038s
I20260129 03:00:10.009807 20582 raft_consensus.cc:1240] T 0f20af7fbe5d4db89a6f17948d98912c P 1c50de89a5a845668696dcb28c3badd6 [term 7 FOLLOWER]: Rejecting Update request from peer 974b802141bd4f13beffef2aa03f88f9 for earlier term 6. Current term is 7. Ops: [6.4-6.4]
W20260129 03:00:10.009958 20479 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.211.1:42027, user_credentials={real_user=slave}} blocked reactor thread for 100213us
I20260129 03:00:10.011736 20582 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35752 (request call id 1683) took 1682 ms. Trace:
W20260129 03:00:10.010953 20407 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.211.2:34517, user_credentials={real_user=slave}} blocked reactor thread for 100216us
I20260129 03:00:10.012120 20582 rpcz_store.cc:276] 0129 03:00:08.329747 (+ 0us) service_pool.cc:168] Inserting onto call queue
0129 03:00:09.958148 (+1628401us) service_pool.cc:225] Handling call
0129 03:00:10.009490 (+ 51342us) raft_consensus.cc:1471] Updating replica for 1 ops
0129 03:00:10.011472 (+ 1982us) raft_consensus.cc:1697] Filling consensus response to leader.
0129 03:00:10.011679 (+ 207us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260129 03:00:10.013587 20588 raft_consensus.cc:1275] T 0f20af7fbe5d4db89a6f17948d98912c P 1c50de89a5a845668696dcb28c3badd6 [term 7 FOLLOWER]: Refusing update from remote peer 197b880fa39d432a9b472845876bfaf0: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 7 index: 5. (index mismatch)
I20260129 03:00:09.928304 23398 raft_consensus.cc:493] T 18f30ebaa4174fd28c958e72d0cc7e24 P 1c50de89a5a845668696dcb28c3badd6 [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260129 03:00:09.923718 23389 raft_consensus.cc:2804] T 8055232cf0004d40a85f3bf983fe00a9 P 197b880fa39d432a9b472845876bfaf0 [term 5 FOLLOWER]: Leader pre-election won for term 6
W20260129 03:00:09.932682 23352 consensus_meta.cc:287] T 87d5fe48a44a408aaf03e894a67ecb2f P 197b880fa39d432a9b472845876bfaf0: Time spent flushing consensus metadata: real 0.989s user 0.011s sys 0.000s
I20260129 03:00:10.014361 20588 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35754 (request call id 1814) took 1464 ms. Trace:
W20260129 03:00:09.995410 20435 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.575s user 0.005s sys 0.006s
I20260129 03:00:10.021222 20588 rpcz_store.cc:276] 0129 03:00:08.550565 (+ 0us) service_pool.cc:168] Inserting onto call queue
0129 03:00:09.974063 (+1423498us) service_pool.cc:225] Handling call
0129 03:00:10.013275 (+ 39212us) raft_consensus.cc:1471] Updating replica for 0 ops
0129 03:00:10.014116 (+ 841us) raft_consensus.cc:1697] Filling consensus response to leader.
0129 03:00:10.014308 (+ 192us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20260129 03:00:10.025563 23328 log.cc:927] Time spent T 75be4bd6bd45474795fd472eff2953ac P 974b802141bd4f13beffef2aa03f88f9: Append to log took a long time: real 2.726s user 0.007s sys 0.023s
W20260129 03:00:09.995131 23035 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.701s user 0.009s sys 0.009s
W20260129 03:00:10.056033 20555 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260129 03:00:09.981573 23203 raft_consensus.cc:2804] T 18f30ebaa4174fd28c958e72d0cc7e24 P 7a9daf5d42a84615a6383cfec9a10608 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260129 03:00:09.985138 23298 raft_consensus.cc:493] T a5a05fbcbbcf4916bbc01fd1a5b5e90d P 1c50de89a5a845668696dcb28c3badd6 [term 7 FOLLOWER]: Starting leader election (detected failure of leader 974b802141bd4f13beffef2aa03f88f9)
I20260129 03:00:10.057096 20588 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3e2d15f8640b407195210c476e09df91" candidate_uuid: "974b802141bd4f13beffef2aa03f88f9" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "1c50de89a5a845668696dcb28c3badd6" is_pre_election: true
W20260129 03:00:10.022588 23182 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.548s user 0.000s sys 0.021s
I20260129 03:00:10.009660 23087 raft_consensus.cc:515] T 42d27a28d89b409e8807ec0efd97af69 P 197b880fa39d432a9b472845876bfaf0 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } }
I20260129 03:00:10.064677 20511 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57600 (request call id 1909) took 3462 ms. Trace:
I20260129 03:00:10.020421 23398 raft_consensus.cc:3060] T 18f30ebaa4174fd28c958e72d0cc7e24 P 1c50de89a5a845668696dcb28c3badd6 [term 7 FOLLOWER]: Advancing to term 8
I20260129 03:00:10.021124 23389 raft_consensus.cc:493] T 8055232cf0004d40a85f3bf983fe00a9 P 197b880fa39d432a9b472845876bfaf0 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260129 03:00:10.070367 20582 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "75be4bd6bd45474795fd472eff2953ac" candidate_uuid: "974b802141bd4f13beffef2aa03f88f9" candidate_term: 11 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "1c50de89a5a845668696dcb28c3badd6" is_pre_election: true
I20260129 03:00:09.982329 23280 raft_consensus.cc:515] T 3906b080a5b34a74b8b64128635e8997 P 1c50de89a5a845668696dcb28c3badd6 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
W20260129 03:00:10.068130 20480 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.211.3:37735, user_credentials={real_user=slave}} blocked reactor thread for 300676us
I20260129 03:00:10.015436 23215 raft_consensus.cc:2804] T 0678b8b468bd43cdbc3e485900844821 P 974b802141bd4f13beffef2aa03f88f9 [term 8 FOLLOWER]: Leader pre-election won for term 9
W20260129 03:00:10.080595 20407 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.211.2:34517, user_credentials={real_user=slave}} blocked reactor thread for 66438.9us
W20260129 03:00:10.105376 20480 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.18.211.3:37735 timed out after 1.539s (SENT) [suppressed 14 similar messages]
I20260129 03:00:10.080155 20511 rpcz_store.cc:276] 0129 03:00:06.602576 (+ 0us) service_pool.cc:168] Inserting onto call queue
0129 03:00:06.758369 (+155793us) service_pool.cc:225] Handling call
0129 03:00:06.790065 (+ 31696us) raft_consensus.cc:1471] Updating replica for 1 ops
0129 03:00:06.829513 (+ 39448us) raft_consensus.cc:1534] Early marking committed up to index 6
0129 03:00:06.849096 (+ 19583us) log.cc:844] Serialized 14 byte log entry
0129 03:00:07.095545 (+246449us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0129 03:00:07.113562 (+ 18017us) log.cc:844] Serialized 26 byte log entry
0129 03:00:07.113747 (+ 185us) raft_consensus.cc:1645] Marking committed up to 6
0129 03:00:07.113799 (+ 52us) raft_consensus.cc:1697] Filling consensus response to leader.
0129 03:00:07.113896 (+ 97us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0129 03:00:09.930931 (+2817035us) spinlock_profiling.cc:243] Waited 2.57 ms on lock 0x7b440002304c. stack: 00007f8fe044e373 00007f8fe044e270 00007f8fdfda0574 00007f8fe9a5c6b4 00007f8fe9a5c649 00007f8fe02c08b1 00007f8fe038f005 00007f8fe038fa22 00007f8fe30d957b 00007f8fe30d4e4f 00007f8fe30c8c0e 00007f8fe30c7f56 00007f8fe30c7b78 00007f8fe30d15c1 00007f8fe30d1579 00007f8fe30d1509
0129 03:00:10.060558 (+129627us) raft_consensus.cc:1685] finished
0129 03:00:10.064471 (+ 3913us) raft_consensus.cc:1691] UpdateReplicas() finished
0129 03:00:10.064615 (+ 144us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":39700,"prepare.queue_time_us":16318,"prepare.run_cpu_time_us":46,"prepare.run_wall_time_us":48,"raft.queue_time_us":33782,"raft.run_cpu_time_us":290,"raft.run_wall_time_us":20501,"spinlock_wait_cycles":5899648,"thread_start_us":210757,"threads_started":1,"wal-append.queue_time_us":293600}
I20260129 03:00:10.061297 22932 leader_election.cc:290] T febffe4a8b394368b69b1d164308fb9a P 7a9daf5d42a84615a6383cfec9a10608 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 197b880fa39d432a9b472845876bfaf0 (127.18.211.1:42027), 974b802141bd4f13beffef2aa03f88f9 (127.18.211.2:34517)
I20260129 03:00:10.102157 20582 raft_consensus.cc:2468] T 75be4bd6bd45474795fd472eff2953ac P 1c50de89a5a845668696dcb28c3badd6 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 974b802141bd4f13beffef2aa03f88f9 in term 10.
W20260129 03:00:10.103302 20479 connection.cc:232] RPC call timeout handler was delayed by -1.4128363459999491s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2026s for a response
W20260129 03:00:10.075276 22855 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.610s user 0.005s sys 0.020s
I20260129 03:00:10.065709 23400 raft_consensus.cc:2804] T c891d2bff4ce4568861bbc992e91fbb0 P 197b880fa39d432a9b472845876bfaf0 [term 8 FOLLOWER]: Leader pre-election won for term 9
W20260129 03:00:10.078032 23182 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.603s user 0.000s sys 0.021s
I20260129 03:00:10.120008 20436 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e7d83d4ccf684e76acb29a0dc844e1b8" candidate_uuid: "974b802141bd4f13beffef2aa03f88f9" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "197b880fa39d432a9b472845876bfaf0"
I20260129 03:00:10.120784 20436 raft_consensus.cc:3060] T e7d83d4ccf684e76acb29a0dc844e1b8 P 197b880fa39d432a9b472845876bfaf0 [term 6 FOLLOWER]: Advancing to term 7
I20260129 03:00:10.121834 20434 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0f20af7fbe5d4db89a6f17948d98912c" candidate_uuid: "974b802141bd4f13beffef2aa03f88f9" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "197b880fa39d432a9b472845876bfaf0" is_pre_election: true
I20260129 03:00:10.077204 23203 raft_consensus.cc:493] T 18f30ebaa4174fd28c958e72d0cc7e24 P 7a9daf5d42a84615a6383cfec9a10608 [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260129 03:00:10.126216 20405 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.211.3:37735, user_credentials={real_user=slave}} blocked reactor thread for 397399us
I20260129 03:00:10.035873 23352 raft_consensus.cc:515] T 87d5fe48a44a408aaf03e894a67ecb2f P 197b880fa39d432a9b472845876bfaf0 [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } }
I20260129 03:00:10.137459 20516 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:48544 (request call id 313) took 3087 ms. Trace:
I20260129 03:00:10.137976 20516 rpcz_store.cc:276] 0129 03:00:07.049634 (+ 0us) service_pool.cc:168] Inserting onto call queue
0129 03:00:07.674827 (+625193us) service_pool.cc:225] Handling call
0129 03:00:07.676343 (+ 1516us) raft_consensus.cc:1471] Updating replica for 0 ops
0129 03:00:07.827645 (+151302us) raft_consensus.cc:1697] Filling consensus response to leader.
0129 03:00:10.137154 (+2309509us) spinlock_profiling.cc:243] Waited 149 ms on lock 0x7b5400315454. stack: 00007f8fe044e373 00007f8fe044e270 00007f8fdfda0574 00007f8fe9a5c6b4 00007f8fe9a5c649 00007f8fe34975a1 00007f8fe352ea81 00007f8fe352d4be 00007f8fe84e1f49 00007f8fe1a6ed34 00007f8fe1a6ecc2 00007f8fe1a6ebf1 00007f8fe1a6eb6c 00007f8fe1a6de22 00007f8fe186d34c 00007f8fe186c786
0129 03:00:10.137381 (+ 227us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":342586112}
I20260129 03:00:10.131834 20437 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "87d5fe48a44a408aaf03e894a67ecb2f" candidate_uuid: "974b802141bd4f13beffef2aa03f88f9" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "197b880fa39d432a9b472845876bfaf0"
W20260129 03:00:10.133085 20480 connection.cc:232] RPC call timeout handler was delayed by -1.0383573980000165s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1887s for a response
I20260129 03:00:10.077860 23298 raft_consensus.cc:3060] T a5a05fbcbbcf4916bbc01fd1a5b5e90d P 1c50de89a5a845668696dcb28c3badd6 [term 7 FOLLOWER]: Advancing to term 8
I20260129 03:00:10.134323 20588 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7a9a69cd82724b1faed1d10aa091fc4a" candidate_uuid: "974b802141bd4f13beffef2aa03f88f9" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "1c50de89a5a845668696dcb28c3badd6" is_pre_election: true
I20260129 03:00:10.141422 20588 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0f20af7fbe5d4db89a6f17948d98912c" candidate_uuid: "974b802141bd4f13beffef2aa03f88f9" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "1c50de89a5a845668696dcb28c3badd6" is_pre_election: true
I20260129 03:00:10.142001 20588 raft_consensus.cc:2468] T 0f20af7fbe5d4db89a6f17948d98912c P 1c50de89a5a845668696dcb28c3badd6 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 974b802141bd4f13beffef2aa03f88f9 in term 7.
I20260129 03:00:10.142216 20582 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e7d83d4ccf684e76acb29a0dc844e1b8" candidate_uuid: "974b802141bd4f13beffef2aa03f88f9" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "1c50de89a5a845668696dcb28c3badd6"
I20260129 03:00:10.143059 20582 raft_consensus.cc:3060] T e7d83d4ccf684e76acb29a0dc844e1b8 P 1c50de89a5a845668696dcb28c3badd6 [term 6 FOLLOWER]: Advancing to term 7
I20260129 03:00:10.144089 20434 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "febffe4a8b394368b69b1d164308fb9a" candidate_uuid: "7a9daf5d42a84615a6383cfec9a10608" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "197b880fa39d432a9b472845876bfaf0" is_pre_election: true
I20260129 03:00:10.104821 23389 raft_consensus.cc:3060] T 8055232cf0004d40a85f3bf983fe00a9 P 197b880fa39d432a9b472845876bfaf0 [term 5 FOLLOWER]: Advancing to term 6
I20260129 03:00:10.135219 20432 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c2633a6fc34c44579de348d208b0c01e" candidate_uuid: "974b802141bd4f13beffef2aa03f88f9" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "197b880fa39d432a9b472845876bfaf0"
I20260129 03:00:10.167559 20432 raft_consensus.cc:3060] T c2633a6fc34c44579de348d208b0c01e P 197b880fa39d432a9b472845876bfaf0 [term 4 FOLLOWER]: Advancing to term 5
I20260129 03:00:10.113878 23215 raft_consensus.cc:493] T 0678b8b468bd43cdbc3e485900844821 P 974b802141bd4f13beffef2aa03f88f9 [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260129 03:00:10.170071 20511 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a609ac02fae84725ab743f95f51d33ce" candidate_uuid: "197b880fa39d432a9b472845876bfaf0" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "974b802141bd4f13beffef2aa03f88f9"
I20260129 03:00:10.170918 20511 raft_consensus.cc:3060] T a609ac02fae84725ab743f95f51d33ce P 974b802141bd4f13beffef2aa03f88f9 [term 8 FOLLOWER]: Advancing to term 9
I20260129 03:00:10.124715 23400 raft_consensus.cc:493] T c891d2bff4ce4568861bbc992e91fbb0 P 197b880fa39d432a9b472845876bfaf0 [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260129 03:00:10.175176 21101 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.540s user 0.038s sys 0.046s
W20260129 03:00:10.175622 21101 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.540s user 0.038s sys 0.046s
W20260129 03:00:10.134770 21100 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.211.3:37735, user_credentials={real_user=slave}} blocked reactor thread for 445478us
I20260129 03:00:10.115496 22694 raft_consensus.cc:493] T 17d6431cc77a491e8464d68db4081a51 P 974b802141bd4f13beffef2aa03f88f9 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260129 03:00:10.189211 20516 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e7d83d4ccf684e76acb29a0dc844e1b8" candidate_uuid: "197b880fa39d432a9b472845876bfaf0" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "974b802141bd4f13beffef2aa03f88f9" is_pre_election: true
I20260129 03:00:10.128551 23404 raft_consensus.cc:493] T 0340229d521b432580ae926f6062e3f7 P 197b880fa39d432a9b472845876bfaf0 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 974b802141bd4f13beffef2aa03f88f9)
I20260129 03:00:10.152307 20434 raft_consensus.cc:2393] T febffe4a8b394368b69b1d164308fb9a P 197b880fa39d432a9b472845876bfaf0 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 7a9daf5d42a84615a6383cfec9a10608 in current term 5: Already voted for candidate 197b880fa39d432a9b472845876bfaf0 in this term.
I20260129 03:00:10.102537 23208 consensus_queue.cc:1048] T a1a39dcd328e4268ae41b988c66b4a75 P 974b802141bd4f13beffef2aa03f88f9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.046s
I20260129 03:00:10.138222 23203 raft_consensus.cc:3060] T 18f30ebaa4174fd28c958e72d0cc7e24 P 7a9daf5d42a84615a6383cfec9a10608 [term 7 FOLLOWER]: Advancing to term 8
W20260129 03:00:10.114512 22855 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.665s user 0.005s sys 0.021s
W20260129 03:00:10.184824 20405 connection.cc:232] RPC call timeout handler was delayed by -1.1439161079999849s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.21960000000000002s for a response
I20260129 03:00:10.184826 20514 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:48544 (request call id 296) took 3819 ms. Trace:
I20260129 03:00:10.190069 20516 raft_consensus.cc:2393] T e7d83d4ccf684e76acb29a0dc844e1b8 P 974b802141bd4f13beffef2aa03f88f9 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 197b880fa39d432a9b472845876bfaf0 in current term 7: Already voted for candidate 974b802141bd4f13beffef2aa03f88f9 in this term.
I20260129 03:00:10.190834 20437 raft_consensus.cc:2368] T 87d5fe48a44a408aaf03e894a67ecb2f P 197b880fa39d432a9b472845876bfaf0 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 974b802141bd4f13beffef2aa03f88f9 for earlier term 6. Current term is 7.
I20260129 03:00:10.197991 20514 rpcz_store.cc:276] 0129 03:00:06.365937 (+ 0us) service_pool.cc:168] Inserting onto call queue
0129 03:00:06.742814 (+376877us) service_pool.cc:225] Handling call
0129 03:00:07.648002 (+905188us) raft_consensus.cc:1471] Updating replica for 0 ops
0129 03:00:07.782092 (+134090us) raft_consensus.cc:1697] Filling consensus response to leader.
0129 03:00:10.184634 (+2402542us) spinlock_profiling.cc:243] Waited 905 ms on lock 0x7b54002ee350. stack: 00007f8fe044e373 00007f8fe044e270 00007f8fdfda0574 00007f8fe9a5c6b4 00007f8fe9a5c649 00007f8fe34975a1 00007f8fe352d52d 00007f8fe84e1f49 00007f8fe1a6ed34 00007f8fe1a6ecc2 00007f8fe1a6ebf1 00007f8fe1a6eb6c 00007f8fe1a6de22 00007f8fe186d34c 00007f8fe186c786 00007f8fe186c12f
0129 03:00:10.184768 (+ 134us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2080999168}
I20260129 03:00:10.114297 23205 consensus_queue.cc:1059] T febffe4a8b394368b69b1d164308fb9a P 974b802141bd4f13beffef2aa03f88f9 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 }, Status: INVALID_TERM, Last received: 3.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.059s
I20260129 03:00:10.146122 22932 raft_consensus.cc:493] T c934d31efd3f4e5195605ddc354d14e1 P 7a9daf5d42a84615a6383cfec9a10608 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 197b880fa39d432a9b472845876bfaf0)
I20260129 03:00:10.173789 23215 raft_consensus.cc:3060] T 0678b8b468bd43cdbc3e485900844821 P 974b802141bd4f13beffef2aa03f88f9 [term 8 FOLLOWER]: Advancing to term 9
I20260129 03:00:10.079948 22891 consensus_queue.cc:237] T 36ff33be666a4c82954731691dbcf327 P 1c50de89a5a845668696dcb28c3badd6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 03:00:10.209246 20436 raft_consensus.cc:2468] T e7d83d4ccf684e76acb29a0dc844e1b8 P 197b880fa39d432a9b472845876bfaf0 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 974b802141bd4f13beffef2aa03f88f9 in term 7.
I20260129 03:00:10.202667 20434 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7d9494f5584d4d5db8b9adf1c8fc888b" candidate_uuid: "974b802141bd4f13beffef2aa03f88f9" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "197b880fa39d432a9b472845876bfaf0" is_pre_election: true
I20260129 03:00:10.259145 20514 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a5a05fbcbbcf4916bbc01fd1a5b5e90d" candidate_uuid: "197b880fa39d432a9b472845876bfaf0" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "974b802141bd4f13beffef2aa03f88f9" is_pre_election: true
I20260129 03:00:10.218451 20510 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2ac659d732314e1288849f08fe5e074e" candidate_uuid: "197b880fa39d432a9b472845876bfaf0" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "974b802141bd4f13beffef2aa03f88f9" is_pre_election: true
I20260129 03:00:10.307159 20586 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:35752 (request call id 1696) took 1064 ms. Trace:
I20260129 03:00:10.307497 20510 raft_consensus.cc:2468] T 2ac659d732314e1288849f08fe5e074e P 974b802141bd4f13beffef2aa03f88f9 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 197b880fa39d432a9b472845876bfaf0 in term 5.
I20260129 03:00:10.307787 20434 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3906b080a5b34a74b8b64128635e8997" candidate_uuid: "1c50de89a5a845668696dcb28c3badd6" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "197b880fa39d432a9b472845876bfaf0" is_pre_election: true
W20260129 03:00:10.282917 20406 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260129 03:00:10.259938 20508 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "febffe4a8b394368b69b1d164308fb9a" candidate_uuid: "197b880fa39d432a9b472845876bfaf0" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "974b802141bd4f13beffef2aa03f88f9"
W20260129 03:00:10.258642 20482 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260129 03:00:10.259516 20515 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "36ff33be666a4c82954731691dbcf327" candidate_uuid: "197b880fa39d432a9b472845876bfaf0" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "974b802141bd4f13beffef2aa03f88f9" is_pre_election: true
I20260129 03:00:10.260339 20513 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fcddab6bca684adf8ea032289a00f90e" candidate_uuid: "197b880fa39d432a9b472845876bfaf0" candidate_term: 10 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "974b802141bd4f13beffef2aa03f88f9" is_pre_election: true
W20260129 03:00:10.221669 20407 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.211.2:34517, user_credentials={real_user=slave}} blocked reactor thread for 87110.5us
I20260129 03:00:10.202625 20582 raft_consensus.cc:2468] T e7d83d4ccf684e76acb29a0dc844e1b8 P 1c50de89a5a845668696dcb28c3badd6 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 974b802141bd4f13beffef2aa03f88f9 in term 7.
I20260129 03:00:10.336423 20582 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fcddab6bca684adf8ea032289a00f90e" candidate_uuid: "197b880fa39d432a9b472845876bfaf0" candidate_term: 10 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "1c50de89a5a845668696dcb28c3badd6" is_pre_election: true
I20260129 03:00:10.307576 20586 rpcz_store.cc:276] 0129 03:00:09.242747 (+ 0us) service_pool.cc:168] Inserting onto call queue
0129 03:00:09.446042 (+203295us) service_pool.cc:225] Handling call
0129 03:00:10.306985 (+860943us) spinlock_profiling.cc:243] Waited 1.13 ms on lock 0x7b54000c8d54. stack: 00007f8fe044e373 00007f8fe044e270 00007f8fdfda0574 00007f8fe9a5c6b4 00007f8fe9a5c649 00007f8fe34975a1 00007f8fe3531f7a 00007f8fe84e3647 00007f8fe1a73834 00007f8fe1a737c2 00007f8fe1a736f1 00007f8fe1a7366c 00007f8fe1a72922 00007f8fe186d34c 00007f8fe186c786 00007f8fe186c12f
0129 03:00:10.307104 (+ 119us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2595200}
I20260129 03:00:10.337781 20507 raft_consensus.cc:1275] T a0fcc58d6fb6473cafbd9d34d5195269 P 974b802141bd4f13beffef2aa03f88f9 [term 5 FOLLOWER]: Refusing update from remote peer 197b880fa39d432a9b472845876bfaf0: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 5 index: 4. (index mismatch)
I20260129 03:00:10.338526 20586 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "dc2f4cb752cf4d59b5f6bc689f5d8104" candidate_uuid: "7a9daf5d42a84615a6383cfec9a10608" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "1c50de89a5a845668696dcb28c3badd6" is_pre_election: true
I20260129 03:00:10.339838 20507 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "febffe4a8b394368b69b1d164308fb9a" candidate_uuid: "7a9daf5d42a84615a6383cfec9a10608" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "974b802141bd4f13beffef2aa03f88f9" is_pre_election: true
I20260129 03:00:10.199378 23304 leader_election.cc:290] T dc2f4cb752cf4d59b5f6bc689f5d8104 P 1c50de89a5a845668696dcb28c3badd6 [CANDIDATE]: Term 6 election: Requested vote from peers 197b880fa39d432a9b472845876bfaf0 (127.18.211.1:42027), 7a9daf5d42a84615a6383cfec9a10608 (127.18.211.4:40439)
W20260129 03:00:10.292596 23141 consensus_meta.cc:287] T 78aa39096868415d9afc2ffcb248e198 P 7a9daf5d42a84615a6383cfec9a10608: Time spent flushing consensus metadata: real 0.792s user 0.010s sys 0.001s
I20260129 03:00:10.308496 20434 raft_consensus.cc:2468] T 3906b080a5b34a74b8b64128635e8997 P 197b880fa39d432a9b472845876bfaf0 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1c50de89a5a845668696dcb28c3badd6 in term 4.
W20260129 03:00:10.218578 20405 leader_election.cc:336] T 75be4bd6bd45474795fd472eff2953ac P 197b880fa39d432a9b472845876bfaf0 [CANDIDATE]: Term 10 pre-election: RPC error from VoteRequest() call to peer 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3:37735): Timed out: RequestConsensusVote RPC to 127.18.211.3:37735 timed out after 1.744s (SENT)
I20260129 03:00:10.341288 20432 raft_consensus.cc:2468] T c2633a6fc34c44579de348d208b0c01e P 197b880fa39d432a9b472845876bfaf0 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 974b802141bd4f13beffef2aa03f88f9 in term 5.
W20260129 03:00:10.292409 23407 log.cc:927] Time spent T add8d4923d1a489e86882c3a25692b95 P 7a9daf5d42a84615a6383cfec9a10608: Append to log took a long time: real 0.176s user 0.000s sys 0.000s
I20260129 03:00:10.394726 20407 leader_election.cc:304] T 2ac659d732314e1288849f08fe5e074e P 197b880fa39d432a9b472845876bfaf0 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 197b880fa39d432a9b472845876bfaf0, 974b802141bd4f13beffef2aa03f88f9; no voters:
I20260129 03:00:10.334151 20436 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "dc2f4cb752cf4d59b5f6bc689f5d8104" candidate_uuid: "7a9daf5d42a84615a6383cfec9a10608" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "197b880fa39d432a9b472845876bfaf0" is_pre_election: true
I20260129 03:00:10.345043 20515 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "87d5fe48a44a408aaf03e894a67ecb2f" candidate_uuid: "197b880fa39d432a9b472845876bfaf0" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "974b802141bd4f13beffef2aa03f88f9"
I20260129 03:00:10.397043 20436 raft_consensus.cc:2468] T dc2f4cb752cf4d59b5f6bc689f5d8104 P 197b880fa39d432a9b472845876bfaf0 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7a9daf5d42a84615a6383cfec9a10608 in term 4.
I20260129 03:00:10.397230 20515 raft_consensus.cc:3060] T 87d5fe48a44a408aaf03e894a67ecb2f P 974b802141bd4f13beffef2aa03f88f9 [term 6 FOLLOWER]: Advancing to term 7
I20260129 03:00:10.340526 20479 leader_election.cc:304] T e7d83d4ccf684e76acb29a0dc844e1b8 P 974b802141bd4f13beffef2aa03f88f9 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 197b880fa39d432a9b472845876bfaf0, 974b802141bd4f13beffef2aa03f88f9; no voters:
W20260129 03:00:10.291769 23406 log.cc:927] Time spent T 4f5a77e94fef4b16a2ba7b067c8b144a P 7a9daf5d42a84615a6383cfec9a10608: Append to log took a long time: real 0.117s user 0.000s sys 0.000s
I20260129 03:00:10.196554 23333 leader_election.cc:290] T cb4727386a3e48aca5d71cc6c611c197 P 974b802141bd4f13beffef2aa03f88f9 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 197b880fa39d432a9b472845876bfaf0 (127.18.211.1:42027), 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3:37735)
I20260129 03:00:10.347621 20434 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fcddab6bca684adf8ea032289a00f90e" candidate_uuid: "974b802141bd4f13beffef2aa03f88f9" candidate_term: 10 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "197b880fa39d432a9b472845876bfaf0"
I20260129 03:00:10.411561 20434 raft_consensus.cc:3060] T fcddab6bca684adf8ea032289a00f90e P 197b880fa39d432a9b472845876bfaf0 [term 9 FOLLOWER]: Advancing to term 10
I20260129 03:00:10.340031 20586 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e7d83d4ccf684e76acb29a0dc844e1b8" candidate_uuid: "197b880fa39d432a9b472845876bfaf0" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "1c50de89a5a845668696dcb28c3badd6" is_pre_election: true
I20260129 03:00:10.412928 20586 raft_consensus.cc:2393] T e7d83d4ccf684e76acb29a0dc844e1b8 P 1c50de89a5a845668696dcb28c3badd6 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 197b880fa39d432a9b472845876bfaf0 in current term 7: Already voted for candidate 974b802141bd4f13beffef2aa03f88f9 in this term.
I20260129 03:00:10.423733 20436 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "96609ac68bdf4993a12e1c23355ca013" candidate_uuid: "974b802141bd4f13beffef2aa03f88f9" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "197b880fa39d432a9b472845876bfaf0" is_pre_election: true
I20260129 03:00:10.216962 23402 raft_consensus.cc:493] T 505580c93c694bbb8ea96fdf9b17ab46 P 974b802141bd4f13beffef2aa03f88f9 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260129 03:00:10.349123 20582 raft_consensus.cc:2468] T fcddab6bca684adf8ea032289a00f90e P 1c50de89a5a845668696dcb28c3badd6 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 197b880fa39d432a9b472845876bfaf0 in term 9.
I20260129 03:00:10.337325 20513 raft_consensus.cc:2393] T fcddab6bca684adf8ea032289a00f90e P 974b802141bd4f13beffef2aa03f88f9 [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 197b880fa39d432a9b472845876bfaf0 in current term 10: Already voted for candidate 974b802141bd4f13beffef2aa03f88f9 in this term.
I20260129 03:00:10.335183 20511 raft_consensus.cc:2468] T a609ac02fae84725ab743f95f51d33ce P 974b802141bd4f13beffef2aa03f88f9 [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 197b880fa39d432a9b472845876bfaf0 in term 9.
W20260129 03:00:10.252292 23405 log.cc:927] Time spent T 4408006eb5174de1896b037cf3f18457 P 1c50de89a5a845668696dcb28c3badd6: Append to log took a long time: real 0.189s user 0.000s sys 0.000s
I20260129 03:00:10.474777 20436 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4e420f90884f4796997b5ea41e49e411" candidate_uuid: "974b802141bd4f13beffef2aa03f88f9" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "197b880fa39d432a9b472845876bfaf0" is_pre_election: true
I20260129 03:00:10.239717 23032 raft_consensus.cc:493] T cb0b3f259ebc4310927338a8740e87b8 P 974b802141bd4f13beffef2aa03f88f9 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 7a9daf5d42a84615a6383cfec9a10608)
I20260129 03:00:10.261426 23205 raft_consensus.cc:3055] T febffe4a8b394368b69b1d164308fb9a P 974b802141bd4f13beffef2aa03f88f9 [term 4 LEADER]: Stepping down as leader of term 4
W20260129 03:00:10.439564 20479 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.211.1:42027, user_credentials={real_user=slave}} blocked reactor thread for 99860.5us
W20260129 03:00:10.446377 20405 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.211.3:37735, user_credentials={real_user=slave}} blocked reactor thread for 246885us
I20260129 03:00:10.250984 23087 leader_election.cc:290] T 42d27a28d89b409e8807ec0efd97af69 P 197b880fa39d432a9b472845876bfaf0 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3:37735), 7a9daf5d42a84615a6383cfec9a10608 (127.18.211.4:40439)
I20260129 03:00:10.199775 22694 raft_consensus.cc:515] T 17d6431cc77a491e8464d68db4081a51 P 974b802141bd4f13beffef2aa03f88f9 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 03:00:10.448757 20589 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:48526 (request call id 380) took 2529 ms. Trace:
I20260129 03:00:10.293424 23280 leader_election.cc:290] T 3906b080a5b34a74b8b64128635e8997 P 1c50de89a5a845668696dcb28c3badd6 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 197b880fa39d432a9b472845876bfaf0 (127.18.211.1:42027), 974b802141bd4f13beffef2aa03f88f9 (127.18.211.2:34517)
I20260129 03:00:10.294863 23352 leader_election.cc:290] T 87d5fe48a44a408aaf03e894a67ecb2f P 197b880fa39d432a9b472845876bfaf0 [CANDIDATE]: Term 7 election: Requested vote from peers 974b802141bd4f13beffef2aa03f88f9 (127.18.211.2:34517), 7a9daf5d42a84615a6383cfec9a10608 (127.18.211.4:40439)
I20260129 03:00:10.182621 23400 raft_consensus.cc:3060] T c891d2bff4ce4568861bbc992e91fbb0 P 197b880fa39d432a9b472845876bfaf0 [term 8 FOLLOWER]: Advancing to term 9
I20260129 03:00:10.468129 20433 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:48356 (request call id 420) took 2010 ms. Trace:
I20260129 03:00:10.469707 20432 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "dc2f4cb752cf4d59b5f6bc689f5d8104" candidate_uuid: "1c50de89a5a845668696dcb28c3badd6" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "197b880fa39d432a9b472845876bfaf0"
I20260129 03:00:10.571439 20432 raft_consensus.cc:3060] T dc2f4cb752cf4d59b5f6bc689f5d8104 P 197b880fa39d432a9b472845876bfaf0 [term 4 FOLLOWER]: Advancing to term 6
W20260129 03:00:10.467811 20439 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55672 (request call id 1697) took 3619 ms (client timeout 2026 ms). Trace:
I20260129 03:00:10.347803 20553 leader_election.cc:304] T 3906b080a5b34a74b8b64128635e8997 P 1c50de89a5a845668696dcb28c3badd6 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 197b880fa39d432a9b472845876bfaf0, 1c50de89a5a845668696dcb28c3badd6; no voters:
I20260129 03:00:10.448863 22989 raft_consensus.cc:493] T 75f45538dc704d9a8ec4ff8ad9f3ea5b P 974b802141bd4f13beffef2aa03f88f9 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 7a9daf5d42a84615a6383cfec9a10608)
W20260129 03:00:10.572175 20439 rpcz_store.cc:269] 0129 03:00:06.847922 (+ 0us) service_pool.cc:168] Inserting onto call queue
0129 03:00:06.848526 (+ 604us) service_pool.cc:225] Handling call
0129 03:00:10.467555 (+3619029us) spinlock_profiling.cc:243] Waited 1.22 s on lock 0x7b54003d1654. stack: 00007f8fe044e373 00007f8fe044e270 00007f8fdfda0574 00007f8fe9a5c6b4 00007f8fe9a5c649 00007f8fe34975a1 00007f8fe3531f7a 00007f8fe84e3647 00007f8fe1a73834 00007f8fe1a737c2 00007f8fe1a736f1 00007f8fe1a7366c 00007f8fe1a72922 00007f8fe186d34c 00007f8fe186c786 00007f8fe186c12f
0129 03:00:10.467735 (+ 180us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2805798784}
I20260129 03:00:10.448372 23429 raft_consensus.cc:2804] T e7d83d4ccf684e76acb29a0dc844e1b8 P 974b802141bd4f13beffef2aa03f88f9 [term 7 FOLLOWER]: Leader election won for term 7
I20260129 03:00:10.401189 23141 raft_consensus.cc:2749] T 78aa39096868415d9afc2ffcb248e198 P 7a9daf5d42a84615a6383cfec9a10608 [term 7 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260129 03:00:10.488294 20582 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "505580c93c694bbb8ea96fdf9b17ab46" candidate_uuid: "7a9daf5d42a84615a6383cfec9a10608" candidate_term: 8 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "1c50de89a5a845668696dcb28c3badd6" is_pre_election: true
I20260129 03:00:10.487591 23428 raft_consensus.cc:2804] T 2ac659d732314e1288849f08fe5e074e P 197b880fa39d432a9b472845876bfaf0 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260129 03:00:10.241612 22932 raft_consensus.cc:515] T c934d31efd3f4e5195605ddc354d14e1 P 7a9daf5d42a84615a6383cfec9a10608 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } }
I20260129 03:00:10.571166 20433 rpcz_store.cc:276] 0129 03:00:08.457995 (+ 0us) service_pool.cc:168] Inserting onto call queue
0129 03:00:08.477271 (+ 19276us) service_pool.cc:225] Handling call
0129 03:00:08.530397 (+ 53126us) raft_consensus.cc:1471] Updating replica for 0 ops
0129 03:00:09.576340 (+1045943us) raft_consensus.cc:1697] Filling consensus response to leader.
0129 03:00:10.467841 (+891501us) spinlock_profiling.cc:243] Waited 1.04 s on lock 0x7b54002fc954. stack: 00007f8fe044e373 00007f8fe044e270 00007f8fdfda0574 00007f8fe9a5c6b4 00007f8fe9a5c649 00007f8fe34975a1 00007f8fe352ea81 00007f8fe352d4be 00007f8fe84e1f49 00007f8fe1a6ed34 00007f8fe1a6ecc2 00007f8fe1a6ebf1 00007f8fe1a6eb6c 00007f8fe1a6de22 00007f8fe186d34c 00007f8fe186c786
0129 03:00:10.468073 (+ 232us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2389246336}
I20260129 03:00:10.212550 23404 raft_consensus.cc:515] T 0340229d521b432580ae926f6062e3f7 P 197b880fa39d432a9b472845876bfaf0 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 03:00:10.342268 20507 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "505580c93c694bbb8ea96fdf9b17ab46" candidate_uuid: "7a9daf5d42a84615a6383cfec9a10608" candidate_term: 8 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "974b802141bd4f13beffef2aa03f88f9" is_pre_election: true
W20260129 03:00:10.561758 20479 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260129 03:00:10.293740 23091 leader_election.cc:290] T a5a05fbcbbcf4916bbc01fd1a5b5e90d P 197b880fa39d432a9b472845876bfaf0 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 974b802141bd4f13beffef2aa03f88f9 (127.18.211.2:34517), 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3:37735)
W20260129 03:00:10.799291 20553 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.211.1:42027, user_credentials={real_user=slave}} blocked reactor thread for 451527us
I20260129 03:00:10.640293 20511 raft_consensus.cc:1275] T add8d4923d1a489e86882c3a25692b95 P 974b802141bd4f13beffef2aa03f88f9 [term 7 FOLLOWER]: Refusing update from remote peer 7a9daf5d42a84615a6383cfec9a10608: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 7 index: 6. (index mismatch)
I20260129 03:00:10.597306 20583 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35754 (request call id 1797) took 3174 ms. Trace:
W20260129 03:00:10.606784 21101 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260129 03:00:10.573303 20586 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "8055232cf0004d40a85f3bf983fe00a9" candidate_uuid: "197b880fa39d432a9b472845876bfaf0" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "1c50de89a5a845668696dcb28c3badd6" is_pre_election: true
I20260129 03:00:10.802160 20586 raft_consensus.cc:2468] T 8055232cf0004d40a85f3bf983fe00a9 P 1c50de89a5a845668696dcb28c3badd6 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 197b880fa39d432a9b472845876bfaf0 in term 5.
I20260129 03:00:10.570588 20589 rpcz_store.cc:276] 0129 03:00:07.928114 (+ 0us) service_pool.cc:168] Inserting onto call queue
0129 03:00:09.618670 (+1690556us) service_pool.cc:225] Handling call
0129 03:00:09.630007 (+ 11337us) raft_consensus.cc:1471] Updating replica for 1 ops
0129 03:00:09.630465 (+ 458us) raft_consensus.cc:1534] Early marking committed up to index 0
0129 03:00:09.630507 (+ 42us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0129 03:00:09.631171 (+ 664us) log.cc:844] Serialized 26 byte log entry
0129 03:00:10.067760 (+436589us) raft_consensus.cc:1645] Marking committed up to 5
0129 03:00:10.067821 (+ 61us) raft_consensus.cc:1697] Filling consensus response to leader.
0129 03:00:10.067930 (+ 109us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0129 03:00:10.448464 (+380534us) raft_consensus.cc:1685] finished
0129 03:00:10.448488 (+ 24us) raft_consensus.cc:1691] UpdateReplicas() finished
0129 03:00:10.448676 (+ 188us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":1286,"raft.run_cpu_time_us":131,"raft.run_wall_time_us":132,"thread_start_us":431466,"threads_started":1,"wal-append.queue_time_us":711425}
I20260129 03:00:10.797245 20434 raft_consensus.cc:2468] T fcddab6bca684adf8ea032289a00f90e P 197b880fa39d432a9b472845876bfaf0 [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 974b802141bd4f13beffef2aa03f88f9 in term 10.
W20260129 03:00:10.639937 20405 leader_election.cc:336] T e33a417cf9c54cf5af73a4ee3aac4d3a P 197b880fa39d432a9b472845876bfaf0 [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3:37735): Timed out: RequestConsensusVote RPC to 127.18.211.3:37735 timed out after 1.859s (SENT)
I20260129 03:00:10.800673 20583 rpcz_store.cc:276] 0129 03:00:07.422946 (+ 0us) service_pool.cc:168] Inserting onto call queue
0129 03:00:07.434400 (+ 11454us) service_pool.cc:225] Handling call
0129 03:00:07.450893 (+ 16493us) raft_consensus.cc:1471] Updating replica for 0 ops
0129 03:00:08.085448 (+634555us) raft_consensus.cc:1697] Filling consensus response to leader.
0129 03:00:10.596938 (+2511490us) spinlock_profiling.cc:243] Waited 615 ms on lock 0x7b54003b0954. stack: 00007f8fe044e373 00007f8fe044e270 00007f8fdfda0574 00007f8fe9a5c6b4 00007f8fe9a5c649 00007f8fe34975a1 00007f8fe352ea81 00007f8fe352d4be 00007f8fe84e1f49 00007f8fe1a6ed34 00007f8fe1a6ecc2 00007f8fe1a6ebf1 00007f8fe1a6eb6c 00007f8fe1a6de22 00007f8fe186d34c 00007f8fe186c786
0129 03:00:10.597213 (+ 275us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1413728128}
I20260129 03:00:10.797039 20436 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "cb4727386a3e48aca5d71cc6c611c197" candidate_uuid: "974b802141bd4f13beffef2aa03f88f9" candidate_term: 6 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "197b880fa39d432a9b472845876bfaf0" is_pre_election: true
I20260129 03:00:10.799052 20515 raft_consensus.cc:2468] T 87d5fe48a44a408aaf03e894a67ecb2f P 974b802141bd4f13beffef2aa03f88f9 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 197b880fa39d432a9b472845876bfaf0 in term 7.
W20260129 03:00:10.604991 20555 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260129 03:00:10.806861 20405 leader_election.cc:304] T e33a417cf9c54cf5af73a4ee3aac4d3a P 197b880fa39d432a9b472845876bfaf0 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 197b880fa39d432a9b472845876bfaf0; no voters: 1c50de89a5a845668696dcb28c3badd6, 974b802141bd4f13beffef2aa03f88f9
I20260129 03:00:10.798461 20480 leader_election.cc:304] T c934d31efd3f4e5195605ddc354d14e1 P 974b802141bd4f13beffef2aa03f88f9 [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 974b802141bd4f13beffef2aa03f88f9; no voters: 197b880fa39d432a9b472845876bfaf0, 7a9daf5d42a84615a6383cfec9a10608
I20260129 03:00:10.474895 23333 raft_consensus.cc:493] T a4c1226b987b468fa84dfc6ba0e2dbe1 P 974b802141bd4f13beffef2aa03f88f9 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 197b880fa39d432a9b472845876bfaf0)
W20260129 03:00:10.817548 20509 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.517s user 0.014s sys 0.006s
W20260129 03:00:10.824610 20509 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.524s user 0.014s sys 0.006s
I20260129 03:00:10.828881 20432 raft_consensus.cc:2468] T dc2f4cb752cf4d59b5f6bc689f5d8104 P 197b880fa39d432a9b472845876bfaf0 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1c50de89a5a845668696dcb28c3badd6 in term 6.
I20260129 03:00:10.831317 20434 raft_consensus.cc:1275] T 368953c9102643b2a21f99af36b51905 P 197b880fa39d432a9b472845876bfaf0 [term 8 FOLLOWER]: Refusing update from remote peer 974b802141bd4f13beffef2aa03f88f9: Log matching property violated. Preceding OpId in replica: term: 6 index: 4. Preceding OpId from leader: term: 8 index: 5. (index mismatch)
I20260129 03:00:10.836311 21134 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "87d5fe48a44a408aaf03e894a67ecb2f" candidate_uuid: "974b802141bd4f13beffef2aa03f88f9" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "7a9daf5d42a84615a6383cfec9a10608"
I20260129 03:00:10.835701 20439 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c891d2bff4ce4568861bbc992e91fbb0" candidate_uuid: "974b802141bd4f13beffef2aa03f88f9" candidate_term: 8 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "197b880fa39d432a9b472845876bfaf0"
I20260129 03:00:10.540131 23205 raft_consensus.cc:740] T febffe4a8b394368b69b1d164308fb9a P 974b802141bd4f13beffef2aa03f88f9 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 974b802141bd4f13beffef2aa03f88f9, State: Running, Role: LEADER
I20260129 03:00:10.835758 21131 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "96609ac68bdf4993a12e1c23355ca013" candidate_uuid: "974b802141bd4f13beffef2aa03f88f9" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "7a9daf5d42a84615a6383cfec9a10608" is_pre_election: true
I20260129 03:00:10.841312 20433 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "926a250fa5834e88b96ccecbefbf5086" candidate_uuid: "974b802141bd4f13beffef2aa03f88f9" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "197b880fa39d432a9b472845876bfaf0" is_pre_election: true
I20260129 03:00:10.840423 20582 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7d9494f5584d4d5db8b9adf1c8fc888b" candidate_uuid: "974b802141bd4f13beffef2aa03f88f9" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "1c50de89a5a845668696dcb28c3badd6" is_pre_election: true
I20260129 03:00:10.752141 23403 raft_consensus.cc:3055] T c934d31efd3f4e5195605ddc354d14e1 P 197b880fa39d432a9b472845876bfaf0 [term 5 LEADER]: Stepping down as leader of term 5
I20260129 03:00:10.662834 23304 raft_consensus.cc:2804] T 3906b080a5b34a74b8b64128635e8997 P 1c50de89a5a845668696dcb28c3badd6 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260129 03:00:10.636770 23087 raft_consensus.cc:493] T 76f713c3331540d2a5b76164103e9eb1 P 197b880fa39d432a9b472845876bfaf0 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260129 03:00:10.843277 21134 raft_consensus.cc:2393] T 87d5fe48a44a408aaf03e894a67ecb2f P 7a9daf5d42a84615a6383cfec9a10608 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 974b802141bd4f13beffef2aa03f88f9 in current term 6: Already voted for candidate 7a9daf5d42a84615a6383cfec9a10608 in this term.
I20260129 03:00:10.804083 23428 raft_consensus.cc:493] T 2ac659d732314e1288849f08fe5e074e P 197b880fa39d432a9b472845876bfaf0 [term 5 FOLLOWER]: Starting leader election (detected failure of leader 7a9daf5d42a84615a6383cfec9a10608)
I20260129 03:00:10.847255 20583 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a644c02b16af47679824ccd060ac10fd" candidate_uuid: "974b802141bd4f13beffef2aa03f88f9" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "1c50de89a5a845668696dcb28c3badd6" is_pre_election: true
W20260129 03:00:10.995679 20406 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260129 03:00:10.846503 22645 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.513s user 0.009s sys 0.010s
I20260129 03:00:10.997134 23417 raft_consensus.cc:493] T 0678b8b468bd43cdbc3e485900844821 P 7a9daf5d42a84615a6383cfec9a10608 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260129 03:00:10.997367 22645 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.685s user 0.010s sys 0.011s
W20260129 03:00:10.835995 23046 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.568s user 0.010s sys 0.010s
I20260129 03:00:10.918761 20586 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c2633a6fc34c44579de348d208b0c01e" candidate_uuid: "974b802141bd4f13beffef2aa03f88f9" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "1c50de89a5a845668696dcb28c3badd6"
I20260129 03:00:10.918908 20582 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "96609ac68bdf4993a12e1c23355ca013" candidate_uuid: "974b802141bd4f13beffef2aa03f88f9" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "1c50de89a5a845668696dcb28c3badd6" is_pre_election: true
I20260129 03:00:11.069636 20582 raft_consensus.cc:2468] T 96609ac68bdf4993a12e1c23355ca013 P 1c50de89a5a845668696dcb28c3badd6 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 974b802141bd4f13beffef2aa03f88f9 in term 6.
I20260129 03:00:10.488957 23402 raft_consensus.cc:515] T 505580c93c694bbb8ea96fdf9b17ab46 P 974b802141bd4f13beffef2aa03f88f9 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } }
I20260129 03:00:10.834095 23282 raft_consensus.cc:493] T 75b0b6e82ddc4f92b1fe1d94f9a0b73b P 1c50de89a5a845668696dcb28c3badd6 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 7a9daf5d42a84615a6383cfec9a10608)
I20260129 03:00:11.111176 23282 raft_consensus.cc:515] T 75b0b6e82ddc4f92b1fe1d94f9a0b73b P 1c50de89a5a845668696dcb28c3badd6 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } }
I20260129 03:00:11.112686 23402 leader_election.cc:290] T 505580c93c694bbb8ea96fdf9b17ab46 P 974b802141bd4f13beffef2aa03f88f9 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3:37735), 7a9daf5d42a84615a6383cfec9a10608 (127.18.211.4:40439)
I20260129 03:00:11.113411 23402 raft_consensus.cc:2749] T c934d31efd3f4e5195605ddc354d14e1 P 974b802141bd4f13beffef2aa03f88f9 [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20260129 03:00:10.795682 22989 raft_consensus.cc:515] T 75f45538dc704d9a8ec4ff8ad9f3ea5b P 974b802141bd4f13beffef2aa03f88f9 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } }
I20260129 03:00:10.916855 23403 raft_consensus.cc:740] T c934d31efd3f4e5195605ddc354d14e1 P 197b880fa39d432a9b472845876bfaf0 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 197b880fa39d432a9b472845876bfaf0, State: Running, Role: LEADER
I20260129 03:00:11.114661 23403 consensus_queue.cc:260] T c934d31efd3f4e5195605ddc354d14e1 P 197b880fa39d432a9b472845876bfaf0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 5.7, Last appended by leader: 7, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } }
I20260129 03:00:11.115314 22989 leader_election.cc:290] T 75f45538dc704d9a8ec4ff8ad9f3ea5b P 974b802141bd4f13beffef2aa03f88f9 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3:37735), 7a9daf5d42a84615a6383cfec9a10608 (127.18.211.4:40439)
I20260129 03:00:10.832535 23429 raft_consensus.cc:697] T e7d83d4ccf684e76acb29a0dc844e1b8 P 974b802141bd4f13beffef2aa03f88f9 [term 7 LEADER]: Becoming Leader. State: Replica: 974b802141bd4f13beffef2aa03f88f9, State: Running, Role: LEADER
I20260129 03:00:11.116175 20507 raft_consensus.cc:2468] T 505580c93c694bbb8ea96fdf9b17ab46 P 974b802141bd4f13beffef2aa03f88f9 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7a9daf5d42a84615a6383cfec9a10608 in term 7.
I20260129 03:00:11.116501 23429 consensus_queue.cc:237] T e7d83d4ccf684e76acb29a0dc844e1b8 P 974b802141bd4f13beffef2aa03f88f9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.3, Last appended by leader: 3, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
W20260129 03:00:11.117221 20405 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.211.3:37735, user_credentials={real_user=slave}} blocked reactor thread for 556659us
I20260129 03:00:11.040469 20583 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c891d2bff4ce4568861bbc992e91fbb0" candidate_uuid: "197b880fa39d432a9b472845876bfaf0" candidate_term: 9 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "1c50de89a5a845668696dcb28c3badd6" is_pre_election: true
I20260129 03:00:11.120203 20583 raft_consensus.cc:2468] T c891d2bff4ce4568861bbc992e91fbb0 P 1c50de89a5a845668696dcb28c3badd6 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 197b880fa39d432a9b472845876bfaf0 in term 8.
I20260129 03:00:10.994397 23087 raft_consensus.cc:515] T 76f713c3331540d2a5b76164103e9eb1 P 197b880fa39d432a9b472845876bfaf0 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 03:00:11.121948 20583 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35754 (request call id 1815) took 2180 ms. Trace:
I20260129 03:00:11.122385 20583 rpcz_store.cc:276] 0129 03:00:08.941479 (+ 0us) service_pool.cc:168] Inserting onto call queue
0129 03:00:11.120902 (+2179423us) service_pool.cc:225] Handling call
0129 03:00:11.121323 (+ 421us) raft_consensus.cc:1471] Updating replica for 0 ops
0129 03:00:11.121554 (+ 231us) raft_consensus.cc:1534] Early marking committed up to index 0
0129 03:00:11.121585 (+ 31us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0129 03:00:11.121622 (+ 37us) raft_consensus.cc:1645] Marking committed up to 3
0129 03:00:11.121680 (+ 58us) raft_consensus.cc:1697] Filling consensus response to leader.
0129 03:00:11.121749 (+ 69us) raft_consensus.cc:1691] UpdateReplicas() finished
0129 03:00:11.121882 (+ 133us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260129 03:00:11.123485 23087 leader_election.cc:290] T 76f713c3331540d2a5b76164103e9eb1 P 197b880fa39d432a9b472845876bfaf0 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 974b802141bd4f13beffef2aa03f88f9 (127.18.211.2:34517), 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3:37735)
I20260129 03:00:10.567503 23032 raft_consensus.cc:515] T cb0b3f259ebc4310927338a8740e87b8 P 974b802141bd4f13beffef2aa03f88f9 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } }
I20260129 03:00:11.067909 20581 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35754 (request call id 1801) took 3458 ms. Trace:
I20260129 03:00:11.109226 20586 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4e3805ffa70242e8ade66da260dddf66" candidate_uuid: "197b880fa39d432a9b472845876bfaf0" candidate_term: 7 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "1c50de89a5a845668696dcb28c3badd6" is_pre_election: true
I20260129 03:00:11.109671 23346 raft_consensus.cc:493] T e4d18c35b0a7493e85d0a4b92956a48f P 197b880fa39d432a9b472845876bfaf0 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 974b802141bd4f13beffef2aa03f88f9)
I20260129 03:00:11.110212 20582 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "368953c9102643b2a21f99af36b51905" candidate_uuid: "197b880fa39d432a9b472845876bfaf0" candidate_term: 9 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "1c50de89a5a845668696dcb28c3badd6" is_pre_election: true
I20260129 03:00:11.154068 20582 raft_consensus.cc:2468] T 368953c9102643b2a21f99af36b51905 P 1c50de89a5a845668696dcb28c3badd6 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 197b880fa39d432a9b472845876bfaf0 in term 8.
I20260129 03:00:11.152776 20581 rpcz_store.cc:276] 0129 03:00:07.664271 (+ 0us) service_pool.cc:168] Inserting onto call queue
0129 03:00:07.685807 (+ 21536us) service_pool.cc:225] Handling call
0129 03:00:07.686165 (+ 358us) raft_consensus.cc:1471] Updating replica for 2 ops
0129 03:00:07.816421 (+130256us) raft_consensus.cc:1534] Early marking committed up to index 0
0129 03:00:07.816452 (+ 31us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0129 03:00:07.825115 (+ 8663us) log.cc:844] Serialized 52 byte log entry
0129 03:00:08.957123 (+1132008us) spinlock_profiling.cc:243] Waited 20 ms on lock 0x7b180049fcc0. stack: 00007f8fe044e373 00007f8fe044e270 00007f8fdfda0574 00007f8fe9a5c6b4 00007f8fe9a5c649 00007f8fe92d4c91 00007f8fe92e1671 00007f8fe0462c67 00007f8fe924b709 00007f8fe047f57b 00007f8fe047d81f 00007f8fe047fae2 00007f8fe30c75ab 00007f8fe30cd0c3 00007f8fe30cd471 00007f8fe34e973f
0129 03:00:09.927139 (+970016us) raft_consensus.cc:1645] Marking committed up to 3
0129 03:00:10.170533 (+243394us) log.cc:844] Serialized 14 byte log entry
0129 03:00:10.171573 (+ 1040us) log.cc:844] Serialized 14 byte log entry
0129 03:00:10.182158 (+ 10585us) raft_consensus.cc:1697] Filling consensus response to leader.
0129 03:00:10.182262 (+ 104us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0129 03:00:10.999512 (+817250us) raft_consensus.cc:1685] finished
0129 03:00:10.999566 (+ 54us) raft_consensus.cc:1691] UpdateReplicas() finished
0129 03:00:11.067784 (+ 68218us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":79424,"prepare.queue_time_us":413870,"prepare.run_cpu_time_us":55,"prepare.run_wall_time_us":57,"raft.queue_time_us":126508,"raft.run_cpu_time_us":578,"raft.run_wall_time_us":102166,"spinlock_wait_cycles":80128000,"thread_start_us":1130267,"threads_started":2,"wal-append.queue_time_us":2124045}
I20260129 03:00:11.155752 20582 raft_consensus.cc:1275] T cedce331e1584ac8b636c9b8a72b2ba0 P 1c50de89a5a845668696dcb28c3badd6 [term 8 FOLLOWER]: Refusing update from remote peer 197b880fa39d432a9b472845876bfaf0: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 8 index: 5. (index mismatch)
I20260129 03:00:11.156466 20582 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35754 (request call id 1819) took 1981 ms. Trace:
W20260129 03:00:11.068521 23046 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.829s user 0.011s sys 0.014s
I20260129 03:00:11.113494 23282 leader_election.cc:290] T 75b0b6e82ddc4f92b1fe1d94f9a0b73b P 1c50de89a5a845668696dcb28c3badd6 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 974b802141bd4f13beffef2aa03f88f9 (127.18.211.2:34517), 7a9daf5d42a84615a6383cfec9a10608 (127.18.211.4:40439)
I20260129 03:00:11.124568 21134 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "505580c93c694bbb8ea96fdf9b17ab46" candidate_uuid: "1c50de89a5a845668696dcb28c3badd6" candidate_term: 7 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "7a9daf5d42a84615a6383cfec9a10608"
I20260129 03:00:11.125327 20583 raft_consensus.cc:1275] T b189f08e16a74a11980cc55d99359a0a P 1c50de89a5a845668696dcb28c3badd6 [term 9 FOLLOWER]: Refusing update from remote peer 197b880fa39d432a9b472845876bfaf0: Log matching property violated. Preceding OpId in replica: term: 8 index: 4. Preceding OpId from leader: term: 9 index: 5. (index mismatch)
I20260129 03:00:11.125592 21129 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1299723a4476420ab02fd8bfd7aa0645" candidate_uuid: "1c50de89a5a845668696dcb28c3badd6" candidate_term: 10 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "7a9daf5d42a84615a6383cfec9a10608" is_pre_election: true
I20260129 03:00:11.126643 21133 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4f5a77e94fef4b16a2ba7b067c8b144a" candidate_uuid: "1c50de89a5a845668696dcb28c3badd6" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "7a9daf5d42a84615a6383cfec9a10608" is_pre_election: true
I20260129 03:00:11.128952 22932 leader_election.cc:290] T c934d31efd3f4e5195605ddc354d14e1 P 7a9daf5d42a84615a6383cfec9a10608 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 197b880fa39d432a9b472845876bfaf0 (127.18.211.1:42027), 974b802141bd4f13beffef2aa03f88f9 (127.18.211.2:34517)
I20260129 03:00:10.837167 23333 raft_consensus.cc:515] T a4c1226b987b468fa84dfc6ba0e2dbe1 P 974b802141bd4f13beffef2aa03f88f9 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 03:00:10.997648 23417 raft_consensus.cc:515] T 0678b8b468bd43cdbc3e485900844821 P 7a9daf5d42a84615a6383cfec9a10608 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } }
I20260129 03:00:11.141775 21127 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4e420f90884f4796997b5ea41e49e411" candidate_uuid: "974b802141bd4f13beffef2aa03f88f9" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "7a9daf5d42a84615a6383cfec9a10608" is_pre_election: true
I20260129 03:00:11.132694 23100 raft_consensus.cc:493] T 00e79d8f25a34676b1d866eb2068a2b2 P 7a9daf5d42a84615a6383cfec9a10608 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260129 03:00:10.883440 20589 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fcddab6bca684adf8ea032289a00f90e" candidate_uuid: "974b802141bd4f13beffef2aa03f88f9" candidate_term: 10 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "1c50de89a5a845668696dcb28c3badd6"
I20260129 03:00:11.193374 20589 raft_consensus.cc:3060] T fcddab6bca684adf8ea032289a00f90e P 1c50de89a5a845668696dcb28c3badd6 [term 9 FOLLOWER]: Advancing to term 10
I20260129 03:00:10.836885 23091 raft_consensus.cc:2749] T e33a417cf9c54cf5af73a4ee3aac4d3a P 197b880fa39d432a9b472845876bfaf0 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260129 03:00:11.146536 21130 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "78aa39096868415d9afc2ffcb248e198" candidate_uuid: "1c50de89a5a845668696dcb28c3badd6" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "7a9daf5d42a84615a6383cfec9a10608" is_pre_election: true
I20260129 03:00:11.199411 21134 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4e3805ffa70242e8ade66da260dddf66" candidate_uuid: "1c50de89a5a845668696dcb28c3badd6" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "7a9daf5d42a84615a6383cfec9a10608" is_pre_election: true
W20260129 03:00:11.147003 23425 log.cc:927] Time spent T 2ac659d732314e1288849f08fe5e074e P 197b880fa39d432a9b472845876bfaf0: Append to log took a long time: real 0.228s user 0.000s sys 0.000s
I20260129 03:00:10.994642 23428 raft_consensus.cc:3060] T 2ac659d732314e1288849f08fe5e074e P 197b880fa39d432a9b472845876bfaf0 [term 5 FOLLOWER]: Advancing to term 6
I20260129 03:00:11.148216 21132 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "45cc5f3d2688486d9b6dbe7e666dedde" candidate_uuid: "1c50de89a5a845668696dcb28c3badd6" candidate_term: 6 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "7a9daf5d42a84615a6383cfec9a10608" is_pre_election: true
I20260129 03:00:10.916450 23304 raft_consensus.cc:493] T 3906b080a5b34a74b8b64128635e8997 P 1c50de89a5a845668696dcb28c3badd6 [term 4 FOLLOWER]: Starting leader election (detected failure of leader 974b802141bd4f13beffef2aa03f88f9)
I20260129 03:00:11.147732 21135 raft_consensus.cc:1275] T 505580c93c694bbb8ea96fdf9b17ab46 P 7a9daf5d42a84615a6383cfec9a10608 [term 7 FOLLOWER]: Refusing update from remote peer 1c50de89a5a845668696dcb28c3badd6: Log matching property violated. Preceding OpId in replica: term: 6 index: 8. Preceding OpId from leader: term: 7 index: 9. (index mismatch)
W20260129 03:00:11.149505 20482 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260129 03:00:11.139487 20407 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.667s user 0.022s sys 0.051s
W20260129 03:00:11.217564 20407 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.773s user 0.022s sys 0.051s
I20260129 03:00:11.156889 20582 rpcz_store.cc:276] 0129 03:00:09.179110 (+ 0us) service_pool.cc:168] Inserting onto call queue
0129 03:00:11.154963 (+1975853us) service_pool.cc:225] Handling call
0129 03:00:11.155349 (+ 386us) raft_consensus.cc:1471] Updating replica for 0 ops
0129 03:00:11.156179 (+ 830us) raft_consensus.cc:1697] Filling consensus response to leader.
0129 03:00:11.156407 (+ 228us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260129 03:00:11.218638 21134 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "dc2f4cb752cf4d59b5f6bc689f5d8104" candidate_uuid: "1c50de89a5a845668696dcb28c3badd6" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "7a9daf5d42a84615a6383cfec9a10608" is_pre_election: true
I20260129 03:00:11.219187 20582 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35752 (request call id 1692) took 1977 ms. Trace:
I20260129 03:00:11.219246 21134 raft_consensus.cc:2468] T dc2f4cb752cf4d59b5f6bc689f5d8104 P 7a9daf5d42a84615a6383cfec9a10608 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1c50de89a5a845668696dcb28c3badd6 in term 5.
W20260129 03:00:11.110601 23418 log.cc:927] Time spent T 75b0b6e82ddc4f92b1fe1d94f9a0b73b P 7a9daf5d42a84615a6383cfec9a10608: Append to log took a long time: real 0.474s user 0.001s sys 0.000s
I20260129 03:00:11.153275 20586 raft_consensus.cc:2410] T 4e3805ffa70242e8ade66da260dddf66 P 1c50de89a5a845668696dcb28c3badd6 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 197b880fa39d432a9b472845876bfaf0 for term 7 because replica has last-logged OpId of term: 5 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 6.
I20260129 03:00:11.158120 20581 raft_consensus.cc:1240] T 724a236524cb4e7ba3428880681d6cc2 P 1c50de89a5a845668696dcb28c3badd6 [term 6 FOLLOWER]: Rejecting Update request from peer 974b802141bd4f13beffef2aa03f88f9 for earlier term 5. Current term is 6. Ops: []
I20260129 03:00:11.222643 20581 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35752 (request call id 1691) took 1981 ms. Trace:
I20260129 03:00:11.222999 20581 rpcz_store.cc:276] 0129 03:00:09.241123 (+ 0us) service_pool.cc:168] Inserting onto call queue
0129 03:00:11.155521 (+1914398us) service_pool.cc:225] Handling call
0129 03:00:11.157921 (+ 2400us) raft_consensus.cc:1471] Updating replica for 0 ops
0129 03:00:11.222424 (+ 64503us) raft_consensus.cc:1697] Filling consensus response to leader.
0129 03:00:11.222593 (+ 169us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20260129 03:00:11.162058 20556 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.211.2:34517, user_credentials={real_user=slave}} blocked reactor thread for 354650us
W20260129 03:00:11.152901 20513 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.559s user 0.015s sys 0.020s
W20260129 03:00:11.224005 20513 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.630s user 0.015s sys 0.020s
I20260129 03:00:11.224164 20405 leader_election.cc:304] T 4f8f3a38f6e24ba4857875181f447442 P 197b880fa39d432a9b472845876bfaf0 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 197b880fa39d432a9b472845876bfaf0, 1c50de89a5a845668696dcb28c3badd6; no voters: 974b802141bd4f13beffef2aa03f88f9
I20260129 03:00:11.044417 21128 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a644c02b16af47679824ccd060ac10fd" candidate_uuid: "974b802141bd4f13beffef2aa03f88f9" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "7a9daf5d42a84615a6383cfec9a10608" is_pre_election: true
I20260129 03:00:11.229404 20586 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:35754 (request call id 1824) took 1043 ms. Trace:
I20260129 03:00:11.173763 20583 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35754 (request call id 1818) took 2113 ms. Trace:
I20260129 03:00:11.230297 20583 rpcz_store.cc:276] 0129 03:00:09.060573 (+ 0us) service_pool.cc:168] Inserting onto call queue
0129 03:00:11.123489 (+2062916us) service_pool.cc:225] Handling call
0129 03:00:11.124064 (+ 575us) raft_consensus.cc:1471] Updating replica for 0 ops
0129 03:00:11.173487 (+ 49423us) raft_consensus.cc:1697] Filling consensus response to leader.
0129 03:00:11.173702 (+ 215us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260129 03:00:11.131603 23404 leader_election.cc:290] T 0340229d521b432580ae926f6062e3f7 P 197b880fa39d432a9b472845876bfaf0 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 974b802141bd4f13beffef2aa03f88f9 (127.18.211.2:34517), 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3:37735)
W20260129 03:00:11.184126 20440 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55672 (request call id 1712) took 3777 ms (client timeout 2078 ms). Trace:
I20260129 03:00:11.143751 22996 raft_consensus.cc:3055] T 76f713c3331540d2a5b76164103e9eb1 P 1c50de89a5a845668696dcb28c3badd6 [term 5 LEADER]: Stepping down as leader of term 5
I20260129 03:00:11.197044 19276 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260129 03:00:11.220108 23304 raft_consensus.cc:3060] T 3906b080a5b34a74b8b64128635e8997 P 1c50de89a5a845668696dcb28c3badd6 [term 4 FOLLOWER]: Advancing to term 5
I20260129 03:00:11.250371 21132 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "42d27a28d89b409e8807ec0efd97af69" candidate_uuid: "1c50de89a5a845668696dcb28c3badd6" candidate_term: 9 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "7a9daf5d42a84615a6383cfec9a10608" is_pre_election: true
I20260129 03:00:11.188711 23282 raft_consensus.cc:493] T add8d4923d1a489e86882c3a25692b95 P 1c50de89a5a845668696dcb28c3badd6 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260129 03:00:11.254033 23032 leader_election.cc:290] T cb0b3f259ebc4310927338a8740e87b8 P 974b802141bd4f13beffef2aa03f88f9 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3:37735), 7a9daf5d42a84615a6383cfec9a10608 (127.18.211.4:40439)
I20260129 03:00:11.219669 20582 rpcz_store.cc:276] 0129 03:00:09.241466 (+ 0us) service_pool.cc:168] Inserting onto call queue
0129 03:00:11.218438 (+1976972us) service_pool.cc:225] Handling call
0129 03:00:11.218706 (+ 268us) raft_consensus.cc:1471] Updating replica for 0 ops
0129 03:00:11.218900 (+ 194us) raft_consensus.cc:1534] Early marking committed up to index 0
0129 03:00:11.218925 (+ 25us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0129 03:00:11.218965 (+ 40us) raft_consensus.cc:1645] Marking committed up to 4
0129 03:00:11.218993 (+ 28us) raft_consensus.cc:1697] Filling consensus response to leader.
0129 03:00:11.219042 (+ 49us) raft_consensus.cc:1691] UpdateReplicas() finished
0129 03:00:11.219136 (+ 94us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260129 03:00:11.243552 21128 raft_consensus.cc:2468] T a644c02b16af47679824ccd060ac10fd P 7a9daf5d42a84615a6383cfec9a10608 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 974b802141bd4f13beffef2aa03f88f9 in term 5.
I20260129 03:00:11.255496 20582 raft_consensus.cc:1275] T 913fc7d751ed4b7388d889aac30a5f9b P 1c50de89a5a845668696dcb28c3badd6 [term 5 FOLLOWER]: Refusing update from remote peer 974b802141bd4f13beffef2aa03f88f9: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260129 03:00:11.216193 21130 raft_consensus.cc:2468] T 78aa39096868415d9afc2ffcb248e198 P 7a9daf5d42a84615a6383cfec9a10608 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1c50de89a5a845668696dcb28c3badd6 in term 7.
W20260129 03:00:11.217919 20407 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.211.2:34517, user_credentials={real_user=slave}} blocked reactor thread for 823325us
I20260129 03:00:11.229912 20586 rpcz_store.cc:276] 0129 03:00:10.186447 (+ 0us) service_pool.cc:168] Inserting onto call queue
0129 03:00:11.069140 (+882693us) service_pool.cc:225] Handling call
0129 03:00:11.229343 (+160203us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260129 03:00:11.258013 21134 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "18f30ebaa4174fd28c958e72d0cc7e24" candidate_uuid: "1c50de89a5a845668696dcb28c3badd6" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "7a9daf5d42a84615a6383cfec9a10608" is_pre_election: true
I20260129 03:00:11.232086 21133 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3e2d15f8640b407195210c476e09df91" candidate_uuid: "1c50de89a5a845668696dcb28c3badd6" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "7a9daf5d42a84615a6383cfec9a10608" is_pre_election: true
I20260129 03:00:11.184340 23346 raft_consensus.cc:515] T e4d18c35b0a7493e85d0a4b92956a48f P 197b880fa39d432a9b472845876bfaf0 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
W20260129 03:00:11.234793 20440 rpcz_store.cc:269] 0129 03:00:07.406808 (+ 0us) service_pool.cc:168] Inserting onto call queue
0129 03:00:07.447193 (+ 40385us) service_pool.cc:225] Handling call
0129 03:00:11.183784 (+3736591us) spinlock_profiling.cc:243] Waited 698 ms on lock 0x7b54002fd854. stack: 00007f8fe044e373 00007f8fe044e270 00007f8fdfda0574 00007f8fe9a5c6b4 00007f8fe9a5c649 00007f8fe34975a1 00007f8fe3531f7a 00007f8fe84e3647 00007f8fe1a73834 00007f8fe1a737c2 00007f8fe1a736f1 00007f8fe1a7366c 00007f8fe1a72922 00007f8fe186d34c 00007f8fe186c786 00007f8fe186c12f
0129 03:00:11.184026 (+ 242us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1605654912}
I20260129 03:00:11.153057 23205 consensus_queue.cc:260] T febffe4a8b394368b69b1d164308fb9a P 974b802141bd4f13beffef2aa03f88f9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 4.6, Last appended by leader: 6, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } }
I20260129 03:00:11.264760 20586 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:48526 (request call id 382) took 1804 ms. Trace:
I20260129 03:00:11.265286 20586 rpcz_store.cc:276] 0129 03:00:09.460591 (+ 0us) service_pool.cc:168] Inserting onto call queue
0129 03:00:11.258546 (+1797955us) service_pool.cc:225] Handling call
0129 03:00:11.264147 (+ 5601us) raft_consensus.cc:1471] Updating replica for 0 ops
0129 03:00:11.264359 (+ 212us) raft_consensus.cc:1534] Early marking committed up to index 0
0129 03:00:11.264399 (+ 40us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0129 03:00:11.264453 (+ 54us) raft_consensus.cc:1645] Marking committed up to 6
0129 03:00:11.264498 (+ 45us) raft_consensus.cc:1697] Filling consensus response to leader.
0129 03:00:11.264568 (+ 70us) raft_consensus.cc:1691] UpdateReplicas() finished
0129 03:00:11.264690 (+ 122us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260129 03:00:11.241962 21135 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "75f45538dc704d9a8ec4ff8ad9f3ea5b" candidate_uuid: "1c50de89a5a845668696dcb28c3badd6" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "7a9daf5d42a84615a6383cfec9a10608" is_pre_election: true
I20260129 03:00:11.216357 23431 raft_consensus.cc:493] T 724a236524cb4e7ba3428880681d6cc2 P 197b880fa39d432a9b472845876bfaf0 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 974b802141bd4f13beffef2aa03f88f9)
W20260129 03:00:11.228833 23298 consensus_meta.cc:287] T a5a05fbcbbcf4916bbc01fd1a5b5e90d P 1c50de89a5a845668696dcb28c3badd6: Time spent flushing consensus metadata: real 1.056s user 0.011s sys 0.001s
I20260129 03:00:11.249387 20581 raft_consensus.cc:1275] T 0340229d521b432580ae926f6062e3f7 P 1c50de89a5a845668696dcb28c3badd6 [term 7 FOLLOWER]: Refusing update from remote peer 974b802141bd4f13beffef2aa03f88f9: Log matching property violated. Preceding OpId in replica: term: 5 index: 4. Preceding OpId from leader: term: 7 index: 5. (index mismatch)
I20260129 03:00:11.234782 22694 leader_election.cc:290] T 17d6431cc77a491e8464d68db4081a51 P 974b802141bd4f13beffef2aa03f88f9 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 197b880fa39d432a9b472845876bfaf0 (127.18.211.1:42027), 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3:37735)
I20260129 03:00:11.268791 20581 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35752 (request call id 1694) took 2026 ms. Trace:
I20260129 03:00:11.269238 20581 rpcz_store.cc:276] 0129 03:00:09.242121 (+ 0us) service_pool.cc:168] Inserting onto call queue
0129 03:00:11.223661 (+1981540us) service_pool.cc:225] Handling call
0129 03:00:11.248929 (+ 25268us) raft_consensus.cc:1471] Updating replica for 0 ops
0129 03:00:11.268471 (+ 19542us) raft_consensus.cc:1697] Filling consensus response to leader.
0129 03:00:11.268722 (+ 251us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260129 03:00:11.242717 22996 raft_consensus.cc:740] T 76f713c3331540d2a5b76164103e9eb1 P 1c50de89a5a845668696dcb28c3badd6 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 1c50de89a5a845668696dcb28c3badd6, State: Running, Role: LEADER
I20260129 03:00:11.253849 21132 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "96609ac68bdf4993a12e1c23355ca013" candidate_uuid: "1c50de89a5a845668696dcb28c3badd6" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "7a9daf5d42a84615a6383cfec9a10608" is_pre_election: true
I20260129 03:00:11.256117 20582 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35752 (request call id 1702) took 1859 ms. Trace:
I20260129 03:00:11.271414 20582 rpcz_store.cc:276] 0129 03:00:09.397227 (+ 0us) service_pool.cc:168] Inserting onto call queue
0129 03:00:11.254940 (+1857713us) service_pool.cc:225] Handling call
0129 03:00:11.255220 (+ 280us) raft_consensus.cc:1471] Updating replica for 0 ops
0129 03:00:11.255844 (+ 624us) raft_consensus.cc:1697] Filling consensus response to leader.
0129 03:00:11.256065 (+ 221us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260129 03:00:11.200886 23100 raft_consensus.cc:515] T 00e79d8f25a34676b1d866eb2068a2b2 P 7a9daf5d42a84615a6383cfec9a10608 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } }
I20260129 03:00:11.232388 23437 raft_consensus.cc:2804] T 4f8f3a38f6e24ba4857875181f447442 P 197b880fa39d432a9b472845876bfaf0 [term 6 FOLLOWER]: Leader pre-election won for term 7
W20260129 03:00:11.267629 20512 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:57584 (request call id 1460) took 2014 ms (client timeout 1872 ms). Trace:
W20260129 03:00:11.275504 20512 rpcz_store.cc:269] 0129 03:00:09.253435 (+ 0us) service_pool.cc:168] Inserting onto call queue
0129 03:00:11.267437 (+2014002us) service_pool.cc:212] Skipping call since client already timed out
0129 03:00:11.267560 (+ 123us) inbound_call.cc:173] Queueing failure response
Metrics: {}
I20260129 03:00:11.233031 23091 consensus_queue.cc:1048] T 1ce4ddf5b10e4ae8b193b13b2e5936ee P 197b880fa39d432a9b472845876bfaf0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 1, Time since last communication: 0.031s
I20260129 03:00:11.280767 20581 raft_consensus.cc:1240] T cb0b3f259ebc4310927338a8740e87b8 P 1c50de89a5a845668696dcb28c3badd6 [term 8 FOLLOWER]: Rejecting Update request from peer 7a9daf5d42a84615a6383cfec9a10608 for earlier term 7. Current term is 8. Ops: []
I20260129 03:00:11.281699 20581 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:48526 (request call id 385) took 1819 ms. Trace:
I20260129 03:00:11.282334 20581 rpcz_store.cc:276] 0129 03:00:09.462787 (+ 0us) service_pool.cc:168] Inserting onto call queue
0129 03:00:11.270163 (+1807376us) service_pool.cc:225] Handling call
0129 03:00:11.280557 (+ 10394us) raft_consensus.cc:1471] Updating replica for 0 ops
0129 03:00:11.281404 (+ 847us) raft_consensus.cc:1697] Filling consensus response to leader.
0129 03:00:11.281639 (+ 235us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260129 03:00:11.283172 20407 leader_election.cc:304] T a609ac02fae84725ab743f95f51d33ce P 197b880fa39d432a9b472845876bfaf0 [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 197b880fa39d432a9b472845876bfaf0, 974b802141bd4f13beffef2aa03f88f9; no voters:
W20260129 03:00:11.208199 20480 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.211.4:40439, user_credentials={real_user=slave}} blocked reactor thread for 409824us
I20260129 03:00:11.288287 20553 leader_election.cc:304] T dc2f4cb752cf4d59b5f6bc689f5d8104 P 1c50de89a5a845668696dcb28c3badd6 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 197b880fa39d432a9b472845876bfaf0, 1c50de89a5a845668696dcb28c3badd6; no voters:
I20260129 03:00:11.288308 20480 leader_election.cc:304] T a609ac02fae84725ab743f95f51d33ce P 974b802141bd4f13beffef2aa03f88f9 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 974b802141bd4f13beffef2aa03f88f9; no voters: 197b880fa39d432a9b472845876bfaf0, 7a9daf5d42a84615a6383cfec9a10608
I20260129 03:00:11.225203 20405 leader_election.cc:304] T a1a39dcd328e4268ae41b988c66b4a75 P 197b880fa39d432a9b472845876bfaf0 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 197b880fa39d432a9b472845876bfaf0, 1c50de89a5a845668696dcb28c3badd6; no voters: 974b802141bd4f13beffef2aa03f88f9
I20260129 03:00:11.284773 20586 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:48526 (request call id 383) took 1823 ms. Trace:
I20260129 03:00:11.301771 20589 raft_consensus.cc:2468] T fcddab6bca684adf8ea032289a00f90e P 1c50de89a5a845668696dcb28c3badd6 [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 974b802141bd4f13beffef2aa03f88f9 in term 10.
I20260129 03:00:11.354410 20589 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:35752 (request call id 1721) took 1003 ms. Trace:
W20260129 03:00:11.357515 20405 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.211.3:37735, user_credentials={real_user=slave}} blocked reactor thread for 132349us
I20260129 03:00:11.359273 20405 leader_election.cc:304] T c2633a6fc34c44579de348d208b0c01e P 197b880fa39d432a9b472845876bfaf0 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 197b880fa39d432a9b472845876bfaf0; no voters: 1c50de89a5a845668696dcb28c3badd6, 974b802141bd4f13beffef2aa03f88f9
I20260129 03:00:11.281751 23091 consensus_queue.cc:1243] T 1ce4ddf5b10e4ae8b193b13b2e5936ee P 197b880fa39d432a9b472845876bfaf0 [LEADER]: Peer 1c50de89a5a845668696dcb28c3badd6 log is divergent from this leader: its last log entry 4.3 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 1
I20260129 03:00:11.313015 20586 rpcz_store.cc:276] 0129 03:00:09.461384 (+ 0us) service_pool.cc:168] Inserting onto call queue
0129 03:00:11.266382 (+1804998us) service_pool.cc:225] Handling call
0129 03:00:11.284084 (+ 17702us) raft_consensus.cc:1471] Updating replica for 0 ops
0129 03:00:11.284324 (+ 240us) raft_consensus.cc:1534] Early marking committed up to index 0
0129 03:00:11.284361 (+ 37us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0129 03:00:11.284440 (+ 79us) raft_consensus.cc:1645] Marking committed up to 5
0129 03:00:11.284489 (+ 49us) raft_consensus.cc:1697] Filling consensus response to leader.
0129 03:00:11.284572 (+ 83us) raft_consensus.cc:1691] UpdateReplicas() finished
0129 03:00:11.284709 (+ 137us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260129 03:00:11.254456 23282 raft_consensus.cc:515] T add8d4923d1a489e86882c3a25692b95 P 1c50de89a5a845668696dcb28c3badd6 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } }
W20260129 03:00:11.363407 20554 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.211.4:40439, user_credentials={real_user=slave}} blocked reactor thread for 141612us
W20260129 03:00:11.372736 20406 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260129 03:00:11.380923 21102 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260129 03:00:11.381935 20581 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35752 (request call id 1706) took 1774 ms. Trace:
W20260129 03:00:11.373345 20479 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.569s user 0.002s sys 0.011s
I20260129 03:00:11.382459 20581 rpcz_store.cc:276] 0129 03:00:09.607403 (+ 0us) service_pool.cc:168] Inserting onto call queue
0129 03:00:11.283419 (+1676016us) service_pool.cc:225] Handling call
0129 03:00:11.370543 (+ 87124us) raft_consensus.cc:1471] Updating replica for 0 ops
0129 03:00:11.381574 (+ 11031us) raft_consensus.cc:1534] Early marking committed up to index 0
0129 03:00:11.381610 (+ 36us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0129 03:00:11.381661 (+ 51us) raft_consensus.cc:1645] Marking committed up to 2
0129 03:00:11.381698 (+ 37us) raft_consensus.cc:1697] Filling consensus response to leader.
0129 03:00:11.381762 (+ 64us) raft_consensus.cc:1691] UpdateReplicas() finished
0129 03:00:11.381871 (+ 109us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":10607}
W20260129 03:00:11.383285 20479 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.579s user 0.003s sys 0.011s
W20260129 03:00:11.373364 20407 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.211.2:34517, user_credentials={real_user=slave}} blocked reactor thread for 57625.2us
I20260129 03:00:11.275182 23437 raft_consensus.cc:493] T 4f8f3a38f6e24ba4857875181f447442 P 197b880fa39d432a9b472845876bfaf0 [term 6 FOLLOWER]: Starting leader election (detected failure of leader 974b802141bd4f13beffef2aa03f88f9)
I20260129 03:00:11.360469 23087 raft_consensus.cc:2804] T a1a39dcd328e4268ae41b988c66b4a75 P 197b880fa39d432a9b472845876bfaf0 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260129 03:00:11.376961 20441 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:48356 (request call id 417) took 3156 ms. Trace:
I20260129 03:00:11.274430 23298 raft_consensus.cc:515] T a5a05fbcbbcf4916bbc01fd1a5b5e90d P 1c50de89a5a845668696dcb28c3badd6 [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 03:00:11.389858 20441 rpcz_store.cc:276] 0129 03:00:08.220330 (+ 0us) service_pool.cc:168] Inserting onto call queue
0129 03:00:08.236825 (+ 16495us) service_pool.cc:225] Handling call
0129 03:00:08.238094 (+ 1269us) raft_consensus.cc:1471] Updating replica for 0 ops
0129 03:00:09.472810 (+1234716us) raft_consensus.cc:1697] Filling consensus response to leader.
0129 03:00:11.376677 (+1903867us) spinlock_profiling.cc:243] Waited 1.17 s on lock 0x7b54002670d4. stack: 00007f8fe044e373 00007f8fe044e270 00007f8fdfda0574 00007f8fe9a5c6b4 00007f8fe9a5c649 00007f8fe34975a1 00007f8fe352ea81 00007f8fe352d4be 00007f8fe84e1f49 00007f8fe1a6ed34 00007f8fe1a6ecc2 00007f8fe1a6ebf1 00007f8fe1a6eb6c 00007f8fe1a6de22 00007f8fe186d34c 00007f8fe186c786
0129 03:00:11.376889 (+ 212us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2698685824}
I20260129 03:00:11.360554 21127 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "78aa39096868415d9afc2ffcb248e198" candidate_uuid: "1c50de89a5a845668696dcb28c3badd6" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "7a9daf5d42a84615a6383cfec9a10608"
W20260129 03:00:11.387306 20553 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.211.1:42027, user_credentials={real_user=slave}} blocked reactor thread for 99282us
I20260129 03:00:11.371541 23404 raft_consensus.cc:2804] T a609ac02fae84725ab743f95f51d33ce P 197b880fa39d432a9b472845876bfaf0 [term 9 FOLLOWER]: Leader election won for term 9
I20260129 03:00:11.397085 20407 leader_election.cc:304] T 87d5fe48a44a408aaf03e894a67ecb2f P 197b880fa39d432a9b472845876bfaf0 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 197b880fa39d432a9b472845876bfaf0, 974b802141bd4f13beffef2aa03f88f9; no voters:
I20260129 03:00:11.380823 23224 raft_consensus.cc:2749] T c2633a6fc34c44579de348d208b0c01e P 197b880fa39d432a9b472845876bfaf0 [term 5 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260129 03:00:11.386909 23437 raft_consensus.cc:3060] T 4f8f3a38f6e24ba4857875181f447442 P 197b880fa39d432a9b472845876bfaf0 [term 6 FOLLOWER]: Advancing to term 7
I20260129 03:00:11.354879 20589 rpcz_store.cc:276] 0129 03:00:10.350974 (+ 0us) service_pool.cc:168] Inserting onto call queue
0129 03:00:10.829817 (+478843us) service_pool.cc:225] Handling call
0129 03:00:11.354317 (+524500us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260129 03:00:11.400785 20586 raft_consensus.cc:1275] T 368953c9102643b2a21f99af36b51905 P 1c50de89a5a845668696dcb28c3badd6 [term 8 FOLLOWER]: Refusing update from remote peer 974b802141bd4f13beffef2aa03f88f9: Log matching property violated. Preceding OpId in replica: term: 6 index: 4. Preceding OpId from leader: term: 8 index: 5. (index mismatch)
I20260129 03:00:11.402506 21099 leader_election.cc:304] T dc2f4cb752cf4d59b5f6bc689f5d8104 P 7a9daf5d42a84615a6383cfec9a10608 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 197b880fa39d432a9b472845876bfaf0, 7a9daf5d42a84615a6383cfec9a10608; no voters: 1c50de89a5a845668696dcb28c3badd6
I20260129 03:00:11.409454 20586 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35752 (request call id 1708) took 1746 ms. Trace:
I20260129 03:00:11.392135 23087 raft_consensus.cc:493] T a1a39dcd328e4268ae41b988c66b4a75 P 197b880fa39d432a9b472845876bfaf0 [term 5 FOLLOWER]: Starting leader election (detected failure of leader 974b802141bd4f13beffef2aa03f88f9)
I20260129 03:00:11.405045 20588 raft_consensus.cc:1240] T 36ff33be666a4c82954731691dbcf327 P 1c50de89a5a845668696dcb28c3badd6 [term 4 LEADER]: Rejecting Update request from peer 974b802141bd4f13beffef2aa03f88f9 for earlier term 3. Current term is 4. Ops: []
I20260129 03:00:11.311363 23436 consensus_queue.cc:1048] T a0fcc58d6fb6473cafbd9d34d5195269 P 197b880fa39d432a9b472845876bfaf0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.114s
I20260129 03:00:11.409829 20586 rpcz_store.cc:276] 0129 03:00:09.663639 (+ 0us) service_pool.cc:168] Inserting onto call queue
0129 03:00:11.379664 (+1716025us) service_pool.cc:225] Handling call
0129 03:00:11.400394 (+ 20730us) raft_consensus.cc:1471] Updating replica for 0 ops
0129 03:00:11.409212 (+ 8818us) raft_consensus.cc:1697] Filling consensus response to leader.
0129 03:00:11.409397 (+ 185us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260129 03:00:11.422014 20515 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0340229d521b432580ae926f6062e3f7" candidate_uuid: "197b880fa39d432a9b472845876bfaf0" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "974b802141bd4f13beffef2aa03f88f9" is_pre_election: true
I20260129 03:00:11.423319 20512 raft_consensus.cc:1240] T c934d31efd3f4e5195605ddc354d14e1 P 974b802141bd4f13beffef2aa03f88f9 [term 6 FOLLOWER]: Rejecting Update request from peer 197b880fa39d432a9b472845876bfaf0 for earlier term 5. Current term is 6. Ops: []
I20260129 03:00:11.430365 20508 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "76f713c3331540d2a5b76164103e9eb1" candidate_uuid: "197b880fa39d432a9b472845876bfaf0" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "974b802141bd4f13beffef2aa03f88f9" is_pre_election: true
I20260129 03:00:11.402937 22996 consensus_queue.cc:260] T 76f713c3331540d2a5b76164103e9eb1 P 1c50de89a5a845668696dcb28c3badd6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 5.4, Last appended by leader: 4, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 03:00:11.448021 20436 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55672 (request call id 1742) took 1265 ms. Trace:
I20260129 03:00:11.404351 23333 leader_election.cc:290] T a4c1226b987b468fa84dfc6ba0e2dbe1 P 974b802141bd4f13beffef2aa03f88f9 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 197b880fa39d432a9b472845876bfaf0 (127.18.211.1:42027), 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3:37735)
I20260129 03:00:11.449863 20436 rpcz_store.cc:276] 0129 03:00:10.182898 (+ 0us) service_pool.cc:168] Inserting onto call queue
0129 03:00:10.835843 (+652945us) service_pool.cc:225] Handling call
0129 03:00:10.836210 (+ 367us) raft_consensus.cc:1471] Updating replica for 1 ops
0129 03:00:10.836453 (+ 243us) raft_consensus.cc:1534] Early marking committed up to index 0
0129 03:00:10.836480 (+ 27us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0129 03:00:10.836968 (+ 488us) log.cc:844] Serialized 26 byte log entry
0129 03:00:11.236173 (+399205us) raft_consensus.cc:1645] Marking committed up to 1
0129 03:00:11.236223 (+ 50us) raft_consensus.cc:1697] Filling consensus response to leader.
0129 03:00:11.236311 (+ 88us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0129 03:00:11.447647 (+211336us) raft_consensus.cc:1685] finished
0129 03:00:11.447681 (+ 34us) raft_consensus.cc:1691] UpdateReplicas() finished
0129 03:00:11.447914 (+ 233us) inbound_call.cc:173] Queueing success response
Metrics: {"thread_start_us":241672,"threads_started":1,"wal-append.queue_time_us":514498}
W20260129 03:00:11.456776 23432 log.cc:927] Time spent T cb4727386a3e48aca5d71cc6c611c197 P 197b880fa39d432a9b472845876bfaf0: Append to log took a long time: real 0.073s user 0.000s sys 0.000s
W20260129 03:00:11.461983 23215 consensus_meta.cc:287] T 0678b8b468bd43cdbc3e485900844821 P 974b802141bd4f13beffef2aa03f88f9: Time spent flushing consensus metadata: real 1.194s user 0.010s sys 0.001s
I20260129 03:00:11.462631 23215 raft_consensus.cc:515] T 0678b8b468bd43cdbc3e485900844821 P 974b802141bd4f13beffef2aa03f88f9 [term 9 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } }
I20260129 03:00:11.463683 23346 leader_election.cc:290] T e4d18c35b0a7493e85d0a4b92956a48f P 197b880fa39d432a9b472845876bfaf0 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 974b802141bd4f13beffef2aa03f88f9 (127.18.211.2:34517), 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3:37735)
I20260129 03:00:11.464334 23346 raft_consensus.cc:2804] T 87d5fe48a44a408aaf03e894a67ecb2f P 197b880fa39d432a9b472845876bfaf0 [term 7 FOLLOWER]: Leader election won for term 7
I20260129 03:00:11.464767 23346 raft_consensus.cc:697] T 87d5fe48a44a408aaf03e894a67ecb2f P 197b880fa39d432a9b472845876bfaf0 [term 7 LEADER]: Becoming Leader. State: Replica: 197b880fa39d432a9b472845876bfaf0, State: Running, Role: LEADER
I20260129 03:00:11.465593 23346 consensus_queue.cc:237] T 87d5fe48a44a408aaf03e894a67ecb2f P 197b880fa39d432a9b472845876bfaf0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 5.7, Last appended by leader: 7, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } }
I20260129 03:00:11.425266 23087 raft_consensus.cc:3060] T a1a39dcd328e4268ae41b988c66b4a75 P 197b880fa39d432a9b472845876bfaf0 [term 5 FOLLOWER]: Advancing to term 6
I20260129 03:00:11.448922 23439 consensus_queue.cc:1059] T a609ac02fae84725ab743f95f51d33ce P 7a9daf5d42a84615a6383cfec9a10608 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 }, Status: INVALID_TERM, Last received: 8.8, Next index: 9, Last known committed idx: 7, Time since last communication: 0.033s
I20260129 03:00:11.468433 23100 leader_election.cc:290] T 00e79d8f25a34676b1d866eb2068a2b2 P 7a9daf5d42a84615a6383cfec9a10608 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 197b880fa39d432a9b472845876bfaf0 (127.18.211.1:42027), 974b802141bd4f13beffef2aa03f88f9 (127.18.211.2:34517)
I20260129 03:00:11.311813 23431 raft_consensus.cc:515] T 724a236524cb4e7ba3428880681d6cc2 P 197b880fa39d432a9b472845876bfaf0 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 03:00:11.468977 23100 raft_consensus.cc:3055] T a609ac02fae84725ab743f95f51d33ce P 7a9daf5d42a84615a6383cfec9a10608 [term 8 LEADER]: Stepping down as leader of term 8
I20260129 03:00:11.469328 23100 raft_consensus.cc:740] T a609ac02fae84725ab743f95f51d33ce P 7a9daf5d42a84615a6383cfec9a10608 [term 8 LEADER]: Becoming Follower/Learner. State: Replica: 7a9daf5d42a84615a6383cfec9a10608, State: Running, Role: LEADER
I20260129 03:00:11.470453 23431 leader_election.cc:290] T 724a236524cb4e7ba3428880681d6cc2 P 197b880fa39d432a9b472845876bfaf0 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 974b802141bd4f13beffef2aa03f88f9 (127.18.211.2:34517), 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3:37735)
I20260129 03:00:11.471365 23431 raft_consensus.cc:493] T 1299723a4476420ab02fd8bfd7aa0645 P 197b880fa39d432a9b472845876bfaf0 [term 10 FOLLOWER]: Starting pre-election (detected failure of leader 7a9daf5d42a84615a6383cfec9a10608)
I20260129 03:00:11.406280 23458 raft_consensus.cc:493] T 7a9a69cd82724b1faed1d10aa091fc4a P 1c50de89a5a845668696dcb28c3badd6 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 197b880fa39d432a9b472845876bfaf0)
I20260129 03:00:11.470137 23100 consensus_queue.cc:260] T a609ac02fae84725ab743f95f51d33ce P 7a9daf5d42a84615a6383cfec9a10608 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 8.8, Last appended by leader: 8, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } }
I20260129 03:00:11.473618 20438 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55676 (request call id 1472) took 1356 ms. Trace:
I20260129 03:00:11.479136 20438 rpcz_store.cc:276] 0129 03:00:10.117404 (+ 0us) service_pool.cc:168] Inserting onto call queue
0129 03:00:10.123111 (+ 5707us) service_pool.cc:225] Handling call
0129 03:00:10.123470 (+ 359us) raft_consensus.cc:1471] Updating replica for 1 ops
0129 03:00:10.136022 (+ 12552us) raft_consensus.cc:1534] Early marking committed up to index 0
0129 03:00:10.136066 (+ 44us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0129 03:00:10.146792 (+ 10726us) log.cc:844] Serialized 26 byte log entry
0129 03:00:10.472086 (+325294us) raft_consensus.cc:1645] Marking committed up to 1
0129 03:00:10.472142 (+ 56us) raft_consensus.cc:1697] Filling consensus response to leader.
0129 03:00:10.472238 (+ 96us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0129 03:00:11.473310 (+1001072us) raft_consensus.cc:1685] finished
0129 03:00:11.473340 (+ 30us) raft_consensus.cc:1691] UpdateReplicas() finished
0129 03:00:11.473535 (+ 195us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":30984,"raft.queue_time_us":12385,"raft.run_cpu_time_us":180,"raft.run_wall_time_us":32081,"thread_start_us":270682,"threads_started":1,"wal-append.queue_time_us":1083875}
W20260129 03:00:11.448271 23434 log.cc:927] Time spent T 75be4bd6bd45474795fd472eff2953ac P 197b880fa39d432a9b472845876bfaf0: Append to log took a long time: real 0.223s user 0.001s sys 0.000s
I20260129 03:00:11.423552 23141 raft_consensus.cc:2804] T dc2f4cb752cf4d59b5f6bc689f5d8104 P 7a9daf5d42a84615a6383cfec9a10608 [term 5 FOLLOWER]: Leader pre-election won for term 6
W20260129 03:00:11.523337 20406 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260129 03:00:11.430784 23417 leader_election.cc:290] T 0678b8b468bd43cdbc3e485900844821 P 7a9daf5d42a84615a6383cfec9a10608 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 197b880fa39d432a9b472845876bfaf0 (127.18.211.1:42027), 974b802141bd4f13beffef2aa03f88f9 (127.18.211.2:34517)
W20260129 03:00:11.481662 23435 log.cc:927] Time spent T b189f08e16a74a11980cc55d99359a0a P 197b880fa39d432a9b472845876bfaf0: Append to log took a long time: real 0.239s user 0.001s sys 0.001s
I20260129 03:00:11.471851 23282 leader_election.cc:290] T add8d4923d1a489e86882c3a25692b95 P 1c50de89a5a845668696dcb28c3badd6 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 974b802141bd4f13beffef2aa03f88f9 (127.18.211.2:34517), 7a9daf5d42a84615a6383cfec9a10608 (127.18.211.4:40439)
I20260129 03:00:11.529279 20508 tablet_service.cc:2066] Received LeaderStepDown RPC: tablet_id: "4f8f3a38f6e24ba4857875181f447442"
dest_uuid: "974b802141bd4f13beffef2aa03f88f9"
mode: GRACEFUL
new_leader_uuid: "1c50de89a5a845668696dcb28c3badd6"
from {username='slave'} at 127.0.0.1:52008
I20260129 03:00:11.476908 22891 raft_consensus.cc:493] T 913fc7d751ed4b7388d889aac30a5f9b P 1c50de89a5a845668696dcb28c3badd6 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260129 03:00:11.448900 23447 raft_consensus.cc:2804] T dc2f4cb752cf4d59b5f6bc689f5d8104 P 1c50de89a5a845668696dcb28c3badd6 [term 6 FOLLOWER]: Leader election won for term 6
I20260129 03:00:11.430516 23456 raft_consensus.cc:493] T a1a39dcd328e4268ae41b988c66b4a75 P 1c50de89a5a845668696dcb28c3badd6 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 974b802141bd4f13beffef2aa03f88f9)
I20260129 03:00:11.401436 23460 consensus_queue.cc:1048] T a644c02b16af47679824ccd060ac10fd P 1c50de89a5a845668696dcb28c3badd6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.085s
I20260129 03:00:11.537005 20508 raft_consensus.cc:606] T 4f8f3a38f6e24ba4857875181f447442 P 974b802141bd4f13beffef2aa03f88f9 [term 6 LEADER]: Received request to transfer leadership to TS 1c50de89a5a845668696dcb28c3badd6
I20260129 03:00:11.537427 20433 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55672 (request call id 1744) took 1269 ms. Trace:
I20260129 03:00:11.531215 23404 raft_consensus.cc:697] T a609ac02fae84725ab743f95f51d33ce P 197b880fa39d432a9b472845876bfaf0 [term 9 LEADER]: Becoming Leader. State: Replica: 197b880fa39d432a9b472845876bfaf0, State: Running, Role: LEADER
W20260129 03:00:11.536352 20553 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.211.1:42027, user_credentials={real_user=slave}} blocked reactor thread for 139683us
I20260129 03:00:11.537803 20584 raft_consensus.cc:1240] T e33a417cf9c54cf5af73a4ee3aac4d3a P 1c50de89a5a845668696dcb28c3badd6 [term 7 FOLLOWER]: Rejecting Update request from peer 974b802141bd4f13beffef2aa03f88f9 for earlier term 6. Current term is 7. Ops: []
W20260129 03:00:11.541572 20587 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:35754 (request call id 1788) took 4410 ms (client timeout 1859 ms). Trace:
W20260129 03:00:11.634636 20587 rpcz_store.cc:269] 0129 03:00:07.127419 (+ 0us) service_pool.cc:168] Inserting onto call queue
0129 03:00:07.127575 (+ 156us) service_pool.cc:225] Handling call
0129 03:00:11.537466 (+4409891us) spinlock_profiling.cc:243] Waited 1.26 s on lock 0x7b54000c9254. stack: 00007f8fe044e373 00007f8fe044e270 00007f8fdfda0574 00007f8fe9a5c6b4 00007f8fe9a5c649 00007f8fe34975a1 00007f8fe3531f7a 00007f8fe84e3647 00007f8fe1a73834 00007f8fe1a737c2 00007f8fe1a736f1 00007f8fe1a7366c 00007f8fe1a72922 00007f8fe186d34c 00007f8fe186c786 00007f8fe186c12f
0129 03:00:11.537601 (+ 135us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2906190080}
I20260129 03:00:11.493716 23141 raft_consensus.cc:493] T dc2f4cb752cf4d59b5f6bc689f5d8104 P 7a9daf5d42a84615a6383cfec9a10608 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260129 03:00:11.431768 23224 consensus_queue.cc:1048] T 75be4bd6bd45474795fd472eff2953ac P 197b880fa39d432a9b472845876bfaf0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.104s
I20260129 03:00:11.448644 22932 consensus_queue.cc:1048] T 4e3805ffa70242e8ade66da260dddf66 P 7a9daf5d42a84615a6383cfec9a10608 [LEADER]: Connected to new peer: Peer: permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 6, Time since last communication: 0.084s
W20260129 03:00:11.566758 20480 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.211.4:40439, user_credentials={real_user=slave}} blocked reactor thread for 278564us
I20260129 03:00:11.620422 20433 rpcz_store.cc:276] 0129 03:00:10.268206 (+ 0us) service_pool.cc:168] Inserting onto call queue
0129 03:00:10.844855 (+576649us) service_pool.cc:225] Handling call
0129 03:00:10.995252 (+150397us) raft_consensus.cc:1471] Updating replica for 0 ops
0129 03:00:10.995458 (+ 206us) raft_consensus.cc:1534] Early marking committed up to index 5
0129 03:00:11.270684 (+275226us) log.cc:844] Serialized 14 byte log entry
0129 03:00:11.536230 (+265546us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0129 03:00:11.536326 (+ 96us) raft_consensus.cc:1645] Marking committed up to 5
0129 03:00:11.536370 (+ 44us) raft_consensus.cc:1697] Filling consensus response to leader.
0129 03:00:11.536440 (+ 70us) raft_consensus.cc:1691] UpdateReplicas() finished
0129 03:00:11.537339 (+ 899us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":2630,"prepare.queue_time_us":261901,"prepare.run_cpu_time_us":56,"prepare.run_wall_time_us":3988,"raft.queue_time_us":72016,"raft.run_cpu_time_us":91,"raft.run_wall_time_us":91,"thread_start_us":400773,"threads_started":3,"wal-append.queue_time_us":223506}
I20260129 03:00:11.641441 20438 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a5a05fbcbbcf4916bbc01fd1a5b5e90d" candidate_uuid: "1c50de89a5a845668696dcb28c3badd6" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "197b880fa39d432a9b472845876bfaf0"
I20260129 03:00:11.567255 20479 leader_election.cc:304] T c2633a6fc34c44579de348d208b0c01e P 974b802141bd4f13beffef2aa03f88f9 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 197b880fa39d432a9b472845876bfaf0, 974b802141bd4f13beffef2aa03f88f9; no voters:
I20260129 03:00:11.539222 23100 raft_consensus.cc:3060] T a609ac02fae84725ab743f95f51d33ce P 7a9daf5d42a84615a6383cfec9a10608 [term 8 FOLLOWER]: Advancing to term 9
I20260129 03:00:11.568951 23298 leader_election.cc:290] T a5a05fbcbbcf4916bbc01fd1a5b5e90d P 1c50de89a5a845668696dcb28c3badd6 [CANDIDATE]: Term 8 election: Requested vote from peers 197b880fa39d432a9b472845876bfaf0 (127.18.211.1:42027), 974b802141bd4f13beffef2aa03f88f9 (127.18.211.2:34517)
I20260129 03:00:11.583444 22694 raft_consensus.cc:2749] T a609ac02fae84725ab743f95f51d33ce P 974b802141bd4f13beffef2aa03f88f9 [term 9 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260129 03:00:11.572922 23215 leader_election.cc:290] T 0678b8b468bd43cdbc3e485900844821 P 974b802141bd4f13beffef2aa03f88f9 [CANDIDATE]: Term 9 election: Requested vote from peers 197b880fa39d432a9b472845876bfaf0 (127.18.211.1:42027), 7a9daf5d42a84615a6383cfec9a10608 (127.18.211.4:40439)
I20260129 03:00:11.575568 20509 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57600 (request call id 1943) took 1317 ms. Trace:
I20260129 03:00:11.578289 23443 raft_consensus.cc:493] T 1299723a4476420ab02fd8bfd7aa0645 P 974b802141bd4f13beffef2aa03f88f9 [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260129 03:00:11.650009 23141 raft_consensus.cc:3060] T dc2f4cb752cf4d59b5f6bc689f5d8104 P 7a9daf5d42a84615a6383cfec9a10608 [term 5 FOLLOWER]: Advancing to term 6
I20260129 03:00:11.579902 20513 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57600 (request call id 1946) took 1236 ms. Trace:
I20260129 03:00:11.684870 20508 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3e2d15f8640b407195210c476e09df91" candidate_uuid: "1c50de89a5a845668696dcb28c3badd6" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "974b802141bd4f13beffef2aa03f88f9" is_pre_election: true
I20260129 03:00:11.580184 20511 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "45cc5f3d2688486d9b6dbe7e666dedde" candidate_uuid: "1c50de89a5a845668696dcb28c3badd6" candidate_term: 6 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "974b802141bd4f13beffef2aa03f88f9" is_pre_election: true
I20260129 03:00:11.686345 20587 raft_consensus.cc:1275] T 4f5a77e94fef4b16a2ba7b067c8b144a P 1c50de89a5a845668696dcb28c3badd6 [term 5 FOLLOWER]: Refusing update from remote peer 7a9daf5d42a84615a6383cfec9a10608: Log matching property violated. Preceding OpId in replica: term: 3 index: 5. Preceding OpId from leader: term: 5 index: 6. (index mismatch)
I20260129 03:00:11.686964 20511 raft_consensus.cc:2393] T 45cc5f3d2688486d9b6dbe7e666dedde P 974b802141bd4f13beffef2aa03f88f9 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1c50de89a5a845668696dcb28c3badd6 in current term 6: Already voted for candidate 7a9daf5d42a84615a6383cfec9a10608 in this term.
I20260129 03:00:11.581053 20512 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a4c1226b987b468fa84dfc6ba0e2dbe1" candidate_uuid: "1c50de89a5a845668696dcb28c3badd6" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "974b802141bd4f13beffef2aa03f88f9" is_pre_election: true
I20260129 03:00:11.581470 23208 raft_consensus.cc:493] T 00e79d8f25a34676b1d866eb2068a2b2 P 974b802141bd4f13beffef2aa03f88f9 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 197b880fa39d432a9b472845876bfaf0)
I20260129 03:00:11.581761 20515 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4f5a77e94fef4b16a2ba7b067c8b144a" candidate_uuid: "1c50de89a5a845668696dcb28c3badd6" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "974b802141bd4f13beffef2aa03f88f9" is_pre_election: true
I20260129 03:00:11.582561 20510 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b189f08e16a74a11980cc55d99359a0a" candidate_uuid: "1c50de89a5a845668696dcb28c3badd6" candidate_term: 10 candidate_status { last_received { term: 8 index: 4 } } ignore_live_leader: false dest_uuid: "974b802141bd4f13beffef2aa03f88f9" is_pre_election: true
I20260129 03:00:11.485016 23458 raft_consensus.cc:515] T 7a9a69cd82724b1faed1d10aa091fc4a P 1c50de89a5a845668696dcb28c3badd6 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 03:00:11.586122 23436 raft_consensus.cc:493] T 0678b8b468bd43cdbc3e485900844821 P 197b880fa39d432a9b472845876bfaf0 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260129 03:00:11.588160 23203 consensus_meta.cc:287] T 18f30ebaa4174fd28c958e72d0cc7e24 P 7a9daf5d42a84615a6383cfec9a10608: Time spent flushing consensus metadata: real 1.372s user 0.009s sys 0.003s
I20260129 03:00:11.696920 20441 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c934d31efd3f4e5195605ddc354d14e1" candidate_uuid: "7a9daf5d42a84615a6383cfec9a10608" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "197b880fa39d432a9b472845876bfaf0" is_pre_election: true
W20260129 03:00:11.624326 20482 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260129 03:00:11.471833 23431 raft_consensus.cc:515] T 1299723a4476420ab02fd8bfd7aa0645 P 197b880fa39d432a9b472845876bfaf0 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } }
I20260129 03:00:11.654696 20438 raft_consensus.cc:3060] T a5a05fbcbbcf4916bbc01fd1a5b5e90d P 197b880fa39d432a9b472845876bfaf0 [term 7 FOLLOWER]: Advancing to term 8
W20260129 03:00:11.655156 20479 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.211.1:42027, user_credentials={real_user=slave}} blocked reactor thread for 87954.4us
I20260129 03:00:11.583424 23087 raft_consensus.cc:515] T a1a39dcd328e4268ae41b988c66b4a75 P 197b880fa39d432a9b472845876bfaf0 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } } peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } }
W20260129 03:00:11.657900 20480 connection.cc:232] RPC call timeout handler was delayed by -1.8697591999999759s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.19230000000000003s for a response
I20260129 03:00:11.644677 20436 raft_consensus.cc:1240] T c2633a6fc34c44579de348d208b0c01e P 197b880fa39d432a9b472845876bfaf0 [term 5 FOLLOWER]: Rejecting Update request from peer 1c50de89a5a845668696dcb28c3badd6 for earlier term 4. Current term is 5. Ops: []
W20260129 03:00:11.572220 23444 log.cc:927] Time spent T 00e79d8f25a34676b1d866eb2068a2b2 P 974b802141bd4f13beffef2aa03f88f9: Append to log took a long time: real 0.180s user 0.000s sys 0.001s
I20260129 03:00:11.679754 20509 rpcz_store.cc:276] 0129 03:00:10.258190 (+ 0us) service_pool.cc:168] Inserting onto call queue
0129 03:00:10.260565 (+ 2375us) service_pool.cc:225] Handling call
0129 03:00:10.285425 (+ 24860us) raft_consensus.cc:1471] Updating replica for 0 ops
0129 03:00:10.285705 (+ 280us) raft_consensus.cc:1534] Early marking committed up to index 9
0129 03:00:10.833699 (+547994us) log.cc:844] Serialized 14 byte log entry
0129 03:00:11.185595 (+351896us) log.cc:844] Serialized 14 byte log entry
0129 03:00:11.483385 (+297790us) spinlock_profiling.cc:243] Waited 4.61 ms on lock 0x7b18004fb220. stack: 00007f8fe044e373 00007f8fe044e270 00007f8fdfda0574 00007f8fe9a5c6b4 00007f8fe9a5c649 00007f8fe92d4c91 00007f8fe92e1671 00007f8fe03f546a 00007f8fe9a7b5c9 00007f8fe02b5160 00007f8fe047d3c4 00007f8fe047d335 00007f8fe3525ce0 00007f8fe3538396 00007f8fe3548f07 00007f8fe3548e88
0129 03:00:11.483857 (+ 472us) log.cc:844] Serialized 14 byte log entry
0129 03:00:11.575121 (+ 91264us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0129 03:00:11.575176 (+ 55us) raft_consensus.cc:1645] Marking committed up to 9
0129 03:00:11.575222 (+ 46us) raft_consensus.cc:1697] Filling consensus response to leader.
0129 03:00:11.575300 (+ 78us) raft_consensus.cc:1691] UpdateReplicas() finished
0129 03:00:11.575500 (+ 200us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":505089,"prepare.queue_time_us":840255,"prepare.run_cpu_time_us":32,"prepare.run_wall_time_us":33,"raft.queue_time_us":392773,"raft.run_cpu_time_us":433,"raft.run_wall_time_us":16427,"spinlock_wait_cycles":33599488,"thread_start_us":875547,"threads_started":2,"wal-append.queue_time_us":477982}
I20260129 03:00:11.678597 21102 leader_election.cc:304] T febffe4a8b394368b69b1d164308fb9a P 7a9daf5d42a84615a6383cfec9a10608 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7a9daf5d42a84615a6383cfec9a10608; no voters: 197b880fa39d432a9b472845876bfaf0, 974b802141bd4f13beffef2aa03f88f9
I20260129 03:00:11.706771 20587 raft_consensus.cc:1275] T 45cc5f3d2688486d9b6dbe7e666dedde P 1c50de89a5a845668696dcb28c3badd6 [term 6 FOLLOWER]: Refusing update from remote peer 7a9daf5d42a84615a6383cfec9a10608: Log matching property violated. Preceding OpId in replica: term: 3 index: 5. Preceding OpId from leader: term: 6 index: 6. (index mismatch)
I20260129 03:00:11.588536 21127 raft_consensus.cc:2468] T 78aa39096868415d9afc2ffcb248e198 P 7a9daf5d42a84615a6383cfec9a10608 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1c50de89a5a845668696dcb28c3badd6 in term 7.
I20260129 03:00:11.685189 20513 rpcz_store.cc:276] 0129 03:00:10.343741 (+ 0us) service_pool.cc:168] Inserting onto call queue
0129 03:00:10.480566 (+136825us) service_pool.cc:225] Handling call
0129 03:00:10.480998 (+ 432us) raft_consensus.cc:1471] Updating replica for 0 ops
0129 03:00:10.481274 (+ 276us) raft_consensus.cc:1534] Early marking committed up to index 5
0129 03:00:11.224364 (+743090us) log.cc:844] Serialized 14 byte log entry
0129 03:00:11.579290 (+354926us) log.cc:844] Serialized 14 byte log entry
0129 03:00:11.579469 (+ 179us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0129 03:00:11.579521 (+ 52us) raft_consensus.cc:1645] Marking committed up to 5
0129 03:00:11.579582 (+ 61us) raft_consensus.cc:1697] Filling consensus response to leader.
0129 03:00:11.579667 (+ 85us) raft_consensus.cc:1691] UpdateReplicas() finished
0129 03:00:11.579814 (+ 147us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":187384,"prepare.queue_time_us":649266,"prepare.run_cpu_time_us":81,"prepare.run_wall_time_us":3533,"raft.queue_time_us":190206,"raft.run_cpu_time_us":160,"raft.run_wall_time_us":162,"thread_start_us":797242,"threads_started":2}
I20260129 03:00:11.689440 20586 raft_consensus.cc:1275] T add8d4923d1a489e86882c3a25692b95 P 1c50de89a5a845668696dcb28c3badd6 [term 7 FOLLOWER]: Refusing update from remote peer 7a9daf5d42a84615a6383cfec9a10608: Log matching property violated. Preceding OpId in replica: term: 5 index: 7. Preceding OpId from leader: term: 7 index: 6. (term mismatch)
I20260129 03:00:11.708196 20586 pending_rounds.cc:85] T add8d4923d1a489e86882c3a25692b95 P 1c50de89a5a845668696dcb28c3badd6: Aborting all ops after (but not including) 5
I20260129 03:00:11.708424 20586 pending_rounds.cc:107] T add8d4923d1a489e86882c3a25692b95 P 1c50de89a5a845668696dcb28c3badd6: Aborting uncommitted NO_OP operation due to leader change: 3.6
I20260129 03:00:11.708624 20586 raft_consensus.cc:2889] T add8d4923d1a489e86882c3a25692b95 P 1c50de89a5a845668696dcb28c3badd6 [term 7 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260129 03:00:11.708907 20586 pending_rounds.cc:107] T add8d4923d1a489e86882c3a25692b95 P 1c50de89a5a845668696dcb28c3badd6: Aborting uncommitted NO_OP operation due to leader change: 5.7
I20260129 03:00:11.709160 20586 raft_consensus.cc:2889] T add8d4923d1a489e86882c3a25692b95 P 1c50de89a5a845668696dcb28c3badd6 [term 7 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260129 03:00:11.633867 23447 raft_consensus.cc:697] T dc2f4cb752cf4d59b5f6bc689f5d8104 P 1c50de89a5a845668696dcb28c3badd6 [term 6 LEADER]: Becoming Leader. State: Replica: 1c50de89a5a845668696dcb28c3badd6, State: Running, Role: LEADER
W20260129 03:00:11.694908 20405 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.211.3:37735, user_credentials={real_user=slave}} blocked reactor thread for 316273us
I20260129 03:00:11.699179 20515 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "724a236524cb4e7ba3428880681d6cc2" candidate_uuid: "1c50de89a5a845668696dcb28c3badd6" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "974b802141bd4f13beffef2aa03f88f9"
I20260129 03:00:11.564253 22891 raft_consensus.cc:515] T 913fc7d751ed4b7388d889aac30a5f9b P 1c50de89a5a845668696dcb28c3badd6 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 03:00:11.701869 20479 leader_election.cc:304] T fcddab6bca684adf8ea032289a00f90e P 974b802141bd4f13beffef2aa03f88f9 [CANDIDATE]: Term 10 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 197b880fa39d432a9b472845876bfaf0, 974b802141bd4f13beffef2aa03f88f9; no voters:
W20260129 03:00:11.743992 20480 connection.cc:232] RPC call timeout handler was delayed by -1.7760776000000078s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1769s for a response
W20260129 03:00:11.735078 21102 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.211.2:34517, user_credentials={real_user=slave}} blocked reactor thread for 56577.2us
W20260129 03:00:11.744798 20480 connection.cc:232] RPC call timeout handler was delayed by -1.7285402779999686s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2014s for a response
I20260129 03:00:11.696274 23443 raft_consensus.cc:515] T 1299723a4476420ab02fd8bfd7aa0645 P 974b802141bd4f13beffef2aa03f88f9 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } attrs { replace: true } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } }
I20260129 03:00:11.701449 23208 raft_consensus.cc:515] T 00e79d8f25a34676b1d866eb2068a2b2 P 974b802141bd4f13beffef2aa03f88f9 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } }
W20260129 03:00:11.745317 20480 leader_election.cc:336] T b189f08e16a74a11980cc55d99359a0a P 974b802141bd4f13beffef2aa03f88f9 [CANDIDATE]: Term 10 pre-election: RPC error from VoteRequest() call to peer 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3:37735): Timed out: RequestConsensusVote RPC to 127.18.211.3:37735 timed out after 1.681s (SENT)
I20260129 03:00:11.733738 20510 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "926a250fa5834e88b96ccecbefbf5086" candidate_uuid: "1c50de89a5a845668696dcb28c3badd6" candidate_term: 8 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "974b802141bd4f13beffef2aa03f88f9" is_pre_election: true
I20260129 03:00:11.746821 20480 leader_election.cc:304] T b189f08e16a74a11980cc55d99359a0a P 974b802141bd4f13beffef2aa03f88f9 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 974b802141bd4f13beffef2aa03f88f9; no voters: 197b880fa39d432a9b472845876bfaf0, 1c50de89a5a845668696dcb28c3badd6
I20260129 03:00:11.677274 23449 raft_consensus.cc:2804] T c2633a6fc34c44579de348d208b0c01e P 974b802141bd4f13beffef2aa03f88f9 [term 5 FOLLOWER]: Leader election won for term 5
I20260129 03:00:11.710409 20513 raft_consensus.cc:1275] T cb4727386a3e48aca5d71cc6c611c197 P 974b802141bd4f13beffef2aa03f88f9 [term 5 FOLLOWER]: Refusing update from remote peer 1c50de89a5a845668696dcb28c3badd6: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 5 index: 2. (index mismatch)
I20260129 03:00:11.746452 20405 leader_election.cc:304] T 926a250fa5834e88b96ccecbefbf5086 P 197b880fa39d432a9b472845876bfaf0 [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 197b880fa39d432a9b472845876bfaf0, 1c50de89a5a845668696dcb28c3badd6; no voters: 974b802141bd4f13beffef2aa03f88f9
I20260129 03:00:11.750087 23447 consensus_queue.cc:237] T dc2f4cb752cf4d59b5f6bc689f5d8104 P 1c50de89a5a845668696dcb28c3badd6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } }
I20260129 03:00:11.703816 23203 raft_consensus.cc:515] T 18f30ebaa4174fd28c958e72d0cc7e24 P 7a9daf5d42a84615a6383cfec9a10608 [term 8 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } }
I20260129 03:00:11.754205 21134 raft_consensus.cc:2393] T 18f30ebaa4174fd28c958e72d0cc7e24 P 7a9daf5d42a84615a6383cfec9a10608 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1c50de89a5a845668696dcb28c3badd6 in current term 8: Already voted for candidate 7a9daf5d42a84615a6383cfec9a10608 in this term.
W20260129 03:00:11.700223 23389 consensus_meta.cc:287] T 8055232cf0004d40a85f3bf983fe00a9 P 197b880fa39d432a9b472845876bfaf0: Time spent flushing consensus metadata: real 1.510s user 0.012s sys 0.000s
I20260129 03:00:11.756798 23203 leader_election.cc:290] T 18f30ebaa4174fd28c958e72d0cc7e24 P 7a9daf5d42a84615a6383cfec9a10608 [CANDIDATE]: Term 8 election: Requested vote from peers 197b880fa39d432a9b472845876bfaf0 (127.18.211.1:42027), 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3:37735)
I20260129 03:00:11.710474 23436 raft_consensus.cc:515] T 0678b8b468bd43cdbc3e485900844821 P 197b880fa39d432a9b472845876bfaf0 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } }
I20260129 03:00:11.734474 20509 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e33a417cf9c54cf5af73a4ee3aac4d3a" candidate_uuid: "1c50de89a5a845668696dcb28c3badd6" candidate_term: 7 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "974b802141bd4f13beffef2aa03f88f9"
I20260129 03:00:11.710697 23215 raft_consensus.cc:493] T add8d4923d1a489e86882c3a25692b95 P 974b802141bd4f13beffef2aa03f88f9 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260129 03:00:11.747435 20510 raft_consensus.cc:2410] T 926a250fa5834e88b96ccecbefbf5086 P 974b802141bd4f13beffef2aa03f88f9 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1c50de89a5a845668696dcb28c3badd6 for term 8 because replica has last-logged OpId of term: 6 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 4 index: 3.
I20260129 03:00:11.760572 20511 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "75f45538dc704d9a8ec4ff8ad9f3ea5b" candidate_uuid: "1c50de89a5a845668696dcb28c3badd6" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "974b802141bd4f13beffef2aa03f88f9" is_pre_election: true
I20260129 03:00:11.761286 20512 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e4d18c35b0a7493e85d0a4b92956a48f" candidate_uuid: "1c50de89a5a845668696dcb28c3badd6" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "974b802141bd4f13beffef2aa03f88f9" is_pre_election: true
I20260129 03:00:11.761934 20508 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "cedce331e1584ac8b636c9b8a72b2ba0" candidate_uuid: "1c50de89a5a845668696dcb28c3badd6" candidate_term: 8 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "974b802141bd4f13beffef2aa03f88f9" is_pre_election: true
I20260129 03:00:11.762019 20405 leader_election.cc:304] T e7d83d4ccf684e76acb29a0dc844e1b8 P 197b880fa39d432a9b472845876bfaf0 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 197b880fa39d432a9b472845876bfaf0; no voters: 1c50de89a5a845668696dcb28c3badd6, 974b802141bd4f13beffef2aa03f88f9
I20260129 03:00:11.747431 23443 leader_election.cc:290] T 1299723a4476420ab02fd8bfd7aa0645 P 974b802141bd4f13beffef2aa03f88f9 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 197b880fa39d432a9b472845876bfaf0 (127.18.211.1:42027), 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3:37735), 7a9daf5d42a84615a6383cfec9a10608 (127.18.211.4:40439)
I20260129 03:00:11.762125 20513 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "368953c9102643b2a21f99af36b51905" candidate_uuid: "1c50de89a5a845668696dcb28c3badd6" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "974b802141bd4f13beffef2aa03f88f9"
I20260129 03:00:11.777454 20508 raft_consensus.cc:2393] T cedce331e1584ac8b636c9b8a72b2ba0 P 974b802141bd4f13beffef2aa03f88f9 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1c50de89a5a845668696dcb28c3badd6 in current term 8: Already voted for candidate 197b880fa39d432a9b472845876bfaf0 in this term.
I20260129 03:00:11.762576 23449 raft_consensus.cc:697] T c2633a6fc34c44579de348d208b0c01e P 974b802141bd4f13beffef2aa03f88f9 [term 5 LEADER]: Becoming Leader. State: Replica: 974b802141bd4f13beffef2aa03f88f9, State: Running, Role: LEADER
W20260129 03:00:11.764448 20480 leader_election.cc:336] T 75be4bd6bd45474795fd472eff2953ac P 974b802141bd4f13beffef2aa03f88f9 [CANDIDATE]: Term 11 pre-election: RPC error from VoteRequest() call to peer 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3:37735): Timed out: RequestConsensusVote RPC to 127.18.211.3:37735 timed out after 1.689s (SENT)
I20260129 03:00:11.756402 22891 leader_election.cc:290] T 913fc7d751ed4b7388d889aac30a5f9b P 1c50de89a5a845668696dcb28c3badd6 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 197b880fa39d432a9b472845876bfaf0 (127.18.211.1:42027), 974b802141bd4f13beffef2aa03f88f9 (127.18.211.2:34517)
I20260129 03:00:11.681368 23035 consensus_queue.cc:1048] T 2ac659d732314e1288849f08fe5e074e P 7a9daf5d42a84615a6383cfec9a10608 [LEADER]: Connected to new peer: Peer: permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.053s
W20260129 03:00:11.758740 20479 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.211.1:42027, user_credentials={real_user=slave}} blocked reactor thread for 56934.5us
I20260129 03:00:11.779217 20480 leader_election.cc:304] T 75be4bd6bd45474795fd472eff2953ac P 974b802141bd4f13beffef2aa03f88f9 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 974b802141bd4f13beffef2aa03f88f9; no voters: 197b880fa39d432a9b472845876bfaf0, 1c50de89a5a845668696dcb28c3badd6
W20260129 03:00:11.789918 20480 leader_election.cc:336] T e7d83d4ccf684e76acb29a0dc844e1b8 P 974b802141bd4f13beffef2aa03f88f9 [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3:37735): Timed out: RequestConsensusVote RPC to 127.18.211.3:37735 timed out after 1.744s (SENT)
I20260129 03:00:11.790033 20433 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0678b8b468bd43cdbc3e485900844821" candidate_uuid: "7a9daf5d42a84615a6383cfec9a10608" candidate_term: 9 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "197b880fa39d432a9b472845876bfaf0" is_pre_election: true
I20260129 03:00:11.633134 23456 raft_consensus.cc:515] T a1a39dcd328e4268ae41b988c66b4a75 P 1c50de89a5a845668696dcb28c3badd6 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } } peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } }
I20260129 03:00:11.793426 20405 leader_election.cc:304] T fcddab6bca684adf8ea032289a00f90e P 197b880fa39d432a9b472845876bfaf0 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 197b880fa39d432a9b472845876bfaf0, 1c50de89a5a845668696dcb28c3badd6; no voters: 974b802141bd4f13beffef2aa03f88f9
I20260129 03:00:11.701123 23304 raft_consensus.cc:515] T 3906b080a5b34a74b8b64128635e8997 P 1c50de89a5a845668696dcb28c3badd6 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 03:00:11.759414 23389 raft_consensus.cc:515] T 8055232cf0004d40a85f3bf983fe00a9 P 197b880fa39d432a9b472845876bfaf0 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 03:00:11.791981 20440 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00e79d8f25a34676b1d866eb2068a2b2" candidate_uuid: "7a9daf5d42a84615a6383cfec9a10608" candidate_term: 9 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "197b880fa39d432a9b472845876bfaf0" is_pre_election: true
W20260129 03:00:11.787952 21100 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.211.3:37735, user_credentials={real_user=slave}} blocked reactor thread for 100992us
W20260129 03:00:11.791013 20480 leader_election.cc:336] T 4b1876f9a70b49edb3aa74105e66d3b3 P 974b802141bd4f13beffef2aa03f88f9 [CANDIDATE]: Term 6 election: RPC error from VoteRequest() call to peer 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3:37735): Timed out: RequestConsensusVote RPC to 127.18.211.3:37735 timed out after 1.769s (SENT)
I20260129 03:00:11.803247 20480 leader_election.cc:304] T 4b1876f9a70b49edb3aa74105e66d3b3 P 974b802141bd4f13beffef2aa03f88f9 [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 974b802141bd4f13beffef2aa03f88f9; no voters: 197b880fa39d432a9b472845876bfaf0, 1c50de89a5a845668696dcb28c3badd6
I20260129 03:00:11.804008 20405 leader_election.cc:304] T 368953c9102643b2a21f99af36b51905 P 197b880fa39d432a9b472845876bfaf0 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 197b880fa39d432a9b472845876bfaf0, 1c50de89a5a845668696dcb28c3badd6; no voters: 974b802141bd4f13beffef2aa03f88f9
W20260129 03:00:11.794052 21102 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.211.2:34517, user_credentials={real_user=slave}} blocked reactor thread for 46659us
I20260129 03:00:11.789117 20436 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "18f30ebaa4174fd28c958e72d0cc7e24" candidate_uuid: "7a9daf5d42a84615a6383cfec9a10608" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "197b880fa39d432a9b472845876bfaf0"
I20260129 03:00:11.797943 20512 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1ce4ddf5b10e4ae8b193b13b2e5936ee" candidate_uuid: "1c50de89a5a845668696dcb28c3badd6" candidate_term: 8 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "974b802141bd4f13beffef2aa03f88f9" is_pre_election: true
I20260129 03:00:11.767666 23450 raft_consensus.cc:2804] T fcddab6bca684adf8ea032289a00f90e P 974b802141bd4f13beffef2aa03f88f9 [term 10 FOLLOWER]: Leader election won for term 10
I20260129 03:00:11.768169 22645 raft_consensus.cc:2804] T 926a250fa5834e88b96ccecbefbf5086 P 197b880fa39d432a9b472845876bfaf0 [term 8 FOLLOWER]: Leader election won for term 8
I20260129 03:00:11.791302 20433 raft_consensus.cc:2468] T 0678b8b468bd43cdbc3e485900844821 P 197b880fa39d432a9b472845876bfaf0 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7a9daf5d42a84615a6383cfec9a10608 in term 8.
I20260129 03:00:11.780815 23439 raft_consensus.cc:3060] T febffe4a8b394368b69b1d164308fb9a P 7a9daf5d42a84615a6383cfec9a10608 [term 4 FOLLOWER]: Advancing to term 5
I20260129 03:00:11.804911 20586 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "18f30ebaa4174fd28c958e72d0cc7e24" candidate_uuid: "7a9daf5d42a84615a6383cfec9a10608" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "1c50de89a5a845668696dcb28c3badd6"
I20260129 03:00:11.805063 20513 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "724a236524cb4e7ba3428880681d6cc2" candidate_uuid: "197b880fa39d432a9b472845876bfaf0" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "974b802141bd4f13beffef2aa03f88f9" is_pre_election: true
I20260129 03:00:11.805078 20436 raft_consensus.cc:3060] T 18f30ebaa4174fd28c958e72d0cc7e24 P 197b880fa39d432a9b472845876bfaf0 [term 7 FOLLOWER]: Advancing to term 8
I20260129 03:00:11.793030 23443 raft_consensus.cc:2749] T 75be4bd6bd45474795fd472eff2953ac P 974b802141bd4f13beffef2aa03f88f9 [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
W20260129 03:00:11.801997 23428 consensus_meta.cc:287] T 2ac659d732314e1288849f08fe5e074e P 197b880fa39d432a9b472845876bfaf0: Time spent flushing consensus metadata: real 0.574s user 0.006s sys 0.004s
I20260129 03:00:11.812287 20510 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e7d83d4ccf684e76acb29a0dc844e1b8" candidate_uuid: "1c50de89a5a845668696dcb28c3badd6" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "974b802141bd4f13beffef2aa03f88f9" is_pre_election: true
I20260129 03:00:11.813347 20583 raft_consensus.cc:1240] T 3906b080a5b34a74b8b64128635e8997 P 1c50de89a5a845668696dcb28c3badd6 [term 5 FOLLOWER]: Rejecting Update request from peer 974b802141bd4f13beffef2aa03f88f9 for earlier term 4. Current term is 5. Ops: []
I20260129 03:00:11.781811 23215 raft_consensus.cc:515] T add8d4923d1a489e86882c3a25692b95 P 974b802141bd4f13beffef2aa03f88f9 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } }
I20260129 03:00:11.803553 23344 raft_consensus.cc:2764] T fcddab6bca684adf8ea032289a00f90e P 197b880fa39d432a9b472845876bfaf0 [term 10 FOLLOWER]: Leader pre-election decision vote started in defunct term 9: won
I20260129 03:00:11.786675 23466 consensus_queue.cc:1048] T 0f20af7fbe5d4db89a6f17948d98912c P 197b880fa39d432a9b472845876bfaf0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 3, Time since last communication: 0.039s
I20260129 03:00:11.839073 20512 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "96609ac68bdf4993a12e1c23355ca013" candidate_uuid: "1c50de89a5a845668696dcb28c3badd6" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "974b802141bd4f13beffef2aa03f88f9" is_pre_election: true
W20260129 03:00:11.846796 21099 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.211.1:42027, user_credentials={real_user=slave}} blocked reactor thread for 44852us
I20260129 03:00:11.851735 20512 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "36ff33be666a4c82954731691dbcf327" candidate_uuid: "1c50de89a5a845668696dcb28c3badd6" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "974b802141bd4f13beffef2aa03f88f9"
I20260129 03:00:11.780390 23404 consensus_queue.cc:237] T a609ac02fae84725ab743f95f51d33ce P 197b880fa39d432a9b472845876bfaf0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 8.8, Last appended by leader: 8, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } }
I20260129 03:00:11.830093 22989 raft_consensus.cc:2749] T 4b1876f9a70b49edb3aa74105e66d3b3 P 974b802141bd4f13beffef2aa03f88f9 [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20260129 03:00:11.809015 23427 consensus_queue.cc:1048] T 368953c9102643b2a21f99af36b51905 P 974b802141bd4f13beffef2aa03f88f9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.049s
W20260129 03:00:11.885521 20405 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.211.3:37735, user_credentials={real_user=slave}} blocked reactor thread for 81702.4us
W20260129 03:00:11.870157 23398 consensus_meta.cc:287] T 18f30ebaa4174fd28c958e72d0cc7e24 P 1c50de89a5a845668696dcb28c3badd6: Time spent flushing consensus metadata: real 1.753s user 0.006s sys 0.008s
W20260129 03:00:11.911341 20406 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260129 03:00:11.829850 23428 raft_consensus.cc:515] T 2ac659d732314e1288849f08fe5e074e P 197b880fa39d432a9b472845876bfaf0 [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } }
W20260129 03:00:11.917492 23455 log.cc:927] Time spent T 0f20af7fbe5d4db89a6f17948d98912c P 974b802141bd4f13beffef2aa03f88f9: Append to log took a long time: real 0.127s user 0.001s sys 0.000s
I20260129 03:00:11.917708 23428 leader_election.cc:290] T 2ac659d732314e1288849f08fe5e074e P 197b880fa39d432a9b472845876bfaf0 [CANDIDATE]: Term 6 election: Requested vote from peers 974b802141bd4f13beffef2aa03f88f9 (127.18.211.2:34517), 7a9daf5d42a84615a6383cfec9a10608 (127.18.211.4:40439)
W20260129 03:00:11.918740 23453 log.cc:927] Time spent T a1a39dcd328e4268ae41b988c66b4a75 P 197b880fa39d432a9b472845876bfaf0: Append to log took a long time: real 0.217s user 0.000s sys 0.001s
I20260129 03:00:11.908174 23087 leader_election.cc:290] T a1a39dcd328e4268ae41b988c66b4a75 P 197b880fa39d432a9b472845876bfaf0 [CANDIDATE]: Term 6 election: Requested vote from peers 974b802141bd4f13beffef2aa03f88f9 (127.18.211.2:34517), 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3:37735)
W20260129 03:00:11.920408 23477 log.cc:927] Time spent T 1299723a4476420ab02fd8bfd7aa0645 P 1c50de89a5a845668696dcb28c3badd6: Append to log took a long time: real 0.119s user 0.000s sys 0.001s
I20260129 03:00:11.920526 23087 consensus_queue.cc:1048] T cedce331e1584ac8b636c9b8a72b2ba0 P 197b880fa39d432a9b472845876bfaf0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 3, Time since last communication: 0.001s
I20260129 03:00:11.812258 23474 consensus_queue.cc:1059] T c2633a6fc34c44579de348d208b0c01e P 1c50de89a5a845668696dcb28c3badd6 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 }, Status: INVALID_TERM, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.078s
I20260129 03:00:11.922751 20589 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:48526 (request call id 391) took 1716 ms. Trace:
I20260129 03:00:11.930364 20589 rpcz_store.cc:276] 0129 03:00:10.206185 (+ 0us) service_pool.cc:168] Inserting onto call queue
0129 03:00:11.408066 (+1201881us) service_pool.cc:225] Handling call
0129 03:00:11.408362 (+ 296us) raft_consensus.cc:1471] Updating replica for 1 ops
0129 03:00:11.408554 (+ 192us) raft_consensus.cc:1534] Early marking committed up to index 0
0129 03:00:11.408577 (+ 23us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0129 03:00:11.409018 (+ 441us) log.cc:844] Serialized 26 byte log entry
0129 03:00:11.688834 (+279816us) raft_consensus.cc:1645] Marking committed up to 8
0129 03:00:11.688887 (+ 53us) raft_consensus.cc:1697] Filling consensus response to leader.
0129 03:00:11.688991 (+ 104us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0129 03:00:11.922509 (+233518us) raft_consensus.cc:1685] finished
0129 03:00:11.922532 (+ 23us) raft_consensus.cc:1691] UpdateReplicas() finished
0129 03:00:11.922689 (+ 157us) inbound_call.cc:173] Queueing success response
Metrics: {"thread_start_us":265326,"threads_started":1,"wal-append.queue_time_us":341054}
W20260129 03:00:11.928275 23472 log.cc:927] Time spent T 75b0b6e82ddc4f92b1fe1d94f9a0b73b P 1c50de89a5a845668696dcb28c3badd6: Append to log took a long time: real 0.091s user 0.001s sys 0.000s
I20260129 03:00:11.890829 22645 raft_consensus.cc:697] T 926a250fa5834e88b96ccecbefbf5086 P 197b880fa39d432a9b472845876bfaf0 [term 8 LEADER]: Becoming Leader. State: Replica: 197b880fa39d432a9b472845876bfaf0, State: Running, Role: LEADER
I20260129 03:00:11.941378 23473 consensus_queue.cc:1059] T c934d31efd3f4e5195605ddc354d14e1 P 197b880fa39d432a9b472845876bfaf0 [NON_LEADER]: Peer responded invalid term: Peer: permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 }, Status: INVALID_TERM, Last received: 5.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.031s
I20260129 03:00:11.908818 23485 raft_consensus.cc:2804] T 368953c9102643b2a21f99af36b51905 P 197b880fa39d432a9b472845876bfaf0 [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260129 03:00:11.876174 23203 consensus_queue.cc:1048] T 4f5a77e94fef4b16a2ba7b067c8b144a P 7a9daf5d42a84615a6383cfec9a10608 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.056s
I20260129 03:00:11.956430 20587 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a5a05fbcbbcf4916bbc01fd1a5b5e90d" candidate_uuid: "197b880fa39d432a9b472845876bfaf0" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "1c50de89a5a845668696dcb28c3badd6" is_pre_election: true
I20260129 03:00:11.955688 20432 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55672 (request call id 1745) took 1618 ms. Trace:
I20260129 03:00:11.955760 20436 raft_consensus.cc:2468] T 18f30ebaa4174fd28c958e72d0cc7e24 P 197b880fa39d432a9b472845876bfaf0 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7a9daf5d42a84615a6383cfec9a10608 in term 8.
I20260129 03:00:11.968708 20589 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "42d27a28d89b409e8807ec0efd97af69" candidate_uuid: "197b880fa39d432a9b472845876bfaf0" candidate_term: 9 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "1c50de89a5a845668696dcb28c3badd6" is_pre_election: true
I20260129 03:00:11.980623 20436 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "87d5fe48a44a408aaf03e894a67ecb2f" candidate_uuid: "974b802141bd4f13beffef2aa03f88f9" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "197b880fa39d432a9b472845876bfaf0"
I20260129 03:00:11.968869 20587 raft_consensus.cc:2393] T a5a05fbcbbcf4916bbc01fd1a5b5e90d P 1c50de89a5a845668696dcb28c3badd6 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 197b880fa39d432a9b472845876bfaf0 in current term 8: Already voted for candidate 1c50de89a5a845668696dcb28c3badd6 in this term.
I20260129 03:00:11.932474 23431 leader_election.cc:290] T 1299723a4476420ab02fd8bfd7aa0645 P 197b880fa39d432a9b472845876bfaf0 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3:37735), 7a9daf5d42a84615a6383cfec9a10608 (127.18.211.4:40439)
W20260129 03:00:11.970407 20480 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.211.3:37735, user_credentials={real_user=slave}} blocked reactor thread for 161857us
I20260129 03:00:11.976358 20581 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:48526 (request call id 390) took 1779 ms. Trace:
W20260129 03:00:11.978121 20407 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.211.2:34517, user_credentials={real_user=slave}} blocked reactor thread for 145235us
I20260129 03:00:11.989889 20587 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b189f08e16a74a11980cc55d99359a0a" candidate_uuid: "197b880fa39d432a9b472845876bfaf0" candidate_term: 9 candidate_status { last_received { term: 8 index: 4 } } ignore_live_leader: false dest_uuid: "1c50de89a5a845668696dcb28c3badd6"
I20260129 03:00:11.996554 20589 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "724a236524cb4e7ba3428880681d6cc2" candidate_uuid: "197b880fa39d432a9b472845876bfaf0" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "1c50de89a5a845668696dcb28c3badd6" is_pre_election: true
I20260129 03:00:11.989534 22645 consensus_queue.cc:237] T 926a250fa5834e88b96ccecbefbf5086 P 197b880fa39d432a9b472845876bfaf0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.4, Last appended by leader: 4, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 03:00:11.889019 23449 consensus_queue.cc:237] T c2633a6fc34c44579de348d208b0c01e P 974b802141bd4f13beffef2aa03f88f9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 03:00:11.990684 23389 leader_election.cc:290] T 8055232cf0004d40a85f3bf983fe00a9 P 197b880fa39d432a9b472845876bfaf0 [CANDIDATE]: Term 6 election: Requested vote from peers 974b802141bd4f13beffef2aa03f88f9 (127.18.211.2:34517), 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3:37735)
I20260129 03:00:11.936038 23344 consensus_queue.cc:1048] T b189f08e16a74a11980cc55d99359a0a P 197b880fa39d432a9b472845876bfaf0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.049s
I20260129 03:00:11.912742 23398 raft_consensus.cc:515] T 18f30ebaa4174fd28c958e72d0cc7e24 P 1c50de89a5a845668696dcb28c3badd6 [term 8 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } }
I20260129 03:00:11.968324 23475 raft_consensus.cc:2749] T e7d83d4ccf684e76acb29a0dc844e1b8 P 197b880fa39d432a9b472845876bfaf0 [term 7 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260129 03:00:11.994457 23141 raft_consensus.cc:515] T dc2f4cb752cf4d59b5f6bc689f5d8104 P 7a9daf5d42a84615a6383cfec9a10608 [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } }
I20260129 03:00:11.997304 20589 raft_consensus.cc:2393] T 724a236524cb4e7ba3428880681d6cc2 P 1c50de89a5a845668696dcb28c3badd6 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 197b880fa39d432a9b472845876bfaf0 in current term 6: Already voted for candidate 1c50de89a5a845668696dcb28c3badd6 in this term.
W20260129 03:00:11.983681 20480 connection.cc:232] RPC call timeout handler was delayed by -1.6606822039999543s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.3295s for a response
I20260129 03:00:11.985523 20510 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0f20af7fbe5d4db89a6f17948d98912c" candidate_uuid: "1c50de89a5a845668696dcb28c3badd6" candidate_term: 8 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "974b802141bd4f13beffef2aa03f88f9" is_pre_election: true
W20260129 03:00:12.010134 20480 connection.cc:232] RPC call timeout handler was delayed by -1.6408856379999861s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.20270000000000002s for a response
W20260129 03:00:11.977520 21102 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.211.2:34517, user_credentials={real_user=slave}} blocked reactor thread for 171352us
I20260129 03:00:12.011561 20510 raft_consensus.cc:2410] T 0f20af7fbe5d4db89a6f17948d98912c P 974b802141bd4f13beffef2aa03f88f9 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1c50de89a5a845668696dcb28c3badd6 for term 8 because replica has last-logged OpId of term: 7 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 4 index: 3.
W20260129 03:00:12.012799 20480 connection.cc:232] RPC call timeout handler was delayed by -1.5891299229998594s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2069s for a response
I20260129 03:00:11.982661 20436 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a4c1226b987b468fa84dfc6ba0e2dbe1" candidate_uuid: "974b802141bd4f13beffef2aa03f88f9" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "197b880fa39d432a9b472845876bfaf0" is_pre_election: true
I20260129 03:00:11.985476 20581 rpcz_store.cc:276] 0129 03:00:10.197071 (+ 0us) service_pool.cc:168] Inserting onto call queue
0129 03:00:11.383493 (+1186422us) service_pool.cc:225] Handling call
0129 03:00:11.395833 (+ 12340us) raft_consensus.cc:1471] Updating replica for 0 ops
0129 03:00:11.396092 (+ 259us) raft_consensus.cc:1534] Early marking committed up to index 9
0129 03:00:11.482613 (+ 86521us) log.cc:844] Serialized 14 byte log entry
0129 03:00:11.975778 (+493165us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0129 03:00:11.975885 (+ 107us) raft_consensus.cc:1645] Marking committed up to 9
0129 03:00:11.975957 (+ 72us) raft_consensus.cc:1697] Filling consensus response to leader.
0129 03:00:11.976083 (+ 126us) raft_consensus.cc:1691] UpdateReplicas() finished
0129 03:00:11.976278 (+ 195us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":184711,"prepare.run_cpu_time_us":31,"prepare.run_wall_time_us":32,"raft.queue_time_us":3559,"thread_start_us":570956,"threads_started":2,"wal-append.queue_time_us":490627}
W20260129 03:00:11.999382 20553 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.211.1:42027, user_credentials={real_user=slave}} blocked reactor thread for 287580us
I20260129 03:00:11.999229 20438 raft_consensus.cc:2468] T a5a05fbcbbcf4916bbc01fd1a5b5e90d P 197b880fa39d432a9b472845876bfaf0 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1c50de89a5a845668696dcb28c3badd6 in term 8.
I20260129 03:00:11.969132 20432 rpcz_store.cc:276] 0129 03:00:10.337740 (+ 0us) service_pool.cc:168] Inserting onto call queue
0129 03:00:10.919349 (+581609us) service_pool.cc:225] Handling call
0129 03:00:11.109007 (+189658us) raft_consensus.cc:1471] Updating replica for 1 ops
0129 03:00:11.144514 (+ 35507us) raft_consensus.cc:1534] Early marking committed up to index 0
0129 03:00:11.144560 (+ 46us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0129 03:00:11.154077 (+ 9517us) log.cc:844] Serialized 26 byte log entry
0129 03:00:11.288549 (+134472us) raft_consensus.cc:1645] Marking committed up to 2
0129 03:00:11.288606 (+ 57us) raft_consensus.cc:1697] Filling consensus response to leader.
0129 03:00:11.288698 (+ 92us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0129 03:00:11.955390 (+666692us) raft_consensus.cc:1685] finished
0129 03:00:11.955418 (+ 28us) raft_consensus.cc:1691] UpdateReplicas() finished
0129 03:00:11.955610 (+ 192us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":50436,"raft.queue_time_us":35143,"raft.run_cpu_time_us":209,"raft.run_wall_time_us":23921,"thread_start_us":81811,"threads_started":1,"wal-append.queue_time_us":375853}
I20260129 03:00:11.978588 23485 raft_consensus.cc:493] T 368953c9102643b2a21f99af36b51905 P 197b880fa39d432a9b472845876bfaf0 [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260129 03:00:11.996711 20587 raft_consensus.cc:2376] T b189f08e16a74a11980cc55d99359a0a P 1c50de89a5a845668696dcb28c3badd6 [term 9 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 197b880fa39d432a9b472845876bfaf0 in term 9. Re-sending same reply.
I20260129 03:00:12.002215 20512 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c891d2bff4ce4568861bbc992e91fbb0" candidate_uuid: "1c50de89a5a845668696dcb28c3badd6" candidate_term: 9 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "974b802141bd4f13beffef2aa03f88f9" is_pre_election: true
W20260129 03:00:12.004873 20514 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.528s user 0.019s sys 0.014s
I20260129 03:00:12.017664 20440 raft_consensus.cc:1240] T a5a05fbcbbcf4916bbc01fd1a5b5e90d P 197b880fa39d432a9b472845876bfaf0 [term 8 FOLLOWER]: Rejecting Update request from peer 974b802141bd4f13beffef2aa03f88f9 for earlier term 7. Current term is 8. Ops: []
I20260129 03:00:12.018913 20438 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0678b8b468bd43cdbc3e485900844821" candidate_uuid: "974b802141bd4f13beffef2aa03f88f9" candidate_term: 9 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "197b880fa39d432a9b472845876bfaf0"
I20260129 03:00:12.020159 20436 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1299723a4476420ab02fd8bfd7aa0645" candidate_uuid: "974b802141bd4f13beffef2aa03f88f9" candidate_term: 11 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "197b880fa39d432a9b472845876bfaf0" is_pre_election: true
I20260129 03:00:12.023087 20438 raft_consensus.cc:3060] T 0678b8b468bd43cdbc3e485900844821 P 197b880fa39d432a9b472845876bfaf0 [term 8 FOLLOWER]: Advancing to term 9
I20260129 03:00:12.023278 20436 raft_consensus.cc:1813] T 1299723a4476420ab02fd8bfd7aa0645 P 197b880fa39d432a9b472845876bfaf0 [term 10 FOLLOWER]: Handling vote request from an unknown peer 974b802141bd4f13beffef2aa03f88f9
W20260129 03:00:12.023777 20480 connection.cc:232] RPC call timeout handler was delayed by -1.5284854029998769s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2136s for a response
W20260129 03:00:12.023555 20514 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.547s user 0.019s sys 0.014s
W20260129 03:00:12.024325 20480 connection.cc:232] RPC call timeout handler was delayed by -1.4122373720000496s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2938s for a response
W20260129 03:00:12.024773 20480 connection.cc:232] RPC call timeout handler was delayed by -1.2054556339999181s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.24940000000000004s for a response
I20260129 03:00:12.024997 20436 raft_consensus.cc:2410] T 1299723a4476420ab02fd8bfd7aa0645 P 197b880fa39d432a9b472845876bfaf0 [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 974b802141bd4f13beffef2aa03f88f9 for term 11 because replica has last-logged OpId of term: 10 index: 9, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 4.
W20260129 03:00:12.025203 20480 connection.cc:232] RPC call timeout handler was delayed by -1.1634604749999653s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.3058s for a response
I20260129 03:00:12.022115 20512 raft_consensus.cc:2468] T c891d2bff4ce4568861bbc992e91fbb0 P 974b802141bd4f13beffef2aa03f88f9 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1c50de89a5a845668696dcb28c3badd6 in term 8.
I20260129 03:00:11.982710 23304 leader_election.cc:290] T 3906b080a5b34a74b8b64128635e8997 P 1c50de89a5a845668696dcb28c3badd6 [CANDIDATE]: Term 5 election: Requested vote from peers 197b880fa39d432a9b472845876bfaf0 (127.18.211.1:42027), 974b802141bd4f13beffef2aa03f88f9 (127.18.211.2:34517)
I20260129 03:00:11.969141 23450 raft_consensus.cc:697] T fcddab6bca684adf8ea032289a00f90e P 974b802141bd4f13beffef2aa03f88f9 [term 10 LEADER]: Becoming Leader. State: Replica: 974b802141bd4f13beffef2aa03f88f9, State: Running, Role: LEADER
I20260129 03:00:12.029088 20436 raft_consensus.cc:1275] T 76f713c3331540d2a5b76164103e9eb1 P 197b880fa39d432a9b472845876bfaf0 [term 6 FOLLOWER]: Refusing update from remote peer 974b802141bd4f13beffef2aa03f88f9: Log matching property violated. Preceding OpId in replica: term: 5 index: 4. Preceding OpId from leader: term: 6 index: 5. (index mismatch)
I20260129 03:00:12.029470 22694 raft_consensus.cc:493] T 45cc5f3d2688486d9b6dbe7e666dedde P 974b802141bd4f13beffef2aa03f88f9 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260129 03:00:12.029944 20405 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.211.3:37735, user_credentials={real_user=slave}} blocked reactor thread for 69260.6us
I20260129 03:00:12.034746 20436 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "913fc7d751ed4b7388d889aac30a5f9b" candidate_uuid: "1c50de89a5a845668696dcb28c3badd6" candidate_term: 6 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "197b880fa39d432a9b472845876bfaf0" is_pre_election: true
I20260129 03:00:12.036485 20436 raft_consensus.cc:1240] T 36ff33be666a4c82954731691dbcf327 P 197b880fa39d432a9b472845876bfaf0 [term 4 FOLLOWER]: Rejecting Update request from peer 974b802141bd4f13beffef2aa03f88f9 for earlier term 3. Current term is 4. Ops: []
I20260129 03:00:12.041108 23141 leader_election.cc:290] T dc2f4cb752cf4d59b5f6bc689f5d8104 P 7a9daf5d42a84615a6383cfec9a10608 [CANDIDATE]: Term 6 election: Requested vote from peers 197b880fa39d432a9b472845876bfaf0 (127.18.211.1:42027), 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3:37735)
I20260129 03:00:12.041199 20553 leader_election.cc:304] T a5a05fbcbbcf4916bbc01fd1a5b5e90d P 1c50de89a5a845668696dcb28c3badd6 [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 197b880fa39d432a9b472845876bfaf0, 1c50de89a5a845668696dcb28c3badd6; no voters:
I20260129 03:00:12.042101 20589 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "76f713c3331540d2a5b76164103e9eb1" candidate_uuid: "974b802141bd4f13beffef2aa03f88f9" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "1c50de89a5a845668696dcb28c3badd6"
I20260129 03:00:12.045507 20586 raft_consensus.cc:2393] T 18f30ebaa4174fd28c958e72d0cc7e24 P 1c50de89a5a845668696dcb28c3badd6 [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate 7a9daf5d42a84615a6383cfec9a10608 in current term 8: Already voted for candidate 1c50de89a5a845668696dcb28c3badd6 in this term.
I20260129 03:00:12.045850 23492 consensus_queue.cc:1048] T 76f713c3331540d2a5b76164103e9eb1 P 974b802141bd4f13beffef2aa03f88f9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20260129 03:00:12.046577 23490 consensus_queue.cc:1059] T 36ff33be666a4c82954731691dbcf327 P 974b802141bd4f13beffef2aa03f88f9 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 }, Status: INVALID_TERM, Last received: 3.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260129 03:00:12.047892 22694 raft_consensus.cc:515] T 45cc5f3d2688486d9b6dbe7e666dedde P 974b802141bd4f13beffef2aa03f88f9 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } }
I20260129 03:00:12.049583 23458 leader_election.cc:290] T 7a9a69cd82724b1faed1d10aa091fc4a P 1c50de89a5a845668696dcb28c3badd6 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 197b880fa39d432a9b472845876bfaf0 (127.18.211.1:42027), 974b802141bd4f13beffef2aa03f88f9 (127.18.211.2:34517)
I20260129 03:00:12.049896 22694 leader_election.cc:290] T 45cc5f3d2688486d9b6dbe7e666dedde P 974b802141bd4f13beffef2aa03f88f9 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3:37735), 7a9daf5d42a84615a6383cfec9a10608 (127.18.211.4:40439)
W20260129 03:00:12.051183 20407 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.211.2:34517, user_credentials={real_user=slave}} blocked reactor thread for 48862.7us
I20260129 03:00:12.051978 23368 leader_election.cc:290] T 36ff33be666a4c82954731691dbcf327 P 197b880fa39d432a9b472845876bfaf0 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 974b802141bd4f13beffef2aa03f88f9 (127.18.211.2:34517), 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3:37735)
I20260129 03:00:12.055809 20509 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "75b0b6e82ddc4f92b1fe1d94f9a0b73b" candidate_uuid: "1c50de89a5a845668696dcb28c3badd6" candidate_term: 5 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "974b802141bd4f13beffef2aa03f88f9" is_pre_election: true
I20260129 03:00:12.056465 20509 raft_consensus.cc:2468] T 75b0b6e82ddc4f92b1fe1d94f9a0b73b P 974b802141bd4f13beffef2aa03f88f9 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1c50de89a5a845668696dcb28c3badd6 in term 4.
I20260129 03:00:12.058696 20432 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "17d6431cc77a491e8464d68db4081a51" candidate_uuid: "974b802141bd4f13beffef2aa03f88f9" candidate_term: 6 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "197b880fa39d432a9b472845876bfaf0" is_pre_election: true
I20260129 03:00:12.059520 20432 raft_consensus.cc:2468] T 17d6431cc77a491e8464d68db4081a51 P 197b880fa39d432a9b472845876bfaf0 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 974b802141bd4f13beffef2aa03f88f9 in term 5.
I20260129 03:00:12.060618 20479 leader_election.cc:304] T 17d6431cc77a491e8464d68db4081a51 P 974b802141bd4f13beffef2aa03f88f9 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 197b880fa39d432a9b472845876bfaf0, 974b802141bd4f13beffef2aa03f88f9; no voters:
I20260129 03:00:12.060670 20587 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "cedce331e1584ac8b636c9b8a72b2ba0" candidate_uuid: "197b880fa39d432a9b472845876bfaf0" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "1c50de89a5a845668696dcb28c3badd6" is_pre_election: true
I20260129 03:00:12.061461 20587 raft_consensus.cc:2376] T cedce331e1584ac8b636c9b8a72b2ba0 P 1c50de89a5a845668696dcb28c3badd6 [term 8 FOLLOWER]: Leader pre-election vote request: Already granted yes vote for candidate 197b880fa39d432a9b472845876bfaf0 in term 8. Re-sending same reply.
I20260129 03:00:12.065084 20509 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4f5a77e94fef4b16a2ba7b067c8b144a" candidate_uuid: "1c50de89a5a845668696dcb28c3badd6" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "974b802141bd4f13beffef2aa03f88f9" is_pre_election: true
I20260129 03:00:12.066030 20509 raft_consensus.cc:2393] T 4f5a77e94fef4b16a2ba7b067c8b144a P 974b802141bd4f13beffef2aa03f88f9 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1c50de89a5a845668696dcb28c3badd6 in current term 5: Already voted for candidate 7a9daf5d42a84615a6383cfec9a10608 in this term.
I20260129 03:00:12.056792 20405 leader_election.cc:304] T a5a05fbcbbcf4916bbc01fd1a5b5e90d P 197b880fa39d432a9b472845876bfaf0 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 197b880fa39d432a9b472845876bfaf0; no voters: 1c50de89a5a845668696dcb28c3badd6, 974b802141bd4f13beffef2aa03f88f9
I20260129 03:00:12.070544 20512 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0340229d521b432580ae926f6062e3f7" candidate_uuid: "1c50de89a5a845668696dcb28c3badd6" candidate_term: 8 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "974b802141bd4f13beffef2aa03f88f9" is_pre_election: true
I20260129 03:00:12.072609 20581 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "cb0b3f259ebc4310927338a8740e87b8" candidate_uuid: "974b802141bd4f13beffef2aa03f88f9" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "1c50de89a5a845668696dcb28c3badd6" is_pre_election: true
I20260129 03:00:12.073554 20581 raft_consensus.cc:2393] T cb0b3f259ebc4310927338a8740e87b8 P 1c50de89a5a845668696dcb28c3badd6 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 974b802141bd4f13beffef2aa03f88f9 in current term 8: Already voted for candidate 1c50de89a5a845668696dcb28c3badd6 in this term.
W20260129 03:00:12.074846 21100 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.211.3:37735, user_credentials={real_user=slave}} blocked reactor thread for 270235us
I20260129 03:00:12.075485 20509 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3e2d15f8640b407195210c476e09df91" candidate_uuid: "1c50de89a5a845668696dcb28c3badd6" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "974b802141bd4f13beffef2aa03f88f9" is_pre_election: true
I20260129 03:00:12.076683 20509 raft_consensus.cc:2410] T 3e2d15f8640b407195210c476e09df91 P 974b802141bd4f13beffef2aa03f88f9 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1c50de89a5a845668696dcb28c3badd6 for term 5 because replica has last-logged OpId of term: 4 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 5.
I20260129 03:00:12.078043 20438 raft_consensus.cc:2468] T 0678b8b468bd43cdbc3e485900844821 P 197b880fa39d432a9b472845876bfaf0 [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 974b802141bd4f13beffef2aa03f88f9 in term 9.
I20260129 03:00:12.078945 20510 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a5a05fbcbbcf4916bbc01fd1a5b5e90d" candidate_uuid: "1c50de89a5a845668696dcb28c3badd6" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "974b802141bd4f13beffef2aa03f88f9" is_pre_election: true
I20260129 03:00:12.079300 20438 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3906b080a5b34a74b8b64128635e8997" candidate_uuid: "1c50de89a5a845668696dcb28c3badd6" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "197b880fa39d432a9b472845876bfaf0"
I20260129 03:00:12.082074 20587 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "36ff33be666a4c82954731691dbcf327" candidate_uuid: "197b880fa39d432a9b472845876bfaf0" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "1c50de89a5a845668696dcb28c3badd6" is_pre_election: true
I20260129 03:00:12.083053 20511 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3906b080a5b34a74b8b64128635e8997" candidate_uuid: "1c50de89a5a845668696dcb28c3badd6" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "974b802141bd4f13beffef2aa03f88f9" is_pre_election: true
I20260129 03:00:12.026661 23485 raft_consensus.cc:3060] T 368953c9102643b2a21f99af36b51905 P 197b880fa39d432a9b472845876bfaf0 [term 8 FOLLOWER]: Advancing to term 9
I20260129 03:00:12.085248 20432 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7a9a69cd82724b1faed1d10aa091fc4a" candidate_uuid: "1c50de89a5a845668696dcb28c3badd6" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "197b880fa39d432a9b472845876bfaf0" is_pre_election: true
I20260129 03:00:12.085855 20512 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "17d6431cc77a491e8464d68db4081a51" candidate_uuid: "1c50de89a5a845668696dcb28c3badd6" candidate_term: 5 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "974b802141bd4f13beffef2aa03f88f9" is_pre_election: true
I20260129 03:00:12.086066 20479 leader_election.cc:304] T 0678b8b468bd43cdbc3e485900844821 P 974b802141bd4f13beffef2aa03f88f9 [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 197b880fa39d432a9b472845876bfaf0, 974b802141bd4f13beffef2aa03f88f9; no voters:
I20260129 03:00:12.090965 20509 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a5a05fbcbbcf4916bbc01fd1a5b5e90d" candidate_uuid: "1c50de89a5a845668696dcb28c3badd6" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "974b802141bd4f13beffef2aa03f88f9"
I20260129 03:00:12.067777 20480 leader_election.cc:304] T 75b0b6e82ddc4f92b1fe1d94f9a0b73b P 974b802141bd4f13beffef2aa03f88f9 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 974b802141bd4f13beffef2aa03f88f9; no voters: 1c50de89a5a845668696dcb28c3badd6, 7a9daf5d42a84615a6383cfec9a10608
I20260129 03:00:12.092725 22694 raft_consensus.cc:2749] T 75b0b6e82ddc4f92b1fe1d94f9a0b73b P 974b802141bd4f13beffef2aa03f88f9 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260129 03:00:12.093416 20480 leader_election.cc:400] T 1299723a4476420ab02fd8bfd7aa0645 P 974b802141bd4f13beffef2aa03f88f9 [CANDIDATE]: Term 9 pre-election: Vote denied by peer 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3:37735) with higher term. Message: Invalid argument: T 1299723a4476420ab02fd8bfd7aa0645 P 1c50de89a5a845668696dcb28c3badd6 [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 974b802141bd4f13beffef2aa03f88f9 for earlier term 9. Current term is 10.
I20260129 03:00:12.094882 20509 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a4c1226b987b468fa84dfc6ba0e2dbe1" candidate_uuid: "1c50de89a5a845668696dcb28c3badd6" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "974b802141bd4f13beffef2aa03f88f9" is_pre_election: true
W20260129 03:00:12.095619 21102 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260129 03:00:12.096406 20480 leader_election.cc:304] T a0fcc58d6fb6473cafbd9d34d5195269 P 974b802141bd4f13beffef2aa03f88f9 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 974b802141bd4f13beffef2aa03f88f9; no voters: 197b880fa39d432a9b472845876bfaf0, 1c50de89a5a845668696dcb28c3badd6
I20260129 03:00:12.097608 20480 leader_election.cc:304] T 8055232cf0004d40a85f3bf983fe00a9 P 974b802141bd4f13beffef2aa03f88f9 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1c50de89a5a845668696dcb28c3badd6, 974b802141bd4f13beffef2aa03f88f9; no voters: 197b880fa39d432a9b472845876bfaf0
W20260129 03:00:12.098785 21161 heartbeater.cc:646] Failed to heartbeat to 127.18.211.62:34285 (0 consecutive failures): Timed out: Failed to send heartbeat to master: TSHeartbeat RPC to 127.18.211.62:34285 timed out after 15.000s (SENT)
I20260129 03:00:12.099574 20480 leader_election.cc:304] T 4408006eb5174de1896b037cf3f18457 P 974b802141bd4f13beffef2aa03f88f9 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 974b802141bd4f13beffef2aa03f88f9; no voters: 197b880fa39d432a9b472845876bfaf0, 1c50de89a5a845668696dcb28c3badd6
I20260129 03:00:12.101461 22989 raft_consensus.cc:2804] T 8055232cf0004d40a85f3bf983fe00a9 P 974b802141bd4f13beffef2aa03f88f9 [term 5 FOLLOWER]: Leader election won for term 5
I20260129 03:00:12.101956 22989 raft_consensus.cc:697] T 8055232cf0004d40a85f3bf983fe00a9 P 974b802141bd4f13beffef2aa03f88f9 [term 5 LEADER]: Becoming Leader. State: Replica: 974b802141bd4f13beffef2aa03f88f9, State: Running, Role: LEADER
I20260129 03:00:12.102104 20480 leader_election.cc:304] T 3e2d15f8640b407195210c476e09df91 P 974b802141bd4f13beffef2aa03f88f9 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 974b802141bd4f13beffef2aa03f88f9; no voters: 1c50de89a5a845668696dcb28c3badd6, 7a9daf5d42a84615a6383cfec9a10608
I20260129 03:00:12.103123 22694 raft_consensus.cc:2749] T 3e2d15f8640b407195210c476e09df91 P 974b802141bd4f13beffef2aa03f88f9 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260129 03:00:12.102957 22989 consensus_queue.cc:237] T 8055232cf0004d40a85f3bf983fe00a9 P 974b802141bd4f13beffef2aa03f88f9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 03:00:12.102123 23333 raft_consensus.cc:2749] T 4408006eb5174de1896b037cf3f18457 P 974b802141bd4f13beffef2aa03f88f9 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260129 03:00:12.067896 23458 raft_consensus.cc:2804] T a5a05fbcbbcf4916bbc01fd1a5b5e90d P 1c50de89a5a845668696dcb28c3badd6 [term 8 FOLLOWER]: Leader election won for term 8
I20260129 03:00:12.069976 23402 raft_consensus.cc:2804] T 17d6431cc77a491e8464d68db4081a51 P 974b802141bd4f13beffef2aa03f88f9 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260129 03:00:12.110397 23035 consensus_queue.cc:1048] T 45cc5f3d2688486d9b6dbe7e666dedde P 7a9daf5d42a84615a6383cfec9a10608 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.014s
W20260129 03:00:12.125147 23400 consensus_meta.cc:287] T c891d2bff4ce4568861bbc992e91fbb0 P 197b880fa39d432a9b472845876bfaf0: Time spent flushing consensus metadata: real 1.457s user 0.006s sys 0.006s
I20260129 03:00:12.125555 23400 raft_consensus.cc:515] T c891d2bff4ce4568861bbc992e91fbb0 P 197b880fa39d432a9b472845876bfaf0 [term 9 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 03:00:12.114542 20587 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "18f30ebaa4174fd28c958e72d0cc7e24" candidate_uuid: "197b880fa39d432a9b472845876bfaf0" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "1c50de89a5a845668696dcb28c3badd6"
I20260129 03:00:12.114780 20510 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b189f08e16a74a11980cc55d99359a0a" candidate_uuid: "1c50de89a5a845668696dcb28c3badd6" candidate_term: 10 candidate_status { last_received { term: 8 index: 4 } } ignore_live_leader: false dest_uuid: "974b802141bd4f13beffef2aa03f88f9" is_pre_election: true
I20260129 03:00:12.109249 20509 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "724a236524cb4e7ba3428880681d6cc2" candidate_uuid: "1c50de89a5a845668696dcb28c3badd6" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "974b802141bd4f13beffef2aa03f88f9"
I20260129 03:00:12.114991 20581 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4b1876f9a70b49edb3aa74105e66d3b3" candidate_uuid: "197b880fa39d432a9b472845876bfaf0" candidate_term: 6 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "1c50de89a5a845668696dcb28c3badd6" is_pre_election: true
I20260129 03:00:12.129498 20581 raft_consensus.cc:2393] T 4b1876f9a70b49edb3aa74105e66d3b3 P 1c50de89a5a845668696dcb28c3badd6 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 197b880fa39d432a9b472845876bfaf0 in current term 6: Already voted for candidate 974b802141bd4f13beffef2aa03f88f9 in this term.
I20260129 03:00:12.128278 20439 raft_consensus.cc:2368] T c891d2bff4ce4568861bbc992e91fbb0 P 197b880fa39d432a9b472845876bfaf0 [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate 974b802141bd4f13beffef2aa03f88f9 for earlier term 8. Current term is 9.
I20260129 03:00:12.120975 20511 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "75f45538dc704d9a8ec4ff8ad9f3ea5b" candidate_uuid: "1c50de89a5a845668696dcb28c3badd6" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "974b802141bd4f13beffef2aa03f88f9" is_pre_election: true
I20260129 03:00:12.118124 20480 leader_election.cc:304] T 7a9a69cd82724b1faed1d10aa091fc4a P 974b802141bd4f13beffef2aa03f88f9 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 974b802141bd4f13beffef2aa03f88f9; no voters: 197b880fa39d432a9b472845876bfaf0, 1c50de89a5a845668696dcb28c3badd6
I20260129 03:00:12.078352 23436 leader_election.cc:290] T 0678b8b468bd43cdbc3e485900844821 P 197b880fa39d432a9b472845876bfaf0 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 974b802141bd4f13beffef2aa03f88f9 (127.18.211.2:34517), 7a9daf5d42a84615a6383cfec9a10608 (127.18.211.4:40439)
I20260129 03:00:12.150499 20509 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "add8d4923d1a489e86882c3a25692b95" candidate_uuid: "1c50de89a5a845668696dcb28c3badd6" candidate_term: 8 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "974b802141bd4f13beffef2aa03f88f9" is_pre_election: true
I20260129 03:00:12.131244 20511 raft_consensus.cc:2468] T 75f45538dc704d9a8ec4ff8ad9f3ea5b P 974b802141bd4f13beffef2aa03f88f9 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1c50de89a5a845668696dcb28c3badd6 in term 4.
I20260129 03:00:12.117802 20512 raft_consensus.cc:2468] T 17d6431cc77a491e8464d68db4081a51 P 974b802141bd4f13beffef2aa03f88f9 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1c50de89a5a845668696dcb28c3badd6 in term 5.
I20260129 03:00:12.169672 20509 raft_consensus.cc:2468] T add8d4923d1a489e86882c3a25692b95 P 974b802141bd4f13beffef2aa03f88f9 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1c50de89a5a845668696dcb28c3badd6 in term 7.
I20260129 03:00:12.176612 20587 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fcddab6bca684adf8ea032289a00f90e" candidate_uuid: "974b802141bd4f13beffef2aa03f88f9" candidate_term: 10 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "1c50de89a5a845668696dcb28c3badd6" is_pre_election: true
W20260129 03:00:12.177234 20480 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.211.3:37735, user_credentials={real_user=slave}} blocked reactor thread for 59240.1us
I20260129 03:00:12.205742 20480 leader_election.cc:304] T 0f20af7fbe5d4db89a6f17948d98912c P 974b802141bd4f13beffef2aa03f88f9 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1c50de89a5a845668696dcb28c3badd6, 974b802141bd4f13beffef2aa03f88f9; no voters: 197b880fa39d432a9b472845876bfaf0
I20260129 03:00:12.116298 23427 raft_consensus.cc:2804] T 0678b8b468bd43cdbc3e485900844821 P 974b802141bd4f13beffef2aa03f88f9 [term 9 FOLLOWER]: Leader election won for term 9
I20260129 03:00:12.203529 20581 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0340229d521b432580ae926f6062e3f7" candidate_uuid: "197b880fa39d432a9b472845876bfaf0" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "1c50de89a5a845668696dcb28c3badd6" is_pre_election: true
I20260129 03:00:12.215304 20581 raft_consensus.cc:2468] T 0340229d521b432580ae926f6062e3f7 P 1c50de89a5a845668696dcb28c3badd6 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 197b880fa39d432a9b472845876bfaf0 in term 7.
I20260129 03:00:12.216373 20587 raft_consensus.cc:2376] T fcddab6bca684adf8ea032289a00f90e P 1c50de89a5a845668696dcb28c3badd6 [term 10 FOLLOWER]: Leader pre-election vote request: Already granted yes vote for candidate 974b802141bd4f13beffef2aa03f88f9 in term 10. Re-sending same reply.
I20260129 03:00:12.113570 23215 leader_election.cc:290] T add8d4923d1a489e86882c3a25692b95 P 974b802141bd4f13beffef2aa03f88f9 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3:37735), 7a9daf5d42a84615a6383cfec9a10608 (127.18.211.4:40439)
I20260129 03:00:12.217794 20510 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "913fc7d751ed4b7388d889aac30a5f9b" candidate_uuid: "1c50de89a5a845668696dcb28c3badd6" candidate_term: 6 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "974b802141bd4f13beffef2aa03f88f9" is_pre_election: true
I20260129 03:00:12.176314 22694 raft_consensus.cc:2749] T 7a9a69cd82724b1faed1d10aa091fc4a P 974b802141bd4f13beffef2aa03f88f9 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260129 03:00:12.220211 20581 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7d9494f5584d4d5db8b9adf1c8fc888b" candidate_uuid: "197b880fa39d432a9b472845876bfaf0" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "1c50de89a5a845668696dcb28c3badd6" is_pre_election: true
I20260129 03:00:12.223567 20511 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e33a417cf9c54cf5af73a4ee3aac4d3a" candidate_uuid: "1c50de89a5a845668696dcb28c3badd6" candidate_term: 7 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "974b802141bd4f13beffef2aa03f88f9"
I20260129 03:00:12.224009 20581 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1299723a4476420ab02fd8bfd7aa0645" candidate_uuid: "197b880fa39d432a9b472845876bfaf0" candidate_term: 11 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "1c50de89a5a845668696dcb28c3badd6" is_pre_election: true
W20260129 03:00:12.229254 20590 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:35754 (request call id 1817) took 3173 ms (client timeout 2120 ms). Trace:
W20260129 03:00:12.229755 20590 rpcz_store.cc:269] 0129 03:00:09.055658 (+ 0us) service_pool.cc:168] Inserting onto call queue
0129 03:00:09.451339 (+395681us) service_pool.cc:225] Handling call
0129 03:00:12.229051 (+2777712us) spinlock_profiling.cc:243] Waited 366 ms on lock 0x7b5400108e54. stack: 00007f8fe044e373 00007f8fe044e270 00007f8fdfda0574 00007f8fe9a5c6b4 00007f8fe9a5c649 00007f8fe34975a1 00007f8fe3531f7a 00007f8fe84e3647 00007f8fe1a73834 00007f8fe1a737c2 00007f8fe1a736f1 00007f8fe1a7366c 00007f8fe1a72922 00007f8fe186d34c 00007f8fe186c786 00007f8fe186c12f
0129 03:00:12.229186 (+ 135us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":841308032}
I20260129 03:00:12.231401 20581 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2e58e9a5a8aa4cecafb464addb4f35ed" candidate_uuid: "197b880fa39d432a9b472845876bfaf0" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "1c50de89a5a845668696dcb28c3badd6" is_pre_election: true
I20260129 03:00:12.205422 23032 raft_consensus.cc:2749] T b189f08e16a74a11980cc55d99359a0a P 974b802141bd4f13beffef2aa03f88f9 [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20260129 03:00:12.167413 23458 raft_consensus.cc:697] T a5a05fbcbbcf4916bbc01fd1a5b5e90d P 1c50de89a5a845668696dcb28c3badd6 [term 8 LEADER]: Becoming Leader. State: Replica: 1c50de89a5a845668696dcb28c3badd6, State: Running, Role: LEADER
I20260129 03:00:12.236852 20590 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a0fcc58d6fb6473cafbd9d34d5195269" candidate_uuid: "197b880fa39d432a9b472845876bfaf0" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "1c50de89a5a845668696dcb28c3badd6"
I20260129 03:00:12.241883 20590 raft_consensus.cc:2376] T a0fcc58d6fb6473cafbd9d34d5195269 P 1c50de89a5a845668696dcb28c3badd6 [term 5 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 197b880fa39d432a9b472845876bfaf0 in term 5. Re-sending same reply.
I20260129 03:00:12.239965 23439 raft_consensus.cc:2749] T febffe4a8b394368b69b1d164308fb9a P 7a9daf5d42a84615a6383cfec9a10608 [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260129 03:00:12.232862 23427 raft_consensus.cc:697] T 0678b8b468bd43cdbc3e485900844821 P 974b802141bd4f13beffef2aa03f88f9 [term 9 LEADER]: Becoming Leader. State: Replica: 974b802141bd4f13beffef2aa03f88f9, State: Running, Role: LEADER
I20260129 03:00:12.264382 23427 consensus_queue.cc:237] T 0678b8b468bd43cdbc3e485900844821 P 974b802141bd4f13beffef2aa03f88f9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 7.8, Last appended by leader: 8, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } }
I20260129 03:00:12.246078 23382 consensus_queue.cc:1048] T add8d4923d1a489e86882c3a25692b95 P 7a9daf5d42a84615a6383cfec9a10608 [LEADER]: Connected to new peer: Peer: permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.172s
I20260129 03:00:12.247423 23400 leader_election.cc:290] T c891d2bff4ce4568861bbc992e91fbb0 P 197b880fa39d432a9b472845876bfaf0 [CANDIDATE]: Term 9 election: Requested vote from peers 974b802141bd4f13beffef2aa03f88f9 (127.18.211.2:34517), 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3:37735)
I20260129 03:00:12.242695 23215 raft_consensus.cc:2804] T 0f20af7fbe5d4db89a6f17948d98912c P 974b802141bd4f13beffef2aa03f88f9 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260129 03:00:12.252766 23398 leader_election.cc:290] T 18f30ebaa4174fd28c958e72d0cc7e24 P 1c50de89a5a845668696dcb28c3badd6 [CANDIDATE]: Term 8 election: Requested vote from peers 197b880fa39d432a9b472845876bfaf0 (127.18.211.1:42027), 7a9daf5d42a84615a6383cfec9a10608 (127.18.211.4:40439)
W20260129 03:00:12.253854 23476 log.cc:927] Time spent T 4e3805ffa70242e8ade66da260dddf66 P 1c50de89a5a845668696dcb28c3badd6: Append to log took a long time: real 0.327s user 0.001s sys 0.000s
W20260129 03:00:12.255012 23489 log.cc:927] Time spent T 4e3805ffa70242e8ade66da260dddf66 P 197b880fa39d432a9b472845876bfaf0: Append to log took a long time: real 0.173s user 0.000s sys 0.000s
I20260129 03:00:12.233886 23490 raft_consensus.cc:3055] T 36ff33be666a4c82954731691dbcf327 P 974b802141bd4f13beffef2aa03f88f9 [term 3 LEADER]: Stepping down as leader of term 3
W20260129 03:00:12.262763 23498 log.cc:927] Time spent T 724a236524cb4e7ba3428880681d6cc2 P 197b880fa39d432a9b472845876bfaf0: Append to log took a long time: real 0.060s user 0.000s sys 0.000s
I20260129 03:00:12.262987 20590 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "76f713c3331540d2a5b76164103e9eb1" candidate_uuid: "197b880fa39d432a9b472845876bfaf0" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "1c50de89a5a845668696dcb28c3badd6" is_pre_election: true
I20260129 03:00:12.266187 20511 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "926a250fa5834e88b96ccecbefbf5086" candidate_uuid: "1c50de89a5a845668696dcb28c3badd6" candidate_term: 8 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "974b802141bd4f13beffef2aa03f88f9" is_pre_election: true
I20260129 03:00:12.207751 23456 leader_election.cc:290] T a1a39dcd328e4268ae41b988c66b4a75 P 1c50de89a5a845668696dcb28c3badd6 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 974b802141bd4f13beffef2aa03f88f9 (127.18.211.2:34517), 197b880fa39d432a9b472845876bfaf0 (127.18.211.1:42027)
I20260129 03:00:12.267846 20581 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "cb4727386a3e48aca5d71cc6c611c197" candidate_uuid: "197b880fa39d432a9b472845876bfaf0" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "1c50de89a5a845668696dcb28c3badd6"
I20260129 03:00:12.177484 23402 raft_consensus.cc:493] T 17d6431cc77a491e8464d68db4081a51 P 974b802141bd4f13beffef2aa03f88f9 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260129 03:00:12.225492 23437 consensus_meta.cc:287] T 4f8f3a38f6e24ba4857875181f447442 P 197b880fa39d432a9b472845876bfaf0: Time spent flushing consensus metadata: real 0.801s user 0.011s sys 0.000s
I20260129 03:00:12.276454 20587 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "75f45538dc704d9a8ec4ff8ad9f3ea5b" candidate_uuid: "974b802141bd4f13beffef2aa03f88f9" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "1c50de89a5a845668696dcb28c3badd6" is_pre_election: true
I20260129 03:00:12.176867 23450 consensus_queue.cc:237] T fcddab6bca684adf8ea032289a00f90e P 974b802141bd4f13beffef2aa03f88f9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 8.5, Last appended by leader: 5, Current term: 10, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 03:00:12.309644 20439 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55672 (request call id 1746) took 1511 ms. Trace:
I20260129 03:00:12.310756 20514 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a5a05fbcbbcf4916bbc01fd1a5b5e90d" candidate_uuid: "1c50de89a5a845668696dcb28c3badd6" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "974b802141bd4f13beffef2aa03f88f9" is_pre_election: true
I20260129 03:00:12.328974 20439 rpcz_store.cc:276] 0129 03:00:10.798219 (+ 0us) service_pool.cc:168] Inserting onto call queue
0129 03:00:10.800072 (+ 1853us) service_pool.cc:225] Handling call
0129 03:00:12.309419 (+1509347us) spinlock_profiling.cc:243] Waited 1.29 s on lock 0x7b54003d1654. stack: 00007f8fe044e373 00007f8fe044e270 00007f8fdfda0574 00007f8fe9a5c6b4 00007f8fe9a5c649 00007f8fe34975a1 00007f8fe3531f7a 00007f8fe84e3647 00007f8fe1a73834 00007f8fe1a737c2 00007f8fe1a736f1 00007f8fe1a7366c 00007f8fe1a72922 00007f8fe186d34c 00007f8fe186c786 00007f8fe186c12f
0129 03:00:12.309576 (+ 157us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2965434624}
I20260129 03:00:12.329690 20514 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "45cc5f3d2688486d9b6dbe7e666dedde" candidate_uuid: "1c50de89a5a845668696dcb28c3badd6" candidate_term: 6 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "974b802141bd4f13beffef2aa03f88f9" is_pre_election: true
I20260129 03:00:12.330206 20439 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a1a39dcd328e4268ae41b988c66b4a75" candidate_uuid: "1c50de89a5a845668696dcb28c3badd6" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "197b880fa39d432a9b472845876bfaf0" is_pre_election: true
I20260129 03:00:12.330475 20514 raft_consensus.cc:2393] T 45cc5f3d2688486d9b6dbe7e666dedde P 974b802141bd4f13beffef2aa03f88f9 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1c50de89a5a845668696dcb28c3badd6 in current term 6: Already voted for candidate 7a9daf5d42a84615a6383cfec9a10608 in this term.
I20260129 03:00:12.331087 23437 raft_consensus.cc:515] T 4f8f3a38f6e24ba4857875181f447442 P 197b880fa39d432a9b472845876bfaf0 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 03:00:12.281378 23026 raft_consensus.cc:493] T 4b1876f9a70b49edb3aa74105e66d3b3 P 1c50de89a5a845668696dcb28c3badd6 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260129 03:00:12.332180 23026 raft_consensus.cc:515] T 4b1876f9a70b49edb3aa74105e66d3b3 P 1c50de89a5a845668696dcb28c3badd6 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 03:00:12.333036 20434 raft_consensus.cc:1275] T 36ff33be666a4c82954731691dbcf327 P 197b880fa39d432a9b472845876bfaf0 [term 4 FOLLOWER]: Refusing update from remote peer 1c50de89a5a845668696dcb28c3badd6: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20260129 03:00:12.281270 23215 raft_consensus.cc:493] T 0f20af7fbe5d4db89a6f17948d98912c P 974b802141bd4f13beffef2aa03f88f9 [term 7 FOLLOWER]: Starting leader election (detected failure of leader 197b880fa39d432a9b472845876bfaf0)
I20260129 03:00:12.296161 23480 consensus_queue.cc:1048] T add8d4923d1a489e86882c3a25692b95 P 7a9daf5d42a84615a6383cfec9a10608 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.033s
I20260129 03:00:12.317998 23485 raft_consensus.cc:515] T 368953c9102643b2a21f99af36b51905 P 197b880fa39d432a9b472845876bfaf0 [term 9 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 03:00:12.319568 23458 consensus_queue.cc:237] T a5a05fbcbbcf4916bbc01fd1a5b5e90d P 1c50de89a5a845668696dcb28c3badd6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 7.5, Last appended by leader: 5, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
W20260129 03:00:12.322651 20479 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260129 03:00:12.323441 20583 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35752 (request call id 1698) took 3080 ms. Trace:
I20260129 03:00:12.327330 20511 raft_consensus.cc:2410] T 926a250fa5834e88b96ccecbefbf5086 P 974b802141bd4f13beffef2aa03f88f9 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1c50de89a5a845668696dcb28c3badd6 for term 8 because replica has last-logged OpId of term: 6 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 4 index: 3.
I20260129 03:00:12.331350 20439 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "18f30ebaa4174fd28c958e72d0cc7e24" candidate_uuid: "1c50de89a5a845668696dcb28c3badd6" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "197b880fa39d432a9b472845876bfaf0"
W20260129 03:00:12.311014 20480 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.211.3:37735, user_credentials={real_user=slave}} blocked reactor thread for 105376us
I20260129 03:00:12.333634 20436 raft_consensus.cc:1240] T 4f8f3a38f6e24ba4857875181f447442 P 197b880fa39d432a9b472845876bfaf0 [term 7 FOLLOWER]: Rejecting Update request from peer 974b802141bd4f13beffef2aa03f88f9 for earlier term 6. Current term is 7. Ops: []
I20260129 03:00:12.506233 23304 raft_consensus.cc:493] T 75be4bd6bd45474795fd472eff2953ac P 1c50de89a5a845668696dcb28c3badd6 [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260129 03:00:12.372013 20407 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.211.2:34517, user_credentials={real_user=slave}} blocked reactor thread for 308193us
I20260129 03:00:12.514506 20584 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35752 (request call id 1679) took 4364 ms. Trace:
I20260129 03:00:12.514261 23304 raft_consensus.cc:515] T 75be4bd6bd45474795fd472eff2953ac P 1c50de89a5a845668696dcb28c3badd6 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 03:00:12.514978 20584 rpcz_store.cc:276] 0129 03:00:08.150375 (+ 0us) service_pool.cc:168] Inserting onto call queue
0129 03:00:09.897321 (+1746946us) service_pool.cc:225] Handling call
0129 03:00:11.537615 (+1640294us) raft_consensus.cc:1471] Updating replica for 0 ops
0129 03:00:11.634356 (+ 96741us) raft_consensus.cc:1697] Filling consensus response to leader.
0129 03:00:12.514303 (+879947us) spinlock_profiling.cc:243] Waited 1.64 s on lock 0x7b54000c9250. stack: 00007f8fe044e373 00007f8fe044e270 00007f8fdfda0574 00007f8fe9a5c6b4 00007f8fe9a5c649 00007f8fe34975a1 00007f8fe352d52d 00007f8fe84e1f49 00007f8fe1a6ed34 00007f8fe1a6ecc2 00007f8fe1a6ebf1 00007f8fe1a6eb6c 00007f8fe1a6de22 00007f8fe186d34c 00007f8fe186c786 00007f8fe186c12f
0129 03:00:12.514443 (+ 140us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":3771687168}
I20260129 03:00:12.515986 21099 leader_election.cc:304] T 0678b8b468bd43cdbc3e485900844821 P 7a9daf5d42a84615a6383cfec9a10608 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 197b880fa39d432a9b472845876bfaf0, 7a9daf5d42a84615a6383cfec9a10608; no voters:
I20260129 03:00:12.516831 23365 raft_consensus.cc:2804] T 0678b8b468bd43cdbc3e485900844821 P 7a9daf5d42a84615a6383cfec9a10608 [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260129 03:00:12.518013 21099 leader_election.cc:304] T 18f30ebaa4174fd28c958e72d0cc7e24 P 7a9daf5d42a84615a6383cfec9a10608 [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 197b880fa39d432a9b472845876bfaf0, 7a9daf5d42a84615a6383cfec9a10608; no voters:
I20260129 03:00:12.373908 20516 raft_consensus.cc:1275] T cedce331e1584ac8b636c9b8a72b2ba0 P 974b802141bd4f13beffef2aa03f88f9 [term 8 FOLLOWER]: Refusing update from remote peer 197b880fa39d432a9b472845876bfaf0: Log matching property violated. Preceding OpId in replica: term: 6 index: 4. Preceding OpId from leader: term: 8 index: 5. (index mismatch)
I20260129 03:00:12.518653 20584 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a4c1226b987b468fa84dfc6ba0e2dbe1" candidate_uuid: "974b802141bd4f13beffef2aa03f88f9" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "1c50de89a5a845668696dcb28c3badd6" is_pre_election: true
I20260129 03:00:12.519336 20583 rpcz_store.cc:276] 0129 03:00:09.243377 (+ 0us) service_pool.cc:168] Inserting onto call queue
0129 03:00:11.231146 (+1987769us) service_pool.cc:225] Handling call
0129 03:00:11.248445 (+ 17299us) raft_consensus.cc:1471] Updating replica for 0 ops
0129 03:00:11.829711 (+581266us) raft_consensus.cc:1697] Filling consensus response to leader.
0129 03:00:12.323123 (+493412us) spinlock_profiling.cc:243] Waited 562 ms on lock 0x7b540010a754. stack: 00007f8fe044e373 00007f8fe044e270 00007f8fdfda0574 00007f8fe9a5c6b4 00007f8fe9a5c649 00007f8fe34975a1 00007f8fe352ea81 00007f8fe352d4be 00007f8fe84e1f49 00007f8fe1a6ed34 00007f8fe1a6ecc2 00007f8fe1a6ebf1 00007f8fe1a6eb6c 00007f8fe1a6de22 00007f8fe186d34c 00007f8fe186c786
0129 03:00:12.323362 (+ 239us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1292575104}
I20260129 03:00:12.478590 20581 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a8479d1fe0bd41e5946186e3a6ca8c7a" candidate_uuid: "197b880fa39d432a9b472845876bfaf0" candidate_term: 6 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "1c50de89a5a845668696dcb28c3badd6" is_pre_election: true
I20260129 03:00:12.465721 20514 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3906b080a5b34a74b8b64128635e8997" candidate_uuid: "1c50de89a5a845668696dcb28c3badd6" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "974b802141bd4f13beffef2aa03f88f9"
I20260129 03:00:12.522334 20514 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7a9a69cd82724b1faed1d10aa091fc4a" candidate_uuid: "1c50de89a5a845668696dcb28c3badd6" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "974b802141bd4f13beffef2aa03f88f9" is_pre_election: true
I20260129 03:00:12.522876 20480 leader_election.cc:304] T 7d9494f5584d4d5db8b9adf1c8fc888b P 974b802141bd4f13beffef2aa03f88f9 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 974b802141bd4f13beffef2aa03f88f9; no voters: 197b880fa39d432a9b472845876bfaf0, 1c50de89a5a845668696dcb28c3badd6
I20260129 03:00:12.460817 20512 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3906b080a5b34a74b8b64128635e8997" candidate_uuid: "1c50de89a5a845668696dcb28c3badd6" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "974b802141bd4f13beffef2aa03f88f9" is_pre_election: true
I20260129 03:00:12.459622 20510 raft_consensus.cc:1240] T c2633a6fc34c44579de348d208b0c01e P 974b802141bd4f13beffef2aa03f88f9 [term 5 LEADER]: Rejecting Update request from peer 1c50de89a5a845668696dcb28c3badd6 for earlier term 4. Current term is 5. Ops: []
I20260129 03:00:12.518927 20516 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57600 (request call id 1937) took 2304 ms. Trace:
I20260129 03:00:12.350580 20587 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a1a39dcd328e4268ae41b988c66b4a75" candidate_uuid: "197b880fa39d432a9b472845876bfaf0" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "1c50de89a5a845668696dcb28c3badd6"
I20260129 03:00:12.524068 20439 raft_consensus.cc:2393] T 18f30ebaa4174fd28c958e72d0cc7e24 P 197b880fa39d432a9b472845876bfaf0 [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate 1c50de89a5a845668696dcb28c3badd6 in current term 8: Already voted for candidate 7a9daf5d42a84615a6383cfec9a10608 in this term.
I20260129 03:00:12.546482 20516 rpcz_store.cc:276] 0129 03:00:10.214412 (+ 0us) service_pool.cc:168] Inserting onto call queue
0129 03:00:12.373101 (+2158689us) service_pool.cc:225] Handling call
0129 03:00:12.373503 (+ 402us) raft_consensus.cc:1471] Updating replica for 0 ops
0129 03:00:12.518685 (+145182us) raft_consensus.cc:1697] Filling consensus response to leader.
0129 03:00:12.518873 (+ 188us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260129 03:00:12.526741 20511 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a1a39dcd328e4268ae41b988c66b4a75" candidate_uuid: "1c50de89a5a845668696dcb28c3badd6" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "974b802141bd4f13beffef2aa03f88f9" is_pre_election: true
I20260129 03:00:12.392001 20590 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "17d6431cc77a491e8464d68db4081a51" candidate_uuid: "197b880fa39d432a9b472845876bfaf0" candidate_term: 5 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "1c50de89a5a845668696dcb28c3badd6"
W20260129 03:00:12.390285 20405 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.211.3:37735, user_credentials={real_user=slave}} blocked reactor thread for 333548us
I20260129 03:00:12.525059 20584 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7a9a69cd82724b1faed1d10aa091fc4a" candidate_uuid: "197b880fa39d432a9b472845876bfaf0" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "1c50de89a5a845668696dcb28c3badd6"
I20260129 03:00:12.493408 20432 raft_consensus.cc:1240] T a1a39dcd328e4268ae41b988c66b4a75 P 197b880fa39d432a9b472845876bfaf0 [term 6 FOLLOWER]: Rejecting Update request from peer 974b802141bd4f13beffef2aa03f88f9 for earlier term 5. Current term is 6. Ops: []
I20260129 03:00:12.548305 20511 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2ac659d732314e1288849f08fe5e074e" candidate_uuid: "197b880fa39d432a9b472845876bfaf0" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "974b802141bd4f13beffef2aa03f88f9"
I20260129 03:00:12.467315 20582 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:48526 (request call id 386) took 3003 ms. Trace:
I20260129 03:00:12.549602 20511 raft_consensus.cc:3060] T 2ac659d732314e1288849f08fe5e074e P 974b802141bd4f13beffef2aa03f88f9 [term 5 FOLLOWER]: Advancing to term 6
I20260129 03:00:12.549521 20582 rpcz_store.cc:276] 0129 03:00:09.463448 (+ 0us) service_pool.cc:168] Inserting onto call queue
0129 03:00:11.272362 (+1808914us) service_pool.cc:225] Handling call
0129 03:00:11.272796 (+ 434us) raft_consensus.cc:1471] Updating replica for 1 ops
0129 03:00:11.361940 (+ 89144us) raft_consensus.cc:1534] Early marking committed up to index 0
0129 03:00:11.362221 (+ 281us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0129 03:00:11.380440 (+ 18219us) log.cc:844] Serialized 26 byte log entry
0129 03:00:11.767716 (+387276us) raft_consensus.cc:1645] Marking committed up to 7
0129 03:00:11.767785 (+ 69us) raft_consensus.cc:1697] Filling consensus response to leader.
0129 03:00:11.767904 (+ 119us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0129 03:00:12.467077 (+699173us) raft_consensus.cc:1685] finished
0129 03:00:12.467106 (+ 29us) raft_consensus.cc:1691] UpdateReplicas() finished
0129 03:00:12.467249 (+ 143us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":2807,"raft.queue_time_us":146356,"raft.run_cpu_time_us":142,"raft.run_wall_time_us":6585,"thread_start_us":441965,"threads_started":2,"wal-append.queue_time_us":412634}
I20260129 03:00:12.439635 20435 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:48356 (request call id 419) took 3991 ms. Trace:
I20260129 03:00:12.548250 20584 raft_consensus.cc:2376] T 7a9a69cd82724b1faed1d10aa091fc4a P 1c50de89a5a845668696dcb28c3badd6 [term 7 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 197b880fa39d432a9b472845876bfaf0 in term 7. Re-sending same reply.
I20260129 03:00:12.548377 20405 leader_election.cc:304] T 724a236524cb4e7ba3428880681d6cc2 P 197b880fa39d432a9b472845876bfaf0 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 197b880fa39d432a9b472845876bfaf0; no voters: 1c50de89a5a845668696dcb28c3badd6, 974b802141bd4f13beffef2aa03f88f9
I20260129 03:00:12.554620 20584 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "78aa39096868415d9afc2ffcb248e198" candidate_uuid: "197b880fa39d432a9b472845876bfaf0" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "1c50de89a5a845668696dcb28c3badd6" is_pre_election: true
I20260129 03:00:12.555541 20590 raft_consensus.cc:2376] T 17d6431cc77a491e8464d68db4081a51 P 1c50de89a5a845668696dcb28c3badd6 [term 5 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 197b880fa39d432a9b472845876bfaf0 in term 5. Re-sending same reply.
I20260129 03:00:12.557650 20516 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0340229d521b432580ae926f6062e3f7" candidate_uuid: "1c50de89a5a845668696dcb28c3badd6" candidate_term: 8 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "974b802141bd4f13beffef2aa03f88f9" is_pre_election: true
I20260129 03:00:12.517251 23365 raft_consensus.cc:493] T 0678b8b468bd43cdbc3e485900844821 P 7a9daf5d42a84615a6383cfec9a10608 [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260129 03:00:12.561920 20514 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "368953c9102643b2a21f99af36b51905" candidate_uuid: "1c50de89a5a845668696dcb28c3badd6" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "974b802141bd4f13beffef2aa03f88f9"
I20260129 03:00:12.567149 20479 leader_election.cc:400] T c891d2bff4ce4568861bbc992e91fbb0 P 974b802141bd4f13beffef2aa03f88f9 [CANDIDATE]: Term 8 election: Vote denied by peer 197b880fa39d432a9b472845876bfaf0 (127.18.211.1:42027) with higher term. Message: Invalid argument: T c891d2bff4ce4568861bbc992e91fbb0 P 197b880fa39d432a9b472845876bfaf0 [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate 974b802141bd4f13beffef2aa03f88f9 for earlier term 8. Current term is 9.
I20260129 03:00:12.573963 20479 leader_election.cc:403] T c891d2bff4ce4568861bbc992e91fbb0 P 974b802141bd4f13beffef2aa03f88f9 [CANDIDATE]: Term 8 election: Cancelling election due to peer responding with higher term
I20260129 03:00:12.559543 20590 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c891d2bff4ce4568861bbc992e91fbb0" candidate_uuid: "974b802141bd4f13beffef2aa03f88f9" candidate_term: 8 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "1c50de89a5a845668696dcb28c3badd6"
W20260129 03:00:12.472828 23429 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 1.347s user 0.004s sys 0.017s
I20260129 03:00:12.325126 23490 raft_consensus.cc:740] T 36ff33be666a4c82954731691dbcf327 P 974b802141bd4f13beffef2aa03f88f9 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 974b802141bd4f13beffef2aa03f88f9, State: Running, Role: LEADER
W20260129 03:00:12.594986 21102 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260129 03:00:12.637919 20480 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.211.3:37735, user_credentials={real_user=slave}} blocked reactor thread for 59607.4us
I20260129 03:00:12.466087 23380 raft_consensus.cc:3060] T cb0b3f259ebc4310927338a8740e87b8 P 7a9daf5d42a84615a6383cfec9a10608 [term 7 FOLLOWER]: Advancing to term 8
I20260129 03:00:12.643492 20516 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a1a39dcd328e4268ae41b988c66b4a75" candidate_uuid: "197b880fa39d432a9b472845876bfaf0" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "974b802141bd4f13beffef2aa03f88f9"
I20260129 03:00:12.552112 20435 rpcz_store.cc:276] 0129 03:00:08.448808 (+ 0us) service_pool.cc:168] Inserting onto call queue
0129 03:00:08.449001 (+ 193us) service_pool.cc:225] Handling call
0129 03:00:08.449396 (+ 395us) raft_consensus.cc:1471] Updating replica for 0 ops
0129 03:00:09.391710 (+942314us) raft_consensus.cc:1534] Early marking committed up to index 6
0129 03:00:10.040993 (+649283us) log.cc:844] Serialized 14 byte log entry
0129 03:00:10.217314 (+176321us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0129 03:00:10.217390 (+ 76us) raft_consensus.cc:1645] Marking committed up to 6
0129 03:00:10.217450 (+ 60us) raft_consensus.cc:1697] Filling consensus response to leader.
0129 03:00:12.439286 (+2221836us) spinlock_profiling.cc:243] Waited 942 ms on lock 0x7b54002ae754. stack: 00007f8fe044e373 00007f8fe044e270 00007f8fdfda0574 00007f8fe9a5c6b4 00007f8fe9a5c649 00007f8fe34975a1 00007f8fe352ea81 00007f8fe352d4be 00007f8fe84e1f49 00007f8fe1a6ed34 00007f8fe1a6ecc2 00007f8fe1a6ebf1 00007f8fe1a6eb6c 00007f8fe1a6de22 00007f8fe186d34c 00007f8fe186c786
0129 03:00:12.439355 (+ 69us) raft_consensus.cc:1691] UpdateReplicas() finished
0129 03:00:12.439556 (+ 201us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":19772,"prepare.queue_time_us":722024,"prepare.run_cpu_time_us":42,"prepare.run_wall_time_us":43,"raft.queue_time_us":105619,"raft.run_cpu_time_us":120,"raft.run_wall_time_us":4779,"spinlock_wait_cycles":2166476544,"thread_start_us":740821,"threads_started":2,"wal-append.queue_time_us":749171,"wal-append.run_cpu_time_us":2032,"wal-append.run_wall_time_us":1397906}
I20260129 03:00:12.576835 20512 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "8055232cf0004d40a85f3bf983fe00a9" candidate_uuid: "197b880fa39d432a9b472845876bfaf0" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "974b802141bd4f13beffef2aa03f88f9"
I20260129 03:00:12.576256 20584 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "cedce331e1584ac8b636c9b8a72b2ba0" candidate_uuid: "197b880fa39d432a9b472845876bfaf0" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "1c50de89a5a845668696dcb28c3badd6"
I20260129 03:00:12.568970 20582 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "cb4727386a3e48aca5d71cc6c611c197" candidate_uuid: "197b880fa39d432a9b472845876bfaf0" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "1c50de89a5a845668696dcb28c3badd6" is_pre_election: true
I20260129 03:00:12.642691 20438 raft_consensus.cc:1240] T 368953c9102643b2a21f99af36b51905 P 197b880fa39d432a9b472845876bfaf0 [term 9 FOLLOWER]: Rejecting Update request from peer 974b802141bd4f13beffef2aa03f88f9 for earlier term 8. Current term is 9. Ops: [8.5-8.5]
W20260129 03:00:12.575338 20479 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.211.1:42027, user_credentials={real_user=slave}} blocked reactor thread for 52026.5us
I20260129 03:00:12.459390 23437 leader_election.cc:290] T 4f8f3a38f6e24ba4857875181f447442 P 197b880fa39d432a9b472845876bfaf0 [CANDIDATE]: Term 7 election: Requested vote from peers 974b802141bd4f13beffef2aa03f88f9 (127.18.211.2:34517), 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3:37735)
I20260129 03:00:12.525400 23215 raft_consensus.cc:3060] T 0f20af7fbe5d4db89a6f17948d98912c P 974b802141bd4f13beffef2aa03f88f9 [term 7 FOLLOWER]: Advancing to term 8
I20260129 03:00:12.648433 20439 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "75be4bd6bd45474795fd472eff2953ac" candidate_uuid: "1c50de89a5a845668696dcb28c3badd6" candidate_term: 11 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "197b880fa39d432a9b472845876bfaf0" is_pre_election: true
I20260129 03:00:12.524857 23375 raft_consensus.cc:2804] T 18f30ebaa4174fd28c958e72d0cc7e24 P 7a9daf5d42a84615a6383cfec9a10608 [term 8 FOLLOWER]: Leader election won for term 8
I20260129 03:00:12.577135 23365 raft_consensus.cc:3060] T 0678b8b468bd43cdbc3e485900844821 P 7a9daf5d42a84615a6383cfec9a10608 [term 8 FOLLOWER]: Advancing to term 9
I20260129 03:00:12.650825 20582 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "505580c93c694bbb8ea96fdf9b17ab46" candidate_uuid: "974b802141bd4f13beffef2aa03f88f9" candidate_term: 8 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "1c50de89a5a845668696dcb28c3badd6" is_pre_election: true
I20260129 03:00:12.691733 20582 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "8055232cf0004d40a85f3bf983fe00a9" candidate_uuid: "197b880fa39d432a9b472845876bfaf0" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "1c50de89a5a845668696dcb28c3badd6"
I20260129 03:00:12.655290 20435 raft_consensus.cc:1275] T dc2f4cb752cf4d59b5f6bc689f5d8104 P 197b880fa39d432a9b472845876bfaf0 [term 6 FOLLOWER]: Refusing update from remote peer 1c50de89a5a845668696dcb28c3badd6: Log matching property violated. Preceding OpId in replica: term: 4 index: 6. Preceding OpId from leader: term: 6 index: 7. (index mismatch)
I20260129 03:00:12.693459 20434 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "dc2f4cb752cf4d59b5f6bc689f5d8104" candidate_uuid: "7a9daf5d42a84615a6383cfec9a10608" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "197b880fa39d432a9b472845876bfaf0"
I20260129 03:00:12.694418 20434 raft_consensus.cc:2393] T dc2f4cb752cf4d59b5f6bc689f5d8104 P 197b880fa39d432a9b472845876bfaf0 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 7a9daf5d42a84615a6383cfec9a10608 in current term 6: Already voted for candidate 1c50de89a5a845668696dcb28c3badd6 in this term.
I20260129 03:00:12.333446 23402 raft_consensus.cc:3060] T 17d6431cc77a491e8464d68db4081a51 P 974b802141bd4f13beffef2aa03f88f9 [term 5 FOLLOWER]: Advancing to term 6
I20260129 03:00:12.646162 20584 raft_consensus.cc:2376] T cedce331e1584ac8b636c9b8a72b2ba0 P 1c50de89a5a845668696dcb28c3badd6 [term 8 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 197b880fa39d432a9b472845876bfaf0 in term 8. Re-sending same reply.
I20260129 03:00:12.696974 20584 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "17d6431cc77a491e8464d68db4081a51" candidate_uuid: "974b802141bd4f13beffef2aa03f88f9" candidate_term: 6 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "1c50de89a5a845668696dcb28c3badd6" is_pre_election: true
I20260129 03:00:12.697602 20584 raft_consensus.cc:2468] T 17d6431cc77a491e8464d68db4081a51 P 1c50de89a5a845668696dcb28c3badd6 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 974b802141bd4f13beffef2aa03f88f9 in term 5.
I20260129 03:00:12.698458 20584 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "dc2f4cb752cf4d59b5f6bc689f5d8104" candidate_uuid: "7a9daf5d42a84615a6383cfec9a10608" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "1c50de89a5a845668696dcb28c3badd6"
I20260129 03:00:12.699337 23087 consensus_queue.cc:1048] T cedce331e1584ac8b636c9b8a72b2ba0 P 197b880fa39d432a9b472845876bfaf0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 3, Time since last communication: 0.000s
I20260129 03:00:12.699517 20584 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "913fc7d751ed4b7388d889aac30a5f9b" candidate_uuid: "974b802141bd4f13beffef2aa03f88f9" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "1c50de89a5a845668696dcb28c3badd6"
W20260129 03:00:12.700295 20405 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.211.3:37735, user_credentials={real_user=slave}} blocked reactor thread for 152007us
I20260129 03:00:12.700691 23087 raft_consensus.cc:3060] T 724a236524cb4e7ba3428880681d6cc2 P 197b880fa39d432a9b472845876bfaf0 [term 5 FOLLOWER]: Advancing to term 6
I20260129 03:00:12.700774 20584 raft_consensus.cc:2376] T 913fc7d751ed4b7388d889aac30a5f9b P 1c50de89a5a845668696dcb28c3badd6 [term 5 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 974b802141bd4f13beffef2aa03f88f9 in term 5. Re-sending same reply.
I20260129 03:00:12.702030 20405 leader_election.cc:304] T 36ff33be666a4c82954731691dbcf327 P 197b880fa39d432a9b472845876bfaf0 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 197b880fa39d432a9b472845876bfaf0; no voters: 1c50de89a5a845668696dcb28c3badd6, 974b802141bd4f13beffef2aa03f88f9
I20260129 03:00:12.702159 20584 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "cb4727386a3e48aca5d71cc6c611c197" candidate_uuid: "974b802141bd4f13beffef2aa03f88f9" candidate_term: 6 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1c50de89a5a845668696dcb28c3badd6" is_pre_election: true
I20260129 03:00:12.702971 23089 raft_consensus.cc:2749] T 36ff33be666a4c82954731691dbcf327 P 197b880fa39d432a9b472845876bfaf0 [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260129 03:00:12.703183 20405 leader_election.cc:304] T 18f30ebaa4174fd28c958e72d0cc7e24 P 197b880fa39d432a9b472845876bfaf0 [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 197b880fa39d432a9b472845876bfaf0; no voters: 1c50de89a5a845668696dcb28c3badd6, 7a9daf5d42a84615a6383cfec9a10608
I20260129 03:00:12.703477 20584 tablet_service.cc:2066] Received LeaderStepDown RPC: tablet_id: "7d9494f5584d4d5db8b9adf1c8fc888b"
dest_uuid: "1c50de89a5a845668696dcb28c3badd6"
mode: GRACEFUL
new_leader_uuid: "197b880fa39d432a9b472845876bfaf0"
from {username='slave'} at 127.0.0.1:54860
I20260129 03:00:12.704027 23346 raft_consensus.cc:2749] T 18f30ebaa4174fd28c958e72d0cc7e24 P 197b880fa39d432a9b472845876bfaf0 [term 8 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
I20260129 03:00:12.704095 20584 raft_consensus.cc:606] T 7d9494f5584d4d5db8b9adf1c8fc888b P 1c50de89a5a845668696dcb28c3badd6 [term 4 LEADER]: Received request to transfer leadership to TS 197b880fa39d432a9b472845876bfaf0
I20260129 03:00:12.704531 20515 raft_consensus.cc:1275] T 505580c93c694bbb8ea96fdf9b17ab46 P 974b802141bd4f13beffef2aa03f88f9 [term 7 FOLLOWER]: Refusing update from remote peer 1c50de89a5a845668696dcb28c3badd6: Log matching property violated. Preceding OpId in replica: term: 5 index: 7. Preceding OpId from leader: term: 7 index: 9. (index mismatch)
I20260129 03:00:12.705705 20584 raft_consensus.cc:1217] T 1ce4ddf5b10e4ae8b193b13b2e5936ee P 1c50de89a5a845668696dcb28c3badd6 [term 7 FOLLOWER]: Deduplicated request from leader. Original: 1.1->[2.2-7.4] Dedup: 2.2->[5.3-7.4]
I20260129 03:00:12.705930 20515 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57584 (request call id 1466) took 1127 ms. Trace:
I20260129 03:00:12.706202 20584 pending_rounds.cc:85] T 1ce4ddf5b10e4ae8b193b13b2e5936ee P 1c50de89a5a845668696dcb28c3badd6: Aborting all ops after (but not including) 2
I20260129 03:00:12.706508 20584 pending_rounds.cc:107] T 1ce4ddf5b10e4ae8b193b13b2e5936ee P 1c50de89a5a845668696dcb28c3badd6: Aborting uncommitted NO_OP operation due to leader change: 4.3
I20260129 03:00:12.657783 20514 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4f8f3a38f6e24ba4857875181f447442" candidate_uuid: "197b880fa39d432a9b472845876bfaf0" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "974b802141bd4f13beffef2aa03f88f9"
I20260129 03:00:12.592789 20583 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3906b080a5b34a74b8b64128635e8997" candidate_uuid: "197b880fa39d432a9b472845876bfaf0" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "1c50de89a5a845668696dcb28c3badd6" is_pre_election: true
I20260129 03:00:12.706379 20515 rpcz_store.cc:276] 0129 03:00:11.578360 (+ 0us) service_pool.cc:168] Inserting onto call queue
0129 03:00:11.744683 (+166323us) service_pool.cc:225] Handling call
0129 03:00:12.704233 (+959550us) raft_consensus.cc:1471] Updating replica for 0 ops
0129 03:00:12.704920 (+ 687us) raft_consensus.cc:1697] Filling consensus response to leader.
0129 03:00:12.705751 (+ 831us) spinlock_profiling.cc:243] Waited 959 ms on lock 0x7b5400129150. stack: 00007f8fe044e373 00007f8fe044e270 00007f8fdfda0574 00007f8fe9a5c6b4 00007f8fe9a5c649 00007f8fe34975a1 00007f8fe352d52d 00007f8fe84e1f49 00007f8fe1a6ed34 00007f8fe1a6ecc2 00007f8fe1a6ebf1 00007f8fe1a6eb6c 00007f8fe1a6de22 00007f8fe186d34c 00007f8fe186c786 00007f8fe186c12f
0129 03:00:12.705867 (+ 116us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2206004096}
I20260129 03:00:12.706930 20405 leader_election.cc:304] T 4b1876f9a70b49edb3aa74105e66d3b3 P 197b880fa39d432a9b472845876bfaf0 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 197b880fa39d432a9b472845876bfaf0; no voters: 1c50de89a5a845668696dcb28c3badd6, 974b802141bd4f13beffef2aa03f88f9
I20260129 03:00:12.729574 20515 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c891d2bff4ce4568861bbc992e91fbb0" candidate_uuid: "1c50de89a5a845668696dcb28c3badd6" candidate_term: 9 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "974b802141bd4f13beffef2aa03f88f9" is_pre_election: true
I20260129 03:00:12.729861 20405 leader_election.cc:304] T 0340229d521b432580ae926f6062e3f7 P 197b880fa39d432a9b472845876bfaf0 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 197b880fa39d432a9b472845876bfaf0, 1c50de89a5a845668696dcb28c3badd6; no voters: 974b802141bd4f13beffef2aa03f88f9
W20260129 03:00:12.704129 20507 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:48544 (request call id 323) took 2388 ms (client timeout 1954 ms). Trace:
I20260129 03:00:12.730998 20405 leader_election.cc:304] T 7d9494f5584d4d5db8b9adf1c8fc888b P 197b880fa39d432a9b472845876bfaf0 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 197b880fa39d432a9b472845876bfaf0; no voters: 1c50de89a5a845668696dcb28c3badd6, 974b802141bd4f13beffef2aa03f88f9
W20260129 03:00:12.730885 20507 rpcz_store.cc:269] 0129 03:00:10.315704 (+ 0us) service_pool.cc:168] Inserting onto call queue
0129 03:00:10.341971 (+ 26267us) service_pool.cc:225] Handling call
0129 03:00:12.703953 (+2361982us) spinlock_profiling.cc:243] Waited 317 ms on lock 0x7b5400129154. stack: 00007f8fe044e373 00007f8fe044e270 00007f8fdfda0574 00007f8fe9a5c6b4 00007f8fe9a5c649 00007f8fe34975a1 00007f8fe3531f7a 00007f8fe84e3647 00007f8fe1a73834 00007f8fe1a737c2 00007f8fe1a736f1 00007f8fe1a7366c 00007f8fe1a72922 00007f8fe186d34c 00007f8fe186c786 00007f8fe186c12f
0129 03:00:12.704062 (+ 109us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":728308992}
I20260129 03:00:12.738086 20583 raft_consensus.cc:2393] T 3906b080a5b34a74b8b64128635e8997 P 1c50de89a5a845668696dcb28c3badd6 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 197b880fa39d432a9b472845876bfaf0 in current term 5: Already voted for candidate 1c50de89a5a845668696dcb28c3badd6 in this term.
I20260129 03:00:12.720104 20510 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "75be4bd6bd45474795fd472eff2953ac" candidate_uuid: "1c50de89a5a845668696dcb28c3badd6" candidate_term: 11 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "974b802141bd4f13beffef2aa03f88f9" is_pre_election: true
I20260129 03:00:12.637391 23450 raft_consensus.cc:3060] T c891d2bff4ce4568861bbc992e91fbb0 P 974b802141bd4f13beffef2aa03f88f9 [term 8 FOLLOWER]: Advancing to term 9
I20260129 03:00:12.366839 23026 leader_election.cc:290] T 4b1876f9a70b49edb3aa74105e66d3b3 P 1c50de89a5a845668696dcb28c3badd6 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 197b880fa39d432a9b472845876bfaf0 (127.18.211.1:42027), 974b802141bd4f13beffef2aa03f88f9 (127.18.211.2:34517)
I20260129 03:00:12.649483 20516 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c891d2bff4ce4568861bbc992e91fbb0" candidate_uuid: "197b880fa39d432a9b472845876bfaf0" candidate_term: 9 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "974b802141bd4f13beffef2aa03f88f9"
I20260129 03:00:12.745569 20405 leader_election.cc:304] T 2e58e9a5a8aa4cecafb464addb4f35ed P 197b880fa39d432a9b472845876bfaf0 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 197b880fa39d432a9b472845876bfaf0; no voters: 1c50de89a5a845668696dcb28c3badd6, 7a9daf5d42a84615a6383cfec9a10608
W20260129 03:00:12.674959 20406 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260129 03:00:12.369163 23378 leader_election.cc:290] T 2ac659d732314e1288849f08fe5e074e P 197b880fa39d432a9b472845876bfaf0 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 974b802141bd4f13beffef2aa03f88f9 (127.18.211.2:34517), 7a9daf5d42a84615a6383cfec9a10608 (127.18.211.4:40439)
I20260129 03:00:12.364156 23456 consensus_queue.cc:1048] T 36ff33be666a4c82954731691dbcf327 P 1c50de89a5a845668696dcb28c3badd6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.312s
I20260129 03:00:12.675783 22748 raft_consensus.cc:493] T 8055232cf0004d40a85f3bf983fe00a9 P 1c50de89a5a845668696dcb28c3badd6 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260129 03:00:12.575067 23514 raft_consensus.cc:2749] T 7d9494f5584d4d5db8b9adf1c8fc888b P 974b802141bd4f13beffef2aa03f88f9 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260129 03:00:12.767194 20516 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4b1876f9a70b49edb3aa74105e66d3b3" candidate_uuid: "1c50de89a5a845668696dcb28c3badd6" candidate_term: 7 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "974b802141bd4f13beffef2aa03f88f9" is_pre_election: true
I20260129 03:00:12.572933 23304 leader_election.cc:290] T 75be4bd6bd45474795fd472eff2953ac P 1c50de89a5a845668696dcb28c3badd6 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 197b880fa39d432a9b472845876bfaf0 (127.18.211.1:42027), 974b802141bd4f13beffef2aa03f88f9 (127.18.211.2:34517)
I20260129 03:00:12.472651 23344 raft_consensus.cc:2749] T a5a05fbcbbcf4916bbc01fd1a5b5e90d P 197b880fa39d432a9b472845876bfaf0 [term 8 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260129 03:00:12.680214 23490 consensus_queue.cc:260] T 36ff33be666a4c82954731691dbcf327 P 974b802141bd4f13beffef2aa03f88f9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 03:00:12.647305 23426 consensus_queue.cc:1048] T 913fc7d751ed4b7388d889aac30a5f9b P 974b802141bd4f13beffef2aa03f88f9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 2, Time since last communication: 0.035s
W20260129 03:00:12.682699 23411 log.cc:927] Time spent T a8479d1fe0bd41e5946186e3a6ca8c7a P 1c50de89a5a845668696dcb28c3badd6: Append to log took a long time: real 2.390s user 0.011s sys 0.014s
I20260129 03:00:12.683799 23427 consensus_queue.cc:1059] T e33a417cf9c54cf5af73a4ee3aac4d3a P 974b802141bd4f13beffef2aa03f88f9 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 }, Status: INVALID_TERM, Last received: 6.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
W20260129 03:00:12.684387 20437 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55672 (request call id 1735) took 2565 ms (client timeout 1510 ms). Trace:
W20260129 03:00:12.834707 20437 rpcz_store.cc:269] 0129 03:00:10.118656 (+ 0us) service_pool.cc:168] Inserting onto call queue
0129 03:00:10.131532 (+ 12876us) service_pool.cc:225] Handling call
0129 03:00:12.684126 (+2552594us) spinlock_profiling.cc:243] Waited 50.4 ms on lock 0x7b54000a64d4. stack: 00007f8fe044e373 00007f8fe044e270 00007f8fdfda0574 00007f8fe9a5c6b4 00007f8fe9a5c649 00007f8fe34975a1 00007f8fe3531f7a 00007f8fe84e3647 00007f8fe1a73834 00007f8fe1a737c2 00007f8fe1a736f1 00007f8fe1a7366c 00007f8fe1a72922 00007f8fe186d34c 00007f8fe186c786 00007f8fe186c12f
0129 03:00:12.684304 (+ 178us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":115875200}
I20260129 03:00:12.836164 20437 raft_consensus.cc:1240] T c2633a6fc34c44579de348d208b0c01e P 197b880fa39d432a9b472845876bfaf0 [term 5 FOLLOWER]: Rejecting Update request from peer 1c50de89a5a845668696dcb28c3badd6 for earlier term 4. Current term is 5. Ops: [4.3-4.3]
I20260129 03:00:12.686095 23283 consensus_queue.cc:1048] T cb4727386a3e48aca5d71cc6c611c197 P 1c50de89a5a845668696dcb28c3badd6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260129 03:00:12.690057 23208 leader_election.cc:290] T 00e79d8f25a34676b1d866eb2068a2b2 P 974b802141bd4f13beffef2aa03f88f9 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 197b880fa39d432a9b472845876bfaf0 (127.18.211.1:42027), 7a9daf5d42a84615a6383cfec9a10608 (127.18.211.4:40439)
I20260129 03:00:12.690300 20432 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4b1876f9a70b49edb3aa74105e66d3b3" candidate_uuid: "1c50de89a5a845668696dcb28c3badd6" candidate_term: 7 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "197b880fa39d432a9b472845876bfaf0" is_pre_election: true
I20260129 03:00:12.730217 20514 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0678b8b468bd43cdbc3e485900844821" candidate_uuid: "197b880fa39d432a9b472845876bfaf0" candidate_term: 9 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "974b802141bd4f13beffef2aa03f88f9" is_pre_election: true
I20260129 03:00:12.839069 20514 raft_consensus.cc:1275] T 87d5fe48a44a408aaf03e894a67ecb2f P 974b802141bd4f13beffef2aa03f88f9 [term 7 FOLLOWER]: Refusing update from remote peer 197b880fa39d432a9b472845876bfaf0: Log matching property violated. Preceding OpId in replica: term: 5 index: 7. Preceding OpId from leader: term: 7 index: 8. (index mismatch)
I20260129 03:00:12.740424 20510 raft_consensus.cc:2410] T 75be4bd6bd45474795fd472eff2953ac P 974b802141bd4f13beffef2aa03f88f9 [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1c50de89a5a845668696dcb28c3badd6 for term 11 because replica has last-logged OpId of term: 10 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 9 index: 6.
I20260129 03:00:12.664901 20511 raft_consensus.cc:2468] T 2ac659d732314e1288849f08fe5e074e P 974b802141bd4f13beffef2aa03f88f9 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 197b880fa39d432a9b472845876bfaf0 in term 6.
I20260129 03:00:12.669734 20587 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e4d18c35b0a7493e85d0a4b92956a48f" candidate_uuid: "197b880fa39d432a9b472845876bfaf0" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "1c50de89a5a845668696dcb28c3badd6" is_pre_election: true
W20260129 03:00:12.577450 23429 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 1.458s user 0.004s sys 0.026s
I20260129 03:00:12.753301 20507 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e4d18c35b0a7493e85d0a4b92956a48f" candidate_uuid: "1c50de89a5a845668696dcb28c3badd6" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "974b802141bd4f13beffef2aa03f88f9" is_pre_election: true
I20260129 03:00:12.706851 20584 raft_consensus.cc:2889] T 1ce4ddf5b10e4ae8b193b13b2e5936ee P 1c50de89a5a845668696dcb28c3badd6 [term 7 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260129 03:00:12.721128 22989 raft_consensus.cc:993] T 4f8f3a38f6e24ba4857875181f447442 P 974b802141bd4f13beffef2aa03f88f9: : Instructing follower 1c50de89a5a845668696dcb28c3badd6 to start an election
I20260129 03:00:12.756655 20405 leader_election.cc:304] T 76f713c3331540d2a5b76164103e9eb1 P 197b880fa39d432a9b472845876bfaf0 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 197b880fa39d432a9b472845876bfaf0; no voters: 1c50de89a5a845668696dcb28c3badd6, 974b802141bd4f13beffef2aa03f88f9
W20260129 03:00:12.856184 20405 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.211.3:37735, user_credentials={real_user=slave}} blocked reactor thread for 99637.3us
I20260129 03:00:12.857157 20405 leader_election.cc:304] T cb4727386a3e48aca5d71cc6c611c197 P 197b880fa39d432a9b472845876bfaf0 [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 197b880fa39d432a9b472845876bfaf0; no voters: 1c50de89a5a845668696dcb28c3badd6, 974b802141bd4f13beffef2aa03f88f9
I20260129 03:00:12.858563 20405 leader_election.cc:304] T a8479d1fe0bd41e5946186e3a6ca8c7a P 197b880fa39d432a9b472845876bfaf0 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 197b880fa39d432a9b472845876bfaf0; no voters: 1c50de89a5a845668696dcb28c3badd6, 974b802141bd4f13beffef2aa03f88f9
I20260129 03:00:12.757915 19276 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 2
I20260129 03:00:12.859406 20407 leader_election.cc:304] T 2ac659d732314e1288849f08fe5e074e P 197b880fa39d432a9b472845876bfaf0 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 197b880fa39d432a9b472845876bfaf0, 974b802141bd4f13beffef2aa03f88f9; no voters:
I20260129 03:00:12.579466 23449 consensus_queue.cc:1059] T 724a236524cb4e7ba3428880681d6cc2 P 974b802141bd4f13beffef2aa03f88f9 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 }, Status: INVALID_TERM, Last received: 0.0, Next index: 4, Last known committed idx: 2, Time since last communication: 0.154s
I20260129 03:00:12.711711 23403 raft_consensus.cc:3060] T c934d31efd3f4e5195605ddc354d14e1 P 197b880fa39d432a9b472845876bfaf0 [term 5 FOLLOWER]: Advancing to term 6
I20260129 03:00:12.709561 23470 raft_consensus.cc:3055] T c2633a6fc34c44579de348d208b0c01e P 1c50de89a5a845668696dcb28c3badd6 [term 4 LEADER]: Stepping down as leader of term 4
I20260129 03:00:12.714310 23474 raft_consensus.cc:493] T 4408006eb5174de1896b037cf3f18457 P 1c50de89a5a845668696dcb28c3badd6 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 197b880fa39d432a9b472845876bfaf0)
I20260129 03:00:12.735121 23466 raft_consensus.cc:3060] T 4b1876f9a70b49edb3aa74105e66d3b3 P 197b880fa39d432a9b472845876bfaf0 [term 5 FOLLOWER]: Advancing to term 6
W20260129 03:00:12.879421 20479 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.211.1:42027, user_credentials={real_user=slave}} blocked reactor thread for 232529us
I20260129 03:00:12.864631 23490 raft_consensus.cc:3060] T 36ff33be666a4c82954731691dbcf327 P 974b802141bd4f13beffef2aa03f88f9 [term 3 FOLLOWER]: Advancing to term 4
I20260129 03:00:12.816272 20590 raft_consensus.cc:2468] T c891d2bff4ce4568861bbc992e91fbb0 P 1c50de89a5a845668696dcb28c3badd6 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 974b802141bd4f13beffef2aa03f88f9 in term 8.
I20260129 03:00:12.656774 23492 consensus_queue.cc:1048] T 368953c9102643b2a21f99af36b51905 P 974b802141bd4f13beffef2aa03f88f9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.106s
I20260129 03:00:12.818179 21131 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34046 (request call id 571) took 1925 ms. Trace:
I20260129 03:00:12.758126 23089 raft_consensus.cc:2804] T 0340229d521b432580ae926f6062e3f7 P 197b880fa39d432a9b472845876bfaf0 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260129 03:00:12.820140 21099 leader_election.cc:304] T dc2f4cb752cf4d59b5f6bc689f5d8104 P 7a9daf5d42a84615a6383cfec9a10608 [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7a9daf5d42a84615a6383cfec9a10608; no voters: 197b880fa39d432a9b472845876bfaf0, 1c50de89a5a845668696dcb28c3badd6
I20260129 03:00:12.866780 23436 raft_consensus.cc:2749] T 76f713c3331540d2a5b76164103e9eb1 P 197b880fa39d432a9b472845876bfaf0 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260129 03:00:12.766783 23026 consensus_queue.cc:1048] T dc2f4cb752cf4d59b5f6bc689f5d8104 P 1c50de89a5a845668696dcb28c3badd6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.061s
I20260129 03:00:12.867240 23378 raft_consensus.cc:2804] T 2ac659d732314e1288849f08fe5e074e P 197b880fa39d432a9b472845876bfaf0 [term 6 FOLLOWER]: Leader election won for term 6
I20260129 03:00:12.776834 22748 raft_consensus.cc:515] T 8055232cf0004d40a85f3bf983fe00a9 P 1c50de89a5a845668696dcb28c3badd6 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 03:00:12.761777 23366 raft_consensus.cc:2749] T 2e58e9a5a8aa4cecafb464addb4f35ed P 197b880fa39d432a9b472845876bfaf0 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260129 03:00:12.762449 23375 raft_consensus.cc:697] T 18f30ebaa4174fd28c958e72d0cc7e24 P 7a9daf5d42a84615a6383cfec9a10608 [term 8 LEADER]: Becoming Leader. State: Replica: 7a9daf5d42a84615a6383cfec9a10608, State: Running, Role: LEADER
I20260129 03:00:12.842496 20587 raft_consensus.cc:2468] T e4d18c35b0a7493e85d0a4b92956a48f P 1c50de89a5a845668696dcb28c3badd6 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 197b880fa39d432a9b472845876bfaf0 in term 5.
W20260129 03:00:12.876339 20480 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.211.3:37735, user_credentials={real_user=slave}} blocked reactor thread for 230940us
I20260129 03:00:12.878136 20585 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35752 (request call id 1676) took 4801 ms. Trace:
I20260129 03:00:12.768551 20516 raft_consensus.cc:2468] T 4b1876f9a70b49edb3aa74105e66d3b3 P 974b802141bd4f13beffef2aa03f88f9 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1c50de89a5a845668696dcb28c3badd6 in term 6.
W20260129 03:00:12.881351 20556 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.211.2:34517, user_credentials={real_user=slave}} blocked reactor thread for 212449us
I20260129 03:00:12.900162 20556 leader_election.cc:304] T 1ce4ddf5b10e4ae8b193b13b2e5936ee P 1c50de89a5a845668696dcb28c3badd6 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1c50de89a5a845668696dcb28c3badd6; no voters: 197b880fa39d432a9b472845876bfaf0, 974b802141bd4f13beffef2aa03f88f9
I20260129 03:00:12.901752 20556 leader_election.cc:304] T e7d83d4ccf684e76acb29a0dc844e1b8 P 1c50de89a5a845668696dcb28c3badd6 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1c50de89a5a845668696dcb28c3badd6; no voters: 197b880fa39d432a9b472845876bfaf0, 974b802141bd4f13beffef2aa03f88f9
I20260129 03:00:12.902653 23385 raft_consensus.cc:2749] T a8479d1fe0bd41e5946186e3a6ca8c7a P 197b880fa39d432a9b472845876bfaf0 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
W20260129 03:00:12.865834 23515 log.cc:927] Time spent T c2633a6fc34c44579de348d208b0c01e P 974b802141bd4f13beffef2aa03f88f9: Append to log took a long time: real 0.123s user 0.001s sys 0.000s
I20260129 03:00:12.903780 22748 leader_election.cc:290] T 8055232cf0004d40a85f3bf983fe00a9 P 1c50de89a5a845668696dcb28c3badd6 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 197b880fa39d432a9b472845876bfaf0 (127.18.211.1:42027), 974b802141bd4f13beffef2aa03f88f9 (127.18.211.2:34517)
I20260129 03:00:12.904984 23375 consensus_queue.cc:237] T 18f30ebaa4174fd28c958e72d0cc7e24 P 7a9daf5d42a84615a6383cfec9a10608 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 6.7, Last appended by leader: 7, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } }
I20260129 03:00:12.905539 23484 consensus_queue.cc:1048] T 87d5fe48a44a408aaf03e894a67ecb2f P 197b880fa39d432a9b472845876bfaf0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.027s
W20260129 03:00:12.907402 23516 log.cc:927] Time spent T e7d83d4ccf684e76acb29a0dc844e1b8 P 974b802141bd4f13beffef2aa03f88f9: Append to log took a long time: real 0.073s user 0.000s sys 0.000s
I20260129 03:00:12.909550 23205 raft_consensus.cc:3060] T febffe4a8b394368b69b1d164308fb9a P 974b802141bd4f13beffef2aa03f88f9 [term 4 FOLLOWER]: Advancing to term 5
I20260129 03:00:12.884027 20405 leader_election.cc:304] T 17d6431cc77a491e8464d68db4081a51 P 197b880fa39d432a9b472845876bfaf0 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 197b880fa39d432a9b472845876bfaf0, 1c50de89a5a845668696dcb28c3badd6; no voters: 974b802141bd4f13beffef2aa03f88f9
I20260129 03:00:12.897420 20585 rpcz_store.cc:276] 0129 03:00:08.077269 (+ 0us) service_pool.cc:168] Inserting onto call queue
0129 03:00:09.688742 (+1611473us) service_pool.cc:225] Handling call
0129 03:00:09.732187 (+ 43445us) raft_consensus.cc:1471] Updating replica for 1 ops
0129 03:00:09.740663 (+ 8476us) raft_consensus.cc:1534] Early marking committed up to index 0
0129 03:00:09.740717 (+ 54us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0129 03:00:09.741275 (+ 558us) log.cc:844] Serialized 26 byte log entry
0129 03:00:10.080778 (+339503us) raft_consensus.cc:1645] Marking committed up to 2
0129 03:00:10.080834 (+ 56us) raft_consensus.cc:1697] Filling consensus response to leader.
0129 03:00:10.080938 (+ 104us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0129 03:00:12.682092 (+2601154us) spinlock_profiling.cc:243] Waited 3.5 ms on lock 0x7b440002304c. stack: 00007f8fe044e373 00007f8fe044e270 00007f8fdfda0574 00007f8fe9a5c6b4 00007f8fe9a5c649 00007f8fe02c08b1 00007f8fe038f005 00007f8fe038fa22 00007f8fe30d957b 00007f8fe30d4e4f 00007f8fe30c8c0e 00007f8fe30c7f56 00007f8fe30c7b78 00007f8fe30d15c1 00007f8fe30d1579 00007f8fe30d1509
0129 03:00:12.870151 (+188059us) raft_consensus.cc:1685] finished
0129 03:00:12.870185 (+ 34us) raft_consensus.cc:1691] UpdateReplicas() finished
0129 03:00:12.878062 (+ 7877us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":8033,"raft.queue_time_us":12465,"raft.run_cpu_time_us":94,"raft.run_wall_time_us":95,"spinlock_wait_cycles":9211520,"thread_start_us":326309,"threads_started":1,"wal-append.queue_time_us":507054}
I20260129 03:00:12.912396 20480 leader_election.cc:304] T a4c1226b987b468fa84dfc6ba0e2dbe1 P 974b802141bd4f13beffef2aa03f88f9 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 974b802141bd4f13beffef2aa03f88f9; no voters: 197b880fa39d432a9b472845876bfaf0, 1c50de89a5a845668696dcb28c3badd6
I20260129 03:00:12.916848 20556 leader_election.cc:304] T 96609ac68bdf4993a12e1c23355ca013 P 1c50de89a5a845668696dcb28c3badd6 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 1c50de89a5a845668696dcb28c3badd6; no voters: 197b880fa39d432a9b472845876bfaf0, 974b802141bd4f13beffef2aa03f88f9
I20260129 03:00:12.918522 20405 leader_election.cc:304] T 78aa39096868415d9afc2ffcb248e198 P 197b880fa39d432a9b472845876bfaf0 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 197b880fa39d432a9b472845876bfaf0; no voters: 1c50de89a5a845668696dcb28c3badd6, 7a9daf5d42a84615a6383cfec9a10608
I20260129 03:00:12.645895 22694 consensus_queue.cc:1048] T 0340229d521b432580ae926f6062e3f7 P 974b802141bd4f13beffef2aa03f88f9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.108s
W20260129 03:00:12.887676 21099 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.211.1:42027, user_credentials={real_user=slave}} blocked reactor thread for 70409.3us
I20260129 03:00:12.760567 23437 raft_consensus.cc:2749] T 7d9494f5584d4d5db8b9adf1c8fc888b P 197b880fa39d432a9b472845876bfaf0 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260129 03:00:12.867069 22645 raft_consensus.cc:2749] T cb4727386a3e48aca5d71cc6c611c197 P 197b880fa39d432a9b472845876bfaf0 [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
W20260129 03:00:12.828565 23517 log.cc:927] Time spent T 0678b8b468bd43cdbc3e485900844821 P 974b802141bd4f13beffef2aa03f88f9: Append to log took a long time: real 0.086s user 0.000s sys 0.000s
I20260129 03:00:12.867384 23470 raft_consensus.cc:740] T c2633a6fc34c44579de348d208b0c01e P 1c50de89a5a845668696dcb28c3badd6 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 1c50de89a5a845668696dcb28c3badd6, State: Running, Role: LEADER
I20260129 03:00:12.885934 21131 rpcz_store.cc:276] 0129 03:00:10.892755 (+ 0us) service_pool.cc:168] Inserting onto call queue
0129 03:00:10.892952 (+ 197us) service_pool.cc:225] Handling call
0129 03:00:12.329452 (+1436500us) spinlock_profiling.cc:243] Waited 3.67 ms on lock 0x7b5800111f98. stack: 00007f8fe044e373 00007f8fe044e270 00007f8fdfda0574 00007f8fe9a5c6b4 00007f8fe9a5c649 00007f8fe02c08b1 00007f8fe0442db1 00007f8fe0442c62 00007f8fe92eddd9 00007f8fe846d260 00007f8fe8535dbb 00007f8fe8535e41 00007f8fe84f7af0 00007f8fe84e2459 00007f8fe84e1f02 00007f8fe1a6ed34
0129 03:00:12.329646 (+ 194us) raft_consensus.cc:1471] Updating replica for 0 ops
0129 03:00:12.329868 (+ 222us) raft_consensus.cc:1534] Early marking committed up to index 7
0129 03:00:12.507541 (+177673us) log.cc:844] Serialized 14 byte log entry
0129 03:00:12.817815 (+310274us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0129 03:00:12.817868 (+ 53us) raft_consensus.cc:1645] Marking committed up to 7
0129 03:00:12.817917 (+ 49us) raft_consensus.cc:1697] Filling consensus response to leader.
0129 03:00:12.818003 (+ 86us) raft_consensus.cc:1691] UpdateReplicas() finished
0129 03:00:12.818124 (+ 121us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":54136,"prepare.queue_time_us":307774,"prepare.run_cpu_time_us":51,"prepare.run_wall_time_us":54,"raft.queue_time_us":54268,"raft.run_cpu_time_us":158,"raft.run_wall_time_us":12456,"spinlock_wait_cycles":8435584,"thread_start_us":433025,"threads_started":2}
I20260129 03:00:12.904997 20582 raft_consensus.cc:3060] T 8055232cf0004d40a85f3bf983fe00a9 P 1c50de89a5a845668696dcb28c3badd6 [term 5 FOLLOWER]: Advancing to term 6
I20260129 03:00:12.886845 23089 raft_consensus.cc:493] T 0340229d521b432580ae926f6062e3f7 P 197b880fa39d432a9b472845876bfaf0 [term 7 FOLLOWER]: Starting leader election (detected failure of leader 974b802141bd4f13beffef2aa03f88f9)
I20260129 03:00:12.928094 23398 raft_consensus.cc:2749] T 96609ac68bdf4993a12e1c23355ca013 P 1c50de89a5a845668696dcb28c3badd6 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260129 03:00:12.891264 22932 raft_consensus.cc:2749] T dc2f4cb752cf4d59b5f6bc689f5d8104 P 7a9daf5d42a84615a6383cfec9a10608 [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20260129 03:00:12.881922 23474 raft_consensus.cc:515] T 4408006eb5174de1896b037cf3f18457 P 1c50de89a5a845668696dcb28c3badd6 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 03:00:12.927584 20556 leader_election.cc:304] T 0f20af7fbe5d4db89a6f17948d98912c P 1c50de89a5a845668696dcb28c3badd6 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1c50de89a5a845668696dcb28c3badd6; no voters: 197b880fa39d432a9b472845876bfaf0, 974b802141bd4f13beffef2aa03f88f9
W20260129 03:00:12.998060 20556 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.211.2:34517, user_credentials={real_user=slave}} blocked reactor thread for 70581.8us
I20260129 03:00:12.999140 20556 leader_election.cc:304] T 75b0b6e82ddc4f92b1fe1d94f9a0b73b P 1c50de89a5a845668696dcb28c3badd6 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1c50de89a5a845668696dcb28c3badd6, 974b802141bd4f13beffef2aa03f88f9; no voters:
I20260129 03:00:13.002498 20556 leader_election.cc:304] T a4c1226b987b468fa84dfc6ba0e2dbe1 P 1c50de89a5a845668696dcb28c3badd6 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1c50de89a5a845668696dcb28c3badd6; no voters: 197b880fa39d432a9b472845876bfaf0, 974b802141bd4f13beffef2aa03f88f9
I20260129 03:00:12.966265 20480 leader_election.cc:304] T cb4727386a3e48aca5d71cc6c611c197 P 974b802141bd4f13beffef2aa03f88f9 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 974b802141bd4f13beffef2aa03f88f9; no voters: 197b880fa39d432a9b472845876bfaf0, 1c50de89a5a845668696dcb28c3badd6
I20260129 03:00:12.924353 22748 raft_consensus.cc:2749] T e7d83d4ccf684e76acb29a0dc844e1b8 P 1c50de89a5a845668696dcb28c3badd6 [term 7 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260129 03:00:12.948971 20405 leader_election.cc:304] T a1a39dcd328e4268ae41b988c66b4a75 P 197b880fa39d432a9b472845876bfaf0 [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 197b880fa39d432a9b472845876bfaf0; no voters: 1c50de89a5a845668696dcb28c3badd6, 974b802141bd4f13beffef2aa03f88f9
I20260129 03:00:12.910955 23449 raft_consensus.cc:1081] T 4f8f3a38f6e24ba4857875181f447442 P 974b802141bd4f13beffef2aa03f88f9 [term 6 LEADER]: Signalling peer 1c50de89a5a845668696dcb28c3badd6 to start an election
I20260129 03:00:12.926753 23475 raft_consensus.cc:2804] T 17d6431cc77a491e8464d68db4081a51 P 197b880fa39d432a9b472845876bfaf0 [term 5 FOLLOWER]: Leader election won for term 5
I20260129 03:00:13.012447 20510 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c934d31efd3f4e5195605ddc354d14e1" candidate_uuid: "7a9daf5d42a84615a6383cfec9a10608" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "974b802141bd4f13beffef2aa03f88f9" is_pre_election: true
I20260129 03:00:12.924887 23378 raft_consensus.cc:697] T 2ac659d732314e1288849f08fe5e074e P 197b880fa39d432a9b472845876bfaf0 [term 6 LEADER]: Becoming Leader. State: Replica: 197b880fa39d432a9b472845876bfaf0, State: Running, Role: LEADER
W20260129 03:00:13.017001 20405 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.211.3:37735, user_credentials={real_user=slave}} blocked reactor thread for 68168.5us
I20260129 03:00:12.965191 23469 raft_consensus.cc:2749] T 78aa39096868415d9afc2ffcb248e198 P 197b880fa39d432a9b472845876bfaf0 [term 7 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260129 03:00:13.020988 20514 raft_consensus.cc:1275] T 45cc5f3d2688486d9b6dbe7e666dedde P 974b802141bd4f13beffef2aa03f88f9 [term 6 FOLLOWER]: Refusing update from remote peer 7a9daf5d42a84615a6383cfec9a10608: Log matching property violated. Preceding OpId in replica: term: 3 index: 5. Preceding OpId from leader: term: 6 index: 6. (index mismatch)
I20260129 03:00:12.972234 23497 consensus_queue.cc:1059] T c2633a6fc34c44579de348d208b0c01e P 1c50de89a5a845668696dcb28c3badd6 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 }, Status: INVALID_TERM, Last received: 2.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.087s
I20260129 03:00:13.014966 23292 raft_consensus.cc:2804] T 75b0b6e82ddc4f92b1fe1d94f9a0b73b P 1c50de89a5a845668696dcb28c3badd6 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260129 03:00:12.973141 23089 raft_consensus.cc:3060] T 0340229d521b432580ae926f6062e3f7 P 197b880fa39d432a9b472845876bfaf0 [term 7 FOLLOWER]: Advancing to term 8
I20260129 03:00:13.050258 20510 raft_consensus.cc:2393] T c934d31efd3f4e5195605ddc354d14e1 P 974b802141bd4f13beffef2aa03f88f9 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 7a9daf5d42a84615a6383cfec9a10608 in current term 6: Already voted for candidate 974b802141bd4f13beffef2aa03f88f9 in this term.
W20260129 03:00:13.050716 20556 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.211.2:34517, user_credentials={real_user=slave}} blocked reactor thread for 48296.6us
I20260129 03:00:13.082543 20556 leader_election.cc:304] T 724a236524cb4e7ba3428880681d6cc2 P 1c50de89a5a845668696dcb28c3badd6 [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1c50de89a5a845668696dcb28c3badd6; no voters: 197b880fa39d432a9b472845876bfaf0, 974b802141bd4f13beffef2aa03f88f9
I20260129 03:00:13.084120 20556 leader_election.cc:304] T b189f08e16a74a11980cc55d99359a0a P 1c50de89a5a845668696dcb28c3badd6 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1c50de89a5a845668696dcb28c3badd6; no voters: 197b880fa39d432a9b472845876bfaf0, 974b802141bd4f13beffef2aa03f88f9
I20260129 03:00:13.085620 20556 leader_election.cc:304] T 75f45538dc704d9a8ec4ff8ad9f3ea5b P 1c50de89a5a845668696dcb28c3badd6 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1c50de89a5a845668696dcb28c3badd6, 974b802141bd4f13beffef2aa03f88f9; no voters:
I20260129 03:00:13.059120 20514 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "505580c93c694bbb8ea96fdf9b17ab46" candidate_uuid: "7a9daf5d42a84615a6383cfec9a10608" candidate_term: 8 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "974b802141bd4f13beffef2aa03f88f9" is_pre_election: true
I20260129 03:00:13.087000 20514 raft_consensus.cc:2468] T 505580c93c694bbb8ea96fdf9b17ab46 P 974b802141bd4f13beffef2aa03f88f9 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7a9daf5d42a84615a6383cfec9a10608 in term 7.
I20260129 03:00:13.087237 20582 raft_consensus.cc:2468] T 8055232cf0004d40a85f3bf983fe00a9 P 1c50de89a5a845668696dcb28c3badd6 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 197b880fa39d432a9b472845876bfaf0 in term 6.
I20260129 03:00:13.088230 20514 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0678b8b468bd43cdbc3e485900844821" candidate_uuid: "7a9daf5d42a84615a6383cfec9a10608" candidate_term: 9 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "974b802141bd4f13beffef2aa03f88f9" is_pre_election: true
I20260129 03:00:13.089605 20514 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "368953c9102643b2a21f99af36b51905" candidate_uuid: "197b880fa39d432a9b472845876bfaf0" candidate_term: 9 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "974b802141bd4f13beffef2aa03f88f9"
I20260129 03:00:13.090924 20514 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00e79d8f25a34676b1d866eb2068a2b2" candidate_uuid: "7a9daf5d42a84615a6383cfec9a10608" candidate_term: 9 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "974b802141bd4f13beffef2aa03f88f9" is_pre_election: true
I20260129 03:00:12.957531 23208 consensus_queue.cc:1059] T 3906b080a5b34a74b8b64128635e8997 P 974b802141bd4f13beffef2aa03f88f9 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 }, Status: INVALID_TERM, Last received: 4.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.061s
I20260129 03:00:13.093089 23526 raft_consensus.cc:2749] T 724a236524cb4e7ba3428880681d6cc2 P 1c50de89a5a845668696dcb28c3badd6 [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20260129 03:00:13.094043 23526 raft_consensus.cc:2749] T b189f08e16a74a11980cc55d99359a0a P 1c50de89a5a845668696dcb28c3badd6 [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20260129 03:00:13.094784 23526 raft_consensus.cc:2804] T 75f45538dc704d9a8ec4ff8ad9f3ea5b P 1c50de89a5a845668696dcb28c3badd6 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260129 03:00:13.095194 23526 raft_consensus.cc:493] T 75f45538dc704d9a8ec4ff8ad9f3ea5b P 1c50de89a5a845668696dcb28c3badd6 [term 4 FOLLOWER]: Starting leader election (detected failure of leader 7a9daf5d42a84615a6383cfec9a10608)
I20260129 03:00:13.095605 23526 raft_consensus.cc:3060] T 75f45538dc704d9a8ec4ff8ad9f3ea5b P 1c50de89a5a845668696dcb28c3badd6 [term 4 FOLLOWER]: Advancing to term 5
I20260129 03:00:13.104732 20514 raft_consensus.cc:1240] T 2ac659d732314e1288849f08fe5e074e P 974b802141bd4f13beffef2aa03f88f9 [term 6 FOLLOWER]: Rejecting Update request from peer 7a9daf5d42a84615a6383cfec9a10608 for earlier term 5. Current term is 6. Ops: [5.6-5.6]
W20260129 03:00:13.113576 20480 connection.cc:232] RPC call timeout handler was delayed by -1.3884421359999806s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.21320000000000003s for a response
I20260129 03:00:13.055393 23344 raft_consensus.cc:2749] T a1a39dcd328e4268ae41b988c66b4a75 P 197b880fa39d432a9b472845876bfaf0 [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20260129 03:00:13.050765 23492 raft_consensus.cc:2749] T a4c1226b987b468fa84dfc6ba0e2dbe1 P 974b802141bd4f13beffef2aa03f88f9 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
W20260129 03:00:13.115353 20553 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.211.1:42027, user_credentials={real_user=slave}} blocked reactor thread for 262876us
I20260129 03:00:13.056977 23026 raft_consensus.cc:493] T a0fcc58d6fb6473cafbd9d34d5195269 P 1c50de89a5a845668696dcb28c3badd6 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260129 03:00:13.064905 22748 raft_consensus.cc:2749] T a4c1226b987b468fa84dfc6ba0e2dbe1 P 1c50de89a5a845668696dcb28c3badd6 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260129 03:00:13.123821 20510 raft_consensus.cc:1275] T a609ac02fae84725ab743f95f51d33ce P 974b802141bd4f13beffef2aa03f88f9 [term 9 FOLLOWER]: Refusing update from remote peer 197b880fa39d432a9b472845876bfaf0: Log matching property violated. Preceding OpId in replica: term: 6 index: 7. Preceding OpId from leader: term: 9 index: 9. (index mismatch)
I20260129 03:00:13.134575 20510 raft_consensus.cc:1275] T 926a250fa5834e88b96ccecbefbf5086 P 974b802141bd4f13beffef2aa03f88f9 [term 8 FOLLOWER]: Refusing update from remote peer 197b880fa39d432a9b472845876bfaf0: Log matching property violated. Preceding OpId in replica: term: 6 index: 4. Preceding OpId from leader: term: 8 index: 5. (index mismatch)
I20260129 03:00:13.128413 20440 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55672 (request call id 1749) took 1316 ms. Trace:
I20260129 03:00:13.135404 20440 rpcz_store.cc:276] 0129 03:00:11.811613 (+ 0us) service_pool.cc:168] Inserting onto call queue
0129 03:00:11.813938 (+ 2325us) service_pool.cc:225] Handling call
0129 03:00:12.017431 (+203493us) raft_consensus.cc:1471] Updating replica for 0 ops
0129 03:00:12.022629 (+ 5198us) raft_consensus.cc:1697] Filling consensus response to leader.
0129 03:00:13.128142 (+1105513us) spinlock_profiling.cc:243] Waited 181 ms on lock 0x7b54002cf6d0. stack: 00007f8fe044e373 00007f8fe044e270 00007f8fdfda0574 00007f8fe9a5c6b4 00007f8fe9a5c649 00007f8fe34975a1 00007f8fe352d52d 00007f8fe84e1f49 00007f8fe1a6ed34 00007f8fe1a6ecc2 00007f8fe1a6ebf1 00007f8fe1a6eb6c 00007f8fe1a6de22 00007f8fe186d34c 00007f8fe186c786 00007f8fe186c12f
0129 03:00:13.128331 (+ 189us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":415819648}
W20260129 03:00:13.143790 20406 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260129 03:00:13.073777 23292 raft_consensus.cc:493] T 75b0b6e82ddc4f92b1fe1d94f9a0b73b P 1c50de89a5a845668696dcb28c3badd6 [term 4 FOLLOWER]: Starting leader election (detected failure of leader 7a9daf5d42a84615a6383cfec9a10608)
I20260129 03:00:13.074411 23475 raft_consensus.cc:697] T 17d6431cc77a491e8464d68db4081a51 P 197b880fa39d432a9b472845876bfaf0 [term 5 LEADER]: Becoming Leader. State: Replica: 197b880fa39d432a9b472845876bfaf0, State: Running, Role: LEADER
I20260129 03:00:13.112584 23485 leader_election.cc:290] T 368953c9102643b2a21f99af36b51905 P 197b880fa39d432a9b472845876bfaf0 [CANDIDATE]: Term 9 election: Requested vote from peers 974b802141bd4f13beffef2aa03f88f9 (127.18.211.2:34517), 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3:37735)
I20260129 03:00:13.081385 23524 raft_consensus.cc:2749] T cb4727386a3e48aca5d71cc6c611c197 P 974b802141bd4f13beffef2aa03f88f9 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
W20260129 03:00:13.206959 20553 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260129 03:00:13.208815 20440 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "8055232cf0004d40a85f3bf983fe00a9" candidate_uuid: "1c50de89a5a845668696dcb28c3badd6" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "197b880fa39d432a9b472845876bfaf0" is_pre_election: true
I20260129 03:00:13.229482 20440 raft_consensus.cc:2393] T 8055232cf0004d40a85f3bf983fe00a9 P 197b880fa39d432a9b472845876bfaf0 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1c50de89a5a845668696dcb28c3badd6 in current term 6: Already voted for candidate 197b880fa39d432a9b472845876bfaf0 in this term.
W20260129 03:00:13.242249 21102 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.211.2:34517, user_credentials={real_user=slave}} blocked reactor thread for 248891us
I20260129 03:00:13.205866 23292 raft_consensus.cc:3060] T 75b0b6e82ddc4f92b1fe1d94f9a0b73b P 1c50de89a5a845668696dcb28c3badd6 [term 4 FOLLOWER]: Advancing to term 5
W20260129 03:00:13.257633 21134 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34046 (request call id 586) took 2041 ms (client timeout 1985 ms). Trace:
W20260129 03:00:13.258104 21134 rpcz_store.cc:269] 0129 03:00:11.222088 (+ 0us) service_pool.cc:168] Inserting onto call queue
0129 03:00:11.236163 (+ 14075us) service_pool.cc:225] Handling call
0129 03:00:13.257382 (+2021219us) spinlock_profiling.cc:243] Waited 495 ms on lock 0x7b5400670554. stack: 00007f8fe044e373 00007f8fe044e270 00007f8fdfda0574 00007f8fe9a5c6b4 00007f8fe9a5c649 00007f8fe34975a1 00007f8fe3531f7a 00007f8fe84e3647 00007f8fe1a73834 00007f8fe1a737c2 00007f8fe1a736f1 00007f8fe1a7366c 00007f8fe1a72922 00007f8fe186d34c 00007f8fe186c786 00007f8fe186c12f
0129 03:00:13.257561 (+ 179us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1138996096}
I20260129 03:00:13.259269 20433 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55672 (request call id 1762) took 1207 ms. Trace:
I20260129 03:00:13.259799 20433 rpcz_store.cc:276] 0129 03:00:12.052151 (+ 0us) service_pool.cc:168] Inserting onto call queue
0129 03:00:12.393736 (+341585us) service_pool.cc:225] Handling call
0129 03:00:12.394096 (+ 360us) raft_consensus.cc:1471] Updating replica for 1 ops
0129 03:00:12.697668 (+303572us) raft_consensus.cc:1534] Early marking committed up to index 0
0129 03:00:12.697716 (+ 48us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0129 03:00:12.698254 (+ 538us) log.cc:844] Serialized 26 byte log entry
0129 03:00:12.836449 (+138195us) raft_consensus.cc:1645] Marking committed up to 4
0129 03:00:12.836489 (+ 40us) raft_consensus.cc:1697] Filling consensus response to leader.
0129 03:00:12.836554 (+ 65us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0129 03:00:13.258993 (+422439us) raft_consensus.cc:1685] finished
0129 03:00:13.259024 (+ 31us) raft_consensus.cc:1691] UpdateReplicas() finished
0129 03:00:13.259194 (+ 170us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":303163,"raft.queue_time_us":304443,"raft.run_cpu_time_us":78,"raft.run_wall_time_us":79,"thread_start_us":137827,"threads_started":1,"wal-append.queue_time_us":433138}
I20260129 03:00:13.265205 20434 raft_consensus.cc:1240] T 76f713c3331540d2a5b76164103e9eb1 P 197b880fa39d432a9b472845876bfaf0 [term 6 FOLLOWER]: Rejecting Update request from peer 1c50de89a5a845668696dcb28c3badd6 for earlier term 5. Current term is 6. Ops: []
I20260129 03:00:13.171411 23026 raft_consensus.cc:515] T a0fcc58d6fb6473cafbd9d34d5195269 P 1c50de89a5a845668696dcb28c3badd6 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 03:00:13.278609 23026 leader_election.cc:290] T a0fcc58d6fb6473cafbd9d34d5195269 P 1c50de89a5a845668696dcb28c3badd6 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 197b880fa39d432a9b472845876bfaf0 (127.18.211.1:42027), 974b802141bd4f13beffef2aa03f88f9 (127.18.211.2:34517)
I20260129 03:00:13.232574 22748 raft_consensus.cc:493] T 2e58e9a5a8aa4cecafb464addb4f35ed P 1c50de89a5a845668696dcb28c3badd6 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 7a9daf5d42a84615a6383cfec9a10608)
W20260129 03:00:13.280023 23519 log.cc:927] Time spent T fcddab6bca684adf8ea032289a00f90e P 974b802141bd4f13beffef2aa03f88f9: Append to log took a long time: real 0.345s user 0.001s sys 0.000s
W20260129 03:00:13.283430 23529 log.cc:927] Time spent T 0f20af7fbe5d4db89a6f17948d98912c P 197b880fa39d432a9b472845876bfaf0: Append to log took a long time: real 0.208s user 0.000s sys 0.000s
I20260129 03:00:13.279565 22748 raft_consensus.cc:515] T 2e58e9a5a8aa4cecafb464addb4f35ed P 1c50de89a5a845668696dcb28c3badd6 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } }
I20260129 03:00:13.285826 22748 leader_election.cc:290] T 2e58e9a5a8aa4cecafb464addb4f35ed P 1c50de89a5a845668696dcb28c3badd6 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 197b880fa39d432a9b472845876bfaf0 (127.18.211.1:42027), 7a9daf5d42a84615a6383cfec9a10608 (127.18.211.4:40439)
I20260129 03:00:13.294354 23089 raft_consensus.cc:515] T 0340229d521b432580ae926f6062e3f7 P 197b880fa39d432a9b472845876bfaf0 [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 03:00:13.297147 23089 leader_election.cc:290] T 0340229d521b432580ae926f6062e3f7 P 197b880fa39d432a9b472845876bfaf0 [CANDIDATE]: Term 8 election: Requested vote from peers 974b802141bd4f13beffef2aa03f88f9 (127.18.211.2:34517), 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3:37735)
W20260129 03:00:13.297377 23433 log.cc:927] Time spent T 76f713c3331540d2a5b76164103e9eb1 P 974b802141bd4f13beffef2aa03f88f9: Append to log took a long time: real 2.150s user 0.003s sys 0.009s
I20260129 03:00:13.304518 20436 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55672 (request call id 1756) took 1426 ms. Trace:
I20260129 03:00:13.305071 20436 rpcz_store.cc:276] 0129 03:00:11.877889 (+ 0us) service_pool.cc:168] Inserting onto call queue
0129 03:00:12.037333 (+159444us) service_pool.cc:225] Handling call
0129 03:00:12.038091 (+ 758us) raft_consensus.cc:1471] Updating replica for 0 ops
0129 03:00:12.513802 (+475711us) raft_consensus.cc:1697] Filling consensus response to leader.
0129 03:00:13.304177 (+790375us) spinlock_profiling.cc:243] Waited 295 ms on lock 0x7b5400075dd4. stack: 00007f8fe044e373 00007f8fe044e270 00007f8fdfda0574 00007f8fe9a5c6b4 00007f8fe9a5c649 00007f8fe34975a1 00007f8fe352ea81 00007f8fe352d4be 00007f8fe84e1f49 00007f8fe1a6ed34 00007f8fe1a6ecc2 00007f8fe1a6ebf1 00007f8fe1a6eb6c 00007f8fe1a6de22 00007f8fe186d34c 00007f8fe186c786
0129 03:00:13.304438 (+ 261us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":679220608}
I20260129 03:00:13.309104 22996 raft_consensus.cc:3060] T 76f713c3331540d2a5b76164103e9eb1 P 1c50de89a5a845668696dcb28c3badd6 [term 5 FOLLOWER]: Advancing to term 6
I20260129 03:00:13.316490 23474 leader_election.cc:290] T 4408006eb5174de1896b037cf3f18457 P 1c50de89a5a845668696dcb28c3badd6 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 197b880fa39d432a9b472845876bfaf0 (127.18.211.1:42027), 974b802141bd4f13beffef2aa03f88f9 (127.18.211.2:34517)
I20260129 03:00:13.317585 23470 consensus_queue.cc:260] T c2633a6fc34c44579de348d208b0c01e P 1c50de89a5a845668696dcb28c3badd6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 4.3, Last appended by leader: 3, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
W20260129 03:00:13.318277 20553 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260129 03:00:13.323554 20436 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2e58e9a5a8aa4cecafb464addb4f35ed" candidate_uuid: "1c50de89a5a845668696dcb28c3badd6" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "197b880fa39d432a9b472845876bfaf0" is_pre_election: true
I20260129 03:00:13.324692 20436 raft_consensus.cc:2468] T 2e58e9a5a8aa4cecafb464addb4f35ed P 197b880fa39d432a9b472845876bfaf0 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1c50de89a5a845668696dcb28c3badd6 in term 4.
I20260129 03:00:13.326411 20436 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4408006eb5174de1896b037cf3f18457" candidate_uuid: "1c50de89a5a845668696dcb28c3badd6" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "197b880fa39d432a9b472845876bfaf0" is_pre_election: true
I20260129 03:00:13.326745 20435 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a0fcc58d6fb6473cafbd9d34d5195269" candidate_uuid: "1c50de89a5a845668696dcb28c3badd6" candidate_term: 6 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "197b880fa39d432a9b472845876bfaf0" is_pre_election: true
I20260129 03:00:13.328799 23427 raft_consensus.cc:3055] T 724a236524cb4e7ba3428880681d6cc2 P 974b802141bd4f13beffef2aa03f88f9 [term 5 LEADER]: Stepping down as leader of term 5
I20260129 03:00:13.329165 23427 raft_consensus.cc:740] T 724a236524cb4e7ba3428880681d6cc2 P 974b802141bd4f13beffef2aa03f88f9 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 974b802141bd4f13beffef2aa03f88f9, State: Running, Role: LEADER
I20260129 03:00:13.329912 23427 consensus_queue.cc:260] T 724a236524cb4e7ba3428880681d6cc2 P 974b802141bd4f13beffef2aa03f88f9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 5.4, Last appended by leader: 4, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 03:00:13.330900 23509 raft_consensus.cc:493] T febffe4a8b394368b69b1d164308fb9a P 974b802141bd4f13beffef2aa03f88f9 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260129 03:00:13.331178 23427 raft_consensus.cc:3060] T 724a236524cb4e7ba3428880681d6cc2 P 974b802141bd4f13beffef2aa03f88f9 [term 5 FOLLOWER]: Advancing to term 6
I20260129 03:00:13.331401 23509 raft_consensus.cc:515] T febffe4a8b394368b69b1d164308fb9a P 974b802141bd4f13beffef2aa03f88f9 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } }
I20260129 03:00:13.342213 20553 leader_election.cc:304] T 2e58e9a5a8aa4cecafb464addb4f35ed P 1c50de89a5a845668696dcb28c3badd6 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 197b880fa39d432a9b472845876bfaf0, 1c50de89a5a845668696dcb28c3badd6; no voters:
I20260129 03:00:13.338709 20589 raft_consensus.cc:2468] T 76f713c3331540d2a5b76164103e9eb1 P 1c50de89a5a845668696dcb28c3badd6 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 974b802141bd4f13beffef2aa03f88f9 in term 6.
W20260129 03:00:13.270727 23522 log.cc:927] Time spent T cb4727386a3e48aca5d71cc6c611c197 P 197b880fa39d432a9b472845876bfaf0: Append to log took a long time: real 0.384s user 0.001s sys 0.000s
I20260129 03:00:13.255210 23524 raft_consensus.cc:3055] T 3906b080a5b34a74b8b64128635e8997 P 974b802141bd4f13beffef2aa03f88f9 [term 4 LEADER]: Stepping down as leader of term 4
I20260129 03:00:13.248232 23437 consensus_queue.cc:1048] T a609ac02fae84725ab743f95f51d33ce P 197b880fa39d432a9b472845876bfaf0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 7, Time since last communication: 0.108s
I20260129 03:00:13.377501 20510 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0340229d521b432580ae926f6062e3f7" candidate_uuid: "197b880fa39d432a9b472845876bfaf0" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "974b802141bd4f13beffef2aa03f88f9"
I20260129 03:00:13.348719 23304 raft_consensus.cc:493] T 76f713c3331540d2a5b76164103e9eb1 P 1c50de89a5a845668696dcb28c3badd6 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260129 03:00:13.384280 21134 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "dc2f4cb752cf4d59b5f6bc689f5d8104" candidate_uuid: "1c50de89a5a845668696dcb28c3badd6" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "7a9daf5d42a84615a6383cfec9a10608"
W20260129 03:00:13.363968 23402 consensus_meta.cc:287] T 17d6431cc77a491e8464d68db4081a51 P 974b802141bd4f13beffef2aa03f88f9: Time spent flushing consensus metadata: real 0.587s user 0.010s sys 0.000s
I20260129 03:00:13.385370 21134 raft_consensus.cc:2393] T dc2f4cb752cf4d59b5f6bc689f5d8104 P 7a9daf5d42a84615a6383cfec9a10608 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 1c50de89a5a845668696dcb28c3badd6 in current term 6: Already voted for candidate 7a9daf5d42a84615a6383cfec9a10608 in this term.
I20260129 03:00:13.269485 23100 consensus_queue.cc:1048] T 45cc5f3d2688486d9b6dbe7e666dedde P 7a9daf5d42a84615a6383cfec9a10608 [LEADER]: Connected to new peer: Peer: permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.103s
I20260129 03:00:13.326083 23526 raft_consensus.cc:515] T 75f45538dc704d9a8ec4ff8ad9f3ea5b P 1c50de89a5a845668696dcb28c3badd6 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } }
I20260129 03:00:13.386885 21131 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "75b0b6e82ddc4f92b1fe1d94f9a0b73b" candidate_uuid: "1c50de89a5a845668696dcb28c3badd6" candidate_term: 5 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "7a9daf5d42a84615a6383cfec9a10608" is_pre_election: true
W20260129 03:00:13.386071 23365 consensus_meta.cc:287] T 0678b8b468bd43cdbc3e485900844821 P 7a9daf5d42a84615a6383cfec9a10608: Time spent flushing consensus metadata: real 0.695s user 0.007s sys 0.003s
I20260129 03:00:13.364760 23524 raft_consensus.cc:740] T 3906b080a5b34a74b8b64128635e8997 P 974b802141bd4f13beffef2aa03f88f9 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 974b802141bd4f13beffef2aa03f88f9, State: Running, Role: LEADER
I20260129 03:00:13.446388 20437 raft_consensus.cc:1240] T c2633a6fc34c44579de348d208b0c01e P 197b880fa39d432a9b472845876bfaf0 [term 5 FOLLOWER]: Rejecting Update request from peer 1c50de89a5a845668696dcb28c3badd6 for earlier term 4. Current term is 5. Ops: [4.3-4.3]
I20260129 03:00:13.370546 23325 raft_consensus.cc:2804] T 2e58e9a5a8aa4cecafb464addb4f35ed P 1c50de89a5a845668696dcb28c3badd6 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260129 03:00:13.455225 21130 raft_consensus.cc:1275] T 78aa39096868415d9afc2ffcb248e198 P 7a9daf5d42a84615a6383cfec9a10608 [term 7 FOLLOWER]: Refusing update from remote peer 1c50de89a5a845668696dcb28c3badd6: Log matching property violated. Preceding OpId in replica: term: 5 index: 6. Preceding OpId from leader: term: 7 index: 7. (index mismatch)
I20260129 03:00:13.455112 21129 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "cb0b3f259ebc4310927338a8740e87b8" candidate_uuid: "1c50de89a5a845668696dcb28c3badd6" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "7a9daf5d42a84615a6383cfec9a10608"
I20260129 03:00:13.447017 21127 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "add8d4923d1a489e86882c3a25692b95" candidate_uuid: "1c50de89a5a845668696dcb28c3badd6" candidate_term: 8 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "7a9daf5d42a84615a6383cfec9a10608" is_pre_election: true
I20260129 03:00:13.456427 21134 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3e2d15f8640b407195210c476e09df91" candidate_uuid: "1c50de89a5a845668696dcb28c3badd6" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "7a9daf5d42a84615a6383cfec9a10608" is_pre_election: true
I20260129 03:00:13.251147 23378 consensus_queue.cc:237] T 2ac659d732314e1288849f08fe5e074e P 197b880fa39d432a9b472845876bfaf0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 5.6, Last appended by leader: 6, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } }
I20260129 03:00:13.472234 21134 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "75b0b6e82ddc4f92b1fe1d94f9a0b73b" candidate_uuid: "1c50de89a5a845668696dcb28c3badd6" candidate_term: 5 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "7a9daf5d42a84615a6383cfec9a10608" is_pre_election: true
I20260129 03:00:13.464367 21131 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "dc2f4cb752cf4d59b5f6bc689f5d8104" candidate_uuid: "1c50de89a5a845668696dcb28c3badd6" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "7a9daf5d42a84615a6383cfec9a10608" is_pre_election: true
I20260129 03:00:13.461139 23325 raft_consensus.cc:493] T 2e58e9a5a8aa4cecafb464addb4f35ed P 1c50de89a5a845668696dcb28c3badd6 [term 4 FOLLOWER]: Starting leader election (detected failure of leader 7a9daf5d42a84615a6383cfec9a10608)
I20260129 03:00:13.445602 23492 raft_consensus.cc:493] T a609ac02fae84725ab743f95f51d33ce P 974b802141bd4f13beffef2aa03f88f9 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260129 03:00:13.475804 21131 raft_consensus.cc:2393] T dc2f4cb752cf4d59b5f6bc689f5d8104 P 7a9daf5d42a84615a6383cfec9a10608 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1c50de89a5a845668696dcb28c3badd6 in current term 6: Already voted for candidate 7a9daf5d42a84615a6383cfec9a10608 in this term.
I20260129 03:00:13.485953 21130 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "75f45538dc704d9a8ec4ff8ad9f3ea5b" candidate_uuid: "1c50de89a5a845668696dcb28c3badd6" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "7a9daf5d42a84615a6383cfec9a10608" is_pre_election: true
I20260129 03:00:13.372320 23091 consensus_queue.cc:1048] T 926a250fa5834e88b96ccecbefbf5086 P 197b880fa39d432a9b472845876bfaf0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 3, Time since last communication: 0.088s
W20260129 03:00:13.459017 23513 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.566s user 0.030s sys 0.004s
I20260129 03:00:13.395138 23304 raft_consensus.cc:515] T 76f713c3331540d2a5b76164103e9eb1 P 1c50de89a5a845668696dcb28c3badd6 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 03:00:13.526974 21134 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "febffe4a8b394368b69b1d164308fb9a" candidate_uuid: "197b880fa39d432a9b472845876bfaf0" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "7a9daf5d42a84615a6383cfec9a10608"
I20260129 03:00:13.528170 21134 raft_consensus.cc:2410] T febffe4a8b394368b69b1d164308fb9a P 7a9daf5d42a84615a6383cfec9a10608 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 197b880fa39d432a9b472845876bfaf0 for term 5 because replica has last-logged OpId of term: 4 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 5.
I20260129 03:00:13.522966 20437 raft_consensus.cc:1275] T a5a05fbcbbcf4916bbc01fd1a5b5e90d P 197b880fa39d432a9b472845876bfaf0 [term 8 FOLLOWER]: Refusing update from remote peer 1c50de89a5a845668696dcb28c3badd6: Log matching property violated. Preceding OpId in replica: term: 7 index: 5. Preceding OpId from leader: term: 8 index: 6. (index mismatch)
I20260129 03:00:13.444890 23402 raft_consensus.cc:515] T 17d6431cc77a491e8464d68db4081a51 P 974b802141bd4f13beffef2aa03f88f9 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 03:00:13.487090 23325 raft_consensus.cc:3060] T 2e58e9a5a8aa4cecafb464addb4f35ed P 1c50de89a5a845668696dcb28c3badd6 [term 4 FOLLOWER]: Advancing to term 5
I20260129 03:00:13.505903 21128 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "18f30ebaa4174fd28c958e72d0cc7e24" candidate_uuid: "1c50de89a5a845668696dcb28c3badd6" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "7a9daf5d42a84615a6383cfec9a10608"
I20260129 03:00:13.533191 21131 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "78aa39096868415d9afc2ffcb248e198" candidate_uuid: "1c50de89a5a845668696dcb28c3badd6" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "7a9daf5d42a84615a6383cfec9a10608" is_pre_election: true
W20260129 03:00:13.520123 23513 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.630s user 0.031s sys 0.004s
I20260129 03:00:13.537822 20586 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:48526 (request call id 398) took 1733 ms. Trace:
I20260129 03:00:13.540423 20586 rpcz_store.cc:276] 0129 03:00:11.804531 (+ 0us) service_pool.cc:168] Inserting onto call queue
0129 03:00:11.804680 (+ 149us) service_pool.cc:225] Handling call
0129 03:00:13.537615 (+1732935us) spinlock_profiling.cc:243] Waited 223 ms on lock 0x7b540039dad4. stack: 00007f8fe044e373 00007f8fe044e270 00007f8fdfda0574 00007f8fe9a5c6b4 00007f8fe9a5c649 00007f8fe34975a1 00007f8fe3531f7a 00007f8fe84e3647 00007f8fe1a73834 00007f8fe1a737c2 00007f8fe1a736f1 00007f8fe1a7366c 00007f8fe1a72922 00007f8fe186d34c 00007f8fe186c786 00007f8fe186c12f
0129 03:00:13.537761 (+ 146us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":512556928}
I20260129 03:00:13.541046 21134 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2e58e9a5a8aa4cecafb464addb4f35ed" candidate_uuid: "1c50de89a5a845668696dcb28c3badd6" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "7a9daf5d42a84615a6383cfec9a10608" is_pre_election: true
I20260129 03:00:13.541574 20586 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "96609ac68bdf4993a12e1c23355ca013" candidate_uuid: "974b802141bd4f13beffef2aa03f88f9" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "1c50de89a5a845668696dcb28c3badd6" is_pre_election: true
I20260129 03:00:13.515776 23091 consensus_queue.cc:1243] T 926a250fa5834e88b96ccecbefbf5086 P 197b880fa39d432a9b472845876bfaf0 [LEADER]: Peer 974b802141bd4f13beffef2aa03f88f9 log is divergent from this leader: its last log entry 6.4 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 3
W20260129 03:00:13.550974 20554 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260129 03:00:13.538422 21131 raft_consensus.cc:2376] T 78aa39096868415d9afc2ffcb248e198 P 7a9daf5d42a84615a6383cfec9a10608 [term 7 FOLLOWER]: Leader pre-election vote request: Already granted yes vote for candidate 1c50de89a5a845668696dcb28c3badd6 in term 7. Re-sending same reply.
W20260129 03:00:13.575376 20406 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260129 03:00:13.701521 20405 thread.cc:641] dns-resolver [worker] (thread pool) Time spent creating pthread: real 0.632s user 0.034s sys 0.029s
W20260129 03:00:13.702013 20405 thread.cc:608] dns-resolver [worker] (thread pool) Time spent starting thread: real 0.633s user 0.034s sys 0.029s
W20260129 03:00:13.702381 20405 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.211.4:40439, user_credentials={real_user=slave}} blocked reactor thread for 633184us
I20260129 03:00:13.542227 20586 raft_consensus.cc:2468] T 96609ac68bdf4993a12e1c23355ca013 P 1c50de89a5a845668696dcb28c3badd6 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 974b802141bd4f13beffef2aa03f88f9 in term 6.
I20260129 03:00:13.717614 20586 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "926a250fa5834e88b96ccecbefbf5086" candidate_uuid: "974b802141bd4f13beffef2aa03f88f9" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "1c50de89a5a845668696dcb28c3badd6" is_pre_election: true
I20260129 03:00:13.718667 20586 raft_consensus.cc:2393] T 926a250fa5834e88b96ccecbefbf5086 P 1c50de89a5a845668696dcb28c3badd6 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 974b802141bd4f13beffef2aa03f88f9 in current term 8: Already voted for candidate 197b880fa39d432a9b472845876bfaf0 in this term.
W20260129 03:00:13.721663 20553 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260129 03:00:13.459525 23365 raft_consensus.cc:515] T 0678b8b468bd43cdbc3e485900844821 P 7a9daf5d42a84615a6383cfec9a10608 [term 9 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } }
I20260129 03:00:13.788075 20586 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4f5a77e94fef4b16a2ba7b067c8b144a" candidate_uuid: "974b802141bd4f13beffef2aa03f88f9" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "1c50de89a5a845668696dcb28c3badd6" is_pre_election: true
W20260129 03:00:13.798866 20553 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260129 03:00:13.788810 21102 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.211.2:34517, user_credentials={real_user=slave}} blocked reactor thread for 58607.9us
I20260129 03:00:13.784682 23165 raft_consensus.cc:493] T 87d5fe48a44a408aaf03e894a67ecb2f P 7a9daf5d42a84615a6383cfec9a10608 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260129 03:00:13.791335 23215 consensus_meta.cc:287] T 0f20af7fbe5d4db89a6f17948d98912c P 974b802141bd4f13beffef2aa03f88f9: Time spent flushing consensus metadata: real 1.107s user 0.008s sys 0.004s
I20260129 03:00:13.717232 23485 raft_consensus.cc:493] T e33a417cf9c54cf5af73a4ee3aac4d3a P 197b880fa39d432a9b472845876bfaf0 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 974b802141bd4f13beffef2aa03f88f9)
I20260129 03:00:13.793479 21134 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "18f30ebaa4174fd28c958e72d0cc7e24" candidate_uuid: "1c50de89a5a845668696dcb28c3badd6" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "7a9daf5d42a84615a6383cfec9a10608" is_pre_election: true
I20260129 03:00:13.716315 23458 consensus_queue.cc:1048] T a5a05fbcbbcf4916bbc01fd1a5b5e90d P 1c50de89a5a845668696dcb28c3badd6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.093s
I20260129 03:00:13.798956 20586 raft_consensus.cc:2393] T 4f5a77e94fef4b16a2ba7b067c8b144a P 1c50de89a5a845668696dcb28c3badd6 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 974b802141bd4f13beffef2aa03f88f9 in current term 5: Already voted for candidate 7a9daf5d42a84615a6383cfec9a10608 in this term.
I20260129 03:00:13.802372 23509 leader_election.cc:290] T febffe4a8b394368b69b1d164308fb9a P 974b802141bd4f13beffef2aa03f88f9 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 197b880fa39d432a9b472845876bfaf0 (127.18.211.1:42027), 7a9daf5d42a84615a6383cfec9a10608 (127.18.211.4:40439)
W20260129 03:00:13.790484 20556 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.704s user 0.037s sys 0.034s
I20260129 03:00:13.807796 21131 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "42d27a28d89b409e8807ec0efd97af69" candidate_uuid: "1c50de89a5a845668696dcb28c3badd6" candidate_term: 9 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "7a9daf5d42a84615a6383cfec9a10608" is_pre_election: true
W20260129 03:00:13.719098 23403 consensus_meta.cc:287] T c934d31efd3f4e5195605ddc354d14e1 P 197b880fa39d432a9b472845876bfaf0: Time spent flushing consensus metadata: real 0.837s user 0.008s sys 0.001s
W20260129 03:00:13.727548 23087 consensus_meta.cc:287] T 724a236524cb4e7ba3428880681d6cc2 P 197b880fa39d432a9b472845876bfaf0: Time spent flushing consensus metadata: real 1.007s user 0.013s sys 0.000s
I20260129 03:00:13.387674 23475 consensus_queue.cc:237] T 17d6431cc77a491e8464d68db4081a51 P 197b880fa39d432a9b472845876bfaf0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 4.2, Last appended by leader: 2, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
W20260129 03:00:13.813112 20480 thread.cc:641] dns-resolver [worker] (thread pool) Time spent creating pthread: real 0.695s user 0.009s sys 0.069s
W20260129 03:00:13.828269 20480 thread.cc:608] dns-resolver [worker] (thread pool) Time spent starting thread: real 0.710s user 0.009s sys 0.070s
W20260129 03:00:13.828648 20480 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.211.4:40439, user_credentials={real_user=slave}} blocked reactor thread for 710732us
I20260129 03:00:13.815678 21134 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "dc2f4cb752cf4d59b5f6bc689f5d8104" candidate_uuid: "1c50de89a5a845668696dcb28c3badd6" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "7a9daf5d42a84615a6383cfec9a10608"
I20260129 03:00:13.509018 23492 raft_consensus.cc:515] T a609ac02fae84725ab743f95f51d33ce P 974b802141bd4f13beffef2aa03f88f9 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } }
W20260129 03:00:13.818569 20556 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.732s user 0.037s sys 0.034s
W20260129 03:00:13.831655 20556 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.211.2:34517, user_credentials={real_user=slave}} blocked reactor thread for 746068us
I20260129 03:00:13.832363 20556 leader_election.cc:304] T add8d4923d1a489e86882c3a25692b95 P 1c50de89a5a845668696dcb28c3badd6 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1c50de89a5a845668696dcb28c3badd6, 974b802141bd4f13beffef2aa03f88f9; no voters:
I20260129 03:00:13.816637 20586 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "cedce331e1584ac8b636c9b8a72b2ba0" candidate_uuid: "974b802141bd4f13beffef2aa03f88f9" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "1c50de89a5a845668696dcb28c3badd6" is_pre_election: true
W20260129 03:00:13.820214 20554 leader_election.cc:336] T 4f5a77e94fef4b16a2ba7b067c8b144a P 1c50de89a5a845668696dcb28c3badd6 [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer 7a9daf5d42a84615a6383cfec9a10608 (127.18.211.4:40439): Timed out: RequestConsensusVote RPC to 127.18.211.4:40439 timed out after 1.553s (SENT)
I20260129 03:00:13.785482 23089 raft_consensus.cc:493] T e7d83d4ccf684e76acb29a0dc844e1b8 P 197b880fa39d432a9b472845876bfaf0 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260129 03:00:13.532651 23524 consensus_queue.cc:260] T 3906b080a5b34a74b8b64128635e8997 P 974b802141bd4f13beffef2aa03f88f9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 03:00:13.833797 20554 leader_election.cc:304] T 4f5a77e94fef4b16a2ba7b067c8b144a P 1c50de89a5a845668696dcb28c3badd6 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1c50de89a5a845668696dcb28c3badd6; no voters: 7a9daf5d42a84615a6383cfec9a10608, 974b802141bd4f13beffef2aa03f88f9
I20260129 03:00:13.834897 20586 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "45cc5f3d2688486d9b6dbe7e666dedde" candidate_uuid: "974b802141bd4f13beffef2aa03f88f9" candidate_term: 7 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "1c50de89a5a845668696dcb28c3badd6" is_pre_election: true
W20260129 03:00:13.815270 23380 consensus_meta.cc:287] T cb0b3f259ebc4310927338a8740e87b8 P 7a9daf5d42a84615a6383cfec9a10608: Time spent flushing consensus metadata: real 1.169s user 0.012s sys 0.000s
I20260129 03:00:13.845342 21134 raft_consensus.cc:2393] T dc2f4cb752cf4d59b5f6bc689f5d8104 P 7a9daf5d42a84615a6383cfec9a10608 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 1c50de89a5a845668696dcb28c3badd6 in current term 6: Already voted for candidate 7a9daf5d42a84615a6383cfec9a10608 in this term.
I20260129 03:00:13.846741 20556 leader_election.cc:304] T e33a417cf9c54cf5af73a4ee3aac4d3a P 1c50de89a5a845668696dcb28c3badd6 [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1c50de89a5a845668696dcb28c3badd6; no voters: 197b880fa39d432a9b472845876bfaf0, 974b802141bd4f13beffef2aa03f88f9
I20260129 03:00:13.852109 20586 raft_consensus.cc:2468] T 45cc5f3d2688486d9b6dbe7e666dedde P 1c50de89a5a845668696dcb28c3badd6 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 974b802141bd4f13beffef2aa03f88f9 in term 6.
W20260129 03:00:13.853224 20480 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260129 03:00:13.854005 20480 connection.cc:232] RPC call timeout handler was delayed by -1.021242136000069s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.254s for a response
I20260129 03:00:13.855298 21102 leader_election.cc:304] T 505580c93c694bbb8ea96fdf9b17ab46 P 7a9daf5d42a84615a6383cfec9a10608 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 7a9daf5d42a84615a6383cfec9a10608, 974b802141bd4f13beffef2aa03f88f9; no voters: 1c50de89a5a845668696dcb28c3badd6
W20260129 03:00:13.827327 20590 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.712s user 0.009s sys 0.029s
W20260129 03:00:13.856573 20590 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.742s user 0.009s sys 0.029s
I20260129 03:00:13.855355 20586 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "368953c9102643b2a21f99af36b51905" candidate_uuid: "974b802141bd4f13beffef2aa03f88f9" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "1c50de89a5a845668696dcb28c3badd6" is_pre_election: true
W20260129 03:00:13.860888 20480 leader_election.cc:336] T 76f713c3331540d2a5b76164103e9eb1 P 974b802141bd4f13beffef2aa03f88f9 [CANDIDATE]: Term 6 election: RPC error from VoteRequest() call to peer 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3:37735): Timed out: RequestConsensusVote RPC to 127.18.211.3:37735 timed out after 1.539s (SENT)
I20260129 03:00:13.864277 21131 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "87d5fe48a44a408aaf03e894a67ecb2f" candidate_uuid: "197b880fa39d432a9b472845876bfaf0" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "7a9daf5d42a84615a6383cfec9a10608"
W20260129 03:00:13.865125 20554 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.211.4:40439, user_credentials={real_user=slave}} blocked reactor thread for 44948.7us
I20260129 03:00:13.865473 21136 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "cb0b3f259ebc4310927338a8740e87b8" candidate_uuid: "1c50de89a5a845668696dcb28c3badd6" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "7a9daf5d42a84615a6383cfec9a10608"
W20260129 03:00:13.867305 21136 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34046 (request call id 574) took 2742 ms (client timeout 1625 ms). Trace:
I20260129 03:00:13.864313 20556 leader_election.cc:304] T 913fc7d751ed4b7388d889aac30a5f9b P 1c50de89a5a845668696dcb28c3badd6 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1c50de89a5a845668696dcb28c3badd6; no voters: 197b880fa39d432a9b472845876bfaf0, 974b802141bd4f13beffef2aa03f88f9
I20260129 03:00:13.869406 20586 raft_consensus.cc:2393] T 368953c9102643b2a21f99af36b51905 P 1c50de89a5a845668696dcb28c3badd6 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 974b802141bd4f13beffef2aa03f88f9 in current term 8: Already voted for candidate 1c50de89a5a845668696dcb28c3badd6 in this term.
I20260129 03:00:13.870570 20586 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1299723a4476420ab02fd8bfd7aa0645" candidate_uuid: "974b802141bd4f13beffef2aa03f88f9" candidate_term: 11 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "1c50de89a5a845668696dcb28c3badd6" is_pre_election: true
I20260129 03:00:13.871392 20586 raft_consensus.cc:1813] T 1299723a4476420ab02fd8bfd7aa0645 P 1c50de89a5a845668696dcb28c3badd6 [term 10 FOLLOWER]: Handling vote request from an unknown peer 974b802141bd4f13beffef2aa03f88f9
I20260129 03:00:13.872435 20586 raft_consensus.cc:2410] T 1299723a4476420ab02fd8bfd7aa0645 P 1c50de89a5a845668696dcb28c3badd6 [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 974b802141bd4f13beffef2aa03f88f9 for term 11 because replica has last-logged OpId of term: 10 index: 9, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 4.
W20260129 03:00:13.842556 23390 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.610s user 0.023s sys 0.012s
I20260129 03:00:13.876148 21134 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "78aa39096868415d9afc2ffcb248e198" candidate_uuid: "1c50de89a5a845668696dcb28c3badd6" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "7a9daf5d42a84615a6383cfec9a10608"
I20260129 03:00:13.878736 20556 leader_election.cc:304] T 17d6431cc77a491e8464d68db4081a51 P 1c50de89a5a845668696dcb28c3badd6 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1c50de89a5a845668696dcb28c3badd6, 974b802141bd4f13beffef2aa03f88f9; no voters: 197b880fa39d432a9b472845876bfaf0
W20260129 03:00:13.867748 21136 rpcz_store.cc:269] 0129 03:00:11.124867 (+ 0us) service_pool.cc:168] Inserting onto call queue
0129 03:00:11.136888 (+ 12021us) service_pool.cc:225] Handling call
0129 03:00:13.865134 (+2728246us) spinlock_profiling.cc:243] Waited 7 ms on lock 0x7b5800111f98. stack: 00007f8fe044e373 00007f8fe044e270 00007f8fdfda0574 00007f8fe9a5c6b4 00007f8fe9a5c649 00007f8fe02c08b1 00007f8fe0442db1 00007f8fe0442c62 00007f8fe92eddd9 00007f8fe846d260 00007f8fe8535dbb 00007f8fe8535e41 00007f8fe84e3bd0 00007f8fe84e34b4 00007f8fe1a73834 00007f8fe1a737c2
0129 03:00:13.867217 (+ 2083us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":16099072}
I20260129 03:00:13.882120 21134 raft_consensus.cc:2376] T 78aa39096868415d9afc2ffcb248e198 P 7a9daf5d42a84615a6383cfec9a10608 [term 7 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 1c50de89a5a845668696dcb28c3badd6 in term 7. Re-sending same reply.
I20260129 03:00:13.888041 20586 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "add8d4923d1a489e86882c3a25692b95" candidate_uuid: "974b802141bd4f13beffef2aa03f88f9" candidate_term: 8 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "1c50de89a5a845668696dcb28c3badd6" is_pre_election: true
I20260129 03:00:13.889312 20480 leader_election.cc:304] T 926a250fa5834e88b96ccecbefbf5086 P 974b802141bd4f13beffef2aa03f88f9 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 974b802141bd4f13beffef2aa03f88f9; no voters: 197b880fa39d432a9b472845876bfaf0, 1c50de89a5a845668696dcb28c3badd6
I20260129 03:00:13.844525 21127 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "42d27a28d89b409e8807ec0efd97af69" candidate_uuid: "197b880fa39d432a9b472845876bfaf0" candidate_term: 9 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "7a9daf5d42a84615a6383cfec9a10608" is_pre_election: true
I20260129 03:00:13.853616 23428 raft_consensus.cc:493] T c934d31efd3f4e5195605ddc354d14e1 P 197b880fa39d432a9b472845876bfaf0 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260129 03:00:13.849300 23087 raft_consensus.cc:2749] T 724a236524cb4e7ba3428880681d6cc2 P 197b880fa39d432a9b472845876bfaf0 [term 6 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260129 03:00:13.813374 23215 raft_consensus.cc:515] T 0f20af7fbe5d4db89a6f17948d98912c P 974b802141bd4f13beffef2aa03f88f9 [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 03:00:13.886744 23474 raft_consensus.cc:2749] T 4f5a77e94fef4b16a2ba7b067c8b144a P 1c50de89a5a845668696dcb28c3badd6 [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260129 03:00:13.816236 23485 raft_consensus.cc:515] T e33a417cf9c54cf5af73a4ee3aac4d3a P 197b880fa39d432a9b472845876bfaf0 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
W20260129 03:00:13.889489 23390 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.654s user 0.024s sys 0.012s
I20260129 03:00:13.904258 20480 leader_election.cc:304] T 4f5a77e94fef4b16a2ba7b067c8b144a P 974b802141bd4f13beffef2aa03f88f9 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 974b802141bd4f13beffef2aa03f88f9; no voters: 1c50de89a5a845668696dcb28c3badd6, 7a9daf5d42a84615a6383cfec9a10608
I20260129 03:00:13.916049 20480 leader_election.cc:304] T cedce331e1584ac8b636c9b8a72b2ba0 P 974b802141bd4f13beffef2aa03f88f9 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 974b802141bd4f13beffef2aa03f88f9; no voters: 197b880fa39d432a9b472845876bfaf0, 1c50de89a5a845668696dcb28c3badd6
I20260129 03:00:13.818427 23165 raft_consensus.cc:515] T 87d5fe48a44a408aaf03e894a67ecb2f P 7a9daf5d42a84615a6383cfec9a10608 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } }
I20260129 03:00:13.864545 23419 raft_consensus.cc:493] T cb0b3f259ebc4310927338a8740e87b8 P 7a9daf5d42a84615a6383cfec9a10608 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260129 03:00:13.849685 23446 raft_consensus.cc:2804] T add8d4923d1a489e86882c3a25692b95 P 1c50de89a5a845668696dcb28c3badd6 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260129 03:00:13.876766 23290 raft_consensus.cc:2749] T e33a417cf9c54cf5af73a4ee3aac4d3a P 1c50de89a5a845668696dcb28c3badd6 [term 7 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
I20260129 03:00:13.909094 21134 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a609ac02fae84725ab743f95f51d33ce" candidate_uuid: "197b880fa39d432a9b472845876bfaf0" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "7a9daf5d42a84615a6383cfec9a10608"
I20260129 03:00:13.853142 21130 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1299723a4476420ab02fd8bfd7aa0645" candidate_uuid: "1c50de89a5a845668696dcb28c3badd6" candidate_term: 10 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "7a9daf5d42a84615a6383cfec9a10608" is_pre_election: true
I20260129 03:00:13.853463 23089 raft_consensus.cc:515] T e7d83d4ccf684e76acb29a0dc844e1b8 P 197b880fa39d432a9b472845876bfaf0 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 03:00:13.885207 23380 raft_consensus.cc:493] T fd8311b779af4e1fa1402b2a02a602f2 P 7a9daf5d42a84615a6383cfec9a10608 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 1c50de89a5a845668696dcb28c3badd6)
I20260129 03:00:13.920347 20480 leader_election.cc:304] T 45cc5f3d2688486d9b6dbe7e666dedde P 974b802141bd4f13beffef2aa03f88f9 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1c50de89a5a845668696dcb28c3badd6, 974b802141bd4f13beffef2aa03f88f9; no voters:
W20260129 03:00:13.921627 20587 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.648s user 0.003s sys 0.014s
W20260129 03:00:13.960397 20587 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.688s user 0.004s sys 0.015s
I20260129 03:00:13.908295 23344 raft_consensus.cc:493] T c2633a6fc34c44579de348d208b0c01e P 197b880fa39d432a9b472845876bfaf0 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260129 03:00:13.910506 23526 leader_election.cc:290] T 75f45538dc704d9a8ec4ff8ad9f3ea5b P 1c50de89a5a845668696dcb28c3badd6 [CANDIDATE]: Term 5 election: Requested vote from peers 974b802141bd4f13beffef2aa03f88f9 (127.18.211.2:34517), 7a9daf5d42a84615a6383cfec9a10608 (127.18.211.4:40439)
W20260129 03:00:13.963763 20511 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.596s user 0.011s sys 0.040s
W20260129 03:00:13.964263 20511 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.714s user 0.011s sys 0.040s
W20260129 03:00:13.921241 20584 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.810s user 0.013s sys 0.005s
W20260129 03:00:13.945497 23538 log.cc:927] Time spent T cedce331e1584ac8b636c9b8a72b2ba0 P 1c50de89a5a845668696dcb28c3badd6: Append to log took a long time: real 0.424s user 0.000s sys 0.001s
I20260129 03:00:13.946478 21131 raft_consensus.cc:3060] T 87d5fe48a44a408aaf03e894a67ecb2f P 7a9daf5d42a84615a6383cfec9a10608 [term 6 FOLLOWER]: Advancing to term 7
W20260129 03:00:13.944805 23542 log.cc:927] Time spent T 1299723a4476420ab02fd8bfd7aa0645 P 197b880fa39d432a9b472845876bfaf0: Append to log took a long time: real 0.069s user 0.000s sys 0.000s
I20260129 03:00:13.904179 22996 raft_consensus.cc:2749] T 913fc7d751ed4b7388d889aac30a5f9b P 1c50de89a5a845668696dcb28c3badd6 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260129 03:00:13.905587 23524 raft_consensus.cc:3060] T 3906b080a5b34a74b8b64128635e8997 P 974b802141bd4f13beffef2aa03f88f9 [term 4 FOLLOWER]: Advancing to term 5
I20260129 03:00:13.953964 20512 raft_consensus.cc:1240] T 0f20af7fbe5d4db89a6f17948d98912c P 974b802141bd4f13beffef2aa03f88f9 [term 8 FOLLOWER]: Rejecting Update request from peer 197b880fa39d432a9b472845876bfaf0 for earlier term 7. Current term is 8. Ops: []
I20260129 03:00:13.923641 21136 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "505580c93c694bbb8ea96fdf9b17ab46" candidate_uuid: "1c50de89a5a845668696dcb28c3badd6" candidate_term: 7 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "7a9daf5d42a84615a6383cfec9a10608"
I20260129 03:00:13.906183 23536 raft_consensus.cc:2749] T 926a250fa5834e88b96ccecbefbf5086 P 974b802141bd4f13beffef2aa03f88f9 [term 8 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
W20260129 03:00:13.961712 20479 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 1.068s user 0.009s sys 0.013s
W20260129 03:00:13.955950 23530 log.cc:927] Time spent T 4e3805ffa70242e8ade66da260dddf66 P 7a9daf5d42a84615a6383cfec9a10608: Append to log took a long time: real 0.469s user 0.000s sys 0.001s
I20260129 03:00:13.973572 20510 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c934d31efd3f4e5195605ddc354d14e1" candidate_uuid: "7a9daf5d42a84615a6383cfec9a10608" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "974b802141bd4f13beffef2aa03f88f9" is_pre_election: true
I20260129 03:00:13.958159 23262 raft_consensus.cc:2764] T 17d6431cc77a491e8464d68db4081a51 P 1c50de89a5a845668696dcb28c3badd6 [term 5 FOLLOWER]: Leader pre-election decision vote started in defunct term 4: won
W20260129 03:00:13.975792 20584 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.865s user 0.014s sys 0.005s
W20260129 03:00:13.981407 20479 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 1.087s user 0.010s sys 0.013s
I20260129 03:00:13.909588 23428 raft_consensus.cc:515] T c934d31efd3f4e5195605ddc354d14e1 P 197b880fa39d432a9b472845876bfaf0 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } }
W20260129 03:00:14.034205 20479 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.211.1:42027, user_credentials={real_user=slave}} blocked reactor thread for 1.15382e+06us
I20260129 03:00:13.982589 20510 raft_consensus.cc:2393] T c934d31efd3f4e5195605ddc354d14e1 P 974b802141bd4f13beffef2aa03f88f9 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 7a9daf5d42a84615a6383cfec9a10608 in current term 6: Already voted for candidate 974b802141bd4f13beffef2aa03f88f9 in this term.
I20260129 03:00:13.966203 23344 raft_consensus.cc:515] T c2633a6fc34c44579de348d208b0c01e P 197b880fa39d432a9b472845876bfaf0 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
W20260129 03:00:13.991313 20406 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260129 03:00:14.010939 23449 raft_consensus.cc:493] T 75be4bd6bd45474795fd472eff2953ac P 974b802141bd4f13beffef2aa03f88f9 [term 10 FOLLOWER]: Starting pre-election (detected failure of leader 197b880fa39d432a9b472845876bfaf0)
I20260129 03:00:13.968168 23380 raft_consensus.cc:515] T fd8311b779af4e1fa1402b2a02a602f2 P 7a9daf5d42a84615a6383cfec9a10608 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } }
I20260129 03:00:14.020501 21130 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2ac659d732314e1288849f08fe5e074e" candidate_uuid: "197b880fa39d432a9b472845876bfaf0" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "7a9daf5d42a84615a6383cfec9a10608"
I20260129 03:00:14.025022 23365 leader_election.cc:290] T 0678b8b468bd43cdbc3e485900844821 P 7a9daf5d42a84615a6383cfec9a10608 [CANDIDATE]: Term 9 election: Requested vote from peers 197b880fa39d432a9b472845876bfaf0 (127.18.211.1:42027), 974b802141bd4f13beffef2aa03f88f9 (127.18.211.2:34517)
I20260129 03:00:13.966572 23509 raft_consensus.cc:2804] T 45cc5f3d2688486d9b6dbe7e666dedde P 974b802141bd4f13beffef2aa03f88f9 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260129 03:00:13.956404 23419 raft_consensus.cc:515] T cb0b3f259ebc4310927338a8740e87b8 P 7a9daf5d42a84615a6383cfec9a10608 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } }
I20260129 03:00:14.028203 21136 raft_consensus.cc:2468] T 505580c93c694bbb8ea96fdf9b17ab46 P 7a9daf5d42a84615a6383cfec9a10608 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1c50de89a5a845668696dcb28c3badd6 in term 7.
W20260129 03:00:14.031147 23450 consensus_meta.cc:287] T c891d2bff4ce4568861bbc992e91fbb0 P 974b802141bd4f13beffef2aa03f88f9: Time spent flushing consensus metadata: real 1.270s user 0.008s sys 0.005s
I20260129 03:00:13.919407 23458 consensus_queue.cc:1048] T 505580c93c694bbb8ea96fdf9b17ab46 P 1c50de89a5a845668696dcb28c3badd6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.056s
I20260129 03:00:14.025863 20516 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57600 (request call id 1970) took 1052 ms. Trace:
I20260129 03:00:14.051738 20516 rpcz_store.cc:276] 0129 03:00:12.973289 (+ 0us) service_pool.cc:168] Inserting onto call queue
0129 03:00:12.973511 (+ 222us) service_pool.cc:225] Handling call
0129 03:00:13.011407 (+ 37896us) raft_consensus.cc:1471] Updating replica for 1 ops
0129 03:00:13.021109 (+ 9702us) raft_consensus.cc:1534] Early marking committed up to index 0
0129 03:00:13.021154 (+ 45us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0129 03:00:13.021712 (+ 558us) log.cc:844] Serialized 26 byte log entry
0129 03:00:13.525121 (+503409us) raft_consensus.cc:1645] Marking committed up to 7
0129 03:00:13.525174 (+ 53us) raft_consensus.cc:1697] Filling consensus response to leader.
0129 03:00:13.525266 (+ 92us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0129 03:00:14.025649 (+500383us) raft_consensus.cc:1685] finished
0129 03:00:14.025671 (+ 22us) raft_consensus.cc:1691] UpdateReplicas() finished
0129 03:00:14.025803 (+ 132us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":4539,"raft.queue_time_us":109997,"raft.run_cpu_time_us":175,"raft.run_wall_time_us":72034,"thread_start_us":412448,"threads_started":1,"wal-append.queue_time_us":889338}
I20260129 03:00:13.947091 23446 raft_consensus.cc:493] T add8d4923d1a489e86882c3a25692b95 P 1c50de89a5a845668696dcb28c3badd6 [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260129 03:00:14.039896 21134 raft_consensus.cc:2468] T a609ac02fae84725ab743f95f51d33ce P 7a9daf5d42a84615a6383cfec9a10608 [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 197b880fa39d432a9b472845876bfaf0 in term 9.
I20260129 03:00:14.025799 23165 leader_election.cc:290] T 87d5fe48a44a408aaf03e894a67ecb2f P 7a9daf5d42a84615a6383cfec9a10608 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 197b880fa39d432a9b472845876bfaf0 (127.18.211.1:42027), 974b802141bd4f13beffef2aa03f88f9 (127.18.211.2:34517)
I20260129 03:00:14.043198 23089 leader_election.cc:290] T e7d83d4ccf684e76acb29a0dc844e1b8 P 197b880fa39d432a9b472845876bfaf0 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 974b802141bd4f13beffef2aa03f88f9 (127.18.211.2:34517), 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3:37735)
I20260129 03:00:14.045081 20441 raft_consensus.cc:2468] T c934d31efd3f4e5195605ddc354d14e1 P 197b880fa39d432a9b472845876bfaf0 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7a9daf5d42a84615a6383cfec9a10608 in term 6.
I20260129 03:00:14.040058 21127 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1299723a4476420ab02fd8bfd7aa0645" candidate_uuid: "197b880fa39d432a9b472845876bfaf0" candidate_term: 11 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "7a9daf5d42a84615a6383cfec9a10608" is_pre_election: true
I20260129 03:00:14.045102 23509 raft_consensus.cc:493] T 45cc5f3d2688486d9b6dbe7e666dedde P 974b802141bd4f13beffef2aa03f88f9 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260129 03:00:14.055476 21134 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4e3805ffa70242e8ade66da260dddf66" candidate_uuid: "197b880fa39d432a9b472845876bfaf0" candidate_term: 7 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "7a9daf5d42a84615a6383cfec9a10608" is_pre_election: true
W20260129 03:00:14.050572 20583 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.805s user 0.004s sys 0.012s
W20260129 03:00:14.068356 20583 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.832s user 0.004s sys 0.013s
W20260129 03:00:14.060170 20433 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.616s user 0.026s sys 0.090s
I20260129 03:00:14.053853 21136 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "45cc5f3d2688486d9b6dbe7e666dedde" candidate_uuid: "1c50de89a5a845668696dcb28c3badd6" candidate_term: 6 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "7a9daf5d42a84615a6383cfec9a10608" is_pre_election: true
W20260129 03:00:14.068598 20507 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.611s user 0.009s sys 0.009s
I20260129 03:00:14.066241 21131 raft_consensus.cc:2468] T 87d5fe48a44a408aaf03e894a67ecb2f P 7a9daf5d42a84615a6383cfec9a10608 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 197b880fa39d432a9b472845876bfaf0 in term 7.
W20260129 03:00:14.070247 20507 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.613s user 0.009s sys 0.009s
I20260129 03:00:14.071925 20507 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:48544 (request call id 328) took 1065 ms. Trace:
I20260129 03:00:14.074641 20507 rpcz_store.cc:276] 0129 03:00:13.006784 (+ 0us) service_pool.cc:168] Inserting onto call queue
0129 03:00:13.006949 (+ 165us) service_pool.cc:225] Handling call
0129 03:00:13.016022 (+ 9073us) raft_consensus.cc:1471] Updating replica for 0 ops
0129 03:00:13.016331 (+ 309us) raft_consensus.cc:1534] Early marking committed up to index 7
0129 03:00:13.457214 (+440883us) log.cc:844] Serialized 14 byte log entry
0129 03:00:14.071476 (+614262us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0129 03:00:14.071548 (+ 72us) raft_consensus.cc:1645] Marking committed up to 7
0129 03:00:14.071613 (+ 65us) raft_consensus.cc:1697] Filling consensus response to leader.
0129 03:00:14.071703 (+ 90us) raft_consensus.cc:1691] UpdateReplicas() finished
0129 03:00:14.071842 (+ 139us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":868432,"prepare.run_cpu_time_us":37,"prepare.run_wall_time_us":38,"thread_start_us":1013604,"threads_started":2}
I20260129 03:00:14.051545 23450 raft_consensus.cc:2749] T c891d2bff4ce4568861bbc992e91fbb0 P 974b802141bd4f13beffef2aa03f88f9 [term 9 FOLLOWER]: Leader election lost for term 8. Reason: Vote denied by peer 197b880fa39d432a9b472845876bfaf0 (127.18.211.1:42027) with higher term. Message: Invalid argument: T c891d2bff4ce4568861bbc992e91fbb0 P 197b880fa39d432a9b472845876bfaf0 [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate 974b802141bd4f13beffef2aa03f88f9 for earlier term 8. Current term is 9.
I20260129 03:00:14.086160 21136 raft_consensus.cc:1275] T dc2f4cb752cf4d59b5f6bc689f5d8104 P 7a9daf5d42a84615a6383cfec9a10608 [term 6 FOLLOWER]: Refusing update from remote peer 1c50de89a5a845668696dcb28c3badd6: Log matching property violated. Preceding OpId in replica: term: 4 index: 6. Preceding OpId from leader: term: 6 index: 7. (index mismatch)
W20260129 03:00:14.070248 20433 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.626s user 0.026s sys 0.091s
I20260129 03:00:14.056701 23446 raft_consensus.cc:3060] T add8d4923d1a489e86882c3a25692b95 P 1c50de89a5a845668696dcb28c3badd6 [term 7 FOLLOWER]: Advancing to term 8
I20260129 03:00:14.043521 23449 raft_consensus.cc:515] T 75be4bd6bd45474795fd472eff2953ac P 974b802141bd4f13beffef2aa03f88f9 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 03:00:14.093494 20515 raft_consensus.cc:2468] T c891d2bff4ce4568861bbc992e91fbb0 P 974b802141bd4f13beffef2aa03f88f9 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1c50de89a5a845668696dcb28c3badd6 in term 9.
I20260129 03:00:14.094069 21134 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0678b8b468bd43cdbc3e485900844821" candidate_uuid: "197b880fa39d432a9b472845876bfaf0" candidate_term: 9 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "7a9daf5d42a84615a6383cfec9a10608" is_pre_election: true
I20260129 03:00:14.067288 23165 raft_consensus.cc:2804] T 505580c93c694bbb8ea96fdf9b17ab46 P 7a9daf5d42a84615a6383cfec9a10608 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260129 03:00:14.096912 21134 raft_consensus.cc:2393] T 0678b8b468bd43cdbc3e485900844821 P 7a9daf5d42a84615a6383cfec9a10608 [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 197b880fa39d432a9b472845876bfaf0 in current term 9: Already voted for candidate 7a9daf5d42a84615a6383cfec9a10608 in this term.
I20260129 03:00:14.070305 23509 raft_consensus.cc:3060] T 45cc5f3d2688486d9b6dbe7e666dedde P 974b802141bd4f13beffef2aa03f88f9 [term 6 FOLLOWER]: Advancing to term 7
W20260129 03:00:14.100838 20585 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.737s user 0.013s sys 0.006s
W20260129 03:00:14.101796 20585 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.738s user 0.014s sys 0.007s
I20260129 03:00:14.109442 23449 leader_election.cc:290] T 75be4bd6bd45474795fd472eff2953ac P 974b802141bd4f13beffef2aa03f88f9 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 197b880fa39d432a9b472845876bfaf0 (127.18.211.1:42027), 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3:37735)
I20260129 03:00:14.110070 23449 consensus_queue.cc:1059] T a5a05fbcbbcf4916bbc01fd1a5b5e90d P 974b802141bd4f13beffef2aa03f88f9 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 }, Status: INVALID_TERM, Last received: 7.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260129 03:00:14.096016 23215 leader_election.cc:290] T 0f20af7fbe5d4db89a6f17948d98912c P 974b802141bd4f13beffef2aa03f88f9 [CANDIDATE]: Term 8 election: Requested vote from peers 197b880fa39d432a9b472845876bfaf0 (127.18.211.1:42027), 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3:37735)
I20260129 03:00:14.098383 23165 raft_consensus.cc:493] T 505580c93c694bbb8ea96fdf9b17ab46 P 7a9daf5d42a84615a6383cfec9a10608 [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260129 03:00:14.112323 23165 raft_consensus.cc:3060] T 505580c93c694bbb8ea96fdf9b17ab46 P 7a9daf5d42a84615a6383cfec9a10608 [term 7 FOLLOWER]: Advancing to term 8
W20260129 03:00:14.115993 20553 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260129 03:00:14.117081 23380 leader_election.cc:290] T fd8311b779af4e1fa1402b2a02a602f2 P 7a9daf5d42a84615a6383cfec9a10608 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 197b880fa39d432a9b472845876bfaf0 (127.18.211.1:42027), 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3:37735)
I20260129 03:00:14.119912 20438 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55672 (request call id 1760) took 2135 ms. Trace:
I20260129 03:00:14.120328 20438 rpcz_store.cc:276] 0129 03:00:11.984369 (+ 0us) service_pool.cc:168] Inserting onto call queue
0129 03:00:12.081108 (+ 96739us) service_pool.cc:225] Handling call
0129 03:00:12.081453 (+ 345us) raft_consensus.cc:1471] Updating replica for 1 ops
0129 03:00:12.646637 (+565184us) raft_consensus.cc:1697] Filling consensus response to leader.
0129 03:00:14.119606 (+1472969us) spinlock_profiling.cc:243] Waited 561 ms on lock 0x7b54002aec54. stack: 00007f8fe044e373 00007f8fe044e270 00007f8fdfda0574 00007f8fe9a5c6b4 00007f8fe9a5c649 00007f8fe34975a1 00007f8fe352ea81 00007f8fe352d4be 00007f8fe84e1f49 00007f8fe1a6ed34 00007f8fe1a6ecc2 00007f8fe1a6ebf1 00007f8fe1a6eb6c 00007f8fe1a6de22 00007f8fe186d34c 00007f8fe186c786
0129 03:00:14.119826 (+ 220us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1289934592}
W20260129 03:00:14.132095 23490 consensus_meta.cc:287] T 36ff33be666a4c82954731691dbcf327 P 974b802141bd4f13beffef2aa03f88f9: Time spent flushing consensus metadata: real 1.242s user 0.007s sys 0.005s
I20260129 03:00:14.134531 23490 raft_consensus.cc:3055] T a5a05fbcbbcf4916bbc01fd1a5b5e90d P 974b802141bd4f13beffef2aa03f88f9 [term 7 LEADER]: Stepping down as leader of term 7
I20260129 03:00:14.134910 23490 raft_consensus.cc:740] T a5a05fbcbbcf4916bbc01fd1a5b5e90d P 974b802141bd4f13beffef2aa03f88f9 [term 7 LEADER]: Becoming Follower/Learner. State: Replica: 974b802141bd4f13beffef2aa03f88f9, State: Running, Role: LEADER
I20260129 03:00:14.136276 20509 raft_consensus.cc:1275] T 36ff33be666a4c82954731691dbcf327 P 974b802141bd4f13beffef2aa03f88f9 [term 4 FOLLOWER]: Refusing update from remote peer 1c50de89a5a845668696dcb28c3badd6: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20260129 03:00:14.137463 20509 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57584 (request call id 1498) took 1917 ms. Trace:
I20260129 03:00:14.137813 20509 rpcz_store.cc:276] 0129 03:00:12.220462 (+ 0us) service_pool.cc:168] Inserting onto call queue
0129 03:00:12.220618 (+ 156us) service_pool.cc:225] Handling call
0129 03:00:12.221006 (+ 388us) raft_consensus.cc:1471] Updating replica for 0 ops
0129 03:00:14.136674 (+1915668us) raft_consensus.cc:1697] Filling consensus response to leader.
0129 03:00:14.137244 (+ 570us) spinlock_profiling.cc:243] Waited 1.91 s on lock 0x7b5400168354. stack: 00007f8fe044e373 00007f8fe044e270 00007f8fdfda0574 00007f8fe9a5c6b4 00007f8fe9a5c649 00007f8fe34975a1 00007f8fe352ea81 00007f8fe352d4be 00007f8fe84e1f49 00007f8fe1a6ed34 00007f8fe1a6ecc2 00007f8fe1a6ebf1 00007f8fe1a6eb6c 00007f8fe1a6de22 00007f8fe186d34c 00007f8fe186c786
0129 03:00:14.137404 (+ 160us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":4403898752}
I20260129 03:00:14.141026 23429 raft_consensus.cc:3055] T e33a417cf9c54cf5af73a4ee3aac4d3a P 974b802141bd4f13beffef2aa03f88f9 [term 6 LEADER]: Stepping down as leader of term 6
I20260129 03:00:14.141459 23429 raft_consensus.cc:740] T e33a417cf9c54cf5af73a4ee3aac4d3a P 974b802141bd4f13beffef2aa03f88f9 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 974b802141bd4f13beffef2aa03f88f9, State: Running, Role: LEADER
I20260129 03:00:14.142318 23429 consensus_queue.cc:260] T e33a417cf9c54cf5af73a4ee3aac4d3a P 974b802141bd4f13beffef2aa03f88f9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 6.3, Last appended by leader: 3, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 03:00:14.082609 23325 raft_consensus.cc:515] T 2e58e9a5a8aa4cecafb464addb4f35ed P 1c50de89a5a845668696dcb28c3badd6 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } }
I20260129 03:00:14.147387 23325 leader_election.cc:290] T 2e58e9a5a8aa4cecafb464addb4f35ed P 1c50de89a5a845668696dcb28c3badd6 [CANDIDATE]: Term 5 election: Requested vote from peers 197b880fa39d432a9b472845876bfaf0 (127.18.211.1:42027), 7a9daf5d42a84615a6383cfec9a10608 (127.18.211.4:40439)
I20260129 03:00:14.148373 23485 leader_election.cc:290] T e33a417cf9c54cf5af73a4ee3aac4d3a P 197b880fa39d432a9b472845876bfaf0 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 974b802141bd4f13beffef2aa03f88f9 (127.18.211.2:34517), 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3:37735)
I20260129 03:00:14.150827 20438 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2e58e9a5a8aa4cecafb464addb4f35ed" candidate_uuid: "1c50de89a5a845668696dcb28c3badd6" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "197b880fa39d432a9b472845876bfaf0"
I20260129 03:00:14.151590 20438 raft_consensus.cc:3060] T 2e58e9a5a8aa4cecafb464addb4f35ed P 197b880fa39d432a9b472845876bfaf0 [term 4 FOLLOWER]: Advancing to term 5
W20260129 03:00:14.122721 23292 consensus_meta.cc:287] T 75b0b6e82ddc4f92b1fe1d94f9a0b73b P 1c50de89a5a845668696dcb28c3badd6: Time spent flushing consensus metadata: real 0.831s user 0.008s sys 0.001s
I20260129 03:00:14.082885 23536 consensus_queue.cc:1059] T a1a39dcd328e4268ae41b988c66b4a75 P 974b802141bd4f13beffef2aa03f88f9 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 }, Status: INVALID_TERM, Last received: 5.4, Next index: 5, Last known committed idx: 2, Time since last communication: 0.062s
I20260129 03:00:14.143651 23429 raft_consensus.cc:3060] T e33a417cf9c54cf5af73a4ee3aac4d3a P 974b802141bd4f13beffef2aa03f88f9 [term 6 FOLLOWER]: Advancing to term 7
I20260129 03:00:14.158108 23344 leader_election.cc:290] T c2633a6fc34c44579de348d208b0c01e P 197b880fa39d432a9b472845876bfaf0 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 974b802141bd4f13beffef2aa03f88f9 (127.18.211.2:34517), 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3:37735)
I20260129 03:00:14.099959 23492 leader_election.cc:290] T a609ac02fae84725ab743f95f51d33ce P 974b802141bd4f13beffef2aa03f88f9 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 197b880fa39d432a9b472845876bfaf0 (127.18.211.1:42027), 7a9daf5d42a84615a6383cfec9a10608 (127.18.211.4:40439)
I20260129 03:00:14.169078 20438 raft_consensus.cc:2468] T 2e58e9a5a8aa4cecafb464addb4f35ed P 197b880fa39d432a9b472845876bfaf0 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1c50de89a5a845668696dcb28c3badd6 in term 5.
W20260129 03:00:14.203475 20481 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 1.122s user 0.022s sys 0.053s
W20260129 03:00:14.203922 20481 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 1.123s user 0.022s sys 0.053s
I20260129 03:00:14.159763 23292 raft_consensus.cc:515] T 75b0b6e82ddc4f92b1fe1d94f9a0b73b P 1c50de89a5a845668696dcb28c3badd6 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } }
W20260129 03:00:14.223416 23556 log.cc:927] Time spent T 1ce4ddf5b10e4ae8b193b13b2e5936ee P 1c50de89a5a845668696dcb28c3badd6: Append to log took a long time: real 0.065s user 0.001s sys 0.000s
I20260129 03:00:14.135756 23490 consensus_queue.cc:260] T a5a05fbcbbcf4916bbc01fd1a5b5e90d P 974b802141bd4f13beffef2aa03f88f9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 7.5, Last appended by leader: 5, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 03:00:14.208390 23536 raft_consensus.cc:493] T 4408006eb5174de1896b037cf3f18457 P 974b802141bd4f13beffef2aa03f88f9 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 197b880fa39d432a9b472845876bfaf0)
I20260129 03:00:14.240569 20588 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35752 (request call id 1686) took 5323 ms. Trace:
I20260129 03:00:14.241142 20588 rpcz_store.cc:276] 0129 03:00:08.917581 (+ 0us) service_pool.cc:168] Inserting onto call queue
0129 03:00:10.142695 (+1225114us) service_pool.cc:225] Handling call
0129 03:00:10.188109 (+ 45414us) raft_consensus.cc:1471] Updating replica for 0 ops
0129 03:00:11.420658 (+1232549us) raft_consensus.cc:1697] Filling consensus response to leader.
0129 03:00:14.240240 (+2819582us) spinlock_profiling.cc:243] Waited 1.22 s on lock 0x7b54003b1354. stack: 00007f8fe044e373 00007f8fe044e270 00007f8fdfda0574 00007f8fe9a5c6b4 00007f8fe9a5c649 00007f8fe34975a1 00007f8fe352ea81 00007f8fe352d4be 00007f8fe84e1f49 00007f8fe1a6ed34 00007f8fe1a6ecc2 00007f8fe1a6ebf1 00007f8fe1a6eb6c 00007f8fe1a6de22 00007f8fe186d34c 00007f8fe186c786
0129 03:00:14.240493 (+ 253us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2798209408}
W20260129 03:00:14.245990 20514 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 1.009s user 0.029s sys 0.009s
W20260129 03:00:14.246376 20514 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 1.009s user 0.029s sys 0.009s
I20260129 03:00:14.206890 23553 raft_consensus.cc:493] T b189f08e16a74a11980cc55d99359a0a P 974b802141bd4f13beffef2aa03f88f9 [term 9 FOLLOWER]: Starting pre-election (detected failure of leader 197b880fa39d432a9b472845876bfaf0)
I20260129 03:00:14.225250 23485 raft_consensus.cc:493] T fcddab6bca684adf8ea032289a00f90e P 197b880fa39d432a9b472845876bfaf0 [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260129 03:00:14.246522 20556 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.211.2:34517, user_credentials={real_user=slave}} blocked reactor thread for 367858us
I20260129 03:00:14.253916 20556 leader_election.cc:304] T 3906b080a5b34a74b8b64128635e8997 P 1c50de89a5a845668696dcb28c3badd6 [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1c50de89a5a845668696dcb28c3badd6; no voters: 197b880fa39d432a9b472845876bfaf0, 974b802141bd4f13beffef2aa03f88f9
I20260129 03:00:14.263190 20433 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55676 (request call id 1495) took 1145 ms. Trace:
I20260129 03:00:14.263620 20433 rpcz_store.cc:276] 0129 03:00:13.117638 (+ 0us) service_pool.cc:168] Inserting onto call queue
0129 03:00:13.261760 (+144122us) service_pool.cc:225] Handling call
0129 03:00:13.262205 (+ 445us) raft_consensus.cc:1471] Updating replica for 1 ops
0129 03:00:13.267433 (+ 5228us) raft_consensus.cc:1534] Early marking committed up to index 0
0129 03:00:13.267516 (+ 83us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0129 03:00:13.352621 (+ 85105us) log.cc:844] Serialized 26 byte log entry
0129 03:00:14.088938 (+736317us) raft_consensus.cc:1645] Marking committed up to 6
0129 03:00:14.088994 (+ 56us) raft_consensus.cc:1697] Filling consensus response to leader.
0129 03:00:14.089083 (+ 89us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0129 03:00:14.262976 (+173893us) raft_consensus.cc:1685] finished
0129 03:00:14.262995 (+ 19us) raft_consensus.cc:1691] UpdateReplicas() finished
0129 03:00:14.263134 (+ 139us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":6629,"raft.run_cpu_time_us":152,"raft.run_wall_time_us":40355,"thread_start_us":644255,"threads_started":1,"wal-append.queue_time_us":878478}
I20260129 03:00:14.264753 23428 leader_election.cc:290] T c934d31efd3f4e5195605ddc354d14e1 P 197b880fa39d432a9b472845876bfaf0 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 974b802141bd4f13beffef2aa03f88f9 (127.18.211.2:34517), 7a9daf5d42a84615a6383cfec9a10608 (127.18.211.4:40439)
I20260129 03:00:14.253548 23427 raft_consensus.cc:2749] T 4f5a77e94fef4b16a2ba7b067c8b144a P 974b802141bd4f13beffef2aa03f88f9 [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
W20260129 03:00:14.271562 20554 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.211.4:40439, user_credentials={real_user=slave}} blocked reactor thread for 390204us
W20260129 03:00:14.270552 23466 consensus_meta.cc:287] T 4b1876f9a70b49edb3aa74105e66d3b3 P 197b880fa39d432a9b472845876bfaf0: Time spent flushing consensus metadata: real 1.389s user 0.013s sys 0.000s
W20260129 03:00:14.285797 20482 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260129 03:00:14.284790 20479 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.211.1:42027, user_credentials={real_user=slave}} blocked reactor thread for 249125us
W20260129 03:00:14.275279 23527 log.cc:927] Time spent T 78aa39096868415d9afc2ffcb248e198 P 197b880fa39d432a9b472845876bfaf0: Append to log took a long time: real 1.028s user 0.004s sys 0.010s
I20260129 03:00:14.252691 23536 raft_consensus.cc:515] T 4408006eb5174de1896b037cf3f18457 P 974b802141bd4f13beffef2aa03f88f9 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 03:00:14.291194 23536 leader_election.cc:290] T 4408006eb5174de1896b037cf3f18457 P 974b802141bd4f13beffef2aa03f88f9 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 197b880fa39d432a9b472845876bfaf0 (127.18.211.1:42027), 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3:37735)
I20260129 03:00:14.254675 23485 raft_consensus.cc:515] T fcddab6bca684adf8ea032289a00f90e P 197b880fa39d432a9b472845876bfaf0 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
W20260129 03:00:14.296217 20480 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260129 03:00:14.276160 23428 raft_consensus.cc:493] T 96609ac68bdf4993a12e1c23355ca013 P 197b880fa39d432a9b472845876bfaf0 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 7a9daf5d42a84615a6383cfec9a10608)
I20260129 03:00:14.298501 21134 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "75f45538dc704d9a8ec4ff8ad9f3ea5b" candidate_uuid: "974b802141bd4f13beffef2aa03f88f9" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "7a9daf5d42a84615a6383cfec9a10608" is_pre_election: true
W20260129 03:00:14.286015 23342 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 4.000s user 0.026s sys 0.042s
I20260129 03:00:14.284309 23439 raft_consensus.cc:493] T febffe4a8b394368b69b1d164308fb9a P 7a9daf5d42a84615a6383cfec9a10608 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260129 03:00:14.302706 23492 consensus_queue.cc:1059] T 4f8f3a38f6e24ba4857875181f447442 P 974b802141bd4f13beffef2aa03f88f9 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 }, Status: INVALID_TERM, Last received: 6.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260129 03:00:14.303854 23215 raft_consensus.cc:3055] T 4f8f3a38f6e24ba4857875181f447442 P 974b802141bd4f13beffef2aa03f88f9 [term 6 LEADER]: Stepping down as leader of term 6
I20260129 03:00:14.304253 23215 raft_consensus.cc:740] T 4f8f3a38f6e24ba4857875181f447442 P 974b802141bd4f13beffef2aa03f88f9 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 974b802141bd4f13beffef2aa03f88f9, State: Running, Role: LEADER
I20260129 03:00:14.305058 23215 consensus_queue.cc:260] T 4f8f3a38f6e24ba4857875181f447442 P 974b802141bd4f13beffef2aa03f88f9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 6.4, Last appended by leader: 4, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 03:00:14.305768 21129 raft_consensus.cc:2468] T cb0b3f259ebc4310927338a8740e87b8 P 7a9daf5d42a84615a6383cfec9a10608 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1c50de89a5a845668696dcb28c3badd6 in term 8.
I20260129 03:00:14.306325 23215 raft_consensus.cc:3060] T 4f8f3a38f6e24ba4857875181f447442 P 974b802141bd4f13beffef2aa03f88f9 [term 6 FOLLOWER]: Advancing to term 7
I20260129 03:00:14.254853 23553 raft_consensus.cc:515] T b189f08e16a74a11980cc55d99359a0a P 974b802141bd4f13beffef2aa03f88f9 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 03:00:14.308526 20439 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55676 (request call id 1488) took 1795 ms. Trace:
W20260129 03:00:14.308604 20515 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:57584 (request call id 1509) took 1751 ms (client timeout 2295 ms). Trace:
I20260129 03:00:14.297353 23537 raft_consensus.cc:2749] T cedce331e1584ac8b636c9b8a72b2ba0 P 974b802141bd4f13beffef2aa03f88f9 [term 8 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260129 03:00:14.286628 23466 raft_consensus.cc:2749] T 4b1876f9a70b49edb3aa74105e66d3b3 P 197b880fa39d432a9b472845876bfaf0 [term 6 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
W20260129 03:00:14.309064 20515 rpcz_store.cc:269] 0129 03:00:12.557724 (+ 0us) service_pool.cc:168] Inserting onto call queue
0129 03:00:12.729221 (+171497us) service_pool.cc:225] Handling call
0129 03:00:14.308452 (+1579231us) spinlock_profiling.cc:243] Waited 1.36 s on lock 0x7b5400316854. stack: 00007f8fe044e373 00007f8fe044e270 00007f8fdfda0574 00007f8fe9a5c6b4 00007f8fe9a5c649 00007f8fe34975a1 00007f8fe3531f7a 00007f8fe84e3647 00007f8fe1a73834 00007f8fe1a737c2 00007f8fe1a736f1 00007f8fe1a7366c 00007f8fe1a72922 00007f8fe186d34c 00007f8fe186c786 00007f8fe186c12f
0129 03:00:14.308549 (+ 97us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":3134674816}
I20260129 03:00:14.308960 20439 rpcz_store.cc:276] 0129 03:00:12.513617 (+ 0us) service_pool.cc:168] Inserting onto call queue
0129 03:00:12.657594 (+143977us) service_pool.cc:225] Handling call
0129 03:00:12.688385 (+ 30791us) raft_consensus.cc:1471] Updating replica for 1 ops
0129 03:00:12.688684 (+ 299us) raft_consensus.cc:1534] Early marking committed up to index 0
0129 03:00:12.688720 (+ 36us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0129 03:00:12.689310 (+ 590us) log.cc:844] Serialized 26 byte log entry
0129 03:00:12.891750 (+202440us) raft_consensus.cc:1645] Marking committed up to 6
0129 03:00:12.891788 (+ 38us) raft_consensus.cc:1697] Filling consensus response to leader.
0129 03:00:12.891856 (+ 68us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0129 03:00:14.267970 (+1376114us) spinlock_profiling.cc:243] Waited 4.3 ms on lock 0x7b440002304c. stack: 00007f8fe044e373 00007f8fe044e270 00007f8fdfda0574 00007f8fe9a5c6b4 00007f8fe9a5c649 00007f8fe02c08b1 00007f8fe038f005 00007f8fe038fa22 00007f8fe30d957b 00007f8fe30d4e4f 00007f8fe30c8c0e 00007f8fe30c7f56 00007f8fe30c7b78 00007f8fe30d15c1 00007f8fe30d1579 00007f8fe30d1509
0129 03:00:14.308239 (+ 40269us) raft_consensus.cc:1685] finished
0129 03:00:14.308264 (+ 25us) raft_consensus.cc:1691] UpdateReplicas() finished
0129 03:00:14.308467 (+ 203us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":9894144,"thread_start_us":201951,"threads_started":1,"wal-append.queue_time_us":425565}
I20260129 03:00:14.321552 20514 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:48544 (request call id 335) took 1271 ms. Trace:
I20260129 03:00:14.323275 20581 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "913fc7d751ed4b7388d889aac30a5f9b" candidate_uuid: "197b880fa39d432a9b472845876bfaf0" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "1c50de89a5a845668696dcb28c3badd6" is_pre_election: true
I20260129 03:00:14.322599 20432 raft_consensus.cc:2468] T 4b1876f9a70b49edb3aa74105e66d3b3 P 197b880fa39d432a9b472845876bfaf0 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1c50de89a5a845668696dcb28c3badd6 in term 6.
I20260129 03:00:14.323150 23450 consensus_queue.cc:1059] T 368953c9102643b2a21f99af36b51905 P 974b802141bd4f13beffef2aa03f88f9 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 }, Status: INVALID_TERM, Last received: 6.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.010s
W20260129 03:00:14.334790 20479 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260129 03:00:14.335518 20581 raft_consensus.cc:2393] T 913fc7d751ed4b7388d889aac30a5f9b P 1c50de89a5a845668696dcb28c3badd6 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 197b880fa39d432a9b472845876bfaf0 in current term 5: Already voted for candidate 974b802141bd4f13beffef2aa03f88f9 in this term.
I20260129 03:00:14.333359 23333 raft_consensus.cc:3055] T a1a39dcd328e4268ae41b988c66b4a75 P 974b802141bd4f13beffef2aa03f88f9 [term 5 LEADER]: Stepping down as leader of term 5
I20260129 03:00:14.337127 20438 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "febffe4a8b394368b69b1d164308fb9a" candidate_uuid: "974b802141bd4f13beffef2aa03f88f9" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "197b880fa39d432a9b472845876bfaf0" is_pre_election: true
I20260129 03:00:14.338819 20433 raft_consensus.cc:1275] T e7d83d4ccf684e76acb29a0dc844e1b8 P 197b880fa39d432a9b472845876bfaf0 [term 7 FOLLOWER]: Refusing update from remote peer 974b802141bd4f13beffef2aa03f88f9: Log matching property violated. Preceding OpId in replica: term: 5 index: 3. Preceding OpId from leader: term: 7 index: 4. (index mismatch)
I20260129 03:00:14.342443 20436 raft_consensus.cc:1275] T fcddab6bca684adf8ea032289a00f90e P 197b880fa39d432a9b472845876bfaf0 [term 10 FOLLOWER]: Refusing update from remote peer 974b802141bd4f13beffef2aa03f88f9: Log matching property violated. Preceding OpId in replica: term: 8 index: 5. Preceding OpId from leader: term: 10 index: 6. (index mismatch)
W20260129 03:00:14.353667 20479 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260129 03:00:14.354085 20480 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260129 03:00:14.353916 20581 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:35754 (request call id 1844) took 2389 ms (client timeout 2019 ms). Trace:
W20260129 03:00:14.354300 20479 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260129 03:00:14.352396 21134 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "87d5fe48a44a408aaf03e894a67ecb2f" candidate_uuid: "974b802141bd4f13beffef2aa03f88f9" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "7a9daf5d42a84615a6383cfec9a10608"
W20260129 03:00:14.354856 20479 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260129 03:00:14.355387 20438 raft_consensus.cc:2468] T febffe4a8b394368b69b1d164308fb9a P 197b880fa39d432a9b472845876bfaf0 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 974b802141bd4f13beffef2aa03f88f9 in term 5.
I20260129 03:00:14.332561 20514 rpcz_store.cc:276] 0129 03:00:13.050539 (+ 0us) service_pool.cc:168] Inserting onto call queue
0129 03:00:13.106985 (+ 56446us) service_pool.cc:225] Handling call
0129 03:00:13.107320 (+ 335us) raft_consensus.cc:1471] Updating replica for 1 ops
0129 03:00:13.107586 (+ 266us) raft_consensus.cc:1534] Early marking committed up to index 0
0129 03:00:13.107621 (+ 35us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0129 03:00:13.122581 (+ 14960us) log.cc:844] Serialized 26 byte log entry
0129 03:00:14.246701 (+1124120us) raft_consensus.cc:1645] Marking committed up to 5
0129 03:00:14.252130 (+ 5429us) raft_consensus.cc:1697] Filling consensus response to leader.
0129 03:00:14.252251 (+ 121us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0129 03:00:14.321309 (+ 69058us) raft_consensus.cc:1685] finished
0129 03:00:14.321329 (+ 20us) raft_consensus.cc:1691] UpdateReplicas() finished
0129 03:00:14.321488 (+ 159us) inbound_call.cc:173] Queueing success response
Metrics: {"thread_start_us":1009474,"threads_started":1,"wal-append.queue_time_us":1134006}
I20260129 03:00:14.356806 20438 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "75be4bd6bd45474795fd472eff2953ac" candidate_uuid: "974b802141bd4f13beffef2aa03f88f9" candidate_term: 11 candidate_status { last_received { term: 10 index: 7 } } ignore_live_leader: false dest_uuid: "197b880fa39d432a9b472845876bfaf0" is_pre_election: true
I20260129 03:00:14.359192 20438 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4408006eb5174de1896b037cf3f18457" candidate_uuid: "974b802141bd4f13beffef2aa03f88f9" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "197b880fa39d432a9b472845876bfaf0" is_pre_election: true
I20260129 03:00:14.361258 20438 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a609ac02fae84725ab743f95f51d33ce" candidate_uuid: "974b802141bd4f13beffef2aa03f88f9" candidate_term: 10 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "197b880fa39d432a9b472845876bfaf0" is_pre_election: true
W20260129 03:00:14.302348 23342 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 4.018s user 0.026s sys 0.042s
W20260129 03:00:14.354527 20581 rpcz_store.cc:269] 0129 03:00:11.964298 (+ 0us) service_pool.cc:168] Inserting onto call queue
0129 03:00:12.546959 (+582661us) service_pool.cc:225] Handling call
0129 03:00:14.323115 (+1776156us) spinlock_profiling.cc:243] Waited 9.99 ms on lock 0x7b58001cb398. stack: 00007f8fe044e373 00007f8fe044e270 00007f8fdfda0574 00007f8fe9a5c6b4 00007f8fe9a5c649 00007f8fe02c08b1 00007f8fe0442db1 00007f8fe0442c62 00007f8fe92eddd9 00007f8fe846d260 00007f8fe8535dbb 00007f8fe8535e41 00007f8fe84e3bd0 00007f8fe84e34b4 00007f8fe1a73834 00007f8fe1a737c2
0129 03:00:14.353840 (+ 30725us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":22984064}
W20260129 03:00:14.345271 20441 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:48356 (request call id 427) took 2649 ms (client timeout 1857 ms). Trace:
W20260129 03:00:14.365062 20441 rpcz_store.cc:269] 0129 03:00:11.696242 (+ 0us) service_pool.cc:168] Inserting onto call queue
0129 03:00:11.696611 (+ 369us) service_pool.cc:225] Handling call
0129 03:00:14.345042 (+2648431us) spinlock_profiling.cc:243] Waited 2.35 s on lock 0x7b54002e2054. stack: 00007f8fe044e373 00007f8fe044e270 00007f8fdfda0574 00007f8fe9a5c6b4 00007f8fe9a5c649 00007f8fe34975a1 00007f8fe3531f7a 00007f8fe84e3647 00007f8fe1a73834 00007f8fe1a737c2 00007f8fe1a736f1 00007f8fe1a7366c 00007f8fe1a72922 00007f8fe186d34c 00007f8fe186c786 00007f8fe186c12f
0129 03:00:14.345190 (+ 148us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":5398451072}
I20260129 03:00:14.372290 20439 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00e79d8f25a34676b1d866eb2068a2b2" candidate_uuid: "974b802141bd4f13beffef2aa03f88f9" candidate_term: 9 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "197b880fa39d432a9b472845876bfaf0" is_pre_election: true
I20260129 03:00:14.374318 20435 raft_consensus.cc:1240] T 2ac659d732314e1288849f08fe5e074e P 197b880fa39d432a9b472845876bfaf0 [term 6 LEADER]: Rejecting Update request from peer 7a9daf5d42a84615a6383cfec9a10608 for earlier term 5. Current term is 6. Ops: []
I20260129 03:00:14.325158 23485 leader_election.cc:290] T fcddab6bca684adf8ea032289a00f90e P 197b880fa39d432a9b472845876bfaf0 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 974b802141bd4f13beffef2aa03f88f9 (127.18.211.2:34517), 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3:37735)
I20260129 03:00:14.326205 23292 leader_election.cc:290] T 75b0b6e82ddc4f92b1fe1d94f9a0b73b P 1c50de89a5a845668696dcb28c3badd6 [CANDIDATE]: Term 5 election: Requested vote from peers 974b802141bd4f13beffef2aa03f88f9 (127.18.211.2:34517), 7a9daf5d42a84615a6383cfec9a10608 (127.18.211.4:40439)
I20260129 03:00:14.325599 21127 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "505580c93c694bbb8ea96fdf9b17ab46" candidate_uuid: "974b802141bd4f13beffef2aa03f88f9" candidate_term: 8 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "7a9daf5d42a84615a6383cfec9a10608" is_pre_election: true
I20260129 03:00:14.376618 20433 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0f20af7fbe5d4db89a6f17948d98912c" candidate_uuid: "974b802141bd4f13beffef2aa03f88f9" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "197b880fa39d432a9b472845876bfaf0"
I20260129 03:00:14.397318 20405 leader_election.cc:304] T 0678b8b468bd43cdbc3e485900844821 P 197b880fa39d432a9b472845876bfaf0 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 197b880fa39d432a9b472845876bfaf0; no voters: 7a9daf5d42a84615a6383cfec9a10608, 974b802141bd4f13beffef2aa03f88f9
I20260129 03:00:14.401819 20480 leader_election.cc:304] T 87d5fe48a44a408aaf03e894a67ecb2f P 974b802141bd4f13beffef2aa03f88f9 [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 974b802141bd4f13beffef2aa03f88f9; no voters: 197b880fa39d432a9b472845876bfaf0, 7a9daf5d42a84615a6383cfec9a10608
I20260129 03:00:14.370812 23304 leader_election.cc:290] T 76f713c3331540d2a5b76164103e9eb1 P 1c50de89a5a845668696dcb28c3badd6 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 197b880fa39d432a9b472845876bfaf0 (127.18.211.1:42027), 974b802141bd4f13beffef2aa03f88f9 (127.18.211.2:34517)
W20260129 03:00:14.394232 21102 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.507s user 0.010s sys 0.024s
I20260129 03:00:14.395407 21134 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "cb0b3f259ebc4310927338a8740e87b8" candidate_uuid: "974b802141bd4f13beffef2aa03f88f9" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "7a9daf5d42a84615a6383cfec9a10608" is_pre_election: true
I20260129 03:00:14.403568 21134 raft_consensus.cc:2393] T cb0b3f259ebc4310927338a8740e87b8 P 7a9daf5d42a84615a6383cfec9a10608 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 974b802141bd4f13beffef2aa03f88f9 in current term 8: Already voted for candidate 1c50de89a5a845668696dcb28c3badd6 in this term.
W20260129 03:00:14.404448 21102 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.527s user 0.010s sys 0.024s
W20260129 03:00:14.405128 21102 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.211.2:34517, user_credentials={real_user=slave}} blocked reactor thread for 549867us
I20260129 03:00:14.354773 23333 raft_consensus.cc:740] T a1a39dcd328e4268ae41b988c66b4a75 P 974b802141bd4f13beffef2aa03f88f9 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 974b802141bd4f13beffef2aa03f88f9, State: Running, Role: LEADER
W20260129 03:00:14.406749 20405 leader_election.cc:336] T 3906b080a5b34a74b8b64128635e8997 P 197b880fa39d432a9b472845876bfaf0 [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3:37735): Timed out: RequestConsensusVote RPC to 127.18.211.3:37735 timed out after 1.980s (SENT)
I20260129 03:00:14.338511 23439 raft_consensus.cc:515] T febffe4a8b394368b69b1d164308fb9a P 7a9daf5d42a84615a6383cfec9a10608 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } }
I20260129 03:00:14.417408 20514 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0678b8b468bd43cdbc3e485900844821" candidate_uuid: "7a9daf5d42a84615a6383cfec9a10608" candidate_term: 9 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "974b802141bd4f13beffef2aa03f88f9" is_pre_election: true
I20260129 03:00:14.417693 20515 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00e79d8f25a34676b1d866eb2068a2b2" candidate_uuid: "7a9daf5d42a84615a6383cfec9a10608" candidate_term: 9 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "974b802141bd4f13beffef2aa03f88f9" is_pre_election: true
I20260129 03:00:14.374214 23152 raft_consensus.cc:3055] T 368953c9102643b2a21f99af36b51905 P 974b802141bd4f13beffef2aa03f88f9 [term 8 LEADER]: Stepping down as leader of term 8
I20260129 03:00:14.418001 20405 leader_election.cc:304] T 3906b080a5b34a74b8b64128635e8997 P 197b880fa39d432a9b472845876bfaf0 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 197b880fa39d432a9b472845876bfaf0; no voters: 1c50de89a5a845668696dcb28c3badd6, 974b802141bd4f13beffef2aa03f88f9
I20260129 03:00:14.396685 23443 raft_consensus.cc:2749] T a0fcc58d6fb6473cafbd9d34d5195269 P 974b802141bd4f13beffef2aa03f88f9 [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260129 03:00:14.418164 20509 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0678b8b468bd43cdbc3e485900844821" candidate_uuid: "7a9daf5d42a84615a6383cfec9a10608" candidate_term: 9 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "974b802141bd4f13beffef2aa03f88f9"
I20260129 03:00:14.309582 23553 leader_election.cc:290] T b189f08e16a74a11980cc55d99359a0a P 974b802141bd4f13beffef2aa03f88f9 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 197b880fa39d432a9b472845876bfaf0 (127.18.211.1:42027), 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3:37735)
I20260129 03:00:14.419831 20480 leader_election.cc:304] T cb0b3f259ebc4310927338a8740e87b8 P 974b802141bd4f13beffef2aa03f88f9 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 974b802141bd4f13beffef2aa03f88f9; no voters: 1c50de89a5a845668696dcb28c3badd6, 7a9daf5d42a84615a6383cfec9a10608
I20260129 03:00:14.333626 23428 raft_consensus.cc:515] T 96609ac68bdf4993a12e1c23355ca013 P 197b880fa39d432a9b472845876bfaf0 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } attrs { replace: true } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } }
I20260129 03:00:14.425545 21134 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00e79d8f25a34676b1d866eb2068a2b2" candidate_uuid: "974b802141bd4f13beffef2aa03f88f9" candidate_term: 9 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "7a9daf5d42a84615a6383cfec9a10608" is_pre_election: true
I20260129 03:00:14.429222 20507 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "87d5fe48a44a408aaf03e894a67ecb2f" candidate_uuid: "7a9daf5d42a84615a6383cfec9a10608" candidate_term: 7 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "974b802141bd4f13beffef2aa03f88f9" is_pre_election: true
I20260129 03:00:14.408491 23485 raft_consensus.cc:2749] T 0678b8b468bd43cdbc3e485900844821 P 197b880fa39d432a9b472845876bfaf0 [term 9 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
W20260129 03:00:14.435937 20405 leader_election.cc:336] T 913fc7d751ed4b7388d889aac30a5f9b P 197b880fa39d432a9b472845876bfaf0 [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3:37735): Timed out: RequestConsensusVote RPC to 127.18.211.3:37735 timed out after 2.019s (SENT)
I20260129 03:00:14.436406 20405 leader_election.cc:304] T 913fc7d751ed4b7388d889aac30a5f9b P 197b880fa39d432a9b472845876bfaf0 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 197b880fa39d432a9b472845876bfaf0; no voters: 1c50de89a5a845668696dcb28c3badd6, 974b802141bd4f13beffef2aa03f88f9
I20260129 03:00:14.406913 23165 raft_consensus.cc:515] T 505580c93c694bbb8ea96fdf9b17ab46 P 7a9daf5d42a84615a6383cfec9a10608 [term 8 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } }
I20260129 03:00:14.438786 23165 leader_election.cc:290] T 505580c93c694bbb8ea96fdf9b17ab46 P 7a9daf5d42a84615a6383cfec9a10608 [CANDIDATE]: Term 8 election: Requested vote from peers 974b802141bd4f13beffef2aa03f88f9 (127.18.211.2:34517), 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3:37735)
I20260129 03:00:14.422079 23215 raft_consensus.cc:2749] T 87d5fe48a44a408aaf03e894a67ecb2f P 974b802141bd4f13beffef2aa03f88f9 [term 7 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20260129 03:00:14.422561 23152 raft_consensus.cc:740] T 368953c9102643b2a21f99af36b51905 P 974b802141bd4f13beffef2aa03f88f9 [term 8 LEADER]: Becoming Follower/Learner. State: Replica: 974b802141bd4f13beffef2aa03f88f9, State: Running, Role: LEADER
I20260129 03:00:14.440425 23152 consensus_queue.cc:260] T 368953c9102643b2a21f99af36b51905 P 974b802141bd4f13beffef2aa03f88f9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 8.5, Last appended by leader: 5, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 03:00:14.441550 23152 raft_consensus.cc:3060] T 368953c9102643b2a21f99af36b51905 P 974b802141bd4f13beffef2aa03f88f9 [term 8 FOLLOWER]: Advancing to term 9
I20260129 03:00:14.444239 23342 raft_consensus.cc:2749] T 3906b080a5b34a74b8b64128635e8997 P 197b880fa39d432a9b472845876bfaf0 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260129 03:00:14.444618 23475 raft_consensus.cc:2749] T 913fc7d751ed4b7388d889aac30a5f9b P 197b880fa39d432a9b472845876bfaf0 [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260129 03:00:14.446677 21127 raft_consensus.cc:2393] T 505580c93c694bbb8ea96fdf9b17ab46 P 7a9daf5d42a84615a6383cfec9a10608 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 974b802141bd4f13beffef2aa03f88f9 in current term 8: Already voted for candidate 7a9daf5d42a84615a6383cfec9a10608 in this term.
W20260129 03:00:14.448238 20405 leader_election.cc:336] T cedce331e1584ac8b636c9b8a72b2ba0 P 197b880fa39d432a9b472845876bfaf0 [CANDIDATE]: Term 8 election: RPC error from VoteRequest() call to peer 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3:37735): Timed out: RequestConsensusVote RPC to 127.18.211.3:37735 timed out after 2.077s (SENT)
I20260129 03:00:14.453368 23439 leader_election.cc:290] T febffe4a8b394368b69b1d164308fb9a P 7a9daf5d42a84615a6383cfec9a10608 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 197b880fa39d432a9b472845876bfaf0 (127.18.211.1:42027), 974b802141bd4f13beffef2aa03f88f9 (127.18.211.2:34517)
I20260129 03:00:14.458153 23446 raft_consensus.cc:515] T add8d4923d1a489e86882c3a25692b95 P 1c50de89a5a845668696dcb28c3badd6 [term 8 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } }
I20260129 03:00:14.459805 20586 raft_consensus.cc:2393] T add8d4923d1a489e86882c3a25692b95 P 1c50de89a5a845668696dcb28c3badd6 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 974b802141bd4f13beffef2aa03f88f9 in current term 8: Already voted for candidate 1c50de89a5a845668696dcb28c3badd6 in this term.
I20260129 03:00:14.459915 23333 consensus_queue.cc:260] T a1a39dcd328e4268ae41b988c66b4a75 P 974b802141bd4f13beffef2aa03f88f9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 5.4, Last appended by leader: 4, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } } peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } }
I20260129 03:00:14.467134 23402 leader_election.cc:290] T 17d6431cc77a491e8464d68db4081a51 P 974b802141bd4f13beffef2aa03f88f9 [CANDIDATE]: Term 6 election: Requested vote from peers 197b880fa39d432a9b472845876bfaf0 (127.18.211.1:42027), 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3:37735)
I20260129 03:00:14.432258 23492 raft_consensus.cc:3060] T cb0b3f259ebc4310927338a8740e87b8 P 974b802141bd4f13beffef2aa03f88f9 [term 7 FOLLOWER]: Advancing to term 8
I20260129 03:00:14.467293 23333 raft_consensus.cc:3060] T a1a39dcd328e4268ae41b988c66b4a75 P 974b802141bd4f13beffef2aa03f88f9 [term 5 FOLLOWER]: Advancing to term 6
I20260129 03:00:14.476994 23428 leader_election.cc:290] T 96609ac68bdf4993a12e1c23355ca013 P 197b880fa39d432a9b472845876bfaf0 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 974b802141bd4f13beffef2aa03f88f9 (127.18.211.2:34517), 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3:37735), 7a9daf5d42a84615a6383cfec9a10608 (127.18.211.4:40439)
I20260129 03:00:14.478327 21134 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c934d31efd3f4e5195605ddc354d14e1" candidate_uuid: "197b880fa39d432a9b472845876bfaf0" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "7a9daf5d42a84615a6383cfec9a10608" is_pre_election: true
I20260129 03:00:14.480157 21134 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "42d27a28d89b409e8807ec0efd97af69" candidate_uuid: "197b880fa39d432a9b472845876bfaf0" candidate_term: 9 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "7a9daf5d42a84615a6383cfec9a10608" is_pre_election: true
I20260129 03:00:14.434269 23570 consensus_queue.cc:1059] T 2ac659d732314e1288849f08fe5e074e P 7a9daf5d42a84615a6383cfec9a10608 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 }, Status: INVALID_TERM, Last received: 3.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.061s
I20260129 03:00:14.488731 23553 raft_consensus.cc:493] T 36ff33be666a4c82954731691dbcf327 P 974b802141bd4f13beffef2aa03f88f9 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260129 03:00:14.489442 23553 raft_consensus.cc:515] T 36ff33be666a4c82954731691dbcf327 P 974b802141bd4f13beffef2aa03f88f9 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 03:00:14.490262 23468 raft_consensus.cc:993] T 7d9494f5584d4d5db8b9adf1c8fc888b P 1c50de89a5a845668696dcb28c3badd6: : Instructing follower 197b880fa39d432a9b472845876bfaf0 to start an election
I20260129 03:00:14.492938 20508 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e4d18c35b0a7493e85d0a4b92956a48f" candidate_uuid: "197b880fa39d432a9b472845876bfaf0" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "974b802141bd4f13beffef2aa03f88f9" is_pre_election: true
W20260129 03:00:14.493896 20508 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:57600 (request call id 1953) took 2692 ms (client timeout 2140 ms). Trace:
W20260129 03:00:14.494330 20508 rpcz_store.cc:269] 0129 03:00:11.801006 (+ 0us) service_pool.cc:168] Inserting onto call queue
0129 03:00:11.806696 (+ 5690us) service_pool.cc:225] Handling call
0129 03:00:14.492726 (+2686030us) spinlock_profiling.cc:243] Waited 1.8 ms on lock 0x7b5800254d98. stack: 00007f8fe044e373 00007f8fe044e270 00007f8fdfda0574 00007f8fe9a5c6b4 00007f8fe9a5c649 00007f8fe02c08b1 00007f8fe0442db1 00007f8fe0442c62 00007f8fe92eddd9 00007f8fe846d260 00007f8fe8535dbb 00007f8fe8535e41 00007f8fe84e3bd0 00007f8fe84e34b4 00007f8fe1a73834 00007f8fe1a737c2
0129 03:00:14.493807 (+ 1081us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":4131072}
I20260129 03:00:14.494709 20405 leader_election.cc:304] T 42d27a28d89b409e8807ec0efd97af69 P 197b880fa39d432a9b472845876bfaf0 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 197b880fa39d432a9b472845876bfaf0; no voters: 1c50de89a5a845668696dcb28c3badd6, 7a9daf5d42a84615a6383cfec9a10608
I20260129 03:00:14.496343 21134 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "febffe4a8b394368b69b1d164308fb9a" candidate_uuid: "974b802141bd4f13beffef2aa03f88f9" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "7a9daf5d42a84615a6383cfec9a10608" is_pre_election: true
I20260129 03:00:14.497189 21134 raft_consensus.cc:2468] T febffe4a8b394368b69b1d164308fb9a P 7a9daf5d42a84615a6383cfec9a10608 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 974b802141bd4f13beffef2aa03f88f9 in term 5.
I20260129 03:00:14.498934 20509 pending_rounds.cc:85] T 926a250fa5834e88b96ccecbefbf5086 P 974b802141bd4f13beffef2aa03f88f9: Aborting all ops after (but not including) 3
I20260129 03:00:14.499290 20509 pending_rounds.cc:107] T 926a250fa5834e88b96ccecbefbf5086 P 974b802141bd4f13beffef2aa03f88f9: Aborting uncommitted NO_OP operation due to leader change: 6.4
I20260129 03:00:14.499552 20509 raft_consensus.cc:2889] T 926a250fa5834e88b96ccecbefbf5086 P 974b802141bd4f13beffef2aa03f88f9 [term 8 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260129 03:00:14.505225 20480 leader_election.cc:304] T febffe4a8b394368b69b1d164308fb9a P 974b802141bd4f13beffef2aa03f88f9 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7a9daf5d42a84615a6383cfec9a10608, 974b802141bd4f13beffef2aa03f88f9; no voters:
I20260129 03:00:14.511390 20441 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "36ff33be666a4c82954731691dbcf327" candidate_uuid: "974b802141bd4f13beffef2aa03f88f9" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "197b880fa39d432a9b472845876bfaf0" is_pre_election: true
I20260129 03:00:14.514744 21134 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "febffe4a8b394368b69b1d164308fb9a" candidate_uuid: "197b880fa39d432a9b472845876bfaf0" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "7a9daf5d42a84615a6383cfec9a10608"
I20260129 03:00:14.515522 21134 raft_consensus.cc:2410] T febffe4a8b394368b69b1d164308fb9a P 7a9daf5d42a84615a6383cfec9a10608 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 197b880fa39d432a9b472845876bfaf0 for term 5 because replica has last-logged OpId of term: 4 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 5.
I20260129 03:00:14.517716 20479 leader_election.cc:304] T 00e79d8f25a34676b1d866eb2068a2b2 P 974b802141bd4f13beffef2aa03f88f9 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 974b802141bd4f13beffef2aa03f88f9; no voters: 197b880fa39d432a9b472845876bfaf0, 7a9daf5d42a84615a6383cfec9a10608
I20260129 03:00:14.519085 20405 leader_election.cc:304] T febffe4a8b394368b69b1d164308fb9a P 197b880fa39d432a9b472845876bfaf0 [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 197b880fa39d432a9b472845876bfaf0; no voters: 7a9daf5d42a84615a6383cfec9a10608, 974b802141bd4f13beffef2aa03f88f9
I20260129 03:00:14.519080 20507 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e33a417cf9c54cf5af73a4ee3aac4d3a" candidate_uuid: "197b880fa39d432a9b472845876bfaf0" candidate_term: 7 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "974b802141bd4f13beffef2aa03f88f9" is_pre_election: true
I20260129 03:00:14.502418 23352 raft_consensus.cc:2749] T 42d27a28d89b409e8807ec0efd97af69 P 197b880fa39d432a9b472845876bfaf0 [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260129 03:00:14.522541 20515 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c2633a6fc34c44579de348d208b0c01e" candidate_uuid: "197b880fa39d432a9b472845876bfaf0" candidate_term: 6 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "974b802141bd4f13beffef2aa03f88f9" is_pre_election: true
I20260129 03:00:14.523900 20439 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "17d6431cc77a491e8464d68db4081a51" candidate_uuid: "974b802141bd4f13beffef2aa03f88f9" candidate_term: 6 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "197b880fa39d432a9b472845876bfaf0"
I20260129 03:00:14.504510 23553 leader_election.cc:290] T 36ff33be666a4c82954731691dbcf327 P 974b802141bd4f13beffef2aa03f88f9 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 197b880fa39d432a9b472845876bfaf0 (127.18.211.1:42027), 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3:37735)
I20260129 03:00:14.527634 20433 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b189f08e16a74a11980cc55d99359a0a" candidate_uuid: "974b802141bd4f13beffef2aa03f88f9" candidate_term: 10 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "197b880fa39d432a9b472845876bfaf0" is_pre_election: true
I20260129 03:00:14.529896 20514 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e7d83d4ccf684e76acb29a0dc844e1b8" candidate_uuid: "197b880fa39d432a9b472845876bfaf0" candidate_term: 8 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "974b802141bd4f13beffef2aa03f88f9" is_pre_election: true
W20260129 03:00:14.532306 21102 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.211.2:34517, user_credentials={real_user=slave}} blocked reactor thread for 99816.7us
I20260129 03:00:14.533558 20515 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fcddab6bca684adf8ea032289a00f90e" candidate_uuid: "197b880fa39d432a9b472845876bfaf0" candidate_term: 11 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "974b802141bd4f13beffef2aa03f88f9" is_pre_election: true
I20260129 03:00:14.508215 23402 raft_consensus.cc:493] T c934d31efd3f4e5195605ddc354d14e1 P 974b802141bd4f13beffef2aa03f88f9 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260129 03:00:14.513664 23572 raft_consensus.cc:2804] T febffe4a8b394368b69b1d164308fb9a P 974b802141bd4f13beffef2aa03f88f9 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260129 03:00:14.537030 21134 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "75f45538dc704d9a8ec4ff8ad9f3ea5b" candidate_uuid: "974b802141bd4f13beffef2aa03f88f9" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "7a9daf5d42a84615a6383cfec9a10608" is_pre_election: true
I20260129 03:00:14.540328 20480 leader_election.cc:304] T 75f45538dc704d9a8ec4ff8ad9f3ea5b P 974b802141bd4f13beffef2aa03f88f9 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 974b802141bd4f13beffef2aa03f88f9; no voters: 1c50de89a5a845668696dcb28c3badd6, 7a9daf5d42a84615a6383cfec9a10608
I20260129 03:00:14.544701 20515 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "96609ac68bdf4993a12e1c23355ca013" candidate_uuid: "197b880fa39d432a9b472845876bfaf0" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "974b802141bd4f13beffef2aa03f88f9" is_pre_election: true
I20260129 03:00:14.545310 20515 raft_consensus.cc:2468] T 96609ac68bdf4993a12e1c23355ca013 P 974b802141bd4f13beffef2aa03f88f9 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 197b880fa39d432a9b472845876bfaf0 in term 6.
I20260129 03:00:14.543901 21102 leader_election.cc:304] T 00e79d8f25a34676b1d866eb2068a2b2 P 7a9daf5d42a84615a6383cfec9a10608 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7a9daf5d42a84615a6383cfec9a10608; no voters: 197b880fa39d432a9b472845876bfaf0, 974b802141bd4f13beffef2aa03f88f9
I20260129 03:00:14.549953 20432 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55676 (request call id 1486) took 2072 ms. Trace:
I20260129 03:00:14.550480 20432 rpcz_store.cc:276] 0129 03:00:12.477818 (+ 0us) service_pool.cc:168] Inserting onto call queue
0129 03:00:12.649000 (+171182us) service_pool.cc:225] Handling call
0129 03:00:14.549699 (+1900699us) spinlock_profiling.cc:243] Waited 1.48 s on lock 0x7b54002ae254. stack: 00007f8fe044e373 00007f8fe044e270 00007f8fdfda0574 00007f8fe9a5c6b4 00007f8fe9a5c649 00007f8fe34975a1 00007f8fe3531f7a 00007f8fe84e3647 00007f8fe1a73834 00007f8fe1a737c2 00007f8fe1a736f1 00007f8fe1a7366c 00007f8fe1a72922 00007f8fe186d34c 00007f8fe186c786 00007f8fe186c12f
0129 03:00:14.549842 (+ 143us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":3415436672}
I20260129 03:00:14.552881 21134 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "96609ac68bdf4993a12e1c23355ca013" candidate_uuid: "197b880fa39d432a9b472845876bfaf0" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "7a9daf5d42a84615a6383cfec9a10608" is_pre_election: true
I20260129 03:00:14.532635 23485 raft_consensus.cc:2749] T febffe4a8b394368b69b1d164308fb9a P 197b880fa39d432a9b472845876bfaf0 [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20260129 03:00:14.538012 23582 raft_consensus.cc:2749] T 00e79d8f25a34676b1d866eb2068a2b2 P 974b802141bd4f13beffef2aa03f88f9 [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260129 03:00:14.525414 23443 consensus_queue.cc:1048] T e7d83d4ccf684e76acb29a0dc844e1b8 P 974b802141bd4f13beffef2aa03f88f9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.068s
I20260129 03:00:14.617403 20514 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c934d31efd3f4e5195605ddc354d14e1" candidate_uuid: "197b880fa39d432a9b472845876bfaf0" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "974b802141bd4f13beffef2aa03f88f9" is_pre_election: true
I20260129 03:00:14.625739 20439 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c934d31efd3f4e5195605ddc354d14e1" candidate_uuid: "7a9daf5d42a84615a6383cfec9a10608" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "197b880fa39d432a9b472845876bfaf0" is_pre_election: true
I20260129 03:00:14.538544 23572 raft_consensus.cc:493] T febffe4a8b394368b69b1d164308fb9a P 974b802141bd4f13beffef2aa03f88f9 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260129 03:00:14.635046 20439 raft_consensus.cc:2468] T c934d31efd3f4e5195605ddc354d14e1 P 197b880fa39d432a9b472845876bfaf0 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7a9daf5d42a84615a6383cfec9a10608 in term 6.
I20260129 03:00:14.535740 23152 consensus_queue.cc:1048] T fcddab6bca684adf8ea032289a00f90e P 974b802141bd4f13beffef2aa03f88f9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.094s
I20260129 03:00:14.653662 21134 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a644c02b16af47679824ccd060ac10fd" candidate_uuid: "974b802141bd4f13beffef2aa03f88f9" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "7a9daf5d42a84615a6383cfec9a10608" is_pre_election: true
I20260129 03:00:14.656472 20507 raft_consensus.cc:2468] T e33a417cf9c54cf5af73a4ee3aac4d3a P 974b802141bd4f13beffef2aa03f88f9 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 197b880fa39d432a9b472845876bfaf0 in term 7.
I20260129 03:00:14.627761 23536 raft_consensus.cc:2749] T 75f45538dc704d9a8ec4ff8ad9f3ea5b P 974b802141bd4f13beffef2aa03f88f9 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
W20260129 03:00:14.633801 23509 consensus_meta.cc:287] T 45cc5f3d2688486d9b6dbe7e666dedde P 974b802141bd4f13beffef2aa03f88f9: Time spent flushing consensus metadata: real 0.531s user 0.007s sys 0.009s
I20260129 03:00:14.666407 20433 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "87d5fe48a44a408aaf03e894a67ecb2f" candidate_uuid: "7a9daf5d42a84615a6383cfec9a10608" candidate_term: 7 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "197b880fa39d432a9b472845876bfaf0" is_pre_election: true
I20260129 03:00:14.667584 20480 leader_election.cc:304] T a644c02b16af47679824ccd060ac10fd P 974b802141bd4f13beffef2aa03f88f9 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 974b802141bd4f13beffef2aa03f88f9; no voters: 1c50de89a5a845668696dcb28c3badd6, 7a9daf5d42a84615a6383cfec9a10608
I20260129 03:00:14.625751 23553 raft_consensus.cc:493] T 75b0b6e82ddc4f92b1fe1d94f9a0b73b P 974b802141bd4f13beffef2aa03f88f9 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 7a9daf5d42a84615a6383cfec9a10608)
I20260129 03:00:14.547299 23402 raft_consensus.cc:515] T c934d31efd3f4e5195605ddc354d14e1 P 974b802141bd4f13beffef2aa03f88f9 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } }
W20260129 03:00:14.653843 23579 log.cc:927] Time spent T a0fcc58d6fb6473cafbd9d34d5195269 P 974b802141bd4f13beffef2aa03f88f9: Append to log took a long time: real 0.127s user 0.000s sys 0.001s
I20260129 03:00:14.668910 20432 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0678b8b468bd43cdbc3e485900844821" candidate_uuid: "7a9daf5d42a84615a6383cfec9a10608" candidate_term: 9 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "197b880fa39d432a9b472845876bfaf0"
I20260129 03:00:14.638989 23572 raft_consensus.cc:3060] T febffe4a8b394368b69b1d164308fb9a P 974b802141bd4f13beffef2aa03f88f9 [term 5 FOLLOWER]: Advancing to term 6
I20260129 03:00:14.679605 20515 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "505580c93c694bbb8ea96fdf9b17ab46" candidate_uuid: "7a9daf5d42a84615a6383cfec9a10608" candidate_term: 8 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "974b802141bd4f13beffef2aa03f88f9"
I20260129 03:00:14.680160 20437 raft_consensus.cc:1240] T 2e58e9a5a8aa4cecafb464addb4f35ed P 197b880fa39d432a9b472845876bfaf0 [term 5 FOLLOWER]: Rejecting Update request from peer 7a9daf5d42a84615a6383cfec9a10608 for earlier term 4. Current term is 5. Ops: []
I20260129 03:00:14.680346 20515 raft_consensus.cc:3060] T 505580c93c694bbb8ea96fdf9b17ab46 P 974b802141bd4f13beffef2aa03f88f9 [term 7 FOLLOWER]: Advancing to term 8
I20260129 03:00:14.671772 20439 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fd8311b779af4e1fa1402b2a02a602f2" candidate_uuid: "7a9daf5d42a84615a6383cfec9a10608" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "197b880fa39d432a9b472845876bfaf0" is_pre_election: true
I20260129 03:00:14.632601 23569 raft_consensus.cc:2749] T 00e79d8f25a34676b1d866eb2068a2b2 P 7a9daf5d42a84615a6383cfec9a10608 [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260129 03:00:14.676729 20432 raft_consensus.cc:2393] T 0678b8b468bd43cdbc3e485900844821 P 197b880fa39d432a9b472845876bfaf0 [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate 7a9daf5d42a84615a6383cfec9a10608 in current term 9: Already voted for candidate 974b802141bd4f13beffef2aa03f88f9 in this term.
I20260129 03:00:14.684484 21099 leader_election.cc:304] T c934d31efd3f4e5195605ddc354d14e1 P 7a9daf5d42a84615a6383cfec9a10608 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 197b880fa39d432a9b472845876bfaf0, 7a9daf5d42a84615a6383cfec9a10608; no voters: 974b802141bd4f13beffef2aa03f88f9
I20260129 03:00:14.682060 20441 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "febffe4a8b394368b69b1d164308fb9a" candidate_uuid: "7a9daf5d42a84615a6383cfec9a10608" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "197b880fa39d432a9b472845876bfaf0" is_pre_election: true
I20260129 03:00:14.686820 20441 raft_consensus.cc:2468] T febffe4a8b394368b69b1d164308fb9a P 197b880fa39d432a9b472845876bfaf0 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7a9daf5d42a84615a6383cfec9a10608 in term 5.
I20260129 03:00:14.691556 20516 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "febffe4a8b394368b69b1d164308fb9a" candidate_uuid: "7a9daf5d42a84615a6383cfec9a10608" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "974b802141bd4f13beffef2aa03f88f9" is_pre_election: true
I20260129 03:00:14.694307 23567 raft_consensus.cc:2804] T c934d31efd3f4e5195605ddc354d14e1 P 7a9daf5d42a84615a6383cfec9a10608 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260129 03:00:14.694797 23567 raft_consensus.cc:493] T c934d31efd3f4e5195605ddc354d14e1 P 7a9daf5d42a84615a6383cfec9a10608 [term 5 FOLLOWER]: Starting leader election (detected failure of leader 197b880fa39d432a9b472845876bfaf0)
I20260129 03:00:14.695191 23567 raft_consensus.cc:3060] T c934d31efd3f4e5195605ddc354d14e1 P 7a9daf5d42a84615a6383cfec9a10608 [term 5 FOLLOWER]: Advancing to term 6
W20260129 03:00:14.696115 23576 log.cc:927] Time spent T 00e79d8f25a34676b1d866eb2068a2b2 P 7a9daf5d42a84615a6383cfec9a10608: Append to log took a long time: real 0.055s user 0.001s sys 0.000s
I20260129 03:00:14.697388 21130 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34060 (request call id 543) took 1229 ms. Trace:
I20260129 03:00:14.699000 23398 raft_consensus.cc:2749] T 0f20af7fbe5d4db89a6f17948d98912c P 1c50de89a5a845668696dcb28c3badd6 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
W20260129 03:00:14.698367 23577 log.cc:927] Time spent T 17d6431cc77a491e8464d68db4081a51 P 197b880fa39d432a9b472845876bfaf0: Append to log took a long time: real 0.102s user 0.000s sys 0.000s
I20260129 03:00:14.699630 20513 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57600 (request call id 1955) took 2887 ms. Trace:
I20260129 03:00:14.700569 23402 leader_election.cc:290] T c934d31efd3f4e5195605ddc354d14e1 P 974b802141bd4f13beffef2aa03f88f9 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 197b880fa39d432a9b472845876bfaf0 (127.18.211.1:42027), 7a9daf5d42a84615a6383cfec9a10608 (127.18.211.4:40439)
I20260129 03:00:14.700021 20513 rpcz_store.cc:276] 0129 03:00:11.812632 (+ 0us) service_pool.cc:168] Inserting onto call queue
0129 03:00:11.818400 (+ 5768us) service_pool.cc:225] Handling call
0129 03:00:11.818756 (+ 356us) raft_consensus.cc:1471] Updating replica for 1 ops
0129 03:00:14.199472 (+2380716us) spinlock_profiling.cc:243] Waited 1.35 ms on lock 0x7b18004fb400. stack: 00007f8fe044e373 00007f8fe044e270 00007f8fdfda0574 00007f8fe9a5c6b4 00007f8fe9a5c649 00007f8fe92d4c91 00007f8fe92e1671 00007f8fe30d585c 00007f8fe04632eb 00007f8fe924b709 00007f8fe047f57b 00007f8fe047d81f 00007f8fe047d335 00007f8fe3525ce0 00007f8fe352687b 00007f8fe35312cb
0129 03:00:14.199661 (+ 189us) raft_consensus.cc:1534] Early marking committed up to index 0
0129 03:00:14.199710 (+ 49us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0129 03:00:14.206115 (+ 6405us) log.cc:844] Serialized 26 byte log entry
0129 03:00:14.393819 (+187704us) raft_consensus.cc:1645] Marking committed up to 3
0129 03:00:14.393856 (+ 37us) raft_consensus.cc:1697] Filling consensus response to leader.
0129 03:00:14.393937 (+ 81us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0129 03:00:14.699351 (+305414us) raft_consensus.cc:1685] finished
0129 03:00:14.699372 (+ 21us) raft_consensus.cc:1691] UpdateReplicas() finished
0129 03:00:14.699557 (+ 185us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":19274,"raft.run_cpu_time_us":166,"raft.run_wall_time_us":18068,"spinlock_wait_cycles":3096832,"thread_start_us":178938,"threads_started":2,"wal-append.queue_time_us":300010}
I20260129 03:00:14.704222 20585 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35754 (request call id 1858) took 2645 ms. Trace:
I20260129 03:00:14.704651 21134 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1299723a4476420ab02fd8bfd7aa0645" candidate_uuid: "197b880fa39d432a9b472845876bfaf0" candidate_term: 11 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "7a9daf5d42a84615a6383cfec9a10608" is_pre_election: true
I20260129 03:00:14.705717 20513 raft_consensus.cc:1240] T 17d6431cc77a491e8464d68db4081a51 P 974b802141bd4f13beffef2aa03f88f9 [term 6 FOLLOWER]: Rejecting Update request from peer 197b880fa39d432a9b472845876bfaf0 for earlier term 5. Current term is 6. Ops: []
I20260129 03:00:14.704756 20585 rpcz_store.cc:276] 0129 03:00:12.058399 (+ 0us) service_pool.cc:168] Inserting onto call queue
0129 03:00:12.911686 (+853287us) service_pool.cc:225] Handling call
0129 03:00:12.921025 (+ 9339us) raft_consensus.cc:1471] Updating replica for 1 ops
0129 03:00:12.921469 (+ 444us) raft_consensus.cc:1534] Early marking committed up to index 0
0129 03:00:12.921512 (+ 43us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0129 03:00:12.922056 (+ 544us) log.cc:844] Serialized 26 byte log entry
0129 03:00:13.251022 (+328966us) raft_consensus.cc:1645] Marking committed up to 7
0129 03:00:14.102235 (+851213us) log.cc:844] Serialized 14 byte log entry
0129 03:00:14.703875 (+601640us) raft_consensus.cc:1697] Filling consensus response to leader.
0129 03:00:14.703998 (+ 123us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0129 03:00:14.704044 (+ 46us) raft_consensus.cc:1685] finished
0129 03:00:14.704057 (+ 13us) raft_consensus.cc:1691] UpdateReplicas() finished
0129 03:00:14.704172 (+ 115us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":622817,"prepare.queue_time_us":868165,"prepare.run_cpu_time_us":32,"prepare.run_wall_time_us":32,"raft.queue_time_us":5603,"raft.run_cpu_time_us":187,"raft.run_wall_time_us":33016,"thread_start_us":984541,"threads_started":2,"wal-append.queue_time_us":567017}
I20260129 03:00:14.668048 23583 raft_consensus.cc:493] T 3e2d15f8640b407195210c476e09df91 P 974b802141bd4f13beffef2aa03f88f9 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 7a9daf5d42a84615a6383cfec9a10608)
I20260129 03:00:14.704169 20514 raft_consensus.cc:2468] T c934d31efd3f4e5195605ddc354d14e1 P 974b802141bd4f13beffef2aa03f88f9 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 197b880fa39d432a9b472845876bfaf0 in term 6.
I20260129 03:00:14.706760 20585 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fd8311b779af4e1fa1402b2a02a602f2" candidate_uuid: "7a9daf5d42a84615a6383cfec9a10608" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "1c50de89a5a845668696dcb28c3badd6" is_pre_election: true
I20260129 03:00:14.700591 20583 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35754 (request call id 1851) took 2734 ms. Trace:
I20260129 03:00:14.697772 21130 rpcz_store.cc:276] 0129 03:00:13.467712 (+ 0us) service_pool.cc:168] Inserting onto call queue
0129 03:00:14.093064 (+625352us) service_pool.cc:225] Handling call
0129 03:00:14.106477 (+ 13413us) raft_consensus.cc:1471] Updating replica for 1 ops
0129 03:00:14.106874 (+ 397us) raft_consensus.cc:1534] Early marking committed up to index 0
0129 03:00:14.106913 (+ 39us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0129 03:00:14.107452 (+ 539us) log.cc:844] Serialized 26 byte log entry
0129 03:00:14.421650 (+314198us) raft_consensus.cc:1645] Marking committed up to 6
0129 03:00:14.429425 (+ 7775us) raft_consensus.cc:1697] Filling consensus response to leader.
0129 03:00:14.435581 (+ 6156us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0129 03:00:14.697138 (+261557us) raft_consensus.cc:1685] finished
0129 03:00:14.697163 (+ 25us) raft_consensus.cc:1691] UpdateReplicas() finished
0129 03:00:14.697324 (+ 161us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":7380,"raft.run_cpu_time_us":66,"raft.run_wall_time_us":66,"thread_start_us":294464,"threads_started":1,"wal-append.queue_time_us":419680}
I20260129 03:00:14.705888 21134 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "96609ac68bdf4993a12e1c23355ca013" candidate_uuid: "974b802141bd4f13beffef2aa03f88f9" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "7a9daf5d42a84615a6383cfec9a10608" is_pre_election: true
I20260129 03:00:14.698315 20590 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35754 (request call id 1854) took 2641 ms. Trace:
I20260129 03:00:14.713264 20590 rpcz_store.cc:276] 0129 03:00:12.057292 (+ 0us) service_pool.cc:168] Inserting onto call queue
0129 03:00:12.893293 (+836001us) service_pool.cc:225] Handling call
0129 03:00:12.893641 (+ 348us) raft_consensus.cc:1471] Updating replica for 2 ops
0129 03:00:12.927111 (+ 33470us) raft_consensus.cc:1534] Early marking committed up to index 0
0129 03:00:12.927158 (+ 47us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0129 03:00:12.963317 (+ 36159us) log.cc:844] Serialized 52 byte log entry
0129 03:00:13.857131 (+893814us) raft_consensus.cc:1645] Marking committed up to 5
0129 03:00:13.904802 (+ 47671us) log.cc:844] Serialized 14 byte log entry
0129 03:00:13.959717 (+ 54915us) log.cc:844] Serialized 14 byte log entry
0129 03:00:14.697716 (+737999us) spinlock_profiling.cc:243] Waited 2.35 ms on lock 0x7b18004b8620. stack: 00007f8fe044e373 00007f8fe044e270 00007f8fdfda0574 00007f8fe9a5c6b4 00007f8fe9a5c649 00007f8fe92d4c91 00007f8fe92e1671 00007f8fe03f546a 00007f8fe9a7b5c9 00007f8fe02b5160 00007f8fe047d3c4 00007f8fe047d335 00007f8fe3525ce0 00007f8fe3538396 00007f8fe3548f07 00007f8fe3548e88
0129 03:00:14.697902 (+ 186us) raft_consensus.cc:1697] Filling consensus response to leader.
0129 03:00:14.698001 (+ 99us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0129 03:00:14.698140 (+ 139us) raft_consensus.cc:1685] finished
0129 03:00:14.698155 (+ 15us) raft_consensus.cc:1691] UpdateReplicas() finished
0129 03:00:14.698251 (+ 96us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":683569,"prepare.queue_time_us":461709,"prepare.run_cpu_time_us":55,"prepare.run_wall_time_us":56,"raft.queue_time_us":804784,"raft.run_cpu_time_us":343,"raft.run_wall_time_us":12039,"spinlock_wait_cycles":5394176,"thread_start_us":742125,"threads_started":1,"wal-append.queue_time_us":897776}
I20260129 03:00:14.710309 20583 rpcz_store.cc:276] 0129 03:00:11.966544 (+ 0us) service_pool.cc:168] Inserting onto call queue
0129 03:00:12.738843 (+772299us) service_pool.cc:225] Handling call
0129 03:00:12.764265 (+ 25422us) raft_consensus.cc:1471] Updating replica for 1 ops
0129 03:00:12.764422 (+ 157us) raft_consensus.cc:1534] Early marking committed up to index 0
0129 03:00:12.764437 (+ 15us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0129 03:00:12.813740 (+ 49303us) log.cc:844] Serialized 26 byte log entry
0129 03:00:13.120985 (+307245us) raft_consensus.cc:1645] Marking committed up to 5
0129 03:00:14.068960 (+947975us) log.cc:844] Serialized 14 byte log entry
0129 03:00:14.140192 (+ 71232us) spinlock_profiling.cc:243] Waited 3.54 ms on lock 0x7b180036eaa0. stack: 00007f8fe044e373 00007f8fe044e270 00007f8fdfda0574 00007f8fe9a5c6b4 00007f8fe9a5c649 00007f8fe92d4c91 00007f8fe92e1671 00007f8fe04802d3 00007f8fe04826a1 00007f8fe0482659 00007f8fe04825e9 00007f8fe04825b1 00007f8fe04818ad 0000000000379254 0000000000379099 00007f8fe046369a
0129 03:00:14.700205 (+560013us) raft_consensus.cc:1697] Filling consensus response to leader.
0129 03:00:14.700280 (+ 75us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0129 03:00:14.700322 (+ 42us) raft_consensus.cc:1685] finished
0129 03:00:14.700422 (+ 100us) raft_consensus.cc:1691] UpdateReplicas() finished
0129 03:00:14.700537 (+ 115us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":630945,"prepare.queue_time_us":209819,"prepare.run_cpu_time_us":45,"prepare.run_wall_time_us":46,"spinlock_wait_cycles":8146304,"thread_start_us":1070271,"threads_started":2,"wal-append.queue_time_us":513382}
I20260129 03:00:14.698315 20587 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35754 (request call id 1856) took 2640 ms. Trace:
I20260129 03:00:14.717365 20480 leader_election.cc:304] T 96609ac68bdf4993a12e1c23355ca013 P 974b802141bd4f13beffef2aa03f88f9 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 1c50de89a5a845668696dcb28c3badd6, 974b802141bd4f13beffef2aa03f88f9; no voters: 197b880fa39d432a9b472845876bfaf0, 7a9daf5d42a84615a6383cfec9a10608
I20260129 03:00:14.694278 20515 raft_consensus.cc:2468] T 505580c93c694bbb8ea96fdf9b17ab46 P 974b802141bd4f13beffef2aa03f88f9 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7a9daf5d42a84615a6383cfec9a10608 in term 8.
W20260129 03:00:14.721079 20553 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.211.1:42027, user_credentials={real_user=slave}} blocked reactor thread for 565884us
I20260129 03:00:14.717165 20587 rpcz_store.cc:276] 0129 03:00:12.057828 (+ 0us) service_pool.cc:168] Inserting onto call queue
0129 03:00:12.897502 (+839674us) service_pool.cc:225] Handling call
0129 03:00:12.908485 (+ 10983us) raft_consensus.cc:1471] Updating replica for 2 ops
0129 03:00:12.908878 (+ 393us) raft_consensus.cc:1534] Early marking committed up to index 0
0129 03:00:12.908907 (+ 29us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0129 03:00:12.909425 (+ 518us) log.cc:844] Serialized 52 byte log entry
0129 03:00:13.203488 (+294063us) raft_consensus.cc:1645] Marking committed up to 4
0129 03:00:13.961511 (+758023us) log.cc:844] Serialized 14 byte log entry
0129 03:00:14.698006 (+736495us) raft_consensus.cc:1697] Filling consensus response to leader.
0129 03:00:14.698082 (+ 76us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0129 03:00:14.698126 (+ 44us) raft_consensus.cc:1685] finished
0129 03:00:14.698137 (+ 11us) raft_consensus.cc:1691] UpdateReplicas() finished
0129 03:00:14.698251 (+ 114us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":769331,"prepare.queue_time_us":915435,"prepare.run_cpu_time_us":42,"prepare.run_wall_time_us":43,"raft.queue_time_us":738615,"raft.run_cpu_time_us":218,"raft.run_wall_time_us":7000,"thread_start_us":982425,"threads_started":2,"wal-append.queue_time_us":548111}
I20260129 03:00:14.668241 23446 leader_election.cc:290] T add8d4923d1a489e86882c3a25692b95 P 1c50de89a5a845668696dcb28c3badd6 [CANDIDATE]: Term 8 election: Requested vote from peers 974b802141bd4f13beffef2aa03f88f9 (127.18.211.2:34517), 7a9daf5d42a84615a6383cfec9a10608 (127.18.211.4:40439)
I20260129 03:00:14.706604 23583 raft_consensus.cc:515] T 3e2d15f8640b407195210c476e09df91 P 974b802141bd4f13beffef2aa03f88f9 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } }
I20260129 03:00:14.724912 21102 leader_election.cc:304] T 505580c93c694bbb8ea96fdf9b17ab46 P 7a9daf5d42a84615a6383cfec9a10608 [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7a9daf5d42a84615a6383cfec9a10608, 974b802141bd4f13beffef2aa03f88f9; no voters:
I20260129 03:00:14.720117 23450 raft_consensus.cc:2749] T 96609ac68bdf4993a12e1c23355ca013 P 974b802141bd4f13beffef2aa03f88f9 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260129 03:00:14.726132 21134 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "87d5fe48a44a408aaf03e894a67ecb2f" candidate_uuid: "197b880fa39d432a9b472845876bfaf0" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "7a9daf5d42a84615a6383cfec9a10608"
W20260129 03:00:14.726446 20556 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.211.2:34517, user_credentials={real_user=slave}} blocked reactor thread for 472602us
I20260129 03:00:14.731324 20556 leader_election.cc:304] T 926a250fa5834e88b96ccecbefbf5086 P 1c50de89a5a845668696dcb28c3badd6 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1c50de89a5a845668696dcb28c3badd6; no voters: 197b880fa39d432a9b472845876bfaf0, 974b802141bd4f13beffef2aa03f88f9
W20260129 03:00:14.727113 20554 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.211.4:40439, user_credentials={real_user=slave}} blocked reactor thread for 450240us
I20260129 03:00:14.732604 20590 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "505580c93c694bbb8ea96fdf9b17ab46" candidate_uuid: "7a9daf5d42a84615a6383cfec9a10608" candidate_term: 8 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "1c50de89a5a845668696dcb28c3badd6"
I20260129 03:00:14.724557 20583 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:48526 (request call id 402) took 2397 ms. Trace:
I20260129 03:00:14.733294 20583 rpcz_store.cc:276] 0129 03:00:12.327239 (+ 0us) service_pool.cc:168] Inserting onto call queue
0129 03:00:14.723687 (+2396448us) service_pool.cc:225] Handling call
0129 03:00:14.724053 (+ 366us) raft_consensus.cc:1471] Updating replica for 0 ops
0129 03:00:14.724238 (+ 185us) raft_consensus.cc:1534] Early marking committed up to index 0
0129 03:00:14.724263 (+ 25us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0129 03:00:14.724307 (+ 44us) raft_consensus.cc:1645] Marking committed up to 9
0129 03:00:14.724342 (+ 35us) raft_consensus.cc:1697] Filling consensus response to leader.
0129 03:00:14.724399 (+ 57us) raft_consensus.cc:1691] UpdateReplicas() finished
0129 03:00:14.724499 (+ 100us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260129 03:00:14.734067 20590 raft_consensus.cc:1240] T a5a05fbcbbcf4916bbc01fd1a5b5e90d P 1c50de89a5a845668696dcb28c3badd6 [term 8 LEADER]: Rejecting Update request from peer 974b802141bd4f13beffef2aa03f88f9 for earlier term 7. Current term is 8. Ops: []
I20260129 03:00:14.682979 23536 raft_consensus.cc:2749] T a644c02b16af47679824ccd060ac10fd P 974b802141bd4f13beffef2aa03f88f9 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260129 03:00:14.734673 20590 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35752 (request call id 1742) took 1661 ms. Trace:
I20260129 03:00:14.738173 20590 rpcz_store.cc:276] 0129 03:00:13.073161 (+ 0us) service_pool.cc:168] Inserting onto call queue
0129 03:00:14.733629 (+1660468us) service_pool.cc:225] Handling call
0129 03:00:14.733906 (+ 277us) raft_consensus.cc:1471] Updating replica for 0 ops
0129 03:00:14.734443 (+ 537us) raft_consensus.cc:1697] Filling consensus response to leader.
0129 03:00:14.734620 (+ 177us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260129 03:00:14.732616 20554 leader_election.cc:304] T 3e2d15f8640b407195210c476e09df91 P 1c50de89a5a845668696dcb28c3badd6 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1c50de89a5a845668696dcb28c3badd6; no voters: 7a9daf5d42a84615a6383cfec9a10608, 974b802141bd4f13beffef2aa03f88f9
I20260129 03:00:14.720994 23325 raft_consensus.cc:493] T e7d83d4ccf684e76acb29a0dc844e1b8 P 1c50de89a5a845668696dcb28c3badd6 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260129 03:00:14.720708 23497 raft_consensus.cc:493] T a8479d1fe0bd41e5946186e3a6ca8c7a P 1c50de89a5a845668696dcb28c3badd6 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 974b802141bd4f13beffef2aa03f88f9)
I20260129 03:00:14.668538 23509 raft_consensus.cc:515] T 45cc5f3d2688486d9b6dbe7e666dedde P 974b802141bd4f13beffef2aa03f88f9 [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } }
I20260129 03:00:14.737810 23165 raft_consensus.cc:2804] T 505580c93c694bbb8ea96fdf9b17ab46 P 7a9daf5d42a84615a6383cfec9a10608 [term 8 FOLLOWER]: Leader election won for term 8
I20260129 03:00:14.685472 23553 raft_consensus.cc:515] T 75b0b6e82ddc4f92b1fe1d94f9a0b73b P 974b802141bd4f13beffef2aa03f88f9 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } }
I20260129 03:00:14.760897 23165 raft_consensus.cc:697] T 505580c93c694bbb8ea96fdf9b17ab46 P 7a9daf5d42a84615a6383cfec9a10608 [term 8 LEADER]: Becoming Leader. State: Replica: 7a9daf5d42a84615a6383cfec9a10608, State: Running, Role: LEADER
I20260129 03:00:14.741967 23565 raft_consensus.cc:2749] T 3906b080a5b34a74b8b64128635e8997 P 1c50de89a5a845668696dcb28c3badd6 [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20260129 03:00:14.758610 23325 raft_consensus.cc:515] T e7d83d4ccf684e76acb29a0dc844e1b8 P 1c50de89a5a845668696dcb28c3badd6 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 03:00:14.768774 23283 raft_consensus.cc:1081] T 7d9494f5584d4d5db8b9adf1c8fc888b P 1c50de89a5a845668696dcb28c3badd6 [term 4 LEADER]: Signalling peer 197b880fa39d432a9b472845876bfaf0 to start an election
I20260129 03:00:14.777050 23583 leader_election.cc:290] T 3e2d15f8640b407195210c476e09df91 P 974b802141bd4f13beffef2aa03f88f9 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3:37735), 7a9daf5d42a84615a6383cfec9a10608 (127.18.211.4:40439)
I20260129 03:00:14.777436 23470 raft_consensus.cc:3060] T c2633a6fc34c44579de348d208b0c01e P 1c50de89a5a845668696dcb28c3badd6 [term 4 FOLLOWER]: Advancing to term 5
I20260129 03:00:14.780879 23325 leader_election.cc:290] T e7d83d4ccf684e76acb29a0dc844e1b8 P 1c50de89a5a845668696dcb28c3badd6 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 197b880fa39d432a9b472845876bfaf0 (127.18.211.1:42027), 974b802141bd4f13beffef2aa03f88f9 (127.18.211.2:34517)
I20260129 03:00:14.760190 23497 raft_consensus.cc:515] T a8479d1fe0bd41e5946186e3a6ca8c7a P 1c50de89a5a845668696dcb28c3badd6 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 03:00:14.782531 23497 leader_election.cc:290] T a8479d1fe0bd41e5946186e3a6ca8c7a P 1c50de89a5a845668696dcb28c3badd6 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 197b880fa39d432a9b472845876bfaf0 (127.18.211.1:42027), 974b802141bd4f13beffef2aa03f88f9 (127.18.211.2:34517)
I20260129 03:00:14.784291 23509 leader_election.cc:290] T 45cc5f3d2688486d9b6dbe7e666dedde P 974b802141bd4f13beffef2aa03f88f9 [CANDIDATE]: Term 7 election: Requested vote from peers 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3:37735), 7a9daf5d42a84615a6383cfec9a10608 (127.18.211.4:40439)
I20260129 03:00:14.784699 23490 raft_consensus.cc:3060] T a5a05fbcbbcf4916bbc01fd1a5b5e90d P 974b802141bd4f13beffef2aa03f88f9 [term 7 FOLLOWER]: Advancing to term 8
I20260129 03:00:14.785884 23572 raft_consensus.cc:515] T febffe4a8b394368b69b1d164308fb9a P 974b802141bd4f13beffef2aa03f88f9 [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } }
I20260129 03:00:14.786715 23553 leader_election.cc:290] T 75b0b6e82ddc4f92b1fe1d94f9a0b73b P 974b802141bd4f13beffef2aa03f88f9 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3:37735), 7a9daf5d42a84615a6383cfec9a10608 (127.18.211.4:40439)
I20260129 03:00:14.787936 23572 leader_election.cc:290] T febffe4a8b394368b69b1d164308fb9a P 974b802141bd4f13beffef2aa03f88f9 [CANDIDATE]: Term 6 election: Requested vote from peers 197b880fa39d432a9b472845876bfaf0 (127.18.211.1:42027), 7a9daf5d42a84615a6383cfec9a10608 (127.18.211.4:40439)
I20260129 03:00:14.788478 23492 raft_consensus.cc:2749] T cb0b3f259ebc4310927338a8740e87b8 P 974b802141bd4f13beffef2aa03f88f9 [term 8 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260129 03:00:14.788823 23567 raft_consensus.cc:515] T c934d31efd3f4e5195605ddc354d14e1 P 7a9daf5d42a84615a6383cfec9a10608 [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } }
I20260129 03:00:14.791555 23567 leader_election.cc:290] T c934d31efd3f4e5195605ddc354d14e1 P 7a9daf5d42a84615a6383cfec9a10608 [CANDIDATE]: Term 6 election: Requested vote from peers 197b880fa39d432a9b472845876bfaf0 (127.18.211.1:42027), 974b802141bd4f13beffef2aa03f88f9 (127.18.211.2:34517)
I20260129 03:00:14.793478 23480 raft_consensus.cc:3055] T 2ac659d732314e1288849f08fe5e074e P 7a9daf5d42a84615a6383cfec9a10608 [term 5 LEADER]: Stepping down as leader of term 5
I20260129 03:00:14.793835 23480 raft_consensus.cc:740] T 2ac659d732314e1288849f08fe5e074e P 7a9daf5d42a84615a6383cfec9a10608 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 7a9daf5d42a84615a6383cfec9a10608, State: Running, Role: LEADER
I20260129 03:00:14.794533 23480 consensus_queue.cc:260] T 2ac659d732314e1288849f08fe5e074e P 7a9daf5d42a84615a6383cfec9a10608 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 5.6, Last appended by leader: 6, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } }
I20260129 03:00:14.795521 23480 raft_consensus.cc:3060] T 2ac659d732314e1288849f08fe5e074e P 7a9daf5d42a84615a6383cfec9a10608 [term 5 FOLLOWER]: Advancing to term 6
I20260129 03:00:14.802179 23419 leader_election.cc:290] T cb0b3f259ebc4310927338a8740e87b8 P 7a9daf5d42a84615a6383cfec9a10608 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 974b802141bd4f13beffef2aa03f88f9 (127.18.211.2:34517), 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3:37735)
W20260129 03:00:14.824805 23574 log.cc:927] Time spent T cedce331e1584ac8b636c9b8a72b2ba0 P 974b802141bd4f13beffef2aa03f88f9: Append to log took a long time: real 0.387s user 0.004s sys 0.000s
W20260129 03:00:14.825045 23578 log.cc:927] Time spent T 36ff33be666a4c82954731691dbcf327 P 197b880fa39d432a9b472845876bfaf0: Append to log took a long time: real 0.361s user 0.001s sys 0.002s
I20260129 03:00:14.871284 20434 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55676 (request call id 1491) took 2133 ms. Trace:
W20260129 03:00:14.872390 20407 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.211.2:34517, user_credentials={real_user=slave}} blocked reactor thread for 190487us
I20260129 03:00:14.872342 20434 rpcz_store.cc:276] 0129 03:00:12.737253 (+ 0us) service_pool.cc:168] Inserting onto call queue
0129 03:00:12.754794 (+ 17541us) service_pool.cc:225] Handling call
0129 03:00:13.264969 (+510175us) raft_consensus.cc:1471] Updating replica for 0 ops
0129 03:00:13.271961 (+ 6992us) raft_consensus.cc:1697] Filling consensus response to leader.
0129 03:00:14.870441 (+1598480us) spinlock_profiling.cc:243] Waited 510 ms on lock 0x7b5400330750. stack: 00007f8fe044e373 00007f8fe044e270 00007f8fdfda0574 00007f8fe9a5c6b4 00007f8fe9a5c649 00007f8fe34975a1 00007f8fe352d52d 00007f8fe84e1f49 00007f8fe1a6ed34 00007f8fe1a6ecc2 00007f8fe1a6ebf1 00007f8fe1a6eb6c 00007f8fe1a6de22 00007f8fe186d34c 00007f8fe186c786 00007f8fe186c12f
0129 03:00:14.870896 (+ 455us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1171875072}
I20260129 03:00:14.876328 20440 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55676 (request call id 1493) took 1759 ms. Trace:
I20260129 03:00:14.876545 20433 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "76f713c3331540d2a5b76164103e9eb1" candidate_uuid: "1c50de89a5a845668696dcb28c3badd6" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "197b880fa39d432a9b472845876bfaf0" is_pre_election: true
I20260129 03:00:14.877005 20440 rpcz_store.cc:276] 0129 03:00:13.117052 (+ 0us) service_pool.cc:168] Inserting onto call queue
0129 03:00:13.230946 (+113894us) service_pool.cc:225] Handling call
0129 03:00:13.231397 (+ 451us) raft_consensus.cc:1471] Updating replica for 1 ops
0129 03:00:13.244688 (+ 13291us) raft_consensus.cc:1534] Early marking committed up to index 0
0129 03:00:13.244727 (+ 39us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0129 03:00:13.250475 (+ 5748us) log.cc:844] Serialized 26 byte log entry
0129 03:00:14.147207 (+896732us) spinlock_profiling.cc:243] Waited 1.52 ms on lock 0x7b18003f1120. stack: 00007f8fe044e373 00007f8fe044e270 00007f8fdfda0574 00007f8fe9a5c6b4 00007f8fe9a5c649 00007f8fe92d4c91 00007f8fe92e1671 00007f8fe0462c67 00007f8fe924b709 00007f8fe047f57b 00007f8fe047d81f 00007f8fe047fae2 00007f8fe30c75ab 00007f8fe30cd0c3 00007f8fe30cd471 00007f8fe34e973f
0129 03:00:14.369888 (+222681us) raft_consensus.cc:1645] Marking committed up to 3
0129 03:00:14.369945 (+ 57us) raft_consensus.cc:1697] Filling consensus response to leader.
0129 03:00:14.370033 (+ 88us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0129 03:00:14.824326 (+454293us) spinlock_profiling.cc:243] Waited 39.8 ms on lock 0x7b440002304c. stack: 00007f8fe044e373 00007f8fe044e270 00007f8fdfda0574 00007f8fe9a5c6b4 00007f8fe9a5c649 00007f8fe02c08b1 00007f8fe038f005 00007f8fe038fa22 00007f8fe30d957b 00007f8fe30d4e4f 00007f8fe30c8c0e 00007f8fe30c7f56 00007f8fe30c7b78 00007f8fe30d15c1 00007f8fe30d1579 00007f8fe30d1509
0129 03:00:14.875711 (+ 51385us) raft_consensus.cc:1685] finished
0129 03:00:14.875804 (+ 93us) raft_consensus.cc:1691] UpdateReplicas() finished
0129 03:00:14.876080 (+ 276us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":12853,"raft.queue_time_us":13382,"raft.run_cpu_time_us":149,"raft.run_wall_time_us":6502,"spinlock_wait_cycles":94977152,"thread_start_us":222442,"threads_started":1,"wal-append.queue_time_us":1206865}
I20260129 03:00:14.878655 20435 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:48356 (request call id 442) took 1495 ms. Trace:
I20260129 03:00:14.880002 20405 leader_election.cc:304] T 1299723a4476420ab02fd8bfd7aa0645 P 197b880fa39d432a9b472845876bfaf0 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 197b880fa39d432a9b472845876bfaf0; no voters: 1c50de89a5a845668696dcb28c3badd6, 7a9daf5d42a84615a6383cfec9a10608
I20260129 03:00:14.881489 19276 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
W20260129 03:00:14.896029 20482 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260129 03:00:14.904488 20480 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260129 03:00:14.940062 23514 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.505s user 0.000s sys 0.006s
I20260129 03:00:14.892095 20435 rpcz_store.cc:276] 0129 03:00:13.382797 (+ 0us) service_pool.cc:168] Inserting onto call queue
0129 03:00:13.385986 (+ 3189us) service_pool.cc:225] Handling call
0129 03:00:13.386360 (+ 374us) raft_consensus.cc:1471] Updating replica for 0 ops
0129 03:00:14.374821 (+988461us) raft_consensus.cc:1697] Filling consensus response to leader.
0129 03:00:14.878057 (+503236us) spinlock_profiling.cc:243] Waited 988 ms on lock 0x7b54002ae754. stack: 00007f8fe044e373 00007f8fe044e270 00007f8fdfda0574 00007f8fe9a5c6b4 00007f8fe9a5c649 00007f8fe34975a1 00007f8fe352ea81 00007f8fe352d4be 00007f8fe84e1f49 00007f8fe1a6ed34 00007f8fe1a6ecc2 00007f8fe1a6ebf1 00007f8fe1a6eb6c 00007f8fe1a6de22 00007f8fe186d34c 00007f8fe186c786
0129 03:00:14.878464 (+ 407us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2271679360}
I20260129 03:00:14.900564 20512 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57600 (request call id 1962) took 2327 ms. Trace:
I20260129 03:00:14.898360 20510 raft_consensus.cc:1240] T 45cc5f3d2688486d9b6dbe7e666dedde P 974b802141bd4f13beffef2aa03f88f9 [term 7 FOLLOWER]: Rejecting Update request from peer 7a9daf5d42a84615a6383cfec9a10608 for earlier term 6. Current term is 7. Ops: [6.6-6.6]
I20260129 03:00:14.905692 20516 raft_consensus.cc:2393] T febffe4a8b394368b69b1d164308fb9a P 974b802141bd4f13beffef2aa03f88f9 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 7a9daf5d42a84615a6383cfec9a10608 in current term 6: Already voted for candidate 974b802141bd4f13beffef2aa03f88f9 in this term.
I20260129 03:00:14.940943 20512 rpcz_store.cc:276] 0129 03:00:12.573022 (+ 0us) service_pool.cc:168] Inserting onto call queue
0129 03:00:12.855141 (+282119us) service_pool.cc:225] Handling call
0129 03:00:12.855460 (+ 319us) raft_consensus.cc:1471] Updating replica for 0 ops
0129 03:00:13.981724 (+1126264us) raft_consensus.cc:1697] Filling consensus response to leader.
0129 03:00:14.899801 (+918077us) spinlock_profiling.cc:243] Waited 1.09 s on lock 0x7b54001ae354. stack: 00007f8fe044e373 00007f8fe044e270 00007f8fdfda0574 00007f8fe9a5c6b4 00007f8fe9a5c649 00007f8fe34975a1 00007f8fe352ea81 00007f8fe352d4be 00007f8fe84e1f49 00007f8fe1a6ed34 00007f8fe1a6ecc2 00007f8fe1a6ebf1 00007f8fe1a6eb6c 00007f8fe1a6de22 00007f8fe186d34c 00007f8fe186c786
0129 03:00:14.900314 (+ 513us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2516795776}
W20260129 03:00:14.903272 20479 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.211.1:42027, user_credentials={real_user=slave}} blocked reactor thread for 385453us
I20260129 03:00:14.898504 20511 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57600 (request call id 1967) took 2158 ms. Trace:
I20260129 03:00:14.951110 23165 consensus_queue.cc:237] T 505580c93c694bbb8ea96fdf9b17ab46 P 7a9daf5d42a84615a6383cfec9a10608 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 6.8, Last appended by leader: 8, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } }
I20260129 03:00:14.952723 21132 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34046 (request call id 591) took 1579 ms. Trace:
I20260129 03:00:14.953269 21132 rpcz_store.cc:276] 0129 03:00:13.373275 (+ 0us) service_pool.cc:168] Inserting onto call queue
0129 03:00:13.373696 (+ 421us) service_pool.cc:225] Handling call
0129 03:00:13.983832 (+610136us) spinlock_profiling.cc:243] Waited 1.18 ms on lock 0x7b5800111f98. stack: 00007f8fe044e373 00007f8fe044e270 00007f8fdfda0574 00007f8fe9a5c6b4 00007f8fe9a5c649 00007f8fe02c08b1 00007f8fe0442db1 00007f8fe0442c62 00007f8fe92eddd9 00007f8fe846d260 00007f8fe8535dbb 00007f8fe8535e41 00007f8fe84f7af0 00007f8fe84e2459 00007f8fe84e1f02 00007f8fe1a6ed34
0129 03:00:14.025486 (+ 41654us) raft_consensus.cc:1471] Updating replica for 1 ops
0129 03:00:14.277127 (+251641us) raft_consensus.cc:1534] Early marking committed up to index 0
0129 03:00:14.277166 (+ 39us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0129 03:00:14.288216 (+ 11050us) log.cc:844] Serialized 26 byte log entry
0129 03:00:14.503783 (+215567us) raft_consensus.cc:1645] Marking committed up to 7
0129 03:00:14.553367 (+ 49584us) log.cc:844] Serialized 14 byte log entry
0129 03:00:14.618095 (+ 64728us) raft_consensus.cc:1697] Filling consensus response to leader.
0129 03:00:14.618790 (+ 695us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0129 03:00:14.951991 (+333201us) raft_consensus.cc:1685] finished
0129 03:00:14.952196 (+ 205us) raft_consensus.cc:1691] UpdateReplicas() finished
0129 03:00:14.952459 (+ 263us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":15552,"prepare.queue_time_us":177908,"prepare.run_cpu_time_us":40,"prepare.run_wall_time_us":42,"raft.queue_time_us":159834,"raft.run_cpu_time_us":238,"raft.run_wall_time_us":3810,"spinlock_wait_cycles":2712960,"thread_start_us":459451,"threads_started":3,"wal-append.queue_time_us":367704}
I20260129 03:00:14.956553 21130 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0678b8b468bd43cdbc3e485900844821" candidate_uuid: "974b802141bd4f13beffef2aa03f88f9" candidate_term: 9 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "7a9daf5d42a84615a6383cfec9a10608"
I20260129 03:00:14.957906 21130 raft_consensus.cc:2393] T 0678b8b468bd43cdbc3e485900844821 P 7a9daf5d42a84615a6383cfec9a10608 [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate 974b802141bd4f13beffef2aa03f88f9 in current term 9: Already voted for candidate 7a9daf5d42a84615a6383cfec9a10608 in this term.
I20260129 03:00:14.959288 21130 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c934d31efd3f4e5195605ddc354d14e1" candidate_uuid: "974b802141bd4f13beffef2aa03f88f9" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "7a9daf5d42a84615a6383cfec9a10608" is_pre_election: true
I20260129 03:00:14.960448 23475 consensus_queue.cc:1059] T 17d6431cc77a491e8464d68db4081a51 P 197b880fa39d432a9b472845876bfaf0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 }, Status: INVALID_TERM, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260129 03:00:14.962044 23475 raft_consensus.cc:493] T a8479d1fe0bd41e5946186e3a6ca8c7a P 197b880fa39d432a9b472845876bfaf0 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 974b802141bd4f13beffef2aa03f88f9)
I20260129 03:00:14.968108 22645 raft_consensus.cc:3055] T 17d6431cc77a491e8464d68db4081a51 P 197b880fa39d432a9b472845876bfaf0 [term 5 LEADER]: Stepping down as leader of term 5
I20260129 03:00:14.968786 22645 raft_consensus.cc:740] T 17d6431cc77a491e8464d68db4081a51 P 197b880fa39d432a9b472845876bfaf0 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 197b880fa39d432a9b472845876bfaf0, State: Running, Role: LEADER
W20260129 03:00:14.969879 20405 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.211.4:40439, user_credentials={real_user=slave}} blocked reactor thread for 90344.6us
I20260129 03:00:14.971678 23466 raft_consensus.cc:2749] T 1299723a4476420ab02fd8bfd7aa0645 P 197b880fa39d432a9b472845876bfaf0 [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
W20260129 03:00:14.974526 21100 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260129 03:00:14.974797 23555 raft_consensus.cc:493] T 7d9494f5584d4d5db8b9adf1c8fc888b P 974b802141bd4f13beffef2aa03f88f9 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 1c50de89a5a845668696dcb28c3badd6)
I20260129 03:00:14.976050 23555 raft_consensus.cc:515] T 7d9494f5584d4d5db8b9adf1c8fc888b P 974b802141bd4f13beffef2aa03f88f9 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 03:00:14.962966 23475 raft_consensus.cc:515] T a8479d1fe0bd41e5946186e3a6ca8c7a P 197b880fa39d432a9b472845876bfaf0 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 03:00:14.986857 21127 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0678b8b468bd43cdbc3e485900844821" candidate_uuid: "974b802141bd4f13beffef2aa03f88f9" candidate_term: 9 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "7a9daf5d42a84615a6383cfec9a10608"
I20260129 03:00:14.988005 20587 raft_consensus.cc:1275] T e7d83d4ccf684e76acb29a0dc844e1b8 P 1c50de89a5a845668696dcb28c3badd6 [term 7 FOLLOWER]: Refusing update from remote peer 974b802141bd4f13beffef2aa03f88f9: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 5 index: 3. (term mismatch)
I20260129 03:00:14.988480 20587 pending_rounds.cc:85] T e7d83d4ccf684e76acb29a0dc844e1b8 P 1c50de89a5a845668696dcb28c3badd6: Aborting all ops after (but not including) 2
I20260129 03:00:14.988842 20587 pending_rounds.cc:107] T e7d83d4ccf684e76acb29a0dc844e1b8 P 1c50de89a5a845668696dcb28c3badd6: Aborting uncommitted NO_OP operation due to leader change: 4.3
I20260129 03:00:14.989153 20587 raft_consensus.cc:2889] T e7d83d4ccf684e76acb29a0dc844e1b8 P 1c50de89a5a845668696dcb28c3badd6 [term 7 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
W20260129 03:00:14.989187 20553 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260129 03:00:14.991498 20553 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260129 03:00:15.005662 20554 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.211.4:40439, user_credentials={real_user=slave}} blocked reactor thread for 272961us
W20260129 03:00:15.008083 20582 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:35754 (request call id 1855) took 2858 ms (client timeout 2084 ms). Trace:
W20260129 03:00:15.008760 20582 rpcz_store.cc:269] 0129 03:00:12.057565 (+ 0us) service_pool.cc:168] Inserting onto call queue
0129 03:00:12.691483 (+633918us) service_pool.cc:225] Handling call
0129 03:00:14.915786 (+2224303us) spinlock_profiling.cc:243] Waited 213 ms on lock 0x7b54000c9c54. stack: 00007f8fe044e373 00007f8fe044e270 00007f8fdfda0574 00007f8fe9a5c6b4 00007f8fe9a5c649 00007f8fe34975a1 00007f8fe3531f7a 00007f8fe84e3647 00007f8fe1a73834 00007f8fe1a737c2 00007f8fe1a736f1 00007f8fe1a7366c 00007f8fe1a72922 00007f8fe186d34c 00007f8fe186c786 00007f8fe186c12f
0129 03:00:14.916106 (+ 320us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":489111424}
I20260129 03:00:15.009613 23564 raft_consensus.cc:493] T c2633a6fc34c44579de348d208b0c01e P 1c50de89a5a845668696dcb28c3badd6 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260129 03:00:15.010839 20582 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "cb0b3f259ebc4310927338a8740e87b8" candidate_uuid: "7a9daf5d42a84615a6383cfec9a10608" candidate_term: 9 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "1c50de89a5a845668696dcb28c3badd6" is_pre_election: true
I20260129 03:00:15.010537 23564 raft_consensus.cc:515] T c2633a6fc34c44579de348d208b0c01e P 1c50de89a5a845668696dcb28c3badd6 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 03:00:15.014036 21129 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "cb0b3f259ebc4310927338a8740e87b8" candidate_uuid: "974b802141bd4f13beffef2aa03f88f9" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "7a9daf5d42a84615a6383cfec9a10608" is_pre_election: true
I20260129 03:00:15.015398 21129 raft_consensus.cc:2393] T cb0b3f259ebc4310927338a8740e87b8 P 7a9daf5d42a84615a6383cfec9a10608 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 974b802141bd4f13beffef2aa03f88f9 in current term 8: Already voted for candidate 1c50de89a5a845668696dcb28c3badd6 in this term.
W20260129 03:00:15.019129 20556 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.211.2:34517, user_credentials={real_user=slave}} blocked reactor thread for 287723us
W20260129 03:00:15.020820 20405 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.211.4:40439, user_credentials={real_user=slave}} blocked reactor thread for 49255.2us
I20260129 03:00:15.022235 21131 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34060 (request call id 547) took 1487 ms. Trace:
I20260129 03:00:15.022696 21131 rpcz_store.cc:276] 0129 03:00:13.534675 (+ 0us) service_pool.cc:168] Inserting onto call queue
0129 03:00:13.829261 (+294586us) service_pool.cc:225] Handling call
0129 03:00:15.021771 (+1192510us) spinlock_profiling.cc:243] Waited 79.7 ms on lock 0x7b54005631d4. stack: 00007f8fe044e373 00007f8fe044e270 00007f8fdfda0574 00007f8fe9a5c6b4 00007f8fe9a5c649 00007f8fe34975a1 00007f8fe3531f7a 00007f8fe84e3647 00007f8fe1a73834 00007f8fe1a737c2 00007f8fe1a736f1 00007f8fe1a7366c 00007f8fe1a72922 00007f8fe186d34c 00007f8fe186c786 00007f8fe186c12f
0129 03:00:15.022034 (+ 263us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":183362944}
I20260129 03:00:15.024257 23485 raft_consensus.cc:493] T fd8311b779af4e1fa1402b2a02a602f2 P 197b880fa39d432a9b472845876bfaf0 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 1c50de89a5a845668696dcb28c3badd6)
I20260129 03:00:14.995427 21099 leader_election.cc:304] T fd8311b779af4e1fa1402b2a02a602f2 P 7a9daf5d42a84615a6383cfec9a10608 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7a9daf5d42a84615a6383cfec9a10608; no voters: 197b880fa39d432a9b472845876bfaf0, 1c50de89a5a845668696dcb28c3badd6
I20260129 03:00:15.025072 23485 raft_consensus.cc:515] T fd8311b779af4e1fa1402b2a02a602f2 P 197b880fa39d432a9b472845876bfaf0 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } }
I20260129 03:00:15.029552 22891 raft_consensus.cc:493] T 724a236524cb4e7ba3428880681d6cc2 P 1c50de89a5a845668696dcb28c3badd6 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260129 03:00:15.030493 22891 raft_consensus.cc:515] T 724a236524cb4e7ba3428880681d6cc2 P 1c50de89a5a845668696dcb28c3badd6 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 03:00:15.032886 21133 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4e3805ffa70242e8ade66da260dddf66" candidate_uuid: "1c50de89a5a845668696dcb28c3badd6" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "7a9daf5d42a84615a6383cfec9a10608" is_pre_election: true
W20260129 03:00:15.034068 21133 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34046 (request call id 593) took 1660 ms (client timeout 1602 ms). Trace:
W20260129 03:00:15.034499 21133 rpcz_store.cc:269] 0129 03:00:13.373829 (+ 0us) service_pool.cc:168] Inserting onto call queue
0129 03:00:13.373938 (+ 109us) service_pool.cc:225] Handling call
0129 03:00:15.032436 (+1658498us) spinlock_profiling.cc:243] Waited 2.11 ms on lock 0x7b5800111f98. stack: 00007f8fe044e373 00007f8fe044e270 00007f8fdfda0574 00007f8fe9a5c6b4 00007f8fe9a5c649 00007f8fe02c08b1 00007f8fe0442db1 00007f8fe0442c62 00007f8fe92eddd9 00007f8fe846d260 00007f8fe8535dbb 00007f8fe8535e41 00007f8fe84e3bd0 00007f8fe84e34b4 00007f8fe1a73834 00007f8fe1a737c2
0129 03:00:15.033966 (+ 1530us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":4851456}
I20260129 03:00:15.036262 23469 raft_consensus.cc:493] T 4e420f90884f4796997b5ea41e49e411 P 197b880fa39d432a9b472845876bfaf0 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 7a9daf5d42a84615a6383cfec9a10608)
I20260129 03:00:15.037096 23469 raft_consensus.cc:515] T 4e420f90884f4796997b5ea41e49e411 P 197b880fa39d432a9b472845876bfaf0 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } }
I20260129 03:00:15.037567 23475 leader_election.cc:290] T a8479d1fe0bd41e5946186e3a6ca8c7a P 197b880fa39d432a9b472845876bfaf0 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 974b802141bd4f13beffef2aa03f88f9 (127.18.211.2:34517), 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3:37735)
I20260129 03:00:14.991668 20587 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35752 (request call id 1734) took 1923 ms. Trace:
I20260129 03:00:15.040428 23262 raft_consensus.cc:2749] T 3e2d15f8640b407195210c476e09df91 P 1c50de89a5a845668696dcb28c3badd6 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260129 03:00:15.040551 23290 consensus_queue.cc:1048] T 78aa39096868415d9afc2ffcb248e198 P 1c50de89a5a845668696dcb28c3badd6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.001s
I20260129 03:00:15.041692 23262 raft_consensus.cc:2749] T 926a250fa5834e88b96ccecbefbf5086 P 1c50de89a5a845668696dcb28c3badd6 [term 8 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
W20260129 03:00:15.048413 20479 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.211.1:42027, user_credentials={real_user=slave}} blocked reactor thread for 105474us
I20260129 03:00:15.071684 21136 raft_consensus.cc:1275] T 87d5fe48a44a408aaf03e894a67ecb2f P 7a9daf5d42a84615a6383cfec9a10608 [term 7 FOLLOWER]: Refusing update from remote peer 197b880fa39d432a9b472845876bfaf0: Log matching property violated. Preceding OpId in replica: term: 4 index: 6. Preceding OpId from leader: term: 7 index: 8. (index mismatch)
I20260129 03:00:15.073783 21136 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34060 (request call id 553) took 1246 ms. Trace:
I20260129 03:00:15.074260 21136 rpcz_store.cc:276] 0129 03:00:13.827753 (+ 0us) service_pool.cc:168] Inserting onto call queue
0129 03:00:14.088602 (+260849us) service_pool.cc:225] Handling call
0129 03:00:15.070853 (+982251us) raft_consensus.cc:1471] Updating replica for 0 ops
0129 03:00:15.072310 (+ 1457us) raft_consensus.cc:1697] Filling consensus response to leader.
0129 03:00:15.073476 (+ 1166us) spinlock_profiling.cc:243] Waited 957 ms on lock 0x7b54005631d0. stack: 00007f8fe044e373 00007f8fe044e270 00007f8fdfda0574 00007f8fe9a5c6b4 00007f8fe9a5c649 00007f8fe34975a1 00007f8fe352d52d 00007f8fe84e1f49 00007f8fe1a6ed34 00007f8fe1a6ecc2 00007f8fe1a6ebf1 00007f8fe1a6eb6c 00007f8fe1a6de22 00007f8fe186d34c 00007f8fe186c786 00007f8fe186c12f
0129 03:00:15.073694 (+ 218us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2202136448}
I20260129 03:00:15.076828 23403 raft_consensus.cc:493] T 3ccac82a01a2445fa5b0ce6c0fcf2295 P 197b880fa39d432a9b472845876bfaf0 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 7a9daf5d42a84615a6383cfec9a10608)
I20260129 03:00:15.039709 20587 rpcz_store.cc:276] 0129 03:00:13.068102 (+ 0us) service_pool.cc:168] Inserting onto call queue
0129 03:00:14.722478 (+1654376us) service_pool.cc:225] Handling call
0129 03:00:14.738869 (+ 16391us) raft_consensus.cc:1471] Updating replica for 0 ops
0129 03:00:14.990498 (+251629us) raft_consensus.cc:1697] Filling consensus response to leader.
0129 03:00:14.991295 (+ 797us) spinlock_profiling.cc:243] Waited 177 ms on lock 0x7b540027fbd4. stack: 00007f8fe044e373 00007f8fe044e270 00007f8fdfda0574 00007f8fe9a5c6b4 00007f8fe9a5c649 00007f8fe34975a1 00007f8fe352ea81 00007f8fe352d4be 00007f8fe84e1f49 00007f8fe1a6ed34 00007f8fe1a6ecc2 00007f8fe1a6ebf1 00007f8fe1a6eb6c 00007f8fe1a6de22 00007f8fe186d34c 00007f8fe186c786
0129 03:00:14.991549 (+ 254us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":406081920}
I20260129 03:00:15.011783 20582 raft_consensus.cc:2468] T cb0b3f259ebc4310927338a8740e87b8 P 1c50de89a5a845668696dcb28c3badd6 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7a9daf5d42a84615a6383cfec9a10608 in term 8.
I20260129 03:00:15.039682 23469 leader_election.cc:290] T 4e420f90884f4796997b5ea41e49e411 P 197b880fa39d432a9b472845876bfaf0 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 974b802141bd4f13beffef2aa03f88f9 (127.18.211.2:34517), 7a9daf5d42a84615a6383cfec9a10608 (127.18.211.4:40439)
I20260129 03:00:14.949082 23224 raft_consensus.cc:493] T 36ff33be666a4c82954731691dbcf327 P 197b880fa39d432a9b472845876bfaf0 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 1c50de89a5a845668696dcb28c3badd6)
I20260129 03:00:15.086969 21127 raft_consensus.cc:2393] T 0678b8b468bd43cdbc3e485900844821 P 7a9daf5d42a84615a6383cfec9a10608 [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate 974b802141bd4f13beffef2aa03f88f9 in current term 9: Already voted for candidate 7a9daf5d42a84615a6383cfec9a10608 in this term.
I20260129 03:00:15.095419 20508 raft_consensus.cc:1275] T 2ac659d732314e1288849f08fe5e074e P 974b802141bd4f13beffef2aa03f88f9 [term 6 FOLLOWER]: Refusing update from remote peer 197b880fa39d432a9b472845876bfaf0: Log matching property violated. Preceding OpId in replica: term: 3 index: 5. Preceding OpId from leader: term: 6 index: 7. (index mismatch)
I20260129 03:00:15.097057 21133 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4e420f90884f4796997b5ea41e49e411" candidate_uuid: "974b802141bd4f13beffef2aa03f88f9" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "7a9daf5d42a84615a6383cfec9a10608" is_pre_election: true
W20260129 03:00:15.099155 20482 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260129 03:00:14.970697 22645 consensus_queue.cc:260] T 17d6431cc77a491e8464d68db4081a51 P 197b880fa39d432a9b472845876bfaf0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 5.3, Last appended by leader: 3, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 03:00:15.106756 21129 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "45cc5f3d2688486d9b6dbe7e666dedde" candidate_uuid: "974b802141bd4f13beffef2aa03f88f9" candidate_term: 7 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "7a9daf5d42a84615a6383cfec9a10608" is_pre_election: true
I20260129 03:00:15.108587 21129 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "add8d4923d1a489e86882c3a25692b95" candidate_uuid: "974b802141bd4f13beffef2aa03f88f9" candidate_term: 8 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "7a9daf5d42a84615a6383cfec9a10608" is_pre_election: true
I20260129 03:00:15.110530 21129 raft_consensus.cc:1275] T 0678b8b468bd43cdbc3e485900844821 P 7a9daf5d42a84615a6383cfec9a10608 [term 9 FOLLOWER]: Refusing update from remote peer 974b802141bd4f13beffef2aa03f88f9: Log matching property violated. Preceding OpId in replica: term: 7 index: 8. Preceding OpId from leader: term: 9 index: 9. (index mismatch)
I20260129 03:00:15.113727 20556 leader_election.cc:304] T a1a39dcd328e4268ae41b988c66b4a75 P 1c50de89a5a845668696dcb28c3badd6 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1c50de89a5a845668696dcb28c3badd6; no voters: 197b880fa39d432a9b472845876bfaf0, 974b802141bd4f13beffef2aa03f88f9
I20260129 03:00:15.115371 22996 raft_consensus.cc:2749] T a1a39dcd328e4268ae41b988c66b4a75 P 1c50de89a5a845668696dcb28c3badd6 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260129 03:00:15.116305 23511 raft_consensus.cc:2749] T fd8311b779af4e1fa1402b2a02a602f2 P 7a9daf5d42a84615a6383cfec9a10608 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260129 03:00:15.124722 20510 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:48544 (request call id 340) took 1165 ms. Trace:
I20260129 03:00:15.124480 21128 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2ac659d732314e1288849f08fe5e074e" candidate_uuid: "197b880fa39d432a9b472845876bfaf0" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "7a9daf5d42a84615a6383cfec9a10608" is_pre_election: true
I20260129 03:00:15.125170 20510 rpcz_store.cc:276] 0129 03:00:13.959021 (+ 0us) service_pool.cc:168] Inserting onto call queue
0129 03:00:14.050096 (+ 91075us) service_pool.cc:225] Handling call
0129 03:00:14.050457 (+ 361us) raft_consensus.cc:1471] Updating replica for 1 ops
0129 03:00:14.941414 (+890957us) raft_consensus.cc:1697] Filling consensus response to leader.
0129 03:00:15.124417 (+183003us) spinlock_profiling.cc:243] Waited 847 ms on lock 0x7b5400314f54. stack: 00007f8fe044e373 00007f8fe044e270 00007f8fdfda0574 00007f8fe9a5c6b4 00007f8fe9a5c649 00007f8fe34975a1 00007f8fe352ea81 00007f8fe352d4be 00007f8fe84e1f49 00007f8fe1a6ed34 00007f8fe1a6ecc2 00007f8fe1a6ebf1 00007f8fe1a6eb6c 00007f8fe1a6de22 00007f8fe186d34c 00007f8fe186c786
0129 03:00:15.124651 (+ 234us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1948133248}
I20260129 03:00:15.125712 21128 raft_consensus.cc:2468] T 2ac659d732314e1288849f08fe5e074e P 7a9daf5d42a84615a6383cfec9a10608 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 197b880fa39d432a9b472845876bfaf0 in term 6.
I20260129 03:00:15.126574 21128 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34060 (request call id 552) took 1302 ms. Trace:
I20260129 03:00:15.127012 21128 rpcz_store.cc:276] 0129 03:00:13.824303 (+ 0us) service_pool.cc:168] Inserting onto call queue
0129 03:00:13.853918 (+ 29615us) service_pool.cc:225] Handling call
0129 03:00:14.995657 (+1141739us) spinlock_profiling.cc:243] Waited 4.24 ms on lock 0x7b5800111f98. stack: 00007f8fe044e373 00007f8fe044e270 00007f8fdfda0574 00007f8fe9a5c6b4 00007f8fe9a5c649 00007f8fe02c08b1 00007f8fe0442db1 00007f8fe044312e 000000000035a3c9 00007f8fe846d560 00007f8fe8535dd4 00007f8fe8535e41 00007f8fe84e3bd0 00007f8fe84e34b4 00007f8fe1a73834 00007f8fe1a737c2
0129 03:00:15.126497 (+130840us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":9753984}
W20260129 03:00:15.128957 21099 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.211.1:42027, user_credentials={real_user=slave}} blocked reactor thread for 134024us
I20260129 03:00:15.130493 21099 leader_election.cc:304] T febffe4a8b394368b69b1d164308fb9a P 7a9daf5d42a84615a6383cfec9a10608 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 197b880fa39d432a9b472845876bfaf0, 7a9daf5d42a84615a6383cfec9a10608; no voters:
W20260129 03:00:15.133826 20589 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:35752 (request call id 1725) took 3158 ms (client timeout 1539 ms). Trace:
W20260129 03:00:15.134317 20589 rpcz_store.cc:269] 0129 03:00:11.975203 (+ 0us) service_pool.cc:168] Inserting onto call queue
0129 03:00:12.012903 (+ 37700us) service_pool.cc:225] Handling call
0129 03:00:15.133364 (+3120461us) spinlock_profiling.cc:243] Waited 1.29 s on lock 0x7b540018b5d4. stack: 00007f8fe044e373 00007f8fe044e270 00007f8fdfda0574 00007f8fe9a5c6b4 00007f8fe9a5c649 00007f8fe34975a1 00007f8fe3531f7a 00007f8fe84e3647 00007f8fe1a73834 00007f8fe1a737c2 00007f8fe1a736f1 00007f8fe1a7366c 00007f8fe1a72922 00007f8fe186d34c 00007f8fe186c786 00007f8fe186c12f
0129 03:00:15.133579 (+ 215us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2959975680}
I20260129 03:00:15.136260 23224 raft_consensus.cc:515] T 36ff33be666a4c82954731691dbcf327 P 197b880fa39d432a9b472845876bfaf0 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 03:00:15.138257 21131 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "96609ac68bdf4993a12e1c23355ca013" candidate_uuid: "1c50de89a5a845668696dcb28c3badd6" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "7a9daf5d42a84615a6383cfec9a10608" is_pre_election: true
I20260129 03:00:15.140026 23480 raft_consensus.cc:2804] T febffe4a8b394368b69b1d164308fb9a P 7a9daf5d42a84615a6383cfec9a10608 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260129 03:00:15.140569 23480 raft_consensus.cc:493] T febffe4a8b394368b69b1d164308fb9a P 7a9daf5d42a84615a6383cfec9a10608 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260129 03:00:15.140923 23480 raft_consensus.cc:3060] T febffe4a8b394368b69b1d164308fb9a P 7a9daf5d42a84615a6383cfec9a10608 [term 5 FOLLOWER]: Advancing to term 6
I20260129 03:00:15.142769 20586 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:35752 (request call id 1737) took 2007 ms. Trace:
I20260129 03:00:15.143293 20586 rpcz_store.cc:276] 0129 03:00:13.071313 (+ 0us) service_pool.cc:168] Inserting onto call queue
0129 03:00:13.887712 (+816399us) service_pool.cc:225] Handling call
0129 03:00:15.063575 (+1175863us) spinlock_profiling.cc:243] Waited 571 ms on lock 0x7b540009bad4. stack: 00007f8fe044e373 00007f8fe044e270 00007f8fdfda0574 00007f8fe9a5c6b4 00007f8fe9a5c649 00007f8fe34975a1 00007f8fe3531f7a 00007f8fe84e3647 00007f8fe1a73834 00007f8fe1a737c2 00007f8fe1a736f1 00007f8fe1a7366c 00007f8fe1a72922 00007f8fe186d34c 00007f8fe186c786 00007f8fe186c12f
0129 03:00:15.078894 (+ 15319us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1312681728}
I20260129 03:00:15.151901 21127 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "45cc5f3d2688486d9b6dbe7e666dedde" candidate_uuid: "974b802141bd4f13beffef2aa03f88f9" candidate_term: 7 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "7a9daf5d42a84615a6383cfec9a10608"
I20260129 03:00:15.081666 23403 raft_consensus.cc:515] T 3ccac82a01a2445fa5b0ce6c0fcf2295 P 197b880fa39d432a9b472845876bfaf0 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } }
I20260129 03:00:15.135643 20583 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35752 (request call id 1743) took 2062 ms. Trace:
I20260129 03:00:15.188738 20583 rpcz_store.cc:276] 0129 03:00:13.073442 (+ 0us) service_pool.cc:168] Inserting onto call queue
0129 03:00:14.734146 (+1660704us) service_pool.cc:225] Handling call
0129 03:00:15.133627 (+399481us) raft_consensus.cc:1471] Updating replica for 0 ops
0129 03:00:15.134287 (+ 660us) raft_consensus.cc:1534] Early marking committed up to index 0
0129 03:00:15.134327 (+ 40us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0129 03:00:15.134396 (+ 69us) raft_consensus.cc:1645] Marking committed up to 4
0129 03:00:15.134438 (+ 42us) raft_consensus.cc:1697] Filling consensus response to leader.
0129 03:00:15.134529 (+ 91us) raft_consensus.cc:1691] UpdateReplicas() finished
0129 03:00:15.135271 (+ 742us) spinlock_profiling.cc:243] Waited 399 ms on lock 0x7b540018b5d0. stack: 00007f8fe044e373 00007f8fe044e270 00007f8fdfda0574 00007f8fe9a5c6b4 00007f8fe9a5c649 00007f8fe34975a1 00007f8fe352d52d 00007f8fe84e1f49 00007f8fe1a6ed34 00007f8fe1a6ecc2 00007f8fe1a6ebf1 00007f8fe1a6eb6c 00007f8fe1a6de22 00007f8fe186d34c 00007f8fe186c786 00007f8fe186c12f
0129 03:00:15.135419 (+ 148us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":16255,"raft.run_cpu_time_us":170,"raft.run_wall_time_us":172,"spinlock_wait_cycles":917767424}
I20260129 03:00:15.190304 23403 leader_election.cc:290] T 3ccac82a01a2445fa5b0ce6c0fcf2295 P 197b880fa39d432a9b472845876bfaf0 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 974b802141bd4f13beffef2aa03f88f9 (127.18.211.2:34517), 7a9daf5d42a84615a6383cfec9a10608 (127.18.211.4:40439)
I20260129 03:00:15.162593 21100 leader_election.cc:304] T cb0b3f259ebc4310927338a8740e87b8 P 7a9daf5d42a84615a6383cfec9a10608 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1c50de89a5a845668696dcb28c3badd6, 7a9daf5d42a84615a6383cfec9a10608; no voters:
I20260129 03:00:15.163695 20553 leader_election.cc:304] T 2e58e9a5a8aa4cecafb464addb4f35ed P 1c50de89a5a845668696dcb28c3badd6 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 197b880fa39d432a9b472845876bfaf0, 1c50de89a5a845668696dcb28c3badd6; no voters:
W20260129 03:00:15.191661 21100 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.211.3:37735, user_credentials={real_user=slave}} blocked reactor thread for 86092.4us
I20260129 03:00:15.192627 22996 raft_consensus.cc:2804] T 2e58e9a5a8aa4cecafb464addb4f35ed P 1c50de89a5a845668696dcb28c3badd6 [term 5 FOLLOWER]: Leader election won for term 5
I20260129 03:00:15.167554 23466 raft_consensus.cc:493] T cb4727386a3e48aca5d71cc6c611c197 P 197b880fa39d432a9b472845876bfaf0 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 1c50de89a5a845668696dcb28c3badd6)
I20260129 03:00:15.193354 22996 raft_consensus.cc:697] T 2e58e9a5a8aa4cecafb464addb4f35ed P 1c50de89a5a845668696dcb28c3badd6 [term 5 LEADER]: Becoming Leader. State: Replica: 1c50de89a5a845668696dcb28c3badd6, State: Running, Role: LEADER
I20260129 03:00:15.193811 23466 raft_consensus.cc:515] T cb4727386a3e48aca5d71cc6c611c197 P 197b880fa39d432a9b472845876bfaf0 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 03:00:15.194669 22996 consensus_queue.cc:237] T 2e58e9a5a8aa4cecafb464addb4f35ed P 1c50de89a5a845668696dcb28c3badd6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } }
I20260129 03:00:15.195676 23568 raft_consensus.cc:2804] T cb0b3f259ebc4310927338a8740e87b8 P 7a9daf5d42a84615a6383cfec9a10608 [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260129 03:00:15.196275 23568 raft_consensus.cc:493] T cb0b3f259ebc4310927338a8740e87b8 P 7a9daf5d42a84615a6383cfec9a10608 [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260129 03:00:15.196674 23568 raft_consensus.cc:3060] T cb0b3f259ebc4310927338a8740e87b8 P 7a9daf5d42a84615a6383cfec9a10608 [term 8 FOLLOWER]: Advancing to term 9
W20260129 03:00:15.171361 20554 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.211.4:40439, user_credentials={real_user=slave}} blocked reactor thread for 163820us
W20260129 03:00:15.172261 20479 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.211.1:42027, user_credentials={real_user=slave}} blocked reactor thread for 66234us
I20260129 03:00:14.943181 20511 rpcz_store.cc:276] 0129 03:00:12.739663 (+ 0us) service_pool.cc:168] Inserting onto call queue
0129 03:00:12.841459 (+101796us) service_pool.cc:225] Handling call
0129 03:00:12.841799 (+ 340us) raft_consensus.cc:1471] Updating replica for 1 ops
0129 03:00:13.237868 (+396069us) raft_consensus.cc:1534] Early marking committed up to index 4
0129 03:00:13.964735 (+726867us) log.cc:844] Serialized 14 byte log entry
0129 03:00:14.288267 (+323532us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0129 03:00:14.295042 (+ 6775us) log.cc:844] Serialized 26 byte log entry
0129 03:00:14.295224 (+ 182us) raft_consensus.cc:1645] Marking committed up to 4
0129 03:00:14.295274 (+ 50us) raft_consensus.cc:1697] Filling consensus response to leader.
0129 03:00:14.295368 (+ 94us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0129 03:00:14.824026 (+528658us) spinlock_profiling.cc:243] Waited 64.8 ms on lock 0x7b440002304c. stack: 00007f8fe044e373 00007f8fe044e270 00007f8fdfda0574 00007f8fe9a5c6b4 00007f8fe9a5c649 00007f8fe02c08b1 00007f8fe038f005 00007f8fe038fa22 00007f8fe30d957b 00007f8fe30d4e4f 00007f8fe30c8c0e 00007f8fe30c7f56 00007f8fe30c7b78 00007f8fe30d15c1 00007f8fe30d1579 00007f8fe30d1509
0129 03:00:14.897744 (+ 73718us) raft_consensus.cc:1685] finished
0129 03:00:14.897932 (+ 188us) raft_consensus.cc:1691] UpdateReplicas() finished
0129 03:00:14.898255 (+ 323us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":21478,"prepare.queue_time_us":684076,"prepare.run_cpu_time_us":41,"prepare.run_wall_time_us":41,"raft.queue_time_us":43754,"raft.run_cpu_time_us":342,"raft.run_wall_time_us":19157,"spinlock_wait_cycles":148985728,"thread_start_us":1326744,"threads_started":3,"wal-append.queue_time_us":443250}
I20260129 03:00:15.202263 21128 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "febffe4a8b394368b69b1d164308fb9a" candidate_uuid: "974b802141bd4f13beffef2aa03f88f9" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "7a9daf5d42a84615a6383cfec9a10608"
I20260129 03:00:15.173851 23224 leader_election.cc:290] T 36ff33be666a4c82954731691dbcf327 P 197b880fa39d432a9b472845876bfaf0 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 974b802141bd4f13beffef2aa03f88f9 (127.18.211.2:34517), 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3:37735)
I20260129 03:00:15.174794 20433 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c934d31efd3f4e5195605ddc354d14e1" candidate_uuid: "7a9daf5d42a84615a6383cfec9a10608" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "197b880fa39d432a9b472845876bfaf0"
I20260129 03:00:15.175984 23564 leader_election.cc:290] T c2633a6fc34c44579de348d208b0c01e P 1c50de89a5a845668696dcb28c3badd6 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 197b880fa39d432a9b472845876bfaf0 (127.18.211.1:42027), 974b802141bd4f13beffef2aa03f88f9 (127.18.211.2:34517)
I20260129 03:00:15.204489 21127 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "75b0b6e82ddc4f92b1fe1d94f9a0b73b" candidate_uuid: "974b802141bd4f13beffef2aa03f88f9" candidate_term: 5 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "7a9daf5d42a84615a6383cfec9a10608" is_pre_election: true
I20260129 03:00:15.205559 21099 leader_election.cc:304] T 0678b8b468bd43cdbc3e485900844821 P 7a9daf5d42a84615a6383cfec9a10608 [CANDIDATE]: Term 9 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7a9daf5d42a84615a6383cfec9a10608; no voters: 197b880fa39d432a9b472845876bfaf0, 974b802141bd4f13beffef2aa03f88f9
I20260129 03:00:15.176846 20407 leader_election.cc:304] T e33a417cf9c54cf5af73a4ee3aac4d3a P 197b880fa39d432a9b472845876bfaf0 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 197b880fa39d432a9b472845876bfaf0, 974b802141bd4f13beffef2aa03f88f9; no voters:
I20260129 03:00:15.177922 23485 leader_election.cc:290] T fd8311b779af4e1fa1402b2a02a602f2 P 197b880fa39d432a9b472845876bfaf0 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3:37735), 7a9daf5d42a84615a6383cfec9a10608 (127.18.211.4:40439)
I20260129 03:00:15.180914 23511 consensus_queue.cc:1059] T 45cc5f3d2688486d9b6dbe7e666dedde P 7a9daf5d42a84615a6383cfec9a10608 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 }, Status: INVALID_TERM, Last received: 3.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
I20260129 03:00:15.181473 20512 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "cb0b3f259ebc4310927338a8740e87b8" candidate_uuid: "7a9daf5d42a84615a6383cfec9a10608" candidate_term: 9 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "974b802141bd4f13beffef2aa03f88f9" is_pre_election: true
I20260129 03:00:15.183653 22645 raft_consensus.cc:3060] T 17d6431cc77a491e8464d68db4081a51 P 197b880fa39d432a9b472845876bfaf0 [term 5 FOLLOWER]: Advancing to term 6
I20260129 03:00:15.102108 23555 leader_election.cc:290] T 7d9494f5584d4d5db8b9adf1c8fc888b P 974b802141bd4f13beffef2aa03f88f9 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 197b880fa39d432a9b472845876bfaf0 (127.18.211.1:42027), 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3:37735)
I20260129 03:00:15.184417 20510 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c934d31efd3f4e5195605ddc354d14e1" candidate_uuid: "7a9daf5d42a84615a6383cfec9a10608" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "974b802141bd4f13beffef2aa03f88f9"
I20260129 03:00:15.186801 20586 tablet_service.cc:2038] Received Run Leader Election RPC: tablet_id: "4f8f3a38f6e24ba4857875181f447442"
dest_uuid: "1c50de89a5a845668696dcb28c3badd6"
from {username='slave'} at 127.0.0.1:35752
I20260129 03:00:15.187227 21135 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4f5a77e94fef4b16a2ba7b067c8b144a" candidate_uuid: "1c50de89a5a845668696dcb28c3badd6" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "7a9daf5d42a84615a6383cfec9a10608" is_pre_election: true
I20260129 03:00:15.139943 21131 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3e2d15f8640b407195210c476e09df91" candidate_uuid: "974b802141bd4f13beffef2aa03f88f9" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "7a9daf5d42a84615a6383cfec9a10608" is_pre_election: true
W20260129 03:00:15.188035 20556 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.211.2:34517, user_credentials={real_user=slave}} blocked reactor thread for 74612.3us
I20260129 03:00:15.213222 21132 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "505580c93c694bbb8ea96fdf9b17ab46" candidate_uuid: "974b802141bd4f13beffef2aa03f88f9" candidate_term: 8 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "7a9daf5d42a84615a6383cfec9a10608" is_pre_election: true
W20260129 03:00:15.215164 21135 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34046 (request call id 592) took 1841 ms (client timeout 1553 ms). Trace:
I20260129 03:00:15.215548 20512 raft_consensus.cc:2468] T cb0b3f259ebc4310927338a8740e87b8 P 974b802141bd4f13beffef2aa03f88f9 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7a9daf5d42a84615a6383cfec9a10608 in term 8.
W20260129 03:00:15.215708 21135 rpcz_store.cc:269] 0129 03:00:13.373555 (+ 0us) service_pool.cc:168] Inserting onto call queue
0129 03:00:13.373785 (+ 230us) service_pool.cc:225] Handling call
0129 03:00:15.138005 (+1764220us) spinlock_profiling.cc:243] Waited 1.32 ms on lock 0x7b5800111f98. stack: 00007f8fe044e373 00007f8fe044e270 00007f8fdfda0574 00007f8fe9a5c6b4 00007f8fe9a5c649 00007f8fe02c08b1 00007f8fe0442db1 00007f8fe0442c62 00007f8fe92eddd9 00007f8fe846d260 00007f8fe8535dbb 00007f8fe8535e41 00007f8fe84e3bd0 00007f8fe84e34b4 00007f8fe1a73834 00007f8fe1a737c2
0129 03:00:15.215048 (+ 77043us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":3034496}
I20260129 03:00:15.220747 20553 leader_election.cc:304] T 4b1876f9a70b49edb3aa74105e66d3b3 P 1c50de89a5a845668696dcb28c3badd6 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 197b880fa39d432a9b472845876bfaf0, 1c50de89a5a845668696dcb28c3badd6; no voters:
I20260129 03:00:15.223299 23564 raft_consensus.cc:2804] T 4b1876f9a70b49edb3aa74105e66d3b3 P 1c50de89a5a845668696dcb28c3badd6 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260129 03:00:15.223785 23564 raft_consensus.cc:493] T 4b1876f9a70b49edb3aa74105e66d3b3 P 1c50de89a5a845668696dcb28c3badd6 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260129 03:00:15.224212 23564 raft_consensus.cc:3060] T 4b1876f9a70b49edb3aa74105e66d3b3 P 1c50de89a5a845668696dcb28c3badd6 [term 6 FOLLOWER]: Advancing to term 7
I20260129 03:00:15.224352 20510 raft_consensus.cc:2393] T c934d31efd3f4e5195605ddc354d14e1 P 974b802141bd4f13beffef2aa03f88f9 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 7a9daf5d42a84615a6383cfec9a10608 in current term 6: Already voted for candidate 974b802141bd4f13beffef2aa03f88f9 in this term.
I20260129 03:00:15.229812 20438 tablet_service.cc:2038] Received Run Leader Election RPC: tablet_id: "7d9494f5584d4d5db8b9adf1c8fc888b"
dest_uuid: "197b880fa39d432a9b472845876bfaf0"
from {username='slave'} at 127.0.0.1:55676
I20260129 03:00:15.230504 20438 raft_consensus.cc:493] T 7d9494f5584d4d5db8b9adf1c8fc888b P 197b880fa39d432a9b472845876bfaf0 [term 4 FOLLOWER]: Starting forced leader election (received explicit request)
I20260129 03:00:15.230934 20438 raft_consensus.cc:3060] T 7d9494f5584d4d5db8b9adf1c8fc888b P 197b880fa39d432a9b472845876bfaf0 [term 4 FOLLOWER]: Advancing to term 5
I20260129 03:00:15.237026 23485 raft_consensus.cc:2804] T e33a417cf9c54cf5af73a4ee3aac4d3a P 197b880fa39d432a9b472845876bfaf0 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260129 03:00:15.237375 20516 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4e420f90884f4796997b5ea41e49e411" candidate_uuid: "197b880fa39d432a9b472845876bfaf0" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "974b802141bd4f13beffef2aa03f88f9" is_pre_election: true
W20260129 03:00:15.237936 20405 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.18.211.3:37735 timed out after 2.107s (SENT) [suppressed 83 similar messages]
I20260129 03:00:15.237556 23485 raft_consensus.cc:493] T e33a417cf9c54cf5af73a4ee3aac4d3a P 197b880fa39d432a9b472845876bfaf0 [term 6 FOLLOWER]: Starting leader election (detected failure of leader 974b802141bd4f13beffef2aa03f88f9)
W20260129 03:00:15.238157 20407 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.211.2:34517, user_credentials={real_user=slave}} blocked reactor thread for 61519.1us
I20260129 03:00:15.238760 23485 raft_consensus.cc:3060] T e33a417cf9c54cf5af73a4ee3aac4d3a P 197b880fa39d432a9b472845876bfaf0 [term 6 FOLLOWER]: Advancing to term 7
W20260129 03:00:15.238982 20405 leader_election.cc:336] T dc2f4cb752cf4d59b5f6bc689f5d8104 P 197b880fa39d432a9b472845876bfaf0 [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3:37735): Timed out: RequestConsensusVote RPC to 127.18.211.3:37735 timed out after 2.120s (SENT)
I20260129 03:00:15.239480 20407 leader_election.cc:304] T c934d31efd3f4e5195605ddc354d14e1 P 197b880fa39d432a9b472845876bfaf0 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 197b880fa39d432a9b472845876bfaf0, 974b802141bd4f13beffef2aa03f88f9; no voters: 7a9daf5d42a84615a6383cfec9a10608
I20260129 03:00:15.241878 23469 consensus_queue.cc:1048] T 87d5fe48a44a408aaf03e894a67ecb2f P 197b880fa39d432a9b472845876bfaf0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 6, Time since last communication: 0.000s
I20260129 03:00:15.246351 22891 leader_election.cc:290] T 724a236524cb4e7ba3428880681d6cc2 P 1c50de89a5a845668696dcb28c3badd6 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 197b880fa39d432a9b472845876bfaf0 (127.18.211.1:42027), 974b802141bd4f13beffef2aa03f88f9 (127.18.211.2:34517)
W20260129 03:00:15.246503 20335 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:33836 (request call id 153) took 18389 ms (client timeout 15000 ms). Trace:
I20260129 03:00:14.894371 19276 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260129 03:00:15.249122 20435 raft_consensus.cc:1275] T c2633a6fc34c44579de348d208b0c01e P 197b880fa39d432a9b472845876bfaf0 [term 5 FOLLOWER]: Refusing update from remote peer 974b802141bd4f13beffef2aa03f88f9: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 5 index: 3. (index mismatch)
I20260129 03:00:15.249641 23466 leader_election.cc:290] T cb4727386a3e48aca5d71cc6c611c197 P 197b880fa39d432a9b472845876bfaf0 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 974b802141bd4f13beffef2aa03f88f9 (127.18.211.2:34517), 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3:37735)
I20260129 03:00:15.250476 23511 raft_consensus.cc:2749] T 0678b8b468bd43cdbc3e485900844821 P 7a9daf5d42a84615a6383cfec9a10608 [term 9 FOLLOWER]: Leader election lost for term 9. Reason: could not achieve majority
I20260129 03:00:15.251223 23570 raft_consensus.cc:3055] T 45cc5f3d2688486d9b6dbe7e666dedde P 7a9daf5d42a84615a6383cfec9a10608 [term 6 LEADER]: Stepping down as leader of term 6
I20260129 03:00:15.251396 20440 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c934d31efd3f4e5195605ddc354d14e1" candidate_uuid: "974b802141bd4f13beffef2aa03f88f9" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "197b880fa39d432a9b472845876bfaf0" is_pre_election: true
W20260129 03:00:15.251737 21099 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.211.1:42027, user_credentials={real_user=slave}} blocked reactor thread for 46356.5us
I20260129 03:00:15.253084 23365 consensus_queue.cc:1059] T 2e58e9a5a8aa4cecafb464addb4f35ed P 7a9daf5d42a84615a6383cfec9a10608 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 }, Status: INVALID_TERM, Last received: 4.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.001s
I20260129 03:00:15.254521 22932 raft_consensus.cc:3055] T 2e58e9a5a8aa4cecafb464addb4f35ed P 7a9daf5d42a84615a6383cfec9a10608 [term 4 LEADER]: Stepping down as leader of term 4
I20260129 03:00:15.239552 20405 leader_election.cc:304] T dc2f4cb752cf4d59b5f6bc689f5d8104 P 197b880fa39d432a9b472845876bfaf0 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 197b880fa39d432a9b472845876bfaf0; no voters: 1c50de89a5a845668696dcb28c3badd6, 7a9daf5d42a84615a6383cfec9a10608
I20260129 03:00:15.254935 22932 raft_consensus.cc:740] T 2e58e9a5a8aa4cecafb464addb4f35ed P 7a9daf5d42a84615a6383cfec9a10608 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 7a9daf5d42a84615a6383cfec9a10608, State: Running, Role: LEADER
I20260129 03:00:15.280824 21135 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4e420f90884f4796997b5ea41e49e411" candidate_uuid: "197b880fa39d432a9b472845876bfaf0" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "7a9daf5d42a84615a6383cfec9a10608" is_pre_election: true
I20260129 03:00:15.280459 21130 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a609ac02fae84725ab743f95f51d33ce" candidate_uuid: "974b802141bd4f13beffef2aa03f88f9" candidate_term: 10 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "7a9daf5d42a84615a6383cfec9a10608" is_pre_election: true
I20260129 03:00:15.294220 20511 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a8479d1fe0bd41e5946186e3a6ca8c7a" candidate_uuid: "197b880fa39d432a9b472845876bfaf0" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "974b802141bd4f13beffef2aa03f88f9" is_pre_election: true
I20260129 03:00:15.296303 21136 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4e3805ffa70242e8ade66da260dddf66" candidate_uuid: "197b880fa39d432a9b472845876bfaf0" candidate_term: 7 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "7a9daf5d42a84615a6383cfec9a10608" is_pre_election: true
W20260129 03:00:15.307379 20479 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260129 03:00:15.303552 21130 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:47844 (request call id 396) took 1022 ms. Trace:
I20260129 03:00:15.251761 23570 raft_consensus.cc:740] T 45cc5f3d2688486d9b6dbe7e666dedde P 7a9daf5d42a84615a6383cfec9a10608 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 7a9daf5d42a84615a6383cfec9a10608, State: Running, Role: LEADER
I20260129 03:00:15.311973 21133 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1299723a4476420ab02fd8bfd7aa0645" candidate_uuid: "974b802141bd4f13beffef2aa03f88f9" candidate_term: 11 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "7a9daf5d42a84615a6383cfec9a10608" is_pre_election: true
I20260129 03:00:15.307899 21130 rpcz_store.cc:276] 0129 03:00:14.281037 (+ 0us) service_pool.cc:168] Inserting onto call queue
0129 03:00:15.039168 (+758131us) service_pool.cc:225] Handling call
0129 03:00:15.280233 (+241065us) spinlock_profiling.cc:243] Waited 40.3 ms on lock 0x7b5800111f98. stack: 00007f8fe044e373 00007f8fe044e270 00007f8fdfda0574 00007f8fe9a5c6b4 00007f8fe9a5c649 00007f8fe02c08b1 00007f8fe0442db1 00007f8fe0442c62 00007f8fe92eddd9 00007f8fe846d260 00007f8fe8535dbb 00007f8fe8535e41 00007f8fe84e3bd0 00007f8fe84e34b4 00007f8fe1a73834 00007f8fe1a737c2
0129 03:00:15.303485 (+ 23252us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":92677504}
I20260129 03:00:15.319998 21133 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:47844 (request call id 390) took 1040 ms. Trace:
I20260129 03:00:15.320412 21133 rpcz_store.cc:276] 0129 03:00:14.279287 (+ 0us) service_pool.cc:168] Inserting onto call queue
0129 03:00:15.108323 (+829036us) service_pool.cc:225] Handling call
0129 03:00:15.232655 (+124332us) spinlock_profiling.cc:243] Waited 91.4 ms on lock 0x7b5800111f98. stack: 00007f8fe044e373 00007f8fe044e270 00007f8fdfda0574 00007f8fe9a5c6b4 00007f8fe9a5c649 00007f8fe02c08b1 00007f8fe0442db1 00007f8fe0442c62 00007f8fe92eddd9 00007f8fe846d260 00007f8fe8535dbb 00007f8fe8535e41 00007f8fe84e3bd0 00007f8fe84e34b4 00007f8fe1a73834 00007f8fe1a737c2
0129 03:00:15.303617 (+ 70962us) spinlock_profiling.cc:243] Waited 5.63 ms on lock 0x7b5800111f98. stack: 00007f8fe044e373 00007f8fe044e270 00007f8fdfda0574 00007f8fe9a5c6b4 00007f8fe9a5c649 00007f8fe02c08b1 00007f8fe04430dc 00007f8fe0442c9b 00007f8fe92eddd9 00007f8fe846d260 00007f8fe8535dbb 00007f8fe8535e41 00007f8fe84e3bd0 00007f8fe84e34b4 00007f8fe1a73834 00007f8fe1a737c2
0129 03:00:15.319935 (+ 16318us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":223145088}
I20260129 03:00:15.321254 21129 raft_consensus.cc:1275] T a609ac02fae84725ab743f95f51d33ce P 7a9daf5d42a84615a6383cfec9a10608 [term 9 FOLLOWER]: Refusing update from remote peer 197b880fa39d432a9b472845876bfaf0: Log matching property violated. Preceding OpId in replica: term: 8 index: 8. Preceding OpId from leader: term: 9 index: 9. (index mismatch)
I20260129 03:00:15.322340 23485 raft_consensus.cc:515] T e33a417cf9c54cf5af73a4ee3aac4d3a P 197b880fa39d432a9b472845876bfaf0 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 03:00:15.322472 23480 raft_consensus.cc:515] T febffe4a8b394368b69b1d164308fb9a P 7a9daf5d42a84615a6383cfec9a10608 [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } }
I20260129 03:00:15.323824 21128 raft_consensus.cc:2393] T febffe4a8b394368b69b1d164308fb9a P 7a9daf5d42a84615a6383cfec9a10608 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 974b802141bd4f13beffef2aa03f88f9 in current term 6: Already voted for candidate 7a9daf5d42a84615a6383cfec9a10608 in this term.
I20260129 03:00:15.299932 20586 raft_consensus.cc:493] T 4f8f3a38f6e24ba4857875181f447442 P 1c50de89a5a845668696dcb28c3badd6 [term 6 FOLLOWER]: Starting forced leader election (received explicit request)
W20260129 03:00:15.330854 20479 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260129 03:00:15.331275 21132 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fd8311b779af4e1fa1402b2a02a602f2" candidate_uuid: "197b880fa39d432a9b472845876bfaf0" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "7a9daf5d42a84615a6383cfec9a10608" is_pre_election: true
I20260129 03:00:15.324151 20480 leader_election.cc:304] T 4e420f90884f4796997b5ea41e49e411 P 974b802141bd4f13beffef2aa03f88f9 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 974b802141bd4f13beffef2aa03f88f9; no voters: 197b880fa39d432a9b472845876bfaf0, 7a9daf5d42a84615a6383cfec9a10608
I20260129 03:00:15.331425 20556 leader_election.cc:304] T 0340229d521b432580ae926f6062e3f7 P 1c50de89a5a845668696dcb28c3badd6 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1c50de89a5a845668696dcb28c3badd6; no voters: 197b880fa39d432a9b472845876bfaf0, 974b802141bd4f13beffef2aa03f88f9
I20260129 03:00:15.330587 21134 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a609ac02fae84725ab743f95f51d33ce" candidate_uuid: "197b880fa39d432a9b472845876bfaf0" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "7a9daf5d42a84615a6383cfec9a10608"
I20260129 03:00:15.341601 20586 raft_consensus.cc:3060] T 4f8f3a38f6e24ba4857875181f447442 P 1c50de89a5a845668696dcb28c3badd6 [term 6 FOLLOWER]: Advancing to term 7
I20260129 03:00:15.324818 23485 leader_election.cc:290] T e33a417cf9c54cf5af73a4ee3aac4d3a P 197b880fa39d432a9b472845876bfaf0 [CANDIDATE]: Term 7 election: Requested vote from peers 974b802141bd4f13beffef2aa03f88f9 (127.18.211.2:34517), 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3:37735)
I20260129 03:00:15.336663 21135 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3ccac82a01a2445fa5b0ce6c0fcf2295" candidate_uuid: "197b880fa39d432a9b472845876bfaf0" candidate_term: 5 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "7a9daf5d42a84615a6383cfec9a10608" is_pre_election: true
W20260129 03:00:15.350240 20479 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260129 03:00:15.315248 23466 raft_consensus.cc:2749] T dc2f4cb752cf4d59b5f6bc689f5d8104 P 197b880fa39d432a9b472845876bfaf0 [term 6 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260129 03:00:15.353704 20438 raft_consensus.cc:515] T 7d9494f5584d4d5db8b9adf1c8fc888b P 197b880fa39d432a9b472845876bfaf0 [term 5 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 03:00:15.360181 21134 raft_consensus.cc:2376] T a609ac02fae84725ab743f95f51d33ce P 7a9daf5d42a84615a6383cfec9a10608 [term 9 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 197b880fa39d432a9b472845876bfaf0 in term 9. Re-sending same reply.
I20260129 03:00:15.369503 20438 leader_election.cc:290] T 7d9494f5584d4d5db8b9adf1c8fc888b P 197b880fa39d432a9b472845876bfaf0 [CANDIDATE]: Term 5 election: Requested vote from peers 974b802141bd4f13beffef2aa03f88f9 (127.18.211.2:34517), 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3:37735)
I20260129 03:00:15.361863 21132 raft_consensus.cc:2468] T fd8311b779af4e1fa1402b2a02a602f2 P 7a9daf5d42a84615a6383cfec9a10608 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 197b880fa39d432a9b472845876bfaf0 in term 5.
W20260129 03:00:15.378296 20554 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.211.4:40439, user_credentials={real_user=slave}} blocked reactor thread for 171286us
I20260129 03:00:15.363381 21099 leader_election.cc:304] T 87d5fe48a44a408aaf03e894a67ecb2f P 7a9daf5d42a84615a6383cfec9a10608 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7a9daf5d42a84615a6383cfec9a10608; no voters: 197b880fa39d432a9b472845876bfaf0, 974b802141bd4f13beffef2aa03f88f9
W20260129 03:00:15.386802 20405 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.211.3:37735, user_credentials={real_user=slave}} blocked reactor thread for 147918us
W20260129 03:00:15.387486 20407 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.211.2:34517, user_credentials={real_user=slave}} blocked reactor thread for 148282us
W20260129 03:00:15.397210 21102 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.211.2:34517, user_credentials={real_user=slave}} blocked reactor thread for 75023.7us
I20260129 03:00:15.397821 23456 raft_consensus.cc:2749] T 1ce4ddf5b10e4ae8b193b13b2e5936ee P 1c50de89a5a845668696dcb28c3badd6 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260129 03:00:15.398445 20440 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "febffe4a8b394368b69b1d164308fb9a" candidate_uuid: "974b802141bd4f13beffef2aa03f88f9" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "197b880fa39d432a9b472845876bfaf0"
I20260129 03:00:15.399317 20440 raft_consensus.cc:3060] T febffe4a8b394368b69b1d164308fb9a P 197b880fa39d432a9b472845876bfaf0 [term 5 FOLLOWER]: Advancing to term 6
I20260129 03:00:15.404146 23604 raft_consensus.cc:2749] T 0340229d521b432580ae926f6062e3f7 P 1c50de89a5a845668696dcb28c3badd6 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260129 03:00:15.408236 23450 raft_consensus.cc:493] T 2ac659d732314e1288849f08fe5e074e P 974b802141bd4f13beffef2aa03f88f9 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260129 03:00:15.408670 20582 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:48526 (request call id 404) took 1621 ms. Trace:
I20260129 03:00:15.408792 23450 raft_consensus.cc:515] T 2ac659d732314e1288849f08fe5e074e P 974b802141bd4f13beffef2aa03f88f9 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } }
I20260129 03:00:15.409130 20582 rpcz_store.cc:276] 0129 03:00:13.787178 (+ 0us) service_pool.cc:168] Inserting onto call queue
0129 03:00:15.296378 (+1509200us) service_pool.cc:225] Handling call
0129 03:00:15.376688 (+ 80310us) raft_consensus.cc:1471] Updating replica for 0 ops
0129 03:00:15.408290 (+ 31602us) raft_consensus.cc:1534] Early marking committed up to index 0
0129 03:00:15.408334 (+ 44us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0129 03:00:15.408379 (+ 45us) raft_consensus.cc:1645] Marking committed up to 7
0129 03:00:15.408411 (+ 32us) raft_consensus.cc:1697] Filling consensus response to leader.
0129 03:00:15.408476 (+ 65us) raft_consensus.cc:1691] UpdateReplicas() finished
0129 03:00:15.408595 (+ 119us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260129 03:00:15.411067 20582 raft_consensus.cc:1240] T e33a417cf9c54cf5af73a4ee3aac4d3a P 1c50de89a5a845668696dcb28c3badd6 [term 7 FOLLOWER]: Rejecting Update request from peer 974b802141bd4f13beffef2aa03f88f9 for earlier term 6. Current term is 7. Ops: []
I20260129 03:00:15.411257 23450 leader_election.cc:290] T 2ac659d732314e1288849f08fe5e074e P 974b802141bd4f13beffef2aa03f88f9 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 197b880fa39d432a9b472845876bfaf0 (127.18.211.1:42027), 7a9daf5d42a84615a6383cfec9a10608 (127.18.211.4:40439)
I20260129 03:00:15.412207 20441 raft_consensus.cc:1275] T 0678b8b468bd43cdbc3e485900844821 P 197b880fa39d432a9b472845876bfaf0 [term 9 FOLLOWER]: Refusing update from remote peer 974b802141bd4f13beffef2aa03f88f9: Log matching property violated. Preceding OpId in replica: term: 5 index: 7. Preceding OpId from leader: term: 9 index: 9. (index mismatch)
I20260129 03:00:15.412287 22932 consensus_queue.cc:260] T 2e58e9a5a8aa4cecafb464addb4f35ed P 7a9daf5d42a84615a6383cfec9a10608 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } }
I20260129 03:00:15.413808 20441 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a8479d1fe0bd41e5946186e3a6ca8c7a" candidate_uuid: "1c50de89a5a845668696dcb28c3badd6" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "197b880fa39d432a9b472845876bfaf0" is_pre_election: true
I20260129 03:00:15.414418 20438 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e7d83d4ccf684e76acb29a0dc844e1b8" candidate_uuid: "1c50de89a5a845668696dcb28c3badd6" candidate_term: 8 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "197b880fa39d432a9b472845876bfaf0" is_pre_election: true
I20260129 03:00:15.414752 20554 leader_election.cc:304] T 42d27a28d89b409e8807ec0efd97af69 P 1c50de89a5a845668696dcb28c3badd6 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1c50de89a5a845668696dcb28c3badd6; no voters: 197b880fa39d432a9b472845876bfaf0, 7a9daf5d42a84615a6383cfec9a10608
W20260129 03:00:15.418248 21099 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.211.1:42027, user_credentials={real_user=slave}} blocked reactor thread for 115162us
I20260129 03:00:15.420495 23570 consensus_queue.cc:260] T 45cc5f3d2688486d9b6dbe7e666dedde P 7a9daf5d42a84615a6383cfec9a10608 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 6.6, Last appended by leader: 6, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } }
I20260129 03:00:15.411732 20582 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35752 (request call id 1749) took 1543 ms. Trace:
I20260129 03:00:15.488958 20582 rpcz_store.cc:276] 0129 03:00:13.867813 (+ 0us) service_pool.cc:168] Inserting onto call queue
0129 03:00:15.410215 (+1542402us) service_pool.cc:225] Handling call
0129 03:00:15.410664 (+ 449us) raft_consensus.cc:1471] Updating replica for 0 ops
0129 03:00:15.411443 (+ 779us) raft_consensus.cc:1697] Filling consensus response to leader.
0129 03:00:15.411656 (+ 213us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260129 03:00:15.413827 22932 raft_consensus.cc:3060] T 2e58e9a5a8aa4cecafb464addb4f35ed P 7a9daf5d42a84615a6383cfec9a10608 [term 4 FOLLOWER]: Advancing to term 5
I20260129 03:00:15.415915 20438 raft_consensus.cc:1240] T 7d9494f5584d4d5db8b9adf1c8fc888b P 197b880fa39d432a9b472845876bfaf0 [term 5 FOLLOWER]: Rejecting Update request from peer 1c50de89a5a845668696dcb28c3badd6 for earlier term 4. Current term is 5. Ops: []
W20260129 03:00:15.247020 20335 rpcz_store.cc:269] 0129 02:59:56.857362 (+ 0us) service_pool.cc:168] Inserting onto call queue
0129 02:59:56.858363 (+ 1001us) service_pool.cc:225] Handling call
0129 03:00:15.246255 (+18387892us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0129 03:00:07.368055 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0129 03:00:07.611500 (+243445us) write_op.cc:432] Acquiring schema lock in shared mode
0129 03:00:07.692822 (+ 81322us) write_op.cc:435] Acquired schema lock
0129 03:00:07.737963 (+ 45141us) tablet.cc:660] Decoding operations
0129 03:00:08.200741 (+462778us) write_op.cc:620] Acquiring the partition lock for write op
0129 03:00:08.200874 (+ 133us) write_op.cc:641] Partition lock acquired for write op
0129 03:00:08.200903 (+ 29us) tablet.cc:683] Acquiring locks for 13 operations
0129 03:00:08.520299 (+319396us) tablet.cc:699] Row locks acquired
0129 03:00:08.520338 (+ 39us) write_op.cc:260] PREPARE: finished
0129 03:00:08.520577 (+ 239us) write_op.cc:270] Start()
0129 03:00:08.520789 (+ 212us) write_op.cc:276] Timestamp: P: 1769655608520501 usec, L: 0
0129 03:00:08.520826 (+ 37us) op_driver.cc:348] REPLICATION: starting
0129 03:00:08.521687 (+ 861us) log.cc:844] Serialized 5094 byte log entry
0129 03:00:10.485731 (+1964044us) op_driver.cc:464] REPLICATION: finished
0129 03:00:11.250718 (+764987us) write_op.cc:301] APPLY: starting
0129 03:00:11.260016 (+ 9298us) tablet.cc:1365] starting BulkCheckPresence
0129 03:00:11.385794 (+125778us) tablet.cc:1368] finished BulkCheckPresence
0129 03:00:11.402829 (+ 17035us) tablet.cc:1370] starting ApplyRowOperation cycle
0129 03:00:11.575216 (+172387us) tablet.cc:1381] finished ApplyRowOperation cycle
0129 03:00:11.575586 (+ 370us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=13
0129 03:00:11.575648 (+ 62us) write_op.cc:312] APPLY: finished
0129 03:00:11.576067 (+ 419us) log.cc:844] Serialized 95 byte log entry
0129 03:00:11.785828 (+209761us) write_op.cc:489] Releasing partition, row and schema locks
0129 03:00:11.796394 (+ 10566us) write_op.cc:454] Released schema lock
0129 03:00:11.802958 (+ 6564us) write_op.cc:341] FINISH: Updating metrics
0129 03:00:11.873529 (+ 70571us) write_op.cc:489] Releasing partition, row and schema locks
0129 03:00:11.878287 (+ 4758us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":8543,"reported_tablets":22,"child_traces":[["op",{"apply.queue_time_us":715379,"apply.run_cpu_time_us":52044,"apply.run_wall_time_us":613570,"num_ops":13,"prepare.queue_time_us":413767,"prepare.run_cpu_time_us":21257,"prepare.run_wall_time_us":1705525,"raft.queue_time_us":489283,"raft.run_cpu_time_us":63043,"raft.run_wall_time_us":589647,"replication_time_us":1790949,"thread_start_us":1588504,"threads_started":5,"wal-append.queue_time_us":1203034,"wal-append.run_cpu_time_us":45646,"wal-append.run_wall_time_us":2805193}]]}
I20260129 03:00:15.454022 20581 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:48526 (request call id 401) took 3077 ms. Trace:
I20260129 03:00:15.389256 22989 raft_consensus.cc:493] T 724a236524cb4e7ba3428880681d6cc2 P 974b802141bd4f13beffef2aa03f88f9 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260129 03:00:15.419696 23603 raft_consensus.cc:2749] T 42d27a28d89b409e8807ec0efd97af69 P 1c50de89a5a845668696dcb28c3badd6 [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260129 03:00:15.439497 21127 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2ac659d732314e1288849f08fe5e074e" candidate_uuid: "197b880fa39d432a9b472845876bfaf0" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "7a9daf5d42a84615a6383cfec9a10608" is_pre_election: true
I20260129 03:00:15.403746 20433 raft_consensus.cc:2468] T c934d31efd3f4e5195605ddc354d14e1 P 197b880fa39d432a9b472845876bfaf0 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7a9daf5d42a84615a6383cfec9a10608 in term 6.
I20260129 03:00:15.515523 23163 raft_consensus.cc:2749] T 87d5fe48a44a408aaf03e894a67ecb2f P 7a9daf5d42a84615a6383cfec9a10608 [term 7 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260129 03:00:15.515879 20432 raft_consensus.cc:1240] T 8055232cf0004d40a85f3bf983fe00a9 P 197b880fa39d432a9b472845876bfaf0 [term 6 FOLLOWER]: Rejecting Update request from peer 974b802141bd4f13beffef2aa03f88f9 for earlier term 5. Current term is 6. Ops: []
I20260129 03:00:15.515995 20439 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7d9494f5584d4d5db8b9adf1c8fc888b" candidate_uuid: "974b802141bd4f13beffef2aa03f88f9" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "197b880fa39d432a9b472845876bfaf0" is_pre_election: true
I20260129 03:00:15.515544 23427 raft_consensus.cc:2749] T 4e420f90884f4796997b5ea41e49e411 P 974b802141bd4f13beffef2aa03f88f9 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260129 03:00:15.517393 20438 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c2633a6fc34c44579de348d208b0c01e" candidate_uuid: "1c50de89a5a845668696dcb28c3badd6" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "197b880fa39d432a9b472845876bfaf0" is_pre_election: true
I20260129 03:00:15.518170 20438 raft_consensus.cc:2468] T c2633a6fc34c44579de348d208b0c01e P 197b880fa39d432a9b472845876bfaf0 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1c50de89a5a845668696dcb28c3badd6 in term 5.
I20260129 03:00:15.519037 21127 raft_consensus.cc:2468] T 2ac659d732314e1288849f08fe5e074e P 7a9daf5d42a84615a6383cfec9a10608 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 197b880fa39d432a9b472845876bfaf0 in term 6.
I20260129 03:00:15.519443 20584 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35754 (request call id 1850) took 3470 ms. Trace:
I20260129 03:00:15.519887 20584 rpcz_store.cc:276] 0129 03:00:11.966216 (+ 0us) service_pool.cc:168] Inserting onto call queue
0129 03:00:12.705043 (+738827us) service_pool.cc:225] Handling call
0129 03:00:12.705504 (+ 461us) raft_consensus.cc:1471] Updating replica for 3 ops
0129 03:00:12.848351 (+142847us) raft_consensus.cc:1534] Early marking committed up to index 2
0129 03:00:13.065602 (+217251us) log.cc:844] Serialized 14 byte log entry
0129 03:00:14.034097 (+968495us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0129 03:00:14.034647 (+ 550us) log.cc:844] Serialized 52 byte log entry
0129 03:00:14.034785 (+ 138us) raft_consensus.cc:1645] Marking committed up to 4
0129 03:00:14.269649 (+234864us) log.cc:844] Serialized 14 byte log entry
0129 03:00:14.727668 (+458019us) log.cc:844] Serialized 14 byte log entry
0129 03:00:15.317189 (+589521us) raft_consensus.cc:1697] Filling consensus response to leader.
0129 03:00:15.381771 (+ 64582us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0129 03:00:15.391430 (+ 9659us) raft_consensus.cc:1685] finished
0129 03:00:15.391450 (+ 20us) raft_consensus.cc:1691] UpdateReplicas() finished
0129 03:00:15.436111 (+ 44661us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":1029127,"prepare.queue_time_us":604993,"prepare.run_cpu_time_us":157,"prepare.run_wall_time_us":4328,"raft.queue_time_us":798752,"raft.run_cpu_time_us":478,"raft.run_wall_time_us":25701,"thread_start_us":1636698,"threads_started":4,"wal-append.queue_time_us":1047734}
I20260129 03:00:15.518008 23555 raft_consensus.cc:493] T 3ccac82a01a2445fa5b0ce6c0fcf2295 P 974b802141bd4f13beffef2aa03f88f9 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 7a9daf5d42a84615a6383cfec9a10608)
I20260129 03:00:15.522164 23555 raft_consensus.cc:515] T 3ccac82a01a2445fa5b0ce6c0fcf2295 P 974b802141bd4f13beffef2aa03f88f9 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } }
I20260129 03:00:15.522935 20584 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35752 (request call id 1751) took 1642 ms. Trace:
I20260129 03:00:15.524327 23514 raft_consensus.cc:493] T 4f8f3a38f6e24ba4857875181f447442 P 974b802141bd4f13beffef2aa03f88f9 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260129 03:00:15.524824 23600 log.cc:927] Time spent T 45cc5f3d2688486d9b6dbe7e666dedde P 1c50de89a5a845668696dcb28c3badd6: Append to log took a long time: real 0.089s user 0.000s sys 0.000s
I20260129 03:00:15.526539 20588 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:48526 (request call id 400) took 3397 ms. Trace:
I20260129 03:00:15.526938 20588 rpcz_store.cc:276] 0129 03:00:12.129528 (+ 0us) service_pool.cc:168] Inserting onto call queue
0129 03:00:14.242389 (+2112861us) service_pool.cc:225] Handling call
0129 03:00:14.259336 (+ 16947us) raft_consensus.cc:1471] Updating replica for 1 ops
0129 03:00:14.727038 (+467702us) raft_consensus.cc:1534] Early marking committed up to index 0
0129 03:00:14.727071 (+ 33us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0129 03:00:14.732880 (+ 5809us) log.cc:844] Serialized 26 byte log entry
0129 03:00:14.818208 (+ 85328us) spinlock_profiling.cc:243] Waited 10.4 ms on lock 0x7b480018c258. stack: 00007f8fe044e373 00007f8fe044e270 00007f8fdfda0574 00007f8fe9a5c6b4 00007f8fe9a5c649 00007f8fe8447e21 00007f8fe843eeb7 00007f8fe843ed4a 00007f8fe8535f7f 00007f8fe8535c47 00007f8fe854a26c 00007f8fe854a218 00007f8fe854a198 00007f8fe854a153 00007f8fe85493fc 00007f8fe74845b6
0129 03:00:15.185880 (+367672us) raft_consensus.cc:1645] Marking committed up to 5
0129 03:00:15.185949 (+ 69us) raft_consensus.cc:1697] Filling consensus response to leader.
0129 03:00:15.186044 (+ 95us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0129 03:00:15.526072 (+340028us) raft_consensus.cc:1685] finished
0129 03:00:15.526094 (+ 22us) raft_consensus.cc:1691] UpdateReplicas() finished
0129 03:00:15.526343 (+ 249us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":694900,"raft.queue_time_us":483517,"raft.run_cpu_time_us":2623,"raft.run_wall_time_us":71658,"spinlock_wait_cycles":23904000,"thread_start_us":173129,"threads_started":1,"wal-append.queue_time_us":558612}
I20260129 03:00:15.532028 23469 raft_consensus.cc:2804] T c934d31efd3f4e5195605ddc354d14e1 P 197b880fa39d432a9b472845876bfaf0 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260129 03:00:15.533964 23469 raft_consensus.cc:493] T c934d31efd3f4e5195605ddc354d14e1 P 197b880fa39d432a9b472845876bfaf0 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260129 03:00:15.534420 23469 raft_consensus.cc:3060] T c934d31efd3f4e5195605ddc354d14e1 P 197b880fa39d432a9b472845876bfaf0 [term 6 FOLLOWER]: Advancing to term 7
W20260129 03:00:15.537593 20480 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.211.4:40439, user_credentials={real_user=slave}} blocked reactor thread for 237301us
I20260129 03:00:15.540216 23333 consensus_queue.cc:1048] T 0678b8b468bd43cdbc3e485900844821 P 974b802141bd4f13beffef2aa03f88f9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.001s
I20260129 03:00:15.544378 22989 raft_consensus.cc:515] T 724a236524cb4e7ba3428880681d6cc2 P 974b802141bd4f13beffef2aa03f88f9 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 03:00:15.546734 20479 leader_election.cc:304] T c934d31efd3f4e5195605ddc354d14e1 P 974b802141bd4f13beffef2aa03f88f9 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 974b802141bd4f13beffef2aa03f88f9; no voters: 197b880fa39d432a9b472845876bfaf0, 7a9daf5d42a84615a6383cfec9a10608
I20260129 03:00:15.548143 23450 consensus_queue.cc:1048] T c2633a6fc34c44579de348d208b0c01e P 974b802141bd4f13beffef2aa03f88f9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260129 03:00:15.548451 20554 leader_election.cc:304] T 18f30ebaa4174fd28c958e72d0cc7e24 P 1c50de89a5a845668696dcb28c3badd6 [CANDIDATE]: Term 8 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1c50de89a5a845668696dcb28c3badd6; no voters: 197b880fa39d432a9b472845876bfaf0, 7a9daf5d42a84615a6383cfec9a10608
I20260129 03:00:15.549793 23450 raft_consensus.cc:2749] T c934d31efd3f4e5195605ddc354d14e1 P 974b802141bd4f13beffef2aa03f88f9 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
W20260129 03:00:15.550151 20554 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.211.4:40439, user_credentials={real_user=slave}} blocked reactor thread for 50854.1us
I20260129 03:00:15.551410 23603 raft_consensus.cc:2749] T 18f30ebaa4174fd28c958e72d0cc7e24 P 1c50de89a5a845668696dcb28c3badd6 [term 8 FOLLOWER]: Leader election lost for term 8. Reason: could not achieve majority
I20260129 03:00:15.551828 20440 raft_consensus.cc:2468] T febffe4a8b394368b69b1d164308fb9a P 197b880fa39d432a9b472845876bfaf0 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 974b802141bd4f13beffef2aa03f88f9 in term 6.
I20260129 03:00:15.519279 20436 raft_consensus.cc:1240] T 0340229d521b432580ae926f6062e3f7 P 197b880fa39d432a9b472845876bfaf0 [term 8 FOLLOWER]: Rejecting Update request from peer 974b802141bd4f13beffef2aa03f88f9 for earlier term 7. Current term is 8. Ops: []
I20260129 03:00:15.544552 20582 raft_consensus.cc:1275] T c2633a6fc34c44579de348d208b0c01e P 1c50de89a5a845668696dcb28c3badd6 [term 5 FOLLOWER]: Refusing update from remote peer 974b802141bd4f13beffef2aa03f88f9: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 5 index: 3. (term mismatch)
W20260129 03:00:15.552634 20482 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260129 03:00:15.553064 21102 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.211.2:34517, user_credentials={real_user=slave}} blocked reactor thread for 43640.9us
I20260129 03:00:15.551573 20553 leader_election.cc:304] T c2633a6fc34c44579de348d208b0c01e P 1c50de89a5a845668696dcb28c3badd6 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 197b880fa39d432a9b472845876bfaf0, 1c50de89a5a845668696dcb28c3badd6; no voters:
I20260129 03:00:15.523336 20584 rpcz_store.cc:276] 0129 03:00:13.880474 (+ 0us) service_pool.cc:168] Inserting onto call queue
0129 03:00:15.521910 (+1641436us) service_pool.cc:225] Handling call
0129 03:00:15.522352 (+ 442us) raft_consensus.cc:1471] Updating replica for 0 ops
0129 03:00:15.522604 (+ 252us) raft_consensus.cc:1534] Early marking committed up to index 0
0129 03:00:15.522630 (+ 26us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0129 03:00:15.522700 (+ 70us) raft_consensus.cc:1645] Marking committed up to 3
0129 03:00:15.522733 (+ 33us) raft_consensus.cc:1697] Filling consensus response to leader.
0129 03:00:15.522786 (+ 53us) raft_consensus.cc:1691] UpdateReplicas() finished
0129 03:00:15.522881 (+ 95us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20260129 03:00:15.566882 20556 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.211.2:34517, user_credentials={real_user=slave}} blocked reactor thread for 250991us
W20260129 03:00:15.594349 20479 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.211.1:42027, user_credentials={real_user=slave}} blocked reactor thread for 47724.4us
I20260129 03:00:15.580060 20582 pending_rounds.cc:85] T c2633a6fc34c44579de348d208b0c01e P 1c50de89a5a845668696dcb28c3badd6: Aborting all ops after (but not including) 2
I20260129 03:00:15.606485 20582 pending_rounds.cc:107] T c2633a6fc34c44579de348d208b0c01e P 1c50de89a5a845668696dcb28c3badd6: Aborting uncommitted NO_OP operation due to leader change: 4.3
I20260129 03:00:15.605875 20441 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "724a236524cb4e7ba3428880681d6cc2" candidate_uuid: "1c50de89a5a845668696dcb28c3badd6" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "197b880fa39d432a9b472845876bfaf0" is_pre_election: true
I20260129 03:00:15.607905 21132 raft_consensus.cc:1275] T 2ac659d732314e1288849f08fe5e074e P 7a9daf5d42a84615a6383cfec9a10608 [term 6 FOLLOWER]: Refusing update from remote peer 197b880fa39d432a9b472845876bfaf0: Log matching property violated. Preceding OpId in replica: term: 5 index: 6. Preceding OpId from leader: term: 6 index: 7. (index mismatch)
I20260129 03:00:15.606781 20582 raft_consensus.cc:2889] T c2633a6fc34c44579de348d208b0c01e P 1c50de89a5a845668696dcb28c3badd6 [term 5 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
W20260129 03:00:15.620471 21100 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260129 03:00:15.525138 23514 raft_consensus.cc:515] T 4f8f3a38f6e24ba4857875181f447442 P 974b802141bd4f13beffef2aa03f88f9 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 03:00:15.537611 23555 leader_election.cc:290] T 3ccac82a01a2445fa5b0ce6c0fcf2295 P 974b802141bd4f13beffef2aa03f88f9 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 197b880fa39d432a9b472845876bfaf0 (127.18.211.1:42027), 7a9daf5d42a84615a6383cfec9a10608 (127.18.211.4:40439)
I20260129 03:00:15.495107 20581 rpcz_store.cc:276] 0129 03:00:12.313285 (+ 0us) service_pool.cc:168] Inserting onto call queue
0129 03:00:14.364848 (+2051563us) service_pool.cc:225] Handling call
0129 03:00:14.370777 (+ 5929us) raft_consensus.cc:1471] Updating replica for 0 ops
0129 03:00:14.370997 (+ 220us) raft_consensus.cc:1534] Early marking committed up to index 9
0129 03:00:14.393712 (+ 22715us) log.cc:844] Serialized 14 byte log entry
0129 03:00:15.308575 (+914863us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0129 03:00:15.342549 (+ 33974us) raft_consensus.cc:1645] Marking committed up to 9
0129 03:00:15.368270 (+ 25721us) raft_consensus.cc:1697] Filling consensus response to leader.
0129 03:00:15.380788 (+ 12518us) raft_consensus.cc:1691] UpdateReplicas() finished
0129 03:00:15.390173 (+ 9385us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":513633,"prepare.queue_time_us":25456,"prepare.run_cpu_time_us":58,"prepare.run_wall_time_us":45,"raft.queue_time_us":536318,"raft.run_cpu_time_us":342,"raft.run_wall_time_us":342,"thread_start_us":350818,"threads_started":2,"wal-append.queue_time_us":232731}
W20260129 03:00:15.681082 20482 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260129 03:00:15.691310 20479 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.211.1:42027, user_credentials={real_user=slave}} blocked reactor thread for 95986.4us
W20260129 03:00:15.695483 20553 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.211.1:42027, user_credentials={real_user=slave}} blocked reactor thread for 46039.4us
I20260129 03:00:15.704835 20433 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2ac659d732314e1288849f08fe5e074e" candidate_uuid: "974b802141bd4f13beffef2aa03f88f9" candidate_term: 7 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "197b880fa39d432a9b472845876bfaf0" is_pre_election: true
I20260129 03:00:15.707878 20583 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35752 (request call id 1748) took 1840 ms. Trace:
I20260129 03:00:15.708469 20583 rpcz_store.cc:276] 0129 03:00:13.867505 (+ 0us) service_pool.cc:168] Inserting onto call queue
0129 03:00:15.190189 (+1322684us) service_pool.cc:225] Handling call
0129 03:00:15.190662 (+ 473us) raft_consensus.cc:1471] Updating replica for 0 ops
0129 03:00:15.191106 (+ 444us) raft_consensus.cc:1534] Early marking committed up to index 3
0129 03:00:15.209724 (+ 18618us) log.cc:844] Serialized 14 byte log entry
0129 03:00:15.707374 (+497650us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0129 03:00:15.707462 (+ 88us) raft_consensus.cc:1645] Marking committed up to 3
0129 03:00:15.707532 (+ 70us) raft_consensus.cc:1697] Filling consensus response to leader.
0129 03:00:15.707622 (+ 90us) raft_consensus.cc:1691] UpdateReplicas() finished
0129 03:00:15.707781 (+ 159us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":39597,"prepare.queue_time_us":5608,"prepare.run_cpu_time_us":45,"prepare.run_wall_time_us":45,"thread_start_us":324578,"threads_started":1}
I20260129 03:00:15.721067 20441 raft_consensus.cc:2410] T 724a236524cb4e7ba3428880681d6cc2 P 197b880fa39d432a9b472845876bfaf0 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1c50de89a5a845668696dcb28c3badd6 for term 7 because replica has last-logged OpId of term: 5 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 4 index: 3.
W20260129 03:00:15.729344 20553 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260129 03:00:15.742695 20407 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.211.2:34517, user_credentials={real_user=slave}} blocked reactor thread for 136920us
I20260129 03:00:15.672480 23475 raft_consensus.cc:493] T a1a39dcd328e4268ae41b988c66b4a75 P 197b880fa39d432a9b472845876bfaf0 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260129 03:00:15.680336 22996 raft_consensus.cc:493] T 17d6431cc77a491e8464d68db4081a51 P 1c50de89a5a845668696dcb28c3badd6 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260129 03:00:15.607142 23466 consensus_queue.cc:1059] T 0f20af7fbe5d4db89a6f17948d98912c P 197b880fa39d432a9b472845876bfaf0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 }, Status: INVALID_TERM, Last received: 7.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.123s
I20260129 03:00:15.729265 23570 raft_consensus.cc:3060] T 45cc5f3d2688486d9b6dbe7e666dedde P 7a9daf5d42a84615a6383cfec9a10608 [term 6 FOLLOWER]: Advancing to term 7
I20260129 03:00:15.720911 23480 leader_election.cc:290] T febffe4a8b394368b69b1d164308fb9a P 7a9daf5d42a84615a6383cfec9a10608 [CANDIDATE]: Term 6 election: Requested vote from peers 197b880fa39d432a9b472845876bfaf0 (127.18.211.1:42027), 974b802141bd4f13beffef2aa03f88f9 (127.18.211.2:34517)
I20260129 03:00:15.774565 20582 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35752 (request call id 1750) took 1894 ms. Trace:
I20260129 03:00:15.778658 20582 rpcz_store.cc:276] 0129 03:00:13.880082 (+ 0us) service_pool.cc:168] Inserting onto call queue
0129 03:00:15.509817 (+1629735us) service_pool.cc:225] Handling call
0129 03:00:15.544174 (+ 34357us) raft_consensus.cc:1471] Updating replica for 0 ops
0129 03:00:15.774227 (+230053us) raft_consensus.cc:1697] Filling consensus response to leader.
0129 03:00:15.774500 (+ 273us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20260129 03:00:15.785076 20480 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.211.4:40439, user_credentials={real_user=slave}} blocked reactor thread for 212723us
I20260129 03:00:15.791370 20556 leader_election.cc:304] T 7a9a69cd82724b1faed1d10aa091fc4a P 1c50de89a5a845668696dcb28c3badd6 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1c50de89a5a845668696dcb28c3badd6; no voters: 197b880fa39d432a9b472845876bfaf0, 974b802141bd4f13beffef2aa03f88f9
W20260129 03:00:15.796326 21100 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.211.3:37735, user_credentials={real_user=slave}} blocked reactor thread for 103733us
I20260129 03:00:15.722035 23333 consensus_queue.cc:1048] T 0678b8b468bd43cdbc3e485900844821 P 974b802141bd4f13beffef2aa03f88f9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 7, Time since last communication: 0.055s
W20260129 03:00:15.816727 20556 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.211.2:34517, user_credentials={real_user=slave}} blocked reactor thread for 74467.6us
I20260129 03:00:15.817150 20480 leader_election.cc:304] T 505580c93c694bbb8ea96fdf9b17ab46 P 974b802141bd4f13beffef2aa03f88f9 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 974b802141bd4f13beffef2aa03f88f9; no voters: 1c50de89a5a845668696dcb28c3badd6, 7a9daf5d42a84615a6383cfec9a10608
I20260129 03:00:15.817800 20556 leader_election.cc:304] T 368953c9102643b2a21f99af36b51905 P 1c50de89a5a845668696dcb28c3badd6 [CANDIDATE]: Term 8 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1c50de89a5a845668696dcb28c3badd6; no voters: 197b880fa39d432a9b472845876bfaf0, 974b802141bd4f13beffef2aa03f88f9
W20260129 03:00:15.819214 20407 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.211.2:34517, user_credentials={real_user=slave}} blocked reactor thread for 58628.1us
W20260129 03:00:15.820307 20554 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.211.4:40439, user_credentials={real_user=slave}} blocked reactor thread for 188737us
I20260129 03:00:15.818475 23601 raft_consensus.cc:2749] T 7a9a69cd82724b1faed1d10aa091fc4a P 1c50de89a5a845668696dcb28c3badd6 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260129 03:00:15.827673 23604 raft_consensus.cc:2749] T 368953c9102643b2a21f99af36b51905 P 1c50de89a5a845668696dcb28c3badd6 [term 8 FOLLOWER]: Leader election lost for term 8. Reason: could not achieve majority
W20260129 03:00:15.831701 20434 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.502s user 0.107s sys 0.189s
W20260129 03:00:15.832743 20434 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.503s user 0.107s sys 0.189s
I20260129 03:00:15.835451 20405 leader_election.cc:304] T 4e3805ffa70242e8ade66da260dddf66 P 197b880fa39d432a9b472845876bfaf0 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 197b880fa39d432a9b472845876bfaf0; no voters: 1c50de89a5a845668696dcb28c3badd6, 7a9daf5d42a84615a6383cfec9a10608
I20260129 03:00:15.764816 23450 consensus_queue.cc:1059] T 8055232cf0004d40a85f3bf983fe00a9 P 974b802141bd4f13beffef2aa03f88f9 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 }, Status: INVALID_TERM, Last received: 0.0, Next index: 3, Last known committed idx: 1, Time since last communication: 0.056s
I20260129 03:00:15.879292 20479 leader_election.cc:304] T febffe4a8b394368b69b1d164308fb9a P 974b802141bd4f13beffef2aa03f88f9 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 197b880fa39d432a9b472845876bfaf0, 974b802141bd4f13beffef2aa03f88f9; no voters:
I20260129 03:00:15.775333 23475 raft_consensus.cc:515] T a1a39dcd328e4268ae41b988c66b4a75 P 197b880fa39d432a9b472845876bfaf0 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } } peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } }
I20260129 03:00:15.841957 23362 raft_consensus.cc:493] T a644c02b16af47679824ccd060ac10fd P 7a9daf5d42a84615a6383cfec9a10608 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 1c50de89a5a845668696dcb28c3badd6)
I20260129 03:00:15.908052 20583 raft_consensus.cc:1240] T 75f45538dc704d9a8ec4ff8ad9f3ea5b P 1c50de89a5a845668696dcb28c3badd6 [term 5 FOLLOWER]: Rejecting Update request from peer 7a9daf5d42a84615a6383cfec9a10608 for earlier term 4. Current term is 5. Ops: []
I20260129 03:00:15.834619 22989 leader_election.cc:290] T 724a236524cb4e7ba3428880681d6cc2 P 974b802141bd4f13beffef2aa03f88f9 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 197b880fa39d432a9b472845876bfaf0 (127.18.211.1:42027), 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3:37735)
I20260129 03:00:15.920253 23362 raft_consensus.cc:515] T a644c02b16af47679824ccd060ac10fd P 7a9daf5d42a84615a6383cfec9a10608 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } }
W20260129 03:00:15.921222 23564 consensus_meta.cc:287] T 4b1876f9a70b49edb3aa74105e66d3b3 P 1c50de89a5a845668696dcb28c3badd6: Time spent flushing consensus metadata: real 0.696s user 0.007s sys 0.008s
I20260129 03:00:15.921766 23564 raft_consensus.cc:515] T 4b1876f9a70b49edb3aa74105e66d3b3 P 1c50de89a5a845668696dcb28c3badd6 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 03:00:15.923113 23362 leader_election.cc:290] T a644c02b16af47679824ccd060ac10fd P 7a9daf5d42a84615a6383cfec9a10608 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 974b802141bd4f13beffef2aa03f88f9 (127.18.211.2:34517), 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3:37735)
W20260129 03:00:15.924417 20556 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.211.2:34517, user_credentials={real_user=slave}} blocked reactor thread for 67757.2us
I20260129 03:00:15.923940 23427 raft_consensus.cc:2749] T 505580c93c694bbb8ea96fdf9b17ab46 P 974b802141bd4f13beffef2aa03f88f9 [term 8 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260129 03:00:15.929312 23602 raft_consensus.cc:2804] T c2633a6fc34c44579de348d208b0c01e P 1c50de89a5a845668696dcb28c3badd6 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260129 03:00:15.929636 23428 consensus_queue.cc:1048] T 2ac659d732314e1288849f08fe5e074e P 197b880fa39d432a9b472845876bfaf0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 5, Time since last communication: 0.001s
I20260129 03:00:15.924165 20336 catalog_manager.cc:5654] T 4e3805ffa70242e8ade66da260dddf66 P 1c50de89a5a845668696dcb28c3badd6 reported cstate change: term changed from 2 to 5, leader changed from 7a9daf5d42a84615a6383cfec9a10608 (127.18.211.4) to 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3). New cstate: current_term: 5 leader_uuid: "1c50de89a5a845668696dcb28c3badd6" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20260129 03:00:15.936493 23304 consensus_queue.cc:1048] T 505580c93c694bbb8ea96fdf9b17ab46 P 1c50de89a5a845668696dcb28c3badd6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 7, Time since last communication: 0.001s
W20260129 03:00:15.937347 20404 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260129 03:00:15.937521 23224 raft_consensus.cc:2749] T 4e3805ffa70242e8ade66da260dddf66 P 197b880fa39d432a9b472845876bfaf0 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260129 03:00:15.834589 23531 consensus_queue.cc:1059] T 7d9494f5584d4d5db8b9adf1c8fc888b P 1c50de89a5a845668696dcb28c3badd6 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 }, Status: INVALID_TERM, Last received: 4.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.339s
I20260129 03:00:15.939273 23531 raft_consensus.cc:3055] T 7d9494f5584d4d5db8b9adf1c8fc888b P 1c50de89a5a845668696dcb28c3badd6 [term 4 LEADER]: Stepping down as leader of term 4
I20260129 03:00:15.939759 23531 raft_consensus.cc:740] T 7d9494f5584d4d5db8b9adf1c8fc888b P 1c50de89a5a845668696dcb28c3badd6 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 1c50de89a5a845668696dcb28c3badd6, State: Running, Role: LEADER
I20260129 03:00:15.941097 23046 raft_consensus.cc:2804] T febffe4a8b394368b69b1d164308fb9a P 974b802141bd4f13beffef2aa03f88f9 [term 6 FOLLOWER]: Leader election won for term 6
I20260129 03:00:15.940869 23531 consensus_queue.cc:260] T 7d9494f5584d4d5db8b9adf1c8fc888b P 1c50de89a5a845668696dcb28c3badd6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 03:00:15.941625 23046 raft_consensus.cc:697] T febffe4a8b394368b69b1d164308fb9a P 974b802141bd4f13beffef2aa03f88f9 [term 6 LEADER]: Becoming Leader. State: Replica: 974b802141bd4f13beffef2aa03f88f9, State: Running, Role: LEADER
I20260129 03:00:15.942493 23531 raft_consensus.cc:3060] T 7d9494f5584d4d5db8b9adf1c8fc888b P 1c50de89a5a845668696dcb28c3badd6 [term 4 FOLLOWER]: Advancing to term 5
I20260129 03:00:15.942720 23046 consensus_queue.cc:237] T febffe4a8b394368b69b1d164308fb9a P 974b802141bd4f13beffef2aa03f88f9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 4.6, Last appended by leader: 6, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } }
I20260129 03:00:15.945931 20589 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35752 (request call id 1746) took 2079 ms. Trace:
I20260129 03:00:15.947979 23602 raft_consensus.cc:493] T c2633a6fc34c44579de348d208b0c01e P 1c50de89a5a845668696dcb28c3badd6 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260129 03:00:15.948385 23579 log.cc:927] Time spent T a0fcc58d6fb6473cafbd9d34d5195269 P 974b802141bd4f13beffef2aa03f88f9: Append to log took a long time: real 0.382s user 0.001s sys 0.000s
I20260129 03:00:15.948540 23602 raft_consensus.cc:3060] T c2633a6fc34c44579de348d208b0c01e P 1c50de89a5a845668696dcb28c3badd6 [term 5 FOLLOWER]: Advancing to term 6
I20260129 03:00:15.949388 20582 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:48526 (request call id 409) took 1246 ms. Trace:
I20260129 03:00:15.949568 23475 leader_election.cc:290] T a1a39dcd328e4268ae41b988c66b4a75 P 197b880fa39d432a9b472845876bfaf0 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 974b802141bd4f13beffef2aa03f88f9 (127.18.211.2:34517), 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3:37735)
I20260129 03:00:15.949817 20582 rpcz_store.cc:276] 0129 03:00:14.702578 (+ 0us) service_pool.cc:168] Inserting onto call queue
0129 03:00:15.784297 (+1081719us) service_pool.cc:225] Handling call
0129 03:00:15.857787 (+ 73490us) raft_consensus.cc:1471] Updating replica for 0 ops
0129 03:00:15.949004 (+ 91217us) raft_consensus.cc:1534] Early marking committed up to index 0
0129 03:00:15.949031 (+ 27us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0129 03:00:15.949090 (+ 59us) raft_consensus.cc:1645] Marking committed up to 5
0129 03:00:15.949140 (+ 50us) raft_consensus.cc:1697] Filling consensus response to leader.
0129 03:00:15.949199 (+ 59us) raft_consensus.cc:1691] UpdateReplicas() finished
0129 03:00:15.949313 (+ 114us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20260129 03:00:15.938366 21101 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260129 03:00:15.952209 20581 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:48526 (request call id 407) took 1213 ms. Trace:
I20260129 03:00:15.952705 20581 rpcz_store.cc:276] 0129 03:00:14.701966 (+ 0us) service_pool.cc:168] Inserting onto call queue
0129 03:00:15.736021 (+1034055us) service_pool.cc:225] Handling call
0129 03:00:15.821015 (+ 84994us) raft_consensus.cc:1471] Updating replica for 0 ops
0129 03:00:15.902444 (+ 81429us) raft_consensus.cc:1534] Early marking committed up to index 0
0129 03:00:15.902478 (+ 34us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0129 03:00:15.902536 (+ 58us) raft_consensus.cc:1645] Marking committed up to 6
0129 03:00:15.914462 (+ 11926us) raft_consensus.cc:1697] Filling consensus response to leader.
0129 03:00:15.914551 (+ 89us) raft_consensus.cc:1691] UpdateReplicas() finished
0129 03:00:15.915225 (+ 674us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260129 03:00:15.954053 20583 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:48526 (request call id 408) took 1213 ms. Trace:
I20260129 03:00:15.954530 20583 rpcz_store.cc:276] 0129 03:00:14.702307 (+ 0us) service_pool.cc:168] Inserting onto call queue
0129 03:00:15.729845 (+1027538us) service_pool.cc:225] Handling call
0129 03:00:15.821182 (+ 91337us) raft_consensus.cc:1471] Updating replica for 0 ops
0129 03:00:15.915378 (+ 94196us) raft_consensus.cc:1697] Filling consensus response to leader.
0129 03:00:15.915589 (+ 211us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260129 03:00:15.962267 20584 raft_consensus.cc:1240] T 75b0b6e82ddc4f92b1fe1d94f9a0b73b P 1c50de89a5a845668696dcb28c3badd6 [term 5 FOLLOWER]: Rejecting Update request from peer 7a9daf5d42a84615a6383cfec9a10608 for earlier term 4. Current term is 5. Ops: []
I20260129 03:00:15.963246 20584 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:48526 (request call id 405) took 1261 ms. Trace:
I20260129 03:00:15.963666 20584 rpcz_store.cc:276] 0129 03:00:14.701402 (+ 0us) service_pool.cc:168] Inserting onto call queue
0129 03:00:15.747301 (+1045899us) service_pool.cc:225] Handling call
0129 03:00:15.837550 (+ 90249us) raft_consensus.cc:1471] Updating replica for 0 ops
0129 03:00:15.962909 (+125359us) raft_consensus.cc:1697] Filling consensus response to leader.
0129 03:00:15.963177 (+ 268us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260129 03:00:15.834852 22996 raft_consensus.cc:515] T 17d6431cc77a491e8464d68db4081a51 P 1c50de89a5a845668696dcb28c3badd6 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 03:00:15.994263 20336 catalog_manager.cc:5654] T 75f45538dc704d9a8ec4ff8ad9f3ea5b P 1c50de89a5a845668696dcb28c3badd6 reported cstate change: term changed from 2 to 4. New cstate: current_term: 4 leader_uuid: "7a9daf5d42a84615a6383cfec9a10608" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } } }
I20260129 03:00:15.946431 20589 rpcz_store.cc:276] 0129 03:00:13.866872 (+ 0us) service_pool.cc:168] Inserting onto call queue
0129 03:00:15.135285 (+1268413us) service_pool.cc:225] Handling call
0129 03:00:15.135838 (+ 553us) raft_consensus.cc:1471] Updating replica for 1 ops
0129 03:00:15.158457 (+ 22619us) raft_consensus.cc:1534] Early marking committed up to index 0
0129 03:00:15.158491 (+ 34us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0129 03:00:15.159094 (+ 603us) log.cc:844] Serialized 26 byte log entry
0129 03:00:15.335755 (+176661us) raft_consensus.cc:1645] Marking committed up to 5
0129 03:00:15.363812 (+ 28057us) log.cc:844] Serialized 14 byte log entry
0129 03:00:15.375013 (+ 11201us) raft_consensus.cc:1697] Filling consensus response to leader.
0129 03:00:15.375109 (+ 96us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0129 03:00:15.945654 (+570545us) raft_consensus.cc:1685] finished
0129 03:00:15.945693 (+ 39us) raft_consensus.cc:1691] UpdateReplicas() finished
0129 03:00:15.945860 (+ 167us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":4900,"prepare.queue_time_us":28126,"prepare.run_cpu_time_us":58,"prepare.run_wall_time_us":2321,"raft.queue_time_us":6373,"raft.run_cpu_time_us":293,"raft.run_wall_time_us":4887,"thread_start_us":175128,"threads_started":1,"wal-append.queue_time_us":779968}
I20260129 03:00:16.005319 20582 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a644c02b16af47679824ccd060ac10fd" candidate_uuid: "7a9daf5d42a84615a6383cfec9a10608" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "1c50de89a5a845668696dcb28c3badd6" is_pre_election: true
I20260129 03:00:16.017517 20336 catalog_manager.cc:5654] T a5a05fbcbbcf4916bbc01fd1a5b5e90d P 1c50de89a5a845668696dcb28c3badd6 reported cstate change: term changed from 5 to 7. New cstate: current_term: 7 leader_uuid: "974b802141bd4f13beffef2aa03f88f9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } } }
I20260129 03:00:16.020555 20556 leader_election.cc:304] T e4d18c35b0a7493e85d0a4b92956a48f P 1c50de89a5a845668696dcb28c3badd6 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1c50de89a5a845668696dcb28c3badd6; no voters: 197b880fa39d432a9b472845876bfaf0, 974b802141bd4f13beffef2aa03f88f9
I20260129 03:00:16.026576 20587 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35752 (request call id 1745) took 2913 ms. Trace:
I20260129 03:00:16.027009 20581 raft_consensus.cc:1240] T 2e58e9a5a8aa4cecafb464addb4f35ed P 1c50de89a5a845668696dcb28c3badd6 [term 5 LEADER]: Rejecting Update request from peer 7a9daf5d42a84615a6383cfec9a10608 for earlier term 4. Current term is 5. Ops: []
I20260129 03:00:16.027679 20581 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:48526 (request call id 410) took 1324 ms. Trace:
I20260129 03:00:16.028317 20436 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4b1876f9a70b49edb3aa74105e66d3b3" candidate_uuid: "1c50de89a5a845668696dcb28c3badd6" candidate_term: 7 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "197b880fa39d432a9b472845876bfaf0"
I20260129 03:00:16.029078 20436 raft_consensus.cc:3060] T 4b1876f9a70b49edb3aa74105e66d3b3 P 197b880fa39d432a9b472845876bfaf0 [term 6 FOLLOWER]: Advancing to term 7
I20260129 03:00:16.030823 23480 raft_consensus.cc:493] T dc2f4cb752cf4d59b5f6bc689f5d8104 P 7a9daf5d42a84615a6383cfec9a10608 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260129 03:00:16.031482 23480 raft_consensus.cc:515] T dc2f4cb752cf4d59b5f6bc689f5d8104 P 7a9daf5d42a84615a6383cfec9a10608 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } }
I20260129 03:00:16.033493 20583 raft_consensus.cc:1240] T add8d4923d1a489e86882c3a25692b95 P 1c50de89a5a845668696dcb28c3badd6 [term 8 FOLLOWER]: Rejecting Update request from peer 7a9daf5d42a84615a6383cfec9a10608 for earlier term 7. Current term is 8. Ops: [7.6-7.6]
I20260129 03:00:16.033712 23480 leader_election.cc:290] T dc2f4cb752cf4d59b5f6bc689f5d8104 P 7a9daf5d42a84615a6383cfec9a10608 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 197b880fa39d432a9b472845876bfaf0 (127.18.211.1:42027), 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3:37735)
I20260129 03:00:16.034260 20583 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:48526 (request call id 412) took 1330 ms. Trace:
I20260129 03:00:16.034639 20583 rpcz_store.cc:276] 0129 03:00:14.703542 (+ 0us) service_pool.cc:168] Inserting onto call queue
0129 03:00:15.955394 (+1251852us) service_pool.cc:225] Handling call
0129 03:00:16.033129 (+ 77735us) raft_consensus.cc:1471] Updating replica for 1 ops
0129 03:00:16.034006 (+ 877us) raft_consensus.cc:1697] Filling consensus response to leader.
0129 03:00:16.034206 (+ 200us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20260129 03:00:16.037986 22645 consensus_meta.cc:287] T 17d6431cc77a491e8464d68db4081a51 P 197b880fa39d432a9b472845876bfaf0: Time spent flushing consensus metadata: real 0.819s user 0.011s sys 0.000s
I20260129 03:00:16.041358 22996 leader_election.cc:290] T 17d6431cc77a491e8464d68db4081a51 P 1c50de89a5a845668696dcb28c3badd6 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 197b880fa39d432a9b472845876bfaf0 (127.18.211.1:42027), 974b802141bd4f13beffef2aa03f88f9 (127.18.211.2:34517)
I20260129 03:00:16.042470 23514 leader_election.cc:290] T 4f8f3a38f6e24ba4857875181f447442 P 974b802141bd4f13beffef2aa03f88f9 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 197b880fa39d432a9b472845876bfaf0 (127.18.211.1:42027), 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3:37735)
I20260129 03:00:16.001075 23365 raft_consensus.cc:493] T a609ac02fae84725ab743f95f51d33ce P 7a9daf5d42a84615a6383cfec9a10608 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260129 03:00:16.044301 23365 raft_consensus.cc:515] T a609ac02fae84725ab743f95f51d33ce P 7a9daf5d42a84615a6383cfec9a10608 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } }
I20260129 03:00:16.046454 23365 leader_election.cc:290] T a609ac02fae84725ab743f95f51d33ce P 7a9daf5d42a84615a6383cfec9a10608 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 197b880fa39d432a9b472845876bfaf0 (127.18.211.1:42027), 974b802141bd4f13beffef2aa03f88f9 (127.18.211.2:34517)
W20260129 03:00:16.048677 20480 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.211.4:40439, user_credentials={real_user=slave}} blocked reactor thread for 231590us
W20260129 03:00:16.048914 23568 consensus_meta.cc:287] T cb0b3f259ebc4310927338a8740e87b8 P 7a9daf5d42a84615a6383cfec9a10608: Time spent flushing consensus metadata: real 0.852s user 0.009s sys 0.004s
I20260129 03:00:16.049335 23568 raft_consensus.cc:515] T cb0b3f259ebc4310927338a8740e87b8 P 7a9daf5d42a84615a6383cfec9a10608 [term 9 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } }
I20260129 03:00:16.049718 20480 leader_election.cc:304] T a609ac02fae84725ab743f95f51d33ce P 974b802141bd4f13beffef2aa03f88f9 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 974b802141bd4f13beffef2aa03f88f9; no voters: 197b880fa39d432a9b472845876bfaf0, 7a9daf5d42a84615a6383cfec9a10608
I20260129 03:00:16.051239 20480 leader_election.cc:304] T 1299723a4476420ab02fd8bfd7aa0645 P 974b802141bd4f13beffef2aa03f88f9 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 974b802141bd4f13beffef2aa03f88f9; no voters: 197b880fa39d432a9b472845876bfaf0, 7a9daf5d42a84615a6383cfec9a10608
I20260129 03:00:16.051939 20441 raft_consensus.cc:1275] T 2e58e9a5a8aa4cecafb464addb4f35ed P 197b880fa39d432a9b472845876bfaf0 [term 5 FOLLOWER]: Refusing update from remote peer 1c50de89a5a845668696dcb28c3badd6: Log matching property violated. Preceding OpId in replica: term: 4 index: 6. Preceding OpId from leader: term: 5 index: 7. (index mismatch)
W20260129 03:00:16.052947 20554 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.211.4:40439, user_credentials={real_user=slave}} blocked reactor thread for 168557us
I20260129 03:00:16.053766 20554 leader_election.cc:304] T 1299723a4476420ab02fd8bfd7aa0645 P 1c50de89a5a845668696dcb28c3badd6 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1c50de89a5a845668696dcb28c3badd6; no voters: 197b880fa39d432a9b472845876bfaf0, 7a9daf5d42a84615a6383cfec9a10608
I20260129 03:00:16.027031 20587 rpcz_store.cc:276] 0129 03:00:13.113584 (+ 0us) service_pool.cc:168] Inserting onto call queue
0129 03:00:15.142131 (+2028547us) service_pool.cc:225] Handling call
0129 03:00:15.168720 (+ 26589us) raft_consensus.cc:1471] Updating replica for 1 ops
0129 03:00:15.169485 (+ 765us) raft_consensus.cc:1534] Early marking committed up to index 0
0129 03:00:15.169526 (+ 41us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0129 03:00:15.170210 (+ 684us) log.cc:844] Serialized 26 byte log entry
0129 03:00:15.353456 (+183246us) raft_consensus.cc:1645] Marking committed up to 4
0129 03:00:15.353491 (+ 35us) raft_consensus.cc:1697] Filling consensus response to leader.
0129 03:00:15.353551 (+ 60us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0129 03:00:16.026324 (+672773us) raft_consensus.cc:1685] finished
0129 03:00:16.026347 (+ 23us) raft_consensus.cc:1691] UpdateReplicas() finished
0129 03:00:16.026509 (+ 162us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":8276,"raft.run_cpu_time_us":116,"raft.run_wall_time_us":116,"thread_start_us":182818,"threads_started":1,"wal-append.queue_time_us":830424}
I20260129 03:00:16.056349 20585 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:48526 (request call id 403) took 3535 ms. Trace:
I20260129 03:00:16.056795 20585 rpcz_store.cc:276] 0129 03:00:12.520889 (+ 0us) service_pool.cc:168] Inserting onto call queue
0129 03:00:14.718529 (+2197640us) service_pool.cc:225] Handling call
0129 03:00:14.718830 (+ 301us) raft_consensus.cc:1471] Updating replica for 1 ops
0129 03:00:15.329497 (+610667us) raft_consensus.cc:1534] Early marking committed up to index 0
0129 03:00:15.329558 (+ 61us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0129 03:00:15.330181 (+ 623us) log.cc:844] Serialized 26 byte log entry
0129 03:00:15.728311 (+398130us) raft_consensus.cc:1645] Marking committed up to 5
0129 03:00:15.728366 (+ 55us) raft_consensus.cc:1697] Filling consensus response to leader.
0129 03:00:15.745573 (+ 17207us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0129 03:00:16.031670 (+286097us) raft_consensus.cc:1685] finished
0129 03:00:16.031689 (+ 19us) raft_consensus.cc:1691] UpdateReplicas() finished
0129 03:00:16.056167 (+ 24478us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":308940,"raft.queue_time_us":324288,"raft.run_cpu_time_us":185,"raft.run_wall_time_us":184,"thread_start_us":625861,"threads_started":2,"wal-append.queue_time_us":699332}
I20260129 03:00:16.028045 20581 rpcz_store.cc:276] 0129 03:00:14.702870 (+ 0us) service_pool.cc:168] Inserting onto call queue
0129 03:00:15.953802 (+1250932us) service_pool.cc:225] Handling call
0129 03:00:16.026815 (+ 73013us) raft_consensus.cc:1471] Updating replica for 0 ops
0129 03:00:16.027439 (+ 624us) raft_consensus.cc:1697] Filling consensus response to leader.
0129 03:00:16.027625 (+ 186us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260129 03:00:16.055406 20336 catalog_manager.cc:5654] T 76f713c3331540d2a5b76164103e9eb1 P 1c50de89a5a845668696dcb28c3badd6 reported cstate change: term changed from 2 to 5. New cstate: current_term: 5 leader_uuid: "1c50de89a5a845668696dcb28c3badd6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } health_report { overall_health: HEALTHY } } }
I20260129 03:00:16.051575 23568 leader_election.cc:290] T cb0b3f259ebc4310927338a8740e87b8 P 7a9daf5d42a84615a6383cfec9a10608 [CANDIDATE]: Term 9 election: Requested vote from peers 974b802141bd4f13beffef2aa03f88f9 (127.18.211.2:34517), 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3:37735)
I20260129 03:00:16.089972 20441 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "17d6431cc77a491e8464d68db4081a51" candidate_uuid: "1c50de89a5a845668696dcb28c3badd6" candidate_term: 6 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "197b880fa39d432a9b472845876bfaf0" is_pre_election: true
W20260129 03:00:16.085023 20586 consensus_meta.cc:287] T 4f8f3a38f6e24ba4857875181f447442 P 1c50de89a5a845668696dcb28c3badd6: Time spent flushing consensus metadata: real 0.702s user 0.010s sys 0.000s
W20260129 03:00:16.093796 20482 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260129 03:00:16.095633 20482 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260129 03:00:16.084669 20554 leader_election.cc:304] T 45cc5f3d2688486d9b6dbe7e666dedde P 1c50de89a5a845668696dcb28c3badd6 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1c50de89a5a845668696dcb28c3badd6; no voters: 7a9daf5d42a84615a6383cfec9a10608, 974b802141bd4f13beffef2aa03f88f9
I20260129 03:00:16.091465 20586 raft_consensus.cc:515] T 4f8f3a38f6e24ba4857875181f447442 P 1c50de89a5a845668696dcb28c3badd6 [term 7 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 03:00:16.022337 23475 raft_consensus.cc:493] T 76f713c3331540d2a5b76164103e9eb1 P 197b880fa39d432a9b472845876bfaf0 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 974b802141bd4f13beffef2aa03f88f9)
I20260129 03:00:16.106958 20441 raft_consensus.cc:2410] T 17d6431cc77a491e8464d68db4081a51 P 197b880fa39d432a9b472845876bfaf0 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1c50de89a5a845668696dcb28c3badd6 for term 6 because replica has last-logged OpId of term: 5 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 4 index: 2.
I20260129 03:00:16.108012 23304 consensus_queue.cc:1048] T 2e58e9a5a8aa4cecafb464addb4f35ed P 1c50de89a5a845668696dcb28c3badd6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.005s
I20260129 03:00:16.109179 20336 catalog_manager.cc:5654] T 7d9494f5584d4d5db8b9adf1c8fc888b P 1c50de89a5a845668696dcb28c3badd6 reported cstate change: term changed from 2 to 4. New cstate: current_term: 4 leader_uuid: "1c50de89a5a845668696dcb28c3badd6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } health_report { overall_health: HEALTHY } } }
I20260129 03:00:16.133441 20586 leader_election.cc:290] T 4f8f3a38f6e24ba4857875181f447442 P 1c50de89a5a845668696dcb28c3badd6 [CANDIDATE]: Term 7 election: Requested vote from peers 197b880fa39d432a9b472845876bfaf0 (127.18.211.1:42027), 974b802141bd4f13beffef2aa03f88f9 (127.18.211.2:34517)
I20260129 03:00:16.090765 22996 raft_consensus.cc:2749] T e4d18c35b0a7493e85d0a4b92956a48f P 1c50de89a5a845668696dcb28c3badd6 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260129 03:00:16.066643 23333 raft_consensus.cc:2749] T a609ac02fae84725ab743f95f51d33ce P 974b802141bd4f13beffef2aa03f88f9 [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20260129 03:00:16.084316 23262 raft_consensus.cc:2749] T 1299723a4476420ab02fd8bfd7aa0645 P 1c50de89a5a845668696dcb28c3badd6 [term 10 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20260129 03:00:16.088528 23564 leader_election.cc:290] T 4b1876f9a70b49edb3aa74105e66d3b3 P 1c50de89a5a845668696dcb28c3badd6 [CANDIDATE]: Term 7 election: Requested vote from peers 197b880fa39d432a9b472845876bfaf0 (127.18.211.1:42027), 974b802141bd4f13beffef2aa03f88f9 (127.18.211.2:34517)
I20260129 03:00:16.095814 23514 raft_consensus.cc:2749] T 1299723a4476420ab02fd8bfd7aa0645 P 974b802141bd4f13beffef2aa03f88f9 [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20260129 03:00:16.138967 20583 raft_consensus.cc:3055] T 505580c93c694bbb8ea96fdf9b17ab46 P 1c50de89a5a845668696dcb28c3badd6 [term 7 LEADER]: Stepping down as leader of term 7
I20260129 03:00:16.148686 20583 raft_consensus.cc:740] T 505580c93c694bbb8ea96fdf9b17ab46 P 1c50de89a5a845668696dcb28c3badd6 [term 7 LEADER]: Becoming Follower/Learner. State: Replica: 1c50de89a5a845668696dcb28c3badd6, State: Running, Role: LEADER
I20260129 03:00:16.154174 20586 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RunLeaderElection from 127.0.0.1:35752 (request call id 1747) took 2267 ms. Trace:
W20260129 03:00:16.156847 20480 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260129 03:00:16.158247 20581 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "cb0b3f259ebc4310927338a8740e87b8" candidate_uuid: "7a9daf5d42a84615a6383cfec9a10608" candidate_term: 9 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "1c50de89a5a845668696dcb28c3badd6"
I20260129 03:00:16.158990 20581 raft_consensus.cc:3060] T cb0b3f259ebc4310927338a8740e87b8 P 1c50de89a5a845668696dcb28c3badd6 [term 8 FOLLOWER]: Advancing to term 9
I20260129 03:00:16.087838 23365 consensus_queue.cc:1059] T 75b0b6e82ddc4f92b1fe1d94f9a0b73b P 7a9daf5d42a84615a6383cfec9a10608 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 }, Status: INVALID_TERM, Last received: 4.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.052s
I20260129 03:00:16.153637 20585 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "dc2f4cb752cf4d59b5f6bc689f5d8104" candidate_uuid: "7a9daf5d42a84615a6383cfec9a10608" candidate_term: 7 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "1c50de89a5a845668696dcb28c3badd6" is_pre_election: true
I20260129 03:00:16.154675 20586 rpcz_store.cc:276] 0129 03:00:13.867228 (+ 0us) service_pool.cc:168] Inserting onto call queue
0129 03:00:15.144399 (+1277171us) service_pool.cc:225] Handling call
0129 03:00:16.134059 (+989660us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260129 03:00:16.161728 21131 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3ccac82a01a2445fa5b0ce6c0fcf2295" candidate_uuid: "974b802141bd4f13beffef2aa03f88f9" candidate_term: 5 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "7a9daf5d42a84615a6383cfec9a10608" is_pre_election: true
I20260129 03:00:16.162642 21132 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2ac659d732314e1288849f08fe5e074e" candidate_uuid: "974b802141bd4f13beffef2aa03f88f9" candidate_term: 7 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "7a9daf5d42a84615a6383cfec9a10608" is_pre_election: true
I20260129 03:00:16.165997 21132 raft_consensus.cc:2410] T 2ac659d732314e1288849f08fe5e074e P 7a9daf5d42a84615a6383cfec9a10608 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 974b802141bd4f13beffef2aa03f88f9 for term 7 because replica has last-logged OpId of term: 5 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 5.
I20260129 03:00:16.132853 23456 raft_consensus.cc:2749] T 45cc5f3d2688486d9b6dbe7e666dedde P 1c50de89a5a845668696dcb28c3badd6 [term 6 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260129 03:00:16.096192 23480 consensus_queue.cc:1059] T add8d4923d1a489e86882c3a25692b95 P 7a9daf5d42a84615a6383cfec9a10608 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.053s
I20260129 03:00:16.151556 23109 consensus_queue.cc:1059] T 75f45538dc704d9a8ec4ff8ad9f3ea5b P 7a9daf5d42a84615a6383cfec9a10608 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 }, Status: INVALID_TERM, Last received: 4.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.084s
W20260129 03:00:16.173313 20553 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260129 03:00:16.175560 20554 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.211.4:40439, user_credentials={real_user=slave}} blocked reactor thread for 106561us
I20260129 03:00:16.199987 20510 raft_consensus.cc:1275] T 505580c93c694bbb8ea96fdf9b17ab46 P 974b802141bd4f13beffef2aa03f88f9 [term 8 FOLLOWER]: Refusing update from remote peer 7a9daf5d42a84615a6383cfec9a10608: Log matching property violated. Preceding OpId in replica: term: 5 index: 7. Preceding OpId from leader: term: 8 index: 9. (index mismatch)
I20260129 03:00:16.200258 20507 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a644c02b16af47679824ccd060ac10fd" candidate_uuid: "7a9daf5d42a84615a6383cfec9a10608" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "974b802141bd4f13beffef2aa03f88f9" is_pre_election: true
I20260129 03:00:16.201069 20507 raft_consensus.cc:2468] T a644c02b16af47679824ccd060ac10fd P 974b802141bd4f13beffef2aa03f88f9 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7a9daf5d42a84615a6383cfec9a10608 in term 5.
I20260129 03:00:16.162227 23427 raft_consensus.cc:493] T a5a05fbcbbcf4916bbc01fd1a5b5e90d P 974b802141bd4f13beffef2aa03f88f9 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260129 03:00:16.204206 23427 raft_consensus.cc:515] T a5a05fbcbbcf4916bbc01fd1a5b5e90d P 974b802141bd4f13beffef2aa03f88f9 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 03:00:16.206277 23427 leader_election.cc:290] T a5a05fbcbbcf4916bbc01fd1a5b5e90d P 974b802141bd4f13beffef2aa03f88f9 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 197b880fa39d432a9b472845876bfaf0 (127.18.211.1:42027), 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3:37735)
I20260129 03:00:16.206817 23427 raft_consensus.cc:493] T a1a39dcd328e4268ae41b988c66b4a75 P 974b802141bd4f13beffef2aa03f88f9 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260129 03:00:16.207360 23427 raft_consensus.cc:515] T a1a39dcd328e4268ae41b988c66b4a75 P 974b802141bd4f13beffef2aa03f88f9 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } } peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } }
I20260129 03:00:16.209301 23362 raft_consensus.cc:3055] T 75f45538dc704d9a8ec4ff8ad9f3ea5b P 7a9daf5d42a84615a6383cfec9a10608 [term 4 LEADER]: Stepping down as leader of term 4
I20260129 03:00:16.209812 23362 raft_consensus.cc:740] T 75f45538dc704d9a8ec4ff8ad9f3ea5b P 7a9daf5d42a84615a6383cfec9a10608 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 7a9daf5d42a84615a6383cfec9a10608, State: Running, Role: LEADER
I20260129 03:00:16.210075 23427 leader_election.cc:290] T a1a39dcd328e4268ae41b988c66b4a75 P 974b802141bd4f13beffef2aa03f88f9 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3:37735), 197b880fa39d432a9b472845876bfaf0 (127.18.211.1:42027)
I20260129 03:00:16.210718 23427 raft_consensus.cc:493] T 926a250fa5834e88b96ccecbefbf5086 P 974b802141bd4f13beffef2aa03f88f9 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 197b880fa39d432a9b472845876bfaf0)
W20260129 03:00:16.212182 21100 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.211.3:37735, user_credentials={real_user=slave}} blocked reactor thread for 60141.3us
I20260129 03:00:16.210186 20516 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "febffe4a8b394368b69b1d164308fb9a" candidate_uuid: "7a9daf5d42a84615a6383cfec9a10608" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "974b802141bd4f13beffef2aa03f88f9"
I20260129 03:00:16.153162 23555 raft_consensus.cc:493] T 3906b080a5b34a74b8b64128635e8997 P 974b802141bd4f13beffef2aa03f88f9 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260129 03:00:16.178443 20336 catalog_manager.cc:5654] T a0fcc58d6fb6473cafbd9d34d5195269 P 1c50de89a5a845668696dcb28c3badd6 reported cstate change: term changed from 2 to 3, leader changed from 197b880fa39d432a9b472845876bfaf0 (127.18.211.1) to 974b802141bd4f13beffef2aa03f88f9 (127.18.211.2). New cstate: current_term: 3 leader_uuid: "974b802141bd4f13beffef2aa03f88f9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } } }
W20260129 03:00:16.221001 20556 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.211.2:34517, user_credentials={real_user=slave}} blocked reactor thread for 245725us
I20260129 03:00:16.222010 20556 leader_election.cc:304] T 75be4bd6bd45474795fd472eff2953ac P 1c50de89a5a845668696dcb28c3badd6 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1c50de89a5a845668696dcb28c3badd6; no voters: 197b880fa39d432a9b472845876bfaf0, 974b802141bd4f13beffef2aa03f88f9
I20260129 03:00:16.222345 23109 raft_consensus.cc:3055] T 75b0b6e82ddc4f92b1fe1d94f9a0b73b P 7a9daf5d42a84615a6383cfec9a10608 [term 4 LEADER]: Stepping down as leader of term 4
I20260129 03:00:16.222851 23109 raft_consensus.cc:740] T 75b0b6e82ddc4f92b1fe1d94f9a0b73b P 7a9daf5d42a84615a6383cfec9a10608 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 7a9daf5d42a84615a6383cfec9a10608, State: Running, Role: LEADER
I20260129 03:00:16.213424 23485 raft_consensus.cc:493] T 3906b080a5b34a74b8b64128635e8997 P 197b880fa39d432a9b472845876bfaf0 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 974b802141bd4f13beffef2aa03f88f9)
I20260129 03:00:16.224136 23456 consensus_queue.cc:1048] T dc2f4cb752cf4d59b5f6bc689f5d8104 P 1c50de89a5a845668696dcb28c3badd6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.001s
I20260129 03:00:16.224064 23564 raft_consensus.cc:2749] T 75be4bd6bd45474795fd472eff2953ac P 1c50de89a5a845668696dcb28c3badd6 [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20260129 03:00:16.223795 23109 consensus_queue.cc:260] T 75b0b6e82ddc4f92b1fe1d94f9a0b73b P 7a9daf5d42a84615a6383cfec9a10608 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 4.7, Last appended by leader: 7, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } }
I20260129 03:00:16.225626 20441 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4f8f3a38f6e24ba4857875181f447442" candidate_uuid: "1c50de89a5a845668696dcb28c3badd6" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: true dest_uuid: "197b880fa39d432a9b472845876bfaf0"
W20260129 03:00:16.226078 23469 consensus_meta.cc:287] T c934d31efd3f4e5195605ddc354d14e1 P 197b880fa39d432a9b472845876bfaf0: Time spent flushing consensus metadata: real 0.681s user 0.014s sys 0.000s
I20260129 03:00:16.196560 20583 consensus_queue.cc:260] T 505580c93c694bbb8ea96fdf9b17ab46 P 1c50de89a5a845668696dcb28c3badd6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 7.9, Last appended by leader: 9, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } }
I20260129 03:00:16.226804 20441 raft_consensus.cc:2393] T 4f8f3a38f6e24ba4857875181f447442 P 197b880fa39d432a9b472845876bfaf0 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 1c50de89a5a845668696dcb28c3badd6 in current term 7: Already voted for candidate 197b880fa39d432a9b472845876bfaf0 in this term.
I20260129 03:00:16.226569 23469 raft_consensus.cc:515] T c934d31efd3f4e5195605ddc354d14e1 P 197b880fa39d432a9b472845876bfaf0 [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } }
I20260129 03:00:16.228891 23469 leader_election.cc:290] T c934d31efd3f4e5195605ddc354d14e1 P 197b880fa39d432a9b472845876bfaf0 [CANDIDATE]: Term 7 election: Requested vote from peers 974b802141bd4f13beffef2aa03f88f9 (127.18.211.2:34517), 7a9daf5d42a84615a6383cfec9a10608 (127.18.211.4:40439)
I20260129 03:00:16.229048 23109 raft_consensus.cc:3060] T 75b0b6e82ddc4f92b1fe1d94f9a0b73b P 7a9daf5d42a84615a6383cfec9a10608 [term 4 FOLLOWER]: Advancing to term 5
I20260129 03:00:16.166869 23475 raft_consensus.cc:515] T 76f713c3331540d2a5b76164103e9eb1 P 197b880fa39d432a9b472845876bfaf0 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 03:00:16.233273 20336 catalog_manager.cc:5654] T 1ce4ddf5b10e4ae8b193b13b2e5936ee P 1c50de89a5a845668696dcb28c3badd6 reported cstate change: term changed from 2 to 6. New cstate: current_term: 6 leader_uuid: "1c50de89a5a845668696dcb28c3badd6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } health_report { overall_health: HEALTHY } } }
I20260129 03:00:16.226981 20583 raft_consensus.cc:3060] T 505580c93c694bbb8ea96fdf9b17ab46 P 1c50de89a5a845668696dcb28c3badd6 [term 7 FOLLOWER]: Advancing to term 8
I20260129 03:00:16.224483 23485 raft_consensus.cc:515] T 3906b080a5b34a74b8b64128635e8997 P 197b880fa39d432a9b472845876bfaf0 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 03:00:16.235716 20516 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a609ac02fae84725ab743f95f51d33ce" candidate_uuid: "7a9daf5d42a84615a6383cfec9a10608" candidate_term: 10 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "974b802141bd4f13beffef2aa03f88f9" is_pre_election: true
I20260129 03:00:16.236829 20516 raft_consensus.cc:2410] T a609ac02fae84725ab743f95f51d33ce P 974b802141bd4f13beffef2aa03f88f9 [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 7a9daf5d42a84615a6383cfec9a10608 for term 10 because replica has last-logged OpId of term: 9 index: 9, which is greater than that of the candidate, which has last-logged OpId of term: 8 index: 8.
I20260129 03:00:16.211225 23427 raft_consensus.cc:515] T 926a250fa5834e88b96ccecbefbf5086 P 974b802141bd4f13beffef2aa03f88f9 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 03:00:16.240937 23456 raft_consensus.cc:493] T e33a417cf9c54cf5af73a4ee3aac4d3a P 1c50de89a5a845668696dcb28c3badd6 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260129 03:00:16.241484 23456 raft_consensus.cc:515] T e33a417cf9c54cf5af73a4ee3aac4d3a P 1c50de89a5a845668696dcb28c3badd6 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 03:00:16.231629 23475 leader_election.cc:290] T 76f713c3331540d2a5b76164103e9eb1 P 197b880fa39d432a9b472845876bfaf0 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 974b802141bd4f13beffef2aa03f88f9 (127.18.211.2:34517), 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3:37735)
I20260129 03:00:16.196040 23514 raft_consensus.cc:493] T 4b1876f9a70b49edb3aa74105e66d3b3 P 974b802141bd4f13beffef2aa03f88f9 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260129 03:00:16.216238 22645 raft_consensus.cc:493] T 42d27a28d89b409e8807ec0efd97af69 P 197b880fa39d432a9b472845876bfaf0 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 7a9daf5d42a84615a6383cfec9a10608)
W20260129 03:00:16.252888 20553 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260129 03:00:16.247484 20513 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "cb0b3f259ebc4310927338a8740e87b8" candidate_uuid: "7a9daf5d42a84615a6383cfec9a10608" candidate_term: 9 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "974b802141bd4f13beffef2aa03f88f9"
I20260129 03:00:16.243629 23456 leader_election.cc:290] T e33a417cf9c54cf5af73a4ee3aac4d3a P 1c50de89a5a845668696dcb28c3badd6 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 197b880fa39d432a9b472845876bfaf0 (127.18.211.1:42027), 974b802141bd4f13beffef2aa03f88f9 (127.18.211.2:34517)
I20260129 03:00:16.252744 20554 leader_election.cc:304] T cb0b3f259ebc4310927338a8740e87b8 P 1c50de89a5a845668696dcb28c3badd6 [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1c50de89a5a845668696dcb28c3badd6, 7a9daf5d42a84615a6383cfec9a10608; no voters: 974b802141bd4f13beffef2aa03f88f9
I20260129 03:00:16.253441 20513 raft_consensus.cc:3060] T cb0b3f259ebc4310927338a8740e87b8 P 974b802141bd4f13beffef2aa03f88f9 [term 8 FOLLOWER]: Advancing to term 9
I20260129 03:00:16.198005 23333 raft_consensus.cc:493] T cedce331e1584ac8b636c9b8a72b2ba0 P 974b802141bd4f13beffef2aa03f88f9 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 197b880fa39d432a9b472845876bfaf0)
I20260129 03:00:16.251114 20556 leader_election.cc:304] T c891d2bff4ce4568861bbc992e91fbb0 P 1c50de89a5a845668696dcb28c3badd6 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1c50de89a5a845668696dcb28c3badd6, 974b802141bd4f13beffef2aa03f88f9; no voters: 197b880fa39d432a9b472845876bfaf0
I20260129 03:00:16.254870 20441 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e33a417cf9c54cf5af73a4ee3aac4d3a" candidate_uuid: "1c50de89a5a845668696dcb28c3badd6" candidate_term: 8 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "197b880fa39d432a9b472845876bfaf0" is_pre_election: true
I20260129 03:00:16.257059 20581 raft_consensus.cc:2468] T cb0b3f259ebc4310927338a8740e87b8 P 1c50de89a5a845668696dcb28c3badd6 [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7a9daf5d42a84615a6383cfec9a10608 in term 9.
I20260129 03:00:16.257489 20441 raft_consensus.cc:2468] T e33a417cf9c54cf5af73a4ee3aac4d3a P 197b880fa39d432a9b472845876bfaf0 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1c50de89a5a845668696dcb28c3badd6 in term 7.
I20260129 03:00:16.211047 23362 consensus_queue.cc:260] T 75f45538dc704d9a8ec4ff8ad9f3ea5b P 7a9daf5d42a84615a6383cfec9a10608 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } }
I20260129 03:00:16.250586 23564 consensus_queue.cc:1048] T 36ff33be666a4c82954731691dbcf327 P 1c50de89a5a845668696dcb28c3badd6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.006s
I20260129 03:00:16.252158 21102 leader_election.cc:304] T a644c02b16af47679824ccd060ac10fd P 7a9daf5d42a84615a6383cfec9a10608 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 7a9daf5d42a84615a6383cfec9a10608, 974b802141bd4f13beffef2aa03f88f9; no voters: 1c50de89a5a845668696dcb28c3badd6
I20260129 03:00:16.258194 20336 catalog_manager.cc:5654] T 368953c9102643b2a21f99af36b51905 P 1c50de89a5a845668696dcb28c3badd6 reported cstate change: term changed from 2 to 6, leader changed from 197b880fa39d432a9b472845876bfaf0 (127.18.211.1) to 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3). New cstate: current_term: 6 leader_uuid: "1c50de89a5a845668696dcb28c3badd6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } health_report { overall_health: HEALTHY } } }
I20260129 03:00:16.261996 20553 leader_election.cc:304] T e33a417cf9c54cf5af73a4ee3aac4d3a P 1c50de89a5a845668696dcb28c3badd6 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 197b880fa39d432a9b472845876bfaf0, 1c50de89a5a845668696dcb28c3badd6; no voters:
I20260129 03:00:16.263906 21100 leader_election.cc:304] T cb0b3f259ebc4310927338a8740e87b8 P 7a9daf5d42a84615a6383cfec9a10608 [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1c50de89a5a845668696dcb28c3badd6, 7a9daf5d42a84615a6383cfec9a10608; no voters:
I20260129 03:00:16.270562 20436 raft_consensus.cc:2468] T 4b1876f9a70b49edb3aa74105e66d3b3 P 197b880fa39d432a9b472845876bfaf0 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1c50de89a5a845668696dcb28c3badd6 in term 7.
I20260129 03:00:16.273618 20553 leader_election.cc:304] T 4b1876f9a70b49edb3aa74105e66d3b3 P 1c50de89a5a845668696dcb28c3badd6 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 197b880fa39d432a9b472845876bfaf0, 1c50de89a5a845668696dcb28c3badd6; no voters:
I20260129 03:00:16.254562 23570 consensus_queue.cc:1048] T 505580c93c694bbb8ea96fdf9b17ab46 P 7a9daf5d42a84615a6383cfec9a10608 [LEADER]: Connected to new peer: Peer: permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 7, Time since last communication: 0.009s
I20260129 03:00:16.259909 22932 raft_consensus.cc:3055] T add8d4923d1a489e86882c3a25692b95 P 7a9daf5d42a84615a6383cfec9a10608 [term 7 LEADER]: Stepping down as leader of term 7
I20260129 03:00:16.269104 23262 raft_consensus.cc:2804] T c891d2bff4ce4568861bbc992e91fbb0 P 1c50de89a5a845668696dcb28c3badd6 [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260129 03:00:16.259824 23333 raft_consensus.cc:515] T cedce331e1584ac8b636c9b8a72b2ba0 P 974b802141bd4f13beffef2aa03f88f9 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 03:00:16.277096 22996 raft_consensus.cc:2804] T e33a417cf9c54cf5af73a4ee3aac4d3a P 1c50de89a5a845668696dcb28c3badd6 [term 7 FOLLOWER]: Leader pre-election won for term 8
W20260129 03:00:16.292198 20554 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.211.4:40439, user_credentials={real_user=slave}} blocked reactor thread for 53943.2us
I20260129 03:00:16.251449 23602 raft_consensus.cc:515] T c2633a6fc34c44579de348d208b0c01e P 1c50de89a5a845668696dcb28c3badd6 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
W20260129 03:00:16.313982 20407 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.211.2:34517, user_credentials={real_user=slave}} blocked reactor thread for 475072us
I20260129 03:00:16.263607 22645 raft_consensus.cc:515] T 42d27a28d89b409e8807ec0efd97af69 P 197b880fa39d432a9b472845876bfaf0 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } }
I20260129 03:00:16.270690 23365 raft_consensus.cc:2804] T cb0b3f259ebc4310927338a8740e87b8 P 7a9daf5d42a84615a6383cfec9a10608 [term 9 FOLLOWER]: Leader election won for term 9
I20260129 03:00:16.250547 23555 raft_consensus.cc:515] T 3906b080a5b34a74b8b64128635e8997 P 974b802141bd4f13beffef2aa03f88f9 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 03:00:16.293213 23485 leader_election.cc:290] T 3906b080a5b34a74b8b64128635e8997 P 197b880fa39d432a9b472845876bfaf0 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 974b802141bd4f13beffef2aa03f88f9 (127.18.211.2:34517), 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3:37735)
I20260129 03:00:16.293735 23531 raft_consensus.cc:2804] T 4b1876f9a70b49edb3aa74105e66d3b3 P 1c50de89a5a845668696dcb28c3badd6 [term 7 FOLLOWER]: Leader election won for term 7
I20260129 03:00:16.255666 23514 raft_consensus.cc:515] T 4b1876f9a70b49edb3aa74105e66d3b3 P 974b802141bd4f13beffef2aa03f88f9 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 03:00:16.311611 20588 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35752 (request call id 1752) took 2430 ms. Trace:
W20260129 03:00:16.320212 23163 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.546s user 0.020s sys 0.049s
I20260129 03:00:16.381646 20583 raft_consensus.cc:1275] T 505580c93c694bbb8ea96fdf9b17ab46 P 1c50de89a5a845668696dcb28c3badd6 [term 8 FOLLOWER]: Refusing update from remote peer 7a9daf5d42a84615a6383cfec9a10608: Log matching property violated. Preceding OpId in replica: term: 7 index: 9. Preceding OpId from leader: term: 8 index: 9. (term mismatch)
I20260129 03:00:16.377442 20588 rpcz_store.cc:276] 0129 03:00:13.880791 (+ 0us) service_pool.cc:168] Inserting onto call queue
0129 03:00:15.528562 (+1647771us) service_pool.cc:225] Handling call
0129 03:00:15.529000 (+ 438us) raft_consensus.cc:1471] Updating replica for 1 ops
0129 03:00:15.529304 (+ 304us) raft_consensus.cc:1534] Early marking committed up to index 0
0129 03:00:15.529328 (+ 24us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0129 03:00:15.529846 (+ 518us) log.cc:844] Serialized 26 byte log entry
0129 03:00:15.989134 (+459288us) raft_consensus.cc:1645] Marking committed up to 4
0129 03:00:16.061283 (+ 72149us) log.cc:844] Serialized 14 byte log entry
0129 03:00:16.086393 (+ 25110us) raft_consensus.cc:1697] Filling consensus response to leader.
0129 03:00:16.086497 (+ 104us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0129 03:00:16.311344 (+224847us) raft_consensus.cc:1685] finished
0129 03:00:16.311368 (+ 24us) raft_consensus.cc:1691] UpdateReplicas() finished
0129 03:00:16.311542 (+ 174us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":60376,"prepare.queue_time_us":76816,"prepare.run_cpu_time_us":68,"prepare.run_wall_time_us":5940,"raft.queue_time_us":28489,"raft.run_cpu_time_us":130,"raft.run_wall_time_us":133,"thread_start_us":421821,"threads_started":1,"wal-append.queue_time_us":778604}
I20260129 03:00:16.301891 23262 raft_consensus.cc:493] T c891d2bff4ce4568861bbc992e91fbb0 P 1c50de89a5a845668696dcb28c3badd6 [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260129 03:00:16.263311 23456 raft_consensus.cc:2764] T cb0b3f259ebc4310927338a8740e87b8 P 1c50de89a5a845668696dcb28c3badd6 [term 9 FOLLOWER]: Leader election decision vote started in defunct term 8: won
W20260129 03:00:16.355007 20479 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.211.1:42027, user_credentials={real_user=slave}} blocked reactor thread for 524612us
W20260129 03:00:16.319976 20439 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.509s user 0.000s sys 0.010s
W20260129 03:00:16.356019 20556 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.211.2:34517, user_credentials={real_user=slave}} blocked reactor thread for 104979us
I20260129 03:00:16.277676 23480 raft_consensus.cc:2804] T a644c02b16af47679824ccd060ac10fd P 7a9daf5d42a84615a6383cfec9a10608 [term 5 FOLLOWER]: Leader pre-election won for term 6
W20260129 03:00:16.372252 20404 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260129 03:00:16.350325 23565 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.611s user 0.020s sys 0.068s
I20260129 03:00:16.385373 20336 catalog_manager.cc:5654] T 505580c93c694bbb8ea96fdf9b17ab46 P 1c50de89a5a845668696dcb28c3badd6 reported cstate change: term changed from 2 to 5, leader changed from 974b802141bd4f13beffef2aa03f88f9 (127.18.211.2) to 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3). New cstate: current_term: 5 leader_uuid: "1c50de89a5a845668696dcb28c3badd6" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20260129 03:00:16.295614 22932 raft_consensus.cc:740] T add8d4923d1a489e86882c3a25692b95 P 7a9daf5d42a84615a6383cfec9a10608 [term 7 LEADER]: Becoming Follower/Learner. State: Replica: 7a9daf5d42a84615a6383cfec9a10608, State: Running, Role: LEADER
W20260129 03:00:16.384233 23163 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.620s user 0.021s sys 0.055s
I20260129 03:00:16.392344 23262 raft_consensus.cc:3060] T c891d2bff4ce4568861bbc992e91fbb0 P 1c50de89a5a845668696dcb28c3badd6 [term 8 FOLLOWER]: Advancing to term 9
I20260129 03:00:16.382200 20583 pending_rounds.cc:85] T 505580c93c694bbb8ea96fdf9b17ab46 P 1c50de89a5a845668696dcb28c3badd6: Aborting all ops after (but not including) 8
I20260129 03:00:16.408066 20583 pending_rounds.cc:107] T 505580c93c694bbb8ea96fdf9b17ab46 P 1c50de89a5a845668696dcb28c3badd6: Aborting uncommitted NO_OP operation due to leader change: 7.9
I20260129 03:00:16.408382 20583 raft_consensus.cc:2889] T 505580c93c694bbb8ea96fdf9b17ab46 P 1c50de89a5a845668696dcb28c3badd6 [term 8 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260129 03:00:16.302534 22996 raft_consensus.cc:493] T e33a417cf9c54cf5af73a4ee3aac4d3a P 1c50de89a5a845668696dcb28c3badd6 [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260129 03:00:16.384923 23362 raft_consensus.cc:3060] T 75f45538dc704d9a8ec4ff8ad9f3ea5b P 7a9daf5d42a84615a6383cfec9a10608 [term 4 FOLLOWER]: Advancing to term 5
W20260129 03:00:16.409368 21100 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.211.3:37735, user_credentials={real_user=slave}} blocked reactor thread for 140301us
I20260129 03:00:16.408365 20336 catalog_manager.cc:5654] T 724a236524cb4e7ba3428880681d6cc2 P 1c50de89a5a845668696dcb28c3badd6 reported cstate change: term changed from 1 to 4, leader changed from 974b802141bd4f13beffef2aa03f88f9 (127.18.211.2) to 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3). New cstate: current_term: 4 leader_uuid: "1c50de89a5a845668696dcb28c3badd6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } health_report { overall_health: HEALTHY } } }
I20260129 03:00:16.384070 23365 raft_consensus.cc:697] T cb0b3f259ebc4310927338a8740e87b8 P 7a9daf5d42a84615a6383cfec9a10608 [term 9 LEADER]: Becoming Leader. State: Replica: 7a9daf5d42a84615a6383cfec9a10608, State: Running, Role: LEADER
I20260129 03:00:16.416865 20513 raft_consensus.cc:2468] T cb0b3f259ebc4310927338a8740e87b8 P 974b802141bd4f13beffef2aa03f88f9 [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7a9daf5d42a84615a6383cfec9a10608 in term 9.
I20260129 03:00:16.426257 23568 consensus_queue.cc:1048] T 505580c93c694bbb8ea96fdf9b17ab46 P 7a9daf5d42a84615a6383cfec9a10608 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 7, Time since last communication: 0.000s
W20260129 03:00:16.397284 23565 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.746s user 0.020s sys 0.074s
I20260129 03:00:16.396739 23480 raft_consensus.cc:493] T a644c02b16af47679824ccd060ac10fd P 7a9daf5d42a84615a6383cfec9a10608 [term 5 FOLLOWER]: Starting leader election (detected failure of leader 1c50de89a5a845668696dcb28c3badd6)
I20260129 03:00:16.432822 20336 catalog_manager.cc:5654] T 75be4bd6bd45474795fd472eff2953ac P 1c50de89a5a845668696dcb28c3badd6 reported cstate change: term changed from 7 to 9. New cstate: current_term: 9 leader_uuid: "1c50de89a5a845668696dcb28c3badd6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } health_report { overall_health: HEALTHY } } }
I20260129 03:00:16.397570 23531 raft_consensus.cc:697] T 4b1876f9a70b49edb3aa74105e66d3b3 P 1c50de89a5a845668696dcb28c3badd6 [term 7 LEADER]: Becoming Leader. State: Replica: 1c50de89a5a845668696dcb28c3badd6, State: Running, Role: LEADER
I20260129 03:00:16.438585 20407 leader_election.cc:304] T 4e420f90884f4796997b5ea41e49e411 P 197b880fa39d432a9b472845876bfaf0 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 197b880fa39d432a9b472845876bfaf0; no voters: 7a9daf5d42a84615a6383cfec9a10608, 974b802141bd4f13beffef2aa03f88f9
I20260129 03:00:16.437222 20336 catalog_manager.cc:5654] T add8d4923d1a489e86882c3a25692b95 P 1c50de89a5a845668696dcb28c3badd6 reported cstate change: term changed from 1 to 3. New cstate: current_term: 3 leader_uuid: "1c50de89a5a845668696dcb28c3badd6" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20260129 03:00:16.440721 20554 leader_election.cc:304] T 4e3805ffa70242e8ade66da260dddf66 P 1c50de89a5a845668696dcb28c3badd6 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1c50de89a5a845668696dcb28c3badd6; no voters: 197b880fa39d432a9b472845876bfaf0, 7a9daf5d42a84615a6383cfec9a10608
I20260129 03:00:16.417357 22996 raft_consensus.cc:3060] T e33a417cf9c54cf5af73a4ee3aac4d3a P 1c50de89a5a845668696dcb28c3badd6 [term 7 FOLLOWER]: Advancing to term 8
I20260129 03:00:16.424275 23427 leader_election.cc:290] T 926a250fa5834e88b96ccecbefbf5086 P 974b802141bd4f13beffef2aa03f88f9 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 197b880fa39d432a9b472845876bfaf0 (127.18.211.1:42027), 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3:37735)
I20260129 03:00:16.438656 23480 raft_consensus.cc:3060] T a644c02b16af47679824ccd060ac10fd P 7a9daf5d42a84615a6383cfec9a10608 [term 5 FOLLOWER]: Advancing to term 6
I20260129 03:00:16.455310 20516 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "cb4727386a3e48aca5d71cc6c611c197" candidate_uuid: "197b880fa39d432a9b472845876bfaf0" candidate_term: 6 candidate_status { last_received { term: 5 index: 2 } } ignore_live_leader: false dest_uuid: "974b802141bd4f13beffef2aa03f88f9" is_pre_election: true
I20260129 03:00:16.460638 20436 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c2633a6fc34c44579de348d208b0c01e" candidate_uuid: "1c50de89a5a845668696dcb28c3badd6" candidate_term: 6 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "197b880fa39d432a9b472845876bfaf0"
I20260129 03:00:16.463057 20436 raft_consensus.cc:3060] T c2633a6fc34c44579de348d208b0c01e P 197b880fa39d432a9b472845876bfaf0 [term 5 FOLLOWER]: Advancing to term 6
I20260129 03:00:16.464378 20511 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "36ff33be666a4c82954731691dbcf327" candidate_uuid: "197b880fa39d432a9b472845876bfaf0" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "974b802141bd4f13beffef2aa03f88f9" is_pre_election: true
W20260129 03:00:16.460032 23624 log.cc:927] Time spent T 913fc7d751ed4b7388d889aac30a5f9b P 1c50de89a5a845668696dcb28c3badd6: Append to log took a long time: real 0.071s user 0.000s sys 0.002s
I20260129 03:00:16.451756 20336 catalog_manager.cc:5654] T c2633a6fc34c44579de348d208b0c01e P 1c50de89a5a845668696dcb28c3badd6 reported cstate change: term changed from 1 to 2, leader changed from 197b880fa39d432a9b472845876bfaf0 (127.18.211.1) to 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3). New cstate: current_term: 2 leader_uuid: "1c50de89a5a845668696dcb28c3badd6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } health_report { overall_health: HEALTHY } } }
I20260129 03:00:16.450233 23469 raft_consensus.cc:2749] T 4e420f90884f4796997b5ea41e49e411 P 197b880fa39d432a9b472845876bfaf0 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260129 03:00:16.474871 20516 raft_consensus.cc:2468] T cb4727386a3e48aca5d71cc6c611c197 P 974b802141bd4f13beffef2aa03f88f9 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 197b880fa39d432a9b472845876bfaf0 in term 5.
I20260129 03:00:16.492851 23630 consensus_queue.cc:1059] T 0340229d521b432580ae926f6062e3f7 P 974b802141bd4f13beffef2aa03f88f9 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 }, Status: INVALID_TERM, Last received: 7.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.048s
I20260129 03:00:16.493999 23630 raft_consensus.cc:3055] T 0340229d521b432580ae926f6062e3f7 P 974b802141bd4f13beffef2aa03f88f9 [term 7 LEADER]: Stepping down as leader of term 7
I20260129 03:00:16.494379 23630 raft_consensus.cc:740] T 0340229d521b432580ae926f6062e3f7 P 974b802141bd4f13beffef2aa03f88f9 [term 7 LEADER]: Becoming Follower/Learner. State: Replica: 974b802141bd4f13beffef2aa03f88f9, State: Running, Role: LEADER
I20260129 03:00:16.500877 23514 leader_election.cc:290] T 4b1876f9a70b49edb3aa74105e66d3b3 P 974b802141bd4f13beffef2aa03f88f9 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 197b880fa39d432a9b472845876bfaf0 (127.18.211.1:42027), 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3:37735)
I20260129 03:00:16.501737 23531 consensus_queue.cc:237] T 4b1876f9a70b49edb3aa74105e66d3b3 P 1c50de89a5a845668696dcb28c3badd6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 4.4, Last appended by leader: 4, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 03:00:16.508670 23333 leader_election.cc:290] T cedce331e1584ac8b636c9b8a72b2ba0 P 974b802141bd4f13beffef2aa03f88f9 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 197b880fa39d432a9b472845876bfaf0 (127.18.211.1:42027), 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3:37735)
I20260129 03:00:16.479075 20508 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3ccac82a01a2445fa5b0ce6c0fcf2295" candidate_uuid: "197b880fa39d432a9b472845876bfaf0" candidate_term: 5 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "974b802141bd4f13beffef2aa03f88f9" is_pre_election: true
I20260129 03:00:16.523676 21099 leader_election.cc:304] T c934d31efd3f4e5195605ddc354d14e1 P 7a9daf5d42a84615a6383cfec9a10608 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 197b880fa39d432a9b472845876bfaf0, 7a9daf5d42a84615a6383cfec9a10608; no voters: 974b802141bd4f13beffef2aa03f88f9
I20260129 03:00:16.517164 20511 raft_consensus.cc:2468] T 36ff33be666a4c82954731691dbcf327 P 974b802141bd4f13beffef2aa03f88f9 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 197b880fa39d432a9b472845876bfaf0 in term 4.
I20260129 03:00:16.528152 20516 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e33a417cf9c54cf5af73a4ee3aac4d3a" candidate_uuid: "197b880fa39d432a9b472845876bfaf0" candidate_term: 7 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "974b802141bd4f13beffef2aa03f88f9"
W20260129 03:00:16.530134 20554 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.211.4:40439, user_credentials={real_user=slave}} blocked reactor thread for 99953.7us
I20260129 03:00:16.480064 23602 leader_election.cc:290] T c2633a6fc34c44579de348d208b0c01e P 1c50de89a5a845668696dcb28c3badd6 [CANDIDATE]: Term 6 election: Requested vote from peers 197b880fa39d432a9b472845876bfaf0 (127.18.211.1:42027), 974b802141bd4f13beffef2aa03f88f9 (127.18.211.2:34517)
I20260129 03:00:16.454578 22932 consensus_queue.cc:260] T add8d4923d1a489e86882c3a25692b95 P 7a9daf5d42a84615a6383cfec9a10608 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 7.6, Last appended by leader: 6, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } }
I20260129 03:00:16.495276 23630 consensus_queue.cc:260] T 0340229d521b432580ae926f6062e3f7 P 974b802141bd4f13beffef2aa03f88f9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 7.5, Last appended by leader: 5, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 03:00:16.542852 23627 raft_consensus.cc:493] T 18f30ebaa4174fd28c958e72d0cc7e24 P 197b880fa39d432a9b472845876bfaf0 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 7a9daf5d42a84615a6383cfec9a10608)
W20260129 03:00:16.545310 20555 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260129 03:00:16.547061 22645 leader_election.cc:290] T 42d27a28d89b409e8807ec0efd97af69 P 197b880fa39d432a9b472845876bfaf0 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3:37735), 7a9daf5d42a84615a6383cfec9a10608 (127.18.211.4:40439)
I20260129 03:00:16.518723 23365 consensus_queue.cc:237] T cb0b3f259ebc4310927338a8740e87b8 P 7a9daf5d42a84615a6383cfec9a10608 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 7.8, Last appended by leader: 8, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } }
I20260129 03:00:16.541188 23570 raft_consensus.cc:2804] T c934d31efd3f4e5195605ddc354d14e1 P 7a9daf5d42a84615a6383cfec9a10608 [term 6 FOLLOWER]: Leader election won for term 6
W20260129 03:00:16.556154 20589 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.506s user 0.003s sys 0.024s
W20260129 03:00:16.564404 20589 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.515s user 0.003s sys 0.024s
W20260129 03:00:16.548466 23224 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.610s user 0.025s sys 0.022s
I20260129 03:00:16.564786 20336 catalog_manager.cc:5654] T 0340229d521b432580ae926f6062e3f7 P 1c50de89a5a845668696dcb28c3badd6 reported cstate change: term changed from 3 to 5, leader changed from 197b880fa39d432a9b472845876bfaf0 (127.18.211.1) to 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3). New cstate: current_term: 5 leader_uuid: "1c50de89a5a845668696dcb28c3badd6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } health_report { overall_health: HEALTHY } } }
I20260129 03:00:16.567680 20508 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "36ff33be666a4c82954731691dbcf327" candidate_uuid: "1c50de89a5a845668696dcb28c3badd6" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "974b802141bd4f13beffef2aa03f88f9"
I20260129 03:00:16.574899 20511 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c934d31efd3f4e5195605ddc354d14e1" candidate_uuid: "197b880fa39d432a9b472845876bfaf0" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "974b802141bd4f13beffef2aa03f88f9"
I20260129 03:00:16.577509 20511 raft_consensus.cc:3060] T c934d31efd3f4e5195605ddc354d14e1 P 974b802141bd4f13beffef2aa03f88f9 [term 6 FOLLOWER]: Advancing to term 7
I20260129 03:00:16.566520 20510 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a1a39dcd328e4268ae41b988c66b4a75" candidate_uuid: "197b880fa39d432a9b472845876bfaf0" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "974b802141bd4f13beffef2aa03f88f9" is_pre_election: true
W20260129 03:00:16.561540 23466 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.693s user 0.023s sys 0.026s
I20260129 03:00:16.580597 20510 raft_consensus.cc:2468] T a1a39dcd328e4268ae41b988c66b4a75 P 974b802141bd4f13beffef2aa03f88f9 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 197b880fa39d432a9b472845876bfaf0 in term 6.
W20260129 03:00:16.573841 20405 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.670s user 0.017s sys 0.034s
W20260129 03:00:16.591572 20405 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.688s user 0.017s sys 0.034s
W20260129 03:00:16.571895 23046 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.622s user 0.015s sys 0.013s
I20260129 03:00:16.596778 20512 raft_consensus.cc:1240] T 76f713c3331540d2a5b76164103e9eb1 P 974b802141bd4f13beffef2aa03f88f9 [term 6 LEADER]: Rejecting Update request from peer 1c50de89a5a845668696dcb28c3badd6 for earlier term 5. Current term is 6. Ops: []
I20260129 03:00:16.576339 23630 raft_consensus.cc:3060] T 0340229d521b432580ae926f6062e3f7 P 974b802141bd4f13beffef2aa03f88f9 [term 7 FOLLOWER]: Advancing to term 8
W20260129 03:00:16.589810 23224 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.640s user 0.026s sys 0.023s
W20260129 03:00:16.595921 23466 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.736s user 0.024s sys 0.027s
I20260129 03:00:16.572942 23570 raft_consensus.cc:697] T c934d31efd3f4e5195605ddc354d14e1 P 7a9daf5d42a84615a6383cfec9a10608 [term 6 LEADER]: Becoming Leader. State: Replica: 7a9daf5d42a84615a6383cfec9a10608, State: Running, Role: LEADER
I20260129 03:00:16.609136 23570 consensus_queue.cc:237] T c934d31efd3f4e5195605ddc354d14e1 P 7a9daf5d42a84615a6383cfec9a10608 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 5.7, Last appended by leader: 7, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } }
I20260129 03:00:16.610769 20515 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "cedce331e1584ac8b636c9b8a72b2ba0" candidate_uuid: "1c50de89a5a845668696dcb28c3badd6" candidate_term: 8 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "974b802141bd4f13beffef2aa03f88f9" is_pre_election: true
I20260129 03:00:16.611639 20515 raft_consensus.cc:2393] T cedce331e1584ac8b636c9b8a72b2ba0 P 974b802141bd4f13beffef2aa03f88f9 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1c50de89a5a845668696dcb28c3badd6 in current term 8: Already voted for candidate 197b880fa39d432a9b472845876bfaf0 in this term.
I20260129 03:00:16.603940 20510 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "8055232cf0004d40a85f3bf983fe00a9" candidate_uuid: "1c50de89a5a845668696dcb28c3badd6" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "974b802141bd4f13beffef2aa03f88f9" is_pre_election: true
I20260129 03:00:16.612381 20514 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7d9494f5584d4d5db8b9adf1c8fc888b" candidate_uuid: "197b880fa39d432a9b472845876bfaf0" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: true dest_uuid: "974b802141bd4f13beffef2aa03f88f9"
I20260129 03:00:16.612483 20515 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4408006eb5174de1896b037cf3f18457" candidate_uuid: "1c50de89a5a845668696dcb28c3badd6" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "974b802141bd4f13beffef2aa03f88f9" is_pre_election: true
I20260129 03:00:16.613049 20514 raft_consensus.cc:3060] T 7d9494f5584d4d5db8b9adf1c8fc888b P 974b802141bd4f13beffef2aa03f88f9 [term 4 FOLLOWER]: Advancing to term 5
I20260129 03:00:16.612998 20510 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "75f45538dc704d9a8ec4ff8ad9f3ea5b" candidate_uuid: "1c50de89a5a845668696dcb28c3badd6" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "974b802141bd4f13beffef2aa03f88f9"
I20260129 03:00:16.613639 20515 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a0fcc58d6fb6473cafbd9d34d5195269" candidate_uuid: "1c50de89a5a845668696dcb28c3badd6" candidate_term: 6 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "974b802141bd4f13beffef2aa03f88f9" is_pre_election: true
W20260129 03:00:16.599613 23450 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.660s user 0.016s sys 0.042s
W20260129 03:00:16.615552 23450 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.677s user 0.017s sys 0.042s
I20260129 03:00:16.615947 23450 consensus_queue.cc:1243] T 8055232cf0004d40a85f3bf983fe00a9 P 974b802141bd4f13beffef2aa03f88f9 [LEADER]: Peer 197b880fa39d432a9b472845876bfaf0 log is divergent from this leader: its last log entry 4.3 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 1
W20260129 03:00:16.616147 20405 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.211.4:40439, user_credentials={real_user=slave}} blocked reactor thread for 796033us
W20260129 03:00:16.596939 23046 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.647s user 0.016s sys 0.013s
I20260129 03:00:16.617275 22932 raft_consensus.cc:3060] T add8d4923d1a489e86882c3a25692b95 P 7a9daf5d42a84615a6383cfec9a10608 [term 7 FOLLOWER]: Advancing to term 8
I20260129 03:00:16.617967 20405 leader_election.cc:304] T fd8311b779af4e1fa1402b2a02a602f2 P 197b880fa39d432a9b472845876bfaf0 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 197b880fa39d432a9b472845876bfaf0, 7a9daf5d42a84615a6383cfec9a10608; no voters:
I20260129 03:00:16.618840 23466 consensus_queue.cc:1048] T a609ac02fae84725ab743f95f51d33ce P 197b880fa39d432a9b472845876bfaf0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.001s
I20260129 03:00:16.622277 23485 raft_consensus.cc:2804] T fd8311b779af4e1fa1402b2a02a602f2 P 197b880fa39d432a9b472845876bfaf0 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260129 03:00:16.622720 23485 raft_consensus.cc:493] T fd8311b779af4e1fa1402b2a02a602f2 P 197b880fa39d432a9b472845876bfaf0 [term 5 FOLLOWER]: Starting leader election (detected failure of leader 1c50de89a5a845668696dcb28c3badd6)
I20260129 03:00:16.623111 23485 raft_consensus.cc:3060] T fd8311b779af4e1fa1402b2a02a602f2 P 197b880fa39d432a9b472845876bfaf0 [term 5 FOLLOWER]: Advancing to term 6
W20260129 03:00:16.634717 20553 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260129 03:00:16.591149 23475 raft_consensus.cc:493] T 724a236524cb4e7ba3428880681d6cc2 P 197b880fa39d432a9b472845876bfaf0 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260129 03:00:16.641688 20554 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.211.4:40439, user_credentials={real_user=slave}} blocked reactor thread for 110746us
I20260129 03:00:16.647075 20510 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "75b0b6e82ddc4f92b1fe1d94f9a0b73b" candidate_uuid: "1c50de89a5a845668696dcb28c3badd6" candidate_term: 5 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "974b802141bd4f13beffef2aa03f88f9"
I20260129 03:00:16.597236 23555 leader_election.cc:290] T 3906b080a5b34a74b8b64128635e8997 P 974b802141bd4f13beffef2aa03f88f9 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 197b880fa39d432a9b472845876bfaf0 (127.18.211.1:42027), 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3:37735)
I20260129 03:00:16.650283 20441 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "febffe4a8b394368b69b1d164308fb9a" candidate_uuid: "7a9daf5d42a84615a6383cfec9a10608" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "197b880fa39d432a9b472845876bfaf0"
W20260129 03:00:16.650791 21102 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.211.2:34517, user_credentials={real_user=slave}} blocked reactor thread for 382455us
I20260129 03:00:16.651149 20441 raft_consensus.cc:2393] T febffe4a8b394368b69b1d164308fb9a P 197b880fa39d432a9b472845876bfaf0 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 7a9daf5d42a84615a6383cfec9a10608 in current term 6: Already voted for candidate 974b802141bd4f13beffef2aa03f88f9 in this term.
I20260129 03:00:16.563082 23627 raft_consensus.cc:515] T 18f30ebaa4174fd28c958e72d0cc7e24 P 197b880fa39d432a9b472845876bfaf0 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } }
I20260129 03:00:16.634359 23086 raft_consensus.cc:3055] T 0f20af7fbe5d4db89a6f17948d98912c P 197b880fa39d432a9b472845876bfaf0 [term 7 LEADER]: Stepping down as leader of term 7
W20260129 03:00:16.666290 20482 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260129 03:00:16.681993 23627 leader_election.cc:290] T 18f30ebaa4174fd28c958e72d0cc7e24 P 197b880fa39d432a9b472845876bfaf0 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3:37735), 7a9daf5d42a84615a6383cfec9a10608 (127.18.211.4:40439)
I20260129 03:00:16.652052 23466 consensus_queue.cc:1048] T 2ac659d732314e1288849f08fe5e074e P 197b880fa39d432a9b472845876bfaf0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.031s
I20260129 03:00:16.685425 20510 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "76f713c3331540d2a5b76164103e9eb1" candidate_uuid: "1c50de89a5a845668696dcb28c3badd6" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "974b802141bd4f13beffef2aa03f88f9" is_pre_election: true
I20260129 03:00:16.686456 20509 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "76f713c3331540d2a5b76164103e9eb1" candidate_uuid: "197b880fa39d432a9b472845876bfaf0" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "974b802141bd4f13beffef2aa03f88f9" is_pre_election: true
W20260129 03:00:16.702740 20405 connection.cc:232] RPC call timeout handler was delayed by -1.0472419200000331s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1552s for a response
W20260129 03:00:16.703472 20405 connection.cc:232] RPC call timeout handler was delayed by -1.0301419200000055s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.15710000000000002s for a response
I20260129 03:00:16.703503 20512 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3906b080a5b34a74b8b64128635e8997" candidate_uuid: "197b880fa39d432a9b472845876bfaf0" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "974b802141bd4f13beffef2aa03f88f9" is_pre_election: true
I20260129 03:00:16.708963 20512 raft_consensus.cc:2468] T 3906b080a5b34a74b8b64128635e8997 P 974b802141bd4f13beffef2aa03f88f9 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 197b880fa39d432a9b472845876bfaf0 in term 5.
W20260129 03:00:16.710994 20482 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260129 03:00:16.713205 20436 raft_consensus.cc:2468] T c2633a6fc34c44579de348d208b0c01e P 197b880fa39d432a9b472845876bfaf0 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1c50de89a5a845668696dcb28c3badd6 in term 6.
I20260129 03:00:16.714753 21135 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "42d27a28d89b409e8807ec0efd97af69" candidate_uuid: "197b880fa39d432a9b472845876bfaf0" candidate_term: 9 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "7a9daf5d42a84615a6383cfec9a10608" is_pre_election: true
I20260129 03:00:16.715723 21132 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c934d31efd3f4e5195605ddc354d14e1" candidate_uuid: "197b880fa39d432a9b472845876bfaf0" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "7a9daf5d42a84615a6383cfec9a10608"
I20260129 03:00:16.717183 20553 leader_election.cc:304] T c2633a6fc34c44579de348d208b0c01e P 1c50de89a5a845668696dcb28c3badd6 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 197b880fa39d432a9b472845876bfaf0, 1c50de89a5a845668696dcb28c3badd6; no voters:
I20260129 03:00:16.719331 20514 raft_consensus.cc:2468] T 7d9494f5584d4d5db8b9adf1c8fc888b P 974b802141bd4f13beffef2aa03f88f9 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 197b880fa39d432a9b472845876bfaf0 in term 5.
I20260129 03:00:16.724766 20584 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:48526 (request call id 414) took 1158 ms. Trace:
I20260129 03:00:16.730196 20584 rpcz_store.cc:276] 0129 03:00:15.566442 (+ 0us) service_pool.cc:168] Inserting onto call queue
0129 03:00:15.993738 (+427296us) service_pool.cc:225] Handling call
0129 03:00:16.032313 (+ 38575us) raft_consensus.cc:1471] Updating replica for 0 ops
0129 03:00:16.032623 (+ 310us) raft_consensus.cc:1534] Early marking committed up to index 8
0129 03:00:16.157761 (+125138us) log.cc:844] Serialized 14 byte log entry
0129 03:00:16.724267 (+566506us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0129 03:00:16.724337 (+ 70us) raft_consensus.cc:1645] Marking committed up to 8
0129 03:00:16.724399 (+ 62us) raft_consensus.cc:1697] Filling consensus response to leader.
0129 03:00:16.724520 (+ 121us) raft_consensus.cc:1691] UpdateReplicas() finished
0129 03:00:16.724689 (+ 169us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":62835,"prepare.queue_time_us":119186,"prepare.run_cpu_time_us":73,"prepare.run_wall_time_us":6077,"thread_start_us":497870,"threads_started":1}
W20260129 03:00:16.750715 20435 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.548s user 0.006s sys 0.019s
W20260129 03:00:16.751447 20435 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.548s user 0.006s sys 0.020s
I20260129 03:00:16.672624 22989 raft_consensus.cc:3055] T 8055232cf0004d40a85f3bf983fe00a9 P 974b802141bd4f13beffef2aa03f88f9 [term 5 LEADER]: Stepping down as leader of term 5
I20260129 03:00:16.760715 20514 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "add8d4923d1a489e86882c3a25692b95" candidate_uuid: "1c50de89a5a845668696dcb28c3badd6" candidate_term: 8 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "974b802141bd4f13beffef2aa03f88f9"
I20260129 03:00:16.764196 20436 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "dc2f4cb752cf4d59b5f6bc689f5d8104" candidate_uuid: "7a9daf5d42a84615a6383cfec9a10608" candidate_term: 7 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "197b880fa39d432a9b472845876bfaf0" is_pre_election: true
I20260129 03:00:16.766160 21134 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "18f30ebaa4174fd28c958e72d0cc7e24" candidate_uuid: "197b880fa39d432a9b472845876bfaf0" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "7a9daf5d42a84615a6383cfec9a10608" is_pre_election: true
I20260129 03:00:16.673333 23086 raft_consensus.cc:740] T 0f20af7fbe5d4db89a6f17948d98912c P 197b880fa39d432a9b472845876bfaf0 [term 7 LEADER]: Becoming Follower/Learner. State: Replica: 197b880fa39d432a9b472845876bfaf0, State: Running, Role: LEADER
I20260129 03:00:16.721235 23627 raft_consensus.cc:493] T 17d6431cc77a491e8464d68db4081a51 P 197b880fa39d432a9b472845876bfaf0 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260129 03:00:16.718286 23555 raft_consensus.cc:493] T 7a9a69cd82724b1faed1d10aa091fc4a P 974b802141bd4f13beffef2aa03f88f9 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 197b880fa39d432a9b472845876bfaf0)
I20260129 03:00:16.724627 23565 raft_consensus.cc:2749] T 4e3805ffa70242e8ade66da260dddf66 P 1c50de89a5a845668696dcb28c3badd6 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260129 03:00:16.772696 23163 raft_consensus.cc:493] T 2e58e9a5a8aa4cecafb464addb4f35ed P 7a9daf5d42a84615a6383cfec9a10608 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260129 03:00:16.755947 23333 raft_consensus.cc:493] T 87d5fe48a44a408aaf03e894a67ecb2f P 974b802141bd4f13beffef2aa03f88f9 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 197b880fa39d432a9b472845876bfaf0)
I20260129 03:00:16.760641 22989 raft_consensus.cc:740] T 8055232cf0004d40a85f3bf983fe00a9 P 974b802141bd4f13beffef2aa03f88f9 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 974b802141bd4f13beffef2aa03f88f9, State: Running, Role: LEADER
I20260129 03:00:16.773062 23398 raft_consensus.cc:2804] T c2633a6fc34c44579de348d208b0c01e P 1c50de89a5a845668696dcb28c3badd6 [term 6 FOLLOWER]: Leader election won for term 6
I20260129 03:00:16.784219 20441 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a609ac02fae84725ab743f95f51d33ce" candidate_uuid: "7a9daf5d42a84615a6383cfec9a10608" candidate_term: 10 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "197b880fa39d432a9b472845876bfaf0" is_pre_election: true
I20260129 03:00:16.756139 23046 raft_consensus.cc:493] T a0fcc58d6fb6473cafbd9d34d5195269 P 974b802141bd4f13beffef2aa03f88f9 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 197b880fa39d432a9b472845876bfaf0)
W20260129 03:00:16.756974 23624 log.cc:927] Time spent T 913fc7d751ed4b7388d889aac30a5f9b P 1c50de89a5a845668696dcb28c3badd6: Append to log took a long time: real 0.090s user 0.000s sys 0.000s
I20260129 03:00:16.667431 23475 raft_consensus.cc:515] T 724a236524cb4e7ba3428880681d6cc2 P 197b880fa39d432a9b472845876bfaf0 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 03:00:16.805547 21133 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "add8d4923d1a489e86882c3a25692b95" candidate_uuid: "1c50de89a5a845668696dcb28c3badd6" candidate_term: 8 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "7a9daf5d42a84615a6383cfec9a10608"
W20260129 03:00:16.801776 20437 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.627s user 0.013s sys 0.026s
W20260129 03:00:16.806854 20437 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.859s user 0.013s sys 0.027s
I20260129 03:00:16.796850 20508 raft_consensus.cc:2468] T 36ff33be666a4c82954731691dbcf327 P 974b802141bd4f13beffef2aa03f88f9 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1c50de89a5a845668696dcb28c3badd6 in term 4.
I20260129 03:00:16.812526 21135 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2e58e9a5a8aa4cecafb464addb4f35ed" candidate_uuid: "1c50de89a5a845668696dcb28c3badd6" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "7a9daf5d42a84615a6383cfec9a10608"
I20260129 03:00:16.812918 21132 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "75f45538dc704d9a8ec4ff8ad9f3ea5b" candidate_uuid: "1c50de89a5a845668696dcb28c3badd6" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "7a9daf5d42a84615a6383cfec9a10608"
W20260129 03:00:16.860952 20480 thread.cc:641] dns-resolver [worker] (thread pool) Time spent creating pthread: real 0.583s user 0.023s sys 0.030s
W20260129 03:00:16.863747 20480 thread.cc:608] dns-resolver [worker] (thread pool) Time spent starting thread: real 0.586s user 0.023s sys 0.030s
W20260129 03:00:16.864017 20553 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260129 03:00:16.865214 20437 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55676 (request call id 1514) took 1336 ms. Trace:
W20260129 03:00:16.868389 20480 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.211.3:37735, user_credentials={real_user=slave}} blocked reactor thread for 614976us
I20260129 03:00:16.871929 20480 leader_election.cc:304] T 2ac659d732314e1288849f08fe5e074e P 974b802141bd4f13beffef2aa03f88f9 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 974b802141bd4f13beffef2aa03f88f9; no voters: 197b880fa39d432a9b472845876bfaf0, 7a9daf5d42a84615a6383cfec9a10608
W20260129 03:00:16.891703 20405 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.211.3:37735, user_credentials={real_user=slave}} blocked reactor thread for 87475.5us
I20260129 03:00:16.891978 21134 raft_consensus.cc:1240] T 505580c93c694bbb8ea96fdf9b17ab46 P 7a9daf5d42a84615a6383cfec9a10608 [term 8 LEADER]: Rejecting Update request from peer 1c50de89a5a845668696dcb28c3badd6 for earlier term 7. Current term is 8. Ops: [7.9-7.9]
I20260129 03:00:16.899286 20590 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35752 (request call id 1744) took 3522 ms. Trace:
I20260129 03:00:16.866878 20437 rpcz_store.cc:276] 0129 03:00:15.528966 (+ 0us) service_pool.cc:168] Inserting onto call queue
0129 03:00:15.565528 (+ 36562us) service_pool.cc:225] Handling call
0129 03:00:15.618391 (+ 52863us) raft_consensus.cc:1471] Updating replica for 0 ops
0129 03:00:15.811548 (+193157us) raft_consensus.cc:1534] Early marking committed up to index 7
0129 03:00:15.859033 (+ 47485us) log.cc:844] Serialized 14 byte log entry
0129 03:00:16.864763 (+1005730us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0129 03:00:16.864836 (+ 73us) raft_consensus.cc:1645] Marking committed up to 7
0129 03:00:16.864902 (+ 66us) raft_consensus.cc:1697] Filling consensus response to leader.
0129 03:00:16.864995 (+ 93us) raft_consensus.cc:1691] UpdateReplicas() finished
0129 03:00:16.865148 (+ 153us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":98063,"prepare.run_cpu_time_us":48,"prepare.run_wall_time_us":51,"thread_start_us":859078,"threads_started":1}
I20260129 03:00:16.898694 21129 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "75b0b6e82ddc4f92b1fe1d94f9a0b73b" candidate_uuid: "1c50de89a5a845668696dcb28c3badd6" candidate_term: 5 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "7a9daf5d42a84615a6383cfec9a10608"
I20260129 03:00:16.859246 23398 raft_consensus.cc:697] T c2633a6fc34c44579de348d208b0c01e P 1c50de89a5a845668696dcb28c3badd6 [term 6 LEADER]: Becoming Leader. State: Replica: 1c50de89a5a845668696dcb28c3badd6, State: Running, Role: LEADER
I20260129 03:00:16.787719 23627 raft_consensus.cc:515] T 17d6431cc77a491e8464d68db4081a51 P 197b880fa39d432a9b472845876bfaf0 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 03:00:16.916183 21099 leader_election.cc:304] T febffe4a8b394368b69b1d164308fb9a P 7a9daf5d42a84615a6383cfec9a10608 [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7a9daf5d42a84615a6383cfec9a10608; no voters: 197b880fa39d432a9b472845876bfaf0, 974b802141bd4f13beffef2aa03f88f9
W20260129 03:00:16.931003 20404 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260129 03:00:16.799798 23163 raft_consensus.cc:515] T 2e58e9a5a8aa4cecafb464addb4f35ed P 7a9daf5d42a84615a6383cfec9a10608 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } }
I20260129 03:00:16.813916 23046 raft_consensus.cc:515] T a0fcc58d6fb6473cafbd9d34d5195269 P 974b802141bd4f13beffef2aa03f88f9 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 03:00:16.989511 20510 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e7d83d4ccf684e76acb29a0dc844e1b8" candidate_uuid: "1c50de89a5a845668696dcb28c3badd6" candidate_term: 8 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "974b802141bd4f13beffef2aa03f88f9" is_pre_election: true
I20260129 03:00:16.860461 23555 raft_consensus.cc:515] T 7a9a69cd82724b1faed1d10aa091fc4a P 974b802141bd4f13beffef2aa03f88f9 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 03:00:16.814231 23333 raft_consensus.cc:515] T 87d5fe48a44a408aaf03e894a67ecb2f P 974b802141bd4f13beffef2aa03f88f9 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } }
I20260129 03:00:16.900197 20590 rpcz_store.cc:276] 0129 03:00:13.077487 (+ 0us) service_pool.cc:168] Inserting onto call queue
0129 03:00:14.738885 (+1661398us) service_pool.cc:225] Handling call
0129 03:00:14.739200 (+ 315us) raft_consensus.cc:1471] Updating replica for 2 ops
0129 03:00:15.369193 (+629993us) raft_consensus.cc:1534] Early marking committed up to index 0
0129 03:00:15.375915 (+ 6722us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0129 03:00:15.553342 (+177427us) log.cc:844] Serialized 52 byte log entry
0129 03:00:16.061537 (+508195us) raft_consensus.cc:1645] Marking committed up to 4
0129 03:00:16.566136 (+504599us) log.cc:844] Serialized 14 byte log entry
0129 03:00:16.594905 (+ 28769us) log.cc:844] Serialized 14 byte log entry
0129 03:00:16.595185 (+ 280us) raft_consensus.cc:1697] Filling consensus response to leader.
0129 03:00:16.599898 (+ 4713us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0129 03:00:16.599947 (+ 49us) raft_consensus.cc:1685] finished
0129 03:00:16.599957 (+ 10us) raft_consensus.cc:1691] UpdateReplicas() finished
0129 03:00:16.600084 (+ 127us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":411517,"prepare.queue_time_us":113552,"prepare.run_cpu_time_us":99,"prepare.run_wall_time_us":6506,"raft.queue_time_us":307124,"raft.run_cpu_time_us":410,"raft.run_wall_time_us":16472,"thread_start_us":1015818,"threads_started":3,"wal-append.queue_time_us":660315}
W20260129 03:00:16.983937 23636 log.cc:927] Time spent T 18f30ebaa4174fd28c958e72d0cc7e24 P 1c50de89a5a845668696dcb28c3badd6: Append to log took a long time: real 0.065s user 0.001s sys 0.000s
I20260129 03:00:16.993461 21099 leader_election.cc:304] T dc2f4cb752cf4d59b5f6bc689f5d8104 P 7a9daf5d42a84615a6383cfec9a10608 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7a9daf5d42a84615a6383cfec9a10608; no voters: 197b880fa39d432a9b472845876bfaf0, 1c50de89a5a845668696dcb28c3badd6
W20260129 03:00:17.011610 20479 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.574s user 0.011s sys 0.005s
W20260129 03:00:17.014302 20479 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.579s user 0.011s sys 0.005s
W20260129 03:00:17.014719 20479 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.211.1:42027, user_credentials={real_user=slave}} blocked reactor thread for 600399us
I20260129 03:00:16.915727 20407 leader_election.cc:304] T cb4727386a3e48aca5d71cc6c611c197 P 197b880fa39d432a9b472845876bfaf0 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 197b880fa39d432a9b472845876bfaf0, 974b802141bd4f13beffef2aa03f88f9; no voters:
W20260129 03:00:17.018985 20407 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.211.2:34517, user_credentials={real_user=slave}} blocked reactor thread for 103342us
I20260129 03:00:16.910308 23262 raft_consensus.cc:515] T c891d2bff4ce4568861bbc992e91fbb0 P 1c50de89a5a845668696dcb28c3badd6 [term 9 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
W20260129 03:00:17.039093 21099 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.211.1:42027, user_credentials={real_user=slave}} blocked reactor thread for 92371.4us
W20260129 03:00:17.011708 20585 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.552s user 0.009s sys 0.007s
W20260129 03:00:17.042645 20585 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.583s user 0.009s sys 0.007s
I20260129 03:00:17.043567 20407 leader_election.cc:304] T 36ff33be666a4c82954731691dbcf327 P 197b880fa39d432a9b472845876bfaf0 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 197b880fa39d432a9b472845876bfaf0, 974b802141bd4f13beffef2aa03f88f9; no voters:
W20260129 03:00:17.000963 23632 log.cc:927] Time spent T a609ac02fae84725ab743f95f51d33ce P 197b880fa39d432a9b472845876bfaf0: Append to log took a long time: real 0.198s user 0.000s sys 0.001s
I20260129 03:00:17.048113 20407 leader_election.cc:304] T 3ccac82a01a2445fa5b0ce6c0fcf2295 P 197b880fa39d432a9b472845876bfaf0 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 197b880fa39d432a9b472845876bfaf0; no voters: 7a9daf5d42a84615a6383cfec9a10608, 974b802141bd4f13beffef2aa03f88f9
I20260129 03:00:17.048764 20510 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c2633a6fc34c44579de348d208b0c01e" candidate_uuid: "1c50de89a5a845668696dcb28c3badd6" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "974b802141bd4f13beffef2aa03f88f9" is_pre_election: true
I20260129 03:00:17.000069 23046 leader_election.cc:290] T a0fcc58d6fb6473cafbd9d34d5195269 P 974b802141bd4f13beffef2aa03f88f9 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 197b880fa39d432a9b472845876bfaf0 (127.18.211.1:42027), 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3:37735)
I20260129 03:00:17.063426 20588 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c891d2bff4ce4568861bbc992e91fbb0" candidate_uuid: "197b880fa39d432a9b472845876bfaf0" candidate_term: 9 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "1c50de89a5a845668696dcb28c3badd6"
I20260129 03:00:17.064483 20588 raft_consensus.cc:2393] T c891d2bff4ce4568861bbc992e91fbb0 P 1c50de89a5a845668696dcb28c3badd6 [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate 197b880fa39d432a9b472845876bfaf0 in current term 9: Already voted for candidate 1c50de89a5a845668696dcb28c3badd6 in this term.
W20260129 03:00:17.073279 20480 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260129 03:00:17.073786 20581 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4f8f3a38f6e24ba4857875181f447442" candidate_uuid: "197b880fa39d432a9b472845876bfaf0" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "1c50de89a5a845668696dcb28c3badd6"
I20260129 03:00:17.074698 20581 raft_consensus.cc:2393] T 4f8f3a38f6e24ba4857875181f447442 P 1c50de89a5a845668696dcb28c3badd6 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 197b880fa39d432a9b472845876bfaf0 in current term 7: Already voted for candidate 1c50de89a5a845668696dcb28c3badd6 in this term.
I20260129 03:00:16.924031 23640 raft_consensus.cc:493] T 00e79d8f25a34676b1d866eb2068a2b2 P 7a9daf5d42a84615a6383cfec9a10608 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 197b880fa39d432a9b472845876bfaf0)
I20260129 03:00:17.080798 20510 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e33a417cf9c54cf5af73a4ee3aac4d3a" candidate_uuid: "1c50de89a5a845668696dcb28c3badd6" candidate_term: 8 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "974b802141bd4f13beffef2aa03f88f9" is_pre_election: true
I20260129 03:00:17.050701 23466 raft_consensus.cc:2804] T cb4727386a3e48aca5d71cc6c611c197 P 197b880fa39d432a9b472845876bfaf0 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260129 03:00:17.044849 23568 raft_consensus.cc:493] T 87d5fe48a44a408aaf03e894a67ecb2f P 7a9daf5d42a84615a6383cfec9a10608 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 197b880fa39d432a9b472845876bfaf0)
W20260129 03:00:17.065461 20439 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.651s user 0.000s sys 0.042s
I20260129 03:00:17.063019 23531 raft_consensus.cc:493] T a1a39dcd328e4268ae41b988c66b4a75 P 1c50de89a5a845668696dcb28c3badd6 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 974b802141bd4f13beffef2aa03f88f9)
W20260129 03:00:17.086007 20482 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260129 03:00:17.101800 20439 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.691s user 0.000s sys 0.042s
W20260129 03:00:17.102634 20516 consensus_meta.cc:287] T e33a417cf9c54cf5af73a4ee3aac4d3a P 974b802141bd4f13beffef2aa03f88f9: Time spent flushing consensus metadata: real 0.572s user 0.006s sys 0.001s
I20260129 03:00:17.052225 23109 raft_consensus.cc:2749] T dc2f4cb752cf4d59b5f6bc689f5d8104 P 7a9daf5d42a84615a6383cfec9a10608 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260129 03:00:17.058015 23224 raft_consensus.cc:2804] T 36ff33be666a4c82954731691dbcf327 P 197b880fa39d432a9b472845876bfaf0 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260129 03:00:17.109119 20516 raft_consensus.cc:2468] T e33a417cf9c54cf5af73a4ee3aac4d3a P 974b802141bd4f13beffef2aa03f88f9 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 197b880fa39d432a9b472845876bfaf0 in term 7.
I20260129 03:00:17.123057 20514 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4b1876f9a70b49edb3aa74105e66d3b3" candidate_uuid: "1c50de89a5a845668696dcb28c3badd6" candidate_term: 7 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "974b802141bd4f13beffef2aa03f88f9"
I20260129 03:00:17.122983 20583 raft_consensus.cc:1275] T a0fcc58d6fb6473cafbd9d34d5195269 P 1c50de89a5a845668696dcb28c3badd6 [term 5 FOLLOWER]: Refusing update from remote peer 197b880fa39d432a9b472845876bfaf0: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 5 index: 4. (index mismatch)
I20260129 03:00:17.123922 20514 raft_consensus.cc:3060] T 4b1876f9a70b49edb3aa74105e66d3b3 P 974b802141bd4f13beffef2aa03f88f9 [term 6 FOLLOWER]: Advancing to term 7
I20260129 03:00:17.075737 22645 raft_consensus.cc:2749] T 3ccac82a01a2445fa5b0ce6c0fcf2295 P 197b880fa39d432a9b472845876bfaf0 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260129 03:00:17.128726 20516 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a8479d1fe0bd41e5946186e3a6ca8c7a" candidate_uuid: "1c50de89a5a845668696dcb28c3badd6" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "974b802141bd4f13beffef2aa03f88f9" is_pre_election: true
I20260129 03:00:16.997278 22989 consensus_queue.cc:260] T 8055232cf0004d40a85f3bf983fe00a9 P 974b802141bd4f13beffef2aa03f88f9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 5.3, Last appended by leader: 3, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 03:00:17.122849 20589 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:48526 (request call id 415) took 1353 ms. Trace:
I20260129 03:00:17.132064 20510 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "17d6431cc77a491e8464d68db4081a51" candidate_uuid: "1c50de89a5a845668696dcb28c3badd6" candidate_term: 6 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "974b802141bd4f13beffef2aa03f88f9" is_pre_election: true
I20260129 03:00:17.132212 20589 rpcz_store.cc:276] 0129 03:00:15.769469 (+ 0us) service_pool.cc:168] Inserting onto call queue
0129 03:00:16.021577 (+252108us) service_pool.cc:225] Handling call
0129 03:00:16.048339 (+ 26762us) raft_consensus.cc:1471] Updating replica for 1 ops
0129 03:00:16.048657 (+ 318us) raft_consensus.cc:1534] Early marking committed up to index 0
0129 03:00:16.048697 (+ 40us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0129 03:00:16.049273 (+ 576us) log.cc:844] Serialized 26 byte log entry
0129 03:00:16.573993 (+524720us) raft_consensus.cc:1645] Marking committed up to 7
0129 03:00:16.574036 (+ 43us) raft_consensus.cc:1697] Filling consensus response to leader.
0129 03:00:16.574125 (+ 89us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0129 03:00:17.122606 (+548481us) raft_consensus.cc:1685] finished
0129 03:00:17.122626 (+ 20us) raft_consensus.cc:1691] UpdateReplicas() finished
0129 03:00:17.122789 (+ 163us) inbound_call.cc:173] Queueing success response
Metrics: {"thread_start_us":524295,"threads_started":1,"wal-append.queue_time_us":738512}
I20260129 03:00:17.136075 20588 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "dc2f4cb752cf4d59b5f6bc689f5d8104" candidate_uuid: "197b880fa39d432a9b472845876bfaf0" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "1c50de89a5a845668696dcb28c3badd6" is_pre_election: true
I20260129 03:00:17.098754 23046 raft_consensus.cc:493] T 368953c9102643b2a21f99af36b51905 P 974b802141bd4f13beffef2aa03f88f9 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260129 03:00:17.088377 23466 raft_consensus.cc:493] T cb4727386a3e48aca5d71cc6c611c197 P 197b880fa39d432a9b472845876bfaf0 [term 5 FOLLOWER]: Starting leader election (detected failure of leader 1c50de89a5a845668696dcb28c3badd6)
I20260129 03:00:17.173029 20509 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "724a236524cb4e7ba3428880681d6cc2" candidate_uuid: "1c50de89a5a845668696dcb28c3badd6" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "974b802141bd4f13beffef2aa03f88f9" is_pre_election: true
I20260129 03:00:17.173391 20589 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "368953c9102643b2a21f99af36b51905" candidate_uuid: "197b880fa39d432a9b472845876bfaf0" candidate_term: 9 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "1c50de89a5a845668696dcb28c3badd6"
I20260129 03:00:17.153415 20516 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4f8f3a38f6e24ba4857875181f447442" candidate_uuid: "1c50de89a5a845668696dcb28c3badd6" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: true dest_uuid: "974b802141bd4f13beffef2aa03f88f9"
W20260129 03:00:17.132534 22996 consensus_meta.cc:287] T e33a417cf9c54cf5af73a4ee3aac4d3a P 1c50de89a5a845668696dcb28c3badd6: Time spent flushing consensus metadata: real 0.662s user 0.010s sys 0.000s
W20260129 03:00:17.175489 21102 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.211.2:34517, user_credentials={real_user=slave}} blocked reactor thread for 257952us
W20260129 03:00:17.140455 23514 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.544s user 0.003s sys 0.023s
I20260129 03:00:17.114095 23224 raft_consensus.cc:493] T 36ff33be666a4c82954731691dbcf327 P 197b880fa39d432a9b472845876bfaf0 [term 4 FOLLOWER]: Starting leader election (detected failure of leader 1c50de89a5a845668696dcb28c3badd6)
I20260129 03:00:17.176924 20509 raft_consensus.cc:2410] T 724a236524cb4e7ba3428880681d6cc2 P 974b802141bd4f13beffef2aa03f88f9 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1c50de89a5a845668696dcb28c3badd6 for term 7 because replica has last-logged OpId of term: 5 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 4 index: 3.
I20260129 03:00:17.133355 21129 raft_consensus.cc:2468] T 75b0b6e82ddc4f92b1fe1d94f9a0b73b P 7a9daf5d42a84615a6383cfec9a10608 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1c50de89a5a845668696dcb28c3badd6 in term 5.
I20260129 03:00:17.107066 23640 raft_consensus.cc:515] T 00e79d8f25a34676b1d866eb2068a2b2 P 7a9daf5d42a84615a6383cfec9a10608 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } }
I20260129 03:00:17.193384 20589 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0340229d521b432580ae926f6062e3f7" candidate_uuid: "197b880fa39d432a9b472845876bfaf0" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "1c50de89a5a845668696dcb28c3badd6"
I20260129 03:00:17.194854 20510 raft_consensus.cc:2393] T 17d6431cc77a491e8464d68db4081a51 P 974b802141bd4f13beffef2aa03f88f9 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1c50de89a5a845668696dcb28c3badd6 in current term 6: Already voted for candidate 974b802141bd4f13beffef2aa03f88f9 in this term.
I20260129 03:00:17.114226 23531 raft_consensus.cc:515] T a1a39dcd328e4268ae41b988c66b4a75 P 1c50de89a5a845668696dcb28c3badd6 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } } peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } }
I20260129 03:00:17.110294 23568 raft_consensus.cc:515] T 87d5fe48a44a408aaf03e894a67ecb2f P 7a9daf5d42a84615a6383cfec9a10608 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } }
I20260129 03:00:17.201290 20516 raft_consensus.cc:2468] T 4f8f3a38f6e24ba4857875181f447442 P 974b802141bd4f13beffef2aa03f88f9 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1c50de89a5a845668696dcb28c3badd6 in term 7.
I20260129 03:00:17.176941 23466 raft_consensus.cc:3060] T cb4727386a3e48aca5d71cc6c611c197 P 197b880fa39d432a9b472845876bfaf0 [term 5 FOLLOWER]: Advancing to term 6
W20260129 03:00:17.206707 20480 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.211.3:37735, user_credentials={real_user=slave}} blocked reactor thread for 117149us
I20260129 03:00:17.207123 23641 raft_consensus.cc:2749] T febffe4a8b394368b69b1d164308fb9a P 7a9daf5d42a84615a6383cfec9a10608 [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20260129 03:00:17.207581 23362 raft_consensus.cc:493] T 45cc5f3d2688486d9b6dbe7e666dedde P 7a9daf5d42a84615a6383cfec9a10608 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260129 03:00:17.208302 23634 log.cc:927] Time spent T febffe4a8b394368b69b1d164308fb9a P 974b802141bd4f13beffef2aa03f88f9: Append to log took a long time: real 0.211s user 0.000s sys 0.000s
I20260129 03:00:17.208362 23163 leader_election.cc:290] T 2e58e9a5a8aa4cecafb464addb4f35ed P 7a9daf5d42a84615a6383cfec9a10608 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 197b880fa39d432a9b472845876bfaf0 (127.18.211.1:42027), 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3:37735)
W20260129 03:00:17.209404 23485 consensus_meta.cc:287] T fd8311b779af4e1fa1402b2a02a602f2 P 197b880fa39d432a9b472845876bfaf0: Time spent flushing consensus metadata: real 0.586s user 0.010s sys 0.000s
I20260129 03:00:17.209937 23475 leader_election.cc:290] T 724a236524cb4e7ba3428880681d6cc2 P 197b880fa39d432a9b472845876bfaf0 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 974b802141bd4f13beffef2aa03f88f9 (127.18.211.2:34517), 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3:37735)
I20260129 03:00:17.210870 23262 leader_election.cc:290] T c891d2bff4ce4568861bbc992e91fbb0 P 1c50de89a5a845668696dcb28c3badd6 [CANDIDATE]: Term 9 election: Requested vote from peers 197b880fa39d432a9b472845876bfaf0 (127.18.211.1:42027), 974b802141bd4f13beffef2aa03f88f9 (127.18.211.2:34517)
I20260129 03:00:17.211102 23531 leader_election.cc:290] T a1a39dcd328e4268ae41b988c66b4a75 P 1c50de89a5a845668696dcb28c3badd6 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 974b802141bd4f13beffef2aa03f88f9 (127.18.211.2:34517), 197b880fa39d432a9b472845876bfaf0 (127.18.211.1:42027)
I20260129 03:00:17.218878 20581 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e7d83d4ccf684e76acb29a0dc844e1b8" candidate_uuid: "197b880fa39d432a9b472845876bfaf0" candidate_term: 8 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "1c50de89a5a845668696dcb28c3badd6" is_pre_election: true
I20260129 03:00:17.237401 20581 raft_consensus.cc:2468] T e7d83d4ccf684e76acb29a0dc844e1b8 P 1c50de89a5a845668696dcb28c3badd6 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 197b880fa39d432a9b472845876bfaf0 in term 7.
I20260129 03:00:17.218961 20589 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e33a417cf9c54cf5af73a4ee3aac4d3a" candidate_uuid: "197b880fa39d432a9b472845876bfaf0" candidate_term: 7 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "1c50de89a5a845668696dcb28c3badd6" is_pre_election: true
I20260129 03:00:17.183431 23469 raft_consensus.cc:493] T 913fc7d751ed4b7388d889aac30a5f9b P 197b880fa39d432a9b472845876bfaf0 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 974b802141bd4f13beffef2aa03f88f9)
W20260129 03:00:17.191316 23514 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.602s user 0.003s sys 0.023s
I20260129 03:00:17.103473 23086 consensus_queue.cc:260] T 0f20af7fbe5d4db89a6f17948d98912c P 197b880fa39d432a9b472845876bfaf0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 7.5, Last appended by leader: 5, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 03:00:17.194990 23224 raft_consensus.cc:3060] T 36ff33be666a4c82954731691dbcf327 P 197b880fa39d432a9b472845876bfaf0 [term 4 FOLLOWER]: Advancing to term 5
I20260129 03:00:17.224429 20510 raft_consensus.cc:1240] T 505580c93c694bbb8ea96fdf9b17ab46 P 974b802141bd4f13beffef2aa03f88f9 [term 8 FOLLOWER]: Rejecting Update request from peer 1c50de89a5a845668696dcb28c3badd6 for earlier term 7. Current term is 8. Ops: [6.8-7.9]
I20260129 03:00:17.113435 23398 consensus_queue.cc:237] T c2633a6fc34c44579de348d208b0c01e P 1c50de89a5a845668696dcb28c3badd6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 3, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 03:00:17.180366 22996 raft_consensus.cc:515] T e33a417cf9c54cf5af73a4ee3aac4d3a P 1c50de89a5a845668696dcb28c3badd6 [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 03:00:17.217582 22989 raft_consensus.cc:3060] T 8055232cf0004d40a85f3bf983fe00a9 P 974b802141bd4f13beffef2aa03f88f9 [term 5 FOLLOWER]: Advancing to term 6
I20260129 03:00:17.172835 23046 raft_consensus.cc:515] T 368953c9102643b2a21f99af36b51905 P 974b802141bd4f13beffef2aa03f88f9 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 03:00:17.221352 23627 leader_election.cc:290] T 17d6431cc77a491e8464d68db4081a51 P 197b880fa39d432a9b472845876bfaf0 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 974b802141bd4f13beffef2aa03f88f9 (127.18.211.2:34517), 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3:37735)
W20260129 03:00:17.282975 20479 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.211.1:42027, user_credentials={real_user=slave}} blocked reactor thread for 237361us
I20260129 03:00:17.271500 20514 raft_consensus.cc:2468] T 4b1876f9a70b49edb3aa74105e66d3b3 P 974b802141bd4f13beffef2aa03f88f9 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1c50de89a5a845668696dcb28c3badd6 in term 7.
I20260129 03:00:17.285126 20441 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c891d2bff4ce4568861bbc992e91fbb0" candidate_uuid: "1c50de89a5a845668696dcb28c3badd6" candidate_term: 9 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "197b880fa39d432a9b472845876bfaf0"
I20260129 03:00:17.284216 20439 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55676 (request call id 1513) took 1755 ms. Trace:
W20260129 03:00:17.287386 20553 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260129 03:00:17.303141 20482 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260129 03:00:17.242110 23141 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.727s user 0.005s sys 0.041s
I20260129 03:00:17.272465 23450 raft_consensus.cc:2749] T 2ac659d732314e1288849f08fe5e074e P 974b802141bd4f13beffef2aa03f88f9 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
W20260129 03:00:17.321748 23141 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.987s user 0.005s sys 0.042s
I20260129 03:00:17.287027 20439 rpcz_store.cc:276] 0129 03:00:15.528593 (+ 0us) service_pool.cc:168] Inserting onto call queue
0129 03:00:15.565438 (+ 36845us) service_pool.cc:225] Handling call
0129 03:00:15.651160 (+ 85722us) raft_consensus.cc:1471] Updating replica for 0 ops
0129 03:00:15.748281 (+ 97121us) raft_consensus.cc:1534] Early marking committed up to index 6
0129 03:00:16.389077 (+640796us) log.cc:844] Serialized 14 byte log entry
0129 03:00:17.283644 (+894567us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0129 03:00:17.283716 (+ 72us) raft_consensus.cc:1645] Marking committed up to 6
0129 03:00:17.283773 (+ 57us) raft_consensus.cc:1697] Filling consensus response to leader.
0129 03:00:17.283930 (+ 157us) raft_consensus.cc:1691] UpdateReplicas() finished
0129 03:00:17.284132 (+ 202us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":61672,"prepare.run_cpu_time_us":35,"prepare.run_wall_time_us":37,"raft.queue_time_us":107309,"raft.run_cpu_time_us":131,"raft.run_wall_time_us":7008,"thread_start_us":1446983,"threads_started":3,"wal-append.queue_time_us":808384}
W20260129 03:00:17.322791 23643 log.cc:927] Time spent T 0678b8b468bd43cdbc3e485900844821 P 7a9daf5d42a84615a6383cfec9a10608: Append to log took a long time: real 0.238s user 0.000s sys 0.000s
W20260129 03:00:17.653955 20507 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.594s user 0.000s sys 0.008s
W20260129 03:00:17.286764 23633 log.cc:927] Time spent T 78aa39096868415d9afc2ffcb248e198 P 197b880fa39d432a9b472845876bfaf0: Append to log took a long time: real 0.278s user 0.000s sys 0.001s
I20260129 03:00:17.235410 23485 raft_consensus.cc:515] T fd8311b779af4e1fa1402b2a02a602f2 P 197b880fa39d432a9b472845876bfaf0 [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } }
W20260129 03:00:17.789857 20482 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260129 03:00:17.790222 20436 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a1a39dcd328e4268ae41b988c66b4a75" candidate_uuid: "1c50de89a5a845668696dcb28c3badd6" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "197b880fa39d432a9b472845876bfaf0" is_pre_election: true
W20260129 03:00:17.792059 20404 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260129 03:00:17.819115 20436 raft_consensus.cc:2393] T a1a39dcd328e4268ae41b988c66b4a75 P 197b880fa39d432a9b472845876bfaf0 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1c50de89a5a845668696dcb28c3badd6 in current term 6: Already voted for candidate 197b880fa39d432a9b472845876bfaf0 in this term.
W20260129 03:00:17.331089 23639 log.cc:927] Time spent T 4e3805ffa70242e8ade66da260dddf66 P 1c50de89a5a845668696dcb28c3badd6: Append to log took a long time: real 0.276s user 0.000s sys 0.001s
W20260129 03:00:17.587823 20405 thread.cc:641] dns-resolver [worker] (thread pool) Time spent creating pthread: real 0.602s user 0.006s sys 0.001s
I20260129 03:00:17.273152 23469 raft_consensus.cc:515] T 913fc7d751ed4b7388d889aac30a5f9b P 197b880fa39d432a9b472845876bfaf0 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 03:00:17.283959 23403 raft_consensus.cc:493] T a5a05fbcbbcf4916bbc01fd1a5b5e90d P 197b880fa39d432a9b472845876bfaf0 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 1c50de89a5a845668696dcb28c3badd6)
W20260129 03:00:17.582676 20407 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.523s user 0.002s sys 0.006s
W20260129 03:00:17.844936 20407 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.785s user 0.004s sys 0.008s
W20260129 03:00:17.845360 20407 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.211.2:34517, user_credentials={real_user=slave}} blocked reactor thread for 786005us
I20260129 03:00:17.228127 23362 raft_consensus.cc:515] T 45cc5f3d2688486d9b6dbe7e666dedde P 7a9daf5d42a84615a6383cfec9a10608 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } }
I20260129 03:00:17.334223 23333 leader_election.cc:290] T 87d5fe48a44a408aaf03e894a67ecb2f P 974b802141bd4f13beffef2aa03f88f9 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 197b880fa39d432a9b472845876bfaf0 (127.18.211.1:42027), 7a9daf5d42a84615a6383cfec9a10608 (127.18.211.4:40439)
I20260129 03:00:17.349191 20441 raft_consensus.cc:2393] T c891d2bff4ce4568861bbc992e91fbb0 P 197b880fa39d432a9b472845876bfaf0 [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate 1c50de89a5a845668696dcb28c3badd6 in current term 9: Already voted for candidate 197b880fa39d432a9b472845876bfaf0 in this term.
I20260129 03:00:17.795722 20581 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c2633a6fc34c44579de348d208b0c01e" candidate_uuid: "197b880fa39d432a9b472845876bfaf0" candidate_term: 6 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "1c50de89a5a845668696dcb28c3badd6" is_pre_election: true
I20260129 03:00:17.796327 20588 raft_consensus.cc:1275] T 926a250fa5834e88b96ccecbefbf5086 P 1c50de89a5a845668696dcb28c3badd6 [term 8 FOLLOWER]: Refusing update from remote peer 197b880fa39d432a9b472845876bfaf0: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 8 index: 5. (index mismatch)
W20260129 03:00:17.799151 20511 consensus_meta.cc:287] T c934d31efd3f4e5195605ddc354d14e1 P 974b802141bd4f13beffef2aa03f88f9: Time spent flushing consensus metadata: real 1.221s user 0.014s sys 0.013s
I20260129 03:00:17.869159 20511 raft_consensus.cc:2468] T c934d31efd3f4e5195605ddc354d14e1 P 974b802141bd4f13beffef2aa03f88f9 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 197b880fa39d432a9b472845876bfaf0 in term 7.
W20260129 03:00:17.330447 23656 log.cc:927] Time spent T 4b1876f9a70b49edb3aa74105e66d3b3 P 1c50de89a5a845668696dcb28c3badd6: Append to log took a long time: real 0.105s user 0.000s sys 0.001s
I20260129 03:00:17.800375 23640 leader_election.cc:290] T 00e79d8f25a34676b1d866eb2068a2b2 P 7a9daf5d42a84615a6383cfec9a10608 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 197b880fa39d432a9b472845876bfaf0 (127.18.211.1:42027), 974b802141bd4f13beffef2aa03f88f9 (127.18.211.2:34517)
I20260129 03:00:17.801669 20500 tablet_service.cc:1552] Processing DeleteTablet for tablet 1299723a4476420ab02fd8bfd7aa0645 with delete_type TABLET_DATA_TOMBSTONED (TS 974b802141bd4f13beffef2aa03f88f9 not found in new config with opid_index 5) from {username='slave'} at 127.0.0.1:57580
I20260129 03:00:17.794090 19276 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260129 03:00:17.885960 20440 raft_consensus.cc:1240] T fd8311b779af4e1fa1402b2a02a602f2 P 197b880fa39d432a9b472845876bfaf0 [term 6 FOLLOWER]: Rejecting Update request from peer 1c50de89a5a845668696dcb28c3badd6 for earlier term 5. Current term is 6. Ops: []
I20260129 03:00:17.328662 23555 leader_election.cc:290] T 7a9a69cd82724b1faed1d10aa091fc4a P 974b802141bd4f13beffef2aa03f88f9 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 197b880fa39d432a9b472845876bfaf0 (127.18.211.1:42027), 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3:37735)
W20260129 03:00:17.887658 20480 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.211.3:37735, user_credentials={real_user=slave}} blocked reactor thread for 659703us
W20260129 03:00:17.668109 23602 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.608s user 0.010s sys 0.006s
W20260129 03:00:17.889035 23602 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.829s user 0.013s sys 0.008s
W20260129 03:00:17.733415 20438 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.612s user 0.011s sys 0.007s
W20260129 03:00:17.889854 20438 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.784s user 0.011s sys 0.008s
W20260129 03:00:17.807307 20507 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.758s user 0.000s sys 0.010s
I20260129 03:00:17.817348 20512 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57600 (request call id 2018) took 1027 ms. Trace:
I20260129 03:00:17.818295 20589 raft_consensus.cc:2368] T e33a417cf9c54cf5af73a4ee3aac4d3a P 1c50de89a5a845668696dcb28c3badd6 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 197b880fa39d432a9b472845876bfaf0 for earlier term 7. Current term is 8.
W20260129 03:00:17.818301 21132 consensus_meta.cc:287] T 75f45538dc704d9a8ec4ff8ad9f3ea5b P 7a9daf5d42a84615a6383cfec9a10608: Time spent flushing consensus metadata: real 1.005s user 0.017s sys 0.010s
W20260129 03:00:17.823568 21135 consensus_meta.cc:287] T 2e58e9a5a8aa4cecafb464addb4f35ed P 7a9daf5d42a84615a6383cfec9a10608: Time spent flushing consensus metadata: real 0.811s user 0.017s sys 0.009s
W20260129 03:00:17.827518 20405 thread.cc:608] dns-resolver [worker] (thread pool) Time spent starting thread: real 0.842s user 0.009s sys 0.001s
I20260129 03:00:17.827229 20435 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3ccac82a01a2445fa5b0ce6c0fcf2295" candidate_uuid: "974b802141bd4f13beffef2aa03f88f9" candidate_term: 5 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "197b880fa39d432a9b472845876bfaf0" is_pre_election: true
I20260129 03:00:17.846007 20586 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0f20af7fbe5d4db89a6f17948d98912c" candidate_uuid: "974b802141bd4f13beffef2aa03f88f9" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "1c50de89a5a845668696dcb28c3badd6"
W20260129 03:00:17.899024 20553 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260129 03:00:17.900905 21135 raft_consensus.cc:2468] T 2e58e9a5a8aa4cecafb464addb4f35ed P 7a9daf5d42a84615a6383cfec9a10608 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1c50de89a5a845668696dcb28c3badd6 in term 5.
I20260129 03:00:17.900594 21132 raft_consensus.cc:2468] T 75f45538dc704d9a8ec4ff8ad9f3ea5b P 7a9daf5d42a84615a6383cfec9a10608 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1c50de89a5a845668696dcb28c3badd6 in term 5.
I20260129 03:00:17.901871 21132 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34046 (request call id 611) took 1113 ms. Trace:
I20260129 03:00:17.902163 21132 rpcz_store.cc:276] 0129 03:00:16.788501 (+ 0us) service_pool.cc:168] Inserting onto call queue
0129 03:00:16.799177 (+ 10676us) service_pool.cc:225] Handling call
0129 03:00:17.901814 (+1102637us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20260129 03:00:17.904134 23653 log.cc:927] Time spent T cedce331e1584ac8b636c9b8a72b2ba0 P 197b880fa39d432a9b472845876bfaf0: Append to log took a long time: real 0.085s user 0.001s sys 0.000s
I20260129 03:00:17.904848 20587 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fcddab6bca684adf8ea032289a00f90e" candidate_uuid: "197b880fa39d432a9b472845876bfaf0" candidate_term: 11 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "1c50de89a5a845668696dcb28c3badd6" is_pre_election: true
W20260129 03:00:17.903116 20405 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.211.3:37735, user_credentials={real_user=slave}} blocked reactor thread for 969067us
I20260129 03:00:17.906551 20441 raft_consensus.cc:1240] T c2633a6fc34c44579de348d208b0c01e P 197b880fa39d432a9b472845876bfaf0 [term 6 FOLLOWER]: Rejecting Update request from peer 974b802141bd4f13beffef2aa03f88f9 for earlier term 5. Current term is 6. Ops: [5.3-5.3]
W20260129 03:00:17.863302 20479 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260129 03:00:17.870530 20407 leader_election.cc:304] T a1a39dcd328e4268ae41b988c66b4a75 P 197b880fa39d432a9b472845876bfaf0 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 197b880fa39d432a9b472845876bfaf0, 974b802141bd4f13beffef2aa03f88f9; no voters:
I20260129 03:00:17.911054 20587 raft_consensus.cc:2468] T fcddab6bca684adf8ea032289a00f90e P 1c50de89a5a845668696dcb28c3badd6 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 197b880fa39d432a9b472845876bfaf0 in term 10.
I20260129 03:00:17.911988 20582 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4408006eb5174de1896b037cf3f18457" candidate_uuid: "974b802141bd4f13beffef2aa03f88f9" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "1c50de89a5a845668696dcb28c3badd6" is_pre_election: true
W20260129 03:00:17.880753 21136 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.604s user 0.009s sys 0.036s
I20260129 03:00:17.882863 20511 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:57600 (request call id 2013) took 1301 ms. Trace:
W20260129 03:00:17.913532 21136 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.641s user 0.010s sys 0.037s
I20260129 03:00:17.913730 20511 rpcz_store.cc:276] 0129 03:00:16.568248 (+ 0us) service_pool.cc:168] Inserting onto call queue
0129 03:00:16.568486 (+ 238us) service_pool.cc:225] Handling call
0129 03:00:17.869763 (+1301277us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260129 03:00:17.884850 20583 raft_consensus.cc:1240] T 4f8f3a38f6e24ba4857875181f447442 P 1c50de89a5a845668696dcb28c3badd6 [term 7 FOLLOWER]: Rejecting Update request from peer 974b802141bd4f13beffef2aa03f88f9 for earlier term 6. Current term is 7. Ops: []
I20260129 03:00:17.800385 23086 raft_consensus.cc:3060] T 0f20af7fbe5d4db89a6f17948d98912c P 197b880fa39d432a9b472845876bfaf0 [term 7 FOLLOWER]: Advancing to term 8
I20260129 03:00:17.916234 20407 leader_election.cc:304] T 3906b080a5b34a74b8b64128635e8997 P 197b880fa39d432a9b472845876bfaf0 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 197b880fa39d432a9b472845876bfaf0, 974b802141bd4f13beffef2aa03f88f9; no voters:
I20260129 03:00:17.916436 20587 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "17d6431cc77a491e8464d68db4081a51" candidate_uuid: "974b802141bd4f13beffef2aa03f88f9" candidate_term: 6 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "1c50de89a5a845668696dcb28c3badd6"
I20260129 03:00:17.917084 20587 raft_consensus.cc:3060] T 17d6431cc77a491e8464d68db4081a51 P 1c50de89a5a845668696dcb28c3badd6 [term 5 FOLLOWER]: Advancing to term 6
I20260129 03:00:17.917388 20588 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "75be4bd6bd45474795fd472eff2953ac" candidate_uuid: "974b802141bd4f13beffef2aa03f88f9" candidate_term: 11 candidate_status { last_received { term: 10 index: 7 } } ignore_live_leader: false dest_uuid: "1c50de89a5a845668696dcb28c3badd6" is_pre_election: true
I20260129 03:00:17.890720 20512 rpcz_store.cc:276] 0129 03:00:16.775914 (+ 0us) service_pool.cc:168] Inserting onto call queue
0129 03:00:16.776089 (+ 175us) service_pool.cc:225] Handling call
0129 03:00:16.786143 (+ 10054us) raft_consensus.cc:1471] Updating replica for 0 ops
0129 03:00:16.786381 (+ 238us) raft_consensus.cc:1534] Early marking committed up to index 5
0129 03:00:17.110223 (+323842us) log.cc:844] Serialized 14 byte log entry
0129 03:00:17.803273 (+693050us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0129 03:00:17.803336 (+ 63us) raft_consensus.cc:1645] Marking committed up to 5
0129 03:00:17.803391 (+ 55us) raft_consensus.cc:1697] Filling consensus response to leader.
0129 03:00:17.803466 (+ 75us) raft_consensus.cc:1691] UpdateReplicas() finished
0129 03:00:17.803593 (+ 127us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":455696,"prepare.run_wall_time_us":40,"raft.queue_time_us":74586,"raft.run_cpu_time_us":218,"raft.run_wall_time_us":3273,"spinlock_wait_cycles":77579264,"thread_start_us":476430,"threads_started":2,"wal-append.queue_time_us":673525}
I20260129 03:00:17.918026 20588 raft_consensus.cc:2468] T 75be4bd6bd45474795fd472eff2953ac P 1c50de89a5a845668696dcb28c3badd6 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 974b802141bd4f13beffef2aa03f88f9 in term 10.
W20260129 03:00:17.889642 22932 consensus_meta.cc:287] T add8d4923d1a489e86882c3a25692b95 P 7a9daf5d42a84615a6383cfec9a10608: Time spent flushing consensus metadata: real 1.272s user 0.013s sys 0.007s
I20260129 03:00:17.922962 20439 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "724a236524cb4e7ba3428880681d6cc2" candidate_uuid: "974b802141bd4f13beffef2aa03f88f9" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "197b880fa39d432a9b472845876bfaf0" is_pre_election: true
I20260129 03:00:17.916839 23680 raft_consensus.cc:493] T 1ce4ddf5b10e4ae8b193b13b2e5936ee P 1c50de89a5a845668696dcb28c3badd6 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 197b880fa39d432a9b472845876bfaf0)
I20260129 03:00:17.930434 20586 raft_consensus.cc:3060] T 0f20af7fbe5d4db89a6f17948d98912c P 1c50de89a5a845668696dcb28c3badd6 [term 7 FOLLOWER]: Advancing to term 8
I20260129 03:00:17.936396 21133 raft_consensus.cc:2410] T add8d4923d1a489e86882c3a25692b95 P 7a9daf5d42a84615a6383cfec9a10608 [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate 1c50de89a5a845668696dcb28c3badd6 for term 8 because replica has last-logged OpId of term: 7 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 5.
I20260129 03:00:17.936524 20441 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4f8f3a38f6e24ba4857875181f447442" candidate_uuid: "974b802141bd4f13beffef2aa03f88f9" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "197b880fa39d432a9b472845876bfaf0" is_pre_election: true
I20260129 03:00:17.936621 20435 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a5a05fbcbbcf4916bbc01fd1a5b5e90d" candidate_uuid: "974b802141bd4f13beffef2aa03f88f9" candidate_term: 9 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "197b880fa39d432a9b472845876bfaf0" is_pre_election: true
I20260129 03:00:17.925149 20439 raft_consensus.cc:2468] T 724a236524cb4e7ba3428880681d6cc2 P 197b880fa39d432a9b472845876bfaf0 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 974b802141bd4f13beffef2aa03f88f9 in term 6.
I20260129 03:00:17.944830 20581 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b189f08e16a74a11980cc55d99359a0a" candidate_uuid: "974b802141bd4f13beffef2aa03f88f9" candidate_term: 10 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "1c50de89a5a845668696dcb28c3badd6" is_pre_election: true
I20260129 03:00:17.945508 20441 raft_consensus.cc:2468] T 4f8f3a38f6e24ba4857875181f447442 P 197b880fa39d432a9b472845876bfaf0 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 974b802141bd4f13beffef2aa03f88f9 in term 7.
I20260129 03:00:17.946333 20583 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "36ff33be666a4c82954731691dbcf327" candidate_uuid: "974b802141bd4f13beffef2aa03f88f9" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "1c50de89a5a845668696dcb28c3badd6" is_pre_election: true
I20260129 03:00:17.950716 21133 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34046 (request call id 614) took 1161 ms. Trace:
I20260129 03:00:17.951273 20582 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3e2d15f8640b407195210c476e09df91" candidate_uuid: "974b802141bd4f13beffef2aa03f88f9" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "1c50de89a5a845668696dcb28c3badd6" is_pre_election: true
I20260129 03:00:17.952186 20588 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "45cc5f3d2688486d9b6dbe7e666dedde" candidate_uuid: "974b802141bd4f13beffef2aa03f88f9" candidate_term: 7 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "1c50de89a5a845668696dcb28c3badd6"
I20260129 03:00:17.952939 20588 raft_consensus.cc:3060] T 45cc5f3d2688486d9b6dbe7e666dedde P 1c50de89a5a845668696dcb28c3badd6 [term 6 FOLLOWER]: Advancing to term 7
I20260129 03:00:17.951134 21133 rpcz_store.cc:276] 0129 03:00:16.789275 (+ 0us) service_pool.cc:168] Inserting onto call queue
0129 03:00:16.789563 (+ 288us) service_pool.cc:225] Handling call
0129 03:00:17.944832 (+1155269us) spinlock_profiling.cc:243] Waited 1.13 s on lock 0x7b54005712d4. stack: 00007f8fe044e373 00007f8fe044e270 00007f8fdfda0574 00007f8fe9a5c6b4 00007f8fe9a5c649 00007f8fe34975a1 00007f8fe3531f7a 00007f8fe84e3647 00007f8fe1a73834 00007f8fe1a737c2 00007f8fe1a736f1 00007f8fe1a7366c 00007f8fe1a72922 00007f8fe186d34c 00007f8fe186c786 00007f8fe186c12f
0129 03:00:17.950623 (+ 5791us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2590994176}
I20260129 03:00:17.912248 23602 raft_consensus.cc:493] T 368953c9102643b2a21f99af36b51905 P 1c50de89a5a845668696dcb28c3badd6 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 974b802141bd4f13beffef2aa03f88f9)
W20260129 03:00:17.964308 20405 leader_election.cc:336] T 8055232cf0004d40a85f3bf983fe00a9 P 197b880fa39d432a9b472845876bfaf0 [CANDIDATE]: Term 6 election: RPC error from VoteRequest() call to peer 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3:37735): Timed out: RequestConsensusVote RPC to 127.18.211.3:37735 timed out after 2.084s (SENT)
I20260129 03:00:17.968808 20441 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "926a250fa5834e88b96ccecbefbf5086" candidate_uuid: "974b802141bd4f13beffef2aa03f88f9" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "197b880fa39d432a9b472845876bfaf0" is_pre_election: true
I20260129 03:00:17.933977 23680 raft_consensus.cc:515] T 1ce4ddf5b10e4ae8b193b13b2e5936ee P 1c50de89a5a845668696dcb28c3badd6 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 03:00:17.970865 20439 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a1a39dcd328e4268ae41b988c66b4a75" candidate_uuid: "974b802141bd4f13beffef2aa03f88f9" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "197b880fa39d432a9b472845876bfaf0" is_pre_election: true
I20260129 03:00:17.971684 20439 raft_consensus.cc:2468] T a1a39dcd328e4268ae41b988c66b4a75 P 197b880fa39d432a9b472845876bfaf0 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 974b802141bd4f13beffef2aa03f88f9 in term 6.
I20260129 03:00:17.881633 23403 raft_consensus.cc:515] T a5a05fbcbbcf4916bbc01fd1a5b5e90d P 197b880fa39d432a9b472845876bfaf0 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 03:00:17.977593 20405 leader_election.cc:304] T 8055232cf0004d40a85f3bf983fe00a9 P 197b880fa39d432a9b472845876bfaf0 [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 197b880fa39d432a9b472845876bfaf0; no voters: 1c50de89a5a845668696dcb28c3badd6, 974b802141bd4f13beffef2aa03f88f9
I20260129 03:00:17.977943 20581 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "75b0b6e82ddc4f92b1fe1d94f9a0b73b" candidate_uuid: "974b802141bd4f13beffef2aa03f88f9" candidate_term: 5 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "1c50de89a5a845668696dcb28c3badd6" is_pre_election: true
I20260129 03:00:17.982123 20581 raft_consensus.cc:2393] T 75b0b6e82ddc4f92b1fe1d94f9a0b73b P 1c50de89a5a845668696dcb28c3badd6 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 974b802141bd4f13beffef2aa03f88f9 in current term 5: Already voted for candidate 1c50de89a5a845668696dcb28c3badd6 in this term.
I20260129 03:00:17.984654 20513 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57600 (request call id 2006) took 1457 ms. Trace:
I20260129 03:00:17.981925 20434 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4b1876f9a70b49edb3aa74105e66d3b3" candidate_uuid: "974b802141bd4f13beffef2aa03f88f9" candidate_term: 7 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "197b880fa39d432a9b472845876bfaf0" is_pre_election: true
I20260129 03:00:17.985126 20513 rpcz_store.cc:276] 0129 03:00:16.527365 (+ 0us) service_pool.cc:168] Inserting onto call queue
0129 03:00:16.534598 (+ 7233us) service_pool.cc:225] Handling call
0129 03:00:16.543963 (+ 9365us) raft_consensus.cc:1471] Updating replica for 0 ops
0129 03:00:16.552843 (+ 8880us) raft_consensus.cc:1534] Early marking committed up to index 5
0129 03:00:17.038742 (+485899us) log.cc:844] Serialized 14 byte log entry
0129 03:00:17.807915 (+769173us) log.cc:844] Serialized 14 byte log entry
0129 03:00:17.983291 (+175376us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0129 03:00:17.983340 (+ 49us) raft_consensus.cc:1645] Marking committed up to 5
0129 03:00:17.983393 (+ 53us) raft_consensus.cc:1697] Filling consensus response to leader.
0129 03:00:17.984488 (+ 1095us) raft_consensus.cc:1691] UpdateReplicas() finished
0129 03:00:17.984599 (+ 111us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":79777,"prepare.queue_time_us":1257304,"prepare.run_cpu_time_us":100,"prepare.run_wall_time_us":103,"raft.queue_time_us":55280,"spinlock_wait_cycles":16767360,"thread_start_us":805645,"threads_started":3,"wal-append.queue_time_us":911268}
I20260129 03:00:17.988786 20583 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "8055232cf0004d40a85f3bf983fe00a9" candidate_uuid: "197b880fa39d432a9b472845876bfaf0" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "1c50de89a5a845668696dcb28c3badd6"
I20260129 03:00:17.992624 20434 raft_consensus.cc:2393] T 4b1876f9a70b49edb3aa74105e66d3b3 P 197b880fa39d432a9b472845876bfaf0 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 974b802141bd4f13beffef2aa03f88f9 in current term 7: Already voted for candidate 1c50de89a5a845668696dcb28c3badd6 in this term.
I20260129 03:00:17.995251 20583 raft_consensus.cc:2376] T 8055232cf0004d40a85f3bf983fe00a9 P 1c50de89a5a845668696dcb28c3badd6 [term 6 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 197b880fa39d432a9b472845876bfaf0 in term 6. Re-sending same reply.
W20260129 03:00:17.982400 23480 consensus_meta.cc:287] T a644c02b16af47679824ccd060ac10fd P 7a9daf5d42a84615a6383cfec9a10608: Time spent flushing consensus metadata: real 1.502s user 0.016s sys 0.014s
I20260129 03:00:17.996446 20581 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e4d18c35b0a7493e85d0a4b92956a48f" candidate_uuid: "197b880fa39d432a9b472845876bfaf0" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "1c50de89a5a845668696dcb28c3badd6" is_pre_election: true
I20260129 03:00:17.997467 20435 raft_consensus.cc:2410] T a5a05fbcbbcf4916bbc01fd1a5b5e90d P 197b880fa39d432a9b472845876bfaf0 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 974b802141bd4f13beffef2aa03f88f9 for term 9 because replica has last-logged OpId of term: 8 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 7 index: 5.
I20260129 03:00:17.964552 23568 leader_election.cc:290] T 87d5fe48a44a408aaf03e894a67ecb2f P 7a9daf5d42a84615a6383cfec9a10608 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 197b880fa39d432a9b472845876bfaf0 (127.18.211.1:42027), 974b802141bd4f13beffef2aa03f88f9 (127.18.211.2:34517)
W20260129 03:00:18.066859 20555 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260129 03:00:18.075834 20590 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:48526 (request call id 430) took 1064 ms. Trace:
I20260129 03:00:18.058414 22996 leader_election.cc:290] T e33a417cf9c54cf5af73a4ee3aac4d3a P 1c50de89a5a845668696dcb28c3badd6 [CANDIDATE]: Term 8 election: Requested vote from peers 197b880fa39d432a9b472845876bfaf0 (127.18.211.1:42027), 974b802141bd4f13beffef2aa03f88f9 (127.18.211.2:34517)
I20260129 03:00:18.003456 20582 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7d9494f5584d4d5db8b9adf1c8fc888b" candidate_uuid: "974b802141bd4f13beffef2aa03f88f9" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "1c50de89a5a845668696dcb28c3badd6" is_pre_election: true
I20260129 03:00:18.056002 20441 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3906b080a5b34a74b8b64128635e8997" candidate_uuid: "974b802141bd4f13beffef2aa03f88f9" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "197b880fa39d432a9b472845876bfaf0" is_pre_election: true
I20260129 03:00:18.068292 20583 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "724a236524cb4e7ba3428880681d6cc2" candidate_uuid: "974b802141bd4f13beffef2aa03f88f9" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "1c50de89a5a845668696dcb28c3badd6" is_pre_election: true
I20260129 03:00:18.095573 20590 rpcz_store.cc:276] 0129 03:00:17.011468 (+ 0us) service_pool.cc:168] Inserting onto call queue
0129 03:00:18.067182 (+1055714us) service_pool.cc:225] Handling call
0129 03:00:18.075145 (+ 7963us) raft_consensus.cc:1471] Updating replica for 0 ops
0129 03:00:18.075385 (+ 240us) raft_consensus.cc:1534] Early marking committed up to index 0
0129 03:00:18.075424 (+ 39us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0129 03:00:18.075481 (+ 57us) raft_consensus.cc:1645] Marking committed up to 6
0129 03:00:18.075530 (+ 49us) raft_consensus.cc:1697] Filling consensus response to leader.
0129 03:00:18.075624 (+ 94us) raft_consensus.cc:1691] UpdateReplicas() finished
0129 03:00:18.075762 (+ 138us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260129 03:00:17.979368 23602 raft_consensus.cc:515] T 368953c9102643b2a21f99af36b51905 P 1c50de89a5a845668696dcb28c3badd6 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 03:00:18.091440 20581 raft_consensus.cc:2468] T e4d18c35b0a7493e85d0a4b92956a48f P 1c50de89a5a845668696dcb28c3badd6 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 197b880fa39d432a9b472845876bfaf0 in term 5.
I20260129 03:00:18.087544 20587 raft_consensus.cc:2468] T 17d6431cc77a491e8464d68db4081a51 P 1c50de89a5a845668696dcb28c3badd6 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 974b802141bd4f13beffef2aa03f88f9 in term 6.
I20260129 03:00:18.056787 20432 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a0fcc58d6fb6473cafbd9d34d5195269" candidate_uuid: "974b802141bd4f13beffef2aa03f88f9" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "197b880fa39d432a9b472845876bfaf0" is_pre_election: true
I20260129 03:00:18.096894 20583 raft_consensus.cc:2468] T 724a236524cb4e7ba3428880681d6cc2 P 1c50de89a5a845668696dcb28c3badd6 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 974b802141bd4f13beffef2aa03f88f9 in term 6.
I20260129 03:00:18.055548 22645 raft_consensus.cc:2749] T 8055232cf0004d40a85f3bf983fe00a9 P 197b880fa39d432a9b472845876bfaf0 [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20260129 03:00:18.097080 20582 raft_consensus.cc:2468] T 7d9494f5584d4d5db8b9adf1c8fc888b P 1c50de89a5a845668696dcb28c3badd6 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 974b802141bd4f13beffef2aa03f88f9 in term 5.
I20260129 03:00:18.101939 20439 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "cedce331e1584ac8b636c9b8a72b2ba0" candidate_uuid: "974b802141bd4f13beffef2aa03f88f9" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "197b880fa39d432a9b472845876bfaf0" is_pre_election: true
W20260129 03:00:18.087780 22989 consensus_meta.cc:287] T 8055232cf0004d40a85f3bf983fe00a9 P 974b802141bd4f13beffef2aa03f88f9: Time spent flushing consensus metadata: real 0.802s user 0.031s sys 0.005s
I20260129 03:00:18.111339 20587 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4f8f3a38f6e24ba4857875181f447442" candidate_uuid: "974b802141bd4f13beffef2aa03f88f9" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "1c50de89a5a845668696dcb28c3badd6" is_pre_election: true
W20260129 03:00:18.114583 21099 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.211.1:42027, user_credentials={real_user=slave}} blocked reactor thread for 147419us
I20260129 03:00:18.076630 23046 leader_election.cc:290] T 368953c9102643b2a21f99af36b51905 P 974b802141bd4f13beffef2aa03f88f9 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 197b880fa39d432a9b472845876bfaf0 (127.18.211.1:42027), 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3:37735)
I20260129 03:00:18.116487 20587 raft_consensus.cc:2468] T 4f8f3a38f6e24ba4857875181f447442 P 1c50de89a5a845668696dcb28c3badd6 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 974b802141bd4f13beffef2aa03f88f9 in term 7.
I20260129 03:00:18.060256 23480 raft_consensus.cc:515] T a644c02b16af47679824ccd060ac10fd P 7a9daf5d42a84615a6383cfec9a10608 [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } }
I20260129 03:00:18.121573 20586 raft_consensus.cc:2468] T 0f20af7fbe5d4db89a6f17948d98912c P 1c50de89a5a845668696dcb28c3badd6 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 974b802141bd4f13beffef2aa03f88f9 in term 8.
W20260129 03:00:18.120110 20407 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.211.2:34517, user_credentials={real_user=slave}} blocked reactor thread for 203956us
I20260129 03:00:18.123754 20590 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a5a05fbcbbcf4916bbc01fd1a5b5e90d" candidate_uuid: "974b802141bd4f13beffef2aa03f88f9" candidate_term: 9 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "1c50de89a5a845668696dcb28c3badd6" is_pre_election: true
I20260129 03:00:18.122822 23362 leader_election.cc:290] T 45cc5f3d2688486d9b6dbe7e666dedde P 7a9daf5d42a84615a6383cfec9a10608 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 974b802141bd4f13beffef2aa03f88f9 (127.18.211.2:34517), 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3:37735)
I20260129 03:00:18.120105 23485 leader_election.cc:290] T fd8311b779af4e1fa1402b2a02a602f2 P 197b880fa39d432a9b472845876bfaf0 [CANDIDATE]: Term 6 election: Requested vote from peers 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3:37735), 7a9daf5d42a84615a6383cfec9a10608 (127.18.211.4:40439)
I20260129 03:00:18.125800 23480 leader_election.cc:290] T a644c02b16af47679824ccd060ac10fd P 7a9daf5d42a84615a6383cfec9a10608 [CANDIDATE]: Term 6 election: Requested vote from peers 974b802141bd4f13beffef2aa03f88f9 (127.18.211.2:34517), 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3:37735)
I20260129 03:00:18.127136 20587 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a1a39dcd328e4268ae41b988c66b4a75" candidate_uuid: "974b802141bd4f13beffef2aa03f88f9" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "1c50de89a5a845668696dcb28c3badd6" is_pre_election: true
I20260129 03:00:18.135174 20587 raft_consensus.cc:2468] T a1a39dcd328e4268ae41b988c66b4a75 P 1c50de89a5a845668696dcb28c3badd6 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 974b802141bd4f13beffef2aa03f88f9 in term 5.
I20260129 03:00:18.137056 20583 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "96609ac68bdf4993a12e1c23355ca013" candidate_uuid: "197b880fa39d432a9b472845876bfaf0" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "1c50de89a5a845668696dcb28c3badd6" is_pre_election: true
I20260129 03:00:18.128894 20439 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7a9a69cd82724b1faed1d10aa091fc4a" candidate_uuid: "974b802141bd4f13beffef2aa03f88f9" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "197b880fa39d432a9b472845876bfaf0" is_pre_election: true
I20260129 03:00:18.125124 20432 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "87d5fe48a44a408aaf03e894a67ecb2f" candidate_uuid: "974b802141bd4f13beffef2aa03f88f9" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "197b880fa39d432a9b472845876bfaf0" is_pre_election: true
I20260129 03:00:18.131722 20586 raft_consensus.cc:1240] T 17d6431cc77a491e8464d68db4081a51 P 1c50de89a5a845668696dcb28c3badd6 [term 6 FOLLOWER]: Rejecting Update request from peer 197b880fa39d432a9b472845876bfaf0 for earlier term 5. Current term is 6. Ops: []
I20260129 03:00:18.124331 20407 leader_election.cc:304] T 7d9494f5584d4d5db8b9adf1c8fc888b P 197b880fa39d432a9b472845876bfaf0 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 197b880fa39d432a9b472845876bfaf0, 974b802141bd4f13beffef2aa03f88f9; no voters:
I20260129 03:00:18.131435 20582 raft_consensus.cc:1240] T 8055232cf0004d40a85f3bf983fe00a9 P 1c50de89a5a845668696dcb28c3badd6 [term 6 FOLLOWER]: Rejecting Update request from peer 974b802141bd4f13beffef2aa03f88f9 for earlier term 5. Current term is 6. Ops: []
W20260129 03:00:18.116210 23224 consensus_meta.cc:287] T 36ff33be666a4c82954731691dbcf327 P 197b880fa39d432a9b472845876bfaf0: Time spent flushing consensus metadata: real 0.828s user 0.015s sys 0.007s
I20260129 03:00:18.145605 20581 raft_consensus.cc:1275] T fcddab6bca684adf8ea032289a00f90e P 1c50de89a5a845668696dcb28c3badd6 [term 10 FOLLOWER]: Refusing update from remote peer 974b802141bd4f13beffef2aa03f88f9: Log matching property violated. Preceding OpId in replica: term: 4 index: 4. Preceding OpId from leader: term: 10 index: 6. (index mismatch)
W20260129 03:00:18.148753 20405 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.211.3:37735, user_credentials={real_user=slave}} blocked reactor thread for 206722us
I20260129 03:00:18.150467 23665 raft_consensus.cc:2804] T 3906b080a5b34a74b8b64128635e8997 P 197b880fa39d432a9b472845876bfaf0 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260129 03:00:18.155031 23669 raft_consensus.cc:493] T 3906b080a5b34a74b8b64128635e8997 P 1c50de89a5a845668696dcb28c3badd6 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260129 03:00:18.155341 23670 log.cc:927] Time spent T 926a250fa5834e88b96ccecbefbf5086 P 974b802141bd4f13beffef2aa03f88f9: Append to log took a long time: real 0.077s user 0.000s sys 0.000s
I20260129 03:00:18.155778 23669 raft_consensus.cc:515] T 3906b080a5b34a74b8b64128635e8997 P 1c50de89a5a845668696dcb28c3badd6 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
W20260129 03:00:18.156526 23683 log.cc:927] Time spent T a609ac02fae84725ab743f95f51d33ce P 974b802141bd4f13beffef2aa03f88f9: Append to log took a long time: real 0.201s user 0.001s sys 0.001s
I20260129 03:00:18.140803 23674 raft_consensus.cc:2804] T a1a39dcd328e4268ae41b988c66b4a75 P 197b880fa39d432a9b472845876bfaf0 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260129 03:00:18.157331 20583 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "926a250fa5834e88b96ccecbefbf5086" candidate_uuid: "974b802141bd4f13beffef2aa03f88f9" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "1c50de89a5a845668696dcb28c3badd6" is_pre_election: true
W20260129 03:00:18.158039 23690 log.cc:927] Time spent T b189f08e16a74a11980cc55d99359a0a P 1c50de89a5a845668696dcb28c3badd6: Append to log took a long time: real 0.070s user 0.000s sys 0.000s
I20260129 03:00:18.158380 20583 raft_consensus.cc:2468] T 926a250fa5834e88b96ccecbefbf5086 P 1c50de89a5a845668696dcb28c3badd6 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 974b802141bd4f13beffef2aa03f88f9 in term 8.
W20260129 03:00:18.158736 23647 log.cc:927] Time spent T 505580c93c694bbb8ea96fdf9b17ab46 P 1c50de89a5a845668696dcb28c3badd6: Append to log took a long time: real 0.799s user 0.000s sys 0.011s
W20260129 03:00:18.159346 23672 log.cc:927] Time spent T 78aa39096868415d9afc2ffcb248e198 P 7a9daf5d42a84615a6383cfec9a10608: Append to log took a long time: real 0.053s user 0.000s sys 0.000s
I20260129 03:00:18.159582 20432 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e33a417cf9c54cf5af73a4ee3aac4d3a" candidate_uuid: "1c50de89a5a845668696dcb28c3badd6" candidate_term: 8 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "197b880fa39d432a9b472845876bfaf0"
I20260129 03:00:18.160250 20432 raft_consensus.cc:3060] T e33a417cf9c54cf5af73a4ee3aac4d3a P 197b880fa39d432a9b472845876bfaf0 [term 7 FOLLOWER]: Advancing to term 8
W20260129 03:00:18.160578 23466 consensus_meta.cc:287] T cb4727386a3e48aca5d71cc6c611c197 P 197b880fa39d432a9b472845876bfaf0: Time spent flushing consensus metadata: real 0.921s user 0.016s sys 0.006s
I20260129 03:00:18.153810 23224 raft_consensus.cc:515] T 36ff33be666a4c82954731691dbcf327 P 197b880fa39d432a9b472845876bfaf0 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 03:00:18.160995 23466 raft_consensus.cc:515] T cb4727386a3e48aca5d71cc6c611c197 P 197b880fa39d432a9b472845876bfaf0 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 03:00:18.162460 21134 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34046 (request call id 617) took 1249 ms. Trace:
I20260129 03:00:18.162643 23224 leader_election.cc:290] T 36ff33be666a4c82954731691dbcf327 P 197b880fa39d432a9b472845876bfaf0 [CANDIDATE]: Term 5 election: Requested vote from peers 974b802141bd4f13beffef2aa03f88f9 (127.18.211.2:34517), 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3:37735)
I20260129 03:00:18.162968 23466 leader_election.cc:290] T cb4727386a3e48aca5d71cc6c611c197 P 197b880fa39d432a9b472845876bfaf0 [CANDIDATE]: Term 6 election: Requested vote from peers 974b802141bd4f13beffef2aa03f88f9 (127.18.211.2:34517), 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3:37735)
I20260129 03:00:18.163776 22645 raft_consensus.cc:2804] T 7d9494f5584d4d5db8b9adf1c8fc888b P 197b880fa39d432a9b472845876bfaf0 [term 5 FOLLOWER]: Leader election won for term 5
I20260129 03:00:18.162848 21134 rpcz_store.cc:276] 0129 03:00:16.913432 (+ 0us) service_pool.cc:168] Inserting onto call queue
0129 03:00:16.913573 (+ 141us) service_pool.cc:225] Handling call
0129 03:00:16.939720 (+ 26147us) raft_consensus.cc:1471] Updating replica for 1 ops
0129 03:00:17.005857 (+ 66137us) raft_consensus.cc:1534] Early marking committed up to index 0
0129 03:00:17.005887 (+ 30us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0129 03:00:17.017151 (+ 11264us) log.cc:844] Serialized 26 byte log entry
0129 03:00:17.283648 (+266497us) raft_consensus.cc:1645] Marking committed up to 6
0129 03:00:17.283687 (+ 39us) raft_consensus.cc:1697] Filling consensus response to leader.
0129 03:00:17.283752 (+ 65us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0129 03:00:18.162241 (+878489us) raft_consensus.cc:1685] finished
0129 03:00:18.162262 (+ 21us) raft_consensus.cc:1691] UpdateReplicas() finished
0129 03:00:18.162400 (+ 138us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":32988,"raft.run_cpu_time_us":140,"raft.run_wall_time_us":12404,"thread_start_us":198109,"threads_started":1,"wal-append.queue_time_us":1006961}
I20260129 03:00:18.164840 20584 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35754 (request call id 1859) took 1119 ms. Trace:
I20260129 03:00:18.165308 20435 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3906b080a5b34a74b8b64128635e8997" candidate_uuid: "1c50de89a5a845668696dcb28c3badd6" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "197b880fa39d432a9b472845876bfaf0" is_pre_election: true
I20260129 03:00:18.166731 20435 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "368953c9102643b2a21f99af36b51905" candidate_uuid: "1c50de89a5a845668696dcb28c3badd6" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "197b880fa39d432a9b472845876bfaf0" is_pre_election: true
W20260129 03:00:18.167105 21100 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.211.3:37735, user_credentials={real_user=slave}} blocked reactor thread for 99673.6us
I20260129 03:00:18.167526 20435 raft_consensus.cc:2393] T 368953c9102643b2a21f99af36b51905 P 197b880fa39d432a9b472845876bfaf0 [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1c50de89a5a845668696dcb28c3badd6 in current term 9: Already voted for candidate 197b880fa39d432a9b472845876bfaf0 in this term.
I20260129 03:00:18.168570 20437 raft_consensus.cc:1240] T cb4727386a3e48aca5d71cc6c611c197 P 197b880fa39d432a9b472845876bfaf0 [term 6 FOLLOWER]: Rejecting Update request from peer 1c50de89a5a845668696dcb28c3badd6 for earlier term 5. Current term is 6. Ops: []
I20260129 03:00:18.169426 23669 leader_election.cc:290] T 3906b080a5b34a74b8b64128635e8997 P 1c50de89a5a845668696dcb28c3badd6 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 197b880fa39d432a9b472845876bfaf0 (127.18.211.1:42027), 974b802141bd4f13beffef2aa03f88f9 (127.18.211.2:34517)
I20260129 03:00:18.170751 20585 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:48526 (request call id 424) took 1742 ms. Trace:
I20260129 03:00:18.171227 20585 rpcz_store.cc:276] 0129 03:00:16.428626 (+ 0us) service_pool.cc:168] Inserting onto call queue
0129 03:00:16.428790 (+ 164us) service_pool.cc:225] Handling call
0129 03:00:16.429227 (+ 437us) raft_consensus.cc:1471] Updating replica for 1 ops
0129 03:00:16.431053 (+ 1826us) raft_consensus.cc:1534] Early marking committed up to index 8
0129 03:00:16.450315 (+ 19262us) log.cc:844] Serialized 14 byte log entry
0129 03:00:17.245405 (+795090us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0129 03:00:17.245841 (+ 436us) log.cc:844] Serialized 26 byte log entry
0129 03:00:17.245970 (+ 129us) raft_consensus.cc:1645] Marking committed up to 8
0129 03:00:17.246002 (+ 32us) raft_consensus.cc:1697] Filling consensus response to leader.
0129 03:00:17.246065 (+ 63us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0129 03:00:18.170539 (+924474us) raft_consensus.cc:1685] finished
0129 03:00:18.170561 (+ 22us) raft_consensus.cc:1691] UpdateReplicas() finished
0129 03:00:18.170694 (+ 133us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":6801,"prepare.queue_time_us":22214,"prepare.run_cpu_time_us":55,"prepare.run_wall_time_us":56,"raft.queue_time_us":154159,"raft.run_cpu_time_us":234,"raft.run_wall_time_us":16160,"spinlock_wait_cycles":29952,"thread_start_us":774313,"threads_started":2,"wal-append.queue_time_us":764923}
I20260129 03:00:18.173142 20585 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a644c02b16af47679824ccd060ac10fd" candidate_uuid: "7a9daf5d42a84615a6383cfec9a10608" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "1c50de89a5a845668696dcb28c3badd6"
I20260129 03:00:18.174027 20587 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2e58e9a5a8aa4cecafb464addb4f35ed" candidate_uuid: "7a9daf5d42a84615a6383cfec9a10608" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "1c50de89a5a845668696dcb28c3badd6" is_pre_election: true
I20260129 03:00:18.155408 23665 raft_consensus.cc:493] T 3906b080a5b34a74b8b64128635e8997 P 197b880fa39d432a9b472845876bfaf0 [term 4 FOLLOWER]: Starting leader election (detected failure of leader 974b802141bd4f13beffef2aa03f88f9)
I20260129 03:00:18.175675 23665 raft_consensus.cc:3060] T 3906b080a5b34a74b8b64128635e8997 P 197b880fa39d432a9b472845876bfaf0 [term 4 FOLLOWER]: Advancing to term 5
I20260129 03:00:18.170799 20440 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55676 (request call id 1528) took 1522 ms. Trace:
I20260129 03:00:18.181730 20440 rpcz_store.cc:276] 0129 03:00:16.648768 (+ 0us) service_pool.cc:168] Inserting onto call queue
0129 03:00:16.649052 (+ 284us) service_pool.cc:225] Handling call
0129 03:00:16.684146 (+ 35094us) raft_consensus.cc:1471] Updating replica for 0 ops
0129 03:00:17.886410 (+1202264us) raft_consensus.cc:1697] Filling consensus response to leader.
0129 03:00:18.170605 (+284195us) spinlock_profiling.cc:243] Waited 1.2 s on lock 0x7b54002e1654. stack: 00007f8fe044e373 00007f8fe044e270 00007f8fdfda0574 00007f8fe9a5c6b4 00007f8fe9a5c649 00007f8fe34975a1 00007f8fe352ea81 00007f8fe352d4be 00007f8fe84e1f49 00007f8fe1a6ed34 00007f8fe1a6ecc2 00007f8fe1a6ebf1 00007f8fe1a6eb6c 00007f8fe1a6de22 00007f8fe186d34c 00007f8fe186c786
0129 03:00:18.170746 (+ 141us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2763489152}
I20260129 03:00:18.184125 23680 leader_election.cc:290] T 1ce4ddf5b10e4ae8b193b13b2e5936ee P 1c50de89a5a845668696dcb28c3badd6 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 197b880fa39d432a9b472845876bfaf0 (127.18.211.1:42027), 974b802141bd4f13beffef2aa03f88f9 (127.18.211.2:34517)
I20260129 03:00:18.185175 23469 leader_election.cc:290] T 913fc7d751ed4b7388d889aac30a5f9b P 197b880fa39d432a9b472845876bfaf0 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 974b802141bd4f13beffef2aa03f88f9 (127.18.211.2:34517), 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3:37735)
I20260129 03:00:18.185297 23362 raft_consensus.cc:493] T add8d4923d1a489e86882c3a25692b95 P 7a9daf5d42a84615a6383cfec9a10608 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260129 03:00:18.157353 23674 raft_consensus.cc:493] T a1a39dcd328e4268ae41b988c66b4a75 P 197b880fa39d432a9b472845876bfaf0 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260129 03:00:18.186131 23674 raft_consensus.cc:3060] T a1a39dcd328e4268ae41b988c66b4a75 P 197b880fa39d432a9b472845876bfaf0 [term 6 FOLLOWER]: Advancing to term 7
I20260129 03:00:18.185844 23362 raft_consensus.cc:515] T add8d4923d1a489e86882c3a25692b95 P 7a9daf5d42a84615a6383cfec9a10608 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } }
I20260129 03:00:18.165233 20584 rpcz_store.cc:276] 0129 03:00:17.045228 (+ 0us) service_pool.cc:168] Inserting onto call queue
0129 03:00:17.045350 (+ 122us) service_pool.cc:225] Handling call
0129 03:00:17.060805 (+ 15455us) raft_consensus.cc:1471] Updating replica for 1 ops
0129 03:00:17.796886 (+736081us) raft_consensus.cc:1534] Early marking committed up to index 0
0129 03:00:17.796948 (+ 62us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0129 03:00:17.797700 (+ 752us) log.cc:844] Serialized 26 byte log entry
0129 03:00:17.865229 (+ 67529us) raft_consensus.cc:1645] Marking committed up to 5
0129 03:00:17.907850 (+ 42621us) log.cc:844] Serialized 14 byte log entry
0129 03:00:18.093257 (+185407us) raft_consensus.cc:1697] Filling consensus response to leader.
0129 03:00:18.093354 (+ 97us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0129 03:00:18.164594 (+ 71240us) raft_consensus.cc:1685] finished
0129 03:00:18.164633 (+ 39us) raft_consensus.cc:1691] UpdateReplicas() finished
0129 03:00:18.164780 (+ 147us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":44910,"prepare.run_cpu_time_us":66,"prepare.run_wall_time_us":7971,"raft.queue_time_us":139224,"raft.run_cpu_time_us":168,"raft.run_wall_time_us":46668,"spinlock_wait_cycles":21828480,"thread_start_us":596257,"threads_started":3,"wal-append.queue_time_us":190152}
I20260129 03:00:18.188668 20439 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1ce4ddf5b10e4ae8b193b13b2e5936ee" candidate_uuid: "1c50de89a5a845668696dcb28c3badd6" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "197b880fa39d432a9b472845876bfaf0" is_pre_election: true
I20260129 03:00:18.191555 23362 leader_election.cc:290] T add8d4923d1a489e86882c3a25692b95 P 7a9daf5d42a84615a6383cfec9a10608 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 974b802141bd4f13beffef2aa03f88f9 (127.18.211.2:34517), 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3:37735)
I20260129 03:00:18.194211 20583 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "45cc5f3d2688486d9b6dbe7e666dedde" candidate_uuid: "7a9daf5d42a84615a6383cfec9a10608" candidate_term: 8 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "1c50de89a5a845668696dcb28c3badd6" is_pre_election: true
W20260129 03:00:18.195559 23667 log.cc:927] Time spent T 2ac659d732314e1288849f08fe5e074e P 974b802141bd4f13beffef2aa03f88f9: Append to log took a long time: real 0.068s user 0.001s sys 0.001s
I20260129 03:00:18.197715 20508 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57600 (request call id 2020) took 1327 ms. Trace:
W20260129 03:00:18.199426 23652 log.cc:927] Time spent T c934d31efd3f4e5195605ddc354d14e1 P 7a9daf5d42a84615a6383cfec9a10608: Append to log took a long time: real 0.089s user 0.000s sys 0.000s
I20260129 03:00:18.198092 20508 rpcz_store.cc:276] 0129 03:00:16.870009 (+ 0us) service_pool.cc:168] Inserting onto call queue
0129 03:00:16.870436 (+ 427us) service_pool.cc:225] Handling call
0129 03:00:16.870927 (+ 491us) raft_consensus.cc:1471] Updating replica for 2 ops
0129 03:00:16.871445 (+ 518us) raft_consensus.cc:1534] Early marking committed up to index 0
0129 03:00:16.871484 (+ 39us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0129 03:00:16.876428 (+ 4944us) log.cc:844] Serialized 52 byte log entry
0129 03:00:17.217322 (+340894us) raft_consensus.cc:1645] Marking committed up to 6
0129 03:00:17.239299 (+ 21977us) log.cc:844] Serialized 14 byte log entry
0129 03:00:17.243048 (+ 3749us) raft_consensus.cc:1697] Filling consensus response to leader.
0129 03:00:17.243147 (+ 99us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0129 03:00:18.194713 (+951566us) spinlock_profiling.cc:243] Waited 18.1 ms on lock 0x7b440002304c. stack: 00007f8fe044e373 00007f8fe044e270 00007f8fdfda0574 00007f8fe9a5c6b4 00007f8fe9a5c649 00007f8fe02c08b1 00007f8fe038f005 00007f8fe038fa22 00007f8fe30d957b 00007f8fe30d4e4f 00007f8fe30c8c0e 00007f8fe30c7f56 00007f8fe30c7b78 00007f8fe30d15c1 00007f8fe30d1579 00007f8fe30d1509
0129 03:00:18.197409 (+ 2696us) raft_consensus.cc:1685] finished
0129 03:00:18.197507 (+ 98us) raft_consensus.cc:1691] UpdateReplicas() finished
0129 03:00:18.197654 (+ 147us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":49095,"prepare.queue_time_us":105183,"prepare.run_cpu_time_us":9199,"prepare.run_wall_time_us":479194,"raft.queue_time_us":3945,"raft.run_cpu_time_us":360,"raft.run_wall_time_us":52162,"spinlock_wait_cycles":41535872,"thread_start_us":220927,"threads_started":1,"wal-append.queue_time_us":1207305}
I20260129 03:00:18.199995 23333 raft_consensus.cc:493] T 75f45538dc704d9a8ec4ff8ad9f3ea5b P 974b802141bd4f13beffef2aa03f88f9 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 7a9daf5d42a84615a6383cfec9a10608)
I20260129 03:00:18.200554 23333 raft_consensus.cc:515] T 75f45538dc704d9a8ec4ff8ad9f3ea5b P 974b802141bd4f13beffef2aa03f88f9 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } }
W20260129 03:00:18.200894 20480 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.211.3:37735, user_credentials={real_user=slave}} blocked reactor thread for 311500us
I20260129 03:00:18.201752 20480 leader_election.cc:304] T add8d4923d1a489e86882c3a25692b95 P 974b802141bd4f13beffef2aa03f88f9 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 974b802141bd4f13beffef2aa03f88f9; no voters: 1c50de89a5a845668696dcb28c3badd6, 7a9daf5d42a84615a6383cfec9a10608
I20260129 03:00:18.202448 21127 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:47844 (request call id 412) took 2067 ms. Trace:
I20260129 03:00:18.202651 23333 leader_election.cc:290] T 75f45538dc704d9a8ec4ff8ad9f3ea5b P 974b802141bd4f13beffef2aa03f88f9 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3:37735), 7a9daf5d42a84615a6383cfec9a10608 (127.18.211.4:40439)
W20260129 03:00:18.204411 20405 leader_election.cc:336] T e4d18c35b0a7493e85d0a4b92956a48f P 197b880fa39d432a9b472845876bfaf0 [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3:37735): Timed out: RequestConsensusVote RPC to 127.18.211.3:37735 timed out after 2.140s (SENT)
I20260129 03:00:18.204716 20507 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57600 (request call id 2011) took 1655 ms. Trace:
I20260129 03:00:18.205030 20405 leader_election.cc:304] T e4d18c35b0a7493e85d0a4b92956a48f P 197b880fa39d432a9b472845876bfaf0 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 197b880fa39d432a9b472845876bfaf0; no voters: 1c50de89a5a845668696dcb28c3badd6, 974b802141bd4f13beffef2aa03f88f9
I20260129 03:00:18.205065 20507 rpcz_store.cc:276] 0129 03:00:16.549370 (+ 0us) service_pool.cc:168] Inserting onto call queue
0129 03:00:16.549511 (+ 141us) service_pool.cc:225] Handling call
0129 03:00:16.558357 (+ 8846us) raft_consensus.cc:1471] Updating replica for 0 ops
0129 03:00:16.562656 (+ 4299us) raft_consensus.cc:1534] Early marking committed up to index 9
0129 03:00:17.049241 (+486585us) log.cc:844] Serialized 14 byte log entry
0129 03:00:18.204231 (+1154990us) log.cc:844] Serialized 14 byte log entry
0129 03:00:18.204404 (+ 173us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0129 03:00:18.204456 (+ 52us) raft_consensus.cc:1645] Marking committed up to 9
0129 03:00:18.204498 (+ 42us) raft_consensus.cc:1697] Filling consensus response to leader.
0129 03:00:18.204559 (+ 61us) raft_consensus.cc:1691] UpdateReplicas() finished
0129 03:00:18.204665 (+ 106us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":341656,"prepare.queue_time_us":643551,"prepare.run_cpu_time_us":36,"prepare.run_wall_time_us":37,"thread_start_us":1280754,"threads_started":2,"wal-append.queue_time_us":838228}
I20260129 03:00:18.204377 23333 consensus_queue.cc:1048] T e7d83d4ccf684e76acb29a0dc844e1b8 P 974b802141bd4f13beffef2aa03f88f9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 2, Time since last communication: 0.001s
I20260129 03:00:18.138417 22996 raft_consensus.cc:493] T cb0b3f259ebc4310927338a8740e87b8 P 1c50de89a5a845668696dcb28c3badd6 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260129 03:00:18.214172 20588 raft_consensus.cc:2410] T 45cc5f3d2688486d9b6dbe7e666dedde P 1c50de89a5a845668696dcb28c3badd6 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 974b802141bd4f13beffef2aa03f88f9 for term 7 because replica has last-logged OpId of term: 6 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 5.
I20260129 03:00:18.126292 23602 leader_election.cc:290] T 368953c9102643b2a21f99af36b51905 P 1c50de89a5a845668696dcb28c3badd6 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 197b880fa39d432a9b472845876bfaf0 (127.18.211.1:42027), 974b802141bd4f13beffef2aa03f88f9 (127.18.211.2:34517)
W20260129 03:00:18.219046 20553 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260129 03:00:18.203624 23679 log.cc:927] Time spent T 36ff33be666a4c82954731691dbcf327 P 974b802141bd4f13beffef2aa03f88f9: Append to log took a long time: real 0.391s user 0.003s sys 0.000s
I20260129 03:00:18.228654 20587 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "add8d4923d1a489e86882c3a25692b95" candidate_uuid: "7a9daf5d42a84615a6383cfec9a10608" candidate_term: 9 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "1c50de89a5a845668696dcb28c3badd6" is_pre_election: true
I20260129 03:00:18.226631 20590 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a8479d1fe0bd41e5946186e3a6ca8c7a" candidate_uuid: "197b880fa39d432a9b472845876bfaf0" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "1c50de89a5a845668696dcb28c3badd6" is_pre_election: true
I20260129 03:00:18.229589 20587 raft_consensus.cc:2468] T add8d4923d1a489e86882c3a25692b95 P 1c50de89a5a845668696dcb28c3badd6 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7a9daf5d42a84615a6383cfec9a10608 in term 8.
I20260129 03:00:18.230008 20590 raft_consensus.cc:2468] T a8479d1fe0bd41e5946186e3a6ca8c7a P 1c50de89a5a845668696dcb28c3badd6 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 197b880fa39d432a9b472845876bfaf0 in term 5.
I20260129 03:00:18.212980 23514 raft_consensus.cc:3060] T add8d4923d1a489e86882c3a25692b95 P 974b802141bd4f13beffef2aa03f88f9 [term 7 FOLLOWER]: Advancing to term 8
I20260129 03:00:18.191462 23678 raft_consensus.cc:493] T 7a9a69cd82724b1faed1d10aa091fc4a P 1c50de89a5a845668696dcb28c3badd6 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 197b880fa39d432a9b472845876bfaf0)
I20260129 03:00:18.218739 22645 raft_consensus.cc:697] T 7d9494f5584d4d5db8b9adf1c8fc888b P 197b880fa39d432a9b472845876bfaf0 [term 5 LEADER]: Becoming Leader. State: Replica: 197b880fa39d432a9b472845876bfaf0, State: Running, Role: LEADER
I20260129 03:00:18.244702 20582 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4b1876f9a70b49edb3aa74105e66d3b3" candidate_uuid: "974b802141bd4f13beffef2aa03f88f9" candidate_term: 7 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "1c50de89a5a845668696dcb28c3badd6" is_pre_election: true
W20260129 03:00:18.256289 20407 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.211.2:34517, user_credentials={real_user=slave}} blocked reactor thread for 43551.5us
W20260129 03:00:18.257648 20404 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260129 03:00:18.233852 23469 raft_consensus.cc:2749] T e4d18c35b0a7493e85d0a4b92956a48f P 197b880fa39d432a9b472845876bfaf0 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260129 03:00:18.260670 20407 leader_election.cc:304] T e33a417cf9c54cf5af73a4ee3aac4d3a P 197b880fa39d432a9b472845876bfaf0 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 197b880fa39d432a9b472845876bfaf0, 974b802141bd4f13beffef2aa03f88f9; no voters:
I20260129 03:00:18.261941 20584 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "cedce331e1584ac8b636c9b8a72b2ba0" candidate_uuid: "974b802141bd4f13beffef2aa03f88f9" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "1c50de89a5a845668696dcb28c3badd6" is_pre_election: true
W20260129 03:00:18.264497 20405 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.211.3:37735, user_credentials={real_user=slave}} blocked reactor thread for 60131.4us
W20260129 03:00:18.266228 20480 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.211.3:37735, user_credentials={real_user=slave}} blocked reactor thread for 46814.4us
I20260129 03:00:18.266619 21100 leader_election.cc:304] T add8d4923d1a489e86882c3a25692b95 P 7a9daf5d42a84615a6383cfec9a10608 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1c50de89a5a845668696dcb28c3badd6, 7a9daf5d42a84615a6383cfec9a10608; no voters:
I20260129 03:00:18.268384 20590 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c891d2bff4ce4568861bbc992e91fbb0" candidate_uuid: "197b880fa39d432a9b472845876bfaf0" candidate_term: 9 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "1c50de89a5a845668696dcb28c3badd6"
I20260129 03:00:18.269177 20590 raft_consensus.cc:2393] T c891d2bff4ce4568861bbc992e91fbb0 P 1c50de89a5a845668696dcb28c3badd6 [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate 197b880fa39d432a9b472845876bfaf0 in current term 9: Already voted for candidate 1c50de89a5a845668696dcb28c3badd6 in this term.
I20260129 03:00:18.268946 20584 raft_consensus.cc:2468] T cedce331e1584ac8b636c9b8a72b2ba0 P 1c50de89a5a845668696dcb28c3badd6 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 974b802141bd4f13beffef2aa03f88f9 in term 8.
I20260129 03:00:18.226040 22996 raft_consensus.cc:515] T cb0b3f259ebc4310927338a8740e87b8 P 1c50de89a5a845668696dcb28c3badd6 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } }
W20260129 03:00:18.271961 20479 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260129 03:00:18.272403 20479 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260129 03:00:18.271173 20581 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3906b080a5b34a74b8b64128635e8997" candidate_uuid: "974b802141bd4f13beffef2aa03f88f9" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "1c50de89a5a845668696dcb28c3badd6" is_pre_election: true
I20260129 03:00:18.273131 20581 raft_consensus.cc:2468] T 3906b080a5b34a74b8b64128635e8997 P 1c50de89a5a845668696dcb28c3badd6 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 974b802141bd4f13beffef2aa03f88f9 in term 5.
I20260129 03:00:18.275776 20432 raft_consensus.cc:2468] T e33a417cf9c54cf5af73a4ee3aac4d3a P 197b880fa39d432a9b472845876bfaf0 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1c50de89a5a845668696dcb28c3badd6 in term 8.
I20260129 03:00:18.242681 23304 consensus_queue.cc:1059] T fd8311b779af4e1fa1402b2a02a602f2 P 1c50de89a5a845668696dcb28c3badd6 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 }, Status: INVALID_TERM, Last received: 5.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.025s
I20260129 03:00:18.279755 20587 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fd8311b779af4e1fa1402b2a02a602f2" candidate_uuid: "197b880fa39d432a9b472845876bfaf0" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "1c50de89a5a845668696dcb28c3badd6" is_pre_election: true
I20260129 03:00:18.279686 20553 leader_election.cc:304] T e33a417cf9c54cf5af73a4ee3aac4d3a P 1c50de89a5a845668696dcb28c3badd6 [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 197b880fa39d432a9b472845876bfaf0, 1c50de89a5a845668696dcb28c3badd6; no voters:
I20260129 03:00:18.264209 23450 raft_consensus.cc:493] T 0340229d521b432580ae926f6062e3f7 P 974b802141bd4f13beffef2aa03f88f9 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260129 03:00:18.264083 23555 raft_consensus.cc:493] T 4e420f90884f4796997b5ea41e49e411 P 974b802141bd4f13beffef2aa03f88f9 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 7a9daf5d42a84615a6383cfec9a10608)
I20260129 03:00:18.272487 23165 raft_consensus.cc:2804] T add8d4923d1a489e86882c3a25692b95 P 7a9daf5d42a84615a6383cfec9a10608 [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260129 03:00:18.245232 23678 raft_consensus.cc:515] T 7a9a69cd82724b1faed1d10aa091fc4a P 1c50de89a5a845668696dcb28c3badd6 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 03:00:18.277931 23485 raft_consensus.cc:2764] T e33a417cf9c54cf5af73a4ee3aac4d3a P 197b880fa39d432a9b472845876bfaf0 [term 8 FOLLOWER]: Leader election decision vote started in defunct term 7: won
I20260129 03:00:18.328923 21134 raft_consensus.cc:1240] T a644c02b16af47679824ccd060ac10fd P 7a9daf5d42a84615a6383cfec9a10608 [term 6 FOLLOWER]: Rejecting Update request from peer 1c50de89a5a845668696dcb28c3badd6 for earlier term 5. Current term is 6. Ops: []
I20260129 03:00:18.329653 20581 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "36ff33be666a4c82954731691dbcf327" candidate_uuid: "197b880fa39d432a9b472845876bfaf0" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "1c50de89a5a845668696dcb28c3badd6" is_pre_election: true
I20260129 03:00:18.334210 23703 tablet_replica.cc:333] T 1299723a4476420ab02fd8bfd7aa0645 P 974b802141bd4f13beffef2aa03f88f9: stopping tablet replica
I20260129 03:00:18.335312 20509 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57584 (request call id 1534) took 1398 ms. Trace:
I20260129 03:00:18.335681 20509 rpcz_store.cc:276] 0129 03:00:16.936713 (+ 0us) service_pool.cc:168] Inserting onto call queue
0129 03:00:17.207848 (+271135us) service_pool.cc:225] Handling call
0129 03:00:17.208250 (+ 402us) raft_consensus.cc:1471] Updating replica for 1 ops
0129 03:00:17.208681 (+ 431us) raft_consensus.cc:1534] Early marking committed up to index 0
0129 03:00:17.208714 (+ 33us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0129 03:00:17.209215 (+ 501us) log.cc:844] Serialized 26 byte log entry
0129 03:00:17.792346 (+583131us) raft_consensus.cc:1645] Marking committed up to 4
0129 03:00:17.848306 (+ 55960us) log.cc:844] Serialized 14 byte log entry
0129 03:00:18.192891 (+344585us) spinlock_profiling.cc:243] Waited 15 ms on lock 0x7b440002304c. stack: 00007f8fe044e373 00007f8fe044e270 00007f8fdfda0574 00007f8fe9a5c6b4 00007f8fe9a5c649 00007f8fe02c08b1 00007f8fe038f005 00007f8fe038fa22 00007f8fe30d957b 00007f8fe30d4e4f 00007f8fe30c8c0e 00007f8fe30c7f56 00007f8fe30c7b78 00007f8fe30d15c1 00007f8fe30d1579 00007f8fe30d1509
0129 03:00:18.199170 (+ 6279us) spinlock_profiling.cc:243] Waited 941 us on lock 0x7b18003e3ce0. stack: 00007f8fe044e373 00007f8fe044e270 00007f8fdfda0574 00007f8fe9a5c6b4 00007f8fe9a5c649 00007f8fe92d4c91 00007f8fe92e1671 00007f8fe03f546a 00007f8fe9a7b5c9 00007f8fe02b5160 00007f8fe047d3c4 00007f8fe047d335 00007f8fe3525ce0 00007f8fe3538396 00007f8fe3548f07 00007f8fe3548e88
0129 03:00:18.293873 (+ 94703us) raft_consensus.cc:1697] Filling consensus response to leader.
0129 03:00:18.293992 (+ 119us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0129 03:00:18.335134 (+ 41142us) raft_consensus.cc:1685] finished
0129 03:00:18.335154 (+ 20us) raft_consensus.cc:1691] UpdateReplicas() finished
0129 03:00:18.335265 (+ 111us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":45048,"prepare.queue_time_us":124684,"prepare.run_cpu_time_us":54,"prepare.run_wall_time_us":2419,"raft.queue_time_us":352572,"raft.run_cpu_time_us":168,"raft.run_wall_time_us":167,"spinlock_wait_cycles":36569856,"thread_start_us":553734,"threads_started":2,"wal-append.queue_time_us":592906}
I20260129 03:00:18.341562 20432 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "368953c9102643b2a21f99af36b51905" candidate_uuid: "974b802141bd4f13beffef2aa03f88f9" candidate_term: 10 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "197b880fa39d432a9b472845876bfaf0" is_pre_election: true
I20260129 03:00:18.342366 20432 raft_consensus.cc:2468] T 368953c9102643b2a21f99af36b51905 P 197b880fa39d432a9b472845876bfaf0 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 974b802141bd4f13beffef2aa03f88f9 in term 9.
I20260129 03:00:18.343035 23695 raft_consensus.cc:493] T 00e79d8f25a34676b1d866eb2068a2b2 P 974b802141bd4f13beffef2aa03f88f9 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 197b880fa39d432a9b472845876bfaf0)
I20260129 03:00:18.343591 23695 raft_consensus.cc:515] T 00e79d8f25a34676b1d866eb2068a2b2 P 974b802141bd4f13beffef2aa03f88f9 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } }
I20260129 03:00:18.345494 23695 leader_election.cc:290] T 00e79d8f25a34676b1d866eb2068a2b2 P 974b802141bd4f13beffef2aa03f88f9 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 197b880fa39d432a9b472845876bfaf0 (127.18.211.1:42027), 7a9daf5d42a84615a6383cfec9a10608 (127.18.211.4:40439)
I20260129 03:00:18.202883 21127 rpcz_store.cc:276] 0129 03:00:16.135218 (+ 0us) service_pool.cc:168] Inserting onto call queue
0129 03:00:16.147792 (+ 12574us) service_pool.cc:225] Handling call
0129 03:00:16.218526 (+ 70734us) raft_consensus.cc:1471] Updating replica for 1 ops
0129 03:00:16.251296 (+ 32770us) raft_consensus.cc:1534] Early marking committed up to index 0
0129 03:00:16.252266 (+ 970us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0129 03:00:16.290923 (+ 38657us) spinlock_profiling.cc:243] Waited 2.77 ms on lock 0x7b2c000472a8. stack: 00007f8fe044e373 00007f8fe044e270 00007f8fdfda0574 00007f8fe9a5c6b4 00007f8fe9a5c649 00007f8fe1e2c9b1 00007f8fe1e3bfe2 00007f8fe3569851 00007f8fe352e0ba 00007f8fe352d4be 00007f8fe84e1f49 00007f8fe1a6ed34 00007f8fe1a6ecc2 00007f8fe1a6ebf1 00007f8fe1a6eb6c 00007f8fe1a6de22
0129 03:00:16.291419 (+ 496us) log.cc:844] Serialized 26 byte log entry
0129 03:00:16.758032 (+466613us) raft_consensus.cc:1645] Marking committed up to 8
0129 03:00:16.758094 (+ 62us) raft_consensus.cc:1697] Filling consensus response to leader.
0129 03:00:16.759211 (+ 1117us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0129 03:00:18.202134 (+1442923us) spinlock_profiling.cc:243] Waited 3.1 ms on lock 0x7b2c001d8798. stack: 00007f8fe044e373 00007f8fe044e270 00007f8fdfda0574 00007f8fe9a5c6b4 00007f8fe9a5c649 00007f8fe17e9c81 00007f8fe18044fa 00007f8fe352712d 00007f8fe352ec0e 00007f8fe352d4be 00007f8fe84e1f49 00007f8fe1a6ed34 00007f8fe1a6ecc2 00007f8fe1a6ebf1 00007f8fe1a6eb6c 00007f8fe1a6de22
0129 03:00:18.202227 (+ 93us) raft_consensus.cc:1685] finished
0129 03:00:18.202247 (+ 20us) raft_consensus.cc:1691] UpdateReplicas() finished
0129 03:00:18.202386 (+ 139us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":18124,"raft.queue_time_us":25203,"raft.run_cpu_time_us":115,"raft.run_wall_time_us":2229,"spinlock_wait_cycles":13520128,"thread_start_us":443973,"threads_started":1,"wal-append.queue_time_us":705085}
I20260129 03:00:18.323556 23629 raft_consensus.cc:493] T febffe4a8b394368b69b1d164308fb9a P 197b880fa39d432a9b472845876bfaf0 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260129 03:00:18.323351 23165 raft_consensus.cc:493] T add8d4923d1a489e86882c3a25692b95 P 7a9daf5d42a84615a6383cfec9a10608 [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260129 03:00:18.346654 20587 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4f8f3a38f6e24ba4857875181f447442" candidate_uuid: "197b880fa39d432a9b472845876bfaf0" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "1c50de89a5a845668696dcb28c3badd6"
I20260129 03:00:18.347421 20587 raft_consensus.cc:2393] T 4f8f3a38f6e24ba4857875181f447442 P 1c50de89a5a845668696dcb28c3badd6 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 197b880fa39d432a9b472845876bfaf0 in current term 7: Already voted for candidate 1c50de89a5a845668696dcb28c3badd6 in this term.
I20260129 03:00:18.348127 23629 raft_consensus.cc:515] T febffe4a8b394368b69b1d164308fb9a P 197b880fa39d432a9b472845876bfaf0 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } }
I20260129 03:00:18.349686 23665 raft_consensus.cc:515] T 3906b080a5b34a74b8b64128635e8997 P 197b880fa39d432a9b472845876bfaf0 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 03:00:18.350325 23629 leader_election.cc:290] T febffe4a8b394368b69b1d164308fb9a P 197b880fa39d432a9b472845876bfaf0 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 974b802141bd4f13beffef2aa03f88f9 (127.18.211.2:34517), 7a9daf5d42a84615a6383cfec9a10608 (127.18.211.4:40439)
I20260129 03:00:18.350649 23403 leader_election.cc:290] T a5a05fbcbbcf4916bbc01fd1a5b5e90d P 197b880fa39d432a9b472845876bfaf0 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 974b802141bd4f13beffef2aa03f88f9 (127.18.211.2:34517), 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3:37735)
I20260129 03:00:18.353847 20439 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2e58e9a5a8aa4cecafb464addb4f35ed" candidate_uuid: "7a9daf5d42a84615a6383cfec9a10608" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "197b880fa39d432a9b472845876bfaf0" is_pre_election: true
I20260129 03:00:18.355101 20441 raft_consensus.cc:2468] T 3906b080a5b34a74b8b64128635e8997 P 197b880fa39d432a9b472845876bfaf0 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 974b802141bd4f13beffef2aa03f88f9 in term 5.
I20260129 03:00:18.282357 23631 consensus_queue.cc:1048] T c2633a6fc34c44579de348d208b0c01e P 974b802141bd4f13beffef2aa03f88f9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.058s
I20260129 03:00:18.365348 23693 consensus_queue.cc:1059] T c2633a6fc34c44579de348d208b0c01e P 974b802141bd4f13beffef2aa03f88f9 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 }, Status: INVALID_TERM, Last received: 2.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260129 03:00:18.292646 23450 raft_consensus.cc:515] T 0340229d521b432580ae926f6062e3f7 P 974b802141bd4f13beffef2aa03f88f9 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 03:00:18.335075 23555 raft_consensus.cc:515] T 4e420f90884f4796997b5ea41e49e411 P 974b802141bd4f13beffef2aa03f88f9 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } }
I20260129 03:00:18.354553 23165 raft_consensus.cc:3060] T add8d4923d1a489e86882c3a25692b95 P 7a9daf5d42a84615a6383cfec9a10608 [term 8 FOLLOWER]: Advancing to term 9
I20260129 03:00:18.335711 22645 consensus_queue.cc:237] T 7d9494f5584d4d5db8b9adf1c8fc888b P 197b880fa39d432a9b472845876bfaf0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 03:00:18.367846 23693 raft_consensus.cc:3055] T c2633a6fc34c44579de348d208b0c01e P 974b802141bd4f13beffef2aa03f88f9 [term 5 LEADER]: Stepping down as leader of term 5
I20260129 03:00:18.385675 23693 raft_consensus.cc:740] T c2633a6fc34c44579de348d208b0c01e P 974b802141bd4f13beffef2aa03f88f9 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 974b802141bd4f13beffef2aa03f88f9, State: Running, Role: LEADER
I20260129 03:00:18.386655 20589 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:35754 (request call id 1868) took 1168 ms. Trace:
I20260129 03:00:18.386510 23693 consensus_queue.cc:260] T c2633a6fc34c44579de348d208b0c01e P 974b802141bd4f13beffef2aa03f88f9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 5.3, Last appended by leader: 3, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 03:00:18.387238 20589 rpcz_store.cc:276] 0129 03:00:17.218390 (+ 0us) service_pool.cc:168] Inserting onto call queue
0129 03:00:17.218578 (+ 188us) service_pool.cc:225] Handling call
0129 03:00:18.386453 (+1167875us) spinlock_profiling.cc:243] Waited 91 ms on lock 0x7b54000c9254. stack: 00007f8fe044e373 00007f8fe044e270 00007f8fdfda0574 00007f8fe9a5c6b4 00007f8fe9a5c649 00007f8fe34975a1 00007f8fe3531f7a 00007f8fe84e3647 00007f8fe1a73834 00007f8fe1a737c2 00007f8fe1a736f1 00007f8fe1a7366c 00007f8fe1a72922 00007f8fe186d34c 00007f8fe186c786 00007f8fe186c12f
0129 03:00:18.386578 (+ 125us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":209428480}
I20260129 03:00:18.369539 20438 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:48356 (request call id 464) took 1770 ms. Trace:
I20260129 03:00:18.388834 20438 rpcz_store.cc:276] 0129 03:00:16.598769 (+ 0us) service_pool.cc:168] Inserting onto call queue
0129 03:00:16.603366 (+ 4597us) service_pool.cc:225] Handling call
0129 03:00:16.609636 (+ 6270us) raft_consensus.cc:1471] Updating replica for 0 ops
0129 03:00:16.707690 (+ 98054us) raft_consensus.cc:1534] Early marking committed up to index 8
0129 03:00:16.742928 (+ 35238us) log.cc:844] Serialized 14 byte log entry
0129 03:00:17.890220 (+1147292us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0129 03:00:17.890279 (+ 59us) raft_consensus.cc:1645] Marking committed up to 8
0129 03:00:17.890330 (+ 51us) raft_consensus.cc:1697] Filling consensus response to leader.
0129 03:00:18.369302 (+478972us) spinlock_profiling.cc:243] Waited 97.8 ms on lock 0x7b54002af154. stack: 00007f8fe044e373 00007f8fe044e270 00007f8fdfda0574 00007f8fe9a5c6b4 00007f8fe9a5c649 00007f8fe34975a1 00007f8fe352ea81 00007f8fe352d4be 00007f8fe84e1f49 00007f8fe1a6ed34 00007f8fe1a6ecc2 00007f8fe1a6ebf1 00007f8fe1a6eb6c 00007f8fe1a6de22 00007f8fe186d34c 00007f8fe186c786
0129 03:00:18.369351 (+ 49us) raft_consensus.cc:1691] UpdateReplicas() finished
0129 03:00:18.369477 (+ 126us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":35438,"prepare.run_cpu_time_us":67,"prepare.run_wall_time_us":1802,"raft.queue_time_us":84636,"raft.run_cpu_time_us":127,"raft.run_wall_time_us":10525,"spinlock_wait_cycles":224990080,"thread_start_us":1146814,"threads_started":2,"wal-append.queue_time_us":1236029}
I20260129 03:00:18.389835 23674 raft_consensus.cc:515] T a1a39dcd328e4268ae41b988c66b4a75 P 197b880fa39d432a9b472845876bfaf0 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } } peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } }
I20260129 03:00:18.391170 20441 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7a9a69cd82724b1faed1d10aa091fc4a" candidate_uuid: "1c50de89a5a845668696dcb28c3badd6" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "197b880fa39d432a9b472845876bfaf0" is_pre_election: true
I20260129 03:00:18.369690 23450 leader_election.cc:290] T 0340229d521b432580ae926f6062e3f7 P 974b802141bd4f13beffef2aa03f88f9 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 197b880fa39d432a9b472845876bfaf0 (127.18.211.1:42027), 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3:37735)
I20260129 03:00:18.370152 20437 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55676 (request call id 1531) took 1094 ms. Trace:
I20260129 03:00:18.392755 20437 rpcz_store.cc:276] 0129 03:00:17.275451 (+ 0us) service_pool.cc:168] Inserting onto call queue
0129 03:00:17.275621 (+ 170us) service_pool.cc:225] Handling call
0129 03:00:17.305851 (+ 30230us) raft_consensus.cc:1471] Updating replica for 0 ops
0129 03:00:18.168959 (+863108us) raft_consensus.cc:1697] Filling consensus response to leader.
0129 03:00:18.369936 (+200977us) spinlock_profiling.cc:243] Waited 862 ms on lock 0x7b54002fd354. stack: 00007f8fe044e373 00007f8fe044e270 00007f8fdfda0574 00007f8fe9a5c6b4 00007f8fe9a5c649 00007f8fe34975a1 00007f8fe352ea81 00007f8fe352d4be 00007f8fe84e1f49 00007f8fe1a6ed34 00007f8fe1a6ecc2 00007f8fe1a6ebf1 00007f8fe1a6eb6c 00007f8fe1a6de22 00007f8fe186d34c 00007f8fe186c786
0129 03:00:18.370096 (+ 160us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1983632768}
I20260129 03:00:18.393886 21133 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "febffe4a8b394368b69b1d164308fb9a" candidate_uuid: "197b880fa39d432a9b472845876bfaf0" candidate_term: 7 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "7a9daf5d42a84615a6383cfec9a10608" is_pre_election: true
I20260129 03:00:18.340492 20554 leader_election.cc:304] T 75b0b6e82ddc4f92b1fe1d94f9a0b73b P 1c50de89a5a845668696dcb28c3badd6 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1c50de89a5a845668696dcb28c3badd6, 7a9daf5d42a84615a6383cfec9a10608; no voters:
I20260129 03:00:18.358990 20479 leader_election.cc:304] T 3ccac82a01a2445fa5b0ce6c0fcf2295 P 974b802141bd4f13beffef2aa03f88f9 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 974b802141bd4f13beffef2aa03f88f9; no voters: 197b880fa39d432a9b472845876bfaf0, 7a9daf5d42a84615a6383cfec9a10608
I20260129 03:00:18.378144 20480 leader_election.cc:304] T 4408006eb5174de1896b037cf3f18457 P 974b802141bd4f13beffef2aa03f88f9 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 974b802141bd4f13beffef2aa03f88f9; no voters: 197b880fa39d432a9b472845876bfaf0, 1c50de89a5a845668696dcb28c3badd6
I20260129 03:00:18.359225 23678 leader_election.cc:290] T 7a9a69cd82724b1faed1d10aa091fc4a P 1c50de89a5a845668696dcb28c3badd6 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 197b880fa39d432a9b472845876bfaf0 (127.18.211.1:42027), 974b802141bd4f13beffef2aa03f88f9 (127.18.211.2:34517)
I20260129 03:00:18.360270 20590 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e33a417cf9c54cf5af73a4ee3aac4d3a" candidate_uuid: "197b880fa39d432a9b472845876bfaf0" candidate_term: 7 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "1c50de89a5a845668696dcb28c3badd6"
I20260129 03:00:18.353541 23665 leader_election.cc:290] T 3906b080a5b34a74b8b64128635e8997 P 197b880fa39d432a9b472845876bfaf0 [CANDIDATE]: Term 5 election: Requested vote from peers 974b802141bd4f13beffef2aa03f88f9 (127.18.211.2:34517), 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3:37735)
I20260129 03:00:18.400758 20479 leader_election.cc:304] T 724a236524cb4e7ba3428880681d6cc2 P 974b802141bd4f13beffef2aa03f88f9 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 197b880fa39d432a9b472845876bfaf0, 974b802141bd4f13beffef2aa03f88f9; no voters:
I20260129 03:00:18.401633 20479 leader_election.cc:304] T 4f8f3a38f6e24ba4857875181f447442 P 974b802141bd4f13beffef2aa03f88f9 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 197b880fa39d432a9b472845876bfaf0, 974b802141bd4f13beffef2aa03f88f9; no voters:
I20260129 03:00:18.402150 21134 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fd8311b779af4e1fa1402b2a02a602f2" candidate_uuid: "197b880fa39d432a9b472845876bfaf0" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "7a9daf5d42a84615a6383cfec9a10608"
I20260129 03:00:18.402923 20479 leader_election.cc:304] T a1a39dcd328e4268ae41b988c66b4a75 P 974b802141bd4f13beffef2aa03f88f9 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 197b880fa39d432a9b472845876bfaf0, 974b802141bd4f13beffef2aa03f88f9; no voters:
I20260129 03:00:18.373144 22996 leader_election.cc:290] T cb0b3f259ebc4310927338a8740e87b8 P 1c50de89a5a845668696dcb28c3badd6 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 974b802141bd4f13beffef2aa03f88f9 (127.18.211.2:34517), 7a9daf5d42a84615a6383cfec9a10608 (127.18.211.4:40439)
I20260129 03:00:18.384976 23668 raft_consensus.cc:3055] T fd8311b779af4e1fa1402b2a02a602f2 P 1c50de89a5a845668696dcb28c3badd6 [term 5 LEADER]: Stepping down as leader of term 5
W20260129 03:00:18.381105 23659 log.cc:927] Time spent T a609ac02fae84725ab743f95f51d33ce P 7a9daf5d42a84615a6383cfec9a10608: Append to log took a long time: real 1.025s user 0.001s sys 0.010s
I20260129 03:00:18.404264 23668 raft_consensus.cc:740] T fd8311b779af4e1fa1402b2a02a602f2 P 1c50de89a5a845668696dcb28c3badd6 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 1c50de89a5a845668696dcb28c3badd6, State: Running, Role: LEADER
I20260129 03:00:18.404717 20480 leader_election.cc:304] T b189f08e16a74a11980cc55d99359a0a P 974b802141bd4f13beffef2aa03f88f9 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 974b802141bd4f13beffef2aa03f88f9; no voters: 197b880fa39d432a9b472845876bfaf0, 1c50de89a5a845668696dcb28c3badd6
I20260129 03:00:18.405308 23668 consensus_queue.cc:260] T fd8311b779af4e1fa1402b2a02a602f2 P 1c50de89a5a845668696dcb28c3badd6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 5.7, Last appended by leader: 7, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } }
I20260129 03:00:18.404271 23646 raft_consensus.cc:2749] T 3ccac82a01a2445fa5b0ce6c0fcf2295 P 974b802141bd4f13beffef2aa03f88f9 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260129 03:00:18.406008 20480 leader_election.cc:304] T 3e2d15f8640b407195210c476e09df91 P 974b802141bd4f13beffef2aa03f88f9 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 974b802141bd4f13beffef2aa03f88f9; no voters: 1c50de89a5a845668696dcb28c3badd6, 7a9daf5d42a84615a6383cfec9a10608
I20260129 03:00:18.406477 21131 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34060 (request call id 572) took 1691 ms. Trace:
I20260129 03:00:18.406627 23646 raft_consensus.cc:2804] T 724a236524cb4e7ba3428880681d6cc2 P 974b802141bd4f13beffef2aa03f88f9 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260129 03:00:18.407480 23630 raft_consensus.cc:2804] T 4f8f3a38f6e24ba4857875181f447442 P 974b802141bd4f13beffef2aa03f88f9 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260129 03:00:18.383401 20582 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a1a39dcd328e4268ae41b988c66b4a75" candidate_uuid: "197b880fa39d432a9b472845876bfaf0" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "1c50de89a5a845668696dcb28c3badd6" is_pre_election: true
I20260129 03:00:18.408162 20582 raft_consensus.cc:2468] T a1a39dcd328e4268ae41b988c66b4a75 P 1c50de89a5a845668696dcb28c3badd6 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 197b880fa39d432a9b472845876bfaf0 in term 5.
I20260129 03:00:18.408895 23695 raft_consensus.cc:2804] T a1a39dcd328e4268ae41b988c66b4a75 P 974b802141bd4f13beffef2aa03f88f9 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260129 03:00:18.409025 20582 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e33a417cf9c54cf5af73a4ee3aac4d3a" candidate_uuid: "197b880fa39d432a9b472845876bfaf0" candidate_term: 7 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "1c50de89a5a845668696dcb28c3badd6" is_pre_election: true
I20260129 03:00:18.409308 23695 raft_consensus.cc:493] T a1a39dcd328e4268ae41b988c66b4a75 P 974b802141bd4f13beffef2aa03f88f9 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260129 03:00:18.409657 23695 raft_consensus.cc:3060] T a1a39dcd328e4268ae41b988c66b4a75 P 974b802141bd4f13beffef2aa03f88f9 [term 6 FOLLOWER]: Advancing to term 7
I20260129 03:00:18.410180 20582 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fd8311b779af4e1fa1402b2a02a602f2" candidate_uuid: "197b880fa39d432a9b472845876bfaf0" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "1c50de89a5a845668696dcb28c3badd6" is_pre_election: true
I20260129 03:00:18.383704 20588 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "76f713c3331540d2a5b76164103e9eb1" candidate_uuid: "197b880fa39d432a9b472845876bfaf0" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "1c50de89a5a845668696dcb28c3badd6" is_pre_election: true
I20260129 03:00:18.410434 20583 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "18f30ebaa4174fd28c958e72d0cc7e24" candidate_uuid: "197b880fa39d432a9b472845876bfaf0" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "1c50de89a5a845668696dcb28c3badd6" is_pre_election: true
I20260129 03:00:18.411031 20588 raft_consensus.cc:2468] T 76f713c3331540d2a5b76164103e9eb1 P 1c50de89a5a845668696dcb28c3badd6 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 197b880fa39d432a9b472845876bfaf0 in term 6.
I20260129 03:00:18.411681 20583 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "96609ac68bdf4993a12e1c23355ca013" candidate_uuid: "197b880fa39d432a9b472845876bfaf0" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "1c50de89a5a845668696dcb28c3badd6" is_pre_election: true
I20260129 03:00:18.411962 20588 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "cb4727386a3e48aca5d71cc6c611c197" candidate_uuid: "197b880fa39d432a9b472845876bfaf0" candidate_term: 6 candidate_status { last_received { term: 5 index: 2 } } ignore_live_leader: false dest_uuid: "1c50de89a5a845668696dcb28c3badd6"
I20260129 03:00:18.412164 23707 raft_consensus.cc:2749] T 4408006eb5174de1896b037cf3f18457 P 974b802141bd4f13beffef2aa03f88f9 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260129 03:00:18.395113 23693 raft_consensus.cc:3060] T c2633a6fc34c44579de348d208b0c01e P 974b802141bd4f13beffef2aa03f88f9 [term 5 FOLLOWER]: Advancing to term 6
I20260129 03:00:18.385222 20581 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "cb4727386a3e48aca5d71cc6c611c197" candidate_uuid: "197b880fa39d432a9b472845876bfaf0" candidate_term: 6 candidate_status { last_received { term: 5 index: 2 } } ignore_live_leader: false dest_uuid: "1c50de89a5a845668696dcb28c3badd6" is_pre_election: true
I20260129 03:00:18.414487 20581 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "17d6431cc77a491e8464d68db4081a51" candidate_uuid: "197b880fa39d432a9b472845876bfaf0" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "1c50de89a5a845668696dcb28c3badd6" is_pre_election: true
I20260129 03:00:18.415145 20581 raft_consensus.cc:2468] T 17d6431cc77a491e8464d68db4081a51 P 1c50de89a5a845668696dcb28c3badd6 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 197b880fa39d432a9b472845876bfaf0 in term 6.
I20260129 03:00:18.416121 20581 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fd8311b779af4e1fa1402b2a02a602f2" candidate_uuid: "197b880fa39d432a9b472845876bfaf0" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "1c50de89a5a845668696dcb28c3badd6"
I20260129 03:00:18.384059 20585 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "42d27a28d89b409e8807ec0efd97af69" candidate_uuid: "197b880fa39d432a9b472845876bfaf0" candidate_term: 9 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "1c50de89a5a845668696dcb28c3badd6" is_pre_election: true
I20260129 03:00:18.373342 23671 raft_consensus.cc:493] T 96609ac68bdf4993a12e1c23355ca013 P 974b802141bd4f13beffef2aa03f88f9 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 7a9daf5d42a84615a6383cfec9a10608)
I20260129 03:00:18.383086 20586 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3906b080a5b34a74b8b64128635e8997" candidate_uuid: "197b880fa39d432a9b472845876bfaf0" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "1c50de89a5a845668696dcb28c3badd6" is_pre_election: true
I20260129 03:00:18.417532 20586 raft_consensus.cc:2393] T 3906b080a5b34a74b8b64128635e8997 P 1c50de89a5a845668696dcb28c3badd6 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 197b880fa39d432a9b472845876bfaf0 in current term 5: Already voted for candidate 1c50de89a5a845668696dcb28c3badd6 in this term.
I20260129 03:00:18.356328 23427 raft_consensus.cc:493] T cb4727386a3e48aca5d71cc6c611c197 P 974b802141bd4f13beffef2aa03f88f9 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 1c50de89a5a845668696dcb28c3badd6)
I20260129 03:00:18.413169 23691 raft_consensus.cc:2749] T 3e2d15f8640b407195210c476e09df91 P 974b802141bd4f13beffef2aa03f88f9 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260129 03:00:18.371747 23555 leader_election.cc:290] T 4e420f90884f4796997b5ea41e49e411 P 974b802141bd4f13beffef2aa03f88f9 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 197b880fa39d432a9b472845876bfaf0 (127.18.211.1:42027), 7a9daf5d42a84615a6383cfec9a10608 (127.18.211.4:40439)
W20260129 03:00:18.396467 20405 connection.cc:232] RPC call timeout handler was delayed by -1.4243132750000314s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2882s for a response
W20260129 03:00:18.421056 20405 connection.cc:232] RPC call timeout handler was delayed by -1.3855161410000392s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.3058s for a response
I20260129 03:00:18.406872 21131 rpcz_store.cc:276] 0129 03:00:16.714651 (+ 0us) service_pool.cc:168] Inserting onto call queue
0129 03:00:16.715366 (+ 715us) service_pool.cc:225] Handling call
0129 03:00:16.715696 (+ 330us) raft_consensus.cc:1471] Updating replica for 1 ops
0129 03:00:16.716120 (+ 424us) raft_consensus.cc:1534] Early marking committed up to index 0
0129 03:00:16.716150 (+ 30us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0129 03:00:16.716624 (+ 474us) log.cc:844] Serialized 26 byte log entry
0129 03:00:17.107967 (+391343us) raft_consensus.cc:1645] Marking committed up to 9
0129 03:00:17.788360 (+680393us) log.cc:844] Serialized 14 byte log entry
0129 03:00:17.788564 (+ 204us) raft_consensus.cc:1697] Filling consensus response to leader.
0129 03:00:17.788638 (+ 74us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0129 03:00:18.165028 (+376390us) spinlock_profiling.cc:243] Waited 5.6 ms on lock 0x7b180039e020. stack: 00007f8fe044e373 00007f8fe044e270 00007f8fdfda0574 00007f8fe9a5c6b4 00007f8fe9a5c649 00007f8fe92d4c91 00007f8fe92e1671 00007f8fe04802d3 00007f8fe04826a1 00007f8fe0482659 00007f8fe04825e9 00007f8fe04825b1 00007f8fe04818ad 0000000000379254 0000000000379099 00007f8fe046369a
0129 03:00:18.166703 (+ 1675us) spinlock_profiling.cc:243] Waited 7.95 ms on lock 0x7b180039e020. stack: 00007f8fe044e373 00007f8fe044e270 00007f8fdfda0574 00007f8fe9a5c6b4 00007f8fe9a5c649 00007f8fe92d4c91 00007f8fe92e1671 00007f8fe03f546a 00007f8fe9a7b5c9 00007f8fe74038ef 00007f8fe0480451 00007f8fe04826a1 00007f8fe0482659 00007f8fe04825e9 00007f8fe04825b1 00007f8fe04818ad
0129 03:00:18.380211 (+213508us) spinlock_profiling.cc:243] Waited 27.2 ms on lock 0x7b440002304c. stack: 00007f8fe044e373 00007f8fe044e270 00007f8fdfda0574 00007f8fe9a5c6b4 00007f8fe9a5c649 00007f8fe02c08b1 00007f8fe038f005 00007f8fe038fa22 00007f8fe30d957b 00007f8fe30d4e4f 00007f8fe30c8c0e 00007f8fe30c7f56 00007f8fe30c7b78 00007f8fe30d15c1 00007f8fe30d1579 00007f8fe30d1509
0129 03:00:18.406272 (+ 26061us) raft_consensus.cc:1685] finished
0129 03:00:18.406294 (+ 22us) raft_consensus.cc:1691] UpdateReplicas() finished
0129 03:00:18.406420 (+ 126us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":11240,"prepare.queue_time_us":795616,"prepare.run_cpu_time_us":41,"prepare.run_wall_time_us":42,"raft.queue_time_us":97819,"raft.run_cpu_time_us":265,"raft.run_wall_time_us":27895,"spinlock_wait_cycles":93653632,"thread_start_us":799639,"threads_started":2,"wal-append.queue_time_us":499590}
I20260129 03:00:18.407276 20480 leader_election.cc:304] T 75b0b6e82ddc4f92b1fe1d94f9a0b73b P 974b802141bd4f13beffef2aa03f88f9 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 974b802141bd4f13beffef2aa03f88f9; no voters: 1c50de89a5a845668696dcb28c3badd6, 7a9daf5d42a84615a6383cfec9a10608
I20260129 03:00:18.412752 20583 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "36ff33be666a4c82954731691dbcf327" candidate_uuid: "197b880fa39d432a9b472845876bfaf0" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "1c50de89a5a845668696dcb28c3badd6"
I20260129 03:00:18.397522 20584 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7d9494f5584d4d5db8b9adf1c8fc888b" candidate_uuid: "197b880fa39d432a9b472845876bfaf0" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: true dest_uuid: "1c50de89a5a845668696dcb28c3badd6"
W20260129 03:00:18.421567 20405 connection.cc:232] RPC call timeout handler was delayed by -1.3640568309999708s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.17290000000000003s for a response
I20260129 03:00:18.413102 20588 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "368953c9102643b2a21f99af36b51905" candidate_uuid: "197b880fa39d432a9b472845876bfaf0" candidate_term: 9 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "1c50de89a5a845668696dcb28c3badd6"
I20260129 03:00:18.435245 20588 raft_consensus.cc:3060] T 368953c9102643b2a21f99af36b51905 P 1c50de89a5a845668696dcb28c3badd6 [term 8 FOLLOWER]: Advancing to term 9
W20260129 03:00:18.438889 20554 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.211.4:40439, user_credentials={real_user=slave}} blocked reactor thread for 101190us
I20260129 03:00:18.442806 20437 raft_consensus.cc:1240] T 36ff33be666a4c82954731691dbcf327 P 197b880fa39d432a9b472845876bfaf0 [term 5 FOLLOWER]: Rejecting Update request from peer 1c50de89a5a845668696dcb28c3badd6 for earlier term 4. Current term is 5. Ops: []
I20260129 03:00:18.455276 23262 consensus_queue.cc:1059] T 36ff33be666a4c82954731691dbcf327 P 1c50de89a5a845668696dcb28c3badd6 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 }, Status: INVALID_TERM, Last received: 4.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20260129 03:00:18.456740 23304 raft_consensus.cc:3055] T 36ff33be666a4c82954731691dbcf327 P 1c50de89a5a845668696dcb28c3badd6 [term 4 LEADER]: Stepping down as leader of term 4
I20260129 03:00:18.457216 23304 raft_consensus.cc:740] T 36ff33be666a4c82954731691dbcf327 P 1c50de89a5a845668696dcb28c3badd6 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 1c50de89a5a845668696dcb28c3badd6, State: Running, Role: LEADER
I20260129 03:00:18.458199 23304 consensus_queue.cc:260] T 36ff33be666a4c82954731691dbcf327 P 1c50de89a5a845668696dcb28c3badd6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 4.4, Last appended by leader: 4, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 03:00:18.459483 23304 raft_consensus.cc:3060] T 36ff33be666a4c82954731691dbcf327 P 1c50de89a5a845668696dcb28c3badd6 [term 4 FOLLOWER]: Advancing to term 5
W20260129 03:00:18.478909 20405 leader_election.cc:336] T c891d2bff4ce4568861bbc992e91fbb0 P 197b880fa39d432a9b472845876bfaf0 [CANDIDATE]: Term 9 election: RPC error from VoteRequest() call to peer 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3:37735): Timed out: RequestConsensusVote RPC to 127.18.211.3:37735 timed out after 2.003s (SENT)
I20260129 03:00:18.479664 20405 leader_election.cc:304] T c891d2bff4ce4568861bbc992e91fbb0 P 197b880fa39d432a9b472845876bfaf0 [CANDIDATE]: Term 9 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 197b880fa39d432a9b472845876bfaf0; no voters: 1c50de89a5a845668696dcb28c3badd6, 974b802141bd4f13beffef2aa03f88f9
I20260129 03:00:18.480793 23469 raft_consensus.cc:2749] T c891d2bff4ce4568861bbc992e91fbb0 P 197b880fa39d432a9b472845876bfaf0 [term 9 FOLLOWER]: Leader election lost for term 9. Reason: could not achieve majority
W20260129 03:00:18.482395 20405 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260129 03:00:18.414417 23660 log.cc:927] Time spent T 2ac659d732314e1288849f08fe5e074e P 7a9daf5d42a84615a6383cfec9a10608: Append to log took a long time: real 0.432s user 0.000s sys 0.001s
I20260129 03:00:18.407140 23646 raft_consensus.cc:493] T 724a236524cb4e7ba3428880681d6cc2 P 974b802141bd4f13beffef2aa03f88f9 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260129 03:00:18.494303 20405 connection.cc:232] RPC call timeout handler was delayed by -1.0238568309999891s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.21070000000000003s for a response
I20260129 03:00:18.496799 20581 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "913fc7d751ed4b7388d889aac30a5f9b" candidate_uuid: "197b880fa39d432a9b472845876bfaf0" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "1c50de89a5a845668696dcb28c3badd6" is_pre_election: true
I20260129 03:00:18.497470 20581 raft_consensus.cc:2468] T 913fc7d751ed4b7388d889aac30a5f9b P 1c50de89a5a845668696dcb28c3badd6 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 197b880fa39d432a9b472845876bfaf0 in term 5.
I20260129 03:00:18.498848 20588 raft_consensus.cc:2410] T 368953c9102643b2a21f99af36b51905 P 1c50de89a5a845668696dcb28c3badd6 [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate 197b880fa39d432a9b472845876bfaf0 for term 9 because replica has last-logged OpId of term: 8 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 6 index: 4.
W20260129 03:00:18.498930 20480 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.211.3:37735, user_credentials={real_user=slave}} blocked reactor thread for 91722.7us
I20260129 03:00:18.500759 20480 leader_election.cc:304] T 7d9494f5584d4d5db8b9adf1c8fc888b P 974b802141bd4f13beffef2aa03f88f9 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1c50de89a5a845668696dcb28c3badd6, 974b802141bd4f13beffef2aa03f88f9; no voters: 197b880fa39d432a9b472845876bfaf0
I20260129 03:00:18.502027 20584 raft_consensus.cc:2468] T 7d9494f5584d4d5db8b9adf1c8fc888b P 1c50de89a5a845668696dcb28c3badd6 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 197b880fa39d432a9b472845876bfaf0 in term 5.
I20260129 03:00:18.407423 23668 raft_consensus.cc:3060] T fd8311b779af4e1fa1402b2a02a602f2 P 1c50de89a5a845668696dcb28c3badd6 [term 5 FOLLOWER]: Advancing to term 6
I20260129 03:00:18.412807 23707 raft_consensus.cc:2749] T b189f08e16a74a11980cc55d99359a0a P 974b802141bd4f13beffef2aa03f88f9 [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20260129 03:00:18.512272 20587 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a5a05fbcbbcf4916bbc01fd1a5b5e90d" candidate_uuid: "197b880fa39d432a9b472845876bfaf0" candidate_term: 9 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "1c50de89a5a845668696dcb28c3badd6" is_pre_election: true
I20260129 03:00:18.420938 23630 raft_consensus.cc:493] T 4f8f3a38f6e24ba4857875181f447442 P 974b802141bd4f13beffef2aa03f88f9 [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260129 03:00:18.505734 20586 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "724a236524cb4e7ba3428880681d6cc2" candidate_uuid: "197b880fa39d432a9b472845876bfaf0" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "1c50de89a5a845668696dcb28c3badd6" is_pre_election: true
I20260129 03:00:18.518611 20586 raft_consensus.cc:2468] T 724a236524cb4e7ba3428880681d6cc2 P 1c50de89a5a845668696dcb28c3badd6 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 197b880fa39d432a9b472845876bfaf0 in term 6.
I20260129 03:00:18.416536 23674 leader_election.cc:290] T a1a39dcd328e4268ae41b988c66b4a75 P 197b880fa39d432a9b472845876bfaf0 [CANDIDATE]: Term 7 election: Requested vote from peers 974b802141bd4f13beffef2aa03f88f9 (127.18.211.2:34517), 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3:37735)
I20260129 03:00:18.514043 20554 leader_election.cc:304] T 75f45538dc704d9a8ec4ff8ad9f3ea5b P 1c50de89a5a845668696dcb28c3badd6 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1c50de89a5a845668696dcb28c3badd6, 7a9daf5d42a84615a6383cfec9a10608; no voters:
I20260129 03:00:18.508404 20479 leader_election.cc:304] T 368953c9102643b2a21f99af36b51905 P 974b802141bd4f13beffef2aa03f88f9 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 197b880fa39d432a9b472845876bfaf0, 974b802141bd4f13beffef2aa03f88f9; no voters:
I20260129 03:00:18.516382 20509 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c2633a6fc34c44579de348d208b0c01e" candidate_uuid: "1c50de89a5a845668696dcb28c3badd6" candidate_term: 6 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "974b802141bd4f13beffef2aa03f88f9"
I20260129 03:00:18.517712 20511 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "368953c9102643b2a21f99af36b51905" candidate_uuid: "1c50de89a5a845668696dcb28c3badd6" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "974b802141bd4f13beffef2aa03f88f9" is_pre_election: true
I20260129 03:00:18.522365 20511 raft_consensus.cc:2468] T 368953c9102643b2a21f99af36b51905 P 974b802141bd4f13beffef2aa03f88f9 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1c50de89a5a845668696dcb28c3badd6 in term 9.
I20260129 03:00:18.524597 20514 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3906b080a5b34a74b8b64128635e8997" candidate_uuid: "1c50de89a5a845668696dcb28c3badd6" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "974b802141bd4f13beffef2aa03f88f9" is_pre_election: true
I20260129 03:00:18.525406 20514 raft_consensus.cc:2468] T 3906b080a5b34a74b8b64128635e8997 P 974b802141bd4f13beffef2aa03f88f9 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1c50de89a5a845668696dcb28c3badd6 in term 5.
I20260129 03:00:18.526008 23703 raft_consensus.cc:2243] T 1299723a4476420ab02fd8bfd7aa0645 P 974b802141bd4f13beffef2aa03f88f9 [term 10 FOLLOWER]: Raft consensus shutting down.
I20260129 03:00:18.526372 20513 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e33a417cf9c54cf5af73a4ee3aac4d3a" candidate_uuid: "1c50de89a5a845668696dcb28c3badd6" candidate_term: 8 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "974b802141bd4f13beffef2aa03f88f9"
I20260129 03:00:18.526991 20513 raft_consensus.cc:3060] T e33a417cf9c54cf5af73a4ee3aac4d3a P 974b802141bd4f13beffef2aa03f88f9 [term 7 FOLLOWER]: Advancing to term 8
I20260129 03:00:18.493643 23678 consensus_queue.cc:1059] T cb4727386a3e48aca5d71cc6c611c197 P 1c50de89a5a845668696dcb28c3badd6 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 }, Status: INVALID_TERM, Last received: 5.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.034s
W20260129 03:00:18.522717 23086 consensus_meta.cc:287] T 0f20af7fbe5d4db89a6f17948d98912c P 197b880fa39d432a9b472845876bfaf0: Time spent flushing consensus metadata: real 0.600s user 0.011s sys 0.000s
I20260129 03:00:18.532193 20507 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a1a39dcd328e4268ae41b988c66b4a75" candidate_uuid: "1c50de89a5a845668696dcb28c3badd6" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "974b802141bd4f13beffef2aa03f88f9" is_pre_election: true
I20260129 03:00:18.497233 23646 raft_consensus.cc:3060] T 724a236524cb4e7ba3428880681d6cc2 P 974b802141bd4f13beffef2aa03f88f9 [term 6 FOLLOWER]: Advancing to term 7
I20260129 03:00:18.533551 20437 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00e79d8f25a34676b1d866eb2068a2b2" candidate_uuid: "7a9daf5d42a84615a6383cfec9a10608" candidate_term: 9 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "197b880fa39d432a9b472845876bfaf0" is_pre_election: true
I20260129 03:00:18.536636 20511 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1ce4ddf5b10e4ae8b193b13b2e5936ee" candidate_uuid: "1c50de89a5a845668696dcb28c3badd6" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "974b802141bd4f13beffef2aa03f88f9" is_pre_election: true
I20260129 03:00:18.519887 23630 raft_consensus.cc:3060] T 4f8f3a38f6e24ba4857875181f447442 P 974b802141bd4f13beffef2aa03f88f9 [term 7 FOLLOWER]: Advancing to term 8
I20260129 03:00:18.537312 20511 raft_consensus.cc:2468] T 1ce4ddf5b10e4ae8b193b13b2e5936ee P 974b802141bd4f13beffef2aa03f88f9 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1c50de89a5a845668696dcb28c3badd6 in term 7.
I20260129 03:00:18.539382 20511 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7a9a69cd82724b1faed1d10aa091fc4a" candidate_uuid: "1c50de89a5a845668696dcb28c3badd6" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "974b802141bd4f13beffef2aa03f88f9" is_pre_election: true
I20260129 03:00:18.540201 23663 raft_consensus.cc:493] T 0f20af7fbe5d4db89a6f17948d98912c P 197b880fa39d432a9b472845876bfaf0 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260129 03:00:18.540709 23663 raft_consensus.cc:515] T 0f20af7fbe5d4db89a6f17948d98912c P 197b880fa39d432a9b472845876bfaf0 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 03:00:18.543715 23165 raft_consensus.cc:515] T add8d4923d1a489e86882c3a25692b95 P 7a9daf5d42a84615a6383cfec9a10608 [term 9 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } }
I20260129 03:00:18.545388 20514 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "cb0b3f259ebc4310927338a8740e87b8" candidate_uuid: "1c50de89a5a845668696dcb28c3badd6" candidate_term: 10 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "974b802141bd4f13beffef2aa03f88f9" is_pre_election: true
I20260129 03:00:18.546083 20514 raft_consensus.cc:2468] T cb0b3f259ebc4310927338a8740e87b8 P 974b802141bd4f13beffef2aa03f88f9 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1c50de89a5a845668696dcb28c3badd6 in term 9.
I20260129 03:00:18.525900 20585 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3906b080a5b34a74b8b64128635e8997" candidate_uuid: "197b880fa39d432a9b472845876bfaf0" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "1c50de89a5a845668696dcb28c3badd6"
I20260129 03:00:18.547066 20585 raft_consensus.cc:2393] T 3906b080a5b34a74b8b64128635e8997 P 1c50de89a5a845668696dcb28c3badd6 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 197b880fa39d432a9b472845876bfaf0 in current term 5: Already voted for candidate 1c50de89a5a845668696dcb28c3badd6 in this term.
I20260129 03:00:18.510156 23224 raft_consensus.cc:493] T 1299723a4476420ab02fd8bfd7aa0645 P 197b880fa39d432a9b472845876bfaf0 [term 10 FOLLOWER]: Starting pre-election (detected failure of leader 7a9daf5d42a84615a6383cfec9a10608)
I20260129 03:00:18.547956 20587 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "add8d4923d1a489e86882c3a25692b95" candidate_uuid: "7a9daf5d42a84615a6383cfec9a10608" candidate_term: 9 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "1c50de89a5a845668696dcb28c3badd6"
I20260129 03:00:18.548559 20587 raft_consensus.cc:3060] T add8d4923d1a489e86882c3a25692b95 P 1c50de89a5a845668696dcb28c3badd6 [term 8 FOLLOWER]: Advancing to term 9
I20260129 03:00:18.551167 20513 raft_consensus.cc:2468] T e33a417cf9c54cf5af73a4ee3aac4d3a P 974b802141bd4f13beffef2aa03f88f9 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1c50de89a5a845668696dcb28c3badd6 in term 8.
I20260129 03:00:18.547956 23224 raft_consensus.cc:515] T 1299723a4476420ab02fd8bfd7aa0645 P 197b880fa39d432a9b472845876bfaf0 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } }
I20260129 03:00:18.553754 23703 raft_consensus.cc:2272] T 1299723a4476420ab02fd8bfd7aa0645 P 974b802141bd4f13beffef2aa03f88f9 [term 10 FOLLOWER]: Raft consensus is shut down!
W20260129 03:00:18.552071 20480 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.211.3:37735, user_credentials={real_user=slave}} blocked reactor thread for 51379.7us
I20260129 03:00:18.552757 23708 raft_consensus.cc:2764] T 7d9494f5584d4d5db8b9adf1c8fc888b P 974b802141bd4f13beffef2aa03f88f9 [term 5 FOLLOWER]: Leader pre-election decision vote started in defunct term 4: won
I20260129 03:00:18.496717 22996 raft_consensus.cc:2804] T 75b0b6e82ddc4f92b1fe1d94f9a0b73b P 1c50de89a5a845668696dcb28c3badd6 [term 5 FOLLOWER]: Leader election won for term 5
I20260129 03:00:18.556460 20479 leader_election.cc:304] T 3906b080a5b34a74b8b64128635e8997 P 974b802141bd4f13beffef2aa03f88f9 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 197b880fa39d432a9b472845876bfaf0, 974b802141bd4f13beffef2aa03f88f9; no voters:
I20260129 03:00:18.561543 20508 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c891d2bff4ce4568861bbc992e91fbb0" candidate_uuid: "1c50de89a5a845668696dcb28c3badd6" candidate_term: 9 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "974b802141bd4f13beffef2aa03f88f9"
I20260129 03:00:18.561996 20480 leader_election.cc:304] T a5a05fbcbbcf4916bbc01fd1a5b5e90d P 974b802141bd4f13beffef2aa03f88f9 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 974b802141bd4f13beffef2aa03f88f9; no voters: 197b880fa39d432a9b472845876bfaf0, 1c50de89a5a845668696dcb28c3badd6
W20260129 03:00:18.563161 20554 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.211.4:40439, user_credentials={real_user=slave}} blocked reactor thread for 69831.5us
I20260129 03:00:18.565490 20432 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4e420f90884f4796997b5ea41e49e411" candidate_uuid: "974b802141bd4f13beffef2aa03f88f9" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "197b880fa39d432a9b472845876bfaf0" is_pre_election: true
I20260129 03:00:18.566103 20436 raft_consensus.cc:1275] T febffe4a8b394368b69b1d164308fb9a P 197b880fa39d432a9b472845876bfaf0 [term 6 FOLLOWER]: Refusing update from remote peer 974b802141bd4f13beffef2aa03f88f9: Log matching property violated. Preceding OpId in replica: term: 3 index: 5. Preceding OpId from leader: term: 6 index: 7. (index mismatch)
I20260129 03:00:18.566809 20480 leader_election.cc:304] T 926a250fa5834e88b96ccecbefbf5086 P 974b802141bd4f13beffef2aa03f88f9 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1c50de89a5a845668696dcb28c3badd6, 974b802141bd4f13beffef2aa03f88f9; no voters: 197b880fa39d432a9b472845876bfaf0
I20260129 03:00:18.569248 20512 raft_consensus.cc:1275] T 4b1876f9a70b49edb3aa74105e66d3b3 P 974b802141bd4f13beffef2aa03f88f9 [term 7 FOLLOWER]: Refusing update from remote peer 1c50de89a5a845668696dcb28c3badd6: Log matching property violated. Preceding OpId in replica: term: 4 index: 4. Preceding OpId from leader: term: 7 index: 5. (index mismatch)
I20260129 03:00:18.571134 20432 raft_consensus.cc:1240] T c934d31efd3f4e5195605ddc354d14e1 P 197b880fa39d432a9b472845876bfaf0 [term 7 FOLLOWER]: Rejecting Update request from peer 7a9daf5d42a84615a6383cfec9a10608 for earlier term 6. Current term is 7. Ops: []
I20260129 03:00:18.617444 20480 leader_election.cc:304] T 45cc5f3d2688486d9b6dbe7e666dedde P 974b802141bd4f13beffef2aa03f88f9 [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 974b802141bd4f13beffef2aa03f88f9; no voters: 1c50de89a5a845668696dcb28c3badd6, 7a9daf5d42a84615a6383cfec9a10608
I20260129 03:00:18.619295 20513 raft_consensus.cc:1240] T 0f20af7fbe5d4db89a6f17948d98912c P 974b802141bd4f13beffef2aa03f88f9 [term 8 FOLLOWER]: Rejecting Update request from peer 197b880fa39d432a9b472845876bfaf0 for earlier term 7. Current term is 8. Ops: []
I20260129 03:00:18.585565 20587 raft_consensus.cc:2468] T add8d4923d1a489e86882c3a25692b95 P 1c50de89a5a845668696dcb28c3badd6 [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7a9daf5d42a84615a6383cfec9a10608 in term 9.
I20260129 03:00:18.619235 20440 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00e79d8f25a34676b1d866eb2068a2b2" candidate_uuid: "974b802141bd4f13beffef2aa03f88f9" candidate_term: 9 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "197b880fa39d432a9b472845876bfaf0" is_pre_election: true
I20260129 03:00:18.622045 21100 leader_election.cc:304] T add8d4923d1a489e86882c3a25692b95 P 7a9daf5d42a84615a6383cfec9a10608 [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1c50de89a5a845668696dcb28c3badd6, 7a9daf5d42a84615a6383cfec9a10608; no voters:
W20260129 03:00:18.623732 20554 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.211.4:40439, user_credentials={real_user=slave}} blocked reactor thread for 59661.1us
I20260129 03:00:18.556030 23694 raft_consensus.cc:3060] T 75b0b6e82ddc4f92b1fe1d94f9a0b73b P 974b802141bd4f13beffef2aa03f88f9 [term 4 FOLLOWER]: Advancing to term 5
I20260129 03:00:18.625123 20510 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "724a236524cb4e7ba3428880681d6cc2" candidate_uuid: "197b880fa39d432a9b472845876bfaf0" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "974b802141bd4f13beffef2aa03f88f9" is_pre_election: true
I20260129 03:00:18.496686 23427 raft_consensus.cc:515] T cb4727386a3e48aca5d71cc6c611c197 P 974b802141bd4f13beffef2aa03f88f9 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 03:00:18.628340 20438 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0340229d521b432580ae926f6062e3f7" candidate_uuid: "974b802141bd4f13beffef2aa03f88f9" candidate_term: 9 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "197b880fa39d432a9b472845876bfaf0" is_pre_election: true
I20260129 03:00:18.555819 23224 leader_election.cc:290] T 1299723a4476420ab02fd8bfd7aa0645 P 197b880fa39d432a9b472845876bfaf0 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3:37735), 7a9daf5d42a84615a6383cfec9a10608 (127.18.211.4:40439)
I20260129 03:00:18.628855 20438 raft_consensus.cc:2468] T 0340229d521b432580ae926f6062e3f7 P 197b880fa39d432a9b472845876bfaf0 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 974b802141bd4f13beffef2aa03f88f9 in term 8.
I20260129 03:00:18.519579 23046 raft_consensus.cc:493] T a4c1226b987b468fa84dfc6ba0e2dbe1 P 974b802141bd4f13beffef2aa03f88f9 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 197b880fa39d432a9b472845876bfaf0)
I20260129 03:00:18.569550 23631 raft_consensus.cc:2749] T a5a05fbcbbcf4916bbc01fd1a5b5e90d P 974b802141bd4f13beffef2aa03f88f9 [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
W20260129 03:00:18.631094 20553 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260129 03:00:18.635062 20514 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "17d6431cc77a491e8464d68db4081a51" candidate_uuid: "197b880fa39d432a9b472845876bfaf0" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "974b802141bd4f13beffef2aa03f88f9" is_pre_election: true
I20260129 03:00:18.569759 23262 raft_consensus.cc:2804] T 75f45538dc704d9a8ec4ff8ad9f3ea5b P 1c50de89a5a845668696dcb28c3badd6 [term 5 FOLLOWER]: Leader election won for term 5
I20260129 03:00:18.655541 23262 raft_consensus.cc:697] T 75f45538dc704d9a8ec4ff8ad9f3ea5b P 1c50de89a5a845668696dcb28c3badd6 [term 5 LEADER]: Becoming Leader. State: Replica: 1c50de89a5a845668696dcb28c3badd6, State: Running, Role: LEADER
I20260129 03:00:18.656562 23262 consensus_queue.cc:237] T 75f45538dc704d9a8ec4ff8ad9f3ea5b P 1c50de89a5a845668696dcb28c3badd6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } }
I20260129 03:00:18.557307 21128 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34060 (request call id 574) took 1842 ms. Trace:
I20260129 03:00:18.630087 20511 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "36ff33be666a4c82954731691dbcf327" candidate_uuid: "197b880fa39d432a9b472845876bfaf0" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "974b802141bd4f13beffef2aa03f88f9"
I20260129 03:00:18.557636 23555 raft_consensus.cc:2804] T 368953c9102643b2a21f99af36b51905 P 974b802141bd4f13beffef2aa03f88f9 [term 9 FOLLOWER]: Leader pre-election won for term 10
I20260129 03:00:18.639268 21131 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "cb0b3f259ebc4310927338a8740e87b8" candidate_uuid: "1c50de89a5a845668696dcb28c3badd6" candidate_term: 10 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "7a9daf5d42a84615a6383cfec9a10608" is_pre_election: true
I20260129 03:00:18.526407 23566 raft_consensus.cc:493] T fd8311b779af4e1fa1402b2a02a602f2 P 7a9daf5d42a84615a6383cfec9a10608 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 1c50de89a5a845668696dcb28c3badd6)
I20260129 03:00:18.640390 20405 leader_election.cc:304] T e7d83d4ccf684e76acb29a0dc844e1b8 P 197b880fa39d432a9b472845876bfaf0 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 197b880fa39d432a9b472845876bfaf0, 1c50de89a5a845668696dcb28c3badd6; no voters: 974b802141bd4f13beffef2aa03f88f9
I20260129 03:00:18.658293 21128 rpcz_store.cc:276] 0129 03:00:16.715219 (+ 0us) service_pool.cc:168] Inserting onto call queue
0129 03:00:16.716440 (+ 1221us) service_pool.cc:225] Handling call
0129 03:00:16.720722 (+ 4282us) raft_consensus.cc:1471] Updating replica for 1 ops
0129 03:00:16.780400 (+ 59678us) raft_consensus.cc:1534] Early marking committed up to index 0
0129 03:00:16.780441 (+ 41us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0129 03:00:16.781056 (+ 615us) log.cc:844] Serialized 26 byte log entry
0129 03:00:17.166704 (+385648us) raft_consensus.cc:1645] Marking committed up to 6
0129 03:00:17.169861 (+ 3157us) raft_consensus.cc:1697] Filling consensus response to leader.
0129 03:00:17.177448 (+ 7587us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0129 03:00:18.413673 (+1236225us) spinlock_profiling.cc:243] Waited 6.16 ms on lock 0x7b440002304c. stack: 00007f8fe044e373 00007f8fe044e270 00007f8fdfda0574 00007f8fe9a5c6b4 00007f8fe9a5c649 00007f8fe02c08b1 00007f8fe038f005 00007f8fe038fa22 00007f8fe30d957b 00007f8fe30d4e4f 00007f8fe30c8c0e 00007f8fe30c7f56 00007f8fe30c7b78 00007f8fe30d15c1 00007f8fe30d1579 00007f8fe30d1509
0129 03:00:18.557007 (+143334us) raft_consensus.cc:1685] finished
0129 03:00:18.557036 (+ 29us) raft_consensus.cc:1691] UpdateReplicas() finished
0129 03:00:18.557248 (+ 212us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":241,"raft.queue_time_us":12283,"raft.run_cpu_time_us":95,"raft.run_wall_time_us":98,"spinlock_wait_cycles":14164736,"thread_start_us":345961,"threads_started":1,"wal-append.queue_time_us":1113215}
I20260129 03:00:18.660151 23566 raft_consensus.cc:515] T fd8311b779af4e1fa1402b2a02a602f2 P 7a9daf5d42a84615a6383cfec9a10608 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } }
I20260129 03:00:18.637198 22996 raft_consensus.cc:697] T 75b0b6e82ddc4f92b1fe1d94f9a0b73b P 1c50de89a5a845668696dcb28c3badd6 [term 5 LEADER]: Becoming Leader. State: Replica: 1c50de89a5a845668696dcb28c3badd6, State: Running, Role: LEADER
I20260129 03:00:18.661499 22996 consensus_queue.cc:237] T 75b0b6e82ddc4f92b1fe1d94f9a0b73b P 1c50de89a5a845668696dcb28c3badd6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 4.7, Last appended by leader: 7, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } }
I20260129 03:00:18.662307 23566 leader_election.cc:290] T fd8311b779af4e1fa1402b2a02a602f2 P 7a9daf5d42a84615a6383cfec9a10608 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 197b880fa39d432a9b472845876bfaf0 (127.18.211.1:42027), 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3:37735)
W20260129 03:00:18.642019 23715 log.cc:927] Time spent T 505580c93c694bbb8ea96fdf9b17ab46 P 7a9daf5d42a84615a6383cfec9a10608: Append to log took a long time: real 0.109s user 0.000s sys 0.000s
I20260129 03:00:18.643400 23046 raft_consensus.cc:515] T a4c1226b987b468fa84dfc6ba0e2dbe1 P 974b802141bd4f13beffef2aa03f88f9 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 03:00:18.664036 23709 raft_consensus.cc:2804] T 926a250fa5834e88b96ccecbefbf5086 P 974b802141bd4f13beffef2aa03f88f9 [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260129 03:00:18.642712 23398 consensus_queue.cc:1059] T a644c02b16af47679824ccd060ac10fd P 1c50de89a5a845668696dcb28c3badd6 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false }, Status: INVALID_TERM, Last received: 5.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.001s
I20260129 03:00:18.623266 23304 raft_consensus.cc:493] T 0340229d521b432580ae926f6062e3f7 P 1c50de89a5a845668696dcb28c3badd6 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 974b802141bd4f13beffef2aa03f88f9)
I20260129 03:00:18.665021 23046 leader_election.cc:290] T a4c1226b987b468fa84dfc6ba0e2dbe1 P 974b802141bd4f13beffef2aa03f88f9 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 197b880fa39d432a9b472845876bfaf0 (127.18.211.1:42027), 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3:37735)
I20260129 03:00:18.532260 23699 raft_consensus.cc:2804] T e33a417cf9c54cf5af73a4ee3aac4d3a P 1c50de89a5a845668696dcb28c3badd6 [term 8 FOLLOWER]: Leader election won for term 8
I20260129 03:00:18.665373 23398 raft_consensus.cc:3055] T a644c02b16af47679824ccd060ac10fd P 1c50de89a5a845668696dcb28c3badd6 [term 5 LEADER]: Stepping down as leader of term 5
I20260129 03:00:18.665762 23699 raft_consensus.cc:697] T e33a417cf9c54cf5af73a4ee3aac4d3a P 1c50de89a5a845668696dcb28c3badd6 [term 8 LEADER]: Becoming Leader. State: Replica: 1c50de89a5a845668696dcb28c3badd6, State: Running, Role: LEADER
I20260129 03:00:18.665513 23304 raft_consensus.cc:515] T 0340229d521b432580ae926f6062e3f7 P 1c50de89a5a845668696dcb28c3badd6 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 03:00:18.626277 23450 raft_consensus.cc:493] T 4f5a77e94fef4b16a2ba7b067c8b144a P 974b802141bd4f13beffef2aa03f88f9 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 7a9daf5d42a84615a6383cfec9a10608)
I20260129 03:00:18.645270 23686 consensus_queue.cc:1048] T a0fcc58d6fb6473cafbd9d34d5195269 P 197b880fa39d432a9b472845876bfaf0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.008s
I20260129 03:00:18.645735 20516 tablet_service.cc:2066] Received LeaderStepDown RPC: tablet_id: "724a236524cb4e7ba3428880681d6cc2"
dest_uuid: "974b802141bd4f13beffef2aa03f88f9"
mode: GRACEFUL
new_leader_uuid: "197b880fa39d432a9b472845876bfaf0"
from {username='slave'} at 127.0.0.1:52008
I20260129 03:00:18.648461 20480 leader_election.cc:304] T 4b1876f9a70b49edb3aa74105e66d3b3 P 974b802141bd4f13beffef2aa03f88f9 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 974b802141bd4f13beffef2aa03f88f9; no voters: 197b880fa39d432a9b472845876bfaf0, 1c50de89a5a845668696dcb28c3badd6
I20260129 03:00:18.652938 20512 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "913fc7d751ed4b7388d889aac30a5f9b" candidate_uuid: "197b880fa39d432a9b472845876bfaf0" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "974b802141bd4f13beffef2aa03f88f9" is_pre_election: true
I20260129 03:00:18.564739 23165 leader_election.cc:290] T add8d4923d1a489e86882c3a25692b95 P 7a9daf5d42a84615a6383cfec9a10608 [CANDIDATE]: Term 9 election: Requested vote from peers 974b802141bd4f13beffef2aa03f88f9 (127.18.211.2:34517), 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3:37735)
W20260129 03:00:18.653440 20555 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260129 03:00:18.653858 20509 raft_consensus.cc:2410] T c2633a6fc34c44579de348d208b0c01e P 974b802141bd4f13beffef2aa03f88f9 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 1c50de89a5a845668696dcb28c3badd6 for term 6 because replica has last-logged OpId of term: 5 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 2.
I20260129 03:00:18.654991 20514 raft_consensus.cc:2468] T 17d6431cc77a491e8464d68db4081a51 P 974b802141bd4f13beffef2aa03f88f9 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 197b880fa39d432a9b472845876bfaf0 in term 6.
I20260129 03:00:18.664654 23709 raft_consensus.cc:493] T 926a250fa5834e88b96ccecbefbf5086 P 974b802141bd4f13beffef2aa03f88f9 [term 8 FOLLOWER]: Starting leader election (detected failure of leader 197b880fa39d432a9b472845876bfaf0)
I20260129 03:00:18.681957 20585 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fd8311b779af4e1fa1402b2a02a602f2" candidate_uuid: "7a9daf5d42a84615a6383cfec9a10608" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "1c50de89a5a845668696dcb28c3badd6" is_pre_election: true
I20260129 03:00:18.682140 20439 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "87d5fe48a44a408aaf03e894a67ecb2f" candidate_uuid: "7a9daf5d42a84615a6383cfec9a10608" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "197b880fa39d432a9b472845876bfaf0" is_pre_election: true
I20260129 03:00:18.566048 23707 raft_consensus.cc:2804] T 3906b080a5b34a74b8b64128635e8997 P 974b802141bd4f13beffef2aa03f88f9 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260129 03:00:18.644495 20513 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "cb4727386a3e48aca5d71cc6c611c197" candidate_uuid: "197b880fa39d432a9b472845876bfaf0" candidate_term: 6 candidate_status { last_received { term: 5 index: 2 } } ignore_live_leader: false dest_uuid: "974b802141bd4f13beffef2aa03f88f9"
I20260129 03:00:18.672290 23466 raft_consensus.cc:2804] T e7d83d4ccf684e76acb29a0dc844e1b8 P 197b880fa39d432a9b472845876bfaf0 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260129 03:00:18.672978 20405 leader_election.cc:304] T c2633a6fc34c44579de348d208b0c01e P 197b880fa39d432a9b472845876bfaf0 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 197b880fa39d432a9b472845876bfaf0; no voters: 1c50de89a5a845668696dcb28c3badd6, 974b802141bd4f13beffef2aa03f88f9
I20260129 03:00:18.709380 20405 leader_election.cc:304] T fcddab6bca684adf8ea032289a00f90e P 197b880fa39d432a9b472845876bfaf0 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 197b880fa39d432a9b472845876bfaf0, 1c50de89a5a845668696dcb28c3badd6; no voters: 974b802141bd4f13beffef2aa03f88f9
W20260129 03:00:18.674999 20404 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260129 03:00:18.488931 23671 raft_consensus.cc:515] T 96609ac68bdf4993a12e1c23355ca013 P 974b802141bd4f13beffef2aa03f88f9 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } attrs { replace: true } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } }
I20260129 03:00:18.633479 23669 raft_consensus.cc:493] T 45cc5f3d2688486d9b6dbe7e666dedde P 1c50de89a5a845668696dcb28c3badd6 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260129 03:00:18.635794 23163 raft_consensus.cc:2804] T add8d4923d1a489e86882c3a25692b95 P 7a9daf5d42a84615a6383cfec9a10608 [term 9 FOLLOWER]: Leader election won for term 9
I20260129 03:00:18.679952 20433 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:48356 (request call id 461) took 2110 ms. Trace:
I20260129 03:00:18.665846 23398 raft_consensus.cc:740] T a644c02b16af47679824ccd060ac10fd P 1c50de89a5a845668696dcb28c3badd6 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 1c50de89a5a845668696dcb28c3badd6, State: Running, Role: LEADER
I20260129 03:00:18.634750 23680 raft_consensus.cc:3055] T cb4727386a3e48aca5d71cc6c611c197 P 1c50de89a5a845668696dcb28c3badd6 [term 5 LEADER]: Stepping down as leader of term 5
I20260129 03:00:18.685886 20480 leader_election.cc:304] T cedce331e1584ac8b636c9b8a72b2ba0 P 974b802141bd4f13beffef2aa03f88f9 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1c50de89a5a845668696dcb28c3badd6, 974b802141bd4f13beffef2aa03f88f9; no voters: 197b880fa39d432a9b472845876bfaf0
I20260129 03:00:18.618723 23691 consensus_queue.cc:1048] T fcddab6bca684adf8ea032289a00f90e P 974b802141bd4f13beffef2aa03f88f9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 4, Time since last communication: 0.105s
I20260129 03:00:18.681849 23450 raft_consensus.cc:515] T 4f5a77e94fef4b16a2ba7b067c8b144a P 974b802141bd4f13beffef2aa03f88f9 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } }
I20260129 03:00:18.677397 23333 raft_consensus.cc:2749] T 45cc5f3d2688486d9b6dbe7e666dedde P 974b802141bd4f13beffef2aa03f88f9 [term 7 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
I20260129 03:00:18.679903 23627 raft_consensus.cc:493] T 4e3805ffa70242e8ade66da260dddf66 P 197b880fa39d432a9b472845876bfaf0 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 7a9daf5d42a84615a6383cfec9a10608)
I20260129 03:00:18.678418 23693 raft_consensus.cc:493] T 8055232cf0004d40a85f3bf983fe00a9 P 974b802141bd4f13beffef2aa03f88f9 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260129 03:00:18.690204 23514 raft_consensus.cc:2749] T add8d4923d1a489e86882c3a25692b95 P 974b802141bd4f13beffef2aa03f88f9 [term 8 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260129 03:00:18.687831 23046 raft_consensus.cc:2749] T 4b1876f9a70b49edb3aa74105e66d3b3 P 974b802141bd4f13beffef2aa03f88f9 [term 7 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260129 03:00:18.737084 23427 leader_election.cc:290] T cb4727386a3e48aca5d71cc6c611c197 P 974b802141bd4f13beffef2aa03f88f9 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 197b880fa39d432a9b472845876bfaf0 (127.18.211.1:42027), 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3:37735)
I20260129 03:00:18.687723 23707 raft_consensus.cc:493] T 3906b080a5b34a74b8b64128635e8997 P 974b802141bd4f13beffef2aa03f88f9 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260129 03:00:18.696463 23711 log.cc:927] Time spent T e7d83d4ccf684e76acb29a0dc844e1b8 P 197b880fa39d432a9b472845876bfaf0: Append to log took a long time: real 0.062s user 0.000s sys 0.000s
I20260129 03:00:18.698043 23703 ts_tablet_manager.cc:1916] T 1299723a4476420ab02fd8bfd7aa0645 P 974b802141bd4f13beffef2aa03f88f9: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260129 03:00:18.635478 23708 consensus_queue.cc:1048] T febffe4a8b394368b69b1d164308fb9a P 974b802141bd4f13beffef2aa03f88f9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 5, Time since last communication: 0.064s
I20260129 03:00:18.714130 20508 raft_consensus.cc:2468] T c891d2bff4ce4568861bbc992e91fbb0 P 974b802141bd4f13beffef2aa03f88f9 [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1c50de89a5a845668696dcb28c3badd6 in term 9.
I20260129 03:00:18.748646 20508 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "febffe4a8b394368b69b1d164308fb9a" candidate_uuid: "197b880fa39d432a9b472845876bfaf0" candidate_term: 7 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "974b802141bd4f13beffef2aa03f88f9" is_pre_election: true
W20260129 03:00:18.715116 20479 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.211.1:42027, user_credentials={real_user=slave}} blocked reactor thread for 88275.4us
I20260129 03:00:18.715451 20556 leader_election.cc:304] T cedce331e1584ac8b636c9b8a72b2ba0 P 1c50de89a5a845668696dcb28c3badd6 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1c50de89a5a845668696dcb28c3badd6; no voters: 197b880fa39d432a9b472845876bfaf0, 974b802141bd4f13beffef2aa03f88f9
W20260129 03:00:18.716322 20554 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.211.4:40439, user_credentials={real_user=slave}} blocked reactor thread for 80930.8us
I20260129 03:00:18.667212 23555 raft_consensus.cc:493] T 368953c9102643b2a21f99af36b51905 P 974b802141bd4f13beffef2aa03f88f9 [term 9 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260129 03:00:18.725611 20440 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0340229d521b432580ae926f6062e3f7" candidate_uuid: "1c50de89a5a845668696dcb28c3badd6" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "197b880fa39d432a9b472845876bfaf0" is_pre_election: true
I20260129 03:00:18.754729 20440 raft_consensus.cc:2393] T 0340229d521b432580ae926f6062e3f7 P 197b880fa39d432a9b472845876bfaf0 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1c50de89a5a845668696dcb28c3badd6 in current term 8: Already voted for candidate 197b880fa39d432a9b472845876bfaf0 in this term.
I20260129 03:00:18.727412 20511 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00e79d8f25a34676b1d866eb2068a2b2" candidate_uuid: "7a9daf5d42a84615a6383cfec9a10608" candidate_term: 9 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "974b802141bd4f13beffef2aa03f88f9" is_pre_election: true
I20260129 03:00:18.734725 23693 raft_consensus.cc:515] T 8055232cf0004d40a85f3bf983fe00a9 P 974b802141bd4f13beffef2aa03f88f9 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 03:00:18.687331 23709 raft_consensus.cc:3060] T 926a250fa5834e88b96ccecbefbf5086 P 974b802141bd4f13beffef2aa03f88f9 [term 8 FOLLOWER]: Advancing to term 9
I20260129 03:00:18.695619 23646 raft_consensus.cc:515] T 724a236524cb4e7ba3428880681d6cc2 P 974b802141bd4f13beffef2aa03f88f9 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 03:00:18.721557 20433 rpcz_store.cc:276] 0129 03:00:16.570027 (+ 0us) service_pool.cc:168] Inserting onto call queue
0129 03:00:16.570176 (+ 149us) service_pool.cc:225] Handling call
0129 03:00:16.570517 (+ 341us) raft_consensus.cc:1471] Updating replica for 0 ops
0129 03:00:16.588858 (+ 18341us) raft_consensus.cc:1534] Early marking committed up to index 8
0129 03:00:16.605780 (+ 16922us) log.cc:844] Serialized 14 byte log entry
0129 03:00:18.679491 (+2073711us) spinlock_profiling.cc:243] Waited 21.7 ms on lock 0x7b180029c140. stack: 00007f8fe044e373 00007f8fe044e270 00007f8fdfda0574 00007f8fe9a5c6b4 00007f8fe9a5c649 00007f8fe92d4c91 00007f8fe92e1671 00007f8fe30d585c 00007f8fe04632eb 00007f8fe924b709 00007f8fe047f57b 00007f8fe047d81f 00007f8fe047d335 00007f8fe3525ce0 00007f8fe3538396 00007f8fe3548f07
0129 03:00:18.679589 (+ 98us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0129 03:00:18.679652 (+ 63us) raft_consensus.cc:1645] Marking committed up to 8
0129 03:00:18.679693 (+ 41us) raft_consensus.cc:1697] Filling consensus response to leader.
0129 03:00:18.679757 (+ 64us) raft_consensus.cc:1691] UpdateReplicas() finished
0129 03:00:18.679898 (+ 141us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":2618,"prepare.queue_time_us":22068,"prepare.run_cpu_time_us":32,"prepare.run_wall_time_us":33,"raft.queue_time_us":15462,"raft.run_cpu_time_us":215,"raft.run_wall_time_us":11178,"spinlock_wait_cycles":49846528,"thread_start_us":535497,"threads_started":2,"wal-append.queue_time_us":536386,"wal-append.run_cpu_time_us":1385,"wal-append.run_wall_time_us":1042908}
I20260129 03:00:18.708199 23466 raft_consensus.cc:493] T e7d83d4ccf684e76acb29a0dc844e1b8 P 197b880fa39d432a9b472845876bfaf0 [term 7 FOLLOWER]: Starting leader election (detected failure of leader 974b802141bd4f13beffef2aa03f88f9)
I20260129 03:00:18.720168 23224 raft_consensus.cc:2749] T c2633a6fc34c44579de348d208b0c01e P 197b880fa39d432a9b472845876bfaf0 [term 6 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260129 03:00:18.768396 20438 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a609ac02fae84725ab743f95f51d33ce" candidate_uuid: "7a9daf5d42a84615a6383cfec9a10608" candidate_term: 10 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "197b880fa39d432a9b472845876bfaf0" is_pre_election: true
W20260129 03:00:18.744825 23716 log.cc:927] Time spent T c2633a6fc34c44579de348d208b0c01e P 197b880fa39d432a9b472845876bfaf0: Append to log took a long time: real 0.057s user 0.000s sys 0.001s
I20260129 03:00:18.746317 20437 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fd8311b779af4e1fa1402b2a02a602f2" candidate_uuid: "7a9daf5d42a84615a6383cfec9a10608" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "197b880fa39d432a9b472845876bfaf0" is_pre_election: true
I20260129 03:00:18.684872 22645 consensus_queue.cc:1048] T 926a250fa5834e88b96ccecbefbf5086 P 197b880fa39d432a9b472845876bfaf0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 3, Time since last communication: 0.049s
I20260129 03:00:18.756197 20511 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "87d5fe48a44a408aaf03e894a67ecb2f" candidate_uuid: "7a9daf5d42a84615a6383cfec9a10608" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "974b802141bd4f13beffef2aa03f88f9" is_pre_election: true
I20260129 03:00:18.666884 23699 consensus_queue.cc:237] T e33a417cf9c54cf5af73a4ee3aac4d3a P 1c50de89a5a845668696dcb28c3badd6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 6.3, Last appended by leader: 3, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 03:00:18.790963 20511 raft_consensus.cc:2468] T 87d5fe48a44a408aaf03e894a67ecb2f P 974b802141bd4f13beffef2aa03f88f9 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7a9daf5d42a84615a6383cfec9a10608 in term 7.
I20260129 03:00:18.757899 21099 leader_election.cc:304] T 2e58e9a5a8aa4cecafb464addb4f35ed P 7a9daf5d42a84615a6383cfec9a10608 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7a9daf5d42a84615a6383cfec9a10608; no voters: 197b880fa39d432a9b472845876bfaf0, 1c50de89a5a845668696dcb28c3badd6
I20260129 03:00:18.792753 20437 raft_consensus.cc:2393] T fd8311b779af4e1fa1402b2a02a602f2 P 197b880fa39d432a9b472845876bfaf0 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 7a9daf5d42a84615a6383cfec9a10608 in current term 6: Already voted for candidate 197b880fa39d432a9b472845876bfaf0 in this term.
I20260129 03:00:18.724192 23680 raft_consensus.cc:740] T cb4727386a3e48aca5d71cc6c611c197 P 1c50de89a5a845668696dcb28c3badd6 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 1c50de89a5a845668696dcb28c3badd6, State: Running, Role: LEADER
I20260129 03:00:18.728369 23631 raft_consensus.cc:2804] T cedce331e1584ac8b636c9b8a72b2ba0 P 974b802141bd4f13beffef2aa03f88f9 [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260129 03:00:18.761456 20479 leader_election.cc:304] T 0340229d521b432580ae926f6062e3f7 P 974b802141bd4f13beffef2aa03f88f9 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 197b880fa39d432a9b472845876bfaf0, 974b802141bd4f13beffef2aa03f88f9; no voters:
I20260129 03:00:18.761569 20508 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3906b080a5b34a74b8b64128635e8997" candidate_uuid: "197b880fa39d432a9b472845876bfaf0" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "974b802141bd4f13beffef2aa03f88f9"
I20260129 03:00:18.765426 20407 leader_election.cc:304] T c934d31efd3f4e5195605ddc354d14e1 P 197b880fa39d432a9b472845876bfaf0 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 197b880fa39d432a9b472845876bfaf0, 974b802141bd4f13beffef2aa03f88f9; no voters: 7a9daf5d42a84615a6383cfec9a10608
I20260129 03:00:18.739300 23163 raft_consensus.cc:697] T add8d4923d1a489e86882c3a25692b95 P 7a9daf5d42a84615a6383cfec9a10608 [term 9 LEADER]: Becoming Leader. State: Replica: 7a9daf5d42a84615a6383cfec9a10608, State: Running, Role: LEADER
I20260129 03:00:18.767163 20405 leader_election.cc:304] T a8479d1fe0bd41e5946186e3a6ca8c7a P 197b880fa39d432a9b472845876bfaf0 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 197b880fa39d432a9b472845876bfaf0, 1c50de89a5a845668696dcb28c3badd6; no voters: 974b802141bd4f13beffef2aa03f88f9
I20260129 03:00:18.771347 23602 raft_consensus.cc:2749] T cedce331e1584ac8b636c9b8a72b2ba0 P 1c50de89a5a845668696dcb28c3badd6 [term 8 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260129 03:00:18.772260 20512 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a5a05fbcbbcf4916bbc01fd1a5b5e90d" candidate_uuid: "197b880fa39d432a9b472845876bfaf0" candidate_term: 9 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "974b802141bd4f13beffef2aa03f88f9" is_pre_election: true
I20260129 03:00:18.743765 23707 raft_consensus.cc:3060] T 3906b080a5b34a74b8b64128635e8997 P 974b802141bd4f13beffef2aa03f88f9 [term 5 FOLLOWER]: Advancing to term 6
I20260129 03:00:18.759398 23555 raft_consensus.cc:3060] T 368953c9102643b2a21f99af36b51905 P 974b802141bd4f13beffef2aa03f88f9 [term 9 FOLLOWER]: Advancing to term 10
I20260129 03:00:18.795300 20516 raft_consensus.cc:606] T 724a236524cb4e7ba3428880681d6cc2 P 974b802141bd4f13beffef2aa03f88f9 [term 7 FOLLOWER]: Received request to transfer leadership to TS 197b880fa39d432a9b472845876bfaf0
I20260129 03:00:18.724035 23669 raft_consensus.cc:515] T 45cc5f3d2688486d9b6dbe7e666dedde P 1c50de89a5a845668696dcb28c3badd6 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } }
I20260129 03:00:18.821465 20516 raft_consensus.cc:614] T 724a236524cb4e7ba3428880681d6cc2 P 974b802141bd4f13beffef2aa03f88f9 [term 7 FOLLOWER]: Rejecting request to transer leadership while not leader
W20260129 03:00:18.821833 20405 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.211.3:37735, user_credentials={real_user=slave}} blocked reactor thread for 54754.4us
I20260129 03:00:18.822643 20510 raft_consensus.cc:2393] T 724a236524cb4e7ba3428880681d6cc2 P 974b802141bd4f13beffef2aa03f88f9 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 197b880fa39d432a9b472845876bfaf0 in current term 7: Already voted for candidate 974b802141bd4f13beffef2aa03f88f9 in this term.
W20260129 03:00:18.767221 23679 log.cc:927] Time spent T 36ff33be666a4c82954731691dbcf327 P 974b802141bd4f13beffef2aa03f88f9: Append to log took a long time: real 0.429s user 0.004s sys 0.000s
I20260129 03:00:18.794621 23570 raft_consensus.cc:2749] T 2e58e9a5a8aa4cecafb464addb4f35ed P 7a9daf5d42a84615a6383cfec9a10608 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260129 03:00:18.815702 20512 raft_consensus.cc:2468] T a5a05fbcbbcf4916bbc01fd1a5b5e90d P 974b802141bd4f13beffef2aa03f88f9 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 197b880fa39d432a9b472845876bfaf0 in term 8.
I20260129 03:00:18.817662 20589 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0340229d521b432580ae926f6062e3f7" candidate_uuid: "197b880fa39d432a9b472845876bfaf0" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "1c50de89a5a845668696dcb28c3badd6"
I20260129 03:00:18.891709 20587 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "17d6431cc77a491e8464d68db4081a51" candidate_uuid: "974b802141bd4f13beffef2aa03f88f9" candidate_term: 6 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "1c50de89a5a845668696dcb28c3badd6"
I20260129 03:00:18.801540 20511 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a644c02b16af47679824ccd060ac10fd" candidate_uuid: "7a9daf5d42a84615a6383cfec9a10608" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "974b802141bd4f13beffef2aa03f88f9"
I20260129 03:00:18.794245 23695 raft_consensus.cc:515] T a1a39dcd328e4268ae41b988c66b4a75 P 974b802141bd4f13beffef2aa03f88f9 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } } peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } }
W20260129 03:00:18.881018 20407 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.211.2:34517, user_credentials={real_user=slave}} blocked reactor thread for 59075.8us
I20260129 03:00:18.892627 20587 raft_consensus.cc:2376] T 17d6431cc77a491e8464d68db4081a51 P 1c50de89a5a845668696dcb28c3badd6 [term 6 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 974b802141bd4f13beffef2aa03f88f9 in term 6. Re-sending same reply.
I20260129 03:00:18.768823 23466 raft_consensus.cc:3060] T e7d83d4ccf684e76acb29a0dc844e1b8 P 197b880fa39d432a9b472845876bfaf0 [term 7 FOLLOWER]: Advancing to term 8
I20260129 03:00:18.754045 23627 raft_consensus.cc:515] T 4e3805ffa70242e8ade66da260dddf66 P 197b880fa39d432a9b472845876bfaf0 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } }
I20260129 03:00:18.890065 20437 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a4c1226b987b468fa84dfc6ba0e2dbe1" candidate_uuid: "974b802141bd4f13beffef2aa03f88f9" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "197b880fa39d432a9b472845876bfaf0" is_pre_election: true
I20260129 03:00:18.875063 20438 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "cb4727386a3e48aca5d71cc6c611c197" candidate_uuid: "974b802141bd4f13beffef2aa03f88f9" candidate_term: 6 candidate_status { last_received { term: 5 index: 2 } } ignore_live_leader: false dest_uuid: "197b880fa39d432a9b472845876bfaf0" is_pre_election: true
I20260129 03:00:18.815080 23640 consensus_queue.cc:1059] T c934d31efd3f4e5195605ddc354d14e1 P 7a9daf5d42a84615a6383cfec9a10608 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 }, Status: INVALID_TERM, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.005s
I20260129 03:00:18.892134 20589 raft_consensus.cc:3060] T 0340229d521b432580ae926f6062e3f7 P 1c50de89a5a845668696dcb28c3badd6 [term 7 FOLLOWER]: Advancing to term 8
I20260129 03:00:18.894160 20407 leader_election.cc:304] T 17d6431cc77a491e8464d68db4081a51 P 197b880fa39d432a9b472845876bfaf0 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 197b880fa39d432a9b472845876bfaf0, 974b802141bd4f13beffef2aa03f88f9; no voters:
I20260129 03:00:18.794333 23631 raft_consensus.cc:493] T cedce331e1584ac8b636c9b8a72b2ba0 P 974b802141bd4f13beffef2aa03f88f9 [term 8 FOLLOWER]: Starting leader election (detected failure of leader 197b880fa39d432a9b472845876bfaf0)
I20260129 03:00:18.902663 20438 raft_consensus.cc:2393] T cb4727386a3e48aca5d71cc6c611c197 P 197b880fa39d432a9b472845876bfaf0 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 974b802141bd4f13beffef2aa03f88f9 in current term 6: Already voted for candidate 197b880fa39d432a9b472845876bfaf0 in this term.
I20260129 03:00:18.815209 23673 raft_consensus.cc:2804] T c934d31efd3f4e5195605ddc354d14e1 P 197b880fa39d432a9b472845876bfaf0 [term 7 FOLLOWER]: Leader election won for term 7
I20260129 03:00:18.817327 23427 raft_consensus.cc:2804] T 0340229d521b432580ae926f6062e3f7 P 974b802141bd4f13beffef2aa03f88f9 [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260129 03:00:18.900456 20587 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0f20af7fbe5d4db89a6f17948d98912c" candidate_uuid: "974b802141bd4f13beffef2aa03f88f9" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "1c50de89a5a845668696dcb28c3badd6"
I20260129 03:00:18.901383 20586 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "75be4bd6bd45474795fd472eff2953ac" candidate_uuid: "974b802141bd4f13beffef2aa03f88f9" candidate_term: 11 candidate_status { last_received { term: 10 index: 7 } } ignore_live_leader: false dest_uuid: "1c50de89a5a845668696dcb28c3badd6" is_pre_election: true
I20260129 03:00:18.896270 23450 leader_election.cc:290] T 4f5a77e94fef4b16a2ba7b067c8b144a P 974b802141bd4f13beffef2aa03f88f9 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3:37735), 7a9daf5d42a84615a6383cfec9a10608 (127.18.211.4:40439)
I20260129 03:00:18.874680 23403 raft_consensus.cc:2804] T a8479d1fe0bd41e5946186e3a6ca8c7a P 197b880fa39d432a9b472845876bfaf0 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260129 03:00:18.907173 20585 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a0fcc58d6fb6473cafbd9d34d5195269" candidate_uuid: "974b802141bd4f13beffef2aa03f88f9" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "1c50de89a5a845668696dcb28c3badd6" is_pre_election: true
I20260129 03:00:18.907079 20511 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "45cc5f3d2688486d9b6dbe7e666dedde" candidate_uuid: "7a9daf5d42a84615a6383cfec9a10608" candidate_term: 8 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "974b802141bd4f13beffef2aa03f88f9" is_pre_election: true
W20260129 03:00:18.908649 20556 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.211.2:34517, user_credentials={real_user=slave}} blocked reactor thread for 228489us
I20260129 03:00:18.910344 20587 raft_consensus.cc:2376] T 0f20af7fbe5d4db89a6f17948d98912c P 1c50de89a5a845668696dcb28c3badd6 [term 8 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 974b802141bd4f13beffef2aa03f88f9 in term 8. Re-sending same reply.
I20260129 03:00:18.912999 20586 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "36ff33be666a4c82954731691dbcf327" candidate_uuid: "974b802141bd4f13beffef2aa03f88f9" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "1c50de89a5a845668696dcb28c3badd6" is_pre_election: true
I20260129 03:00:18.913902 20584 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7a9a69cd82724b1faed1d10aa091fc4a" candidate_uuid: "974b802141bd4f13beffef2aa03f88f9" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "1c50de89a5a845668696dcb28c3badd6" is_pre_election: true
I20260129 03:00:18.915081 20588 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "368953c9102643b2a21f99af36b51905" candidate_uuid: "974b802141bd4f13beffef2aa03f88f9" candidate_term: 10 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "1c50de89a5a845668696dcb28c3badd6" is_pre_election: true
I20260129 03:00:18.916502 23640 raft_consensus.cc:3055] T c934d31efd3f4e5195605ddc354d14e1 P 7a9daf5d42a84615a6383cfec9a10608 [term 6 LEADER]: Stepping down as leader of term 6
I20260129 03:00:18.908068 23631 raft_consensus.cc:3060] T cedce331e1584ac8b636c9b8a72b2ba0 P 974b802141bd4f13beffef2aa03f88f9 [term 8 FOLLOWER]: Advancing to term 9
I20260129 03:00:18.922400 20511 raft_consensus.cc:2468] T 45cc5f3d2688486d9b6dbe7e666dedde P 974b802141bd4f13beffef2aa03f88f9 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7a9daf5d42a84615a6383cfec9a10608 in term 7.
I20260129 03:00:18.925015 20584 raft_consensus.cc:2468] T 7a9a69cd82724b1faed1d10aa091fc4a P 1c50de89a5a845668696dcb28c3badd6 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 974b802141bd4f13beffef2aa03f88f9 in term 7.
I20260129 03:00:18.925053 20588 raft_consensus.cc:2468] T 368953c9102643b2a21f99af36b51905 P 1c50de89a5a845668696dcb28c3badd6 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 974b802141bd4f13beffef2aa03f88f9 in term 9.
I20260129 03:00:18.956831 20440 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "8055232cf0004d40a85f3bf983fe00a9" candidate_uuid: "974b802141bd4f13beffef2aa03f88f9" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "197b880fa39d432a9b472845876bfaf0" is_pre_election: true
I20260129 03:00:18.957535 20507 raft_consensus.cc:2368] T a1a39dcd328e4268ae41b988c66b4a75 P 974b802141bd4f13beffef2aa03f88f9 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1c50de89a5a845668696dcb28c3badd6 for earlier term 6. Current term is 7.
I20260129 03:00:18.956831 20512 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "add8d4923d1a489e86882c3a25692b95" candidate_uuid: "7a9daf5d42a84615a6383cfec9a10608" candidate_term: 9 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "974b802141bd4f13beffef2aa03f88f9" is_pre_election: true
I20260129 03:00:18.790612 23398 consensus_queue.cc:260] T a644c02b16af47679824ccd060ac10fd P 1c50de89a5a845668696dcb28c3badd6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 5.7, Last appended by leader: 7, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } }
I20260129 03:00:18.926414 20585 raft_consensus.cc:2468] T a0fcc58d6fb6473cafbd9d34d5195269 P 1c50de89a5a845668696dcb28c3badd6 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 974b802141bd4f13beffef2aa03f88f9 in term 5.
I20260129 03:00:19.042106 21127 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4e420f90884f4796997b5ea41e49e411" candidate_uuid: "974b802141bd4f13beffef2aa03f88f9" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "7a9daf5d42a84615a6383cfec9a10608" is_pre_election: true
I20260129 03:00:19.046161 20437 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "96609ac68bdf4993a12e1c23355ca013" candidate_uuid: "974b802141bd4f13beffef2aa03f88f9" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "197b880fa39d432a9b472845876bfaf0" is_pre_election: true
I20260129 03:00:19.022080 21133 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00e79d8f25a34676b1d866eb2068a2b2" candidate_uuid: "974b802141bd4f13beffef2aa03f88f9" candidate_term: 9 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "7a9daf5d42a84615a6383cfec9a10608" is_pre_election: true
I20260129 03:00:19.010429 21131 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "87d5fe48a44a408aaf03e894a67ecb2f" candidate_uuid: "974b802141bd4f13beffef2aa03f88f9" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "7a9daf5d42a84615a6383cfec9a10608" is_pre_election: true
I20260129 03:00:19.043304 20441 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "724a236524cb4e7ba3428880681d6cc2" candidate_uuid: "974b802141bd4f13beffef2aa03f88f9" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "197b880fa39d432a9b472845876bfaf0"
I20260129 03:00:19.025336 21136 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4f5a77e94fef4b16a2ba7b067c8b144a" candidate_uuid: "974b802141bd4f13beffef2aa03f88f9" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "7a9daf5d42a84615a6383cfec9a10608" is_pre_election: true
I20260129 03:00:19.050043 20441 raft_consensus.cc:3060] T 724a236524cb4e7ba3428880681d6cc2 P 197b880fa39d432a9b472845876bfaf0 [term 6 FOLLOWER]: Advancing to term 7
I20260129 03:00:19.015838 21134 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "75f45538dc704d9a8ec4ff8ad9f3ea5b" candidate_uuid: "974b802141bd4f13beffef2aa03f88f9" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "7a9daf5d42a84615a6383cfec9a10608" is_pre_election: true
I20260129 03:00:19.051726 20512 raft_consensus.cc:2468] T add8d4923d1a489e86882c3a25692b95 P 974b802141bd4f13beffef2aa03f88f9 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7a9daf5d42a84615a6383cfec9a10608 in term 8.
I20260129 03:00:18.965144 21102 leader_election.cc:304] T 00e79d8f25a34676b1d866eb2068a2b2 P 7a9daf5d42a84615a6383cfec9a10608 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7a9daf5d42a84615a6383cfec9a10608; no voters: 197b880fa39d432a9b472845876bfaf0, 974b802141bd4f13beffef2aa03f88f9
I20260129 03:00:19.052953 21134 raft_consensus.cc:2393] T 75f45538dc704d9a8ec4ff8ad9f3ea5b P 7a9daf5d42a84615a6383cfec9a10608 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 974b802141bd4f13beffef2aa03f88f9 in current term 5: Already voted for candidate 1c50de89a5a845668696dcb28c3badd6 in this term.
I20260129 03:00:18.929503 20556 leader_election.cc:304] T 8055232cf0004d40a85f3bf983fe00a9 P 1c50de89a5a845668696dcb28c3badd6 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1c50de89a5a845668696dcb28c3badd6; no voters: 197b880fa39d432a9b472845876bfaf0, 974b802141bd4f13beffef2aa03f88f9
I20260129 03:00:18.959403 20511 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "add8d4923d1a489e86882c3a25692b95" candidate_uuid: "7a9daf5d42a84615a6383cfec9a10608" candidate_term: 9 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "974b802141bd4f13beffef2aa03f88f9"
I20260129 03:00:19.059707 20511 raft_consensus.cc:3060] T add8d4923d1a489e86882c3a25692b95 P 974b802141bd4f13beffef2aa03f88f9 [term 8 FOLLOWER]: Advancing to term 9
I20260129 03:00:18.925871 20581 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "75f45538dc704d9a8ec4ff8ad9f3ea5b" candidate_uuid: "974b802141bd4f13beffef2aa03f88f9" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "1c50de89a5a845668696dcb28c3badd6" is_pre_election: true
I20260129 03:00:18.958534 20584 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4f5a77e94fef4b16a2ba7b067c8b144a" candidate_uuid: "974b802141bd4f13beffef2aa03f88f9" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "1c50de89a5a845668696dcb28c3badd6" is_pre_election: true
I20260129 03:00:19.049218 21133 raft_consensus.cc:2468] T 00e79d8f25a34676b1d866eb2068a2b2 P 7a9daf5d42a84615a6383cfec9a10608 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 974b802141bd4f13beffef2aa03f88f9 in term 8.
I20260129 03:00:18.925212 20586 raft_consensus.cc:2410] T 36ff33be666a4c82954731691dbcf327 P 1c50de89a5a845668696dcb28c3badd6 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 974b802141bd4f13beffef2aa03f88f9 for term 5 because replica has last-logged OpId of term: 4 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 3.
I20260129 03:00:18.941107 20587 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0340229d521b432580ae926f6062e3f7" candidate_uuid: "974b802141bd4f13beffef2aa03f88f9" candidate_term: 9 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "1c50de89a5a845668696dcb28c3badd6" is_pre_election: true
I20260129 03:00:19.012503 20480 leader_election.cc:304] T 17d6431cc77a491e8464d68db4081a51 P 974b802141bd4f13beffef2aa03f88f9 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1c50de89a5a845668696dcb28c3badd6, 974b802141bd4f13beffef2aa03f88f9; no voters: 197b880fa39d432a9b472845876bfaf0
I20260129 03:00:18.958024 20440 raft_consensus.cc:2468] T 8055232cf0004d40a85f3bf983fe00a9 P 197b880fa39d432a9b472845876bfaf0 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 974b802141bd4f13beffef2aa03f88f9 in term 6.
I20260129 03:00:18.917871 23427 raft_consensus.cc:493] T 0340229d521b432580ae926f6062e3f7 P 974b802141bd4f13beffef2aa03f88f9 [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260129 03:00:19.078135 20479 leader_election.cc:304] T 8055232cf0004d40a85f3bf983fe00a9 P 974b802141bd4f13beffef2aa03f88f9 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 197b880fa39d432a9b472845876bfaf0, 974b802141bd4f13beffef2aa03f88f9; no voters:
I20260129 03:00:19.077983 20584 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a4c1226b987b468fa84dfc6ba0e2dbe1" candidate_uuid: "974b802141bd4f13beffef2aa03f88f9" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "1c50de89a5a845668696dcb28c3badd6" is_pre_election: true
I20260129 03:00:19.076300 23669 leader_election.cc:290] T 45cc5f3d2688486d9b6dbe7e666dedde P 1c50de89a5a845668696dcb28c3badd6 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 974b802141bd4f13beffef2aa03f88f9 (127.18.211.2:34517), 7a9daf5d42a84615a6383cfec9a10608 (127.18.211.4:40439)
I20260129 03:00:18.921538 23403 raft_consensus.cc:493] T a8479d1fe0bd41e5946186e3a6ca8c7a P 197b880fa39d432a9b472845876bfaf0 [term 5 FOLLOWER]: Starting leader election (detected failure of leader 974b802141bd4f13beffef2aa03f88f9)
I20260129 03:00:19.003580 21128 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2e58e9a5a8aa4cecafb464addb4f35ed" candidate_uuid: "1c50de89a5a845668696dcb28c3badd6" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "7a9daf5d42a84615a6383cfec9a10608"
I20260129 03:00:19.091331 20516 raft_consensus.cc:1240] T c934d31efd3f4e5195605ddc354d14e1 P 974b802141bd4f13beffef2aa03f88f9 [term 7 FOLLOWER]: Rejecting Update request from peer 7a9daf5d42a84615a6383cfec9a10608 for earlier term 6. Current term is 7. Ops: []
I20260129 03:00:19.088346 20585 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "8055232cf0004d40a85f3bf983fe00a9" candidate_uuid: "974b802141bd4f13beffef2aa03f88f9" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "1c50de89a5a845668696dcb28c3badd6" is_pre_election: true
I20260129 03:00:19.095811 20588 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "cb4727386a3e48aca5d71cc6c611c197" candidate_uuid: "974b802141bd4f13beffef2aa03f88f9" candidate_term: 6 candidate_status { last_received { term: 5 index: 2 } } ignore_live_leader: false dest_uuid: "1c50de89a5a845668696dcb28c3badd6" is_pre_election: true
I20260129 03:00:19.096306 20585 raft_consensus.cc:2468] T 8055232cf0004d40a85f3bf983fe00a9 P 1c50de89a5a845668696dcb28c3badd6 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 974b802141bd4f13beffef2aa03f88f9 in term 6.
W20260129 03:00:19.070852 21102 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.211.2:34517, user_credentials={real_user=slave}} blocked reactor thread for 105806us
I20260129 03:00:18.923365 23686 raft_consensus.cc:2804] T 17d6431cc77a491e8464d68db4081a51 P 197b880fa39d432a9b472845876bfaf0 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260129 03:00:18.998677 23691 raft_consensus.cc:493] T a609ac02fae84725ab743f95f51d33ce P 974b802141bd4f13beffef2aa03f88f9 [term 9 FOLLOWER]: Starting pre-election (detected failure of leader 197b880fa39d432a9b472845876bfaf0)
W20260129 03:00:19.097965 20480 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.211.3:37735, user_credentials={real_user=slave}} blocked reactor thread for 85577.7us
W20260129 03:00:19.084264 20405 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.211.3:37735, user_credentials={real_user=slave}} blocked reactor thread for 189494us
I20260129 03:00:18.917339 23693 leader_election.cc:290] T 8055232cf0004d40a85f3bf983fe00a9 P 974b802141bd4f13beffef2aa03f88f9 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 197b880fa39d432a9b472845876bfaf0 (127.18.211.1:42027), 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3:37735)
I20260129 03:00:19.052316 23646 leader_election.cc:290] T 724a236524cb4e7ba3428880681d6cc2 P 974b802141bd4f13beffef2aa03f88f9 [CANDIDATE]: Term 7 election: Requested vote from peers 197b880fa39d432a9b472845876bfaf0 (127.18.211.1:42027), 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3:37735)
I20260129 03:00:18.925704 23640 raft_consensus.cc:740] T c934d31efd3f4e5195605ddc354d14e1 P 7a9daf5d42a84615a6383cfec9a10608 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 7a9daf5d42a84615a6383cfec9a10608, State: Running, Role: LEADER
I20260129 03:00:19.005077 23682 raft_consensus.cc:493] T dc2f4cb752cf4d59b5f6bc689f5d8104 P 7a9daf5d42a84615a6383cfec9a10608 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 1c50de89a5a845668696dcb28c3badd6)
I20260129 03:00:19.097746 20512 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a1a39dcd328e4268ae41b988c66b4a75" candidate_uuid: "197b880fa39d432a9b472845876bfaf0" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "974b802141bd4f13beffef2aa03f88f9"
I20260129 03:00:19.100167 21102 leader_election.cc:304] T 87d5fe48a44a408aaf03e894a67ecb2f P 7a9daf5d42a84615a6383cfec9a10608 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 7a9daf5d42a84615a6383cfec9a10608, 974b802141bd4f13beffef2aa03f88f9; no voters: 197b880fa39d432a9b472845876bfaf0
I20260129 03:00:19.078917 23570 raft_consensus.cc:493] T 75b0b6e82ddc4f92b1fe1d94f9a0b73b P 7a9daf5d42a84615a6383cfec9a10608 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260129 03:00:19.119374 20480 leader_election.cc:304] T 75be4bd6bd45474795fd472eff2953ac P 974b802141bd4f13beffef2aa03f88f9 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 974b802141bd4f13beffef2aa03f88f9; no voters: 197b880fa39d432a9b472845876bfaf0, 1c50de89a5a845668696dcb28c3badd6
I20260129 03:00:19.101508 21133 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "45cc5f3d2688486d9b6dbe7e666dedde" candidate_uuid: "1c50de89a5a845668696dcb28c3badd6" candidate_term: 8 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "7a9daf5d42a84615a6383cfec9a10608" is_pre_election: true
I20260129 03:00:19.130612 21133 raft_consensus.cc:2468] T 45cc5f3d2688486d9b6dbe7e666dedde P 7a9daf5d42a84615a6383cfec9a10608 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1c50de89a5a845668696dcb28c3badd6 in term 7.
I20260129 03:00:19.131811 20554 leader_election.cc:304] T 45cc5f3d2688486d9b6dbe7e666dedde P 1c50de89a5a845668696dcb28c3badd6 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1c50de89a5a845668696dcb28c3badd6, 7a9daf5d42a84615a6383cfec9a10608; no voters:
I20260129 03:00:19.087067 23427 raft_consensus.cc:3060] T 0340229d521b432580ae926f6062e3f7 P 974b802141bd4f13beffef2aa03f88f9 [term 8 FOLLOWER]: Advancing to term 9
I20260129 03:00:19.077924 23688 raft_consensus.cc:2749] T 00e79d8f25a34676b1d866eb2068a2b2 P 7a9daf5d42a84615a6383cfec9a10608 [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
W20260129 03:00:19.127408 20481 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260129 03:00:19.094200 23403 raft_consensus.cc:3060] T a8479d1fe0bd41e5946186e3a6ca8c7a P 197b880fa39d432a9b472845876bfaf0 [term 5 FOLLOWER]: Advancing to term 6
I20260129 03:00:19.101181 23398 raft_consensus.cc:3060] T a644c02b16af47679824ccd060ac10fd P 1c50de89a5a845668696dcb28c3badd6 [term 5 FOLLOWER]: Advancing to term 6
I20260129 03:00:19.102317 23717 raft_consensus.cc:2804] T 17d6431cc77a491e8464d68db4081a51 P 974b802141bd4f13beffef2aa03f88f9 [term 6 FOLLOWER]: Leader election won for term 6
I20260129 03:00:19.139951 20405 leader_election.cc:304] T 4f8f3a38f6e24ba4857875181f447442 P 197b880fa39d432a9b472845876bfaf0 [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 197b880fa39d432a9b472845876bfaf0; no voters: 1c50de89a5a845668696dcb28c3badd6, 974b802141bd4f13beffef2aa03f88f9
I20260129 03:00:19.149690 20511 raft_consensus.cc:2468] T add8d4923d1a489e86882c3a25692b95 P 974b802141bd4f13beffef2aa03f88f9 [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7a9daf5d42a84615a6383cfec9a10608 in term 9.
I20260129 03:00:19.145964 23570 raft_consensus.cc:515] T 75b0b6e82ddc4f92b1fe1d94f9a0b73b P 7a9daf5d42a84615a6383cfec9a10608 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } }
I20260129 03:00:18.965345 23163 consensus_queue.cc:237] T add8d4923d1a489e86882c3a25692b95 P 7a9daf5d42a84615a6383cfec9a10608 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 7.6, Last appended by leader: 6, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } }
I20260129 03:00:19.115188 23333 raft_consensus.cc:2804] T 8055232cf0004d40a85f3bf983fe00a9 P 974b802141bd4f13beffef2aa03f88f9 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260129 03:00:19.166321 20512 raft_consensus.cc:1275] T cb0b3f259ebc4310927338a8740e87b8 P 974b802141bd4f13beffef2aa03f88f9 [term 9 FOLLOWER]: Refusing update from remote peer 7a9daf5d42a84615a6383cfec9a10608: Log matching property violated. Preceding OpId in replica: term: 7 index: 8. Preceding OpId from leader: term: 9 index: 9. (index mismatch)
W20260129 03:00:19.167091 20556 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.211.2:34517, user_credentials={real_user=slave}} blocked reactor thread for 244536us
I20260129 03:00:18.965080 23680 consensus_queue.cc:260] T cb4727386a3e48aca5d71cc6c611c197 P 1c50de89a5a845668696dcb28c3badd6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 5.2, Last appended by leader: 2, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 03:00:19.170671 20437 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a1a39dcd328e4268ae41b988c66b4a75" candidate_uuid: "974b802141bd4f13beffef2aa03f88f9" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "197b880fa39d432a9b472845876bfaf0"
W20260129 03:00:19.173074 20480 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.211.3:37735, user_credentials={real_user=slave}} blocked reactor thread for 53780.9us
W20260129 03:00:19.145030 20407 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.211.2:34517, user_credentials={real_user=slave}} blocked reactor thread for 251154us
W20260129 03:00:19.175328 21135 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34046 (request call id 612) took 2376 ms (client timeout 1645 ms). Trace:
I20260129 03:00:19.120740 23673 raft_consensus.cc:697] T c934d31efd3f4e5195605ddc354d14e1 P 197b880fa39d432a9b472845876bfaf0 [term 7 LEADER]: Becoming Leader. State: Replica: 197b880fa39d432a9b472845876bfaf0, State: Running, Role: LEADER
I20260129 03:00:19.120324 23686 raft_consensus.cc:493] T 17d6431cc77a491e8464d68db4081a51 P 197b880fa39d432a9b472845876bfaf0 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260129 03:00:19.155814 23695 leader_election.cc:290] T a1a39dcd328e4268ae41b988c66b4a75 P 974b802141bd4f13beffef2aa03f88f9 [CANDIDATE]: Term 7 election: Requested vote from peers 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3:37735), 197b880fa39d432a9b472845876bfaf0 (127.18.211.1:42027)
I20260129 03:00:19.153988 22996 raft_consensus.cc:2749] T 8055232cf0004d40a85f3bf983fe00a9 P 1c50de89a5a845668696dcb28c3badd6 [term 6 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260129 03:00:19.127856 23691 raft_consensus.cc:515] T a609ac02fae84725ab743f95f51d33ce P 974b802141bd4f13beffef2aa03f88f9 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } }
I20260129 03:00:19.169080 23646 raft_consensus.cc:2749] T 75be4bd6bd45474795fd472eff2953ac P 974b802141bd4f13beffef2aa03f88f9 [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
W20260129 03:00:19.245754 21135 rpcz_store.cc:269] 0129 03:00:16.788788 (+ 0us) service_pool.cc:168] Inserting onto call queue
0129 03:00:16.798908 (+ 10120us) service_pool.cc:225] Handling call
0129 03:00:19.165547 (+2366639us) spinlock_profiling.cc:243] Waited 200 ms on lock 0x7b54004c95d4. stack: 00007f8fe044e373 00007f8fe044e270 00007f8fdfda0574 00007f8fe9a5c6b4 00007f8fe9a5c649 00007f8fe34975a1 00007f8fe3531f7a 00007f8fe84e3647 00007f8fe1a73834 00007f8fe1a737c2 00007f8fe1a736f1 00007f8fe1a7366c 00007f8fe1a72922 00007f8fe186d34c 00007f8fe186c786 00007f8fe186c12f
0129 03:00:19.165659 (+ 112us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":459040512}
I20260129 03:00:19.247171 20437 raft_consensus.cc:2393] T a1a39dcd328e4268ae41b988c66b4a75 P 197b880fa39d432a9b472845876bfaf0 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 974b802141bd4f13beffef2aa03f88f9 in current term 7: Already voted for candidate 197b880fa39d432a9b472845876bfaf0 in this term.
I20260129 03:00:19.248682 20407 leader_election.cc:304] T a5a05fbcbbcf4916bbc01fd1a5b5e90d P 197b880fa39d432a9b472845876bfaf0 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 197b880fa39d432a9b472845876bfaf0, 974b802141bd4f13beffef2aa03f88f9; no voters:
W20260129 03:00:19.178196 20555 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260129 03:00:19.165460 23531 raft_consensus.cc:2804] T 45cc5f3d2688486d9b6dbe7e666dedde P 1c50de89a5a845668696dcb28c3badd6 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260129 03:00:19.237936 20480 leader_election.cc:304] T 0f20af7fbe5d4db89a6f17948d98912c P 974b802141bd4f13beffef2aa03f88f9 [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1c50de89a5a845668696dcb28c3badd6, 974b802141bd4f13beffef2aa03f88f9; no voters: 197b880fa39d432a9b472845876bfaf0
I20260129 03:00:19.146488 23682 raft_consensus.cc:515] T dc2f4cb752cf4d59b5f6bc689f5d8104 P 7a9daf5d42a84615a6383cfec9a10608 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } }
I20260129 03:00:19.264707 21129 raft_consensus.cc:1275] T 2e58e9a5a8aa4cecafb464addb4f35ed P 7a9daf5d42a84615a6383cfec9a10608 [term 5 FOLLOWER]: Refusing update from remote peer 1c50de89a5a845668696dcb28c3badd6: Log matching property violated. Preceding OpId in replica: term: 4 index: 6. Preceding OpId from leader: term: 5 index: 7. (index mismatch)
I20260129 03:00:19.222365 23333 raft_consensus.cc:493] T 8055232cf0004d40a85f3bf983fe00a9 P 974b802141bd4f13beffef2aa03f88f9 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260129 03:00:19.173863 23717 raft_consensus.cc:697] T 17d6431cc77a491e8464d68db4081a51 P 974b802141bd4f13beffef2aa03f88f9 [term 6 LEADER]: Becoming Leader. State: Replica: 974b802141bd4f13beffef2aa03f88f9, State: Running, Role: LEADER
I20260129 03:00:19.272476 20556 leader_election.cc:304] T 4408006eb5174de1896b037cf3f18457 P 1c50de89a5a845668696dcb28c3badd6 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1c50de89a5a845668696dcb28c3badd6; no voters: 197b880fa39d432a9b472845876bfaf0, 974b802141bd4f13beffef2aa03f88f9
W20260129 03:00:19.281653 21100 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.211.3:37735, user_credentials={real_user=slave}} blocked reactor thread for 168145us
I20260129 03:00:19.224889 23692 raft_consensus.cc:2804] T 87d5fe48a44a408aaf03e894a67ecb2f P 7a9daf5d42a84615a6383cfec9a10608 [term 7 FOLLOWER]: Leader pre-election won for term 8
W20260129 03:00:19.273749 20554 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.211.4:40439, user_credentials={real_user=slave}} blocked reactor thread for 95149.9us
W20260129 03:00:19.278005 21102 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.211.2:34517, user_credentials={real_user=slave}} blocked reactor thread for 177941us
W20260129 03:00:19.262737 23668 consensus_meta.cc:287] T fd8311b779af4e1fa1402b2a02a602f2 P 1c50de89a5a845668696dcb28c3badd6: Time spent flushing consensus metadata: real 0.732s user 0.010s sys 0.002s
I20260129 03:00:19.231185 23686 raft_consensus.cc:3060] T 17d6431cc77a491e8464d68db4081a51 P 197b880fa39d432a9b472845876bfaf0 [term 6 FOLLOWER]: Advancing to term 7
I20260129 03:00:19.284833 21102 leader_election.cc:304] T a644c02b16af47679824ccd060ac10fd P 7a9daf5d42a84615a6383cfec9a10608 [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7a9daf5d42a84615a6383cfec9a10608; no voters: 1c50de89a5a845668696dcb28c3badd6, 974b802141bd4f13beffef2aa03f88f9
W20260129 03:00:19.288306 20405 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.211.3:37735, user_credentials={real_user=slave}} blocked reactor thread for 168716us
I20260129 03:00:19.260017 20589 raft_consensus.cc:2468] T 0340229d521b432580ae926f6062e3f7 P 1c50de89a5a845668696dcb28c3badd6 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 197b880fa39d432a9b472845876bfaf0 in term 8.
I20260129 03:00:19.289834 21135 raft_consensus.cc:1275] T 75f45538dc704d9a8ec4ff8ad9f3ea5b P 7a9daf5d42a84615a6383cfec9a10608 [term 5 FOLLOWER]: Refusing update from remote peer 1c50de89a5a845668696dcb28c3badd6: Log matching property violated. Preceding OpId in replica: term: 4 index: 6. Preceding OpId from leader: term: 5 index: 7. (index mismatch)
I20260129 03:00:19.168946 23671 leader_election.cc:290] T 96609ac68bdf4993a12e1c23355ca013 P 974b802141bd4f13beffef2aa03f88f9 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 197b880fa39d432a9b472845876bfaf0 (127.18.211.1:42027), 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3:37735), 7a9daf5d42a84615a6383cfec9a10608 (127.18.211.4:40439)
I20260129 03:00:19.250535 23086 raft_consensus.cc:2749] T 4f8f3a38f6e24ba4857875181f447442 P 197b880fa39d432a9b472845876bfaf0 [term 7 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
I20260129 03:00:19.284806 20582 raft_consensus.cc:2468] T fd8311b779af4e1fa1402b2a02a602f2 P 1c50de89a5a845668696dcb28c3badd6 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 197b880fa39d432a9b472845876bfaf0 in term 6.
W20260129 03:00:19.277483 23724 log.cc:927] Time spent T dc2f4cb752cf4d59b5f6bc689f5d8104 P 7a9daf5d42a84615a6383cfec9a10608: Append to log took a long time: real 0.150s user 0.000s sys 0.001s
I20260129 03:00:19.264346 23708 raft_consensus.cc:493] T 2ac659d732314e1288849f08fe5e074e P 974b802141bd4f13beffef2aa03f88f9 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 197b880fa39d432a9b472845876bfaf0)
I20260129 03:00:19.271166 23531 raft_consensus.cc:493] T 45cc5f3d2688486d9b6dbe7e666dedde P 1c50de89a5a845668696dcb28c3badd6 [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260129 03:00:19.295015 20405 leader_election.cc:304] T 18f30ebaa4174fd28c958e72d0cc7e24 P 197b880fa39d432a9b472845876bfaf0 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 197b880fa39d432a9b472845876bfaf0; no voters: 1c50de89a5a845668696dcb28c3badd6, 7a9daf5d42a84615a6383cfec9a10608
I20260129 03:00:19.281224 23333 raft_consensus.cc:3060] T 8055232cf0004d40a85f3bf983fe00a9 P 974b802141bd4f13beffef2aa03f88f9 [term 6 FOLLOWER]: Advancing to term 7
I20260129 03:00:19.273823 23727 raft_consensus.cc:2804] T a5a05fbcbbcf4916bbc01fd1a5b5e90d P 197b880fa39d432a9b472845876bfaf0 [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260129 03:00:19.283571 23692 raft_consensus.cc:493] T 87d5fe48a44a408aaf03e894a67ecb2f P 7a9daf5d42a84615a6383cfec9a10608 [term 7 FOLLOWER]: Starting leader election (detected failure of leader 197b880fa39d432a9b472845876bfaf0)
I20260129 03:00:19.312914 20589 tablet_service.cc:2066] Received LeaderStepDown RPC: tablet_id: "75be4bd6bd45474795fd472eff2953ac"
dest_uuid: "1c50de89a5a845668696dcb28c3badd6"
mode: GRACEFUL
new_leader_uuid: "197b880fa39d432a9b472845876bfaf0"
from {username='slave'} at 127.0.0.1:54860
W20260129 03:00:19.294322 23720 log.cc:927] Time spent T 75f45538dc704d9a8ec4ff8ad9f3ea5b P 1c50de89a5a845668696dcb28c3badd6: Append to log took a long time: real 0.283s user 0.001s sys 0.000s
I20260129 03:00:19.317831 20589 raft_consensus.cc:606] T 75be4bd6bd45474795fd472eff2953ac P 1c50de89a5a845668696dcb28c3badd6 [term 10 FOLLOWER]: Received request to transfer leadership to TS 197b880fa39d432a9b472845876bfaf0
I20260129 03:00:19.318213 20589 raft_consensus.cc:614] T 75be4bd6bd45474795fd472eff2953ac P 1c50de89a5a845668696dcb28c3badd6 [term 10 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260129 03:00:19.249820 23640 consensus_queue.cc:260] T c934d31efd3f4e5195605ddc354d14e1 P 7a9daf5d42a84615a6383cfec9a10608 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 6.8, Last appended by leader: 8, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } }
I20260129 03:00:19.296473 23680 raft_consensus.cc:3060] T cb4727386a3e48aca5d71cc6c611c197 P 1c50de89a5a845668696dcb28c3badd6 [term 5 FOLLOWER]: Advancing to term 6
W20260129 03:00:19.320535 20556 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.211.2:34517, user_credentials={real_user=slave}} blocked reactor thread for 70994.5us
W20260129 03:00:19.320060 20480 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.211.3:37735, user_credentials={real_user=slave}} blocked reactor thread for 82257.9us
I20260129 03:00:19.300165 23531 raft_consensus.cc:3060] T 45cc5f3d2688486d9b6dbe7e666dedde P 1c50de89a5a845668696dcb28c3badd6 [term 7 FOLLOWER]: Advancing to term 8
I20260129 03:00:19.297421 23602 raft_consensus.cc:2749] T 4408006eb5174de1896b037cf3f18457 P 1c50de89a5a845668696dcb28c3badd6 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260129 03:00:19.327041 20480 leader_election.cc:304] T 7a9a69cd82724b1faed1d10aa091fc4a P 974b802141bd4f13beffef2aa03f88f9 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1c50de89a5a845668696dcb28c3badd6, 974b802141bd4f13beffef2aa03f88f9; no voters: 197b880fa39d432a9b472845876bfaf0
I20260129 03:00:19.343055 20480 leader_election.cc:304] T a0fcc58d6fb6473cafbd9d34d5195269 P 974b802141bd4f13beffef2aa03f88f9 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1c50de89a5a845668696dcb28c3badd6, 974b802141bd4f13beffef2aa03f88f9; no voters: 197b880fa39d432a9b472845876bfaf0
W20260129 03:00:19.335299 20554 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260129 03:00:19.346326 21135 raft_consensus.cc:1275] T 75b0b6e82ddc4f92b1fe1d94f9a0b73b P 7a9daf5d42a84615a6383cfec9a10608 [term 5 FOLLOWER]: Refusing update from remote peer 1c50de89a5a845668696dcb28c3badd6: Log matching property violated. Preceding OpId in replica: term: 4 index: 7. Preceding OpId from leader: term: 5 index: 8. (index mismatch)
I20260129 03:00:19.347620 23695 raft_consensus.cc:2804] T 7a9a69cd82724b1faed1d10aa091fc4a P 974b802141bd4f13beffef2aa03f88f9 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260129 03:00:19.348166 23695 raft_consensus.cc:493] T 7a9a69cd82724b1faed1d10aa091fc4a P 974b802141bd4f13beffef2aa03f88f9 [term 7 FOLLOWER]: Starting leader election (detected failure of leader 197b880fa39d432a9b472845876bfaf0)
I20260129 03:00:19.348618 23695 raft_consensus.cc:3060] T 7a9a69cd82724b1faed1d10aa091fc4a P 974b802141bd4f13beffef2aa03f88f9 [term 7 FOLLOWER]: Advancing to term 8
I20260129 03:00:19.352118 23735 raft_consensus.cc:2804] T a0fcc58d6fb6473cafbd9d34d5195269 P 974b802141bd4f13beffef2aa03f88f9 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260129 03:00:19.352741 23735 raft_consensus.cc:493] T a0fcc58d6fb6473cafbd9d34d5195269 P 974b802141bd4f13beffef2aa03f88f9 [term 5 FOLLOWER]: Starting leader election (detected failure of leader 197b880fa39d432a9b472845876bfaf0)
I20260129 03:00:19.353214 23735 raft_consensus.cc:3060] T a0fcc58d6fb6473cafbd9d34d5195269 P 974b802141bd4f13beffef2aa03f88f9 [term 5 FOLLOWER]: Advancing to term 6
I20260129 03:00:19.361536 20480 leader_election.cc:304] T 36ff33be666a4c82954731691dbcf327 P 974b802141bd4f13beffef2aa03f88f9 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 974b802141bd4f13beffef2aa03f88f9; no voters: 197b880fa39d432a9b472845876bfaf0, 1c50de89a5a845668696dcb28c3badd6
I20260129 03:00:19.368418 23671 raft_consensus.cc:3060] T 36ff33be666a4c82954731691dbcf327 P 974b802141bd4f13beffef2aa03f88f9 [term 4 FOLLOWER]: Advancing to term 5
I20260129 03:00:19.377571 20480 leader_election.cc:304] T a4c1226b987b468fa84dfc6ba0e2dbe1 P 974b802141bd4f13beffef2aa03f88f9 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 974b802141bd4f13beffef2aa03f88f9; no voters: 197b880fa39d432a9b472845876bfaf0, 1c50de89a5a845668696dcb28c3badd6
I20260129 03:00:19.378906 23646 raft_consensus.cc:2749] T a4c1226b987b468fa84dfc6ba0e2dbe1 P 974b802141bd4f13beffef2aa03f88f9 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260129 03:00:19.385583 23695 raft_consensus.cc:515] T 7a9a69cd82724b1faed1d10aa091fc4a P 974b802141bd4f13beffef2aa03f88f9 [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 03:00:19.389906 23671 raft_consensus.cc:2749] T 36ff33be666a4c82954731691dbcf327 P 974b802141bd4f13beffef2aa03f88f9 [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
W20260129 03:00:19.323915 23721 log.cc:927] Time spent T 75b0b6e82ddc4f92b1fe1d94f9a0b73b P 1c50de89a5a845668696dcb28c3badd6: Append to log took a long time: real 0.275s user 0.001s sys 0.000s
I20260129 03:00:19.435740 20434 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55672 (request call id 1806) took 1379 ms. Trace:
I20260129 03:00:19.320611 23692 raft_consensus.cc:3060] T 87d5fe48a44a408aaf03e894a67ecb2f P 7a9daf5d42a84615a6383cfec9a10608 [term 7 FOLLOWER]: Advancing to term 8
W20260129 03:00:19.433560 20405 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.211.3:37735, user_credentials={real_user=slave}} blocked reactor thread for 138621us
I20260129 03:00:19.436362 20434 rpcz_store.cc:276] 0129 03:00:18.056237 (+ 0us) service_pool.cc:168] Inserting onto call queue
0129 03:00:18.057961 (+ 1724us) service_pool.cc:225] Handling call
0129 03:00:18.068515 (+ 10554us) raft_consensus.cc:1471] Updating replica for 0 ops
0129 03:00:18.068749 (+ 234us) raft_consensus.cc:1534] Early marking committed up to index 6
0129 03:00:18.088173 (+ 19424us) log.cc:844] Serialized 14 byte log entry
0129 03:00:19.174652 (+1086479us) spinlock_profiling.cc:243] Waited 8.1 ms on lock 0x7b1800419320. stack: 00007f8fe044e373 00007f8fe044e270 00007f8fdfda0574 00007f8fe9a5c6b4 00007f8fe9a5c649 00007f8fe92d4c91 00007f8fe92e1671 00007f8fe0462c67 00007f8fe924b709 00007f8fe047f57b 00007f8fe047d81f 00007f8fe047fae2 00007f8fe30c75ab 00007f8fe30cd0c3 00007f8fe30cd5da 00007f8fe35386aa
0129 03:00:19.434488 (+259836us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0129 03:00:19.435300 (+ 812us) raft_consensus.cc:1645] Marking committed up to 6
0129 03:00:19.435395 (+ 95us) raft_consensus.cc:1697] Filling consensus response to leader.
0129 03:00:19.435486 (+ 91us) raft_consensus.cc:1691] UpdateReplicas() finished
0129 03:00:19.435650 (+ 164us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":11553,"prepare.queue_time_us":19271,"prepare.run_cpu_time_us":80,"prepare.run_wall_time_us":6224,"raft.queue_time_us":34225,"raft.run_cpu_time_us":134,"raft.run_wall_time_us":5879,"spinlock_wait_cycles":18635136,"thread_start_us":241101,"threads_started":2,"wal-append.queue_time_us":1228339}
W20260129 03:00:19.443373 21100 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.211.3:37735, user_credentials={real_user=slave}} blocked reactor thread for 107356us
I20260129 03:00:19.315105 23727 raft_consensus.cc:493] T a5a05fbcbbcf4916bbc01fd1a5b5e90d P 197b880fa39d432a9b472845876bfaf0 [term 8 FOLLOWER]: Starting leader election (detected failure of leader 1c50de89a5a845668696dcb28c3badd6)
W20260129 03:00:19.396019 20556 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.211.2:34517, user_credentials={real_user=slave}} blocked reactor thread for 69766.2us
I20260129 03:00:19.450381 20556 leader_election.cc:304] T a0fcc58d6fb6473cafbd9d34d5195269 P 1c50de89a5a845668696dcb28c3badd6 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1c50de89a5a845668696dcb28c3badd6; no voters: 197b880fa39d432a9b472845876bfaf0, 974b802141bd4f13beffef2aa03f88f9
I20260129 03:00:19.310777 23732 raft_consensus.cc:2804] T 0f20af7fbe5d4db89a6f17948d98912c P 974b802141bd4f13beffef2aa03f88f9 [term 8 FOLLOWER]: Leader election won for term 8
I20260129 03:00:19.324245 23673 consensus_queue.cc:237] T c934d31efd3f4e5195605ddc354d14e1 P 197b880fa39d432a9b472845876bfaf0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 5.7, Last appended by leader: 7, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } }
I20260129 03:00:19.401464 20587 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "724a236524cb4e7ba3428880681d6cc2" candidate_uuid: "974b802141bd4f13beffef2aa03f88f9" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "1c50de89a5a845668696dcb28c3badd6"
I20260129 03:00:19.298395 23708 raft_consensus.cc:515] T 2ac659d732314e1288849f08fe5e074e P 974b802141bd4f13beffef2aa03f88f9 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } }
I20260129 03:00:19.324151 23726 raft_consensus.cc:2749] T 18f30ebaa4174fd28c958e72d0cc7e24 P 197b880fa39d432a9b472845876bfaf0 [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
W20260129 03:00:19.400573 21102 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.211.2:34517, user_credentials={real_user=slave}} blocked reactor thread for 115819us
I20260129 03:00:19.471972 21102 leader_election.cc:304] T 45cc5f3d2688486d9b6dbe7e666dedde P 7a9daf5d42a84615a6383cfec9a10608 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 7a9daf5d42a84615a6383cfec9a10608, 974b802141bd4f13beffef2aa03f88f9; no voters: 1c50de89a5a845668696dcb28c3badd6
I20260129 03:00:19.399935 23570 leader_election.cc:290] T 75b0b6e82ddc4f92b1fe1d94f9a0b73b P 7a9daf5d42a84615a6383cfec9a10608 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 974b802141bd4f13beffef2aa03f88f9 (127.18.211.2:34517), 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3:37735)
I20260129 03:00:19.403102 23640 raft_consensus.cc:3060] T c934d31efd3f4e5195605ddc354d14e1 P 7a9daf5d42a84615a6383cfec9a10608 [term 6 FOLLOWER]: Advancing to term 7
I20260129 03:00:19.441921 20405 leader_election.cc:304] T 76f713c3331540d2a5b76164103e9eb1 P 197b880fa39d432a9b472845876bfaf0 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 197b880fa39d432a9b472845876bfaf0, 1c50de89a5a845668696dcb28c3badd6; no voters: 974b802141bd4f13beffef2aa03f88f9
I20260129 03:00:19.475131 20405 leader_election.cc:304] T 96609ac68bdf4993a12e1c23355ca013 P 197b880fa39d432a9b472845876bfaf0 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 197b880fa39d432a9b472845876bfaf0, 974b802141bd4f13beffef2aa03f88f9; no voters: 1c50de89a5a845668696dcb28c3badd6, 7a9daf5d42a84615a6383cfec9a10608
I20260129 03:00:19.462394 20480 leader_election.cc:304] T 4e420f90884f4796997b5ea41e49e411 P 974b802141bd4f13beffef2aa03f88f9 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 974b802141bd4f13beffef2aa03f88f9; no voters: 197b880fa39d432a9b472845876bfaf0, 7a9daf5d42a84615a6383cfec9a10608
I20260129 03:00:19.465922 20587 raft_consensus.cc:3060] T 724a236524cb4e7ba3428880681d6cc2 P 1c50de89a5a845668696dcb28c3badd6 [term 6 FOLLOWER]: Advancing to term 7
I20260129 03:00:19.466542 20585 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "96609ac68bdf4993a12e1c23355ca013" candidate_uuid: "974b802141bd4f13beffef2aa03f88f9" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "1c50de89a5a845668696dcb28c3badd6" is_pre_election: true
W20260129 03:00:19.402149 23723 log.cc:927] Time spent T 505580c93c694bbb8ea96fdf9b17ab46 P 974b802141bd4f13beffef2aa03f88f9: Append to log took a long time: real 0.224s user 0.001s sys 0.000s
I20260129 03:00:19.304476 23678 consensus_queue.cc:1048] T 75f45538dc704d9a8ec4ff8ad9f3ea5b P 1c50de89a5a845668696dcb28c3badd6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.100s
W20260129 03:00:19.467743 20407 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.211.2:34517, user_credentials={real_user=slave}} blocked reactor thread for 188293us
I20260129 03:00:19.432746 20589 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a1a39dcd328e4268ae41b988c66b4a75" candidate_uuid: "974b802141bd4f13beffef2aa03f88f9" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "1c50de89a5a845668696dcb28c3badd6"
I20260129 03:00:19.484225 20589 raft_consensus.cc:3060] T a1a39dcd328e4268ae41b988c66b4a75 P 1c50de89a5a845668696dcb28c3badd6 [term 5 FOLLOWER]: Advancing to term 7
I20260129 03:00:19.467095 23674 raft_consensus.cc:2804] T fcddab6bca684adf8ea032289a00f90e P 197b880fa39d432a9b472845876bfaf0 [term 10 FOLLOWER]: Leader pre-election won for term 11
W20260129 03:00:19.468345 23724 log.cc:927] Time spent T dc2f4cb752cf4d59b5f6bc689f5d8104 P 7a9daf5d42a84615a6383cfec9a10608: Append to log took a long time: real 0.073s user 0.001s sys 0.000s
I20260129 03:00:19.465634 23727 raft_consensus.cc:3060] T a5a05fbcbbcf4916bbc01fd1a5b5e90d P 197b880fa39d432a9b472845876bfaf0 [term 8 FOLLOWER]: Advancing to term 9
I20260129 03:00:19.490875 20587 raft_consensus.cc:2468] T 724a236524cb4e7ba3428880681d6cc2 P 1c50de89a5a845668696dcb28c3badd6 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 974b802141bd4f13beffef2aa03f88f9 in term 7.
I20260129 03:00:19.402544 23262 consensus_queue.cc:1048] T 75b0b6e82ddc4f92b1fe1d94f9a0b73b P 1c50de89a5a845668696dcb28c3badd6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.072s
I20260129 03:00:19.496260 20480 leader_election.cc:304] T 4f5a77e94fef4b16a2ba7b067c8b144a P 974b802141bd4f13beffef2aa03f88f9 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 974b802141bd4f13beffef2aa03f88f9; no voters: 1c50de89a5a845668696dcb28c3badd6, 7a9daf5d42a84615a6383cfec9a10608
I20260129 03:00:19.496646 20585 tablet_service.cc:2066] Received LeaderStepDown RPC: tablet_id: "dc2f4cb752cf4d59b5f6bc689f5d8104"
dest_uuid: "1c50de89a5a845668696dcb28c3badd6"
mode: GRACEFUL
new_leader_uuid: "197b880fa39d432a9b472845876bfaf0"
from {username='slave'} at 127.0.0.1:54860
I20260129 03:00:19.498787 20585 raft_consensus.cc:606] T dc2f4cb752cf4d59b5f6bc689f5d8104 P 1c50de89a5a845668696dcb28c3badd6 [term 6 LEADER]: Received request to transfer leadership to TS 197b880fa39d432a9b472845876bfaf0
W20260129 03:00:19.484122 23630 consensus_meta.cc:287] T 4f8f3a38f6e24ba4857875181f447442 P 974b802141bd4f13beffef2aa03f88f9: Time spent flushing consensus metadata: real 0.946s user 0.009s sys 0.003s
I20260129 03:00:19.497938 20587 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "75b0b6e82ddc4f92b1fe1d94f9a0b73b" candidate_uuid: "7a9daf5d42a84615a6383cfec9a10608" candidate_term: 6 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "1c50de89a5a845668696dcb28c3badd6" is_pre_election: true
I20260129 03:00:19.506156 20589 raft_consensus.cc:2468] T a1a39dcd328e4268ae41b988c66b4a75 P 1c50de89a5a845668696dcb28c3badd6 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 974b802141bd4f13beffef2aa03f88f9 in term 7.
W20260129 03:00:19.510720 20556 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.211.2:34517, user_credentials={real_user=slave}} blocked reactor thread for 60415.7us
I20260129 03:00:19.486435 23224 raft_consensus.cc:2804] T 76f713c3331540d2a5b76164103e9eb1 P 197b880fa39d432a9b472845876bfaf0 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260129 03:00:19.483426 23699 raft_consensus.cc:2749] T a0fcc58d6fb6473cafbd9d34d5195269 P 1c50de89a5a845668696dcb28c3badd6 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260129 03:00:19.490674 23570 raft_consensus.cc:2749] T a644c02b16af47679824ccd060ac10fd P 7a9daf5d42a84615a6383cfec9a10608 [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20260129 03:00:19.494602 23663 leader_election.cc:290] T 0f20af7fbe5d4db89a6f17948d98912c P 197b880fa39d432a9b472845876bfaf0 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 974b802141bd4f13beffef2aa03f88f9 (127.18.211.2:34517), 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3:37735)
I20260129 03:00:19.556620 21130 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34046 (request call id 619) took 2323 ms. Trace:
I20260129 03:00:19.556838 22989 raft_consensus.cc:2749] T 4e420f90884f4796997b5ea41e49e411 P 974b802141bd4f13beffef2aa03f88f9 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260129 03:00:19.557746 22989 raft_consensus.cc:2749] T 4f5a77e94fef4b16a2ba7b067c8b144a P 974b802141bd4f13beffef2aa03f88f9 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260129 03:00:19.557096 21130 rpcz_store.cc:276] 0129 03:00:17.233397 (+ 0us) service_pool.cc:168] Inserting onto call queue
0129 03:00:17.233519 (+ 122us) service_pool.cc:225] Handling call
0129 03:00:17.236020 (+ 2501us) raft_consensus.cc:1471] Updating replica for 1 ops
0129 03:00:17.240747 (+ 4727us) raft_consensus.cc:1534] Early marking committed up to index 0
0129 03:00:17.240791 (+ 44us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0129 03:00:18.753699 (+1512908us) spinlock_profiling.cc:243] Waited 33.8 ms on lock 0x7b2c000472a8. stack: 00007f8fe044e373 00007f8fe044e270 00007f8fdfda0574 00007f8fe9a5c6b4 00007f8fe9a5c649 00007f8fe1e2c9b1 00007f8fe1e3bfe2 00007f8fe3569851 00007f8fe352e0ba 00007f8fe352d4be 00007f8fe84e1f49 00007f8fe1a6ed34 00007f8fe1a6ecc2 00007f8fe1a6ebf1 00007f8fe1a6eb6c 00007f8fe1a6de22
0129 03:00:18.759151 (+ 5452us) log.cc:844] Serialized 26 byte log entry
0129 03:00:18.920826 (+161675us) raft_consensus.cc:1645] Marking committed up to 7
0129 03:00:19.000379 (+ 79553us) log.cc:844] Serialized 14 byte log entry
0129 03:00:19.004915 (+ 4536us) raft_consensus.cc:1697] Filling consensus response to leader.
0129 03:00:19.005007 (+ 92us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0129 03:00:19.543512 (+538505us) raft_consensus.cc:1685] finished
0129 03:00:19.556388 (+ 12876us) raft_consensus.cc:1691] UpdateReplicas() finished
0129 03:00:19.556537 (+ 149us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":81867,"prepare.run_cpu_time_us":66,"prepare.run_wall_time_us":68,"raft.queue_time_us":117579,"raft.run_cpu_time_us":9705,"raft.run_wall_time_us":471146,"spinlock_wait_cycles":77754112,"thread_start_us":159343,"threads_started":1,"wal-append.queue_time_us":267418}
I20260129 03:00:19.557036 20512 raft_consensus.cc:1275] T 7d9494f5584d4d5db8b9adf1c8fc888b P 974b802141bd4f13beffef2aa03f88f9 [term 5 FOLLOWER]: Refusing update from remote peer 197b880fa39d432a9b472845876bfaf0: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 5 index: 4. (index mismatch)
W20260129 03:00:19.546589 20479 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.211.1:42027, user_credentials={real_user=slave}} blocked reactor thread for 102144us
W20260129 03:00:19.576750 21102 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.211.2:34517, user_credentials={real_user=slave}} blocked reactor thread for 104867us
I20260129 03:00:19.515534 23742 raft_consensus.cc:2749] T 96609ac68bdf4993a12e1c23355ca013 P 197b880fa39d432a9b472845876bfaf0 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260129 03:00:19.516877 23224 raft_consensus.cc:493] T 76f713c3331540d2a5b76164103e9eb1 P 197b880fa39d432a9b472845876bfaf0 [term 6 FOLLOWER]: Starting leader election (detected failure of leader 974b802141bd4f13beffef2aa03f88f9)
I20260129 03:00:19.541747 21128 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "96609ac68bdf4993a12e1c23355ca013" candidate_uuid: "974b802141bd4f13beffef2aa03f88f9" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "7a9daf5d42a84615a6383cfec9a10608" is_pre_election: true
I20260129 03:00:19.481236 23735 raft_consensus.cc:515] T a0fcc58d6fb6473cafbd9d34d5195269 P 974b802141bd4f13beffef2aa03f88f9 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
W20260129 03:00:19.572867 23736 log.cc:927] Time spent T fcddab6bca684adf8ea032289a00f90e P 197b880fa39d432a9b472845876bfaf0: Append to log took a long time: real 0.170s user 0.001s sys 0.000s
I20260129 03:00:19.433610 23717 consensus_queue.cc:237] T 17d6431cc77a491e8464d68db4081a51 P 974b802141bd4f13beffef2aa03f88f9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 4.2, Last appended by leader: 2, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
W20260129 03:00:19.583469 20405 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.211.3:37735, user_credentials={real_user=slave}} blocked reactor thread for 108415us
I20260129 03:00:19.597064 20405 leader_election.cc:304] T cb4727386a3e48aca5d71cc6c611c197 P 197b880fa39d432a9b472845876bfaf0 [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 197b880fa39d432a9b472845876bfaf0; no voters: 1c50de89a5a845668696dcb28c3badd6, 974b802141bd4f13beffef2aa03f88f9
I20260129 03:00:19.590605 20513 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0f20af7fbe5d4db89a6f17948d98912c" candidate_uuid: "197b880fa39d432a9b472845876bfaf0" candidate_term: 9 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "974b802141bd4f13beffef2aa03f88f9" is_pre_election: true
I20260129 03:00:19.515021 23630 raft_consensus.cc:515] T 4f8f3a38f6e24ba4857875181f447442 P 974b802141bd4f13beffef2aa03f88f9 [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
W20260129 03:00:19.600574 20480 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.211.4:40439, user_credentials={real_user=slave}} blocked reactor thread for 105545us
I20260129 03:00:19.608335 20440 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7a9a69cd82724b1faed1d10aa091fc4a" candidate_uuid: "974b802141bd4f13beffef2aa03f88f9" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "197b880fa39d432a9b472845876bfaf0"
I20260129 03:00:19.608886 20437 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a609ac02fae84725ab743f95f51d33ce" candidate_uuid: "974b802141bd4f13beffef2aa03f88f9" candidate_term: 10 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "197b880fa39d432a9b472845876bfaf0" is_pre_election: true
I20260129 03:00:19.602002 20480 leader_election.cc:304] T 87d5fe48a44a408aaf03e894a67ecb2f P 974b802141bd4f13beffef2aa03f88f9 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 974b802141bd4f13beffef2aa03f88f9; no voters: 197b880fa39d432a9b472845876bfaf0, 7a9daf5d42a84615a6383cfec9a10608
I20260129 03:00:19.585793 23224 raft_consensus.cc:3060] T 76f713c3331540d2a5b76164103e9eb1 P 197b880fa39d432a9b472845876bfaf0 [term 6 FOLLOWER]: Advancing to term 7
W20260129 03:00:19.595826 23651 log.cc:927] Time spent T cb4727386a3e48aca5d71cc6c611c197 P 974b802141bd4f13beffef2aa03f88f9: Append to log took a long time: real 1.767s user 0.010s sys 0.000s
I20260129 03:00:19.611760 20511 raft_consensus.cc:1240] T a0fcc58d6fb6473cafbd9d34d5195269 P 974b802141bd4f13beffef2aa03f88f9 [term 6 FOLLOWER]: Rejecting Update request from peer 197b880fa39d432a9b472845876bfaf0 for earlier term 5. Current term is 6. Ops: []
I20260129 03:00:19.515211 23692 raft_consensus.cc:515] T 87d5fe48a44a408aaf03e894a67ecb2f P 7a9daf5d42a84615a6383cfec9a10608 [term 8 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } }
I20260129 03:00:19.585402 23691 leader_election.cc:290] T a609ac02fae84725ab743f95f51d33ce P 974b802141bd4f13beffef2aa03f88f9 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 197b880fa39d432a9b472845876bfaf0 (127.18.211.1:42027), 7a9daf5d42a84615a6383cfec9a10608 (127.18.211.4:40439)
I20260129 03:00:19.611001 20480 leader_election.cc:304] T 00e79d8f25a34676b1d866eb2068a2b2 P 974b802141bd4f13beffef2aa03f88f9 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 7a9daf5d42a84615a6383cfec9a10608, 974b802141bd4f13beffef2aa03f88f9; no voters: 197b880fa39d432a9b472845876bfaf0
W20260129 03:00:19.636181 20441 consensus_meta.cc:287] T 724a236524cb4e7ba3428880681d6cc2 P 197b880fa39d432a9b472845876bfaf0: Time spent flushing consensus metadata: real 0.558s user 0.010s sys 0.001s
I20260129 03:00:19.631973 23753 raft_consensus.cc:493] T 0678b8b468bd43cdbc3e485900844821 P 7a9daf5d42a84615a6383cfec9a10608 [term 9 FOLLOWER]: Starting pre-election (detected failure of leader 974b802141bd4f13beffef2aa03f88f9)
I20260129 03:00:19.662652 20437 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2ac659d732314e1288849f08fe5e074e" candidate_uuid: "974b802141bd4f13beffef2aa03f88f9" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "197b880fa39d432a9b472845876bfaf0" is_pre_election: true
I20260129 03:00:19.662824 23753 raft_consensus.cc:515] T 0678b8b468bd43cdbc3e485900844821 P 7a9daf5d42a84615a6383cfec9a10608 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } }
I20260129 03:00:19.621837 23663 raft_consensus.cc:2749] T cb4727386a3e48aca5d71cc6c611c197 P 197b880fa39d432a9b472845876bfaf0 [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
W20260129 03:00:19.666194 23728 log.cc:927] Time spent T febffe4a8b394368b69b1d164308fb9a P 197b880fa39d432a9b472845876bfaf0: Append to log took a long time: real 0.372s user 0.001s sys 0.000s
W20260129 03:00:19.666831 23729 log.cc:927] Time spent T e7d83d4ccf684e76acb29a0dc844e1b8 P 1c50de89a5a845668696dcb28c3badd6: Append to log took a long time: real 0.255s user 0.001s sys 0.000s
I20260129 03:00:19.667704 20510 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:57600 (request call id 2025) took 1071 ms. Trace:
I20260129 03:00:19.668111 20510 rpcz_store.cc:276] 0129 03:00:18.596352 (+ 0us) service_pool.cc:168] Inserting onto call queue
0129 03:00:18.596499 (+ 147us) service_pool.cc:225] Handling call
0129 03:00:19.667503 (+1071004us) spinlock_profiling.cc:243] Waited 197 ms on lock 0x7b54000fbc54. stack: 00007f8fe044e373 00007f8fe044e270 00007f8fdfda0574 00007f8fe9a5c6b4 00007f8fe9a5c649 00007f8fe34975a1 00007f8fe3531f7a 00007f8fe84e3647 00007f8fe1a73834 00007f8fe1a737c2 00007f8fe1a736f1 00007f8fe1a7366c 00007f8fe1a72922 00007f8fe186d34c 00007f8fe186c786 00007f8fe186c12f
0129 03:00:19.667644 (+ 141us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":452208896}
W20260129 03:00:19.670049 21102 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.211.2:34517, user_credentials={real_user=slave}} blocked reactor thread for 53371.3us
W20260129 03:00:19.639256 23746 log.cc:927] Time spent T e33a417cf9c54cf5af73a4ee3aac4d3a P 1c50de89a5a845668696dcb28c3badd6: Append to log took a long time: real 0.098s user 0.000s sys 0.000s
I20260129 03:00:19.640091 23740 raft_consensus.cc:2804] T 45cc5f3d2688486d9b6dbe7e666dedde P 7a9daf5d42a84615a6383cfec9a10608 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260129 03:00:19.640779 23733 raft_consensus.cc:493] T febffe4a8b394368b69b1d164308fb9a P 7a9daf5d42a84615a6383cfec9a10608 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 974b802141bd4f13beffef2aa03f88f9)
I20260129 03:00:19.641227 23710 raft_consensus.cc:2749] T 87d5fe48a44a408aaf03e894a67ecb2f P 974b802141bd4f13beffef2aa03f88f9 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260129 03:00:19.621433 23695 leader_election.cc:290] T 7a9a69cd82724b1faed1d10aa091fc4a P 974b802141bd4f13beffef2aa03f88f9 [CANDIDATE]: Term 8 election: Requested vote from peers 197b880fa39d432a9b472845876bfaf0 (127.18.211.1:42027), 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3:37735)
W20260129 03:00:19.648622 23738 log.cc:927] Time spent T fcddab6bca684adf8ea032289a00f90e P 1c50de89a5a845668696dcb28c3badd6: Append to log took a long time: real 0.162s user 0.001s sys 0.000s
I20260129 03:00:19.653015 23735 leader_election.cc:290] T a0fcc58d6fb6473cafbd9d34d5195269 P 974b802141bd4f13beffef2aa03f88f9 [CANDIDATE]: Term 6 election: Requested vote from peers 197b880fa39d432a9b472845876bfaf0 (127.18.211.1:42027), 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3:37735)
I20260129 03:00:19.657222 20441 raft_consensus.cc:2468] T 724a236524cb4e7ba3428880681d6cc2 P 197b880fa39d432a9b472845876bfaf0 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 974b802141bd4f13beffef2aa03f88f9 in term 7.
I20260129 03:00:19.661568 20515 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57584 (request call id 1520) took 3050 ms. Trace:
I20260129 03:00:19.681032 20509 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:57584 (request call id 1535) took 1165 ms. Trace:
I20260129 03:00:19.691529 20509 rpcz_store.cc:276] 0129 03:00:18.515926 (+ 0us) service_pool.cc:168] Inserting onto call queue
0129 03:00:18.516075 (+ 149us) service_pool.cc:225] Handling call
0129 03:00:19.674495 (+1158420us) spinlock_profiling.cc:243] Waited 132 ms on lock 0x7b54003feb54. stack: 00007f8fe044e373 00007f8fe044e270 00007f8fdfda0574 00007f8fe9a5c6b4 00007f8fe9a5c649 00007f8fe34975a1 00007f8fe3531f7a 00007f8fe84e3647 00007f8fe1a73834 00007f8fe1a737c2 00007f8fe1a736f1 00007f8fe1a7366c 00007f8fe1a72922 00007f8fe186d34c 00007f8fe186c786 00007f8fe186c12f
0129 03:00:19.680963 (+ 6468us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":302666112}
I20260129 03:00:19.644027 23730 consensus_queue.cc:1048] T 7d9494f5584d4d5db8b9adf1c8fc888b P 197b880fa39d432a9b472845876bfaf0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.031s
W20260129 03:00:19.682983 20480 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.211.4:40439, user_credentials={real_user=slave}} blocked reactor thread for 72063.8us
I20260129 03:00:19.684478 23708 leader_election.cc:290] T 2ac659d732314e1288849f08fe5e074e P 974b802141bd4f13beffef2aa03f88f9 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 197b880fa39d432a9b472845876bfaf0 (127.18.211.1:42027), 7a9daf5d42a84615a6383cfec9a10608 (127.18.211.4:40439)
I20260129 03:00:19.686061 23727 raft_consensus.cc:515] T a5a05fbcbbcf4916bbc01fd1a5b5e90d P 197b880fa39d432a9b472845876bfaf0 [term 9 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
W20260129 03:00:19.693184 20405 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.211.3:37735, user_credentials={real_user=slave}} blocked reactor thread for 96245.6us
I20260129 03:00:19.694041 20479 leader_election.cc:304] T 724a236524cb4e7ba3428880681d6cc2 P 974b802141bd4f13beffef2aa03f88f9 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 197b880fa39d432a9b472845876bfaf0, 974b802141bd4f13beffef2aa03f88f9; no voters:
I20260129 03:00:19.706445 20405 leader_election.cc:304] T 36ff33be666a4c82954731691dbcf327 P 197b880fa39d432a9b472845876bfaf0 [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 197b880fa39d432a9b472845876bfaf0; no voters: 1c50de89a5a845668696dcb28c3badd6, 974b802141bd4f13beffef2aa03f88f9
I20260129 03:00:19.695497 20437 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a0fcc58d6fb6473cafbd9d34d5195269" candidate_uuid: "974b802141bd4f13beffef2aa03f88f9" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "197b880fa39d432a9b472845876bfaf0"
I20260129 03:00:19.697053 20581 raft_consensus.cc:1217] T c2633a6fc34c44579de348d208b0c01e P 1c50de89a5a845668696dcb28c3badd6 [term 6 LEADER]: Deduplicated request from leader. Original: 2.2->[5.3-5.3] Dedup: 5.3->[]
I20260129 03:00:19.707196 20581 raft_consensus.cc:1240] T c2633a6fc34c44579de348d208b0c01e P 1c50de89a5a845668696dcb28c3badd6 [term 6 LEADER]: Rejecting Update request from peer 974b802141bd4f13beffef2aa03f88f9 for earlier term 5. Current term is 6. Ops: [5.3-5.3]
I20260129 03:00:19.691088 20515 rpcz_store.cc:276] 0129 03:00:16.611375 (+ 0us) service_pool.cc:168] Inserting onto call queue
0129 03:00:16.614425 (+ 3050us) service_pool.cc:225] Handling call
0129 03:00:16.614706 (+ 281us) raft_consensus.cc:1471] Updating replica for 1 ops
0129 03:00:16.615085 (+ 379us) raft_consensus.cc:1534] Early marking committed up to index 0
0129 03:00:16.615109 (+ 24us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0129 03:00:16.615587 (+ 478us) log.cc:844] Serialized 26 byte log entry
0129 03:00:17.083104 (+467517us) raft_consensus.cc:1645] Marking committed up to 2
0129 03:00:17.803744 (+720640us) log.cc:844] Serialized 14 byte log entry
0129 03:00:17.862156 (+ 58412us) raft_consensus.cc:1697] Filling consensus response to leader.
0129 03:00:17.862265 (+ 109us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0129 03:00:17.916902 (+ 54637us) spinlock_profiling.cc:243] Waited 17.6 ms on lock 0x7b18000cf460. stack: 00007f8fe044e373 00007f8fe044e270 00007f8fdfda0574 00007f8fe9a5c6b4 00007f8fe9a5c649 00007f8fe92d4c91 00007f8fe92e1671 00007f8fe0480402 00007f8fe04826a1 00007f8fe0482659 00007f8fe04825e9 00007f8fe04825b1 00007f8fe04818ad 0000000000379254 0000000000379099 00007f8fe046369a
0129 03:00:19.402258 (+1485356us) spinlock_profiling.cc:243] Waited 10.8 ms on lock 0x7b440002304c. stack: 00007f8fe044e373 00007f8fe044e270 00007f8fdfda0574 00007f8fe9a5c6b4 00007f8fe9a5c649 00007f8fe02c08b1 00007f8fe038f005 00007f8fe038fa22 00007f8fe30d957b 00007f8fe30d4e4f 00007f8fe30c8c0e 00007f8fe30c7f56 00007f8fe30c7b78 00007f8fe30d15c1 00007f8fe30d1579 00007f8fe30d1509
0129 03:00:19.661345 (+259087us) raft_consensus.cc:1685] finished
0129 03:00:19.661365 (+ 20us) raft_consensus.cc:1691] UpdateReplicas() finished
0129 03:00:19.661491 (+ 126us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":8888,"prepare.queue_time_us":189305,"prepare.run_cpu_time_us":43,"prepare.run_wall_time_us":45,"raft.queue_time_us":397949,"raft.run_cpu_time_us":177,"raft.run_wall_time_us":178,"spinlock_wait_cycles":65386112,"thread_start_us":702432,"threads_started":3,"wal-append.queue_time_us":613476}
W20260129 03:00:19.709177 23631 consensus_meta.cc:287] T cedce331e1584ac8b636c9b8a72b2ba0 P 974b802141bd4f13beffef2aa03f88f9: Time spent flushing consensus metadata: real 0.658s user 0.011s sys 0.000s
I20260129 03:00:19.709622 23631 raft_consensus.cc:515] T cedce331e1584ac8b636c9b8a72b2ba0 P 974b802141bd4f13beffef2aa03f88f9 [term 9 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 03:00:19.712479 23630 leader_election.cc:290] T 4f8f3a38f6e24ba4857875181f447442 P 974b802141bd4f13beffef2aa03f88f9 [CANDIDATE]: Term 8 election: Requested vote from peers 197b880fa39d432a9b472845876bfaf0 (127.18.211.1:42027), 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3:37735)
I20260129 03:00:19.681684 23570 consensus_queue.cc:1048] T cb0b3f259ebc4310927338a8740e87b8 P 7a9daf5d42a84615a6383cfec9a10608 [LEADER]: Connected to new peer: Peer: permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.031s
W20260129 03:00:19.716079 20481 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260129 03:00:19.696007 20480 leader_election.cc:304] T 75f45538dc704d9a8ec4ff8ad9f3ea5b P 974b802141bd4f13beffef2aa03f88f9 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 974b802141bd4f13beffef2aa03f88f9; no voters: 1c50de89a5a845668696dcb28c3badd6, 7a9daf5d42a84615a6383cfec9a10608
I20260129 03:00:19.675359 23740 raft_consensus.cc:493] T 45cc5f3d2688486d9b6dbe7e666dedde P 7a9daf5d42a84615a6383cfec9a10608 [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260129 03:00:19.688333 23646 raft_consensus.cc:2804] T 00e79d8f25a34676b1d866eb2068a2b2 P 974b802141bd4f13beffef2aa03f88f9 [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260129 03:00:19.711663 23631 leader_election.cc:290] T cedce331e1584ac8b636c9b8a72b2ba0 P 974b802141bd4f13beffef2aa03f88f9 [CANDIDATE]: Term 9 election: Requested vote from peers 197b880fa39d432a9b472845876bfaf0 (127.18.211.1:42027), 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3:37735)
I20260129 03:00:19.727015 20405 leader_election.cc:304] T 42d27a28d89b409e8807ec0efd97af69 P 197b880fa39d432a9b472845876bfaf0 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 197b880fa39d432a9b472845876bfaf0; no voters: 1c50de89a5a845668696dcb28c3badd6, 7a9daf5d42a84615a6383cfec9a10608
I20260129 03:00:19.730029 23744 raft_consensus.cc:2749] T 36ff33be666a4c82954731691dbcf327 P 197b880fa39d432a9b472845876bfaf0 [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20260129 03:00:19.735488 20405 leader_election.cc:304] T febffe4a8b394368b69b1d164308fb9a P 197b880fa39d432a9b472845876bfaf0 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 197b880fa39d432a9b472845876bfaf0; no voters: 7a9daf5d42a84615a6383cfec9a10608, 974b802141bd4f13beffef2aa03f88f9
I20260129 03:00:19.735605 21099 leader_election.cc:304] T a609ac02fae84725ab743f95f51d33ce P 7a9daf5d42a84615a6383cfec9a10608 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7a9daf5d42a84615a6383cfec9a10608; no voters: 197b880fa39d432a9b472845876bfaf0, 974b802141bd4f13beffef2aa03f88f9
I20260129 03:00:19.737120 21099 leader_election.cc:304] T fd8311b779af4e1fa1402b2a02a602f2 P 7a9daf5d42a84615a6383cfec9a10608 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7a9daf5d42a84615a6383cfec9a10608; no voters: 197b880fa39d432a9b472845876bfaf0, 1c50de89a5a845668696dcb28c3badd6
I20260129 03:00:19.723721 23740 raft_consensus.cc:3060] T 45cc5f3d2688486d9b6dbe7e666dedde P 7a9daf5d42a84615a6383cfec9a10608 [term 7 FOLLOWER]: Advancing to term 8
I20260129 03:00:19.738791 20480 leader_election.cc:304] T 96609ac68bdf4993a12e1c23355ca013 P 974b802141bd4f13beffef2aa03f88f9 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 974b802141bd4f13beffef2aa03f88f9; no voters: 197b880fa39d432a9b472845876bfaf0, 7a9daf5d42a84615a6383cfec9a10608
I20260129 03:00:19.740779 23744 raft_consensus.cc:2749] T febffe4a8b394368b69b1d164308fb9a P 197b880fa39d432a9b472845876bfaf0 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260129 03:00:19.755313 23710 raft_consensus.cc:2749] T 96609ac68bdf4993a12e1c23355ca013 P 974b802141bd4f13beffef2aa03f88f9 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260129 03:00:19.755725 23531 raft_consensus.cc:515] T 45cc5f3d2688486d9b6dbe7e666dedde P 1c50de89a5a845668696dcb28c3badd6 [term 8 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } }
I20260129 03:00:19.740988 23756 raft_consensus.cc:493] T 75f45538dc704d9a8ec4ff8ad9f3ea5b P 7a9daf5d42a84615a6383cfec9a10608 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260129 03:00:19.757411 23756 raft_consensus.cc:515] T 75f45538dc704d9a8ec4ff8ad9f3ea5b P 7a9daf5d42a84615a6383cfec9a10608 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } }
I20260129 03:00:19.741407 20437 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4f8f3a38f6e24ba4857875181f447442" candidate_uuid: "974b802141bd4f13beffef2aa03f88f9" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "197b880fa39d432a9b472845876bfaf0"
I20260129 03:00:19.758759 20437 raft_consensus.cc:3060] T 4f8f3a38f6e24ba4857875181f447442 P 197b880fa39d432a9b472845876bfaf0 [term 7 FOLLOWER]: Advancing to term 8
I20260129 03:00:19.687795 23733 raft_consensus.cc:515] T febffe4a8b394368b69b1d164308fb9a P 7a9daf5d42a84615a6383cfec9a10608 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } }
I20260129 03:00:19.759955 23756 leader_election.cc:290] T 75f45538dc704d9a8ec4ff8ad9f3ea5b P 7a9daf5d42a84615a6383cfec9a10608 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 974b802141bd4f13beffef2aa03f88f9 (127.18.211.2:34517), 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3:37735)
I20260129 03:00:19.741132 20439 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "cedce331e1584ac8b636c9b8a72b2ba0" candidate_uuid: "974b802141bd4f13beffef2aa03f88f9" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "197b880fa39d432a9b472845876bfaf0"
W20260129 03:00:19.741150 23403 consensus_meta.cc:287] T a8479d1fe0bd41e5946186e3a6ca8c7a P 197b880fa39d432a9b472845876bfaf0: Time spent flushing consensus metadata: real 0.578s user 0.002s sys 0.009s
I20260129 03:00:19.761130 23403 raft_consensus.cc:515] T a8479d1fe0bd41e5946186e3a6ca8c7a P 197b880fa39d432a9b472845876bfaf0 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 03:00:19.761482 23727 leader_election.cc:290] T a5a05fbcbbcf4916bbc01fd1a5b5e90d P 197b880fa39d432a9b472845876bfaf0 [CANDIDATE]: Term 9 election: Requested vote from peers 974b802141bd4f13beffef2aa03f88f9 (127.18.211.2:34517), 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3:37735)
I20260129 03:00:19.762121 20433 raft_consensus.cc:1240] T a8479d1fe0bd41e5946186e3a6ca8c7a P 197b880fa39d432a9b472845876bfaf0 [term 6 FOLLOWER]: Rejecting Update request from peer 974b802141bd4f13beffef2aa03f88f9 for earlier term 5. Current term is 6. Ops: []
I20260129 03:00:19.749531 23692 leader_election.cc:290] T 87d5fe48a44a408aaf03e894a67ecb2f P 7a9daf5d42a84615a6383cfec9a10608 [CANDIDATE]: Term 8 election: Requested vote from peers 197b880fa39d432a9b472845876bfaf0 (127.18.211.1:42027), 974b802141bd4f13beffef2aa03f88f9 (127.18.211.2:34517)
W20260129 03:00:19.748441 23709 consensus_meta.cc:287] T 926a250fa5834e88b96ccecbefbf5086 P 974b802141bd4f13beffef2aa03f88f9: Time spent flushing consensus metadata: real 0.975s user 0.011s sys 0.008s
I20260129 03:00:19.751407 23743 raft_consensus.cc:2749] T 42d27a28d89b409e8807ec0efd97af69 P 197b880fa39d432a9b472845876bfaf0 [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
W20260129 03:00:19.754293 23466 consensus_meta.cc:287] T e7d83d4ccf684e76acb29a0dc844e1b8 P 197b880fa39d432a9b472845876bfaf0: Time spent flushing consensus metadata: real 0.846s user 0.017s sys 0.001s
I20260129 03:00:19.757786 23531 leader_election.cc:290] T 45cc5f3d2688486d9b6dbe7e666dedde P 1c50de89a5a845668696dcb28c3badd6 [CANDIDATE]: Term 8 election: Requested vote from peers 974b802141bd4f13beffef2aa03f88f9 (127.18.211.2:34517), 7a9daf5d42a84615a6383cfec9a10608 (127.18.211.4:40439)
W20260129 03:00:19.759259 23427 consensus_meta.cc:287] T 0340229d521b432580ae926f6062e3f7 P 974b802141bd4f13beffef2aa03f88f9: Time spent flushing consensus metadata: real 0.608s user 0.008s sys 0.000s
I20260129 03:00:19.745430 23706 raft_consensus.cc:2804] T 724a236524cb4e7ba3428880681d6cc2 P 974b802141bd4f13beffef2aa03f88f9 [term 7 FOLLOWER]: Leader election won for term 7
I20260129 03:00:19.737195 23570 raft_consensus.cc:2749] T a609ac02fae84725ab743f95f51d33ce P 7a9daf5d42a84615a6383cfec9a10608 [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20260129 03:00:19.771340 20510 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a5a05fbcbbcf4916bbc01fd1a5b5e90d" candidate_uuid: "197b880fa39d432a9b472845876bfaf0" candidate_term: 9 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "974b802141bd4f13beffef2aa03f88f9"
I20260129 03:00:19.750968 23750 raft_consensus.cc:3060] T fd8311b779af4e1fa1402b2a02a602f2 P 7a9daf5d42a84615a6383cfec9a10608 [term 5 FOLLOWER]: Advancing to term 6
I20260129 03:00:19.775594 20510 raft_consensus.cc:3060] T a5a05fbcbbcf4916bbc01fd1a5b5e90d P 974b802141bd4f13beffef2aa03f88f9 [term 8 FOLLOWER]: Advancing to term 9
I20260129 03:00:19.767351 20440 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "87d5fe48a44a408aaf03e894a67ecb2f" candidate_uuid: "7a9daf5d42a84615a6383cfec9a10608" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "197b880fa39d432a9b472845876bfaf0"
I20260129 03:00:19.742022 20405 leader_election.cc:304] T fd8311b779af4e1fa1402b2a02a602f2 P 197b880fa39d432a9b472845876bfaf0 [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 197b880fa39d432a9b472845876bfaf0; no voters: 1c50de89a5a845668696dcb28c3badd6, 7a9daf5d42a84615a6383cfec9a10608
I20260129 03:00:19.728061 23646 raft_consensus.cc:493] T 00e79d8f25a34676b1d866eb2068a2b2 P 974b802141bd4f13beffef2aa03f88f9 [term 8 FOLLOWER]: Starting leader election (detected failure of leader 197b880fa39d432a9b472845876bfaf0)
I20260129 03:00:19.771095 20515 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "75b0b6e82ddc4f92b1fe1d94f9a0b73b" candidate_uuid: "7a9daf5d42a84615a6383cfec9a10608" candidate_term: 6 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "974b802141bd4f13beffef2aa03f88f9" is_pre_election: true
I20260129 03:00:19.772699 21135 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a609ac02fae84725ab743f95f51d33ce" candidate_uuid: "974b802141bd4f13beffef2aa03f88f9" candidate_term: 10 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "7a9daf5d42a84615a6383cfec9a10608" is_pre_election: true
I20260129 03:00:19.773058 20509 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "75f45538dc704d9a8ec4ff8ad9f3ea5b" candidate_uuid: "7a9daf5d42a84615a6383cfec9a10608" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "974b802141bd4f13beffef2aa03f88f9" is_pre_election: true
I20260129 03:00:19.730298 23224 raft_consensus.cc:515] T 76f713c3331540d2a5b76164103e9eb1 P 197b880fa39d432a9b472845876bfaf0 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 03:00:19.774008 21133 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2ac659d732314e1288849f08fe5e074e" candidate_uuid: "974b802141bd4f13beffef2aa03f88f9" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "7a9daf5d42a84615a6383cfec9a10608" is_pre_election: true
I20260129 03:00:19.765511 23695 raft_consensus.cc:3060] T 75f45538dc704d9a8ec4ff8ad9f3ea5b P 974b802141bd4f13beffef2aa03f88f9 [term 4 FOLLOWER]: Advancing to term 5
W20260129 03:00:19.771042 23694 consensus_meta.cc:287] T 75b0b6e82ddc4f92b1fe1d94f9a0b73b P 974b802141bd4f13beffef2aa03f88f9: Time spent flushing consensus metadata: real 1.138s user 0.006s sys 0.007s
I20260129 03:00:19.780995 23706 raft_consensus.cc:697] T 724a236524cb4e7ba3428880681d6cc2 P 974b802141bd4f13beffef2aa03f88f9 [term 7 LEADER]: Becoming Leader. State: Replica: 974b802141bd4f13beffef2aa03f88f9, State: Running, Role: LEADER
I20260129 03:00:19.785920 23706 consensus_queue.cc:237] T 724a236524cb4e7ba3428880681d6cc2 P 974b802141bd4f13beffef2aa03f88f9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 5.4, Last appended by leader: 4, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 03:00:19.772317 23709 raft_consensus.cc:515] T 926a250fa5834e88b96ccecbefbf5086 P 974b802141bd4f13beffef2aa03f88f9 [term 9 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 03:00:19.779213 23304 leader_election.cc:290] T 0340229d521b432580ae926f6062e3f7 P 1c50de89a5a845668696dcb28c3badd6 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 197b880fa39d432a9b472845876bfaf0 (127.18.211.1:42027), 974b802141bd4f13beffef2aa03f88f9 (127.18.211.2:34517)
I20260129 03:00:19.789636 23709 leader_election.cc:290] T 926a250fa5834e88b96ccecbefbf5086 P 974b802141bd4f13beffef2aa03f88f9 [CANDIDATE]: Term 9 election: Requested vote from peers 197b880fa39d432a9b472845876bfaf0 (127.18.211.1:42027), 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3:37735)
W20260129 03:00:19.778441 23555 consensus_meta.cc:287] T 368953c9102643b2a21f99af36b51905 P 974b802141bd4f13beffef2aa03f88f9: Time spent flushing consensus metadata: real 0.901s user 0.007s sys 0.004s
I20260129 03:00:19.790325 23555 raft_consensus.cc:515] T 368953c9102643b2a21f99af36b51905 P 974b802141bd4f13beffef2aa03f88f9 [term 10 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 03:00:19.779392 23646 raft_consensus.cc:3060] T 00e79d8f25a34676b1d866eb2068a2b2 P 974b802141bd4f13beffef2aa03f88f9 [term 8 FOLLOWER]: Advancing to term 9
I20260129 03:00:19.792382 23555 leader_election.cc:290] T 368953c9102643b2a21f99af36b51905 P 974b802141bd4f13beffef2aa03f88f9 [CANDIDATE]: Term 10 election: Requested vote from peers 197b880fa39d432a9b472845876bfaf0 (127.18.211.1:42027), 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3:37735)
I20260129 03:00:19.793814 20434 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "368953c9102643b2a21f99af36b51905" candidate_uuid: "974b802141bd4f13beffef2aa03f88f9" candidate_term: 10 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "197b880fa39d432a9b472845876bfaf0"
I20260129 03:00:19.794445 20434 raft_consensus.cc:3060] T 368953c9102643b2a21f99af36b51905 P 197b880fa39d432a9b472845876bfaf0 [term 9 FOLLOWER]: Advancing to term 10
I20260129 03:00:19.778709 23427 raft_consensus.cc:515] T 0340229d521b432580ae926f6062e3f7 P 974b802141bd4f13beffef2aa03f88f9 [term 9 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 03:00:19.796257 20440 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "926a250fa5834e88b96ccecbefbf5086" candidate_uuid: "974b802141bd4f13beffef2aa03f88f9" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "197b880fa39d432a9b472845876bfaf0"
W20260129 03:00:19.799602 23398 consensus_meta.cc:287] T a644c02b16af47679824ccd060ac10fd P 1c50de89a5a845668696dcb28c3badd6: Time spent flushing consensus metadata: real 0.636s user 0.010s sys 0.000s
I20260129 03:00:19.799690 20588 raft_consensus.cc:2468] T cb4727386a3e48aca5d71cc6c611c197 P 1c50de89a5a845668696dcb28c3badd6 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 974b802141bd4f13beffef2aa03f88f9 in term 6.
I20260129 03:00:19.795140 20437 raft_consensus.cc:2468] T 4f8f3a38f6e24ba4857875181f447442 P 197b880fa39d432a9b472845876bfaf0 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 974b802141bd4f13beffef2aa03f88f9 in term 8.
I20260129 03:00:19.803606 20440 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0340229d521b432580ae926f6062e3f7" candidate_uuid: "974b802141bd4f13beffef2aa03f88f9" candidate_term: 9 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "197b880fa39d432a9b472845876bfaf0"
I20260129 03:00:19.777295 23686 raft_consensus.cc:515] T 17d6431cc77a491e8464d68db4081a51 P 197b880fa39d432a9b472845876bfaf0 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 03:00:19.818023 20440 raft_consensus.cc:3060] T 0340229d521b432580ae926f6062e3f7 P 197b880fa39d432a9b472845876bfaf0 [term 8 FOLLOWER]: Advancing to term 9
I20260129 03:00:19.804167 23427 leader_election.cc:290] T 0340229d521b432580ae926f6062e3f7 P 974b802141bd4f13beffef2aa03f88f9 [CANDIDATE]: Term 9 election: Requested vote from peers 197b880fa39d432a9b472845876bfaf0 (127.18.211.1:42027), 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3:37735)
I20260129 03:00:19.805989 23333 raft_consensus.cc:515] T 8055232cf0004d40a85f3bf983fe00a9 P 974b802141bd4f13beffef2aa03f88f9 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 03:00:19.812695 21135 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1299723a4476420ab02fd8bfd7aa0645" candidate_uuid: "197b880fa39d432a9b472845876bfaf0" candidate_term: 11 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "7a9daf5d42a84615a6383cfec9a10608" is_pre_election: true
I20260129 03:00:19.781236 20480 leader_election.cc:304] T a609ac02fae84725ab743f95f51d33ce P 974b802141bd4f13beffef2aa03f88f9 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 974b802141bd4f13beffef2aa03f88f9; no voters: 197b880fa39d432a9b472845876bfaf0, 7a9daf5d42a84615a6383cfec9a10608
I20260129 03:00:19.807193 23694 raft_consensus.cc:2749] T 75b0b6e82ddc4f92b1fe1d94f9a0b73b P 974b802141bd4f13beffef2aa03f88f9 [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260129 03:00:19.814945 21129 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34046 (request call id 618) took 2583 ms. Trace:
I20260129 03:00:19.819643 20479 leader_election.cc:304] T 4f8f3a38f6e24ba4857875181f447442 P 974b802141bd4f13beffef2aa03f88f9 [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 197b880fa39d432a9b472845876bfaf0, 974b802141bd4f13beffef2aa03f88f9; no voters:
I20260129 03:00:19.780798 23764 raft_consensus.cc:2749] T fd8311b779af4e1fa1402b2a02a602f2 P 197b880fa39d432a9b472845876bfaf0 [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20260129 03:00:19.840107 21129 rpcz_store.cc:276] 0129 03:00:17.231598 (+ 0us) service_pool.cc:168] Inserting onto call queue
0129 03:00:17.241757 (+ 10159us) service_pool.cc:225] Handling call
0129 03:00:19.173788 (+1932031us) raft_consensus.cc:1471] Updating replica for 0 ops
0129 03:00:19.275118 (+101330us) raft_consensus.cc:1697] Filling consensus response to leader.
0129 03:00:19.814761 (+539643us) spinlock_profiling.cc:243] Waited 1.36 s on lock 0x7b54004c95d0. stack: 00007f8fe044e373 00007f8fe044e270 00007f8fdfda0574 00007f8fe9a5c6b4 00007f8fe9a5c649 00007f8fe34975a1 00007f8fe352d52d 00007f8fe84e1f49 00007f8fe1a6ed34 00007f8fe1a6ecc2 00007f8fe1a6ebf1 00007f8fe1a6eb6c 00007f8fe1a6de22 00007f8fe186d34c 00007f8fe186c786 00007f8fe186c12f
0129 03:00:19.814890 (+ 129us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":3131523328}
W20260129 03:00:19.842352 20405 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.211.3:37735, user_credentials={real_user=slave}} blocked reactor thread for 58748.7us
I20260129 03:00:19.822024 23740 raft_consensus.cc:515] T 45cc5f3d2688486d9b6dbe7e666dedde P 7a9daf5d42a84615a6383cfec9a10608 [term 8 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } }
I20260129 03:00:19.818321 23765 raft_consensus.cc:493] T 2ac659d732314e1288849f08fe5e074e P 7a9daf5d42a84615a6383cfec9a10608 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 197b880fa39d432a9b472845876bfaf0)
I20260129 03:00:19.847959 23693 raft_consensus.cc:2804] T 4f8f3a38f6e24ba4857875181f447442 P 974b802141bd4f13beffef2aa03f88f9 [term 8 FOLLOWER]: Leader election won for term 8
I20260129 03:00:19.848503 23693 raft_consensus.cc:697] T 4f8f3a38f6e24ba4857875181f447442 P 974b802141bd4f13beffef2aa03f88f9 [term 8 LEADER]: Becoming Leader. State: Replica: 974b802141bd4f13beffef2aa03f88f9, State: Running, Role: LEADER
I20260129 03:00:19.774564 23466 raft_consensus.cc:515] T e7d83d4ccf684e76acb29a0dc844e1b8 P 197b880fa39d432a9b472845876bfaf0 [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 03:00:19.826668 23703 ts_tablet_manager.cc:1929] T 1299723a4476420ab02fd8bfd7aa0645 P 974b802141bd4f13beffef2aa03f88f9: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.4
I20260129 03:00:19.830822 20510 raft_consensus.cc:2468] T a5a05fbcbbcf4916bbc01fd1a5b5e90d P 974b802141bd4f13beffef2aa03f88f9 [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 197b880fa39d432a9b472845876bfaf0 in term 9.
I20260129 03:00:19.833961 20434 raft_consensus.cc:2468] T 368953c9102643b2a21f99af36b51905 P 197b880fa39d432a9b472845876bfaf0 [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 974b802141bd4f13beffef2aa03f88f9 in term 10.
I20260129 03:00:19.855145 20407 leader_election.cc:304] T a5a05fbcbbcf4916bbc01fd1a5b5e90d P 197b880fa39d432a9b472845876bfaf0 [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 197b880fa39d432a9b472845876bfaf0, 974b802141bd4f13beffef2aa03f88f9; no voters:
W20260129 03:00:19.845990 20480 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.211.4:40439, user_credentials={real_user=slave}} blocked reactor thread for 64845.3us
W20260129 03:00:19.825419 23707 consensus_meta.cc:287] T 3906b080a5b34a74b8b64128635e8997 P 974b802141bd4f13beffef2aa03f88f9: Time spent flushing consensus metadata: real 0.950s user 0.009s sys 0.002s
I20260129 03:00:19.848325 23765 raft_consensus.cc:515] T 2ac659d732314e1288849f08fe5e074e P 7a9daf5d42a84615a6383cfec9a10608 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } }
I20260129 03:00:19.856045 20479 leader_election.cc:304] T 368953c9102643b2a21f99af36b51905 P 974b802141bd4f13beffef2aa03f88f9 [CANDIDATE]: Term 10 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 197b880fa39d432a9b472845876bfaf0, 974b802141bd4f13beffef2aa03f88f9; no voters:
I20260129 03:00:19.855123 20515 raft_consensus.cc:2468] T 75b0b6e82ddc4f92b1fe1d94f9a0b73b P 974b802141bd4f13beffef2aa03f88f9 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7a9daf5d42a84615a6383cfec9a10608 in term 5.
I20260129 03:00:19.857832 20510 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "87d5fe48a44a408aaf03e894a67ecb2f" candidate_uuid: "7a9daf5d42a84615a6383cfec9a10608" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "974b802141bd4f13beffef2aa03f88f9"
I20260129 03:00:19.855240 20440 raft_consensus.cc:2468] T 0340229d521b432580ae926f6062e3f7 P 197b880fa39d432a9b472845876bfaf0 [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 974b802141bd4f13beffef2aa03f88f9 in term 9.
I20260129 03:00:19.861948 23693 consensus_queue.cc:237] T 4f8f3a38f6e24ba4857875181f447442 P 974b802141bd4f13beffef2aa03f88f9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 6.4, Last appended by leader: 4, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 03:00:19.869560 20507 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:57584 (request call id 1536) took 1353 ms. Trace:
W20260129 03:00:19.862255 23737 log.cc:927] Time spent T 76f713c3331540d2a5b76164103e9eb1 P 1c50de89a5a845668696dcb28c3badd6: Append to log took a long time: real 0.318s user 0.000s sys 0.002s
I20260129 03:00:19.869999 20507 rpcz_store.cc:276] 0129 03:00:18.516248 (+ 0us) service_pool.cc:168] Inserting onto call queue
0129 03:00:18.516947 (+ 699us) service_pool.cc:225] Handling call
0129 03:00:19.869353 (+1352406us) spinlock_profiling.cc:243] Waited 390 ms on lock 0x7b5400166f54. stack: 00007f8fe044e373 00007f8fe044e270 00007f8fdfda0574 00007f8fe9a5c6b4 00007f8fe9a5c649 00007f8fe34975a1 00007f8fe3531f7a 00007f8fe84e3647 00007f8fe1a73834 00007f8fe1a737c2 00007f8fe1a736f1 00007f8fe1a7366c 00007f8fe1a72922 00007f8fe186d34c 00007f8fe186c786 00007f8fe186c12f
0129 03:00:19.869496 (+ 143us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":897709440}
I20260129 03:00:19.862519 23694 raft_consensus.cc:2804] T 368953c9102643b2a21f99af36b51905 P 974b802141bd4f13beffef2aa03f88f9 [term 10 FOLLOWER]: Leader election won for term 10
I20260129 03:00:19.871373 23694 raft_consensus.cc:697] T 368953c9102643b2a21f99af36b51905 P 974b802141bd4f13beffef2aa03f88f9 [term 10 LEADER]: Becoming Leader. State: Replica: 974b802141bd4f13beffef2aa03f88f9, State: Running, Role: LEADER
I20260129 03:00:19.863046 23765 leader_election.cc:290] T 2ac659d732314e1288849f08fe5e074e P 7a9daf5d42a84615a6383cfec9a10608 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 197b880fa39d432a9b472845876bfaf0 (127.18.211.1:42027), 974b802141bd4f13beffef2aa03f88f9 (127.18.211.2:34517)
I20260129 03:00:19.864017 23333 leader_election.cc:290] T 8055232cf0004d40a85f3bf983fe00a9 P 974b802141bd4f13beffef2aa03f88f9 [CANDIDATE]: Term 7 election: Requested vote from peers 197b880fa39d432a9b472845876bfaf0 (127.18.211.1:42027), 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3:37735)
I20260129 03:00:19.860777 23707 raft_consensus.cc:515] T 3906b080a5b34a74b8b64128635e8997 P 974b802141bd4f13beffef2aa03f88f9 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 03:00:19.872481 23694 consensus_queue.cc:237] T 368953c9102643b2a21f99af36b51905 P 974b802141bd4f13beffef2aa03f88f9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 8.5, Last appended by leader: 5, Current term: 10, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 03:00:19.873095 23224 leader_election.cc:290] T 76f713c3331540d2a5b76164103e9eb1 P 197b880fa39d432a9b472845876bfaf0 [CANDIDATE]: Term 7 election: Requested vote from peers 974b802141bd4f13beffef2aa03f88f9 (127.18.211.2:34517), 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3:37735)
I20260129 03:00:19.873632 23707 leader_election.cc:290] T 3906b080a5b34a74b8b64128635e8997 P 974b802141bd4f13beffef2aa03f88f9 [CANDIDATE]: Term 6 election: Requested vote from peers 197b880fa39d432a9b472845876bfaf0 (127.18.211.1:42027), 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3:37735)
I20260129 03:00:19.864313 23686 leader_election.cc:290] T 17d6431cc77a491e8464d68db4081a51 P 197b880fa39d432a9b472845876bfaf0 [CANDIDATE]: Term 7 election: Requested vote from peers 974b802141bd4f13beffef2aa03f88f9 (127.18.211.2:34517), 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3:37735)
I20260129 03:00:19.864809 20439 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2ac659d732314e1288849f08fe5e074e" candidate_uuid: "7a9daf5d42a84615a6383cfec9a10608" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "197b880fa39d432a9b472845876bfaf0" is_pre_election: true
I20260129 03:00:19.865192 23764 raft_consensus.cc:2804] T a5a05fbcbbcf4916bbc01fd1a5b5e90d P 197b880fa39d432a9b472845876bfaf0 [term 9 FOLLOWER]: Leader election won for term 9
I20260129 03:00:19.865816 23740 leader_election.cc:290] T 45cc5f3d2688486d9b6dbe7e666dedde P 7a9daf5d42a84615a6383cfec9a10608 [CANDIDATE]: Term 8 election: Requested vote from peers 974b802141bd4f13beffef2aa03f88f9 (127.18.211.2:34517), 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3:37735)
I20260129 03:00:19.870687 20510 raft_consensus.cc:1275] T add8d4923d1a489e86882c3a25692b95 P 974b802141bd4f13beffef2aa03f88f9 [term 9 FOLLOWER]: Refusing update from remote peer 7a9daf5d42a84615a6383cfec9a10608: Log matching property violated. Preceding OpId in replica: term: 7 index: 6. Preceding OpId from leader: term: 9 index: 7. (index mismatch)
I20260129 03:00:19.876417 20510 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2ac659d732314e1288849f08fe5e074e" candidate_uuid: "7a9daf5d42a84615a6383cfec9a10608" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "974b802141bd4f13beffef2aa03f88f9" is_pre_election: true
I20260129 03:00:19.876883 23466 leader_election.cc:290] T e7d83d4ccf684e76acb29a0dc844e1b8 P 197b880fa39d432a9b472845876bfaf0 [CANDIDATE]: Term 8 election: Requested vote from peers 974b802141bd4f13beffef2aa03f88f9 (127.18.211.2:34517), 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3:37735)
I20260129 03:00:19.852298 23755 raft_consensus.cc:2749] T a609ac02fae84725ab743f95f51d33ce P 974b802141bd4f13beffef2aa03f88f9 [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
W20260129 03:00:19.880823 20582 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:35754 (request call id 1896) took 1494 ms (client timeout 1600 ms). Trace:
W20260129 03:00:19.881270 20582 rpcz_store.cc:269] 0129 03:00:18.386485 (+ 0us) service_pool.cc:168] Inserting onto call queue
0129 03:00:18.409911 (+ 23426us) service_pool.cc:225] Handling call
0129 03:00:19.880649 (+1470738us) spinlock_profiling.cc:243] Waited 874 ms on lock 0x7b540009bfd4. stack: 00007f8fe044e373 00007f8fe044e270 00007f8fdfda0574 00007f8fe9a5c6b4 00007f8fe9a5c649 00007f8fe34975a1 00007f8fe3531f7a 00007f8fe84e3647 00007f8fe1a73834 00007f8fe1a737c2 00007f8fe1a736f1 00007f8fe1a7366c 00007f8fe1a72922 00007f8fe186d34c 00007f8fe186c786 00007f8fe186c12f
0129 03:00:19.880761 (+ 112us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2010029824}
I20260129 03:00:19.883123 20437 raft_consensus.cc:1240] T e7d83d4ccf684e76acb29a0dc844e1b8 P 197b880fa39d432a9b472845876bfaf0 [term 8 FOLLOWER]: Rejecting Update request from peer 974b802141bd4f13beffef2aa03f88f9 for earlier term 7. Current term is 8. Ops: []
I20260129 03:00:19.876725 20508 raft_consensus.cc:2368] T 3906b080a5b34a74b8b64128635e8997 P 974b802141bd4f13beffef2aa03f88f9 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 197b880fa39d432a9b472845876bfaf0 for earlier term 5. Current term is 6.
I20260129 03:00:19.879743 23682 leader_election.cc:290] T dc2f4cb752cf4d59b5f6bc689f5d8104 P 7a9daf5d42a84615a6383cfec9a10608 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 197b880fa39d432a9b472845876bfaf0 (127.18.211.1:42027), 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3:37735)
I20260129 03:00:19.893414 23646 raft_consensus.cc:515] T 00e79d8f25a34676b1d866eb2068a2b2 P 974b802141bd4f13beffef2aa03f88f9 [term 9 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } }
I20260129 03:00:19.901722 20435 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0678b8b468bd43cdbc3e485900844821" candidate_uuid: "7a9daf5d42a84615a6383cfec9a10608" candidate_term: 10 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "197b880fa39d432a9b472845876bfaf0" is_pre_election: true
I20260129 03:00:19.896176 23733 leader_election.cc:290] T febffe4a8b394368b69b1d164308fb9a P 7a9daf5d42a84615a6383cfec9a10608 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 197b880fa39d432a9b472845876bfaf0 (127.18.211.1:42027), 974b802141bd4f13beffef2aa03f88f9 (127.18.211.2:34517)
I20260129 03:00:19.903177 23646 leader_election.cc:290] T 00e79d8f25a34676b1d866eb2068a2b2 P 974b802141bd4f13beffef2aa03f88f9 [CANDIDATE]: Term 9 election: Requested vote from peers 197b880fa39d432a9b472845876bfaf0 (127.18.211.1:42027), 7a9daf5d42a84615a6383cfec9a10608 (127.18.211.4:40439)
I20260129 03:00:19.894397 23764 raft_consensus.cc:697] T a5a05fbcbbcf4916bbc01fd1a5b5e90d P 197b880fa39d432a9b472845876bfaf0 [term 9 LEADER]: Becoming Leader. State: Replica: 197b880fa39d432a9b472845876bfaf0, State: Running, Role: LEADER
I20260129 03:00:19.891685 23753 leader_election.cc:290] T 0678b8b468bd43cdbc3e485900844821 P 7a9daf5d42a84615a6383cfec9a10608 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 197b880fa39d432a9b472845876bfaf0 (127.18.211.1:42027), 974b802141bd4f13beffef2aa03f88f9 (127.18.211.2:34517)
I20260129 03:00:19.896184 20479 leader_election.cc:304] T 0340229d521b432580ae926f6062e3f7 P 974b802141bd4f13beffef2aa03f88f9 [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 197b880fa39d432a9b472845876bfaf0, 974b802141bd4f13beffef2aa03f88f9; no voters:
I20260129 03:00:19.897336 20439 raft_consensus.cc:1240] T 76f713c3331540d2a5b76164103e9eb1 P 197b880fa39d432a9b472845876bfaf0 [term 7 FOLLOWER]: Rejecting Update request from peer 974b802141bd4f13beffef2aa03f88f9 for earlier term 6. Current term is 7. Ops: []
I20260129 03:00:19.898535 23695 raft_consensus.cc:2749] T 75f45538dc704d9a8ec4ff8ad9f3ea5b P 974b802141bd4f13beffef2aa03f88f9 [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260129 03:00:19.891801 23627 leader_election.cc:290] T 4e3805ffa70242e8ade66da260dddf66 P 197b880fa39d432a9b472845876bfaf0 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3:37735), 7a9daf5d42a84615a6383cfec9a10608 (127.18.211.4:40439)
I20260129 03:00:19.899974 20434 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3906b080a5b34a74b8b64128635e8997" candidate_uuid: "974b802141bd4f13beffef2aa03f88f9" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "197b880fa39d432a9b472845876bfaf0"
I20260129 03:00:19.900709 20432 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "dc2f4cb752cf4d59b5f6bc689f5d8104" candidate_uuid: "7a9daf5d42a84615a6383cfec9a10608" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "197b880fa39d432a9b472845876bfaf0" is_pre_election: true
I20260129 03:00:19.916837 20434 raft_consensus.cc:3060] T 3906b080a5b34a74b8b64128635e8997 P 197b880fa39d432a9b472845876bfaf0 [term 5 FOLLOWER]: Advancing to term 6
I20260129 03:00:19.871300 20480 leader_election.cc:304] T 2ac659d732314e1288849f08fe5e074e P 974b802141bd4f13beffef2aa03f88f9 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 974b802141bd4f13beffef2aa03f88f9; no voters: 197b880fa39d432a9b472845876bfaf0, 7a9daf5d42a84615a6383cfec9a10608
I20260129 03:00:19.901289 20436 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "febffe4a8b394368b69b1d164308fb9a" candidate_uuid: "7a9daf5d42a84615a6383cfec9a10608" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "197b880fa39d432a9b472845876bfaf0" is_pre_election: true
I20260129 03:00:19.900624 20440 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "8055232cf0004d40a85f3bf983fe00a9" candidate_uuid: "974b802141bd4f13beffef2aa03f88f9" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "197b880fa39d432a9b472845876bfaf0"
I20260129 03:00:19.918025 20440 raft_consensus.cc:3060] T 8055232cf0004d40a85f3bf983fe00a9 P 197b880fa39d432a9b472845876bfaf0 [term 6 FOLLOWER]: Advancing to term 7
I20260129 03:00:19.925215 20435 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00e79d8f25a34676b1d866eb2068a2b2" candidate_uuid: "974b802141bd4f13beffef2aa03f88f9" candidate_term: 9 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "197b880fa39d432a9b472845876bfaf0"
I20260129 03:00:19.887107 23703 log.cc:1199] T 1299723a4476420ab02fd8bfd7aa0645 P 974b802141bd4f13beffef2aa03f88f9: Deleting WAL directory at /tmp/dist-test-taskrTP_Jo/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1769655460890924-19276-0/minicluster-data/ts-1-root/wals/1299723a4476420ab02fd8bfd7aa0645
I20260129 03:00:19.927295 20509 raft_consensus.cc:2468] T 75f45538dc704d9a8ec4ff8ad9f3ea5b P 974b802141bd4f13beffef2aa03f88f9 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7a9daf5d42a84615a6383cfec9a10608 in term 5.
I20260129 03:00:19.916447 23646 raft_consensus.cc:2804] T 0340229d521b432580ae926f6062e3f7 P 974b802141bd4f13beffef2aa03f88f9 [term 9 FOLLOWER]: Leader election won for term 9
I20260129 03:00:19.939105 23646 raft_consensus.cc:697] T 0340229d521b432580ae926f6062e3f7 P 974b802141bd4f13beffef2aa03f88f9 [term 9 LEADER]: Becoming Leader. State: Replica: 974b802141bd4f13beffef2aa03f88f9, State: Running, Role: LEADER
W20260129 03:00:19.939195 20480 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.211.4:40439, user_credentials={real_user=slave}} blocked reactor thread for 67994.5us
I20260129 03:00:19.940178 23646 consensus_queue.cc:237] T 0340229d521b432580ae926f6062e3f7 P 974b802141bd4f13beffef2aa03f88f9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 7.5, Last appended by leader: 5, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 03:00:19.939208 21135 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4e3805ffa70242e8ade66da260dddf66" candidate_uuid: "197b880fa39d432a9b472845876bfaf0" candidate_term: 7 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "7a9daf5d42a84615a6383cfec9a10608" is_pre_election: true
I20260129 03:00:19.941376 23764 consensus_queue.cc:237] T a5a05fbcbbcf4916bbc01fd1a5b5e90d P 197b880fa39d432a9b472845876bfaf0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 8.6, Last appended by leader: 6, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 03:00:19.939221 23450 raft_consensus.cc:2749] T 2ac659d732314e1288849f08fe5e074e P 974b802141bd4f13beffef2aa03f88f9 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260129 03:00:19.965621 21133 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00e79d8f25a34676b1d866eb2068a2b2" candidate_uuid: "974b802141bd4f13beffef2aa03f88f9" candidate_term: 9 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "7a9daf5d42a84615a6383cfec9a10608"
I20260129 03:00:19.938469 20440 raft_consensus.cc:2468] T 8055232cf0004d40a85f3bf983fe00a9 P 197b880fa39d432a9b472845876bfaf0 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 974b802141bd4f13beffef2aa03f88f9 in term 7.
I20260129 03:00:19.969197 20434 raft_consensus.cc:2468] T 3906b080a5b34a74b8b64128635e8997 P 197b880fa39d432a9b472845876bfaf0 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 974b802141bd4f13beffef2aa03f88f9 in term 6.
I20260129 03:00:19.970449 20479 leader_election.cc:304] T 8055232cf0004d40a85f3bf983fe00a9 P 974b802141bd4f13beffef2aa03f88f9 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 197b880fa39d432a9b472845876bfaf0, 974b802141bd4f13beffef2aa03f88f9; no voters:
I20260129 03:00:19.984427 20480 leader_election.cc:304] T 00e79d8f25a34676b1d866eb2068a2b2 P 974b802141bd4f13beffef2aa03f88f9 [CANDIDATE]: Term 9 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 974b802141bd4f13beffef2aa03f88f9; no voters: 197b880fa39d432a9b472845876bfaf0, 7a9daf5d42a84615a6383cfec9a10608
I20260129 03:00:19.965220 23688 consensus_queue.cc:1048] T add8d4923d1a489e86882c3a25692b95 P 7a9daf5d42a84615a6383cfec9a10608 [LEADER]: Connected to new peer: Peer: permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.022s
I20260129 03:00:19.986622 20507 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0678b8b468bd43cdbc3e485900844821" candidate_uuid: "7a9daf5d42a84615a6383cfec9a10608" candidate_term: 10 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "974b802141bd4f13beffef2aa03f88f9" is_pre_election: true
I20260129 03:00:19.987984 20512 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "45cc5f3d2688486d9b6dbe7e666dedde" candidate_uuid: "7a9daf5d42a84615a6383cfec9a10608" candidate_term: 8 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "974b802141bd4f13beffef2aa03f88f9"
I20260129 03:00:19.988582 20512 raft_consensus.cc:3060] T 45cc5f3d2688486d9b6dbe7e666dedde P 974b802141bd4f13beffef2aa03f88f9 [term 7 FOLLOWER]: Advancing to term 8
I20260129 03:00:19.993065 23703 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:57580 (request call id 491) took 2557 ms. Trace:
I20260129 03:00:19.993607 23703 rpcz_store.cc:276] 0129 03:00:17.800851 (+ 0us) service_pool.cc:168] Inserting onto call queue
0129 03:00:17.801310 (+ 459us) service_pool.cc:225] Handling call
0129 03:00:18.333801 (+532491us) ts_tablet_manager.cc:1151] Deleting tablet 1299723a4476420ab02fd8bfd7aa0645
0129 03:00:18.333870 (+ 69us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
0129 03:00:19.747771 (+1413901us) tablet_metadata.cc:641] Metadata flushed
0129 03:00:19.826390 (+ 78619us) tablet_metadata.cc:641] Metadata flushed
0129 03:00:19.992963 (+166573us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":127575,"tablet-delete.queue_time_us":420825,"thread_start_us":270257,"threads_started":1}
W20260129 03:00:20.000599 23771 log.cc:927] Time spent T 724a236524cb4e7ba3428880681d6cc2 P 974b802141bd4f13beffef2aa03f88f9: Append to log took a long time: real 0.062s user 0.000s sys 0.000s
I20260129 03:00:20.005267 23674 raft_consensus.cc:493] T fcddab6bca684adf8ea032289a00f90e P 197b880fa39d432a9b472845876bfaf0 [term 10 FOLLOWER]: Starting leader election (detected failure of leader 974b802141bd4f13beffef2aa03f88f9)
I20260129 03:00:20.005916 23674 raft_consensus.cc:3060] T fcddab6bca684adf8ea032289a00f90e P 197b880fa39d432a9b472845876bfaf0 [term 10 FOLLOWER]: Advancing to term 11
I20260129 03:00:20.012691 23750 raft_consensus.cc:2749] T fd8311b779af4e1fa1402b2a02a602f2 P 7a9daf5d42a84615a6383cfec9a10608 [term 6 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260129 03:00:19.988319 20510 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "febffe4a8b394368b69b1d164308fb9a" candidate_uuid: "7a9daf5d42a84615a6383cfec9a10608" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "974b802141bd4f13beffef2aa03f88f9" is_pre_election: true
I20260129 03:00:19.933808 23710 consensus_queue.cc:1059] T 76f713c3331540d2a5b76164103e9eb1 P 974b802141bd4f13beffef2aa03f88f9 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 }, Status: INVALID_TERM, Last received: 6.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.036s
I20260129 03:00:20.023099 23694 raft_consensus.cc:2749] T 00e79d8f25a34676b1d866eb2068a2b2 P 974b802141bd4f13beffef2aa03f88f9 [term 9 FOLLOWER]: Leader election lost for term 9. Reason: could not achieve majority
I20260129 03:00:20.023289 23706 raft_consensus.cc:2804] T 8055232cf0004d40a85f3bf983fe00a9 P 974b802141bd4f13beffef2aa03f88f9 [term 7 FOLLOWER]: Leader election won for term 7
W20260129 03:00:20.023123 20479 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.211.1:42027, user_credentials={real_user=slave}} blocked reactor thread for 52788.5us
I20260129 03:00:20.023813 23706 raft_consensus.cc:697] T 8055232cf0004d40a85f3bf983fe00a9 P 974b802141bd4f13beffef2aa03f88f9 [term 7 LEADER]: Becoming Leader. State: Replica: 974b802141bd4f13beffef2aa03f88f9, State: Running, Role: LEADER
I20260129 03:00:20.025269 20479 leader_election.cc:304] T 3906b080a5b34a74b8b64128635e8997 P 974b802141bd4f13beffef2aa03f88f9 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 197b880fa39d432a9b472845876bfaf0, 974b802141bd4f13beffef2aa03f88f9; no voters:
I20260129 03:00:20.025520 23706 consensus_queue.cc:237] T 8055232cf0004d40a85f3bf983fe00a9 P 974b802141bd4f13beffef2aa03f88f9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 5.3, Last appended by leader: 3, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 03:00:20.026247 20325 catalog_manager.cc:4985] TS 974b802141bd4f13beffef2aa03f88f9 (127.18.211.2:34517): tablet 1299723a4476420ab02fd8bfd7aa0645 (table test-workload [id=de73611ac8e3485a98aecc5add247032]) successfully deleted
I20260129 03:00:20.024323 23710 raft_consensus.cc:3055] T 76f713c3331540d2a5b76164103e9eb1 P 974b802141bd4f13beffef2aa03f88f9 [term 6 LEADER]: Stepping down as leader of term 6
I20260129 03:00:20.027459 23710 raft_consensus.cc:740] T 76f713c3331540d2a5b76164103e9eb1 P 974b802141bd4f13beffef2aa03f88f9 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 974b802141bd4f13beffef2aa03f88f9, State: Running, Role: LEADER
I20260129 03:00:20.027709 20514 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:48544 (request call id 374) took 1300 ms. Trace:
I20260129 03:00:20.028344 23710 consensus_queue.cc:260] T 76f713c3331540d2a5b76164103e9eb1 P 974b802141bd4f13beffef2aa03f88f9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 6.5, Last appended by leader: 5, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 03:00:20.029903 23710 raft_consensus.cc:3060] T 76f713c3331540d2a5b76164103e9eb1 P 974b802141bd4f13beffef2aa03f88f9 [term 6 FOLLOWER]: Advancing to term 7
I20260129 03:00:20.028148 20514 rpcz_store.cc:276] 0129 03:00:18.727472 (+ 0us) service_pool.cc:168] Inserting onto call queue
0129 03:00:18.731459 (+ 3987us) service_pool.cc:225] Handling call
0129 03:00:18.731739 (+ 280us) raft_consensus.cc:1471] Updating replica for 0 ops
0129 03:00:18.741664 (+ 9925us) raft_consensus.cc:1534] Early marking committed up to index 0
0129 03:00:18.741707 (+ 43us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0129 03:00:18.741752 (+ 45us) raft_consensus.cc:1645] Marking committed up to 5
0129 03:00:18.741792 (+ 40us) raft_consensus.cc:1697] Filling consensus response to leader.
0129 03:00:20.027472 (+1285680us) spinlock_profiling.cc:243] Waited 9.71 ms on lock 0x7b54000fcb54. stack: 00007f8fe044e373 00007f8fe044e270 00007f8fdfda0574 00007f8fe9a5c6b4 00007f8fe9a5c649 00007f8fe34975a1 00007f8fe352ea81 00007f8fe352d4be 00007f8fe84e1f49 00007f8fe1a6ed34 00007f8fe1a6ecc2 00007f8fe1a6ebf1 00007f8fe1a6eb6c 00007f8fe1a6de22 00007f8fe186d34c 00007f8fe186c786
0129 03:00:20.027519 (+ 47us) raft_consensus.cc:1691] UpdateReplicas() finished
0129 03:00:20.027641 (+ 122us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":22331648}
I20260129 03:00:20.037159 23450 raft_consensus.cc:493] T c2633a6fc34c44579de348d208b0c01e P 974b802141bd4f13beffef2aa03f88f9 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260129 03:00:20.038168 23450 raft_consensus.cc:515] T c2633a6fc34c44579de348d208b0c01e P 974b802141bd4f13beffef2aa03f88f9 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 03:00:20.039301 23694 raft_consensus.cc:2804] T 3906b080a5b34a74b8b64128635e8997 P 974b802141bd4f13beffef2aa03f88f9 [term 6 FOLLOWER]: Leader election won for term 6
I20260129 03:00:20.041746 21102 leader_election.cc:304] T 75b0b6e82ddc4f92b1fe1d94f9a0b73b P 7a9daf5d42a84615a6383cfec9a10608 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 7a9daf5d42a84615a6383cfec9a10608, 974b802141bd4f13beffef2aa03f88f9; no voters: 1c50de89a5a845668696dcb28c3badd6
W20260129 03:00:20.056468 23732 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.588s user 0.029s sys 0.044s
W20260129 03:00:20.063696 23732 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.598s user 0.029s sys 0.044s
W20260129 03:00:20.064352 20590 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:35754 (request call id 1883) took 1712 ms (client timeout 1536 ms). Trace:
W20260129 03:00:20.064935 20590 rpcz_store.cc:269] 0129 03:00:18.353495 (+ 0us) service_pool.cc:168] Inserting onto call queue
0129 03:00:18.359960 (+ 6465us) service_pool.cc:225] Handling call
0129 03:00:20.064069 (+1704109us) spinlock_profiling.cc:243] Waited 1.06 s on lock 0x7b54000c9254. stack: 00007f8fe044e373 00007f8fe044e270 00007f8fdfda0574 00007f8fe9a5c6b4 00007f8fe9a5c649 00007f8fe34975a1 00007f8fe3531f7a 00007f8fe84e3647 00007f8fe1a73834 00007f8fe1a737c2 00007f8fe1a736f1 00007f8fe1a7366c 00007f8fe1a72922 00007f8fe186d34c 00007f8fe186c786 00007f8fe186c12f
0129 03:00:20.064260 (+ 191us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2428347648}
I20260129 03:00:20.064209 23732 raft_consensus.cc:697] T 0f20af7fbe5d4db89a6f17948d98912c P 974b802141bd4f13beffef2aa03f88f9 [term 8 LEADER]: Becoming Leader. State: Replica: 974b802141bd4f13beffef2aa03f88f9, State: Running, Role: LEADER
I20260129 03:00:20.067381 23732 consensus_queue.cc:237] T 0f20af7fbe5d4db89a6f17948d98912c P 974b802141bd4f13beffef2aa03f88f9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 7.5, Last appended by leader: 5, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
W20260129 03:00:20.068715 20554 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.211.4:40439, user_credentials={real_user=slave}} blocked reactor thread for 571225us
W20260129 03:00:20.070873 20554 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260129 03:00:20.046437 23694 raft_consensus.cc:697] T 3906b080a5b34a74b8b64128635e8997 P 974b802141bd4f13beffef2aa03f88f9 [term 6 LEADER]: Becoming Leader. State: Replica: 974b802141bd4f13beffef2aa03f88f9, State: Running, Role: LEADER
I20260129 03:00:20.047583 23740 raft_consensus.cc:2804] T 75b0b6e82ddc4f92b1fe1d94f9a0b73b P 7a9daf5d42a84615a6383cfec9a10608 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260129 03:00:20.074828 20584 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "cedce331e1584ac8b636c9b8a72b2ba0" candidate_uuid: "974b802141bd4f13beffef2aa03f88f9" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "1c50de89a5a845668696dcb28c3badd6"
I20260129 03:00:20.094034 23740 raft_consensus.cc:493] T 75b0b6e82ddc4f92b1fe1d94f9a0b73b P 7a9daf5d42a84615a6383cfec9a10608 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260129 03:00:20.094192 20584 raft_consensus.cc:3060] T cedce331e1584ac8b636c9b8a72b2ba0 P 1c50de89a5a845668696dcb28c3badd6 [term 8 FOLLOWER]: Advancing to term 9
I20260129 03:00:20.094477 23740 raft_consensus.cc:3060] T 75b0b6e82ddc4f92b1fe1d94f9a0b73b P 7a9daf5d42a84615a6383cfec9a10608 [term 5 FOLLOWER]: Advancing to term 6
I20260129 03:00:20.097497 20582 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a0fcc58d6fb6473cafbd9d34d5195269" candidate_uuid: "974b802141bd4f13beffef2aa03f88f9" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "1c50de89a5a845668696dcb28c3badd6"
I20260129 03:00:20.098058 20582 raft_consensus.cc:3060] T a0fcc58d6fb6473cafbd9d34d5195269 P 1c50de89a5a845668696dcb28c3badd6 [term 5 FOLLOWER]: Advancing to term 6
I20260129 03:00:20.080440 20587 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "368953c9102643b2a21f99af36b51905" candidate_uuid: "974b802141bd4f13beffef2aa03f88f9" candidate_term: 10 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "1c50de89a5a845668696dcb28c3badd6"
I20260129 03:00:20.100296 20587 raft_consensus.cc:3060] T 368953c9102643b2a21f99af36b51905 P 1c50de89a5a845668696dcb28c3badd6 [term 9 FOLLOWER]: Advancing to term 10
I20260129 03:00:20.080554 20480 leader_election.cc:304] T a1a39dcd328e4268ae41b988c66b4a75 P 974b802141bd4f13beffef2aa03f88f9 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1c50de89a5a845668696dcb28c3badd6, 974b802141bd4f13beffef2aa03f88f9; no voters: 197b880fa39d432a9b472845876bfaf0
I20260129 03:00:20.081102 23450 leader_election.cc:290] T c2633a6fc34c44579de348d208b0c01e P 974b802141bd4f13beffef2aa03f88f9 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 197b880fa39d432a9b472845876bfaf0 (127.18.211.1:42027), 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3:37735)
I20260129 03:00:20.081446 20583 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4f8f3a38f6e24ba4857875181f447442" candidate_uuid: "974b802141bd4f13beffef2aa03f88f9" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "1c50de89a5a845668696dcb28c3badd6"
I20260129 03:00:20.108592 20583 raft_consensus.cc:3060] T 4f8f3a38f6e24ba4857875181f447442 P 1c50de89a5a845668696dcb28c3badd6 [term 7 FOLLOWER]: Advancing to term 8
I20260129 03:00:20.081446 20588 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7a9a69cd82724b1faed1d10aa091fc4a" candidate_uuid: "974b802141bd4f13beffef2aa03f88f9" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "1c50de89a5a845668696dcb28c3badd6"
I20260129 03:00:20.109294 20588 raft_consensus.cc:3060] T 7a9a69cd82724b1faed1d10aa091fc4a P 1c50de89a5a845668696dcb28c3badd6 [term 7 FOLLOWER]: Advancing to term 8
I20260129 03:00:20.083847 20589 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0340229d521b432580ae926f6062e3f7" candidate_uuid: "974b802141bd4f13beffef2aa03f88f9" candidate_term: 9 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "1c50de89a5a845668696dcb28c3badd6"
I20260129 03:00:20.084223 21135 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "45cc5f3d2688486d9b6dbe7e666dedde" candidate_uuid: "1c50de89a5a845668696dcb28c3badd6" candidate_term: 8 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "7a9daf5d42a84615a6383cfec9a10608"
I20260129 03:00:20.086570 23694 consensus_queue.cc:237] T 3906b080a5b34a74b8b64128635e8997 P 974b802141bd4f13beffef2aa03f88f9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 03:00:20.088913 23747 raft_consensus.cc:493] T 18f30ebaa4174fd28c958e72d0cc7e24 P 1c50de89a5a845668696dcb28c3badd6 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 7a9daf5d42a84615a6383cfec9a10608)
W20260129 03:00:20.089901 20555 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260129 03:00:20.061210 23695 raft_consensus.cc:493] T a644c02b16af47679824ccd060ac10fd P 974b802141bd4f13beffef2aa03f88f9 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 1c50de89a5a845668696dcb28c3badd6)
I20260129 03:00:20.092264 20581 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35752 (request call id 1784) took 1176 ms. Trace:
W20260129 03:00:20.092796 20553 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.211.1:42027, user_credentials={real_user=slave}} blocked reactor thread for 573499us
I20260129 03:00:20.120353 20581 rpcz_store.cc:276] 0129 03:00:18.918136 (+ 0us) service_pool.cc:168] Inserting onto call queue
0129 03:00:19.090945 (+172809us) service_pool.cc:225] Handling call
0129 03:00:19.115375 (+ 24430us) raft_consensus.cc:1471] Updating replica for 1 ops
0129 03:00:19.707586 (+592211us) raft_consensus.cc:1697] Filling consensus response to leader.
0129 03:00:20.092044 (+384458us) spinlock_profiling.cc:243] Waited 581 ms on lock 0x7b54001540d4. stack: 00007f8fe044e373 00007f8fe044e270 00007f8fdfda0574 00007f8fe9a5c6b4 00007f8fe9a5c649 00007f8fe34975a1 00007f8fe352ea81 00007f8fe352d4be 00007f8fe84e1f49 00007f8fe1a6ed34 00007f8fe1a6ecc2 00007f8fe1a6ebf1 00007f8fe1a6eb6c 00007f8fe1a6de22 00007f8fe186d34c 00007f8fe186c786
0129 03:00:20.092208 (+ 164us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1337368448}
I20260129 03:00:20.102732 20584 raft_consensus.cc:2468] T cedce331e1584ac8b636c9b8a72b2ba0 P 1c50de89a5a845668696dcb28c3badd6 [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 974b802141bd4f13beffef2aa03f88f9 in term 9.
I20260129 03:00:20.107033 20512 raft_consensus.cc:2468] T 45cc5f3d2688486d9b6dbe7e666dedde P 974b802141bd4f13beffef2aa03f88f9 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7a9daf5d42a84615a6383cfec9a10608 in term 8.
I20260129 03:00:20.110211 20589 raft_consensus.cc:3060] T 0340229d521b432580ae926f6062e3f7 P 1c50de89a5a845668696dcb28c3badd6 [term 8 FOLLOWER]: Advancing to term 9
I20260129 03:00:20.123050 20590 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3906b080a5b34a74b8b64128635e8997" candidate_uuid: "974b802141bd4f13beffef2aa03f88f9" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "1c50de89a5a845668696dcb28c3badd6"
I20260129 03:00:20.123759 20590 raft_consensus.cc:3060] T 3906b080a5b34a74b8b64128635e8997 P 1c50de89a5a845668696dcb28c3badd6 [term 5 FOLLOWER]: Advancing to term 6
I20260129 03:00:20.114470 20586 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35752 (request call id 1783) took 1198 ms. Trace:
I20260129 03:00:20.115265 21135 raft_consensus.cc:2393] T 45cc5f3d2688486d9b6dbe7e666dedde P 7a9daf5d42a84615a6383cfec9a10608 [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate 1c50de89a5a845668696dcb28c3badd6 in current term 8: Already voted for candidate 7a9daf5d42a84615a6383cfec9a10608 in this term.
W20260129 03:00:20.091284 23763 log.cc:927] Time spent T 2ac659d732314e1288849f08fe5e074e P 974b802141bd4f13beffef2aa03f88f9: Append to log took a long time: real 0.329s user 0.001s sys 0.000s
I20260129 03:00:20.117983 20585 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "926a250fa5834e88b96ccecbefbf5086" candidate_uuid: "974b802141bd4f13beffef2aa03f88f9" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "1c50de89a5a845668696dcb28c3badd6"
I20260129 03:00:20.110864 20480 leader_election.cc:304] T cb4727386a3e48aca5d71cc6c611c197 P 974b802141bd4f13beffef2aa03f88f9 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1c50de89a5a845668696dcb28c3badd6, 974b802141bd4f13beffef2aa03f88f9; no voters: 197b880fa39d432a9b472845876bfaf0
I20260129 03:00:20.124284 20586 rpcz_store.cc:276] 0129 03:00:18.915994 (+ 0us) service_pool.cc:168] Inserting onto call queue
0129 03:00:19.072089 (+156095us) service_pool.cc:225] Handling call
0129 03:00:19.072526 (+ 437us) raft_consensus.cc:1471] Updating replica for 1 ops
0129 03:00:19.072826 (+ 300us) raft_consensus.cc:1534] Early marking committed up to index 0
0129 03:00:19.072889 (+ 63us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0129 03:00:19.099351 (+ 26462us) log.cc:844] Serialized 26 byte log entry
0129 03:00:19.295674 (+196323us) raft_consensus.cc:1645] Marking committed up to 5
0129 03:00:19.839723 (+544049us) spinlock_profiling.cc:243] Waited 10.7 ms on lock 0x7b440002304c. stack: 00007f8fe044e373 00007f8fe044e270 00007f8fdfda0574 00007f8fe9a5c6b4 00007f8fe9a5c649 00007f8fe02c08b1 00007f8fe038f005 00007f8fe038fa22 00007f8fe30d957b 00007f8fe30d4e4f 00007f8fe30c8c0e 00007f8fe30c7f56 00007f8fe30c7b78 00007f8fe30d15c1 00007f8fe30d1579 00007f8fe30d1509
0129 03:00:20.026995 (+187272us) spinlock_profiling.cc:243] Waited 9.44 ms on lock 0x7b18000d91e0. stack: 00007f8fe044e373 00007f8fe044e270 00007f8fdfda0574 00007f8fe9a5c6b4 00007f8fe9a5c649 00007f8fe92d4c91 00007f8fe92e1671 00007f8fe30d585c 00007f8fe04632eb 00007f8fe924b709 00007f8fe047f57b 00007f8fe047d81f 00007f8fe047d335 00007f8fe6f28317 00007f8fe3538627 00007f8fe3547d3f
0129 03:00:20.027146 (+ 151us) log.cc:844] Serialized 14 byte log entry
0129 03:00:20.114164 (+ 87018us) raft_consensus.cc:1697] Filling consensus response to leader.
0129 03:00:20.114241 (+ 77us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0129 03:00:20.114278 (+ 37us) raft_consensus.cc:1685] finished
0129 03:00:20.114288 (+ 10us) raft_consensus.cc:1691] UpdateReplicas() finished
0129 03:00:20.114412 (+ 124us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":86752,"prepare.queue_time_us":175139,"prepare.run_cpu_time_us":68,"prepare.run_wall_time_us":1717,"spinlock_wait_cycles":46296192,"thread_start_us":292147,"threads_started":2,"wal-append.queue_time_us":329378}
I20260129 03:00:20.126394 20585 raft_consensus.cc:3060] T 926a250fa5834e88b96ccecbefbf5086 P 1c50de89a5a845668696dcb28c3badd6 [term 8 FOLLOWER]: Advancing to term 9
I20260129 03:00:20.132639 21102 leader_election.cc:304] T 45cc5f3d2688486d9b6dbe7e666dedde P 7a9daf5d42a84615a6383cfec9a10608 [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7a9daf5d42a84615a6383cfec9a10608, 974b802141bd4f13beffef2aa03f88f9; no voters:
I20260129 03:00:20.134234 20589 raft_consensus.cc:2468] T 0340229d521b432580ae926f6062e3f7 P 1c50de89a5a845668696dcb28c3badd6 [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 974b802141bd4f13beffef2aa03f88f9 in term 9.
I20260129 03:00:20.136701 20581 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "8055232cf0004d40a85f3bf983fe00a9" candidate_uuid: "974b802141bd4f13beffef2aa03f88f9" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "1c50de89a5a845668696dcb28c3badd6"
I20260129 03:00:20.137307 20581 raft_consensus.cc:3060] T 8055232cf0004d40a85f3bf983fe00a9 P 1c50de89a5a845668696dcb28c3badd6 [term 6 FOLLOWER]: Advancing to term 7
I20260129 03:00:20.142445 20583 raft_consensus.cc:2468] T 4f8f3a38f6e24ba4857875181f447442 P 1c50de89a5a845668696dcb28c3badd6 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 974b802141bd4f13beffef2aa03f88f9 in term 8.
I20260129 03:00:20.146718 20587 raft_consensus.cc:2468] T 368953c9102643b2a21f99af36b51905 P 1c50de89a5a845668696dcb28c3badd6 [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 974b802141bd4f13beffef2aa03f88f9 in term 10.
I20260129 03:00:20.148806 23777 raft_consensus.cc:2804] T a1a39dcd328e4268ae41b988c66b4a75 P 974b802141bd4f13beffef2aa03f88f9 [term 7 FOLLOWER]: Leader election won for term 7
W20260129 03:00:20.159025 20481 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260129 03:00:20.160702 23741 raft_consensus.cc:2804] T 45cc5f3d2688486d9b6dbe7e666dedde P 7a9daf5d42a84615a6383cfec9a10608 [term 8 FOLLOWER]: Leader election won for term 8
I20260129 03:00:20.161285 23741 raft_consensus.cc:697] T 45cc5f3d2688486d9b6dbe7e666dedde P 7a9daf5d42a84615a6383cfec9a10608 [term 8 LEADER]: Becoming Leader. State: Replica: 7a9daf5d42a84615a6383cfec9a10608, State: Running, Role: LEADER
I20260129 03:00:20.161453 20588 raft_consensus.cc:2468] T 7a9a69cd82724b1faed1d10aa091fc4a P 1c50de89a5a845668696dcb28c3badd6 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 974b802141bd4f13beffef2aa03f88f9 in term 8.
I20260129 03:00:20.162225 23741 consensus_queue.cc:237] T 45cc5f3d2688486d9b6dbe7e666dedde P 7a9daf5d42a84615a6383cfec9a10608 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 6.6, Last appended by leader: 6, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } }
I20260129 03:00:20.165264 20581 raft_consensus.cc:2468] T 8055232cf0004d40a85f3bf983fe00a9 P 1c50de89a5a845668696dcb28c3badd6 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 974b802141bd4f13beffef2aa03f88f9 in term 7.
I20260129 03:00:20.169486 21102 leader_election.cc:304] T 75f45538dc704d9a8ec4ff8ad9f3ea5b P 7a9daf5d42a84615a6383cfec9a10608 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7a9daf5d42a84615a6383cfec9a10608, 974b802141bd4f13beffef2aa03f88f9; no voters:
W20260129 03:00:20.176221 20556 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.211.2:34517, user_credentials={real_user=slave}} blocked reactor thread for 664714us
I20260129 03:00:20.177711 20556 leader_election.cc:304] T 76f713c3331540d2a5b76164103e9eb1 P 1c50de89a5a845668696dcb28c3badd6 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1c50de89a5a845668696dcb28c3badd6; no voters: 197b880fa39d432a9b472845876bfaf0, 974b802141bd4f13beffef2aa03f88f9
W20260129 03:00:20.181293 20405 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.211.4:40439, user_credentials={real_user=slave}} blocked reactor thread for 248530us
I20260129 03:00:20.169540 20433 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55672 (request call id 1821) took 1247 ms. Trace:
I20260129 03:00:20.184700 20433 rpcz_store.cc:276] 0129 03:00:18.922272 (+ 0us) service_pool.cc:168] Inserting onto call queue
0129 03:00:18.922598 (+ 326us) service_pool.cc:225] Handling call
0129 03:00:18.968014 (+ 45416us) raft_consensus.cc:1471] Updating replica for 0 ops
0129 03:00:19.762480 (+794466us) raft_consensus.cc:1697] Filling consensus response to leader.
0129 03:00:20.169294 (+406814us) spinlock_profiling.cc:243] Waited 794 ms on lock 0x7b54002b4b54. stack: 00007f8fe044e373 00007f8fe044e270 00007f8fdfda0574 00007f8fe9a5c6b4 00007f8fe9a5c649 00007f8fe34975a1 00007f8fe352ea81 00007f8fe352d4be 00007f8fe84e1f49 00007f8fe1a6ed34 00007f8fe1a6ecc2 00007f8fe1a6ebf1 00007f8fe1a6eb6c 00007f8fe1a6de22 00007f8fe186d34c 00007f8fe186c786
0129 03:00:20.169458 (+ 164us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1825895296}
W20260129 03:00:20.188017 20407 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.211.2:34517, user_credentials={real_user=slave}} blocked reactor thread for 167184us
I20260129 03:00:20.189683 23742 consensus_queue.cc:1059] T a0fcc58d6fb6473cafbd9d34d5195269 P 197b880fa39d432a9b472845876bfaf0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 }, Status: INVALID_TERM, Last received: 5.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260129 03:00:20.190492 23742 raft_consensus.cc:3055] T a0fcc58d6fb6473cafbd9d34d5195269 P 197b880fa39d432a9b472845876bfaf0 [term 5 LEADER]: Stepping down as leader of term 5
I20260129 03:00:20.190882 23742 raft_consensus.cc:740] T a0fcc58d6fb6473cafbd9d34d5195269 P 197b880fa39d432a9b472845876bfaf0 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 197b880fa39d432a9b472845876bfaf0, State: Running, Role: LEADER
I20260129 03:00:20.192418 20438 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c2633a6fc34c44579de348d208b0c01e" candidate_uuid: "974b802141bd4f13beffef2aa03f88f9" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "197b880fa39d432a9b472845876bfaf0" is_pre_election: true
I20260129 03:00:20.194235 20437 raft_consensus.cc:1240] T 17d6431cc77a491e8464d68db4081a51 P 197b880fa39d432a9b472845876bfaf0 [term 7 FOLLOWER]: Rejecting Update request from peer 974b802141bd4f13beffef2aa03f88f9 for earlier term 6. Current term is 7. Ops: []
I20260129 03:00:20.197239 23602 raft_consensus.cc:2749] T 76f713c3331540d2a5b76164103e9eb1 P 1c50de89a5a845668696dcb28c3badd6 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260129 03:00:20.197139 23740 raft_consensus.cc:515] T 75b0b6e82ddc4f92b1fe1d94f9a0b73b P 7a9daf5d42a84615a6383cfec9a10608 [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } }
I20260129 03:00:20.187465 23627 raft_consensus.cc:493] T c891d2bff4ce4568861bbc992e91fbb0 P 197b880fa39d432a9b472845876bfaf0 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260129 03:00:20.198547 23627 raft_consensus.cc:515] T c891d2bff4ce4568861bbc992e91fbb0 P 197b880fa39d432a9b472845876bfaf0 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 03:00:20.199069 20556 leader_election.cc:304] T add8d4923d1a489e86882c3a25692b95 P 1c50de89a5a845668696dcb28c3badd6 [CANDIDATE]: Term 8 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1c50de89a5a845668696dcb28c3badd6; no voters: 7a9daf5d42a84615a6383cfec9a10608, 974b802141bd4f13beffef2aa03f88f9
I20260129 03:00:20.200398 20433 raft_consensus.cc:1240] T a5a05fbcbbcf4916bbc01fd1a5b5e90d P 197b880fa39d432a9b472845876bfaf0 [term 9 LEADER]: Rejecting Update request from peer 1c50de89a5a845668696dcb28c3badd6 for earlier term 8. Current term is 9. Ops: []
I20260129 03:00:20.201023 23627 leader_election.cc:290] T c891d2bff4ce4568861bbc992e91fbb0 P 197b880fa39d432a9b472845876bfaf0 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 974b802141bd4f13beffef2aa03f88f9 (127.18.211.2:34517), 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3:37735)
I20260129 03:00:20.191587 23742 consensus_queue.cc:260] T a0fcc58d6fb6473cafbd9d34d5195269 P 197b880fa39d432a9b472845876bfaf0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 5.4, Last appended by leader: 4, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 03:00:20.202155 23742 raft_consensus.cc:3060] T a0fcc58d6fb6473cafbd9d34d5195269 P 197b880fa39d432a9b472845876bfaf0 [term 5 FOLLOWER]: Advancing to term 6
I20260129 03:00:20.203325 20586 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "45cc5f3d2688486d9b6dbe7e666dedde" candidate_uuid: "7a9daf5d42a84615a6383cfec9a10608" candidate_term: 8 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "1c50de89a5a845668696dcb28c3badd6"
I20260129 03:00:20.204104 20586 raft_consensus.cc:2393] T 45cc5f3d2688486d9b6dbe7e666dedde P 1c50de89a5a845668696dcb28c3badd6 [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate 7a9daf5d42a84615a6383cfec9a10608 in current term 8: Already voted for candidate 1c50de89a5a845668696dcb28c3badd6 in this term.
I20260129 03:00:20.205827 20586 raft_consensus.cc:1275] T add8d4923d1a489e86882c3a25692b95 P 1c50de89a5a845668696dcb28c3badd6 [term 9 FOLLOWER]: Refusing update from remote peer 7a9daf5d42a84615a6383cfec9a10608: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 9 index: 7. (index mismatch)
I20260129 03:00:20.207638 20441 raft_consensus.cc:1275] T e33a417cf9c54cf5af73a4ee3aac4d3a P 197b880fa39d432a9b472845876bfaf0 [term 8 FOLLOWER]: Refusing update from remote peer 1c50de89a5a845668696dcb28c3badd6: Log matching property violated. Preceding OpId in replica: term: 6 index: 3. Preceding OpId from leader: term: 8 index: 4. (index mismatch)
W20260129 03:00:20.207830 20508 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:57600 (request call id 2032) took 1459 ms (client timeout 1859 ms). Trace:
I20260129 03:00:20.137588 23747 raft_consensus.cc:515] T 18f30ebaa4174fd28c958e72d0cc7e24 P 1c50de89a5a845668696dcb28c3badd6 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } }
W20260129 03:00:20.208449 20405 leader_election.cc:336] T 0340229d521b432580ae926f6062e3f7 P 197b880fa39d432a9b472845876bfaf0 [CANDIDATE]: Term 8 election: RPC error from VoteRequest() call to peer 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3:37735): Timed out: RequestConsensusVote RPC to 127.18.211.3:37735 timed out after 1.552s (SENT)
W20260129 03:00:20.208230 20508 rpcz_store.cc:269] 0129 03:00:18.748693 (+ 0us) service_pool.cc:168] Inserting onto call queue
0129 03:00:18.749517 (+ 824us) service_pool.cc:225] Handling call
0129 03:00:20.207644 (+1458127us) spinlock_profiling.cc:243] Waited 1.08 s on lock 0x7b540026f054. stack: 00007f8fe044e373 00007f8fe044e270 00007f8fdfda0574 00007f8fe9a5c6b4 00007f8fe9a5c649 00007f8fe34975a1 00007f8fe3531f7a 00007f8fe84e3647 00007f8fe1a73834 00007f8fe1a737c2 00007f8fe1a736f1 00007f8fe1a7366c 00007f8fe1a72922 00007f8fe186d34c 00007f8fe186c786 00007f8fe186c12f
0129 03:00:20.207775 (+ 131us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2487409536}
I20260129 03:00:20.208986 20405 leader_election.cc:304] T 0340229d521b432580ae926f6062e3f7 P 197b880fa39d432a9b472845876bfaf0 [CANDIDATE]: Term 8 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 197b880fa39d432a9b472845876bfaf0; no voters: 1c50de89a5a845668696dcb28c3badd6, 974b802141bd4f13beffef2aa03f88f9
I20260129 03:00:20.209880 23747 leader_election.cc:290] T 18f30ebaa4174fd28c958e72d0cc7e24 P 1c50de89a5a845668696dcb28c3badd6 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 197b880fa39d432a9b472845876bfaf0 (127.18.211.1:42027), 7a9daf5d42a84615a6383cfec9a10608 (127.18.211.4:40439)
I20260129 03:00:20.127760 23695 raft_consensus.cc:515] T a644c02b16af47679824ccd060ac10fd P 974b802141bd4f13beffef2aa03f88f9 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } }
W20260129 03:00:20.210409 20405 leader_election.cc:336] T e33a417cf9c54cf5af73a4ee3aac4d3a P 197b880fa39d432a9b472845876bfaf0 [CANDIDATE]: Term 7 election: RPC error from VoteRequest() call to peer 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3:37735): Timed out: RequestConsensusVote RPC to 127.18.211.3:37735 timed out after 1.536s (SENT)
I20260129 03:00:20.210988 23403 leader_election.cc:290] T a8479d1fe0bd41e5946186e3a6ca8c7a P 197b880fa39d432a9b472845876bfaf0 [CANDIDATE]: Term 6 election: Requested vote from peers 974b802141bd4f13beffef2aa03f88f9 (127.18.211.2:34517), 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3:37735)
W20260129 03:00:20.211495 20405 leader_election.cc:336] T fd8311b779af4e1fa1402b2a02a602f2 P 197b880fa39d432a9b472845876bfaf0 [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3:37735): Timed out: RequestConsensusVote RPC to 127.18.211.3:37735 timed out after 1.600s (SENT)
I20260129 03:00:20.221264 21135 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a644c02b16af47679824ccd060ac10fd" candidate_uuid: "974b802141bd4f13beffef2aa03f88f9" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "7a9daf5d42a84615a6383cfec9a10608" is_pre_election: true
I20260129 03:00:20.218221 20510 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e7d83d4ccf684e76acb29a0dc844e1b8" candidate_uuid: "197b880fa39d432a9b472845876bfaf0" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "974b802141bd4f13beffef2aa03f88f9"
I20260129 03:00:20.254379 21135 raft_consensus.cc:2393] T a644c02b16af47679824ccd060ac10fd P 7a9daf5d42a84615a6383cfec9a10608 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 974b802141bd4f13beffef2aa03f88f9 in current term 6: Already voted for candidate 7a9daf5d42a84615a6383cfec9a10608 in this term.
I20260129 03:00:20.221063 20512 raft_consensus.cc:1240] T 00e79d8f25a34676b1d866eb2068a2b2 P 974b802141bd4f13beffef2aa03f88f9 [term 9 FOLLOWER]: Rejecting Update request from peer 197b880fa39d432a9b472845876bfaf0 for earlier term 8. Current term is 9. Ops: []
I20260129 03:00:20.216886 20582 raft_consensus.cc:2468] T a0fcc58d6fb6473cafbd9d34d5195269 P 1c50de89a5a845668696dcb28c3badd6 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 974b802141bd4f13beffef2aa03f88f9 in term 6.
I20260129 03:00:20.220214 20508 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a8479d1fe0bd41e5946186e3a6ca8c7a" candidate_uuid: "197b880fa39d432a9b472845876bfaf0" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "974b802141bd4f13beffef2aa03f88f9"
I20260129 03:00:20.218717 20515 raft_consensus.cc:1240] T 7a9a69cd82724b1faed1d10aa091fc4a P 974b802141bd4f13beffef2aa03f88f9 [term 8 FOLLOWER]: Rejecting Update request from peer 197b880fa39d432a9b472845876bfaf0 for earlier term 7. Current term is 8. Ops: []
I20260129 03:00:20.213829 21099 leader_election.cc:304] T 87d5fe48a44a408aaf03e894a67ecb2f P 7a9daf5d42a84615a6383cfec9a10608 [CANDIDATE]: Term 8 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7a9daf5d42a84615a6383cfec9a10608; no voters: 197b880fa39d432a9b472845876bfaf0, 974b802141bd4f13beffef2aa03f88f9
I20260129 03:00:20.257618 20587 raft_consensus.cc:1275] T cb0b3f259ebc4310927338a8740e87b8 P 1c50de89a5a845668696dcb28c3badd6 [term 9 FOLLOWER]: Refusing update from remote peer 7a9daf5d42a84615a6383cfec9a10608: Log matching property violated. Preceding OpId in replica: term: 7 index: 8. Preceding OpId from leader: term: 9 index: 9. (index mismatch)
I20260129 03:00:20.260143 20583 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "75f45538dc704d9a8ec4ff8ad9f3ea5b" candidate_uuid: "7a9daf5d42a84615a6383cfec9a10608" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "1c50de89a5a845668696dcb28c3badd6" is_pre_election: true
I20260129 03:00:20.219394 20511 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "17d6431cc77a491e8464d68db4081a51" candidate_uuid: "197b880fa39d432a9b472845876bfaf0" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "974b802141bd4f13beffef2aa03f88f9"
I20260129 03:00:20.218945 20516 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "76f713c3331540d2a5b76164103e9eb1" candidate_uuid: "197b880fa39d432a9b472845876bfaf0" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "974b802141bd4f13beffef2aa03f88f9"
I20260129 03:00:20.244777 20584 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "dc2f4cb752cf4d59b5f6bc689f5d8104" candidate_uuid: "7a9daf5d42a84615a6383cfec9a10608" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "1c50de89a5a845668696dcb28c3badd6" is_pre_election: true
I20260129 03:00:20.266243 20590 raft_consensus.cc:2468] T 3906b080a5b34a74b8b64128635e8997 P 1c50de89a5a845668696dcb28c3badd6 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 974b802141bd4f13beffef2aa03f88f9 in term 6.
W20260129 03:00:20.277419 20553 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260129 03:00:20.277486 20585 raft_consensus.cc:2468] T 926a250fa5834e88b96ccecbefbf5086 P 1c50de89a5a845668696dcb28c3badd6 [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 974b802141bd4f13beffef2aa03f88f9 in term 9.
I20260129 03:00:20.211599 23403 raft_consensus.cc:2749] T 0340229d521b432580ae926f6062e3f7 P 197b880fa39d432a9b472845876bfaf0 [term 9 FOLLOWER]: Leader election lost for term 8. Reason: could not achieve majority
I20260129 03:00:20.279345 20407 leader_election.cc:400] T 3906b080a5b34a74b8b64128635e8997 P 197b880fa39d432a9b472845876bfaf0 [CANDIDATE]: Term 5 election: Vote denied by peer 974b802141bd4f13beffef2aa03f88f9 (127.18.211.2:34517) with higher term. Message: Invalid argument: T 3906b080a5b34a74b8b64128635e8997 P 974b802141bd4f13beffef2aa03f88f9 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 197b880fa39d432a9b472845876bfaf0 for earlier term 5. Current term is 6.
I20260129 03:00:20.286121 20407 leader_election.cc:403] T 3906b080a5b34a74b8b64128635e8997 P 197b880fa39d432a9b472845876bfaf0 [CANDIDATE]: Term 5 election: Cancelling election due to peer responding with higher term
W20260129 03:00:20.285908 20479 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.211.1:42027, user_credentials={real_user=slave}} blocked reactor thread for 97415us
W20260129 03:00:20.279541 21099 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.211.1:42027, user_credentials={real_user=slave}} blocked reactor thread for 65785.4us
I20260129 03:00:20.140727 23758 raft_consensus.cc:2804] T cb4727386a3e48aca5d71cc6c611c197 P 974b802141bd4f13beffef2aa03f88f9 [term 5 FOLLOWER]: Leader pre-election won for term 6
W20260129 03:00:20.291673 20554 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.211.4:40439, user_credentials={real_user=slave}} blocked reactor thread for 204424us
I20260129 03:00:20.298151 20585 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a644c02b16af47679824ccd060ac10fd" candidate_uuid: "974b802141bd4f13beffef2aa03f88f9" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "1c50de89a5a845668696dcb28c3badd6" is_pre_election: true
I20260129 03:00:20.299081 20585 raft_consensus.cc:2468] T a644c02b16af47679824ccd060ac10fd P 1c50de89a5a845668696dcb28c3badd6 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 974b802141bd4f13beffef2aa03f88f9 in term 6.
W20260129 03:00:20.299253 20556 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.211.2:34517, user_credentials={real_user=slave}} blocked reactor thread for 100262us
I20260129 03:00:20.260205 23777 raft_consensus.cc:697] T a1a39dcd328e4268ae41b988c66b4a75 P 974b802141bd4f13beffef2aa03f88f9 [term 7 LEADER]: Becoming Leader. State: Replica: 974b802141bd4f13beffef2aa03f88f9, State: Running, Role: LEADER
I20260129 03:00:20.212119 23695 leader_election.cc:290] T a644c02b16af47679824ccd060ac10fd P 974b802141bd4f13beffef2aa03f88f9 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3:37735), 7a9daf5d42a84615a6383cfec9a10608 (127.18.211.4:40439)
I20260129 03:00:20.276799 23680 raft_consensus.cc:2749] T add8d4923d1a489e86882c3a25692b95 P 1c50de89a5a845668696dcb28c3badd6 [term 9 FOLLOWER]: Leader election lost for term 8. Reason: could not achieve majority
I20260129 03:00:20.380725 20582 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c2633a6fc34c44579de348d208b0c01e" candidate_uuid: "974b802141bd4f13beffef2aa03f88f9" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "1c50de89a5a845668696dcb28c3badd6" is_pre_election: true
I20260129 03:00:20.381333 20441 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "18f30ebaa4174fd28c958e72d0cc7e24" candidate_uuid: "1c50de89a5a845668696dcb28c3badd6" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "197b880fa39d432a9b472845876bfaf0" is_pre_election: true
I20260129 03:00:20.487284 20441 raft_consensus.cc:2468] T 18f30ebaa4174fd28c958e72d0cc7e24 P 197b880fa39d432a9b472845876bfaf0 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1c50de89a5a845668696dcb28c3badd6 in term 8.
W20260129 03:00:20.413643 20406 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260129 03:00:20.488368 20479 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.211.1:42027, user_credentials={real_user=slave}} blocked reactor thread for 201562us
I20260129 03:00:20.291842 23758 raft_consensus.cc:493] T cb4727386a3e48aca5d71cc6c611c197 P 974b802141bd4f13beffef2aa03f88f9 [term 5 FOLLOWER]: Starting leader election (detected failure of leader 1c50de89a5a845668696dcb28c3badd6)
I20260129 03:00:20.489500 23758 raft_consensus.cc:3060] T cb4727386a3e48aca5d71cc6c611c197 P 974b802141bd4f13beffef2aa03f88f9 [term 5 FOLLOWER]: Advancing to term 6
I20260129 03:00:20.489949 23695 consensus_queue.cc:1059] T a8479d1fe0bd41e5946186e3a6ca8c7a P 974b802141bd4f13beffef2aa03f88f9 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 }, Status: INVALID_TERM, Last received: 5.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260129 03:00:20.491587 23646 consensus_queue.cc:1059] T 17d6431cc77a491e8464d68db4081a51 P 974b802141bd4f13beffef2aa03f88f9 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 }, Status: INVALID_TERM, Last received: 0.0, Next index: 3, Last known committed idx: 1, Time since last communication: 0.000s
I20260129 03:00:20.492337 23646 consensus_queue.cc:1243] T 17d6431cc77a491e8464d68db4081a51 P 974b802141bd4f13beffef2aa03f88f9 [LEADER]: Peer 197b880fa39d432a9b472845876bfaf0 log is divergent from this leader: its last log entry 5.3 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 1
I20260129 03:00:20.495271 23730 consensus_queue.cc:1059] T 00e79d8f25a34676b1d866eb2068a2b2 P 197b880fa39d432a9b472845876bfaf0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 }, Status: INVALID_TERM, Last received: 8.9, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20260129 03:00:20.496302 23730 consensus_queue.cc:1059] T 7a9a69cd82724b1faed1d10aa091fc4a P 197b880fa39d432a9b472845876bfaf0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 }, Status: INVALID_TERM, Last received: 7.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260129 03:00:20.496745 23671 raft_consensus.cc:3055] T 17d6431cc77a491e8464d68db4081a51 P 974b802141bd4f13beffef2aa03f88f9 [term 6 LEADER]: Stepping down as leader of term 6
I20260129 03:00:20.497090 23403 raft_consensus.cc:3055] T 00e79d8f25a34676b1d866eb2068a2b2 P 197b880fa39d432a9b472845876bfaf0 [term 8 LEADER]: Stepping down as leader of term 8
W20260129 03:00:20.497254 20480 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.211.3:37735, user_credentials={real_user=slave}} blocked reactor thread for 201394us
I20260129 03:00:20.497689 23403 raft_consensus.cc:740] T 00e79d8f25a34676b1d866eb2068a2b2 P 197b880fa39d432a9b472845876bfaf0 [term 8 LEADER]: Becoming Follower/Learner. State: Replica: 197b880fa39d432a9b472845876bfaf0, State: Running, Role: LEADER
I20260129 03:00:20.498382 20480 leader_election.cc:304] T cedce331e1584ac8b636c9b8a72b2ba0 P 974b802141bd4f13beffef2aa03f88f9 [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1c50de89a5a845668696dcb28c3badd6, 974b802141bd4f13beffef2aa03f88f9; no voters: 197b880fa39d432a9b472845876bfaf0
I20260129 03:00:20.498772 23403 consensus_queue.cc:260] T 00e79d8f25a34676b1d866eb2068a2b2 P 197b880fa39d432a9b472845876bfaf0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 8.9, Last appended by leader: 9, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } }
I20260129 03:00:20.500108 23427 raft_consensus.cc:2804] T cedce331e1584ac8b636c9b8a72b2ba0 P 974b802141bd4f13beffef2aa03f88f9 [term 9 FOLLOWER]: Leader election won for term 9
I20260129 03:00:20.500615 23403 raft_consensus.cc:3060] T 00e79d8f25a34676b1d866eb2068a2b2 P 197b880fa39d432a9b472845876bfaf0 [term 8 FOLLOWER]: Advancing to term 9
I20260129 03:00:20.303977 21130 raft_consensus.cc:1240] T 75b0b6e82ddc4f92b1fe1d94f9a0b73b P 7a9daf5d42a84615a6383cfec9a10608 [term 6 FOLLOWER]: Rejecting Update request from peer 1c50de89a5a845668696dcb28c3badd6 for earlier term 5. Current term is 6. Ops: [5.8-5.8]
I20260129 03:00:20.301990 23725 consensus_queue.cc:1048] T add8d4923d1a489e86882c3a25692b95 P 7a9daf5d42a84615a6383cfec9a10608 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 5, Time since last communication: 0.115s
I20260129 03:00:20.285398 23674 raft_consensus.cc:515] T fcddab6bca684adf8ea032289a00f90e P 197b880fa39d432a9b472845876bfaf0 [term 11 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 03:00:20.297365 23514 consensus_queue.cc:1059] T e7d83d4ccf684e76acb29a0dc844e1b8 P 974b802141bd4f13beffef2aa03f88f9 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 }, Status: INVALID_TERM, Last received: 7.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.120s
W20260129 03:00:20.475898 21099 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.211.1:42027, user_credentials={real_user=slave}} blocked reactor thread for 187234us
W20260129 03:00:20.474926 21100 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.211.3:37735, user_credentials={real_user=slave}} blocked reactor thread for 173440us
I20260129 03:00:20.503651 21099 leader_election.cc:304] T 2ac659d732314e1288849f08fe5e074e P 7a9daf5d42a84615a6383cfec9a10608 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7a9daf5d42a84615a6383cfec9a10608; no voters: 197b880fa39d432a9b472845876bfaf0, 974b802141bd4f13beffef2aa03f88f9
I20260129 03:00:20.504315 23674 leader_election.cc:290] T fcddab6bca684adf8ea032289a00f90e P 197b880fa39d432a9b472845876bfaf0 [CANDIDATE]: Term 11 election: Requested vote from peers 974b802141bd4f13beffef2aa03f88f9 (127.18.211.2:34517), 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3:37735)
I20260129 03:00:20.475150 23753 consensus_queue.cc:1048] T cb0b3f259ebc4310927338a8740e87b8 P 7a9daf5d42a84615a6383cfec9a10608 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20260129 03:00:20.504796 21099 leader_election.cc:304] T 0678b8b468bd43cdbc3e485900844821 P 7a9daf5d42a84615a6383cfec9a10608 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7a9daf5d42a84615a6383cfec9a10608; no voters: 197b880fa39d432a9b472845876bfaf0, 974b802141bd4f13beffef2aa03f88f9
I20260129 03:00:20.505934 21099 leader_election.cc:304] T dc2f4cb752cf4d59b5f6bc689f5d8104 P 7a9daf5d42a84615a6383cfec9a10608 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7a9daf5d42a84615a6383cfec9a10608; no voters: 197b880fa39d432a9b472845876bfaf0, 1c50de89a5a845668696dcb28c3badd6
I20260129 03:00:20.506048 23566 raft_consensus.cc:2749] T 2ac659d732314e1288849f08fe5e074e P 7a9daf5d42a84615a6383cfec9a10608 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260129 03:00:20.497126 23671 raft_consensus.cc:740] T 17d6431cc77a491e8464d68db4081a51 P 974b802141bd4f13beffef2aa03f88f9 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 974b802141bd4f13beffef2aa03f88f9, State: Running, Role: LEADER
I20260129 03:00:20.507004 21099 leader_election.cc:304] T febffe4a8b394368b69b1d164308fb9a P 7a9daf5d42a84615a6383cfec9a10608 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7a9daf5d42a84615a6383cfec9a10608; no voters: 197b880fa39d432a9b472845876bfaf0, 974b802141bd4f13beffef2aa03f88f9
I20260129 03:00:20.507225 23566 raft_consensus.cc:2749] T 0678b8b468bd43cdbc3e485900844821 P 7a9daf5d42a84615a6383cfec9a10608 [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20260129 03:00:20.507812 23753 raft_consensus.cc:2749] T dc2f4cb752cf4d59b5f6bc689f5d8104 P 7a9daf5d42a84615a6383cfec9a10608 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260129 03:00:20.507936 23566 raft_consensus.cc:2749] T febffe4a8b394368b69b1d164308fb9a P 7a9daf5d42a84615a6383cfec9a10608 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260129 03:00:20.508831 23709 raft_consensus.cc:3055] T a8479d1fe0bd41e5946186e3a6ca8c7a P 974b802141bd4f13beffef2aa03f88f9 [term 5 LEADER]: Stepping down as leader of term 5
I20260129 03:00:20.509279 23709 raft_consensus.cc:740] T a8479d1fe0bd41e5946186e3a6ca8c7a P 974b802141bd4f13beffef2aa03f88f9 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 974b802141bd4f13beffef2aa03f88f9, State: Running, Role: LEADER
I20260129 03:00:20.510155 20480 leader_election.cc:304] T 7a9a69cd82724b1faed1d10aa091fc4a P 974b802141bd4f13beffef2aa03f88f9 [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1c50de89a5a845668696dcb28c3badd6, 974b802141bd4f13beffef2aa03f88f9; no voters: 197b880fa39d432a9b472845876bfaf0
I20260129 03:00:20.510234 23709 consensus_queue.cc:260] T a8479d1fe0bd41e5946186e3a6ca8c7a P 974b802141bd4f13beffef2aa03f88f9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.3, Last appended by leader: 3, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 03:00:20.511363 23709 raft_consensus.cc:3060] T a8479d1fe0bd41e5946186e3a6ca8c7a P 974b802141bd4f13beffef2aa03f88f9 [term 5 FOLLOWER]: Advancing to term 6
I20260129 03:00:20.511940 20480 leader_election.cc:304] T a0fcc58d6fb6473cafbd9d34d5195269 P 974b802141bd4f13beffef2aa03f88f9 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1c50de89a5a845668696dcb28c3badd6, 974b802141bd4f13beffef2aa03f88f9; no voters: 197b880fa39d432a9b472845876bfaf0
I20260129 03:00:20.497296 23730 raft_consensus.cc:3055] T 7a9a69cd82724b1faed1d10aa091fc4a P 197b880fa39d432a9b472845876bfaf0 [term 7 LEADER]: Stepping down as leader of term 7
I20260129 03:00:20.520309 23646 raft_consensus.cc:2804] T a0fcc58d6fb6473cafbd9d34d5195269 P 974b802141bd4f13beffef2aa03f88f9 [term 6 FOLLOWER]: Leader election won for term 6
W20260129 03:00:20.480949 20405 leader_election.cc:336] T 36ff33be666a4c82954731691dbcf327 P 197b880fa39d432a9b472845876bfaf0 [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3:37735): Timed out: RequestConsensusVote RPC to 127.18.211.3:37735 timed out after 1.735s (SENT)
I20260129 03:00:20.521180 23646 raft_consensus.cc:697] T a0fcc58d6fb6473cafbd9d34d5195269 P 974b802141bd4f13beffef2aa03f88f9 [term 6 LEADER]: Becoming Leader. State: Replica: 974b802141bd4f13beffef2aa03f88f9, State: Running, Role: LEADER
W20260129 03:00:20.521730 20405 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.18.211.3:37735 timed out after 1.859s (SENT) [suppressed 14 similar messages]
I20260129 03:00:20.522059 23758 raft_consensus.cc:515] T cb4727386a3e48aca5d71cc6c611c197 P 974b802141bd4f13beffef2aa03f88f9 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 03:00:20.522819 23674 raft_consensus.cc:493] T 4b1876f9a70b49edb3aa74105e66d3b3 P 197b880fa39d432a9b472845876bfaf0 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260129 03:00:20.523363 23674 raft_consensus.cc:515] T 4b1876f9a70b49edb3aa74105e66d3b3 P 197b880fa39d432a9b472845876bfaf0 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 03:00:20.525445 23674 leader_election.cc:290] T 4b1876f9a70b49edb3aa74105e66d3b3 P 197b880fa39d432a9b472845876bfaf0 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 974b802141bd4f13beffef2aa03f88f9 (127.18.211.2:34517), 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3:37735)
I20260129 03:00:20.525738 20441 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "cb4727386a3e48aca5d71cc6c611c197" candidate_uuid: "974b802141bd4f13beffef2aa03f88f9" candidate_term: 6 candidate_status { last_received { term: 5 index: 2 } } ignore_live_leader: false dest_uuid: "197b880fa39d432a9b472845876bfaf0"
I20260129 03:00:20.526705 20441 raft_consensus.cc:2393] T cb4727386a3e48aca5d71cc6c611c197 P 197b880fa39d432a9b472845876bfaf0 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 974b802141bd4f13beffef2aa03f88f9 in current term 6: Already voted for candidate 197b880fa39d432a9b472845876bfaf0 in this term.
I20260129 03:00:20.298691 23365 raft_consensus.cc:2749] T 87d5fe48a44a408aaf03e894a67ecb2f P 7a9daf5d42a84615a6383cfec9a10608 [term 8 FOLLOWER]: Leader election lost for term 8. Reason: could not achieve majority
I20260129 03:00:20.300362 23304 raft_consensus.cc:493] T 36ff33be666a4c82954731691dbcf327 P 1c50de89a5a845668696dcb28c3badd6 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260129 03:00:20.529450 23304 raft_consensus.cc:515] T 36ff33be666a4c82954731691dbcf327 P 1c50de89a5a845668696dcb28c3badd6 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 03:00:20.531486 23304 leader_election.cc:290] T 36ff33be666a4c82954731691dbcf327 P 1c50de89a5a845668696dcb28c3badd6 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 197b880fa39d432a9b472845876bfaf0 (127.18.211.1:42027), 974b802141bd4f13beffef2aa03f88f9 (127.18.211.2:34517)
I20260129 03:00:20.298246 23701 raft_consensus.cc:2749] T 3906b080a5b34a74b8b64128635e8997 P 197b880fa39d432a9b472845876bfaf0 [term 6 FOLLOWER]: Leader election lost for term 5. Reason: Vote denied by peer 974b802141bd4f13beffef2aa03f88f9 (127.18.211.2:34517) with higher term. Message: Invalid argument: T 3906b080a5b34a74b8b64128635e8997 P 974b802141bd4f13beffef2aa03f88f9 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 197b880fa39d432a9b472845876bfaf0 for earlier term 5. Current term is 6.
I20260129 03:00:20.486274 23747 raft_consensus.cc:493] T 75be4bd6bd45474795fd472eff2953ac P 1c50de89a5a845668696dcb28c3badd6 [term 10 FOLLOWER]: Starting pre-election (detected failure of leader 197b880fa39d432a9b472845876bfaf0)
W20260129 03:00:20.489107 20482 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260129 03:00:20.532321 21128 raft_consensus.cc:1275] T c934d31efd3f4e5195605ddc354d14e1 P 7a9daf5d42a84615a6383cfec9a10608 [term 7 FOLLOWER]: Refusing update from remote peer 197b880fa39d432a9b472845876bfaf0: Log matching property violated. Preceding OpId in replica: term: 6 index: 8. Preceding OpId from leader: term: 7 index: 8. (term mismatch)
I20260129 03:00:20.532761 21128 pending_rounds.cc:85] T c934d31efd3f4e5195605ddc354d14e1 P 7a9daf5d42a84615a6383cfec9a10608: Aborting all ops after (but not including) 7
I20260129 03:00:20.532991 21128 pending_rounds.cc:107] T c934d31efd3f4e5195605ddc354d14e1 P 7a9daf5d42a84615a6383cfec9a10608: Aborting uncommitted NO_OP operation due to leader change: 6.8
I20260129 03:00:20.533214 21128 raft_consensus.cc:2889] T c934d31efd3f4e5195605ddc354d14e1 P 7a9daf5d42a84615a6383cfec9a10608 [term 7 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260129 03:00:20.500618 23427 raft_consensus.cc:697] T cedce331e1584ac8b636c9b8a72b2ba0 P 974b802141bd4f13beffef2aa03f88f9 [term 9 LEADER]: Becoming Leader. State: Replica: 974b802141bd4f13beffef2aa03f88f9, State: Running, Role: LEADER
I20260129 03:00:20.522241 23646 consensus_queue.cc:237] T a0fcc58d6fb6473cafbd9d34d5195269 P 974b802141bd4f13beffef2aa03f88f9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 5.4, Last appended by leader: 4, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 03:00:20.537663 23427 consensus_queue.cc:237] T cedce331e1584ac8b636c9b8a72b2ba0 P 974b802141bd4f13beffef2aa03f88f9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 8.5, Last appended by leader: 5, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 03:00:20.503755 23693 raft_consensus.cc:3055] T e7d83d4ccf684e76acb29a0dc844e1b8 P 974b802141bd4f13beffef2aa03f88f9 [term 7 LEADER]: Stepping down as leader of term 7
I20260129 03:00:20.544281 23693 raft_consensus.cc:740] T e7d83d4ccf684e76acb29a0dc844e1b8 P 974b802141bd4f13beffef2aa03f88f9 [term 7 LEADER]: Becoming Follower/Learner. State: Replica: 974b802141bd4f13beffef2aa03f88f9, State: Running, Role: LEADER
I20260129 03:00:20.545315 23693 consensus_queue.cc:260] T e7d83d4ccf684e76acb29a0dc844e1b8 P 974b802141bd4f13beffef2aa03f88f9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 7.4, Last appended by leader: 4, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 03:00:20.551550 20511 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c891d2bff4ce4568861bbc992e91fbb0" candidate_uuid: "197b880fa39d432a9b472845876bfaf0" candidate_term: 10 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "974b802141bd4f13beffef2aa03f88f9" is_pre_election: true
I20260129 03:00:20.482383 20516 raft_consensus.cc:2468] T 76f713c3331540d2a5b76164103e9eb1 P 974b802141bd4f13beffef2aa03f88f9 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 197b880fa39d432a9b472845876bfaf0 in term 7.
I20260129 03:00:20.512548 23710 raft_consensus.cc:2804] T 7a9a69cd82724b1faed1d10aa091fc4a P 974b802141bd4f13beffef2aa03f88f9 [term 8 FOLLOWER]: Leader election won for term 8
I20260129 03:00:20.515570 20434 raft_consensus.cc:1275] T 368953c9102643b2a21f99af36b51905 P 197b880fa39d432a9b472845876bfaf0 [term 10 FOLLOWER]: Refusing update from remote peer 974b802141bd4f13beffef2aa03f88f9: Log matching property violated. Preceding OpId in replica: term: 6 index: 4. Preceding OpId from leader: term: 10 index: 6. (index mismatch)
I20260129 03:00:20.477298 23777 consensus_queue.cc:237] T a1a39dcd328e4268ae41b988c66b4a75 P 974b802141bd4f13beffef2aa03f88f9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 5.4, Last appended by leader: 4, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } } peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } }
I20260129 03:00:20.515846 20437 raft_consensus.cc:1275] T 4f8f3a38f6e24ba4857875181f447442 P 197b880fa39d432a9b472845876bfaf0 [term 8 FOLLOWER]: Refusing update from remote peer 974b802141bd4f13beffef2aa03f88f9: Log matching property violated. Preceding OpId in replica: term: 6 index: 4. Preceding OpId from leader: term: 8 index: 5. (index mismatch)
I20260129 03:00:20.516685 20433 raft_consensus.cc:1240] T fcddab6bca684adf8ea032289a00f90e P 197b880fa39d432a9b472845876bfaf0 [term 11 FOLLOWER]: Rejecting Update request from peer 974b802141bd4f13beffef2aa03f88f9 for earlier term 10. Current term is 11. Ops: []
I20260129 03:00:20.516839 20438 raft_consensus.cc:1275] T 724a236524cb4e7ba3428880681d6cc2 P 197b880fa39d432a9b472845876bfaf0 [term 7 FOLLOWER]: Refusing update from remote peer 974b802141bd4f13beffef2aa03f88f9: Log matching property violated. Preceding OpId in replica: term: 5 index: 4. Preceding OpId from leader: term: 7 index: 5. (index mismatch)
I20260129 03:00:20.520512 23730 raft_consensus.cc:740] T 7a9a69cd82724b1faed1d10aa091fc4a P 197b880fa39d432a9b472845876bfaf0 [term 7 LEADER]: Becoming Follower/Learner. State: Replica: 197b880fa39d432a9b472845876bfaf0, State: Running, Role: LEADER
I20260129 03:00:20.552316 20511 raft_consensus.cc:2468] T c891d2bff4ce4568861bbc992e91fbb0 P 974b802141bd4f13beffef2aa03f88f9 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 197b880fa39d432a9b472845876bfaf0 in term 9.
I20260129 03:00:20.507511 23671 consensus_queue.cc:260] T 17d6431cc77a491e8464d68db4081a51 P 974b802141bd4f13beffef2aa03f88f9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 6.3, Last appended by leader: 3, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 03:00:20.554081 20480 leader_election.cc:304] T 926a250fa5834e88b96ccecbefbf5086 P 974b802141bd4f13beffef2aa03f88f9 [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1c50de89a5a845668696dcb28c3badd6, 974b802141bd4f13beffef2aa03f88f9; no voters: 197b880fa39d432a9b472845876bfaf0
W20260129 03:00:20.561664 20553 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.211.1:42027, user_credentials={real_user=slave}} blocked reactor thread for 270213us
W20260129 03:00:20.477720 20556 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.211.2:34517, user_credentials={real_user=slave}} blocked reactor thread for 177550us
I20260129 03:00:20.585803 20556 leader_election.cc:304] T e7d83d4ccf684e76acb29a0dc844e1b8 P 1c50de89a5a845668696dcb28c3badd6 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1c50de89a5a845668696dcb28c3badd6; no voters: 197b880fa39d432a9b472845876bfaf0, 974b802141bd4f13beffef2aa03f88f9
I20260129 03:00:20.563226 21133 raft_consensus.cc:1240] T 87d5fe48a44a408aaf03e894a67ecb2f P 7a9daf5d42a84615a6383cfec9a10608 [term 8 FOLLOWER]: Rejecting Update request from peer 197b880fa39d432a9b472845876bfaf0 for earlier term 7. Current term is 8. Ops: []
I20260129 03:00:20.481117 23740 leader_election.cc:290] T 75b0b6e82ddc4f92b1fe1d94f9a0b73b P 7a9daf5d42a84615a6383cfec9a10608 [CANDIDATE]: Term 6 election: Requested vote from peers 974b802141bd4f13beffef2aa03f88f9 (127.18.211.2:34517), 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3:37735)
I20260129 03:00:20.474862 20335 catalog_manager.cc:5654] T 0678b8b468bd43cdbc3e485900844821 P 974b802141bd4f13beffef2aa03f88f9 reported cstate change: term changed from 4 to 9, leader changed from 7a9daf5d42a84615a6383cfec9a10608 (127.18.211.4) to 974b802141bd4f13beffef2aa03f88f9 (127.18.211.2). New cstate: current_term: 9 leader_uuid: "974b802141bd4f13beffef2aa03f88f9" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20260129 03:00:20.569073 20512 raft_consensus.cc:1275] T c934d31efd3f4e5195605ddc354d14e1 P 974b802141bd4f13beffef2aa03f88f9 [term 7 FOLLOWER]: Refusing update from remote peer 197b880fa39d432a9b472845876bfaf0: Log matching property violated. Preceding OpId in replica: term: 5 index: 7. Preceding OpId from leader: term: 7 index: 8. (index mismatch)
I20260129 03:00:20.583696 20440 raft_consensus.cc:1275] T 0340229d521b432580ae926f6062e3f7 P 197b880fa39d432a9b472845876bfaf0 [term 9 FOLLOWER]: Refusing update from remote peer 974b802141bd4f13beffef2aa03f88f9: Log matching property violated. Preceding OpId in replica: term: 7 index: 5. Preceding OpId from leader: term: 9 index: 6. (index mismatch)
I20260129 03:00:20.590402 20510 raft_consensus.cc:1240] T 926a250fa5834e88b96ccecbefbf5086 P 974b802141bd4f13beffef2aa03f88f9 [term 9 FOLLOWER]: Rejecting Update request from peer 197b880fa39d432a9b472845876bfaf0 for earlier term 8. Current term is 9. Ops: []
I20260129 03:00:20.601222 20335 catalog_manager.cc:5654] T a609ac02fae84725ab743f95f51d33ce P 974b802141bd4f13beffef2aa03f88f9 reported cstate change: term changed from 6 to 9, leader changed from 974b802141bd4f13beffef2aa03f88f9 (127.18.211.2) to 197b880fa39d432a9b472845876bfaf0 (127.18.211.1). New cstate: current_term: 9 leader_uuid: "197b880fa39d432a9b472845876bfaf0" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } } }
I20260129 03:00:20.604542 20407 leader_election.cc:304] T c891d2bff4ce4568861bbc992e91fbb0 P 197b880fa39d432a9b472845876bfaf0 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 197b880fa39d432a9b472845876bfaf0, 974b802141bd4f13beffef2aa03f88f9; no voters:
I20260129 03:00:20.566668 23165 raft_consensus.cc:2804] T 75f45538dc704d9a8ec4ff8ad9f3ea5b P 7a9daf5d42a84615a6383cfec9a10608 [term 5 FOLLOWER]: Leader pre-election won for term 6
W20260129 03:00:20.608609 20480 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.211.3:37735, user_credentials={real_user=slave}} blocked reactor thread for 89035.5us
I20260129 03:00:20.609678 20480 leader_election.cc:304] T a644c02b16af47679824ccd060ac10fd P 974b802141bd4f13beffef2aa03f88f9 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1c50de89a5a845668696dcb28c3badd6, 974b802141bd4f13beffef2aa03f88f9; no voters: 7a9daf5d42a84615a6383cfec9a10608
I20260129 03:00:20.604758 20335 catalog_manager.cc:5654] T a4c1226b987b468fa84dfc6ba0e2dbe1 P 974b802141bd4f13beffef2aa03f88f9 reported cstate change: term changed from 2 to 4. New cstate: current_term: 4 leader_uuid: "197b880fa39d432a9b472845876bfaf0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } } }
I20260129 03:00:20.615166 20335 catalog_manager.cc:5654] T a0fcc58d6fb6473cafbd9d34d5195269 P 974b802141bd4f13beffef2aa03f88f9 reported cstate change: term changed from 3 to 5, leader changed from 974b802141bd4f13beffef2aa03f88f9 (127.18.211.2) to 197b880fa39d432a9b472845876bfaf0 (127.18.211.1). New cstate: current_term: 5 leader_uuid: "197b880fa39d432a9b472845876bfaf0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } } }
I20260129 03:00:20.618316 20514 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4b1876f9a70b49edb3aa74105e66d3b3" candidate_uuid: "197b880fa39d432a9b472845876bfaf0" candidate_term: 8 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "974b802141bd4f13beffef2aa03f88f9" is_pre_election: true
I20260129 03:00:20.581900 23365 raft_consensus.cc:493] T 2e58e9a5a8aa4cecafb464addb4f35ed P 7a9daf5d42a84615a6383cfec9a10608 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260129 03:00:20.627326 20556 leader_election.cc:304] T a8479d1fe0bd41e5946186e3a6ca8c7a P 1c50de89a5a845668696dcb28c3badd6 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1c50de89a5a845668696dcb28c3badd6; no voters: 197b880fa39d432a9b472845876bfaf0, 974b802141bd4f13beffef2aa03f88f9
I20260129 03:00:20.617476 20335 catalog_manager.cc:5654] T 76f713c3331540d2a5b76164103e9eb1 P 974b802141bd4f13beffef2aa03f88f9 reported cstate change: term changed from 5 to 6, leader changed from 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3) to 974b802141bd4f13beffef2aa03f88f9 (127.18.211.2). New cstate: current_term: 6 leader_uuid: "974b802141bd4f13beffef2aa03f88f9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } health_report { overall_health: UNKNOWN } } }
I20260129 03:00:20.619241 20514 raft_consensus.cc:2468] T 4b1876f9a70b49edb3aa74105e66d3b3 P 974b802141bd4f13beffef2aa03f88f9 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 197b880fa39d432a9b472845876bfaf0 in term 7.
I20260129 03:00:20.620157 20508 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fcddab6bca684adf8ea032289a00f90e" candidate_uuid: "197b880fa39d432a9b472845876bfaf0" candidate_term: 11 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "974b802141bd4f13beffef2aa03f88f9"
I20260129 03:00:20.621646 20553 leader_election.cc:304] T 18f30ebaa4174fd28c958e72d0cc7e24 P 1c50de89a5a845668696dcb28c3badd6 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 197b880fa39d432a9b472845876bfaf0, 1c50de89a5a845668696dcb28c3badd6; no voters:
I20260129 03:00:20.639808 23707 raft_consensus.cc:493] T e33a417cf9c54cf5af73a4ee3aac4d3a P 974b802141bd4f13beffef2aa03f88f9 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260129 03:00:20.602211 23262 consensus_queue.cc:1059] T 75b0b6e82ddc4f92b1fe1d94f9a0b73b P 1c50de89a5a845668696dcb28c3badd6 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false }, Status: INVALID_TERM, Last received: 4.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.039s
I20260129 03:00:20.576261 23602 consensus_queue.cc:1059] T a5a05fbcbbcf4916bbc01fd1a5b5e90d P 1c50de89a5a845668696dcb28c3badd6 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 }, Status: INVALID_TERM, Last received: 8.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.050s
I20260129 03:00:20.625082 21128 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "18f30ebaa4174fd28c958e72d0cc7e24" candidate_uuid: "1c50de89a5a845668696dcb28c3badd6" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "7a9daf5d42a84615a6383cfec9a10608" is_pre_election: true
I20260129 03:00:20.641948 23398 raft_consensus.cc:2749] T a8479d1fe0bd41e5946186e3a6ca8c7a P 1c50de89a5a845668696dcb28c3badd6 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260129 03:00:20.642971 23398 raft_consensus.cc:2804] T 18f30ebaa4174fd28c958e72d0cc7e24 P 1c50de89a5a845668696dcb28c3badd6 [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260129 03:00:20.643473 23398 raft_consensus.cc:493] T 18f30ebaa4174fd28c958e72d0cc7e24 P 1c50de89a5a845668696dcb28c3badd6 [term 8 FOLLOWER]: Starting leader election (detected failure of leader 7a9daf5d42a84615a6383cfec9a10608)
I20260129 03:00:20.644003 23398 raft_consensus.cc:3060] T 18f30ebaa4174fd28c958e72d0cc7e24 P 1c50de89a5a845668696dcb28c3badd6 [term 8 FOLLOWER]: Advancing to term 9
I20260129 03:00:20.643739 23365 raft_consensus.cc:515] T 2e58e9a5a8aa4cecafb464addb4f35ed P 7a9daf5d42a84615a6383cfec9a10608 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } }
I20260129 03:00:20.646580 23365 leader_election.cc:290] T 2e58e9a5a8aa4cecafb464addb4f35ed P 7a9daf5d42a84615a6383cfec9a10608 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 197b880fa39d432a9b472845876bfaf0 (127.18.211.1:42027), 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3:37735)
I20260129 03:00:20.619724 23691 consensus_queue.cc:1059] T fcddab6bca684adf8ea032289a00f90e P 974b802141bd4f13beffef2aa03f88f9 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 }, Status: INVALID_TERM, Last received: 10.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.029s
I20260129 03:00:20.648360 20582 tablet_service.cc:2066] Received LeaderStepDown RPC: tablet_id: "e33a417cf9c54cf5af73a4ee3aac4d3a"
dest_uuid: "1c50de89a5a845668696dcb28c3badd6"
mode: GRACEFUL
new_leader_uuid: "197b880fa39d432a9b472845876bfaf0"
from {username='slave'} at 127.0.0.1:54860
I20260129 03:00:20.648926 23691 raft_consensus.cc:2804] T a644c02b16af47679824ccd060ac10fd P 974b802141bd4f13beffef2aa03f88f9 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260129 03:00:20.649295 20582 raft_consensus.cc:606] T e33a417cf9c54cf5af73a4ee3aac4d3a P 1c50de89a5a845668696dcb28c3badd6 [term 8 LEADER]: Received request to transfer leadership to TS 197b880fa39d432a9b472845876bfaf0
I20260129 03:00:20.649509 23691 raft_consensus.cc:493] T a644c02b16af47679824ccd060ac10fd P 974b802141bd4f13beffef2aa03f88f9 [term 5 FOLLOWER]: Starting leader election (detected failure of leader 1c50de89a5a845668696dcb28c3badd6)
I20260129 03:00:20.587205 23710 raft_consensus.cc:697] T 7a9a69cd82724b1faed1d10aa091fc4a P 974b802141bd4f13beffef2aa03f88f9 [term 8 LEADER]: Becoming Leader. State: Replica: 974b802141bd4f13beffef2aa03f88f9, State: Running, Role: LEADER
I20260129 03:00:20.651906 20335 catalog_manager.cc:5654] T cedce331e1584ac8b636c9b8a72b2ba0 P 974b802141bd4f13beffef2aa03f88f9 reported cstate change: term changed from 4 to 8, leader changed from 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3) to 197b880fa39d432a9b472845876bfaf0 (127.18.211.1). New cstate: current_term: 8 leader_uuid: "197b880fa39d432a9b472845876bfaf0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } } }
W20260129 03:00:20.655405 20336 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:55376 (request call id 243) took 17627 ms (client timeout 15000 ms). Trace:
I20260129 03:00:20.658018 20438 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "36ff33be666a4c82954731691dbcf327" candidate_uuid: "1c50de89a5a845668696dcb28c3badd6" candidate_term: 6 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "197b880fa39d432a9b472845876bfaf0" is_pre_election: true
W20260129 03:00:20.673074 20554 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260129 03:00:20.672976 20335 catalog_manager.cc:5654] T a1a39dcd328e4268ae41b988c66b4a75 P 974b802141bd4f13beffef2aa03f88f9 reported cstate change: term changed from 2 to 5, leader changed from 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3) to 974b802141bd4f13beffef2aa03f88f9 (127.18.211.2). New cstate: current_term: 5 leader_uuid: "974b802141bd4f13beffef2aa03f88f9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } health_report { overall_health: HEALTHY } } }
I20260129 03:00:20.640661 23707 raft_consensus.cc:515] T e33a417cf9c54cf5af73a4ee3aac4d3a P 974b802141bd4f13beffef2aa03f88f9 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 03:00:20.620697 23732 raft_consensus.cc:2804] T 926a250fa5834e88b96ccecbefbf5086 P 974b802141bd4f13beffef2aa03f88f9 [term 9 FOLLOWER]: Leader election won for term 9
I20260129 03:00:20.572561 23669 consensus_queue.cc:1048] T 2e58e9a5a8aa4cecafb464addb4f35ed P 1c50de89a5a845668696dcb28c3badd6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.053s
I20260129 03:00:20.592650 23709 consensus_queue.cc:1048] T 368953c9102643b2a21f99af36b51905 P 974b802141bd4f13beffef2aa03f88f9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 4, Time since last communication: 0.072s
I20260129 03:00:20.579239 23747 raft_consensus.cc:515] T 75be4bd6bd45474795fd472eff2953ac P 1c50de89a5a845668696dcb28c3badd6 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 03:00:20.629554 23466 raft_consensus.cc:2804] T c891d2bff4ce4568861bbc992e91fbb0 P 197b880fa39d432a9b472845876bfaf0 [term 9 FOLLOWER]: Leader pre-election won for term 10
I20260129 03:00:20.650111 23691 raft_consensus.cc:3060] T a644c02b16af47679824ccd060ac10fd P 974b802141bd4f13beffef2aa03f88f9 [term 5 FOLLOWER]: Advancing to term 6
I20260129 03:00:20.671623 23514 raft_consensus.cc:3055] T fcddab6bca684adf8ea032289a00f90e P 974b802141bd4f13beffef2aa03f88f9 [term 10 LEADER]: Stepping down as leader of term 10
I20260129 03:00:20.592505 23680 raft_consensus.cc:2749] T e7d83d4ccf684e76acb29a0dc844e1b8 P 1c50de89a5a845668696dcb28c3badd6 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260129 03:00:20.615785 23694 consensus_queue.cc:1048] T 4f8f3a38f6e24ba4857875181f447442 P 974b802141bd4f13beffef2aa03f88f9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.046s
W20260129 03:00:20.670377 20555 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260129 03:00:20.673213 20438 raft_consensus.cc:2468] T 36ff33be666a4c82954731691dbcf327 P 197b880fa39d432a9b472845876bfaf0 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1c50de89a5a845668696dcb28c3badd6 in term 5.
I20260129 03:00:20.624775 23755 consensus_queue.cc:1048] T 724a236524cb4e7ba3428880681d6cc2 P 974b802141bd4f13beffef2aa03f88f9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.042s
I20260129 03:00:20.677860 20440 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2e58e9a5a8aa4cecafb464addb4f35ed" candidate_uuid: "7a9daf5d42a84615a6383cfec9a10608" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "197b880fa39d432a9b472845876bfaf0" is_pre_election: true
I20260129 03:00:20.678568 20335 catalog_manager.cc:5654] T b189f08e16a74a11980cc55d99359a0a P 974b802141bd4f13beffef2aa03f88f9 reported cstate change: term changed from 4 to 9. New cstate: current_term: 9 leader_uuid: "197b880fa39d432a9b472845876bfaf0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } } }
W20260129 03:00:20.682663 20405 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.211.3:37735, user_credentials={real_user=slave}} blocked reactor thread for 160956us
W20260129 03:00:20.731938 20405 leader_election.cc:336] T 368953c9102643b2a21f99af36b51905 P 197b880fa39d432a9b472845876bfaf0 [CANDIDATE]: Term 9 election: RPC error from VoteRequest() call to peer 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3:37735): Timed out: RequestConsensusVote RPC to 127.18.211.3:37735 timed out after 1.986s (SENT)
I20260129 03:00:20.732471 20405 leader_election.cc:304] T 368953c9102643b2a21f99af36b51905 P 197b880fa39d432a9b472845876bfaf0 [CANDIDATE]: Term 9 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 197b880fa39d432a9b472845876bfaf0; no voters: 1c50de89a5a845668696dcb28c3badd6, 974b802141bd4f13beffef2aa03f88f9
I20260129 03:00:20.652144 23668 consensus_queue.cc:1048] T e33a417cf9c54cf5af73a4ee3aac4d3a P 1c50de89a5a845668696dcb28c3badd6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.019s
W20260129 03:00:20.684209 20556 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.211.2:34517, user_credentials={real_user=slave}} blocked reactor thread for 56980.9us
I20260129 03:00:20.733585 20556 leader_election.cc:304] T 17d6431cc77a491e8464d68db4081a51 P 1c50de89a5a845668696dcb28c3badd6 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1c50de89a5a845668696dcb28c3badd6; no voters: 197b880fa39d432a9b472845876bfaf0, 974b802141bd4f13beffef2aa03f88f9
I20260129 03:00:20.630148 23758 leader_election.cc:290] T cb4727386a3e48aca5d71cc6c611c197 P 974b802141bd4f13beffef2aa03f88f9 [CANDIDATE]: Term 6 election: Requested vote from peers 197b880fa39d432a9b472845876bfaf0 (127.18.211.1:42027), 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3:37735)
I20260129 03:00:20.670131 23531 raft_consensus.cc:3055] T 75b0b6e82ddc4f92b1fe1d94f9a0b73b P 1c50de89a5a845668696dcb28c3badd6 [term 5 LEADER]: Stepping down as leader of term 5
I20260129 03:00:20.676970 23766 raft_consensus.cc:493] T fcddab6bca684adf8ea032289a00f90e P 1c50de89a5a845668696dcb28c3badd6 [term 10 FOLLOWER]: Starting pre-election (detected failure of leader 974b802141bd4f13beffef2aa03f88f9)
I20260129 03:00:20.706387 23707 leader_election.cc:290] T e33a417cf9c54cf5af73a4ee3aac4d3a P 974b802141bd4f13beffef2aa03f88f9 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 197b880fa39d432a9b472845876bfaf0 (127.18.211.1:42027), 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3:37735)
I20260129 03:00:20.692229 23732 raft_consensus.cc:697] T 926a250fa5834e88b96ccecbefbf5086 P 974b802141bd4f13beffef2aa03f88f9 [term 9 LEADER]: Becoming Leader. State: Replica: 974b802141bd4f13beffef2aa03f88f9, State: Running, Role: LEADER
I20260129 03:00:20.692807 23514 raft_consensus.cc:740] T fcddab6bca684adf8ea032289a00f90e P 974b802141bd4f13beffef2aa03f88f9 [term 10 LEADER]: Becoming Follower/Learner. State: Replica: 974b802141bd4f13beffef2aa03f88f9, State: Running, Role: LEADER
I20260129 03:00:20.717161 23709 consensus_queue.cc:1048] T 0340229d521b432580ae926f6062e3f7 P 974b802141bd4f13beffef2aa03f88f9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
I20260129 03:00:20.678535 23602 consensus_queue.cc:1243] T a5a05fbcbbcf4916bbc01fd1a5b5e90d P 1c50de89a5a845668696dcb28c3badd6 [LEADER]: Peer 197b880fa39d432a9b472845876bfaf0 log is divergent from this leader: its last log entry 9.7 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 6
I20260129 03:00:20.732095 20335 catalog_manager.cc:5654] T fcddab6bca684adf8ea032289a00f90e P 974b802141bd4f13beffef2aa03f88f9 reported cstate change: term changed from 4 to 10, leader changed from 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3) to 974b802141bd4f13beffef2aa03f88f9 (127.18.211.2). New cstate: current_term: 10 leader_uuid: "974b802141bd4f13beffef2aa03f88f9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } health_report { overall_health: UNKNOWN } } }
W20260129 03:00:20.672479 20336 rpcz_store.cc:269] 0129 03:00:03.028370 (+ 0us) service_pool.cc:168] Inserting onto call queue
0129 03:00:03.029444 (+ 1074us) service_pool.cc:225] Handling call
0129 03:00:20.655323 (+17625879us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0129 03:00:17.801044 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0129 03:00:17.801710 (+ 666us) write_op.cc:432] Acquiring schema lock in shared mode
0129 03:00:17.801761 (+ 51us) write_op.cc:435] Acquired schema lock
0129 03:00:17.801776 (+ 15us) tablet.cc:660] Decoding operations
0129 03:00:17.939423 (+137647us) write_op.cc:620] Acquiring the partition lock for write op
0129 03:00:17.977497 (+ 38074us) write_op.cc:641] Partition lock acquired for write op
0129 03:00:17.982062 (+ 4565us) tablet.cc:683] Acquiring locks for 14 operations
0129 03:00:18.196798 (+214736us) tablet.cc:699] Row locks acquired
0129 03:00:18.196824 (+ 26us) write_op.cc:260] PREPARE: finished
0129 03:00:18.197042 (+ 218us) write_op.cc:270] Start()
0129 03:00:18.197210 (+ 168us) write_op.cc:276] Timestamp: P: 1769655618196963 usec, L: 0
0129 03:00:18.197246 (+ 36us) op_driver.cc:348] REPLICATION: starting
0129 03:00:18.198059 (+ 813us) log.cc:844] Serialized 5524 byte log entry
0129 03:00:18.677197 (+479138us) op_driver.cc:464] REPLICATION: finished
0129 03:00:19.222034 (+544837us) write_op.cc:301] APPLY: starting
0129 03:00:19.250702 (+ 28668us) tablet.cc:1365] starting BulkCheckPresence
0129 03:00:19.335663 (+ 84961us) tablet.cc:1368] finished BulkCheckPresence
0129 03:00:19.471235 (+135572us) tablet.cc:1370] starting ApplyRowOperation cycle
0129 03:00:19.748983 (+277748us) tablet.cc:1381] finished ApplyRowOperation cycle
0129 03:00:19.749324 (+ 341us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=14
0129 03:00:19.749369 (+ 45us) write_op.cc:312] APPLY: finished
0129 03:00:19.749761 (+ 392us) log.cc:844] Serialized 101 byte log entry
0129 03:00:20.034682 (+284921us) spinlock_profiling.cc:243] Waited 18.9 ms on lock 0x7b440002304c. stack: 00007f8fe044e373 00007f8fe044e270 00007f8fdfda0574 00007f8fe9a5c6b4 00007f8fe9a5c649 00007f8fe02c08b1 00007f8fe038f005 00007f8fe038fa22 00007f8fe30d957b 00007f8fe30d4e4f 00007f8fe0462f0c 00007f8fe924b709 00007f8fe047f57b 00007f8fe047d81f 00007f8fe047fae2 00007f8fe30c75ab
0129 03:00:20.136432 (+101750us) write_op.cc:489] Releasing partition, row and schema locks
0129 03:00:20.202970 (+ 66538us) write_op.cc:454] Released schema lock
0129 03:00:20.203062 (+ 92us) write_op.cc:341] FINISH: Updating metrics
0129 03:00:20.203514 (+ 452us) write_op.cc:489] Releasing partition, row and schema locks
0129 03:00:20.203544 (+ 30us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":172314,"reported_tablets":41,"child_traces":[["op",{"apply.queue_time_us":429589,"apply.run_cpu_time_us":26037,"apply.run_wall_time_us":1035799,"num_ops":14,"prepare.queue_time_us":1026663,"prepare.run_cpu_time_us":65164,"prepare.run_wall_time_us":532693,"raft.queue_time_us":197414,"raft.run_cpu_time_us":104193,"raft.run_wall_time_us":512686,"replication_time_us":429058,"spinlock_wait_cycles":43489024,"thread_start_us":1256644,"threads_started":5,"wal-append.queue_time_us":564358,"wal-append.run_cpu_time_us":34751,"wal-append.run_wall_time_us":1151528}]]}
I20260129 03:00:20.685618 23466 raft_consensus.cc:493] T c891d2bff4ce4568861bbc992e91fbb0 P 197b880fa39d432a9b472845876bfaf0 [term 9 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260129 03:00:20.662909 23730 consensus_queue.cc:260] T 7a9a69cd82724b1faed1d10aa091fc4a P 197b880fa39d432a9b472845876bfaf0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 7.5, Last appended by leader: 5, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 03:00:20.758514 20335 catalog_manager.cc:5654] T 4f8f3a38f6e24ba4857875181f447442 P 974b802141bd4f13beffef2aa03f88f9 reported cstate change: term changed from 3 to 6. New cstate: current_term: 6 leader_uuid: "974b802141bd4f13beffef2aa03f88f9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } health_report { overall_health: HEALTHY } } }
I20260129 03:00:20.765468 20335 catalog_manager.cc:5654] T 36ff33be666a4c82954731691dbcf327 P 974b802141bd4f13beffef2aa03f88f9 reported cstate change: term changed from 1 to 3. New cstate: current_term: 3 leader_uuid: "974b802141bd4f13beffef2aa03f88f9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } health_report { overall_health: HEALTHY } } }
W20260129 03:00:20.768146 21100 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.211.3:37735, user_credentials={real_user=slave}} blocked reactor thread for 262633us
I20260129 03:00:20.738919 23744 raft_consensus.cc:2749] T 368953c9102643b2a21f99af36b51905 P 197b880fa39d432a9b472845876bfaf0 [term 10 FOLLOWER]: Leader election lost for term 9. Reason: could not achieve majority
I20260129 03:00:20.773850 20586 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2e58e9a5a8aa4cecafb464addb4f35ed" candidate_uuid: "7a9daf5d42a84615a6383cfec9a10608" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "1c50de89a5a845668696dcb28c3badd6" is_pre_election: true
I20260129 03:00:20.767472 20335 catalog_manager.cc:5654] T 75be4bd6bd45474795fd472eff2953ac P 974b802141bd4f13beffef2aa03f88f9 reported cstate change: term changed from 9 to 10, leader changed from 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3) to 197b880fa39d432a9b472845876bfaf0 (127.18.211.1). New cstate: current_term: 10 leader_uuid: "197b880fa39d432a9b472845876bfaf0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } } }
I20260129 03:00:20.739606 23531 raft_consensus.cc:740] T 75b0b6e82ddc4f92b1fe1d94f9a0b73b P 1c50de89a5a845668696dcb28c3badd6 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 1c50de89a5a845668696dcb28c3badd6, State: Running, Role: LEADER
W20260129 03:00:20.796038 20556 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.211.2:34517, user_credentials={real_user=slave}} blocked reactor thread for 62532.1us
I20260129 03:00:20.799674 20556 leader_election.cc:304] T 724a236524cb4e7ba3428880681d6cc2 P 1c50de89a5a845668696dcb28c3badd6 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1c50de89a5a845668696dcb28c3badd6; no voters: 197b880fa39d432a9b472845876bfaf0, 974b802141bd4f13beffef2aa03f88f9
I20260129 03:00:20.797617 20335 catalog_manager.cc:5654] T c934d31efd3f4e5195605ddc354d14e1 P 974b802141bd4f13beffef2aa03f88f9 reported cstate change: term changed from 2 to 5, leader changed from 7a9daf5d42a84615a6383cfec9a10608 (127.18.211.4) to 197b880fa39d432a9b472845876bfaf0 (127.18.211.1). New cstate: current_term: 5 leader_uuid: "197b880fa39d432a9b472845876bfaf0" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } } }
I20260129 03:00:20.763310 23466 raft_consensus.cc:3060] T c891d2bff4ce4568861bbc992e91fbb0 P 197b880fa39d432a9b472845876bfaf0 [term 9 FOLLOWER]: Advancing to term 10
I20260129 03:00:20.758189 23627 raft_consensus.cc:493] T e33a417cf9c54cf5af73a4ee3aac4d3a P 197b880fa39d432a9b472845876bfaf0 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260129 03:00:20.846658 23730 raft_consensus.cc:3060] T 7a9a69cd82724b1faed1d10aa091fc4a P 197b880fa39d432a9b472845876bfaf0 [term 7 FOLLOWER]: Advancing to term 8
I20260129 03:00:20.766551 23602 raft_consensus.cc:2749] T 17d6431cc77a491e8464d68db4081a51 P 1c50de89a5a845668696dcb28c3badd6 [term 6 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260129 03:00:20.724507 23710 consensus_queue.cc:237] T 7a9a69cd82724b1faed1d10aa091fc4a P 974b802141bd4f13beffef2aa03f88f9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 7.5, Last appended by leader: 5, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
W20260129 03:00:20.806252 23742 consensus_meta.cc:287] T a0fcc58d6fb6473cafbd9d34d5195269 P 197b880fa39d432a9b472845876bfaf0: Time spent flushing consensus metadata: real 0.600s user 0.013s sys 0.000s
I20260129 03:00:20.748768 23766 raft_consensus.cc:515] T fcddab6bca684adf8ea032289a00f90e P 1c50de89a5a845668696dcb28c3badd6 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 03:00:20.800928 23403 consensus_queue.cc:1048] T c934d31efd3f4e5195605ddc354d14e1 P 197b880fa39d432a9b472845876bfaf0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.040s
I20260129 03:00:20.892828 20513 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:57600 (request call id 2044) took 1380 ms. Trace:
I20260129 03:00:20.893263 20513 rpcz_store.cc:276] 0129 03:00:19.512702 (+ 0us) service_pool.cc:168] Inserting onto call queue
0129 03:00:19.575542 (+ 62840us) service_pool.cc:225] Handling call
0129 03:00:20.889456 (+1313914us) spinlock_profiling.cc:243] Waited 988 ms on lock 0x7b54001ae354. stack: 00007f8fe044e373 00007f8fe044e270 00007f8fdfda0574 00007f8fe9a5c6b4 00007f8fe9a5c649 00007f8fe34975a1 00007f8fe3531f7a 00007f8fe84e3647 00007f8fe1a73834 00007f8fe1a737c2 00007f8fe1a736f1 00007f8fe1a7366c 00007f8fe1a72922 00007f8fe186d34c 00007f8fe186c786 00007f8fe186c12f
0129 03:00:20.892772 (+ 3316us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2272050432}
I20260129 03:00:20.841308 23514 consensus_queue.cc:260] T fcddab6bca684adf8ea032289a00f90e P 974b802141bd4f13beffef2aa03f88f9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 10.6, Last appended by leader: 6, Current term: 10, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 03:00:20.845338 23668 raft_consensus.cc:2749] T 724a236524cb4e7ba3428880681d6cc2 P 1c50de89a5a845668696dcb28c3badd6 [term 7 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260129 03:00:20.868517 23747 leader_election.cc:290] T 75be4bd6bd45474795fd472eff2953ac P 1c50de89a5a845668696dcb28c3badd6 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 197b880fa39d432a9b472845876bfaf0 (127.18.211.1:42027), 974b802141bd4f13beffef2aa03f88f9 (127.18.211.2:34517)
I20260129 03:00:20.863899 23732 consensus_queue.cc:237] T 926a250fa5834e88b96ccecbefbf5086 P 974b802141bd4f13beffef2aa03f88f9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 8.5, Last appended by leader: 5, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 03:00:20.832901 20556 leader_election.cc:304] T 4f8f3a38f6e24ba4857875181f447442 P 1c50de89a5a845668696dcb28c3badd6 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1c50de89a5a845668696dcb28c3badd6, 974b802141bd4f13beffef2aa03f88f9; no voters: 197b880fa39d432a9b472845876bfaf0
I20260129 03:00:20.875746 20335 catalog_manager.cc:5654] T a8479d1fe0bd41e5946186e3a6ca8c7a P 974b802141bd4f13beffef2aa03f88f9 reported cstate change: term changed from 2 to 5. New cstate: current_term: 5 leader_uuid: "974b802141bd4f13beffef2aa03f88f9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } health_report { overall_health: UNKNOWN } } }
I20260129 03:00:20.885361 23693 raft_consensus.cc:3060] T e7d83d4ccf684e76acb29a0dc844e1b8 P 974b802141bd4f13beffef2aa03f88f9 [term 7 FOLLOWER]: Advancing to term 8
W20260129 03:00:20.888796 20553 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.211.1:42027, user_credentials={real_user=slave}} blocked reactor thread for 202053us
I20260129 03:00:20.850690 23744 consensus_queue.cc:1059] T 87d5fe48a44a408aaf03e894a67ecb2f P 197b880fa39d432a9b472845876bfaf0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false }, Status: INVALID_TERM, Last received: 7.8, Next index: 9, Last known committed idx: 8, Time since last communication: 0.040s
I20260129 03:00:20.896312 20407 leader_election.cc:304] T 4b1876f9a70b49edb3aa74105e66d3b3 P 197b880fa39d432a9b472845876bfaf0 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 197b880fa39d432a9b472845876bfaf0, 974b802141bd4f13beffef2aa03f88f9; no voters:
W20260129 03:00:20.914520 20556 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.211.2:34517, user_credentials={real_user=slave}} blocked reactor thread for 81740.3us
I20260129 03:00:20.915957 20335 catalog_manager.cc:5654] T e4d18c35b0a7493e85d0a4b92956a48f P 974b802141bd4f13beffef2aa03f88f9 reported cstate change: term changed from 2 to 5. New cstate: current_term: 5 leader_uuid: "974b802141bd4f13beffef2aa03f88f9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } health_report { overall_health: UNKNOWN } } }
W20260129 03:00:20.922034 20480 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.211.3:37735, user_credentials={real_user=slave}} blocked reactor thread for 312479us
W20260129 03:00:20.922837 20482 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260129 03:00:20.930234 20406 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260129 03:00:20.947500 20515 raft_consensus.cc:1240] T cedce331e1584ac8b636c9b8a72b2ba0 P 974b802141bd4f13beffef2aa03f88f9 [term 9 LEADER]: Rejecting Update request from peer 197b880fa39d432a9b472845876bfaf0 for earlier term 8. Current term is 9. Ops: []
I20260129 03:00:20.945727 20335 catalog_manager.cc:5654] T 3906b080a5b34a74b8b64128635e8997 P 974b802141bd4f13beffef2aa03f88f9 reported cstate change: term changed from 1 to 4, leader changed from 197b880fa39d432a9b472845876bfaf0 (127.18.211.1) to 974b802141bd4f13beffef2aa03f88f9 (127.18.211.2). New cstate: current_term: 4 leader_uuid: "974b802141bd4f13beffef2aa03f88f9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } health_report { overall_health: HEALTHY } } }
I20260129 03:00:20.939817 20513 raft_consensus.cc:1275] T a5a05fbcbbcf4916bbc01fd1a5b5e90d P 974b802141bd4f13beffef2aa03f88f9 [term 9 FOLLOWER]: Refusing update from remote peer 197b880fa39d432a9b472845876bfaf0: Log matching property violated. Preceding OpId in replica: term: 7 index: 5. Preceding OpId from leader: term: 9 index: 7. (index mismatch)
I20260129 03:00:20.949447 20335 catalog_manager.cc:5654] T 17d6431cc77a491e8464d68db4081a51 P 974b802141bd4f13beffef2aa03f88f9 reported cstate change: term changed from 1 to 4. New cstate: current_term: 4 leader_uuid: "974b802141bd4f13beffef2aa03f88f9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } health_report { overall_health: UNKNOWN } } }
W20260129 03:00:20.951195 20482 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260129 03:00:20.952229 20407 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.211.2:34517, user_credentials={real_user=slave}} blocked reactor thread for 56001.3us
I20260129 03:00:20.870891 23531 consensus_queue.cc:260] T 75b0b6e82ddc4f92b1fe1d94f9a0b73b P 1c50de89a5a845668696dcb28c3badd6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 5.8, Last appended by leader: 8, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } }
I20260129 03:00:20.939829 20480 leader_election.cc:304] T c2633a6fc34c44579de348d208b0c01e P 974b802141bd4f13beffef2aa03f88f9 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 974b802141bd4f13beffef2aa03f88f9; no voters: 197b880fa39d432a9b472845876bfaf0, 1c50de89a5a845668696dcb28c3badd6
I20260129 03:00:20.882187 23627 raft_consensus.cc:515] T e33a417cf9c54cf5af73a4ee3aac4d3a P 197b880fa39d432a9b472845876bfaf0 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 03:00:20.955921 20553 leader_election.cc:304] T 36ff33be666a4c82954731691dbcf327 P 1c50de89a5a845668696dcb28c3badd6 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 197b880fa39d432a9b472845876bfaf0, 1c50de89a5a845668696dcb28c3badd6; no voters:
I20260129 03:00:20.961591 20556 leader_election.cc:304] T 3906b080a5b34a74b8b64128635e8997 P 1c50de89a5a845668696dcb28c3badd6 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1c50de89a5a845668696dcb28c3badd6, 974b802141bd4f13beffef2aa03f88f9; no voters: 197b880fa39d432a9b472845876bfaf0
I20260129 03:00:20.967403 20440 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "75be4bd6bd45474795fd472eff2953ac" candidate_uuid: "1c50de89a5a845668696dcb28c3badd6" candidate_term: 11 candidate_status { last_received { term: 10 index: 7 } } ignore_live_leader: false dest_uuid: "197b880fa39d432a9b472845876bfaf0" is_pre_election: true
I20260129 03:00:20.968819 20585 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "75b0b6e82ddc4f92b1fe1d94f9a0b73b" candidate_uuid: "7a9daf5d42a84615a6383cfec9a10608" candidate_term: 6 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "1c50de89a5a845668696dcb28c3badd6"
I20260129 03:00:20.969498 20556 leader_election.cc:304] T 368953c9102643b2a21f99af36b51905 P 1c50de89a5a845668696dcb28c3badd6 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1c50de89a5a845668696dcb28c3badd6, 974b802141bd4f13beffef2aa03f88f9; no voters: 197b880fa39d432a9b472845876bfaf0
I20260129 03:00:20.970685 20556 leader_election.cc:304] T 1ce4ddf5b10e4ae8b193b13b2e5936ee P 1c50de89a5a845668696dcb28c3badd6 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1c50de89a5a845668696dcb28c3badd6, 974b802141bd4f13beffef2aa03f88f9; no voters: 197b880fa39d432a9b472845876bfaf0
I20260129 03:00:20.939273 23747 raft_consensus.cc:2764] T 4f8f3a38f6e24ba4857875181f447442 P 1c50de89a5a845668696dcb28c3badd6 [term 8 FOLLOWER]: Leader election decision vote started in defunct term 7: won
I20260129 03:00:20.974180 20407 leader_election.cc:304] T 76f713c3331540d2a5b76164103e9eb1 P 197b880fa39d432a9b472845876bfaf0 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 197b880fa39d432a9b472845876bfaf0, 974b802141bd4f13beffef2aa03f88f9; no voters:
I20260129 03:00:20.907946 23686 consensus_queue.cc:1059] T 926a250fa5834e88b96ccecbefbf5086 P 197b880fa39d432a9b472845876bfaf0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 }, Status: INVALID_TERM, Last received: 8.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.052s
I20260129 03:00:20.954643 23740 raft_consensus.cc:493] T c934d31efd3f4e5195605ddc354d14e1 P 7a9daf5d42a84615a6383cfec9a10608 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260129 03:00:20.981824 23086 raft_consensus.cc:2804] T 76f713c3331540d2a5b76164103e9eb1 P 197b880fa39d432a9b472845876bfaf0 [term 7 FOLLOWER]: Leader election won for term 7
W20260129 03:00:20.982034 20554 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.211.4:40439, user_credentials={real_user=slave}} blocked reactor thread for 164862us
W20260129 03:00:20.983028 20553 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260129 03:00:20.958251 23514 raft_consensus.cc:3060] T fcddab6bca684adf8ea032289a00f90e P 974b802141bd4f13beffef2aa03f88f9 [term 10 FOLLOWER]: Advancing to term 11
I20260129 03:00:20.914992 23742 consensus_queue.cc:1048] T c934d31efd3f4e5195605ddc354d14e1 P 197b880fa39d432a9b472845876bfaf0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.051s
I20260129 03:00:20.959870 23485 raft_consensus.cc:2804] T 4b1876f9a70b49edb3aa74105e66d3b3 P 197b880fa39d432a9b472845876bfaf0 [term 7 FOLLOWER]: Leader pre-election won for term 8
W20260129 03:00:20.979548 21100 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.211.3:37735, user_credentials={real_user=slave}} blocked reactor thread for 180906us
I20260129 03:00:20.991851 20586 raft_consensus.cc:1275] T 4f8f3a38f6e24ba4857875181f447442 P 1c50de89a5a845668696dcb28c3badd6 [term 8 FOLLOWER]: Refusing update from remote peer 974b802141bd4f13beffef2aa03f88f9: Log matching property violated. Preceding OpId in replica: term: 6 index: 4. Preceding OpId from leader: term: 8 index: 5. (index mismatch)
I20260129 03:00:20.991746 20587 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "cb4727386a3e48aca5d71cc6c611c197" candidate_uuid: "974b802141bd4f13beffef2aa03f88f9" candidate_term: 6 candidate_status { last_received { term: 5 index: 2 } } ignore_live_leader: false dest_uuid: "1c50de89a5a845668696dcb28c3badd6"
W20260129 03:00:20.995711 20480 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260129 03:00:20.979300 23427 raft_consensus.cc:2749] T c2633a6fc34c44579de348d208b0c01e P 974b802141bd4f13beffef2aa03f88f9 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260129 03:00:21.023782 21100 leader_election.cc:304] T 2e58e9a5a8aa4cecafb464addb4f35ed P 7a9daf5d42a84615a6383cfec9a10608 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7a9daf5d42a84615a6383cfec9a10608; no voters: 197b880fa39d432a9b472845876bfaf0, 1c50de89a5a845668696dcb28c3badd6
I20260129 03:00:21.017812 20584 raft_consensus.cc:1275] T 17d6431cc77a491e8464d68db4081a51 P 1c50de89a5a845668696dcb28c3badd6 [term 6 FOLLOWER]: Refusing update from remote peer 974b802141bd4f13beffef2aa03f88f9: Log matching property violated. Preceding OpId in replica: term: 4 index: 2. Preceding OpId from leader: term: 6 index: 3. (index mismatch)
I20260129 03:00:21.032366 20583 raft_consensus.cc:1275] T 724a236524cb4e7ba3428880681d6cc2 P 1c50de89a5a845668696dcb28c3badd6 [term 7 FOLLOWER]: Refusing update from remote peer 974b802141bd4f13beffef2aa03f88f9: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 7 index: 5. (index mismatch)
I20260129 03:00:21.036105 20582 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e33a417cf9c54cf5af73a4ee3aac4d3a" candidate_uuid: "974b802141bd4f13beffef2aa03f88f9" candidate_term: 9 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "1c50de89a5a845668696dcb28c3badd6" is_pre_election: true
W20260129 03:00:21.067557 20554 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260129 03:00:21.058148 20479 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.211.1:42027, user_credentials={real_user=slave}} blocked reactor thread for 79234.9us
I20260129 03:00:21.053628 23631 consensus_queue.cc:1048] T 4f8f3a38f6e24ba4857875181f447442 P 974b802141bd4f13beffef2aa03f88f9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.019s
I20260129 03:00:21.023309 20515 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "75b0b6e82ddc4f92b1fe1d94f9a0b73b" candidate_uuid: "7a9daf5d42a84615a6383cfec9a10608" candidate_term: 6 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "974b802141bd4f13beffef2aa03f88f9"
I20260129 03:00:21.080717 20515 raft_consensus.cc:3060] T 75b0b6e82ddc4f92b1fe1d94f9a0b73b P 974b802141bd4f13beffef2aa03f88f9 [term 5 FOLLOWER]: Advancing to term 6
I20260129 03:00:21.073278 23165 raft_consensus.cc:493] T 75f45538dc704d9a8ec4ff8ad9f3ea5b P 7a9daf5d42a84615a6383cfec9a10608 [term 5 FOLLOWER]: Starting leader election (detected failure of leader 1c50de89a5a845668696dcb28c3badd6)
W20260129 03:00:21.084038 23793 log.cc:927] Time spent T a0fcc58d6fb6473cafbd9d34d5195269 P 974b802141bd4f13beffef2aa03f88f9: Append to log took a long time: real 0.100s user 0.001s sys 0.000s
I20260129 03:00:21.033618 23262 raft_consensus.cc:3055] T a5a05fbcbbcf4916bbc01fd1a5b5e90d P 1c50de89a5a845668696dcb28c3badd6 [term 8 LEADER]: Stepping down as leader of term 8
I20260129 03:00:21.043423 23743 raft_consensus.cc:3055] T 87d5fe48a44a408aaf03e894a67ecb2f P 197b880fa39d432a9b472845876bfaf0 [term 7 LEADER]: Stepping down as leader of term 7
I20260129 03:00:21.020068 20586 raft_consensus.cc:1275] T 0f20af7fbe5d4db89a6f17948d98912c P 1c50de89a5a845668696dcb28c3badd6 [term 8 FOLLOWER]: Refusing update from remote peer 974b802141bd4f13beffef2aa03f88f9: Log matching property violated. Preceding OpId in replica: term: 7 index: 5. Preceding OpId from leader: term: 8 index: 6. (index mismatch)
I20260129 03:00:21.084708 20587 raft_consensus.cc:2468] T cb4727386a3e48aca5d71cc6c611c197 P 1c50de89a5a845668696dcb28c3badd6 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 974b802141bd4f13beffef2aa03f88f9 in term 6.
I20260129 03:00:21.088286 20440 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e33a417cf9c54cf5af73a4ee3aac4d3a" candidate_uuid: "974b802141bd4f13beffef2aa03f88f9" candidate_term: 9 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "197b880fa39d432a9b472845876bfaf0" is_pre_election: true
W20260129 03:00:21.091702 21100 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.211.3:37735, user_credentials={real_user=slave}} blocked reactor thread for 68880.4us
I20260129 03:00:20.992261 23086 raft_consensus.cc:697] T 76f713c3331540d2a5b76164103e9eb1 P 197b880fa39d432a9b472845876bfaf0 [term 7 LEADER]: Becoming Leader. State: Replica: 197b880fa39d432a9b472845876bfaf0, State: Running, Role: LEADER
I20260129 03:00:20.992508 23485 raft_consensus.cc:493] T 4b1876f9a70b49edb3aa74105e66d3b3 P 197b880fa39d432a9b472845876bfaf0 [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260129 03:00:20.989508 23668 raft_consensus.cc:2764] T 3906b080a5b34a74b8b64128635e8997 P 1c50de89a5a845668696dcb28c3badd6 [term 6 FOLLOWER]: Leader pre-election decision vote started in defunct term 5: won
I20260129 03:00:21.130120 20433 raft_consensus.cc:1275] T 8055232cf0004d40a85f3bf983fe00a9 P 197b880fa39d432a9b472845876bfaf0 [term 7 FOLLOWER]: Refusing update from remote peer 974b802141bd4f13beffef2aa03f88f9: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 7 index: 4. (index mismatch)
I20260129 03:00:20.995118 23686 raft_consensus.cc:3055] T 926a250fa5834e88b96ccecbefbf5086 P 197b880fa39d432a9b472845876bfaf0 [term 8 LEADER]: Stepping down as leader of term 8
I20260129 03:00:20.992632 23747 raft_consensus.cc:2764] T 368953c9102643b2a21f99af36b51905 P 1c50de89a5a845668696dcb28c3badd6 [term 10 FOLLOWER]: Leader pre-election decision vote started in defunct term 8: won
I20260129 03:00:20.990590 23798 raft_consensus.cc:2804] T 36ff33be666a4c82954731691dbcf327 P 1c50de89a5a845668696dcb28c3badd6 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260129 03:00:21.140524 23798 raft_consensus.cc:493] T 36ff33be666a4c82954731691dbcf327 P 1c50de89a5a845668696dcb28c3badd6 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260129 03:00:21.140906 23798 raft_consensus.cc:3060] T 36ff33be666a4c82954731691dbcf327 P 1c50de89a5a845668696dcb28c3badd6 [term 5 FOLLOWER]: Advancing to term 6
I20260129 03:00:21.143242 23466 raft_consensus.cc:515] T c891d2bff4ce4568861bbc992e91fbb0 P 197b880fa39d432a9b472845876bfaf0 [term 10 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 03:00:21.146225 23466 leader_election.cc:290] T c891d2bff4ce4568861bbc992e91fbb0 P 197b880fa39d432a9b472845876bfaf0 [CANDIDATE]: Term 10 election: Requested vote from peers 974b802141bd4f13beffef2aa03f88f9 (127.18.211.2:34517), 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3:37735)
I20260129 03:00:21.148257 20581 raft_consensus.cc:1275] T 368953c9102643b2a21f99af36b51905 P 1c50de89a5a845668696dcb28c3badd6 [term 10 FOLLOWER]: Refusing update from remote peer 974b802141bd4f13beffef2aa03f88f9: Log matching property violated. Preceding OpId in replica: term: 8 index: 5. Preceding OpId from leader: term: 10 index: 6. (index mismatch)
I20260129 03:00:21.116298 20583 raft_consensus.cc:1275] T 8055232cf0004d40a85f3bf983fe00a9 P 1c50de89a5a845668696dcb28c3badd6 [term 7 FOLLOWER]: Refusing update from remote peer 974b802141bd4f13beffef2aa03f88f9: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 7 index: 4. (index mismatch)
I20260129 03:00:21.049288 23602 raft_consensus.cc:493] T 4408006eb5174de1896b037cf3f18457 P 1c50de89a5a845668696dcb28c3badd6 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 197b880fa39d432a9b472845876bfaf0)
I20260129 03:00:21.032881 23800 raft_consensus.cc:2804] T 1ce4ddf5b10e4ae8b193b13b2e5936ee P 1c50de89a5a845668696dcb28c3badd6 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260129 03:00:21.152774 20480 leader_election.cc:304] T cb4727386a3e48aca5d71cc6c611c197 P 974b802141bd4f13beffef2aa03f88f9 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1c50de89a5a845668696dcb28c3badd6, 974b802141bd4f13beffef2aa03f88f9; no voters: 197b880fa39d432a9b472845876bfaf0
I20260129 03:00:21.109656 23165 raft_consensus.cc:3060] T 75f45538dc704d9a8ec4ff8ad9f3ea5b P 7a9daf5d42a84615a6383cfec9a10608 [term 5 FOLLOWER]: Advancing to term 6
I20260129 03:00:20.990998 23740 raft_consensus.cc:515] T c934d31efd3f4e5195605ddc354d14e1 P 7a9daf5d42a84615a6383cfec9a10608 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } }
I20260129 03:00:21.113745 23262 raft_consensus.cc:740] T a5a05fbcbbcf4916bbc01fd1a5b5e90d P 1c50de89a5a845668696dcb28c3badd6 [term 8 LEADER]: Becoming Follower/Learner. State: Replica: 1c50de89a5a845668696dcb28c3badd6, State: Running, Role: LEADER
I20260129 03:00:21.169582 23766 leader_election.cc:290] T fcddab6bca684adf8ea032289a00f90e P 1c50de89a5a845668696dcb28c3badd6 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 197b880fa39d432a9b472845876bfaf0 (127.18.211.1:42027), 974b802141bd4f13beffef2aa03f88f9 (127.18.211.2:34517)
I20260129 03:00:21.169734 23262 consensus_queue.cc:260] T a5a05fbcbbcf4916bbc01fd1a5b5e90d P 1c50de89a5a845668696dcb28c3badd6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 8.6, Last appended by leader: 6, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 03:00:21.170113 23086 consensus_queue.cc:237] T 76f713c3331540d2a5b76164103e9eb1 P 197b880fa39d432a9b472845876bfaf0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 6.5, Last appended by leader: 5, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 03:00:21.132176 23485 raft_consensus.cc:3060] T 4b1876f9a70b49edb3aa74105e66d3b3 P 197b880fa39d432a9b472845876bfaf0 [term 7 FOLLOWER]: Advancing to term 8
I20260129 03:00:21.177855 23627 leader_election.cc:290] T e33a417cf9c54cf5af73a4ee3aac4d3a P 197b880fa39d432a9b472845876bfaf0 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 974b802141bd4f13beffef2aa03f88f9 (127.18.211.2:34517), 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3:37735)
I20260129 03:00:21.161052 23646 consensus_queue.cc:1048] T 0f20af7fbe5d4db89a6f17948d98912c P 974b802141bd4f13beffef2aa03f88f9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.018s
I20260129 03:00:21.153486 23800 raft_consensus.cc:493] T 1ce4ddf5b10e4ae8b193b13b2e5936ee P 1c50de89a5a845668696dcb28c3badd6 [term 7 FOLLOWER]: Starting leader election (detected failure of leader 197b880fa39d432a9b472845876bfaf0)
I20260129 03:00:21.182811 23800 raft_consensus.cc:3060] T 1ce4ddf5b10e4ae8b193b13b2e5936ee P 1c50de89a5a845668696dcb28c3badd6 [term 7 FOLLOWER]: Advancing to term 8
I20260129 03:00:21.105768 23706 consensus_queue.cc:1048] T 724a236524cb4e7ba3428880681d6cc2 P 974b802141bd4f13beffef2aa03f88f9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 2, Time since last communication: 0.059s
I20260129 03:00:21.190399 20434 raft_consensus.cc:1275] T a0fcc58d6fb6473cafbd9d34d5195269 P 197b880fa39d432a9b472845876bfaf0 [term 6 FOLLOWER]: Refusing update from remote peer 974b802141bd4f13beffef2aa03f88f9: Log matching property violated. Preceding OpId in replica: term: 5 index: 4. Preceding OpId from leader: term: 6 index: 5. (index mismatch)
I20260129 03:00:21.124845 23743 raft_consensus.cc:740] T 87d5fe48a44a408aaf03e894a67ecb2f P 197b880fa39d432a9b472845876bfaf0 [term 7 LEADER]: Becoming Follower/Learner. State: Replica: 197b880fa39d432a9b472845876bfaf0, State: Running, Role: LEADER
I20260129 03:00:21.139631 23686 raft_consensus.cc:740] T 926a250fa5834e88b96ccecbefbf5086 P 197b880fa39d432a9b472845876bfaf0 [term 8 LEADER]: Becoming Follower/Learner. State: Replica: 197b880fa39d432a9b472845876bfaf0, State: Running, Role: LEADER
I20260129 03:00:21.175338 23691 raft_consensus.cc:515] T a644c02b16af47679824ccd060ac10fd P 974b802141bd4f13beffef2aa03f88f9 [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } }
W20260129 03:00:21.241416 20479 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.211.1:42027, user_credentials={real_user=slave}} blocked reactor thread for 55556.3us
W20260129 03:00:21.270867 20479 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260129 03:00:21.273327 20434 raft_consensus.cc:3055] T cedce331e1584ac8b636c9b8a72b2ba0 P 197b880fa39d432a9b472845876bfaf0 [term 8 LEADER]: Stepping down as leader of term 8
I20260129 03:00:21.273835 20434 raft_consensus.cc:740] T cedce331e1584ac8b636c9b8a72b2ba0 P 197b880fa39d432a9b472845876bfaf0 [term 8 LEADER]: Becoming Follower/Learner. State: Replica: 197b880fa39d432a9b472845876bfaf0, State: Running, Role: LEADER
I20260129 03:00:21.273847 20583 raft_consensus.cc:1275] T a0fcc58d6fb6473cafbd9d34d5195269 P 1c50de89a5a845668696dcb28c3badd6 [term 6 FOLLOWER]: Refusing update from remote peer 974b802141bd4f13beffef2aa03f88f9: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 6 index: 5. (index mismatch)
I20260129 03:00:21.271814 20479 leader_election.cc:304] T e33a417cf9c54cf5af73a4ee3aac4d3a P 974b802141bd4f13beffef2aa03f88f9 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 974b802141bd4f13beffef2aa03f88f9; no voters: 197b880fa39d432a9b472845876bfaf0, 1c50de89a5a845668696dcb28c3badd6
I20260129 03:00:21.185909 23514 raft_consensus.cc:493] T c934d31efd3f4e5195605ddc354d14e1 P 974b802141bd4f13beffef2aa03f88f9 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260129 03:00:21.285241 19276 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 4
I20260129 03:00:21.286823 20439 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fcddab6bca684adf8ea032289a00f90e" candidate_uuid: "1c50de89a5a845668696dcb28c3badd6" candidate_term: 11 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "197b880fa39d432a9b472845876bfaf0" is_pre_election: true
I20260129 03:00:21.287510 19276 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20260129 03:00:21.288080 20439 raft_consensus.cc:2393] T fcddab6bca684adf8ea032289a00f90e P 197b880fa39d432a9b472845876bfaf0 [term 11 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1c50de89a5a845668696dcb28c3badd6 in current term 11: Already voted for candidate 197b880fa39d432a9b472845876bfaf0 in this term.
W20260129 03:00:21.286007 20407 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.211.2:34517, user_credentials={real_user=slave}} blocked reactor thread for 299412us
I20260129 03:00:21.291002 20439 raft_consensus.cc:1240] T a5a05fbcbbcf4916bbc01fd1a5b5e90d P 197b880fa39d432a9b472845876bfaf0 [term 9 LEADER]: Rejecting Update request from peer 1c50de89a5a845668696dcb28c3badd6 for earlier term 8. Current term is 9. Ops: []
I20260129 03:00:21.290658 19276 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
W20260129 03:00:21.303638 20556 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.211.2:34517, user_credentials={real_user=slave}} blocked reactor thread for 333032us
I20260129 03:00:21.304783 20556 leader_election.cc:304] T 7a9a69cd82724b1faed1d10aa091fc4a P 1c50de89a5a845668696dcb28c3badd6 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1c50de89a5a845668696dcb28c3badd6; no voters: 197b880fa39d432a9b472845876bfaf0, 974b802141bd4f13beffef2aa03f88f9
I20260129 03:00:21.305801 23678 raft_consensus.cc:2749] T 7a9a69cd82724b1faed1d10aa091fc4a P 1c50de89a5a845668696dcb28c3badd6 [term 8 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260129 03:00:21.307802 20556 leader_election.cc:304] T cb0b3f259ebc4310927338a8740e87b8 P 1c50de89a5a845668696dcb28c3badd6 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1c50de89a5a845668696dcb28c3badd6, 974b802141bd4f13beffef2aa03f88f9; no voters: 7a9daf5d42a84615a6383cfec9a10608
I20260129 03:00:21.311496 23766 consensus_queue.cc:1048] T 4b1876f9a70b49edb3aa74105e66d3b3 P 1c50de89a5a845668696dcb28c3badd6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260129 03:00:21.311560 20556 leader_election.cc:304] T c891d2bff4ce4568861bbc992e91fbb0 P 1c50de89a5a845668696dcb28c3badd6 [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1c50de89a5a845668696dcb28c3badd6, 974b802141bd4f13beffef2aa03f88f9; no voters: 197b880fa39d432a9b472845876bfaf0
I20260129 03:00:21.274802 20434 consensus_queue.cc:260] T cedce331e1584ac8b636c9b8a72b2ba0 P 197b880fa39d432a9b472845876bfaf0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 8.5, Last appended by leader: 5, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 03:00:21.237824 23758 raft_consensus.cc:2804] T cb4727386a3e48aca5d71cc6c611c197 P 974b802141bd4f13beffef2aa03f88f9 [term 6 FOLLOWER]: Leader election won for term 6
I20260129 03:00:21.159614 23602 raft_consensus.cc:515] T 4408006eb5174de1896b037cf3f18457 P 1c50de89a5a845668696dcb28c3badd6 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 03:00:21.318779 20556 leader_election.cc:400] T a1a39dcd328e4268ae41b988c66b4a75 P 1c50de89a5a845668696dcb28c3badd6 [CANDIDATE]: Term 6 pre-election: Vote denied by peer 974b802141bd4f13beffef2aa03f88f9 (127.18.211.2:34517) with higher term. Message: Invalid argument: T a1a39dcd328e4268ae41b988c66b4a75 P 974b802141bd4f13beffef2aa03f88f9 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1c50de89a5a845668696dcb28c3badd6 for earlier term 6. Current term is 7.
W20260129 03:00:21.319741 20480 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.211.3:37735, user_credentials={real_user=slave}} blocked reactor thread for 115054us
I20260129 03:00:21.316428 20434 raft_consensus.cc:3060] T cedce331e1584ac8b636c9b8a72b2ba0 P 197b880fa39d432a9b472845876bfaf0 [term 8 FOLLOWER]: Advancing to term 9
I20260129 03:00:21.330251 20556 leader_election.cc:403] T a1a39dcd328e4268ae41b988c66b4a75 P 1c50de89a5a845668696dcb28c3badd6 [CANDIDATE]: Term 6 pre-election: Cancelling election due to peer responding with higher term
I20260129 03:00:21.277072 23682 raft_consensus.cc:493] T a644c02b16af47679824ccd060ac10fd P 7a9daf5d42a84615a6383cfec9a10608 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260129 03:00:21.359934 22645 consensus_queue.cc:1048] T a5a05fbcbbcf4916bbc01fd1a5b5e90d P 197b880fa39d432a9b472845876bfaf0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 5, Time since last communication: 0.006s
W20260129 03:00:21.370757 20479 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.211.1:42027, user_credentials={real_user=slave}} blocked reactor thread for 99053.5us
W20260129 03:00:21.376516 20407 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260129 03:00:21.377174 20407 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260129 03:00:21.377970 20406 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260129 03:00:21.377970 20511 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e33a417cf9c54cf5af73a4ee3aac4d3a" candidate_uuid: "197b880fa39d432a9b472845876bfaf0" candidate_term: 9 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "974b802141bd4f13beffef2aa03f88f9" is_pre_election: true
I20260129 03:00:21.366163 23531 raft_consensus.cc:3060] T 75b0b6e82ddc4f92b1fe1d94f9a0b73b P 1c50de89a5a845668696dcb28c3badd6 [term 5 FOLLOWER]: Advancing to term 6
I20260129 03:00:21.367479 20581 raft_consensus.cc:1275] T 0340229d521b432580ae926f6062e3f7 P 1c50de89a5a845668696dcb28c3badd6 [term 9 FOLLOWER]: Refusing update from remote peer 974b802141bd4f13beffef2aa03f88f9: Log matching property violated. Preceding OpId in replica: term: 7 index: 5. Preceding OpId from leader: term: 9 index: 6. (index mismatch)
I20260129 03:00:21.367789 23602 leader_election.cc:290] T 4408006eb5174de1896b037cf3f18457 P 1c50de89a5a845668696dcb28c3badd6 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 197b880fa39d432a9b472845876bfaf0 (127.18.211.1:42027), 974b802141bd4f13beffef2aa03f88f9 (127.18.211.2:34517)
I20260129 03:00:21.370110 20439 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4408006eb5174de1896b037cf3f18457" candidate_uuid: "1c50de89a5a845668696dcb28c3badd6" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "197b880fa39d432a9b472845876bfaf0" is_pre_election: true
I20260129 03:00:21.401369 20434 raft_consensus.cc:1275] T cedce331e1584ac8b636c9b8a72b2ba0 P 197b880fa39d432a9b472845876bfaf0 [term 9 FOLLOWER]: Refusing update from remote peer 974b802141bd4f13beffef2aa03f88f9: Log matching property violated. Preceding OpId in replica: term: 8 index: 5. Preceding OpId from leader: term: 9 index: 6. (index mismatch)
I20260129 03:00:21.327383 23803 raft_consensus.cc:2804] T c891d2bff4ce4568861bbc992e91fbb0 P 1c50de89a5a845668696dcb28c3badd6 [term 9 FOLLOWER]: Leader election won for term 9
I20260129 03:00:21.353919 23747 raft_consensus.cc:2749] T a1a39dcd328e4268ae41b988c66b4a75 P 1c50de89a5a845668696dcb28c3badd6 [term 7 FOLLOWER]: Leader pre-election lost for term 6. Reason: Vote denied by peer 974b802141bd4f13beffef2aa03f88f9 (127.18.211.2:34517) with higher term. Message: Invalid argument: T a1a39dcd328e4268ae41b988c66b4a75 P 974b802141bd4f13beffef2aa03f88f9 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1c50de89a5a845668696dcb28c3badd6 for earlier term 6. Current term is 7.
I20260129 03:00:21.409848 20510 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c891d2bff4ce4568861bbc992e91fbb0" candidate_uuid: "197b880fa39d432a9b472845876bfaf0" candidate_term: 10 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "974b802141bd4f13beffef2aa03f88f9"
W20260129 03:00:21.411450 21102 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.211.2:34517, user_credentials={real_user=slave}} blocked reactor thread for 83879.3us
I20260129 03:00:21.412820 20510 raft_consensus.cc:3060] T c891d2bff4ce4568861bbc992e91fbb0 P 974b802141bd4f13beffef2aa03f88f9 [term 9 FOLLOWER]: Advancing to term 10
I20260129 03:00:21.383037 23695 raft_consensus.cc:493] T 36ff33be666a4c82954731691dbcf327 P 974b802141bd4f13beffef2aa03f88f9 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260129 03:00:21.382752 23694 raft_consensus.cc:2749] T e33a417cf9c54cf5af73a4ee3aac4d3a P 974b802141bd4f13beffef2aa03f88f9 [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260129 03:00:21.423205 20581 raft_consensus.cc:1240] T cedce331e1584ac8b636c9b8a72b2ba0 P 1c50de89a5a845668696dcb28c3badd6 [term 9 FOLLOWER]: Rejecting Update request from peer 197b880fa39d432a9b472845876bfaf0 for earlier term 8. Current term is 9. Ops: []
I20260129 03:00:21.423413 20587 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a1a39dcd328e4268ae41b988c66b4a75" candidate_uuid: "197b880fa39d432a9b472845876bfaf0" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "1c50de89a5a845668696dcb28c3badd6"
I20260129 03:00:21.445036 20587 raft_consensus.cc:2393] T a1a39dcd328e4268ae41b988c66b4a75 P 1c50de89a5a845668696dcb28c3badd6 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 197b880fa39d432a9b472845876bfaf0 in current term 7: Already voted for candidate 974b802141bd4f13beffef2aa03f88f9 in this term.
I20260129 03:00:21.424544 20586 raft_consensus.cc:1240] T 7a9a69cd82724b1faed1d10aa091fc4a P 1c50de89a5a845668696dcb28c3badd6 [term 8 FOLLOWER]: Rejecting Update request from peer 197b880fa39d432a9b472845876bfaf0 for earlier term 7. Current term is 8. Ops: []
I20260129 03:00:21.446116 20586 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7d9494f5584d4d5db8b9adf1c8fc888b" candidate_uuid: "197b880fa39d432a9b472845876bfaf0" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: true dest_uuid: "1c50de89a5a845668696dcb28c3badd6"
I20260129 03:00:21.353600 23710 consensus_queue.cc:1048] T 8055232cf0004d40a85f3bf983fe00a9 P 974b802141bd4f13beffef2aa03f88f9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 1, Time since last communication: 0.059s
I20260129 03:00:21.428633 20584 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e33a417cf9c54cf5af73a4ee3aac4d3a" candidate_uuid: "197b880fa39d432a9b472845876bfaf0" candidate_term: 7 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "1c50de89a5a845668696dcb28c3badd6"
I20260129 03:00:21.452571 20584 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "36ff33be666a4c82954731691dbcf327" candidate_uuid: "197b880fa39d432a9b472845876bfaf0" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "1c50de89a5a845668696dcb28c3badd6" is_pre_election: true
W20260129 03:00:21.420501 20482 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260129 03:00:21.322501 23514 raft_consensus.cc:515] T c934d31efd3f4e5195605ddc354d14e1 P 974b802141bd4f13beffef2aa03f88f9 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } }
W20260129 03:00:21.454402 20407 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.211.2:34517, user_credentials={real_user=slave}} blocked reactor thread for 60238.6us
I20260129 03:00:21.353600 23631 consensus_queue.cc:1048] T 368953c9102643b2a21f99af36b51905 P 974b802141bd4f13beffef2aa03f88f9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 4, Time since last communication: 0.059s
I20260129 03:00:21.446812 20586 raft_consensus.cc:2376] T 7d9494f5584d4d5db8b9adf1c8fc888b P 1c50de89a5a845668696dcb28c3badd6 [term 5 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 197b880fa39d432a9b472845876bfaf0 in term 5. Re-sending same reply.
I20260129 03:00:21.457119 20586 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fcddab6bca684adf8ea032289a00f90e" candidate_uuid: "197b880fa39d432a9b472845876bfaf0" candidate_term: 11 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "1c50de89a5a845668696dcb28c3badd6"
I20260129 03:00:21.457691 20586 raft_consensus.cc:3060] T fcddab6bca684adf8ea032289a00f90e P 1c50de89a5a845668696dcb28c3badd6 [term 10 FOLLOWER]: Advancing to term 11
I20260129 03:00:21.425660 23799 raft_consensus.cc:493] T fd8311b779af4e1fa1402b2a02a602f2 P 1c50de89a5a845668696dcb28c3badd6 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260129 03:00:21.411362 23566 raft_consensus.cc:2749] T 2e58e9a5a8aa4cecafb464addb4f35ed P 7a9daf5d42a84615a6383cfec9a10608 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260129 03:00:21.359958 23646 consensus_queue.cc:1048] T a0fcc58d6fb6473cafbd9d34d5195269 P 974b802141bd4f13beffef2aa03f88f9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 3, Time since last communication: 0.059s
I20260129 03:00:21.382584 23403 raft_consensus.cc:493] T 42d27a28d89b409e8807ec0efd97af69 P 197b880fa39d432a9b472845876bfaf0 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 7a9daf5d42a84615a6383cfec9a10608)
I20260129 03:00:21.453737 20581 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a8479d1fe0bd41e5946186e3a6ca8c7a" candidate_uuid: "197b880fa39d432a9b472845876bfaf0" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "1c50de89a5a845668696dcb28c3badd6"
I20260129 03:00:21.475172 20581 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c891d2bff4ce4568861bbc992e91fbb0" candidate_uuid: "197b880fa39d432a9b472845876bfaf0" candidate_term: 10 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "1c50de89a5a845668696dcb28c3badd6"
W20260129 03:00:21.475606 20555 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260129 03:00:21.448151 20587 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "17d6431cc77a491e8464d68db4081a51" candidate_uuid: "197b880fa39d432a9b472845876bfaf0" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "1c50de89a5a845668696dcb28c3badd6"
I20260129 03:00:21.476786 20587 raft_consensus.cc:3060] T 17d6431cc77a491e8464d68db4081a51 P 1c50de89a5a845668696dcb28c3badd6 [term 6 FOLLOWER]: Advancing to term 7
I20260129 03:00:21.447220 23627 raft_consensus.cc:493] T a0fcc58d6fb6473cafbd9d34d5195269 P 197b880fa39d432a9b472845876bfaf0 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260129 03:00:21.469092 20511 raft_consensus.cc:2468] T e33a417cf9c54cf5af73a4ee3aac4d3a P 974b802141bd4f13beffef2aa03f88f9 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 197b880fa39d432a9b472845876bfaf0 in term 8.
I20260129 03:00:21.457293 23809 raft_consensus.cc:493] T 96609ac68bdf4993a12e1c23355ca013 P 197b880fa39d432a9b472845876bfaf0 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 7a9daf5d42a84615a6383cfec9a10608)
I20260129 03:00:21.442638 23678 raft_consensus.cc:2804] T cb0b3f259ebc4310927338a8740e87b8 P 1c50de89a5a845668696dcb28c3badd6 [term 9 FOLLOWER]: Leader pre-election won for term 10
I20260129 03:00:21.443506 23803 raft_consensus.cc:697] T c891d2bff4ce4568861bbc992e91fbb0 P 1c50de89a5a845668696dcb28c3badd6 [term 9 LEADER]: Becoming Leader. State: Replica: 1c50de89a5a845668696dcb28c3badd6, State: Running, Role: LEADER
I20260129 03:00:21.386061 23682 raft_consensus.cc:515] T a644c02b16af47679824ccd060ac10fd P 7a9daf5d42a84615a6383cfec9a10608 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } }
I20260129 03:00:21.373473 23686 consensus_queue.cc:260] T 926a250fa5834e88b96ccecbefbf5086 P 197b880fa39d432a9b472845876bfaf0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 8.5, Last appended by leader: 5, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 03:00:21.476002 20510 raft_consensus.cc:2468] T c891d2bff4ce4568861bbc992e91fbb0 P 974b802141bd4f13beffef2aa03f88f9 [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 197b880fa39d432a9b472845876bfaf0 in term 10.
I20260129 03:00:21.389740 23743 consensus_queue.cc:260] T 87d5fe48a44a408aaf03e894a67ecb2f P 197b880fa39d432a9b472845876bfaf0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 7.8, Last appended by leader: 8, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } }
I20260129 03:00:21.481998 20405 leader_election.cc:304] T a8479d1fe0bd41e5946186e3a6ca8c7a P 197b880fa39d432a9b472845876bfaf0 [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 197b880fa39d432a9b472845876bfaf0; no voters: 1c50de89a5a845668696dcb28c3badd6, 974b802141bd4f13beffef2aa03f88f9
W20260129 03:00:21.502727 20479 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.211.1:42027, user_credentials={real_user=slave}} blocked reactor thread for 130558us
I20260129 03:00:21.449849 23695 raft_consensus.cc:515] T 36ff33be666a4c82954731691dbcf327 P 974b802141bd4f13beffef2aa03f88f9 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 03:00:21.525739 20587 raft_consensus.cc:2468] T 17d6431cc77a491e8464d68db4081a51 P 1c50de89a5a845668696dcb28c3badd6 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 197b880fa39d432a9b472845876bfaf0 in term 7.
I20260129 03:00:21.467610 23717 consensus_queue.cc:1048] T 8055232cf0004d40a85f3bf983fe00a9 P 974b802141bd4f13beffef2aa03f88f9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 1, Time since last communication: 0.047s
W20260129 03:00:21.509898 23398 consensus_meta.cc:287] T 18f30ebaa4174fd28c958e72d0cc7e24 P 1c50de89a5a845668696dcb28c3badd6: Time spent flushing consensus metadata: real 0.865s user 0.006s sys 0.008s
I20260129 03:00:21.527706 20405 leader_election.cc:304] T 17d6431cc77a491e8464d68db4081a51 P 197b880fa39d432a9b472845876bfaf0 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 197b880fa39d432a9b472845876bfaf0, 1c50de89a5a845668696dcb28c3badd6; no voters: 974b802141bd4f13beffef2aa03f88f9
I20260129 03:00:21.540503 20586 raft_consensus.cc:2468] T fcddab6bca684adf8ea032289a00f90e P 1c50de89a5a845668696dcb28c3badd6 [term 11 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 197b880fa39d432a9b472845876bfaf0 in term 11.
I20260129 03:00:21.467963 23799 raft_consensus.cc:515] T fd8311b779af4e1fa1402b2a02a602f2 P 1c50de89a5a845668696dcb28c3badd6 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } }
I20260129 03:00:21.549296 20405 leader_election.cc:304] T fcddab6bca684adf8ea032289a00f90e P 197b880fa39d432a9b472845876bfaf0 [CANDIDATE]: Term 11 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 197b880fa39d432a9b472845876bfaf0, 1c50de89a5a845668696dcb28c3badd6; no voters: 974b802141bd4f13beffef2aa03f88f9
I20260129 03:00:21.551543 23758 raft_consensus.cc:697] T cb4727386a3e48aca5d71cc6c611c197 P 974b802141bd4f13beffef2aa03f88f9 [term 6 LEADER]: Becoming Leader. State: Replica: 974b802141bd4f13beffef2aa03f88f9, State: Running, Role: LEADER
I20260129 03:00:21.550360 20587 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4e3805ffa70242e8ade66da260dddf66" candidate_uuid: "197b880fa39d432a9b472845876bfaf0" candidate_term: 7 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "1c50de89a5a845668696dcb28c3badd6" is_pre_election: true
W20260129 03:00:21.552963 21099 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.211.1:42027, user_credentials={real_user=slave}} blocked reactor thread for 140057us
I20260129 03:00:21.553764 20434 raft_consensus.cc:1275] T 3906b080a5b34a74b8b64128635e8997 P 197b880fa39d432a9b472845876bfaf0 [term 6 FOLLOWER]: Refusing update from remote peer 974b802141bd4f13beffef2aa03f88f9: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 6 index: 4. (index mismatch)
I20260129 03:00:21.551561 20586 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a644c02b16af47679824ccd060ac10fd" candidate_uuid: "974b802141bd4f13beffef2aa03f88f9" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "1c50de89a5a845668696dcb28c3badd6"
I20260129 03:00:21.556250 20587 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3906b080a5b34a74b8b64128635e8997" candidate_uuid: "197b880fa39d432a9b472845876bfaf0" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "1c50de89a5a845668696dcb28c3badd6"
I20260129 03:00:21.559062 23631 consensus_queue.cc:1048] T cedce331e1584ac8b636c9b8a72b2ba0 P 974b802141bd4f13beffef2aa03f88f9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260129 03:00:21.559355 20587 raft_consensus.cc:2368] T 3906b080a5b34a74b8b64128635e8997 P 1c50de89a5a845668696dcb28c3badd6 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 197b880fa39d432a9b472845876bfaf0 for earlier term 5. Current term is 6.
I20260129 03:00:21.556663 23758 consensus_queue.cc:237] T cb4727386a3e48aca5d71cc6c611c197 P 974b802141bd4f13beffef2aa03f88f9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 5.2, Last appended by leader: 2, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 03:00:21.563294 20587 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e33a417cf9c54cf5af73a4ee3aac4d3a" candidate_uuid: "197b880fa39d432a9b472845876bfaf0" candidate_term: 9 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "1c50de89a5a845668696dcb28c3badd6" is_pre_election: true
I20260129 03:00:21.566416 20587 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1299723a4476420ab02fd8bfd7aa0645" candidate_uuid: "197b880fa39d432a9b472845876bfaf0" candidate_term: 11 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "1c50de89a5a845668696dcb28c3badd6" is_pre_election: true
I20260129 03:00:21.483987 23403 raft_consensus.cc:515] T 42d27a28d89b409e8807ec0efd97af69 P 197b880fa39d432a9b472845876bfaf0 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } }
I20260129 03:00:21.491653 23706 consensus_queue.cc:1048] T a0fcc58d6fb6473cafbd9d34d5195269 P 974b802141bd4f13beffef2aa03f88f9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.059s
I20260129 03:00:21.572252 21130 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a644c02b16af47679824ccd060ac10fd" candidate_uuid: "974b802141bd4f13beffef2aa03f88f9" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "7a9daf5d42a84615a6383cfec9a10608"
I20260129 03:00:21.572455 20587 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a5a05fbcbbcf4916bbc01fd1a5b5e90d" candidate_uuid: "197b880fa39d432a9b472845876bfaf0" candidate_term: 9 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "1c50de89a5a845668696dcb28c3badd6"
I20260129 03:00:21.584888 21130 raft_consensus.cc:2393] T a644c02b16af47679824ccd060ac10fd P 7a9daf5d42a84615a6383cfec9a10608 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 974b802141bd4f13beffef2aa03f88f9 in current term 6: Already voted for candidate 7a9daf5d42a84615a6383cfec9a10608 in this term.
I20260129 03:00:21.492887 23627 raft_consensus.cc:515] T a0fcc58d6fb6473cafbd9d34d5195269 P 197b880fa39d432a9b472845876bfaf0 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 03:00:21.546291 23717 consensus_queue.cc:1243] T 8055232cf0004d40a85f3bf983fe00a9 P 974b802141bd4f13beffef2aa03f88f9 [LEADER]: Peer 197b880fa39d432a9b472845876bfaf0 log is divergent from this leader: its last log entry 4.3 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 1
W20260129 03:00:21.569810 23820 log.cc:927] Time spent T 4f8f3a38f6e24ba4857875181f447442 P 197b880fa39d432a9b472845876bfaf0: Append to log took a long time: real 0.081s user 0.000s sys 0.001s
W20260129 03:00:21.569275 20407 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.211.2:34517, user_credentials={real_user=slave}} blocked reactor thread for 95054.4us
I20260129 03:00:21.584192 20586 raft_consensus.cc:2468] T a644c02b16af47679824ccd060ac10fd P 1c50de89a5a845668696dcb28c3badd6 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 974b802141bd4f13beffef2aa03f88f9 in term 6.
I20260129 03:00:21.576876 20515 raft_consensus.cc:2468] T 75b0b6e82ddc4f92b1fe1d94f9a0b73b P 974b802141bd4f13beffef2aa03f88f9 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7a9daf5d42a84615a6383cfec9a10608 in term 6.
I20260129 03:00:21.591475 20514 raft_consensus.cc:1275] T c2633a6fc34c44579de348d208b0c01e P 974b802141bd4f13beffef2aa03f88f9 [term 6 FOLLOWER]: Refusing update from remote peer 1c50de89a5a845668696dcb28c3badd6: Log matching property violated. Preceding OpId in replica: term: 5 index: 3. Preceding OpId from leader: term: 6 index: 3. (term mismatch)
I20260129 03:00:21.600555 20514 pending_rounds.cc:85] T c2633a6fc34c44579de348d208b0c01e P 974b802141bd4f13beffef2aa03f88f9: Aborting all ops after (but not including) 2
I20260129 03:00:21.600896 20514 pending_rounds.cc:107] T c2633a6fc34c44579de348d208b0c01e P 974b802141bd4f13beffef2aa03f88f9: Aborting uncommitted NO_OP operation due to leader change: 5.3
I20260129 03:00:21.601158 20514 raft_consensus.cc:2889] T c2633a6fc34c44579de348d208b0c01e P 974b802141bd4f13beffef2aa03f88f9 [term 6 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260129 03:00:21.594632 20510 raft_consensus.cc:1275] T 75f45538dc704d9a8ec4ff8ad9f3ea5b P 974b802141bd4f13beffef2aa03f88f9 [term 5 FOLLOWER]: Refusing update from remote peer 1c50de89a5a845668696dcb28c3badd6: Log matching property violated. Preceding OpId in replica: term: 4 index: 6. Preceding OpId from leader: term: 5 index: 7. (index mismatch)
I20260129 03:00:21.549464 23466 raft_consensus.cc:2749] T a8479d1fe0bd41e5946186e3a6ca8c7a P 197b880fa39d432a9b472845876bfaf0 [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20260129 03:00:21.604264 20586 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e7d83d4ccf684e76acb29a0dc844e1b8" candidate_uuid: "197b880fa39d432a9b472845876bfaf0" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "1c50de89a5a845668696dcb28c3badd6"
I20260129 03:00:21.606660 20436 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c934d31efd3f4e5195605ddc354d14e1" candidate_uuid: "7a9daf5d42a84615a6383cfec9a10608" candidate_term: 8 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "197b880fa39d432a9b472845876bfaf0" is_pre_election: true
I20260129 03:00:21.607854 20586 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0f20af7fbe5d4db89a6f17948d98912c" candidate_uuid: "197b880fa39d432a9b472845876bfaf0" candidate_term: 9 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "1c50de89a5a845668696dcb28c3badd6" is_pre_election: true
I20260129 03:00:21.609319 20513 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "45cc5f3d2688486d9b6dbe7e666dedde" candidate_uuid: "1c50de89a5a845668696dcb28c3badd6" candidate_term: 8 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "974b802141bd4f13beffef2aa03f88f9"
I20260129 03:00:21.610002 20513 raft_consensus.cc:2393] T 45cc5f3d2688486d9b6dbe7e666dedde P 974b802141bd4f13beffef2aa03f88f9 [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate 1c50de89a5a845668696dcb28c3badd6 in current term 8: Already voted for candidate 7a9daf5d42a84615a6383cfec9a10608 in this term.
I20260129 03:00:21.573328 23807 raft_consensus.cc:2804] T 17d6431cc77a491e8464d68db4081a51 P 197b880fa39d432a9b472845876bfaf0 [term 7 FOLLOWER]: Leader election won for term 7
I20260129 03:00:21.610685 20586 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "913fc7d751ed4b7388d889aac30a5f9b" candidate_uuid: "197b880fa39d432a9b472845876bfaf0" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "1c50de89a5a845668696dcb28c3badd6" is_pre_election: true
I20260129 03:00:21.610162 20510 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0340229d521b432580ae926f6062e3f7" candidate_uuid: "1c50de89a5a845668696dcb28c3badd6" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "974b802141bd4f13beffef2aa03f88f9" is_pre_election: true
I20260129 03:00:21.564523 23691 leader_election.cc:290] T a644c02b16af47679824ccd060ac10fd P 974b802141bd4f13beffef2aa03f88f9 [CANDIDATE]: Term 6 election: Requested vote from peers 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3:37735), 7a9daf5d42a84615a6383cfec9a10608 (127.18.211.4:40439)
I20260129 03:00:21.612726 20407 leader_election.cc:304] T c891d2bff4ce4568861bbc992e91fbb0 P 197b880fa39d432a9b472845876bfaf0 [CANDIDATE]: Term 10 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 197b880fa39d432a9b472845876bfaf0, 974b802141bd4f13beffef2aa03f88f9; no voters:
I20260129 03:00:21.571552 23791 raft_consensus.cc:493] T 00e79d8f25a34676b1d866eb2068a2b2 P 7a9daf5d42a84615a6383cfec9a10608 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 197b880fa39d432a9b472845876bfaf0)
I20260129 03:00:21.613147 20508 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "45cc5f3d2688486d9b6dbe7e666dedde" candidate_uuid: "1c50de89a5a845668696dcb28c3badd6" candidate_term: 8 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "974b802141bd4f13beffef2aa03f88f9" is_pre_election: true
I20260129 03:00:21.618629 20508 raft_consensus.cc:2393] T 45cc5f3d2688486d9b6dbe7e666dedde P 974b802141bd4f13beffef2aa03f88f9 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1c50de89a5a845668696dcb28c3badd6 in current term 8: Already voted for candidate 7a9daf5d42a84615a6383cfec9a10608 in this term.
I20260129 03:00:21.620218 20586 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "76f713c3331540d2a5b76164103e9eb1" candidate_uuid: "197b880fa39d432a9b472845876bfaf0" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "1c50de89a5a845668696dcb28c3badd6"
I20260129 03:00:21.620867 20586 raft_consensus.cc:3060] T 76f713c3331540d2a5b76164103e9eb1 P 1c50de89a5a845668696dcb28c3badd6 [term 6 FOLLOWER]: Advancing to term 7
I20260129 03:00:21.630168 20586 raft_consensus.cc:2468] T 76f713c3331540d2a5b76164103e9eb1 P 1c50de89a5a845668696dcb28c3badd6 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 197b880fa39d432a9b472845876bfaf0 in term 7.
I20260129 03:00:21.564056 23398 raft_consensus.cc:515] T 18f30ebaa4174fd28c958e72d0cc7e24 P 1c50de89a5a845668696dcb28c3badd6 [term 9 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } }
I20260129 03:00:21.634713 20586 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a5a05fbcbbcf4916bbc01fd1a5b5e90d" candidate_uuid: "197b880fa39d432a9b472845876bfaf0" candidate_term: 9 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "1c50de89a5a845668696dcb28c3badd6" is_pre_election: true
I20260129 03:00:21.632507 20513 raft_consensus.cc:1240] T 75b0b6e82ddc4f92b1fe1d94f9a0b73b P 974b802141bd4f13beffef2aa03f88f9 [term 6 FOLLOWER]: Rejecting Update request from peer 1c50de89a5a845668696dcb28c3badd6 for earlier term 5. Current term is 6. Ops: []
I20260129 03:00:21.587944 23631 consensus_queue.cc:1048] T 0340229d521b432580ae926f6062e3f7 P 974b802141bd4f13beffef2aa03f88f9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.036s
I20260129 03:00:21.581988 23755 consensus_queue.cc:1048] T 3906b080a5b34a74b8b64128635e8997 P 974b802141bd4f13beffef2aa03f88f9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.040s
I20260129 03:00:21.636744 20586 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c891d2bff4ce4568861bbc992e91fbb0" candidate_uuid: "197b880fa39d432a9b472845876bfaf0" candidate_term: 10 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "1c50de89a5a845668696dcb28c3badd6" is_pre_election: true
I20260129 03:00:21.513896 23809 raft_consensus.cc:515] T 96609ac68bdf4993a12e1c23355ca013 P 197b880fa39d432a9b472845876bfaf0 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } attrs { replace: true } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } }
I20260129 03:00:21.631655 23466 raft_consensus.cc:2804] T fcddab6bca684adf8ea032289a00f90e P 197b880fa39d432a9b472845876bfaf0 [term 11 FOLLOWER]: Leader election won for term 11
I20260129 03:00:21.628441 23807 raft_consensus.cc:697] T 17d6431cc77a491e8464d68db4081a51 P 197b880fa39d432a9b472845876bfaf0 [term 7 LEADER]: Becoming Leader. State: Replica: 197b880fa39d432a9b472845876bfaf0, State: Running, Role: LEADER
I20260129 03:00:21.646763 20508 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "36ff33be666a4c82954731691dbcf327" candidate_uuid: "1c50de89a5a845668696dcb28c3badd6" candidate_term: 6 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "974b802141bd4f13beffef2aa03f88f9" is_pre_election: true
I20260129 03:00:21.671130 20588 raft_consensus.cc:1240] T 18f30ebaa4174fd28c958e72d0cc7e24 P 1c50de89a5a845668696dcb28c3badd6 [term 9 FOLLOWER]: Rejecting Update request from peer 7a9daf5d42a84615a6383cfec9a10608 for earlier term 8. Current term is 9. Ops: []
I20260129 03:00:21.671478 20508 raft_consensus.cc:2468] T 36ff33be666a4c82954731691dbcf327 P 974b802141bd4f13beffef2aa03f88f9 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1c50de89a5a845668696dcb28c3badd6 in term 5.
I20260129 03:00:21.647763 23701 raft_consensus.cc:2804] T c891d2bff4ce4568861bbc992e91fbb0 P 197b880fa39d432a9b472845876bfaf0 [term 10 FOLLOWER]: Leader election won for term 10
I20260129 03:00:21.679055 21130 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c934d31efd3f4e5195605ddc354d14e1" candidate_uuid: "974b802141bd4f13beffef2aa03f88f9" candidate_term: 8 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "7a9daf5d42a84615a6383cfec9a10608" is_pre_election: true
I20260129 03:00:21.640012 20405 leader_election.cc:304] T 4e3805ffa70242e8ade66da260dddf66 P 197b880fa39d432a9b472845876bfaf0 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 197b880fa39d432a9b472845876bfaf0; no voters: 1c50de89a5a845668696dcb28c3badd6, 7a9daf5d42a84615a6383cfec9a10608
W20260129 03:00:21.664677 20555 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260129 03:00:21.681347 20436 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c934d31efd3f4e5195605ddc354d14e1" candidate_uuid: "974b802141bd4f13beffef2aa03f88f9" candidate_term: 8 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "197b880fa39d432a9b472845876bfaf0" is_pre_election: true
I20260129 03:00:21.646157 23682 leader_election.cc:290] T a644c02b16af47679824ccd060ac10fd P 7a9daf5d42a84615a6383cfec9a10608 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 974b802141bd4f13beffef2aa03f88f9 (127.18.211.2:34517), 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3:37735)
I20260129 03:00:21.668505 20586 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "724a236524cb4e7ba3428880681d6cc2" candidate_uuid: "197b880fa39d432a9b472845876bfaf0" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "1c50de89a5a845668696dcb28c3badd6" is_pre_election: true
I20260129 03:00:21.677265 20508 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "75be4bd6bd45474795fd472eff2953ac" candidate_uuid: "1c50de89a5a845668696dcb28c3badd6" candidate_term: 11 candidate_status { last_received { term: 10 index: 7 } } ignore_live_leader: false dest_uuid: "974b802141bd4f13beffef2aa03f88f9" is_pre_election: true
I20260129 03:00:21.686754 20586 raft_consensus.cc:2393] T 724a236524cb4e7ba3428880681d6cc2 P 1c50de89a5a845668696dcb28c3badd6 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 197b880fa39d432a9b472845876bfaf0 in current term 7: Already voted for candidate 974b802141bd4f13beffef2aa03f88f9 in this term.
I20260129 03:00:21.689391 23627 leader_election.cc:290] T a0fcc58d6fb6473cafbd9d34d5195269 P 197b880fa39d432a9b472845876bfaf0 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 974b802141bd4f13beffef2aa03f88f9 (127.18.211.2:34517), 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3:37735)
W20260129 03:00:21.692329 23821 log.cc:927] Time spent T 926a250fa5834e88b96ccecbefbf5086 P 974b802141bd4f13beffef2aa03f88f9: Append to log took a long time: real 0.061s user 0.000s sys 0.000s
I20260129 03:00:21.694689 20586 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a1a39dcd328e4268ae41b988c66b4a75" candidate_uuid: "197b880fa39d432a9b472845876bfaf0" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "1c50de89a5a845668696dcb28c3badd6"
I20260129 03:00:21.695176 23403 leader_election.cc:290] T 42d27a28d89b409e8807ec0efd97af69 P 197b880fa39d432a9b472845876bfaf0 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3:37735), 7a9daf5d42a84615a6383cfec9a10608 (127.18.211.4:40439)
I20260129 03:00:21.698165 20586 raft_consensus.cc:2393] T a1a39dcd328e4268ae41b988c66b4a75 P 1c50de89a5a845668696dcb28c3badd6 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 197b880fa39d432a9b472845876bfaf0 in current term 7: Already voted for candidate 974b802141bd4f13beffef2aa03f88f9 in this term.
W20260129 03:00:21.700888 23730 consensus_meta.cc:287] T 7a9a69cd82724b1faed1d10aa091fc4a P 197b880fa39d432a9b472845876bfaf0: Time spent flushing consensus metadata: real 0.820s user 0.006s sys 0.005s
W20260129 03:00:21.701117 23693 consensus_meta.cc:287] T e7d83d4ccf684e76acb29a0dc844e1b8 P 974b802141bd4f13beffef2aa03f88f9: Time spent flushing consensus metadata: real 0.755s user 0.009s sys 0.001s
I20260129 03:00:21.700863 20586 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4b1876f9a70b49edb3aa74105e66d3b3" candidate_uuid: "197b880fa39d432a9b472845876bfaf0" candidate_term: 8 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "1c50de89a5a845668696dcb28c3badd6" is_pre_election: true
I20260129 03:00:21.702390 20510 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a0fcc58d6fb6473cafbd9d34d5195269" candidate_uuid: "197b880fa39d432a9b472845876bfaf0" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "974b802141bd4f13beffef2aa03f88f9" is_pre_election: true
I20260129 03:00:21.637849 23810 raft_consensus.cc:493] T fd8311b779af4e1fa1402b2a02a602f2 P 197b880fa39d432a9b472845876bfaf0 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260129 03:00:21.704057 20405 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.211.3:37735, user_credentials={real_user=slave}} blocked reactor thread for 64147us
W20260129 03:00:21.704360 20441 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.554s user 0.005s sys 0.014s
W20260129 03:00:21.722280 20441 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.572s user 0.005s sys 0.014s
I20260129 03:00:21.722241 20405 leader_election.cc:400] T 3906b080a5b34a74b8b64128635e8997 P 197b880fa39d432a9b472845876bfaf0 [CANDIDATE]: Term 5 election: Vote denied by peer 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3:37735) with higher term. Message: Invalid argument: T 3906b080a5b34a74b8b64128635e8997 P 1c50de89a5a845668696dcb28c3badd6 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 197b880fa39d432a9b472845876bfaf0 for earlier term 5. Current term is 6.
W20260129 03:00:21.708757 23165 consensus_meta.cc:287] T 75f45538dc704d9a8ec4ff8ad9f3ea5b P 7a9daf5d42a84615a6383cfec9a10608: Time spent flushing consensus metadata: real 0.538s user 0.010s sys 0.000s
W20260129 03:00:21.710685 23804 log.cc:927] Time spent T cb0b3f259ebc4310927338a8740e87b8 P 1c50de89a5a845668696dcb28c3badd6: Append to log took a long time: real 0.259s user 0.000s sys 0.001s
I20260129 03:00:21.677790 23514 leader_election.cc:290] T c934d31efd3f4e5195605ddc354d14e1 P 974b802141bd4f13beffef2aa03f88f9 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 197b880fa39d432a9b472845876bfaf0 (127.18.211.1:42027), 7a9daf5d42a84615a6383cfec9a10608 (127.18.211.4:40439)
I20260129 03:00:21.711465 23840 raft_consensus.cc:2749] T 4e3805ffa70242e8ade66da260dddf66 P 197b880fa39d432a9b472845876bfaf0 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260129 03:00:21.714051 23799 leader_election.cc:290] T fd8311b779af4e1fa1402b2a02a602f2 P 1c50de89a5a845668696dcb28c3badd6 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 197b880fa39d432a9b472845876bfaf0 (127.18.211.1:42027), 7a9daf5d42a84615a6383cfec9a10608 (127.18.211.4:40439)
I20260129 03:00:21.717096 23398 leader_election.cc:290] T 18f30ebaa4174fd28c958e72d0cc7e24 P 1c50de89a5a845668696dcb28c3badd6 [CANDIDATE]: Term 9 election: Requested vote from peers 197b880fa39d432a9b472845876bfaf0 (127.18.211.1:42027), 7a9daf5d42a84615a6383cfec9a10608 (127.18.211.4:40439)
W20260129 03:00:21.670534 23805 log.cc:927] Time spent T add8d4923d1a489e86882c3a25692b95 P 1c50de89a5a845668696dcb28c3badd6: Append to log took a long time: real 0.265s user 0.001s sys 0.000s
W20260129 03:00:21.718770 23800 consensus_meta.cc:287] T 1ce4ddf5b10e4ae8b193b13b2e5936ee P 1c50de89a5a845668696dcb28c3badd6: Time spent flushing consensus metadata: real 0.535s user 0.009s sys 0.000s
W20260129 03:00:21.719873 20555 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260129 03:00:21.679011 23803 consensus_queue.cc:237] T c891d2bff4ce4568861bbc992e91fbb0 P 1c50de89a5a845668696dcb28c3badd6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 6.5, Last appended by leader: 5, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
W20260129 03:00:21.720506 23814 log.cc:927] Time spent T a1a39dcd328e4268ae41b988c66b4a75 P 974b802141bd4f13beffef2aa03f88f9: Append to log took a long time: real 0.207s user 0.000s sys 0.000s
I20260129 03:00:21.724690 20405 leader_election.cc:304] T 1299723a4476420ab02fd8bfd7aa0645 P 197b880fa39d432a9b472845876bfaf0 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 197b880fa39d432a9b472845876bfaf0; no voters: 1c50de89a5a845668696dcb28c3badd6, 7a9daf5d42a84615a6383cfec9a10608
I20260129 03:00:21.732335 20510 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4408006eb5174de1896b037cf3f18457" candidate_uuid: "1c50de89a5a845668696dcb28c3badd6" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "974b802141bd4f13beffef2aa03f88f9" is_pre_election: true
I20260129 03:00:21.719800 23701 raft_consensus.cc:697] T c891d2bff4ce4568861bbc992e91fbb0 P 197b880fa39d432a9b472845876bfaf0 [term 10 LEADER]: Becoming Leader. State: Replica: 197b880fa39d432a9b472845876bfaf0, State: Running, Role: LEADER
W20260129 03:00:21.747077 23806 log.cc:927] Time spent T 7a9a69cd82724b1faed1d10aa091fc4a P 974b802141bd4f13beffef2aa03f88f9: Append to log took a long time: real 0.305s user 0.000s sys 0.001s
I20260129 03:00:21.744719 20440 raft_consensus.cc:1275] T a1a39dcd328e4268ae41b988c66b4a75 P 197b880fa39d432a9b472845876bfaf0 [term 7 FOLLOWER]: Refusing update from remote peer 974b802141bd4f13beffef2aa03f88f9: Log matching property violated. Preceding OpId in replica: term: 5 index: 4. Preceding OpId from leader: term: 7 index: 5. (index mismatch)
I20260129 03:00:21.745496 23669 consensus_queue.cc:1048] T c2633a6fc34c44579de348d208b0c01e P 1c50de89a5a845668696dcb28c3badd6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260129 03:00:21.746117 23680 consensus_queue.cc:1048] T 75f45538dc704d9a8ec4ff8ad9f3ea5b P 1c50de89a5a845668696dcb28c3badd6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
W20260129 03:00:21.708855 23798 consensus_meta.cc:287] T 36ff33be666a4c82954731691dbcf327 P 1c50de89a5a845668696dcb28c3badd6: Time spent flushing consensus metadata: real 0.567s user 0.010s sys 0.000s
I20260129 03:00:21.751345 20508 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fcddab6bca684adf8ea032289a00f90e" candidate_uuid: "1c50de89a5a845668696dcb28c3badd6" candidate_term: 11 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "974b802141bd4f13beffef2aa03f88f9" is_pre_election: true
I20260129 03:00:21.696259 23466 raft_consensus.cc:697] T fcddab6bca684adf8ea032289a00f90e P 197b880fa39d432a9b472845876bfaf0 [term 11 LEADER]: Becoming Leader. State: Replica: 197b880fa39d432a9b472845876bfaf0, State: Running, Role: LEADER
I20260129 03:00:21.628698 23791 raft_consensus.cc:515] T 00e79d8f25a34676b1d866eb2068a2b2 P 7a9daf5d42a84615a6383cfec9a10608 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } }
I20260129 03:00:21.737293 20556 leader_election.cc:304] T 0340229d521b432580ae926f6062e3f7 P 1c50de89a5a845668696dcb28c3badd6 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1c50de89a5a845668696dcb28c3badd6; no voters: 197b880fa39d432a9b472845876bfaf0, 974b802141bd4f13beffef2aa03f88f9
W20260129 03:00:21.822245 20556 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.211.2:34517, user_credentials={real_user=slave}} blocked reactor thread for 85074.6us
W20260129 03:00:21.791435 20405 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.211.3:37735, user_credentials={real_user=slave}} blocked reactor thread for 66856.3us
I20260129 03:00:21.823098 20556 leader_election.cc:304] T 45cc5f3d2688486d9b6dbe7e666dedde P 1c50de89a5a845668696dcb28c3badd6 [CANDIDATE]: Term 8 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1c50de89a5a845668696dcb28c3badd6; no voters: 7a9daf5d42a84615a6383cfec9a10608, 974b802141bd4f13beffef2aa03f88f9
I20260129 03:00:21.823697 20405 leader_election.cc:304] T e7d83d4ccf684e76acb29a0dc844e1b8 P 197b880fa39d432a9b472845876bfaf0 [CANDIDATE]: Term 8 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 197b880fa39d432a9b472845876bfaf0; no voters: 1c50de89a5a845668696dcb28c3badd6, 974b802141bd4f13beffef2aa03f88f9
I20260129 03:00:21.825225 20405 leader_election.cc:304] T 0f20af7fbe5d4db89a6f17948d98912c P 197b880fa39d432a9b472845876bfaf0 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 197b880fa39d432a9b472845876bfaf0; no voters: 1c50de89a5a845668696dcb28c3badd6, 974b802141bd4f13beffef2aa03f88f9
I20260129 03:00:21.826207 20556 leader_election.cc:304] T 75be4bd6bd45474795fd472eff2953ac P 1c50de89a5a845668696dcb28c3badd6 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1c50de89a5a845668696dcb28c3badd6; no voters: 197b880fa39d432a9b472845876bfaf0, 974b802141bd4f13beffef2aa03f88f9
I20260129 03:00:21.801024 23701 consensus_queue.cc:237] T c891d2bff4ce4568861bbc992e91fbb0 P 197b880fa39d432a9b472845876bfaf0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 7.6, Last appended by leader: 6, Current term: 10, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
W20260129 03:00:21.786646 23816 log.cc:927] Time spent T 2e58e9a5a8aa4cecafb464addb4f35ed P 7a9daf5d42a84615a6383cfec9a10608: Append to log took a long time: real 0.307s user 0.001s sys 0.000s
I20260129 03:00:21.781349 23800 raft_consensus.cc:515] T 1ce4ddf5b10e4ae8b193b13b2e5936ee P 1c50de89a5a845668696dcb28c3badd6 [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 03:00:21.816905 23627 raft_consensus.cc:2749] T 1299723a4476420ab02fd8bfd7aa0645 P 197b880fa39d432a9b472845876bfaf0 [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20260129 03:00:21.790483 23398 raft_consensus.cc:493] T 4f5a77e94fef4b16a2ba7b067c8b144a P 1c50de89a5a845668696dcb28c3badd6 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 7a9daf5d42a84615a6383cfec9a10608)
I20260129 03:00:21.735195 23810 raft_consensus.cc:515] T fd8311b779af4e1fa1402b2a02a602f2 P 197b880fa39d432a9b472845876bfaf0 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } }
W20260129 03:00:21.772609 23485 consensus_meta.cc:287] T 4b1876f9a70b49edb3aa74105e66d3b3 P 197b880fa39d432a9b472845876bfaf0: Time spent flushing consensus metadata: real 0.598s user 0.010s sys 0.000s
I20260129 03:00:21.805713 23807 consensus_queue.cc:237] T 17d6431cc77a491e8464d68db4081a51 P 197b880fa39d432a9b472845876bfaf0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 5.3, Last appended by leader: 3, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 03:00:21.812086 20585 raft_consensus.cc:2410] T 75b0b6e82ddc4f92b1fe1d94f9a0b73b P 1c50de89a5a845668696dcb28c3badd6 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 7a9daf5d42a84615a6383cfec9a10608 for term 6 because replica has last-logged OpId of term: 5 index: 8, which is greater than that of the candidate, which has last-logged OpId of term: 4 index: 7.
I20260129 03:00:21.859522 20582 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a644c02b16af47679824ccd060ac10fd" candidate_uuid: "7a9daf5d42a84615a6383cfec9a10608" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "1c50de89a5a845668696dcb28c3badd6" is_pre_election: true
W20260129 03:00:21.813233 20406 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260129 03:00:21.817981 20508 raft_consensus.cc:2468] T fcddab6bca684adf8ea032289a00f90e P 974b802141bd4f13beffef2aa03f88f9 [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1c50de89a5a845668696dcb28c3badd6 in term 11.
I20260129 03:00:21.800539 20440 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "18f30ebaa4174fd28c958e72d0cc7e24" candidate_uuid: "1c50de89a5a845668696dcb28c3badd6" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "197b880fa39d432a9b472845876bfaf0"
I20260129 03:00:21.834538 20405 leader_election.cc:304] T 913fc7d751ed4b7388d889aac30a5f9b P 197b880fa39d432a9b472845876bfaf0 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 197b880fa39d432a9b472845876bfaf0; no voters: 1c50de89a5a845668696dcb28c3badd6, 974b802141bd4f13beffef2aa03f88f9
W20260129 03:00:21.844985 20589 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.660s user 0.010s sys 0.025s
I20260129 03:00:21.744134 23165 raft_consensus.cc:515] T 75f45538dc704d9a8ec4ff8ad9f3ea5b P 7a9daf5d42a84615a6383cfec9a10608 [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } }
I20260129 03:00:21.860368 20582 raft_consensus.cc:2468] T a644c02b16af47679824ccd060ac10fd P 1c50de89a5a845668696dcb28c3badd6 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7a9daf5d42a84615a6383cfec9a10608 in term 6.
I20260129 03:00:21.865885 20586 raft_consensus.cc:1240] T 1ce4ddf5b10e4ae8b193b13b2e5936ee P 1c50de89a5a845668696dcb28c3badd6 [term 8 FOLLOWER]: Rejecting Update request from peer 197b880fa39d432a9b472845876bfaf0 for earlier term 7. Current term is 8. Ops: []
I20260129 03:00:21.850646 20590 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:48526 (request call id 470) took 1079 ms. Trace:
I20260129 03:00:21.865810 20582 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "36ff33be666a4c82954731691dbcf327" candidate_uuid: "974b802141bd4f13beffef2aa03f88f9" candidate_term: 6 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "1c50de89a5a845668696dcb28c3badd6" is_pre_election: true
W20260129 03:00:21.863394 20589 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.679s user 0.010s sys 0.025s
I20260129 03:00:21.863204 20405 leader_election.cc:304] T 724a236524cb4e7ba3428880681d6cc2 P 197b880fa39d432a9b472845876bfaf0 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 197b880fa39d432a9b472845876bfaf0; no voters: 1c50de89a5a845668696dcb28c3badd6, 974b802141bd4f13beffef2aa03f88f9
I20260129 03:00:21.851974 20556 leader_election.cc:304] T 4408006eb5174de1896b037cf3f18457 P 1c50de89a5a845668696dcb28c3badd6 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1c50de89a5a845668696dcb28c3badd6; no voters: 197b880fa39d432a9b472845876bfaf0, 974b802141bd4f13beffef2aa03f88f9
I20260129 03:00:21.871553 20437 raft_consensus.cc:1275] T 7a9a69cd82724b1faed1d10aa091fc4a P 197b880fa39d432a9b472845876bfaf0 [term 8 FOLLOWER]: Refusing update from remote peer 974b802141bd4f13beffef2aa03f88f9: Log matching property violated. Preceding OpId in replica: term: 7 index: 5. Preceding OpId from leader: term: 8 index: 6. (index mismatch)
I20260129 03:00:21.828677 23836 raft_consensus.cc:2749] T e7d83d4ccf684e76acb29a0dc844e1b8 P 197b880fa39d432a9b472845876bfaf0 [term 8 FOLLOWER]: Leader election lost for term 8. Reason: could not achieve majority
W20260129 03:00:21.874620 21102 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.211.2:34517, user_credentials={real_user=slave}} blocked reactor thread for 460933us
I20260129 03:00:21.867023 20590 rpcz_store.cc:276] 0129 03:00:20.771165 (+ 0us) service_pool.cc:168] Inserting onto call queue
0129 03:00:20.772602 (+ 1437us) service_pool.cc:225] Handling call
0129 03:00:20.773052 (+ 450us) raft_consensus.cc:1471] Updating replica for 1 ops
0129 03:00:20.789849 (+ 16797us) raft_consensus.cc:1534] Early marking committed up to index 0
0129 03:00:20.789899 (+ 50us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0129 03:00:20.802051 (+ 12152us) log.cc:844] Serialized 26 byte log entry
0129 03:00:21.096585 (+294534us) raft_consensus.cc:1645] Marking committed up to 9
0129 03:00:21.403477 (+306892us) log.cc:844] Serialized 14 byte log entry
0129 03:00:21.434820 (+ 31343us) raft_consensus.cc:1697] Filling consensus response to leader.
0129 03:00:21.434927 (+ 107us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0129 03:00:21.850405 (+415478us) raft_consensus.cc:1685] finished
0129 03:00:21.850427 (+ 22us) raft_consensus.cc:1691] UpdateReplicas() finished
0129 03:00:21.850578 (+ 151us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":27786,"prepare.queue_time_us":424024,"prepare.run_cpu_time_us":29,"prepare.run_wall_time_us":29,"raft.queue_time_us":74079,"raft.run_cpu_time_us":301,"raft.run_wall_time_us":31078,"thread_start_us":529067,"threads_started":2,"wal-append.queue_time_us":516746}
I20260129 03:00:21.880843 20582 raft_consensus.cc:1240] T a0fcc58d6fb6473cafbd9d34d5195269 P 1c50de89a5a845668696dcb28c3badd6 [term 6 FOLLOWER]: Rejecting Update request from peer 197b880fa39d432a9b472845876bfaf0 for earlier term 5. Current term is 6. Ops: [5.4-5.4]
I20260129 03:00:21.845005 23766 raft_consensus.cc:2749] T 0340229d521b432580ae926f6062e3f7 P 1c50de89a5a845668696dcb28c3badd6 [term 9 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260129 03:00:21.833177 23695 leader_election.cc:290] T 36ff33be666a4c82954731691dbcf327 P 974b802141bd4f13beffef2aa03f88f9 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 197b880fa39d432a9b472845876bfaf0 (127.18.211.1:42027), 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3:37735)
I20260129 03:00:21.876960 20441 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fd8311b779af4e1fa1402b2a02a602f2" candidate_uuid: "1c50de89a5a845668696dcb28c3badd6" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "197b880fa39d432a9b472845876bfaf0" is_pre_election: true
I20260129 03:00:21.851330 23799 raft_consensus.cc:2749] T 45cc5f3d2688486d9b6dbe7e666dedde P 1c50de89a5a845668696dcb28c3badd6 [term 8 FOLLOWER]: Leader election lost for term 8. Reason: could not achieve majority
I20260129 03:00:21.850984 23841 raft_consensus.cc:2749] T 0f20af7fbe5d4db89a6f17948d98912c P 197b880fa39d432a9b472845876bfaf0 [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260129 03:00:21.887064 20582 raft_consensus.cc:1275] T 7d9494f5584d4d5db8b9adf1c8fc888b P 1c50de89a5a845668696dcb28c3badd6 [term 5 FOLLOWER]: Refusing update from remote peer 197b880fa39d432a9b472845876bfaf0: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 5 index: 4. (index mismatch)
I20260129 03:00:21.887822 21133 raft_consensus.cc:1240] T 75f45538dc704d9a8ec4ff8ad9f3ea5b P 7a9daf5d42a84615a6383cfec9a10608 [term 6 FOLLOWER]: Rejecting Update request from peer 1c50de89a5a845668696dcb28c3badd6 for earlier term 5. Current term is 6. Ops: []
I20260129 03:00:21.888533 20441 raft_consensus.cc:2468] T fd8311b779af4e1fa1402b2a02a602f2 P 197b880fa39d432a9b472845876bfaf0 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1c50de89a5a845668696dcb28c3badd6 in term 6.
I20260129 03:00:21.859961 23812 raft_consensus.cc:2749] T 75be4bd6bd45474795fd472eff2953ac P 1c50de89a5a845668696dcb28c3badd6 [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20260129 03:00:21.876248 23602 raft_consensus.cc:2749] T 4408006eb5174de1896b037cf3f18457 P 1c50de89a5a845668696dcb28c3badd6 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260129 03:00:21.832796 23466 consensus_queue.cc:237] T fcddab6bca684adf8ea032289a00f90e P 197b880fa39d432a9b472845876bfaf0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 10.6, Last appended by leader: 6, Current term: 11, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 03:00:21.864632 23485 raft_consensus.cc:515] T 4b1876f9a70b49edb3aa74105e66d3b3 P 197b880fa39d432a9b472845876bfaf0 [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 03:00:21.897367 23165 leader_election.cc:290] T 75f45538dc704d9a8ec4ff8ad9f3ea5b P 7a9daf5d42a84615a6383cfec9a10608 [CANDIDATE]: Term 6 election: Requested vote from peers 974b802141bd4f13beffef2aa03f88f9 (127.18.211.2:34517), 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3:37735)
I20260129 03:00:21.872406 23627 raft_consensus.cc:2749] T 913fc7d751ed4b7388d889aac30a5f9b P 197b880fa39d432a9b472845876bfaf0 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260129 03:00:21.898726 23165 raft_consensus.cc:493] T fd8311b779af4e1fa1402b2a02a602f2 P 7a9daf5d42a84615a6383cfec9a10608 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260129 03:00:21.899139 23485 leader_election.cc:290] T 4b1876f9a70b49edb3aa74105e66d3b3 P 197b880fa39d432a9b472845876bfaf0 [CANDIDATE]: Term 8 election: Requested vote from peers 974b802141bd4f13beffef2aa03f88f9 (127.18.211.2:34517), 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3:37735)
I20260129 03:00:21.900074 20435 raft_consensus.cc:1240] T 4b1876f9a70b49edb3aa74105e66d3b3 P 197b880fa39d432a9b472845876bfaf0 [term 8 FOLLOWER]: Rejecting Update request from peer 1c50de89a5a845668696dcb28c3badd6 for earlier term 7. Current term is 8. Ops: []
I20260129 03:00:21.899206 23165 raft_consensus.cc:515] T fd8311b779af4e1fa1402b2a02a602f2 P 7a9daf5d42a84615a6383cfec9a10608 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } }
I20260129 03:00:21.902621 20589 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:48526 (request call id 469) took 1131 ms. Trace:
I20260129 03:00:21.826511 23798 raft_consensus.cc:515] T 36ff33be666a4c82954731691dbcf327 P 1c50de89a5a845668696dcb28c3badd6 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 03:00:21.904452 20589 rpcz_store.cc:276] 0129 03:00:20.770867 (+ 0us) service_pool.cc:168] Inserting onto call queue
0129 03:00:20.771050 (+ 183us) service_pool.cc:225] Handling call
0129 03:00:20.771533 (+ 483us) raft_consensus.cc:1471] Updating replica for 2 ops
0129 03:00:20.772028 (+ 495us) raft_consensus.cc:1534] Early marking committed up to index 0
0129 03:00:20.772070 (+ 42us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0129 03:00:20.797008 (+ 24938us) log.cc:844] Serialized 52 byte log entry
0129 03:00:21.117265 (+320257us) raft_consensus.cc:1645] Marking committed up to 6
0129 03:00:21.867312 (+750047us) log.cc:844] Serialized 14 byte log entry
0129 03:00:21.867523 (+ 211us) raft_consensus.cc:1697] Filling consensus response to leader.
0129 03:00:21.867591 (+ 68us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0129 03:00:21.902405 (+ 34814us) raft_consensus.cc:1685] finished
0129 03:00:21.902428 (+ 23us) raft_consensus.cc:1691] UpdateReplicas() finished
0129 03:00:21.902554 (+ 126us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":11257,"prepare.queue_time_us":408792,"prepare.run_cpu_time_us":37,"prepare.run_wall_time_us":37,"raft.queue_time_us":25017,"raft.run_cpu_time_us":298,"raft.run_wall_time_us":23972,"thread_start_us":933725,"threads_started":2,"wal-append.queue_time_us":483512}
I20260129 03:00:21.910607 20589 raft_consensus.cc:1240] T 926a250fa5834e88b96ccecbefbf5086 P 1c50de89a5a845668696dcb28c3badd6 [term 9 FOLLOWER]: Rejecting Update request from peer 197b880fa39d432a9b472845876bfaf0 for earlier term 8. Current term is 9. Ops: [5.4-8.5]
W20260129 03:00:21.932452 21102 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.211.2:34517, user_credentials={real_user=slave}} blocked reactor thread for 52018.4us
I20260129 03:00:21.876303 23398 raft_consensus.cc:515] T 4f5a77e94fef4b16a2ba7b067c8b144a P 1c50de89a5a845668696dcb28c3badd6 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } }
I20260129 03:00:21.935457 20584 raft_consensus.cc:2368] T 36ff33be666a4c82954731691dbcf327 P 1c50de89a5a845668696dcb28c3badd6 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 197b880fa39d432a9b472845876bfaf0 for earlier term 5. Current term is 6.
I20260129 03:00:21.901262 23730 raft_consensus.cc:2749] T 724a236524cb4e7ba3428880681d6cc2 P 197b880fa39d432a9b472845876bfaf0 [term 7 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260129 03:00:21.895778 23836 raft_consensus.cc:493] T cb4727386a3e48aca5d71cc6c611c197 P 197b880fa39d432a9b472845876bfaf0 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260129 03:00:21.895403 23840 raft_consensus.cc:493] T 00e79d8f25a34676b1d866eb2068a2b2 P 197b880fa39d432a9b472845876bfaf0 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260129 03:00:21.940873 21102 leader_election.cc:304] T 75b0b6e82ddc4f92b1fe1d94f9a0b73b P 7a9daf5d42a84615a6383cfec9a10608 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7a9daf5d42a84615a6383cfec9a10608, 974b802141bd4f13beffef2aa03f88f9; no voters:
I20260129 03:00:21.972168 23829 raft_consensus.cc:2804] T 75b0b6e82ddc4f92b1fe1d94f9a0b73b P 7a9daf5d42a84615a6383cfec9a10608 [term 6 FOLLOWER]: Leader election won for term 6
I20260129 03:00:21.972568 23810 leader_election.cc:290] T fd8311b779af4e1fa1402b2a02a602f2 P 197b880fa39d432a9b472845876bfaf0 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3:37735), 7a9daf5d42a84615a6383cfec9a10608 (127.18.211.4:40439)
I20260129 03:00:21.973037 23829 raft_consensus.cc:697] T 75b0b6e82ddc4f92b1fe1d94f9a0b73b P 7a9daf5d42a84615a6383cfec9a10608 [term 6 LEADER]: Becoming Leader. State: Replica: 7a9daf5d42a84615a6383cfec9a10608, State: Running, Role: LEADER
I20260129 03:00:21.974085 23671 raft_consensus.cc:3060] T 17d6431cc77a491e8464d68db4081a51 P 974b802141bd4f13beffef2aa03f88f9 [term 6 FOLLOWER]: Advancing to term 7
I20260129 03:00:21.974296 23829 consensus_queue.cc:237] T 75b0b6e82ddc4f92b1fe1d94f9a0b73b P 7a9daf5d42a84615a6383cfec9a10608 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 4.7, Last appended by leader: 7, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } }
I20260129 03:00:21.976768 23809 leader_election.cc:290] T 96609ac68bdf4993a12e1c23355ca013 P 197b880fa39d432a9b472845876bfaf0 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 974b802141bd4f13beffef2aa03f88f9 (127.18.211.2:34517), 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3:37735), 7a9daf5d42a84615a6383cfec9a10608 (127.18.211.4:40439)
I20260129 03:00:21.980775 23798 leader_election.cc:290] T 36ff33be666a4c82954731691dbcf327 P 1c50de89a5a845668696dcb28c3badd6 [CANDIDATE]: Term 6 election: Requested vote from peers 197b880fa39d432a9b472845876bfaf0 (127.18.211.1:42027), 974b802141bd4f13beffef2aa03f88f9 (127.18.211.2:34517)
I20260129 03:00:21.981570 23836 raft_consensus.cc:515] T cb4727386a3e48aca5d71cc6c611c197 P 197b880fa39d432a9b472845876bfaf0 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
W20260129 03:00:21.982517 23849 log.cc:927] Time spent T cedce331e1584ac8b636c9b8a72b2ba0 P 197b880fa39d432a9b472845876bfaf0: Append to log took a long time: real 0.079s user 0.001s sys 0.000s
W20260129 03:00:21.982656 20585 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:48526 (request call id 468) took 1212 ms (client timeout 1601 ms). Trace:
I20260129 03:00:21.984540 23791 leader_election.cc:290] T 00e79d8f25a34676b1d866eb2068a2b2 P 7a9daf5d42a84615a6383cfec9a10608 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 197b880fa39d432a9b472845876bfaf0 (127.18.211.1:42027), 974b802141bd4f13beffef2aa03f88f9 (127.18.211.2:34517)
W20260129 03:00:21.983253 20585 rpcz_store.cc:269] 0129 03:00:20.770465 (+ 0us) service_pool.cc:168] Inserting onto call queue
0129 03:00:20.772707 (+ 2242us) service_pool.cc:225] Handling call
0129 03:00:20.968648 (+195941us) spinlock_profiling.cc:243] Waited 1.51 ms on lock 0x7b58001cb398. stack: 00007f8fe044e373 00007f8fe044e270 00007f8fdfda0574 00007f8fe9a5c6b4 00007f8fe9a5c649 00007f8fe02c08b1 00007f8fe0442db1 00007f8fe0442c62 00007f8fe92eddd9 00007f8fe846d260 00007f8fe8535dbb 00007f8fe8535e41 00007f8fe84e3bd0 00007f8fe84e34b4 00007f8fe1a73834 00007f8fe1a737c2
0129 03:00:21.982390 (+1013742us) spinlock_profiling.cc:243] Waited 842 ms on lock 0x7b540003f2d4. stack: 00007f8fe044e373 00007f8fe044e270 00007f8fdfda0574 00007f8fe9a5c6b4 00007f8fe9a5c649 00007f8fe34975a1 00007f8fe3531f7a 00007f8fe84e3647 00007f8fe1a73834 00007f8fe1a737c2 00007f8fe1a736f1 00007f8fe1a7366c 00007f8fe1a72922 00007f8fe186d34c 00007f8fe186c786 00007f8fe186c12f
0129 03:00:21.982576 (+ 186us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1940257408}
W20260129 03:00:21.989073 21102 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.211.2:34517, user_credentials={real_user=slave}} blocked reactor thread for 48294.3us
W20260129 03:00:22.014078 23851 log.cc:927] Time spent T 45cc5f3d2688486d9b6dbe7e666dedde P 7a9daf5d42a84615a6383cfec9a10608: Append to log took a long time: real 0.069s user 0.002s sys 0.000s
I20260129 03:00:21.983798 23836 leader_election.cc:290] T cb4727386a3e48aca5d71cc6c611c197 P 197b880fa39d432a9b472845876bfaf0 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 974b802141bd4f13beffef2aa03f88f9 (127.18.211.2:34517), 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3:37735)
I20260129 03:00:21.984930 23840 raft_consensus.cc:515] T 00e79d8f25a34676b1d866eb2068a2b2 P 197b880fa39d432a9b472845876bfaf0 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } }
I20260129 03:00:22.022662 20507 raft_consensus.cc:1275] T e33a417cf9c54cf5af73a4ee3aac4d3a P 974b802141bd4f13beffef2aa03f88f9 [term 8 FOLLOWER]: Refusing update from remote peer 1c50de89a5a845668696dcb28c3badd6: Log matching property violated. Preceding OpId in replica: term: 6 index: 3. Preceding OpId from leader: term: 8 index: 4. (index mismatch)
I20260129 03:00:22.023265 23840 leader_election.cc:290] T 00e79d8f25a34676b1d866eb2068a2b2 P 197b880fa39d432a9b472845876bfaf0 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 974b802141bd4f13beffef2aa03f88f9 (127.18.211.2:34517), 7a9daf5d42a84615a6383cfec9a10608 (127.18.211.4:40439)
I20260129 03:00:22.038283 20434 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "36ff33be666a4c82954731691dbcf327" candidate_uuid: "1c50de89a5a845668696dcb28c3badd6" candidate_term: 6 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "197b880fa39d432a9b472845876bfaf0"
I20260129 03:00:22.039099 20434 raft_consensus.cc:3060] T 36ff33be666a4c82954731691dbcf327 P 197b880fa39d432a9b472845876bfaf0 [term 5 FOLLOWER]: Advancing to term 6
I20260129 03:00:22.052395 23800 leader_election.cc:290] T 1ce4ddf5b10e4ae8b193b13b2e5936ee P 1c50de89a5a845668696dcb28c3badd6 [CANDIDATE]: Term 8 election: Requested vote from peers 197b880fa39d432a9b472845876bfaf0 (127.18.211.1:42027), 974b802141bd4f13beffef2aa03f88f9 (127.18.211.2:34517)
I20260129 03:00:22.052726 20553 leader_election.cc:304] T fd8311b779af4e1fa1402b2a02a602f2 P 1c50de89a5a845668696dcb28c3badd6 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 197b880fa39d432a9b472845876bfaf0, 1c50de89a5a845668696dcb28c3badd6; no voters:
I20260129 03:00:22.053442 23398 leader_election.cc:290] T 4f5a77e94fef4b16a2ba7b067c8b144a P 1c50de89a5a845668696dcb28c3badd6 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 974b802141bd4f13beffef2aa03f88f9 (127.18.211.2:34517), 7a9daf5d42a84615a6383cfec9a10608 (127.18.211.4:40439)
I20260129 03:00:22.054054 23398 raft_consensus.cc:2804] T fd8311b779af4e1fa1402b2a02a602f2 P 1c50de89a5a845668696dcb28c3badd6 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260129 03:00:22.054399 23398 raft_consensus.cc:493] T fd8311b779af4e1fa1402b2a02a602f2 P 1c50de89a5a845668696dcb28c3badd6 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260129 03:00:22.054741 23398 raft_consensus.cc:3060] T fd8311b779af4e1fa1402b2a02a602f2 P 1c50de89a5a845668696dcb28c3badd6 [term 6 FOLLOWER]: Advancing to term 7
I20260129 03:00:22.027753 23165 leader_election.cc:290] T fd8311b779af4e1fa1402b2a02a602f2 P 7a9daf5d42a84615a6383cfec9a10608 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 197b880fa39d432a9b472845876bfaf0 (127.18.211.1:42027), 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3:37735)
W20260129 03:00:22.139001 20555 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260129 03:00:22.219655 20441 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1ce4ddf5b10e4ae8b193b13b2e5936ee" candidate_uuid: "1c50de89a5a845668696dcb28c3badd6" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "197b880fa39d432a9b472845876bfaf0"
I20260129 03:00:22.230240 20434 raft_consensus.cc:2468] T 36ff33be666a4c82954731691dbcf327 P 197b880fa39d432a9b472845876bfaf0 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1c50de89a5a845668696dcb28c3badd6 in term 6.
I20260129 03:00:22.231384 20553 leader_election.cc:304] T 36ff33be666a4c82954731691dbcf327 P 1c50de89a5a845668696dcb28c3badd6 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 197b880fa39d432a9b472845876bfaf0, 1c50de89a5a845668696dcb28c3badd6; no voters:
I20260129 03:00:22.233604 23798 raft_consensus.cc:2804] T 36ff33be666a4c82954731691dbcf327 P 1c50de89a5a845668696dcb28c3badd6 [term 6 FOLLOWER]: Leader election won for term 6
I20260129 03:00:22.234210 23798 raft_consensus.cc:697] T 36ff33be666a4c82954731691dbcf327 P 1c50de89a5a845668696dcb28c3badd6 [term 6 LEADER]: Becoming Leader. State: Replica: 1c50de89a5a845668696dcb28c3badd6, State: Running, Role: LEADER
I20260129 03:00:22.235157 23798 consensus_queue.cc:237] T 36ff33be666a4c82954731691dbcf327 P 1c50de89a5a845668696dcb28c3badd6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 4.4, Last appended by leader: 4, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 03:00:22.246239 21100 leader_election.cc:304] T a644c02b16af47679824ccd060ac10fd P 7a9daf5d42a84615a6383cfec9a10608 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1c50de89a5a845668696dcb28c3badd6, 7a9daf5d42a84615a6383cfec9a10608; no voters:
W20260129 03:00:22.254036 20405 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.211.3:37735, user_credentials={real_user=slave}} blocked reactor thread for 390839us
I20260129 03:00:22.255137 20405 leader_election.cc:304] T a1a39dcd328e4268ae41b988c66b4a75 P 197b880fa39d432a9b472845876bfaf0 [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 197b880fa39d432a9b472845876bfaf0; no voters: 1c50de89a5a845668696dcb28c3badd6, 974b802141bd4f13beffef2aa03f88f9
I20260129 03:00:22.260037 20434 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fd8311b779af4e1fa1402b2a02a602f2" candidate_uuid: "7a9daf5d42a84615a6383cfec9a10608" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "197b880fa39d432a9b472845876bfaf0" is_pre_election: true
I20260129 03:00:22.260689 20434 raft_consensus.cc:2468] T fd8311b779af4e1fa1402b2a02a602f2 P 197b880fa39d432a9b472845876bfaf0 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7a9daf5d42a84615a6383cfec9a10608 in term 6.
I20260129 03:00:22.251921 23852 raft_consensus.cc:2804] T a644c02b16af47679824ccd060ac10fd P 7a9daf5d42a84615a6383cfec9a10608 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260129 03:00:22.260778 23808 raft_consensus.cc:493] T 75b0b6e82ddc4f92b1fe1d94f9a0b73b P 1c50de89a5a845668696dcb28c3badd6 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260129 03:00:22.269647 23808 raft_consensus.cc:515] T 75b0b6e82ddc4f92b1fe1d94f9a0b73b P 1c50de89a5a845668696dcb28c3badd6 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } }
I20260129 03:00:22.258914 23858 raft_consensus.cc:2749] T a1a39dcd328e4268ae41b988c66b4a75 P 197b880fa39d432a9b472845876bfaf0 [term 7 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
I20260129 03:00:22.263289 23840 consensus_queue.cc:1048] T 7d9494f5584d4d5db8b9adf1c8fc888b P 197b880fa39d432a9b472845876bfaf0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.012s
I20260129 03:00:22.263206 23852 raft_consensus.cc:493] T a644c02b16af47679824ccd060ac10fd P 7a9daf5d42a84615a6383cfec9a10608 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260129 03:00:22.272127 21129 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "42d27a28d89b409e8807ec0efd97af69" candidate_uuid: "197b880fa39d432a9b472845876bfaf0" candidate_term: 9 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "7a9daf5d42a84615a6383cfec9a10608" is_pre_election: true
I20260129 03:00:22.282158 20434 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00e79d8f25a34676b1d866eb2068a2b2" candidate_uuid: "7a9daf5d42a84615a6383cfec9a10608" candidate_term: 9 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "197b880fa39d432a9b472845876bfaf0" is_pre_election: true
I20260129 03:00:22.282986 20434 raft_consensus.cc:2468] T 00e79d8f25a34676b1d866eb2068a2b2 P 197b880fa39d432a9b472845876bfaf0 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7a9daf5d42a84615a6383cfec9a10608 in term 9.
I20260129 03:00:22.285513 20513 raft_consensus.cc:1217] T a5a05fbcbbcf4916bbc01fd1a5b5e90d P 974b802141bd4f13beffef2aa03f88f9 [term 9 FOLLOWER]: Deduplicated request from leader. Original: 7.5->[8.6-8.6] Dedup: 8.6->[]
I20260129 03:00:22.286003 20513 raft_consensus.cc:1240] T a5a05fbcbbcf4916bbc01fd1a5b5e90d P 974b802141bd4f13beffef2aa03f88f9 [term 9 FOLLOWER]: Rejecting Update request from peer 1c50de89a5a845668696dcb28c3badd6 for earlier term 8. Current term is 9. Ops: [8.6-8.6]
I20260129 03:00:22.277365 23852 raft_consensus.cc:3060] T a644c02b16af47679824ccd060ac10fd P 7a9daf5d42a84615a6383cfec9a10608 [term 6 FOLLOWER]: Advancing to term 7
I20260129 03:00:22.310184 20556 leader_election.cc:304] T fcddab6bca684adf8ea032289a00f90e P 1c50de89a5a845668696dcb28c3badd6 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1c50de89a5a845668696dcb28c3badd6, 974b802141bd4f13beffef2aa03f88f9; no voters: 197b880fa39d432a9b472845876bfaf0
I20260129 03:00:22.311048 23262 raft_consensus.cc:3060] T a5a05fbcbbcf4916bbc01fd1a5b5e90d P 1c50de89a5a845668696dcb28c3badd6 [term 8 FOLLOWER]: Advancing to term 9
I20260129 03:00:22.312078 23304 raft_consensus.cc:2764] T fcddab6bca684adf8ea032289a00f90e P 1c50de89a5a845668696dcb28c3badd6 [term 11 FOLLOWER]: Leader pre-election decision vote started in defunct term 10: won
W20260129 03:00:22.290514 20555 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260129 03:00:22.290869 21130 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fd8311b779af4e1fa1402b2a02a602f2" candidate_uuid: "1c50de89a5a845668696dcb28c3badd6" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "7a9daf5d42a84615a6383cfec9a10608" is_pre_election: true
I20260129 03:00:22.316147 21130 raft_consensus.cc:2468] T fd8311b779af4e1fa1402b2a02a602f2 P 7a9daf5d42a84615a6383cfec9a10608 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1c50de89a5a845668696dcb28c3badd6 in term 6.
I20260129 03:00:22.271844 23808 leader_election.cc:290] T 75b0b6e82ddc4f92b1fe1d94f9a0b73b P 1c50de89a5a845668696dcb28c3badd6 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 974b802141bd4f13beffef2aa03f88f9 (127.18.211.2:34517), 7a9daf5d42a84615a6383cfec9a10608 (127.18.211.4:40439)
I20260129 03:00:22.294077 23450 raft_consensus.cc:493] T 75f45538dc704d9a8ec4ff8ad9f3ea5b P 974b802141bd4f13beffef2aa03f88f9 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260129 03:00:22.317390 23450 raft_consensus.cc:515] T 75f45538dc704d9a8ec4ff8ad9f3ea5b P 974b802141bd4f13beffef2aa03f88f9 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } }
I20260129 03:00:22.319442 23450 leader_election.cc:290] T 75f45538dc704d9a8ec4ff8ad9f3ea5b P 974b802141bd4f13beffef2aa03f88f9 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3:37735), 7a9daf5d42a84615a6383cfec9a10608 (127.18.211.4:40439)
I20260129 03:00:22.296516 21133 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34046 (request call id 634) took 1220 ms. Trace:
I20260129 03:00:22.310009 20588 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:48526 (request call id 478) took 1310 ms. Trace:
I20260129 03:00:22.297962 20581 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "42d27a28d89b409e8807ec0efd97af69" candidate_uuid: "197b880fa39d432a9b472845876bfaf0" candidate_term: 9 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "1c50de89a5a845668696dcb28c3badd6" is_pre_election: true
I20260129 03:00:22.321465 20581 raft_consensus.cc:2468] T 42d27a28d89b409e8807ec0efd97af69 P 1c50de89a5a845668696dcb28c3badd6 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 197b880fa39d432a9b472845876bfaf0 in term 8.
I20260129 03:00:22.321168 20588 rpcz_store.cc:276] 0129 03:00:20.999979 (+ 0us) service_pool.cc:168] Inserting onto call queue
0129 03:00:21.027351 (+ 27372us) service_pool.cc:225] Handling call
0129 03:00:21.027663 (+ 312us) raft_consensus.cc:1471] Updating replica for 0 ops
0129 03:00:21.671562 (+643899us) raft_consensus.cc:1697] Filling consensus response to leader.
0129 03:00:22.309774 (+638212us) spinlock_profiling.cc:243] Waited 643 ms on lock 0x7b540039dad4. stack: 00007f8fe044e373 00007f8fe044e270 00007f8fdfda0574 00007f8fe9a5c6b4 00007f8fe9a5c649 00007f8fe34975a1 00007f8fe352ea81 00007f8fe352d4be 00007f8fe84e1f49 00007f8fe1a6ed34 00007f8fe1a6ecc2 00007f8fe1a6ebf1 00007f8fe1a6eb6c 00007f8fe1a6de22 00007f8fe186d34c 00007f8fe186c786
0129 03:00:22.309951 (+ 177us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1479423744}
I20260129 03:00:22.322633 20581 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fd8311b779af4e1fa1402b2a02a602f2" candidate_uuid: "7a9daf5d42a84615a6383cfec9a10608" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "1c50de89a5a845668696dcb28c3badd6" is_pre_election: true
I20260129 03:00:22.322863 20588 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fd8311b779af4e1fa1402b2a02a602f2" candidate_uuid: "197b880fa39d432a9b472845876bfaf0" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "1c50de89a5a845668696dcb28c3badd6" is_pre_election: true
I20260129 03:00:22.280467 23800 raft_consensus.cc:493] T add8d4923d1a489e86882c3a25692b95 P 1c50de89a5a845668696dcb28c3badd6 [term 9 FOLLOWER]: Starting pre-election (detected failure of leader 7a9daf5d42a84615a6383cfec9a10608)
I20260129 03:00:22.298578 20480 leader_election.cc:304] T a644c02b16af47679824ccd060ac10fd P 974b802141bd4f13beffef2aa03f88f9 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1c50de89a5a845668696dcb28c3badd6, 974b802141bd4f13beffef2aa03f88f9; no voters: 7a9daf5d42a84615a6383cfec9a10608
I20260129 03:00:22.302804 20508 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "36ff33be666a4c82954731691dbcf327" candidate_uuid: "1c50de89a5a845668696dcb28c3badd6" candidate_term: 6 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "974b802141bd4f13beffef2aa03f88f9"
I20260129 03:00:22.324355 23800 raft_consensus.cc:515] T add8d4923d1a489e86882c3a25692b95 P 1c50de89a5a845668696dcb28c3badd6 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } }
I20260129 03:00:22.324931 20508 raft_consensus.cc:3060] T 36ff33be666a4c82954731691dbcf327 P 974b802141bd4f13beffef2aa03f88f9 [term 5 FOLLOWER]: Advancing to term 6
I20260129 03:00:22.325551 23678 raft_consensus.cc:493] T cb0b3f259ebc4310927338a8740e87b8 P 1c50de89a5a845668696dcb28c3badd6 [term 9 FOLLOWER]: Starting leader election (detected failure of leader 7a9daf5d42a84615a6383cfec9a10608)
I20260129 03:00:22.326076 23678 raft_consensus.cc:3060] T cb0b3f259ebc4310927338a8740e87b8 P 1c50de89a5a845668696dcb28c3badd6 [term 9 FOLLOWER]: Advancing to term 10
I20260129 03:00:22.326121 20588 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a0fcc58d6fb6473cafbd9d34d5195269" candidate_uuid: "197b880fa39d432a9b472845876bfaf0" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "1c50de89a5a845668696dcb28c3badd6" is_pre_election: true
I20260129 03:00:22.326331 23800 leader_election.cc:290] T add8d4923d1a489e86882c3a25692b95 P 1c50de89a5a845668696dcb28c3badd6 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 974b802141bd4f13beffef2aa03f88f9 (127.18.211.2:34517), 7a9daf5d42a84615a6383cfec9a10608 (127.18.211.4:40439)
I20260129 03:00:22.303525 20510 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4f5a77e94fef4b16a2ba7b067c8b144a" candidate_uuid: "1c50de89a5a845668696dcb28c3badd6" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "974b802141bd4f13beffef2aa03f88f9" is_pre_election: true
I20260129 03:00:22.304195 20515 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1ce4ddf5b10e4ae8b193b13b2e5936ee" candidate_uuid: "1c50de89a5a845668696dcb28c3badd6" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "974b802141bd4f13beffef2aa03f88f9"
I20260129 03:00:22.329053 20515 raft_consensus.cc:1275] T 76f713c3331540d2a5b76164103e9eb1 P 974b802141bd4f13beffef2aa03f88f9 [term 7 FOLLOWER]: Refusing update from remote peer 197b880fa39d432a9b472845876bfaf0: Log matching property violated. Preceding OpId in replica: term: 6 index: 5. Preceding OpId from leader: term: 7 index: 6. (index mismatch)
I20260129 03:00:22.320715 21133 rpcz_store.cc:276] 0129 03:00:21.075700 (+ 0us) service_pool.cc:168] Inserting onto call queue
0129 03:00:21.075843 (+ 143us) service_pool.cc:225] Handling call
0129 03:00:21.082839 (+ 6996us) raft_consensus.cc:1471] Updating replica for 0 ops
0129 03:00:21.890541 (+807702us) raft_consensus.cc:1697] Filling consensus response to leader.
0129 03:00:22.296283 (+405742us) spinlock_profiling.cc:243] Waited 805 ms on lock 0x7b540068fbd4. stack: 00007f8fe044e373 00007f8fe044e270 00007f8fdfda0574 00007f8fe9a5c6b4 00007f8fe9a5c649 00007f8fe34975a1 00007f8fe352ea81 00007f8fe352d4be 00007f8fe84e1f49 00007f8fe1a6ed34 00007f8fe1a6ecc2 00007f8fe1a6ebf1 00007f8fe1a6eb6c 00007f8fe1a6de22 00007f8fe186d34c 00007f8fe186c786
0129 03:00:22.296458 (+ 175us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1850834176}
I20260129 03:00:22.290086 20585 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "75f45538dc704d9a8ec4ff8ad9f3ea5b" candidate_uuid: "7a9daf5d42a84615a6383cfec9a10608" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "1c50de89a5a845668696dcb28c3badd6"
I20260129 03:00:22.305059 23646 raft_consensus.cc:493] T 00e79d8f25a34676b1d866eb2068a2b2 P 974b802141bd4f13beffef2aa03f88f9 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260129 03:00:22.304858 20509 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "75b0b6e82ddc4f92b1fe1d94f9a0b73b" candidate_uuid: "1c50de89a5a845668696dcb28c3badd6" candidate_term: 7 candidate_status { last_received { term: 5 index: 8 } } ignore_live_leader: false dest_uuid: "974b802141bd4f13beffef2aa03f88f9" is_pre_election: true
W20260129 03:00:22.305830 23837 log.cc:927] Time spent T 0f20af7fbe5d4db89a6f17948d98912c P 1c50de89a5a845668696dcb28c3badd6: Append to log took a long time: real 0.613s user 0.005s sys 0.009s
I20260129 03:00:22.331563 20509 raft_consensus.cc:2468] T 75b0b6e82ddc4f92b1fe1d94f9a0b73b P 974b802141bd4f13beffef2aa03f88f9 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1c50de89a5a845668696dcb28c3badd6 in term 6.
I20260129 03:00:22.331343 23646 raft_consensus.cc:515] T 00e79d8f25a34676b1d866eb2068a2b2 P 974b802141bd4f13beffef2aa03f88f9 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } }
I20260129 03:00:22.326771 20588 raft_consensus.cc:2468] T a0fcc58d6fb6473cafbd9d34d5195269 P 1c50de89a5a845668696dcb28c3badd6 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 197b880fa39d432a9b472845876bfaf0 in term 6.
I20260129 03:00:22.307488 23398 raft_consensus.cc:515] T fd8311b779af4e1fa1402b2a02a602f2 P 1c50de89a5a845668696dcb28c3badd6 [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } }
I20260129 03:00:22.335501 23398 leader_election.cc:290] T fd8311b779af4e1fa1402b2a02a602f2 P 1c50de89a5a845668696dcb28c3badd6 [CANDIDATE]: Term 7 election: Requested vote from peers 197b880fa39d432a9b472845876bfaf0 (127.18.211.1:42027), 7a9daf5d42a84615a6383cfec9a10608 (127.18.211.4:40439)
I20260129 03:00:22.310063 21099 leader_election.cc:304] T fd8311b779af4e1fa1402b2a02a602f2 P 7a9daf5d42a84615a6383cfec9a10608 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 197b880fa39d432a9b472845876bfaf0, 7a9daf5d42a84615a6383cfec9a10608; no voters:
I20260129 03:00:22.336347 23669 raft_consensus.cc:493] T 4e3805ffa70242e8ade66da260dddf66 P 1c50de89a5a845668696dcb28c3badd6 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 7a9daf5d42a84615a6383cfec9a10608)
I20260129 03:00:22.337066 21099 leader_election.cc:304] T 00e79d8f25a34676b1d866eb2068a2b2 P 7a9daf5d42a84615a6383cfec9a10608 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 197b880fa39d432a9b472845876bfaf0, 7a9daf5d42a84615a6383cfec9a10608; no voters:
I20260129 03:00:22.336927 23669 raft_consensus.cc:515] T 4e3805ffa70242e8ade66da260dddf66 P 1c50de89a5a845668696dcb28c3badd6 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } }
I20260129 03:00:22.307987 20582 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "96609ac68bdf4993a12e1c23355ca013" candidate_uuid: "197b880fa39d432a9b472845876bfaf0" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "1c50de89a5a845668696dcb28c3badd6" is_pre_election: true
I20260129 03:00:22.338383 23640 raft_consensus.cc:2804] T fd8311b779af4e1fa1402b2a02a602f2 P 7a9daf5d42a84615a6383cfec9a10608 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260129 03:00:22.338716 20438 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55676 (request call id 1565) took 1372 ms. Trace:
I20260129 03:00:22.332684 20583 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35752 (request call id 1823) took 1003 ms. Trace:
I20260129 03:00:22.338980 23640 raft_consensus.cc:493] T fd8311b779af4e1fa1402b2a02a602f2 P 7a9daf5d42a84615a6383cfec9a10608 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260129 03:00:22.339668 23640 raft_consensus.cc:3060] T fd8311b779af4e1fa1402b2a02a602f2 P 7a9daf5d42a84615a6383cfec9a10608 [term 6 FOLLOWER]: Advancing to term 7
I20260129 03:00:22.339969 23688 raft_consensus.cc:2804] T 00e79d8f25a34676b1d866eb2068a2b2 P 7a9daf5d42a84615a6383cfec9a10608 [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260129 03:00:22.340466 23688 raft_consensus.cc:493] T 00e79d8f25a34676b1d866eb2068a2b2 P 7a9daf5d42a84615a6383cfec9a10608 [term 8 FOLLOWER]: Starting leader election (detected failure of leader 197b880fa39d432a9b472845876bfaf0)
I20260129 03:00:22.340900 23688 raft_consensus.cc:3060] T 00e79d8f25a34676b1d866eb2068a2b2 P 7a9daf5d42a84615a6383cfec9a10608 [term 8 FOLLOWER]: Advancing to term 9
I20260129 03:00:22.339332 20583 rpcz_store.cc:276] 0129 03:00:21.329458 (+ 0us) service_pool.cc:168] Inserting onto call queue
0129 03:00:21.330002 (+ 544us) service_pool.cc:225] Handling call
0129 03:00:21.380968 (+ 50966us) raft_consensus.cc:1471] Updating replica for 1 ops
0129 03:00:21.412530 (+ 31562us) raft_consensus.cc:1534] Early marking committed up to index 0
0129 03:00:21.412559 (+ 29us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0129 03:00:21.412997 (+ 438us) log.cc:844] Serialized 26 byte log entry
0129 03:00:21.619469 (+206472us) raft_consensus.cc:1645] Marking committed up to 5
0129 03:00:21.619511 (+ 42us) raft_consensus.cc:1697] Filling consensus response to leader.
0129 03:00:21.619584 (+ 73us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0129 03:00:22.305543 (+685959us) spinlock_profiling.cc:243] Waited 16.3 ms on lock 0x7b240018d448. stack: 00007f8fe044e373 00007f8fe044e270 00007f8fdfda0574 00007f8fe9a5c6b4 00007f8fe9a5c649 00007f8fe02c08b1 00007f8fe02d4ad4 00007f8fe02d2bdb 00007f8fe0361ad9 00007f8fe0361765 00007f8fe0360dad 00007f8fe30b14ae 00007f8fe30c8c2c 00007f8fe30c7f56 00007f8fe30c7b78 00007f8fe30d15c1
0129 03:00:22.332379 (+ 26836us) raft_consensus.cc:1685] finished
0129 03:00:22.332400 (+ 21us) raft_consensus.cc:1691] UpdateReplicas() finished
0129 03:00:22.332619 (+ 219us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":14227,"raft.queue_time_us":27032,"raft.run_cpu_time_us":216,"raft.run_wall_time_us":34568,"spinlock_wait_cycles":37393152,"thread_start_us":155706,"threads_started":1,"wal-append.queue_time_us":245714}
I20260129 03:00:22.339206 20438 rpcz_store.cc:276] 0129 03:00:20.966123 (+ 0us) service_pool.cc:168] Inserting onto call queue
0129 03:00:20.966594 (+ 471us) service_pool.cc:225] Handling call
0129 03:00:20.966972 (+ 378us) raft_consensus.cc:1471] Updating replica for 1 ops
0129 03:00:21.328891 (+361919us) raft_consensus.cc:1534] Early marking committed up to index 0
0129 03:00:21.328951 (+ 60us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0129 03:00:21.329742 (+ 791us) log.cc:844] Serialized 26 byte log entry
0129 03:00:21.527305 (+197563us) raft_consensus.cc:1645] Marking committed up to 3
0129 03:00:21.527348 (+ 43us) raft_consensus.cc:1697] Filling consensus response to leader.
0129 03:00:22.338347 (+810999us) spinlock_profiling.cc:243] Waited 20 ms on lock 0x7b540026e654. stack: 00007f8fe044e373 00007f8fe044e270 00007f8fdfda0574 00007f8fe9a5c6b4 00007f8fe9a5c649 00007f8fe34975a1 00007f8fe352ea81 00007f8fe352d4be 00007f8fe84e1f49 00007f8fe1a6ed34 00007f8fe1a6ecc2 00007f8fe1a6ebf1 00007f8fe1a6eb6c 00007f8fe1a6de22 00007f8fe186d34c 00007f8fe186c786
0129 03:00:22.338393 (+ 46us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0129 03:00:22.338441 (+ 48us) raft_consensus.cc:1685] finished
0129 03:00:22.338455 (+ 14us) raft_consensus.cc:1691] UpdateReplicas() finished
0129 03:00:22.338660 (+ 205us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":86525,"raft.run_cpu_time_us":132,"raft.run_wall_time_us":12295,"spinlock_wait_cycles":45946624,"thread_start_us":444436,"threads_started":2,"wal-append.queue_time_us":222766}
I20260129 03:00:22.342840 20583 raft_consensus.cc:1275] T 7a9a69cd82724b1faed1d10aa091fc4a P 1c50de89a5a845668696dcb28c3badd6 [term 8 FOLLOWER]: Refusing update from remote peer 974b802141bd4f13beffef2aa03f88f9: Log matching property violated. Preceding OpId in replica: term: 7 index: 5. Preceding OpId from leader: term: 8 index: 6. (index mismatch)
I20260129 03:00:22.307462 23707 raft_consensus.cc:493] T e7d83d4ccf684e76acb29a0dc844e1b8 P 974b802141bd4f13beffef2aa03f88f9 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260129 03:00:22.332758 21129 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4f5a77e94fef4b16a2ba7b067c8b144a" candidate_uuid: "1c50de89a5a845668696dcb28c3badd6" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "7a9daf5d42a84615a6383cfec9a10608" is_pre_election: true
W20260129 03:00:22.360370 20480 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.211.3:37735, user_credentials={real_user=slave}} blocked reactor thread for 61891.7us
I20260129 03:00:22.332687 20479 leader_election.cc:304] T c934d31efd3f4e5195605ddc354d14e1 P 974b802141bd4f13beffef2aa03f88f9 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 974b802141bd4f13beffef2aa03f88f9; no voters: 197b880fa39d432a9b472845876bfaf0, 7a9daf5d42a84615a6383cfec9a10608
I20260129 03:00:22.333701 21133 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "18f30ebaa4174fd28c958e72d0cc7e24" candidate_uuid: "1c50de89a5a845668696dcb28c3badd6" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "7a9daf5d42a84615a6383cfec9a10608"
I20260129 03:00:22.309125 23691 raft_consensus.cc:493] T a8479d1fe0bd41e5946186e3a6ca8c7a P 974b802141bd4f13beffef2aa03f88f9 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260129 03:00:22.307963 20441 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "36ff33be666a4c82954731691dbcf327" candidate_uuid: "974b802141bd4f13beffef2aa03f88f9" candidate_term: 6 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "197b880fa39d432a9b472845876bfaf0" is_pre_election: true
I20260129 03:00:22.336189 20581 raft_consensus.cc:2393] T fd8311b779af4e1fa1402b2a02a602f2 P 1c50de89a5a845668696dcb28c3badd6 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 7a9daf5d42a84615a6383cfec9a10608 in current term 7: Already voted for candidate 1c50de89a5a845668696dcb28c3badd6 in this term.
I20260129 03:00:22.317515 23808 consensus_queue.cc:1059] T 4b1876f9a70b49edb3aa74105e66d3b3 P 1c50de89a5a845668696dcb28c3badd6 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 }, Status: INVALID_TERM, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260129 03:00:22.332587 20510 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4b1876f9a70b49edb3aa74105e66d3b3" candidate_uuid: "197b880fa39d432a9b472845876bfaf0" candidate_term: 8 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "974b802141bd4f13beffef2aa03f88f9"
I20260129 03:00:22.365269 20441 raft_consensus.cc:2393] T 36ff33be666a4c82954731691dbcf327 P 197b880fa39d432a9b472845876bfaf0 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 974b802141bd4f13beffef2aa03f88f9 in current term 6: Already voted for candidate 1c50de89a5a845668696dcb28c3badd6 in this term.
I20260129 03:00:22.339435 20582 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4b1876f9a70b49edb3aa74105e66d3b3" candidate_uuid: "197b880fa39d432a9b472845876bfaf0" candidate_term: 8 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "1c50de89a5a845668696dcb28c3badd6"
I20260129 03:00:22.365540 20407 leader_election.cc:304] T e33a417cf9c54cf5af73a4ee3aac4d3a P 197b880fa39d432a9b472845876bfaf0 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 197b880fa39d432a9b472845876bfaf0, 974b802141bd4f13beffef2aa03f88f9; no voters: 1c50de89a5a845668696dcb28c3badd6
I20260129 03:00:22.366866 20582 raft_consensus.cc:1275] T a1a39dcd328e4268ae41b988c66b4a75 P 1c50de89a5a845668696dcb28c3badd6 [term 7 FOLLOWER]: Refusing update from remote peer 974b802141bd4f13beffef2aa03f88f9: Log matching property violated. Preceding OpId in replica: term: 5 index: 4. Preceding OpId from leader: term: 7 index: 5. (index mismatch)
I20260129 03:00:22.337970 20585 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "cb4727386a3e48aca5d71cc6c611c197" candidate_uuid: "197b880fa39d432a9b472845876bfaf0" candidate_term: 7 candidate_status { last_received { term: 5 index: 2 } } ignore_live_leader: false dest_uuid: "1c50de89a5a845668696dcb28c3badd6" is_pre_election: true
I20260129 03:00:22.368083 21130 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "75b0b6e82ddc4f92b1fe1d94f9a0b73b" candidate_uuid: "1c50de89a5a845668696dcb28c3badd6" candidate_term: 7 candidate_status { last_received { term: 5 index: 8 } } ignore_live_leader: false dest_uuid: "7a9daf5d42a84615a6383cfec9a10608" is_pre_election: true
I20260129 03:00:22.368661 20585 raft_consensus.cc:2468] T cb4727386a3e48aca5d71cc6c611c197 P 1c50de89a5a845668696dcb28c3badd6 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 197b880fa39d432a9b472845876bfaf0 in term 6.
I20260129 03:00:22.351688 20509 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "96609ac68bdf4993a12e1c23355ca013" candidate_uuid: "197b880fa39d432a9b472845876bfaf0" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "974b802141bd4f13beffef2aa03f88f9" is_pre_election: true
I20260129 03:00:22.336220 20432 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fd8311b779af4e1fa1402b2a02a602f2" candidate_uuid: "1c50de89a5a845668696dcb28c3badd6" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "197b880fa39d432a9b472845876bfaf0"
I20260129 03:00:22.370484 20432 raft_consensus.cc:3060] T fd8311b779af4e1fa1402b2a02a602f2 P 197b880fa39d432a9b472845876bfaf0 [term 6 FOLLOWER]: Advancing to term 7
I20260129 03:00:22.347677 20588 raft_consensus.cc:1275] T cedce331e1584ac8b636c9b8a72b2ba0 P 1c50de89a5a845668696dcb28c3badd6 [term 9 FOLLOWER]: Refusing update from remote peer 974b802141bd4f13beffef2aa03f88f9: Log matching property violated. Preceding OpId in replica: term: 8 index: 5. Preceding OpId from leader: term: 9 index: 6. (index mismatch)
I20260129 03:00:22.346433 23646 leader_election.cc:290] T 00e79d8f25a34676b1d866eb2068a2b2 P 974b802141bd4f13beffef2aa03f88f9 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 197b880fa39d432a9b472845876bfaf0 (127.18.211.1:42027), 7a9daf5d42a84615a6383cfec9a10608 (127.18.211.4:40439)
I20260129 03:00:22.353941 20556 leader_election.cc:304] T 1ce4ddf5b10e4ae8b193b13b2e5936ee P 1c50de89a5a845668696dcb28c3badd6 [CANDIDATE]: Term 8 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1c50de89a5a845668696dcb28c3badd6; no voters: 197b880fa39d432a9b472845876bfaf0, 974b802141bd4f13beffef2aa03f88f9
W20260129 03:00:22.354511 20405 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260129 03:00:22.347112 20516 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:48544 (request call id 411) took 1250 ms. Trace:
I20260129 03:00:22.372155 20507 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "cb4727386a3e48aca5d71cc6c611c197" candidate_uuid: "197b880fa39d432a9b472845876bfaf0" candidate_term: 7 candidate_status { last_received { term: 5 index: 2 } } ignore_live_leader: false dest_uuid: "974b802141bd4f13beffef2aa03f88f9" is_pre_election: true
I20260129 03:00:22.381022 20509 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00e79d8f25a34676b1d866eb2068a2b2" candidate_uuid: "197b880fa39d432a9b472845876bfaf0" candidate_term: 10 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "974b802141bd4f13beffef2aa03f88f9" is_pre_election: true
I20260129 03:00:22.382030 20585 raft_consensus.cc:1275] T 76f713c3331540d2a5b76164103e9eb1 P 1c50de89a5a845668696dcb28c3badd6 [term 7 FOLLOWER]: Refusing update from remote peer 197b880fa39d432a9b472845876bfaf0: Log matching property violated. Preceding OpId in replica: term: 6 index: 5. Preceding OpId from leader: term: 7 index: 6. (index mismatch)
I20260129 03:00:22.382465 20508 raft_consensus.cc:2468] T 36ff33be666a4c82954731691dbcf327 P 974b802141bd4f13beffef2aa03f88f9 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1c50de89a5a845668696dcb28c3badd6 in term 6.
I20260129 03:00:22.383698 20508 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "add8d4923d1a489e86882c3a25692b95" candidate_uuid: "1c50de89a5a845668696dcb28c3badd6" candidate_term: 10 candidate_status { last_received { term: 9 index: 7 } } ignore_live_leader: false dest_uuid: "974b802141bd4f13beffef2aa03f88f9" is_pre_election: true
I20260129 03:00:22.381641 20509 raft_consensus.cc:2468] T 00e79d8f25a34676b1d866eb2068a2b2 P 974b802141bd4f13beffef2aa03f88f9 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 197b880fa39d432a9b472845876bfaf0 in term 9.
I20260129 03:00:22.379300 20516 rpcz_store.cc:276] 0129 03:00:21.097128 (+ 0us) service_pool.cc:168] Inserting onto call queue
0129 03:00:21.097260 (+ 132us) service_pool.cc:225] Handling call
0129 03:00:21.097550 (+ 290us) raft_consensus.cc:1471] Updating replica for 0 ops
0129 03:00:21.097805 (+ 255us) raft_consensus.cc:1534] Early marking committed up to index 7
0129 03:00:21.279379 (+181574us) log.cc:844] Serialized 14 byte log entry
0129 03:00:22.319782 (+1040403us) spinlock_profiling.cc:243] Waited 2.33 ms on lock 0x7b180032fd40. stack: 00007f8fe044e373 00007f8fe044e270 00007f8fdfda0574 00007f8fe9a5c6b4 00007f8fe9a5c649 00007f8fe92d4c91 00007f8fe92e1671 00007f8fe30d585c 00007f8fe04632eb 00007f8fe924b709 00007f8fe047f57b 00007f8fe047d81f 00007f8fe047d335 00007f8fe3525ce0 00007f8fe3538396 00007f8fe3548f07
0129 03:00:22.341543 (+ 21761us) spinlock_profiling.cc:243] Waited 7.82 ms on lock 0x7b180032fd40. stack: 00007f8fe044e373 00007f8fe044e270 00007f8fdfda0574 00007f8fe9a5c6b4 00007f8fe9a5c649 00007f8fe92d4c91 00007f8fe92e1671 00007f8fe0480402 00007f8fe04826a1 00007f8fe0482659 00007f8fe04825e9 00007f8fe04825b1 00007f8fe04818ad 0000000000379254 0000000000379099 00007f8fe046369a
0129 03:00:22.346656 (+ 5113us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0129 03:00:22.346738 (+ 82us) raft_consensus.cc:1645] Marking committed up to 7
0129 03:00:22.346803 (+ 65us) raft_consensus.cc:1697] Filling consensus response to leader.
0129 03:00:22.346906 (+ 103us) raft_consensus.cc:1691] UpdateReplicas() finished
0129 03:00:22.347042 (+ 136us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":121046,"prepare.queue_time_us":344402,"prepare.run_cpu_time_us":67,"prepare.run_wall_time_us":5076,"raft.queue_time_us":134291,"raft.run_cpu_time_us":161,"raft.run_wall_time_us":25010,"spinlock_wait_cycles":23347968,"thread_start_us":263876,"threads_started":2}
I20260129 03:00:22.376305 23665 raft_consensus.cc:2804] T e33a417cf9c54cf5af73a4ee3aac4d3a P 197b880fa39d432a9b472845876bfaf0 [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260129 03:00:22.398599 20516 raft_consensus.cc:1275] T c891d2bff4ce4568861bbc992e91fbb0 P 974b802141bd4f13beffef2aa03f88f9 [term 10 FOLLOWER]: Refusing update from remote peer 197b880fa39d432a9b472845876bfaf0: Log matching property violated. Preceding OpId in replica: term: 6 index: 5. Preceding OpId from leader: term: 10 index: 7. (index mismatch)
I20260129 03:00:22.399832 21130 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fd8311b779af4e1fa1402b2a02a602f2" candidate_uuid: "1c50de89a5a845668696dcb28c3badd6" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "7a9daf5d42a84615a6383cfec9a10608"
I20260129 03:00:22.396978 20439 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4e3805ffa70242e8ade66da260dddf66" candidate_uuid: "1c50de89a5a845668696dcb28c3badd6" candidate_term: 7 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "197b880fa39d432a9b472845876bfaf0" is_pre_election: true
I20260129 03:00:22.369753 23466 raft_consensus.cc:493] T 7a9a69cd82724b1faed1d10aa091fc4a P 197b880fa39d432a9b472845876bfaf0 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260129 03:00:22.407096 21129 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "add8d4923d1a489e86882c3a25692b95" candidate_uuid: "1c50de89a5a845668696dcb28c3badd6" candidate_term: 10 candidate_status { last_received { term: 9 index: 7 } } ignore_live_leader: false dest_uuid: "7a9daf5d42a84615a6383cfec9a10608" is_pre_election: true
I20260129 03:00:22.409067 20432 raft_consensus.cc:2468] T fd8311b779af4e1fa1402b2a02a602f2 P 197b880fa39d432a9b472845876bfaf0 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1c50de89a5a845668696dcb28c3badd6 in term 7.
W20260129 03:00:22.411185 20479 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.211.1:42027, user_credentials={real_user=slave}} blocked reactor thread for 102149us
I20260129 03:00:22.411695 21134 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00e79d8f25a34676b1d866eb2068a2b2" candidate_uuid: "197b880fa39d432a9b472845876bfaf0" candidate_term: 10 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "7a9daf5d42a84615a6383cfec9a10608" is_pre_election: true
I20260129 03:00:22.412659 21135 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "96609ac68bdf4993a12e1c23355ca013" candidate_uuid: "197b880fa39d432a9b472845876bfaf0" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "7a9daf5d42a84615a6383cfec9a10608" is_pre_election: true
I20260129 03:00:22.386026 23777 raft_consensus.cc:2804] T a644c02b16af47679824ccd060ac10fd P 974b802141bd4f13beffef2aa03f88f9 [term 6 FOLLOWER]: Leader election won for term 6
I20260129 03:00:22.433934 20554 leader_election.cc:304] T 18f30ebaa4174fd28c958e72d0cc7e24 P 1c50de89a5a845668696dcb28c3badd6 [CANDIDATE]: Term 9 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1c50de89a5a845668696dcb28c3badd6; no voters: 197b880fa39d432a9b472845876bfaf0, 7a9daf5d42a84615a6383cfec9a10608
I20260129 03:00:22.435613 20554 leader_election.cc:304] T 4f5a77e94fef4b16a2ba7b067c8b144a P 1c50de89a5a845668696dcb28c3badd6 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1c50de89a5a845668696dcb28c3badd6; no voters: 7a9daf5d42a84615a6383cfec9a10608, 974b802141bd4f13beffef2aa03f88f9
I20260129 03:00:22.392923 23795 raft_consensus.cc:493] T 17d6431cc77a491e8464d68db4081a51 P 974b802141bd4f13beffef2aa03f88f9 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260129 03:00:22.373147 23707 raft_consensus.cc:515] T e7d83d4ccf684e76acb29a0dc844e1b8 P 974b802141bd4f13beffef2aa03f88f9 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
W20260129 03:00:22.444726 20556 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.211.2:34517, user_credentials={real_user=slave}} blocked reactor thread for 90887.7us
I20260129 03:00:22.446470 20584 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:35754 (request call id 1908) took 1022 ms. Trace:
I20260129 03:00:22.379918 23691 raft_consensus.cc:515] T a8479d1fe0bd41e5946186e3a6ca8c7a P 974b802141bd4f13beffef2aa03f88f9 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 03:00:22.397650 23665 raft_consensus.cc:493] T e33a417cf9c54cf5af73a4ee3aac4d3a P 197b880fa39d432a9b472845876bfaf0 [term 8 FOLLOWER]: Starting leader election (detected failure of leader 1c50de89a5a845668696dcb28c3badd6)
I20260129 03:00:22.467185 20584 rpcz_store.cc:276] 0129 03:00:21.423931 (+ 0us) service_pool.cc:168] Inserting onto call queue
0129 03:00:21.452324 (+ 28393us) service_pool.cc:225] Handling call
0129 03:00:22.446221 (+993897us) spinlock_profiling.cc:243] Waited 482 ms on lock 0x7b54003b1354. stack: 00007f8fe044e373 00007f8fe044e270 00007f8fdfda0574 00007f8fe9a5c6b4 00007f8fe9a5c649 00007f8fe34975a1 00007f8fe3531f7a 00007f8fe84e3647 00007f8fe1a73834 00007f8fe1a737c2 00007f8fe1a736f1 00007f8fe1a7366c 00007f8fe1a72922 00007f8fe186d34c 00007f8fe186c786 00007f8fe186c12f
0129 03:00:22.446394 (+ 173us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1108596480}
W20260129 03:00:22.505822 20406 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260129 03:00:22.546743 20407 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.211.2:34517, user_credentials={real_user=slave}} blocked reactor thread for 166375us
I20260129 03:00:22.506213 23740 leader_election.cc:290] T c934d31efd3f4e5195605ddc354d14e1 P 7a9daf5d42a84615a6383cfec9a10608 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 197b880fa39d432a9b472845876bfaf0 (127.18.211.1:42027), 974b802141bd4f13beffef2aa03f88f9 (127.18.211.2:34517)
I20260129 03:00:22.501271 23743 raft_consensus.cc:3060] T 87d5fe48a44a408aaf03e894a67ecb2f P 197b880fa39d432a9b472845876bfaf0 [term 7 FOLLOWER]: Advancing to term 8
I20260129 03:00:22.417678 23466 raft_consensus.cc:515] T 7a9a69cd82724b1faed1d10aa091fc4a P 197b880fa39d432a9b472845876bfaf0 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 03:00:22.506639 23669 leader_election.cc:290] T 4e3805ffa70242e8ade66da260dddf66 P 1c50de89a5a845668696dcb28c3badd6 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 197b880fa39d432a9b472845876bfaf0 (127.18.211.1:42027), 7a9daf5d42a84615a6383cfec9a10608 (127.18.211.4:40439)
I20260129 03:00:22.431849 23847 consensus_queue.cc:1059] T 18f30ebaa4174fd28c958e72d0cc7e24 P 7a9daf5d42a84615a6383cfec9a10608 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 }, Status: INVALID_TERM, Last received: 8.8, Next index: 9, Last known committed idx: 8, Time since last communication: 0.076s
I20260129 03:00:22.444159 23710 consensus_queue.cc:1048] T 7a9a69cd82724b1faed1d10aa091fc4a P 974b802141bd4f13beffef2aa03f88f9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.070s
I20260129 03:00:22.448570 23799 consensus_queue.cc:1059] T 75f45538dc704d9a8ec4ff8ad9f3ea5b P 1c50de89a5a845668696dcb28c3badd6 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false }, Status: INVALID_TERM, Last received: 5.7, Next index: 8, Last known committed idx: 6, Time since last communication: 0.062s
I20260129 03:00:22.387931 23086 consensus_queue.cc:1048] T 76f713c3331540d2a5b76164103e9eb1 P 197b880fa39d432a9b472845876bfaf0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.044s
I20260129 03:00:22.595391 23838 consensus_queue.cc:1048] T cedce331e1584ac8b636c9b8a72b2ba0 P 974b802141bd4f13beffef2aa03f88f9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 4, Time since last communication: 0.000s
I20260129 03:00:22.595927 23803 raft_consensus.cc:993] T e33a417cf9c54cf5af73a4ee3aac4d3a P 1c50de89a5a845668696dcb28c3badd6: : Instructing follower 197b880fa39d432a9b472845876bfaf0 to start an election
I20260129 03:00:22.408221 23531 raft_consensus.cc:3055] T 4b1876f9a70b49edb3aa74105e66d3b3 P 1c50de89a5a845668696dcb28c3badd6 [term 7 LEADER]: Stepping down as leader of term 7
W20260129 03:00:22.546217 20405 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.211.4:40439, user_credentials={real_user=slave}} blocked reactor thread for 100372us
I20260129 03:00:22.597909 23531 raft_consensus.cc:740] T 4b1876f9a70b49edb3aa74105e66d3b3 P 1c50de89a5a845668696dcb28c3badd6 [term 7 LEADER]: Becoming Follower/Learner. State: Replica: 1c50de89a5a845668696dcb28c3badd6, State: Running, Role: LEADER
I20260129 03:00:22.598788 23531 consensus_queue.cc:260] T 4b1876f9a70b49edb3aa74105e66d3b3 P 1c50de89a5a845668696dcb28c3badd6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 7.5, Last appended by leader: 5, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 03:00:22.599100 20405 leader_election.cc:304] T 96609ac68bdf4993a12e1c23355ca013 P 197b880fa39d432a9b472845876bfaf0 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 197b880fa39d432a9b472845876bfaf0; no voters: 7a9daf5d42a84615a6383cfec9a10608, 974b802141bd4f13beffef2aa03f88f9
I20260129 03:00:22.599972 23531 raft_consensus.cc:3060] T 4b1876f9a70b49edb3aa74105e66d3b3 P 1c50de89a5a845668696dcb28c3badd6 [term 7 FOLLOWER]: Advancing to term 8
W20260129 03:00:22.548101 20479 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.211.1:42027, user_credentials={real_user=slave}} blocked reactor thread for 117857us
I20260129 03:00:22.548825 20556 leader_election.cc:304] T 75b0b6e82ddc4f92b1fe1d94f9a0b73b P 1c50de89a5a845668696dcb28c3badd6 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1c50de89a5a845668696dcb28c3badd6, 974b802141bd4f13beffef2aa03f88f9; no voters:
W20260129 03:00:22.550453 21100 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.211.3:37735, user_credentials={real_user=slave}} blocked reactor thread for 119308us
I20260129 03:00:22.552225 19276 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260129 03:00:22.554701 23800 raft_consensus.cc:2749] T 1ce4ddf5b10e4ae8b193b13b2e5936ee P 1c50de89a5a845668696dcb28c3badd6 [term 8 FOLLOWER]: Leader election lost for term 8. Reason: could not achieve majority
I20260129 03:00:22.570952 23466 leader_election.cc:290] T 7a9a69cd82724b1faed1d10aa091fc4a P 197b880fa39d432a9b472845876bfaf0 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 974b802141bd4f13beffef2aa03f88f9 (127.18.211.2:34517), 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3:37735)
I20260129 03:00:22.551056 23602 raft_consensus.cc:2749] T 18f30ebaa4174fd28c958e72d0cc7e24 P 1c50de89a5a845668696dcb28c3badd6 [term 9 FOLLOWER]: Leader election lost for term 9. Reason: could not achieve majority
I20260129 03:00:22.573406 23777 raft_consensus.cc:697] T a644c02b16af47679824ccd060ac10fd P 974b802141bd4f13beffef2aa03f88f9 [term 6 LEADER]: Becoming Leader. State: Replica: 974b802141bd4f13beffef2aa03f88f9, State: Running, Role: LEADER
I20260129 03:00:22.552462 23665 raft_consensus.cc:3060] T e33a417cf9c54cf5af73a4ee3aac4d3a P 197b880fa39d432a9b472845876bfaf0 [term 8 FOLLOWER]: Advancing to term 9
I20260129 03:00:22.497627 23795 raft_consensus.cc:515] T 17d6431cc77a491e8464d68db4081a51 P 974b802141bd4f13beffef2aa03f88f9 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 03:00:22.580626 20407 leader_election.cc:304] T 00e79d8f25a34676b1d866eb2068a2b2 P 197b880fa39d432a9b472845876bfaf0 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 197b880fa39d432a9b472845876bfaf0, 974b802141bd4f13beffef2aa03f88f9; no voters:
W20260129 03:00:22.652537 20556 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.211.2:34517, user_credentials={real_user=slave}} blocked reactor thread for 198248us
W20260129 03:00:22.594641 20480 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.211.3:37735, user_credentials={real_user=slave}} blocked reactor thread for 44428.1us
I20260129 03:00:22.618140 20553 leader_election.cc:304] T fd8311b779af4e1fa1402b2a02a602f2 P 1c50de89a5a845668696dcb28c3badd6 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 197b880fa39d432a9b472845876bfaf0, 1c50de89a5a845668696dcb28c3badd6; no voters:
I20260129 03:00:22.573417 23707 leader_election.cc:290] T e7d83d4ccf684e76acb29a0dc844e1b8 P 974b802141bd4f13beffef2aa03f88f9 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 197b880fa39d432a9b472845876bfaf0 (127.18.211.1:42027), 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3:37735)
I20260129 03:00:22.554657 23755 consensus_queue.cc:1048] T a1a39dcd328e4268ae41b988c66b4a75 P 974b802141bd4f13beffef2aa03f88f9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.059s
W20260129 03:00:22.677404 21100 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.211.3:37735, user_credentials={real_user=slave}} blocked reactor thread for 64786.5us
I20260129 03:00:22.568732 23668 raft_consensus.cc:2749] T 4f5a77e94fef4b16a2ba7b067c8b144a P 1c50de89a5a845668696dcb28c3badd6 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
W20260129 03:00:22.679361 20407 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.211.2:34517, user_credentials={real_user=slave}} blocked reactor thread for 98896.4us
I20260129 03:00:22.694545 20587 raft_consensus.cc:2468] T a5a05fbcbbcf4916bbc01fd1a5b5e90d P 1c50de89a5a845668696dcb28c3badd6 [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 197b880fa39d432a9b472845876bfaf0 in term 9.
I20260129 03:00:22.576823 23766 raft_consensus.cc:493] T 505580c93c694bbb8ea96fdf9b17ab46 P 1c50de89a5a845668696dcb28c3badd6 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 7a9daf5d42a84615a6383cfec9a10608)
I20260129 03:00:22.687661 20479 leader_election.cc:304] T 36ff33be666a4c82954731691dbcf327 P 974b802141bd4f13beffef2aa03f88f9 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 974b802141bd4f13beffef2aa03f88f9; no voters: 197b880fa39d432a9b472845876bfaf0, 1c50de89a5a845668696dcb28c3badd6
W20260129 03:00:22.698810 20553 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.211.1:42027, user_credentials={real_user=slave}} blocked reactor thread for 80787.4us
W20260129 03:00:22.730180 20555 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260129 03:00:22.734634 21100 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.211.3:37735, user_credentials={real_user=slave}} blocked reactor thread for 49342.5us
I20260129 03:00:22.613299 23744 raft_consensus.cc:2749] T 96609ac68bdf4993a12e1c23355ca013 P 197b880fa39d432a9b472845876bfaf0 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260129 03:00:22.652424 23669 raft_consensus.cc:1081] T e33a417cf9c54cf5af73a4ee3aac4d3a P 1c50de89a5a845668696dcb28c3badd6 [term 8 LEADER]: Signalling peer 197b880fa39d432a9b472845876bfaf0 to start an election
W20260129 03:00:22.761214 20554 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.211.4:40439, user_credentials={real_user=slave}} blocked reactor thread for 325624us
W20260129 03:00:22.761973 20405 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.211.3:37735, user_credentials={real_user=slave}} blocked reactor thread for 157778us
I20260129 03:00:22.686663 23798 raft_consensus.cc:2804] T 75b0b6e82ddc4f92b1fe1d94f9a0b73b P 1c50de89a5a845668696dcb28c3badd6 [term 6 FOLLOWER]: Leader pre-election won for term 7
W20260129 03:00:22.777278 20480 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.211.3:37735, user_credentials={real_user=slave}} blocked reactor thread for 123367us
I20260129 03:00:22.786702 20509 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a644c02b16af47679824ccd060ac10fd" candidate_uuid: "7a9daf5d42a84615a6383cfec9a10608" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "974b802141bd4f13beffef2aa03f88f9" is_pre_election: true
I20260129 03:00:22.786944 20510 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "75f45538dc704d9a8ec4ff8ad9f3ea5b" candidate_uuid: "7a9daf5d42a84615a6383cfec9a10608" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "974b802141bd4f13beffef2aa03f88f9"
I20260129 03:00:22.759613 23800 raft_consensus.cc:493] T 1299723a4476420ab02fd8bfd7aa0645 P 1c50de89a5a845668696dcb28c3badd6 [term 10 FOLLOWER]: Starting pre-election (detected failure of leader 7a9daf5d42a84615a6383cfec9a10608)
I20260129 03:00:22.733949 23398 raft_consensus.cc:493] T a5a05fbcbbcf4916bbc01fd1a5b5e90d P 1c50de89a5a845668696dcb28c3badd6 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260129 03:00:22.747462 23747 raft_consensus.cc:2804] T fd8311b779af4e1fa1402b2a02a602f2 P 1c50de89a5a845668696dcb28c3badd6 [term 7 FOLLOWER]: Leader election won for term 7
I20260129 03:00:22.792181 20516 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00e79d8f25a34676b1d866eb2068a2b2" candidate_uuid: "7a9daf5d42a84615a6383cfec9a10608" candidate_term: 9 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "974b802141bd4f13beffef2aa03f88f9" is_pre_election: true
I20260129 03:00:22.794646 20516 raft_consensus.cc:2393] T 00e79d8f25a34676b1d866eb2068a2b2 P 974b802141bd4f13beffef2aa03f88f9 [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 7a9daf5d42a84615a6383cfec9a10608 in current term 9: Already voted for candidate 974b802141bd4f13beffef2aa03f88f9 in this term.
I20260129 03:00:22.779228 20588 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7a9a69cd82724b1faed1d10aa091fc4a" candidate_uuid: "197b880fa39d432a9b472845876bfaf0" candidate_term: 9 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "1c50de89a5a845668696dcb28c3badd6" is_pre_election: true
I20260129 03:00:22.779881 20585 raft_consensus.cc:1275] T fcddab6bca684adf8ea032289a00f90e P 1c50de89a5a845668696dcb28c3badd6 [term 11 FOLLOWER]: Refusing update from remote peer 197b880fa39d432a9b472845876bfaf0: Log matching property violated. Preceding OpId in replica: term: 10 index: 6. Preceding OpId from leader: term: 11 index: 7. (index mismatch)
I20260129 03:00:22.795387 20588 raft_consensus.cc:2468] T 7a9a69cd82724b1faed1d10aa091fc4a P 1c50de89a5a845668696dcb28c3badd6 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 197b880fa39d432a9b472845876bfaf0 in term 8.
I20260129 03:00:22.780591 20584 raft_consensus.cc:1275] T 17d6431cc77a491e8464d68db4081a51 P 1c50de89a5a845668696dcb28c3badd6 [term 7 FOLLOWER]: Refusing update from remote peer 197b880fa39d432a9b472845876bfaf0: Log matching property violated. Preceding OpId in replica: term: 4 index: 2. Preceding OpId from leader: term: 7 index: 4. (index mismatch)
I20260129 03:00:22.747121 23693 consensus_queue.cc:1048] T a1a39dcd328e4268ae41b988c66b4a75 P 974b802141bd4f13beffef2aa03f88f9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 2, Time since last communication: 0.048s
I20260129 03:00:22.747794 23812 raft_consensus.cc:493] T 3e2d15f8640b407195210c476e09df91 P 1c50de89a5a845668696dcb28c3badd6 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 7a9daf5d42a84615a6383cfec9a10608)
W20260129 03:00:22.782501 20479 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.211.1:42027, user_credentials={real_user=slave}} blocked reactor thread for 94926.1us
I20260129 03:00:22.755488 23847 raft_consensus.cc:493] T 87d5fe48a44a408aaf03e894a67ecb2f P 7a9daf5d42a84615a6383cfec9a10608 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260129 03:00:22.755370 23262 raft_consensus.cc:493] T 7d9494f5584d4d5db8b9adf1c8fc888b P 1c50de89a5a845668696dcb28c3badd6 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260129 03:00:22.802320 20516 raft_consensus.cc:1275] T 45cc5f3d2688486d9b6dbe7e666dedde P 974b802141bd4f13beffef2aa03f88f9 [term 8 FOLLOWER]: Refusing update from remote peer 7a9daf5d42a84615a6383cfec9a10608: Log matching property violated. Preceding OpId in replica: term: 3 index: 5. Preceding OpId from leader: term: 6 index: 6. (index mismatch)
I20260129 03:00:22.805431 20441 raft_consensus.cc:1217] T 8055232cf0004d40a85f3bf983fe00a9 P 197b880fa39d432a9b472845876bfaf0 [term 7 FOLLOWER]: Deduplicated request from leader. Original: 1.1->[3.2-7.4] Dedup: 3.2->[5.3-7.4]
I20260129 03:00:22.686398 23701 raft_consensus.cc:2804] T 00e79d8f25a34676b1d866eb2068a2b2 P 197b880fa39d432a9b472845876bfaf0 [term 9 FOLLOWER]: Leader pre-election won for term 10
I20260129 03:00:22.810112 20441 pending_rounds.cc:85] T 8055232cf0004d40a85f3bf983fe00a9 P 197b880fa39d432a9b472845876bfaf0: Aborting all ops after (but not including) 2
I20260129 03:00:22.810516 20441 pending_rounds.cc:107] T 8055232cf0004d40a85f3bf983fe00a9 P 197b880fa39d432a9b472845876bfaf0: Aborting uncommitted NO_OP operation due to leader change: 4.3
I20260129 03:00:22.759850 23707 raft_consensus.cc:493] T 3ccac82a01a2445fa5b0ce6c0fcf2295 P 974b802141bd4f13beffef2aa03f88f9 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 7a9daf5d42a84615a6383cfec9a10608)
I20260129 03:00:22.763500 23691 leader_election.cc:290] T a8479d1fe0bd41e5946186e3a6ca8c7a P 974b802141bd4f13beffef2aa03f88f9 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 197b880fa39d432a9b472845876bfaf0 (127.18.211.1:42027), 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3:37735)
I20260129 03:00:22.736155 23602 consensus_queue.cc:1048] T e33a417cf9c54cf5af73a4ee3aac4d3a P 1c50de89a5a845668696dcb28c3badd6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.057s
I20260129 03:00:22.810833 20441 raft_consensus.cc:2889] T 8055232cf0004d40a85f3bf983fe00a9 P 197b880fa39d432a9b472845876bfaf0 [term 7 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260129 03:00:22.814829 20516 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c934d31efd3f4e5195605ddc354d14e1" candidate_uuid: "7a9daf5d42a84615a6383cfec9a10608" candidate_term: 8 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "974b802141bd4f13beffef2aa03f88f9" is_pre_election: true
I20260129 03:00:22.891913 20513 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57584 (request call id 1553) took 1255 ms. Trace:
I20260129 03:00:22.810209 23701 raft_consensus.cc:493] T 00e79d8f25a34676b1d866eb2068a2b2 P 197b880fa39d432a9b472845876bfaf0 [term 9 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260129 03:00:22.816051 23810 raft_consensus.cc:493] T e4d18c35b0a7493e85d0a4b92956a48f P 197b880fa39d432a9b472845876bfaf0 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 974b802141bd4f13beffef2aa03f88f9)
W20260129 03:00:22.848867 23872 log.cc:927] Time spent T 4b1876f9a70b49edb3aa74105e66d3b3 P 974b802141bd4f13beffef2aa03f88f9: Append to log took a long time: real 0.113s user 0.000s sys 0.000s
W20260129 03:00:22.849689 23868 log.cc:927] Time spent T 0340229d521b432580ae926f6062e3f7 P 197b880fa39d432a9b472845876bfaf0: Append to log took a long time: real 0.282s user 0.000s sys 0.001s
I20260129 03:00:22.751730 23838 consensus_queue.cc:1048] T 7a9a69cd82724b1faed1d10aa091fc4a P 974b802141bd4f13beffef2aa03f88f9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.050s
I20260129 03:00:22.791401 23798 raft_consensus.cc:493] T 75b0b6e82ddc4f92b1fe1d94f9a0b73b P 1c50de89a5a845668696dcb28c3badd6 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260129 03:00:22.852244 20438 raft_consensus.cc:1275] T cb4727386a3e48aca5d71cc6c611c197 P 197b880fa39d432a9b472845876bfaf0 [term 6 FOLLOWER]: Refusing update from remote peer 974b802141bd4f13beffef2aa03f88f9: Log matching property violated. Preceding OpId in replica: term: 5 index: 2. Preceding OpId from leader: term: 6 index: 3. (index mismatch)
I20260129 03:00:22.853065 20584 tablet_service.cc:2066] Received LeaderStepDown RPC: tablet_id: "42d27a28d89b409e8807ec0efd97af69"
dest_uuid: "1c50de89a5a845668696dcb28c3badd6"
mode: GRACEFUL
new_leader_uuid: "197b880fa39d432a9b472845876bfaf0"
from {username='slave'} at 127.0.0.1:54860
I20260129 03:00:22.799531 23398 raft_consensus.cc:515] T a5a05fbcbbcf4916bbc01fd1a5b5e90d P 1c50de89a5a845668696dcb28c3badd6 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 03:00:22.790603 23710 raft_consensus.cc:2749] T 36ff33be666a4c82954731691dbcf327 P 974b802141bd4f13beffef2aa03f88f9 [term 6 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260129 03:00:22.933832 20583 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35752 (request call id 1831) took 1278 ms. Trace:
I20260129 03:00:22.788683 23845 raft_consensus.cc:493] T 76f713c3331540d2a5b76164103e9eb1 P 974b802141bd4f13beffef2aa03f88f9 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260129 03:00:22.875003 20405 leader_election.cc:304] T 42d27a28d89b409e8807ec0efd97af69 P 197b880fa39d432a9b472845876bfaf0 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 197b880fa39d432a9b472845876bfaf0, 1c50de89a5a845668696dcb28c3badd6; no voters: 7a9daf5d42a84615a6383cfec9a10608
W20260129 03:00:22.955580 20405 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.211.3:37735, user_credentials={real_user=slave}} blocked reactor thread for 80688us
I20260129 03:00:22.956454 20438 tablet_service.cc:2038] Received Run Leader Election RPC: tablet_id: "e33a417cf9c54cf5af73a4ee3aac4d3a"
dest_uuid: "197b880fa39d432a9b472845876bfaf0"
from {username='slave'} at 127.0.0.1:55676
I20260129 03:00:22.810787 23812 raft_consensus.cc:515] T 3e2d15f8640b407195210c476e09df91 P 1c50de89a5a845668696dcb28c3badd6 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } }
I20260129 03:00:22.822945 20432 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00e79d8f25a34676b1d866eb2068a2b2" candidate_uuid: "974b802141bd4f13beffef2aa03f88f9" candidate_term: 10 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "197b880fa39d432a9b472845876bfaf0" is_pre_election: true
I20260129 03:00:22.904053 20516 raft_consensus.cc:1275] T 75b0b6e82ddc4f92b1fe1d94f9a0b73b P 974b802141bd4f13beffef2aa03f88f9 [term 6 FOLLOWER]: Refusing update from remote peer 7a9daf5d42a84615a6383cfec9a10608: Log matching property violated. Preceding OpId in replica: term: 4 index: 7. Preceding OpId from leader: term: 6 index: 8. (index mismatch)
I20260129 03:00:22.892814 20513 rpcz_store.cc:276] 0129 03:00:21.636932 (+ 0us) service_pool.cc:168] Inserting onto call queue
0129 03:00:21.637077 (+ 145us) service_pool.cc:225] Handling call
0129 03:00:22.285411 (+648334us) raft_consensus.cc:1471] Updating replica for 1 ops
0129 03:00:22.288009 (+ 2598us) raft_consensus.cc:1697] Filling consensus response to leader.
0129 03:00:22.891593 (+603584us) spinlock_profiling.cc:243] Waited 648 ms on lock 0x7b5400228b50. stack: 00007f8fe044e373 00007f8fe044e270 00007f8fdfda0574 00007f8fe9a5c6b4 00007f8fe9a5c649 00007f8fe34975a1 00007f8fe352d52d 00007f8fe84e1f49 00007f8fe1a6ed34 00007f8fe1a6ecc2 00007f8fe1a6ebf1 00007f8fe1a6eb6c 00007f8fe1a6de22 00007f8fe186d34c 00007f8fe186c786 00007f8fe186c12f
0129 03:00:22.891836 (+ 243us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1490328960}
I20260129 03:00:22.789502 23836 consensus_queue.cc:1048] T c891d2bff4ce4568861bbc992e91fbb0 P 197b880fa39d432a9b472845876bfaf0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 5, Time since last communication: 0.096s
I20260129 03:00:22.922605 20584 raft_consensus.cc:606] T 42d27a28d89b409e8807ec0efd97af69 P 1c50de89a5a845668696dcb28c3badd6 [term 8 FOLLOWER]: Received request to transfer leadership to TS 197b880fa39d432a9b472845876bfaf0
I20260129 03:00:22.978317 20584 raft_consensus.cc:614] T 42d27a28d89b409e8807ec0efd97af69 P 1c50de89a5a845668696dcb28c3badd6 [term 8 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260129 03:00:22.854918 20440 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e7d83d4ccf684e76acb29a0dc844e1b8" candidate_uuid: "974b802141bd4f13beffef2aa03f88f9" candidate_term: 9 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "197b880fa39d432a9b472845876bfaf0" is_pre_election: true
I20260129 03:00:22.979424 20440 raft_consensus.cc:2468] T e7d83d4ccf684e76acb29a0dc844e1b8 P 197b880fa39d432a9b472845876bfaf0 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 974b802141bd4f13beffef2aa03f88f9 in term 8.
I20260129 03:00:22.886121 23674 raft_consensus.cc:493] T 368953c9102643b2a21f99af36b51905 P 197b880fa39d432a9b472845876bfaf0 [term 10 FOLLOWER]: Starting pre-election (detected failure of leader 974b802141bd4f13beffef2aa03f88f9)
I20260129 03:00:22.792438 23800 raft_consensus.cc:515] T 1299723a4476420ab02fd8bfd7aa0645 P 1c50de89a5a845668696dcb28c3badd6 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } }
I20260129 03:00:22.873502 23680 raft_consensus.cc:493] T 4f8f3a38f6e24ba4857875181f447442 P 1c50de89a5a845668696dcb28c3badd6 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 974b802141bd4f13beffef2aa03f88f9)
I20260129 03:00:22.933050 20586 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35752 (request call id 1830) took 1388 ms. Trace:
I20260129 03:00:22.938721 20583 rpcz_store.cc:276] 0129 03:00:21.655706 (+ 0us) service_pool.cc:168] Inserting onto call queue
0129 03:00:22.350493 (+694787us) service_pool.cc:225] Handling call
0129 03:00:22.350803 (+ 310us) raft_consensus.cc:1471] Updating replica for 1 ops
0129 03:00:22.360471 (+ 9668us) raft_consensus.cc:1534] Early marking committed up to index 0
0129 03:00:22.360501 (+ 30us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0129 03:00:22.360976 (+ 475us) log.cc:844] Serialized 26 byte log entry
0129 03:00:22.615381 (+254405us) raft_consensus.cc:1645] Marking committed up to 5
0129 03:00:22.615418 (+ 37us) raft_consensus.cc:1697] Filling consensus response to leader.
0129 03:00:22.615477 (+ 59us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0129 03:00:22.933574 (+318097us) raft_consensus.cc:1685] finished
0129 03:00:22.933604 (+ 30us) raft_consensus.cc:1691] UpdateReplicas() finished
0129 03:00:22.933760 (+ 156us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":5665,"raft.queue_time_us":7889,"spinlock_wait_cycles":17768320,"thread_start_us":254131,"threads_started":1,"wal-append.queue_time_us":570479}
I20260129 03:00:22.937697 21135 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00e79d8f25a34676b1d866eb2068a2b2" candidate_uuid: "974b802141bd4f13beffef2aa03f88f9" candidate_term: 10 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "7a9daf5d42a84615a6383cfec9a10608" is_pre_election: true
I20260129 03:00:22.991353 20586 rpcz_store.cc:276] 0129 03:00:21.544875 (+ 0us) service_pool.cc:168] Inserting onto call queue
0129 03:00:22.349913 (+805038us) service_pool.cc:225] Handling call
0129 03:00:22.350180 (+ 267us) raft_consensus.cc:1471] Updating replica for 1 ops
0129 03:00:22.350578 (+ 398us) raft_consensus.cc:1534] Early marking committed up to index 4
0129 03:00:22.350606 (+ 28us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0129 03:00:22.351073 (+ 467us) log.cc:844] Serialized 26 byte log entry
0129 03:00:22.606631 (+255558us) raft_consensus.cc:1645] Marking committed up to 4
0129 03:00:22.606671 (+ 40us) raft_consensus.cc:1697] Filling consensus response to leader.
0129 03:00:22.606733 (+ 62us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0129 03:00:22.932815 (+326082us) raft_consensus.cc:1685] finished
0129 03:00:22.932837 (+ 22us) raft_consensus.cc:1691] UpdateReplicas() finished
0129 03:00:22.932985 (+ 148us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":25996,"raft.queue_time_us":2534,"raft.run_cpu_time_us":112,"raft.run_wall_time_us":9887,"thread_start_us":255223,"threads_started":1,"wal-append.queue_time_us":579019}
I20260129 03:00:22.937196 21133 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "75f45538dc704d9a8ec4ff8ad9f3ea5b" candidate_uuid: "974b802141bd4f13beffef2aa03f88f9" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "7a9daf5d42a84615a6383cfec9a10608" is_pre_election: true
I20260129 03:00:22.885679 23691 raft_consensus.cc:493] T 96609ac68bdf4993a12e1c23355ca013 P 974b802141bd4f13beffef2aa03f88f9 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 7a9daf5d42a84615a6383cfec9a10608)
I20260129 03:00:22.883805 22645 raft_consensus.cc:493] T 0340229d521b432580ae926f6062e3f7 P 197b880fa39d432a9b472845876bfaf0 [term 9 FOLLOWER]: Starting pre-election (detected failure of leader 974b802141bd4f13beffef2aa03f88f9)
I20260129 03:00:22.956616 20405 leader_election.cc:304] T a0fcc58d6fb6473cafbd9d34d5195269 P 197b880fa39d432a9b472845876bfaf0 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 197b880fa39d432a9b472845876bfaf0, 1c50de89a5a845668696dcb28c3badd6; no voters: 974b802141bd4f13beffef2aa03f88f9
I20260129 03:00:22.902853 23701 raft_consensus.cc:3060] T 00e79d8f25a34676b1d866eb2068a2b2 P 197b880fa39d432a9b472845876bfaf0 [term 9 FOLLOWER]: Advancing to term 10
I20260129 03:00:22.954241 20435 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a8479d1fe0bd41e5946186e3a6ca8c7a" candidate_uuid: "974b802141bd4f13beffef2aa03f88f9" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "197b880fa39d432a9b472845876bfaf0" is_pre_election: true
I20260129 03:00:23.001044 20435 raft_consensus.cc:2468] T a8479d1fe0bd41e5946186e3a6ca8c7a P 197b880fa39d432a9b472845876bfaf0 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 974b802141bd4f13beffef2aa03f88f9 in term 6.
I20260129 03:00:22.751626 23766 raft_consensus.cc:515] T 505580c93c694bbb8ea96fdf9b17ab46 P 1c50de89a5a845668696dcb28c3badd6 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } }
I20260129 03:00:22.957681 21102 leader_election.cc:304] T 75f45538dc704d9a8ec4ff8ad9f3ea5b P 7a9daf5d42a84615a6383cfec9a10608 [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7a9daf5d42a84615a6383cfec9a10608; no voters: 1c50de89a5a845668696dcb28c3badd6, 974b802141bd4f13beffef2aa03f88f9
W20260129 03:00:22.961054 23874 log.cc:927] Time spent T 36ff33be666a4c82954731691dbcf327 P 1c50de89a5a845668696dcb28c3badd6: Append to log took a long time: real 0.072s user 0.000s sys 0.000s
I20260129 03:00:22.816695 23262 raft_consensus.cc:515] T 7d9494f5584d4d5db8b9adf1c8fc888b P 1c50de89a5a845668696dcb28c3badd6 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
W20260129 03:00:22.965090 21100 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.211.3:37735, user_credentials={real_user=slave}} blocked reactor thread for 173788us
I20260129 03:00:22.816850 23847 raft_consensus.cc:515] T 87d5fe48a44a408aaf03e894a67ecb2f P 7a9daf5d42a84615a6383cfec9a10608 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } }
I20260129 03:00:22.870808 23707 raft_consensus.cc:515] T 3ccac82a01a2445fa5b0ce6c0fcf2295 P 974b802141bd4f13beffef2aa03f88f9 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } }
I20260129 03:00:22.926395 23798 raft_consensus.cc:3060] T 75b0b6e82ddc4f92b1fe1d94f9a0b73b P 1c50de89a5a845668696dcb28c3badd6 [term 6 FOLLOWER]: Advancing to term 7
I20260129 03:00:22.998607 21133 raft_consensus.cc:2393] T 75f45538dc704d9a8ec4ff8ad9f3ea5b P 7a9daf5d42a84615a6383cfec9a10608 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 974b802141bd4f13beffef2aa03f88f9 in current term 6: Already voted for candidate 7a9daf5d42a84615a6383cfec9a10608 in this term.
I20260129 03:00:22.999444 20513 raft_consensus.cc:1275] T 36ff33be666a4c82954731691dbcf327 P 974b802141bd4f13beffef2aa03f88f9 [term 6 FOLLOWER]: Refusing update from remote peer 1c50de89a5a845668696dcb28c3badd6: Log matching property violated. Preceding OpId in replica: term: 4 index: 4. Preceding OpId from leader: term: 6 index: 5. (index mismatch)
W20260129 03:00:23.001753 20407 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260129 03:00:23.005347 20405 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.211.3:37735, user_credentials={real_user=slave}} blocked reactor thread for 48807.2us
I20260129 03:00:23.017547 20405 leader_election.cc:304] T 4b1876f9a70b49edb3aa74105e66d3b3 P 197b880fa39d432a9b472845876bfaf0 [CANDIDATE]: Term 8 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 197b880fa39d432a9b472845876bfaf0; no voters: 1c50de89a5a845668696dcb28c3badd6, 974b802141bd4f13beffef2aa03f88f9
I20260129 03:00:23.017902 23888 raft_consensus.cc:493] T add8d4923d1a489e86882c3a25692b95 P 974b802141bd4f13beffef2aa03f88f9 [term 9 FOLLOWER]: Starting pre-election (detected failure of leader 7a9daf5d42a84615a6383cfec9a10608)
W20260129 03:00:23.005353 21102 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.211.2:34517, user_credentials={real_user=slave}} blocked reactor thread for 47775.8us
I20260129 03:00:23.017905 23847 leader_election.cc:290] T 87d5fe48a44a408aaf03e894a67ecb2f P 7a9daf5d42a84615a6383cfec9a10608 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 197b880fa39d432a9b472845876bfaf0 (127.18.211.1:42027), 974b802141bd4f13beffef2aa03f88f9 (127.18.211.2:34517)
I20260129 03:00:23.018594 23888 raft_consensus.cc:515] T add8d4923d1a489e86882c3a25692b95 P 974b802141bd4f13beffef2aa03f88f9 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } }
I20260129 03:00:23.020046 23629 consensus_queue.cc:1059] T 1ce4ddf5b10e4ae8b193b13b2e5936ee P 197b880fa39d432a9b472845876bfaf0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 }, Status: INVALID_TERM, Last received: 7.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260129 03:00:23.021055 23629 raft_consensus.cc:2749] T 4b1876f9a70b49edb3aa74105e66d3b3 P 197b880fa39d432a9b472845876bfaf0 [term 8 FOLLOWER]: Leader election lost for term 8. Reason: could not achieve majority
I20260129 03:00:23.021683 23629 raft_consensus.cc:3055] T 1ce4ddf5b10e4ae8b193b13b2e5936ee P 197b880fa39d432a9b472845876bfaf0 [term 7 LEADER]: Stepping down as leader of term 7
I20260129 03:00:23.021996 23629 raft_consensus.cc:740] T 1ce4ddf5b10e4ae8b193b13b2e5936ee P 197b880fa39d432a9b472845876bfaf0 [term 7 LEADER]: Becoming Follower/Learner. State: Replica: 197b880fa39d432a9b472845876bfaf0, State: Running, Role: LEADER
I20260129 03:00:23.022163 23262 leader_election.cc:290] T 7d9494f5584d4d5db8b9adf1c8fc888b P 1c50de89a5a845668696dcb28c3badd6 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 197b880fa39d432a9b472845876bfaf0 (127.18.211.1:42027), 974b802141bd4f13beffef2aa03f88f9 (127.18.211.2:34517)
I20260129 03:00:22.991039 23674 raft_consensus.cc:515] T 368953c9102643b2a21f99af36b51905 P 197b880fa39d432a9b472845876bfaf0 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 03:00:23.022727 23629 consensus_queue.cc:260] T 1ce4ddf5b10e4ae8b193b13b2e5936ee P 197b880fa39d432a9b472845876bfaf0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 7.4, Last appended by leader: 4, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 03:00:23.024475 23674 leader_election.cc:290] T 368953c9102643b2a21f99af36b51905 P 197b880fa39d432a9b472845876bfaf0 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 974b802141bd4f13beffef2aa03f88f9 (127.18.211.2:34517), 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3:37735)
I20260129 03:00:23.024358 23602 consensus_queue.cc:1048] T 36ff33be666a4c82954731691dbcf327 P 1c50de89a5a845668696dcb28c3badd6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20260129 03:00:23.005931 23691 raft_consensus.cc:515] T 96609ac68bdf4993a12e1c23355ca013 P 974b802141bd4f13beffef2aa03f88f9 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } attrs { replace: true } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } }
I20260129 03:00:23.027522 23802 raft_consensus.cc:2749] T 75f45538dc704d9a8ec4ff8ad9f3ea5b P 7a9daf5d42a84615a6383cfec9a10608 [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20260129 03:00:22.971401 23865 raft_consensus.cc:2804] T 42d27a28d89b409e8807ec0efd97af69 P 197b880fa39d432a9b472845876bfaf0 [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260129 03:00:23.028570 23865 raft_consensus.cc:493] T 42d27a28d89b409e8807ec0efd97af69 P 197b880fa39d432a9b472845876bfaf0 [term 8 FOLLOWER]: Starting leader election (detected failure of leader 7a9daf5d42a84615a6383cfec9a10608)
I20260129 03:00:23.029063 23865 raft_consensus.cc:3060] T 42d27a28d89b409e8807ec0efd97af69 P 197b880fa39d432a9b472845876bfaf0 [term 8 FOLLOWER]: Advancing to term 9
I20260129 03:00:23.011180 23744 raft_consensus.cc:2804] T a0fcc58d6fb6473cafbd9d34d5195269 P 197b880fa39d432a9b472845876bfaf0 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260129 03:00:23.031594 23744 raft_consensus.cc:493] T a0fcc58d6fb6473cafbd9d34d5195269 P 197b880fa39d432a9b472845876bfaf0 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260129 03:00:23.032012 23744 raft_consensus.cc:3060] T a0fcc58d6fb6473cafbd9d34d5195269 P 197b880fa39d432a9b472845876bfaf0 [term 6 FOLLOWER]: Advancing to term 7
I20260129 03:00:23.034627 21102 leader_election.cc:304] T c934d31efd3f4e5195605ddc354d14e1 P 7a9daf5d42a84615a6383cfec9a10608 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7a9daf5d42a84615a6383cfec9a10608; no voters: 197b880fa39d432a9b472845876bfaf0, 974b802141bd4f13beffef2aa03f88f9
W20260129 03:00:23.010744 20406 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260129 03:00:23.012037 20514 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57584 (request call id 1558) took 1276 ms. Trace:
I20260129 03:00:23.037554 20514 rpcz_store.cc:276] 0129 03:00:21.735724 (+ 0us) service_pool.cc:168] Inserting onto call queue
0129 03:00:21.735935 (+ 211us) service_pool.cc:225] Handling call
0129 03:00:21.753342 (+ 17407us) raft_consensus.cc:1471] Updating replica for 1 ops
0129 03:00:22.303647 (+550305us) raft_consensus.cc:1534] Early marking committed up to index 0
0129 03:00:22.303679 (+ 32us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0129 03:00:22.304184 (+ 505us) log.cc:844] Serialized 26 byte log entry
0129 03:00:22.527114 (+222930us) raft_consensus.cc:1645] Marking committed up to 4
0129 03:00:22.527175 (+ 61us) raft_consensus.cc:1697] Filling consensus response to leader.
0129 03:00:22.527275 (+ 100us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0129 03:00:23.011788 (+484513us) raft_consensus.cc:1685] finished
0129 03:00:23.011812 (+ 24us) raft_consensus.cc:1691] UpdateReplicas() finished
0129 03:00:23.011974 (+ 162us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":549839,"raft.queue_time_us":551007,"raft.run_cpu_time_us":96,"raft.run_wall_time_us":95,"thread_start_us":222471,"threads_started":1,"wal-append.queue_time_us":269826}
I20260129 03:00:23.041224 23800 leader_election.cc:290] T 1299723a4476420ab02fd8bfd7aa0645 P 1c50de89a5a845668696dcb28c3badd6 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 197b880fa39d432a9b472845876bfaf0 (127.18.211.1:42027), 7a9daf5d42a84615a6383cfec9a10608 (127.18.211.4:40439)
I20260129 03:00:23.018589 20405 leader_election.cc:304] T cb4727386a3e48aca5d71cc6c611c197 P 197b880fa39d432a9b472845876bfaf0 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 197b880fa39d432a9b472845876bfaf0, 1c50de89a5a845668696dcb28c3badd6; no voters: 974b802141bd4f13beffef2aa03f88f9
I20260129 03:00:23.044747 23864 raft_consensus.cc:2804] T cb4727386a3e48aca5d71cc6c611c197 P 197b880fa39d432a9b472845876bfaf0 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260129 03:00:23.045146 23864 raft_consensus.cc:493] T cb4727386a3e48aca5d71cc6c611c197 P 197b880fa39d432a9b472845876bfaf0 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260129 03:00:23.045450 23864 raft_consensus.cc:3060] T cb4727386a3e48aca5d71cc6c611c197 P 197b880fa39d432a9b472845876bfaf0 [term 6 FOLLOWER]: Advancing to term 7
I20260129 03:00:23.048215 23836 consensus_queue.cc:1048] T 76f713c3331540d2a5b76164103e9eb1 P 197b880fa39d432a9b472845876bfaf0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260129 03:00:23.048882 23888 leader_election.cc:290] T add8d4923d1a489e86882c3a25692b95 P 974b802141bd4f13beffef2aa03f88f9 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3:37735), 7a9daf5d42a84615a6383cfec9a10608 (127.18.211.4:40439)
I20260129 03:00:23.049197 20589 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35754 (request call id 1924) took 1619 ms. Trace:
I20260129 03:00:23.049896 20405 leader_election.cc:304] T 7a9a69cd82724b1faed1d10aa091fc4a P 197b880fa39d432a9b472845876bfaf0 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 197b880fa39d432a9b472845876bfaf0, 1c50de89a5a845668696dcb28c3badd6; no voters:
I20260129 03:00:23.051844 23707 leader_election.cc:290] T 3ccac82a01a2445fa5b0ce6c0fcf2295 P 974b802141bd4f13beffef2aa03f88f9 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 197b880fa39d432a9b472845876bfaf0 (127.18.211.1:42027), 7a9daf5d42a84615a6383cfec9a10608 (127.18.211.4:40439)
I20260129 03:00:23.052157 23691 leader_election.cc:290] T 96609ac68bdf4993a12e1c23355ca013 P 974b802141bd4f13beffef2aa03f88f9 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 197b880fa39d432a9b472845876bfaf0 (127.18.211.1:42027), 1c50de89a5a845668696dcb28c3badd6 (127.18.211.3:37735), 7a9daf5d42a84615a6383cfec9a10608 (127.18.211.4:40439)
I20260129 03:00:23.053011 23674 raft_consensus.cc:2804] T 7a9a69cd82724b1faed1d10aa091fc4a P 197b880fa39d432a9b472845876bfaf0 [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260129 03:00:23.053399 23674 raft_consensus.cc:493] T 7a9a69cd82724b1faed1d10aa091fc4a P 197b880fa39d432a9b472845876bfaf0 [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260129 03:00:23.053845 23674 raft_consensus.cc:3060] T 7a9a69cd82724b1faed1d10aa091fc4a P 197b880fa39d432a9b472845876bfaf0 [term 8 FOLLOWER]: Advancing to term 9
I20260129 03:00:23.049569 20589 rpcz_store.cc:276] 0129 03:00:21.430246 (+ 0us) service_pool.cc:168] Inserting onto call queue
0129 03:00:21.911435 (+481189us) service_pool.cc:225] Handling call
0129 03:00:21.940913 (+ 29478us) raft_consensus.cc:1471] Updating replica for 0 ops
0129 03:00:22.329911 (+388998us) raft_consensus.cc:1534] Early marking committed up to index 0
0129 03:00:22.329943 (+ 32us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0129 03:00:22.329985 (+ 42us) raft_consensus.cc:1645] Marking committed up to 7
0129 03:00:22.330019 (+ 34us) raft_consensus.cc:1697] Filling consensus response to leader.
0129 03:00:23.048970 (+718951us) spinlock_profiling.cc:243] Waited 389 ms on lock 0x7b540018bfd4. stack: 00007f8fe044e373 00007f8fe044e270 00007f8fdfda0574 00007f8fe9a5c6b4 00007f8fe9a5c649 00007f8fe34975a1 00007f8fe352ea81 00007f8fe352d4be 00007f8fe84e1f49 00007f8fe1a6ed34 00007f8fe1a6ecc2 00007f8fe1a6ebf1 00007f8fe1a6eb6c 00007f8fe1a6de22 00007f8fe186d34c 00007f8fe186c786
0129 03:00:23.049013 (+ 43us) raft_consensus.cc:1691] UpdateReplicas() finished
0129 03:00:23.049144 (+ 131us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":894148480}
I20260129 03:00:23.055117 23686 raft_consensus.cc:3060] T 926a250fa5834e88b96ccecbefbf5086 P 197b880fa39d432a9b472845876bfaf0 [term 8 FOLLOWER]: Advancing to term 9
I20260129 03:00:23.055660 23398 leader_election.cc:290] T a5a05fbcbbcf4916bbc01fd1a5b5e90d P 1c50de89a5a845668696dcb28c3badd6 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 197b880fa39d432a9b472845876bfaf0 (127.18.211.1:42027), 974b802141bd4f13beffef2aa03f88f9 (127.18.211.2:34517)
I20260129 03:00:23.054852 23798 raft_consensus.cc:515] T 75b0b6e82ddc4f92b1fe1d94f9a0b73b P 1c50de89a5a845668696dcb28c3badd6 [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4" port: 40439 } attrs { promote: false } }
I20260129 03:00:23.018052 20433 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55672 (request call id 1863) took 1863 ms. Trace:
I20260129 03:00:23.036027 23825 consensus_queue.cc:1048] T 45cc5f3d2688486d9b6dbe7e666dedde P 7a9daf5d42a84615a6383cfec9a10608 [LEADER]: Connected to new peer: Peer: permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 5, Time since last communication: 0.001s
I20260129 03:00:23.056919 20433 rpcz_store.cc:276] 0129 03:00:21.155026 (+ 0us) service_pool.cc:168] Inserting onto call queue
0129 03:00:21.155302 (+ 276us) service_pool.cc:225] Handling call
0129 03:00:21.155721 (+ 419us) raft_consensus.cc:1471] Updating replica for 1 ops
0129 03:00:21.156585 (+ 864us) raft_consensus.cc:1534] Early marking committed up to index 0
0129 03:00:21.156637 (+ 52us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0129 03:00:22.226175 (+1069538us) spinlock_profiling.cc:243] Waited 5.49 ms on lock 0x7b2c000014c8. stack: 00007f8fe044e373 00007f8fe044e270 00007f8fdfda0574 00007f8fe9a5c6b4 00007f8fe9a5c649 00007f8fe1e2c9b1 00007f8fe1e3bfe2 00007f8fe3569851 00007f8fe352e0ba 00007f8fe352d4be 00007f8fe84e1f49 00007f8fe1a6ed34 00007f8fe1a6ecc2 00007f8fe1a6ebf1 00007f8fe1a6eb6c 00007f8fe1a6de22
0129 03:00:22.229170 (+ 2995us) log.cc:844] Serialized 26 byte log entry
0129 03:00:22.389901 (+160731us) raft_consensus.cc:1645] Marking committed up to 5
0129 03:00:22.389939 (+ 38us) raft_consensus.cc:1697] Filling consensus response to leader.
0129 03:00:22.390004 (+ 65us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0129 03:00:23.009925 (+619921us) raft_consensus.cc:1685] finished
0129 03:00:23.009954 (+ 29us) raft_consensus.cc:1691] UpdateReplicas() finished
0129 03:00:23.017993 (+ 8039us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":399,"raft.queue_time_us":518,"raft.run_cpu_time_us":152,"raft.run_wall_time_us":8670,"spinlock_wait_cycles":12617984,"thread_start_us":143755,"threads_started":1,"wal-append.queue_time_us":207314}
I20260129 03:00:23.049592 23836 consensus_queue.cc:1048] T fcddab6bca684adf8ea032289a00f90e P 197b880fa39d432a9b472845876bfaf0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.007s
I20260129 03:00:23.058924 20514 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7d9494f5584d4d5db8b9adf1c8fc888b" candidate_uuid: "1c50de89a5a845668696dcb28c3badd6" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "974b802141bd4f13beffef2aa03f88f9" is_pre_election: true
I20260129 03:00:23.059445 23836 consensus_queue.cc:1048] T 17d6431cc77a491e8464d68db4081a51 P 197b880fa39d432a9b472845876bfaf0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 2, Time since last communication: 0.000s
I20260129 03:00:23.063146 23744 raft_consensus.cc:515] T a0fcc58d6fb6473cafbd9d34d5195269 P 197b880fa39d432a9b472845876bfaf0 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "1c50de89a5a845668696dcb28c3badd6" member_type: VOTER last_known_addr { host: "127.18.211.3" port: 37735 } }
I20260129 03:00:23.065239 23812 leader_election.cc:290] T 3e2d15f8640b407195210c476e09df91 P 1c50de89a5a845668696dcb28c3badd6 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 974b802141bd4f13beffef2aa03f88f9 (127.18.211.2:34517), 7a9daf5d42a84615a6383cfec9a10608 (127.18.211.4:40439)
W20260129 03:00:23.066603 23688 consensus_meta.cc:287] T 00e79d8f25a34676b1d866eb2068a2b2 P 7a9daf5d42a84615a6383cfec9a10608: Time spent flushing consensus metadata: real 0.725s user 0.010s sys 0.001s
I20260129 03:00:23.067044 23688 raft_consensus.cc:515] T 00e79d8f25a34676b1d866eb2068a2b2 P 7a9daf5d42a84615a6383cfec9a10608 [term 9 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "197b880fa39d432a9b472845876bfaf0" member_type: VOTER last_known_addr { host: "127.18.211.1" port: 42027 } } peers { permanent_uuid: "974b802141bd4f13beffef2aa03f88f9" member_type: VOTER last_known_addr { host: "127.18.211.2" port: 34517 } } peers { permanent_uuid: "7a9daf5d42a84615a6383cfec9a10608" member_type: VOTER last_known_addr { host: "127.18.211.4"